From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED492C433EF for ; Thu, 7 Apr 2022 04:26:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4CE906B0071; Thu, 7 Apr 2022 00:26:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 47E0E6B0073; Thu, 7 Apr 2022 00:26:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36C4D6B0074; Thu, 7 Apr 2022 00:26:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0238.hostedemail.com [216.40.44.238]) by kanga.kvack.org (Postfix) with ESMTP id 2831C6B0071 for ; Thu, 7 Apr 2022 00:26:13 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id CF06D8249980 for ; Thu, 7 Apr 2022 04:26:02 +0000 (UTC) X-FDA: 79328795364.22.A88DDAE Received: from mail-ua1-f50.google.com (mail-ua1-f50.google.com [209.85.222.50]) by imf25.hostedemail.com (Postfix) with ESMTP id 6C4BDA0005 for ; Thu, 7 Apr 2022 04:26:02 +0000 (UTC) Received: by mail-ua1-f50.google.com with SMTP id o10so3141680uar.0 for ; Wed, 06 Apr 2022 21:26:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ZzyMjesxZrkAFOB6DXZrDGwnfcNbPaMjKymwklBUkhg=; b=bBlnksobnNuVqb1lJSZBEikWKn0QnbqnyoWc8tn/yYpawolkxbsPXp75oMXMM3jB0x lz76wTi/i95MtZa6WUC/fEzeFzkuyvHICKzTx2HgkwMkwug2BVpD8VMiVgqR16ZYWVvb xXtd4S2jIBgPSFLk0yQmcHlXPk+03y7DyVcMeDORgsfWgw0RoG88z//Mk9yxZk+Fj56w +v6JFkQ3X63htBVUzOQHfyJNA9yL49cW7V8vanIzGs3mgzHJJRC7dM/sfmujnDEnysTh Fxo7iN2poGhN7hqvHMuWUlIxTjWcOxwzga84nwQjaU3nFSUtdwUKrBgY71SUMXoMKy7B AZeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZzyMjesxZrkAFOB6DXZrDGwnfcNbPaMjKymwklBUkhg=; b=xZeXiYIoMMWG/NykgmQC9GGuk+s72BQOoOf+pfVxWMYqAdBOvV6g8yoP3h2MIo9j7B E8Zpbqr5AlLnIrWpCgPlBA+cXoYmxD4qTH6aCPBTKWnp8X2Yz2ZUuBS5r1LnwN4Y60QL 2rtyNAllO+y/pCF/STfzXesN66dtuPH04ojTnmkYakrAboFjhuuuXet1L/HiBrcnKaHJ 2X+JVvo3rSLVD5WrsTF6vCUHgOiW0Fsnecsf4NCaLncwOJz61KNc10irxStV+wFJmHMu tDP6sUko14u/9mRbPu6Dxmv4viYOdmavm+GLuXr2CK/Mff4aqWufuPCcJL7YeSrKijZs f9lg== X-Gm-Message-State: AOAM5301xErt+thoKr65VeamO5BaJhF5AOl2704IO/yU6H75rEwd1ner asDoE0kgWbAXRIbCUpwydkpxii8OQnvy/s8poQw= X-Google-Smtp-Source: ABdhPJx/B2frk/TmaPzK1fHRk6zyCRzDsnUQtuYpFVwPiWeuW5ya0fj7g4a6WE3i5ToSqfdmeneldXBq5WwO3tdQPuo= X-Received: by 2002:a9f:2046:0:b0:35d:bfc:2c9 with SMTP id 64-20020a9f2046000000b0035d0bfc02c9mr74389uam.119.1649305561699; Wed, 06 Apr 2022 21:26:01 -0700 (PDT) MIME-Version: 1.0 References: <673D708E-2DFA-4812-BB63-6A437E0C72EE@oracle.com> <11f319-c9a-4648-bfbb-dc5a83c774@google.com> In-Reply-To: <11f319-c9a-4648-bfbb-dc5a83c774@google.com> From: Mark Hemment Date: Thu, 7 Apr 2022 05:25:49 +0100 Message-ID: Subject: Re: Regression in xfstests on tmpfs-backed NFS exports To: Hugh Dickins Cc: Chuck Lever III , Andrew Morton , Patrice CHOTARD , Mikulas Patocka , Lukas Czerner , Christoph Hellwig , "Darrick J. Wong" , Linux-MM , Linux NFS Mailing List , linux-fsdevel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=googlemail.com header.s=20210112 header.b=bBlnksob; spf=pass (imf25.hostedemail.com: domain of markhemm@googlemail.com designates 209.85.222.50 as permitted sender) smtp.mailfrom=markhemm@googlemail.com; dmarc=pass (policy=quarantine) header.from=googlemail.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 6C4BDA0005 X-Stat-Signature: ek3txoh7z8dri5imkbgx96mr5a9cqip1 X-HE-Tag: 1649305562-29959 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, 7 Apr 2022 at 01:19, Hugh Dickins wrote: > > On Wed, 6 Apr 2022, Chuck Lever III wrote: > > > Good day, Hugh- > > Huh! If you were really wishing me a good day, would you tell me this ;-? > > > > > I noticed that several fsx-related tests in the xfstests suite are > > failing after updating my NFS server to v5.18-rc1. I normally test > > against xfs, ext4, btrfs, and tmpfs exports. tmpfs is the only export > > that sees these new failures: > > [...] > > generic/075 fails almost immediately without any NFS-level errors. > > Likely this is data corruption rather than an overt I/O error. > > That's sad. Thanks for bisecting and reporting. Sorry for the nuisance. > > I suspect this patch is heading for a revert, because I shall not have > time to debug and investigate. Cc'ing fsdevel and a few people who have > an interest in it, to warn of that likely upcoming revert. > > But if it's okay with everyone, please may we leave it in for -rc2? > Given that having it in -rc1 already smoked out another issue (problem > of SetPageUptodate(ZERO_PAGE(0)) without CONFIG_MMU), I think keeping > it in a little longer might smoke out even more. > > The xfstests info above doesn't actually tell very much, beyond that > generic/075 generic/091 generic/112 generic/127, each a test with fsx, > all fall at their first hurdle. If you have time, please rerun and > tar up the results/generic directory (maybe filter just those failing) > and send as attachment. But don't go to any trouble, it's unlikely > that I shall even untar it - it would be mainly to go on record if > anyone has time to look into it later. And, frankly, it's unlikely > to tell us anything more enlightening, than that the data seen was > not as expected: which we do already know. > > I've had no problem with xfstests generic 075,091,112,127 testing > tmpfs here, not before and not in the month or two I've had that > patch in: so it's something in the way that NFS exercises tmpfs > that reveals it. If I had time to duplicate your procedure, I'd be > asking for detailed instructions: but no, I won't have a chance. > > But I can sit here and try to guess. I notice fs/nfsd checks > file->f_op->splice_read, and employs fallback if not available: > if you have time, please try rerunning those xfstests on an -rc1 > kernel, but with mm/shmem.c's .splice_read line commented out. > My guess is that will then pass the tests, and we shall know more. > > What could be going wrong there? I've thought of two possibilities. > A minor, hopefully easily fixed, issue would be if fs/nfsd has > trouble with seeing the same page twice in a row: since tmpfs is > now using the ZERO_PAGE(0) for all pages of a hole, and I think I > caught sight of code which looks to see if the latest page is the > same as the one before. It's easy to imagine that might go wrong. When I worked at Veritas, data corruption over NFS was hit when sending the same page in succession. This was triggered via VxFS's shared page cache, after a file had been dedup'ed. I can't remember all the details (~15yrs ago), but the core issue was skb_can_coalesce() returning a false-positive for the 'same page' case (no check for crossing a page boundary). > A more difficult issue would be, if fsx is racing writes and reads, > in a way that it can guarantee the correct result, but that correct > result is no longer delivered: because the writes go into freshly > allocated tmpfs cache pages, while reads are still delivering > stale ZERO_PAGEs from the pipe. I'm hazy on the guarantees there. > > But unless someone has time to help out, we're heading for a revert. > > Thanks, > Hugh Cheers, Mark