From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A94A0C433F5 for ; Sat, 19 Mar 2022 09:36:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 260CF8D0003; Sat, 19 Mar 2022 05:36:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 210098D0001; Sat, 19 Mar 2022 05:36:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D77A8D0003; Sat, 19 Mar 2022 05:36:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0223.hostedemail.com [216.40.44.223]) by kanga.kvack.org (Postfix) with ESMTP id F3F358D0001 for ; Sat, 19 Mar 2022 05:36:26 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A7BE5A45B7 for ; Sat, 19 Mar 2022 09:36:26 +0000 (UTC) X-FDA: 79260630372.31.CF3CB64 Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) by imf28.hostedemail.com (Postfix) with ESMTP id 39204C0030 for ; Sat, 19 Mar 2022 09:36:26 +0000 (UTC) Received: by mail-ot1-f51.google.com with SMTP id x8-20020a9d6288000000b005b22c373759so7152288otk.8 for ; Sat, 19 Mar 2022 02:36:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Uy+1eKTB+JOqH9ugZeFOQ3QEkLKT7rVLVRCW8ChDEEI=; b=CI/DdxTDhHuesgA4pEAjrcH349fvDHT7gnrLCuodcRRzG9/lFd0zpSTiKU2dAl8sIa oLhyCSaplVepH7CSHVtjmTMMptUtS1GvVfNXJa/cvmg1CU1/bVwqKXO2JaTGIe6z9mzP xehpZQtTHAkpAar+OwRyYzfR5FXIMhUiQDLasJ8spbS84/IvqNGD5O6v9m0Cs8wCiYd8 2C2wCk2zrXsAGT1x1v7RqMyZgiCpkx3qaRGLZI8IYpEoKZeJAi9aZkKbWU7Mbx4A46Cv s9nnhm0ejUU5UtXmM1QeyGIoayX8Jsim2iystSAZMPcLr/WUpDumv1TXJKcMDLpk3Bz/ kMbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Uy+1eKTB+JOqH9ugZeFOQ3QEkLKT7rVLVRCW8ChDEEI=; b=VpPPVqBLLVFb1Hv7yo4xYt0TLCKyLu29D6tX6uKes0st952hlEtstvIInQ/OfppyKG saXXWXPdWqF467jmy9uxANcUDkX7m0OfQ/xYLMXOCZrF72KSO3ED54VoWTRk75LQ8h3x yUAjeyAFtaYC2mR0+v3S5YM6nJLYePD6Cx6kxuTYDTR1o546ToD+aBSVbAztZJhWeNuN WVNhXIF1799yrye29BZlo2GzVZw2wr9amh7zr3hb5cuGoymwK6Wd3wRtTmsZa//c9NK5 STQAd7KjpGpK/Bn7zsWkJmbZmtliZY9F5Rxwi/6byoPkCmn+mhlHT4q/QkFjZqvxB+ng KRuw== X-Gm-Message-State: AOAM533lYPB6rxZ67QekCr6ICyX+ehOdhJ8yGOMpBDKskAEQhTxCqjF1 AzkXuo1AUl0o/8CNTkylVTMeo4Nf6IBv0WNnwFo= X-Google-Smtp-Source: ABdhPJxttmgU2KO+KHNJuxjRV3sNDDnjxXwzWANgANwl+IjiPqEEuo/Gvb1gWA56jP1Z4Yfcz3WYRsELGeVWQeG28Zo= X-Received: by 2002:a9d:5cc8:0:b0:5b2:35ae:7ad6 with SMTP id r8-20020a9d5cc8000000b005b235ae7ad6mr4491355oti.275.1647682585482; Sat, 19 Mar 2022 02:36:25 -0700 (PDT) MIME-Version: 1.0 References: <20220319001635.4097742-1-khazhy@google.com> In-Reply-To: From: Amir Goldstein Date: Sat, 19 Mar 2022 11:36:13 +0200 Message-ID: Subject: Re: [PATCH RFC] nfsd: avoid recursive locking through fsnotify To: Trond Myklebust , Jan Kara Cc: "bfields@fieldses.org" , "khazhy@google.com" , "chuck.lever@oracle.com" , "linux-mm@kvack.org" , "linux-nfs@vger.kernel.org" , "linux-kernel@vger.kernel.org" , Jeff Layton Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 39204C0030 X-Rspam-User: Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="CI/DdxTD"; spf=pass (imf28.hostedemail.com: domain of amir73il@gmail.com designates 209.85.210.51 as permitted sender) smtp.mailfrom=amir73il@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: c4guukrbgusnng77whe9j8tb5x9kwi3d X-Rspamd-Server: rspam07 X-HE-Tag: 1647682586-911425 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sat, Mar 19, 2022 at 9:02 AM Trond Myklebust wrote: > > On Fri, 2022-03-18 at 17:16 -0700, Khazhismel Kumykov wrote: > > fsnotify_add_inode_mark may allocate with GFP_KERNEL, which may > > result > > in recursing back into nfsd, resulting in deadlock. See below stack. > > > > nfsd D 0 1591536 2 0x80004080 > > Call Trace: > > __schedule+0x497/0x630 > > schedule+0x67/0x90 > > schedule_preempt_disabled+0xe/0x10 > > __mutex_lock+0x347/0x4b0 > > fsnotify_destroy_mark+0x22/0xa0 > > nfsd_file_free+0x79/0xd0 [nfsd] > > nfsd_file_put_noref+0x7c/0x90 [nfsd] > > nfsd_file_lru_dispose+0x6d/0xa0 [nfsd] > > nfsd_file_lru_scan+0x57/0x80 [nfsd] > > do_shrink_slab+0x1f2/0x330 > > shrink_slab+0x244/0x2f0 > > shrink_node+0xd7/0x490 > > do_try_to_free_pages+0x12f/0x3b0 > > try_to_free_pages+0x43f/0x540 > > __alloc_pages_slowpath+0x6ab/0x11c0 > > __alloc_pages_nodemask+0x274/0x2c0 > > alloc_slab_page+0x32/0x2e0 > > new_slab+0xa6/0x8b0 > > ___slab_alloc+0x34b/0x520 > > kmem_cache_alloc+0x1c4/0x250 > > fsnotify_add_mark_locked+0x18d/0x4c0 > > fsnotify_add_mark+0x48/0x70 > > nfsd_file_acquire+0x570/0x6f0 [nfsd] > > nfsd_read+0xa7/0x1c0 [nfsd] > > nfsd3_proc_read+0xc1/0x110 [nfsd] > > nfsd_dispatch+0xf7/0x240 [nfsd] > > svc_process_common+0x2f4/0x610 [sunrpc] > > svc_process+0xf9/0x110 [sunrpc] > > nfsd+0x10e/0x180 [nfsd] > > kthread+0x130/0x140 > > ret_from_fork+0x35/0x40 > > > > Signed-off-by: Khazhismel Kumykov > > --- > > fs/nfsd/filecache.c | 4 ++++ > > 1 file changed, 4 insertions(+) > > > > Marking this RFC since I haven't actually had a chance to test this, > > we > > we're seeing this deadlock for some customers. > > > > diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c > > index fdf89fcf1a0c..a14760f9b486 100644 > > --- a/fs/nfsd/filecache.c > > +++ b/fs/nfsd/filecache.c > > @@ -121,6 +121,7 @@ nfsd_file_mark_find_or_create(struct nfsd_file > > *nf) > > struct fsnotify_mark *mark; > > struct nfsd_file_mark *nfm = NULL, *new; > > struct inode *inode = nf->nf_inode; > > + unsigned int pflags; > > > > do { > > mutex_lock(&nfsd_file_fsnotify_group->mark_mutex); > > @@ -149,7 +150,10 @@ nfsd_file_mark_find_or_create(struct nfsd_file > > *nf) > > new->nfm_mark.mask = FS_ATTRIB|FS_DELETE_SELF; > > refcount_set(&new->nfm_ref, 1); > > > > + /* fsnotify allocates, avoid recursion back into nfsd > > */ > > + pflags = memalloc_nofs_save(); > > err = fsnotify_add_inode_mark(&new->nfm_mark, inode, > > 0); > > + memalloc_nofs_restore(pflags); > > > > /* > > * If the add was successful, then return the object. > > Isn't that stack trace showing a slab direct reclaim, and not a > filesystem writeback situation? > > Does memalloc_nofs_save()/restore() really fix this problem? It seems > to me that it cannot, particularly since knfsd is not a filesystem, and > so does not ever handle writeback of dirty pages. > Maybe NOFS throttles direct reclaims to the point that the problem is harder to hit? This report came in at good timing for me. It demonstrates an issue I did not predict for "volatile"' fanotify marks [1]. As far as I can tell, nfsd filecache is currently the only fsnotify backend that frees fsnotify marks in memory shrinker. "volatile" fanotify marks would also be evictable in that way, so they would expose fanotify to this deadlock. For the short term, maybe nfsd filecache can avoid the problem by checking mutex_is_locked(&nfsd_file_fsnotify_group->mark_mutex) and abort the shrinker. I wonder if there is a place for a helper mutex_is_locked_by_me()? Jan, A relatively simple fix would be to allocate fsnotify_mark_connector in fsnotify_add_mark() and free it, if a connector already exists for the object. I don't think there is a good reason to optimize away this allocation for the case of a non-first group to set a mark on an object? Thanks, Amir. [1] https://lore.kernel.org/linux-fsdevel/20220307155741.1352405-1-amir73il@gmail.com/