From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5A82C433F5 for ; Thu, 10 Mar 2022 11:10:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 275F48D0003; Thu, 10 Mar 2022 06:10:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 226238D0001; Thu, 10 Mar 2022 06:10:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 115908D0003; Thu, 10 Mar 2022 06:10:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0110.hostedemail.com [216.40.44.110]) by kanga.kvack.org (Postfix) with ESMTP id 004558D0001 for ; Thu, 10 Mar 2022 06:10:55 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B92A79CD50 for ; Thu, 10 Mar 2022 11:10:55 +0000 (UTC) X-FDA: 79228209270.21.5629FD3 Received: from support.corp-email.com (support.corp-email.com [222.73.234.235]) by imf14.hostedemail.com (Postfix) with ESMTP id 42903100020 for ; Thu, 10 Mar 2022 11:10:53 +0000 (UTC) Received: from ([114.119.32.142]) by support.corp-email.com ((D)) with ASMTP (SSL) id FGV00041; Thu, 10 Mar 2022 19:10:41 +0800 Received: from localhost.localdomain (172.16.35.8) by GCY-MBS-28.TCL.com (10.136.3.28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Thu, 10 Mar 2022 19:10:42 +0800 From: Rokudo Yan To: CC: , , , , , , Subject: Re: [fuse-devel] [PATCH] fuse: avoid deadlock when write fuse inode Date: Thu, 10 Mar 2022 19:10:26 +0800 Message-ID: <20220310111026.684924-1-wu-yan@tcl.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [172.16.35.8] X-ClientProxiedBy: GCY-EXS-05.TCL.com (10.74.128.155) To GCY-MBS-28.TCL.com (10.136.3.28) tUid: 20223101910411d16907f6fcc7c4fc6f62671d6c235dc X-Abuse-Reports-To: service@corp-email.com Abuse-Reports-To: service@corp-email.com X-Complaints-To: service@corp-email.com X-Report-Abuse-To: service@corp-email.com X-Rspamd-Queue-Id: 42903100020 X-Stat-Signature: npqkkx17g515i4zxyzd6swh9byk8uzjq Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of wu-yan@tcl.com designates 222.73.234.235 as permitted sender) smtp.mailfrom=wu-yan@tcl.com; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1646910653-334233 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, Miklos The similar issue occurs in our Android device(4G RAM + 3G zram + 8 arm c= ores + kernel-4.14) too. Under the monkey test, kswapd and fuse daemon thread deadlocked when free= pages is extreme low (less than 1/2 of the min watermark), the backtrace of the 2 threads is a= s follows. kswapd try to evict inode to free some memory(blocked at inode_wait_for_writebac= k), and fuse daemon thread handle the fuse inode write request, which is throttled when do direct re= claim in page allocation slow path(blocked at throttle_direct_reclaim). As the __GFP_FS is set, th= e thread is throttled until kswapd free enough pages until watermark ok(check allow_direct_reclaim), = which cause the deadlock. Although the kernel version is 4.14, the same issue exists in the upstrea= m kernel too. kswapd0 D 26485194.538158 157 1287917 23577482 0x1a20840 0x0 157 = 438599862461462 __switch_to+0x134/0x150 __schedule+0xd5c/0x1100 schedule+0x70/0x90 bit_wait+0x14/0x54 __wait_on_bit+0x74/0xe0 inode_wait_for_writeback+0xa0/0xe4 evict+0xa4/0x284 iput+0x25c/0x2ac dentry_unlink_inode+0xd8/0xe4 __dentry_kill+0xe8/0x22c shrink_dentry_list+0x19c/0x3b0 prune_dcache_sb+0x54/0x80 super_cache_scan+0x114/0x164 shrink_slab+0x454/0x528 shrink_node+0x144/0x318 kswapd+0x830/0x9e0 kthread+0x17c/0x18c ret_from_fork+0x10/0x18 0xffffffffffffffff Thread-19 D 7542.719029 2888 24823 5064 0x1404840 0x1000008 24235 4= 38599754021693 __switch_to+0x134/0x150 __schedule+0xd5c/0x1100 schedule+0x70/0x90 try_to_free_pages+0x264/0x4b0 __alloc_pages_nodemask+0x7a4/0x10d0 fuse_copy_fill+0x15c/0x210 fuse_dev_do_read+0x434/0xc24 fuse_dev_splice_read+0x84/0x1d8 SyS_splice+0x67c/0x8bc el0_svc_naked+0x34/0x38 0xffffffffffffffff code snippet: static bool throttle_direct_reclaim(...) { ... /* * If the caller cannot enter the filesystem, it's possible that it * is due to the caller holding an FS lock or performing a journal * transaction in the case of a filesystem like ext[3|4]. In this case, * it is not safe to block on pfmemalloc_wait as kswapd could be * blocked waiting on the same lock. Instead, throttle for up to a * second before continuing. */ if (!(gfp_mask & __GFP_FS)) { wait_event_interruptible_timeout(pgdat->pfmemalloc_wait, allow_direct_reclaim(pgdat), HZ); goto check_pending; } /* Throttle until kswapd wakes the process */ wait_event_killable(zone->zone_pgdat->pfmemalloc_wait, allow_direct_reclaim(pgdat)); ... } Thanks, yanwu On Wed, 24 Mar 2021 16:28:35 +0100 Miklos Szeredi via = wrote: > On what kernel are you seeing this? > I don't see how it can happen on upstream kernels, since there's a >"write_inode_now(inode, 1)" call in fuse_release() and nothing can > dirty the inode after the file has been released. > Thanks, > Miklos >On Tue, Feb 2, 2021 at 5:41 AM Huang Jianan via fuse-devel > wrote: >> >> We found the following deadlock situations in low memory scenarios: >> Thread A Thread B >> - __writeback_single_inode >> - fuse_write_inode >> - fuse_simple_request >> - __fuse_request_send >> - request_wait_answer >> - fuse_dev_splice_read >> - fuse_copy_fill >> - __alloc_pages_direct_reclaim >> - do_shrink_slab >> - super_cache_scan >> - shrink_dentry_list >> - dentry_unlink_inode >> - iput_final >> - inode_wait_for_writeback