From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 651E7C433DB for ; Wed, 24 Feb 2021 17:04:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D015864F04 for ; Wed, 24 Feb 2021 17:04:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D015864F04 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3AF246B006C; Wed, 24 Feb 2021 12:04:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 35EE56B0070; Wed, 24 Feb 2021 12:04:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2739A6B0071; Wed, 24 Feb 2021 12:04:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0059.hostedemail.com [216.40.44.59]) by kanga.kvack.org (Postfix) with ESMTP id 0E4076B006C for ; Wed, 24 Feb 2021 12:04:01 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C771A824C447 for ; Wed, 24 Feb 2021 17:04:00 +0000 (UTC) X-FDA: 77853783840.25.A02FBE3 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf18.hostedemail.com (Postfix) with ESMTP id 0E87720003AA for ; Wed, 24 Feb 2021 17:03:59 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 1F1BEADDB; Wed, 24 Feb 2021 17:03:52 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id AF7841E14EE; Wed, 24 Feb 2021 18:03:50 +0100 (CET) Date: Wed, 24 Feb 2021 18:03:50 +0100 From: Jan Kara To: syzbot Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, syzkaller-bugs@googlegroups.com, viro@zeniv.linux.org.uk, linux-mm@kvack.org Subject: Re: possible deadlock in evict Message-ID: <20210224170350.GC849@quack2.suse.cz> References: <000000000000de58b305bb355903@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <000000000000de58b305bb355903@google.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 0E87720003AA X-Stat-Signature: 4uhyiw4g4kyaienukdocpj9xo5jm4egh Received-SPF: none (suse.cz>: No applicable sender policy available) receiver=imf18; identity=mailfrom; envelope-from=""; helo=mx2.suse.de; client-ip=195.135.220.15 X-HE-DKIM-Result: none/none X-HE-Tag: 1614186239-882307 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi! On Sat 13-02-21 02:38:18, syzbot wrote: > syzbot found the following issue on: > > HEAD commit: c6d8570e Merge tag 'io_uring-5.11-2021-02-12' of git://git.. > git tree: upstream > console output: https://syzkaller.appspot.com/x/log.txt?x=123a4be2d00000 > kernel config: https://syzkaller.appspot.com/x/.config?x=bec717fd4ac4bf03 > dashboard link: https://syzkaller.appspot.com/bug?extid=1b2c6989ec12e467d65c > > Unfortunately, I don't have any reproducer for this issue yet. > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > Reported-by: syzbot+1b2c6989ec12e467d65c@syzkaller.appspotmail.com > > ====================================================== > WARNING: possible circular locking dependency detected > 5.11.0-rc7-syzkaller #0 Not tainted > ------------------------------------------------------ > kswapd0/2232 is trying to acquire lock: > ffff88801f552650 (sb_internal){.+.+}-{0:0}, at: evict+0x2ed/0x6b0 fs/inode.c:577 > > but task is already holding lock: > ffffffff8be89240 (fs_reclaim){+.+.}-{0:0}, at: __fs_reclaim_acquire+0x0/0x30 mm/page_alloc.c:5195 > > which lock already depends on the new lock. So this is an interesting problem. The stacktrace below shows that we can end up with inode reclaim deleting inode. It likely happens like: CPU1 CPU2 prune_icache_sb() ... inode_lru_isolate() if (inode_has_buffers(inode) || inode->i_data.nrpages) { __iget(inode); ... unlink(inode); d_delete(dentry); ... iput(inode) ...going to delete inode... And this introduces interesting lock dependency with filesystem freezing - fs reclaim can block on filesystem being frozen. That inherently means that anything between freeze_super() and thaw_super() that enters fs reclaim is a potential deadlock. But among lot of kernel code in there, there's also userspace running inbetween those so we have no sane way to avoid entering fs reclaim there. So I belive the only sane way of avoiding this deadlock is to really avoiding deleting inode from fs reclaim. And the best idea how to achieve that I have is to simply avoid the 'inode_has_buffers(inode) || inode->i_data.nrpages' branch if we are running in direct reclaim. Any better idea? > stack backtrace: > CPU: 3 PID: 2232 Comm: kswapd0 Not tainted 5.11.0-rc7-syzkaller #0 > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014 > Call Trace: > __dump_stack lib/dump_stack.c:79 [inline] > dump_stack+0x107/0x163 lib/dump_stack.c:120 > check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2117 > check_prev_add kernel/locking/lockdep.c:2868 [inline] > check_prevs_add kernel/locking/lockdep.c:2993 [inline] > validate_chain kernel/locking/lockdep.c:3608 [inline] > __lock_acquire+0x2b26/0x54f0 kernel/locking/lockdep.c:4832 > lock_acquire kernel/locking/lockdep.c:5442 [inline] > lock_acquire+0x1a8/0x720 kernel/locking/lockdep.c:5407 > percpu_down_read include/linux/percpu-rwsem.h:51 [inline] > __sb_start_write include/linux/fs.h:1592 [inline] > sb_start_intwrite include/linux/fs.h:1709 [inline] > ext4_evict_inode+0xe6f/0x1940 fs/ext4/inode.c:241 > evict+0x2ed/0x6b0 fs/inode.c:577 > iput_final fs/inode.c:1653 [inline] > iput.part.0+0x57e/0x810 fs/inode.c:1679 > iput fs/inode.c:1669 [inline] > inode_lru_isolate+0x301/0x4f0 fs/inode.c:778 > __list_lru_walk_one+0x178/0x5c0 mm/list_lru.c:222 > list_lru_walk_one+0x99/0xd0 mm/list_lru.c:266 > list_lru_shrink_walk include/linux/list_lru.h:195 [inline] > prune_icache_sb+0xdc/0x140 fs/inode.c:803 > super_cache_scan+0x38d/0x590 fs/super.c:107 > do_shrink_slab+0x3e4/0x9f0 mm/vmscan.c:511 > shrink_slab+0x16f/0x5d0 mm/vmscan.c:672 > shrink_node_memcgs mm/vmscan.c:2665 [inline] > shrink_node+0x8cc/0x1de0 mm/vmscan.c:2780 > kswapd_shrink_node mm/vmscan.c:3523 [inline] > balance_pgdat+0x745/0x1270 mm/vmscan.c:3681 > kswapd+0x5b1/0xdb0 mm/vmscan.c:3938 > kthread+0x3b1/0x4a0 kernel/kthread.c:292 > ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:296 Honza -- Jan Kara SUSE Labs, CR