From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB8A8C021B2 for ; Fri, 21 Feb 2025 02:46:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 567F96B00B5; Thu, 20 Feb 2025 21:46:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 53DD86B00B6; Thu, 20 Feb 2025 21:46:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 40567280006; Thu, 20 Feb 2025 21:46:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 21CCC6B00B5 for ; Thu, 20 Feb 2025 21:46:49 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 95135C0577 for ; Fri, 21 Feb 2025 02:46:48 +0000 (UTC) X-FDA: 83142414096.24.506FDB8 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf20.hostedemail.com (Postfix) with ESMTP id F406E1C000F for ; Fri, 21 Feb 2025 02:46:46 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=CTIvs9OX; spf=pass (imf20.hostedemail.com: domain of dennis@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=dennis@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740106007; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rGjNckgsSdQ2RlIhf2/1/eS6bT49C0Fb34Z5JiIB9ZQ=; b=ig2gmQiKlGe6vOFumrfzGB6J/r1mvvvyOlnYhBva8K+qz5pEUckd3ybBe363I/VnuOuzYg k8PpMeeFSm68ZyOs92XJ6sQpjHrg1R7CJLXuXWLN3pNR4ehMBZBJ9e5bGnN22wU2Sfxs3j Ckj3r5bF565C9HLZYsZXn4AE4t4FRNs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740106007; a=rsa-sha256; cv=none; b=1a8flSpkHEEJSBwT1QX1YtN+kVS2mfaWE+qIDiGhAHEnWOWmOim3HPV9fymg8LJvFgY8E7 ESNiq/ikoYy8gwVHLGqiYsfiYbbxq4dmrDuMdx5VLLriprDJGDc7PJAqTGQS656HkHP2Tg FqP/fbOZ7RLSC4gV9HdB/N6B6FpJgLo= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=CTIvs9OX; spf=pass (imf20.hostedemail.com: domain of dennis@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=dennis@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id C86735C5B6E; Fri, 21 Feb 2025 02:46:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3B1C7C4CED1; Fri, 21 Feb 2025 02:46:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1740106005; bh=LrI5oWPR8oNl4lN4e7r09ioHzvcbuFW/ylm3nzocmTc=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=CTIvs9OXZlE6pfloivJgY5MW19+rrh40uJplxWFPxrzX53nGdEXYXfJQZX5sNUcgC icF4nDFHBOLOIV/1qzQNv4CGPQ69Bmm7xEr0ANISE13SaKeqHnKyHQTJd5aMGb9xSJ 57AM153EzHQpwwEvbmXizZYoEdeg9WzXd9qCACj5e13mDmQJWtOOEgXL+2AdcU+lil m5bV+G6LqBTT/BmOVqzW8SdgfdaC4F2OwkwWL1geElBj6VrGIzNuchX+ibgML3tTkV gCotTh20AYSPDbyBDGiFGiZfPOIswloMP9rEbgWLIPpHTWA8lZX9x8m2aF/BEslfWh Jp3UuBP180HRA== Date: Thu, 20 Feb 2025 18:46:43 -0800 From: Dennis Zhou To: Kent Overstreet Cc: Vlastimil Babka , Alan Huang , linux-bcachefs@vger.kernel.org, syzbot+fe63f377148a6371a9db@syzkaller.appspotmail.com, linux-mm@kvack.org, Tejun Heo , Christoph Lameter , Michal Hocko Subject: Re: [PATCH] bcachefs: Use alloc_percpu_gfp to avoid deadlock Message-ID: References: <20250212100625.55860-1-mmpgouride@gmail.com> <25FBAAE5-8BC6-41F3-9A6D-65911BA5A5D7@gmail.com> <78d954b5-e33f-4bbc-855b-e91e96278bef@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Queue-Id: F406E1C000F X-Rspamd-Server: rspam07 X-Stat-Signature: q7oxoub5pf3xzkc1bmia8ih9akrq35g1 X-HE-Tag: 1740106006-337898 X-HE-Meta: U2FsdGVkX1/JNygguuiCrTs6oy6Ri2YN0XNQKFMW+yE0Y8EAHaHuWSEaltI0aq2KNbo7VVnqfYoenfoq+QCB2cAB6BTyb0zteh+HmfgIrBDqYnuTArl1jlCHrfgn7sn9nS968xSYjmtHGpzo3o1ll3zWGb+SvdDYrweooyuxy55wTGlPidgYgrkzpqcG2fRkooE+yCTmmr1gCUW+CAhUQz1rDzMV5Xd9uuJuD1DfeO0mxrAB73RItegFs56QuPG51uBZ9cS31OE/WAxX//bt6EyM3Q0ZBkeWt0xtmIWyoBcaBawTcKZ9mADK5DOZc3Way7e/Yp/qjyfvskI5RA8iI8GNvv83OmExA4Avh5aw/0w2ZdxbLLjUUzskE7IEJsM2BDIW1BKhO3o0e7ILoA8AA7SwOp2Z3jntGqLklYb9+k0jzVT2Td5KPe9HI5Ud9+lj6C2G5ZbWWBriAdP9lvQyeL+2ZXtS3eJo42p0xKdPU6jd48+k3OmyxSsQcNm+01oFM6SwJhfhNga4CvYXZCbwRsCSNwSEnRkNwvEtXkQTTBFRt/LvMtbLHf5pZwxfmBYkntS1Pw1DXdNqLLIahnRimza7bJ5qTSWeVNxTLxnhQFHQRHc2JtZQ0LXLOr9qMNboRD906Ax0xwRCfqos1r5UqxgTFJLIhBXfB+JqN0D12lSRS1JPDlk15WHPOaPIyhCFFRby8SNyzrgTCuxhz53hwOOHABGkq2+RKwq0tE/89MEJq9bJULAnH86RK0hmf2v1iarZ2pn6tML5KfqrL4Zp4rSiSpOj30vdCZ676i1goQrjo1dXckASjoapQS7n5TomZPqX8iKODipQBFsybsgclj6xgAW55Fz626vyH08Cy1il94zNQVrhZr34/MeCCg7Un2rQzyEOEdYtytt0uaar3R2QjpafQp0+4snEOOb8k3AsZYZ+oVA6aJjwWpOQ+XhMxLBzuGEGv7CbvCXPljH tRF2C9OT tbitH/nprfmS/1zTsaaUb5EN91Ehpy+4Bl5JuHh/mXpuy/r4cx5NDEeckfNmzC5kV1XZKl8s6oucY74iC0eJxOcySvtvsvT7ZTwTxtGRMJn8TywsuFpcdeOzvbI+0lw2SY9OKX9JXW9MYZeDxk/un07Twyw7zOOnGJwDWoJZkJnJnUeLdg590Ia1kk/2x+57AX9NlTcArnivCZoN1Z6RBwMi8THV/kDsbp16FrxcbaXnNGfb0b8vJMgHF5071PVnfghuFWVVEQaN+NogEuZgTBMFxNpH1P0tg6ZEjR3L0sXhZqkhTsw7U1t3LC0LFo4COadeRjFj51mzTx7zqHScKAB794z7SH2hRAN8F/oNnDor+dblEYFJ6Rw9O9Yz5ml0OqpyBm2IRlk6Yv2cP0DTg4q0B4hZI4XtRrVTHDCeAVs9Ku/QTz8r3bp6GAnUzbxwAgSZHav98sOD4uwyvFRkkmq6g41FdFvPxpfeFnQjYUzCSwGVAjX2vSpZr/A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hello, On Thu, Feb 20, 2025 at 03:37:26PM -0500, Kent Overstreet wrote: > On Thu, Feb 20, 2025 at 06:16:43PM +0100, Vlastimil Babka wrote: > > On 2/20/25 11:57, Alan Huang wrote: > > > Ping > > > > > >> On Feb 12, 2025, at 22:27, Kent Overstreet wrote: > > >> > > >> Adding pcpu people to the CC > > >> > > >> On Wed, Feb 12, 2025 at 06:06:25PM +0800, Alan Huang wrote: > > >>> The cycle: > > >>> > > >>> CPU0: CPU1: > > >>> bc->lock pcpu_alloc_mutex > > >>> pcpu_alloc_mutex bc->lock > > >>> > > >>> Reported-by: syzbot+fe63f377148a6371a9db@syzkaller.appspotmail.com > > >>> Tested-by: syzbot+fe63f377148a6371a9db@syzkaller.appspotmail.com > > >>> Signed-off-by: Alan Huang > > >> > > >> So pcpu_alloc_mutex -> fs_reclaim? > > >> > > >> That's really awkward; seems like something that might invite more > > >> issues. We can apply your fix if we need to, but I want to hear with the > > >> percpu people have to say first. > > >> > > >> ====================================================== > > >> WARNING: possible circular locking dependency detected > > >> 6.14.0-rc2-syzkaller-00039-g09fbf3d50205 #0 Not tainted > > >> ------------------------------------------------------ > > >> syz.0.21/5625 is trying to acquire lock: > > >> ffffffff8ea19608 (pcpu_alloc_mutex){+.+.}-{4:4}, at: pcpu_alloc_noprof+0x293/0x1760 mm/percpu.c:1782 > > >> > > >> but task is already holding lock: > > >> ffff888051401c68 (&bc->lock){+.+.}-{4:4}, at: bch2_btree_node_mem_alloc+0x559/0x16f0 fs/bcachefs/btree_cache.c:804 > > >> > > >> which lock already depends on the new lock. > > >> > > >> > > >> the existing dependency chain (in reverse order) is: > > >> > > >> -> #2 (&bc->lock){+.+.}-{4:4}: > > >> lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5851 > > >> __mutex_lock_common kernel/locking/mutex.c:585 [inline] > > >> __mutex_lock+0x19c/0x1010 kernel/locking/mutex.c:730 > > >> bch2_btree_cache_scan+0x184/0xec0 fs/bcachefs/btree_cache.c:482 > > >> do_shrink_slab+0x72d/0x1160 mm/shrinker.c:437 > > >> shrink_slab+0x1093/0x14d0 mm/shrinker.c:664 > > >> shrink_one+0x43b/0x850 mm/vmscan.c:4868 > > >> shrink_many mm/vmscan.c:4929 [inline] > > >> lru_gen_shrink_node mm/vmscan.c:5007 [inline] > > >> shrink_node+0x37c5/0x3e50 mm/vmscan.c:5978 > > >> kswapd_shrink_node mm/vmscan.c:6807 [inline] > > >> balance_pgdat mm/vmscan.c:6999 [inline] > > >> kswapd+0x20f3/0x3b10 mm/vmscan.c:7264 > > >> kthread+0x7a9/0x920 kernel/kthread.c:464 > > >> ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:148 > > >> ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244 > > >> > > >> -> #1 (fs_reclaim){+.+.}-{0:0}: > > >> lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5851 > > >> __fs_reclaim_acquire mm/page_alloc.c:3853 [inline] > > >> fs_reclaim_acquire+0x88/0x130 mm/page_alloc.c:3867 > > >> might_alloc include/linux/sched/mm.h:318 [inline] > > >> slab_pre_alloc_hook mm/slub.c:4066 [inline] > > >> slab_alloc_node mm/slub.c:4144 [inline] > > >> __do_kmalloc_node mm/slub.c:4293 [inline] > > >> __kmalloc_noprof+0xae/0x4c0 mm/slub.c:4306 > > >> kmalloc_noprof include/linux/slab.h:905 [inline] > > >> kzalloc_noprof include/linux/slab.h:1037 [inline] > > >> pcpu_mem_zalloc mm/percpu.c:510 [inline] > > >> pcpu_alloc_chunk mm/percpu.c:1430 [inline] > > >> pcpu_create_chunk+0x57/0xbc0 mm/percpu-vm.c:338 > > >> pcpu_balance_populated mm/percpu.c:2063 [inline] > > >> pcpu_balance_workfn+0xc4d/0xd40 mm/percpu.c:2200 > > >> process_one_work kernel/workqueue.c:3236 [inline] > > >> process_scheduled_works+0xa66/0x1840 kernel/workqueue.c:3317 > > >> worker_thread+0x870/0xd30 kernel/workqueue.c:3398 > > >> kthread+0x7a9/0x920 kernel/kthread.c:464 > > >> ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:148 > > >> ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244 > > > > Seeing this as part of the chain (fs reclaim from a worker doing > > pcpu_balance_workfn) makes me think Michal's patch could be a fix to this: > > > > https://lore.kernel.org/all/20250206122633.167896-1-mhocko@kernel.org/ > > Thanks for the link - that does look like just the thing. Sorry I missed the first email asking to weigh in. Michal's problem is a little bit different than what's happening here. He's having an issue where a alloc_percpu_gfp(NOFS/NOIO) is considered atomic and failing during probing. This is because we don't have enough percpu memory backed to fulfill the "atomic" requests. Historically we've considered any allocation that's not GFP_KERNEL to be atomic. Here it seems like the alloc_percpu() behind the bc->lock() should have been an "atomic" allocation to prevent the lock cycle? Thanks, Dennis