From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59771C4345F for ; Fri, 26 Apr 2024 21:20:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC6B46B0083; Fri, 26 Apr 2024 17:20:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A75DB6B0085; Fri, 26 Apr 2024 17:20:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93DE46B0087; Fri, 26 Apr 2024 17:20:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 765FF6B0083 for ; Fri, 26 Apr 2024 17:20:12 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 26AA941449 for ; Fri, 26 Apr 2024 21:20:12 +0000 (UTC) X-FDA: 82052951064.18.B1DEDA0 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf02.hostedemail.com (Postfix) with ESMTP id 2F73B80012 for ; Fri, 26 Apr 2024 21:20:09 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=fromorbit-com.20230601.gappssmtp.com header.s=20230601 header.b=FHRquKKT; dmarc=pass (policy=quarantine) header.from=fromorbit.com; spf=pass (imf02.hostedemail.com: domain of david@fromorbit.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=david@fromorbit.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714166409; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7tVzNK4uUjr+QTuz4dYOgYNVPVhfLm2jvPaMneK04Hc=; b=pE0hxfN8C1xmm9Ko7tIRIL9cuct2wnfE04yQOVP0D9UMBolwBtGLYSNnZ56Ia/IYyKcAhj 0GiB+SL5uiMlV8ORa1OsZ1UwLm44N01P9p8UMAwk7wu8tA1677yWJ93r2p0P1+v36CQCf4 uyj+jRwzXWe/N1Tz8iSEYV/3VDmS4G4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714166409; a=rsa-sha256; cv=none; b=gnjT9K92Z85TbSMJYUuU6Lo2f5QK3QNFO+PoqjZFHPtyQonwhKQmFmDLFEaMZ2haNGNVPO yd8kXFbxl7yHD4jUjEoWAwRXXfdXE3kOr9R50+qJIwJ9RkTN7U4EmkVf59b3O5ZoAAV5p5 paez4PoXcQVNo8FdSuc3XVgMGRD7j+E= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=fromorbit-com.20230601.gappssmtp.com header.s=20230601 header.b=FHRquKKT; dmarc=pass (policy=quarantine) header.from=fromorbit.com; spf=pass (imf02.hostedemail.com: domain of david@fromorbit.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=david@fromorbit.com Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6ed0e9ccca1so2547929b3a.0 for ; Fri, 26 Apr 2024 14:20:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20230601.gappssmtp.com; s=20230601; t=1714166407; x=1714771207; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=7tVzNK4uUjr+QTuz4dYOgYNVPVhfLm2jvPaMneK04Hc=; b=FHRquKKTAm/FMZ+8r/efk8ERdQB0Py/Za03MQD9rCl7mTu0q3D68p6P5le+lEQ3pLX ajiKmPAWXBKk8I95ZZyAWL5/pnwkzsr5SjeUM43ufQITKyvELyAkuWUpPpzmWQ6pF1yP VTZgE3w+/MT74yQQmoOE5IDahRk3ssBLe/sgnesXZtALZH/E0ugJGrwE8C4lBDum4cNh rggApZFy+hzYi6LJH6feD5FTrSIhlYyctpW8TRlP9knHbWT+nqDILS4nTfIY9DSKEnX0 0HOb3iV1DnmYMxuyHVDJvqJL9lMvAHwvYDodXFlYBbWzxeN+kFss/swuEyQ70KexBp55 XR2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714166407; x=1714771207; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=7tVzNK4uUjr+QTuz4dYOgYNVPVhfLm2jvPaMneK04Hc=; b=QZAAJ3sk7CMnhDridi38uaVR9nGdqXQPt0RTHcnm1jyHOfdOqKDOX2oErlCg7Vj0Ap EjtR9k7CDyh0iuztc9OO1A50ccp5ehAysupD/XuPv43iCn9YhShOvwCPNnUePEYuWaAo Zj+gUz9Hf+8vbnH0GMxDktFwuLS1ndXo3XZffWe3IeaLBllTQ2u9j/35PX+MD+uVd6Pc XPWlBt7CVC/6cuZzjUrH/QfT3BfGlljgvEL26oxRa1qG4MCLk44ofoXn0eqzgdLsMcDh R932W4U2F2NaGZBVmri529SaW17XrbZIypmG0vvMsbo18NHQ374smPWjBlmVVAUuGySw vXdQ== X-Forwarded-Encrypted: i=1; AJvYcCWMx2S3ofVT4uD7pRnCPHDoDPikhjVz8pcILaOFUsPBJPXv+M1zEffFfkVQ4MWpE8CHlLtjFkMAXvdgg91Xn2BNK4Q= X-Gm-Message-State: AOJu0YwvXO24H4pDXwM2g2Xz1Z3hYxoN+vOO2edX8RZIwNpXGT5qpyfe o2mBEdwUYNIY0Ng6vO4uYFe/uKTNvb40+3dvKxOfuWyHYJumJSyGbPFijiW76sY= X-Google-Smtp-Source: AGHT+IHy9Iy0EdDWZlyUCOK3+Kb0ToI9DOIrnuCr0zR65zLK7tOKbRa3HTeFMC4u53qQuIJWpz35JA== X-Received: by 2002:a05:6a00:a0f:b0:6ec:ebf4:3e8a with SMTP id p15-20020a056a000a0f00b006ecebf43e8amr5035898pfh.15.1714166406757; Fri, 26 Apr 2024 14:20:06 -0700 (PDT) Received: from dread.disaster.area (pa49-181-56-237.pa.nsw.optusnet.com.au. [49.181.56.237]) by smtp.gmail.com with ESMTPSA id g14-20020a62f94e000000b006f2d97c3e87sm9259099pfm.125.2024.04.26.14.20.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Apr 2024 14:20:06 -0700 (PDT) Received: from dave by dread.disaster.area with local (Exim 4.96) (envelope-from ) id 1s0SzD-00Bzh6-0B; Sat, 27 Apr 2024 07:20:03 +1000 Date: Sat, 27 Apr 2024 07:20:03 +1000 From: Dave Chinner To: "Darrick J. Wong" Cc: syzbot , chandan.babu@oracle.com, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org, syzkaller-bugs@googlegroups.com, linux-mm@kvack.org, Andrey Ryabinin Subject: Re: [syzbot] [xfs?] possible deadlock in xfs_ilock_data_map_shared Message-ID: References: <00000000000028dd9a0616ecda61@google.com> <20240426163228.GP360919@frogsfrogsfrogs> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240426163228.GP360919@frogsfrogsfrogs> X-Rspamd-Queue-Id: 2F73B80012 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: n456x7jnxsjeoq7fz6yadq9pfit4imwd X-HE-Tag: 1714166408-788074 X-HE-Meta: U2FsdGVkX19GsuqkgkIPVlyNHWksDbNDKbPGKmBgad31AxU0u7S1hozkEmDcm2Pfd7RFx0mS7vfKcGV/l8BYLfJ/0cjdiBNpGJ67eREJnzeL0UIzdEvLDhzbXHfVu9qsPetg5V3UuuKQDQvSu8S8N2b0fgJscxj5iflT2zOJ69fvYYQsBUgkebMM13f6xI3KPrX1sslPUeo5gw53Om/rYRap64wUOznqSmQt9z6VRHgk7v+mxWljw08FrtKErzUvlPgxoz6TKp1VYBnkS0dSHty3qDyhZxwEoGzOgJMriBZrhIr/56ifphT8THGAzuxaSrPGn4ZVQToaH1lUpZ8s1z5M44cACDLM15u4w3XzabuUFzBAruEKioKG3JcrcFuPxq5KuWdG0AWICRpl1F8Fnyou2i73bw+gOQEulL3jGxhdXOBsAK6EyvB9FsuGa577Baw7dKL8Kz1jCrkuprR85N/QwrOvh6p+ZEBYPr28+N1tFZpokq8UnmhqmIfG5zLKTYDykkPB/gI1pZKvbUoUUiGeVZjvoTqjnv76a/CLj8FC/Eoh4NOe2t8FwCSmEi/kvTEmjS/rrZXE0gWage2j0tg/bKqcqOVes/LoW9LJkPX1GD/9bP49cJXn/475kfBg5GGNSwyhCTFxCqnePLqJO6xPyT24t23j8aiOnqvOSyRGYmuSu/B6sQauNK1JUjdLW0Ka7NL3myI/QHe74i2wLeHaSNGMe7aZ6jtLxzfdmGdfANVMmXr24v8jYd4bsjSdsHjT6pgg1OL79B5LGSNvAGh42JiTm6HVdKQy8tyFCMjt46FGPMJRtlS6J2MuqnDNJ9c6ODqrhzkMVPMTA0uJx08yovrJF6VTk4DhUTqaGXB1NLb/Zl3SHqCCzIURKdx1iPNQCl5CofuUTIduf+s/u+VH3ulsr8hAEPNxF38MQVQs6pUy2bgMClSsYFufmYJIUh1R5Nn1RXROm0N0l0q xaeltU40 bieHYZNBSsG3YkbWQ4bzXLxj8cKYOYiy8VkEA0s0UP5p0Ub+1hLYgh0fkS+nHOAG7mWesplDE8T8AOwGYqqDWPWLzYIAXQlpG0KE+OF2aC/B1ebNcpP8Dwoe4zsIRmAxdtlgQn0CAZ+SE7Py2OZy6R09ZtHo0BR6p6z1vxZROKrWtjSYsQjUpAOtpNcCUB4Pjq3kYgT0jitH6SwBKr7eEBf5+v8hUoNJgfnrd0DRu3LFXXXF63jDQKTkyTm7NlhFkwZ5PRxyhDBw122OkbZ+tUd6Ss+iAMUa38B8MozX/AnvpGq9KfAGpTxB3llh/B9PkQtqRIOFSuPqpY6uKargCDZTc7az7tb5i2nJKJhJ/PTNvYFg95E9SqpfSuxUP+zJdgkLIRFKUvRk8CxYACvNr4+li2nmygWjmTK1oYrZV+lnOjkZ8q36Wv/Losn8KaUuu7tDvuEsh4RtOOiz6rpgBYPXRUz4x9csd8GqY0jPZFB965LiKXJS4jpM+dzqmVh0aJfkV+Z8SC3EsNX+BMN+IuFPB3wjP4MRLWjm1oOU7WyngYzkglbUHzGVx0xODOTy5GzpJn+m12kq7dgutrGJNbXhtSiQctaFurRFsRjhhScQ2OgQqmGQ+GDi/Az2Ffrw/gVWSI80RwZ1T2+y5s6kI1Y8c9G6qqHWkmlakNVxJdL2AyTiSR2rrXgf7z29u7k/S44wELh8wDCNymqb+TUuhs85Qfx6Sftm/1uEVnORV5T4YOgnG5BixM/dCrsHwivmoETlNKJctOxTk6kH5UnimoeJso1Up5rjYU0l+5NYIUNDjmMIB4wdBfJnFmxrKKlv9sujqbecYhGCFK0MXvDuHUM8Z0wPAV3lFOnsztf0KPLwAmIjOQqCGrNkfg6/NOcqaEjlob6Hk0HqmO5QO9E3Ya011PBtmNBQ9sU/wAUgkDJv0yLAOrLg2oAs9htJM+f854k/UkUrI4JGaO5IY5TkhKPEhSLVW YMdyU98V PY4B99QPD2aEfGcHAI3R/Mi7dHeOAWN9RU7doMiCtBgwm2ziHeY+B+C6H+tZI+HzMcWi75jxCBDZjFLLHt9QCw4RjRYF/Se9liiQY67JvU7kCd35Ne730Fdl2CzOhBXXOyIvyklzJco5Km7agbFUIOH6LndwdD3qQqB5JxaHrGffsM/0/Keq49qmkPjeV8J/OTCun46rilSbq7tsRoQKHLU0oMEbvzOoj9vYUty7OGEaHbvQHdSlbNMp31yBGLel X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: [cc linux-mm@kvack.org] On Fri, Apr 26, 2024 at 09:32:28AM -0700, Darrick J. Wong wrote: > On Thu, Apr 25, 2024 at 07:46:28AM -0700, syzbot wrote: > > Hello, > > > > syzbot found the following issue on: > > > > HEAD commit: 977b1ef51866 Merge tag 'block-6.9-20240420' of git://git.k.. > > git tree: upstream > > console output: https://syzkaller.appspot.com/x/log.txt?x=126497cd180000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=d239903bd07761e5 > > dashboard link: https://syzkaller.appspot.com/bug?extid=b7e8d799f0ab724876f9 > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > Downloadable assets: > > disk image: https://storage.googleapis.com/syzbot-assets/08d7b6e107aa/disk-977b1ef5.raw.xz > > vmlinux: https://storage.googleapis.com/syzbot-assets/9c5e543ffdcf/vmlinux-977b1ef5.xz > > kernel image: https://storage.googleapis.com/syzbot-assets/04a6d79d2f69/bzImage-977b1ef5.xz > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > Reported-by: syzbot+b7e8d799f0ab724876f9@syzkaller.appspotmail.com > > > > XFS (loop2): Ending clean mount > > ====================================================== > > WARNING: possible circular locking dependency detected > > 6.9.0-rc4-syzkaller-00266-g977b1ef51866 #0 Not tainted > > ------------------------------------------------------ > > syz-executor.2/7915 is trying to acquire lock: > > ffffffff8e42a800 (fs_reclaim){+.+.}-{0:0}, at: might_alloc include/linux/sched/mm.h:312 [inline] > > ffffffff8e42a800 (fs_reclaim){+.+.}-{0:0}, at: slab_pre_alloc_hook mm/slub.c:3746 [inline] > > ffffffff8e42a800 (fs_reclaim){+.+.}-{0:0}, at: slab_alloc_node mm/slub.c:3827 [inline] > > ffffffff8e42a800 (fs_reclaim){+.+.}-{0:0}, at: kmalloc_trace+0x47/0x360 mm/slub.c:3992 > > > > but task is already holding lock: > > ffff888056da8118 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4f/0x70 fs/xfs/xfs_inode.c:114 > > > > which lock already depends on the new lock. > > > > > > the existing dependency chain (in reverse order) is: > > > > -> #1 (&xfs_dir_ilock_class){++++}-{3:3}: > > lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754 > > down_write_nested+0x3d/0x50 kernel/locking/rwsem.c:1695 > > xfs_reclaim_inode fs/xfs/xfs_icache.c:945 [inline] > > xfs_icwalk_process_inode fs/xfs/xfs_icache.c:1631 [inline] > > xfs_icwalk_ag+0x120e/0x1ad0 fs/xfs/xfs_icache.c:1713 > > xfs_icwalk fs/xfs/xfs_icache.c:1762 [inline] > > xfs_reclaim_inodes_nr+0x257/0x360 fs/xfs/xfs_icache.c:1011 > > super_cache_scan+0x411/0x4b0 fs/super.c:227 > > do_shrink_slab+0x707/0x1160 mm/shrinker.c:435 > > shrink_slab+0x1092/0x14d0 mm/shrinker.c:662 > > shrink_one+0x453/0x880 mm/vmscan.c:4774 > > shrink_many mm/vmscan.c:4835 [inline] > > lru_gen_shrink_node mm/vmscan.c:4935 [inline] > > shrink_node+0x3b17/0x4310 mm/vmscan.c:5894 > > kswapd_shrink_node mm/vmscan.c:6704 [inline] > > balance_pgdat mm/vmscan.c:6895 [inline] > > kswapd+0x1882/0x38a0 mm/vmscan.c:7164 > > kthread+0x2f2/0x390 kernel/kthread.c:388 > > ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147 > > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244 > > > > -> #0 (fs_reclaim){+.+.}-{0:0}: > > check_prev_add kernel/locking/lockdep.c:3134 [inline] > > check_prevs_add kernel/locking/lockdep.c:3253 [inline] > > validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869 > > __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137 > > lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754 > > __fs_reclaim_acquire mm/page_alloc.c:3698 [inline] > > fs_reclaim_acquire+0x88/0x140 mm/page_alloc.c:3712 > > might_alloc include/linux/sched/mm.h:312 [inline] > > slab_pre_alloc_hook mm/slub.c:3746 [inline] > > slab_alloc_node mm/slub.c:3827 [inline] > > kmalloc_trace+0x47/0x360 mm/slub.c:3992 > > kmalloc include/linux/slab.h:628 [inline] > > add_stack_record_to_list mm/page_owner.c:177 [inline] > > inc_stack_record_count mm/page_owner.c:219 [inline] > > __set_page_owner+0x561/0x810 mm/page_owner.c:334 > > set_page_owner include/linux/page_owner.h:32 [inline] > > post_alloc_hook+0x1ea/0x210 mm/page_alloc.c:1534 > > prep_new_page mm/page_alloc.c:1541 [inline] > > get_page_from_freelist+0x3410/0x35b0 mm/page_alloc.c:3317 > > __alloc_pages+0x256/0x6c0 mm/page_alloc.c:4575 > > __alloc_pages_bulk+0x729/0xd40 mm/page_alloc.c:4523 > > alloc_pages_bulk_array include/linux/gfp.h:202 [inline] > > xfs_buf_alloc_pages+0x1a7/0x860 fs/xfs/xfs_buf.c:398 > > xfs_buf_find_insert+0x19a/0x1540 fs/xfs/xfs_buf.c:650 > > xfs_buf_get_map+0x149c/0x1ae0 fs/xfs/xfs_buf.c:755 > > xfs_buf_read_map+0x111/0xa60 fs/xfs/xfs_buf.c:860 > > xfs_trans_read_buf_map+0x260/0xad0 fs/xfs/xfs_trans_buf.c:289 > > xfs_da_read_buf+0x2b1/0x470 fs/xfs/libxfs/xfs_da_btree.c:2674 > > xfs_dir3_block_read+0x92/0x1a0 fs/xfs/libxfs/xfs_dir2_block.c:145 > > xfs_dir2_block_lookup_int+0x109/0x7d0 fs/xfs/libxfs/xfs_dir2_block.c:700 > > xfs_dir2_block_lookup+0x19a/0x630 fs/xfs/libxfs/xfs_dir2_block.c:650 > > xfs_dir_lookup+0x633/0xaf0 fs/xfs/libxfs/xfs_dir2.c:399 > > Hm. We've taken an ILOCK in xfs_dir_lookup, and now we're reading a > directory block. We don't have PF_MEMALLOC_NOFS set, nor do we pass > GFP_NOFS when allocating the xfs_buf pages. > > Nothing in this code path sets PF_MEMALLOC_NOFS explicitly, nor does it > create a xfs_trans_alloc_empty, which would set that. Prior to the > removal of kmem_alloc, I think we were much more aggressive about > GFP_NOFS usage. This isn't an XFS bug. The XFS code is correct - the callsite in the buffer cache is using GFP_KERNEL | __GFP_NOLOCKDEP explicitly to avoid these sorts of false positives. Please take a closer look at the stack trace - there's a second memory allocation taking place there way below the XFS memory allocation inside the page owner tracking code itself: static void add_stack_record_to_list(struct stack_record *stack_record, gfp_t gfp_mask) { unsigned long flags; struct stack *stack; /* Filter gfp_mask the same way stackdepot does, for consistency */ gfp_mask &= ~GFP_ZONEMASK; gfp_mask &= (GFP_ATOMIC | GFP_KERNEL); gfp_mask |= __GFP_NOWARN; set_current_in_page_owner(); stack = kmalloc(sizeof(*stack), gfp_mask); if (!stack) { unset_current_in_page_owner(); return; } unset_current_in_page_owner(); ..... Look familiar? That exactly the same gfp mask filtering that the stackdepot code was doing that caused this issue with KASAN: https://lore.kernel.org/linux-xfs/000000000000fbf10e06164f3695@google.com/ Which was fixed with this patch: https://lore.kernel.org/linux-xfs/20240418141133.22950-1-ryabinin.a.a@gmail.com/ Essentially, we're now playing whack-a-mole with internal kernel debug code that doesn't honor __GFP_NOLOCKDEP.... MM-people: can you please do an audit of all the nested allocations that occur inside the public high level allocation API and ensure that they all obey __GFP_NOLOCKDEP so we don't have syzbot keep tripping over them one at a time? -Dave. -- Dave Chinner david@fromorbit.com