From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E892C5B543 for ; Tue, 10 Jun 2025 12:19:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B8BB6B007B; Tue, 10 Jun 2025 08:19:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 990B96B0089; Tue, 10 Jun 2025 08:19:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87F5A6B0098; Tue, 10 Jun 2025 08:19:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6987F6B007B for ; Tue, 10 Jun 2025 08:19:53 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0B3C0100294 for ; Tue, 10 Jun 2025 12:19:53 +0000 (UTC) X-FDA: 83539397466.10.03C017A Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by imf22.hostedemail.com (Postfix) with ESMTP id 06DD7C000D for ; Tue, 10 Jun 2025 12:19:50 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YgTnuZp6; spf=pass (imf22.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749557991; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eQAN1VoZVMU0LFg5gjayj+lCmiFTFvo/2om6CUAJUKA=; b=HjLptH9eG/i7nweURxDzVUw22USCAcgkXU+AASrcvuD7Tt8t8CiP5EHl4ssqtcpZimNKvJ ihGTl59JoXyEYKWe/CnDtzuZyoCYfE5PkpIz8xuvXKLikpAIkNQ9Qa7gUyBZz2t+v8ZRaK RvneOvPYC3neuARUcz0vFgUVWgdwBNI= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YgTnuZp6; spf=pass (imf22.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749557991; a=rsa-sha256; cv=none; b=wTYbrTr/M/9f3o0mXGFup8m9A0mprf7S4Y3PV77DJMDN+qOpe4yF3KaciolBGrdXBYel6i RDDOSv86P7IAx+bkKLr0SHDLK4DO6SNhTAzlO/N6H60GShXU3kpiPd+Y7+N1hU3DRR1wZ2 uCvMESUvM+73v15rsz8PcO/tEG1bhnM= Received: by mail-lf1-f51.google.com with SMTP id 2adb3069b0e04-5536b9be5f6so3004686e87.1 for ; Tue, 10 Jun 2025 05:19:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749557989; x=1750162789; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:from:to:cc:subject:date:message-id:reply-to; bh=eQAN1VoZVMU0LFg5gjayj+lCmiFTFvo/2om6CUAJUKA=; b=YgTnuZp62goVqGbXj3/tlczNIrjaIsig2F8cAPpB7gGuftcxTy5U0BzkA5+GnSCDyY eP9AbDPuUZE4y62ATtCML5KzOt4Rml3Ia/XkjqYdphjAGxKJehgl1JetPTrQ01YouJCD 9zdNfnsoobPsiVx8vFSuKycHLFCkxv2IL85ftFXHgp79thsq3rmHOJngofZocq+m0WIK buCc99Neu+BTvUOZSR8qFqlsnyB77k1INgHW5OKKeMZIt13dFOvu+RdjHMlq3cyvRsbS viQdfmLnabkrtxvOs5+WXtEGtJ6u/J+akXozI3MW0kjVLfa23XPtQ+cUsHWAhNzrgUS4 0Mag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749557989; x=1750162789; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=eQAN1VoZVMU0LFg5gjayj+lCmiFTFvo/2om6CUAJUKA=; b=jJnY7JHsf1w0fKlbt4m82UlCyqyp0VjPO2JwPXSNtTLi0BVHf35a3lZGoixqtNOSCg qDQTc/3VRMffyvAeflzIobyl958VH75/3LUTlesG0HT0Z4IVZzS+2izgH8w4atAvPhdL j6KDGeQHPwNPjz7Bmjs0/uWk61tBX6+yXB+Tv3s1Om6zqKTm5Ly7Yvjs8VKk+K5iKA9Y EtgwIiyAEMsTq8N522RN4Hw0HdZ3vy0PDhzLP38OSN53X0Oaa8HqePMq2sqBt6n1TsdX OfqOAozS8RUbB/pbrR1Ohb4dI1+uybDjzta2AtihR1LBlDYWX1XLBG82fLQBMixHuSaf qLuQ== X-Forwarded-Encrypted: i=1; AJvYcCWJLjsIzzrNUYnLkN9vdV9dtShm3ixTvswkLxLqGYaPaE4i+2u5/x4L0yGjQHFjybhiHtI6TFBpoA==@kvack.org X-Gm-Message-State: AOJu0YwKx7awxEicq/DEgoTOSdqxhQhenaCIMpX6V0aZnlCxcdX5N6Fq 8aoFX0kwJ7007qmQ7b+fQIia/3mWCAYdOCtkXfTdJwvVWxWyMpitiydn X-Gm-Gg: ASbGncsovXYHC0hd7u5lj7VLjoPQEdWbKIFVtME32+rNGS79B2MpCilfecpIC9sAR5j HWjVmZebnsdXhD7GGEHvxNzZO5VbmqlHGvmhecLqZv+l+Y4NBxXe0Bawv74mZhA1xFePTAF62c5 5iALhbDsYIcmKG1VXnpxL0ViKJf/6FJOl1yLw8WykhjTKlYvt0hlCiJobtnUx/CIUS91p/P3Ptl rL2V2XZgjGk+GSUJvQsmrF8eMTsk15aBi/99UJtSWBxbEMlVKJKHWpC+ImuH7cl+8Hqs6Vrls3J kGbSmPfmm9x41c4EVkegraHuO92iqeL+KPkyazXaDG3lqH2dZ0XQaGp1IPTttPkc+kzbiLQH4g2 b7r0X16UDiu4= X-Google-Smtp-Source: AGHT+IEOAiThqyr/g3IbOy9bU1mIQFiH29X+QkzD7LOn1vfWuf50BiAf8qQ83WrZqQTXXFrnlRL34Q== X-Received: by 2002:a05:6512:ba9:b0:553:29a6:3398 with SMTP id 2adb3069b0e04-55366e349b3mr4481056e87.55.1749557988857; Tue, 10 Jun 2025 05:19:48 -0700 (PDT) Received: from pc636 (host-95-203-1-180.mobileonline.telia.com. [95.203.1.180]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-55367722516sm1497993e87.106.2025.06.10.05.19.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Jun 2025 05:19:48 -0700 (PDT) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Tue, 10 Jun 2025 14:19:45 +0200 To: Joel Fernandes Cc: paulmck@kernel.org, Uladzislau Rezki , Kent Overstreet , syzbot , akpm@linux-foundation.org, josh@joshtriplett.org, linux-bcachefs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, rcu@vger.kernel.org, syzkaller-bugs@googlegroups.com Subject: Re: [syzbot] [rcu?] [bcachefs?] BUG: unable to handle kernel NULL pointer dereference in rcu_core (3) Message-ID: References: <67a2b20a.050a0220.50516.0003.GAE@google.com> <9694d40a-072e-47c2-a950-3b258bbe04f5@paulmck-laptop> <602bb1be-f4a4-4194-803f-856e95711870@paulmck-laptop> <2f762834-d143-4b84-9ab2-8bff4688ae66@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2f762834-d143-4b84-9ab2-8bff4688ae66@nvidia.com> X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 06DD7C000D X-Stat-Signature: hqtzo1gdz1oj6z5eop3ti41dmz8j7zbm X-Rspam-User: X-HE-Tag: 1749557990-433415 X-HE-Meta: U2FsdGVkX1+qL1ZpGB1EVwm/zwwQqBWISzL6NXP354bEP7SrolQzE9aRlaBFuFBfhqnqRgU2HXiwoO9ghc1nrqMPxIF9Cr40qkcKjcrBfUAV9MjOZ3BYHKyJjjwubWBXI+z37YyMtlbyhmRLuVTPRYUrv5YPf1xC5gxoc4PEDQ3gYHg0bLkm/nqKPhdCS2n1nEMstg8At4lLfUGNP8pBzOs2me3F+DTyYS62fVbldNL3JU2WHIuE8O3eOpfRd/1tiDRZ6649Op1RyOvzPrTQG7d5zg1HOLgpqyppYLyr3kZDUirHF5GpkEp4NiZcEY3MKJJd4rSpfFjfPtsv888MAaHPEYhNeC+EOPtqn3/bzc9w8jWrHTujlwnt+MjTwHOd+6unDML9Gic62q3rl4rkase7u6XYmB7slGhTQqJ2sCf8izrqfC09o5bu6Q2143q+AIuaQ8u8JtuapnZkpVNmvAU+qsJAZlSLdyXDEK0qArAUB5Yd1nI7bI9dgF6PAMwBFpoKsYROztOZX094EJ9Ei9mW+sJM0imi0H/cuX08O3PgLcT9YEfJTHia1BajNQhPcA0rCgUvK2mBAwOUZ8Qo1h7YBOjAfdcbO/kUW70S8etXrThe6R1432EHyvcZ2vmHXNMHO1nEDlsbYK/URj7XCsjYIxf9/gYk8pYIo68oXtE1eIKRGfNjcrAMNpeJa1x31jsZ2Rvd2+PrEmRXp4rATRMPPI11bHju1IqcsI6VXaUJnvNnnwEEQk/iXfQmITNi/oN3+62DhaFFe0sN1TEuujDUilDJJ31p1U0IEIceBU7wfjJuCybnHWzvdvYrQdNTvY+pax1doGqicBXkNvD2wkBl5w1qlwsXpjLnljjOLSE+bnpPhWVCnlWPevVzDhTvxuCxEZF4ydMWrxilsr8ygnSKR1mD6W/J7DMwmsZRm2RqD4x4PrqZvLgSbgMslQXIxzROH6YyG6aV+lv2kmH N1l3dBsE MRrdYhIXPu6wWgWblQ7jgu76/oAX/rm33NGA3mxLKhCmb3sJcQpblgAHbAYidcdBu0/6THdEt+hy0oyVq/KJb4u7SJLv2QwLUbUBgtB8LXrLWgZvvAII2QHHYus3VktPQljPj0lC094nfzlzYn6/oipMvAciNXX2e5Y90vCH2Z3qN8B6xxsLQQLRbe3VnsmGS1NCUE2cBC+n1ZvuORqTkjY42JxYwuv4mJaiziRKL3BAFy96MmToaR1LGp1ZxD9NZSlB8rpnfX6yPZDx75FptxazoxB47ZzElS6Qktp5P+pg832U3yF7QmBQeRSdvRjLjOrkkQpbP6bn52DzKr6YYLnNJDj875WSINLH1hoyhG/wb70YXSWoznHgh6lpRoutIjoVIy8FeB6DyJ+e5zAkyHRCXA9w26/bAF24xkYs4HhumPPr9Ro/V1Uk5xsp0XHjeasQqhHTEymgedQ7wBzq2Q5ayYHURIGvCzuMSOQupzUuEbyU0PJOl0qqq3uz+qAefpKMJYuaG2nTKMiWwuwlrXiM6JUtjFCvAJ1CCTxEHjjkQofhRRe/dD5pcAckywmtXUsgJkbAS6RzFtIjPx8RQOQP7sAwpPkANl+/o6rVBHIGbhyJwakGdIKBX6Hfp/OO5HGh7tONjUbRmv4F+Z15OO4yFAWPYaA5Xyxu0Ej2jDbc09HpundXMLyJd3Vmp+h7BM4KINuZ9cLXsYA2HXIAtWmtt0kcib9i7Jo9M46l+q8jNmLl2xUXDGVD3U58tidXsIDXmRPe9Ei2kW1RwAznVtayGtsYGJy5q9bJAmld0W3hYvdFNUa0AJCExhBbPas5eLyTAyYVAHyEiWbUw+lJ9Q0h7Fb6/49prm0sCxcEpzTXj33kXTtj31QHPZO7Y7FlzQZa1mbEhTEqt955E0KwrQ2+bB5LpsuK0juv7oLejT90VT3yttlZOdUUJxGCE5lDTCG+r68VIY5sd6pR5/Huvde/zR+n2 2iUW7yak t3xwtcXAJyhILXUIoVkKSNEtTS/9ZLPy1Gu+Ffu06fc7SAiEF3u7vg5B5Gg7I1B5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jun 09, 2025 at 10:20:58AM -0400, Joel Fernandes wrote: > > > On 6/9/2025 5:47 AM, Paul E. McKenney wrote: > > On Mon, Jun 09, 2025 at 10:35:34AM +0200, Uladzislau Rezki wrote: > >> On Sun, Jun 08, 2025 at 05:25:05PM -0700, Paul E. McKenney wrote: > >>> On Sun, Jun 08, 2025 at 08:23:36PM +0200, Uladzislau Rezki wrote: > >>>> On Sun, Jun 08, 2025 at 11:26:28AM -0400, Kent Overstreet wrote: > >>>>> On Wed, Feb 05, 2025 at 06:56:19AM -0800, Paul E. McKenney wrote: > >>>>>> On Tue, Feb 04, 2025 at 04:34:18PM -0800, syzbot wrote: > >>>>>>> Hello, > >>>>>>> > >>>>>>> syzbot found the following issue on: > >>>>>>> > >>>>>>> HEAD commit: 0de63bb7d919 Merge tag 'pull-fix' of git://git.kernel.org/.. > >>>>>>> git tree: upstream > >>>>>>> console output: https://syzkaller.appspot.com/x/log.txt?x=10faf5f8580000 > >>>>>>> kernel config: https://syzkaller.appspot.com/x/.config?x=1909f2f0d8e641ce > >>>>>>> dashboard link: https://syzkaller.appspot.com/bug?extid=80e5d6f453f14a53383a > >>>>>>> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > >>>>>>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=16b69d18580000 > >>>>>>> > >>>>>>> Downloadable assets: > >>>>>>> disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7feb34a89c2a/non_bootable_disk-0de63bb7.raw.xz > >>>>>>> vmlinux: https://storage.googleapis.com/syzbot-assets/1142009a30a7/vmlinux-0de63bb7.xz > >>>>>>> kernel image: https://storage.googleapis.com/syzbot-assets/5d9e46a8998d/bzImage-0de63bb7.xz > >>>>>>> mounted in repro: https://storage.googleapis.com/syzbot-assets/526692501242/mount_0.gz > >>>>>>> > >>>>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit: > >>>>>>> Reported-by: syzbot+80e5d6f453f14a53383a@syzkaller.appspotmail.com > >>>>>>> > >>>>>>> slab radix_tree_node start ffff88803bf382c0 pointer offset 24 size 576 > >>>>>>> BUG: kernel NULL pointer dereference, address: 0000000000000000 > >>>>>>> #PF: supervisor instruction fetch in kernel mode > >>>>>>> #PF: error_code(0x0010) - not-present page > >>>>>>> PGD 0 P4D 0 > >>>>>>> Oops: Oops: 0010 [#1] PREEMPT SMP KASAN NOPTI > >>>>>>> CPU: 0 UID: 0 PID: 5705 Comm: syz-executor Not tainted 6.14.0-rc1-syzkaller-00020-g0de63bb7d919 #0 > >>>>>>> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 > >>>>>>> RIP: 0010:0x0 > >>>>>>> Code: Unable to access opcode bytes at 0xffffffffffffffd6. > >>>>>>> RSP: 0018:ffffc90000007bd8 EFLAGS: 00010246 > >>>>>>> RAX: dffffc0000000000 RBX: 1ffff110077e705c RCX: 23438dd059a4b100 > >>>>>>> RDX: 0000000000000100 RSI: 0000000000000000 RDI: ffff88803bf382d8 > >>>>>>> RBP: ffffc90000007e10 R08: ffffffff819f146c R09: 1ffff11003f8519a > >>>>>>> R10: dffffc0000000000 R11: 0000000000000000 R12: ffffffff81a6d507 > >>>>>>> R13: ffff88803bf382e0 R14: 0000000000000000 R15: ffff88803bf382d8 > >>>>>>> FS: 0000555567992500(0000) GS:ffff88801fc00000(0000) knlGS:0000000000000000 > >>>>>>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > >>>>>>> CR2: ffffffffffffffd6 CR3: 000000004da38000 CR4: 0000000000352ef0 > >>>>>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > >>>>>>> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > >>>>>>> Call Trace: > >>>>>>> > >>>>>>> rcu_do_batch kernel/rcu/tree.c:2546 [inline] > >>>>>> > >>>>>> The usual way that this happens is that someone clobbers the rcu_head > >>>>>> structure of something that has been passed to call_rcu(). The most > >>>>>> popular way of clobbering this structure is to pass the same something to > >>>>>> call_rcu() twice in a row, but other creative arrangements are possible. > >>>>>> > >>>>>> Building your kernel with CONFIG_DEBUG_OBJECTS_RCU_HEAD=y can usually > >>>>>> spot invoking call_rcu() twice in a row. > >>>>> > >>>>> I don't think it's that - syzbot's .config already has that enabled. > >>>>> KASAN, too. > >>>>> > >>>>> And the only place we do call_rcu() is from rcu_pending.c, where we've > >>>>> got a rearming rcu callback - but we track whether it's outstanding, and > >>>>> we do all relevant operations with a lock held. > >>>>> > >>>>> And we only use rcu_pending.c with SRCU, not regular RCU. > >>>>> > >>>>> We do use kfree_rcu() in a few places (all boring, I expect), but that > >>>>> doesn't (generally?) use the rcu callback list. > >>>>> > >>>> Right, kvfree_rcu() does not intersect with regular callbacks, it has > >>>> its own path. > >>>> > >>>> It looks like the problem is here: > >>>> > >>>> > >>>> f = rhp->func; > >>>> debug_rcu_head_callback(rhp); > >>>> WRITE_ONCE(rhp->func, (rcu_callback_t)0L); > >>>> f(rhp); > >>>> > >>>> > >>>> we do not check if callback, "f", is a NULL. If it is, the kernel bug > >>>> is triggered right away. For example: > >>>> > >>>> call_rcu(&rh, NULL); > >>>> > >>>> @Paul, do you think it makes sense to narrow callers which apparently > >>>> pass NULL as a callback? To me it seems the case of this bug. But we > >>>> do not know the source. > >>>> > >>>> It would give at least a stack-trace of caller which passes a NULL. > >>> > >>> Adding a check for NULL func passed to __call_rcu_common(), you mean? > >>> > >> Yes. Currently there is no any check. So passing a NULL just triggers > >> kernel panic. > >> > >>> > >>> That wouldn't hurt, and would either (as you say) catch the culprit > >>> or show that the problem is elsewhere. > >>> > >> I can add it then and send out the patch if no objections. > > > > No objections from me! > > Me neither! And I can push that into an -rc release as well once I have it > (since it is related to a potential bug). > I will prepare it and send out today. -- Uladzislau Rezki