From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7DC3D6B6B4 for ; Wed, 30 Oct 2024 23:04:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7703A6B0088; Wed, 30 Oct 2024 19:04:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 71FD66B0089; Wed, 30 Oct 2024 19:04:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 599606B008C; Wed, 30 Oct 2024 19:04:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 38D486B0088 for ; Wed, 30 Oct 2024 19:04:31 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DD6EE810C4 for ; Wed, 30 Oct 2024 23:04:30 +0000 (UTC) X-FDA: 82731798912.05.D3B9C88 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf11.hostedemail.com (Postfix) with ESMTP id 772C64001A for ; Wed, 30 Oct 2024 23:03:56 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MoXotpW1; spf=pass (imf11.hostedemail.com: domain of boqun.feng@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=boqun.feng@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730329338; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PLJJBAKyQj+SNUNmxz2AeNTsEKRK6TwiIX+0O6pGNdQ=; b=grR57nHdZUKwPW+c/8zZ6J7DyCxV6sToiVbADb9+kVbfJhwIYvh8y1vG9cylwwC56k6IHf TT4t4C4icS+PVBPj3rCGtWsESjwHvnfY8AlEiV0yrmIpq3WfWVWY80hVO/oHOixLdo/3Or iRF1HSLHg0J4dDv+SYYjHU7MOK9MA9M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730329338; a=rsa-sha256; cv=none; b=INvbXV04G+Glw5/k6TgkBlLcGHzooC76UjGrP28tO9oMIra+LCl/8q72bjVnSjM9aAXzpR gdAnkx53eSdHubWMD5bHR5q2h0Lrxx0O2ZpR+PoHLLvsKS3/pN+xK/g/BltbTBCgSrZM7r 38OT/San5T/5q8s83c0IuIgoM4zDZzI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MoXotpW1; spf=pass (imf11.hostedemail.com: domain of boqun.feng@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=boqun.feng@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-46094b68e30so2661211cf.0 for ; Wed, 30 Oct 2024 16:04:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730329468; x=1730934268; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:feedback-id:from:to:cc:subject:date :message-id:reply-to; bh=PLJJBAKyQj+SNUNmxz2AeNTsEKRK6TwiIX+0O6pGNdQ=; b=MoXotpW1R3ASD2WVsp4hto+/O5qH72qpmZZLe3Orkur9xZk/mDSRtg6HKGLmhPlvZd O79GVD7uEcfqdAl74Rf/4L9z/2pEV9F0w4KiExL5JM64U4T60+lvBesn2fAOnj8OjCSM Fd76XVbAJ9gt9uK5vtnQG0366Kz5bZmr9ih65iKG1gVamt+DT9QywT4fw3fWDBjA+D2I BdkAZjY9j90vY2Gl4s9TmzEWXvschFAJ0VTbGJYHy/52MuhJIqiQyRImBzw0Y1e3rcJf YLTOKkp1gx0hessmWGhomt27itTtSdFItBDkRr438VzU3p5qU+/xpHeoK9t0VTc72YV5 Ju/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730329468; x=1730934268; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:feedback-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PLJJBAKyQj+SNUNmxz2AeNTsEKRK6TwiIX+0O6pGNdQ=; b=B0Hbt4cSV3aq1t/2MMa98HPr3aYRiAmMg2IEeHPPEg+Unnd7fRO1OL8/o4DbyH672y f75/poSL5w7nALA/WO+DEVRMcoCUpZFOnTIic2dqHh+JQcnT12FDnoVYZ4K/K665Xp6O XELvgkEFvb1lDc1M4xxrgqTGZ2aCFF6YvOo97YeRBAilTzJwQsDWW8dvgQshPGMge7fM RrHPgLTxr8ksxC+sWmc2v7vajb4VF+uiEg7W+pBEDcMI2KUUlQl6D/c40hcMHdmE6el2 9+jiWm38XDCM1k7lAw1CNfx7RG3SVMvPFeRTCs5PJfBIcdG0O8rizh68aAndcQS5fSJK Sa+Q== X-Forwarded-Encrypted: i=1; AJvYcCXigjx0S6N5o+unQBqRvfjg1aLZafDap9An0Sw3CedxsRc0wdGDsglfXVaQnUfYBlYHQiPnJWRWHA==@kvack.org X-Gm-Message-State: AOJu0YwmnyFXQAy4VXULiWuVeRdW/Pt8oM0TbpGhyhfZ01STKUFUGJAS 81DQRXDgnsYMGKqXnT7cxfXRIECKy5qsNP7CKPIo+Mx6qRc8OuUN X-Google-Smtp-Source: AGHT+IFMAcf+dFBcoD7SGdRauKrebHUvmAG2ECPiTluSnfadQE+H/b/qksVg722bKVaJXipBpbHcLw== X-Received: by 2002:a05:622a:58e:b0:461:148b:1884 with SMTP id d75a77b69052e-4613bfd0547mr250096891cf.11.1730329467740; Wed, 30 Oct 2024 16:04:27 -0700 (PDT) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-462ad19b328sm1220441cf.85.2024.10.30.16.04.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 16:04:27 -0700 (PDT) Received: from phl-compute-03.internal (phl-compute-03.phl.internal [10.202.2.43]) by mailfauth.phl.internal (Postfix) with ESMTP id 72ED31200066; Wed, 30 Oct 2024 19:04:26 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-03.internal (MEProxy); Wed, 30 Oct 2024 19:04:26 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdekgedgtdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnh htshculddquddttddmnecujfgurhepfffhvfevuffkfhggtggujgesthdtredttddtvden ucfhrhhomhepuehoqhhunhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrd gtohhmqeenucggtffrrghtthgvrhhnpeevgfejgfevfeevteegffdvhedtgfekvefgledv teffgeffveeiuedvieethfdugeenucffohhmrghinhepqhgvmhhurdhorhhgnecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqhhunhdomhgv shhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieegqddujeejkeehhe ehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvgdrnhgrmhgv pdhnsggprhgtphhtthhopeduiedpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepvg hlvhgvrhesghhoohhglhgvrdgtohhmpdhrtghpthhtohepvhgsrggskhgrsehsuhhsvgdr tgiipdhrtghpthhtohepphgruhhlmhgtkheskhgvrhhnvghlrdhorhhgpdhrtghpthhtoh eplhhinhhugidqnhgvgihtsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohep lhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtoh epkhgrshgrnhdquggvvhesghhoohhglhgvghhrohhuphhsrdgtohhmpdhrtghpthhtohep lhhinhhugidqmhhmsehkvhgrtghkrdhorhhgpdhrtghpthhtohepshhfrhestggrnhgsrd gruhhughdrohhrghdrrghupdhrtghpthhtohepsghighgvrghshieslhhinhhuthhrohhn ihigrdguvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 30 Oct 2024 19:04:25 -0400 (EDT) Date: Wed, 30 Oct 2024 16:04:24 -0700 From: Boqun Feng To: Marco Elver Cc: Vlastimil Babka , paulmck@kernel.org, linux-next@vger.kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, sfr@canb.auug.org.au, bigeasy@linutronix.de, longman@redhat.com, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org Subject: Re: [BUG] -next lockdep invalid wait context Message-ID: References: <41619255-cdc2-4573-a360-7794fc3614f7@paulmck-laptop> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 772C64001A X-Stat-Signature: z91tmnup3udozkdd37n49fz9cgtjm73t X-HE-Tag: 1730329436-788414 X-HE-Meta: U2FsdGVkX19L9m/gNOfRekGQrfHYue7a6XwW6nPbINKZNOlkpEpbS5Az8PA0YOFOJcWT7YWSzEedkbNfFdPc2Iw6rp3EOc9hgxrCSaAyJo33Q7KX9sUaAqvoREyZGrT3S5+rWX0cKx6/65RmNtOn9YjXMdb2pSGf1l1HLAwVQytgGJZDUQJ7KlVnmRQ5zZCYzrcVDrApo8q1w1lpk6N35PDmDXAhRuBd6beU68oqrV3AMx5VNdSHNFlTzu5lQTgu3Mnjqu99Sd5oMMs4NZaq2+vbmMTATkSruA6GqxvoQXklYUzMc4cspYOs9toxCJZOFtjuRu23vhnCa1ImpEIUL6ER53ui5Jb3VXX4hZ1kHGFgdJkEO+ofYg0uEv1bEdU50pqpnCc/jMA8GtlrvVFP7k40fPSBpMIJCcS/1UlyrQKFFOM45rOepbqCCYKJ+jdmac+F/IThJ5NYosWdTXkqAQETiOJ6TlvDt42Mo5pzzibUmN6ZVCQzzYR5pdwUaDFb4pkPdptRafX0MMR6CFOSVzgMelaQ4j1i83ri0vp+cp9F+9JUJ3ONxmxcJgZY15KK63xdlwO95b+PtFs39fe9PAIzOwEtYu80CYcOf9HOpT2aVcC7nQ7XbGg/QJwxIKujzATY6ukTmUo7ZNirPV7WTkCI/sOXdJVKU1ioaSMxzYktrlGvquVcszeg8w4LbbZq8pcKe2ksRqu6D02512EgOnRA5yYvRsvW0vHdqdFqeoKPymu4YML3DgTsrjea1GREbs7P0PjTFZHySsP10ysF/7RbUWppZQHXaYNz7V0QZEH99VMLysPfTB5x7VpjxS2lgF/Vy4wuHgeLAGqew5CGWwTpauo6jhhq124xyLssGWBlOmD8yvDpaWaYbULJquoeIU81/wUUX23pnPFo7PGJHIl/wRLlTc56M/E7fwLN2n1HLG3Um8Q1PZWofVvfXjtJeJjPvwQGeU3YOqKIPSj dkFv21W2 gWgtO61sVmNe7TCYw0it2vv7CLaDuFIkVFxzj5qogw05hY8ALRhV66A6jZGWX9IFHBVzJXt95GlQKn26HFoLi1BmyxDPyg1v1EzfrdERMcHI6mObvpSJ/6pe/Z3Dh2jiU3YFU2uZ9PVGeHomFTrApT8R7RbWlXfjpQr2spd6x2jn0TmvgI1ZVINg+yTj618Rwx2kbigJubOWsAz0gJKL+g9DGv/MRGwjtx/5YMrurtlrgjQ+RDoOTrx04KgUMwGaB03evgDA7+yk+FJwNwHYaLGq69HkOrGvFNXNrmLd7trNFZmeueDLtnA2uweOqboQn2d+bzvoYGF5oUGpD7VrY91lQpxqsrLU9EUphhSRW5/L+40dDD9hO3xIjvPb8KVEIcNGD/nq40+s9jugTQRlGN+iw4prx28RJrlq6HhHuoBLESnI7hu3fYVcmY6x1OH7IKsB4gHJo22Pe9XBwxaI1SiwMQ8K3FZpYTjpLXpXue4VvhHAFEc0lw2+2z+DpjDW8J2/N X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Oct 30, 2024 at 11:34:08PM +0100, Marco Elver wrote: > On Wed, Oct 30, 2024 at 10:48PM +0100, Vlastimil Babka wrote: > > On 10/30/24 22:05, Paul E. McKenney wrote: > > > Hello! > > > > Hi! > > > > > The next-20241030 release gets the splat shown below when running > > > scftorture in a preemptible kernel. This bisects to this commit: > > > > > > 560af5dc839e ("lockdep: Enable PROVE_RAW_LOCK_NESTING with PROVE_LOCKING") > > > > > > Except that all this is doing is enabling lockdep to find the problem. > > > > > > The obvious way to fix this is to make the kmem_cache structure's > > > cpu_slab field's ->lock be a raw spinlock, but this might not be what > > > we want for real-time response. > > > > But it's a local_lock, not spinlock and it's doing local_lock_irqsave(). I'm > > confused what's happening here, the code has been like this for years now. > > > > > This can be reproduced deterministically as follows: > > > > > > tools/testing/selftests/rcutorture/bin/kvm.sh --torture scf --allcpus --duration 2 --configs PREEMPT --kconfig CONFIG_NR_CPUS=64 --memory 7G --trust-make --kasan --bootargs "scftorture.nthreads=64 torture.disable_onoff_at_boot csdlock_debug=1" > > > > > > I doubt that the number of CPUs or amount of memory makes any difference, > > > but that is what I used. > > > > > > Thoughts? > > > > > > Thanx, Paul > > > > > > ------------------------------------------------------------------------ > > > > > > [ 35.659746] ============================= > > > [ 35.659746] [ BUG: Invalid wait context ] > > > [ 35.659746] 6.12.0-rc5-next-20241029 #57233 Not tainted > > > [ 35.659746] ----------------------------- > > > [ 35.659746] swapper/37/0 is trying to lock: > > > [ 35.659746] ffff8881ff4bf2f0 (&c->lock){....}-{3:3}, at: put_cpu_partial+0x49/0x1b0 > > > [ 35.659746] other info that might help us debug this: > > > [ 35.659746] context-{2:2} > > > [ 35.659746] no locks held by swapper/37/0. > > > [ 35.659746] stack backtrace: > > > [ 35.659746] CPU: 37 UID: 0 PID: 0 Comm: swapper/37 Not tainted 6.12.0-rc5-next-20241029 #57233 > > > [ 35.659746] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 > > > [ 35.659746] Call Trace: > > > [ 35.659746] > > > [ 35.659746] dump_stack_lvl+0x68/0xa0 > > > [ 35.659746] __lock_acquire+0x8fd/0x3b90 > > > [ 35.659746] ? start_secondary+0x113/0x210 > > > [ 35.659746] ? __pfx___lock_acquire+0x10/0x10 > > > [ 35.659746] ? __pfx___lock_acquire+0x10/0x10 > > > [ 35.659746] ? __pfx___lock_acquire+0x10/0x10 > > > [ 35.659746] ? __pfx___lock_acquire+0x10/0x10 > > > [ 35.659746] lock_acquire+0x19b/0x520 > > > [ 35.659746] ? put_cpu_partial+0x49/0x1b0 > > > [ 35.659746] ? __pfx_lock_acquire+0x10/0x10 > > > [ 35.659746] ? __pfx_lock_release+0x10/0x10 > > > [ 35.659746] ? lock_release+0x20f/0x6f0 > > > [ 35.659746] ? __pfx_lock_release+0x10/0x10 > > > [ 35.659746] ? lock_release+0x20f/0x6f0 > > > [ 35.659746] ? kasan_save_track+0x14/0x30 > > > [ 35.659746] put_cpu_partial+0x52/0x1b0 > > > [ 35.659746] ? put_cpu_partial+0x49/0x1b0 > > > [ 35.659746] ? __pfx_scf_handler_1+0x10/0x10 > > > [ 35.659746] __flush_smp_call_function_queue+0x2d2/0x600 > > > > How did we even get to put_cpu_partial directly from flushing smp calls? > > SLUB doesn't use them, it uses queue_work_on)_ for flushing and that > > flushing doesn't involve put_cpu_partial() AFAIK. > > > > I think only slab allocation or free can lead to put_cpu_partial() that > > would mean the backtrace is missing something. And that somebody does a slab > > alloc/free from a smp callback, which I'd then assume isn't allowed? > I think in this particular case, it is queuing a callback for smp_call_function_single() which doesn't have an interrupt handle thread AKAICT, that means the callback will be executed in non-threaded hardirq context, and that makes locks must be taken with real interrupt disabled. Using irq_work might be fine, because it has a handler thread (but the torture is for s(mp) c(all) f(unction), so replacing with irq_work is not really fixing it ;-)). Regards, Boqun > Tail-call optimization is hiding the caller. Compiling with > -fno-optimize-sibling-calls exposes the caller. This gives the full > picture: > > [ 40.321505] ============================= > [ 40.322711] [ BUG: Invalid wait context ] > [ 40.323927] 6.12.0-rc5-next-20241030-dirty #4 Not tainted > [ 40.325502] ----------------------------- > [ 40.326653] cpuhp/47/253 is trying to lock: > [ 40.327869] ffff8881ff9bf2f0 (&c->lock){....}-{3:3}, at: put_cpu_partial+0x48/0x1a0 > [ 40.330081] other info that might help us debug this: > [ 40.331540] context-{2:2} > [ 40.332305] 3 locks held by cpuhp/47/253: > [ 40.333468] #0: ffffffffae6e6910 (cpu_hotplug_lock){++++}-{0:0}, at: cpuhp_thread_fun+0xe0/0x590 > [ 40.336048] #1: ffffffffae6e9060 (cpuhp_state-down){+.+.}-{0:0}, at: cpuhp_thread_fun+0xe0/0x590 > [ 40.338607] #2: ffff8881002a6948 (&root->kernfs_rwsem){++++}-{4:4}, at: kernfs_remove_by_name_ns+0x78/0x100 > [ 40.341454] stack backtrace: > [ 40.342291] CPU: 47 UID: 0 PID: 253 Comm: cpuhp/47 Not tainted 6.12.0-rc5-next-20241030-dirty #4 > [ 40.344807] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 > [ 40.347482] Call Trace: > [ 40.348199] > [ 40.348827] dump_stack_lvl+0x6b/0xa0 > [ 40.349899] dump_stack+0x10/0x20 > [ 40.350850] __lock_acquire+0x900/0x4010 > [ 40.360290] lock_acquire+0x191/0x4f0 > [ 40.364850] put_cpu_partial+0x51/0x1a0 > [ 40.368341] scf_handler+0x1bd/0x290 > [ 40.370590] scf_handler_1+0x4e/0xb0 > [ 40.371630] __flush_smp_call_function_queue+0x2dd/0x600 > [ 40.373142] generic_smp_call_function_single_interrupt+0xe/0x20 > [ 40.374801] __sysvec_call_function_single+0x50/0x280 > [ 40.376214] sysvec_call_function_single+0x6c/0x80 > [ 40.377543] > [ 40.378142] > > And scf_handler does indeed tail-call kfree: > > static void scf_handler(void *scfc_in) > { > [...] > } else { > kfree(scfcp); > } > }