From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02A1CD6B6DA for ; Wed, 30 Oct 2024 22:34:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90AEC6B0096; Wed, 30 Oct 2024 18:34:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 893706B0098; Wed, 30 Oct 2024 18:34:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 733006B0099; Wed, 30 Oct 2024 18:34:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5436D6B0096 for ; Wed, 30 Oct 2024 18:34:20 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F2AB1121192 for ; Wed, 30 Oct 2024 22:34:19 +0000 (UTC) X-FDA: 82731721674.20.BC9BB36 Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by imf03.hostedemail.com (Postfix) with ESMTP id C72832000C for ; Wed, 30 Oct 2024 22:34:05 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jlktrJqr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of elver@google.com designates 209.85.221.53 as permitted sender) smtp.mailfrom=elver@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730327602; a=rsa-sha256; cv=none; b=VbHO1vwLgikFDozgIKek/jAb54gILONEYuioYepsB4OT492NIAb/FOrwG3+hoiASECKNHI C7hLVBRfIMa5sSQyjbMby/FEfyrK0TEF+Ipjyaidt6tprhVNccy49YQ/LALJeCKoi5LwNx sz0vJJi/ixg0lBstowP7vn6rE6pH+BI= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jlktrJqr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of elver@google.com designates 209.85.221.53 as permitted sender) smtp.mailfrom=elver@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730327602; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ni3lKjy3PY8T0mBefZFxGJ1yDDh/2Xy0r8Ncbm02QOA=; b=U9fSq1p93cJu7kD9oU1uezKeLKtV2Q149w5sjSxg9dam9BXeRpdkZYQ7LfKr3lewv8sKu9 NVB36vO66udlALK+ULO6+IswLA522MfWPiH1B0AuzFm+KTkgcM1KK8jBqBd2hzUwAQUYy2 5OI3VFlEL30sgNPzCeZSTd0r9X57qSs= Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-37d47eff9acso224213f8f.3 for ; Wed, 30 Oct 2024 15:34:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1730327656; x=1730932456; darn=kvack.org; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=Ni3lKjy3PY8T0mBefZFxGJ1yDDh/2Xy0r8Ncbm02QOA=; b=jlktrJqrZDOHiO9+fnzQTIna7+JfCwS8FJBvJh42w61QAmxb18VKKlRac1Oc5V3eqP wmv4vj9mQeKGw1CKSgG8xuwZbwiGkRIW3kA25T7Ow54HKWDp2gbwuzwP+RIisSdWKCEh BMqYKsGUaeYM6qnHsHdcLu+C1xcvZnWtzl21lRvHYlLAitsNE4fGQIkv/cC6Y106ExR+ M+l/9+hYzj2Q+F7kUxKX8srNNp26X5c/Peg3oBYa7GtS868LIXaM2UWoFtflcCTeZfFD unWFMI1i3dq97LWxXTNhxhywPAZdSILDCXJBiF3GuIO/ZltdihkuN9YK+AeTeOfJF5NV yARw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730327656; x=1730932456; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ni3lKjy3PY8T0mBefZFxGJ1yDDh/2Xy0r8Ncbm02QOA=; b=pSng9TP6g0ap/JMSGFSKgtAz6oGazqRFRaYXTVGsHHGrEsA7+cqRgo7gmbY9HBgEC3 Hvh8jExdaGJJJbm3Mb5KtRS0ewqeQGfWd8sFjQt5BvUNCgiATpT35ml6vTX8H+cOkSXu AIGSFSmwLWJ2U1B8/qNIV5oG+X2rFWxPQsmpYwdBGRRkfyehnI6tltaTKmi0/j7CkPD/ c2rH6jb1vhNwUirO4rufd24EtjdsW44myv346r5epZA+zCXTfNg6EsF+XyMM3GnhfqZx 3OT8dcRINWYm4YuQkyrDW4HVDDDYeUnhfwEtQluzdmlo9ZtOcuI2Nmk22rvnLvjztyPQ M6FQ== X-Forwarded-Encrypted: i=1; AJvYcCXe0lKVG5NqCsdcluV3qLlIsOeFMLzIpJV45KzKNk8sE0EUnneaQ1ZiHNFwBak9GUnCH4xyiUmSlA==@kvack.org X-Gm-Message-State: AOJu0Yz6V3Ncgb02J80TJ+YLltjsKS4BLn7/8WqNCrqYWKTqMNOzbLHV D+WAB+iKPDXI5GIumBIzJG9mwCaZPv0UZFMlH7HWrFnxoinxeddLZi8fqp9FGQ== X-Google-Smtp-Source: AGHT+IFmGsbAbkZzs0MbotIQ1EenBGrw3itupW8ZgVwCnUy7Fpt7cnZeTzVFA5Njzc0+8H3RJxO/QA== X-Received: by 2002:a5d:494f:0:b0:37d:4846:3d29 with SMTP id ffacd0b85a97d-38061162e68mr11660878f8f.28.1730327656290; Wed, 30 Oct 2024 15:34:16 -0700 (PDT) Received: from elver.google.com ([2a00:79e0:9c:201:ca43:df8b:ca42:54da]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c10b7b75sm250245f8f.15.2024.10.30.15.34.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 15:34:14 -0700 (PDT) Date: Wed, 30 Oct 2024 23:34:08 +0100 From: Marco Elver To: Vlastimil Babka Cc: paulmck@kernel.org, linux-next@vger.kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, sfr@canb.auug.org.au, bigeasy@linutronix.de, longman@redhat.com, boqun.feng@gmail.com, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org Subject: Re: [BUG] -next lockdep invalid wait context Message-ID: References: <41619255-cdc2-4573-a360-7794fc3614f7@paulmck-laptop> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/2.2.12 (2023-09-09) X-Rspam-User: X-Rspamd-Queue-Id: C72832000C X-Rspamd-Server: rspam01 X-Stat-Signature: zwj57p3jxbbxn8ah9cokcq4ja4mf5mq7 X-HE-Tag: 1730327645-368129 X-HE-Meta: U2FsdGVkX18NsMOFOxctMPI44BSSV+HTPFqR6Ky+OUezLaewZGIrx/KNdarqaE2P/8ki2WjEOjw6s8nw3t/IgLBWnFwOlyTlUDHkhIv9wG5fbpLS068vPf7tka0v4/fYlXK7cFiqsXIPPcPcpL8WnDA2rBCm5ZTQtyl5q1Vp9q0eaLwDRwnTfjD90suCNzYJP+wo36bdqfK6pGTC9saNiAKNPwFvPKzGfwgiJHqge99jpbWgDOZlJoS5eY8YLKXK0+4uXL7/adKALiT9He/coedJBK2dAzT7Jw6bmMQKqhH4d/hw8WGGg8wTI7ZQto3L2gKF8zT+kn/TSVd6FJppYfK1mtLxXHQHAVDp0LT/KjqDoNxrUpBzIG5iJLeU2skk2cvmPmEadgT/QNlDiqwGe8atJ7ZpmktCiDzgJdsPYqzGqvA2IbMcCeJWW5BiURD86AnEA9HRIxYvoLFphZMVEHpaGIb+Kfz/N86e56b9x+82wPxrQNn+5N8Sq8yvLfaezgA6HV+v8f1BA2aCxYATziwJJcQXJii9v0r2ITExzQX8o4p6HsYUUwh6rWtBnC9GXD3KkVtob5pteH0KU/ybzo3QDIwXbGrkNqyldwi5Bkze+IcuBSibOybzYaSnO4g/SrfzQXYfoXxdCXZ2mjk82e6RAVKcQxAh7CoBnt5rtI5/0/+TYqSWD57forg1PjyOki4+/QD72w2zEzHmKs5zTbqF9yi9qgwSWURLwKvRdRhE+tsA+nlfanxSBjgiXgVaGgzNZ3jTfhFaZOTOjxnPyMwvTY9isV1Pz4CYnxrNrC1+suc2YeCxlCXlk1bP9wOOJ1TdGeuM2bwxUWnPMSwZY4q0W3khXb5XZN5nDdIZWSozDYTxikuE4Zkz7f5XzVQu8y3MsgDd87s0hsLmFwMCf8D4gNRMYdcO2qm1N8G9xFhzG5bCxWUH5pBoFEVYdNbclmUcvLV67oUID6FFgFT LIIhN5uP 6SlCOugoXh98WudjYvQM0nPwD4Dcqn5ov3+nJBBrjuWcOTsOlrIvdibUv0rjOnK82vf7GDQJILaORuHCb8C5HgqXGrTO2xdVKb1GwRvnkTH4ZOuE76sAomIKf8Hbs69LX9naEe8HDun8V2GXVEg2Bz69QrQGH9NxOVRZZDtdXuijxBiPyK3UGH8ccvdT+JNUWuQGQhDIyMSm4rNdO4Y3Og/hGGwOiq97twIielh0mY+svPivbgxyrFy7m/q5m216Tm0pE3Afl1G97IkcID3GrLLJViDr+CUGU6dLPZgcgCiGUXn308GutkwT9us4x7LdE0DedINf6mJytrFDaetGzvARr9IwLRAyqAqaxV94uBmoeKfqEpAaWmz8k8d4jjQxzaBvb9OAl63iGCJskXKnDqu01aDT8Svdo4rGvKKGgEHVeRcXl4DDNeafSNan1it3APiKwUGpB6OvhUspL6JFR/A/5cpb3d80gfyLk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Oct 30, 2024 at 10:48PM +0100, Vlastimil Babka wrote: > On 10/30/24 22:05, Paul E. McKenney wrote: > > Hello! > > Hi! > > > The next-20241030 release gets the splat shown below when running > > scftorture in a preemptible kernel. This bisects to this commit: > > > > 560af5dc839e ("lockdep: Enable PROVE_RAW_LOCK_NESTING with PROVE_LOCKING") > > > > Except that all this is doing is enabling lockdep to find the problem. > > > > The obvious way to fix this is to make the kmem_cache structure's > > cpu_slab field's ->lock be a raw spinlock, but this might not be what > > we want for real-time response. > > But it's a local_lock, not spinlock and it's doing local_lock_irqsave(). I'm > confused what's happening here, the code has been like this for years now. > > > This can be reproduced deterministically as follows: > > > > tools/testing/selftests/rcutorture/bin/kvm.sh --torture scf --allcpus --duration 2 --configs PREEMPT --kconfig CONFIG_NR_CPUS=64 --memory 7G --trust-make --kasan --bootargs "scftorture.nthreads=64 torture.disable_onoff_at_boot csdlock_debug=1" > > > > I doubt that the number of CPUs or amount of memory makes any difference, > > but that is what I used. > > > > Thoughts? > > > > Thanx, Paul > > > > ------------------------------------------------------------------------ > > > > [ 35.659746] ============================= > > [ 35.659746] [ BUG: Invalid wait context ] > > [ 35.659746] 6.12.0-rc5-next-20241029 #57233 Not tainted > > [ 35.659746] ----------------------------- > > [ 35.659746] swapper/37/0 is trying to lock: > > [ 35.659746] ffff8881ff4bf2f0 (&c->lock){....}-{3:3}, at: put_cpu_partial+0x49/0x1b0 > > [ 35.659746] other info that might help us debug this: > > [ 35.659746] context-{2:2} > > [ 35.659746] no locks held by swapper/37/0. > > [ 35.659746] stack backtrace: > > [ 35.659746] CPU: 37 UID: 0 PID: 0 Comm: swapper/37 Not tainted 6.12.0-rc5-next-20241029 #57233 > > [ 35.659746] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 > > [ 35.659746] Call Trace: > > [ 35.659746] > > [ 35.659746] dump_stack_lvl+0x68/0xa0 > > [ 35.659746] __lock_acquire+0x8fd/0x3b90 > > [ 35.659746] ? start_secondary+0x113/0x210 > > [ 35.659746] ? __pfx___lock_acquire+0x10/0x10 > > [ 35.659746] ? __pfx___lock_acquire+0x10/0x10 > > [ 35.659746] ? __pfx___lock_acquire+0x10/0x10 > > [ 35.659746] ? __pfx___lock_acquire+0x10/0x10 > > [ 35.659746] lock_acquire+0x19b/0x520 > > [ 35.659746] ? put_cpu_partial+0x49/0x1b0 > > [ 35.659746] ? __pfx_lock_acquire+0x10/0x10 > > [ 35.659746] ? __pfx_lock_release+0x10/0x10 > > [ 35.659746] ? lock_release+0x20f/0x6f0 > > [ 35.659746] ? __pfx_lock_release+0x10/0x10 > > [ 35.659746] ? lock_release+0x20f/0x6f0 > > [ 35.659746] ? kasan_save_track+0x14/0x30 > > [ 35.659746] put_cpu_partial+0x52/0x1b0 > > [ 35.659746] ? put_cpu_partial+0x49/0x1b0 > > [ 35.659746] ? __pfx_scf_handler_1+0x10/0x10 > > [ 35.659746] __flush_smp_call_function_queue+0x2d2/0x600 > > How did we even get to put_cpu_partial directly from flushing smp calls? > SLUB doesn't use them, it uses queue_work_on)_ for flushing and that > flushing doesn't involve put_cpu_partial() AFAIK. > > I think only slab allocation or free can lead to put_cpu_partial() that > would mean the backtrace is missing something. And that somebody does a slab > alloc/free from a smp callback, which I'd then assume isn't allowed? Tail-call optimization is hiding the caller. Compiling with -fno-optimize-sibling-calls exposes the caller. This gives the full picture: [ 40.321505] ============================= [ 40.322711] [ BUG: Invalid wait context ] [ 40.323927] 6.12.0-rc5-next-20241030-dirty #4 Not tainted [ 40.325502] ----------------------------- [ 40.326653] cpuhp/47/253 is trying to lock: [ 40.327869] ffff8881ff9bf2f0 (&c->lock){....}-{3:3}, at: put_cpu_partial+0x48/0x1a0 [ 40.330081] other info that might help us debug this: [ 40.331540] context-{2:2} [ 40.332305] 3 locks held by cpuhp/47/253: [ 40.333468] #0: ffffffffae6e6910 (cpu_hotplug_lock){++++}-{0:0}, at: cpuhp_thread_fun+0xe0/0x590 [ 40.336048] #1: ffffffffae6e9060 (cpuhp_state-down){+.+.}-{0:0}, at: cpuhp_thread_fun+0xe0/0x590 [ 40.338607] #2: ffff8881002a6948 (&root->kernfs_rwsem){++++}-{4:4}, at: kernfs_remove_by_name_ns+0x78/0x100 [ 40.341454] stack backtrace: [ 40.342291] CPU: 47 UID: 0 PID: 253 Comm: cpuhp/47 Not tainted 6.12.0-rc5-next-20241030-dirty #4 [ 40.344807] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 40.347482] Call Trace: [ 40.348199] [ 40.348827] dump_stack_lvl+0x6b/0xa0 [ 40.349899] dump_stack+0x10/0x20 [ 40.350850] __lock_acquire+0x900/0x4010 [ 40.360290] lock_acquire+0x191/0x4f0 [ 40.364850] put_cpu_partial+0x51/0x1a0 [ 40.368341] scf_handler+0x1bd/0x290 [ 40.370590] scf_handler_1+0x4e/0xb0 [ 40.371630] __flush_smp_call_function_queue+0x2dd/0x600 [ 40.373142] generic_smp_call_function_single_interrupt+0xe/0x20 [ 40.374801] __sysvec_call_function_single+0x50/0x280 [ 40.376214] sysvec_call_function_single+0x6c/0x80 [ 40.377543] [ 40.378142] And scf_handler does indeed tail-call kfree: static void scf_handler(void *scfc_in) { [...] } else { kfree(scfcp); } }