From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DD95AC982C9 for ; Fri, 16 Jan 2026 14:41:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3CE6A6B00B0; Fri, 16 Jan 2026 09:41:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 37BAF6B00B3; Fri, 16 Jan 2026 09:41:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2734D6B00B0; Fri, 16 Jan 2026 09:41:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0FF626B00B0 for ; Fri, 16 Jan 2026 09:41:29 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DD2A5B5B45 for ; Fri, 16 Jan 2026 14:41:28 +0000 (UTC) X-FDA: 84338090256.01.D836EF2 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf17.hostedemail.com (Postfix) with ESMTP id AB58640009 for ; Fri, 16 Jan 2026 14:41:26 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=YIg7LUNR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=XHSg3RXI; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=YIg7LUNR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=XHSg3RXI; dmarc=none; spf=pass (imf17.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1768574487; a=rsa-sha256; cv=none; b=w7NBwqIGJ9K4HdVPYEcd57Ar+suY2i5p6sj7EiIkClfybcw8jsIF5L+EPQzt0DX9GeByVJ 4Ved6e6zA4hn3NBI7fLPRxeTTg3p8q40EEwEPCGzhNR5O1EERVPxgEetn2rj/b34Qcevpk 2zwSbRgeKoJAtYyR8mAt/JgfWFFYXfc= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=YIg7LUNR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=XHSg3RXI; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=YIg7LUNR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=XHSg3RXI; dmarc=none; spf=pass (imf17.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1768574487; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7lW8Ood/bwUVgrzq8dslY//7lpLYcu/ZEOKOtqQvI2s=; b=SAXRczQ87WKgUR83nWKcJVojoWmmz5bRJna4jef4q5ZgKtd4dSQ9laadDyLwSP7SkyNNAp ZrUP1f1LEHH56cBp7hoe/XovLkjK02Q9uT3yHIrtmvpGqFtz/zQ3NHmvteEsq7Hp+L69XD IJHmFnErff8I1OHr+6PKTMNh6qzWu+A= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 34C6C337CD; Fri, 16 Jan 2026 14:40:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1768574438; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7lW8Ood/bwUVgrzq8dslY//7lpLYcu/ZEOKOtqQvI2s=; b=YIg7LUNRkDHtoX03L9UVgGmG4iwJ4yZCzcerQmIQhyO9lRGh/i5HutJbaVXIMYn7dvbcDI HNegctP8xYh8uDisNhcHoXlPLiKCzITawoyK/JIKfEKDaJGtY+asraHZTKVh0jv9Yi2KKD vUxq4ztWQy2NS2vaGGww7GNSQToFGjk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1768574438; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7lW8Ood/bwUVgrzq8dslY//7lpLYcu/ZEOKOtqQvI2s=; b=XHSg3RXIpj1ZmrTF+dnVIlvUn+uJq1uN/ZrcQSYhgFOMvcx3pCQN/kWUJRjBDVSQ4li67g D0Pms52i3TwTSRDg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1768574438; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7lW8Ood/bwUVgrzq8dslY//7lpLYcu/ZEOKOtqQvI2s=; b=YIg7LUNRkDHtoX03L9UVgGmG4iwJ4yZCzcerQmIQhyO9lRGh/i5HutJbaVXIMYn7dvbcDI HNegctP8xYh8uDisNhcHoXlPLiKCzITawoyK/JIKfEKDaJGtY+asraHZTKVh0jv9Yi2KKD vUxq4ztWQy2NS2vaGGww7GNSQToFGjk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1768574438; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7lW8Ood/bwUVgrzq8dslY//7lpLYcu/ZEOKOtqQvI2s=; b=XHSg3RXIpj1ZmrTF+dnVIlvUn+uJq1uN/ZrcQSYhgFOMvcx3pCQN/kWUJRjBDVSQ4li67g D0Pms52i3TwTSRDg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 16EB53EA65; Fri, 16 Jan 2026 14:40:38 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id wChTBeZNamnydgAAD6G6ig (envelope-from ); Fri, 16 Jan 2026 14:40:38 +0000 From: Vlastimil Babka Date: Fri, 16 Jan 2026 15:40:34 +0100 Subject: [PATCH v3 14/21] slab: simplify kmalloc_nolock() MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260116-sheaves-for-all-v3-14-5595cb000772@suse.cz> References: <20260116-sheaves-for-all-v3-0-5595cb000772@suse.cz> In-Reply-To: <20260116-sheaves-for-all-v3-0-5595cb000772@suse.cz> To: Harry Yoo , Petr Tesarik , Christoph Lameter , David Rientjes , Roman Gushchin Cc: Hao Li , Andrew Morton , Uladzislau Rezki , "Liam R. Howlett" , Suren Baghdasaryan , Sebastian Andrzej Siewior , Alexei Starovoitov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, bpf@vger.kernel.org, kasan-dev@googlegroups.com, Vlastimil Babka X-Mailer: b4 0.14.3 X-Rspamd-Action: no action X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: AB58640009 X-Stat-Signature: 9hcsrm1e3wsg3hxdzd9nopaif8oiz963 X-Rspam-User: X-HE-Tag: 1768574486-668615 X-HE-Meta: U2FsdGVkX19dLwinK+Oetwfv/1jaKmiWawTfwNjgX0DnmNdbYsFg5/nadIfGEiptlF2KulHmArjYKC3de3ImNHYhr61SEIIAx3mGXv7qgcai1+GLwLJRLTC004qWwUbL8PTF+IcUlZJo7VQx1647uYMxzDUIi4KZ3B/wQQR8kPaoBeJqiW1ZBcDAqNkuIhIkGeOOfh4a/A2nLVkyPCdA8TGhw2iENHtScp/DJTNmMgSrJVLvcxfs1UzhZQ3InuVOePgHLOpzJEMzCHz7PAHhLWZWZZC2KI7GaviZvmrgFc0aFyRRdJ1piYZP8f4R0SkHG0XZRVevPqJZMF/GkQchZvoCoq2rjdclXvycRTGXlO64dQs5yy274w0X9EyQxXgG2S7A+pDxhzkXKm98IMvAEg/b/qqdv3xs+EfYsA9KL6hPH6qli+cGmVwis+z/afWATx80w5lRoQmyXf4sOjJSWs7venDBmiDQMSoCVVf78g8voO/stljmHlh5oxne0wp1QNKmjPqxWb/Rvjux70pDHZ55aDn73hmIiGQcIeMMpF6kMSONCn3TspE/xvGQyEoG2Gut7o4MAgMtyMyejPPTh1wx4HBCo++oaxpV7UDbThqRrkiaNraFwgWCpHozSv3SdJ4pfVEftPAeCCK02qWgP6jyzqvDkkj3BSADM8ejDRnPEQ3KBP+Y1eh0y+jLXsDc4KfLTQXu5ZpHyc9OQnAdp6WoCxNpZp0tXFlL7XqTW1Turt6J8dbqGYf0DMpDv5m9d5OT8okIlUF4fviwABMPNKtD8WgZhSMQRjX3fP8JhTLQUN/eaOnX6lK32t2IvaM/ttajcerXL15iGya9L/h66m+AhJQ8Nl6NiR7loq4RPhcnefFKyj4gIcIQYJtoQcI2PQVOin2pX6avIrAiEXSWzHsK/uBNsqYoHLSvbadsZRLiEjZVmx1Q9TqtiHCpkt8LapKjIFUKnhnxVm2Qpky h18hkLXY ALdYNAAASD5OmP+NuYKGOT4xM7YkgFZcL6WEUt35AUcKPvgGJo61Sdu1ccjHIE1+cB/thpjx9XVfs2HImAkam2EvyVSEL229bLeo7kXZCxV68AIhej254nDscwow26AIXzPiPNh7KlIgd5IAr1fURBJlJ8maaTzDGlcKli1PVH8C8VHHAD+zNm06qVDo2Rqt0jIGlAuLfTzAglkJZD93ObpT8+00J/HydxivrANjuaRLXy/GnvEXBvQjW96U4LgDiVW3DwynwBdmvi/le+7lZguPxx5x6D+bIfxozkhM+3ddt7aw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The kmalloc_nolock() implementation has several complications and restrictions due to SLUB's cpu slab locking, lockless fastpath and PREEMPT_RT differences. With cpu slab usage removed, we can simplify things: - relax the PREEMPT_RT context checks as they were before commit a4ae75d1b6a2 ("slab: fix kmalloc_nolock() context check for PREEMPT_RT") and also reference the explanation comment in the page allocator - the local_lock_cpu_slab() macros became unused, remove them - we no longer need to set up lockdep classes on PREEMPT_RT - we no longer need to annotate ___slab_alloc as NOKPROBE_SYMBOL since there's no lockless cpu freelist manipulation anymore - __slab_alloc_node() can be called from kmalloc_nolock_noprof() unconditionally. It can also no longer return EBUSY. But trylock failures can still happen so retry with the larger bucket if the allocation fails for any reason. Note that we still need __CMPXCHG_DOUBLE, because while it was removed we don't use cmpxchg16b on cpu freelist anymore, we still use it on slab freelist, and the alternative is slab_lock() which can be interrupted by a nmi. Clarify the comment to mention it specifically. Signed-off-by: Vlastimil Babka --- mm/slab.h | 1 - mm/slub.c | 144 +++++++++++++------------------------------------------------- 2 files changed, 29 insertions(+), 116 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 4efec41b6445..e9a0738133ed 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -190,7 +190,6 @@ struct kmem_cache_order_objects { */ struct kmem_cache { struct kmem_cache_cpu __percpu *cpu_slab; - struct lock_class_key lock_key; struct slub_percpu_sheaves __percpu *cpu_sheaves; /* Used for retrieving partial slabs, etc. */ slab_flags_t flags; diff --git a/mm/slub.c b/mm/slub.c index 33f218c0e8d6..8746d9d3f3a3 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3694,29 +3694,12 @@ static inline unsigned int init_tid(int cpu) static void init_kmem_cache_cpus(struct kmem_cache *s) { -#ifdef CONFIG_PREEMPT_RT - /* - * Register lockdep key for non-boot kmem caches to avoid - * WARN_ON_ONCE(static_obj(key))) in lockdep_register_key() - */ - bool finegrain_lockdep = !init_section_contains(s, 1); -#else - /* - * Don't bother with different lockdep classes for each - * kmem_cache, since we only use local_trylock_irqsave(). - */ - bool finegrain_lockdep = false; -#endif int cpu; struct kmem_cache_cpu *c; - if (finegrain_lockdep) - lockdep_register_key(&s->lock_key); for_each_possible_cpu(cpu) { c = per_cpu_ptr(s->cpu_slab, cpu); local_trylock_init(&c->lock); - if (finegrain_lockdep) - lockdep_set_class(&c->lock, &s->lock_key); c->tid = init_tid(cpu); } } @@ -3803,47 +3786,6 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, } } -/* - * ___slab_alloc()'s caller is supposed to check if kmem_cache::kmem_cache_cpu::lock - * can be acquired without a deadlock before invoking the function. - * - * Without LOCKDEP we trust the code to be correct. kmalloc_nolock() is - * using local_lock_is_locked() properly before calling local_lock_cpu_slab(), - * and kmalloc() is not used in an unsupported context. - * - * With LOCKDEP, on PREEMPT_RT lockdep does its checking in local_lock_irqsave(). - * On !PREEMPT_RT we use trylock to avoid false positives in NMI, but - * lockdep_assert() will catch a bug in case: - * #1 - * kmalloc() -> ___slab_alloc() -> irqsave -> NMI -> bpf -> kmalloc_nolock() - * or - * #2 - * kmalloc() -> ___slab_alloc() -> irqsave -> tracepoint/kprobe -> bpf -> kmalloc_nolock() - * - * On PREEMPT_RT an invocation is not possible from IRQ-off or preempt - * disabled context. The lock will always be acquired and if needed it - * block and sleep until the lock is available. - * #1 is possible in !PREEMPT_RT only. - * #2 is possible in both with a twist that irqsave is replaced with rt_spinlock: - * kmalloc() -> ___slab_alloc() -> rt_spin_lock(kmem_cache_A) -> - * tracepoint/kprobe -> bpf -> kmalloc_nolock() -> rt_spin_lock(kmem_cache_B) - * - * local_lock_is_locked() prevents the case kmem_cache_A == kmem_cache_B - */ -#if defined(CONFIG_PREEMPT_RT) || !defined(CONFIG_LOCKDEP) -#define local_lock_cpu_slab(s, flags) \ - local_lock_irqsave(&(s)->cpu_slab->lock, flags) -#else -#define local_lock_cpu_slab(s, flags) \ - do { \ - bool __l = local_trylock_irqsave(&(s)->cpu_slab->lock, flags); \ - lockdep_assert(__l); \ - } while (0) -#endif - -#define local_unlock_cpu_slab(s, flags) \ - local_unlock_irqrestore(&(s)->cpu_slab->lock, flags) - static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c) { unsigned long flags; @@ -4402,20 +4344,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, return freelist; } -/* - * We disallow kprobes in ___slab_alloc() to prevent reentrance - * - * kmalloc() -> ___slab_alloc() -> local_lock_cpu_slab() protected part of - * ___slab_alloc() manipulating c->freelist -> kprobe -> bpf -> - * kmalloc_nolock() or kfree_nolock() -> __update_cpu_freelist_fast() - * manipulating c->freelist without lock. - * - * This does not prevent kprobe in functions called from ___slab_alloc() such as - * local_lock_irqsave() itself, and that is fine, we only need to protect the - * c->freelist manipulation in ___slab_alloc() itself. - */ -NOKPROBE_SYMBOL(___slab_alloc); - static __always_inline void *__slab_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node, unsigned long addr, size_t orig_size) { @@ -5253,13 +5181,13 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node) if (unlikely(!size)) return ZERO_SIZE_PTR; - if (IS_ENABLED(CONFIG_PREEMPT_RT) && !preemptible()) - /* - * kmalloc_nolock() in PREEMPT_RT is not supported from - * non-preemptible context because local_lock becomes a - * sleeping lock on RT. - */ + /* + * See the comment for the same check in + * alloc_frozen_pages_nolock_noprof() + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq())) return NULL; + retry: if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) return NULL; @@ -5268,10 +5196,11 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node) if (!(s->flags & __CMPXCHG_DOUBLE) && !kmem_cache_debug(s)) /* * kmalloc_nolock() is not supported on architectures that - * don't implement cmpxchg16b, but debug caches don't use - * per-cpu slab and per-cpu partial slabs. They rely on - * kmem_cache_node->list_lock, so kmalloc_nolock() can - * attempt to allocate from debug caches by + * don't implement cmpxchg16b and thus need slab_lock() + * which could be preempted by a nmi. + * But debug caches don't use that and only rely on + * kmem_cache_node->list_lock, so kmalloc_nolock() can attempt + * to allocate from debug caches by * spin_trylock_irqsave(&n->list_lock, ...) */ return NULL; @@ -5280,42 +5209,31 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node) if (ret) goto success; - ret = ERR_PTR(-EBUSY); - /* * Do not call slab_alloc_node(), since trylock mode isn't * compatible with slab_pre_alloc_hook/should_failslab and * kfence_alloc. Hence call __slab_alloc_node() (at most twice) * and slab_post_alloc_hook() directly. - * - * In !PREEMPT_RT ___slab_alloc() manipulates (freelist,tid) pair - * in irq saved region. It assumes that the same cpu will not - * __update_cpu_freelist_fast() into the same (freelist,tid) pair. - * Therefore use in_nmi() to check whether particular bucket is in - * irq protected section. - * - * If in_nmi() && local_lock_is_locked(s->cpu_slab) then it means that - * this cpu was interrupted somewhere inside ___slab_alloc() after - * it did local_lock_irqsave(&s->cpu_slab->lock, flags). - * In this case fast path with __update_cpu_freelist_fast() is not safe. */ - if (!in_nmi() || !local_lock_is_locked(&s->cpu_slab->lock)) - ret = __slab_alloc_node(s, alloc_gfp, node, _RET_IP_, size); + ret = __slab_alloc_node(s, alloc_gfp, node, _RET_IP_, size); - if (PTR_ERR(ret) == -EBUSY) { - if (can_retry) { - /* pick the next kmalloc bucket */ - size = s->object_size + 1; - /* - * Another alternative is to - * if (memcg) alloc_gfp &= ~__GFP_ACCOUNT; - * else if (!memcg) alloc_gfp |= __GFP_ACCOUNT; - * to retry from bucket of the same size. - */ - can_retry = false; - goto retry; - } - ret = NULL; + /* + * It's possible we failed due to trylock as we preempted someone with + * the sheaves locked, and the list_lock is also held by another cpu. + * But it should be rare that multiple kmalloc buckets would have + * sheaves locked, so try a larger one. + */ + if (!ret && can_retry) { + /* pick the next kmalloc bucket */ + size = s->object_size + 1; + /* + * Another alternative is to + * if (memcg) alloc_gfp &= ~__GFP_ACCOUNT; + * else if (!memcg) alloc_gfp |= __GFP_ACCOUNT; + * to retry from bucket of the same size. + */ + can_retry = false; + goto retry; } success: @@ -7334,10 +7252,6 @@ void __kmem_cache_release(struct kmem_cache *s) cache_random_seq_destroy(s); if (s->cpu_sheaves) pcs_destroy(s); -#ifdef CONFIG_PREEMPT_RT - if (s->cpu_slab) - lockdep_unregister_key(&s->lock_key); -#endif free_percpu(s->cpu_slab); free_kmem_cache_nodes(s); } -- 2.52.0