From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5816ECCD1BE for ; Thu, 23 Oct 2025 13:53:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A3428E002C; Thu, 23 Oct 2025 09:53:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 253B68E0002; Thu, 23 Oct 2025 09:53:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11C0C8E002C; Thu, 23 Oct 2025 09:53:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F05D78E0002 for ; Thu, 23 Oct 2025 09:53:01 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 95A43160B73 for ; Thu, 23 Oct 2025 13:53:01 +0000 (UTC) X-FDA: 84029520162.24.CB71F62 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf08.hostedemail.com (Postfix) with ESMTP id 44EE716000A for ; Thu, 23 Oct 2025 13:52:59 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JNhOt4Mr; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=h8AklegF; spf=pass (imf08.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1761227579; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QwO+ywC/2qNyrdzTEnXXbwTudyXSN4BHY9D8LplJsaM=; b=3gJ7kmyrvacMt3tsE6idFpnB5YHR59vtez5vSS8gyB1dBHrl5X7QILWg9QUaeTQVkcYuKP x/qd86EWV7bmJO0+3+F/s4R7AwmxU3vuJsPBDlhPFS5Icmc4WkRSrS4VgA56zxyCo23dbp ivuos7XAD5LSV4QJEuSeRXUhp/s3vTQ= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JNhOt4Mr; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=h8AklegF; spf=pass (imf08.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1761227579; a=rsa-sha256; cv=none; b=s9JfXhFd2bYV7wTDBGjJUoDffyP62jLBKgiRIuH/CY+fGPbXPxHbhDo3GTtD2R7xG2XJgQ SicunwxLPH28PiNzGJAyfJZqusToqW+fI4Melifc1stalvqd0zbi3ziH7+22Y1BjBwJtXD E4H2hsfzbJvBpdA5GDq0P+l0cExoRmM= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 67B0B1F7E5; Thu, 23 Oct 2025 13:52:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1761227573; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QwO+ywC/2qNyrdzTEnXXbwTudyXSN4BHY9D8LplJsaM=; b=JNhOt4MrW/r4AHKbxbNBxMY252YT69WdoS5qlcZ98vDein1GymZZC/OkvC1cX3BhqI9wET jmLrHWTVps8FR4oQTgiGCkeh5OS4yzsyuRzCjlOI8ZXP6yqz899skfZ/NkFH+3yOuO+Fts zt4BvG/cbhtZF1g/gdwWsEdS5lDNUuM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1761227573; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QwO+ywC/2qNyrdzTEnXXbwTudyXSN4BHY9D8LplJsaM=; b=h8AklegFTOmVDSiZshRItoNehLZvS5hrrsunxLyHas1HC89CFJt15BLTFIoFjiRJwdBSfE Bb7kDe2fvyKpeYAQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 4A26513AAC; Thu, 23 Oct 2025 13:52:53 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id gMrQETUz+mjvQQAAD6G6ig (envelope-from ); Thu, 23 Oct 2025 13:52:53 +0000 From: Vlastimil Babka Date: Thu, 23 Oct 2025 15:52:25 +0200 Subject: [PATCH RFC 03/19] slub: remove CONFIG_SLUB_TINY specific code paths MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20251023-sheaves-for-all-v1-3-6ffa2c9941c0@suse.cz> References: <20251023-sheaves-for-all-v1-0-6ffa2c9941c0@suse.cz> In-Reply-To: <20251023-sheaves-for-all-v1-0-6ffa2c9941c0@suse.cz> To: Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , Harry Yoo Cc: Uladzislau Rezki , "Liam R. Howlett" , Suren Baghdasaryan , Sebastian Andrzej Siewior , Alexei Starovoitov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, bpf@vger.kernel.org, kasan-dev@googlegroups.com, Vlastimil Babka X-Mailer: b4 0.14.3 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 44EE716000A X-Stat-Signature: 6yxdmft611cwi1c1d4h4fz7eayzow9bk X-Rspam-User: X-HE-Tag: 1761227579-954523 X-HE-Meta: U2FsdGVkX19WsnbqumbGu11GUXYxm/SGH4T6/ES7e02Byt2yaTyXyAFORCXZq4FYw5b4c3+GyBIG/jo9OCllyAXWljSdHokxnoZ5hnkce4rFoNEoTW3bDEKAeGmzMeJG8HOj5zuGdM2F11PjgMW5nMf35QVa8ULOA+xi9LO4zCeIPKLj/pXigIqYHKIq9v/Oqsw9wi8rJuRPSH+KRIzFNIlToRlDCOmG4kCCJ9rsGwv8uk5ORnbzoqrVdwoyjwQgkcMXEIgvUg5xYHCqOW/WX/7t9w8J12/keWyrwbD8oLxmANS/lTC6vc6/chZNpH82T+DmStFORdJVIiGzURI5kBuqCj3DCTRydLIM2ek6JaK5SBe0xw6EVZy+Ddts00yCys7IFhUbCTc/TRF+DBVJTrJu71WyHeQuuiv59bTawle9bTZGXE4RI73+or2s+t+Yfuk541KUwOdFGVrE87YZMZWD/+ALopesyYRNHycpFpSE4gW4dV9SNRp14YJXgIJGUbQytLhsqfDRug4EJtbrpfzacuqbNV5AkxjgZtdso2JI07JZzkT6mLHYDOMqq17n0X+pur/f9lnoc/jxQuGCnsI/Jy8Fv1+/Z8CJ2BH2NnHHKX3xj1OyDyOq3sVCujxKBx9DeEWKFIpVGhfpIKKWAweRClidHHYUYmSCvepCx1dSnzXHUGZjZ+Q6FrheSX046h1hdQWUvqTOmXXEeNZkCb4shSSzAM0JIdlUydJT+SL4sNI5mS4XG7hj9Clg4jnrs7cd1zm4XBKY3bH/lXvkTBAHstRyPnVtpmpdzauyvMaelJa1ozO6H93TuTP49yNQj3hXnF5mgu3zCUCSUBGPGvIVhJAz8AJWM9VCFVPljA9me7I0dLkeRBPGk+l7LPX0LVHXZ7D2QeW0TwW7B0weI4m8KfL4XAj77bs6u26m5+MTYx/Vndln+kurfexft0odWTULJTf13SZoeLFZN05 yq0xNJBo VDJ7AME9Q/H9voPKzC7XXoKhr5VDdIL4Ab1QAPDTEgfNyLxRB1fZNBgB7GilIwTwLt0GEIBlSDji06aGNVBWr4akoAMMkpSTZqGlntobV/D1tREEMusC9CtZKGlRxsif5ZVKqSGoBGdq04+WoSxky+j7AzOPVmhK/FbBlx/bVnG4cOnTtXeSQna9mEFXapFzOPyABOyM+vpiqWgs2c63Igyw2KPRE92qER1KD9KJUXQw850UBW1N1zspCiFUwvu2ZYy/YCTlXw4DN7joJCg0TVS7acLFHlGS6q/I4yGb5iLuc2IBemQe+NXDEvJ7rqKL2AREZ7SW2NZhAJS4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: CONFIG_SLUB_TINY minimizes the SLUB's memory overhead in multiple ways, mainly by avoiding percpu caching of slabs and objects. It also reduces code size by replacing some code paths with simplified ones through ifdefs, but the benefits of that are smaller and would complicate the upcoming changes. Thus remove these code paths and associated ifdefs and simplify the code base. Signed-off-by: Vlastimil Babka --- mm/slab.h | 2 -- mm/slub.c | 107 +++----------------------------------------------------------- 2 files changed, 4 insertions(+), 105 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 078daecc7cf5..f7b8df56727d 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -236,10 +236,8 @@ struct kmem_cache_order_objects { * Slab cache management. */ struct kmem_cache { -#ifndef CONFIG_SLUB_TINY struct kmem_cache_cpu __percpu *cpu_slab; struct lock_class_key lock_key; -#endif struct slub_percpu_sheaves __percpu *cpu_sheaves; /* Used for retrieving partial slabs, etc. */ slab_flags_t flags; diff --git a/mm/slub.c b/mm/slub.c index ab03f29dc3bf..68867cd52c4f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -410,7 +410,6 @@ enum stat_item { NR_SLUB_STAT_ITEMS }; -#ifndef CONFIG_SLUB_TINY /* * When changing the layout, make sure freelist and tid are still compatible * with this_cpu_cmpxchg_double() alignment requirements. @@ -432,7 +431,6 @@ struct kmem_cache_cpu { unsigned int stat[NR_SLUB_STAT_ITEMS]; #endif }; -#endif /* CONFIG_SLUB_TINY */ static inline void stat(const struct kmem_cache *s, enum stat_item si) { @@ -597,12 +595,10 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object) return freelist_ptr_decode(s, p, ptr_addr); } -#ifndef CONFIG_SLUB_TINY static void prefetch_freepointer(const struct kmem_cache *s, void *object) { prefetchw(object + s->offset); } -#endif /* * When running under KMSAN, get_freepointer_safe() may return an uninitialized @@ -714,10 +710,12 @@ static inline unsigned int slub_get_cpu_partial(struct kmem_cache *s) return s->cpu_partial_slabs; } #else +#ifdef SLAB_SUPPORTS_SYSFS static inline void slub_set_cpu_partial(struct kmem_cache *s, unsigned int nr_objects) { } +#endif static inline unsigned int slub_get_cpu_partial(struct kmem_cache *s) { @@ -2026,13 +2024,11 @@ static inline void inc_slabs_node(struct kmem_cache *s, int node, int objects) {} static inline void dec_slabs_node(struct kmem_cache *s, int node, int objects) {} -#ifndef CONFIG_SLUB_TINY static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab, void **freelist, void *nextfree) { return false; } -#endif #endif /* CONFIG_SLUB_DEBUG */ #ifdef CONFIG_SLAB_OBJ_EXT @@ -3617,8 +3613,6 @@ static struct slab *get_partial(struct kmem_cache *s, int node, return get_any_partial(s, pc); } -#ifndef CONFIG_SLUB_TINY - #ifdef CONFIG_PREEMPTION /* * Calculate the next globally unique transaction for disambiguation @@ -4018,12 +4012,6 @@ static bool has_cpu_slab(int cpu, struct kmem_cache *s) return c->slab || slub_percpu_partial(c); } -#else /* CONFIG_SLUB_TINY */ -static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) { } -static inline bool has_cpu_slab(int cpu, struct kmem_cache *s) { return false; } -static inline void flush_this_cpu_slab(struct kmem_cache *s) { } -#endif /* CONFIG_SLUB_TINY */ - static bool has_pcs_used(int cpu, struct kmem_cache *s) { struct slub_percpu_sheaves *pcs; @@ -4364,7 +4352,6 @@ static inline bool pfmemalloc_match(struct slab *slab, gfp_t gfpflags) return true; } -#ifndef CONFIG_SLUB_TINY static inline bool __update_cpu_freelist_fast(struct kmem_cache *s, void *freelist_old, void *freelist_new, @@ -4628,7 +4615,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, pc.orig_size = orig_size; slab = get_partial(s, node, &pc); if (slab) { - if (kmem_cache_debug(s)) { + if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) { freelist = pc.object; /* * For debug caches here we had to go through @@ -4666,7 +4653,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, stat(s, ALLOC_SLAB); - if (kmem_cache_debug(s)) { + if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) { freelist = alloc_single_from_new_slab(s, slab, orig_size, gfpflags); if (unlikely(!freelist)) @@ -4874,32 +4861,6 @@ static __always_inline void *__slab_alloc_node(struct kmem_cache *s, return object; } -#else /* CONFIG_SLUB_TINY */ -static void *__slab_alloc_node(struct kmem_cache *s, - gfp_t gfpflags, int node, unsigned long addr, size_t orig_size) -{ - struct partial_context pc; - struct slab *slab; - void *object; - - pc.flags = gfpflags; - pc.orig_size = orig_size; - slab = get_partial(s, node, &pc); - - if (slab) - return pc.object; - - slab = new_slab(s, gfpflags, node); - if (unlikely(!slab)) { - slab_out_of_memory(s, gfpflags, node); - return NULL; - } - - object = alloc_single_from_new_slab(s, slab, orig_size, gfpflags); - - return object; -} -#endif /* CONFIG_SLUB_TINY */ /* * If the object has been wiped upon free, make sure it's fully initialized by @@ -5746,9 +5707,7 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node) * it did local_lock_irqsave(&s->cpu_slab->lock, flags). * In this case fast path with __update_cpu_freelist_fast() is not safe. */ -#ifndef CONFIG_SLUB_TINY if (!in_nmi() || !local_lock_is_locked(&s->cpu_slab->lock)) -#endif ret = __slab_alloc_node(s, alloc_gfp, node, _RET_IP_, size); if (PTR_ERR(ret) == -EBUSY) { @@ -6511,14 +6470,10 @@ static void free_deferred_objects(struct irq_work *work) llist_for_each_safe(pos, t, llnode) { struct slab *slab = container_of(pos, struct slab, llnode); -#ifdef CONFIG_SLUB_TINY - free_slab(slab->slab_cache, slab); -#else if (slab->frozen) deactivate_slab(slab->slab_cache, slab, slab->flush_freelist); else free_slab(slab->slab_cache, slab); -#endif } } @@ -6554,7 +6509,6 @@ void defer_free_barrier(void) irq_work_sync(&per_cpu_ptr(&defer_free_objects, cpu)->work); } -#ifndef CONFIG_SLUB_TINY /* * Fastpath with forced inlining to produce a kfree and kmem_cache_free that * can perform fastpath freeing without additional function calls. @@ -6647,14 +6601,6 @@ static __always_inline void do_slab_free(struct kmem_cache *s, } stat_add(s, FREE_FASTPATH, cnt); } -#else /* CONFIG_SLUB_TINY */ -static void do_slab_free(struct kmem_cache *s, - struct slab *slab, void *head, void *tail, - int cnt, unsigned long addr) -{ - __slab_free(s, slab, head, tail, cnt, addr); -} -#endif /* CONFIG_SLUB_TINY */ static __fastpath_inline void slab_free(struct kmem_cache *s, struct slab *slab, void *object, @@ -6932,11 +6878,7 @@ void kfree_nolock(const void *object) * since kasan quarantine takes locks and not supported from NMI. */ kasan_slab_free(s, x, false, false, /* skip quarantine */true); -#ifndef CONFIG_SLUB_TINY do_slab_free(s, slab, x, x, 0, _RET_IP_); -#else - defer_free(s, x); -#endif } EXPORT_SYMBOL_GPL(kfree_nolock); @@ -7386,7 +7328,6 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p) } EXPORT_SYMBOL(kmem_cache_free_bulk); -#ifndef CONFIG_SLUB_TINY static inline int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p) @@ -7451,35 +7392,6 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, return 0; } -#else /* CONFIG_SLUB_TINY */ -static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, - size_t size, void **p) -{ - int i; - - for (i = 0; i < size; i++) { - void *object = kfence_alloc(s, s->object_size, flags); - - if (unlikely(object)) { - p[i] = object; - continue; - } - - p[i] = __slab_alloc_node(s, flags, NUMA_NO_NODE, - _RET_IP_, s->object_size); - if (unlikely(!p[i])) - goto error; - - maybe_wipe_obj_freeptr(s, p[i]); - } - - return i; - -error: - __kmem_cache_free_bulk(s, i, p); - return 0; -} -#endif /* CONFIG_SLUB_TINY */ /* Note that interrupts must be enabled when calling this function. */ int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, @@ -7698,7 +7610,6 @@ init_kmem_cache_node(struct kmem_cache_node *n, struct node_barn *barn) barn_init(barn); } -#ifndef CONFIG_SLUB_TINY static inline int alloc_kmem_cache_cpus(struct kmem_cache *s) { BUILD_BUG_ON(PERCPU_DYNAMIC_EARLY_SIZE < @@ -7719,12 +7630,6 @@ static inline int alloc_kmem_cache_cpus(struct kmem_cache *s) return 1; } -#else -static inline int alloc_kmem_cache_cpus(struct kmem_cache *s) -{ - return 1; -} -#endif /* CONFIG_SLUB_TINY */ static int init_percpu_sheaves(struct kmem_cache *s) { @@ -7814,13 +7719,11 @@ void __kmem_cache_release(struct kmem_cache *s) cache_random_seq_destroy(s); if (s->cpu_sheaves) pcs_destroy(s); -#ifndef CONFIG_SLUB_TINY #ifdef CONFIG_PREEMPT_RT if (s->cpu_slab) lockdep_unregister_key(&s->lock_key); #endif free_percpu(s->cpu_slab); -#endif free_kmem_cache_nodes(s); } @@ -8563,10 +8466,8 @@ void __init kmem_cache_init(void) void __init kmem_cache_init_late(void) { -#ifndef CONFIG_SLUB_TINY flushwq = alloc_workqueue("slub_flushwq", WQ_MEM_RECLAIM, 0); WARN_ON(!flushwq); -#endif } struct kmem_cache * -- 2.51.1