From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8BBDFC9EC94 for ; Mon, 12 Jan 2026 15:17:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EDCE56B00A3; Mon, 12 Jan 2026 10:17:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E872D6B00A4; Mon, 12 Jan 2026 10:17:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D72C36B00A6; Mon, 12 Jan 2026 10:17:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C39E36B00A3 for ; Mon, 12 Jan 2026 10:17:19 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9517F1AD22D for ; Mon, 12 Jan 2026 15:17:19 +0000 (UTC) X-FDA: 84323665398.25.1D6C1DD Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf09.hostedemail.com (Postfix) with ESMTP id 676D3140013 for ; Mon, 12 Jan 2026 15:17:17 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; spf=pass (imf09.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1768231037; a=rsa-sha256; cv=none; b=ur+Qyjov9ZMgTzYU4KuypmXV1vuWzJaHZY8ujAlFbjxbMbvD6zpM5pCIcJ/rhzFRgOsUlj Ds5JggB0Fw+V5KMr0z9+UaoMQ4kr4LckXAr8hLCgFeP9EokPdF2SFAoCRt8cpNGwEbYg6Z 8Z6B0ANbM3AL6P/qrmCJxEc4qrMC5Eg= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1768231037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v6cRCZ2uht5FrarNg/yCMPHI/FNHADk8HmE9kFOaEj8=; b=L2qhTgMfdKW0+IPafWwUjlw21ZK2Q6D4X6Mdqf55fM5+wwV5Fswk+wmFG1lrZBZrOG5DNi IabHyzUHyJqYN2s9Mrv5unnnEQdX3JhVlg3PixojZwqYxshciW0pYnpi4wj4L9VpxuZBy+ WUxhITWGW5T/RKLRBes4FmKWoKDdQ4Y= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 361813368C; Mon, 12 Jan 2026 15:16:58 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 167FB3EA63; Mon, 12 Jan 2026 15:16:58 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id CE42BWoQZWn7FgAAD6G6ig (envelope-from ); Mon, 12 Jan 2026 15:16:58 +0000 From: Vlastimil Babka Date: Mon, 12 Jan 2026 16:17:01 +0100 Subject: [PATCH RFC v2 07/20] slab: handle kmalloc sheaves bootstrap MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260112-sheaves-for-all-v2-7-98225cfb50cf@suse.cz> References: <20260112-sheaves-for-all-v2-0-98225cfb50cf@suse.cz> In-Reply-To: <20260112-sheaves-for-all-v2-0-98225cfb50cf@suse.cz> To: Harry Yoo , Petr Tesarik , Christoph Lameter , David Rientjes , Roman Gushchin Cc: Hao Li , Andrew Morton , Uladzislau Rezki , "Liam R. Howlett" , Suren Baghdasaryan , Sebastian Andrzej Siewior , Alexei Starovoitov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, bpf@vger.kernel.org, kasan-dev@googlegroups.com, Vlastimil Babka X-Mailer: b4 0.14.3 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Stat-Signature: ogxofcfnbzi4dqf7roymw8c6fxawx3sg X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 676D3140013 X-Rspam-User: X-HE-Tag: 1768231037-849693 X-HE-Meta: U2FsdGVkX18ByZP4t5MncUuMoLaGkRF8S+UIVzXCis9QPpnURZWyLjaRtWK8/UDT8fJFArb2EATmGonX3sbYg2aSMfVcuBha1sOSnGaKX6EUbU1dLGvckOvVn6EOxQZ/tAhcaxPFToAlGCsCTJ3Zp503vTP4lsoDtr4JqhfTYsu38Gel1Kbk+8Z/r80RIk6pv6sLs/CjJcZQ+YqhRAdSyktEw6/gWhJD6vmGFhwjjaxv6m3yaMKbqgpQ4IDOgw+X9o+/x0D0RNY0nkSRRAJlJGRhmBgrSG46BOU4il6y6Ew9h5doxj/WsvkQSjycK/NnOH5gyKQ34QL12AnS5iohNB1TgmsMplubyo2JwiXhfUiyvgesDAmFizN+R2OY1gQYlFSs5GKB3wkrxkiQG3XCVF9j1kAJnPNeaXe31Dnk+hUwaD0GJ0t7LWwW6ZKmSDUwGgTOsd1c2XnuIU6hrzyEVP8ZDjEqBVrijSUUoPkNgqLCjEU5ek1IxhSwF1kyNzl/n9D91n2y58hoCQEFpegdKnP6+MZ3NRF6mX78WCy+PQDl+ufcCMyrwiLVE/WY+x/wTGc4Ymda6px7nQbn8krCpZUG5hzbaOR6yOExMkBCyYVN3cxKhdhpB5Gc1mEu3ZSFgVY6Nw8/Vz5Xjt8C4f08Slwv5rw1AyP5Vla0L9M5bhhYujbQb548ALLIonN705qBrme9sIXaz5pe+Ph2g0cnAx+z5TJg56hYnXb20BMC9ZYuwy/L7YLXsU5VnI++GO2ERXK9VaDyuDc441IQdPIOgnz3mnX48/2q8IQqEv3pnvQDGZEOYoSoRniCuWQg77+u9oljWIoqRHMQ03pkWAbmwSILU6RAol50kn/pIzDmLjvQDJFLMpQCu69AsEDNG9Nc3d23VubAROOxhQx4ndvHztpkw1W83+k+Su6PQFU51G703ZVU8H91+VI9w9YiQxVNFiJxvkoxFWA2igYTe1N j82eBUcY azLPc2LuEmOO8iK44jM28G7xtEUzlNO7FTGJeaXXVrxWKrcY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Enable sheaves for kmalloc caches. For other types than KMALLOC_NORMAL, we can simply allow them in calculate_sizes() as they are created later than KMALLOC_NORMAL caches and can allocate sheaves and barns from those. For KMALLOC_NORMAL caches we perform additional step after first creating them without sheaves. Then bootstrap_cache_sheaves() simply allocates and initializes barns and sheaves and finally sets s->sheaf_capacity to make them actually used. Afterwards the only caches left without sheaves (unless SLUB_TINY or debugging is enabled) are kmem_cache and kmem_cache_node. These are only used when creating or destroying other kmem_caches. Thus they are not performance critical and we can simply leave it that way. Signed-off-by: Vlastimil Babka --- mm/slub.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 84 insertions(+), 4 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 0177a654a06a..f2de44f8bda4 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2593,7 +2593,8 @@ static void *setup_object(struct kmem_cache *s, void *object) return object; } -static struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp) +static struct slab_sheaf *__alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp, + unsigned int capacity) { struct slab_sheaf *sheaf; size_t sheaf_size; @@ -2611,7 +2612,7 @@ static struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp) if (s->flags & SLAB_KMALLOC) gfp |= __GFP_NO_OBJ_EXT; - sheaf_size = struct_size(sheaf, objects, s->sheaf_capacity); + sheaf_size = struct_size(sheaf, objects, capacity); sheaf = kzalloc(sheaf_size, gfp); if (unlikely(!sheaf)) @@ -2624,6 +2625,12 @@ static struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp) return sheaf; } +static inline struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s, + gfp_t gfp) +{ + return __alloc_empty_sheaf(s, gfp, s->sheaf_capacity); +} + static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf) { kfree(sheaf); @@ -8117,8 +8124,11 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s) if (s->flags & SLAB_RECLAIM_ACCOUNT) s->allocflags |= __GFP_RECLAIMABLE; - /* kmalloc caches need extra care to support sheaves */ - if (!is_kmalloc_cache(s)) + /* + * For KMALLOC_NORMAL caches we enable sheaves later by + * bootstrap_kmalloc_sheaves() to avoid recursion + */ + if (!is_kmalloc_normal(s)) s->sheaf_capacity = calculate_sheaf_capacity(s, args); /* @@ -8613,6 +8623,74 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) return s; } +/* + * Finish the sheaves initialization done normally by init_percpu_sheaves() and + * init_kmem_cache_nodes(). For normal kmalloc caches we have to bootstrap it + * since sheaves and barns are allocated by kmalloc. + */ +static void __init bootstrap_cache_sheaves(struct kmem_cache *s) +{ + struct kmem_cache_args empty_args = {}; + unsigned int capacity; + bool failed = false; + int node, cpu; + + capacity = calculate_sheaf_capacity(s, &empty_args); + + /* capacity can be 0 due to debugging or SLUB_TINY */ + if (!capacity) + return; + + for_each_node_mask(node, slab_nodes) { + struct node_barn *barn; + + barn = kmalloc_node(sizeof(*barn), GFP_KERNEL, node); + + if (!barn) { + failed = true; + goto out; + } + + barn_init(barn); + get_node(s, node)->barn = barn; + } + + for_each_possible_cpu(cpu) { + struct slub_percpu_sheaves *pcs; + + pcs = per_cpu_ptr(s->cpu_sheaves, cpu); + + pcs->main = __alloc_empty_sheaf(s, GFP_KERNEL, capacity); + + if (!pcs->main) { + failed = true; + break; + } + } + +out: + /* + * It's still early in boot so treat this like same as a failure to + * create the kmalloc cache in the first place + */ + if (failed) + panic("Out of memory when creating kmem_cache %s\n", s->name); + + s->sheaf_capacity = capacity; +} + +static void __init bootstrap_kmalloc_sheaves(void) +{ + enum kmalloc_cache_type type; + + for (type = KMALLOC_NORMAL; type <= KMALLOC_RANDOM_END; type++) { + for (int idx = 0; idx < KMALLOC_SHIFT_HIGH + 1; idx++) { + if (kmalloc_caches[type][idx]) + bootstrap_cache_sheaves(kmalloc_caches[type][idx]); + } + } +} + void __init kmem_cache_init(void) { static __initdata struct kmem_cache boot_kmem_cache, @@ -8656,6 +8734,8 @@ void __init kmem_cache_init(void) setup_kmalloc_cache_index_table(); create_kmalloc_caches(); + bootstrap_kmalloc_sheaves(); + /* Setup random freelists for each cache */ init_freelist_randomization(); -- 2.52.0