From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B07A0D2ED0F for ; Tue, 20 Jan 2026 06:33:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A7906B036D; Tue, 20 Jan 2026 01:33:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0548E6B036E; Tue, 20 Jan 2026 01:33:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E78326B036F; Tue, 20 Jan 2026 01:33:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D8B6E6B036D for ; Tue, 20 Jan 2026 01:33:52 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 640C813ADDE for ; Tue, 20 Jan 2026 06:33:52 +0000 (UTC) X-FDA: 84351376704.14.38B6FA9 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf23.hostedemail.com (Postfix) with ESMTP id D569C14000A for ; Tue, 20 Jan 2026 06:33:49 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=HgCjMk+E; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="/RqIpGWo"; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ScNhbrR+; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=CwkFapkR; spf=pass (imf23.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1768890830; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VK5LAWZJGesYmfMe3akrFE9Qr+u5i9eRqUPU6M9O84A=; b=lzo98JZ7pZQiW4WLW/okKxGLRE/uZqWcbObgXykbwJXHGDylBxmocYrrryrul/eetCz/j9 t9rYEXcnAUpyfKPLw0X41G5tS+QcLESSqmxLH7pksg0rar3l10h5SpjKbniv6ilmsfPpxd Hz7sB8XCve1DDSE4IrlbQAg2QIdCK18= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=HgCjMk+E; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="/RqIpGWo"; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ScNhbrR+; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=CwkFapkR; spf=pass (imf23.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1768890830; a=rsa-sha256; cv=none; b=YWeHeNMR4BPIIGTkgixTbcL3fW1DnKr9RILkx4qhzv+MXlRVr+Yv5xWdCsxe71dQoucis/ nmjU2A3eQUyeZ6dVc4I6sHMxq2aW5PMqRdl6aGHX8NCS+EjSgUlcXiQvy07rQluSswS1z4 PN+qK/H5ngVdsr5EQTLnoiZeDFoBKEQ= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id DDF775BCC9; Tue, 20 Jan 2026 06:33:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1768890828; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=VK5LAWZJGesYmfMe3akrFE9Qr+u5i9eRqUPU6M9O84A=; b=HgCjMk+Eeeek0ViYMGJZHfXPrpt91EQyv/WZhxIssU7+VTqcZMCHLR9PaARjSrRv+9y81C Ca4VPpdPw9/s9SAMX8K74Bow/H7G4J1T66yl6jr018FUgpRii1fIc4hf/o+xR/mhO9Y7yx TqTse/uDn4KB4O3xziG1Dy8tjTdNjMA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1768890828; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=VK5LAWZJGesYmfMe3akrFE9Qr+u5i9eRqUPU6M9O84A=; b=/RqIpGWoYhLlSBZI1+coiVcbhpn4cXVVNMgcAF05LnsnR+0A8Cb3jXGIV577L/BBhgS35b yInvfJ1pkLwYXGAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1768890827; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=VK5LAWZJGesYmfMe3akrFE9Qr+u5i9eRqUPU6M9O84A=; b=ScNhbrR+3/oaBrjT9QPc7vNyJBqtT/kLM6kXRBYM6ATBcu66I99UJZwUcVJP4hGjGwtNzm CFrLQ7Cr6zNqv7hOSVpSPfR2rq6cczVcyLcsAg5tLP/WScyx7vlsqVcI12gWSY4vTHEi6U toEujIvHQrKUOT8TeyqvXJLX7KqZcHA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1768890827; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=VK5LAWZJGesYmfMe3akrFE9Qr+u5i9eRqUPU6M9O84A=; b=CwkFapkRy8eNu4Yhm45k1te576OuxQByC7CIkaBzdUH+y1blptJ2vChjRTZae9SXOw6Mdo H9pm0J2cedcycyCg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id AC09A3EA63; Tue, 20 Jan 2026 06:33:47 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id c0s8KMshb2lLAgAAD6G6ig (envelope-from ); Tue, 20 Jan 2026 06:33:47 +0000 Message-ID: <2232564a-b3f7-4591-abe2-8f1711590e6e@suse.cz> Date: Tue, 20 Jan 2026 07:33:47 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 09/21] slab: add optimized sheaf refill from partial list Content-Language: en-US To: Harry Yoo Cc: Petr Tesarik , Christoph Lameter , David Rientjes , Roman Gushchin , Hao Li , Andrew Morton , Uladzislau Rezki , "Liam R. Howlett" , Suren Baghdasaryan , Sebastian Andrzej Siewior , Alexei Starovoitov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, bpf@vger.kernel.org, kasan-dev@googlegroups.com References: <20260116-sheaves-for-all-v3-0-5595cb000772@suse.cz> <20260116-sheaves-for-all-v3-9-5595cb000772@suse.cz> From: Vlastimil Babka Autocrypt: addr=vbabka@suse.cz; keydata= xsFNBFZdmxYBEADsw/SiUSjB0dM+vSh95UkgcHjzEVBlby/Fg+g42O7LAEkCYXi/vvq31JTB KxRWDHX0R2tgpFDXHnzZcQywawu8eSq0LxzxFNYMvtB7sV1pxYwej2qx9B75qW2plBs+7+YB 87tMFA+u+L4Z5xAzIimfLD5EKC56kJ1CsXlM8S/LHcmdD9Ctkn3trYDNnat0eoAcfPIP2OZ+ 9oe9IF/R28zmh0ifLXyJQQz5ofdj4bPf8ecEW0rhcqHfTD8k4yK0xxt3xW+6Exqp9n9bydiy tcSAw/TahjW6yrA+6JhSBv1v2tIm+itQc073zjSX8OFL51qQVzRFr7H2UQG33lw2QrvHRXqD Ot7ViKam7v0Ho9wEWiQOOZlHItOOXFphWb2yq3nzrKe45oWoSgkxKb97MVsQ+q2SYjJRBBH4 8qKhphADYxkIP6yut/eaj9ImvRUZZRi0DTc8xfnvHGTjKbJzC2xpFcY0DQbZzuwsIZ8OPJCc LM4S7mT25NE5kUTG/TKQCk922vRdGVMoLA7dIQrgXnRXtyT61sg8PG4wcfOnuWf8577aXP1x 6mzw3/jh3F+oSBHb/GcLC7mvWreJifUL2gEdssGfXhGWBo6zLS3qhgtwjay0Jl+kza1lo+Cv BB2T79D4WGdDuVa4eOrQ02TxqGN7G0Biz5ZLRSFzQSQwLn8fbwARAQABzSBWbGFzdGltaWwg QmFia2EgPHZiYWJrYUBzdXNlLmN6PsLBlAQTAQoAPgIbAwULCQgHAwUVCgkICwUWAgMBAAIe AQIXgBYhBKlA1DSZLC6OmRA9UCJPp+fMgqZkBQJnyBr8BQka0IFQAAoJECJPp+fMgqZkqmMQ AIbGN95ptUMUvo6aAdhxaOCHXp1DfIBuIOK/zpx8ylY4pOwu3GRe4dQ8u4XS9gaZ96Gj4bC+ jwWcSmn+TjtKW3rH1dRKopvC07tSJIGGVyw7ieV/5cbFffA8NL0ILowzVg8w1ipnz1VTkWDr 2zcfslxJsJ6vhXw5/npcY0ldeC1E8f6UUoa4eyoskd70vO0wOAoGd02ZkJoox3F5ODM0kjHu Y97VLOa3GG66lh+ZEelVZEujHfKceCw9G3PMvEzyLFbXvSOigZQMdKzQ8D/OChwqig8wFBmV QCPS4yDdmZP3oeDHRjJ9jvMUKoYODiNKsl2F+xXwyRM2qoKRqFlhCn4usVd1+wmv9iLV8nPs 2Db1ZIa49fJet3Sk3PN4bV1rAPuWvtbuTBN39Q/6MgkLTYHb84HyFKw14Rqe5YorrBLbF3rl M51Dpf6Egu1yTJDHCTEwePWug4XI11FT8lK0LNnHNpbhTCYRjX73iWOnFraJNcURld1jL1nV r/LRD+/e2gNtSTPK0Qkon6HcOBZnxRoqtazTU6YQRmGlT0v+rukj/cn5sToYibWLn+RoV1CE Qj6tApOiHBkpEsCzHGu+iDQ1WT0Idtdynst738f/uCeCMkdRu4WMZjteQaqvARFwCy3P/jpK uvzMtves5HvZw33ZwOtMCgbpce00DaET4y/UzsBNBFsZNTUBCACfQfpSsWJZyi+SHoRdVyX5 J6rI7okc4+b571a7RXD5UhS9dlVRVVAtrU9ANSLqPTQKGVxHrqD39XSw8hxK61pw8p90pg4G /N3iuWEvyt+t0SxDDkClnGsDyRhlUyEWYFEoBrrCizbmahOUwqkJbNMfzj5Y7n7OIJOxNRkB IBOjPdF26dMP69BwePQao1M8Acrrex9sAHYjQGyVmReRjVEtv9iG4DoTsnIR3amKVk6si4Ea X/mrapJqSCcBUVYUFH8M7bsm4CSxier5ofy8jTEa/CfvkqpKThTMCQPNZKY7hke5qEq1CBk2 wxhX48ZrJEFf1v3NuV3OimgsF2odzieNABEBAAHCwXwEGAEKACYCGwwWIQSpQNQ0mSwujpkQ PVAiT6fnzIKmZAUCZ8gcVAUJFhTonwAKCRAiT6fnzIKmZLY8D/9uo3Ut9yi2YCuASWxr7QQZ lJCViArjymbxYB5NdOeC50/0gnhK4pgdHlE2MdwF6o34x7TPFGpjNFvycZqccSQPJ/gibwNA zx3q9vJT4Vw+YbiyS53iSBLXMweeVV1Jd9IjAoL+EqB0cbxoFXvnjkvP1foiiF5r73jCd4PR rD+GoX5BZ7AZmFYmuJYBm28STM2NA6LhT0X+2su16f/HtummENKcMwom0hNu3MBNPUOrujtW khQrWcJNAAsy4yMoJ2Lw51T/5X5Hc7jQ9da9fyqu+phqlVtn70qpPvgWy4HRhr25fCAEXZDp xG4RNmTm+pqorHOqhBkI7wA7P/nyPo7ZEc3L+ZkQ37u0nlOyrjbNUniPGxPxv1imVq8IyycG AN5FaFxtiELK22gvudghLJaDiRBhn8/AhXc642/Z/yIpizE2xG4KU4AXzb6C+o7LX/WmmsWP Ly6jamSg6tvrdo4/e87lUedEqCtrp2o1xpn5zongf6cQkaLZKQcBQnPmgHO5OG8+50u88D9I rywqgzTUhHFKKF6/9L/lYtrNcHU8Z6Y4Ju/MLUiNYkmtrGIMnkjKCiRqlRrZE/v5YFHbayRD dJKXobXTtCBYpLJM4ZYRpGZXne/FAtWNe4KbNJJqxMvrTOrnIatPj8NhBVI0RSJRsbilh6TE m6M14QORSWTLRg== In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Stat-Signature: 41e7p6uen115y5rc5isuz9edfuboa5sb X-Rspamd-Queue-Id: D569C14000A X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1768890829-282626 X-HE-Meta: U2FsdGVkX19Ss/ND0wsBd1NnFy8gRgRd6eHEzMuEly9IJTmRNvsTbpC3VX2B/cTrgTr2aYw1+QuHiNXMoXT2RfYGIV1JeZVCqgE1OLaUbwCiI2W3L6gpAsOECmofET26gGpBVEDui716yr59ipPXlPHl8NS1mIUAAZfirFXf2h1cEnorPy+mzHL7wDB9uUpA1K5gymy93o4WCH3kOI4Y7W8pjhZB9ArHluO9uKgpxYQcjw6QOdPLpzfJ5COMf6Wh2s7ZSDVlKPE6bOxT2MiESDMX/wAcgKxXdFKMeYeyrFbUpdtRr5c5nRcNGLCfQ0+cPydAKrY11c0DImXHOXNBFOqX9wTElX3zcP4vc4J+NM9Hb8yQ1m1qv90KMLyT1HaKndW/V8LyBmELV8tFJj42f4/ad9aQsIbTPDRw6OH/C8E/dVr5SY5JiRV2yprkTp94FBkJdIGVanG3rd/RDwp9gw6Frjbzwb6MqZ9FESW2sWPHNki/pSn6aDsJZa5I75ua4jDqlNJK2wiolbuqTcOPoWrO8ezj/4Nw2Xd4twdkOV0HHQb+W4VS1O0eBVwzuqi0r3S1ipBQUv1VgtnRGLh6Pc3NBa7ykvJs73Lbw9xDPHDWIwMRdsaoN+jC9WHQNrhKYreRmyuYSfRbgEIv7OnvrsdR5ka51YWENQv3oFPIiRhkykTM9KchqMS0JzNOkYhAC24ISed+wDN9SBeYg7F9NEqu2AjXjiLqUeHbi3kTT2x9Ib+UZ5Hd6VweKE+gfvKC0KD2z7EymM76RO2VVq9lOoNcD/LvJP+A9fW5+ceGShf8LH7y5eqs5ELSg1Li89KFhqaoOEu4LGOYklGTGuUqiqzcYiyCpKvY35q7hPwwvTF1ijXztYVMeQiZEK6ca+M525v9j6F9MwHqMIC8UBnXdLpLUT6sS5tltVXLNHsH/3NLNm6doPdhNwTrqQ3ny1GVvzEifPm/WuoDwV3rSyX SNhxO2T5 xXjf0Z8/WOib0VHWxxLSysCE8OrjyElnp09xFvDbyJpafoQemKNoYmy6ba4iFcG1RSMpuDKuR5gytjSXcejC1i4j/YajVwcVVIOIJAFmHvplJnZv+h6i89vxZ2j9LyLfGpuVYKCLOq4BNhr5TEgCbY28U3lPhKuO8lOipiEFs5OrbCD6iE665tF1V/vQujqlTWly9UsTOkNQXBp0iDkmRbOMv/0M3BBBTcjkCLEnFpQbfkXnaDfK5m5eX9XYpbVTjSmmLPj34wuyaykC4Ggcsz/MRT0+4nZRGeasf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 1/20/26 03:32, Harry Yoo wrote: > On Fri, Jan 16, 2026 at 03:40:29PM +0100, Vlastimil Babka wrote: >> At this point we have sheaves enabled for all caches, but their refill >> is done via __kmem_cache_alloc_bulk() which relies on cpu (partial) >> slabs - now a redundant caching layer that we are about to remove. >> >> The refill will thus be done from slabs on the node partial list. >> Introduce new functions that can do that in an optimized way as it's >> easier than modifying the __kmem_cache_alloc_bulk() call chain. >> >> Extend struct partial_context so it can return a list of slabs from the >> partial list with the sum of free objects in them within the requested >> min and max. >> >> Introduce get_partial_node_bulk() that removes the slabs from freelist >> and returns them in the list. >> >> Introduce get_freelist_nofreeze() which grabs the freelist without >> freezing the slab. >> >> Introduce alloc_from_new_slab() which can allocate multiple objects from >> a newly allocated slab where we don't need to synchronize with freeing. >> In some aspects it's similar to alloc_single_from_new_slab() but assumes >> the cache is a non-debug one so it can avoid some actions. >> >> Introduce __refill_objects() that uses the functions above to fill an >> array of objects. It has to handle the possibility that the slabs will >> contain more objects that were requested, due to concurrent freeing of >> objects to those slabs. When no more slabs on partial lists are >> available, it will allocate new slabs. It is intended to be only used >> in context where spinning is allowed, so add a WARN_ON_ONCE check there. >> >> Finally, switch refill_sheaf() to use __refill_objects(). Sheaves are >> only refilled from contexts that allow spinning, or even blocking. >> >> Signed-off-by: Vlastimil Babka >> --- >> mm/slub.c | 284 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++----- >> 1 file changed, 264 insertions(+), 20 deletions(-) >> >> diff --git a/mm/slub.c b/mm/slub.c >> index 9bea8a65e510..dce80463f92c 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -246,6 +246,9 @@ struct partial_context { >> gfp_t flags; >> unsigned int orig_size; >> void *object; >> + unsigned int min_objects; >> + unsigned int max_objects; >> + struct list_head slabs; >> }; >> >> static inline bool kmem_cache_debug(struct kmem_cache *s) >> @@ -2663,8 +2666,8 @@ static int refill_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf, >> if (!to_fill) >> return 0; >> >> - filled = __kmem_cache_alloc_bulk(s, gfp, to_fill, >> - &sheaf->objects[sheaf->size]); >> + filled = __refill_objects(s, &sheaf->objects[sheaf->size], gfp, >> + to_fill, to_fill); > > nit: perhaps handling min and max separately is unnecessary > if it's always min == max? we could have simply one 'count' or 'size'? Right, so the plan was to set min to some fraction of max when refilling sheaves, with the goal of maximizing the chance that once we grab a slab from the partial list, we almost certainly fully use it and don't have to return it back. But I didn't get to there yet. It seems worthwile to try though so we can leave the implementation prepared for it? > Otherwise LGTM! >