From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D735FC3ABCC for ; Wed, 14 May 2025 13:07:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5C3E6B0145; Wed, 14 May 2025 09:07:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE5986B0147; Wed, 14 May 2025 09:07:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5E1C6B0148; Wed, 14 May 2025 09:07:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 94AF16B0145 for ; Wed, 14 May 2025 09:07:35 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BFC071CA140 for ; Wed, 14 May 2025 13:07:36 +0000 (UTC) X-FDA: 83441540112.19.D7BDAEE Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf26.hostedemail.com (Postfix) with ESMTP id 60A83140002 for ; Wed, 14 May 2025 13:07:34 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=X9cDcykG; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NapuXXhH; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=X9cDcykG; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NapuXXhH; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747228054; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sTevUoX1NJ86upLyJVVYXZfr97nb3JES0EX6Dz2Mnio=; b=a8n05osyVT9bTw9XWHvnQznsuscuU8D2/vZNN610UWk481Gp+f/ByIakmGqRigPVIcmeWs Cf/8D1y18nHhYyQgu0xxMEoeHWdQGYm/T4vl/mWjZKD1cVQcQ0nyvvqGVMzlqV7yFKre3k 1WchI0D6Vlphtp0v8hYRytEGstD6TCA= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=X9cDcykG; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NapuXXhH; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=X9cDcykG; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NapuXXhH; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747228054; a=rsa-sha256; cv=none; b=E0XkMGO7iq6PLefOMM0CceYpEl5KXz7u53jpqUL0cXNAVfrHoB2JTT6hnnB7FjbcrhGd5s 29tZJRRaI7xjZVUFvHGMWuT44QgCyy4ua4kYUDP/KPW3NLZDC6PhxLINcGNnXq/EXdMI97 g+0nYYe0rkpKh+uqyWx0sEpYHVx7mPE= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id A9FA01F393; Wed, 14 May 2025 13:07:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1747228052; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sTevUoX1NJ86upLyJVVYXZfr97nb3JES0EX6Dz2Mnio=; b=X9cDcykGfbu+gNwgzknPZbrouKsVTC3LaMBQka+T9chfiUl8i/I8XW+ULnzosyQQnIaUYH nR6Ksxe424xmq5kuCdiaRS3m3uQSJ7Nxymedb5DlxW8eQXBjn00fM/0J95RNRGuASYV2/q tmCF4tWzzcNsnV3s3Nke4d4eey1QK7A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1747228052; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sTevUoX1NJ86upLyJVVYXZfr97nb3JES0EX6Dz2Mnio=; b=NapuXXhH2L4gtzp0cy/t6W3tRvxuLVoYqMEcyxzS0I6Doy3V3EpP0MFXYwOQmb+UTbo/x+ Odcwvl7xqejZfaBQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1747228052; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sTevUoX1NJ86upLyJVVYXZfr97nb3JES0EX6Dz2Mnio=; b=X9cDcykGfbu+gNwgzknPZbrouKsVTC3LaMBQka+T9chfiUl8i/I8XW+ULnzosyQQnIaUYH nR6Ksxe424xmq5kuCdiaRS3m3uQSJ7Nxymedb5DlxW8eQXBjn00fM/0J95RNRGuASYV2/q tmCF4tWzzcNsnV3s3Nke4d4eey1QK7A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1747228052; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sTevUoX1NJ86upLyJVVYXZfr97nb3JES0EX6Dz2Mnio=; b=NapuXXhH2L4gtzp0cy/t6W3tRvxuLVoYqMEcyxzS0I6Doy3V3EpP0MFXYwOQmb+UTbo/x+ Odcwvl7xqejZfaBQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 8D22E137E8; Wed, 14 May 2025 13:07:32 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id wnInIpSVJGj7dQAAD6G6ig (envelope-from ); Wed, 14 May 2025 13:07:32 +0000 Message-ID: <505bfde6-6338-4747-b29f-0e80d6fa8fff@suse.cz> Date: Wed, 14 May 2025 15:07:32 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 2/9] slab: add sheaf support for batching kfree_rcu() operations Content-Language: en-US To: Harry Yoo Cc: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes , Roman Gushchin , Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org References: <20250425-slub-percpu-caches-v4-0-8a636982b4a4@suse.cz> <20250425-slub-percpu-caches-v4-2-8a636982b4a4@suse.cz> From: Vlastimil Babka In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Action: no action X-Stat-Signature: msufdk761nq9w65teo4inirqoq1wouf1 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 60A83140002 X-HE-Tag: 1747228054-136709 X-HE-Meta: U2FsdGVkX19sxhnSCgMSZLcPw8Y5lamXtoTTDyHhfVkGXGfa84tmViCp1qJuuaNxGGlwZ3VVhV87Y4DdwqSsYLCvfta5hkF9McmY1b/u/q1rLsKuifd8atUUp6DAIHoVzGuzAJjhN5nA9jc/zee0H1aWGPTqd+4lmL0VYpgWx7ySwHTFfJJYPY71N5M56FbtwnCoyKKKp4KaYZN2LqEaRv65rtzrgKQuJ1Q+TF+zIaYteNzCt3pRQO/6rNIYgqHLjTuSBk41PXV74ubU3DKFz99Tr2npO4rLBCbF6kiA8g7XnEnMGTxAYnBvZ5hiTO5kCRI83g4ixCGzM4vkciSV0SVrG0G/j3ZmSagI2iAB3asYpZLRG0oPCGdGXBwMrxXtZOCJWw4GQYGg7zrfm8Eq9t6uuRGwpqwdTFlKu5DpVnQXy4J+pmokHbR++vS7U7UZSe8uS3CxCgxAjLhQWez55+uQvFcMM3U58U6qbkfiR3jhsuEzmUu95xCeg9yNRga2DieusCyzHaiTRgHBa5MFzRLnCWydeMvxlu61zAy8Lel1JbPyeXZTTHJVAivkRwWKQmZEnERHLQZjJEjpIThW4kP7mIdjybTfv1m6PQk5+Ji3zESofomPFgXmvJfHmek1V+MPsfqk2hJRsH2/XZDinEbfwDcFL/YGSzTEgX+ENTsuY0O6xkmGVlxaG9XQ8CSLw97PxAh9sxmiz7DVWL/qmNru4vZ14r0sSJLREu1AdOqciq+vTGmylGodoeNkMFvyp+OJbfha5AAAZD4eX7HUPaLdJdGL1rWUrP9RHghb74oqpGHWuvArPmf70E+KhD64+EVbNly3uz3+CNqdbaWV7dpQZucpEHIyVwpeLhn28AsPpgr0ko1Q5Qj4eQb72x8Sl4D3T5Prh7X3pBVhPwIQihk/FmENLXENSBsr9jifsSVRuyb1Zl776WMAj7m6E3PJ99BprDxEjQLjet7qTbx hSMHMDmH FOfpGStaXLSieXzy9+tQ2CialkSotjr7WPZgZMn98bCFo3wzpwD1G0gX5gIvMrAUoPDCbS5oqSANS1s85dCKYBI6VCRN5kauN8kE7cZCTxe4LmzcOrCK82a//miRXFdI9T4bA1PJC+fxYGNgJmtmuePutMl//RtVUootVD+lrA3aT4MOwi8d3k1ETXWM3U1uiWHmbDpvQUAzTUajmlfTvxPckJBU9aC8sYvl6UE15b0rxasVbX+LslJZGZPp7kMz3Y7JC6PMpa9jVWKPWUF5sOFBbDngsvqW4LCOJJJNpMHRWszJuzODogM+pX0VzZIsU0uga6Tx4xWuCyVh6bW0dG4t2G4c9ZPlr9NMcZJk6uGmY0IM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 4/29/25 09:36, Harry Yoo wrote: > On Fri, Apr 25, 2025 at 10:27:22AM +0200, Vlastimil Babka wrote: >> Extend the sheaf infrastructure for more efficient kfree_rcu() handling. >> For caches with sheaves, on each cpu maintain a rcu_free sheaf in >> addition to main and spare sheaves. >> >> kfree_rcu() operations will try to put objects on this sheaf. Once full, >> the sheaf is detached and submitted to call_rcu() with a handler that >> will try to put it in the barn, or flush to slab pages using bulk free, >> when the barn is full. Then a new empty sheaf must be obtained to put >> more objects there. >> >> It's possible that no free sheaves are available to use for a new >> rcu_free sheaf, and the allocation in kfree_rcu() context can only use >> GFP_NOWAIT and thus may fail. In that case, fall back to the existing >> kfree_rcu() implementation. >> >> Expected advantages: >> - batching the kfree_rcu() operations, that could eventually replace the >> existing batching >> - sheaves can be reused for allocations via barn instead of being >> flushed to slabs, which is more efficient >> - this includes cases where only some cpus are allowed to process rcu >> callbacks (Android) >> >> Possible disadvantage: >> - objects might be waiting for more than their grace period (it is >> determined by the last object freed into the sheaf), increasing memory >> usage - but the existing batching does that too. >> >> Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny >> implementation favors smaller memory footprint over performance. >> >> Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to >> count how many kfree_rcu() used the rcu_free sheaf successfully and how >> many had to fall back to the existing implementation. >> >> Signed-off-by: Vlastimil Babka >> --- > > Looks good to me, > Reviewed-by: Harry Yoo Thanks! > >> +/* Legal flag mask for kmem_cache_create(), for various configurations */ > > nit: I think now this line should be removed? Yeah looks like rebasing mistake. Removed. >> #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \ >> SLAB_CACHE_DMA32 | SLAB_PANIC | \ >> SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \ >> diff --git a/mm/slab_common.c b/mm/slab_common.c >> index 4f295bdd2d42355af6311a799955301005f8a532..6c3b90f03cb79b57f426824450f576a977d85c53 100644 >> --- a/mm/slab_common.c >> +++ b/mm/slab_common.c >> diff --git a/mm/slub.c b/mm/slub.c >> index ae3e80ad9926ca15601eef2f2aa016ca059498f8..6f31a27b5d47fa6621fa8af6d6842564077d4b60 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -5304,6 +5340,140 @@ bool free_to_pcs(struct kmem_cache *s, void *object) >> return true; >> } >> >> +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) >> +{ >> + struct slub_percpu_sheaves *pcs; >> + struct slab_sheaf *rcu_sheaf; >> + >> + if (!local_trylock(&s->cpu_sheaves->lock)) >> + goto fail; >> + >> + pcs = this_cpu_ptr(s->cpu_sheaves); >> + >> + if (unlikely(!pcs->rcu_free)) { >> + >> + struct slab_sheaf *empty; > > nit: should we grab the spare sheaf here if it's empty? Hmm yeah why not. But only completely empty. Done, thanks! >> + >> + empty = barn_get_empty_sheaf(pcs->barn); >> + >> + if (empty) { >> + pcs->rcu_free = empty; >> + goto do_free; >> + } >> + >> + local_unlock(&s->cpu_sheaves->lock); >> + >> + empty = alloc_empty_sheaf(s, GFP_NOWAIT); >> + >> + if (!empty) >> + goto fail; >> + >> /* >> * Bulk free objects to the percpu sheaves. >> * Unlike free_to_pcs() this includes the calls to all necessary hooks >