From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67059C3ABC9 for ; Thu, 15 May 2025 08:59:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26BC88D0002; Thu, 15 May 2025 04:59:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 21AA08D0001; Thu, 15 May 2025 04:59:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E3C58D0002; Thu, 15 May 2025 04:59:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E4FC88D0001 for ; Thu, 15 May 2025 04:59:18 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 238D7BF1E2 for ; Thu, 15 May 2025 08:59:19 +0000 (UTC) X-FDA: 83444543238.02.736901E Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf23.hostedemail.com (Postfix) with ESMTP id 8DB1114000A for ; Thu, 15 May 2025 08:59:16 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Pnphta3p; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NmIXdFnU; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Pnphta3p; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NmIXdFnU; spf=pass (imf23.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747299557; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QFkGUgAQs5DlUjM0ydlG9GUUFq/7KcW6NX7M7pmvjeQ=; b=jj25zhpJvUkvs9CmWsBrUqKnW8PZE+jh5yrjC4zc4AclMl7y6BtcWZFipeVHZ5tjf8UeD7 WhCL1PQK0dCGSS83Xswg/eekSXgHNiO1LUQt9KktY1IVZ0NVys+ToB5RsMkzjSHLItNSV9 HnoC8JhrHMtOYIFtkKxwAJA1MhFp1j4= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Pnphta3p; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NmIXdFnU; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Pnphta3p; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NmIXdFnU; spf=pass (imf23.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747299557; a=rsa-sha256; cv=none; b=Fp7m9AkDd1hl5XAj4o9+qrhRbCAGeKjGq6tavYs81CKR54rApLhc/HHjhCbZhcg2yLKiL4 2OfSdhlasVXkbHWuCpAV+D0ibDlwHmIJz4Sr2dPX36kqBlZM/DSJNPPa2YmEz/t/TShAQM iDK2RwLTO+tAOf+lO0DEq5R0jwOdjgk= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id B45651F391; Thu, 15 May 2025 08:59:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1747299554; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QFkGUgAQs5DlUjM0ydlG9GUUFq/7KcW6NX7M7pmvjeQ=; b=Pnphta3pj/hPd36vEObXKs7oZ0yRXLXPve+d4n7PRI8BWxbpSlvygZrWnAcKZ3FH9Xl2c1 Be8QO7NDCZVnPH+8GY5eKp+0IXQbjQVeZqJw169CCHTYPbe+EqedilZ9Qqzsq/lb3ybn3C LZkD6mMSDDZZ+dUG0KyyHODqR+TR9VI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1747299554; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QFkGUgAQs5DlUjM0ydlG9GUUFq/7KcW6NX7M7pmvjeQ=; b=NmIXdFnUXuSNu01AoL3nca3Ijs1g4XLQ1IEDXnBpVbRnnWbWWz48AeD2HD4wmnz+mbOu+O 5dBO7cFrMvmfO4Cw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1747299554; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QFkGUgAQs5DlUjM0ydlG9GUUFq/7KcW6NX7M7pmvjeQ=; b=Pnphta3pj/hPd36vEObXKs7oZ0yRXLXPve+d4n7PRI8BWxbpSlvygZrWnAcKZ3FH9Xl2c1 Be8QO7NDCZVnPH+8GY5eKp+0IXQbjQVeZqJw169CCHTYPbe+EqedilZ9Qqzsq/lb3ybn3C LZkD6mMSDDZZ+dUG0KyyHODqR+TR9VI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1747299554; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QFkGUgAQs5DlUjM0ydlG9GUUFq/7KcW6NX7M7pmvjeQ=; b=NmIXdFnUXuSNu01AoL3nca3Ijs1g4XLQ1IEDXnBpVbRnnWbWWz48AeD2HD4wmnz+mbOu+O 5dBO7cFrMvmfO4Cw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 9DBEB139D0; Thu, 15 May 2025 08:59:14 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id D5A2JuKsJWgTRAAAD6G6ig (envelope-from ); Thu, 15 May 2025 08:59:14 +0000 Message-ID: <938fa0c7-86c3-44d4-b583-0612458aed98@suse.cz> Date: Thu, 15 May 2025 10:59:14 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 9/9] mm, slub: skip percpu sheaves for remote object freeing Content-Language: en-US To: Harry Yoo Cc: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes , Roman Gushchin , Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org References: <20250425-slub-percpu-caches-v4-0-8a636982b4a4@suse.cz> <20250425-slub-percpu-caches-v4-9-8a636982b4a4@suse.cz> From: Vlastimil Babka In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Action: no action X-Rspam-User: X-Rspamd-Queue-Id: 8DB1114000A X-Rspamd-Server: rspam09 X-Stat-Signature: 4dgoz1p33kb3hz3hmnwnpbnz61iyfqfe X-HE-Tag: 1747299556-399727 X-HE-Meta: U2FsdGVkX19Ia7vQaoBD1etKK55H80E/QmatWnmQ400DTodZqbm/zYaWi3qBVEIwoo7mj8Jq3zn252MEi4jnXg1tHPO/wXb5q5o89KnXxZiSARly+l/UXmN+1wuJftuDdtA7y6Vg2+Io83RJVabMqi41nOjaJww4FL0Z8jqDrYteQ9swbMKN9T4ygBLyzGEqtciYzr9kVzEJHuhdxW6EzkJifJXAhhnBbHjJ5seWWrov+3V4XwohDQ5FRUHxdKRcuaDDf+izRwZxlzJB9eGbgDWpAWYRGItyCiOAI1h6+xb1Qmci/NGtyqKziPiw+DzUvh+nLep8dwAakzFMsTj9e6R66pak1OJi5j98E4cJd13pn12ifer6APJHAAsFkKgoNtoUEMiMvy80ofNRyjjAazwPplTnoGHKNoSqNL+aqf4rg8mbUBimp59Jfs7+Uay1EVr5Elihs3FRub/umFSAQmWV3zVPwDD8htI0ZGl1ObCZTO+7liFtv0T8pJL2Wxjy47x9fIeWLB2YojwAdeDS1OaGT6XF8NIP5ss2uy7h9HsxqsKaTD/9/VW+TmOQ5tW/7g6BIQpmYOZtM1ym0kZyOn/RL7147U9RDqn5hien0fkJfN5yCUC6Uf/do98jgg08xK0WRsHUsfoALLH+zLMB9hXLSASg8H2cvS0kygEEU9aysWD2Le9xqYNJZ/CsCFDuwmj2AobMQuixRme28MpYAESoBIrjlLfQGWPjDYVl/EsoCRv0ti/92l015OskVBzaOBFYdVY8v0GkoeGRoKXJHKyavxdgYyIDXUZGZ10DLlSuZGpIYm/T69cSThnyu+apk+u23IxXk8kD7wX77gom4iE8Wnp2PIvQMz9Y9zeyCZe/WmtEADVM6gYzNEOhWS3MpkKwe6K7skUyp78rjmNdm3ZAdhYvTFR3sQXbAAnh+tLIlcPJj/XAdk+pB8c8/z3gRaRVBXrjowSsH5MpjM7 tPd6m3mr zkGVm8jcs+9yDDzU//uy980XA6gnwE+SDB1mrK0zxmFftxG+nCiXzzwWhQ0YbYIgTJYM4vs85icLOth1pUo7uv2oSCYeTES4zCJhBx8HF7qHu6oTVy+4wbrwnXh4jzkDinHZC3otAO1Rx+MggmBVYquA6yXD4IApcSOZhTBR9enkfjjrdIEPWTgNV44XXVDiBQf8U5E/AMZQhPV8dtkns7/cNgpSMmoabND3LmD35vhnb8QTEj9W2UThai2uG97p63uCJ7jpip+PPCUl8xHjDVZSLf3OREqQxXPWG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 5/7/25 12:39, Harry Yoo wrote: > On Fri, Apr 25, 2025 at 10:27:29AM +0200, Vlastimil Babka wrote: >> Since we don't control the NUMA locality of objects in percpu sheaves, >> allocations with node restrictions bypass them. Allocations without >> restrictions may however still expect to get local objects with high >> probability, and the introduction of sheaves can decrease it due to >> freed object from a remote node ending up in percpu sheaves. >> >> The fraction of such remote frees seems low (5% on an 8-node machine) >> but it can be expected that some cache or workload specific corner cases >> exist. We can either conclude that this is not a problem due to the low >> fraction, or we can make remote frees bypass percpu sheaves and go >> directly to their slabs. This will make the remote frees more expensive, >> but if if's only a small fraction, most frees will still benefit from >> the lower overhead of percpu sheaves. >> >> This patch thus makes remote object freeing bypass percpu sheaves, >> including bulk freeing, and kfree_rcu() via the rcu_free sheaf. However >> it's not intended to be 100% guarantee that percpu sheaves will only >> contain local objects. The refill from slabs does not provide that >> guarantee in the first place, and there might be cpu migrations >> happening when we need to unlock the local_lock. Avoiding all that could >> be possible but complicated so we can leave it for later investigation >> whether it would be worth it. It can be expected that the more selective >> freeing will itself prevent accumulation of remote objects in percpu >> sheaves so any such violations would have only short-term effects. >> >> Another possible optimization to investigate is whether it would be >> beneficial for node-restricted or strict_numa allocations to attempt to >> obtain an object from percpu sheaves if the node or mempolicy (i.e. >> MPOL_LOCAL) happens to want the local node of the allocating cpu. Right >> now such allocations bypass sheaves, but they could probably look first >> whether the first available object in percpu sheaves is local, and with >> high probability succeed - and only bypass the sheaves in cases it's >> not local. >> >> Signed-off-by: Vlastimil Babka >> --- >> mm/slab_common.c | 7 +++++-- >> mm/slub.c | 43 +++++++++++++++++++++++++++++++++++++------ >> 2 files changed, 42 insertions(+), 8 deletions(-) >> >> diff --git a/mm/slub.c b/mm/slub.c >> index cc273cc45f632e16644355831132cdc391219cec..2bf83e2b85b23f4db2b311edaded4bef6b7d01de 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -5924,8 +5948,15 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, >> if (unlikely(!slab_free_hook(s, object, slab_want_init_on_free(s), false))) >> return; >> >> - if (!s->cpu_sheaves || !free_to_pcs(s, object)) >> - do_slab_free(s, slab, object, object, 1, addr); >> + if (s->cpu_sheaves) { >> + if (likely(!IS_ENABLED(CONFIG_NUMA) || >> + slab_nid(slab) == numa_node_id())) { >> + free_to_pcs(s, object); > > Shouldn't it call do_slab_free() when free_to_pcs() failed? Oops yes, thanks! > >> + return; >> + } >> + } >> + >> + do_slab_free(s, slab, object, object, 1, addr); >> } >> >> #ifdef CONFIG_MEMCG >> >> -- >> 2.49.0 >> >> >