From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24F92C369C2 for ; Fri, 25 Apr 2025 08:28:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58DDE6B0089; Fri, 25 Apr 2025 04:28:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 515916B0088; Fri, 25 Apr 2025 04:28:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 392536B0089; Fri, 25 Apr 2025 04:28:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 10E406B0088 for ; Fri, 25 Apr 2025 04:28:13 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0C82481753 for ; Fri, 25 Apr 2025 08:28:15 +0000 (UTC) X-FDA: 83371888950.01.C9856F9 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf01.hostedemail.com (Postfix) with ESMTP id 1010340008 for ; Fri, 25 Apr 2025 08:28:12 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf01.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745569693; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GSIBSQvAw8T+6mt/7dd5JmuBDVoOtfBsjS6uA/BmTlQ=; b=HkSYRddSh0KgHMduFsxx1eWF9aGTBlSKUwM8K+3hD5L2u4nZ0pYhmhYiVrdAxmcnDhQuRF qqIjtWIbUusuVi97H/a1soweNi2afOF08K7OY6um/wOUx8r7Imbj8hLfGVsTP6rVWSVK6D fTqFy5za42Y+QPDh68BdBBCyp7qpSs8= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf01.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745569693; a=rsa-sha256; cv=none; b=WmN4iNNf0hybsCFZpBj8FsXzZlRjV1EUzXtEhpFCdhmXNhv1L3CJnrCjKRbSNdtp6mwuMC dXliUkyxf5Ckepexv6s+6cSCo03dRWVuxfIKx6yUhDzVfr/ulc1m8gcyBrdTs/CkeixeOB MJySD+M8NGIhHJb3SQomwLlZr655dpM= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 46AC12116B; Fri, 25 Apr 2025 08:27:53 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 2EA7813A8B; Fri, 25 Apr 2025 08:27:53 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id mKEVC4lHC2htfwAAD6G6ig (envelope-from ); Fri, 25 Apr 2025 08:27:53 +0000 From: Vlastimil Babka Date: Fri, 25 Apr 2025 10:27:27 +0200 Subject: [PATCH v4 7/9] maple_tree: use percpu sheaves for maple_node_cache MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20250425-slub-percpu-caches-v4-7-8a636982b4a4@suse.cz> References: <20250425-slub-percpu-caches-v4-0-8a636982b4a4@suse.cz> In-Reply-To: <20250425-slub-percpu-caches-v4-0-8a636982b4a4@suse.cz> To: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes Cc: Roman Gushchin , Harry Yoo , Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, vbabka@suse.cz X-Mailer: b4 0.14.2 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 1010340008 X-Stat-Signature: b7d4nruitmzk6x3nfc1c8o9pbqe8547g X-HE-Tag: 1745569692-977752 X-HE-Meta: U2FsdGVkX18lX/YeLMPt/UavYRnQNOJLdXxeiStH2xxwxhe5SONNgINc2cwsGSC+CUh6ZSV0ePUVj9NbBZOHpXntO4d6x+2OGPxJ4dvbA6+URKEmFg8jwJqeHbCEDQCp41n1J9MS+FWkxR0jzUh6YAw+tpEnSLRWdyMZwydWlnwKAeYesKM2HkMm4R3z3GsVHB5HINEcpKova+LFZ7FvZvOgjZd0MNf35zVF6Hya5/kG9X7/XHY52pCxU3TJCmNSUUXDAGfZwRbAjUQhKTVVOTLn/gUPl5UKZ5hKy4QKwMyssta9MYePEzldCYBzYvkngASiJxKxmhAEYEfkxFLdqNXbl3/CUNGcAASxAR2gvb/QnEg/1CpUX3y2EPs1BNhcbmHIEhoVzdN6s8ca/EzaarbzQu8dH3yBgF1oGKkyYOZG4Idpl7YCbkqAjGE6gcz5nzbQ70IicEsh3+LQTYLxL/Y8HzK0dMlGUr/pskK2zlUCL8rGWGhGKieUE9dHXWo/AcacK0mzRywt8rRd8D7kDYm3C4ed5rcz0UAI9VJ7WMSnw8wWy+1s/MrVK2/oRyS+gsjA/QNx5+RLpPNYllsinEsic2sYY/nzxTXbattXIVLR6UXhjo/i65vzXAU+HAIelirt4bCm86+P0XYagiykv22KLq5idKYFDOXRhJoPvV91um6Ni4YX02G68B3j3t9MFMQ7ufbbb642uGcWKZUP+8X+Hy/5PriHLGBR/IfaFltM9b7xZNeDLSFPSLGvo7K0s0qKEsBXcVLRXLVDF/rNAQmFXbk6IhMm4rfjOzqt57O0NN1u7WUiWNvQ2JvbzcMP3MfiyOMwC+UfbiX2s5DII8Hfs42kvRnq9IZRW5adjmE7rItur4xi0D1Od0LTJV3v8tafwZMFgRJIpi0CFQQohIY0okNgrudvV6vY2x39qEnQlFWsv4FWnOU8ce6il50nSRlOwegh+QTurTaJM6a uAzhHAtX ClilZYHW9bSvn1LnmhlrR37FUdXRupDYHH+dF8yTKEl+wgX1t8ySr7SSD3MAyrPu2LGVU/LRzm80+6OFV8+bTg3DuAgBLcgxW32didekPYcNM0bygr1M6/Yrnb+y9Ap5CbeCjAHaEhdA6Bek7mF8gg2s1uL6YeRw2oDdZiyTMk93u1jDrasSecufEkQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Setup the maple_node_cache with percpu sheaves of size 32 to hopefully improve its performance. Change the single node rcu freeing in ma_free_rcu() to use kfree_rcu() instead of the custom callback, which allows the rcu_free sheaf batching to be used. Note there are other users of mt_free_rcu() where larger parts of maple tree are submitted to call_rcu() as a whole, and that cannot use the rcu_free sheaf. But it's still possible for maple nodes freed this way to be reused via the barn, even if only some cpus are allowed to process rcu callbacks. Signed-off-by: Vlastimil Babka Reviewed-by: Suren Baghdasaryan --- lib/maple_tree.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index d0bea23fa4bc9fdd0ca4803a108d3c943f6a0c73..812ba155f3577d1b6ecc779ce9e4e7ded3085d8b 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -208,7 +208,7 @@ static void mt_free_rcu(struct rcu_head *head) static void ma_free_rcu(struct maple_node *node) { WARN_ON(node->parent != ma_parent_ptr(node)); - call_rcu(&node->rcu, mt_free_rcu); + kfree_rcu(node, rcu); } static void mas_set_height(struct ma_state *mas) @@ -6254,9 +6254,14 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) void __init maple_tree_init(void) { + struct kmem_cache_args args = { + .align = sizeof(struct maple_node), + .sheaf_capacity = 32, + }; + maple_node_cache = kmem_cache_create("maple_node", - sizeof(struct maple_node), sizeof(struct maple_node), - SLAB_PANIC, NULL); + sizeof(struct maple_node), &args, + SLAB_PANIC); } /** -- 2.49.0