From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F29A1C7619A for ; Wed, 12 Apr 2023 08:12:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F41C900008; Wed, 12 Apr 2023 04:12:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 854C7900002; Wed, 12 Apr 2023 04:12:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F562900008; Wed, 12 Apr 2023 04:12:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5F957900002 for ; Wed, 12 Apr 2023 04:12:45 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 24726120F8A for ; Wed, 12 Apr 2023 08:12:45 +0000 (UTC) X-FDA: 80672022690.23.4930CF9 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf03.hostedemail.com (Postfix) with ESMTP id 593E020003 for ; Wed, 12 Apr 2023 08:12:43 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=linuxfoundation.org header.s=korg header.b=v7sKqu0O; spf=pass (imf03.hostedemail.com: domain of gregkh@linuxfoundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=gregkh@linuxfoundation.org; dmarc=pass (policy=none) header.from=linuxfoundation.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681287163; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:dkim-signature; bh=Cuk3mPZo2lVhl3LvW3fLyLTeaAVSdLP825QRJwxkg+Y=; b=RTFKjdtUr7I4N8yM3/mXyKKLeoUoKbR+1r4QOMkQ93S5Heype4mFJaWnqkoumDeCQg/66t x3TBnMYSFL8ilVYjRUeEdVM3Wx9mDCThdOVHT1afrDJ+bI4a1y5892uMSUBgzKU83dM4o5 VkiQD9mojU/CT+lfg1zotrMjXwyoOfc= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=linuxfoundation.org header.s=korg header.b=v7sKqu0O; spf=pass (imf03.hostedemail.com: domain of gregkh@linuxfoundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=gregkh@linuxfoundation.org; dmarc=pass (policy=none) header.from=linuxfoundation.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681287163; a=rsa-sha256; cv=none; b=ZehP/0KqZQuwsz4V35oeASrQlsJGHw9822KsZBy72En2AfkKGoBseZteluOAepks5niz2R aeJACqDOpzUKfC6xY/cugVLnYBqWaqXee6BbhzXT5XlOnpjUaB5/CWKkZMIQIoZyNo/OKc h0EBSQ6QNYVB0ucXcDPvHmaBYJt8Rb0= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9031E62929; Wed, 12 Apr 2023 08:12:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A154AC433D2; Wed, 12 Apr 2023 08:12:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1681287162; bh=8j2QUhbaRGQ0R4sEUEYTNzqbJsqWI/ZMYpEd3iETdQk=; h=Subject:To:Cc:From:Date:In-Reply-To:From; b=v7sKqu0OTfHnHLl7AXudzojZri4gZvMtZXiAXjZMspf1Tujj6M1kpNZVoR+mcQQdO 2uubZZyLvB3b8MPwbs8YsprIQADEafs87qFtiT+Q57emsMptp5/6AVzeMmLyOYlHAd VBMBr5Rfr6lUwUkNTeLPKWcs5DVdmkbiJ2uPPVlU= Subject: Patch "maple_tree: remove GFP_ZERO from kmem_cache_alloc() and kmem_cache_alloc_bulk()" has been added to the 6.2-stable tree To: Liam.Howlett@Oracle.com,Liam.Howlett@oracle.com,gregkh@linuxfoundation.org,jhladky@redhat.com,linux-mm@kvack.org,maple-tree@lists.infradead.org,willy@infradead.org Cc: From: Date: Wed, 12 Apr 2023 10:12:10 +0200 In-Reply-To: <20230411151055.2910579-2-Liam.Howlett@oracle.com> Message-ID: <2023041210-frown-gibberish-f20a@gregkh> MIME-Version: 1.0 Content-Type: text/plain; charset=ANSI_X3.4-1968 Content-Transfer-Encoding: 8bit X-stable: commit X-Patchwork-Hint: ignore X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: deootwraqdhorx17riky1se6o9fp13bk X-Rspamd-Queue-Id: 593E020003 X-HE-Tag: 1681287163-158654 X-HE-Meta: U2FsdGVkX1+qBHKI8WHWT9F+p5JtUkcmUGEqnuC0BvC0dCKMwN/JXp8SfyAonE012rgTQlqZHcluEilSAAutUGYdZc9cmDm9PKgGkSKe1qzEdNfYL4M5BGrUCb9z/dKW3t0EGMvKOWuUi6pH7SRrsaTC2X4fhmm1WTIrruwgCUYKwfhDFNLkI0pCNMZZ0Dx36djjL/4f+AsX5wCAUV06jIz3VG8s/iHyq5o2ge+FxVDiDYl/LJf8bZLNb8K8atcS8MPPJPwDrfcgXfhkiVyb/BWS4h+mvbsDG6ZBwFbTP3jL0jMuUniGElYIWPl+/TPRFzBp9xVxW2UPkbUQaBYBBGq2wSaSRrJmrQ1GjNjPUoZDPwUfbFr4kJUVOh2KPLHgh/I5STLy1ysPf5VxhkWCaDhMlVtS/UaXIDrSIRJ8blnMHa+mzJ/Z1PjbkdmqgvDSzhGVkZkhzbClgTQzuuQFqv5bHUYk45gzs1Lon4aPkJt7xdEcCXzxfuQvoLDXZoHM7yJMpKwMhEawabbr9BWbNrBIn/W2u9FUvTbTQeswUFxz2xMtIcYHp4K8QYe+ShGXxRTE/4z7bIhl9R6Qdt0kvWI6csbt+tv8M4/U0zusFSwGvJ2st1lwX9rLXjUweTSKiPTYhcIMdw9siaCSKYKPWl4BoBIoANDAc5dV0t+uHLZNZDKD0qTGYj3Trw7j2PNt4X9M0wwPGIOLl4F17XvRyhCg7MnV6UKDP1uQBGerCYtM7l1LGviu37s0VzKPsrlYqV+Wijlt6QSLQBz2MfWq2GZNxJqacATkauUKehFXtDrkMpu+7+8O5cUdB+kx2f7E0rIvvj1Z7sBJHcJcA0MU/gkhhr24bddLmlQ/Hq7s7fYaJGBkoH+CuUbnPdrinVbTieWCFAtOQqsxHFmTQXyN5EqHcOSmXAX8cojQGg9WWNNlBmOx4/Q5M+wDJOk/bX33U4u2dFrl03u1f1YlZhx DVodNmht zywVum8HbfVES/ROxWNqCVIsCHJlZlQ8+KI49lLuI1+xx91pAkfYp6pMQLSp3hMZ9k9BvMbA2glSn0Xa/d010N5e3/HgiGtATNxXChRUChpPQXtmOSlcQaM7BSw3Kdxnv4ErNZac1hen+cFnJhvcPsVCzU8Q3oALiheGFtrhjZcvEtQpnuPIez7P5Z2go9B5JsFvrTHWfSY3DVXww/3NqGWbinV/Nlo4L5cnqB5aOPei4ntsMxaHUtt9E3V1Z+Vnp9IW0DR2kUabbIh1WFS8IVlM6SvNsh8Ps3v5icLprSq8Nd7T2gEtMGYJtIGPNJ48PgQTiTcHHfrcBZtWkjsFOeJZlUoCJ+UN2u6jzOCw9qF/ZJJQtw71MV8lhZMRlxpv9FNeqpEWEaLlJsyMKMC1A2dzoIqo76TVnsMMjwne8oiRs5L1Q5Tc/pHVWlI1uCePMTr3OnLpqVyUNKsdVHNacp0NVzTS1/F+xHDKmt+SHfaQNZDgPkhW8tdL+YtQBsT3Lx9o9ViUA1IXKCrZkoy7/PiUSsYs7mGvbCc0KBBzjs2DeiY+C8J9r+1k8huo5zo9fCVHwl4U5Xbv0fYee17Zz/SSHQ4vhSjYjg3b1YkwJtyzikj5VMqzYmZhz8ZnAnmVs77zQT1LAgwmlhyp8J5/FsjuS9A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a note to let you know that I've just added the patch titled maple_tree: remove GFP_ZERO from kmem_cache_alloc() and kmem_cache_alloc_bulk() to the 6.2-stable tree which can be found at: http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary The filename of the patch is: maple_tree-remove-gfp_zero-from-kmem_cache_alloc-and-kmem_cache_alloc_bulk.patch and it can be found in the queue-6.2 subdirectory. If you, or anyone else, feels it should not be added to the stable tree, please let know about it. >From stable-owner@vger.kernel.org Tue Apr 11 17:13:02 2023 From: "Liam R. Howlett" Date: Tue, 11 Apr 2023 11:10:42 -0400 Subject: maple_tree: remove GFP_ZERO from kmem_cache_alloc() and kmem_cache_alloc_bulk() To: Greg Kroah-Hartman , stable@vger.kernel.org Cc: maple-tree@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Liam R. Howlett" , Liam Howlett , Jirka Hladky , Matthew Wilcox Message-ID: <20230411151055.2910579-2-Liam.Howlett@oracle.com> From: "Liam R. Howlett" commit 541e06b772c1aaffb3b6a245ccface36d7107af2 upstream. Preallocations are common in the VMA code to avoid allocating under certain locking conditions. The preallocations must also cover the worst-case scenario. Removing the GFP_ZERO flag from the kmem_cache_alloc() (and bulk variant) calls will reduce the amount of time spent zeroing memory that may not be used. Only zero out the necessary area to keep track of the allocations in the maple state. Zero the entire node prior to using it in the tree. This required internal changes to node counting on allocation, so the test code is also updated. This restores some micro-benchmark performance: up to +9% in mmtests mmap1 by my testing +10% to +20% in mmap, mmapaddr, mmapmany tests reported by Red Hat Link: https://bugzilla.redhat.com/show_bug.cgi?id=2149636 Link: https://lkml.kernel.org/r/20230105160427.2988454-1-Liam.Howlett@oracle.com Cc: stable@vger.kernel.org Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Reported-by: Jirka Hladky Suggested-by: Matthew Wilcox (Oracle) Signed-off-by: Greg Kroah-Hartman --- lib/maple_tree.c | 80 ++++++++++++++++++++------------------- tools/testing/radix-tree/maple.c | 18 ++++---- 2 files changed, 52 insertions(+), 46 deletions(-) --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -149,13 +149,12 @@ struct maple_subtree_state { /* Functions */ static inline struct maple_node *mt_alloc_one(gfp_t gfp) { - return kmem_cache_alloc(maple_node_cache, gfp | __GFP_ZERO); + return kmem_cache_alloc(maple_node_cache, gfp); } static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes) { - return kmem_cache_alloc_bulk(maple_node_cache, gfp | __GFP_ZERO, size, - nodes); + return kmem_cache_alloc_bulk(maple_node_cache, gfp, size, nodes); } static inline void mt_free_bulk(size_t size, void __rcu **nodes) @@ -1128,9 +1127,10 @@ static inline struct maple_node *mas_pop { struct maple_alloc *ret, *node = mas->alloc; unsigned long total = mas_allocated(mas); + unsigned int req = mas_alloc_req(mas); /* nothing or a request pending. */ - if (unlikely(!total)) + if (WARN_ON(!total)) return NULL; if (total == 1) { @@ -1140,27 +1140,25 @@ static inline struct maple_node *mas_pop goto single_node; } - if (!node->node_count) { + if (node->node_count == 1) { /* Single allocation in this node. */ mas->alloc = node->slot[0]; - node->slot[0] = NULL; mas->alloc->total = node->total - 1; ret = node; goto new_head; } - node->total--; - ret = node->slot[node->node_count]; - node->slot[node->node_count--] = NULL; + ret = node->slot[--node->node_count]; + node->slot[node->node_count] = NULL; single_node: new_head: - ret->total = 0; - ret->node_count = 0; - if (ret->request_count) { - mas_set_alloc_req(mas, ret->request_count + 1); - ret->request_count = 0; + if (req) { + req++; + mas_set_alloc_req(mas, req); } + + memset(ret, 0, sizeof(*ret)); return (struct maple_node *)ret; } @@ -1179,21 +1177,20 @@ static inline void mas_push_node(struct unsigned long count; unsigned int requested = mas_alloc_req(mas); - memset(reuse, 0, sizeof(*reuse)); count = mas_allocated(mas); - if (count && (head->node_count < MAPLE_ALLOC_SLOTS - 1)) { - if (head->slot[0]) - head->node_count++; - head->slot[head->node_count] = reuse; + reuse->request_count = 0; + reuse->node_count = 0; + if (count && (head->node_count < MAPLE_ALLOC_SLOTS)) { + head->slot[head->node_count++] = reuse; head->total++; goto done; } reuse->total = 1; if ((head) && !((unsigned long)head & 0x1)) { - head->request_count = 0; reuse->slot[0] = head; + reuse->node_count = 1; reuse->total += head->total; } @@ -1212,7 +1209,6 @@ static inline void mas_alloc_nodes(struc { struct maple_alloc *node; unsigned long allocated = mas_allocated(mas); - unsigned long success = allocated; unsigned int requested = mas_alloc_req(mas); unsigned int count; void **slots = NULL; @@ -1228,24 +1224,29 @@ static inline void mas_alloc_nodes(struc WARN_ON(!allocated); } - if (!allocated || mas->alloc->node_count == MAPLE_ALLOC_SLOTS - 1) { + if (!allocated || mas->alloc->node_count == MAPLE_ALLOC_SLOTS) { node = (struct maple_alloc *)mt_alloc_one(gfp); if (!node) goto nomem_one; - if (allocated) + if (allocated) { node->slot[0] = mas->alloc; + node->node_count = 1; + } else { + node->node_count = 0; + } - success++; mas->alloc = node; + node->total = ++allocated; requested--; } node = mas->alloc; + node->request_count = 0; while (requested) { max_req = MAPLE_ALLOC_SLOTS; - if (node->slot[0]) { - unsigned int offset = node->node_count + 1; + if (node->node_count) { + unsigned int offset = node->node_count; slots = (void **)&node->slot[offset]; max_req -= offset; @@ -1259,15 +1260,13 @@ static inline void mas_alloc_nodes(struc goto nomem_bulk; node->node_count += count; - /* zero indexed. */ - if (slots == (void **)&node->slot) - node->node_count--; - - success += count; + allocated += count; node = node->slot[0]; + node->node_count = 0; + node->request_count = 0; requested -= count; } - mas->alloc->total = success; + mas->alloc->total = allocated; return; nomem_bulk: @@ -1276,7 +1275,7 @@ nomem_bulk: nomem_one: mas_set_alloc_req(mas, requested); if (mas->alloc && !(((unsigned long)mas->alloc & 0x1))) - mas->alloc->total = success; + mas->alloc->total = allocated; mas_set_err(mas, -ENOMEM); return; @@ -5725,6 +5724,7 @@ int mas_preallocate(struct ma_state *mas void mas_destroy(struct ma_state *mas) { struct maple_alloc *node; + unsigned long total; /* * When using mas_for_each() to insert an expected number of elements, @@ -5747,14 +5747,20 @@ void mas_destroy(struct ma_state *mas) } mas->mas_flags &= ~(MA_STATE_BULK|MA_STATE_PREALLOC); - while (mas->alloc && !((unsigned long)mas->alloc & 0x1)) { + total = mas_allocated(mas); + while (total) { node = mas->alloc; mas->alloc = node->slot[0]; - if (node->node_count > 0) - mt_free_bulk(node->node_count, - (void __rcu **)&node->slot[1]); + if (node->node_count > 1) { + size_t count = node->node_count - 1; + + mt_free_bulk(count, (void __rcu **)&node->slot[1]); + total -= count; + } kmem_cache_free(maple_node_cache, node); + total--; } + mas->alloc = NULL; } EXPORT_SYMBOL_GPL(mas_destroy); --- a/tools/testing/radix-tree/maple.c +++ b/tools/testing/radix-tree/maple.c @@ -173,11 +173,11 @@ static noinline void check_new_node(stru if (!MAPLE_32BIT) { if (i >= 35) - e = i - 35; + e = i - 34; else if (i >= 5) - e = i - 5; + e = i - 4; else if (i >= 2) - e = i - 2; + e = i - 1; } else { if (i >= 4) e = i - 4; @@ -305,17 +305,17 @@ static noinline void check_new_node(stru MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM)); MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL)); MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1); - MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1); + MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS); mn = mas_pop_node(&mas); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS); - MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 2); + MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1); mas_push_node(&mas, mn); MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1); - MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1); + MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS); /* Check the limit of pop/push/pop */ mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 2); /* Request */ @@ -323,14 +323,14 @@ static noinline void check_new_node(stru MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM)); MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL)); MT_BUG_ON(mt, mas_alloc_req(&mas)); - MT_BUG_ON(mt, mas.alloc->node_count); + MT_BUG_ON(mt, mas.alloc->node_count != 1); MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1); - MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1); + MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS); mas_push_node(&mas, mn); - MT_BUG_ON(mt, mas.alloc->node_count); + MT_BUG_ON(mt, mas.alloc->node_count != 1); MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); Patches currently in stable-queue which might be from stable-owner@vger.kernel.org are queue-6.2/maple_tree-fix-potential-rcu-issue.patch queue-6.2/maple_tree-add-smp_rmb-to-dead-node-detection.patch queue-6.2/maple_tree-add-rcu-lock-checking-to-rcu-callback-functions.patch queue-6.2/maple_tree-fix-handle-of-invalidated-state-in-mas_wr_store_setup.patch queue-6.2/maple_tree-reduce-user-error-potential.patch queue-6.2/maple_tree-fix-mas_prev-and-mas_find-state-handling.patch queue-6.2/maple_tree-remove-gfp_zero-from-kmem_cache_alloc-and-kmem_cache_alloc_bulk.patch queue-6.2/maple_tree-be-more-cautious-about-dead-nodes.patch queue-6.2/mm-enable-maple-tree-rcu-mode-by-default.patch queue-6.2/maple_tree-detect-dead-nodes-in-mas_start.patch queue-6.2/maple_tree-fix-freeing-of-nodes-in-rcu-mode.patch queue-6.2/maple_tree-remove-extra-smp_wmb-from-mas_dead_leaves.patch queue-6.2/maple_tree-refine-ma_state-init-from-mas_start.patch