From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>,
Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>,
Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>,
Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Date: Tue, 2 Dec 2025 16:48:17 +0800 [thread overview]
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos> (raw)
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:31:49 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 3509
Lines: 119
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:33:21 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 8250
Lines: 265
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:31:49 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 3509
Lines: 119
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:44:16 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 17732
Lines: 557
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:31:49 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 3509
Lines: 119
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:33:21 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 8250
Lines: 265
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:31:49 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 3509
Lines: 119
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:45:09 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 36697
Lines: 1141
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:31:49 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 3509
Lines: 119
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:33:21 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 8250
Lines: 265
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:31:49 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 3509
Lines: 119
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:44:16 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 17732
Lines: 557
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:31:49 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 3509
Lines: 119
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:33:21 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 8250
Lines: 265
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
From haoli.tcs@gmail.com Tue Dec 2 16:24:49 2025
Date: Tue, 2 Dec 2025 16:31:49 +0800
From: Hao Li <haoli.tcs@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Suren Baghdasaryan <surenb@google.com>,
"Liam R. Howlett" <Liam.Howlett@oracle.com>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>, Roman Gushchin <roman.gushchin@linux.dev>,
Harry Yoo <harry.yoo@oracle.com>, Uladzislau Rezki <urezki@gmail.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, maple-tree@lists.infradead.org,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Subject: [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf
Message-ID: <fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos>
Mutt-References: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
<20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>
Mutt-Fcc: ~/sent
Status: RO
Content-Length: 3509
Lines: 119
Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from
the per-node barn without requiring an empty sheaf in exchange.
Use this helper in __pcs_replace_empty_main() to change how an empty main
per-CPU sheaf is handled:
- If pcs->spare is NULL and pcs->main is empty, first try to obtain a
full sheaf from the barn via barn_get_full_sheaf(). On success, park
the empty main sheaf in pcs->spare and install the full sheaf as the
new pcs->main.
- If pcs->spare already exists and has objects, keep the existing
behavior of simply swapping pcs->main and pcs->spare.
- Only when both pcs->main and pcs->spare are empty do we fall back to
barn_replace_empty_sheaf() and trade the empty main sheaf into the
barn in exchange for a full one.
This makes the empty-main path more symmetric with __pcs_replace_full_main(),
which for a full main sheaf parks the full sheaf in pcs->spare and pulls an
empty sheaf from the barn. It also matches the documented design more closely:
"When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn."
Signed-off-by: Hao Li <haoli.tcs@gmail.com>
---
* This patch is based on b4/sheaves-for-all branch
mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index a94c64f56504..1fd28aa204e1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s)
s->cpu_sheaves = NULL;
}
+static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn,
+ bool allow_spin)
+{
+ struct slab_sheaf *full = NULL;
+ unsigned long flags;
+
+ if (!data_race(barn->nr_full))
+ return NULL;
+
+ if (likely(allow_spin))
+ spin_lock_irqsave(&barn->lock, flags);
+ else if (!spin_trylock_irqsave(&barn->lock, flags))
+ return NULL;
+
+ if (likely(barn->nr_full)) {
+ full = list_first_entry(&barn->sheaves_full,
+ struct slab_sheaf, barn_list);
+ list_del(&full->barn_list);
+ barn->nr_full--;
+ }
+
+ spin_unlock_irqrestore(&barn->lock, flags);
+
+ return full;
+}
+
static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
bool allow_spin)
{
@@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
struct node_barn *barn;
- bool can_alloc;
+ bool can_alloc, allow_spin;
lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
@@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- if (pcs->spare && pcs->spare->size > 0) {
- swap(pcs->main, pcs->spare);
- return pcs;
- }
+ allow_spin = gfpflags_allow_spinning(gfp);
barn = get_barn(s);
if (!barn) {
@@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ if (!pcs->spare) {
+ full = barn_get_full_sheaf(barn, allow_spin);
+ if (full) {
+ pcs->spare = pcs->main;
+ pcs->main = full;
+ return pcs;
+ }
+ } else if (pcs->spare->size > 0) {
+ swap(pcs->main, pcs->spare);
+ return pcs;
+ }
+
+ /* both main and spare are empty */
+
+ full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
if (full) {
stat(s, BARN_GET);
--
2.50.1
next prev parent reply other threads:[~2025-12-02 8:48 UTC|newest]
Thread overview: 95+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-09-10 8:01 [PATCH v8 00/23] SLUB percpu sheaves Vlastimil Babka
2025-09-10 8:01 ` [PATCH v8 01/23] locking/local_lock: Expose dep_map in local_trylock_t Vlastimil Babka
2025-09-24 16:49 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 02/23] slab: simplify init_kmem_cache_nodes() error handling Vlastimil Babka
2025-09-24 16:52 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 03/23] slab: add opt-in caching layer of percpu sheaves Vlastimil Babka
2025-12-02 8:48 ` Hao Li [this message]
2025-12-02 8:55 ` [PATCH] slub: add barn_get_full_sheaf() and refine empty-main sheaf Hao Li
2025-12-02 9:00 ` slub: add barn_get_full_sheaf() and refine empty-main sheaf replacement Hao Li
2025-12-03 5:46 ` Harry Yoo
2025-12-03 11:15 ` Hao Li
2025-09-10 8:01 ` [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu() operations Vlastimil Babka
2025-09-12 0:38 ` Sergey Senozhatsky
2025-09-12 7:03 ` Vlastimil Babka
2025-09-17 8:30 ` Harry Yoo
2025-09-17 9:55 ` Vlastimil Babka
2025-09-17 11:32 ` Harry Yoo
2025-09-17 12:05 ` Vlastimil Babka
2025-09-17 13:07 ` Harry Yoo
2025-09-17 13:21 ` Vlastimil Babka
2025-09-17 13:34 ` Harry Yoo
2025-09-17 14:14 ` Vlastimil Babka
2025-09-18 8:09 ` Vlastimil Babka
2025-09-19 6:47 ` Harry Yoo
2025-09-19 7:02 ` Vlastimil Babka
2025-09-19 8:59 ` Harry Yoo
2025-09-25 4:35 ` Suren Baghdasaryan
2025-09-25 8:52 ` Harry Yoo
2025-09-25 13:38 ` Suren Baghdasaryan
2025-09-26 10:08 ` Vlastimil Babka
2025-09-26 15:41 ` Suren Baghdasaryan
2025-09-17 11:36 ` Paul E. McKenney
2025-09-17 12:13 ` Vlastimil Babka
2025-10-31 21:32 ` Daniel Gomez
2025-11-03 3:17 ` Harry Yoo
2025-11-05 11:25 ` Vlastimil Babka
2025-11-27 14:00 ` Daniel Gomez
2025-11-27 19:29 ` Suren Baghdasaryan
2025-11-28 11:37 ` [PATCH V1] mm/slab: introduce kvfree_rcu_barrier_on_cache() for cache destruction Harry Yoo
2025-11-28 12:22 ` Harry Yoo
2025-11-28 12:38 ` Daniel Gomez
2025-12-02 9:29 ` Jon Hunter
2025-12-02 10:18 ` Harry Yoo
2025-11-27 11:38 ` [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu() operations Jon Hunter
2025-11-27 11:50 ` Jon Hunter
2025-11-27 12:33 ` Harry Yoo
2025-11-27 12:48 ` Harry Yoo
2025-11-28 8:57 ` Jon Hunter
2025-12-01 6:55 ` Harry Yoo
2025-11-27 13:18 ` Vlastimil Babka
2025-11-28 8:59 ` Jon Hunter
2025-09-10 8:01 ` [PATCH v8 05/23] slab: sheaf prefilling for guaranteed allocations Vlastimil Babka
2025-09-10 8:01 ` [PATCH v8 06/23] slab: determine barn status racily outside of lock Vlastimil Babka
2025-09-10 8:01 ` [PATCH v8 07/23] slab: skip percpu sheaves for remote object freeing Vlastimil Babka
2025-09-25 16:14 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 08/23] slab: allow NUMA restricted allocations to use percpu sheaves Vlastimil Babka
2025-09-25 16:27 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 09/23] maple_tree: remove redundant __GFP_NOWARN Vlastimil Babka
2025-09-10 8:01 ` [PATCH v8 10/23] tools/testing/vma: clean up stubs in vma_internal.h Vlastimil Babka
2025-09-10 8:01 ` [PATCH v8 11/23] maple_tree: Drop bulk insert support Vlastimil Babka
2025-09-25 16:38 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 12/23] tools/testing/vma: Implement vm_refcnt reset Vlastimil Babka
2025-09-25 16:38 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 13/23] tools/testing: Add support for changes to slab for sheaves Vlastimil Babka
2025-09-26 23:28 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 14/23] mm, vma: use percpu sheaves for vm_area_struct cache Vlastimil Babka
2025-09-10 8:01 ` [PATCH v8 15/23] maple_tree: use percpu sheaves for maple_node_cache Vlastimil Babka
2025-09-12 2:20 ` Liam R. Howlett
2025-10-16 15:16 ` D, Suneeth
2025-10-16 16:15 ` Vlastimil Babka
2025-10-17 18:26 ` D, Suneeth
2025-09-10 8:01 ` [PATCH v8 16/23] tools/testing: include maple-shim.c in maple.c Vlastimil Babka
2025-09-26 23:45 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 17/23] testing/radix-tree/maple: Hack around kfree_rcu not existing Vlastimil Babka
2025-09-26 23:53 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 18/23] maple_tree: Use kfree_rcu in ma_free_rcu Vlastimil Babka
2025-09-17 11:46 ` Harry Yoo
2025-09-27 0:05 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 19/23] maple_tree: Replace mt_free_one() with kfree() Vlastimil Babka
2025-09-27 0:06 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 20/23] tools/testing: Add support for prefilled slab sheafs Vlastimil Babka
2025-09-27 0:28 ` Suren Baghdasaryan
2025-09-10 8:01 ` [PATCH v8 21/23] maple_tree: Prefilled sheaf conversion and testing Vlastimil Babka
2025-09-27 1:08 ` Suren Baghdasaryan
2025-09-29 7:30 ` Vlastimil Babka
2025-09-29 16:51 ` Liam R. Howlett
2025-09-10 8:01 ` [PATCH v8 22/23] maple_tree: Add single node allocation support to maple state Vlastimil Babka
2025-09-27 1:17 ` Suren Baghdasaryan
2025-09-29 7:39 ` Vlastimil Babka
2025-09-10 8:01 ` [PATCH v8 23/23] maple_tree: Convert forking to use the sheaf interface Vlastimil Babka
2025-10-07 6:34 ` [PATCH v8 00/23] SLUB percpu sheaves Christoph Hellwig
2025-10-07 8:03 ` Vlastimil Babka
2025-10-08 6:04 ` Christoph Hellwig
2025-10-15 8:32 ` Vlastimil Babka
2025-10-22 6:47 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=fqbhzbjw3nvbylvxayboyug7qayfref4xti7udztygd6zbhcun@b7s2yjihrwos \
--to=haoli.tcs@gmail.com \
--cc=Liam.Howlett@oracle.com \
--cc=cl@gentwo.org \
--cc=harry.yoo@oracle.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=maple-tree@lists.infradead.org \
--cc=rcu@vger.kernel.org \
--cc=rientjes@google.com \
--cc=roman.gushchin@linux.dev \
--cc=sidhartha.kumar@oracle.com \
--cc=surenb@google.com \
--cc=urezki@gmail.com \
--cc=vbabka@suse.cz \
--cc=venkat88@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox