From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5B5CC021B5 for ; Sat, 22 Feb 2025 23:08:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3183E6B007B; Sat, 22 Feb 2025 18:08:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C7F16B0082; Sat, 22 Feb 2025 18:08:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 142C06B0083; Sat, 22 Feb 2025 18:08:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EAA076B007B for ; Sat, 22 Feb 2025 18:08:56 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6499A1CBA33 for ; Sat, 22 Feb 2025 23:08:56 +0000 (UTC) X-FDA: 83149122672.30.9AE5B67 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf13.hostedemail.com (Postfix) with ESMTP id 8652120002 for ; Sat, 22 Feb 2025 23:08:54 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2XooBDPi; spf=pass (imf13.hostedemail.com: domain of surenb@google.com designates 209.85.160.175 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740265734; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/Th+w5zeXQufZOVmAeN72xK4zjFpK20IFV72ncsNhk0=; b=H2XlYBww6azQTH0WJ7hCEjfHcQ+AsCUwi+1MCuT/4/tXD1Eb/Zf/pTZIIhxQmX4C9RlWPY LspUlNCny4j4kBB59t2USXXxZfP/IQlDCpjZb8lRX8v20inhSvi2VBylbZEtZ+ziI/4D63 j6dHTmc67+RAXx65ND6byK+FQOLLoGo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2XooBDPi; spf=pass (imf13.hostedemail.com: domain of surenb@google.com designates 209.85.160.175 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740265734; a=rsa-sha256; cv=none; b=8WfyI+Me3J/Bd9GYvOBa8FX37N215IIIVsFXjdHjJydV1gNQh9qkoVdKryMpT8mwUladmf 7iHrgItHe+kdHTxvN92PVGrwmaduo+f4JgwDwmSlXzdqw3BTSk0X4ce+xVJszZyf2l93ZB 053F3y8Gpx0SozCc6J9ZL5rpdDfBCs0= Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-472098e6e75so220241cf.1 for ; Sat, 22 Feb 2025 15:08:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740265733; x=1740870533; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=/Th+w5zeXQufZOVmAeN72xK4zjFpK20IFV72ncsNhk0=; b=2XooBDPiEpUmHhqhGk1C1tDF0jK0576drfJW6jW/gnuc4RVqov0HqhiTMMRjL8baS5 DWGc/h9+1W6cJcEqLKu0QTUwp5hXZI79LwrOfkn17KUpH+myQ6LPPwlnBzktdOmQiVOA 6m+hjymI28BFBYXGsNSQJUamwp149JM+lacXdiBXA6o2nF3+lCdXtWfhZZ6i0pGVCxGm XEdgq/y5ICdOlpWIJYm1nbZnWH4DhnV3gF2ms8EfX+Go5lgqrEJtPsRS3659fd+lZ1rk fUztBRo99EDDZ2wQfSxiCdQInPSDm+kzfT/HeGblc9uFq6uto0rlbSwj5ZiM85QuaSIk Mocg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740265733; x=1740870533; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/Th+w5zeXQufZOVmAeN72xK4zjFpK20IFV72ncsNhk0=; b=dhePF2w1FfeAPHe6Z5McIAxSfBY7tSVbVqYvhn8Dsfpd7BHWaz+n+5FohyUUQ7gixN 5judX/hUtRUAq8rr3wvvP0qo98mbcNyHvEg5Nmo+/qzyAt+k+aTStiqy51VVXv58CKBK Mgxp1KWMCDvcvJJ7/wP/4FsNEgJoROfJj5YsytGIomjlSWjmwoYxBiXIcbHUMyrd7ev+ /mPXyCKgabks6eg9RNWPkQlygsQNVClwx3ecjKF5grBd5jDPgJGSjGvZ5kMoB+oca0qG obCOkrEZx/dgFB5RrXjdnT9fazUYc/ym3PgKEMDQPSk6jQYvYmrU48HHXlg2xquko4Ts irKg== X-Forwarded-Encrypted: i=1; AJvYcCVFA6OvsB08Pz/AO8dE1+wdwIQNSvaztKOkMcmlAnMXs5VurNKzLSpWhtxVs5idkFRYC5MsY3vbIA==@kvack.org X-Gm-Message-State: AOJu0YwvXjftuR1vLLoFJ1LN7WtTaJiJEuS6u/8zfPAzWYOOlfTgpAEv QNqat1QBZkiAgozoC0iDEAH7tI92RQaaYCsjxx7/9Z/S5npnuGu/TxNyamjaspoB6EqOWnjsGwL GsygmA463lOk5cXph/uR02OAX2Ln9wLnRv5f+ X-Gm-Gg: ASbGncsTps9X3zBrRxtbndkevVs5+DGvnGXP7LfWn0pqtI+YV1J4cg9Ta6c/o3/JRUe 8FmkwJJlzD3woSVIjkyQzu8M0Yu3Wy9AJDCoHd1CLvRPAyuq3VYq1ja0IMAZvqfBJdVxUjO62Wl enJJ7/GKQ= X-Google-Smtp-Source: AGHT+IGNmc3p+y2LRfccUPqS/dEYqzeVeZnJMCY+O0oWmFmdfIWdyts0kUci8NBzSSMS6XMIVzXdNfQnoRCVW8gkc6s= X-Received: by 2002:a05:622a:1306:b0:467:7c30:3446 with SMTP id d75a77b69052e-47234cc8f6cmr2083641cf.25.1740265733382; Sat, 22 Feb 2025 15:08:53 -0800 (PST) MIME-Version: 1.0 References: <20250214-slub-percpu-caches-v2-0-88592ee0966a@suse.cz> <20250214-slub-percpu-caches-v2-2-88592ee0966a@suse.cz> In-Reply-To: <20250214-slub-percpu-caches-v2-2-88592ee0966a@suse.cz> From: Suren Baghdasaryan Date: Sat, 22 Feb 2025 15:08:42 -0800 X-Gm-Features: AWEUYZmJu6rhsB0CYiL4W_PcRtfIH2WDT70rCYAHCu0JXCyZCkXAvuLC2iVFtIE Message-ID: Subject: Re: [PATCH RFC v2 02/10] slab: add sheaf support for batching kfree_rcu() operations To: Vlastimil Babka Cc: "Liam R. Howlett" , Christoph Lameter , David Rientjes , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Stat-Signature: yhfrnonpjt9dqtgug1x58eudsmg5upwp X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8652120002 X-HE-Tag: 1740265734-969435 X-HE-Meta: U2FsdGVkX1+idWqoVLAY9Kv7jwXA37bvthoWGLOTg7OHC6CiqHy02euBc3JvUhyAyZo8BA1b0GLmejz0lbHNqF0bW/fxPvYRzGdIgkMYiF3Ps4mG63Pk6YFpfZb9NxDKtzyU1juK2DnstH8flwovJ1ELQ9W+4U4IKHMXauR5/k/JdBP0xoiu9/elmZS6iIYVUUtQZmEM8Zi2qA/E69P+SBVo9suQ53ehqjxU6AOfqJpOA1afSTz4Y/m/1eTgZneEpJ1dlI30GcGuX8bBmpNcd++6GCmOIGe76I70JBE4AI9onip14S/YnsFGCeUZfgavrdcHnfDuq0CJvg2wnfJSFGBalRcSUEEZHagq2L0zwIZtNc/JNRkQpamNmz9YzrZc8i+dokvYVqnY0ghT385OebPjn2ppphevkbtpe0hnJWo4s94u9IWGn1uUIroUyZk7Vb/RZdxaEmYn20rwbUjSHqCEwuhVYTTD+pxslvuUwaHVQwzPL95lz2ff8ZwHogB7oXPdMgf3yXL2usZ+FXGjMDSMBwWCJFlYDYmqNN14s3N0U1kHCvOnETFcBHT292WqjSQ2KxIUIvR2D5qYb8O4ynekjvGuyYwyxRSEJFGSME4PxPaMe0JQV0jUj718p/PBd4TiV9qU02lNn/6oRUv4Bf1/StDNRbxdSqZPXbxjBD4jhhUT1rK8BcYDv17bKP973870KtAa/PLNxobPJWKz9vUMqf40E73Q9MNcfoHwvZia7eR0iyHbus7UgZRm08sHGCpF8v3rJU3rqxVNczatwkYFQ4+A2xr5mtHGwUpfEi361qyaMpbJ4+vcD3ovNgtj2wFGJIYgvNLi36sfTQMcNFOm11xghSPDaDIFN8jWwKVmaWqvnDEkErZViRmBfqtwgTTa3Pn37VAYt3B3ZEcBwRjBP0kC+rbkyVxgNJldRx4yTzcBVThu3ihpsaGYt2N/qNR6XmeZaVml2tNA6fg /IJxpREz KC4pX5TtYO4elxlOIFwOXwvcLDcNHjawthZZVdj13aApSUiFGkQIUqteNe59T/wXOU0Nq2vIlrK1OGOYNPwOjUmiQATfKg5nlvw6VXCIiaGnDoNY+Udw52yKocXTY10Ch9FgG3xWFeB+miXFxhAZy5BoLJW+IFrE4BnKDB9J7BWjf9hZ5LKdUVLXG2MRqBNOHRvwU895MeDC0Ye+RUPtFh7IQtiUGECQdqWKXm2btKUKxhakgoNlRFFFBVp34RTAjRLaUgWKZXKVzlArMq3jH6+vjqKgsoYuPPRpID9xTl1iHpjaX3AQfa/ZnoUBG5KC/JU3SLdEh+ntpt/t3waKjV2B0lw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Feb 14, 2025 at 8:27=E2=80=AFAM Vlastimil Babka wr= ote: > > Extend the sheaf infrastructure for more efficient kfree_rcu() handling. > For caches with sheaves, on each cpu maintain a rcu_free sheaf in > addition to main and spare sheaves. > > kfree_rcu() operations will try to put objects on this sheaf. Once full, > the sheaf is detached and submitted to call_rcu() with a handler that > will try to put in in the barn, or flush to slab pages using bulk free, s/in in/it in > when the barn is full. Then a new empty sheaf must be obtained to put > more objects there. > > It's possible that no free sheaves are available to use for a new > rcu_free sheaf, and the allocation in kfree_rcu() context can only use > GFP_NOWAIT and thus may fail. In that case, fall back to the existing > kfree_rcu() machinery. > > Expected advantages: > - batching the kfree_rcu() operations, that could eventually replace the > existing batching > - sheaves can be reused for allocations via barn instead of being > flushed to slabs, which is more efficient > - this includes cases where only some cpus are allowed to process rcu > callbacks (Android) > > Possible disadvantage: > - objects might be waiting for more than their grace period (it is > determined by the last object freed into the sheaf), increasing memory > usage - but the existing batching does that too? > > Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny > implementation favors smaller memory footprint over performance. > > Signed-off-by: Vlastimil Babka Reviewed-by: Suren Baghdasaryan > --- > mm/slab.h | 2 + > mm/slab_common.c | 21 ++++++++ > mm/slub.c | 151 +++++++++++++++++++++++++++++++++++++++++++++++++= ++++-- > 3 files changed, 170 insertions(+), 4 deletions(-) > > diff --git a/mm/slab.h b/mm/slab.h > index 8daaec53b6ecfc44171191d421adb12e5cba2c58..94e9959e1aefa350d3d74e3f5= 309fde7a5cf2ec8 100644 > --- a/mm/slab.h > +++ b/mm/slab.h > @@ -459,6 +459,8 @@ static inline bool is_kmalloc_normal(struct kmem_cach= e *s) > return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACC= OUNT)); > } > > +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj); > + > /* Legal flag mask for kmem_cache_create(), for various configurations *= / > #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \ > SLAB_CACHE_DMA32 | SLAB_PANIC | \ > diff --git a/mm/slab_common.c b/mm/slab_common.c > index ceeefb287899a82f30ad79b403556001c1860311..c6853450ed74160cfcb497c09= f92c1f9f7b12629 100644 > --- a/mm/slab_common.c > +++ b/mm/slab_common.c > @@ -1613,6 +1613,24 @@ static void kfree_rcu_work(struct work_struct *wor= k) > kvfree_rcu_list(head); > } > > +static bool kfree_rcu_sheaf(void *obj) > +{ > + struct kmem_cache *s; > + struct folio *folio; > + struct slab *slab; > + > + folio =3D virt_to_folio(obj); > + if (unlikely(!folio_test_slab(folio))) > + return false; > + > + slab =3D folio_slab(folio); > + s =3D slab->slab_cache; > + if (s->cpu_sheaves) > + return __kfree_rcu_sheaf(s, obj); > + > + return false; > +} > + > static bool > need_offload_krc(struct kfree_rcu_cpu *krcp) > { > @@ -1957,6 +1975,9 @@ void kvfree_call_rcu(struct rcu_head *head, void *p= tr) > if (!head) > might_sleep(); > > + if (kfree_rcu_sheaf(ptr)) > + return; > + > // Queue the object but don't yet schedule the batch. > if (debug_rcu_head_queue(ptr)) { > // Probable double kfree_rcu(), just leak. > diff --git a/mm/slub.c b/mm/slub.c > index c06734912972b799f537359f7fe6a750918ffe9e..40175747212fefb27137309b2= 7571abe8d0966e2 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -350,6 +350,8 @@ enum stat_item { > ALLOC_FASTPATH, /* Allocation from cpu slab */ > ALLOC_SLOWPATH, /* Allocation by getting a new cpu slab *= / > FREE_PCS, /* Free to percpu sheaf */ > + FREE_RCU_SHEAF, /* Free to rcu_free sheaf */ > + FREE_RCU_SHEAF_FAIL, /* Failed to free to a rcu_free sheaf */ > FREE_FASTPATH, /* Free to cpu slab */ > FREE_SLOWPATH, /* Freeing not to cpu slab */ > FREE_FROZEN, /* Freeing to frozen slab */ > @@ -2569,6 +2571,24 @@ static void sheaf_flush(struct kmem_cache *s, stru= ct slab_sheaf *sheaf) > sheaf->size =3D 0; > } > > +static void __rcu_free_sheaf_prepare(struct kmem_cache *s, > + struct slab_sheaf *sheaf); > + > +static void rcu_free_sheaf_nobarn(struct rcu_head *head) > +{ > + struct slab_sheaf *sheaf; > + struct kmem_cache *s; > + > + sheaf =3D container_of(head, struct slab_sheaf, rcu_head); > + s =3D sheaf->cache; Ah, that's where you are using sheaf->cache. Maybe you should introduce it in this patch? > + > + __rcu_free_sheaf_prepare(s, sheaf); > + > + sheaf_flush(s, sheaf); > + > + free_empty_sheaf(s, sheaf); > +} > + > /* > * Caller needs to make sure migration is disabled in order to fully flu= sh > * single cpu's sheaves > @@ -2598,8 +2618,8 @@ static void pcs_flush_all(struct kmem_cache *s) > free_empty_sheaf(s, spare); > } > > - // TODO: handle rcu_free > - BUG_ON(rcu_free); > + if (rcu_free) > + call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn); > > sheaf_flush_main(s); > } > @@ -2616,8 +2636,10 @@ static void __pcs_flush_all_cpu(struct kmem_cache = *s, unsigned int cpu) > pcs->spare =3D NULL; > } > > - // TODO: handle rcu_free > - BUG_ON(pcs->rcu_free); > + if (pcs->rcu_free) { > + call_rcu(&pcs->rcu_free->rcu_head, rcu_free_sheaf_nobarn)= ; > + pcs->rcu_free =3D NULL; > + } > > sheaf_flush_main(s); > } > @@ -5192,6 +5214,118 @@ void free_to_pcs(struct kmem_cache *s, void *obje= ct) > stat(s, FREE_PCS); > } > > +static void __rcu_free_sheaf_prepare(struct kmem_cache *s, > + struct slab_sheaf *sheaf) > +{ > + bool init =3D slab_want_init_on_free(s); > + void **p =3D &sheaf->objects[0]; > + unsigned int i =3D 0; > + > + while (i < sheaf->size) { > + struct slab *slab =3D virt_to_slab(p[i]); > + > + memcg_slab_free_hook(s, slab, p + i, 1); > + alloc_tagging_slab_free_hook(s, slab, p + i, 1); > + > + if (unlikely(!slab_free_hook(s, p[i], init, false))) { > + p[i] =3D p[--sheaf->size]; > + continue; > + } > + > + i++; > + } > +} > + > +static void rcu_free_sheaf(struct rcu_head *head) > +{ > + struct slab_sheaf *sheaf; > + struct node_barn *barn; > + struct kmem_cache *s; > + > + sheaf =3D container_of(head, struct slab_sheaf, rcu_head); > + > + s =3D sheaf->cache; > + > + __rcu_free_sheaf_prepare(s, sheaf); > + > + barn =3D get_node(s, numa_mem_id())->barn; > + > + /* due to slab_free_hook() */ > + if (unlikely(sheaf->size =3D=3D 0)) > + goto empty; > + > + if (!barn_put_full_sheaf(barn, sheaf, false)) > + return; > + > + sheaf_flush(s, sheaf); > + > +empty: > + if (!barn_put_empty_sheaf(barn, sheaf, false)) > + return; > + > + free_empty_sheaf(s, sheaf); > +} > + > +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) > +{ > + struct slub_percpu_sheaves *pcs; > + struct slab_sheaf *rcu_sheaf; > + unsigned long flags; > + > + local_lock_irqsave(&s->cpu_sheaves->lock, flags); > + pcs =3D this_cpu_ptr(s->cpu_sheaves); > + > + if (unlikely(!pcs->rcu_free)) { > + > + struct slab_sheaf *empty; > + > + empty =3D barn_get_empty_sheaf(pcs->barn); > + > + if (empty) { > + pcs->rcu_free =3D empty; > + goto do_free; > + } > + > + local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); > + > + empty =3D alloc_empty_sheaf(s, GFP_NOWAIT); > + > + if (!empty) { > + stat(s, FREE_RCU_SHEAF_FAIL); > + return false; > + } > + > + local_lock_irqsave(&s->cpu_sheaves->lock, flags); > + pcs =3D this_cpu_ptr(s->cpu_sheaves); > + > + if (unlikely(pcs->rcu_free)) > + barn_put_empty_sheaf(pcs->barn, empty, true); > + else > + pcs->rcu_free =3D empty; > + } > + > +do_free: > + > + rcu_sheaf =3D pcs->rcu_free; > + > + rcu_sheaf->objects[rcu_sheaf->size++] =3D obj; > + > + if (likely(rcu_sheaf->size < s->sheaf_capacity)) { > + local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); > + stat(s, FREE_RCU_SHEAF); > + return true; > + } > + > + pcs->rcu_free =3D NULL; > + local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); > + > + call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf); > + > + stat(s, FREE_RCU_SHEAF); > + > + return true; > +} > + > /* > * Bulk free objects to the percpu sheaves. > * Unlike free_to_pcs() this includes the calls to all necessary hooks > @@ -6522,6 +6656,11 @@ int __kmem_cache_shutdown(struct kmem_cache *s) > struct kmem_cache_node *n; > > flush_all_cpus_locked(s); > + > + /* we might have rcu sheaves in flight */ > + if (s->cpu_sheaves) > + rcu_barrier(); > + > /* Attempt to free all objects */ > for_each_kmem_cache_node(s, node, n) { > if (n->barn) > @@ -7927,6 +8066,8 @@ STAT_ATTR(ALLOC_PCS, alloc_cpu_sheaf); > STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath); > STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath); > STAT_ATTR(FREE_PCS, free_cpu_sheaf); > +STAT_ATTR(FREE_RCU_SHEAF, free_rcu_sheaf); > +STAT_ATTR(FREE_RCU_SHEAF_FAIL, free_rcu_sheaf_fail); > STAT_ATTR(FREE_FASTPATH, free_fastpath); > STAT_ATTR(FREE_SLOWPATH, free_slowpath); > STAT_ATTR(FREE_FROZEN, free_frozen); > @@ -8022,6 +8163,8 @@ static struct attribute *slab_attrs[] =3D { > &alloc_fastpath_attr.attr, > &alloc_slowpath_attr.attr, > &free_cpu_sheaf_attr.attr, > + &free_rcu_sheaf_attr.attr, > + &free_rcu_sheaf_fail_attr.attr, > &free_fastpath_attr.attr, > &free_slowpath_attr.attr, > &free_frozen_attr.attr, > > -- > 2.48.1 >