From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BB7AC3ABCC for ; Wed, 14 May 2025 14:01:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 963736B015B; Wed, 14 May 2025 10:01:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 914086B015D; Wed, 14 May 2025 10:01:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B34F6B015E; Wed, 14 May 2025 10:01:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5CE336B015B for ; Wed, 14 May 2025 10:01:22 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 158001CD078 for ; Wed, 14 May 2025 14:01:23 +0000 (UTC) X-FDA: 83441675646.23.48A5446 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf01.hostedemail.com (Postfix) with ESMTP id 2B0864000B for ; Wed, 14 May 2025 14:01:19 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=UjBuT4qm; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=HzAefIuD; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=UjBuT4qm; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=HzAefIuD; spf=pass (imf01.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747231280; a=rsa-sha256; cv=none; b=Dq/lkTmtM1jIwWXYSCk95p1vTYgyS3WBKcbBQDFAWGCrxB5MO3FVK+a/vn7IxxjweLxvip uO2weJJqvsZ47yMZh2QT83gB4/2DjZxXPBHp3JzuBt5JdFh19AWrqxwBjEJTJ0UqL3IALX YvySUx5a/1pD4JhTGNdQOcszNKkJdnw= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=UjBuT4qm; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=HzAefIuD; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=UjBuT4qm; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=HzAefIuD; spf=pass (imf01.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747231280; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7pjNJhG3L0zAzXf2BefC48+ZaUDK3Dac9PaRcrtBMX4=; b=1liwVxf3riauyrlN+xIpY4X054BoL2cGzpX7KTDkPibmOzcfyLByvJgTloziJRarm7Axjp 4tBk3FTnSRwpS4q4t4W1+bdY7NAb46UNfmqK/2XEO4WbgznqGTmhV8GLRQ/9SzBKzDyIWJ ip0G18qj508DxAsv9Avx5GQAQXTYhTo= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 569271F7C4; Wed, 14 May 2025 14:01:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1747231278; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7pjNJhG3L0zAzXf2BefC48+ZaUDK3Dac9PaRcrtBMX4=; b=UjBuT4qmfCVgyPdGfEiUzIcpAH9aqdhkB2npjDFPiLuooOks6LoICam3u8FNuZYglWoJpz Rtfd9zK/unJ0ualUFt3nE30Kn0fwb9X51nqL9x1UVC5kGBmpXi81uMwGllyMCbdkNmZm1P tgIh9LR4dOSwUdlF9AupDdv74PfeYFM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1747231278; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7pjNJhG3L0zAzXf2BefC48+ZaUDK3Dac9PaRcrtBMX4=; b=HzAefIuD6NVp9PhTTI8Nnk7j+1w3CbHemXo7wpeK1Zfh/3BsFJZdtXwAkAZ4wjhBmu16wu NrLr009Y1TBYwOCg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1747231278; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7pjNJhG3L0zAzXf2BefC48+ZaUDK3Dac9PaRcrtBMX4=; b=UjBuT4qmfCVgyPdGfEiUzIcpAH9aqdhkB2npjDFPiLuooOks6LoICam3u8FNuZYglWoJpz Rtfd9zK/unJ0ualUFt3nE30Kn0fwb9X51nqL9x1UVC5kGBmpXi81uMwGllyMCbdkNmZm1P tgIh9LR4dOSwUdlF9AupDdv74PfeYFM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1747231278; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7pjNJhG3L0zAzXf2BefC48+ZaUDK3Dac9PaRcrtBMX4=; b=HzAefIuD6NVp9PhTTI8Nnk7j+1w3CbHemXo7wpeK1Zfh/3BsFJZdtXwAkAZ4wjhBmu16wu NrLr009Y1TBYwOCg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 3BD5A139A2; Wed, 14 May 2025 14:01:18 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id PLp3DS6iJGjbCgAAD6G6ig (envelope-from ); Wed, 14 May 2025 14:01:18 +0000 Message-ID: Date: Wed, 14 May 2025 16:01:17 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 2/9] slab: add sheaf support for batching kfree_rcu() operations Content-Language: en-US To: Suren Baghdasaryan Cc: "Liam R. Howlett" , Christoph Lameter , David Rientjes , Roman Gushchin , Harry Yoo , Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org References: <20250425-slub-percpu-caches-v4-0-8a636982b4a4@suse.cz> <20250425-slub-percpu-caches-v4-2-8a636982b4a4@suse.cz> From: Vlastimil Babka In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Action: no action X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 2B0864000B X-Stat-Signature: dmfkxmgdy5rgiyd3o1c4463xnxgb6hjg X-HE-Tag: 1747231279-529980 X-HE-Meta: U2FsdGVkX19pTbbe7PK3iMcis2nkxe6nWvT86kFFnMOphvGvRp5dCIp1IGvlvNUQjsNXINHrH4/tVZI6k2e4HFSTL/In5K5PCaOaTX01J4DoYx/uMOVfJBPBmw7OmOPmV/ZYiIS5IPVqwPirRWgHFepzLZ0AvLjeCuue6k9LyXbWH5NkLfF7XuL3Y1AWSjONxuk8qC1u3R/0kfWP38JVtpn6d4ist0NfxI/ROPfXMrD49GSNqnoPZT1Zlu2HfXES4YLNw2ILWmo2rQS1JiJIdBN6QV8SsklAhJ1BtniId0j/UJWzdyFHl9j4r9lBrHcfIPpb46yukVU3PZRKwVCNOfA17vxuzbbVQizDjbThhrwox14FkxqC/zYNb6yidRRJlBrux0Kk36NjAp7n7IiGfysFwL4l4sM6zb64e7jqnfYZAUUsU1dveJP57+D8BZIsonCaSZjew/GWF2MTgNymurvHRQom5UtNTV4LqFfQioHjAf8Gw2gc4VhCpNa6c8A46ibUQLlP+hJERbDWQqrPpAgIvdyKz+lxWhSvVGbfL/FqL7MYdclPK2ApWrUVQSpS05x/LBzRJVwSvJDisqJachP3i5ABGwCg3n1pSRKtFrtHsFxXTqHWQcIEmxPrjZ0oxXJk5bfKky173MlX/zHmTs7qYrZ5oN2UGgUqRMSJvbZfWRfOfRjukffJmMCrrCqAnY8j+chkj/2bDPu03+O+b+wSIIUqSxFtjhWfpDVit8rTR3MPjEV7n72kEfzK+g6+D6Kfmt6rQfQPdMR1aiAKM8v7hnQlKOv2pvSjBLlBBM5sR+IhtrEeKWz/Fdu0t0smJXtBtF5xErVvWwDD5dEC4N5z9wrFb7KvhtvwJtj3zrHF1XviuhrG4QL/8XheRNNNkga95oqC7cxpHiEezof0mUj+5kuIgVkxlIRNfNrwhzYGbznWc8h3syzJPyRzRbMh6FrsNbUjfqcA+vALHfq A2tZ9lsf vOEnlGvuq/7Smf88fTVwGSTN2Cym+3wR7npZC1kKNxXeren07P/AhrPyTjsks8YPGoySLYGLIO4fKg8fmE/dgRuiST07teiMu8Hz/IOiFUuK3bEnzxggyC7Y2DKqXd93UXy4HlfhrBCCosiBI32t8FJyhm8C8cEqiCvn9XEM+/AP2hM7bl1owHmTmiOZiCl5oWREbEJu6ef+TfxpRbLzhrCL9gtkkbHmAkzMLBygCPS5XZhYKtiIQw6YLIcyHzdGo7bJfiyv9yJLzND2loIGo0zHTmIGubkXLp2fM/EFiE572KYhzPXxmkpUFaPRpJ6fvmbRN1S8AlysGFCaaJ/SJV8NwPHtJVORnPKqLAb9Tgy7W1uU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 5/6/25 23:34, Suren Baghdasaryan wrote: > On Fri, Apr 25, 2025 at 1:27 AM Vlastimil Babka wrote: >> @@ -2631,6 +2637,24 @@ static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf) >> sheaf->size = 0; >> } >> >> +static void __rcu_free_sheaf_prepare(struct kmem_cache *s, >> + struct slab_sheaf *sheaf); > > I think you could safely move __rcu_free_sheaf_prepare() here and > avoid the above forward declaration. Right, done. >> @@ -5304,6 +5340,140 @@ bool free_to_pcs(struct kmem_cache *s, void *object) >> return true; >> } >> >> +static void __rcu_free_sheaf_prepare(struct kmem_cache *s, >> + struct slab_sheaf *sheaf) > > This function seems to be an almost exact copy of free_to_pcs_bulk() > from your previous patch. Maybe they can be consolidated? True, I've extracted it to __kmem_cache_free_bulk_prepare(). >> +{ >> + bool init = slab_want_init_on_free(s); >> + void **p = &sheaf->objects[0]; >> + unsigned int i = 0; >> + >> + while (i < sheaf->size) { >> + struct slab *slab = virt_to_slab(p[i]); >> + >> + memcg_slab_free_hook(s, slab, p + i, 1); >> + alloc_tagging_slab_free_hook(s, slab, p + i, 1); >> + >> + if (unlikely(!slab_free_hook(s, p[i], init, true))) { >> + p[i] = p[--sheaf->size]; >> + continue; >> + } >> + >> + i++; >> + } >> +} >> + >> +static void rcu_free_sheaf(struct rcu_head *head) >> +{ >> + struct slab_sheaf *sheaf; >> + struct node_barn *barn; >> + struct kmem_cache *s; >> + >> + sheaf = container_of(head, struct slab_sheaf, rcu_head); >> + >> + s = sheaf->cache; >> + >> + /* >> + * This may reduce the number of objects that the sheaf is no longer >> + * technically full, but it's easier to treat it that way (unless it's > > I don't understand the sentence above. Could you please clarify and > maybe reword it? Is this more clear? /* * This may remove some objects due to slab_free_hook() returning false, * so that the sheaf might no longer be completely full. But it's easier * to handle it as full (unless it became completely empty), as the code * handles it fine. The only downside is that sheaf will serve fewer * allocations when reused. It only happens due to debugging, which is a * performance hit anyway. */ >> + >> + if (!local_trylock(&s->cpu_sheaves->lock)) > > Aren't you leaking `empty` sheaf on this failure? Right! Fixed, thanks. >> + goto fail; >> + >> + pcs = this_cpu_ptr(s->cpu_sheaves); >> + >> + if (unlikely(pcs->rcu_free)) >> + barn_put_empty_sheaf(pcs->barn, empty); >> + else >> + pcs->rcu_free = empty; >> + } >> + >> +do_free: >> + >> + rcu_sheaf = pcs->rcu_free; >> + >> + rcu_sheaf->objects[rcu_sheaf->size++] = obj; >> + >> + if (likely(rcu_sheaf->size < s->sheaf_capacity)) >> + rcu_sheaf = NULL; >> + else >> + pcs->rcu_free = NULL; >> + >> + local_unlock(&s->cpu_sheaves->lock); >> + >> + if (rcu_sheaf) >> + call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf); >> + >> + stat(s, FREE_RCU_SHEAF); >> + return true; >> + >> +fail: >> + stat(s, FREE_RCU_SHEAF_FAIL); >> + return false; >> +} >> + >> /* >> * Bulk free objects to the percpu sheaves. >> * Unlike free_to_pcs() this includes the calls to all necessary hooks >> @@ -6802,6 +6972,11 @@ int __kmem_cache_shutdown(struct kmem_cache *s) >> struct kmem_cache_node *n; >> >> flush_all_cpus_locked(s); >> + >> + /* we might have rcu sheaves in flight */ >> + if (s->cpu_sheaves) >> + rcu_barrier(); >> + >> /* Attempt to free all objects */ >> for_each_kmem_cache_node(s, node, n) { >> if (n->barn) >> @@ -8214,6 +8389,8 @@ STAT_ATTR(ALLOC_PCS, alloc_cpu_sheaf); >> STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath); >> STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath); >> STAT_ATTR(FREE_PCS, free_cpu_sheaf); >> +STAT_ATTR(FREE_RCU_SHEAF, free_rcu_sheaf); >> +STAT_ATTR(FREE_RCU_SHEAF_FAIL, free_rcu_sheaf_fail); >> STAT_ATTR(FREE_FASTPATH, free_fastpath); >> STAT_ATTR(FREE_SLOWPATH, free_slowpath); >> STAT_ATTR(FREE_FROZEN, free_frozen); >> @@ -8312,6 +8489,8 @@ static struct attribute *slab_attrs[] = { >> &alloc_fastpath_attr.attr, >> &alloc_slowpath_attr.attr, >> &free_cpu_sheaf_attr.attr, >> + &free_rcu_sheaf_attr.attr, >> + &free_rcu_sheaf_fail_attr.attr, >> &free_fastpath_attr.attr, >> &free_slowpath_attr.attr, >> &free_frozen_attr.attr, >> >> -- >> 2.49.0 >>