From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D85BC369D1 for ; Fri, 25 Apr 2025 08:28:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFC786B002F; Fri, 25 Apr 2025 04:28:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B5FAB6B0030; Fri, 25 Apr 2025 04:28:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B3DA6B0031; Fri, 25 Apr 2025 04:28:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 72AE36B002F for ; Fri, 25 Apr 2025 04:28:00 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 69989BA106 for ; Fri, 25 Apr 2025 08:28:02 +0000 (UTC) X-FDA: 83371888404.02.7AA8529 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf12.hostedemail.com (Postfix) with ESMTP id 486BC40003 for ; Fri, 25 Apr 2025 08:28:00 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf12.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745569680; a=rsa-sha256; cv=none; b=tcxbGYno8xxp9ICvYgGHNzb5/f2OeBdweI20H/az4b9KQjiV6wMZtsqsQXKAAegFgUeJUk lxLs6hnD8LDqXYmhkph0Py9HVwr9DGlKfUsorTZ+ziFsVWRiiT4JDD67FKaGxAYyLrVDC8 ZxscnaFVz2X/Cn0FhZNQ14TwkagKyV4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf12.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745569680; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lE5cW5ooznFo6+6TvsGlsSjYfh9HheQR0UP2bw/0+GM=; b=evCsfmJXaVOKiuZf1IZzNJ80GqiYTZSaHTAQ9Nq4c+kQMW0KyIkVvQ7TLkvk+oflvmBIuO q7agPzTmOvt2zTefeTVbQswvCWT+/Mb2Z1KeByKibZFFJDh2eYQGwjXsxiwLIS+qavY04Y WeLkP6APZAirZJ22sU3Kg/xUFxdy9bI= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 04ED91F38E; Fri, 25 Apr 2025 08:27:53 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id E27E813A80; Fri, 25 Apr 2025 08:27:52 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id aEQDN4hHC2htfwAAD6G6ig (envelope-from ); Fri, 25 Apr 2025 08:27:52 +0000 From: Vlastimil Babka Date: Fri, 25 Apr 2025 10:27:24 +0200 Subject: [PATCH v4 4/9] slab: determine barn status racily outside of lock MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20250425-slub-percpu-caches-v4-4-8a636982b4a4@suse.cz> References: <20250425-slub-percpu-caches-v4-0-8a636982b4a4@suse.cz> In-Reply-To: <20250425-slub-percpu-caches-v4-0-8a636982b4a4@suse.cz> To: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes Cc: Roman Gushchin , Harry Yoo , Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, vbabka@suse.cz X-Mailer: b4 0.14.2 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 486BC40003 X-Stat-Signature: 49371neguj9h4zqdcopf7mf8n9zkwrgj X-Rspam-User: X-HE-Tag: 1745569680-977870 X-HE-Meta: U2FsdGVkX18hD4fQS9URWsXth1OBXWqskHVr4SRKH5dJsPhQqsjyOq3wCwmMIeRWn73CWd6MgO88dGkv5X0fVJfiopcSzhpnQgRODt5LGEJ66Lcy+sT/jiyt6HzNzqgTkq/bZ700pbNcMVe4FnK5thUQYPqbTiT6cRibteaYEarU9kvv3ZHScZy2ua2BNhkMz3lqfGm++qMec+xdQbYFeoc3tUb9KkxuVJfO1WM/jnETSCuOZ0F601AzX0WCx9tpPmC0LxA6LMmqKDibaVw52vQtx8ZPIYv5GDz3GicOsceDPMcIhCZV6ILKFc3O7a3wyM2Urnss+jlF29ZMvyI0siv9sgAZVGB68G6eJtPDYaf8k1p2LyTE8dP+luSCQpp6iFRKHE4d8x6oGNugnPR3ofv6I6Ay9f+pJP16hk59VhR3Q+6l/CzBGf3lhtj1G18DWPsZcaLsB6mWlmfcLO3a5qqn95ePu92pvRKZF4sygV/uZWyDXxUotApk23fK7w4sjx5ZGyPwJ1/Ma1hNLnh6Ah7Fzwr647Pl+PCihelYAmnzvT/qG6F4tbDFCYzQ3ip+4lz/e/fc1KghlfiveqyK0s3M8qIwHpThNDN99IUsvpcGjOMbrJDcgaJhGiS9utb7fPRbH/WjlVvelNsUzHOYSIDM6fNugThMJVcGKMDvu+OX53jVzmP7pi3V9siHvp2h9FvCruEAX3UBHd/Q68jX0Pl1J6LT6Iu3u+rYBKyQqc/y0GQx/TK3e5EAM33uaZvRr3KnyLEev0CYX6YwMX7SfaqGjM+R//8H7JUAfZP3rR6rd5Vd5ZbQnbBFINdu/eCLixXe+ANu77zOuyGqiQqHmEhBmkJCzIVLQyuSSqW49vOQvSKi+TFdQ8oM7C9mfSBGi5QjqLn+ZSVNgZbnWTz3MbUh41WbRi/e7iU3l5oc83oodZr5qEEYVNKygB69FeCM4YSwke6O+nhPA089GSR 49HcRUlk CVIKJ/u8R6YhcXvfw5X3stsiLYkDn7YJlwO4cTt1t0KXwNPUGbP7ZHj4SnQ0eSSRQDOY5TrvuuhOXNX5rkmJo/qEfP4tsUW419vRTOUlbwvXBuO9a3HzgZ0K0avqIldJz6NvI3jGGLLvWs9kUGeNfbYF048yzZEJdIewu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The possibility of many barn operations is determined by the current number of full or empty sheaves. Taking the barn->lock just to find out that e.g. there are no empty sheaves results in unnecessary overhead and lock contention. Thus perform these checks outside of the lock with a data_race() annotated variable read and fail quickly without taking the lock. Checks for sheaf availability that racily succeed have to be obviously repeated under the lock for correctness, but we can skip repeating checks if there are too many sheaves on the given list as the limits don't need to be strict. Signed-off-by: Vlastimil Babka Reviewed-by: Suren Baghdasaryan Reviewed-by: Harry Yoo --- mm/slub.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 724266fdd996c091f1f0b34012c5179f17dfa422..cc273cc45f632e16644355831132cdc391219cec 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2758,9 +2758,12 @@ static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn) struct slab_sheaf *empty = NULL; unsigned long flags; + if (!data_race(barn->nr_empty)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); - if (barn->nr_empty) { + if (likely(barn->nr_empty)) { empty = list_first_entry(&barn->sheaves_empty, struct slab_sheaf, barn_list); list_del(&empty->barn_list); @@ -2807,6 +2810,9 @@ static struct slab_sheaf *barn_get_full_or_empty_sheaf(struct node_barn *barn) struct slab_sheaf *sheaf = NULL; unsigned long flags; + if (!data_race(barn->nr_full) && !data_race(barn->nr_empty)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); if (barn->nr_full) { @@ -2837,9 +2843,12 @@ barn_replace_empty_sheaf(struct node_barn *barn, struct slab_sheaf *empty) struct slab_sheaf *full = NULL; unsigned long flags; + if (!data_race(barn->nr_full)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); - if (barn->nr_full) { + if (likely(barn->nr_full)) { full = list_first_entry(&barn->sheaves_full, struct slab_sheaf, barn_list); list_del(&full->barn_list); @@ -2862,19 +2871,23 @@ barn_replace_full_sheaf(struct node_barn *barn, struct slab_sheaf *full) struct slab_sheaf *empty; unsigned long flags; + /* we don't repeat this check under barn->lock as it's not critical */ + if (data_race(barn->nr_full) >= MAX_FULL_SHEAVES) + return ERR_PTR(-E2BIG); + if (!data_race(barn->nr_empty)) + return ERR_PTR(-ENOMEM); + spin_lock_irqsave(&barn->lock, flags); - if (barn->nr_full >= MAX_FULL_SHEAVES) { - empty = ERR_PTR(-E2BIG); - } else if (!barn->nr_empty) { - empty = ERR_PTR(-ENOMEM); - } else { + if (likely(barn->nr_empty)) { empty = list_first_entry(&barn->sheaves_empty, struct slab_sheaf, barn_list); list_del(&empty->barn_list); list_add(&full->barn_list, &barn->sheaves_full); barn->nr_empty--; barn->nr_full++; + } else { + empty = ERR_PTR(-ENOMEM); } spin_unlock_irqrestore(&barn->lock, flags); -- 2.49.0