From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22138C021B2 for ; Sun, 23 Feb 2025 04:01:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74E606B007B; Sat, 22 Feb 2025 23:01:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FDE26B0082; Sat, 22 Feb 2025 23:01:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C55D6B0083; Sat, 22 Feb 2025 23:01:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 400EA6B007B for ; Sat, 22 Feb 2025 23:01:02 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D79F95060E for ; Sun, 23 Feb 2025 04:01:01 +0000 (UTC) X-FDA: 83149858722.03.960B01C Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf25.hostedemail.com (Postfix) with ESMTP id EB7A1A0010 for ; Sun, 23 Feb 2025 04:00:59 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=NYgAv5tO; spf=pass (imf25.hostedemail.com: domain of surenb@google.com designates 209.85.160.175 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740283260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aY9CdaTP4RbjlHaRC0ponJc9BbyfmorYxqjI77oHIeQ=; b=s2cbluC7vwSAnEmbgLtpOE8SNRyHK93e9RRmSfa8K3mbJIedFnivC/5zxSq4GiOLBfYxLn g6/oLJpONjzkHILfl/ri2d2a/Zc/NoHWRPVJJh0/JY3Q3GK2uM2zuavjFe/b7hYOCAlpDt gcwHSBFnsDfJHGxLhqGjulTw7pV7lX8= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=NYgAv5tO; spf=pass (imf25.hostedemail.com: domain of surenb@google.com designates 209.85.160.175 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740283260; a=rsa-sha256; cv=none; b=0LJjQd4C8L/GyrzdIjY9b0kDgzD1fGV/H83NF5/lvhaNBlMkYCJtocYWiCPewhad8rG3Ew 4l4a3MKRqCItJLansfNOF7FKXbJLJoxG4yRGUxLVvTSw3bVy0hzZX08gdWyeBnToH7HbNP Z8gaWCPt494g2LZwM2Xnm94QR9HFaiM= Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-471fbfe8b89so304821cf.0 for ; Sat, 22 Feb 2025 20:00:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740283259; x=1740888059; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=aY9CdaTP4RbjlHaRC0ponJc9BbyfmorYxqjI77oHIeQ=; b=NYgAv5tOuhLpg7BrrpWJqA4ykmjSN/Sm7XlOUm/xO0/hbRExo2TcyrJmBUMo7tWsTp tlQOph6UzMq6+BDOTfUNjTE3pkrGwQsQ6nWpAZghEzSXNxg6RBrH+e+4gqC6VbBLBoIi GvArkbzrHy9/n3/AVAP9DC1QIJS0AVP6P57+r+NNcm+TZlox2LtBdWK7WjhJVilo2Xo0 eXmdOU6J45iRwgEhNsPPdbGWuBxeElL6ReSi4FP/VtGTkPMr8XbzNce5hV4xQ8MPy3sG /zS36BwiH/Fc+95y2QKuge5xukEaaXy4FkpoW1TIuSHUA8yVCrCTKd65jhuzd0VlWNk6 fQSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740283259; x=1740888059; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aY9CdaTP4RbjlHaRC0ponJc9BbyfmorYxqjI77oHIeQ=; b=Y1F8G05jOntr7u5abOrI5Lq7yCBLpY5useR5qVnrVaRcbBinrUz6cZ/RqfJeeER9yG XyfnfyMvmBzVJZGCPCPwY8WTgBxbmddTOp0LWnU6Yz/VxMzCsXDk9LH4xgtEMlD2A9nQ Ct/vD0Urlr2G9xhoUFHxjR1f2wgTxdusIbwK2fq8g8tmovFJbIQ6FYFOUFLBvtGhAqHw OLqr1aZx9ogCbMAO+fqJSYqkFHP2JL3imMFRk0ETTmIBZktRhNCAC7FCPryh743ixgLA CFhzGEA2/Ls/RhhjxqrlpzCN2R3Pi3yH9CCErgzQXJmBoiPoZ3IEitZyxqNhjwvzSz1/ 02SQ== X-Forwarded-Encrypted: i=1; AJvYcCWGN/VgntG3zaqYgbgtGtukn+IY/tgD1pSR5LEvdgT7IuBpmEmKdRw4Xzhik7SiczMy6IWnd/AMgg==@kvack.org X-Gm-Message-State: AOJu0YyECbxGIGyA3iJ2G9qrhdEjnwDZz5QFbs3NDNHecOOPiVa+edqF h47zILStcJ24lc48pYDicOjqau+yi20e+n6zRCuaDfW6/gGzU8miVVniB0SQQKrl5FnHCJtzBuS oNbMVB4HYTwsYJDtrNUzgvaDPq01Wu8IiBbkRcWwuI1/GKU4XGrIn X-Gm-Gg: ASbGncunwhP+2x1jsfL2AiUnsupvHedkhyVvOATEso05hgVhvuRXk3wkL0NzV0sSqKL LavL1ZHJS9kmw94iAF8rVJZyzVAKQow02rt0YffWMdcbgCSLrAB3oEzAsuEZDgja+pZUsl9e5qV YvdRLPRQk= X-Google-Smtp-Source: AGHT+IHetWQEdxRiILOaUBpw/NeyYJ/gqHkNcZyWUSeSwMhL2AOEG5FmX2oXvR+sBPsWcfju++oIs/B2IWKj7AV9jXc= X-Received: by 2002:ac8:5dce:0:b0:447:e59b:54eb with SMTP id d75a77b69052e-4723358478emr3373221cf.26.1740283258617; Sat, 22 Feb 2025 20:00:58 -0800 (PST) MIME-Version: 1.0 References: <20250214-slub-percpu-caches-v2-0-88592ee0966a@suse.cz> <20250214-slub-percpu-caches-v2-7-88592ee0966a@suse.cz> In-Reply-To: <20250214-slub-percpu-caches-v2-7-88592ee0966a@suse.cz> From: Suren Baghdasaryan Date: Sat, 22 Feb 2025 20:00:47 -0800 X-Gm-Features: AWEUYZk951RWW7__fhix8bBdYAk5asCgJ9Ex815rLz1fssOu7vW6UIhbzIZEhBs Message-ID: Subject: Re: [PATCH RFC v2 07/10] slab: determine barn status racily outside of lock To: Vlastimil Babka Cc: "Liam R. Howlett" , Christoph Lameter , David Rientjes , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: EB7A1A0010 X-Stat-Signature: 1kdi6t8zk1hfnzrgq6httob8ae8njeq9 X-HE-Tag: 1740283259-667708 X-HE-Meta: U2FsdGVkX1+knlP/TRKGeGYZ2r6FkWaStnmh7abo5C18dNMUzIt6OGK9Pa1YBov7A/yqTFx6ohF8UXTqbB7n7cB3BJQxt7NnRLRu+hY+43QQX3qeIkbSVbgZPMHqEs4CREYs9g5uGrQGtxla5eSSQxFEyPUj7QN0yS6/++JwuHc1ONroTiDRMaHXjDLDAWir6t8vi0jr2X13rAyIV3AeLBhE+cJnJ2DBbxVYoMztQERsynVN5XEYbMEA/+OUiSKijg9H8f4LEBA0P+dQWCrpcKux6nDha0DUEGJk5o4+KmAyKQylgvuJYYxdaxGepvepTizANkUd5hdVTVqc8j0mfhPUSgBkl7cXEQZ90iqiLty4608p4lSu6/13lEaHwVcOW8ijxwDmwhBVKmEITsuBFHYTSUTWo4sBA9GzpbxaXIYUbRge3SjUFWN+VFzxX9nzp7whKns0Pl5pSk39i01p5VCBTNnLX+mBW2axY55Fp1Kk0HBM1m3tF5OVmOSagX25n21ymuN0Fd+3r1276DS0c3A1Mmk/TzZz9BsFB3nM7yZgUM0w8rcUG1nfdkovtDfDAPYcedJYeU4S6gRAi+vimGo/H1rfpdBAFcCTDMaxUv2exPvHozMpwWRDzJw1sTH2TVzmXPkJoD4+VSPATP+VbJHUPFHH/xlNEjTS15/0E6UR4S86PcvaHwZWjEwjmkH6rnlWN7MqxJEVKbJRmmEcqNzKluyt8F5q3w1xKrBQiJ6oIKOGOjFS0YWJYpRgNg28njFImk0q0L7uUihwnmfgkcRBTBi7gzM2B/FvGtOlN/BL4u2NmjWZeasFBrxMGiB8Qe2O+uIjb2L88uvubAr0kwrx9athrrQytwsfvzJtsz8GURAu7cY/CdaOQL/OhUjPNipKTIfQMnkSpzD7iJalHgGW6rLdta7UMLoiK51qyQj1rhPhQvWwwevBBxx8URGLh0RIV0r4+SXF2qjTfTa 4WGn+bQg 0h2AkSizJZ3Izww5g8809hqK1nAutAPCddkC+IlqkXlemvxvdVVsbFJYz/HQbA1uyVsMrX3eys5zg4SJx/XlmeH2vZi0JSVA0yX3XDmmjyRqJDCLLajrEOay+8Mqbefguxkx3CWX0h9Jrb32/FOwt94Tq01zPfycDUibNtW3raj8rGlCkg77tHGzwgeHdDexb/zj9UeZuzB5MPq68wZ1PUbLI1+jqoVVbfwIGWUoRo8kEjSviu5t65D0V8oI+aJANvFLMHRhv5EYAeQ01MUTw1Zq0UiWXjT87da4J X-Bogosity: Ham, tests=bogofilter, spamicity=0.001151, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Feb 14, 2025 at 8:27=E2=80=AFAM Vlastimil Babka wr= ote: > > The possibility of many barn operations is determined by the current > number of full or empty sheaves. Taking the barn->lock just to find out > that e.g. there are no empty sheaves results in unnecessary overhead and > lock contention. Thus perform these checks outside of the lock with a > data_race() annotated variable read and fail quickly without taking the > lock. > > Checks for sheaf availability that racily succeed have to be obviously > repeated under the lock for correctness, but we can skip repeating > checks if there are too many sheaves on the given list as the limits > don't need to be strict. > > Signed-off-by: Vlastimil Babka Reviewed-by: Suren Baghdasaryan > --- > mm/slub.c | 57 ++++++++++++++++++++++++++++++++++----------------------- > 1 file changed, 34 insertions(+), 23 deletions(-) > > diff --git a/mm/slub.c b/mm/slub.c > index c1df7cf22267f28f743404531bef921e25fac086..72e6437f1d74bfacbb1cd7642= af42929c48cc66a 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -2685,9 +2685,12 @@ static struct slab_sheaf *barn_get_empty_sheaf(str= uct node_barn *barn) > struct slab_sheaf *empty =3D NULL; > unsigned long flags; > > + if (!data_race(barn->nr_empty)) > + return NULL; > + > spin_lock_irqsave(&barn->lock, flags); > > - if (barn->nr_empty) { > + if (likely(barn->nr_empty)) { > empty =3D list_first_entry(&barn->sheaves_empty, > struct slab_sheaf, barn_list); > list_del(&empty->barn_list); > @@ -2703,38 +2706,36 @@ static int barn_put_empty_sheaf(struct node_barn = *barn, > struct slab_sheaf *sheaf, bool ignore_lim= it) > { > unsigned long flags; > - int ret =3D 0; > + > + /* we don't repeat the check under barn->lock as it's not critica= l */ > + if (!ignore_limit && data_race(barn->nr_empty) >=3D MAX_EMPTY_SHE= AVES) > + return -E2BIG; > > spin_lock_irqsave(&barn->lock, flags); > > - if (!ignore_limit && barn->nr_empty >=3D MAX_EMPTY_SHEAVES) { > - ret =3D -E2BIG; > - } else { > - list_add(&sheaf->barn_list, &barn->sheaves_empty); > - barn->nr_empty++; > - } > + list_add(&sheaf->barn_list, &barn->sheaves_empty); > + barn->nr_empty++; > > spin_unlock_irqrestore(&barn->lock, flags); > - return ret; > + return 0; > } > > static int barn_put_full_sheaf(struct node_barn *barn, struct slab_sheaf= *sheaf, > bool ignore_limit) > { > unsigned long flags; > - int ret =3D 0; > + > + /* we don't repeat the check under barn->lock as it's not critica= l */ > + if (!ignore_limit && data_race(barn->nr_full) >=3D MAX_FULL_SHEAV= ES) > + return -E2BIG; > > spin_lock_irqsave(&barn->lock, flags); > > - if (!ignore_limit && barn->nr_full >=3D MAX_FULL_SHEAVES) { > - ret =3D -E2BIG; > - } else { > - list_add(&sheaf->barn_list, &barn->sheaves_full); > - barn->nr_full++; > - } > + list_add(&sheaf->barn_list, &barn->sheaves_full); > + barn->nr_full++; > > spin_unlock_irqrestore(&barn->lock, flags); > - return ret; > + return 0; > } > > static struct slab_sheaf *barn_get_full_or_empty_sheaf(struct node_barn = *barn) > @@ -2742,6 +2743,9 @@ static struct slab_sheaf *barn_get_full_or_empty_sh= eaf(struct node_barn *barn) > struct slab_sheaf *sheaf =3D NULL; > unsigned long flags; > > + if (!data_race(barn->nr_full) && !data_race(barn->nr_empty)) > + return NULL; > + > spin_lock_irqsave(&barn->lock, flags); > > if (barn->nr_full) { > @@ -2772,9 +2776,12 @@ barn_replace_empty_sheaf(struct node_barn *barn, s= truct slab_sheaf *empty) > struct slab_sheaf *full =3D NULL; > unsigned long flags; > > + if (!data_race(barn->nr_full)) > + return NULL; > + > spin_lock_irqsave(&barn->lock, flags); > > - if (barn->nr_full) { > + if (likely(barn->nr_full)) { > full =3D list_first_entry(&barn->sheaves_full, struct sla= b_sheaf, > barn_list); > list_del(&full->barn_list); > @@ -2797,19 +2804,23 @@ barn_replace_full_sheaf(struct node_barn *barn, s= truct slab_sheaf *full) > struct slab_sheaf *empty; > unsigned long flags; > > + /* we don't repeat this check under barn->lock as it's not critic= al */ > + if (data_race(barn->nr_full) >=3D MAX_FULL_SHEAVES) > + return ERR_PTR(-E2BIG); > + if (!data_race(barn->nr_empty)) > + return ERR_PTR(-ENOMEM); > + > spin_lock_irqsave(&barn->lock, flags); > > - if (barn->nr_full >=3D MAX_FULL_SHEAVES) { > - empty =3D ERR_PTR(-E2BIG); > - } else if (!barn->nr_empty) { > - empty =3D ERR_PTR(-ENOMEM); > - } else { > + if (likely(barn->nr_empty)) { > empty =3D list_first_entry(&barn->sheaves_empty, struct s= lab_sheaf, > barn_list); > list_del(&empty->barn_list); > list_add(&full->barn_list, &barn->sheaves_full); > barn->nr_empty--; > barn->nr_full++; > + } else { > + empty =3D ERR_PTR(-ENOMEM); > } > > spin_unlock_irqrestore(&barn->lock, flags); > > -- > 2.48.1 >