From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8FA7C7EE30 for ; Thu, 26 Jun 2025 20:04:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B0986B00A1; Thu, 26 Jun 2025 16:04:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75FD86B00A2; Thu, 26 Jun 2025 16:04:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64E766B00A3; Thu, 26 Jun 2025 16:04:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 508CF6B00A1 for ; Thu, 26 Jun 2025 16:04:11 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EA1E3C03B7 for ; Thu, 26 Jun 2025 20:04:10 +0000 (UTC) X-FDA: 83598628260.14.B6D242A Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) by imf03.hostedemail.com (Postfix) with ESMTP id 04BD220009 for ; Thu, 26 Jun 2025 20:04:08 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Z8JmUw5T; spf=pass (imf03.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.221.49 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1750968249; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rQCbS/HjgZKKichcZaS63aazCIIPAdL8cR0fR9IrlVI=; b=Ry6SC6BumV4XQDrttLNygyW2Stv3Jlv6LE/d6mlYw9Ktsvp4SmjVsd/V/VWnY8KMqsFdLX 7j9HBX0NIuxgvbNNn0SPccruuGc+0MFVgcib0jcwwDpumJyai4KnAmjYrCGQvxaSJNaxep GeGrNhJQ5UfP+wceEmvzWzA6xVZzmA0= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Z8JmUw5T; spf=pass (imf03.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.221.49 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1750968249; a=rsa-sha256; cv=none; b=oBJFuPLtCDyrWpW7eBDGVODN/XCqtV7fOJ+YecFDQSRAOaAHnBY7+eKFkS6If80gpaDviJ bUVp6QWNrz5+XDWvq0q0ZnrRc0wg+0LhL8+tOg480C1wpOutcu++dZYvzHbbm9FfH4vZlZ 7IcpOB0SVa4KdQCOUdub6irgZ9KlRnM= Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-3a582e09144so899827f8f.1 for ; Thu, 26 Jun 2025 13:04:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750968247; x=1751573047; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=rQCbS/HjgZKKichcZaS63aazCIIPAdL8cR0fR9IrlVI=; b=Z8JmUw5Tdm1W37+PokfeDcJGaDm9jiG1O9iauEu+3hsaxQ8k5gjRUFZE050iwbc1lm OfmsNf2O8D42MU9rG0esHibTNuRU5ju2tP5olMYbq49vMFAvN3tnBV+f5eOiOFjocYUX k0tYDXsf4uZWDARAFD6++pexj23yc+pXk1p73nmpV3N4PdN748ynCbxh35qpZqULNTgZ qcdH1XRvYBdOqLhv1laWTimOr/rWYSX8SY+zJsHLj/0NyJafgG2YwMeLDIH+WPhKGlGY LSgjmnTP0QGtOfA+e+x4AzUzwZe4XX4tl5S5GpKhPclThm/2BUvB6TqHddreVPo+pBKT hzIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750968247; x=1751573047; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rQCbS/HjgZKKichcZaS63aazCIIPAdL8cR0fR9IrlVI=; b=AYMIbMbvx52vdKDM0aWF/dkBl4FhfIr5DlZlB0bYblVhlVNk3cQTfn54v4vuIISaWc O1FKxkGJo5+ewNsKksVMDCYXVHEpYFAg9W/IiR5SG7QclCxsgrOwak3KaNsoTmAcLYtA 0QQ3BfZZLy4JpEXa/V13H63A7/tuVIC7SYwzXNrdN9ljRM+XbikRfqfnikcTKgwHl4Wf ZYModDwCAsAWBL88E42APdt0qInzm9ezgGPkSTYP0llgzL60YPBDhs3adqj6zixUU9Ba leCU3J0J2ASr7uK00lVSeyO60hp1ujQYubFrZW9f5EI9QJjQfh6R5PMq7HdOx3wKJgI2 fCXQ== X-Forwarded-Encrypted: i=1; AJvYcCXKuMcJGg1ks595P5HL95IFWMtDYqXUM/0513VFEbZbBC1FyjBBc6XCbZ05vWLIDjTW/aIwITE/tg==@kvack.org X-Gm-Message-State: AOJu0YxDT+f/jRi1x7Lw9y0jHfMcKQHrFVT1dir0GPPneSRs0OhDjrpk V/zZeqcNJgH6WWdn5BR/ejI2AcDhgpxi/buMDhRIc+wI2gbHwgMHWGzSMiHjHyGx9iEO/DBCvvn 93QzgT6itwRZCS1QiILlJ3ROTsE/X9Cc= X-Gm-Gg: ASbGncsV2I+69ykBD3aTMPXhvsgD8eSQJc/KBpmrSDnqKhPbMWRCNwttfbygaANQLXI DJRsShbb7k+wQ4S5tjXcCJDkpUqTMhHW9rmSgBHPKniXh/kIGvciTe1pRt4nTA20o7bZxkvBN8R +SumBcFxwzL97YCbCruGRz8owLw6fW62A4iW4mSZQiIQsDc+CqpSZULX3pZqsobOtrR57vHH/KZ dYSBsK6Jbk= X-Google-Smtp-Source: AGHT+IHNsEyH4zXsR8XhUHCuIlAny2GHVSvoSwZ3KwdYJ7vl4ofvMDE2bRkJ4IU245CbPVCBaieFZi5SqWppHKQ8a2U= X-Received: by 2002:a05:6000:270e:b0:3a4:e609:dc63 with SMTP id ffacd0b85a97d-3a8fdb2a345mr531042f8f.20.1750968247103; Thu, 26 Jun 2025 13:04:07 -0700 (PDT) MIME-Version: 1.0 References: <20250501032718.65476-1-alexei.starovoitov@gmail.com> <20250501032718.65476-7-alexei.starovoitov@gmail.com> In-Reply-To: From: Alexei Starovoitov Date: Thu, 26 Jun 2025 13:03:56 -0700 X-Gm-Features: Ac12FXzYRjgVGeObofZlNcvYhCYwFL26bL4avlqFpoSTBUZRSJ7eZFkwdTTjh9A Message-ID: Subject: Re: SLAB_NO_CMPXCHG was:: [PATCH 6/6] slab: Introduce kmalloc_nolock() and kfree_nolock(). To: Harry Yoo Cc: bpf , linux-mm , Vlastimil Babka , Shakeel Butt , Michal Hocko , Sebastian Sewior , Andrii Nakryiko , Kumar Kartikeya Dwivedi , Andrew Morton , Peter Zijlstra , Steven Rostedt , Johannes Weiner , Matthew Wilcox Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam11 X-Rspam-User: X-Rspamd-Queue-Id: 04BD220009 X-Stat-Signature: aszem8hef4eynsgdeb3rsbtsxp8sjm7q X-HE-Tag: 1750968248-810648 X-HE-Meta: U2FsdGVkX19lh1YvbYoCjlwzXaJ075TcipSx4eiOSGTJOCk9+d/1mNpEKKg/SD6kFPCRqx7K56tmHI94NpyfC6YCAEQwBeK9+uPimCejRlf1JlSoGsC/ThrMuefUb54A7+a3QhsfsE8cCSG0rliRpdP3jHV5fasRzrf3x//n1EwihS06dQqYscO12D5o4gPNDGimv3qxeqS87ntoFgtJv+MnwdF+PuY+0hkKcAKewGZjg9/QZAHyjJtdIS/M/JYoNYbA41P4DL8k7sPddYZFK99S4H6ThguXhXTR88nJTkO3B4WJwt6A+Hk504oSvgcWAsmn7bTHC7yiERO/9JbQ5YxHvN97qPlZe5RqgpNMu6/wIWMNoxAU4FjaPZd9NboVLE3srMSVIY/+YrMHiuZwFP5q7t7Pn17vv0HZBS1gvJopgHGJyiFRJAj29ErCkt1QuHz2eG691eCSHAnxpkYD1IRVXTis8WEtoVOW22HAxxxFKeMhqzo5uwCH/wRw+9ihdcs714JAecGxZH88MACtfiAsSk1xtGuKub4bEAJtG/MdoasDj7U2hEUD3gV8ez/vwzdsKxy+6vy59eDBUcR3NYbVE14fA06G3d48uLTWrbeLI35Zi2jMRe/BfADpU1rGWTavlZGsm3LR7HXCOOtJwG1jjzqMEKe4Z2m6t31g5siTZzub6SNMjlObMhXMkN8VYCvYfXWcyyk4SzlFLJKSFFIp7CczhfpIzjNixIUd1niyUVCUvxRGCW4kHII9ZoI9F47FMMr0fa8uDivjeIZYzYt7tmNVbtO1lp11Vjoh1QNQGZ08tw+wEMjrfZCZzhhj+99QbTpqmvQdmipnlVLlUtfzq1X/LOkfGaV6O2R0DyQEruS6vXz5jSBZFshK9+hAxMPKqk+nTAw64oJRBhKtUNgxZETb3rd+Imffwb5zadEULV24gGE04oOp4GKj/pY5/Qs5pUz04OpaK8zZQVY Z1iS6nb1 zig2Aad0gc/qJxWUYIdsg2bs4PgK1MJ2G010aBNP8eD9eGErZea4WptNRSmDyggDZ14R7Pj/lIye4EU/k7sFLSafXfZPflMfjLq7kz2h/38y2xXqYVYjqMmlKKaR3MPvB0BLi5NLnYuSUFOC32D5ZrQTMBqtp5vXBBydff9GGqPE+0/gG5vADM7R27VR4+LUHynnPqt0hvsXn6wXSaaHw1I408MDp6+FJQ0dHpGdzZa/huI1FxASadshuGjbfrtqirdy2RJklJOHUfvWyoYT+gpAQV2PER+1q+IGV7Poz2jf0zhHwHB4XWHCL4BHaZklCtfN203/QGTx+I2RZhcAYwU2htEQap0nwt/PsjbBzgRIFP6GhV9o1zJrtu1VFzKXJ1RDT7EoZhUshTItn+QtP5VjLgoEHvDbM3hYx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jun 25, 2025 at 4:38=E2=80=AFAM Harry Yoo wr= ote: > > On Tue, Jun 24, 2025 at 10:13:49AM -0700, Alexei Starovoitov wrote: > > On Thu, May 8, 2025 at 6:03=E2=80=AFPM Harry Yoo = wrote: > > > > + s =3D kmalloc_slab(size, NULL, alloc_gfp, _RET_IP_); > > > > + > > > > + if (!(s->flags & __CMPXCHG_DOUBLE)) > > > > + /* > > > > + * kmalloc_nolock() is not supported on architectures= that > > > > + * don't implement cmpxchg16b. > > > > + */ > > > > + return NULL; > > > > > > Hmm when someone uses slab debugging flags (e.g., passing boot > > > parameter slab_debug=3DFPZ as a hardening option on production [1], o= r > > > just for debugging), __CMPXCHG_DOUBLE is not set even when the arch > > > supports it. > > > > > > Is it okay to fail all kmalloc_nolock() calls in such cases? > > > > I studied the code and the git history. > > Looks like slub doesn't have to disable cmpxchg mode when slab_debug is= on. > > A slight correction; Debug caches do not use cmpxchg mode at all by > design. If a future change enables cmpxchg mode for them, it will cause > the same consistency issue. > > > The commit 41bec7c33f37 ("mm/slub: remove slab_lock() usage for debug > > operations") > > removed slab_lock from debug validation checks. > > An excellent point! > > Yes, SLUB does not maintain cpu slab and percpu partial slabs on > debug caches. Ohh. That was a crucial detail I was missing. Now I see that 90% of ___slab_alloc() logic is not executed for debug slabs :) > Alloc/free is done under n->list_lock, so no need for > cmpxchg double and slab_lock() at all :) > > > So right now slab_lock() only serializes slab->freelist/counter update. > > It's still necessary on arch-s that don't have cmpxchg, but that's it. > > Only __update_freelist_slow() is using it. > > Yes. > > > The comment next to SLAB_NO_CMPXCHG is obsolete as well. > > It's been there since days that slab_lock() was taken during > > consistency checks. > > Yes. > > > I think the following diff is appropriate: > > > > diff --git a/mm/slub.c b/mm/slub.c > > index 044e43ee3373..9d615cfd1b6f 100644 > > --- a/mm/slub.c > > +++ b/mm/slub.c > > @@ -286,14 +286,6 @@ static inline bool > > kmem_cache_has_cpu_partial(struct kmem_cache *s) > > #define DEBUG_DEFAULT_FLAGS (SLAB_CONSISTENCY_CHECKS | SLAB_RED_ZONE |= \ > > SLAB_POISON | SLAB_STORE_USER) > > > > -/* > > - * These debug flags cannot use CMPXCHG because there might be consist= ency > > - * issues when checking or reading debug information > > - */ > > -#define SLAB_NO_CMPXCHG (SLAB_CONSISTENCY_CHECKS | SLAB_STORE_USER | \ > > - SLAB_TRACE) > > - > > - > > > > /* > > * Debugging flags that require metadata to be stored in the slab. Th= ese get > > * disabled when slab_debug=3DO is used and a cache's min order increa= ses with > > @@ -6654,7 +6646,7 @@ int do_kmem_cache_create(struct kmem_cache *s, > > const char *name, > > } > > > > #ifdef system_has_freelist_aba > > - if (system_has_freelist_aba() && !(s->flags & SLAB_NO_CMPXCHG))= { > > + if (system_has_freelist_aba()) { > > /* Enable fast mode */ > > s->flags |=3D __CMPXCHG_DOUBLE; > > } > > > > It survived my stress tests. > > Thoughts? > > Perhaps it's better to change the condition in > kmalloc_nolock_noprof() from; > > if (!(s->flags & __CMPXCHG_DOUBLE)) > return NULL; > > to; > > if (!(s->flags & __CMPXCHG_DOUBLE) && !kmem_cache_debug(s)) > return NULL; > > Because debug caches do not use cmpxchg double (and that's why > it survived your test), it is not accurate to set __CMPXCHG_DOUBLE. > > And it'll get into trouble anyway if debug caches use cmpxchg double. All makes sense. Tested this suggestion. Indeed behaves as you described :) It's quite a relief, since I was also worried that __CMPXCHG_DOUBLE limitation of kmalloc_nolock() might hurt debug_slab cases. Only slub_tiny case left to do before I can post v2.