From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F3D8C25B75 for ; Fri, 31 May 2024 16:44:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 245596B00B0; Fri, 31 May 2024 12:44:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1CD976B00B1; Fri, 31 May 2024 12:44:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06D766B00B2; Fri, 31 May 2024 12:44:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DA6646B00B0 for ; Fri, 31 May 2024 12:44:08 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 86D9E81233 for ; Fri, 31 May 2024 16:44:08 +0000 (UTC) X-FDA: 82179263376.29.B10B9F3 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by imf30.hostedemail.com (Postfix) with ESMTP id 8D2A58000E for ; Fri, 31 May 2024 16:44:06 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HagZHoYs; spf=pass (imf30.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717173846; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GgrPSqavQhhCwRo/IO4y9sSd78oBJ0BfHadESybGIRk=; b=5M3yXCu1MoS47cMIElHMzZPSF94iRu2MinKx/KPVU3ZuQ7oW2otbrwheDh5Ft59QHViQgk JnPXOSDGOJZufZJ8RNsO7pCHpVTWQsXZ8l5yyvBJhNkz3hgG7KH59bpXWpeYGDnNe4JPne cM4J0zoZjvpu4P096gZm2zVytKt4PBg= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HagZHoYs; spf=pass (imf30.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717173846; a=rsa-sha256; cv=none; b=uj8G67OHSEwZ2f2x+E4hKCqRn4ZaG4HmWVCvw+Rv9QoczWVeCyG262hNLc7dADr0q5sfvU HaJT2V5eDv605MtgR9NQyvONQztj+x2Vf23bCI1MKNvrUtIDc1WjJgn/C87sMsWzLQXlmK /cRAvm3iBRkA0wRe/5kLUWAhiyeOIRI= Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-421208c97a2so21008795e9.1 for ; Fri, 31 May 2024 09:44:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717173845; x=1717778645; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=GgrPSqavQhhCwRo/IO4y9sSd78oBJ0BfHadESybGIRk=; b=HagZHoYsh/wcmDkIuNba4KxxSRSZAokXGqcCo4MzRmaMSUdJCUNxACNwwUpPVaCYQP CEs5mmCyVuylSRv2SSkO9NutG1OZuTIqM4YGicOaZGjC02FM0/phK+ujl+lUNXcKqiDy rN6im1qvCaZIYOpNd0wCl4J9Pqy2pwdZQke0zfrmyPt38uTIcDCFT9AWJtNytVkZ/9Jt f6/i4PQMVmMt608W6RqTH7WT//gGn3IZqu821jFaZFzg1LIcH7W7OYh9ASMGmm1JNWWr xdWzii7BuoxqD1dEMlsA0CyA+MF0mbOMbdclI6TKeGDFBFngRiS3YEr/Edkafe/ul7Tn f0Eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717173845; x=1717778645; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GgrPSqavQhhCwRo/IO4y9sSd78oBJ0BfHadESybGIRk=; b=XDBrf9167+D7A+djsdnOvhUcdoCdvaTpX2snb/nQN8PbC9Cwu74iZNTQMBqmT0pyv+ UgMV1sBK4tYFfHyfbwP83Q0z2y4SMP8HL5zstAu+i8pY1KBXdbepOEc74shxwQ3sH+/M VaHG2cp4Tc31tknlEquBPxfV+3AvilUjkjjIsiS0OSmoLW/bSdMfwRUqikuuApFCGKYH Ua9ftGQGpFcde5lQzEf/7FOdqACaBgqStLigmVaQadbfI9rX4Z7GoKREXvxfuhVc2GtA AI/V/VT28YP3qojUwVy9vTiRlSwoJdQl4WjyAImrTuLarDXY0inlW+RBWLhhkiPTCvDG ax9w== X-Forwarded-Encrypted: i=1; AJvYcCUN0knB52n4gT4IU5QWSrZ5/aKruVWYMiakjF8Zf8PXbF1gz5/w8zsw9nNmOF39D/J7XoUl5nWChOeDB0UkWrWe31s= X-Gm-Message-State: AOJu0YzCbl2BuUT22mSW3qeZtCo1XKE9t4NWCjKIvmXPdVvYxt5Wb9dv zSRVxj7TcrKXFbQASN0K/A3KGu5VSnyCwvRfKs7qe14I8oLTFU7IiwOJBxqLmgj3Odlx7jcpOsK 4UzqwlzlxyB6mvkT1FajGN1582o4= X-Google-Smtp-Source: AGHT+IFTytoJamLJRvTQO3BYC2MBDxCEuBj6lgJTYnD/N3Qp380NOujMSa5JG5AI3SYo5H23tb7dmLf0PjIWpX8mTVk= X-Received: by 2002:a05:600c:3541:b0:41a:c86e:a4db with SMTP id 5b1f17b1804b1-4212e04732amr18706075e9.9.1717173844917; Fri, 31 May 2024 09:44:04 -0700 (PDT) MIME-Version: 1.0 References: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> <20240531-fault-injection-statickeys-v1-3-a513fd0a9614@suse.cz> In-Reply-To: <20240531-fault-injection-statickeys-v1-3-a513fd0a9614@suse.cz> From: Alexei Starovoitov Date: Fri, 31 May 2024 09:43:53 -0700 Message-ID: Subject: Re: [PATCH RFC 3/4] mm, slab: add static key for should_failslab() To: Vlastimil Babka Cc: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland , Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, LKML , linux-mm , bpf , linux-trace-kernel Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: zbap31uh43yj7qxbgqo5ni867ns6xx97 X-Rspamd-Queue-Id: 8D2A58000E X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1717173846-764500 X-HE-Meta: U2FsdGVkX18gkefXY2KG9CWl+lSLYCDkSg7lJDxmJser5vp6BGhI5RGVv4q9Ga+TxQRpT/ll5c8b8JO7ZkArpSFsVTddlgyrgBBtLMXObsjgvk9WOR7iDyFo38FUIokzk4E/Vjt99oRboflrii2qTsRPAW46AABvr+/Fn+/5fPBUVT7beJhxyhp5QTZA2cOySgTJBva/QnPMRtZCZbi+dhLrV7oY1MNkGMjXA/jMfDJY1dbE4/idLUa7oSeplQyaOVJ5zHunr0C7pUzgBgQgf3HGdMIEjjEhGf8vxae75RVaEBC2VL09rlWiqgf3Yau79c1fwQ5zjJFNi45LcQO0g//89IgJzuJNbfQQvWIr7atkC8Jev7lnEEI3HlgWW6vPm29mi7QmiFAFgyLj6m9GHq8++VcfGAn3vy3rJMjeoJRuKxEfEbzsrNrBn9uMvWHV0gChxrVPos0/ENV5O/MJw/w7bGBj0PHbxulEYlROO9TCCH7TWjwqFLl+21mE6pLGmLUaqAyl18kpe0S9Ed75Ym0XD3qTpdFylXdpDsdzaQYD6GxL4IrMlcMcA56WfQk/pdZ4lEuBttbLCB38jkydz3BZzmGSeQQ4mJ4ixqaIn8YRT+DuOD5nYJAS3Dxj8Lamh71UlHSHB9NYTM1L43EbpFYxSyiGm4KIG0Bnp02pQRdV3h58y3bNaRIaTbwi11W06D/Udb+M1dJxtiqGutIouS0zWVD3HYxYPts8FtmhUlIYFE+75pLbaBGlnjhGofmbpxSiIlhFDEqHsSeuOmkH4a3UltxVsj6pQKdm5Zff6vJ3Cu4+2R7JbpSg0oIKhQZ2j2ladacEsgivExVGdcnWDY8ep5NsBbGwziZF/UNNgNsTU5Pqev8hTeL7ZJqGmXLOwOTQsxTxDMYIlpR8ywuJkqJFfV3Wfj578ZwMFLn3KvX9T2MaAa7fCLxM9oSgkqRZcyE/P7vTqmkUEbxTDLj 5AHaTmiv OI783YhHfBA/qlc4CFzCS873+DTgPUybv4nrGpHUya8l1ONQp5STEkoORS0tp8ldnEa6gFdOEzgxQzo9bAxIoh74vVkzob5+AxXP1bLOqN7YIOCnmKW759KBRj8Cac/QyFoCruiKrIQaU9U9JKBvC3EkaQESFFA0MGYnAtm80S5oAc8WPRCqqrx1dB9JXPsu0qmiXEoxhzvAxbDIXs6WVGHrWkBrVU45r5I/zXiR6TPH2w2xXiuXKM9xz7Jwitr8Vv/3lYm8nQEtVisrz0Dr7ZqeoxVCXa9Cn4QcYAP2aksPz+pBOL+NeCFyIoSltzHUwoRuLeV7XmuD9fjuawY6n6JpIoNLd8xqmXSnM0PXudbTjVrOfRkuOj5iyGPB4aMyWPx/SF8n1WnBim+/nitWH/hUvtXVeY5RfF8pdvnmHPQFrkTKH2ySRotzjnMlcEbKqn3pNwcGF1dXy8z1NSbK4yOX8pxKQD4qZMWRjPGrHqlvtU6Ki/oG5lSsfQldviddC0onmaOKD2OXIiT7EidQcnFUp9olbNSbQ08/nkzixGwrckzL/zqNtvKuB2oRDddO6DHqLWx48lHF/xqIVQsi7cmaMZg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, May 31, 2024 at 2:33=E2=80=AFAM Vlastimil Babka wr= ote: > > Since commit 4f6923fbb352 ("mm: make should_failslab always available for > fault injection") should_failslab() is unconditionally a noinline > function. This adds visible overhead to the slab allocation hotpath, > even if the function is empty. With CONFIG_FAILSLAB=3Dy there's additiona= l > overhead when the functionality is not enabled by a boot parameter or > debugfs. > > The overhead can be eliminated with a static key around the callsite. > Fault injection and error injection frameworks can now be told that the > this function has a static key associated, and are able to enable and > disable it accordingly. > > Signed-off-by: Vlastimil Babka > --- > mm/failslab.c | 2 +- > mm/slab.h | 3 +++ > mm/slub.c | 10 +++++++--- > 3 files changed, 11 insertions(+), 4 deletions(-) > > diff --git a/mm/failslab.c b/mm/failslab.c > index ffc420c0e767..878fd08e5dac 100644 > --- a/mm/failslab.c > +++ b/mm/failslab.c > @@ -9,7 +9,7 @@ static struct { > bool ignore_gfp_reclaim; > bool cache_filter; > } failslab =3D { > - .attr =3D FAULT_ATTR_INITIALIZER, > + .attr =3D FAULT_ATTR_INITIALIZER_KEY(&should_failslab_active.key)= , > .ignore_gfp_reclaim =3D true, > .cache_filter =3D false, > }; > diff --git a/mm/slab.h b/mm/slab.h > index 5f8f47c5bee0..792e19cb37b8 100644 > --- a/mm/slab.h > +++ b/mm/slab.h > @@ -11,6 +11,7 @@ > #include > #include > #include > +#include > > /* > * Internal slab definitions > @@ -160,6 +161,8 @@ static_assert(IS_ALIGNED(offsetof(struct slab, freeli= st), sizeof(freelist_aba_t) > */ > #define slab_page(s) folio_page(slab_folio(s), 0) > > +DECLARE_STATIC_KEY_FALSE(should_failslab_active); > + > /* > * If network-based swap is enabled, sl*b must keep track of whether pag= es > * were allocated from pfmemalloc reserves. > diff --git a/mm/slub.c b/mm/slub.c > index 0809760cf789..3bb579760a37 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -3874,13 +3874,15 @@ static __always_inline void maybe_wipe_obj_freept= r(struct kmem_cache *s, > 0, sizeof(void *)); > } > > +DEFINE_STATIC_KEY_FALSE(should_failslab_active); > + > noinline int should_failslab(struct kmem_cache *s, gfp_t gfpflags) > { > if (__should_failslab(s, gfpflags)) > return -ENOMEM; > return 0; > } > -ALLOW_ERROR_INJECTION(should_failslab, ERRNO); > +ALLOW_ERROR_INJECTION_KEY(should_failslab, ERRNO, &should_failslab_activ= e); > > static __fastpath_inline > struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags= ) > @@ -3889,8 +3891,10 @@ struct kmem_cache *slab_pre_alloc_hook(struct kmem= _cache *s, gfp_t flags) > > might_alloc(flags); > > - if (unlikely(should_failslab(s, flags))) > - return NULL; > + if (static_branch_unlikely(&should_failslab_active)) { > + if (should_failslab(s, flags)) > + return NULL; > + } makes sense. Acked-by: Alexei Starovoitov Do you have any microbenchmark numbers before/after this optimization?