From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 337E1C25B7E for ; Fri, 31 May 2024 17:18:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 968276B00AD; Fri, 31 May 2024 13:18:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 918446B00AE; Fri, 31 May 2024 13:18:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E01F6B00AF; Fri, 31 May 2024 13:18:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6244E6B00AD for ; Fri, 31 May 2024 13:18:40 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 099F912167F for ; Fri, 31 May 2024 17:18:40 +0000 (UTC) X-FDA: 82179350400.05.C4AABFB Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) by imf23.hostedemail.com (Postfix) with ESMTP id 2A49E140002 for ; Fri, 31 May 2024 17:18:36 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xZIxAGPS; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717175917; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=knmodrcY3ruf2qGktj4YBIa4Hu+EWJ5K27FOck3RpeQ=; b=3KxA2wNaT//l0lCWs0jYSaIERrep3PsC1j6Ory6U9mmEr/oIKENP6AV3HH1ljfBWTVZWjZ b1roYJjKV/B4aqkj0d8YpxAWq9djyGIOX3b6x4lk1lc40xt0sX3CXyNTG/iRN0szdlsiIT mUZTA2ReS6SvxOP4AZE9A0PfA4Y9Umg= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xZIxAGPS; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717175917; a=rsa-sha256; cv=none; b=nZrjl++z/HpXZ0Bc/fbXjoEbzPNeh/t986+voOzPQYjJkcXhY6YATMtMY16B37fY2nnRxP wJ8Dggflv74izrj3tBOK9Dsw81XIXEd73BLyJuhE7PZqJjsEtBo0awInlM5DTlATIggoKY bDN4U5GCnnLV4ZKhXgMhYILRh14JlUE= Received: by mail-ej1-f54.google.com with SMTP id a640c23a62f3a-a688ebb75b9so63355166b.3 for ; Fri, 31 May 2024 10:18:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717175916; x=1717780716; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=knmodrcY3ruf2qGktj4YBIa4Hu+EWJ5K27FOck3RpeQ=; b=xZIxAGPS1dO9cS2Yy1P3oGr7UvOp2k6i/CuluajPbak3nzposyXMTXmY7WoU52qiMO njcTMeyxdjpRxE10DLo3/dHvm60CF1LpCbUPZESS84IPl5e4jDT8NPbUBoznbwvTMTWj QWMYaHqHk9tPcxuvRWSfjnWBUUICAl0RaS9RkOyqC/vzhXWVogZzwuXU6oesPXN52IAQ UPwSDmUmc30TbqEap3/rqPJemy3eflwyZ9w2rX65kV6DpzfSA5NIrvKiJS2317IwFiG/ Bp6ttDrW+fRPYgK5zFPYl5eR1jU2jRbcY46tbz/5vSyatLOzaG1fzwJKI+VHOq/588eF MDRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717175916; x=1717780716; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=knmodrcY3ruf2qGktj4YBIa4Hu+EWJ5K27FOck3RpeQ=; b=GXhvuC+PJjqqtn9CwIoXQ64tXU6da4AOtJ411w2CRKaG3NFDKkUmw/d+tD6umTFiD2 n95kApiEHmcCICMRJ4zG6BQmSEWM7UE2XM62k6c71ntQJUgYx1xfg7CRJi1Fnet8nTzU LEVdIWzNSysUdIqpfXI0HZMpBe2cZSNNIFHpqdRw19AunO5vDGXWIuhM2pjjsQs3nCy3 q4ll5TQ/97BD2p1IGeGobKarWs49fe8y0s43oIRgv94MqYCEu2PO4QvCzsYBSXiJGteq UwQFuMSXggWb4bjtbw71HyO7mvBNkZuRv4m99O+wCUjdilynQ8vTMIjt1U7v3T3qK3QH +zqQ== X-Forwarded-Encrypted: i=1; AJvYcCVmw/w8p0rWRyvDEbaaFtvj51EBNTwhumcQRiOKEK9TeBudl9ifFNAJCPyeiUiSsFdPVx6tghkeA3tTbJXqLSbp5fI= X-Gm-Message-State: AOJu0YzV/2krUzHVInT8K3QBKYuFh2zD3f0kUIB3DMqPXhTnlX2ZxcAN HI1UGLPvo/kYk/ffNBmRedIXpHt+F7xxVxNFbAPA+hQwN5o9OWqHsMWP09UVXZgk41nxpFLollm mlSogrTjGibtRC7sO5rTLtMlisRZWzuc8WX4b X-Google-Smtp-Source: AGHT+IFq57opgQgUB+TA+18B1Szt8imqxEQZcxRZEyIW0myS+jg9WzpqrrZiKo7rqzk0TmUBniiGQ6+5D9+jhENDG1U= X-Received: by 2002:a17:906:a1c5:b0:a67:b440:e50f with SMTP id a640c23a62f3a-a68224472c9mr177860866b.63.1717175915378; Fri, 31 May 2024 10:18:35 -0700 (PDT) MIME-Version: 1.0 References: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> <20240531-fault-injection-statickeys-v1-3-a513fd0a9614@suse.cz> In-Reply-To: From: Yosry Ahmed Date: Fri, 31 May 2024 10:17:57 -0700 Message-ID: Subject: Re: [PATCH RFC 3/4] mm, slab: add static key for should_failslab() To: Alexei Starovoitov Cc: Vlastimil Babka , Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland , Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, LKML , linux-mm , bpf , linux-trace-kernel Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 2A49E140002 X-Stat-Signature: mcczyy5xjyn3pco8kw3bxpzsgknrb179 X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1717175916-994494 X-HE-Meta: U2FsdGVkX1/GTo8jptzbP9880EglfZeT6NhqaHDNNco1V8puob7otekrr40nDHL+TTROdiAVKswAa++97CJmK0/x8OXiCTaN5bv2k5a2BsJEpaOD5tPivrp+xlKsb0Al9Bd7ZGJNQrRK0jhE+BqOsM/7f1W8VGq4UaSm2uOOesXXxK4sq+PMEiBTBsyM/1EQKgD7pKKhIFfAF5hBBeFjg5kp28V43gUUmYMmvo1AKAHHHWMdYkCplDH3A1N671HkD4JZTVCqPWtI2f9i3tGqeMK5S0KgcB4koI8zbQGYRoq8jzzV99ztDZVR96NjlPG9O+9HHFaDYKNolQEe2Ll+cp0PmpMWfCD971mCW0zFylzXt8lGW+5yP7zeLEdZYOQmTWEi4Tr4B0hsanfBMLdNOnudRacHyB4RLEkwjygbT+9CZFMWCLMkMahIPTvPXUa/Xq36haIJNw4rDGZxuDVofhauZzenxw0yVowhYRdmT9DZ33WZe4sAoql3MQaR0T3t/BM9QTYKbg5mIcZpKtYZXejMUyBZhR8xItnD016A1NER7tNDooysKNTFhM3veJMSM0GzKM8IRM+prxcTss8wz4fBi1We7T4kQ4fVceRbS7NX5Xd6vkJc/d48a8z7r4AaEgEvqMd4r3OofOUOWpuaMfGRCa86ZkAN0Ty/L/lycAnL47Di4NKKn8dhfG9RHcy1YCpsJqiPEkx/spwcKFHN/v/H5RnT85TkjQ++bVdNXfhqKlNM2fG4Ku/NeaRl7bTFhbRNnP83+j/sWz8z0ytsdIEEvnN6mdobDXI0RPSVItMBQlwvVryt18uBKT/P2y8Up59Mvi3C9HJF9nKDNaxT/stit155CQh+WF2wKqXYfb3UY9P55xTs1TFmlglLUkLGQy0tvxXjMDuTrtMyi8yJQpanj5ArY2W8gj5hfgWzmdzmVbOcsu4zzWEL4jEpQxGEJgJvAE8PS+lJpZcIAak pdDtkseG vH9qKCQUXEsf71h+EeFtcBsckEHiRTwcuGF9fEIWbRxFl3k10y4lEo5t4uRJIKuQQ5fDUONwvfoxM/DU8MzmOO2Fm5KOn72+PQwym7nkdz3vfndNDy104kZvIs0ed05Ue2frRU16vKacUJFJ1tTpyv2xM92gxeS98f7E3aUqdRdEGCaOhBilPCCR3cVGHuJdztsKANjWvxYXd1r1291L6ayDAOfTHBGEx16QfDhWTwi75xcHoDrmBlVP/dGXI9HSN8QUuQTTgBWsR0TcOu56r6H4PKZbbEqNRvNjs2q7J6fWn473RxhOcGGF0EhbJh7KJT2rzqXui7KsPMNVR0eulBeg8SZftJVun4T1dQQFPXeKGL6xyzNIPFgB2OlmphmfhOWpsSZ3lXB3qB/7McwvbHNOTDSqY4EPJkNnwDhXXnRKoeNc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, May 31, 2024 at 9:44=E2=80=AFAM Alexei Starovoitov wrote: > > On Fri, May 31, 2024 at 2:33=E2=80=AFAM Vlastimil Babka = wrote: > > > > Since commit 4f6923fbb352 ("mm: make should_failslab always available f= or > > fault injection") should_failslab() is unconditionally a noinline > > function. This adds visible overhead to the slab allocation hotpath, > > even if the function is empty. With CONFIG_FAILSLAB=3Dy there's additio= nal > > overhead when the functionality is not enabled by a boot parameter or > > debugfs. > > > > The overhead can be eliminated with a static key around the callsite. > > Fault injection and error injection frameworks can now be told that the > > this function has a static key associated, and are able to enable and > > disable it accordingly. > > > > Signed-off-by: Vlastimil Babka > > --- > > mm/failslab.c | 2 +- > > mm/slab.h | 3 +++ > > mm/slub.c | 10 +++++++--- > > 3 files changed, 11 insertions(+), 4 deletions(-) > > > > diff --git a/mm/failslab.c b/mm/failslab.c > > index ffc420c0e767..878fd08e5dac 100644 > > --- a/mm/failslab.c > > +++ b/mm/failslab.c > > @@ -9,7 +9,7 @@ static struct { > > bool ignore_gfp_reclaim; > > bool cache_filter; > > } failslab =3D { > > - .attr =3D FAULT_ATTR_INITIALIZER, > > + .attr =3D FAULT_ATTR_INITIALIZER_KEY(&should_failslab_active.ke= y), > > .ignore_gfp_reclaim =3D true, > > .cache_filter =3D false, > > }; > > diff --git a/mm/slab.h b/mm/slab.h > > index 5f8f47c5bee0..792e19cb37b8 100644 > > --- a/mm/slab.h > > +++ b/mm/slab.h > > @@ -11,6 +11,7 @@ > > #include > > #include > > #include > > +#include > > > > /* > > * Internal slab definitions > > @@ -160,6 +161,8 @@ static_assert(IS_ALIGNED(offsetof(struct slab, free= list), sizeof(freelist_aba_t) > > */ > > #define slab_page(s) folio_page(slab_folio(s), 0) > > > > +DECLARE_STATIC_KEY_FALSE(should_failslab_active); > > + > > /* > > * If network-based swap is enabled, sl*b must keep track of whether p= ages > > * were allocated from pfmemalloc reserves. > > diff --git a/mm/slub.c b/mm/slub.c > > index 0809760cf789..3bb579760a37 100644 > > --- a/mm/slub.c > > +++ b/mm/slub.c > > @@ -3874,13 +3874,15 @@ static __always_inline void maybe_wipe_obj_free= ptr(struct kmem_cache *s, > > 0, sizeof(void *)); > > } > > > > +DEFINE_STATIC_KEY_FALSE(should_failslab_active); > > + > > noinline int should_failslab(struct kmem_cache *s, gfp_t gfpflags) > > { > > if (__should_failslab(s, gfpflags)) > > return -ENOMEM; > > return 0; > > } > > -ALLOW_ERROR_INJECTION(should_failslab, ERRNO); > > +ALLOW_ERROR_INJECTION_KEY(should_failslab, ERRNO, &should_failslab_act= ive); > > > > static __fastpath_inline > > struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t fla= gs) > > @@ -3889,8 +3891,10 @@ struct kmem_cache *slab_pre_alloc_hook(struct km= em_cache *s, gfp_t flags) > > > > might_alloc(flags); > > > > - if (unlikely(should_failslab(s, flags))) > > - return NULL; > > + if (static_branch_unlikely(&should_failslab_active)) { > > + if (should_failslab(s, flags)) > > + return NULL; > > + } > > makes sense. > Acked-by: Alexei Starovoitov > > Do you have any microbenchmark numbers before/after this optimization? There are numbers in the cover letter for the entire series: https://lore.kernel.org/lkml/20240531-fault-injection-statickeys-v1-0-a513f= d0a9614@suse.cz/