From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14734C02192 for ; Sat, 1 Feb 2025 23:18:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A02BD6B0083; Sat, 1 Feb 2025 18:18:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B1A86B0085; Sat, 1 Feb 2025 18:18:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 805056B0088; Sat, 1 Feb 2025 18:18:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 60CD46B0083 for ; Sat, 1 Feb 2025 18:18:11 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0C1AB140F44 for ; Sat, 1 Feb 2025 23:18:11 +0000 (UTC) X-FDA: 83072941182.24.8EA88AD Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf02.hostedemail.com (Postfix) with ESMTP id 3F09180009 for ; Sat, 1 Feb 2025 23:18:09 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FfKAqiPc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3sKueZwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3sKueZwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738451889; a=rsa-sha256; cv=none; b=SQg4nOJcRbgncvOYF4g2+gBVZzNxtyfNb5qbn4g8EHLsLqH0Twaboz0pZn6savDv8bNLtx 3hiT+GeH8EnnU1lZWqwFYL8zeFGsSgtvKOnuX6dYZ2RvCRAWO92X613Yxc0ru1HUXo7bpY RHvCjQnkzTdKWzo00wfvEoAl//pmZdY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FfKAqiPc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3sKueZwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3sKueZwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738451889; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+LHGz0Xil5zeJ4evV+ex8j8EcdekSjxZZZh0JGQMV40=; b=B8lKTncyD6XPqpTayZ+jJkMrR8rPPZypcnHC8L2sqGbl4pvnVlvr396uSPKVG7QuZ5efn8 LGCI6J6UxOqUwyF4V9mM8Km1KzwGHsoBpN3gEaiTayxCEjFC7ofgu9REc/WyKvFywdexjz CxDA3u6iCaPklcuUtFLS4dpFJH1cTgw= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2166f9f52fbso99676015ad.2 for ; Sat, 01 Feb 2025 15:18:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738451888; x=1739056688; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+LHGz0Xil5zeJ4evV+ex8j8EcdekSjxZZZh0JGQMV40=; b=FfKAqiPcCMCM2hI4YsFSV9cAyLzKAYL9DkzrPk5nFSDcYsuTN/Teu8lZlEoYc9d96W b4LbyshEyHXYn6agVTZs8EeVQf2vpdViQqkMuBZIgoPiaD6B80C8ZyL1Bwcp12RZvzuJ b7kx+2FG6RokxaUagVyLx+mvSklob6kljnKByzpmUUGkz7lF7I5pbjBzQm8+7uA9zORu bb/XIvb9eQ54pAtIKRd/NvdBwIYo4nYaotQl8Zi/biYxJVaK1fxOGo2PIcFVE7h1d6p5 mHT9udKtZE8gumT4mznbDnR+D16iiZxhxBSyg3wC2qZrDy+LY6n2NKYMBnCwI1Fk7koQ zqnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738451888; x=1739056688; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+LHGz0Xil5zeJ4evV+ex8j8EcdekSjxZZZh0JGQMV40=; b=a2dTwYTn3Uf79IiXYK/GEpI0CuJHN31hNToAnXPXfen3vHNUt1R3QL9A/Fz35JVWk9 +RS6EaAg2Cl05nNl4YVSDJT1Gwj5Vj3betW//lVCBwtjKh7diB9cBztkSFqbJ3qWu6TP BM3xOLudr85TVl63mtudNT4ebEb1HJOxbJsnHu+Wx33vgPfSyiz4MuL0qeehb6GjI49V U6PqByO3ZhmCAiKUbyFIVyZUrKB3Xn9uBxdnqtg8vRW/AgZPfb2DudkVHfmi8k1O2dg4 rf2TF5RnCwgPJ9sUBTaQEArbfrOp8qIGeR3gaVYCr49YiMGRk2OUd448jUtICgrujQOn E3zw== X-Forwarded-Encrypted: i=1; AJvYcCUeNanfD1d7n3ld3rKnVh0xesp2OV4c21hTeCQeYOIV1809e2d8CXGBbBPG4+eFUe/70A8yO/x6NQ==@kvack.org X-Gm-Message-State: AOJu0YzXI6meECvK581sjqtw4B3LU/yjoo9lst/rvjpLyA72r8WvjpVX wQeSdHpKb/6jDow4cNj9jnIR9HO57mj/4+6UyYpSrWPqL9iFxzQOVStKTTnqctWqF60M1J9X8w0 oGw== X-Google-Smtp-Source: AGHT+IE2vxYeAR9419oUgZ2ihycmI1jsvSuPTWn2lYlIeN6Pbw/ZzCEAbsoRXT2NV98TB1fIQxgW3UhypMQ= X-Received: from pjbpa13.prod.google.com ([2002:a17:90b:264d:b0:2ef:8055:93d9]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:c941:b0:215:5ea2:654b with SMTP id d9443c01a7336-21dd7c49718mr257314855ad.1.1738451888074; Sat, 01 Feb 2025 15:18:08 -0800 (PST) Date: Sat, 1 Feb 2025 15:18:01 -0800 In-Reply-To: <20250201231803.2661189-1-surenb@google.com> Mime-Version: 1.0 References: <20250201231803.2661189-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog Message-ID: <20250201231803.2661189-2-surenb@google.com> Subject: [PATCH v2 2/3] alloc_tag: uninline code gated by mem_alloc_profiling_key in slab allocator From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, vbabka@suse.cz, rostedt@goodmis.org, peterz@infradead.org, yuzhao@google.com, minchan@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, 00107082@163.com, quic_zhenhuah@quicinc.com, surenb@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3F09180009 X-Stat-Signature: 9t9ah4j5jd6t1wnphdfxngtghjzyu67x X-Rspam-User: X-HE-Tag: 1738451889-449091 X-HE-Meta: U2FsdGVkX184ApUNld7+UoXY64PBNkEJiHCQt6MXzdvike+dZNM1aoDROi2LsVMKLF8mldG4GoqFAMjR7lcj71BkoNrP8GVNPp4JX1tQyAE2dXu1yYtTpAbIXc66N6jc48j/A+GmwqHgHcHqOo/Pi5MhwGibxFTQpADHG8mf7s+3soow4OOCeS6CabY7lm38R8+YVqrReAn6EBtoZoz//gyShIdbX6UYPA4+0ueXzUvHF42C6PHFIggSr5a5U2jTgjRRZlTJuF+nubWo9/2i6QtvSZfWoNxLHi/8SZ3kWUMU0ftbwFWA/R3TiNnSvCabYOSFhoX3XCg1sR515q92daPtPRue11GkcP1ntId7LPSmoKVTVqG6syVkcmAB+dr+bMcjUUwhAf/qlGoGQmtXyXQRnbatHCeuMKZ2d8Bk8hvhSRzD0HWG9b7pRcpzd1VNdedsqM8K9KYMJ4kuMHDntPqxZ8xgyKVG1B0DK+hRKB5yGMoRLP+CDvXSUzIQ4o+oevFSpabaKzvn1stE/cafiFWrgLJiygKqDcyJIX/MGVE7EDWiYTvHuax3xTXFahG1inR4JyeZBT1Sct/jUAIChkxPhUJ4TIcQiRHXcDd1yDJNdMe5X6BH19lXDno3y3bq/W9uEWr+Tn6dw/V11XDoHj8HDTNnzb9gv8CA+ta4jo+4RZ6SQTLx+EsS0R1ue5rGxrTfm0NFHmJOIY9zFI/WpXXvO8XKkYbS3+n4k2DIuceR6caDyXjZzcORIoKBthcq2HibV70qwqpEWB6Hc77PVSauGwIVm7BVeA8ndRJYfmKF6AAnarZmZARy4LtIlFs1Q2/iTRW0IgFoPSXF5SnHSQAHNB7Z7cbsHBdABedxJLDuphSm222oTEss9bn8nktI+11nWHT0yx/Bro58SwRv4aWk0NLApupeP+KnJH0GAb/KoGW+V1nqzHcsxtJLjUFWn1MUFhATy/nl3hNvOfo X6m2Yf8p FKtDJT35/hgmhfpK82EmaMdCswf4i8yp826zZeqiv4RiiCJG6/gTog5Mrp9Bm0rZUW1iSxFZcvlxDdZX3WVqVmIGB7tail9Ztj/d8/UqF01lsqepVqZRAcOtZ52UF4t4m/kBIT2CXCLbDtU9CI3MjMvTa7Z5LFOrDDB+EhBHVBfAdfNauATZuhY3xMdxQox3IB/gGEdL/DcI62qRBNfvP8EQ0d+DQFnzfqqvjRRbF5MCi5agAI8/BJDHVS5mGMSuG3kEfBgEu7evL8fQPBhh8emYO8ALjLB+3mQ70Dl+Vi1nBDuhZoyvyQH0wKQJvghZ1xfH99kXsOTPGEMz/HRu9aIxQRrEb/WxOpRkYlU8n4jKAttxtM852xAq1qoYcKcFIMl/uX17VDqR35x1SiJxnC2y5p4UdSvWcU9K6Dx3lI9Yp43ni4wi94jGe6K7eFpGQ9ZM6NT8vYbQCp1/c31YZHh2zG0Cfh3QX1dmyEmV0zloj3y/go9GyEn3ySFWOgrs6BmnAcN2MQQH0SiEAMGqkQ7/QmXNsE3pka2qtUkfvf1y9MoKvLMF2Yjb7lG1lW3r0lbDyxSRJl6xgzFMyPI/P7FUdWApw+6VdlbTMsBKeIzloHM8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000047, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a sizable code section is protected by a disabled static key, that code gets into the instruction cache even though it's not executed and consumes the cache, increasing cache misses. This can be remedied by moving such code into a separate uninlined function. On a Pixel6 phone, slab allocation profiling overhead measured with CONFIG_MEM_ALLOC_PROFILING=y and profiling disabled is: baseline modified Big core 3.31% 0.17% Medium core 3.79% 0.57% Little core 6.68% 1.28% This improvement comes at the expense of the configuration when profiling gets enabled, since there is now an additional function call. The overhead from this additional call on Pixel6 is: Big core 0.66% Middle core 1.23% Little core 2.42% However this is negligible when compared with the overall overhead of the memory allocation profiling when it is enabled. On x86 this patch does not make noticeable difference because the overhead with mem_alloc_profiling_key disabled is much lower (under 1%) to start with, so any improvement is less visible and hard to distinguish from the noise. The overhead from additional call when profiling is enabled is also within noise levels. Signed-off-by: Suren Baghdasaryan --- Changes since v1 [1]: - Removed inline_if_mem_alloc_prof and uninlined the gated code unconditionally, per Steven Rostedt and Vlastimil Babka - Updated changelog to include overhead when profiling is enabled [1] https://lore.kernel.org/all/20250126070206.381302-2-surenb@google.com/ mm/slub.c | 51 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 19 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 1f50129dcfb3..184fd2b14758 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2000,7 +2000,8 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, return 0; } -static inline void free_slab_obj_exts(struct slab *slab) +/* Should be called only if mem_alloc_profiling_enabled() */ +static noinline void free_slab_obj_exts(struct slab *slab) { struct slabobj_ext *obj_exts; @@ -2077,33 +2078,37 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) return slab_obj_exts(slab) + obj_to_index(s, slab, p); } -static inline void -alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) +/* Should be called only if mem_alloc_profiling_enabled() */ +static noinline void +__alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) { - if (need_slab_obj_ext()) { - struct slabobj_ext *obj_exts; + struct slabobj_ext *obj_exts; - obj_exts = prepare_slab_obj_exts_hook(s, flags, object); - /* - * Currently obj_exts is used only for allocation profiling. - * If other users appear then mem_alloc_profiling_enabled() - * check should be added before alloc_tag_add(). - */ - if (likely(obj_exts)) - alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); - } + obj_exts = prepare_slab_obj_exts_hook(s, flags, object); + /* + * Currently obj_exts is used only for allocation profiling. + * If other users appear then mem_alloc_profiling_enabled() + * check should be added before alloc_tag_add(). + */ + if (likely(obj_exts)) + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); } static inline void -alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, - int objects) +alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) +{ + if (need_slab_obj_ext()) + __alloc_tagging_slab_alloc_hook(s, object, flags); +} + +/* Should be called only if mem_alloc_profiling_enabled() */ +static noinline void +__alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, + int objects) { struct slabobj_ext *obj_exts; int i; - if (!mem_alloc_profiling_enabled()) - return; - /* slab->obj_exts might not be NULL if it was created for MEMCG accounting. */ if (s->flags & (SLAB_NO_OBJ_EXT | SLAB_NOLEAKTRACE)) return; @@ -2119,6 +2124,14 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, } } +static inline void +alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, + int objects) +{ + if (mem_alloc_profiling_enabled()) + __alloc_tagging_slab_free_hook(s, slab, p, objects); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void -- 2.48.1.362.g079036d154-goog