From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BF49C5478C for ; Tue, 27 Feb 2024 13:07:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C24B28000A; Tue, 27 Feb 2024 08:07:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 77245280007; Tue, 27 Feb 2024 08:07:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C47E28000A; Tue, 27 Feb 2024 08:07:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 480A3280007 for ; Tue, 27 Feb 2024 08:07:16 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1D1D340BF6 for ; Tue, 27 Feb 2024 13:07:16 +0000 (UTC) X-FDA: 81837609672.01.7106184 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf09.hostedemail.com (Postfix) with ESMTP id A307F140017 for ; Tue, 27 Feb 2024 13:07:12 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=lLjXBkwS; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=QuOdtqlw; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=TDeXG6qf; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=x7kVZZ2b; dmarc=none; spf=pass (imf09.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709039233; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cW70zYu78ce27du6nFlEM9h9LcRqy3qIdoJOEMgxHrU=; b=jmH3uRxXZEVOmWIffyA/1JwH1yggag+TV4L1pFk2Tf+ZJ0Z0CBov+7SgO5Edf11krFKMn+ 1e2qTMWwcIGADd/WD4ccNEVcxnvXFDrE0uWwZVN0y1X5BDn8urGIo171HwYIdtNCRNJCJH w+Lg26kXcyToU7p6buJqwY5EpkEyZS0= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=lLjXBkwS; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=QuOdtqlw; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=TDeXG6qf; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=x7kVZZ2b; dmarc=none; spf=pass (imf09.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709039233; a=rsa-sha256; cv=none; b=AgKx9PxwMBM1bdn1q5oXlI1bSmnZSB67dp1UgeDTamAhNX7lxLZKhIjaLzp1x4Pr0utU13 XHEyqpXW1oFhN6xzTEESHc2c/pjh+CeJSoiYZVASn6e4VldFWhcL5LIky2+rgGcNPQ/5jl p0byHh9PH0ACLoRUWrWe2KaOXC/57q8= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id A36851F7B9; Tue, 27 Feb 2024 13:07:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1709039230; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cW70zYu78ce27du6nFlEM9h9LcRqy3qIdoJOEMgxHrU=; b=lLjXBkwSq4zVWDpU6/ofgbzfBCnuGD7WdSXfD8ViQQRE1+HKJbO34+Gzh29oHqfhywajAJ xd6VpHYHeA+lMIDc2X0LUuilIJLgFgRUWpX0+2USkX8CkBSSuM5wQ8HcvRNtGEJDU8zQzg B1EiPMFl21up4fU4/nhvYVWOow9hpcQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1709039230; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cW70zYu78ce27du6nFlEM9h9LcRqy3qIdoJOEMgxHrU=; b=QuOdtqlwB2n5UpUvZAQXn1p6d3jd41mAMNMufuBBkSELyBp/2B9PV6bswWPZs/NGwzSwma MfH9Uc8gUFNHdnBQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1709039228; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cW70zYu78ce27du6nFlEM9h9LcRqy3qIdoJOEMgxHrU=; b=TDeXG6qfFssifgl/IwZPUo2hz8sRgPKz7Rc+MiJVTP4aR1BZWy7Q6jkFcRNhFXSoq9UfSu cN35aZnq5noMPiPIwcTJ6Wbc9eN5lKdjPWO9jqaZdLUwn/xvaDQoSSCKZCpxQKC8uBTm+U soVbxLp9C4/rdFn1gbl5nYmN0zVuniQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1709039228; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cW70zYu78ce27du6nFlEM9h9LcRqy3qIdoJOEMgxHrU=; b=x7kVZZ2bAkzoDmE//U9l56lfDmWmTUJkTS193DKe/lp6VGXut63+mJFIV0tJi7b7AxADyk Pb79/ZLGUtSXyIBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id F129813A58; Tue, 27 Feb 2024 13:07:06 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id XexAOnre3WWJKQAAD6G6ig (envelope-from ); Tue, 27 Feb 2024 13:07:06 +0000 Message-ID: <4a0e40e5-3542-4d47-bb2b-c0666f6a904d@suse.cz> Date: Tue, 27 Feb 2024 14:07:38 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 22/36] mm/slab: add allocation accounting into slab allocation and free paths Content-Language: en-US To: Suren Baghdasaryan , akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org References: <20240221194052.927623-1-surenb@google.com> <20240221194052.927623-23-surenb@google.com> From: Vlastimil Babka In-Reply-To: <20240221194052.927623-23-surenb@google.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: A307F140017 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 97z9cwduyme1mnyk6rykjkh4p5yf5k5o X-HE-Tag: 1709039232-392759 X-HE-Meta: U2FsdGVkX1/Msl1yomhEQs+/p8SwWUnBHLgZYYV3KC+D6LsdyDtHxPVIT2ayxHR4ZVXWhgldUzhNdnDv9MsKRF3fMrzdEuxz1eJW9x/gFZnvdvhwP+CpWzF/ZVKeJfKjYkmvje2GrS53xWISqgSuFVo8Qf5n834dSlgSX1cuqcW/PtPrkP904LRR07k7aCQL3AbLrl15zdk1OY2GSbIYYIUOjBNxBLqpnculnmWUUGvjFcL0cUgexX9wj6Ofc8znjw19jfDugjXXWTa8oLqnoeBZjjKspxd/zuiSxBR/onSevZm9tnqe+dyCgLcuxQDpnzTMCpHrbva70VsFQ8ENmH7R6/S/yE4UR00ByUPsDw0NdQH3VrP+/f7HwS7gDxZTPNQkVJeVeI8RoAZES1bq/0DfxOpPO97bAVVXuv4Qdt6L4dlF84cZEebEc+MAwq8Q82eTG8fiGGOXc3QXOX2IQuI+LnwHXe43sio9iq06GK9R+mrfhXnyjOgTSQhGN2xDvDGhbga5aSmt1L2hjZhbJnYh1YNEGxHczyVjn3quw7+E3YGFxzzKw6/PnceEe1UEbBZvVUVYFVVNS1SHeyuYpb56RUxBac0We/+gGmt8VAA7Xo2xYbiPgjNoOAh9ZeCmh+xRs3auHtm540KSf/OGq4/pb00/d5bcdzaApnblVu9QTOwWO/XkJcuNJEdbeSDqBaiaWa5pyodfBKsEALYzd9YOShP7UBb41woT9S04ytbFN1yeLG/hR6XneDNJibzEhq9XcE/qZxEenWK0Mzfpk7qUvtu2b6VH+EBR5vzJa/rb9gbNS8jU7M/OO9VS9Rr1Y14c8Y42iOt+DIDnJcePLmKMkGHBXno+Hptb+879I0uUZtGfuUGff2YPNiF7HpuUv/gvPzToBEQOxuCi3tqJRZXlvLfXrAqjlD0CORLKAqOHKb/Nn5uYJ0zeAJT7XC65CcGGlDfyFKd8PZwDUx8 732JyyAf ncQrS7aeP971pjUw8etQVWVRdhECkwTfrYjIS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2/21/24 20:40, Suren Baghdasaryan wrote: > Account slab allocations using codetag reference embedded into slabobj_ext. > > Signed-off-by: Suren Baghdasaryan > Co-developed-by: Kent Overstreet > Signed-off-by: Kent Overstreet > Reviewed-by: Kees Cook > --- > mm/slab.h | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ > mm/slub.c | 9 ++++++++ > 2 files changed, 75 insertions(+) > > diff --git a/mm/slab.h b/mm/slab.h > index 13b6ba2abd74..c4bd0d5348cb 100644 > --- a/mm/slab.h > +++ b/mm/slab.h > @@ -567,6 +567,46 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) > int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, > gfp_t gfp, bool new_slab); > > +static inline bool need_slab_obj_ext(void) > +{ > +#ifdef CONFIG_MEM_ALLOC_PROFILING > + if (mem_alloc_profiling_enabled()) > + return true; > +#endif > + /* > + * CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally > + * inside memcg_slab_post_alloc_hook. No other users for now. > + */ > + return false; > +} > + > +static inline struct slabobj_ext * > +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) > +{ > + struct slab *slab; > + > + if (!p) > + return NULL; > + > + if (!need_slab_obj_ext()) > + return NULL; > + > + if (s->flags & SLAB_NO_OBJ_EXT) > + return NULL; > + > + if (flags & __GFP_NO_OBJ_EXT) > + return NULL; > + > + slab = virt_to_slab(p); > + if (!slab_obj_exts(slab) && > + WARN(alloc_slab_obj_exts(slab, s, flags, false), > + "%s, %s: Failed to create slab extension vector!\n", > + __func__, s->name)) > + return NULL; > + > + return slab_obj_exts(slab) + obj_to_index(s, slab, p); > +} > + > #else /* CONFIG_SLAB_OBJ_EXT */ > > static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) > @@ -589,6 +629,32 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) > > #endif /* CONFIG_SLAB_OBJ_EXT */ > > +#ifdef CONFIG_MEM_ALLOC_PROFILING > + > +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, > + void **p, int objects) Only used from mm/slub.c so could move? > +{ > + struct slabobj_ext *obj_exts; > + int i; > + > + obj_exts = slab_obj_exts(slab); > + if (!obj_exts) > + return; > + > + for (i = 0; i < objects; i++) { > + unsigned int off = obj_to_index(s, slab, p[i]); > + > + alloc_tag_sub(&obj_exts[off].ref, s->size); > + } > +} > + > +#else > + > +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, > + void **p, int objects) {} > + > +#endif /* CONFIG_MEM_ALLOC_PROFILING */ > + > #ifdef CONFIG_MEMCG_KMEM > void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, > enum node_stat_item idx, int nr); > diff --git a/mm/slub.c b/mm/slub.c > index 5dc7beda6c0d..a69b6b4c8df6 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -3826,6 +3826,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, > unsigned int orig_size) > { > unsigned int zero_size = s->object_size; > + struct slabobj_ext *obj_exts; > bool kasan_init = init; > size_t i; > gfp_t init_flags = flags & gfp_allowed_mask; > @@ -3868,6 +3869,12 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, > kmemleak_alloc_recursive(p[i], s->object_size, 1, > s->flags, init_flags); > kmsan_slab_alloc(s, p[i], init_flags); > + obj_exts = prepare_slab_obj_exts_hook(s, flags, p[i]); > +#ifdef CONFIG_MEM_ALLOC_PROFILING > + /* obj_exts can be allocated for other reasons */ > + if (likely(obj_exts) && mem_alloc_profiling_enabled()) > + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); > +#endif I think that like in the page allocator, this could be better guarded by mem_alloc_profiling_enabled() as the outermost thing. > } > > memcg_slab_post_alloc_hook(s, objcg, flags, size, p); > @@ -4346,6 +4353,7 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, > unsigned long addr) > { > memcg_slab_free_hook(s, slab, &object, 1); > + alloc_tagging_slab_free_hook(s, slab, &object, 1); Same here, the static key is not even inside of this? > > if (likely(slab_free_hook(s, object, slab_want_init_on_free(s)))) > do_slab_free(s, slab, object, object, 1, addr); > @@ -4356,6 +4364,7 @@ void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, > void *tail, void **p, int cnt, unsigned long addr) > { > memcg_slab_free_hook(s, slab, p, cnt); > + alloc_tagging_slab_free_hook(s, slab, p, cnt); Ditto. > /* > * With KASAN enabled slab_free_freelist_hook modifies the freelist > * to remove objects, whose reuse must be delayed.