From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C02EC54798 for ; Tue, 27 Feb 2024 16:15:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07E63280029; Tue, 27 Feb 2024 11:15:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 02E5C6B0096; Tue, 27 Feb 2024 11:15:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1150280029; Tue, 27 Feb 2024 11:15:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D30166B0095 for ; Tue, 27 Feb 2024 11:15:36 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0EAB680D41 for ; Tue, 27 Feb 2024 16:15:36 +0000 (UTC) X-FDA: 81838084272.25.96D273C Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com [209.85.219.178]) by imf21.hostedemail.com (Postfix) with ESMTP id 31A1F1C001E for ; Tue, 27 Feb 2024 16:15:33 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XmW7JWyb; spf=pass (imf21.hostedemail.com: domain of surenb@google.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709050534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZscKwOxqDJIaAnxf3DOisSB4kKJVJaPdUBtyCsbS8zY=; b=ZCJXDLBXmcUW3FbLSTJCsE/4V6SmeI7x3i0g46tLNgmAjT0pcAiVLg1qEkpKo31DFMShmA 3TT1B4aXSrsaGPxekXWGiFQL+5ionngAhWi4tBZpRQisZJ9QpUU8ImhBj8wDdI/SqWvP+5 dHTWiYqIktekaHfPVxjGoNiy4i2WVQ0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709050534; a=rsa-sha256; cv=none; b=sYFcS1hPXF5TiJllNzRxTnmiFQvLmQtOmv96L1d14LGNbzRwg5s0FBThc0Fnx3iAp121be qkrRg8x7v3U296xi4FzunWT+FclcWenv38Sg51YVAxHNWnwRErhUXy9B40cFu01kAalGuc JFqE5JwIxFfLL6lQyA2I1C8IkvZHGPg= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XmW7JWyb; spf=pass (imf21.hostedemail.com: domain of surenb@google.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f178.google.com with SMTP id 3f1490d57ef6-dbed0710c74so4032390276.1 for ; Tue, 27 Feb 2024 08:15:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709050533; x=1709655333; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ZscKwOxqDJIaAnxf3DOisSB4kKJVJaPdUBtyCsbS8zY=; b=XmW7JWybgi4DsrUypPWbtBj+j8z5PDZ6NtGqYs/Ogtf2HAxQG1QMMPNHbFEmw8tOR4 CZneBF6CqRxVtq/Tjv1IAJaTL2wrdRLuKZj1cMbOGTOLXZJ0upcR1YDp4Wn9OC7qHMtV ddlmsShCbYKXDrXY2L6Vpg+UD5mytBdmTgwOqMxA4xJ6iTG3BH/dmFS54aQbYc1Z6GE1 7xiFuW1OXK/cGLH24L75qUZuAO2dfCFbwMRgRMeseRXj7uoqy/xy9lPLD5TT3pL0/0lc LC2V2rdW3XYxKN+r2c7qYmiPaoLm5BinF0VWT8AmmGl+va7IZjKaGUnThPgqwZo/AOVH v5Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709050533; x=1709655333; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZscKwOxqDJIaAnxf3DOisSB4kKJVJaPdUBtyCsbS8zY=; b=r4RYtrNqqRH+4j8OuVNylLiOXVsZg65IXk5xgC9v762sfWdg674laVv0IiWp+vIjum VFvGMPmC8c0e4G0OfuDtxOVx4QDg72RDd9ooB74+LUq5Pf/pYRsWjLC21SXynAd/FAGE kDDJrzpF7hexmbz+LmXMb2raXEQ2hc0v1CIMzFDMUE0PDnBYePB/FFVH2QHKkyhNH6+y v/VOmhKJlJjOrrXJCy9mXaSZXz5/9SGEMAyzk9kRQ2X5/TDfCZhS7+kYoNMCSUaSHGud yu8O5xRlyHhSibr+Gt6/K9xTVKmEru49nVqlPB+aLxT/8n6q3cukJGTZVTg6LXuuYXR1 UEIQ== X-Forwarded-Encrypted: i=1; AJvYcCWCMsOzbhv8yd8AfJLSkvxCEz+rErH8LkVXHIQDOsYo2lh5Z6WntjOX4I/asmuM5drqSlfLRLJhMUelO1OAKCSdB5E= X-Gm-Message-State: AOJu0Yy3YTWpkzoBbl4BY1uXV7U+ryxPS28skGJBypjdhQfrunFb4Neq DV36zop+RUvmoG64qC5RApBW10gURWGOky/7N67JncSGrTzriD7dfNyT1ARUfL6rD0JpLGWgBLm ohOn2+JszyAvaqI9rPZCbQ/JF8zwG0zh9gSaq X-Google-Smtp-Source: AGHT+IEw+VgUM8rjFF4tarET9LYSoU4rt/YwPutl48cPMakVPIe4FNGy4MH5hQJmtrlQlJwMQOaq0wT1r6aJOfixfFE= X-Received: by 2002:a25:aac5:0:b0:dcc:4b84:67cd with SMTP id t63-20020a25aac5000000b00dcc4b8467cdmr2309188ybi.9.1709050532850; Tue, 27 Feb 2024 08:15:32 -0800 (PST) MIME-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> <20240221194052.927623-23-surenb@google.com> <4a0e40e5-3542-4d47-bb2b-c0666f6a904d@suse.cz> In-Reply-To: <4a0e40e5-3542-4d47-bb2b-c0666f6a904d@suse.cz> From: Suren Baghdasaryan Date: Tue, 27 Feb 2024 08:15:21 -0800 Message-ID: Subject: Re: [PATCH v4 22/36] mm/slab: add allocation accounting into slab allocation and free paths To: Vlastimil Babka Cc: akpm@linux-foundation.org, kent.overstreet@linux.dev, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 31A1F1C001E X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: bkfned41h7a7xwp1uqddgupbpknpn55z X-HE-Tag: 1709050533-296751 X-HE-Meta: U2FsdGVkX1+5rswK3bNbjuWVXw3WAxnI3jhCAHzj7hX8wAerJ+sXDNT6gKxBBKbe698JZLdCyDYGI8VppjhmzGD60mpiEFrYhztXtFsG89t8FB48Gy3vLw+hy0gCIn1UFdg4pwjJscR/xRvKsMAeNA1dKQoTzlwleIRj7GwtxjMVnbV8Ys8t5OYoIy+/KbXhKXUS/F6GeomrdJipWfoeor222oESWjpP1MJx5e7THz9wAcqz/TIkUjQm77T+0hSCEEiVzl3AE5bhysksGEd+8TMSV6j5pqg8UqfaAOtfZwkf2/2rCfL8zecC3YQCBt3+ktPY4PNnuOgp1rWkafyOa4t6YpIuFXmJTyuv9hOCiflWdoQh/Yjzfy9VVTlVPXeb3P1w9yjJZo5XepkB2os8lVuIwy8r4hbwXUTLoWzgz8lC8gPOEdUVGxSzf2aPQEEk/PpK9oPhtjhbnegNEOAZKAYiSZSxOkrdmFLzSkhwWkVCj65WBtcj9nL2Ami/5+C+r1SC0P0qYVEmBC3v75QE2vorWlt0AOL+/Esub6ddwBpJOUnGvDTr5x53CFG+EOuqCF+TR0k3rVQq5HcLHxrUaLHPmQa7XOovRwODrC5B30sPWfw1akvFqfqHZDSBFJzY/ITmmKS/GbLG9QoMu+gq9tHzxttJ1Bfr91XOzH7SJ8fQiPy6z7CdgEpAA8cRPg1JYh/VRWXqw1MAHwqcxTRvmfEnJ4V8eAtE/VjC03lj2YahqZl85M8WelUqDb8WsS4JEsT81DKHgnTcJmtdCIvcftcwI1P8Jb//1u0GRMf5Dm8s17ktAgn7BW60z2BtSEFrvAfgFPi+4PtcthUBgUmn3p+JyLb+IVHQ4L1kv1ch10NKXd2jlb4Suc/zmclZGCW+wNmRoYm64xDI5dW0LPMzGyByaNwu+mKc6vdBNmcR7NYzk/wlaCR5fwaZ0LFrynypFFM69VNvKpM1CP0mpVb I9M+jqjX 9NWVH21PnonejYo/RybuLalkd3mG9USTN0kGwv/yJtN8ImOj/QWbZUfv4/ROvom7UShxRz0wKNTopqZnc4ddFasgu7g5GGzhL+vQzEcYZpv5v0DOxzRn+S64mQnqcBuIdeeUlozUljBYgsPvgL3Tw64XZsp52VbZM+kb2OCJWQbvG+GAQ8+YgpLJ0Xwp3gyJ/V8ntmKCDXehb3DGvxNHGePlTK4ca7w9FqaWQftVvcKsjat/iCWgiUyRLg8C2jZcHuZVJlrZTDexMwf9r2YLKRZgmy27/TSlXKwG1r+FPV7el+6qz7B0xGIJk8s4yaxI/3hO05jmbP1Ks7yOsW7LYysrzj9N+375FJ2xZlMR0YnrU/tFo4LgCEG7josX+HqQC3eniEiXEjb4VuXU4wIvCoWsz+mb36qPu6bBZwj59I22u6rI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Feb 27, 2024 at 5:07=E2=80=AFAM Vlastimil Babka wr= ote: > > > > On 2/21/24 20:40, Suren Baghdasaryan wrote: > > Account slab allocations using codetag reference embedded into slabobj_= ext. > > > > Signed-off-by: Suren Baghdasaryan > > Co-developed-by: Kent Overstreet > > Signed-off-by: Kent Overstreet > > Reviewed-by: Kees Cook > > --- > > mm/slab.h | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > mm/slub.c | 9 ++++++++ > > 2 files changed, 75 insertions(+) > > > > diff --git a/mm/slab.h b/mm/slab.h > > index 13b6ba2abd74..c4bd0d5348cb 100644 > > --- a/mm/slab.h > > +++ b/mm/slab.h > > @@ -567,6 +567,46 @@ static inline struct slabobj_ext *slab_obj_exts(st= ruct slab *slab) > > int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, > > gfp_t gfp, bool new_slab); > > > > +static inline bool need_slab_obj_ext(void) > > +{ > > +#ifdef CONFIG_MEM_ALLOC_PROFILING > > + if (mem_alloc_profiling_enabled()) > > + return true; > > +#endif > > + /* > > + * CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditi= onally > > + * inside memcg_slab_post_alloc_hook. No other users for now. > > + */ > > + return false; > > +} > > + > > +static inline struct slabobj_ext * > > +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) > > +{ > > + struct slab *slab; > > + > > + if (!p) > > + return NULL; > > + > > + if (!need_slab_obj_ext()) > > + return NULL; > > + > > + if (s->flags & SLAB_NO_OBJ_EXT) > > + return NULL; > > + > > + if (flags & __GFP_NO_OBJ_EXT) > > + return NULL; > > + > > + slab =3D virt_to_slab(p); > > + if (!slab_obj_exts(slab) && > > + WARN(alloc_slab_obj_exts(slab, s, flags, false), > > + "%s, %s: Failed to create slab extension vector!\n", > > + __func__, s->name)) > > + return NULL; > > + > > + return slab_obj_exts(slab) + obj_to_index(s, slab, p); > > +} > > + > > #else /* CONFIG_SLAB_OBJ_EXT */ > > > > static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) > > @@ -589,6 +629,32 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, g= fp_t flags, void *p) > > > > #endif /* CONFIG_SLAB_OBJ_EXT */ > > > > +#ifdef CONFIG_MEM_ALLOC_PROFILING > > + > > +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, = struct slab *slab, > > + void **p, int objects) > > Only used from mm/slub.c so could move? Ack. > > > +{ > > + struct slabobj_ext *obj_exts; > > + int i; > > + > > + obj_exts =3D slab_obj_exts(slab); > > + if (!obj_exts) > > + return; > > + > > + for (i =3D 0; i < objects; i++) { > > + unsigned int off =3D obj_to_index(s, slab, p[i]); > > + > > + alloc_tag_sub(&obj_exts[off].ref, s->size); > > + } > > +} > > + > > +#else > > + > > +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, = struct slab *slab, > > + void **p, int objects) {} > > + > > +#endif /* CONFIG_MEM_ALLOC_PROFILING */ > > + > > #ifdef CONFIG_MEMCG_KMEM > > void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgd= at, > > enum node_stat_item idx, int nr); > > diff --git a/mm/slub.c b/mm/slub.c > > index 5dc7beda6c0d..a69b6b4c8df6 100644 > > --- a/mm/slub.c > > +++ b/mm/slub.c > > @@ -3826,6 +3826,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, s= truct obj_cgroup *objcg, > > unsigned int orig_size) > > { > > unsigned int zero_size =3D s->object_size; > > + struct slabobj_ext *obj_exts; > > bool kasan_init =3D init; > > size_t i; > > gfp_t init_flags =3D flags & gfp_allowed_mask; > > @@ -3868,6 +3869,12 @@ void slab_post_alloc_hook(struct kmem_cache *s, = struct obj_cgroup *objcg, > > kmemleak_alloc_recursive(p[i], s->object_size, 1, > > s->flags, init_flags); > > kmsan_slab_alloc(s, p[i], init_flags); > > + obj_exts =3D prepare_slab_obj_exts_hook(s, flags, p[i]); > > +#ifdef CONFIG_MEM_ALLOC_PROFILING > > + /* obj_exts can be allocated for other reasons */ > > + if (likely(obj_exts) && mem_alloc_profiling_enabled()) > > + alloc_tag_add(&obj_exts->ref, current->alloc_tag,= s->size); > > +#endif > > I think that like in the page allocator, this could be better guarded by > mem_alloc_profiling_enabled() as the outermost thing. Oops, missed it. Will fix. > > > } > > > > memcg_slab_post_alloc_hook(s, objcg, flags, size, p); > > @@ -4346,6 +4353,7 @@ void slab_free(struct kmem_cache *s, struct slab = *slab, void *object, > > unsigned long addr) > > { > > memcg_slab_free_hook(s, slab, &object, 1); > > + alloc_tagging_slab_free_hook(s, slab, &object, 1); > > Same here, the static key is not even inside of this? Ack. > > > > > if (likely(slab_free_hook(s, object, slab_want_init_on_free(s)))) > > do_slab_free(s, slab, object, object, 1, addr); > > @@ -4356,6 +4364,7 @@ void slab_free_bulk(struct kmem_cache *s, struct = slab *slab, void *head, > > void *tail, void **p, int cnt, unsigned long addr) > > { > > memcg_slab_free_hook(s, slab, p, cnt); > > + alloc_tagging_slab_free_hook(s, slab, p, cnt); > > Ditto. Ack. > > > /* > > * With KASAN enabled slab_free_freelist_hook modifies the freeli= st > > * to remove objects, whose reuse must be delayed. > > -- > To unsubscribe from this group and stop receiving emails from it, send an= email to kernel-team+unsubscribe@android.com. >