From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78B48C4167B for ; Tue, 5 Dec 2023 08:19:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01DF16B0075; Tue, 5 Dec 2023 03:19:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F0FB56B0078; Tue, 5 Dec 2023 03:19:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB0786B007B; Tue, 5 Dec 2023 03:19:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C73476B0075 for ; Tue, 5 Dec 2023 03:19:37 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9DBBC14020A for ; Tue, 5 Dec 2023 08:19:37 +0000 (UTC) X-FDA: 81532065594.28.F3FBEE1 Received: from out-189.mta0.migadu.com (out-189.mta0.migadu.com [91.218.175.189]) by imf24.hostedemail.com (Postfix) with ESMTP id B48E718001A for ; Tue, 5 Dec 2023 08:19:35 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=AtV3fzAe; spf=pass (imf24.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701764375; a=rsa-sha256; cv=none; b=syTAf6X0XXCEZYnhvr56cqhaB16wtskOQ+MuIeVfN40XXUI6QiV3nkR90BTJ7hr7Sm5tZU zM9Zx/+yrImZj9gPKDAf71Bi0NISw8u+RGZyM6S/Cls6cYzRG0dv/QFG1GMMr+oKYOcyNK 1GmNn1J2L4vQqIHHsBnhupCUtJX6fNo= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=AtV3fzAe; spf=pass (imf24.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701764375; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1msEm8akAYRx9kjUOA91nsxPgIb6zvoJEQIqcVegQCI=; b=jC72Bl6lNz/7hf8ZTKgZJGx2Iev6+mk0lti/+hap5O8j/x3s2ccqLtBRGs7f8lAoTFTWGi +oFP/2iGcEc+EYlHhSeZz7M5QucCRzl/rLtBcgTaB8WzxXCGn+dGw39WVvSHDs29q0Uy3t OYA8w1rBZ3hF2dsY5I49hS4R6cFT754= Message-ID: <30f88452-740b-441f-bb4f-a2d946e35cf5@linux.dev> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1701764373; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1msEm8akAYRx9kjUOA91nsxPgIb6zvoJEQIqcVegQCI=; b=AtV3fzAeFmZ4JdXWxE03Prk3MZ1wZv43S0/xlV9BSbUTbS4gpzfPNZcJ8hnkvLbnvyQ/uu jdzhUwZUufJ0BcrKBBvLcMlMyveGsVUDjDk/FESYXgTpwridpsKWff7xPk+5s1zThdAMky 6MUHbCZDjgykCWbP9LXWTvdIX0CSgLM= Date: Tue, 5 Dec 2023 16:19:27 +0800 MIME-Version: 1.0 Subject: Re: [PATCH 2/4] mm/slub: introduce __kmem_cache_free_bulk() without free hooks Content-Language: en-US To: Vlastimil Babka , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alexander Potapenko , Marco Elver , Dmitry Vyukov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com References: <20231204-slub-cleanup-hooks-v1-0-88b65f7cd9d5@suse.cz> <20231204-slub-cleanup-hooks-v1-2-88b65f7cd9d5@suse.cz> X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou In-Reply-To: <20231204-slub-cleanup-hooks-v1-2-88b65f7cd9d5@suse.cz> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: B48E718001A X-Stat-Signature: te4ai8gug3rq9jek55jjsxsi9mutqemz X-Rspam-User: X-HE-Tag: 1701764375-88559 X-HE-Meta: U2FsdGVkX19Bx5AaFQXsY+ZvwxmSa61HGAjMps9tnnaC3084naDRYtKRTnuZZ7nqQZ1ECNRUvVFcDZXZk9j7Dp7gd5c7DX/HLFWZGu4nl7Xvww+nG6yPTd8UpCN55W6LWwTrQKM9q83r1AX7SLB2//I4NZJf4u/v/JacNI9chCZ2MEoS6ywT8OVp5YoQLliMBtuJ1V4FQCofzoFL/JtCH5ZvInPdi9Gw9yXn4K7W0q7kIDg4khIkwVKcHEffDjvsCB0V1MrxA4RFop8v2DdsmbxymZQ4/3ApQl6inM5Eig5MsyToZ77A9DMVjsJ8WTX13ZX8NMfJc5ojBUNhZT2lwCIV+4mmqpMMWnQTjDaR/hT9miPeJWRQ2FElo/k5aQTa2B2YxZT6K/+YfX3XyzWDdA29p0xHdRfRm1uY4I1/M3Y1hYFIM7ZXramH1zs35100FsNjLw90Yj+bqKFzaKq348YRsfQD+6huS2RutvyXBZVGnsErcvoJZMnOjACh1dFYjbob7lLPM47EHRLUk6lQgTEl6i4dTa889i3UqBYKoR9IhtfF28XGRtMLQ/C4F2croz8Jigi+AoDCh70xEY2NHi57WE9X7Dno7tfL7WC00N+iZV40CmnmZ1helb9zTS/LvxpDLOJ8lf0nWrv4MRoH/n4htCVwAmgzVGLa+BtMxkPXWZW3TeRmmzm2cHN2zaSfmk3AGxbDTT2nAnMgsrTg45ftfU3b0Yu33iM2JgRy1rryg6m3Qi6hTaXCicLAXLQeTHkQ/3LpaiycTTeTHQ0IFBJiY8DOCQIqHzf9m0BMMwbyVjPs5yQVlsDBXybThdq392vXHmjyz5qQ8KCN5C9sGCkFd0GFIQBP13Iiwstyt68YPPA23fMORE4qfJrpBRbmKHLKo9IfbVOcFcK4FcVip3wqs2p3iWPkm9pDVwEEhn/Z9gyL3fPMJmrSEnWEEk6AfM5dYSVWawWDl5/MMH/ PzCZwEIR S+HQ7tDAUAp5V28X0CvS+EJ6YJ41YSD70bAI/138iqkItTqyKH81BDXcIb1hThIC0w9d+C9ixQ62MlotsyTPAwDWo9nnjS0eNHKfKyDTFxWbieUl8TEdbxTPIMSz+J9ksotEouolcO1xdnkeHLpi6Ljo4Tl2IiKNSNbkShvoh0RCtYDT91Kn8qeByA0QWa7+hDe9mCpsEGvscUCNf8qG+hznRdw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2023/12/5 03:34, Vlastimil Babka wrote: > Currently, when __kmem_cache_alloc_bulk() fails, it frees back the > objects that were allocated before the failure, using > kmem_cache_free_bulk(). Because kmem_cache_free_bulk() calls the free > hooks (KASAN etc.) and those expect objects that were processed by the > post alloc hooks, slab_post_alloc_hook() is called before > kmem_cache_free_bulk(). > > This is wasteful, although not a big concern in practice for the rare > error path. But in order to efficiently handle percpu array batch refill > and free in the near future, we will also need a variant of > kmem_cache_free_bulk() that avoids the free hooks. So introduce it now > and use it for the failure path. > > As a consequence, __kmem_cache_alloc_bulk() no longer needs the objcg > parameter, remove it. The objects may have been charged before, but it seems __kmem_cache_alloc_bulk() forget to uncharge them? I can't find "uncharge" in do_slab_free(), or maybe the bulk interface won't be used on chargeable slab? Thanks. > > Signed-off-by: Vlastimil Babka > --- > mm/slub.c | 33 ++++++++++++++++++++++++++------- > 1 file changed, 26 insertions(+), 7 deletions(-) > > diff --git a/mm/slub.c b/mm/slub.c > index d7b0ca6012e0..0742564c4538 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -4478,6 +4478,27 @@ int build_detached_freelist(struct kmem_cache *s, size_t size, > return same; > } > > +/* > + * Internal bulk free of objects that were not initialised by the post alloc > + * hooks and thus should not be processed by the free hooks > + */ > +static void __kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p) > +{ > + if (!size) > + return; > + > + do { > + struct detached_freelist df; > + > + size = build_detached_freelist(s, size, p, &df); > + if (!df.slab) > + continue; > + > + do_slab_free(df.s, df.slab, df.freelist, df.tail, df.cnt, > + _RET_IP_); > + } while (likely(size)); > +} > + > /* Note that interrupts must be enabled when calling this function. */ > void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p) > { > @@ -4499,7 +4520,7 @@ EXPORT_SYMBOL(kmem_cache_free_bulk); > > #ifndef CONFIG_SLUB_TINY > static inline int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, > - size_t size, void **p, struct obj_cgroup *objcg) > + size_t size, void **p) > { > struct kmem_cache_cpu *c; > unsigned long irqflags; > @@ -4563,14 +4584,13 @@ static inline int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, > > error: > slub_put_cpu_ptr(s->cpu_slab); > - slab_post_alloc_hook(s, objcg, flags, i, p, false, s->object_size); > - kmem_cache_free_bulk(s, i, p); > + __kmem_cache_free_bulk(s, i, p); > return 0; > > } > #else /* CONFIG_SLUB_TINY */ > static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, > - size_t size, void **p, struct obj_cgroup *objcg) > + size_t size, void **p) > { > int i; > > @@ -4593,8 +4613,7 @@ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, > return i; > > error: > - slab_post_alloc_hook(s, objcg, flags, i, p, false, s->object_size); > - kmem_cache_free_bulk(s, i, p); > + __kmem_cache_free_bulk(s, i, p); > return 0; > } > #endif /* CONFIG_SLUB_TINY */ > @@ -4614,7 +4633,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, > if (unlikely(!s)) > return 0; > > - i = __kmem_cache_alloc_bulk(s, flags, size, p, objcg); > + i = __kmem_cache_alloc_bulk(s, flags, size, p); > > /* > * memcg and kmem_cache debug support and memory initialization. >