From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D62EDC4167B for ; Tue, 5 Dec 2023 04:38:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E5236B0085; Mon, 4 Dec 2023 23:38:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 394736B0092; Mon, 4 Dec 2023 23:38:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 25DF36B0095; Mon, 4 Dec 2023 23:38:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 17F716B0085 for ; Mon, 4 Dec 2023 23:38:43 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id DE30116055A for ; Tue, 5 Dec 2023 04:38:42 +0000 (UTC) X-FDA: 81531508884.18.13AEC8D Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf07.hostedemail.com (Postfix) with ESMTP id 0457940007 for ; Tue, 5 Dec 2023 04:38:40 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Vd6gSkvD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701751121; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z8za1LL9JYBxMQqZEFu+Ip+HiV0+LDHs+kobZXAkLSM=; b=HnZPmhjfME+VG2p+oEzTOm51wvEfAaB9iaTLqn7FkB9TWiIYMO2WoSfveuPbYMmXyMY0R4 iZ3cgA9Wr/O48KkUu9wT0Dmm1HnfiYryUT0ww8/UddIufHb3yEZzVoVIyqs9FIPvbj6Gh+ jz4OaoLNZqKhirT/bAzB8VdUUcwFh30= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Vd6gSkvD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701751121; a=rsa-sha256; cv=none; b=EPFWAO/nNVn0xXVuXrfHB6kaMPrFsCucF+JxaLTtq4x//3CGpJ1sd8Py5ZBCEVd5ZIuZ8a 1Iqil/gj93bwxYbnowuGSo/0ynwx9/TKFFLlPoRRJ6pull0iuoh2stFkJabM7O1/Wkgzzu Q4n6Rs1WVn33WQjYGS64ffLe4ED2sY0= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6ce33234fd7so1370828b3a.0 for ; Mon, 04 Dec 2023 20:38:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701751120; x=1702355920; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=z8za1LL9JYBxMQqZEFu+Ip+HiV0+LDHs+kobZXAkLSM=; b=Vd6gSkvDOrJkIYTYYat9T1xZ+TicGqnvTef43luhqvX6lCJZaVgAlp7Veb8fC9BdSk hAlEvRmxy0Z1gtYVI7U5B+NCOGLYZ9E+PUMUyfzLp9/2UkfhBVQ3pWWvnSq3r0YjiqMz oBDERd3P1+Yu6ST5Enj2LATXq5dQH5jGsm5gewpAjfSfBG4NIvOD6lHnYeyj9jPiFwSS j5lGBu58ykkl+LDUvwV8Ros6K9ll9sqcSZfTnoBwnbizByfl4XVA2ZvQxK7CktsgS1vO BquHGKOPQfyDKa15vAjDfHkO3W88Ztxq8bxhH/D9mIoYI4Ya6wt9XOnXddMVoTZ9Smez q23A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701751120; x=1702355920; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=z8za1LL9JYBxMQqZEFu+Ip+HiV0+LDHs+kobZXAkLSM=; b=SrW2oN0dciwNr5/+ZLtDqO3pOwk02OvOxDumNlVxzR/JSktEE+y2MAD/QDTUD7/xW9 bQwEAlmCLEafj5uDAzW+DcBMNaCsHvpxczfDA1Cr3IA0p21IZVvfUCce+ZOgTmfdFVlq SeGvVIvmBPKmfcliaKVrmcIjV86A0rlVjDW245dybOQbRTkcFtCW2Hn/BrxXqCY2Z/9C 8kIog66oKDOm2liT96FuyxruU1N//hw8yeHXCLNV0/nYSXpW0zjxEWvdhO6uPlzvOdF0 Gh5ezWkBxuPw/CV94KUan9qPBTp0DYp6By6LyMy3E/tpIMwGJBcel4nnU3Vf7iWQhjIl rI9A== X-Gm-Message-State: AOJu0Yw5ZW9Fwix37WCySzFv5ZJ1LhQ1XOfWHdWMCq4zgQ4LXwqP3+OQ UecVbhwpLBJpwjxTGDTvMDk= X-Google-Smtp-Source: AGHT+IHSG+1Tgf/q/ZSVPAa40DZqAi1IiJXzl6rcBNCixCdC/TwLHWLbumoyWO7hnbLLdrrVcadqwg== X-Received: by 2002:a05:6a20:1586:b0:18f:97c:5b84 with SMTP id h6-20020a056a20158600b0018f097c5b84mr2423300pzj.82.1701751119663; Mon, 04 Dec 2023 20:38:39 -0800 (PST) Received: from localhost.localdomain ([1.245.180.67]) by smtp.gmail.com with ESMTPSA id c13-20020a170903234d00b001cfcf3dd317sm9212738plh.61.2023.12.04.20.38.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 20:38:38 -0800 (PST) Date: Tue, 5 Dec 2023 13:38:21 +0900 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Vlastimil Babka Cc: David Rientjes , Christoph Lameter , Pekka Enberg , Joonsoo Kim , Andrew Morton , Roman Gushchin , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Marco Elver , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Kees Cook , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, linux-hardening@vger.kernel.org Subject: Re: [PATCH v2 16/21] mm/slab: move kfree() from slab_common.c to slub.c Message-ID: References: <20231120-slab-remove-slab-v2-0-9c9c70177183@suse.cz> <20231120-slab-remove-slab-v2-16-9c9c70177183@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20231120-slab-remove-slab-v2-16-9c9c70177183@suse.cz> X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 0457940007 X-Stat-Signature: 4azxnskit8ez7nkoemb8b573ed1kadut X-HE-Tag: 1701751120-119653 X-HE-Meta: U2FsdGVkX19QiPQJNwQK6X//CZVuza/yXQrOZX1683Bqv8STt7tl4oLcC2FR+N8YXOTog7L5BDqAMJFyeXAD+J59xs9kCzEtHtCDzphbhrMxcQRMcJrCwyaDhTR3waI9Ze6qpyMkjvTwSO+ttZSq3o/J8mRmiYEKKT9lE8XAAb6lWlDfcccRZlUEU3RFnJVUd422TC94cygIdvdMjlyyLOTlLMVFXavsj3XNoSE6RixaPVrHHuMBnJFHhnjYNBJnsn6R7+oxOnQwWm8oqWYJzS78gUlXeO7o+z87m1jyJscYo3pSZ9ax2lKskiRipS/SeAZe3dMzTjIs8zZNyPa2NnIAvzJvv7x425VfquqHYXufzy4lWEgBd52Qs5M1qcNnVt/l3B+XTpR4URePGuSJjPlfQM3G+k7qiT332THHnd0KjdPp8MauHSUYvOvmPDUddGSWxwnP0TGVTR3iW2go6ouIADP5GpKnCo9nO8J5mpQz+zwI9s+6kGM3HkzdhHxwqg0rynujH5nETlIPppLSqwHcn/bwWU5uWFpovX5krcNiglPM1U1/KP/aGaK7vd85j5JFRCXDoRCewSkSMrtBcvImfWZmanWnVZi0bsqSfR0IJdfKy+DUwaJhELjxU4MlZvRqE4wy6sMDEn+ftjdRMpMIt6L3oJhHooQ5m5Np1L5P2uKwVcrje757tFAIFzmBFffuzKkfVsF8xxK+WhrAzTIqS84YzvrnGbblqDYv7IzVrplzJTyqO9mIUHk3MQ5D2n0jzCkOOy4omiBwR9utzt/Ge31ISHR6RrTGFQREQ+2jT+ek/EkRR8qZHv3lqv6y21GzM2X5kT6oTDThjHF9zctbeHYvdqUWByC520qc3D+hDQXuRW/7ZefCa2ncYtn9BjXPecRCy1Z38yKtamwCoCOlEbQlbqI0YHBUUfzwY/hNAcBagbpbIKeXHt7DW52XTzCqBVA5MoJ6vMBNNQ7 P1cdGa+e q+dATPkjmlcfId4LUbjvBAzQbowqH179xPYpNEuzWfor0QDY0uH4+uGYGGP+jXlQdUvrtyL070Ao+JMvsSQcqbpD0kLaN0afnbrpkFOJr27BSf9XUA7UGcQDa2hxNXk/l5Jqpf44Jbi9Vyw86mT9u0Kk5YuZOYPVe35LnIK/zbK89SXc2QlqaEo3ugTZZFnnj1V+NULXQ/ERsSGn1zbX8+4R+Ji4tZexqQNWgrsH1KOsG50rwRJ3t6uH+znCDrLGSpYompFbv8rSxmRobGp7f3Z65NPG3qp5yi7VKE6W41c8UpAdSfHrBjFbDdhufaD2wE5bY6dpfl7DLSnBINeWdmPA+SL4OpuIaZG13I8oH2ruB7uQIMU0i7B/O9AhTf577pRN1SRUNDgAFsWByzs6Tkt/f+d/7QsIlm+rnd/czIMJyg4Iaftg1TLUlS3U7iN/RkC+TjxTIYgbx/2nl91A0uSiNXT1HF5m3UMGU0F2X5i0sA6uS+xNcjHB9j3LZ/OTk9+pLtyOGUQWkF91e9IC6ulxENQWTAXzrPxMXYYUJ4e2cmq4aadZBt91ijolkVyUlaBuHpKCEDpUbQQMHlttMbA0B1GPXB4Ei0x1GurZGU3xc8zsyHe8p/SW0gw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Nov 20, 2023 at 07:34:27PM +0100, Vlastimil Babka wrote: > This should result in better code. Currently kfree() makes a function > call between compilation units to __kmem_cache_free() which does its own > virt_to_slab(), throwing away the struct slab pointer we already had in > kfree(). Now it can be reused. Additionally kfree() can now inline the > whole SLUB freeing fastpath. > > Also move over free_large_kmalloc() as the only callsites are now in > slub.c, and make it static. > > Reviewed-by: Kees Cook > Signed-off-by: Vlastimil Babka > --- > mm/slab.h | 4 ---- > mm/slab_common.c | 45 --------------------------------------------- > mm/slub.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++----- > 3 files changed, 46 insertions(+), 54 deletions(-) > > diff --git a/mm/slab.h b/mm/slab.h > index 5ae6a978e9c2..35a55c4a407d 100644 > --- a/mm/slab.h > +++ b/mm/slab.h > @@ -395,8 +395,6 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags, unsigned long caller); > void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, > int node, size_t orig_size, > unsigned long caller); > -void __kmem_cache_free(struct kmem_cache *s, void *x, unsigned long caller); > - > gfp_t kmalloc_fix_flags(gfp_t flags); > > /* Functions provided by the slab allocators */ > @@ -559,8 +557,6 @@ static inline int memcg_alloc_slab_cgroups(struct slab *slab, > } > #endif /* CONFIG_MEMCG_KMEM */ > > -void free_large_kmalloc(struct folio *folio, void *object); > - > size_t __ksize(const void *objp); > > static inline size_t slab_ksize(const struct kmem_cache *s) > diff --git a/mm/slab_common.c b/mm/slab_common.c > index bbc2e3f061f1..f4f275613d2a 100644 > --- a/mm/slab_common.c > +++ b/mm/slab_common.c > @@ -963,22 +963,6 @@ void __init create_kmalloc_caches(slab_flags_t flags) > slab_state = UP; > } > > -void free_large_kmalloc(struct folio *folio, void *object) > -{ > - unsigned int order = folio_order(folio); > - > - if (WARN_ON_ONCE(order == 0)) > - pr_warn_once("object pointer: 0x%p\n", object); > - > - kmemleak_free(object); > - kasan_kfree_large(object); > - kmsan_kfree_large(object); > - > - mod_lruvec_page_state(folio_page(folio, 0), NR_SLAB_UNRECLAIMABLE_B, > - -(PAGE_SIZE << order)); > - __free_pages(folio_page(folio, 0), order); > -} > - > static void *__kmalloc_large_node(size_t size, gfp_t flags, int node); > static __always_inline > void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller) > @@ -1023,35 +1007,6 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags, > } > EXPORT_SYMBOL(__kmalloc_node_track_caller); > > -/** > - * kfree - free previously allocated memory > - * @object: pointer returned by kmalloc() or kmem_cache_alloc() > - * > - * If @object is NULL, no operation is performed. > - */ > -void kfree(const void *object) > -{ > - struct folio *folio; > - struct slab *slab; > - struct kmem_cache *s; > - > - trace_kfree(_RET_IP_, object); > - > - if (unlikely(ZERO_OR_NULL_PTR(object))) > - return; > - > - folio = virt_to_folio(object); > - if (unlikely(!folio_test_slab(folio))) { > - free_large_kmalloc(folio, (void *)object); > - return; > - } > - > - slab = folio_slab(folio); > - s = slab->slab_cache; > - __kmem_cache_free(s, (void *)object, _RET_IP_); > -} > -EXPORT_SYMBOL(kfree); > - > /** > * __ksize -- Report full size of underlying allocation > * @object: pointer to the object > diff --git a/mm/slub.c b/mm/slub.c > index cc801f8258fe..2baa9e94d9df 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -4197,11 +4197,6 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) > return cachep; > } > > -void __kmem_cache_free(struct kmem_cache *s, void *x, unsigned long caller) > -{ > - slab_free(s, virt_to_slab(x), x, NULL, &x, 1, caller); > -} > - > /** > * kmem_cache_free - Deallocate an object > * @s: The cache the allocation was from. > @@ -4220,6 +4215,52 @@ void kmem_cache_free(struct kmem_cache *s, void *x) > } > EXPORT_SYMBOL(kmem_cache_free); > > +static void free_large_kmalloc(struct folio *folio, void *object) > +{ > + unsigned int order = folio_order(folio); > + > + if (WARN_ON_ONCE(order == 0)) > + pr_warn_once("object pointer: 0x%p\n", object); > + > + kmemleak_free(object); > + kasan_kfree_large(object); > + kmsan_kfree_large(object); > + > + mod_lruvec_page_state(folio_page(folio, 0), NR_SLAB_UNRECLAIMABLE_B, > + -(PAGE_SIZE << order)); > + __free_pages(folio_page(folio, 0), order); > +} > + > +/** > + * kfree - free previously allocated memory > + * @object: pointer returned by kmalloc() or kmem_cache_alloc() > + * > + * If @object is NULL, no operation is performed. > + */ > +void kfree(const void *object) > +{ > + struct folio *folio; > + struct slab *slab; > + struct kmem_cache *s; > + void *x = (void *)object; > + > + trace_kfree(_RET_IP_, object); > + > + if (unlikely(ZERO_OR_NULL_PTR(object))) > + return; > + > + folio = virt_to_folio(object); > + if (unlikely(!folio_test_slab(folio))) { > + free_large_kmalloc(folio, (void *)object); > + return; > + } > + > + slab = folio_slab(folio); > + s = slab->slab_cache; > + slab_free(s, slab, x, NULL, &x, 1, _RET_IP_); > +} > +EXPORT_SYMBOL(kfree); > + > struct detached_freelist { > struct slab *slab; > void *tail; Looks good to me, Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> nit: mm/kfence/report.c checks if a function name starts with "__kmem_cache_free" which is removed by this patch. > > -- > 2.42.1 > >