From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87B38ECAAA1 for ; Thu, 27 Oct 2022 12:12:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCDAC8E0002; Thu, 27 Oct 2022 08:12:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7E588E0001; Thu, 27 Oct 2022 08:12:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B46308E0002; Thu, 27 Oct 2022 08:12:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A60448E0001 for ; Thu, 27 Oct 2022 08:12:00 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 285D2C023B for ; Thu, 27 Oct 2022 12:12:00 +0000 (UTC) X-FDA: 80066616000.06.15A0C0E Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf05.hostedemail.com (Postfix) with ESMTP id 9FCE6100021 for ; Thu, 27 Oct 2022 12:11:58 +0000 (UTC) Received: by mail-pj1-f54.google.com with SMTP id c15-20020a17090a1d0f00b0021365864446so1260029pjd.4 for ; Thu, 27 Oct 2022 05:11:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=F6Ci2e1GctkVuBTZcZj2eOapSo4Y9VSQb5VfOkL4ECQ=; b=I1BrW8L4ftDKz7+Ha5iaHWSl0PQkJLe9nd7F4aZFl6znTYgH3oUmR8iLi7IUccm7uR H3lVXMkh7opU7Ylg8CssHwAvXjVsgkxssw0XM+B2P3vucup510f9Z8Rh3ChdIbSjJyjQ nyyfpHvv6/Vx4koh31Dg9bKDaC3o6tLmUO4lMcqoKhJYKweBXPDeB5+Mxc7ul5DGQ+qT vYEtva9OUTJQOyxzkK7WwxvsgqhXY15uMNhcI2XgCk9GgkIHi8m1id0YRwB8tzHGWEES 8kh5tO6+oUHZIJ2RcRZTzGF315rgUGJXQSg7TNPmIAu2/p72aFq5Rr/VobYc1mMSgigS pnPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=F6Ci2e1GctkVuBTZcZj2eOapSo4Y9VSQb5VfOkL4ECQ=; b=KAkub6AXcXGvBOyLRiBL+XECXgJ4GXxAOO29NlbeXkJ4AzoSncvyAS8V0wqTcTDU4r lqVB3+u4DHu6uY24OwEzUHKKenctFUpK5bYj7TQjRDmf5IM4Wipkrt3qpTabBOcvZt5Z W9umHoqEtS8kGdaEu3NzW+VD0mPlfrBE/EPlMPNQLLDG3YTzxxPDOGl5M9PVWq24d3IK Y/vzb8BKxSqWwo2TAarf7yUL+3GkF5a8cq5ABGH+7TFHl8QcOibaAv4wAmsQ+SCZqSmE fN4o12uFEj2kIw4EM3S16hCSR/dWSKouhl7fHXOSfbc3YUIhd6yXi2anpHAlRZ128zzH 6NNg== X-Gm-Message-State: ACrzQf1NB9rPKhGlXTK7nXswdUjRRdbWYDve2QCNItD9953VDusCs0c3 DJ2Un0FvVq/NTQ0xrK4W49M= X-Google-Smtp-Source: AMsMyM4nffcpt1vkSfU7BlBDrYyw1AhchiK11XevHkvIemdpwdIBMJbqBGjSR5GoiwCY0WaDKWNOrg== X-Received: by 2002:a17:903:2596:b0:186:a395:c4bd with SMTP id jb22-20020a170903259600b00186a395c4bdmr22258036plb.60.1666872717473; Thu, 27 Oct 2022 05:11:57 -0700 (PDT) Received: from hyeyoo ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id q4-20020aa79824000000b0056bb7d90f0fsm1063961pfl.182.2022.10.27.05.11.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Oct 2022 05:11:56 -0700 (PDT) Date: Thu, 27 Oct 2022 21:11:49 +0900 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Catalin Marinas Cc: Linus Torvalds , Arnd Bergmann , Will Deacon , Marc Zyngier , Greg Kroah-Hartman , Andrew Morton , Herbert Xu , Ard Biesheuvel , Christoph Hellwig , Isaac Manjarres , Saravana Kannan , linux-mm@kvack.org, Vlastimil Babka , David Rientjes , Christoph Lameter , Roman Gushchin , linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v2 1/2] mm: slab: Introduce __GFP_PACKED for smaller kmalloc() alignments Message-ID: References: <20221025205247.3264568-1-catalin.marinas@arm.com> <20221025205247.3264568-2-catalin.marinas@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20221025205247.3264568-2-catalin.marinas@arm.com> ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666872718; a=rsa-sha256; cv=none; b=rKKiyZP0TAqS12jWG1tIxzIUawLETiu9i7xAAplckLMNtrB+2sxG4UcNZ/JAEuZa26JOlq ZzNKnG/9mcM1TVcfUZVwcdWstc4NVy77eSoRTNNWVzbmCqejX33usqBrkEvWHVmkFGdEGK nLixiikLpk8HIJOw1UBkJ4R2K8HJOMc= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=I1BrW8L4; spf=pass (imf05.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666872718; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F6Ci2e1GctkVuBTZcZj2eOapSo4Y9VSQb5VfOkL4ECQ=; b=xQR4kIH4K2yNUOhn6TevkBpd2bVidNJtSq6+czzb/YI6k5b2s2Powxkhxz8Ieh2YLUts60 jJjJMKwTJ0hBtGf4NAqPOXTjb2TpdcEr2ecfsliNplI9su6ByLRq3+MBvt+v2zROyJPwZn ItY66Q8qq/v7nn1BZIi8DgISpIwnshQ= X-Rspamd-Queue-Id: 9FCE6100021 X-Rspam-User: Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=I1BrW8L4; spf=pass (imf05.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam04 X-Stat-Signature: 5thx7eahzm9g3ehmhtqqtdzjwwn3w4w6 X-HE-Tag: 1666872718-954821 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Oct 25, 2022 at 09:52:46PM +0100, Catalin Marinas wrote: > By default kmalloc() returns objects aligned to ARCH_KMALLOC_MINALIGN. > This can be somewhat large on architectures defining ARCH_DMA_MINALIGN > (e.g. 128 on arm64) and significant memory is wasted through small > kmalloc() allocations. > > Reduce the minimum alignment for kmalloc() to the default > KMALLOC_MIN_SIZE (8 for slub, 32 for slab) but align the > requested size to the bigger ARCH_KMALLOC_MINALIGN unless a newly added > __GFP_PACKED flag is passed. > > With this gfp flag, the alignment is > reduced to KMALLOC_PACKED_ALIGN, at least sizeof(unsigned long long). > > There's no slob support. > Thank you for pushing it forward! > Signed-off-by: Catalin Marinas > --- > include/linux/gfp_types.h | 10 ++++++++-- > include/linux/slab.h | 22 ++++++++++++++++++---- > mm/slab_common.c | 3 ++- > 3 files changed, 28 insertions(+), 7 deletions(-) > > diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h > index d88c46ca82e1..305cb8cb6f8b 100644 > --- a/include/linux/gfp_types.h > +++ b/include/linux/gfp_types.h > @@ -55,8 +55,9 @@ typedef unsigned int __bitwise gfp_t; > #define ___GFP_SKIP_KASAN_UNPOISON 0 > #define ___GFP_SKIP_KASAN_POISON 0 > #endif > +#define ___GFP_PACKED 0x8000000u > #ifdef CONFIG_LOCKDEP > -#define ___GFP_NOLOCKDEP 0x8000000u > +#define ___GFP_NOLOCKDEP 0x10000000u > #else > #define ___GFP_NOLOCKDEP 0 > #endif > @@ -243,6 +244,10 @@ typedef unsigned int __bitwise gfp_t; > * > * %__GFP_SKIP_KASAN_POISON makes KASAN skip poisoning on page deallocation. > * Typically, used for userspace pages. Only effective in HW_TAGS mode. > + * > + * %__GFP_PACKED returns a pointer aligned to the possibly smaller > + * KMALLOC_PACKED_ALIGN rather than ARCH_KMALLOC_MINALIGN. Useful for small > + * object allocation on architectures that define large ARCH_DMA_MINALIGN. > */ > #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) > #define __GFP_COMP ((__force gfp_t)___GFP_COMP) > @@ -251,12 +256,13 @@ typedef unsigned int __bitwise gfp_t; > #define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO) > #define __GFP_SKIP_KASAN_UNPOISON ((__force gfp_t)___GFP_SKIP_KASAN_UNPOISON) > #define __GFP_SKIP_KASAN_POISON ((__force gfp_t)___GFP_SKIP_KASAN_POISON) > +#define __GFP_PACKED ((__force gfp_t)___GFP_PACKED) > > /* Disable lockdep for GFP context tracking */ > #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) > > /* Room for N __GFP_FOO bits */ > -#define __GFP_BITS_SHIFT (27 + IS_ENABLED(CONFIG_LOCKDEP)) > +#define __GFP_BITS_SHIFT (28 + IS_ENABLED(CONFIG_LOCKDEP)) > #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) > > /** > diff --git a/include/linux/slab.h b/include/linux/slab.h > index 90877fcde70b..0f59585b5fbf 100644 > --- a/include/linux/slab.h > +++ b/include/linux/slab.h > @@ -223,8 +223,6 @@ void kmem_dump_obj(void *object); > */ > #if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8 > #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN > -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN > -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN) > #else > #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long) > #endif > @@ -310,6 +308,11 @@ static inline unsigned int arch_slab_minalign(void) > #define KMALLOC_MIN_SIZE (1 << KMALLOC_SHIFT_LOW) > #endif > > +/* > + * This alignment should be at least sizeof(unsigned long long). > + */ > +#define KMALLOC_PACKED_ALIGN (KMALLOC_MIN_SIZE) > + I think __assume_kmalloc_alignment should be changed as well, to avoid compiler making wrong decision. > /* > * This restriction comes from byte sized index implementation. > * Page size is normally 2^12 bytes and, in this case, if we want to use > @@ -382,6 +385,17 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) > return KMALLOC_CGROUP; > } > > +/* > + * Align the size to ARCH_KMALLOC_MINALIGN unless __GFP_PACKED is passed. > + */ > +static __always_inline size_t kmalloc_size_align(size_t size, gfp_t flags) > +{ > + if (ARCH_KMALLOC_MINALIGN > KMALLOC_PACKED_ALIGN && > + !(flags & __GFP_PACKED)) > + size = ALIGN(size, ARCH_KMALLOC_MINALIGN); > + return size; > +} > + > /* > * Figure out which kmalloc slab an allocation of a certain size > * belongs to. > @@ -568,7 +582,7 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) > if (size > KMALLOC_MAX_CACHE_SIZE) > return kmalloc_large(size, flags); > #ifndef CONFIG_SLOB > - index = kmalloc_index(size); > + index = kmalloc_index(kmalloc_size_align(size, flags)); > > if (!index) > return ZERO_SIZE_PTR; > @@ -590,7 +604,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla > if (size > KMALLOC_MAX_CACHE_SIZE) > return kmalloc_large_node(size, flags, node); > > - index = kmalloc_index(size); > + index = kmalloc_index(kmalloc_size_align(size, flags)); > > if (!index) > return ZERO_SIZE_PTR; > > diff --git a/mm/slab_common.c b/mm/slab_common.c > index 33b1886b06eb..0e4ea396cd4f 100644 > --- a/mm/slab_common.c > +++ b/mm/slab_common.c > @@ -627,7 +627,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, > unsigned int useroffset, unsigned int usersize) > { > int err; > - unsigned int align = ARCH_KMALLOC_MINALIGN; > + unsigned int align = KMALLOC_PACKED_ALIGN; > > s->name = name; > s->size = s->object_size = size; > @@ -720,6 +720,7 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) > { > unsigned int index; > > + size = kmalloc_size_align(size, flags); > > > if (size <= 192) { > if (!size) > return ZERO_SIZE_PTR; > -- Thanks, Hyeonggon