From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF14DC2BD09 for ; Wed, 3 Jul 2024 08:12:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D7716B0098; Wed, 3 Jul 2024 04:12:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 35EA26B0099; Wed, 3 Jul 2024 04:12:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D8966B009A; Wed, 3 Jul 2024 04:12:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id ED9506B0098 for ; Wed, 3 Jul 2024 04:12:23 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6EBEC80910 for ; Wed, 3 Jul 2024 08:12:23 +0000 (UTC) X-FDA: 82297724166.21.1C9A02D Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by imf13.hostedemail.com (Postfix) with ESMTP id A1E882001E for ; Wed, 3 Jul 2024 08:12:21 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=peX9aEYf; spf=pass (imf13.hostedemail.com: domain of aliceryhl@google.com designates 209.85.221.52 as permitted sender) smtp.mailfrom=aliceryhl@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719994323; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EYZQ6hvzbyuQ9dkJY/vs/6RIOJzsf5Qm2425goUaxvE=; b=KhmQ21/RqrAjRD4iRhCsWfwN6eoMF6FzEMlJhXf1cHEczLx3LlPqK7z3V81nWyFdnrHaiO hEA4E2nuW4gK0vOtsWYISrP6gooWly9Jh+ZMq0k4I9cNt6hRD+Azw0pZ3R7kpbKicFATuw +VwFvOolI05Yf54chlAdHEIkCOknoB0= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=peX9aEYf; spf=pass (imf13.hostedemail.com: domain of aliceryhl@google.com designates 209.85.221.52 as permitted sender) smtp.mailfrom=aliceryhl@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719994323; a=rsa-sha256; cv=none; b=pWYqZvjwYyIlfFqhGetUMbAzUmRSADMv/9ZL6E5ZCQk7f9WQ3a2NNbHQvm+6AE09BC67RS bpphPqLcH47BJEcpB3krjwmM0Ac5b3FAd7gF9pEZF+XG3bIr8gsUVXVOANR7dGCvhg6V4m 2sZKThE1Nimzm9iSoeVrFBw3mUgANh4= Received: by mail-wr1-f52.google.com with SMTP id ffacd0b85a97d-361785bfa71so3501627f8f.2 for ; Wed, 03 Jul 2024 01:12:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1719994340; x=1720599140; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=EYZQ6hvzbyuQ9dkJY/vs/6RIOJzsf5Qm2425goUaxvE=; b=peX9aEYfVIOCCYe3k52gcl24Oo+VAzB1qL3FHw+olg+M/CaRVcP1b5rJe70l9sVoFS qqpRvXysXHszLDU8Fl82eeAQZWXWgLHnjQmyKNFuzKwAVOtQVr97LWayZPebj8qj2XfG BzVgMtJ/o1lZQfbzzSNuN8rL+BkQ9STawkJWXoFMO4GQ5GFi0j1j3jcyBeZoMfvXZpNE 4zOhc3bPUCiUF/uCoIN73x5kbv7ZVOqTsKpI1d4NPG/IPHBVeSruVpTjyQsv7LZayBMv eysb02onO4WQzVDMqZIr9nPAaHsLMzztWZUQXJYSWO1kpp8k31v/RCmibgRL7j+3eOuY Wa0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719994340; x=1720599140; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EYZQ6hvzbyuQ9dkJY/vs/6RIOJzsf5Qm2425goUaxvE=; b=ZxzlsS3FNP8+iOsiEurWOY6b4DzAv3YbLNpR44doG6BQpjtex8F5+zntDNWT/qZCMQ jrw83tx+2J9Otyx9WMYgQbgMHBL3VsXQo/ZnruAbSiW2tT+3uaxbIwopMtvvRyh/+ltw OlnA8us2bvBYRmpbucFidaDA7W3ef1WHAsglEMTGPpIYAutD9ajG4n0aXOfpHWM5zIcr ZWSrAqmAJoyZbx6Je/xaZYIgFNdcmOsb1zdJ/GDNDdFkv3eXZ4VMpoHGGWOVEOk9U7Mu IOMoYAz0s1dZ+nYEfD4f7gbSNQhjfMUJLTEvrF+dCoTkOacYXyF/xheBm4ep4Hg0m5uy tjyA== X-Gm-Message-State: AOJu0Yx9GMAyYFvtD7US792xpXKJyxvQzC8kbgNs9EtYtc/XcV4QgVM5 CICRftN/20Q26Xgqk68IOfsB94nfwnzP5DPh3idwlN1VF/2jlZ5zPvIxQ8YAjH5KEBiQ9Y35mFT GHq/yBQikP1RjJtFx8USqWMKZmrXQXNRWWfhQ X-Google-Smtp-Source: AGHT+IEICDH5BfKPspvMCdmvkvvtMFo63HbRNY8nyE4K1FX/Ysrh7tFANjRp15gsI9lcNyfLPvRWNd79Gsa/BRnUtVg= X-Received: by 2002:adf:f510:0:b0:35f:2366:12c5 with SMTP id ffacd0b85a97d-367756aade1mr7017890f8f.23.1719994339769; Wed, 03 Jul 2024 01:12:19 -0700 (PDT) MIME-Version: 1.0 References: <20240703072520.45837-2-vbabka@suse.cz> In-Reply-To: <20240703072520.45837-2-vbabka@suse.cz> From: Alice Ryhl Date: Wed, 3 Jul 2024 10:12:07 +0200 Message-ID: Subject: Re: [PATCH v2] slab, rust: extend kmalloc() alignment guarantees to remove Rust padding To: Vlastimil Babka Cc: linux-mm@kvack.org, David Rientjes , Christoph Lameter , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , Kees Cook , Boqun Feng , rust-for-linux@vger.kernel.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: gc8inn5pqouf84k4qpjrr867er3zdd7c X-Rspam-User: X-Rspamd-Queue-Id: A1E882001E X-Rspamd-Server: rspam02 X-HE-Tag: 1719994341-956224 X-HE-Meta: U2FsdGVkX1+nG47tiFlTN0GLNPR7Eo4MMG7ReWETQy1D3nKRxzGD1yFu6ruS/rIMgBYI2Hx2oT0OCou2L/w7VM29Rc07/AaNgK7bNH293QBBRqddLYIUAWby2Ly3Bq1v7sdYpMufDxX0esslj0exDdwZHa6YLMPqWgDFc0jjtWGiqWfnb6IF1Xb/boZ+1j8A2KqoLKYcViNVLOqbyI5JmqFX/eCILIVF9mWDSHGjjgxNIA9TX61RiP/Tu571RGjnsVhDCvetyW8mVEWylfVufey/XO+Q3AqgSDZFFhSmxlHAQc541r3oMHVoZRdpyQzjduY80YESScBntvvQIt453hxmrjClKM+ak7BZ46NbAQIRyHv8tQnYcSXnyc5nsycN53vI70R9npHahfz+SwBTBk//flX5N/0NmoMd47yafExsxQKwpkfY+5hi5P+J8Jv2JdsbS3IzwdQsDDkZfxe06FUCsGLiehBHjhxUvkfoa605A27a0C7sOQAb2hbUgR9Acwr8Mykbfui5LtwRzoIb/u1tLSreuKdUF4NHpNXDvE4zTJhQLywYn3YPzCPGobOQOCcfiWzC6n63PW0D/hKsS2AdGUFE2cC45Hs7groD11GxnQoFhMpNIahKwKqfYAeeXvMswc2Uk4PEww85081sPVXdl16Yii6HxfyWdqcTpt4ec9j1vwDeUX91pb91v84KNYFHVeUAUFpz6tIPZo6W1GOih3sYPOTCxl//0l9aAueZGhZ1yojV37TL4MUxihKtepugqvxnIUpzYM/ESJZ/YCmyyKZx5L176uL3GgDqPTS1pSwPWjsCn3so2nwbIxHFKGA81MKToUNuyv92YO3HbR28nYMDgbS2pmhG+uq86VqURHTRE0FKdl6BYJjWZZ4FgnEFVpPdYy73xKwDBE0jW7afOptEkl73WViJlnafDOzpDvXNckOhoc9UHH93oduv15LEXeq4hAkubYDtLaq Xi76PszT zaL7q/DQ/gKrnfEw/wsmGZc2fbnLaZ3d1r8r73KlL5VORndRflRUNn/I2xeYGjPFnM9eaMMp6MMCxgRlFLHEkOD49JvCtbBtkBNML31rO+eyZcQ0ZxqXBVr8HACfEGu34T04/UDdtYYPwJ0sVoRBo0ATGCkwNaxJQCUkrdKGnDUTKUHYm3aVhNwgA1NfSJUi83pGraD3+A89wb0vkiWd+5G4nULPlU+s2erORtBV70m9aXr9x0rPzLNLhgPTkK1rmYODM8DAlNQsGx0mO19XW7BaX7cwieizweD4qPnl+U2+DMtwuOS2U85RD51wPfEFOKquQAMigoVY/ARXm7fghzumc86s/jQLtmYtL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jul 3, 2024 at 9:25=E2=80=AFAM Vlastimil Babka wro= te: > > Slab allocators have been guaranteeing natural alignment for > power-of-two sizes since commit 59bb47985c1d ("mm, sl[aou]b: guarantee > natural alignment for kmalloc(power-of-two)"), while any other sizes are > guaranteed to be aligned only to ARCH_KMALLOC_MINALIGN bytes (although > in practice are aligned more than that in non-debug scenarios). > > Rust's allocator API specifies size and alignment per allocation, which > have to satisfy the following rules, per Alice Ryhl [1]: > > 1. The alignment is a power of two. > 2. The size is non-zero. > 3. When you round up the size to the next multiple of the alignment, > then it must not overflow the signed type isize / ssize_t. > > In order to map this to kmalloc()'s guarantees, some requested > allocation sizes have to be padded to the next power-of-two size [2]. > For example, an allocation of size 96 and alignment of 32 will be padded > to an allocation of size 128, because the existing kmalloc-96 bucket > doesn't guarantee alignent above ARCH_KMALLOC_MINALIGN. Without slab > debugging active, the layout of the kmalloc-96 slabs however naturally > align the objects to 32 bytes, so extending the size to 128 bytes is > wasteful. > > To improve the situation we can extend the kmalloc() alignment > guarantees in a way that > > 1) doesn't change the current slab layout (and thus does not increase > internal fragmentation) when slab debugging is not active > 2) reduces waste in the Rust allocator use case > 3) is a superset of the current guarantee for power-of-two sizes. > > The extended guarantee is that alignment is at least the largest > power-of-two divisor of the requested size. For power-of-two sizes the > largest divisor is the size itself, but let's keep this case documented > separately for clarity. > > For current kmalloc size buckets, it means kmalloc-96 will guarantee > alignment of 32 bytes and kmalloc-196 will guarantee 64 bytes. > > This covers the rules 1 and 2 above of Rust's API as long as the size is > a multiple of the alignment. The Rust layer should now only need to > round up the size to the next multiple if it isn't, while enforcing the > rule 3. > > Implementation-wise, this changes the alignment calculation in > create_boot_cache(). While at it also do the calulation only for caches > with the SLAB_KMALLOC flag, because the function is also used to create > the initial kmem_cache and kmem_cache_node caches, where no alignment > guarantee is necessary. > > In the Rust allocator's krealloc_aligned(), remove the code that padded > sizes to the next power of two (suggested by Alice Ryhl) as it's no > longer necessary with the new guarantees. > > Reported-by: Alice Ryhl > Reported-by: Boqun Feng > Link: https://lore.kernel.org/all/CAH5fLggjrbdUuT-H-5vbQfMazjRDpp2%2Bk3%3= DYhPyS17ezEqxwcw@mail.gmail.com/ [1] > Link: https://lore.kernel.org/all/CAH5fLghsZRemYUwVvhk77o6y1foqnCeDzW4WZv= 6ScEWna2+_jw@mail.gmail.com/ [2] > Signed-off-by: Vlastimil Babka > Reviewed-by: Boqun Feng > Acked-by: Roman Gushchin Reviewed-by: Alice Ryhl > --- > v2: - add Rust side change as suggested by Alice, also thanks Boqun for f= ixups > - clarify that the alignment already existed (unless debugging) but was > not guaranteed, so there's no extra fragmentation in slab > - add r-b, a-b thanks tO Boqun and Roman > > If it's fine with Rust folks, I can put this in the slab.git tree. > > Documentation/core-api/memory-allocation.rst | 6 ++++-- > include/linux/slab.h | 3 ++- > mm/slab_common.c | 9 +++++---- > rust/kernel/alloc/allocator.rs | 19 ++++++------------- > 4 files changed, 17 insertions(+), 20 deletions(-) > > diff --git a/Documentation/core-api/memory-allocation.rst b/Documentation= /core-api/memory-allocation.rst > index 1c58d883b273..8b84eb4bdae7 100644 > --- a/Documentation/core-api/memory-allocation.rst > +++ b/Documentation/core-api/memory-allocation.rst > @@ -144,8 +144,10 @@ configuration, but it is a good practice to use `kma= lloc` for objects > smaller than page size. > > The address of a chunk allocated with `kmalloc` is aligned to at least > -ARCH_KMALLOC_MINALIGN bytes. For sizes which are a power of two, the > -alignment is also guaranteed to be at least the respective size. > +ARCH_KMALLOC_MINALIGN bytes. For sizes which are a power of two, the > +alignment is also guaranteed to be at least the respective size. For oth= er > +sizes, the alignment is guaranteed to be at least the largest power-of-t= wo > +divisor of the size. > > Chunks allocated with kmalloc() can be resized with krealloc(). Similarl= y > to kmalloc_array(): a helper for resizing arrays is provided in the form= of > diff --git a/include/linux/slab.h b/include/linux/slab.h > index ed6bee5ec2b6..640cea6e6323 100644 > --- a/include/linux/slab.h > +++ b/include/linux/slab.h > @@ -604,7 +604,8 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t = flags, int node) > * > * The allocated object address is aligned to at least ARCH_KMALLOC_MINA= LIGN > * bytes. For @size of power of two bytes, the alignment is also guarant= eed > - * to be at least to the size. > + * to be at least to the size. For other sizes, the alignment is guarant= eed to > + * be at least the largest power-of-two divisor of @size. > * > * The @flags argument may be one of the GFP flags defined at > * include/linux/gfp_types.h and described at > diff --git a/mm/slab_common.c b/mm/slab_common.c > index 1560a1546bb1..7272ef7bc55f 100644 > --- a/mm/slab_common.c > +++ b/mm/slab_common.c > @@ -617,11 +617,12 @@ void __init create_boot_cache(struct kmem_cache *s,= const char *name, > s->size =3D s->object_size =3D size; > > /* > - * For power of two sizes, guarantee natural alignment for kmallo= c > - * caches, regardless of SL*B debugging options. > + * kmalloc caches guarantee alignment of at least the largest > + * power-of-two divisor of the size. For power-of-two sizes, > + * it is the size itself. > */ > - if (is_power_of_2(size)) > - align =3D max(align, size); > + if (flags & SLAB_KMALLOC) > + align =3D max(align, 1U << (ffs(size) - 1)); > s->align =3D calculate_alignment(flags, align, size); > > #ifdef CONFIG_HARDENED_USERCOPY > diff --git a/rust/kernel/alloc/allocator.rs b/rust/kernel/alloc/allocator= .rs > index 229642960cd1..e6ea601f38c6 100644 > --- a/rust/kernel/alloc/allocator.rs > +++ b/rust/kernel/alloc/allocator.rs > @@ -18,23 +18,16 @@ pub(crate) unsafe fn krealloc_aligned(ptr: *mut u8, n= ew_layout: Layout, flags: F > // Customized layouts from `Layout::from_size_align()` can have size= < align, so pad first. > let layout =3D new_layout.pad_to_align(); > > - let mut size =3D layout.size(); > - > - if layout.align() > bindings::ARCH_SLAB_MINALIGN { > - // The alignment requirement exceeds the slab guarantee, thus tr= y to enlarge the size > - // to use the "power-of-two" size/alignment guarantee (see comme= nts in `kmalloc()` for > - // more information). > - // > - // Note that `layout.size()` (after padding) is guaranteed to be= a multiple of > - // `layout.align()`, so `next_power_of_two` gives enough alignme= nt guarantee. > - size =3D size.next_power_of_two(); > - } > + // Note that `layout.size()` (after padding) is guaranteed to be a m= ultiple of `layout.align()` > + // which together with the slab guarantees means the `krealloc` will= return a properly aligned > + // object (see comments in `kmalloc()` for more information). > + let size =3D layout.size(); The size is a multiple of the alignment due to the `pad_to_align` call above, which rounds up the size to ensure that this is the case. Alice