From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1842C83F0A for ; Wed, 9 Jul 2025 17:25:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 909AE6B00A3; Wed, 9 Jul 2025 13:25:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 893496B00A5; Wed, 9 Jul 2025 13:25:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A9266B00AC; Wed, 9 Jul 2025 13:25:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 66C366B00A3 for ; Wed, 9 Jul 2025 13:25:05 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2AB2280341 for ; Wed, 9 Jul 2025 17:25:05 +0000 (UTC) X-FDA: 83645401770.30.D022F8B Received: from mailrelay-egress16.pub.mailoutpod3-cph3.one.com (mailrelay-egress16.pub.mailoutpod3-cph3.one.com [46.30.212.3]) by imf18.hostedemail.com (Postfix) with ESMTP id 4960C1C000A for ; Wed, 9 Jul 2025 17:25:03 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=konsulko.se header.s=rsa1 header.b=QnFONWGg; dkim=pass header.d=konsulko.se header.s=ed1 header.b=sg0+3uas ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752081903; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J9nxjDaqlzWYUB0phI7+TkY/6yrU4R7qwJ3m2vmI6YU=; b=DnxW9DRylZm1ZCGpZgw4jOZv2RQV62iCI2N32AuKiKBE1zby87RY+DttSrG5tbKMJAiEkg O6+L8df3vA7lETODgshTLR50BbH4/UUBCe1qOYrmJE5uFXOAStqF4uqrQlqidBWRAVjAcC YPc9haDedi2gYVOHSquhNI9Y3BmncDs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752081903; a=rsa-sha256; cv=none; b=FuyI//3Pk1qXnpVy5AWRSBeA8wEK9t3pKf52Pvph8IroMNAWj2SbVbpbVPJ2bH9nuxe6DZ QgkncJSohI59RHe0JjnV1HsVDVT7NIHMU0cS51Er53Vv/i+BzVpJrqchPFqkqK4pEK0qir U0CswjME9Shoaad2SABo8+6R6Oojdmo= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=konsulko.se header.s=rsa1 header.b=QnFONWGg; dkim=pass header.d=konsulko.se header.s=ed1 header.b=sg0+3uas; dmarc=none; spf=none (imf18.hostedemail.com: domain of vitaly.wool@konsulko.se has no SPF policy when checking 46.30.212.3) smtp.mailfrom=vitaly.wool@konsulko.se DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1752081902; x=1752686702; d=konsulko.se; s=rsa1; h=content-transfer-encoding:mime-version:references:in-reply-to:message-id:date: subject:cc:to:from:from; bh=J9nxjDaqlzWYUB0phI7+TkY/6yrU4R7qwJ3m2vmI6YU=; b=QnFONWGgD7brrb3xqESrfA6dx6MLDmxc6pMiPJhLUgQyb2Nywu5lQPl7H2rOQFNrdUQT58f/V80ts e42O2BdMoM6omyo8f+zmwdC8loxv2ukXaqTGkbY/BXpjf9hUhETzOv+WOvXX+t2LYS+sAWNxHpZlNO DvHcJKicixwPh6/2jeEtAfG+wVdqT3L9X7eVa+9AJ8fkbD/FUZt06L8DeJqTWl2g7K4/X1gXnppw/s Srp84NQeka0q7WLXszB5xftZzTru6ZF2SdUc1M/bLEAxsgcmepkFxpmwKT7YCsFz0nw+bIBrnt5btk e78p5G/kHj/3RseAMNTa+YV91ErIIfA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; t=1752081902; x=1752686702; d=konsulko.se; s=ed1; h=content-transfer-encoding:mime-version:references:in-reply-to:message-id:date: subject:cc:to:from:from; bh=J9nxjDaqlzWYUB0phI7+TkY/6yrU4R7qwJ3m2vmI6YU=; b=sg0+3uasecWCBVg5FqY1owP+oURyn+kmAEi+gunS8CpLiXcYWvuZBDGhEMy9PjZxssTPgaoxPXKc7 TkpDDhmCw== X-HalOne-ID: a42e7db0-5ce9-11f0-9a3e-f3c0f7fef5ee Received: from slottsdator.home (host-90-238-19-233.mobileonline.telia.com [90.238.19.233]) by mailrelay4.pub.mailoutpod2-cph3.one.com (Halon) with ESMTPSA id a42e7db0-5ce9-11f0-9a3e-f3c0f7fef5ee; Wed, 09 Jul 2025 17:25:01 +0000 (UTC) From: Vitaly Wool To: linux-mm@kvack.org Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, Uladzislau Rezki , Danilo Krummrich , Alice Ryhl , Vlastimil Babka , rust-for-linux@vger.kernel.org, Lorenzo Stoakes , "Liam R . Howlett" , Kent Overstreet , linux-bcachefs@vger.kernel.org, bpf@vger.kernel.org, Herbert Xu , Jann Horn , Pedro Falcato , Vitaly Wool Subject: [PATCH v12 3/4] rust: add support for NUMA ids in allocations Date: Wed, 9 Jul 2025 19:24:58 +0200 Message-Id: <20250709172458.1032040-1-vitaly.wool@konsulko.se> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250709172345.1031907-1-vitaly.wool@konsulko.se> References: <20250709172345.1031907-1-vitaly.wool@konsulko.se> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4960C1C000A X-Stat-Signature: 9zi9pkxfb5t7wpqjxxnub1ergrppme8c X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1752081903-615398 X-HE-Meta: U2FsdGVkX1+xvqA2/lD3BJgvgpSt+PUuQLTxG4dbva9ceJ0Zx3My5j8iz9Sz2Fo0C2c7WjQy/O36Je9d+4o/0vUq4v6OfzSYaH3ZWEliGvpiJKF+mKzEpKA+c2XzQZ9mo6rK7wgnuuH15/6O9heTaXlyR1AUgTUIoIoZRl2ngYcvHoF1SFln6u3lDJUYfAvopdBhs2zXF2mbjPiTmyszLToyA4ab4inOiU0sJh7uwWZCIOEhTMetMk/l+FwuJCe4GOu6y8dqrGyzeLpnNBmMio2HFeh9gEnPOBtr224hpb3FlokWkSkYL/tcxmrZ3zMnYsrq53OHHI3jd/RBE7goJK05Ps+Qv34SkMCcB/6QDeKV1fYDXkZbq0muRH+qvbICLkUUXOANHRoaO3xe6dKJw68ECMnsNc+osah0wfB+xtuBc+kvVsbbkoFW5X6wNNSU3Gzdzd2PK12Bbg6Ke4aLKhUymQ4vkKsmaCWHhS3tY8I348d24JxAwChGuCc23IXNHqi+1lncgCIKp6wKgRjDUyp1bN2763Eru4eTyV4X/X1CqjCkDc9k9LRxf0+rj6R0dMIEDGqjyOInkHg11ynn+SINkic+gs1iipLLsa/oQspjUX6+Gu6i6+/TCejVRlgROzzru8hojIsPYwlwItN4rNEHqc6287FLftvGT3zdKseFz0z5AcCJeyDM5RQYUFQ1gmJOE1mix7xDWKy3f6RVlbcDh1tz0xPx2Os4DjoolxD4AR03+ITZ/Zb2N4oNnMF9hUfDDS03ZVEdFE7oGQhkMJ8qOR5HGe15AwXnZsSv8AF10bvCvp4GnZ1sYJHtid1Ez3angMoNW3a/F2PAbFmBXxb1g848KZR6QqEXpO+2+Vj8Sq3pbNtIvt1MTP1kOlcQ6IdFirANI+TfCfuXDOKo61fSgSdgR0scopnH7wBtpTZGKplV3giN7N4QfUi4WkZdQUa/cBq5s4g2Q6Wu+oj ztjCXN+Y kf+/Md8ac890HDxGKVz2WgMKd9HBmDnqblfLOofp2lRLq0XRlkrVf3MLX1zjT8K9CcAlx4dkfYSeqmfXE8MYsJJKqfnsiUQNx8e7s/Eq2aXehG0plZZ+LoxxsQ6kumFtENaKAVRh2aqU1ZQMYnTj+O+/VA2yvHJ9k33+ZRUDkQDsEVY1SfPqVIkreaA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a new type to support specifying NUMA identifiers in Rust allocators and extend the allocators to have NUMA id as a parameter. Thus, modify ReallocFunc to use the new extended realloc primitives from the C side of the kernel (i. e. k[v]realloc_node_align/vrealloc_node_align) and add the new function alloc_node to the Allocator trait while keeping the existing one (alloc) for backward compatibility. This will allow to specify node to use for allocation of e. g. {KV}Box, as well as for future NUMA aware users of the API. Signed-off-by: Vitaly Wool --- rust/helpers/slab.c | 8 +++--- rust/helpers/vmalloc.c | 4 +-- rust/kernel/alloc.rs | 52 ++++++++++++++++++++++++++++++---- rust/kernel/alloc/allocator.rs | 35 ++++++++++++++--------- rust/kernel/alloc/kbox.rs | 4 +-- rust/kernel/alloc/kvec.rs | 11 +++++-- 6 files changed, 86 insertions(+), 28 deletions(-) diff --git a/rust/helpers/slab.c b/rust/helpers/slab.c index a842bfbddcba..8472370a4338 100644 --- a/rust/helpers/slab.c +++ b/rust/helpers/slab.c @@ -3,13 +3,13 @@ #include void * __must_check __realloc_size(2) -rust_helper_krealloc(const void *objp, size_t new_size, gfp_t flags) +rust_helper_krealloc_node(const void *objp, size_t new_size, gfp_t flags, int node) { - return krealloc(objp, new_size, flags); + return krealloc_node(objp, new_size, flags, node); } void * __must_check __realloc_size(2) -rust_helper_kvrealloc(const void *p, size_t size, gfp_t flags) +rust_helper_kvrealloc_node(const void *p, size_t size, gfp_t flags, int node) { - return kvrealloc(p, size, flags); + return kvrealloc_node(p, size, flags, node); } diff --git a/rust/helpers/vmalloc.c b/rust/helpers/vmalloc.c index 80d34501bbc0..62d30db9a1a6 100644 --- a/rust/helpers/vmalloc.c +++ b/rust/helpers/vmalloc.c @@ -3,7 +3,7 @@ #include void * __must_check __realloc_size(2) -rust_helper_vrealloc(const void *p, size_t size, gfp_t flags) +rust_helper_vrealloc_node(const void *p, size_t size, gfp_t flags, int node) { - return vrealloc(p, size, flags); + return vrealloc_node(p, size, flags, node); } diff --git a/rust/kernel/alloc.rs b/rust/kernel/alloc.rs index a2c49e5494d3..6ba1675c9da0 100644 --- a/rust/kernel/alloc.rs +++ b/rust/kernel/alloc.rs @@ -28,6 +28,8 @@ /// Indicates an allocation error. #[derive(Copy, Clone, PartialEq, Eq, Debug)] pub struct AllocError; + +use crate::error::{code::EINVAL, Result}; use core::{alloc::Layout, ptr::NonNull}; /// Flags to be used when allocating memory. @@ -115,6 +117,29 @@ pub mod flags { pub const __GFP_NOWARN: Flags = Flags(bindings::__GFP_NOWARN); } +/// Non Uniform Memory Access (NUMA) node identifier +#[derive(Clone, Copy, PartialEq)] +pub struct NumaNode(i32); + +impl NumaNode { + /// create a new NUMA node identifer (non-negative integer) + /// returns EINVAL if a negative id or an id exceeding MAX_NUMNODES is specified + pub fn new(node: i32) -> Result { + // SAFETY: MAX_NUMNODES never exceeds 2**10 because NODES_SHIFT is 0..10 + if node < 0 || node >= bindings::MAX_NUMNODES as i32 { + return Err(EINVAL); + } + Ok(Self(node)) + } +} + +/// Specify necessary constant to pass the information to Allocator that the caller doesn't care +/// about the NUMA node to allocate memory from. +impl NumaNode { + /// No node preference. + pub const NO_NODE: NumaNode = NumaNode(bindings::NUMA_NO_NODE); +} + /// The kernel's [`Allocator`] trait. /// /// An implementation of [`Allocator`] can allocate, re-allocate and free memory buffers described @@ -137,7 +162,7 @@ pub mod flags { /// - Implementers must ensure that all trait functions abide by the guarantees documented in the /// `# Guarantees` sections. pub unsafe trait Allocator { - /// Allocate memory based on `layout` and `flags`. + /// Allocate memory based on `layout`, `flags` and `nid`. /// /// On success, returns a buffer represented as `NonNull<[u8]>` that satisfies the layout /// constraints (i.e. minimum size and alignment as specified by `layout`). @@ -153,13 +178,21 @@ pub unsafe trait Allocator { /// /// Additionally, `Flags` are honored as documented in /// . - fn alloc(layout: Layout, flags: Flags) -> Result, AllocError> { + fn alloc(layout: Layout, flags: Flags, nid: NumaNode) -> Result, AllocError> { // SAFETY: Passing `None` to `realloc` is valid by its safety requirements and asks for a // new memory allocation. - unsafe { Self::realloc(None, layout, Layout::new::<()>(), flags) } + unsafe { Self::realloc(None, layout, Layout::new::<()>(), flags, nid) } } - /// Re-allocate an existing memory allocation to satisfy the requested `layout`. + /// Re-allocate an existing memory allocation to satisfy the requested `layout` and + /// a specific NUMA node request to allocate the memory for. + /// + /// Systems employing a Non Uniform Memory Access (NUMA) architecture contain collections of + /// hardware resources including processors, memory, and I/O buses, that comprise what is + /// commonly known as a NUMA node. + /// + /// `nid` stands for NUMA id, i. e. NUMA node identifier, which is a non-negative integer + /// if a node needs to be specified, or [`NumaNode::NO_NODE`] if the caller doesn't care. /// /// If the requested size is zero, `realloc` behaves equivalent to `free`. /// @@ -196,6 +229,7 @@ unsafe fn realloc( layout: Layout, old_layout: Layout, flags: Flags, + nid: NumaNode, ) -> Result, AllocError>; /// Free an existing memory allocation. @@ -211,7 +245,15 @@ unsafe fn free(ptr: NonNull, layout: Layout) { // SAFETY: The caller guarantees that `ptr` points at a valid allocation created by this // allocator. We are passing a `Layout` with the smallest possible alignment, so it is // smaller than or equal to the alignment previously used with this allocation. - let _ = unsafe { Self::realloc(Some(ptr), Layout::new::<()>(), layout, Flags(0)) }; + let _ = unsafe { + Self::realloc( + Some(ptr), + Layout::new::<()>(), + layout, + Flags(0), + NumaNode::NO_NODE, + ) + }; } } diff --git a/rust/kernel/alloc/allocator.rs b/rust/kernel/alloc/allocator.rs index aa2dfa9dca4c..8af7e04e3cc6 100644 --- a/rust/kernel/alloc/allocator.rs +++ b/rust/kernel/alloc/allocator.rs @@ -13,7 +13,7 @@ use core::ptr; use core::ptr::NonNull; -use crate::alloc::{AllocError, Allocator}; +use crate::alloc::{AllocError, Allocator, NumaNode}; use crate::bindings; use crate::pr_warn; @@ -56,20 +56,25 @@ fn aligned_size(new_layout: Layout) -> usize { /// # Invariants /// -/// One of the following: `krealloc`, `vrealloc`, `kvrealloc`. +/// One of the following: `krealloc_node`, `vrealloc_node`, `kvrealloc_node`. struct ReallocFunc( - unsafe extern "C" fn(*const crate::ffi::c_void, usize, u32) -> *mut crate::ffi::c_void, + unsafe extern "C" fn( + *const crate::ffi::c_void, + usize, + u32, + crate::ffi::c_int, + ) -> *mut crate::ffi::c_void, ); impl ReallocFunc { - // INVARIANT: `krealloc` satisfies the type invariants. - const KREALLOC: Self = Self(bindings::krealloc); + // INVARIANT: `krealloc_node` satisfies the type invariants. + const KREALLOC: Self = Self(bindings::krealloc_node); - // INVARIANT: `vrealloc` satisfies the type invariants. - const VREALLOC: Self = Self(bindings::vrealloc); + // INVARIANT: `vrealloc_node` satisfies the type invariants. + const VREALLOC: Self = Self(bindings::vrealloc_node); - // INVARIANT: `kvrealloc` satisfies the type invariants. - const KVREALLOC: Self = Self(bindings::kvrealloc); + // INVARIANT: `kvrealloc_node` satisfies the type invariants. + const KVREALLOC: Self = Self(bindings::kvrealloc_node); /// # Safety /// @@ -87,6 +92,7 @@ unsafe fn call( layout: Layout, old_layout: Layout, flags: Flags, + nid: NumaNode, ) -> Result, AllocError> { let size = aligned_size(layout); let ptr = match ptr { @@ -110,7 +116,7 @@ unsafe fn call( // - Those functions provide the guarantees of this function. let raw_ptr = unsafe { // If `size == 0` and `ptr != NULL` the memory behind the pointer is freed. - self.0(ptr.cast(), size, flags.0).cast() + self.0(ptr.cast(), size, flags.0, nid.0).cast() }; let ptr = if size == 0 { @@ -134,9 +140,10 @@ unsafe fn realloc( layout: Layout, old_layout: Layout, flags: Flags, + nid: NumaNode, ) -> Result, AllocError> { // SAFETY: `ReallocFunc::call` has the same safety requirements as `Allocator::realloc`. - unsafe { ReallocFunc::KREALLOC.call(ptr, layout, old_layout, flags) } + unsafe { ReallocFunc::KREALLOC.call(ptr, layout, old_layout, flags, nid) } } } @@ -151,6 +158,7 @@ unsafe fn realloc( layout: Layout, old_layout: Layout, flags: Flags, + nid: NumaNode, ) -> Result, AllocError> { // TODO: Support alignments larger than PAGE_SIZE. if layout.align() > bindings::PAGE_SIZE { @@ -160,7 +168,7 @@ unsafe fn realloc( // SAFETY: If not `None`, `ptr` is guaranteed to point to valid memory, which was previously // allocated with this `Allocator`. - unsafe { ReallocFunc::VREALLOC.call(ptr, layout, old_layout, flags) } + unsafe { ReallocFunc::VREALLOC.call(ptr, layout, old_layout, flags, nid) } } } @@ -175,6 +183,7 @@ unsafe fn realloc( layout: Layout, old_layout: Layout, flags: Flags, + nid: NumaNode, ) -> Result, AllocError> { // TODO: Support alignments larger than PAGE_SIZE. if layout.align() > bindings::PAGE_SIZE { @@ -184,6 +193,6 @@ unsafe fn realloc( // SAFETY: If not `None`, `ptr` is guaranteed to point to valid memory, which was previously // allocated with this `Allocator`. - unsafe { ReallocFunc::KVREALLOC.call(ptr, layout, old_layout, flags) } + unsafe { ReallocFunc::KVREALLOC.call(ptr, layout, old_layout, flags, nid) } } } diff --git a/rust/kernel/alloc/kbox.rs b/rust/kernel/alloc/kbox.rs index c386ff771d50..5c0b020fb2a4 100644 --- a/rust/kernel/alloc/kbox.rs +++ b/rust/kernel/alloc/kbox.rs @@ -4,7 +4,7 @@ #[allow(unused_imports)] // Used in doc comments. use super::allocator::{KVmalloc, Kmalloc, Vmalloc}; -use super::{AllocError, Allocator, Flags}; +use super::{AllocError, Allocator, Flags, NumaNode}; use core::alloc::Layout; use core::fmt; use core::marker::PhantomData; @@ -271,7 +271,7 @@ pub fn new(x: T, flags: Flags) -> Result { /// ``` pub fn new_uninit(flags: Flags) -> Result, A>, AllocError> { let layout = Layout::new::>(); - let ptr = A::alloc(layout, flags)?; + let ptr = A::alloc(layout, flags, NumaNode::NO_NODE)?; // INVARIANT: `ptr` is either a dangling pointer or points to memory allocated with `A`, // which is sufficient in size and alignment for storing a `T`. diff --git a/rust/kernel/alloc/kvec.rs b/rust/kernel/alloc/kvec.rs index 1a0dd852a468..aa5d27176d9c 100644 --- a/rust/kernel/alloc/kvec.rs +++ b/rust/kernel/alloc/kvec.rs @@ -5,7 +5,7 @@ use super::{ allocator::{KVmalloc, Kmalloc, Vmalloc}, layout::ArrayLayout, - AllocError, Allocator, Box, Flags, + AllocError, Allocator, Box, Flags, NumaNode, }; use core::{ fmt, @@ -633,6 +633,7 @@ pub fn reserve(&mut self, additional: usize, flags: Flags) -> Result<(), AllocEr layout.into(), self.layout.into(), flags, + NumaNode::NO_NODE, )? }; @@ -1058,7 +1059,13 @@ pub fn collect(self, flags: Flags) -> Vec { // the type invariant to be smaller than `cap`. Depending on `realloc` this operation // may shrink the buffer or leave it as it is. ptr = match unsafe { - A::realloc(Some(buf.cast()), layout.into(), old_layout.into(), flags) + A::realloc( + Some(buf.cast()), + layout.into(), + old_layout.into(), + flags, + NumaNode::NO_NODE, + ) } { // If we fail to shrink, which likely can't even happen, continue with the existing // buffer. -- 2.39.2