From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EDEBC77B7F for ; Fri, 27 Jun 2025 09:39:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB58A8D000A; Fri, 27 Jun 2025 05:39:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E65138D0006; Fri, 27 Jun 2025 05:39:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D54D88D000A; Fri, 27 Jun 2025 05:39:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C0F738D0006 for ; Fri, 27 Jun 2025 05:39:06 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 61636C05F1 for ; Fri, 27 Jun 2025 09:39:06 +0000 (UTC) X-FDA: 83600681892.15.4E2919B Received: from mailrelay-egress12.pub.mailoutpod2-cph3.one.com (mailrelay-egress12.pub.mailoutpod2-cph3.one.com [46.30.211.187]) by imf24.hostedemail.com (Postfix) with ESMTP id 993A1180003 for ; Fri, 27 Jun 2025 09:39:04 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=konsulko.se header.s=rsa1 header.b=SLC8WGqY; dkim=pass header.d=konsulko.se header.s=ed1 header.b=XIuLFsgY; spf=none (imf24.hostedemail.com: domain of vitaly.wool@konsulko.se has no SPF policy when checking 46.30.211.187) smtp.mailfrom=vitaly.wool@konsulko.se; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751017144; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Zx3LglPu0mTTlfXiukstk1SD/1VGL40wbfi6XAP+H6c=; b=Yw6r6I9ktUpWTPLBP3ZZCEX2u2oIYjbZm44IfyIp2/+dmD5pdbwuZRPZ8+nhrsKm8uN9zL 9DC+iPtMeDdYTn+QXsb+04SFk1+OirduBafoIO4CrJV8Wdiyb65PMju9wxZUcQbmfsiA8L /TLM4t9LWTm3W3rmkhOcM2rdYYgaYBs= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=konsulko.se header.s=rsa1 header.b=SLC8WGqY; dkim=pass header.d=konsulko.se header.s=ed1 header.b=XIuLFsgY; spf=none (imf24.hostedemail.com: domain of vitaly.wool@konsulko.se has no SPF policy when checking 46.30.211.187) smtp.mailfrom=vitaly.wool@konsulko.se; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751017144; a=rsa-sha256; cv=none; b=HixWdR/nFUfgbCZDAYgI2hTXYXJL654kLDdBwyW/RbAcBQk3FkDjSuf08dk/7EvBnSVNJE AvXf3Jb5LicqQMvjZ4Uz0WMlOEiOmhIZ+VPiVgy2/e5ZjH/icbO1IHoI3v4a+GtYjOJaeX EVWVMoWeRs7+AxfKKptkVfZYJfe1WL0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1751017143; x=1751621943; d=konsulko.se; s=rsa1; h=content-transfer-encoding:mime-version:references:in-reply-to:message-id:date: subject:cc:to:from:from; bh=Zx3LglPu0mTTlfXiukstk1SD/1VGL40wbfi6XAP+H6c=; b=SLC8WGqYSlOuExKlSLNnuwYPyRqCrtKmjCo9cjzxxalOzpHfwynKd2FQ2lKRiaUD2nfleTlQFiBzB znNqjffyPyFMoziA0lGrMWUAzNvJeexYr0li0pNn9oYprRygXhBIkVnUvl1PfLso+oW3DcJ+Xxz3yd 70SyL21vuMpIbXhaVNE2qLqNYLoARkhwvxcVkf7ngVbKh7Vo/KM7XhgdBV+DB/26FjUNkjKBGQLebM eU7764KD5weZ7miuSeMGJBn3MnTzrQcBGHH5yuUX5swX4DJgWWEy+McK3L4dDwEbStW11tO93pyiZH YqPYIwsyzZ1RVpgX4Qs2Q0owUqwYjwA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; t=1751017143; x=1751621943; d=konsulko.se; s=ed1; h=content-transfer-encoding:mime-version:references:in-reply-to:message-id:date: subject:cc:to:from:from; bh=Zx3LglPu0mTTlfXiukstk1SD/1VGL40wbfi6XAP+H6c=; b=XIuLFsgYciRqdOrFWVTDTZ2QHucN29Inmmd4QCdcnzLjTC2FfY7gG8Ec7G87pVKgTiIuuNhIwsiZz Y62j114DA== X-HalOne-ID: 8e647328-533a-11f0-b070-4f541c8bf1cc Received: from slottsdator.home (host-90-238-19-233.mobileonline.telia.com [90.238.19.233]) by mailrelay3.pub.mailoutpod2-cph3.one.com (Halon) with ESMTPSA id 8e647328-533a-11f0-b070-4f541c8bf1cc; Fri, 27 Jun 2025 09:39:02 +0000 (UTC) From: Vitaly Wool To: linux-mm@kvack.org Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, Uladzislau Rezki , Danilo Krummrich , Alice Ryhl , rust-for-linux@vger.kernel.org, Vitaly Wool Subject: [PATCH v5 3/4] rust: add support for NUMA ids in allocations Date: Fri, 27 Jun 2025 11:38:58 +0200 Message-Id: <20250627093858.413855-1-vitaly.wool@konsulko.se> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250627092901.356909-1-vitaly.wool@konsulko.se> References: <20250627092901.356909-1-vitaly.wool@konsulko.se> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam11 X-Rspam-User: X-Rspamd-Queue-Id: 993A1180003 X-Stat-Signature: ztz59nf9xnupf88jk4a9jz4yg1ypytp3 X-HE-Tag: 1751017144-416779 X-HE-Meta: U2FsdGVkX1+vB5kNYBqcjIvJviYPvjA9L291XBYeok9IwqB8NbcUOPUAfP6GtEY8c78n6mgfyw4O464t7ZgQwjIGrIpeSZXHa47iYtI4hI4Iw4NxVqJeuzz3CfTffuuf9cA7C1S2wP4MPduko3OckLfuvawaobJzAm8tQS3jvBVReZaw94iNEgB6mhlW8vFuCMgNMo5QacRC214bOyiO+irDBl4pONwWl1UF57jIuhRMAB+kDGo+WBpEHPa3V3YizBKsyIxiMhvpfIoB3PPIr6SuRk5uVgwGZHLEFAMttsrAtXvFCwy86LTx1wj1e1OYpyxOtHjORgbIqxdUq/iiuO2njRqNdei8EJCVnFnib8aGlVg3E7dvqyBQ2hNHDyg1rFjYDCAJOXOZTqpqeRxMWK8OvxpWfIojWrEj296XY8WL7H53XGmt7+zaldqOkPfZMA2Ukza2h0GVNybXgKtv6HFvd0N4l/uWK8+/s1IhUU1m3k/Y/mkGArywOWaCvQIOvhuPFhgBE5ED7coG6X1jwmhVhEEyWzVWpM9gX7BNSh2gx4GmAR9cDpXwY2gzzfJ7MPnfqU69Iw8sDYHJriFzi4mjhRc3orHLukwoMdRBr0WMuHgnst/j152ZA+notx6qmfT/YrGFZjGCVD6EJf4AfqToYmo+7Kit5YD/ab9db+Y4XalIVgNSqFiEdflqLH/8/MBFuOpL/A+kEwcPZO6bn708aYUK31aqfmA1NpEwFUd1lQiYN+Z1nbImHiCkZM4KYJrFH/ggOlO46V66tyPtb4lS/18G0wgHW/JtVWkHFgEe3GPL4yXtNkAGi7XLbAAyHJrjLYgVjXnbg/vqLIXySlPI7s7GbPvs6FCqvs9Oka3k06BvpzHUrxC5oOJPKFGf/aq3vdvqP+U3xsqeocOzKHmrRmX6XIvl0ShUduDQHjS4ot9HS6thxcIFOFKW6fTqco9UMSqCZhYTTBbI9DE bPiBqs0f z+IE64Eq5lsW5dP26sxOVgRAX7kuTsSvLxqx7ob4lukTCCfFOIcuNDugiQXcXu0NXQQLmBZzMY78cMJOdnRedhD8T/YsJmRJnn95WWyTRnZmRIwEG61jWA9DDNZRCI0KGqTKj5QFzIdM9q0PmXhvAaTyyS4BxXBeKV+iy3gpzP1KioyVjCXKvadJ3uX4NDHX3dam3MFvXK11SotVzMmWKYJYC4R3Ty9mopUBR2bY3TjkVdTA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add support for specifying NUMA ids in Rust allocators as an Option (i. e. providing `None` as nid corresponds to NUMA_NO_NODE). To do this, modify ReallocFunc to use the new extended realloc primitives from the C side of the kernel (i. e. k[v]realloc_node/vrealloc_node) and add the new function alloc_node to the Allocator trait while keeping the existing one (alloc) for backward compatibility. This will allow to specify node to use for allocation of e. g. {KV}Box, as well as for future NUMA aware users of the API. Signed-off-by: Vitaly Wool --- rust/helpers/slab.c | 13 +++++++++++++ rust/helpers/vmalloc.c | 6 ++++++ rust/kernel/alloc.rs | 28 ++++++++++++++++++++++++++-- rust/kernel/alloc/allocator.rs | 33 ++++++++++++++++++++++----------- rust/kernel/alloc/kvec.rs | 3 ++- 5 files changed, 69 insertions(+), 14 deletions(-) diff --git a/rust/helpers/slab.c b/rust/helpers/slab.c index a842bfbddcba..5db9450e9348 100644 --- a/rust/helpers/slab.c +++ b/rust/helpers/slab.c @@ -13,3 +13,16 @@ rust_helper_kvrealloc(const void *p, size_t size, gfp_t flags) { return kvrealloc(p, size, flags); } + +void * __must_check __realloc_size(2) +rust_helper_krealloc_node(const void *objp, size_t new_size, unsigned long align, + gfp_t flags, int node) +{ + return krealloc_node(objp, new_size, align, flags, node); +} + +void * __must_check __realloc_size(2) +rust_helper_kvrealloc_node(const void *p, size_t size, unsigned long align, gfp_t flags, int node) +{ + return kvrealloc_node(p, size, align, flags, node); +} diff --git a/rust/helpers/vmalloc.c b/rust/helpers/vmalloc.c index 80d34501bbc0..7a71427a1e78 100644 --- a/rust/helpers/vmalloc.c +++ b/rust/helpers/vmalloc.c @@ -7,3 +7,9 @@ rust_helper_vrealloc(const void *p, size_t size, gfp_t flags) { return vrealloc(p, size, flags); } + +void * __must_check __realloc_size(2) +rust_helper_vrealloc_node(const void *p, size_t size, unsigned long align, gfp_t flags, int node) +{ + return vrealloc_node(p, size, align, flags, node); +} diff --git a/rust/kernel/alloc.rs b/rust/kernel/alloc.rs index a2c49e5494d3..1e26c2a7f47c 100644 --- a/rust/kernel/alloc.rs +++ b/rust/kernel/alloc.rs @@ -156,7 +156,30 @@ pub unsafe trait Allocator { fn alloc(layout: Layout, flags: Flags) -> Result, AllocError> { // SAFETY: Passing `None` to `realloc` is valid by its safety requirements and asks for a // new memory allocation. - unsafe { Self::realloc(None, layout, Layout::new::<()>(), flags) } + unsafe { Self::realloc(None, layout, Layout::new::<()>(), flags, None) } + } + + /// Allocate memory based on `layout`, `flags` and `nid`. + /// + /// On success, returns a buffer represented as `NonNull<[u8]>` that satisfies the layout + /// constraints (i.e. minimum size and alignment as specified by `layout`). + /// + /// This function is equivalent to `realloc` when called with `None`. + /// + /// # Guarantees + /// + /// When the return value is `Ok(ptr)`, then `ptr` is + /// - valid for reads and writes for `layout.size()` bytes, until it is passed to + /// [`Allocator::free`] or [`Allocator::realloc`], + /// - aligned to `layout.align()`, + /// + /// Additionally, `Flags` are honored as documented in + /// . + fn alloc_node(layout: Layout, flags: Flags, nid: Option) + -> Result, AllocError> { + // SAFETY: Passing `None` to `realloc` is valid by its safety requirements and asks for a + // new memory allocation. + unsafe { Self::realloc(None, layout, Layout::new::<()>(), flags, nid) } } /// Re-allocate an existing memory allocation to satisfy the requested `layout`. @@ -196,6 +219,7 @@ unsafe fn realloc( layout: Layout, old_layout: Layout, flags: Flags, + nid: Option, ) -> Result, AllocError>; /// Free an existing memory allocation. @@ -211,7 +235,7 @@ unsafe fn free(ptr: NonNull, layout: Layout) { // SAFETY: The caller guarantees that `ptr` points at a valid allocation created by this // allocator. We are passing a `Layout` with the smallest possible alignment, so it is // smaller than or equal to the alignment previously used with this allocation. - let _ = unsafe { Self::realloc(Some(ptr), Layout::new::<()>(), layout, Flags(0)) }; + let _ = unsafe { Self::realloc(Some(ptr), Layout::new::<()>(), layout, Flags(0), None) }; } } diff --git a/rust/kernel/alloc/allocator.rs b/rust/kernel/alloc/allocator.rs index aa2dfa9dca4c..4f0fe2b67593 100644 --- a/rust/kernel/alloc/allocator.rs +++ b/rust/kernel/alloc/allocator.rs @@ -58,18 +58,20 @@ fn aligned_size(new_layout: Layout) -> usize { /// /// One of the following: `krealloc`, `vrealloc`, `kvrealloc`. struct ReallocFunc( - unsafe extern "C" fn(*const crate::ffi::c_void, usize, u32) -> *mut crate::ffi::c_void, + unsafe extern "C" fn(*const crate::ffi::c_void, usize, crate::ffi::c_ulong, u32, + crate::ffi::c_int + ) -> *mut crate::ffi::c_void, ); impl ReallocFunc { - // INVARIANT: `krealloc` satisfies the type invariants. - const KREALLOC: Self = Self(bindings::krealloc); + // INVARIANT: `krealloc_node` satisfies the type invariants. + const KREALLOC_NODE: Self = Self(bindings::krealloc_node); - // INVARIANT: `vrealloc` satisfies the type invariants. - const VREALLOC: Self = Self(bindings::vrealloc); + // INVARIANT: `vrealloc_node` satisfies the type invariants. + const VREALLOC_NODE: Self = Self(bindings::vrealloc_node); - // INVARIANT: `kvrealloc` satisfies the type invariants. - const KVREALLOC: Self = Self(bindings::kvrealloc); + // INVARIANT: `kvrealloc_node` satisfies the type invariants. + const KVREALLOC_NODE: Self = Self(bindings::kvrealloc_node); /// # Safety /// @@ -87,6 +89,7 @@ unsafe fn call( layout: Layout, old_layout: Layout, flags: Flags, + nid: Option, ) -> Result, AllocError> { let size = aligned_size(layout); let ptr = match ptr { @@ -100,6 +103,11 @@ unsafe fn call( None => ptr::null(), }; + let c_nid = match nid { + None => bindings::NUMA_NO_NODE, + Some(n) => n, + }; + // SAFETY: // - `self.0` is one of `krealloc`, `vrealloc`, `kvrealloc` and thus only requires that // `ptr` is NULL or valid. @@ -110,7 +118,7 @@ unsafe fn call( // - Those functions provide the guarantees of this function. let raw_ptr = unsafe { // If `size == 0` and `ptr != NULL` the memory behind the pointer is freed. - self.0(ptr.cast(), size, flags.0).cast() + self.0(ptr.cast(), size, 1, flags.0, c_nid).cast() }; let ptr = if size == 0 { @@ -134,9 +142,10 @@ unsafe fn realloc( layout: Layout, old_layout: Layout, flags: Flags, + nid: Option, ) -> Result, AllocError> { // SAFETY: `ReallocFunc::call` has the same safety requirements as `Allocator::realloc`. - unsafe { ReallocFunc::KREALLOC.call(ptr, layout, old_layout, flags) } + unsafe { ReallocFunc::KREALLOC_NODE.call(ptr, layout, old_layout, flags, nid) } } } @@ -151,6 +160,7 @@ unsafe fn realloc( layout: Layout, old_layout: Layout, flags: Flags, + nid: Option, ) -> Result, AllocError> { // TODO: Support alignments larger than PAGE_SIZE. if layout.align() > bindings::PAGE_SIZE { @@ -160,7 +170,7 @@ unsafe fn realloc( // SAFETY: If not `None`, `ptr` is guaranteed to point to valid memory, which was previously // allocated with this `Allocator`. - unsafe { ReallocFunc::VREALLOC.call(ptr, layout, old_layout, flags) } + unsafe { ReallocFunc::VREALLOC_NODE.call(ptr, layout, old_layout, flags, nid) } } } @@ -175,6 +185,7 @@ unsafe fn realloc( layout: Layout, old_layout: Layout, flags: Flags, + nid: Option, ) -> Result, AllocError> { // TODO: Support alignments larger than PAGE_SIZE. if layout.align() > bindings::PAGE_SIZE { @@ -184,6 +195,6 @@ unsafe fn realloc( // SAFETY: If not `None`, `ptr` is guaranteed to point to valid memory, which was previously // allocated with this `Allocator`. - unsafe { ReallocFunc::KVREALLOC.call(ptr, layout, old_layout, flags) } + unsafe { ReallocFunc::KVREALLOC_NODE.call(ptr, layout, old_layout, flags, nid) } } } diff --git a/rust/kernel/alloc/kvec.rs b/rust/kernel/alloc/kvec.rs index 1a0dd852a468..ef4f977ba012 100644 --- a/rust/kernel/alloc/kvec.rs +++ b/rust/kernel/alloc/kvec.rs @@ -633,6 +633,7 @@ pub fn reserve(&mut self, additional: usize, flags: Flags) -> Result<(), AllocEr layout.into(), self.layout.into(), flags, + None, )? }; @@ -1058,7 +1059,7 @@ pub fn collect(self, flags: Flags) -> Vec { // the type invariant to be smaller than `cap`. Depending on `realloc` this operation // may shrink the buffer or leave it as it is. ptr = match unsafe { - A::realloc(Some(buf.cast()), layout.into(), old_layout.into(), flags) + A::realloc(Some(buf.cast()), layout.into(), old_layout.into(), flags, None) } { // If we fail to shrink, which likely can't even happen, continue with the existing // buffer. -- 2.39.2