From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80761C77B7C for ; Wed, 2 Jul 2025 16:09:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2405A900004; Wed, 2 Jul 2025 12:09:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 217C58E0002; Wed, 2 Jul 2025 12:09:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12DA5900004; Wed, 2 Jul 2025 12:09:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F3DB88E0002 for ; Wed, 2 Jul 2025 12:09:05 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 94C85160316 for ; Wed, 2 Jul 2025 16:09:05 +0000 (UTC) X-FDA: 83619808650.02.3AE532C Received: from mailrelay4-3.pub.mailoutpod3-cph3.one.com (mailrelay4-3.pub.mailoutpod3-cph3.one.com [46.30.212.3]) by imf28.hostedemail.com (Postfix) with ESMTP id 9819FC0002 for ; Wed, 2 Jul 2025 16:09:03 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=konsulko.se header.s=rsa1 header.b=2LISNcjr; dkim=pass header.d=konsulko.se header.s=ed1 header.b="I/AVUUqw" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751472543; a=rsa-sha256; cv=none; b=40DOHkymKy9rEBKN4o8GKl9SxDKxdQjOjuGS2Ls7TkDjFcI0cmE+osKVbyBcNeWpWcsfUQ XTHAclf9pZhCM8Lzm/Lc34r1LbSbAJ1p9YJpL2AtEBYCvbMszFyq+vZBx4z9olfidmX9zx EOUQNERjv91TQ7uTxKB2VQIoXyE0bZY= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=konsulko.se header.s=rsa1 header.b=2LISNcjr; dkim=pass header.d=konsulko.se header.s=ed1 header.b="I/AVUUqw"; dmarc=none; spf=none (imf28.hostedemail.com: domain of vitaly.wool@konsulko.se has no SPF policy when checking 46.30.212.3) smtp.mailfrom=vitaly.wool@konsulko.se ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751472543; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HMcuy67aBnM2gEG1I5gjH0uFFk4qS0eIAy3dUVDgrME=; b=Tp/zuGER8K8fH9hLpOQuMGkXCpXuH9SmmMLPc7raSFQ5fdwCpRSi7ABZRgt6AnNLaZuRgH W74LvaTe371wZUiOk2YphalfZ4btVirj60l8CvZ7XsWlOy1YXUM0fi8Hh1C1Mwsq5bYMBv cuOpYLXKptjK0GAiSzKnMrm/GEb+x+k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1751472541; x=1752077341; d=konsulko.se; s=rsa1; h=content-transfer-encoding:mime-version:references:in-reply-to:message-id:date: subject:cc:to:from:from; bh=HMcuy67aBnM2gEG1I5gjH0uFFk4qS0eIAy3dUVDgrME=; b=2LISNcjrDeAbkEcyApMF0DPFGO6jtof7cvnnF7rsP9cnn4qYXcmYt/DO2HbSJThrFH4Uh6CGCeAUA vs6zFxzVsF+l7Gbg3sTr3F9JfCiUVgRRS2yDLJYkRZw5LzTbtKnrZH/fq2gEIW5o3CS0dWTVkELWyc oIVnCg3xklDFPseRdwa81kr3ygCa8kYhXrbfq4RpAes7JIMivXjrEDJ8i1pijxnwhCY1WZ940rcsn1 bsouvJdE4rTPvmPc+QYzh0AcpBFdKTi3ttygoficPhJ04xLBO+jTKtIiinxTVo/JU63GNRnxrbEPMh poBGQ3lIc7byP8zoMhnZgkICGnrJ02Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; t=1751472541; x=1752077341; d=konsulko.se; s=ed1; h=content-transfer-encoding:mime-version:references:in-reply-to:message-id:date: subject:cc:to:from:from; bh=HMcuy67aBnM2gEG1I5gjH0uFFk4qS0eIAy3dUVDgrME=; b=I/AVUUqwPj0KDZGjWlsmuvzgBzSXCLj7K5/M2Kljmnw/grAu+jmZO6EbcqowOKVnE/AqnLbpbEEJl xHw++rnBA== X-HalOne-ID: dd55c8f4-575e-11f0-8e17-29b2d794c87d Received: from slottsdator.home (host-90-238-19-233.mobileonline.telia.com [90.238.19.233]) by mailrelay4.pub.mailoutpod3-cph3.one.com (Halon) with ESMTPSA id dd55c8f4-575e-11f0-8e17-29b2d794c87d; Wed, 02 Jul 2025 16:09:00 +0000 (UTC) From: Vitaly Wool To: linux-mm@kvack.org Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, Uladzislau Rezki , Danilo Krummrich , Alice Ryhl , rust-for-linux@vger.kernel.org, Vitaly Wool Subject: [PATCH v10 2/4] mm/slub: allow to set node and align in k[v]realloc Date: Wed, 2 Jul 2025 18:08:57 +0200 Message-Id: <20250702160857.3610105-1-vitaly.wool@konsulko.se> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250702160758.3609992-1-vitaly.wool@konsulko.se> References: <20250702160758.3609992-1-vitaly.wool@konsulko.se> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: awne5nrgeekhkmcmsbz1f611th88f8wm X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 9819FC0002 X-Rspam-User: X-HE-Tag: 1751472543-959724 X-HE-Meta: U2FsdGVkX1/lNIlAlwePNq7Q0zGUmivaCDJKWXxLRlvZvyBPRfMgTWXE63k8MjNrE6usM3BSXcx6KnBQz4LluOPRQyRLRgTFRAZsOVvH4199OOxwHPr/jZ6umBq3RZbu5F8r8j+xbSiPxAv9iXdr9HOq+rXCXHsdU9Wng97juoh+SmcwwYw5uflP250YbAmMsBY76urVFJ7tesgavfQlH1l/QI0+/+DONE5K8n6uPop/Uek42BQ/NfY19uuX2B1axON+5qIUOzfY9ZkAu/iOmEt6BqcurOyKZKBU9oQT2wCpmZhLsmzGDHBRoZBawD+rSXahPNtu8bsmzpQKeUswTsQnZy02jofxXIt4fCMDoI+7DbIxtWZ7P8iTB6VtopN8Vcas0eIH5855NnWcoGOk12i6RBtXZdAFPhmkUMKhZLuBUIL3OoenRGDvYNUlC9zHAmxQf2bq1C1lh/XYD/ZxqfHBCn6QRSeq70n5hIuNH1u4xvMSTYrOk3zXtfgC+01Ff0h4wrj4r5EltA/rIzGU2q1/hgoHku+dZBN6GmGL6RyRIkgyiSxIyOgD8gTST0OnkakHcYT94zclETn3RMQTezhYvZKJKpQ8LJmgXhp+tbCHfpCC2KfS9LnyNpVoacMAhZ4ojEwUfaimydP0Et+CcaarEVVHkaRIpT1UgZkQSp1NBEaEIRoo/vkxPw7pYbVVGBH/QpmfJ/uxy1zXrARr1gSId0q/JzQFVKSkn7D6WK2NR+ogmV3iJ0Huuq/q7U7CZt9HYqGvAzEHbse2/t/nnmjuCzUkRmGWZmPalbW+jQyA1M4e6APrfLH+hQtbnfTl16NYEL3jSil+Z8BhOUbkqOXOLj5y9zBUVFdRmeqrlptmoKaWGFCt7w7nrhPdy4yKu7l3iJSSthoWx6wcK5m9KGs7qc4czJu5XsbctQNyxZT4v5ZOp5qc0T10b6nYgnj0UC0BtuGI4GmC6Vg/SL1 95W4Le3u WH9OLhkyousYklUxgwPO53kPGnS6lZj8zrr9bVmysp9bhmywIGnVieNorJbJSjXvqcNv4WRqWidqr3km23bR+FrjwMFBPlZeZHfUiaJXoaz0KN4I33y2C124ZZrZelsGRq7yIbV3we0cW3OfGNowEJuwABYVckACDmCNMVQQwHMplqbQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Reimplement k[v]realloc_node() to be able to set node and alignment should a user need to do so. In order to do that while retaining the maximal backward compatibility, add k[v]realloc_node_align() functions and redefine the rest of API using these new ones. With that change we also provide the ability for the Rust part of the kernel to set node and alignment in its K[v]xxx [re]allocations. Signed-off-by: Vitaly Wool --- include/linux/slab.h | 40 ++++++++++++++++++--------- mm/slub.c | 64 ++++++++++++++++++++++++++++++-------------- 2 files changed, 71 insertions(+), 33 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index d5a8ab98035c..13abcf4ada22 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -465,9 +465,15 @@ int kmem_cache_shrink(struct kmem_cache *s); /* * Common kmalloc functions provided by all allocators */ -void * __must_check krealloc_noprof(const void *objp, size_t new_size, - gfp_t flags) __realloc_size(2); -#define krealloc(...) alloc_hooks(krealloc_noprof(__VA_ARGS__)) +void * __must_check krealloc_node_align_noprof(const void *objp, size_t new_size, + unsigned long align, + gfp_t flags, int nid) __realloc_size(2); +#define krealloc_node_noprof(_p, _s, _f, _n) \ + krealloc_node_align_noprof(_p, _s, 1, _f, _n) +#define krealloc_noprof(...) krealloc_node_noprof(__VA_ARGS__, NUMA_NO_NODE) +#define krealloc_node_align(...) alloc_hooks(krealloc_node_align_noprof(__VA_ARGS__)) +#define krealloc_node(...) alloc_hooks(krealloc_node_noprof(__VA_ARGS__)) +#define krealloc(...) alloc_hooks(krealloc_noprof(__VA_ARGS__)) void kfree(const void *objp); void kfree_sensitive(const void *objp); @@ -1041,18 +1047,23 @@ static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) #define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) #define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) __alloc_size(1); -#define kvmalloc_node_noprof(size, flags, node) \ - __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node) +void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align, + gfp_t flags, int node) __alloc_size(1); +#define kvmalloc_node_align_noprof(_size, _align, _flags, _node) \ + __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, NULL), _align, _flags, _node) +#define kvmalloc_node_noprof(_size, _flags, _node) \ + kvmalloc_node_align_noprof(_size, 1, _flags, _node) +#define kvmalloc_node_align(...) \ + alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__)) #define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) -#define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) -#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE) +#define kvmalloc_noprof(...) kvmalloc_node_noprof(__VA_ARGS__, NUMA_NO_NODE) +#define kvmalloc(...) alloc_hooks(kvmalloc_noprof(__VA_ARGS__)) #define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO) -#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node) +#define kvzalloc_node(_s, _f, _n) kvmalloc_node(_s, (_f)|__GFP_ZERO, _n) #define kmem_buckets_valloc(_b, _size, _flags) \ - alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) + alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), 1, _flags, NUMA_NO_NODE)) static inline __alloc_size(1, 2) void * kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) @@ -1068,13 +1079,16 @@ kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) #define kvmalloc_array_noprof(...) kvmalloc_array_node_noprof(__VA_ARGS__, NUMA_NO_NODE) #define kvcalloc_node_noprof(_n,_s,_f,_node) kvmalloc_array_node_noprof(_n,_s,(_f)|__GFP_ZERO,_node) #define kvcalloc_noprof(...) kvcalloc_node_noprof(__VA_ARGS__, NUMA_NO_NODE) - #define kvmalloc_array(...) alloc_hooks(kvmalloc_array_noprof(__VA_ARGS__)) #define kvcalloc_node(...) alloc_hooks(kvcalloc_node_noprof(__VA_ARGS__)) #define kvcalloc(...) alloc_hooks(kvcalloc_noprof(__VA_ARGS__)) -void *kvrealloc_noprof(const void *p, size_t size, gfp_t flags) - __realloc_size(2); +void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align, + gfp_t flags, int nid) __realloc_size(2); +#define kvrealloc_node_align(...) kvrealloc_node_align_noprof(__VA_ARGS__) +#define kvrealloc_node_noprof(_p, _s, _f, _n) kvrealloc_node_align_noprof(_p, _s, 1, _f, _n) +#define kvrealloc_node(...) alloc_hooks(kvrealloc_node_noprof(__VA_ARGS__)) +#define kvrealloc_noprof(...) kvrealloc_node_noprof(__VA_ARGS__, NUMA_NO_NODE) #define kvrealloc(...) alloc_hooks(kvrealloc_noprof(__VA_ARGS__)) extern void kvfree(const void *addr); diff --git a/mm/slub.c b/mm/slub.c index c4b64821e680..881244c357dd 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4845,7 +4845,7 @@ void kfree(const void *object) EXPORT_SYMBOL(kfree); static __always_inline __realloc_size(2) void * -__do_krealloc(const void *p, size_t new_size, gfp_t flags) +__do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid) { void *ret; size_t ks = 0; @@ -4859,6 +4859,20 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) if (!kasan_check_byte(p)) return NULL; + /* refuse to proceed if alignment is bigger than what kmalloc() provides */ + if (!IS_ALIGNED((unsigned long)p, align) || new_size < align) + return NULL; + + /* + * If reallocation is not necessary (e. g. the new size is less + * than the current allocated size), the current allocation will be + * preserved unless __GFP_THISNODE is set. In the latter case a new + * allocation on the requested node will be attempted. + */ + if (unlikely(flags & __GFP_THISNODE) && nid != NUMA_NO_NODE && + nid != page_to_nid(vmalloc_to_page(p))) + goto alloc_new; + if (is_kfence_address(p)) { ks = orig_size = kfence_ksize(p); } else { @@ -4903,7 +4917,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) return (void *)p; alloc_new: - ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); + ret = kmalloc_node_track_caller_noprof(new_size, flags, nid, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); @@ -4915,10 +4929,12 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) } /** - * krealloc - reallocate memory. The contents will remain unchanged. + * krealloc_node_align - reallocate memory. The contents will remain unchanged. * @p: object to reallocate memory for. * @new_size: how many bytes of memory are required. + * @align: desired alignment. * @flags: the type of memory to allocate. + * @nid: NUMA node or NUMA_NO_NODE * * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size * is 0 and @p is not a %NULL pointer, the object pointed to is freed. @@ -4947,7 +4963,8 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) * * Return: pointer to the allocated memory or %NULL in case of error */ -void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) +void *krealloc_node_align_noprof(const void *p, size_t new_size, unsigned long align, + gfp_t flags, int nid) { void *ret; @@ -4956,13 +4973,13 @@ void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) return ZERO_SIZE_PTR; } - ret = __do_krealloc(p, new_size, flags); + ret = __do_krealloc(p, new_size, align, flags, nid); if (ret && kasan_reset_tag(p) != kasan_reset_tag(ret)) kfree(p); return ret; } -EXPORT_SYMBOL(krealloc_noprof); +EXPORT_SYMBOL(krealloc_node_align_noprof); static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size) { @@ -4993,6 +5010,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size) * failure, fall back to non-contiguous (vmalloc) allocation. * @size: size of the request. * @b: which set of kmalloc buckets to allocate from. + * @align: desired alignment. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. * @node: numa node to allocate from * @@ -5005,19 +5023,22 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size) * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) +void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align, + gfp_t flags, int node) { void *ret; /* * It doesn't really make sense to fallback to vmalloc for sub page - * requests + * requests and small alignments */ - ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), - kmalloc_gfp_adjust(flags, size), - node, _RET_IP_); - if (ret || size <= PAGE_SIZE) - return ret; + if (size >= align) { + ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), + kmalloc_gfp_adjust(flags, size), + node, _RET_IP_); + if (ret || size <= PAGE_SIZE) + return ret; + } /* non-sleeping allocations are not supported by vmalloc */ if (!gfpflags_allow_blocking(flags)) @@ -5035,7 +5056,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) * about the resulting pointer, and cannot play * protection games. */ - return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, align, VMALLOC_START, VMALLOC_END, flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } @@ -5079,10 +5100,12 @@ void kvfree_sensitive(const void *addr, size_t len) EXPORT_SYMBOL(kvfree_sensitive); /** - * kvrealloc - reallocate memory; contents remain unchanged + * kvrealloc_node_align - reallocate memory; contents remain unchanged * @p: object to reallocate memory for * @size: the size to reallocate + * @align: desired alignment * @flags: the flags for the page level allocator + * @nid: NUMA node id * * If @p is %NULL, kvrealloc() behaves exactly like kvmalloc(). If @size is 0 * and @p is not a %NULL pointer, the object pointed to is freed. @@ -5100,17 +5123,18 @@ EXPORT_SYMBOL(kvfree_sensitive); * * Return: pointer to the allocated memory or %NULL in case of error */ -void *kvrealloc_noprof(const void *p, size_t size, gfp_t flags) +void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align, + gfp_t flags, int nid) { void *n; if (is_vmalloc_addr(p)) - return vrealloc_noprof(p, size, flags); + return vrealloc_node_align_noprof(p, size, align, flags, nid); - n = krealloc_noprof(p, size, kmalloc_gfp_adjust(flags, size)); + n = krealloc_node_align_noprof(p, size, align, kmalloc_gfp_adjust(flags, size), nid); if (!n) { /* We failed to krealloc(), fall back to kvmalloc(). */ - n = kvmalloc_noprof(size, flags); + n = kvmalloc_node_align_noprof(size, align, flags, nid); if (!n) return NULL; @@ -5126,7 +5150,7 @@ void *kvrealloc_noprof(const void *p, size_t size, gfp_t flags) return n; } -EXPORT_SYMBOL(kvrealloc_noprof); +EXPORT_SYMBOL(kvrealloc_node_align_noprof); struct detached_freelist { struct slab *slab; -- 2.39.2