Re: [PATCH v3] SLUB: Add support for per object memory policies

[Vlastimil Babka <vbabka@suse.cz>]

On 10/1/24 21:08, Christoph Lameter via B4 Relay wrote:
> From: Christoph Lameter <cl@gentwo.org>
> 
>     The old SLAB allocator used to support memory policies on a per
>     allocation bases. In SLUB the memory policies are applied on a
>     per page frame / folio bases. Doing so avoids having to check memory
>     policies in critical code paths for kmalloc and friends.
> 
>     This worked on general well on Intel/AMD/PowerPC because the
>     interconnect technology is mature and can minimize the latencies
>     through intelligent caching even if a small object is not
>     placed optimally.
> 
>     However, on ARM we have an emergence of new NUMA interconnect
>     technology based more on embedded devices. Caching of remote content
>     can currently be ineffective using the standard building blocks / mesh
>     available on that platform. Such architectures benefit if each slab
>     object is individually placed according to memory policies
>     and other restrictions.
> 
>     This patch adds another kernel parameter
> 
>             slab_strict_numa
> 
>     If that is set then a static branch is activated that will cause
>     the hotpaths of the allocator to evaluate the current memory
>     allocation policy. Each object will be properly placed by
>     paying the price of extra processing and SLUB will no longer
>     defer to the page allocator to apply memory policies at the
>     folio level.
> 
>     This patch improves performance of memcached running
>     on Ampere Altra 2P system (ARM Neoverse N1 processor)
>     by 3.6% due to accurate placement of small kernel objects.
> 
> Tested-by: Huang Shijie <shijie@os.amperecomputing.com>
> Signed-off-by: Christoph Lameter (Ampere) <cl@gentwo.org>

OK, but we should document this parameter in:
Documentation/admin-guide/kernel-parameters.rst
Documentation/mm/slab.rst

Thanks,
Vlastimil

> ---
> Changes in v3:
> - Make the static key a static in slub.c
> - Use pr_warn / pr_info instead of printk
> - Link to v2: https://lore.kernel.org/r/20240906-strict_numa-v2-1-f104e6de6d1e@gentwo.org
> 
> Changes in v2:
> - Fix various issues
> - Testing
> ---
>  mm/slub.c | 42 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 42 insertions(+)
> 
> diff --git a/mm/slub.c b/mm/slub.c
> index 21f71cb6cc06..7ae94f79740d 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -218,6 +218,10 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled);
>  #endif
>  #endif		/* CONFIG_SLUB_DEBUG */
>  
> +#ifdef CONFIG_NUMA
> +static DEFINE_STATIC_KEY_FALSE(strict_numa);
> +#endif
> +
>  /* Structure holding parameters for get_partial() call chain */
>  struct partial_context {
>  	gfp_t flags;
> @@ -3957,6 +3961,28 @@ static __always_inline void *__slab_alloc_node(struct kmem_cache *s,
>  	object = c->freelist;
>  	slab = c->slab;
>  
> +#ifdef CONFIG_NUMA
> +	if (static_branch_unlikely(&strict_numa) &&
> +			node == NUMA_NO_NODE) {
> +
> +		struct mempolicy *mpol = current->mempolicy;
> +
> +		if (mpol) {
> +			/*
> +			 * Special BIND rule support. If existing slab
> +			 * is in permitted set then do not redirect
> +			 * to a particular node.
> +			 * Otherwise we apply the memory policy to get
> +			 * the node we need to allocate on.
> +			 */
> +			if (mpol->mode != MPOL_BIND || !slab ||
> +					!node_isset(slab_nid(slab), mpol->nodes))
> +
> +				node = mempolicy_slab_node();
> +		}
> +	}
> +#endif
> +
>  	if (!USE_LOCKLESS_FAST_PATH() ||
>  	    unlikely(!object || !slab || !node_match(slab, node))) {
>  		object = __slab_alloc(s, gfpflags, node, addr, c, orig_size);
> @@ -5601,6 +5627,22 @@ static int __init setup_slub_min_objects(char *str)
>  __setup("slab_min_objects=", setup_slub_min_objects);
>  __setup_param("slub_min_objects=", slub_min_objects, setup_slub_min_objects, 0);
>  
> +#ifdef CONFIG_NUMA
> +static int __init setup_slab_strict_numa(char *str)
> +{
> +	if (nr_node_ids > 1) {
> +		static_branch_enable(&strict_numa);
> +		pr_info("SLUB: Strict NUMA enabled.\n");
> +	} else
> +		pr_warn("slab_strict_numa parameter set on non NUMA system.\n");
> +
> +	return 1;
> +}
> +
> +__setup("slab_strict_numa", setup_slab_strict_numa);
> +#endif
> +
> +
>  #ifdef CONFIG_HARDENED_USERCOPY
>  /*
>   * Rejects incorrectly sized objects and objects that are to be copied
> 
> ---
> base-commit: e32cde8d2bd7d251a8f9b434143977ddf13dcec6
> change-id: 20240819-strict_numa-fc59b33123a2
> 
> Best regards,