Re: [PATCH] powerpc/numa: Associate numa node to its cpu earlier

[Srikar Dronamraju <srikar@linux.vnet.ibm.com>]

* Oscar Salvador <osalvador@suse.de> [2022-04-11 09:49:34]:

> powerpc is the only platform that do not rely on
> cpu_up()->try_online_node() to bring up a numa node,
> and special cases it, instead, deep in its own machinery:
> 
> dlpar_online_cpu
>  find_and_online_cpu_nid
>   try_online_node
> 
> This should not be needed, but the thing is that the try_online_node()
> from cpu_up() will not apply on the right node, because cpu_to_node()
> will return the old mapping numa<->cpu that gets set on boot stage
> for all possible cpus.
> 
> That can be seen easily if we try to print out the numa node passed
> to try_online_node() in cpu_up().
> 
> The thing is that the numa<->cpu mapping does not get updated till a much
> later stage in start_secondary:
> 
> start_secondary:
>  set_numa_node(numa_cpu_lookup_table[cpu])
> 
> But we do not really care, as we already now the
> CPU <-> NUMA associativity back in find_and_online_cpu_nid(),
> so let us make use of that and set the proper numa<->cpu mapping,
> so cpu_to_node() in cpu_up() returns the right node and
> try_online_node() can do its work.
> 
> Signed-off-by: Oscar Salvador <osalvador@suse.de>
> Reviewed-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
> Tested-by: Geetika Moolchandani <Geetika.Moolchandani1@ibm.com>
> ---
>  arch/powerpc/include/asm/topology.h          |  8 ++-----
>  arch/powerpc/mm/numa.c                       | 31 +++++++---------------------
>  arch/powerpc/platforms/pseries/hotplug-cpu.c |  2 +-
>  3 files changed, 11 insertions(+), 30 deletions(-)
> 
> diff --git a/arch/powerpc/include/asm/topology.h b/arch/powerpc/include/asm/topology.h
> index 36fcafb1fd6d..6ae1b2dce83e 100644
> --- a/arch/powerpc/include/asm/topology.h
> +++ b/arch/powerpc/include/asm/topology.h
> @@ -111,14 +111,10 @@ static inline void unmap_cpu_from_node(unsigned long cpu) {}
>  #endif /* CONFIG_NUMA */
> 
>  #if defined(CONFIG_NUMA) && defined(CONFIG_PPC_SPLPAR)
> -extern int find_and_online_cpu_nid(int cpu);
> +extern void find_and_update_cpu_nid(int cpu);
>  extern int cpu_to_coregroup_id(int cpu);
>  #else
> -static inline int find_and_online_cpu_nid(int cpu)
> -{
> -	return 0;
> -}
> -
> +static inline void find_and_update_cpu_nid(int cpu) {}
>  static inline int cpu_to_coregroup_id(int cpu)
>  {
>  #ifdef CONFIG_SMP
> diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
> index b9b7fefbb64b..b5bc8b1a833d 100644
> --- a/arch/powerpc/mm/numa.c
> +++ b/arch/powerpc/mm/numa.c
> @@ -1423,43 +1423,28 @@ static long vphn_get_associativity(unsigned long cpu,
>  	return rc;
>  }
> 
> -int find_and_online_cpu_nid(int cpu)
> +void find_and_update_cpu_nid(int cpu)
>  {
>  	__be32 associativity[VPHN_ASSOC_BUFSIZE] = {0};
>  	int new_nid;
> 
>  	/* Use associativity from first thread for all siblings */
>  	if (vphn_get_associativity(cpu, associativity))
> -		return cpu_to_node(cpu);
> +		return;
> 
> +	/* Do not have previous associativity, so find it now. */
>  	new_nid = associativity_to_nid(associativity);
> +
>  	if (new_nid < 0 || !node_possible(new_nid))
>  		new_nid = first_online_node;
> -
> -	if (NODE_DATA(new_nid) == NULL) {

Given that my patch got accepted into powerpc tree
https://git.kernel.org/powerpc/c/e4ff77598a109bd36789ad5e80aba66fc53d0ffb
is now part of Linus tree, this line may need a slight tweak.

> -#ifdef CONFIG_MEMORY_HOTPLUG
> -		/*
> -		 * Need to ensure that NODE_DATA is initialized for a node from
> -		 * available memory (see memblock_alloc_try_nid). If unable to
> -		 * init the node, then default to nearest node that has memory
> -		 * installed. Skip onlining a node if the subsystems are not
> -		 * yet initialized.
> -		 */
> -		if (!topology_inited || try_online_node(new_nid))
> -			new_nid = first_online_node;
> -#else
> -		/*
> -		 * Default to using the nearest node that has memory installed.
> -		 * Otherwise, it would be necessary to patch the kernel MM code
> -		 * to deal with more memoryless-node error conditions.
> +	else
> +		/* Associate node <-> cpu, so cpu_up() calls
> +		 * try_online_node() on the right node.
>  		 */
> -		new_nid = first_online_node;
> -#endif
> -	}
> +		set_cpu_numa_node(cpu, new_nid);
> 
>  	pr_debug("%s:%d cpu %d nid %d\n", __FUNCTION__, __LINE__,
>  		cpu, new_nid);
> -	return new_nid;
>  }
> 
>  int cpu_to_coregroup_id(int cpu)
> diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c b/arch/powerpc/platforms/pseries/hotplug-cpu.c
> index b81fc846d99c..0f8cd8b06432 100644
> --- a/arch/powerpc/platforms/pseries/hotplug-cpu.c
> +++ b/arch/powerpc/platforms/pseries/hotplug-cpu.c
> @@ -398,7 +398,7 @@ static int dlpar_online_cpu(struct device_node *dn)
>  			if (get_hard_smp_processor_id(cpu) != thread)
>  				continue;
>  			cpu_maps_update_done();
> -			find_and_online_cpu_nid(cpu);
> +			find_and_update_cpu_nid(cpu);
>  			rc = device_online(get_cpu_device(cpu));
>  			if (rc) {
>  				dlpar_offline_cpu(dn);
> -- 
> 2.16.4
>