Re: [PATCH v4 24/26] arch_numa: switch over to numa_memblks

Re: [PATCH v4 24/26] arch_numa: switch over to numa_memblks

[Marc Zyngier]

Hi Mike,

Sorry for reviving a rather old thread.

On Wed, 07 Aug 2024 07:41:08 +0100,
Mike Rapoport <rppt@kernel.org> wrote:
> 
> From: "Mike Rapoport (Microsoft)" <rppt@kernel.org>
> 
> Until now arch_numa was directly translating firmware NUMA information
> to memblock.
> 
> Using numa_memblks as an intermediate step has a few advantages:
> * alignment with more battle tested x86 implementation
> * availability of NUMA emulation
> * maintaining node information for not yet populated memory
> 
> Adjust a few places in numa_memblks to compile with 32-bit phys_addr_t
> and replace current functionality related to numa_add_memblk() and
> __node_distance() in arch_numa with the implementation based on
> numa_memblks and add functions required by numa_emulation.
> 
> Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
> Tested-by: Zi Yan <ziy@nvidia.com> # for x86_64 and arm64
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> [arm64 + CXL via QEMU]
> Acked-by: Dan Williams <dan.j.williams@intel.com>
> Acked-by: David Hildenbrand <david@redhat.com>
> ---
>  drivers/base/Kconfig       |   1 +
>  drivers/base/arch_numa.c   | 201 +++++++++++--------------------------
>  include/asm-generic/numa.h |   6 +-
>  mm/numa_memblks.c          |  17 ++--
>  4 files changed, 75 insertions(+), 150 deletions(-)
>

[...]

>  static int __init numa_register_nodes(void)
>  {
>  	int nid;
> -	struct memblock_region *mblk;
> -
> -	/* Check that valid nid is set to memblks */
> -	for_each_mem_region(mblk) {
> -		int mblk_nid = memblock_get_region_node(mblk);
> -		phys_addr_t start = mblk->base;
> -		phys_addr_t end = mblk->base + mblk->size - 1;
> -
> -		if (mblk_nid == NUMA_NO_NODE || mblk_nid >= MAX_NUMNODES) {
> -			pr_warn("Warning: invalid memblk node %d [mem %pap-%pap]\n",
> -				mblk_nid, &start, &end);
> -			return -EINVAL;
> -		}
> -	}
>  

This hunk has the unfortunate side effect of killing my ThunderX
extremely early at boot time, as this sorry excuse for a machine
really relies on the kernel recognising that whatever NUMA information
the FW offers is BS.

Reverting this hunk restores happiness (sort of).

FWIW, I've posted a patch with such revert at [1].

Thanks,

	M.

[1] https://lore.kernel.org/r/20241127193000.3702637-1-maz@kernel.org

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v4 24/26] arch_numa: switch over to numa_memblks

[Mike Rapoport]

Hi,

On Mon, Aug 26, 2024 at 06:17:22PM +0000, Bruno Faccini wrote:
> > On 7 Aug 2024, at 2:41, Mike Rapoport wrote:
> > 
> > From: "Mike Rapoport (Microsoft)" <rppt@kernel.org>
> > 
> > Until now arch_numa was directly translating firmware NUMA information
> > to memblock.
> > 
> > Using numa_memblks as an intermediate step has a few advantages:
> > * alignment with more battle tested x86 implementation
> > * availability of NUMA emulation
> > * maintaining node information for not yet populated memory
> > 
> > Adjust a few places in numa_memblks to compile with 32-bit phys_addr_t
> > and replace current functionality related to numa_add_memblk() and
> > __node_distance() in arch_numa with the implementation based on
> > numa_memblks and add functions required by numa_emulation.
> > 
> > Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
> > Tested-by: Zi Yan <ziy@nvidia.com> # for x86_64 and arm64
> > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> [arm64 + CXL via
> > QEMU]
> > Acked-by: Dan Williams <dan.j.williams@intel.com>
> > Acked-by: David Hildenbrand <david@redhat.com>
> > ---
> >   drivers/base/Kconfig       |   1 +
> >   drivers/base/arch_numa.c   | 201 +++++++++++--------------------------
> >   include/asm-generic/numa.h |   6 +-
> >   mm/numa_memblks.c          |  17 ++--
> >   4 files changed, 75 insertions(+), 150 deletions(-)
> >  
> > <snip>
> > 
> > +
> > +u64 __init numa_emu_dma_end(void)
> > +{
> > +             return PFN_PHYS(memblock_start_of_DRAM() + SZ_4G);
> > +}
> > +
> 
> PFN_PHYS() translation is unnecessary here, as
> memblock_start_of_DRAM() + SZ_4G is already a
> memory size.
> 
> This should fix it:
>  
> diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c
> index 8d49893c0e94..e18701676426 100644
> --- a/drivers/base/arch_numa.c
> +++ b/drivers/base/arch_numa.c
> @@ -346,7 +346,7 @@ void __init numa_emu_update_cpu_to_node(int
> *emu_nid_to_phys,
> 
> u64 __init numa_emu_dma_end(void)
> {
> -              return PFN_PHYS(memblock_start_of_DRAM() + SZ_4G);
> +             return memblock_start_of_DRAM() + SZ_4G;
> }
> 
> void debug_cpumask_set_cpu(unsigned int cpu, int node, bool enable)

Right, I've missed that. Thanks for the fix!

Andrew, can you please apply this (with fixed formatting)

diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c
index 8d49893c0e94..e18701676426 100644
--- a/drivers/base/arch_numa.c
+++ b/drivers/base/arch_numa.c
@@ -346,7 +346,7 @@ void __init numa_emu_update_cpu_to_node(int *emu_nid_to_phys,
 
 u64 __init numa_emu_dma_end(void)
 {
-	return PFN_PHYS(memblock_start_of_DRAM() + SZ_4G);
+	return memblock_start_of_DRAM() + SZ_4G;
 }
 
 void debug_cpumask_set_cpu(unsigned int cpu, int node, bool enable)

-- 
Sincerely yours,
Mike.

Re: [PATCH v4 24/26] arch_numa: switch over to numa_memblks

[Bruno Faccini]

On 7 Aug 2024, at 2:41, Mike Rapoport wrote:

From: "Mike Rapoport (Microsoft)" <rppt@kernel.org>

Until now arch_numa was directly translating firmware NUMA information
to memblock.

Using numa_memblks as an intermediate step has a few advantages:
* alignment with more battle tested x86 implementation
* availability of NUMA emulation
* maintaining node information for not yet populated memory

Adjust a few places in numa_memblks to compile with 32-bit phys_addr_t
and replace current functionality related to numa_add_memblk() and
__node_distance() in arch_numa with the implementation based on
numa_memblks and add functions required by numa_emulation.

Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Tested-by: Zi Yan <ziy@nvidia.com> # for x86_64 and arm64
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> [arm64 + CXL via QEMU]
Acked-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: David Hildenbrand <david@redhat.com>
---
  drivers/base/Kconfig       |   1 +
  drivers/base/arch_numa.c   | 201 +++++++++++--------------------------
  include/asm-generic/numa.h |   6 +-
  mm/numa_memblks.c          |  17 ++--
  4 files changed, 75 insertions(+), 150 deletions(-)


<snip>

+
+u64 __init numa_emu_dma_end(void)
+{
+             return PFN_PHYS(memblock_start_of_DRAM() + SZ_4G);
+}
+

PFN_PHYS() translation is unnecessary here, as
memblock_start_of_DRAM() + SZ_4G is already a
memory size.

This should fix it:
====================================================
diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c
index 8d49893c0e94..e18701676426 100644
--- a/drivers/base/arch_numa.c
+++ b/drivers/base/arch_numa.c
@@ -346,7 +346,7 @@ void __init numa_emu_update_cpu_to_node(int *emu_nid_to_phys,

u64 __init numa_emu_dma_end(void)
{
-              return PFN_PHYS(memblock_start_of_DRAM() + SZ_4G);
+             return memblock_start_of_DRAM() + SZ_4G;
}

void debug_cpumask_set_cpu(unsigned int cpu, int node, bool enable)
====================================================



!!! I had a lot of trouble to send in plain text from Outlook on my Mac, sorry for the noise and the duplicate copies !!!

Re: [PATCH v4 24/26] arch_numa: switch over to numa_memblks

[Bruno Faccini]

[-- Attachment #1: Type: text/plain, Size: 2252 bytes --]

On 7 Aug 2024, at 2:41, Mike Rapoport wrote:

From: "Mike Rapoport (Microsoft)" <rppt@kernel.org<mailto:rppt@kernel.org>>

Until now arch_numa was directly translating firmware NUMA information
to memblock.

Using numa_memblks as an intermediate step has a few advantages:
* alignment with more battle tested x86 implementation
* availability of NUMA emulation
* maintaining node information for not yet populated memory

Adjust a few places in numa_memblks to compile with 32-bit phys_addr_t
and replace current functionality related to numa_add_memblk() and
__node_distance() in arch_numa with the implementation based on
numa_memblks and add functions required by numa_emulation.

Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org<mailto:rppt@kernel.org>>
Tested-by: Zi Yan <ziy@nvidia.com<mailto:ziy@nvidia.com>> # for x86_64 and arm64
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com<mailto:Jonathan.Cameron@huawei.com>>
Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com<mailto:Jonathan.Cameron@huawei.com>> [arm64 + CXL via QEMU]
Acked-by: Dan Williams <dan.j.williams@intel.com<mailto:dan.j.williams@intel.com>>
Acked-by: David Hildenbrand <david@redhat.com<mailto:david@redhat.com>>
---
  drivers/base/Kconfig       |   1 +
  drivers/base/arch_numa.c   | 201 +++++++++++--------------------------
  include/asm-generic/numa.h |   6 +-
  mm/numa_memblks.c          |  17 ++--
  4 files changed, 75 insertions(+), 150 deletions(-)


<snip>

+
+u64 __init numa_emu_dma_end(void)
+{
+             return PFN_PHYS(memblock_start_of_DRAM() + SZ_4G);
+}
+

PFN_PHYS() translation is unnecessary here, as
memblock_start_of_DRAM() + SZ_4G is already a
memory size.

This should fix it:

diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c
index 8d49893c0e94..e18701676426 100644
--- a/drivers/base/arch_numa.c
+++ b/drivers/base/arch_numa.c
@@ -346,7 +346,7 @@ void __init numa_emu_update_cpu_to_node(int *emu_nid_to_phys,

u64 __init numa_emu_dma_end(void)
{
-              return PFN_PHYS(memblock_start_of_DRAM() + SZ_4G);
+             return memblock_start_of_DRAM() + SZ_4G;
}

void debug_cpumask_set_cpu(unsigned int cpu, int node, bool enable)



[-- Attachment #2: Type: text/html, Size: 8981 bytes --]