memory_block_size reduction in memory hot plug, can we reduced it to 1GB,currently it is 4GB

memory_block_size reduction in memory hot plug, can we reduced it to 1GB,currently it is 4GB

[Ranjan, Vikash]

[-- Attachment #1.1: Type: text/plain, Size: 564 bytes --]

  Hello Kernel Memory Team,


We are trying to enable memory hot plug for our device's based on arm64, is it possible to reduce memory_block_size to 1 GB or lesser as currently it is 4GB.

If you have any info on updated patch or any other way we can do this, please let us know.
 After applying attached test patch (for reduction of memory_block_size to 1 GB),we are getting below error. Probe is working fine in case of 4GB.
 root@sys:/sys/devices/system/memory# echo 0x980000000  > /sys/devices/system/memory/probe
echo: write error

Thanks,
Vikash

[-- Attachment #1.2: Type: text/html, Size: 2733 bytes --]

[-- Attachment #2: hotplug.patch --]
[-- Type: application/octet-stream, Size: 1729 bytes --]

diff --git a/arch/arm64/configs/exynosauto9_linux_vm_defconfig b/arch/arm64/configs/exynosauto9_linux_vm_defconfig
index 2567f5bc24f1..54bf4618c9ca 100644
--- a/arch/arm64/configs/exynosauto9_linux_vm_defconfig
+++ b/arch/arm64/configs/exynosauto9_linux_vm_defconfig

+CONFIG_SPARSEMEM=y
+CONFIG_MEMORY_HOTPLUG=y
+CONFIG_ARCH_MEMORY_PROBE=y
+CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 5a8c430fb8ff..3d2988b4bfed 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -469,7 +469,7 @@ static DEVICE_ATTR_RW(auto_online_blocks);
  * as well as ppc64 will do all of their discovery in userspace
  * and will require this interface.
  */
-#ifdef CONFIG_ARCH_MEMORY_PROBE
+//#ifdef CONFIG_ARCH_MEMORY_PROBE
 static ssize_t probe_store(struct device *dev, struct device_attribute *attr,
                           const char *buf, size_t count)
 {
@@ -490,7 +490,7 @@ static ssize_t probe_store(struct device *dev, struct device_attribute *attr,

        nid = memory_add_physaddr_to_nid(phys_addr);
        ret = __add_memory(nid, phys_addr,
-                          MIN_MEMORY_BLOCK_SIZE * sections_per_block);
+                          (MIN_MEMORY_BLOCK_SIZE/4) * sections_per_block);

        if (ret)
                goto out;
@@ -502,7 +502,7 @@ static ssize_t probe_store(struct device *dev, struct device_attribute *attr,
 }

 static DEVICE_ATTR_WO(probe);
-#endif
+//#endif

 #ifdef CONFIG_MEMORY_FAILURE
 /*
@@ -756,9 +756,9 @@ bool is_memblock_offlined(struct memory_block *mem)
 }

 static struct attribute *memory_root_attrs[] = {
-#ifdef CONFIG_ARCH_MEMORY_PROBE
+//#ifdef CONFIG_ARCH_MEMORY_PROBE
        &dev_attr_probe.attr,
-#endif
+//#endif

Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB,currently it is 4GB

[David Hildenbrand]

Hi

> 
> We are trying to enable memory hot plug for our device's based on arm64,
> is it possible to reduce memory_block_size to 1 GB or lesser as
> currently it is 4GB.

... are you sure it's 4GB? I never heard that magic number on arm64
before. The section size is 512 MiB on arm64 with 64k and 128 MiB with 4k.

> 
> If you have any info on updated patch or any other way we can do this,
> please let us know.
> 
>  After applying attached test patch (for reduction of memory_block_size
> to 1 GB),we are getting below error. Probe is working fine in case of 4GB.
> 

I think I am missing something important.

#define MIN_MEMORY_BLOCK_SIZE     (1UL << SECTION_SIZE_BITS)

arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 29
arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

-> 512 MiB / 128 MiB


arm64 does not override memory_block_size_bytes(), so it matches exactly
these numbers.


-- 
Thanks,

David / dhildenb

RE: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

[Ranjan, Vikash]

Hi David,

You were right , in my source code 
arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 30 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

I need your input on below.
1) currently due to code ( if (phys_addr & ((pages_per_block << PAGE_SHIFT) - 1)) return -EINVAL;) as of now we could give start addr in the increasing step of 40000000 such as 940000000,980000000,9C0000000. However we cannot give  95000000,990000000,9d0000000. Can we give start address which is not in step of 40000000

2) You were right MIN_MEMORY_BLOCK_SIZE is 1 GB, can we further reduce memory region length to 128 MB or it will be good if we can make it customizable( even smaller then 128 MB).  Like in our system I want to hot-plug memory which is starting from 9e0000000 to 9e3000000, As we can see memory region is dependent on MIN_MEMORY_BLOCK_SIZE in below code.

ret = __add_memory(nid, phys_addr,
                           MIN_MEMORY_BLOCK_SIZE * sections_per_block);


My ask for above is due to non-availability of non -reserved memory in continuous bigger chunk in our system. 

Please let me know your input if we can do the same.

Thanks,
Vikash


-----Original Message-----
From: David Hildenbrand <david@redhat.com> 
Sent: Sunday, January 9, 2022 5:56 PM
To: Ranjan, Vikash <Vikash.Ranjan@harman.com>; linux-mm@kvack.org
Subject: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

[EXTERNAL EMAIL]

Hi

> 
> We are trying to enable memory hot plug for our device's based on 
> arm64, is it possible to reduce memory_block_size to 1 GB or lesser as 
> currently it is 4GB.

... are you sure it's 4GB? I never heard that magic number on arm64 before. The section size is 512 MiB on arm64 with 64k and 128 MiB with 4k.
Vikash--- yes David you were right. 
> 
> If you have any info on updated patch or any other way we can do this, 
> please let us know.
> 
>  After applying attached test patch (for reduction of 
> memory_block_size to 1 GB),we are getting below error. Probe is working fine in case of 4GB.
> 

I think I am missing something important.

#define MIN_MEMORY_BLOCK_SIZE     (1UL << SECTION_SIZE_BITS)

arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 29 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

-> 512 MiB / 128 MiB
arm64 does not override memory_block_size_bytes(), so it matches exactly these numbers.

1) 
--
Thanks,

David / dhildenb

Re: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

[David Hildenbrand]

On 10.01.22 17:07, Ranjan, Vikash wrote:
> Hi David,
> 

Hi,

> You were right , in my source code 
> arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 30 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

Ever since f0b13ee23241 ("arm64/sparsemem: reduce SECTION_SIZE_BITS"),
it should be 512 MiB / 128 MiB. That one should be easy to backport also
to older kernels IIRC.

> 
> I need your input on below.
> 1) currently due to code ( if (phys_addr & ((pages_per_block << PAGE_SHIFT) - 1)) return -EINVAL;) as of now we could give start addr in the increasing step of 40000000 such as 940000000,980000000,9C0000000. However we cannot give  95000000,990000000,9d0000000. Can we give start address which is not in step of 40000000

Right, in your kernel it should be 1 GiB-aligned chunks. Upstream it
should be 512 MiB / 128 MiB-aligned chunks.

Memory blocks always have to be aligned, starting in the middle of one
is impossible. So with a section size of 512 MiB, best you can do is
probing 512 MiB aligned and sized memory blocks. Similarly, with 128 MiB.

> 
> 2) You were right MIN_MEMORY_BLOCK_SIZE is 1 GB, can we further reduce memory region length to 128 MB or it will be good if we can make it customizable( even smaller then 128 MB).  Like in our system I want to hot-plug memory which is starting from 9e0000000 to 9e3000000, As we can see memory region is dependent on MIN_MEMORY_BLOCK_SIZE in below code.

Upstream, with 4k pages, 128 MiB should work. Anything smaller than that
is impossible to hotplug/probe.

There are certain restrictions that impose this limit: for example, the
pageblock size of 512 MiB limits on 64k limits the section size on 64k
to 512 MiB as well.

With 4k, we're using 128 MiB, because it's the smallest possible value
that still allows for having huge pages to store the vmemmap easily: the
memmap of 128 MiB is exactly 2 MiB, corresponding to one huge page on arm64.

We usually don't want smaller memory blocks, because it makes
/sys/devices/system/memory/ explode in size, can degrade performance
(e.g., vmemmap), and might result in other issues (sub-section hot-add
for ZONE_DEVICE)

> 
> ret = __add_memory(nid, phys_addr,
>                            MIN_MEMORY_BLOCK_SIZE * sections_per_block);
> 
> 
> My ask for above is due to non-availability of non -reserved memory in continuous bigger chunk in our system. 

If you can use 4k base pages, with f0b13ee23241 ("arm64/sparsemem:
reduce SECTION_SIZE_BITS") you'll get 128 MiB sections and consequently
128 MiB MIN_MEMORY_BLOCK_SIZE. With that, you can probe 9e0000000 to
9e3000000 by probing 4 individual 128 MiB-sized memory blocks.

-- 
Thanks,

David / dhildenb

RE: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

[Ranjan, Vikash]

Thank You David for your input, let me backport/implement your advice. Will update you on this.

Thanks,
Vikash

-----Original Message-----
From: David Hildenbrand <david@redhat.com> 
Sent: Monday, January 10, 2022 9:52 PM
To: Ranjan, Vikash <Vikash.Ranjan@harman.com>; linux-mm@kvack.org
Subject: Re: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

[EXTERNAL EMAIL]

On 10.01.22 17:07, Ranjan, Vikash wrote:
> Hi David,
> 

Hi,

> You were right , in my source code
> arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 30 
> arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

Ever since f0b13ee23241 ("arm64/sparsemem: reduce SECTION_SIZE_BITS"), it should be 512 MiB / 128 MiB. That one should be easy to backport also to older kernels IIRC.

> 
> I need your input on below.
> 1) currently due to code ( if (phys_addr & ((pages_per_block << 
> PAGE_SHIFT) - 1)) return -EINVAL;) as of now we could give start addr 
> in the increasing step of 40000000 such as 
> 940000000,980000000,9C0000000. However we cannot give  
> 95000000,990000000,9d0000000. Can we give start address which is not 
> in step of 40000000

Right, in your kernel it should be 1 GiB-aligned chunks. Upstream it should be 512 MiB / 128 MiB-aligned chunks.

Memory blocks always have to be aligned, starting in the middle of one is impossible. So with a section size of 512 MiB, best you can do is probing 512 MiB aligned and sized memory blocks. Similarly, with 128 MiB.

> 
> 2) You were right MIN_MEMORY_BLOCK_SIZE is 1 GB, can we further reduce memory region length to 128 MB or it will be good if we can make it customizable( even smaller then 128 MB).  Like in our system I want to hot-plug memory which is starting from 9e0000000 to 9e3000000, As we can see memory region is dependent on MIN_MEMORY_BLOCK_SIZE in below code.

Upstream, with 4k pages, 128 MiB should work. Anything smaller than that is impossible to hotplug/probe.

There are certain restrictions that impose this limit: for example, the pageblock size of 512 MiB limits on 64k limits the section size on 64k to 512 MiB as well.

With 4k, we're using 128 MiB, because it's the smallest possible value that still allows for having huge pages to store the vmemmap easily: the memmap of 128 MiB is exactly 2 MiB, corresponding to one huge page on arm64.

We usually don't want smaller memory blocks, because it makes /sys/devices/system/memory/ explode in size, can degrade performance (e.g., vmemmap), and might result in other issues (sub-section hot-add for ZONE_DEVICE)

> 
> ret = __add_memory(nid, phys_addr,
>                            MIN_MEMORY_BLOCK_SIZE * 
> sections_per_block);
> 
> 
> My ask for above is due to non-availability of non -reserved memory in continuous bigger chunk in our system. 

If you can use 4k base pages, with f0b13ee23241 ("arm64/sparsemem:
reduce SECTION_SIZE_BITS") you'll get 128 MiB sections and consequently
128 MiB MIN_MEMORY_BLOCK_SIZE. With that, you can probe 9e0000000 to
9e3000000 by probing 4 individual 128 MiB-sized memory blocks.

--
Thanks,

David / dhildenb