Re: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

[David Hildenbrand <david@redhat.com>]

On 10.01.22 17:07, Ranjan, Vikash wrote:
> Hi David,
> 

Hi,

> You were right , in my source code 
> arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 30 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

Ever since f0b13ee23241 ("arm64/sparsemem: reduce SECTION_SIZE_BITS"),
it should be 512 MiB / 128 MiB. That one should be easy to backport also
to older kernels IIRC.

> 
> I need your input on below.
> 1) currently due to code ( if (phys_addr & ((pages_per_block << PAGE_SHIFT) - 1)) return -EINVAL;) as of now we could give start addr in the increasing step of 40000000 such as 940000000,980000000,9C0000000. However we cannot give  95000000,990000000,9d0000000. Can we give start address which is not in step of 40000000

Right, in your kernel it should be 1 GiB-aligned chunks. Upstream it
should be 512 MiB / 128 MiB-aligned chunks.

Memory blocks always have to be aligned, starting in the middle of one
is impossible. So with a section size of 512 MiB, best you can do is
probing 512 MiB aligned and sized memory blocks. Similarly, with 128 MiB.

> 
> 2) You were right MIN_MEMORY_BLOCK_SIZE is 1 GB, can we further reduce memory region length to 128 MB or it will be good if we can make it customizable( even smaller then 128 MB).  Like in our system I want to hot-plug memory which is starting from 9e0000000 to 9e3000000, As we can see memory region is dependent on MIN_MEMORY_BLOCK_SIZE in below code.

Upstream, with 4k pages, 128 MiB should work. Anything smaller than that
is impossible to hotplug/probe.

There are certain restrictions that impose this limit: for example, the
pageblock size of 512 MiB limits on 64k limits the section size on 64k
to 512 MiB as well.

With 4k, we're using 128 MiB, because it's the smallest possible value
that still allows for having huge pages to store the vmemmap easily: the
memmap of 128 MiB is exactly 2 MiB, corresponding to one huge page on arm64.

We usually don't want smaller memory blocks, because it makes
/sys/devices/system/memory/ explode in size, can degrade performance
(e.g., vmemmap), and might result in other issues (sub-section hot-add
for ZONE_DEVICE)

> 
> ret = __add_memory(nid, phys_addr,
>                            MIN_MEMORY_BLOCK_SIZE * sections_per_block);
> 
> 
> My ask for above is due to non-availability of non -reserved memory in continuous bigger chunk in our system. 

If you can use 4k base pages, with f0b13ee23241 ("arm64/sparsemem:
reduce SECTION_SIZE_BITS") you'll get 128 MiB sections and consequently
128 MiB MIN_MEMORY_BLOCK_SIZE. With that, you can probe 9e0000000 to
9e3000000 by probing 4 individual 128 MiB-sized memory blocks.

-- 
Thanks,

David / dhildenb