RE: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

["Ranjan, Vikash" <Vikash.Ranjan@harman.com>]

Thank You David for your input, let me backport/implement your advice. Will update you on this.

Thanks,
Vikash

-----Original Message-----
From: David Hildenbrand <david@redhat.com> 
Sent: Monday, January 10, 2022 9:52 PM
To: Ranjan, Vikash <Vikash.Ranjan@harman.com>; linux-mm@kvack.org
Subject: Re: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

[EXTERNAL EMAIL]

On 10.01.22 17:07, Ranjan, Vikash wrote:
> Hi David,
> 

Hi,

> You were right , in my source code
> arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 30 
> arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

Ever since f0b13ee23241 ("arm64/sparsemem: reduce SECTION_SIZE_BITS"), it should be 512 MiB / 128 MiB. That one should be easy to backport also to older kernels IIRC.

> 
> I need your input on below.
> 1) currently due to code ( if (phys_addr & ((pages_per_block << 
> PAGE_SHIFT) - 1)) return -EINVAL;) as of now we could give start addr 
> in the increasing step of 40000000 such as 
> 940000000,980000000,9C0000000. However we cannot give  
> 95000000,990000000,9d0000000. Can we give start address which is not 
> in step of 40000000

Right, in your kernel it should be 1 GiB-aligned chunks. Upstream it should be 512 MiB / 128 MiB-aligned chunks.

Memory blocks always have to be aligned, starting in the middle of one is impossible. So with a section size of 512 MiB, best you can do is probing 512 MiB aligned and sized memory blocks. Similarly, with 128 MiB.

> 
> 2) You were right MIN_MEMORY_BLOCK_SIZE is 1 GB, can we further reduce memory region length to 128 MB or it will be good if we can make it customizable( even smaller then 128 MB).  Like in our system I want to hot-plug memory which is starting from 9e0000000 to 9e3000000, As we can see memory region is dependent on MIN_MEMORY_BLOCK_SIZE in below code.

Upstream, with 4k pages, 128 MiB should work. Anything smaller than that is impossible to hotplug/probe.

There are certain restrictions that impose this limit: for example, the pageblock size of 512 MiB limits on 64k limits the section size on 64k to 512 MiB as well.

With 4k, we're using 128 MiB, because it's the smallest possible value that still allows for having huge pages to store the vmemmap easily: the memmap of 128 MiB is exactly 2 MiB, corresponding to one huge page on arm64.

We usually don't want smaller memory blocks, because it makes /sys/devices/system/memory/ explode in size, can degrade performance (e.g., vmemmap), and might result in other issues (sub-section hot-add for ZONE_DEVICE)

> 
> ret = __add_memory(nid, phys_addr,
>                            MIN_MEMORY_BLOCK_SIZE * 
> sections_per_block);
> 
> 
> My ask for above is due to non-availability of non -reserved memory in continuous bigger chunk in our system. 

If you can use 4k base pages, with f0b13ee23241 ("arm64/sparsemem:
reduce SECTION_SIZE_BITS") you'll get 128 MiB sections and consequently
128 MiB MIN_MEMORY_BLOCK_SIZE. With that, you can probe 9e0000000 to
9e3000000 by probing 4 individual 128 MiB-sized memory blocks.

--
Thanks,

David / dhildenb