Re: [PATCH 0/4] Support dynamic (de)configuration of memory

[David Hildenbrand <david@redhat.com>]

On 26.09.25 15:15, Sumanth Korikkar wrote:
> Hi,

Hi,

> 
> Patchset provides a new interface for dynamic configuration and
> deconfiguration of hotplug memory on s390, allowing with/without
> memmap_on_memory support. It is a follow up on the discussion with David
> when introducing memmap_on_memory support for s390 and support dynamic
> (de)configuration of memory:
> https://lore.kernel.org/all/ee492da8-74b4-4a97-8b24-73e07257f01d@redhat.com/
> https://lore.kernel.org/all/20241202082732.3959803-1-sumanthk@linux.ibm.com/
> 
> The original motivation for introducing memmap_on_memory on s390 was to
> avoid using online memory to store struct pages metadata, particularly
> for standby memory blocks. This became critical in cases where there was
> an imbalance between standby and online memory, potentially leading to
> boot failures due to insufficient memory for metadata allocation.
> 
> To address this, memmap_on_memory was utilized on s390. However, in its
> current form, it adds struct pages metadata at the start of each memory
> block at the time of addition (only standby memory), and this
> configuration is static. It cannot be changed at runtime  (When the user
> needs continuous physical memory).
> 
> Inorder to provide more flexibility to the user and overcome the above
> limitation, add an option to dynamically configure and deconfigure
> hotpluggable memory block with/without memmap_on_memory.

This will cleanly add/remove the memory, including the directmap and 
other tracking data, so I like it.

> 
> With the new interface, s390 will not add all possible hotplug memory in
> advance, like before, to make it visible in sysfs for online/offline
> actions. Instead, before memory block can be set online, it has to be
> configured via a new interface in /sys/firmware/memory/memoryX/config,
> which makes s390 similar to others.  i.e. Adding of hotpluggable memory is
> controlled by the user instead of adding it at boottime.

Before I dig into the details, will onlining/offling still trigger 
hypervisor action, or does that now really happen when memory is 
added/removed?

That would be really nice, because it would remove the whole need for 
"standby" memory, and having to treat hotplugged memory differently 
under LPAR/z/VM than anywhere else (-> keep it offline).

> 
> s390 kernel sysfs interface to configure/deconfigure memory with
> memmap_on_memory (with upcoming lsmem changes):
>      
> * Initial memory layout:
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE                 SIZE   STATE BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x7fffffff   2G  online 0-15  yes        no
> 0x80000000-0xffffffff   2G offline 16-31 no         yes

Could we instead modify "STATE" to reflect that it is "not added" / "not 
configured" / "disabled" etc?

Like

lsmem -o RANGE,SIZE,STATE,BLOCK,MEMMAP_ON_MEMORY
RANGE                 SIZE    STATE BLOCK
0x00000000-0x7fffffff   2G   online 0-15
0x80000000-0xffffffff   2G disabled 16-31

Or is that an attempt to maintain backwards compatibility?

> 
> * Configure memory
> echo 1 > /sys/firmware/memory/memory16/config

The granularity here is also memory_block_size_bytes(), correct?

> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE                  SIZE  STATE   BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x7fffffff    2G  online  0-15  yes        no
> 0x80000000-0x87ffffff  128M offline    16  yes        yes
> 0x88000000-0xffffffff  1.9G offline 17-31  no         yes
> 
> * Deconfigure memory
> echo 0 > /sys/firmware/memory/memory16/config
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE                 SIZE   STATE BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x7fffffff   2G  online 0-15  yes        no
> 0x80000000-0xffffffff   2G offline 16-31 no         yes
> 
> * Enable memmap_on_memory and online it.
> (Deconfigure first)
> echo 0 > /sys/devices/system/memory/memory5/online
> echo 0 > /sys/firmware/memory/memory5/config
> 
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE                  SIZE  STATE  BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x27ffffff  640M  online 0-4   yes        no
> 0x28000000-0x2fffffff  128M offline 5     no         no
> 0x30000000-0x7fffffff  1.3G  online 6-15  yes        no
> 0x80000000-0xffffffff    2G offline 16-31 no         yes
> 
> (Enable memmap_on_memory and online it)
> echo 1 > /sys/firmware/memory/memory5/memmap_on_memory
> echo 1 > /sys/firmware/memory/memory5/config
> echo 1 > /sys/devices/system/memory/memory5/online

I guess the use for memmap_on_memory would now be limited to making 
hotplug more likely to succeed in OOM scenarios.

> 
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE                  SIZE  STATE   BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x27ffffff  640M  online  0-4   yes        no
> 0x28000000-0x2fffffff  128M  online  5     yes        yes
> 0x30000000-0x7fffffff  1.3G  online  6-15  yes        no
> 0x80000000-0xffffffff    2G  offline 16-31 no         yes
> 
> * Disable memmap_on_memory and online it.
> (Deconfigure first)
> echo 0 > /sys/devices/system/memory/memory5/online
> echo 0 > /sys/firmware/memory/memory5/config
> 
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE                  SIZE  STATE  BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x27ffffff  640M  online 0-4   yes        no
> 0x28000000-0x2fffffff  128M offline 5     no         yes
> 0x30000000-0x7fffffff  1.3G  online 6-15  yes        no
> 0x80000000-0xffffffff    2G offline 16-31 no         yes
> 
> (Disable memmap_on_memory and online it)
> echo 0 > /sys/firmware/memory/memory5/memmap_on_memory
> echo 1 > /sys/firmware/memory/memory5/config
> echo 1 > /sys/devices/system/memory/memory5/online
> 
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE                  SIZE  STATE   BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x7fffffff  2G    online  0-15  yes        no
> 0x80000000-0xffffffff  2G    offline 16-31 no         yes
> 
> * Userspace changes:
> lsmem/chmem tool is also changed to use the new interface. I will send
> it to util-linux soon.
> 
> Patch 1 adds support for removal of boot-allocated memory blocks.
> 
> Patch 2 provides option to dynamically configure and deconfigure memory
> with/without memmap_on_memory.
> 
> Patch 3 removes MHP_OFFLINE_INACCESSIBLE from s390. The mhp flag was
> used to mark memory as not accessible until memory hotplug online phase
> begins.  However, with patch 2, it is no longer essential. Memory can be
> brought to accessible state before adding memory, as the memory is added
> during runttime now instead of boottime.

Nice.

> 
> Patch 4 removes the MEM_PREPARE_ONLINE/MEM_FINISH_OFFLINE notifiers. It
> is no longer needed.  Memory can be brought to accessible state before
> adding memory now, with runtime (de)configuration of memory.

Nice.


-- 
Cheers

David / dhildenb