Re: CXL Boot to Bash - Section 3: Memory (block) Hotplug

[Gregory Price <gourry@gourry.net>]

On Wed, Feb 19, 2025 at 09:53:07AM +0100, David Hildenbrand wrote:
... snip ...
> 
> Would the following make it better or worse?
> 

> The question is whether you want that memory to be hidden from MemTotal
> (carveout?) or treated just like allocated memory (allocation?).
> 
> If you treat the memmap as "just a memory allocation after early boot"
> and "memap_on_memory" telling you to allocate that memory from the
> hotplugged memory instead of the buddy, then "carveout"
> might be more of an internal implementation detail to achieve that memory
> allocation.
> 

Probably this is fine either way, but I'll poke at the patch you
provided and let you know.

> Right, it only currently works with ZONE_NORMAL, because 1 GiB pages are
> considered unmovable in practice (try reliably finding a 1 GiB area to
> migrate the memory to during memory unplug ... when these hugetlb things are
> unswappable etc.).
> 
> I documented it under https://www.kernel.org/doc/html/latest/admin-guide/mm/memory-hotplug.html
> 
> "Gigantic pages are unmovable, resulting in user space consuming a lot of unmovable memory."
>

Ah, I'm so used to seeing "1GiB Huge Pages" I missed that parts of the
kernel refer to them as "gigantic".  Completely missed this line in the
docs.  So a subtle choice being made by onlining memory into zone
movable is the exclusion of this memory region from being used for
gigantic pages (for now, assuming it never changes).

This is good to know.

> > I appreciate the clarifications here, sorry for the incorrect info and
> > the increasing confusing.
> 
> No worries. If you have ideas on what to improve in the memory hotplug
> docs, please let me know!
> 

I think that clears up most of my misconceptions.

The end-goal of this series is essentially 2-4 "basic choices" for
onlining CXL memory - the implicit decisions being made - while
identifying some ambiguity.  

For example: driver-setup into ZONE_MOVABLE w/o memmap_on_memory means
  1) ZONE_NORMAL page-struct use
  2) no gigantic page support
  3) limited kernel allocation support
  4) changeable configuration without reboot (in theory)
  5) Additioanl ras capabilities.

Thanks again for you patience and help as I work through this.
~Gregory