[RFC] hugetlb: add memory-hotplug notifier to only allocate for online nodes

[RFC] hugetlb: add memory-hotplug notifier to only allocate for online nodes

[Swaraj Gaikwad]

This patch is a RFC on a proposed change to the hugetlb cgroup subsystem’s
css allocation function.

The existing hugetlb_cgroup_css_alloc() uses for_each_node() to allocate
nodeinfo for all nodes, including those which are not online yet
(or never will be). This can waste considerable memory on large-node systems.
The documentation already lists this as a TODO.

Proposed Change:
    Introduce a memory hotplug notifier that listens for MEM_ONLINE
    events. When a node becomes online, we call the same allocation function
    but insted of for_each_node(),using for_each_online_node(). This means
    memory is only allocated for nodes which are online, thus reducing waste.

Feedback Requested:
    - Where in the codebase (which file or section) is it most appropriate to
      implement and register the memory hotplug notifier for this subsystem?
    - Are there best practices or patterns for handling the notifier lifecycle,
      especially for unregistering during cgroup or subsystem teardown?
    - What are the standard methods or tools to test memory hotplug scenarios
      for cgroups? Are there ways to reliably trigger node online/offline events
      in a development environment?
    - Are there existing test cases or utilities in the kernel tree that would help
      to verify correct behavior of this change?
    - Any suggestions for implementation improvements or cleaner API usage?

Thanks,
Swaraj Gaikwad <swarajgaikwad1925@gmail.com>

Signed-off-by: Swaraj Gaikwad <swarajgaikwad1925@gmail.com>

Re: [RFC] hugetlb: add memory-hotplug notifier to only allocate for online nodes

[David Hildenbrand (Red Hat)]

On 06.11.25 09:56, Swaraj Gaikwad wrote:
> This patch is a RFC on a proposed change to the hugetlb cgroup subsystem’s
> css allocation function.
> 
> The existing hugetlb_cgroup_css_alloc() uses for_each_node() to allocate
> nodeinfo for all nodes, including those which are not online yet
> (or never will be). This can waste considerable memory on large-node systems.
> The documentation already lists this as a TODO.

We're talking about the

kzalloc_node(sizeof(struct hugetlb_cgroup_per_node), GFP_KERNEL, node_to_alloc);

$ pahole mm/hugetlb_cgroup.o

struct hugetlb_cgroup_per_node {
         long unsigned int          usage[2];             /*     0    16 */

         /* size: 16, cachelines: 1, members: 1 */
         /* last cacheline: 16 bytes */
};

16 bytes on x86_64. So nobody should care here.

Of course, it depends on HUGE_MAX_HSTATE.

IIRC only HUGE_MAX_HSTATE goes crazy on that with effectively 15 entries.

15*8 ~128 bytes.

So with 1024 nodes we would be allocating 128 KiB.


And given that this is for each cgroup (right?) I assume it can add up.

> 
> Proposed Change:
>      Introduce a memory hotplug notifier that listens for MEM_ONLINE
>      events. When a node becomes online, we call the same allocation function
>      but insted of for_each_node(),using for_each_online_node(). This means
>      memory is only allocated for nodes which are online, thus reducing waste.

We have a NODE_ADDING_FIRST_MEMORY now, I'd assume that is more suitable?

> 
> Feedback Requested:
>      - Where in the codebase (which file or section) is it most appropriate to
>        implement and register the memory hotplug notifier for this subsystem?

I'd assume you would have to register in hugetlb_cgroup_css_alloc() and
free in hugetlb_cgroup_css_free().

>      - Are there best practices or patterns for handling the notifier lifecycle,
>        especially for unregistering during cgroup or subsystem teardown?

Not that I can think of some :)

>      - What are the standard methods or tools to test memory hotplug scenarios
>        for cgroups? Are there ways to reliably trigger node online/offline events
>        in a development environment?

You can use QEMU to hotplug memory (pc-dimm device) to a CPU+memory-less node and
to then remove it again. If you disable automatic memory onlining, you should be able to
trigger this multiple times without any issues.

>      - Are there existing test cases or utilities in the kernel tree that would help
>        to verify correct behavior of this change?

Don't think so.

>      - Any suggestions for implementation improvements or cleaner API usage?

I'd assume you'd want to look into NODE_ADDING_FIRST_MEMORY.

-- 
Cheers

David