Re: Percpu allocator: CPU hotplug support

[Pratik Sampat <psampat@linux.ibm.com>]

Hello

On 22/04/21 6:14 am, Alexey Makhalov wrote:
> Current implementation of percpu allocator uses total possible number of CPUs (nr_cpu_ids) to
> get number of units to allocate per chunk. Every alloc_percpu() request of N bytes will allocate
> N*nr_cpu_ids bytes even if the number of present CPUs is much less. Percpu allocator grows by
> number of chunks keeping number of units per chunk constant. This is done in that way to
> simplify CPU hotplug/remove to have per-cpu area preallocated.
>
> Problem: This behavior can lead to inefficient memory usage for big server machines and VMs,
> where nr_cpu_ids is huge.
>
> Example from my experiment:
> 2 vCPU VM with hotplug support (up to 128):
> [    0.105989] smpboot: Allowing 128 CPUs, 126 hotplug CPUs
> By creating huge amount of active or/and dying memory cgroups, I can generate active percpu
> allocations of 100 MB (per single CPU) including fragmentation overhead. But in that case total
> percpu memory consumption (reported in /proc/meminfo) will be 12.8 GB. BTW, chunks are
> filled by ~75% in my experiment, so fragmentation is not a concern.
> Out of 12.8 GB:
>   - 0.2 GB are actually used by present vCPUs, and
>   - 12.6 GB are "wasted"!
>
> I've seen production VMs consuming 16-20 GB of memory by Percpu. Roman reported 100 GB.
> There are solutions to reduce "wasted" memory overhead such as: disabling CPU hotplug; reducing
> number of maximum CPUs reported by hypervisor or/and firmware; using possible_cpus= kernel
> parameter. But it won't eliminate fundamental issue with "wasted" memory.
>
> Suggestion: To support percpu chunks scaling by number of units there. To allocate/deallocate new
> units for existing chunks on CPU hotplug/remove event.
>
> Any thoughts? Thanks! --Alexey
>
>
I've run some traces around memory cgroups to determine memory consumption by
the Percpu allocator and the major contributers to these allocations by either
creating an empty memory cgroup or an empty container.

There are 4 memcg percpu allocation charges I see when I create a cgroup
attached to a memory controller. They seem to belong to mm/memcontrol.c's
lruvec_stat and vmstats.

I've run this experiment in 2 configurations on a POWER9 box
1. cpus=16 (present), maxcpus=16   (possible)
2. cpus=16 (present), maxcpus=1024 (possible)

On system boot,
Maxcpus    Sum percpu charges(MB)
16         2.4979
1024       159.86

0 MB container setup (empty parallel container setup that just spawns and spins)
Maxcpus    per container avg(MB)
16         0.0398
1024       2.5507

The difference in cgroup charges, although in absolute numbers is quite small,
wastes memory proportionally when the cgroup or the container setup is scaled
to say 10,000 containers.

If memory cgroup is the point of focus then would it make sense to attempt to
optimize only those callers to be hotplug aware than to attempt to optimize the
whole percpu allocator?

Thanks,
Pratik