Re: high kswapd CPU usage with symmetrical swap in/out pattern with multi-gen LRU

[Yu Zhao <yuzhao@google.com>]

Hi Jaroslav,

On Wed, Nov 8, 2023 at 6:35 AM Jaroslav Pulchart
<jaroslav.pulchart@gooddata.com> wrote:
>
> Hello,
>
> I would like to report to you an unpleasant behavior of multi-gen LRU
> with strange swap in/out usage on my Dell 7525 two socket AMD 74F3
> system (16numa domains).

Kernel version please?

> Symptoms of my issue are
>
> /A/ if mult-gen LRU is enabled
> 1/ [kswapd3] is consuming 100% CPU

Just thinking out loud: kswapd3 means the fourth node was under memory pressure.

>     top - 15:03:11 up 34 days,  1:51,  2 users,  load average: 23.34,
> 18.26, 15.01
>     Tasks: 1226 total,   2 running, 1224 sleeping,   0 stopped,   0 zombie
>     %Cpu(s): 12.5 us,  4.7 sy,  0.0 ni, 82.1 id,  0.0 wa,  0.4 hi,
> 0.4 si,  0.0 st
>     MiB Mem : 1047265.+total,  28382.7 free, 1021308.+used,    767.6 buff/cache
>     MiB Swap:   8192.0 total,   8187.7 free,      4.2 used.  25956.7 avail Mem
>     ...
>         765 root      20   0       0      0      0 R  98.3   0.0
> 34969:04 kswapd3
>     ...
> 2/ swap space usage is low about ~4MB from 8GB as swap in zram (was
> observed with swap disk as well and cause IO latency issues due to
> some kind of locking)
> 3/ swap In/Out is huge and symmetrical ~12MB/s in and ~12MB/s out
>
>
> /B/ if mult-gen LRU is disabled
> 1/ [kswapd3] is consuming 3%-10% CPU
>     top - 15:02:49 up 34 days,  1:51,  2 users,  load average: 23.05,
> 17.77, 14.77
>     Tasks: 1226 total,   1 running, 1225 sleeping,   0 stopped,   0 zombie
>     %Cpu(s): 14.7 us,  2.8 sy,  0.0 ni, 81.8 id,  0.0 wa,  0.4 hi,
> 0.4 si,  0.0 st
>     MiB Mem : 1047265.+total,  28378.5 free, 1021313.+used,    767.3 buff/cache
>     MiB Swap:   8192.0 total,   8189.0 free,      3.0 used.  25952.4 avail Mem
>     ...
>        765 root      20   0       0      0      0 S   3.6   0.0
> 34966:46 [kswapd3]
>     ...
> 2/ swap space usage is low (4MB)
> 3/ swap In/Out is huge and symmetrical ~500kB/s in and ~500kB/s out
>
> Both situations are wrong as they are using swap in/out extensively,
> however the multi-gen LRU situation is 10times worse.

From the stats below, node 3 had the lowest free memory. So I think in
both cases, the reclaim activities were as expected.

> Could I ask for any suggestions on how to avoid the kswapd utilization
> pattern?

The easiest way is to disable NUMA domain so that there would be only
two nodes with 8x more memory. IOW, you have fewer pools but each pool
has more memory and therefore they are less likely to become empty.

> There is a free RAM in each numa node for the few MB used in
> swap:
>     NUMA stats:
>     NUMA nodes: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
>     MemTotal: 65048 65486 65486 65486 65486 65486 65486 65469 65486
> 65486 65486 65486 65486 65486 65486 65424
>     MemFree: 468 601 1200 302 548 1879 2321 2478 1967 2239 1453 2417
> 2623 2833 2530 2269
> the in/out usage does not make sense for me nor the CPU utilization by
> multi-gen LRU.

My questions:
1. Were there any OOM kills with either case?
2. Was THP enabled?
MGLRU might have spent the extra CPU cycles just to void OOM kills or
produce more THPs.

If disabling the NUMA domain isn't an option, I'd recommend:
1. Try the latest kernel (6.6.1) if you haven't.
2. Disable THP if it was enabled, to verify whether it has an impact.

Thanks.