Re: high kswapd CPU usage with symmetrical swap in/out pattern with multi-gen LRU

[Yu Zhao <yuzhao@google.com>]

On Wed, Nov 8, 2023 at 10:39 PM Jaroslav Pulchart
<jaroslav.pulchart@gooddata.com> wrote:
>
> >
> > On Wed, Nov 8, 2023 at 12:04 PM Jaroslav Pulchart
> > <jaroslav.pulchart@gooddata.com> wrote:
> > >
> > > >
> > > > Hi Jaroslav,
> > >
> > > Hi Yu Zhao
> > >
> > > thanks for response, see answers inline:
> > >
> > > >
> > > > On Wed, Nov 8, 2023 at 6:35 AM Jaroslav Pulchart
> > > > <jaroslav.pulchart@gooddata.com> wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > I would like to report to you an unpleasant behavior of multi-gen LRU
> > > > > with strange swap in/out usage on my Dell 7525 two socket AMD 74F3
> > > > > system (16numa domains).
> > > >
> > > > Kernel version please?
> > >
> > > 6.5.y, but we saw it sooner as it is in investigation from 23th May
> > > (6.4.y and maybe even the 6.3.y).
> >
> > v6.6 has a few critical fixes for MGLRU, I can backport them to v6.5
> > for you if you run into other problems with v6.6.
> >
>
> I will give it a try using 6.6.y. When it will work we can switch to
> 6.6.y instead of backporting the stuff to 6.5.y.
>
> > > > > Symptoms of my issue are
> > > > >
> > > > > /A/ if mult-gen LRU is enabled
> > > > > 1/ [kswapd3] is consuming 100% CPU
> > > >
> > > > Just thinking out loud: kswapd3 means the fourth node was under memory pressure.
> > > >
> > > > >     top - 15:03:11 up 34 days,  1:51,  2 users,  load average: 23.34,
> > > > > 18.26, 15.01
> > > > >     Tasks: 1226 total,   2 running, 1224 sleeping,   0 stopped,   0 zombie
> > > > >     %Cpu(s): 12.5 us,  4.7 sy,  0.0 ni, 82.1 id,  0.0 wa,  0.4 hi,
> > > > > 0.4 si,  0.0 st
> > > > >     MiB Mem : 1047265.+total,  28382.7 free, 1021308.+used,    767.6 buff/cache
> > > > >     MiB Swap:   8192.0 total,   8187.7 free,      4.2 used.  25956.7 avail Mem
> > > > >     ...
> > > > >         765 root      20   0       0      0      0 R  98.3   0.0
> > > > > 34969:04 kswapd3
> > > > >     ...
> > > > > 2/ swap space usage is low about ~4MB from 8GB as swap in zram (was
> > > > > observed with swap disk as well and cause IO latency issues due to
> > > > > some kind of locking)
> > > > > 3/ swap In/Out is huge and symmetrical ~12MB/s in and ~12MB/s out
> > > > >
> > > > >
> > > > > /B/ if mult-gen LRU is disabled
> > > > > 1/ [kswapd3] is consuming 3%-10% CPU
> > > > >     top - 15:02:49 up 34 days,  1:51,  2 users,  load average: 23.05,
> > > > > 17.77, 14.77
> > > > >     Tasks: 1226 total,   1 running, 1225 sleeping,   0 stopped,   0 zombie
> > > > >     %Cpu(s): 14.7 us,  2.8 sy,  0.0 ni, 81.8 id,  0.0 wa,  0.4 hi,
> > > > > 0.4 si,  0.0 st
> > > > >     MiB Mem : 1047265.+total,  28378.5 free, 1021313.+used,    767.3 buff/cache
> > > > >     MiB Swap:   8192.0 total,   8189.0 free,      3.0 used.  25952.4 avail Mem
> > > > >     ...
> > > > >        765 root      20   0       0      0      0 S   3.6   0.0
> > > > > 34966:46 [kswapd3]
> > > > >     ...
> > > > > 2/ swap space usage is low (4MB)
> > > > > 3/ swap In/Out is huge and symmetrical ~500kB/s in and ~500kB/s out
> > > > >
> > > > > Both situations are wrong as they are using swap in/out extensively,
> > > > > however the multi-gen LRU situation is 10times worse.
> > > >
> > > > From the stats below, node 3 had the lowest free memory. So I think in
> > > > both cases, the reclaim activities were as expected.
> > >
> > > I do not see a reason for the memory pressure and reclaims. This node
> > > has the lowest free memory of all nodes (~302MB free) that is true,
> > > however the swap space usage is just 4MB (still going in and out). So
> > > what can be the reason for that behaviour?
> >
> > The best analogy is that refuel (reclaim) happens before the tank
> > becomes empty, and it happens even sooner when there is a long road
> > ahead (high order allocations).
> >
> > > The workers/application is running in pre-allocated HugePages and the
> > > rest is used for a small set of system services and drivers of
> > > devices. It is static and not growing. The issue persists when I stop
> > > the system services and free the memory.
> >
> > Yes, this helps.
> >  Also could you attach /proc/buddyinfo from the moment
> > you hit the problem?
> >
>
> I can. The problem is continuous, it is 100% of time continuously
> doing in/out and consuming 100% of CPU and locking IO.
>
> The output of /proc/buddyinfo is:
>
> # cat /proc/buddyinfo
> Node 0, zone      DMA      7      2      2      1      1      2      1
>      1      1      2      1
> Node 0, zone    DMA32   4567   3395   1357    846    439    190     93
>     61     43     23      4
> Node 0, zone   Normal     19    190    140    129    136     75     66
>     41      9      1      5
> Node 1, zone   Normal    194   1210   2080   1800    715    255    111
>     56     42     36     55
> Node 2, zone   Normal    204    768   3766   3394   1742    468    185
>    194    238     47     74
> Node 3, zone   Normal   1622   2137   1058    846    388    208     97
>     44     14     42     10

Again, thinking out loud: there is only one zone on node 3, i.e., the
normal zone, and this excludes the problem commit
669281ee7ef731fb5204df9d948669bf32a5e68d ("Multi-gen LRU: fix per-zone
reclaim") fixed in v6.6.

> Node 4, zone   Normal    282    705    623    274    184     90     63
>     41     11      1     28
> Node 5, zone   Normal    505    620   6180   3706   1724   1083    592
>    410    417    168     70
> Node 6, zone   Normal   1120    357   3314   3437   2264    872    606
>    209    215    123    265
> Node 7, zone   Normal    365   5499  12035   7486   3845   1743    635
>    243    309    292     78
> Node 8, zone   Normal    248    740   2280   1094   1225   2087    846
>    308    192     65     55
> Node 9, zone   Normal    356    763   1625    944    740   1920   1174
>    696    217    235    111
> Node 10, zone   Normal    727   1479   7002   6114   2487   1084
> 407    269    157     78     16
> Node 11, zone   Normal    189   3287   9141   5039   2560   1183
> 1247    693    506    252      8
> Node 12, zone   Normal    142    378   1317    466   1512   1568
> 646    359    248    264    228
> Node 13, zone   Normal    444   1977   3173   2625   2105   1493
> 931    600    369    266    230
> Node 14, zone   Normal    376    221    120    360   2721   2378
> 1521    826    442    204     59
> Node 15, zone   Normal   1210    966    922   2046   4128   2904
> 1518    744    352    102     58
>
>
> > > > > Could I ask for any suggestions on how to avoid the kswapd utilization
> > > > > pattern?
> > > >
> > > > The easiest way is to disable NUMA domain so that there would be only
> > > > two nodes with 8x more memory. IOW, you have fewer pools but each pool
> > > > has more memory and therefore they are less likely to become empty.
> > > >
> > > > > There is a free RAM in each numa node for the few MB used in
> > > > > swap:
> > > > >     NUMA stats:
> > > > >     NUMA nodes: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> > > > >     MemTotal: 65048 65486 65486 65486 65486 65486 65486 65469 65486
> > > > > 65486 65486 65486 65486 65486 65486 65424
> > > > >     MemFree: 468 601 1200 302 548 1879 2321 2478 1967 2239 1453 2417
> > > > > 2623 2833 2530 2269
> > > > > the in/out usage does not make sense for me nor the CPU utilization by
> > > > > multi-gen LRU.
> > > >
> > > > My questions:
> > > > 1. Were there any OOM kills with either case?
> > >
> > > There is no OOM. The memory usage is not growing nor the swap space
> > > usage, it is still a few MB there.
> > >
> > > > 2. Was THP enabled?
> > >
> > > Both situations with enabled and with disabled THP.
> >
> > My suspicion is that you packed the node 3 too perfectly :) And that
> > might have triggered a known but currently a low priority problem in
> > MGLRU. I'm attaching a patch for v6.6 and hoping you could verify it
> > for me in case v6.6 by itself still has the problem?
> >
>
> I would not focus just to node3, we had issues on different servers
> with node0 and node2 both in parallel, but mostly it is the node3.
>
> How our setup looks like:
> * each node has 64GB of RAM,
> * 61GB from it is in 1GB Huge Pages,
> * rest 3GB is used by host system
>
> There are running kvm VMs vCPUs pinned to the NUMA domains and using
> the Huge Pages (topology is exposed to VMs, no-overcommit, no-shared
> cpus), the qemu-kvm threads are pinned to the same numa domain as the
> vCPUs. System services are not pinned, I'm not sure why the node3 is
> used at most as the vms are balanced and the host's system services
> can move between domains.
>
> > > > MGLRU might have spent the extra CPU cycles just to void OOM kills or
> > > > produce more THPs.
> > > >
> > > > If disabling the NUMA domain isn't an option, I'd recommend:
> > >
> > > Disabling numa is not an option. However we are now testing a setup
> > > with -1GB in HugePages per each numa.
> > >
> > > > 1. Try the latest kernel (6.6.1) if you haven't.
> > >
> > > Not yet, the 6.6.1 was released today.
> > >
> > > > 2. Disable THP if it was enabled, to verify whether it has an impact.
> > >
> > > I try disabling THP without any effect.
> >
> > Gochat. Please try the patch with MGLRU and let me know. Thanks!
> >
> > (Also CC Charan @ Qualcomm who initially reported the problem that
> > ended up with the attached patch.)
>
> I can try it. Will let you know.

Great, thanks!