Re: high kswapd CPU usage with symmetrical swap in/out pattern with multi-gen LRU

[Jaroslav Pulchart <jaroslav.pulchart@gooddata.com>]

>
> On Wed, Nov 8, 2023 at 12:04 PM Jaroslav Pulchart
> <jaroslav.pulchart@gooddata.com> wrote:
> >
> > >
> > > Hi Jaroslav,
> >
> > Hi Yu Zhao
> >
> > thanks for response, see answers inline:
> >
> > >
> > > On Wed, Nov 8, 2023 at 6:35 AM Jaroslav Pulchart
> > > <jaroslav.pulchart@gooddata.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I would like to report to you an unpleasant behavior of multi-gen LRU
> > > > with strange swap in/out usage on my Dell 7525 two socket AMD 74F3
> > > > system (16numa domains).
> > >
> > > Kernel version please?
> >
> > 6.5.y, but we saw it sooner as it is in investigation from 23th May
> > (6.4.y and maybe even the 6.3.y).
>
> v6.6 has a few critical fixes for MGLRU, I can backport them to v6.5
> for you if you run into other problems with v6.6.
>

I will give it a try using 6.6.y. When it will work we can switch to
6.6.y instead of backporting the stuff to 6.5.y.

> > > > Symptoms of my issue are
> > > >
> > > > /A/ if mult-gen LRU is enabled
> > > > 1/ [kswapd3] is consuming 100% CPU
> > >
> > > Just thinking out loud: kswapd3 means the fourth node was under memory pressure.
> > >
> > > >     top - 15:03:11 up 34 days,  1:51,  2 users,  load average: 23.34,
> > > > 18.26, 15.01
> > > >     Tasks: 1226 total,   2 running, 1224 sleeping,   0 stopped,   0 zombie
> > > >     %Cpu(s): 12.5 us,  4.7 sy,  0.0 ni, 82.1 id,  0.0 wa,  0.4 hi,
> > > > 0.4 si,  0.0 st
> > > >     MiB Mem : 1047265.+total,  28382.7 free, 1021308.+used,    767.6 buff/cache
> > > >     MiB Swap:   8192.0 total,   8187.7 free,      4.2 used.  25956.7 avail Mem
> > > >     ...
> > > >         765 root      20   0       0      0      0 R  98.3   0.0
> > > > 34969:04 kswapd3
> > > >     ...
> > > > 2/ swap space usage is low about ~4MB from 8GB as swap in zram (was
> > > > observed with swap disk as well and cause IO latency issues due to
> > > > some kind of locking)
> > > > 3/ swap In/Out is huge and symmetrical ~12MB/s in and ~12MB/s out
> > > >
> > > >
> > > > /B/ if mult-gen LRU is disabled
> > > > 1/ [kswapd3] is consuming 3%-10% CPU
> > > >     top - 15:02:49 up 34 days,  1:51,  2 users,  load average: 23.05,
> > > > 17.77, 14.77
> > > >     Tasks: 1226 total,   1 running, 1225 sleeping,   0 stopped,   0 zombie
> > > >     %Cpu(s): 14.7 us,  2.8 sy,  0.0 ni, 81.8 id,  0.0 wa,  0.4 hi,
> > > > 0.4 si,  0.0 st
> > > >     MiB Mem : 1047265.+total,  28378.5 free, 1021313.+used,    767.3 buff/cache
> > > >     MiB Swap:   8192.0 total,   8189.0 free,      3.0 used.  25952.4 avail Mem
> > > >     ...
> > > >        765 root      20   0       0      0      0 S   3.6   0.0
> > > > 34966:46 [kswapd3]
> > > >     ...
> > > > 2/ swap space usage is low (4MB)
> > > > 3/ swap In/Out is huge and symmetrical ~500kB/s in and ~500kB/s out
> > > >
> > > > Both situations are wrong as they are using swap in/out extensively,
> > > > however the multi-gen LRU situation is 10times worse.
> > >
> > > From the stats below, node 3 had the lowest free memory. So I think in
> > > both cases, the reclaim activities were as expected.
> >
> > I do not see a reason for the memory pressure and reclaims. This node
> > has the lowest free memory of all nodes (~302MB free) that is true,
> > however the swap space usage is just 4MB (still going in and out). So
> > what can be the reason for that behaviour?
>
> The best analogy is that refuel (reclaim) happens before the tank
> becomes empty, and it happens even sooner when there is a long road
> ahead (high order allocations).
>
> > The workers/application is running in pre-allocated HugePages and the
> > rest is used for a small set of system services and drivers of
> > devices. It is static and not growing. The issue persists when I stop
> > the system services and free the memory.
>
> Yes, this helps.
>  Also could you attach /proc/buddyinfo from the moment
> you hit the problem?
>

I can. The problem is continuous, it is 100% of time continuously
doing in/out and consuming 100% of CPU and locking IO.

The output of /proc/buddyinfo is:

# cat /proc/buddyinfo
Node 0, zone      DMA      7      2      2      1      1      2      1
     1      1      2      1
Node 0, zone    DMA32   4567   3395   1357    846    439    190     93
    61     43     23      4
Node 0, zone   Normal     19    190    140    129    136     75     66
    41      9      1      5
Node 1, zone   Normal    194   1210   2080   1800    715    255    111
    56     42     36     55
Node 2, zone   Normal    204    768   3766   3394   1742    468    185
   194    238     47     74
Node 3, zone   Normal   1622   2137   1058    846    388    208     97
    44     14     42     10
Node 4, zone   Normal    282    705    623    274    184     90     63
    41     11      1     28
Node 5, zone   Normal    505    620   6180   3706   1724   1083    592
   410    417    168     70
Node 6, zone   Normal   1120    357   3314   3437   2264    872    606
   209    215    123    265
Node 7, zone   Normal    365   5499  12035   7486   3845   1743    635
   243    309    292     78
Node 8, zone   Normal    248    740   2280   1094   1225   2087    846
   308    192     65     55
Node 9, zone   Normal    356    763   1625    944    740   1920   1174
   696    217    235    111
Node 10, zone   Normal    727   1479   7002   6114   2487   1084
407    269    157     78     16
Node 11, zone   Normal    189   3287   9141   5039   2560   1183
1247    693    506    252      8
Node 12, zone   Normal    142    378   1317    466   1512   1568
646    359    248    264    228
Node 13, zone   Normal    444   1977   3173   2625   2105   1493
931    600    369    266    230
Node 14, zone   Normal    376    221    120    360   2721   2378
1521    826    442    204     59
Node 15, zone   Normal   1210    966    922   2046   4128   2904
1518    744    352    102     58


> > > > Could I ask for any suggestions on how to avoid the kswapd utilization
> > > > pattern?
> > >
> > > The easiest way is to disable NUMA domain so that there would be only
> > > two nodes with 8x more memory. IOW, you have fewer pools but each pool
> > > has more memory and therefore they are less likely to become empty.
> > >
> > > > There is a free RAM in each numa node for the few MB used in
> > > > swap:
> > > >     NUMA stats:
> > > >     NUMA nodes: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> > > >     MemTotal: 65048 65486 65486 65486 65486 65486 65486 65469 65486
> > > > 65486 65486 65486 65486 65486 65486 65424
> > > >     MemFree: 468 601 1200 302 548 1879 2321 2478 1967 2239 1453 2417
> > > > 2623 2833 2530 2269
> > > > the in/out usage does not make sense for me nor the CPU utilization by
> > > > multi-gen LRU.
> > >
> > > My questions:
> > > 1. Were there any OOM kills with either case?
> >
> > There is no OOM. The memory usage is not growing nor the swap space
> > usage, it is still a few MB there.
> >
> > > 2. Was THP enabled?
> >
> > Both situations with enabled and with disabled THP.
>
> My suspicion is that you packed the node 3 too perfectly :) And that
> might have triggered a known but currently a low priority problem in
> MGLRU. I'm attaching a patch for v6.6 and hoping you could verify it
> for me in case v6.6 by itself still has the problem?
>

I would not focus just to node3, we had issues on different servers
with node0 and node2 both in parallel, but mostly it is the node3.

How our setup looks like:
* each node has 64GB of RAM,
* 61GB from it is in 1GB Huge Pages,
* rest 3GB is used by host system

There are running kvm VMs vCPUs pinned to the NUMA domains and using
the Huge Pages (topology is exposed to VMs, no-overcommit, no-shared
cpus), the qemu-kvm threads are pinned to the same numa domain as the
vCPUs. System services are not pinned, I'm not sure why the node3 is
used at most as the vms are balanced and the host's system services
can move between domains.

> > > MGLRU might have spent the extra CPU cycles just to void OOM kills or
> > > produce more THPs.
> > >
> > > If disabling the NUMA domain isn't an option, I'd recommend:
> >
> > Disabling numa is not an option. However we are now testing a setup
> > with -1GB in HugePages per each numa.
> >
> > > 1. Try the latest kernel (6.6.1) if you haven't.
> >
> > Not yet, the 6.6.1 was released today.
> >
> > > 2. Disable THP if it was enabled, to verify whether it has an impact.
> >
> > I try disabling THP without any effect.
>
> Gochat. Please try the patch with MGLRU and let me know. Thanks!
>
> (Also CC Charan @ Qualcomm who initially reported the problem that
> ended up with the attached patch.)

I can try it. Will let you know.