Re: [PATCH v1] mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaim

[Shakeel Butt <shakeel.butt@linux.dev>]

Hi Jiayuan,

Sorry for late reply due to holidays/break. I will still be slow to
respond this week but will be fully back after one more week. Anyways,
let me respond below.

On Tue, Dec 23, 2025 at 08:22:43AM +0000, Jiayuan Chen wrote:
> December 23, 2025 at 14:11, "Shakeel Butt" <shakeel.butt@linux.dev mailto:shakeel.butt@linux.dev?to=%22Shakeel%20Butt%22%20%3Cshakeel.butt%40linux.dev%3E > wrote:
> 
> 
> > 
> > On Tue, Dec 23, 2025 at 01:42:37AM +0000, Jiayuan Chen wrote:
> > 
> > > 
> > > December 23, 2025 at 05:15, "Shakeel Butt" <shakeel.butt@linux.dev mailto:shakeel.butt@linux.dev?to=%22Shakeel%20Butt%22%20%3Cshakeel.butt%40linux.dev%3E > wrote:
> > > 
> > [...]
> > 
> > > 
> > > > 
> > >  I don't think kswapd is an issue here. The system is out of memory and
> > >  most of the memory is unreclaimable. Either change the workload to use
> > >  less memory or enable swap (or zswap) to have more reclaimable memory.
> > >  
> > >  
> > >  Hi,
> > >  Thanks for looking into this.
> > >  
> > >  Sorry, I didn't describe the scenario clearly enough in the original patch. Let me clarify:
> > >  
> > >  This is a multi-NUMA system where the memory pressure is not global but node-local. The key observation is:
> > >  
> > >  Node 0: Under memory pressure, most memory is anonymous (unreclaimable without swap)
> > >  Node 1: Has plenty of reclaimable memory (~60GB file cache out of 125GB total)
> > > 
> > Thanks and now the situation is much more clear. IIUC you are running
> > multiple workloads (pods) on the system. How is the memcg limits
> > configured for these workloads. You mentioned memory.high, what about
> 
> Thanks for the questions. We have pods configured with memory.high and pods configured with memory.max.
> 
> Actually, memory.max itself causes heavy I/O issues for us, because it keeps trying to reclaim hot
> pages within the cgroup aggressively without killing the process. 
> 
> So we configured some pods with memory.high instead, since it performs reclaim in resume_user_mode_work,
> which somewhat throttles the memory allocation of user processes.
> 
> > memory.max? Also are you using cpusets to limit the pods to individual
> > nodes (cpu & memory) or they can run on any node?
> 
> Yes, we have cpusets(only cpuset.cpus not cpuset.mems) configured for our cgroups, binding
> them to specific NUMA nodes. But I don't think this is directly related to the issue - the
> problem can occur with or without cpusets. Even without cpusets.cpus, the kernel prefers
> to allocate memory from the node where the process is running, so if a process happens to
> run on a CPU belonging to Node 0, the behavior would be similar.

Are you limiting (using cpuset.cpus) the workloads to single respective
nodes or the individual workloads can still run on multiple nodes? For
example do you have a workload which can run on both (or more) nodes?

> 
> 
> > 
> > Overall I still think it is unbalanced numa nodes in terms of memory and
> > may for cpu as well. Anyways let's talk about kswapd.
> > > 
> > > Node 0's kswapd runs continuously but cannot reclaim anything
> > >  Direct reclaim succeeds by reclaiming from Node 1
> > >  Direct reclaim resets kswapd_failures,
> > > 
> > So successful reclaim on one node does not reset kswapd_failures on
> > other node. The kernel reclaims each node one by one, so if Node 0
> > direct reclaim was successfull only then kernel allows to reset the
> > kswapd_failures of Node 0 to be reset.
> 
> Let me dig deeper into this.
> 
> When either memory.max or memory.high is reached, direct reclaim is
> triggered. The memory being reclaimed depends on the CPU where the
> process is running.
> 
> When the problem occurred, we had workloads continuously hitting 
> memory.max and workloads continuously hitting memory.high:
> 
> reclaim_high    ->   -> try_to_free_mem_cgroup_pages
>                    ^      do_try_to_free_pages(zone of current node)
>                    |         shrink_zones()
> try_charge_memcg  -              shrink_node()
>                                      kswapd_failures = 0
> 
> Although the pages are hot, if we scan aggressively enough, they will eventually
> be reclaimed, and then kswapd_failures gets reset to 0 - because even reclaiming
> a single page resets kswapd_failures to 0.
> 
> The end result is that we most workloads, which didn't even hit their high
> or max limits, experiencing continuous refaults, causing heavy I/O.
> 

So, the decision to reset kswapd_failures on memcg reclaim can be
re-evaluated but I think that is not the root cause here. The
kswapd_failures mechanism is for situations where kswapd is unable to
reclaim and then punting on the direct reclaimers but in your situation
the workloads are not numa memory bound and thus there really is not any
numa level direct reclaimers. Also the lack of reclaimable memory is
making the situation worse.


> Thanks.
> 
> > > 
> > > preventing Node 0's kswapd from stopping
> > >  The few file pages on Node 0 are hot and keep refaulting, causing heavy I/O
> > > 
> > Have you tried numa balancing? Though I think it would be better to
> > schedule upfront in a way that one node is not overcommitted but numa
> > balancing provides a dynamic way to adjust the load on each node.
> 
> Yes, we have tried it. Actually, I submitted a patch about a month ago to improve
> its observability:
> https://lore.kernel.org/all/20251124153331.465306a2@gandalf.local.home/
> (though only Steven replied, a bit awkward :( ).
> 
> We found that the default settings didn't work well for our workloads. When we tried
> to increase scan_size to make it more aggressive, we noticed the system load started
> to increase. So we haven't fully adopted it yet.
> 

I feel the numa balancing will not help as well as or it might make it
worse as the workloads may have allocated some memory on the other node
which numa balancing might try to move to the node which is already
under pressure.

Let me say what I think is the issue. You have the situation where node
0 is overcommitted and is mostly filled with unreclaimable memory. The
workloads running on node 0 have their workingset continuously getting
reclaimed due to node 0 being OOM.

I think the simplest solution for you is to enable swap to have more
reclaimable memory on the system. Hopefully you will have workingset of
the workloads fully in memory on each node.

You can try to change application/workload to be more numa aware and
balance their anon memory on the given nodes but I think that would much
more involved and error prone.

Shakeel