Re: [PATCH v1] mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaim

["Jiayuan Chen" <jiayuan.chen@linux.dev>]

January 5, 2026 at 12:51, "Shakeel Butt" <shakeel.butt@linux.dev mailto:shakeel.butt@linux.dev?to=%22Shakeel%20Butt%22%20%3Cshakeel.butt%40linux.dev%3E > wrote:


> 
> Hi Jiayuan,
> 
> Sorry for late reply due to holidays/break. I will still be slow to
> respond this week but will be fully back after one more week. Anyways,
> let me respond below.

No worries about the delay - happy holidays!

> On Tue, Dec 23, 2025 at 08:22:43AM +0000, Jiayuan Chen wrote:
> 
> > 
> > December 23, 2025 at 14:11, "Shakeel Butt" <shakeel.butt@linux.dev mailto:shakeel.butt@linux.dev?to=%22Shakeel%20Butt%22%20%3Cshakeel.butt%40linux.dev%3E > wrote:
> >  
> >  
> >  
> >  On Tue, Dec 23, 2025 at 01:42:37AM +0000, Jiayuan Chen wrote:
> >  
> >  > 
> >  > December 23, 2025 at 05:15, "Shakeel Butt" <shakeel.butt@linux.dev mailto:shakeel.butt@linux.dev?to=%22Shakeel%20Butt%22%20%3Cshakeel.butt%40linux.dev%3E > wrote:
> >  > 
> >  [...]
> >  
> >  > 
> >  > > 
> >  > I don't think kswapd is an issue here. The system is out of memory and
> >  > most of the memory is unreclaimable. Either change the workload to use
> >  > less memory or enable swap (or zswap) to have more reclaimable memory.
> >  > 
> >  > 
> >  > Hi,
> >  > Thanks for looking into this.
> >  > 
> >  > Sorry, I didn't describe the scenario clearly enough in the original patch. Let me clarify:
> >  > 
> >  > This is a multi-NUMA system where the memory pressure is not global but node-local. The key observation is:
> >  > 
> >  > Node 0: Under memory pressure, most memory is anonymous (unreclaimable without swap)
> >  > Node 1: Has plenty of reclaimable memory (~60GB file cache out of 125GB total)
> >  > 
> >  Thanks and now the situation is much more clear. IIUC you are running
> >  multiple workloads (pods) on the system. How is the memcg limits
> >  configured for these workloads. You mentioned memory.high, what about
> >  
> >  Thanks for the questions. We have pods configured with memory.high and pods configured with memory.max.
> >  
> >  Actually, memory.max itself causes heavy I/O issues for us, because it keeps trying to reclaim hot
> >  pages within the cgroup aggressively without killing the process. 
> >  
> >  So we configured some pods with memory.high instead, since it performs reclaim in resume_user_mode_work,
> >  which somewhat throttles the memory allocation of user processes.
> >  
> >  memory.max? Also are you using cpusets to limit the pods to individual
> >  nodes (cpu & memory) or they can run on any node?
> >  
> >  Yes, we have cpusets(only cpuset.cpus not cpuset.mems) configured for our cgroups, binding
> >  them to specific NUMA nodes. But I don't think this is directly related to the issue - the
> >  problem can occur with or without cpusets. Even without cpusets.cpus, the kernel prefers
> >  to allocate memory from the node where the process is running, so if a process happens to
> >  run on a CPU belonging to Node 0, the behavior would be similar.
> > 
> Are you limiting (using cpuset.cpus) the workloads to single respective
> nodes or the individual workloads can still run on multiple nodes? For
> example do you have a workload which can run on both (or more) nodes?

We have many workloads. Some performance-sensitive ones have cpuset.cpus configured to
bind to a specific node, while others don't.

> > 
> > Overall I still think it is unbalanced numa nodes in terms of memory and
> >  may for cpu as well. Anyways let's talk about kswapd.
> >  > 
> >  > Node 0's kswapd runs continuously but cannot reclaim anything
> >  > Direct reclaim succeeds by reclaiming from Node 1
> >  > Direct reclaim resets kswapd_failures,
> >  > 
> >  So successful reclaim on one node does not reset kswapd_failures on
> >  other node. The kernel reclaims each node one by one, so if Node 0
> >  direct reclaim was successfull only then kernel allows to reset the
> >  kswapd_failures of Node 0 to be reset.
> >  
> >  Let me dig deeper into this.
> >  
> >  When either memory.max or memory.high is reached, direct reclaim is
> >  triggered. The memory being reclaimed depends on the CPU where the
> >  process is running.
> >  
> >  When the problem occurred, we had workloads continuously hitting 
> >  memory.max and workloads continuously hitting memory.high:
> >  
> >  reclaim_high -> -> try_to_free_mem_cgroup_pages
> >  ^ do_try_to_free_pages(zone of current node)
> >  | shrink_zones()
> >  try_charge_memcg - shrink_node()
> >  kswapd_failures = 0
> >  
> >  Although the pages are hot, if we scan aggressively enough, they will eventually
> >  be reclaimed, and then kswapd_failures gets reset to 0 - because even reclaiming
> >  a single page resets kswapd_failures to 0.
> >  
> >  The end result is that we most workloads, which didn't even hit their high
> >  or max limits, experiencing continuous refaults, causing heavy I/O.
> > 
> So, the decision to reset kswapd_failures on memcg reclaim can be
> re-evaluated but I think that is not the root cause here. The


The workloads triggering direct reclaim have their memory spread across multiple nodes,
since we don't set cpuset.mems, so the cgroup can reclaim memory from multiple nodes.
In particular, complex applications have many threads, different threads allocating and
freeing large amounts of memory (both anonymous and file pages), and these allocations
can consume memory from nodes that are above the low watermark.

You're right that multiple factors contribute to the issue I described. This patch addresses
one of them, just like the boost_watermark patch I submitted before, and the recent patch
about memory.high causing high I/O. There are other scenarios as well that I'm still trying
to reproduce.

That said, I believe this patch is still a valid fix on its own - resetting kswapd_failures
when the node is not actually balanced doesn't seem like correct behavior regardless of the
broader context.

> kswapd_failures mechanism is for situations where kswapd is unable to
> reclaim and then punting on the direct reclaimers but in your situation
> the workloads are not numa memory bound and thus there really is not any
> numa level direct reclaimers. Also the lack of reclaimable memory is
> making the situation worse.


> > 
> > Thanks.
> >  
> >  > 
> >  > preventing Node 0's kswapd from stopping
> >  > The few file pages on Node 0 are hot and keep refaulting, causing heavy I/O
> >  > 
> >  Have you tried numa balancing? Though I think it would be better to
> >  schedule upfront in a way that one node is not overcommitted but numa
> >  balancing provides a dynamic way to adjust the load on each node.
> >  
> >  Yes, we have tried it. Actually, I submitted a patch about a month ago to improve
> >  its observability:
> >  https://lore.kernel.org/all/20251124153331.465306a2@gandalf.local.home/
> >  (though only Steven replied, a bit awkward :( ).
> >  
> >  We found that the default settings didn't work well for our workloads. When we tried
> >  to increase scan_size to make it more aggressive, we noticed the system load started
> >  to increase. So we haven't fully adopted it yet.
> > 
> I feel the numa balancing will not help as well as or it might make it
> worse as the workloads may have allocated some memory on the other node
> which numa balancing might try to move to the node which is already
> under pressure.

Agreed.

> Let me say what I think is the issue. You have the situation where node
> 0 is overcommitted and is mostly filled with unreclaimable memory. The
> workloads running on node 0 have their workingset continuously getting
> reclaimed due to node 0 being OOM.

From our monitoring, only a single cgroup triggered direct reclaim - some
hitting memory.high and some hitting memory.max (we have tracepoints for monitoring).

> I think the simplest solution for you is to enable swap to have more
> reclaimable memory on the system. Hopefully you will have workingset of
> the workloads fully in memory on each node.
> 
> You can try to change application/workload to be more numa aware and
> balance their anon memory on the given nodes but I think that would much
> more involved and error prone.

Enabling swap is one solution, but due to historical reasons we haven't
enabled it - our disk performance is relatively poor. zram is also an
option, but the migration would take significant time.

Thanks