Re: [PATCH] memcg: Always call cond_resched() after fn()

[Shakeel Butt <shakeel.butt@linux.dev>]

On Fri, May 23, 2025 at 10:21:06AM -0700, Breno Leitao wrote:
> I am seeing soft lockup on certain machine types when a cgroup
> OOMs. This is happening because killing the process in certain machine
> might be very slow, which causes the soft lockup and RCU stalls. This
> happens usually when the cgroup has MANY processes and memory.oom.group
> is set.
> 
> Example I am seeing in real production:
> 
>        [462012.244552] Memory cgroup out of memory: Killed process 3370438 (crosvm) ....
>        ....
>        [462037.318059] Memory cgroup out of memory: Killed process 4171372 (adb) ....
>        [462037.348314] watchdog: BUG: soft lockup - CPU#64 stuck for 26s! [stat_manager-ag:1618982]
>        ....
> 

> Quick look at why this is so slow, it seems to be related to serial
> flush for certain machine types. For all the crashes I saw, the target
> CPU was at console_flush_all().
> 
> In the case above, there are thousands of processes in the cgroup, and
> it is soft locking up before it reaches the 1024 limit in the code
> (which would call the cond_resched()). So, cond_resched() in 1024 blocks
> is not sufficient.
> 
> Remove the counter-based conditional rescheduling logic and call
> cond_resched() unconditionally after each task iteration, after fn() is
> called. This avoids the lockup independently of how slow fn() is.
> 
> Cc: Michael van der Westhuizen <rmikey@meta.com>
> Cc: Usama Arif <usamaarif642@gmail.com>
> Cc: Pavel Begunkov <asml.silence@gmail.com>
> Suggested-by: Rik van Riel <riel@surriel.com>
> Signed-off-by: Breno Leitao <leitao@debian.org>
> Fixes: 46576834291869457 ("memcg: fix soft lockup in the OOM process")

Can you share the call stack but I think from the above, it seems to be
from oom_kill_memcg_member(). Have you tried making __oom_kill_process
not chatty? Basically instead of dumping to serial directly, use local
buffer and then dump once it is full.

Anyways, that would be a bit more involved and until then this seems
fine to me.

Acked-by: Shakeel Butt <shakeel.butt@linux.dev>