Environment: A machine with a fast SSD and a high core count (e.g., Google Cloud's N2-standard-128).
Workload: Concurrently generate a large number of files (e.g., 2 million) using multiple services managed by systemd-run. This creates significant I/O and cgroup churn.
Trigger: After the file generation completes, terminate the systemd-run services.
Result: Shortly after the services are killed, the system's CPU load spikes, leading to a massive number of kworker/+inode_switch_wbs threads and a system-wide hang/livelock where the machine becomes unresponsive (20s - 300s).
1. The initial commit The process begins with a worker that can get stuck busy-waiting on a spinlock.
Commit: ("writeback, cgroup: release dying cgwbs by switching attached inodes")
Effect: This introduced the inode_switch_wbs_work_fn worker to clean up cgroup writeback structures. Under our test load, this worker appears to hit a highly contended wb->list_lock spinlock, causing it to burn 100% CPU without sleeping.
2. The Kworker Explosion A subsequent change misinterprets the spinning worker from Stage 1, leading to a runaway feedback loop of worker creation.
Commit: 616db8779b1e ("workqueue: Automatically mark CPU-hogging work items CPU_INTENSIVE")
Effect: This logic sees the spinning worker, marks it as CPU_INTENSIVE, and excludes it from concurrency management. To handle the work backlog, it spawns a new kworker, which then also gets stuck on the same lock, repeating the cycle. This directly causes the kworker count to explode from <50 to 100-2000+.
3. The System-Wide Lockdown The final piece allows this localized worker explosion to saturate the entire system.
Commit: 8639ecebc9b1 ("workqueue: Implement non-strict affinity scope for unbound workqueues")
Effect: This change introduced non-strict affinity as the default. It allows the hundreds of kworkers created in Stage 2 to be spread by the scheduler across all available CPU cores, turning the problem into a system-wide hang.
Mainline Status: On the latest mainline kernel, the hang is far less frequent and the kworker counts are reduced back to normal (<50), suggesting other changes have partially mitigated the issue. However, the hang still occurs, and when it does, the kworker count still explodes (e.g., 300+), indicating the underlying feedback loop remains.
Workaround: A reliable mitigation is to revert to the old workqueue behavior by setting affinity_strict to 1. This contains the kworker proliferation to a single CPU pod, preventing the system-wide hang.
Given that the issue is not fully resolved, could you please provide some guidance?
Is this a known issue, and are there patches in development that might fully address the underlying spinlock contention or the kworker feedback loop?
Is there a better long-term mitigation we can apply other than forcing strict affinity?
Thank you for your time and help.
Best regards,
Chenglong