Re: [PATCH 0/2] Fix 100% CPU usage issue in IOU worker threads

[Jens Axboe <axboe@kernel.dk>]

On 4/22/25 8:10 AM, ??? wrote:
> On Tue, Apr 22, 2025 at 9:35?PM Jens Axboe <axboe@kernel.dk> wrote:
>>
>> On 4/22/25 4:45 AM, Zhiwei Jiang wrote:
>>> In the Firecracker VM scenario, sporadically encountered threads with
>>> the UN state in the following call stack:
>>> [<0>] io_wq_put_and_exit+0xa1/0x210
>>> [<0>] io_uring_clean_tctx+0x8e/0xd0
>>> [<0>] io_uring_cancel_generic+0x19f/0x370
>>> [<0>] __io_uring_cancel+0x14/0x20
>>> [<0>] do_exit+0x17f/0x510
>>> [<0>] do_group_exit+0x35/0x90
>>> [<0>] get_signal+0x963/0x970
>>> [<0>] arch_do_signal_or_restart+0x39/0x120
>>> [<0>] syscall_exit_to_user_mode+0x206/0x260
>>> [<0>] do_syscall_64+0x8d/0x170
>>> [<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80
>>> The cause is a large number of IOU kernel threads saturating the CPU
>>> and not exiting. When the issue occurs, CPU usage 100% and can only
>>> be resolved by rebooting. Each thread's appears as follows:
>>> iou-wrk-44588  [kernel.kallsyms]  [k] ret_from_fork_asm
>>> iou-wrk-44588  [kernel.kallsyms]  [k] ret_from_fork
>>> iou-wrk-44588  [kernel.kallsyms]  [k] io_wq_worker
>>> iou-wrk-44588  [kernel.kallsyms]  [k] io_worker_handle_work
>>> iou-wrk-44588  [kernel.kallsyms]  [k] io_wq_submit_work
>>> iou-wrk-44588  [kernel.kallsyms]  [k] io_issue_sqe
>>> iou-wrk-44588  [kernel.kallsyms]  [k] io_write
>>> iou-wrk-44588  [kernel.kallsyms]  [k] blkdev_write_iter
>>> iou-wrk-44588  [kernel.kallsyms]  [k] iomap_file_buffered_write
>>> iou-wrk-44588  [kernel.kallsyms]  [k] iomap_write_iter
>>> iou-wrk-44588  [kernel.kallsyms]  [k] fault_in_iov_iter_readable
>>> iou-wrk-44588  [kernel.kallsyms]  [k] fault_in_readable
>>> iou-wrk-44588  [kernel.kallsyms]  [k] asm_exc_page_fault
>>> iou-wrk-44588  [kernel.kallsyms]  [k] exc_page_fault
>>> iou-wrk-44588  [kernel.kallsyms]  [k] do_user_addr_fault
>>> iou-wrk-44588  [kernel.kallsyms]  [k] handle_mm_fault
>>> iou-wrk-44588  [kernel.kallsyms]  [k] hugetlb_fault
>>> iou-wrk-44588  [kernel.kallsyms]  [k] hugetlb_no_page
>>> iou-wrk-44588  [kernel.kallsyms]  [k] hugetlb_handle_userfault
>>> iou-wrk-44588  [kernel.kallsyms]  [k] handle_userfault
>>> iou-wrk-44588  [kernel.kallsyms]  [k] schedule
>>> iou-wrk-44588  [kernel.kallsyms]  [k] __schedule
>>> iou-wrk-44588  [kernel.kallsyms]  [k] __raw_spin_unlock_irq
>>> iou-wrk-44588  [kernel.kallsyms]  [k] io_wq_worker_sleeping
>>>
>>> I tracked the address that triggered the fault and the related function
>>> graph, as well as the wake-up side of the user fault, and discovered this
>>> : In the IOU worker, when fault in a user space page, this space is
>>> associated with a userfault but does not sleep. This is because during
>>> scheduling, the judgment in the IOU worker context leads to early return.
>>> Meanwhile, the listener on the userfaultfd user side never performs a COPY
>>> to respond, causing the page table entry to remain empty. However, due to
>>> the early return, it does not sleep and wait to be awakened as in a normal
>>> user fault, thus continuously faulting at the same address,so CPU loop.
>>> Therefore, I believe it is necessary to specifically handle user faults by
>>> setting a new flag to allow schedule function to continue in such cases,
>>> make sure the thread to sleep.
>>>
>>> Patch 1  io_uring: Add new functions to handle user fault scenarios
>>> Patch 2  userfaultfd: Set the corresponding flag in IOU worker context
>>>
>>>  fs/userfaultfd.c |  7 ++++++
>>>  io_uring/io-wq.c | 57 +++++++++++++++---------------------------------
>>>  io_uring/io-wq.h | 45 ++++++++++++++++++++++++++++++++++++--
>>>  3 files changed, 68 insertions(+), 41 deletions(-)
>>
>> Do you have a test case for this? I don't think the proposed solution is
>> very elegant, userfaultfd should not need to know about thread workers.
>> I'll ponder this a bit...
>>
>> --
>> Jens Axboe
> Sorry,The issue occurs very infrequently, and I can't manually
> reproduce it. It's not very elegant, but for corner cases, it seems
> necessary to make some compromises.

I'm going to see if I can create one. Not sure I fully understand the
issue yet, but I'd be surprised if there isn't a more appropriate and
elegant solution rather than exposing the io-wq guts and having
userfaultfd manipulate them. That really should not be necessary.

-- 
Jens Axboe