From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0BA8C369C2 for ; Tue, 22 Apr 2025 14:11:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2AC7F6B0005; Tue, 22 Apr 2025 10:10:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 25B5D6B0006; Tue, 22 Apr 2025 10:10:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14ABB6B0007; Tue, 22 Apr 2025 10:10:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EB18E6B0005 for ; Tue, 22 Apr 2025 10:10:58 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CBBA514129D for ; Tue, 22 Apr 2025 14:11:00 +0000 (UTC) X-FDA: 83361866280.18.864E3DD Received: from mail-ot1-f50.google.com (mail-ot1-f50.google.com [209.85.210.50]) by imf21.hostedemail.com (Postfix) with ESMTP id E50191C001F for ; Tue, 22 Apr 2025 14:10:58 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=O4YvbfWc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of qq282012236@gmail.com designates 209.85.210.50 as permitted sender) smtp.mailfrom=qq282012236@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745331059; a=rsa-sha256; cv=none; b=NHqPKgr+g1fWeWOKLWzXKklRJhvHqp7fnM9Jx0Sy9SB4mylA31YEYir50ave7ucqOrUl0v F85Jlf9G4rFxpy6A0Wt2+ETGzUJyP6HYFAsKZNH3cTiYZtDzTrGRfXVsHsOENrj4rhi+on hD52AlRlbf5fVrVkHKiIZyk1E4Qfogg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745331058; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ENmNvcG+i4cA95dff7/W/R/4xtydjhp0oR96uzNkfLE=; b=yqrgLXDFjq1Bks7r87fpa5LgwuNBD/ZFLWVs3spPPiAmnyyizA2B992+fkxnoj3oIZrkDp qLs0QxKctk4In5BiMBkU9mR1UvEKBtCbS2x+bL34+r8mhC+RaH9DyLj+gGju5pGHGvQ7/q uW8TFn5FxNTgxWzM5uKdJFOgxW6d9KI= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=O4YvbfWc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of qq282012236@gmail.com designates 209.85.210.50 as permitted sender) smtp.mailfrom=qq282012236@gmail.com Received: by mail-ot1-f50.google.com with SMTP id 46e09a7af769-7301c227512so1599281a34.2 for ; Tue, 22 Apr 2025 07:10:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1745331058; x=1745935858; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ENmNvcG+i4cA95dff7/W/R/4xtydjhp0oR96uzNkfLE=; b=O4YvbfWcp1Ql+p0HRcyCxXjoLmZEueUOvbeHyQReIe6JIAOKhjVErjUhJN8/xW9qiZ J7+G+w2wehC9NHO+ZSnXJW4QiFQvvhQqvDWu8vQfe4TYR2zaSuG7Rc9GPS++Xgxkp4Xz JsCZjhMkRQGy3V1QmnmoTj0Tw5aHRDu1Zs26wQ8GUaB7yTq48zkl9qLPutV6liAfnN1s GhlRxRKBFdE5AurRn9Z2V1Adg2cuGJSfTciQXnP/mk/psw8bVX/+zzZNw2DOaj6sebRE XUW2mGL1iLejVCKg6oJ8bIbPOz87PeKJCc5O/wzywyCxsfJvMU3W8OqW+z3qZPL3u007 bbdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745331058; x=1745935858; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ENmNvcG+i4cA95dff7/W/R/4xtydjhp0oR96uzNkfLE=; b=fZO4kMV0GD0O2SmhlqmS5Wk46oqnBw/PgXD/d6b5LPTmb8W0evrgTDx35ZMe5srvHk Hc+23xja1RiqD33+t+iNkUgsbjJtPD/CPnNQUkXtrjFUuBB3AUaaCfXMxC31oP8xgKUn V6NakNdBY5qrVprsXSSnA4iIOKV6UFIsiWuBo6KIxhoJwjMpfpB1ZWt4IcHNf4sXUPSu Err7/o0dEuOOauO3vZ5npuHHPNjHECt+TjK9sszSyBjWQXBr5UcycJZnxQBnOIjnyyA4 b4DLoBHs3TccJI4QQgYjkTOg5mgxnrnGGRGtDqWaNSXmQQwWfTuPUTZtofLvky8DQrRm 3bCg== X-Forwarded-Encrypted: i=1; AJvYcCXC4jsorttzBE0Z9tAgYFydx+NL72ZEeLOgUcRlBJpJowIJXkYKdDy0iSjdTodQl1y1+eKEv86LPw==@kvack.org X-Gm-Message-State: AOJu0YwUKxzEq59mTvTZgT9M3oNZaWHpolWgN3WE0bMoL7QglmwLnmt2 itp0baVEF0okrRCphU4RZXciBNMxrNKmwZNAcy4J3ZWI01RvmnBZMe/2BGqtovdbFnBCAMwwIyv svWrrZBeBA81Z20nMvM6SZ5qbYWo= X-Gm-Gg: ASbGnctYlxiWcsLJ+sGD7Oka7vcz8zVfmTMFlVwLWBN1uo4HWjgff9J1JarGvv9BpTK 4iPVc7Ig1hje0I0IvGWmBkW6UO8nVXlj6z+kQSGQ+zMBgqTNaa7WvwYR/UwelToDexyifhi/f9i 3On48Kcjl/MG+X+Zk0R9+clOA= X-Google-Smtp-Source: AGHT+IHs52QH1e1EcEp6l4JLqsVOeDSJ3Ng/TlhNfJ91v6SCzBDysQnzdm5i0GiYHIM+K+KBtAUyFHpvlpyYlRyyIQM= X-Received: by 2002:a05:6871:530d:b0:28c:8476:dd76 with SMTP id 586e51a60fabf-2d526ee4ad6mr11088969fac.29.1745331057649; Tue, 22 Apr 2025 07:10:57 -0700 (PDT) MIME-Version: 1.0 References: <20250422104545.1199433-1-qq282012236@gmail.com> In-Reply-To: From: =?UTF-8?B?5aec5pm65Lyf?= Date: Tue, 22 Apr 2025 22:10:45 +0800 X-Gm-Features: ATxdqUFY0D9ZhIoPAnO4wqwO_j8l65R--mu-PkgmPt3dkQ_eilqKjmLgt8z5KSc Message-ID: Subject: Re: [PATCH 0/2] Fix 100% CPU usage issue in IOU worker threads To: Jens Axboe Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, akpm@linux-foundation.org, peterx@redhat.com, asml.silence@gmail.com, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, io-uring@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E50191C001F X-Stat-Signature: 3qh9wbzbms937oem5wzr34ufwacqkw1m X-Rspam-User: X-HE-Tag: 1745331058-55754 X-HE-Meta: U2FsdGVkX1/eSaoyfB7lMJjcEqS0izwcz4OTGUNsxjRQrvGZtWZxl3/WF++irNWINVrTxf7c5lRflXgIS9AVvxS+VM8oW1wl92ZTezS5oxuqfulch/jGMSOIfMi/uKsWx3KmLCDhlaaH5PcCeW+dFfzAcNQTK/LZcY+Ve2lxUpqIDa0566tWE74WusZawEV64IEndieR0In43p7xj9nB1tMNvf9EgfSSuAIyd9OlcNyWWlzQJhPpp33V2+40VpVAXB8G2bxYpa2sgrjGs/rnCtHv1EZg3WsUQdfuoC+rnzl+X6hh9f4V95lfXM/nSujqMdkW8dYaQ5boaRHTy43UjdmfXvuTaFgOt/pN8ANzYYOi9js6N//R6RueaVdaunEyd6axTYxh3u96FLtzaJ/mQMTYJpciWIF1gwW8nCzqH5X172KehyR9aARpKGZYypN+al36I12qGr9wXMR9mytumYBYr5p7K8uoK8BzXskGbCiwWX2skQ2YGpEc2WhYfONicRNoJFcoNMIR7KSWo2embxzoCNRS1fMNpX7yXJB8qiwKpYXsjNzrjAq2NllvFHOjPOfTddQgjLgtbhrTp2EOHWMShI7DQjc1J7yLDii5FyQbCwfTf13OHRYbxAiN6onSeDRdKvuqAtZblqf6s4ESpsDSm3tCtZwxP73LKWZ/3HtTGol88Y4m7Au7LznNElEm16gH1WaohmMZJxxCbvWlBAUSfr5V5JLI+9NBf87x/BlqMgS/5z15K2kHwY2RLrWMnmNSWwUtjvlCJLKic+u0puwTR1CkUyuCqoBOKHnywwdOKM1X+RHf9xUswfiZRvbryhJT6/jZCGKz4RPYHigPuNbQGF91ZHkawTJkyVANezH9wuzrx+ZMbbzkhycKLG5fWD0UL+lgyi2mBs2I4JWOcS//0NmeG7FeY1VYwaeMWvbCaC/mzKw04W7oi9IWNrm/kHTPIWlaetLQX1V9il6 ir0t/2JW EIK8g9aohuT7S9ps86D4Y49Pa21Ja7d87y5EZ6Yj2M/C44Hjk37f663i44fBXQizlIdezOgvrKEjqgU5Qxt850LXDALkMGmfzi4WBM5BCqeUM0OnGNQlJKztMf4i8JzuMNowck2Uyeq3aCBms0CiXuZnmebqcdfWCahK3GIomNndHC4ERptx8BATYSKJdwuHDhW/4mUR+2wDjoxr3d4CMcB/wMxtulBUsoX72Mzxb9hRdSMTbH1sX26vAXZ3cLVuNeE2j37kcZNw/W/dSIR+m/JOYEkhNQSFetLL6lu3mZDUdgf39Tv/pucOkutBxo2t4nPDeccYsyGtF1BE3dKeaEQ/4lACXSKd/z0/7gKAiGqT7O5no6RVBid+ZnLjhZTrqo1vpGPvjBmh6HQs4Of8/eEW7D3Aw9xVSc88L X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Apr 22, 2025 at 9:35=E2=80=AFPM Jens Axboe wrote: > > On 4/22/25 4:45 AM, Zhiwei Jiang wrote: > > In the Firecracker VM scenario, sporadically encountered threads with > > the UN state in the following call stack: > > [<0>] io_wq_put_and_exit+0xa1/0x210 > > [<0>] io_uring_clean_tctx+0x8e/0xd0 > > [<0>] io_uring_cancel_generic+0x19f/0x370 > > [<0>] __io_uring_cancel+0x14/0x20 > > [<0>] do_exit+0x17f/0x510 > > [<0>] do_group_exit+0x35/0x90 > > [<0>] get_signal+0x963/0x970 > > [<0>] arch_do_signal_or_restart+0x39/0x120 > > [<0>] syscall_exit_to_user_mode+0x206/0x260 > > [<0>] do_syscall_64+0x8d/0x170 > > [<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80 > > The cause is a large number of IOU kernel threads saturating the CPU > > and not exiting. When the issue occurs, CPU usage 100% and can only > > be resolved by rebooting. Each thread's appears as follows: > > iou-wrk-44588 [kernel.kallsyms] [k] ret_from_fork_asm > > iou-wrk-44588 [kernel.kallsyms] [k] ret_from_fork > > iou-wrk-44588 [kernel.kallsyms] [k] io_wq_worker > > iou-wrk-44588 [kernel.kallsyms] [k] io_worker_handle_work > > iou-wrk-44588 [kernel.kallsyms] [k] io_wq_submit_work > > iou-wrk-44588 [kernel.kallsyms] [k] io_issue_sqe > > iou-wrk-44588 [kernel.kallsyms] [k] io_write > > iou-wrk-44588 [kernel.kallsyms] [k] blkdev_write_iter > > iou-wrk-44588 [kernel.kallsyms] [k] iomap_file_buffered_write > > iou-wrk-44588 [kernel.kallsyms] [k] iomap_write_iter > > iou-wrk-44588 [kernel.kallsyms] [k] fault_in_iov_iter_readable > > iou-wrk-44588 [kernel.kallsyms] [k] fault_in_readable > > iou-wrk-44588 [kernel.kallsyms] [k] asm_exc_page_fault > > iou-wrk-44588 [kernel.kallsyms] [k] exc_page_fault > > iou-wrk-44588 [kernel.kallsyms] [k] do_user_addr_fault > > iou-wrk-44588 [kernel.kallsyms] [k] handle_mm_fault > > iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_fault > > iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_no_page > > iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_handle_userfault > > iou-wrk-44588 [kernel.kallsyms] [k] handle_userfault > > iou-wrk-44588 [kernel.kallsyms] [k] schedule > > iou-wrk-44588 [kernel.kallsyms] [k] __schedule > > iou-wrk-44588 [kernel.kallsyms] [k] __raw_spin_unlock_irq > > iou-wrk-44588 [kernel.kallsyms] [k] io_wq_worker_sleeping > > > > I tracked the address that triggered the fault and the related function > > graph, as well as the wake-up side of the user fault, and discovered th= is > > : In the IOU worker, when fault in a user space page, this space is > > associated with a userfault but does not sleep. This is because during > > scheduling, the judgment in the IOU worker context leads to early retur= n. > > Meanwhile, the listener on the userfaultfd user side never performs a C= OPY > > to respond, causing the page table entry to remain empty. However, due = to > > the early return, it does not sleep and wait to be awakened as in a nor= mal > > user fault, thus continuously faulting at the same address,so CPU loop. > > Therefore, I believe it is necessary to specifically handle user faults= by > > setting a new flag to allow schedule function to continue in such cases= , > > make sure the thread to sleep. > > > > Patch 1 io_uring: Add new functions to handle user fault scenarios > > Patch 2 userfaultfd: Set the corresponding flag in IOU worker context > > > > fs/userfaultfd.c | 7 ++++++ > > io_uring/io-wq.c | 57 +++++++++++++++--------------------------------- > > io_uring/io-wq.h | 45 ++++++++++++++++++++++++++++++++++++-- > > 3 files changed, 68 insertions(+), 41 deletions(-) > > Do you have a test case for this? I don't think the proposed solution is > very elegant, userfaultfd should not need to know about thread workers. > I'll ponder this a bit... > > -- > Jens Axboe Sorry,The issue occurs very infrequently, and I can't manually reproduce it. It's not very elegant, but for corner cases, it seems necessary to make some compromises.