From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F32DC369D3 for ; Tue, 22 Apr 2025 14:13:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7EC776B0007; Tue, 22 Apr 2025 10:13:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 79CDD6B000A; Tue, 22 Apr 2025 10:13:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63CD46B000C; Tue, 22 Apr 2025 10:13:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 473D56B0007 for ; Tue, 22 Apr 2025 10:13:25 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4490581408 for ; Tue, 22 Apr 2025 14:13:25 +0000 (UTC) X-FDA: 83361872370.12.E1DE192 Received: from mail-il1-f182.google.com (mail-il1-f182.google.com [209.85.166.182]) by imf24.hostedemail.com (Postfix) with ESMTP id 2EBC818000C for ; Tue, 22 Apr 2025 14:13:23 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=3YX87OTf; dmarc=none; spf=pass (imf24.hostedemail.com: domain of axboe@kernel.dk designates 209.85.166.182 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745331203; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QR8zq+WeURJcQJZtbdR5JK1T9fRJKOc9/cP5khZOT8Q=; b=px6trgOSLGUTZzBy7FxXqpHHSmPWTLRT2OwiKnCE4JNMYExVDgeb/olabl7oOGy0nKS3HM T3KMaS/pjES4nd2m+IHQjFxaYgDhv1mD9RArKCrI9AOuzsWBT3uGbkSlLVH1qIM9uAPEb8 f/rafHJkjhg6ccbzK8zcBOJI3eieEpg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745331203; a=rsa-sha256; cv=none; b=e7oBcwKhRIwmrXo/WmdA75J9EBZPuIZlrUJ3PAlcCH4cI8H21njU9ik1BuVozMXa4hNCGB JrIuMQfJaTK9I1px1tyUWcdTtX7zXiZZlEZfWqInbL8scJ5zE8GWVK5VyfuHD1MwgeCPC5 LtMUXkzsWgabqxbNfZPahO2xsy4m+rw= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=3YX87OTf; dmarc=none; spf=pass (imf24.hostedemail.com: domain of axboe@kernel.dk designates 209.85.166.182 as permitted sender) smtp.mailfrom=axboe@kernel.dk Received: by mail-il1-f182.google.com with SMTP id e9e14a558f8ab-3d7f4cb7636so14770585ab.3 for ; Tue, 22 Apr 2025 07:13:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1745331202; x=1745936002; darn=kvack.org; h=content-transfer-encoding:in-reply-to:content-language:from :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=QR8zq+WeURJcQJZtbdR5JK1T9fRJKOc9/cP5khZOT8Q=; b=3YX87OTf+S694AAfRuZO0UDafJMqc1A9TQqSMNmyn82yzmg2RrIno+T9RTijRxr9Td TvtdRyO3VYVsGHxH18W0WJ3yiX7id78yVQwhoXAWfezRYj4ZkFkJOySESZSV7sbreJSo drosxZ51S6vXJJt5fhXWJ863dtehiyxotOYYVzqx2e8RF/lzd0AtScRKFGKIH+etrJOx 3S638aZToQw8MQ8Zrqxb2wHwh7My1yNpPWhnNqLWcT/u1iJwMyOHXY/NXo9ZD2SRkQda xG9k2SzejonqsvQ+RKqeQFUG+XBFSWHXUZWGYTBIZG3i62GZ1TqpDP5Kvszuc6U2FrUr d52A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745331202; x=1745936002; h=content-transfer-encoding:in-reply-to:content-language:from :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QR8zq+WeURJcQJZtbdR5JK1T9fRJKOc9/cP5khZOT8Q=; b=Oalm+y4vtwk1SZRvUHezFIOxG+Mx8Mmv7tcgoLNULj/RjRtyghjuY1rL7Y4ZA2achB tSkU0Gd9HLgaXyMqTPyEobcGMxsqvNyohhYB0ibgp4N+1k6mCG1362WBPSCLatF0dvBi IbRKgsMnNtuQDmo8nrx3DoBVBg0clzWVR1F+k1u/0ovSfv4lbK0/h0D0iwcLrnWTdgXT sjwZ/ilJrFpL0fL0zQNdDUORcheUUMKHSNA0aBpGToDIsNoiw/YSLqtAaKTYXX/0ObwU PZw5polbSweVSYHUsWaLUC/NFD8oX9nuS5mpylefNVxmjf4pBqgaPAIGspe4eygFanLa VsqA== X-Forwarded-Encrypted: i=1; AJvYcCUPnMJ+Vhgt6bGUygIYRVtkCA6wdez5SN5bLQ26j0/YDSFBDfH6qRc+9BAfyzd/rjsCFXGlpdGNhA==@kvack.org X-Gm-Message-State: AOJu0YyFuVfacDqGJng1cXtOUwyO4/ugjHY7vB6S+D1esuJHV0+uK+VX deJwUFp+XAkuZke4h4B5AR5o3enMjr7c6g829zZetAPcZip5ss7EHg0LgAJHFqs= X-Gm-Gg: ASbGncvsHPP/s9jiL271a6XpJWEwEPO284u9bBH0AUq0VVZAXZ14cuJTNnFDLDfMP9u zGRkB6FaIBWdkU/AGbpFaiHdQciaWKJjZ62lLzlAFq1kI1g2LoQpXmGpk6N4Xnpde+7SPPwAlGg dZsQa/kSt1d2r42DM9sdeRSdSFCvBsFDcCQXbk7vkMpDzy+gKRDSZUQfp/u5i+T+mJasviIZBg2 SO0ciVszaMZgyF3rIdHrUfUfXu06R7qBJBAZGUbsxARkohM7fpV/UNX+rAbtDVt+QdqXJu3fTtr 8Y7vcAO5lAANzV9S2Enzc8q0q5JZ7lzIEry8Nw== X-Google-Smtp-Source: AGHT+IHWzI0Wn75AyMi9DGxMOkS03WlRiBtxZ7rlcgm58ABO0oLpJVq/2ieK8hF5G5HQ2V0a9+w1wA== X-Received: by 2002:a05:6e02:3183:b0:3d8:1d7c:e197 with SMTP id e9e14a558f8ab-3d88ed95743mr178064425ab.7.1745331202189; Tue, 22 Apr 2025 07:13:22 -0700 (PDT) Received: from [192.168.1.150] ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4f6a37cbb4fsm2354342173.16.2025.04.22.07.13.21 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 22 Apr 2025 07:13:21 -0700 (PDT) Message-ID: <028b4791-b6fc-47e3-9220-907180967d3a@kernel.dk> Date: Tue, 22 Apr 2025 08:13:20 -0600 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/2] Fix 100% CPU usage issue in IOU worker threads To: =?UTF-8?B?5aec5pm65Lyf?= Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, akpm@linux-foundation.org, peterx@redhat.com, asml.silence@gmail.com, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, io-uring@vger.kernel.org References: <20250422104545.1199433-1-qq282012236@gmail.com> From: Jens Axboe Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2EBC818000C X-Rspam-User: X-Stat-Signature: jqy3dzq7ypxh674bajpd5dema6wzk3md X-HE-Tag: 1745331203-411532 X-HE-Meta: U2FsdGVkX1/6Ky96F8yRcHzNdiSqjG0GiqhDiSr2t/Ya3gbjENCFGGJGrF8PlHddO9MqPr/u6TbhtzAbowsNWHFMAt3jcRm86DJ9Q7yI48wCzzj8VSDSKAl+L5nWon0gFwf2G25yHADMR6zlauIUArX5a6SD5lFigrzagGPTDcrFH83zsGYxdJ0IapkSfUgLDSKg/wub3MIoheCAoalLtOUgGle1XCKzfJJtPmoEQny4yZo6NwPAjfQTEkpGAPfjNCDmssM9bQM0sdXhbsIhwvChv9voyq0Vt/r4auxlMnw7e5jwM5sSRP3VpIbyUYA/w24Zqsqh7K+od6RbEhc3ZPSBGEwcbNdL668ML3HXl1xredu6+C+PJCughAVNQfqvvZ8uQxaShwRWyOuyYGuiVUxtDfcCn2tf8ecDRc3I8o8a8aH/L3BWQRc5xRBMID3cjKLiaKDy+8bM9iIsoLLJNFJ1WvfkBx6bjkq+3JRUJkA/ttB2USXvGVBkFXgaCm+wm4Vv075TAKLQdX6K3YmpY07iJRwsMwlmLerxg+QcCruJSQvb7X/ofMKCW2O60n7ZxR6pO4f4sh6VVXJZj4GVl4RzRzF3teM4BZKmA9joxvUtVl5aDSJkrL7UJPSxzLYdfOhw5yAyiCIh6itx0oskvhFrkl3rnmEguUdMikfHaFeiU2dNlOOxF8y6lCaB/D/eWlrCggXCTRfdx1aTFz4e9Eqvr5fRL5KZU8aqwEjbyhCGlL+D3hkCT9a5QKhmp5MzfbcA91Mn3bWR6vtmDv6Hm3xD8AbBhhk7F/HHSM1WLd4B1xI3/CcXPwwtPOY67m+1ZCfHLam3XxofYBnQax+aDKIajLfyS8K+u8TXLgUv0UDVb1dhXp1O03GPLTp3NV0pjzdLPsL5iezG07D2QjY43QP4JKBlnyVVHPkGAYu+0J6roVNkOAQ6q5cdaqm9qYotODxE+ofMuw9+PgO+H98 FTgAXPAE RqxkWvXyob6UEG/Sf5u1qCLBMHyzMCMhyrMxwUN1zZ9nn1+tcvGpCXNPPDdlBHpEztmokyR+pMfhaPctLaNYcSh1Idrs8NeVh5G/sGaylrIy9x6/kfs+qA9t+d0h7Mg1Hx+A7j5pALbhizy5WqrEDJbiAjnbnubHxrwkS1tTA81W8ji2vD8AzSfBKrDP5U5E3P8WxQgzQRLdtHTiTEcWKtYkQF4gwATdSnRkpbd4Ozjehw3ANM8St96KgODky5fUN2jUCWR3pYQqUW3+Rcyq2mKMUQL1RqFXik4VgfUj8Qv3W3FKdDxgdgxU776R/SFDJv09zlKOUlqX9mUmu1kje+hDNq9jeQ2yu/HG3xhmseOZ/7QsCC2EvdfCcBuc7fr9WKVNJInKfbAtkeBw/CJSfux3DKur1VETLIJYJ3Bv6puboVfk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 4/22/25 8:10 AM, ??? wrote: > On Tue, Apr 22, 2025 at 9:35?PM Jens Axboe wrote: >> >> On 4/22/25 4:45 AM, Zhiwei Jiang wrote: >>> In the Firecracker VM scenario, sporadically encountered threads with >>> the UN state in the following call stack: >>> [<0>] io_wq_put_and_exit+0xa1/0x210 >>> [<0>] io_uring_clean_tctx+0x8e/0xd0 >>> [<0>] io_uring_cancel_generic+0x19f/0x370 >>> [<0>] __io_uring_cancel+0x14/0x20 >>> [<0>] do_exit+0x17f/0x510 >>> [<0>] do_group_exit+0x35/0x90 >>> [<0>] get_signal+0x963/0x970 >>> [<0>] arch_do_signal_or_restart+0x39/0x120 >>> [<0>] syscall_exit_to_user_mode+0x206/0x260 >>> [<0>] do_syscall_64+0x8d/0x170 >>> [<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80 >>> The cause is a large number of IOU kernel threads saturating the CPU >>> and not exiting. When the issue occurs, CPU usage 100% and can only >>> be resolved by rebooting. Each thread's appears as follows: >>> iou-wrk-44588 [kernel.kallsyms] [k] ret_from_fork_asm >>> iou-wrk-44588 [kernel.kallsyms] [k] ret_from_fork >>> iou-wrk-44588 [kernel.kallsyms] [k] io_wq_worker >>> iou-wrk-44588 [kernel.kallsyms] [k] io_worker_handle_work >>> iou-wrk-44588 [kernel.kallsyms] [k] io_wq_submit_work >>> iou-wrk-44588 [kernel.kallsyms] [k] io_issue_sqe >>> iou-wrk-44588 [kernel.kallsyms] [k] io_write >>> iou-wrk-44588 [kernel.kallsyms] [k] blkdev_write_iter >>> iou-wrk-44588 [kernel.kallsyms] [k] iomap_file_buffered_write >>> iou-wrk-44588 [kernel.kallsyms] [k] iomap_write_iter >>> iou-wrk-44588 [kernel.kallsyms] [k] fault_in_iov_iter_readable >>> iou-wrk-44588 [kernel.kallsyms] [k] fault_in_readable >>> iou-wrk-44588 [kernel.kallsyms] [k] asm_exc_page_fault >>> iou-wrk-44588 [kernel.kallsyms] [k] exc_page_fault >>> iou-wrk-44588 [kernel.kallsyms] [k] do_user_addr_fault >>> iou-wrk-44588 [kernel.kallsyms] [k] handle_mm_fault >>> iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_fault >>> iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_no_page >>> iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_handle_userfault >>> iou-wrk-44588 [kernel.kallsyms] [k] handle_userfault >>> iou-wrk-44588 [kernel.kallsyms] [k] schedule >>> iou-wrk-44588 [kernel.kallsyms] [k] __schedule >>> iou-wrk-44588 [kernel.kallsyms] [k] __raw_spin_unlock_irq >>> iou-wrk-44588 [kernel.kallsyms] [k] io_wq_worker_sleeping >>> >>> I tracked the address that triggered the fault and the related function >>> graph, as well as the wake-up side of the user fault, and discovered this >>> : In the IOU worker, when fault in a user space page, this space is >>> associated with a userfault but does not sleep. This is because during >>> scheduling, the judgment in the IOU worker context leads to early return. >>> Meanwhile, the listener on the userfaultfd user side never performs a COPY >>> to respond, causing the page table entry to remain empty. However, due to >>> the early return, it does not sleep and wait to be awakened as in a normal >>> user fault, thus continuously faulting at the same address,so CPU loop. >>> Therefore, I believe it is necessary to specifically handle user faults by >>> setting a new flag to allow schedule function to continue in such cases, >>> make sure the thread to sleep. >>> >>> Patch 1 io_uring: Add new functions to handle user fault scenarios >>> Patch 2 userfaultfd: Set the corresponding flag in IOU worker context >>> >>> fs/userfaultfd.c | 7 ++++++ >>> io_uring/io-wq.c | 57 +++++++++++++++--------------------------------- >>> io_uring/io-wq.h | 45 ++++++++++++++++++++++++++++++++++++-- >>> 3 files changed, 68 insertions(+), 41 deletions(-) >> >> Do you have a test case for this? I don't think the proposed solution is >> very elegant, userfaultfd should not need to know about thread workers. >> I'll ponder this a bit... >> >> -- >> Jens Axboe > Sorry,The issue occurs very infrequently, and I can't manually > reproduce it. It's not very elegant, but for corner cases, it seems > necessary to make some compromises. I'm going to see if I can create one. Not sure I fully understand the issue yet, but I'd be surprised if there isn't a more appropriate and elegant solution rather than exposing the io-wq guts and having userfaultfd manipulate them. That really should not be necessary. -- Jens Axboe