From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 956ABC369C2 for ; Tue, 22 Apr 2025 13:35:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6FDB56B000C; Tue, 22 Apr 2025 09:35:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6ADED6B000D; Tue, 22 Apr 2025 09:35:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 575D66B000E; Tue, 22 Apr 2025 09:35:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3D3BC6B000C for ; Tue, 22 Apr 2025 09:35:03 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 75E10C1713 for ; Tue, 22 Apr 2025 13:35:03 +0000 (UTC) X-FDA: 83361775686.05.4351BC7 Received: from mail-il1-f171.google.com (mail-il1-f171.google.com [209.85.166.171]) by imf06.hostedemail.com (Postfix) with ESMTP id 56358180014 for ; Tue, 22 Apr 2025 13:35:01 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=VIxTU8Mz; dmarc=none; spf=pass (imf06.hostedemail.com: domain of axboe@kernel.dk designates 209.85.166.171 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745328901; a=rsa-sha256; cv=none; b=ksb8Kt3t/ivQrIBiWhvtOw/ksj7MT+6MFe5Z+3S/Lu5cgpLXEKVCwCBqsPx2gj93EuCBuC 1KnJ75hPxtcefof6QKppBHm5oYKUoTkSRd7YU5RVVDJN3j+PX39pcpIoKPa0xaLzUKBu0y WGJeCU7PZzhIDKpFHactavM7Q7rJEzA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=VIxTU8Mz; dmarc=none; spf=pass (imf06.hostedemail.com: domain of axboe@kernel.dk designates 209.85.166.171 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745328901; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ng1AJph91LR1w58lbhVEsRYbLXwmFH+WlPGVdw4yh0A=; b=i6N1Tovd/EyS5qg1Knw8TNmARP45TLOim3badMwlbaSlZ9uQKnQx8oOEEin5yYAn7Kqvc+ UqqTlEEdHmb624cS6clZOENZ89Z42DEENNz0ewh8LhhPfXeKcrR35yiOBEYBAKym++ENxS excWKXyZjVDnKe2l+ZnjVngV6C0eV6M= Received: by mail-il1-f171.google.com with SMTP id e9e14a558f8ab-3d92585938aso357475ab.2 for ; Tue, 22 Apr 2025 06:35:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1745328900; x=1745933700; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=ng1AJph91LR1w58lbhVEsRYbLXwmFH+WlPGVdw4yh0A=; b=VIxTU8MzPHbVOI4v7xcnhpbHPtkNhEKagNEFpe+0iS+GoVZlA1V3lmwU/kvlrjDcY1 umM74I/4jLTwIxz5ylsw1BFjaB/Iw8nRDNpW2sym8E7BPiYpGQ5mZIvZEWsfNOGvI64j ZkLKgUicCrm40e83sHL4CLg48zzvFWiE1BTcaljeQJgkU/Cmi8Z4nEGj9wwX1yfIRc8c RF1ln24hDyldful2lccB1BdqzYmoAwYTUFIN1n5X1aZT07uc7cX7rDQGsSBOOQWRMlk8 t/voDwp6W3c+K0XqhI+8B00WsYBKmIbnBDGyhyiK8q5C5q2rx1ffcdPRM2TgPCMKNHga dLgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745328900; x=1745933700; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ng1AJph91LR1w58lbhVEsRYbLXwmFH+WlPGVdw4yh0A=; b=gTjGU2fuCCaieZyOALeDTFs3Jd2eo7EVcGudNfgeu5pi7tCGYfBiAz4lGApKt9wj8P EXLbCK/QD4zllrxrlSLHmKpKRQXIeWGF+YwOzkIeglWDhENld9pE252RbHyKT5KHX1Ad dJw/LDMTN6xj3R7+5XGmh75rEg9HdxNhY5mZ1ewqLyJHBRbQxSA0213U9q9TAV2qyp+3 typnduIltVbZdGBPX6TCOTFd0IkorHB+k+yaR/o1I6eJN2MH0JtFtH1L00BrqpT21D9i M7PnoXqgS5kB2wyxEc7v2JeWp0E/tvmwWOp5A14R1jk69r7upQ5x9exQPwUgrixzXvzW 7j9A== X-Forwarded-Encrypted: i=1; AJvYcCV+YN3Z9HP1VNOeu/Pxj7xU57GDOPTEOW3yFfdRaEe7MxEE39HwALYSocwRtxbKijxfmTw4vJK1Og==@kvack.org X-Gm-Message-State: AOJu0YzLhkCe7vKZA3Dp3WKCjuvGzO0MOg+1+Whe4y66qpY8d4+dkf7a sCGiF0hfkMNSOAfyRoBuf6qnXlsSi/975ClJpC5g4ROqTZILI7hqM78+lh7nxoU= X-Gm-Gg: ASbGnctPOO7iIxqCRcx+5CKGP6WMvaQmvCN7BllGlOHxJlS8HCUFbS8rUyVakJSAjUd 7gVVg59QFdJVFgtg+/KzJBMw6fSmy2UsjKdjWDC645R8EfUB/N5ULPSKETQFoFOV7/gM6kKnQK4 ZQhID3//5WfmKeI4C5OnZAHHPQJAy5z5CqkMIl9kdlavxOul6VAnKmFRVEfqKwcZaKy/MGCg1Oz LLivRve9wM2l9/XcNIyr0CHkc9wJ5FL44sRbCq5qm6cFuEfVByOo2UKLKFywb9+i/eEWwqMOLOx RAISBbHsYwdz02G+15ls0cdzuRVzyZMlslxNr7z7hsM47G6g X-Google-Smtp-Source: AGHT+IHIKm2ycnI/f/1vykV7XxOgEYKLsp1CQzjQB79xzDp79m4CqVmAxO17uvtvLxPVOJiUh4ZBpg== X-Received: by 2002:a05:6e02:214f:b0:3d3:dcc4:a58e with SMTP id e9e14a558f8ab-3d88ed7c050mr142584975ab.8.1745328900022; Tue, 22 Apr 2025 06:35:00 -0700 (PDT) Received: from [192.168.1.150] ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4f6a3958e2esm2316031173.122.2025.04.22.06.34.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 22 Apr 2025 06:34:59 -0700 (PDT) Message-ID: Date: Tue, 22 Apr 2025 07:34:57 -0600 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/2] Fix 100% CPU usage issue in IOU worker threads To: Zhiwei Jiang , viro@zeniv.linux.org.uk Cc: brauner@kernel.org, jack@suse.cz, akpm@linux-foundation.org, peterx@redhat.com, asml.silence@gmail.com, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, io-uring@vger.kernel.org References: <20250422104545.1199433-1-qq282012236@gmail.com> Content-Language: en-US From: Jens Axboe In-Reply-To: <20250422104545.1199433-1-qq282012236@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 56358180014 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: zsih1efoojrs6xgg6mf4qf1ysu6aega1 X-HE-Tag: 1745328901-50421 X-HE-Meta: U2FsdGVkX18z+qPioTvLbarOWM87Kk61ktq2hgdeglLVzVlO21N9TT5Wxaqos/7iotIzDDhqbVA2bZ1wDJUCCuuHQHR39Eb61D5zw6kBKPpAihrup5J+0S9JTwqeAnqJe4uKCStd01KEGg7vWTKpHPZ1048uiivfs8MhASo8ue0ytByGxGbyKkIQStDtPuSfJSdVXUsx85HeQw+i5jiLbWYDG2FaHNCentHhhjZjLMkTni0IzSYYj7nif9A3zKnjd2t+Su0mjBOUZZyLgly5MJF92bOIIbMZEP6KRu3mecLg8dVpOYWTna/BuIQupaJcnmJfLNa9qoR2v3ZHt9pO2RNj44YTt3LcgRujjtvuV2LoPQ1HhqXGFXZLuiFJwHIwyHTHxNIp24qmGl1JAUFcS6mjbfh5jpV1UzgK1s8kyJ6HMjiFV6Ibk7wIMhXHOSmnieARAD8wh6sQmEJv7lrLiy2O9IF2uXJXZAh126bpmBsfO1fWDzkxJZJiMEY3W2KHBE2QjOragmxF4s3y5sKbrVtHMczSi7rvNX8cCz75cPDQDW85aufgX46t1yEGeFl10zZ7uRVEgdhgR3falFIrHIdzziMeec3qGvR3BLNKjIYv9Tz/sP3Qa1iWI7Bl+0W09T+VzY7nybAmKtL8sg/8PQnBbPELpAdziJmZ/znRmtSZlyVmmqqhpCxEuaLTkMF6FLay8kbmzeEFrSjjxa0O/MweklQbZPwUcyeRY8WWJB3kng55YlOIrTY0p0m+Fca6D1yJF/f7DK1TrbLh6eUP/Ixw6zI17aE/Ryi+IqrNQ6VVh/oLrBVxRszYHE2qdR+JwoThzyRFoQDQYmIdxSx9AqErtXYHQFvrxfs3xSxPBgtwJU+1k7JSnqZj5Sa8DJPzBtqS24pE4YaEf/7yw/+7qzgBU3IJLBSFTUATb7/xvnnzoh9TFqSNMy32j9h09TSG8+sGN9N42r0nr/QaFOv UG7+Z6+C i21ZAOPAcfRcPgW0sy/STcHP676Cra8cl6WdBBNtl6CO8Cyn0LbXX5fMl3Yr17iwrvnF2huBPnu9rk+z0JKVF6G1k/LIin8VTX6Cn8bUYjn4Iav+r3aLW0lGig3hkzgjj/esV2xY48WeundwGP3Mbwvod4MugBaNzYlM1kNm4f5BfAqUsvveRmg5LiH5wbI+wt9Vy4Y+0vyGqJTgOSqmAkXT/6VfsiDwgqTSGCNTF+sBnIrjdqHsTaOA0ulxChDYmBxbs06o8mpbvIhyFiSJ4bT0dMJRBPXoI27InQ78nZA2JiqSvrJP+dsZXazPT9J/2CJK78q2r6c0gSkZl95RPjs6X2kHpxxvgiOv1p4mvmeKxX7t2/vCfS/ipBQtbRlmvKxxvKZM7wrVpwHvrTAvxr3ZoiIGL7PMyEMPbR/jenb2etxs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 4/22/25 4:45 AM, Zhiwei Jiang wrote: > In the Firecracker VM scenario, sporadically encountered threads with > the UN state in the following call stack: > [<0>] io_wq_put_and_exit+0xa1/0x210 > [<0>] io_uring_clean_tctx+0x8e/0xd0 > [<0>] io_uring_cancel_generic+0x19f/0x370 > [<0>] __io_uring_cancel+0x14/0x20 > [<0>] do_exit+0x17f/0x510 > [<0>] do_group_exit+0x35/0x90 > [<0>] get_signal+0x963/0x970 > [<0>] arch_do_signal_or_restart+0x39/0x120 > [<0>] syscall_exit_to_user_mode+0x206/0x260 > [<0>] do_syscall_64+0x8d/0x170 > [<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80 > The cause is a large number of IOU kernel threads saturating the CPU > and not exiting. When the issue occurs, CPU usage 100% and can only > be resolved by rebooting. Each thread's appears as follows: > iou-wrk-44588 [kernel.kallsyms] [k] ret_from_fork_asm > iou-wrk-44588 [kernel.kallsyms] [k] ret_from_fork > iou-wrk-44588 [kernel.kallsyms] [k] io_wq_worker > iou-wrk-44588 [kernel.kallsyms] [k] io_worker_handle_work > iou-wrk-44588 [kernel.kallsyms] [k] io_wq_submit_work > iou-wrk-44588 [kernel.kallsyms] [k] io_issue_sqe > iou-wrk-44588 [kernel.kallsyms] [k] io_write > iou-wrk-44588 [kernel.kallsyms] [k] blkdev_write_iter > iou-wrk-44588 [kernel.kallsyms] [k] iomap_file_buffered_write > iou-wrk-44588 [kernel.kallsyms] [k] iomap_write_iter > iou-wrk-44588 [kernel.kallsyms] [k] fault_in_iov_iter_readable > iou-wrk-44588 [kernel.kallsyms] [k] fault_in_readable > iou-wrk-44588 [kernel.kallsyms] [k] asm_exc_page_fault > iou-wrk-44588 [kernel.kallsyms] [k] exc_page_fault > iou-wrk-44588 [kernel.kallsyms] [k] do_user_addr_fault > iou-wrk-44588 [kernel.kallsyms] [k] handle_mm_fault > iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_fault > iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_no_page > iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_handle_userfault > iou-wrk-44588 [kernel.kallsyms] [k] handle_userfault > iou-wrk-44588 [kernel.kallsyms] [k] schedule > iou-wrk-44588 [kernel.kallsyms] [k] __schedule > iou-wrk-44588 [kernel.kallsyms] [k] __raw_spin_unlock_irq > iou-wrk-44588 [kernel.kallsyms] [k] io_wq_worker_sleeping > > I tracked the address that triggered the fault and the related function > graph, as well as the wake-up side of the user fault, and discovered this > : In the IOU worker, when fault in a user space page, this space is > associated with a userfault but does not sleep. This is because during > scheduling, the judgment in the IOU worker context leads to early return. > Meanwhile, the listener on the userfaultfd user side never performs a COPY > to respond, causing the page table entry to remain empty. However, due to > the early return, it does not sleep and wait to be awakened as in a normal > user fault, thus continuously faulting at the same address,so CPU loop. > Therefore, I believe it is necessary to specifically handle user faults by > setting a new flag to allow schedule function to continue in such cases, > make sure the thread to sleep. > > Patch 1 io_uring: Add new functions to handle user fault scenarios > Patch 2 userfaultfd: Set the corresponding flag in IOU worker context > > fs/userfaultfd.c | 7 ++++++ > io_uring/io-wq.c | 57 +++++++++++++++--------------------------------- > io_uring/io-wq.h | 45 ++++++++++++++++++++++++++++++++++++-- > 3 files changed, 68 insertions(+), 41 deletions(-) Do you have a test case for this? I don't think the proposed solution is very elegant, userfaultfd should not need to know about thread workers. I'll ponder this a bit... -- Jens Axboe