From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67136C41513 for ; Wed, 26 Jul 2023 19:30:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B02AE6B0071; Wed, 26 Jul 2023 15:30:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AB33C6B0072; Wed, 26 Jul 2023 15:30:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A16E8D0001; Wed, 26 Jul 2023 15:30:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8BD186B0071 for ; Wed, 26 Jul 2023 15:30:51 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5C7B2B23AB for ; Wed, 26 Jul 2023 19:30:51 +0000 (UTC) X-FDA: 81054755502.09.C993F6F Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf04.hostedemail.com (Postfix) with ESMTP id 25ABF4000B for ; Wed, 26 Jul 2023 19:30:48 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=SVgBXvcr; spf=pass (imf04.hostedemail.com: domain of akpm@linux-foundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690399849; a=rsa-sha256; cv=none; b=UI9BeA4TEQMi2zVLbXNmz3vcubXiVfTXUwYdjr65PruTmeVp5O5U8gO5lPTPkzWRtEWZfT zo6Rlrdsm1G1SPzcLzndAE23po1YrqArXuZqPV0Zl/bhXUvkGeH4z7m0X9gTkORHcbWASl JjNURiqNSrfD7dESD1nexec9VF8S40s= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=SVgBXvcr; spf=pass (imf04.hostedemail.com: domain of akpm@linux-foundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690399849; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7f1qxe+nsWg+W9Z0H2K567+nQnYT0vl2apEw8vfFYwA=; b=fcjagBkfpn3RvwK8YCMMb+FQ4C4oloqXyayJ/GXwONFX4B2fBfw/Tt+ADJx95rZeWBgibJ rkXS7Bcm5h/Epwq9mm2BkB/BU9FCw8Avsv4Jj7xp9NkmoIf8BX3vlRUeS6hTNl6d5xdDNc /VfFuYWrDsGz4igX7YrMNTXoNE6jLEw= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2AFA561CA5; Wed, 26 Jul 2023 19:30:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EBEC3C433C8; Wed, 26 Jul 2023 19:30:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1690399847; bh=RV8tXL7RQ7wnolCqR1y4PyPALTKooCyEOhzhJRXcm/I=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=SVgBXvcrYNhex0+1z06NCyli0sUblvDQ90P7N/xe5kJ0+Q0OeP9lBlMuWohBd599E X0HwNQlpDq4CCkcsveKBraRfvaJNSyGF5bPHYv15pSAWcbsia2p99BVqHMVZ9ZvqOY 0doZpYRXWnSGvBupT8LtftvO3Yru9jMJIrV7gPkQ= Date: Wed, 26 Jul 2023 12:30:46 -0700 From: Andrew Morton To: Ming Lei Cc: David Howells , linux-block , "Theodore Ts'o" , Song Liu , Christoph Hellwig , Alasdair Kergon , linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, dm-devel@redhat.com, linux-ext4@vger.kernel.org, Ming Lei Subject: Re: [dm-devel] Processes hung in "D" state in ext4, mm, md and dmcrypt Message-Id: <20230726123046.a001b6963da19ca883045271@linux-foundation.org> In-Reply-To: References: <4919.1690365747@warthog.procyon.org.uk> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 25ABF4000B X-Stat-Signature: yexrrhpj5cozfufsbihzbi3oqdakojzc X-Rspam-User: X-HE-Tag: 1690399848-573490 X-HE-Meta: U2FsdGVkX19OcxXtH7IXxAy4w1OFY1wUe+Rj4iiCG2UBvC1sZgpNm92DXArRORR10GcKR3bYICvhPbIz8tHljEvQYcR7sPRLrrqAoONfBXbPZMVdVRWwWz1z41Kk23rlAp/TrGM1rgSAjsmUokYX4stTlwn88ioe+jALG1jia8gtBepY8/8wfXHDYFkpYNms83MeJaj2iKy2I3bQuxkWw4towFflsArGABJXCYVEIZy7EEpplVq+5cH/GVOEXpfFZdYavnHW+ekioq9x7wTuKv2r4IF2WLKSn4m/nokgIFpRRwvhuzvUKswTs0yfUsmQCjS3f69mQzPjirL0izUMU7ss9f+VAMoCRRWHLjAkw592kRP6sBnIAUdRnKcwTdTSAZGdufGuo8hdOAwAGVWJZZXIYzeWNkF7KbAKXX46yY+q/qKWeevxB7vIMkyTvtm2kM555jC5BCttKlIz2iOqRYF81hSVEfylFkeqzHwqEt8MFjehKm48dBpd8XjWT+9xgVh0nrmocYNgO30zEZTO7jU3Gns7heyWNmedp6MP07O52rB6gcXGvFZGcdfe8EiiWouUs+Bf1whE/GxcxvtnBgKcfjXXj+4O2yqlzXvPwU6jPi1bheEQ90t3hoku1NTnd5ynGpy58V6IZkfElcjPLYtxFtUIqWP7czClKYaD5schP8URoOpYZfhxd83tn9zUSSc1qCWhe80JVNaPEHld1pSSPMUC7R7CTGGzvIxuMMlIZY0lSKz6xRZbX24vQ0DXV6faGcYIu824bjquGOjO24GwOZWb3QgkIyepxBDvRiLpTgfCk4OL41bMaoqQTURQjiQSgATeDcPwPX+dc43M0EyQ1JUIZiC4BawLdkc84mQjWxBlHOshavH70IwnYWDam60YNnNry/nnkJwGjQB/oGwsX8gmggqLXkXZlIrJ4L8XcqjuNSW0Wa0H24e8a2/5ZXUMwLKhHbvxMZZ7LfN DynHoIP5 KLeT/7D8WQyji9xvx5XR9yBTB76+KS/NsX2nRSzZtVx41eG82BVLc9qSHljMAzAc2hNzOkwE6+anhRrQ6wrYoDjqpJT7Uup5x7mk3ntFofCh7QTGZINHVEC3QLiVn2tD5EcGs6YfX3DcGdAVSWAu1gdlMmocEQf6rf7d9BaQNz9m6M/4v2cJoXqn6FGyljyZDxRB1KxhpfIgrefVwvnAnzEtewy8S8l9Soz5Bb0bBWiL0MZLBWrgizp4I+twwnu3Evx+cNhssPjaPUMXXUpfWVL/vUjOKtmLXUT8aPA/ZhraXLKkhv8kC3TsvvxiGcp1cCrgPTNbK0huVbcokkAtdo+g+5Qdx1MZ1mN3mKl/Jgv+a+3gH70bN1Q62Iz3x9UU1v/1zq1UBgt/fHDeIBQvsXi7OMA6xj4mfM6hPh4txKP08ys6zkXra7tb15FPxrZ50iCCe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 26 Jul 2023 23:29:51 +0800 Ming Lei wrote: > On Wed, Jul 26, 2023 at 6:02 PM David Howells wrote: > > > > Hi, > > > > With 6.5-rc2 (6.5.0-0.rc2.20230721gitf7e3a1bafdea.20.fc39.x86_64), I'm seeing > > a bunch of processes getting stuck in the D state on my desktop after a few > > hours of reading email and compiling stuff. It's happened every day this week > > so far and I managed to grab stack traces of the stuck processes this morning > > (see attached). > > > > There are two blockdevs involved below, /dev/md2 and /dev/md3. md3 is a raid1 > > array with two partitions with an ext4 partition on it. md2 is similar but > > it's dm-crypted and ext4 is on top of that. > > > ... > > > ===117547=== > > PID TTY STAT TIME COMMAND > > 117547 ? D 5:12 [kworker/u16:8+flush-9:3] > > [<0>] blk_mq_get_tag+0x11e/0x2b0 > > [<0>] __blk_mq_alloc_requests+0x1bc/0x350 > > [<0>] blk_mq_submit_bio+0x2c7/0x680 > > [<0>] __submit_bio+0x8b/0x170 > > [<0>] submit_bio_noacct_nocheck+0x159/0x370 > > [<0>] __block_write_full_folio+0x1e1/0x400 > > [<0>] writepage_cb+0x1a/0x70 > > [<0>] write_cache_pages+0x144/0x3b0 > > [<0>] do_writepages+0x164/0x1e0 > > [<0>] __writeback_single_inode+0x3d/0x360 > > [<0>] writeback_sb_inodes+0x1ed/0x4b0 > > [<0>] __writeback_inodes_wb+0x4c/0xf0 > > [<0>] wb_writeback+0x298/0x310 > > [<0>] wb_workfn+0x35b/0x510 > > [<0>] process_one_work+0x1de/0x3f0 > > [<0>] worker_thread+0x51/0x390 > > [<0>] kthread+0xe5/0x120 > > [<0>] ret_from_fork+0x31/0x50 > > [<0>] ret_from_fork_asm+0x1b/0x30 > > BTW, -rc3 fixes one similar issue on the above code path, so please try -rc3. > > 106397376c03 sbitmap: fix batching wakeup That patch really needs a Fixes:, please. And consideration for a -stable backport. Looking at what has changed recently in sbitmap, it seems unlikely that 106397376c03 fixes an issue that just appeared in 6.5-rcX. But maybe the issue you have identified has recently become easier to hit; we'll see.