From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 771DEC001DC for ; Wed, 26 Jul 2023 15:30:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F056E6B0071; Wed, 26 Jul 2023 11:30:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB50D6B0072; Wed, 26 Jul 2023 11:30:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7CD26B0074; Wed, 26 Jul 2023 11:30:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B11026B0071 for ; Wed, 26 Jul 2023 11:30:07 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 659DF1C9B8B for ; Wed, 26 Jul 2023 15:30:07 +0000 (UTC) X-FDA: 81054148854.20.CB989CF Received: from mail-yw1-f174.google.com (mail-yw1-f174.google.com [209.85.128.174]) by imf17.hostedemail.com (Postfix) with ESMTP id 44B7A40023 for ; Wed, 26 Jul 2023 15:30:03 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=IfQOikjT; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of tom.leiming@gmail.com designates 209.85.128.174 as permitted sender) smtp.mailfrom=tom.leiming@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690385404; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Lrg+45f8OyZpsYrZsCtiNxFu62qT5zsFL2TVU9KzV9Y=; b=c4sXV46JW6H7uELZJRENmdgH6j7d3+t0FFn0XaOsMyCDnxCuKcaEEhbk5Nuj8T6/xYh4NG uJNr1omOEwVOCDT/FUrqzffxFJ6rCu0CRpHsJ3YTzPqldNlupNdONby14K4VOn++RNKIMk OJfAxWlUkEYUlOiGYUUBMKiXe7n1hGw= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=IfQOikjT; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of tom.leiming@gmail.com designates 209.85.128.174 as permitted sender) smtp.mailfrom=tom.leiming@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690385404; a=rsa-sha256; cv=none; b=hvczTdGiDHMErlJ/ZJb/jkoF/jQCR269VqTqbGnraleAr9IPQUUI21ADHeCpvjABAzGavX qvhEcSZHEHrUTslF47H0nh1nANb6Xy/0avqMstmMIYHxQY7FYRyyoI5KpdeMqDKU2K5YvZ wckNDhiE+GWkQmTpSS3K3Wb0iwXxBkE= Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-583f65806f8so41313117b3.0 for ; Wed, 26 Jul 2023 08:30:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690385403; x=1690990203; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Lrg+45f8OyZpsYrZsCtiNxFu62qT5zsFL2TVU9KzV9Y=; b=IfQOikjTb54JcoQOnhR/3Fq0FlEKu8EnycAFmIQLw1zLeRNxqDoNAwmyVIhZkPF/QT +JKHfyxiDvW04jhJpcOFcj0C2PKdMmWCrunMR2AzxG0XfyRJJBMCaQhTRoLGmYm3IYXd lGIO55xgdc1kvjTJ6XJE1yNt1XIOO0MtUoCMwigMn0OAQWsmSvhujLJ15v6B0t2yd6pw X0MidQmtlMJG6RVSarnypuX/llFdlPvZ6IDlkQkBN9OSZUjnDHzhCWSbuvBlNcp7FRAP sfVGXpcdfaxj9zK9FMZMTyJODEA5x5SDojiVCcamwVuuOafYQuYqurs6a8cIWKqW1bCF zRVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690385403; x=1690990203; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Lrg+45f8OyZpsYrZsCtiNxFu62qT5zsFL2TVU9KzV9Y=; b=d3dkCDjZWQZilCJA3+B+i5W+9r3wx6tiXciz3sNrpeP24qA7BWu51YpWxt01um0RyC wnzucXQZtXowHHPdplpVP1tkw4IsoMD+GcAF9gOo390PE3wgU3bd58mcCTzzonl1mfcN 1BkdHV7kkDgpb9tx7w0tHW7ptzvd++qTkqY+8+aDlunDlT3QoUlgguwtlbi/i+6TTsOz 9Anq/BsJ/oTTM2hCKQw5oVL4okx61X3VMB25kXTKvrWYJhADhbTovpY3s1jJ/devFmTR AbnmiaqFOrF2JNSSDPEm2giGSc91j0LSLTGwMl0CrPpWExmkQPSDjxPa+80MKLA0Oju5 T++Q== X-Gm-Message-State: ABy/qLZspdqwRfZyNNEcMRZPbExBinRKO64L/41m1y9fluzDuXKpJbPd qcvcT7qNca6DflLzgklDntKa2urlTC9+Mh6P620= X-Google-Smtp-Source: APBJJlH9c/Cm1Gn1k/bqGcIifEtT5WA6JYO4NWDRh17eGkNuN//LPGYMjb5PrsKWtVwk5jtZLub4b2CzwC+F8JFg/Zw= X-Received: by 2002:a81:a151:0:b0:584:5e7e:40c1 with SMTP id y78-20020a81a151000000b005845e7e40c1mr533351ywg.28.1690385402927; Wed, 26 Jul 2023 08:30:02 -0700 (PDT) MIME-Version: 1.0 References: <4919.1690365747@warthog.procyon.org.uk> In-Reply-To: <4919.1690365747@warthog.procyon.org.uk> From: Ming Lei Date: Wed, 26 Jul 2023 23:29:51 +0800 Message-ID: Subject: Re: [dm-devel] Processes hung in "D" state in ext4, mm, md and dmcrypt To: David Howells , linux-block Cc: "Theodore Ts'o" , Andrew Morton , Song Liu , Christoph Hellwig , Alasdair Kergon , linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, dm-devel@redhat.com, linux-ext4@vger.kernel.org, Ming Lei Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 44B7A40023 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ys13s7pg6ukos6jfx7y4tagat9dg494u X-HE-Tag: 1690385403-263488 X-HE-Meta: U2FsdGVkX1/7HYrjyp0y/jQYHN6UNJ8BZdkgbVvSLYBpc0X3QPszmoQ+7rhucdKaG90+CdHGNPmlENFMr2lqGeOPgk/0TODuzEXLR/RI4ROjKS9XrbCTbxuctQFI6UMPsdUQupWajtGAWUwxxcJ/R76G960SEnIydWvqHkuFEnYzCLgCWUdNzg5Yyk+6gYQJjNguoDvvcFul8PKvQ2mp4DEcMggHoD/6JB9xaSTY3bnazMfBWA8rMcb+0j3m8hJVONuOAPuvKIYvUiNkT5r7deCF4nP16Hlaalz/z1y9ZatuFMFafZ3eAyy6P8RkHvNLa/jTxS5sSDkOhjZVjbJw+hHObzSZ9V1USyqLGMj2sWq81ixyVzGAEbfmMJnY7WDt7XH54777OpKrOMk9IZEe+Hi72FBeG5/e2Dnfmq/EwMitfU3dBl17dONWTLEPs1qbQjp760JA7uPpVTBcpAyXETwSy/zuNNcOX6LgbAnqLbzKGutfpfiMhq1CTz8cWHLtPo3ta5uRku6Svd+30GkwrQxnWn2Z527Y4hNqXKPSKWopni3kobw/oZCZgAbCedq2wY0K5pUh9F3s/2LOqF52h7Fat34r7GXjMwWw8KpT9cS7Ya5g+05deEhgX4xZjTCpk7Or2zQFMIf64rw8z7s0stMmaeyL0a4SS7XRVeu2pal4l2XR5Dz6Bqg2hYlHj8BC7OrfXNTnd9bMWXuXU3/TWL//hDP+O5ag/I2hPZbPJGnZ3cxTzEjVTgVF7QzpIiMZQEPG84W222eW29vSKvJc8Dm8tBTUIoWrt8b98MbVjU+0VlWY9qGHyxMexFElgyM3/RGKlPxCyz54L2CCP7qfgEP4QoeXIPjNiHXgc94lcHskf3LUuOuT/oSAbZGJP3ZqgP9cMMFn/DsSE7h7zTn6/aD66HcKlQAuO5z1c9YkCMu3eOUl7K8upm7dkWoyqHR9j4MugtuZn6+H6eYN+IR tCOlIa3D HFF04hk3XXOTTtPV0uMaWuXqpknGfQFlwoxLcVibJtcag6D5ZwUkEp4XpmB9wDhD18AoQhQq1qH/EsOHiFLxpL9R5Hfb3BG+1j4+hqkfzLDSfP4rsQ7iuphXgsEDkjsZZciVFm55vNb5dU0nmnO9BK59qL+uSdRjskEyhEc8ykkLvmrKlfgg6SOxfhvMNxLm0ys10Y3ww48CUliE/R1PU7lAw6tvs1VZQXk55wjiGBEIk7xMccU2olNLzOoQqqjzZku7qzNS0USEXZsBnd7/EA3ZV/8TDs2jBvUlOcI5p08/P/JcOh9MJ0wPXGzaHmzNzQlS/JRhRmKPbUMqkEykN2lgHVpspKT3zRyQF8YxHGOYmxojKwrD9eu5Ka0WozvucipkY4tsf1q62p+FXWaiLS9Kxcy41gr9oFg8kvOrjB5UrydL1eJ3mXpF3ifphxbJ8TOOz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000134, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jul 26, 2023 at 6:02=E2=80=AFPM David Howells = wrote: > > Hi, > > With 6.5-rc2 (6.5.0-0.rc2.20230721gitf7e3a1bafdea.20.fc39.x86_64), I'm se= eing > a bunch of processes getting stuck in the D state on my desktop after a f= ew > hours of reading email and compiling stuff. It's happened every day this= week > so far and I managed to grab stack traces of the stuck processes this mor= ning > (see attached). > > There are two blockdevs involved below, /dev/md2 and /dev/md3. md3 is a = raid1 > array with two partitions with an ext4 partition on it. md2 is similar b= ut > it's dm-crypted and ext4 is on top of that. > ... > =3D=3D=3D117547=3D=3D=3D > PID TTY STAT TIME COMMAND > 117547 ? D 5:12 [kworker/u16:8+flush-9:3] > [<0>] blk_mq_get_tag+0x11e/0x2b0 > [<0>] __blk_mq_alloc_requests+0x1bc/0x350 > [<0>] blk_mq_submit_bio+0x2c7/0x680 > [<0>] __submit_bio+0x8b/0x170 > [<0>] submit_bio_noacct_nocheck+0x159/0x370 > [<0>] __block_write_full_folio+0x1e1/0x400 > [<0>] writepage_cb+0x1a/0x70 > [<0>] write_cache_pages+0x144/0x3b0 > [<0>] do_writepages+0x164/0x1e0 > [<0>] __writeback_single_inode+0x3d/0x360 > [<0>] writeback_sb_inodes+0x1ed/0x4b0 > [<0>] __writeback_inodes_wb+0x4c/0xf0 > [<0>] wb_writeback+0x298/0x310 > [<0>] wb_workfn+0x35b/0x510 > [<0>] process_one_work+0x1de/0x3f0 > [<0>] worker_thread+0x51/0x390 > [<0>] kthread+0xe5/0x120 > [<0>] ret_from_fork+0x31/0x50 > [<0>] ret_from_fork_asm+0x1b/0x30 BTW, -rc3 fixes one similar issue on the above code path, so please try -rc= 3. 106397376c03 sbitmap: fix batching wakeup Thanks, Ming Lei