From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A29EEC7618E for ; Thu, 27 Apr 2023 02:20:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0D126B0071; Wed, 26 Apr 2023 22:20:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E97306B0072; Wed, 26 Apr 2023 22:20:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5E8F6B0074; Wed, 26 Apr 2023 22:20:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C6FB06B0071 for ; Wed, 26 Apr 2023 22:20:46 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9F44B1C5E0C for ; Thu, 27 Apr 2023 02:20:46 +0000 (UTC) X-FDA: 80725567692.06.F92BF6F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id EC68F1A0012 for ; Thu, 27 Apr 2023 02:20:44 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XP42qjAw; spf=pass (imf19.hostedemail.com: domain of ming.lei@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=ming.lei@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682562045; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=GCPcaFdJ9SvoZ93cdbG0lFwwi9Thl1ubA5vESylOj3g=; b=QRQ24z9SYtG++WF7NJq6Mw4y7uQSLw5SAxx8LhZIaRTxAQtpNbQZgOOBnMZ7c6OZvVNzBz etbxEogfRrrb3HyJV8rphHjaJQfhEfBbiSK57dEtYV5vAnE4ghvjNQGOgL5rsvyOxDinVH yfy6TOTM3pxGTlAKuO82IbGV8QlrXx4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XP42qjAw; spf=pass (imf19.hostedemail.com: domain of ming.lei@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=ming.lei@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682562045; a=rsa-sha256; cv=none; b=i6/TOOxIJudjB0RDEIpgYVRl16kJzfyvh5GeVmOcdgMFqYw4ZS6unqJxk+TQUqDQMEogwu YL5i3+gF1ECRhShHuS0GueoWEZQPYihWppXNEfd1Zrf3S9iycTG2kocupVVUO2h7VbJaYc UK++q0BXA9kFAatwBze3dnOVBvOptgQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682562044; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=GCPcaFdJ9SvoZ93cdbG0lFwwi9Thl1ubA5vESylOj3g=; b=XP42qjAwM3lzgyaYuWSOHp9o1lxcxWVCnK4zHL5ozhraMfCE3kEEhFu2FLnhY1mkJExG0g g9Q8bs6cxwLeFZmJ1r84a/ty8TWOLpLwVwMDEM1AD5lCjVHo6WZ3Ey42z4m5SRZKuaiwKs s0TZsrOtVsJfE/DOgx1LsyauQv8mNTQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-605-aRep1uVGPCu1PlVejb15nA-1; Wed, 26 Apr 2023 22:20:40 -0400 X-MC-Unique: aRep1uVGPCu1PlVejb15nA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5842B29A9D28; Thu, 27 Apr 2023 02:20:40 +0000 (UTC) Received: from ovpn-8-24.pek2.redhat.com (ovpn-8-24.pek2.redhat.com [10.72.8.24]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C35681121314; Thu, 27 Apr 2023 02:20:33 +0000 (UTC) Date: Thu, 27 Apr 2023 10:20:28 +0800 From: Ming Lei To: Theodore Ts'o , linux-ext4@vger.kernel.org Cc: ming.lei@redhat.com, Andreas Dilger , linux-block@vger.kernel.org, Andrew Morton , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, Dave Chinner , Eric Sandeen , Christoph Hellwig , Zhang Yi Subject: [ext4 io hang] buffered write io hang in balance_dirty_pages Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: EC68F1A0012 X-Stat-Signature: smie59xjitcdygux7cuuu84bzcbk7jgr X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1682562044-715753 X-HE-Meta: U2FsdGVkX19y4mQlu5ONz/EEbgeOfuzFuW4W6jsKUP17mORKq6TLUpcU1snInV8dgR0Ff66oQYa1wTcnKtr6rx52PB3rl0sBi9L/W36mcV+gjlCqWgEjKBBXl6EOhQ0BivW88EDyJQJVH34hnBHk9cZYWhaLwKSc/C2UOBV5GWhcXxQeaAX+jGmSlDubYgf8jcUi1HZTbG/K18vpCyHaR5onkWRZkUNrh8+yn47i6jJqa+GQgdwjm1s+n3IJMaXVFw54Nbg8qXmpEIxMHMUTOhPQ1z8MhCWISbiCNlujMKDS304becLJ1ELuxyUr/hF0C55V6Fubm7rSegS93j6+werW+85pyPgU0V2Ugeh1Dsu+gWJko8YKR4ywqNnU8ILUXK0MeK1D40cBiuKZAkitnpD7Tlzvp8FLP+tf2nSx/CaMuBMxwm5UMQX/AnZXNob9sSzP1d8GNj44ujTRLXCHqAXGU+asmMvtZ8ul19FbBw9ozLeTZ//VOg7MA3a5yGcldrrqfp8qxADDABQpR8T4+QfIathMMX8VirYPZ6MMLGMYH48WDQ+tGYijS071WRFTS/8+RBldAsQ0uuI1LzSfWkiN1Cew4kQTiJMbkfOgyhB0kW8ggZiU9t+r4pxtiKTVYlUP5Ryv1MmE9+d8B7aKmTeoEdoNYDNvJ7n9OOt5lBu2YRBS5KeeuAuZjZ2UWrXpfZyk0IzDSxs9CVOQi3a5EV23aOdGt4qEqLx7CB1htA7itwsbrQyCUbdgaeVmsr25iYbIA+v5vA0k5/vmDVCbKiXJDlWYGpP/3RrBux24X/ClchtasSfQuRzyRgWblTSmQEGf5vhfdOfmSe+JrXxnX9N/nXcYg5H3QhXZJJmPc+ldJ3eF8hwdvlW1hSocQRETocNxbyRRxBvTuoG3DrVgr77vwC++7hKhCmjgGXMPv0+MunNk6797pj1YF1UK+OZsddTxo8hw6CD2eZ7/vC/ U2X2tVJF iByuo2nWdQ3Ou00NZEB1WFjgSrGoGdETY8+Y4NoS5KcryZGW6gEOQSKTyWjXpazLIvTJdcUE4mIQsprhRtSUR+ryAGZZTs32fRBruvt0lNmTAHA+k0kdBUHeM3x5bUXHfeoZuKBHzu+VTuPR089KmfeXweO3TIpx/+1vKv9218JT+pMiRZ4+MKVl4eESAUp6gsXC6jhzRxZIJWtkBIyiPjVh2vg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello Guys, I got one report in which buffered write IO hangs in balance_dirty_pages, after one nvme block device is unplugged physically, then umount can't succeed. Turns out it is one long-term issue, and it can be triggered at least since v5.14 until the latest v6.3. And the issue can be reproduced reliably in KVM guest: 1) run the following script inside guest: mkfs.ext4 -F /dev/nvme0n1 mount /dev/nvme0n1 /mnt dd if=/dev/zero of=/mnt/z.img& sleep 10 echo 1 > /sys/block/nvme0n1/device/device/remove 2) dd hang is observed and /dev/nvme0n1 is gone actually [root@ktest-09 ~]# ps -ax | grep dd 1348 pts/0 D 0:33 dd if=/dev/zero of=/mnt/z.img 1365 pts/0 S+ 0:00 grep --color=auto dd [root@ktest-09 ~]# cat /proc/1348/stack [<0>] balance_dirty_pages+0x649/0x2500 [<0>] balance_dirty_pages_ratelimited_flags+0x4c6/0x5d0 [<0>] generic_perform_write+0x310/0x4c0 [<0>] ext4_buffered_write_iter+0x130/0x2c0 [ext4] [<0>] new_sync_write+0x28e/0x4a0 [<0>] vfs_write+0x62a/0x920 [<0>] ksys_write+0xf9/0x1d0 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd [root@ktest-09 ~]# lsblk | grep nvme [root@ktest-09 ~]# BTW, my VM sets 2G ram, and the nvme disk size is 40GB. So far only observed on ext4 FS, not see it on XFS. I guess it isn't related with disk type, and not tried such test on other type of disks yet, but will do. Seems like dirty pages aren't cleaned after ext4 bio is failed in this situation? Thanks, Ming