From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFCFBC4332F for ; Wed, 1 Nov 2023 10:15:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 245CC6B0206; Wed, 1 Nov 2023 06:15:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F47B6B020C; Wed, 1 Nov 2023 06:15:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06F4B6B020B; Wed, 1 Nov 2023 06:15:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E7AFA6B01FF for ; Wed, 1 Nov 2023 06:15:09 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B73641CA93F for ; Wed, 1 Nov 2023 10:15:09 +0000 (UTC) X-FDA: 81408977538.16.84C4181 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf20.hostedemail.com (Postfix) with ESMTP id 84CBC1C000F for ; Wed, 1 Nov 2023 10:15:06 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=JBrOF4N+; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=b83TNYrC; spf=pass (imf20.hostedemail.com: domain of hare@suse.de designates 195.135.220.29 as permitted sender) smtp.mailfrom=hare@suse.de; dmarc=pass (policy=none) header.from=suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698833706; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9jjfndRN0o/qN0WgqtkcVkame02NBTvybODA5DUFQdM=; b=32BrZ528SesWZH6POAhSyadgZYrwLtnhLvA1Zej2/+yeDn9zL6RqxpHL5bK9cyMFUkO1Gk nNceR45yIOImdBTpf8Qn9MvEFi38N75QxwqzyBziae3uCRfCqhfPMxHTmpiXGczuUbC7bm HfNpAOMiXtNdsueZyx+vy2H0bfANLjY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=JBrOF4N+; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=b83TNYrC; spf=pass (imf20.hostedemail.com: domain of hare@suse.de designates 195.135.220.29 as permitted sender) smtp.mailfrom=hare@suse.de; dmarc=pass (policy=none) header.from=suse.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698833706; a=rsa-sha256; cv=none; b=D0ECqH0XZSwn2AKHJO7tFEsrSUlNr0TbVb/ZmFwuiiaqrEY0c/E5K76y1VP9zvUBbizPFC 4xaI61SK+ZsPqXai0oIc+rUWpcTkGHobMb2eS5OdP4cmBJzPkFr+O7do0euwZMhuIRlaU4 ix9dHHbTLyHrrWS1q83AqlemZjk1Bjk= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id B28341F74D; Wed, 1 Nov 2023 10:15:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1698833704; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9jjfndRN0o/qN0WgqtkcVkame02NBTvybODA5DUFQdM=; b=JBrOF4N+H7Qu68rgXIVpiR4ij70g9zGB4RB2XwdJTf9uHOMn1LSON24WmYk6DmjD9/juzp q5pHyWnCi8n4KVVn4jYMiJoaOe3x/rdleeCkZnIvzhtyf+onXKJoMs9E/VUeEB/jtWok5g IYQ8IOVD9xWipD4chrwK72kEDtPtOuo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1698833704; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9jjfndRN0o/qN0WgqtkcVkame02NBTvybODA5DUFQdM=; b=b83TNYrCuySV4g9tpLH7UrvpqfJGJ9NVzmB5hA8RmFun7QJ+jVo/tD9DOF1USrlTG9hFqA 8U6fVqecOGuBhhDw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 41DD41348D; Wed, 1 Nov 2023 10:15:03 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 5ZCsDCclQmVCGwAAMHmgww (envelope-from ); Wed, 01 Nov 2023 10:15:03 +0000 Message-ID: Date: Wed, 1 Nov 2023 11:15:02 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 Content-Language: en-US To: Ming Lei , =?UTF-8?Q?Marek_Marczykowski-G=C3=B3recki?= Cc: Jan Kara , Mikulas Patocka , Vlastimil Babka , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org, ming.lei@redhat.com References: <20231030155603.k3kejytq2e4vnp7z@quack3> <98aefaa9-1ac-a0e4-fb9a-89ded456750@redhat.com> <20231031140136.25bio5wajc5pmdtl@quack3> From: Hannes Reinecke In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 84CBC1C000F X-Rspam-User: X-Stat-Signature: yg5b544p9gkhoy3or6w8krjqj37zfbsc X-Rspamd-Server: rspam01 X-HE-Tag: 1698833706-364057 X-HE-Meta: U2FsdGVkX1+QWKV8RiBnQg4pqPgfg5nnASJT1HCTJ6BBcXlX43gRULofAlQ1is5DlChiB+d94KTxsIB9ggTBHccNkYHJiuc5AqKDPOQDC/asVtXGE8300lRGEU4DJEqc4Pmen0F0Fo6z/Ru0O95Fag9HarpOquz0GaMnJNz4JqIpQwalsNTJCEG8B+pal5URZs/Wlgfk5WZDcQS0lHgpk/9XJaHoM96B+jreHIQ8QAH584P8tDbCbiXxjREkIQ7artVbVG5Sl8PM9nbg3QLLrtR1UHbUXa+PuunmCtKj1fSVij+cXt0kxJzJoIif9MbUQU5rmr2w6gnT9+bfNMZaI5WCyfxoJYC6OVNpuEkaZ0Z7b10IZOtWeuQR8+EE5DvKt2iKt4UCad5Nn6URmGsTF0zYUjUbMkO6wdrYqIV6Jc57PCH9wjRdDZUb2iF9RB6gPzn32J810JaaPv7Gq+ak1VEuEBrsXzytvPpDQ7/fgFGzrFop4cexPKMYIqXFqxoq3z7NsT5TWvTWd47TfJkg3atgGVdQVLmiVSsVrljHDOHuNt7N2eP/461DaY2RPnoU5T456hkEBLzodpFmP/c6XvUHKn4uEJJD33C9E6R0rQHYxTbqv6RxN4AUYqcvJDBvMbKYfRSd55iQ0g6fYnRZGiPvAyYEeXJidRFYMy4G3lTzI38TqaGjaWsjAMY8tuEMfdByHBjhXsmK0RDJgSCPVrpg6ZkPNR9SqsryplIaP0GDHZYyzRDmcKxX8ibW9YcSUMLojdFbmhmBYsrGzvUM0LBP5B60UntrGUkKpTuijsN6dCVkcDcNeBVTz1ZCMBa1ibSfbf1Mmb+LxUmwhsrytrtgKRsrlafazZ4gMemjA5Jjk8hOnm+vWU2lPZDi0YngStvaUryyBnbErNIs3M5iELTpMB4Rg3cK5lK4mu6heFzdPS0u0BY1wazqAmnvq8CizapStUe/i4sAIZQWtWr Fk+UBtGZ K7C8fgQrTwYC8IzetMK+dTEnQHEEWX3ioSz7wFyCNnMIOAw4b9UymE3u2ujdiygsR0F1+tLAsqiLJLSu+yXiCMwIknxgsa/sfROWVx+SELaybFYBN8iIgnEOIky9zP44mbmcO7+0mN82rTaCY2LMf+h6OM9y1mP3ghbK9+SPLatNelHaK98LyRTLArBj16XBwffGdSSM7+waSdoe5+eCCNgAUyUrCJE8qX9um8goIeaYselLGsLKUVy/8PDx7iU+u/XbmMpuL/5WBauDvdNH/EPuII0e7HGWjVisRT9Zjjt7lT1LVNGZliXeTHKujVCMQUf4mdi5D/YYVMdVT8fJjKN1nKCrmD+H51d8QSzeu2cHUajmBtQ7ofx7iiq1OUjLBQVLTtCJEIcFKSyc3g2R5nG4Nfck77V2Yqpq4CliAzNBFF9mIav7/5kIkcg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 11/1/23 04:24, Ming Lei wrote: > On Wed, Nov 01, 2023 at 03:14:22AM +0100, Marek Marczykowski-Górecki wrote: >> On Wed, Nov 01, 2023 at 09:27:24AM +0800, Ming Lei wrote: >>> On Tue, Oct 31, 2023 at 11:42 PM Marek Marczykowski-Górecki >>> wrote: >>>> >>>> On Tue, Oct 31, 2023 at 03:01:36PM +0100, Jan Kara wrote: >>>>> On Tue 31-10-23 04:48:44, Marek Marczykowski-Górecki wrote: >>>>>> Then tried: >>>>>> - PAGE_ALLOC_COSTLY_ORDER=4, order=4 - cannot reproduce, >>>>>> - PAGE_ALLOC_COSTLY_ORDER=4, order=5 - cannot reproduce, >>>>>> - PAGE_ALLOC_COSTLY_ORDER=4, order=6 - freeze rather quickly >>>>>> >>>>>> I've retried the PAGE_ALLOC_COSTLY_ORDER=4,order=5 case several times >>>>>> and I can't reproduce the issue there. I'm confused... >>>>> >>>>> And this kind of confirms that allocations > PAGE_ALLOC_COSTLY_ORDER >>>>> causing hangs is most likely just a coincidence. Rather something either in >>>>> the block layer or in the storage driver has problems with handling bios >>>>> with sufficiently high order pages attached. This is going to be a bit >>>>> painful to debug I'm afraid. How long does it take for you trigger the >>>>> hang? I'm asking to get rough estimate how heavy tracing we can afford so >>>>> that we don't overwhelm the system... >>>> >>>> Sometimes it freezes just after logging in, but in worst case it takes >>>> me about 10min of more or less `tar xz` + `dd`. >>> >>> blk-mq debugfs is usually helpful for hang issue in block layer or >>> underlying drivers: >>> >>> (cd /sys/kernel/debug/block && find . -type f -exec grep -aH . {} \;) >>> >>> BTW, you can just collect logs of the exact disks if you know what >>> are behind dm-crypt, >>> which can be figured out by `lsblk`, and it has to be collected after >>> the hang is triggered. >> >> dm-crypt lives on the nvme disk, this is what I collected when it >> hanged: >> > ... >> nvme0n1/hctx4/cpu4/default_rq_list:000000000d41998f {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=65, .internal_tag=-1} >> nvme0n1/hctx4/cpu4/default_rq_list:00000000d0d04ed2 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=70, .internal_tag=-1} > > Two requests stays in sw queue, but not related with this issue. > >> nvme0n1/hctx4/type:default >> nvme0n1/hctx4/dispatch_busy:9 > > non-zero dispatch_busy means BLK_STS_RESOURCE is returned from > nvme_queue_rq() recently and mostly. > >> nvme0n1/hctx4/active:0 >> nvme0n1/hctx4/run:20290468 > > ... > >> nvme0n1/hctx4/tags:nr_tags=1023 >> nvme0n1/hctx4/tags:nr_reserved_tags=0 >> nvme0n1/hctx4/tags:active_queues=0 >> nvme0n1/hctx4/tags:bitmap_tags: >> nvme0n1/hctx4/tags:depth=1023 >> nvme0n1/hctx4/tags:busy=3 > > Just three requests in-flight, two are in sw queue, another is in hctx->dispatch. > > ... > >> nvme0n1/hctx4/dispatch:00000000b335fa89 {.op=WRITE, .cmd_flags=NOMERGE, .rq_flags=DONTPREP|IO_STAT, .state=idle, .tag=78, .internal_tag=-1} >> nvme0n1/hctx4/flags:alloc_policy=FIFO SHOULD_MERGE >> nvme0n1/hctx4/state:SCHED_RESTART > > The request staying in hctx->dispatch can't move on, and nvme_queue_rq() > returns -BLK_STS_RESOURCE constantly, and you can verify with > the following bpftrace when the hang is triggered: > > bpftrace -e 'kretfunc:nvme_queue_rq { @[retval, kstack]=count() }' > > It is very likely that memory allocation inside nvme_queue_rq() > can't be done successfully, then blk-mq just have to retry by calling > nvme_queue_rq() on the above request. > And that is something I've been wondering (for quite some time now): What _is_ the appropriate error handling for -ENOMEM? At this time, we assume it to be a retryable error and re-run the queue in the hope that things will sort itself out. But if they don't we're stuck. Can we somehow figure out if we make progress during submission, and (at least) issue a warning once we detect a stall? Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman