From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09C25C4332F for ; Thu, 2 Nov 2023 12:22:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 55E086B0146; Thu, 2 Nov 2023 08:22:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E3CD6B0148; Thu, 2 Nov 2023 08:22:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3356A6B0149; Thu, 2 Nov 2023 08:22:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1A6526B0146 for ; Thu, 2 Nov 2023 08:22:01 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DBB9A1CB380 for ; Thu, 2 Nov 2023 12:22:00 +0000 (UTC) X-FDA: 81412926000.22.3D18FD2 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf08.hostedemail.com (Postfix) with ESMTP id B2FAA160030 for ; Thu, 2 Nov 2023 12:21:57 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="d/bJkBWz"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=7+I9n1AP; spf=pass (imf08.hostedemail.com: domain of jack@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=jack@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698927718; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5mAwtuufov9lTyy9sGroezcbI4J+0d92k5eAe06+d7k=; b=hBMvrqZjwfmxHr5PCzyjk5hc68HIC2iJq/qJmLwAmkJ9caZZdL2yBkfykWML/neZ0a977u UrsX3ZmkUdlk9hXJjpV2AGZCF3LyUSmeclBS0Q+6kig5ZrD35Ufb4HLBCb9errSwt7Q5Ad t/FNMX4pb5OoeXHrD4Hg5ytSsgFDQg8= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="d/bJkBWz"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=7+I9n1AP; spf=pass (imf08.hostedemail.com: domain of jack@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=jack@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698927718; a=rsa-sha256; cv=none; b=DE6/MGJDydL6ZDUz3+gNQAy0nla6YwWe3S+Sq0FzI8+XzMI/84EgaL91w61x2pdVNgzyBU K9DTksfmTmfC+8rujS0YECNiWvPIDaeS0FK6qsdoo71qiQRUSHwG7uIdz13OtGQ6RTk9M2 gNs5FSJ7NVKCvL7SHpBvugQCPTiDEBE= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 7C5A121A04; Thu, 2 Nov 2023 12:21:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1698927715; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5mAwtuufov9lTyy9sGroezcbI4J+0d92k5eAe06+d7k=; b=d/bJkBWzLkto/7fGyqVLPb1234f6tUyvUNiGGAK7Qoa8qcSX2xy+GlFZUIjtizxRoIhTQw a+VGgqPncn2LHxZGvcVnzn5XRAp9g4+PoYAslRKjrVPjc0vY9sdDyCr+iTruuzwCIYi1Kw 3VbDzjjOhoWdWqh6HVj4Noo1SIELCOw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1698927715; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5mAwtuufov9lTyy9sGroezcbI4J+0d92k5eAe06+d7k=; b=7+I9n1APdWA4llCvO7g/6lEagQD4M+yeaMcPW4WCfXzozzeg8KR3dU9aGgwa4GrCFmWM6c u3lBNfzxJp1TuKDg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 6DCAF138EC; Thu, 2 Nov 2023 12:21:55 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id vJjEGmOUQ2VQRQAAMHmgww (envelope-from ); Thu, 02 Nov 2023 12:21:55 +0000 Received: by quack3.suse.cz (Postfix, from userid 1000) id E5402A06E3; Thu, 2 Nov 2023 13:21:54 +0100 (CET) Date: Thu, 2 Nov 2023 13:21:54 +0100 From: Jan Kara To: Mikulas Patocka Cc: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Jan Kara , Vlastimil Babka , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 Message-ID: <20231102122154.jtwcl6l4f4pebqqx@quack3> References: <20231030155603.k3kejytq2e4vnp7z@quack3> <98aefaa9-1ac-a0e4-fb9a-89ded456750@redhat.com> <20231031140136.25bio5wajc5pmdtl@quack3> <8a35cdea-3a1a-e859-1f7c-55d1c864a48@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: B2FAA160030 X-Rspam-User: X-Stat-Signature: god17j1cpdymuo35c7hj44dzcusp9zn9 X-Rspamd-Server: rspam01 X-HE-Tag: 1698927717-805165 X-HE-Meta: U2FsdGVkX1/2jRLAAc9A24Q0Cwq5UxZ97YsmoKzeQes41F+5v0f8XOdHc8d9fjSbM98QgjZgDJgy8ug/Ix1kbmo40xMwgKFkfcr+SFaIZ7J0Gp29b1E46CMATuTtGwRBHppcv0UUjUfm/t6BVio6G1E9KKmyq6Y1U8achS5SxvMlE43lfhmJ6wHD2mOPYQypHc6B2F9N3UA0VjvegCvTQb79TE1UU7ZVhA3Z6WSwTAqfIT9+XKAC/Od8LPKHe0JWOjkkDJ3p94QR/zzEUjFg5rPU+yq95Tp/yG5iLS7qApsgrd/rGoCq5gnO1VDIRsUypU+2+aSe4EVUtIcPhfKrROBmdKtuyAAjH9ry3V81C8wjwHxDBi/iAEgXvquGqrhRC+7TxCi3uRf5CN7PXJqbiOULPg/SxOhjohZrvJ+R9pNMXqWI9yo/+Is5sNCNsq4NUWMXUaX8XCNg3I/s5F9FVNcWKmEG6/k1CPHO7utiYSzaRoiHf4HlGoDsv3E4Rn5zYaknHoE0AZkv8kfDoWABzIeqBH2ZZQd4OfsmlFAocdqawCZNTMjiUZqMdS5n68XXNY3W/O4xVmmq1zu7SgYfRrLZvUDcxA9L43C+/bwWodwasWExwZlsc93Dkfx7WGcHz+d6Wx9y/YA4RUNiQv9nda0RzvKi1NLyBWer3ZJa01MAMu6Rx5apFWYT9AMxwmWOmpu9n+yB6fEQ2i/7ASacc1YJW0ETNrzjUplC4mRKpdVaSsgvxKYW4pTa/Wktf5OHJU1k8qL7S8j1oTYNecsuHUDrr8WwcCuPJF6HSJs6ZeL3RGbegOOuXE8jPUzuXyX5xoLanZ3pbDDLYCAt2XKAkY78Ej/SNvkUTS9vlEw1Hlr9TfY7vbqCKw5r20bIvaPiz7Mq5fCXguGRrbjn/0OsCet6kFusi3+LNPjvDXI2EGpCcZF86HyQQPw/Ouj45Xa17yFmQGBQGBoeauwfO/Z qr6LV98U V2+2mPoIRcjZYiigWjGwD0edFyzTdBHILD5pFvfShbGwWVyiSVfwwQ2oUA9JktRLBklULj1+HgF6PVbZNxgugxvIK9dwAWPHgw3IFLbwrtvsjj9o8Do6suvy+A+oiix0VF1eFqwQixuqu4HjxamqvwbUCbmBZOi4Ay6STgiFguOSwdCpFk+AnLmZHi2pJfEtTfz28g0eYULq46dM+Xs83Nt/27zt1zoDf1zm7YUNTF7X7jh89FW41zQtE1nMeOIJNpSmnqtHTodLlDUaOlEZoBgeAFROGzOqgTFR4Wbx8eaRmuqTE6DeCmu/UGa3VBivabGNKi6mK6kBCX0x1PTl7tJnd/c9IYDUPcmaJl+z+5cfE/q4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu 02-11-23 10:28:57, Mikulas Patocka wrote: > On Thu, 2 Nov 2023, Marek Marczykowski-Górecki wrote: > > On Tue, Oct 31, 2023 at 06:24:19PM +0100, Mikulas Patocka wrote: > > > > > > Hi > > > > > > > > I would like to ask you to try this patch. Revert the changes to "order" > > > > and "PAGE_ALLOC_COSTLY_ORDER" back to normal and apply this patch on a > > > > clean upstream kernel. > > > > > > > > Does it deadlock? > > > > > > > > There is a bug in dm-crypt that it doesn't account large pages in > > > > cc->n_allocated_pages, this patch fixes the bug. > > > > This patch did not help. > > > > > If the previous patch didn't fix it, try this patch (on a clean upstream > > > kernel). > > > > > > This patch allocates large pages, but it breaks them up into single-page > > > entries when adding them to the bio. > > > > But this does help. > > Thanks. So we can stop blaming the memory allocator and start blaming the > NVMe subsystem. ;-) > I added NVMe maintainers to this thread - the summary of the problem is: > In dm-crypt, we allocate a large compound page and add this compound page > to the bio as a single big vector entry. Marek reports that on his system > it causes deadlocks, the deadlocks look like a lost bio that was never > completed. When I chop the large compound page to individual pages in > dm-crypt and add bio vector for each of them, Marek reports that there are > no longer any deadlocks. So, we have a problem (either hardware or > software) that the NVMe subsystem doesn't like bio vectors with large > bv_len. This is the original bug report: > https://lore.kernel.org/stable/ZTNH0qtmint%2FzLJZ@mail-itl/ Actually, Ming Lei has already identified [1] that we are apparently looping in an endless retry loop in nvme_queue_rq(), always ending up the attempt with BLK_STS_RESOURCE. Honza [1] https://lore.kernel.org/all/ZUHE52SznRaZQxnG@fedora -- Jan Kara SUSE Labs, CR