From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B759C4332F for ; Mon, 30 Oct 2023 11:28:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A65C36B019A; Mon, 30 Oct 2023 07:28:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A15526B019B; Mon, 30 Oct 2023 07:28:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DCD06B019C; Mon, 30 Oct 2023 07:28:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7D6576B019A for ; Mon, 30 Oct 2023 07:28:50 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4BAF8C014B for ; Mon, 30 Oct 2023 11:28:50 +0000 (UTC) X-FDA: 81401905620.16.140316A Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf24.hostedemail.com (Postfix) with ESMTP id E8112180031 for ; Mon, 30 Oct 2023 11:28:47 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=XW8axk8U; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=jq7LbyhL; dmarc=none; spf=pass (imf24.hostedemail.com: domain of jack@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=jack@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698665328; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7+MAdxAuVinBXZcCJL4/EKLK4pRjoLXrIPh+TEUOPZE=; b=zLsy6kZXco6Ygs/qdKUI9Wey+S1Qm5U6Yk0j12qWxSL/5c+ZQBH09vGKf/+7ZsKOTt/wy1 3Tk4eZZFGWguflixX+x+Zbyp6opHPtACXq10ajcAnRIYpzuGziDa3sdjAP8U2Y4JKgf9Es vFogdtX9v6PRpOgm4ckTd7flamfGfo8= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=XW8axk8U; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=jq7LbyhL; dmarc=none; spf=pass (imf24.hostedemail.com: domain of jack@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=jack@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698665328; a=rsa-sha256; cv=none; b=XOqfl+E/KxsBMQS5IYMfl6LHZO3LX3j2/mUOVci6aJq17J8MDADQ3qVSMuVX9nTEyt5GKV +CzR2MG9om5x16A8EooCiDg5rdnp4A68bvfqZ3wLR8cB+Rh6WQxvE9u5sH9RP0TvQQen7t kj11OT9Su54fjFW7/36B3NhbSEyphtg= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 2F4B31FEF0; Mon, 30 Oct 2023 11:28:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1698665325; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=7+MAdxAuVinBXZcCJL4/EKLK4pRjoLXrIPh+TEUOPZE=; b=XW8axk8UmIhsqHSxUQBcewcGOXoBAH7r5DaLZDdsYsO40AQIXr1e2csYN7ueAdM98cJqjY QvwuJOH0ShR5RlQMLqeczeRT08BQ+fmpU8anq/BVCGjnnuSOIM8c8gLwEdN02J6kPc+G/C YD9Y7DTL1iT18fOkDWUVq74tTvjcFYI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1698665325; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=7+MAdxAuVinBXZcCJL4/EKLK4pRjoLXrIPh+TEUOPZE=; b=jq7LbyhL8aFf8kmwRFv8/QL9PD6AwV0DeKhn++1TmLbMFdAU+e4G2mxN4gqZ80nQCsvgbD F/URD+2Qq9zbtWCA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 1C55E138F8; Mon, 30 Oct 2023 11:28:45 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 4ZvkBm2TP2VeTgAAMHmgww (envelope-from ); Mon, 30 Oct 2023 11:28:45 +0000 Received: by quack3.suse.cz (Postfix, from userid 1000) id 90A8BA05BC; Mon, 30 Oct 2023 12:28:44 +0100 (CET) Date: Mon, 30 Oct 2023 12:28:44 +0100 From: Jan Kara To: Vlastimil Babka Cc: Mikulas Patocka , Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org, Jan Kara Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 Message-ID: <20231030112844.g7b76cm2xxpovt6e@quack3> References: <89320668-67a2-2a41-e577-a2f561e3dfdd@suse.cz> <818a23f2-c242-1c51-232d-d479c3bcbb6@redhat.com> <18a38935-3031-1f35-bc36-40406e2e6fd2@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <18a38935-3031-1f35-bc36-40406e2e6fd2@suse.cz> X-Rspamd-Queue-Id: E8112180031 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: hozeketpu8g9xdcecjm4y9rxnr96og9j X-HE-Tag: 1698665327-840165 X-HE-Meta: U2FsdGVkX1/Nvlq6T+fkpeAz89p6TXFzIACrPDe/x/11q+nXxvMtc46aJhBTduSUzhIZ1UA/nDsdueE39D6WfvMJ0gN2LTBL3LOwZB2Jdd/U2hCol8PPFTposyvBqF/uSLF2hu7Cr7YPZFnrfe1P8Jpq+OJgTmzKrzI5e2urgRjsEaqo9b/fdXjdZlw2atByYuRKaziFQPci4FCyq4fkNNBYl20u3QtJw37t5EP1zg4JC9BsL7TWRs8aTLnsySFhSa+dWfVM1DFe2Gfkj7drnVb1ez1alfqUiqwnXeOKqjFsPBFb9+A2qsN2AscVGuY2Gahlr4/kE8LXSkdANy/rubQOeR7jy03s6HZQuyh2WXd7OLV5MJuk4dkqti6b6XRb1DDOFLNm56WwbjkcGEroskfvizyou0wuV7T/5Ydn7vKWbDu71BKx8r0QQ8cACnNlu8MNKfZ8EFP4FrvbyHraDWfqkcbZwCykHy0mp4Ahw00iBRU8dmb7ZSS3gukJpKvuGNhbRJdrHeZYWYMpum6YsFISQ6nY6nvJP9kPblqGgh1kZiAa5GJPCtLgOZALIrpUa26QbN5eQpgmCBSA/NSu0YqrCLUSfJPBrWUUDOQVRzston2CF3EB4pBaFVzvQmTrjJ9IReyJBsQ39KNScBn7NT+E73ctMe0ufVDRovx5fsYruBIjIBkTd8Zm65xr/7WdiNjT/3iF1hGKWmeBxjpl1aK1PBOd97paVkWyR7lzBYorvCFjuHQaljvVx0xZPt4O0g5bQtvVisAcEJSUTk+eN2J5ulAOjIYcxMBxFlLbn/hK60FPLJRYousIQQKGn75jFP5WL2CKK9YqorhFxde/PFVKYXleOSG8w8aHSuKYvxJ/CKmBvol824dT46NuzxUhqteH2r4vLCqqszYg8RpcYryVa/55DM/GuIARFG7gV/QI4YDTLuWJ3PAtAgNWy/9wC5d9pWQaE2vUPOFocTQ XBFaKSw2 pc5J1Jt5/Qz62kZKN554EtEHeXWdk+CLbrZYzEn2RrYsnZbkxjFHS4ii41DxNaiXAHfKXJuZ4DCxnN8nSYkc6yYhvF1EH/wh3I46u5NZ2164d/EFjZyp1Hd3oKpWq0U0fa9xLWlqB63xReN8epyF/3WPBCfQhaYIrt9QYDXLf4M/ZpdQuYKacmG8yrre8Mc/V3eQHsUPPXT9+bgCPXOFngULFFQj0poHnAu1O9HSvd85VPmZOSH+Q572EWcfhSgPxT8BFVb3bQbjBwhGU7jIooLnlvxMHFZ+0BCo16W++ROZ9mjc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon 30-10-23 09:37:10, Vlastimil Babka wrote: > On 10/30/23 08:37, Mikulas Patocka wrote: > > On Sun, 29 Oct 2023, Vlastimil Babka wrote: > > > >> Haven't found any. However I'd like to point out some things I noticed in > >> crypt_alloc_buffer(), although they are probably not related. > >> > >> > static struct bio *crypt_alloc_buffer(struct dm_crypt_io *io, unsigned int size) > >> > { > >> > struct crypt_config *cc = io->cc; > >> > struct bio *clone; > >> > unsigned int nr_iovecs = (size + PAGE_SIZE - 1) >> PAGE_SHIFT; > >> > gfp_t gfp_mask = GFP_NOWAIT | __GFP_HIGHMEM; > >> > unsigned int remaining_size; > >> > unsigned int order = MAX_ORDER - 1; > >> > > >> > retry: > >> > if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM)) > >> > mutex_lock(&cc->bio_alloc_lock); > >> > >> What if we end up in "goto retry" more than once? I don't see a matching > > > > It is impossible. Before we jump to the retry label, we set > > __GFP_DIRECT_RECLAIM. mempool_alloc can't ever fail if > > __GFP_DIRECT_RECLAIM is present (it will just wait until some other task > > frees some objects into the mempool). > > Ah, missed that. And the traces don't show that we would be waiting for > that. I'm starting to think the allocation itself is really not the issue > here. Also I don't think it deprives something else of large order pages, as > per the sysrq listing they still existed. > > What I rather suspect is what happens next to the allocated bio such that it > works well with order-0 or up to costly_order pages, but there's some > problem causing a deadlock if the bio contains larger pages than that? Hum, so in all the backtraces presented we see that we are waiting for page writeback to complete but I don't see anything that would be preventing the bios from completing. Page writeback can submit quite large bios so it kind of makes sense that it trips up some odd behavior. Looking at the code I can see one possible problem in crypt_alloc_buffer() but it doesn't explain why reducing initial page order would help. Anyway: Are we guaranteed mempool has enough pages for arbitrarily large bio that can enter crypt_alloc_buffer()? I can see crypt_page_alloc() does limit the number of pages in the mempool to dm_crypt_pages_per_client plus I assume the percpu counter bias in cc->n_allocated_pages can limit the really available number of pages even further. So if a single bio is large enough to trip percpu_counter_read_positive(&cc->n_allocated_pages) >= dm_crypt_pages_per_client condition in crypt_page_alloc(), we can loop forever? But maybe this cannot happen for some reason... If this is not it, I think we need to find out why the writeback bios are not completeting. Probably I'd start with checking what is kcryptd, presumably responsible for processing these bios, doing. Honza -- Jan Kara SUSE Labs, CR