From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8222C4167D for ; Mon, 30 Oct 2023 15:56:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E68C6B01F9; Mon, 30 Oct 2023 11:56:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 496AC6B01FF; Mon, 30 Oct 2023 11:56:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 35F696B020A; Mon, 30 Oct 2023 11:56:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 27A156B01F9 for ; Mon, 30 Oct 2023 11:56:09 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F370780792 for ; Mon, 30 Oct 2023 15:56:08 +0000 (UTC) X-FDA: 81402579216.30.B041D0A Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf04.hostedemail.com (Postfix) with ESMTP id C72F240023 for ; Mon, 30 Oct 2023 15:56:05 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=kR38w7x3; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=YmGYY48H; dmarc=none; spf=pass (imf04.hostedemail.com: domain of jack@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=jack@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698681366; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E3XSA6WYBW8wrapbDS0LhBDnP8RGWsuPkduvqjedEqs=; b=BCmpt+51iB250zdaxFvdpOrA+i8UZnzZGDD9uKQx/rhwx0nfL9Y2qcnwcBaZYnrGQ/oOYK 1+V8gyv49mrF5vqC83cDT/F2lyQeT8ZXWhI/5cKlv9yKdkHnG8RUrPv96gBdtQMO0Om7B3 AIf0c161NeTgzORiKxoHT8nLGjvMz10= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=kR38w7x3; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=YmGYY48H; dmarc=none; spf=pass (imf04.hostedemail.com: domain of jack@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=jack@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698681366; a=rsa-sha256; cv=none; b=7Ls1TlukrAxG36oPms6qiUQwEst0TXl7JhQdWAwPknoKj9hpUa9E6/Edc2wDeBAAIvLxss 0/3cLKAFKw0RH1z8CDo1CzvuQKq2csirV1i9+leNZe2jD9inSmY9K7vVmER05FA2v2d+Y7 PWIpa6jbglsuqBM+Rc1ZqPToCkxkWfw= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id B1A051F7AB; Mon, 30 Oct 2023 15:56:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1698681363; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E3XSA6WYBW8wrapbDS0LhBDnP8RGWsuPkduvqjedEqs=; b=kR38w7x3BQ7gPL3vFPD9UWK/Ssfj/pyDTaDMbLv2qaWIQteL3Hwcf4Vbkrc9on90TpC1F4 lr8Cbs1p4oNm14I8vNHRq7HSAU0ulEbAVm42YXxdQLgCb8cqyxqwxdry+JgM5ymFXtVjeq xgoLjsKgHtKY/Y9oK+NdacDzYqsOeOA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1698681363; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E3XSA6WYBW8wrapbDS0LhBDnP8RGWsuPkduvqjedEqs=; b=YmGYY48HATodz9rPRuIhG053z/Hc+GzKDncp3XcDsaOpjoqlYV+AVRowhJZE1aLiHA9Zwa gMionwcRsqOM67AA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 9F9E8138EF; Mon, 30 Oct 2023 15:56:03 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id lVjgJhPSP2XqegAAMHmgww (envelope-from ); Mon, 30 Oct 2023 15:56:03 +0000 Received: by quack3.suse.cz (Postfix, from userid 1000) id 293D2A05BC; Mon, 30 Oct 2023 16:56:03 +0100 (CET) Date: Mon, 30 Oct 2023 16:56:03 +0100 From: Jan Kara To: Mikulas Patocka Cc: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= , Jan Kara , Vlastimil Babka , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 Message-ID: <20231030155603.k3kejytq2e4vnp7z@quack3> References: <89320668-67a2-2a41-e577-a2f561e3dfdd@suse.cz> <818a23f2-c242-1c51-232d-d479c3bcbb6@redhat.com> <18a38935-3031-1f35-bc36-40406e2e6fd2@suse.cz> <3514c87f-c87f-f91f-ca90-1616428f6317@redhat.com> <1a47fa28-3968-51df-5b0b-a19c675cc289@suse.cz> <20231030122513.6gds75hxd65gu747@quack3> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C72F240023 X-Stat-Signature: 8mxbj7aoen7u41j9gbe46ir7yf6tsoro X-Rspam-User: X-HE-Tag: 1698681365-881900 X-HE-Meta: U2FsdGVkX19kStBSAdAc9FqJBrcTRB/83pRo0fEb+6duFv2R++iI3k3AiVQvioV+UU8c5syciZHwkAwwXj0fRXKcNAs5s1v7SgbHdTmZMxGsOSlFvcPuKGPZP5CtdyVfR3PlaQzSlQ/8UOYNbMz7YXJWnSR12RLDgTCsNVJNrhU9ak4FNOQvwgio0LgMAUe01EMZ/7C1kdvHCRCfVlFoLQhSAUp5rauQ13tkRn9HIlAkYDL+13q2TuOW7wRIQwAhs3aHyspiHeCS1yAQ0PqRMQ6aLUYE0Nb5t/fBg+pe4qDtljs7cYP707zQDanG/GpQk6+24DkiLC9AvLAornhKmUCsWmxFXRzX2Hrf7PMB1G6Upd/LlteIVxYUGJeJVuoJjo+XAKEpagENRpb941jkuGjRepBdoT/0PlPFE3QTcz+Hnf/QqK62iWpF7a4wPeuNReUHFKPsXOrhI0JNHTd+RyPumgKQGDzNMeh2ORkjgAL6pPD8RP64qwxjypZweynSGnELWXr62grQ/HQeyjKwDcflccLHQ0kAVBa3MozUPBUmQbWeFizrmIXsoQ39wlIMehlm2PrgLmX/EMHYnr+Z73yuiJmQUG+WXSFYquLRIPZF5QWN36PiJGNuAbj8MSCA7czgPMSexYxL2QMSrxpVPJttAqXHkOrfVHkPJm5hsm5I1+92f5mYxTQg7wNJFOhE8M5h/+44k3CgfYquovygUSaHItmPp2DtaJbm9DOu7cJbNKFWsOx/WHdK3jUl/IxDSOQ1vGuiSgPYRT3cf9MotKuELOLqBnef7741/gg3JSPg1drOIYMMnLdcF1C1n7Igx6u4CUS+VEjhxcBM9260WB/VwZYXP3DmGDhGFIFimIhQIF+81qZjZGQb3aXWWVDAnXWGUCmP/kfX1EPvL8AOzB0/PuPtxEeVRk7ie4Tj02bXcduLvIedRJ6ecpYrGYZnbNASYeAe3EMfQfhCoxL JwcOPvZI jVfu0F4fwUDItFk63/cJPW6gf6ItwlsyMUCHhwglAji+XgDQG4HZyTBPsQdn6T4PLV2PKrCD+NDyVacEhKH5sY955Io5s17CLsp04r6AWbTM7O0E2+gRgy/Imd19AjgFRGUg1+ToRlmCcoYh8Vcm2G30pb4Yiso6bh2vhgIF9aM9QkAydvMGuIuQqySACnAucmz20KJimZPIRdHkLzDbduDH7xNonNQ0bG5n8pIFSPmEqQ0AlLe1MzEJpsaZt0V8E2Ra3g1YesxnIoSuyOqa0oFz8A4eTIGldbLm8bu32rDm6V2w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon 30-10-23 15:08:56, Mikulas Patocka wrote: > On Mon, 30 Oct 2023, Marek Marczykowski-Górecki wrote: > > > > Well, it would be possible that larger pages in a bio would trip e.g. bio > > > splitting due to maximum segment size the disk supports (which can be e.g. > > > 0xffff) and that upsets something somewhere. But this is pure > > > speculation. We definitely need more debug data to be able to tell more. > > > > I can collect more info, but I need some guidance how :) Some patch > > adding extra debug messages? > > Note I collect those via serial console (writing to disk doesn't work > > when it freezes), and that has some limits in the amount of data I can > > extract especially when printed quickly. For example sysrq-t is too much. > > Or maybe there is some trick to it, like increasing log_bug_len? > > If you can do more tests, I would suggest this: > > We already know that it works with order 3 and doesn't work with order 4. > > So, in the file include/linux/mmzone.h, change PAGE_ALLOC_COSTLY_ORDER > from 3 to 4 and in the file drivers/md/dm-crypt.c leave "unsigned int > order = PAGE_ALLOC_COSTLY_ORDER" there. > > Does it deadlock or not? > > So, that we can see whether the deadlock depends on > PAGE_ALLOC_COSTLY_ORDER or whether it is just a coincidence. Good idea. Also if the kernel hangs, please find kcryptd processes. In what state are they? If they are sleeping, please send what's in /proc//stack. Thanks! Honza -- Jan Kara SUSE Labs, CR