From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1F3CC4332F for ; Fri, 3 Nov 2023 15:10:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 491298D00BE; Fri, 3 Nov 2023 11:10:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 441498D000C; Fri, 3 Nov 2023 11:10:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 309508D00BE; Fri, 3 Nov 2023 11:10:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 239338D000C for ; Fri, 3 Nov 2023 11:10:29 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DCB6A120F79 for ; Fri, 3 Nov 2023 15:10:28 +0000 (UTC) X-FDA: 81416979336.26.2A569FA Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf05.hostedemail.com (Postfix) with ESMTP id 9766510000C for ; Fri, 3 Nov 2023 15:10:26 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TJt0EXAN; spf=pass (imf05.hostedemail.com: domain of kbusch@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=kbusch@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699024227; a=rsa-sha256; cv=none; b=t6QirHBGTcP0IMRTvxvZy/9rooUYnvm95KJyCNEou7tgGUcKRqU2F+8HlMw4GsQRrbUUZF 3bxljGDEQyZV24XGkQkCyKHOMw6EHwmet3hLsCtzhFxiOyqIvscvCCDqYShZS7kJFu5qvC AsiO5TqmBRmLJAB/G5V8yW0HQbzOPoE= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TJt0EXAN; spf=pass (imf05.hostedemail.com: domain of kbusch@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=kbusch@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699024227; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2yrp500kpumJbNMCo79rQhVZ/F79oInbinHfFhbav9Y=; b=8JJWqcIkMfTeJbAhB51y5sdjXwyffmT5/Q6QxO6/DaHJF9sNqWTobjoBq6wbt1exSKVX/O vnHcfrFGJPVI8kaIgCFHfmhqxhvR2P1BpPwzsfvgpgxn08N8BH6CuAo3Oy++YXwpkJTn85 PDfpnQpPUWSkhfx+4W/b+q5x5B5gEik= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 2A22ECE2295; Fri, 3 Nov 2023 15:10:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5F993C433C8; Fri, 3 Nov 2023 15:10:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699024222; bh=I4xDSq1ezZOAkiMznffHlRhBFTdfQ6d+HVaQqg2tD5g=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=TJt0EXANZ1OjfdJU6AU6q3gklQYkIFtCEkQ6esqW9Ysjw3fvvXwbWlHFHaTYSFw1K uQZfo4O4dRUsOkzPj6GiBmoXmlCMfLnGqlVYCZfI5XXYfDh7XmV4vcysYh8/mn1svt zCzu55NYX212Yjf85v6fzHa318WO3Qlx16h5Fc7u2GK28yT+UNxYZ4e2QIlt14NSXi FwOkwSi/CUSnvfGPGgrirXyfi438VI6PdLIlucX9Tmx1M5X7F9GOebRAYHdLWUmggC 4EgDk9PoupSazA4bPMMR3OYd9W9c49KEINlQQr6nF56UiUjyWzgPyl/hfj5wKZtwHI zLNZ4I8iJCg8w== Date: Fri, 3 Nov 2023 09:10:18 -0600 From: Keith Busch To: Marek =?us-ascii?Q?Marczykowski-G'orecki?= Cc: Mikulas Patocka , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Jan Kara , Vlastimil Babka , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 Message-ID: References: <20231031140136.25bio5wajc5pmdtl@quack3> <8a35cdea-3a1a-e859-1f7c-55d1c864a48@redhat.com> <3cb4133c-b6db-9187-a678-11ed8c9456e@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 9766510000C X-Stat-Signature: noz95r73hzbxf3z4mc1xrqsmjnqaonpf X-Rspam-User: X-HE-Tag: 1699024226-818850 X-HE-Meta: U2FsdGVkX19G2sddrz2T6E0UMK0L/wH3nWSLKF0IPOTfzVhZKfrANjjzVgavx+bRt1m7AjwWpKE74mBnAA9SvrmMgmMgVxNBaqVgXVRKftf4X44hnuSpz57YZI//nR8qotvJTrln4B3eSuaPVNpyyWwiISGOOlc1DTucDyX0eF2lAR42uhVCmJNSyilU/9YUTMZgBmlm5iPbNYgLIWBoNaBptb0kyGnhDCZThjdZ6msyGE0x08U5Ibz+5UxwHljyB4KU64xxkNmqmOFOMy9NmsP1Bm4B8ONuwG3PwcKdbi/29VdhOaIbjKOUHuhx3Wpole8/mAxKcB6FbDeRLbgi6wzGYJkG0BbgM4HnOLo0/xT4VSHvlX2HmhbuUsd/gwKN2QknX9Jmbd8Mjeh3S1nmg3vSe1U1aeXHKn4RtiCMPP0Ps4pR6TqsEtF0Z1p+xXZuosmy8tKtscG5Rk/RrWWnOHDBo5uutkc2pdttAtP2enQ40SYquKCjZqkISkKowScdWZHk3aD4efVB6R3MdYTBclV+EkY6d03Al9XVtvvkNHhTa1dia47fZFCXUvmUkTZdYGltkgTSgWbC2zE7UyklYzCw4TMGg54M++vF21a48Wc1WMnrRpGJIJZkSx+jvasf4t+e6cophHDlA/UaDSTy2thVLjLw59LBCyf6cuuyfQNRhRa25x0g2P4sfquunejv+yK3UMImlT4AyPaQiYiKznqlVw/nqLjbQ4FcXXjBdXh0as30pHdN8EBQDrRBHPQLQAuJOWIbYz8OftYghrqm2SOQG1vNK/MOLmlwMPkYVI+7Nf5SM4cpi4S66TS7hdA+oCRFXuRfaxtonB6/krB1AfwFO2Hif98WGH7FLDPZO6E1bGtOJHtxiwtTpGYyKmxsCG6w+wSqoUXwNgIJG4s5e9eWImEC4//+K5rdJgRj14xyeLDQ734qHgWQjdyJ7lAE256gKKTAEVqyl19yEgb 28FUtXv3 +mqqBNs0Q9s6Zkz//33kJIWO3tx8n59Si/17DrmM7mqWCbgbOm5PmeE20pfVZW58VeNMTssMBYuhI5o83WbYhhFseCsppUs5zPRf1aMwLZEVyx/TSpKlTtdNF+vKqk2oXjOIhUntCYsVi0TLdOGg+cKJjYQfWK/Ei3jLqFx/+6g/I2InzTYkJuYOqva20Yt9GeEDX/TgkV6IvfT1g5D2riXqS/YstJ0Zbq43R X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Nov 03, 2023 at 04:01:19PM +0100, Marek Marczykowski-G'orecki wrote: > On Thu, Nov 02, 2023 at 06:06:33PM +0100, Mikulas Patocka wrote: > > On Thu, 2 Nov 2023, Marek Marczykowski-G'orecki wrote: > > > > > On Thu, Nov 02, 2023 at 10:28:57AM +0100, Mikulas Patocka wrote: > > > > > > > Try lowring /sys/block/nvme0n1/queue/max_sectors_kb to some small value > > > > (for example 64) and test if it helps. > > > > > > Yes, this helps too. > > > > On a plain upstream kernel with no other modifications (and with default > > max_sectors_kb), set the value /sys/module/nvme/parameters/sgl_threshold > > to "0" and test it if it deadlocks. Then, set this value to "1" and test > > it again. > > Got deadlock wit both values. > > > Revert sgl_threshold back to the default (32768). Boot the kernel with the > > option "iommu=panic". Reproduce the deadlock and if you get a kernel > > panic, send us the panic log. > > This is a Xen PV, so Linux is not in charge of IOMMU here. And there is > SWIOTLB involved (64MB of it), I'm not sure if for every DMA, but > definitely for some. So it's using xen_swiotlb_dma_ops, right? That doesn't implmeent .opt_mapping_size, and I'm guessing it should be equal to swiotlb_max_mapping_size(). > > Then, try this patch (without "iommu=panic"), reproduce the deadlock and > > tell us which one of the "printk" statements is triggered during the > > deadlock. > > I'll try this next. Placing my bet now: you'll see a DMA mapping error.