From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DD4EECE582 for ; Tue, 10 Sep 2024 17:28:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B99AA8D009A; Tue, 10 Sep 2024 13:28:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B485E8D0056; Tue, 10 Sep 2024 13:28:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A371E8D009A; Tue, 10 Sep 2024 13:28:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 86A888D0056 for ; Tue, 10 Sep 2024 13:28:42 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1216A1C44EC for ; Tue, 10 Sep 2024 17:28:42 +0000 (UTC) X-FDA: 82549513284.20.105050A Received: from sender4-pp-f112.zoho.com (sender4-pp-f112.zoho.com [136.143.188.112]) by imf04.hostedemail.com (Postfix) with ESMTP id EC2F84000E for ; Tue, 10 Sep 2024 17:28:39 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=collabora.com header.s=zohomail header.b=FcyR+0II; spf=pass (imf04.hostedemail.com: domain of bob.beckett@collabora.com designates 136.143.188.112 as permitted sender) smtp.mailfrom=bob.beckett@collabora.com; dmarc=pass (policy=none) header.from=collabora.com; arc=pass ("zohomail.com:s=zohoarc:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1725989268; a=rsa-sha256; cv=pass; b=iD3EEoDz0LMDDaxpHO9ZVH/ybbrci5ZheoYfZ+VUsRG/rba5Q8eznyL5gY8TokUtE0MciJ qItKl918cjsu36/fKmMp2BvZ3NbrSwRo9sr9KmPNDPkhlndO2m2O2v928GJlhM8wqqq5ut Lv5draVoMguqnCgnsSsWvl1VBx7P9lo= ARC-Authentication-Results: i=2; imf04.hostedemail.com; dkim=pass header.d=collabora.com header.s=zohomail header.b=FcyR+0II; spf=pass (imf04.hostedemail.com: domain of bob.beckett@collabora.com designates 136.143.188.112 as permitted sender) smtp.mailfrom=bob.beckett@collabora.com; dmarc=pass (policy=none) header.from=collabora.com; arc=pass ("zohomail.com:s=zohoarc:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725989268; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xIDalGuglFCsWPANE/RXImM1MbzLrLepclcprYWZhd0=; b=GGPW1eCHc7AG3KZferdM7QDZHIJDR5KKX/OjFG0+X108evd65mXU3YPs+iT4rhdijvGi0j 5yMDycAxrQDI4Mo3SFycgU7S0lI/ssE4z5g/Qh5Yq1Dg21YyCeqA6heO85JcYJw2Ezxmaq UftgKz4ktUU+aQLs4671o7N4wItMsSk= ARC-Seal: i=1; a=rsa-sha256; t=1725989307; cv=none; d=zohomail.com; s=zohoarc; b=OQFU74OFQ/T13NhMVadu6N4xXSEYqdsXZyTWpB5Uo9eJ15JdPxCT53UVfhYsCtZXCao9zgBE0B0TmM2y3oZBCgQG6z1WmA5MDqZRRtpIWBRiVfsj/u7A7j4hfypdvts1zk5lj88OrndjlrK0iRmoCiye2qxuF/PyLv5uwUUSM/g= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1725989307; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=xIDalGuglFCsWPANE/RXImM1MbzLrLepclcprYWZhd0=; b=iJNj0k2w2QNof0U0RGnBPdE/iYHyEuechjUerGj6/E+j727JU6aXjL/fWDMPkoJsdQyuWO8CrXrsSPOlY0WCiNhjJld+vRh+TBH7aPSd7eeLLPF6dRE8X6QZ4DST7FWdAQ1OVm1EzQeRpAlxjPgVHT5TukCfmIwG6kIWJx5tRFk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=collabora.com; spf=pass smtp.mailfrom=bob.beckett@collabora.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1725989307; s=zohomail; d=collabora.com; i=bob.beckett@collabora.com; h=Date:Date:From:From:To:To:Cc:Cc:Message-ID:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; bh=xIDalGuglFCsWPANE/RXImM1MbzLrLepclcprYWZhd0=; b=FcyR+0IIksMy0xlPrDUWUF44qbJjNSBS/GjPEbc704cAbzXXiwUqWC0lIqLT1D4P /dWYTupEBKR9Tg30LAfjKQFzacc3Oxrmw5ATJVL9p5JwK0NYptmoU3YvhY/BQYziy8o aBDar8naHC7ehvdJA1TtQMej3nN0wD75H2OWOfYA= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1725989275740601.1685810015596; Tue, 10 Sep 2024 10:27:55 -0700 (PDT) Date: Tue, 10 Sep 2024 18:27:55 +0100 From: Robert Beckett To: "Keith Busch" Cc: "linux-nvme" , "Jens Axboe" , "Christoph Hellwig" , "Sagi Grimberg" , "Andrew Morton" , "linux-mm" Message-ID: <191dcfa4846.bb18f3291189856.1624418308692137124@collabora.com> In-Reply-To: <191db450152.e0b28690987786.6989198174827147639@collabora.com> References: <191d810a4e3.fcc6066c765804.973611676137075390@collabora.com> <191db450152.e0b28690987786.6989198174827147639@collabora.com> Subject: Re: possible regression fs corruption on 64GB nvme MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Stat-Signature: acpreobhmccory6wyirwswhfkz3r3bme X-Rspamd-Queue-Id: EC2F84000E X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1725989319-947233 X-HE-Meta: U2FsdGVkX1+qVFtxc0rE/EpvVmefMtqw7xrsrg2pForNWP+1/VO/p4yC+V9hEaFPEg5mVYU7rQun9wqdsvNJeRvmCcN67iZJb+nBMNq9NKPMy/ZYWnQNBQmlk6jCl7o+PxvK794dDHiO4if0VHMmKgleRwZFcTPjdLzM9C18ghLSdUvRxqBWLKHVjXwIjyROZ0VnJTp5EeG4BQxzWh9sLPsqdvBxINrTWiun5bnyG/tuqfiZUYYmVMPrvlokfcqnDHQc3vQOuyMXHZtSYhSLSO62sRkcW5Kx+QQidBra3vjgCy81pHnVOtMBEECdmaWyu4NhSf5nmIrQjbcQImDQDLpxuEb915OLUHz/25p/+jCaMTxEieiNAoaeGGPIcVhQLLtf9Y7eyTpkYSgcwGQ7K0R6Iw8o7v7NE+9HexZyXt306Mv5zQ244f7wGtozP8ltemIJu4XWyTJP2cgCinZZIe1Z5jWjdfU4fR8jaITAox3VYrKLmhlec1Q5xbHI6TbkPFRKfwGtt05S5IFvzv735pehcBJr+ht6pQo/wHGtDHbEW7p1M619oM0Tacog0QgmAWsU+v8v+QNEaEtkJm54//lPMclXgJ/59Uei1oCM15ToNZGqvMNbT8hggFFZatE38xBXNQU5VI0OB2TLAK+Vp9lMOWYGSnqxk04P6lBIHXJKM2PMSxzUcwS0lz1A2ZXUxYd7jvKWvtjhG5vwbNzuknh+2/gA2Co9Y76eOcc6FgovvCXlASlUaEzpy1y6mgdmh+Kq0taUvfGZc+T1nm5CxwATkI0+XEkO2+78EzP3HXBbX/m8TJ2yoKlvJm4z9mZ/oZOs9pS3YQ5q97wIBEIYe3W+8iEDJgLAZjdXhQkbVmCPs4akp3rnYH0Di+9EkRLS1xlS2k3dZ1svHXg4yGEyFRgylJ/RVPXt07AdRg7mC1N1N9+v2kLFbhE/PTPiuKQ2Ylej/O1NUCs6iQd2YEr 60DqrVEr ljWeGDI/mRQ/KbnbPyF/kyP6m9b6tRiBoJrPZ2x/U+22FslnZ94qE5O2XBB8GnEY5F6c5+B9+fufd6cUaTJpVzduC220zNuB83jAiGFWEX4RMmZMTWag3Mh8NRg6HEbsq3U2B/ElQaw5MAY6ABxIMEucMqI8nteehukMbzROG4nSxsm+hSA7QJfh+Tnprl2gGKMD19BK+Lp3OHhxiv5PFvRjJaQ/td3gVIeSpQyFf7RPShvkN6G6mV4PWe9sBd4R8BfTKqd84CaxiFk82/lyfgTAjUUQtzKhTA8bfZjrB+J76tSN7GAbW/s7opQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: ---- On Tue, 10 Sep 2024 10:30:18 +0100 Robert Beckett wrote --- > > > > > > ---- On Mon, 09 Sep 2024 21:31:41 +0100 Keith Busch wrote --- > > On Mon, Sep 09, 2024 at 02:29:14PM -0600, Keith Busch wrote: > > > As a test, could you try kernel parameter "nvme.io_queue_depth_set=2"? > > > > Err, I mean "nvme.io_queue_depth=2". > > > > Thanks, I'll give it a try along with your other questions and report back. > > For clarity, the repro steps dropped a step. They should have included the make command: > > > $ dd if=/dev/urandom of=test_file bs=1M count=10240 > $ desync make test_file.caibx test_file > $ sudo bash -c "echo 3 > /proc/sys/vm/drop_caches" > $ desync verify-index test_file.caibx test_file > CONFIG_SLUB_DEBUG_ON showed no debug output. nvme.io_queue_depth=2 appears to fix it. Could you explain the implications of this? I assume it is limiting to 2 outstanding requests concurrently. Does it suggest an issue with the specific device's FW? I assume this would suggest that it is not actually anything wrong with the dmapool, it was just exposing the issue of the device/fw? Any advice for handling this and/or investigating further? My initial speculation was that maybe the disk fw is signalling completion of an access before it has actually finished making it's way to ram. I checked the code and saw that the dmapool appears to be used for storing the buffer page addresses, so I imagine that is not updated by the disk at all, which would rule out my assumption. I'd appreciate any insight you could give on the usage of the dmapools in the driver and whether you would expect them to be significant in this issue, or if they are just making a device/fw bug more observable. Thanks Bob p.s. Here is an transcript of the issue seen in testing. To my knowledge, if everything is working as it should, nothing should be able to produce this output, that dropping caches and re-priming the page cache via a linear read it fixes things. $ dd if=/dev/urandom of=test_file bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB, 10 GiB) copied, 111.609 s, 96.2 MB/s $ desync make test_file.caibx test_file Chunking [=======================================================================================================================================] 100.00% 18s $ sudo bash -c "echo 3 > /proc/sys/vm/drop_caches" $ desync verify-index test_file.caibx test_file [=============>-----------------------------------------------------------------------------------------------------------------------------------] 9.00% 4s Error: seed index for test_file doesn't match its data $ md5sum test_file ce4f1cca0b3dfd63ea2adfd745e4bfc1 test_file $ sudo bash -c "echo 3 > /proc/sys/vm/drop_caches" $ md5sum test_file 1edb3eaf5ae57b6187cc0be843ed2e5c test_file $ desync verify-index test_file.caibx test_file [=================================================================================================================================================] 100.00% 5s