From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C5AFC3ABB5 for ; Mon, 16 Sep 2024 00:00:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4806C6B0089; Sun, 15 Sep 2024 20:00:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 409366B008A; Sun, 15 Sep 2024 20:00:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 282FC6B008C; Sun, 15 Sep 2024 20:00:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 053DC6B0089 for ; Sun, 15 Sep 2024 20:00:27 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 71A56120117 for ; Mon, 16 Sep 2024 00:00:27 +0000 (UTC) X-FDA: 82568644494.22.0900021 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf03.hostedemail.com (Postfix) with ESMTP id 613F920013 for ; Mon, 16 Sep 2024 00:00:24 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=fromorbit-com.20230601.gappssmtp.com header.s=20230601 header.b=sQS68LbN; dmarc=pass (policy=quarantine) header.from=fromorbit.com; spf=pass (imf03.hostedemail.com: domain of david@fromorbit.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=david@fromorbit.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726444703; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yiTneBRUQz0c7oUx85pbwoHLabaV+URnFTxx3H0XSE8=; b=i0SccTDen995XIqj/Lbk70hy2UUX45HBZDEgmGnGzSjh8BYYRqRMO8MZC2b1cTsAboX5JI ya55QcbFialrxGDHsa5CZL9SQZdpYHjJyeictaU8uBOq27+3Zm9GDK/RuKrpQhV44SMLEc yoKxlRZew5rou4sfvollpxqM/qZUt1s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726444703; a=rsa-sha256; cv=none; b=vYMgYSO/rbeo5CeBavNwfNgCmTWw6neOxNYVmx3O5qyFbLdanMu/P8SNZicUg/GHZ1oZvc s7Lzm8QhYvdYgVSVhyot9wUwMxtTe8c1NvkANCMDDB7FHk8rKkFFtXFd+d+MvxFie4T8lX jm0yr9e2kjOZI5nvP2gmHNA2gqBmXE4= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=fromorbit-com.20230601.gappssmtp.com header.s=20230601 header.b=sQS68LbN; dmarc=pass (policy=quarantine) header.from=fromorbit.com; spf=pass (imf03.hostedemail.com: domain of david@fromorbit.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=david@fromorbit.com Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-206b9455460so29063925ad.0 for ; Sun, 15 Sep 2024 17:00:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20230601.gappssmtp.com; s=20230601; t=1726444823; x=1727049623; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=yiTneBRUQz0c7oUx85pbwoHLabaV+URnFTxx3H0XSE8=; b=sQS68LbNKMgkfUCqO/9N3BEeMgnkU+wpxEwpX2Uqtd2FKFNG+SFUzkhK0jtnDQzfFA izRgTKbTNH2RxpTW4QdkTwELy6U5n4Bpb0R/2FO6BmMtUIFHp+hyLQ5X9mMVt5iYsbQU G7Zy7X1wFbkWDKDR08eYxIJhBKg+kPZYEM49vRd9DpMnSYjKmXh5E7xvUZpsQ/JRes5h DCz5bly96k6bC5lmxm8zdweM1FSBkqUOI2ubR7HdqaCdcdZ1NM331TXDSxfU0LYoy99m jTtLl+NJ6ScSj0JDFvZ5/EKSZJ7Upzv9kNCH0Mx66rC+l4DRLa+7bqGXetxaicQjgmHN cWyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726444823; x=1727049623; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=yiTneBRUQz0c7oUx85pbwoHLabaV+URnFTxx3H0XSE8=; b=brSzMrE8MXJzGXdMDXzTB7RkM5YBPREHavFW5wV6mpqywGH61guCC3FutljhQ3UjDx ODLL5UX+Ra6ywVb4AnHk4IQ3qZhU6l+0QKp3cndTEfemLDGghl1NOQpk8byQi2F+2AkV aKPCU3w7wJIH0Paoffg+7uUxczb6Iy7ZAfSljoKiCbHG6CkLhjScMfnKJ5lzY2wzijMr LZd3HQKYYSleExMHNz34OwLzwIafKT5y4BmL6GI5flEszNsUD5GyjodwX7La5taSQqOO Ng08Pdixi4Wz/qGMv88qCD1KZy3/S7nnG6oRt0axQX48RHWM5OPqaXZyQPvdzBUi74GJ zK+A== X-Forwarded-Encrypted: i=1; AJvYcCUaQ4kHxuwN+ppehx7QTqyzb1n5I9vHUsdWXfjBwvlP4j0PNjiMEGsbr62+ItEQQVS344W+ZevgmQ==@kvack.org X-Gm-Message-State: AOJu0YyjbmNES4BfBrwQdSm5eZoLWYbpBY2AtrXtF+hJGT+HInwaeBUG /gYRZzveJa+9GXdrV5RwUcv6NGhFqhXCUTYqiWi3DVhsCJ6wIal2VgemOCuK1uE= X-Google-Smtp-Source: AGHT+IEL1eQKv3NuUXB18zqkV0UW1Ie5TJSdwTXsRLrQ2G5lCXlhwJhgyjhFRZZ66d6/+lADVVDKkA== X-Received: by 2002:a17:902:ce92:b0:207:6d2:1aa5 with SMTP id d9443c01a7336-2076e591737mr218814085ad.13.1726444822748; Sun, 15 Sep 2024 17:00:22 -0700 (PDT) Received: from dread.disaster.area (pa49-179-78-197.pa.nsw.optusnet.com.au. [49.179.78.197]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-207945dc76asm26908985ad.42.2024.09.15.17.00.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Sep 2024 17:00:22 -0700 (PDT) Received: from dave by dread.disaster.area with local (Exim 4.96) (envelope-from ) id 1spzAA-005hUa-2C; Mon, 16 Sep 2024 10:00:18 +1000 Date: Mon, 16 Sep 2024 10:00:18 +1000 From: Dave Chinner To: Linus Torvalds Cc: Jens Axboe , Matthew Wilcox , Christian Theune , linux-mm@kvack.org, "linux-xfs@vger.kernel.org" , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Dao , clm@meta.com, regressions@lists.linux.dev, regressions@leemhuis.info Subject: Re: Known and unfixed active data loss bug in MM + XFS with large folios since Dec 2021 (any kernel from 6.1 upwards) Message-ID: References: <0fc8c3e7-e5d2-40db-8661-8c7199f84e43@kernel.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 613F920013 X-Stat-Signature: d6ij15bfp5bj5m8yjbsx9i65ug387msx X-Rspam-User: X-HE-Tag: 1726444824-655392 X-HE-Meta: U2FsdGVkX1+b0dF4IBdcR9pP7pykA8kGcEwkIeGmaUhpWGfNZKAAXrv1djquJi91QLDUEXVYuKpLKpqAXOEPW0igKUbUit+ZRg/xB02tYrCRl0yMpWsbz70VvitQR5YNX+OhPb5agrpxZ9axQExrpcIYH/+NZ2jpVDtGdQuS1YgsBmIPQxm5ayh8h0bBrWQkI7yFqgXMfrfXYeQAgBPn7Xn2LvspzwZ+eXZgfxQNHhqdCpHv2lTgOeT8anAxzqXMG4CDSPjA5FrG2UYFNsNgf8g0QynPuhfnwXvNLwRO6z2eAdYRfZK5MtHb9MyAAS0XrEHlg6i5fKVMjI0zhpjNfZrfqK7ykjuoJNw6k0PYHEHOObKZXVkiYSrE7ZILNE5F43sYWmacESniMd8WQ58pyNZLtK6/4svlwJBlAbDlr1i4rxQmRmY1JJkbNt+vefn9Dh2s63rkrS2LV8uepJ55vvR3D/Cao2dPXV7Lac6DF8jASvg0KAhbV8CotQY3MZBjyEvOvas3jgcgbgGULbHbKwaq0YT+1BuAdnfdzSjJuW7tn70OOcWXeR6N/eFngCA2tFhEogMoTRP7LTU/w3xveLFAElHv2kepQeN5pQyg0YurF346MGUcHapkL3M/gKSexAmZVyqKSqJG0u1vC0DafSfkUT1PEs686sfiVqTaGTD2bc2vTCwu3rzbAlAUatDuetTAv3xfJpHQJ+//GzBnDAAWq60eY82vUr3k+yEP9WAZNIED9mWWspBmTuLHcKRZFCl1WtKZjO1wQHXpbkueSCwXJVuyOW0Kw9224auWon1cUIvDboLCc2EWw9Eyk7Peg05CRR+02Q4A3Ec09u2iAf79YGAyf6d+eKaneadXxQra824kjYn9EveFaeCcL2jjO1dqOM1YbPLdRij/8KQL1j/zN9zUFu9dZqaz96DuOusnkAXhNOpL/EUli42h12R2vmD/foFV0vXJ6o+icYm akI4aNSX vJsjOAfW5yghE2jL5QCAIxvP7duT1K0M5WqROD33yhKVBUCdOsVHTxqgBgoMARB/FjJ/7i+ldPJG1VjTFRod0FjCLO+VjVrofSnE1KW591E93odAc8gP9+BQoJnvGzqsEPaLWlkFahCieJ+II+w6twqH8bRcddI5u6zVgSL8xm35W4s40FQF8i7dZ3GfymNdDQ82s+2/1t2ZHPeTEaYKiQxO7fthV+W9YY3JnBmv5BQ7SRNoF0EibJdsLjx6I5ba82xkmpd7HLhUQ8sGlUAA4GjPezHQy/GSysJWseEzqYcbnZ4HYQaVSZuS24KqlkfyP56zw0WeTTHTcWMrLbPPlwwGR7P6ANsd0jQCaVxEDuteosz/E+onwCAV+QbwMGyfTaw2LiS0l19XL1zX1lE2JS6qkF16CVfq9KO90Vpd+ys5No2b4Ac0rBcmQHw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Sep 12, 2024 at 03:25:50PM -0700, Linus Torvalds wrote: > On Thu, 12 Sept 2024 at 15:12, Jens Axboe wrote: > Honestly, the fact that it hasn't been reverted after apparently > people knowing about it for months is a bit shocking to me. Filesystem > people tend to take unknown corruption issues as a big deal. What > makes this so special? Is it because the XFS people don't consider it > an XFS issue, so... I don't think this is a data corruption/loss problem - it certainly hasn't ever appeared that way to me. The "data loss" appeared to be in incomplete postgres dump files after the system was rebooted and this is exactly what would happen when you randomly crash the system. i.e. dirty data in memory is lost, and application data being written at the time is in an inconsistent state after the system recovers. IOWs, there was no clear evidence of actual data corruption occuring, and data loss is definitely expected when the page cache iteration hangs and the system is forcibly rebooted without being able to sync or unmount the filesystems... All the hangs seem to be caused by folio lookup getting stuck on a rogue xarray entry in truncate or readahead. If we find an invalid entry or a folio from a different mapping or with a unexpected index, we skip it and try again. Hence this does not appear to be a data corruption vector, either - it results in a livelock from endless retry because of the bad entry in the xarray. This endless retry livelock appears to be what is being reported. IOWs, there is no evidence of real runtime data corruption or loss from this pagecache livelock bug. We also haven't heard of any random file data corruption events since we've enabled large folios on XFS. Hence there really is no evidence to indicate that there is a large folio xarray lookup bug that results in data corruption in the existing code, and therefore there is no obvious reason for turning off the functionality we are already building significant new functionality on top of. It's been 10 months since I asked Christain to help isolate a reproducer so we can track this down. Nothing came from that, so we're still at exactly where we were at back in november 2023 - waiting for information on a way to reproduce this issue more reliably. -Dave. -- Dave Chinner david@fromorbit.com