From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 317D1CDD1A9 for ; Mon, 30 Sep 2024 19:25:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C166E280023; Mon, 30 Sep 2024 15:25:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B9F88280017; Mon, 30 Sep 2024 15:25:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A18AD280023; Mon, 30 Sep 2024 15:25:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7EAAB280017 for ; Mon, 30 Sep 2024 15:25:52 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CB7121A0501 for ; Mon, 30 Sep 2024 19:25:51 +0000 (UTC) X-FDA: 82622384502.21.35B2238 Received: from mail.flyingcircus.io (mail.flyingcircus.io [212.122.41.197]) by imf24.hostedemail.com (Postfix) with ESMTP id 04329180006 for ; Mon, 30 Sep 2024 19:25:48 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=flyingcircus.io header.s=mail header.b=pl7GyzWA; spf=pass (imf24.hostedemail.com: domain of ct@flyingcircus.io designates 212.122.41.197 as permitted sender) smtp.mailfrom=ct@flyingcircus.io; dmarc=pass (policy=reject) header.from=flyingcircus.io ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727724285; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/oqYHO23CtlMs38L/l795oaos4aeK2PzyY47DMjMR0g=; b=ZTbqXhLuhpLnVYfk7qmJsOosHB2gKkHu1SRBqP2I8oXWngINYCMuy0veNa3s5boVUWcKV1 EL/nY7U9BUzuB1oazpIscqLH64mYaBgaQp6u9/07IlYMYBr1cG+boc76kvLELIDj/gQKjk FhYgRhnQWz/+mPvYoT05WjC5meFMJcU= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=flyingcircus.io header.s=mail header.b=pl7GyzWA; spf=pass (imf24.hostedemail.com: domain of ct@flyingcircus.io designates 212.122.41.197 as permitted sender) smtp.mailfrom=ct@flyingcircus.io; dmarc=pass (policy=reject) header.from=flyingcircus.io ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727724285; a=rsa-sha256; cv=none; b=TCmULGXatC+VKyBW9hM9HZOdH/MI2k16VGDMAJGWiXaFj5C9LDuTQdiPcoFiSYSAFMQK4N 0jfX6hK+thzbTcnXEyFUWtq1kM/bSKwAgjYr9sACS9zZxuQhfHVg3yNl4eoFwj9gCMvJB1 eCDygKbfm2xXPYJPlc1gTVa4FpIat5I= Content-Type: text/plain; charset=utf-8 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=flyingcircus.io; s=mail; t=1727724344; bh=/oqYHO23CtlMs38L/l795oaos4aeK2PzyY47DMjMR0g=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=pl7GyzWA3rCcYvPx25TKfcekTSb/0hWU288mSYTsN0c4t6kbB4raN4nCg69qC6Q+K FtYSNwSGnaBefcatTqxZdPQrmZoE9LdVprrzZ44yv9MipNZ7w5ULsiFRDzasOHmQn2 TpljjdeQIHBXsN8it8IH2fsPeiuUC3PRDpvpbPt8= Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3818.100.11.1.3\)) Subject: Re: Known and unfixed active data loss bug in MM + XFS with large folios since Dec 2021 (any kernel from 6.1 upwards) From: Christian Theune In-Reply-To: Date: Mon, 30 Sep 2024 21:25:22 +0200 Cc: Dave Chinner , Matthew Wilcox , Chris Mason , Jens Axboe , linux-mm@kvack.org, "linux-xfs@vger.kernel.org" , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Dao , regressions@lists.linux.dev, regressions@leemhuis.info Content-Transfer-Encoding: quoted-printable Message-Id: <295BE120-8BF4-41AE-A506-3D6B10965F2B@flyingcircus.io> References: <74cceb67-2e71-455f-a4d4-6c5185ef775b@meta.com> <52d45d22-e108-400e-a63f-f50ef1a0ae1a@meta.com> <5bee194c-9cd3-47e7-919b-9f352441f855@kernel.dk> <459beb1c-defd-4836-952c-589203b7005c@meta.com> <02121707-E630-4E7E-837B-8F53B4C28721@flyingcircus.io> To: Linus Torvalds X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 04329180006 X-Stat-Signature: qffxijy8die9aj34skfg19op17x4g4qg X-HE-Tag: 1727724348-828900 X-HE-Meta: U2FsdGVkX18VV2Nth1yWkq0rHJ1vreTHWKqGZ0gax8qd9UCJ9sQPhnJgm9JXk4ahqZ85atyzbxV1iF7R8t4ueUGCTtch2xSxIINll7W2MT0DrXBsPdHQb5KY2M05C4tpad9YNpXKZfhEdJOHPc8LSsMNn8QPec4CsLXyDj1kkGGojF/VHxP4KBk68jH0N72AQcjzHxL2IKraAvNfyAcKdiYoRS4JM4YewPnrapJVt76hKB2BUYW6dJW3NxciJg+vFwDRM58u1ckzLXzHUNB+3PpaKWwczc24EovVbHxABNkRRGvHpQtf1ilE1+PcijuVmhg8SPf1iv7h4WUAkFWVa3xYIkNCOiAk+F/V56LWpHSXz6D+SfTIvWgitpDlHyvp/RMbmNWyYyu2pUzQPni8DeOBxdJ4x8eSikXWzXP+GEs9MOVHYBqErObb2mtcCykzUMuQyy5X+/ywuwdrx0MpQ5PgoMDf0GlW1yAWn0mfnWXWL0ZQ2d8AbofrLgRXxnrf89Pecs3ePC+hStFpFh1hQtTLk07M8Ut2FEHhTYG+shi3Lyzenl7S8Ta6nLNZD/bshjxdHM9p6dgxkwtswB4xTitHQfrbpLsyNZKgOKvG50NSlmu6Ip6jqRLzEG4QIzj0IlEnln8RiW4GVmp1RY2FQtBeCVru7w/aE2NYlKuTL5wBCwL4Ibvc+30/0CQrLRW0A6pzHlSWpUZ4B7Z5kByqzGrB4DMT78dxGcSNFPXclhLNfWyM8ifsdMF172oLnQscml6CUkUf8GxX1++PrNHugtRKiMxIrUQ+NpKNQhCMq9kkvzW0Z09pDW4XNnGSLjdLwPWDTPVYXTZTLg2XnjVD2HYsd9XR4XZW9Mza64Brg9nwSxmefwE5GyKSZqsFlK4Z6XIjbVxcfKqRslApYfiMeYntw7bXDLPmHAG8mO/fnGqrM57oFjQ2ezsPUpM7l2HQ99XFdjJc/xmkSAwhYLp V3ZC+5zD 8YLmAVg+BK6f3Ja1+fu0iXuBQOTz16AxmGWSlMBidYBJbu91IT8gu5c2tFtEF9JqOQtbHNxEwaR3YiaZXdcRwDUnZC00nFWQ1yb8DvqCm2lH/XmRpl9OyqigYeYSjy8sy2rsnhcNz/0B3qGZwkMNUuJlKhbmIffp9y1LBRaa1S8/u3XTVVVLvJ9RDRFtiYvC0eykIIZstx3D8gSJx8+zxmmbdt5WANBYFC3ejsU6OqdJ7S9qDNArtqYGmxbHHaiCkcnRemWdVU04j628eJ27/5H0iKqfZkpTXTdGipV5qKgFtXPdr2kqkb2XHI7hrrTPnLravMM9oVWh2VmM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.014410, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: > On 30. Sep 2024, at 20:46, Linus Torvalds = wrote: >=20 > On Mon, 30 Sept 2024 at 10:35, Christian Theune = wrote: >>=20 >> Sep 27 00:51:20 13 kernel: = folio_wait_bit_common+0x13f/0x340 >> Sep 27 00:51:20 13 kernel: = folio_wait_writeback+0x2b/0x80 >=20 > Gaah. Every single case you point to is that folio_wait_writeback() = case. >=20 > And this might be an old old annoyance. I=E2=80=99m being told that I=E2=80=99m somewhat of a truffle pig for = dirty code =E2=80=A6 how long ago does =E2=80=9Cold old=E2=80=9D refer = to, btw? > [=E2=80=A6] > IOW, this code is known-broken and might have extreme unfairness > issues (although I had blissfully forgotten about it), because while > the actual writeback *bit* itself is set and cleared atomically, the > wakeup for the bit is asynchronous and can be delayed almost > arbitrarily, so you can get basically spurious wakeups that were from > a previous bit clear. I wonder whether the extreme unfairness gets exacerbated when in a = cgroup throttled context =E2=80=A6 It=E2=80=99s a limited number of = workloads we=20 have seen this with, some of which are parallelized and others aren=E2=80=99= t. (and I guess non-parallelized code shouldn=E2=80=99t suffer much from = this?) Maybe I can reproduce this more easily and ... > So the code here is questionable, and might cause some issues, but the > starvation of folio_wait_writeback() can't explain _all_ the cases you > see. =E2=80=A6 also get you more data and dig for maybe more cases more = systematically. Anything particular you=E2=80=99d like me to look for? Any specific = additional data points that would help? We=E2=80=99re going to keep with 6.11 in staging and avoid rolling it = out to the production machines for now. Christian --=20 Christian Theune =C2=B7 ct@flyingcircus.io =C2=B7 +49 345 219401 0 Flying Circus Internet Operations GmbH =C2=B7 https://flyingcircus.io Leipziger Str. 70/71 =C2=B7 06108 Halle (Saale) =C2=B7 Deutschland HR Stendal HRB 21169 =C2=B7 Gesch=C3=A4ftsf=C3=BChrer: Christian Theune, = Christian Zagrodnick