From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB4A2C05027 for ; Sun, 12 Feb 2023 16:52:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 00C916B0073; Sun, 12 Feb 2023 11:52:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EFEDC6B0074; Sun, 12 Feb 2023 11:52:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D79636B0075; Sun, 12 Feb 2023 11:52:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C5D0E6B0073 for ; Sun, 12 Feb 2023 11:52:07 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8C08A405AD for ; Sun, 12 Feb 2023 16:52:07 +0000 (UTC) X-FDA: 80459232294.29.E84BCEA Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by imf05.hostedemail.com (Postfix) with ESMTP id C6FD7100010 for ; Sun, 12 Feb 2023 16:52:03 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=scylladb.com header.s=google header.b=O7peFQ7A; spf=pass (imf05.hostedemail.com: domain of avi@scylladb.com designates 209.85.221.43 as permitted sender) smtp.mailfrom=avi@scylladb.com; dmarc=pass (policy=reject) header.from=scylladb.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676220724; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BUNO3zH56v7+aWv1bzAzvIAzAdGa8qraFAdDc+iV4Cw=; b=cxz5GmOCZYobWXnt/qA1iF/zPSqpAzmyMCT83vQtCodPf9c1xtzBjDX3688d08H8FIoswm nb2Wv3wvT+7cK7s/2TPsDgaZxtHj4NpLsGz3etL4RItqyyPYk00pP7uz0KRQ4+Ty2o9eA1 sJ1utq8tI3HaGb704d0tbt6d0X371Ng= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=scylladb.com header.s=google header.b=O7peFQ7A; spf=pass (imf05.hostedemail.com: domain of avi@scylladb.com designates 209.85.221.43 as permitted sender) smtp.mailfrom=avi@scylladb.com; dmarc=pass (policy=reject) header.from=scylladb.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676220724; a=rsa-sha256; cv=none; b=HlF+1pPcUy/6qpV6pCsINCkznY+ZOTuwUbXReyqVPcMmy1JERJ8WdiPuqFtFlmGrjKobgZ dhYJi90akHIJ6up6by+xeJo6x4a1To3cuZzzSag/18Xg7vXTIRPWjNGHcbbsuCO43Ay5oX O5+pOG5FuRCBmImdvbErAiSuQ92HUIc= Received: by mail-wr1-f43.google.com with SMTP id m14so9953901wrg.13 for ; Sun, 12 Feb 2023 08:52:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=scylladb.com; s=google; h=mime-version:user-agent:references:in-reply-to:date:cc:to:from :subject:message-id:from:to:cc:subject:date:message-id:reply-to; bh=BUNO3zH56v7+aWv1bzAzvIAzAdGa8qraFAdDc+iV4Cw=; b=O7peFQ7ALZmPYOlGrbBta5Stu1lI82NyAekBLG8bMExgYoH23Tb1NqqHlT62oEUTg3 K+QN61JSUliDVt5yWipvfqOBAqfluXYrbmrHa1a9IICj6xmX+XTbTtDjhxQ+kqtRUErR NbjxrrAikkvQh3nGMSnVqGtnsj4e45QwbX4Q9gpofM0jThbbpm6fI/xphHx27wRWd3go 4tgqVJX/61XTvyTBvPOqiF1wdQBziCWKkuAB6aMwukQbI+nLBgv8ZCkJMvbSU3LL0ljF gHQy+VgUtTQwmRQBB2YT1S5SYx9Hws4/QD0bngCdrZme5jjgUCcc9W1iWoBWQF1GC1Jx ifBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:user-agent:references:in-reply-to:date:cc:to:from :subject:message-id:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=BUNO3zH56v7+aWv1bzAzvIAzAdGa8qraFAdDc+iV4Cw=; b=KoD06NOtrZnqkIYnXV89+nEZ1/Ffx1EP/zapjuJ2NYuH2HOgPEmA1Loh1ov7kM6gAJ pg89Eu5fu1vcBfQ+Ge+hdVwrllh//qGa4VHsjCdnWywPcY2QgttmP63nkorP/HMstQat JmWMqkAVdU/g5L0PQlEjzkEHUsF5GgVc1u0zmKO2daoA9rDuU6HPbE02RYalIEKz6PMw 26lJV1u/wT2IlWPpJQh1LWKHRe1NAuluprv9Oynh0uDflZBFCPwmo7xV9WODUvNdlFXa Fg89dWiwBE2PWnL09aaUXRXDbzIr1o7GzaPfPJ5vo7e/Oz1lOanfMJXmvF0i2tGZn4yz h/oA== X-Gm-Message-State: AO0yUKXArMh5vEau3CUmh3HDz5+GrFttkA0a4Tiqbcn3rVlN7sjCicW+ tICihiQgl+9fRuq/3CHXLh1D4hfcRAKfVrqawEOMgIrq+kSmvCe2QHqHdCLg2BFni4P9VH6kZs9 L/X6SdtKJ5q27sczEZb3CzGtrdBq8K/gj661fwZ93vGN9IB3+mWieF+H1OEqjSHySYZkXgAnxXe ftQe5St/6DqprImp/qS2Sp0zM= X-Google-Smtp-Source: AK7set8SQFrPfFoLiIeier4Nvn34CwL/N4gwSAgrurTUMXn7+OmmJVKGDOMZV+UtqDxVesj1NRkjuQ== X-Received: by 2002:adf:fd02:0:b0:2c5:4c7f:c91 with SMTP id e2-20020adffd02000000b002c54c7f0c91mr5895722wrr.66.1676220722226; Sun, 12 Feb 2023 08:52:02 -0800 (PST) Received: from avi.scylladb.com (system.cloudius-systems.com. [199.203.229.89]) by smtp.gmail.com with ESMTPSA id d14-20020adff2ce000000b002bc7fcf08ddsm8582999wrp.103.2023.02.12.08.52.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 12 Feb 2023 08:52:01 -0800 (PST) Message-ID: <5003b225c065596992d1016bf378bd561f8d55f6.camel@scylladb.com> Subject: Re: BUG: kernel NULL pointer dereference, address: 0000000000000042 From: Avi Kivity To: Matthew Wilcox , Dave Chinner Cc: linux-xfs@vger.kernel.org, linux-mm@kvack.org Date: Sun, 12 Feb 2023 18:52:15 +0200 In-Reply-To: References: <412ef57499e8ad13c815516f11cd00479a35587a.camel@scylladb.com> <20230209213002.GF360264@dread.disaster.area> Content-Type: multipart/alternative; boundary="=-nxNXkuki7hlDMJCsqHgD" User-Agent: Evolution 3.46.3 (3.46.3-1.fc37) MIME-Version: 1.0 X-CLOUD-SEC-AV-Sent: true X-CLOUD-SEC-AV-Info: scylladb,google_mail,monitor X-Gm-Spam: 0 X-Gm-Phishy: 0 X-Rspamd-Queue-Id: C6FD7100010 X-Stat-Signature: 4pc9y57o8c7cyt7ou4ig3kh6gy7ag8mq X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1676220723-166494 X-HE-Meta: U2FsdGVkX190ETGqHD2SEOCICVyRtS7lRYqpc+O0uWL3ngSWM8OWlbbVk/HXEB2BAUeEb0n2hhN37UB/GCeozN65/brhXpfGMx5MU9q+ecZU2hQDu6cMiVs1T650N0pnXDv3EfifrqGcdA6dcmxNLSw70Rj4G47IQLmlrJjQW/HePA23yRS66vF9TxEgNbKdIBW8qzQV8El9cCBmYLeKvaCnHYHH+Ip4a7XnsMvPKMZRfFFCZ8QEvIm2672yMap8Hk+OuMVwC8Km3aINgBIQVYEKOLwhUQCv1u8xkWlhEoTzOC8vXU2LO5B7yvFaVN5gUpf8qIccA73E9RUzo938dv7B6wNYB0Q5dxXdEdZiYp7Zo1OXTwK9g24MuTLTmtNiSUry3BF873kfEJosXOw0wqzUm/iBm9BHmVw9FvufTehd4KpmhrcoC7eaIBoY/7EjY2RLlVrMQdFD4eML57EDOIbx4bSTRiXtq4suSDTFyXnixmqRNapuqIFEYigJm2GgXkXEM7dCw9rFxbtujQMXI2ZASPk03gdpRbRLiLmMxEMcxsSsqlY9yxqwuib3xkDtPx892vwhynCX2QhbEJQuDqGUXhyiLvGsYFo3lYcha0KZByyzqXboGiHkmqeg3o9MaUuEIFS5tUEJkFZ80SRTeIRn3WwMoSa++krf2ewCpJyR4stlc0Oz9T46ovEMx7F7CWVRlEhWTiVME5Qelbw5Q5JRPFuxl58R/3snvBBihgPmo+8/5XAItxQsI3usj8hGdcFxQ0rB2Ev5x1hvtLczIYlXAMK6n+vh7E/xRVvbW9iw8LjF6xrig8MJuKsnRbMAum1XqXkvluDsnMmh4BmBN5rBEDsuoCpzFRS3HjkoTyFh17OTCp3JrkcOBoR6avhndcL4ARjFpLRJPamxgC/itd0ZmNXAYFL4eqlBsLX2CrY/yL6Y4oLy1P8S21FlIlqGH8jQBcMgR37dAKlTiqe Hhso4A9F eX7op6L5SWOKkqYjSpzZibj3Lb4snhFNwjbwYdubRpvgLadMVlUukuByDq5gLYdRCUPACcwdUWG/t2UkQtrdLug3Kijvov5725X0zXOm04xEQpojNU2YUjVeztjiRTS6B3Ng35Si8u/5QEoS3CLReq+Pm+xzoR2D0bbCQXpenpsT+Fj4gCi8oPX6hKy7TY2eQwHZajxXM84J5p6AYyJkbqh9/10LaTWoSvHKe4PvBEq6tWW01Ug3p0ccU533O6kJ3mfZuCyxsAqz0gj4KlqtGX034XKaYvpDzM8cioJhbkcxl5onrAiBKQQ1790AR475iXnUrS5IH6ShDYG+vplbHiC0p0x3jSZzWQ10rjUenRRSkeIiW0pz7+XVH7+d/BhirhD4ACljy/jQzAxcHrzMx/oJ3qoC9hIq2MzLetqIRnee0GA8kPf16U1OX99QPu0ei4F1aslGMqTdAlqfAq7WMA6pP4JCm9Pd0SOYjgLKe+7r61bvO/x7wCzTSusD72D1JOhJrCH6Enq7i+ml6GA/z5ERdpk6gkc7DRHvFDG/Q90eBWWCRmrLc8jw6Jg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --=-nxNXkuki7hlDMJCsqHgD Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, 2023-02-09 at 21:50 +0000, Matthew Wilcox wrote: > On Fri, Feb 10, 2023 at 08:30:02AM +1100, Dave Chinner wrote: > > [cc willy, linux-mm, as it crashed walking the page cache in the > > generic fault code] >=20 > I've seen this one occasionally, and I'm not sure what's going on. > I've never been able to reproduce it myself, and it seems to > disappear > for the people who have been able to reproduce it ;-( >=20 > It is 100% my fault and definitely caused by large folios.=C2=A0 In the > XArray, large folios are represented by a folio pointer in the lowest > index occupied by that folio and sibling entries in every other > index, > which redirect lookups to the canonical (ie lowest) entry.=C2=A0 This 0x4= 2 > that you've managed to find in the XArray is a sibling entry.=C2=A0 It > says that the entry we're actually looking for is at offset 0x10 of > the node we're in.=C2=A0=20 >=20 > Something similar was fixed in commit 63b1898fffcd, but that was a > sibling entry that ended up pointing to a node.=C2=A0 You've *presumably* > hit some kind of temporary situation where the original sibling entry > is no > longer pointing to the folio entry that it should be.=C2=A0 However, > there's > another possibility, which is that this is not a temporary RCU- > induced > state, but we have corruption in the tree.=C2=A0 If we do have corruption= , > then you'll see an infinite loop instead of a crash. >=20 > If it's a temporary situation, this will fix it. >=20 I'm unfortunately not in a position to test a fix.=C2=A0 > diff --git a/lib/xarray.c b/lib/xarray.c > index ea9ce1f0b386..4237a9647a6a 100644 > --- a/lib/xarray.c > +++ b/lib/xarray.c > @@ -207,7 +207,8 @@ static void *xas_descend(struct xa_state *xas, > struct xa_node *node) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (xa_is_sibling(entry))= { > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0offset =3D xa_to_sibling(entry); > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0entry =3D xa_entry(xas->xa, node, offset); > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0if (node->shift && xa_is_node(entry)) > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0if (xa_is_sibling(entry) || > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (node->shift && xa_is_node(entry))) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0ent= ry =3D XA_RETRY_ENTRY; > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0} > =C2=A0 > Please do let me know ... you say it's happened twice, but how many > machine-hours did it take to hit twice? That's hard to say. There are ~5 machines doing this work, the kernel was installed in early February, so around 1000 machine-hours, but what part of the time they were busy and how much of that they were running the triggering workload, I can't say. >=20 > > On Thu, Feb 09, 2023 at 10:43:10AM +0200, Avi Kivity wrote: > > > Workload: compilation and running unit tests. The task that > > > crashed is > > > a unit test. > > >=20 > > > Kernel: 6.1.8-200.fc37.x86_64 > > >=20 > > > Previously known stable on 5.8.9-200.fc32.x86_64. Two crashes > > > seen so > > > far. > > >=20 > > >=20 > > > Feb=C2=A0 7 17:19:33 localhost kernel: BUG: kernel NULL pointer > > > dereference, > > > address: 0000000000000042 > > > Feb=C2=A0 7 17:19:33 localhost kernel: #PF: supervisor read access in > > > kernel > > > mode > > > Feb=C2=A0 7 17:19:33 localhost kernel: #PF: error_code(0x0000) - not- > > > present > > > page > > > Feb=C2=A0 7 17:19:33 localhost kernel: PGD 80000001cbb1f067 P4D > > > 80000001cbb1f067 PUD 9cbb75067 PMD 0=20 > > > Feb=C2=A0 7 17:19:33 localhost kernel: Oops: 0000 [#1] PREEMPT SMP PT= I > > > Feb=C2=A0 7 17:19:33 localhost kernel: CPU: 24 PID: 3718328 Comm: > > > transport_test Tainted: G S=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 6.1.8-200.fc37.x86_6= 4 > > > #1 > > > Feb=C2=A0 7 17:19:33 localhost kernel: Hardware name: Dell Inc. > > > PowerEdge > > > R730/0599V5, BIOS 2.9.1 12/04/2018 > > > Feb=C2=A0 7 17:19:33 localhost kernel: RIP: > > > 0010:next_uptodate_page+0x46/0x200 > > > Feb=C2=A0 7 17:19:33 localhost kernel: Code: 0f 84 3f 01 00 00 48 81 > > > ff 06 > > > 04 00 00 0f 84 b3 00 00 00 48 81 ff 02 04 00 00 0f 84 37 01 00 00 > > > 40 f6 > > > c7 01 0f 85 9c 00 00 00 <48> 8b 07 a8 01 0f 85 91 00 00 00 8b 47 > > > 34 85 > > > c0 0f 84 86 00 00 00 > > > Feb=C2=A0 7 17:19:33 localhost kernel: RSP: 0000:ffffa83e4ed67cc8 > > > EFLAGS: > > > 00010246 > > > Feb=C2=A0 7 17:19:33 localhost kernel: RAX: 0000000000000042 RBX: > > > ffffa83e4ed67e00 RCX: 000000000000146e > > > Feb=C2=A0 7 17:19:33 localhost kernel: RDX: ffffa83e4ed67d20 RSI: > > > ffff94a9046316b0 RDI: 0000000000000042 > > > Feb=C2=A0 7 17:19:33 localhost kernel: RBP: ffffa83e4ed67d20 R08: > > > 000000000000146e R09: 0000000000dfd000 > > > Feb=C2=A0 7 17:19:33 localhost kernel: R10: 000000000000145f R11: > > > ffff94978b85960c R12: ffff94a9046316b0 > > > Feb=C2=A0 7 17:19:33 localhost kernel: R13: 000000000000146e R14: > > > ffff94a9046316b0 R15: ffff948f8bb1f000 > > > Feb=C2=A0 7 17:19:33 localhost kernel: FS:=C2=A0 00007fd68fcb9d40(000= 0) > > > GS:ffff949dffd00000(0000) knlGS:0000000000000000 > > > Feb=C2=A0 7 17:19:33 localhost kernel: CS:=C2=A0 0010 DS: 0000 ES: 00= 00 > > > CR0: > > > 0000000080050033 > > > Feb=C2=A0 7 17:19:33 localhost kernel: CR2: 0000000000000042 CR3: > > > 00000001dc1be005 CR4: 00000000001706e0 > > > Feb=C2=A0 7 17:19:33 localhost kernel: Call Trace: > > > Feb=C2=A0 7 17:19:33 localhost kernel: > > > Feb=C2=A0 7 17:19:33 localhost kernel: filemap_map_pages+0x9f/0x7b0 > > > Feb=C2=A0 7 17:19:33 localhost kernel: xfs_filemap_map_pages+0x41/0x6= 0 > > > [xfs] > > > Feb=C2=A0 7 17:19:33 localhost kernel: do_fault+0x1bf/0x430 > > > Feb=C2=A0 7 17:19:33 localhost kernel: __handle_mm_fault+0x63d/0xe40 > > > Feb=C2=A0 7 17:19:33 localhost kernel: ? do_sigaction+0x11a/0x240 > > > Feb=C2=A0 7 17:19:33 localhost kernel: handle_mm_fault+0xdb/0x2d0 > > > Feb=C2=A0 7 17:19:33 localhost kernel: do_user_addr_fault+0x1cd/0x690 > > > Feb=C2=A0 7 17:19:33 localhost kernel: exc_page_fault+0x70/0x170 > > > Feb=C2=A0 7 17:19:33 localhost kernel: asm_exc_page_fault+0x22/0x30 > > > Feb=C2=A0 7 17:19:33 localhost kernel: RIP: 0033:0x1666350 > > > Feb=C2=A0 7 17:19:33 localhost kernel: Code: Unable to access opcode > > > bytes > > > at 0x1666326. > > > Feb=C2=A0 7 17:19:33 localhost kernel: RSP: 002b:00007ffde7fa86d8 > > > EFLAGS: > > > 00010212 > > > Feb=C2=A0 7 17:19:33 localhost kernel: RAX: 0000000000000000 RBX: > > > 00007ffde7fa8748 RCX: 0000000002ed4468 > > > Feb=C2=A0 7 17:19:33 localhost kernel: RDX: 00006000000c4f50 RSI: > > > 00007ffde7fa8748 RDI: 0000000000000012 > > > Feb=C2=A0 7 17:19:33 localhost kernel: RBP: 0000000000000012 R08: > > > 0000000000000001 R09: 0000000002f46860 > > > Feb=C2=A0 7 17:19:33 localhost kernel: R10: 00007fd69219cac0 R11: > > > 00007fd69224e670 R12: 0000000000000000 > > > Feb=C2=A0 7 17:19:33 localhost kernel: R13: 00006000000c4f50 R14: > > > 0000000002ed4470 R15: 00007fd693be0000 > > > Feb=C2=A0 7 17:19:33 localhost kernel: > > > Feb=C2=A0 7 17:19:33 localhost kernel: Modules linked in: xsk_diag > > > veth tls > > > xt_conntrack xt_MASQUERADE nf_conntrack_netlink xt_addrtype > > > nft_compat > > > br_netfilter bridge stp llc intel_rapl_msr dell_wmi iTCO_wdt > > > dell_smbios intel_pmc_bxt iTCO_vendor_support dell_wmi_descriptor > > > ledtrig_audio sparse_keymap video dcdbas intel_rapl_common > > > sb_edac > > > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm > > > ipmi_ssif > > > irqbypass rapl intel_cstate intel_uncore ipmi_si ipmi_devintf > > > ipmi_msghandler nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib > > > nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct > > > nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 > > > rfkill > > > overlay ip_set nf_tables nfnetlink qrtr acpi_power_meter mxm_wmi > > > mei_me > > > mei lpc_ich auth_rpcgss ip6_tables ip_tables sunrpc zram xfs > > > crct10dif_pclmul crc32_pclmul nvme crc32c_intel polyval_clmulni > > > polyval_generic ixgbe ghash_clmulni_intel nvme_core sha512_ssse3 > > > megaraid_sas tg3 mgag200 mdio nvme_common dca wmi scsi_dh_rdac > > > scsi_dh_emc scsi_dh_alua > > > Feb=C2=A0 7 17:19:33 localhost kernel: dm_multipath fuse > > > Feb=C2=A0 7 17:19:33 localhost kernel: CR2: 0000000000000042 > > > Feb=C2=A0 7 17:19:33 localhost kernel: ---[ end trace 000000000000000= 0 > > > ]--- > > > Feb=C2=A0 7 17:19:33 localhost kernel: RIP: > > > 0010:next_uptodate_page+0x46/0x200 > > > Feb=C2=A0 7 17:19:33 localhost kernel: Code: 0f 84 3f 01 00 00 48 81 > > > ff 06 > > > 04 00 00 0f 84 b3 00 00 00 48 81 ff 02 04 00 00 0f 84 37 01 00 00 > > > 40 f6 > > > c7 01 0f 85 9c 00 00 00 <48> 8b 07 a8 01 0f 85 91 00 00 00 8b 47 > > > 34 85 > > > c0 0f 84 86 00 00 00 > > > Feb=C2=A0 7 17:19:33 localhost kernel: RSP: 0000:ffffa83e4ed67cc8 > > > EFLAGS: > > > 00010246 > > > Feb=C2=A0 7 17:19:33 localhost kernel: RAX: 0000000000000042 RBX: > > > ffffa83e4ed67e00 RCX: 000000000000146e > > > Feb=C2=A0 7 17:19:33 localhost kernel: RDX: ffffa83e4ed67d20 RSI: > > > ffff94a9046316b0 RDI: 0000000000000042 > > > Feb=C2=A0 7 17:19:33 localhost kernel: RBP: ffffa83e4ed67d20 R08: > > > 000000000000146e R09: 0000000000dfd000 > > > Feb=C2=A0 7 17:19:33 localhost kernel: R10: 000000000000145f R11: > > > ffff94978b85960c R12: ffff94a9046316b0 > > > Feb=C2=A0 7 17:19:33 localhost kernel: R13: 000000000000146e R14: > > > ffff94a9046316b0 R15: ffff948f8bb1f000 > > > Feb=C2=A0 7 17:19:33 localhost kernel: FS:=C2=A0 00007fd68fcb9d40(000= 0) > > > GS:ffff949dffd00000(0000) knlGS:0000000000000000 > > >=20 > >=20 > > --=20 > > Dave Chinner > > david@fromorbit.com > >=20 --=-nxNXkuki7hlDMJCsqHgD Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
On Thu, 2023-02-09 at 21:50 +0000, Matthew Wi= lcox wrote:
On Fri, Feb 10, 2023 at 08= :30:02AM +1100, Dave Chinner wrote:
[cc willy, linux-mm, as it crashed walking the page cache in the
generic fault code]

I've se= en this one occasionally, and I'm not sure what's going on.
I= 've never been able to reproduce it myself, and it seems to disappear
for the people who have been able to reproduce it ;-(

It is 100% my fault and definitely caused by large folios.=   In the
XArray, large folios are represented by a folio= pointer in the lowest
index occupied by that folio and sibli= ng entries in every other index,
which redirect lookups to th= e canonical (ie lowest) entry.  This 0x42
that you've ma= naged to find in the XArray is a sibling entry.  It
says= that the entry we're actually looking for is at offset 0x10 of
the node we're in. 

Something similar= was fixed in commit 63b1898fffcd, but that was a
sibling ent= ry that ended up pointing to a node.  You've *presumably*
hit some kind of temporary situation where the original sibling entry is = no
longer pointing to the folio entry that it should be. = ; However, there's
another possibility, which is that this is= not a temporary RCU-induced
state, but we have corruption in= the tree.  If we do have corruption,
then you'll see an= infinite loop instead of a crash.

If it's a t= emporary situation, this will fix it.

=

I'm unfortunately not in a position to test a fix. = ;

diff --git a/lib/xarr= ay.c b/lib/xarray.c
index ea9ce1f0b386..4237a9647a6a 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
<= div>@@ -207,7 +207,8 @@ static void *xas_descend(struct xa_state *xas, stru= ct xa_node *node)
       &= nbsp;if (xa_is_sibling(entry)) {
    &nbs= p;           offset = =3D xa_to_sibling(entry);
      = ;          entry =3D xa_e= ntry(xas->xa, node, offset);
-    &nbs= p;          if (node->= shift && xa_is_node(entry))
+    =            if (xa_is= _sibling(entry) ||
+       = ;            (node-&= gt;shift && xa_is_node(entry)))
   &nb= sp;            =         entry =3D XA_RETRY_ENTRY;
        }
 
Please do let me know ... you say it's happened twic= e, but how many
machine-hours did it take to hit twice?


That's hard to say. The= re are ~5 machines doing this work, the kernel was installed in early Febru= ary, so around 1000 machine-hours, but what part of the time they were busy= and how much of that they were running the triggering workload, I can't sa= y.


On Thu, Feb 09, 2023 at 10:43:10AM +0200, Avi Kivity = wrote:
Workload: compilation and r= unning unit tests. The task that crashed is
a unit test.
<= /div>

Kernel: 6.1.8-200.fc37.x86_64

Previously known stable on 5.8.9-200.fc32.x86_64. Two crashes seen= so
far.


Feb = ; 7 17:19:33 localhost kernel: BUG: kernel NULL pointer dereference,
address: 0000000000000042
Feb  7 17:19:33 localh= ost kernel: #PF: supervisor read access in kernel
mode
Feb  7 17:19:33 localhost kernel: #PF: error_code(0x0000) - no= t-present
page
Feb  7 17:19:33 localhost k= ernel: PGD 80000001cbb1f067 P4D
80000001cbb1f067 PUD 9cbb7506= 7 PMD 0
Feb  7 17:19:33 localhost kernel: Oops: 0000 [#= 1] PREEMPT SMP PTI
Feb  7 17:19:33 localhost kernel: CPU= : 24 PID: 3718328 Comm:
transport_test Tainted: G S &nbs= p;            &= nbsp;  6.1.8-200.fc37.x86_64 #1
Feb  7 17:19:33 loc= alhost kernel: Hardware name: Dell Inc. PowerEdge
R730/0599V5= , BIOS 2.9.1 12/04/2018
Feb  7 17:19:33 localhost kernel= : RIP:
0010:next_uptodate_page+0x46/0x200
Feb&n= bsp; 7 17:19:33 localhost kernel: Code: 0f 84 3f 01 00 00 48 81 ff 06
04 00 00 0f 84 b3 00 00 00 48 81 ff 02 04 00 00 0f 84 37 01 00 00 = 40 f6
c7 01 0f 85 9c 00 00 00 <48> 8b 07 a8 01 0f 85 91= 00 00 00 8b 47 34 85
c0 0f 84 86 00 00 00
Feb&= nbsp; 7 17:19:33 localhost kernel: RSP: 0000:ffffa83e4ed67cc8 EFLAGS:
00010246
Feb  7 17:19:33 localhost kernel: RAX:= 0000000000000042 RBX:
ffffa83e4ed67e00 RCX: 000000000000146e=
Feb  7 17:19:33 localhost kernel: RDX: ffffa83e4ed67d20= RSI:
ffff94a9046316b0 RDI: 0000000000000042
Fe= b  7 17:19:33 localhost kernel: RBP: ffffa83e4ed67d20 R08:
000000000000146e R09: 0000000000dfd000
Feb  7 17:19:3= 3 localhost kernel: R10: 000000000000145f R11:
ffff94978b8596= 0c R12: ffff94a9046316b0
Feb  7 17:19:33 localhost kerne= l: R13: 000000000000146e R14:
ffff94a9046316b0 R15: ffff948f8= bb1f000
Feb  7 17:19:33 localhost kernel: FS:  0000= 7fd68fcb9d40(0000)
GS:ffff949dffd00000(0000) knlGS:0000000000= 000000
Feb  7 17:19:33 localhost kernel: CS:  0010 = DS: 0000 ES: 0000 CR0:
0000000080050033
Feb&nbs= p; 7 17:19:33 localhost kernel: CR2: 0000000000000042 CR3:
00= 000001dc1be005 CR4: 00000000001706e0
Feb  7 17:19:33 loc= alhost kernel: Call Trace:
Feb  7 17:19:33 localhost ker= nel: <TASK>
Feb  7 17:19:33 localhost kernel: file= map_map_pages+0x9f/0x7b0
Feb  7 17:19:33 localhost kerne= l: xfs_filemap_map_pages+0x41/0x60 [xfs]
Feb  7 17:19:33= localhost kernel: do_fault+0x1bf/0x430
Feb  7 17:19:33 = localhost kernel: __handle_mm_fault+0x63d/0xe40
Feb  7 1= 7:19:33 localhost kernel: ? do_sigaction+0x11a/0x240
Feb = ; 7 17:19:33 localhost kernel: handle_mm_fault+0xdb/0x2d0
Feb=   7 17:19:33 localhost kernel: do_user_addr_fault+0x1cd/0x690
Feb  7 17:19:33 localhost kernel: exc_page_fault+0x70/0x170
<= /div>
Feb  7 17:19:33 localhost kernel: asm_exc_page_fault+0x22/0x= 30
Feb  7 17:19:33 localhost kernel: RIP: 0033:0x1666350=
Feb  7 17:19:33 localhost kernel: Code: Unable to acces= s opcode bytes
at 0x1666326.
Feb  7 17:19:= 33 localhost kernel: RSP: 002b:00007ffde7fa86d8 EFLAGS:
00010= 212
Feb  7 17:19:33 localhost kernel: RAX: 0000000000000= 000 RBX:
00007ffde7fa8748 RCX: 0000000002ed4468
Feb  7 17:19:33 localhost kernel: RDX: 00006000000c4f50 RSI:
00007ffde7fa8748 RDI: 0000000000000012
Feb  7 17:1= 9:33 localhost kernel: RBP: 0000000000000012 R08:
00000000000= 00001 R09: 0000000002f46860
Feb  7 17:19:33 localhost ke= rnel: R10: 00007fd69219cac0 R11:
00007fd69224e670 R12: 000000= 0000000000
Feb  7 17:19:33 localhost kernel: R13: 000060= 00000c4f50 R14:
0000000002ed4470 R15: 00007fd693be0000
Feb  7 17:19:33 localhost kernel: </TASK>
= Feb  7 17:19:33 localhost kernel: Modules linked in: xsk_diag veth tls=
xt_conntrack xt_MASQUERADE nf_conntrack_netlink xt_addrtype = nft_compat
br_netfilter bridge stp llc intel_rapl_msr dell_wm= i iTCO_wdt
dell_smbios intel_pmc_bxt iTCO_vendor_support dell= _wmi_descriptor
ledtrig_audio sparse_keymap video dcdbas inte= l_rapl_common sb_edac
x86_pkg_temp_thermal intel_powerclamp c= oretemp kvm_intel kvm ipmi_ssif
irqbypass rapl intel_cstate i= ntel_uncore ipmi_si ipmi_devintf
ipmi_msghandler nft_fib_inet= nft_fib_ipv4 nft_fib_ipv6 nft_fib
nft_reject_inet nf_reject_= ipv4 nf_reject_ipv6 nft_reject nft_ct
nft_chain_nat nf_nat nf= _conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill
overlay ip_se= t nf_tables nfnetlink qrtr acpi_power_meter mxm_wmi mei_me
me= i lpc_ich auth_rpcgss ip6_tables ip_tables sunrpc zram xfs
cr= ct10dif_pclmul crc32_pclmul nvme crc32c_intel polyval_clmulni
polyval_generic ixgbe ghash_clmulni_intel nvme_core sha512_ssse3
=
megaraid_sas tg3 mgag200 mdio nvme_common dca wmi scsi_dh_rdac
scsi_dh_emc scsi_dh_alua
Feb  7 17:19:33 localhos= t kernel: dm_multipath fuse
Feb  7 17:19:33 localhost ke= rnel: CR2: 0000000000000042
Feb  7 17:19:33 localhost ke= rnel: ---[ end trace 0000000000000000 ]---
Feb  7 17:19:= 33 localhost kernel: RIP:
0010:next_uptodate_page+0x46/0x200<= br>
Feb  7 17:19:33 localhost kernel: Code: 0f 84 3f 01 00 0= 0 48 81 ff 06
04 00 00 0f 84 b3 00 00 00 48 81 ff 02 04 00 00= 0f 84 37 01 00 00 40 f6
c7 01 0f 85 9c 00 00 00 <48> 8= b 07 a8 01 0f 85 91 00 00 00 8b 47 34 85
c0 0f 84 86 00 00 00=
Feb  7 17:19:33 localhost kernel: RSP: 0000:ffffa83e4ed= 67cc8 EFLAGS:
00010246
Feb  7 17:19:33 loc= alhost kernel: RAX: 0000000000000042 RBX:
ffffa83e4ed67e00 RC= X: 000000000000146e
Feb  7 17:19:33 localhost kernel: RD= X: ffffa83e4ed67d20 RSI:
ffff94a9046316b0 RDI: 00000000000000= 42
Feb  7 17:19:33 localhost kernel: RBP: ffffa83e4ed67d= 20 R08:
000000000000146e R09: 0000000000dfd000
= Feb  7 17:19:33 localhost kernel: R10: 000000000000145f R11:
=
ffff94978b85960c R12: ffff94a9046316b0
Feb  7 17:19= :33 localhost kernel: R13: 000000000000146e R14:
ffff94a90463= 16b0 R15: ffff948f8bb1f000
Feb  7 17:19:33 localhost ker= nel: FS:  00007fd68fcb9d40(0000)
GS:ffff949dffd00000(000= 0) knlGS:0000000000000000


--
Dave Chinner


--=-nxNXkuki7hlDMJCsqHgD--