From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0C80CA0EC0 for ; Thu, 29 Aug 2024 21:55:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2ED166B007B; Thu, 29 Aug 2024 17:55:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 29CB56B0083; Thu, 29 Aug 2024 17:55:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 165A06B0085; Thu, 29 Aug 2024 17:55:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E91E56B007B for ; Thu, 29 Aug 2024 17:55:05 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9A798412B2 for ; Thu, 29 Aug 2024 21:55:05 +0000 (UTC) X-FDA: 82506638970.13.F2EEAB8 Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) by imf12.hostedemail.com (Postfix) with ESMTP id C8B6140005 for ; Thu, 29 Aug 2024 21:55:03 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qNbYVsyx; spf=pass (imf12.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.52 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724968482; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X2J1Zruavdx6XZtyjF3WfuYLy3+4x0oPlle13EcoFtA=; b=NPVBb8sWwrLq/pHkgEUSCGKVa6uJx2Uloz8lFXI75dn1qRfd1CnzqxpG7QaMK4aQdsa3r/ smefRMgZ2Zm/s8wl8ArwW0ewxTG8/E4mpwENeQEk2oCZl75yK94PlZ9KmGe76km5uMy2dP yiWD1REdyvfk1rFTDl82gZj+4Ypc9a8= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qNbYVsyx; spf=pass (imf12.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.52 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724968482; a=rsa-sha256; cv=none; b=ntNAqZZzI5VJx/hhjxUWIALOyQMSB2TX5U7/JsaGeJHpHvt6JPI8RQ2KvkeLGd+2Ukujb/ fsPyG1ABBEn8P8taltiDcbVScV57nBcgK56l07o6nXO4V1pcgyuvRhw1/o3C9F++tQGe2n +lhNyK8f9VKiqIhsGV+U7ECfhrvKb8w= Received: by mail-ed1-f52.google.com with SMTP id 4fb4d7f45d1cf-5bebd3b7c22so3628099a12.0 for ; Thu, 29 Aug 2024 14:55:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1724968502; x=1725573302; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=X2J1Zruavdx6XZtyjF3WfuYLy3+4x0oPlle13EcoFtA=; b=qNbYVsyxFJDbMJYl/kTDNqeOI8ssMPUuCdy+OCocR/+cteURurGGePQc8GboOCGzXa ls5LmEeEdc6iPvIPa+y+tk+UcqxiKNrjh9rUiTFDYO8Zo0QVcWBludavf9DEjqzVOfkd 2fCUmc4JZvvxGhHDE6xcpdHpQrJV+FN4w2T2XSU08x+wbo8rgynWZlyvSBNcde4AUzQh 8LcI/OFQKwAbcdTao6QC6/76vvSGHMeeLk54EULZ4rp3G8PL+tjcPdpDajuWRPOYTrbp 4edYRnH/ayP/fIaKH8V8P+A7duhUMIF8FdwNvzwG4rYjU86tSau1iQtDczCHq0ifz/UP K8dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724968502; x=1725573302; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X2J1Zruavdx6XZtyjF3WfuYLy3+4x0oPlle13EcoFtA=; b=XGl203aQINt8yJ/MhCTo78bsfT1Bz8Z0nDmGiExT20k04nfq+T/0AlU6iHHtDSdL9t yK7z9PbN36+f6mSmQrGuCaCtYl2snBqwntTTTO1bJPEYgYdTv3J3cf+BWiJPzAep8G9q 1apW5kRuL8M69AM2PnzEro+QG5lgDZ85spDKCnyaTZuKgdGhk3LqELU1A/FPE2JykW4t 9gLhvxlmpxc6tJvWEZMsJR9bjJTEGNaW2upxJw3bTRSQGu0iqHkbou8D7yfiHoemf/AG g9sNlKK/ovOUV/N45+hewfmS7riPe6JFMCAI/+28nweGNXByzWPWqEncCXnBH/pLpWhP y++g== X-Forwarded-Encrypted: i=1; AJvYcCXMCEchT3gDu/7QDZDwt4vF6skbTSN/mo9Hu3KcMl2khAaVUT3in8DTtR94BMOK8S94MfsPMNUI5g==@kvack.org X-Gm-Message-State: AOJu0YwgCNcJ5IHmIJ7jyKuF5UTVbhI80oOrlGkHgW5dqP3V8lNAf8Fb pudpOTmW1tKzq8OdRFKRuqVh+Lzh4VyGw5yhR9BWFuPUkGk3V8vXqJ3OK7NLvas7lSDQR3aMbe2 3HsU7YKCX31mkA/Xr+oRxjcKHYE6tOdHpClcD X-Google-Smtp-Source: AGHT+IH4ZiYY7Gh1I4yhkeVflwkseKNXmSds8HXCeIwpuTOHYtBHBjH+rd6088PWHQ/1mRdQhrMjnIDBUnNE0bVXLxU= X-Received: by 2002:a17:906:f5aa:b0:a86:96da:afb with SMTP id a640c23a62f3a-a89a2584293mr20082666b.10.1724968501505; Thu, 29 Aug 2024 14:55:01 -0700 (PDT) MIME-Version: 1.0 References: <6f65e3a6-5f1a-4fda-b406-17598f4a72d5@leemhuis.info> <02D2DA66-4A91-4033-8B98-ED25FC2E0CD6@gmail.com> <3D1B8F1F-2C41-4CCD-A5D7-41CF412F99DE@gmail.com> In-Reply-To: From: Yosry Ahmed Date: Thu, 29 Aug 2024 14:54:25 -0700 Message-ID: Subject: Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000") To: Piotr Oniszczuk Cc: Pedro Falcato , Nhat Pham , Matthew Wilcox , Linux regressions mailing list , LKML , Johannes Weiner , Linux-MM Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Stat-Signature: 9hhzkef4bgcpjxxou3xdugf8n4gm9143 X-Rspamd-Queue-Id: C8B6140005 X-Rspamd-Server: rspam11 X-HE-Tag: 1724968503-532620 X-HE-Meta: U2FsdGVkX18D+JcAYSd8CGkZzpuzy0SHwnfcIrIJvP7QwF9TMzk+0h3BNzXjjPhHv0hZ93SC38GCv0h4vfwHw3Izr/auSWOs9wLzaYWnH6txW4R2N311lRVUAGI3P1MMzpT9jlaTa+8KOhjmak4YSNLnJ7lrKYrQtw97BdUDTErZy4xpP4pyivEMzi/xwSJT0wY6hjb4IjbTbAv2R21SDlzjBTtzPuJiRgJDMIVDQKKuLghyi7dxosUL+fjby66LnyHClUPeFmBp1DJcXs6STv0tcySK/GwTtVVfrayIyDWzN3yh2MCjH0l6eSEtdZL4xW+4jSzl7vUhbyhQ3ZQpBDsp/q5UuVEmwZ5/vlEwy1V+PkjMTtOKI9OQUCU8ytevKESkDf/QG9HTe6/bWfHGGApsHdFWexds/sgoaY+BI2AWhrWaMppaF9+TR74h44xh8lvUKZpb6MXn3/jakmfajrl6MzbQprQbcs60P5nRFglsecMnCfvAK5/lUJ9/cFu0yHDVyPhbr/WLAJXu8FlmyqKGi9f9osKM9Fk0eP6AUo1lFgY2H4nZwkoX6GoUMG9mXKefy7Ym9bvb+fj8jk5SwDFB+PbdaRh2UM0Jup4l/+513+duug7lOP+35k666ADDDK8mSBAfXmDj2p2YHG1RxYB73xtraNqCz3PK8gGn6mzN6YC0qYF6C4MVEv8WC4AMEUH7zTSPzwy+4esY4DCen9JLClGMueMuPLCvIVm6eCINOtaR2S/TQUlm0y+Q8kKlpuAu4iCxuSD5AZdo85VcpiGIzTu9i9fNHc95qghhvZ9LMKb3+6dEMiWLy5MOF1a6oGgPh92agjPCWnACP8pBamPLxh2xfAuViG3W2oWaBB4/t3Iz7eyTBsDHXQmkqI/mCvsgyUeLbvNmH0EpZli5sT9cNUWPsVkHNjpuusLjDkxoaxtbVbmhp37D0LrKVw35t2Uu37SNW++lUEx5NH9 TIxTaxrx DT/YaJopOs0QG13Yl25TmLInn/o73uy2N0ppxfQlH9ivv9+37fJFsXy2h/Peb4YwGtUuJP0bi3JPuN4OudLt16/IokcSn5lW8XRfGV7Xn0Wsi/Sr5kct94cjJ4qvIoGk6/McLGVugmANATHkZfJ+fyBdFE8tVsoq7L4MpRo5RWi/T/dF3bNjVfRQxbVmylPuxOjvPbAqYWB5kVaqG17W0jpa9VOwNDIgIG19oLufJ6w6Fiv/SPBORaZmhBxH+7rydjEw84hfn3mc1mZrCs+EX0RWL8hYyPLSqXNkJxWmbUM+/y/v1gqHbo4xycEmEt+TCyQ8wp55guVlWyaM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000417, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Aug 29, 2024 at 8:51=E2=80=AFAM Piotr Oniszczuk wrote: > > > > > Wiadomo=C5=9B=C4=87 napisana przez Yosry Ahmed = w dniu 27.08.2024, o godz. 20:48: > > > > On Sun, Aug 25, 2024 at 9:24=E2=80=AFAM Piotr Oniszczuk > > wrote: > >> > >> > >> > >>> Wiadomo=C5=9B=C4=87 napisana przez Pedro Falcato w dniu 25.08.2024, o godz. 17:05: > >>> > >>> Also, could you try a memtest86 on your machine, to shake out potenti= al hardware problems? > >> > >> > >> I found less time consuming way to trigger issue: 12c24t cross compile= of llvm with =E2=80=9Eonly 16G=E2=80=9D of ram - as this triggers many hea= vy swappings (top swap usage gets 8-9G out of 16G swap part) > >> > >> With such setup - on 6.9.12 - i=E2=80=99m getting not available system= (due cpu soft lockup) just in 1..3h > >> (usually first or second compile iteration; i wrote simple scrip compi= ling in loop + counting interations) > > > > Are we sure that the soft lockup problem is related to the originally > > reported problem? It seems like in v6.10 you hit a BUG in zswap > > (corruption?), and in v6.9 you hit a soft lockup with a zswap lock > > showing up in the splat. Not sure how they are relevant. > > If so then i=E2=80=99m interpreting this as: > > a\ 2 different bugs > > or > > b\ 6.10 issue is result of 6.9 bug > > In such case i think we may: > > 1. fix 6.9 first (=3Dget it stable for let say 30h continuous compil.) > 2. apply fix to 6.10 then test stability on 6.10 > > > > > Is the soft lockup reproducible in v6.10 as well? > > > > Since you have a narrow window (6.8.2 to 6.9) and a reproducer for the > > soft lockup problem, can you try bisecting? > > > > Thanks! > > > > May you pls help me with reducing amount of work here? > > 1. by narrowing # of bisect iternations? My information about the good (v6.8) and bad (v6.9) versions come from your report. I am not sure how I can help narrow down the number of bisect iterations. Do you mind elaborating? > On my side each iteration is like > -build arch pkg > -install on builder > -compile till first hang (2..3h probably for bad) or 20h (for good) > this means days and i=E2=80=99m a bit short with time as all this is my h= obby (so competes with all rest of my life...) > > or > > 2. Ideally will be to have list of revert 6.9 commit candidates (starting= from most probable falling commit) > i=E2=80=99ll revert and test Looking at the zswap commits between 6.8 and 6.9, ignoring cleanups and seemingly irrelevant patches (e.g. swapoff fixups), I think the some likely candidates could be the following, but this is not really based on any scientific methodology: 44c7c734a5132 mm/zswap: split zswap rb-tree c2e2ba770200b mm/zswap: only support zswap_exclusive_loads_enabled a230c20e63efe mm/zswap: zswap entry doesn't need refcount anymore 8409a385a6b41 mm/zswap: improve with alloc_workqueue() call 0827a1fb143fa mm/zswap: invalidate zswap entry when swap entry free I also noticed that you are using z3fold as the zpool. Is the problem reproducible with zsmalloc? I wouldn't be surprised if there's a z3fold bug somewhere. > > i=E2=80=99ll really appreciate help here=E2=80=A6. >