From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A3CACD37B5 for ; Tue, 3 Sep 2024 22:43:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 258D38D01EC; Tue, 3 Sep 2024 18:43:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 209328D01E4; Tue, 3 Sep 2024 18:43:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F7E08D01EC; Tue, 3 Sep 2024 18:43:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E076B8D01E4 for ; Tue, 3 Sep 2024 18:43:57 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8EE9FA0A0D for ; Tue, 3 Sep 2024 22:43:57 +0000 (UTC) X-FDA: 82524906114.14.2E27C1A Received: from mail-vk1-f173.google.com (mail-vk1-f173.google.com [209.85.221.173]) by imf17.hostedemail.com (Postfix) with ESMTP id B32C240008 for ; Tue, 3 Sep 2024 22:43:55 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BIwV1vkA; spf=pass (imf17.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.221.173 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725403387; a=rsa-sha256; cv=none; b=zfZWAzMNzQW/oXet9/8bFj04oTSbaZ8ubeOfNse/J1LGcxVsW2EL2XTAYasNwHXuPUKv0E GwHF/dMjMwLxhKKJo6T5ite8EPK4VahuMBOdCx8LUIpvmQn75L5mDRyQWjPkEiyjfeHfv3 J5DdkQ1N5c1Ltvag/pFPe/eqWisWU+U= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BIwV1vkA; spf=pass (imf17.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.221.173 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725403387; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8FiFXXA/l3FkeAr8G3sWENhyWD2nRnjVPnBxOLfEYnc=; b=7ADlCfnI2c2eh39xITKQQDK47qv1OkfThEePaGN/7EzvkNR1kL6uV0WNSc1ZMmd2hnqXbw mk60uobnPqsSSKF9PmeIJ8YrUHxy44VOTlYN9aMPOO5OqPGFTIf5QLFsu4m7hPMdKga/Oj +lhTmzY1u0ZOsFbDv1MqEiqLv+i0RgU= Received: by mail-vk1-f173.google.com with SMTP id 71dfb90a1353d-4fd117e0008so1773693e0c.3 for ; Tue, 03 Sep 2024 15:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725403435; x=1726008235; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=8FiFXXA/l3FkeAr8G3sWENhyWD2nRnjVPnBxOLfEYnc=; b=BIwV1vkAO193kbCtFaiodL59BsuhkkL5irVstEgWjio2ONABji/J0nLmwZ8VDE2tUd 5bfiK6S+FoylefIW8igx8YUclx/MyJoatfponP8OYkX8mogHzhWqJZuFCYX/RTnCmkGq Ktu94ocMwn1zH0zDLs2c7EDe7u+tZqryQWa3sWAzT4e+d+1Up1vBw55lS3mHF4hm+P57 xyLIZ3iTdCW1jStTlaqbUoevKDmIN7ghuAZtD7nVADXF6skXfv+TaTOtEwHT+t7kZUVN Fht65CcKIOWvHv1AY8EU9EIa5Eb3KOEwJUO+QDy4JWx0HpYou4yV7Mn70vxk9w1djhmc dp8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725403435; x=1726008235; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8FiFXXA/l3FkeAr8G3sWENhyWD2nRnjVPnBxOLfEYnc=; b=icX3XwOxT5OKu0On3eFjZ7vS7n5lO9FydUNc0jQOLYVFNuBXv82WjnbD/Kf5h9l3xZ ydrSm65kdzTEB3ZQJnv6XojByDzUrZWaOFtuH2qWtA/1/t6byitXG8DNbsOfmNgjtufN orNYTy9/YFD87y0O1p1Jp+3RGiX9Uorj3uoOoLul4akTQySqWquX7nKTLcRvL4OADc2Y yYXsMieE3SGfVBdpC+ml9x05Nhx/vb3kRSvfrZ+Na3NXOObs7wFUn6vgfLkpX8/4r/Vn Q5wlpOxBBKTTVnfEykyCFipLai2KNA4UMiccox4D2vJSEwtzMCLsJSRVJHp5bW2dH43y WLVA== X-Forwarded-Encrypted: i=1; AJvYcCXoYJ8cdg2aUWyRSpUMf7whTrQ+boD6qC544CcnI7bNi0X1TJtpLVBoriIJNmkZe3B93LfZYwAF9A==@kvack.org X-Gm-Message-State: AOJu0YxGbRbr6NqM1xFKDT5iiCfeyMYC3wKosfSDcQM1/tNbFAH1+8xg 8tiBSZksPw8iF+WHiSALquwtNp0Fz8hvbNTZ3IR3JBPSOcl564JpM4Yyd1fYnFe9vt1cNic1Sfu ioVn28B/bgahvdxqC/5gnlMGjdw4= X-Google-Smtp-Source: AGHT+IENyUbCLbadEWFLpOh2q/6waOSadl+medXd1iZVdEMn6jiqqDqBGZRw1xe7jx6KhhamVY4Jg5rPOa45zS2xjGA= X-Received: by 2002:a05:6122:3115:b0:4f6:aa3e:aa4c with SMTP id 71dfb90a1353d-5009b1b6015mr10311639e0c.3.1725403434612; Tue, 03 Sep 2024 15:43:54 -0700 (PDT) MIME-Version: 1.0 References: <6f65e3a6-5f1a-4fda-b406-17598f4a72d5@leemhuis.info> <02D2DA66-4A91-4033-8B98-ED25FC2E0CD6@gmail.com> <3D1B8F1F-2C41-4CCD-A5D7-41CF412F99DE@gmail.com> <277CDE7C-7ED8-4840-9C30-533C9327B028@gmail.com> In-Reply-To: From: Nhat Pham Date: Tue, 3 Sep 2024 15:43:43 -0700 Message-ID: Subject: Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000") To: Yosry Ahmed Cc: Piotr Oniszczuk , Pedro Falcato , Matthew Wilcox , Linux regressions mailing list , LKML , Johannes Weiner , Linux-MM Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: 6k67daewi6fqqzrejoy56ht7gck4qzcm X-Rspamd-Queue-Id: B32C240008 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1725403435-126700 X-HE-Meta: U2FsdGVkX19WVYet/iFKH8I+WNZOIcn1fN5bb2IKisrZHkbxZBYnvxsHsD9k43pu/rZzI/FrV6qMj+fXmydmxJkoohznpEagUHXMq41Ll9AroHIBpSM59kJKeU3+Oj/iQh25z2nCJJAlE4719bkJVb1uHXV+7gr1SyQY9Gc2mXOgbSG/jzgGhoKG0qp2QrvZ3VdYZIxX0EZ51rBZNrz0l+oESDzMP1icIWEc9daW53F9H7m8+hpnEAlmhD0wc+GFSji5Kyg7t6exBW8aafDxZo0k/Bk3SDiHMXwRiEw4qb22U5mtM9D3lN/bYYO0+mtz1XOOdsHWUHU8ExxHizD3mSwelgBiuGZZN0prInv7iJnGdnnDWB/LC/eoo75jpmwfhP4igBOGAE8gn1QEltGvbIt3oHFGE2BiqxHL0M5yT79Z0xCjfIZlooR9TjORtT14JL+uSuf3C9ReHgriOjvqU4UzLvtOtZ65OD8pEd4/2k0MW5VG6boBML4L7x8r86cluRX1xC8QKgAvmmX7ejDd3yBvG/KSQL+iVKGF3A58yVoragkP9ERdkg8TbIk9aTkn4u9jBaCzQkYkQsPQWEdFmN5o1g2DdLItLaHzr9gfeiDGdRetZMHl9pUkvxWYg9CbQG7JAlrBSe/RAdf5uDjXZWo7c7DhSp9vISy8ircvrYHIiFZwR5vmuYhQiRLWRXyBa6RFmClQNJstjNO2Y3luK+IvoG2kw9GDjHsxchs4xxXNE3rtQhHdB71a4/QevyYlEYUiRqn7ImswYOp5B1pMclYP54MmalgJ0z766h9ctYWx5Go3qR+X5j1y8iVDclA4SEP45EqoaaI0Je6Q0UEAjItVjy2XaaGT3ygr4Tblj0fWZ/mWJXrV/U+F4RzhgXJVmoWv0R5DUFemwbn1HbDTJUEWCt1fZVXInGF/1JYwnrDsuy+SfECjj9PHhzy71kGkaPauxvRlL2eTsm5Um6t N7DEu8FP k7/xQkAkUdHc1Hq9ZvUuErfbRzePxndP7m491GseVmmkvew7E5g7VKwuIS9bkarLsVvUgDuBxxadqaqDIl3HWp2JIFbqj/fxkJIVcEz2zcPgYeHiZhRsWsdwI3MxINSImp3pHtnit0/hqvkUo45BiOAOPM2F+QsbwytnvdRI6t94Pjj1V7FvLOJswPJPAfwZ9g6kOa+LPsis3DQBBrnhq6p6ShivOtBPsEsEP+/BuKYnOYjMEwPP0dB0EdAtTYzu+2DhIQm4wP3LEaReWnXQ54cnyRMDqB5hPs60m7j2+WNJ2D0fabTPiVh0gk/KjAOxGjD0FTbIU0u1j2rInQzKfM//RvCjMx0H06o+2EUq8feHHdZv7NesC3llpxbvELV16AnJk1gzg6MvR1uZRpFOczFzaeHmE3rXuDnj7T59DvNtH9sbJzxgL7GsISYg9kZ8HaM/61TG0QNaYxac5zcsHJophy3zLqNkEG93t8v/r13S9YAd0c8NuhFwenFWtD6aAIwO9t3+nYPUeSAbLhHqGoFSdhFdtukcn6vxkYTY5jiPuOQXSxJSg4fpc1D1CCA9e+IkY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000092, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Sep 3, 2024 at 10:49=E2=80=AFAM Yosry Ahmed = wrote: > > On Mon, Sep 2, 2024 at 1:58=E2=80=AFAM Piotr Oniszczuk > wrote: > > > > > > > > > Wiadomo=C5=9B=C4=87 napisana przez Yosry Ahmed w dniu 31.08.2024, o godz. 19:23: > > > > > > On Sat, Aug 31, 2024 at 2:41=E2=80=AFAM Piotr Oniszczuk > > > wrote: > > >> > > >> > > >> > > >>> Wiadomo=C5=9B=C4=87 napisana przez Yosry Ahmed w dniu 29.08.2024, o godz. 23:54: > > >>> > > >>> I also noticed that you are using z3fold as the zpool. Is the probl= em > > >>> reproducible with zsmalloc? I wouldn't be surprised if there's a > > >>> z3fold bug somewhere. > > >>> > > >> > > >> Hmm - yesterday i recompiled 6.9.12 with zsmalloc and =E2=80=A6. aft= er 16h of continuous tests I can=E2=80=99t reproduce issue. > > >> With zsmalloc 6.9.12 looks to me like stable. > > > > > > Interesting, and a little bit what I hoped for tbh. > > > > :-) > > > > I tested mainline 6.10.7 with 26h test and also it is stable with zsmal= loc > > > > > > > >> > > >> With this - what will be your advice to move forward? > > > > > > Well, it's possible that some zswap change was not fully compatible > > > with z3fold, or surfaced a dormant bug in z3fold. Either way, my > > > recommendation is to use zsmalloc. > > > I have been trying to deprecate > > > > IMHO - isn=E2=80=99t bug in this report + difficulties to reproduce->fi= x enough to depreciate z3fold? > > I would say this bug report is yet another reason why we should deprecate= it. +100000. This is precisely why I was asking which allocator was being used here. We have also accidentally selected z3fold internally a couple times in the past, which had bitten us as well. > > > > > > z3fold, and honestly you are the only person I have seen use z3fold i= n > > > a while -- which is probably why no one else reported such a problem. > > > > Well - in fact this is ArchLinux - not me. > > I=E2=80=99m using Arch and kernel in builder machine with ArchLinux con= fig + packaging > > According to [1], zsmalloc should be the default allocator for zswap > on ArchLinux. Anyway, I initially thought that no one was using z3fold > and it was bitrot, but apparently some people are using it and it's > actively harming them. > > [1]https://wiki.archlinux.org/title/Zswap > > > > > > > > > > I see benefits already: on very memory demanding qtwebkit compile: > > z3fold: swap frequently gets 6..8G from 16G available > > zsmalloc: can=E2=80=99t see more than 1..2G Exactly :) zsmalloc is better than z3fold in a lot of workloads that I have observed. > > > > > doubt that you (or anyone) wants to spend time debugging a z3fold > > > problem :) > > > > lets depreciate it! > > I tried deprecating it before [2] and performed some analysis [3], but > there was some.. resistance. Maybe I will try again and use this bug > report as yet another argument for deprecating z3fold :) > > [2] https://lore.kernel.org/linux-mm/20240112193103.3798287-1-yosryahmed@= google.com/ > [3] https://lore.kernel.org/lkml/CAJD7tkbRF6od-2x_L8-A1QL3=3D2Ww13sCj4S3i= 4bNndqF+3+_Vg@mail.gmail.com/ I don't wanna sound like a broken record. But this has been the nth time we need to spend extra engineering time and effort unnecessarily because we have not deprecated z3fold. If you need more datapoint - here's our last conversation where z3fold was a problem: https://lore.kernel.org/lkml/CAKEwX=3DMo+EaaxBYcLMTHYADB4WhqC3QmWV3WQ0h2KM4= 91FRuQA@mail.gmail.com/