From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4ACDCD4F4A for ; Wed, 4 Sep 2024 23:37:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 530216B00C6; Wed, 4 Sep 2024 19:37:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DD5E6B00D7; Wed, 4 Sep 2024 19:37:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 37E946B00E6; Wed, 4 Sep 2024 19:37:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 175BA6B00C6 for ; Wed, 4 Sep 2024 19:37:08 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8AE624182B for ; Wed, 4 Sep 2024 23:37:07 +0000 (UTC) X-FDA: 82528668894.30.43BF8F1 Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) by imf13.hostedemail.com (Postfix) with ESMTP id 9E68F20004 for ; Wed, 4 Sep 2024 23:37:05 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=b3xdIp7k; spf=pass (imf13.hostedemail.com: domain of yosryahmed@google.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725492949; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2i7jdujG8eIGFsZ+R4bmgK7Z9BWm3NOZw8YsCrlObMA=; b=pZODojtn02IhbgIMglxvkXsQ/aJUwfdawIzoZV5b32tGZTMf9sYqG+330ixNPmI0pkQvSs 9JCTsVvo79H9e2Nx61swMIQWq4ZO8GJBDhDfzyk4RGMkUYX6Ok4xWXI26PUuqRONJ2W3Bk VcsP/ksX9bCkifP0e7rTJOijWLmBh3o= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=b3xdIp7k; spf=pass (imf13.hostedemail.com: domain of yosryahmed@google.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725492949; a=rsa-sha256; cv=none; b=K7fYdv/6hTwDUYG9nGeMZchBXSsIDQQ50OJigB5gA7uMTT8op9dY7CBxzHno1OcDqL9mP/ bRDaK9XVgnrRhCpNgwEVAYz0J3U/sK4DvTaGwT+tsXsimbiSYCQs/F//O5JEXwuzf4sZOL xsIbwye/gctA/UXa7v1yyWcyY7zzF28= Received: by mail-lf1-f43.google.com with SMTP id 2adb3069b0e04-533521cd1c3so127642e87.1 for ; Wed, 04 Sep 2024 16:37:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725493024; x=1726097824; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=2i7jdujG8eIGFsZ+R4bmgK7Z9BWm3NOZw8YsCrlObMA=; b=b3xdIp7kNwfbLiX1hN+EgmeRWyZ/erLeeEJ8vvX9PEPZFhWJF78Ngcj+Os1pOc0p+c qBbOPFJGGfUX9kt2CDs4eAmNHZiZCGhtc9yv669dnqeDHycasPuaKyLGqQb0v4LujwWf 4rlFa9pOLorRQe5mwO6fHkgRe+Ek9BBU5pXOBEkuQVWpNc6z+pcwC0xp0kfXgBn5SRWx gj2nFdsNlEimuZlr0cT2iYA+z8Kew9/fId4qQxViyyG/rkY1kzZwLEHq5XPq2KxbY3iv JobAGl6fPNqmVKwuchzSqxhKUgyOFHvcVDTwwkMOxVVpLs0FM6Wm1dwkY8Id94zlhpRC JGoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725493024; x=1726097824; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2i7jdujG8eIGFsZ+R4bmgK7Z9BWm3NOZw8YsCrlObMA=; b=bz92i7hcJxhjf0kdXieGM6Kris80fytosArYZw97juQNRVo0g3vbq9AbWi1vrOg+MG slXPeXL1ZJp0dxYb2GrYSVKlqBs6r8/QVlDBxXV3rzPZ4U50+EK9fWACeRWTR5rUGI0z Socj8uheJae/mT0Ee6k6FcPepwY7YTsRRiPz530eTI1mj/J4naQ3w8bOlaxZ5MuEO/VQ bR9Go5+L8udRiLpVrmzk9LFL/e5E2NdcP6nd4IL5tkPJU/+rqFZHwTcjoK21IAQ2UmBd I7kuFZeXv1ZIpitkWLE0Br5M8T08oUYd6/jdKlxxNNFr7kAqx8gLDjDdQ6Y7oLcFy5fc V2tA== X-Forwarded-Encrypted: i=1; AJvYcCX4VWEIt/Y1CvMdEt+De9/T/OTa9dggCFFMo216baMe5iLeHr4huvlSw0ag4wfZ0Nbkb3PYZEGGFA==@kvack.org X-Gm-Message-State: AOJu0YxT8MUqjY+kRNRivgiCzvfBxGcQQ1jsMXgrVxmVDYqV8w5s5nea ce1yQDkBRDzusD1Fl+y2aUa6/5592ICD3jQHUxalms4T9JK0i3GNZEfVG8i/918FtkcLKDjbM6l arvYf8X+NnDy5W9G/S/RhBopaniJ/7219BGE0 X-Google-Smtp-Source: AGHT+IFqfOcvoCbUS5vOqB2O1EnmAuLB1CPavvPvZTFX9sDAQaTS7Ezm3gmoT4QSh8MVYb0iQpBoMJ7C62D7EK12AuY= X-Received: by 2002:a05:6512:3a94:b0:52e:936e:a237 with SMTP id 2adb3069b0e04-53546b345a1mr17100135e87.16.1725493023268; Wed, 04 Sep 2024 16:37:03 -0700 (PDT) MIME-Version: 1.0 References: <6f65e3a6-5f1a-4fda-b406-17598f4a72d5@leemhuis.info> <02D2DA66-4A91-4033-8B98-ED25FC2E0CD6@gmail.com> <3D1B8F1F-2C41-4CCD-A5D7-41CF412F99DE@gmail.com> <277CDE7C-7ED8-4840-9C30-533C9327B028@gmail.com> In-Reply-To: From: Yosry Ahmed Date: Wed, 4 Sep 2024 16:36:27 -0700 Message-ID: Subject: Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000") To: Nhat Pham Cc: Piotr Oniszczuk , Pedro Falcato , Matthew Wilcox , Linux regressions mailing list , LKML , Johannes Weiner , Linux-MM Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: a4khxpnt478zsmdzxfyn84543yabne13 X-Rspam-User: X-Rspamd-Queue-Id: 9E68F20004 X-Rspamd-Server: rspam02 X-HE-Tag: 1725493025-880555 X-HE-Meta: U2FsdGVkX1/v0btoVZIEsZbdWcTbr/Lb5DTCbrQEwVKy4/d/2qQppE19JsEgjYpFn8lW9+/NYVqBL6WW/HopEHpzGFjUunkOuwZjje5vPETwc4W6xI1ZaujVJSa8P4AjANWQr0X+BE4I8OQKnikYSJJQBOohCblDPIgxyVeECxn3/t77WJHhQ6YWq55XQwM1CTFp8oQv8XN73cUbkoV+vkWnd7yQyNtZYHGvpqN28Spj0wxh+PPtDxZ2IZJ4X2jWK5KVtnA0sZdmMZf82s9aOVzNZ3+GK2mz76XeiWPYyGjoGGdZcfIkinubUnnPAa5iHNp+b0aJubDfKgFfEvuTQn0b1OkLprbY6VuENNIcra9xcL2N3eFeUxia/YhUdEps2jVGbwA4aGUauT/21Q8ckkGuqhvMH7y/uTqRHBQGslfju65sC9PQqhQfxv27rggY75MCwI0r0V+HHp0uuF0KheKeRSQdiiDBlM/BsP7Dpxep4kVpmjr+HFx7rI4kFfco2bOeMiYZdL0XunjdkOziks99/8kPgk+FsWpVebXjXK18ohRFV3qqPGAcGAQBOiT0Do3eZ9hlhSC8VAOSsIcRVQOrTeSrawA2wMfixTLuPbklvcMlHWsID2/LMjVHncyfpmpz0HC/pjtQUfXgM7bz1HwL+ToENR/0sRoVWJtV6Yx1mVGXnbQGJFXRVGDmnWxgQ9j/aoLVNpGNWZxsi2mf8E4z+KfEHUQSHGFl+csVxap9PsauQR7fgMjSHgQ03vRPZYkmsdaDlvRU5OtuARXNvV6NvlPzyDbSWdit6zL2Q4dHOo/Bj41Ah9AG97LcwfhN6w2CAaeXUANJyfyyetmC0dyM7GoRBRDiQoiL/YyfWsIv2h/kvgtDFEu9+iao4g6XbVFRTOr+dEXzgVuCreXjU3OCjeGI9zmuTnIiKsXRPAc0SiXUxd7Zh3pmyoMCKI+BHhLxrQQQy+Q02KvJUM7 YtozfSoi UvmCTLFM+e5u22JgJ7yndDpUrcg6SXGLJtKhwmljo9qryBxOC61WZNpHXiQU0ffrMwZCpxXZ3kpqyvNc6HjEsteb2bCvx3dpcfC/5VGgzm2HA6sfTrTuHndS1vqgERKng/sUHt8p2DE0Uj4xr8usH4OHcI4qke7UWgfywn+gQTF6maMFXV3UGQ8FG1LpkSQ/5cGe0oMRy/SPqrh34FIiJTbee1Fo6y/DLykbf/oxdN+LLS2ewLIZl+/Ial1ZPysJMJvsqqlkhPeCRF9od78LmYljnlJRpKpbkaw9rrAAlmY8PQYeO+1hjakxOEvV3RQ1MuBfR+5mymtwElXEhQNRRVSwg2F7DhIb7v2Y1BZ2EuTn1isfJDj0dc+ZYQ9szG34o2HhOFUJo1FPNkQZw4x+ByhBhDwJSvYWTsJoEJPsB+3JhvRUxBVH8iXCLpAOFP7Y1ue1y X-Bogosity: Ham, tests=bogofilter, spamicity=0.003799, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Sep 3, 2024 at 3:43=E2=80=AFPM Nhat Pham wrote: > > On Tue, Sep 3, 2024 at 10:49=E2=80=AFAM Yosry Ahmed wrote: > > > > On Mon, Sep 2, 2024 at 1:58=E2=80=AFAM Piotr Oniszczuk > > wrote: > > > > > > > > > > > > > Wiadomo=C5=9B=C4=87 napisana przez Yosry Ahmed w dniu 31.08.2024, o godz. 19:23: > > > > > > > > On Sat, Aug 31, 2024 at 2:41=E2=80=AFAM Piotr Oniszczuk > > > > wrote: > > > >> > > > >> > > > >> > > > >>> Wiadomo=C5=9B=C4=87 napisana przez Yosry Ahmed w dniu 29.08.2024, o godz. 23:54: > > > >>> > > > >>> I also noticed that you are using z3fold as the zpool. Is the pro= blem > > > >>> reproducible with zsmalloc? I wouldn't be surprised if there's a > > > >>> z3fold bug somewhere. > > > >>> > > > >> > > > >> Hmm - yesterday i recompiled 6.9.12 with zsmalloc and =E2=80=A6. a= fter 16h of continuous tests I can=E2=80=99t reproduce issue. > > > >> With zsmalloc 6.9.12 looks to me like stable. > > > > > > > > Interesting, and a little bit what I hoped for tbh. > > > > > > :-) > > > > > > I tested mainline 6.10.7 with 26h test and also it is stable with zsm= alloc > > > > > > > > > > >> > > > >> With this - what will be your advice to move forward? > > > > > > > > Well, it's possible that some zswap change was not fully compatible > > > > with z3fold, or surfaced a dormant bug in z3fold. Either way, my > > > > recommendation is to use zsmalloc. > > > > I have been trying to deprecate > > > > > > IMHO - isn=E2=80=99t bug in this report + difficulties to reproduce->= fix enough to depreciate z3fold? > > > > I would say this bug report is yet another reason why we should depreca= te it. > > +100000. > > This is precisely why I was asking which allocator was being used > here. We have also accidentally selected z3fold internally a couple > times in the past, which had bitten us as well. > > > > > > > > > > z3fold, and honestly you are the only person I have seen use z3fold= in > > > > a while -- which is probably why no one else reported such a proble= m. > > > > > > Well - in fact this is ArchLinux - not me. > > > I=E2=80=99m using Arch and kernel in builder machine with ArchLinux c= onfig + packaging > > > > According to [1], zsmalloc should be the default allocator for zswap > > on ArchLinux. Anyway, I initially thought that no one was using z3fold > > and it was bitrot, but apparently some people are using it and it's > > actively harming them. > > > > [1]https://wiki.archlinux.org/title/Zswap > > > > > > > > > > > > > > > I see benefits already: on very memory demanding qtwebkit compile: > > > z3fold: swap frequently gets 6..8G from 16G available > > > zsmalloc: can=E2=80=99t see more than 1..2G > > Exactly :) zsmalloc is better than z3fold in a lot of workloads that I > have observed. > > > > > > > > doubt that you (or anyone) wants to spend time debugging a z3fold > > > > problem :) > > > > > > lets depreciate it! > > > > I tried deprecating it before [2] and performed some analysis [3], but > > there was some.. resistance. Maybe I will try again and use this bug > > report as yet another argument for deprecating z3fold :) > > > > [2] https://lore.kernel.org/linux-mm/20240112193103.3798287-1-yosryahme= d@google.com/ > > [3] https://lore.kernel.org/lkml/CAJD7tkbRF6od-2x_L8-A1QL3=3D2Ww13sCj4S= 3i4bNndqF+3+_Vg@mail.gmail.com/ > > I don't wanna sound like a broken record. But this has been the nth > time we need to spend extra engineering time and effort unnecessarily > because we have not deprecated z3fold. > > If you need more datapoint - here's our last conversation where z3fold > was a problem: > > https://lore.kernel.org/lkml/CAKEwX=3DMo+EaaxBYcLMTHYADB4WhqC3QmWV3WQ0h2K= M491FRuQA@mail.gmail.com/ I sent a v2 of the z3fold deprecation attempt: https://lore.kernel.org/lkml/20240904233343.933462-1-yosryahmed@google.com/= .