From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35FEEC52D7C for ; Fri, 23 Aug 2024 18:07:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE729800BE; Fri, 23 Aug 2024 14:07:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B1FA6800B4; Fri, 23 Aug 2024 14:07:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E77F800BE; Fri, 23 Aug 2024 14:07:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7BBBC800B4 for ; Fri, 23 Aug 2024 14:07:08 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 11930AA426 for ; Fri, 23 Aug 2024 18:07:08 +0000 (UTC) X-FDA: 82484291736.11.6ED2579 Received: from mail-oo1-f49.google.com (mail-oo1-f49.google.com [209.85.161.49]) by imf17.hostedemail.com (Postfix) with ESMTP id 338B640018 for ; Fri, 23 Aug 2024 18:07:05 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mxzg9W84; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.161.49 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724436359; a=rsa-sha256; cv=none; b=3QXpXkhA3moGMSSQEDM6s/uoBbKIQHNG4IfpzwwGTvvsHmIPBkmLViKt4zX9A5DVhpaWu/ pP/d34NGWUbdasae3TPCLCx/3H6p8qJeDIGZnDsHVKbYraYl5M4ASqeGphEAQuM8iQsTXx 29nwIpdGUBIf0TrFpBbL9aUgwMdz9dE= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mxzg9W84; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.161.49 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724436359; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fpgh31z9NlGdlITa1fFDfgoc4AKpZdcCLAm16vs4/MM=; b=WrrPa2pQLL/ASPcPXwCBzw07pMiSgO7jqRU507sU7GBNa2yqZ4J3Vmpg6Z5YVUcUKyQuPX xMBZSjsBOGXruxFfv+ROKI+99v/+Qg8Plbw60u65ISbyRcLuR5/ZMdYnOeX2WYeWtvTTT+ sLkrasn3BUotl+bVFclX6Id43W3cOhY= Received: by mail-oo1-f49.google.com with SMTP id 006d021491bc7-5d5e1c86b83so1785721eaf.3 for ; Fri, 23 Aug 2024 11:07:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724436425; x=1725041225; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=fpgh31z9NlGdlITa1fFDfgoc4AKpZdcCLAm16vs4/MM=; b=mxzg9W84FuH7QLDc/ObDfxyV9l7RJYQgVPPNq7HELH5gxUb3WWpfkOGWrxLPup+Boo hnNRVs+5oI+XQafGg/MKclNo1PZdv1wOpDOXwpD+R+LA3k+T5RIOYHG44bFsrV2qLFks 6lWPJiZjkBFBFTHhTK83xaR/1T/cpQrAL/9xOMWVyZICpjv25qkg2uuHQAwGW52u1Lgq clkOwiOrK04qAMNnjy70MSs3ubBJGLdkH9NZnPa0aynhLh3ClqhQH0PG2glNZn0XVLhQ RJxw8yADAZDarO7hDV7ufrF6wBht6pyu2+eB1GX4U6DnvjbQuNVbqrARaLrW17fIw2+t 2Ffg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724436425; x=1725041225; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fpgh31z9NlGdlITa1fFDfgoc4AKpZdcCLAm16vs4/MM=; b=fCpPgmZM6cu/kbkG/UdxuQ1x7L8M/SAkzevO+KhKH0f4rCmatGJ1dccHbmRV+13a9G ddArC2+0UiRNc+OZmN7z1Of0ZTSg0xPM0fbSHcIpDxkWEPyuCPPhmCzFND0Nw26iVzUE /PvGX/M/vCh05tNSG/wSTHnQDgj0VijujO5w0Icyoe55sTHRyAZ0QUUyD85m6LbaWhOG rpXVvja9PU+db9OPG7A9PBVYpY2ja5kVH0kyzUQ3HTMImHMjbzEQ9d0fprHb8nsX2M5N 2EYMOpTHcZFvi8CVl3hBPepZAIpBKfOAQpfkWsB/Fob0i0JGyOtvRQNEKmx0PSNqJKj9 pwAA== X-Forwarded-Encrypted: i=1; AJvYcCXR8LCAim6MyuDeLGmZw/dutX6T+XAc5jgRkbbkMnTh/B+7mjEGAo8t8JV/c6sIKq0Y7+eYs95bDg==@kvack.org X-Gm-Message-State: AOJu0Yxfolw8NXjs0OJFecTjQOYnWetseyA6FD/GaqaangHOehjyreLH bdnTtc+AxAwf9PeZ0ecIGMWDsDLylE/rq8ARuOSSdWoUxFQHrsLJ1cQaGCvwY8E9lGmg4k2yR/C NLRKsdedGnrntMWGGpYSgLnKCphw= X-Google-Smtp-Source: AGHT+IGEXqYJ4XE5UaIgMWPe4gXgslzxa2GlWa13l20lAXEX1G3nCklEFxaTPJ4BX5NEP4cBklJ3DR9np92KXeFhKNg= X-Received: by 2002:a05:6820:206:b0:5d6:ae6:a852 with SMTP id 006d021491bc7-5dcc62754c8mr3354442eaf.6.1724436424921; Fri, 23 Aug 2024 11:07:04 -0700 (PDT) MIME-Version: 1.0 References: <6f65e3a6-5f1a-4fda-b406-17598f4a72d5@leemhuis.info> <02D2DA66-4A91-4033-8B98-ED25FC2E0CD6@gmail.com> <9793DBCA-13F4-4B47-AD57-12A62F7DD8DD@gmail.com> In-Reply-To: <9793DBCA-13F4-4B47-AD57-12A62F7DD8DD@gmail.com> From: Nhat Pham Date: Fri, 23 Aug 2024 14:06:53 -0400 Message-ID: Subject: Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000") To: Piotr Oniszczuk Cc: Matthew Wilcox , Linux regressions mailing list , LKML , Johannes Weiner , Yosry Ahmed , Linux-MM Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 338B640018 X-Stat-Signature: heb3p6p34kfugyze57q7o6rbqfh5rcjc X-Rspam-User: X-HE-Tag: 1724436425-74213 X-HE-Meta: U2FsdGVkX1+g9EjWUy6QkPpw1N57wNHr1DucY3OTV4X80nUNbe67wRoUdvOkml0vqcbW80E9oZ3j9y8j0DV/iDvdVqBgUmL1ZDTD6e6NlTV5L03pH/OvwBAHmBPs8r39Ca38f14CiVYxNzBzJylpYaPBCSLtautC46bURv9CpceCSoCtN5xB2Sflkl9LlE5NsdIX6pf2ANrihHrCk8ealfARdbFfvaTBopnV8ODIiMQjU5aBU1zUBvnWklRomNLYqjUAFs8rJ6IVYoYVvJEuDhXZWC1VD//q9VFvxgCHUM/CkDOnc/Ud39/XZ9/rEE30eaT5P+xMIfkeIkUcL4gL3z/JAcvwMpWf+8OMb7Ab6TD/A5/JYbe8D547V/2BqJdmz1VupMdcK4QrUHmnUn/dZdo5jTmkY4SzHTEfNGxZxA6iCifAQZkTU+haBQCyfDl1Galq+neimuQZK5Bp8d5R4iLX2FAOJ7FP5xUOauG8KYpyYRtOxvgkxQV0oK5y2FZi5K2OrdCShcdc71QAwaF6kqJ1iCPQis5UhTpEbiUarj8Tr9FBu96d9KaGM1zA4vxwPkacnbmt+WlGiJ9IJFhT76AoaWyOEZievyaNOmuDjsBe5TDfv8ASBaX2VeuusOQ+FQWbydnp0605SqRKNZMKUzBd/iq/GCIvNmt/HzekrQIa5++T6VdQNZLHCuk4zbZODSQ3xx8hT+cRon5IN7KUsZNpKeNAN/yk/lSoVk838nNH8BrNOUQBrnAOJVlQoXSHgcGlbVIEzpWp+RPdd5pMHrOv9P3T/CS+yXOai8Fd+JsfHvWehpmIXwOGtOdLQC8/il0bKhTCduSz6nebJY4cHQafDQFNwzNCrX3Qf6FWnR5WTvkKK6awLLL/Y0QMX+2UchFMAV//pfnU5EHCYpYd3FTdN53KYK/hP2d0s57wECFrSfffONiNTkdd27dlx2nJ3Ugcyv52ehjMIiGemf2 qeJoY+qr 2tfU1ycDnOq7wLAQCAbXmaRGSG4z5HssHWUG0Y6tGclnSKr6qgwx6IIOocALE2qGrsl8vjlIBDvpyW2r3Fut0ZQfHCkx36+ZrApF134GUsJbevGaS8E79o/qwy5sO5pyN+Ue32sq4fpZsnQs75e9eHWtlnVzLI22iw6owO7YICrTg+bJOtwK5Hh7cbeOYWB0nvuZlAi87uIK6xPXdy4aAUxVtA1oy9ktcbWPvGIJAZsql+uUF83hLyqOT6UseeMy0sfgw6nosiGK0NkHMU5v5UqMSqNFJkJJ50YzA0Lpp1Fmxks8lD+3MuRXHQFBn8fEe+/UuPaxoSR6RGr6WYCHQJQanrIpJxeA6cd2iTe8Lueh9tKfRHQm1jBq5T+o6ZHeuFxCxjQIJECX3sNQucKeuzWWj7DtOVb/+BBJLdpD/X2ewh7py/dPEVd2204Ddn2Q4yADa+Ho/v40GpwFV8J9PpcXn9UAtvdCgjSGo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000774, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Aug 23, 2024 at 1:24=E2=80=AFPM Piotr Oniszczuk wrote: > > > > > Wiadomo=C5=9B=C4=87 napisana przez Nhat Pham w dniu= 23.08.2024, o godz. 18:16: > > > > Have you tried with 6.9 yet? IIRC, there are two major changes to > > zswap architecture in recent versions. > > No. But now building vanilla 6.9.12. Will install and see=E2=80=A6 > (This will take some time as catching issue needs days of compilation) > > > > > 1. In 6.9, we range-partition zswap's rbtrees to reduce lock contention= . > > > > 2. In 6.10, we replace zswap's rbtrees with xarrays. > > > > If 6.9 is fine, then the latter is the suspect, and vice versa. Of > > course, the minor changes are still suspect - but you get the idea :) > > > >> > >> btw: we can go with elimination strategy. > >> So what i need to change/disable to be closer to finding root cause? > > > > Could you let me know more about the setup? A couple things come to my = mind: > > > > 1. zswap configs (allocator - is it zsmalloc? compressor?) > > Well - I=E2=80=99m not using zswap. But the bug happens in zswap path? :) Could you do: grep . /sys/module/zswap/parameters/* > > [root@minimyth2-aarch64-next piotro]# swapon -s > Filename Type Size U= sed Priority > /dev/nvme0n1p3 partition 16776188 2= 94164 -2 > > > > > 2. Is mTHP enabled? mTHP swapout was merged in 6.10, and there seems > > I don=E2=80=99t have used config at the moment, but /sys/kernel/mm/transp= arent_hugepage in I see: > > =E2=94=82/hugepages-1024kB > =E2=94=82/hugepages-128kB > =E2=94=82/hugepages-16kB > =E2=94=82/hugepages-2048kB > =E2=94=82/hugepages-256kB > =E2=94=82/hugepages-32kB > =E2=94=82/hugepages-512kB > =E2=94=82/hugepages-64kB > > > > to be some conflicts with zswap, but Yosry will know more about this > > than me... > > > > 3. Is there any proprietary driver etc.? > > > > Only 2, both ryzen9 monitoring related: > https://github.com/leogx9r/ryzen_smu/commits/master > https://github.com/ocerman/zenpower/commits/master > The reason I asked this is because I've seen proprietary error screwing with memory in the past - it was an NVIDIA one though. https://lore.kernel.org/linux-mm/CAKbZUD1-kqfuV0U+KDKPkQbm=3DRwzD_A1H3qk_c+= bw92CqtMbuw@mail.gmail.com/ Also decompression step failure (albeit in the writeback path)