From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A0A2C4345F for ; Tue, 16 Apr 2024 23:29:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6799C6B0089; Tue, 16 Apr 2024 19:29:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 62B2E6B008C; Tue, 16 Apr 2024 19:29:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4CA486B0092; Tue, 16 Apr 2024 19:29:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2F1C36B0089 for ; Tue, 16 Apr 2024 19:29:54 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id BDCF9A0C73 for ; Tue, 16 Apr 2024 23:29:53 +0000 (UTC) X-FDA: 82016989866.06.47D5596 Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) by imf09.hostedemail.com (Postfix) with ESMTP id 27901140011 for ; Tue, 16 Apr 2024 23:29:51 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DBIg+hQM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.219.44 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713310192; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XLy1Ua1b77r4qsxuukriMKLfQ+NQtqpcoJqaT5UdrRA=; b=Uu+ep+FadNB0g28AiWgjH2T07wOm3DBx9fUvH6xC/phPuftIl1L4p8HFhO59BklGt4qSAX +FXUdA0RWJoh/KSdHNoXFU/ZauDDbSXrsZf9ZxZYi1kjhgAroLA1epBfyYgovgIzwyqjTe fcI0ekQFx3Q4bRkTpCDAl0r3FLxnfuY= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DBIg+hQM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.219.44 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713310192; a=rsa-sha256; cv=none; b=Z1MSy+EpatCTgypZkCwZAxx8dayJy4hS2D0FyGQ0fHxZMDIPB0imcf/GjppmXJGJoUDlYz THHb87UIEuH6A/lIn+ZzdJk+NgCZ163tSD4kc81A/WxDRNv3EujVtfWZYUV9zY402YjpnC E7SzI0er2rP2bv4UMJ86usDcmTrUtc4= Received: by mail-qv1-f44.google.com with SMTP id 6a1803df08f44-69b44071a07so38185836d6.3 for ; Tue, 16 Apr 2024 16:29:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713310191; x=1713914991; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=XLy1Ua1b77r4qsxuukriMKLfQ+NQtqpcoJqaT5UdrRA=; b=DBIg+hQMMBHRXiPubWtV28vOxm7U+/10DhlAUv3Bhj/IehLOW9Yt/Z7JgTW95ZJkBB yp+9BxwErBlTOLWY147oEDTGivzpAi9Rnw4P9IacCXWC3Pn+MHLSrliBlyvtvdVQDsjq 4mKjIj39wbTYIk6m/G2B2clN/pwn8o5WnHgd8RgO3H04sa4OFHogTCR+qO7Dr5Xm4ne8 oOe+ohGec7S7EB6Z+j1cbh7W7amDzcTP3vKB1fuhYSHTt1JJkIRMHEFZ1wSXIIYj+Tom 6fmSdY0sBLdJpJraHdCLfFge3YvjafcD5zMcZtCuKViIqzocAULj7OQBZ2u6+t+hH/mX 9QVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713310191; x=1713914991; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XLy1Ua1b77r4qsxuukriMKLfQ+NQtqpcoJqaT5UdrRA=; b=OqbUwF9PWY5+ZX4R36TKTEik80OPbM8HU9XHu0dWpeiwOh1VylKVgi/ixvTgUlL6U5 QwDiL5k/ElHUiRg8OSxocx1BHP1EDzNDvUJbR5yAeENWWvnV6ZjX4x8cgMVgL9YqHxGX 0aLiuag4hAW3mOZ5O+i5PthVqpjSsLQ/PjZRtYcU5Se5nvos12drm+baJxM47xzC+vpx 2FpoUlvgHaI0pqIbX4nEe9rzRYLZhkq9T0Zrf1LiZ118Yhpjn7nZREhBKWykDFaO6ACj ofk51Qkq48lQRk59ERiw/9B3Wh+fokaB3jIyaCxVJ4A+wf1xnnePAI+H5L8W1nphvny7 hM7w== X-Forwarded-Encrypted: i=1; AJvYcCVLFdKhwb2ThZjh5ZoqLHoDtEccbmaV0zGpB4HkojAwGDyxlLSbTk6b4+CTAKkMnrsqieOXuGipYYt0BnGPtay6eyk= X-Gm-Message-State: AOJu0Yxs1WV7VyrzMB/ptqpxL3S/Vdn0bMt00nDqd8Qq9GWi5Mq041YD nBRKnc9eRRysdFxZWNbAgyYT21tCTzBvytKknx8X3Uyo+fNanezOTEkuO8WXXAhMQfYTg75rWuK PXnE6c0k+G2IuwBaGWXRP7p1VC0Q= X-Google-Smtp-Source: AGHT+IF+EtI1xd1QjYng9Ds05N6w9StWVfIdI1Sei5Hyuup1KvXrmFZUzese74UopTTaLAlxrHD/vxCIeP/TAVL0fGI= X-Received: by 2002:a0c:ea33:0:b0:69b:7c1e:ac02 with SMTP id t19-20020a0cea33000000b0069b7c1eac02mr6282293qvp.18.1713310191203; Tue, 16 Apr 2024 16:29:51 -0700 (PDT) MIME-Version: 1.0 References: <3iccc6vjl5gminut3lvpl4va2lbnsgku5ei2d7ylftoofy3n2v@gcfdvtsq6dx2> In-Reply-To: From: Nhat Pham Date: Tue, 16 Apr 2024 16:29:39 -0700 Message-ID: Subject: Re: [REGRESSION] Null pointer dereference while shrinking zswap To: Christian Heusel Cc: Seth Jennings , Dan Streetman , Vitaly Wool , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, David Runge , "Richard W.M. Jones" , Mark W , regressions@lists.linux.dev, Johannes Weiner , Yosry Ahmed , Chengming Zhou Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 27901140011 X-Stat-Signature: mwy78fix758ymfn5xzq71srfu8bws9yn X-HE-Tag: 1713310191-790087 X-HE-Meta: U2FsdGVkX19GFOEw6NYgG7+maOTE8jfGINs8zWvVb5upz1jR6CDatDLDl6HOP2hNIb/QLI09WDiPj46rx2iWrXUYrQpXcF2HowEdFrtWyLpbrGgneb19OmCPZzc906zqCdZfJazgkGeLBi7GcDj3FWc5J6RKpAsrF+ZukuuigrsSq61JZFeOvR/cQmK3imzA7nDmQwQN9SLh22xwhB8l96Kh2Q8stLi2jtbv3Jyf8a+rzb1rpXgKI/lhApbfbCr6pfbnx0by/YGsT1b5/4uPnjCb7GFwNqEYACl5cKbWYrBqb4a0aqasy5IpWwvocwwmnZjXMLK7xiOLjZ49kmGuT8olGiht4JCV7FvXSf+YS9NTZvXzSLZ9WSYrlsE+4Q5zyTfUFgYRPhPYgQ2+aPOfXsvMuxXjjbYWnN9SxN3rQ1yVxiASkhRCUxg5YfwtdtEm68Kq+hbhP5bwKm7Sb++FBwSq7VTRAGMsxwJHUnwTl5DSEQ2znTYks9sV1xt+DU/RNKTqB71oylyj+KejLgW0fPogew50BpVVQKDdYB+UiUXI7t9IoxtcCpOybvFBxKICAgEsgQ6fIJlQTyi7LKPo+b+uPKokgKWEq9lOTNYw3TRXkotWelEd7AvZZNUlcriY4kGLlspnYGK2Bg6T90D/WbKFwjc5hHcnMolQK2GHS7+Uko7450AlFVHTPit7RtEms/mDHuTa9zGAKeaHXVEPg6Zevk5xuj6yn3TY/pswqODQydFbZX/e3J4YFCw+IUD+naNTnnN8Yv7nMnnnq95ByPY7yN3BDaLbEUOALDYpjpN5TWOM2d7f+yIzbio9Af/uGB5EeLpFtZ2J8czIl6eV9F/8cBrMyNuX5NkneGKx3h3QzTjWrImXeycLnS0lhM4/e6JxP/5duxGuWWgsFmsqAZorZys/ntcbV3jQtFGw+ZE6Zp83Nh6AnHqAozZhz7PYVxRzUw+2yzHUtpaLmcR 5hpNkwh9 UaP8cEG0w1EsECev8xmGVrZMG8NbMRNoP8NMG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Apr 16, 2024 at 3:14=E2=80=AFPM Nhat Pham wrote= : > > On Tue, Apr 16, 2024 at 5:19=E2=80=AFAM Christian Heusel wrote: > > > > Hello everyone, > > Thanks for the report, Christian! Looking at it now. > > > > > while rebuilding a few packages in Arch Linux we have recently come > > across a regression in the linux kernel which was made visible by a tes= t > > failure in libguestfs[0], where the booted kernel showed a Call Trace > > like the following one: > > > > [ 218.738568] CPU: 0 PID: 167 Comm: guestfsd Not tainted 6.7.0-rc4-1-m= ainline-00158-gb5ba474f3f51 #1 bf39861cf50acae7a79c534e25532f28afe4e593^M > > Is this one of the kernel versions that was broken? That looks a bit > odd, as zswap shrinker landed on 6.8... Ah ignore this - I understand the versioning now... > > > [ 218.739007] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS= Arch Linux 1.16.3-1-1 04/01/2014^M > > [ 218.739787] RIP: 0010:memcg_page_state+0x9/0x30^M > > [ 218.740299] Code: 0d b8 ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90= 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00= <48> 8b 87 00 06 00 00 48 63 f6 31 d2 48 8b 04 f0 48 85 c0 48 0f 48^M > > [ 218.740727] RSP: 0018:ffffb5fa808dfc10 EFLAGS: 00000202^M > > [ 218.740862] RAX: 0000000000000000 RBX: ffffb5fa808dfce0 RCX: 0000000= 000000002^M > > [ 218.741016] RDX: 0000000000000001 RSI: 0000000000000033 RDI: 0000000= 000000000^M > > [ 218.741168] RBP: 0000000000000000 R08: ffff976681ff8000 R09: 0000000= 000000000^M > > [ 218.741322] R10: 0000000000000001 R11: ffff9766833f9d00 R12: ffff976= 6ffffe780^M > > [ 218.742167] R13: 0000000000000000 R14: ffff976680cc1800 R15: ffff976= 682204d80^M > > [ 218.742376] FS: 00007f1479d9f540(0000) GS:ffff9766fbc00000(0000) kn= lGS:0000000000000000^M > > [ 218.742569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033^M > > [ 218.743256] CR2: 0000000000000600 CR3: 0000000103606000 CR4: 0000000= 000750ef0^M > > [ 218.743494] PKRU: 55555554^M > > [ 218.743593] Call Trace:^M > > [ 218.743733] ^M > > [ 218.743847] ? __die+0x23/0x70^M > > [ 218.743957] ? page_fault_oops+0x171/0x4e0^M > > [ 218.744056] ? free_unref_page+0xf6/0x180^M > > [ 218.744458] ? exc_page_fault+0x7f/0x180^M > > [ 218.744551] ? asm_exc_page_fault+0x26/0x30^M > > [ 218.744684] ? memcg_page_state+0x9/0x30^M > > [ 218.744779] zswap_shrinker_count+0x9d/0x110^M > > [ 218.744896] do_shrink_slab+0x3a/0x360^M > > [ 218.744990] shrink_slab+0xc7/0x3c0^M > > [ 218.745609] drop_slab+0x85/0x140^M > > [ 218.745691] drop_caches_sysctl_handler+0x7e/0xd0^M > > [ 218.745799] proc_sys_call_handler+0x1c0/0x2e0^M > > [ 218.745912] vfs_write+0x23d/0x400^M > > [ 218.745998] ksys_write+0x6f/0xf0^M > > [ 218.746080] do_syscall_64+0x64/0xe0^M > > [ 218.746169] ? exit_to_user_mode_prepare+0x132/0x1f0^M > > [ 218.746873] entry_SYSCALL_64_after_hwframe+0x6e/0x76^M > > > > The regression is present in the mainline kernel and also was > > independently reported to the redhat bugtracker[1]. > > > > I have bisected (see log[2]) the regression between v6.9-rc4 and v6.6 > > and have landed on the following results (removed unrelated test commit= ) > > as remainders since some of the commits were not buildable for me: > > - 7108cc3f765c ("mm: memcg: add per-memcg zswap writeback stat") > > - a65b0e7607cc ("zswap: make shrinking memcg-aware") > > - b5ba474f3f51 ("zswap: shrink zswap pool based on memory pressure") Hmmm, this smells like a memcg reference counting bug to me. The memcg was successfully accessed just a couple of LoCs above, but failed after the stats flushing. My suspicion right now is the new memcg refcnt dance to select an memcg for the global, capacity-based, shrinker, introduced in this patch series. That dance looks solid to me tho - not sure where it goes wrong. And, if it's a reference counting issue, that should pop up in different places (i.e other mysterious memcg traces)...