From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE583D10C11 for ; Sun, 27 Oct 2024 03:14:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30AC96B008A; Sat, 26 Oct 2024 23:14:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BA956B008C; Sat, 26 Oct 2024 23:14:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15B136B0092; Sat, 26 Oct 2024 23:14:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EBBED6B008A for ; Sat, 26 Oct 2024 23:14:48 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0C2E9141AC7 for ; Sun, 27 Oct 2024 03:14:26 +0000 (UTC) X-FDA: 82717914678.11.2E9C5A7 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by imf16.hostedemail.com (Postfix) with ESMTP id 3D49D180005 for ; Sun, 27 Oct 2024 03:14:25 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=QDGQCMSM; spf=pass (imf16.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.219.46 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729998731; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yv89R+T0Nd6b55exm0Ax1BdIKWRCFhuzO4ARUxjR5i0=; b=jStSPg7ktXa4Zj21aBDuiMng8p4H/rgwG4NZdo8kFUz3JvsB9lBlzgguhLy9/PZ+Vrd0F8 siCRVJRksnVP0Km8Iltw6u5CqezL0j0mOrop8G2svDprKVqlJQNiPCgLmgEzsV5Lhb+0fd 59iRnrxvEOKUauVIXq4w6nxG8M4Eqkg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729998731; a=rsa-sha256; cv=none; b=zGetBCWw72jqe1vjzm2aPtT0KLyCCRcj9aFdHnCBFniaCVQGwnhcQZj4O4k4Put0E1vgi8 OQcsGl5P0IAblynAhA2o0lCZ9zGW5cMTNw2pU6TcWIsfLp/HLwfZwTcnYHpxDF3q4SIsGl LFYP06HLWgr2ARlN/eTmm7PQ8Ve+7Mc= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=QDGQCMSM; spf=pass (imf16.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.219.46 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-6cbd550b648so24602096d6.0 for ; Sat, 26 Oct 2024 20:14:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729998886; x=1730603686; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=yv89R+T0Nd6b55exm0Ax1BdIKWRCFhuzO4ARUxjR5i0=; b=QDGQCMSMyXozINn9Tj/3AGz1KA7st91QRlBKmtI4fTrdHgR1wGFbrB5SHIJKEEPz7q /e0hW9/mVEQiY92vtr71VXLdGBcUXf3UsbJxrNhRt3o7SA21ZuS5r9dWsDqi25fvq8sw 9T6Crs3+wBR8DQY0iWrPXjA+dN6XjfnU5j1VGPXoX6U8lx5dcC6BEGnbLZY0MuE8Zrb5 4+UVyyFQyNDSLL0hZiWRxsnvaQHXPQPxbA1jlMGMgfo7eiFx0dNJeFPIQRQhvKA6poWi scYB2n4QT59dJZM9TzFdIN3M2jSMcGZOOt8IdC4sV0yeQVVw7DXQgtQDSfmPILC7nrE6 PfcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729998886; x=1730603686; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yv89R+T0Nd6b55exm0Ax1BdIKWRCFhuzO4ARUxjR5i0=; b=JWxqGdhr8ZwLzk/93zSzjaA3n2llDsrw1jUrT+2/GwfEJSz18pJgLSoPkeyN/c0o+D JgrUgpES4QDPa4f8yf37ZrVtzL4WlqM44ZbENEE8dgX6OmMW5ZqWVl52nOmtPG/kAe7K DqnoSUBSV57tqTW9RmJHAZZMTo/1OY9T6c7gmPnMdfTwTIjsMNetahS/wWTpml0N3yY9 ilbGnuWnuZvRa5FdX9f8tR4zfMSMkZqSjd/CwvXtuAkBfdU1GyeYuXtDj14PGNxKWjbm gnhlaO3lzJywQ8mO4CtHhQIvvy3HHofFaKDybNVElqMQn0HD1DvtWrAVtiDwB8HTJOQD r3Og== X-Forwarded-Encrypted: i=1; AJvYcCV5d6VJCR/UMfsCSkJh2kPbE7o6TYqqM1RsY69sE3kkLe0L6UdNiCg98k9SaZETJ0VvLEDH20SpCQ==@kvack.org X-Gm-Message-State: AOJu0YwB3H03vbzj+Y3BJ6iGKzxz1/yt8yk8D5cjg6zE/pp7PmsSi6qB paq/RGbriYPmeP7DP35KdYGp5WH1UhX/m8+btTb72H1up/nOXl2SD9yTstPdp/wMoBhVZReYmM2 TkMvzetEr8lbgJmlR8AN2DZSbuHLJP1tO X-Google-Smtp-Source: AGHT+IHNhCZnpajk8BFTTEN1xqxgVoK2TpKxfAQSC3jjLENulxmqOs5oi0XBe4c0JkZrU5J2nZ2UEBSzmy2thWReG4Q= X-Received: by 2002:a05:6214:448e:b0:6cb:99db:bdd5 with SMTP id 6a1803df08f44-6d185853395mr82468746d6.39.1729998885796; Sat, 26 Oct 2024 20:14:45 -0700 (PDT) MIME-Version: 1.0 References: <7d873b2682a7513a4d5aad5ac09f5c78851efbb6.camel@yandex.ru> <28352fb75268060f2e78a32325f02e46737a16ea.camel@yandex.ru> In-Reply-To: From: Nhat Pham Date: Sat, 26 Oct 2024 20:14:34 -0700 Message-ID: Subject: Re: [BUG] ZSwap leaks memory upon being disabled To: Konstantin Kharlamov Cc: Yosry Ahmed , linux-mm@kvack.org, Johannes Weiner , Chengming Zhou Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 3D49D180005 X-Stat-Signature: hmtpnppk4bm5ir4hb71w3i691duofj8h X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1729998865-53728 X-HE-Meta: U2FsdGVkX1+GfvhfHSMF2d8U+DH9xm/ZEQJA3hF30JKZnbgVGFOw18kYz66oXo4p3VFAWa2DWjLTw/f5K10i+Nm02wFzmq1/uhZuA69gusFEXq2+CSp1FrccEkowCCl6CegkqMKFj3xvPLyJYREFZxDvfw4ZX+om5a6S/03A7zvGMwTIVnklY7+YaS038b9Ih5MSDu3o/+tAVSZKA+ZoQfS2/P5RP7ecxl32IzdEv/t1wgeNrMmXJO+k+m1jlfIxRxBPe/cY/Iqk5/4LdQ1FMlLdESf4/ETm/6NMaafp01QX1pLnHYwTSQubgtg4SEGDvEoMbVUMoiPs85Tv03Ety635BzJRgVV9iOtpztrN46oSlSLKttSUoey7NS75pAiIP4EEJemqP6sAXCcyRib6K6wHRu+pt2jncmWMMFAMfiOoi2CLplP+lwMAu1zCS5qIu2JW8+JNQftBywMMAvMFSRl6Htks7ToIDJCJOMvO81Ts2w+/4XSff0Sts5SoxpFTXSDA+qCv4Wj6LYMnaLywe+ZOjAW8fN77bWzJ4IEc/tXUAPysfyYext27n6g54ADrYJgY9gG0Zf9J6AJbdAVP7p4zk3ssguDAGoBy0GFXrSgGaxZcwyFzotkZDje4x+rr25tj9ccWZwHOlRrFdt6mXipYclAZqM+xWPH+f/zab1KIb5j/rYhq/xyX1UfrHDkEQ4UkTygaFQfOl9GVtX3PsQCKaEncxHAKCkzWXbx4esf6icSTL1jMsBjRXuoTpg2XJ8060amCaXBBOQ9v7h5gUwzjJGdtl3OnwlU8TIGJCtOzboFf5Eq8XJXy2JovkurVEgsELbqhoo1MYrgKcerkMHg+Tt6vtzm5Dd1sl4fAk3OSWB8dZ3WblnBmDVHsrwmMK5zHbAyrrpVpEWcmKJtXWbdK2Y9DPeDm/wbkGy/4KjEn/NCQMN6PK4WIcS4U80IdH+QfSt8loMjfoqx7mFZ TleTpLCq MVVZ5mTB0z3nqRAJVRwdf1US8KcjIFbkbQqGos9nntdFAAv96gHFCdz2d+iENvIwksuuVzva6PWma+p0geGSNL/JvrAQycXPxtvsHmCXQmWA8isS7XJb2i1DjT8A7qypxk6XKUX0LOfpI720qlTkTcWGaXNkuij8UGc5Pk5sVppQz9pIdT439DHJFGwejYYuU/BsBei+QFk5r0OSETZJf1mWMC+UbFb9kMznNbksY6q6fWMFUEYf/w/7EeE0za37fYwBcTkhge1wpbpTUgygBa/MItQcf3NRbrBAkorBL85LHFeh9I968klkyp9o9hy/82N/tgpNVtLWnrLLWt7ZI7Ct4Xg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000364, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, Oct 26, 2024 at 5:29=E2=80=AFPM Konstantin Kharlamov wrote: > > That was a good idea! The > `/sys/fs/cgroup/system.slice/memory.swap.current` seems to have the > missing half of the SWAP memory. From my understanding of the > `systemctl status` graph `sytem.slice` and `user.slice` groups do not > intersect, and by adding up `system.slice/=E2=80=A6` + `user.slice/=E2=80= =A6` I get > around 8G. > > However, I'm still unclear what does this memory belong to. > `system.slice/memory.swap.current` is 4.4G currently, that's a lot and > I'm not seeing anything that could take so much memory. I assume you do not have any proactive memory reclaimer? :) I believe the top utility can display swap usage by process. Have you tried that? There are a couple of edge cases - for instance, if you disable zswap writeback and zswap at the same time. We will allocate slots on swapfile, and store it at the page table entry, but we cannot store the page's content in zswap or the swapfile, so the page remains in memory. You're occupying swap space, but are not really saving any memory usage. IIRC, there is also an edge case where a page is faulted back into memory from swap, but the associated swap space cannot be immediately released. This should be temporary though - memory reclaimer will attempt to release these pages later on, or they can be released when we scan the swapfile for slots during swap out. > > An even larger related mystery is why does this memory not show up in > `smem` numbers for individual applications (which calculates it by > going over `/proc/$pid/smaps` for every pid). > > > The other possibility is that the pages are swapped out from the root > > cgroup, in which case they won't show up in memory.swap.current as > > they are basically unaccounted. Although typically user processes > > should not be running in the root cgroup. > > > > > "phantom swap memory" is hidden in `user.slice`, because if I wait > > > till > > > OOM-killer gets triggered and kills some app, my user-systemd gets > > > crashed for some reason, taking down the entire user session, and > > > afterwards SWAP is almost free. > > > > Did you check the OOM logs? It is possible that the OOM killer kills > > some system process that has some memory in swap as well. > > I did, logs are pretty uninteresting. OOM kills `electron` (of element- > desktop), but I tried closing it before the OOM, that didn't have much > influence. Just an arbitrary victim. Then a few lines later a `Process > 560296 (systemd) of user 1000 terminated abnormally with signal > 11/SEGV`. Wasn't able to get stacktrace for systemd with Archlinux's > debuginfo servers. And then everything gets down with systemd. > > I just tried closing every application I have open and I still got 5.5 > in SWAP. Well, obviously there are services still running, Plasma, > i3wm=E2=80=A6 Not many suspects left though. This beats me. I don't know the process situation in your laptop. Sorry :)