From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A897D10BF7 for ; Sat, 26 Oct 2024 11:33:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4362A6B007B; Sat, 26 Oct 2024 07:33:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E4B46B0082; Sat, 26 Oct 2024 07:33:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D3696B0083; Sat, 26 Oct 2024 07:33:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0DE406B007B for ; Sat, 26 Oct 2024 07:33:12 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 965C7C08AF for ; Sat, 26 Oct 2024 11:32:49 +0000 (UTC) X-FDA: 82715541552.10.B5A23F7 Received: from forward502d.mail.yandex.net (forward502d.mail.yandex.net [178.154.239.210]) by imf15.hostedemail.com (Postfix) with ESMTP id 5BE1EA002E for ; Sat, 26 Oct 2024 11:32:48 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=yandex.ru header.s=mail header.b=UEiXupyO; dmarc=pass (policy=none) header.from=yandex.ru; spf=pass (imf15.hostedemail.com: domain of Hi-Angel@yandex.ru designates 178.154.239.210 as permitted sender) smtp.mailfrom=Hi-Angel@yandex.ru ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729942218; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mqbQ+Vj+cN+9OkBGsm6aX9gkN4IKN25kisw5S47C7EI=; b=bLZrLoesPwLNlU+dWP47cCQaYBqTH7oFCYnCPVpICagZkdb9mw78Q/NU1W84qrJGCeWIrF 1gmpP+C6HC6XTYsArMF9L6PBlKcxJ/UGZh3M3e5259m1Gj75O5jFfoLbifzINzZcniA6PZ mahvrEedqg+kRP81nMGYz+i3xeoO8QA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729942218; a=rsa-sha256; cv=none; b=wDPswrGpCZPq4P0N3vPR9YP8todb5OSCJARHBudjJeEpDgKvj8/TosQ2DFUJrHAVV7HwEp UhMftTeGpi7aqQuKLHtIMYLhtE5wYxrV6gk9yClRE349BgnwwNyST+D9kX43fA+rbaG9dP ngLFluDJRuOaQMP87gcrLz2b7ZSsKJQ= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=yandex.ru header.s=mail header.b=UEiXupyO; dmarc=pass (policy=none) header.from=yandex.ru; spf=pass (imf15.hostedemail.com: domain of Hi-Angel@yandex.ru designates 178.154.239.210 as permitted sender) smtp.mailfrom=Hi-Angel@yandex.ru Received: from mail-nwsmtp-smtp-production-main-72.klg.yp-c.yandex.net (mail-nwsmtp-smtp-production-main-72.klg.yp-c.yandex.net [IPv6:2a02:6b8:c42:2cca:0:640:9416:0]) by forward502d.mail.yandex.net (Yandex) with ESMTPS id 58F9661025; Sat, 26 Oct 2024 14:33:07 +0300 (MSK) Received: by mail-nwsmtp-smtp-production-main-72.klg.yp-c.yandex.net (smtp/Yandex) with ESMTPSA id 5XUU5w6LgOs0-iq4GWTfy; Sat, 26 Oct 2024 14:33:06 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1729942386; bh=mqbQ+Vj+cN+9OkBGsm6aX9gkN4IKN25kisw5S47C7EI=; h=References:Date:In-Reply-To:Cc:To:From:Subject:Message-ID; b=UEiXupyOEiv+eyNdfhUOLT1/IIsgCOF8SkFC7YAWADjuk5OXnqIhpAOYGZO6vDm8w 2fRY2/6pHZ8Ea/SXlErgyzQvybBYdeeo1Tb//jjf/w72nW8mwXAUj55JsuMDa4vAmV UfItm6ALX+QrafGBLh8F7V5W3veVs+cuJkSdPa2E= Message-ID: Subject: Re: [BUG] ZSwap leaks memory upon being disabled From: Konstantin Kharlamov To: Yosry Ahmed Cc: linux-mm@kvack.org, Johannes Weiner , Nhat Pham , Chengming Zhou Date: Sat, 26 Oct 2024 14:33:05 +0300 In-Reply-To: References: <7d873b2682a7513a4d5aad5ac09f5c78851efbb6.camel@yandex.ru> <28352fb75268060f2e78a32325f02e46737a16ea.camel@yandex.ru> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.54.0 MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5BE1EA002E X-Stat-Signature: 4w4dmnqq4npy6utxjbiuebtmxp1k4jmp X-Rspam-User: X-HE-Tag: 1729942368-866757 X-HE-Meta: U2FsdGVkX18+vq5XBBqnMsY6v40I5wQwOIptbZQeITfUFiGHitZeHMl6lp6470k41Dmv9LvMsELoWag93lj76DAC1/CthcH87dnEusP9o58fy8EAvseCF7TKC1xKRNqZ0vMRyMSod2oL21aif1j8EYrqAM9YLkb1BQ3w7/mwO3iK1S4dO+EO2MZuJmG1NRjRr705KLHoLuZV/87t9p2umJPJxqz0EwDRdN3aD1GXdtODCa5OPcIekPy1ofOS0CNLPfQC9yT/sWmQMHyoD8e09DQd+6NWNoEwT/01zSETLrJZnCM4U5YFVSAZAMZJv4/236tf4OmGr9E511uTm0PjDrZzTbV7vUIeCjeQOgbF8datSV+P3hmszKoUAqmWs5m3/9hBbgc+TayOiUxfSfH8lJyOL0Iw9msqf9ZEvbYklggGnx7zV42MukeqSGlTElnLmyVkAmObY9+hAmXFXT51oMCNZJrLfng3TKKL0H6frF2lTFj+xfpzAov99oMQjUj/6QSkeyrbNRD1jKDhXwHORMADwlSw4t1oB9S25MYu21g2YKLdlEpg8YiJ8vZR9m/P3eglbhLzfY7mjf8LE5W4kjcw+U8/gYA4GvQ/+LA8Eihjdds8770CvdWrjqNT1oSqD1wGDVuLFxry1KkJ3QqxSad771jcrwue3WeVe5gB9ZHtCaERRfXuA3byn6ztWpspZgwcDn/iFUQq4n3rSolkl2Q487m3Y4QSAFd/RS+HtGdeqq6y0kEhTkeBhB+gzz2l+fvNdnH4u9GxCzqtmV0S4yNyRkGNrceqJgZsZ72/qZ3R/g9angLcEIJ4aEKyTpetcB5Z2BeTA91vVl5Gmngn788eVADLOlDfg11fS4hXamCSj5lT/0TB2xwDq4P+GafDBC9AaLDLijwQA31tLo5Qt1w3bGYvGbPjdypWiGzhFryhq4etx176ccQ8yX5vzJmllqAySAgkwmizNxi0Wlg BcM2b9Mg /FbXPI3qUF2sRxnmLLnManYHd17nDix2oYRbdjC28QBzM3lQtjyLL4Ii1GvlnWrddwGda7cZ5bSSHrTKv8PxnFXy0+LAc/tCCQsCinXqmYTecuYMu0LpTj7a9/gUC6Hw+KzZdS8hrewxJgO1/mDnokFfWa7GbbfydSaeJiM4uwI+2njpRGJzdKqG1ZPQD0zHPV6JvvrOq5oQ0lrjLQA1f7xiraR+55oOQb8BSTwdEtTTI9DzVk/TqmjOBBhayuRA0xOKB8jK5zcMam4I9ImLrNqFWVxazbeIgCXmylM+NvEj22JycLD84c2KSrfJN1O+q1dcGSZvZq+uwIOtJCsmkpQMnTNh5lYY0HaOrFBcrLrjnbwwgunvF/mJzJ6k1WG6dfoxPL6NdJE24Uo4kITYsubq+KU/CY0Sn63TgncUl0c44rQg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, 2024-10-25 at 00:50 -0700, Yosry Ahmed wrote: > On Thu, Oct 24, 2024 at 11:41=E2=80=AFPM Konstantin Kharlamov > wrote: > >=20 > > On Thu, 2024-10-24 at 13:47 -0700, Yosry Ahmed wrote: > > > On Thu, Oct 24, 2024 at 6:02=E2=80=AFAM Konstantin Kharlamov > > > wrote: > > > >=20 > > > > When ZSWAP is disabled, the `Zswap` and `Zswapped` in meminfo > > > > are > > > > still non-zero. > > > > IOW, ZSWAP doesn't free memory upon being disabled. > > > >=20 > > > > Stumbled upon this while trying to figure out where did =E2=89=884G= of > > > > my > > > > SWAP memory > > > > disappear. Been seeing some unknown memory in SWAP for years, > > > > now I > > > > suspect ZSWAP > > > > might be the culprit. But no way to know for sure because of > > > > this > > > > bug. > > > >=20 > > > > # Steps to reproduce > > > >=20 > > > > 1. Enable ZSWAP > > > > 2. Wait for `grep Zswap /proc/meminfo` to become non-zero > > > > 3. Disable ZSWAP via `sudo sh -c "echo 0 > > > > > /sys/module/zswap/parameters/enabled"` > > > > 4. Look at `grep Zswap /proc/meminfo` > > > >=20 > > > > ## Expected > > > >=20 > > > > The rows are zero because ZSWAP is disabled. > > >=20 > > > Not really, the expected behavior is that further swapouts will > > > not > > > go > > > to zswap, but pages that are already compressed in zswap will not > > > be > > > written out to the backing swapfile or swapped back to memory. A > > > swapoff would be required for the latter. > > >=20 > > > This is documented in: > > > https://docs.kernel.org/admin-guide/mm/zswap.html#overview. > >=20 > > Oh, I see, thank you, sorry for the noise. > >=20 > > Then, I'm curious, is it correct to assume that this `Zswap`- > > prefixed > > memory mentioned in meminfo is never the one that is in SWAP? I > > mean, > > Zswap being a buffer before data goes to swap kind of implies that > > yes, > > the data *either* in zswap or in swap. But just wanted to hear that > > explicitly. >=20 > I know this makes sense, but unfortunately no. Zswap is currently > transparent to the rest of the system. For all intents and purposes, > pages in zswap are considered in swap. You cannot even use zswap with > an actual swapfile. So the zswap stats should be a subset of the swap > stats. >=20 > FWIW, Nhat is working on restructuring this to have zswap be its own > entity, separate from any swapfiles. >=20 > >=20 > > The background to my question is that I'm trying to find the > > culprit > > some "phantom memory" eventually filling up my SWAP. This memory is > > not > > one accounted to apps (as calculated via `smem`), nor to tmpfs. So > > my > > next suspect was something related to ZSwap. > > >=20 >=20 > As I mentioned, zswap should be transparent to the rest of the > system, > so it shouldn't make a difference in this case whether the pages are > in zswap or in the swapfile. >=20 > You can use the memory.swap.current counter to find out which memory > cgroup currently has swapped out pages (in zswap or in the swapfile). > This should help find the application that has memory in swap. If you > want to find the exact type of memory (e.g. anon vs tmpfs), that > would > be more tricky. Perhaps you can swapoff and see what counters > increase > in memory.stat of the relevant memory cgroup? Thank you, so, I've waited till my SWAP gets almost full again (apparently my new workflow triggers that a lot). It is 7.5G out of 8 in total. 437M is taken by tmpfs'es, let's subtract for simplicity, so I have 7G taken by something else. Now I'm looking at `/sys/fs/cgroup/user.slice/memory.swap.current` and it's 4422422528 =3D 4.1G. That's a lot less than 7G. I'm certain this "phantom swap memory" is hidden in `user.slice`, because if I wait till OOM-killer gets triggered and kills some app, my user-systemd gets crashed for some reason, taking down the entire user session, and afterwards SWAP is almost free. I think this memory.swap.current isn't much different compared to just asking `smem` for SWAP taken by individual apps. As of writing the words that's 4.6G for the entire system, as calculated by: sudo smem -c "name user pid vss pss rss swap" | awk '{total+=3D$7} END {print "Swap memory: " total "K"}' So 7 - 4.6 =3D 2.4G of some "phantom" memory.