From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4657FD13589 for ; Sun, 27 Oct 2024 22:13:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E4426B0089; Sun, 27 Oct 2024 18:13:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7930A6B008A; Sun, 27 Oct 2024 18:13:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65B296B008C; Sun, 27 Oct 2024 18:13:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 46B926B0089 for ; Sun, 27 Oct 2024 18:13:49 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AB3BC160566 for ; Sun, 27 Oct 2024 22:13:23 +0000 (UTC) X-FDA: 82720784496.06.C92676F Received: from forward502b.mail.yandex.net (forward502b.mail.yandex.net [178.154.239.146]) by imf05.hostedemail.com (Postfix) with ESMTP id E6D7F10000D for ; Sun, 27 Oct 2024 22:13:03 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=yandex.ru header.s=mail header.b=V7QsgxDR; spf=pass (imf05.hostedemail.com: domain of Hi-Angel@yandex.ru designates 178.154.239.146 as permitted sender) smtp.mailfrom=Hi-Angel@yandex.ru; dmarc=pass (policy=none) header.from=yandex.ru ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730067069; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hlMz59wKP885yunV0EWYMzPX/xrnWaIjWb1q2PHc28U=; b=ON3Uhlj7+iOElIPLn3Tv7gYx+KEES/OFOzpilxrxOINUsm3epbo0nhyq4Lb//pfOubCxIY 3yCg+60bsTXihkcBU0hZ17gyKRDbLf7iXLzSixw0zRVWJe+FBRQrwpt9JSqBfnzwa0D5xl b6X2XP0P+CncFe4xpLaXQJDXpMrHNl8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730067069; a=rsa-sha256; cv=none; b=cU6G4VEOyeMQBEFhiewyuomE0WX7nDv0yOLrC9VCTk6vTgCnDfWq6/kvQSueaa3Icl3CKK m5R3dAbCVQ/XyTvkDtaXHfvrzRZodmnVMISY6xETPaAgeDAckMc5OlEoqlrffnAS3hv9AQ FSp1N3LFH4766Sfk6f2+EYR7hY8/Qww= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=yandex.ru header.s=mail header.b=V7QsgxDR; spf=pass (imf05.hostedemail.com: domain of Hi-Angel@yandex.ru designates 178.154.239.146 as permitted sender) smtp.mailfrom=Hi-Angel@yandex.ru; dmarc=pass (policy=none) header.from=yandex.ru Received: from mail-nwsmtp-smtp-production-main-46.sas.yp-c.yandex.net (mail-nwsmtp-smtp-production-main-46.sas.yp-c.yandex.net [IPv6:2a02:6b8:c07:109:0:640:efe1:0]) by forward502b.mail.yandex.net (Yandex) with ESMTPS id 1EEEB60DBE; Mon, 28 Oct 2024 01:13:44 +0300 (MSK) Received: by mail-nwsmtp-smtp-production-main-46.sas.yp-c.yandex.net (smtp/Yandex) with ESMTPSA id gDg0AtPJZCg0-aKfSpEW1; Mon, 28 Oct 2024 01:13:43 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1730067223; bh=hlMz59wKP885yunV0EWYMzPX/xrnWaIjWb1q2PHc28U=; h=References:Date:In-Reply-To:Cc:To:From:Subject:Message-ID; b=V7QsgxDRjYvYHd5qcia04hD4PpRL2mQr9ei/mo8pA+bxSKkXblZy1KgwqEvr0xNRh gxQ00DCQ0lNIEx+zAUWeU4VslMPZ9D/3CfH3qds7aWiTJMp2MRizoyFjS+JuFn+774 0tWinPZYV8TlcgAtlR8Qu/M86A07m6bz+INTGxGc= Message-ID: Subject: Re: phantom memory in a cgroup (was [BUG] ZSwap leaks memory upon being disabled) From: Konstantin Kharlamov To: Yosry Ahmed Cc: Nhat Pham , linux-mm@kvack.org, Johannes Weiner , Chengming Zhou Date: Mon, 28 Oct 2024 01:13:42 +0300 In-Reply-To: References: <7d873b2682a7513a4d5aad5ac09f5c78851efbb6.camel@yandex.ru> <28352fb75268060f2e78a32325f02e46737a16ea.camel@yandex.ru> <656f9dad3edba22b3b8ae16b97050d3f189ff1dc.camel@yandex.ru> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.54.0 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E6D7F10000D X-Stat-Signature: rijjyhbygup8nu9ofyjjyaej1g9c6dmr X-HE-Tag: 1730067183-54885 X-HE-Meta: U2FsdGVkX18cPAov4ZfHA6Qi6uoFmnV3Qe7u2OwA3tZokAMZGTO46U6z4gN5z6g51JVIDxdSIGo9PR2yRRAyTz5gfRRe/u+TKuJ73AEgQ+shKK0bQFEr2aKARriL2C7OHNqsXJSYXnUT92gFpUTGccY7S3pttGw7BK+p4FgEbt13oxymLWk7cANNM49Pu0Hl8OAeL3+sNTltKH05SRhnydJTCuQ9wZFFaUA2sgliKeeQLtv40MZjJhS/WW8Us/+P/6SXTXXPZX2oXOGPOoP+TMS87sjxW9z83Nm/OKTW/ziLfK6AnREuYUfpVHk4Mb/a/rultewBfzTl16JSrDgdRJHyQn4uDeg7l++wiH/tAfbVE2cTGsN2hKSl90NvyQ7fZyH8Boje/g+xNwOCF4q2hs7EHmHY2TViP7J2cGDd3aSluamqBSFSJPfCvil2jKmMgFSGni/9acEyw3zpPIXD1pl6G+7MehJcM6Bnh2zCasOaMCylIoUjs+bDXaPcrCU1qEpilEvMutF3Ra4y6W3nM9F97eDaObCp4J6ua83F+KddprFSZA+HFoDqJ9v7NB+Wf6CSlyyCFdpnGPgE1upbjGMBf3iGAko3oA0XDavBu/RZb6XNgKK3NBKokhHacspiCct7KyrIeu6fVh5YvJEScsJ/ndDrZUVE8uk7upP81gqXeYERwodEwtNfYPiM9Rq0hy9NURVhHInzy87kddjMqbEnSHD16R1HCICcw/s87pAj4L3mtWUf/0MY7YXB846lIB2CUcExv70WN6DnvwZzd+6tq6JRz9wSO1d+20TbTZNKVlEwNUPhMX6ypIN2a0qQVwfmVFMmMN7iZ3PfhwILz0E6T/OX6cDI9o0oq0hCLGhh8KlRJuve/hD1Rc9xdibjLP55FgpTI6Al++hxfnvylOItGIWV1q6tx12ANIw6mT0gt2inJbKv6w/mTf3/cl+JUkmhLarQ1z4DINkXQhJ nBEFOR1j 2V7DoIvNyojUPxFfiQI9jKeUhQRPap2kb81Y5oJULS4lle2UoKPKmzfuYwMYphk8iaNqFKieBCcDv0Khdm+SlCcBSvYMSrc2Vq8V66+Dh3VPC/J+xXAkEqnQQqOzKKBJuQ8L0qARhA54D9rodKrd4lPph3waWVDNPs14ywPnm/EBgkHZOK5Em1Qd7BpBp1QP2qE8jjBdYF0MFGxfjLlDYD+2OTzTynIZNOy+pJJDrGv6SC6bVlqidFKpA7lws/iLsgUcTt4hJVUtTHA8Q0d3lM3C+3OoH7dJ4pIgWDXho8YmMEsO2f4TSE+r020L/fUtMKdjk95z+Npcg1ARYQCO240AuZXlekqFQ9gy5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sun, 2024-10-27 at 12:31 -0700, Yosry Ahmed wrote: > On Sun, Oct 27, 2024 at 4:28=E2=80=AFAM Konstantin Kharlamov > wrote: > >=20 > > On Sun, 2024-10-27 at 13:32 +0300, Konstantin Kharlamov wrote: > > > On Sun, 2024-10-27 at 13:11 +0300, Konstantin Kharlamov wrote: > > > > On Sat, 2024-10-26 at 23:46 -0700, Yosry Ahmed wrote: > > > > > I don't think this is an edge case. I think when we swapin a > > > > > page > > > > > we > > > > > generally leave it in the swapcache if there is no pressure > > > > > on > > > > > swap > > > > > space. In that case the memory is not really swapped out, but > > > > > because > > > > > it remains in the swapcache it is still reserving a swap > > > > > slot, so > > > > > it > > > > > shows up as swap usage. > > > > >=20 > > > > > Konstantin, could you check the amount of swapcache you have, > > > > > whether > > > > > through /proc/vmstat or memory.stat on both user and system > > > > > slices? > > > >=20 > > > > Sure > > > >=20 > > > > =C2=A0=C2=A0=C2=A0 =CE=BB grep cache /sys/fs/cgroup/*/memory.stat > > > > =C2=A0=C2=A0=C2=A0 =E2=80=A6 > > > > =C2=A0=C2=A0=C2=A0 /sys/fs/cgroup/system.slice/memory.stat:swapcach= ed > > > > 434917376 > > > > =C2=A0=C2=A0=C2=A0 /sys/fs/cgroup/user.slice/memory.stat:swapcached= 15478784 > > > >=20 > > > > `434917376` is a 0.4G, not much. In comparison, > > > > `system.slice/memory.swap.current` is currently `4764139520 =3D > > > > 4.4G`. > > >=20 > > > I figured since 434917376 is 10 numbers, I'd grep everything in > > > memory.stat that has ten digits: > > >=20 > > > =C2=A0=C2=A0=C2=A0 =CE=BB grep -P "\d{10}$" /sys/fs/cgroup/system.sli= ce/memory.stat > > > =C2=A0=C2=A0=C2=A0 file 2671874048 > > > =C2=A0=C2=A0=C2=A0 shmem 2592768000 > > > =C2=A0=C2=A0=C2=A0 zswapped 2997760000 > > > =C2=A0=C2=A0=C2=A0 active_anon 1491247104 > > > =C2=A0=C2=A0=C2=A0 unevictable 1269555200 > > >=20 > > > well, to me personally this isn't helpful, but perhaps am I > > > missing > > > something=E2=80=A6 > >=20 > > I found the process the "phantom memory" belongs to! I just > > realized > > that I can see `memory.swap.current` for individual processes in a > > cgroup too, and it turns out currently 4.3G belong to sddm: > >=20 > > =C2=A0 > > /sys/fs/cgroup/system.slice/sddm.service/memory.swap.current:472378 > > 1632 > >=20 > > systemctl confirms this: > >=20 > > =C2=A0 =CE=BB systemctl status sddm > > =C2=A0 =E2=97=8F sddm.service - Simple Desktop Display Manager > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Loaded: loaded (/usr/lib/systemd/s= ystem/sddm.service; > > enabled; preset: disabled) > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Active: active (running) since Wed= 2024-10-16 15:59:10 MSK; > > 1 week 3 days ago > > =C2=A0=C2=A0 Invocation: daadb3ed391b421b90b216122339be83 > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Docs: man:sddm(1) > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 man:sddm.conf(5) > > =C2=A0=C2=A0=C2=A0=C2=A0 Main PID: 720 (sddm) > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Tasks: 10 (limit: 18621) > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Memory: 3.3G (peak: 4.1G swap: 4.3= G swap peak: 5.8G zswap: > > 67.6M) > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU: 21h 30min 5= 6.309s > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CGroup: /system.slice/sddm.service > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 =E2=94=9C=E2=94=80720 /usr/bin/sddm > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 =E2=94=94=E2=94=80724 /usr/lib/Xorg -nolisten tcp -backgrou= nd none - > > seat seat0 vt2 -auth /run/sddm/xauth_IKXVXT -noreset -displayfd 16 > >=20 > > Note the `swap: 4.3G` sentence. > >=20 > > So, this is good news, but still doesn't answer the question where > > did this memory > > go. Out of the 2 processes in the group, `smem` shows 2.1M for sddm > > and 88M for Xorg. > >=20 > > I even tried manually calculating: > >=20 > > =C2=A0 =CE=BB sudo grep Swap /proc/72{0,4}/smaps | awk '{total+=3D$2} E= ND > > {print "Swap memory: " total "K"}' > > =C2=A0 Swap memory: 184656K > >=20 > > That's 180M, for some reason very different, but whatever, still > > very far from 4.3G. FTR, the reason I got "very different 180M" is I by mistake added up SwapPSS as well. > I think smaps will only show you swapped out mapped memory. It could > be tmpfs. >=20 > One thing you can do is take a snapshot of memory.stat when > memory.swap.current is at a high value (for sddm), then swapoff, then > take another snapshot of memory.stat. >=20 > We should see an increase in either anon or shmem, which will tell us > which type of memory was swapped out. Okay. I will have to wait, because the session got killed by OOM. But I think it's gonna reproduce in just a few days, my new workflow seems to be triggering that a lot. I took this chance to rename the thread as well, otherwise I'm gonna forget it upon writing the next email.