Re: phantom memory in a cgroup (was [BUG] ZSwap leaks memory upon being disabled)

[Konstantin Kharlamov <Hi-Angel@yandex.ru>]

On Sun, 2024-10-27 at 12:31 -0700, Yosry Ahmed wrote:
> On Sun, Oct 27, 2024 at 4:28 AM Konstantin Kharlamov
> <Hi-Angel@yandex.ru> wrote:
> > 
> > On Sun, 2024-10-27 at 13:32 +0300, Konstantin Kharlamov wrote:
> > > On Sun, 2024-10-27 at 13:11 +0300, Konstantin Kharlamov wrote:
> > > > On Sat, 2024-10-26 at 23:46 -0700, Yosry Ahmed wrote:
> > > > > I don't think this is an edge case. I think when we swapin a
> > > > > page
> > > > > we
> > > > > generally leave it in the swapcache if there is no pressure
> > > > > on
> > > > > swap
> > > > > space. In that case the memory is not really swapped out, but
> > > > > because
> > > > > it remains in the swapcache it is still reserving a swap
> > > > > slot, so
> > > > > it
> > > > > shows up as swap usage.
> > > > > 
> > > > > Konstantin, could you check the amount of swapcache you have,
> > > > > whether
> > > > > through /proc/vmstat or memory.stat on both user and system
> > > > > slices?
> > > > 
> > > > Sure
> > > > 
> > > >     λ grep cache /sys/fs/cgroup/*/memory.stat
> > > >     …
> > > >     /sys/fs/cgroup/system.slice/memory.stat:swapcached
> > > > 434917376
> > > >     /sys/fs/cgroup/user.slice/memory.stat:swapcached 15478784
> > > > 
> > > > `434917376` is a 0.4G, not much. In comparison,
> > > > `system.slice/memory.swap.current` is currently `4764139520 =
> > > > 4.4G`.
> > > 
> > > I figured since 434917376 is 10 numbers, I'd grep everything in
> > > memory.stat that has ten digits:
> > > 
> > >     λ grep -P "\d{10}$" /sys/fs/cgroup/system.slice/memory.stat
> > >     file 2671874048
> > >     shmem 2592768000
> > >     zswapped 2997760000
> > >     active_anon 1491247104
> > >     unevictable 1269555200
> > > 
> > > well, to me personally this isn't helpful, but perhaps am I
> > > missing
> > > something…
> > 
> > I found the process the "phantom memory" belongs to! I just
> > realized
> > that I can see `memory.swap.current` for individual processes in a
> > cgroup too, and it turns out currently 4.3G belong to sddm:
> > 
> >  
> > /sys/fs/cgroup/system.slice/sddm.service/memory.swap.current:472378
> > 1632
> > 
> > systemctl confirms this:
> > 
> >   λ systemctl status sddm
> >   ● sddm.service - Simple Desktop Display Manager
> >        Loaded: loaded (/usr/lib/systemd/system/sddm.service;
> > enabled; preset: disabled)
> >        Active: active (running) since Wed 2024-10-16 15:59:10 MSK;
> > 1 week 3 days ago
> >    Invocation: daadb3ed391b421b90b216122339be83
> >          Docs: man:sddm(1)
> >                man:sddm.conf(5)
> >      Main PID: 720 (sddm)
> >         Tasks: 10 (limit: 18621)
> >        Memory: 3.3G (peak: 4.1G swap: 4.3G swap peak: 5.8G zswap:
> > 67.6M)
> >           CPU: 21h 30min 56.309s
> >        CGroup: /system.slice/sddm.service
> >                ├─720 /usr/bin/sddm
> >                └─724 /usr/lib/Xorg -nolisten tcp -background none -
> > seat seat0 vt2 -auth /run/sddm/xauth_IKXVXT -noreset -displayfd 16
> > 
> > Note the `swap: 4.3G` sentence.
> > 
> > So, this is good news, but still doesn't answer the question where
> > did this memory
> > go. Out of the 2 processes in the group, `smem` shows 2.1M for sddm
> > and 88M for Xorg.
> > 
> > I even tried manually calculating:
> > 
> >   λ sudo grep Swap /proc/72{0,4}/smaps | awk '{total+=$2} END
> > {print "Swap memory: " total "K"}'
> >   Swap memory: 184656K
> > 
> > That's 180M, for some reason very different, but whatever, still
> > very far from 4.3G.

FTR, the reason I got "very different 180M" is I by mistake added up
SwapPSS as well.

> I think smaps will only show you swapped out mapped memory. It could
> be tmpfs.
> 
> One thing you can do is take a snapshot of memory.stat when
> memory.swap.current is at a high value (for sddm), then swapoff, then
> take another snapshot of memory.stat.
> 
> We should see an increase in either anon or shmem, which will tell us
> which type of memory was swapped out.

Okay. I will have to wait, because the session got killed by OOM. But I
think it's gonna reproduce in just a few days, my new workflow seems to
be triggering that a lot.

I took this chance to rename the thread as well, otherwise I'm gonna
forget it upon writing the next email.