From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D198D6B6C4 for ; Wed, 30 Oct 2024 19:45:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0E838D0002; Wed, 30 Oct 2024 15:45:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9BE178D0001; Wed, 30 Oct 2024 15:45:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 85EAF8D0002; Wed, 30 Oct 2024 15:45:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 67E158D0001 for ; Wed, 30 Oct 2024 15:45:23 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EAABAC0F66 for ; Wed, 30 Oct 2024 19:45:22 +0000 (UTC) X-FDA: 82731296970.12.2EAC1A1 Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) by imf25.hostedemail.com (Postfix) with ESMTP id 53125A0024 for ; Wed, 30 Oct 2024 19:45:02 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=UUygFRQq; spf=pass (imf25.hostedemail.com: domain of yosryahmed@google.com designates 209.85.219.49 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730317362; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=orRTKDkqlxEuFvnDEpRROSbi4+sJldR0A+ySRqO6gpg=; b=BBxwpX6K87f1CNIdbGk23Wp7ga158HGohHxVYJy9/5PuZD1MddppQDsKX7R0gdUcb6NnHD SQPMBFJboeQz3lP8eKL1h6tGj+dcX+FBf6tPst9YmyEtw6KNPYtCzKh6SdBEHOFmpsKbLy oTS2xHBOFPBb+zMBUpsL8/9MjDMxmss= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730317362; a=rsa-sha256; cv=none; b=E1A4+CeSLVk2qTllYnbtpETdUkA/UJ9liFmvS0ekHnBx+zCurkELb0Yq3VDEmFMV8UgYPK QO/lsXvPzlt1pJwWVEQR04XjFa1aR0ivdKAaEwcrK1LAMVwrc3z0/jSO8CrvhTKMzPXNOW mOGpHk/AYfDDwriKCugJrRmpttKGTAM= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=UUygFRQq; spf=pass (imf25.hostedemail.com: domain of yosryahmed@google.com designates 209.85.219.49 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-qv1-f49.google.com with SMTP id 6a1803df08f44-6cbd00dd21cso1322586d6.3 for ; Wed, 30 Oct 2024 12:45:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1730317520; x=1730922320; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=orRTKDkqlxEuFvnDEpRROSbi4+sJldR0A+ySRqO6gpg=; b=UUygFRQq75eHgtCZy4OmxxuagBazeQIcFQBNg7OH/OwjzOKOf3jhJomJdUtQRMQ8VF qhnnyJvdfsq/magLePZfxfsrFp1MuJR70jVCB6VtTk4c3ho+b4/OfQlIt2iohK35duHM OC/Uz7UO1nnVw+0xdwNTeOPvCyW+pwoT+jAO0zlCHDW0TAcoZ+fTcFitNFqw+pVNzXKg qbUrjRaZai+v7jIsGfNTay8LV2wTQFF/Sy3ImR6BKEEucG/OFCzA6KuRl1T5WbjSlJMA EWjXzseuXQftif6wGm8WdSLdPi38Bdo3d1saywEFGkFz6RpnSavv2YHxsLeK8yP9aeeA nK8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730317520; x=1730922320; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=orRTKDkqlxEuFvnDEpRROSbi4+sJldR0A+ySRqO6gpg=; b=XsIDsSmLnOCOP5Ln8BskGsiZoGopNOt7K4V9oLf2Rf4y0havJ4G4p9PxV13HNSwOxc bV25w1pNRQfKoohYTm7l1+gySeUAXnY7e/h/6mQhgRic+Lu0AaQ64xj7cDUoDw0DtL1V DcccBoTxlfgTOEDpUXzVzHvRuD6WG5c9vs/q+62JIOWxkMudo80ZBYwLM2lOkPV6Mu67 peEOoGicZkvS/Mc6ivutFyozfkuev+EZbOH9d/iqXnun1nVlUXw9Rq4a4Jy2SJfjVOCD 3Qe5ceJxlSs1yS6iIdIotc/GAAv65SKOvDZk8Z8J6Z53FOCejJ++rK2qpLlX/emtTqAk tj+Q== X-Forwarded-Encrypted: i=1; AJvYcCXXpwUraHUGx5fQfZJWp6TKTi7t8MQ8oo28gXYCB9JAivXkRYVvQMe9scKJi7O8JeacQ9C/0diLYA==@kvack.org X-Gm-Message-State: AOJu0Yw4bBtgV1G46FlNtASbdPeF38sfon3UEUatQWV6RtuXIpOhxEC2 4FUxBMLK8Ro7e8f8fdt6AUHwM+8o5CuFgrrtYQSmUN7LTlvlSh9y3SbVphI7ukNMPi2F1WdQIyT 0wH6HHgMTAGr4Kwun09uTGkKufSJNwilRGgfT X-Google-Smtp-Source: AGHT+IHk8j/Q1NlPeDl9nS9pSvnyrf3g6M4rjDUGJoZoPpW7vB0Hfhn+jRj1xzhQVh8C+nZkgrAoalqy9lzBPJ3Mnw8= X-Received: by 2002:a05:6214:4343:b0:6cb:6187:7c93 with SMTP id 6a1803df08f44-6d345fae071mr52873246d6.8.1730317519907; Wed, 30 Oct 2024 12:45:19 -0700 (PDT) MIME-Version: 1.0 References: <7d873b2682a7513a4d5aad5ac09f5c78851efbb6.camel@yandex.ru> <28352fb75268060f2e78a32325f02e46737a16ea.camel@yandex.ru> <656f9dad3edba22b3b8ae16b97050d3f189ff1dc.camel@yandex.ru> <861b38be2afcc1678cc2178a20f85b8fa1842bcc.camel@yandex.ru> In-Reply-To: <861b38be2afcc1678cc2178a20f85b8fa1842bcc.camel@yandex.ru> From: Yosry Ahmed Date: Wed, 30 Oct 2024 12:44:43 -0700 Message-ID: Subject: Re: phantom memory in a cgroup (was [BUG] ZSwap leaks memory upon being disabled) To: Konstantin Kharlamov Cc: Nhat Pham , linux-mm@kvack.org, Johannes Weiner , Chengming Zhou Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 53125A0024 X-Stat-Signature: 9xcfniszii9c7hxcn1xq14d71dhwfcdc X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1730317502-355930 X-HE-Meta: U2FsdGVkX1/0zgwtGDbQDLD0k/2MhBw13f/u38o3ngjcd3jdqk1Vnzs+HUbupe8KCD+16HtM8BTUtDxlRMDxhN51qMgJoY2cUh3BccG3/T4t4FJPAUcfOIRASmSXulR3lraX9eLJtcgFYC4egtxoNFslmE9FPfmkVwo2cySWqF+CuQWjKMK2PcL1+b10qh9mMHq2K/9tLhnuXLTAekr/9cuHOJt3v9txPqL7azhpIqaP/MF5/7Po1dHBd7r7P1uniMmh1+nbjsZ1Lj+etIU1Sc6CgQJPtkbPbtIYWKxTC6bOIZEnccVdF3Ldw4lpcb/dJFlI3yh8IBLl4vyX/lJVglvqpdxDHnrh+0jk/IaCKeGAjBcOJdXDGeMUcfu2pj0VPlO75QDLtq/99dKjZtR3oAKeq6REj0rhz7Dp54syAcbkhPf8kv0CV0v0z13Skx/TaRp+SD835uQdTHgc7JpVA0oCrLP4NdPkVJtJaJLfeBFlzzWH8kvS8G3TEwDIVUYy4yZEVHVyEYUfNHH7yTmIaYMK831nRP4zOBKrJ16CO2tdigKMHu5BAuojTOEFTYUjRB8MiLPaQ1VGAwlhpTWt+/KY4sszeF55emZPoUCo5wIWV95FYDuoJLKrAIxmqsgpjMU/ueAB4XR218g5T1I2bsb30aCw19NQ2Fy6fZo5Gpj6m3PFamji98bzjWE/8JB3agYH9syY+N2jquVYcGXm72UJnGa1C4yHQtriH5ovHbKr+iRg2BSIVK9wLJblhpW7TdBp2yUYF/ko2+s+90ovzAD0faTFIzDOjYsbXV/cW4oyEdvpufOAct7jUL8G4M61S7CPNtz3FHtR4PZAg3IRLP1r/nFArUWw4mQ06s+4oI6h24bm8dusgjKHcqpZyM0Wl1NwbtIdPsMK9FGQlu3pE+4p2gLc4rUA1Wy+h+o8ObLpdGrPvVfRPJsGc5nnaUifuqclwjR3lTw6eMPsdIy CaAHJhx/ VkeOcS47+GdsvX6ZzkdVhBiXCL5MOMSq9Qm0qvQJpPsXCDIRr6BdXXTbOVNvPbsB7mwWpQVEqoD/wEFR0nxfIuJOojgQ0jhzQk50/S/bmNOYiorovquiIjjse8HVj6iwdNvdwGo2e4D7k33QGWGWPBPVNGvJKSfnImHyfhSQjDq+r5kMHxQtwwgfLKMGfyvZWj50FrbYspkhugj5RYC8DGDv/2w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Oct 30, 2024 at 7:41=E2=80=AFAM Konstantin Kharlamov wrote: > > On Mon, 2024-10-28 at 01:13 +0300, Konstantin Kharlamov wrote: > > On Sun, 2024-10-27 at 12:31 -0700, Yosry Ahmed wrote: > > > One thing you can do is take a snapshot of memory.stat when > > > memory.swap.current is at a high value (for sddm), then swapoff, > > > then > > > take another snapshot of memory.stat. > > > > > > We should see an increase in either anon or shmem, which will tell > > > us > > > which type of memory was swapped out. > > > > Okay. I will have to wait, because the session got killed by OOM. But > > I > > think it's gonna reproduce in just a few days, my new workflow seems > > to > > be triggering that a lot. > > Done. I missed one cycle, which again got my session killed by OOM =F0=9F= =98=85 > Now I caught this in time. The information was retrieved by: > > (systemctl status sddm && cat /sys/fs/cgroup/system.slice/sddm.servic= e/memory.stat) > ~/Projects/cgroups-mem-leak/"$(date -R)".log > > I wasn't sure how to represent it in email, and decided to post a diff > of "before `swapoff -a`" and "after =E2=80=A6", to be viewed with `diffr`= or > with `perl /path/to/diff-highlight` of git or similar. > > Diff follows: > > --- "Wed, 30 Oct 2024 17:27:38 +0300.log" 2024-10-30 17:27:38.40129= 0017 +0300 > +++ "Wed, 30 Oct 2024 17:28:12 +0300.log" 2024-10-30 17:28:12.39769= 5798 +0300 > @@ -6,8 +6,8 @@ > man:sddm.conf(5) > Main PID: 710 (sddm) > Tasks: 9 (limit: 18621) > - Memory: 1.2G (peak: 2.8G swap: 1.7G swap peak: 3.2G zswap: 58.3M) > - CPU: 6h 10min 7.847s > + Memory: 2.8G (peak: 2.8G swap: 0B swap peak: 3.2G) > + CPU: 6h 10min 14.748s > CGroup: /system.slice/sddm.service > =E2=94=9C=E2=94=80710 /usr/bin/sddm > =E2=94=94=E2=94=80746 /usr/lib/Xorg -nolisten tcp -backgrou= nd none -seat seat0 vt2 -auth /run/sddm/xauth_GXHGRA -noreset -displayfd 20 > @@ -22,36 +22,36 @@ > =D0=BE=D0=BA=D1=82 28 09:22:36 dell-g15 sddm-helper[925]: Writing cookie= to "/tmp/xauth_RKKlcB" > =D0=BE=D0=BA=D1=82 28 09:22:36 dell-g15 sddm-helper[925]: Starting X11 s= ession: "" "/usr/share/sddm/scripts/Xsession \"env KDEWM=3D/usr/bin/i3 /usr= /bin/startplasma-x11\"" > =D0=BE=D0=BA=D1=82 28 09:22:36 dell-g15 sddm[710]: Session started true > -anon 42807296 > -file 1150423040 > -kernel 79376384 > +anon 93822976 Anonymous memory increased, but not by too much. > +file 2957750272 > +kernel 18210816 > kernel_stack 147456 > pagetables 7204864 > sec_pagetables 0 > percpu 2184 > sock 0 > vmalloc 12288 > -shmem 1150795776 > -zswap 61173438 > -zswapped 1751408640 > +shmem 2958123008 shmem increased by a lot (~1.8G). So this looks like it could be the answer to your question about where the swap usage is coming from. I would try to find what tmpfs files are used by this application. > +zswap 0 > +zswapped 0 > file_mapped 4108288 > file_dirty 0 > file_writeback 0 > -swapcached 17666048 > +swapcached 0 > anon_thp 2097152 > file_thp 0 > shmem_thp 0 > -inactive_anon 445014016 > -active_anon 625201152 > +inactive_anon 589209600 > +active_anon 2321489920 > inactive_file 2895872 > active_file 2244608 > -unevictable 140836864 > -slab_reclaimable 8166656 > -slab_unreclaimable 2618128 > -slab 10784784 > -workingset_refault_anon 69854 > +unevictable 141144064 > +slab_reclaimable 8169032 > +slab_unreclaimable 2625208 > +slab 10794240 > +workingset_refault_anon 177253 > workingset_refault_file 12496 > -workingset_activate_anon 33476 > +workingset_activate_anon 41579 > workingset_activate_file 2372 > workingset_restore_anon 12558 > workingset_restore_file 2132 > @@ -64,14 +64,14 @@ > pgsteal_kswapd 1243374 > pgsteal_direct 348876 > pgsteal_khugepaged 9149 > -pgfault 626853 > +pgfault 626941 > pgmajfault 11521 > pgrefill 560417 > pgactivate 85087 > pgdeactivate 0 > pglazyfree 0 > pglazyfreed 0 > -zswpin 87568 > +zswpin 515158 > zswpout 1395410 > zswpwb 211559 > thp_fault_alloc 8