* The percpu memory used by memcg cannot be cleared
@ 2024-08-29 17:06 Liu Jing
2024-08-30 7:08 ` Michal Hocko
0 siblings, 1 reply; 4+ messages in thread
From: Liu Jing @ 2024-08-29 17:06 UTC (permalink / raw)
To: akpm; +Cc: linux-mm, linux-kernel, Liu Jing
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 3587 bytes --]
hello，linux boss
I found a problem in the process of using linux memcg，When I turned swap off, the memcg memory I created with the following script could not be deleted with echo 0 > memory.force_empty, as explained below。
----------------------------------------------------------------------------------------------------------
step1：swapoff -a
step2：use this script to create memcg
#!/bin/bash
mkdir -p /tmp/test
for i in 'seq 2000'
do
sudo mkdir -p /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}
sudo echo $$ > /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}/tasks
sudo echo 'data' > /tmp/test/test$ {i}
sudo echo $$ > /sys/fs/cgroup/memory/user.slice/user-0.slice/tasks
sudo rmdir /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}
done
step3：view /proc/cgroup and /proc/meminfo files
[root@localhost ~]# cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 10 1 1
cpu 4 1 1
cpuacct 4 1 1
blkio 13 1 1
memory 14 2009 1
devices 6 94 1
[root@localhost ~]# cat /proc/meminfo | grep Percpu
Percpu: 600576 kB
step4：when I use "echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty", I find the num_cgroups of memory and percpu have no changed
[root@localhost ~]# echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty
[root@localhost ~]# cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 10 1 1
cpu 4 1 1
cpuacct 4 1 1
blkio 13 1 1
memory 14 2039 1
devices 6 87 1
[root@localhost ~]# cat /proc/meminfo | grep Percpu
Percpu: 600576 kB
step 5: when I use swapon -a to open swap, then echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty again
[root@localhost ~]# swapon -a
[root@localhost ~]# echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty
step 6: view /proc/cgroup and /proc/meminfo files ,I found the the num_cgroups of memory and percpu have been reduced.
[root@localhost ~]# cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 10 1 1
cpu 4 1 1
cpuacct 4 1 1
blkio 13 1 1
memory 14 185 1
devices 6 87 1
freezer 9 1 1
[root@localhost ~]# cat /proc/meminfo | grep Percpu
Percpu: 120832 kB
--------------------------------------------------------------------------------------------------------
Therefore, I want to know why swap affects memcg memory reclamation, echo 0 > memory.force_empty this interface should force the memory used by the cgroup to be reclaimed.
I want to know why ,I look forward to hearing back from the community.
Thanks.
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: The percpu memory used by memcg cannot be cleared
2024-08-29 17:06 The percpu memory used by memcg cannot be cleared Liu Jing
@ 2024-08-30 7:08 ` Michal Hocko
0 siblings, 0 replies; 4+ messages in thread
From: Michal Hocko @ 2024-08-30 7:08 UTC (permalink / raw)
To: Liu Jing; +Cc: akpm, linux-mm, linux-kernel
On Fri 30-08-24 01:06:44, Liu Jing wrote:
> hello，linux boss
>
> I found a problem in the process of using linux memcg，When I turned swap off, the memcg memory I created with the following script could not be deleted with echo 0 > memory.force_empty, as explained below。
>
> ----------------------------------------------------------------------------------------------------------
> step1：swapoff -a
>
>
> step2：use this script to create memcg
>
> #!/bin/bash
> mkdir -p /tmp/test
> for i in 'seq 2000'
> do
> sudo mkdir -p /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}
> sudo echo $$ > /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}/tasks
> sudo echo 'data' > /tmp/test/test$ {i}
> sudo echo $$ > /sys/fs/cgroup/memory/user.slice/user-0.slice/tasks
> sudo rmdir /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}
> done
[...]
I assume you /tmp is tmpfs backed.
> Therefore, I want to know why swap affects memcg memory reclamation,
> echo 0 > memory.force_empty this interface should force the memory
> used by the cgroup to be reclaimed.
If the above is true then you simply do not have any backing storage to
reclaim to. You need swap to reclaim tmpfs/shmem memory.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 4+ messages in thread
* The percpu memory used by memcg cannot be cleared
@ 2024-08-30 3:22 liujing
2024-08-30 7:13 ` Yosry Ahmed
0 siblings, 1 reply; 4+ messages in thread
From: liujing @ 2024-08-30 3:22 UTC (permalink / raw)
To: akpm; +Cc: linux-mm, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 3936 bytes --]
\r\r\r\r\r\rhello，linux boss\r\r\r\r\r I found a problem in the process of using linux memcg，When I turned swap off, the memcg memory I created with the following script could not be deleted with echo 0 > memory.force_empty, as explained below。\r\r\r\r\r----------------------------------------------------------------------------------------------------------\r\rstep1：swapoff -a\r\r\r\r\r\r\r\rstep2：use this script to create memcg\r\r\r\r\r\r\r#!/bin/bash\r\rmkdir -p /tmp/test\r\rfor i in 'seq 2000'\r\rdo\r\r sudo mkdir -p /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}\r\r sudo echo $$ > /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}/tasks\r\r sudo echo 'data' > /tmp/test/test$ {i}\r\r sudo echo $$ > /sys/fs/cgroup/memory/user.slice/user-0.slice/tasks\r\r sudo rmdir /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}\r\rdone\r\r\r\r\r\r\r\rstep3：view /proc/cgroup and /proc/meminfo files\r\r\r\r\r\r\r[root@localhost ~]# cat /proc/cgroups\r\r#subsys_name hierarchy num_cgroups enabled\r\rcpuset 10 1 1\r\rcpu 4 1 1\r\rcpuacct 4 1 1\r\rblkio 13 1 1\r\rmemory 14 2009 1\r\rdevices 6 94 1\r\r\r\r\r\r\r[root@localhost ~]# cat /proc/meminfo | grep Percpu\r\rPercpu: 600576 kB\r\r\r\r\r\r\r\rstep4：when I use "echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty", I find the num_cgroups of memory and percpu have no changed\r\r\r\r\r\r\r[root@localhost ~]# echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty\r\r[root@localhost ~]# cat /proc/cgroups\r\r#subsys_name hierarchy num_cgroups enabled\r\rcpuset 10 1 1\r\rcpu 4 1 1\r\rcpuacct 4 1 1\r\rblkio 13 1 1\r\rmemory 14 2039 1\r\rdevices 6 87 1\r\r\r\r\r\r\r[root@localhost ~]# cat /proc/meminfo | grep Percpu\r\rPercpu: 600576 kB\r\r\r\r\r\r\r\rstep 5: when I use swapon -a to open swap, then echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty again\r\r\r\r\r\r\r[root@localhost ~]# swapon -a\r\r[root@localhost ~]# echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty\r\r\r\r\r\r\r\rstep 6: view /proc/cgroup and /proc/meminfo files ,I found the the num_cgroups of memory and percpu have been reduced. \r\r\r\r[root@localhost ~]# cat /proc/cgroups\r\r#subsys_name hierarchy num_cgroups enabled\r\rcpuset 10 1 1\r\rcpu 4 1 1\r\rcpuacct 4 1 1\r\rblkio 13 1 1\r\rmemory 14 185 1\r\rdevices 6 87 1\r\rfreezer 9 1 1\r\r\r\r\r\r\r[root@localhost ~]# cat /proc/meminfo | grep Percpu\r\rPercpu: 120832 kB\r\r--------------------------------------------------------------------------------------------------------\r\r\r\r\r\r\r\rTherefore, I want to know why swap affects memcg memory reclamation, echo 0 > memory.force_empty this interface should force the memory used by the cgroup to be reclaimed. \r\rI want to know why ,I look forward to hearing back from the community.\r\rThanks.\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r-----------------------------------------\r\r刘婧 计算产品部\r\r\r\r手机：18896725597\r\r邮箱：liujing@cmss.chinamobile.com\r\r中移（苏州）软件技术有限公司\r\r\r
[-- Attachment #2: Type: text/html, Size: 16267 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: The percpu memory used by memcg cannot be cleared
2024-08-30 3:22 liujing
@ 2024-08-30 7:13 ` Yosry Ahmed
0 siblings, 0 replies; 4+ messages in thread
From: Yosry Ahmed @ 2024-08-30 7:13 UTC (permalink / raw)
To: liujing
Cc: akpm, linux-mm, linux-kernel, Johannes Weiner, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song
On Thu, Aug 29, 2024 at 8:22 PM liujing <liujing@cmss.chinamobile.com> wrote:
>
> hello，linux boss
>
> I found a problem in the process of using linux memcg，When I turned swap off, the memcg memory I created with the following script could not be deleted with echo 0 > memory.force_empty, as explained below。
(Adding memcg maintainers in case they are interested)
It's not a problem, it's the way the linux kernel currently behaves in
terms of handling deleted memcgs that are still referenced in the
kernel (i.e. offline/dying/zombie memcgs).
>
> ----------------------------------------------------------------------------------------------------------
> step1：swapoff -a
>
>
> step2：use this script to create memcg
>
> #!/bin/bash
> mkdir -p /tmp/test
> for i in 'seq 2000'
> do
> sudo mkdir -p /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}
> sudo echo $$ > /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}/tasks
> sudo echo 'data' > /tmp/test/test$ {i}
Assuming /tmp is a tmpfs mount, here you created 2000 child memcgs and
allocated one tmpfs page in each of them. So each of those child
memcgs is charged for one page of memory, and each charge holds a
reference to the the respective memcg.
> sudo echo $$ > /sys/fs/cgroup/memory/user.slice/user-0.slice/tasks
> sudo rmdir /sys/fs/cgroup/memory/user.slice/user-0.slice/test$ {i}
Then you deleted those memcgs, but the kernel cannot free them yet
because the tmpfs memory you allocated above is still charged to them.
> done
>
>
> step3：view /proc/cgroup and /proc/meminfo files
>
> [root@localhost ~]# cat /proc/cgroups
> #subsys_name hierarchy num_cgroups enabled
> cpuset 10 1 1
> cpu 4 1 1
> cpuacct 4 1 1
> blkio 13 1 1
> memory 14 2009 1
Here you can see the cgroups you deleted still exist in the kernel.
> devices 6 94 1
>
> [root@localhost ~]# cat /proc/meminfo | grep Percpu
> Percpu: 600576 kB
The percpu memory you observe here is likely the per-CPU metadata that
the kernel uses to keep track of each memcg. Since the memcgs are not
freed, the metadata is not freed either.
>
>
> step4：when I use "echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty", I find the num_cgroups of memory and percpu have no changed
Yes, because at this point there is no swap, so the tmpfs memory
charged to the deleted memcg cannot be reclaimed and cannot be freed,
and the refs they hold cannot be dropped.
>
> [root@localhost ~]# echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty
> [root@localhost ~]# cat /proc/cgroups
> #subsys_name hierarchy num_cgroups enabled
> cpuset 10 1 1
> cpu 4 1 1
> cpuacct 4 1 1
> blkio 13 1 1
> memory 14 2039 1
> devices 6 87 1
>
> [root@localhost ~]# cat /proc/meminfo | grep Percpu
> Percpu: 600576 kB
>
>
> step 5: when I use swapon -a to open swap, then echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty again
>
> [root@localhost ~]# swapon -a
> [root@localhost ~]# echo 0 > /sys/fs/cgroup/memory/user.slice/user-0.slice/memory.force_empty
When you add a swapfile and try to reclaim memory from the cgroups
again, the kernel is able to reclaim the tmpfs memory by swapping it
out. The kernel is smart enough at this point to not charge the swap
slots to the deleted cgroups, but to their living/online parent. At
this point, the tmpfs memory is uncharged and freed, and the refs to
the deleted cgroups are dropped. Now they can be deleted by the
kernel.
>
>
> step 6: view /proc/cgroup and /proc/meminfo files ,I found the the num_cgroups of memory and percpu have been reduced.
> [root@localhost ~]# cat /proc/cgroups
> #subsys_name hierarchy num_cgroups enabled
> cpuset 10 1 1
> cpu 4 1 1
> cpuacct 4 1 1
> blkio 13 1 1
> memory 14 185 1
> devices 6 87 1
> freezer 9 1 1
>
> [root@localhost ~]# cat /proc/meminfo | grep Percpu
> Percpu: 120832 kB
Now the memcgs are freed, and their associated per-CPU metadata is also freed.
> --------------------------------------------------------------------------------------------------------
>
>
> Therefore, I want to know why swap affects memcg memory reclamation, echo 0 > memory.force_empty this interface should force the memory used by the cgroup to be reclaimed.
> I want to know why ,I look forward to hearing back from the community.
I hope it's now clear that the per-CPU memory cannot be freed when you
use memory.force_empty on the parent memcg, because the per-CPU memory
is the metadata of the deleted memcgs, and those cannot be freed until
the memory charged to them is freed (which needs swap, because it's
tmpfs not a regular file).
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-08-30 7:14 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-08-29 17:06 The percpu memory used by memcg cannot be cleared Liu Jing
2024-08-30 7:08 ` Michal Hocko
2024-08-30 3:22 liujing
2024-08-30 7:13 ` Yosry Ahmed
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox