low memory killer

low memory killer

[zhiyuan_zhu]

[-- Attachment #1: Type: text/plain, Size: 2049 bytes --]

hi Johannes

My name is Zhiyuan zhu, an android development engineer in htc.
We encounter a lowmemorykiller’s problem.
Coluld you help to kindly support? Thank you.

Problem describtion:
We noticed that gap of dumping cached value from “lowmemorykiller” and “/proc/meminfo/” are different apparently. Like example below, lowmemorykiller showed cache only has 72460kB while launcher was killed, but /proc/meminfo showed cached still has 142448kB. Please check why the gap of cache value between lowmemorykill and /proc/meminfo are huge.

kernel_e0058_0001_20141107_204711_LC4ABYA00200_htc_a31ul_0.54.999.1.txt at LC4ABYA00200
        6      161030.084144       2014-11-07 21:44:53.304        lowmemorykiller: Killing 'om.htc.launcher' (4486), adj 294,
6      161030.084144       2014-11-07 21:44:53.304           to free 47856kB on behalf of 'kworker/u8:14' (20594) because
6      161030.084144       2014-11-07 21:44:53.304           cache 72460kB is below limit 73728kB for oom_score_adj 235
6      161030.084144       2014-11-07 21:44:53.304           Free memory is 51304kB above reserved
4      161030.084797       2014-11-07 21:44:53.304        MemFree:           55676 kB
4      161030.084797       2014-11-07 21:44:53.304        Buffers:            1240 kB
4      161030.084797       2014-11-07 21:44:53.304        Cached:           142448 kB

Lowmemorykiller calculated cache value is 72460kB, but the /proc/meminfo’s cached is 142448 kB

After checked the code, I found that:
Lowmemorykiller’s memory information is comes from /proc/zoneinfo file’s nr_file_pages
So I want to know how different the /proc/zoneinfo file’s nr_file_pages and /proc/meminfo file’s Cached ?

Thanks
Zhiyuan zhu

MEMORY RESOURCE CONTROLLER
M: Johannes Weiner <hannes@cmpxchg.org>
M: Michal Hocko <mhocko@suse.cz>
M: Balbir Singh <bsingharora@gmail.com>
M: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
L:   cgroups@vger.kernel.org
L:   linux-mm@kvack.org
S:  Maintained
F:  mm/memcontrol.c
F:  mm/page_cgroup.c

[-- Attachment #2: Type: text/html, Size: 34203 bytes --]

Re: low memory killer

[Michal Hocko]

[Adding Greg to CC as the current maintainer of the staging and dropping
majordomo]

I am not Johannes but,

On Thu 20-11-14 09:26:05, zhiyuan_zhu@htc.com wrote:
> hi Johannes
> 
> My name is Zhiyuan zhu, an android development engineer in htc.
> We encounter a lowmemorykillera??s problem.
> Coluld you help to kindly support? Thank you.

Please note that lowmemory killer is not part of Memory cgroup resource
controller. It is a staging driver which is supposed to be supported by
support by Android people. I am not sure about the current state but the
implementation and some concepts used to be broken in many ways.
Anyway I think the driver should be dropped from the tree or try to get
promoted to the regular tree _after_ it passes a proper review.
 
> Problem describtion:
> We noticed that gap of dumping cached value from a??lowmemorykillera??
> and a??/proc/meminfo/a?? are different apparently. Like example below,
> lowmemorykiller showed cache only has 72460kB while launcher was
> killed, but /proc/meminfo showed cached still has 142448kB. Please
> check why the gap of cache value between lowmemorykill and
> /proc/meminfo are huge.

lowmem_scan prints:
other_file = global_page_state(NR_FILE_PAGES) - global_page_state(NR_SHMEM) - total_swapcache_pages();

as per meminfo_proc_show(), Cached value corresponds to:
        cached = global_page_state(NR_FILE_PAGES) - total_swapcache_pages() - i.bufferram;

So those two values are quite different. E.g. lowmem killer ignores
shmem pages. This can be considerable amount of memory.

> kernel_e0058_0001_20141107_204711_LC4ABYA00200_htc_a31ul_0.54.999.1.txt at LC4ABYA00200
>         6      161030.084144       2014-11-07 21:44:53.304        lowmemorykiller: Killing 'om.htc.launcher' (4486), adj 294,
> 6      161030.084144       2014-11-07 21:44:53.304           to free 47856kB on behalf of 'kworker/u8:14' (20594) because
> 6      161030.084144       2014-11-07 21:44:53.304           cache 72460kB is below limit 73728kB for oom_score_adj 235
> 6      161030.084144       2014-11-07 21:44:53.304           Free memory is 51304kB above reserved
> 4      161030.084797       2014-11-07 21:44:53.304        MemFree:           55676 kB
> 4      161030.084797       2014-11-07 21:44:53.304        Buffers:            1240 kB
> 4      161030.084797       2014-11-07 21:44:53.304        Cached:           142448 kB

I do not see any code in drivers/staging/android/lowmemorykiller.c that
would print such an information in the current tree.

> Lowmemorykiller calculated cache value is 72460kB, but the
> /proc/meminfoa??s cached is 142448 kB
> 
> After checked the code, I found that:
> Lowmemorykillera??s memory information is comes from /proc/zoneinfo
> filea??s nr_file_pages So I want to know how different the
> /proc/zoneinfo filea??s nr_file_pages and /proc/meminfo filea??s
> Cached ?

See the above.

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

答复: low memory killer

[zhiyuan_zhu]

[-- Attachment #1: Type: text/plain, Size: 3999 bytes --]

Hi Michal
Thanks for your kindly support.
I got a device, and dump the /proc/meminfo and /proc/vmstat files, they are the Linux standard proc files.
I found that: Cached = 339880 KB, but nr_free_pages=14675*4 = 58700KB and nr_shmem = 508*4=2032KB

nr_shmem is just a little memory, and nr free pages + nr_shmem is largely less than Cached.
So why nr_free_pages is largely less than Cached? Thank you.

$ adb shell cat /proc/meminfo
MemTotal:        1366904 kB
MemFree:           59040 kB
Buffers:           54600 kB
Cached:           339880 kB

$ adb shell cat /proc/vmstat
nr_free_pages 14675
nr_inactive_anon 73520
nr_shmem 508

Thanks
Zhiyuan zhu

-----邮件原件-----
发件人: Michal Hocko [mailto:mhocko@suse.cz] 
发送时间: 2014年11月20日 17:58
收件人: Zhiyuan Zhu(朱志遠)
抄送: hannes@cmpxchg.org; Future Zhou(周未來); Rachel Zhang(張瑩); bsingharora@gmail.com; kamezawa.hiroyu@jp.fujitsu.com; cgroups@vger.kernel.org; linux-mm@kvack.org; Greg KH
主题: Re: low memory killer

[Adding Greg to CC as the current maintainer of the staging and dropping majordomo]

I am not Johannes but,

On Thu 20-11-14 09:26:05, zhiyuan_zhu@htc.com wrote:
> hi Johannes
> 
> My name is Zhiyuan zhu, an android development engineer in htc.
> We encounter a lowmemorykiller’s problem.
> Coluld you help to kindly support? Thank you.

Please note that lowmemory killer is not part of Memory cgroup resource controller. It is a staging driver which is supposed to be supported by support by Android people. I am not sure about the current state but the implementation and some concepts used to be broken in many ways.
Anyway I think the driver should be dropped from the tree or try to get promoted to the regular tree _after_ it passes a proper review.
 
> Problem describtion:
> We noticed that gap of dumping cached value from “lowmemorykiller”
> and “/proc/meminfo/” are different apparently. Like example below, 
> lowmemorykiller showed cache only has 72460kB while launcher was 
> killed, but /proc/meminfo showed cached still has 142448kB. Please 
> check why the gap of cache value between lowmemorykill and 
> /proc/meminfo are huge.

lowmem_scan prints:
other_file = global_page_state(NR_FILE_PAGES) - global_page_state(NR_SHMEM) - total_swapcache_pages();

as per meminfo_proc_show(), Cached value corresponds to:
        cached = global_page_state(NR_FILE_PAGES) - total_swapcache_pages() - i.bufferram;

So those two values are quite different. E.g. lowmem killer ignores shmem pages. This can be considerable amount of memory.

> kernel_e0058_0001_20141107_204711_LC4ABYA00200_htc_a31ul_0.54.999.1.txt at LC4ABYA00200
>         6      161030.084144       2014-11-07 21:44:53.304        lowmemorykiller: Killing 'om.htc.launcher' (4486), adj 294,
> 6      161030.084144       2014-11-07 21:44:53.304           to free 47856kB on behalf of 'kworker/u8:14' (20594) because
> 6      161030.084144       2014-11-07 21:44:53.304           cache 72460kB is below limit 73728kB for oom_score_adj 235
> 6      161030.084144       2014-11-07 21:44:53.304           Free memory is 51304kB above reserved
> 4      161030.084797       2014-11-07 21:44:53.304        MemFree:           55676 kB
> 4      161030.084797       2014-11-07 21:44:53.304        Buffers:            1240 kB
> 4      161030.084797       2014-11-07 21:44:53.304        Cached:           142448 kB

I do not see any code in drivers/staging/android/lowmemorykiller.c that would print such an information in the current tree.

> Lowmemorykiller calculated cache value is 72460kB, but the 
> /proc/meminfo’s cached is 142448 kB
> 
> After checked the code, I found that:
> Lowmemorykiller’s memory information is comes from /proc/zoneinfo 
> file’s nr_file_pages So I want to know how different the 
> /proc/zoneinfo file’s nr_file_pages and /proc/meminfo file’s Cached ?

See the above.

--
Michal Hocko
SUSE Labs

[-- Attachment #2: Type: text/html, Size: 7613 bytes --]

Re: 答复: low memory killer

[Michal Hocko]

On Thu 20-11-14 10:09:25, zhiyuan_zhu@htc.com wrote:
> Hi Michal
> Thanks for your kindly support.
> I got a device, and dump the /proc/meminfo and /proc/vmstat files,
> they are the Linux standard proc files. 
> I found that: Cached = 339880 KB, but nr_free_pages=14675*4 = 58700KB
> and nr_shmem = 508*4=2032KB
>
> nr_shmem is just a little memory, and nr free pages + nr_shmem is
> largely less than Cached.  So why nr_free_pages is largely less than
> Cached? Thank you.

nr_free_pages refers to pages which are not allocated. Cached referes to
a used memory which is easily reclaimable so it can be reused should
there be a need and free memory drops down. So this is a normal
situation. How is this related to the lowmemory killer question posted
previously?

[...]
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

答复: 答复: low memory killer

[zhiyuan_zhu]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="gb2312", Size: 3599 bytes --]

Hi Greg/Michal
Very sorry I have a mistake in previous mail. (It should be nr_file_pages not nr_free_pages)
I rearrange this problem.

// *********** log begin **********
6      161030.084144       2014-11-07 21:44:53.304        lowmemorykiller: Killing 'om.htc.launcher' (4486), adj 294,
6      161030.084144       2014-11-07 21:44:53.304           to free 47856kB on behalf of 'kworker/u8:14' (20594) because
6      161030.084144       2014-11-07 21:44:53.304           cache 72460kB is below limit 73728kB for oom_score_adj 235
//  **** /proc/meminfo 's result
4      161030.084797       2014-11-07 21:44:53.304        Cached:           142448 kB
// *********** log end **********

After I checked the android's low memory strategy: kernel/drivers/staging/android/lowmemorykiller.c

// ****** code begin *********
		other_file = global_page_state(NR_FILE_PAGES) -
						global_page_state(NR_SHMEM) -
						total_swapcache_pages();

		lowmem_print(1, "Killing '%s' (%d), adj %hd,\n" \
				"   to free %ldkB on behalf of '%s' (%d) because\n" \
				"   cache %ldkB is below limit %ldkB for oom_score_adj %hd\n" \
				"   Free memory is %ldkB above reserved\n",
			     selected->comm, selected->pid,
			     selected_oom_score_adj,
			     selected_tasksize * (long)(PAGE_SIZE / 1024),
			     current->comm, current->pid,
			     other_file * (long)(PAGE_SIZE / 1024),
			     minfree * (long)(PAGE_SIZE / 1024),
			     min_score_adj,
			     other_free * (long)(PAGE_SIZE / 1024));
// ******* code end ************

So android's strategy's free memory is = other_file = (nr file pages - nr shmem - total swapcache pages) * 4K = [cache 72460kB]
But the system's free memory is: Cached:        142448 kB  // from /proc/meminfo

And system's free memory is: Cached + MemFree + Buffers is largely than the memory which anroid lowmemorykiller calculated memory [cache 72460K]
At this time point, system will kill some important processes, but system have enough memory.
This is android's lowmemorykiller defect? or Linux kernel memory's defect?

So I have some questions:
I have a question: what's the nr file pages mean? What different between nr_file_pages from Cached (from /proc/meminfo)?
And nr shmem, swapcache pages are small, so I think this is the key problem why android's stragegy calculated free memory is largely less than /proc/meminfo Cached's value.

Thanks
Zhiyuan zhu

-----ÓʼþÔ­¼þ-----
·¢¼þÈË: Michal Hocko [mailto:mhocko@suse.cz] 
·¢ËÍʱ¼ä: 2014Äê11ÔÂ20ÈÕ 18:19
ÊÕ¼þÈË: Zhiyuan Zhu(ÖìÖ¾ßh)
³­ËÍ: hannes@cmpxchg.org; Future Zhou(ÖÜδí); Rachel Zhang(ˆ¬“); bsingharora@gmail.com; kamezawa.hiroyu@jp.fujitsu.com; cgroups@vger.kernel.org; linux-mm@kvack.org; greg@kroah.com
Ö÷Ìâ: Re: ´ð¸´: low memory killer

On Thu 20-11-14 10:09:25, zhiyuan_zhu@htc.com wrote:
> Hi Michal
> Thanks for your kindly support.
> I got a device, and dump the /proc/meminfo and /proc/vmstat files, 
> they are the Linux standard proc files.
> I found that: Cached = 339880 KB, but nr_free_pages=14675*4 = 58700KB 
> and nr_shmem = 508*4=2032KB
>
> nr_shmem is just a little memory, and nr free pages + nr_shmem is 
> largely less than Cached.  So why nr_free_pages is largely less than 
> Cached? Thank you.

nr_free_pages refers to pages which are not allocated. Cached referes to a used memory which is easily reclaimable so it can be reused should there be a need and free memory drops down. So this is a normal situation. How is this related to the lowmemory killer question posted previously?

[...]
--
Michal Hocko
SUSE Labs

[-- Attachment #2: Type: text/html, Size: 6775 bytes --]

Re: 答复: 答复: low memory killer

[Weijie Yang]

[-- Attachment #1: Type: text/plain, Size: 4843 bytes --]

On Thu, Nov 20, 2014 at 8:05 PM, <zhiyuan_zhu@htc.com> wrote:

> Hi Greg/Michal
> Very sorry I have a mistake in previous mail. (It should be nr_file_pages not nr_free_pages)
> I rearrange this problem.
>
> // *********** log begin **********
> 6      161030.084144       2014-11-07 21:44:53.304        lowmemorykiller: Killing 'om.htc.launcher' (4486), adj 294,
> 6      161030.084144       2014-11-07 21:44:53.304           to free 47856kB on behalf of 'kworker/u8:14' (20594) because
> 6      161030.084144       2014-11-07 21:44:53.304           cache 72460kB is below limit 73728kB for oom_score_adj 235
> //  **** /proc/meminfo 's result
> 4      161030.084797       2014-11-07 21:44:53.304        Cached:           142448 kB
> // *********** log end **********
>
> After I checked the android's low memory strategy: kernel/drivers/staging/android/lowmemorykiller.c
>
> // ****** code begin *********
> other_file = global_page_state(NR_FILE_PAGES) -
> global_page_state(NR_SHMEM) -
> total_swapcache_pages();
>
> lowmem_print(1, "Killing '%s' (%d), adj %hd,\n" \
> "   to free %ldkB on behalf of '%s' (%d) because\n" \
> "   cache %ldkB is below limit %ldkB for oom_score_adj %hd\n" \
> "   Free memory is %ldkB above reserved\n",
>      selected->comm, selected->pid,
>      selected_oom_score_adj,
>      selected_tasksize * (long)(PAGE_SIZE / 1024),
>      current->comm, current->pid,
>      other_file * (long)(PAGE_SIZE / 1024),
>      minfree * (long)(PAGE_SIZE / 1024),
>      min_score_adj,
>      other_free * (long)(PAGE_SIZE / 1024));
> // ******* code end ************
>
> So android's strategy's free memory is = other_file = (nr file pages - nr shmem - total swapcache pages) * 4K = [cache 72460kB]
> But the system's free memory is: Cached:        142448 kB  // from /proc/meminfo
>
> And system's free memory is: Cached + MemFree + Buffers is largely than the memory which anroid lowmemorykiller calculated memory [cache 72460K]
> At this time point, system will kill some important processes, but system have enough memory.
> This is android's lowmemorykiller defect? or Linux kernel memory's defect?
>
> So I have some questions:
> I have a question: what's the nr file pages mean? What different between nr_file_pages from Cached (from /proc/meminfo)?
> And nr shmem, swapcache pages are small, so I think this is the key problem why android's stragegy calculated free memory is largely less than /proc/meminfo Cached's value.
>
>
Why lowmemkiller -total_swapcache_pages()? see commit 058dbde92:

staging: android: lowmemorykiller: neglect swap cached pages in other_file
With ZRAM enabled it is observed that lowmemory killer
doesn't trigger properly. swap cached pages are
accounted in NR_FILE, and lowmemorykiller considers
this as reclaimable and adds to other_file. But these
pages can't be reclaimed unless lowmemorykiller triggers.
So subtract swap pages from other_file.


and commit 31d59a4198f will also make help, please check it.


> Thanks
> Zhiyuan zhu
>
> -----邮件原件-----
> 发件人: Michal Hocko [mailto:mhocko@suse.cz]
> 发送时间: 2014年11月20日 18:19
> 收件人: Zhiyuan Zhu(朱志遠)
> 抄送: hannes@cmpxchg.org; Future Zhou(周未來); Rachel Zhang(張瑩); bsingharora@gmail.com; kamezawa.hiroyu@jp.fujitsu.com; cgroups@vger.kernel.org; linux-mm@kvack.org; greg@kroah.com
> 主题: Re: 答复: low memory killer
>
> On Thu 20-11-14 10:09:25, zhiyuan_zhu@htc.com wrote:
> > Hi Michal
> > Thanks for your kindly support.
> > I got a device, and dump the /proc/meminfo and /proc/vmstat files,
> > they are the Linux standard proc files.
> > I found that: Cached = 339880 KB, but nr_free_pages=14675*4 = 58700KB
> > and nr_shmem = 508*4=2032KB
> >
> > nr_shmem is just a little memory, and nr free pages + nr_shmem is
> > largely less than Cached.  So why nr_free_pages is largely less than
> > Cached? Thank you.
>
> nr_free_pages refers to pages which are not allocated. Cached referes to a used memory which is easily reclaimable so it can be reused should there be a need and free memory drops down. So this is a normal situation. How is this related to the lowmemory killer question posted previously?
>
> [...]
> --
> Michal Hocko
> SUSE Labs
>
>
> CONFIDENTIALITY NOTE : The information in this e-mail is confidential and
> privileged; it is intended for use solely by the individual or entity named
> as the recipient hereof. Disclosure, copying, distribution, or use of the
> contents of this e-mail by persons other than the intended recipient is
> strictly prohibited and may violate applicable laws. If you have received
> this e-mail in error, please delete the original message and notify us by
> return email or collect call immediately. Thank you. HTC Corporation
>

[-- Attachment #2: Type: text/html, Size: 6104 bytes --]

Re: 答复: 答复: low memory killer

[Michal Hocko]

On Thu 20-11-14 12:05:00, zhiyuan_zhu@htc.com wrote:
> Hi Greg/Michal
> Very sorry I have a mistake in previous mail. (It should be nr_file_pages not nr_free_pages)
> I rearrange this problem.
> 
> // *********** log begin **********
> 6      161030.084144       2014-11-07 21:44:53.304        lowmemorykiller: Killing 'om.htc.launcher' (4486), adj 294,
> 6      161030.084144       2014-11-07 21:44:53.304           to free 47856kB on behalf of 'kworker/u8:14' (20594) because
> 6      161030.084144       2014-11-07 21:44:53.304           cache 72460kB is below limit 73728kB for oom_score_adj 235

This is missing the last line which dumps Free memory (at least from the
snippet below, because nothing like that is in the current driver code
anymore)

> //  **** /proc/meminfo 's result
> 4      161030.084797       2014-11-07 21:44:53.304        Cached:           142448 kB
> // *********** log end **********
> 
> After I checked the android's low memory strategy: kernel/drivers/staging/android/lowmemorykiller.c
> 
> // ****** code begin *********
> 		other_file = global_page_state(NR_FILE_PAGES) -
> 						global_page_state(NR_SHMEM) -
> 						total_swapcache_pages();
> 
> 		lowmem_print(1, "Killing '%s' (%d), adj %hd,\n" \
> 				"   to free %ldkB on behalf of '%s' (%d) because\n" \
> 				"   cache %ldkB is below limit %ldkB for oom_score_adj %hd\n" \
> 				"   Free memory is %ldkB above reserved\n",
> 			     selected->comm, selected->pid,
> 			     selected_oom_score_adj,
> 			     selected_tasksize * (long)(PAGE_SIZE / 1024),
> 			     current->comm, current->pid,
> 			     other_file * (long)(PAGE_SIZE / 1024),
> 			     minfree * (long)(PAGE_SIZE / 1024),
> 			     min_score_adj,
> 			     other_free * (long)(PAGE_SIZE / 1024));
> // ******* code end ************
> 
> So android's strategy's free memory is = other_file = (nr file pages - nr shmem - total swapcache pages) * 4K = [cache 72460kB]
> But the system's free memory is: Cached:        142448 kB  // from /proc/meminfo

I have already pointed you at meminfo_proc_show() which calculates the
shown value as:
	cached = global_page_state(NR_FILE_PAGES) -
	                        total_swapcache_pages() - i.bufferram;

That means that the two differ by NR_SHMEM - i.bufferram. 

> And system's free memory is: Cached + MemFree + Buffers is largely
> than the memory which anroid lowmemorykiller calculated memory [cache
> 72460K] At this time point, system will kill some important processes,
> but system have enough memory.  This is android's lowmemorykiller
> defect? or Linux kernel memory's defect?

It looks most likely be lowmemory killer bug but this is hard to tell as
I do not know which kernel version are you looking at and what are the
additional patch on top of Vanilla kernel.
The upstream code doesn't care about Cached value as presented by
/proc/meminfo

> So I have some questions:
> I have a question: what's the nr file pages mean?

It counts page cache pages (aka pages backed by a file). This includes
also SHMEM pages.

> What different between nr_file_pages from Cached (from /proc/meminfo)?

See above about meminfo_proc_show.

> And nr shmem, swapcache pages are small, so I think this is the key
> problem why android's stragegy calculated free memory is largely less
> than /proc/meminfo Cached's value.

I do not see the value of shmem and swapcache from the time when the
situation was evaluated.

Anyway I do not think there would be problem with counters but rather
the way how they are used by lowmemory killer and I would be looking
into that code first before checking the core kernel.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>