Re: "make -j" with memory.(memsw.)limit_in_bytes smaller than required -> livelock, even for unlimited processes

[KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>]

On Tue, 21 Jun 2011 16:51:18 +0200
Lutz Vieweg <lvml@5t9.de> wrote:

> Dear Memory Ressource Controller maintainers,
> 
> by using per-user control groups with a limit on memory (and swap) I am
> trying to secure a shared development server against memory exhaustion
> by any one single user - as it happened before when somebody imprudently
> issued "make -j" (which has the infamous habit to spawn an unlimited
> number of processes) on a large software project with many source files.
> 
> The memory limitation using control groups works just fine when
> only a few processes sum up to a usage that exceeds the limits - the
> processes are OOM-killed, then, and the others users are unaffected.
> 
> But the original cause, a "make -j" on many source files, leads to
> the following ugly symptom:
> 
> - make starts numerous (~ 100 < x < 200) gcc processes
> 
> - some of those gcc processes get OOM-killed quickly, then
>    a few more are killed, but with increasing pauses in between
> 
> - then after a few seconds, no more gcc processes are killed, but
>    the "make" process and its childs do not show any progress anymore
> 

This is a famous fork-bomb problem. I posted fork-bomb-killer patch sets once
but not welcomed. And, there are OOM-killer trouble in kernel, too.
(I think it's recently fixed.)



Don't you use your test set under some cpu cgroup ?
If so, you can see  deadlock in some versions of kernel.


Then, you can stop oom-kill by echo 1 > .../memory.oom_control.
All processes under memcg will be blocked. you can kill all process under memcg
by you hands.

> - at this time, top indicates 100% "system" CPU usage, mostly by
>    "[kworker/*]" threads (one per CPU). But processes from other
>    users, that only require CPU, proceed to run.
> 

This is a known bug and it's now fixed.


> - but also at this time, if any other user (who has not exhausted
>    his memory limits) tries to access any file (at least on /tmp/,
>    as e.g. gcc does), even a simple "ls /tmp/", this operation
>    waits forever. (But "iostat" does not indicate any I/O activity.)
> 

Hmm, it means your 'ls' gets some lock and wait for it. Then, what lock
you wait for ? what w_chan is shown in 'ps -elf' ?


> - as soon as you press "CTRL-C" to abort the "make -j", everything
>    goes back to normal, quickly - also the other users' processes proceed.
> 

yes.


> 
> To reproduce the problem, the attached "Makefile" to a directory
> on a filesystem with at least 70MB free space, then
> 
>   mount -o memory none /cgroup
>   mkdir /cgroup/test
>   echo 64M >/cgroup/test/memory.limit_in_bytes
>   echo 64M >/cgroup/test/memory.memsw.limit_in_bytes
> 

64M is crazy small limit for make -j , I use 300M for my test...



>   cd /somewhere/with/70mb/free
>   echo $$ >/cgroup/test/tasks
>   make sources
>   make -j compile
> 
> Notice that "make sources" will create 200 bogus "*.c" files from
> /dev/urandom to make sure that "gcc" will use up some memory.
> 
> The "make -j compile" reliably reproduces the above mentioned syndrome,
> here.
> 
> Please notice that the livelock does happen only with a significant
> number of parallel compiler runs - it did e.g. not happen with
> only 100 for me, and it also did not happen when I started "make"
> with "strace" - so timing seems to be an issue, here.
> 
> Thanks for any hints towards a solution of this issue in advance!
> 

I think the most of problem comes from oom-killer logic.

Anyway, please post oom-killer log.

and plesse see what hapeens when 

 echo 1 > /memory.oom_control
 (See Documentation/cgroup/memory.txt)




Thsnks,
-Kame



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>