[PATCH 0/3] memcg oom: bail out from the charge path if no victim found

[Yafang Shao <laoar.shao@gmail.com>]

Without considering the manually triggered OOM, if no victim found in
system OOM, the system will be deadlocked on memory, however if no
victim found in memcg OOM, it can charge successfully and runs well.
This behavior in memcg oom is not proper because that can prevent the
memcg from being limited.

Take an easy example.
        $ cd /sys/fs/cgroup/foo/
        $ echo $$ > cgroup.procs
        $ echo 200M > memory.max
        $ cat memory.max
        209715200
        $ echo -1000 > /proc/$$/oom_score_adj
Then, let's run a memhog task in memcg foo, which will allocate 1G
memory and keeps running.
        $ /home/yafang/test/memhog &
Then memory.current will be greater than memory.max. Run bellow command
in another shell.
        $ cat /sys/fs/cgroup/foo/memory.current
        1097228288
The tasks which have already allocated memory and won't allocate new
memory still runs well. This behavior makes nonsense.

This patch is to improve it.
If no victim found in memcg oom, we should force the current task to
wait until there's available pages. That is similar with the behavior in
memcg1 when oom_kill_disable is set.

Patch #1 and #2 are the preparation of patch #3.

Yafang Shao (3):
  mm: change the return type of out_of_memory()
  mm, memcg: introduce a new helper task_in_memcg_oom_set()
  memcg oom: bail out from the charge path if no victim found

 include/linux/memcontrol.h | 30 ++++++++++++++++++
 include/linux/oom.h        |  9 +++++-
 include/linux/sched.h      |  1 +
 mm/memcontrol.c            | 63 ++++++++++++++++++++------------------
 mm/oom_kill.c              | 36 +++++++++++++++++-----
 mm/page_alloc.c            |  3 +-
 6 files changed, 104 insertions(+), 38 deletions(-)

-- 
2.18.2