[PATCH] memcg: handle panic_on_oom=always case

[PATCH] memcg: handle panic_on_oom=always case

[KAMEZAWA Hiroyuki]

tested on mmotm-Feb11.

Balbir-san, Nishimura-san, I want review from both of you.

==

From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

Now, if panic_on_oom=2, the whole system panics even if the oom happend
in some special situation (as cpuset, mempolicy....).
Then, panic_on_oom=2 means painc_on_oom_always.

Now, memcg doesn't check panic_on_oom flag. This patch adds a check.

Maybe someone doubts how it's useful. kdump+panic_on_oom=2 is the
last tool to investigate what happens in oom-ed system. If a task is killed,
the sysytem recovers and used memory were freed, there will be few hint
to know what happnes. In mission critical system, oom should never happen.
Then, investigation after OOM is very important.
Then, panic_on_oom=2+kdump is useful to avoid next OOM by knowing
precise information via snapshot.

TODO:
 - For memcg, it's for isolate system's memory usage, oom-notiifer and
   freeze_at_oom (or rest_at_oom) should be implemented. Then, management
   daemon can do similar jobs (as kdump) in safer way or taking snapshot
   per cgroup.

CC: Balbir Singh <balbir@linux.vnet.ibm.com>
CC: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
CC: David Rientjes <rientjes@google.com>
Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
---
 Documentation/cgroups/memory.txt |    2 ++
 Documentation/sysctl/vm.txt      |    5 ++++-
 mm/oom_kill.c                    |    2 ++
 3 files changed, 8 insertions(+), 1 deletion(-)

Index: mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
===================================================================
--- mmotm-2.6.33-Feb11.orig/Documentation/cgroups/memory.txt
+++ mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
@@ -182,6 +182,8 @@ list.
 NOTE: Reclaim does not work for the root cgroup, since we cannot set any
 limits on the root cgroup.
 
+Note2: When panic_on_oom is set to "2", the whole system will panic.
+
 2. Locking
 
 The memory controller uses the following hierarchy
Index: mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
===================================================================
--- mmotm-2.6.33-Feb11.orig/Documentation/sysctl/vm.txt
+++ mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
@@ -573,11 +573,14 @@ Because other nodes' memory may be free.
 may be not fatal yet.
 
 If this is set to 2, the kernel panics compulsorily even on the
-above-mentioned.
+above-mentioned. Even oom happens under memoyr cgroup, the whole
+system panics.
 
 The default value is 0.
 1 and 2 are for failover of clustering. Please select either
 according to your policy of failover.
+2 seems too strong but panic_on_oom=2+kdump gives you very strong
+tool to investigate a system which should never cause OOM.
 
 =============================================================
 
Index: mmotm-2.6.33-Feb11/mm/oom_kill.c
===================================================================
--- mmotm-2.6.33-Feb11.orig/mm/oom_kill.c
+++ mmotm-2.6.33-Feb11/mm/oom_kill.c
@@ -471,6 +471,8 @@ void mem_cgroup_out_of_memory(struct mem
 	unsigned long points = 0;
 	struct task_struct *p;
 
+	if (sysctl_panic_on_oom == 2)
+		panic("out of memory(memcg). panic_on_oom is selected.\n");
 	read_lock(&tasklist_lock);
 retry:
 	p = select_bad_process(&points, mem);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memcg: handle panic_on_oom=always case

[Daisuke Nishimura]

On Wed, 17 Feb 2010 15:04:45 +0900, KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> tested on mmotm-Feb11.
> 
> Balbir-san, Nishimura-san, I want review from both of you.
> 
I've read only part of the original patch set yet, but I agree to the direction
of making memcg's oom panic the system on panic_on_oom==2, not panic on panic_on_oom==1.

	Reviewed-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>

Thanks,
Daisuke Nishimura.

> ==
> 
> From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> 
> Now, if panic_on_oom=2, the whole system panics even if the oom happend
> in some special situation (as cpuset, mempolicy....).
> Then, panic_on_oom=2 means painc_on_oom_always.
> 
> Now, memcg doesn't check panic_on_oom flag. This patch adds a check.
> 
> Maybe someone doubts how it's useful. kdump+panic_on_oom=2 is the
> last tool to investigate what happens in oom-ed system. If a task is killed,
> the sysytem recovers and used memory were freed, there will be few hint
> to know what happnes. In mission critical system, oom should never happen.
> Then, investigation after OOM is very important.
> Then, panic_on_oom=2+kdump is useful to avoid next OOM by knowing
> precise information via snapshot.
> 
> TODO:
>  - For memcg, it's for isolate system's memory usage, oom-notiifer and
>    freeze_at_oom (or rest_at_oom) should be implemented. Then, management
>    daemon can do similar jobs (as kdump) in safer way or taking snapshot
>    per cgroup.
> 
> CC: Balbir Singh <balbir@linux.vnet.ibm.com>
> CC: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
> CC: David Rientjes <rientjes@google.com>
> Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> ---
>  Documentation/cgroups/memory.txt |    2 ++
>  Documentation/sysctl/vm.txt      |    5 ++++-
>  mm/oom_kill.c                    |    2 ++
>  3 files changed, 8 insertions(+), 1 deletion(-)
> 
> Index: mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/Documentation/cgroups/memory.txt
> +++ mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> @@ -182,6 +182,8 @@ list.
>  NOTE: Reclaim does not work for the root cgroup, since we cannot set any
>  limits on the root cgroup.
>  
> +Note2: When panic_on_oom is set to "2", the whole system will panic.
> +
>  2. Locking
>  
>  The memory controller uses the following hierarchy
> Index: mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/Documentation/sysctl/vm.txt
> +++ mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> @@ -573,11 +573,14 @@ Because other nodes' memory may be free.
>  may be not fatal yet.
>  
>  If this is set to 2, the kernel panics compulsorily even on the
> -above-mentioned.
> +above-mentioned. Even oom happens under memoyr cgroup, the whole
> +system panics.
>  
>  The default value is 0.
>  1 and 2 are for failover of clustering. Please select either
>  according to your policy of failover.
> +2 seems too strong but panic_on_oom=2+kdump gives you very strong
> +tool to investigate a system which should never cause OOM.
>  
>  =============================================================
>  
> Index: mmotm-2.6.33-Feb11/mm/oom_kill.c
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/mm/oom_kill.c
> +++ mmotm-2.6.33-Feb11/mm/oom_kill.c
> @@ -471,6 +471,8 @@ void mem_cgroup_out_of_memory(struct mem
>  	unsigned long points = 0;
>  	struct task_struct *p;
>  
> +	if (sysctl_panic_on_oom == 2)
> +		panic("out of memory(memcg). panic_on_oom is selected.\n");
>  	read_lock(&tasklist_lock);
>  retry:
>  	p = select_bad_process(&points, mem);
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memcg: handle panic_on_oom=always case

[Nick Piggin]

On Wed, Feb 17, 2010 at 03:04:45PM +0900, KAMEZAWA Hiroyuki wrote:
> tested on mmotm-Feb11.
> 
> Balbir-san, Nishimura-san, I want review from both of you.
> 
> ==
> 
> From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> 
> Now, if panic_on_oom=2, the whole system panics even if the oom happend
> in some special situation (as cpuset, mempolicy....).
> Then, panic_on_oom=2 means painc_on_oom_always.
> 
> Now, memcg doesn't check panic_on_oom flag. This patch adds a check.
> 
> Maybe someone doubts how it's useful. kdump+panic_on_oom=2 is the
> last tool to investigate what happens in oom-ed system. If a task is killed,
> the sysytem recovers and used memory were freed, there will be few hint
> to know what happnes. In mission critical system, oom should never happen.
> Then, investigation after OOM is very important.
> Then, panic_on_oom=2+kdump is useful to avoid next OOM by knowing
> precise information via snapshot.

No I don't doubt it is useful, and I think this probably is the simplest
and most useful semantic. So thanks for doing this.

I hate to pick nits in a trivial patch but I will anyway:


> TODO:
>  - For memcg, it's for isolate system's memory usage, oom-notiifer and
>    freeze_at_oom (or rest_at_oom) should be implemented. Then, management
>    daemon can do similar jobs (as kdump) in safer way or taking snapshot
>    per cgroup.
> 
> CC: Balbir Singh <balbir@linux.vnet.ibm.com>
> CC: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
> CC: David Rientjes <rientjes@google.com>
> Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> ---
>  Documentation/cgroups/memory.txt |    2 ++
>  Documentation/sysctl/vm.txt      |    5 ++++-
>  mm/oom_kill.c                    |    2 ++
>  3 files changed, 8 insertions(+), 1 deletion(-)
> 
> Index: mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/Documentation/cgroups/memory.txt
> +++ mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> @@ -182,6 +182,8 @@ list.
>  NOTE: Reclaim does not work for the root cgroup, since we cannot set any
>  limits on the root cgroup.
>  
> +Note2: When panic_on_oom is set to "2", the whole system will panic.
> +

Maybe:

NOTE2: When panic_on_oom is set to "2", the whole system will panic in
case of an oom event in any cgroup.

>  2. Locking
>  
>  The memory controller uses the following hierarchy
> Index: mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/Documentation/sysctl/vm.txt
> +++ mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> @@ -573,11 +573,14 @@ Because other nodes' memory may be free.
>  may be not fatal yet.
>  
>  If this is set to 2, the kernel panics compulsorily even on the
> -above-mentioned.
> +above-mentioned. Even oom happens under memoyr cgroup, the whole
> +system panics.
                                           memory

>  
>  The default value is 0.
>  1 and 2 are for failover of clustering. Please select either
>  according to your policy of failover.
> +2 seems too strong but panic_on_oom=2+kdump gives you very strong
> +tool to investigate a system which should never cause OOM.

I don't think you need say 2 seems too strong because as you rightfully
say, it has real uses. The hint about using it to investigate OOM
conditions is good though.

>  
>  =============================================================
>  
> Index: mmotm-2.6.33-Feb11/mm/oom_kill.c
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/mm/oom_kill.c
> +++ mmotm-2.6.33-Feb11/mm/oom_kill.c
> @@ -471,6 +471,8 @@ void mem_cgroup_out_of_memory(struct mem
>  	unsigned long points = 0;
>  	struct task_struct *p;
>  
> +	if (sysctl_panic_on_oom == 2)
> +		panic("out of memory(memcg). panic_on_oom is selected.\n");
>  	read_lock(&tasklist_lock);
>  retry:
>  	p = select_bad_process(&points, mem);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memcg: handle panic_on_oom=always case

[KAMEZAWA Hiroyuki]

On Wed, 17 Feb 2010 19:45:26 +1100
Nick Piggin <npiggin@suse.de> wrote:

> On Wed, Feb 17, 2010 at 03:04:45PM +0900, KAMEZAWA Hiroyuki wrote:
> > tested on mmotm-Feb11.
> > 
> > Balbir-san, Nishimura-san, I want review from both of you.
> > 
> > ==
> > 
> > From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> > 
> > Now, if panic_on_oom=2, the whole system panics even if the oom happend
> > in some special situation (as cpuset, mempolicy....).
> > Then, panic_on_oom=2 means painc_on_oom_always.
> > 
> > Now, memcg doesn't check panic_on_oom flag. This patch adds a check.
> > 
> > Maybe someone doubts how it's useful. kdump+panic_on_oom=2 is the
> > last tool to investigate what happens in oom-ed system. If a task is killed,
> > the sysytem recovers and used memory were freed, there will be few hint
> > to know what happnes. In mission critical system, oom should never happen.
> > Then, investigation after OOM is very important.
> > Then, panic_on_oom=2+kdump is useful to avoid next OOM by knowing
> > precise information via snapshot.
> 
> No I don't doubt it is useful, and I think this probably is the simplest
> and most useful semantic. So thanks for doing this.
> 
Thank you for review.


> I hate to pick nits in a trivial patch but I will anyway:
> 
> 
> > TODO:
> >  - For memcg, it's for isolate system's memory usage, oom-notiifer and
> >    freeze_at_oom (or rest_at_oom) should be implemented. Then, management
> >    daemon can do similar jobs (as kdump) in safer way or taking snapshot
> >    per cgroup.
> > 
> > CC: Balbir Singh <balbir@linux.vnet.ibm.com>
> > CC: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
> > CC: David Rientjes <rientjes@google.com>
> > Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> > ---
> >  Documentation/cgroups/memory.txt |    2 ++
> >  Documentation/sysctl/vm.txt      |    5 ++++-
> >  mm/oom_kill.c                    |    2 ++
> >  3 files changed, 8 insertions(+), 1 deletion(-)
> > 
> > Index: mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> > ===================================================================
> > --- mmotm-2.6.33-Feb11.orig/Documentation/cgroups/memory.txt
> > +++ mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> > @@ -182,6 +182,8 @@ list.
> >  NOTE: Reclaim does not work for the root cgroup, since we cannot set any
> >  limits on the root cgroup.
> >  
> > +Note2: When panic_on_oom is set to "2", the whole system will panic.
> > +
> 
> Maybe:
> 
> NOTE2: When panic_on_oom is set to "2", the whole system will panic in
> case of an oom event in any cgroup.
> 

ok.


> >  2. Locking
> >  
> >  The memory controller uses the following hierarchy
> > Index: mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> > ===================================================================
> > --- mmotm-2.6.33-Feb11.orig/Documentation/sysctl/vm.txt
> > +++ mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> > @@ -573,11 +573,14 @@ Because other nodes' memory may be free.
> >  may be not fatal yet.
> >  
> >  If this is set to 2, the kernel panics compulsorily even on the
> > -above-mentioned.
> > +above-mentioned. Even oom happens under memoyr cgroup, the whole
> > +system panics.
>                                            memory
> 
> >  
> >  The default value is 0.
> >  1 and 2 are for failover of clustering. Please select either
> >  according to your policy of failover.
> > +2 seems too strong but panic_on_oom=2+kdump gives you very strong
> > +tool to investigate a system which should never cause OOM.
> 
> I don't think you need say 2 seems too strong because as you rightfully
> say, it has real uses. The hint about using it to investigate OOM
> conditions is good though.
> 

ok. I'll update this patch.

Thanks,
-Kame

> >  
> >  =============================================================
> >  
> > Index: mmotm-2.6.33-Feb11/mm/oom_kill.c
> > ===================================================================
> > --- mmotm-2.6.33-Feb11.orig/mm/oom_kill.c
> > +++ mmotm-2.6.33-Feb11/mm/oom_kill.c
> > @@ -471,6 +471,8 @@ void mem_cgroup_out_of_memory(struct mem
> >  	unsigned long points = 0;
> >  	struct task_struct *p;
> >  
> > +	if (sysctl_panic_on_oom == 2)
> > +		panic("out of memory(memcg). panic_on_oom is selected.\n");
> >  	read_lock(&tasklist_lock);
> >  retry:
> >  	p = select_bad_process(&points, mem);
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH] memcg: handle panic_on_oom=always case v2

[KAMEZAWA Hiroyuki]

Documenation is updated.
==
From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

Now, if panic_on_oom=2, the whole system panics even if the oom happend
in some special situation (as cpuset, mempolicy....).
Then, panic_on_oom=2 means painc_on_oom_always.

Now, memcg doesn't check panic_on_oom flag. This patch adds a check.

BTW, how it's useful ?

kdump+panic_on_oom=2 is the last tool to investigate what happens in oom-ed
system. When a task is killed, the sysytem recovers and there will be few hint
to know what happnes. In mission critical system, oom should never happen.
Then, panic_on_oom=2+kdump is useful to avoid next OOM by knowing
precise information via snapshot.

TODO:
 - For memcg, it's for isolate system's memory usage, oom-notiifer and
   freeze_at_oom (or rest_at_oom) should be implemented. Then, management
   daemon can do similar jobs (as kdump) or taking snapshot per cgroup.

Changelg:
 - rewrote documentations.

CC: Balbir Singh <balbir@linux.vnet.ibm.com>
CC: David Rientjes <rientjes@google.com>
CC: Nick Piggin <npiggin@suse.de>
Reviewed-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
---
 Documentation/cgroups/memory.txt |    5 ++++-
 Documentation/sysctl/vm.txt      |    5 ++++-
 mm/oom_kill.c                    |    2 ++
 3 files changed, 10 insertions(+), 2 deletions(-)

Index: mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
===================================================================
--- mmotm-2.6.33-Feb11.orig/Documentation/cgroups/memory.txt
+++ mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
@@ -182,6 +182,8 @@ list.
 NOTE: Reclaim does not work for the root cgroup, since we cannot set any
 limits on the root cgroup.
 
+Note2: When panic_on_oom is set to "2", the whole system will panic.
+
 2. Locking
 
 The memory controller uses the following hierarchy
@@ -379,7 +381,8 @@ The feature can be disabled by
 NOTE1: Enabling/disabling will fail if the cgroup already has other
 cgroups created below it.
 
-NOTE2: This feature can be enabled/disabled per subtree.
+NOTE2: When panic_on_oom is set to "2", the whole system will panic in
+case of an oom event in any cgroup.
 
 7. Soft limits
 
Index: mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
===================================================================
--- mmotm-2.6.33-Feb11.orig/Documentation/sysctl/vm.txt
+++ mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
@@ -573,11 +573,14 @@ Because other nodes' memory may be free.
 may be not fatal yet.
 
 If this is set to 2, the kernel panics compulsorily even on the
-above-mentioned.
+above-mentioned. Even oom happens under memory cgroup, the whole
+system panics.
 
 The default value is 0.
 1 and 2 are for failover of clustering. Please select either
 according to your policy of failover.
+panic_on_oom=2+kdump gives you very strong tool to investigate
+why oom happens. You can get snapshot.
 
 =============================================================
 
Index: mmotm-2.6.33-Feb11/mm/oom_kill.c
===================================================================
--- mmotm-2.6.33-Feb11.orig/mm/oom_kill.c
+++ mmotm-2.6.33-Feb11/mm/oom_kill.c
@@ -471,6 +471,8 @@ void mem_cgroup_out_of_memory(struct mem
 	unsigned long points = 0;
 	struct task_struct *p;
 
+	if (sysctl_panic_on_oom == 2)
+		panic("out of memory(memcg). panic_on_oom is selected.\n");
 	read_lock(&tasklist_lock);
 retry:
 	p = select_bad_process(&points, mem);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>