* Re: [PATCH] memcg: handle panic_on_oom=always case
2010-02-17 6:04 [PATCH] memcg: handle panic_on_oom=always case KAMEZAWA Hiroyuki
@ 2010-02-17 6:50 ` Daisuke Nishimura
2010-02-17 8:45 ` Nick Piggin
2010-02-17 9:04 ` [PATCH] memcg: handle panic_on_oom=always case v2 KAMEZAWA Hiroyuki
2 siblings, 0 replies; 5+ messages in thread
From: Daisuke Nishimura @ 2010-02-17 6:50 UTC (permalink / raw)
To: KAMEZAWA Hiroyuki
Cc: linux-mm, linux-kernel, balbir, rientjes, akpm, npiggin,
Daisuke Nishimura
On Wed, 17 Feb 2010 15:04:45 +0900, KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> tested on mmotm-Feb11.
>
> Balbir-san, Nishimura-san, I want review from both of you.
>
I've read only part of the original patch set yet, but I agree to the direction
of making memcg's oom panic the system on panic_on_oom==2, not panic on panic_on_oom==1.
Reviewed-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
Thanks,
Daisuke Nishimura.
> ==
>
> From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
>
> Now, if panic_on_oom=2, the whole system panics even if the oom happend
> in some special situation (as cpuset, mempolicy....).
> Then, panic_on_oom=2 means painc_on_oom_always.
>
> Now, memcg doesn't check panic_on_oom flag. This patch adds a check.
>
> Maybe someone doubts how it's useful. kdump+panic_on_oom=2 is the
> last tool to investigate what happens in oom-ed system. If a task is killed,
> the sysytem recovers and used memory were freed, there will be few hint
> to know what happnes. In mission critical system, oom should never happen.
> Then, investigation after OOM is very important.
> Then, panic_on_oom=2+kdump is useful to avoid next OOM by knowing
> precise information via snapshot.
>
> TODO:
> - For memcg, it's for isolate system's memory usage, oom-notiifer and
> freeze_at_oom (or rest_at_oom) should be implemented. Then, management
> daemon can do similar jobs (as kdump) in safer way or taking snapshot
> per cgroup.
>
> CC: Balbir Singh <balbir@linux.vnet.ibm.com>
> CC: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
> CC: David Rientjes <rientjes@google.com>
> Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> ---
> Documentation/cgroups/memory.txt | 2 ++
> Documentation/sysctl/vm.txt | 5 ++++-
> mm/oom_kill.c | 2 ++
> 3 files changed, 8 insertions(+), 1 deletion(-)
>
> Index: mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/Documentation/cgroups/memory.txt
> +++ mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> @@ -182,6 +182,8 @@ list.
> NOTE: Reclaim does not work for the root cgroup, since we cannot set any
> limits on the root cgroup.
>
> +Note2: When panic_on_oom is set to "2", the whole system will panic.
> +
> 2. Locking
>
> The memory controller uses the following hierarchy
> Index: mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/Documentation/sysctl/vm.txt
> +++ mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> @@ -573,11 +573,14 @@ Because other nodes' memory may be free.
> may be not fatal yet.
>
> If this is set to 2, the kernel panics compulsorily even on the
> -above-mentioned.
> +above-mentioned. Even oom happens under memoyr cgroup, the whole
> +system panics.
>
> The default value is 0.
> 1 and 2 are for failover of clustering. Please select either
> according to your policy of failover.
> +2 seems too strong but panic_on_oom=2+kdump gives you very strong
> +tool to investigate a system which should never cause OOM.
>
> =============================================================
>
> Index: mmotm-2.6.33-Feb11/mm/oom_kill.c
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/mm/oom_kill.c
> +++ mmotm-2.6.33-Feb11/mm/oom_kill.c
> @@ -471,6 +471,8 @@ void mem_cgroup_out_of_memory(struct mem
> unsigned long points = 0;
> struct task_struct *p;
>
> + if (sysctl_panic_on_oom == 2)
> + panic("out of memory(memcg). panic_on_oom is selected.\n");
> read_lock(&tasklist_lock);
> retry:
> p = select_bad_process(&points, mem);
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH] memcg: handle panic_on_oom=always case
2010-02-17 6:04 [PATCH] memcg: handle panic_on_oom=always case KAMEZAWA Hiroyuki
2010-02-17 6:50 ` Daisuke Nishimura
@ 2010-02-17 8:45 ` Nick Piggin
2010-02-17 8:51 ` KAMEZAWA Hiroyuki
2010-02-17 9:04 ` [PATCH] memcg: handle panic_on_oom=always case v2 KAMEZAWA Hiroyuki
2 siblings, 1 reply; 5+ messages in thread
From: Nick Piggin @ 2010-02-17 8:45 UTC (permalink / raw)
To: KAMEZAWA Hiroyuki
Cc: linux-mm, linux-kernel, balbir, nishimura, rientjes, akpm
On Wed, Feb 17, 2010 at 03:04:45PM +0900, KAMEZAWA Hiroyuki wrote:
> tested on mmotm-Feb11.
>
> Balbir-san, Nishimura-san, I want review from both of you.
>
> ==
>
> From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
>
> Now, if panic_on_oom=2, the whole system panics even if the oom happend
> in some special situation (as cpuset, mempolicy....).
> Then, panic_on_oom=2 means painc_on_oom_always.
>
> Now, memcg doesn't check panic_on_oom flag. This patch adds a check.
>
> Maybe someone doubts how it's useful. kdump+panic_on_oom=2 is the
> last tool to investigate what happens in oom-ed system. If a task is killed,
> the sysytem recovers and used memory were freed, there will be few hint
> to know what happnes. In mission critical system, oom should never happen.
> Then, investigation after OOM is very important.
> Then, panic_on_oom=2+kdump is useful to avoid next OOM by knowing
> precise information via snapshot.
No I don't doubt it is useful, and I think this probably is the simplest
and most useful semantic. So thanks for doing this.
I hate to pick nits in a trivial patch but I will anyway:
> TODO:
> - For memcg, it's for isolate system's memory usage, oom-notiifer and
> freeze_at_oom (or rest_at_oom) should be implemented. Then, management
> daemon can do similar jobs (as kdump) in safer way or taking snapshot
> per cgroup.
>
> CC: Balbir Singh <balbir@linux.vnet.ibm.com>
> CC: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
> CC: David Rientjes <rientjes@google.com>
> Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> ---
> Documentation/cgroups/memory.txt | 2 ++
> Documentation/sysctl/vm.txt | 5 ++++-
> mm/oom_kill.c | 2 ++
> 3 files changed, 8 insertions(+), 1 deletion(-)
>
> Index: mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/Documentation/cgroups/memory.txt
> +++ mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> @@ -182,6 +182,8 @@ list.
> NOTE: Reclaim does not work for the root cgroup, since we cannot set any
> limits on the root cgroup.
>
> +Note2: When panic_on_oom is set to "2", the whole system will panic.
> +
Maybe:
NOTE2: When panic_on_oom is set to "2", the whole system will panic in
case of an oom event in any cgroup.
> 2. Locking
>
> The memory controller uses the following hierarchy
> Index: mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/Documentation/sysctl/vm.txt
> +++ mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> @@ -573,11 +573,14 @@ Because other nodes' memory may be free.
> may be not fatal yet.
>
> If this is set to 2, the kernel panics compulsorily even on the
> -above-mentioned.
> +above-mentioned. Even oom happens under memoyr cgroup, the whole
> +system panics.
memory
>
> The default value is 0.
> 1 and 2 are for failover of clustering. Please select either
> according to your policy of failover.
> +2 seems too strong but panic_on_oom=2+kdump gives you very strong
> +tool to investigate a system which should never cause OOM.
I don't think you need say 2 seems too strong because as you rightfully
say, it has real uses. The hint about using it to investigate OOM
conditions is good though.
>
> =============================================================
>
> Index: mmotm-2.6.33-Feb11/mm/oom_kill.c
> ===================================================================
> --- mmotm-2.6.33-Feb11.orig/mm/oom_kill.c
> +++ mmotm-2.6.33-Feb11/mm/oom_kill.c
> @@ -471,6 +471,8 @@ void mem_cgroup_out_of_memory(struct mem
> unsigned long points = 0;
> struct task_struct *p;
>
> + if (sysctl_panic_on_oom == 2)
> + panic("out of memory(memcg). panic_on_oom is selected.\n");
> read_lock(&tasklist_lock);
> retry:
> p = select_bad_process(&points, mem);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH] memcg: handle panic_on_oom=always case
2010-02-17 8:45 ` Nick Piggin
@ 2010-02-17 8:51 ` KAMEZAWA Hiroyuki
0 siblings, 0 replies; 5+ messages in thread
From: KAMEZAWA Hiroyuki @ 2010-02-17 8:51 UTC (permalink / raw)
To: Nick Piggin; +Cc: linux-mm, linux-kernel, balbir, nishimura, rientjes, akpm
On Wed, 17 Feb 2010 19:45:26 +1100
Nick Piggin <npiggin@suse.de> wrote:
> On Wed, Feb 17, 2010 at 03:04:45PM +0900, KAMEZAWA Hiroyuki wrote:
> > tested on mmotm-Feb11.
> >
> > Balbir-san, Nishimura-san, I want review from both of you.
> >
> > ==
> >
> > From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> >
> > Now, if panic_on_oom=2, the whole system panics even if the oom happend
> > in some special situation (as cpuset, mempolicy....).
> > Then, panic_on_oom=2 means painc_on_oom_always.
> >
> > Now, memcg doesn't check panic_on_oom flag. This patch adds a check.
> >
> > Maybe someone doubts how it's useful. kdump+panic_on_oom=2 is the
> > last tool to investigate what happens in oom-ed system. If a task is killed,
> > the sysytem recovers and used memory were freed, there will be few hint
> > to know what happnes. In mission critical system, oom should never happen.
> > Then, investigation after OOM is very important.
> > Then, panic_on_oom=2+kdump is useful to avoid next OOM by knowing
> > precise information via snapshot.
>
> No I don't doubt it is useful, and I think this probably is the simplest
> and most useful semantic. So thanks for doing this.
>
Thank you for review.
> I hate to pick nits in a trivial patch but I will anyway:
>
>
> > TODO:
> > - For memcg, it's for isolate system's memory usage, oom-notiifer and
> > freeze_at_oom (or rest_at_oom) should be implemented. Then, management
> > daemon can do similar jobs (as kdump) in safer way or taking snapshot
> > per cgroup.
> >
> > CC: Balbir Singh <balbir@linux.vnet.ibm.com>
> > CC: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
> > CC: David Rientjes <rientjes@google.com>
> > Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> > ---
> > Documentation/cgroups/memory.txt | 2 ++
> > Documentation/sysctl/vm.txt | 5 ++++-
> > mm/oom_kill.c | 2 ++
> > 3 files changed, 8 insertions(+), 1 deletion(-)
> >
> > Index: mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> > ===================================================================
> > --- mmotm-2.6.33-Feb11.orig/Documentation/cgroups/memory.txt
> > +++ mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
> > @@ -182,6 +182,8 @@ list.
> > NOTE: Reclaim does not work for the root cgroup, since we cannot set any
> > limits on the root cgroup.
> >
> > +Note2: When panic_on_oom is set to "2", the whole system will panic.
> > +
>
> Maybe:
>
> NOTE2: When panic_on_oom is set to "2", the whole system will panic in
> case of an oom event in any cgroup.
>
ok.
> > 2. Locking
> >
> > The memory controller uses the following hierarchy
> > Index: mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> > ===================================================================
> > --- mmotm-2.6.33-Feb11.orig/Documentation/sysctl/vm.txt
> > +++ mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
> > @@ -573,11 +573,14 @@ Because other nodes' memory may be free.
> > may be not fatal yet.
> >
> > If this is set to 2, the kernel panics compulsorily even on the
> > -above-mentioned.
> > +above-mentioned. Even oom happens under memoyr cgroup, the whole
> > +system panics.
> memory
>
> >
> > The default value is 0.
> > 1 and 2 are for failover of clustering. Please select either
> > according to your policy of failover.
> > +2 seems too strong but panic_on_oom=2+kdump gives you very strong
> > +tool to investigate a system which should never cause OOM.
>
> I don't think you need say 2 seems too strong because as you rightfully
> say, it has real uses. The hint about using it to investigate OOM
> conditions is good though.
>
ok. I'll update this patch.
Thanks,
-Kame
> >
> > =============================================================
> >
> > Index: mmotm-2.6.33-Feb11/mm/oom_kill.c
> > ===================================================================
> > --- mmotm-2.6.33-Feb11.orig/mm/oom_kill.c
> > +++ mmotm-2.6.33-Feb11/mm/oom_kill.c
> > @@ -471,6 +471,8 @@ void mem_cgroup_out_of_memory(struct mem
> > unsigned long points = 0;
> > struct task_struct *p;
> >
> > + if (sysctl_panic_on_oom == 2)
> > + panic("out of memory(memcg). panic_on_oom is selected.\n");
> > read_lock(&tasklist_lock);
> > retry:
> > p = select_bad_process(&points, mem);
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH] memcg: handle panic_on_oom=always case v2
2010-02-17 6:04 [PATCH] memcg: handle panic_on_oom=always case KAMEZAWA Hiroyuki
2010-02-17 6:50 ` Daisuke Nishimura
2010-02-17 8:45 ` Nick Piggin
@ 2010-02-17 9:04 ` KAMEZAWA Hiroyuki
2 siblings, 0 replies; 5+ messages in thread
From: KAMEZAWA Hiroyuki @ 2010-02-17 9:04 UTC (permalink / raw)
To: KAMEZAWA Hiroyuki
Cc: linux-mm, linux-kernel, balbir, nishimura, rientjes, akpm, npiggin
Documenation is updated.
==
From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Now, if panic_on_oom=2, the whole system panics even if the oom happend
in some special situation (as cpuset, mempolicy....).
Then, panic_on_oom=2 means painc_on_oom_always.
Now, memcg doesn't check panic_on_oom flag. This patch adds a check.
BTW, how it's useful ?
kdump+panic_on_oom=2 is the last tool to investigate what happens in oom-ed
system. When a task is killed, the sysytem recovers and there will be few hint
to know what happnes. In mission critical system, oom should never happen.
Then, panic_on_oom=2+kdump is useful to avoid next OOM by knowing
precise information via snapshot.
TODO:
- For memcg, it's for isolate system's memory usage, oom-notiifer and
freeze_at_oom (or rest_at_oom) should be implemented. Then, management
daemon can do similar jobs (as kdump) or taking snapshot per cgroup.
Changelg:
- rewrote documentations.
CC: Balbir Singh <balbir@linux.vnet.ibm.com>
CC: David Rientjes <rientjes@google.com>
CC: Nick Piggin <npiggin@suse.de>
Reviewed-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
---
Documentation/cgroups/memory.txt | 5 ++++-
Documentation/sysctl/vm.txt | 5 ++++-
mm/oom_kill.c | 2 ++
3 files changed, 10 insertions(+), 2 deletions(-)
Index: mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
===================================================================
--- mmotm-2.6.33-Feb11.orig/Documentation/cgroups/memory.txt
+++ mmotm-2.6.33-Feb11/Documentation/cgroups/memory.txt
@@ -182,6 +182,8 @@ list.
NOTE: Reclaim does not work for the root cgroup, since we cannot set any
limits on the root cgroup.
+Note2: When panic_on_oom is set to "2", the whole system will panic.
+
2. Locking
The memory controller uses the following hierarchy
@@ -379,7 +381,8 @@ The feature can be disabled by
NOTE1: Enabling/disabling will fail if the cgroup already has other
cgroups created below it.
-NOTE2: This feature can be enabled/disabled per subtree.
+NOTE2: When panic_on_oom is set to "2", the whole system will panic in
+case of an oom event in any cgroup.
7. Soft limits
Index: mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
===================================================================
--- mmotm-2.6.33-Feb11.orig/Documentation/sysctl/vm.txt
+++ mmotm-2.6.33-Feb11/Documentation/sysctl/vm.txt
@@ -573,11 +573,14 @@ Because other nodes' memory may be free.
may be not fatal yet.
If this is set to 2, the kernel panics compulsorily even on the
-above-mentioned.
+above-mentioned. Even oom happens under memory cgroup, the whole
+system panics.
The default value is 0.
1 and 2 are for failover of clustering. Please select either
according to your policy of failover.
+panic_on_oom=2+kdump gives you very strong tool to investigate
+why oom happens. You can get snapshot.
=============================================================
Index: mmotm-2.6.33-Feb11/mm/oom_kill.c
===================================================================
--- mmotm-2.6.33-Feb11.orig/mm/oom_kill.c
+++ mmotm-2.6.33-Feb11/mm/oom_kill.c
@@ -471,6 +471,8 @@ void mem_cgroup_out_of_memory(struct mem
unsigned long points = 0;
struct task_struct *p;
+ if (sysctl_panic_on_oom == 2)
+ panic("out of memory(memcg). panic_on_oom is selected.\n");
read_lock(&tasklist_lock);
retry:
p = select_bad_process(&points, mem);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 5+ messages in thread