From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yh0-f48.google.com (mail-yh0-f48.google.com [209.85.213.48]) by kanga.kvack.org (Postfix) with ESMTP id E9F636B00B2 for ; Sat, 7 Dec 2013 11:38:22 -0500 (EST) Received: by mail-yh0-f48.google.com with SMTP id f73so1424651yha.21 for ; Sat, 07 Dec 2013 08:38:22 -0800 (PST) Received: from mail-pb0-x234.google.com (mail-pb0-x234.google.com [2607:f8b0:400e:c01::234]) by mx.google.com with ESMTPS id r49si2623469yho.267.2013.12.07.08.38.21 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 07 Dec 2013 08:38:21 -0800 (PST) Received: by mail-pb0-f52.google.com with SMTP id uo5so2825417pbc.11 for ; Sat, 07 Dec 2013 08:38:20 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <20131206173438.GE21724@cmpxchg.org> References: <20131120152251.GA18809@dhcp22.suse.cz> <20131128115458.GK2761@dhcp22.suse.cz> <20131204054533.GZ3556@cmpxchg.org> <20131205025026.GA26777@htj.dyndns.org> <20131206173438.GE21724@cmpxchg.org> Date: Sat, 7 Dec 2013 08:38:20 -0800 Message-ID: Subject: Re: [patch 7/8] mm, memcg: allow processes handling oom notifications to access reserves From: Tim Hockin Content-Type: multipart/alternative; boundary=bcaec520ea03f803a304ecf46301 Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Michal Hocko , Li Zefan , KAMEZAWA Hiroyuki , Tejun Heo , Christoph Lameter , David Rientjes , linux-mm@kvack.org, Rik van Riel , Pekka Enberg , cgroups@vger.kernel.org, Mel Gorman , Andrew Morton , linux-kernel@vger.kernel.org --bcaec520ea03f803a304ecf46301 Content-Type: text/plain; charset=UTF-8 We actually started with kernel patches all h these lines - per-memcg scores and all of our crazy policy requirements. It turns out that changing policies is hard. When David offered the opportunity to manage it all in user space it sounded like a great idea. If this can be made to work as a high prio daemon with access to reserves, we would like it. Tim On Dec 6, 2013 9:36 AM, "Johannes Weiner" wrote: > On Thu, Dec 05, 2013 at 03:49:57PM -0800, David Rientjes wrote: > > On Wed, 4 Dec 2013, Tejun Heo wrote: > > > > > Hello, > > > > > > > Tejun, how are you? > > > > > Umm.. without delving into details, aren't you basically creating a > > > memory cgroup inside a memory cgroup? Doesn't sound like a > > > particularly well thought-out plan to me. > > > > > > > I agree that we wouldn't need such support if we are only addressing > memcg > > oom conditions. We could do things like A/memory.limit_in_bytes == 128M > > and A/b/memory.limit_in_bytes == 126MB and then attach the process > waiting > > on A/b/memory.oom_control to A and that would work perfect. > > > > However, we also need to discuss system oom handling. We have an > interest > > in being able to allow userspace to handle system oom conditions since > the > > policy will differ depending on machine and we can't encode every > possible > > mechanism into the kernel. For example, on system oom we want to kill a > > process from the lowest priority top-level memcg. We lack that ability > > entirely in the kernel and since the sum of our top-level memcgs > > memory.limit_in_bytes exceeds the amount of present RAM, we run into > these > > oom conditions a _lot_. > > A simple and natural solution to this is to have the global OOM killer > respect cgroups. You go through all the effort of carefully grouping > tasks into bigger entities that you then arrange hierarchically. The > global OOM killer should not just treat all tasks as equal peers. > > We can add a per-cgroup OOM priority knob and have the global OOM > handler pick victim tasks from the one or more groups that have the > lowest priority. > > Out of the box, every cgroup has the same priority, which means we can > add this feature without changing the default behavior. > > > So the first step, in my opinion, is to add a system oom notification on > > the root memcg's memory.oom_control which currently allows registering an > > eventfd() notification but never actually triggers. I did that in a > patch > > and it is was merged into -mm but was pulled out for later discussion. > > > > Then, we need to ensure that the userspace that is registered to handle > > such events and that is difficult to do when the system is oom. The > > proposal is to allow such processes, now marked as PF_OOM_HANDLER, to be > > able to access pre-defined per-zone memory reserves in the page > allocator. > > The only special handling for PF_OOM_HANDLER in the page allocator itself > > would be under such oom conditions (memcg oom conditions have no problem > > allocating the memory, only charging it). The amount of reserves would > be > > defined as memory.oom_reserve_in_bytes from within the root memcg as > > defined by this patch, i.e. allow this amount of memory to be allocated > in > > the page allocator for PF_OOM_HANDLER below the per-zone min watermarks. > > > > This, I believe, is the cleanest interface for users who choose to use a > > non-default policy by setting memory.oom_reserve_in_bytes and constrains > > all of the code to memcg which you have to configure for such support. > > > > The system oom condition is not addressed in this patch series, although > > the PF_OOM_HANDLER bit can be used for that purpose. I didn't post that > > patch because the notification on the root memcg's memory.oom_control in > > such conditions is currently being debated, so we need to solve that > issue > > first. > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > --bcaec520ea03f803a304ecf46301 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

We actually started with kernel patches all h these lines - = per-memcg scores and all of our crazy policy requirements.

It turns out that changing policies is hard.

When David offered the opportunity to manage it all in user = space it sounded like a great idea.

If this can be made to work as a high prio daemon with acces= s to reserves, we would like it.

Tim

On Dec 6, 2013 9:36 AM, "Johannes Weiner&qu= ot; <hannes@cmpxchg.org> wr= ote:
On Thu, Dec 05, 2013 at 03:49:57PM -0800, David Rientjes wrote:
> On Wed, 4 Dec 2013, Tejun Heo wrote:
>
> > Hello,
> >
>
> Tejun, how are you?
>
> > Umm.. without delving into details, aren't you basically crea= ting a
> > memory cgroup inside a memory cgroup? =C2=A0Doesn't sound lik= e a
> > particularly well thought-out plan to me.
> >
>
> I agree that we wouldn't need such support if we are only addressi= ng memcg
> oom conditions. =C2=A0We could do things like A/memory.limit_in_bytes = =3D=3D 128M
> and A/b/memory.limit_in_bytes =3D=3D 126MB and then attach the process= waiting
> on A/b/memory.oom_control to A and that would work perfect.
>
> However, we also need to discuss system oom handling. =C2=A0We have an= interest
> in being able to allow userspace to handle system oom conditions since= the
> policy will differ depending on machine and we can't encode every = possible
> mechanism into the kernel. =C2=A0For example, on system oom we want to= kill a
> process from the lowest priority top-level memcg. =C2=A0We lack that a= bility
> entirely in the kernel and since the sum of our top-level memcgs
> memory.limit_in_bytes exceeds the amount of present RAM, we run into t= hese
> oom conditions a _lot_.

A simple and natural solution to this is to have the global OOM killer
respect cgroups. =C2=A0You go through all the effort of carefully grouping<= br> tasks into bigger entities that you then arrange hierarchically. =C2=A0The<= br> global OOM killer should not just treat all tasks as equal peers.

We can add a per-cgroup OOM priority knob and have the global OOM
handler pick victim tasks from the one or more groups that have the
lowest priority.

Out of the box, every cgroup has the same priority, which means we can
add this feature without changing the default behavior.

> So the first step, in my opinion, is to add a system oom notification = on
> the root memcg's memory.oom_control which currently allows registe= ring an
> eventfd() notification but never actually triggers. =C2=A0I did that i= n a patch
> and it is was merged into -mm but was pulled out for later discussion.=
>
> Then, we need to ensure that the userspace that is registered to handl= e
> such events and that is difficult to do when the system is oom. =C2=A0= The
> proposal is to allow such processes, now marked as PF_OOM_HANDLER, to = be
> able to access pre-defined per-zone memory reserves in the page alloca= tor.
> The only special handling for PF_OOM_HANDLER in the page allocator its= elf
> would be under such oom conditions (memcg oom conditions have no probl= em
> allocating the memory, only charging it). =C2=A0The amount of reserves= would be
> defined as memory.oom_reserve_in_bytes from within the root memcg as > defined by this patch, i.e. allow this amount of memory to be allocate= d in
> the page allocator for PF_OOM_HANDLER below the per-zone min watermark= s.
>
> This, I believe, is the cleanest interface for users who choose to use= a
> non-default policy by setting memory.oom_reserve_in_bytes and constrai= ns
> all of the code to memcg which you have to configure for such support.=
>
> The system oom condition is not addressed in this patch series, althou= gh
> the PF_OOM_HANDLER bit can be used for that purpose. =C2=A0I didn'= t post that
> patch because the notification on the root memcg's memory.oom_cont= rol in
> such conditions is currently being debated, so we need to solve that i= ssue
> first.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel= " in
the body of a message to major= domo@vger.kernel.org
More majordomo info at =C2=A0http://vger.kernel.org/majordomo-info.html Please read the FAQ at =C2=A0http://www.tux.org/lkml/
--bcaec520ea03f803a304ecf46301-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org