Re: [PATCH V6 00/10] memcg: per cgroup background reclaim

[Johannes Weiner <hannes@cmpxchg.org>]

On Wed, Apr 27, 2011 at 10:41:47AM -0700, Ying Han wrote:
> On Wed, Apr 27, 2011 at 12:36 AM, Johannes Weiner <hannes@cmpxchg.org>wrote:
> 
> > On Fri, Apr 22, 2011 at 08:33:58PM -0700, Ying Han wrote:
> > > On Fri, Apr 22, 2011 at 7:34 PM, Johannes Weiner <hannes@cmpxchg.org>
> > wrote:
> > >
> > > > On Fri, Apr 22, 2011 at 07:10:25PM -0700, Ying Han wrote: >
> > > > However, i still think there is a need from the admin to have some
> > > > controls > of which memcg to do background reclaim proactively
> > > > (before global memory > pressure) and that was the initial logic
> > > > behind the API.
> > > >
> > > > That sounds more interesting.  Do you have a specific use case
> > > > that requires this?
> > >
> > > There might be more interesting use cases there, and here is one I
> > > can think of:
> > >
> > > let's say we three jobs A, B and C, and one host with 32G of RAM. We
> > > configure each job's hard_limit as their peak memory usage.
> > > A: 16G
> > > B: 16G
> > > C: 10G
> > >
> > > 1. we start running A with hard_limit 15G, and start running B with
> > > hard_limit 15G.
> > > 2. we set A and B's soft_limit based on their "hot" memory. Let's say
> > > setting A's soft_limit 10G and B's soft_limit 10G.
> > > (The soft_limit will be changing based on their runtime memory usage)
> > >
> > > If no more jobs running on the system, A and B will easily fill up the
> > whole
> > > system with pagecache pages. Since we are not over-committing the machine
> > > with their hard_limit, there will be no pressure to push their memory
> > usage
> > > down to soft_limit.
> > >
> > > Now we would like to launch another job C, since we know there are A(16G
> > -
> > > 10G) + B(16G - 10G)  = 12G "cold" memory can be reclaimed (w/o impacting
> > the
> > > A and B's performance). So what will happen
> > >
> > > 1. start running C on the host, which triggers global memory pressure
> > right
> > > away. If the reclaim is fast, C start growing with the free pages from A
> > and
> > > B.
> > >
> > > However, it might be possible that the reclaim can not catch-up with the
> > > job's page allocation. We end up with either OOM condition or performance
> > > spike on any of the running jobs.
> >
> > If background reclaim can not catch up, C will go into direct reclaim,
> > which will have exactly the same effect, only that C will have to do
> > the work itself.
> >
> > > One way to improve it is to set a wmark on either A/B to be proactively
> > > reclaiming pages before launching C. The global memory pressure won't
> > help
> > > much here since we won't trigger that.
> >
> > Ok, so you want to use the watermarks to push back and limit the usage
> > of A and B to make room for C.  Isn't this exactly what the hard limit
> > is for?
> 
> similar, but not exactly the same. there is no need to hard cap the memory
> usage for A and B in that case.
> what we need is to have some period of time that A and B slowly reclaim
> pages and leaves some room to
> launch C smoothly.

I think we are going in circles now.

Since starting with C the machine is overcommitted, the problem is no
longer memcg-internal latency but latency of global memory scarcity.

My suggestion to that was, and still is, to fix global background
reclaim, which should apply pressure equally to all memcgs until the
_global_ watermarks are met again.

This would do the right thing for this case: C starts up, the global
watermark is breached sooner or later and background reclaim will push
back A and B, hopefully before anyone has to go into direct
reclaim. ('Hopefully' because the allocations may still happen faster
than background reclaim can keep up freeing pages.  But this applies
to your scenario as well.)

I think this should work out of the box, without tweaking obscure
knobs from userspace.

Anyway, at this point I can only repeat myself, so I will shut up now.

	Hannes

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>