Re: [patch 2/2] mm: memcg: hierarchical soft limit reclaim

[Michal Hocko <mhocko@suse.cz>]

On Tue 10-01-12 16:02:52, Johannes Weiner wrote:
> Right now, memcg soft limits are implemented by having a sorted tree
> of memcgs that are in excess of their limits.  Under global memory
> pressure, kswapd first reclaims from the biggest excessor and then
> proceeds to do regular global reclaim.  The result of this is that
> pages are reclaimed from all memcgs, but more scanning happens against
> those above their soft limit.
> 
> With global reclaim doing memcg-aware hierarchical reclaim by default,
> this is a lot easier to implement: everytime a memcg is reclaimed
> from, scan more aggressively (per tradition with a priority of 0) if
> it's above its soft limit.  With the same end result of scanning
> everybody, but soft limit excessors a bit more.
> 
> Advantages:
> 
>   o smoother reclaim: soft limit reclaim is a separate stage before
>     global reclaim, whose result is not communicated down the line and
>     so overreclaim of the groups in excess is very likely.  After this
>     patch, soft limit reclaim is fully integrated into regular reclaim
>     and each memcg is considered exactly once per cycle.
> 
>   o true hierarchy support: soft limits are only considered when
>     kswapd does global reclaim, but after this patch, targetted
>     reclaim of a memcg will mind the soft limit settings of its child
>     groups.

Yes it makes sense. At first I was thinking that soft limit should be
considered only under global mem. pressure (at least documentation says
so) but now it makes sense.
We can push on over-soft limit groups more because they told us they
could sacrifice something...  Anyway documentation needs an update as
well.

But we have to be little bit careful here. I am still quite confuses how
we should handle hierarchies vs. subtrees. See bellow.

> 
>   o code size: soft limit reclaim requires a lot of code to maintain
>     the per-node per-zone rb-trees to quickly find the biggest
>     offender, dedicated paths for soft limit reclaim etc. while this
>     new implementation gets away without all that.

on my i386 pae setup (including swap extension enabled):
Before
   text    data     bss     dec     hex filename
    310086   29970   35372  375428   5ba84 mm/built-in.o
After
size mm/built-in.o 
   text    data     bss     dec     hex filename
    309048   30030   35372  374450   5b6b2 mm/built-in.o

I would expect a bigger difference but still good.

> Test:

Will look into results later.
[...]

> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
> ---
>  include/linux/memcontrol.h |   18 +--
>  mm/memcontrol.c            |  412 ++++----------------------------------------
>  mm/vmscan.c                |   80 +--------
>  3 files changed, 48 insertions(+), 462 deletions(-)

Really nice to see

[...]
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 170dff4..d4f7ae5 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
[...]
> @@ -1318,6 +1123,36 @@ static unsigned long mem_cgroup_margin(struct mem_cgroup *memcg)
>  	return margin >> PAGE_SHIFT;
>  }
>  
> +/**
> + * mem_cgroup_over_softlimit
> + * @root: hierarchy root
> + * @memcg: child of @root to test
> + *
> + * Returns %true if @memcg exceeds its own soft limit or contributes
> + * to the soft limit excess of one of its parents up to and including
> + * @root.
> + */
> +bool mem_cgroup_over_softlimit(struct mem_cgroup *root,
> +			       struct mem_cgroup *memcg)
> +{
> +	if (mem_cgroup_disabled())
> +		return false;
> +
> +	if (!root)
> +		root = root_mem_cgroup;
> +
> +	for (; memcg; memcg = parent_mem_cgroup(memcg)) {
> +		/* root_mem_cgroup does not have a soft limit */
> +		if (memcg == root_mem_cgroup)
> +			break;
> +		if (res_counter_soft_limit_excess(&memcg->res))
> +			return true;
> +		if (memcg == root)
> +			break;
> +	}
> +	return false;
> +}

Well, this might be little bit tricky. We do not check whether memcg and
root are in a hierarchy (in terms of use_hierarchy) relation. 

If we are under global reclaim then we iterate over all memcgs and so
there is no guarantee that there is a hierarchical relation between the
given memcg and its parent. While, on the other hand, if we are doing
memcg reclaim then we have this guarantee.

Why should we punish a group (subtree) which is perfectly under its soft
limit just because some other subtree contributes to the common parent's
usage and makes it over its limit?
Should we check memcg->use_hierarchy here?

Does it even makes sense to setup soft limit on a parent group without
hierarchies?
Well I have to admit that hierarchies makes me headache.

> +
>  int mem_cgroup_swappiness(struct mem_cgroup *memcg)
>  {
>  	struct cgroup *cgrp = memcg->css.cgroup;
[...]
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index e3fd8a7..4279549 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -2121,8 +2121,16 @@ static void shrink_zone(int priority, struct zone *zone,
>  			.mem_cgroup = memcg,
>  			.zone = zone,
>  		};
> +		int epriority = priority;
> +		/*
> +		 * Put more pressure on hierarchies that exceed their
> +		 * soft limit, to push them back harder than their
> +		 * well-behaving siblings.
> +		 */
> +		if (mem_cgroup_over_softlimit(root, memcg))
> +			epriority = 0;

This sounds too aggressive to me. Shouldn't we just double the pressure
or something like that?
Previously we always had nr_to_reclaim == SWAP_CLUSTER_MAX when we did
memcg reclaim but this is not the case now. For the kswapd we have
nr_to_reclaim == ULONG_MAX so we will not break out of the reclaim early
and we have to scan a lot.
Direct reclaim (shrink or hard limit) shouldn't be affected here.

>  
> -		shrink_mem_cgroup_zone(priority, &mz, sc);
> +		shrink_mem_cgroup_zone(epriority, &mz, sc);
>  
>  		mem_cgroup_account_reclaim(root, memcg,
>  					   sc->nr_reclaimed - nr_reclaimed,

-- 
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>