Re: [LSF/MM TOPIC] dying memory cgroups and slab reclaim issues

[Dave Chinner <david@fromorbit.com>]

On Thu, Feb 28, 2019 at 08:30:49PM +0000, Roman Gushchin wrote:
> On Thu, Feb 21, 2019 at 02:46:17PM -0800, Roman Gushchin wrote:
> > On Wed, Feb 20, 2019 at 06:27:07PM +1100, Dave Chinner wrote:
> > > On Wed, Feb 20, 2019 at 04:50:31PM +1100, Dave Chinner wrote:
> > > > I'm just going to fix the original regression in the shrinker
> > > > algorithm by restoring the gradual accumulation behaviour, and this
> > > > whole series of problems can be put to bed.
> > > 
> > > Something like this lightly smoke tested patch below. It may be
> > > slightly more agressive than the original code for really small
> > > freeable values (i.e. < 100) but otherwise should be roughly
> > > equivalent to historic accumulation behaviour.
> > > 
> > > Cheers,
> > > 
> > > Dave.
> > > -- 
> > > Dave Chinner
> > > david@fromorbit.com
> > > 
> > > mm: fix shrinker scan accumulation regression
> > > 
> > > From: Dave Chinner <dchinner@redhat.com>
> > 
> > JFYI: I'm testing this patch in our environment for fixing
> > the memcg memory leak.
> > 
> > It will take a couple of days to get reliable results.
> > 
> 
> So unfortunately the proposed patch is not solving the dying memcg reclaim
> issue. I've tested it as is, with s/ilog2()/fls(), suggested by Johannes,
> and also with more a aggressive zero-seek slabs reclaim (always scanning
> at least SHRINK_BATCH for zero-seeks shrinkers).

Which makes sense if it's inodes and/or dentries shared across
multiple memcgs and actively referenced by non-owner memcgs that
prevent dying memcg reclaim. i.e. the shrinkers will not reclaim
frequently referenced objects unless there is extreme memory
pressure put on them.

> In all cases the number
> of outstanding memory cgroups grew almost linearly with time and didn't show
> any signs of plateauing.

What happend to the amount of memory pinned by those dying memcgs?
Did that change in any way? Did the rate of reclaim of objects
referencing dying memcgs improve? What type of objects are still
pinning those dying memcgs? did you run any traces to see how big
those pinned caches were and how much deferal and scanning work was
actually being done on them?

i.e. if all you measured is the number of memcgs over time, then we
don't have any information that tells us whether this patch has had
any effect on the reclaimable memory footprint of those dying memcgs
or what is actually pinning them in memory.

IOWs, we need to know if this patch reduces the dying memcg
references down to just the objects that non-owner memcgs are
keeping active in cache and hence preventing the dying memcgs from
being freed. If this patch does that, then the shrinkers are doing
exactly what they should be doing, and the remaining problem to
solve is reparenting actively referenced objects pinning the dying
memcgs...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com