Re: [LSF/MM TOPIC] dying memory cgroups and slab reclaim issues

[Roman Gushchin <guro@fb.com>]

On Fri, Mar 01, 2019 at 08:30:32AM +1100, Dave Chinner wrote:
> On Thu, Feb 28, 2019 at 08:30:49PM +0000, Roman Gushchin wrote:
> > On Thu, Feb 21, 2019 at 02:46:17PM -0800, Roman Gushchin wrote:
> > > On Wed, Feb 20, 2019 at 06:27:07PM +1100, Dave Chinner wrote:
> > > > On Wed, Feb 20, 2019 at 04:50:31PM +1100, Dave Chinner wrote:
> > > > > I'm just going to fix the original regression in the shrinker
> > > > > algorithm by restoring the gradual accumulation behaviour, and this
> > > > > whole series of problems can be put to bed.
> > > > 
> > > > Something like this lightly smoke tested patch below. It may be
> > > > slightly more agressive than the original code for really small
> > > > freeable values (i.e. < 100) but otherwise should be roughly
> > > > equivalent to historic accumulation behaviour.
> > > > 
> > > > Cheers,
> > > > 
> > > > Dave.
> > > > -- 
> > > > Dave Chinner
> > > > david@fromorbit.com
> > > > 
> > > > mm: fix shrinker scan accumulation regression
> > > > 
> > > > From: Dave Chinner <dchinner@redhat.com>
> > > 
> > > JFYI: I'm testing this patch in our environment for fixing
> > > the memcg memory leak.
> > > 
> > > It will take a couple of days to get reliable results.
> > > 
> > 
> > So unfortunately the proposed patch is not solving the dying memcg reclaim
> > issue. I've tested it as is, with s/ilog2()/fls(), suggested by Johannes,
> > and also with more a aggressive zero-seek slabs reclaim (always scanning
> > at least SHRINK_BATCH for zero-seeks shrinkers).
> 
> Which makes sense if it's inodes and/or dentries shared across
> multiple memcgs and actively referenced by non-owner memcgs that
> prevent dying memcg reclaim. i.e. the shrinkers will not reclaim
> frequently referenced objects unless there is extreme memory
> pressure put on them.
> 
> > In all cases the number
> > of outstanding memory cgroups grew almost linearly with time and didn't show
> > any signs of plateauing.
> 
> What happend to the amount of memory pinned by those dying memcgs?
> Did that change in any way? Did the rate of reclaim of objects
> referencing dying memcgs improve? What type of objects are still
> pinning those dying memcgs? did you run any traces to see how big
> those pinned caches were and how much deferal and scanning work was
> actually being done on them?

The amount of pinned memory is approximately proportional to the number
of dying cgroups, in other words it also grows almost linearly.
The rate of reclaim is better than without any patches, and it's
approximately on pair with a version with Rik's patches.

> 
> i.e. if all you measured is the number of memcgs over time, then we
> don't have any information that tells us whether this patch has had
> any effect on the reclaimable memory footprint of those dying memcgs
> or what is actually pinning them in memory.

I'm not saying that the patch is bad, I'm saying it's not sufficient
in our environment.

> 
> IOWs, we need to know if this patch reduces the dying memcg
> references down to just the objects that non-owner memcgs are
> keeping active in cache and hence preventing the dying memcgs from
> being freed. If this patch does that, then the shrinkers are doing
> exactly what they should be doing, and the remaining problem to
> solve is reparenting actively referenced objects pinning the dying
> memcgs...

Yes, I agree. I'll take a look.

Thanks!