Re: memcgroup lruvec_lru_size scaling issue

[Michal Hocko <mhocko@kernel.org>]

On Mon 14-10-19 11:06:24, Tim Chen wrote:
> On 10/14/19 10:59 AM, Michal Hocko wrote:
> > On Mon 14-10-19 10:49:49, Dave Hansen wrote:
> >> On 10/14/19 10:37 AM, Michal Hocko wrote:
> >>>> for_each_possible_cpu(cpu)
> >>>> 	x += per_cpu(pn->lruvec_stat_local->count[idx], cpu);
> >>>>
> >>>> It is costly looping through all the cpus to get the lru vec size info.
> >>>> And doing this on our workload with 96 cpu threads and 500 mem cgroups
> >>>> makes things much worse.  We might end up having 96 cpus * 500 cgroups * 2 (main) LRUs pagevecs,
> >>>> which is a lot of data structures to be running through all the time.
> >>> Why does the number of cgroup matter?
> >>
> >> I was thinking purely of the cache footprint.  If it's reading
> >> pn->lruvec_stat_local->count[idx] is three separate cachelines, so 192
> >> bytes of cache *96 CPUs = 18k of data, mostly read-only.  1 cgroup would
> >> be 18k of data for the whole system and the caching would be pretty
> >> efficient and all 18k would probably survive a tight page fault loop in
> >> the L1.  500 cgroups would be ~90k of data per CPU thread which doesn't
> >> fit in the L1 and probably wouldn't survive a tight page fault loop if
> >> both logical threads were banging on different cgroups.
> >>
> >> It's just a theory, but it's why I noted the number of cgroups when I
> >> initially saw this show up in profiles.
> > 
> > Yes, the cache traffic might be really high but I still find it a bit
> > surprising that it makes such a large footprint because this should be
> > mostly called from slow paths (reclaim) and the real work done should
> > just be larger - at least that's my intuition which might be quite off
> > here. How much is that 25% of the system time in the total time btw?
> > 
> 
> About 7% of total cpu cycles.

OK, 7% on a highly direct reclaim bound workload sounds more realistic.
Please put all that to the changelog. It would be also great to see
whether that really scales with the number of cgroups if that is easy to
check with your benchmark.
-- 
Michal Hocko
SUSE Labs