Re: memcgroup lruvec_lru_size scaling issue

[Michal Hocko <mhocko@kernel.org>]

On Mon 14-10-19 10:49:49, Dave Hansen wrote:
> On 10/14/19 10:37 AM, Michal Hocko wrote:
> >> for_each_possible_cpu(cpu)
> >> 	x += per_cpu(pn->lruvec_stat_local->count[idx], cpu);
> >>
> >> It is costly looping through all the cpus to get the lru vec size info.
> >> And doing this on our workload with 96 cpu threads and 500 mem cgroups
> >> makes things much worse.  We might end up having 96 cpus * 500 cgroups * 2 (main) LRUs pagevecs,
> >> which is a lot of data structures to be running through all the time.
> > Why does the number of cgroup matter?
> 
> I was thinking purely of the cache footprint.  If it's reading
> pn->lruvec_stat_local->count[idx] is three separate cachelines, so 192
> bytes of cache *96 CPUs = 18k of data, mostly read-only.  1 cgroup would
> be 18k of data for the whole system and the caching would be pretty
> efficient and all 18k would probably survive a tight page fault loop in
> the L1.  500 cgroups would be ~90k of data per CPU thread which doesn't
> fit in the L1 and probably wouldn't survive a tight page fault loop if
> both logical threads were banging on different cgroups.
> 
> It's just a theory, but it's why I noted the number of cgroups when I
> initially saw this show up in profiles.

Yes, the cache traffic might be really high but I still find it a bit
surprising that it makes such a large footprint because this should be
mostly called from slow paths (reclaim) and the real work done should
just be larger - at least that's my intuition which might be quite off
here. How much is that 25% of the system time in the total time btw?

-- 
Michal Hocko
SUSE Labs