Re: [RFC] memcg rstat flushing optimization

[Tejun Heo <tj@kernel.org>]

Hello,

On Wed, Oct 05, 2022 at 11:02:23AM -0700, Yosry Ahmed wrote:
> > I was thinking more that being done inside the flush function.
> 
> I think the flush function already does that in some sense if
> might_sleep is true, right? The problem here is that we are using

Oh I forgot about that. Right.

...
> I took a couple of crashed machines kdumps and ran a script to
> traverse updated memcgs and check how many cpus have updates and how
> many updates are there on each cpu. I found that on average only a
> couple of stats are updated per-cpu per-cgroup, and less than 25% of
> cpus (but this is on a large machine, I expect the number to go higher
> on smaller machines). Which is why I suggested a bitmask. I understand
> though that this depends on whatever workloads were running on those
> machines, and that in case where most stats are updated the bitmask
> will actually make things slightly worse.

One worry I have about selective flushing is that it's only gonna improve
things by some multiples while we can reasonably increase the problem size
by orders of magnitude.

The only real ways out I can think of are:

* Implement a periodic flusher which keeps the stats needed in irqsafe path
  acceptably uptodate to avoid flushing with irq disabled. We can make this
  adaptive too - no reason to do all this if the number to flush isn't huge.

* Shift some work to the updaters. e.g. in many cases, propagating per-cpu
  updates a couple levels up from update path will significantly reduce the
  fanouts and thus the number of entries which need to be flushed later. It
  does add on-going overhead, so it prolly should adaptive or configurable,
  hopefully the former.

Thanks.

-- 
tejun