Re: [RFC] memcg rstat flushing optimization

[Yosry Ahmed <yosryahmed@google.com>]

On Mon, Oct 17, 2022 at 11:52 AM Michal Koutný <mkoutny@suse.com> wrote:
>
> Hello.
>
> On Tue, Oct 04, 2022 at 06:17:40PM -0700, Yosry Ahmed <yosryahmed@google.com> wrote:
> > Sorry for the long email :)
>
> (I'll get to other parts sometime in the future. Sorry for my latency :)
>
> > We have recently ran into a hard lockup on a machine with hundreds of
> > CPUs and thousands of memcgs during an rstat flush.
> > [...]
>
> I only respond with some remarks to this particular case.
>
>
> > As you can imagine, with a sufficiently large number of
> > memcgs and cpus, a call to mem_cgroup_flush_stats() might be slow, or
> > in an extreme case like the one we ran into, cause a hard lockup
> > (despite periodically flushing every 4 seconds).
>
> Is this your modification from the upstream value of FLUSH_TIME (that's
> every 2 s)?

It's actually once every 4s like upstream, I got confused by
flush_next_time multiplying the flush interval by 2.

>
> In the mailthread, you also mention >10s for hard-lockups. That sounds
> scary (even with the once per 4 seconds) since with large enough update
> tree (and update activity) periodic flush couldn't keep up.
> Also, it seems to be kind of bad feedback, the longer a (periodic) flush
> takes, the lower is the frequency of them and the more updates may
> accumulate. I.e. one spike in update activity can get the system into
> a spiral of long flushes that won't recover once the activity doesn't
> drop much more.

Yeah it is scary and shouldn't be likely to happen, but it did :(

We can keep coming up with mitigations to try and make it less likely,
but I was hoping we can find something more fundamental like keeping
track of what we really need to flush or avoiding all flushing in
non-sleepable contexts if possible.

>
> (2nd point should have been about some memcg_check_events() optimization
> or THRESHOLDS_EVENTS_TARGET justifying delayed flush but I've found none to be applicable.
> Just noting that v2 fortunetly doesn't have the threshold
> notifications.)

I think even without that, we can still run into the same problem in
other non-sleepable flushing contexts.

>
> Regards,
> Michal