Re: Memory allocation profiling warnings in memory bound systems

[Suren Baghdasaryan <surenb@google.com>]

On Mon, May 19, 2025 at 9:08 AM Johannes Weiner <hannes@cmpxchg.org> wrote:
>
> On Mon, May 19, 2025 at 08:50:28AM -0700, Suren Baghdasaryan wrote:
> > On Mon, May 19, 2025 at 6:33 AM Usama Arif <usamaarif642@gmail.com> wrote:
> > >
> > >
> > > +cc Vlad
> > >
> > > On 19/05/2025 14:31, Usama Arif wrote:
> > > > Hi,
> > > >
> > > > We have started enabling memory allocation profiling (with kernel 6.13) in our fleet
> > > > and are seeing a large number of warnings (reported by Vlad Poenaru) due to failure
> > > > in allocation of slab object extensions on services that are memory bound. I have attached
> > > > one of the logs at the end.
> > > >
> > > > Does it make sense to change the slabobj_ext to be allocated via kvcalloc and also change
> > > > the WARN to WARN_ONCE (or maybe even pr_debug?) like the diff below? A large number of
> > > > prints for this in a short time may mask any real issues in the system during memory
> > > > pressure being reported in dmesg. I tried to see if there were any changes after 6.13
> > > > to this code but didn't find any, but thought will check before sending below as a patch.
> > > >
> > > > diff --git a/mm/slub.c b/mm/slub.c
> > > > index c2151c9fee22..4595ca190cd9 100644
> > > > --- a/mm/slub.c
> > > > +++ b/mm/slub.c
> > > > @@ -1961,7 +1961,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
> > > >         gfp &= ~OBJCGS_CLEAR_MASK;
> > > >         /* Prevent recursive extension vector allocation */
> > > >         gfp |= __GFP_NO_OBJ_EXT;
> > > > -       vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp,
> > > > +       vec = kvcalloc_node(objects, sizeof(struct slabobj_ext), gfp,
> >
> > Hi Usama,
> > Is the allocation larger than page size? IIUC, unless allocation size
> > is over PAGE_SIZE, kvcalloc_node() will not fall back to vmalloc (see:
> > https://elixir.bootlin.com/linux/v6.14.7/source/mm/util.c#L668). How
> > big is the allocation when it fails in your case?
>
> Digging through the reports, it appears we're encountering both. We've
> seen a zswap slab where the slab is order-0 and slabext is
> higher-order (8 byte objects, 512 objsperslab, 1 pageperslab), but
> also biovec-max where it's the other way round (4k byte objects, 8
> objsperslab, 8 pagesperslab).
>
> In the first case, vmalloc would help. In the second it wouldn't.

Ok, then I don't see any downside to changing to kvcalloc_node() here.
Let's do it.

>
> The second case is interesting. The higher-order slab succeeds because
> bios use a mempool; but the system is so depleted that the order-0 for
> the slabext fails.

I see.

>
> I'm not sure there is much we can do about this tbh. It would seem
> overkill to add a mempool or grant the tracking access to system-wide
> emergency reserves.

Yeah, with the system under so much memory pressure we probably have
bigger issues than extension vector allocation failures.

>
> A warn-once would probably make sense nonetheless.

Agree.

>
> It might also make sense to flag the line item for that callsite in
> the reporting file, to make it obvious that the counter is compromised
> and is missing allocations?

Good idea. We could output something like 'X' instead of the number if
the value is known to be invalid. I can look into it. Will also have
to raise the file version so that parsers can handle this change.

>