Re: slabinfo shows incorrect active_objs ???

[Hyeonggon Yoo <42.hyeyoo@gmail.com>]

On Tue, Feb 22, 2022 at 12:22:02PM +0300, Vasily Averin wrote:
> Dear all,
> 
> I've found that /proc/slabinfo shows inadequate numbers of in-use slab objects.
> it assumes that all objects stored in cpu caches are always 100% in use.
> 
> for example:
>  slabinfo shows that all 20 objects are in use.
> 
> [root@fc34-vvs linux]# uname -a
> Linux fc34-vvs.sw.ru 5.17.0-rc3+ #42 SMP PREEMPT Mon Feb 21 20:14:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> 
> [root@fc34-vvs linux]# cat /proc/slabinfo
> slabinfo - version: 2.1
> # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
> ...
> kmalloc-cg-8k         20     20   8192    4    8 : tunables    0    0    0 : slabdata      5      5      0
> 
> At the same time crash said that only 2 objects are in use.
> 
> crash> kmem -s kmalloc-cg-8k
> CACHE             OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE  NAME
> ffff8f4840043b00     8192          2        20      5    32k  kmalloc-cg-8k
> 
> And this looks like true, see kmem -S output below.
> 
> Is it a bug or perhaps a well-known feature that I missed?
> 

This is not a bug..

TL;DR: Some of slabs are locked by CPU and SLUB does not accurately account them.
(I guess its purpose is to reduce cacheline usages in fastpath?)

There are 3 status of slab. A slab can be:
	1) a cpu slab or 2) cpu partial slabs:
		They're locked (frozen) by cpu to avoid per-node
		locking overhead.

	3) node partial slabs:
		They exist in node's partial slab list. Any CPU can take them
		using spinlock. Usually SLUB takes slab from node
		partial slabs when there is no cpu slab and cpu partial
		slabs.

Only free objects in node partial slabs are calculated as free objects.

> Numbers are counted in mm/slub.c, see below,
> but count_partial() doe not includes free objects of cpu caches
> 
> Moreover adequate statistic is not showed in any other interfaces too
> /sys/kerenl/slab/ read cpu slab caches but does not output these numbers.
> 
> Thank you,
> 	Vasily Averin
> 
> #ifdef CONFIG_SLUB_DEBUG
> void get_slabinfo(struct kmem_cache *s, struct slabinfo *sinfo)
> {
>         unsigned long nr_slabs = 0;
>         unsigned long nr_objs = 0;
>         unsigned long nr_free = 0;
>         int node;
>         struct kmem_cache_node *n;
> 
>         for_each_kmem_cache_node(s, node, n) {
>                 nr_slabs += node_nr_slabs(n);
>                 nr_objs += node_nr_objs(n);
>                 nr_free += count_partial(n, count_free);
>         }
> 
>         sinfo->active_objs = nr_objs - nr_free;

This code assumes all objects in frozen slabs (cpu slab or cpu
partial slabs) are not free.

With current implementation of SLUB, there is no easy way to accurately
account objects in cpu slab and cpu partial slabs' objects because SLUB
sets slab->inuse = slab->objects when taking slab from node partial slabs.

They are accounted again only when going back to node partial slabs.
(deactivated).

Thanks,
Hyeonggon

>         sinfo->num_objs = nr_objs;
> 
> 
> 
> crash> kmem -S kmalloc-cg-8k
> CACHE             OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE  NAME
> ffff8f4840043b00     8192          2        20      5    32k  kmalloc-cg-8k
> CPU 0 KMEM_CACHE_CPU:
>   ffff8f4b58236360
> CPU 0 SLAB:
>   (empty)
> CPU 0 PARTIAL:
>   (empty)
> CPU 1 KMEM_CACHE_CPU:
>   ffff8f4b58276360
> CPU 1 SLAB:
>   SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
>   ffffed3f842af400  ffff8f484abd0000     0      4          1     3
>   FREE / [ALLOCATED]
>    ffff8f484abd0000  (cpu 1 cache)
>    ffff8f484abd2000  (cpu 1 cache)
>    ffff8f484abd4000  (cpu 1 cache)
>   [ffff8f484abd6000]
> CPU 1 PARTIAL:
>   (empty)
> CPU 2 KMEM_CACHE_CPU:
>   ffff8f4b582b6360
> CPU 2 SLAB:
>   (empty)
> CPU 2 PARTIAL:
>   (empty)
> CPU 3 KMEM_CACHE_CPU:
>   ffff8f4b582f6360
> CPU 3 SLAB:
>   SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
>   ffffed3f842ce600  ffff8f484b398000     0      4          0     4
>   FREE / [ALLOCATED]
>    ffff8f484b398000  (cpu 3 cache)
>    ffff8f484b39a000  (cpu 3 cache)
>    ffff8f484b39c000  (cpu 3 cache)
>    ffff8f484b39e000  (cpu 3 cache)
> CPU 3 PARTIAL:
>   (empty)
> CPU 4 KMEM_CACHE_CPU:
>   ffff8f4b58336360
> CPU 4 SLAB:
>   SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
>   ffffed3f8418c200  ffff8f4846308000     0      4          0     4
>   FREE / [ALLOCATED]
>    ffff8f4846308000  (cpu 4 cache)
>    ffff8f484630a000  (cpu 4 cache)
>    ffff8f484630c000  (cpu 4 cache)
>    ffff8f484630e000  (cpu 4 cache)
> CPU 4 PARTIAL:
>   (empty)
> CPU 5 KMEM_CACHE_CPU:
>   ffff8f4b58376360
> CPU 5 SLAB:
>   (empty)
> CPU 5 PARTIAL:
>   (empty)
> CPU 6 KMEM_CACHE_CPU:
>   ffff8f4b583b6360
> CPU 6 SLAB:
>   SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
>   ffffed3f8412d000  ffff8f4844b40000     0      4          0     4
>   FREE / [ALLOCATED]
>    ffff8f4844b40000  (cpu 6 cache)
>    ffff8f4844b42000  (cpu 6 cache)
>    ffff8f4844b44000  (cpu 6 cache)
>    ffff8f4844b46000  (cpu 6 cache)
> CPU 6 PARTIAL:
>   (empty)
> CPU 7 KMEM_CACHE_CPU:
>   ffff8f4b583f6360
> CPU 7 SLAB:
>   SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
>   ffffed3f84124000  ffff8f4844900000     0      4          1     3
>   FREE / [ALLOCATED]
>    ffff8f4844900000  (cpu 7 cache)
>    ffff8f4844902000  (cpu 7 cache)
>   [ffff8f4844904000]
>    ffff8f4844906000  (cpu 7 cache)
> CPU 7 PARTIAL:
>   (empty)
> KMEM_CACHE_NODE   NODE  SLABS  PARTIAL  PER-CPU
> ffff8f48400416c0     0      5        0        5
> NODE 0 PARTIAL:
>   (empty)
> NODE 0 FULL:
>   (not tracked)
> 
> 
>