Jesper/Christop

Thanks for getting back.

This is Marvel Armada dual core ARMV7 and it is 32bit CPU.

The problem is NOT seen if maxcpus=1 used in command line.

The HAVE_ALIGNED_STRUCT_PAGE and CONFIG_HAVE_CMPXCHG_DOUBLE is NOT defined
for the board where problem is seen.

The problem is profoundly seen on kmalloc-8192 slab.
The slab size and object size is displayed below.
Also the object partial is growing and active slabs are growing.

It seems one core is trying to allocate the buffers while other core is freeing the buffer and causing this.
I have to add some debug to confirm the theory.

I also turned the SLUB_DEBUG_CMPXCHG and it is flooding the console.
Some of those messages pasted below.

Let me know if you need more info.

Thanks
Ajay


================== CPU info ============================
:/# cat /proc/cpuinfo
processor       : 0
model name      : ARMv7 Processor rev 1 (v7l)
Features        : swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x4
CPU part        : 0xc09
CPU revision    : 1

processor       : 1
model name      : ARMv7 Processor rev 1 (v7l)
Features        : swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x4
CPU part        : 0xc09
CPU revision    : 1

Hardware        : Marvell Armada 380/385/390/398

:/#



#:/sys/kernel/slab/kmalloc-8192# cat slab_size
8384
#:/sys/kernel/slab/kmalloc-8192# cat object_size
8192
=====================Initial state of counters  =====================
:/sys/kernel/slab/kmalloc-8192# cat objects_partial
2
:/sys/kernel/slab/kmalloc-8192# cat /proc/slabinfo | grep 8192
kmalloc-8192       32786  32790   8384    3    8 : tunables    0 0    0 : slabdata  10930  10930      0

===================== counters after some time ============
#:/sys/kernel/slab/kmalloc-8192# cat /proc/slabinfo | grep 8192
kmalloc-8192       32789  44712   8384    3    8 : tunables    0 0    0 : slabdata  14904  14904      0

#:/sys/kernel/slab/kmalloc-8192# cat objects_partial
15006



======================= Debug messages ======================

[03/30/2016 13:19:21.9400] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:22.0700] __slab_free skbuff_head_cache: cmpxchg double redo acquire_slab names_cache: cmpxchg double redo
[03/30/2016 13:19:22.2100] __slab_free names_cache: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:22.3400] acquire_slab skbuff_head_cache: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:22.5000] __slab_free filp: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:22.6300] acquire_slab skbuff_head_cache: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:22.7900] acquire_slab skbuff_head_cache: cmpxchg double redo __slab_free kmalloc-2048: cmpxchg double redo
[03/30/2016 13:19:22.9500] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:23.0900] acquire_slab skbuff_head_cache: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:23.2300] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:23.3600] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:23.4900] acquire_slab skbuff_head_cache: cmpxchg double redo __slab_free skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:23.6500] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:23.7900] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:23.9200] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:24.0500] __slab_free skbuff_head_cache: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:24.2300] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:24.3600] acquire_slab kmalloc-8192: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:24.5000] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:24.6300] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:24.7600] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:24.9000] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:25.0300] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:25.1700] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:25.3000] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:25.4300] acquire_slab kmalloc-8192: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:25.5600] unfreezing slab kmalloc-8192: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:25.6900] acquire_slab skbuff_head_cache: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:25.8300] unfreezing slab kmalloc-8192: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:25.9600] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:26.0900] unfreezing slab kmalloc-8192: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:26.2300] unfreezing slab kmalloc-8192: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:26.3600] __slab_free kmalloc-8192: cmpxchg double redo __slab_free kmalloc-256: cmpxchg double redo
[03/30/2016 13:19:26.5000] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:26.6900] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:26.8200] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:26.9400] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:27.0700] unfreezing slab skbuff_head_cache: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:27.2100] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:27.3400] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:27.4800] acquire_slab skbuff_head_cache: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:27.6100] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:27.7400] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:27.8700] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:28.0000] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:28.1300] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:28.2600] acquire_slab kmalloc-4096: cmpxchg double redo __slab_free kmalloc-4096: cmpxchg double redo
[03/30/2016 13:19:28.4300] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:28.5700] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:28.7000] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:28.8400] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:28.9700] __slab_free kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:29.0900] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:29.2300] acquire_slab skbuff_head_cache: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:29.3600] unfreezing slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:29.5000] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:29.6300] acquire_slab skbuff_head_cache: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:29.7700] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:29.9000] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:30.0300] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:30.1700] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:30.3000] acquire_slab skbuff_head_cache: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:30.4400] __slab_free kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:30.5700] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:30.7000] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:30.8300] __slab_free skbuff_head_cache: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:30.9800] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:31.1100] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:31.2400] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:31.3700] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:31.5000] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:31.6300] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:31.7700] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:31.9000] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:32.0300] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:32.1600] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:32.2900] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:32.4300] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:32.5600] acquire_slab skbuff_head_cache: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:32.7000] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:32.8300] acquire_slab skbuff_head_cache: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:32.9700] acquire_slab skbuff_head_cache: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:33.1200] __slab_free skbuff_head_cache: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:33.2600] acquire_slab skbuff_head_cache: cmpxchg double redo __slab_free kmalloc-64: cmpxchg double redo
[03/30/2016 13:19:33.5900] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:33.7300] acquire_slab kmalloc-8192: cmpxchg double redo unfreezing slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:33.8700] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:34.0000] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:34.2300] acquire_slab kmalloc-8192: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:34.3700] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:34.5000] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:34.6300] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:34.7600] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:34.9000] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:35.0300] acquire_slab skbuff_head_cache: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:35.1700] acquire_slab skbuff_head_cache: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:35.3000] __slab_free kmalloc-8192: cmpxchg double redo unfreezing slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:35.4400] unfreezing slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:35.5700] __slab_free kmalloc-8192: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:35.7000] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:35.8400] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:35.9600] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:36.0900] unfreezing slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-4096: cmpxchg double redo
[03/30/2016 13:19:36.2300] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:36.3500] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:36.4900] acquire_slab skbuff_head_cache: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:36.6200] __slab_free kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:36.7500] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:36.8800] __slab_free skbuff_head_cache: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:37.0200] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:37.1500] acquire_slab kmalloc-8192: cmpxchg double redo unfreezing slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:37.2900] acquire_slab kmalloc-8192: cmpxchg double redo acquire_slab skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:37.4200] acquire_slab skbuff_head_cache: cmpxchg double redo acquire_slab kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:37.5500] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free skbuff_head_cache: cmpxchg double redo
[03/30/2016 13:19:37.7500] acquire_slab kmalloc-8192: cmpxchg double redo __slab_free kmalloc-8192: cmpxchg double redo
[03/30/2016 13:19:37.8900] acquire_slab kmalloc-8192: cmpxchg double redo ac





On Wed, Mar 30, 2016 at 1:58 AM, Jesper Dangaard Brouer <brouer@redhat.com> wrote:

Hi Ajay,

Could you please provide info on kernel .config settings via commands:

 grep HAVE_ALIGNED_STRUCT_PAGE .config
 grep CONFIG_HAVE_CMPXCHG_DOUBLE .config

You can try to further debug your problem by defining SLUB_DEBUG_CMPXCHG
manually in mm/slub.c to get some verbose output on the cmpxchg failures.

Is the "Marvell Armada dual core ARMV7" a 32-bit CPU?

--Jesper

On Tue, 29 Mar 2016 15:32:26 -0700 Ajay Patel <patela@gmail.com> wrote:

> We have custom board with Marvell Armada dual core ARMV7.
> The driver uses buffers from kmalloc-8192 slab heavily.
> When slub_debug is enabled, the kmalloc-8192 active slabs are
> increasing. The slub stats shows  cmpxchg_double_fail and objects_partial
> are increasing too. Eventually system panics on oom.
>
> Following patch fixes the issue.
> Has anybody encountered this issue?
> Is this right fix?
>
> I am not in mailing list please cc me.
>
> Thanks
> Ajay
>
>
> --- slub.c.orig Tue Mar 29 11:54:42 2016
> +++ slub.c      Tue Mar 29 15:08:30 2016
> @@ -1562,9 +1562,12 @@
>         void *freelist;
>         unsigned long counters;
>         struct page new;
> +       int retry_count = 0;
> +#define RETRY_COUNT 10
>
>         lockdep_assert_held(&n->list_lock);
>
> +again:
>         /*
>          * Zap the freelist and set the frozen bit.
>          * The old freelist is the list of objects for the
> @@ -1587,8 +1590,13 @@
>         if (!__cmpxchg_double_slab(s, page,
>                         freelist, counters,
>                         new.freelist, new.counters,
> -                       "acquire_slab"))
> +                       "acquire_slab")) {
> +               if (retry_count++ < RETRY_COUNT) {
> +                       new.frozen = 0;
> +                       goto again;
> +               }
>                 return NULL;
> +       }
>
>         remove_partial(n, page);
>         WARN_ON(!freelist);



--
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer