Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Naresh Kamboju <naresh.kamboju@linaro.org>]

Hi Alexander,

On Mon, 24 Jul 2023 at 17:40, Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
>
> On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@google.com> wrote:
> >
> > On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
> > <torvalds@linux-foundation.org> wrote:
> > >
> > > [ Removed the stable reviewers, bringing in the kfence people ]
> > >
> > > See
> > >
> > >   https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
> > >
> > > for the original report. The warning was introduced in 8f0b36497303
> > > ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
> > > any other cases of this.
> > >
> > > Anybody?
> > >
> > >                     Linus
> > >
> >
> >
> > > > NOTE:
> > > > The following kernel warning was noticed while booting qemu-arm64
> > > > with these configs enabled on stable rc 6.4.5-rc1.
> > > >
> > > >  CONFIG_ARM64_64K_PAGES=y
> > > >  CONFIG_KFENCE=y
> >
> > Is there a full config somewhere?
>
> Please find build details
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/config
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/vmlinux.xz
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/System.map
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/Image.gz
>
> >
> > > > This crash is not easily reproducible.
> >
> > CONFIG_KFENCE_SAMPLE_INTERVAL=10
> > CONFIG_KFENCE_NUM_OBJECTS=2048
> >
> > might improve reproducibility.
>
> The above test have following Kconfigs enabled.
>
> CONFIG_HAVE_ARCH_KASAN=y
> CONFIG_HAVE_ARCH_KASAN_SW_TAGS=y
> CONFIG_HAVE_ARCH_KASAN_HW_TAGS=y
> CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
> CONFIG_CC_HAS_KASAN_GENERIC=y
> CONFIG_CC_HAS_KASAN_SW_TAGS=y
> CONFIG_CC_HAS_WORKING_NOSANITIZE_ADDRESS=y
> # CONFIG_KASAN is not set
> CONFIG_HAVE_ARCH_KFENCE=y
> CONFIG_KFENCE=y
> CONFIG_KFENCE_SAMPLE_INTERVAL=100
> CONFIG_KFENCE_NUM_OBJECTS=255
> # CONFIG_KFENCE_DEFERRABLE is not set
> CONFIG_KFENCE_STRESS_TEST_FAULTS=0
>
> >
> > > >
> > > > boot logs:
> > > > --------
> > > > [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
> > > > [    0.000000] Linux version 6.4.5-rc1 (tuxmake@tuxmake)
> > > > (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils
> > > > for Debian) 2.40) #1 SMP PREEMPT @1689957802
> > > > [    0.000000] random: crng init done
> > > > [    0.000000] Machine model: linux,dummy-virt
> > > > ...
> > > > [    0.006821] kfence: initialized - using 33554432 bytes for 255
> > > > objects at 0x(____ptrval____)-0x(____ptrval____)
> > > > ...
> > > > [    7.726994] ------------[ cut here ]------------
> > > > [    7.727704] WARNING: CPU: 1 PID: 1 at mm/kfence/core.c:1097
> > > > __kfence_free+0x84/0xc8
> > > ...
> > > > [    7.746478] Call trace:
> > > > [    7.746776]  __kfence_free+0x84/0xc8
> > > > [    7.747134]  __slab_free+0x490/0x508
> > > > [    7.748063]  __kmem_cache_free+0x2b4/0x2d0
> > > > [    7.748377]  kfree+0x78/0x140
> > > > [    7.748638]  single_release+0x40/0x60
> > > > [    7.750664]  __fput+0x78/0x260
> > > > [    7.751065]  ____fput+0x18/0x30
> > > > [    7.752086]  task_work_run+0x80/0xe0
> > > > [    7.753122]  do_notify_resume+0x200/0x1398
> > > > [    7.754292]  el0_svc+0xec/0x100
> > > > [    7.754573]  el0t_64_sync_handler+0xf4/0x120
> > > > [    7.755559]  el0t_64_sync+0x190/0x198
> >
> > It would be interesting to see the contents of
> > /sys/kernel/debug/kfence/objects together with the object address.
> > Would it be possible to boot the kernel with no_hash_pointers and add
> > a line printing the object address:
> >
> > diff --git a/mm/kfence/core.c b/mm/kfence/core.c
> > index dad3c0eb70a01..23f27f6cb18cf 100644
> > --- a/mm/kfence/core.c
> > +++ b/mm/kfence/core.c
> > @@ -1094,7 +1094,10 @@ void __kfence_free(void *addr)
> >         struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr);
> >
> >  #ifdef CONFIG_MEMCG
> > -       KFENCE_WARN_ON(meta->objcg);
> > +       if (meta->objcg) {
> > +               pr_err("ADDR: %px\n", addr);
> > +               KFENCE_WARN_ON(1);
> > +       }
> >  #endif
> >         /*
> >          * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing
> >
> > , and then dump /sys/kernel/debug/kfence/objects?
> >
>
> This testing is running on CI loops and however, I will try to reproduce
> this locally.

I have applied the above debug patch and tested in a loop but the issues
did not reproduce yet.

- Naresh