Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Linus Torvalds]

[ Removed the stable reviewers, bringing in the kfence people ]

See

  https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/

for the original report. The warning was introduced in 8f0b36497303
("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
any other cases of this.

Anybody?

                    Linus

On Sat, 22 Jul 2023 at 01:06, Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
>
> Results from Linaro’s test farm.
> No regressions on arm64, arm, x86_64, and i386.
>
> Tested-by: Linux Kernel Functional Testing <lkft@linaro.org>
>
> NOTE:
> The following kernel warning was noticed while booting qemu-arm64
> with these configs enabled on stable rc 6.4.5-rc1.
>
>  CONFIG_ARM64_64K_PAGES=y
>  CONFIG_KFENCE=y
>
> This crash is not easily reproducible.
>
> boot logs:
> --------
> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
> [    0.000000] Linux version 6.4.5-rc1 (tuxmake@tuxmake)
> (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils
> for Debian) 2.40) #1 SMP PREEMPT @1689957802
> [    0.000000] random: crng init done
> [    0.000000] Machine model: linux,dummy-virt
> ...
> [    0.006821] kfence: initialized - using 33554432 bytes for 255
> objects at 0x(____ptrval____)-0x(____ptrval____)
> ...
> [    7.726994] ------------[ cut here ]------------
> [    7.727704] WARNING: CPU: 1 PID: 1 at mm/kfence/core.c:1097
> __kfence_free+0x84/0xc8
...
> [    7.746478] Call trace:
> [    7.746776]  __kfence_free+0x84/0xc8
> [    7.747134]  __slab_free+0x490/0x508
> [    7.748063]  __kmem_cache_free+0x2b4/0x2d0
> [    7.748377]  kfree+0x78/0x140
> [    7.748638]  single_release+0x40/0x60
> [    7.750664]  __fput+0x78/0x260
> [    7.751065]  ____fput+0x18/0x30
> [    7.752086]  task_work_run+0x80/0xe0
> [    7.753122]  do_notify_resume+0x200/0x1398
> [    7.754292]  el0_svc+0xec/0x100
> [    7.754573]  el0t_64_sync_handler+0xf4/0x120
> [    7.755559]  el0t_64_sync+0x190/0x198

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Alexander Potapenko]

On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> [ Removed the stable reviewers, bringing in the kfence people ]
>
> See
>
>   https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
>
> for the original report. The warning was introduced in 8f0b36497303
> ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
> any other cases of this.
>
> Anybody?
>
>                     Linus
>


> > NOTE:
> > The following kernel warning was noticed while booting qemu-arm64
> > with these configs enabled on stable rc 6.4.5-rc1.
> >
> >  CONFIG_ARM64_64K_PAGES=y
> >  CONFIG_KFENCE=y

Is there a full config somewhere?

> > This crash is not easily reproducible.

CONFIG_KFENCE_SAMPLE_INTERVAL=10
CONFIG_KFENCE_NUM_OBJECTS=2048

might improve reproducibility.

> >
> > boot logs:
> > --------
> > [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
> > [    0.000000] Linux version 6.4.5-rc1 (tuxmake@tuxmake)
> > (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils
> > for Debian) 2.40) #1 SMP PREEMPT @1689957802
> > [    0.000000] random: crng init done
> > [    0.000000] Machine model: linux,dummy-virt
> > ...
> > [    0.006821] kfence: initialized - using 33554432 bytes for 255
> > objects at 0x(____ptrval____)-0x(____ptrval____)
> > ...
> > [    7.726994] ------------[ cut here ]------------
> > [    7.727704] WARNING: CPU: 1 PID: 1 at mm/kfence/core.c:1097
> > __kfence_free+0x84/0xc8
> ...
> > [    7.746478] Call trace:
> > [    7.746776]  __kfence_free+0x84/0xc8
> > [    7.747134]  __slab_free+0x490/0x508
> > [    7.748063]  __kmem_cache_free+0x2b4/0x2d0
> > [    7.748377]  kfree+0x78/0x140
> > [    7.748638]  single_release+0x40/0x60
> > [    7.750664]  __fput+0x78/0x260
> > [    7.751065]  ____fput+0x18/0x30
> > [    7.752086]  task_work_run+0x80/0xe0
> > [    7.753122]  do_notify_resume+0x200/0x1398
> > [    7.754292]  el0_svc+0xec/0x100
> > [    7.754573]  el0t_64_sync_handler+0xf4/0x120
> > [    7.755559]  el0t_64_sync+0x190/0x198

It would be interesting to see the contents of
/sys/kernel/debug/kfence/objects together with the object address.
Would it be possible to boot the kernel with no_hash_pointers and add
a line printing the object address:

diff --git a/mm/kfence/core.c b/mm/kfence/core.c
index dad3c0eb70a01..23f27f6cb18cf 100644
--- a/mm/kfence/core.c
+++ b/mm/kfence/core.c
@@ -1094,7 +1094,10 @@ void __kfence_free(void *addr)
        struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr);

 #ifdef CONFIG_MEMCG
-       KFENCE_WARN_ON(meta->objcg);
+       if (meta->objcg) {
+               pr_err("ADDR: %px\n", addr);
+               KFENCE_WARN_ON(1);
+       }
 #endif
        /*
         * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing

, and then dump /sys/kernel/debug/kfence/objects?

Knowing the kfence pool location (the line starting with "kfence:
initialized") and the object address, we can probably understand from
the allocation stack in sysfs, whether the object is supposed to be
deleted.

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Naresh Kamboju]

On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@google.com> wrote:
>
> On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > [ Removed the stable reviewers, bringing in the kfence people ]
> >
> > See
> >
> >   https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
> >
> > for the original report. The warning was introduced in 8f0b36497303
> > ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
> > any other cases of this.
> >
> > Anybody?
> >
> >                     Linus
> >
>
>
> > > NOTE:
> > > The following kernel warning was noticed while booting qemu-arm64
> > > with these configs enabled on stable rc 6.4.5-rc1.
> > >
> > >  CONFIG_ARM64_64K_PAGES=y
> > >  CONFIG_KFENCE=y
>
> Is there a full config somewhere?

Please find build details
 - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/
 - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/config
 - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/vmlinux.xz
 - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/System.map
 - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/Image.gz

>
> > > This crash is not easily reproducible.
>
> CONFIG_KFENCE_SAMPLE_INTERVAL=10
> CONFIG_KFENCE_NUM_OBJECTS=2048
>
> might improve reproducibility.

The above test have following Kconfigs enabled.

CONFIG_HAVE_ARCH_KASAN=y
CONFIG_HAVE_ARCH_KASAN_SW_TAGS=y
CONFIG_HAVE_ARCH_KASAN_HW_TAGS=y
CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
CONFIG_CC_HAS_KASAN_GENERIC=y
CONFIG_CC_HAS_KASAN_SW_TAGS=y
CONFIG_CC_HAS_WORKING_NOSANITIZE_ADDRESS=y
# CONFIG_KASAN is not set
CONFIG_HAVE_ARCH_KFENCE=y
CONFIG_KFENCE=y
CONFIG_KFENCE_SAMPLE_INTERVAL=100
CONFIG_KFENCE_NUM_OBJECTS=255
# CONFIG_KFENCE_DEFERRABLE is not set
CONFIG_KFENCE_STRESS_TEST_FAULTS=0

>
> > >
> > > boot logs:
> > > --------
> > > [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
> > > [    0.000000] Linux version 6.4.5-rc1 (tuxmake@tuxmake)
> > > (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils
> > > for Debian) 2.40) #1 SMP PREEMPT @1689957802
> > > [    0.000000] random: crng init done
> > > [    0.000000] Machine model: linux,dummy-virt
> > > ...
> > > [    0.006821] kfence: initialized - using 33554432 bytes for 255
> > > objects at 0x(____ptrval____)-0x(____ptrval____)
> > > ...
> > > [    7.726994] ------------[ cut here ]------------
> > > [    7.727704] WARNING: CPU: 1 PID: 1 at mm/kfence/core.c:1097
> > > __kfence_free+0x84/0xc8
> > ...
> > > [    7.746478] Call trace:
> > > [    7.746776]  __kfence_free+0x84/0xc8
> > > [    7.747134]  __slab_free+0x490/0x508
> > > [    7.748063]  __kmem_cache_free+0x2b4/0x2d0
> > > [    7.748377]  kfree+0x78/0x140
> > > [    7.748638]  single_release+0x40/0x60
> > > [    7.750664]  __fput+0x78/0x260
> > > [    7.751065]  ____fput+0x18/0x30
> > > [    7.752086]  task_work_run+0x80/0xe0
> > > [    7.753122]  do_notify_resume+0x200/0x1398
> > > [    7.754292]  el0_svc+0xec/0x100
> > > [    7.754573]  el0t_64_sync_handler+0xf4/0x120
> > > [    7.755559]  el0t_64_sync+0x190/0x198
>
> It would be interesting to see the contents of
> /sys/kernel/debug/kfence/objects together with the object address.
> Would it be possible to boot the kernel with no_hash_pointers and add
> a line printing the object address:
>
> diff --git a/mm/kfence/core.c b/mm/kfence/core.c
> index dad3c0eb70a01..23f27f6cb18cf 100644
> --- a/mm/kfence/core.c
> +++ b/mm/kfence/core.c
> @@ -1094,7 +1094,10 @@ void __kfence_free(void *addr)
>         struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr);
>
>  #ifdef CONFIG_MEMCG
> -       KFENCE_WARN_ON(meta->objcg);
> +       if (meta->objcg) {
> +               pr_err("ADDR: %px\n", addr);
> +               KFENCE_WARN_ON(1);
> +       }
>  #endif
>         /*
>          * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing
>
> , and then dump /sys/kernel/debug/kfence/objects?
>

This testing is running on CI loops and however, I will try to reproduce
this locally.

> Knowing the kfence pool location (the line starting with "kfence:
> initialized") and the object address, we can probably understand from
> the allocation stack in sysfs, whether the object is supposed to be
> deleted.

- Naresh

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Naresh Kamboju]

Hi Alexander,

On Mon, 24 Jul 2023 at 17:40, Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
>
> On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@google.com> wrote:
> >
> > On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
> > <torvalds@linux-foundation.org> wrote:
> > >
> > > [ Removed the stable reviewers, bringing in the kfence people ]
> > >
> > > See
> > >
> > >   https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
> > >
> > > for the original report. The warning was introduced in 8f0b36497303
> > > ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
> > > any other cases of this.
> > >
> > > Anybody?
> > >
> > >                     Linus
> > >
> >
> >
> > > > NOTE:
> > > > The following kernel warning was noticed while booting qemu-arm64
> > > > with these configs enabled on stable rc 6.4.5-rc1.
> > > >
> > > >  CONFIG_ARM64_64K_PAGES=y
> > > >  CONFIG_KFENCE=y
> >
> > Is there a full config somewhere?
>
> Please find build details
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/config
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/vmlinux.xz
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/System.map
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/Image.gz
>
> >
> > > > This crash is not easily reproducible.
> >
> > CONFIG_KFENCE_SAMPLE_INTERVAL=10
> > CONFIG_KFENCE_NUM_OBJECTS=2048
> >
> > might improve reproducibility.
>
> The above test have following Kconfigs enabled.
>
> CONFIG_HAVE_ARCH_KASAN=y
> CONFIG_HAVE_ARCH_KASAN_SW_TAGS=y
> CONFIG_HAVE_ARCH_KASAN_HW_TAGS=y
> CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
> CONFIG_CC_HAS_KASAN_GENERIC=y
> CONFIG_CC_HAS_KASAN_SW_TAGS=y
> CONFIG_CC_HAS_WORKING_NOSANITIZE_ADDRESS=y
> # CONFIG_KASAN is not set
> CONFIG_HAVE_ARCH_KFENCE=y
> CONFIG_KFENCE=y
> CONFIG_KFENCE_SAMPLE_INTERVAL=100
> CONFIG_KFENCE_NUM_OBJECTS=255
> # CONFIG_KFENCE_DEFERRABLE is not set
> CONFIG_KFENCE_STRESS_TEST_FAULTS=0
>
> >
> > > >
> > > > boot logs:
> > > > --------
> > > > [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
> > > > [    0.000000] Linux version 6.4.5-rc1 (tuxmake@tuxmake)
> > > > (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils
> > > > for Debian) 2.40) #1 SMP PREEMPT @1689957802
> > > > [    0.000000] random: crng init done
> > > > [    0.000000] Machine model: linux,dummy-virt
> > > > ...
> > > > [    0.006821] kfence: initialized - using 33554432 bytes for 255
> > > > objects at 0x(____ptrval____)-0x(____ptrval____)
> > > > ...
> > > > [    7.726994] ------------[ cut here ]------------
> > > > [    7.727704] WARNING: CPU: 1 PID: 1 at mm/kfence/core.c:1097
> > > > __kfence_free+0x84/0xc8
> > > ...
> > > > [    7.746478] Call trace:
> > > > [    7.746776]  __kfence_free+0x84/0xc8
> > > > [    7.747134]  __slab_free+0x490/0x508
> > > > [    7.748063]  __kmem_cache_free+0x2b4/0x2d0
> > > > [    7.748377]  kfree+0x78/0x140
> > > > [    7.748638]  single_release+0x40/0x60
> > > > [    7.750664]  __fput+0x78/0x260
> > > > [    7.751065]  ____fput+0x18/0x30
> > > > [    7.752086]  task_work_run+0x80/0xe0
> > > > [    7.753122]  do_notify_resume+0x200/0x1398
> > > > [    7.754292]  el0_svc+0xec/0x100
> > > > [    7.754573]  el0t_64_sync_handler+0xf4/0x120
> > > > [    7.755559]  el0t_64_sync+0x190/0x198
> >
> > It would be interesting to see the contents of
> > /sys/kernel/debug/kfence/objects together with the object address.
> > Would it be possible to boot the kernel with no_hash_pointers and add
> > a line printing the object address:
> >
> > diff --git a/mm/kfence/core.c b/mm/kfence/core.c
> > index dad3c0eb70a01..23f27f6cb18cf 100644
> > --- a/mm/kfence/core.c
> > +++ b/mm/kfence/core.c
> > @@ -1094,7 +1094,10 @@ void __kfence_free(void *addr)
> >         struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr);
> >
> >  #ifdef CONFIG_MEMCG
> > -       KFENCE_WARN_ON(meta->objcg);
> > +       if (meta->objcg) {
> > +               pr_err("ADDR: %px\n", addr);
> > +               KFENCE_WARN_ON(1);
> > +       }
> >  #endif
> >         /*
> >          * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing
> >
> > , and then dump /sys/kernel/debug/kfence/objects?
> >
>
> This testing is running on CI loops and however, I will try to reproduce
> this locally.

I have applied the above debug patch and tested in a loop but the issues
did not reproduce yet.

- Naresh

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Alexander Potapenko]

On Mon, Jul 24, 2023 at 2:10 PM Naresh Kamboju
<naresh.kamboju@linaro.org> wrote:
>
> On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@google.com> wrote:
> >
> > On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
> > <torvalds@linux-foundation.org> wrote:
> > >
> > > [ Removed the stable reviewers, bringing in the kfence people ]
> > >
> > > See
> > >
> > >   https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
> > >
> > > for the original report. The warning was introduced in 8f0b36497303
> > > ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
> > > any other cases of this.
> > >
> > > Anybody?
> > >
> > >                     Linus
> > >
> >
> >
> > > > NOTE:
> > > > The following kernel warning was noticed while booting qemu-arm64
> > > > with these configs enabled on stable rc 6.4.5-rc1.
> > > >
> > > >  CONFIG_ARM64_64K_PAGES=y
> > > >  CONFIG_KFENCE=y
> >
> > Is there a full config somewhere?
>
> Please find build details
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/config
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/vmlinux.xz
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/System.map
>  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/Image.gz

I am afraid it still doesn't help much.
I installed tuxmake into a virtualenv and tried running:

 $ tuxmake --runtime podman --target-arch arm64 --toolchain gcc-12
--kconfig https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/config

, but after downloading some Docker stuff it died with the following error:

Error: copying system image from manifest list: writing blob: adding
layer with blob
"sha256:faef57eae888cbe4a5613eca6741b5e48d768b83f6088858aee9a5a2834f8151":
processing tar file(potentially insufficient UIDs or GIDs available in
user namespace (requested 0:42 for /etc/gshadow): Check /etc/subuid
and /etc/subgid if configured locally and run podman-system-migrate:
lchown /etc/gshadow: invalid argument): exit status 1

I also tried building with the config you provided, but the resulting
kernel didn't boot with my existing rootfs.

The third attempt was to take a config that's known to work, and
enable CONFIG_ARM64_64K_PAGES=y in it, and boot on a rootfs that's
known to work, but it didn't boot either, claiming that "/sbin/init
exists but couldn't execute it".

Are there any additional requirements to rootfs related to 64k pages?
Or maybe there's a system image somewhere at tuxsite.com that I can use?

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Alexander Potapenko]

On Tue, Jul 25, 2023 at 11:59 AM Alexander Potapenko <glider@google.com> wrote:
>
> On Mon, Jul 24, 2023 at 2:10 PM Naresh Kamboju
> <naresh.kamboju@linaro.org> wrote:
> >
> > On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@google.com> wrote:
> > >
> > > On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
> > > <torvalds@linux-foundation.org> wrote:
> > > >
> > > > [ Removed the stable reviewers, bringing in the kfence people ]
> > > >
> > > > See
> > > >
> > > >   https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
> > > >
> > > > for the original report. The warning was introduced in 8f0b36497303
> > > > ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
> > > > any other cases of this.
> > > >
> > > > Anybody?
> > > >
> > > >                     Linus
> > > >
> > >
> > >
> > > > > NOTE:
> > > > > The following kernel warning was noticed while booting qemu-arm64
> > > > > with these configs enabled on stable rc 6.4.5-rc1.
> > > > >
> > > > >  CONFIG_ARM64_64K_PAGES=y
> > > > >  CONFIG_KFENCE=y
> > >
> > > Is there a full config somewhere?
> >
> > Please find build details
> >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/
> >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/config
> >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/vmlinux.xz
> >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/System.map
> >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/Image.gz
>
> I am afraid it still doesn't help much.
> I installed tuxmake into a virtualenv and tried running:
Ok, my buildroot indeed lacked the BR2_ARM64_PAGE_SIZE_64K. I am now
able to boot a kernel with your config, trying to reproduce the
problem locally...

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Naresh Kamboju]

On Tue, 25 Jul 2023 at 17:22, Alexander Potapenko <glider@google.com> wrote:
>
> On Tue, Jul 25, 2023 at 11:59 AM Alexander Potapenko <glider@google.com> wrote:
> >
> > On Mon, Jul 24, 2023 at 2:10 PM Naresh Kamboju
> > <naresh.kamboju@linaro.org> wrote:
> > >
> > > On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@google.com> wrote:
> > > >
> > > > On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
> > > > <torvalds@linux-foundation.org> wrote:
> > > > >
> > > > > [ Removed the stable reviewers, bringing in the kfence people ]
> > > > >
> > > > > See
> > > > >
> > > > >   https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
> > > > >
> > > > > for the original report. The warning was introduced in 8f0b36497303
> > > > > ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
> > > > > any other cases of this.
> > > > >
> > > > > Anybody?
> > > > >
> > > > >                     Linus
> > > > >
> > > >
> > > >
> > > > > > NOTE:
> > > > > > The following kernel warning was noticed while booting qemu-arm64
> > > > > > with these configs enabled on stable rc 6.4.5-rc1.
> > > > > >
> > > > > >  CONFIG_ARM64_64K_PAGES=y
> > > > > >  CONFIG_KFENCE=y
> > > >
> > > > Is there a full config somewhere?
> > >
> > > Please find build details
> > >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/
> > >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/config
> > >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/vmlinux.xz
> > >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/System.map
> > >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/Image.gz
> >
> > I am afraid it still doesn't help much.
> > I installed tuxmake into a virtualenv and tried running:
> Ok, my buildroot indeed lacked the BR2_ARM64_PAGE_SIZE_64K. I am now
> able to boot a kernel with your config, trying to reproduce the
> problem locally...

Great to know that, boot successfully with a 64k page size kernel image.

- Naresh

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Alexander Potapenko]

On Tue, Jul 25, 2023 at 3:39 PM Naresh Kamboju
<naresh.kamboju@linaro.org> wrote:
>
> On Tue, 25 Jul 2023 at 17:22, Alexander Potapenko <glider@google.com> wrote:
> >
> > On Tue, Jul 25, 2023 at 11:59 AM Alexander Potapenko <glider@google.com> wrote:
> > >
> > > On Mon, Jul 24, 2023 at 2:10 PM Naresh Kamboju
> > > <naresh.kamboju@linaro.org> wrote:
> > > >
> > > > On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@google.com> wrote:
> > > > >
> > > > > On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
> > > > > <torvalds@linux-foundation.org> wrote:
> > > > > >
> > > > > > [ Removed the stable reviewers, bringing in the kfence people ]
> > > > > >
> > > > > > See
> > > > > >
> > > > > >   https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
> > > > > >
> > > > > > for the original report. The warning was introduced in 8f0b36497303
> > > > > > ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
> > > > > > any other cases of this.
> > > > > >
> > > > > > Anybody?
> > > > > >
> > > > > >                     Linus
> > > > > >
> > > > >

Muchun, any chance you know under what circumstances a KFENCE object
has its meta->objcg set to a non-NULL value?
It seems to be a quite rare case, and I've only seen it in live
radix_tree_node objects.
Since the check here:
https://elixir.bootlin.com/linux/latest/source/mm/kfence/core.c#L1097
ensures that this value is NULL when the object is freed, where is the
code that is supposed to zero it?
Could there be a race somewhere?

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Alexander Potapenko]

On Tue, Jul 25, 2023 at 6:21 PM Alexander Potapenko <glider@google.com> wrote:
>
> On Tue, Jul 25, 2023 at 3:39 PM Naresh Kamboju
> <naresh.kamboju@linaro.org> wrote:
> >
> > On Tue, 25 Jul 2023 at 17:22, Alexander Potapenko <glider@google.com> wrote:
> > >
> > > On Tue, Jul 25, 2023 at 11:59 AM Alexander Potapenko <glider@google.com> wrote:
> > > >
> > > > On Mon, Jul 24, 2023 at 2:10 PM Naresh Kamboju
> > > > <naresh.kamboju@linaro.org> wrote:
> > > > >
> > > > > On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@google.com> wrote:
> > > > > >
> > > > > > On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
> > > > > > <torvalds@linux-foundation.org> wrote:
> > > > > > >
> > > > > > > [ Removed the stable reviewers, bringing in the kfence people ]
> > > > > > >
> > > > > > > See
> > > > > > >
> > > > > > >   https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
> > > > > > >
> > > > > > > for the original report. The warning was introduced in 8f0b36497303
> > > > > > > ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
> > > > > > > any other cases of this.
> > > > > > >
> > > > > > > Anybody?
> > > > > > >
> > > > > > >                     Linus
> > > > > > >
> > > > > >
>
> Muchun, any chance you know under what circumstances a KFENCE object
> has its meta->objcg set to a non-NULL value?
> It seems to be a quite rare case, and I've only seen it in live
> radix_tree_node objects.
> Since the check here:
> https://elixir.bootlin.com/linux/latest/source/mm/kfence/core.c#L1097
> ensures that this value is NULL when the object is freed, where is the
> code that is supposed to zero it?
> Could there be a race somewhere?


I am still puzzled about what is going on.

As far as I can see, when KFENCE pool is initialized, for ith object
page in the pool its page_slab()->memcg_data is set to a value derived
from kfence_metadata[i].objcg
Because KFENCE objects always occupy one page, no two objects are
expected to share memcg_data at any time.

When slab_alloc_node() is called, it first invokes
slab_pre_alloc_hook(), figures out the obj_cgroup and charges it for
the allocated memory. The obj_cgroup is returned to slab_alloc_node()
and after KFENCE allocation succeeds is passed to
slab_post_alloc_hook(), which then writes obj_cgroup to
*(page_slab(object)->memcg_data).

When an object is deallocated, slab_free() calls
memcg_slab_free_hook(), which zeroes *(page_slab(object)->memcg_data)
and passes the object to kfence_free().
At this point the object's meta->objcg must be NULL, so the warning
should not be firing.



--
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Muchun Song]

> On Jul 27, 2023, at 00:52, Alexander Potapenko <glider@google.com> wrote:
> 
> On Tue, Jul 25, 2023 at 6:21 PM Alexander Potapenko <glider@google.com> wrote:
>> 
>> On Tue, Jul 25, 2023 at 3:39 PM Naresh Kamboju
>> <naresh.kamboju@linaro.org> wrote:
>>> 
>>> On Tue, 25 Jul 2023 at 17:22, Alexander Potapenko <glider@google.com> wrote:
>>>> 
>>>> On Tue, Jul 25, 2023 at 11:59 AM Alexander Potapenko <glider@google.com> wrote:
>>>>> 
>>>>> On Mon, Jul 24, 2023 at 2:10 PM Naresh Kamboju
>>>>> <naresh.kamboju@linaro.org> wrote:
>>>>>> 
>>>>>> On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@google.com> wrote:
>>>>>>> 
>>>>>>> On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
>>>>>>> <torvalds@linux-foundation.org> wrote:
>>>>>>>> 
>>>>>>>> [ Removed the stable reviewers, bringing in the kfence people ]
>>>>>>>> 
>>>>>>>> See
>>>>>>>> 
>>>>>>>>  https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
>>>>>>>> 
>>>>>>>> for the original report. The warning was introduced in 8f0b36497303
>>>>>>>> ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
>>>>>>>> any other cases of this.
>>>>>>>> 
>>>>>>>> Anybody?
>>>>>>>> 
>>>>>>>>                    Linus
>>>>>>>> 
>>>>>>> 
>> 
>> Muchun, any chance you know under what circumstances a KFENCE object
>> has its meta->objcg set to a non-NULL value?
>> It seems to be a quite rare case, and I've only seen it in live
>> radix_tree_node objects.
>> Since the check here:
>> https://elixir.bootlin.com/linux/latest/source/mm/kfence/core.c#L1097
>> ensures that this value is NULL when the object is freed, where is the
>> code that is supposed to zero it?
>> Could there be a race somewhere?
> 
> 
> I am still puzzled about what is going on.
> 
> As far as I can see, when KFENCE pool is initialized, for ith object
> page in the pool its page_slab()->memcg_data is set to a value derived
> from kfence_metadata[i].objcg
> Because KFENCE objects always occupy one page, no two objects are
> expected to share memcg_data at any time.
> 
> When slab_alloc_node() is called, it first invokes
> slab_pre_alloc_hook(), figures out the obj_cgroup and charges it for
> the allocated memory. The obj_cgroup is returned to slab_alloc_node()
> and after KFENCE allocation succeeds is passed to
> slab_post_alloc_hook(), which then writes obj_cgroup to
> *(page_slab(object)->memcg_data).
> 
> When an object is deallocated, slab_free() calls
> memcg_slab_free_hook(), which zeroes *(page_slab(object)->memcg_data)
> and passes the object to kfence_free().
> At this point the object's meta->objcg must be NULL, so the warning
> should not be firing.

At least, totally agree. This call  stack comes from slab_free() which
makes sure memcg_slab_free_hook() is called before kfence_free(), so
meta->objcg must be NULL. Otherwise, seems something is corrupted. So
I really want to know what's the value of "meta->objcg" when the warning
is firing (e.g. whether it is a valid pointer or does the last bit is
set with MEMCG_DATA_OBJCGS). Maybe we could improve the warning message,
e.g. print the current value of "meta->objcg".

Thanks.

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Muchun Song]

> On Jul 27, 2023, at 15:02, Muchun Song <muchun.song@linux.dev> wrote:
> 
> 
> 
>> On Jul 27, 2023, at 00:52, Alexander Potapenko <glider@google.com> wrote:
>> 
>> On Tue, Jul 25, 2023 at 6:21 PM Alexander Potapenko <glider@google.com> wrote:
>>> 
>>> On Tue, Jul 25, 2023 at 3:39 PM Naresh Kamboju
>>> <naresh.kamboju@linaro.org> wrote:
>>>> 
>>>> On Tue, 25 Jul 2023 at 17:22, Alexander Potapenko <glider@google.com> wrote:
>>>>> 
>>>>> On Tue, Jul 25, 2023 at 11:59 AM Alexander Potapenko <glider@google.com> wrote:
>>>>>> 
>>>>>> On Mon, Jul 24, 2023 at 2:10 PM Naresh Kamboju
>>>>>> <naresh.kamboju@linaro.org> wrote:
>>>>>>> 
>>>>>>> On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@google.com> wrote:
>>>>>>>> 
>>>>>>>> On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
>>>>>>>> <torvalds@linux-foundation.org> wrote:
>>>>>>>>> 
>>>>>>>>> [ Removed the stable reviewers, bringing in the kfence people ]
>>>>>>>>> 
>>>>>>>>> See
>>>>>>>>> 
>>>>>>>>> https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
>>>>>>>>> 
>>>>>>>>> for the original report. The warning was introduced in 8f0b36497303
>>>>>>>>> ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
>>>>>>>>> any other cases of this.
>>>>>>>>> 
>>>>>>>>> Anybody?
>>>>>>>>> 
>>>>>>>>>                   Linus
>>>>>>>>> 
>>>>>>>> 
>>> 
>>> Muchun, any chance you know under what circumstances a KFENCE object
>>> has its meta->objcg set to a non-NULL value?
>>> It seems to be a quite rare case, and I've only seen it in live
>>> radix_tree_node objects.
>>> Since the check here:
>>> https://elixir.bootlin.com/linux/latest/source/mm/kfence/core.c#L1097
>>> ensures that this value is NULL when the object is freed, where is the
>>> code that is supposed to zero it?
>>> Could there be a race somewhere?
>> 
>> 
>> I am still puzzled about what is going on.
>> 
>> As far as I can see, when KFENCE pool is initialized, for ith object
>> page in the pool its page_slab()->memcg_data is set to a value derived
>> from kfence_metadata[i].objcg
>> Because KFENCE objects always occupy one page, no two objects are
>> expected to share memcg_data at any time.
>> 
>> When slab_alloc_node() is called, it first invokes
>> slab_pre_alloc_hook(), figures out the obj_cgroup and charges it for
>> the allocated memory. The obj_cgroup is returned to slab_alloc_node()
>> and after KFENCE allocation succeeds is passed to
>> slab_post_alloc_hook(), which then writes obj_cgroup to
>> *(page_slab(object)->memcg_data).
>> 
>> When an object is deallocated, slab_free() calls
>> memcg_slab_free_hook(), which zeroes *(page_slab(object)->memcg_data)
>> and passes the object to kfence_free().
>> At this point the object's meta->objcg must be NULL, so the warning
>> should not be firing.
> 
> At least, totally agree. This call  stack comes from slab_free() which
> makes sure memcg_slab_free_hook() is called before kfence_free(), so
> meta->objcg must be NULL. Otherwise, seems something is corrupted. So
> I really want to know what's the value of "meta->objcg" when the warning
> is firing (e.g. whether it is a valid pointer or does the last bit is
> set with MEMCG_DATA_OBJCGS). Maybe we could improve the warning message,

Sorry for the confusing, meta->objcg should be a objcg pointer, it
cannot be set with MEMCG_DATA_OBJCGS.

> e.g. print the current value of "meta->objcg".
> 
> Thanks.

Re: [PATCH 6.4 000/292] 6.4.5-rc1 review

[Naresh Kamboju]

On Tue, 25 Jul 2023 at 15:29, Alexander Potapenko <glider@google.com> wrote:
>
> On Mon, Jul 24, 2023 at 2:10 PM Naresh Kamboju
> <naresh.kamboju@linaro.org> wrote:
> >
> > On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@google.com> wrote:
> > >
> > > On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds
> > > <torvalds@linux-foundation.org> wrote:
> > > >
> > > > [ Removed the stable reviewers, bringing in the kfence people ]
> > > >
> > > > See
> > > >
> > > >   https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@mail.gmail.com/
> > > >
> > > > for the original report. The warning was introduced in 8f0b36497303
> > > > ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find
> > > > any other cases of this.
> > > >
> > > > Anybody?
> > > >
> > > >                     Linus
> > > >
> > >
> > >
> > > > > NOTE:
> > > > > The following kernel warning was noticed while booting qemu-arm64
> > > > > with these configs enabled on stable rc 6.4.5-rc1.
> > > > >
> > > > >  CONFIG_ARM64_64K_PAGES=y
> > > > >  CONFIG_KFENCE=y
> > >
> > > Is there a full config somewhere?
> >
> > Please find build details
> >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/
> >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/config
> >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/vmlinux.xz
> >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/System.map
> >  - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/Image.gz
>
> I am afraid it still doesn't help much.
> I installed tuxmake into a virtualenv and tried running:
>
>  $ tuxmake --runtime podman --target-arch arm64 --toolchain gcc-12
> --kconfig https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/config
>
> , but after downloading some Docker stuff it died with the following error:
>
> Error: copying system image from manifest list: writing blob: adding
> layer with blob
> "sha256:faef57eae888cbe4a5613eca6741b5e48d768b83f6088858aee9a5a2834f8151":
> processing tar file(potentially insufficient UIDs or GIDs available in
> user namespace (requested 0:42 for /etc/gshadow): Check /etc/subuid
> and /etc/subgid if configured locally and run podman-system-migrate:
> lchown /etc/gshadow: invalid argument): exit status 1

This might helpful,

# Some images require this range in the user namespace

echo "{username}:4000:165535" > /etc/subuid
echo "{username}:4000:165535" > /etc/subgid

Replace {username} with the username in your machine.

- Naresh