* HW-KASAN and CONFIG_SLUB_DEBUG_ON=y screams about redzone corruption
@ 2023-06-28 15:47 Will Deacon
2023-07-04 7:41 ` Alexander Potapenko
0 siblings, 1 reply; 3+ messages in thread
From: Will Deacon @ 2023-06-28 15:47 UTC (permalink / raw)
To: catalin.marinas
Cc: ryabinin.a.a, andreyknvl, pcc, kasan-dev, linux-mm, linux-arm-kernel
Hi memory tagging folks,
While debugging something else, I ended up running v6.4 on an arm64 (v9)
fastmodel with both CONFIG_SLUB_DEBUG_ON=y and CONFIG_KASAN_HW_TAGS=y.
This makes the system pretty unusable, as I see a tonne of kmalloc
Redzone corruption messages pretty much straight out of startup (example
below).
Please can you take a look?
Cheers,
Will
--->8
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[ 0.000000] =============================================================================
[ 0.000000] BUG kmalloc-128 (Not tainted): kmalloc Redzone overwritten
[ 0.000000] -----------------------------------------------------------------------------
[ 0.000000]
[ 0.000000] 0xffff00080001a9b0-0xf1ff00080001a9ff @offset=2480. First byte 0x0 instead of 0xcc
[ 0.000000] Allocated in apply_wqattrs_prepare+0x90/0x2a4 age=0 cpu=0 pid=0
[ 0.000000] kmalloc_trace+0x34/0x6c
[ 0.000000] apply_wqattrs_prepare+0x90/0x2a4
[ 0.000000] apply_workqueue_attrs+0x5c/0xb4
[ 0.000000] alloc_workqueue+0x368/0x4f8
[ 0.000000] workqueue_init_early+0x2e8/0x3ac
[ 0.000000] start_kernel+0x168/0x394
[ 0.000000] __primary_switched+0xbc/0xc4
[ 0.000000] Slab 0xfffffc0020000680 objects=21 used=8 fp=0xffff00080001ac80 flags=0xbfffc0000010200(slab|head|node=0|zone=2|lastcpupid=0xffff|kasantag=0x0)
[ 0.000000] Object 0xf1ff00080001a980 @offset=17437937757178562944 fp=0x0000000000000000
[ 0.000000]
[ 0.000000] Redzone ffff00080001a900: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a910: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a920: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a930: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a940: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a950: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a960: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a970: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Object ffff00080001a980: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Redzone ffff00080001aa00: cc cc cc cc cc cc cc cc ........
[ 0.000000] Padding ffff00080001aa54: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 0.000000] Padding ffff00080001aa64: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 0.000000] Padding ffff00080001aa74: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZ
[ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.4.0-00001-g56e11237836c #1
[ 0.000000] Hardware name: FVP Base RevC (DT)
[ 0.000000] Call trace:
[ 0.000000] dump_backtrace+0xec/0x108
[ 0.000000] show_stack+0x18/0x2c
[ 0.000000] dump_stack_lvl+0x50/0x68
[ 0.000000] dump_stack+0x18/0x24
[ 0.000000] print_trailer+0x1ec/0x230
[ 0.000000] check_bytes_and_report+0x110/0x154
[ 0.000000] check_object+0x31c/0x360
[ 0.000000] free_to_partial_list+0x174/0x5d8
[ 0.000000] __slab_free+0x220/0x28c
[ 0.000000] __kmem_cache_free+0x364/0x3dc
[ 0.000000] kfree+0x50/0x70
[ 0.000000] apply_wqattrs_prepare+0x244/0x2a4
[ 0.000000] apply_workqueue_attrs+0x5c/0xb4
[ 0.000000] alloc_workqueue+0x368/0x4f8
[ 0.000000] workqueue_init_early+0x2e8/0x3ac
[ 0.000000] start_kernel+0x168/0x394
[ 0.000000] __primary_switched+0xbc/0xc4
[ 0.000000] Disabling lock debugging due to kernel taint
[ 0.000000] FIX kmalloc-128: Restoring kmalloc Redzone 0xffff00080001a9b0-0xf1ff00080001a9ff=0xcc
[ 0.000000] FIX kmalloc-128: Object at 0xf1ff00080001a980 not freed
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: HW-KASAN and CONFIG_SLUB_DEBUG_ON=y screams about redzone corruption
2023-06-28 15:47 HW-KASAN and CONFIG_SLUB_DEBUG_ON=y screams about redzone corruption Will Deacon
@ 2023-07-04 7:41 ` Alexander Potapenko
2023-07-04 15:29 ` Andrey Konovalov
0 siblings, 1 reply; 3+ messages in thread
From: Alexander Potapenko @ 2023-07-04 7:41 UTC (permalink / raw)
To: Will Deacon
Cc: catalin.marinas, ryabinin.a.a, andreyknvl, pcc, kasan-dev,
linux-mm, linux-arm-kernel
On Wed, Jun 28, 2023 at 5:47 PM Will Deacon <will@kernel.org> wrote:
>
> Hi memory tagging folks,
>
> While debugging something else, I ended up running v6.4 on an arm64 (v9)
> fastmodel with both CONFIG_SLUB_DEBUG_ON=y and CONFIG_KASAN_HW_TAGS=y.
> This makes the system pretty unusable, as I see a tonne of kmalloc
> Redzone corruption messages pretty much straight out of startup (example
> below).
>
> Please can you take a look?
>
> Cheers,
Does the problem reproduce with CONFIG_KASAN_SW_TAGS?
Also, any chance you could share the file:line info for the stack trace below?
I myself haven't expected KASAN to work together with SLUB_DEBUG...
>
> Will
>
> --->8
>
> [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
> [ 0.000000] =============================================================================
> [ 0.000000] BUG kmalloc-128 (Not tainted): kmalloc Redzone overwritten
> [ 0.000000] -----------------------------------------------------------------------------
> [ 0.000000]
> [ 0.000000] 0xffff00080001a9b0-0xf1ff00080001a9ff @offset=2480. First byte 0x0 instead of 0xcc
> [ 0.000000] Allocated in apply_wqattrs_prepare+0x90/0x2a4 age=0 cpu=0 pid=0
> [ 0.000000] kmalloc_trace+0x34/0x6c
> [ 0.000000] apply_wqattrs_prepare+0x90/0x2a4
> [ 0.000000] apply_workqueue_attrs+0x5c/0xb4
> [ 0.000000] alloc_workqueue+0x368/0x4f8
> [ 0.000000] workqueue_init_early+0x2e8/0x3ac
> [ 0.000000] start_kernel+0x168/0x394
> [ 0.000000] __primary_switched+0xbc/0xc4
> [ 0.000000] Slab 0xfffffc0020000680 objects=21 used=8 fp=0xffff00080001ac80 flags=0xbfffc0000010200(slab|head|node=0|zone=2|lastcpupid=0xffff|kasantag=0x0)
> [ 0.000000] Object 0xf1ff00080001a980 @offset=17437937757178562944 fp=0x0000000000000000
> [ 0.000000]
> [ 0.000000] Redzone ffff00080001a900: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
> [ 0.000000] Redzone ffff00080001a910: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
> [ 0.000000] Redzone ffff00080001a920: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
> [ 0.000000] Redzone ffff00080001a930: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
> [ 0.000000] Redzone ffff00080001a940: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
> [ 0.000000] Redzone ffff00080001a950: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
> [ 0.000000] Redzone ffff00080001a960: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
> [ 0.000000] Redzone ffff00080001a970: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
> [ 0.000000] Object ffff00080001a980: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00 ................
> [ 0.000000] Object ffff00080001a990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [ 0.000000] Object ffff00080001a9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [ 0.000000] Object ffff00080001a9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [ 0.000000] Object ffff00080001a9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [ 0.000000] Object ffff00080001a9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [ 0.000000] Object ffff00080001a9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [ 0.000000] Object ffff00080001a9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [ 0.000000] Redzone ffff00080001aa00: cc cc cc cc cc cc cc cc ........
> [ 0.000000] Padding ffff00080001aa54: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 0.000000] Padding ffff00080001aa64: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 0.000000] Padding ffff00080001aa74: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZ
> [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.4.0-00001-g56e11237836c #1
> [ 0.000000] Hardware name: FVP Base RevC (DT)
> [ 0.000000] Call trace:
> [ 0.000000] dump_backtrace+0xec/0x108
> [ 0.000000] show_stack+0x18/0x2c
> [ 0.000000] dump_stack_lvl+0x50/0x68
> [ 0.000000] dump_stack+0x18/0x24
> [ 0.000000] print_trailer+0x1ec/0x230
> [ 0.000000] check_bytes_and_report+0x110/0x154
> [ 0.000000] check_object+0x31c/0x360
> [ 0.000000] free_to_partial_list+0x174/0x5d8
> [ 0.000000] __slab_free+0x220/0x28c
> [ 0.000000] __kmem_cache_free+0x364/0x3dc
> [ 0.000000] kfree+0x50/0x70
> [ 0.000000] apply_wqattrs_prepare+0x244/0x2a4
> [ 0.000000] apply_workqueue_attrs+0x5c/0xb4
> [ 0.000000] alloc_workqueue+0x368/0x4f8
> [ 0.000000] workqueue_init_early+0x2e8/0x3ac
> [ 0.000000] start_kernel+0x168/0x394
> [ 0.000000] __primary_switched+0xbc/0xc4
> [ 0.000000] Disabling lock debugging due to kernel taint
> [ 0.000000] FIX kmalloc-128: Restoring kmalloc Redzone 0xffff00080001a9b0-0xf1ff00080001a9ff=0xcc
> [ 0.000000] FIX kmalloc-128: Object at 0xf1ff00080001a980 not freed
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/20230628154714.GB22090%40willie-the-truck.
--
Alexander Potapenko
Software Engineer
Google Germany GmbH
Erika-Mann-Straße, 33
80636 München
Geschäftsführer: Paul Manicle, Liana Sebastian
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: HW-KASAN and CONFIG_SLUB_DEBUG_ON=y screams about redzone corruption
2023-07-04 7:41 ` Alexander Potapenko
@ 2023-07-04 15:29 ` Andrey Konovalov
0 siblings, 0 replies; 3+ messages in thread
From: Andrey Konovalov @ 2023-07-04 15:29 UTC (permalink / raw)
To: Alexander Potapenko, Will Deacon
Cc: catalin.marinas, ryabinin.a.a, pcc, kasan-dev, linux-mm,
linux-arm-kernel
On Tue, Jul 4, 2023 at 9:42 AM Alexander Potapenko <glider@google.com> wrote:
>
> > While debugging something else, I ended up running v6.4 on an arm64 (v9)
> > fastmodel with both CONFIG_SLUB_DEBUG_ON=y and CONFIG_KASAN_HW_TAGS=y.
> > This makes the system pretty unusable, as I see a tonne of kmalloc
> > Redzone corruption messages pretty much straight out of startup (example
> > below).
I've reproduced the issue, looking into the root cause.
> Does the problem reproduce with CONFIG_KASAN_SW_TAGS?
Looks like SW_TAGS is not affected.
> Also, any chance you could share the file:line info for the stack trace below?
>
> I myself haven't expected KASAN to work together with SLUB_DEBUG...
This was implemented at some point, so it should work.
Thanks!
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-07-04 15:30 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-06-28 15:47 HW-KASAN and CONFIG_SLUB_DEBUG_ON=y screams about redzone corruption Will Deacon
2023-07-04 7:41 ` Alexander Potapenko
2023-07-04 15:29 ` Andrey Konovalov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox