From: Breno Leitao <leitao@debian.org>
To: Catalin Marinas <catalin.marinas@arm.com>
Cc: Gu Bowen <gubowen5@huawei.com>,
Andrew Morton <akpm@linux-foundation.org>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Waiman Long <llong@redhat.com>,
stable@vger.kernel.org, linux-mm@kvack.org,
John Ogness <john.ogness@linutronix.de>,
Lu Jialin <lujialin4@huawei.com>
Subject: Re: [PATCH v5] mm: Fix possible deadlock in kmemleak
Date: Fri, 26 Sep 2025 07:57:37 -0700 [thread overview]
Message-ID: <kuq7guzalpqj5bxe2vt6s3kirrq4sg5ozwcim6ewnzpxhuxm4l@yfgb44nbcisz> (raw)
In-Reply-To: <aNVSsmY86yi-cV_e@arm.com>
hello Catalin,
On Thu, Sep 25, 2025 at 03:33:22PM +0100, Catalin Marinas wrote:
> On Fri, Sep 19, 2025 at 03:37:27AM -0700, Breno Leitao wrote:
> > On Fri, Aug 22, 2025 at 03:35:41PM +0800, Gu Bowen wrote:
> > > To solve this problem, switch to printk_safe mode before printing warning
> > > message, this will redirect all printk()-s to a special per-CPU buffer,
> > > which will be flushed later from a safe context (irq work), and this
> > > deadlock problem can be avoided.
> >
> > I am still thinking about this problem, given I got another deadlock
> > issue that I was not able to debug further given I do not have the
> > crashdump.
>
> Do you have some kernel log? I thought we covered all cases in
> kmemleak.c (well, might have missed some).
I do have, but, I do not have the vmcore anymore, thus, I was not able
to investigate much, but, here is a kernel message and dump of what all
CPUs were doing. Let me start with the lock/crash:
watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [kworker/u144:2:526241]
CPU#8 Utilization every 4s during lockup:
#1: 45% system, 55% softirq, 2% hardirq, 0% idle
#2: 39% system, 61% softirq, 2% hardirq, 0% idle
#3: 28% system, 73% softirq, 2% hardirq, 0% idle
#4: 49% system, 51% softirq, 2% hardirq, 0% idle
#5: 42% system, 58% softirq, 2% hardirq, 0% idle
Modules linked in: sunrpc(E) squashfs(E) sch_fq(E) tls(E) act_gact(E) tcp_diag(E) inet_diag(E) cls_bpf(E) intel_uncore_frequency(E) intel_uncore_frequency_common(E) skx_edac(E) skx_edac_common(E) nfit(E) libnvdimm(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) iTCO_wdt(E) iTCO_vendor_support(E) xhci_pci(E) kvm(E) acpi_cpufreq(E) irqbypass(E) i2c_i801(E) xhci_hcd(E) i2c_smbus(E) wmi(E) ipmi_si(E) ipmi_devintf(E) evdev(E) ipmi_msghandler(E) button(E) sch_fq_codel(E) bpf_preload(E) vhost_net(E) tun(E) vhost(E) vhost_iotlb(E) tap(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) loop(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) autofs4(E) efivarfs(E)
irq event stamp: 0
hardirqs last enabled at (0): [<0000000000000000>] 0x0
hardirqs last disabled at (0): [<ffffffff812f4fb2>] copy_process+0x362/0x1410
softirqs last enabled at (0): [<ffffffff812f4fb2>] copy_process+0x362/0x1410
softirqs last disabled at (0): [<0000000000000000>] 0x0
CPU: 8 UID: 0 PID: 526241 Comm: kworker/u144:2 Kdump: loaded Tainted: G S E N 6.17.0-0_fbk701_debug_rc0_0_gf83ec76bf285 #1 PREEMPT(none)
Tainted: [S]=CPU_OUT_OF_SPEC, [E]=UNSIGNED_MODULE, [N]=TEST
Hardware name: Quanta Twin Lakes MP/Twin Lakes Passive MP, BIOS F09_3A23 12/08/2020
Workqueue: events_unbound btrfs_reclaim_bgs_work
RIP: 0010:_raw_spin_unlock_irqrestore+0x5d/0xa0
Code: e0 fe f7 c3 00 02 00 00 74 05 e8 4e 5d f0 fe 9c 8f 04 24 f7 04 24 00 02 00 00 75 35 f7 c3 00 02 00 00 74 01 fb bf 01 00 00 00 <e8> 4e b3 d9 fe 65 8b 05 57 00 35 02 85 c0 74 26 65 48 8b 05 33 00
RSP: 0018:ffffc900004106d0 EFLAGS: 00000206
RAX: 24717fb88e5a6f00 RBX: 0000000000000282 RCX: 24717fb88e5a6f00
RDX: 0000000000000000 RSI: ffffffff82a3a8dc RDI: 0000000000000001
RBP: 0000000000000000 R08: ffff888f1cb85ab1 R09: ffff888f1cb85ab8
R10: ffff888f1cb85ab0 R11: ffff888f1cb85ac0 R12: ffffea003b1dc780
R13: ffffea003b1dc780 R14: ffffffff83e0bfd0 R15: ffffffff823c3954
FS: 0000000000000000(0000) GS:ffff88909fb25000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7e0ea1d1e0 CR3: 0000001073627001 CR4: 00000000007726f0
PKRU: 55555554
Call Trace:
<IRQ>
delete_object_full+0xba/0xe0
kmem_cache_free+0x108/0x3a0
? skb_release_data+0x132/0x150
tcp_rcv_established+0x2b4/0x720
tcp_v6_do_rcv+0x18e/0x520
tcp_v6_rcv+0x10e5/0x1260
? raw6_local_deliver+0x68/0x3e0
ip6_protocol_deliver_rcu+0x358/0x5c0
? NF_HOOK+0x36/0x1e0
? ip6_input+0x160/0x160
ip6_input_finish+0x58/0xf0
? ip6_input+0x160/0x160
NF_HOOK+0x170/0x1e0
? lock_release+0x4a/0x3e0
? ip6_dst_check+0x48/0x2b0
? ip6_dst_check+0x19c/0x2b0
? ip6_input+0x1a/0x160
ip6_input+0xb5/0x160
ip6_sublist_rcv+0x331/0x520
? ip6_rcv_core+0x4b7/0x6a0
ipv6_list_rcv+0x159/0x180
__netif_receive_skb_list_core+0x1eb/0x260
netif_receive_skb_list_internal+0x26b/0x380
? netif_receive_skb_list_internal+0x74/0x380
? dev_gro_receive+0x393/0x810
? rcu_is_watching+0xd/0x50
gro_complete+0x8a/0x290
dev_gro_receive+0x6df/0x810
gro_receive_skb+0xf6/0x2e0
bnxt_rx_pkt+0x112e/0x17a0
? lock_release+0x4a/0x3e0
? irqtime_account_irq+0x71/0x100
__bnxt_poll_work+0x195/0x390
? debug_check_no_obj_freed+0x27b/0x2a0
bnxt_poll+0x7f/0x320
? rcu_is_watching+0xd/0x50
? rcu_is_watching+0xd/0x50
__napi_poll+0x28/0x130
net_rx_action+0x233/0x4a0
? ktime_get+0x25/0x110
? sched_clock+0xc/0x20
? sched_clock_cpu+0xc/0x1d0
handle_softirqs+0x171/0x4f0
? __irq_exit_rcu+0x76/0x160
__irq_exit_rcu+0x76/0x160
irq_exit_rcu+0xa/0x20
sysvec_apic_timer_interrupt+0xa0/0xc0
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x16/0x20
RIP: 0010:_raw_spin_unlock_irqrestore+0x5d/0xa0
Code: e0 fe f7 c3 00 02 00 00 74 05 e8 4e 5d f0 fe 9c 8f 04 24 f7 04 24 00 02 00 00 75 35 f7 c3 00 02 00 00 74 01 fb bf 01 00 00 00 <e8> 4e b3 d9 fe 65 8b 05 57 00 35 02 85 c0 74 26 65 48 8b 05 33 00
RSP: 0018:ffffc9003c3ef9d0 EFLAGS: 00000206
RAX: 24717fb88e5a6f00 RBX: 0000000000000282 RCX: 24717fb88e5a6f00
RDX: ffff888c3074c5d8 RSI: ffffffff82a3a8dc RDI: 0000000000000001
RBP: 0000000000000000 R08: ffff888c3074c5e0 R09: ffff888c3074c5e8
R10: 0000000000000001 R11: 8080000000000000 R12: ffffea000a532e40
R13: ffffea000a532e40 R14: ffffffff83e0bfd0 R15: ffffffff81b3a016
? set_extent_bit.llvm.5807687024302053429+0x9b6/0xbd0
delete_object_full+0xba/0xe0
kmem_cache_free+0x108/0x3a0
set_extent_bit.llvm.5807687024302053429+0x9b6/0xbd0
? btrfs_lock_extent_bits+0x4a/0x2d0
btrfs_set_extent_delalloc+0x14f/0x170
relocate_file_extent_cluster+0x54b/0xa20
relocate_block_group+0x346/0x4a0
btrfs_relocate_block_group+0x22d/0x4b0
btrfs_relocate_chunk+0x64/0x220
btrfs_reclaim_bgs_work+0x375/0x630
? process_scheduled_works+0x28a/0x670
process_scheduled_works+0x2e8/0x670
worker_thread+0x26a/0x380
? pr_cont_work+0x1c0/0x1c0
kthread+0x227/0x250
? kthread_blkcg+0x30/0x30
ret_from_fork+0x155/0x260
? kthread_blkcg+0x30/0x30
ret_from_fork_asm+0x11/0x20
</TASK>
Kernel panic - not syncing: softlockup: hung tasks
This is the list of what each CPU was doing. The lines are based in
commit f83ec76bf285bea5727f478a68b894f5543ca76e ("Linux 6.17-rc6")
Crashing CPU: 8
Running tasks:
CPU: 3 PID: 37 COMM: ksoftirqd/3
#0 queued_spin_lock_slowpath(lock=kmemleak_lock, node=0xffff8890241d4b00, tail=0x100000, next=0, __PTR=0xffff8890241d4b08) (kernel/locking/qspinlock.c:291:3)
#1 queued_spin_lock (./include/asm-generic/qspinlock.h:114:2)
#2 do_raw_spin_lock (kernel/locking/spinlock_debug.c:116:2)
#3 __raw_spin_lock_irqsave(flags=130) (./include/linux/spinlock_api_smp.h:111:2)
#4 _raw_spin_lock_irqsave(lock=kmemleak_lock) (kernel/locking/spinlock.c:162:9)
#5 __find_and_get_object(ptr=0xffff888efe905c00[slab object: kmalloc-cg-128], alias=0, objflags=0) (mm/kmemleak.c:581:2)
#6 paint_ptr(_entry_ptr=paint_ptr._entry, color=-1) (mm/kmemleak.c:928:11)
#7 kvfree_call_rcu(__already_done=0, ptr=0xffff888efe905c00[slab object: kmalloc-cg-128]) (mm/slab_common.c:1987:2)
#8 bpf_selem_free_list(__warned=0, selem=0xffff888efe905c00[slab object: kmalloc-cg-128], n=0xffff888f163abb28[slab object: kmalloc-cg-256+0x28], smap=0xffff8882ae87b000[slab object: kmalloc-cg-1k]) (kernel/bpf/bpf_local_storage.c:277:3)
#9 bpf_local_storage_destroy(__warned=0, local_storage=0xffff888f21824f00[slab object: kmalloc-cg-256], storage_smap=0xffff8882ae87b000[slab object: kmalloc-cg-1k], flags=0x282) (kernel/bpf/bpf_local_storage.c:769:2)
#10 bpf_sk_storage_free(__warned=0, sk_storage=0xffff888f21824f00[slab object: kmalloc-cg-256]) (net/core/bpf_sk_storage.c:59:2)
#11 __sk_destruct(__warned=0, head=0xffff888c34b7aca8[slab object: TCPv6+0x4e8], sk=0xffff888c34b7a7c0[slab object: TCPv6], net=init_net, filter=0) (net/core/sock.c:2351:2)
#12 skb_release_head_state(skb=0xffff888f25ce2600[slab object: skbuff_head_cache]) (net/core/skbuff.c:1138:3)
#13 skb_release_all(reason=2) (net/core/skbuff.c:1149:2)
#14 __kfree_skb (net/core/skbuff.c:1165:2)
#15 kfree_skb_reason (./include/linux/skbuff.h:1275:2)
#16 dev_kfree_skb_any_reason (net/core/dev.c:3442:3)
#17 dev_consume_skb_any (./include/linux/netdevice.h:4148:2)
#18 __bnxt_tx_int(bp=0xffff8882a1db4b80, txr=0xffff8882c067b000[slab object: kmalloc-4k], txq=0xffff8882a3220000, tx_bytes=0x4a3f, tx_pkts=236, rc=0, tx_buf=0xffffc90005bfe450[vmap: 0xffffc90005bfd000-0xffffc90005c12000 caller bnxt_alloc_ring+0x1e3]) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:875:3)
#19 bnxt_tx_int(bp=0xffff8882a1db4b80, bnapi=0xffff88828e490040, i=0, txr=0xffff8882c067b000[slab object: kmalloc-4k]) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:895:12)
#20 __bnxt_poll_work_done (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3108:3)
#21 bnxt_poll_work(bp=0xffff8882a1db4b80, cpr=0xffff88828e490268, budget=64, bnapi=0xffff88828e490040, rx_pkts=64) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3138:2)
#22 bnxt_poll(napi=0xffff88828e490040, budget=64, bnapi=0xffff88828e490040, bp=0xffff8882a1db4b80, cpr=0xffff88828e490268, work_done=0) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3230:16)
#23 __napi_poll(__print_once=0, __already_done=0, _entry_ptr=__napi_poll._entry, n=0xffff88828e490040, repoll=0xffffc900002f7d58[vmap stack: 37 (ksoftirqd/3) +0x3d58], weight=64, work=0) (net/core/dev.c:7506:10)
#24 napi_poll(n=0xffff88828e490040, repoll=0xffffc900002f7d38[vmap stack: 37 (ksoftirqd/3) +0x3d38], do_repoll=0, have=0xffff88828e490040) (net/core/dev.c:7569:9)
#25 net_rx_action(sd=0xffff8890241d5180, budget=0x12c, time_limit=0x1009ed990, n=0xffff88828e490040) (net/core/dev.c:7696:13)
#26 handle_softirqs(_entry_ptr=handle_softirqs._entry, ksirqd=1, end=0xfffffffeff612670, old_flags=0x4208040, max_restart=10, pending=34, in_hardirq=1, h=softirq_vec+0x18, softirq_bit=2, vec_nr=3) (kernel/softirq.c:579:3)
#27 run_ksoftirqd (kernel/softirq.c:968:3)
#28 smpboot_thread_fn(data=0xffff8882881664b0[slab object: kmalloc-16], td=0xffff8882881664b0[slab object: kmalloc-16], ht=softirq_threads) (kernel/smpboot.c:160:4)
#29 kthread(threadfn=smpboot_thread_fn, data=0xffff8882881664b0[slab object: kmalloc-16], self=0xffff888288a7ba00[slab object: kmalloc-512], ret=-4) (kernel/kthread.c:463:9)
#30 ret_from_fork(regs=0xffffc900002f7f58[vmap stack: 37 (ksoftirqd/3) +0x3f58], fn=kthread, fn_arg=0xffff88828956fdc0[slab object: kmalloc-64]) (arch/x86/kernel/process.c:148:3)
CPU: 5 PID: 0 COMM: swapper/5
#0 get_stack_info_noinstr(stack=0xffffc900003748d8[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102], task=0xffff888289880000[slab object: task_struct], info=0xffffc90000374900[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102]) (arch/x86/kernel/dumpstack_64.c:172:0)
#1 get_stack_info(__already_done=0, _entry_ptr=get_stack_info._entry, task=0xffff888289880000[slab object: task_struct], info=0xffffc90000374900[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102], visit_mask=0xffffc90000374920[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102]) (arch/x86/kernel/dumpstack_64.c:199:7)
#2 __unwind_start(state=0xffffc90000374900[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102], first_frame=0xffffc90000374998[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102]) (arch/x86/kernel/unwind_orc.c:727:6)
#3 unwind_start(state=0xffffc90000374900[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102], task=0xffff888289880000[slab object: task_struct], regs=0) (./arch/x86/include/asm/unwind.h:64:2)
#4 arch_stack_walk(consume_entry=stack_trace_consume_entry, cookie=0xffffc900003749a8[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102], task=0xffff888289880000[slab object: task_struct], regs=0) (arch/x86/kernel/stacktrace.c:24:7)
#5 stack_trace_save (kernel/stacktrace.c:122:2)
#6 set_track_prepare (mm/kmemleak.c:655:15)
#7 __alloc_object(_entry_ptr=__alloc_object._entry) (mm/kmemleak.c:701:25)
#8 __create_object(ptr=0xffff888f205db700[slab object: skbuff_small_head]) (mm/kmemleak.c:779:11)
#9 kmemleak_alloc_recursive(ptr=0xffff888f205db700[slab object: skbuff_small_head], min_count=1) (./include/linux/kmemleak.h:44:3)
#10 slab_post_alloc_hook(s=0xffff88828a294900[slab object: kmem_cache], lru=0, flags=0x82820, size=1, p=0xffffc90000374af0[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102]) (mm/slub.c:4195:3)
#11 slab_alloc_node(s=0xffff88828a294900[slab object: kmem_cache], lru=0, gfpflags=0x82820, addr=kmalloc_reserve+0x12d, orig_size=0x280, object=0xffff888f205db700[slab object: skbuff_small_head]) (mm/slub.c:4240:2)
#12 kmem_cache_alloc_node_noprof(s=0xffff88828a294900[slab object: kmem_cache], gfpflags=0x82820, node=-1) (mm/slub.c:4292:14)
#13 kmalloc_reserve(size=0xffffc90000374b84[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102], flags=0x2820, node=-1, pfmemalloc=0xffffc90000374b83[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102], ret_pfmemalloc=0, _old=0) (net/core/skbuff.c:578:9)
#14 __alloc_skb(size=150, gfp_mask=0x2820, flags=6, node=-1, cache=0xffff88828a294700[slab object: kmem_cache], skb=0xffff888f1f491a00[slab object: skbuff_head_cache], pfmemalloc=0) (net/core/skbuff.c:669:9)
#15 napi_alloc_skb(napi=0xffff88828e4919c0, len=150, gfp_mask=0x2820) (net/core/skbuff.c:811:9)
#16 bnxt_copy_data(data=0xffff888f14b5d040, bp=0xffff8882a1db4b80, pdev=0xffff88829058f000[slab object: kmalloc-4k]) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:1395:8)
#17 bnxt_copy_skb(data=0xffff888f14b5d040) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:1417:9)
#18 bnxt_rx_pkt(__print_once=0, bp=0xffff8882a1db4b80, cpr=0xffff88828e491be8, raw_cons=0xffffc90000374d7c[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102], event=0xffffc90000374d7b[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102], bnapi=0xffff88828e4919c0, rxr=0xffff88828e48e280, dev=0xffff8882a1db4000, xdp_active=0, rc=0, rxcmp=0xffff8882956a8820, cmp_type=17, rxcmp1=0xffff8882956a8830, prod=0x8f52, cons=0xf53, rx_buf=0xffffc90005bb1fc8[vmap: 0xffffc90005b9b000-0xffffc90005bcc000 caller bnxt_alloc_ring+0x1e3], data=0xffff888f14b5d000, misc=0x560601, agg_bufs=0, flags=0x562411) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:2245:10)
#19 __bnxt_poll_work(bp=0xffff8882a1db4b80, cpr=0xffff88828e491be8, budget=64, bnapi=0xffff88828e4919c0, raw_cons=0x13b7482, rx_pkts=0, event=1, txcmp=0xffff8882956a8820) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3053:10)
#20 bnxt_poll_work(bp=0xffff8882a1db4b80, cpr=0xffff88828e491be8, budget=64, bnapi=0xffff88828e4919c0) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3130:12)
#21 bnxt_poll(napi=0xffff88828e4919c0, budget=64, bnapi=0xffff88828e4919c0, bp=0xffff8882a1db4b80, cpr=0xffff88828e491be8, work_done=0) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3230:16)
#22 __napi_poll(__print_once=0, __already_done=0, _entry_ptr=__napi_poll._entry, n=0xffff88828e4919c0, repoll=0xffffc90000374ea8[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102], weight=64, work=0) (net/core/dev.c:7506:10)
#23 napi_poll(n=0xffff88828e4919c0, repoll=0xffffc90000374e88[vmap: 0xffffc90000371000-0xffffc90000376000 caller irq_init_percpu_irqstack+0x102], do_repoll=0, have=0xffff88828e4919c0) (net/core/dev.c:7569:9)
#24 net_rx_action(sd=0xffff8890242d5180, budget=0x12c, time_limit=0x1009ed9d6, n=0xffff88828e4919c0) (net/core/dev.c:7696:13)
#25 handle_softirqs(_entry_ptr=handle_softirqs._entry, ksirqd=0, end=0xfffffffeff61262a, old_flags=0x4200042, max_restart=10, pending=8, in_hardirq=0, h=softirq_vec+0x18, softirq_bit=4, vec_nr=3) (kernel/softirq.c:579:3)
#26 __do_softirq (kernel/softirq.c:613:2)
#27 invoke_softirq (kernel/softirq.c:453:3)
#28 __irq_exit_rcu (kernel/softirq.c:680:3)
#29 irq_exit_rcu (kernel/softirq.c:696:2)
#30 common_interrupt(regs=0xffffc9000017fdc8[vmap: 0xffffc9000017c000-0xffffc90000181000 caller copy_process+0x172]) (arch/x86/kernel/irq.c:318:1)
#31 asm_common_interrupt (./arch/x86/include/asm/idtentry.h:693:0)
#32 cpuidle_enter_state(__already_done=0, dev=0xffff8890242ddf88, drv=intel_idle_driver, index=2, time_start=0x9bf5c92f347, entered_state=2, time_end=0x9bf5c9548a9) (drivers/cpuidle/cpuidle.c:292:20)
#33 cpuidle_enter(drv=intel_idle_driver, dev=0xffff8890242ddf88, index=2, ret=0) (drivers/cpuidle/cpuidle.c:389:9)
#34 call_cpuidle(drv=intel_idle_driver, dev=0xffff8890242ddf88, next_state=2) (kernel/sched/idle.c:160:9)
#35 cpuidle_idle_call(dev=0xffff8890242ddf88, drv=intel_idle_driver, next_state=2) (kernel/sched/idle.c:235:19)
#36 do_idle(cpu=5) (kernel/sched/idle.c:330:4)
#37 cpu_startup_entry(state=146) (kernel/sched/idle.c:428:3)
#38 start_secondary (arch/x86/kernel/smpboot.c:315:2)
CPU: 8 PID: 526241 COMM: kworker/u144:2
#0 crash_setup_regs(newregs=0xffffc90000410328[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102], oldregs=0) (./arch/x86/include/asm/kexec.h:108:15)
#1 __crash_kexec(regs=0) (kernel/crash_core.c:133:4)
#2 vpanic(_entry_ptr=vpanic._entry, _entry_ptr=vpanic._entry, _entry_ptr=vpanic._entry, i_next=0, state=0, _crash_kexec_post_notifiers=0) (kernel/panic.c:449:3)
#3 panic (kernel/panic.c:566:2)
#4 watchdog_timer_fn(_entry_ptr=watchdog_timer_fn._entry, regs=0xffffc90000410628[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102], softlockup_all_cpu_backtrace=0, duration=23, flags=2) (kernel/watchdog.c:832:4)
#5 __run_hrtimer(cpu_base=0xffff889024431c40, base=0xffff889024431cc0, timer=0xffff889024432c80, flags=130, fn=watchdog_timer_fn, expires_in_hardirq=0) (kernel/time/hrtimer.c:1761:12)
#6 __hrtimer_run_queues(cpu_base=0xffff889024431c40, now=0x9be2bdcdb88, flags=130, base=0xffff889024431cc0, timer=0xffff889024432c80) (kernel/time/hrtimer.c:1825:4)
#7 hrtimer_interrupt(__already_done=0, _entry_ptr=hrtimer_interrupt._entry, cpu_base=0xffff889024431c40, retries=0, flags=130, now=0x9be2bdcdb88, entry_time=0x9be2bdcdb88) (kernel/time/hrtimer.c:1887:2)
#8 local_apic_timer_interrupt(evt=0xffff88902441be40) (arch/x86/kernel/apic/apic.c:1039:2)
#9 __sysvec_apic_timer_interrupt(old_regs=0) (arch/x86/kernel/apic/apic.c:1056:2)
#10 instr_sysvec_apic_timer_interrupt(regs=0xffffc90000410628[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102]) (arch/x86/kernel/apic/apic.c:1050:1)
#11 sysvec_apic_timer_interrupt(regs=0xffffc90000410628[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102]) (arch/x86/kernel/apic/apic.c:1050:1)
#12 asm_sysvec_apic_timer_interrupt (./arch/x86/include/asm/idtentry.h:702:0)
#13 __raw_spin_unlock_irqrestore(lock=kmemleak_lock, flags=0x282) (./include/linux/spinlock_api_smp.h:152:2)
#14 _raw_spin_unlock_irqrestore(lock=kmemleak_lock, flags=0x282) (kernel/locking/spinlock.c:194:2)
#15 find_and_remove_object(alias=0, objflags=0, flags=0x282, object=0xffff88873889fa48[slab object: kmemleak_object]) (mm/kmemleak.c:637:2)
#16 delete_object_full(objflags=0) (mm/kmemleak.c:839:11)
#17 kmemleak_free_recursive(ptr=0xffff888ec771ec00[slab object: skbuff_head_cache]) (./include/linux/kmemleak.h:50:3)
#18 slab_free_hook(s=0xffff88828a294700[slab object: kmem_cache], x=0xffff888ec771ec00[slab object: skbuff_head_cache], init=0, after_rcu_delay=0) (mm/slub.c:2347:2)
#19 slab_free(s=0xffff88828a294700[slab object: kmem_cache], slab=0xffffea003b1dc780, object=0xffff888ec771ec00[slab object: skbuff_head_cache], addr=tcp_rcv_established+0x2b4) (mm/slub.c:4695:6)
#20 kmem_cache_free(s=0xffff88828a294700[slab object: kmem_cache]) (mm/slub.c:4797:2)
#21 tcp_rcv_established(sk=0xffff888c47b41a80[slab object: TCPv6], skb=0xffff888ec771ec00[slab object: skbuff_head_cache], reason=2, th=0xffff888f15bb1976[slab object: skbuff_small_head+0x76], len=32, delta=0, tcp_header_len=32) (net/ipv4/tcp_input.c:6121:5)
#22 tcp_v6_do_rcv(__warned=0, sk=0xffff888c47b41a80[slab object: TCPv6], skb=0xffff888ec771ec00[slab object: skbuff_head_cache], np=0xffff888c47b426c0[slab object: TCPv6+0xc40], opt_skb=0, dst=0xffff88839f77c800[slab object: ip6_dst_cache]) (net/ipv6/tcp_ipv6.c:1647:3)
#23 tcp_v6_rcv(skb=0xffff888ec771ec00[slab object: skbuff_head_cache], net=init_net, dif=2, sk=0xffff888c47b41a80[slab object: TCPv6], drop_reason=0, ret=0) (net/ipv6/tcp_ipv6.c:1916:9)
#24 ip6_protocol_deliver_rcu(__warned=0, net=init_net, skb=0xffff888ec771ec00[slab object: skbuff_head_cache], nexthdr=6, have_final=1, reason=2, idev=0xffff88828a090800[slab object: kmalloc-cg-2k], ipprot=net_hotdata+0xf0) (net/ipv6/ip6_input.c:438:9)
#25 ip6_input_finish (net/ipv6/ip6_input.c:489:2)
#26 NF_HOOK(pf=10, net=init_net, sk=0, skb=0xffff888ec771ec00[slab object: skbuff_head_cache], in=0xffff8882a1db4000, out=0, okfn=ip6_input_finish, ret=1) (./include/linux/netfilter.h:318:9)
#27 ip6_input(skb=0xffff888ec771ec00[slab object: skbuff_head_cache]) (net/ipv6/ip6_input.c:500:8)
#28 dst_input(skb=0xffff888ec771ec00[slab object: skbuff_head_cache]) (./include/net/dst.h:471:9)
#29 ip6_sublist_rcv_finish(head=0xffffc900004109d0[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102], skb=0xffff888ec771ec00[slab object: skbuff_head_cache]) (net/ipv6/ip6_input.c:88:3)
#30 ip6_list_rcv_finish(net=init_net, sk=0, head=0xffffc90000410a50[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102], hint=0xffff888ec771f600[slab object: skbuff_head_cache], curr_dst=0xffff88839f77c800[slab object: ip6_dst_cache], skb=0xffffc90000410a50[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102]) (net/ipv6/ip6_input.c:145:2)
#31 ip6_sublist_rcv(head=0xffffc90000410a50[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102], net=init_net) (net/ipv6/ip6_input.c:321:2)
#32 ipv6_list_rcv(head=0xffffc90000410ac0[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102], skb=0xffffc90000410ac0[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102]) (net/ipv6/ip6_input.c:355:3)
#33 __netif_receive_skb_list_ptype(head=0xffffc90000410ac0[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102], pt_prev=ipv6_packet_type, orig_dev=0xffff8882a1db4000) (net/core/dev.c:6034:3)
#34 __netif_receive_skb_list_core(head=0xffff88828e491f08, pt_curr=ipv6_packet_type, od_curr=0xffff8882a1db4000, skb=0xffff88828e491f08) (net/core/dev.c:6081:2)
#35 __netif_receive_skb_list(head=0xffff88828e491f08, skb=0xffff88828e491f08) (net/core/dev.c:6133:3)
#36 netif_receive_skb_list_internal(head=0xffff88828e491f08) (net/core/dev.c:6224:2)
#37 gro_normal_list(gro=0xffff88828e491e40) (./include/net/gro.h:532:2)
#38 gro_normal_one(gro=0xffff88828e491e40, skb=0xffff888ec771f800[slab object: skbuff_head_cache]) (./include/net/gro.h:552:3)
#39 gro_complete(__warned=0, gro=0xffff88828e491e40, skb=0xffff888ec771f800[slab object: skbuff_head_cache]) (net/core/gro.c:286:2)
#40 dev_gro_receive(__warned=0, gro=0xffff88828e491e40, skb=0xffff888ec771e900[slab object: skbuff_head_cache], bucket=7, gro_list=0xffff88828e491ef0, type=0xdd86, pp=0xffff888ec771f800[slab object: skbuff_head_cache]) (net/core/gro.c:529:3)
#41 gro_receive_skb(gro=0xffff88828e491e40, skb=0xffff888ec771e900[slab object: skbuff_head_cache]) (net/core/gro.c:631:33)
#42 bnxt_rx_pkt(__print_once=0, bp=0xffff8882a1db4b80, cpr=0xffff88828e492028, raw_cons=0xffffc90000410d7c[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102], event=0xffffc90000410d7b[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102], bnapi=0xffff88828e491e00, rxr=0xffff88828e48e840, dev=0xffff8882a1db4000, rc=0, rxcmp=0xffff88829e4b00a0, cmp_type=17, rxcmp1=0xffff88829e4b00b0, prod=0x435a, cons=0x35b, data=0xffff888f28925000, misc=0x560600, flags=0x33b2411, skb=0xffff888ec771e900[slab object: skbuff_head_cache]) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:2343:2)
#43 __bnxt_poll_work(bp=0xffff8882a1db4b80, cpr=0xffff88828e492028, budget=64, bnapi=0xffff88828e491e00, raw_cons=0x161830a, rx_pkts=8, event=17, txcmp=0xffff88829e4b00a0) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3053:10)
#44 bnxt_poll_work(bp=0xffff8882a1db4b80, cpr=0xffff88828e492028, budget=64, bnapi=0xffff88828e491e00) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3130:12)
#45 bnxt_poll(napi=0xffff88828e491e00, budget=64, bnapi=0xffff88828e491e00, bp=0xffff8882a1db4b80, cpr=0xffff88828e492028, work_done=0) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3230:16)
#46 __napi_poll(__print_once=0, __already_done=0, _entry_ptr=__napi_poll._entry, n=0xffff88828e491e00, repoll=0xffffc90000410ea8[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102], weight=64, work=0) (net/core/dev.c:7506:10)
#47 napi_poll(n=0xffff88828e491e00, repoll=0xffffc90000410e88[vmap: 0xffffc9000040d000-0xffffc90000412000 caller irq_init_percpu_irqstack+0x102], do_repoll=0, have=0xffff88828e491e00) (net/core/dev.c:7569:9)
#48 net_rx_action(sd=0xffff889024455180, budget=0x12c, time_limit=0x1009ed0fa, n=0xffff88828e491e00) (net/core/dev.c:7696:13)
#49 handle_softirqs(_entry_ptr=handle_softirqs._entry, ksirqd=0, end=0xfffffffeff612f06, old_flags=0x4208060, max_restart=10, pending=0x208, in_hardirq=0, h=softirq_vec+0x18, softirq_bit=4, vec_nr=3) (kernel/softirq.c:579:3)
#50 __do_softirq (kernel/softirq.c:613:2)
#51 invoke_softirq (kernel/softirq.c:453:3)
#52 __irq_exit_rcu (kernel/softirq.c:680:3)
#53 irq_exit_rcu (kernel/softirq.c:696:2)
#54 instr_sysvec_apic_timer_interrupt(regs=0xffffc9003c3ef928[vmap stack: 526241 (kworker/u144:2) +0x3928]) (arch/x86/kernel/apic/apic.c:1050:1)
#55 sysvec_apic_timer_interrupt(regs=0xffffc9003c3ef928[vmap stack: 526241 (kworker/u144:2) +0x3928]) (arch/x86/kernel/apic/apic.c:1050:1)
#56 asm_sysvec_apic_timer_interrupt (./arch/x86/include/asm/idtentry.h:702:0)
#57 __raw_spin_unlock_irqrestore(lock=kmemleak_lock, flags=0x282) (./include/linux/spinlock_api_smp.h:152:2)
#58 _raw_spin_unlock_irqrestore(lock=kmemleak_lock, flags=0x282) (kernel/locking/spinlock.c:194:2)
#59 find_and_remove_object(alias=0, objflags=0, flags=0x282, object=0xffff888c3074dde8[slab object: kmemleak_object]) (mm/kmemleak.c:637:2)
#60 delete_object_full(objflags=0) (mm/kmemleak.c:839:11)
#61 kmemleak_free_recursive(ptr=0xffff888294cb9400[slab object: btrfs_extent_state]) (./include/linux/kmemleak.h:50:3)
#62 slab_free_hook(s=0xffff88828da43300[slab object: kmem_cache], x=0xffff888294cb9400[slab object: btrfs_extent_state], init=0, after_rcu_delay=0) (mm/slub.c:2347:2)
#63 slab_free(s=0xffff88828da43300[slab object: kmem_cache], slab=0xffffea000a532e40, object=0xffff888294cb9400[slab object: btrfs_extent_state], addr=set_extent_bit.llvm.5807687024302053429+0x9b6) (mm/slub.c:4695:6)
#64 kmem_cache_free(s=0xffff88828da43300[slab object: kmem_cache]) (mm/slub.c:4797:2)
#65 set_extent_bit(_entry_ptr=set_extent_bit._entry, _entry_ptr=set_extent_bit._entry, tree=0xffff8882beccc1e8[slab object: btrfs_inode+0xa8], end=0xf37bfff, failed_start=0, failed_state=0, prealloc=0xffff888294cb9400[slab object: btrfs_extent_state], ret=0, exclusive_bits=0) (fs/btrfs/extent-io-tree.c:1288:2)
#66 btrfs_set_extent_bit(tree=0xffff8882beccc1e8[slab object: btrfs_inode+0xa8], end=0xf37bfff) (fs/btrfs/extent-io-tree.c:1297:9)
#67 btrfs_set_extent_delalloc(end=0xf37bfff) (fs/btrfs/inode.c:2722:9)
#68 relocate_one_folio(rc=0xffff888cef79f000[slab object: kmalloc-2k], ra=0xffff888f0c881c40[slab object: kmalloc-32], cluster=0xffff888cef79f128[slab object: kmalloc-2k+0x128], inode=0xffff8882beccc4e0[slab object: btrfs_inode+0x3a0], fs_info=0xffff88828a10c000, offset=0x2689d00000, folio=0xffffea0006669280, folio_start=0xf37b000, folio_end=0xf37bfff, cur=0xf37b000, cached_state=0xffff888294cb9300[slab object: btrfs_extent_state], extent_end=0x10180fff, clamped_start=0xf37b000, clamped_len=0x1000, clamped_end=0xf37bfff) (fs/btrfs/relocation.c:2889:9)
#69 relocate_file_extent_cluster(rc=0xffff888cef79f000[slab object: kmalloc-2k], cluster=0xffff888cef79f128[slab object: kmalloc-2k+0x128], offset=0x2689d00000, cluster_nr=0, ret=0) (fs/btrfs/relocation.c:2979:9)
#70 relocate_block_group(rc=0xffff888cef79f000[slab object: kmalloc-2k], fs_info=0xffff88828a10c000, trans=0xffff8882e407d790[free slab object: btrfs_trans_handle], path=0xffff8882e354d2a0[slab object: btrfs_path]) (fs/btrfs/relocation.c:3641:9)
#71 btrfs_relocate_block_group(_entry_ptr=btrfs_relocate_block_group._entry, fs_info=0xffff88828a10c000, verbose=1, rw=1, err=0, bg=0xffff888c30852800[slab object: kmalloc-1k], rc=0xffff888cef79f000[slab object: kmalloc-2k]) (fs/btrfs/relocation.c:3997:9)
#72 btrfs_relocate_chunk(fs_info=0xffff88828a10c000, chunk_offset=0x2689d00000, root=0xffff8882b5a88000[slab object: kmalloc-4k]) (fs/btrfs/volumes.c:3451:8)
#73 btrfs_reclaim_bgs_work(work=0xffff88828a10dfb0, fs_info=0xffff88828a10c000, bg=0xffff888c30852800[slab object: kmalloc-1k], space_info=0xffff8882b5a92800[slab object: kmalloc-1k], reserved=0, used=0x10181000) (fs/btrfs/block-group.c:1981:9)
#74 process_one_work(worker=0xffff8885d7f02f00[slab object: kmalloc-192], work=0xffff88828a10dfb0, pwq=0xffff888288208c00[slab object: pool_workqueue], pool=0xffff888284a60800[slab object: kmalloc-2k], work_data=0xffff888288208c05[slab object: pool_workqueue+0x5], rcu_start_depth=0, lockdep_start_depth=0) (kernel/workqueue.c:3236:2)
#75 process_scheduled_works(worker=0xffff8885d7f02f00[slab object: kmalloc-192]) (kernel/workqueue.c:3319:3)
#76 worker_thread(__worker=0xffff8885d7f02f00[slab object: kmalloc-192], worker=0xffff8885d7f02f00[slab object: kmalloc-192], pool=0xffff888284a60800[slab object: kmalloc-2k]) (kernel/workqueue.c:3400:4)
#77 kthread(threadfn=worker_thread, data=0xffff8885d7f02f00[slab object: kmalloc-192], self=0xffff888753ef8400[slab object: kmalloc-512], ret=-4) (kernel/kthread.c:463:9)
#78 ret_from_fork(regs=0xffffc9003c3eff58[vmap stack: 526241 (kworker/u144:2) +0x3f58], fn=kthread, fn_arg=0xffff88873b51ddc0[free slab object: kmalloc-64]) (arch/x86/kernel/process.c:148:3)
CPU: 9 PID: 73 COMM: ksoftirqd/9
#0 queued_spin_lock_slowpath(lock=kmemleak_lock, node=0xffff8890244d4b00, tail=0x280000) (kernel/locking/qspinlock.c:328:8)
#1 queued_spin_lock (./include/asm-generic/qspinlock.h:114:2)
#2 do_raw_spin_lock (kernel/locking/spinlock_debug.c:116:2)
#3 __raw_spin_lock_irqsave(flags=0x282) (./include/linux/spinlock_api_smp.h:111:2)
#4 _raw_spin_lock_irqsave(lock=kmemleak_lock) (kernel/locking/spinlock.c:162:9)
#5 find_and_remove_object(ptr=0xffff888f22fb0c80[slab object: skbuff_small_head], alias=0) (mm/kmemleak.c:635:2)
#6 delete_object_full(ptr=0xffff888f22fb0c80[slab object: skbuff_small_head]) (mm/kmemleak.c:839:11)
#7 kmemleak_free_recursive(ptr=0xffff888f22fb0c80[slab object: skbuff_small_head]) (./include/linux/kmemleak.h:50:3)
#8 slab_free_hook(s=0xffff88828a294900[slab object: kmem_cache], x=0xffff888f22fb0c80[slab object: skbuff_small_head], init=0, after_rcu_delay=0) (mm/slub.c:2347:2)
#9 slab_free(s=0xffff88828a294900[slab object: kmem_cache], slab=0xffffea003c8bec00, object=0xffff888f22fb0c80[slab object: skbuff_small_head], addr=skb_release_data+0x132) (mm/slub.c:4695:6)
#10 kmem_cache_free(s=0xffff88828a294900[slab object: kmem_cache]) (mm/slub.c:4797:2)
#11 skb_release_data(skb=0xffff888f0da9ece0[slab object: skbuff_fclone_cache+0xe0], reason=2, shinfo=0xffff888f22fb0dc0[slab object: skbuff_small_head+0x140]) (net/core/skbuff.c:1086:2)
#12 skb_release_all(skb=0xffff888f0da9ece0[slab object: skbuff_fclone_cache+0xe0], reason=2) (net/core/skbuff.c:1151:3)
#13 __kfree_skb(skb=0xffff888f0da9ece0[slab object: skbuff_fclone_cache+0xe0]) (net/core/skbuff.c:1165:2)
#14 kfree_skb_reason (./include/linux/skbuff.h:1275:2)
#15 dev_kfree_skb_any_reason (net/core/dev.c:3442:3)
#16 dev_consume_skb_any (./include/linux/netdevice.h:4148:2)
#17 __bnxt_tx_int(bp=0xffff8882a1db4b80, txr=0xffff8882c067b300[slab object: kmalloc-4k+0x300], txq=0xffff8882a3220380, tx_bytes=0x146f6, tx_pkts=0x13f, rc=0, tx_buf=0xffffc90005c29be8[vmap: 0xffffc90005c27000-0xffffc90005c3c000 caller bnxt_alloc_ring+0x1e3]) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:875:3)
#18 bnxt_tx_int(bp=0xffff8882a1db4b80, bnapi=0xffff88828e4908c0, i=0, txr=0xffff8882c067b300[slab object: kmalloc-4k+0x300]) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:895:12)
#19 __bnxt_poll_work_done (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3108:3)
#20 bnxt_poll_work(bp=0xffff8882a1db4b80, cpr=0xffff88828e490ae8, budget=64, bnapi=0xffff88828e4908c0, rx_pkts=64) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3138:2)
#21 bnxt_poll(napi=0xffff88828e4908c0, budget=64, bnapi=0xffff88828e4908c0, bp=0xffff8882a1db4b80, cpr=0xffff88828e490ae8, work_done=0) (drivers/net/ethernet/broadcom/bnxt/bnxt.c:3230:16)
#22 __napi_poll(__print_once=0, __already_done=0, _entry_ptr=__napi_poll._entry, n=0xffff88828e4908c0, repoll=0xffffc9000042fd58[vmap stack: 73 (ksoftirqd/9) +0x3d58], weight=64, work=0) (net/core/dev.c:7506:10)
#23 napi_poll(n=0xffff88828e4908c0, repoll=0xffffc9000042fd38[vmap stack: 73 (ksoftirqd/9) +0x3d38], do_repoll=0, have=0xffff88828e4908c0) (net/core/dev.c:7569:9)
#24 net_rx_action(sd=0xffff8890244d5180, budget=0x12c, time_limit=0x1009ed992, n=0xffff88828e4908c0) (net/core/dev.c:7696:13)
#25 handle_softirqs(_entry_ptr=handle_softirqs._entry, ksirqd=1, end=0xfffffffeff61266e, old_flags=0x4208040, max_restart=10, pending=8, in_hardirq=1, h=softirq_vec+0x18, softirq_bit=4, vec_nr=3) (kernel/softirq.c:579:3)
#26 run_ksoftirqd (kernel/softirq.c:968:3)
#27 smpboot_thread_fn(data=0xffff888288166810[slab object: kmalloc-16], td=0xffff888288166810[slab object: kmalloc-16], ht=softirq_threads) (kernel/smpboot.c:160:4)
#28 kthread(threadfn=smpboot_thread_fn, data=0xffff888288166810[slab object: kmalloc-16], self=0xffff888289ac0c00[slab object: kmalloc-512], ret=-4) (kernel/kthread.c:463:9)
#29 ret_from_fork(regs=0xffffc9000042ff58[vmap stack: 73 (ksoftirqd/9) +0x3f58], fn=kthread, fn_arg=0xffff88828998fa00[slab object: kmalloc-64]) (arch/x86/kernel/process.c:148:3)
CPU: 11 PID: 7324 COMM: bar
#0 queued_spin_lock_slowpath(lock=kmemleak_lock, node=0xffff8890245d4b00, tail=0x300000, next=0, __PTR=0xffff8890245d4b08) (kernel/locking/qspinlock.c:291:3)
#1 queued_spin_lock (./include/asm-generic/qspinlock.h:114:2)
#2 do_raw_spin_lock (kernel/locking/spinlock_debug.c:116:2)
#3 __raw_spin_lock_irqsave(flags=0x282) (./include/linux/spinlock_api_smp.h:111:2)
#4 _raw_spin_lock_irqsave(lock=kmemleak_lock) (kernel/locking/spinlock.c:162:9)
#5 __create_object(ptr=0xffff888f0e530000[slab object: skbuff_fclone_cache], size=0x1c8, min_count=1, objflags=0) (mm/kmemleak.c:783:2)
#6 kmemleak_alloc_recursive(ptr=0xffff888f0e530000[slab object: skbuff_fclone_cache], min_count=1) (./include/linux/kmemleak.h:44:3)
#7 slab_post_alloc_hook(s=0xffff88828a294800[slab object: kmem_cache], lru=0, flags=0xcc0, size=1, p=0xffffc9000c93fa88[vmap stack: 7324 (ScribeIO0) +0x3a88]) (mm/slub.c:4195:3)
#8 slab_alloc_node(s=0xffff88828a294800[slab object: kmem_cache], lru=0, gfpflags=0xcc0, addr=__alloc_skb+0x17e, orig_size=0x1c8, object=0xffff888f0e530000[slab object: skbuff_fclone_cache]) (mm/slub.c:4240:2)
#9 kmem_cache_alloc_node_noprof(s=0xffff88828a294800[slab object: kmem_cache], gfpflags=0xcc0, node=-1) (mm/slub.c:4292:14)
#10 __alloc_skb(gfp_mask=0xcc0, flags=1, node=-1, cache=0xffff88828a294800[slab object: kmem_cache], _old=0) (net/core/skbuff.c:659:9)
#11 alloc_skb_fclone(size=0x100) (./include/linux/skbuff.h:1386:9)
#12 tcp_stream_alloc_skb(sk=0xffff88866ca58000[slab object: TCPv6], force_schedule=1) (net/ipv4/tcp.c:892:8)
#13 tcp_sendmsg_locked(sk=0xffff88866ca58000[slab object: TCPv6], msg=0xffffc9000c93fe68[vmap stack: 7324 (ScribeIO0) +0x3e68], binding=0, tp=0xffff88866ca58000[slab object: TCPv6], uarg=0, copied=0, mss_now=0x7c00, process_backlog=0, zc=0, skb=0) (net/ipv4/tcp.c:1198:10)
#14 tcp_sendmsg(sk=0xffff88866ca58000[slab object: TCPv6], msg=0xffffc9000c93fe68[vmap stack: 7324 (ScribeIO0) +0x3e68], size=64) (net/ipv4/tcp.c:1393:8)
#15 sock_sendmsg_nosec(sock=0xffff888d1bf33180[slab object: sock_inode_cache], msg=0xffffc9000c93fe68[vmap stack: 7324 (ScribeIO0) +0x3e68]) (net/socket.c:714:12)
#16 __sock_sendmsg(sock=0xffff888d1bf33180[slab object: sock_inode_cache], msg=0xffffc9000c93fe68[vmap stack: 7324 (ScribeIO0) +0x3e68]) (net/socket.c:729:16)
#17 ____sys_sendmsg(sock=0xffff888d1bf33180[slab object: sock_inode_cache], msg_sys=0xffffc9000c93fe68[vmap stack: 7324 (ScribeIO0) +0x3e68], used_address=0, ctl_buf=0xffffc9000c93fc68[vmap stack: 7324 (ScribeIO0) +0x3c68], ctl_len=0) (net/socket.c:2614:8)
#18 ___sys_sendmsg(sock=0xffff888d1bf33180[slab object: sock_inode_cache], msg_sys=0xffffc9000c93fe68[vmap stack: 7324 (ScribeIO0) +0x3e68], flags=0x4040, used_address=0, allowed_msghdr_flags=0, iov=0, err=0) (net/socket.c:2668:8)
#19 __sys_sendmsg(msg=0x7efd4fbf6170, flags=0x4040, forbid_cmsg_compat=1) (net/socket.c:2700:9)
#20 __do_sys_sendmsg(msg=0x7efd4fbf6170, flags=0x4040) (net/socket.c:2705:9)
#21 __se_sys_sendmsg(msg=0x7efd4fbf6170) (net/socket.c:2703:1)
#22 __x64_sys_sendmsg (net/socket.c:2703:1)
#23 do_syscall_x64(regs=0xffffc9000c93ff58[vmap stack: 7324 (ScribeIO0) +0x3f58], nr=46) (arch/x86/entry/syscall_64.c:63:14)
#24 do_syscall_64(regs=0xffffc9000c93ff58[vmap stack: 7324 (ScribeIO0) +0x3f58], nr=46) (arch/x86/entry/syscall_64.c:94:7)
#25 entry_SYSCALL_64 (arch/x86/entry/entry_64.S:121:0)
CPU: 17 PID: 501192 COMM: blah
#0 memset_orig (arch/x86/lib/memset_64.S:70:0)
#1 INIT_HLIST_NODE (./include/linux/list.h:937:11)
#2 inode_init_once (fs/inode.c:506:2)
#3 setup_object(s=0xffff8882889e6600[slab object: kmem_cache]) (mm/slub.c:2475:3)
#4 allocate_slab(s=0xffff8882889e6600[slab object: kmem_cache], slab=0xffffea003c025c00, shuffle=0, idx=8) (mm/slub.c:2697:11)
#5 new_slab(s=0xffff8882889e6600[slab object: kmem_cache], node=-1) (mm/slub.c:2714:9)
#6 ___slab_alloc(s=0xffff8882889e6600[slab object: kmem_cache], gfpflags=0xcc0, node=-1, addr=proc_alloc_inode.llvm.9965772860669215490+0xa7, c=0xffff8890248dba60, orig_size=0x4c8, slab=0) (mm/slub.c:3901:9)
#7 __slab_alloc(s=0xffff8882889e6600[slab object: kmem_cache], gfpflags=0xcc0, node=-1, addr=proc_alloc_inode.llvm.9965772860669215490+0xa7, c=0xffff8890248dba60) (mm/slub.c:3992:6)
#8 __slab_alloc_node(s=0xffff8882889e6600[slab object: kmem_cache], gfpflags=0xcc0, node=-1, addr=proc_alloc_inode.llvm.9965772860669215490+0xa7, orig_size=0x4c8, object=0) (mm/slub.c:4067:12)
#9 slab_alloc_node(s=0xffff8882889e6600[slab object: kmem_cache], gfpflags=0xcc0, node=-1, addr=proc_alloc_inode.llvm.9965772860669215490+0xa7, orig_size=0x4c8, init=0) (mm/slub.c:4228:11)
#10 kmem_cache_alloc_lru_noprof(s=0xffff8882889e6600[slab object: kmem_cache], lru=0xffff8883c24b67f8[slab object: kmalloc-4k+0x7f8], gfpflags=0xcc0) (mm/slub.c:4259:14)
#11 proc_alloc_inode(_old=0) (fs/proc/inode.c:57:7)
#12 alloc_inode(sb=0xffff8883c24b6000[slab object: kmalloc-4k], ops=proc_sops) (fs/inode.c:346:11)
#13 new_inode (fs/inode.c:1145:10)
#14 proc_pid_make_inode (fs/proc/base.c:1953:10)
#15 proc_fd_instantiate(dentry=0xffff8882bee20000[slab object: dentry], ptr=0xffffc90036f07e10[vmap stack: 501192 (FuncSched) +0x3e10], data=0xffffc90036f07e10[vmap stack: 501192 (FuncSched) +0x3e10]) (fs/proc/fd.c:209:10)
#16 proc_fill_cache(ctx=0xffffc90036f07e98[vmap stack: 501192 (FuncSched) +0x3e98], name=0xffffc90036f07e1d[vmap stack: 501192 (FuncSched) +0x3e1d], len=4, instantiate=proc_fd_instantiate, ptr=0xffffc90036f07e10[vmap stack: 501192 (FuncSched) +0x3e10], dir=0xffff8885fe54f6d8[slab object: dentry], type=0, ino=1, child=0xffff8882bee20000[slab object: dentry]) (fs/proc/base.c:2138:10)
#17 proc_readfd_common(file=0xffff888cdfe75800[slab object: filp], ctx=0xffffc90036f07e98[vmap stack: 501192 (FuncSched) +0x3e98], instantiate=proc_fd_instantiate, p=0xffff888756c9a480[slab object: task_struct]) (fs/proc/fd.c:275:8)
#18 iterate_dir(file=0xffff888cdfe75800[slab object: filp], ctx=0xffffc90036f07e98[vmap stack: 501192 (FuncSched) +0x3e98], res=-2, inode=0xffff88866ea5bed8[slab object: proc_inode_cache+0x48]) (fs/readdir.c:108:9)
#19 __do_sys_getdents64(dirent=0x7fee18e75030, count=0x8000) (fs/readdir.c:410:10)
#20 __se_sys_getdents64(dirent=0x7fee18e75030, count=0x8000) (fs/readdir.c:396:1)
#21 do_syscall_x64(regs=0xffffc90036f07f58[vmap stack: 501192 (FuncSched) +0x3f58], nr=217) (arch/x86/entry/syscall_64.c:63:14)
#22 do_syscall_64(regs=0xffffc90036f07f58[vmap stack: 501192 (FuncSched) +0x3f58], nr=217) (arch/x86/entry/syscall_64.c:94:7)
#23 entry_SYSCALL_64 (arch/x86/entry/entry_64.S:121:0)
CPU: 18 PID: 11401 COMM: blabla
#0 stack_trace_consume_entry(cookie=0xffffc90021923a78[vmap stack: 11401 (below) +0x3a78], addr=do_sys_openat2+0x6c, c=0xffffc90021923a78[vmap stack: 11401 (below) +0x3a78]) (kernel/stacktrace.c:95:1)
#1 arch_stack_walk(consume_entry=stack_trace_consume_entry, cookie=0xffffc90021923a78[vmap stack: 11401 (below) +0x3a78], task=0xffff8886352bc900[slab object: task_struct]) (arch/x86/kernel/stacktrace.c:27:17)
#2 stack_trace_save (kernel/stacktrace.c:122:2)
#3 set_track_prepare (mm/kmemleak.c:655:15)
#4 __alloc_object(_entry_ptr=__alloc_object._entry) (mm/kmemleak.c:701:25)
#5 __create_object(ptr=0xffff8882b2ec4e80[slab object: seq_file]) (mm/kmemleak.c:779:11)
#6 kmemleak_alloc_recursive(ptr=0xffff8882b2ec4e80[slab object: seq_file], min_count=1) (./include/linux/kmemleak.h:44:3)
#7 slab_post_alloc_hook(s=0xffff8882889e6500[slab object: kmem_cache], lru=0, flags=0xdc0, size=1, p=0xffffc90021923bb8[vmap stack: 11401 (below) +0x3bb8]) (mm/slub.c:4195:3)
#8 slab_alloc_node(s=0xffff8882889e6500[slab object: kmem_cache], lru=0, gfpflags=0xdc0, node=-1, addr=single_open+0x12a, orig_size=232, object=0xffff8882b2ec4e80[slab object: seq_file]) (mm/slub.c:4240:2)
#9 kmem_cache_alloc_noprof(s=0xffff8882889e6500[slab object: kmem_cache], gfpflags=0xdc0) (mm/slub.c:4247:14)
#10 seq_open(file=0xffff888c5bd35e00[slab object: filp], op=0xffff8882b4c509e0[slab object: kmalloc-cg-32], _old=0) (fs/seq_file.c:63:6)
#11 single_open(file=0xffff888c5bd35e00[slab object: filp], data=0xffff8883c9d943a8[slab object: proc_inode_cache+0x48], op=0xffff8882b4c509e0[slab object: kmalloc-cg-32], res=-12) (fs/seq_file.c:583:9)
#12 do_dentry_open(f=0xffff888c5bd35e00[slab object: filp], open=proc_single_open, inode=0xffff8883c9d943a8[slab object: proc_inode_cache+0x48]) (fs/open.c:965:11)
#13 vfs_open(file=0xffff888c5bd35e00[slab object: filp]) (fs/open.c:1095:8)
#14 do_open(nd=0xffffc90021923d78[vmap stack: 11401 (below) +0x3d78], file=0xffff888c5bd35e00[slab object: filp], op=0xffffc90021923e94[vmap stack: 11401 (below) +0x3e94], open_flag=0x8000, idmap=nop_mnt_idmap) (fs/namei.c:3887:11)
#15 path_openat(__warned=0, nd=0xffffc90021923d78[vmap stack: 11401 (below) +0x3d78], op=0xffffc90021923e94[vmap stack: 11401 (below) +0x3e94], flags=0x101, file=0xffff888c5bd35e00[slab object: filp]) (fs/namei.c:4046:12)
#16 do_filp_open(op=0xffffc90021923e94[vmap stack: 11401 (below) +0x3e94], flags=1) (fs/namei.c:4073:9)
#17 do_sys_openat2(dfd=-100, how=0xffffc90021923ed8[vmap stack: 11401 (below) +0x3ed8], tmp=0xffff88828c5bd000[slab object: names_cache], fd=158) (fs/open.c:1435:20)
#18 do_sys_open (fs/open.c:1450:9)
#19 __do_sys_openat (fs/open.c:1466:9)
#20 __se_sys_openat (fs/open.c:1461:1)
#21 __x64_sys_openat (fs/open.c:1461:1)
#22 do_syscall_x64(regs=0xffffc90021923f58[vmap stack: 11401 (below) +0x3f58], nr=0x101) (arch/x86/entry/syscall_64.c:63:14)
#23 do_syscall_64(regs=0xffffc90021923f58[vmap stack: 11401 (below) +0x3f58], nr=0x101) (arch/x86/entry/syscall_64.c:94:7)
#24 entry_SYSCALL_64 (arch/x86/entry/entry_64.S:121:0)
CPU: 20 PID: 7652 COMM: blabla
#0 check_cb_ovld(rdp=0xffff889024a54b40) (kernel/rcu/tree.c:3084:6)
#1 __call_rcu_common(head=0xffff888f268e7d80[slab object: kmemleak_object+0x80], lazy_in=0, flags=0x246, rdp=0xffff889024a54b40, lazy=0) (kernel/rcu/tree.c:3142:2)
#2 call_rcu(head=0xffff888f268e7d80[slab object: kmemleak_object+0x80]) (kernel/rcu/tree.c:3243:2)
#3 kmemleak_free_recursive(ptr=0xffff88871ff7a400[slab object: btrfs_extent_state]) (./include/linux/kmemleak.h:50:3)
#4 slab_free_hook(s=0xffff88828da43300[slab object: kmem_cache], x=0xffff88871ff7a400[slab object: btrfs_extent_state], init=0, after_rcu_delay=0) (mm/slub.c:2347:2)
#5 slab_free(s=0xffff88828da43300[slab object: kmem_cache], slab=0xffffea001c7fde80, object=0xffff88871ff7a400[slab object: btrfs_extent_state], addr=set_extent_bit.llvm.5807687024302053429+0x9b6) (mm/slub.c:4695:6)
#6 kmem_cache_free(s=0xffff88828da43300[slab object: kmem_cache]) (mm/slub.c:4797:2)
#7 set_extent_bit(_entry_ptr=set_extent_bit._entry, _entry_ptr=set_extent_bit._entry, tree=0xffff88866b2fa970[slab object: btrfs_inode+0xa8], end=0x1106fff, failed_start=0, failed_state=0, prealloc=0xffff88871ff7a400[slab object: btrfs_extent_state], ret=0, exclusive_bits=0) (fs/btrfs/extent-io-tree.c:1288:2)
#8 btrfs_set_extent_bit(tree=0xffff88866b2fa970[slab object: btrfs_inode+0xa8], end=0x1106fff) (fs/btrfs/extent-io-tree.c:1297:9)
#9 btrfs_set_extent_delalloc(end=0x1106fff) (fs/btrfs/inode.c:2722:9)
#10 btrfs_dirty_folio(_entry_ptr=btrfs_dirty_folio._entry, _entry_ptr=btrfs_dirty_folio._entry, inode=0xffff88866b2fa8c8[slab object: btrfs_inode], folio=0xffffea0006ea5180, ret=0, fs_info=0xffff88828a10c000, end_pos=0x11063ac, extra_bits=0, isize=0x1106354, start_pos=0x1106000, num_bytes=0x1000, end_of_last_block=0x1106fff) (fs/btrfs/file.c:104:8)
#11 copy_one_range(inode=0xffff88866b2fa8c8[slab object: btrfs_inode], iter=0xffffc9000d2c7e28[vmap stack: 7652 (Collection-25) +0x3e28], start=0x1106354, fs_info=0xffff88828a10c000, cached_state=0xffff88871ff7aa00[slab object: btrfs_extent_state], reserved_start=0x1106000, folio=0xffffea0006ea5180, only_release_metadata=0, bdp_flags=0, reserved_len=0x1000, extents_locked=1, copied=88) (fs/btrfs/file.c:1325:8)
#12 btrfs_buffered_write(iocb=0xffffc9000d2c7e50[vmap stack: 7652 (Collection-25) +0x3e50], iter=0xffffc9000d2c7e28[vmap stack: 7652 (Collection-25) +0x3e28], inode=0xffff88866b2fac68[slab object: btrfs_inode+0x3a0], num_written=0, ilock_flags=0, old_isize=0x1106354, pos=0x1106354) (fs/btrfs/file.c:1389:9)
#13 btrfs_do_write_iter(iocb=0xffffc9000d2c7e50[vmap stack: 7652 (Collection-25) +0x3e50]) (fs/btrfs/file.c:1463:17)
#14 new_sync_write(filp=0xffff888c3b1ed680[slab object: filp], buf=0x7fea6a6f5f80, len=88, ppos=0xffffc9000d2c7eb8[vmap stack: 7652 (Collection-25) +0x3eb8]) (fs/read_write.c:593:8)
#15 vfs_write(file=0xffff888c3b1ed680[slab object: filp], buf=0x7fea6a6f5f80, count=88, pos=0xffffc9000d2c7eb8[vmap stack: 7652 (Collection-25) +0x3eb8]) (fs/read_write.c:686:9)
#16 ksys_write(buf=0x7fea6a6f5f80, count=88, ret=-9, ppos=0xffffc9000d2c7eb8[vmap stack: 7652 (Collection-25) +0x3eb8]) (fs/read_write.c:738:9)
#17 do_syscall_x64(regs=0xffffc9000d2c7f58[vmap stack: 7652 (Collection-25) +0x3f58], nr=1) (arch/x86/entry/syscall_64.c:63:14)
#18 do_syscall_64(regs=0xffffc9000d2c7f58[vmap stack: 7652 (Collection-25) +0x3f58], nr=1) (arch/x86/entry/syscall_64.c:94:7)
#19 entry_SYSCALL_64 (arch/x86/entry/entry_64.S:121:0)
CPU: 22 PID: 493448 COMM: blan
#0 __lookup_object(_entry_ptr=__lookup_object._entry, ptr=0xffff888f3a251180[slab object: skbuff_small_head], alias=0, rb=0xffff888f1f155370[slab object: kmemleak_object+0x68]) (mm/kmemleak.c:426:2)
#1 __find_and_remove_object(ptr=0xffff888f3a251180[slab object: skbuff_small_head], alias=0, objflags=0) (mm/kmemleak.c:617:11)
#2 find_and_remove_object(ptr=0xffff888f3a251180[slab object: skbuff_small_head], alias=0, objflags=0, flags=0x282) (mm/kmemleak.c:636:11)
#3 delete_object_full(ptr=0xffff888f3a251180[slab object: skbuff_small_head], objflags=0) (mm/kmemleak.c:839:11)
#4 kmemleak_free_recursive(ptr=0xffff888f3a251180[slab object: skbuff_small_head]) (./include/linux/kmemleak.h:50:3)
#5 slab_free_hook(s=0xffff88828a294900[slab object: kmem_cache], x=0xffff888f3a251180[slab object: skbuff_small_head], init=0, after_rcu_delay=0) (mm/slub.c:2347:2)
#6 slab_free(s=0xffff88828a294900[slab object: kmem_cache], slab=0xffffea003ce89400, object=0xffff888f3a251180[slab object: skbuff_small_head], addr=skb_release_data+0x132) (mm/slub.c:4695:6)
#7 kmem_cache_free(s=0xffff88828a294900[slab object: kmem_cache]) (mm/slub.c:4797:2)
#8 skb_release_data(skb=0xffff88883e71dc00[slab object: skbuff_fclone_cache], reason=2, shinfo=0xffff888f3a2512c0[slab object: skbuff_small_head+0x140]) (net/core/skbuff.c:1086:2)
#9 skb_release_all(skb=0xffff88883e71dc00[slab object: skbuff_fclone_cache], reason=2) (net/core/skbuff.c:1151:3)
#10 __kfree_skb(skb=0xffff88883e71dc00[slab object: skbuff_fclone_cache]) (net/core/skbuff.c:1165:2)
#11 tcp_wmem_free_skb(sk=0xffff888b18b20000[slab object: TCPv6], skb=0xffff88883e71dc00[slab object: skbuff_fclone_cache]) (./include/net/tcp.h:308:2)
#12 tcp_rtx_queue_purge(sk=0xffff888b18b20000[slab object: TCPv6], p=0, skb=0xffff88883e71dc00[slab object: skbuff_fclone_cache]) (net/ipv4/tcp.c:3304:3)
#13 tcp_write_queue_purge(sk=0xffff888b18b20000[slab object: TCPv6]) (net/ipv4/tcp.c:3317:2)
#14 tcp_v4_destroy_sock(__warned=0, sk=0xffff888b18b20000[slab object: TCPv6], tp=0xffff888b18b20000[slab object: TCPv6]) (net/ipv4/tcp_ipv4.c:2564:2)
#15 inet_csk_destroy_sock(sk=0xffff888b18b20000[slab object: TCPv6]) (net/ipv4/inet_connection_sock.c:1294:2)
#16 __tcp_close(__warned=0, sk=0xffff888b18b20000[slab object: TCPv6]) (net/ipv4/tcp.c:3262:3)
#17 tcp_close(sk=0xffff888b18b20000[slab object: TCPv6], timeout=0) (net/ipv4/tcp.c:3274:2)
#18 inet_release(sock=0xffff888ec7d2f380[slab object: sock_inode_cache], sk=0xffff888b18b20000[slab object: TCPv6]) (net/ipv4/af_inet.c:435:3)
#19 __sock_release(sock=0xffff888ec7d2f380[slab object: sock_inode_cache], inode=0xffff888ec7d2f440[slab object: sock_inode_cache+0xc0], ops=inet6_stream_ops, owner=0) (net/socket.c:649:3)
#20 sock_close(inode=0xffff888ec7d2f440[slab object: sock_inode_cache+0xc0]) (net/socket.c:1439:2)
#21 __fput(file=0xffff888c3b1ed800[slab object: filp], dentry=0xffff888736fa5728[slab object: dentry], mnt=0xffff88828a2dc320[slab object: mnt_cache+0x20], inode=0xffff888ec7d2f440[slab object: sock_inode_cache+0xc0], mode=0xc2e0003) (fs/file_table.c:468:3)
#22 __do_sys_close(file=0xffff888c3b1ed800[slab object: filp], retval=0) (fs/open.c:1587:2)
#23 __se_sys_close (fs/open.c:1572:1)
#24 do_syscall_x64(regs=0xffffc9002b627f58[vmap stack: 493448 (SR_TLSConnectio) +0x3f58], nr=3) (arch/x86/entry/syscall_64.c:63:14)
#25 do_syscall_64(regs=0xffffc9002b627f58[vmap stack: 493448 (SR_TLSConnectio) +0x3f58], nr=3) (arch/x86/entry/syscall_64.c:94:7)
#26 entry_SYSCALL_64 (arch/x86/entry/entry_64.S:121:0)
CPU: 24 PID: 4908 COMM: bla
#0 csd_lock_wait_toolong(csd=0xffffc90009a3bc60[vmap stack: 4908 (dynoKernelMon) +0x3c60], ts0=0x9be54c8a8aa, cpu=-1, flags=17) (kernel/smp.c:241:14)
#1 __csd_lock_wait(csd=0xffffc90009a3bc60[vmap stack: 4908 (dynoKernelMon) +0x3c60], nmessages=0, bug_id=0, ts0=0x9be54c8a8aa, ts1=0x9be54c8a8aa) (kernel/smp.c:328:7)
#2 csd_lock_wait(csd=0xffffc90009a3bc60[vmap stack: 4908 (dynoKernelMon) +0x3c60]) (kernel/smp.c:338:3)
#3 smp_call_function_single(func=__perf_event_read, info=0xffffc90009a3bcf8[vmap stack: 4908 (dynoKernelMon) +0x3cf8], wait=1, csd=0xffffc90009a3bc60[vmap stack: 4908 (dynoKernelMon) +0x3c60], err=0) (kernel/smp.c:687:3)
#4 perf_event_read(event=0xffff88828a42d9a0[slab object: perf_event], group=0, ret=0) (kernel/events/core.c:4862:9)
#5 __perf_event_read_value(event=0xffff88828a42d9a0[slab object: perf_event], enabled=0xffffc90009a3bda8[vmap stack: 4908 (dynoKernelMon) +0x3da8], running=0xffffc90009a3bda0[vmap stack: 4908 (dynoKernelMon) +0x3da0], total=0) (kernel/events/core.c:5890:8)
#6 perf_read_one(event=0xffff88828a42d9a0[slab object: perf_event], read_format=7, buf=0x7f0b9b1bdf98, n=0, enabled=0, running=0) (kernel/events/core.c:6050:16)
#7 __perf_read(event=0xffff88828a42d9a0[slab object: perf_event], buf=0x7f0b9b1bdf98, count=32, read_format=7) (kernel/events/core.c:6103:9)
#8 perf_read(buf=0x7f0b9b1bdf98, count=32, event=0xffff88828a42d9a0[slab object: perf_event], ctx=0xffff88902443e370) (kernel/events/core.c:6120:8)
#9 vfs_read(file=0xffff88833c218780[slab object: filp], buf=0x7f0b9b1bdf98, pos=0xffffc90009a3beb8[vmap stack: 4908 (dynoKernelMon) +0x3eb8]) (fs/read_write.c:570:9)
#10 ksys_read(buf=0x7f0b9b1bdf98, count=32, ret=-9, ppos=0xffffc90009a3beb8[vmap stack: 4908 (dynoKernelMon) +0x3eb8]) (fs/read_write.c:715:9)
#11 do_syscall_x64(regs=0xffffc90009a3bf58[vmap stack: 4908 (dynoKernelMon) +0x3f58], nr=0) (arch/x86/entry/syscall_64.c:63:14)
#12 do_syscall_64(regs=0xffffc90009a3bf58[vmap stack: 4908 (dynoKernelMon) +0x3f58], nr=0) (arch/x86/entry/syscall_64.c:94:7)
#13 entry_SYSCALL_64 (arch/x86/entry/entry_64.S:121:0)
CPU: 28 PID: 480960 COMM: kworker/u144:17
#0 csd_lock_wait_toolong(csd=0xffff88902445c880, ts0=0x9be2c355318, cpu=-1) (kernel/smp.c:238:23)
#1 __csd_lock_wait(csd=0xffff88902445c880, nmessages=0, bug_id=0, ts0=0x9be2c355318, ts1=0x9be2c355318) (kernel/smp.c:328:7)
#2 csd_lock_wait (kernel/smp.c:338:3)
#3 smp_call_function_many_cond(mask=__cpu_online_mask, func=do_sync_core.llvm.7180805200076648557, info=0, scf_flags=3, this_cpu=28, cfd=0xffff889024e54d80) (kernel/smp.c:877:4)
#4 on_each_cpu_cond_mask(cond_func=0, func=do_sync_core.llvm.7180805200076648557, info=0, mask=__cpu_online_mask, scf_flags=3) (kernel/smp.c:1044:2)
#5 on_each_cpu(info=0, wait=1) (./include/linux/smp.h:71:2)
#6 smp_text_poke_sync_each_cpu (arch/x86/kernel/alternative.c:2653:2)
#7 smp_text_poke_batch_finish (arch/x86/kernel/alternative.c:2863:2)
#8 arch_jump_label_transform_apply (arch/x86/kernel/jump_label.c:146:2)
#9 __jump_label_update (kernel/jump_label.c:521:2)
#10 static_key_enable_cpuslocked(key=kfence_allocation_key) (kernel/jump_label.c:210:3)
#11 static_key_enable (kernel/jump_label.c:223:2)
#12 toggle_allocation_gate (mm/kfence/core.c:850:2)
#13 process_one_work(worker=0xffff88873925c840[slab object: kmalloc-192], work=kfence_timer, pwq=0xffff888288209a00[slab object: pool_workqueue], pool=0xffff888284a60800[slab object: kmalloc-2k], work_data=0xffff888288209a05[slab object: pool_workqueue+0x5], rcu_start_depth=0, lockdep_start_depth=0) (kernel/workqueue.c:3236:2)
#14 process_scheduled_works(worker=0xffff88873925c840[slab object: kmalloc-192]) (kernel/workqueue.c:3319:3)
#15 worker_thread(__worker=0xffff88873925c840[slab object: kmalloc-192], worker=0xffff88873925c840[slab object: kmalloc-192], pool=0xffff888284a60800[slab object: kmalloc-2k]) (kernel/workqueue.c:3400:4)
#16 kthread(threadfn=worker_thread, data=0xffff88873925c840[slab object: kmalloc-192], self=0xffff888726e6d800[slab object: kmalloc-512], ret=-4) (kernel/kthread.c:463:9)
#17 ret_from_fork(regs=0xffffc9002acdff58[vmap stack: 480960 (kworker/u144:17) +0x3f58], fn=kthread, fn_arg=0xffff88871eadab00[slab object: kmalloc-64]) (arch/x86/kernel/process.c:148:3)
CPU: 31 PID: 353 COMM: kmemleak
#0 queued_spin_lock_slowpath(lock=kmemleak_lock, node=0xffff889024fd4b00, tail=0x800000, next=0, __PTR=0xffff889024fd4b08) (kernel/locking/qspinlock.c:291:3)
#1 queued_spin_lock (./include/asm-generic/qspinlock.h:114:2)
#2 do_raw_spin_lock (kernel/locking/spinlock_debug.c:116:2)
#3 __raw_spin_lock_irqsave(flags=134) (./include/linux/spinlock_api_smp.h:111:2)
#4 _raw_spin_lock_irqsave(lock=kmemleak_lock) (kernel/locking/spinlock.c:162:9)
#5 scan_block(start=0xffff88893ced2b60[slab object: lsm_inode_cache], end=0xffff88893ced2b79[slab object: lsm_inode_cache+0x19]) (mm/kmemleak.c:1530:2)
#6 scan_object(object=0xffff888c3c736910[slab object: kmemleak_object], flags=0x282) (mm/kmemleak.c:1609:4)
#7 scan_gray_list(object=0xffff888c3c736910[slab object: kmemleak_object]) (mm/kmemleak.c:1648:4)
#8 kmemleak_scan(__warned=0, __warned=0, __warned=0, __warned=0, _entry_ptr=kmemleak_scan._entry, new_leaks=0) (mm/kmemleak.c:1800:2)
#9 kmemleak_scan_thread(first_run=0, _entry_ptr=kmemleak_scan_thread._entry, _entry_ptr=kmemleak_scan_thread._entry, timeout=0x927c0) (mm/kmemleak.c:1903:3)
#10 kthread(threadfn=kmemleak_scan_thread, data=0, self=0xffff88828ba5d200[slab object: kmalloc-512], ret=-4) (kernel/kthread.c:463:9)
#11 ret_from_fork(regs=0xffffc90002177f58[vmap stack: 353 (kmemleak) +0x3f58], fn=kthread, fn_arg=0xffff8890063f0000[slab object: kmemleak_object]) (arch/x86/kernel/process.c:148:3)
>
> > Should we have a wrapper around raw_spin_lock_irqsave(kmemleak_lock,
> > flags), that would defer printk at all?
> >
> > Then, we can simply replace the raw_spin_lock_irqsave() by the helper,
> > avoiding spreading these printk_deferred_enter() in the kmemleak code.
> >
> > For instance, something as this completely untested code, just to show
> > the idea.
> >
> > void kmemleak_lock(unsigned long *flags) {
> > printk_deferred_enter();
> > raw_spin_lock_irqsave(&kmemleak_lock, flags);
> > }
> >
> > void kmemleak_lock(unsigned long flags) {
> > raw_spin_unlock_irqrestore(&kmemleak_lock, flags);
> > printk_deferred_exit();
> > }
>
> The way we added the printk deferring recently is around the actual
> printk() calls. Given that you can't get an interrupt under
> raw_spin_lock_irqsave(),
My concern is when printk() is called with kmemleak_lock held(). Something as:
raw_spin_lock_irqsave(&kmemleak_lock, flags);
-> printk()
This is instant deadlock when netconsole is enabled. Given that
netconsole tries to allocate memory when flushing. Similarly to commit
47b0f6d8f0d2be ("mm/kmemleak: avoid deadlock by moving pr_warn() outside
kmemleak_lock").
The hack above would guarantee that all printks() inside kmemleak_lock
critical area to be deferred, and not executed inline.
> I don't think printk_deferred_exit() would
> trigger a console flush. So we could simply add them around those
> kmemleak_warn() or pr_*() calls rather than together with the spinlocks.
AFAIK printk_deferred_exit() only tells printk that all further
printks() could be executed "inline", instead of being deferred to be
executed on a workqueue. This is done setting an per-cpu
`printk_context`.
While the patch above tells printk to defer all printk a workqueue when
it happens from inside kemleak_lock region.
> But we do need to be able to reproduce the problem and show that any
> potential patch fixes it.
ack!
Thanks for the reply,
--breno
next prev parent reply other threads:[~2025-09-26 14:57 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-08-22 7:35 Gu Bowen
2025-08-26 8:23 ` Breno Leitao
2025-08-26 19:37 ` Catalin Marinas
2025-08-26 23:23 ` Waiman Long
2025-09-19 10:37 ` Breno Leitao
2025-09-25 14:33 ` Catalin Marinas
2025-09-26 14:57 ` Breno Leitao [this message]
2025-09-29 14:07 ` John Ogness
2025-09-29 14:40 ` Breno Leitao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=kuq7guzalpqj5bxe2vt6s3kirrq4sg5ozwcim6ewnzpxhuxm4l@yfgb44nbcisz \
--to=leitao@debian.org \
--cc=akpm@linux-foundation.org \
--cc=catalin.marinas@arm.com \
--cc=gregkh@linuxfoundation.org \
--cc=gubowen5@huawei.com \
--cc=john.ogness@linutronix.de \
--cc=linux-mm@kvack.org \
--cc=llong@redhat.com \
--cc=lujialin4@huawei.com \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox