From: Hannes Reinecke <hare@suse.com>
To: Sagi Grimberg <sagi@grimberg.me>, Matthew Wilcox <willy@infradead.org>
Cc: "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
linux-nvme@lists.infradead.org, linux-mm@kvack.org
Subject: Re: Kernel oops with 6.14 when enabling TLS
Date: Mon, 3 Mar 2025 08:48:09 +0100 [thread overview]
Message-ID: <509dd4d3-85e9-40b2-a967-8c937909a1bf@suse.com> (raw)
In-Reply-To: <08c29e4b-2f71-4b6d-8046-27e407214d8c@suse.com>
On 2/28/25 11:47, Hannes Reinecke wrote:
> Hi Sagi,
>
> enabling TLS on latest linus tree reliably crashes my system:
>
> [ 487.018058] ------------[ cut here ]------------
> [ 487.024046] WARNING: CPU: 9 PID: 6159 at mm/slub.c:4719
> free_large_kmalloc+0x15/0xa0
> [ 487.033549] Modules linked in: tls(E) nvme_tcp(E) af_packet(E)
> iscsi_ibft(E) iscsi_boot_sysfs(E) amd_atl(E) intel_rapl_msr(E)
> intel_rapl_common(E) amd64_edac(E) edac_mce_amd(E) nls_iso8859_1(E)
> nls_cp437(E) dax_hmem(E) vfat(E) cxl_acpi(E) fat(E) kvm_amd(E)
> ipmi_ssif(E) cxl_port(E) xfs(E) tg3(E) cxl_core(E) ipmi_si(E) i40e(E)
> kvm(E) einj(E) wmi_bmof(E) acpi_cpufreq(E) ipmi_devintf(E) libphy(E)
> k10temp(E) libie(E) i2c_piix4(E) i2c_smbus(E) ipmi_msghandler(E)
> i2c_designware_platform(E) i2c_designware_core(E) button(E)
> nvme_fabrics(E) nvme_keyring(E) fuse(E) efi_pstore(E) configfs(E)
> dmi_sysfs(E) ip_tables(E) x_tables(E) ahci(E) libahci(E)
> ghash_clmulni_intel(E) libata(E) sha512_ssse3(E) sd_mod(E)
> sha256_ssse3(E) ast(E) sha1_ssse3(E) drm_client_lib(E) scsi_dh_emc(E)
> i2c_algo_bit(E) aesni_intel(E) drm_shmem_helper(E) scsi_dh_rdac(E)
> crypto_simd(E) cryptd(E) xhci_pci(E) drm_kms_helper(E) scsi_dh_alua(E)
> nvme(E) sg(E) xhci_hcd(E) nvme_core(E) scsi_mod(E) drm(E) nvme_auth(E)
> scsi_common(E) usbcore(E) sp5100_tco(E) ccp(E)
> [ 487.033696] wmi(E) btrfs(E) blake2b_generic(E) xor(E) raid6_pq(E)
> efivarfs(E)
> [ 487.033707] CPU: 9 UID: 0 PID: 6159 Comm: nvme Kdump: loaded Tainted:
> G W E 6.14.0-rc4-default+ #292
> f1e35f01b401c038558e67f3c2d644747de50dbd
> [ 487.033713] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE
> [ 487.033715] Hardware name: Lenovo ThinkSystem SR655V3/SB27B09914,
> BIOS KAE111E-2.10 04/11/2023
> [ 487.033717] RIP: 0010:free_large_kmalloc+0x15/0xa0
> [ 487.033722] Code: 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90
> 90 90 90 90 0f 1f 44 00 00 55 53 48 89 fb 48 83 ec 08 48 8b 07 a8 40 75
> 4b <0f> 0b 80 3d 24 4e 8e 01 00 ba 00 f0 ff ff 74 5d 9c 58 0f 1f 40 00
> [ 487.205280] RSP: 0018:ff4de44e432a7b70 EFLAGS: 00010246
> [ 487.205284] RAX: 000fffffc0000000 RBX: ffd659b280210b80 RCX:
> ff42118489e0cd80
> [ 487.205286] RDX: 0000000000000000 RSI: ff4211848842e000 RDI:
> ffd659b280210b80
> [ 487.205288] RBP: ff4de44e432a7be0 R08: 0000000000000001 R09:
> 0000000000000002
> [ 487.205289] R10: ff4de44e432a7c00 R11: 0000000000000104 R12:
> ffd659b280210b80
> [ 487.205291] R13: ff4211848842e000 R14: ff421186d0696520 R15:
> ff421186e19c4000
> [ 487.205292] FS: 00007f66b8ffd800(0000) GS:ff4211874d980000(0000)
> knlGS:0000000000000000
> [ 487.205294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 487.205296] CR2: 00007f66b9150d7e CR3: 0000000281a5e005 CR4:
> 0000000000771ef0
> [ 487.205298] PKRU: 55555554
> [ 487.205306] Call Trace:
> [ 487.205309] <TASK>
> [ 487.205314] ? __warn+0x85/0x130
> [ 487.296763] ? free_large_kmalloc+0x15/0xa0
> [ 487.296772] ? report_bug+0xf8/0x1e0
> [ 487.296779] ? handle_bug+0x50/0xa0
> [ 487.296783] ? exc_invalid_op+0x13/0x60
> [ 487.296786] ? asm_exc_invalid_op+0x16/0x20
> [ 487.296796] ? free_large_kmalloc+0x15/0xa0
> [ 487.296801] kfree+0x234/0x320
> [ 487.332065] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 487.332075] ? nvmf_connect_admin_queue+0x105/0x1a0 [nvme_fabrics
> 34d997d53c805aa2fae8e8baee6a736e8da38358]
> [ 487.332081] ? nvmf_connect_admin_queue+0xa1/0x1a0 [nvme_fabrics
> 34d997d53c805aa2fae8e8baee6a736e8da38358]
> [ 487.332084] nvmf_connect_admin_queue+0x105/0x1a0 [nvme_fabrics
> 34d997d53c805aa2fae8e8baee6a736e8da38358]
> [ 487.332093] nvme_tcp_start_queue+0x18f/0x310 [nvme_tcp
> 68f6be106f52ac467179f8a0922f02aeb6fa1f1c]
> [ 487.332102] nvme_tcp_setup_ctrl+0xf8/0x700 [nvme_tcp
> 68f6be106f52ac467179f8a0922f02aeb6fa1f1c]
> [ 487.394479] ? nvme_change_ctrl_state+0x99/0x1b0 [nvme_core
> 22f0ce18ead628230226a9b87ebf48eb576bf299]
> [ 487.394495] nvme_tcp_create_ctrl+0x2e3/0x4d0 [nvme_tcp
> 68f6be106f52ac467179f8a0922f02aeb6fa1f1c]
> [ 487.394503] nvmf_dev_write+0x323/0x3d0 [nvme_fabrics
> 34d997d53c805aa2fae8e8baee6a736e8da38358]
> [ 487.394508] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 487.394514] vfs_write+0xd9/0x430
> [ 487.394521] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 487.394523] ? __handle_mm_fault+0x7da/0xca0
> [ 487.394531] ksys_write+0x68/0xe0
> [ 487.394536] do_syscall_64+0x74/0x160
> [ 487.394543] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 487.394545] ? __count_memcg_events+0x98/0x130
> [ 487.394550] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 487.476947] ? count_memcg_events.constprop.163+0x1a/0x30
> [ 487.476956] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 487.476960] ? handle_mm_fault+0xa1/0x290
> [ 487.476966] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 487.476968] ? do_user_addr_fault+0x56b/0x830
> [ 487.476975] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 487.476977] ? exc_page_fault+0x68/0x150
> [ 487.476983] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 487.476989] RIP: 0033:0x7f66b91216f0
> [ 487.476994] Code: 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90
> 90 90 90 90 90 90 90 90 90 80 3d 19 c3 0e 00 00 74 17 b8 01 00 00 00 0f
> 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 48 89
> [ 487.551619] RSP: 002b:00007ffef5393ed8 EFLAGS: 00000202 ORIG_RAX:
> 0000000000000001
> [ 487.551623] RAX: ffffffffffffffda RBX: 000055e61c2113b0 RCX:
> 00007f66b91216f0
> [ 487.551625] RDX: 00000000000000ed RSI: 000055e61c2113b0 RDI:
> 0000000000000003
> [ 487.551627] RBP: 0000000000000003 R08: 00000000000000ed R09:
> 000055e61c2113b0
> [ 487.551628] R10: 0000000000000000 R11: 0000000000000202 R12:
> 00000000000000ed
> [ 487.551630] R13: 00007f66b9380008 R14: 000055e61c20a980 R15:
> 000055e61c20b100
> [ 487.551637] </TASK>
> [ 487.551639] ---[ end trace 0000000000000000 ]---
> [ 487.551642] object pointer: 0x00000000346cb6fc
> [ 487.554112] nvme nvme1: creating 32 I/O queues.
> [ 489.396262] nvme nvme1: mapped 32/0/0 default/read/poll queues.
> [ 489.405197] Oops: general protection fault, probably for non-
> canonical address 0xdead000000000100: 0000 [#1] PREEMPT SMP NOPTI
> [ 489.418790] CPU: 9 UID: 0 PID: 6159 Comm: nvme Kdump: loaded Tainted:
> G W E 6.14.0-rc4-default+ #292
> f1e35f01b401c038558e67f3c2d644747de50dbd
> [ 489.435212] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE
> [ 489.441381] Hardware name: Lenovo ThinkSystem SR655V3/SB27B09914,
> BIOS KAE111E-2.10 04/11/2023
> [ 489.451841] RIP: 0010:__rmqueue_pcplist+0xe1/0xc80
> [ 489.458016] Code: 06 48 83 e8 08 48 89 44 24 70 48 8b 45 00 48 39 c5
> 0f 84 72 01 00 00 48 8b 55 00 48 8b 32 48 8b 4a 08 48 8d 42 f8 48 89 4e
> 08 <48> 89 31 48 b9 00 01 00 00 00 00 ad de 48 89 0a 48 8b 4c 24 20 48
> [ 489.479905] RSP: 0018:ff4de44e432a7688 EFLAGS: 00010293
> [ 489.486567] RAX: ffd659b280210a00 RBX: ff4211874d9bf400 RCX:
> dead000000000100
> [ 489.495370] RDX: ffd659b280210a08 RSI: ffd659b281ed7e08 RDI:
> ff421184fffd60c0
> [ 489.504174] RBP: ff4211874d9bf4b0 R08: ff4211874d9bf400 R09:
> ff4211874d9bf4b0
> [ 489.512976] R10: 0000000000002acd R11: 0000000000000002 R12:
> 0000000000000003
> [ 489.521779] R13: 0000000000000003 R14: 0000000000252800 R15:
> ff421184fffd60c0
> [ 489.530584] FS: 00007f66b8ffd800(0000) GS:ff4211874d980000(0000)
> knlGS:0000000000000000
> [ 489.540460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 489.547700] CR2: 00007f66b9150d7e CR3: 0000000281a5e005 CR4:
> 0000000000771ef0
> [ 489.556502] PKRU: 55555554
> [ 489.560337] Call Trace:
> [ 489.563876] <TASK>
> [ 489.567022] ? __die_body+0x1a/0x60
> [ 489.571728] ? die_addr+0x38/0x60
> [ 489.576241] ? exc_general_protection+0x19e/0x430
> [ 489.582325] ? asm_exc_general_protection+0x22/0x30
> [ 489.588603] ? __rmqueue_pcplist+0xe1/0xc80
> [ 489.594092] ? __rmqueue_pcplist+0x51b/0xc80
> [ 489.599687] get_page_from_freelist+0xe10/0x1680
> [ 489.605675] __alloc_frozen_pages_noprof+0x171/0x340
> [ 489.612048] new_slab+0x90/0x4d0
> [ 489.616466] ___slab_alloc+0x6f3/0xb20
> [ 489.621469] ? sbitmap_init_node+0x77/0x1a0
> [ 489.626961] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 489.633134] ? _get_random_bytes.part.18+0x90/0x120
> [ 489.639409] ? __slab_alloc.isra.98+0x22/0x40
> [ 489.645101] __slab_alloc.isra.98+0x22/0x40
> [ 489.650597] __kmalloc_node_noprof+0x218/0x510
> [ 489.656380] ? sbitmap_init_node+0x77/0x1a0
> [ 489.661874] ? sbitmap_init_node+0x77/0x1a0
> [ 489.667362] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 489.673534] sbitmap_init_node+0x77/0x1a0
> [ 489.678830] sbitmap_queue_init_node+0x24/0x150
> [ 489.684709] blk_mq_init_tags+0x7e/0x110
> [ 489.689915] blk_mq_alloc_map_and_rqs+0x44/0x320
> [ 489.695898] __blk_mq_alloc_map_and_rqs+0x3b/0x60
> [ 489.701973] blk_mq_alloc_tag_set+0x1f1/0x380
> [ 489.707662] nvme_alloc_io_tag_set+0xc2/0x1a0 [nvme_core
> 22f0ce18ead628230226a9b87ebf48eb576bf299]
> [ 489.718534] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 489.724707] ? nvme_tcp_alloc_queue+0x293/0x7b0 [nvme_tcp
> 68f6be106f52ac467179f8a0922f02aeb6fa1f1c]
> [ 489.735676] ? __pfx_nvme_tcp_tls_done+0x10/0x10 [nvme_tcp
> 68f6be106f52ac467179f8a0922f02aeb6fa1f1c]
> [ 489.746737] nvme_tcp_setup_ctrl+0x3ee/0x700 [nvme_tcp
> 68f6be106f52ac467179f8a0922f02aeb6fa1f1c]
> [ 489.757401] nvme_tcp_create_ctrl+0x2e3/0x4d0 [nvme_tcp
> 68f6be106f52ac467179f8a0922f02aeb6fa1f1c]
> [ 489.768165] nvmf_dev_write+0x323/0x3d0 [nvme_fabrics
> 34d997d53c805aa2fae8e8baee6a736e8da38358]
> [ 489.778732] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 489.784906] vfs_write+0xd9/0x430
> [ 489.789422] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 489.795591] ? __handle_mm_fault+0x7da/0xca0
> [ 489.801183] ksys_write+0x68/0xe0
> [ 489.805700] do_syscall_64+0x74/0x160
> [ 489.810616] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 489.816794] ? __count_memcg_events+0x98/0x130
> [ 489.822580] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 489.828750] ? count_memcg_events.constprop.163+0x1a/0x30
> [ 489.835606] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 489.841777] ? handle_mm_fault+0xa1/0x290
> [ 489.847070] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 489.853241] ? do_user_addr_fault+0x56b/0x830
> [ 489.858928] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 489.865101] ? exc_page_fault+0x68/0x150
> [ 489.870302] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 489.876769] RIP: 0033:0x7f66b91216f0
> [ 489.881573] Code: 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90
> 90 90 90 90 90 90 90 90 90 80 3d 19 c3 0e 00 00 74 17 b8 01 00 00 00 0f
> 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 48 89
> [ 489.903462] RSP: 002b:00007ffef5393ed8 EFLAGS: 00000202 ORIG_RAX:
> 0000000000000001
> [ 489.912761] RAX: ffffffffffffffda RBX: 000055e61c2113b0 RCX:
> 00007f66b91216f0
> [ 489.921564] RDX: 00000000000000ed RSI: 000055e61c2113b0 RDI:
> 0000000000000003
> [ 489.930366] RBP: 0000000000000003 R08: 00000000000000ed R09:
> 000055e61c2113b0
> [ 489.939169] R10: 0000000000000000 R11: 0000000000000202 R12:
> 00000000000000ed
> [ 489.947970] R13: 00007f66b9380008 R14: 000055e61c20a980 R15:
> 000055e61c20b100
> [ 489.956783] </TASK>
> [ 489.960024] Modules linked in: tls(E) nvme_tcp(E) af_packet(E)
> iscsi_ibft(E) iscsi_boot_sysfs(E) amd_atl(E) intel_rapl_msr(E)
> intel_rapl_common(E) amd64_edac(E) edac_mce_amd(E) nls_iso8859_1(E)
> nls_cp437(E) dax_hmem(E) vfat(E) cxl_acpi(E) fat(E) kvm_amd(E)
> ipmi_ssif(E) cxl_port(E) xfs(E) tg3(E) cxl_core(E) ipmi_si(E) i40e(E)
> kvm(E) einj(E) wmi_bmof(E) acpi_cpufreq(E) ipmi_devintf(E) libphy(E)
> k10temp(E) libie(E) i2c_piix4(E) i2c_smbus(E) ipmi_msghandler(E)
> i2c_designware_platform(E) i2c_designware_core(E) button(E)
> nvme_fabrics(E) nvme_keyring(E) fuse(E) efi_pstore(E) configfs(E)
> dmi_sysfs(E) ip_tables(E) x_tables(E) ahci(E) libahci(E)
> ghash_clmulni_intel(E) libata(E) sha512_ssse3(E) sd_mod(E)
> sha256_ssse3(E) ast(E) sha1_ssse3(E) drm_client_lib(E) scsi_dh_emc(E)
> i2c_algo_bit(E) aesni_intel(E) drm_shmem_helper(E) scsi_dh_rdac(E)
> crypto_simd(E) cryptd(E) xhci_pci(E) drm_kms_helper(E) scsi_dh_alua(E)
> nvme(E) sg(E) xhci_hcd(E) nvme_core(E) scsi_mod(E) drm(E) nvme_auth(E)
> scsi_common(E) usbcore(E) sp5100_tco(E) ccp(E)
> [ 489.960207] wmi(E) btrfs(E) blake2b_generic(E) xor(E) raid6_pq(E)
> efivarfs(E)
>
> Haven't found a culprit for that one for now, started bisecting.
> Just wanted to report that as a heads-up, maybe you have some idea.
>
bisect is pointing to
9aec2fb0fd5e ("slab: allocate frozen pages")
and, indeed, reverting this patch on top of linus current resolves
the issue.
Sorry Matthew.
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.com +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
next parent reply other threads:[~2025-03-03 7:48 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <08c29e4b-2f71-4b6d-8046-27e407214d8c@suse.com>
2025-03-03 7:48 ` Hannes Reinecke [this message]
2025-03-03 11:06 ` Hannes Reinecke
2025-03-03 12:57 ` Hannes Reinecke
2025-03-03 13:57 ` Matthew Wilcox
2025-03-03 14:05 ` Hannes Reinecke
2025-03-03 14:27 ` Matthew Wilcox
2025-03-03 14:42 ` Matthew Wilcox
2025-03-03 15:12 ` Vlastimil Babka
2025-03-03 15:39 ` Hannes Reinecke
2025-03-03 15:48 ` Matthew Wilcox
2025-03-03 16:15 ` Vlastimil Babka
2025-03-03 22:02 ` Vlastimil Babka
2025-03-04 7:58 ` Hannes Reinecke
2025-03-04 8:18 ` Vlastimil Babka
2025-03-04 10:20 ` Hannes Reinecke
2025-03-04 10:26 ` Vlastimil Babka
2025-03-04 15:11 ` Hannes Reinecke
2025-03-04 15:29 ` Vlastimil Babka
2025-03-04 16:20 ` Hannes Reinecke
2025-03-04 16:14 ` Matthew Wilcox
2025-03-04 16:32 ` Hannes Reinecke
2025-03-04 16:53 ` Matthew Wilcox
2025-03-04 18:05 ` Matthew Wilcox
2025-03-04 18:31 ` Vlastimil Babka
2025-03-04 19:39 ` Hannes Reinecke
2025-03-04 19:44 ` Vlastimil Babka
2025-03-05 7:14 ` Hannes Reinecke
2025-03-05 8:20 ` Hannes Reinecke
2025-03-05 8:58 ` Vlastimil Babka
2025-03-05 11:43 ` Hannes Reinecke
2025-03-05 18:11 ` Networking people smell funny and make poor life choices Matthew Wilcox
2025-03-06 0:46 ` Cong Wang
2025-03-12 15:09 ` Christoph Hellwig
2025-03-12 18:28 ` James R. Bergsten
2025-03-13 9:43 ` David Laight
2025-03-06 9:15 ` Kernel oops with 6.14 when enabling TLS Vlastimil Babka
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=509dd4d3-85e9-40b2-a967-8c937909a1bf@suse.com \
--to=hare@suse.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-nvme@lists.infradead.org \
--cc=sagi@grimberg.me \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox