[regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Linux regression tracking (Thorsten Leemhuis)]

[to the next one replying: please drop stable@vger.kernel.org from the
CC, this is just noise there, as this is most likely a mainline regression]

[thread starts here:
https://lore.kernel.org/all/BD22A15A-9216-4FA0-82DF-C7BBF8EE642E@gmail.com/
]

On 23.08.24 13:21, Piotr Oniszczuk wrote:
> 
> In my development i’m using ryzen9 based builder machine.
> OS is ArchLinux.
> It worked perfectly stable with 6.8.2 kernel.
> 
> Recently I updated to 6.10.6  kernel and….started to have regular oops at heavy compilations (12c/24t loaded 8..12h constantly compiling)
> 
> Only single change is kernel: 6.8.2->6.10.6
> 6.10.6 is vanilla mainline (no any ArchLinux patches)
> 
> When i have ooops - dmesg is like below.
> 
> For me this looks like regression...

Many thx for the report. And yes, it looks like one. I CCed the zswap
and the mm folks, which a bit of luck they might have an idea. But this
sort of error sometimes is caused by changes in other areas of the
kernel and they just manifest in zswap. If that is the case nobody might
look into this unless you are able to provide more details, like the
result of a bisction
(https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html
) -- that's just how it is...

Ciao, Thorsten


> [75864.693223] br0: port 1(enp5s0) entered blocking state
> [75864.693226] br0: port 1(enp5s0) entered forwarding state
> [86041.349844] ------------[ cut here ]------------
> [86041.349850] kernel BUG at mm/zswap.c:1005!
> [86041.349862] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
> [86041.349867] CPU: 5 PID: 2798071 Comm: llvm-tblgen Not tainted 6.10.6-12 #1 349ceb515693b41153483eac7819a5fb2832d2bf
> [86041.349872] Hardware name: To Be Filled By O.E.M. B450M Pro4-F R2.0/B450M Pro4-F R2.0, BIOS P10.08 01/19/2024
> [86041.349876] RIP: 0010:zswap_decompress+0x1ef/0x200
> [86041.349884] Code: ef e8 95 2a ce ff 84 c0 0f 85 1f ff ff ff e9 fb fe ff ff 0f 0b 48 8d 7b 10 e8 0d a9 a4 00 c7 43 10 00 00 00 00 8b 43 30 eb 86 <0f> 0b 0f 0b e8 f8 9b a3 00 0f 1f 84 00 00 00 00 00 90 90 90 90 90
> [86041.349889] RSP: 0000:ffffb98f823ebb90 EFLAGS: 00010282
> [86041.349892] RAX: 00000000ffffffea RBX: ffff9bf22e8c1e08 RCX: ffff9bef137774ba
> [86041.349894] RDX: 0000000000000002 RSI: 0000000000000438 RDI: ffff9bf22e8b2af0
> [86041.349897] RBP: ffff9bef58cd2b98 R08: ffff9bee8baf07e0 R09: ffff9bef13777080
> [86041.349899] R10: 0000000000000022 R11: ffff9bee8baf1000 R12: fffff782422ebc00
> [86041.349902] R13: ffff9bef13777080 R14: ffff9bef01e3d6e0 R15: ffff9bf22e8c1e48
> [86041.349904] FS:  00007f4bda31d280(0000) GS:ffff9bf22e880000(0000) knlGS:0000000000000000
> [86041.349908] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [86041.349910] CR2: 000000001665d010 CR3: 0000000191a2c000 CR4: 0000000000350ef0
> [86041.349914] Call Trace:
> [86041.349918]  <TASK>
> [86041.349920]  ? die+0x36/0x90
> [86041.349925]  ? do_trap+0xdd/0x100
> [86041.349929]  ? zswap_decompress+0x1ef/0x200
> [86041.349932]  ? do_error_trap+0x6a/0x90
> [86041.349935]  ? zswap_decompress+0x1ef/0x200
> [86041.349938]  ? exc_invalid_op+0x50/0x70
> [86041.349943]  ? zswap_decompress+0x1ef/0x200
> [86041.349946]  ? asm_exc_invalid_op+0x1a/0x20
> [86041.349951]  ? zswap_decompress+0x1ef/0x200
> [86041.349955]  zswap_load+0x109/0x120
> [86041.349958]  swap_read_folio+0x64/0x450
> [86041.349963]  swapin_readahead+0x463/0x4e0
> [86041.349967]  do_swap_page+0x436/0xd70
> [86041.349972]  ? __pte_offset_map+0x1b/0x180
> [86041.349976]  __handle_mm_fault+0x85d/0x1070
> [86041.349979]  ? sched_tick+0xee/0x2f0
> [86041.349985]  handle_mm_fault+0x18d/0x320
> [86041.349988]  do_user_addr_fault+0x177/0x6a0
> [86041.349993]  exc_page_fault+0x7e/0x180
> [86041.349996]  asm_exc_page_fault+0x26/0x30
> [86041.350000] RIP: 0033:0x7453b9
> [86041.350019] Code: 00 48 8d 0c 49 4c 8d 04 ca 48 8b 0f 4c 39 c2 75 19 e9 7f 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 48 83 c2 18 49 39 d0 74 6b <48> 39 0a 75 f2 48 89 84 24 90 00 00 00 4c 39 73 10 0f 84 2f 02 00
> [86041.350024] RSP: 002b:00007ffe67b93c80 EFLAGS: 00010206
> [86041.350027] RAX: 0000000016659250 RBX: 00007ffe67b93db0 RCX: 000000000f1aad40
> [86041.350030] RDX: 000000001665d010 RSI: 00007ffe67b93cd8 RDI: 00007ffe67b93cd0
> [86041.350032] RBP: 0000000000000001 R08: 000000001665d088 R09: 0000000000000000
> [86041.350035] R10: 00007f4bda030610 R11: 00007f4bda0d6200 R12: 0000000016661210
> [86041.350038] R13: 00007ffe67b94a58 R14: 000000000ba280a8 R15: 0000000000000006
> [86041.350041]  </TASK>
> [86041.350043] Modules linked in: tls rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs rpcrdma rdma_cm iw_cm ib_cm ib_core br_netfilter iptable_filter xt_physdev tun bridge stp llc ext4 crc16 mbcache jbd2 amd_atl intel_rapl_msr intel_rapl_common cfg80211 edac_mce_amd kvm_amd rfkill kvm crct10dif_pclmul crc32_pclmul polyval_clmulni r8169 polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 realtek sha256_ssse3 sha1_ssse3 aesni_intel mdio_devres crypto_simd sp5100_tco k10temp gpio_amdpt cryptd wmi_bmof pcspkr ccp libphy i2c_piix4 acpi_cpufreq rapl zenpower ryzen_smu gpio_generic mac_hid nfsd auth_rpcgss nfs_acl lockd grace nct6775 nct6775_core hwmon_vid sg sunrpc crypto_user fuse dm_mod loop nfnetlink bpf_preload ip_tables x_tables xfs libcrc32c crc32c_generic drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm drm_exec mxm_wmi nvme crc32c_intel drm_display_helper xhci_pci nvme_core xhci_pci_renesas wmi virtio_net net_failover failover dimlib virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev
> [86041.350106]  [last unloaded: nouveau]
> [86041.350125] ---[ end trace 0000000000000000 ]---
> [86041.350128] RIP: 0010:zswap_decompress+0x1ef/0x200
> [86041.350131] Code: ef e8 95 2a ce ff 84 c0 0f 85 1f ff ff ff e9 fb fe ff ff 0f 0b 48 8d 7b 10 e8 0d a9 a4 00 c7 43 10 00 00 00 00 8b 43 30 eb 86 <0f> 0b 0f 0b e8 f8 9b a3 00 0f 1f 84 00 00 00 00 00 90 90 90 90 90
> [86041.350137] RSP: 0000:ffffb98f823ebb90 EFLAGS: 00010282
> [86041.350139] RAX: 00000000ffffffea RBX: ffff9bf22e8c1e08 RCX: ffff9bef137774ba
> [86041.350142] RDX: 0000000000000002 RSI: 0000000000000438 RDI: ffff9bf22e8b2af0
> [86041.350145] RBP: ffff9bef58cd2b98 R08: ffff9bee8baf07e0 R09: ffff9bef13777080
> [86041.350147] R10: 0000000000000022 R11: ffff9bee8baf1000 R12: fffff782422ebc00
> [86041.350150] R13: ffff9bef13777080 R14: ffff9bef01e3d6e0 R15: ffff9bf22e8c1e48
> [86041.350152] FS:  00007f4bda31d280(0000) GS:ffff9bf22e880000(0000) knlGS:0000000000000000
> [86041.350156] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [86041.350158] CR2: 000000001665d010 CR3: 0000000191a2c000 CR4: 0000000000350ef0
> [86041.350162] ------------[ cut here ]------------
> [86041.350164] WARNING: CPU: 5 PID: 2798071 at kernel/exit.c:825 do_exit+0x88b/0xac0
> [86041.350170] Modules linked in: tls rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs rpcrdma rdma_cm iw_cm ib_cm ib_core br_netfilter iptable_filter xt_physdev tun bridge stp llc ext4 crc16 mbcache jbd2 amd_atl intel_rapl_msr intel_rapl_common cfg80211 edac_mce_amd kvm_amd rfkill kvm crct10dif_pclmul crc32_pclmul polyval_clmulni r8169 polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 realtek sha256_ssse3 sha1_ssse3 aesni_intel mdio_devres crypto_simd sp5100_tco k10temp gpio_amdpt cryptd wmi_bmof pcspkr ccp libphy i2c_piix4 acpi_cpufreq rapl zenpower ryzen_smu gpio_generic mac_hid nfsd auth_rpcgss nfs_acl lockd grace nct6775 nct6775_core hwmon_vid sg sunrpc crypto_user fuse dm_mod loop nfnetlink bpf_preload ip_tables x_tables xfs libcrc32c crc32c_generic drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm drm_exec mxm_wmi nvme crc32c_intel drm_display_helper xhci_pci nvme_core xhci_pci_renesas wmi virtio_net net_failover failover dimlib virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev
> [86041.350211]  [last unloaded: nouveau]
> [86041.350231] CPU: 5 PID: 2798071 Comm: llvm-tblgen Tainted: G      D            6.10.6-12 #1 349ceb515693b41153483eac7819a5fb2832d2bf
> [86041.350236] Hardware name: To Be Filled By O.E.M. B450M Pro4-F R2.0/B450M Pro4-F R2.0, BIOS P10.08 01/19/2024
> [86041.350239] RIP: 0010:do_exit+0x88b/0xac0
> [86041.350242] Code: 89 a3 48 06 00 00 48 89 6c 24 10 48 8b 83 68 08 00 00 e9 ff fd ff ff 48 8b bb 28 06 00 00 31 f6 e8 da e1 ff ff e9 a1 fd ff ff <0f> 0b e9 eb f7 ff ff 4c 89 e6 bf 05 06 00 00 e8 c1 2b 01 00 e9 66
> [86041.350248] RSP: 0000:ffffb98f823ebed8 EFLAGS: 00010282
> [86041.350250] RAX: 0000000400000000 RBX: ffff9bf042adc100 RCX: 0000000000000000
> [86041.350252] RDX: 0000000000000001 RSI: 0000000000002710 RDI: ffff9bef09907380
> [86041.350255] RBP: ffff9bef81c55580 R08: 0000000000000000 R09: 0000000000000003
> [86041.350258] R10: ffffb98f823eb850 R11: ffff9bf23f2ad7a8 R12: 000000000000000b
> [86041.350261] R13: ffff9bef09907380 R14: ffffffffa65fa463 R15: ffffb98f823ebae8
> [86041.350263] FS:  00007f4bda31d280(0000) GS:ffff9bf22e880000(0000) knlGS:0000000000000000
> [86041.350267] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [86041.350269] CR2: 000000001665d010 CR3: 0000000191a2c000 CR4: 0000000000350ef0
> [86041.350272] Call Trace:
> [86041.350274]  <TASK>
> [86041.350276]  ? __warn+0x80/0x120
> [86041.350280]  ? do_exit+0x88b/0xac0
> [86041.350283]  ? report_bug+0x164/0x190
> [86041.350288]  ? handle_bug+0x3c/0x80
> [86041.350291]  ? exc_invalid_op+0x17/0x70
> [86041.350294]  ? asm_exc_invalid_op+0x1a/0x20
> [86041.350297]  ? do_exit+0x88b/0xac0
> [86041.350300]  ? do_exit+0x6f/0xac0
> [86041.350303]  ? do_user_addr_fault+0x177/0x6a0
> [86041.350307]  make_task_dead+0x81/0x170
> [86041.350310]  rewind_stack_and_make_dead+0x16/0x20
> [86041.350314] RIP: 0033:0x7453b9
> [86041.350319] Code: 00 48 8d 0c 49 4c 8d 04 ca 48 8b 0f 4c 39 c2 75 19 e9 7f 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 48 83 c2 18 49 39 d0 74 6b <48> 39 0a 75 f2 48 89 84 24 90 00 00 00 4c 39 73 10 0f 84 2f 02 00
> [86041.350324] RSP: 002b:00007ffe67b93c80 EFLAGS: 00010206
> [86041.350327] RAX: 0000000016659250 RBX: 00007ffe67b93db0 RCX: 000000000f1aad40
> [86041.350330] RDX: 000000001665d010 RSI: 00007ffe67b93cd8 RDI: 00007ffe67b93cd0
> [86041.350332] RBP: 0000000000000001 R08: 000000001665d088 R09: 0000000000000000
> [86041.350335] R10: 00007f4bda030610 R11: 00007f4bda0d6200 R12: 0000000016661210
> [86041.350337] R13: 00007ffe67b94a58 R14: 000000000ba280a8 R15: 0000000000000006
> [86041.350341]  </TASK>
> [86041.350342] ---[ end trace 0000000000000000 ]---
> [86041.579617] BUG: kernel NULL pointer dereference, address: 0000000000000008
> [86041.579627] #PF: supervisor write access in kernel mode
> [86041.579630] #PF: error_code(0x0002) - not-present page
> [86041.579632] PGD 0 P4D 0
> [86041.579636] Oops: Oops: 0002 [#2] PREEMPT SMP NOPTI
> [86041.579640] CPU: 5 PID: 2798071 Comm: llvm-tblgen Tainted: G      D W          6.10.6-12 #1 349ceb515693b41153483eac7819a5fb2832d2bf
> [86041.579645] Hardware name: To Be Filled By O.E.M. B450M Pro4-F R2.0/B450M Pro4-F R2.0, BIOS P10.08 01/19/2024
> [86041.579649] RIP: 0010:__blk_flush_plug+0x89/0x150
> [86041.579655] Code: de 48 89 5c 24 08 48 89 5c 24 10 48 39 c1 74 7c 49 8b 46 20 48 8b 34 24 48 39 c6 74 5b 49 8b 4e 20 49 8b 56 28 48 8b 44 24 08 <48> 89 59 08 48 89 4c 24 08 48 89 02 48 89 50 08 49 89 76 20 49 89
> [86041.579660] RSP: 0018:ffffb98f823ebc30 EFLAGS: 00010286
> [86041.579662] RAX: ffffb98f823ebc38 RBX: ffffb98f823ebc38 RCX: 0000000000000000
> [86041.579665] RDX: 0000000101887e59 RSI: ffffb98f823ebce8 RDI: ffffb98f823ebcc8
> [86041.579667] RBP: 0000000000000001 R08: ffff9bef14e7c248 R09: 0000000000000050
> [86041.579669] R10: 0000000000400023 R11: 0000000000000001 R12: dead000000000122
> [86041.579672] R13: dead000000000100 R14: ffffb98f823ebcc8 R15: 0000000000000000
> [86041.579674] FS:  0000000000000000(0000) GS:ffff9bf22e880000(0000) knlGS:0000000000000000
> [86041.579677] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [86041.579679] CR2: 0000000000000008 CR3: 0000000103bfe000 CR4: 0000000000350ef0
> [86041.579682] Call Trace:
> [86041.579685]  <TASK>
> [86041.579689]  ? __die+0x23/0x70
> [86041.579694]  ? page_fault_oops+0x173/0x5a0
> [86041.579698]  ? exc_page_fault+0x7e/0x180
> [86041.579702]  ? asm_exc_page_fault+0x26/0x30
> [86041.579706]  ? __blk_flush_plug+0x89/0x150
> [86041.579709]  schedule+0x99/0xf0
> [86041.579714]  schedule_preempt_disabled+0x15/0x30
> [86041.579716]  rwsem_down_write_slowpath+0x1eb/0x640
> [86041.579720]  down_write+0x5a/0x60
> [86041.579723]  free_pgtables+0xc6/0x1e0
> [86041.579728]  exit_mmap+0x16b/0x3a0
> [86041.579733]  __mmput+0x3e/0x130
> [86041.579736]  do_exit+0x2ac/0xac0
> [86041.579741]  ? do_user_addr_fault+0x177/0x6a0
> [86041.579743]  make_task_dead+0x81/0x170
> [86041.579746]  rewind_stack_and_make_dead+0x16/0x20
> [86041.579750] RIP: 0033:0x7453b9
> [86041.579768] Code: Unable to access opcode bytes at 0x74538f.
> [86041.579770] RSP: 002b:00007ffe67b93c80 EFLAGS: 00010206
> [86041.579772] RAX: 0000000016659250 RBX: 00007ffe67b93db0 RCX: 000000000f1aad40
> [86041.579774] RDX: 000000001665d010 RSI: 00007ffe67b93cd8 RDI: 00007ffe67b93cd0
> [86041.579776] RBP: 0000000000000001 R08: 000000001665d088 R09: 0000000000000000
> [86041.579778] R10: 00007f4bda030610 R11: 00007f4bda0d6200 R12: 0000000016661210
> [86041.579781] R13: 00007ffe67b94a58 R14: 000000000ba280a8 R15: 0000000000000006
> [86041.579784]  </TASK>
> [86041.579785] Modules linked in: tls rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs rpcrdma rdma_cm iw_cm ib_cm ib_core br_netfilter iptable_filter xt_physdev tun bridge stp llc ext4 crc16 mbcache jbd2 amd_atl intel_rapl_msr intel_rapl_common cfg80211 edac_mce_amd kvm_amd rfkill kvm crct10dif_pclmul crc32_pclmul polyval_clmulni r8169 polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 realtek sha256_ssse3 sha1_ssse3 aesni_intel mdio_devres crypto_simd sp5100_tco k10temp gpio_amdpt cryptd wmi_bmof pcspkr ccp libphy i2c_piix4 acpi_cpufreq rapl zenpower ryzen_smu gpio_generic mac_hid nfsd auth_rpcgss nfs_acl lockd grace nct6775 nct6775_core hwmon_vid sg sunrpc crypto_user fuse dm_mod loop nfnetlink bpf_preload ip_tables x_tables xfs libcrc32c crc32c_generic drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm drm_exec mxm_wmi nvme crc32c_intel drm_display_helper xhci_pci nvme_core xhci_pci_renesas wmi virtio_net net_failover failover dimlib virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev
> [86041.579842]  [last unloaded: nouveau]
> [86041.579858] CR2: 0000000000000008
> [86041.579861] ---[ end trace 0000000000000000 ]---
> [86041.579863] RIP: 0010:zswap_decompress+0x1ef/0x200
> [86041.579867] Code: ef e8 95 2a ce ff 84 c0 0f 85 1f ff ff ff e9 fb fe ff ff 0f 0b 48 8d 7b 10 e8 0d a9 a4 00 c7 43 10 00 00 00 00 8b 43 30 eb 86 <0f> 0b 0f 0b e8 f8 9b a3 00 0f 1f 84 00 00 00 00 00 90 90 90 90 90
> [86041.579872] RSP: 0000:ffffb98f823ebb90 EFLAGS: 00010282
> [86041.579875] RAX: 00000000ffffffea RBX: ffff9bf22e8c1e08 RCX: ffff9bef137774ba
> [86041.579877] RDX: 0000000000000002 RSI: 0000000000000438 RDI: ffff9bf22e8b2af0
> [86041.579880] RBP: ffff9bef58cd2b98 R08: ffff9bee8baf07e0 R09: ffff9bef13777080
> [86041.579882] R10: 0000000000000022 R11: ffff9bee8baf1000 R12: fffff782422ebc00
> [86041.579884] R13: ffff9bef13777080 R14: ffff9bef01e3d6e0 R15: ffff9bf22e8c1e48
> [86041.579886] FS:  0000000000000000(0000) GS:ffff9bf22e880000(0000) knlGS:0000000000000000
> [86041.579889] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [86041.579891] CR2: 0000000000000008 CR3: 0000000103bfe000 CR4: 0000000000350ef0
> [86041.579893] note: llvm-tblgen[2798071] exited with irqs disabled
> [86041.579895] Fixing recursive fault but reboot is needed!
> [86041.579897] BUG: scheduling while atomic: llvm-tblgen/2798071/0x00000000
> [86041.579899] Modules linked in: tls rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs rpcrdma rdma_cm iw_cm ib_cm ib_core br_netfilter iptable_filter xt_physdev tun bridge stp llc ext4 crc16 mbcache jbd2 amd_atl intel_rapl_msr intel_rapl_common cfg80211 edac_mce_amd kvm_amd rfkill kvm crct10dif_pclmul crc32_pclmul polyval_clmulni r8169 polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 realtek sha256_ssse3 sha1_ssse3 aesni_intel mdio_devres crypto_simd sp5100_tco k10temp gpio_amdpt cryptd wmi_bmof pcspkr ccp libphy i2c_piix4 acpi_cpufreq rapl zenpower ryzen_smu gpio_generic mac_hid nfsd auth_rpcgss nfs_acl lockd grace nct6775 nct6775_core hwmon_vid sg sunrpc crypto_user fuse dm_mod loop nfnetlink bpf_preload ip_tables x_tables xfs libcrc32c crc32c_generic drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm drm_exec mxm_wmi nvme crc32c_intel drm_display_helper xhci_pci nvme_core xhci_pci_renesas wmi virtio_net net_failover failover dimlib virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev
> [86041.579933]  [last unloaded: nouveau]
> [86041.579950] CPU: 5 PID: 2798071 Comm: llvm-tblgen Tainted: G      D W          6.10.6-12 #1 349ceb515693b41153483eac7819a5fb2832d2bf
> [86041.579954] Hardware name: To Be Filled By O.E.M. B450M Pro4-F R2.0/B450M Pro4-F R2.0, BIOS P10.08 01/19/2024
> [86041.579957] Call Trace:
> [86041.579959]  <TASK>
> [86041.579960]  dump_stack_lvl+0x64/0x80
> [86041.579965]  __schedule_bug+0x56/0x70
> [86041.579970]  __schedule+0x10d1/0x1520
> [86041.579973]  ? __wake_up_klogd.part.0+0x3c/0x60
> [86041.579978]  ? vprintk_emit+0x176/0x2a0
> [86041.579981]  ? _printk+0x64/0x80
> [86041.579984]  do_task_dead+0x42/0x50
> [86041.579988]  make_task_dead+0x149/0x170
> [86041.579991]  rewind_stack_and_make_dead+0x16/0x20
> [86041.579994] RIP: 0033:0x7453b9
> [86041.579997] Code: Unable to access opcode bytes at 0x74538f.
> [86041.579999] RSP: 002b:00007ffe67b93c80 EFLAGS: 00010206
> [86041.580002] RAX: 0000000016659250 RBX: 00007ffe67b93db0 RCX: 000000000f1aad40
> [86041.580004] RDX: 000000001665d010 RSI: 00007ffe67b93cd8 RDI: 00007ffe67b93cd0
> [86041.580006] RBP: 0000000000000001 R08: 000000001665d088 R09: 0000000000000000
> [86041.580008] R10: 00007f4bda030610 R11: 00007f4bda0d6200 R12: 0000000016661210
> [86041.580011] R13: 00007ffe67b94a58 R14: 000000000ba280a8 R15: 0000000000000006
> [86041.580014]  </TASK>
> [86260.530317] systemd[1]: systemd-journald.service: State 'stop-watchdog' timed out. Killing.
> [86260.530377] systemd[1]: systemd-journald.service: Killing process 483 (systemd-journal) with signal SIGKILL.
> [86350.780590] systemd[1]: systemd-journald.service: Processes still around after SIGKILL. Ignoring.
> [86441.030515] systemd[1]: systemd-journald.service: State 'final-sigterm' timed out. Killing.
> [86441.030574] systemd[1]: systemd-journald.service: Killing process 483 (systemd-journal) with signal SIGKILL.
> [86531.280569] systemd[1]: systemd-journald.service: Processes still around after final SIGKILL. Entering failed mode.
> [86531.280585] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
> [86531.280685] systemd[1]: systemd-journald.service: Unit process 483 (systemd-journal) remains running after unit stopped.
> [86531.289108] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
> [86531.289280] systemd[1]: systemd-journald.service: Found left-over process 483 (systemd-journal) in control group while starting unit. Ignoring.
> [86531.289285] systemd[1]: systemd-journald.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
> [86531.323344] systemd[1]: Starting Journal Service...
> [86531.330820] systemd-journald[2799374]: Collecting audit messages is disabled.
> [86531.331902] systemd-journald[2799374]: File /var/log/journal/1a15c5c01ee34ffb8beb42df7c18ff94/system.journal corrupted or uncleanly shut down, renaming and replacing.
> [86531.338702] systemd[1]: Started Journal Service.
> [root@minimyth2-x8664 piotro]#

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Piotr Oniszczuk]

[-- Attachment #1: Type: text/plain, Size: 536 bytes --]



> Wiadomość napisana przez Linux regression tracking (Thorsten Leemhuis) <regressions@leemhuis.info> w dniu 23.08.2024, o godz. 13:51:
> 
> If that is the case nobody might
> look into this unless you are able to provide more details, like the
> result of a bisction
> (https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html
> ) -- that's just how it is…


oh well - bisecting might be painful as to provoke oops - usually i need  - literally  - multiple days of constant 12c/24t compilation…


[-- Attachment #2: Type: text/html, Size: 4436 bytes --]

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Matthew Wilcox]

On Fri, Aug 23, 2024 at 01:51:41PM +0200, Linux regression tracking (Thorsten Leemhuis) wrote:
> > [75864.693223] br0: port 1(enp5s0) entered blocking state
> > [75864.693226] br0: port 1(enp5s0) entered forwarding state
> > [86041.349844] ------------[ cut here ]------------
> > [86041.349850] kernel BUG at mm/zswap.c:1005!

This is:

        BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait));

so crypto_acomp_decompress() returned an error, and zswap did not handle
it.  I wouldn't be surprised if this were dodgy ram.

That said, zswap could handle this better.  There's no need to panic the
entire machine over being unable to read a page from swap.  Killing just
the process that needed this page is sufficient.

Suggested patch at end after the oops.

> > [86041.349862] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
> > [86041.349867] CPU: 5 PID: 2798071 Comm: llvm-tblgen Not tainted 6.10.6-12 #1 349ceb515693b41153483eac7819a5fb2832d2bf
> > [86041.349872] Hardware name: To Be Filled By O.E.M. B450M Pro4-F R2.0/B450M Pro4-F R2.0, BIOS P10.08 01/19/2024
> > [86041.349876] RIP: 0010:zswap_decompress+0x1ef/0x200
> > [86041.349884] Code: ef e8 95 2a ce ff 84 c0 0f 85 1f ff ff ff e9 fb fe ff ff 0f 0b 48 8d 7b 10 e8 0d a9 a4 00 c7 43 10 00 00 00 00 8b 43 30 eb 86 <0f> 0b 0f 0b e8 f8 9b a3 00 0f 1f 84 00 00 00 00 00 90 90 90 90 90
> > [86041.349889] RSP: 0000:ffffb98f823ebb90 EFLAGS: 00010282
> > [86041.349892] RAX: 00000000ffffffea RBX: ffff9bf22e8c1e08 RCX: ffff9bef137774ba
> > [86041.349894] RDX: 0000000000000002 RSI: 0000000000000438 RDI: ffff9bf22e8b2af0
> > [86041.349897] RBP: ffff9bef58cd2b98 R08: ffff9bee8baf07e0 R09: ffff9bef13777080
> > [86041.349899] R10: 0000000000000022 R11: ffff9bee8baf1000 R12: fffff782422ebc00
> > [86041.349902] R13: ffff9bef13777080 R14: ffff9bef01e3d6e0 R15: ffff9bf22e8c1e48
> > [86041.349904] FS:  00007f4bda31d280(0000) GS:ffff9bf22e880000(0000) knlGS:0000000000000000
> > [86041.349908] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [86041.349910] CR2: 000000001665d010 CR3: 0000000191a2c000 CR4: 0000000000350ef0
> > [86041.349914] Call Trace:
> > [86041.349918]  <TASK>
> > [86041.349920]  ? die+0x36/0x90
> > [86041.349925]  ? do_trap+0xdd/0x100
> > [86041.349929]  ? zswap_decompress+0x1ef/0x200
> > [86041.349932]  ? do_error_trap+0x6a/0x90
> > [86041.349935]  ? zswap_decompress+0x1ef/0x200
> > [86041.349938]  ? exc_invalid_op+0x50/0x70
> > [86041.349943]  ? zswap_decompress+0x1ef/0x200
> > [86041.349946]  ? asm_exc_invalid_op+0x1a/0x20
> > [86041.349951]  ? zswap_decompress+0x1ef/0x200
> > [86041.349955]  zswap_load+0x109/0x120
> > [86041.349958]  swap_read_folio+0x64/0x450
> > [86041.349963]  swapin_readahead+0x463/0x4e0
> > [86041.349967]  do_swap_page+0x436/0xd70
> > [86041.349972]  ? __pte_offset_map+0x1b/0x180
> > [86041.349976]  __handle_mm_fault+0x85d/0x1070
> > [86041.349979]  ? sched_tick+0xee/0x2f0
> > [86041.349985]  handle_mm_fault+0x18d/0x320
> > [86041.349988]  do_user_addr_fault+0x177/0x6a0
> > [86041.349993]  exc_page_fault+0x7e/0x180
> > [86041.349996]  asm_exc_page_fault+0x26/0x30
> > [86041.350000] RIP: 0033:0x7453b9
> > [86041.350019] Code: 00 48 8d 0c 49 4c 8d 04 ca 48 8b 0f 4c 39 c2 75 19 e9 7f 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 48 83 c2 18 49 39 d0 74 6b <48> 39 0a 75 f2 48 89 84 24 90 00 00 00 4c 39 73 10 0f 84 2f 02 00
> > [86041.350024] RSP: 002b:00007ffe67b93c80 EFLAGS: 00010206
> > [86041.350027] RAX: 0000000016659250 RBX: 00007ffe67b93db0 RCX: 000000000f1aad40
> > [86041.350030] RDX: 000000001665d010 RSI: 00007ffe67b93cd8 RDI: 00007ffe67b93cd0
> > [86041.350032] RBP: 0000000000000001 R08: 000000001665d088 R09: 0000000000000000
> > [86041.350035] R10: 00007f4bda030610 R11: 00007f4bda0d6200 R12: 0000000016661210
> > [86041.350038] R13: 00007ffe67b94a58 R14: 000000000ba280a8 R15: 0000000000000006
> > [86041.350041]  </TASK>
> > [86041.350043] Modules linked in: tls rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs rpcrdma rdma_cm iw_cm ib_cm ib_core br_netfilter iptable_filter xt_physdev tun bridge stp llc ext4 crc16 mbcache jbd2 amd_atl intel_rapl_msr intel_rapl_common cfg80211 edac_mce_amd kvm_amd rfkill kvm crct10dif_pclmul crc32_pclmul polyval_clmulni r8169 polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 realtek sha256_ssse3 sha1_ssse3 aesni_intel mdio_devres crypto_simd sp5100_tco k10temp gpio_amdpt cryptd wmi_bmof pcspkr ccp libphy i2c_piix4 acpi_cpufreq rapl zenpower ryzen_smu gpio_generic mac_hid nfsd auth_rpcgss nfs_acl lockd grace nct6775 nct6775_core hwmon_vid sg sunrpc crypto_user fuse dm_mod loop nfnetlink bpf_preload ip_tables x_tables xfs libcrc32c crc32c_generic drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm drm_exec mxm_wmi nvme crc32c_intel drm_display_helper xhci_pci nvme_core xhci_pci_renesas wmi virtio_net net_failover failover dimlib virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev
> > [86041.350106]  [last unloaded: nouveau]
> > [86041.350125] ---[ end trace 0000000000000000 ]---

diff --git a/mm/zswap.c b/mm/zswap.c
index df66ab102d27..186aa4282c93 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -958,12 +958,13 @@ static bool zswap_compress(struct folio *folio, struct zswap_entry *entry)
 	return comp_ret == 0 && alloc_ret == 0;
 }
 
-static void zswap_decompress(struct zswap_entry *entry, struct folio *folio)
+static int zswap_decompress(struct zswap_entry *entry, struct folio *folio)
 {
 	struct zpool *zpool = entry->pool->zpool;
 	struct scatterlist input, output;
 	struct crypto_acomp_ctx *acomp_ctx;
 	u8 *src;
+	int err;
 
 	acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx);
 	mutex_lock(&acomp_ctx->mutex);
@@ -989,12 +990,17 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio)
 	sg_init_table(&output, 1);
 	sg_set_folio(&output, folio, PAGE_SIZE, 0);
 	acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE);
-	BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait));
-	BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE);
+	err = crypto_acomp_decompress(acomp_ctx->req);
+	err = crypto_wait_req(err, &acomp_ctx->wait);;
+	if (WARN_ONCE(err, "Decompression error %d -- corrupted RAM?\n", err))
+		return err;
+	if (acomp_ctx->req->dlen != PAGE_SIZE)
+		return -EIO;
 	mutex_unlock(&acomp_ctx->mutex);
 
 	if (src != acomp_ctx->buffer)
 		zpool_unmap_handle(zpool, entry->handle);
+	return 0;
 }
 
 /*********************************
@@ -1020,6 +1026,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
 	struct folio *folio;
 	struct mempolicy *mpol;
 	bool folio_was_allocated;
+	int err;
 	struct writeback_control wbc = {
 		.sync_mode = WB_SYNC_NONE,
 	};
@@ -1060,7 +1067,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
 		return -ENOMEM;
 	}
 
-	zswap_decompress(entry, folio);
+	err = zswap_decompress(entry, folio);
+	if (err < 0) {
+		folio_unlock(folio);
+		folio_put(folio);
+		return err;
+	}
 
 	count_vm_event(ZSWPWB);
 	if (entry->objcg)
@@ -1601,6 +1613,7 @@ bool zswap_load(struct folio *folio)
 	bool swapcache = folio_test_swapcache(folio);
 	struct xarray *tree = swap_zswap_tree(swp);
 	struct zswap_entry *entry;
+	int err;
 
 	VM_WARN_ON_ONCE(!folio_test_locked(folio));
 
@@ -1638,10 +1651,13 @@ bool zswap_load(struct folio *folio)
 	if (!entry)
 		return false;
 
-	if (entry->length)
-		zswap_decompress(entry, folio);
-	else
+	if (entry->length) {
+		err = zswap_decompress(entry, folio);
+		if (err)
+			return false;
+	} else {
 		zswap_fill_folio(folio, entry->value);
+	}
 
 	count_vm_event(ZSWPIN);
 	if (entry->objcg)

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Nhat Pham]

On Fri, Aug 23, 2024 at 9:13 AM Matthew Wilcox <willy@infradead.org> wrote:
>
>
> That said, zswap could handle this better.  There's no need to panic the
> entire machine over being unable to read a page from swap.  Killing just
> the process that needed this page is sufficient.

Agree 100%. It is silly to kill the entire host for a swap read error,
and extra silly to kill the process because we fail to writeback - for
all we know that page might never be needed by the process again!!!

>
> Suggested patch at end after the oops.
>
> @@ -1601,6 +1613,7 @@ bool zswap_load(struct folio *folio)
>         bool swapcache = folio_test_swapcache(folio);
>         struct xarray *tree = swap_zswap_tree(swp);
>         struct zswap_entry *entry;
> +       int err;
>
>         VM_WARN_ON_ONCE(!folio_test_locked(folio));
>
> @@ -1638,10 +1651,13 @@ bool zswap_load(struct folio *folio)
>         if (!entry)
>                 return false;
>
> -       if (entry->length)
> -               zswap_decompress(entry, folio);
> -       else
> +       if (entry->length) {
> +               err = zswap_decompress(entry, folio);
> +               if (err)
> +                       return false;

Here, if zswap decompression fails and zswap load returns false, the
page_io logic will proceed as if zswap does not have the page and
reads garbage from the backing device instead. This could potentially
lead to silent data/memory corruption right? Or am I missing something
:) Maybe we could be extra careful here and treat it as if there is a
bio read error in the case zswap owns the page, but cannot decompress
it?

The rest seems solid to me :)

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Matthew Wilcox]

On Fri, Aug 23, 2024 at 10:35:19AM -0400, Nhat Pham wrote:
> On Fri, Aug 23, 2024 at 9:13 AM Matthew Wilcox <willy@infradead.org> wrote:
> >
> >
> > That said, zswap could handle this better.  There's no need to panic the
> > entire machine over being unable to read a page from swap.  Killing just
> > the process that needed this page is sufficient.
> 
> Agree 100%. It is silly to kill the entire host for a swap read error,
> and extra silly to kill the process because we fail to writeback - for
> all we know that page might never be needed by the process again!!!
> 
> >
> > Suggested patch at end after the oops.
> >
> > @@ -1601,6 +1613,7 @@ bool zswap_load(struct folio *folio)
> >         bool swapcache = folio_test_swapcache(folio);
> >         struct xarray *tree = swap_zswap_tree(swp);
> >         struct zswap_entry *entry;
> > +       int err;
> >
> >         VM_WARN_ON_ONCE(!folio_test_locked(folio));
> >
> > @@ -1638,10 +1651,13 @@ bool zswap_load(struct folio *folio)
> >         if (!entry)
> >                 return false;
> >
> > -       if (entry->length)
> > -               zswap_decompress(entry, folio);
> > -       else
> > +       if (entry->length) {
> > +               err = zswap_decompress(entry, folio);
> > +               if (err)
> > +                       return false;
> 
> Here, if zswap decompression fails and zswap load returns false, the
> page_io logic will proceed as if zswap does not have the page and
> reads garbage from the backing device instead. This could potentially
> lead to silent data/memory corruption right? Or am I missing something
> :) Maybe we could be extra careful here and treat it as if there is a
> bio read error in the case zswap owns the page, but cannot decompress
> it?

Ah; you know more about how zswap works than I do.  So it's not a
write-through cache?  I guess we need to go a bit further then and
return an errno from zswap_load -- EIO/ENOENT/0 and handle that
appropriately.

> The rest seems solid to me :)

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Piotr Oniszczuk]

> Wiadomość napisana przez Matthew Wilcox <willy@infradead.org> w dniu 23.08.2024, o godz. 15:13:
> 
> I wouldn't be surprised if this were dodgy ram.


Well - that was my initial hypothesis.

in fact i had few of them. Ranked (and ordered) like this:
1. downstream kernel patches
2. hw (ram) issue
3. kernel bug

So full history was:
-build myself archlinux 6.10.2 kernel; upgrade builder OS (only kernel; nothing else)
-run normal devel process and (to my surprise) discover interrupted CI/CD builds by kernel oops
-downgrade to 6.8.2 and done 4 full builds (full takes 8..9h of constant 12c/24/t compile). all good.  
-prepare vanilla 6.10.6 (to exclude potential downstream (ArchLinux) root causes)
-run normal devel process and still discover oops
-make sure hw is ok by week of test with 6.8.2 (recompiling for 3 architectures on 4 OS (3 in kvm). This was almost 5 full days of 12c/24 compiling. All good
-because last steep was all good - decide to go to you :-)

sure - this is possible that 6.8.2 had luck with my ram and 6.10.6 had no luck….but i personally don’t believe this is a case….     

btw: we can go with elimination strategy.
So what i need to change/disable to be closer to finding root cause?
swap?
now it is swapfile on system nvme

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Yosry Ahmed]

On Fri, Aug 23, 2024 at 7:47 AM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Fri, Aug 23, 2024 at 10:35:19AM -0400, Nhat Pham wrote:
> > On Fri, Aug 23, 2024 at 9:13 AM Matthew Wilcox <willy@infradead.org> wrote:
> > >
> > >
> > > That said, zswap could handle this better.  There's no need to panic the
> > > entire machine over being unable to read a page from swap.  Killing just
> > > the process that needed this page is sufficient.
> >
> > Agree 100%. It is silly to kill the entire host for a swap read error,
> > and extra silly to kill the process because we fail to writeback - for
> > all we know that page might never be needed by the process again!!!
> >
> > >
> > > Suggested patch at end after the oops.
> > >
> > > @@ -1601,6 +1613,7 @@ bool zswap_load(struct folio *folio)
> > >         bool swapcache = folio_test_swapcache(folio);
> > >         struct xarray *tree = swap_zswap_tree(swp);
> > >         struct zswap_entry *entry;
> > > +       int err;
> > >
> > >         VM_WARN_ON_ONCE(!folio_test_locked(folio));
> > >
> > > @@ -1638,10 +1651,13 @@ bool zswap_load(struct folio *folio)
> > >         if (!entry)
> > >                 return false;
> > >
> > > -       if (entry->length)
> > > -               zswap_decompress(entry, folio);
> > > -       else
> > > +       if (entry->length) {
> > > +               err = zswap_decompress(entry, folio);
> > > +               if (err)
> > > +                       return false;
> >
> > Here, if zswap decompression fails and zswap load returns false, the
> > page_io logic will proceed as if zswap does not have the page and
> > reads garbage from the backing device instead. This could potentially
> > lead to silent data/memory corruption right? Or am I missing something
> > :) Maybe we could be extra careful here and treat it as if there is a
> > bio read error in the case zswap owns the page, but cannot decompress
> > it?
>
> Ah; you know more about how zswap works than I do.  So it's not a
> write-through cache?  I guess we need to go a bit further then and
> return an errno from zswap_load -- EIO/ENOENT/0 and handle that
> appropriately.

It should work if we just return true without calling
folio_mark_uptodate(), this is what we do if we get a large folio in
zswap_load(). Returning true means that the page was found in zswap,
so we won't fallback to reading from the backing device. Not marking
the folio uptodate will cause an IO error IIUC.

>
> > The rest seems solid to me :)

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Nhat Pham]

On Fri, Aug 23, 2024 at 11:07 AM Piotr Oniszczuk
<piotr.oniszczuk@gmail.com> wrote:
>
>
>
> > Wiadomość napisana przez Matthew Wilcox <willy@infradead.org> w dniu 23.08.2024, o godz. 15:13:
> >
> > I wouldn't be surprised if this were dodgy ram.
>
>
> Well - that was my initial hypothesis.
>
> in fact i had few of them. Ranked (and ordered) like this:
> 1. downstream kernel patches
> 2. hw (ram) issue
> 3. kernel bug
>
> So full history was:
> -build myself archlinux 6.10.2 kernel; upgrade builder OS (only kernel; nothing else)
> -run normal devel process and (to my surprise) discover interrupted CI/CD builds by kernel oops
> -downgrade to 6.8.2 and done 4 full builds (full takes 8..9h of constant 12c/24/t compile). all good.
> -prepare vanilla 6.10.6 (to exclude potential downstream (ArchLinux) root causes)
> -run normal devel process and still discover oops
> -make sure hw is ok by week of test with 6.8.2 (recompiling for 3 architectures on 4 OS (3 in kvm). This was almost 5 full days of 12c/24 compiling. All good
> -because last steep was all good - decide to go to you :-)
>
> sure - this is possible that 6.8.2 had luck with my ram and 6.10.6 had no luck….but i personally don’t believe this is a case….

Have you tried with 6.9 yet? IIRC, there are two major changes to
zswap architecture in recent versions.

1. In 6.9, we range-partition zswap's rbtrees to reduce lock contention.

2. In 6.10, we replace zswap's rbtrees with xarrays.

If 6.9 is fine, then the latter is the suspect, and vice versa. Of
course, the minor changes are still suspect - but you get the idea :)

>
> btw: we can go with elimination strategy.
> So what i need to change/disable to be closer to finding root cause?

Could you let me know more about the setup? A couple things come to my mind:

1. zswap configs (allocator - is it zsmalloc? compressor?)

2. Is mTHP enabled? mTHP swapout was merged in 6.10, and there seems
to be some conflicts with zswap, but Yosry will know more about this
than me...

3. Is there any proprietary driver etc.?

> swap?
> now it is swapfile on system nvme
>

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Piotr Oniszczuk]

> Wiadomość napisana przez Nhat Pham <nphamcs@gmail.com> w dniu 23.08.2024, o godz. 18:16:
> 
> Have you tried with 6.9 yet? IIRC, there are two major changes to
> zswap architecture in recent versions.

No. But now building vanilla 6.9.12. Will install and see…
(This will take some time as catching issue needs days of compilation)

> 
> 1. In 6.9, we range-partition zswap's rbtrees to reduce lock contention.
> 
> 2. In 6.10, we replace zswap's rbtrees with xarrays.
> 
> If 6.9 is fine, then the latter is the suspect, and vice versa. Of
> course, the minor changes are still suspect - but you get the idea :)
> 
>> 
>> btw: we can go with elimination strategy.
>> So what i need to change/disable to be closer to finding root cause?
> 
> Could you let me know more about the setup? A couple things come to my mind:
> 
> 1. zswap configs (allocator - is it zsmalloc? compressor?)

Well - I’m not using zswap.

[root@minimyth2-aarch64-next piotro]# swapon -s
Filename				Type		Size		Used		Priority
/dev/nvme0n1p3                          partition	16776188	294164		-2

> 
> 2. Is mTHP enabled? mTHP swapout was merged in 6.10, and there seems

I don’t have used config at the moment, but /sys/kernel/mm/transparent_hugepage in I see:

│/hugepages-1024kB
│/hugepages-128kB
│/hugepages-16kB
│/hugepages-2048kB
│/hugepages-256kB
│/hugepages-32kB
│/hugepages-512kB
│/hugepages-64kB


> to be some conflicts with zswap, but Yosry will know more about this
> than me...
> 
> 3. Is there any proprietary driver etc.?
> 

Only 2, both ryzen9 monitoring related:
https://github.com/leogx9r/ryzen_smu/commits/master
https://github.com/ocerman/zenpower/commits/master

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Nhat Pham]

On Fri, Aug 23, 2024 at 1:24 PM Piotr Oniszczuk
<piotr.oniszczuk@gmail.com> wrote:
>
>
>
> > Wiadomość napisana przez Nhat Pham <nphamcs@gmail.com> w dniu 23.08.2024, o godz. 18:16:
> >
> > Have you tried with 6.9 yet? IIRC, there are two major changes to
> > zswap architecture in recent versions.
>
> No. But now building vanilla 6.9.12. Will install and see…
> (This will take some time as catching issue needs days of compilation)
>
> >
> > 1. In 6.9, we range-partition zswap's rbtrees to reduce lock contention.
> >
> > 2. In 6.10, we replace zswap's rbtrees with xarrays.
> >
> > If 6.9 is fine, then the latter is the suspect, and vice versa. Of
> > course, the minor changes are still suspect - but you get the idea :)
> >
> >>
> >> btw: we can go with elimination strategy.
> >> So what i need to change/disable to be closer to finding root cause?
> >
> > Could you let me know more about the setup? A couple things come to my mind:
> >
> > 1. zswap configs (allocator - is it zsmalloc? compressor?)
>
> Well - I’m not using zswap.

But the bug happens in zswap path? :)

Could you do:

grep . /sys/module/zswap/parameters/*


>
> [root@minimyth2-aarch64-next piotro]# swapon -s
> Filename                                Type            Size            Used            Priority
> /dev/nvme0n1p3                          partition       16776188        294164          -2
>
> >
> > 2. Is mTHP enabled? mTHP swapout was merged in 6.10, and there seems
>
> I don’t have used config at the moment, but /sys/kernel/mm/transparent_hugepage in I see:
>
> │/hugepages-1024kB
> │/hugepages-128kB
> │/hugepages-16kB
> │/hugepages-2048kB
> │/hugepages-256kB
> │/hugepages-32kB
> │/hugepages-512kB
> │/hugepages-64kB
>
>
> > to be some conflicts with zswap, but Yosry will know more about this
> > than me...
> >
> > 3. Is there any proprietary driver etc.?
> >
>
> Only 2, both ryzen9 monitoring related:
> https://github.com/leogx9r/ryzen_smu/commits/master
> https://github.com/ocerman/zenpower/commits/master
>

The reason I asked this is because I've seen proprietary error
screwing with memory in the past - it was an NVIDIA one though.

https://lore.kernel.org/linux-mm/CAKbZUD1-kqfuV0U+KDKPkQbm=RwzD_A1H3qk_c+bw92CqtMbuw@mail.gmail.com/

Also decompression step failure (albeit in the writeback path)

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Takero Funaki]

2024年8月24日(土) 0:07 Piotr Oniszczuk <piotr.oniszczuk@gmail.com>:
>
>
>
> > Wiadomość napisana przez Matthew Wilcox <willy@infradead.org> w dniu 23.08.2024, o godz. 15:13:
> >
> > I wouldn't be surprised if this were dodgy ram.
>
>
> Well - that was my initial hypothesis.
>
> in fact i had few of them. Ranked (and ordered) like this:
> 1. downstream kernel patches
> 2. hw (ram) issue
> 3. kernel bug
>
> So full history was:
> -build myself archlinux 6.10.2 kernel; upgrade builder OS (only kernel; nothing else)
> -run normal devel process and (to my surprise) discover interrupted CI/CD builds by kernel oops
> -downgrade to 6.8.2 and done 4 full builds (full takes 8..9h of constant 12c/24/t compile). all good.
> -prepare vanilla 6.10.6 (to exclude potential downstream (ArchLinux) root causes)
> -run normal devel process and still discover oops
> -make sure hw is ok by week of test with 6.8.2 (recompiling for 3 architectures on 4 OS (3 in kvm). This was almost 5 full days of 12c/24 compiling. All good
> -because last steep was all good - decide to go to you :-)
>
> sure - this is possible that 6.8.2 had luck with my ram and 6.10.6 had no luck….but i personally don’t believe this is a case….
>
> btw: we can go with elimination strategy.
> So what i need to change/disable to be closer to finding root cause?
> swap?
> now it is swapfile on system nvme
>
>

Hello,

I’m encountering a similar crash and trace in the issue I posted on Bugzilla:
https://bugzilla.kernel.org/show_bug.cgi?id=219154

If this is the same issue caused by virtio_net corrupting memory, you
should be able to reproduce the crash by sending data to the VM over
virtio interface while it is actively allocating memory (e.g., using
iperf3 -s on the VM and running iperf -c from another host).

In my case, as Thorsten suggested, reverting bisected commit
f9dac92ba908 (virtio_ring: enable premapped mode regardless of
use_dma_api) along with two related commits in this series resolved
the issue:
https://lore.kernel.org/all/7774ac707743ad8ce3afeacbd4bee63ac96dd927.1723617902.git.mst@redhat.com/

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Piotr Oniszczuk]

> Wiadomość napisana przez Nhat Pham <nphamcs@gmail.com> w dniu 23.08.2024, o godz. 20:06:
> 
> Could you do:
> 
> grep . /sys/module/zswap/parameters/*


Here it is:

/sys/module/zswap/parameters/accept_threshold_percent:90
/sys/module/zswap/parameters/compressor:lz4
/sys/module/zswap/parameters/enabled:Y
/sys/module/zswap/parameters/max_pool_percent:20
/sys/module/zswap/parameters/non_same_filled_pages_enabled:Y
/sys/module/zswap/parameters/same_filled_pages_enabled:Y
/sys/module/zswap/parameters/shrinker_enabled:N
/sys/module/zswap/parameters/zpool:z3fold

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Piotr Oniszczuk]

> Wiadomość napisana przez Nhat Pham <nphamcs@gmail.com> w dniu 23.08.2024, o godz. 18:16:
> 
> 
> Have you tried with 6.9 yet? IIRC, there are two major changes to
> zswap architecture in recent versions.
> 
> 1. In 6.9, we range-partition zswap's rbtrees to reduce lock contention.
> 

Ok - after 32h of continuous compilation also on 6.9.12 I got series of oops (see below).

[68616.350398] watchdog: BUG: soft lockup - CPU#4 stuck for 596s! [kcompactd0:176]
[68616.350401] Modules linked in: tls rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs rpcrdma rdma_cm iw_cm ib_cm ib_core br_netfilter iptable_filter xt_physdev tun bridge stp llc ext4 crc16 mbcache jbd2 amd_atl intel_rapl_msr intel_rapl_common edac_mce_amd kvm_amd cfg80211 rfkill kvm crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 r8169 aesni_intel crypto_simd cryptd realtek mdio_devres sp5100_tco wmi_bmof k10temp libphy ccp pcspkr rapl i2c_piix4 acpi_cpufreq zenpower ryzen_smu gpio_amdpt gpio_generic mac_hid nfsd auth_rpcgss nfs_acl lockd grace nct6775 nct6775_core hwmon_vid sunrpc sg crypto_user fuse dm_mod loop nfnetlink bpf_preload ip_tables x_tables xfs libcrc32c crc32c_generic drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm drm_exec mxm_wmi nvme drm_display_helper crc32c_intel xhci_pci nvme_core cec xhci_pci_renesas wmi virtio_net net_failover failover virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev
[68616.350434]  [last unloaded: nouveau]
[68616.350448] CPU: 4 PID: 176 Comm: kcompactd0 Tainted: G      D W    L     6.9.12-12 #1 e59bce453550af16b12fd25785f4d449e921764e
[68616.350451] Hardware name: To Be Filled By O.E.M. B450M Pro4-F R2.0/B450M Pro4-F R2.0, BIOS P10.08 01/19/2024
[68616.350454] RIP: 0010:native_queued_spin_lock_slowpath+0x6e/0x2e0
[68616.350457] Code: 77 7f f0 0f ba 2b 08 0f 92 c2 8b 03 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 77 5b 85 c0 74 10 0f b6 03 84 c0 74 09 f3 90 <0f> b6 03 84 c0 75 f7 b8 01 00 00 00 66 89 03 65 48 ff 05 f3 66 02
[68616.350461] RSP: 0018:ffffb268806db858 EFLAGS: 00000202
[68616.350463] RAX: 0000000000000001 RBX: fffffbc30722d4e8 RCX: 0000000000000867
[68616.350465] RDX: 0000000000000000 RSI: 0000000000000001 RDI: fffffbc30722d4e8
[68616.350467] RBP: 00007f6af88b3000 R08: 0000008000000000 R09: 0200000000000080
[68616.350469] R10: 0000000000000000 R11: ffffb268806db9e0 R12: ffff9388c7f63700
[68616.350471] R13: ffff938815651b00 R14: 0000000000000001 R15: ffffff8000000000
[68616.350473] FS:  0000000000000000(0000) GS:ffff938b2e800000(0000) knlGS:0000000000000000
[68616.350475] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[68616.350477] CR2: 000055c339a59150 CR3: 00000002c2c20000 CR4: 0000000000350ef0
[68616.350479] Call Trace:
[68616.350480]  <IRQ>
[68616.350481]  ? watchdog_timer_fn+0x1dd/0x260
[68616.350484]  ? __pfx_watchdog_timer_fn+0x10/0x10
[68616.350487]  ? __hrtimer_run_queues+0x10f/0x2a0
[68616.350490]  ? hrtimer_interrupt+0xfa/0x230
[68616.350492]  ? __sysvec_apic_timer_interrupt+0x55/0x150
[68616.350494]  ? sysvec_apic_timer_interrupt+0x6c/0x90
[68616.350497]  </IRQ>
[68616.350498]  <TASK>
[68616.350500]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[68616.350503]  ? native_queued_spin_lock_slowpath+0x6e/0x2e0
[68616.350506]  _raw_spin_lock+0x29/0x30
[68616.350509]  page_vma_mapped_walk+0x6a2/0x950
[68616.350511]  try_to_migrate_one+0x174/0xbf0
[68616.350515]  rmap_walk_anon+0xb6/0x190
[68616.350518]  try_to_migrate+0xf9/0x110
[68616.350520]  ? __pfx_try_to_migrate_one+0x10/0x10
[68616.350523]  ? __pfx_folio_not_mapped+0x10/0x10
[68616.350526]  ? __pfx_folio_lock_anon_vma_read+0x10/0x10
[68616.350528]  ? __pfx_invalid_migration_vma+0x10/0x10
[68616.350531]  migrate_pages_batch+0x545/0xb80
[68616.350534]  ? __pfx_compaction_free+0x10/0x10
[68616.350536]  ? __pfx_compaction_alloc+0x10/0x10
[68616.350540]  ? __pfx_remove_migration_pte+0x10/0x10
[68616.350542]  migrate_pages+0xada/0xd90
[68616.350545]  ? __pfx_compaction_alloc+0x10/0x10
[68616.350548]  ? __pfx_compaction_free+0x10/0x10
[68616.350551]  ? folio_add_lru+0x5f/0xb0
[68616.350553]  compact_zone+0x9e8/0x10c0
[68616.350556]  ? sched_clock_cpu+0xf/0x190
[68616.350559]  ? raw_spin_rq_lock_nested+0x1c/0x80
[68616.350561]  ? psi_group_change+0x213/0x3c0
[68616.350564]  compact_node+0xb7/0x130
[68616.350568]  kcompactd+0x355/0x430
[68616.350571]  ? __pfx_autoremove_wake_function+0x10/0x10
[68616.350573]  ? __pfx_kcompactd+0x10/0x10
[68616.350576]  kthread+0xcf/0x100
[68616.350578]  ? __pfx_kthread+0x10/0x10
[68616.350580]  ret_from_fork+0x31/0x50
[68616.350583]  ? __pfx_kthread+0x10/0x10
[68616.350585]  ret_from_fork_asm+0x1a/0x30
[68616.350588]  </TASK>
[68620.214362] watchdog: BUG: soft lockup - CPU#0 stuck for 663s! [cc1plus:2979523]
[68620.214367] Modules linked in: tls rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs rpcrdma rdma_cm iw_cm ib_cm ib_core br_netfilter iptable_filter xt_physdev tun bridge stp llc ext4 crc16 mbcache jbd2 amd_atl intel_rapl_msr intel_rapl_common edac_mce_amd kvm_amd cfg80211 rfkill kvm crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 r8169 aesni_intel crypto_simd cryptd realtek mdio_devres sp5100_tco wmi_bmof k10temp libphy ccp pcspkr rapl i2c_piix4 acpi_cpufreq zenpower ryzen_smu gpio_amdpt gpio_generic mac_hid nfsd auth_rpcgss nfs_acl lockd grace nct6775 nct6775_core hwmon_vid sunrpc sg crypto_user fuse dm_mod loop nfnetlink bpf_preload ip_tables x_tables xfs libcrc32c crc32c_generic drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm drm_exec mxm_wmi nvme drm_display_helper crc32c_intel xhci_pci nvme_core cec xhci_pci_renesas wmi virtio_net net_failover failover virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev
[68620.214402]  [last unloaded: nouveau]
[68620.214416] CPU: 0 PID: 2979523 Comm: cc1plus Tainted: G      D W    L     6.9.12-12 #1 e59bce453550af16b12fd25785f4d449e921764e
[68620.214420] Hardware name: To Be Filled By O.E.M. B450M Pro4-F R2.0/B450M Pro4-F R2.0, BIOS P10.08 01/19/2024
[68620.214422] RIP: 0010:native_queued_spin_lock_slowpath+0x21f/0x2e0
[68620.214426] Code: c5 01 41 c1 e4 10 41 c1 e5 12 45 09 ec 44 89 e0 c1 e8 10 66 87 43 02 89 c2 c1 e2 10 81 fa ff ff 00 00 77 5e 31 d2 eb 02 f3 90 <8b> 03 66 85 c0 75 f7 44 39 e0 0f 84 8e 00 00 00 c6 03 01 48 85 d2
[68620.214430] RSP: 0000:ffffb2688397fbe0 EFLAGS: 00000202
[68620.214432] RAX: 00000000000c0101 RBX: ffff9388140cf738 RCX: 0000000000000008
[68620.214434] RDX: 0000000000000000 RSI: 0000000000000101 RDI: ffff9388140cf738
[68620.214436] RBP: ffff938b2e6373c0 R08: ffff938b2e6310c0 R09: 000000000000000a
[68620.214438] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000040000
[68620.214440] R13: 0000000000040000 R14: ffff9388140cf738 R15: ffff9388140cf730
[68620.214442] FS:  00007fc78bf83f00(0000) GS:ffff938b2e600000(0000) knlGS:0000000000000000
[68620.214445] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[68620.214447] CR2: 00007fc75b53f264 CR3: 00000001797f4000 CR4: 0000000000350ef0
[68620.214449] Call Trace:
[68620.214450]  <IRQ>
[68620.214451]  ? watchdog_timer_fn+0x1dd/0x260
[68620.214454]  ? __pfx_watchdog_timer_fn+0x10/0x10
[68620.214457]  ? __hrtimer_run_queues+0x10f/0x2a0
[68620.214460]  ? hrtimer_interrupt+0xfa/0x230
[68620.214462]  ? __sysvec_apic_timer_interrupt+0x55/0x150
[68620.214465]  ? sysvec_apic_timer_interrupt+0x6c/0x90
[68620.214468]  </IRQ>
[68620.214469]  <TASK>
[68620.214470]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[68620.214474]  ? native_queued_spin_lock_slowpath+0x21f/0x2e0
[68620.214477]  _raw_spin_lock+0x29/0x30
[68620.214479]  zswap_load+0x6a/0x160
[68620.214482]  swap_read_folio+0x64/0x450
[68620.214484]  swapin_readahead+0x1ea/0x4e0
[68620.214487]  do_swap_page+0x403/0xd20
[68620.214490]  ? shmem_file_write_iter+0x5e/0x90
[68620.214492]  ? __pte_offset_map+0x1b/0x180
[68620.214494]  __handle_mm_fault+0x868/0xdd0
[68620.214498]  handle_mm_fault+0x18d/0x320
[68620.214500]  do_user_addr_fault+0x175/0x6b0
[68620.214503]  exc_page_fault+0x7e/0x180
[68620.214505]  asm_exc_page_fault+0x26/0x30
[68620.214508] RIP: 0033:0x330a353
[68620.214512] Code: e2 03 48 01 d0 48 8b 00 48 89 45 e8 48 83 7d e8 00 0f 84 11 01 00 00 48 83 7d e8 ff 75 08 8b 45 fc 89 45 f8 eb 42 48 8b 45 e8 <8b> 40 0c 39 45 84 75 36 48 8b 45 e8 8b 40 08 48 8b 55 88 39 d0 75
[68620.214515] RSP: 002b:00007ffc42ae0390 EFLAGS: 00010217
[68620.214517] RAX: 00007fc75b53f258 RBX: 00000000000003e9 RCX: 00000000f9107c14
[68620.214519] RDX: 000000000003e0a0 RSI: 00007ffc42ae04f1 RDI: 0000000027ef6180
[68620.214521] RBP: 00007ffc42ae0410 R08: 0000000000000000 R09: 0000000000000000
[68620.214523] R10: 00007fc78c106ac0 R11: 00007fc78c1073c0 R12: 0000000000000000
[68620.214525] R13: 00007ffc42ae1030 R14: 00007fc78c66f000 R15: 0000000003df8e50
[68620.214528]  </TASK>
[68632.363664] watchdog: BUG: soft lockup - CPU#8 stuck for 648s! [cc1plus:2982130]
[68632.363668] Modules linked in: tls rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs rpcrdma rdma_cm iw_cm ib_cm ib_core br_netfilter iptable_filter xt_physdev tun bridge stp llc ext4 crc16 mbcache jbd2 amd_atl intel_rapl_msr intel_rapl_common edac_mce_amd kvm_amd cfg80211 rfkill kvm crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 r8169 aesni_intel crypto_simd cryptd realtek mdio_devres sp5100_tco wmi_bmof k10temp libphy ccp pcspkr rapl i2c_piix4 acpi_cpufreq zenpower ryzen_smu gpio_amdpt gpio_generic mac_hid nfsd auth_rpcgss nfs_acl lockd grace nct6775 nct6775_core hwmon_vid sunrpc sg crypto_user fuse dm_mod loop nfnetlink bpf_preload ip_tables x_tables xfs libcrc32c crc32c_generic drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm drm_exec mxm_wmi nvme drm_display_helper crc32c_intel xhci_pci nvme_core cec xhci_pci_renesas wmi virtio_net net_failover failover virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev
[68632.363704]  [last unloaded: nouveau]
[68632.363719] CPU: 8 PID: 2982130 Comm: cc1plus Tainted: G      D W    L     6.9.12-12 #1 e59bce453550af16b12fd25785f4d449e921764e
[68632.363722] Hardware name: To Be Filled By O.E.M. B450M Pro4-F R2.0/B450M Pro4-F R2.0, BIOS P10.08 01/19/2024
[68632.363724] RIP: 0010:native_queued_spin_lock_slowpath+0x2a1/0x2e0
[68632.363728] Code: c1 ea 12 83 e0 03 83 ea 01 48 c1 e0 05 48 63 d2 48 05 c0 73 03 00 48 03 04 d5 40 32 91 aa 48 89 28 8b 45 08 85 c0 75 09 f3 90 <8b> 45 08 85 c0 74 f7 48 8b 55 00 48 85 d2 0f 84 6a ff ff ff 0f 0d
[68632.363732] RSP: 0000:ffffb26885e1f450 EFLAGS: 00000246
[68632.363734] RAX: 0000000000000000 RBX: ffff9388140cf738 RCX: fffffbc30f2ad340
[68632.363736] RDX: 0000000000000014 RSI: 0000000000540101 RDI: ffff9388140cf738
[68632.363738] RBP: ffff938b2ea373c0 R08: ffff93876cc2aaa0 R09: 0008c8130ae03aa0
[68632.363740] R10: 020f0008c8130ae0 R11: 0000000000501000 R12: 0000000000240000
[68632.363741] R13: 0000000000240000 R14: 03ffffffffffffff R15: 00000000000005fa
[68632.363743] FS:  00007fd929957f00(0000) GS:ffff938b2ea00000(0000) knlGS:0000000000000000
[68632.363746] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[68632.363748] CR2: 00007fd923600000 CR3: 0000000162ba8000 CR4: 0000000000350ef0
[68632.363750] Call Trace:
[68632.363751]  <IRQ>
[68632.363752]  ? watchdog_timer_fn+0x1dd/0x260
[68632.363755]  ? __pfx_watchdog_timer_fn+0x10/0x10
[68632.363758]  ? __hrtimer_run_queues+0x10f/0x2a0
[68632.363761]  ? hrtimer_interrupt+0xfa/0x230
[68632.363763]  ? __sysvec_apic_timer_interrupt+0x55/0x150
[68632.363766]  ? sysvec_apic_timer_interrupt+0x6c/0x90
[68632.363769]  </IRQ>
[68632.363770]  <TASK>
[68632.363771]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[68632.363775]  ? native_queued_spin_lock_slowpath+0x2a1/0x2e0
[68632.363778]  _raw_spin_lock+0x29/0x30
[68632.363781]  zswap_store+0x623/0xc70
[68632.363783]  swap_writepage+0x33/0xe0
[68632.363786]  pageout+0xc9/0x250
[68632.363790]  shrink_folio_list+0x5e4/0xca0
[68632.363793]  ? vm_mmap_pgoff+0xec/0x1a0
[68632.363796]  ? do_syscall_64+0x82/0x170
[68632.363799]  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
[68632.363803]  evict_folios+0x27a/0x660
[68632.363807]  try_to_shrink_lruvec+0x1a7/0x280
[68632.363810]  shrink_one+0x10a/0x1f0
[68632.363812]  shrink_node+0xab9/0xc00
[68632.363815]  ? zone_reclaimable_pages+0x15c/0x190
[68632.363818]  do_try_to_free_pages+0xca/0x5d0
[68632.363822]  try_to_free_pages+0xdd/0x210
[68632.363825]  __alloc_pages_slowpath.constprop.0+0x316/0xd60
[68632.363828]  ? mmap_region+0x4fc/0x9c0
[68632.363831]  __alloc_pages+0x32d/0x360
[68632.363834]  alloc_pages_mpol+0xd9/0x1e0
[68632.363836]  ? __lruvec_stat_mod_folio+0x81/0xa0
[68632.363839]  mm_get_huge_zero_page+0x74/0x100
[68632.363841]  do_huge_pmd_anonymous_page+0x37f/0x6e0
[68632.363844]  ? syscall_exit_to_user_mode+0x72/0x200
[68632.363846]  __handle_mm_fault+0xb5f/0xdd0
[68632.363850]  handle_mm_fault+0x18d/0x320
[68632.363852]  do_user_addr_fault+0x175/0x6b0
[68632.363855]  exc_page_fault+0x7e/0x180
[68632.363857]  asm_exc_page_fault+0x26/0x30
[68632.363860] RIP: 0033:0xf19e9f
[68632.363864] Code: 08 5d c3 55 48 89 e5 48 89 7d f8 89 75 f4 89 55 f0 89 4d ec 8b 45 f4 25 ff ff ff 7f 89 c2 48 8b 45 f8 89 d1 81 e1 ff ff ff 7f <8b> 10 81 e2 00 00 00 80 09 ca 89 10 8b 45 ec 83 e0 01 48 8b 55 f8
[68632.363868] RSP: 002b:00007ffc3532adf0 EFLAGS: 00010202
[68632.363870] RAX: 00007fd923600000 RBX: ffffffffffffffff RCX: 0000000000000004
[68632.363872] RDX: 0000000000000004 RSI: 0000000000000004 RDI: 00007fd923600000
[68632.363874] RBP: 00007ffc3532adf0 R08: 0000000000000040 R09: 0000000000000060
[68632.363875] R10: 00000000365e5970 R11: 0000000000000000 R12: 00007ffc3532c240
[68632.363877] R13: 00007ffc3532d690 R14: 00007fd92a043000 R15: 0000000003df8e50
[68632.363880]  </TASK>

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Pedro Falcato]

On Sun, Aug 25, 2024 at 07:55:36AM GMT, Piotr Oniszczuk wrote:
> 
> 
> > Wiadomość napisana przez Nhat Pham <nphamcs@gmail.com> w dniu 23.08.2024, o godz. 18:16:
> > 
> > 
> > Have you tried with 6.9 yet? IIRC, there are two major changes to
> > zswap architecture in recent versions.
> > 
> > 1. In 6.9, we range-partition zswap's rbtrees to reduce lock contention.
> > 
> 
> Ok - after 32h of continuous compilation also on 6.9.12 I got series of oops (see below).
>

Since you have a reliable-ish repro: Could you compile a KASAN kernel and run that? Note that
KASAN has a very real performance hit (if you're using this for prod) but it'll probably help
shake out the bug.

> [68616.350398] watchdog: BUG: soft lockup - CPU#4 stuck for 596s! [kcompactd0:176]
<snip>
> [68616.350490]  ? hrtimer_interrupt+0xfa/0x230
> [68616.350492]  ? __sysvec_apic_timer_interrupt+0x55/0x150
> [68616.350494]  ? sysvec_apic_timer_interrupt+0x6c/0x90
> [68616.350497]  </IRQ>
> [68616.350498]  <TASK>
> [68616.350500]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
> [68616.350503]  ? native_queued_spin_lock_slowpath+0x6e/0x2e0
> [68616.350506]  _raw_spin_lock+0x29/0x30
> [68616.350509]  page_vma_mapped_walk+0x6a2/0x950

I don't understand what this is spinning on here. Page table lock?
Could you get the file/line number from this address?

> [68616.350511]  try_to_migrate_one+0x174/0xbf0
> [68616.350515]  rmap_walk_anon+0xb6/0x190
> [68616.350518]  try_to_migrate+0xf9/0x110
> [68616.350520]  ? __pfx_try_to_migrate_one+0x10/0x10
> [68616.350523]  ? __pfx_folio_not_mapped+0x10/0x10
> [68616.350526]  ? __pfx_folio_lock_anon_vma_read+0x10/0x10
> [68616.350528]  ? __pfx_invalid_migration_vma+0x10/0x10
> [68616.350531]  migrate_pages_batch+0x545/0xb80
> [68616.350534]  ? __pfx_compaction_free+0x10/0x10
> [68616.350536]  ? __pfx_compaction_alloc+0x10/0x10
> [68616.350540]  ? __pfx_remove_migration_pte+0x10/0x10
> [68616.350542]  migrate_pages+0xada/0xd90
> [68616.350545]  ? __pfx_compaction_alloc+0x10/0x10
> [68616.350548]  ? __pfx_compaction_free+0x10/0x10
> [68616.350551]  ? folio_add_lru+0x5f/0xb0
> [68616.350553]  compact_zone+0x9e8/0x10c0
<snip>
> [68620.214430] RSP: 0000:ffffb2688397fbe0 EFLAGS: 00000202
> [68620.214432] RAX: 00000000000c0101 RBX: ffff9388140cf738 RCX: 0000000000000008
> [68620.214434] RDX: 0000000000000000 RSI: 0000000000000101 RDI: ffff9388140cf738
> [68620.214436] RBP: ffff938b2e6373c0 R08: ffff938b2e6310c0 R09: 000000000000000a
> [68620.214438] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000040000
> [68620.214440] R13: 0000000000040000 R14: ffff9388140cf738 R15: ffff9388140cf730
> [68620.214442] FS:  00007fc78bf83f00(0000) GS:ffff938b2e600000(0000) knlGS:0000000000000000
> [68620.214445] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [68620.214447] CR2: 00007fc75b53f264 CR3: 00000001797f4000 CR4: 0000000000350ef0
> [68620.214449] Call Trace:
> [68620.214450]  <IRQ>
> [68620.214451]  ? watchdog_timer_fn+0x1dd/0x260
> [68620.214454]  ? __pfx_watchdog_timer_fn+0x10/0x10
> [68620.214457]  ? __hrtimer_run_queues+0x10f/0x2a0
> [68620.214460]  ? hrtimer_interrupt+0xfa/0x230
> [68620.214462]  ? __sysvec_apic_timer_interrupt+0x55/0x150
> [68620.214465]  ? sysvec_apic_timer_interrupt+0x6c/0x90
> [68620.214468]  </IRQ>
> [68620.214469]  <TASK>
> [68620.214470]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
> [68620.214474]  ? native_queued_spin_lock_slowpath+0x21f/0x2e0
> [68620.214477]  _raw_spin_lock+0x29/0x30
> [68620.214479]  zswap_load+0x6a/0x160

... and I don't how a zswap lock could be related to a page table lock (in case it is one).

> [68620.214482]  swap_read_folio+0x64/0x450
> [68620.214484]  swapin_readahead+0x1ea/0x4e0
> [68620.214487]  do_swap_page+0x403/0xd20
> [68620.214490]  ? shmem_file_write_iter+0x5e/0x90
> [68620.214492]  ? __pte_offset_map+0x1b/0x180
> [68620.214494]  __handle_mm_fault+0x868/0xdd0
> [68620.214498]  handle_mm_fault+0x18d/0x320
> [68620.214500]  do_user_addr_fault+0x175/0x6b0
> [68620.214503]  exc_page_fault+0x7e/0x180
> [68620.214505]  asm_exc_page_fault+0x26/0x30
<snip>
> [68620.214508] RIP: 0033:0x330a353
> [68620.214512] Code: e2 03 48 01 d0 48 8b 00 48 89 45 e8 48 83 7d e8 00 0f 84 11 01 00 00 48 83 7d e8 ff 75 08 8b 45 fc 89 45 f8 eb 42 48 8b 45 e8 <8b> 40 0c 39 45 84 75 36 48 8b 45 e8 8b 40 08 48 8b 55 88 39 d0 75
> [68620.214515] RSP: 002b:00007ffc42ae0390 EFLAGS: 00010217
> [68620.214517] RAX: 00007fc75b53f258 RBX: 00000000000003e9 RCX: 00000000f9107c14
> [68620.214519] RDX: 000000000003e0a0 RSI: 00007ffc42ae04f1 RDI: 0000000027ef6180
> [68620.214521] RBP: 00007ffc42ae0410 R08: 0000000000000000 R09: 0000000000000000
> [68620.214523] R10: 00007fc78c106ac0 R11: 00007fc78c1073c0 R12: 0000000000000000
> [68620.214525] R13: 00007ffc42ae1030 R14: 00007fc78c66f000 R15: 0000000003df8e50
> [68620.214528]  </TASK>
> [68632.363664] watchdog: BUG: soft lockup - CPU#8 stuck for 648s! [cc1plus:2982130]
> [68632.363668] Modules linked in: tls rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs rpcrdma rdma_cm iw_cm ib_cm ib_core br_netfilter iptable_filter xt_physdev tun bridge stp llc ext4 crc16 mbcache jbd2 amd_atl intel_rapl_msr intel_rapl_common edac_mce_amd kvm_amd cfg80211 rfkill kvm crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 r8169 aesni_intel crypto_simd cryptd realtek mdio_devres sp5100_tco wmi_bmof k10temp libphy ccp pcspkr rapl i2c_piix4 acpi_cpufreq zenpower ryzen_smu gpio_amdpt gpio_generic mac_hid nfsd auth_rpcgss nfs_acl lockd grace nct6775 nct6775_core hwmon_vid sunrpc sg crypto_user fuse dm_mod loop nfnetlink bpf_preload ip_tables x_tables xfs libcrc32c crc32c_generic drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm drm_exec mxm_wmi nvme drm_display_helper crc32c_intel xhci_pci nvme_core cec xhci_pci_renesas wmi virtio_net net_failover failover virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev
> [68632.363704]  [last unloaded: nouveau]
> [68632.363719] CPU: 8 PID: 2982130 Comm: cc1plus Tainted: G      D W    L     6.9.12-12 #1 e59bce453550af16b12fd25785f4d449e921764e
> [68632.363722] Hardware name: To Be Filled By O.E.M. B450M Pro4-F R2.0/B450M Pro4-F R2.0, BIOS P10.08 01/19/2024
> [68632.363724] RIP: 0010:native_queued_spin_lock_slowpath+0x2a1/0x2e0
> [68632.363728] Code: c1 ea 12 83 e0 03 83 ea 01 48 c1 e0 05 48 63 d2 48 05 c0 73 03 00 48 03 04 d5 40 32 91 aa 48 89 28 8b 45 08 85 c0 75 09 f3 90 <8b> 45 08 85 c0 74 f7 48 8b 55 00 48 85 d2 0f 84 6a ff ff ff 0f 0d
> [68632.363732] RSP: 0000:ffffb26885e1f450 EFLAGS: 00000246
> [68632.363734] RAX: 0000000000000000 RBX: ffff9388140cf738 RCX: fffffbc30f2ad340
> [68632.363736] RDX: 0000000000000014 RSI: 0000000000540101 RDI: ffff9388140cf738
> [68632.363738] RBP: ffff938b2ea373c0 R08: ffff93876cc2aaa0 R09: 0008c8130ae03aa0
> [68632.363740] R10: 020f0008c8130ae0 R11: 0000000000501000 R12: 0000000000240000
> [68632.363741] R13: 0000000000240000 R14: 03ffffffffffffff R15: 00000000000005fa
> [68632.363743] FS:  00007fd929957f00(0000) GS:ffff938b2ea00000(0000) knlGS:0000000000000000
> [68632.363746] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [68632.363748] CR2: 00007fd923600000 CR3: 0000000162ba8000 CR4: 0000000000350ef0
> [68632.363750] Call Trace:
> [68632.363751]  <IRQ>
> [68632.363752]  ? watchdog_timer_fn+0x1dd/0x260
> [68632.363755]  ? __pfx_watchdog_timer_fn+0x10/0x10
> [68632.363758]  ? __hrtimer_run_queues+0x10f/0x2a0
> [68632.363761]  ? hrtimer_interrupt+0xfa/0x230
> [68632.363763]  ? __sysvec_apic_timer_interrupt+0x55/0x150
> [68632.363766]  ? sysvec_apic_timer_interrupt+0x6c/0x90
> [68632.363769]  </IRQ>
> [68632.363770]  <TASK>
> [68632.363771]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
> [68632.363775]  ? native_queued_spin_lock_slowpath+0x2a1/0x2e0
> [68632.363778]  _raw_spin_lock+0x29/0x30
> [68632.363781]  zswap_store+0x623/0xc70

FWIW this is the same zswap lock as above.

Also, could you try a memtest86 on your machine, to shake out potential hardware problems?

All-in-all if the above is a page table lock then this is a weird bug, because I don't see
how a zswap lock could be related to a ptlock through memory corruption, since ptdescs are just
struct pages... Either this is has to be a different bug than the one I reported back then, or
there's some side effect that's non-obvious to me.

-- 
Pedro

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Piotr Oniszczuk]

> Wiadomość napisana przez Pedro Falcato <pedro.falcato@gmail.com> w dniu 25.08.2024, o godz. 17:05:
> 
> Also, could you try a memtest86 on your machine, to shake out potential hardware problems?


I found less time consuming way to trigger issue: 12c24t cross compile of llvm with „only 16G” of ram - as this triggers many heavy swappings (top swap usage gets 8-9G out of 16G swap part)

With such setup - on 6.9.12 - i’m getting not available system (due cpu soft lockup) just in 1..3h
(usually first or second compile iteration; i wrote simple scrip compiling in loop + counting interations)

Then i switched back to 6.8.2 and…. decided interrupt successful test after 14 iterations (it was 19h of non-stop compiling with heavy swapping)
(sure - i can still keep test going - but builder is also needed for my normal development)

So summarising:
-on 6.9+ just 1..3h seems enough to provoke issue
-on 6.8.2 so far never in past + currently can’t provoke issue (last day in 19h test window nor 5 full distro builds nor last 2 weeks of development)

By above i personally don’t believe issue is in hw (ram)


   

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Yosry Ahmed]

On Sun, Aug 25, 2024 at 9:24 AM Piotr Oniszczuk
<piotr.oniszczuk@gmail.com> wrote:
>
>
>
> > Wiadomość napisana przez Pedro Falcato <pedro.falcato@gmail.com> w dniu 25.08.2024, o godz. 17:05:
> >
> > Also, could you try a memtest86 on your machine, to shake out potential hardware problems?
>
>
> I found less time consuming way to trigger issue: 12c24t cross compile of llvm with „only 16G” of ram - as this triggers many heavy swappings (top swap usage gets 8-9G out of 16G swap part)
>
> With such setup - on 6.9.12 - i’m getting not available system (due cpu soft lockup) just in 1..3h
> (usually first or second compile iteration; i wrote simple scrip compiling in loop + counting interations)

Are we sure that the soft lockup problem is related to the originally
reported problem? It seems like in v6.10 you hit a BUG in zswap
(corruption?), and in v6.9 you hit a soft lockup with a zswap lock
showing up in the splat. Not sure how they are relevant.

Is the soft lockup reproducible in v6.10 as well?

Since you have a narrow window (6.8.2 to 6.9) and a reproducer for the
soft lockup problem, can you try bisecting?

Thanks!

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Piotr Oniszczuk]

> Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 27.08.2024, o godz. 20:48:
> 
> On Sun, Aug 25, 2024 at 9:24 AM Piotr Oniszczuk
> <piotr.oniszczuk@gmail.com> wrote:
>> 
>> 
>> 
>>> Wiadomość napisana przez Pedro Falcato <pedro.falcato@gmail.com> w dniu 25.08.2024, o godz. 17:05:
>>> 
>>> Also, could you try a memtest86 on your machine, to shake out potential hardware problems?
>> 
>> 
>> I found less time consuming way to trigger issue: 12c24t cross compile of llvm with „only 16G” of ram - as this triggers many heavy swappings (top swap usage gets 8-9G out of 16G swap part)
>> 
>> With such setup - on 6.9.12 - i’m getting not available system (due cpu soft lockup) just in 1..3h
>> (usually first or second compile iteration; i wrote simple scrip compiling in loop + counting interations)
> 
> Are we sure that the soft lockup problem is related to the originally
> reported problem? It seems like in v6.10 you hit a BUG in zswap
> (corruption?), and in v6.9 you hit a soft lockup with a zswap lock
> showing up in the splat. Not sure how they are relevant.

If so then i’m interpreting this as:

a\ 2 different bugs 

or 

b\ 6.10 issue is result of 6.9 bug

In such case i think we may:

1. fix 6.9 first (=get it stable for let say 30h continuous compil.)
2. apply fix to 6.10 then test stability on 6.10 

> 
> Is the soft lockup reproducible in v6.10 as well?
> 
> Since you have a narrow window (6.8.2 to 6.9) and a reproducer for the
> soft lockup problem, can you try bisecting?
> 
> Thanks!



May you pls help me with reducing amount of work here? 

1. by narrowing # of bisect iternations?
On my side each iteration is like
-build arch pkg
-install on builder
-compile till first hang (2..3h probably for bad) or 20h (for good)
this means days and i’m a bit short with time as all this is my hobby (so competes with all rest of my life...)

or

2. Ideally will be to have list of revert 6.9 commit candidates (starting from most probable falling commit)
i’ll revert and test

i’ll really appreciate help here….

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Yosry Ahmed]

On Thu, Aug 29, 2024 at 8:51 AM Piotr Oniszczuk
<piotr.oniszczuk@gmail.com> wrote:
>
>
>
> > Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 27.08.2024, o godz. 20:48:
> >
> > On Sun, Aug 25, 2024 at 9:24 AM Piotr Oniszczuk
> > <piotr.oniszczuk@gmail.com> wrote:
> >>
> >>
> >>
> >>> Wiadomość napisana przez Pedro Falcato <pedro.falcato@gmail.com> w dniu 25.08.2024, o godz. 17:05:
> >>>
> >>> Also, could you try a memtest86 on your machine, to shake out potential hardware problems?
> >>
> >>
> >> I found less time consuming way to trigger issue: 12c24t cross compile of llvm with „only 16G” of ram - as this triggers many heavy swappings (top swap usage gets 8-9G out of 16G swap part)
> >>
> >> With such setup - on 6.9.12 - i’m getting not available system (due cpu soft lockup) just in 1..3h
> >> (usually first or second compile iteration; i wrote simple scrip compiling in loop + counting interations)
> >
> > Are we sure that the soft lockup problem is related to the originally
> > reported problem? It seems like in v6.10 you hit a BUG in zswap
> > (corruption?), and in v6.9 you hit a soft lockup with a zswap lock
> > showing up in the splat. Not sure how they are relevant.
>
> If so then i’m interpreting this as:
>
> a\ 2 different bugs
>
> or
>
> b\ 6.10 issue is result of 6.9 bug
>
> In such case i think we may:
>
> 1. fix 6.9 first (=get it stable for let say 30h continuous compil.)
> 2. apply fix to 6.10 then test stability on 6.10
>
> >
> > Is the soft lockup reproducible in v6.10 as well?
> >
> > Since you have a narrow window (6.8.2 to 6.9) and a reproducer for the
> > soft lockup problem, can you try bisecting?
> >
> > Thanks!
>
>
>
> May you pls help me with reducing amount of work here?
>
> 1. by narrowing # of bisect iternations?

My information about the good (v6.8) and bad (v6.9) versions come from
your report. I am not sure how I can help narrow down the number of
bisect iterations. Do you mind elaborating?

> On my side each iteration is like
> -build arch pkg
> -install on builder
> -compile till first hang (2..3h probably for bad) or 20h (for good)
> this means days and i’m a bit short with time as all this is my hobby (so competes with all rest of my life...)
>
> or
>
> 2. Ideally will be to have list of revert 6.9 commit candidates (starting from most probable falling commit)
> i’ll revert and test

Looking at the zswap commits between 6.8 and 6.9, ignoring cleanups
and seemingly irrelevant patches (e.g. swapoff fixups), I think the
some likely candidates could be the following, but this is not really
based on any scientific methodology:

44c7c734a5132 mm/zswap: split zswap rb-tree
c2e2ba770200b mm/zswap: only support zswap_exclusive_loads_enabled
a230c20e63efe mm/zswap: zswap entry doesn't need refcount anymore
8409a385a6b41 mm/zswap: improve with alloc_workqueue() call
0827a1fb143fa mm/zswap: invalidate zswap entry when swap entry free

I also noticed that you are using z3fold as the zpool. Is the problem
reproducible with zsmalloc? I wouldn't be surprised if there's a
z3fold bug somewhere.

>
> i’ll really appreciate help here….
>

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Matthew Wilcox]

On Thu, Aug 29, 2024 at 02:54:25PM -0700, Yosry Ahmed wrote:
> Looking at the zswap commits between 6.8 and 6.9, ignoring cleanups
> and seemingly irrelevant patches (e.g. swapoff fixups), I think the
> some likely candidates could be the following, but this is not really
> based on any scientific methodology:
> 
> 44c7c734a5132 mm/zswap: split zswap rb-tree
> c2e2ba770200b mm/zswap: only support zswap_exclusive_loads_enabled
> a230c20e63efe mm/zswap: zswap entry doesn't need refcount anymore
> 8409a385a6b41 mm/zswap: improve with alloc_workqueue() call
> 0827a1fb143fa mm/zswap: invalidate zswap entry when swap entry free
> 
> I also noticed that you are using z3fold as the zpool. Is the problem
> reproducible with zsmalloc? I wouldn't be surprised if there's a
> z3fold bug somewhere.

You're assuming that it's a zswap/zsmalloc/... bug.  If it's a random
scribble, as suggested by Takero Funaki:

https://lore.kernel.org/linux-mm/CAPpoddere2g=kkMzrxuJ1KCG=0Hg1-1v=ppg4dON9wK=pKq2uQ@mail.gmail.com/

then focusing on zswap will not be fruitful.

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Yosry Ahmed]

On Thu, Aug 29, 2024 at 3:29 PM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Thu, Aug 29, 2024 at 02:54:25PM -0700, Yosry Ahmed wrote:
> > Looking at the zswap commits between 6.8 and 6.9, ignoring cleanups
> > and seemingly irrelevant patches (e.g. swapoff fixups), I think the
> > some likely candidates could be the following, but this is not really
> > based on any scientific methodology:
> >
> > 44c7c734a5132 mm/zswap: split zswap rb-tree
> > c2e2ba770200b mm/zswap: only support zswap_exclusive_loads_enabled
> > a230c20e63efe mm/zswap: zswap entry doesn't need refcount anymore
> > 8409a385a6b41 mm/zswap: improve with alloc_workqueue() call
> > 0827a1fb143fa mm/zswap: invalidate zswap entry when swap entry free
> >
> > I also noticed that you are using z3fold as the zpool. Is the problem
> > reproducible with zsmalloc? I wouldn't be surprised if there's a
> > z3fold bug somewhere.
>
> You're assuming that it's a zswap/zsmalloc/... bug.  If it's a random
> scribble, as suggested by Takero Funaki:
>
> https://lore.kernel.org/linux-mm/CAPpoddere2g=kkMzrxuJ1KCG=0Hg1-1v=ppg4dON9wK=pKq2uQ@mail.gmail.com/
>
> then focusing on zswap will not be fruitful.

IIUC that was for the initial bug report. Piotr reported a different
problem for v6.9 in the same thread, a soft lockup. They look
unrelated to me. Also the patch that Takero found through bisection
landed in v6.10, so it cannot be the cause of the soft lockups.

Piotr never confirmed if reverting patch Takero found fixes the
initial problem on v6.10 though, that would be useful.

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Piotr Oniszczuk]

> Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 29.08.2024, o godz. 23:54:
> 
> I also noticed that you are using z3fold as the zpool. Is the problem
> reproducible with zsmalloc? I wouldn't be surprised if there's a
> z3fold bug somewhere.
> 

Hmm - yesterday i recompiled 6.9.12 with zsmalloc and …. after 16h of continuous tests I can’t reproduce issue.
With zsmalloc 6.9.12 looks to me like stable.

With this - what will be your advice to move forward?
Is there any possibility/way to avoid bisecting? (due limited time from my side)

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Yosry Ahmed]

On Sat, Aug 31, 2024 at 2:41 AM Piotr Oniszczuk
<piotr.oniszczuk@gmail.com> wrote:
>
>
>
> > Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 29.08.2024, o godz. 23:54:
> >
> > I also noticed that you are using z3fold as the zpool. Is the problem
> > reproducible with zsmalloc? I wouldn't be surprised if there's a
> > z3fold bug somewhere.
> >
>
> Hmm - yesterday i recompiled 6.9.12 with zsmalloc and …. after 16h of continuous tests I can’t reproduce issue.
> With zsmalloc 6.9.12 looks to me like stable.

Interesting, and a little bit what I hoped for tbh.

>
> With this - what will be your advice to move forward?

Well, it's possible that some zswap change was not fully compatible
with z3fold, or surfaced a dormant bug in z3fold. Either way, my
recommendation is to use zsmalloc. I have been trying to deprecate
z3fold, and honestly you are the only person I have seen use z3fold in
a while -- which is probably why no one else reported such a problem.

> Is there any possibility/way to avoid bisecting? (due limited time from my side)

So unless you have a reason to specifically use z3fold or avoid
zsmalloc, please use zsmalloc. It should be better for you anyway. I
doubt that you (or anyone) wants to spend time debugging a z3fold
problem :)

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Piotr Oniszczuk]

> Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 31.08.2024, o godz. 19:23:
> 
> On Sat, Aug 31, 2024 at 2:41 AM Piotr Oniszczuk
> <piotr.oniszczuk@gmail.com> wrote:
>> 
>> 
>> 
>>> Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 29.08.2024, o godz. 23:54:
>>> 
>>> I also noticed that you are using z3fold as the zpool. Is the problem
>>> reproducible with zsmalloc? I wouldn't be surprised if there's a
>>> z3fold bug somewhere.
>>> 
>> 
>> Hmm - yesterday i recompiled 6.9.12 with zsmalloc and …. after 16h of continuous tests I can’t reproduce issue.
>> With zsmalloc 6.9.12 looks to me like stable.
> 
> Interesting, and a little bit what I hoped for tbh.

:-)

I tested mainline 6.10.7 with 26h test and also it is stable with zsmalloc 

> 
>> 
>> With this - what will be your advice to move forward?
> 
> Well, it's possible that some zswap change was not fully compatible
> with z3fold, or surfaced a dormant bug in z3fold. Either way, my
> recommendation is to use zsmalloc.
> I have been trying to deprecate

IMHO - isn’t bug in this report + difficulties to reproduce->fix enough to depreciate z3fold?  

> z3fold, and honestly you are the only person I have seen use z3fold in
> a while -- which is probably why no one else reported such a problem.

Well - in fact this is ArchLinux - not me.
I’m using Arch and kernel in builder machine with ArchLinux config + packaging

> 
>> Is there any possibility/way to avoid bisecting? (due limited time from my side)
> 
> So unless you have a reason to specifically use z3fold or avoid
> zsmalloc, please use zsmalloc. It should be better for you anyway. I

I see benefits already: on very memory demanding qtwebkit compile:
z3fold: swap frequently gets 6..8G from 16G available
zsmalloc: can’t see more than 1..2G
  
> doubt that you (or anyone) wants to spend time debugging a z3fold
> problem :)

lets depreciate it!

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Yosry Ahmed]

On Mon, Sep 2, 2024 at 1:58 AM Piotr Oniszczuk
<piotr.oniszczuk@gmail.com> wrote:
>
>
>
> > Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 31.08.2024, o godz. 19:23:
> >
> > On Sat, Aug 31, 2024 at 2:41 AM Piotr Oniszczuk
> > <piotr.oniszczuk@gmail.com> wrote:
> >>
> >>
> >>
> >>> Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 29.08.2024, o godz. 23:54:
> >>>
> >>> I also noticed that you are using z3fold as the zpool. Is the problem
> >>> reproducible with zsmalloc? I wouldn't be surprised if there's a
> >>> z3fold bug somewhere.
> >>>
> >>
> >> Hmm - yesterday i recompiled 6.9.12 with zsmalloc and …. after 16h of continuous tests I can’t reproduce issue.
> >> With zsmalloc 6.9.12 looks to me like stable.
> >
> > Interesting, and a little bit what I hoped for tbh.
>
> :-)
>
> I tested mainline 6.10.7 with 26h test and also it is stable with zsmalloc
>
> >
> >>
> >> With this - what will be your advice to move forward?
> >
> > Well, it's possible that some zswap change was not fully compatible
> > with z3fold, or surfaced a dormant bug in z3fold. Either way, my
> > recommendation is to use zsmalloc.
> > I have been trying to deprecate
>
> IMHO - isn’t bug in this report + difficulties to reproduce->fix enough to depreciate z3fold?

I would say this bug report is yet another reason why we should deprecate it.

>
> > z3fold, and honestly you are the only person I have seen use z3fold in
> > a while -- which is probably why no one else reported such a problem.
>
> Well - in fact this is ArchLinux - not me.
> I’m using Arch and kernel in builder machine with ArchLinux config + packaging

According to [1], zsmalloc should be the default allocator for zswap
on ArchLinux. Anyway, I initially thought that no one was using z3fold
and it was bitrot, but apparently some people are using it and it's
actively harming them.

[1]https://wiki.archlinux.org/title/Zswap

>
> >
> >> Is there any possibility/way to avoid bisecting? (due limited time from my side)
> >
> > So unless you have a reason to specifically use z3fold or avoid
> > zsmalloc, please use zsmalloc. It should be better for you anyway. I
>
> I see benefits already: on very memory demanding qtwebkit compile:
> z3fold: swap frequently gets 6..8G from 16G available
> zsmalloc: can’t see more than 1..2G
>
> > doubt that you (or anyone) wants to spend time debugging a z3fold
> > problem :)
>
> lets depreciate it!

I tried deprecating it before [2] and performed some analysis [3], but
there was some.. resistance. Maybe I will try again and use this bug
report as yet another argument for deprecating z3fold :)

[2] https://lore.kernel.org/linux-mm/20240112193103.3798287-1-yosryahmed@google.com/
[3] https://lore.kernel.org/lkml/CAJD7tkbRF6od-2x_L8-A1QL3=2Ww13sCj4S3i4bNndqF+3+_Vg@mail.gmail.com/

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Yosry Ahmed]

On Tue, Sep 3, 2024 at 1:31 AM Aithal, Srikanth <sraithal@amd.com> wrote:
>
> On 8/25/2024 11:25 AM, Piotr Oniszczuk wrote:
> >
> >
> >> Wiadomość napisana przez Nhat Pham <nphamcs@gmail.com> w dniu 23.08.2024, o godz. 18:16:
> >>
> >>
> >> Have you tried with 6.9 yet? IIRC, there are two major changes to
> >> zswap architecture in recent versions.
> >>
> >> 1. In 6.9, we range-partition zswap's rbtrees to reduce lock contention.
> >>
> >
> > Ok - after 32h of continuous compilation also on 6.9.12 I got series of oops (see below).
> >
> I hit similar soft lockup with linuxnext-20240902 build, but I was not
> running anything for that long. Once I hit it while kexecing on
> linuxnext-20240902 and other time was during linuxnext-20240902 boot up.
> I have attached the logs here, I am trying to see if I can recreate it
> on todays linux-next build.

This doesn't look like the same problem to me. I do not see any zswap
functions in the backtrace, I see fuse stuff. Please send a separate
bug report to the relevant mailing lists (probably
linux-fsdevel@vger.kernel.org).

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Nhat Pham]

On Tue, Sep 3, 2024 at 10:49 AM Yosry Ahmed <yosryahmed@google.com> wrote:
>
> On Mon, Sep 2, 2024 at 1:58 AM Piotr Oniszczuk
> <piotr.oniszczuk@gmail.com> wrote:
> >
> >
> >
> > > Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 31.08.2024, o godz. 19:23:
> > >
> > > On Sat, Aug 31, 2024 at 2:41 AM Piotr Oniszczuk
> > > <piotr.oniszczuk@gmail.com> wrote:
> > >>
> > >>
> > >>
> > >>> Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 29.08.2024, o godz. 23:54:
> > >>>
> > >>> I also noticed that you are using z3fold as the zpool. Is the problem
> > >>> reproducible with zsmalloc? I wouldn't be surprised if there's a
> > >>> z3fold bug somewhere.
> > >>>
> > >>
> > >> Hmm - yesterday i recompiled 6.9.12 with zsmalloc and …. after 16h of continuous tests I can’t reproduce issue.
> > >> With zsmalloc 6.9.12 looks to me like stable.
> > >
> > > Interesting, and a little bit what I hoped for tbh.
> >
> > :-)
> >
> > I tested mainline 6.10.7 with 26h test and also it is stable with zsmalloc
> >
> > >
> > >>
> > >> With this - what will be your advice to move forward?
> > >
> > > Well, it's possible that some zswap change was not fully compatible
> > > with z3fold, or surfaced a dormant bug in z3fold. Either way, my
> > > recommendation is to use zsmalloc.
> > > I have been trying to deprecate
> >
> > IMHO - isn’t bug in this report + difficulties to reproduce->fix enough to depreciate z3fold?
>
> I would say this bug report is yet another reason why we should deprecate it.

+100000.

This is precisely why I was asking which allocator was being used
here. We have also accidentally selected z3fold internally a couple
times in the past, which had bitten us as well.

>
> >
> > > z3fold, and honestly you are the only person I have seen use z3fold in
> > > a while -- which is probably why no one else reported such a problem.
> >
> > Well - in fact this is ArchLinux - not me.
> > I’m using Arch and kernel in builder machine with ArchLinux config + packaging
>
> According to [1], zsmalloc should be the default allocator for zswap
> on ArchLinux. Anyway, I initially thought that no one was using z3fold
> and it was bitrot, but apparently some people are using it and it's
> actively harming them.
>
> [1]https://wiki.archlinux.org/title/Zswap
>
> >
> > >
> >
> > I see benefits already: on very memory demanding qtwebkit compile:
> > z3fold: swap frequently gets 6..8G from 16G available
> > zsmalloc: can’t see more than 1..2G

Exactly :) zsmalloc is better than z3fold in a lot of workloads that I
have observed.

> >
> > > doubt that you (or anyone) wants to spend time debugging a z3fold
> > > problem :)
> >
> > lets depreciate it!
>
> I tried deprecating it before [2] and performed some analysis [3], but
> there was some.. resistance. Maybe I will try again and use this bug
> report as yet another argument for deprecating z3fold :)
>
> [2] https://lore.kernel.org/linux-mm/20240112193103.3798287-1-yosryahmed@google.com/
> [3] https://lore.kernel.org/lkml/CAJD7tkbRF6od-2x_L8-A1QL3=2Ww13sCj4S3i4bNndqF+3+_Vg@mail.gmail.com/

I don't wanna sound like a broken record. But this has been the nth
time we need to spend extra engineering time and effort unnecessarily
because we have not deprecated z3fold.

If you need more datapoint - here's our last conversation where z3fold
was a problem:

https://lore.kernel.org/lkml/CAKEwX=Mo+EaaxBYcLMTHYADB4WhqC3QmWV3WQ0h2KM491FRuQA@mail.gmail.com/

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Yosry Ahmed]

On Tue, Sep 3, 2024 at 3:43 PM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Tue, Sep 3, 2024 at 10:49 AM Yosry Ahmed <yosryahmed@google.com> wrote:
> >
> > On Mon, Sep 2, 2024 at 1:58 AM Piotr Oniszczuk
> > <piotr.oniszczuk@gmail.com> wrote:
> > >
> > >
> > >
> > > > Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 31.08.2024, o godz. 19:23:
> > > >
> > > > On Sat, Aug 31, 2024 at 2:41 AM Piotr Oniszczuk
> > > > <piotr.oniszczuk@gmail.com> wrote:
> > > >>
> > > >>
> > > >>
> > > >>> Wiadomość napisana przez Yosry Ahmed <yosryahmed@google.com> w dniu 29.08.2024, o godz. 23:54:
> > > >>>
> > > >>> I also noticed that you are using z3fold as the zpool. Is the problem
> > > >>> reproducible with zsmalloc? I wouldn't be surprised if there's a
> > > >>> z3fold bug somewhere.
> > > >>>
> > > >>
> > > >> Hmm - yesterday i recompiled 6.9.12 with zsmalloc and …. after 16h of continuous tests I can’t reproduce issue.
> > > >> With zsmalloc 6.9.12 looks to me like stable.
> > > >
> > > > Interesting, and a little bit what I hoped for tbh.
> > >
> > > :-)
> > >
> > > I tested mainline 6.10.7 with 26h test and also it is stable with zsmalloc
> > >
> > > >
> > > >>
> > > >> With this - what will be your advice to move forward?
> > > >
> > > > Well, it's possible that some zswap change was not fully compatible
> > > > with z3fold, or surfaced a dormant bug in z3fold. Either way, my
> > > > recommendation is to use zsmalloc.
> > > > I have been trying to deprecate
> > >
> > > IMHO - isn’t bug in this report + difficulties to reproduce->fix enough to depreciate z3fold?
> >
> > I would say this bug report is yet another reason why we should deprecate it.
>
> +100000.
>
> This is precisely why I was asking which allocator was being used
> here. We have also accidentally selected z3fold internally a couple
> times in the past, which had bitten us as well.
>
> >
> > >
> > > > z3fold, and honestly you are the only person I have seen use z3fold in
> > > > a while -- which is probably why no one else reported such a problem.
> > >
> > > Well - in fact this is ArchLinux - not me.
> > > I’m using Arch and kernel in builder machine with ArchLinux config + packaging
> >
> > According to [1], zsmalloc should be the default allocator for zswap
> > on ArchLinux. Anyway, I initially thought that no one was using z3fold
> > and it was bitrot, but apparently some people are using it and it's
> > actively harming them.
> >
> > [1]https://wiki.archlinux.org/title/Zswap
> >
> > >
> > > >
> > >
> > > I see benefits already: on very memory demanding qtwebkit compile:
> > > z3fold: swap frequently gets 6..8G from 16G available
> > > zsmalloc: can’t see more than 1..2G
>
> Exactly :) zsmalloc is better than z3fold in a lot of workloads that I
> have observed.
>
> > >
> > > > doubt that you (or anyone) wants to spend time debugging a z3fold
> > > > problem :)
> > >
> > > lets depreciate it!
> >
> > I tried deprecating it before [2] and performed some analysis [3], but
> > there was some.. resistance. Maybe I will try again and use this bug
> > report as yet another argument for deprecating z3fold :)
> >
> > [2] https://lore.kernel.org/linux-mm/20240112193103.3798287-1-yosryahmed@google.com/
> > [3] https://lore.kernel.org/lkml/CAJD7tkbRF6od-2x_L8-A1QL3=2Ww13sCj4S3i4bNndqF+3+_Vg@mail.gmail.com/
>
> I don't wanna sound like a broken record. But this has been the nth
> time we need to spend extra engineering time and effort unnecessarily
> because we have not deprecated z3fold.
>
> If you need more datapoint - here's our last conversation where z3fold
> was a problem:
>
> https://lore.kernel.org/lkml/CAKEwX=Mo+EaaxBYcLMTHYADB4WhqC3QmWV3WQ0h2KM491FRuQA@mail.gmail.com/

I sent a v2 of the z3fold deprecation attempt:
https://lore.kernel.org/lkml/20240904233343.933462-1-yosryahmed@google.com/.

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Tomáš Trnka]

> Well, it's possible that some zswap change was not fully compatible
> with z3fold, or surfaced a dormant bug in z3fold. Either way, my
> recommendation is to use zsmalloc. I have been trying to deprecate
> z3fold, and honestly you are the only person I have seen use z3fold in
> a while -- which is probably why no one else reported such a problem.

FWIW, I have repeatedly hit this exact BUG (mm/zswap.c:1005) on two of my 
machines on 6.10.x (possibly 6.9.x as well, but I don't have the logs at hand 
to confirm). In both cases, this was also using z3fold under moderate memory 
pressure. I think this fairly conclusively rules out a HW issue.

Additionally, I have hit the following BUG on 6.10.8, which is potentially 
related (note __z3fold_alloc in there):

list_del corruption, ffff977c17128000->next is NULL
------------[ cut here ]------------
kernel BUG at lib/list_debug.c:52!
Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
CPU: 3 PID: 248608 Comm: kworker/u32:3 Tainted: G        W          
6.10.8-100.fc39.x86_64 #1
Hardware name: HP HP EliteBook 850 G6/8549, BIOS R70 Ver. 01.28.00 04/12/2024
Workqueue: zswap12 compact_page_work
RIP: 0010:__list_del_entry_valid_or_report+0x5d/0xc0
Code: 48 8b 01 48 39 f8 75 5a 48 8b 72 08 48 39 f0 75 65 b8 01 00 00 00 c3 cc 
cc cc cc 48 89 fe 48 c7 c7 f0 89 ba ad e8 73 34 8f ff <0f> 0b 48 89 fe 48 c7 
c7 20 8a ba ad e8 62 34 8f ff 0f 0b 48 89 fe
RSP: 0018:ffffac7299f5bdb0 EFLAGS: 00010246
RAX: 0000000000000033 RBX: ffff977c0afd0b08 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff977f2d5a18c0 RDI: ffff977f2d5a18c0
RBP: ffff977c0afd0b00 R08: 0000000000000000 R09: 4e20736920747865
R10: 7478656e3e2d3030 R11: 4c4c554e20736920 R12: ffff977c17128010
R13: 000000000000000a R14: 00000000000000a0 R15: ffff977c17128000
FS:  0000000000000000(0000) GS:ffff977f2d580000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f063638a000 CR3: 0000000179428002 CR4: 00000000003706f0
Call Trace:
 <TASK>
 ? die+0x36/0x90
 ? do_trap+0xdd/0x100
 ? __list_del_entry_valid_or_report+0x5d/0xc0
 ? do_error_trap+0x6a/0x90
 ? __list_del_entry_valid_or_report+0x5d/0xc0
 ? exc_invalid_op+0x50/0x70
 ? __list_del_entry_valid_or_report+0x5d/0xc0
 ? asm_exc_invalid_op+0x1a/0x20
 ? __list_del_entry_valid_or_report+0x5d/0xc0
 __z3fold_alloc+0x4e/0x4b0
 do_compact_page+0x20e/0xa60
 process_one_work+0x17b/0x390
 worker_thread+0x265/0x380
 ? __pfx_worker_thread+0x10/0x10
 kthread+0xcf/0x100
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x31/0x50
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1a/0x30
 </TASK>
Modules linked in: nf_conntrack_netbios_ns nf_conntrack_broadcast lp parport 
ti_usb_3410_5052 hid_logitech_hidpp snd_usb_audio snd_usbmidi_lib snd_ump 
snd_rawmidi hid_logitech_dj r8153_ecm cdc_ether usbnet r8152 mii ib_core 
dimlib tls >
 snd_hda_codec_realtek snd_hda_codec_generic snd_hda_scodec_component 
snd_soc_dmic snd_sof_pci_intel_cnl snd_sof_intel_hda_generic soundwire_intel 
soundwire_cadence snd_sof_intel_hda_common snd_sof_intel_hda_mlink 
snd_sof_intel_hda snd>
 processor_thermal_device_pci_legacy intel_cstate hp_wmi 
processor_thermal_device snd_timer sparse_keymap processor_thermal_wt_hint 
intel_uncore intel_wmi_thunderbolt thunderbolt wmi_bmof cfg80211 snd 
processor_thermal_rfim i2c_i801 sp>
---[ end trace 0000000000000000 ]---
RIP: 0010:__list_del_entry_valid_or_report+0x5d/0xc0
Code: 48 8b 01 48 39 f8 75 5a 48 8b 72 08 48 39 f0 75 65 b8 01 00 00 00 c3 cc 
cc cc cc 48 89 fe 48 c7 c7 f0 89 ba ad e8 73 34 8f ff <0f> 0b 48 89 fe 48 c7 
c7 20 8a ba ad e8 62 34 8f ff 0f 0b 48 89 fe
RSP: 0018:ffffac7299f5bdb0 EFLAGS: 00010246
RAX: 0000000000000033 RBX: ffff977c0afd0b08 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff977f2d5a18c0 RDI: ffff977f2d5a18c0
RBP: ffff977c0afd0b00 R08: 0000000000000000 R09: 4e20736920747865
R10: 7478656e3e2d3030 R11: 4c4c554e20736920 R12: ffff977c17128010
R13: 000000000000000a R14: 00000000000000a0 R15: ffff977c17128000
FS:  0000000000000000(0000) GS:ffff977f2d580000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f063638a000 CR3: 0000000179428002 CR4: 00000000003706f0
note: kworker/u32:3[248608] exited with preempt_count 3

> > Is there any possibility/way to avoid bisecting? (due limited time from my
> > side)> 
> So unless you have a reason to specifically use z3fold or avoid
> zsmalloc, please use zsmalloc. It should be better for you anyway. I
> doubt that you (or anyone) wants to spend time debugging a z3fold
> problem :)

I could conceivably try to bisect this, but since I don't have a quick 
reproducer, it would likely take weeks to finish. I'm wondering whether it's 
worth trying or if z3fold is going out of the door anyway. I don't think it's 
hardware-related so it should be possible to test this in a VM, but that still 
takes some effort to set up.

Best regards,

Tomáš Trnka

Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

[Yosry Ahmed]

On Fri, Sep 13, 2024 at 2:03 AM Tomáš Trnka <trnka@scm.com> wrote:
>
> > Well, it's possible that some zswap change was not fully compatible
> > with z3fold, or surfaced a dormant bug in z3fold. Either way, my
> > recommendation is to use zsmalloc. I have been trying to deprecate
> > z3fold, and honestly you are the only person I have seen use z3fold in
> > a while -- which is probably why no one else reported such a problem.
>
> FWIW, I have repeatedly hit this exact BUG (mm/zswap.c:1005) on two of my
> machines on 6.10.x (possibly 6.9.x as well, but I don't have the logs at hand
> to confirm). In both cases, this was also using z3fold under moderate memory
> pressure. I think this fairly conclusively rules out a HW issue.
>
> Additionally, I have hit the following BUG on 6.10.8, which is potentially
> related (note __z3fold_alloc in there):
>
> list_del corruption, ffff977c17128000->next is NULL
> ------------[ cut here ]------------
> kernel BUG at lib/list_debug.c:52!
> Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 3 PID: 248608 Comm: kworker/u32:3 Tainted: G        W
> 6.10.8-100.fc39.x86_64 #1
> Hardware name: HP HP EliteBook 850 G6/8549, BIOS R70 Ver. 01.28.00 04/12/2024
> Workqueue: zswap12 compact_page_work
> RIP: 0010:__list_del_entry_valid_or_report+0x5d/0xc0
> Code: 48 8b 01 48 39 f8 75 5a 48 8b 72 08 48 39 f0 75 65 b8 01 00 00 00 c3 cc
> cc cc cc 48 89 fe 48 c7 c7 f0 89 ba ad e8 73 34 8f ff <0f> 0b 48 89 fe 48 c7
> c7 20 8a ba ad e8 62 34 8f ff 0f 0b 48 89 fe
> RSP: 0018:ffffac7299f5bdb0 EFLAGS: 00010246
> RAX: 0000000000000033 RBX: ffff977c0afd0b08 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffff977f2d5a18c0 RDI: ffff977f2d5a18c0
> RBP: ffff977c0afd0b00 R08: 0000000000000000 R09: 4e20736920747865
> R10: 7478656e3e2d3030 R11: 4c4c554e20736920 R12: ffff977c17128010
> R13: 000000000000000a R14: 00000000000000a0 R15: ffff977c17128000
> FS:  0000000000000000(0000) GS:ffff977f2d580000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f063638a000 CR3: 0000000179428002 CR4: 00000000003706f0
> Call Trace:
>  <TASK>
>  ? die+0x36/0x90
>  ? do_trap+0xdd/0x100
>  ? __list_del_entry_valid_or_report+0x5d/0xc0
>  ? do_error_trap+0x6a/0x90
>  ? __list_del_entry_valid_or_report+0x5d/0xc0
>  ? exc_invalid_op+0x50/0x70
>  ? __list_del_entry_valid_or_report+0x5d/0xc0
>  ? asm_exc_invalid_op+0x1a/0x20
>  ? __list_del_entry_valid_or_report+0x5d/0xc0
>  __z3fold_alloc+0x4e/0x4b0
>  do_compact_page+0x20e/0xa60
>  process_one_work+0x17b/0x390
>  worker_thread+0x265/0x380
>  ? __pfx_worker_thread+0x10/0x10
>  kthread+0xcf/0x100
>  ? __pfx_kthread+0x10/0x10
>  ret_from_fork+0x31/0x50
>  ? __pfx_kthread+0x10/0x10
>  ret_from_fork_asm+0x1a/0x30
>  </TASK>
> Modules linked in: nf_conntrack_netbios_ns nf_conntrack_broadcast lp parport
> ti_usb_3410_5052 hid_logitech_hidpp snd_usb_audio snd_usbmidi_lib snd_ump
> snd_rawmidi hid_logitech_dj r8153_ecm cdc_ether usbnet r8152 mii ib_core
> dimlib tls >
>  snd_hda_codec_realtek snd_hda_codec_generic snd_hda_scodec_component
> snd_soc_dmic snd_sof_pci_intel_cnl snd_sof_intel_hda_generic soundwire_intel
> soundwire_cadence snd_sof_intel_hda_common snd_sof_intel_hda_mlink
> snd_sof_intel_hda snd>
>  processor_thermal_device_pci_legacy intel_cstate hp_wmi
> processor_thermal_device snd_timer sparse_keymap processor_thermal_wt_hint
> intel_uncore intel_wmi_thunderbolt thunderbolt wmi_bmof cfg80211 snd
> processor_thermal_rfim i2c_i801 sp>
> ---[ end trace 0000000000000000 ]---
> RIP: 0010:__list_del_entry_valid_or_report+0x5d/0xc0
> Code: 48 8b 01 48 39 f8 75 5a 48 8b 72 08 48 39 f0 75 65 b8 01 00 00 00 c3 cc
> cc cc cc 48 89 fe 48 c7 c7 f0 89 ba ad e8 73 34 8f ff <0f> 0b 48 89 fe 48 c7
> c7 20 8a ba ad e8 62 34 8f ff 0f 0b 48 89 fe
> RSP: 0018:ffffac7299f5bdb0 EFLAGS: 00010246
> RAX: 0000000000000033 RBX: ffff977c0afd0b08 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffff977f2d5a18c0 RDI: ffff977f2d5a18c0
> RBP: ffff977c0afd0b00 R08: 0000000000000000 R09: 4e20736920747865
> R10: 7478656e3e2d3030 R11: 4c4c554e20736920 R12: ffff977c17128010
> R13: 000000000000000a R14: 00000000000000a0 R15: ffff977c17128000
> FS:  0000000000000000(0000) GS:ffff977f2d580000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f063638a000 CR3: 0000000179428002 CR4: 00000000003706f0
> note: kworker/u32:3[248608] exited with preempt_count 3
>
> > > Is there any possibility/way to avoid bisecting? (due limited time from my
> > > side)>
> > So unless you have a reason to specifically use z3fold or avoid
> > zsmalloc, please use zsmalloc. It should be better for you anyway. I
> > doubt that you (or anyone) wants to spend time debugging a z3fold
> > problem :)
>
> I could conceivably try to bisect this, but since I don't have a quick
> reproducer, it would likely take weeks to finish. I'm wondering whether it's
> worth trying or if z3fold is going out of the door anyway. I don't think it's
> hardware-related so it should be possible to test this in a VM, but that still
> takes some effort to set up.

z3fold is going out of the door anyway, I already sent a patch to deprecate it:
https://lore.kernel.org/lkml/20240904233343.933462-1-yosryahmed@google.com/

I will send a new version after the merge window, and I will include
your bug report in the list of problems in the commit log :) Thanks
for the report, please don't waste time debugging this and use
zsmalloc!

>
> Best regards,
>
> Tomáš Trnka
>
>