[BUG] soft lockup in filemap_get_read_batch

[antal.nemes@hycu.com]

Hi Matthew,

We have observed intermittent soft lockups on at least seven different hosts:
- six hosts ran 6.2.8.fc37-200
- one host ran 6.0.13.fc37-200

The list of affected hosts is growing.

Stack traces are all similar:

emerg kern kernel - - watchdog: BUG: soft lockup - CPU#7 stuck for 17117s! [postmaster:2238460]
warning kern kernel - - Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock nbd loop nls_utf8 cifs cifs_arc4 cifs_md4 dns_resolver fscache netfs veth iscsi_tcp libiscsi_tcp libiscsi iscsi_target_mod target_core_mod scsi_transport_iscsi nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink sunrpc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua bochs drm_vram_helper drm_ttm_helper ttm crct10dif_pclmul i2c_piix4 crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 virtio_balloon joydev pcspkr xfs crc32c_intel virtio_net serio_raw ata_generic net_failover failover virtio_scsi pata_acpi qemu_fw_cfg fuse [last unloaded: nbd]
warning kern kernel - - CPU: 7 PID: 2238460 Comm: postmaster Kdump: loaded Tainted: G             L     6.2.8-200.fc37.x86_64 #1
warning kern kernel - - Hardware name: Nutanix AHV, BIOS 1.11.0-2.el7 04/01/2014
warning kern kernel - - RIP: 0010:xas_descend+0x28/0x70
warning kern kernel - - Code: 90 90 0f b6 0e 48 8b 57 08 48 d3 ea 83 e2 3f 89 d0 48 83 c0 04 48 8b 44 c6 08 48 89 77 18 48 89 c1 83 e1 03 48 83 f9 02 75 08 <48> 3d fd 00 00 00 76 08 88 57 12 c3 cc cc cc cc 48 c1 e8 02 89 c2
warning kern kernel - - RSP: 0018:ffffab66c9f4bb98 EFLAGS: 00000246
warning kern kernel - - RAX: 00000000000000c2 RBX: ffffab66c9f4bbb8 RCX: 0000000000000002
warning kern kernel - - RDX: 0000000000000032 RSI: ffff89cd6c8cd6d0 RDI: ffffab66c9f4bbb8
warning kern kernel - - RBP: ffff89cd6c8cd6d0 R08: ffffab66c9f4be20 R09: 0000000000000000
warning kern kernel - - R10: 0000000000000001 R11: 0000000000000100 R12: 00000000000000b3
warning kern kernel - - R13: 00000000000000b2 R14: 00000000000000b2 R15: ffffab66c9f4be48
warning kern kernel - - FS:  00007ff1e8bfb540(0000) GS:ffff89d35fbc0000(0000) knlGS:0000000000000000
warning kern kernel - - CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
warning kern kernel - - CR2: 00007ff1e8af0768 CR3: 000000016fdde001 CR4: 00000000003706e0
warning kern kernel - - Call Trace:
warning kern kernel - -  <TASK>
warning kern kernel - -  xas_load+0x3d/0x50
warning kern kernel - -  filemap_get_read_batch+0x179/0x270
warning kern kernel - -  filemap_get_pages+0xa9/0x690
warning kern kernel - -  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
warning kern kernel - -  filemap_read+0xd2/0x340
warning kern kernel - -  ? filemap_read+0x32f/0x340
warning kern kernel - -  xfs_file_buffered_read+0x4f/0xd0 [xfs]
warning kern kernel - -  xfs_file_read_iter+0x70/0xe0 [xfs]
warning kern kernel - -  vfs_read+0x23c/0x310
warning kern kernel - -  ksys_read+0x6b/0xf0
warning kern kernel - -  do_syscall_64+0x5b/0x80
warning kern kernel - -  ? syscall_exit_to_user_mode+0x17/0x40
warning kern kernel - -  ? do_syscall_64+0x67/0x80
warning kern kernel - -  ? do_syscall_64+0x67/0x80
warning kern kernel - -  ? __irq_exit_rcu+0x3d/0x140
warning kern kernel - -  entry_SYSCALL_64_after_hwframe+0x72/0xdc
warning kern kernel - - RIP: 0033:0x7ff1e5b20b25
warning kern kernel - - Code: fe ff ff 50 48 8d 3d 0a c9 06 00 e8 25 ee 01 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 f5 4b 2a 00 8b 00 85 c0 75 0f 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
warning kern kernel - - RSP: 002b:00007ffe1a5d8d78 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
warning kern kernel - - RAX: ffffffffffffffda RBX: 00000000035345c0 RCX: 00007ff1e5b20b25
warning kern kernel - - RDX: 0000000000002000 RSI: 00007ff1dc9c3080 RDI: 0000000000000032
warning kern kernel - - RBP: 0000000000000000 R08: 0000000000000009 R09: 0000000000000000
warning kern kernel - - R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000002000
warning kern kernel - - R13: 00007ff1dc9c3080 R14: 0000000000000000 R15: 0000000001452148
warning kern kernel - -  </TASK>

Lockup is always reported from postgres process with all data and config on a XFS filesystem. 
Because this blocks a postgres process, lockup has a bunch of knock-on effects 
(invalid page errors, hanged or aborted transactions, tuple accumulation, etc). 
All occurrences eventually required a reboot to remedy.

Issue coincided with our rollout with the 6.x kernel. Previously we ran Rocky 
Linux 8 with 4.18.* (clone of RHEL8 kernel), so I recognize that this issue may 
not be new (AFAICT, livelocks were sporadically reported since folio merge in 5.17).

Issue takes anywhere from 2 days to 30+ days since boot to materialize, and lockups
are reported for duration ranging from 1min to 7 hours (the latter until it was 
manually rebooted). This is followed by a  period of relatively high load averages
(~2*#cpus), but low CPU usage. Memory usage was < 70%, so it does not appear 
to be a high-psi condition.

We are unable to reproduce the issue at will (i.e. by load/stress testing), but
the affected hosts have had multiple occurrences across reboots, so we should
be able to observe effects of any patches over a longer span.

From what I can tell, this appears to be similar to what was reported in
https://lore.kernel.org/linux-kernel/CA+wXwBS7YTHUmxGP3JrhcKMnYQJcd6=7HE+E1v-guk01L2K3Zw@mail.gmail.com/
and 
https://lore.kernel.org/linux-fsdevel/CA+wXwBRGab3UqbLqsr8xG=ZL2u9bgyDNNea4RGfTDjqB=J3geQ@mail.gmail.com/

> > We also have a deadlock reading a very specific file on this host. We managed to
> > do a kdump on this host and extracted out the state of the mapping.
>
> This is almost certainly a different bug, but alos XArray related, so
> I'll keep looking at this one.

I am not sure if the deadlock that Daniel observed matches our stack trace. 
Assuming yes, has there been any follow-up on this?

We tried the patch from https://bugzilla.kernel.org/show_bug.cgi?id=216646#c31 , but the
soft lockup reoccurred with the same signature.

Is there anything we can do to further aid in troubleshooting? If this is a folio
lock issue, would it be possible to trace where the lock was taken?

Best regards,
Antal