Since 6.17.5 I have twice had one of my servers lock up in a
state where it still routed network traffic and responded to pings but
no user programs are running.
These are proceeded by kernel splats:
[542551.007650] [ T238] INFO: task php8.3:1373100 blocked for more
than 614 seconds.
[542551.007652] [ T238] Tainted: G S W E 6.17.6 #1
[542551.007653] [ T238] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[542551.007654] [ T238] task:php8.3 state:D stack:0
pid:1373100 tgid:1373100 ppid:5513 task_flags:0x400000 flags:0x00004002
[542551.007657] [ T238] Call Trace:
[542551.007658] [ T238]
[542551.007659] [ T238] __schedule+0x41c/0x16b0
[542551.007661] [ T238] ? _raw_spin_unlock_bh+0x1d/0x30
[542551.007664] [ T238] ? tcp_cleanup_rbuf+0x43/0xa0
[542551.007670] [ T238] schedule+0x20/0xe0
[542551.007671] [ T238] io_schedule+0x4c/0x80
[542551.007673] [ T238] folio_wait_bit+0x102/0x200
[542551.007676] [ T238] ? __pfx_wake_page_function+0x10/0x10
[542551.007678] [ T238] folio_wait_writeback+0x2e/0x80
[542551.007680] [ T238] shmem_swapin_folio+0x58d/0x1210
[542551.007683] [ T238] ? mod_memcg_lruvec_state+0xed/0x2b0
[542551.007686] [ T238] ? xas_load+0x11/0x100
[542551.007688] [ T238] ? filemap_get_entry+0x60/0x1c0
[542551.007690] [ T238] ? __lruvec_stat_mod_folio+0x7a/0xc0
[542551.007693] [ T238] shmem_get_folio_gfp+0x17a/0x5f0
[542551.007696] [ T238] shmem_fault+0x7a/0x390
[542551.007698] [ T238] __do_fault+0x38/0x1b0
[542551.007701] [ T238] do_fault+0x2bc/0x5b0
[542551.007703] [ T238] ? __x64_sys_alarm+0x61/0xb0
[542551.007707] [ T238] __handle_mm_fault+0x439/0xfc0
[542551.007709] [ T238] ? x64_sys_call+0x17e7/0x2330
[542551.007715] [ T238] handle_mm_fault+0xeb/0x2f0
[542551.007718] [ T238] do_user_addr_fault+0x203/0x690
[542551.007720] [ T238] exc_page_fault+0x7f/0x170
[542551.007722] [ T238] asm_exc_page_fault+0x27/0x30
[542551.007724] [ T238] RIP: 0033:0x73c3e46faffd
[542551.007725] [ T238] RSP: 002b:00007ffc77600ea0 EFLAGS: 00010246
[542551.007727] [ T238] RAX: 000073c46cc14150 RBX: 000073c46cc14120
RCX: 000073c46cc141c0
[542551.007728] [ T238] RDX: 00006198bd151188 RSI: 000073c364698118
RDI: 000073c384cf5718
[542551.007729] [ T238] RBP: 00007ffc77600f40 R08: 000000000000014d
R09: 000073c36414e348
[542551.007730] [ T238] R10: 0000000000000000 R11: 00007ffc77600d48
R12: 0000000000000000
[542551.007732] [ T238] R13: 0000000000000000 R14: 000073c46cc14250
R15: 000073c384cfb350
[542551.007734] [ T238]
[542551.008011] [ T238] Kernel panic - not syncing: hung_task:
blocked tasks
[542551.008013] [ T238] CPU: 16 UID: 0 PID: 238 Comm: khungtaskd
Kdump: loaded Tainted: G S W E 6.17.6 #1 NONE
[542551.008017] [ T238] Tainted: [S]=CPU_OUT_OF_SPEC, [W]=WARN,
[E]=UNSIGNED_MODULE
[542551.008018] [ T238] Hardware name: Gigabyte Technology Co., Ltd.
X299X AORUS MASTER/X299X AORUS MASTER, BIOS F3m 12/06/2021
[542551.008019] [ T238] Call Trace:
[542551.008020] [ T238]
[542551.008021] [ T238] dump_stack_lvl+0x5f/0x90
[542551.008026] [ T238] dump_stack+0x10/0x18
[542551.008029] [ T238] vpanic+0x111/0x320
[542551.008032] [ T238] panic+0x67/0x67
[542551.008034] [ T238] ? show_stack+0x3c/0x50
[542551.008038] [ T238] watchdog.cold+0x2c8/0x362
[542551.008042] [ T238] ? __pfx_watchdog+0x10/0x10
[542551.008045] [ T238] kthread+0xf7/0x210
[542551.008049] [ T238] ? __pfx_kthread+0x10/0x10
[542551.008051] [ T238] ret_from_fork+0x1f6/0x220
[542551.008055] [ T238] ? __pfx_kthread+0x10/0x10
[542551.008057] [ T238] ret_from_fork_asm+0x1a/0x30
[542551.008061] [ T238]
Am running self compiled kernel on Ubuntu 24.04 system. The
.config used to generate the kernel is attached. The compiler used is
gcc 15.2. The hardware this is running on is an i9-10980xe based
machine clocked at 4.4Ghz all cores with -8 multiplier offset for
avx512, -3 multiplier offset for AVX2. The machine is equipped with
256G of RAM. There are three RAID1 arrays, one based upon two SN770 WD
Black nvme 1G drives mounted on /root, another based upon 13TB CMD
drives on /space and a third on 13TB SMR drives on /bu. We
use Apache/2.4.65 (Unix) OpenSSL/3.6.0-dev and PHP from 5.6 to 8.3, most
applications on 8.3.