From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 872F7C3DA70 for ; Thu, 25 Jul 2024 23:56:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9BFF6B009B; Thu, 25 Jul 2024 19:56:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C23246B009C; Thu, 25 Jul 2024 19:56:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A75926B009D; Thu, 25 Jul 2024 19:56:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7F9FF6B009B for ; Thu, 25 Jul 2024 19:56:23 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F05D44143B for ; Thu, 25 Jul 2024 23:56:17 +0000 (UTC) X-FDA: 82379936394.17.530E9D8 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) by imf09.hostedemail.com (Postfix) with ESMTP id 1EEF7140008 for ; Thu, 25 Jul 2024 23:56:15 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="mjX/znO7"; spf=pass (imf09.hostedemail.com: domain of anton.eidelman@gmail.com designates 209.85.221.49 as permitted sender) smtp.mailfrom=anton.eidelman@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721951751; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=0F2gh5PJh2Oegyehs6PNMuS+QGVtid8bRvr8mmR77Fs=; b=wRBzEs6sCxlZpEqke4pwtziUMeURntQPfAWFNeNNRfdiofm9lxyJz2mzFoGJV6zQYuUlOM +jrcq17ZZnFK6EZhQXeSCz5LYzyPk3mesECCiHaH/kn+2xW8SVeupOo5q6CkXYHUVMgkmK T98W0NXA5A06iSESfpHUHcVtpaPIYZU= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="mjX/znO7"; spf=pass (imf09.hostedemail.com: domain of anton.eidelman@gmail.com designates 209.85.221.49 as permitted sender) smtp.mailfrom=anton.eidelman@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721951751; a=rsa-sha256; cv=none; b=fvbgxYRCgTzfYwngbXHut0OwwCwq2jbl1gB3WTurrQ8E9znf4E9PCUb3hKP8KrxCF2Haq1 iaYDsn4BboIRiRgc28P0gd/yD5/hRDhDjWY6LTlPrdGhwXIvm21Pi6afypYB446kOLtf9M yrM7oCE+/8DDI59wJdE8qVcI2tNoA64= Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-368663d7f80so814751f8f.3 for ; Thu, 25 Jul 2024 16:56:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721951774; x=1722556574; darn=kvack.org; h=content-transfer-encoding:content-disposition:mime-version :message-id:subject:to:from:date:from:to:cc:subject:date:message-id :reply-to; bh=0F2gh5PJh2Oegyehs6PNMuS+QGVtid8bRvr8mmR77Fs=; b=mjX/znO7ULpR6riSkWSc/cvES4S5T8zscu6E8EWNLIcdMMfwf6x04JAU7f+5d3prO3 3yR+lnUJ7JtETOGiXy0zG69aFYVnnL6qa23oSS4+vWkO0FZlMBzVK1gnbXLuw9JSLzVz 3uuZq6V0V19KpIBhRyu9Z1Cx/PaFC2IBiQVggIg4fgywk2eTf+ctZoHgSVhhRLSk/Pg5 pa3P7pv0gqHsAwF28wU7wYBVF9jqJc3MBZoMRK4vdTS0GRSrooj3M8c70KDg7OjQa0rX xbyatF9wTYw+6NmjDQThFbfbXX7CPoeFw13wvvCmphWPmMLPMyBxuLpzctyDaoxFpvk/ S5HA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721951774; x=1722556574; h=content-transfer-encoding:content-disposition:mime-version :message-id:subject:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0F2gh5PJh2Oegyehs6PNMuS+QGVtid8bRvr8mmR77Fs=; b=buOXOcnylijOQAhiHx5+bOfYSjfb8rTmDBpoTUq562P5TcW1sFfcM6emTxUP8SkmLF czbzFU9W2sjZKKtXJuJyg2OMYd2J6G+dS3Oznx5lW0d/PZ/KFxxIlwepo+Kt0O/a/Aoq UvpxMrUz61N/puaYWJhaT/t0UOEe0bkwFN1Z6w7HUgq2fCLlwecnY/4CJkz+D3k0y5gt QJy61lpyeuvvN9nu7A3brXgXtaUCRbXRQb7hoqi6c6lc4uOIXA2W266Hr1OfNvn8kcKp eMW7ZryMR/s7mLNxDxDHh5B/0E4tMpHuXO8cYysmV16mwB49Lhux7faMgxODiPpvctUo FmpQ== X-Forwarded-Encrypted: i=1; AJvYcCWC6N8nJDTWedVEFBPX00aH8MLTEa8KcimhyZV6o04wrHADx5KQ84uDjnvLaXIs71sLmJ4p3MRzg9X4hoLFGh6JDj0= X-Gm-Message-State: AOJu0YyZbiX7bP+l2hgldFxxVzt32GkFLLU6h42DPxE4fWQZfFXVmaRy KP8h34xZ5z+iRkV7kcM4ERv4LocrcLfStjgCHxH2jTxlznesOrrZ5zUJLEwc X-Google-Smtp-Source: AGHT+IH369M+GbWh90cqj5bxyyi8HZqLgXcun8LPFtRDK+hAUvAoewXFCYLGUMlZ1vLnqAog/dceAw== X-Received: by 2002:a5d:4d07:0:b0:368:7edd:cd80 with SMTP id ffacd0b85a97d-36b319e6b25mr3219633f8f.20.1721951774131; Thu, 25 Jul 2024 16:56:14 -0700 (PDT) Received: from anton-latitude (c-67-166-97-155.hsd1.ut.comcast.net. [67.166.97.155]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42808f684d0sm29554245e9.6.2024.07.25.16.56.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Jul 2024 16:56:13 -0700 (PDT) Date: Thu, 25 Jul 2024 17:56:10 -0600 From: Anton Eidelman To: linux-xfs@vger.kernel.org, linux-mm@kvack.org, willy@infradead.org Subject: [BUG] xfs: soft lockup in xas_descend/xas_load/xas_next from filemap_get_read_batch (rclone) on 6.8 kernel Message-ID: <20240725235610.GA1669344@anton-latitude> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 1EEF7140008 X-Stat-Signature: s94f1bkxjn4ixdynptwmsao6d8bhuzpe X-HE-Tag: 1721951775-607224 X-HE-Meta: U2FsdGVkX1+jh+B6yaSh8/fKWPZ3JCZJI7ObfZxZDWYaZeqwQIbmW9CDiSggZ9nYu2JphA1+gpWBZn9FaS/iRX2SiD9DmQL0nX/NfVUSVoVdoXM7Of31dtbGvFCeBJDGCc6/QJqHhH8C99mA/l1Qx5OF4GJjO5v3RMGczMoTXgZdX2XV4pAHZQgfw+3GPWwpHhuD746evngljxHKLUEDJw6feCfe7y2LHr8Sve4/igt09q/7VFJhyElAmK62drQ9rQw2WqPphFi0xc8N/GULHVJQt8r5xhX3zBVnFYxK+KQahIF9TddPXhI0uFpJLEMoyjQntlzC8uFJq0TGCm9kKTC71kA7OHWAUlsYxgb+dBc+sZSwmsuBbkkrFSl5OrK+LSuEkdH/2ehRMw6DSz9kx4/B07Rd209/SQywcZJSSc5urNeitgDgpVkoKXkp0+XMZibEgD8vwIAfdGP1VAG1Jk54W0RLGc4j+kWcrCfv5yb5vOZo3BHTdufrk4ARSWLXOOKJL/ATD8g9CQOIwJtwiI+3XIZh6jrhEgP+I8ejWIQQvb2aJUlCcRv6nIvZhO3+HuXmB13n5qwhGfz6Md0M4znDLCJv/RbY6H3ZR9gse4Ri0TzSeeNnWhRILZYu8rts7WEtw8LESRBqs0Oft93fOXphmMygS/7+N9qT9cQivCE0Ybg+UjBeWeHk7jKzYTyHIrxiyrb3dviAJJw10JirYFE7LWuRMT7qmHKG1JIqi0h8ku5nML0szkCoKn+Al9ywialfs2BIHjCe/4d/IIoKr2gVAxGUIpd+klLph/dqywnue+/SH4DyqIetG5l+JW+Sb3rRbY5GvqdeO7cKCnn/0lFh8dE/BAE0NVloTGB6NbMplF69D1Abtm9ut3l0+piYhgMraJkOyLBz3LQ1ibwjJSkvQG3MqEErOyTC9R/mbiDjjiFwP6huuDjYhRJ/WO8tYnwgq8J/TG/zr8eQUoz 4O+zkv26 918IKAl9+CjU1S/kBJQ0cDUFbjVbLjbleiEY/4CEN5sIsmLch7RHGjJo2cGye+oLhrgd73T1jZfRqDTMalquirUQxBmnYpZ6nTdKYz6/SqcHAfKb/iQP+LcyLJDRa25rMmUix6GSfN2t1zTDjSEHRsNZvLoR/h2DHB+CElvHqTZp7fMxI4SEnjZ2F64ucEyX1IRNZ3KeoPlz7dTLfMWBXkfeRDlUCtmjUt6FA7eL4JkTQ3oTQ0KvK6uXpr1ioKnlrl/Mom4NG7QaoaNFXSUuGH5zFW+G0ngjAA0cC9YfgRguxM7Sl9ReYdYSiO0DZ5vh/DX9cJ9g/BDtZUmUakGlIrv0sBrUd/nh1WqxFWoE7LfVanVk7FmO7vnZtVcHEqcLrZ733Ob5y3/KMeqfJDMiXEqBjfuOjLijzTZ42Q1Ps17K3crCNCpsHJHzYj3ivW1kmMHiXFWavzW3dsRGS6I8IZvtva/MRhzB6sytI2RY+vTuspQ59ZL4BmBycFWIwRFPfde6WAeeh/HTijOO94XZjOpy3ljZ6KdWPxcgbiXDxe5G298ZTc9GXn6hY7wbBnlkaDsX5kYP0PZPDHnlMUCL4XZxq5ltJxCKv1wphi7GKhDDuuv4HBbGX1tnnLyKnKrJYTMvdzi/csC09sUM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi, We observe a consistent soft lockup while running rclone from and XFS volum= e. The call trace (samples below) look identical the ones reported on 6.8.0 in https://lore.kernel.org/linux-mm/ZcVH+FzS3NaCrsqe@dread.disaster.area/T/ We see in with ubuntu kernel 6.8.0-38-generic and 6.9. The CPU seems to be spinning and the three samples below hit xas_descend, x= as_load, and __xas_next. This also looks just like the one reported here: https://lore.kernel.org/linux-mm/ZRycfLxGP1CSd%2Fud@dread.disaster.area/T/ on 6.0 and 6.2, and the commit that was intended to fix this is present in these 6.8 and above kernels (merged in 6.6): commit cbc02854331edc6dc22d8b77b6e22e38ebc7dd51 Author: Matthew Wilcox (Oracle) Date: Wed Jul 26 22:58:17 2023 -0400 XArray: Do not return sibling entries from xa_load() Please, let me know if you have any suggestion or if I've missed any import= ant details. Thanks a lot, Anton Jul 23 16:01:16 k8-master kernel: watchdog: BUG: soft lockup - CPU#7 stuck = for 108s! [rclone:102291] Jul 23 16:01:16 k8-master kernel: Modules linked in: xfs nvme_tcp nvme_keyr= ing nvme_fabrics nvme_core nvme_auth xt_statistic nf_conntrack_netlink xt_T= PROXY nf_tproxy_ipv6 nf_tproxy_ipv4 ip_set xt_CT cls_bpf sch_ingress vxlan = ip6_udp_tunnel udp_tunnel veth xt_socket nf_socket_ipv4 nf_socket_ipv6 ip6t= able_filter ip6table_raw ip6table_mangle ip6_tables iptable_filter iptable_= raw iptable_mangle iptable_nat xfrm_user xfrm_algo xt_addrtype xt_nat xt_MA= SQUERADE xt_mark ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_chain_nat nf_nat x= t_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_comment nft_compa= t nf_tables qrtr intel_rapl_msr intel_rapl_common intel_uncore_frequency in= tel_uncore_frequency_common sb_edac x86_pkg_temp_thermal intel_powerclamp c= fg80211 coretemp kvm_intel binfmt_misc kvm irqbypass ipmi_ssif rapl intel_c= state i2c_i801 i2c_smbus ast mei_me lpc_ich mei ioatdma ipmi_si acpi_power_= meter acpi_ipmi ipmi_devintf ipmi_msghandler input_leds joydev acpi_pad mac= _hid br_netfilter dm_multipath bridge stp llc overlay msr efi_pstore nfnetl= ink Jul 23 16:01:16 k8-master kernel: dmi_sysfs ip_tables x_tables autofs4 btr= fs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq a= sync_xor async_tx xor raid6_pq libcrc32c raid1 raid0 mlx5_ib ib_uverbs macs= ec ib_core uas usb_storage hid_generic usbhid hid mlx5_core crct10dif_pclmu= l crc32_pclmul polyval_clmulni mlxfw polyval_generic igb psample ghash_clmu= lni_intel sha256_ssse3 tls ahci sha1_ssse3 xhci_pci i2c_algo_bit libahci xh= ci_pci_renesas pci_hyperv_intf wmi dca aesni_intel crypto_simd cryptd Jul 23 16:01:16 k8-master kernel: CPU: 7 PID: 102291 Comm: rclone Tainted: = G L 6.8.0-38-generic #38-Ubuntu Jul 23 16:01:16 k8-master kernel: Hardware name: Supermicro SYS-2028TR-HTR/= X10DRT-H, BIOS 2.0b 08/12/2016 Jul 23 16:01:16 k8-master kernel: RIP: 0010:xas_descend+0x60/0xd0 Jul 23 16:01:16 k8-master kernel: Code: 4d 89 65 18 48 89 c2 83 e2 03 48 83= fa 02 75 08 48 3d fd 00 00 00 76 30 41 88 5d 12 48 83 c4 08 5b 41 5c 41 5d= 41 5e 5d 31 d2 <31> c9 31 f6 31 ff c3 cc cc cc cc 48 83 fa 02 0f 94 c2 48 = 3d fd 00 Jul 23 16:01:16 k8-master kernel: RSP: 0018:ffffba1eee62f870 EFLAGS: 000002= 46 Jul 23 16:01:16 k8-master kernel: RAX: ffffde475fa91000 RBX: ffffba1eee62f8= e8 RCX: 0000000000000000 Jul 23 16:01:16 k8-master kernel: RDX: 0000000000000000 RSI: ffff93df1faf69= 10 RDI: ffffba1eee62f8e8 Jul 23 16:01:16 k8-master kernel: RBP: ffffba1eee62f888 R08: 00000000000000= 00 R09: 0000000000000000 Jul 23 16:01:16 k8-master kernel: R10: 0000000000000000 R11: 00000000000000= 00 R12: ffff93df1faf6910 Jul 23 16:01:16 k8-master kernel: R13: 0000000000024047 R14: ffffba1eee62fc= 00 R15: ffffba1eee62fc00 Jul 23 16:01:16 k8-master kernel: FS: 000000c000d00098(0000) GS:ffff93e0ef= a80000(0000) knlGS:0000000000000000 Jul 23 16:01:16 k8-master kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 00000000= 80050033 Jul 23 16:01:16 k8-master kernel: CR2: 000000c000f2b000 CR3: 00000008b4b4e0= 01 CR4: 00000000003706f0 Jul 23 16:01:16 k8-master kernel: DR0: 0000000000000000 DR1: 00000000000000= 00 DR2: 0000000000000000 Jul 23 16:01:16 k8-master kernel: DR3: 0000000000000000 DR6: 00000000fffe0f= f0 DR7: 0000000000000400 Jul 23 16:01:16 k8-master kernel: Call Trace: Jul 23 16:01:16 k8-master kernel: Jul 23 16:01:16 k8-master kernel: ? show_regs+0x6d/0x80 Jul 23 16:01:16 k8-master kernel: ? watchdog_timer_fn+0x206/0x290 Jul 23 16:01:16 k8-master kernel: ? __pfx_watchdog_timer_fn+0x10/0x10 Jul 23 16:01:16 k8-master kernel: ? __hrtimer_run_queues+0x112/0x2a0 Jul 23 16:01:16 k8-master kernel: ? clockevents_program_event+0xc1/0x150 Jul 23 16:01:16 k8-master kernel: ? hrtimer_interrupt+0xf6/0x250 Jul 23 16:01:16 k8-master kernel: ? __sysvec_apic_timer_interrupt+0x51/0x1= 50 Jul 23 16:01:16 k8-master kernel: ? sysvec_apic_timer_interrupt+0x8d/0xd0 Jul 23 16:01:16 k8-master kernel: Jul 23 16:01:16 k8-master kernel: Jul 23 16:01:16 k8-master kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0= x20 Jul 23 16:01:16 k8-master kernel: ? xas_descend+0x60/0xd0 Jul 23 16:01:16 k8-master kernel: ? xas_load+0x4c/0x60 Jul 23 16:01:16 k8-master kernel: __xas_next+0xa9/0x150 Jul 23 16:01:16 k8-master kernel: filemap_get_read_batch+0x1a3/0x2e0 Jul 23 16:01:16 k8-master kernel: ? __free_one_page+0x67/0x540 Jul 23 16:01:16 k8-master kernel: filemap_get_pages+0xa9/0x3b0 Jul 23 16:01:16 k8-master kernel: filemap_read+0xf7/0x470 Jul 23 16:01:16 k8-master kernel: ? sk_reset_timer+0x18/0x70 Jul 23 16:01:16 k8-master kernel: generic_file_read_iter+0xbb/0x110 Jul 23 16:01:16 k8-master kernel: ? down_read+0x12/0xd0 Jul 23 16:01:16 k8-master kernel: xfs_file_buffered_read+0x57/0xe0 [xfs] Jul 23 16:01:16 k8-master kernel: xfs_file_read_iter+0xb6/0x1c0 [xfs] Jul 23 16:01:16 k8-master kernel: vfs_read+0x258/0x390 Jul 23 16:01:16 k8-master kernel: ksys_read+0x73/0x100 Jul 23 16:01:16 k8-master kernel: __x64_sys_read+0x19/0x30 Jul 23 16:01:16 k8-master kernel: x64_sys_call+0x1ada/0x25c0 Jul 23 16:01:16 k8-master kernel: do_syscall_64+0x7f/0x180 Jul 23 16:01:16 k8-master kernel: ? vfs_write+0x3d6/0x480 Jul 23 16:01:16 k8-master kernel: ? ksys_write+0xe6/0x100 Jul 23 16:01:16 k8-master kernel: ? syscall_exit_to_user_mode+0x89/0x260 Jul 23 16:01:16 k8-master kernel: ? do_syscall_64+0x8c/0x180 Jul 23 16:01:16 k8-master kernel: ? syscall_exit_to_user_mode+0x89/0x260 Jul 23 16:01:16 k8-master kernel: ? do_syscall_64+0x8c/0x180 Jul 23 16:01:16 k8-master kernel: ? syscall_exit_to_user_mode+0x89/0x260 Jul 23 16:01:16 k8-master kernel: ? do_syscall_64+0x8c/0x180 Jul 23 16:01:16 k8-master kernel: ? restore_fpregs_from_fpstate+0x47/0xf0 Jul 23 16:01:16 k8-master kernel: ? switch_fpu_return+0x55/0xf0 Jul 23 16:01:16 k8-master kernel: ? irqentry_exit_to_user_mode+0x7e/0x260 Jul 23 16:01:16 k8-master kernel: ? irqentry_exit+0x43/0x50 Jul 23 16:01:16 k8-master kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80 Jul 23 16:01:16 k8-master kernel: RIP: 0033:0x40708e Jul 23 16:01:16 k8-master kernel: Code: 48 83 ec 38 e8 13 00 00 00 48 83 c4= 38 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce= 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff = ff ff 48 Jul 23 16:01:16 k8-master kernel: RSP: 002b:000000c00092e4f8 EFLAGS: 000002= 06 ORIG_RAX: 0000000000000000 Jul 23 16:01:16 k8-master kernel: RAX: ffffffffffffffda RBX: 00000000000000= 09 RCX: 000000000040708e Jul 23 16:01:16 k8-master kernel: RDX: 0000000000008000 RSI: 000000c000e980= 00 RDI: 0000000000000009 Jul 23 16:01:16 k8-master kernel: RBP: 000000c00092e538 R08: 00000000000000= 00 R09: 0000000000000000 Jul 23 16:01:16 k8-master kernel: R10: 0000000000000000 R11: 00000000000002= 06 R12: 000000c00092e668 Jul 23 16:01:16 k8-master kernel: R13: 0000000000000000 R14: 000000c000ca25= 40 R15: 0000007fffffffff Jul 23 16:01:16 k8-master kernel: Jul 23 16:01:44 k8-master kernel: watchdog: BUG: soft lockup - CPU#7 stuck = for 134s! [rclone:102291] Jul 23 16:01:44 k8-master kernel: Modules linked in: xfs nvme_tcp nvme_keyr= ing nvme_fabrics nvme_core nvme_auth xt_statistic nf_conntrack_netlink xt_T= PROXY nf_tproxy_ipv6 nf_tproxy_ipv4 ip_set xt_CT cls_bpf sch_ingress vxlan = ip6_udp_tunnel udp_tunnel veth xt_socket nf_socket_ipv4 nf_socket_ipv6 ip6t= able_filter ip6table_raw ip6table_mangle ip6_tables iptable_filter iptable_= raw iptable_mangle iptable_nat xfrm_user xfrm_algo xt_addrtype xt_nat xt_MA= SQUERADE xt_mark ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_chain_nat nf_nat x= t_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_comment nft_compa= t nf_tables qrtr intel_rapl_msr intel_rapl_common intel_uncore_frequency in= tel_uncore_frequency_common sb_edac x86_pkg_temp_thermal intel_powerclamp c= fg80211 coretemp kvm_intel binfmt_misc kvm irqbypass ipmi_ssif rapl intel_c= state i2c_i801 i2c_smbus ast mei_me lpc_ich mei ioatdma ipmi_si acpi_power_= meter acpi_ipmi ipmi_devintf ipmi_msghandler input_leds joydev acpi_pad mac= _hid br_netfilter dm_multipath bridge stp llc overlay msr efi_pstore nfnetl= ink Jul 23 16:01:44 k8-master kernel: dmi_sysfs ip_tables x_tables autofs4 btr= fs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq a= sync_xor async_tx xor raid6_pq libcrc32c raid1 raid0 mlx5_ib ib_uverbs macs= ec ib_core uas usb_storage hid_generic usbhid hid mlx5_core crct10dif_pclmu= l crc32_pclmul polyval_clmulni mlxfw polyval_generic igb psample ghash_clmu= lni_intel sha256_ssse3 tls ahci sha1_ssse3 xhci_pci i2c_algo_bit libahci xh= ci_pci_renesas pci_hyperv_intf wmi dca aesni_intel crypto_simd cryptd Jul 23 16:01:44 k8-master kernel: CPU: 7 PID: 102291 Comm: rclone Tainted: = G L 6.8.0-38-generic #38-Ubuntu Jul 23 16:01:44 k8-master kernel: Hardware name: Supermicro SYS-2028TR-HTR/= X10DRT-H, BIOS 2.0b 08/12/2016 Jul 23 16:01:44 k8-master kernel: RIP: 0010:xas_load+0x44/0x60 Jul 23 16:01:44 k8-master kernel: Code: 08 48 3d 00 10 00 00 77 11 5b 41 5c= 5d 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc 0f b6 4b 10 4c 8d 60 fe 38 48 fe= 72 e2 4c 89 e6 <48> 89 df e8 d4 fe ff ff 41 80 3c 24 00 75 bc eb ce 66 66 = 2e 0f 1f Jul 23 16:01:44 k8-master kernel: RSP: 0018:ffffba1eee62f878 EFLAGS: 000002= 06 Jul 23 16:01:44 k8-master kernel: RAX: ffff93df1faf5b62 RBX: ffffba1eee62f8= e8 RCX: 0000000000000000 Jul 23 16:01:44 k8-master kernel: RDX: 0000000000000002 RSI: ffff93df1faf5b= 60 RDI: 0000000000000000 Jul 23 16:01:44 k8-master kernel: RBP: ffffba1eee62f888 R08: 00000000000000= 00 R09: 0000000000000000 Jul 23 16:01:44 k8-master kernel: R10: 0000000000000000 R11: 00000000000000= 00 R12: ffff93df1faf5b60 Jul 23 16:01:44 k8-master kernel: R13: 0000000000024047 R14: ffffba1eee62fc= 00 R15: ffffba1eee62fc00 Jul 23 16:01:44 k8-master kernel: FS: 000000c000d00098(0000) GS:ffff93e0ef= a80000(0000) knlGS:0000000000000000 Jul 23 16:01:44 k8-master kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 00000000= 80050033 Jul 23 16:01:44 k8-master kernel: CR2: 000000c000f2b000 CR3: 00000008b4b4e0= 01 CR4: 00000000003706f0 Jul 23 16:01:44 k8-master kernel: DR0: 0000000000000000 DR1: 00000000000000= 00 DR2: 0000000000000000 Jul 23 16:01:44 k8-master kernel: DR3: 0000000000000000 DR6: 00000000fffe0f= f0 DR7: 0000000000000400 Jul 23 16:01:44 k8-master kernel: Call Trace: Jul 23 16:01:44 k8-master kernel: Jul 23 16:01:44 k8-master kernel: ? show_regs+0x6d/0x80 Jul 23 16:01:44 k8-master kernel: ? watchdog_timer_fn+0x206/0x290 Jul 23 16:01:44 k8-master kernel: ? __pfx_watchdog_timer_fn+0x10/0x10 Jul 23 16:01:44 k8-master kernel: ? __hrtimer_run_queues+0x112/0x2a0 Jul 23 16:01:44 k8-master kernel: ? clockevents_program_event+0xc1/0x150 Jul 23 16:01:44 k8-master kernel: ? hrtimer_interrupt+0xf6/0x250 Jul 23 16:01:44 k8-master kernel: ? __sysvec_apic_timer_interrupt+0x51/0x1= 50 Jul 23 16:01:44 k8-master kernel: ? sysvec_apic_timer_interrupt+0x8d/0xd0 Jul 23 16:01:44 k8-master kernel: Jul 23 16:01:44 k8-master kernel: Jul 23 16:01:44 k8-master kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0= x20 Jul 23 16:01:44 k8-master kernel: ? xas_load+0x44/0x60 Jul 23 16:01:44 k8-master kernel: ? xas_load+0x4c/0x60 Jul 23 16:01:44 k8-master kernel: __xas_next+0xa9/0x150 Jul 23 16:01:44 k8-master kernel: filemap_get_read_batch+0x1a3/0x2e0 Jul 23 16:01:44 k8-master kernel: ? __free_one_page+0x67/0x540 Jul 23 16:01:44 k8-master kernel: filemap_get_pages+0xa9/0x3b0 Jul 23 16:01:44 k8-master kernel: filemap_read+0xf7/0x470 Jul 23 16:01:44 k8-master kernel: ? sk_reset_timer+0x18/0x70 Jul 23 16:01:44 k8-master kernel: generic_file_read_iter+0xbb/0x110 Jul 23 16:01:44 k8-master kernel: ? down_read+0x12/0xd0 Jul 23 16:01:44 k8-master kernel: xfs_file_buffered_read+0x57/0xe0 [xfs] Jul 23 16:01:44 k8-master kernel: xfs_file_read_iter+0xb6/0x1c0 [xfs] Jul 23 16:01:44 k8-master kernel: vfs_read+0x258/0x390 Jul 23 16:01:44 k8-master kernel: ksys_read+0x73/0x100 Jul 23 16:01:44 k8-master kernel: __x64_sys_read+0x19/0x30 Jul 23 16:01:44 k8-master kernel: x64_sys_call+0x1ada/0x25c0 Jul 23 16:01:44 k8-master kernel: do_syscall_64+0x7f/0x180 Jul 23 16:01:44 k8-master kernel: ? vfs_write+0x3d6/0x480 Jul 23 16:01:44 k8-master kernel: ? ksys_write+0xe6/0x100 Jul 23 16:01:44 k8-master kernel: ? syscall_exit_to_user_mode+0x89/0x260 Jul 23 16:01:44 k8-master kernel: ? do_syscall_64+0x8c/0x180 Jul 23 16:01:44 k8-master kernel: ? syscall_exit_to_user_mode+0x89/0x260 Jul 23 16:01:44 k8-master kernel: ? do_syscall_64+0x8c/0x180 Jul 23 16:01:44 k8-master kernel: ? syscall_exit_to_user_mode+0x89/0x260 Jul 23 16:01:44 k8-master kernel: ? do_syscall_64+0x8c/0x180 Jul 23 16:01:44 k8-master kernel: ? restore_fpregs_from_fpstate+0x47/0xf0 Jul 23 16:01:44 k8-master kernel: ? switch_fpu_return+0x55/0xf0 Jul 23 16:01:44 k8-master kernel: ? irqentry_exit_to_user_mode+0x7e/0x260 Jul 23 16:01:44 k8-master kernel: ? irqentry_exit+0x43/0x50 Jul 23 16:01:44 k8-master kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80 Jul 23 16:01:44 k8-master kernel: RIP: 0033:0x40708e Jul 23 16:01:44 k8-master kernel: Code: 48 83 ec 38 e8 13 00 00 00 48 83 c4= 38 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce= 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff = ff ff 48 Jul 23 16:01:44 k8-master kernel: RSP: 002b:000000c00092e4f8 EFLAGS: 000002= 06 ORIG_RAX: 0000000000000000 Jul 23 16:01:44 k8-master kernel: RAX: ffffffffffffffda RBX: 00000000000000= 09 RCX: 000000000040708e Jul 23 16:01:44 k8-master kernel: RDX: 0000000000008000 RSI: 000000c000e980= 00 RDI: 0000000000000009 Jul 23 16:01:44 k8-master kernel: RBP: 000000c00092e538 R08: 00000000000000= 00 R09: 0000000000000000 Jul 23 16:01:44 k8-master kernel: R10: 0000000000000000 R11: 00000000000002= 06 R12: 000000c00092e668 Jul 23 16:01:44 k8-master kernel: R13: 0000000000000000 R14: 000000c000ca25= 40 R15: 0000007fffffffff Jul 23 16:01:44 k8-master kernel: Jul 23 16:02:12 k8-master kernel: watchdog: BUG: soft lockup - CPU#7 stuck = for 160s! [rclone:102291] Jul 23 16:02:12 k8-master kernel: Modules linked in: xfs nvme_tcp nvme_keyr= ing nvme_fabrics nvme_core nvme_auth xt_statistic nf_conntrack_netlink xt_T= PROXY nf_tproxy_ipv6 nf_tproxy_ipv4 ip_set xt_CT cls_bpf sch_ingress vxlan = ip6_udp_tunnel udp_tunnel veth xt_socket nf_socket_ipv4 nf_socket_ipv6 ip6t= able_filter ip6table_raw ip6table_mangle ip6_tables iptable_filter iptable_= raw iptable_mangle iptable_nat xfrm_user xfrm_algo xt_addrtype xt_nat xt_MA= SQUERADE xt_mark ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_chain_nat nf_nat x= t_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_comment nft_compa= t nf_tables qrtr intel_rapl_msr intel_rapl_common intel_uncore_frequency in= tel_uncore_frequency_common sb_edac x86_pkg_temp_thermal intel_powerclamp c= fg80211 coretemp kvm_intel binfmt_misc kvm irqbypass ipmi_ssif rapl intel_c= state i2c_i801 i2c_smbus ast mei_me lpc_ich mei ioatdma ipmi_si acpi_power_= meter acpi_ipmi ipmi_devintf ipmi_msghandler input_leds joydev acpi_pad mac= _hid br_netfilter dm_multipath bridge stp llc overlay msr efi_pstore nfnetl= ink Jul 23 16:02:12 k8-master kernel: dmi_sysfs ip_tables x_tables autofs4 btr= fs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq a= sync_xor async_tx xor raid6_pq libcrc32c raid1 raid0 mlx5_ib ib_uverbs macs= ec ib_core uas usb_storage hid_generic usbhid hid mlx5_core crct10dif_pclmu= l crc32_pclmul polyval_clmulni mlxfw polyval_generic igb psample ghash_clmu= lni_intel sha256_ssse3 tls ahci sha1_ssse3 xhci_pci i2c_algo_bit libahci xh= ci_pci_renesas pci_hyperv_intf wmi dca aesni_intel crypto_simd cryptd Jul 23 16:02:12 k8-master kernel: CPU: 7 PID: 102291 Comm: rclone Tainted: = G L 6.8.0-38-generic #38-Ubuntu Jul 23 16:02:12 k8-master kernel: Hardware name: Supermicro SYS-2028TR-HTR/= X10DRT-H, BIOS 2.0b 08/12/2016 Jul 23 16:02:12 k8-master kernel: RIP: 0010:__xas_next+0x0/0x150 Jul 23 16:02:12 k8-master kernel: Code: e9 7c ff ff ff 4c 89 f6 48 c7 c7 60= 73 1e bc e8 16 70 78 ff e9 b2 fe ff ff 90 90 90 90 90 90 90 90 90 90 90 90= 90 90 90 90 90 <55> 48 89 e5 41 56 41 55 41 54 49 89 fc 53 48 83 ec 10 48 = 8b 5f 18 Jul 23 16:02:12 k8-master kernel: RSP: 0018:ffffba1eee62f8d0 EFLAGS: 000002= 46 Jul 23 16:02:12 k8-master kernel: RAX: 0000000000000000 RBX: ffffde475fa910= 00 RCX: 0000000000000000 Jul 23 16:02:12 k8-master kernel: RDX: 0000000000000000 RSI: 00000000000000= 00 RDI: ffffba1eee62f8e8 Jul 23 16:02:12 k8-master kernel: RBP: ffffba1eee62f950 R08: 00000000000000= 00 R09: 0000000000000000 Jul 23 16:02:12 k8-master kernel: R10: 0000000000000000 R11: 00000000000000= 00 R12: ffffba1eee62fa60 Jul 23 16:02:12 k8-master kernel: R13: 0000000000024047 R14: ffffba1eee62fc= 00 R15: ffffba1eee62fc00 Jul 23 16:02:12 k8-master kernel: FS: 000000c000d00098(0000) GS:ffff93e0ef= a80000(0000) knlGS:0000000000000000 Jul 23 16:02:12 k8-master kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 00000000= 80050033 Jul 23 16:02:12 k8-master kernel: CR2: 000000c000f2b000 CR3: 00000008b4b4e0= 01 CR4: 00000000003706f0 Jul 23 16:02:12 k8-master kernel: DR0: 0000000000000000 DR1: 00000000000000= 00 DR2: 0000000000000000 Jul 23 16:02:12 k8-master kernel: DR3: 0000000000000000 DR6: 00000000fffe0f= f0 DR7: 0000000000000400 Jul 23 16:02:12 k8-master kernel: Call Trace: Jul 23 16:02:12 k8-master kernel: Jul 23 16:02:12 k8-master kernel: ? show_regs+0x6d/0x80 Jul 23 16:02:12 k8-master kernel: ? watchdog_timer_fn+0x206/0x290 Jul 23 16:02:12 k8-master kernel: ? __pfx_watchdog_timer_fn+0x10/0x10 Jul 23 16:02:12 k8-master kernel: ? __hrtimer_run_queues+0x112/0x2a0 Jul 23 16:02:12 k8-master kernel: ? clockevents_program_event+0xc1/0x150 Jul 23 16:02:12 k8-master kernel: ? hrtimer_interrupt+0xf6/0x250 Jul 23 16:02:12 k8-master kernel: ? __sysvec_apic_timer_interrupt+0x51/0x1= 50 Jul 23 16:02:12 k8-master kernel: ? sysvec_apic_timer_interrupt+0x8d/0xd0 Jul 23 16:02:12 k8-master kernel: Jul 23 16:02:12 k8-master kernel: Jul 23 16:02:12 k8-master kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0= x20 Jul 23 16:02:12 k8-master kernel: ? __pfx___xas_next+0x10/0x10 Jul 23 16:02:12 k8-master kernel: ? filemap_get_read_batch+0x1a3/0x2e0 Jul 23 16:02:12 k8-master kernel: ? __free_one_page+0x67/0x540 Jul 23 16:02:12 k8-master kernel: filemap_get_pages+0xa9/0x3b0 Jul 23 16:02:12 k8-master kernel: filemap_read+0xf7/0x470 Jul 23 16:02:12 k8-master kernel: ? sk_reset_timer+0x18/0x70 Jul 23 16:02:12 k8-master kernel: generic_file_read_iter+0xbb/0x110 Jul 23 16:02:12 k8-master kernel: ? down_read+0x12/0xd0 Jul 23 16:02:12 k8-master kernel: xfs_file_buffered_read+0x57/0xe0 [xfs] Jul 23 16:02:12 k8-master kernel: xfs_file_read_iter+0xb6/0x1c0 [xfs] Jul 23 16:02:12 k8-master kernel: vfs_read+0x258/0x390 Jul 23 16:02:12 k8-master kernel: ksys_read+0x73/0x100 Jul 23 16:02:12 k8-master kernel: __x64_sys_read+0x19/0x30 Jul 23 16:02:12 k8-master kernel: x64_sys_call+0x1ada/0x25c0 Jul 23 16:02:12 k8-master kernel: do_syscall_64+0x7f/0x180 Jul 23 16:02:12 k8-master kernel: ? vfs_write+0x3d6/0x480 Jul 23 16:02:12 k8-master kernel: ? ksys_write+0xe6/0x100 Jul 23 16:02:12 k8-master kernel: ? syscall_exit_to_user_mode+0x89/0x260 Jul 23 16:02:12 k8-master kernel: ? do_syscall_64+0x8c/0x180 Jul 23 16:02:12 k8-master kernel: ? syscall_exit_to_user_mode+0x89/0x260 Jul 23 16:02:12 k8-master kernel: ? do_syscall_64+0x8c/0x180 Jul 23 16:02:12 k8-master kernel: ? syscall_exit_to_user_mode+0x89/0x260 Jul 23 16:02:12 k8-master kernel: ? do_syscall_64+0x8c/0x180 Jul 23 16:02:12 k8-master kernel: ? restore_fpregs_from_fpstate+0x47/0xf0 Jul 23 16:02:12 k8-master kernel: ? switch_fpu_return+0x55/0xf0 Jul 23 16:02:12 k8-master kernel: ? irqentry_exit_to_user_mode+0x7e/0x260 Jul 23 16:02:12 k8-master kernel: ? irqentry_exit+0x43/0x50 Jul 23 16:02:12 k8-master kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80 Jul 23 16:02:12 k8-master kernel: RIP: 0033:0x40708e Jul 23 16:02:12 k8-master kernel: Code: 48 83 ec 38 e8 13 00 00 00 48 83 c4= 38 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce= 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff = ff ff 48 Jul 23 16:02:12 k8-master kernel: RSP: 002b:000000c00092e4f8 EFLAGS: 000002= 06 ORIG_RAX: 0000000000000000 Jul 23 16:02:12 k8-master kernel: RAX: ffffffffffffffda RBX: 00000000000000= 09 RCX: 000000000040708e Jul 23 16:02:12 k8-master kernel: RDX: 0000000000008000 RSI: 000000c000e980= 00 RDI: 0000000000000009 Jul 23 16:02:12 k8-master kernel: RBP: 000000c00092e538 R08: 00000000000000= 00 R09: 0000000000000000 Jul 23 16:02:12 k8-master kernel: R10: 0000000000000000 R11: 00000000000002= 06 R12: 000000c00092e668 Jul 23 16:02:12 k8-master kernel: R13: 0000000000000000 R14: 000000c000ca25= 40 R15: 0000007fffffffff Jul 23 16:02:12 k8-master kernel: