From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9ED4E8FDBF for ; Tue, 3 Oct 2023 22:58:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C2748D0099; Tue, 3 Oct 2023 18:58:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 772558D0003; Tue, 3 Oct 2023 18:58:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63A838D0099; Tue, 3 Oct 2023 18:58:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 555848D0003 for ; Tue, 3 Oct 2023 18:58:11 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0F8951604B0 for ; Tue, 3 Oct 2023 22:58:11 +0000 (UTC) X-FDA: 81305665182.17.6E496B1 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf28.hostedemail.com (Postfix) with ESMTP id 23413C0018 for ; Tue, 3 Oct 2023 22:58:08 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=fromorbit-com.20230601.gappssmtp.com header.s=20230601 header.b="x9kKa/DY"; dmarc=pass (policy=quarantine) header.from=fromorbit.com; spf=pass (imf28.hostedemail.com: domain of david@fromorbit.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=david@fromorbit.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696373889; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3JvWESe0lSEDvpuZ7eC9fLLBPWTe4VjlZm52MFoyxHw=; b=Fy1rTAJTywQIYUYZxvByCkbhWN+wQUtx4hBaVRjEwmr+yUdpcuk4pBSG5WT0/fJODdm6Fj 9qYBNp0ZnhNCrJwpI6+GG3rpLev3yPMvB/CjLZOejnWD7TXXPvvgzW0I8ESUxaBhZs6FlZ 9gsSIJnoEJbWPFk9prB8vjNDJPb/1Hc= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=fromorbit-com.20230601.gappssmtp.com header.s=20230601 header.b="x9kKa/DY"; dmarc=pass (policy=quarantine) header.from=fromorbit.com; spf=pass (imf28.hostedemail.com: domain of david@fromorbit.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=david@fromorbit.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696373889; a=rsa-sha256; cv=none; b=HDSGECzZdxS35caLmgf5bSrgOtHXanjNB+oDfRbAE9JBRMq0q20s2ZGWx3D37HbhmghUkx 0HGuhaE/sEJO7zymwWW0Hsv9u9DaEG3uW4jI2E2jCVoL7odK/0tTAJ0XQdx9wWhKh5IDEW +7i3QeAvryOPFuPxi0KI/WQTkSLy16c= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-1c61bde0b4bso12668525ad.3 for ; Tue, 03 Oct 2023 15:58:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20230601.gappssmtp.com; s=20230601; t=1696373888; x=1696978688; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=3JvWESe0lSEDvpuZ7eC9fLLBPWTe4VjlZm52MFoyxHw=; b=x9kKa/DYxsqd8gJhjtHdH5NDMm0xa+rknEqH5GkeCIpMNKDscETAKx8ozs57gE9+0b 3bfT2yqBitaAQB3Ht4Ecb81X570h+PpZS0a/pK8xISes6gKkoYTOByMWi3WcVo8251dx gbaiAs3XFFAqCuypxlRzcOrHxxChh8xOjwxCnMVLPl1m2g2RmMrlPKHdMoOLhGPVqpLW znChixHGk2NgocaYS/oxyjER+NMJ8uYemSUteEJtA50a5YKZqsfE+jusu/ZZvPy88lJ5 fgqf3YHw4EHLdvNUyJqDDS2yWjhvKvIhMKhlUt+zrtcMsWGM5VQWu0EDP+5dYBlwAIdj Cpug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696373888; x=1696978688; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=3JvWESe0lSEDvpuZ7eC9fLLBPWTe4VjlZm52MFoyxHw=; b=VF+u3BVwvK3VsuW4JIoGRFZtmLjc097zhr/D6e5SxWF50+tVP694ZZlr9PW6hT8W1F euU5s85PVrOzrUVRi1pQe1RthvdKngFVANwzL//TWORiyrDde9psrJljT1TeeslUyBb0 y93RJMZtXoTeGH2fqPqx3gzZvmyWgAUcROrNQdp8dhYmw8MkfuAaX0CxMGrBSm4YhRwF Dn7spNaaNWtK17aNqDqpqrdMEpdi8afdvI0EERGi4IvL2uBFlZ/6Y8jSjA7lCfl8Zz1O 6ustKFI47Vsuta4slAoTg/lpaJzycTGXGXvxURLg4G3/Py8PcVOE3JTOQvKn27g/7agZ 0U+w== X-Gm-Message-State: AOJu0YzaEQUUqZWaqbtN+uvrQROUmLCcJMtSp47652EolDCibk0DK72Z ZpVs44F1CDfY4va2Yi7jS0nxt2bFSWh5UYX7TMI= X-Google-Smtp-Source: AGHT+IFW2cwsAODKjPVPHFFeQctUGwjCr1o5jxYyiloHlNhRxWVUe+1LJ99Yu+Woy7PE0LA8njLcqg== X-Received: by 2002:a17:902:ea09:b0:1c5:d747:a124 with SMTP id s9-20020a170902ea0900b001c5d747a124mr1157219plg.9.1696373887906; Tue, 03 Oct 2023 15:58:07 -0700 (PDT) Received: from dread.disaster.area (pa49-180-20-59.pa.nsw.optusnet.com.au. [49.180.20.59]) by smtp.gmail.com with ESMTPSA id jb17-20020a170903259100b001bbdd44bbb6sm2147647plb.136.2023.10.03.15.58.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 15:58:07 -0700 (PDT) Received: from dave by dread.disaster.area with local (Exim 4.96) (envelope-from ) id 1qnoL6-0095u7-2A; Wed, 04 Oct 2023 09:58:04 +1100 Date: Wed, 4 Oct 2023 09:58:04 +1100 From: Dave Chinner To: antal.nemes@hycu.com Cc: Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Daniel Dao Subject: Re: [BUG] soft lockup in filemap_get_read_batch Message-ID: References: <95d6033195a781f81e6ad5bd46026aae@hycu.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <95d6033195a781f81e6ad5bd46026aae@hycu.com> X-Rspamd-Queue-Id: 23413C0018 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ijynjis6u33dd7i5ssnkrdcdirgw3fgt X-HE-Tag: 1696373888-772572 X-HE-Meta: U2FsdGVkX18hN4HS58PyfVzZMKFZMAw+cN38kD8MLEew1RhS3jFy6u1FyIg06rnNolrRrTIR20mebSgzBVxrZZkS6On2P4xfhUUXTeoHNF6VWuhb+3TyP0eSiV/RfdbwNj3UsISmkrK4FXnHsFIm9/lsQQDx7FVIIYoZb7Utf8+3TTaeDHEzRfChZi83AKKyCl8lz8hF7nJAhfs2ZTDcQSrhaFFjItUiwMYj9jw00FpKqTNs/ROnv+XAHvhNeLx+xBrCWVCVuJwy2dTBMPu3ndSpjHFenM6zIE9J8tCMf2gpmHYEgKfXXMu2t61Lhr7Bdbq+09B/1C8H1fnQevw8cM/HE1JIfHkxNymfXWdoLJ/EDCjAU4FkJykKB/59HDDsradk7tckHc4Enudral9VcEgXf8sqED8BGBH9vxL4fFGbxAhTRnw5eY47tOTSUaFqpl38f7g7RwYZq5a979R3XS/cnSqes3+yjTpBXC/iRaqNNHVcJ4YXewB//N9BwqXEqWoHa2K+7C9S+uJbx/+BNwLzxiEfuDxugehDkh/Ajl3kMtF9MX64FTWQ3wY+Uo3/00wquuGgJFiLzkg8ora7rWlEAI7gic2dWGm0SAEuZasL5v7O/Yl/3zoSxOzl+WsXq0QKMPsAOXVKM/jr0tUqp6PDHZqb9upqVYb3RDJxO/QmF5QmCfsCXuq9PceNw+uI8MjkFuguYSzH8hXG3r9LrNq96hSIYqfsjnYqIyAzWMJ4hmW7YeNjOfr35rm3I8cnHckyukL1IF0jET9E9fKXqDOeIQOsy3U2jz85jRyDE8EZMFqbPJEAUamJELu79OdgJfdgrNIbdJqD6HjUsIYX7ypB5axrPsjI7hoU1+VQvy9buqt3Enbj05L45vSgElyrrrhU5KMqj4zuplNzU//jHa5IKcVlZMHmHSRzH+yOf3ZBh4pAC7R9trC2yac4UQiiVkEbsVeSslWU0wGz95s MsxDaxzZ oS9Vyw9roCf8x4ySCYTKRbHxwG4KdmUK2RcqL6LNq5GwEl/snKp06DqJBO1P1hIL+0NHTctqPS6lLWs9ElGJT+Gsj4mXfjITGrWY9ZyrBDkfgRzIVTGjEQ/OCZRdPyT38QdNipq5qJtJnAx8LbOkUJyr/9VpPlNGUYL36Kd6xsTVKCeK3YUam4jGoz0h/FAfcjdho8rgUQ+vK2SxT/5FZsIJU+fvKjkhUYWVGFgxPL4xKSyh+MGV1zxZPQbTCqDkdyC1mm0QfPOgEMXh5X9nUCd7EDyOqVKd5Rx9IttVeF8G6YMcFGmZlUc5I5FJDlP9TBSM3LbVLpWGRVTOUcKidXiM/bh32KXu5OlfsugqZ6gw7a5S3CBOqLLacW2w4PBLPQPrh1xzlwO7R/pP5zS8sH49dfDZTvqdqNHqtbwmMffyFwHM4qM2NE2pabyQO5aYI7UtDjqWToLcUquvhYhEmeeh7WwGZ1pazoLOeD4ZWXMNtFDm1cMIk+e66YSNzbHrr6SsBZuYAvUkEvUDsd94ckblBbdI/pzLvV0GxHZ50BgfA0Sg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000059, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Oct 03, 2023 at 03:48:14PM +0200, antal.nemes@hycu.com wrote: > Hi Matthew, > > We have observed intermittent soft lockups on at least seven different hosts: > - six hosts ran 6.2.8.fc37-200 > - one host ran 6.0.13.fc37-200 > > The list of affected hosts is growing. > > Stack traces are all similar: > > emerg kern kernel - - watchdog: BUG: soft lockup - CPU#7 stuck for 17117s! [postmaster:2238460] > warning kern kernel - - Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock nbd loop nls_utf8 cifs cifs_arc4 cifs_md4 dns_resolver fscache netfs veth iscsi_tcp libiscsi_tcp libiscsi iscsi_target_mod target_core_mod scsi_transport_iscsi nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink sunrpc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua bochs drm_vram_helper drm_ttm_helper ttm crct10dif_pclmul i2c_piix4 crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 virtio_balloon joydev pcspkr xfs crc32c_intel virtio_net serio_raw ata_generic net_failover failover virtio_scsi pata_acpi qemu_fw_cfg fuse [last unloaded: nbd] > warning kern kernel - - CPU: 7 PID: 2238460 Comm: postmaster Kdump: loaded Tainted: G L 6.2.8-200.fc37.x86_64 #1 > warning kern kernel - - Hardware name: Nutanix AHV, BIOS 1.11.0-2.el7 04/01/2014 > warning kern kernel - - RIP: 0010:xas_descend+0x28/0x70 > warning kern kernel - - Code: 90 90 0f b6 0e 48 8b 57 08 48 d3 ea 83 e2 3f 89 d0 48 83 c0 04 48 8b 44 c6 08 48 89 77 18 48 89 c1 83 e1 03 48 83 f9 02 75 08 <48> 3d fd 00 00 00 76 08 88 57 12 c3 cc cc cc cc 48 c1 e8 02 89 c2 > warning kern kernel - - RSP: 0018:ffffab66c9f4bb98 EFLAGS: 00000246 > warning kern kernel - - RAX: 00000000000000c2 RBX: ffffab66c9f4bbb8 RCX: 0000000000000002 > warning kern kernel - - RDX: 0000000000000032 RSI: ffff89cd6c8cd6d0 RDI: ffffab66c9f4bbb8 > warning kern kernel - - RBP: ffff89cd6c8cd6d0 R08: ffffab66c9f4be20 R09: 0000000000000000 > warning kern kernel - - R10: 0000000000000001 R11: 0000000000000100 R12: 00000000000000b3 > warning kern kernel - - R13: 00000000000000b2 R14: 00000000000000b2 R15: ffffab66c9f4be48 > warning kern kernel - - FS: 00007ff1e8bfb540(0000) GS:ffff89d35fbc0000(0000) knlGS:0000000000000000 > warning kern kernel - - CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > warning kern kernel - - CR2: 00007ff1e8af0768 CR3: 000000016fdde001 CR4: 00000000003706e0 > warning kern kernel - - Call Trace: > warning kern kernel - - > warning kern kernel - - xas_load+0x3d/0x50 > warning kern kernel - - filemap_get_read_batch+0x179/0x270 > warning kern kernel - - filemap_get_pages+0xa9/0x690 > warning kern kernel - - ? asm_sysvec_apic_timer_interrupt+0x16/0x20 > warning kern kernel - - filemap_read+0xd2/0x340 > warning kern kernel - - ? filemap_read+0x32f/0x340 > warning kern kernel - - xfs_file_buffered_read+0x4f/0xd0 [xfs] > warning kern kernel - - xfs_file_read_iter+0x70/0xe0 [xfs] > warning kern kernel - - vfs_read+0x23c/0x310 > warning kern kernel - - ksys_read+0x6b/0xf0 > warning kern kernel - - do_syscall_64+0x5b/0x80 > warning kern kernel - - ? syscall_exit_to_user_mode+0x17/0x40 > warning kern kernel - - ? do_syscall_64+0x67/0x80 > warning kern kernel - - ? do_syscall_64+0x67/0x80 > warning kern kernel - - ? __irq_exit_rcu+0x3d/0x140 > warning kern kernel - - entry_SYSCALL_64_after_hwframe+0x72/0xdc Fixed by commit cbc02854331e ("XArray: Do not return sibling entries from xa_load()"). Should already be backported to the lastest stable kernels. -Dave. -- Dave Chinner david@fromorbit.com