From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2739AC27C4F for ; Thu, 30 May 2024 08:57:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 557176B009B; Thu, 30 May 2024 04:57:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 507E46B009C; Thu, 30 May 2024 04:57:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A8516B009D; Thu, 30 May 2024 04:57:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1BB6B6B009B for ; Thu, 30 May 2024 04:57:36 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 924B1A2612 for ; Thu, 30 May 2024 08:57:35 +0000 (UTC) X-FDA: 82174458870.15.D3DF543 Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) by imf05.hostedemail.com (Postfix) with ESMTP id A8153100016 for ; Thu, 30 May 2024 08:57:33 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SSqrSL73; spf=pass (imf05.hostedemail.com: domain of huangzhaoyang@gmail.com designates 209.85.208.176 as permitted sender) smtp.mailfrom=huangzhaoyang@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717059453; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WuofYPH7NjxsjhW4n9VuZ+Yq9c1H81nyri4D+yzFhgs=; b=4xXt7KcXugMWqMaNTrZUv2J6IyA5IERtllpTZw2yA0/zW0JNbyTKXpE2v2Eq1rWwY3rrFH Eaj9u6QMdI0m9xna28JRx1aaltS+otpL74oAgJQqTCI3++HH7UQ9LoMP1K6kiGFtBBRA7H rn+/cvVjeA8xYjwunypg+jsqTpKIbHY= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SSqrSL73; spf=pass (imf05.hostedemail.com: domain of huangzhaoyang@gmail.com designates 209.85.208.176 as permitted sender) smtp.mailfrom=huangzhaoyang@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717059453; a=rsa-sha256; cv=none; b=ihWnKa4N70cvr0K2TY/p24cNm9457YxP+VGnq94r9IrPxk31d20bySb+dJ0J0WB8HQ2VGP uJJoPiI35PLzkj3va+IJX7X0LBbY9k6MbhImHqUNnbA78ZBju6aHEAWiHerhkezsLc2dCJ VKQ3/o20KAWShqiY9JmlTtjx7DCyzso= Received: by mail-lj1-f176.google.com with SMTP id 38308e7fff4ca-2e73359b8fbso8545601fa.2 for ; Thu, 30 May 2024 01:57:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717059452; x=1717664252; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=WuofYPH7NjxsjhW4n9VuZ+Yq9c1H81nyri4D+yzFhgs=; b=SSqrSL73ihd9SHEld1PISFTMoCmSEZJsZziHLD+7PvspsNLF7QzQ3U6ripHwxHURt1 GlFkpwPOfLpkiclz0YzCoDUH6YSJVGS9/07kLpEt4Q0P4lqf+pFAkxPPdJF6xzqdHLBx izV+7uJ24pm+dbDKfQHtbzzcg70btoQjJ1+4bwlGyUMDrx/xW8WN1t+RmGp2vc3i7Ld5 AzNqx+1Xj4n/sj70rvGY6IbMb/EdoVkpOAla+5vta89gEoyCpegBSWe+0v/Dm3iMCedX qJCH52kDHhjFIdNmKScXDKb8uSd8CUAX6SMTyUaRhUOH++rG4MxMdDepq1YAG82AOcrI PnGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717059452; x=1717664252; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WuofYPH7NjxsjhW4n9VuZ+Yq9c1H81nyri4D+yzFhgs=; b=WaA5CcXTn34LxhGQRo/MXvZsV2MKxLkGse9MOzFwV+weSQxb5T4N7/BRBDKOAXA12G BO7LdbkNHQWzvy1knhmS2UnyR8VQbM66eztftuXFylau0GRUkguTOj2f/PlV5iuaCFRO AaT60UZcVlxXdFw03fKAn2wqxYps3Zzwmr0FemLRTT6GMfWRmBDd/xdjOq1VaMpz9xaF 22JMj+hKDNwGdYtK7TQ3lgDvXXetfa8LbVrwDgwFmSqYilsCX+54BEyevuHPqnX9vQwR sUpo44Z1qbHvSpSI4vJtTQIZfHPr0zKIFAzh1fxo9ujvE3pXOrJ+dcIVmWIZOhGqakMh COkA== X-Forwarded-Encrypted: i=1; AJvYcCXQZz8ZkwmVAI6xZNNxRsXSMz4af6awb5fEsL41O59faKHLHYrdaTDcdD2BEe6waWHbzGJiUfYOPx0KuQrGaQgwaXg= X-Gm-Message-State: AOJu0Yy0JGMXVVLCii3KsuOE1S2NZBevteNWJWJITrK3rSmat7z40/l3 PPpcoQ4oqz7nNTGPPZOngvbgNNlhUZhZT2oouCxfwvZHLsmiXttRaFCHD2C+8WU7Li7uGV5HdVN lojVXfs+i+knExVMl3ldRy6rxiLo= X-Google-Smtp-Source: AGHT+IGzt3DWrAaj2GCff7e1OXqv1OpyGAAkzCAJP3+qEmgVZocGI/Ef1fG45//gD6SvPtMktAPG2mhzYvJRy53xCiQ= X-Received: by 2002:a2e:a547:0:b0:2df:ffc6:1a0c with SMTP id 38308e7fff4ca-2ea847aca9amr16876751fa.2.1717059450795; Thu, 30 May 2024 01:57:30 -0700 (PDT) MIME-Version: 1.0 References: <20240412064353.133497-1-zhaoyang.huang@unisoc.com> <20240412143457.5c6c0ae8f6df0f647d7cf0be@linux-foundation.org> In-Reply-To: From: Zhaoyang Huang Date: Thu, 30 May 2024 16:57:19 +0800 Message-ID: Subject: Re: [PATCH 1/1] mm: protect xa split stuff under lruvec->lru_lock during migration To: Yafang Shao Cc: Marcin Wanat , Dave Chinner , Andrew Morton , "zhaoyang.huang" , Alex Shi , "Kirill A . Shutemov" , Hugh Dickins , Baolin Wang , linux-mm@kvack.org, linux-kernel@vger.kernel.org, steve.kang@unisoc.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: 9b951hxtytes3mmqnht4zbkw44nn8rom X-Rspamd-Queue-Id: A8153100016 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1717059453-718010 X-HE-Meta: U2FsdGVkX1/JyMW6dEbmidFQ2V4UoaBkDV4WbqwxABUqXU+IB7rP5YEzXd6tL/K65VD+N2LOWU5PKWPZ14WQg51cm1VTmWkZ4XRHMdIoZwkCtODd+lvLwfT4Oymkz6nBFXVWF0iaLF3iuP08VORk0VmX+oMj78f+NwdeToR97CezYAweU7FZDXPKRhMO3Rj7keY8qgorm76QrBwUtz1GYDMnog89vT0YJuhpldtvpGEPPsR01xTOCmSrXNexoD1hGv09QuAmqITGcSiKEu+Toft17Z/w9tnyvJQi6VW7GJPLHRvmDy7qiZZg6dA8AFxRjZmm0x4UGYilcSLFQVisovOngkbcJQq+SetRZrp9jy4ueGlb0DmIXxnLGibr8xa8RcDeZ3/HsbqPpT1RO9vjXNieiT8TB05+MYvAU8KPpeMFAaQ+tSLkceHCqD/cDR6sXOXyKMnAe+fInqWTo5ibqtDIDgTsFu5ux58ys7RFOAgsTzAau0QDn5xedC+Zq1pkUUBQEqwDVheMNgajIWDeGFiqE9M8JJQh4v1M19ZMr7WJfwm+kWYGwZjTTDOtm7BqiY9r3sOx/Q9rrIOggXbzdnb1yXJzpVU8LGHxuZnEBIgNuJePr6uywbJWQVKME6uSxFDlhapIN3pR7bIlzJeuwHk0cj93yOW6ZE7nbIgGNAn9iKkZlTxujpC2UrkYpqo5Pih2X5ni60v31EeT9vAX/pzTc1P42lZTfHshct+12EbXKojmxw14gNVr1oVPlRFXz9qE7ed5r3M0a6ogIO3jlzny1qBt1J0cVTRBt+KHmDoCd5TCvmjpC9lKzarSsVH83+1F5gQpnL3kcAl7kXVZOH4DHMTn/R0tRLhHJWhU3ruHILeasaWEBs9GaY4MLAAFLlfA8fKkb34Jt7pm9TY/r2kdluNKuaOqFfF1tnurLdEOiSdAMy3MywrtMRg6XIVAzqj3GGFcOrqrtcbzrRN 3yKfaMWd jCv6W0V5s2zmIbPh7Uk9wrasFNBZdq4Y3PSxXcdbggUaf5mlffmj49U4URXf26OaZtxP1gOQtHwPCMiEed9UCjXsOxKjWubXka1bdfctTbRt2EZ7g65/IJdNA11TAz9yBr27gdYG65gKd9VkHdF7SxSF/aURRhmE6dadfNpmWYWQsPVKvheA+6AVbGWxq8+bQzZfRcy7MGUEjqMp64dr7cX9a6rImuXduOUu+aMGSUgP6htQvCqCcxcm2T99/KZJ81vDTdoHRRxhu6ELfetFeKT0o9PWCb0XAVDBcsX291yqw2lV2oAALjPFQDOxjo7Rf5ECO6iPfkGG436rk5O2SAcQck4ud4Y7yyOV4aaNakOss+zbwQ7qnxxC2SIxZ0gjqgvcaRnw7MH7zv4cwmcWVrAyJEZe/eIqy5RAvhAJCUwwebhezMgGQCqoxYJTiq/Pz9SR6ToebGDJQzSKey26p3vQsscXIfgxasKqBqUTC9PgydQz0NDOKThNBzznuStHJK3v7ZT3HXxMQ576Zps0hqMn935LW4Fq+nIYWXUeS2n/TZ4YoRbT75iH8olmJZI/glEvJ5AavdLMYDLJzPlTBdgEl8KNY7Pa9KNMZufahEf4Xv9E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, May 30, 2024 at 4:49=E2=80=AFPM Yafang Shao = wrote: > > On Tue, May 21, 2024 at 3:42=E2=80=AFAM Marcin Wanat wrote: > > > > On 15.04.2024 03:50, Zhaoyang Huang wrote: > > > On Mon, Apr 15, 2024 at 8:09=E2=80=AFAM Dave Chinner > wrote: >> >> On Sat, Apr 13, 2024 at 10:01:27AM +0800, Zhaoyang > > Huang wrote: >>> loop Dave, since he has ever helped set up an > > reproducer in >>> https://lore.kernel.org/linux- >>> > > mm/20221101071721.GV2703033@dread.disaster.area/ @Dave Chinner , >>> I > > would like to ask for your kindly help on if you can verify >>> this > > patch on your environment if convenient. Thanks a lot. >> >> I don't > > have the test environment from 18 months ago available any >> more. > > Also, I haven't seen this problem since that specific test >> > > environment tripped over the issue. Hence I don't have any way of >> > > confirming that the problem is fixed, either, because first I'd >> have > > to reproduce it... > Thanks for the information. I noticed that you > > reported another soft > lockup which is related to xas_load since > > NOV.2023. This patch is > supposed to be helpful for this. With regard > > to the version timing, > this commit is actually a revert of > narrow lru locking> > b6769834aac1d467fa1c71277d15688efcbb4d76 which is > > merged before > v5.15. > > For saving your time, a brief description > > below. IMO, b6769834aa > introduce a potential stall between freeze the > > folio's refcnt and > store it back to 2, which have the > > xas_load->folio_try_get_rcu loops > as livelock if it stalls the > > lru_lock's holder. > > b6769834aa split_huge_page_to_list - > > spin_lock(lru_lock) > xas_split(&xas, folio,order) > > folio_refcnt_freeze(folio, 1 + > folio_nr_pages(folio0) + > > spin_lock(lru_lock) xas_store(&xas, > offset++, head+i) > > page_ref_add(head, 2) spin_unlock(lru_lock) > > Sorry in advance if the > > above doesn't make sense, I am just a > developer who is also suffering > > from this bug and trying to fix it > > I am experiencing a similar error on dozens of hosts, with stack traces > > that are all similar: > > > > [627163.727746] watchdog: BUG: soft lockup - CPU#77 stuck for 22s! > > [file_get:953301] > > [627163.727778] Modules linked in: xt_set ip_set_hash_net ip_set xt_CT > > xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat > > nf_tables nfnetlink sr_mod cdrom rfkill vfat fat intel_rapl_msr > > intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common > > isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal > > intel_powerclamp coretemp ipmi_ssif kvm_intel kvm irqbypass mlx5_ib rap= l > > iTCO_wdt intel_cstate intel_pmc_bxt ib_uverbs iTCO_vendor_support > > dell_smbios dcdbas i2c_i801 intel_uncore uas ses mei_me ib_core > > dell_wmi_descriptor wmi_bmof pcspkr enclosure lpc_ich usb_storage > > i2c_smbus acpi_ipmi mei intel_pch_thermal ipmi_si ipmi_devintf > > ipmi_msghandler acpi_power_meter joydev tcp_bbr fuse xfs libcrc32c raid= 1 > > sd_mod sg mlx5_core crct10dif_pclmul crc32_pclmul crc32c_intel > > polyval_clmulni mgag200 polyval_generic drm_kms_helper mlxfw > > drm_shmem_helper ahci nvme mpt3sas tls libahci ghash_clmulni_intel > > nvme_core psample drm igb t10_pi raid_class pci_hyperv_intf dca libata > > scsi_transport_sas i2c_algo_bit wmi > > [627163.727841] CPU: 77 PID: 953301 Comm: file_get Kdump: loaded > > Tainted: G L 6.6.30.el9 #2 > > [627163.727844] Hardware name: Dell Inc. PowerEdge R740xd/08D89F, BIOS > > 2.21.2 02/19/2024 > > [627163.727847] RIP: 0010:xas_descend+0x1b/0x70 > > [627163.727857] Code: 57 10 48 89 07 48 c1 e8 20 48 89 57 08 c3 cc 0f b= 6 > > 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 > > 08 <48> 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 74 0a 88 4f 12 c3 48 83 > > [627163.727859] RSP: 0018:ffffc90034a67978 EFLAGS: 00000206 > > [627163.727861] RAX: ffff888e4f971242 RBX: ffffc90034a67a98 RCX: > > 0000000000000020 > > [627163.727863] RDX: 0000000000000002 RSI: ffff88a454546d80 RDI: > > ffffc90034a67990 > > [627163.727865] RBP: fffffffffffffffe R08: fffffffffffffffe R09: > > 0000000000008820 > > [627163.727867] R10: 0000000000008820 R11: 0000000000000000 R12: > > ffffc90034a67a20 > > [627163.727868] R13: ffffc90034a67a18 R14: ffffea00873e8000 R15: > > ffffc90034a67a18 > > [627163.727870] FS: 00007fc5e503b740(0000) GS:ffff88bfefd80000(0000) > > knlGS:0000000000000000 > > [627163.727871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [627163.727873] CR2: 000000005fb87b6e CR3: 00000022875e8006 CR4: > > 00000000007706e0 > > [627163.727875] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [627163.727876] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [627163.727878] PKRU: 55555554 > > [627163.727879] Call Trace: > > [627163.727882] > > [627163.727886] ? watchdog_timer_fn+0x22a/0x2a0 > > [627163.727892] ? softlockup_fn+0x70/0x70 > > [627163.727895] ? __hrtimer_run_queues+0x10f/0x2a0 > > [627163.727903] ? hrtimer_interrupt+0x106/0x240 > > [627163.727906] ? __sysvec_apic_timer_interrupt+0x68/0x170 > > [627163.727913] ? sysvec_apic_timer_interrupt+0x9d/0xd0 > > [627163.727917] > > [627163.727918] > > [627163.727920] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 > > [627163.727927] ? xas_descend+0x1b/0x70 > > [627163.727930] xas_load+0x2c/0x40 > > [627163.727933] xas_find+0x161/0x1a0 > > [627163.727937] find_get_entries+0x77/0x1d0 > > [627163.727944] truncate_inode_pages_range+0x244/0x3f0 > > [627163.727950] truncate_pagecache+0x44/0x60 > > [627163.727955] xfs_setattr_size+0x168/0x490 [xfs] > > [627163.728074] xfs_vn_setattr+0x78/0x140 [xfs] > > [627163.728153] notify_change+0x34f/0x4f0 > > [627163.728158] ? _raw_spin_lock+0x13/0x30 > > [627163.728165] ? do_truncate+0x80/0xd0 > > [627163.728169] do_truncate+0x80/0xd0 > > [627163.728172] do_open+0x2ce/0x400 > > [627163.728177] path_openat+0x10d/0x280 > > [627163.728181] do_filp_open+0xb2/0x150 > > [627163.728186] ? check_heap_object+0x34/0x190 > > [627163.728189] ? __check_object_size.part.0+0x5a/0x130 > > [627163.728194] do_sys_openat2+0x92/0xc0 > > [627163.728197] __x64_sys_openat+0x53/0x90 > > [627163.728200] do_syscall_64+0x35/0x80 > > [627163.728206] entry_SYSCALL_64_after_hwframe+0x4b/0xb5 > > [627163.728210] RIP: 0033:0x7fc5e493e7fb > > [627163.728213] Code: 25 00 00 41 00 3d 00 00 41 00 74 4b 64 8b 04 25 1= 8 > > 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f > > 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 2b 14 25 > > [627163.728215] RSP: 002b:00007ffdd4e300e0 EFLAGS: 00000246 ORIG_RAX: > > 0000000000000101 > > [627163.728218] RAX: ffffffffffffffda RBX: 00007ffdd4e30180 RCX: > > 00007fc5e493e7fb > > [627163.728220] RDX: 0000000000000241 RSI: 00007ffdd4e30180 RDI: > > 00000000ffffff9c > > [627163.728221] RBP: 00007ffdd4e30180 R08: 00007fc5e4600040 R09: > > 0000000000000001 > > [627163.728223] R10: 00000000000001b6 R11: 0000000000000246 R12: > > 0000000000000241 > > [627163.728224] R13: 0000000000000000 R14: 00007fc5e4662fa8 R15: > > 0000000000000000 > > [627163.728227] > > > > I have around 50 hosts handling high I/O (each with 20Gbps+ uplinks > > and multiple NVMe drives), running RockyLinux 8/9. The stock RHEL > > kernel 8/9 is NOT affected, and the long-term kernel 5.15.X is NOT affe= cted. > > However, with long-term kernels 6.1.XX and 6.6.XX, > > (tested at least 10 different versions), this lockup always appears > > after 2-30 days, similar to the report in the original thread. > > The more load (for example, copying a lot of local files while > > serving 20Gbps traffic), the higher the chance that the bug will appear= . > > > > I haven't been able to reproduce this during synthetic tests, > > but it always occurs in production on 6.1.X and 6.6.X within 2-30 days. > > We encountered a similar issue several months ago. Some of our > production servers crashed within days after deploying the 6.1.y > stable kernel. The soft lock info as follows, > > [282879.612238] watchdog: BUG: soft lockup - CPU#65 stuck for 101s! > [container-execu:1572375] > [282879.612513] Modules linked in: ebtable_filter ebtables xt_DSCP > iptable_mangle iptable_raw xt_CT cls_bpf sch_ingress raw_diag > unix_diag tcp_diag udp_diag inet_diag iptable_filter bpfilter > xt_conntrack nf_nat nf_conntrack_netlink nfnetlink nf_conntrack > nf_defrag_ipv6 nf_defrag_ipv4 bpf_preload binfmt_misc cuse fuse > overlay af_packet bonding intel_rapl_msr intel_rapl_common > 64_edac kvm_amd kvm irqbypass crct10dif_pclmul crc32_pclmul > polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 > aesni_intel crypto_simd cryptd rapl pcspkr vfat fat xfs mlx5_ib(O) > ib_uverbs(O) input_leds ib_core(O) sg ccp ptdma i2c_piix4 k10temp > acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_cpufreq ip_tables > ext4 mbcache crc32c_intel jbd2 mlx5_core(O) mlxfw(O) pci_hyperv_intf > psample mlxdevm(O) mlx_compat(O) tls nvme ptp pps_core nvme_core > sd_mod t10_pi ahci libahci libata > [282879.612571] CPU: 65 PID: 1572375 Comm: container-execu Kdump: > loaded Tainted: G W O L 6.1.38-rc3 #rc3.pdd > [282879.612574] Hardware name: New H3C Technologies Co., Ltd. H3C > UniServer R4950 G5/RS45M2C9S, BIOS 5.30 06/30/2021 > [282879.612576] RIP: 0010:xas_descend+0x18/0x80 > [282879.612583] Code: b6 e8 ec de 05 00 cc cc cc cc cc cc cc cc cc cc > cc cc 0f b6 0e 48 8b 57 08 48 d3 ea 83 e2 3f 89 d0 48 83 c0 04 48 8b > 44 c6 08 <48> 89 77 18 48 89 c1 83 e1 03 48 83 f9 02 75 08 48 3d fd 00 > 00 00 > [282879.612586] RSP: 0018:ffffad700b247c40 EFLAGS: 00000202 > [282879.612588] RAX: ffff91d247a75d8a RBX: fffffffffffffffe RCX: > 0000000000000006 > [282879.612589] RDX: 0000000000000026 RSI: ffff91d473cb7b30 RDI: > ffffad700b247c68 > [282879.612591] RBP: ffffad700b247c48 R08: 0000000000000003 R09: > fffffffffffffffe > [282879.612592] R10: 0000000000001990 R11: 0000000000000003 R12: > ffffad700b247cf8 > [282879.612593] R13: ffffad700b247d70 R14: ffffad700b247cf8 R15: > ffffdfcd2c778000 > [282879.612594] FS: 00007f5f576fb740(0000) GS:ffff922df0840000(0000) > knlGS:0000000000000000 > [282879.612596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [282879.612597] CR2: 00007fe797100600 CR3: 0000002b2468e000 CR4: > 0000000000350ee0 > [282879.612599] Call Trace: > [282879.612601] > [282879.612605] ? show_regs.cold+0x1a/0x1f > [282879.612610] ? watchdog_timer_fn+0x1c4/0x220 > [282879.612614] ? softlockup_fn+0x30/0x30 > [282879.612616] ? __hrtimer_run_queues+0xa2/0x2b0 > [282879.612620] ? hrtimer_interrupt+0x109/0x220 > [282879.612622] ? __sysvec_apic_timer_interrupt+0x5e/0x110 > [282879.612625] ? sysvec_apic_timer_interrupt+0x7b/0x90 > [282879.612629] > [282879.612630] > [282879.612631] ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 > [282879.612640] ? xas_descend+0x18/0x80 > [282879.612641] ? xas_load+0x35/0x40 > [282879.612643] xas_find+0x197/0x1d0 > [282879.612645] find_get_entries+0x6e/0x170 > [282879.612649] truncate_inode_pages_range+0x294/0x4c0 > [282879.612655] ? __xfs_trans_commit+0x13c/0x3e0 [xfs] > [282879.612787] ? kvfree+0x2c/0x40 > [282879.612791] ? trace_hardirqs_off+0x36/0xf0 > [282879.612795] truncate_inode_pages_final+0x44/0x50 > [282879.612798] evict+0x177/0x190 > [282879.612802] iput.part.0+0x183/0x1e0 > [282879.612804] iput+0x1c/0x30 > [282879.612806] do_unlinkat+0x1c7/0x2c0 > [282879.612810] __x64_sys_unlinkat+0x38/0x70 > [282879.612812] do_syscall_64+0x38/0x90 > [282879.612815] entry_SYSCALL_64_after_hwframe+0x63/0xcd > [282879.612818] RIP: 0033:0x7f5f56cf120d > [282879.612827] Code: 69 5c 2d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e > 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 63 d2 48 63 ff b8 07 01 00 > 00 0f 05 <48> 3d 00 f0 ff ff 77 02 f3 c3 48 8b 15 32 5c 2d 00 f7 d8 64 > 89 02 > [282879.612828] RSP: 002b:00007fff30375c48 EFLAGS: 00000206 ORIG_RAX: > 0000000000000107 > [282879.612830] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: > 00007f5f56cf120d > [282879.612831] RDX: 0000000000000000 RSI: 0000000001640403 RDI: > 0000000000000003 > [282879.612832] RBP: 0000000001640403 R08: 0000000000000000 R09: > 0000000001640403 > [282879.612833] R10: 0000000000000100 R11: 0000000000000206 R12: > 0000000000000003 > [282879.612834] R13: 000000000163c5c0 R14: 00007fff30375c80 R15: > 0000000000000000 > [282879.612836] > > > Unfortunately, we couldn't reproduce the issue on our test servers. We > worked around it by disabling CONFIG_XARRAY_MULTI. Since then, these > production servers have been running smoothly for several months. > > > If anyone can provide a patch, I can test it on multiple machines > > over the next few days. It is highly appreciated that you could help to try below one which works on my v6.6 based android. However, there is a hard lockup reported on an ongoing regression test(not sure if caused by this patch yet). Thank you! mm/huge_memory.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 064fbd90822b..5899906c326a 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2498,7 +2498,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, { struct folio *folio =3D page_folio(page); struct page *head =3D &folio->page; - struct lruvec *lruvec; + struct lruvec *lruvec =3D folio_lruvec(folio); struct address_space *swap_cache =3D NULL; unsigned long offset =3D 0; unsigned int nr =3D thp_nr_pages(head); @@ -2513,9 +2513,6 @@ static void __split_huge_page(struct page *page, struct list_head *list, xa_lock(&swap_cache->i_pages); } - /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ - lruvec =3D folio_lruvec_lock(folio); - ClearPageHasHWPoisoned(head); for (i =3D nr - 1; i >=3D 1; i--) { @@ -2541,9 +2538,6 @@ static void __split_huge_page(struct page *page, struct list_head *list, } ClearPageCompound(head); - unlock_page_lruvec(lruvec); - /* Caller disabled irqs, so they are still disabled here */ - split_page_owner(head, nr); /* See comment in __split_huge_page_tail() */ @@ -2560,7 +2554,6 @@ static void __split_huge_page(struct page *page, struct list_head *list, page_ref_add(head, 2); xa_unlock(&head->mapping->i_pages); } - local_irq_enable(); if (nr_dropped) shmem_uncharge(head->mapping->host, nr_dropped); @@ -2631,6 +2624,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) int extra_pins, ret; pgoff_t end; bool is_hzp; + struct lruvec *lruvec; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); @@ -2714,6 +2708,14 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) /* block interrupt reentry in xa_lock and spinlock */ local_irq_disable(); + + /* + * take lruvec's lock before freeze the folio to prevent the folio + * remains in the page cache with refcnt =3D=3D 0, which could lead to + * find_get_entry enters livelock by iterating the xarray. + */ + lruvec =3D folio_lruvec_lock(folio); + if (mapping) { /* * Check if the folio is present in page cache. @@ -2748,12 +2750,16 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) } __split_huge_page(page, list, end); > > > > > -- > Regards > Yafang