From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C228C5B552 for ; Mon, 9 Jun 2025 08:49:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE5576B0092; Mon, 9 Jun 2025 04:49:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E966E6B0093; Mon, 9 Jun 2025 04:49:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAC4F6B0095; Mon, 9 Jun 2025 04:49:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BA24D6B0092 for ; Mon, 9 Jun 2025 04:49:37 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 34A071A1563 for ; Mon, 9 Jun 2025 08:49:36 +0000 (UTC) X-FDA: 83535238752.24.34AC07A Received: from out30-101.freemail.mail.aliyun.com (out30-101.freemail.mail.aliyun.com [115.124.30.101]) by imf09.hostedemail.com (Postfix) with ESMTP id 28D30140006 for ; Mon, 9 Jun 2025 08:49:32 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=AiL+YaRO; spf=pass (imf09.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.101 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749458974; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5bJmfGljLGyk00an7Cg65VH+uCqXVJGk/LoQfeiR5C8=; b=ZvXEP4hHnBefUY6DG/lU7ALHG+2V1sF4Fm9bPzdQf07X0VFlTXKWzx7CVJh+LniMInpoQI XsM9E+xwBHD5zx9AlWEq5VoHQ7Hn9m2VxCmEpV6kd8/+O0vZfbOLgFMi3ALtD31Up4jw0M 9HvdUQPZc5WV3MrDJxWJox+lYlDfvOw= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=AiL+YaRO; spf=pass (imf09.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.101 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749458974; a=rsa-sha256; cv=none; b=mILmEObMX6wWdPrlUIVc8DtIxXHv/f/0P1Yw0YELMvHiOf+igHnXLu7k/9V9qu5N3hyJuF 6Qyx7iScZgkWQnTSQ1TiUSCIC0ZE8D3Xlk1h5ICFY9c8SXvFQ+L1pR/oxlopPhSEB2j3+x Jb+7gacL1WENKk+mUoiCVch63JSO498= DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1749458966; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; bh=5bJmfGljLGyk00an7Cg65VH+uCqXVJGk/LoQfeiR5C8=; b=AiL+YaRODxz3NY5V8VbaHcxiXTpIp/W9aGtQsinDJIHQBAflC38d+uoi5dPd+VnJ4SwGfHq0ivsb9wN67TADHi15iRKPh06YuOavAaV52u1qHuM6sEZ4CZoRTXBhPdpoWWKKR9A8U5aLGmHBK9OKo70fFdEtGbUzqmQ+211YHvA= Received: from 30.74.144.144(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0WdO7Itq_1749458964 cluster:ay36) by smtp.aliyun-inc.com; Mon, 09 Jun 2025 16:49:25 +0800 Message-ID: <1452d0c6-50ab-4680-9aa9-13290d51177d@linux.alibaba.com> Date: Mon, 9 Jun 2025 16:49:24 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] mm/shmem, swap: fix softlockup with mTHP swapin To: Kairui Song Cc: linux-mm@kvack.org, Andrew Morton , Hugh Dickins , Kemeng Shi , Chris Li , Nhat Pham , Baoquan He , Barry Song , Usama Arif , linux-kernel@vger.kernel.org References: <20250608192713.95875-1-ryncsn@gmail.com> <36f52466-071a-4efb-adc2-8514b11f120c@linux.alibaba.com> From: Baolin Wang In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Stat-Signature: ar863cff5uwkrk4k6gurdkkon8qzaddd X-Rspamd-Queue-Id: 28D30140006 X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1749458972-119765 X-HE-Meta: U2FsdGVkX18dclz0ht8c3WLHHWuiajhCkhdRLxvkst22dyp3KYfJL9kVdZQblSQ8GooAgwHnnlgSTFk2KufBeKNUnAUwCc0Q/TZS049qDVCvilylZ+XdMdefNXdnd/dDbTfaTxLyEFDSldtaHEhsNVli7TsxnEUrSG6lBiK9/VDxl/aY9RXTNmStcH9pVT8X+TzTP/GOEFmHoTFqWuH90LEHegr3qSbyEoruXSvvKQ11i2Kd6GLtlzyv48TB01EIzAFYG9AGZ936ekwCl/F8Eoyx8x1grz9dSVBLU92Mj3Pdk/aXpnCpVshJPDg2jkgJAj7UYqrLmziEuIUocOsWRjOCWLs3oGNY8uGTgjrLyoJLOO3axB2xsNiWPYjC/y6pkiwedE7zv3/kl0cvrtK6CyA+320A6Q1WxbU2Oaw0XutDZ5b3uHwYmymWNc31VzrC1IFIJ4ZxUF6fMVq6vSz80aMO4z20z2p0D3alsIKF2QuNQppRU4AsYqVlkkYiAZyVHiRGeaKqNWwFvUPt0iGXxDnknSlUeuNicEY/BcHVVZai2rmc38+BhPHA7iNGT26rE7nmTCEvPpKEpjc4qo0cJEXMVHsns/eNQdBo0fpgOLmTIckDeZHlvBufidSRv550NQD2UBNZYEkYeqarfGxMZpj9147yEpWM6s8aejCCgxDNXIUGnLp2VUFsyQJRSVVGgFN3XVl+RM+TnwsI+uRBDJtciaYFL9Uz/syGmL52jwKimkm061wMn7OgiEzy0ATFcj+5AbmlbtjpigzAfom3JyLxoS3ml40b5r2ud7XUb3Rl3MMWntuGC+JDs6QJ8cHmxdfZXNrPlFct4mx0/uBY1aOFLYvE+mjqaT5TJe1YchY/Fe0h7Rvg0AnpsVJYumY37dZrmt2u+1uiNR4RvC/pEdKg7SU5uVdwLMQKjF2FPXathrGW14bUpz33g7SZXNxki6ofO1W6/OC64HXqi7B R9Vzje6j pu5gJ1ZDMeKz18ya1z5qGPmInyr8av7RXj+o/Vu+jIIGPfRWGH9aJaZzjt/YlFV7aLc9JBZLkZwQwqkLBavds4Uy+x9xYrWyuXTz7zwMjXrrnqVZn8QO9wqYvyuWlYBc3ydQwB8r/80hY2GdncpiDP2kjnVS6YLhJC8YCa3M1EXAEm2faI1fI6a7HsuRVbxzwFYVbwuMXinqc0KK9731vozGzjxzTcEiBmW8C2vomaKXdWPryF4Ohg2mP8SleaB469CJAYW3UOfh/8sy468vgmk1upBqUlKaA8WnzD7Vruxjqzc/pF6mzedq7SxL8oA3LTOeQwWLLOTq9uksNvoB0k0mPbSbmiZnjI+c1WlIXzZ5FJw4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2025/6/9 16:36, Kairui Song wrote: > On Mon, Jun 9, 2025 at 4:27 PM Baolin Wang > wrote: >> On 2025/6/9 03:27, Kairui Song wrote: >>> From: Kairui Song >>> >>> Following softlockup can be easily reproduced on my test machine with: >>> >>> echo always > /sys/kernel/mm/transparent_hugepage/hugepages-64kB/enabled >>> swapon /dev/zram0 # zram0 is a 48G swap device >>> mkdir -p /sys/fs/cgroup/memory/test >>> echo 1G > /sys/fs/cgroup/test/memory.max >>> echo $BASHPID > /sys/fs/cgroup/test/cgroup.procs >>> while true; do >>> dd if=/dev/zero of=/tmp/test.img bs=1M count=5120 >>> cat /tmp/test.img > /dev/null >>> rm /tmp/test.img >>> done >>> >>> Then after a while: >>> watchdog: BUG: soft lockup - CPU#0 stuck for 763s! [cat:5787] >>> Modules linked in: zram virtiofs >>> CPU: 0 UID: 0 PID: 5787 Comm: cat Kdump: loaded Tainted: G L 6.15.0.orig-gf3021d9246bc-dirty #118 PREEMPT(voluntary)· >>> Tainted: [L]=SOFTLOCKUP >>> Hardware name: Red Hat KVM/RHEL-AV, BIOS 0.0.0 02/06/2015 >>> RIP: 0010:mpol_shared_policy_lookup+0xd/0x70 >>> Code: e9 b8 b4 ff ff 31 c0 c3 cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 41 54 55 53 <48> 8b 1f 48 85 db 74 41 4c 8d 67 08 48 89 fb 48 89 f5 4c 89 e7 e8 >>> RSP: 0018:ffffc90002b1fc28 EFLAGS: 00000202 >>> RAX: 00000000001c20ca RBX: 0000000000724e1e RCX: 0000000000000001 >>> RDX: ffff888118e214c8 RSI: 0000000000057d42 RDI: ffff888118e21518 >>> RBP: 000000000002bec8 R08: 0000000000000001 R09: 0000000000000000 >>> R10: 0000000000000bf4 R11: 0000000000000000 R12: 0000000000000001 >>> R13: 00000000001c20ca R14: 00000000001c20ca R15: 0000000000000000 >>> FS: 00007f03f995c740(0000) GS:ffff88a07ad9a000(0000) knlGS:0000000000000000 >>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> CR2: 00007f03f98f1000 CR3: 0000000144626004 CR4: 0000000000770eb0 >>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >>> PKRU: 55555554 >>> Call Trace: >>> >>> shmem_alloc_folio+0x31/0xc0 >>> shmem_swapin_folio+0x309/0xcf0 >>> ? filemap_get_entry+0x117/0x1e0 >>> ? xas_load+0xd/0xb0 >>> ? filemap_get_entry+0x101/0x1e0 >>> shmem_get_folio_gfp+0x2ed/0x5b0 >>> shmem_file_read_iter+0x7f/0x2e0 >>> vfs_read+0x252/0x330 >>> ksys_read+0x68/0xf0 >>> do_syscall_64+0x4c/0x1c0 >>> entry_SYSCALL_64_after_hwframe+0x76/0x7e >>> RIP: 0033:0x7f03f9a46991 >>> Code: 00 48 8b 15 81 14 10 00 f7 d8 64 89 02 b8 ff ff ff ff eb bd e8 20 ad 01 00 f3 0f 1e fa 80 3d 35 97 10 00 00 74 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 4f c3 66 0f 1f 44 00 00 55 48 89 e5 48 83 ec >>> RSP: 002b:00007fff3c52bd28 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 >>> RAX: ffffffffffffffda RBX: 0000000000040000 RCX: 00007f03f9a46991 >>> RDX: 0000000000040000 RSI: 00007f03f98ba000 RDI: 0000000000000003 >>> RBP: 00007fff3c52bd50 R08: 0000000000000000 R09: 00007f03f9b9a380 >>> R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000040000 >>> R13: 00007f03f98ba000 R14: 0000000000000003 R15: 0000000000000000 >>> >>> >>> The reason is simple, readahead brought some order 0 folio in swap >>> cache, and the swapin mTHP folio being allocated is in confict with it, >>> so swapcache_prepare fails and causes shmem_swap_alloc_folio to return >>> -EEXIST, and shmem simply retries again and again causing this loop. >> >> If swapcache_prepare() fails and retries, the folio's order (order 0) >> getting from swapcache will be different from the order stored in the >> shmem mapping, so we will split the large swap entry by the following >> logic in shmem_swapin_folio(). So I am not sure why causing a softlockup? >> >> } else if (order != folio_order(folio)) { >> /* >> * Swap readahead may swap in order 0 folios into swapcache >> * asynchronously, while the shmem mapping can still stores >> * large swap entries. In such cases, we should split the >> * large swap entry to prevent possible data corruption. >> */ >> split_order = shmem_split_large_entry(inode, index, swap, gfp); >> if (split_order < 0) { >> error = split_order; >> goto failed; >> } >> >> /* >> * If the large swap entry has already been split, it is >> * necessary to recalculate the new swap entry based on >> * the old order alignment. >> */ >> if (split_order > 0) { >> pgoff_t offset = index - round_down(index, 1 << split_order); >> >> swap = swp_entry(swp_type(swap), swp_offset(swap) + offset); >> } >> } > > For example if the swap entry is 0x0 in shmem with order 4 (so it > corresponds to swap entries 0x0 - 0x10), and a order 0 folio is > currently cached with swap entry 0xa, then shmem swapin will try to > use a folio with order 4, that will always fails swapcache_prepare, > but filemap/swapcache lookup use entry 0x0 will return NULL, causing a > loop. OK. Thanks for the explanation. >>> Fix it by applying a similar fix for anon mTHP swapin. >>> >>> The performance change is very slight, time of swapin 10g zero folios >>> (test for 12 times): >>> Before: 2.49s >>> After: 2.52s >>> >>> Fixes: 1dd44c0af4fa1 ("mm: shmem: skip swapcache for swapin of synchronous swap device") >>> Signed-off-by: Kairui Song >>> >>> --- >>> >>> I found this issue while doing a performance comparing of mm-new with >>> swap table series [1] on top of mm-new. This issue no longer exists >>> if the swap table series is applied, because it elimated both >>> SWAP_HAS_CACHE and SWP_SYNCHRONOUS_IO swapin completely while improving >>> the performance and simplify the code, and the race swapin is solved >>> differently by then. >>> >>> (The zero map fix might still need to stay for a while, but could be >>> optimized too later with swap table). >> >> I don't understand why adding zeromap changes, and should explain this >> explicitly. > > To stay in consistency with anon mTHP swapin, swap_zeromap_batch have > it's own comments that a hybird folio with zero and non-zero pages > can't be brought back as a whole. I can mention that in the commit > message. Yes. Thanks.