From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 109F4E7716D for ; Thu, 5 Dec 2024 15:19:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 302C56B00B4; Thu, 5 Dec 2024 10:19:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D7296B00B9; Thu, 5 Dec 2024 10:19:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 476CF6B00AC; Thu, 5 Dec 2024 10:19:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 93E478D0056 for ; Tue, 10 Sep 2024 22:56:42 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3AAC540F78 for ; Wed, 11 Sep 2024 02:56:42 +0000 (UTC) X-FDA: 82550944644.25.F50D818 Received: from mailgw.kylinos.cn (mailgw.kylinos.cn [124.126.103.232]) by imf10.hostedemail.com (Postfix) with ESMTP id EF5E0C000A for ; Wed, 11 Sep 2024 02:56:38 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of liuye@kylinos.cn designates 124.126.103.232 as permitted sender) smtp.mailfrom=liuye@kylinos.cn ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726023285; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p7hRDTQ/ceRsN5NP2v+r+8L5uVnnhoL/RecmP72Gll0=; b=M74ezZp3Cm7ALbGt39ipTrlZVRwk9bB0gsIA8OzOuhk24sz+ktZfjoyXyG3qyrO9ZS2wTg iSAOLygH5YVIorxGOHSnWsYVHt7CaqqXl/KvacD7gC47f/tNmLFAc83ANfO21tbuNWSFX7 YbUpZ1BzThqeH7pITvHVGyWS2Wz+T4k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726023285; a=rsa-sha256; cv=none; b=FuYVZwI5rnUY+kY3Op2T4twq8S3a1JPaGrhJMy6V89TGim/DajQXQxoxk6isQRR2DH1y9U d1pfYTqG69EmE6CGhSCX1XWpYlX7p70crBTGPY+XURPqe68ADlqz6z9nsUMSwz/PwjIUxp g1WM4qs9hgRfsHEmtYTCfitINSOX8Ek= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of liuye@kylinos.cn designates 124.126.103.232 as permitted sender) smtp.mailfrom=liuye@kylinos.cn X-UUID: 721990d06fe911efa216b1d71e6e1362-20240911 X-CID-P-RULE: Release_Ham X-CID-O-INFO: VERSION:1.1.38,REQID:d91a7388-59d8-449b-bf5f-56cbfeb3efb0,IP:0,U RL:0,TC:0,Content:0,EDM:0,RT:0,SF:0,FILE:0,BULK:0,RULE:Release_Ham,ACTION: release,TS:0 X-CID-META: VersionHash:82c5f88,CLOUDID:13e232e019dc0f319d65303b3dfb5d0e,BulkI D:nil,BulkQuantity:0,Recheck:0,SF:102,TC:nil,Content:0|-5,EDM:-3,IP:nil,UR L:0,File:nil,RT:nil,Bulk:nil,QS:nil,BEC:nil,COL:0,OSI:0,OSA:0,AV:0,LES:1,S PR:NO,DKR:0,DKP:0,BRR:0,BRE:0 X-CID-BVR: 0,NGT X-CID-BAS: 0,NGT,0,_ X-CID-FACTOR: TF_CID_SPAM_SNR X-UUID: 721990d06fe911efa216b1d71e6e1362-20240911 Received: from node2.com.cn [(10.44.16.197)] by mailgw.kylinos.cn (envelope-from ) (Generic MTA) with ESMTP id 1610315782; Wed, 11 Sep 2024 10:56:30 +0800 Received: from node2.com.cn (localhost [127.0.0.1]) by node2.com.cn (NSMail) with SMTP id 41186B8075B2; Wed, 11 Sep 2024 10:56:30 +0800 (CST) X-ns-mid: postfix-66E106DE-149549471 Received: from [172.30.70.72] (unknown [172.30.70.72]) by node2.com.cn (NSMail) with ESMTPA id 319BEB8075B2; Wed, 11 Sep 2024 02:56:29 +0000 (UTC) Subject: Re: [PATCH] mm/vmscan: Fix hard LOCKUP in function isolate_lru_folios From: liuye To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20240815025226.8973-1-liuye@kylinos.cn> <20240823020443.7379-1-liuye@kylinos.cn> Message-ID: Date: Wed, 11 Sep 2024 10:56:28 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EF5E0C000A X-Stat-Signature: q76or1pafgqfbhx5yxjya6t1mpq44obr X-Rspam-User: X-HE-Tag: 1726023398-514446 X-HE-Meta: U2FsdGVkX1/xaTrOBZq0+Cz1h/ZhPLLERuVlsaPcvPQfmn4tT4SQd9ZaFmaibfcOEIQMRPej15OCPuMoRjs8/LYq0ZQ2vQq7F889dnQM++E0rEHdcWfxthDGIQRggfKE5tx5UFcNS8Ensq48NzgWn7terBw1tMAurNaukxHr+heb3tQ+tXgRBbEjXFmNPmNV1brVzZFMtFDcr76xUOA0FfZ++cRs+Y/CPCUJ98p3XZlUE0huzdIVE43L9TLqcub3NvXJvxuktjfjqM8HDRSmsfuofms2MKZKNljoe3rpfqmyewIms1IRdXlZghSN/gwZF7tTgWkMgeDzYN7Bso0BLQhM27rLC/GicA/wBHNNQFJuBYYMDYA8UJNxvtU3HM+uxrNzHkkPAv5QfOj9IEfsPCpYagqHh2V7lv7LZ2N97H2p/S/DjSLLDRWoKKg0KLqNMXBlnN3mAEoS8awuzk7KVegkftW8smnfVFgS2pqpspJzrLfP0Kb1TRHOQvwA1M/STNep5WbG5vb5HGm4IZ8wtwD5w0MP8yZ/sxhFsO/e2zRPZfLODtI84lL35KZ4vRM+Huu1CduLRkYciO5LALsqh76tt+XAx1MHLUlrjZ+PW6s8EJDXnuGZVDkJ9mec20rN27ySL/pg/19rwZEIwJfEKAf4cVeW9k7Qzo5lQybFrzC66pZZ8GueimMR9oWiSjrUswHauiNFO01Gpa6c5fhiFvDQlc4um6zCp95xWK+JsdnpI+2PgJZUQI6jFwuzuOk1DRKPWMObLHwnd0UFXUfW3yXaUInA0c5u7Y2tMo7tsgM4cZQNqOvsHSPafFJaVJPfmMfk02SRjwA2P+m2KBhBsWtgkwarNKbTOCTDE06POgeVU0XXXLoGFwLF3ibaCk1BIYKjJNiZkpldWRjYfeTyipO0TcXMfZyQ5Dr6iigGidpuy60K6Iwt6WQFJ97nNrnl3hFijvLo+xzk6xsr6sq s3icp1YU 1EwYDmj2S3mi66xDrGF4utVxVUVPfPaVCYJ4wfRLpYOZ1A1whjkEWdIV/zwYK4ZT/8WONIuRWZoWG0VHUe6W/NpT0WeYuwxxUOJ0vDVEVs3MbPtC/6qCdztINl4NI8Agntjsy0kVjERkT1Tw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Friendly ping. =20 Thanks. On 2024/9/6 =E4=B8=8A=E5=8D=889:16, liuye wrote: >=20 >=20 > On 2024/8/23 =E4=B8=8A=E5=8D=8810:04, liuye wrote: >> I'm sorry to bother you about that, but it looks like the following em= ail send 7 days ago,=20 >> did not receive a response from you. Do you mind having a look at this= =20 >> when you have a bit of free time please? >> >>>>> Fixes: b2e18757f2c9 ("mm, vmscan: begin reclaiming pages on a per-n= ode basis") >>>> >>>> Merged in 2016. >>>> >>>> Under what circumstances does it occur? =20 >>> >>> User processe are requesting a large amount of memory and keep page a= ctive. >>> Then a module continuously requests memory from ZONE_DMA32 area. >>> Memory reclaim will be triggered due to ZONE_DMA32 watermark alarm re= ached. >>> However pages in the LRU(active_anon) list are mostly from=20 >>> the ZONE_NORMAL area. >>> >>>> Can you please describe how to reproduce this? =20 >>> >>> Terminal 1: Construct to continuously increase pages active(anon).=20 >>> mkdir /tmp/memory >>> mount -t tmpfs -o size=3D1024000M tmpfs /tmp/memory >>> dd if=3D/dev/zero of=3D/tmp/memory/block bs=3D4M >>> tail /tmp/memory/block >>> >>> Terminal 2: >>> vmstat -a 1 >>> active will increase. >>> procs -----------memory---------- ---swap-- -----io---- -system-- ---= ----cpu------- >>> r b swpd free inact active si so bi bo in cs us = sy id wa st gu >>> 1 0 0 1445623076 45898836 83646008 0 0 0 0 1807 = 1682 0 0 100 0 0 0 >>> 1 0 0 1445623076 43450228 86094616 0 0 0 0 1677 = 1468 0 0 100 0 0 0 >>> 1 0 0 1445623076 41003480 88541364 0 0 0 0 1985 = 2022 0 0 100 0 0 0 >>> 1 0 0 1445623076 38557088 90987756 0 0 0 4 1731 = 1544 0 0 100 0 0 0 >>> 1 0 0 1445623076 36109688 93435156 0 0 0 0 1755 = 1501 0 0 100 0 0 0 >>> 1 0 0 1445619552 33663256 95881632 0 0 0 0 2015 = 1678 0 0 100 0 0 0 >>> 1 0 0 1445619804 31217140 98327792 0 0 0 0 2058 = 2212 0 0 100 0 0 0 >>> 1 0 0 1445619804 28769988 100774944 0 0 0 0 1729= 1585 0 0 100 0 0 0 >>> 1 0 0 1445619804 26322348 103222584 0 0 0 0 1774= 1575 0 0 100 0 0 0 >>> 1 0 0 1445619804 23875592 105669340 0 0 0 4 1738= 1604 0 0 100 0 0 0 >>> >>> cat /proc/meminfo | head >>> Active(anon) increase. >>> MemTotal: 1579941036 kB >>> MemFree: 1445618500 kB >>> MemAvailable: 1453013224 kB >>> Buffers: 6516 kB >>> Cached: 128653956 kB >>> SwapCached: 0 kB >>> Active: 118110812 kB >>> Inactive: 11436620 kB >>> Active(anon): 115345744 kB =20 >>> Inactive(anon): 945292 kB >>> >>> When the Active(anon) is 115345744 kB, insmod module triggers the ZON= E_DMA32 watermark. >>> >>> perf show nr_scanned=3D28835844.=20 >>> 28835844 * 4k =3D 115343376KB approximately equal to 115345744 kB. >>> >>> perf record -e vmscan:mm_vmscan_lru_isolate -aR >>> perf script >>> isolate_mode=3D0 classzone=3D1 order=3D1 nr_requested=3D32 nr_scanned= =3D2 nr_skipped=3D2 nr_taken=3D0 lru=3Dactive_anon >>> isolate_mode=3D0 classzone=3D1 order=3D1 nr_requested=3D32 nr_scanned= =3D0 nr_skipped=3D0 nr_taken=3D0 lru=3Dactive_anon >>> isolate_mode=3D0 classzone=3D1 order=3D0 nr_requested=3D32 nr_scanned= =3D28835844 nr_skipped=3D28835844 nr_taken=3D0 lru=3Dactive_anon >>> isolate_mode=3D0 classzone=3D1 order=3D1 nr_requested=3D32 nr_scanned= =3D28835844 nr_skipped=3D28835844 nr_taken=3D0 lru=3Dactive_anon >>> isolate_mode=3D0 classzone=3D1 order=3D0 nr_requested=3D32 nr_scanned= =3D29 nr_skipped=3D29 nr_taken=3D0 lru=3Dactive_anon >>> isolate_mode=3D0 classzone=3D1 order=3D0 nr_requested=3D32 nr_scanned= =3D0 nr_skipped=3D0 nr_taken=3D0 lru=3Dactive_anon >>> >>> If increase Active(anon) to 1000G then insmod module triggers the ZON= E_DMA32 watermark. hard lockup will occur. >>> >>> In my device nr_scanned =3D 0000000003e3e937 when hard lockup. Conver= t to memory size 0x0000000003e3e937 * 4KB =3D 261072092 KB. >>> >>> #5 [ffffc90006fb7c28] isolate_lru_folios at ffffffffa597df53 >>> ffffc90006fb7c30: 0000000000000020 0000000000000000=20 >>> ffffc90006fb7c40: ffffc90006fb7d40 ffff88812cbd3000=20 >>> ffffc90006fb7c50: ffffc90006fb7d30 0000000106fb7de8=20 >>> ffffc90006fb7c60: ffffea04a2197008 ffffea0006ed4a48=20 >>> ffffc90006fb7c70: 0000000000000000 0000000000000000=20 >>> ffffc90006fb7c80: 0000000000000000 0000000000000000=20 >>> ffffc90006fb7c90: 0000000000000000 0000000000000000=20 >>> ffffc90006fb7ca0: 0000000000000000 0000000003e3e937=20 >>> ffffc90006fb7cb0: 0000000000000000 0000000000000000=20 >>> ffffc90006fb7cc0: 8d7c0b56b7874b00 ffff88812cbd3000=20 >>> >>>> Why do you think it took eight years to be discovered? >>> >>> The problem requires the following conditions to occur: >>> 1. The device memory should be large enough. >>> 2. Pages in the LRU(active_anon) list are mostly from the ZONE_NORMAL= area. >>> 3. The memory in ZONE_DMA32 needs to reach the watermark. >>> >>> If the memory is not large enough, or if the usage design of ZONE_DMA= 32 area memory is reasonable, this problem is difficult to detect. >>> >>> notes: >>> The problem is most likely to occur in ZONE_DMA32 and ZONE_NORMAL, bu= t other suitable scenarios may also trigger the problem. >>> >>>> It looks like that will fix, but perhaps something more fundamental >>>> needs to be done - we're doing a tremendous amount of pretty pointle= ss >>>> work here. Answers to my above questions will help us resolve this. >>>> >>>> Thanks. >>> >>> Please refer to the above explanation for details. >>> >>> Thanks. >> >> Thanks. >> > Friendly ping.=20 >=20