From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C7A3C4707B for ; Thu, 18 Jan 2024 10:40:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6173E6B0075; Thu, 18 Jan 2024 05:40:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C6F16B0078; Thu, 18 Jan 2024 05:40:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4678E6B007D; Thu, 18 Jan 2024 05:40:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 315EC6B0075 for ; Thu, 18 Jan 2024 05:40:35 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 06FA3140BD5 for ; Thu, 18 Jan 2024 10:40:35 +0000 (UTC) X-FDA: 81692088030.14.3499598 Received: from invmail3.skhynix.com (exvmail3.skhynix.com [166.125.252.90]) by imf10.hostedemail.com (Postfix) with ESMTP id 5E37DC0011 for ; Thu, 18 Jan 2024 10:40:32 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of hyeongtak.ji@sk.com designates 166.125.252.90 as permitted sender) smtp.mailfrom=hyeongtak.ji@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705574433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NPUWTfRLPrnJIVJnTk4VGlQpB2CNy1UZOB5/R0ew/3M=; b=IQumsDGgILbGl02OEH8xRN8mSg3WFZg3Xw/4lw8bfkZQ8aonm2cSiaPqPynZPx0zDRx1li y7ZmB8A7Pt4SNOBu1oWTXcWnshGMsWB0iEV3wihyhYyQeaIYOCIvnnhChpjJCsjlkuVreH anh8GENTIn3AceZJ9R1vVuT7ZiExnB0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of hyeongtak.ji@sk.com designates 166.125.252.90 as permitted sender) smtp.mailfrom=hyeongtak.ji@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705574433; a=rsa-sha256; cv=none; b=SxNNlzAcwPA8tYjqoKwZSRvT/SHSzihIuzqtfGvJ+QiYJRrMBk6v74OtvYZXMTZUBqBiUf xL70reo//FtPz4pmcYnNLmz/trywGC5RjrObuwkC1Ofa0OZhY7uE5vnFVMN+099Xf/u8+R GIHpsCbqhPUJBC5SPm+eHz2NZ2XtYEA= X-AuditID: a67dfc59-bd5ff7000001c92b-70-65a9001d9683 From: Hyeongtak Ji To: sj@kernel.org Cc: akpm@linux-foundation.org, apopple@nvidia.com, baolin.wang@linux.alibaba.com, damon@lists.linux.dev, dave.jiang@intel.com, honggyu.kim@sk.com, hyeongtak.ji@sk.com, kernel_team@skhynix.com, linmiaohe@huawei.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, lizhijian@cn.fujitsu.com, mathieu.desnoyers@efficios.com, mhiramat@kernel.org, rakie.kim@sk.com, rostedt@goodmis.org, surenb@google.com, yangx.jy@fujitsu.com, ying.huang@intel.com, ziy@nvidia.com Subject: Re: [RFC PATCH 0/4] DAMON based 2-tier memory management for CXL memory Date: Thu, 18 Jan 2024 19:40:16 +0900 Message-Id: <20240118104017.2098-1-hyeongtak.ji@sk.com> X-Mailer: git-send-email 2.37.3.windows.1 In-Reply-To: <20240117211103.51806-1-sj@kernel.org> References: <20240117211103.51806-1-sj@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrMIsWRmVeSWpSXmKPExsXC9ZZnka4sw8pUgwe/tCzmrF/DZrHrRojF /73HGC2e/P/NanHiZiObRef3pSwWl3fNYbO4t+Y/q8WR9WdZLNbdArI2nz3DbLF4uZrFvo4H TBaHv75hsph8aQGbxYspZxgtTs6azGIx++g9dgchj/8HJzF7LD39hs1jQxOQaNl3i91jwaZS j5Yjb1k9Fu95yeSxaVUnm8emT5PYPU7M+M3isfOhpceLzTMZPXqb37F5fN4kF8AXxWWTkpqT WZZapG+XwJXx53YnW8EnzYpfkw+yNDBOkOti5OSQEDCRuL7kIjuMvfLICiYQm01AQ2Jaw2Jm EFtEQFCi//EM1i5GLg5mgRksEhf3HAcrEhYIkHg85wxYEYuAqkTLgk4WEJtXwFxi5u41LBBD NSUmbJkDVs8pYCwx+cREsGVCAkYS29smskHUC0qcnPkErJ5ZQF6ieetsZpBlEgKH2CVWbVjD BDFIUuLgihssExj5ZyHpmYWkZwEj0ypGkcy8stzEzBxjveLsjMq8zAq95PzcTYzAqFtW+ydy B+O3C8GHGAU4GJV4eDOeLE8VYk0sK67MPcQowcGsJMLrb7AsVYg3JbGyKrUoP76oNCe1+BCj NAeLkjiv0bfyFCGB9MSS1OzU1ILUIpgsEwenVAOjhPXVy5c7udb/d02LtrKX1TU5sHLSLPZT tyRZr1bv39J+aGZTSLi7gu7+8+71h6/orFrqe/Pmzs/eh9f//OG2a7Kgtprb028lO5e+qyt8 UTZxgsqiKWsuL3cvyfu1I5FjW/qsYnu3x0mV/QXsF6dcEunwaL0+w1L93/nA6NapItFaMxQm KpYsV2Ipzkg01GIuKk4EAA04c1+2AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprNIsWRmVeSWpSXmKPExsXCNUPNU1eWYWWqwcW50hZz1q9hs9h1I8Ti /95jjBZP/v9mtThxs5HN4vOz18wWnU++M1ocnnuS1aLz+1IWi8u75rBZ3Fvzn9XiyPqzLBbr bgFZm8+eYbZYvFzN4tC156wW+zoeMFkc/vqGyWLypQVsFi+mnGG0ODlrMovF7KP32B3EPP4f nMTssfT0GzaPDU1AomXfLXaPBZtKPVqOvGX1WLznJZPHplWdbB6bPk1i9zgx4zeLx86Hlh4v Ns9k9Ohtfsfm8e22h8fiFx+YPD5vkgsQiOKySUnNySxLLdK3S+DK+HO7k63gk2bFr8kHWRoY J8h1MXJySAiYSKw8soIJxGYT0JCY1rCYGcQWERCU6H88g7WLkYuDWWAGi8TFPcfBioQFAiQe zzkDVsQioCrRsqCTBcTmFTCXmLl7DQvEUE2JCVvmgNVzChhLTD4xkR3EFhIwktjeNpENol5Q 4uTMJ2D1zALyEs1bZzNPYOSZhSQ1C0lqASPTKkaRzLyy3MTMHDO94uyMyrzMCr3k/NxNjMAo W1b7Z9IOxm+X3Q8xCnAwKvHwdjxckSrEmlhWXJl7iFGCg1lJhNffYFmqEG9KYmVValF+fFFp TmrxIUZpDhYlcV6v8NQEIYH0xJLU7NTUgtQimCwTB6dUA2NFlpnYb70PZ7i5l0jdVFHMfLD2 Tvp120lf3p2bI/1nh2CruuSsXqYYBu1jn0KbOft/Xw7v6n9RsevFU8VWz0/dRywL/rxiK2Iv 3nrIZOPEu0pHP8bvWbTt/j3+61Uz6sVXhWX/zPX9ED5BZKYoU9pKmxdCBZFntH01rTS5f/Bs +ffZO+XUlDNKLMUZiYZazEXFiQBg223CrgIAAA== X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 5E37DC0011 X-Stat-Signature: kdii8d5eqxfby4zwmngwm4rfnhfhg9uj X-HE-Tag: 1705574432-596948 X-HE-Meta: U2FsdGVkX1+fJYVb7Y/HRljb5qMNQKf8zXTSTtYFYG506M1E3SwwDET9vX9s5snkPpYRxBYU2FRDWguik31hp9bCUdApE4zyzDUbtfkfrzGry/ITQ0NT2uZtxYj6/nwyVBP/NILoLnrDSRrFED/AyFOLoHvnthszXh5DH3Z4sUkjSH+x36uJHbOzDIKfi4nQwvi2+Q0SOrtUH9Q6vEvxDrBU10PZM1nEC0qu6Uw0wCvjrk+tYbqahEORWMZSWPkzZFXQNocHxBxmutBxjmkF9gHCDnJKkZm8npxJ+iz4zVdr36XlYKnmzyIaPKqkvrNkMWBx2xRD4ZUbOp8J+XO0Jx5fwpT7V20cR/GMmjV+Bbj1qfmF5liYwO9pWN6R5hYLFRTFO2kL8r21J4porAVPvz0+kjym6GwdgoH0N/of9ubsslk1hG1BZsI7hZ8HATryPcovfMyAk91HGA3KCf0mF4JPgxyaIPrCJ2d90tfOjisMAZsR5Nid11V1jUHJjhedYwm8zKEqfdg7ZZO3e0gMyNr/kuRRGfBKz5p4Uk1u4cNnpoDvBaNLFbPrYK12tjpsE8oJXTWb0Mlb3mvHHyL8m1bq57RRdaOm944HvvXrfjoMAdLDTvqcyM4IbPLcEc54TPDe2oOgkfy6snU1imnYiIdiIXiQbnVUhXfAVvqgkViKRejGwYn6UgVrzLvb2b70A/KU+aBvfODKG8yCRR3AEBF3ArzOSjveqhD+SKvBa+wjKfjyEm2p6YSFinl24hyZI4/Qqd5IV5lfWm3H+3fUddIbgmaxWAs1ORQIbKLBofEHQwTfBwcHNdGGqPjK7fWZhPkwYDr3yPV4SkZ6w7HHNmNdhU8rPZPSQB9NFbU9Y/8b6h7yR29efAmeC8yeHX5ZBfYkPN99//YCfwcTtSByZETDTi18ICvqf2XwDKaZaXIT1alF0BPNa01UhljhDcL0MSfyl+KR1HX0DtXPXzX LpMz2e9O BwdVqApmRPS99oqRimmZ+OGT088NhAvj1KWyr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi SeongJae, On Wed, 17 Jan 2024 SeongJae Park wrote: [...] >> Let's say there are 3 nodes in the system and the first node0 and node1 >> are the first tier, and node2 is the second tier. >> >> $ cat /sys/devices/virtual/memory_tiering/memory_tier4/nodelist >> 0-1 >> >> $ cat /sys/devices/virtual/memory_tiering/memory_tier22/nodelist >> 2 >> >> Here is the result of partitioning hot/cold memory and I put execution >> command at the right side of numastat result. I initially ran each >> hot_cold program with preferred setting so that they initially allocate >> memory on one of node0 or node2, but they gradually migrated based on >> their access frequencies. >> >> $ numastat -c -p hot_cold >> Per-node process memory usage (in MBs) >> PID Node 0 Node 1 Node 2 Total >> --------------- ------ ------ ------ ----- >> 754 (hot_cold) 1800 0 2000 3800 <- hot_cold 1800 2000 >> 1184 (hot_cold) 300 0 500 800 <- hot_cold 300 500 >> 1818 (hot_cold) 801 0 3199 4000 <- hot_cold 800 3200 >> 30289 (hot_cold) 4 0 5 10 <- hot_cold 3 5 >> 30325 (hot_cold) 31 0 51 81 <- hot_cold 30 50 >> --------------- ------ ------ ------ ----- >> Total 2938 0 5756 8695 >> >> The final node placement result shows that DAMON accurately migrated >> pages by their hotness for multiple processes. > > What was the result when the corner cases handling logics were not applied? This is the result of the same test that Honggyu did, but with an insufficient corner cases handling logics. $ numastat -c -p hot_cold Per-node process memory usage (in MBs) PID Node 0 Node 1 Node 2 Total -------------- ------ ------ ------ ----- 862 (hot_cold) 2256 0 1545 3801 <- hot_cold 1800 2000 863 (hot_cold) 403 0 398 801 <- hot_cold 300 500 864 (hot_cold) 1520 0 2482 4001 <- hot_cold 800 3200 865 (hot_cold) 6 0 3 9 <- hot_cold 3 5 866 (hot_cold) 29 0 52 81 <- hot_cold 30 50 -------------- ------ ------ ------ ----- Total 4215 0 4480 8695 As time goes by, DAMON keeps trying to split the hot/cold region, but it does not seem to be enough. $ numastat -c -p hot_cold Per-node process memory usage (in MBs) PID Node 0 Node 1 Node 2 Total -------------- ------ ------ ------ ----- 862 (hot_cold) 2022 0 1780 3801 <- hot_cold 1800 2000 863 (hot_cold) 351 0 450 801 <- hot_cold 300 500 864 (hot_cold) 1134 0 2868 4001 <- hot_cold 800 3200 865 (hot_cold) 7 0 2 9 <- hot_cold 3 5 866 (hot_cold) 43 0 39 81 <- hot_cold 30 50 -------------- ------ ------ ------ ----- Total 3557 0 5138 8695 > > And, what are the corner cases handling logic that seemed essential? I show > the page granularity active/reference check could indeed provide many > improvements, but that's only my humble assumption. Yes, the page granularity active/reference check is essential. To make the above "insufficient" result, the only thing I did was to promote inactive/not_referenced pages. diff --git a/mm/vmscan.c b/mm/vmscan.c index f03be320f9ad..c2aefb883c54 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1127,9 +1127,7 @@ static unsigned int __promote_folio_list(struct list_head *folio_list, VM_BUG_ON_FOLIO(folio_test_active(folio), folio); references = folio_check_references(folio, sc); - if (references == FOLIOREF_KEEP || - references == FOLIOREF_RECLAIM || - references == FOLIOREF_RECLAIM_CLEAN) + if (references == FOLIOREF_KEEP ) goto keep_locked; /* Relocate its contents to another node. */ > > If the corner cases are indeed better to be applied in page granularity, I > agree we need some more efforts since DAMON monitoring results are not page > granularity aware by the design. Users could increase min_nr_regions to make > it more accurate, and we have plan to support page granularity monitoring, > though. But maybe the overhead could be unacceptable. > > Ideal solution would be making DAMON more accurate while keeping current level > of overhead. We indeed have TODO items for DAMON accuracy improvement, but > this may take some time that might unacceptable for your case. > > If that's the case, I think the additional corner handling (or, page gran > additional access check) could be made as DAMOS filters[1], since DAMOS filters > can be applied in page granularity, and designed for this kind of handling of > information that DAMON monitoring results cannot provide. More specifically, > we could have filters for promotion-qualifying pages and demotion-qualifying > pages. In this way, I think we can keep the action more flexible while the > filters can be applied in creative ways. Making corner handling as a new DAMOS filters is a good idea. I'm just a bit concerned if adding new filters might cause users to care more. Kind regards, Hyeongtak