From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13C72C47258 for ; Wed, 17 Jan 2024 21:11:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E19E6B0083; Wed, 17 Jan 2024 16:11:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 791FC6B0088; Wed, 17 Jan 2024 16:11:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 659096B008A; Wed, 17 Jan 2024 16:11:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 562216B0083 for ; Wed, 17 Jan 2024 16:11:09 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 34FC51C1360 for ; Wed, 17 Jan 2024 21:11:09 +0000 (UTC) X-FDA: 81690048258.01.BC809B7 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf18.hostedemail.com (Postfix) with ESMTP id 7F8791C0005 for ; Wed, 17 Jan 2024 21:11:07 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dRarnbm1; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf18.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705525867; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K2PJueXkS7hqBTQtWp0Uk42HmIXKkmqd1JDLBR8oXFQ=; b=PP85vKMouzL0OO2O+gfEzlC24Dbpvp4Y1F/tb3qZrfBCgkXehUYq/Q0jLu8jKtSvwMGnbK 58j7Hbmic254YV9eY/1719Cl87gnr/4YLz9o4tgLdxn8AMjkOrIO08s4ZeTXCiUybqFuWQ XBOJpkcWnUI6Cn0sxmYbppLz1v8267o= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dRarnbm1; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf18.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705525867; a=rsa-sha256; cv=none; b=DWL1bCHQWrayz5OWDakGWLLEcQzEZv00wvpgGVVU385kZUvz/GS+3qtbtIADJY7+Kyn4Ec mlcLb2O/sEazpxBGLJ4cq0oTBatZ2NfXxJV4DhGOsQuwA/2SZ7whV19uYiqbk+59mStEgt W9f7A8740IT2ihGeo1DxjlrUWiIFwHU= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 721A1616EE; Wed, 17 Jan 2024 21:11:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 94835C433F1; Wed, 17 Jan 2024 21:11:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1705525866; bh=crEbNXwtnLQ/zYOPViojanwrSzcw/HATD8C5YXyuKk8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dRarnbm1O7aKz/AC+ejABEJqmaFm4U1DFVVICOPx6onvfO09XzSkSEMhuE719lFY3 NwfAjMOrCSYZgRRR5WTVLX8+tmhM1HWBSphVfiEYdeQ4LsrXFiBeccVVAGKyJFOBK+ ZHWeToSJMrDuurR1cxTxmJ8I1q//wntNxgwU7eZgBEGfcZG7iJel8Ic6L6pwGatHZK DCAIaCAhpgg+eDuY3m7LZqR0LOVrkCmsC2X148/61qcpiaKNtlILHVagVOOpu63/Ck +QYDIxcEl7WZMP7EbMaG00200dukyCwTnCdQSFP77TrX8Pnc7bgkIXClY7A6CRGPRk 8BPPaESZZt/Wg== From: SeongJae Park To: Honggyu Kim Cc: sj@kernel.org, akpm@linux-foundation.org, apopple@nvidia.com, baolin.wang@linux.alibaba.com, damon@lists.linux.dev, dave.jiang@intel.com, hyeongtak.ji@sk.com, kernel_team@skhynix.com, linmiaohe@huawei.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, lizhijian@cn.fujitsu.com, mathieu.desnoyers@efficios.com, mhiramat@kernel.org, rakie.kim@sk.com, rostedt@goodmis.org, surenb@google.com, yangx.jy@fujitsu.com, ying.huang@intel.com, ziy@nvidia.com Subject: Re: [RFC PATCH 0/4] DAMON based 2-tier memory management for CXL memory Date: Wed, 17 Jan 2024 13:11:03 -0800 Message-Id: <20240117211103.51806-1-sj@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240117114926.1895-1-honggyu.kim@sk.com> References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 7F8791C0005 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: cck9omepjaeb6p4wyx7fon4sk3gnii7e X-HE-Tag: 1705525867-916613 X-HE-Meta: U2FsdGVkX1+MYTZQroWpdDXum0CeBJRH17waNy1wD0O4EXzUDMWVOvC5jTf41yAi7AS1wwBGnWXx+MFKfAEPW8xQasC+wZZccVtCJGc7oumEUgPEAbO5YM7KZ+m0RJ9tni6HEFddiuCMlJPxWgqt+Dgd36er4AbQ50YZX439fIo7oIzlfD/NzCADIIOG1fYdLOKaMt4qWSNSlaboYe5ntdnsH99hJ+4ZGi+mbksvVJ7a98XU5F7/X855owbQrdguXK/T7LvBrVDK+ciV5fNfxmIvj8Pw5Y6svntL+F9d66wPoren8X52j0guu/JQxI5NYx2ZQ/iC+4ywpsDm1VeY8IRugvcUfBDO65vQCa5EZgErZhQPud24waB/iFfbttyvD4NeUD8N7yGto7tUlo3nxubuUIoeNoUT/luXMH4iV/GVQoFrn7afAQunCiaLRYQI63dHU2zWJW76ZtYgfpheYIAXB7fyJPIxQMJStsbKfDiZQ0WExmlPP3/LZOGkf87RW9B9YX5JfAWxkks6v0YhjFaqW5i87IuhgeGRpEyEENPPp0/v5x8nxKP7qBifTT46dnUNZgp3eDLKArYLl/9zLzC6weqxnin4qBGGr9lmdtPRwTZPNaoCDKbN+Ijy0XD/TawJkRq9EVPBmTLaQOttMOhb/OPuo+StaGxgKsSI9sKBwKCwLoTazsTs+u05P6BeTRGpiIkeYMFpWyoAQznFrGUWvEa1e4GQDo43JgcjZeF/BUOaQQaQCuo8T5q2tUrsZiziq4grAqz9gK+rPDcEc6Og1xTFR/wzHNOCNTm0UhOj70QczOS+bdoGOmxh+yR60IjVC7EnuDxYtPosaJ6gzxOkWJ5uzELufNUSkEDTkITuqgYVrEWmqR9WqQENVNGUJbI7QOVFnXlbO9HxEcBhBd+kce6QeAKIh84yFqH8vNW/TegbuNcCvbqwUVIw5ts11u7+lmn5bsc9J09veU0 lokSIbEY l7YMjrYfuCynkB8FRFt+oJgnRSZt0LJgciOJnQTh8ll8oWgTCPGWR4Ug8wn9ccDa2rjBal1XzZPkXfl0PNq7zroStvG9yQF4edfgx1ZOmI8w/u8f7n1Wf/hDORmbDbiSil6BhAvn85aASWmaz7lNYScAWTP410G8pSdNCNOfVgPKnG+gjtiNsWsq3pHsWV6d/ckbLnrFf7xcOjNdOnyWXJyNmUN1GXcrDRCcgj62GCTefna+vChGotuU2qz/9P9kVqWVy0CSeTrSoxFJWy9Cx8oS58QUhCMOp8QtgU2VOT/dxD29GvE3e9hKeHcDlti9UVbqcRRTnHV9ZEVK6wLtRueiVFGy5bNqZVN6ZaBVNsfgzS3M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Honggyu, On Wed, 17 Jan 2024 20:49:25 +0900 Honggyu Kim wrote: > Hi SeongJae, > > Thanks very much for your comments in details. > > On Tue, 16 Jan 2024 12:31:59 -0800 SeongJae Park wrote: > > > Thank you so much for this great patches and the above nice test results. I > > believe the test setup and results make sense, and merging a revised version of > > this patchset would provide real benefits to the users. > > Glad to hear that! > > > In a high level, I think it might better to separate DAMON internal changes > > from DAMON external changes. > > I agree. I can't guarantee but I can move all the external changes > inside mm/damon, but will try that as much as possible. > > > For DAMON part changes, I have no big concern other than trivial coding style > > level comments. > > Sure. I will fix those. > > > For DAMON-external changes that implementing demote_pages() and > > promote_pages(), I'm unsure if the implementation is reusing appropriate > > functions, and if those are placee in right source file. Especially, I'm > > unsure if vmscan.c is the right place for promotion code. Also I don't know if > > there is a good agreement on the promotion/demotion target node decision. That > > should be because I'm not that familiar with the areas and the files, but I > > feel this might because our discussions on the promotion and the demotion > > operations are having rooms for being more matured. Because I'm not very > > faimiliar with the part, I'd like to hear others' comments, too. > > I would also like to hear others' comments, but this might not be needed > if most of external code can be moved to mm/damon. > > > To this end, I feel the problem might be able te be simpler, because this > > patchset is trying to provide two sophisticated operations, while I think a > > simpler approach might be possible. My humble simpler idea is adding a DAMOS > > operation for moving pages to a given node (like sys_move_phy_pages RFC[1]), > > instead of the promote/demote. Because the general pages migration can handle > > multiple cases including the promote/demote in my humble assumption. > > My initial implementation was similar but I found that it's not accurate > enough due to the nature of inaccuracy of DAMON regions. I saw that > many pages were demoted and promoted back and forth because migration > target regions include both hot and cold pages together. > > So I have implemented the demotion and promotion logics based on the > shrink_folio_list, which contains many corner case handling logics for > reclaim. > > Having the current demotion and promotion logics makes the hot/cold > migration pretty accurate as expected. We made a simple program called > "hot_cold" and it receives 2 arguments for hot size and cold size in MB. > For example, "hot_cold 200 500" allocates 200MB of hot memory and 500MB > of cold memory. It basically allocates 2 large blocks of memory with > mmap, then repeat memset for the initial 200MB to make it accessed in an > infinite loop. > > Let's say there are 3 nodes in the system and the first node0 and node1 > are the first tier, and node2 is the second tier. > > $ cat /sys/devices/virtual/memory_tiering/memory_tier4/nodelist > 0-1 > > $ cat /sys/devices/virtual/memory_tiering/memory_tier22/nodelist > 2 > > Here is the result of partitioning hot/cold memory and I put execution > command at the right side of numastat result. I initially ran each > hot_cold program with preferred setting so that they initially allocate > memory on one of node0 or node2, but they gradually migrated based on > their access frequencies. > > $ numastat -c -p hot_cold > Per-node process memory usage (in MBs) > PID Node 0 Node 1 Node 2 Total > --------------- ------ ------ ------ ----- > 754 (hot_cold) 1800 0 2000 3800 <- hot_cold 1800 2000 > 1184 (hot_cold) 300 0 500 800 <- hot_cold 300 500 > 1818 (hot_cold) 801 0 3199 4000 <- hot_cold 800 3200 > 30289 (hot_cold) 4 0 5 10 <- hot_cold 3 5 > 30325 (hot_cold) 31 0 51 81 <- hot_cold 30 50 > --------------- ------ ------ ------ ----- > Total 2938 0 5756 8695 > > The final node placement result shows that DAMON accurately migrated > pages by their hotness for multiple processes. What was the result when the corner cases handling logics were not applied? And, what are the corner cases handling logic that seemed essential? I show the page granularity active/reference check could indeed provide many improvements, but that's only my humble assumption. If the corner cases are indeed better to be applied in page granularity, I agree we need some more efforts since DAMON monitoring results are not page granularity aware by the design. Users could increase min_nr_regions to make it more accurate, and we have plan to support page granularity monitoring, though. But maybe the overhead could be unacceptable. Ideal solution would be making DAMON more accurate while keeping current level of overhead. We indeed have TODO items for DAMON accuracy improvement, but this may take some time that might unacceptable for your case. If that's the case, I think the additional corner handling (or, page gran additional access check) could be made as DAMOS filters[1], since DAMOS filters can be applied in page granularity, and designed for this kind of handling of information that DAMON monitoring results cannot provide. More specifically, we could have filters for promotion-qualifying pages and demotion-qualifying pages. In this way, I think we can keep the action more flexible while the filters can be applied in creative ways. [1] https://git.kernel.org/sj/c/98def236f63c66629fb6b2d4b69cecffc5b46539 > > > In more detail, users could decide which is the appropriate node for promotion > > or demotion and use the new DAMOS action to do promotion and demotion. Users > > would requested to decide which node is the proper promotion/demotion target > > nodes, but that decision wouldn't be that hard in my opinion. > > > > For this, 'struct damos' would need to be updated for such argument-dependent > > actions, like 'struct damos_filter' is haing a union. > > That might be a better solution. I will think about it. More specifically, I think receiving an address range as the argument might more flexible than just NUMA node. Maybe we can imagine proactively migrating cold movable pages from normal zones to movable zones, to avoid normal zone memory pressure. > > > In future, we could extend the operation to the promotion and the demotion > > after the dicussion around the promotion and demotion is matured, if required. > > And assuming DAMON be extended for originating CPU-aware access monitoring, the > > new DAMOS action would also cover more use cases such as general NUMA nodes > > balancing (extending DAMON for CPU-aware monitoring would required), and some > > complex configurations where having both CPU affinity and tiered memory. I > > also think that may well fit with my RFC idea[2] for tiered memory management. > > > > Looking forward to opinions from you and others. I admig I miss many things, > > and more than happy to be enlightened. > > > > [1] https://lwn.net/Articles/944007/ > > [2] https://lore.kernel.org/damon/20231112195602.61525-1-sj@kernel.org/ > > Thanks very much for your comments. I will need a few more days for the > update but will try to address your concerns as much as possible. No problem, please take your time. I'm looking forward to the next version :) Thanks, SJ > > Thanks, > Honggyu