From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7975C46CD2 for ; Tue, 30 Jan 2024 08:53:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3EA316B007B; Tue, 30 Jan 2024 03:53:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A6B06B0082; Tue, 30 Jan 2024 03:53:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 261B16B00A4; Tue, 30 Jan 2024 03:53:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 129296B007B for ; Tue, 30 Jan 2024 03:53:22 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C6E341605A0 for ; Tue, 30 Jan 2024 08:53:21 +0000 (UTC) X-FDA: 81735363402.20.9DA8CFE Received: from esa8.hc1455-7.c3s2.iphmx.com (esa8.hc1455-7.c3s2.iphmx.com [139.138.61.253]) by imf13.hostedemail.com (Postfix) with ESMTP id 4DA1420008 for ; Tue, 30 Jan 2024 08:53:19 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=fujitsu.com header.s=fj2 header.b=ZprmEvAU; dmarc=pass (policy=none) header.from=fujitsu.com; spf=pass (imf13.hostedemail.com: domain of lizhijian@fujitsu.com designates 139.138.61.253 as permitted sender) smtp.mailfrom=lizhijian@fujitsu.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706604799; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oyVPq0ZItCsc4fBbgJzvwPU1TaA0blkwLzAsL/VFwx8=; b=2/7a4cTxSm/uTR+swxKe0U9pGPKLOyzMhU3oqDIoOWq2YZCjE3uoSlTpyahhAAdqB5sBRc 0YrandFydEc6+tq2vB4khqUyWGXJjtxPXyb4u4LiAsgjHzwvSIZy77Yu2/BTmj4n8Azm+u kQvRgIda8H7OymhGO7DUcWp6xUU2ibU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=fujitsu.com header.s=fj2 header.b=ZprmEvAU; dmarc=pass (policy=none) header.from=fujitsu.com; spf=pass (imf13.hostedemail.com: domain of lizhijian@fujitsu.com designates 139.138.61.253 as permitted sender) smtp.mailfrom=lizhijian@fujitsu.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706604799; a=rsa-sha256; cv=none; b=gHcDg0Mk+9fYoORWKdA1kTWIa35lNItPem6BfWPEY4+zCTR9OUlYlFDj1M+oqM0LmGN3T8 sIidIT8h+T72b9/NDdtkmtncZ7ZIYHPcB8y2pjFCtcwMMg2q0DS3hQcAlrDdF+3nAKTfNq mQpldZFnTu6awHDCwCzc0TOLDiAnIzI= DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=fujitsu.com; i=@fujitsu.com; q=dns/txt; s=fj2; t=1706604799; x=1738140799; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=PYrUS3/+Y771ILCs3sTwuOn/MKRq3WY1J8faebJpjzI=; b=ZprmEvAUnM9ju8rXFF6MIw08IMYMI2lPV+/d0KNiNRBaeKccs+im2A5X Bl6qHfdc0wf/9bovEiBnQU4Ps2r0r2i6TsS7JazmtOSxHJgUiAdsqfzOz EJKbbdV6IH+YAFkcnHBT2veoaPXmfYKrDWH3TIOF9oA4mI02+sGcsSRMi tQzT7TaU6nM1Vaa4NW0zOrw8+ENwGngo0cm9H56PJggCUGQ1c3g91QeAH 2O8OhfkHxeZmlu/swVXcWxNCTbBGzJj9ZuDHoVvQV4lUdZ4G/Q0pJ2WSd B4gX9QTm1tI1ee/f/DWoZVh5iTJWeZRzx6rZZ979RBIS//DKbbAVbCCBe Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10968"; a="135778839" X-IronPort-AV: E=Sophos;i="6.05,707,1701097200"; d="scan'208";a="135778839" Received: from unknown (HELO oym-r3.gw.nic.fujitsu.com) ([210.162.30.91]) by esa8.hc1455-7.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jan 2024 17:53:16 +0900 Received: from oym-m3.gw.nic.fujitsu.com (oym-nat-oym-m3.gw.nic.fujitsu.com [192.168.87.60]) by oym-r3.gw.nic.fujitsu.com (Postfix) with ESMTP id 061A3384A3 for ; Tue, 30 Jan 2024 17:53:15 +0900 (JST) Received: from kws-ab4.gw.nic.fujitsu.com (kws-ab4.gw.nic.fujitsu.com [192.51.206.22]) by oym-m3.gw.nic.fujitsu.com (Postfix) with ESMTP id 3F53AD7886 for ; Tue, 30 Jan 2024 17:53:14 +0900 (JST) Received: from edo.cn.fujitsu.com (edo.cn.fujitsu.com [10.167.33.5]) by kws-ab4.gw.nic.fujitsu.com (Postfix) with ESMTP id B2B4FE3693 for ; Tue, 30 Jan 2024 17:53:11 +0900 (JST) Received: from [10.167.226.45] (unknown [10.167.226.45]) by edo.cn.fujitsu.com (Postfix) with ESMTP id E16711A006A; Tue, 30 Jan 2024 16:53:10 +0800 (CST) Message-ID: Date: Tue, 30 Jan 2024 16:53:10 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH RFC 1/4] drivers/base/node: Add demotion_nodes sys infterface Content-Language: en-US To: "Huang, Ying" Cc: Andrew Morton , Greg Kroah-Hartman , rafael@kernel.org, linux-mm@kvack.org, y-goto@fujitsu.com, linux-kernel@vger.kernel.org References: <20231102025648.1285477-1-lizhijian@fujitsu.com> <20231102025648.1285477-2-lizhijian@fujitsu.com> <878r7g3ktj.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Li Zhijian In-Reply-To: <878r7g3ktj.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-TM-AS-Product-Ver: IMSS-9.1.0.1417-9.0.0.1002-28150.005 X-TM-AS-User-Approved-Sender: Yes X-TMASE-Version: IMSS-9.1.0.1417-9.0.1002-28150.005 X-TMASE-Result: 10--28.496600-10.000000 X-TMASE-MatchedRID: UomY2wWC/KyPvrMjLFD6eJTQgFTHgkhZsBSA1tuZVSbtP4RQ+dxonQVM 9kPsaYx4vtqCRGwbz0Fkjvv21BvS6hH6a53AZfDxR+GtoiXVeDHYUDvAr2Y/1zpnHdhQ1BMbe0K jC/8CK0pVZIN0tHD4H+AbxsLrrMxv0FQxfjRHOzjN+qWlu2ZxaLFcDzCo2ZtWCxzju735hWTAwJ WfIE4NHs1j70qfy418woo+ldrMeVCwHb9D99XpM6zGfgakLdjawdP+8YXeAclOaDdl7pggvRFEt 97pQ+C8kf935YO20KhO9mtymj1NEjN1VvnSogA6A9lly13c/gEINLW/nC4hC/dt0QHl1BcpA3DT 3VehETp72aKohPmGR81qGMpNtFmr+UviL1yO5g3NgrlT5Ajc7sMKxxeacM3vKxnRjlf7+WsYODB tbaa9ST+Ok6f9gUlFtdh2oL28A44v+0FNnM7lDR+0aBf65h9MWQ3R4k5PTnC4GyTmeN+AbKDpIN QWjbma6wtGsupCFkWaspefDToLzTcpdZ3fQiLdOX/V8P8ail1ZDL1gLmoa/ALDAYP4AXVR7nY51 lwLq0+8QIu4z6HhEH7cGd19dSFd X-TMASE-SNAP-Result: 1.821001.0001-0-1-22:0,33:0,34:0-0 X-Rspamd-Queue-Id: 4DA1420008 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: cfqfpmsopgtcxj7aqczshzc6p7myo9wc X-HE-Tag: 1706604799-796248 X-HE-Meta: U2FsdGVkX1+JO9j3AUFZRYTEIqciUUim9p/AHJcKONEBT54Jgt1b9f7ynK1stHJ002jN8EE9HxuC062z4xJTEC2659Q2klVSPNKzCK9QDKggxUqml9PbJuJ0/1gHGWv6g24w45YVd4i33GiVtvjf059dIsHVjHNztpT0Ec+k208ZGegQZ1T9XxP8RWqmzBwD5VcnjkqKoZ5NoBS0nX/h1n2FQPQMoD3UyYfgLTIHoGJjALMt+wkX269m4QHytIq7yi28d9ya28oO0eQuz01+Wsoe2hg5UijKvmNDy9NrkNkC5BC4+KSybA8herYQTlecbxTP/hmRQ9aWKEEsbDVlkVxztjy2sOvgWFyeJvDUipl5xtu7ZEntS33XvxmIT4UvYCpHV4hTiZicwvEKbkpd/X3fWq9wtnx0anUq3JdtOdvoKQ5gNgQOo/cXW9CwGTSVeAOVm9FFApI4uQWSw6Nh1IToRAm7gYFFPYc8VzqJkwn6bbVxb8BZj3ry21mJBEUqch6Um44jUHWHMUJEiqM8l0A3bcMKG+vsmnWO7d2lxCgSFiKYDfkjLj7af7zQ7MYMP0GO+FbZ4fmmnaOFlzFAeLKwq078S4WakQDx85Mb37NC0jV+AbeUo+L2qQaVqgt26+MG0Kx7gHWq/4hmnivlNyhlw1XIcCn7AT5vF2Ae0/merjrb1VOyhGRLAu7mKkR+Z4CoenGZ8xKD0sejmBCNsPVvkAszBVerEF0ZmL04k2vZBZLqE3979mwC2adoGcEfq5e09peLLxf1rd3AbSEVkh5hxPMaL8xuk2g4KJzqvE6jU5R/2oBK2SfmCt806d6dKtKCpgNOwlU+NwJvhU5AldOSZnoVHI9I5Z4CtJr9ACVqQpHHCkHVtSmRO4nayo2pILrdU4+mBjkNqkU8FvDEj/gqDiW6nZTWgyzS0+WScvtll3y18lKviy/wqkvRh4gXhqwOl67gr6ETYttB3+e uYFx3Nv+ WpkX6dEddS+e2ki3ZpTudcb/cvIlm2aeVE62xCSTMy8/D6LTsSEvQVsnZX9YmY2eyE7FGxEGzkDL+ShwPEw6nocSLNhFgVak95o9BxlonOsVgJ7FpisVMjEOBjx1Rd07tzFlXJC3KGAq6Foj8pVPz+Q+xwhCBhgGqJa0yi3anCMWP9OqVnyBvbSrGfghhRVdSEO8tgks+GUCTCquwQ/a3/twznA0eu8GvJTSBs1jB2ZcSL0Bi3M7srw90/IGHuUiTOL72PcokRIzfNjICscRXBTOEXIsZNWQCbd7qlnsZbjA9DDc12tY/r6TBJ2jXtz8GBn6cWQL3ivvHlRdcBaLPZGcM//vf/g06IzbyzX1xfl2edGY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Ying I need to pick up this thread/patch again. > We have /sys/devices/virtual/memory_tiering/memory_tier*/nodelist > already. A node in a higher tier can demote to any node in the lower > tiers. What's more need to be displayed in nodeX/demotion_nodes? > Yes, it's believed that /sys/devices/virtual/memory_tiering/memory_tierN/nodelist are intended to show nodes in memory_tierN. But IMHO, it's not enough, especially for the preferred demotion node(s). Currently, when a demotion occurs, it will prioritize selecting a node from the preferred nodes as the destination node for the demotion. If the preferred nodes does not meet the requirements, it will try from all the lower memory tier nodes until it finds a suitable demotion destination node or ultimately fails. However, currently it only lists the nodes of each tier. If the administrators want to know all the possible demotion destinations for a given node, they need to calculate it themselves: Step 1, find the memory tier where the given node is located Step 2, list all nodes under all its lower tiers It will be even more difficult to know the preferred nodes which depend on more factors, distance etc. For the following example, we may have 6 nodes splitting into three memory tiers. For emulated hmat numa topology example: > $ numactl -H > available: 6 nodes (0-5) > node 0 cpus: 0 > node 0 size: 1974 MB > node 0 free: 1767 MB > node 1 cpus: 1 > node 1 size: 1694 MB > node 1 free: 1454 MB > node 2 cpus: > node 2 size: 896 MB > node 2 free: 896 MB > node 3 cpus: > node 3 size: 896 MB > node 3 free: 896 MB > node 4 cpus: > node 4 size: 896 MB > node 4 free: 896 MB > node 5 cpus: > node 5 size: 896 MB > node 5 free: 896 MB > node distances: > node 0 1 2 3 4 5 > 0: 10 31 21 41 21 41 > 1: 31 10 41 21 41 21 > 2: 21 41 10 51 21 51 > 3: 31 21 51 10 51 21 > 4: 21 41 21 51 10 51 > 5: 31 21 51 21 51 10 > > $ cat memory_tier4/nodelist > 0-1 > $ cat memory_tier12/nodelist > 2,5 > $ cat memory_tier54/nodelist > 3-4 For above topology, memory-tier will build the demotion path for each node like this: node[0].preferred = 2 node[0].demotion_targets = 2-5 node[1].preferred = 5 node[1].demotion_targets = 2-5 node[2].preferred = 4 node[2].demotion_targets = 3-4 node[3].preferred = node[3].demotion_targets = node[4].preferred = node[4].demotion_targets = node[5].preferred = 3 node[5].demotion_targets = 3-4 But this demotion path is not explicitly known to administrator. And with the feedback from our customers, they also think it is helpful to know demotion path built by kernel to understand the demotion behaviors. So i think we should have 2 new interfaces for each node: /sys/devices/system/node/nodeN/demotion_allowed_nodes /sys/devices/system/node/nodeN/demotion_preferred_nodes I value your opinion, and I'd like to know what you think about... Thanks Zhijian On 02/11/2023 11:17, Huang, Ying wrote: > Li Zhijian writes: > >> It shows the demotion target nodes of a node. Export this information to >> user directly. >> >> Below is an example where node0 node1 are DRAM, node3 is a PMEM node. >> - Before PMEM is online, no demotion_nodes for node0 and node1. >> $ cat /sys/devices/system/node/node0/demotion_nodes >> >> - After node3 is online as kmem >> $ daxctl reconfigure-device --mode=system-ram --no-online dax0.0 && daxctl online-memory dax0.0 >> [ >> { >> "chardev":"dax0.0", >> "size":1054867456, >> "target_node":3, >> "align":2097152, >> "mode":"system-ram", >> "online_memblocks":0, >> "total_memblocks":7 >> } >> ] >> $ cat /sys/devices/system/node/node0/demotion_nodes >> 3 >> $ cat /sys/devices/system/node/node1/demotion_nodes >> 3 >> $ cat /sys/devices/system/node/node3/demotion_nodes >> > > We have /sys/devices/virtual/memory_tiering/memory_tier*/nodelist > already. A node in a higher tier can demote to any node in the lower > tiers. What's more need to be displayed in nodeX/demotion_nodes? > > -- > Best Regards, > Huang, Ying > >> Signed-off-by: Li Zhijian >> --- >> drivers/base/node.c | 13 +++++++++++++ >> include/linux/memory-tiers.h | 6 ++++++ >> mm/memory-tiers.c | 8 ++++++++ >> 3 files changed, 27 insertions(+) >> >> diff --git a/drivers/base/node.c b/drivers/base/node.c >> index 493d533f8375..27e8502548a7 100644 >> --- a/drivers/base/node.c >> +++ b/drivers/base/node.c >> @@ -7,6 +7,7 @@ >> #include >> #include >> #include >> +#include >> #include >> #include >> #include >> @@ -569,11 +570,23 @@ static ssize_t node_read_distance(struct device *dev, >> } >> static DEVICE_ATTR(distance, 0444, node_read_distance, NULL); >> >> +static ssize_t demotion_nodes_show(struct device *dev, >> + struct device_attribute *attr, char *buf) >> +{ >> + int ret; >> + nodemask_t nmask = next_demotion_nodes(dev->id); >> + >> + ret = sysfs_emit(buf, "%*pbl\n", nodemask_pr_args(&nmask)); >> + return ret; >> +} >> +static DEVICE_ATTR_RO(demotion_nodes); >> + >> static struct attribute *node_dev_attrs[] = { >> &dev_attr_meminfo.attr, >> &dev_attr_numastat.attr, >> &dev_attr_distance.attr, >> &dev_attr_vmstat.attr, >> + &dev_attr_demotion_nodes.attr, >> NULL >> }; >> >> diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h >> index 437441cdf78f..8eb04923f965 100644 >> --- a/include/linux/memory-tiers.h >> +++ b/include/linux/memory-tiers.h >> @@ -38,6 +38,7 @@ void init_node_memory_type(int node, struct memory_dev_type *default_type); >> void clear_node_memory_type(int node, struct memory_dev_type *memtype); >> #ifdef CONFIG_MIGRATION >> int next_demotion_node(int node); >> +nodemask_t next_demotion_nodes(int node); >> void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets); >> bool node_is_toptier(int node); >> #else >> @@ -46,6 +47,11 @@ static inline int next_demotion_node(int node) >> return NUMA_NO_NODE; >> } >> >> +static inline next_demotion_nodes next_demotion_nodes(int node) >> +{ >> + return NODE_MASK_NONE; >> +} >> + >> static inline void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets) >> { >> *targets = NODE_MASK_NONE; >> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c >> index 37a4f59d9585..90047f37d98a 100644 >> --- a/mm/memory-tiers.c >> +++ b/mm/memory-tiers.c >> @@ -282,6 +282,14 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets) >> rcu_read_unlock(); >> } >> >> +nodemask_t next_demotion_nodes(int node) >> +{ >> + if (!node_demotion) >> + return NODE_MASK_NONE; >> + >> + return node_demotion[node].preferred; >> +} >> + >> /** >> * next_demotion_node() - Get the next node in the demotion path >> * @node: The starting node to lookup the next node