From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEC1EC433EF for ; Wed, 8 Jun 2022 07:18:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3B9676B0072; Wed, 8 Jun 2022 03:18:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 342576B0073; Wed, 8 Jun 2022 03:18:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1BBE46B0074; Wed, 8 Jun 2022 03:18:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 096E36B0072 for ; Wed, 8 Jun 2022 03:18:09 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CAB5E350E1 for ; Wed, 8 Jun 2022 07:18:08 +0000 (UTC) X-FDA: 79554214656.14.E81CB7E Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf16.hostedemail.com (Postfix) with ESMTP id 11BDD18004C for ; Wed, 8 Jun 2022 07:18:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1654672688; x=1686208688; h=message-id:subject:from:to:cc:date:in-reply-to: references:mime-version:content-transfer-encoding; bh=wotc8dogZL4LUHpHacn7DgZBf7g5fqOuafBDysmW2Zg=; b=MCz72bRKcXlTsuCW+wKI9ri3EGGCjNEJAVVoy4YmQNNtdWFGiRBUZOOn 60X0CB2+OnmD2WWCiohd9Z6dRMUdii/Y3CKH2of+kXdIx0wUn3aHwUZWe /F1mnjAkTYUBrv3R7fqrfDexnbIi046c4hdEkDy2cbLKY//TCzgofnuOx +GyOomNQfgoPV6zGXU/V8VXZWLQjA3CSBfN5DSlASeh6/sF2QTQhDZes6 QG9l8wLCJrpCRyS4d2uvxPIoHKX4mAJByX/LC2U0AW4hfEbIpQ6ScxFkv 9oms4+paylFR2dmS75KqSjcWNunzryzVcL7FGnZMBFBGHTpwJZC71e5se A==; X-IronPort-AV: E=McAfee;i="6400,9594,10371"; a="276836430" X-IronPort-AV: E=Sophos;i="5.91,285,1647327600"; d="scan'208";a="276836430" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2022 00:18:06 -0700 X-IronPort-AV: E=Sophos;i="5.91,285,1647327600"; d="scan'208";a="907515184" Received: from xding11-mobl.ccr.corp.intel.com ([10.254.214.239]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2022 00:18:02 -0700 Message-ID: Subject: Re: [RFC PATCH v4 2/7] mm/demotion: Expose per node memory tier to sysfs From: Ying Huang To: "Aneesh Kumar K.V" , linux-mm@kvack.org, akpm@linux-foundation.org Cc: Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes Date: Wed, 08 Jun 2022 15:18:00 +0800 In-Reply-To: <20220527122528.129445-3-aneesh.kumar@linux.ibm.com> References: <20220527122528.129445-1-aneesh.kumar@linux.ibm.com> <20220527122528.129445-3-aneesh.kumar@linux.ibm.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 11BDD18004C Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=MCz72bRK; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf16.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 192.55.52.115) smtp.mailfrom=ying.huang@intel.com X-Stat-Signature: hgohm993or4rofrghhziaysbincrotmf X-Rspam-User: X-HE-Tag: 1654672687-36500 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 2022-05-27 at 17:55 +0530, Aneesh Kumar K.V wrote: > From: Jagdish Gediya > > Add support to read/write the memory tierindex for a NUMA node. > > /sys/devices/system/node/nodeN/memtier > > where N = node id > > When read, It list the memory tier that the node belongs to. > > When written, the kernel moves the node into the specified > memory tier, the tier assignment of all other nodes are not > affected. > > If the memory tier does not exist, writing to the above file > create the tier and assign the NUMA node to that tier. > > mutex memory_tier_lock is introduced to protect memory tier > related chanegs as it can happen from sysfs as well on hot > plug events. > > Signed-off-by: Jagdish Gediya > Signed-off-by: Aneesh Kumar K.V > --- >  drivers/base/node.c | 35 ++++++++++++++ >  include/linux/migrate.h | 4 +- >  mm/migrate.c | 103 ++++++++++++++++++++++++++++++++++++++++ >  3 files changed, 141 insertions(+), 1 deletion(-) > > diff --git a/drivers/base/node.c b/drivers/base/node.c > index ec8bb24a5a22..cf4a58446d8c 100644 > --- a/drivers/base/node.c > +++ b/drivers/base/node.c > @@ -20,6 +20,7 @@ >  #include >  #include >  #include > +#include >   > > > >  static struct bus_type node_subsys = { >   .name = "node", > @@ -560,11 +561,45 @@ static ssize_t node_read_distance(struct device *dev, >  } >  static DEVICE_ATTR(distance, 0444, node_read_distance, NULL); >   > > > > +#ifdef CONFIG_TIERED_MEMORY > +static ssize_t memtier_show(struct device *dev, > + struct device_attribute *attr, > + char *buf) > +{ > + int node = dev->id; > + > + return sysfs_emit(buf, "%d\n", node_get_memory_tier(node)); > +} > + > +static ssize_t memtier_store(struct device *dev, > + struct device_attribute *attr, > + const char *buf, size_t count) > +{ > + unsigned long tier; > + int node = dev->id; > + > + int ret = kstrtoul(buf, 10, &tier); > + if (ret) > + return ret; > + > + ret = node_reset_memory_tier(node, tier); > + if (ret) > + return ret; > + > + return count; > +} > + > +static DEVICE_ATTR_RW(memtier); > +#endif > + >  static struct attribute *node_dev_attrs[] = { >   &dev_attr_meminfo.attr, >   &dev_attr_numastat.attr, >   &dev_attr_distance.attr, >   &dev_attr_vmstat.attr, > +#ifdef CONFIG_TIERED_MEMORY > + &dev_attr_memtier.attr, > +#endif >   NULL >  }; >   > > > > diff --git a/include/linux/migrate.h b/include/linux/migrate.h > index 0ec653623565..d37d1d5dee82 100644 > --- a/include/linux/migrate.h > +++ b/include/linux/migrate.h > @@ -177,13 +177,15 @@ enum memory_tier_type { >  }; >   > > > >  int next_demotion_node(int node); > - >  extern void migrate_on_reclaim_init(void); >  #ifdef CONFIG_HOTPLUG_CPU >  extern void set_migration_target_nodes(void); >  #else >  static inline void set_migration_target_nodes(void) {} >  #endif > +int node_get_memory_tier(int node); > +int node_set_memory_tier(int node, int tier); > +int node_reset_memory_tier(int node, int tier); >  #else >  #define numa_demotion_enabled false >  static inline int next_demotion_node(int node) > diff --git a/mm/migrate.c b/mm/migrate.c > index f28ee93fb017..304559ba3372 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -2132,6 +2132,7 @@ static struct bus_type memory_tier_subsys = { >   .dev_name = "memtier", >  }; >   > > > > +DEFINE_MUTEX(memory_tier_lock); >  static struct memory_tier *memory_tiers[MAX_MEMORY_TIERS]; >   > > > >  static ssize_t nodelist_show(struct device *dev, > @@ -2225,6 +2226,108 @@ static const struct attribute_group *memory_tier_attr_groups[] = { >   NULL, >  }; >   > > > > +static int __node_get_memory_tier(int node) > +{ > + int tier; > + > + for (tier = 0; tier < MAX_MEMORY_TIERS; tier++) { > + if (memory_tiers[tier] && node_isset(node, memory_tiers[tier]->nodelist)) > + return tier; > + } > + > + return -1; > +} > + > +int node_get_memory_tier(int node) > +{ > + int tier; > + > + /* > + * Make sure memory tier is not unregistered > + * while it is being read. > + */ > + mutex_lock(&memory_tier_lock); > + > + tier = __node_get_memory_tier(node); > + > + mutex_unlock(&memory_tier_lock); > + > + return tier; > +} > + > +int __node_set_memory_tier(int node, int tier) > +{ > + int ret = 0; > + /* > + * As register_memory_tier() for new tier can fail, > + * try it before modifying existing tier. register > + * tier makes tier visible in sysfs. > + */ > + if (!memory_tiers[tier]) { > + ret = register_memory_tier(tier); > + if (ret) { > + goto out; > + } > + } > + > + node_set(node, memory_tiers[tier]->nodelist); > + > +out: > + return ret; > +} > + > +int node_reset_memory_tier(int node, int tier) I think "reset" isn't a good name here. Maybe something like "change" or "move"? Best Regards, Huang, Ying > +{ > + int current_tier, ret = 0; > + > + mutex_lock(&memory_tier_lock); > + > + current_tier = __node_get_memory_tier(node); > + if (current_tier == tier) > + goto out; > + > + if (current_tier != -1 ) > + node_clear(node, memory_tiers[current_tier]->nodelist); > + > + ret = __node_set_memory_tier(node, tier); > + > + if (!ret) { > + if (nodes_empty(memory_tiers[current_tier]->nodelist)) > + unregister_memory_tier(current_tier); > + } else { > + /* reset it back to older tier */ > + ret = __node_set_memory_tier(node, current_tier); > + } > +out: > + mutex_unlock(&memory_tier_lock); > + > + return ret; > +} > + > +int node_set_memory_tier(int node, int tier) > +{ > + int current_tier, ret = 0; > + > + if (tier >= MAX_MEMORY_TIERS) > + return -EINVAL; > + > + mutex_lock(&memory_tier_lock); > + current_tier = __node_get_memory_tier(node); > + /* > + * if node is already part of the tier proceed with the > + * current tier value, because we might want to establish > + * new migration paths now. The node might be added to a tier > + * before it was made part of N_MEMORY, hence estabilish_migration_targets > + * will have skipped this node. > + */ > + if (current_tier != -1) > + tier = current_tier; > + ret = __node_set_memory_tier(node, tier); > + mutex_unlock(&memory_tier_lock); > + > + return ret; > +} > + >  /* >   * node_demotion[] example: >   *