From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9C58C433EF for ; Wed, 8 Jun 2022 20:24:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65DD76B0071; Wed, 8 Jun 2022 16:24:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 60E4B6B0072; Wed, 8 Jun 2022 16:24:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AF5E6B0073; Wed, 8 Jun 2022 16:24:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3CBC36B0071 for ; Wed, 8 Jun 2022 16:24:48 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id EA2C281344 for ; Wed, 8 Jun 2022 20:24:47 +0000 (UTC) X-FDA: 79556197014.01.DCAD169 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf28.hostedemail.com (Postfix) with ESMTP id 2C457C0004 for ; Wed, 8 Jun 2022 20:24:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1654719885; x=1686255885; h=message-id:subject:from:to:cc:date:in-reply-to: references:mime-version:content-transfer-encoding; bh=dLp0cUT1fUxwoqMLlx+zcnqvmgDGV98fyxDj54aIu5I=; b=g7Mb/WHrEyE2PAgRmgkV+dx1bMBUoORxCnsM/4ZLv4b2RiLgTrcuPfIR HyxXZvpQP4BUPxYZr8kt7usgiAFgBYg9fsdchIJOgaUDYFUc7vn2X76WO os/G1yYuq95R8Cvc+qfMjAh3qk3ZTtv3j3WPAyYWnSfaLccmoZE2M3s96 73tgeOm6zqlhZ/KZFQNii6MGGygzR7FJS9sI4eUi21hQzIcpOvn7yaMoz BJf4oM6+iq5b/6jKMHEfX/mArgoiMioJFS6RAzd2u6EH2QzrUAV/EviBF bYJXktuhMVT1sSnuiAueodHMHjlFH3t3bbxdUt591vfTtiHxjhd3IVaLe A==; X-IronPort-AV: E=McAfee;i="6400,9594,10372"; a="275800438" X-IronPort-AV: E=Sophos;i="5.91,286,1647327600"; d="scan'208";a="275800438" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2022 13:14:56 -0700 X-IronPort-AV: E=Sophos;i="5.91,286,1647327600"; d="scan'208";a="533256642" Received: from schen9-mobl.amr.corp.intel.com ([10.209.124.119]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2022 13:14:55 -0700 Message-ID: <0d8849467053cf48f5d7356de5f1e3e600a85b39.camel@linux.intel.com> Subject: Re: [PATCH v5 9/9] mm/demotion: Update node_is_toptier to work with memory tiers From: Tim Chen To: "Aneesh Kumar K.V" , Ying Huang , linux-mm@kvack.org, akpm@linux-foundation.org Cc: Wei Xu , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes Date: Wed, 08 Jun 2022 13:14:55 -0700 In-Reply-To: <87sfoffcfz.fsf@linux.ibm.com> References: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> <20220603134237.131362-10-aneesh.kumar@linux.ibm.com> <6e94b7e2a6192e4cacba1db3676b5b5cf9b98eac.camel@intel.com> <11f94e0c50f17f4a6a2f974cb69a1ae72853e2be.camel@intel.com> <232817e0-24fd-e022-6c92-c260f7f01f8a@linux.ibm.com> <87sfoffcfz.fsf@linux.ibm.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.34.4 (3.34.4-1.fc31) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1654719885; a=rsa-sha256; cv=none; b=8OXj9qlzGAGwTHrFe401RDozGAQe26MLLtz0ER/Q4u/9tWhCDvS7/PSO0CPSkwRx5knVGv E6uzt2FCz5gWCRgseEJ5aA2a32cpKGq7CPKtRz7twRczLiiTsAmBNZO+34kP2cMcvBItL2 LbQI/V4t8PGeP+2DmzidXfQkOMRB1ro= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="g7Mb/WHr"; spf=none (imf28.hostedemail.com: domain of tim.c.chen@linux.intel.com has no SPF policy when checking 192.55.52.120) smtp.mailfrom=tim.c.chen@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1654719885; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/qNS2/+mjRPXJI7P7YF+0Gz1mau+sTRclHFOVZphUJA=; b=Ut8VRDt4xJ1I4JwhkMjbr7mnQmzRUB1611c5TQeuYWI5rirVq/mNe4oFr+7jq32o7HyIVs ETIoHUcGcpo5ycvar6jFvOB0vhZNY1HrPjDKhP6gT9TOim4YhHRJt9VDoq/TT9pIhNzk8o 4inXOpqXxOelXOmwZdD0cQHmiOfAkzQ= X-Rspamd-Queue-Id: 2C457C0004 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="g7Mb/WHr"; spf=none (imf28.hostedemail.com: domain of tim.c.chen@linux.intel.com has no SPF policy when checking 192.55.52.120) smtp.mailfrom=tim.c.chen@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: 4buoxafjzcqqnqxoou4uo7gb9iqmg4cx X-HE-Tag: 1654719884-595559 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 2022-06-08 at 20:07 +0530, Aneesh Kumar K.V wrote: > > > This is what I am testing now. We still need to closely audit that lock > free access to the NODE_DATA()->memtier. You're refering to this or something else? This is a write so seems okay. > + for_each_node_state(node, N_MEMORY) { > + /* > + * Should be safe to do this early in the boot. > + */ > + NODE_DATA(node)->memtier = memtier; > + node_set(node, memtier->nodelist); > + } > migrate_on_reclaim_init(); > For v6 I will keep this as a > separate patch and once we all agree that it is safe, I will fold it > back. Please update code that uses __node_get_memory_tier(node) to use NODE_DATA(node)->memtier; Otherwise the code looks okay at a first glance. Tim > > diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h > index a388a806b61a..3e733de1a8a0 100644 > --- a/include/linux/memory-tiers.h > +++ b/include/linux/memory-tiers.h > @@ -17,7 +17,6 @@ > #define MAX_MEMORY_TIERS (MAX_STATIC_MEMORY_TIERS + 2) > > extern bool numa_demotion_enabled; > -extern nodemask_t promotion_mask; > int node_create_and_set_memory_tier(int node, int tier); > int next_demotion_node(int node); > int node_set_memory_tier(int node, int tier); > @@ -25,15 +24,7 @@ int node_get_memory_tier_id(int node); > int node_reset_memory_tier(int node, int tier); > void node_remove_from_memory_tier(int node); > void node_get_allowed_targets(int node, nodemask_t *targets); > - > -/* > - * By default all nodes are top tiper. As we create new memory tiers > - * we below top tiers we add them to NON_TOP_TIER state. > - */ > -static inline bool node_is_toptier(int node) > -{ > - return !node_isset(node, promotion_mask); > -} > +bool node_is_toptier(int node); > > #else > #define numa_demotion_enabled false > diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h > index aab70355d64f..c4fcfd2b9980 100644 > --- a/include/linux/mmzone.h > +++ b/include/linux/mmzone.h > @@ -928,6 +928,9 @@ typedef struct pglist_data { > /* Per-node vmstats */ > struct per_cpu_nodestat __percpu *per_cpu_nodestats; > atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS]; > +#ifdef CONFIG_TIERED_MEMORY > + struct memory_tier *memtier; > +#endif > } pg_data_t; > > #define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages) > diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c > index 29a038bb38b0..31ef0fab5f19 100644 > --- a/mm/memory-tiers.c > +++ b/mm/memory-tiers.c > @@ -7,6 +7,7 @@ > #include > #include > #include > +#include > > #include "internal.h" > > @@ -26,7 +27,7 @@ struct demotion_nodes { > static void establish_migration_targets(void); > static DEFINE_MUTEX(memory_tier_lock); > static LIST_HEAD(memory_tiers); > -nodemask_t promotion_mask; > +static int top_tier_rank; > /* > * node_demotion[] examples: > * > @@ -135,7 +136,7 @@ static void memory_tier_device_release(struct device *dev) > if (tier->dev.id >= MAX_STATIC_MEMORY_TIERS) > ida_free(&memtier_dev_id, tier->dev.id); > > - kfree(tier); > + kfree_rcu(tier); > } > > /* > @@ -233,6 +234,70 @@ static struct memory_tier *__get_memory_tier_from_id(int id) > return NULL; > } > > +/* > + * Called with memory_tier_lock. Hence the device references cannot > + * be dropped during this function. > + */ > +static void memtier_node_clear(int node, struct memory_tier *memtier) > +{ > + pg_data_t *pgdat; > + > + pgdat = NODE_DATA(node); > + if (!pgdat) > + return; > + > + rcu_assign_pointer(pgdat->memtier, NULL); > + /* > + * Make sure read side see the NULL value before we clear the node > + * from the nodelist. > + */ > + synchronize_rcu(); > + node_clear(node, memtier->nodelist); > +} > + > +static void memtier_node_set(int node, struct memory_tier *memtier) > +{ > + pg_data_t *pgdat; > + > + pgdat = NODE_DATA(node); > + if (!pgdat) > + return; > + /* > + * Make sure we mark the memtier NULL before we assign the new memory tier > + * to the NUMA node. This make sure that anybody looking at NODE_DATA > + * finds a NULL memtier or the one which is still valid. > + */ > + rcu_assign_pointer(pgdat->memtier, NULL); > + synchronize_rcu(); > + node_set(node, memtier->nodelist); > + rcu_assign_pointer(pgdat->memtier, memtier); > +} > + > +bool node_is_toptier(int node) > +{ > + bool toptier; > + pg_data_t *pgdat; > + struct memory_tier *memtier; > + > + pgdat = NODE_DATA(node); > + if (!pgdat) > + return false; > + > + rcu_read_lock(); > + memtier = rcu_dereference(pgdat->memtier); > + if (!memtier) { > + toptier = true; > + goto out; > + } > + if (memtier->rank >= top_tier_rank) > + toptier = true; > + else > + toptier = false; > +out: > + rcu_read_unlock(); > + return toptier; > +} > + > static int __node_create_and_set_memory_tier(int node, int tier) > { > int ret = 0; > @@ -253,7 +318,7 @@ static int __node_create_and_set_memory_tier(int node, int tier) > goto out; > } > } > - node_set(node, memtier->nodelist); > + memtier_node_set(node, memtier); > out: > return ret; > } > @@ -275,12 +340,12 @@ int node_create_and_set_memory_tier(int node, int tier) > if (current_tier->dev.id == tier) > goto out; > > - node_clear(node, current_tier->nodelist); > + memtier_node_clear(node, current_tier); > > ret = __node_create_and_set_memory_tier(node, tier); > if (ret) { > /* reset it back to older tier */ > - node_set(node, current_tier->nodelist); > + memtier_node_set(node, current_tier); > goto out; > } > > @@ -305,7 +370,7 @@ static int __node_set_memory_tier(int node, int tier) > ret = -EINVAL; > goto out; > } > - node_set(node, memtier->nodelist); > + memtier_node_set(node, memtier); > out: > return ret; > } > @@ -374,12 +439,12 @@ int node_reset_memory_tier(int node, int tier) > if (current_tier->dev.id == tier) > goto out; > > - node_clear(node, current_tier->nodelist); > + memtier_node_clear(node, current_tier); > > ret = __node_set_memory_tier(node, tier); > if (ret) { > /* reset it back to older tier */ > - node_set(node, current_tier->nodelist); > + memtier_node_set(node, current_tier); > goto out; > } > > @@ -407,7 +472,7 @@ void node_remove_from_memory_tier(int node) > * empty then unregister it to make it invisible > * in sysfs. > */ > - node_clear(node, memtier->nodelist); > + memtier_node_clear(node, memtier); > if (nodes_empty(memtier->nodelist)) > unregister_memory_tier(memtier); > > @@ -570,15 +635,13 @@ static void establish_migration_targets(void) > * a memory tier, we consider that tier as top tiper from > * which promotion is not allowed. > */ > - promotion_mask = NODE_MASK_NONE; > list_for_each_entry_reverse(memtier, &memory_tiers, list) { > nodes_and(allowed, node_states[N_CPU], memtier->nodelist); > - if (nodes_empty(allowed)) > - nodes_or(promotion_mask, promotion_mask, memtier->nodelist); > - else > + if (!nodes_empty(allowed)) { > + top_tier_rank = memtier->rank; > break; > + } > } > - > pr_emerg("top tier rank is %d\n", top_tier_rank); > allowed = NODE_MASK_NONE; > /* > @@ -748,7 +811,7 @@ static const struct attribute_group *memory_tier_attr_groups[] = { > > static int __init memory_tier_init(void) > { > - int ret; > + int ret, node; > struct memory_tier *memtier; > > ret = subsys_system_register(&memory_tier_subsys, memory_tier_attr_groups); > @@ -766,7 +829,13 @@ static int __init memory_tier_init(void) > panic("%s() failed to register memory tier: %d\n", __func__, ret); > > /* CPU only nodes are not part of memory tiers. */ > - memtier->nodelist = node_states[N_MEMORY]; > + for_each_node_state(node, N_MEMORY) { > + /* > + * Should be safe to do this early in the boot. > + */ > + NODE_DATA(node)->memtier = memtier; > + node_set(node, memtier->nodelist); > + } > migrate_on_reclaim_init(); > > return 0;