From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A140CC54E5D for ; Mon, 18 Mar 2024 20:13:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 333396B0088; Mon, 18 Mar 2024 16:13:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E4248D0001; Mon, 18 Mar 2024 16:13:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 183FD6B0092; Mon, 18 Mar 2024 16:13:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0202D6B0088 for ; Mon, 18 Mar 2024 16:13:13 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C2268A098A for ; Mon, 18 Mar 2024 20:13:13 +0000 (UTC) X-FDA: 81911259066.25.738BCE5 Received: from mail-yb1-f169.google.com (mail-yb1-f169.google.com [209.85.219.169]) by imf19.hostedemail.com (Postfix) with ESMTP id 786831A0008 for ; Mon, 18 Mar 2024 20:13:10 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=V0kEy5Ah; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf19.hostedemail.com: domain of horenchuang@bytedance.com designates 209.85.219.169 as permitted sender) smtp.mailfrom=horenchuang@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710792791; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2GmiV5M2hXSbQEKR31mdypKhoX7dIAGE7+RzEV5WkEY=; b=zbdtSVbYdmlgj+BcPE5IUIGVGNUizo6bK/nX/d1auIFOYwVYpYZk5A64U9Hdlqd7NN5del iGb9IVvBkLCdO0RZ+90A8WMKtxtzJ3IGztjmiDJCRlXaktFxQunWEDHXYN+DHwGhArBB42 8cg8mhuiElKDLKoeUz9LHIaXuohzC18= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=V0kEy5Ah; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf19.hostedemail.com: domain of horenchuang@bytedance.com designates 209.85.219.169 as permitted sender) smtp.mailfrom=horenchuang@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710792791; a=rsa-sha256; cv=none; b=ZC8oC4rv4JOvKekvJdsUDWaHAXjpTLEfi+mliTfuycw59/35/lTr1ri0BS9Qz/VW/6oFYi +R466Qkxmuv+PEVB0jv2iNIeR970ZTq3HcwR/JYKSLcvekAq9m8aYvW0+GiZ4yWpMV4mD+ aD9Fy96lghxgTdQMlzYjqhD0xACxW7A= Received: by mail-yb1-f169.google.com with SMTP id 3f1490d57ef6-dc25e12cc63so5382944276.0 for ; Mon, 18 Mar 2024 13:13:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710792789; x=1711397589; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=2GmiV5M2hXSbQEKR31mdypKhoX7dIAGE7+RzEV5WkEY=; b=V0kEy5AhHzMkq5Wfp5Mpdsify8OnVP5b5mu6Qo1TLrZTOGE1S+lazguygZZ6/q6o8F 1P1L9Z0gcfjZohsZ6/b3fxUVB0paYEIqJH4ifEofgn/J3WH7CoZkJmdOWVAfg26uKfeS Ch3/HsMhWKZBuKuSRnQxS/s+i+9oTZQzyfJEszAIjSsOOA0LsWneDJaY+vS+mZw3m4KO 3Q6+aWHmMqcgxKFwpJaDOzVsIbscUq3wWEppz2woTWyhAifM1kBq3tB+qZO0dxO3uwiW krOHrxBUM9A01nLtxRhDeYZj3hegcYh2zN4OOaGCYQveDncB9bx3TLHZT/C9Sr5w0JTz EaRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710792789; x=1711397589; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2GmiV5M2hXSbQEKR31mdypKhoX7dIAGE7+RzEV5WkEY=; b=t7Sxt9Uvw6vtH7iPx3TFWNXeN5zg2eLl8mRJsYTUwM24/QXBsWl2GYFFaa6Ia3+3Ey nys1tt47NwDUPEIaTnzRtZJb8ewH2+G58hdIdnCnTPdHMVmVSrp7mK5SgewsQvOTxlss nMnKqjYMF9shzLAV1tfXpohR+pCcYOUj5hA9wV/4jB3nPb2zLPGO9RPbqQ75fOyzkYO8 X+Y7KkTnvO9P/oLQVfZ3LfEjKubXh7fOLr8cX/7eW03q10VInlhHTPujl86wNUinlc1y l5XE/GLh0zWQbSjVqVnWusN4LMgMTWbQACG6u0UFmlkcH7jnfaKWA4Dmb0znZC4965Xp xRfw== X-Forwarded-Encrypted: i=1; AJvYcCWYsfVgdJh0pyWQzgM+gqIxhdPcQ2A5IVDg/gQ8D9oZNfDtZZWctSM8dl/RTvbhRbHAbJAhmKsxk4dNrJ9ZUioFhis= X-Gm-Message-State: AOJu0Yy/GZ2q1R14XUWZGfjOrN3F5D846zIYc1vfJwNssghcyIEDCDuZ fGO2RMLd2cNjxpJ5893jTnT2m31/kKOOBvlKBT5WmxTIwx7mkGYiAy32RR5i8FqmeHISTklYoaX 9RDW0O+U19NWAqGZrPNg1Q+5QM7r8hoAaocHGzQ== X-Google-Smtp-Source: AGHT+IFETTrBJ77lnga2XTx8H37L+r3SUM9K/oqa4D1IjsNxvnj1s00F9NGDxpflePP6ZXGG+xUJkzolNGOfV0V/g0A= X-Received: by 2002:a05:6902:2191:b0:dcf:2697:5ca3 with SMTP id dl17-20020a056902219100b00dcf26975ca3mr607819ybb.5.1710792789278; Mon, 18 Mar 2024 13:13:09 -0700 (PDT) MIME-Version: 1.0 References: <20240312061729.1997111-1-horenchuang@bytedance.com> <20240312061729.1997111-2-horenchuang@bytedance.com> <874jdb4xk8.fsf@yhuang6-desk2.ccr.corp.intel.com> <8734st2qu0.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: <8734st2qu0.fsf@yhuang6-desk2.ccr.corp.intel.com> From: "Ho-Ren (Jack) Chuang" Date: Mon, 18 Mar 2024 13:12:58 -0700 Message-ID: Subject: Re: [External] Re: [PATCH v2 1/1] memory tier: acpi/hmat: create CPUless memory tiers after obtaining HMAT info To: "Huang, Ying" Cc: Gregory Price , aneesh.kumar@linux.ibm.com, mhocko@suse.com, tj@kernel.org, john@jagalactic.com, Eishan Mirakhur , Vinicius Tavares Petrucci , Ravis OpenSrc , Alistair Popple , "Rafael J. Wysocki" , Len Brown , Dan Williams , Vishal Verma , Dave Jiang , Andrew Morton , Jonathan Cameron , linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, nvdimm@lists.linux.dev, linux-cxl@vger.kernel.org, linux-mm@kvack.org, "Ho-Ren (Jack) Chuang" , "Ho-Ren (Jack) Chuang" , qemu-devel@nongnu.org, Hao Xiang , sthanneeru@micron.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 786831A0008 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: hox49q5w6nd6t5dc685m6n66m8fyosfx X-HE-Tag: 1710792790-843790 X-HE-Meta: U2FsdGVkX1+FvtF86va6MW/VGcvWYkPOULmCO3SxMHgNkls33wRZhyn45KdAaVbvsoQvGwdweDWjjvRZrO9sDzLCbF/y9TFb5Px6qDy08681j2dFALqvV5/OuOhl+CSvy4ioyLrpHdkjrxvUmMgI1895ZzjcfHahCYT1glWnRteYvtAyb0DwXIVo3E3oyq4paSAur2PvtNBGXr0JPecwH4RVxznX3RN/34Nvc+TIdraJ/Gu+W61wFLgtEGt8RNFQRxsERvjHHlZ3jSFxYxrFOl/MeChHT2/VH7MQoSWJ/w653qhfWwVZmJOO3kBuDAUZDl2Lwa5NLj5S874y46dCVubAAknYYE5LAl3KEBZXJxS6XwGQQ/0QwCzXRRcS9YrrAPxtjk9mKgiN580L0zL08mgp4QW3vr6famEp6cwSwiaK1TMDKbZU9WLONGadGl0sH5j6kubfGQLn8EJwF36tVvgpm0f5gju9HOus0QeHf+SH8nkLFnnmaCz2FXFz6/cAAKoEljGiBUyNm2FHup1ccb4QYEUOWh4qAVm3WTebQJahh1+Vt0Od8QJ18XpPhLtQCwKruew9ozuVq9nMGR+wR1GQY4+vo17r2VpIsNbrDF4K8/oFKmWnzqsO7WEcrkObD37ky2ZjErAfmtAkmh+/XnwaxrkMtbgK8DjmfCvQ6PPnCIaO+tbfhK9LMJDjFOpoXLXclQaoEPKpgq5AVqeCcAoW1ErPXZ2tguBfC55JNo35R//d4qFKZoVfEkGE2ynWcV3x5HdkgblALokEWdK4wN5eKhaFhhS67Zs5WWgAbR9JCLdR0LjYgWnJPHRqYw+Hv5UC+H7B043KtrXWGvSfsEW9cNluhq5fL0SzuiJ1T0Ve1csfp4H9HOT7b8izKhMxJK3HcOyeg2jFWsnOUY1XXaGhFdeB22Z0FUnt3VIuoNwXzfilO98XSBzDTxDt6wuMZZjAJq2ZV5Cf1ScXnYi yJMZ1/Q2 bc08rK3Mhr6dfRPz82FQS8FQlkQthDFFvkRYdKDyb9ENtU2h1xuJ3OcOqCSo4876kFLo01ld/kZTkditkO9kCiejdvDtaQ2iJOPc8OO0KJNbtaPjb8VrJtI+K48+9yvbEQVNzEDh5ab05TdLCm/48QuXB1+d7kfNGY1m9hfQFmvalpxNW1gIoTa+x9WEFUTOV0LRJiJjKfXJZxZTyjPfz6/NQXZX67J0PNeOvnftqAseVvc4Yww810L/vzCmcDKo14ddlWjjTqyBOYNfqiflsBg9GHYtjdRPiVJSZ2VR3dpHOeD0f3w61BC3uQ7vxmno9cicWplGhA7HlMqOcqInMouEf+GTcHw/nh2kSOZmUc5iqKN8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: I'm working on V3. Thanks for Ying's feedback. cc: sthanneeru@micron.com On Thu, Mar 14, 2024 at 12:54=E2=80=AFAM Huang, Ying = wrote: > > "Ho-Ren (Jack) Chuang" writes: > > > On Tue, Mar 12, 2024 at 2:21=E2=80=AFAM Huang, Ying wrote: > >> > >> "Ho-Ren (Jack) Chuang" writes: > >> > >> > The current implementation treats emulated memory devices, such as > >> > CXL1.1 type3 memory, as normal DRAM when they are emulated as normal= memory > >> > (E820_TYPE_RAM). However, these emulated devices have different > >> > characteristics than traditional DRAM, making it important to > >> > distinguish them. Thus, we modify the tiered memory initialization p= rocess > >> > to introduce a delay specifically for CPUless NUMA nodes. This delay > >> > ensures that the memory tier initialization for these nodes is defer= red > >> > until HMAT information is obtained during the boot process. Finally, > >> > demotion tables are recalculated at the end. > >> > > >> > * Abstract common functions into `find_alloc_memory_type()` > >> > >> We should move kmem_put_memory_types() (renamed to > >> mt_put_memory_types()?) too. This can be put in a separate patch. > >> > > > > Will do! Thanks, > > > > > >> > >> > Since different memory devices require finding or allocating a memor= y type, > >> > these common steps are abstracted into a single function, > >> > `find_alloc_memory_type()`, enhancing code scalability and concisene= ss. > >> > > >> > * Handle cases where there is no HMAT when creating memory tiers > >> > There is a scenario where a CPUless node does not provide HMAT infor= mation. > >> > If no HMAT is specified, it falls back to using the default DRAM tie= r. > >> > > >> > * Change adist calculation code to use another new lock, mt_perf_loc= k. > >> > In the current implementation, iterating through CPUlist nodes requi= res > >> > holding the `memory_tier_lock`. However, `mt_calc_adistance()` will = end up > >> > trying to acquire the same lock, leading to a potential deadlock. > >> > Therefore, we propose introducing a standalone `mt_perf_lock` to pro= tect > >> > `default_dram_perf`. This approach not only avoids deadlock but also > >> > prevents holding a large lock simultaneously. > >> > > >> > Signed-off-by: Ho-Ren (Jack) Chuang > >> > Signed-off-by: Hao Xiang > >> > --- > >> > drivers/acpi/numa/hmat.c | 11 ++++++ > >> > drivers/dax/kmem.c | 13 +------ > >> > include/linux/acpi.h | 6 ++++ > >> > include/linux/memory-tiers.h | 8 +++++ > >> > mm/memory-tiers.c | 70 +++++++++++++++++++++++++++++++++= --- > >> > 5 files changed, 92 insertions(+), 16 deletions(-) > >> > > >> > diff --git a/drivers/acpi/numa/hmat.c b/drivers/acpi/numa/hmat.c > >> > index d6b85f0f6082..28812ec2c793 100644 > >> > --- a/drivers/acpi/numa/hmat.c > >> > +++ b/drivers/acpi/numa/hmat.c > >> > @@ -38,6 +38,8 @@ static LIST_HEAD(targets); > >> > static LIST_HEAD(initiators); > >> > static LIST_HEAD(localities); > >> > > >> > +static LIST_HEAD(hmat_memory_types); > >> > + > >> > >> HMAT isn't a device driver for some memory devices. So I don't think = we > >> should manage memory types in HMAT. > > > > I can put it back in memory-tier.c. How about the list? Do we still > > need to keep a separate list for storing late_inited memory nodes? > > And how about the list name if we need to remove the prefix "hmat_"? > > I don't think we need a separate list for memory-less nodes. Just > iterate all memory-less nodes. > Ok. There is no need to keep a list of memory-less nodes. I will only keep a memory_type list to manage created memory types. > > > >> Instead, if the memory_type of a > >> node isn't set by the driver, we should manage it in memory-tier.c as > >> fallback. > >> > > > > Do you mean some device drivers may init memory tiers between > > memory_tier_init() and late_initcall(memory_tier_late_init);? > > And this is the reason why you mention to exclude > > "node_memory_types[nid].memtype !=3D NULL" in memory_tier_late_init(). > > Is my understanding correct? > > Yes. > Thanks. > >> > static DEFINE_MUTEX(target_lock); > >> > > >> > /* > >> > @@ -149,6 +151,12 @@ int acpi_get_genport_coordinates(u32 uid, > >> > } > >> > EXPORT_SYMBOL_NS_GPL(acpi_get_genport_coordinates, CXL); > >> > > >> > +struct memory_dev_type *hmat_find_alloc_memory_type(int adist) > >> > +{ > >> > + return find_alloc_memory_type(adist, &hmat_memory_types); > >> > +} > >> > +EXPORT_SYMBOL_GPL(hmat_find_alloc_memory_type); > >> > + > >> > static __init void alloc_memory_initiator(unsigned int cpu_pxm) > >> > { > >> > struct memory_initiator *initiator; > >> > @@ -1038,6 +1046,9 @@ static __init int hmat_init(void) > >> > if (!hmat_set_default_dram_perf()) > >> > register_mt_adistance_algorithm(&hmat_adist_nb); > >> > > >> > + /* Post-create CPUless memory tiers after getting HMAT info */ > >> > + memory_tier_late_init(); > >> > + > >> > >> This should be called in memory-tier.c via > >> > >> late_initcall(memory_tier_late_init); > >> > >> Then, we don't need hmat to call it. > >> > > > > Thanks. Learned! > > > > > >> > return 0; > >> > out_put: > >> > hmat_free_structures(); > >> > diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c > >> > index 42ee360cf4e3..aee17ab59f4f 100644 > >> > --- a/drivers/dax/kmem.c > >> > +++ b/drivers/dax/kmem.c > >> > @@ -55,21 +55,10 @@ static LIST_HEAD(kmem_memory_types); > >> > > >> > static struct memory_dev_type *kmem_find_alloc_memory_type(int adis= t) > >> > { > >> > - bool found =3D false; > >> > struct memory_dev_type *mtype; > >> > > >> > mutex_lock(&kmem_memory_type_lock); > >> > - list_for_each_entry(mtype, &kmem_memory_types, list) { > >> > - if (mtype->adistance =3D=3D adist) { > >> > - found =3D true; > >> > - break; > >> > - } > >> > - } > >> > - if (!found) { > >> > - mtype =3D alloc_memory_type(adist); > >> > - if (!IS_ERR(mtype)) > >> > - list_add(&mtype->list, &kmem_memory_types); > >> > - } > >> > + mtype =3D find_alloc_memory_type(adist, &kmem_memory_types); > >> > mutex_unlock(&kmem_memory_type_lock); > >> > > >> > return mtype; > >> > diff --git a/include/linux/acpi.h b/include/linux/acpi.h > >> > index b7165e52b3c6..3f927ff01f02 100644 > >> > --- a/include/linux/acpi.h > >> > +++ b/include/linux/acpi.h > >> > @@ -434,12 +434,18 @@ int thermal_acpi_critical_trip_temp(struct acp= i_device *adev, int *ret_temp); > >> > > >> > #ifdef CONFIG_ACPI_HMAT > >> > int acpi_get_genport_coordinates(u32 uid, struct access_coordinate = *coord); > >> > +struct memory_dev_type *hmat_find_alloc_memory_type(int adist); > >> > #else > >> > static inline int acpi_get_genport_coordinates(u32 uid, > >> > struct access_coordinat= e *coord) > >> > { > >> > return -EOPNOTSUPP; > >> > } > >> > + > >> > +static inline struct memory_dev_type *hmat_find_alloc_memory_type(i= nt adist) > >> > +{ > >> > + return NULL; > >> > +} > >> > #endif > >> > > >> > #ifdef CONFIG_ACPI_NUMA > >> > diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tie= rs.h > >> > index 69e781900082..4bc2596c5774 100644 > >> > --- a/include/linux/memory-tiers.h > >> > +++ b/include/linux/memory-tiers.h > >> > @@ -48,6 +48,9 @@ int mt_calc_adistance(int node, int *adist); > >> > int mt_set_default_dram_perf(int nid, struct access_coordinate *per= f, > >> > const char *source); > >> > int mt_perf_to_adistance(struct access_coordinate *perf, int *adist= ); > >> > +struct memory_dev_type *find_alloc_memory_type(int adist, > >> > + struct list_he= ad *memory_types); > >> > +void memory_tier_late_init(void); > >> > #ifdef CONFIG_MIGRATION > >> > int next_demotion_node(int node); > >> > void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets= ); > >> > @@ -136,5 +139,10 @@ static inline int mt_perf_to_adistance(struct a= ccess_coordinate *perf, int *adis > >> > { > >> > return -EIO; > >> > } > >> > + > >> > +static inline void memory_tier_late_init(void) > >> > +{ > >> > + > >> > +} > >> > #endif /* CONFIG_NUMA */ > >> > #endif /* _LINUX_MEMORY_TIERS_H */ > >> > diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c > >> > index 0537664620e5..79f748d60e6f 100644 > >> > --- a/mm/memory-tiers.c > >> > +++ b/mm/memory-tiers.c > >> > @@ -6,6 +6,7 @@ > >> > #include > >> > #include > >> > #include > >> > +#include > >> > > >> > #include "internal.h" > >> > > >> > @@ -35,6 +36,7 @@ struct node_memory_type_map { > >> > }; > >> > > >> > static DEFINE_MUTEX(memory_tier_lock); > >> > +static DEFINE_MUTEX(mt_perf_lock); > >> > >> Please add comments about what it protects. And put it near the data > >> structure it protects. > >> > > > > Where is better for me to add comments for this? Code? Patch descriptio= n? > > Will put it closer to the protected data. Thanks. > > Just put the comments at the above of the lock in the source code. > Got it. Thanks! > > > > > > >> > static LIST_HEAD(memory_tiers); > >> > static struct node_memory_type_map node_memory_types[MAX_NUMNODES]; > >> > struct memory_dev_type *default_dram_type; > >> > @@ -623,6 +625,58 @@ void clear_node_memory_type(int node, struct me= mory_dev_type *memtype) > >> > } > >> > EXPORT_SYMBOL_GPL(clear_node_memory_type); > >> > > >> > +struct memory_dev_type *find_alloc_memory_type(int adist, struct li= st_head *memory_types) > >> > +{ > >> > + bool found =3D false; > >> > + struct memory_dev_type *mtype; > >> > + > >> > + list_for_each_entry(mtype, memory_types, list) { > >> > + if (mtype->adistance =3D=3D adist) { > >> > + found =3D true; > >> > + break; > >> > + } > >> > + } > >> > + if (!found) { > >> > + mtype =3D alloc_memory_type(adist); > >> > + if (!IS_ERR(mtype)) > >> > + list_add(&mtype->list, memory_types); > >> > + } > >> > + > >> > + return mtype; > >> > +} > >> > +EXPORT_SYMBOL_GPL(find_alloc_memory_type); > >> > + > >> > +static void memory_tier_late_create(int node) > >> > +{ > >> > + struct memory_dev_type *mtype =3D NULL; > >> > + int adist =3D MEMTIER_ADISTANCE_DRAM; > >> > + > >> > + mt_calc_adistance(node, &adist); > >> > + if (adist !=3D MEMTIER_ADISTANCE_DRAM) { > >> > >> We can manage default_dram_type() via find_alloc_memory_type() > >> too. > >> > >> And, if "node_memory_types[node].memtype =3D=3D NULL", we can call > >> mt_calc_adistance(node, &adist) and find_alloc_memory_type() in > >> set_node_memory_tier(). Then, we can cover hotpluged memory node too. > >> > >> > + mtype =3D hmat_find_alloc_memory_type(adist); > >> > + if (!IS_ERR(mtype)) > >> > + __init_node_memory_type(node, mtype); > >> > + else > >> > + pr_err("Failed to allocate a memory type at %s= ()\n", __func__); > >> > + } > >> > + > >> > + set_node_memory_tier(node); > >> > +} > >> > + > >> > +void memory_tier_late_init(void) > >> > +{ > >> > + int nid; > >> > + > >> > + mutex_lock(&memory_tier_lock); > >> > + for_each_node_state(nid, N_MEMORY) > >> > + if (!node_state(nid, N_CPU)) > >> > >> We should exclude "node_memory_types[nid].memtype !=3D NULL". Some me= mory > >> nodes may be onlined by some device drivers and setup memory tiers > >> already. > >> > >> > + memory_tier_late_create(nid); > >> > + > >> > + establish_demotion_targets(); > >> > + mutex_unlock(&memory_tier_lock); > >> > +} > >> > +EXPORT_SYMBOL_GPL(memory_tier_late_init); > >> > + > >> > static void dump_hmem_attrs(struct access_coordinate *coord, const = char *prefix) > >> > { > >> > pr_info( > >> > @@ -636,7 +690,7 @@ int mt_set_default_dram_perf(int nid, struct acc= ess_coordinate *perf, > >> > { > >> > int rc =3D 0; > >> > > >> > - mutex_lock(&memory_tier_lock); > >> > + mutex_lock(&mt_perf_lock); > >> > if (default_dram_perf_error) { > >> > rc =3D -EIO; > >> > goto out; > >> > @@ -684,7 +738,7 @@ int mt_set_default_dram_perf(int nid, struct acc= ess_coordinate *perf, > >> > } > >> > > >> > out: > >> > - mutex_unlock(&memory_tier_lock); > >> > + mutex_unlock(&mt_perf_lock); > >> > return rc; > >> > } > >> > > >> > @@ -700,7 +754,7 @@ int mt_perf_to_adistance(struct access_coordinat= e *perf, int *adist) > >> > perf->read_bandwidth + perf->write_bandwidth =3D=3D 0) > >> > return -EINVAL; > >> > > >> > - mutex_lock(&memory_tier_lock); > >> > + mutex_lock(&mt_perf_lock); > >> > /* > >> > * The abstract distance of a memory node is in direct proport= ion to > >> > * its memory latency (read + write) and inversely proportiona= l to its > >> > @@ -713,7 +767,7 @@ int mt_perf_to_adistance(struct access_coordinat= e *perf, int *adist) > >> > (default_dram_perf.read_latency + default_dram_perf.wr= ite_latency) * > >> > (default_dram_perf.read_bandwidth + default_dram_perf.= write_bandwidth) / > >> > (perf->read_bandwidth + perf->write_bandwidth); > >> > - mutex_unlock(&memory_tier_lock); > >> > + mutex_unlock(&mt_perf_lock); > >> > > >> > return 0; > >> > } > >> > @@ -836,6 +890,14 @@ static int __init memory_tier_init(void) > >> > * types assigned. > >> > */ > >> > for_each_node_state(node, N_MEMORY) { > >> > + if (!node_state(node, N_CPU)) > >> > + /* > >> > + * Defer memory tier initialization on CPUless= numa nodes. > >> > + * These will be initialized when HMAT informa= tion is > >> > >> HMAT is platform specific, we should avoid to mention it in general co= de > >> if possible. > >> > > > > Will fix! Thanks, > > > > > >> > + * available. > >> > + */ > >> > + continue; > >> > + > >> > memtier =3D set_node_memory_tier(node); > >> > if (IS_ERR(memtier)) > >> > /* > >> > > -- > Best Regards, > Huang, Ying -- Best regards, Ho-Ren (Jack) Chuang =E8=8E=8A=E8=B3=80=E4=BB=BB