From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC3FEC67861 for ; Tue, 9 Apr 2024 16:12:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 547FA6B008A; Tue, 9 Apr 2024 12:12:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D13A6B008C; Tue, 9 Apr 2024 12:12:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 371686B0092; Tue, 9 Apr 2024 12:12:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 16A1B6B008A for ; Tue, 9 Apr 2024 12:12:12 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CC9E3C03BB for ; Tue, 9 Apr 2024 16:12:11 +0000 (UTC) X-FDA: 81990485262.09.423A7A3 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf09.hostedemail.com (Postfix) with ESMTP id 83975140010 for ; Tue, 9 Apr 2024 16:12:09 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf09.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712679129; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=REhUFzE+gNWJMCSfIRfQ6gmmv3zw3SEtW38YEIK89Mc=; b=ooIQdI4fxawjGsibp2v9y5Hx22Sj6BWy6SyqKZ1D9p+kLp8xuJW2lBnx9ldNxqAzNwjzU0 NDBe3z9/cER/eGNRmVpzaHBhGP1pb/f8yxY8Xh86UgFr+6JIeQfC0DsmvfOWj8+7UYqX8w MMbRRWrHJtQ2z+ODRGyjzFSBfflclaU= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf09.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712679129; a=rsa-sha256; cv=none; b=C0Kw0psoJUxAvm2SosuXk6X2MzMi/JqSt4AIvNb9IXOk+jKN3A4CySEz67d6hEmDZ6d0Bn BWE0rk4RK4UZvh/AsQzUe7eyETsWNNiY/wQjXD+TBkOiYBjl8l+y5S7WoeLpXsASIzAmJm 4tTipoindayiFD6dv6Q0tF0Oi4akaKc= Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4VDWC83HBWz6J9yq; Wed, 10 Apr 2024 00:10:28 +0800 (CST) Received: from lhrpeml500005.china.huawei.com (unknown [7.191.163.240]) by mail.maildlp.com (Postfix) with ESMTPS id D5F01140519; Wed, 10 Apr 2024 00:12:05 +0800 (CST) Received: from localhost (10.202.227.76) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Tue, 9 Apr 2024 17:12:05 +0100 Date: Tue, 9 Apr 2024 17:12:04 +0100 From: Jonathan Cameron To: "Ho-Ren (Jack) Chuang" CC: "Huang, Ying" , Gregory Price , , , , , Eishan Mirakhur , Vinicius Tavares Petrucci , Ravis OpenSrc , Alistair Popple , Srinivasulu Thanneeru , SeongJae Park , Dan Williams , Vishal Verma , Dave Jiang , "Andrew Morton" , , , , "Linux Memory Management List" , "Ho-Ren (Jack) Chuang" , "Ho-Ren (Jack) Chuang" , , Hao Xiang Subject: Re: [PATCH v11 2/2] memory tier: create CPUless memory tiers after obtaining HMAT info Message-ID: <20240409171204.00001710@Huawei.com> In-Reply-To: References: <20240405000707.2670063-1-horenchuang@bytedance.com> <20240405000707.2670063-3-horenchuang@bytedance.com> <20240405150244.00004b49@Huawei.com> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.1.0 (GTK 3.24.33; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.202.227.76] X-ClientProxiedBy: lhrpeml100003.china.huawei.com (7.191.160.210) To lhrpeml500005.china.huawei.com (7.191.163.240) X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 83975140010 X-Stat-Signature: rdtecn1w65fgek8mbkqcn8x191zrtxu6 X-Rspam-User: X-HE-Tag: 1712679129-368892 X-HE-Meta: U2FsdGVkX1+Gbb+PfzkA9AA1GFi5BcOIHWJqbk//HiWmbANh73nUVPn2uSbtkCmC0udt8Z+bh7R2yTfCVCUI4NlSSdJ65gDVbvnxCmnnBkaXxOwlDrPu/06/ZWXeLD8Yv9pYaihn2fvtzVdv+DmWRLuxSA2cv0tWm9Qj3m7poUzlR+cLL2nn4WP53K72xk2uxgkvQqyyiD3p/X9BuTKWZP90sUtyuiGidkbY33h3TvhTVO03eYD+fUUfiUx3SvE7v9N3NMpUDmWp+FlFDw++gehsjJhjicxhjz2Y2LtfqjDfmGMai9q7hJpybQKnXYPqb1Zp3uIC9Ytae1K0VN9detus6B88DHSMU+jB2urlUi4ikwuP+bIctTPcfK2BHO13VIvAE+JvQiUXO0dPS5IkGyL7xMqq+57ef1UwX8FeXRx0vDuKbz54cJkRL0XpthpqjTnJ9O6Dmq1Bfvu0Lp8vn8D3G1nNwzLtkMyug8RtL9SMR3pQc2z4IxN7+mLZgxxc6j8PMt8iRNwUMAs8r7zaKpeHpTf8LVrwhYO2iOJQLC6MxGWGIVZii0YZ1GJ1gGqLpBXF1tgwY5IyvxJ9tIimRNmIMmEU0XPoJ4/d7A7772jfTJWXKZnaa6YvAq+uu3j8f6VvZDRoghffzKNNL6ys05d5JKtdX+1hLXktgC98qpbyv0pX1tu6QMNCaSqDMRvYrP+i0V1E7SZuEICujWxu2lYgtC8Eacylc47azNj4M2mF99C7CACh041eXfWj3pf8e8T75Xpvcn93P0moE9LbSUyTkjc3v/5AUsHeWVLePc0BLTuHTzmrYMTcAentfC8CnmfOxAcoFva/XIdLjOPmKcjd+yNYeyhWK2kW7VdkFxpAqdkvkYvlqWAzAQKeVbvbE/Gs11G7iyPhK4cUoTZCzBL16KI76K9mIMfxkheOWHjsOZoFOawVrENLkKBShLDJeNkJyDkBje4jfZ7rriA fGd2Dlt3 VHZbmdFVavrna7qZ6EakPKBpbTRE3OuPEIAWUeje00OU8Xc4+xEoROwE0W/KNxhvL7yQSZzQB+GtfVq0dpTbOXrmpgZmYUVpnb565kcfAjVrdWoAwroR4jBkwx+U0S/QENkvY5bedYTpbZzUFOcjE7oiKVZLefkvMzdGjOUaPJoZfAavtG3R9KGqc9QKH7FoTNOl4epon+eG7NzmlH6sqJGRmSiLUNWpcQzElpoA6QIUwuXGkf7upWWqW7LlR8n4Ot7F4dSE9kXRJMS2g18eSSF9cT/Zixpj2azpszFBgnqmykSwl8x3DB9oLizZL5UN+cbhFO36ISM7ChtX3GajV9ZkaJx80ciCCF1xNhAykMsKkza5gEwplQPVjRETJD+X4N89D X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, 5 Apr 2024 15:43:47 -0700 "Ho-Ren (Jack) Chuang" wrote: > On Fri, Apr 5, 2024 at 7:03=E2=80=AFAM Jonathan Cameron > wrote: > > > > On Fri, 5 Apr 2024 00:07:06 +0000 > > "Ho-Ren (Jack) Chuang" wrote: > > =20 > > > The current implementation treats emulated memory devices, such as > > > CXL1.1 type3 memory, as normal DRAM when they are emulated as normal = memory > > > (E820_TYPE_RAM). However, these emulated devices have different > > > characteristics than traditional DRAM, making it important to > > > distinguish them. Thus, we modify the tiered memory initialization pr= ocess > > > to introduce a delay specifically for CPUless NUMA nodes. This delay > > > ensures that the memory tier initialization for these nodes is deferr= ed > > > until HMAT information is obtained during the boot process. Finally, > > > demotion tables are recalculated at the end. > > > > > > * late_initcall(memory_tier_late_init); > > > Some device drivers may have initialized memory tiers between > > > `memory_tier_init()` and `memory_tier_late_init()`, potentially bring= ing > > > online memory nodes and configuring memory tiers. They should be excl= uded > > > in the late init. > > > > > > * Handle cases where there is no HMAT when creating memory tiers > > > There is a scenario where a CPUless node does not provide HMAT inform= ation. > > > If no HMAT is specified, it falls back to using the default DRAM tier. > > > > > > * Introduce another new lock `default_dram_perf_lock` for adist calcu= lation > > > In the current implementation, iterating through CPUlist nodes requir= es > > > holding the `memory_tier_lock`. However, `mt_calc_adistance()` will e= nd up > > > trying to acquire the same lock, leading to a potential deadlock. > > > Therefore, we propose introducing a standalone `default_dram_perf_loc= k` to > > > protect `default_dram_perf_*`. This approach not only avoids deadlock > > > but also prevents holding a large lock simultaneously. > > > > > > * Upgrade `set_node_memory_tier` to support additional cases, includi= ng > > > default DRAM, late CPUless, and hot-plugged initializations. > > > To cover hot-plugged memory nodes, `mt_calc_adistance()` and > > > `mt_find_alloc_memory_type()` are moved into `set_node_memory_tier()`= to > > > handle cases where memtype is not initialized and where HMAT informat= ion is > > > available. > > > > > > * Introduce `default_memory_types` for those memory types that are not > > > initialized by device drivers. > > > Because late initialized memory and default DRAM memory need to be ma= naged, > > > a default memory type is created for storing all memory types that are > > > not initialized by device drivers and as a fallback. > > > > > > Signed-off-by: Ho-Ren (Jack) Chuang > > > Signed-off-by: Hao Xiang > > > Reviewed-by: "Huang, Ying" =20 > > > > Hi - one remaining question. Why can't we delay init for all nodes > > to either drivers or your fallback late_initcall code. > > It would be nice to reduce possible code paths. =20 >=20 > I try not to change too much of the existing code structure in > this patchset. >=20 > To me, postponing/moving all memory tier registrations to > late_initcall() is another possible action item for the next patchset. >=20 > After tier_mem(), hmat_init() is called, which requires registering > `default_dram_type` info. This is when `default_dram_type` is needed. > However, it is indeed possible to postpone the latter part, > set_node_memory_tier(), to `late_init(). So, memory_tier_init() can > indeed be split into two parts, and the latter part can be moved to > late_initcall() to be processed together. >=20 > Doing this all memory-type drivers have to call late_initcall() to > register a memory tier. I=E2=80=99m not sure how many they are? >=20 > What do you guys think? Gut feeling - if you are going to move it for some cases, move it for all of them. Then we only have to test once ;) J >=20 > > > > Jonathan > > > > =20 > > > --- > > > mm/memory-tiers.c | 94 +++++++++++++++++++++++++++++++++++----------= -- > > > 1 file changed, 70 insertions(+), 24 deletions(-) > > > > > > diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c > > > index 516b144fd45a..6632102bd5c9 100644 > > > --- a/mm/memory-tiers.c > > > +++ b/mm/memory-tiers.c =20 > > > > > > =20 > > > @@ -855,7 +892,8 @@ static int __init memory_tier_init(void) > > > * For now we can have 4 faster memory tiers with smaller adist= ance > > > * than default DRAM tier. > > > */ > > > - default_dram_type =3D alloc_memory_type(MEMTIER_ADISTANCE_DRAM); > > > + default_dram_type =3D mt_find_alloc_memory_type(MEMTIER_ADISTAN= CE_DRAM, > > > + &default_memory_t= ypes); > > > if (IS_ERR(default_dram_type)) > > > panic("%s() failed to allocate default DRAM tier\n", __= func__); > > > > > > @@ -865,6 +903,14 @@ static int __init memory_tier_init(void) > > > * types assigned. > > > */ > > > for_each_node_state(node, N_MEMORY) { > > > + if (!node_state(node, N_CPU)) > > > + /* > > > + * Defer memory tier initialization on > > > + * CPUless numa nodes. These will be initialized > > > + * after firmware and devices are initialized. = =20 > > > > Could the comment also say why we can't defer them all? > > > > (In an odd coincidence we have a similar issue for some CPU hotplug > > related bring up where review feedback was move all cases later). > > =20 > > > + */ > > > + continue; > > > + > > > memtier =3D set_node_memory_tier(node); > > > if (IS_ERR(memtier)) > > > /* =20 > > =20 >=20 >=20