From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10CE4C2BD05 for ; Mon, 24 Jun 2024 08:29:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82C6D6B0405; Mon, 24 Jun 2024 04:29:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DC3D6B0406; Mon, 24 Jun 2024 04:29:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A3F66B0407; Mon, 24 Jun 2024 04:29:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4C52C6B0405 for ; Mon, 24 Jun 2024 04:29:34 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C36091A0A0B for ; Mon, 24 Jun 2024 08:29:33 +0000 (UTC) X-FDA: 82265108226.27.8972C81 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.19]) by imf27.hostedemail.com (Postfix) with ESMTP id 0066840017 for ; Mon, 24 Jun 2024 08:29:30 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PTAi3adF; spf=pass (imf27.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.19 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719217765; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C87GHsFZM6mz58JQsFFZK3WTllY5WPuObm0ZwPKgKQs=; b=sWvL9/T3B3VbAO+dD+ugTTWiotnjqdvI1Z7rVy3nZC6s3BR4cc+hIyYS2KfkRtvDZqkCo5 P/Bq0GzRyz/99vu0jdk7kaCXR97eBoXduCT29J9Zpfqw3QiivjlD3t97neVX/HVJQHatmn /ZQ2yNFrplgLfqcLVyJ5maqhGjVjI2s= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PTAi3adF; spf=pass (imf27.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.19 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719217765; a=rsa-sha256; cv=none; b=k3JlHNAspvwWV9QU/80b36fowoaTSsS0TYjBRkLJEXOzNJmBTZ0fG+WgYrCa0x6RhEIfeR 1ukHf/alqNYqIMrJ2Hs30HbP3JtLcTUaC+spLSyAvhRTxbd9pJr5ANgrlVUEWkTnCKmeyy BSKKKhZB4TfnVThBk0jHkekyUgYU8ng= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719217771; x=1750753771; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=4ZK+iiwg1Tx+nIJ69uu/VGxpDLLeClnkryef18dfxok=; b=PTAi3adFk0B3KaAVF3fxU9U2PaG37EP4++qtBsYEmtbHFzqO58I+BAk8 kk1s3sw6snKi/PHvcAFMosKx8+is5T9NPehG2bgpx3vGcdTm0mxscvmFq 3QlaDXo3+V2BPfpZN6KC5B4+St7ZBCXiKTSGpO2d/97i6UjCZzwY3uTba nPO2LDzNrdPPJxckVMBhDLGSoa4QrBXJixJW0YWaJF3vT2HYO9ad6w/PK 3UxaGRSKAOejmoW7TFxFYdU2Wj7RKgk3AnwLig4hc2fCMq9qZE4lRQIY8 DPqJP83C+cnCS+siV4iBo3/jnEHB4NA2U0Gf52TpZ/ksEDbJ7FhPIgSio Q==; X-CSE-ConnectionGUID: 6VqR/FRdT+a0yY8Vtis8bA== X-CSE-MsgGUID: ewWgtgl4RcS7KYg0hh2Yig== X-IronPort-AV: E=McAfee;i="6700,10204,11112"; a="15931392" X-IronPort-AV: E=Sophos;i="6.08,261,1712646000"; d="scan'208";a="15931392" Received: from orviesa004.jf.intel.com ([10.64.159.144]) by fmvoesa113.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jun 2024 01:29:29 -0700 X-CSE-ConnectionGUID: 8aMdpA13R8e2UTrHEJaLtQ== X-CSE-MsgGUID: D8+9q8xURvCuhS7ukgH/YA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,261,1712646000"; d="scan'208";a="48394394" Received: from unknown (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orviesa004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jun 2024 01:29:24 -0700 From: "Huang, Ying" To: "Ho-Ren (Jack) Chuang" Cc: "Jonathan Cameron" , "Gregory Price" , aneesh.kumar@linux.ibm.com, mhocko@suse.com, tj@kernel.org, john@jagalactic.com, "Eishan Mirakhur" , "Vinicius Tavares Petrucci" , "Ravis OpenSrc" , "Alistair Popple" , "Srinivasulu Thanneeru" , "SeongJae Park" , "Rafael J. Wysocki" , Len Brown , Andrew Morton , Dave Jiang , Dan Williams , linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, "Ho-Ren (Jack) Chuang" , "Ho-Ren (Jack) Chuang" , "Ho-Ren (Jack) Chuang" , linux-cxl@vger.kernel.org, qemu-devel@nongnu.org Subject: Re: [PATCH v1] memory tier: consolidate the initialization of memory tiers In-Reply-To: <20240621044833.3953055-1-horen.chuang@linux.dev> (Ho-Ren Chuang's message of "Fri, 21 Jun 2024 04:48:30 +0000") References: <20240621044833.3953055-1-horen.chuang@linux.dev> Date: Mon, 24 Jun 2024 16:27:32 +0800 Message-ID: <87wmmezqx7.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 0066840017 X-Stat-Signature: e5w3wu84nb3454gkj19umbioby8y8t3b X-HE-Tag: 1719217770-793424 X-HE-Meta: U2FsdGVkX1/jPwoap2RoD1sqjUUp0UGpzBWZO8jRuHvKo/3Etis6pQ3t29StbvEUBUwA+n/comAU+rUoDa0sbhotFBxK5ii7/rTe6ekw42DD5XYB4Ktb6hMY+0mykHcSb3yH533jscamaInO3RQqwqVPqnd5slpKic7roZ95NkA7bXFtM2vpvqoL9xAYA5ze+TtWZca8RKcUJU6QvPyvTTtai92u/c5rJ3v7jrt5Q3FhhI+EevRigIixcCUV2yBnExJj0hWv+Ae+tdlwr7oiMkiGPxCO4Y5unWxd7Wfwy5xhYSgFWMUjbF2AQvJiLHh393lY+IvJx8tgWwY0mNcM2clA9EPfOyQHC2QH0pKQCdXYrASi/NQVVpV6PDeubN/XMDirLGVGEsF7yQxaJj/xx8YFPXiKw5cSfLunqxnHaX5UzEdzZk5j0mp5SUiA6xyEjHwk3hLlHk6JZXcJZgkgynRzugueTsvbg7gjMGA10Cch55dS+xHRR4NGZWBQmfV3FClcG7rMn9jVvPdXg4CnmilTfv7C1YJnacVv9tn/m/uDhRsAz2LqTJxb9V5umrih4vvO0ssrxIoXKZjlefvnmT4oXaOIwCvkJIRJEhU+jRCUHrlFlRaFMOBroFuR0xm+p5fkmS91VcpK7Vlp3uDDXGInV7jPvssxjMZH90OahUBpdvUgKiHCqDW69SKlqO95bourXx5JEXzuI/SySErZ/2+jrgUSjOQ02avimhTdKazV/46HJKU6pr6uKvW16qm1VZdcnbeY4PxK0DRT6IzMEpI4oDDlMQQcsa7TOIJeVAx/86COrNbPP4G/y1poNq5+iwCCABuZRaqUE8NyWK81meZstRtT97geVyemQGIvEzexxae4IFc0/q/Qq3fd2TavowMlGcy97ibbWRVKIRiyyg6hVM7mzMU7MjaycZnt3UtUV5cwjkhWUu5z1nF0txIKz5o5w7SlbjO7EkgUNaw CMeqt4MW phKzgxRmo2IdnChBPhc22IsEbWKT6rQpscbI03OR7NC20et23WpKaIqedMVZsVFEwOGryQxDXHevzVr7PpAyaDHEpTbtl9CipwVrMt5zvTCICjrGnXhAaGVUt5SORigyEvugepbY5t+/ZlWf9ssTmu0iXdFU+UjHthw0uw3UC54XmluhuEqTDICC0a2dnAgI3WfLtuGALXBt3Kx4Ftee4Oz5vKEZLDY9Uo6+grARY/CNzi3VFVZBa5RtPgwcabLihXIP/pKqOznktLLKktxSMQfG7MM8bGkLowGxtYk2oE+Qnjhrh1gpoRwR2TxRULXniUYvsz3GAYpcu7473DHviOHr2lebpas/jq0QebUu4Ur/JHsU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi, Jack, Thanks for patch! "Ho-Ren (Jack) Chuang" writes: > If we simply move the set_node_memory_tier() from memory_tier_init() to > late_initcall(), it will result in HMAT not registering the > mt_adistance_algorithm callback function, because set_node_memory_tier() > is not performed during the memory tiering initialization phase, > leading to a lack of correct default_dram information. > > Therefore, we introduced a nodemask to pass the information of the > default DRAM nodes. The reason for not choosing to reuse > default_dram_type->nodes is that it is not clean enough. So in the end, > we use a __initdata variable, which is a variable that is released once > initialization is complete, including both CPU and memory nodes for HMAT > to iterate through. > > Besides, since default_dram_type may be checked/used during the > initialization process of HMAT and drivers, it is better to keep the > allocation of default_dram_type in memory_tier_init(). > > Signed-off-by: Ho-Ren (Jack) Chuang Suggested-by: Jonathan Cameron > --- > Hi all, > > The current memory tier initialization process is distributed across two > different functions, memory_tier_init() and memory_tier_late_init(). This > design is hard to maintain. Thus, this patch is proposed to reduce the > possible code paths by consolidating different initialization patches into one. > > The earlier discussion with Jonathan and Ying is listed here: > https://lore.kernel.org/lkml/20240405150244.00004b49@Huawei.com/ > > If we want to put these two initializations together, they must be placed > together in the later function. Because only at that time, the HMAT information > will be ready, adist between nodes can be calculated, and memory tiering can be > established based on the adist. So we position the initialization at > memory_tier_init() to the memory_tier_late_init() call. > > Moreover, it's natural to keep memory_tier initialization in drivers at > device_initcall() level. > > This patchset is based on commits cf93be18fa1b and a72a30af550c: > [0/2] https://lkml.kernel.org/r/20240405000707.2670063-1-horenchuang@bytedance.com > [1/2] https://lkml.kernel.org/r/20240405000707.2670063-2-horenchuang@bytedance.com > [1/2] https://lkml.kernel.org/r/20240405000707.2670063-3-horenchuang@bytedance.com It appears that you should switch the parts before and after "---". This is the real patch description, as pointed out by Andrew too. > Thanks, > Ho-Ren (Jack) Chuang > > drivers/acpi/numa/hmat.c | 4 ++- > include/linux/memory-tiers.h | 6 ++++ > mm/memory-tiers.c | 70 ++++++++++++++++++------------------ > 3 files changed, 44 insertions(+), 36 deletions(-) > > diff --git a/drivers/acpi/numa/hmat.c b/drivers/acpi/numa/hmat.c > index 2c8ccc91ebe6..31a77a3324a8 100644 > --- a/drivers/acpi/numa/hmat.c > +++ b/drivers/acpi/numa/hmat.c > @@ -939,11 +939,13 @@ static int hmat_set_default_dram_perf(void) > int nid, pxm; > struct memory_target *target; > struct access_coordinate *attrs; > + nodemask_t default_dram_nodes; > > if (!default_dram_type) > return -EIO; > > - for_each_node_mask(nid, default_dram_type->nodes) { > + default_dram_nodes = mt_get_default_dram_nodemask(); > + for_each_node_mask(nid, default_dram_nodes) { We don't need 'default_dram_type' in the function actually. It appears that we can hide it in memory-tiers.c now? > pxm = node_to_pxm(nid); > target = find_mem_target(pxm); > if (!target) > diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h > index 0d70788558f4..1567db7bd40e 100644 > --- a/include/linux/memory-tiers.h > +++ b/include/linux/memory-tiers.h > @@ -51,6 +51,7 @@ int mt_perf_to_adistance(struct access_coordinate *perf, int *adist); > struct memory_dev_type *mt_find_alloc_memory_type(int adist, > struct list_head *memory_types); > void mt_put_memory_types(struct list_head *memory_types); > +nodemask_t mt_get_default_dram_nodemask(void); > #ifdef CONFIG_MIGRATION > int next_demotion_node(int node); > void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets); > @@ -149,5 +150,10 @@ static inline struct memory_dev_type *mt_find_alloc_memory_type(int adist, > static inline void mt_put_memory_types(struct list_head *memory_types) > { > } > + > +static inline nodemask_t mt_get_default_dram_nodemask(void) > +{ > + return NODE_MASK_NONE; > +} > #endif /* CONFIG_NUMA */ > #endif /* _LINUX_MEMORY_TIERS_H */ > diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c > index 6632102bd5c9..7d4b7f53dd8f 100644 > --- a/mm/memory-tiers.c > +++ b/mm/memory-tiers.c > @@ -43,6 +43,7 @@ static LIST_HEAD(memory_tiers); > static LIST_HEAD(default_memory_types); > static struct node_memory_type_map node_memory_types[MAX_NUMNODES]; > struct memory_dev_type *default_dram_type; > +static nodemask_t default_dram_nodes __initdata = NODE_MASK_NONE; > > static const struct bus_type memory_tier_subsys = { > .name = "memory_tiering", > @@ -125,6 +126,11 @@ static inline struct memory_tier *to_memory_tier(struct device *device) > return container_of(device, struct memory_tier, dev); > } > > +nodemask_t __init mt_get_default_dram_nodemask(void) > +{ > + return default_dram_nodes; > +} > + Why not just expose 'default_dram_nodes'? > static __always_inline nodemask_t get_memtier_nodemask(struct memory_tier *memtier) > { > nodemask_t nodes = NODE_MASK_NONE; > @@ -671,27 +677,38 @@ EXPORT_SYMBOL_GPL(mt_put_memory_types); > > /* > * This is invoked via `late_initcall()` to initialize memory tiers for > - * CPU-less memory nodes after driver initialization, which is > - * expected to provide `adistance` algorithms. > + * memory nodes, both with and without CPUs. After the initialization of > + * firmware and devices, adistance algorithms are expected to be provided. > */ > static int __init memory_tier_late_init(void) > { > int nid; > + struct memory_tier *memtier; > > guard(mutex)(&memory_tier_lock); > + /* > + * Look at all the existing and uninitialized N_MEMORY nodes and > + * add them to default memory tier or to a tier if we already have > + * memory types assigned. > + */ > for_each_node_state(nid, N_MEMORY) { During the function run, the node may change between N_MEMORY and !N_MEMORY in theory. So, it appears necessary to get/put_online_mems() in the function? > - /* > - * Some device drivers may have initialized memory tiers > - * between `memory_tier_init()` and `memory_tier_late_init()`, > - * potentially bringing online memory nodes and > - * configuring memory tiers. Exclude them here. > - */ > - if (node_memory_types[nid].memtype) > - continue; > + if (!node_state(nid, N_CPU)) Why? I think that we should "continue" here even if node_state(nid, N_CPU). > + /* > + * Some device drivers may have initialized > + * memory tiers, potentially bringing memory nodes > + * online and configuring memory tiers. > + * Exclude them here. > + */ > + if (node_memory_types[nid].memtype) > + continue; > > - set_node_memory_tier(nid); > + memtier = set_node_memory_tier(nid); > + if (IS_ERR(memtier)) > + /* > + * Continue with memtiers we are able to setup. > + */ > + break; > } > - > establish_demotion_targets(); > > return 0; > @@ -876,7 +893,6 @@ static int __meminit memtier_hotplug_callback(struct notifier_block *self, > static int __init memory_tier_init(void) > { > int ret, node; > - struct memory_tier *memtier; > > ret = subsys_virtual_register(&memory_tier_subsys, NULL); > if (ret) > @@ -887,7 +903,8 @@ static int __init memory_tier_init(void) > GFP_KERNEL); > WARN_ON(!node_demotion); > #endif > - mutex_lock(&memory_tier_lock); > + > + guard(mutex)(&memory_tier_lock); > /* > * For now we can have 4 faster memory tiers with smaller adistance > * than default DRAM tier. > @@ -898,28 +915,11 @@ static int __init memory_tier_init(void) > panic("%s() failed to allocate default DRAM tier\n", __func__); > > /* > - * Look at all the existing N_MEMORY nodes and add them to > - * default memory tier or to a tier if we already have memory > - * types assigned. > + * Record nodes with memory and CPU to set default DRAM performance. > */ For one line comments, we can use /* Record nodes with memory and CPU to set default DRAM performance. */ > - for_each_node_state(node, N_MEMORY) { > - if (!node_state(node, N_CPU)) > - /* > - * Defer memory tier initialization on > - * CPUless numa nodes. These will be initialized > - * after firmware and devices are initialized. > - */ > - continue; > - > - memtier = set_node_memory_tier(node); > - if (IS_ERR(memtier)) > - /* > - * Continue with memtiers we are able to setup > - */ > - break; > - } > - establish_demotion_targets(); > - mutex_unlock(&memory_tier_lock); > + for_each_node_state(node, N_MEMORY) > + if (node_state(node, N_CPU)) > + node_set(node, default_dram_nodes); Why not use nodes_andnot(default_dram_nodes, node_states[N_MEMORY], node_states[N_CPU]); > hotplug_memory_notifier(memtier_hotplug_callback, MEMTIER_HOTPLUG_PRI); > return 0; -- Best Regards, Huang, Ying