From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12C84C5478C for ; Fri, 1 Mar 2024 08:24:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 81DF36B008C; Fri, 1 Mar 2024 03:24:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 77C7B6B0093; Fri, 1 Mar 2024 03:24:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 580E56B0095; Fri, 1 Mar 2024 03:24:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 425CB6B008C for ; Fri, 1 Mar 2024 03:24:06 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 18CF01A02E7 for ; Fri, 1 Mar 2024 08:24:06 +0000 (UTC) X-FDA: 81847782492.15.0A771B2 Received: from mail-qv1-f51.google.com (mail-qv1-f51.google.com [209.85.219.51]) by imf12.hostedemail.com (Postfix) with ESMTP id 390FE4000D for ; Fri, 1 Mar 2024 08:24:03 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IJ31id8D; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf12.hostedemail.com: domain of horenchuang@bytedance.com designates 209.85.219.51 as permitted sender) smtp.mailfrom=horenchuang@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709281444; a=rsa-sha256; cv=none; b=RoSPaLIA683qjAU+yQkWlLshuTEwlb739M1H7DhtCyn4YE+0tuuQYeV4HVN96zcEshmQwp B07+mENDC5yhGs6Zhd87Svni1cCQfQcvgjgB3biDA83kYIUDbCDHKi0MHrMVUuXP3QUJRI 4TTi3ZJYQH3KzM4GcNqVpnyvddGZMoM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IJ31id8D; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf12.hostedemail.com: domain of horenchuang@bytedance.com designates 209.85.219.51 as permitted sender) smtp.mailfrom=horenchuang@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709281444; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/6TShqh1i4+cFFfpKPT0xQIuYg06VEOVkdOiTtXAx9c=; b=IBydgh+AHSACfzU1wn8OuoorlMFOWNRuvWJZPshebJTRTPBcp/zYRWrnpUtcVyf9JNQ/xj 2tr4nf9qozwZk9JSzGD8DVC+4qVpF/fu10Xtt95pCFAxX2Wa//Y83057jAJIFLoBKTk5Wo a2MiMgdXGOyElINOgAgbhDvvK+PNy60= Received: by mail-qv1-f51.google.com with SMTP id 6a1803df08f44-68f901192afso7610376d6.1 for ; Fri, 01 Mar 2024 00:24:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1709281443; x=1709886243; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/6TShqh1i4+cFFfpKPT0xQIuYg06VEOVkdOiTtXAx9c=; b=IJ31id8D+30YYLqrj10h8sBeM55Z5CK+3kRRNC4gy/ObvY/iksOSA203UB1BNCi3Hb z1ma8c+Z0USTQiFVS0UILbvyLJZQEhMxMcpYvLqcRC8d8R60aTy0T9NaC6pG1cvkmZLd h+FLekbnSuuG5nElBJfTiPcBDoIvak7mrxzwInmU1qNaVwpEQppZO+jcuUX6YAcONDsI uyVbMV4awxCafPOobDnN/C3Oz4wzlxgKrKq64anErb5BIbM3J5j0smINQk4Pm0G8b+Wi TcpKuUFVV9mMgIAlcLgy8YrlHBhGgMRprDMB9Yu3RbKKApbYggH2odnDMhUH2J+ntiEN or2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709281443; x=1709886243; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/6TShqh1i4+cFFfpKPT0xQIuYg06VEOVkdOiTtXAx9c=; b=MRWm+vv86MAS+4SvugV0PV/NSfD5BxlRF8bZTXqFRLjwswhn7UDi/AMqlNpCloAcqp /+ncAjYRa2JIMJBMaNkIWO4XnXQjiorngcOW4OC0uU1j9bU0m7riKjBLlif+iIE6DI8g 87E3uAlau2qU3E5P1XO2MrHXLiggVRoE6hMs4qdhoS0+O4JCPqrrZPY0yzhmyieds/Ed MyRuoKC9V4ObwrAIfUe1dRGamGFNZfLmuXL940KH07lCeBrzikS8Zil7hdRi1EBvrPMs mndnuRZUSJPjkzLGNhuIjCSCxwty7JhAWmNA7Vg9fUcVFBhOw5/iN3iLRlsPeTaEorXS gSiw== X-Forwarded-Encrypted: i=1; AJvYcCUMBCndAt4HNMN3Qeft7d/SixHSKinJ2EuIbJhSFJ7gn3/pCepG8sCcEbDWmtzwr3pqtmt/CZcNNPW1/5k3J1a15v0= X-Gm-Message-State: AOJu0YwYquH4wUuGuSo/DVG4pbigpN348YOZ7To/H0cfXt0jBlknWTPv /z8MJmZl73nYGvo98hIlBa4yB5oxgkfm5BknI9VqyMB6XGNyQ9Fz6yoih2A+52U= X-Google-Smtp-Source: AGHT+IFOE//de5gW8/t5EN6iDGtxoPupJsqmHB+gIs7CE75HdwkbfH6RIWi1eK5iB+kKO7THZmG8hQ== X-Received: by 2002:a05:6214:42:b0:68f:280f:14cc with SMTP id c2-20020a056214004200b0068f280f14ccmr982605qvr.35.1709281443055; Fri, 01 Mar 2024 00:24:03 -0800 (PST) Received: from n231-228-171.byted.org ([130.44.215.123]) by smtp.gmail.com with ESMTPSA id y19-20020a0cd993000000b0068fc392f526sm1631907qvj.127.2024.03.01.00.24.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 00:24:02 -0800 (PST) From: "Ho-Ren (Jack) Chuang" To: "Hao Xiang" , "Gregory Price" , aneesh.kumar@linux.ibm.com, mhocko@suse.com, tj@kernel.org, john@jagalactic.com, "Eishan Mirakhur" , "Vinicius Tavares Petrucci" , "Ravis OpenSrc" , "Alistair Popple" , "Rafael J. Wysocki" , Len Brown , Andrew Morton , Dave Jiang , Dan Williams , Jonathan Cameron , Huang Ying , "Ho-Ren (Jack) Chuang" , linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: "Ho-Ren (Jack) Chuang" , "Ho-Ren (Jack) Chuang" , linux-cxl@vger.kernel.org, qemu-devel@nongnu.org Subject: [PATCH v1 1/1] memory tier: acpi/hmat: create CPUless memory tiers after obtaining HMAT info Date: Fri, 1 Mar 2024 08:22:45 +0000 Message-Id: <20240301082248.3456086-2-horenchuang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240301082248.3456086-1-horenchuang@bytedance.com> References: <20240301082248.3456086-1-horenchuang@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 390FE4000D X-Stat-Signature: 6spsqhy9567wk1cigzkyp8fpc1ctm7ux X-HE-Tag: 1709281443-330206 X-HE-Meta: U2FsdGVkX1+yJ5l/Y3XuJPaHpHjC7XRanGykv8LWHaZucb0mTQugSO7yhEkKQXdy9BGb8X39QWD0ycn3WOke1j8Ek5sltztru/yltdxfYx8fcpkelJBLVuPWABAMBy5nKCCRwfv+Rhvrey/6BeqArzHFbSjNI1nbmWXE2vuLkP079LAHPXoLza91oXb5XPNveMmY8GsnbhUANd43cNFre6M4P9mTpiu6ER+jxY+q+YTuVNW9hi4+JT99duphgEu43vkIUtWceKPD5XXErnCCzz6ju5B/HdICa3iTKAILT4jBiuwcdrtdcNRuNlh6sfoXAW0hLfzQfqBCUarhepdG9+J46PsRPqY4QfdsQbozkz93hw/5cwcgxKbBNkYhA4A3jf7AolDULOFjptU8coyLFSs6rp43LTVse2Lq8b9VHOHS3Tj/9F+dg7z2Jj0UYMpcl6IIo6t5qb0HpJRkDaPvj6Q4hSZ4YRbQy+bznjlIAxrlLmQVykowcd7H5AWoRI0UlGHQvl0zYLgWJnRDW01wxxeqMRCKj450a1nMGP4lhEXgdxilsEfyAntnx9AeNneBgGRTihdzeNsLSLGviO2vcDTJ3XQyDzm4RmwjQUcYJvlKgMfiGZWeLlLZg4kIMwL43ehFLqv3KkVgnbzj/6g0MjeLXvpdWFOzCzufzuve5sRsqmtTq6G8knw1hCNGBtco0ISS2kDoPKhYIUz+kAaukHq5LXXsoeihKWIjRLnObPASNtqio3JPoA8HhqcMx7k/Qj49HbfhoT6XTbWuoqnplMVvF25+1CtPFsUMKCwkET183P468Wl0jD5hwc2WDzHDud+tWky/UWtGVPkrw7WGpMcvO3cCrVT/QQgxihWGUYDaplGemjct6Miep4UjzulMAiStrU1i1JJZiLtotFr6dGMumZHAd4vTmyzukdj+QuJOR5+ySpi9OMX3MxBapFuJsb7KhKmN7VwldxNR78Y A7KqTN+p aMBrTuU/1NxNjlvTYrV8mm9pYhzTNR9rIX8MCPyvI2Ru+tPIEYnyGyLQn1mRZkYP/K5ykDiJtsVyXCclC6a8ID1ytWhi/qTXlPFQ2qgyVae5KIx97lHXu1SXriaHu5qsqPvgDLrDNRE+0f2mp31zpLlMkaB0pDMHFD26eJ6QTnw6hdM7ZUYnW09RoRHZqH4rb5xZjVea4lL+1m2vXxjkNwu3MHBpTo+126U2w972YVzlzT6WY4tPbnX8lTuiZrtuL5UbJ/GGu7KFAVFWB+sZv0/CFIKbyOEVdtruFcogqOip/1gc2qKoM1onNBENpmm2dNTUQybDU7ja5XcsvLF51w8TRNmcMrcEDa/KJpIGoZXZRLwxasusZJ0kgMsjaYRgFcEni7AlFNO4tluR3ALwBiZurNTtc/qQV/dm+mooK4orgLsobRnte5J++rD1CGfjU949vVy/R7cYZ52SwcUPzp6FG0g8+h4g558rDAI/5MZ1HxaWa/T6H2g6rKg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: * Introduce `mt_init_with_hmat()` We defer memory tier initialization for those CPUless NUMA nodes until acquiring HMAT info. `mt_init_with_hmat()` is introduced to post-create CPUless memory tiers after obtaining HMAT info. It iterates through each CPUless memory node, creating memory tiers if necessary. Finally, it calculates demotion tables again at the end. * Introduce `hmat_find_alloc_memory_type()` Find or allocate a memory type in the `hmat_memory_types` list. * Make `set_node_memory_tier()` more generic This function can also be used for setting other memory types for a node. To do so, a new argument is added to specify a memory type. * Handle cases where there is no HMAT when creating memory tiers If no HMAT is specified, it falls back to using `default_dram_type`. * Change adist calculation code to use another new lock, mt_perf_lock. Iterating through CPUlist nodes requires holding the `memory_tier_lock`. However, `mt_calc_adistance()` will end up trying to acquire the same lock, leading to a potential deadlock. Therefore, we propose introducing a standalone `mt_perf_lock` to protect `default_dram_perf`. This approach not only avoids deadlock but also prevents holding a large lock simultaneously. Signed-off-by: Ho-Ren (Jack) Chuang Signed-off-by: Hao Xiang --- drivers/acpi/numa/hmat.c | 3 ++ include/linux/memory-tiers.h | 6 +++ mm/memory-tiers.c | 76 ++++++++++++++++++++++++++++++++---- 3 files changed, 77 insertions(+), 8 deletions(-) diff --git a/drivers/acpi/numa/hmat.c b/drivers/acpi/numa/hmat.c index d6b85f0f6082..9f57338b3cb5 100644 --- a/drivers/acpi/numa/hmat.c +++ b/drivers/acpi/numa/hmat.c @@ -1038,6 +1038,9 @@ static __init int hmat_init(void) if (!hmat_set_default_dram_perf()) register_mt_adistance_algorithm(&hmat_adist_nb); + /* Post-create CPUless memory tiers after getting HMAT info */ + mt_init_with_hmat(); + return 0; out_put: hmat_free_structures(); diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h index 69e781900082..2f845e90c033 100644 --- a/include/linux/memory-tiers.h +++ b/include/linux/memory-tiers.h @@ -48,6 +48,7 @@ int mt_calc_adistance(int node, int *adist); int mt_set_default_dram_perf(int nid, struct access_coordinate *perf, const char *source); int mt_perf_to_adistance(struct access_coordinate *perf, int *adist); +void mt_init_with_hmat(void); #ifdef CONFIG_MIGRATION int next_demotion_node(int node); void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets); @@ -136,5 +137,10 @@ static inline int mt_perf_to_adistance(struct access_coordinate *perf, int *adis { return -EIO; } + +static inline void mt_init_with_hmat(void) +{ + +} #endif /* CONFIG_NUMA */ #endif /* _LINUX_MEMORY_TIERS_H */ diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c index 0537664620e5..7a0a579b3deb 100644 --- a/mm/memory-tiers.c +++ b/mm/memory-tiers.c @@ -35,7 +35,9 @@ struct node_memory_type_map { }; static DEFINE_MUTEX(memory_tier_lock); +static DEFINE_MUTEX(mt_perf_lock); static LIST_HEAD(memory_tiers); +static LIST_HEAD(hmat_memory_types); static struct node_memory_type_map node_memory_types[MAX_NUMNODES]; struct memory_dev_type *default_dram_type; @@ -502,7 +504,7 @@ static inline void __init_node_memory_type(int node, struct memory_dev_type *mem } } -static struct memory_tier *set_node_memory_tier(int node) +static struct memory_tier *set_node_memory_tier(int node, struct memory_dev_type *new_memtype) { struct memory_tier *memtier; struct memory_dev_type *memtype; @@ -514,7 +516,7 @@ static struct memory_tier *set_node_memory_tier(int node) if (!node_state(node, N_MEMORY)) return ERR_PTR(-EINVAL); - __init_node_memory_type(node, default_dram_type); + __init_node_memory_type(node, new_memtype); memtype = node_memory_types[node].memtype; node_set(node, memtype->nodes); @@ -623,6 +625,56 @@ void clear_node_memory_type(int node, struct memory_dev_type *memtype) } EXPORT_SYMBOL_GPL(clear_node_memory_type); +static struct memory_dev_type *hmat_find_alloc_memory_type(int adist) +{ + bool found = false; + struct memory_dev_type *mtype; + + list_for_each_entry(mtype, &hmat_memory_types, list) { + if (mtype->adistance == adist) { + found = true; + break; + } + } + if (!found) { + mtype = alloc_memory_type(adist); + if (!IS_ERR(mtype)) + list_add(&mtype->list, &hmat_memory_types); + } + return mtype; +} + +static void mt_create_with_hmat(int node) +{ + struct memory_dev_type *mtype = NULL; + int adist = MEMTIER_ADISTANCE_DRAM; + + mt_calc_adistance(node, &adist); + if (adist != MEMTIER_ADISTANCE_DRAM) { + mtype = hmat_find_alloc_memory_type(adist); + if (IS_ERR(mtype)) + pr_err("%s() failed to allocate a tier\n", __func__); + } else { + mtype = default_dram_type; + } + + set_node_memory_tier(node, mtype); +} + +void mt_init_with_hmat(void) +{ + int nid; + + mutex_lock(&memory_tier_lock); + for_each_node_state(nid, N_MEMORY) + if (!node_state(nid, N_CPU)) + mt_create_with_hmat(nid); + + establish_demotion_targets(); + mutex_unlock(&memory_tier_lock); +} +EXPORT_SYMBOL_GPL(mt_init_with_hmat); + static void dump_hmem_attrs(struct access_coordinate *coord, const char *prefix) { pr_info( @@ -636,7 +688,7 @@ int mt_set_default_dram_perf(int nid, struct access_coordinate *perf, { int rc = 0; - mutex_lock(&memory_tier_lock); + mutex_lock(&mt_perf_lock); if (default_dram_perf_error) { rc = -EIO; goto out; @@ -684,7 +736,7 @@ int mt_set_default_dram_perf(int nid, struct access_coordinate *perf, } out: - mutex_unlock(&memory_tier_lock); + mutex_unlock(&mt_perf_lock); return rc; } @@ -700,7 +752,7 @@ int mt_perf_to_adistance(struct access_coordinate *perf, int *adist) perf->read_bandwidth + perf->write_bandwidth == 0) return -EINVAL; - mutex_lock(&memory_tier_lock); + mutex_lock(&mt_perf_lock); /* * The abstract distance of a memory node is in direct proportion to * its memory latency (read + write) and inversely proportional to its @@ -713,7 +765,7 @@ int mt_perf_to_adistance(struct access_coordinate *perf, int *adist) (default_dram_perf.read_latency + default_dram_perf.write_latency) * (default_dram_perf.read_bandwidth + default_dram_perf.write_bandwidth) / (perf->read_bandwidth + perf->write_bandwidth); - mutex_unlock(&memory_tier_lock); + mutex_unlock(&mt_perf_lock); return 0; } @@ -797,7 +849,7 @@ static int __meminit memtier_hotplug_callback(struct notifier_block *self, break; case MEM_ONLINE: mutex_lock(&memory_tier_lock); - memtier = set_node_memory_tier(arg->status_change_nid); + memtier = set_node_memory_tier(arg->status_change_nid, default_dram_type); if (!IS_ERR(memtier)) establish_demotion_targets(); mutex_unlock(&memory_tier_lock); @@ -836,7 +888,15 @@ static int __init memory_tier_init(void) * types assigned. */ for_each_node_state(node, N_MEMORY) { - memtier = set_node_memory_tier(node); + if (!node_state(node, N_CPU)) + /* + * Defer memory tier initialization on CPUless numa nodes. + * These will be initialized when HMAT information is + * available. + */ + continue; + + memtier = set_node_memory_tier(node, default_dram_type); if (IS_ERR(memtier)) /* * Continue with memtiers we are able to setup -- Hao Xiang and Ho-Ren (Jack) Chuang