From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A50CC46CA1 for ; Thu, 12 Oct 2023 12:52:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 004228D0126; Thu, 12 Oct 2023 08:52:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EF62F8D0002; Thu, 12 Oct 2023 08:52:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DBD668D0126; Thu, 12 Oct 2023 08:52:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C92D98D0002 for ; Thu, 12 Oct 2023 08:52:58 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A362BB5176 for ; Thu, 12 Oct 2023 12:52:58 +0000 (UTC) X-FDA: 81336799236.17.0E3516C Received: from outbound-smtp35.blacknight.com (outbound-smtp35.blacknight.com [46.22.139.218]) by imf25.hostedemail.com (Postfix) with ESMTP id AF736A0010 for ; Thu, 12 Oct 2023 12:52:56 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf25.hostedemail.com: domain of mgorman@techsingularity.net designates 46.22.139.218 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697115177; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+PwQdRauJ/B3bEzMpdv1PnLeYxdMgVWrNCHJMJ2z4bE=; b=KT+OPC6SW+SZVcAagevHvmjthja13wq6YpdGvgkHe1SKoH6Ov+aSpICNBrZkot4YMljbi+ CC0GzVdUOdqV1uZexG78gA2RvPLEHOqvAcVMxIERkzJSaDOhbR7z3HNhf3wfSX6HkEsv52 y/uERdwoXSmofTy3VRDfQMb4cG7YgI8= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf25.hostedemail.com: domain of mgorman@techsingularity.net designates 46.22.139.218 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697115177; a=rsa-sha256; cv=none; b=kFLZ/45D247usT6eu/JFwr0Ou0nVT65/aCHlG1BPY/apIEp/IO0ShMqPqLPox1Vo+q+fH0 9luSAoJn44ugydNOSGLFstrWOtVSFVh2Eolv9v3KUzUgUzJTXbWaNyAShAMdgTADxezz1I gDNO+mAg/gSVSIcVbaJE1JeFI3MPgWQ= Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp35.blacknight.com (Postfix) with ESMTPS id 1D1C626EB for ; Thu, 12 Oct 2023 13:52:55 +0100 (IST) Received: (qmail 5844 invoked from network); 12 Oct 2023 12:52:54 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[84.203.197.19]) by 81.17.254.9 with ESMTPSA (AES256-SHA encrypted, authenticated); 12 Oct 2023 12:52:54 -0000 Date: Thu, 12 Oct 2023 13:52:53 +0100 From: Mel Gorman To: "Huang, Ying" Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Arjan Van De Ven , Sudeep Holla , Andrew Morton , Vlastimil Babka , David Hildenbrand , Johannes Weiner , Dave Hansen , Michal Hocko , Pavel Tatashin , Matthew Wilcox , Christoph Lameter Subject: Re: [PATCH 02/10] cacheinfo: calculate per-CPU data cache size Message-ID: <20231012125253.fpeehd6362c5v2sj@techsingularity.net> References: <20230920061856.257597-1-ying.huang@intel.com> <20230920061856.257597-3-ying.huang@intel.com> <20231011122027.pw3uw32sdxxqjsrq@techsingularity.net> <87h6mwf3gf.fsf@yhuang6-desk2.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <87h6mwf3gf.fsf@yhuang6-desk2.ccr.corp.intel.com> X-Rspam-User: X-Stat-Signature: y3uyyahwcs6xjkubi7xonskzmhj5sr4b X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: AF736A0010 X-HE-Tag: 1697115176-290058 X-HE-Meta: U2FsdGVkX1/S0nmAKlrM5ghiSGWPElX2OEdFBquYU3qimQOtxAoYU/9dn+MNDo4c/lD06BC4/X5C6R6VHNMXXUiNBOVV6aRstgtjhiM0jl06J6s1iYrEx3P4wNBuloeUvJYHS952VR33nhZ9yokmep0KnMeo1vAmPydtOxTXDXDBX4aAZ8JZM/tQ4S4LXXQqRr+ikS6za9BJDwgVqdGQCmQVrX18EthALd+WH8EmKgymz+1bvOoIjeGHb0UY6FfQD+bTowepCdGmNWQQFkg2r7s7Rc+ZMXdZ1UeoY+cfgVtzBS20aCgtd3oNFpkdfdFmR1c7Y0HMbT66KnhOsJxwgwIESLPbcFEL9VC+C/sdszh8K9eTohn56tRIJnivfNkTkAFae8kKbSBaVh2KwxjVXEhwe7yItXOG6SHZMJ60M0J8uC11oVjiYwOYeuV6o9deV4Wi4iPeRDk5KCU7ghuAvU3OWqq8d7wvTtvhRQlTr3Mci2psWpJbzwcMbB+AH4icSIhYvFc2bztU0sOKjIUg0zob9LSx1LMN+Mh/0dBZiysp5x5uW/KpOeW1t9MbFH6Ev5sH3Zork2G6e8xn8C3u1dZsSN3mAzfQJ7yBkhBI3VSW3DPh6P/89kfxBAZNuLIHWfbAhUCQmWKpo4aJMYMnTiAw3hqZHCE3WJ3jCzrMNUVLXqitkoyT7LnFun7PyqTamuEarcklR1krfC1q/jur4LGEilkup3O0KsTssbP+K2xxFJseXx9qHd2ykUL4istwhF9sLVwd7p5i1C53Q2d4ULjLcozvISL1dUV1VLQKFcO3NEj9vCp4n5rDqDhXft/j8g+OelN12If4RKLZRdPvPWC3Ix9zr0fU5TK5MRkgURuCxXJRi7Ce3muVmMbsLs1N3fkFiS6v6CdEIZahaHo/2GbOtQ4C54WtqyBHyHseaItM2K7Xk1+iPYGotY6iiuwGibsDt/6djoqdUFqQ1mH DI6x1HHo BkmKUER1kfN8IZwe/N0FPFJ9+zXpIs/sR/u0v8KToJuYQ4m7eMUwZQgLakUP7VCqewOf/HWsGgmB3PBwHebDZn2LL4UBSDjUSQpPBVwSM7ajjT4Ay2YNZyJvcdkVdvyOwW9mz5TU0IRuRB7N2uyAZfIG5/35tsNRhBOJgfYI5bIno8RIL4Q5DcGFE2wGMxaJvJyuJ4nxdXRpLV/SM0ZrFs3oFgOc77Tr4yeUVk7B3yIPp7z6ldVAetuTlYrQXqLs275/leIivwHlew6fBDNz8vt+uY2mWLO8JdUx4+QIduosJFVV9XyafikXhg/QIOcVvaCih7829qCgdyrALZKZKs6Ei8UqlhjsbKHXCuzxrNQFfQFpDM9QuwWVP1YiFR5Gtv4GM5qnTUezaIdhUFOhZvxU9emker98qoIeeqrc3nq9tNkEUuYMWmHw/bhUvNYLcvhiWMrYJ6Q7YBUz2hhGZbpOmRYDJDWTtRlc4ZHTwZ/m1eZUO9h3wFz5f/QMQGBBwZGaU+hJztK/Ch0wZmQp/ezrXpGA5WHqhffvGyP8j35ZymBUbP0DG/rU5psrb6h4XsGsxqg2USNXTlKP5qV9THjusV4WFAiQtOv6dtX9KMCD4CnSAJclhnWQTto/x8f38/eSu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Oct 12, 2023 at 08:08:32PM +0800, Huang, Ying wrote: > Mel Gorman writes: > > > On Wed, Sep 20, 2023 at 02:18:48PM +0800, Huang Ying wrote: > >> Per-CPU data cache size is useful information. For example, it can be > >> used to determine per-CPU cache size. So, in this patch, the data > >> cache size for each CPU is calculated via data_cache_size / > >> shared_cpu_weight. > >> > >> A brute-force algorithm to iterate all online CPUs is used to avoid > >> to allocate an extra cpumask, especially in offline callback. > >> > >> Signed-off-by: "Huang, Ying" > > > > It's not necessarily relevant to the patch, but at least the scheduler > > also stores some per-cpu topology information such as sd_llc_size -- the > > number of CPUs sharing the same last-level-cache as this CPU. It may be > > worth unifying this at some point if it's common that per-cpu > > information is too fine and per-zone or per-node information is too > > coarse. This would be particularly true when considering locking > > granularity, > > > >> Cc: Sudeep Holla > >> Cc: Andrew Morton > >> Cc: Mel Gorman > >> Cc: Vlastimil Babka > >> Cc: David Hildenbrand > >> Cc: Johannes Weiner > >> Cc: Dave Hansen > >> Cc: Michal Hocko > >> Cc: Pavel Tatashin > >> Cc: Matthew Wilcox > >> Cc: Christoph Lameter > >> --- > >> drivers/base/cacheinfo.c | 42 ++++++++++++++++++++++++++++++++++++++- > >> include/linux/cacheinfo.h | 1 + > >> 2 files changed, 42 insertions(+), 1 deletion(-) > >> > >> diff --git a/drivers/base/cacheinfo.c b/drivers/base/cacheinfo.c > >> index cbae8be1fe52..3e8951a3fbab 100644 > >> --- a/drivers/base/cacheinfo.c > >> +++ b/drivers/base/cacheinfo.c > >> @@ -898,6 +898,41 @@ static int cache_add_dev(unsigned int cpu) > >> return rc; > >> } > >> > >> +static void update_data_cache_size_cpu(unsigned int cpu) > >> +{ > >> + struct cpu_cacheinfo *ci; > >> + struct cacheinfo *leaf; > >> + unsigned int i, nr_shared; > >> + unsigned int size_data = 0; > >> + > >> + if (!per_cpu_cacheinfo(cpu)) > >> + return; > >> + > >> + ci = ci_cacheinfo(cpu); > >> + for (i = 0; i < cache_leaves(cpu); i++) { > >> + leaf = per_cpu_cacheinfo_idx(cpu, i); > >> + if (leaf->type != CACHE_TYPE_DATA && > >> + leaf->type != CACHE_TYPE_UNIFIED) > >> + continue; > >> + nr_shared = cpumask_weight(&leaf->shared_cpu_map); > >> + if (!nr_shared) > >> + continue; > >> + size_data += leaf->size / nr_shared; > >> + } > >> + ci->size_data = size_data; > >> +} > > > > This needs comments. > > > > It would be nice to add a comment on top describing the limitation of > > CACHE_TYPE_UNIFIED here in the context of > > update_data_cache_size_cpu(). > > Sure. Will do that. > Thanks. > > The L2 cache could be unified but much smaller than a L3 or other > > last-level-cache. It's not clear from the code what level of cache is being > > used due to a lack of familiarity of the cpu_cacheinfo code but size_data > > is not the size of a cache, it appears to be the share of a cache a CPU > > would have under ideal circumstances. > > Yes. And it isn't for one specific level of cache. It's sum of per-CPU > shares of all levels of cache. But the calculation is inaccurate. More > details are in the below reply. > > > However, as it appears to also be > > iterating hierarchy then this may not be accurate. Caches may or may not > > allow data to be duplicated between levels so the value may be inaccurate. > > Thank you very much for pointing this out! The cache can be inclusive > or not. So, we cannot calculate the per-CPU slice of all-level caches > via adding them together blindly. I will change this in a follow-on > patch. > Please do, I would strongly suggest basing this on LLC only because it's the only value you can be sure of. This change is the only change that may warrant a respin of the series as the history will be somewhat confusing otherwise. -- Mel Gorman SUSE Labs