From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 616B7C43334 for ; Fri, 17 Jun 2022 10:41:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BADC46B0071; Fri, 17 Jun 2022 06:41:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B5DC66B0073; Fri, 17 Jun 2022 06:41:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9FE016B0074; Fri, 17 Jun 2022 06:41:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 904336B0071 for ; Fri, 17 Jun 2022 06:41:40 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 55B1233071 for ; Fri, 17 Jun 2022 10:41:40 +0000 (UTC) X-FDA: 79587386760.27.AFF9FE5 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf30.hostedemail.com (Postfix) with ESMTP id B4F398008E for ; Fri, 17 Jun 2022 10:41:39 +0000 (UTC) Received: from fraeml706-chm.china.huawei.com (unknown [172.18.147.207]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4LPb8w3WG6z687Rd; Fri, 17 Jun 2022 18:37:52 +0800 (CST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by fraeml706-chm.china.huawei.com (10.206.15.55) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2375.24; Fri, 17 Jun 2022 12:41:37 +0200 Received: from localhost (10.81.209.131) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.2375.24; Fri, 17 Jun 2022 11:41:36 +0100 Date: Fri, 17 Jun 2022 11:41:32 +0100 From: Jonathan Cameron To: Ying Huang CC: Johannes Weiner , Aneesh Kumar K V , , , Wei Xu , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , "Linux Kernel Mailing List" , Hesham Almatary , Dave Hansen , "Alistair Popple" , Dan Williams , "Feng Tang" , Jagdish Gediya , "Baolin Wang" , David Rientjes Subject: Re: [PATCH v5 1/9] mm/demotion: Add support for explicit memory tiers Message-ID: <20220617114132.00000e4b@Huawei.com> In-Reply-To: <2b261518bbb5b8466301f8ab978f408141fa6e68.camel@intel.com> References: <02ee2c97-3bca-8eb6-97d8-1f8743619453@linux.ibm.com> <20220609152243.00000332@Huawei.com> <20220610105708.0000679b@Huawei.com> <4297bd21-e984-9d78-2bca-e70c11749a72@linux.ibm.com> <42f536af-b17d-b001-7b6b-2c6b928f3ecf@linux.ibm.com> <2b261518bbb5b8466301f8ab978f408141fa6e68.camel@intel.com> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.29; i686-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.81.209.131] X-ClientProxiedBy: lhreml709-chm.china.huawei.com (10.201.108.58) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655462500; a=rsa-sha256; cv=none; b=Xo4gvZedE4sLkQXvzIkhPLSSmK/+1b6t50ukYIuLH0MRim0HaY4NizCbmSXplMjAjXp50O /OHh9tGkakv6GBA/lAzdhQ4kvrqRuRlRpUJ6T9vPHIhCsM5lVkM+CoyaK57GKkbH8V/FA8 /DGwFCTh4P0dCazTBETZZAK+5N2PX28= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655462500; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=waRpikxxXe4pdEp3TC329+KQ6BG/oY3vaGXZ+troDoY=; b=0MW/YXsmerS5qEkeFMj+GRLakBvKFlZ8a/jzVWpdKklB5SPxA5pqSPoZaCtVHOIsd5s/1Z 8KL3dvIz+NhmvSqnkVTH2rtr8KIHAVBs8MjXByXCe/AalzK0VaDhdOoeZQ5O0dEuzZ7Dhi t58Uww69rqyTBNQ4VGAHWu26IJYyEoE= X-Stat-Signature: sx96b7yhju7bzrjwcqf9gztky3mrx13a X-Rspamd-Queue-Id: B4F398008E Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com X-Rspamd-Server: rspam07 X-Rspam-User: X-HE-Tag: 1655462499-270348 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, 16 Jun 2022 09:11:24 +0800 Ying Huang wrote: > On Tue, 2022-06-14 at 14:56 -0400, Johannes Weiner wrote: > > On Tue, Jun 14, 2022 at 01:31:37PM +0530, Aneesh Kumar K V wrote: =20 > > > On 6/13/22 9:20 PM, Johannes Weiner wrote: =20 > > > > On Mon, Jun 13, 2022 at 07:53:03PM +0530, Aneesh Kumar K V wrote: = =20 > > > > > If the kernel still can't make the right decision, userspace coul= d rearrange > > > > > them in any order using rank values. Without something like rank,= if > > > > > userspace needs to fix things up, it gets hard with device > > > > > hotplugging. ie, the userspace policy could be that any new PMEM = tier device > > > > > that is hotplugged, park it with a very low-rank value and hence = lowest in > > > > > demotion order by default. (echo 10 > > > > > > /sys/devices/system/memtier/memtier2/rank) . After that userspace= could > > > > > selectively move the new devices to the correct memory tier? =20 > > > >=20 > > > > I had touched on this in the other email. > > > >=20 > > > > This doesn't work if two drivers that should have separate policies > > > > collide into the same tier - which is very likely with just 3 tiers. > > > > So it seems to me the main usecase for having a rank tunable falls > > > > apart rather quickly until tiers are spaced out more widely. And it > > > > does so at the cost of an, IMO, tricky to understand interface. > > > > =20 > > >=20 > > > Considering the kernel has a static map for these tiers, how can two = drivers > > > end up using the same tier? If a new driver is going to manage a memo= ry > > > device that is of different characteristics than the one managed by d= ax/kmem, > > > we will end up adding=20 > > >=20 > > > #define MEMORY_TIER_NEW_DEVICE 4 > > >=20 > > > The new driver will never use MEMORY_TIER_PMEM > > >=20 > > > What can happen is two devices that are managed by DAX/kmem that > > > should be in two memory tiers get assigned the same memory tier > > > because the dax/kmem driver added both the device to the same memory = tier. > > >=20 > > > In the future we would avoid that by using more device properties lik= e HMAT > > > to create additional memory tiers with different rank values. ie, we = would > > > do in the dax/kmem create_tier_from_rank() . =20 > >=20 > > Yes, that's the type of collision I mean. Two GPUs, two CXL-attached > > DRAMs of different speeds etc. > >=20 > > I also like Huang's idea of using latency characteristics instead of > > abstract distances. Though I'm not quite sure how feasible this is in > > the short term, and share some concerns that Jonathan raised. But I > > think a wider possible range to begin with makes sense in any case. > > =20 > > > > In the other email I had suggested the ability to override not just > > > > the per-device distance, but also the driver default for new devices > > > > to handle the hotplug situation. > > > > =20 > > >=20 > > > I understand that the driver override will be done via module paramet= ers. > > > How will we implement device override? For example in case of dax/kme= m driver > > > the device override will be per dax device? What interface will we us= e to set the override? > > >=20 > > > IIUC in the above proposal the dax/kmem will do > > >=20 > > > node_create_and_set_memory_tier(numa_node, get_device_tier_index(dev_= dax)); > > >=20 > > > get_device_tier_index(struct dev_dax *dev) > > > { > > > =A0=A0=A0=A0return dax_kmem_tier_index; // module parameter > > > } > > >=20 > > > Are you suggesting to add a dev_dax property to override the tier def= aults? =20 > >=20 > > I was thinking a new struct memdevice and struct memtype(?). Every > > driver implementing memory devices like this sets those up and > > registers them with generic code and preset parameters. The generic > > code creates sysfs directories and allows overriding the parameters. > >=20 > > struct memdevice { > > struct device dev; > > unsigned long distance; > > struct list_head siblings; > > /* nid? ... */ > > }; > >=20 > > struct memtype { > > struct device_type type; > > unsigned long default_distance; > > struct list_head devices; > > }; > >=20 > > That forms the (tweakable) tree describing physical properties. =20 >=20 > In general, I think memtype is a good idea. I have suggested > something similar before. It can describe the characters of a > specific type of memory (same memory media with different interface > (e.g., CXL, or DIMM) will be different memory types). And they can > be used to provide overriding information. I'm not sure you are suggesting interface as one element of distinguishing types, or as the element - just in case it's as 'the element'. Ignore the next bit if not ;) Memory "interface" isn't going to be enough of a distinction. If you want = to have a default distance it would need to be different for cases where the same 'type' of RAM has very different characteristics. Applies everywhere but given CXL 'defines' a lot of this - if we just have DRAM attached via CXL: 1. 16-lane direct attached DRAM device. (low latency - high bw) 2. 4x 16-lane direct attached DRAM interleaved (low latency - very high bw) 3. 4-lane direct attached DRAM device (low latency - low bandwidth) 4. 16-lane to single switch, 4x 4-lane devices interleaved (mid latency - h= igh bw) 5. 4-lane to single switch, 4x 4-lane devices interleaved (mid latency, mid= bw) 6. 4x 16-lane so 4 switch, each switch to 4 DRAM devices (mid latency, very= high bw) (7. 16 land directed attached nvram. (midish latency, high bw - perf wise m= ight be similarish to 4). It could be a lot more complex, but hopefully that conveys that 'type' is next to useless to characterize things unless we have a very large number of potential subtypes. If we were on current tiering proposal we'd just have the CXL subsystem manage multiple tiers to cover what is attached. >=20 > As for memdevice, I think that we already have "node" to represent > them in sysfs. Do we really need another one? Is it sufficient to > add some links to node in the appropriate directory? For example, > make memtype class device under the physical device (e.g. CXL device), > and create links to node inside the memtype class device directory? >=20 > > From that, the kernel then generates the ordered list of tiers. =20 >=20 > As Jonathan Cameron pointed, we may need the memory tier ID to be > stable if possible. I know this isn't a easy task. At least we can > make the default memory tier (CPU local DRAM) ID stable (for example > make it always 128)? That provides an anchor for users to understand. >=20 > Best Regards, > Huang, Ying >=20 > > > > This should be less policy than before. Driver default and per-devi= ce > > > > distances (both overridable) combined with one tunable to set the > > > > range of distances that get grouped into tiers. > > > > =20 > > >=20 > > > Can you elaborate more on how distance value will be used? The device= /device NUMA node can have > > > different distance value from other NUMA nodes. How do we group them? > > > for ex: earlier discussion did outline three different topologies. Ca= n you > > > ellaborate how we would end up grouping them using distance? > > >=20 > > > For ex: in the topology below node 2 is at distance 30 from Node0 and= 40 from Nodes > > > so how will we classify node 2? > > >=20 > > >=20 > > > Node 0 & 1 are DRAM nodes, node 2 & 3 are PMEM nodes. > > >=20 > > > 20 > > > =A0=A0Node 0 (DRAM) ---- Node 1 (DRAM) > > > =A0=A0=A0=A0=A0=A0=A0| \ / | > > > =A0=A0=A0=A0=A0=A0=A0| 30 40 X 40 | 30 > > > =A0=A0=A0=A0=A0=A0=A0| / \ | > > > =A0=A0Node 2 (PMEM) ---- Node 3 (PMEM) > > > 40 > > >=20 > > > node distances: > > > node 0 1 2 3 > > > =A0=A0=A00 10 20 30 40 > > > =A0=A0=A01 20 10 40 30 > > > =A0=A0=A02 30 40 10 40 > > > =A0=A0=A03 40 30 40 10 =20 > >=20 > > I'm fairly confused by this example. Do all nodes have CPUs? Isn't > > this just classic NUMA, where optimizing for locality makes the most > > sense, rather than tiering? > >=20 > > Forget the interface for a second, I have no idea how tiering on such > > a system would work. One CPU's lower tier can be another CPU's > > toptier. There is no lowest rung from which to actually *reclaim* > > pages. Would the CPUs just demote in circles? > >=20 > > And the coldest pages on one socket would get demoted into another > > socket and displace what that socket considers hot local memory? > >=20 > > I feel like I missing something. > >=20 > > When we're talking about tiered memory, I'm thinking about CPUs > > utilizing more than one memory node. If those other nodes have CPUs, > > you can't reliably establish a singular tier order anymore and it > > becomes classic NUMA, no? =20 >=20 >=20