From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4012AC10F1A for ; Thu, 9 May 2024 17:35:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AED1A6B0092; Thu, 9 May 2024 13:35:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A766F6B0093; Thu, 9 May 2024 13:35:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C8266B0095; Thu, 9 May 2024 13:35:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6C2426B0092 for ; Thu, 9 May 2024 13:35:35 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1CD1A1C131A for ; Thu, 9 May 2024 17:35:35 +0000 (UTC) X-FDA: 82099559430.11.951717C Received: from mailout1.w2.samsung.com (mailout1.w2.samsung.com [211.189.100.11]) by imf22.hostedemail.com (Postfix) with ESMTP id 3E191C000B for ; Thu, 9 May 2024 17:35:32 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=roKHeL+s; spf=pass (imf22.hostedemail.com: domain of a.manzanares@samsung.com designates 211.189.100.11 as permitted sender) smtp.mailfrom=a.manzanares@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715276132; a=rsa-sha256; cv=none; b=sSCacgh9WLAMkwErZ14OQMMT6ayPdBiEDacnmeV45Tny9O2B8XodNfKJi4Iw30khJZW4Sa jYO81/80S56gdDYkQ6G1RrF82Ld43YJgS7yhmJQfYmepGeVE1ksOuq9JjImgMqOEMs29/R 4Euw5fEYGgWrIYS/NdHhIxmFrvFlIyg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=roKHeL+s; spf=pass (imf22.hostedemail.com: domain of a.manzanares@samsung.com designates 211.189.100.11 as permitted sender) smtp.mailfrom=a.manzanares@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715276132; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wUizWTKfjl3Ma8uqS/5vbdnskQUvMGp/AJukB4W0KzI=; b=A2Q5PYgWGl3niB3fRLfVyvbO2jj75yDaz9x4MFHmpPpaPu2th3LdEt0mVfFzhCz9mOhpXG WebNJmv7Y3+FcWrcXEc8w/fyRb+9tpQ/CTEc2rHDjt4hQGdTh5ozIP4My0iiGGsqxP13hj AF5fUMlJFmpp1x1+1SarXx/gmzVovpw= Received: from uscas1p1.samsung.com (unknown [182.198.245.206]) by mailout1.w2.samsung.com (KnoxPortal) with ESMTP id 20240509173530usoutp01620e3097fe2804760445d694d0cf3a5c~N4o10-Vxs1998319983usoutp01z; Thu, 9 May 2024 17:35:30 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w2.samsung.com 20240509173530usoutp01620e3097fe2804760445d694d0cf3a5c~N4o10-Vxs1998319983usoutp01z DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715276130; bh=wUizWTKfjl3Ma8uqS/5vbdnskQUvMGp/AJukB4W0KzI=; h=From:To:CC:Subject:Date:In-Reply-To:References:From; b=roKHeL+szHG4fOGMRqLS2kRFr4f6zeMNMyBdTnFlsNqXRpv+Qzq74nK95V2vHcq0Y 1XoENMx+HSNc0sFv6fsXxJtHQ8Ju8qAFSF6iW2jZ2epuBs4y08eQ3sq8FfsqSCgx6Q CqY1NClpdbGaDwcCfXaJnh385JlpTr8jD9zT75QE= Received: from ussmges1new.samsung.com (u109.gpu85.samsung.co.kr [203.254.195.109]) by uscas1p1.samsung.com (KnoxPortal) with ESMTP id 20240509173530uscas1p1ef3ab35a149d79038a5a45881b811b7a~N4o1nBJZ92885128851uscas1p1K; Thu, 9 May 2024 17:35:30 +0000 (GMT) Received: from uscas1p2.samsung.com ( [182.198.245.207]) by ussmges1new.samsung.com (USCPEMTA) with SMTP id BC.9F.09616.1690D366; Thu, 9 May 2024 13:35:29 -0400 (EDT) Received: from ussmgxs3new.samsung.com (u92.gpu85.samsung.co.kr [203.254.195.92]) by uscas1p1.samsung.com (KnoxPortal) with ESMTP id 20240509173529uscas1p1b6e43b169514d36915cd2bc8aabc4200~N4o1NiboB0744807448uscas1p1O; Thu, 9 May 2024 17:35:29 +0000 (GMT) X-AuditID: cbfec36d-ff7ff70000002590-21-663d0961411b Received: from SSI-EX3.ssi.samsung.com ( [105.128.3.67]) by ussmgxs3new.samsung.com (USCPEXMTA) with SMTP id AB.8D.09511.1690D366; Thu, 9 May 2024 13:35:29 -0400 (EDT) Received: from SSI-EX3.ssi.samsung.com (105.128.2.228) by SSI-EX3.ssi.samsung.com (105.128.2.228) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.1.2507.35; Thu, 9 May 2024 10:35:28 -0700 Received: from SSI-EX3.ssi.samsung.com ([105.128.5.228]) by SSI-EX3.ssi.samsung.com ([105.128.5.228]) with mapi id 15.01.2507.035; Thu, 9 May 2024 10:35:28 -0700 From: Adam Manzanares To: David Rientjes CC: "lsf-pc@lists.linux-foundation.org" , "linux-mm@kvack.org" , Michal Hocko , Dan Williams , John Hubbard , Zi Yan , Bharata B Rao , Dave Jiang , Aneesh Kumar K.V , "Huang, Ying" , Alistair Popple , Christoph Lameter , Andrew Morton , Linus Torvalds , Dave Hansen , Mel Gorman , Jon Grimm , Gregory Price , Wei Xu , Johannes Weiner , SeongJae Park , David Hildenbrand , Davidlohr Bueso , "mcgrof@kernel.org" Subject: Re: [LSF/MM/BPF TOPIC] Locally attached memory tiering Thread-Topic: [LSF/MM/BPF TOPIC] Locally attached memory tiering Thread-Index: AQHaojdIg44HMpIy8UOL06mmtpDI9g== Date: Thu, 9 May 2024 17:35:28 +0000 Message-ID: <6419ec70-ace5-4ded-90fd-ca5ea647f182@nmtadam.samsung> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [105.128.2.176] Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Brightmail-Tracker: H4sIAAAAAAAAA02SfUxTVxjGPfde7m2bsF0Y0gMkbCP8gZMxcMtyGIMx5pa7oJv7kLhlmTZy QSMtpoUNjW7MjiKoFcac5cK08mWRItARioUxaCMWMJGpLRWYfHYrASFthkxAGfXWhP9+7/s+ z3uec3IEeGAbGSo4KMth5TJJVgQpItp6F2+9KhEmZsQOnHgLVTbpSfRnz8fI5Pgc1ZbO4+hy 5a8YOn9uECBXcyFA1ns/kKjhnh6gxaZlHLlW/8FQg2EnanFWUWjZ+Rig+/o1P9Q1+zuOHCVO gMrm/wZoTJ2MVDUtGLIszmFoUv2AQgt2N4X6uDICVVy/TyWHMAW3H5OMZW4BZ/6wyJlr3F8U ozXkMtWdMxhjuFJEMgbPTxRj1awQTElVN2Au9H3ClDkuA+aMcp5kFrpsJHOu4numqdVG7Ar4 UvR2Opt18BtW/lrSPtGBIaURO7wQnld7fQLLBxpYDIQCSL8B2+w2qhiIBIF0PYAPL077igIM 2pQceKZ61O4G/EAPYEGbBuMLN4B3XAN+fFELoInTEl4LScfClRstuJeD6Cio0qyRXhFOmyho 9kxQ3sEL9Duw3flwfZVgXZQMLQ0Er4+BzpMqPy8TdCSsa/wZ90r86RQ4W5LhbQvpJLjssGFe BnQwXOrXP2WcFsPh6YsYnzoAVlV04jwHwyemCZLnl+HY0gzF66OhtsND8pwEH3kWfbwV1l2a fer1X9/TVz5N8N4Q2KNzEN6rQLpeBPu73b4n2g7r8nt9h4XB8/o7PkM2NKnu+vp50DZ7yhci AV5abcJKQCS3ITe3IRO3IRO3IRO3IZMW+F0B4lyFQprJKuJk7LcxColUkSvLjNmfLTWA9a89 8MQibQftw+4YM8AEwAygAI8I8j96LCEj0D9dcuQoK8/eK8/NYhVmECYgIsT+cYl9+wPpTEkO e4hlD7PyZ1NMIAzNx7aP9ff3MC8+X75W8AvW2CpsHU8Zpq9Bwqg+NpFi/aI5LXOreWqT7sQR e+2POdx4mrL9w/zy6AuhXzP4mCdAFZ5PPci2b3FU2wvdxt7xqN1rITuuaraMxL/r0liHQOnr 7pqzH6CROavuo6Ao82jX8er/zmxLwzcZ0xMqpXmjVH1w2GalOO63RmZq84jMWb2nTn1zbXCv rjTyq13vu/ak3vy02TLZ8VJRMtVx2tgRpxvtfDM8kr5xl6o5dXt1JTGao/ZNYkvs6abU8TTb olp7fCTEEF+sy/h325BkKv7Q1axYdmfq2e+0J2+Vt+EDu6VF8+Id7w3FBxFVnz3XPTiTXhhB KA5I4l7B5QrJ/xTD2rJJBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrMKsWRmVeSWpSXmKPExsWS2cDsrJvIaZtmcHmmnMWc9WvYLC4e9LfY dSPEYunEd8wWy+fMZbKYPvUCo8WLDe2MFiduNrJZrL65htHi6/pfzBYv/jxnsli9yddi49NF 7Ba/nv5ltLi35j+rxb7Xe5ktbkx4ymgx+d0zRov7fQ4WbUs2Mlkc/vqGyeJR31t2i/fXPrJb nJw1mcVi9tF77A6SHq2X/rJ5HH7zntlj/+Eij52z7rJ7LNhU6rF4z0smj02rOtk8Nn2axO5x YsZvFo8Jiw4wesw7Gegx+cZyRo/e5ndsHu/3XWXzmDq73mP9lqssAYJRXDYpqTmZZalF+nYJ XBnXm7czFbyXq1h69CFTA+MMiS5GTg4JAROJnzs+MnYxcnEICaxilPjxdyE7hPORUaJr1UQm CGcpo8SdJWtZQFrYBAwkfh/fyAxiiwhoSLTN+M8GUsQssItd4ua+e2wgCWEBe4kdT78BdXMA FTlIHF7NAlGvJ/G0o40VxGYRUJFYtnYKM0gJr4CTxOsJaSBhIQFbie3HF4KVcArYSfy6cZUJ xGYUEJP4fmoNmM0sIC5x68l8JogPBCSW7DnPDGGLSrx8/I8VwlaUuP/9JTtEvY7Egt2f2CBs O4mfn75C2doSyxa+BuvlFRCUODnzCQtEr6TEwRU3WCYwSsxCsm4WklGzkIyahWTULCSjFjCy rmIULy0uzk2vKDbOSy3XK07MLS7NS9dLzs/dxAhMd6f/HY7ZwXjv1ke9Q4xMHIyHGCU4mJVE eKtqrNOEeFMSK6tSi/Lji0pzUosPMUpzsCiJ8959oJEqJJCeWJKanZpakFoEk2Xi4JRqYDr/ uGDicSaL+SdTLe+fZE36ecExxlhK+/vktV6OdxaG/cn68LLN90rwxUk+AkJmwQbi10/7vDw0 Zfqu7Nhda6OfM0Se7QhlSkv/yXfNZpeg8uwAbtu82JgtAcac8gcXKU31fJhTytCmWB+5cq63 hpeo3KmIR93r5Lu+sPLLKG9VXRY6Lc6p2JlXW7C3soCf6fH5ZbUvzPdwHdT1c5nDzhbNvJ2v 80l67OrQyEym25n/dodNl1n9RuzPWrla2/MZk+KXrn4R5B9V0cr2PezwIX2mtYUbek8yzHJ5 1rTG+tyOpWqTku4bRl/fGc6nLruR1V7Ea6IfX4dI7aaUU8t4VWTbH/6ZYKTl1rxrzS0jJZbi jERDLeai4kQAkA/rSeYDAAA= X-CMS-MailID: 20240509173529uscas1p1b6e43b169514d36915cd2bc8aabc4200 CMS-TYPE: 301P X-CMS-RootMailID: 20240509173529uscas1p1b6e43b169514d36915cd2bc8aabc4200 References: X-Stat-Signature: 33h938nztf64bbtmqz6epbhd8wbpg53s X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3E191C000B X-HE-Tag: 1715276132-799794 X-HE-Meta: U2FsdGVkX1/SPBBX5htRmUFT1iqqdgXvCL2zRHZkUHLe7UughLhxnYxlrWE/YSOJjNrAoBTft8T/nMtWt+D7D2ogsARvmOzJhUuwaeok5RHhLVg9JzNQZUEQcFx8ga1ovxTu4STNC1+HLQfGbrukPVQA6e2sCFMzFoWqMzvc5cJGoHeTyAmMuXPM7rWiXUVImOjnEknKmCE3zsulbxw00aJD3FMVBNQVlI42q63i6WjfYCwArKj83xkKslDPiu4y2qQQffJAr/N6cSqa9GwnIvo+AWQZojRMXHHQl40ywNAjdbPWhjWB1GuT3PeHThpBpPSfOe4Icj3ec4KzdY/vqUlhgJ8Vt9b68O8ZuZ13Wj5q97KyVQZ6ZGffbMK+0NUvtuY4dnSyGc5k/+fy3swNjMHFXuVbnvp5RXdt64LQEwBvXRdqoyisA4ESQ4YwIInw2crV6D2bbeQWvEZMjoQxRl+442Su/Num22FIzmJ7mR+q9YbStV+UlujIHjr646GS6rg0xVO93mbUpHVzuxNabLVUnO2ntamddCA9Vkol22nAHXbL8gDVU1yEEaH2HIsnMZKvBwZ//ve/FMdF6Fn4byQt+Xg+hkTZZwv4eOzI5A6cpE6+SA5zO17MggGoW8oOhdBPIMKuhRvF6l0CI7uo9hU3dLk5UN/nmcHAkgYTMdJE+k5BsiZCFhDZorbprIhO4nYm9QqeMrGAo2y2mFClPYMDIrsSDU1MRxLxyRiVp7Qhj4Oah8j2khZkWPwwmpyfKIAVXzVqXXhxgvKNo4vgqavQ7cskR7/k3TBGF5aeaA05CEsyqJ9TnpCMLIoIcfpkllVXi/+Txw1wCYLCQMm4elDE10JrDDykXw2CETOoXf8yMaArY1eJOnV977jeq/royQrDgq/xeijZrD1gQUvfY/YAiPCxxZ+vRVr0W7tylZHMNxhZ0KJMzfQ45zr7EgVM/h0O+08/UXo0cv12lzw pBaVK1/1 K3UKbCytbGhVCgyP8ZB5xYmhzJAT3PPfPE3BI+EFhgXrTTu5rTlnr+KMVi6MhnqZWMWxw0Wj/ohZhXRF9S7nh0duDpw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, May 06, 2024 at 08:37:19PM -0700, David Rientjes wrote: > Hi all, >=20 > I think it would be very worthwhile to have a block set aside for=20 > discussion on locally attached memory tiering extensions at LSF/MM/BPF=20 > 2024. >=20 Agreed. > Primarily interested in discussing Linux enlightenment for CXL 1.1 and=20 > later type-3 memory expansion devices (CXL.mem). I think we could touch= =20 > on CXL 2.0 and later memory pooling architectures if we have time and=20 > there is interest, but the primary focus here would be local attached. Same thought as well, but I tend to decopule CXL specifcation version from= =20 CXL device type. I see the CXL 2.0 feature of hot add/remove as being=20 controversial, but there is not a hard requirement to hot add/remove CXL 2.0 capable devices from systems. Type 3 memory devices are a type of CXL device that can be compatible with different CXL specification versions. What I do like about CXL 2.0 is the push for more OS control of the device and of the CXL hierarchy (HDM decoder programming). IMO the notion of local= ly=20 attached is also not as important as the performance characterics of the li= nk.=20 >=20 > Based on the premise for a Memory Tiering Working Group[1], there is=20 > widespread interest in the foundational topics for generally useful Linux= =20 > enlightenment: >=20 > - Decoupling CPU balancing from memory balancing (or obsoleting CPU > balancing entirely) >=20 > + John Hubbard notes this would be useful for GPUs: >=20 > a) GPUs have their own processors that are invisible to the kernel'= s > NUMA "which tasks are active on which NUMA nodes" calculations, > and >=20 > b) Similar to where CXL is generally going, we have already built > fully memory-coherent hardware, which include memory-only NUMA > nodes. >=20 > - In-kernel hot memory abstraction, informed by hardware hinting drivers > (incl some architectures like Power10), usable as a NUMA Balancing > backend for promotion and other areas of the kernel like transparent > hugepage utilization >=20 > - NUMA and memory tiering enlightenment for accelerators, such as for > optimal use of GPU memory, extremely important for a cloud provider > (hint hint :) >=20 > - Asynchronous memory promotion independent of task_numa_fault() while > considering the cost of page migration (due to identifying cold memory= ) >=20 > - What the role of userspace plays in this decision-making and how we ca= n > extend the default policy and mechanisms in the kernel to allow for it > if necessary >=20 > Additional topics that you find interesting are also very helpful! >=20 > I'm biased toward a generally useful solution that would leverage the=20 > kernel as the ultimate source of truth for page hotness that can be=20 > extended for multiple use caes, one of which is memory tiering support. = =20 > But certainly if there are other approaches, we can discuss that as well. >=20 > A few main goals from this discussion: >=20 > - Ensure that proposals address, or can be extended to address, the=20 > emerging needs of the various use cases that users may have >=20 > - Surface any constraints that stakeholders may find to be prohibitive > for support in the core MM subsystem >=20 > - Alignment and division of work for developers who are actively looking > to contribute to this area Luis has done a great job doing this in the large block effort. If he can join this discussion I think his input would be valuable. >=20 > As I'm just one of many stakeholders for this discussion, I'd nominate=20 > Michal Hocko to moderate it if he's willing to do so. If he's so willing= ,=20 > we'd be in good hands :) >=20 > [1] https://lore.kernel.org/linux-mm/45d850ec-623b-7c07-c266-e948cdbf1f6= 2@linux.com/T/ >=20 > =