From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F5AAC47DDF for ; Thu, 25 Jan 2024 22:28:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94EAF6B0089; Thu, 25 Jan 2024 17:28:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8FE6F6B008C; Thu, 25 Jan 2024 17:28:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79F446B0092; Thu, 25 Jan 2024 17:28:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 667F06B0089 for ; Thu, 25 Jan 2024 17:28:36 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F3E5E1401DE for ; Thu, 25 Jan 2024 22:28:35 +0000 (UTC) X-FDA: 81719273790.29.7C6DD87 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2075.outbound.protection.outlook.com [40.107.244.75]) by imf01.hostedemail.com (Postfix) with ESMTP id 2997540007 for ; Thu, 25 Jan 2024 22:28:32 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=V2qWymzP; spf=pass (imf01.hostedemail.com: domain of gregory.price@memverge.com designates 40.107.244.75 as permitted sender) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=memverge.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1706221713; a=rsa-sha256; cv=pass; b=ZyUVNY2YEV5rcKcGfVDwB/1o6igWvLu6qDJLA6YOLDxxeU+pPhJSUFTeQLG/eI6Qe0+waW jsHMQFIPrtnfCr0Vc83QUHlu+OxDVkOm6+edbIDjEGtwgwz3kZNp9ez8DYXV720oBPc5t5 rfBUokg5Pq1C5rcT73rBzDZbStF5eXw= ARC-Authentication-Results: i=2; imf01.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=V2qWymzP; spf=pass (imf01.hostedemail.com: domain of gregory.price@memverge.com designates 40.107.244.75 as permitted sender) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=memverge.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706221713; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bidxZIqMoszrlJZaWccjT6F9YpzBzOpQXy+ey/dNZew=; b=uxzsb3mh+8SCpQFgn5Pv+Xe2ls5PXrQVtesoFTlHbX5zows2t21WkCwpJQXJBnJZt/jKN5 +21XN2ZoI8OBw9Kj4GnrJYJnTwtrcgPzJR6Jfz/5x/OdJM0xdp7YJQYa8aFerRwhORz35B sJgZMYSywxHJXJjAZ7LBQgQeKvBrYu0= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NUMPv3+WEH8864Hl1gtEAtN4rCgttBt5UA8vfeZChBz7stfw2d+BNoaMIHAg5dRN0Wx7I3gc4CsLBw8/ZK8hreu0hgwDwSLCujLSgZ2SkNIwf19yOVdbGJNi+9ViqkF8tE4uG7GGYUvBb6JB63xYgT4X20EMXxgFIr1r++w9/0UgPt7Hb9wlTUqv0cN4zcjfpzRq5mKBXfndl/gcpcFdlp3TPboWjJozy+YrelRhyJ43E62ZuO1Ej2WUof9MoqN01GdV1mrC6O1WfMBURsHVCqJxnb4HswiasQKsCNC3VmsGq8wl3MIDGfXoIBaomLieEq59lRKJxiXB3IfDyGzkQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bidxZIqMoszrlJZaWccjT6F9YpzBzOpQXy+ey/dNZew=; b=YYmC149HSU8sfkm692aXt+kPfoud32Ya9dUjQGCVxyvdZZ9xVPdT69CsH5PMkqAVdvAgHoBLPdpRaEIjKRk3Jnulfp+HOwXQw/p/NyFggr7iVDGVAHq/bKRq7dG5e+7m3FOVWupijiTCxus05XAaAhh6XNTJSEE8r9tMd+KVg14DzIenoaKrrjLX0D/CCo0o8POvwMVS/ScvMXYlv401aeXHtUD2oCzl5NpC0E+y0QS5+J8I0u2AS42YRpG6aU0S5lG0F8MoGUu7AnpIFxI2d/kn0oDXEXe4P9v6aunG5bGxZuTX72aRGAfTyhNdsjmSYUVIDplOXEJ9Pgg9Bl+6UA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bidxZIqMoszrlJZaWccjT6F9YpzBzOpQXy+ey/dNZew=; b=V2qWymzPyHXqlytllAmBd9JtgbBbsid19CJ6EK70wJlrJKAWy7taCKyvMMJvu4HNqmGPS19MqKXVtnmmDgpwlJZyt5QaZk0I8k9r7rNgqLYe3YvTto0tnn2EUI1Nh6XKDlJ2LnFaVX7jo2ynTrEM565+ihglbOxMkJA86xP+I6o= Received: from SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) by BY3PR17MB5188.namprd17.prod.outlook.com (2603:10b6:a03:3b7::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.22; Thu, 25 Jan 2024 22:28:28 +0000 Received: from SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::7a04:dc86:2799:2f15]) by SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::7a04:dc86:2799:2f15%5]) with mapi id 15.20.7228.023; Thu, 25 Jan 2024 22:28:28 +0000 Date: Thu, 25 Jan 2024 17:28:19 -0500 From: Gregory Price To: David Rientjes Cc: Matthew Wilcox , John Hubbard , Zi Yan , Bharata B Rao , Dave Jiang , "Aneesh Kumar K.V" , "Huang, Ying" , Alistair Popple , Christoph Lameter , Andrew Morton , Linus Torvalds , Dave Hansen , Mel Gorman , Jon Grimm , Gregory Price , Brian Morris , Wei Xu , Johannes Weiner , SeongJae Park , linux-mm@kvack.org Subject: Re: [RFC] Memory tiering kernel alignment Message-ID: References: <75f21150-1e12-4f4b-e578-e170e4fea18b@google.com> <2b29dd3d-bb2c-6a8c-94d2-d5c2e035516a@google.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: SJ0PR13CA0013.namprd13.prod.outlook.com (2603:10b6:a03:2c0::18) To SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR17MB5512:EE_|BY3PR17MB5188:EE_ X-MS-Office365-Filtering-Correlation-Id: 64e0da73-f776-4f54-0a03-08dc1df4f40a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: If6RxLd3cpBaM0ooRzUjHExlLLJZwe4pAul+m9eWvzNlqaIctyXRqmWBpqINFjmHZWlmfyy46H88Gp8V94Rx8kB+Rjg8QrUk1npiMF0AZcMuwMjcGQKO8I58vXPXpCQtsvmxqSkxXiGD8DiF0o9KBHLCSkPOiRMeJSsCoWrRjAqJN5TsdkQOVlWbZy9N01QL7HrwaIGQLaGlwe44e5gakKu6bCR4WKXIY+8ZgYmXNWRhe9IyiU5f0/XHlRZTSIfgCZl7stc5boGQCxcV+Wle01Tt57IcL4qga1SCOgSK4YafoP4pSDQ8IvPhNiWBXUgh2syFJ7wx3KwENmk3zqJSrN1dxzfGGHAonGdYl7cOkecWs/GpEUPItDUUicFAJaC91sciAGisrUCQMM1XslhT8XWqq7KckoYPzPnlxPwhHCdZF7XWnfqX+OC3iwU6xK+7sBcg6jDq4BthnKfeuDTdtrIxYUmCL3YGSPx8mUi2x0gwV6DWjHLtWY46lzb9/HkwqsxVc8fcsj5NG1OrmnX9+Iib9h9MbvUPxgpRXRqBdIbt8QY1E1IHcj8pujrd3H/Dkv3GXQ7Obex1oFU3tD0hpjnS2T23L3OXJEBVTTlZGDA= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR17MB5512.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(396003)(376002)(366004)(346002)(39840400004)(230922051799003)(451199024)(64100799003)(1800799012)(186009)(66899024)(2906002)(38100700002)(6916009)(86362001)(36756003)(66556008)(41300700001)(66946007)(6506007)(6486002)(54906003)(478600001)(2616005)(6512007)(8936002)(44832011)(7416002)(66476007)(5660300002)(4326008)(6666004)(8676002)(316002)(26005)(67856001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?2id5Ag+NmvSoiZbJozYcAHJcKgUpRJNvQ0ISdwAeJ0sAuGUaUr+kE6fuZepx?= =?us-ascii?Q?L/z2bMQWNkX4k4ziUg1U82s6IkZ+Z+LwDLV452ChVXHF7s+m0YG2Lt8nTJbz?= =?us-ascii?Q?i57qbgYboPSRaBpBcXtB1FmkR66tQNXmiy65zWayKn1b72hTmcpuzS7RjI43?= =?us-ascii?Q?fm7QoMmurEN+JYB2cmlZJE1WquSPFyQQJqDiolEqINDEtF9U1V1imfclYcWW?= =?us-ascii?Q?SIYRtd4P1Z8V2Seu1XmEYvHEi/NTOa1BZMJI9CM/CMI0ulkJND55JxU9XiB5?= =?us-ascii?Q?uiKo3FAUXQpFmeB07ItjZgE5XTCfQ7T1vXGbrZJK08taTIh0LCzJ9C5+awx7?= =?us-ascii?Q?+geAV/i+0pefgHAjn9sK37c21JDv1HyyySvHjIrmMZLACA2W+OG6ZE1uVkf6?= =?us-ascii?Q?v+/esA3xVnbV7Ik2IgVxG2hf3+YN1/xaDsuxU94B+AnUtYuv5IrxlOhpqYWU?= =?us-ascii?Q?Zi/1mpWxJ+OOclM2JvKARwNq1z2744V4wMhYSC5DYYv26QEbz/bwzcJ6sy3i?= =?us-ascii?Q?oqq/KZsSghneh6epe+RUew9SsYq54AAy5xmjDtOiLRJnkzvqLEwvPYU+vqHU?= =?us-ascii?Q?n2ud/xWbmQbxr7xbpSgFNwZJUqEcMrTEZdKjU9qQHRNeSTvycg1uQyxoinVf?= =?us-ascii?Q?MbJgs+U9GzqSRUEd0jvBL9TacCPbeX1uo+kXqewwqVHOS5O9YdnLp2b9VthZ?= =?us-ascii?Q?J8NPQ00vb3OqYjBisssa2OyLgV0iZqg7BqBrRXvp+3ukdqQWHNW9nclqrJgW?= =?us-ascii?Q?2TBZEjyHCXv5MOVGNA6TIkMhzkvmmBqOXjSvBvjRFYbzLQtWJbsob58pccAo?= =?us-ascii?Q?xaxZlKKJ58ItfswDa59RzDpy2GpTxuN8+lPA/Sg15/S72YvBhu8ohNOw/uUa?= =?us-ascii?Q?QtYotKUQFtiDv+Ee4YPXaPjqrbYsJj6WRAqUhtM1FCYwaC7e7NDubbsDwOb+?= =?us-ascii?Q?1k0wLgLTYX4cWn7LzEJ/HN9/69z8Z7k2qxyQPCW33qJgWuuXmiOAancemut/?= =?us-ascii?Q?ZvDyVz1nm6cpGKidvF4HOJ47fkjtkQCfBWgoO/6ohbtYvex94iD3Clntwj2Q?= =?us-ascii?Q?OG+MGU+4RshgL5L5theYPcxORKTdGZMN8C6OvrdilHJoX7OYKxLLHHjUD84S?= =?us-ascii?Q?xxqZRP8rLwE+sMuvxAbvXzLduHxsQ6wGILP4MO+LtrBLzPMHg0Dvf88XSbbH?= =?us-ascii?Q?Q/Vg4efRa4O9gMtVqJsi0beOZhzOQHoeicXxqbxErZn6Tdw+itVq2qVqorqq?= =?us-ascii?Q?kxrT3K93BQH6DmeXax7kpz4ti1TBKVcFFmusE21jD3KyOj5F1hgJwFqKEMss?= =?us-ascii?Q?PyXZa0Bcqr5UxSGXvyAS5Emlsbi7fU/Rdkx3PeNra5sYHgJXZGELXK2NL7JG?= =?us-ascii?Q?IPxLVFQO39YJFAbsRWC2htNy6GsWyZm2Xr7ZBPGGs1Evcd8WvAXJ23g0D2xa?= =?us-ascii?Q?GdFzcL3npFjbkqi0Us/FtaF3TUDAlVSEUFu0yTDFNDbReJ6oopGOSIYcXzsn?= =?us-ascii?Q?KuQ0qzs70VnPeFD2vCXju/ScILp2si9fpbolkUG4j9GRCQEsVVF0Y5r7MN7m?= =?us-ascii?Q?1Ri8iLbEeasauf9Yd/P/dujMlVxrO5EN++Q1Q3wnw3y0j33jSIbkc8Gf5Gz5?= =?us-ascii?Q?ug=3D=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: 64e0da73-f776-4f54-0a03-08dc1df4f40a X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2024 22:28:28.5062 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6J2Rl/07/52+7stpwDiZpiDjYd1WFgkoo6PyTSbs0nVMQhbr8Wrf/c2psATKaYq+jBI0LhjOSqs4nHzqxxLbnNcyKENi3QTd9VqALLnVimU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY3PR17MB5188 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2997540007 X-Stat-Signature: cb5yskskaha44ayi7x1sz4qzi7wakmyd X-Rspam-User: X-HE-Tag: 1706221712-603734 X-HE-Meta: U2FsdGVkX19QQDDQgEze1llnIOBqoD9OjTmbTLND3rfpzxZA7IYKha4U9Zezx+LEtUhipgVz5gO8gCEgpvPoZCnSyogtgXEgij0PDzJbHv11PJouS4okEg6A7I0DkPh33zGIfGmQbDHOh7QENpImLC0ct7+xFF/BHvEMHGlyX4ljuNMA3rcTfgYae6yFkD88smR6jhQqOwgoyGcnzVlL0o9hPUhql2C0xkz94Coqm7LCBmEzhT0ysUMaI5P5RJloYCBnDsziBr1WkcxmoBmeh1mGz+bFMsUuaBRngJ8kiOcLAH4mbN4NGf+2er7xn48M/MWVlqs/qlKISqhVT7I2T6rFhc161Gx2cEwbNM+acJhKKH+o/gCwhLXwBde5LwymIwRb0HDC7mWEuQk+/SbL5EVowW4SMmHmDzye4xiDVngESHnCxbQUXdH2M+Tq2BlhJlIBn9yYAE3lKzdyKWr5Fl87Cm/rrYS8VQivnNyboUYgxccDflVWnz9U++RoVaRiTtxkL/uHKvlmHF5cnBxeU97NJQ7TJZnOSEIR8F5lHKv/8r64M+XSPyYsyyNZs9ppIBZgGuS5X6w5iMZhHbNVULHXebwQvzikUkgKtYN/bnuCBCD8Ypaq69YROTD8XInzyFDlwEVcPYwNONlscMuTC4Loy4hlf7eiXXN1u39JMdH303lPk0ABQqoewDGvrJ7vwuLC8eXWSm4dKlivVCmfRxw9MKU7ciI2M7fNGusPqELyLh/9AEx3Rv9daCI3BboVAGDqgkbfEaZJGsrSvh26kA4GRe9nbmCI2ZjHoN7q+KsiXeUYBiupztru6oLHEBKhtzik0VZSbY5s5MHFYZ3jWnk8fQZq1BVL/q5UlebDvxdhmMfFTeCALg4guelMqk6yFTEycZzx1J5I+kZzMOPs2moTHHYSS4S33rFzllpa16gepVFJOwrjF2PIVNL0/SCUefqliX3c1LY+D+c2Mrw wBprkhEO ciQi2cwVFF3eEaboL7Djnxw545p/SjNRqM/uGcH1PZp1nBLcuKyqyJmU+m8guOqic5l8r7i3bqY9NocoSvNDM5x2Iz+XD+gIf+hVvQ8YQMhJOPwAo8wxtO7sJ3MDlmRvZEJ4bXJhRWvK5026gD6XrqL3ckDD5XKQF7Z34dJudl7I3GBEE2O9mKVIbjR2RSnC6gK8obOWZMJ4+/1e2qt8eUkO5ILIoN93tPc5rn5JJ6pCa2JU0fbR3Z/O021mU/UupWt479ZL3P46WUR0TKWvumoha8vQm73JlQj3H X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Jan 25, 2024 at 01:37:02PM -0800, David Rientjes wrote: > > On Thu, 25 Jan 2024, Matthew Wilcox wrote: > > > > That's a huge relief. I was not looking forward to the patches to add > > support for pooling (etc). > > > > Using CXL as cold-data-storage makes a certain amount of sense, although > > I'm not really sure why it offers an advantage over NAND. It's faster > > than NAND, but you still want to bring it back locally before operating > > on it. NAND is denser, and consumes less power while idle. NAND comes > > > > This is **exactly** the type of discussion we're looking to have :) > > There are some things that I've chatted informally with folks about that > I'd like to bring to the forum: > [...snip...] Just going to toss in that tiering from a latency-only perspective is also too narrow a focus. We should also consider bandwidth. Saturating a channel means increased latencies - but the problem can be alleviated by moving hot data *off* of heavily contended devices, even if they are farther away. In a best case scenario, the addition of local CXL devices can bring bandwidth-saturated DRAM back down into the "maximum sustainable" (around ~70% of max), reducing average latencies and increasing CPU utilization. We've been finding there are a non-trivial number of workloads that benefit more from distributing their hot data across their available bandwidth than they do from just jamming it all on the local socket. ~Gregory