From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C900C48BF6 for ; Thu, 29 Feb 2024 04:01:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B3E866B009B; Wed, 28 Feb 2024 23:01:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AEE366B009C; Wed, 28 Feb 2024 23:01:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9690D6B009D; Wed, 28 Feb 2024 23:01:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 83B0C6B009B for ; Wed, 28 Feb 2024 23:01:19 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4A9941A0CA6 for ; Thu, 29 Feb 2024 04:01:19 +0000 (UTC) X-FDA: 81843491478.16.623FBE3 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2042.outbound.protection.outlook.com [40.107.94.42]) by imf02.hostedemail.com (Postfix) with ESMTP id 6306280016 for ; Thu, 29 Feb 2024 04:01:16 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=V5PEiQVU; spf=pass (imf02.hostedemail.com: domain of bharata@amd.com designates 40.107.94.42 as permitted sender) smtp.mailfrom=bharata@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709179276; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sk7fWm9Z1jMORYHnWPLnI8Ywv/M+EAGBDVK0obnTGmA=; b=NaXshLPpLr0Hvb8SSvemoP6WsXhkLLyNuPICw/APXM2SgVDmsKN0lxFTAyrmh1AOGSJQ55 d3Tr6zA8jbVn+Q8F7OK2CuyPu4ozvL/OWrClcurxSLPTcNfRPDy/D3YWRD1tD5RnjgVuL+ nKPb6f2qqxD29506GZPNrVBZhKqp7Yg= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1709179276; a=rsa-sha256; cv=pass; b=W0cyqwvh+brSsZmuSF0+wFoXjo1veMLRAUZvl9l98YsQFMbA7iOtAJVmo9b2c38Uw+Zweo VbYsrcbsXDQmdYGl7v31CtEiYqVdMmQl6Y+J2p+ss4DgVKVSkp17+Sx0xlYhLlG+1MPRv8 wjl2BwHiwJ7aIYAcRils+NnA4UJuoqg= ARC-Authentication-Results: i=2; imf02.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=V5PEiQVU; spf=pass (imf02.hostedemail.com: domain of bharata@amd.com designates 40.107.94.42 as permitted sender) smtp.mailfrom=bharata@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FfU61j2rBGxOA9QkLgnXKFrIFx+F9qGejC6bT0lv153BMh4Pku/sntibA5bnD/lZ5F/3TeBVcaKpx0lxAv9OdlyejFky3I9KXNoUbZMg64YTGeLvfzUY8j8a97WDENv7VQ1j6CYNXXHIosbFkWDvg5nO+Bn5oWOQAa3Yna58gpSAbIquWuzcV896VDXNVtqF5pn14aCV+YWstjFNMrDA3IGjxUobJKk5IZ1WEqA7Csp3NweE0ghNDtIlRZsRLSPxW9OcNhN8cZNMN0qNtPwjIZkY+g2CSFPU/sQcUNqF7T7soSLnVIjIWh3K1ni43wH+Kr+p/OzotfykRHesscOBvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sk7fWm9Z1jMORYHnWPLnI8Ywv/M+EAGBDVK0obnTGmA=; b=cLG/MzU9emYkA8XEzi5bjd+R7xVmRcGnXu7H5w9Zh+0vzMVoF0KpIAOoG7+16oKNr10rST7OgTcRo5bCCiVZl4+gHYQPISBds1SBzHia89lXpzamj7QcCTLVSucJYtDF7ima4OwiV6AuZP9MVUkpXhHl17VIVQwDumcsQH4VkTGpLJPChhqo/DHn0CPChDseUBVdxWvN6RjpcGaMbrflyWFolXnkm6e/Pd5t8yBblIaPD6Im4pF7tJ3CfXCaugNxyWOVR/phcAGkNm7Ul/tT6qPbXQ6Afvir1gqXwXDpVe189EXqFMnDnhM0fLMK6wpDiFZTkOTWDCB6S0jL0UwZdw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sk7fWm9Z1jMORYHnWPLnI8Ywv/M+EAGBDVK0obnTGmA=; b=V5PEiQVUWWj1CAzkOu0qgOftjJhFB1Iu/9V3VTln4NBjGhEcKg0xGGl/g2Nk7GurhbJQ0sDZAn7ETQjxa9/UQuEE3UGEndsn87YzV5HvUdF6DZS0kvj79uxC+M3jAXQSBvAKDTSXlwTrO9OJbOmZVtBPuwZ/Ci9B28rlu6le6gM= Received: from IA1PR12MB6434.namprd12.prod.outlook.com (2603:10b6:208:3ae::10) by BY5PR12MB4308.namprd12.prod.outlook.com (2603:10b6:a03:20a::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.41; Thu, 29 Feb 2024 04:01:13 +0000 Received: from IA1PR12MB6434.namprd12.prod.outlook.com ([fe80::a628:d2dc:a0e9:67e2]) by IA1PR12MB6434.namprd12.prod.outlook.com ([fe80::a628:d2dc:a0e9:67e2%5]) with mapi id 15.20.7316.035; Thu, 29 Feb 2024 04:01:12 +0000 Message-ID: <781b3a7a-082d-46c1-a31a-1484c62123ac@amd.com> Date: Thu, 29 Feb 2024 09:31:02 +0530 User-Agent: Mozilla Thunderbird Subject: Re: [RFC] Memory tiering kernel alignment Content-Language: en-US To: David Rientjes , John Hubbard , Zi Yan , Dave Jiang , "Aneesh Kumar K.V" , "Huang, Ying" , Alistair Popple , Christoph Lameter , Andrew Morton , Linus Torvalds , Dave Hansen , Mel Gorman , Jon Grimm , Gregory Price , Brian Morris , Wei Xu , Johannes Weiner , linux-mm@kvack.org, Adam Manzanares References: <75f21150-1e12-4f4b-e578-e170e4fea18b@google.com> From: Bharata B Rao In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: PN3PR01CA0192.INDPRD01.PROD.OUTLOOK.COM (2603:1096:c01:be::16) To IA1PR12MB6434.namprd12.prod.outlook.com (2603:10b6:208:3ae::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: IA1PR12MB6434:EE_|BY5PR12MB4308:EE_ X-MS-Office365-Filtering-Correlation-Id: 0ccbe8ac-f705-4bdc-ae1e-08dc38db11a1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3sTZqH5uXKFr+juF2bcrGnhMAscAl6RVeB+mmROgKjImT8MbglqtccraNgZNraYQfZHuOS6CwBtN6PXsM/ikUvAfrthTw//vlRHbLP23xjGepew5AlqCJLeRT+fhhpE9GQwDID8kAZtEV53WU2+xoK6DHITYi0ytjNK6+W5H3PAiDB2slt9FNPwDgo5rb1jkKGpmQr93IV1hYdmUrg/xergbYdTwJEuf6bmYC9Tey/DYGo0zbcB4zVrk1hmZJu1gBfJTQpzhRBE9TBOFYnSCPqS3gIfd/Yp8KyU1/hQDekFG2qmVNT7waumtPVWwx9O0RrMTrSfsk2H77FMwvzdjFMxE21rUMyvBFLoqwo3mFldHxv/v3z50E4pF9av0GCA+UB9cnRJwle491/Oj0OyFnGbruaTaJyuwnyKDks7fJkfR4FUYvsFKPwAFB6MU2vAeZsYF8s6D6NfxPxD7kYs2nh3temCieZ/H0P2NhNVpOiRHceyKS1VoCNOiwJM6L9F7lYo5K5suAlkPY+9PZ93sX7hqPgPpy7aSEALHYFAWtGHPEpLjWfN0olTLCmYviOXdeZ8wjfqyTRmekAvLHkGeTHjQAc8tQJfrj/YOapDZ4GjzBoVU74z4son3iDNheClJxHVkCRfK0l9//BLCew8Zl8vs0bHOeUM67SYu/Dh0sr2vFA51hy3acm9PZVPEBn7XjAI11TbdhxetLNU6EI4aQw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:IA1PR12MB6434.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(921011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YWZ6d1hOUm1tNk1SMm1hUC9xanZUVnJtYkVlZ3c1VWV1YU9wbTVnbXhiMDFm?= =?utf-8?B?RmhnZVdhUXBhL2xPQ0ZJb2lTTHJzVUkvb1FpNzBVZ2dFTDQvMzh4TXdvSEdI?= =?utf-8?B?bmhIUmRkbk9RNE1lSVl1UFQyNEI3S1pzcTV2MCt4L2FZWm1aVDZFSE5hc3hM?= =?utf-8?B?YkZzdzlsR1VwY3JHRkE2eFlzSG1MSDk1OVp3SjRocSsvbjR1L1gzcXZXUG8z?= =?utf-8?B?M0lnZ0pOOUNaOHhWcENtOURkRWF6RDVicFN6VFV0bEt0bWZTNkJ2dFZ3TDkv?= =?utf-8?B?RE5TZ1VBNGxVNG53MXU0VXhZczc3SXNiS2w3M1liZ3I0NmQ3Q2FTdWN6OTQz?= =?utf-8?B?SXU5Y044a095VnhYa1hSWXM4UFYzb2wxWW1TK0NLc0hCdHZVR0M3bUN2QTZS?= =?utf-8?B?VmJBOFdOQWxJTlJVeDk5Z0FRZU9GdmVnemNkYmd5VDdpS0FqRGgzRTNTcS9I?= =?utf-8?B?V2pmRWRsbUxDN3ZJYnA5aHZxNDJRNEtkUzdmL0Z1RWlneTJFTllyYmMxcjFB?= =?utf-8?B?a3hEbVNTcVYrV09NdjJHY1E1bGZJaW0yZ0ppWGlMTjFJTDEzdlRMbVppT0kz?= =?utf-8?B?WjUydGpHVk15QnlQUDZQTkFRWDRZQ1ZEVTYwUVRkMFgzdHhBZktTYWtKVEdN?= =?utf-8?B?NkZGRFNLOGxTQkkyNFJGQW96N2dFRUdnV1NtYS9TSU43bjdGRGlmaEcveWl1?= =?utf-8?B?dXZ2djdWMHF0Zzc2S1FyRkhMa0JNWkNqSHE2MU42R3FrcUZiN2c3d21lRmJv?= =?utf-8?B?dDFGQXJmOUxRTmQwRjJIUzUrak1Ebjd5cFpZK204U0QyYTFrL2QxMmFpVWt5?= =?utf-8?B?WHlxL25ZajQ5dS95SHRYNDROZGRtcWRqL1pQSnJrR3JSNFNVdnVaN1lkSFhT?= =?utf-8?B?cXBQOUlWVm90TVB4WHdCNjVmVlRJNUdZRUwvUzFaSk1MQVIzRWVHR0lZWDZo?= =?utf-8?B?MndqbElINWJsbkV1N3lRaUpFUnJJcHkvc1pUejljSXhyMDhrb0tRMm1nRWNt?= =?utf-8?B?Mjl2azlyUEczcUpKQmFkZ043SXlvekRsdmVYR093eUx3RXRxYkpMNGljSG1Y?= =?utf-8?B?L3c5N0o2R2c0K3lwOWxZbTY1Y2J5dkFaYXBMQVozMklKQWo1TTV5VW5zMGFZ?= =?utf-8?B?VVpSdUp2V3h3Y2gzVGxIenN2RVZDbERpdXFEQklEUlVkclFBWmdqRUVuYW1t?= =?utf-8?B?TVdsYU55Ymo5Tnl5R0tCMnh5aDJWamxXVWtCMmlva0EvRDNTa2V5RnY4VzJ6?= =?utf-8?B?QnozMGpDc3k0KzVPM1p2VzNpRGNIYStmMXBsaGFGa2l3dENzTkRIbllzMjAr?= =?utf-8?B?ZGdBRE1nTVEwOUx5YU13bDNoMng2TXBabTgwcnBrRXlyOEVUclBWUXFDSEFJ?= =?utf-8?B?L3BKbm85cklnT3M2MVJnRTZia3BOYWNnM2p1MHV2ODBjZC9IaHl3M2QrYVhK?= =?utf-8?B?S2d1c2ZZUm42Q0lQblVNbnl6M2I4SnJDUVhVS3d1N1lTSm5ZZ0FvQzJqNW9i?= =?utf-8?B?ZlpuTEsrSWVwNFBVMlRId3ZNUmJ0MUhDUkp1KzRaU0VqWG9mcHlKYW1zQUJw?= =?utf-8?B?VG8xM01EaklINzM2ME1vaWpvcUkvVHRzUmphMzVTQXpRbWJFR3RnQmkwZ1Ra?= =?utf-8?B?SnZUV0dCUEhlMjE2SUFJdEpEd2k4STNxcG1qOHc1U0VpSEhIRGRrRzI3RVhQ?= =?utf-8?B?cm1xNldNclZJc0pkZHExZTZQdG1HU01IOURvb0MzSCtzSTlSWFlHZExPQUJV?= =?utf-8?B?MGN2ZTc1Nml4azR2TjVYb0xPYmt6M1E5Q09TMnBPRmszb0lSNmtYbUJreHY1?= =?utf-8?B?YzZ6Z21jMmp2cGFjZTN6UE53MXVrQmtua0YwQzNCdXZJbEZkOEpxR1dFcDFv?= =?utf-8?B?c0xva0VXc2FaaXovNERQZ2J5NDEyTWNDQVFSbzhVdUhFT2NFZnhiZWF1WUc5?= =?utf-8?B?bW1Gd01YbmQybzNXdDA2T3NEeE0vSXgrTTgzVE5idW54RjlKWFF1dnFwVmN2?= =?utf-8?B?WVhpbVlPdDNKUndOZHo5SFpwdmduT3BjZFkvYmRkcFIwNm4xVDJQa2UrV3lN?= =?utf-8?B?Y1ROeUdqa0JBNk5xTXhOWHdtdlczVEhROTJnRFpsOTJ3dmc1YmJHL0MvVWZ6?= =?utf-8?Q?0enEGJMuqGM08U7cF7xXGiyy8?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0ccbe8ac-f705-4bdc-ae1e-08dc38db11a1 X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6434.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Feb 2024 04:01:12.7809 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: o96higXOg8o9CeXB5KeeRcRi/slhPprpP0ckecXfCfT5ERplX5mFXu6gepeMU3FTww+5Gqeb0lF0Uv2RNs0OCA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4308 X-Rspamd-Queue-Id: 6306280016 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: k11wg6ps4nfpnyxo61jzayodbt6muu6m X-HE-Tag: 1709179276-990512 X-HE-Meta: U2FsdGVkX18ym1C3d4S5kqB0Z1qhJzZnUUb02iNWqUEUxkO2+qKNCY7gJMtUj5h85gTzRjPTtAaowgGAg2wC74nqX0KuxUBKGihrDHAe5inU26xlTFs0KwMl/hxukuPB9DF+FfIP/C+gTYMjJNxOmqs0J88wOovuPl/CRutRXxMolp1Oig+4UoKiIBApnVp9E7AldOXDk/KoGJbk/wF5BH2ZdRSzYi6MEdFH1Mv+S7+5rF86tDfpXy4hlRLE+uv4AYW2y1U4BI3BhBC5rT8bkBsU1WHmNHQVh7tYtwbEFIk4dFipmdAWOj271KiA1BavGuQCOlRadLbo4auLHzIJhDIZQuyHKBUb0zFnXK9WKxvaZRWagf4EQgHd7E0gkSuPX9e5OybA+35WyFLdaThNdty4R0AJhK2vTn6r2f6IFlgZKVF5ZcXwzcK2oe0BDH5uX/Pbt9pG03jglCfOxi+sLHSUO89EvYOG24S17QMZqMaLNAvm1tQB9ivpwt1VpRmFxTJU0hTxZtbcS7pAbnEYQ7jIqxHnnNqgTyvO9t2Z2lkL8EUdUU2ReI1wYt3ReXMBMnp2+kq/6JJXTaPnwX6Lf9b6rmT8PqxewjMeBFFGPNxDJ+CJHxhKBHHb/SxWf/X6xJwaN253BycJRep0V3kjQ7VaphwIOfuNYoieaAb9HQYfmJwsXqwIUugQsMfQBt4MJzqOcq120Cq24InLJmY6vkAzErrwBfKGp4FxqF7TJZt8c7mFIKZCo91ZQZaDKDLstO0FQtzCdzYxLCjy/l2EhvTCAGfq8c25ivwpDHqV4ND8XMDspo3PZM32Qr1HS3A+P51wkLrG9b/+u3seaTU9zJSSOmx508fK/ZT1sURGUjfA2eBWLRy//dkFfpWgTQdbKVQrOa3xQGjwm9xO2ogTfVEChiiJn28SghZ8BshLog0u8cc7qNsg+tuNAkWGyFRi/E9uQ5l/s0BCn0NvY4C ahQKXXUN Orp5Af6Saz521rzzeDEJoFwbghmdywspGn97BUMlWdx7cUEUIZXYydjomZu7sDrBZgOXs63xRZco+cKc+UYStnjYMV4l23VhqKU8GEqwkHB9XMHm2zmhwiE7F2LMMZvZBMBfoCzC4RAGljw/7cpgLhE9bqfDK8AHt43PemGdwS3tXr6MmEdsf57nR/aLLJnKRGgmS+A5OkFZNxvMjfS3t23N399Inf+EpnTLKt6eH/Ngr7yPCA2TiucjdAFFply1SPaXF7UtWPL7x0JGU5GX1MqPeT+dKH6MlaZceF6pQlegHqnemlAi9nqpGM5ueyDaK1Lj9AxSVxGC+df0FeQ6lJ3XxUw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 29-Feb-24 7:34 AM, Davidlohr Bueso wrote: > On Thu, 25 Jan 2024, David Rientjes wrote: > >> Some recent discussions have proven that there is widespread interest in >> some very foundational topics for this technology such as: >> >> - Decoupling CPU balancing from memory balancing (or obsoleting CPU >>   balancing entirely) >> >>   + John Hubbard notes this would be useful for GPUs: >> >>      a) GPUs have their own processors that are invisible to the kernel's >>         NUMA "which tasks are active on which NUMA nodes" calculations, >>         and >> >>      b) Similar to where CXL is generally going, we have already built >>         fully memory-coherent hardware, which include memory-only NUMA >>         nodes. >> >> - In-kernel hot memory abstraction, informed by hardware hinting drivers >>   (incl some architectures like Power10), usable as a NUMA Balancing >>   backend for promotion and other areas of the kernel like transparent >>   hugepage utilization > > Regarding the hardware counters, can/will CPU vendors provide something > better for what is currently there for PEBS/IBS - which needs a lot of > stat crunching to make it useful for hot page detection. IBS works independent of PMCs and reports useful information like the virtual and physical address of the access, precise IP, Data Source info (like cache, DRAM, External memory/CXL etc), remote node indication etc. Hence it doesn't really need stat crunching. However it captures and reports access information based on sampling and I have seen that the best sampling interval isn't always good enough to match the number of accesses captured by software based mechanism like NUMA balancing. Regards, Bharata.