From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90B0EC47079 for ; Tue, 9 Jan 2024 17:34:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E9F26B007D; Tue, 9 Jan 2024 12:34:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 09B0D6B0081; Tue, 9 Jan 2024 12:34:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E559B6B0083; Tue, 9 Jan 2024 12:34:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D04DE6B007D for ; Tue, 9 Jan 2024 12:34:36 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AB57A1A0330 for ; Tue, 9 Jan 2024 17:34:36 +0000 (UTC) X-FDA: 81660472152.02.A7B9E34 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2047.outbound.protection.outlook.com [40.107.223.47]) by imf27.hostedemail.com (Postfix) with ESMTP id 1108A4001B for ; Tue, 9 Jan 2024 17:34:33 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=UL4FxFcw; dmarc=pass (policy=none) header.from=memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf27.hostedemail.com: domain of gregory.price@memverge.com designates 40.107.223.47 as permitted sender) smtp.mailfrom=gregory.price@memverge.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704821674; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MjOuvY7UBb25gmCYkb2hCJsWlb7DInLEbKRdAfFxVqw=; b=IdKChLYmu+Ju18i4BYgk1OM2RYqRjwBKTP6fivQ7GwN281/iAKreGO8SsLpb1rZdFcYWQ0 YWI1yrhsYgztWvK/7rK5xLb4/AwYoRXh2iEQQZgSJm2q1FPwvKQBK08YPog3G2Cy4Fg/DE ErzUVKlX1OXhm+n7ZDiPwEPllUNq2E8= ARC-Authentication-Results: i=2; imf27.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=UL4FxFcw; dmarc=pass (policy=none) header.from=memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf27.hostedemail.com: domain of gregory.price@memverge.com designates 40.107.223.47 as permitted sender) smtp.mailfrom=gregory.price@memverge.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1704821674; a=rsa-sha256; cv=pass; b=yDGvV2jP2fwKKkezrvBX6VoTGSYxN30ohMYrVUmCj2MhGeZYtfD2FcBSJdYGd4E+0vxmhr q0RGpkzFLivxkfCmSAk45EB6DK/tGz/I6KfWJy5rcUVQdgk9KKBn3BiD1uV1vmxrUlpo9Q EzrmbTJ/HFCVR8k5s7r6q9Hp1tAdPnw= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=POYMTdYb8hFknrjVmvPlX6puFWQrQxG1AA5X/Hq878aRrV8FHsgNueG7BQlkRGmnCrAckTqFqygYE5RZeYtftN3xG7Jq3Qu2KHEAp6bLHWiO5oXWhaNapQ59ldtiabP3O34zsyfYjF4mKSpMe7hITEJUgKPqi7se7N9rUgctWu2QHaggSoQM/bIvfnir4AyV21K/LY5W35NNI35hsH7U8vReSMCRZ6q1DMLXMibqC5TMQgM3apjjzVSh0sphNyqmC4S3Hq+ngCEMkrFVoOwIZ78IvWNbgP7uTLVGrNot/DTsgRPDtsLTW544CSlvmPFTyTRshFjQb4hwIdOeC+8jIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MjOuvY7UBb25gmCYkb2hCJsWlb7DInLEbKRdAfFxVqw=; b=FWdwrER5qXiUPad/uX89h62EtuNZroVxm/XiOZU2d6YBuP42Yyq606rWkJzOufBq3sr1MOPCLgTTYpFR3s/QlPO+o+kUOWMr9Ky4sEMDP5NOIF0FfvouktWtBzIIIRLFX3fi+VpW73unuGK3KvFTSHe0ugsIrz4hHdUVg/+gDM9uulmVn5WT6soWEv1Q++7h2vsfDwTToX3MgVKgJT8kE3ZJHmkmq/7bNqBCF3W1NDwRCSV3fOzsJvpdrz5EOtv0nISmPjF6pErf4/DCjg1PIoCwQL+86apSdo6jwYQiO2iWbjsQFaPNenG2PTboIoVuRwqh7e5/6bkVM5ElTFZJAQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MjOuvY7UBb25gmCYkb2hCJsWlb7DInLEbKRdAfFxVqw=; b=UL4FxFcw6JUsh4sR+i4GSe6gUiWFJtizBt9TUof+6Csb8iTHmdZFtzNyYfulPPtACuSEMjzixYY0MbooBwN1gXCiS2UdPh7p8QZMVcWPiXAwJ+gMKZ4B8v7wSwiRlhrdp36jZoYSpBrgjT1foZitmrj+7dPoj7u9AB0gOTt0QLE= Received: from SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) by SJ0PR17MB4631.namprd17.prod.outlook.com (2603:10b6:a03:374::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7159.23; Tue, 9 Jan 2024 17:34:28 +0000 Received: from SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::7a04:dc86:2799:2f15]) by SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::7a04:dc86:2799:2f15%5]) with mapi id 15.20.7159.020; Tue, 9 Jan 2024 17:34:28 +0000 Date: Tue, 9 Jan 2024 12:34:24 -0500 From: Gregory Price To: "Huang, Ying" Cc: Srinivasulu Thanneeru , Srinivasulu Opensrc , "linux-cxl@vger.kernel.org" , "linux-mm@kvack.org" , "aneesh.kumar@linux.ibm.com" , "dan.j.williams@intel.com" , "mhocko@suse.com" , "tj@kernel.org" , "john@jagalactic.com" , Eishan Mirakhur , Vinicius Tavares Petrucci , Ravis OpenSrc , "Jonathan.Cameron@huawei.com" , "linux-kernel@vger.kernel.org" , Johannes Weiner , Wei Xu , Hao Xiang , "Ho-Ren (Jack) Chuang" Subject: Re: [EXT] Re: [RFC PATCH v2 0/2] Node migration between memory tiers Message-ID: References: <87fs00njft.fsf@yhuang6-desk2.ccr.corp.intel.com> <87edezc5l1.fsf@yhuang6-desk2.ccr.corp.intel.com> <87a5pmddl5.fsf@yhuang6-desk2.ccr.corp.intel.com> <87wmspbpma.fsf@yhuang6-desk2.ccr.corp.intel.com> <87o7dv897s.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87o7dv897s.fsf@yhuang6-desk2.ccr.corp.intel.com> X-ClientProxiedBy: BYAPR08CA0005.namprd08.prod.outlook.com (2603:10b6:a03:100::18) To SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR17MB5512:EE_|SJ0PR17MB4631:EE_ X-MS-Office365-Filtering-Correlation-Id: 718b7cbf-197d-487b-216d-08dc11393b38 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vZN28pkpuhmJ4jyNt5W6DAjrXyczgwEA2sZLpzpgh45G77HTw/Tho90wKnxx6d1cJOMtP9cMdj+OflGEgfhLa0CLpsTPu6aI+av4fzkdY5fx30lrhPiHKGyHNN1HYXdV8hJj2sW3/rS53yK0dQJvs644ZP5YICzN//m0jL2ARhrrkXS2n7fCHAidIMRqsvVEmwQMMMDk6vl/b6sPSDglZg04QbnjrRD2CD6Au4hwEEmItQwDET36TVJxNHWnCuTzhL6XWK1DMk2bN493qRjtcoT5T1DSNYePIUjIzjCM0kc6lEKGye3Z3VrAEcCIDtDCeJxg0wZDH0VzRisGKzwhiO9FdOiwNFweiyVkvWW3paZegLQHCCpWLRh0FQPO7v//CTGZ7quGxOMvFVboyQrY+6TmvcWNsNYZXebrmoQzJYtbjTWIq7fnItfRLVTwdeMdtoS//7gcV7vTA3Y9RJzaJ9Ms3of5TAZ1TgKkK6KJqagxAIxVEhx4PwUI08hgYSaRJadvFxcKgiDCzL0W04+dKW+3yXyp3ssiPJyOUyiCDeE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR17MB5512.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376002)(396003)(346002)(366004)(39840400004)(136003)(230922051799003)(186009)(451199024)(64100799003)(1800799012)(66946007)(66556008)(66476007)(316002)(4326008)(8676002)(8936002)(478600001)(26005)(66899024)(2616005)(83380400001)(966005)(2906002)(5660300002)(7416002)(41300700001)(38100700002)(6916009)(86362001)(6666004)(6506007)(6512007)(6486002)(54906003)(44832011)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?l593e5e8+m1zaNVUffG5jI/KfHq6r1572WN1s9IflGSmrSrRy5rP6tO/49oI?= =?us-ascii?Q?mOEVph1Mkmz4a0P0tj13iHXtcuXj5lb/4X/7AjhNP9fqhDALxGdX96I7kBKg?= =?us-ascii?Q?l+XL31VVGQKqNuMLBVCO35I+CqL8hc3M5dfMY5Uu4DGk0OIdmYbR8jXmIuWk?= =?us-ascii?Q?+Y1/Ymg5YXRJlRb2a1XIu0M7GZhhr1l55yYwADPTbkJevzqwGPfvZbzxikF2?= =?us-ascii?Q?c6jccY62zrKK/DTHg77O5HBv2Kz08G1K6W015XI3nPknEsxVZwn36BDwjQxT?= =?us-ascii?Q?yqRbXxTNwuXzcflwptJME6Kp7tsqvsRRFv6r6yJy3OQj65HDFZoLfn2qWahV?= =?us-ascii?Q?G4fvmnOIRTYMkNMU03kbIOJH8OhfYyqNgaGZ3Fs4nhOUWDY/DYVXIynPsEAs?= =?us-ascii?Q?Q++ixbqr3x0qt9fFxYrhpMZFkrrzl1NxmGnsBvmgAbvCwCD8i4q2jLSbaZvJ?= =?us-ascii?Q?4zwBrheIJZqbFX6N2xJrU0CRUy1U93X+DcCCIi7xJQY7gXuNKB/3N8ommdvj?= =?us-ascii?Q?zzWznvhUqhN0X3HrynY6FG26tauqeQ58AVYh7lC2XhLO+oEbweIbM5DjGQBT?= =?us-ascii?Q?cLO7Ce/CnZWieVcS1q+sPrrqBBhASkLBnfrM7FVMkcwejxYTOu5KaHI0YTlN?= =?us-ascii?Q?VN4EEcebs6zSequ0d7vRn0jZ6rBNhaq41vKQ3JjIrd92FiDqh5d4Moqa6sbj?= =?us-ascii?Q?UQ+YRbkeT0qB+BY9N4oZg47z6PKCP8putfa+7SKf/cxQQHwVLEnnMoIbqzEu?= =?us-ascii?Q?nrwAWaJv7JBNTLFExLTi/5alejnKr3N33g1nQ4vXQIymEsl8pKWFPc+4NQAF?= =?us-ascii?Q?mKPVwNCrq1ryKodSKXhX+2dxbOC9fiK1UuaAD6Hl/y8Ir9kAQYFmE2Stqyde?= =?us-ascii?Q?5wM8YJTqed75g7vD4MMO4nZIrPbHjDwkHSeOIEdN8r3+keu93Ek4gS4yBzXP?= =?us-ascii?Q?XbJT8XLMnDYdSusGGdmLLDwo1Pk0QsbX6ZrTVvFg3SqakBecLhq4xwnmpYfa?= =?us-ascii?Q?Corg1kJLAibFH92yu0solxep/ve3Q/2l9twAjOS7vNS27Tq9fOuMgwph5KYP?= =?us-ascii?Q?2iH6XqfSIJEZS4PCDRUydJmK8VzKd0yCpq1YWumlVjqdVbcXne1+pTTZT7Gy?= =?us-ascii?Q?4WfEKIsMINRnvzOjTqpjGUMam2vSEb6LxOj9csQy5OvGb8t1oQ5RtnY9DDOh?= =?us-ascii?Q?aJQ9D7MPt/M71Cp4ysQpya++mabqFepT3zmgAIgqMzketMKwNqTpOCJTRKny?= =?us-ascii?Q?ltnFnBJ6iEZETK8WKmZdnIt3BYbKoeif3Opy4IKBrF/ikv04dR6NABvE+n6Y?= =?us-ascii?Q?jDevoyw0IvvX1zayrNoqNOlG0WzjKl0ul+JMjPd4inkbVYgBiKwF1w8fbCDR?= =?us-ascii?Q?vd1qHKJnYmHUqMIVtSb6z3Lphe+jfm1woio0vKG5GLsepmJuQdXKX9HkwNL1?= =?us-ascii?Q?1RhgA1YxK02+c+yAc8/HoJchOwCglbPfEB0SZLW4J4aunHBHtqjipTsOhld0?= =?us-ascii?Q?5w2/NNCv0Yw8QicKWC+5yKT/ApjNigwIvUkeMrng78vRAwOnQwk7AruwfjP+?= =?us-ascii?Q?83WT5nVfcppv7znJuLt+Qgr0mdwKXRLI1lQgMvSdto//QUrftuIuqodbOkfk?= =?us-ascii?Q?0A=3D=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: 718b7cbf-197d-487b-216d-08dc11393b38 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Jan 2024 17:34:28.6214 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fR8uuaP79JKAtxBuJRB4y3jcxxaISCpbvaMuGflvYAbGH9CPNtEW5bOyZgp78TsiiiHKuq6sPep+PkTqTSm+TNAFTTLPHFWBZsZ7ORsE/54= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR17MB4631 X-Rspamd-Queue-Id: 1108A4001B X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: i4o8xtetxboyx6t1n1gasdtn4z9jkhwx X-HE-Tag: 1704821673-539718 X-HE-Meta: U2FsdGVkX1/QoYtnwfFQaUaNw28dWgrZwWKRIkVxhlE/cncwSF7L8+m/CDB0WZtiwPVP2T3UpOy1F3GEhZlD+rdYMxCD4y8iRFggqOVOOVL0palLY4JHO68yl7+MFd222/xSBOv7VHMcs4jvOEuQyxYKqHXwe930U+7LxUL6JKmjjpkFJZqPheCWfX/PQ4wLJD5+k2jjooRvXyvOhC6PBZ/vIYLDF5qUYVS14qnl/7Q60ho9R15is3cKl43NJK22aCmMEm2I9IlXuN/6Vciu8BBgek20OgHM8WuenHnaUUtZcgY8rJGMlyEjmcwGoXFGuuzHNSWU3bJuhI6eIFvZNp/VF/4zgswJ90EljO+6hvnA/dvvPoe+sbFAK1qZaIayZl5LDUl/4fLTf0+1fQKZ8T7QY1c1ZvLgEJCDgwYmzkUrZZdC1izMca13pn18gm5JM72Ub/EQw3WUOU7cz9fNCmbdle6DRwlb5y2VXKkoMtOn0l8w8Xc7uh0OaUsThXJvwZmeNgZTz6ysFD0eZN8yOVIfImzs+aBV36jVWBkNuI7yvSmP5HwcdQRi2caYLEkWR7lNHLzmJToyZuvzV1HdVtFaNixTQTqF9Gb3FinHPSTGhzmiHX35CSyJoR0M2VTnM43SACRvcxKkwSjP6n4zkWzgjLwLEDU5L85y45zFC/oShFbL0LwK/K+rgnhkEM6F0qqkvp2PFCpbc0Aw+KR7d30/HrXKS566ZOiczwTqxVxW2P7hEdfQ1pNC92MY3WOvGXRkKQizIxK6qh6tgIHAHVADPHmOwDPReuNkKhUtrnsA2R/kVPfLtyaMguFgsehQjgmO+fJ+mxtIqI7mO7tjchDfVq22iYtc9WVX+U6w+R8eIdis9Uhf+kW3vs+LDGhew39R0mtFFTzdKGtrWHIyQ1UzTu0S3Mne+X0h7XgMwD0AXiyvnECFj9S4TC1ZnAm45HniNhu+jow= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jan 09, 2024 at 11:41:11AM +0800, Huang, Ying wrote: > Gregory Price writes: > > > On Thu, Jan 04, 2024 at 02:05:01PM +0800, Huang, Ying wrote: > >> > > >> > From https://lpc.events/event/16/contributions/1209/attachments/1042/1995/Live%20In%20a%20World%20With%20Multiple%20Memory%20Types.pdf > >> > abstract_distance_offset: override by users to deal with firmware issue. > >> > > >> > say firmware can configure the cxl node into wrong tiers, similar to > >> > that it may also configure all cxl nodes into single memtype, hence > >> > all these nodes can fall into a single wrong tier. > >> > In this case, per node adistance_offset would be good to have ? > >> > >> I think that it's better to fix the error firmware if possible. And > >> these are only theoretical, not practical issues. Do you have some > >> practical issues? > >> > >> I understand that users may want to move nodes between memory tiers for > >> different policy choices. For that, memory_type based adistance_offset > >> should be good. > >> > > > > There's actually an affirmative case to change memory tiering to allow > > either movement of nodes between tiers, or at least base placement on > > HMAT information. Preferably, membership would be changable to allow > > hotplug/DCD to be managed (there's no guarantee that the memory passed > > through will always be what HMAT says on initial boot). > > IIUC, from Jonathan Cameron as below, the performance of memory > shouldn't change even for DCD devices. > > https://lore.kernel.org/linux-mm/20231103141636.000007e4@Huawei.com/ > > It's possible to change the performance of a NUMA node changed, if we > hot-remove a memory device, then hot-add another different memory > device. It's hoped that the CDAT changes too. > > So, all in all, HMAT + CDAT can help us to put the memory device in > appropriate memory tiers. Now, we have HMAT support in upstream. We > will working on CDAT support. That should be sufficient assuming the `-numa hmat-lb` setting in QEMU does the right thing. I suppose we also need to figure out a way to set CDAT information for a memory device that isn't related to CXL (from the perspective of the guest). I'll take a look if I get cycles. ~Gregory