From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6D9EC4332F for ; Fri, 15 Dec 2023 17:43:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A3BE6B07A9; Fri, 15 Dec 2023 12:43:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 553976B07AA; Fri, 15 Dec 2023 12:43:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41AEC6B07AB; Fri, 15 Dec 2023 12:43:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3168C6B07A9 for ; Fri, 15 Dec 2023 12:43:14 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D82E6A2557 for ; Fri, 15 Dec 2023 17:43:13 +0000 (UTC) X-FDA: 81569773866.04.61F8E2B Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2046.outbound.protection.outlook.com [40.107.220.46]) by imf11.hostedemail.com (Postfix) with ESMTP id EB7C54001E for ; Fri, 15 Dec 2023 17:43:09 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=ML3l9Nfi; dmarc=pass (policy=none) header.from=memverge.com; spf=pass (imf11.hostedemail.com: domain of gregory.price@memverge.com designates 40.107.220.46 as permitted sender) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702662190; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xs8b5T3/it40OjUoOr268t4NxdigYV1DrI7lbLsyghU=; b=SIk5/n5Ow5NNF1O6JFl8xvPNvhVSfNNeHBWanx/Iyu72UbHkzmp4t5DFXGiNxOH3xS+Oq6 UD57R+fMXNGFelo2Kr77WHF9Zy78UKAZPZgAFK8jsK/o8S/ag7fjtQzNXtfgDBR9KuPpiI Ht3OmV9MIbYNIHM5zelW48DZh5Q+0Cc= ARC-Authentication-Results: i=2; imf11.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=ML3l9Nfi; dmarc=pass (policy=none) header.from=memverge.com; spf=pass (imf11.hostedemail.com: domain of gregory.price@memverge.com designates 40.107.220.46 as permitted sender) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1702662190; a=rsa-sha256; cv=pass; b=x8yS7B3enuAz+LbJEjMcjL/zy9YR/6lVW2DLmCqTqwZl3S71A4BCDUELbLYFsrt6lAE4VB m10tCQ514RowcHbfe4xs9xquFmE+DTOaC3Fp3pWksHCa4eB3fbL5YjbqUX2vYw8a5DBmrt O6GVdAU/yHBNpcwnvuh/Hmql1DjuD1U= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GS2gnzJIvVpiUXe21z2Of1uy1st8iHU15YTQ0r+Ejhh6MkoKR3+sAikrnXIP1e89tlMhT85MCk6QTNjC5gNkIW2X27iT4tjw91KQISEgAAMq2EkHbNs5PAHyY7OfjlNwYZnyp3QrUuXefVG2fTRp3JXDLb76hYUYodMLFOe2ClNa6RkYJaCJus9B7g4TYdP8irovcrMf6Fuk3o2KQo7RHcEY6wsI0Z+saIpf10O9m/GL1TlJwtizRBJMbmjFg5VMURUgDH/EQ0MFSdUG8xYkGtw4qyrCnEdY45mVMZnnOmx4IOsJUoQ77KL+YqPvk0j8k8pY8V3KxtJkZt2hFjIuJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Xs8b5T3/it40OjUoOr268t4NxdigYV1DrI7lbLsyghU=; b=HrxYfbllUzHIluYZurZqVsqR/jjkfH3TzROxK6oKzKnnsKHs7Mw2Jg8yszekvyUsgXJqJzCavI2AN7drqlms0c3Iq3wWMSfvw8+tXrsRAM75ETQuoPiFtS2vp5pC/yTaGSuicuPSbHXe740Y7M1s50w1t0v2tLJR18UVZPAYV3fgJdAQv6ai+/vwRgicKZMhbxT5GUqwQ5ujtshNZ7K+N1fbnTIutxhi91l6b92fznG483zzRgEYTAXrZmyIxY6kUv8p41r7QhOfrfmUaSzag6K2seUALanvyNihtNnOb/rXLqDoc5Xklb/vOZi7spnFW2ZfDpfQzlBCcd1uR3NVkw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Xs8b5T3/it40OjUoOr268t4NxdigYV1DrI7lbLsyghU=; b=ML3l9NfiBjm5UEnB0IBxr4wNb2bfy9j4FZTD+LIsNepqVBfHs8WB3mlj2U34Y9/v+ceuzc5+vTzay+MwUGpt/NfY20yt6l7ljP3puTrffdV675mmHRIxus2q1345XSMF8OHc8iFomlcsSoAmmi2zFfBYOWNyzNcFkfwvF8gnEY8= Received: from SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) by BY5PR17MB3873.namprd17.prod.outlook.com (2603:10b6:a03:21e::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.30; Fri, 15 Dec 2023 17:43:05 +0000 Received: from SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::381c:7f11:1028:15f4]) by SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::381c:7f11:1028:15f4%5]) with mapi id 15.20.7091.032; Fri, 15 Dec 2023 17:43:04 +0000 Date: Fri, 15 Dec 2023 12:42:56 -0500 From: Gregory Price To: "Huang, Ying" Cc: sthanneeru.opensrc@micron.com, linux-cxl@vger.kernel.org, linux-mm@kvack.org, sthanneeru@micron.com, aneesh.kumar@linux.ibm.com, dan.j.williams@intel.com, mhocko@suse.com, tj@kernel.org, john@jagalactic.com, emirakhur@micron.com, vtavarespetr@micron.com, Ravis.OpenSrc@micron.com, Jonathan.Cameron@huawei.com, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH v2 0/2] Node migration between memory tiers Message-ID: References: <20231213175329.594-1-sthanneeru.opensrc@micron.com> <87cyv8qcqk.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87cyv8qcqk.fsf@yhuang6-desk2.ccr.corp.intel.com> X-ClientProxiedBy: BY3PR10CA0018.namprd10.prod.outlook.com (2603:10b6:a03:255::23) To SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR17MB5512:EE_|BY5PR17MB3873:EE_ X-MS-Office365-Filtering-Correlation-Id: c17b7672-beb9-4748-0847-08dbfd954a49 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qtNfuqUUGqXw38/361ldaMoSCoW1DFcvLafBgeDBpzcHZlg7JyTsSWKpxmY7pi0g3PqqdPm9j6SjyZnvoCveNYo1rC0aJBt0LbAVefXSyDfi6gsgMtYcKRctJ1lGkKI6FgYwj+KlLNICwJi2sppjo5abmhE9vvh4EtKujhkQS4xfA1qh7VFytgiT9RxrXybsCwzsnopIy4SwTPVI6v7NyRPTG53Y/aszhs4DNYtmcziapNWJ9GTBFjL+g3HgVg8UOk8QLxfrfjQ/MdfHgtlx5cEGO5w0G46fllnAGWaWBjFxw30IvQNamDEKIrLuyXcgRNoHkvn3zPo3NJmIAUE7I6x5BkVDOklt5Dy4QxaM2gNiEhJZFNsVAu+4QjT12xUHHLb1PI/bBTeOonubAgw864Hew3rz+qVwCqHbJWlIOn30zTyQDx+HPL3HhCNS0Dn7nRwT4g+lBxbuNTkExBSUkW27QuejnHJAOxkGo6SuayScYOhMdHejmAbQJZRQStxloqHUZXGRW1s3omo/dd63/mdsGQM3EzdlNs8Bo1TzYwawe1HSi94TaREknnRxoo7EeodnleGAhpuzOC0nHW7ty22o+OudknN+6wXL2tr8qeE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR17MB5512.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(376002)(366004)(39830400003)(346002)(396003)(230922051799003)(64100799003)(1800799012)(451199024)(186009)(66899024)(36756003)(86362001)(38100700002)(316002)(26005)(2616005)(478600001)(6506007)(6666004)(6512007)(8676002)(66946007)(66556008)(66476007)(4326008)(44832011)(6486002)(8936002)(966005)(6916009)(2906002)(41300700001)(5660300002)(7416002)(67856001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?LvHmO1X/jLL9+s8cBAU9ANDDX2acNVj9CPGN+MNUM7u+Ryd6O5R19AaRpaXZ?= =?us-ascii?Q?yKhm/kQO9zaDu2IpOIybDh0nYoaod56qOU/7ei+C2Pul0XZ81nQM70d9Upa5?= =?us-ascii?Q?zXFLw6jXuk0+vbv+YfmGlWMCgUKxRgv1i6082qEfvI/v3XD/1jk9KK0oaaFO?= =?us-ascii?Q?B4M/VuAB+XVp/v6v6R0KdbF+laYGf3Npad7jSfyvYoyhLBYCa5jxMGtdTVXe?= =?us-ascii?Q?0k4LzU0yrIvWRr6glDNKuNIwTSCnXAbf+OvbSfu1ToV+4IlshrciJxWXIRoS?= =?us-ascii?Q?yEZQgjFUvV/E80jKSgnqrpXzSiS3jaqoiD6gz7an4/4eJeOkde1qbxNEpDP7?= =?us-ascii?Q?oWEehBy/cKHDiV6Jm7boL9DMSQkHFgtYYSakq13J4JoIBmv924+Nf0sHJ4nA?= =?us-ascii?Q?lr4zTVyOjRi4/uP31p3naYMxDYzLI/3ckapCcnLe7FF9kVP+KLoPmTUJwhhO?= =?us-ascii?Q?Gn7yupmz+XqpeaaoO/PmGyHAyIq1FalYnmbbB5Bg2Bp0/kDzA1+ajYS8lrZL?= =?us-ascii?Q?B+PZvP0Bb3BsqSHipIc1S+Ta9UPpTmnPNxx+OX79LuLgf7yMPTW2MqiHd0u7?= =?us-ascii?Q?83GaiuTvHgJ11iNH45HjxFDHmNvIWibVIG97dy04jQtZDnwF/r4ZTTl5W7EC?= =?us-ascii?Q?ihL4BO2bdwPuZQpBgMln47Oe+2ewQhjAosRzetTbgK2+F74MOkCGhIKENss0?= =?us-ascii?Q?P+WT7p5cuQYve3XdXopXmm/dnWS46P/v6J3rhBU/DHe+DsMQ3mCfTZRn36OS?= =?us-ascii?Q?3tEiyr/EpLbgktGQ3cwWaTz+l7JRzF/JA28f5rVFktGHIOKpbkYGvzDq+fyD?= =?us-ascii?Q?hM4QtJr21vm/JNHsE2UIOTyui4Zs0dTmDjjV/99RkE07rtsLmdPOzIa74D3R?= =?us-ascii?Q?Ky5yxeXRb9tIlKJl5xrW5QtiSkWuYDQcfjwTFGWjLCcVuxbyQNXXE7BrPl1g?= =?us-ascii?Q?YU3LC3bnEcobkuhJ+k9JHaRZLh+PcdVwGZltoMfOhWIiz2OxNTIAxMaEzwkn?= =?us-ascii?Q?3iNL+FWgr6HAGFON+j8r2FcCLmEsekTrRExTDN4v8r2AmEuSbFZY5yfkVsbd?= =?us-ascii?Q?BITaZZkzHiHTEGUb8scYzJzAv7al7IoDGBJ7MbPgbPohH8/tZznOvgqEkl8x?= =?us-ascii?Q?S8JQMZA3DYrSG+xzpUb+KqnuChe+TAK3G0NqAIHky3VVhHPnkKW5E2RDMx8D?= =?us-ascii?Q?mJH5zVtgWmuEyvw9Eg72vJSdt47LGB3841UvHFBo8re1VR2/sZh8j92PIjEI?= =?us-ascii?Q?YFzdkeQi+521K2GidDs0ucrm5DaUOSvxAIA4qVhrNBzI23wYhapmImjRAowZ?= =?us-ascii?Q?1alSfHyeH/RJUhMI8Royn12gztw6+9hv5X1SCgnpy+r4MLdZib+U0roliJxg?= =?us-ascii?Q?oikYBK6zwZjZn7Wdnk2TPPN22AB7D68w0X/6LTe1qRWq1gmloIZ9TJbtdmJy?= =?us-ascii?Q?9LUUNr2z3LDNynTMZ7CWAJnOBZ62wIU/l1Td/woX6LzIA5BL8NSkY6riQ8hE?= =?us-ascii?Q?ETg2z6WJH/EVqE6fS3od0FQu23aoRSPOPr7NdbW+oCrHQzme0V5Vtd7V6U2O?= =?us-ascii?Q?DQEynX1dE9rsP5bbImodYBGFuDroGjDrFmmpaoDq2vWiD0TuL6Zy/fE3EoBR?= =?us-ascii?Q?WA=3D=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: c17b7672-beb9-4748-0847-08dbfd954a49 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Dec 2023 17:43:04.3528 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +tyNjT48BM55/16fj+TVk5bU+ZTcsTnQeJZsBgCABQB17Rsa5xcTjoprr1B2x/ouiZlDAwGx4CGG+PDGEso2o+vz3+1TpZIh2FudEdK8Zb0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR17MB3873 X-Rspamd-Queue-Id: EB7C54001E X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: e1dkosu33rp6t9euxnaoth7kaahk5ndn X-HE-Tag: 1702662189-599396 X-HE-Meta: U2FsdGVkX18UnB5U0RkP7T9RimE2H4mUIpHnH1rYcwSDaV1RzCFR5Hmw5KLPGylfFqLkI4Rs0IQwsPShLiQLXDgizZ+ILuRPZn8TGtfXkSn6WQOiWWW91M0UVpkBZZNa0/eLZxsSUryVlQyF+akpWJRjS2KCO6JpU9lkbr0XZgiQs4pV4De+zilYuKVoIPYt9vmBUp82MBU5o0F18fWv2YWUtjxZnMTsl6JwtX63ZXLmOfErQkCB1v1LP69L9znh/+jHWpNBqJGcpCFuU4antLN98yv2Pzrx09lEV4Sc/ShEoxUkvGtQbc+rXEOlUBcON/VUF0RLPp3GUNcvL1NF708c4bKMdI9qE6l+KQ0XwZjCMhx2mfVP0PkhxcwJJU68+2ZCvpqLkHXgrwVfyNKLkeSvvPBOWc5Gba6OBb013h6kEdLOvE7SFjTtSHLmte+fAvzHZbkY9hBilluiQJIghjRWuCwpUsaJrRfu9x0MoEmKEGmCJ/nK5oUrNG+zPJ87og2a+D89gTm3FIErdKuxniQziPk+LS/SAcEYb53hA63vr0ax69xFMX122/ckA3dLUlBggIPXBhcZHEBuUarjMum6t1Wt2Q6DXK4cMufC46WStIRScyYTSWbbZQ+GceycZfgKoeLqdNB4a8fTomYXIN9voeOLJRWGLqT1Cf/bD3zLF6a4TPSSJcFQNgGj9c8lmY88lRtd0iEoXdUBGcO3qD/XCZUCVGKdY9cr3Pr5YeWZV/fJ3480YnJe3nA8WQq7F/q6R5aoHYHM+Hp47EmbUiJPJIx1diNVUlPY2QH4ZDWJOqVUCVh4PTiB8E5JTNvhhkoKwPNjLxfLROmnY0+chz/hZnMjgsOt4XVlViKaPRVykuQSZMrA3gzQvakrxQTeMvz7RIwpdMGDHx1EQduyenFH3Bq0ARD8H9ey0+dYYCtH+m2cgNqteKzuxKfANBuC3WHDvY+Ww8Jh4WNkkF1 6WRx7qRc iuZwmN6aZZ9IJjrbnw+N5fH29GnEm6rH/zgpJWqxMmlaHWwnCUwjMcKqnVWd1y+walGN+PfvgNxei4KENeFf1j1cCrue13hyV3UAHwppAEOdJsnC/Tgl3MEAToZYIq7Igc4uE1Yx6XGm7m+Slo7oCVg32ndrRWTualIk5vafjUvSA9dH71C/qMy9ewkjPiHL715yc2N9J6Xu9Yj+5yYM8fRLmlkU55MGH7KXfHYR4DnANsj8TVlGGH/2ubHzxIbODJzY9f89y2N0rawXxe1VyR8cs74f3WldprNGXJ6wk0h7LJ1GeN04QCop8fHhEk/Fmx1TzPI/91PaS6Bdlm8WOdPHy0FktdMS/Ye5o5/C9XWA59WmFhzVUIxf1p9tqP2vc8FaHQX7yqH6I9H0k+JwJ2GZhFzYH3jGcBwia6rSHM5OR/io= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Dec 15, 2023 at 01:02:59PM +0800, Huang, Ying wrote: > writes: > > > ============= > > Version Notes: > > > > V2 : Changed interface to memtier_override from adistance_offset. > > memtier_override was recommended by > > 1. John Groves > > 2. Ravi Shankar > > 3. Brice Goglin > > It appears that you ignored my comments for V1 as follows ... > > https://lore.kernel.org/lkml/87o7f62vur.fsf@yhuang6-desk2.ccr.corp.intel.com/ > https://lore.kernel.org/lkml/87jzpt2ft5.fsf@yhuang6-desk2.ccr.corp.intel.com/ > https://lore.kernel.org/lkml/87a5qp2et0.fsf@yhuang6-desk2.ccr.corp.intel.com/ > Not speaking for the group, just chiming in because i'd discussed it with them. "Memory Type" is a bit nebulous. Is a Micron Type-3 with performance X and an SK Hynix Type-3 with performance Y a "Different type", or are they the "Same Type" given that they're both Type 3 backed by some form of DDR? Is socket placement of those devices relevant for determining "Type"? Is whether they are behind a switch relevant for determining "Type"? "Type" is frustrating when everything we're talking about managing is "Type-3" with difference performance. A concrete example: To the system, a Multi-Headed Single Logical Device (MH-SLD) looks exactly the same as an standard SLD. I may want to have some combination of local memory expansion devices on the majority of my expansion slots, but reserve 1 slot on each socket for a connection to the MH-SLD. As of right now: There is no good way to differentiate the devices in terms of "Type" - and even if you had that, the tiering system would still lump them together. Similarly, an initial run of switches may or may not allow enumeration of devices behind it (depends on the configuration), so you may end up with a static numa node that "looks like" another SLD - despite it being some definition of "GFAM". Do number of hops matter in determining "Type"? So I really don't think "Type" is useful for determining tier placement. As of right now, the system lumps DRAM nodes as one tier, and pretty much everything else as "the other tier". To me, this patch set is an initial pass meant to allow user-control over tier composition while the internal mechanism is sussed out and the environment develops. In general, a release valve that lets you redefine tiers is very welcome for testing and validation of different setups while the industry evolves. Just my two cents. ~Gregory > -- > Best Regards, > Huang, Ying >