From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 057C2C001E0 for ; Fri, 28 Jul 2023 01:21:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80CA86B0075; Thu, 27 Jul 2023 21:21:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BBCA6B0078; Thu, 27 Jul 2023 21:21:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65C7E6B007B; Thu, 27 Jul 2023 21:21:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 52F036B0075 for ; Thu, 27 Jul 2023 21:21:42 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 23C8714066A for ; Fri, 28 Jul 2023 01:21:42 +0000 (UTC) X-FDA: 81059268444.07.2E9E7BB Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2040.outbound.protection.outlook.com [40.107.236.40]) by imf05.hostedemail.com (Postfix) with ESMTP id 40512100005 for ; Fri, 28 Jul 2023 01:21:38 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=KMyO4JKj; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of apopple@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=apopple@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690507299; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u+DZV8Lv76DOb1vqBeMy1Rl2q+T8/vMW333RmpiUqjo=; b=SUrZLtV8GSKNqmHUjFXy8+Ug6xxnSCYJpcvo3QIuNumPK3y40yXVomrDz6y4U5URD7BldB PRbawFP373O/RVwK8HWSFbAyK0XzWNKEapV3G99P9scLw+oxfjX9Mhq+q+zlh3pJPeGtlD Hd1WtmS3RGIcbuk8xq2fdxcUHktnJ0k= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=KMyO4JKj; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of apopple@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=apopple@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1690507299; a=rsa-sha256; cv=pass; b=D2aFcbdA0LZZIeXK3Q/MXfXPXggwfgUi6Snql8tiYsMjwVsVsK6ReunDCrqYGwaOODPNjR 0uTWb9XDBct/oiMcsPKO9SPZ2y6THBOzYeRPUrqq61W1bK90AR0ECKZYHX8NQqmOdmXPP4 7iz0VHvg8TzLpo11WH1jBbB/NsbDwDE= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TpPZBEPIC9Wgqh4o0JGjJEhxCs6a2k7diLKBF0HGMSv11lvyrsF9+UTZ9Yxl+7AAyblmoMStysZ4QMPpycpfn0hBC4BEtaqrhtOU415hjWc7qt5+G3ceD1xruhsOavuT+tqHTCHuoW6bM+Krqzg+ADFbbr+gtvGTgQH5kbQY0RdDN+9EYXiX3uzdQBJedIWk7TvUrF4riFgjuGr0dSrLqaD7BfV/bK/wC3NMw/XPqJamzhJwyo/X8XOavw15HhQu9MV2dyOUFOIOvGCiEKuu7FSApHz5m7tPXtTfw5bCeTAh37HimLVAHGxiy/EsXVh6mkyWfZpw+NbcB/ESLf37BQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=u+DZV8Lv76DOb1vqBeMy1Rl2q+T8/vMW333RmpiUqjo=; b=ZRO9H7L/H1pS25YXztieryZDxXrrUd/3tRTO2Ub/cnThUkPO6A/Oeqs3JzZSCWm3jDf4g1UavNMKxCo2pFgnG0pJeR5DaZIVq0hWr8fb0KRiz8XEejkXvUeGjQiNzeDe4GBk8mpj/R974R83CrNdXhaAp9moO4tCiBddvPUhvgUWJXZ8oqSd3sApsKCZpnhSqX7+lZSSfmGgFwwvDQypDEQGMdrHJr20VtlD9xUOmK6nUkxBvIzxupwEIYfEFcw/sAUAa4rnUfFkbTk10Lv1v2jx0KOU1ItUM+Zug1AtsUmSn+/s9jQGjCxR1uXNhUDp7LmTCKHCGqjX/H4P5P3TwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u+DZV8Lv76DOb1vqBeMy1Rl2q+T8/vMW333RmpiUqjo=; b=KMyO4JKjl0k/MG9gGrCXucqtQhMpiCX+nDiTCYnhO74qkUdnVU4slJUKlzpaRjsmqddFQ0glM7DNa3sQYTtEG0RPdwcGw0WbaxCruijDoK172D1SAIHXAFWnwn+icp1lyVLKRZGwcePUGHyP9lbURn5+THdnEDVzAkhh546Rzwu1gEBCPLr0LKhSXY5HqlkGUezI65YrRMSqd/Ct1HvQ6OuubizbyYh/HWmKl0uUn3kgoujJ9OA/J6ZSXf89kmyYdMfG7xdTANMrpvH5UGR+Au1ggsJivsVrdVPVRs0sV9vi7sKpmvPTkziPU26OZohWzvrHTHA5akhjSwzSReYOjQ== Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by MW6PR12MB8833.namprd12.prod.outlook.com (2603:10b6:303:23f::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6609.24; Fri, 28 Jul 2023 01:21:35 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::c833:9a5c:258e:3351]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::c833:9a5c:258e:3351%4]) with mapi id 15.20.6631.026; Fri, 28 Jul 2023 01:21:35 +0000 References: <20230721012932.190742-1-ying.huang@intel.com> <20230721012932.190742-2-ying.huang@intel.com> <87r0owzqdc.fsf@nvdebian.thelocal> <87r0owy95t.fsf@yhuang6-desk2.ccr.corp.intel.com> <87sf9cxupz.fsf@nvdebian.thelocal> <878rb3xh2x.fsf@yhuang6-desk2.ccr.corp.intel.com> <87351axbk6.fsf@nvdebian.thelocal> <87edkuvw6m.fsf@yhuang6-desk2.ccr.corp.intel.com> <87y1j2vvqw.fsf@nvdebian.thelocal> <87a5vhx664.fsf@yhuang6-desk2.ccr.corp.intel.com> User-agent: mu4e 1.8.13; emacs 28.2 From: Alistair Popple To: "Huang, Ying" Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, linux-acpi@vger.kernel.org, "Aneesh Kumar K . V" , Wei Xu , Dan Williams , Dave Hansen , Davidlohr Bueso , Johannes Weiner , Jonathan Cameron , Michal Hocko , Yang Shi , Rafael J Wysocki , Dave Jiang Subject: Re: [PATCH RESEND 1/4] memory tiering: add abstract distance calculation algorithms management Date: Fri, 28 Jul 2023 11:20:05 +1000 In-reply-to: <87a5vhx664.fsf@yhuang6-desk2.ccr.corp.intel.com> Message-ID: <87lef0x23q.fsf@nvdebian.thelocal> Content-Type: text/plain X-ClientProxiedBy: SY5P282CA0058.AUSP282.PROD.OUTLOOK.COM (2603:10c6:10:20a::17) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|MW6PR12MB8833:EE_ X-MS-Office365-Filtering-Correlation-Id: 9f4b5d1d-f65d-4000-e3f4-08db8f08fbf5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: K3996lDQectkgq0XvMXoX14QM9tXU668dmoINjrblh7lwG+6plR9qmFKDjrJ9gc6ISumh3C4cSAcrOsO+eik5TAWf7Rm+fES82oSfALvD4u+xzIi9+CbtwM2WPOHYentUNgNdHjSKZiM6OCGfrRL/VpSHSdfgRfD/nfM/94fKhaKuV0EqZj06F0td9P9R3BCNlRPdhTE7Ai9/zae+9vfzCuwkNaZV/i3ox0/ZGQ4zUip/3x/3CsReb9HY2fbz4+YRTKH7SGOZcOjUWLCAVg1YebdrhM2dpnt6Kmm+5yAq6Aw7gd0ibcPiK4n2XkasOSvmNI+Y411culDzsVVC6fluXQ6SFRSOXXVWxRpEiMrXUyWqxaI/RCU8wZUWXCI34Jn4wSDgusIpiXzXaxKZBFeEAqH0kLtipxVmh3YKDDTgOIqF1HpnHkNJvhmAf8pn4ags+hD28BYj3J8Urk0C1Dzx7+VVWIp+RD15KMLs7pM7V0aY05+twu9jDlqYkIYGepNKPwBV8oD5oy/BPnS1EtTcrzXrbBYtqPc/s3ewBc+A/u39YkeI6AVfIzNvABnLo8j X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(346002)(136003)(376002)(396003)(366004)(39860400002)(451199021)(8676002)(8936002)(7416002)(5660300002)(316002)(6916009)(41300700001)(4326008)(6512007)(83380400001)(54906003)(2906002)(66946007)(66556008)(66476007)(6666004)(26005)(186003)(6506007)(478600001)(6486002)(9686003)(86362001)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?2pBuRFF0grMABRybRcUU1KoLaqyjB4e9qt2KK3zdQn/5CH5dCrXaayzd6BmH?= =?us-ascii?Q?rD0/lqaZ7DbS+qogyHXNBUisne4Ud34NWqmeWki/Oz0eAFVPHxzPoExHVJae?= =?us-ascii?Q?PwDNyVwI2yQvD87SNdgCT78rpz2w+/fXye6+wiCOvCZY7kcbqTyJaK7t1SUV?= =?us-ascii?Q?tkQfyryEpIy3zOFttbD1dgtf8P+HQF90ykK75uoCbXsA87o6AkCPiD2GI9Cg?= =?us-ascii?Q?Puxc9ijLEP4WlZFtv4QgTpVh+h3tiXe0txFLu0lZZCAGGudqMPmyZuN2eOmd?= =?us-ascii?Q?jDv4eqr0XDLKbfSb3PBlqgGUBsxUICGbDjQbEOmahxAXbKblnhEZ9JgJKJiL?= =?us-ascii?Q?2DnEoG1HHHrNo8SFf/6rlhunwcWWELrhKZKCX7KAr9NhgOtvoPASk2E24UeO?= =?us-ascii?Q?8e1z0vLn70of/5Hsr2BYARpPEz6GzWKUJEPN8cpWq/+Qj8AGF7zlIJKiZDch?= =?us-ascii?Q?Ud+iToswu1L0Vj5DyEqBT52vPhmvZhifnKPglpd2lB4wgKr3io32DU1CLkQ7?= =?us-ascii?Q?2EolfupBkLmSVkIoQcO8Wll0LAtgsAf4OLhYPnOkEUFWY6Ac0kZCrnJ+z/Ab?= =?us-ascii?Q?A9pv8LEYaUF6iwUY1aBGm9UVQYLiWYau5tP5YBiS0BRD6Jg8fSyI2jbegnbl?= =?us-ascii?Q?9u0EDG2IR53fFT//+ReCpEkXVnoQojrOUR+KrjWVbeVX+VA1sWgazY08A98j?= =?us-ascii?Q?dBM1pkeZJfFWLGpiaUgMzHyXoy18m3ic2xS7W07i8W+CS1N5rp1AL47M8j6N?= =?us-ascii?Q?rOCkYEnHOGISFSEzOh3TvKJ+nhGCozTzYyojDIbuJIEUzbaMr2hXLYuP4fIk?= =?us-ascii?Q?ba1GfsQtAyTJYH9t9/At3cjQ8agXnlPYOOyxK3seN0gfaUQmmaQ2qMekcxtu?= =?us-ascii?Q?OQqyE/FJNqjOTdSzB+kJ2x/CDie7IcJ1GQEUaOz8DXgkgi40zrdIf1noaTxg?= =?us-ascii?Q?ZnXyrKomV0sCckf7rEXoVX/iltgaLXo5JW8bs9yediPBB4TRFa8Vh/KC4Npe?= =?us-ascii?Q?8WhVUa73cHdlcCyt37v1xHYGWoyQKQBEx1uEfqCMm/vctC3fnPZ41zqXwnOI?= =?us-ascii?Q?VO4yEl7oy8iWZVuLiLHPz+mQaLMgodJqN7tUImjbWpmMB5+PcvhUDdCGHK+0?= =?us-ascii?Q?lx/s6v4+vVjJ6rLd+Fz5LvbGH22FmR3Vl4ni1+r+oGdcwubTi4K5/rOAWTXa?= =?us-ascii?Q?OWbDc2CpKnUnfVrybFPh71uLPqvGYVfOKsC3V70fSlcuVZyOdphB1Bjp8g/o?= =?us-ascii?Q?4LMFnclqG7L+aKJflhLtP+Nw8So3Ea4wN0QDlYl0CsVSOopaz064fwzycWJ8?= =?us-ascii?Q?EP1mz/Q4o4KEMQ9qKM2Q0NWa4zj3+vvNYpc51z5JTF298p15fvTmwpGUZLry?= =?us-ascii?Q?QfLXqAvrRxoirlKyVyRObsCAHf158CR4BmYZJvAJhkLnxSosy94mjmqWobVk?= =?us-ascii?Q?nDXZkMieZm2CX+/eizjweqy6qL0XWk0sTbFxOk/PlYgA9OR+4VhgIRxz7Dea?= =?us-ascii?Q?ytF4P7qFWjh3RYKD4dQyyYSGq94uzca0ae0Bume/vCcBJfkACYhWeUlS6jj9?= =?us-ascii?Q?0wOdrAWwGz799P7LPdNjBnozjIXmj0bsDPF1ijQ6?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9f4b5d1d-f65d-4000-e3f4-08db8f08fbf5 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jul 2023 01:21:35.4742 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: u0mOLnO3YgQy8Qq7Xt99WXmPjZrt5cqjsFFAizLjTktY04OcVFLK07gb6n1dunk5tbS/UeLQXSBV/hUuwAgvQA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB8833 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 40512100005 X-Stat-Signature: u7uobq5obohwxtek9sdyr6nenz8xr4om X-HE-Tag: 1690507298-135431 X-HE-Meta: U2FsdGVkX193G/liUiIaz/uMcxJ11OTLxk9gkA0N89z2mFN3yhFHLCXj7/ap8NqRAdv6NbIRDgLLPsJSSFa5QsM2rdH0gAQi2vCHVv4lg75OUhN+yX8GCAqL9pg44YXGMLjxlQHOt+P74PWtIhw4R0d7l5EMDas4RCiPxJs4mqGdGeUcqxOBZLGEyFIhwhHW1BOxWonUlsYfvNUAruQ8BobsblQoxpzvdde2Ei/qwKiOL8VacbIPhuUiV5yjIjt3w//abyEPLeUbifONvu8ROL6Uo3/a9c1bDtoc1ALckE92MfviRALiHjcpaxsAXPPfPK8kfRJfg1Vp+ZdlL6gpsnh45afpRI0YEuwc9ygyAo/++jIZXnSo9jWnNv/9F6KGEE5nuk0cf5TMc8UQaWI6JILN0kM1UhEeFbNGY9uYxcl4HmGp1mbCHLa89rAM0LnP/+Yhh6zCeG/DgBMAXR5Wp9qTYlSLYNHAAXaEiXYo4AVBI7R5Bp9fWrZYVgX1gRGhFFVgujWZ5MDi8dnUIRB1DbRHToozXnVbr5iPJahQXN6UNUqzBOCEYMquGZkxhdCjckoaR7xdQI6egfy3FlJa75v6gLDWUkyjjEQVgsTaj2Vblbn/nXlBtFEy/dl7qHlQRD6N5TZbwD2V/KG8w1cwg1lmpJy3+TKLudJPLu9VOmoifr54wypChyCxQGeAyiKYToW50ZqfSINUhEV7LGgt9RQPfaYKwvN28i8N4u6L6w3A2MrbL7XDptaRAkP4paHTY2dvSARulQbWxkNARJN2Q+KPrBYeDbbr3hdMoE4hGBzXcIwxtKlmyMnDoFzK+Nra8Vm5EnROsCS99rbKWWNamO8N4Yxc55wrH+5lTwLBTjvNVG3mB3wf7O4JRk4aR9aj+Dmxf1ndp8Fd9FGKfvqPzYxXH7NXscueg8kh1F4OGHvCnzL2HmFaLEWnctGvV1odP9Ov6gPFHPI1m2uzrk+ EZBkiTed 0x/9UtMKjPN9bxwWaB06gZjw9U8pCkc9cAa/yePVrFHl1huRHNqnE91w7brniQul+kxY6/uvQji1u2xrgmVaQVnZxA369jptjGCDB/fP4joxAuYYHQcsZEXIUZ6gF9rLpPvpIsC08NjxCj6954zs8ZLpQmsbGURK4L6CHc5pZOSSd7oFabosY23KIVjWyITwv4sH+FHfP+0Yrbg/C3XCmGoiZ4MRxlDTvCupNKjeS9JI6tJseFm60VUztzg7hgFPSfqbo0NLDpbnqCZbDLU/0fr/6vA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: "Huang, Ying" writes: > Alistair Popple writes: > >> "Huang, Ying" writes: >> >>> Alistair Popple writes: >>> >>>>>>> While other memory device drivers can use the general notifier chain >>>>>>> interface at the same time. >>>> >>>> How would that work in practice though? The abstract distance as far as >>>> I can tell doesn't have any meaning other than establishing preferences >>>> for memory demotion order. Therefore all calculations are relative to >>>> the rest of the calculations on the system. So if a driver does it's own >>>> thing how does it choose a sensible distance? IHMO the value here is in >>>> coordinating all that through a standard interface, whether that is HMAT >>>> or something else. >>> >>> Only if different algorithms follow the same basic principle. For >>> example, the abstract distance of default DRAM nodes are fixed >>> (MEMTIER_ADISTANCE_DRAM). The abstract distance of the memory device is >>> in linear direct proportion to the memory latency and inversely >>> proportional to the memory bandwidth. Use the memory latency and >>> bandwidth of default DRAM nodes as base. >>> >>> HMAT and CDAT report the raw memory latency and bandwidth. If there are >>> some other methods to report the raw memory latency and bandwidth, we >>> can use them too. >> >> Argh! So we could address my concerns by having drivers feed >> latency/bandwidth numbers into a standard calculation algorithm right? >> Ie. Rather than having drivers calculate abstract distance themselves we >> have the notifier chains return the raw performance data from which the >> abstract distance is derived. > > Now, memory device drivers only need a general interface to get the > abstract distance from the NUMA node ID. In the future, if they need > more interfaces, we can add them. For example, the interface you > suggested above. Huh? Memory device drivers (ie. dax/kmem.c) don't care about abstract distance, it's a meaningless number. The only reason they care about it is so they can pass it to alloc_memory_type(): struct memory_dev_type *alloc_memory_type(int adistance) Instead alloc_memory_type() should be taking bandwidth/latency numbers and the calculation of abstract distance should be done there. That resovles the issues about how drivers are supposed to devine adistance and also means that when CDAT is added we don't have to duplicate the calculation code.