From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB9A2EE49AB for ; Mon, 21 Aug 2023 11:54:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6999A8D000B; Mon, 21 Aug 2023 07:54:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FC508D0002; Mon, 21 Aug 2023 07:54:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44D998D000B; Mon, 21 Aug 2023 07:54:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2E84A8D0002 for ; Mon, 21 Aug 2023 07:54:39 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 072E180C1D for ; Mon, 21 Aug 2023 11:54:39 +0000 (UTC) X-FDA: 81147954678.24.3EDCD3D Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2050.outbound.protection.outlook.com [40.107.94.50]) by imf12.hostedemail.com (Postfix) with ESMTP id 3D9D84000E for ; Mon, 21 Aug 2023 11:54:36 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=bbIx99Jz; spf=pass (imf12.hostedemail.com: domain of apopple@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=apopple@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692618876; a=rsa-sha256; cv=pass; b=4FC9eDJUkQZ2sO65KinHrrjTQ9P8SkHLNw5+hn9wSfshsLLpQo0Ofv1EoyD4Cq0jEd2XfO lk+VPU1WuH8z/Zr6nDvpgD5Ui2zn3ZvBJAA25Rcq8eDjgQZHehmCc89ljDsrMWF6lm6Jg1 z5N2XXY2KOfgsnrLOlOJmT47g0h2epY= ARC-Authentication-Results: i=2; imf12.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=bbIx99Jz; spf=pass (imf12.hostedemail.com: domain of apopple@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=apopple@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692618876; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+7bCijop8ua7fyMKfxf8tPhpv50m9K4/UBtWb7TTlJA=; b=qpq7qiFHsS/j8r/ai6pH7mJqb7x2oCOZ+/TMkRZcDVgKNauOx1a0qVZYvgQ7rkudz6+EvE dqCJcSrD/OvKnCsS4WRrROL+5Tk1eQCmprqXF5FME89NCu6gAfo8UbsgKY+O1Fu7MVBeT0 d6MRmvM4+4/5Z1peqFv1H0kHsEI+uIE= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nipsf7uzGkGdHMP3Lj0XB5h6N4/fM0mnhekONj3mlp9CxB/W8iJSkqq1hMrVupUttKhJSdDu1iYTLN8EyOEcNXIx14Aml9nHzJAzwuj81MgVrekDm8NLPER3GxK6Q6uK5if+pGiVwpf8AaQQuu17is2aHxZKDRfLNlgh7ToZ+HOu3fcQFG9Np+Qd1/bqs8SXj5B9DiV06iMKbb0iEp2Sk0sTRQqDz3tA//RP25mcmLtAu+YpycZfjTQ0OLEgQEn3BS4beHSGH3kni9oegajDv2Di1MF+I4qS06m+wl2bACEaMt3wqHiliLgPkQb58/8QIDB5Uh3VwSkFa1fgyR2PZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+7bCijop8ua7fyMKfxf8tPhpv50m9K4/UBtWb7TTlJA=; b=jHJDByh3InyOnZAOJx9avOx7NUUnsmidhcC0CJOLUImurHQEpO3alPN6ye8+kMuIUFNuEkNMJz26BLeFuwked5vUyFPXWMWm3HcMCIxCnbzd5KW7MiCMl6X0FUur7rDF/cx5B6oO3Q5+W9SO8w/Ar+r8nD5FL78Xgoe90W7AL40AdLTsjVcK3fD27/mipwVXFVF+fNBJDNTwb6k0/g6X+EmPI6+2FVZGdWXWbAm/bxVPkA1I29WIUBwC1tFGUwNzK1pdg3hTAK7AW7rwDahI1iNYkZu9C69hTW5rfAGRlVJ6uUvNVtpMB5HJpxQGOE2KyG/yGxPcjmG0G1JkBfHE8A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+7bCijop8ua7fyMKfxf8tPhpv50m9K4/UBtWb7TTlJA=; b=bbIx99JzieIMNle4ZCFBKFLZGzRU9mSRSCZ/daCRV1B3zYiKoR5hhW7vS7J/XmYe6yhmE4e6MLPjYwSPigYO5n5MQSMgFc+FUKAoye4jaCOhjlMNSnv/hmGuE/uITtXZnIGWH7KU6j/e0Q8vmAwFyexJxqXZbZfZtBtxC4IbC3CCEX/wjJDTlekwXr+nytsICIJNy/r8d0BZk0A9W3+crT+yuUxvUifjGu1y8X/guyv9nWLLe+KdLYD3EGKLFROVis1R1sUnDHouxNEBGqq4Ij/WlgF8BTLfEGrV+I/vJUA//HAQcNr3e1N3jbcz4f86TXmpuMrdV6UZNAHVfvP3/w== Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by SJ0PR12MB5439.namprd12.prod.outlook.com (2603:10b6:a03:3ae::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.24; Mon, 21 Aug 2023 11:54:33 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::3bf6:11f3:64d7:2475]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::3bf6:11f3:64d7:2475%7]) with mapi id 15.20.6699.022; Mon, 21 Aug 2023 11:54:33 +0000 References: <20230721012932.190742-1-ying.huang@intel.com> <20230721012932.190742-4-ying.huang@intel.com> <87ila8zo80.fsf@nvdebian.thelocal> <87h6psxzak.fsf@yhuang6-desk2.ccr.corp.intel.com> User-agent: mu4e 1.8.13; emacs 28.2 From: Alistair Popple To: "Huang, Ying" Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, linux-acpi@vger.kernel.org, "Aneesh Kumar K . V" , Wei Xu , Dan Williams , Dave Hansen , Davidlohr Bueso , Johannes Weiner , Jonathan Cameron , Michal Hocko , Yang Shi , Rafael J Wysocki Subject: Re: [PATCH RESEND 3/4] acpi, hmat: calculate abstract distance with HMAT Date: Mon, 21 Aug 2023 21:53:13 +1000 In-reply-to: <87h6psxzak.fsf@yhuang6-desk2.ccr.corp.intel.com> Message-ID: <878ra4wqz0.fsf@nvdebian.thelocal> Content-Type: text/plain X-ClientProxiedBy: SY5P282CA0090.AUSP282.PROD.OUTLOOK.COM (2603:10c6:10:201::17) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|SJ0PR12MB5439:EE_ X-MS-Office365-Filtering-Correlation-Id: 473b8acb-ea9b-41f0-176d-08dba23d625b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: m6aymPqBS1eWbbuheOlpRkdLubidxmZMTKV3TTXR4mU4AGViXHP6U0/j8Rs7b0/zhcxXwaNcZEJubLoEO0ahoa1/258zblbQrzJCKGPJP/5FCyBUbSZLJCYXUTj7k/3DPznMbLM6g0RaecQL/UQuLHSAV6Hc6NeKB9y3UKr6WiCXZNQAJzqyxpDz74U3VK5mDsf9He4HPy/O9IDVV5Q9bewM/PjLWW6SMpyoqzBuMKc+fOZrHnqyeuZf7RZ+M4sucvMc/ePtcAXQWgAaK4uxLot8ta61a2dTFCRzRO8JfmhkYgB306G5jml3E/jGNG4/Y8gnp2q0WCrWMKAkjgG+lBZnlgnDB8hw3IQINhmRlTaJz1sLmgBN9ITXtfn7sL4HUy/pHzxHXMrYp5frhWL1uN2VbAR9P+QctImY9H7rZ7nDdr9yeesYEfvmUosa/yx9AcU+Mm1VCQWwHY1udqCgdRu9vNrOBS2IysR2+5ccqT6vrt/b7K+zksAnWun1SUcGuoieh+DNpbHUEy+qxgOEg45Gdw3jq9sAIi1Ei5fhraw1EePgNdTdWHmgxWVPZjYE X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(366004)(39860400002)(136003)(396003)(376002)(451199024)(1800799009)(186009)(2906002)(83380400001)(7416002)(38100700002)(6506007)(6486002)(5660300002)(26005)(86362001)(8676002)(8936002)(4326008)(316002)(6512007)(9686003)(66476007)(6916009)(54906003)(66946007)(66556008)(478600001)(6666004)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?9GFIovdmzfUV3Etv8AohJmp8AHZJW6EQ3yjU1qHnu+uJRxhNgAVf/a3Zwx+p?= =?us-ascii?Q?sDxxn8IFpXFgIx4vyG5P9HIU87gK0YyaMIDs+IEKMhtT8HzvOE/XHtW/b5Z/?= =?us-ascii?Q?SJ2CLF+ftdpFXWsd7/4I4n4O2imy2YaGr5xSw7IU5MqD2xUFsbFVOS3FVwUS?= =?us-ascii?Q?fpkt0Cdg54AYr+Zm11RzTaG1DIX+eV6cXgF6VeLHu8kmX5NSg5DemfFfCLWX?= =?us-ascii?Q?Ta+NYoNzdeYn1clh8kuvbNUwlTvKN1Sh3TkFtrQ9ir19L1Kfyj+QhOaVZkEl?= =?us-ascii?Q?eIdfILk21JVVV5BybwLSUHd1nuaogboObuEpYNMZ/VBN4L1luUGMoASqSzPf?= =?us-ascii?Q?5twARvMbdYPlfSiasx6QEhVydpySj6YSHUFuJS+Mm6v9FEIeMjUfV9glx4j7?= =?us-ascii?Q?IQD99QrzJrO3MPMAPy+eW8R/vXJ2lzFMY9Lp+ubh937bkYmJTnYhfNCBdkIz?= =?us-ascii?Q?/PBz1mkN/H+m4A74bcp8rtIicIQjAhAdtoYHIB/XyCz6JqsMH2CR4fAKds3Q?= =?us-ascii?Q?F7zTVvTSCJet27Du7xJ0AAjy4nrQZN1lsSjqarCWBqe1j+wH/i+vuDInb92u?= =?us-ascii?Q?HWEyuF8EI1ue25nbo2cbOdqVPDopn2aFFYdLc6Lsu1M+PllNAJ/KOXpPUtrK?= =?us-ascii?Q?QZ4bT2q/M8tGNqbT64HeFll0Q33x5/ORnL56cA073OcAsce4FxLPXULdW4gI?= =?us-ascii?Q?pq3MlMGylottX1BGcA25nMYMVqptTIJqP7XU4NGPFGsIak4MZuvIf/s0hqRq?= =?us-ascii?Q?bikc+S5suurC/ho5Lvu+io7bBOVkPgeLKEqF9Ra18T3Yct6nzCGeHDEFNkfM?= =?us-ascii?Q?X3Wd7SN/1/UAscESXdB4GqjL5NlGVKtPL85LIB4GxdstCJyn4To6QMxa6IdI?= =?us-ascii?Q?lma5IgjV08vtRCCikq9Y0LURD+96ke4Rt3PuWAmP6k7Px2XMqLfVHopVpqb+?= =?us-ascii?Q?63F2xFjOyIisXHMkhf/zMaqDhiK4FS+WPs6M8KbsSCVLihBrjq9Et3GpsYdC?= =?us-ascii?Q?TmKafpYBVA17i71FHLNqdgEIuv3LvAYVm95HABJhRszybSA8THhSalzZSuWp?= =?us-ascii?Q?7zC8RjPm5roGZ3iKDij9vTD2RaaIKaqWqX0XalC5uJJdzYJsU+g+sS7sP2xp?= =?us-ascii?Q?OFK4l+AHOEwzsd7CcmOHu9lVJzrzdWYdExXYcTPFkV/+wwLuJr1vZhZ7ibx1?= =?us-ascii?Q?Td0H6hNh6/ty5+TNgRqlzHfpQmPmUWB3lmXTQg8oJgBHiUJM4hng+svXMHzq?= =?us-ascii?Q?FefNh1gP2sk5XlKKh2PHMKd6/ejQc8BXzmBD1y3ku0FEksW28Cv/JV3J4RJ1?= =?us-ascii?Q?YBK0smvrA4l4GIy2R1CVFMTPYrwI7YhbIJQlLlrQRGoNA9qeQhsclMtnc+Z4?= =?us-ascii?Q?vAuaTwtZTFOTtcMFtuiubBFuwjrQB+z5igT4dbf9XVRne3TS1sWHMz4oZAIf?= =?us-ascii?Q?uo93Q/A8tUls8AN1Ie5XNpA4m6JjGwNeLuvO7UI+4htnUrdrrA3E2vzM51n8?= =?us-ascii?Q?vW7yXbztYFZUQvHG5J+TmC1C9J3C7n0oiHK2CzVY1PF7qSWGQwLsaHEWCyF9?= =?us-ascii?Q?BeT9oMDuL6NxYon+rvsasc4UUp2yyigDH/xdQ97X?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 473b8acb-ea9b-41f0-176d-08dba23d625b X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Aug 2023 11:54:33.1179 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: WYPZAGVLyOPTV5YUqQNeXhq0rRH+CAYnQc/bTEGNe52JePnWrXQRm9/7r/zr8RO4cWMYTkeWl6dqPJqAuFU9bg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5439 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 3D9D84000E X-Stat-Signature: 941f4qfdqzctem13whsmutdxo3mtb3yj X-Rspam-User: X-HE-Tag: 1692618876-453633 X-HE-Meta: U2FsdGVkX19xsXIhdftL4D/dMk5Nlp53ZqePCTvKqNBVQTTKHVApVFcyHHiL1Z8X8sF98yTbUXgZIrPDtB92fA5OY4BYFR1I/8cRx6F5Ibld3ajY2l+DGeC5oW/BqoxKqhepUOqD9WU14fGm4mmnl9qvPo5LZgjzeocy7MdnQvNQNFHQNtBNXiR1WDohT109fUM492X7KwNLXrjqvt61ehOgQ3pApRXe1EDfGbWAv2MIz/0vUcZexGz/nQGff7Dj44pWZ4eOhrmh47G2v4LymfbvUClL3GDjBSGEM3I8azU/6tSHGim/3fghz9kcNmit9coQX+slM4lJvMFeiKAiM6RcPAsLPxJcLjpTjsLeYV41Bn0W1y7zb1bQTNJZYLpEuqEg4bM8NEEhF4bDVl9Ntawm+o9zThUJQGF/ZiEt79n1s82rhxfHzgS3dT2h2a0M0wewnTrd07mpHvD4AD/uJleOkfpZu1UbP2HzoAd/fj2Q5qShP9JylPlzc4NiQIGzhpfosYwfZViTjSt3mlJxqeJeK+Poh9WgKX8COth1grMH4LhqmaAJN5K8EQ54gPfLmHVEg8nHSPtAqP8NoWqNU94LzqJXeme/e7aT8bHrt/sQ9XeGBgcWhyRzeoVMvpTnJ+4RLCaioff/hxGw8RPiLUIAwXW/KOVArs80JYSgxlCg2eGcYrlhCcLNOMLKWkznY/8ncCxR/4yIG0naGSMj/j06CkuOysLtgoGaO+K4hqH30La2Kb91pYmy2j+TnZBUerzwQpWPo4R6d4zBzCErgYEcTKXkE2eefLH7bgB5WaVLDMZpbj+aTA8PWZMI+tMT+gl2MT7jop93UlP9Bz84ECWkOArkuDVwjmih20+l7Sl6m38OvWsdzpQRRtTdP9Sn+34qwvfMw6cMvhp0wCJy6tkvKRm5KDUJbf9Jug0Uu/ow2uuqdCXwyBTMtkr65LNFSJ8KhkOOt020d0zwSVk 6lbfNS7l HvYUi4uxs2izM7/TZNO7jDDTNVy1Mv7LILjkGu2CFdvTjNTaaHKRO5XvCnc0Qu+bOLkD2G6rjHkxV1SrdwQtJ07N0g4vHFeYV2D3rQKlirYx84iAQyZwpao1Yd3ytsqOPVz3esG4fRHdizq9RkvvpVAYdak8IJIqv9Law/bK9H99+NfJulnYpusdiIIa2k3TEQDQlVOrzl6YtAFy0RmCitttn8g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: "Huang, Ying" writes: > Alistair Popple writes: > >> Huang Ying writes: >> >>> A memory tiering abstract distance calculation algorithm based on ACPI >>> HMAT is implemented. The basic idea is as follows. >>> >>> The performance attributes of system default DRAM nodes are recorded >>> as the base line. Whose abstract distance is MEMTIER_ADISTANCE_DRAM. >>> Then, the ratio of the abstract distance of a memory node (target) to >>> MEMTIER_ADISTANCE_DRAM is scaled based on the ratio of the performance >>> attributes of the node to that of the default DRAM nodes. >> >> The problem I encountered here with the calculations is that HBM memory >> ended up in a lower-tiered node which isn't what I wanted (at least when >> that HBM is attached to a GPU say). > > I have tested the series on a server machine with HBM (pure HBM, not > attached to a GPU). Where, HBM is placed in a higher tier than DRAM. Good to know. >> I suspect this is because the calculations are based on the CPU >> point-of-view (access1) which still sees lower bandwidth to remote HBM >> than local DRAM, even though the remote GPU has higher bandwidth access >> to that memory. Perhaps we need to be considering access0 as well? >> Ie. HBM directly attached to a generic initiator should be in a higher >> tier regardless of CPU access characteristics? > > What's your requirements for memory tiers on the machine? I guess you > want to put GPU attache HBM in a higher tier and put DRAM in a lower > tier. So, cold HBM pages can be demoted to DRAM when there are memory > pressure on HBM? This sounds reasonable from GPU point of view. Yes, that is what I would like to implement. > The above requirements may be satisfied via calculating abstract > distance based on access0 (or combined with access1). But I suspect > this will be a general solution. I guess that any memory devices that > are used mainly by the memory initiators other than CPUs want to put > themselves in a higher memory tier than DRAM, regardless of its > access0. Right. I'm still figuring out how ACPI HMAT fits together but that sounds reasonable. > One solution is to put GPU HBM in the highest memory tier (with smallest > abstract distance) always in GPU device driver regardless its HMAT > performance attributes. Is it possible? It's certainly possible and easy enough to do, although I think it would be good to provide upper and lower bounds for HMAT derived adistances to make that easier. It does make me wonder what the point of HMAT is if we have to ignore it in some scenarios though. But perhaps I need to dig deeper into the GPU values to figure out how it can be applied correctly there. >> That said I'm not entirely convinced the HMAT tables I'm testing against >> are accurate/complete.