From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D924C04A6A for ; Tue, 25 Jul 2023 06:50:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D6E186B0071; Tue, 25 Jul 2023 02:50:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF80E6B0074; Tue, 25 Jul 2023 02:50:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBFF08E0001; Tue, 25 Jul 2023 02:50:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id ABDCA6B0071 for ; Tue, 25 Jul 2023 02:50:15 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 79ACA160C8E for ; Tue, 25 Jul 2023 06:50:15 +0000 (UTC) X-FDA: 81049209990.09.6FBEF12 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf10.hostedemail.com (Postfix) with ESMTP id 5FD9CC0008 for ; Tue, 25 Jul 2023 06:50:12 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=EDuL7YmP; spf=pass (imf10.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690267812; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ioxr2JtPMKHBRfEX6PMu7EWarlC3U+Bci3qDUtqeEEM=; b=ev4zEGP4FuGzJbT08EcIsmLraGIRjf4Q2RHTWYO3o/CcywtJEbrv7pGfMo6IVP4OvjzNXs kQStG7Tr2DekP3BIGE/SSCao/lw/jYIj6+4YtiIyUR9UwuF6aLq7ztwFsahimlkgkAyl3L iQ9l9J52xLwve3ay1bINz0T/lM2UAQQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690267812; a=rsa-sha256; cv=none; b=BTuS1Lbq3Y59pP+zJIaEryiJoqL/+XATFUPMJtXFwi59CtNs0zZh8qqsp+cigj86Mp9ymG UqWEUH6aPv7mfebIjLq8WoGtjvJoRuypwiC3pf54+C9/CNXD7b6GmKI3E5xCI2NVB/ouQi IiHxGuCdWICdRQsc8DzB+nY75kiwtWA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=EDuL7YmP; spf=pass (imf10.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690267812; x=1721803812; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=sMXp8XuVx7sja1Zb9TfJOl/mMXazPG34qUPoSd0z870=; b=EDuL7YmP7zs7p4Ie7yIxYId/a88nTUuFvBLHees3QLQrvKyTfhueKq6p 05P3HuP+poaZl+RZvkM99kwzzogq4Cs9ngj7T+1lhvIOJhlgtgjmCMMt9 VcH6LfB+QUZOGM7NuM9L+61dZwRj1Dw+uvxpSeRiASsC3rje8b9zGUHSb 2dTHVGfGlnlZW776jKXXqKVOkTL606stat1BmWvCmZpJNSPOnLR7cL14M l6HFnkwmBKguvzup6PwfIz+1V+RvdJZiipu1EifNzu2YoTsfLzZlS4YgI 3ZQOOClvPazNtitEuu1WaGoYfUZQoUS4cW4TqRmrpeLM62FaNoWjJhTel A==; X-IronPort-AV: E=McAfee;i="6600,9927,10781"; a="347242148" X-IronPort-AV: E=Sophos;i="6.01,230,1684825200"; d="scan'208";a="347242148" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jul 2023 23:50:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10781"; a="1056698073" X-IronPort-AV: E=Sophos;i="6.01,230,1684825200"; d="scan'208";a="1056698073" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jul 2023 23:49:39 -0700 From: "Huang, Ying" To: Alistair Popple Cc: Andrew Morton , , , , , , "Aneesh Kumar K . V" , Wei Xu , Dan Williams , Dave Hansen , "Davidlohr Bueso" , Johannes Weiner , "Jonathan Cameron" , Michal Hocko , Yang Shi , Rafael J Wysocki Subject: Re: [PATCH RESEND 3/4] acpi, hmat: calculate abstract distance with HMAT In-Reply-To: <87ila8zo80.fsf@nvdebian.thelocal> (Alistair Popple's message of "Tue, 25 Jul 2023 12:45:01 +1000") References: <20230721012932.190742-1-ying.huang@intel.com> <20230721012932.190742-4-ying.huang@intel.com> <87ila8zo80.fsf@nvdebian.thelocal> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) Date: Tue, 25 Jul 2023 14:47:47 +0800 Message-ID: <87h6psxzak.fsf@yhuang6-desk2.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Queue-Id: 5FD9CC0008 X-Rspam-User: X-Stat-Signature: aop5pc5kq8bpepij7otfsok88idmt69a X-Rspamd-Server: rspam03 X-HE-Tag: 1690267812-343840 X-HE-Meta: U2FsdGVkX1+AQMRyrpuVbLNlsRwRHRgsrJ82fL7WPLEnbozeUKRsUsJSvJMuX2k1mAr/nbW2R766VuzsaWbn+qZd+2yYdoLV5GoD4otG3JPdoBBxhk9u9q5KP0ijXJo/RnRSEScGs7rNc0afaLxDIwGIt2DRL5AamRCdCEKEE+ssE7iIH4gBpCrOeAvOJGnE5zoh/LVoqYL6BzsC05QZa/46gk+AkygUOdzkZG8MShFT05qLeuNriI5D+n4CeilIyCyTs1Aro8VdLf1MYSukNXZtahJFopuqFijEev2/tWysKvciU1opCnCnUo49DRWPgxcm/5qfgfk7SDg2n1aPVrSE5hzP3di1qPqSzY6BfUhfsKPnNfkihSiT0s+rNkPhrhCpywR56RrRdNUhPlL31p+mzzi9DqrUkFfmcjcn/ry/qj/L9HFgfOb2wpkvJLusWbnxetjPMqtxMyRvPzi47ZzzDDiT/gBGpdE9RhjD5B/EFsmCJGSG/VK4rFJmJMyIMnPZ7wYyTp3LiI8CRubTxt2WAIrfM5Iz+pAoxjTwr5XNiDcYgnvwIseFKXYCJeov9OM3JNPFf2WvTUh3Xkn4lpDtH8bMJiimjRNbvDzsibHSM9WYKig+EJ6tgg2nzy0oDlnAP/EqgqhwqBGl151oAIaV2+JhIWdWTsRVA703ajThx0F3MMpGdHJi1EY4xjmW+yptRL7QbHunBbzfNFnEERIYg3rTh49/6V8SwAa6hC2b7Zkc2aLzEMx4suKt5JzIMH9pIBrN5O5VJA2eYd8w/QtGuZbMCV7GMK9P+1s2yN8N8UxCszxwpGE4TAwmKs+pTXBDq/7Ca6MhWP3jVsiidFibBBqZEeobADCSVkb01hnPccSTkxeHyvUHf0xsnIMu5s1FGHWjL8/uIP4HO0XEajQYhQyPKr2QghOq75eA4909Gz99Zlky9GPScRLQyx9lm6GxO7RSDVXR22Ys0+K 0kV1FE49 cWnnRtaRsE56g8obmiajx4PIymhNUXksnaBPv5SVwAlPiMJwXq/TT8zbhnhbW6Ec0TIsIyW+GdoL6+W7u5eEPw/DGaP7w3ZxC5z98FQ6YrJEwJ5u2fNot2iSZAanbCgP/xdzJd8OGwBZb29TuJQOWEbnJZKrdQrgRoVV2SMKwVt8Z4I8igoLXb9zOQcYjfHDkN/bnzK+Z/aLntuwujClBfRi5PdsztjPQ3lR4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Alistair Popple writes: > Huang Ying writes: > >> A memory tiering abstract distance calculation algorithm based on ACPI >> HMAT is implemented. The basic idea is as follows. >> >> The performance attributes of system default DRAM nodes are recorded >> as the base line. Whose abstract distance is MEMTIER_ADISTANCE_DRAM. >> Then, the ratio of the abstract distance of a memory node (target) to >> MEMTIER_ADISTANCE_DRAM is scaled based on the ratio of the performance >> attributes of the node to that of the default DRAM nodes. > > The problem I encountered here with the calculations is that HBM memory > ended up in a lower-tiered node which isn't what I wanted (at least when > that HBM is attached to a GPU say). I have tested the series on a server machine with HBM (pure HBM, not attached to a GPU). Where, HBM is placed in a higher tier than DRAM. > I suspect this is because the calculations are based on the CPU > point-of-view (access1) which still sees lower bandwidth to remote HBM > than local DRAM, even though the remote GPU has higher bandwidth access > to that memory. Perhaps we need to be considering access0 as well? > Ie. HBM directly attached to a generic initiator should be in a higher > tier regardless of CPU access characteristics? What's your requirements for memory tiers on the machine? I guess you want to put GPU attache HBM in a higher tier and put DRAM in a lower tier. So, cold HBM pages can be demoted to DRAM when there are memory pressure on HBM? This sounds reasonable from GPU point of view. The above requirements may be satisfied via calculating abstract distance based on access0 (or combined with access1). But I suspect this will be a general solution. I guess that any memory devices that are used mainly by the memory initiators other than CPUs want to put themselves in a higher memory tier than DRAM, regardless of its access0. One solution is to put GPU HBM in the highest memory tier (with smallest abstract distance) always in GPU device driver regardless its HMAT performance attributes. Is it possible? > That said I'm not entirely convinced the HMAT tables I'm testing against > are accurate/complete. -- Best Regards, Huang, Ying