From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BF20C2BB41 for ; Tue, 16 Aug 2022 08:26:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0425A6B0073; Tue, 16 Aug 2022 04:26:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F0D928D0002; Tue, 16 Aug 2022 04:26:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D871E8D0001; Tue, 16 Aug 2022 04:26:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C32416B0073 for ; Tue, 16 Aug 2022 04:26:17 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 97E9616068A for ; Tue, 16 Aug 2022 08:26:17 +0000 (UTC) X-FDA: 79804773594.08.F715AA1 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf30.hostedemail.com (Postfix) with ESMTP id 42D57801A8 for ; Tue, 16 Aug 2022 08:26:17 +0000 (UTC) Received: by mail-pj1-f43.google.com with SMTP id t2-20020a17090a4e4200b001f21572f3a4so8985674pjl.0 for ; Tue, 16 Aug 2022 01:26:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc; bh=dRB3ek2IpNC0KMnCzrclpPy72oUpKT4lQZi90BQ6jjg=; b=QJFyPrueNhZii7jHqZS60xLm2PgDusxYNWy+cySSKI3fXmsDG3G5JXPgZkPhn/6HY1 pZgM3RycunhesOwm+3sEdwmXfpmfCiGBXxfZoNcGqMpdiSEa0QdTleulp5w/FGFb/xQ0 Dfox8I3yRlhSyobXzDNQkF7RcbrOJEFWOY3o4/EbNo2yUGsuO0FNA+MOYampfHkS8Wip Fip69USX/7G9JkKWbsdxyGv3IlcNY5VozJ8qGBeJkbBTN70GHu1lYcX0p6zIC9ed8+fv tQ1mcJ98Afw95XG0qtzmoUL3/OqyDSaNdIbfqff//onYfcI+5XHKnHzfNADR/vf2b+ki UeqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc; bh=dRB3ek2IpNC0KMnCzrclpPy72oUpKT4lQZi90BQ6jjg=; b=DS8RALO2Bz4/8vOCybfXuosq6/ZmlVuLB3FQvXeYTT+iveqkYQewTP7eMMmVlKZHta lRdtUIMLqIYxOXheXmHfE3uF4FJLJWwPR/tQiLcoztq1YKd/IbwNCg21i7iV7GYotHXQ tqdGn+FWZuBB3xlDo2n9eAQDYHgE4i9ESQTof1DI/JXjTR/6YUWbPONXCiFYFkXJ8Ga4 Uuv0mU8lNpq/puUfIw3nmPyEsMzgfO0Z0Te/5kXu+71w7HTcD5oICGccAVCLHQtLodoF gi4sxFk/OMGEyFQMN7bTqkjnjYd8u4WcO2u119xCmvtDu+JuY7487+scCsN5+mOeOQ2s 1TNQ== X-Gm-Message-State: ACgBeo1+qV8hQ+LfryEsq/xpS593fi+g7LFRLNiX0VoMv8DARfR16hZ6 cYJYkwv0bVqOueUakUuiKMbbeoRi347DXNpm7a0= X-Google-Smtp-Source: AA6agR6XkfYmfBwjv/IdordNIWylrsd5Y89YxpRHhuQFNJZLhdsj+xDFduk4v11d5WmXI0Ztk31LBOPt/YWg/uNJKm8= X-Received: by 2002:a17:90b:190b:b0:1fa:a374:f563 with SMTP id mp11-20020a17090b190b00b001faa374f563mr546676pjb.52.1660638376362; Tue, 16 Aug 2022 01:26:16 -0700 (PDT) MIME-Version: 1.0 References: <20220812055710.357820-1-aneesh.kumar@linux.ibm.com> <20220812055710.357820-5-aneesh.kumar@linux.ibm.com> <87wnbacjsh.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: From: huang ying Date: Tue, 16 Aug 2022 16:26:02 +0800 Message-ID: Subject: Re: [PATCH v14 04/10] mm/demotion/dax/kmem: Set node's abstract distance to MEMTIER_DEFAULT_DAX_ADISTANCE To: Bharata B Rao Cc: Aneesh Kumar K V , "Huang, Ying" , linux-mm@kvack.org, akpm@linux-foundation.org, Wei Xu , Yang Shi , Davidlohr Bueso , Tim C Chen , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Johannes Weiner , jvgediya.oss@gmail.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660638377; a=rsa-sha256; cv=none; b=6OjPBATfayTuR9P8OLr4sMDVRaisigMKQRNgRwkLyvCoO8XUDouqNoHH9Rl73Jpk/UHlfh /3szrHiofHxmLm0Fk899JSmsYHYbRvkPaWJc6/PaOwhfRDANZTjcPMQS/kVbV0fmNshw9S P8b73VM7vcXyMXVi8RFxHxNULyyzP28= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=QJFyPrue; spf=pass (imf30.hostedemail.com: domain of huang.ying.caritas@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=huang.ying.caritas@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660638377; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dRB3ek2IpNC0KMnCzrclpPy72oUpKT4lQZi90BQ6jjg=; b=KJaujYo3G1Dd/3z0A9nshLwBOny90kKdbQUS77oDzSBiTD0iwu+BTX2gd6itJCTkPuNjOR qqPFa8CArhLV94Qwjx3MqljsT2qSF64/r8nBR7dGxrhHyBOVRWbcu80jkIba7O3km010xd t6Qg8uniOlVaDLkPewYYHAs3vesndsY= Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=QJFyPrue; spf=pass (imf30.hostedemail.com: domain of huang.ying.caritas@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=huang.ying.caritas@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Stat-Signature: keju8nr9wa4iupntt57joqbyio4ku1dt X-Rspamd-Queue-Id: 42D57801A8 X-Rspamd-Server: rspam06 X-HE-Tag: 1660638377-67712 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Aug 16, 2022 at 4:12 PM Bharata B Rao wrote: > > On 8/16/2022 12:58 PM, huang ying wrote: > > On Tue, Aug 16, 2022 at 1:10 PM Aneesh Kumar K V > > wrote: > >> > >> On 8/15/22 8:09 AM, Huang, Ying wrote: > >>> "Aneesh Kumar K.V" writes: > >>> > > > > [snip] > > > >>>> > >>>> +/* > >>>> + * Default abstract distance assigned to the NUMA node onlined > >>>> + * by DAX/kmem if the low level platform driver didn't initialize > >>>> + * one for this NUMA node. > >>>> + */ > >>>> +#define MEMTIER_DEFAULT_DAX_ADISTANCE (MEMTIER_ADISTANCE_DRAM= * 2) > >>> > >>> If my understanding were correct, this is targeting Optane DCPMM for > >>> now. The measured results in the following paper is, > >>> > >>> https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fa= rxiv.org%2Fpdf%2F2002.06018.pdf&data=3D05%7C01%7Cbharata%40amd.com%7C1c= 5015b55ff849e5275408da7f58e67d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7= C637962317187856589%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2l= uMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3DSxSC8WaKEeT= yfZXoqtI%2FZAoBXXp82PnTeyyavrV%2FGGg%3D&reserved=3D0 > >>> > >>> Section: 2.1 Read/Write Latencies > >>> > >>> " > >>> For read access, the latency of DCPMM was 400.1% higher than that of > >>> DRAM. For write access, it was 407.1% higher. > >>> " > >>> > >>> Section: 2.2 Read/Write Bandwidths > >>> > >>> " > >>> For read access, the throughput of DCPMM was 37.1% of DRAM. For write > >>> access, it was 7.8% > >>> " > >>> > >>> According to the above data, I think the MEMTIER_DEFAULT_DAX_ADISTANC= E > >>> can be "5 * MEMTIER_ADISTANCE_DRAM". > >>> > >> > >> If we look at mapping every 100% increase in latency as a memory tier,= we essentially > >> will have 4 memory tier here. Each memory tier is covering a range of = abstract distance 128. > >> which makes a total adistance increase from MEMTIER_ADISTANCE_DRAM by = 512. This puts > >> DEFAULT_DAX_DISTANCE at 1024 or MEMTIER_ADISTANCE_DRAM * 2 > > > > If my understanding were correct, you are suggesting to use a kind of > > logarithmic mapping from latency to abstract distance? That is, > > > > abstract_distance =3D log2(latency) > > > > While I am suggesting to use a kind of linear mapping from latency to > > abstract distance. That is, > > > > abstract_distance =3D C * latency > > > > I think that linear mapping is easy to understand. > > > > Are there some good reasons to use logarithmic mapping? > > Also, what is the recommendation for using bandwidth measure which > may be available from HMAT for CXL memory? How is bandwidth going > to influence the abstract distance? This is a good question. Per my understanding, latency stands for idle latency by default. But in practice, the latency under some reasonable memory accessing throughput is the "real" latency. So the memory with lower bandwidth should have a larger abstract distance than the memory with higher bandwidth even if the idle latency is the same. But I don't have a perfect formula to combine idle latency and bandwidth into abstract distance. One possibility is to increase abstract distance if the bandwidth of the memory is much lower than that of DRAM. Best Regards, Huang, Ying