From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F18AC433EF for ; Tue, 12 Jul 2022 01:16:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C02746B00BC; Mon, 11 Jul 2022 21:16:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BB223940010; Mon, 11 Jul 2022 21:16:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A79F76B00C0; Mon, 11 Jul 2022 21:16:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 990416B00BC for ; Mon, 11 Jul 2022 21:16:14 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 698BC2031A for ; Tue, 12 Jul 2022 01:16:14 +0000 (UTC) X-FDA: 79676681868.29.2326C1C Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf06.hostedemail.com (Postfix) with ESMTP id 72F5818002A for ; Tue, 12 Jul 2022 01:16:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657588573; x=1689124573; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=sXhI0+wuScURP0q4s6eeIPcqRoIT4rKN0ec56T4agtc=; b=J+PEJ5nP1nJ9Ad6dr82ZcTLEBSITwdAcTR3Lu0FUoMgoEl1O7E4avDlS jDGIDyOyGEcm6iC3TnKFTtRvkne51Ob/wwaTtHuDjtGp2qs7ZoOwZOq6j 1YyCoaScr18tsX+Nh53eE1rOZiHVE0eMjPzRmprFYoM2Cm4bV0XFU67cl T1+GhQEW7fhzS1PLE1GO757n+aqlDMGfCB9mBq6bT4bWNlBU6KFhyqjjU EEbp7zwh9wT5KXRRpZPodwA56ifWWL9fppJuJ1NVdolzee4My7t4wQ7dN W1mT7eM70wRYEnR+sZWYuiTveoryIJYROa9VqzJ8fswjbYThhrJuvdg2I A==; X-IronPort-AV: E=McAfee;i="6400,9594,10405"; a="346493555" X-IronPort-AV: E=Sophos;i="5.92,264,1650956400"; d="scan'208";a="346493555" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jul 2022 18:16:11 -0700 X-IronPort-AV: E=Sophos;i="5.92,264,1650956400"; d="scan'208";a="569996812" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.239.13.94]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jul 2022 18:16:04 -0700 From: "Huang, Ying" To: Aneesh Kumar K V Cc: linux-mm@kvack.org, akpm@linux-foundation.org, Wei Xu , Yang Shi , Davidlohr Bueso , Tim C Chen , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Johannes Weiner , jvgediya.oss@gmail.com Subject: Re: [PATCH v8 00/12] mm/demotion: Memory tiers and demotion References: <20220704070612.299585-1-aneesh.kumar@linux.ibm.com> <87r130b2rh.fsf@yhuang6-desk2.ccr.corp.intel.com> <60e97fa2-0b89-cf42-5307-5a57c956f741@linux.ibm.com> Date: Tue, 12 Jul 2022 09:16:01 +0800 In-Reply-To: <60e97fa2-0b89-cf42-5307-5a57c956f741@linux.ibm.com> (Aneesh Kumar K. V.'s message of "Tue, 5 Jul 2022 10:52:05 +0530") Message-ID: <87r12r5dwu.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657588574; a=rsa-sha256; cv=none; b=htVSpsYbgFEoXjklSll5+Qbe5Eon4BPpmPzpreGAhQMNE9E87cqNul6NO8yC2FpjE59sW5 O1MTQKWHiHmEF6olWbFnq5wtYhYOFnwHJkpTmXX7cFczEWhoDN7WY31Ksner8GKrPEQnv3 hYtplKzsOnKiRipbbJOsNO72nzCMwU0= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=J+PEJ5nP; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf06.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657588574; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=woqDbMih5j8ILrZ+39+43fXYYUunjDTtL0Y6NDD4fZ4=; b=lT1iz4ImKyvRfBsfDjBaxkXIXRFHHyUiblig5F1u+7uCVB+lO3OfZiomKbT+OFZMUCQsCZ mYaATmgrqafL4mRjDqXLVggeaeyOIQ4AMNY1iLH+4EkvluonQ2rPS0Filv579l8EWLIO0F 4etOamPiZ4b2lZ0OupWRnOUyZI5duB0= X-Stat-Signature: nz1jj7fe81x1qy31b3rqbhrjwk7kb3di Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=J+PEJ5nP; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf06.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=ying.huang@intel.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 72F5818002A X-Rspam-User: X-HE-Tag: 1657588573-791454 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Aneesh Kumar K V writes: > On 7/5/22 9:59 AM, Huang, Ying wrote: >> Hi, Aneesh, >> >> "Aneesh Kumar K.V" writes: >> >>> The current kernel has the basic memory tiering support: Inactive >>> pages on a higher tier NUMA node can be migrated (demoted) to a lower >>> tier NUMA node to make room for new allocations on the higher tier >>> NUMA node. Frequently accessed pages on a lower tier NUMA node can be >>> migrated (promoted) to a higher tier NUMA node to improve the >>> performance. >>> >>> In the current kernel, memory tiers are defined implicitly via a >>> demotion path relationship between NUMA nodes, which is created during >>> the kernel initialization and updated when a NUMA node is hot-added or >>> hot-removed. The current implementation puts all nodes with CPU into >>> the top tier, and builds the tier hierarchy tier-by-tier by establishing >>> the per-node demotion targets based on the distances between nodes. >>> >>> This current memory tier kernel interface needs to be improved for >>> several important use cases: >>> >>> * The current tier initialization code always initializes >>> each memory-only NUMA node into a lower tier. But a memory-only >>> NUMA node may have a high performance memory device (e.g. a DRAM >>> device attached via CXL.mem or a DRAM-backed memory-only node on >>> a virtual machine) and should be put into a higher tier. >>> >>> * The current tier hierarchy always puts CPU nodes into the top >>> tier. But on a system with HBM (e.g. GPU memory) devices, these >>> memory-only HBM NUMA nodes should be in the top tier, and DRAM nodes >>> with CPUs are better to be placed into the next lower tier. >>> >>> * Also because the current tier hierarchy always puts CPU nodes >>> into the top tier, when a CPU is hot-added (or hot-removed) and >>> triggers a memory node from CPU-less into a CPU node (or vice >>> versa), the memory tier hierarchy gets changed, even though no >>> memory node is added or removed. This can make the tier >>> hierarchy unstable and make it difficult to support tier-based >>> memory accounting. >>> >>> * A higher tier node can only be demoted to selected nodes on the >>> next lower tier as defined by the demotion path, not any other >>> node from any lower tier. This strict, hard-coded demotion order >>> does not work in all use cases (e.g. some use cases may want to >>> allow cross-socket demotion to another node in the same demotion >>> tier as a fallback when the preferred demotion node is out of >>> space), and has resulted in the feature request for an interface to >>> override the system-wide, per-node demotion order from the >>> userspace. This demotion order is also inconsistent with the page >>> allocation fallback order when all the nodes in a higher tier are >>> out of space: The page allocation can fall back to any node from >>> any lower tier, whereas the demotion order doesn't allow that. >>> >>> * There are no interfaces for the userspace to learn about the memory >>> tier hierarchy in order to optimize its memory allocations. >>> >>> This patch series make the creation of memory tiers explicit under >>> the control of userspace or device driver. >>> >>> Memory Tier Initialization >>> ========================== >>> >>> By default, all memory nodes are assigned to the default tier with >>> tier ID value 200. >>> >>> A device driver can move up or down its memory nodes from the default >>> tier. For example, PMEM can move down its memory nodes below the >>> default tier, whereas GPU can move up its memory nodes above the >>> default tier. >>> >>> The kernel initialization code makes the decision on which exact tier >>> a memory node should be assigned to based on the requests from the >>> device drivers as well as the memory device hardware information >>> provided by the firmware. >>> >>> Hot-adding/removing CPUs doesn't affect memory tier hierarchy. >>> >>> Memory Allocation for Demotion >>> ============================== >>> This patch series keep the demotion target page allocation logic same. >>> The demotion page allocation pick the closest NUMA node in the >>> next lower tier to the current NUMA node allocating pages from. >>> >>> This will be later improved to use the same page allocation strategy >>> using fallback list. >>> >>> Sysfs Interface: >>> ------------- >>> Listing current list of memory tiers details: >>> >>> :/sys/devices/system/memtier$ ls >>> default_tier max_tier memtier1 power uevent >>> :/sys/devices/system/memtier$ cat default_tier >>> memtier200 >>> :/sys/devices/system/memtier$ cat max_tier >>> 400 >>> :/sys/devices/system/memtier$ >>> >>> Per node memory tier details: >>> >>> For a cpu only NUMA node: >>> >>> :/sys/devices/system/node# cat node0/memtier >>> :/sys/devices/system/node# echo 1 > node0/memtier >>> :/sys/devices/system/node# cat node0/memtier >>> :/sys/devices/system/node# >>> >>> For a NUMA node with memory: >>> :/sys/devices/system/node# cat node1/memtier >>> 1 >>> :/sys/devices/system/node# ls ../memtier/ >>> default_tier max_tier memtier1 power uevent >>> :/sys/devices/system/node# echo 2 > node1/memtier >>> :/sys/devices/system/node# >>> :/sys/devices/system/node# ls ../memtier/ >>> default_tier max_tier memtier1 memtier2 power uevent >>> :/sys/devices/system/node# cat node1/memtier >>> 2 >>> :/sys/devices/system/node# >>> >>> Removing a memory tier >>> :/sys/devices/system/node# cat node1/memtier >>> 2 >>> :/sys/devices/system/node# echo 1 > node1/memtier >> >> Thanks a lot for your patchset. >> >> Per my understanding, we haven't reach consensus on >> >> - how to create the default memory tiers in kernel (via abstract >> distance provided by drivers? Or use SLIT as the first step?) >> >> - how to override the default memory tiers from user space >> >> As in the following thread and email, >> >> https://lore.kernel.org/lkml/YqjZyP11O0yCMmiO@cmpxchg.org/ >> >> I think that we need to finalized on that firstly? > > I did list the proposal here > > https://lore.kernel.org/linux-mm/7b72ccf4-f4ae-cb4e-f411-74d055482026@linux.ibm.com > > So both the kernel default and driver-specific default tiers now become kernel parameters that can be updated > if the user wants a different tier topology. > > All memory that is not managed by a driver gets added to default_memory_tier which got a default value of 200 > > For now, the only driver that is updated is dax kmem, which adds the memory it manages to memory tier 100. > Later as we learn more about the device attributes (HMAT or something similar) that we might want to use > to control the tier assignment this can be a range of memory tiers. > > Based on the above, I guess we can merge what is posted in this series and later fine-tune/update > the memory tier assignment based on device attributes. Sorry for late reply. As the first step, it may be better to skip the parts that we haven't reached consensus yet, for example, the user space interface to override the default memory tiers. And we can use 0, 1, 2 as the default memory tier IDs. We can refine/revise the in-kernel implementation, but we cannot change the user space ABI. Best Regards, Huang, Ying