From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5682C433EF for ; Thu, 12 May 2022 03:14:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 72DE56B0075; Wed, 11 May 2022 23:14:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DDC56B0078; Wed, 11 May 2022 23:14:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57F016B007B; Wed, 11 May 2022 23:14:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 47BF56B0075 for ; Wed, 11 May 2022 23:14:05 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id BE44720B08 for ; Thu, 12 May 2022 03:14:04 +0000 (UTC) X-FDA: 79455622008.06.1B4441B Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf09.hostedemail.com (Postfix) with ESMTP id 6F9AB140097 for ; Thu, 12 May 2022 03:13:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652325242; x=1683861242; h=message-id:subject:from:to:cc:date:in-reply-to: references:mime-version:content-transfer-encoding; bh=caZlbfjgZOant41JIEcgL5HRsSvgBxitC5oYYew431M=; b=h5Fh8qxp4hBuKDpygb0KrDYvifF+i6QiB0fBT08ivEpuQZQFaX3PioBA vE+w3Um/P2W14G9JNxTgAFtX8543b9NkJe01MOwJU6JXeYrzQEf99xrFP CxvxkHGht8uRGdyC/n7Yz5yBch0sfIC0eEXEQiKVUATm20Qd0x28oQRAD HGW7ePWJ8EdeIF42R8lcbsmbrsE9Qv+lpGJ/GKGuRIeGSxG0uZmy/4WXm uEiZs2HwU1sVjH2CbUkA5k+Rnk1VQ+dyWrswk5m4AFX0Gm6H8ThwLuHaP bEJO0O6dBppO5qtNAtu1oM4vWZ+ZiGG4quv1wk8u5Sr99v/EwAmbQwjw8 Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10344"; a="295120156" X-IronPort-AV: E=Sophos;i="5.91,218,1647327600"; d="scan'208";a="295120156" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 May 2022 20:13:58 -0700 X-IronPort-AV: E=Sophos;i="5.91,218,1647327600"; d="scan'208";a="594453592" Received: from ruonanwa-mobl.ccr.corp.intel.com ([10.254.212.157]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 May 2022 20:13:53 -0700 Message-ID: Subject: Re: RFC: Memory Tiering Kernel Interfaces From: "ying.huang@intel.com" To: Wei Xu Cc: "Aneesh Kumar K.V" , Alistair Popple , Yang Shi , Andrew Morton , Dave Hansen , Dan Williams , Linux MM , Greg Thelen , Jagdish Gediya , Linux Kernel Mailing List , Davidlohr Bueso , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Jonathan Cameron , Tim Chen Date: Thu, 12 May 2022 11:13:48 +0800 In-Reply-To: References: <87tua3h5r1.fsf@nvdebian.thelocal> <875ymerl81.fsf@nvdebian.thelocal> <87fslhhb2l.fsf@linux.ibm.com> <68333b21a58604f3fd0e660f1a39921ae22849d8.camel@intel.com> <0a92d0040edb3b74ac259062d241b8cd28924edf.camel@intel.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6F9AB140097 X-Stat-Signature: nndtdq58mrhsz9skgoebd831m55q1zcw X-Rspam-User: Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=h5Fh8qxp; spf=none (imf09.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com X-HE-Tag: 1652325234-833922 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 2022-05-11 at 19:39 -0700, Wei Xu wrote: > On Wed, May 11, 2022 at 6:42 PM ying.huang@intel.com > wrote: > > > > On Wed, 2022-05-11 at 10:07 -0700, Wei Xu wrote: > > > On Wed, May 11, 2022 at 12:49 AM ying.huang@intel.com > > > wrote: > > > > > > > > On Tue, 2022-05-10 at 22:30 -0700, Wei Xu wrote: > > > > > On Tue, May 10, 2022 at 4:38 AM Aneesh Kumar K.V > > > > > wrote: > > > > > > > > > > > > Alistair Popple writes: > > > > > > > > > > > > > Wei Xu writes: > > > > > > > > > > > > > > > On Thu, May 5, 2022 at 5:19 PM Alistair Popple wrote: > > > > > > > > > > > > > > > > > > Wei Xu writes: > > > > > > > > > > > > > > > > > > [...] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Tiering Hierarchy Initialization > > > > > > > > > > > > `==============================' > > > > > > > > > > > > > > > > > > > > > > > > By default, all memory nodes are in the top tier (N_TOPTIER_MEMORY). > > > > > > > > > > > > > > > > > > > > > > > > A device driver can remove its memory nodes from the top tier, e.g. > > > > > > > > > > > > a dax driver can remove PMEM nodes from the top tier. > > > > > > > > > > > > > > > > > > > > > > With the topology built by firmware we should not need this. > > > > > > > > > > > > > > > > > > I agree that in an ideal world the hierarchy should be built by firmware based > > > > > > > > > on something like the HMAT. But I also think being able to override this will be > > > > > > > > > useful in getting there. Therefore a way of overriding the generated hierarchy > > > > > > > > > would be good, either via sysfs or kernel boot parameter if we don't want to > > > > > > > > > commit to a particular user interface now. > > > > > > > > > > > > > > > > > > However I'm less sure letting device-drivers override this is a good idea. How > > > > > > > > > for example would a GPU driver make sure it's node is in the top tier? By moving > > > > > > > > > every node that the driver does not know about out of N_TOPTIER_MEMORY? That > > > > > > > > > could get messy if say there were two drivers both of which wanted their node to > > > > > > > > > be in the top tier. > > > > > > > > > > > > > > > > The suggestion is to allow a device driver to opt out its memory > > > > > > > > devices from the top-tier, not the other way around. > > > > > > > > > > > > > > So how would demotion work in the case of accelerators then? In that > > > > > > > case we would want GPU memory to demote to DRAM, but that won't happen > > > > > > > if both DRAM and GPU memory are in N_TOPTIER_MEMORY and it seems the > > > > > > > only override available with this proposal would move GPU memory into a > > > > > > > lower tier, which is the opposite of what's needed there. > > > > > > > > > > > > How about we do 3 tiers now. dax kmem devices can be registered to > > > > > > tier 3. By default all numa nodes can be registered at tier 2 and HBM or > > > > > > GPU can be enabled to register at tier 1. ? > > > > > > > > > > This makes sense. I will send an updated RFC based on the discussions so far. > > > > > > > > Are these tier number fixed? If so, it appears strange that the > > > > smallest tier number is 0 on some machines, but 1 on some other > > > > machines. > > > > > > When the kernel is configured to allow 3 tiers, we can always show all > > > the 3 tiers. It is just that some tiers (e.g. tier 0) may be empty on > > > some machines. > > > > I still think that it's better to have no empty tiers for auto-generated > > memory tiers by kernel. Yes, the tier number will be not absolutely > > stable, but that only happens during system bootup in practice, so it's > > not a big issue IMHO. > > It should not be hard to hide empty tiers (e.g. tier-0) if we prefer. > But even if tier-0 is empty, we should still keep this tier in the > kernel and not move DRAM nodes into this tier. One reason is that a > HBM node might be hot-added into tier-0 at a later time. > Yes. The in-kernel representation and the user space interface could be different. I have thought something like below. We always make the main memory (DRAM here, CPU local) as tier 0. Then the slower memory will be positive, tier 1, 2, 3, ..., and the faster memory will be negative, tier -1, -2, -3, .... Then, GPU driver can regesiter its memory as tier -1. And the tier number could be more stable. But I'm not sure whether users will be happy with negtive tier number. > > And, I still think it's better to make only N-1 tiers writable for > > totally N tiers (or even readable). Considering "tier0" is written, how > > to deal with nodes in "tier0" before but not after writing? One > > possible way is to put them into "tierN". And during a user customize > > the tiers, the union of "N tiers" may be not complete. > > The sysfs interfaces that I have in mind now are: > > * /sys/devices/system/memtier/memtierN/nodelist (N=0, 1, 2) > > This is read-only to list the memory nodes for a specific tier. > > * /sys/devices/system/node/nodeN/memtier. (N=0, 1, ...,) > > This is a read-write interface. When written, the kernel moves the > node into the user-specified tier. No other nodes are affected. > > This interface should be able to avoid the above issue. Yes. This works too. Best Regards, Huang, Ying > > > BTW, the userspace should not assume a specific meaning of a > > > particular tier id because it can change depending on the number of > > > tiers that the kernel is configured with. For example, the userspace > > > should not assume that tier-2 always means PMEM nodes. In a system > > > with 4 tiers, PMEM nodes may be in tier-3, not tier-2. > > > > Yes. This sounds good. > > > > Best Regards, > > Huang, Ying > >