From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7965C433F5 for ; Thu, 12 May 2022 01:42:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 179996B0074; Wed, 11 May 2022 21:42:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 127FC6B0075; Wed, 11 May 2022 21:42:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE4546B0078; Wed, 11 May 2022 21:42:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DA0216B0074 for ; Wed, 11 May 2022 21:42:51 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A5D6B602E1 for ; Thu, 12 May 2022 01:42:51 +0000 (UTC) X-FDA: 79455392142.07.EBBD567 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by imf28.hostedemail.com (Postfix) with ESMTP id 55256C009A for ; Thu, 12 May 2022 01:42:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652319770; x=1683855770; h=message-id:subject:from:to:cc:date:in-reply-to: references:mime-version:content-transfer-encoding; bh=nqS2Yo8wWxSVKw+tz2+wy6FYBMYYW/yKi14tiidRDOI=; b=I65wL9eL5WBOu7sTPCd+EulvQkWeLENNOB1lE6kP/geBEFSTkSCPdc0u wVZYclpNNTxHMJ8NW9c/rtyJhB1rUjTezarTcy2n5SP5QmUiwvWcel0Mq fZf4s/FDL+IiqZ4sRbL6fPZ2FggrkKX2w56t1FhEb7r+63iWHXXGkulD2 F6X40WfkOu7bcEDxMCm8PFGkY5SwJ7BVqUO4UEdw++UWOnXVNob17vf8l hvIceNVv4yUhMHAZ/DK4IFmtkx/PK/XYLwBYiCHT4KKH7RrCyQFLoB2ab B2z+ToRJKeQlwzT0loGYmbTfkjSs1YnMhrzQicd5JVDc1kf4sIp4sghdm Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10344"; a="269531429" X-IronPort-AV: E=Sophos;i="5.91,218,1647327600"; d="scan'208";a="269531429" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 May 2022 18:42:32 -0700 X-IronPort-AV: E=Sophos;i="5.91,218,1647327600"; d="scan'208";a="697821769" Received: from ruonanwa-mobl.ccr.corp.intel.com ([10.254.212.157]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 May 2022 18:42:27 -0700 Message-ID: <0a92d0040edb3b74ac259062d241b8cd28924edf.camel@intel.com> Subject: Re: RFC: Memory Tiering Kernel Interfaces From: "ying.huang@intel.com" To: Wei Xu Cc: "Aneesh Kumar K.V" , Alistair Popple , Yang Shi , Andrew Morton , Dave Hansen , Dan Williams , Linux MM , Greg Thelen , Jagdish Gediya , Linux Kernel Mailing List , Davidlohr Bueso , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Jonathan Cameron , Tim Chen Date: Thu, 12 May 2022 09:42:24 +0800 In-Reply-To: References: <87tua3h5r1.fsf@nvdebian.thelocal> <875ymerl81.fsf@nvdebian.thelocal> <87fslhhb2l.fsf@linux.ibm.com> <68333b21a58604f3fd0e660f1a39921ae22849d8.camel@intel.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=I65wL9eL; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf28.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 134.134.136.24) smtp.mailfrom=ying.huang@intel.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 55256C009A X-Rspam-User: X-Stat-Signature: upbu99withrf797mcas7tin84uzt69ei X-HE-Tag: 1652319750-951909 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 2022-05-11 at 10:07 -0700, Wei Xu wrote: > On Wed, May 11, 2022 at 12:49 AM ying.huang@intel.com > wrote: > > > > On Tue, 2022-05-10 at 22:30 -0700, Wei Xu wrote: > > > On Tue, May 10, 2022 at 4:38 AM Aneesh Kumar K.V > > > wrote: > > > > > > > > Alistair Popple writes: > > > > > > > > > Wei Xu writes: > > > > > > > > > > > On Thu, May 5, 2022 at 5:19 PM Alistair Popple wrote: > > > > > > > > > > > > > > Wei Xu writes: > > > > > > > > > > > > > > [...] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Tiering Hierarchy Initialization > > > > > > > > > > `==============================' > > > > > > > > > > > > > > > > > > > > By default, all memory nodes are in the top tier (N_TOPTIER_MEMORY). > > > > > > > > > > > > > > > > > > > > A device driver can remove its memory nodes from the top tier, e.g. > > > > > > > > > > a dax driver can remove PMEM nodes from the top tier. > > > > > > > > > > > > > > > > > > With the topology built by firmware we should not need this. > > > > > > > > > > > > > > I agree that in an ideal world the hierarchy should be built by firmware based > > > > > > > on something like the HMAT. But I also think being able to override this will be > > > > > > > useful in getting there. Therefore a way of overriding the generated hierarchy > > > > > > > would be good, either via sysfs or kernel boot parameter if we don't want to > > > > > > > commit to a particular user interface now. > > > > > > > > > > > > > > However I'm less sure letting device-drivers override this is a good idea. How > > > > > > > for example would a GPU driver make sure it's node is in the top tier? By moving > > > > > > > every node that the driver does not know about out of N_TOPTIER_MEMORY? That > > > > > > > could get messy if say there were two drivers both of which wanted their node to > > > > > > > be in the top tier. > > > > > > > > > > > > The suggestion is to allow a device driver to opt out its memory > > > > > > devices from the top-tier, not the other way around. > > > > > > > > > > So how would demotion work in the case of accelerators then? In that > > > > > case we would want GPU memory to demote to DRAM, but that won't happen > > > > > if both DRAM and GPU memory are in N_TOPTIER_MEMORY and it seems the > > > > > only override available with this proposal would move GPU memory into a > > > > > lower tier, which is the opposite of what's needed there. > > > > > > > > How about we do 3 tiers now. dax kmem devices can be registered to > > > > tier 3. By default all numa nodes can be registered at tier 2 and HBM or > > > > GPU can be enabled to register at tier 1. ? > > > > > > This makes sense. I will send an updated RFC based on the discussions so far. > > > > Are these tier number fixed? If so, it appears strange that the > > smallest tier number is 0 on some machines, but 1 on some other > > machines. > > When the kernel is configured to allow 3 tiers, we can always show all > the 3 tiers. It is just that some tiers (e.g. tier 0) may be empty on > some machines. I still think that it's better to have no empty tiers for auto-generated memory tiers by kernel. Yes, the tier number will be not absolutely stable, but that only happens during system bootup in practice, so it's not a big issue IMHO. And, I still think it's better to make only N-1 tiers writable for totally N tiers (or even readable). Considering "tier0" is written, how to deal with nodes in "tier0" before but not after writing? One possible way is to put them into "tierN". And during a user customize the tiers, the union of "N tiers" may be not complete. > BTW, the userspace should not assume a specific meaning of a > particular tier id because it can change depending on the number of > tiers that the kernel is configured with. For example, the userspace > should not assume that tier-2 always means PMEM nodes. In a system > with 4 tiers, PMEM nodes may be in tier-3, not tier-2. Yes. This sounds good. Best Regards, Huang, Ying