From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84FADC433EF for ; Fri, 22 Apr 2022 06:11:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 05E6A6B0072; Fri, 22 Apr 2022 02:11:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 00C906B0073; Fri, 22 Apr 2022 02:11:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC8266B0074; Fri, 22 Apr 2022 02:11:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id CCCAC6B0072 for ; Fri, 22 Apr 2022 02:11:45 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 88CA280953 for ; Fri, 22 Apr 2022 06:11:45 +0000 (UTC) X-FDA: 79383493770.19.5778374 Received: from mail-vs1-f45.google.com (mail-vs1-f45.google.com [209.85.217.45]) by imf30.hostedemail.com (Postfix) with ESMTP id B776680026 for ; Fri, 22 Apr 2022 06:11:41 +0000 (UTC) Received: by mail-vs1-f45.google.com with SMTP id r1so6532046vsi.12 for ; Thu, 21 Apr 2022 23:11:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=V1aO3u/qMMSpwo5UKIZhbaMSF4YXFZkRM2MyJq9hJTU=; b=NfrvjDRy6ONak1CtRJJNdL1SXXmYwPMxvh+wSRPwAksEGqUsOTe0Qd0ueft7WJRukT L+6VDBDiKdq+FMXAdVWCzlG037bSC26j5hDqOSky3Ec2gH1jMIrDQafVyBDTFpoh1+wh b4tfKqmuXUp2iY3iqqRGuvtD6G7LeMRqwV0slUtrK0K1rbmPpMAh9wxYK22FeWAkaOJt pS47FtlkSNO2JVj9IWfh61jIBKrfOxoXM+tHh1w27GXmqP93E9LHaeeRDtr7fj9wmogg 0sYtCC/4yZumjwIMobvqWm89Bwu5m1oY37ID4LxYxh2dJBxQRJXYrsK9WHNrNQMT8Xpn yUVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=V1aO3u/qMMSpwo5UKIZhbaMSF4YXFZkRM2MyJq9hJTU=; b=o5SfyaBpN+fuWCY822LfUMMOom26q4+IfTS1j95AwEaaNpIJ50yqF18jHamsLOb1pN 4EcCN0Hp3c6K6K6kItc0hbb7NZVriBRkA56YOMUkHMCLejTaxiiK8tjaPGXHRZY07flc tMijvMKFVFo7QIMKbJoxOgIppjAJX0u4r5CrqW0jZZZcL+7013U57veuO6l5zVEkccCG bW/n5FENgDEXYic4T6B8pUviJlA+LieCba8R6vKyQ5XRaL1mmfLu92BgHbsqwe6/qHpi 4j126c1T1HhGkhcNY3kWDu7Xut0NSthylqeJD2fVWSyvDbIRHBq8RzjkAyAnB8K3Hvtt 3lWA== X-Gm-Message-State: AOAM5321KX4WL0v0oxUXyiFfDx+a/yc5XihnlxZy6u0vsNUYk6I6V0nF x6jPqWJtaoEJB1ewsZLZXtLteJqOH6jUBSDav/Ed3Q== X-Google-Smtp-Source: ABdhPJwXANrPhAL8Cm4jBkXbYDQsL731Sxs5U0gRiarlpMp+nFA4VBRmCeU6gNu7jy2042ILOllaxNlrk3bUW0nXAxI= X-Received: by 2002:a67:fd0b:0:b0:31b:e36d:31b1 with SMTP id f11-20020a67fd0b000000b0031be36d31b1mr977317vsr.44.1650607903975; Thu, 21 Apr 2022 23:11:43 -0700 (PDT) MIME-Version: 1.0 References: <20220413092206.73974-1-jvgediya@linux.ibm.com> <6365983a8fbd8c325bb18959c51e9417fd821c91.camel@intel.com> <610ccaad03f168440ce765ae5570634f3b77555e.camel@intel.com> <8e31c744a7712bb05dbf7ceb2accf1a35e60306a.camel@intel.com> In-Reply-To: <8e31c744a7712bb05dbf7ceb2accf1a35e60306a.camel@intel.com> From: Wei Xu Date: Thu, 21 Apr 2022 23:11:32 -0700 Message-ID: Subject: Re: [PATCH v2 0/5] mm: demotion: Introduce new node state N_DEMOTION_TARGETS To: "ying.huang@intel.com" Cc: Yang Shi , Jagdish Gediya , Linux MM , Linux Kernel Mailing List , Andrew Morton , "Aneesh Kumar K.V" , Baolin Wang , Dave Hansen , Dan Williams , Greg Thelen Content-Type: multipart/alternative; boundary="000000000000c80b0405dd38193f" X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: B776680026 X-Stat-Signature: 84bt9ops17eouhhuzcyfty3o8itu8g98 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NfrvjDRy; spf=pass (imf30.hostedemail.com: domain of weixugc@google.com designates 209.85.217.45 as permitted sender) smtp.mailfrom=weixugc@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-HE-Tag: 1650607901-311840 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --000000000000c80b0405dd38193f Content-Type: text/plain; charset="UTF-8" On Thu, Apr 21, 2022 at 10:40 PM ying.huang@intel.com wrote: > On Thu, 2022-04-21 at 21:46 -0700, Wei Xu wrote: > > On Thu, Apr 21, 2022 at 5:58 PM ying.huang@intel.com > > wrote: > > > > > > On Thu, 2022-04-21 at 11:26 -0700, Wei Xu wrote: > > > > On Thu, Apr 21, 2022 at 12:45 AM ying.huang@intel.com > > > > wrote: > > > > > > > > > > On Thu, 2022-04-21 at 00:29 -0700, Wei Xu wrote: > > > > > > On Thu, Apr 21, 2022 at 12:08 AM ying.huang@intel.com > > > > > > wrote: > > > > > > > > > > > > > > On Wed, 2022-04-20 at 23:49 -0700, Wei Xu wrote: > > > > > > > > On Wed, Apr 20, 2022 at 11:24 PM ying.huang@intel.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > On Wed, 2022-04-20 at 22:41 -0700, Wei Xu wrote: > > > > > > > > > > On Wed, Apr 20, 2022 at 8:12 PM Yang Shi < > shy828301@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 14, 2022 at 12:00 AM ying.huang@intel.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 2022-04-13 at 14:52 +0530, Jagdish Gediya > wrote: > > > > > > > > > > > > > Current implementation to find the demotion > targets works > > > > > > > > > > > > > based on node state N_MEMORY, however some systems > may have > > > > > > > > > > > > > dram only memory numa node which are N_MEMORY but > not the > > > > > > > > > > > > > right choices as demotion targets. > > > > > > > > > > > > > > > > > > > > > > > > > > This patch series introduces the new node state > > > > > > > > > > > > > N_DEMOTION_TARGETS, which is used to distinguish > the nodes which > > > > > > > > > > > > > can be used as demotion targets, > node_states[N_DEMOTION_TARGETS] > > > > > > > > > > > > > is used to hold the list of nodes which can be > used as demotion > > > > > > > > > > > > > targets, support is also added to set the demotion > target > > > > > > > > > > > > > list from user space so that default behavior can > be overridden. > > > > > > > > > > > > > > > > > > > > > > > > It appears that your proposed user space interface > cannot solve all > > > > > > > > > > > > problems. For example, for system as follows, > > > > > > > > > > > > > > > > > > > > > > > > Node 0 & 2 are cpu + dram nodes and node 1 are slow > memory node near > > > > > > > > > > > > node 0, > > > > > > > > > > > > > > > > > > > > > > > > available: 3 nodes (0-2) > > > > > > > > > > > > node 0 cpus: 0 1 > > > > > > > > > > > > node 0 size: n MB > > > > > > > > > > > > node 0 free: n MB > > > > > > > > > > > > node 1 cpus: > > > > > > > > > > > > node 1 size: n MB > > > > > > > > > > > > node 1 free: n MB > > > > > > > > > > > > node 2 cpus: 2 3 > > > > > > > > > > > > node 2 size: n MB > > > > > > > > > > > > node 2 free: n MB > > > > > > > > > > > > node distances: > > > > > > > > > > > > node 0 1 2 > > > > > > > > > > > > 0: 10 40 20 > > > > > > > > > > > > 1: 40 10 80 > > > > > > > > > > > > 2: 20 80 10 > > > > > > > > > > > > > > > > > > > > > > > > Demotion order 1: > > > > > > > > > > > > > > > > > > > > > > > > node demotion_target > > > > > > > > > > > > 0 1 > > > > > > > > > > > > 1 X > > > > > > > > > > > > 2 X > > > > > > > > > > > > > > > > > > > > > > > > Demotion order 2: > > > > > > > > > > > > > > > > > > > > > > > > node demotion_target > > > > > > > > > > > > 0 1 > > > > > > > > > > > > 1 X > > > > > > > > > > > > 2 1 > > > > > > > > > > > > > > > > > > > > > > > > The demotion order 1 is preferred if we want to > reduce cross-socket > > > > > > > > > > > > traffic. While the demotion order 2 is preferred if > we want to take > > > > > > > > > > > > full advantage of the slow memory node. We can take > any choice as > > > > > > > > > > > > automatic-generated order, while make the other > choice possible via user > > > > > > > > > > > > space overridden. > > > > > > > > > > > > > > > > > > > > > > > > I don't know how to implement this via your proposed > user space > > > > > > > > > > > > interface. How about the following user space > interface? > > > > > > > > > > > > > > > > > > > > > > > > 1. Add a file "demotion_order_override" in > > > > > > > > > > > > /sys/devices/system/node/ > > > > > > > > > > > > > > > > > > > > > > > > 2. When read, "1" is output if the demotion order of > the system has been > > > > > > > > > > > > overridden; "0" is output if not. > > > > > > > > > > > > > > > > > > > > > > > > 3. When write "1", the demotion order of the system > will become the > > > > > > > > > > > > overridden mode. When write "0", the demotion order > of the system will > > > > > > > > > > > > become the automatic mode and the demotion order > will be re-generated. > > > > > > > > > > > > > > > > > > > > > > > > 4. Add a file "demotion_targets" for each node in > > > > > > > > > > > > /sys/devices/system/node/nodeX/ > > > > > > > > > > > > > > > > > > > > > > > > 5. When read, the demotion targets of nodeX will be > output. > > > > > > > > > > > > > > > > > > > > > > > > 6. When write a node list to the file, the demotion > targets of nodeX > > > > > > > > > > > > will be set to the written nodes. And the demotion > order of the system > > > > > > > > > > > > will become the overridden mode. > > > > > > > > > > > > > > > > > > > > > > TBH I don't think having override demotion targets in > userspace is > > > > > > > > > > > quite useful in real life for now (it might become > useful in the > > > > > > > > > > > future, I can't tell). Imagine you manage hundred > thousands of > > > > > > > > > > > machines, which may come from different vendors, have > different > > > > > > > > > > > generations of hardware, have different versions of > firmware, it would > > > > > > > > > > > be a nightmare for the users to configure the demotion > targets > > > > > > > > > > > properly. So it would be great to have the kernel > properly configure > > > > > > > > > > > it *without* intervening from the users. > > > > > > > > > > > > > > > > > > > > > > So we should pick up a proper default policy and stick > with that > > > > > > > > > > > policy unless it doesn't work well for the most > workloads. I do > > > > > > > > > > > understand it is hard to make everyone happy. My > proposal is having > > > > > > > > > > > every node in the fast tier has a demotion target (at > least one) if > > > > > > > > > > > the slow tier exists sounds like a reasonable default > policy. I think > > > > > > > > > > > this is also the current implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is reasonable. I agree that with a decent default > policy, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree that a decent default policy is important. As > that was enhanced > > > > > > > > > in [1/5] of this patchset. > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > overriding of per-node demotion targets can be > deferred. The most > > > > > > > > > > important problem here is that we should allow the > configurations > > > > > > > > > > where memory-only nodes are not used as demotion > targets, which this > > > > > > > > > > patch set has already addressed. > > > > > > > > > > > > > > > > > > Do you mean the user space interface proposed by [3/5] of > this patchset? > > > > > > > > > > > > > > > > Yes. > > > > > > > > > > > > > > > > > IMHO, if we want to add a user space interface, I think > that it should > > > > > > > > > be powerful enough to address all existing issues and some > potential > > > > > > > > > future issues, so that it can be stable. I don't think > it's a good idea > > > > > > > > > to define a partial user space interface that works only > for a specific > > > > > > > > > use case and cannot be extended for other use cases. > > > > > > > > > > > > > > > > I actually think that they can be viewed as two separate > problems: one > > > > > > > > is to define which nodes can be used as demotion targets > (this patch > > > > > > > > set), and the other is how to initialize the per-node > demotion path > > > > > > > > (node_demotion[]). We don't have to solve both problems at > the same > > > > > > > > time. > > > > > > > > > > > > > > > > If we decide to go with a per-node demotion path > customization > > > > > > > > interface to indirectly set N_DEMOTION_TARGETS, I'd prefer > that there > > > > > > > > is a single global control to turn off all demotion targets > (for the > > > > > > > > machines that don't use memory-only nodes for demotion). > > > > > > > > > > > > > > > > > > > > > > There's one already. In commit 20b51af15e01 ("mm/migrate: add > sysfs > > > > > > > interface to enable reclaim migration"), a sysfs interface > > > > > > > > > > > > > > /sys/kernel/mm/numa/demotion_enabled > > > > > > > > > > > > > > is added to turn off all demotion targets. > > > > > > > > > > > > IIUC, this sysfs interface only turns off demotion-in-reclaim. > It > > > > > > will be even cleaner if we have an easy way to clear > node_demotion[] > > > > > > and N_DEMOTION_TARGETS so that the userspace (post-boot agent, > not > > > > > > init scripts) can know that the machine doesn't even have memory > > > > > > tiering hardware enabled. > > > > > > > > > > > > > > > > What is the difference? Now we have no interface to show demotion > > > > > targets of a node. That is in-kernel only. What is memory tiering > > > > > hardware? The Optane PMEM? Some information for it is available > via > > > > > ACPI HMAT table. > > > > > > > > > > Except demotion-in-reclaim, what else do you care about? > > > > > > > > There is a difference: one is to indicate the availability of the > > > > memory tiering hardware and the other is to indicate whether > > > > transparent kernel-driven demotion from the reclaim path is > activated. > > > > With /sys/devices/system/node/demote_targets or the per-node demotion > > > > target interface, the userspace can figure out the memory tiering > > > > topology abstracted by the kernel. It is possible to use > > > > application-guided demotion without having to enable reclaim-based > > > > demotion in the kernel. Logically it is also cleaner to me to > > > > decouple the tiering node representation from the actual demotion > > > > mechanism enablement. > > > > > > I am confused here. It appears that you need a way to expose the > > > automatic generated demotion order from kernel to user space interface. > > > We can talk about that if you really need it. > > > > > > But [2-5/5] of this patchset is to override the automatic generated > > > demotion order from user space to kernel interface. > > > > As a side effect of allowing user space to override the default set of > > demotion target nodes, it also provides a sysfs interface to allow > > userspace to read which nodes are currently being designated as > > demotion targets. > > > > The initialization of demotion targets is expected to complete during > > boot (either by kernel or via an init script). After that, the > > userspace processes (e.g. proactive tiering daemon or tiering-aware > > applications) can query this sysfs interface to know if there are any > > tiering nodes present and act accordingly. > > > > It would be even better to expose the per-node demotion order > > (node_demotion[]) via the sysfs interface (e.g. > > /sys/devices/system/node/nodeX/demotion_targets as you have > > suggested). It can be read-only until there are good use cases to > > require overriding the per-node demotion order. > > I am OK to expose the system demotion order to user space. For example, > via /sys/devices/system/node/nodeX/demotion_targets, but read-only. > Sounds good. We can send out a patch for such a read-only interface. > But if we want to add functionality to override system demotion order, > we need to consider the user space interface carefully, at least after > collecting all requirement so far. I don't think the interface proposed > in [2-5/5] of this patchset is sufficient or extensible enough. > The current proposed interface should be sufficient to override which nodes can serve as demotion targets. I agree that it is not sufficient if userspace wants to redefine the per-node demotion targets and a suitable user space interface for that purpose needs to be designed carefully. I also agree that it is better to move out patch 1/5 from this patchset. Best Regards, > Huang, Ying > > > > --000000000000c80b0405dd38193f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Apr 21, 2022 at 10:40 PM ying.huang@intel.com<= /a> <ying.huan= g@intel.com> wrote:
On Thu, 2022-04-21 at 21:46 -0700, Wei Xu wrote:
> On Thu, Apr 21, 2022 at 5:58 PM ying.huang@intel.com
> <ying.hua= ng@intel.com> wrote:
> >
> > On Thu, 2022-04-21 at 11:26 -0700, Wei Xu wrote:
> > > On Thu, Apr 21, 2022 at 12:45 AM ying.huang@intel.com
> > > <ying.huang@intel.com> wrote:
> > > >
> > > > On Thu, 2022-04-21 at 00:29 -0700, Wei Xu wrote:
> > > > > On Thu, Apr 21, 2022 at 12:08 AM ying.huang@intel.com
> > > > > <ying.huang@intel.com> wrote:
> > > > > >
> > > > > > On Wed, 2022-04-20 at 23:49 -0700, Wei Xu wro= te:
> > > > > > > On Wed, Apr 20, 2022 at 11:24 PM ying.huang@intel.com=
> > > > > > > <ying.huang@intel.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, 2022-04-20 at 22:41 -0700, = Wei Xu wrote:
> > > > > > > > > On Wed, Apr 20, 2022 at 8:12 P= M Yang Shi <shy= 828301@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 14, 2022 at 1= 2:00 AM ying.huan= g@intel.com
> > > > > > > > > > <ying.huang@intel.com> wrote: > > > > > > > > > > >
> > > > > > > > > > > On Wed, 2022-04-13 a= t 14:52 +0530, Jagdish Gediya wrote:
> > > > > > > > > > > > Current impleme= ntation to find the demotion targets works
> > > > > > > > > > > > based on node s= tate N_MEMORY, however some systems may have
> > > > > > > > > > > > dram only memor= y numa node which are N_MEMORY but not the
> > > > > > > > > > > > right choices a= s demotion targets.
> > > > > > > > > > > >
> > > > > > > > > > > > This patch seri= es introduces the new node state
> > > > > > > > > > > > N_DEMOTION_TARG= ETS, which is used to distinguish the nodes which
> > > > > > > > > > > > can be used as = demotion targets, node_states[N_DEMOTION_TARGETS]
> > > > > > > > > > > > is used to hold= the list of nodes which can be used as demotion
> > > > > > > > > > > > targets, suppor= t is also added to set the demotion target
> > > > > > > > > > > > list from user = space so that default behavior can be overridden.
> > > > > > > > > > >
> > > > > > > > > > > It appears that your= proposed user space interface cannot solve all
> > > > > > > > > > > problems.=C2=A0 For = example, for system as follows,
> > > > > > > > > > >
> > > > > > > > > > > Node 0 & 2 are c= pu + dram nodes and node 1 are slow memory node near
> > > > > > > > > > > node 0,
> > > > > > > > > > >
> > > > > > > > > > > available: 3 nodes (= 0-2)
> > > > > > > > > > > node 0 cpus: 0 1
> > > > > > > > > > > node 0 size: n MB > > > > > > > > > > > node 0 free: n MB > > > > > > > > > > > node 1 cpus:
> > > > > > > > > > > node 1 size: n MB > > > > > > > > > > > node 1 free: n MB > > > > > > > > > > > node 2 cpus: 2 3
> > > > > > > > > > > node 2 size: n MB > > > > > > > > > > > node 2 free: n MB > > > > > > > > > > > node distances:
> > > > > > > > > > > node=C2=A0 =C2=A00= =C2=A0 =C2=A01=C2=A0 =C2=A02
> > > > > > > > > > > =C2=A0=C2=A00:=C2=A0= 10=C2=A0 40=C2=A0 20
> > > > > > > > > > > =C2=A0=C2=A01:=C2=A0= 40=C2=A0 10=C2=A0 80
> > > > > > > > > > > =C2=A0=C2=A02:=C2=A0= 20=C2=A0 80=C2=A0 10
> > > > > > > > > > >
> > > > > > > > > > > Demotion order 1: > > > > > > > > > > >
> > > > > > > > > > > node=C2=A0 =C2=A0 de= motion_target
> > > > > > > > > > > =C2=A00=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1
> > > > > > > > > > > =C2=A01=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 X
> > > > > > > > > > > =C2=A02=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 X
> > > > > > > > > > >
> > > > > > > > > > > Demotion order 2: > > > > > > > > > > >
> > > > > > > > > > > node=C2=A0 =C2=A0 de= motion_target
> > > > > > > > > > > =C2=A00=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1
> > > > > > > > > > > =C2=A01=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 X
> > > > > > > > > > > =C2=A02=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1
> > > > > > > > > > >
> > > > > > > > > > > The demotion order 1= is preferred if we want to reduce cross-socket
> > > > > > > > > > > traffic.=C2=A0 While= the demotion order 2 is preferred if we want to take
> > > > > > > > > > > full advantage of th= e slow memory node.=C2=A0 We can take any choice as
> > > > > > > > > > > automatic-generated = order, while make the other choice possible via user
> > > > > > > > > > > space overridden. > > > > > > > > > > >
> > > > > > > > > > > I don't know how= to implement this via your proposed user space
> > > > > > > > > > > interface.=C2=A0 How= about the following user space interface?
> > > > > > > > > > >
> > > > > > > > > > > 1. Add a file "= demotion_order_override" in
> > > > > > > > > > > =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0/sys/devices/system/node/
> > > > > > > > > > >
> > > > > > > > > > > 2. When read, "= 1" is output if the demotion order of the system has been
> > > > > > > > > > > overridden; "0&= quot; is output if not.
> > > > > > > > > > >
> > > > > > > > > > > 3. When write "= 1", the demotion order of the system will become the
> > > > > > > > > > > overridden mode.=C2= =A0 When write "0", the demotion order of the system will
> > > > > > > > > > > become the automatic= mode and the demotion order will be re-generated.
> > > > > > > > > > >
> > > > > > > > > > > 4. Add a file "= demotion_targets" for each node in
> > > > > > > > > > > =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0/sys/devices/system/node/nodeX/
> > > > > > > > > > >
> > > > > > > > > > > 5. When read, the de= motion targets of nodeX will be output.
> > > > > > > > > > >
> > > > > > > > > > > 6. When write a node= list to the file, the demotion targets of nodeX
> > > > > > > > > > > will be set to the w= ritten nodes.=C2=A0 And the demotion order of the system
> > > > > > > > > > > will become the over= ridden mode.
> > > > > > > > > >
> > > > > > > > > > TBH I don't think hav= ing override demotion targets in userspace is
> > > > > > > > > > quite useful in real life= for now (it might become useful in the
> > > > > > > > > > future, I can't tell)= . Imagine you manage hundred thousands of
> > > > > > > > > > machines, which may come = from different vendors, have different
> > > > > > > > > > generations of hardware, = have different versions of firmware, it would
> > > > > > > > > > be a nightmare for the us= ers to configure the demotion targets
> > > > > > > > > > properly. So it would be = great to have the kernel properly configure
> > > > > > > > > > it *without* intervening = from the users.
> > > > > > > > > >
> > > > > > > > > > So we should pick up a pr= oper default policy and stick with that
> > > > > > > > > > policy unless it doesn= 9;t work well for the most workloads. I do
> > > > > > > > > > understand it is hard to = make everyone happy. My proposal is having
> > > > > > > > > > every node in the fast ti= er has a demotion target (at least one) if
> > > > > > > > > > the slow tier exists soun= ds like a reasonable default policy. I think
> > > > > > > > > > this is also the current = implementation.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > This is reasonable.=C2=A0 I ag= ree that with a decent default policy,
> > > > > > > > >
> > > > > > > >
> > > > > > > > I agree that a decent default polic= y is important.=C2=A0 As that was enhanced
> > > > > > > > in [1/5] of this patchset.
> > > > > > > >
> > > > > > > > > the
> > > > > > > > > overriding of per-node demotio= n targets can be deferred.=C2=A0 The most
> > > > > > > > > important problem here is that= we should allow the configurations
> > > > > > > > > where memory-only nodes are no= t used as demotion targets, which this
> > > > > > > > > patch set has already addresse= d.
> > > > > > > >
> > > > > > > > Do you mean the user space interfac= e proposed by [3/5] of this patchset?
> > > > > > >
> > > > > > > Yes.
> > > > > > >
> > > > > > > > IMHO, if we want to add a user spac= e interface, I think that it should
> > > > > > > > be powerful enough to address all e= xisting issues and some potential
> > > > > > > > future issues, so that it can be st= able.=C2=A0 I don't think it's a good idea
> > > > > > > > to define a partial user space inte= rface that works only for a specific
> > > > > > > > use case and cannot be extended for= other use cases.
> > > > > > >
> > > > > > > I actually think that they can be viewed= as two separate problems: one
> > > > > > > is to define which nodes can be used as = demotion targets (this patch
> > > > > > > set), and the other is how to initialize= the per-node demotion path
> > > > > > > (node_demotion[]).=C2=A0 We don't ha= ve to solve both problems at the same
> > > > > > > time.
> > > > > > >
> > > > > > > If we decide to go with a per-node demot= ion path customization
> > > > > > > interface to indirectly set N_DEMOTION_T= ARGETS, I'd prefer that there
> > > > > > > is a single global control to turn off a= ll demotion targets (for the
> > > > > > > machines that don't use memory-only = nodes for demotion).
> > > > > > >
> > > > > >
> > > > > > There's one already.=C2=A0 In commit 20b5= 1af15e01 ("mm/migrate: add sysfs
> > > > > > interface to enable reclaim migration"),= a sysfs interface
> > > > > >
> > > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0/sys/kernel/mm/numa/demotion_enabled
> > > > > >
> > > > > > is added to turn off all demotion targets. > > > > >
> > > > > IIUC, this sysfs interface only turns off demotion= -in-reclaim.=C2=A0 It
> > > > > will be even cleaner if we have an easy way to cle= ar node_demotion[]
> > > > > and N_DEMOTION_TARGETS so that the userspace (post= -boot agent, not
> > > > > init scripts) can know that the machine doesn'= t even have memory
> > > > > tiering hardware enabled.
> > > > >
> > > >
> > > > What is the difference?=C2=A0 Now we have no interface = to show demotion
> > > > targets of a node.=C2=A0 That is in-kernel only.=C2=A0 = What is memory tiering
> > > > hardware?=C2=A0 The Optane PMEM?=C2=A0 Some information= for it is available via
> > > > ACPI HMAT table.
> > > >
> > > > Except demotion-in-reclaim, what else do you care about= ?
> > >
> > > There is a difference: one is to indicate the availability o= f the
> > > memory tiering hardware and the other is to indicate whether=
> > > transparent kernel-driven demotion from the reclaim path is = activated.
> > > With /sys/devices/system/node/demote_targets or the per-node= demotion
> > > target interface, the userspace can figure out the memory ti= ering
> > > topology abstracted by the kernel.=C2=A0 It is possible to u= se
> > > application-guided demotion without having to enable reclaim= -based
> > > demotion in the kernel.=C2=A0 Logically it is also cleaner t= o me to
> > > decouple the tiering node representation from the actual dem= otion
> > > mechanism enablement.
> >
> > I am confused here.=C2=A0 It appears that you need a way to expos= e the
> > automatic generated demotion order from kernel to user space inte= rface.
> > We can talk about that if you really need it.
> >
> > But [2-5/5] of this patchset is to override the automatic generat= ed
> > demotion order from user space to kernel interface.
>
> As a side effect of allowing user space to override the default set of=
> demotion target nodes, it also provides a sysfs interface to allow
> userspace to read which nodes are currently being designated as
> demotion targets.
>
> The initialization of demotion targets is expected to complete during<= br> > boot (either by kernel or via an init script).=C2=A0 After that, the > userspace processes (e.g. proactive tiering daemon or tiering-aware > applications) can query this sysfs interface to know if there are any<= br> > tiering nodes present and act accordingly.
>
> It would be even better to expose the per-node demotion order
> (node_demotion[]) via the sysfs interface (e.g.
> /sys/devices/system/node/nodeX/demotion_targets as you have
> suggested). It can be read-only until there are good use cases to
> require overriding the per-node demotion order.

I am OK to expose the system demotion order to user space.=C2=A0 For exampl= e,
via /sys/devices/system/node/nodeX/demotion_targets, but read-only.

Sounds good. We can send out a patch for such = a read-only interface.
=C2=A0
But if we want to add functionality to override system demotion order,
we need to consider the user space interface carefully, at least after
collecting all requirement so far.=C2=A0 I don't think the interface pr= oposed
in [2-5/5] of this patchset is sufficient or extensible enough.

The current proposed interface should be sufficien= t to override which nodes can serve as demotion targets.=C2=A0 I agree that= it is not sufficient if userspace wants to redefine the per-node demotion = targets and a suitable user space interface for that purpose needs to be de= signed carefully.

I also agree that it is better t= o move out patch 1/5 from this patchset.

Best Regards,
Huang, Ying



--000000000000c80b0405dd38193f--