From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 124F0C433EF for ; Fri, 22 Apr 2022 04:46:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 724E26B0072; Fri, 22 Apr 2022 00:46:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6AE906B0073; Fri, 22 Apr 2022 00:46:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 500D66B0074; Fri, 22 Apr 2022 00:46:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 3D6DB6B0072 for ; Fri, 22 Apr 2022 00:46:27 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1573F241C9 for ; Fri, 22 Apr 2022 04:46:27 +0000 (UTC) X-FDA: 79383278814.21.BBA09F3 Received: from mail-vk1-f178.google.com (mail-vk1-f178.google.com [209.85.221.178]) by imf19.hostedemail.com (Postfix) with ESMTP id 56A871A0018 for ; Fri, 22 Apr 2022 04:46:24 +0000 (UTC) Received: by mail-vk1-f178.google.com with SMTP id b81so3297276vkf.1 for ; Thu, 21 Apr 2022 21:46:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ViZ4rS/eycFdyCL0pmim1stXBOTkqYRh9tN/Tz1sHIA=; b=fRrotN+M96X/Eelm6McwhcNGs52xFGZHTiW76h2ddT2c/TYbQvH5ccIj8OaTROdL+W MN9q014IIJasEDcGKIUYbIunY+HNissSqKbGAtXVV+QxuWWpjp46czpiHMQIOx/C7lvT rItmjIrETBrY7kdi0m2MZGl6i7O/m/YTvy5EVaUYsb37nGD3wSdaRhQpic2KSGoA3lNr luXSHpXDQv9W+iM02bAxJt88HQR7Vtr+nPt6NthQkGdLM2IElaTHHCrqX6dCDcJspDbx d9jZPWObZ/IIdfXARxT5Tanoge/VYoCqMT94wIZI3zsquoaYNKRvpFMqpfzPnmSHgEkR BHdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ViZ4rS/eycFdyCL0pmim1stXBOTkqYRh9tN/Tz1sHIA=; b=fuuUR71DDyvbmRF1Ow0sIKD0JGF5P3CKiydwX5c4nNzv6yD/kVbBZPyUNq6cVUzRxF 4vRd06BuA2Xp+/pfDzlhqMRwAfKfMJ2XYzSIYFW/tioE9gwcg0uQdzx1gzdv0N8peR6K lnP72o0oBlkmMRI/jDO2UPTdkQjv5dT5ZCq6zSDoEiwvjX6GCwCfGKqGwE+iRJSV+CGo bp7T8bhDdoYr/13ECFzEY8k3JsjihWXOY0g4Q2/hbMHQGaUN4yIyE+3BSQNjt8me0Fvh L1JHXLihhphR7ezJU0tMrMNQnJENBTTtxr/tkbDZctoDbDMoFKcWQOw8SE+yfNdkncbf XFCg== X-Gm-Message-State: AOAM531VYoMzd2v6GpxarldqoInW2D4z4bzXwvbxfhxrpYGy61cTcab3 ulhJYSFsExvlpew1INf8a6ZQLuIH9WRRQBX/dT3DGA== X-Google-Smtp-Source: ABdhPJyMwHYLSq/3ATEA4EIhyyATrlAhCWPP2V/Sp10XawPq3M6fFbNW14/Bj35ITRZ/7byyBalAiA4PmSmI4vw9N80= X-Received: by 2002:a1f:38c2:0:b0:349:9667:9232 with SMTP id f185-20020a1f38c2000000b0034996679232mr994981vka.16.1650602785473; Thu, 21 Apr 2022 21:46:25 -0700 (PDT) MIME-Version: 1.0 References: <20220413092206.73974-1-jvgediya@linux.ibm.com> <6365983a8fbd8c325bb18959c51e9417fd821c91.camel@intel.com> <610ccaad03f168440ce765ae5570634f3b77555e.camel@intel.com> In-Reply-To: <610ccaad03f168440ce765ae5570634f3b77555e.camel@intel.com> From: Wei Xu Date: Thu, 21 Apr 2022 21:46:14 -0700 Message-ID: Subject: Re: [PATCH v2 0/5] mm: demotion: Introduce new node state N_DEMOTION_TARGETS To: "ying.huang@intel.com" Cc: Yang Shi , Jagdish Gediya , Linux MM , Linux Kernel Mailing List , Andrew Morton , "Aneesh Kumar K.V" , Baolin Wang , Dave Hansen , Dan Williams , Greg Thelen Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: g43p9rya1hn3cyyttazph4hori9rt318 X-Rspam-User: Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=fRrotN+M; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of weixugc@google.com designates 209.85.221.178 as permitted sender) smtp.mailfrom=weixugc@google.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 56A871A0018 X-HE-Tag: 1650602784-771333 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Apr 21, 2022 at 5:58 PM ying.huang@intel.com wrote: > > On Thu, 2022-04-21 at 11:26 -0700, Wei Xu wrote: > > On Thu, Apr 21, 2022 at 12:45 AM ying.huang@intel.com > > wrote: > > > > > > On Thu, 2022-04-21 at 00:29 -0700, Wei Xu wrote: > > > > On Thu, Apr 21, 2022 at 12:08 AM ying.huang@intel.com > > > > wrote: > > > > > > > > > > On Wed, 2022-04-20 at 23:49 -0700, Wei Xu wrote: > > > > > > On Wed, Apr 20, 2022 at 11:24 PM ying.huang@intel.com > > > > > > wrote: > > > > > > > > > > > > > > On Wed, 2022-04-20 at 22:41 -0700, Wei Xu wrote: > > > > > > > > On Wed, Apr 20, 2022 at 8:12 PM Yang Shi wrote: > > > > > > > > > > > > > > > > > > On Thu, Apr 14, 2022 at 12:00 AM ying.huang@intel.com > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > On Wed, 2022-04-13 at 14:52 +0530, Jagdish Gediya wrote: > > > > > > > > > > > Current implementation to find the demotion targets works > > > > > > > > > > > based on node state N_MEMORY, however some systems may have > > > > > > > > > > > dram only memory numa node which are N_MEMORY but not the > > > > > > > > > > > right choices as demotion targets. > > > > > > > > > > > > > > > > > > > > > > This patch series introduces the new node state > > > > > > > > > > > N_DEMOTION_TARGETS, which is used to distinguish the nodes which > > > > > > > > > > > can be used as demotion targets, node_states[N_DEMOTION_TARGETS] > > > > > > > > > > > is used to hold the list of nodes which can be used as demotion > > > > > > > > > > > targets, support is also added to set the demotion target > > > > > > > > > > > list from user space so that default behavior can be overridden. > > > > > > > > > > > > > > > > > > > > It appears that your proposed user space interface cannot solve all > > > > > > > > > > problems. For example, for system as follows, > > > > > > > > > > > > > > > > > > > > Node 0 & 2 are cpu + dram nodes and node 1 are slow memory node near > > > > > > > > > > node 0, > > > > > > > > > > > > > > > > > > > > available: 3 nodes (0-2) > > > > > > > > > > node 0 cpus: 0 1 > > > > > > > > > > node 0 size: n MB > > > > > > > > > > node 0 free: n MB > > > > > > > > > > node 1 cpus: > > > > > > > > > > node 1 size: n MB > > > > > > > > > > node 1 free: n MB > > > > > > > > > > node 2 cpus: 2 3 > > > > > > > > > > node 2 size: n MB > > > > > > > > > > node 2 free: n MB > > > > > > > > > > node distances: > > > > > > > > > > node 0 1 2 > > > > > > > > > > 0: 10 40 20 > > > > > > > > > > 1: 40 10 80 > > > > > > > > > > 2: 20 80 10 > > > > > > > > > > > > > > > > > > > > Demotion order 1: > > > > > > > > > > > > > > > > > > > > node demotion_target > > > > > > > > > > 0 1 > > > > > > > > > > 1 X > > > > > > > > > > 2 X > > > > > > > > > > > > > > > > > > > > Demotion order 2: > > > > > > > > > > > > > > > > > > > > node demotion_target > > > > > > > > > > 0 1 > > > > > > > > > > 1 X > > > > > > > > > > 2 1 > > > > > > > > > > > > > > > > > > > > The demotion order 1 is preferred if we want to reduce cross-socket > > > > > > > > > > traffic. While the demotion order 2 is preferred if we want to take > > > > > > > > > > full advantage of the slow memory node. We can take any choice as > > > > > > > > > > automatic-generated order, while make the other choice possible via user > > > > > > > > > > space overridden. > > > > > > > > > > > > > > > > > > > > I don't know how to implement this via your proposed user space > > > > > > > > > > interface. How about the following user space interface? > > > > > > > > > > > > > > > > > > > > 1. Add a file "demotion_order_override" in > > > > > > > > > > /sys/devices/system/node/ > > > > > > > > > > > > > > > > > > > > 2. When read, "1" is output if the demotion order of the system has been > > > > > > > > > > overridden; "0" is output if not. > > > > > > > > > > > > > > > > > > > > 3. When write "1", the demotion order of the system will become the > > > > > > > > > > overridden mode. When write "0", the demotion order of the system will > > > > > > > > > > become the automatic mode and the demotion order will be re-generated. > > > > > > > > > > > > > > > > > > > > 4. Add a file "demotion_targets" for each node in > > > > > > > > > > /sys/devices/system/node/nodeX/ > > > > > > > > > > > > > > > > > > > > 5. When read, the demotion targets of nodeX will be output. > > > > > > > > > > > > > > > > > > > > 6. When write a node list to the file, the demotion targets of nodeX > > > > > > > > > > will be set to the written nodes. And the demotion order of the system > > > > > > > > > > will become the overridden mode. > > > > > > > > > > > > > > > > > > TBH I don't think having override demotion targets in userspace is > > > > > > > > > quite useful in real life for now (it might become useful in the > > > > > > > > > future, I can't tell). Imagine you manage hundred thousands of > > > > > > > > > machines, which may come from different vendors, have different > > > > > > > > > generations of hardware, have different versions of firmware, it would > > > > > > > > > be a nightmare for the users to configure the demotion targets > > > > > > > > > properly. So it would be great to have the kernel properly configure > > > > > > > > > it *without* intervening from the users. > > > > > > > > > > > > > > > > > > So we should pick up a proper default policy and stick with that > > > > > > > > > policy unless it doesn't work well for the most workloads. I do > > > > > > > > > understand it is hard to make everyone happy. My proposal is having > > > > > > > > > every node in the fast tier has a demotion target (at least one) if > > > > > > > > > the slow tier exists sounds like a reasonable default policy. I think > > > > > > > > > this is also the current implementation. > > > > > > > > > > > > > > > > > > > > > > > > > This is reasonable. I agree that with a decent default policy, > > > > > > > > > > > > > > > > > > > > > > I agree that a decent default policy is important. As that was enhanced > > > > > > > in [1/5] of this patchset. > > > > > > > > > > > > > > > the > > > > > > > > overriding of per-node demotion targets can be deferred. The most > > > > > > > > important problem here is that we should allow the configurations > > > > > > > > where memory-only nodes are not used as demotion targets, which this > > > > > > > > patch set has already addressed. > > > > > > > > > > > > > > Do you mean the user space interface proposed by [3/5] of this patchset? > > > > > > > > > > > > Yes. > > > > > > > > > > > > > IMHO, if we want to add a user space interface, I think that it should > > > > > > > be powerful enough to address all existing issues and some potential > > > > > > > future issues, so that it can be stable. I don't think it's a good idea > > > > > > > to define a partial user space interface that works only for a specific > > > > > > > use case and cannot be extended for other use cases. > > > > > > > > > > > > I actually think that they can be viewed as two separate problems: one > > > > > > is to define which nodes can be used as demotion targets (this patch > > > > > > set), and the other is how to initialize the per-node demotion path > > > > > > (node_demotion[]). We don't have to solve both problems at the same > > > > > > time. > > > > > > > > > > > > If we decide to go with a per-node demotion path customization > > > > > > interface to indirectly set N_DEMOTION_TARGETS, I'd prefer that there > > > > > > is a single global control to turn off all demotion targets (for the > > > > > > machines that don't use memory-only nodes for demotion). > > > > > > > > > > > > > > > > There's one already. In commit 20b51af15e01 ("mm/migrate: add sysfs > > > > > interface to enable reclaim migration"), a sysfs interface > > > > > > > > > > /sys/kernel/mm/numa/demotion_enabled > > > > > > > > > > is added to turn off all demotion targets. > > > > > > > > IIUC, this sysfs interface only turns off demotion-in-reclaim. It > > > > will be even cleaner if we have an easy way to clear node_demotion[] > > > > and N_DEMOTION_TARGETS so that the userspace (post-boot agent, not > > > > init scripts) can know that the machine doesn't even have memory > > > > tiering hardware enabled. > > > > > > > > > > What is the difference? Now we have no interface to show demotion > > > targets of a node. That is in-kernel only. What is memory tiering > > > hardware? The Optane PMEM? Some information for it is available via > > > ACPI HMAT table. > > > > > > Except demotion-in-reclaim, what else do you care about? > > > > There is a difference: one is to indicate the availability of the > > memory tiering hardware and the other is to indicate whether > > transparent kernel-driven demotion from the reclaim path is activated. > > With /sys/devices/system/node/demote_targets or the per-node demotion > > target interface, the userspace can figure out the memory tiering > > topology abstracted by the kernel. It is possible to use > > application-guided demotion without having to enable reclaim-based > > demotion in the kernel. Logically it is also cleaner to me to > > decouple the tiering node representation from the actual demotion > > mechanism enablement. > > I am confused here. It appears that you need a way to expose the > automatic generated demotion order from kernel to user space interface. > We can talk about that if you really need it. > > But [2-5/5] of this patchset is to override the automatic generated > demotion order from user space to kernel interface. As a side effect of allowing user space to override the default set of demotion target nodes, it also provides a sysfs interface to allow userspace to read which nodes are currently being designated as demotion targets. The initialization of demotion targets is expected to complete during boot (either by kernel or via an init script). After that, the userspace processes (e.g. proactive tiering daemon or tiering-aware applications) can query this sysfs interface to know if there are any tiering nodes present and act accordingly. It would be even better to expose the per-node demotion order (node_demotion[]) via the sysfs interface (e.g. /sys/devices/system/node/nodeX/demotion_targets as you have suggested). It can be read-only until there are good use cases to require overriding the per-node demotion order.