From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B38D6C433F5 for ; Sat, 30 Apr 2022 02:21:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0527F6B0072; Fri, 29 Apr 2022 22:21:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 002BB6B0073; Fri, 29 Apr 2022 22:21:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE6136B0074; Fri, 29 Apr 2022 22:21:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id D26C56B0072 for ; Fri, 29 Apr 2022 22:21:13 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C010260F87 for ; Sat, 30 Apr 2022 02:21:13 +0000 (UTC) X-FDA: 79411943226.26.60A89B6 Received: from mail-vs1-f45.google.com (mail-vs1-f45.google.com [209.85.217.45]) by imf17.hostedemail.com (Postfix) with ESMTP id 8F97840013 for ; Sat, 30 Apr 2022 02:21:03 +0000 (UTC) Received: by mail-vs1-f45.google.com with SMTP id c62so9144962vsc.10 for ; Fri, 29 Apr 2022 19:21:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gK3d26AggPNcsYkk3oP2ItRgWODUL1HBUrPTw0t1O2k=; b=ABePdfQLv1VyGE6WM1Eju5dvGiTwUcCXwrai/Eo1D/0yws3gjSA0UmUmVhD3FO9tsK iDT7t9xUF7rktuxv8dtFMGPHEOFG1iMr2pEELbdg/0dDRqsFdYgqCuv6vYdBVrUkLLqd z8g7L0m33fV58f1QRf19Oi9qBc+LowjnnIV5DxszBFtZfXcH0sVsbRaAXrEbBDqheriv U3Cnkuzi0IILFrPO9fSUOU7ZycFPaMOKCYeKenGzY8mS4xGoaASLEY4nsZ09yxUAtVxA QO4D4ysVCdAKeD2vThLZC9CcsQizcPYTiwfHWyD158KYqWWrc//Vfypv+YPo1Wh65WMd +VMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gK3d26AggPNcsYkk3oP2ItRgWODUL1HBUrPTw0t1O2k=; b=Jr7AUsyodIH18MvgvkytIgCPOavOnAs1wF+k4tuF/wVZMu++ZlsIJT7iIvHjLz23wH ttOeKjVHIUAtpwwLADvPjwB6MfAC0aPKkWmI1DSxypPnJRll2qeMld/6doZO0db3mtYW gAzo56cPj2ux6jqOJ9F+WhTX0Ji9Ido3Z85BDxl/OCgykNtbnLaibsVOCIBUO9G4baHP ZBCLaQZViTXe5XFrVJRuspSYMguxvK2RZInwvgabH1jW9/gC5tq7d/n3iAvSqxWJEIv7 cSWw9XbN9iKhG4CuBQYP4J1JEyAvXsluKZLF6M1m1hk1qd+W1RNLT0cAm/PI2wcC2GQr OA1w== X-Gm-Message-State: AOAM530o19Zpbu6Pw7KmO897oMnkVMIeTb4hGHq9P4FUpkqQ6xv+5C1x SyaNuNE3UDvBhPe7/jQLfPO+pgr3Bc68bJMkk6uIlw== X-Google-Smtp-Source: ABdhPJwNcnlz4agwzIEkIQQstRZ5b59FGzGhsMwoAZbH+KbUAYQ07KPASxhwUL4imyloNgve9kK7zIB/ir16DgQUuC4= X-Received: by 2002:a67:fd0b:0:b0:31b:e36d:31b1 with SMTP id f11-20020a67fd0b000000b0031be36d31b1mr607631vsr.44.1651285272306; Fri, 29 Apr 2022 19:21:12 -0700 (PDT) MIME-Version: 1.0 References: <610ccaad03f168440ce765ae5570634f3b77555e.camel@intel.com> <8e31c744a7712bb05dbf7ceb2accf1a35e60306a.camel@intel.com> <78b5f4cfd86efda14c61d515e4db9424e811c5be.camel@intel.com> <200e95cf36c1642512d99431014db8943fed715d.camel@intel.com> In-Reply-To: From: Wei Xu Date: Fri, 29 Apr 2022 19:21:01 -0700 Message-ID: Subject: Re: [PATCH v2 0/5] mm: demotion: Introduce new node state N_DEMOTION_TARGETS To: "Chen, Tim C" Cc: "Huang, Ying" , Jagdish Gediya , Yang Shi , Dave Hansen , "Williams, Dan J" , Davidlohr Bueso , Linux MM , Linux Kernel Mailing List , Andrew Morton , "Aneesh Kumar K.V" , Baolin Wang , Greg Thelen , MichalHocko , Brice Goglin , Tim Chen Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 8F97840013 X-Stat-Signature: h8s4opeutyiq8rb8bntrmcnwcsu9eerk Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ABePdfQL; spf=pass (imf17.hostedemail.com: domain of weixugc@google.com designates 209.85.217.45 as permitted sender) smtp.mailfrom=weixugc@google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1651285263-618975 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Apr 28, 2022 at 12:30 PM Chen, Tim C wrote: > > > > >On Wed, 2022-04-27 at 09:27 -0700, Wei Xu wrote: > >> On Wed, Apr 27, 2022 at 12:11 AM ying.huang@intel.com > >> wrote: > >> > > >> > On Mon, 2022-04-25 at 09:56 -0700, Wei Xu wrote: > >> > > On Sat, Apr 23, 2022 at 8:02 PM ying.huang@intel.com > >> > > wrote: > >> > > > > >> > > > Hi, All, > >> > > > > >> > > > On Fri, 2022-04-22 at 16:30 +0530, Jagdish Gediya wrote: > >> > > > > >> > > > [snip] > >> > > > > >> > > > > I think it is necessary to either have per node demotion > >> > > > > targets configuration or the user space interface supported by > >> > > > > this patch series. As we don't have clear consensus on how the > >> > > > > user interface should look like, we can defer the per node > >> > > > > demotion target set interface to future until the real need arises. > >> > > > > > >> > > > > Current patch series sets N_DEMOTION_TARGET from dax device > >> > > > > kmem driver, it may be possible that some memory node desired > >> > > > > as demotion target is not detected in the system from dax-device > >kmem probe path. > >> > > > > > >> > > > > It is also possible that some of the dax-devices are not > >> > > > > preferred as demotion target e.g. HBM, for such devices, node > >> > > > > shouldn't be set to N_DEMOTION_TARGETS. In future, Support > >> > > > > should be added to distinguish such dax-devices and not mark > >> > > > > them as N_DEMOTION_TARGETS from the kernel, but for now this > >> > > > > user space interface will be useful to avoid such devices as demotion > >targets. > >> > > > > > >> > > > > We can add read only interface to view per node demotion > >> > > > > targets from /sys/devices/system/node/nodeX/demotion_targets, > >> > > > > remove duplicated /sys/kernel/mm/numa/demotion_target > >> > > > > interface and instead make > >/sys/devices/system/node/demotion_targets writable. > >> > > > > > >> > > > > Huang, Wei, Yang, > >> > > > > What do you suggest? > >> > > > > >> > > > We cannot remove a kernel ABI in practice. So we need to make > >> > > > it right at the first time. Let's try to collect some > >> > > > information for the kernel ABI definitation. > >> > > > > >> > > > The below is just a starting point, please add your requirements. > >> > > > > >> > > > 1. Jagdish has some machines with DRAM only NUMA nodes, but they > >> > > > don't want to use that as the demotion targets. But I don't > >> > > > think this is a issue in practice for now, because > >> > > > demote-in-reclaim is disabled by default. > >> > > > > >> > > > 2. For machines with PMEM installed in only 1 of 2 sockets, for > >> > > > example, > >> > > > > >> > > > Node 0 & 2 are cpu + dram nodes and node 1 are slow memory node > >> > > > near node 0, > >> > > > > >> > > > available: 3 nodes (0-2) > >> > > > node 0 cpus: 0 1 > >> > > > node 0 size: n MB > >> > > > node 0 free: n MB > >> > > > node 1 cpus: > >> > > > node 1 size: n MB > >> > > > node 1 free: n MB > >> > > > node 2 cpus: 2 3 > >> > > > node 2 size: n MB > >> > > > node 2 free: n MB > >> > > > node distances: > >> > > > node 0 1 2 > >> > > > 0: 10 40 20 > >> > > > 1: 40 10 80 > >> > > > 2: 20 80 10 > >> > > > > >> > > > We have 2 choices, > >> > > > > >> > > > a) > >> > > > node demotion targets > >> > > > 0 1 > >> > > > 2 1 > >> > > > > >> > > > b) > >> > > > node demotion targets > >> > > > 0 1 > >> > > > 2 X > >> > > > > >> > > > a) is good to take advantage of PMEM. b) is good to reduce > >> > > > cross-socket traffic. Both are OK as defualt configuration. > >> > > > But some users may prefer the other one. So we need a user > >> > > > space ABI to override the default configuration. > >> > > > >> > > I think 2(a) should be the system-wide configuration and 2(b) can > >> > > be achieved with NUMA mempolicy (which needs to be added to > >demotion). > >> > > >> > Unfortunately, some NUMA mempolicy information isn't available at > >> > demotion time, for example, mempolicy enforced via set_mempolicy() > >> > is for thread. But I think that cpusets can work for demotion. > >> > > >> > > In general, we can view the demotion order in a way similar to > >> > > allocation fallback order (after all, if we don't demote or > >> > > demotion lags behind, the allocations will go to these demotion > >> > > target nodes according to the allocation fallback order anyway). > >> > > If we initialize the demotion order in that way (i.e. every node > >> > > can demote to any node in the next tier, and the priority of the > >> > > target nodes is sorted for each source node), we don't need > >> > > per-node demotion order override from the userspace. What we need > >> > > is to specify what nodes should be in each tier and support NUMA > >mempolicy in demotion. > >> > > >> > This sounds interesting. Tier sounds like a natural and general > >> > concept for these memory types. It's attracting to use it for user > >> > space interface too. For example, we may use that for mem_cgroup > >> > limits of a specific memory type (tier). > >> > > >> > And if we take a look at the N_DEMOTION_TARGETS again from the "tier" > >> > point of view. The nodes are divided to 2 classes via > >> > N_DEMOTION_TARGETS. > >> > > >> > - The nodes without N_DEMOTION_TARGETS are top tier (or tier 0). > >> > > >> > - The nodes with N_DEMOTION_TARGETS are non-top tier (or tier 1, 2, > >> > 3, > >> > ...) > >> > > >> > >> Yes, this is one of the main reasons why we (Google) want this interface. > >> > >> > So, another possibility is to fit N_DEMOTION_TARGETS and its > >> > overriding into "tier" concept too. !N_DEMOTION_TARGETS == TIER0. > >> > > >> > - All nodes start with TIER0 > >> > > >> > - TIER0 can be cleared for some nodes via e.g. kmem driver > >> > > >> > TIER0 node list can be read or overriden by the user space via the > >> > following interface, > >> > > >> > /sys/devices/system/node/tier0 > >> > > >> > In the future, if we want to customize more tiers, we can add tier1, > >> > tier2, tier3, ..... For now, we can add just tier0. That is, the > >> > interface is extensible in the future compared with > >> > .../node/demote_targets. > >> > > >> > >> This more explicit tier definition interface works, too. > >> > > > >In addition to make tiering definition explicit, more importantly, this makes it > >much easier to support more than 2 tiers. For example, for a system with > >HBM (High Bandwidth Memory), CPU+DRAM, DRAM only, and PMEM, that is, > >3 tiers, we can put HBM in tier 0, CPU+DRAM and DRAM only in tier 1, and > >PMEM in tier 2, automatically, or via user space overridding. > >N_DEMOTION_TARGETS isn't natural to be extended to support this. > > Agree with Ying that making the tier explicit is fundamental to the rest of the API. > > I think that the tier organization should come before setting the demotion targets, > not the other way round. > > That makes things clear on the demotion direction, (node in tier X > demote to tier Y, X order is only needed when we truly want that level of control or a demotion > order. Otherwise all the higher numbered tiers are valid targets. > Configuring a tier level for each node is a lot easier than fixing up all > demotion targets for each and every node. > > We can prevent demotion target configuration that goes in the wrong > direction by looking at the tier level. > > Tim > I have just posted a RFC on the tier-oriented memory tiering kernel interface based on the discussions here. The RFC proposes a sysfs interface, /sys/devices/system/node/memory_tiers, to display and override the nodes in each memory tier. It also proposes that we rely on the kernel allocation order to select the demotion target node from the next tier and don't expose a userspace overriding interface for per-node demotion order. The RFC drops the approach of CPU nodes as the top-tier by default, too.