From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60DA0C433EF for ; Mon, 25 Apr 2022 16:56:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73AA66B007D; Mon, 25 Apr 2022 12:56:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E8E66B007E; Mon, 25 Apr 2022 12:56:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 589696B0080; Mon, 25 Apr 2022 12:56:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 49FBD6B007D for ; Mon, 25 Apr 2022 12:56:47 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 206E027C5F for ; Mon, 25 Apr 2022 16:56:47 +0000 (UTC) X-FDA: 79396005654.20.31D37A8 Received: from mail-vk1-f177.google.com (mail-vk1-f177.google.com [209.85.221.177]) by imf07.hostedemail.com (Postfix) with ESMTP id 0AA8640044 for ; Mon, 25 Apr 2022 16:56:44 +0000 (UTC) Received: by mail-vk1-f177.google.com with SMTP id o132so7490494vko.11 for ; Mon, 25 Apr 2022 09:56:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=BZbP7hGOBctfriTB8Bv6qGqglSYGULdw7kcbcsIBen8=; b=pi6mzCKN7qkYuXMh1tRBib1eV7FLtozTBRl5Db3XbdIAZajY4LF7yfLSiN0QCGGyEb k0/Bscsj+OVWZdgPV6E4NJjtVk6FZPPCRAqx9Pp3YYTJwnFKXT59Ja1uvZpzE9XlyObG xfi79e0nrRbCUwBdjcYvNABZ1goWj50HhyTWBjE04WmoFH/6+iAg/ODNAEugbVEFOHE2 TDrP9msRr1PpBdbeSXbPUCWPb9loXuz1c6DHdmpXo9tmuWRgJ8fQ1eiZtzzbKbbzFIty V4HnEVRKEgULTvHD516q+ab1GzEGgM8xMMZSqmCOTtEH4xwM0HhOBhUj+0TxMGExQERx 3l5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=BZbP7hGOBctfriTB8Bv6qGqglSYGULdw7kcbcsIBen8=; b=v6bvca+b6i26awjplDlMCRKtG8NnDtqHISOwttggfWIO83S6mxuZvhNy3nk6M0h7CK +grs/QUzw2GDtIF8tJYdGvZofxRcZL0FZ2Y8cLtFdatJJPIYVqa309AKciR5omqgn10h 3GhnJzFFggfE2SQEmNE4uAZG4yE3bMFQMwiSEyRhHELN2aVsvWGi8i5OoNzO+1f6njbT dfkirrPxkZYWy79nQZeMsq8zibtsdSwWe8MMMys8Mxk2rzktekgQmH7jzw7lGGzbXIIA beiqIovTyL+rJerCknOXOInLM2FkRIemanBVjCKATgbrHoeMcEFjizhBj178almSDF5E SQCQ== X-Gm-Message-State: AOAM530zaywzlHAMw5vEKWPvk/DSWS5QSmmVSX3b1auTPKw4Qd/HRL/6 kP+cHCnAwtaW8zgkWH2uRGYljGYFuJLVhkIvlGEevg== X-Google-Smtp-Source: ABdhPJyiayRWMyVF2zMeallpNwbqXi+fqoCQhl7rzCrLovIWwx1eCFHb8KA73JSK47lYk9VSvJp+ga1RKTBTDLxpFh4= X-Received: by 2002:a1f:9b85:0:b0:32d:4d56:cf64 with SMTP id d127-20020a1f9b85000000b0032d4d56cf64mr5370192vke.31.1650905805824; Mon, 25 Apr 2022 09:56:45 -0700 (PDT) MIME-Version: 1.0 References: <610ccaad03f168440ce765ae5570634f3b77555e.camel@intel.com> <8e31c744a7712bb05dbf7ceb2accf1a35e60306a.camel@intel.com> <78b5f4cfd86efda14c61d515e4db9424e811c5be.camel@intel.com> <200e95cf36c1642512d99431014db8943fed715d.camel@intel.com> In-Reply-To: <200e95cf36c1642512d99431014db8943fed715d.camel@intel.com> From: Wei Xu Date: Mon, 25 Apr 2022 09:56:34 -0700 Message-ID: Subject: Re: [PATCH v2 0/5] mm: demotion: Introduce new node state N_DEMOTION_TARGETS To: "ying.huang@intel.com" Cc: Jagdish Gediya , Yang Shi , Dave Hansen , Dan Williams , Davidlohr Bueso , Linux MM , Linux Kernel Mailing List , Andrew Morton , "Aneesh Kumar K.V" , Baolin Wang , Greg Thelen , MichalHocko , Brice Goglin Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: ntuiszbqpamno17pwzq88qx6hofgmak8 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 0AA8640044 X-Rspam-User: Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pi6mzCKN; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of weixugc@google.com designates 209.85.221.177 as permitted sender) smtp.mailfrom=weixugc@google.com X-HE-Tag: 1650905804-141655 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sat, Apr 23, 2022 at 8:02 PM ying.huang@intel.com wrote: > > Hi, All, > > On Fri, 2022-04-22 at 16:30 +0530, Jagdish Gediya wrote: > > [snip] > > > I think it is necessary to either have per node demotion targets > > configuration or the user space interface supported by this patch > > series. As we don't have clear consensus on how the user interface > > should look like, we can defer the per node demotion target set > > interface to future until the real need arises. > > > > Current patch series sets N_DEMOTION_TARGET from dax device kmem > > driver, it may be possible that some memory node desired as demotion > > target is not detected in the system from dax-device kmem probe path. > > > > It is also possible that some of the dax-devices are not preferred as > > demotion target e.g. HBM, for such devices, node shouldn't be set to > > N_DEMOTION_TARGETS. In future, Support should be added to distinguish > > such dax-devices and not mark them as N_DEMOTION_TARGETS from the > > kernel, but for now this user space interface will be useful to avoid > > such devices as demotion targets. > > > > We can add read only interface to view per node demotion targets > > from /sys/devices/system/node/nodeX/demotion_targets, remove > > duplicated /sys/kernel/mm/numa/demotion_target interface and instead > > make /sys/devices/system/node/demotion_targets writable. > > > > Huang, Wei, Yang, > > What do you suggest? > > We cannot remove a kernel ABI in practice. So we need to make it right > at the first time. Let's try to collect some information for the kernel > ABI definitation. > > The below is just a starting point, please add your requirements. > > 1. Jagdish has some machines with DRAM only NUMA nodes, but they don't > want to use that as the demotion targets. But I don't think this is a > issue in practice for now, because demote-in-reclaim is disabled by > default. > > 2. For machines with PMEM installed in only 1 of 2 sockets, for example, > > Node 0 & 2 are cpu + dram nodes and node 1 are slow > memory node near node 0, > > available: 3 nodes (0-2) > node 0 cpus: 0 1 > node 0 size: n MB > node 0 free: n MB > node 1 cpus: > node 1 size: n MB > node 1 free: n MB > node 2 cpus: 2 3 > node 2 size: n MB > node 2 free: n MB > node distances: > node 0 1 2 > 0: 10 40 20 > 1: 40 10 80 > 2: 20 80 10 > > We have 2 choices, > > a) > node demotion targets > 0 1 > 2 1 > > b) > node demotion targets > 0 1 > 2 X > > a) is good to take advantage of PMEM. b) is good to reduce cross-socket > traffic. Both are OK as defualt configuration. But some users may > prefer the other one. So we need a user space ABI to override the > default configuration. I think 2(a) should be the system-wide configuration and 2(b) can be achieved with NUMA mempolicy (which needs to be added to demotion). In general, we can view the demotion order in a way similar to allocation fallback order (after all, if we don't demote or demotion lags behind, the allocations will go to these demotion target nodes according to the allocation fallback order anyway). If we initialize the demotion order in that way (i.e. every node can demote to any node in the next tier, and the priority of the target nodes is sorted for each source node), we don't need per-node demotion order override from the userspace. What we need is to specify what nodes should be in each tier and support NUMA mempolicy in demotion. Cross-socket demotion should not be too big a problem in practice because we can optimize the code to do the demotion from the local CPU node (i.e. local writes to the target node and remote read from the source node). The bigger issue is cross-socket memory access onto the demoted pages from the applications, which is why NUMA mempolicy is important here. > 3. For machines with HBM (High Bandwidth Memory), as in > > https://lore.kernel.org/lkml/39cbe02a-d309-443d-54c9-678a0799342d@gmail.com/ > > > [1] local DDR = 10, remote DDR = 20, local HBM = 31, remote HBM = 41 > > Although HBM has better performance than DDR, in ACPI SLIT, their > distance to CPU is longer. We need to provide a way to fix this. The > user space ABI is one way. The desired result will be to use local DDR > as demotion targets of local HBM. > > Best Regards, > Huang, Ying >