From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 777BBC43334 for ; Wed, 8 Jun 2022 18:16:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABFE66B0071; Wed, 8 Jun 2022 14:16:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A6EDD6B0072; Wed, 8 Jun 2022 14:16:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90F926B0073; Wed, 8 Jun 2022 14:16:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 81CCD6B0071 for ; Wed, 8 Jun 2022 14:16:19 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3F1C03490A for ; Wed, 8 Jun 2022 18:16:19 +0000 (UTC) X-FDA: 79555873278.03.CB07BAC Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf25.hostedemail.com (Postfix) with ESMTP id AC985A0071 for ; Wed, 8 Jun 2022 18:16:18 +0000 (UTC) Received: by mail-qk1-f179.google.com with SMTP id d23so1249707qke.0 for ; Wed, 08 Jun 2022 11:16:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=c9z+xEgn0cvxJ8Erz7demN5NMhZ6h+sFb/q6v76GQl0=; b=fddRxPUuqE8gWzetwxLtNjl49MykwjTpFDUinFEgPSjQFkzdJAt7NY/VtAYr4iwnGl z5kPeprOPAav4Fz4E8rMW+LrnKXaMW9aCKGaFPWG5cDv00yAN6NrZxnlxv7RkNK2GKQA lP2cLVXK58EXz00aXSKCPZ4VOwIyRID8CMZEvPdqiul3j5ORnsQtmkOicZGWNmwufD5Y X05p1hvYzmrGhYagesRNUe6KByYQIwPJQKgfF83DvH8aLGaQvGKKiHEjTropvDB19JZo Kq1/Uec1jrT4BJwMFjMd5Z9QWlGJtqg45x86NalFxbFOTxki4VQcC7HkUxn+XTMnMEuj I/nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=c9z+xEgn0cvxJ8Erz7demN5NMhZ6h+sFb/q6v76GQl0=; b=OH72I77IBnwIv1H9iAuHqyQscHNgqlHOhaFxdcJ9cFIABq/U2kBlJUbETMuBJlQqHl D5494jYcVE5c0WoLWJ0dbg/Uq5vzIeZSPOWuNNoqfrHUVCRw+XohXByS5RA7ubxNr5Za FzFvjuIHJHfJx43+J53PaCz25VC+JyP3LijIeFKghA3AZk/TfzH+ricW3s3+sKGH45vZ uXfIUiUegGQaoaJlzE46tXcUdgTDKEK1oYcT1S50S+nfMvjzgwfSTcFA0W9MgSqL8wTo PuklrGXr2CBBDg7G1TpdtvwxNrliHgL+uDVGWisMtsVIxuReN6mlecVRaxLJ1sLmxY+N ME+w== X-Gm-Message-State: AOAM533kW1OmoxOQuhEpPXI0i1PcOS7D7F8kCoW6bpR46aMRDpanQjnL U9UGT2qhVhzR0A4BRY3HpbTZ2Q== X-Google-Smtp-Source: ABdhPJx67/PgS/fTP5mTmGbXXZzMN8viiztzFKsGtTUDlLt75M8GFJ1/JW1BQmXic0Y5oJ9srgY9Xw== X-Received: by 2002:a37:4549:0:b0:69f:556c:4e38 with SMTP id s70-20020a374549000000b0069f556c4e38mr24302378qka.202.1654712177788; Wed, 08 Jun 2022 11:16:17 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::1:4759]) by smtp.gmail.com with ESMTPSA id bm13-20020a05620a198d00b006a6d83fc9efsm4680398qkb.21.2022.06.08.11.16.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jun 2022 11:16:17 -0700 (PDT) Date: Wed, 8 Jun 2022 14:16:17 -0400 From: Johannes Weiner To: Aneesh Kumar K V Cc: linux-mm@kvack.org, akpm@linux-foundation.org, Wei Xu , Huang Ying , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes Subject: Re: [PATCH v5 1/9] mm/demotion: Add support for explicit memory tiers Message-ID: References: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> <20220603134237.131362-2-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: AC985A0071 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=fddRxPUu; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf25.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.179 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org X-Stat-Signature: jhzyxs8o8inp9rey3w8iffa6erfi3zie X-Rspam-User: X-HE-Tag: 1654712178-814703 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jun 08, 2022 at 09:43:52PM +0530, Aneesh Kumar K V wrote: > On 6/8/22 9:25 PM, Johannes Weiner wrote: > > Hello, > > > > On Wed, Jun 08, 2022 at 10:11:31AM -0400, Johannes Weiner wrote: > > > On Fri, Jun 03, 2022 at 07:12:29PM +0530, Aneesh Kumar K.V wrote: > > > > @@ -0,0 +1,20 @@ > > > > +/* SPDX-License-Identifier: GPL-2.0 */ > > > > +#ifndef _LINUX_MEMORY_TIERS_H > > > > +#define _LINUX_MEMORY_TIERS_H > > > > + > > > > +#ifdef CONFIG_TIERED_MEMORY > > > > + > > > > +#define MEMORY_TIER_HBM_GPU 0 > > > > +#define MEMORY_TIER_DRAM 1 > > > > +#define MEMORY_TIER_PMEM 2 > > > > + > > > > +#define MEMORY_RANK_HBM_GPU 300 > > > > +#define MEMORY_RANK_DRAM 200 > > > > +#define MEMORY_RANK_PMEM 100 > > > > + > > > > +#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM > > > > +#define MAX_MEMORY_TIERS 3 > > > > > > I understand the names are somewhat arbitrary, and the tier ID space > > > can be expanded down the line by bumping MAX_MEMORY_TIERS. > > > > > > But starting out with a packed ID space can get quite awkward for > > > users when new tiers - especially intermediate tiers - show up in > > > existing configurations. I mentioned in the other email that DRAM != > > > DRAM, so new tiers seem inevitable already. > > > > > > It could make sense to start with a bigger address space and spread > > > out the list of kernel default tiers a bit within it: > > > > > > MEMORY_TIER_GPU 0 > > > MEMORY_TIER_DRAM 10 > > > MEMORY_TIER_PMEM 20 > > > > Forgive me if I'm asking a question that has been answered. I went > > back to earlier threads and couldn't work it out - maybe there were > > some off-list discussions? Anyway... > > > > Why is there a distinction between tier ID and rank? I undestand that > > rank was added because tier IDs were too few. But if rank determines > > ordering, what is the use of a separate tier ID? IOW, why not make the > > tier ID space wider and have the kernel pick a few spread out defaults > > based on known hardware, with plenty of headroom to be future proof. > > > > $ ls tiers > > 100 # DEFAULT_TIER > > $ cat tiers/100/nodelist > > 0-1 # conventional numa nodes > > > > > > > > $ grep . tiers/*/nodelist > > tiers/100/nodelist:0-1 # conventional numa > > tiers/200/nodelist:2 # pmem > > > > $ grep . nodes/*/tier > > nodes/0/tier:100 > > nodes/1/tier:100 > > nodes/2/tier:200 > > > > > > > > $ grep . tiers/*/nodelist > > tiers/100/nodelist:0-1,3 > > tiers/200/nodelist:2 > > > > $ echo 300 >nodes/3/tier > > $ grep . tiers/*/nodelist > > tiers/100/nodelist:0-1 > > tiers/200/nodelist:2 > > tiers/300/nodelist:3 > > > > $ echo 200 >nodes/3/tier > > $ grep . tiers/*/nodelist > > tiers/100/nodelist:0-1 > > tiers/200/nodelist:2-3 > > > > etc. > > tier ID is also used as device id memtier.dev.id. It was discussed that we > would need the ability to change the rank value of a memory tier. If we make > rank value same as tier ID or tier device id, we will not be able to support > that. Is the idea that you could change the rank of a collection of nodes in one go? Rather than moving the nodes one by one into a new tier? [ Sorry, I wasn't able to find this discussion. AFAICS the first patches in RFC4 already had the struct device { .id = tier } logic. Could you point me to it? In general it would be really helpful to maintain summarized rationales for such decisions in the coverletter to make sure things don't get lost over many, many threads, conferences, and video calls. ]