From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D50A1C433EF for ; Wed, 8 Jun 2022 15:55:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 13E338D0005; Wed, 8 Jun 2022 11:55:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0EDFC8D0001; Wed, 8 Jun 2022 11:55:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF79B8D0005; Wed, 8 Jun 2022 11:55:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E07238D0001 for ; Wed, 8 Jun 2022 11:55:16 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AF1542146D for ; Wed, 8 Jun 2022 15:55:16 +0000 (UTC) X-FDA: 79555517832.02.EB73C7F Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf27.hostedemail.com (Postfix) with ESMTP id 20C094006B for ; Wed, 8 Jun 2022 15:55:15 +0000 (UTC) Received: by mail-qk1-f173.google.com with SMTP id k6so12746066qkf.4 for ; Wed, 08 Jun 2022 08:55:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=qPt4iZhleoIm2SUINneNwOWCL7eWnDd1KTUorYUI5+8=; b=0vi4pcXfTgM09EU78281Lr4S0ERJFDyhmbMmqENiB7mf4kZug4ZZJ4yL8xLN9+e7xJ PHCM5u6rVIsGfIdv9pDERZF5BnqTufGNvi4yF4O5gt+43loOqdYpLML7haRlBK0ehdlL Qaod6qGB6widDIE5cUAzyF2mBFJMus4CnwCYRU+j3f89q2wy2UbNdUEuC716TU7I8XRc 9077m0OAS8EWKue/D1bXMOS2Pg2Ao2WdT37FrBlX6uc9hnLBL/TOpVt7jS6ZF5IXElVP zw9wn6s6SCel9q2qGmqsclaYdj5lcFIKAkGPetvQQvu5/+Fp1BZ+OSGA6csqW8pLB5JO C65w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=qPt4iZhleoIm2SUINneNwOWCL7eWnDd1KTUorYUI5+8=; b=cCIcLaL6WF5bZwHFYpWJyxv0e3ktju1lcX07/Zg16XzwYZE3FYvtmOjuEARjn9aUJV wpSB4GV0HLizO97AGjYNLCp3Sx/M43GugM0sBqkfaEg1109EJh9CmtnAXrLeZ1ibFLI1 naq2j1C6fo340JNTJPRCSJfIV150aPL+gWCNgbZODUsE+bp7A1Lr63fLYQGMEjOisCQL Mjo360D3WUf1pwN/o8wTx9EjzRESE/yo7wox5u2+5ySHIZ3ifsjV3IYh8gSkCn80OT9q hn0U1mNzzg5/iqjQpwqJkJ1Ae6hX0QvAuaBDGK50AY9/MmJDCmGfOheMvo/uwHDcJG/E +ZBA== X-Gm-Message-State: AOAM532Q0NTT94312X/XPy+MK0Nt637btF83BOW/UqJbgw0MC2JZWAPk XoTNJDtGosSOrS+W3kLUSLDqXg== X-Google-Smtp-Source: ABdhPJyTRv0J9Td1OgSBrw4WrYRRwBBgiPUsV6qhumqhrw6lci8vXx3V2YZ4cXBsiniaGpDhKaLbcg== X-Received: by 2002:a05:620a:2046:b0:6a6:b8d1:7ddf with SMTP id d6-20020a05620a204600b006a6b8d17ddfmr11702965qka.380.1654703715300; Wed, 08 Jun 2022 08:55:15 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::1:4759]) by smtp.gmail.com with ESMTPSA id s3-20020a05620a0bc300b006a66f3d3708sm18349784qki.129.2022.06.08.08.55.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jun 2022 08:55:14 -0700 (PDT) Date: Wed, 8 Jun 2022 11:55:14 -0400 From: Johannes Weiner To: "Aneesh Kumar K.V" Cc: linux-mm@kvack.org, akpm@linux-foundation.org, Wei Xu , Huang Ying , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes Subject: Re: [PATCH v5 1/9] mm/demotion: Add support for explicit memory tiers Message-ID: References: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> <20220603134237.131362-2-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 20C094006B X-Stat-Signature: aejnhacscgzop6f9ao9tninjc8r48dum X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=0vi4pcXf; spf=pass (imf27.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.173 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-HE-Tag: 1654703715-548769 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello, On Wed, Jun 08, 2022 at 10:11:31AM -0400, Johannes Weiner wrote: > On Fri, Jun 03, 2022 at 07:12:29PM +0530, Aneesh Kumar K.V wrote: > > @@ -0,0 +1,20 @@ > > +/* SPDX-License-Identifier: GPL-2.0 */ > > +#ifndef _LINUX_MEMORY_TIERS_H > > +#define _LINUX_MEMORY_TIERS_H > > + > > +#ifdef CONFIG_TIERED_MEMORY > > + > > +#define MEMORY_TIER_HBM_GPU 0 > > +#define MEMORY_TIER_DRAM 1 > > +#define MEMORY_TIER_PMEM 2 > > + > > +#define MEMORY_RANK_HBM_GPU 300 > > +#define MEMORY_RANK_DRAM 200 > > +#define MEMORY_RANK_PMEM 100 > > + > > +#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM > > +#define MAX_MEMORY_TIERS 3 > > I understand the names are somewhat arbitrary, and the tier ID space > can be expanded down the line by bumping MAX_MEMORY_TIERS. > > But starting out with a packed ID space can get quite awkward for > users when new tiers - especially intermediate tiers - show up in > existing configurations. I mentioned in the other email that DRAM != > DRAM, so new tiers seem inevitable already. > > It could make sense to start with a bigger address space and spread > out the list of kernel default tiers a bit within it: > > MEMORY_TIER_GPU 0 > MEMORY_TIER_DRAM 10 > MEMORY_TIER_PMEM 20 Forgive me if I'm asking a question that has been answered. I went back to earlier threads and couldn't work it out - maybe there were some off-list discussions? Anyway... Why is there a distinction between tier ID and rank? I undestand that rank was added because tier IDs were too few. But if rank determines ordering, what is the use of a separate tier ID? IOW, why not make the tier ID space wider and have the kernel pick a few spread out defaults based on known hardware, with plenty of headroom to be future proof. $ ls tiers 100 # DEFAULT_TIER $ cat tiers/100/nodelist 0-1 # conventional numa nodes $ grep . tiers/*/nodelist tiers/100/nodelist:0-1 # conventional numa tiers/200/nodelist:2 # pmem $ grep . nodes/*/tier nodes/0/tier:100 nodes/1/tier:100 nodes/2/tier:200 $ grep . tiers/*/nodelist tiers/100/nodelist:0-1,3 tiers/200/nodelist:2 $ echo 300 >nodes/3/tier $ grep . tiers/*/nodelist tiers/100/nodelist:0-1 tiers/200/nodelist:2 tiers/300/nodelist:3 $ echo 200 >nodes/3/tier $ grep . tiers/*/nodelist tiers/100/nodelist:0-1 tiers/200/nodelist:2-3 etc.