From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B1DBC43334 for ; Thu, 9 Jun 2022 20:41:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8DEB46B00F3; Thu, 9 Jun 2022 16:41:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 881DE6B00F4; Thu, 9 Jun 2022 16:41:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 748D26B00F5; Thu, 9 Jun 2022 16:41:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 644256B00F3 for ; Thu, 9 Jun 2022 16:41:06 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 3ECAB80B56 for ; Thu, 9 Jun 2022 20:41:06 +0000 (UTC) X-FDA: 79559866932.29.07CCF56 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf18.hostedemail.com (Postfix) with ESMTP id 9D8C61C0015 for ; Thu, 9 Jun 2022 20:41:05 +0000 (UTC) Received: by mail-qt1-f176.google.com with SMTP id c8so18060582qtj.1 for ; Thu, 09 Jun 2022 13:41:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=j1i9jnkPCwj7hjptqenXLooIsYDic1cbsYQIxJWV6GE=; b=plm4N/skR66NjRrl4/C5+SMQzqXmUlJjH0J0+883ZKmDFiqwT3OV4Bqij/K5nRcp+A dz2MWYDobAigEnjXlHJokJEUyQvmd4jMYs+PYbX0a/m0aKggTDxNFx+x0HQ+DHp9e2f3 N9nwqNbnXW3JjaMj/W8BSt+frvPMh+frOxCYhT0+HkZeApQDxi1nlOtFJFZXF+cIPZWq tTgrWbL9cM9qq8SvRBFyzTbIBa9G5/TAJrwRyRTWQuB2Hh6uObVrc3+VokZ/0zYSjB6N ivqUGq/Q/gQjLQl9Io1mQ9M83TTOWPYqBe1nF+8fpiC6oExn/9CvbXI0CAWzfb5MckOM qDZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=j1i9jnkPCwj7hjptqenXLooIsYDic1cbsYQIxJWV6GE=; b=lGCl9E8L1NXY4qVsC/Ju6mg16v2GyiEVJMDdQ/I5nV/bnwPw4ULrSYcQZZFoLFhz7z uN58KxC+hAK7oLB0M6HiLG+g3kz9aqIlnQkT+p+59PQV1SHzD5jd6jtJR38u5ee6Tk1u dw7pOO+OMmu8t699ij4TkKsBQjnJTmE6dab8WNkvRvFsw9OaQLHcRqIR5snzNmLdWDRx 9cnlO4TpY+ecAFS5IXHccqo64hpjuCSOCpwy/AiZuEbZgfu4DYUOD7qCdLis4zHSvv3B BAFtHGq+5P8VLJquQd33MZYwRBsUieRLIKGfGxzZxn8bUJYF1TuA08mKlcO9tKTG7LMH NvGQ== X-Gm-Message-State: AOAM533htBhV5Y+KXiJ5hJnXQI5EZGeqkeT5zDzvpRXfUC5S1mvFAloA kZGMBzx4pskJrM3LBTtJTSvTeQ== X-Google-Smtp-Source: ABdhPJxp1k9zfvxgeMouv0XByvqOJbmu7qh92LB53qmBHvgXvd6f2OhwumIXJZDuQs+MM0CyM9KnDw== X-Received: by 2002:a05:622a:130a:b0:304:f156:ec9 with SMTP id v10-20020a05622a130a00b00304f1560ec9mr16605646qtk.226.1654807264737; Thu, 09 Jun 2022 13:41:04 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::1:4759]) by smtp.gmail.com with ESMTPSA id w17-20020a05620a445100b006a7137330e2sm4554017qkp.132.2022.06.09.13.41.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jun 2022 13:41:04 -0700 (PDT) Date: Thu, 9 Jun 2022 16:41:04 -0400 From: Johannes Weiner To: Jonathan Cameron Cc: Aneesh Kumar K V , linux-mm@kvack.org, akpm@linux-foundation.org, Wei Xu , Huang Ying , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes Subject: Re: [PATCH v5 1/9] mm/demotion: Add support for explicit memory tiers Message-ID: References: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> <20220603134237.131362-2-aneesh.kumar@linux.ibm.com> <02ee2c97-3bca-8eb6-97d8-1f8743619453@linux.ibm.com> <20220609152243.00000332@Huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220609152243.00000332@Huawei.com> ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1654807265; a=rsa-sha256; cv=none; b=7vRMSlOg0XyFDNovIVnuzBxY7iw6tJ5RxPzeqXJgdjCwRCALGjOuU8HyYrxaBvVGD8AkSK 105TCpjPJFmR5+lI070bU6jwCKfNo2BDj8C2/aGwtmtLkoDGdKslBibT7cNMj8aOjvsMLx OMeLsRWxMiji7gyI6kHTq8uPwHWIE4w= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b="plm4N/sk"; spf=pass (imf18.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1654807265; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j1i9jnkPCwj7hjptqenXLooIsYDic1cbsYQIxJWV6GE=; b=g9Rn231UGjQbNuSZVAHWzBepbblW1uItQtHpqZ2ZeBMkJXgtL+h1R/DCm6rYhrOx8EUjsh KYfwhBMrzlQbQDSC18A41kIY+FtPsoGovLawAhwML5H88x4BGRFg6dP+WK6PWUDDo0ohM8 8RqJGGAd5UY4p/pQ8svGaWbOI1FtzLQ= X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9D8C61C0015 X-Stat-Signature: 17jgbm5zyucr65mt4xu7rmzx4ye97dnx X-Rspam-User: Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b="plm4N/sk"; spf=pass (imf18.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-HE-Tag: 1654807265-535273 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jun 09, 2022 at 03:22:43PM +0100, Jonathan Cameron wrote: > I think discussion hinged on it making sense to be able to change > rank of a tier rather than create a new tier and move things one by one. > Example was wanting to change the rank of a tier that was created > either by core code or a subsystem. > > E.g. If GPU driver creates a tier, assumption is all similar GPUs will > default to the same tier (if hot plugged later for example) as the > driver subsystem will keep a reference to the created tier. > Hence if user wants to change the order of that relative to > other tiers, the option of creating a new tier and moving the > devices would then require us to have infrastructure to tell the GPU > driver to now use the new tier for additional devices. That's an interesting point, thanks for explaining. But that could still happen when two drivers report the same tier and one of them is wrong, right? You'd still need to separate out by hand to adjust rank, as well as handle hotplug events. Driver colllisions are probable with coarse categories like gpu, dram, pmem. Would it make more sense to have the platform/devicetree/driver provide more fine-grained distance values similar to NUMA distances, and have a driver-scope tunable to override/correct? And then have the distance value function as the unique tier ID and rank in one. That would allow device class reassignments, too, and it would work with driver collisions where simple "tier stickiness" would not. (Although collisions would be less likely to begin with given a broader range of possible distance values.) Going further, it could be useful to separate the business of hardware properties (and configuring quirks) from the business of configuring MM policies that should be applied to the resulting tier hierarchy. They're somewhat orthogonal tuning tasks, and one of them might become obsolete before the other (if the quality of distance values provided by drivers improves before the quality of MM heuristics ;). Separating them might help clarify the interface for both designers and users. E.g. a memdev class scope with a driver-wide distance value, and a memdev scope for per-device values that default to "inherit driver value". The memtier subtree would then have an r/o structure, but allow tuning per-tier interleaving ratio[1], demotion rules etc. [1] https://lore.kernel.org/linux-mm/20220607171949.85796-1-hannes@cmpxchg.org/#t