From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85E46C433EF for ; Fri, 10 Jun 2022 06:15:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C90A08D0079; Fri, 10 Jun 2022 02:15:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BF0708D0064; Fri, 10 Jun 2022 02:15:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A427A8D0079; Fri, 10 Jun 2022 02:15:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8D7A38D0064 for ; Fri, 10 Jun 2022 02:15:46 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 541823597C for ; Fri, 10 Jun 2022 06:15:46 +0000 (UTC) X-FDA: 79561315092.16.170E726 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by imf29.hostedemail.com (Postfix) with ESMTP id AEC0A120049 for ; Fri, 10 Jun 2022 06:15:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1654841743; x=1686377743; h=message-id:subject:from:to:cc:date:in-reply-to: references:mime-version:content-transfer-encoding; bh=Mubq2Tkji6PEsori3NpFWcqeLApjc4Mu831DOd1/+iU=; b=NV0L73i4s7E76hTxT+FuTSl9WuBYRifzilQwsPYqKw7u40ACkMiISvN8 4M0EwqLmXOeCQuWIiVcMo3wz4Fz5UtvP/gsPdqsYE5HgXVbBpQ6MNHFer SGFN+j05sgzsDXWi2DfipUKk8+ScFhzVAaUU1S5/AiKnWWvi655RUb/dA CDWWiEUSGJ832cOrxFUzDuIT5s7GjYL7Ko5mfV4G76jHcWGJ1Kz2EwVpk 3/oiGCL00nkjQ/RO4DGymgKAdJ0iamGywSwqp0oceVdHbED1s+JqxPqzj pp7zqUYt/VSGp5ZkPAgt67KL7wUMlPDvbM7HuFTEEPep/AD0mdpPa3VH8 Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10373"; a="278332954" X-IronPort-AV: E=Sophos;i="5.91,288,1647327600"; d="scan'208";a="278332954" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2022 23:15:40 -0700 X-IronPort-AV: E=Sophos;i="5.91,288,1647327600"; d="scan'208";a="637958331" Received: from qingfen1-mobl1.ccr.corp.intel.com ([10.254.215.73]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2022 23:15:36 -0700 Message-ID: <6e5ecbe5544ca8a434499e95f4d2b9f53a43d91e.camel@intel.com> Subject: Re: [PATCH v5 1/9] mm/demotion: Add support for explicit memory tiers From: Ying Huang To: Johannes Weiner , Jonathan Cameron Cc: Aneesh Kumar K V , linux-mm@kvack.org, akpm@linux-foundation.org, Wei Xu , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes Date: Fri, 10 Jun 2022 14:15:34 +0800 In-Reply-To: References: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> <20220603134237.131362-2-aneesh.kumar@linux.ibm.com> <02ee2c97-3bca-8eb6-97d8-1f8743619453@linux.ibm.com> <20220609152243.00000332@Huawei.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1654841744; a=rsa-sha256; cv=none; b=fZsGvGZ5q/vD0FkZlgZnzkiJxpjOFO6lupggwuIsvWduU0Q7agW4CQzvcav9eUVORM/MYB qqF55Rwb0oJldt99oaAruIf3U39jVIJVypS9pEjOxsh1f6n3reiNfmRzHbGU1hEvh6baFM PUg+fipShqstEE0DJX4dAHcZ8fl9RvE= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NV0L73i4; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf29.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 134.134.136.24) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1654841744; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qsphXuAu9nRn4IVV9gDNp3DMtkZzYcYAG26H1Soka1Y=; b=ZIGj1jkXQYvlAIaOpgloJZdy1bOXJutmKV62EnJOeSLs9FvpHFidw7zC81S2Oz5pC+jz0w PbjcitJlD8EZ/uTAWDV1NOIf1EZQ7rTJwe4kJunNQo4AniMbNuYdIF+ojwp94IBKBZSXj3 tj2cNMWpnZwiEPWYMM4J/4tLkXpzSE8= X-Stat-Signature: xqspxyagukae7b4kfgc8sr17nh4inkbi X-Rspamd-Queue-Id: AEC0A120049 X-Rspam-User: X-Rspamd-Server: rspam10 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NV0L73i4; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf29.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 134.134.136.24) smtp.mailfrom=ying.huang@intel.com X-HE-Tag: 1654841743-57712 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, 2022-06-09 at 16:41 -0400, Johannes Weiner wrote: > On Thu, Jun 09, 2022 at 03:22:43PM +0100, Jonathan Cameron wrote: > > I think discussion hinged on it making sense to be able to change > > rank of a tier rather than create a new tier and move things one by one. > > Example was wanting to change the rank of a tier that was created > > either by core code or a subsystem. > > > > E.g. If GPU driver creates a tier, assumption is all similar GPUs will > > default to the same tier (if hot plugged later for example) as the > > driver subsystem will keep a reference to the created tier. > > Hence if user wants to change the order of that relative to > > other tiers, the option of creating a new tier and moving the > > devices would then require us to have infrastructure to tell the GPU > > driver to now use the new tier for additional devices. > > That's an interesting point, thanks for explaining. I have proposed to use sparse memory tier device ID and remove rank. The response from Wei Xu is as follows, " Using the rank value directly as the device ID has some disadvantages: - It is kind of unconventional to number devices in this way. - We cannot assign DRAM nodes with CPUs with a specific memtier device ID (even though this is not mandated by the "rank" proposal, I expect the device will likely always be memtier1 in practice). - It is possible that we may eventually allow the rank value to be modified as a way to adjust the tier ordering. We cannot do that easily for device IDs. " in https://lore.kernel.org/lkml/CAAPL-u9t=9hYfcXyCZwYFmVTUQGrWVq3cndpN+sqPSm5cwE4Yg@mail.gmail.com/ I think that your proposal below has resolved the latter "disadvantage". So if the former one isn't so important, we can go to remove "rank". That will make memory tier much easier to be understand and use. Best Regards, Huang, Ying > But that could still happen when two drivers report the same tier and > one of them is wrong, right? You'd still need to separate out by hand > to adjust rank, as well as handle hotplug events. Driver colllisions > are probable with coarse categories like gpu, dram, pmem. > > Would it make more sense to have the platform/devicetree/driver > provide more fine-grained distance values similar to NUMA distances, > and have a driver-scope tunable to override/correct? And then have the > distance value function as the unique tier ID and rank in one. > > That would allow device class reassignments, too, and it would work > with driver collisions where simple "tier stickiness" would > not. (Although collisions would be less likely to begin with given a > broader range of possible distance values.) > > Going further, it could be useful to separate the business of hardware > properties (and configuring quirks) from the business of configuring > MM policies that should be applied to the resulting tier hierarchy. > They're somewhat orthogonal tuning tasks, and one of them might become > obsolete before the other (if the quality of distance values provided > by drivers improves before the quality of MM heuristics ;). Separating > them might help clarify the interface for both designers and users. > > E.g. a memdev class scope with a driver-wide distance value, and a > memdev scope for per-device values that default to "inherit driver > value". The memtier subtree would then have an r/o structure, but > allow tuning per-tier interleaving ratio[1], demotion rules etc. > > [1] https://lore.kernel.org/linux-mm/20220607171949.85796-1-hannes@cmpxchg.org/#t