From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56AC3C43334 for ; Thu, 14 Jul 2022 04:57:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB1D3940185; Thu, 14 Jul 2022 00:57:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D5E45940134; Thu, 14 Jul 2022 00:57:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4D28940185; Thu, 14 Jul 2022 00:57:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B58A3940134 for ; Thu, 14 Jul 2022 00:57:19 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 88B7C33898 for ; Thu, 14 Jul 2022 04:57:19 +0000 (UTC) X-FDA: 79684496598.15.EB8C2CD Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf23.hostedemail.com (Postfix) with ESMTP id A5761140080 for ; Thu, 14 Jul 2022 04:57:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657774638; x=1689310638; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=lwQnOGjgivQ5p2oJYc4Kbap67iNh8bkmNF0iKNVM8Oo=; b=Ozl/2PgngoRf2CI7KpfJcSxmLHdz/cwPV56nYLV2gE0Ys6/31vzmeCz8 aHMpol6KgEbMos3NcbcuvkL/lxypN5Rc23ZsBdS/WkEgbNJNp3Orbcd+5 fCwMevPu3IFo0oY2Fh5NGfDRpq4ZVjlqHvAsl8ZUuV16FgCx01cN7SUY+ lQOyw0aZ9w/S8IwUKfxpaU/stOfWAfY+TdDLW+8JCzZkpBpQmfpXVdgzf O7iG4vhhANrozPpU2WgpoqI0KzeQcXRsWSfWj86vJiW00zdQv3Fn9L40L /0VrYcHnNGoKSoO+ZZA+CtEbnoCm22ikr6nh+t4OcH3nKU9CfGPOedA7P w==; X-IronPort-AV: E=McAfee;i="6400,9594,10407"; a="311064104" X-IronPort-AV: E=Sophos;i="5.92,269,1650956400"; d="scan'208";a="311064104" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2022 21:57:17 -0700 X-IronPort-AV: E=Sophos;i="5.92,269,1650956400"; d="scan'208";a="546128526" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.239.13.94]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2022 21:57:13 -0700 From: "Huang, Ying" To: "Aneesh Kumar K.V" Cc: Johannes Weiner , linux-mm@kvack.org, akpm@linux-foundation.org, Wei Xu , Yang Shi , Davidlohr Bueso , Tim C Chen , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , jvgediya.oss@gmail.com Subject: Re: [PATCH v8 00/12] mm/demotion: Memory tiers and demotion References: <20220704070612.299585-1-aneesh.kumar@linux.ibm.com> <87r130b2rh.fsf@yhuang6-desk2.ccr.corp.intel.com> <60e97fa2-0b89-cf42-5307-5a57c956f741@linux.ibm.com> <87r12r5dwu.fsf@yhuang6-desk2.ccr.corp.intel.com> <0a55e48a-b4b7-4477-a72f-73644b5fc4cb@linux.ibm.com> <87mtde6cla.fsf@yhuang6-desk2.ccr.corp.intel.com> <87ilo267jl.fsf@yhuang6-desk2.ccr.corp.intel.com> <87edyp67m1.fsf@yhuang6-desk2.ccr.corp.intel.com> <878roxuz9l.fsf@linux.ibm.com> Date: Thu, 14 Jul 2022 12:56:55 +0800 In-Reply-To: <878roxuz9l.fsf@linux.ibm.com> (Aneesh Kumar K. V.'s message of "Wed, 13 Jul 2022 15:10:06 +0530") Message-ID: <87o7xs47hk.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="Ozl/2Pgn"; spf=none (imf23.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657774639; a=rsa-sha256; cv=none; b=MuDFjJePZW+F6a5WYmOX6Z/JjrqUsLSC9nAxkIXWsMkCh2UrPn7O63WqjNNj0l47+lw74L uEWaT2fQthc7rTPpBv3zY8Xf4InxKRaX1w8dVaIwD4S69mk/uridumWWn5TNNrfv5eHzEg XVnMRCDNRuODM2w0KUR5p6IchcXteJw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657774639; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WM7c+7kDaUxU6x17nu8qbekhuGZhGqJUIWiHcv8BrBg=; b=HDKtI4sbzxsesmR9RqL2Q98y0s6MkGCE/xSB0DdQtcNSFUwnQBYxwe+aD7nSCF6j70ToSX oXgwplnUzJSW73LytFJMr9/cMzK8RorR97O40qJy5e6IuFRICj06S0Nah7UkGyJQY3Xz38 xcs4V1WWQXczrhNQeLs/b74Bn9icUcw= X-Rspamd-Queue-Id: A5761140080 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="Ozl/2Pgn"; spf=none (imf23.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: uo6mxpmx3fxzj8nitshzzzecuzj9u6dy X-HE-Tag: 1657774638-156993 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: "Aneesh Kumar K.V" writes: > "Huang, Ying" writes: [snip] >> >> I believe that sparse memory tier IDs can make memory tier more stable >> in some cases. But this is different from the system suggested by >> Johannes. Per my understanding, with Johannes' system, we will >> >> - one driver may online different memory types (such as kmem_dax may >> online HBM, PMEM, etc.) >> >> - one memory type manages several memory nodes (NUMA nodes) >> >> - one "abstract distance" for each memory type >> >> - the "abstract distance" can be offset by user space override knob >> >> - memory tiers generated dynamic from different memory types according >> "abstract distance" and overridden "offset" >> >> - the granularity to group several memory types into one memory tier can >> be overridden via user space knob >> >> In this way, the memory tiers may be changed totally after user space >> overridden. It may be hard to link memory tiers before/after the >> overridden. So we may need to reset all per-memory-tier configuration, >> such as cgroup paritation limit or interleave weight, etc. > > Making sure we all agree on the details. > > In the proposal https://lore.kernel.org/linux-mm/7b72ccf4-f4ae-cb4e-f411-74d055482026@linux.ibm.com > instead of calling it "abstract distance" I was referring it as device > attributes. > > Johannes also suggested these device attributes/"abstract distance" > to be used to derive the memory tier to which the memory type/memory > device will be assigned. > > So dax kmem would manage different types of memory and based on the device > attributes, we would assign them to different memory tiers (memory tiers > in the range [0-200)). > > Now the additional detail here is that we might add knobs that will be > used by dax kmem to fine-tune memory types to memory tiers assignment. > On updating these knob values, the kernel should rebuild the entire > memory tier hierarchy. (earlier I was considering only newly added > memory devices will get impacted by such a change. But I agree it > makes sense to rebuild the entire hierarchy again) But that rebuilding > will be restricted to dax kmem driver. > Thanks for explanation and pointer. Per my understanding, memory types and memory devices including abstract distances are used to describe the *physical* memory devices, not *policy*. We may add more physical attributes to these memory devices, such as, latency, throughput, etc. I think we can reach consensus on this point? In contrast, memory tiers are more about policy, such as demotion/promotion, interleaving and possible partition among cgroups. How to derive memory tiers from memory types (or devices)? We have multiple choices. Per my understanding, Johannes suggested to use some policy parameters such as distance granularity (e.g., if granularity is 100, then memory devices with abstract distance 0-100, 100-200, 200-300, ... will be put to memory tier 0, 1, 2, ...) to build the memory tiers. Distance granularity may be not flexible enough, we may need something like a set of cutoffs or range, e.g., 50, 100, 200, 500, or 0-50, 50-100, 100-200, 200-500, >500. These policy parameters should be overridable from user space. And per my understanding, you suggested to place memory devices to memory tiers directly via a knob of memory types (or memory devices). e.g., memory_type/memtier can be written to place the memory devices of the memory_type to the specified memtier. Or via memorty_type/distance_offset to do that. Best Regards, Huang, Ying [snip]