From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33903C77B61 for ; Fri, 28 Apr 2023 08:05:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C9EC6B0072; Fri, 28 Apr 2023 04:05:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 679BD6B0074; Fri, 28 Apr 2023 04:05:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 543876B0075; Fri, 28 Apr 2023 04:05:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 436426B0072 for ; Fri, 28 Apr 2023 04:05:53 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 11786120312 for ; Fri, 28 Apr 2023 08:05:53 +0000 (UTC) X-FDA: 80730066186.27.4143829 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf19.hostedemail.com (Postfix) with ESMTP id B58691A0015 for ; Fri, 28 Apr 2023 08:05:49 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=YITd9zy9; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682669150; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bF8EEhkBNjV5ddbSnh4HIB7BrLKzW1/4fyNfVU96mH8=; b=GL+IIEHGWl1/Ykphz4b+j8pcHcsc3oAfkrWApy+3tK6CmFeGQjfsNShUXZk7T1/Lb0Oxxh 1BVajIUmtcuwBk3p8hYwM00sJCg4YribGuU7zH+NqIU7cp2hEeAb9s1sxrA9cCqzNeRh43 PyAKKuwhmbGY02hdDPz7JkBNzNlno7U= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=YITd9zy9; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682669150; a=rsa-sha256; cv=none; b=YYkBBRPc5B4cFXqMMjcP+uUhZwr/WdyfLV/SjsZFvwtDcH6j2rQ2TcCjeCRpQuTLRoGBms vcTZFxBJKMG5vpkIDXi50lD291LDWVKh1qE3aak5lDhCF6roiOjYa5grrq7TjpOSSn4Gga Ov0p5sQtoPdsFOnqw17UiSZ7EmwTe/k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682669149; x=1714205149; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=FyxaUxQFuh/9bk+FGBQu7P/3kapkUh9bw4QE/g25vzM=; b=YITd9zy90Q/SV+QhjkT5kdSINSxiPARM2ZSdLDQvhnIDvJVb3KFBAWyb b2UM+vpR8TNRF+HMzQPYEdBEtwBsn+/7ltU+Spr/K8vAZj6CmTpWppjCD +JD1+nyYsS7cekjwFACbR9onLeB1EaixEMTy+gYRS1wuz5vc+ypwNv95u hg462rTMwSfO7ndok2R65wXU2TrhSNTIBTm2dKLSHxSX954GHEuV1zsA7 Jdk8rkHqJNWt4la/N9HFXvsR8iF9g/OcmlSp+8dWWAbcsXR2IfiCm6rW2 3+oUXHyjsHPnTEjUACfB9trfX7mxSHm6I/qh6AnW0pOdnk3Ehnw2VZv6h w==; X-IronPort-AV: E=McAfee;i="6600,9927,10693"; a="331983996" X-IronPort-AV: E=Sophos;i="5.99,234,1677571200"; d="scan'208";a="331983996" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Apr 2023 01:05:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10693"; a="759553714" X-IronPort-AV: E=Sophos;i="5.99,234,1677571200"; d="scan'208";a="759553714" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Apr 2023 01:05:37 -0700 From: "Huang, Ying" To: David Rientjes Cc: Michal Hocko , Dan Williams , lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, Wei Xu , Frank van der Linden , Johannes Weiner , Dave Hansen , "Aneesh Kumar K.V" , Yang Shi , Davidlohr Bueso , Jon Grimm Subject: Re: [LSF/MM/BPF TOPIC] The future of memory tiering References: <7443f0e6-6be2-3320-60d9-03da0cca2987@google.com> Date: Fri, 28 Apr 2023 16:04:29 +0800 In-Reply-To: <7443f0e6-6be2-3320-60d9-03da0cca2987@google.com> (David Rientjes's message of "Wed, 26 Apr 2023 21:30:54 -0700 (PDT)") Message-ID: <87y1mccu0i.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: B58691A0015 X-Rspam-User: X-Stat-Signature: cwff7mmg65we7jy968133sxnexgr8qym X-HE-Tag: 1682669149-501971 X-HE-Meta: U2FsdGVkX1+I0tA/DvEbjXF8luCVs69oT2H4hN1slUjMI70axI/XRELo6dDsDSoC1J039aa4vnnrGgGTRAMC6kg0so6RGBGMyFxrSGbey6c8KDG+61IE2vkERYpeuj0ExdrXtG9Hmq+EAVFg+yVyhntS+m5PP4MyNAT8gyEjOHpLwsuTYZnO5gTSRcD3Zutt1D4QAi5GuUdVYO2GkP5mu3Y0QoJ9d1CCOyW+Tj1pnxZgXIrJsMv7GlNMR1wv4kv3FKtSVDe/g6zajuDGWHpZjWFabEjhqOBZm0vpCMNDxYYfJPFlOGJYooNOtRreK7h90fbxYFMalKkDLV/JG+zLtAOIFtaXJzSER+phx8spSpZ1RR5FmRIuaGQXGsPFZqcqRBv/FiJm9b7XFyj8BXprPPQv/k/aV9tbYDKZD9yLUjNLkdXOvAyooA8jZUWpn5q2Smh+LlrPd7q0N4cbw/F5QDabXntaj8He+9Mco6FMYJ/jk3qeWT4IvaKDxWbqL7o3Q168Cje08JOfX6sW6omp9nA6R92lQonRv0lI04I1ZL/Y8DaNigp75GDuMRXuxyCZNV+E0F/dd3JefcHcXGbyrPJ0x0FjD1U4JG/L9SR6WnT5NbDuyHrdH8RWtFC+ijUcneRRB1ejXQneWHBPbloVaWc3PfohFsLuYYAXcng8qTmXUNE23sFk+xh5qJi+WGzvt8/AxtU4Zn649rC3ZAwjbPkTOWWxdp2kIjINDbQmSMHEUz6DXwa9aX6n5oTPS96e64H2UlUzvLNwQPVqdpTHe9n82VFcI5D1MrwVPzPgtFeLeQ/G6rXAdnnB9Xstv4NgAiYrAEE66jVdVaYVbvf9nk5BYrOg08I3XlrOLVdofT3qLZWrdcDVgbq0JzunptsTijdZJ7jwnBWNpqIGAVO9HfeDuEXI87bgn13tSBqZdUD8mziuoYjhxIjCAms7Rv/XL4CZ1C8AWrR+PkHA6pV Jt3t/Syt YBuWJq6hdicRh8q2ZkiRWY/ZtVWjVB6tbPlvcS/BhyNKUXW/wpjweZwQQI8Oj+h8M7eJ7LBecz6s33i9o9NH68FpVsVs7q4KotI1MxhP5sqHvuK4cy4qZILA/C6Y7tJffuOXj0ZyHpHx2RaCu0xSoD/4yT+8P/6g2gx9zThfu7jy1EhU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: David Rientjes writes: > Hi everybody, > > As requested, sending along a last minute topic suggestion for > consideration for LSF/MM/BPF 2023 :) > > For a sizable set of emerging technologies, memory tiering presents one of > the most formidable challenges and exicting opportunities for the MM > subsystem today. > > "Memory tiering" can mean many different things based on the user: from > traditional every day NUMA, to swap (to zswap), to NVDIMMs, to HBM, to > locally attached CXL memory, to memory borrowing over PCIe, to memory > pooling with disaggregation, and beyond. Thanks for proposing the topic. I have strong interest in memory tiering topics. > Just as NUMA started out only being useful for the supercomputers, memory > tiering will likely evolve over the next five years to take on an > expanding set of use cases, and likely with rapidly increasing adoption > even beyond hyperscalers. > > I think a discussion about memory tiering would be highly valuable. A few > key questions that I think can drive this discussion: > > - What are the various form factors that must be supported as short-term > goals as well as need to be supported 5+ years into the future? > > - What incremental changes need to be made on top of NUMA support to > fully support the wide range of use cases that will be coming? (Is > memory tiering support built entirely upon NUMA?) Yes. The memory tiers may represent memory compression (such as zswap) too. We may extend it. > - What is the minimum viable *default* support that the MM subsystem > should provide for tiered configs? What are the set of optimizations > that should be left to userspace or BPF to control? > > - What are the various page promotion technqiues that we must plan for > beyond traditional NUMA balancing that will allow us to exploit > hardware innovation? I have interest in hardware innovation too. And I feel it's hard to unite all these methods into one framework. > (And I'm sure there are more topics of discussion that others would > readily add. It would be great to have additional ideas in replies.) > > A key challenge in all of this is to make memory tiering support in the > upstream kernel compatible with the roadmaps of various CPU vendors. A > key goal is to ensure the end user benefits from all of this rapid > innovation with generalized support that is well abstracted and allows for > extensibility. Best Regards, Huang, Ying