From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A05F0C77B61 for ; Thu, 27 Apr 2023 04:30:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C915B6B0071; Thu, 27 Apr 2023 00:30:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C41CB6B0072; Thu, 27 Apr 2023 00:30:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B09896B0074; Thu, 27 Apr 2023 00:30:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A10186B0071 for ; Thu, 27 Apr 2023 00:30:58 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6CCB1C0156 for ; Thu, 27 Apr 2023 04:30:58 +0000 (UTC) X-FDA: 80725895796.20.302592E Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf07.hostedemail.com (Postfix) with ESMTP id B141740013 for ; Thu, 27 Apr 2023 04:30:56 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lxPNQ0WF; spf=pass (imf07.hostedemail.com: domain of rientjes@google.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=rientjes@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682569856; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=1p1cAXIn+g6aitk4SAVhyy4oP2DpmfDnEyuOCIt3phA=; b=5TLdrHsNQ1dFPQi/iUpLEcxxJCmAiyeH8Mq3+Kw2drBYaw0zYc3xFGFtwz0ZVIGs37Nklp ecpSAf+tLYS3TQ87HUXJboOvPTDl+oa5f6IkdGk2bf/cIQqsJCj/4LSIEUHFlBNNHibd13 LEQRLo9AZsw0NZ6EJptk36bjBDCA7wA= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lxPNQ0WF; spf=pass (imf07.hostedemail.com: domain of rientjes@google.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=rientjes@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682569856; a=rsa-sha256; cv=none; b=ibQm/cG8H/Ye1RR9cAvZFc/+FZ+1IZHcd3OHuJNgpulatSdeYVKd05lKVmvQWH6MAAnxS+ v0GGj2gy5kSASvnvnpY6g8yplIaWqDV7+quiZrlvmCSOqzrYLW45rlI9baYvRQ0ynBpCvJ gdl71W3CHF50S5EWIiiT/tAmtkfiwa4= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1a6e5be6224so313915ad.1 for ; Wed, 26 Apr 2023 21:30:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682569855; x=1685161855; h=mime-version:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to; bh=1p1cAXIn+g6aitk4SAVhyy4oP2DpmfDnEyuOCIt3phA=; b=lxPNQ0WFYFnd1KcpdpjMXvecrKa7whgB4uUKfBheSuA23R5jIBZTqKtKosfi1DW+TC ehW0XkoOw3S86s/g1vOdP17o3ZdF1VOLWdJ6QeqKN2pHQ5p/BQCjQrqQsqXOb4Mzw5oZ NmwxPtJfrPG72adNQdIaLT+VnWJPL1bj0kQ+T0iD88lW3AXcQBHDIzO49hRJLm30wNSl gq0Pen/WkD+FNu8ao90U/2qG2Swi0uoj2JPAWyXrh8EeZtxQ9x8T9qiim8D+mf0MUNp0 TP2M/V5J8C7+ylkEUUm2HTO1Pa5n4L+7R1HrA5fSBMSb1W/x90Hld0DRUQXcFaiHgYeo T6/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682569855; x=1685161855; h=mime-version:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=1p1cAXIn+g6aitk4SAVhyy4oP2DpmfDnEyuOCIt3phA=; b=Jl+MN0JKNVjrnrXinFmG8D8VlpLaNu8PtCjaF1PQNOXMThD4BTbhkHyrQu4IpfKsPQ oq8YlYbZ6lEDtvg9k+Lq/9zsPhUaeZ3tpDziMm+Bzq7I1MAm0ZgRA3VtWHrnkjy7oA1y BPI9gFQL9ybRG4kLC4TF581ly5Ix5JLzgKRE7+cEleAoxzErsm2tFDCKlBRBPEuUyOPF VN8haBs0LvrYC6Ko2Ur7kzZc8O4M0aOm19JN4O13+3Urwdu4UKUnorOoeGM4Eu7sP/wO wU8sWh5/xBUTGhXsmgNl7BTWpjwrYzqYqQiGrE1CKtQ5EUMCVt64H+t+r7F/ZR2xH3C9 sLrQ== X-Gm-Message-State: AC+VfDwOdOkLx3OLu1jKwhGjMxXW1Id0zr+9HAydZmWpYJVqtopTPQFN hRtgDnnFL26rEjMiX47MXQPteA== X-Google-Smtp-Source: ACHHUZ7F3lWyCxj79WuklQNAUwRM+CTwZXVCVDFZbM0RnqEZanivJAnMVmtyz/vSHsaPYzzH7L0P0A== X-Received: by 2002:a17:903:1d0:b0:1a6:760c:af3d with SMTP id e16-20020a17090301d000b001a6760caf3dmr41199plh.16.1682569855394; Wed, 26 Apr 2023 21:30:55 -0700 (PDT) Received: from [2620:0:1008:11:2f80:f0f3:b7a2:40a] ([2620:0:1008:11:2f80:f0f3:b7a2:40a]) by smtp.gmail.com with ESMTPSA id a15-20020a65640f000000b0051afa49e07asm10513817pgv.50.2023.04.26.21.30.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Apr 2023 21:30:54 -0700 (PDT) Date: Wed, 26 Apr 2023 21:30:54 -0700 (PDT) From: David Rientjes To: Michal Hocko , Dan Williams cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, Wei Xu , Frank van der Linden , Johannes Weiner , Dave Hansen , Huang Ying , "Aneesh Kumar K.V" , Yang Shi , Davidlohr Bueso , Jon Grimm Subject: [LSF/MM/BPF TOPIC] The future of memory tiering Message-ID: <7443f0e6-6be2-3320-60d9-03da0cca2987@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: B141740013 X-Stat-Signature: abqiwrgmrhswqu3y16ua7y8uro6q78hx X-Rspam-User: X-HE-Tag: 1682569856-154085 X-HE-Meta: U2FsdGVkX189uUj5C618JVbG8UlPz0H/4Vw8cXpOJGfwhGFuyRwOumXMWhC5+EsjCe5JRyuRjlOuboaIJV68+9f7LNpzOVNEy3CTQ9dl+rRgpDnxPj5s00i+4jXOlwsNtaBRDbynXeIYJsC/RKkCvUVlQExOWQNw1WWz4GEWGe6Ex8W02+UtFuRx72pOJrOPcR2FfB2Ndn/xp4aS0KydooaOKqKVw3CfhMhT3VRa6Bp22fkBkzb5EzVP9xJ9/BTFNq+c5+IXGFX5qETMlQmFD+mVvFYlOguXhzQewuir/YsbGZlXh3NcZmTMrL8WYIi1Jhw0G3znEKjLeIF8QXp5LPfjeghQTJScCE/D4I8gQFBboZuftRlgPWz67CkNY2ep+yYLyLLOVCju6Uhv8+AI1Q7WzanmJXCSLjmc4V1YP2ZCoLaj1q7hCHVJeyAYB4vbzG1P5fHG7MALTvsJXN9y+TV6lkKoM+jqrvICBCqZCICLN0jhs9e70ajSEkqEjdCevWrAabMDRnKrRlPoKQUkDtY9/4o8p3IOkOt0czq+GyQhyKjaXzb9yAECgvG/8jul7I7IP0NvX7wpij7kUnbglKyAHZWI9EzHP2Ikl1xk507XqyInaT9FwWvYSQ/MSodynkw7i4yaxKUsaMKlCkoT3Pe9tYj6KvEfIxDCxcQ7Ff/8uIJirVPFRcHdAJnlLINjPQ/06VHHNiVvOZhRVw4sy2gcY3nL/6fdaRqXccIfG1QUTwYvLSQd5O1KEqvkQ4vLHkMiW44O+FvqHb9jBfnrith7VRBS+bRGS9pvL/heggSnyMnWFelFxVrCp7teyKnWN7i3tDn7t8U366Nq54qwRFOt10Btw/twGBIaq4VOFToDwWYmYl3EjY2J4vefAHA2RpQHTeO4XUd7zzhHcZcSqnCMOwJA2wMpPF8yym7Juroyy+/n6P1zQyYYpoqxK9bpUuM2fKA5QIpYYaEKj5y N9Rgmx2Z rgMzkVBuGUqQwpLeQqlhEd5uNybRhboJn7AtNeBJmC5yTSQ1Pm6ua5TAGg/pZ+Xb90731MJmWX8Tf4XGqhtzRhxnKMRKxuDWIak/lHabKwpPULS/dCJjE1+LB3Wq5ZxZ+XYZ0qIHlWofv/B5YPs+TcZC+m5EStDlD2AUQA5N8eApmwJGFcUkoh/Pf8u1Lw9esAMLlTUDtt9D/vWEc+NRuebH/DxUuVlJ9x65KWGor2EjaLVPyV7HBtYaqxwRkLiYOnqg0MtdoFtYPztmGG3i8WhlsDW37tSnml5eeV7+uc0YZ4FtTrC3dWoO6kF3rwO/+hr9yQIf/sHLgytn8Vawk5cpVhqDlS0IdWfZrTxCmxKe8J0s1mnFw7jmNdg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi everybody, As requested, sending along a last minute topic suggestion for consideration for LSF/MM/BPF 2023 :) For a sizable set of emerging technologies, memory tiering presents one of the most formidable challenges and exicting opportunities for the MM subsystem today. "Memory tiering" can mean many different things based on the user: from traditional every day NUMA, to swap (to zswap), to NVDIMMs, to HBM, to locally attached CXL memory, to memory borrowing over PCIe, to memory pooling with disaggregation, and beyond. Just as NUMA started out only being useful for the supercomputers, memory tiering will likely evolve over the next five years to take on an expanding set of use cases, and likely with rapidly increasing adoption even beyond hyperscalers. I think a discussion about memory tiering would be highly valuable. A few key questions that I think can drive this discussion: - What are the various form factors that must be supported as short-term goals as well as need to be supported 5+ years into the future? - What incremental changes need to be made on top of NUMA support to fully support the wide range of use cases that will be coming? (Is memory tiering support built entirely upon NUMA?) - What is the minimum viable *default* support that the MM subsystem should provide for tiered configs? What are the set of optimizations that should be left to userspace or BPF to control? - What are the various page promotion technqiues that we must plan for beyond traditional NUMA balancing that will allow us to exploit hardware innovation? (And I'm sure there are more topics of discussion that others would readily add. It would be great to have additional ideas in replies.) A key challenge in all of this is to make memory tiering support in the upstream kernel compatible with the roadmaps of various CPU vendors. A key goal is to ensure the end user benefits from all of this rapid innovation with generalized support that is well abstracted and allows for extensibility.