From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58DF0C10F1A for ; Tue, 7 May 2024 20:09:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DCD426B008A; Tue, 7 May 2024 16:09:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D7CC36B008C; Tue, 7 May 2024 16:09:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C44196B0092; Tue, 7 May 2024 16:09:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A675D6B008A for ; Tue, 7 May 2024 16:09:30 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4A9C3C0403 for ; Tue, 7 May 2024 20:09:30 +0000 (UTC) X-FDA: 82092689700.11.46CAEBC Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf18.hostedemail.com (Postfix) with ESMTP id 83A611C000B for ; Tue, 7 May 2024 20:09:28 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=205eJWBx; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of rientjes@google.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715112568; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eLq2KEdeSgHFupFQ5aOdwVPPc1VFbSlxs3IcQ2BaxEE=; b=6Sb/R618lbWOYthK66DtUP4LQ6+sSf5adpigSuzI9gPNJx2GfDRs/kRom5yrQZ+BZZimuW Kw/uyBOR/CJdx7MeQXRK27zk5eL7EyIqYrH2YJw42JZchqDN//DBVMdIwNo/jwpulohH2I B0xZ8oHd1TT/lVWz3fU/INd2RklxXQk= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=205eJWBx; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of rientjes@google.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715112568; a=rsa-sha256; cv=none; b=ZCbU3jFR6jokWke8y1xcN3tbcz/3iXd+JNGALSqXYvbPDq2O3+M069sl5KL7NzBK23Sp61 tNBZRMd13MUGA0Ga0QsPp+vZ/I/oV2/Hl9iPeVPzXfDppHDoS9IIdO7YeRSOKDQxsSgc0e 4PpwtT26YqzDir6pJy7t/nL2YHRsHic= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1ee5f3123d8so6185ad.1 for ; Tue, 07 May 2024 13:09:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715112567; x=1715717367; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=eLq2KEdeSgHFupFQ5aOdwVPPc1VFbSlxs3IcQ2BaxEE=; b=205eJWBxjWu4hZCI+uGoc+dUF82Y+I1j4L+HlewB7UU+3K2i/AhNLRdF3XbH+5IbvX AKftMqwc2gOwx8h+bvlatpj+KdD7iBVb72MLSQPqCCHAQs2DnI8MAmIw6nbAE4W4FN1F NzJgQ5zYJBKiQlr7GAD7iFcLPt6NAdy5q97Sfi8AKy+kEoFEstV2umR0D2LNAet8XzVD q/oIEV0FJ8U0LCsWlzbj8bc26z0HnD2wVAjI2Ml7EBjWT0euT2D3ZZZ3uwxkM9fKJp9T QUswb07YqObZ9T1S3EudNLwaz/MPJw91IdGA5AQA9YhlQWeo5YGnWRtQKrjFsR7LtaZg /eJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715112567; x=1715717367; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eLq2KEdeSgHFupFQ5aOdwVPPc1VFbSlxs3IcQ2BaxEE=; b=W+WPkTSqpFbOu3BdPlzCdLcPnSKulXSWaZZ1tTPddDq7yBe8AcjXYo6uaNQ/9omOv8 9gMut8VENrfWGzHAr0cLaXhdzwKILG69FgkHFBV52TQXjBHsQwKGfbSgif2sGVjS2P5L HQpg4tU5Yj2ShoXrYjstmSrmqpA8LaiV34EAxJEadJN/7BBiG/wn4Di7ctVjRTjp5I9i InNoF2mRBJuRTrnBa6NFx6mz3zq9HbXsYgM0LKByVq7UYwMmdCtmBXsQtkbNcVqDptsr voWjYvkLnG4xPDgYDNv1pV5HyCnNrHN/0UtQV/nz0912fgnOmVnTGS6j7o9T63afzh00 w2kg== X-Forwarded-Encrypted: i=1; AJvYcCU36Drk0FccFDIrMKpXrRXg7F1xo/1Gna9ezBKKj04Uf72UbJSGL4M89KhvbwMiB+83n8kp/rrs1D7N8V9EDYraM+Y= X-Gm-Message-State: AOJu0YwDU2qfeEIXY73taXe54ZlIrY700wiZjh90p2c4C2lyUw/6jqgY NQPrBxHN+pe45al0MeDeqC/YjV+urxOBISev2uYMgPAhUL4Tw1OGLXsh8tQkPg== X-Google-Smtp-Source: AGHT+IGJzNvo7hA15kj2mpmx5NS+dHPHot+HZ5z130gW5QGknCxYSIHcQh6k6zYupk+nS49fCWaGRw== X-Received: by 2002:a17:903:2348:b0:1e5:d3d2:1af1 with SMTP id d9443c01a7336-1eeb6fa7e66mr451865ad.25.1715112566843; Tue, 07 May 2024 13:09:26 -0700 (PDT) Received: from [2620:0:1008:15:ac86:3e37:250c:8f8] ([2620:0:1008:15:ac86:3e37:250c:8f8]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b001e446490072sm10675872plb.25.2024.05.07.13.09.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 13:09:26 -0700 (PDT) Date: Tue, 7 May 2024 13:09:25 -0700 (PDT) From: David Rientjes To: Michal Hocko cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, Dan Williams , John Hubbard , Zi Yan , Bharata B Rao , Dave Jiang , "Aneesh Kumar K.V" , "Huang, Ying" , Alistair Popple , Christoph Lameter , Andrew Morton , Linus Torvalds , Dave Hansen , Mel Gorman , Jon Grimm , Gregory Price , Wei Xu , Johannes Weiner , SeongJae Park , David Hildenbrand , Davidlohr Bueso Subject: Re: [LSF/MM/BPF TOPIC] Locally attached memory tiering In-Reply-To: Message-ID: <99aecbfa-8cda-8176-7ca8-7bee18ea45a8@google.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspamd-Server: rspam01 X-Stat-Signature: gpz7atuc6uin7nbmqdfa3s7a8brz8ytz X-Rspam-User: X-Rspamd-Queue-Id: 83A611C000B X-HE-Tag: 1715112568-243212 X-HE-Meta: U2FsdGVkX19YmVKO5c18zDqczDq0hwvywpsp4XpX9sv4beQwdrrZEutT0nA9fb8o5HYUQTYHQcgm/5nh+rVGXeGLUbrYKFWMXIfH1UYcv6uYipG/2qpNCA91HxvkfGicXYCA/BnT7Tv7EPQNZb67JUBlwSIaQ7zdygwBRn3/enEEKlDLIAAjP/57V/T+UtFH9xVMucloyVGaznG4Usu5FeVIszaSg5tEJUUrGxbIXl2akvyt/rtmU+QmcSttJ9jQVPWFNbjaD6n/YsKC+pilyllUPqIlqKBuJ62uHgPIdUwAPvpFD4B11yG95rA6EhD+3uaN/vuZm98CLzmrUDZkcX4bsezXdq2IKlwTOsmhOnOd0jAslbC/4VAkmwzakawzKoh406xbjlo84hAMqNqUA2ekHxfrmN0Kt8iGb8hr5bIo2cqxgToR6TMUBuYqYpKAs/rtIUJDGo7sjwCg5Kwjejs7IunaOBVW+Z6ME6FVMj7RWXCP2xvRc0OrO8xlaOUxfb723FYYZq5q0j0nuUV2owk8Z6G+bkC+4cS1ujMWgo7+YmVk5SL1LQkJfMQjpWPEfLsJ+u+r6LP7Co4OeIevoDYcHeF42D1YnGezCLrGaOpbGAGXsm7eRJYGakn0qvNwpYQVu3vs/3QdMQHoX3gkInuHEmBor6gusjhxlhdrODbZwLQgfnFyH+aMQkVrCRWVp14rGMECi9iBGIXG2MhjRo7GO4kUQAPqwAcefO4UAqDMVibOPbanxqe8+ZB1EmdKf5f1xBsgey8QuG8Iu6czDkft2Z+RKVAsRss36RgT1yVyNValQaOSZIVsXphBvttIz0PmLdO6aGT56j3LgH6qE/uvhFXsmQF7GAOTKsZbaYL5UKbRCMmPKwr4LQwe3kErwvKRu61uP4oLYqRBy7h7WyeIo7ZURwQlCthTuIR82iJv9y/NFDv9XeAIcdmvCd2Z+RxuF/h8HaICudZ8N70 Qm6tUbM3 VzGN8zAIIMJ/tr/n8iQmO7p5QepCyDT6aJVo4K8LCro+w4MRA7AlkQ/fO/eX9l/jf20JO70HmPnq5Orq2a0ytyAWjTVx4KJMarY/vjqo2M3/PXjA61+W/f8MT1eGUTtW1F+l1cCfI8JQOHzl0QzaCHattyh99Dvnoz4a9c20Bd+DuCiEBeYPUp72zKo9VLb45ynguIwGyOm7owFh58lTNI0BfxfOn5uJsT7QslWoKzPwCqfm/+RNCXhp4uau8SCyy4f9ue4f4wy64Sxcig30pvpuvjUHgCRHSRkzCNVMbCv5bkv4AW4p4RiGdx/4jJkgLup5hGms2VxqE8tGhoVYQ71NS/Py0EITgiYmAy3bwuQez2HYcx2gks8DA2FCGPlWY4iC07BHJhlxmE0pfqPN6udXlPFYXmO61k3Hh25pn0q+yhHmB3hPxnWUH4mmV/wvSxCAb4unwvt2mfeOKGCaGT4WRLeJMlmRGfV9z X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, 7 May 2024, Michal Hocko wrote: > On Mon 06-05-24 20:37:19, David Rientjes wrote: > > Hi all, > > > > I think it would be very worthwhile to have a block set aside for > > discussion on locally attached memory tiering extensions at LSF/MM/BPF > > 2024. > > > > Primarily interested in discussing Linux enlightenment for CXL 1.1 and > > later type-3 memory expansion devices (CXL.mem). I think we could touch > > on CXL 2.0 and later memory pooling architectures if we have time and > > there is interest, but the primary focus here would be local attached. > > > > Based on the premise for a Memory Tiering Working Group[1], there is > > widespread interest in the foundational topics for generally useful Linux > > enlightenment: > > > > - Decoupling CPU balancing from memory balancing (or obsoleting CPU > > balancing entirely) > > > > + John Hubbard notes this would be useful for GPUs: > > > > a) GPUs have their own processors that are invisible to the kernel's > > NUMA "which tasks are active on which NUMA nodes" calculations, > > and > > > > b) Similar to where CXL is generally going, we have already built > > fully memory-coherent hardware, which include memory-only NUMA > > nodes. > > > > - In-kernel hot memory abstraction, informed by hardware hinting drivers > > (incl some architectures like Power10), usable as a NUMA Balancing > > backend for promotion and other areas of the kernel like transparent > > hugepage utilization > > > > - NUMA and memory tiering enlightenment for accelerators, such as for > > optimal use of GPU memory, extremely important for a cloud provider > > (hint hint :) > > > > - Asynchronous memory promotion independent of task_numa_fault() while > > considering the cost of page migration (due to identifying cold memory) > > > > - What the role of userspace plays in this decision-making and how we can > > extend the default policy and mechanisms in the kernel to allow for it > > if necessary > > > > Additional topics that you find interesting are also very helpful! > > > > I'm biased toward a generally useful solution that would leverage the > > kernel as the ultimate source of truth for page hotness that can be > > extended for multiple use caes, one of which is memory tiering support. > > But certainly if there are other approaches, we can discuss that as well. > > > > A few main goals from this discussion: > > > > - Ensure that proposals address, or can be extended to address, the > > emerging needs of the various use cases that users may have > > > > - Surface any constraints that stakeholders may find to be prohibitive > > for support in the core MM subsystem > > > > - Alignment and division of work for developers who are actively looking > > to contribute to this area > > Do you think having 2 contigious slots would be sufficient for these > topics? > Yes, I think that makes perfect sense. > > As I'm just one of many stakeholders for this discussion, I'd nominate > > Michal Hocko to moderate it if he's willing to do so. > > Sure I can help out with that. > Thank you Michal! All: if there are additional topics that we discuss in this block beyond what is listed above, that would be great feedback. We can make sure that it is all covered in session, so please expand upon the above for things that we should cover. Thanks!