From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4E53C10F1A for ; Tue, 7 May 2024 11:52:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1ED136B00B0; Tue, 7 May 2024 07:52:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 19DE06B00B1; Tue, 7 May 2024 07:52:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03DA66B00B2; Tue, 7 May 2024 07:52:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DB3546B00B0 for ; Tue, 7 May 2024 07:52:15 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8FAD21A0370 for ; Tue, 7 May 2024 11:52:15 +0000 (UTC) X-FDA: 82091436630.30.41F8E17 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf13.hostedemail.com (Postfix) with ESMTP id 758522000F for ; Tue, 7 May 2024 11:52:13 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=GQ5bq84q; dkim=pass header.d=suse.com header.s=susede1 header.b=GQ5bq84q; spf=pass (imf13.hostedemail.com: domain of mhocko@suse.com designates 195.135.223.131 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715082733; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=REGT61UewgNh1HiTzhkwLWC7OZP/vbULkFtNIWiPHZY=; b=pyBloyMWwoZLMQxNjh/0r5zsuSptXrIDozjHvIUJdIEya+MBuI07Icc9AGqN+zEskrh5jU kMYJfNoAITWzapZ3XjHkgUGuifNyEmrmWMnlanQRJgq5PhJpMNDii9dRkL5S+E/AXkKRLk H4gEQfrERN4SdWsn1l7T5Pq6AmBYGZo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715082733; a=rsa-sha256; cv=none; b=UtJ8br9PE6hqBBzERzMASA4qqvm8YUasrsbOafNpSUgNPAcTQkqQPhX/RHxR0wT1wXhiOu NKm2vx0FNBWNqZjxNNzt8HE7lD6i1ejp6DckcN1zgC/RYgU4sOqK2+SVYv5aNKSr1u4C9d gOeMu5zHkrCm1D8aqB1CLXguGh4VhqU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=GQ5bq84q; dkim=pass header.d=suse.com header.s=susede1 header.b=GQ5bq84q; spf=pass (imf13.hostedemail.com: domain of mhocko@suse.com designates 195.135.223.131 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 0DC3720926; Tue, 7 May 2024 11:52:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1715082732; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=REGT61UewgNh1HiTzhkwLWC7OZP/vbULkFtNIWiPHZY=; b=GQ5bq84q86Ov4jztSlkl6Xn8dI6t+0sYiQkIpd5gX74mxtsRx1nNSIbmt7YFpynyak1Kyw Hck1OSGmejzHUhCwl6AZt8ejlYTSgl9VWo4ynlhYiLuwshlPSfZVl5VKung3eOKs7XRj0x lbftepEoe0Ixm5CdPX6G7GGQ31ZFNQ8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1715082732; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=REGT61UewgNh1HiTzhkwLWC7OZP/vbULkFtNIWiPHZY=; b=GQ5bq84q86Ov4jztSlkl6Xn8dI6t+0sYiQkIpd5gX74mxtsRx1nNSIbmt7YFpynyak1Kyw Hck1OSGmejzHUhCwl6AZt8ejlYTSgl9VWo4ynlhYiLuwshlPSfZVl5VKung3eOKs7XRj0x lbftepEoe0Ixm5CdPX6G7GGQ31ZFNQ8= Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id E35F913A2D; Tue, 7 May 2024 11:52:11 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id nTB9NOsVOmZ8BQAAD6G6ig (envelope-from ); Tue, 07 May 2024 11:52:11 +0000 Date: Tue, 7 May 2024 13:52:11 +0200 From: Michal Hocko To: David Rientjes Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, Dan Williams , John Hubbard , Zi Yan , Bharata B Rao , Dave Jiang , "Aneesh Kumar K.V" , "Huang, Ying" , Alistair Popple , Christoph Lameter , Andrew Morton , Linus Torvalds , Dave Hansen , Mel Gorman , Jon Grimm , Gregory Price , Wei Xu , Johannes Weiner , SeongJae Park , David Hildenbrand , Davidlohr Bueso Subject: Re: [LSF/MM/BPF TOPIC] Locally attached memory tiering Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Action: no action X-Rspam-User: X-Stat-Signature: scxxrmwhy7ru5qjbdma95xjprtx3myox X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 758522000F X-HE-Tag: 1715082733-157511 X-HE-Meta: U2FsdGVkX19yl16T92JpJZNj5SZozHZUsOqisTicyUKyU4siL6m9uYG/mliL2fgC5K7pCC9bsZES3/luSd7iHRVahcs3q6heyTwsNK8I++NUpRZlpGpLBLkNjy+X46Cgni7S8AIAW1eYT4iScFzgMo4Wq8lFbMhIA63ngy9Xt5g0nNqN6ro9gDhzTGsK3gDWletNI5PzboI4gIsQSTO2edMQj9TxGW/QJOi1Sy5DUrXseTQvhGMv5rmpyEz8wY8644fgoHqiOqYsJ29pILYp1+L0gqNnEskOUXce1XHlYqXb9YiSbdfODEpQR8FIgPIL6NhFjRAFlo/QE/Qm1JUDPZBRwB6ZyU/9dNjyrFm52BubRWbPIfgvRvszv+n3VIEDT1qZAyGJkOAueYvLVwa4h/AlrhisDORMOClKJYP6f7lqQHddRkdrwunGALHcEsZh3LqhGvVUV7h7wN/XEtXxwFIrJ7JSqhMUrDXIjykh0GbBGfmweTzHYl7PO3TNtmV1wBPtlXuUigil6qJvXrwqPEWC5zMpNia4SuvBX/rcxt1Fw2OEKaXWyeCSEOS8VCKlu8TNfNGPXVShB5F2pcWFJfXBbtyZzxz0ifGKnbCbdA2Q64yXbyiSjVOZsA9vCFGKtVPNjn6qV04IMB6LAq28XtslG/V/C5K3zL3WXikgF8cHZsOwsJ/nkOuCYhuOSwyLMJIJh6/+x/nreXYWL9gxq6gPsdHMNYrpWy0l3bLYn7QtnGJfynO2yXfKoVgfoCgjwQncjXbtau25ntp14m81YPcWe+ta/eQ+8Ipfh8WoOeRqpqu9pV2NAaaC2ym+jM8FMmSDy2qLFbd8/z6aZnF7RmDLCBpFWACUT3FRnNO28TCDBp/ryLfXqlB+8jqS+5o9zIsrbTxfAXq7CPKuhUssO7dJgDfbAceORszTlrPdIGcSFUnfdz8lPwYPyc27VB5FB6Gh4dr7mtoIlv1s+78 ZfA9VqsH cJBmBx1WcCkBogMVNDFACi83Y0MzBGCIMHc0E1REHmINIYVczbMmUnvBawhlSSL3y3tLCx6HZHkPtWOW/+vs639tg4wPyhd9iqLpsp5have2O8SpclEBwvr0MlVRv5R3wzyxdy9r5MZ8OFJ44jetVmDFtaIP2zNzvbx52ek8A/WhQLsDrJoBMU01sUzFAJ6aSLIEW9o10qZoNYM349Yng4HaB+DwNdzTXAcwNSIkC13lJ3OWHYTgiIBy9GYzCUxjsjGpmeUAYJtt1UIqpfNpde3XkXn9QMzni0lrI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon 06-05-24 20:37:19, David Rientjes wrote: > Hi all, > > I think it would be very worthwhile to have a block set aside for > discussion on locally attached memory tiering extensions at LSF/MM/BPF > 2024. > > Primarily interested in discussing Linux enlightenment for CXL 1.1 and > later type-3 memory expansion devices (CXL.mem). I think we could touch > on CXL 2.0 and later memory pooling architectures if we have time and > there is interest, but the primary focus here would be local attached. > > Based on the premise for a Memory Tiering Working Group[1], there is > widespread interest in the foundational topics for generally useful Linux > enlightenment: > > - Decoupling CPU balancing from memory balancing (or obsoleting CPU > balancing entirely) > > + John Hubbard notes this would be useful for GPUs: > > a) GPUs have their own processors that are invisible to the kernel's > NUMA "which tasks are active on which NUMA nodes" calculations, > and > > b) Similar to where CXL is generally going, we have already built > fully memory-coherent hardware, which include memory-only NUMA > nodes. > > - In-kernel hot memory abstraction, informed by hardware hinting drivers > (incl some architectures like Power10), usable as a NUMA Balancing > backend for promotion and other areas of the kernel like transparent > hugepage utilization > > - NUMA and memory tiering enlightenment for accelerators, such as for > optimal use of GPU memory, extremely important for a cloud provider > (hint hint :) > > - Asynchronous memory promotion independent of task_numa_fault() while > considering the cost of page migration (due to identifying cold memory) > > - What the role of userspace plays in this decision-making and how we can > extend the default policy and mechanisms in the kernel to allow for it > if necessary > > Additional topics that you find interesting are also very helpful! > > I'm biased toward a generally useful solution that would leverage the > kernel as the ultimate source of truth for page hotness that can be > extended for multiple use caes, one of which is memory tiering support. > But certainly if there are other approaches, we can discuss that as well. > > A few main goals from this discussion: > > - Ensure that proposals address, or can be extended to address, the > emerging needs of the various use cases that users may have > > - Surface any constraints that stakeholders may find to be prohibitive > for support in the core MM subsystem > > - Alignment and division of work for developers who are actively looking > to contribute to this area Do you think having 2 contigious slots would be sufficient for these topics? > As I'm just one of many stakeholders for this discussion, I'd nominate > Michal Hocko to moderate it if he's willing to do so. Sure I can help out with that. -- Michal Hocko SUSE Labs