From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8E17C7619A for ; Thu, 30 Mar 2023 22:03:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17F006B0074; Thu, 30 Mar 2023 18:03:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 107EE900002; Thu, 30 Mar 2023 18:03:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9BD86B0078; Thu, 30 Mar 2023 18:02:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D4C796B0074 for ; Thu, 30 Mar 2023 18:02:59 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 99CC2C1005 for ; Thu, 30 Mar 2023 22:02:59 +0000 (UTC) X-FDA: 80626940478.08.BA4E4E0 Received: from dog.elm.relay.mailchannels.net (dog.elm.relay.mailchannels.net [23.83.212.48]) by imf25.hostedemail.com (Postfix) with ESMTP id EC889A001B for ; Thu, 30 Mar 2023 22:02:55 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=stancevic.com header.s=dreamhost header.b=hufabQvn; arc=pass ("mailchannels.net:s=arc-2022:i=1"); spf=pass (imf25.hostedemail.com: domain of dragan@stancevic.com designates 23.83.212.48 as permitted sender) smtp.mailfrom=dragan@stancevic.com; dmarc=none ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680213776; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rDBoPi7u8Hdh3eFhndrLHZE31sc7ofYBzJWm4lYAkos=; b=JfCy9XaTluVpaS7L4eOqSNmbZIo+zTSBJS5yo58cX6ddKnOIz+68BIxWC2BMMkXr1d+lZC puIWWdT20zo/c12L8RGizNUQIj9JLUksWnWPckxbMwKY3m7sRVltVlnEoefDSEODD9Pvgf psExBFLWipTHNZ3geZVCJv9izFoVnrA= ARC-Authentication-Results: i=2; imf25.hostedemail.com; dkim=pass header.d=stancevic.com header.s=dreamhost header.b=hufabQvn; arc=pass ("mailchannels.net:s=arc-2022:i=1"); spf=pass (imf25.hostedemail.com: domain of dragan@stancevic.com designates 23.83.212.48 as permitted sender) smtp.mailfrom=dragan@stancevic.com; dmarc=none ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1680213776; a=rsa-sha256; cv=pass; b=y7/lmkGOFrBx/+XkKz2ddoxvgAixEfwK5BIPZ1qu784KklC9tvriMPjtk0CkV9zbAvV0DV dIDhRnjz0KLX17yfRrUyde4gycfMHgpZKent75UllL/CYGFk1ByZd2zJt3O4JyqtBK7/2T sV93QjA1tH/swGXAdqN9gZpSf0rOO2M= X-Sender-Id: dreamhost|x-authsender|dragan@stancevic.com Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 87705141D2C; Thu, 30 Mar 2023 22:02:54 +0000 (UTC) Received: from pdx1-sub0-mail-a294.dreamhost.com (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id DF39414150F; Thu, 30 Mar 2023 22:02:53 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1680213774; a=rsa-sha256; cv=none; b=GxihehXc/ZVXJRjYYYK5nA0/9U+zrV1T1grQRFO+pYHT0JyXFo4km0cdUFCG/ZLWlaS63R 3rRdTseinHqETPIzEpYPNN6mG6374ORYnRFIlPvc9uS8+3gFWWMOMiscnFgDNIFT868NN5 vQrl9ZqyNBr4vHo6Yct2OKVMxPV7FHXife/Zeisg2X2IOr2seuii7Q32j81xEAjpxK8OIv 45Iu+rJ0COusSkoikL4CcYpA6VGrF5t4Gca+1Y8Voh/x2XVpRXUgrmmNuL85BH5tN6L8lm 37uYFScl+YC66Yg4VMwClmT0M3Qv7sTvJaW8+fCBR3mdwp3o38/D9Nw77GtTVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1680213774; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rDBoPi7u8Hdh3eFhndrLHZE31sc7ofYBzJWm4lYAkos=; b=vovd/zqcchTZbqeOwGNoDYRKB7K5sEtM6adMw5qdRjTTOaYQpFfbgADu/Jjiq2+P8F1M13 aVS2XI3oDnCKsJhpRZgeQiyxAHwzG05Guq0wqWZsgeppQBjQN1KO0vMddKlJyKj7dZz/p1 l96xpNH8v5Oe28UckKsjOao4jYdvnMvO6Gu+UkdQf0E+RgjjkbJGL/WCUZnxUwn+e7IBCP BQFl5M67be5D/3Sm6JabRRnN0XZCVVR/UYVoYBjWTG4WQgdCBSgmmOfTaUBs3xbFquSgVy 8izNbt9jvBRPLyEMx0+hwCtMOQAXzZNpgTFiXJrFw2Q9wCXJgVBk1msOwqzifA== ARC-Authentication-Results: i=1; rspamd-786cb55f77-d9lht; auth=pass smtp.auth=dreamhost smtp.mailfrom=dragan@stancevic.com X-Sender-Id: dreamhost|x-authsender|dragan@stancevic.com X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dragan@stancevic.com X-MailChannels-Auth-Id: dreamhost X-Trail-Thread: 64ee653e6e8b3c6f_1680213774245_3606843821 X-MC-Loop-Signature: 1680213774245:3282326279 X-MC-Ingress-Time: 1680213774244 Received: from pdx1-sub0-mail-a294.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.116.217.198 (trex/6.7.2); Thu, 30 Mar 2023 22:02:54 +0000 Received: from [192.168.1.31] (99-160-136-52.lightspeed.nsvltn.sbcglobal.net [99.160.136.52]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dragan@stancevic.com) by pdx1-sub0-mail-a294.dreamhost.com (Postfix) with ESMTPSA id 4PncqH3jMPzBY; Thu, 30 Mar 2023 15:02:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stancevic.com; s=dreamhost; t=1680213773; bh=rDBoPi7u8Hdh3eFhndrLHZE31sc7ofYBzJWm4lYAkos=; h=Date:Subject:To:Cc:From:Content-Type:Content-Transfer-Encoding; b=hufabQvnYtZ/bTfYroVuIVnOgUVGeVrgJwwLUC/E1om/WAvr35cc2MVPdjqBlmlbH 5iXKY9ianqKhLFUaPMRkzbgeXt8EwNwUcRC4mLwmqoNXYqzBh6nGI9GyNx/qA2awNr OMGjHChH9QQp0Fp0T9NUj2wkDlQkpiFBduBzH57WN9eQIR+qwdHuICnI4JzhsSYb5l NNUlNJcqPs/ZpDwPZZxAu1gtw5EQY+2Kfw8F401ZdAXO1jSIszliO27AhLxkOE5RS/ G61UZpI33yPBGoDqH/1x7dwnVwFgKErOOsmWhRO+nBG/EEAENeax92Wdog1oCC0e75 BQzOdy7sV9xVg== Message-ID: Date: Thu, 30 Mar 2023 17:02:49 -0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Subject: Re: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL Content-Language: en-US To: Kyungsan Kim , lsf-pc@lists.linux-foundation.org Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-cxl@vger.kernel.org, a.manzanares@samsung.com, viacheslav.dubeyko@bytedance.com, nil-migration@lists.linux.dev References: <20230221014114.64888-1-ks0204.kim@samsung.com> From: Dragan Stancevic In-Reply-To: <20230221014114.64888-1-ks0204.kim@samsung.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EC889A001B X-Rspam-User: X-Stat-Signature: 5qyjdswp3b6pzadgmgywt3yy7gzcwdcn X-HE-Tag: 1680213775-812162 X-HE-Meta: U2FsdGVkX1/PAhG3FMjMrbu/mCW48RZSi5j7Tk/ams+kG5DL8qm4xTVc3uBfyzluKiacf7JyPrb4o/LycNN1Uulkf85+CzVWfZOJ9QHr/hmZxMYDohC2yoNd7Uheka6ynAUFGR47MyBs9/qT+BLZdqGrKvxY6O+q6NgW9fB/faAzD3bc6nR0D+p3+XHoeXCbCev2OeIz5B9bCDfNeRLmqOxIccZcvueDo3xKuKj7a5xXty5GiSCjK4mjGHOiBq13QLsIkcehuwfewqUWlfyQApF1gJmegea7kJq2LyvI1mBskRhOLAteMH5jnBlqCSXfrFuYPeVSFWpm8eRZEEaa+z0Z3tyTX9cFk4gZ5gskRyzg/h1wRZAkwq0q+HuAcneQP2azswEnOslLGmkqpmijDUYwhtjy25ZKI2U4TaduIUTLB6OVykdKB3zK7QyFAFLccZ/e0k2VaCOJzmfydFTve4C2eSRVdD/WDV0cpxXb7eOwXCavP7PNAsPLTJI45O/5mU8mwt2UBftgL+l7yV3I5rF/xGVaL0RoRDJkZmC5TSAMS1+TbieECmMY8Ug7WhepxcnmH1TtyvowHa5acyQEGjtrq8/S9SLtPl/XMKclqvwyHiOgzTJ/K9Is/dnwIzJmE9Tu0tGOTLzcq6YdcJ8TraMRWFdtrFeymBQ5ZPSJAeQPVY3mIZAZHQUSKgbQaS7ALPPBylO+f5dNHgPcyzplThc2Wgf0jKHlNe/x7XRja4go/sbNiG7kX8qiUbcRuS9xTDJXtB8Z7I78CttfzfjA1b6JhQEKTilE7ZLFJs6hkKrU13wJx0MWEYHT9N5TviWfFBDKAKB9l18afh5naN6dxyb89iXpr3g08+clvasuxFkKRRT88K8ZsY5R2kDGJzo61WTB8Jq01t/vxls28nEdClGKLsm/7Gs9WbhbHv0w60WzgidbtLV5F8FhLJu6aWdpM8Y3qy2/3qaOIp8umHv KcdfVZjh zYDo1qtsNhZgsM/WOMjV7nMygTkVWOb9yt+Nm2MjOjnjTVjFDMJTxFDV/eKdjJF7iyekGGEOYiHu640s5yoPcgKSIuEZxUkFnbkmYJwE3eF5lPUyMZkVh11oiJonUn0JOqQPIEN8ZtUwlVAXrIr0MRoxilx7FgTmXI2ESjHcefzcdZGRh6xihI5lk0JpE+3ZiB4k1ati0chT/5+qmMRosXZTiqkcYeKWoCJvqCKtCBWhmfWFrAQsDua+9rQI8iCX6x7JGRcQ9pXJttOORA3H3DEx3XCevsyhBSqpxjpEiIhLO+P5dk1ByMhrU1/UB528ABGYSSSo8rL5LRFl+gfkU3OtdjNB7iYlCepx9GIyjArdifMqSaOG3zSv+sg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2/20/23 19:41, Kyungsan Kim wrote: > CXL is a promising technology that leads to fundamental changes in computing architecture. > To facilitate adoption and widespread of CXL memory, we are developing a memory tiering solution, called SMDK[1][2]. > Using SMDK and CXL RAM device, our team has been working with industry and academic partners over last year. > Also, thanks to many researcher's effort, CXL adoption stage is gradually moving forward from basic enablement to real-world composite usecases. > At this moment, based on the researches and experiences gained working on SMDK, we would like to suggest a session at LSF/MM/BFP this year > to propose possible Linux MM changes with a brief of SMDK. > > Adam Manzanares kindly adviced me that it is preferred to discuss implementation details on given problem and consensus at LSF/MM/BFP. > Considering the adoption stage of CXL technology, however, let me suggest a design level discussion on the two MM expansions of SMDK this year. > When we have design consensus with participants, we want to continue follow-up discussions with additional implementation details, hopefully. > > > 1. A new zone, ZONE_EXMEM > We added ZONE_EXMEM to manage CXL RAM device(s), separated from ZONE_NORMAL for usual DRAM due to the three reasons below. Hi Kyungsan- I read through your links and I am very interested in this talk/discussion from the perspective of cloud/virtualization hypervisor loads. The problem that I am starting to tackle is clustering of hypervisors over cxl.mem for high availability of virtual machines. Or live migration of virtual machines between hypervisors using cxl.mem [1]. So I was wondering, with regards to the ZONE_XMEM, has any thought been given to the shared memory across virtual hierarchies [2], where you have cxl.mem access over cxl switches by multiple VH connections. It seems to me that there might be a need for differentiation of direct cxl.mem and switched cxl.mem. At least from the point of view where you have multiple hypervisors sharing the memory over a switch. Where they would potentially have to synchronize state/metadata about the memory. [1] A high-level explanation is at http://nil-migration.org [2] Compute Express Link Specification r3.0, v1.0 8/1/22, Page 51, figure 1-4, black color scheme circle(3) and bars. -- Peace can only come as a natural consequence of universal enlightenment -Dr. Nikola Tesla