From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37DC8C761AF for ; Fri, 31 Mar 2023 11:46:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ACFF26B007D; Fri, 31 Mar 2023 07:46:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A7F136B007E; Fri, 31 Mar 2023 07:46:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FA6A6B0080; Fri, 31 Mar 2023 07:46:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 81B216B007D for ; Fri, 31 Mar 2023 07:46:57 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 543DD40E94 for ; Fri, 31 Mar 2023 11:46:57 +0000 (UTC) X-FDA: 80629016874.29.687C357 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by imf20.hostedemail.com (Postfix) with ESMTP id 4DBB11C0022 for ; Fri, 31 Mar 2023 11:46:53 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=F0mUtfYX; spf=pass (imf20.hostedemail.com: domain of ks0204.kim@samsung.com designates 203.254.224.25 as permitted sender) smtp.mailfrom=ks0204.kim@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680263214; a=rsa-sha256; cv=none; b=1azPk5qrelxvvlaOuHKRHxT/Iqcv7gXZSnUzA/Zht5xSD+hGsKjQ16LwDLe9z65Hw4xyzE Q7Wl8eobU9r4V3LX/xXpqfA2/maehzIalEa4O74w+KQjBMCcYoiLDA4/+L6iqf12nyYDO1 2bFt9pHxXB8OxJnpqcP5zwo+3gdh2+Y= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=F0mUtfYX; spf=pass (imf20.hostedemail.com: domain of ks0204.kim@samsung.com designates 203.254.224.25 as permitted sender) smtp.mailfrom=ks0204.kim@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680263214; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Elvc3NDLcXtB7fip/yBqKkhlU5FfkiSc7O30H4CxdNk=; b=kBgui/hRjXxfYDT6vvqVZSKyoR/XdEEh3LZEyOsoz//9bUtiyTdPwRODXc7v/fTEi/L2xL sgBi4ZEv7pDMcjdLVGn1jIGsMxIkFa0rjZrUnBfJqr09hQhlKnCmZ54ELsn1mhavVnJmDw n2hKJJmB7Aoph8QeFW/M90PxCGOiEjA= Received: from epcas2p4.samsung.com (unknown [182.195.41.56]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230331114651epoutp021b173e71f3b4e3eafa7000d75b7f75f0~Rfnz9bRWJ0494304943epoutp02s for ; Fri, 31 Mar 2023 11:46:51 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230331114651epoutp021b173e71f3b4e3eafa7000d75b7f75f0~Rfnz9bRWJ0494304943epoutp02s DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680263211; bh=Elvc3NDLcXtB7fip/yBqKkhlU5FfkiSc7O30H4CxdNk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F0mUtfYXBJNsYG3nfruPR5Gmhy9fusPyaOEoGFqjZnUk+y+AJ45J4Xk59tzgqT6jf fMwqdvZkFJ5N+6uCyUlUVzXUojSp9kFTfIIMJ4iwnopm2hbWfVlIDzxIp+Bm2TKlc/ OAyi3bVfLGHVaFCLvD1yym51ytnccbUwD2TB9iow= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas2p2.samsung.com (KnoxPortal) with ESMTP id 20230331114650epcas2p28970002302e889ad7a030982bd9dd4ee~Rfnzql2Th1108511085epcas2p2O; Fri, 31 Mar 2023 11:46:50 +0000 (GMT) Received: from epsmges2p2.samsung.com (unknown [182.195.36.101]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Pnz622HVsz4x9Pp; Fri, 31 Mar 2023 11:46:50 +0000 (GMT) Received: from epcas2p1.samsung.com ( [182.195.41.53]) by epsmges2p2.samsung.com (Symantec Messaging Gateway) with SMTP id 28.61.27926.A28C6246; Fri, 31 Mar 2023 20:46:50 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas2p2.samsung.com (KnoxPortal) with ESMTPA id 20230331114649epcas2p23d52cd1d224085e6192a0aaf22948e3e~RfnymVtVK2893028930epcas2p2-; Fri, 31 Mar 2023 11:46:49 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230331114649epsmtrp2ba6137db1fb1281dc84a9aa05d21a935~RfnylDZ8X1084510845epsmtrp2I; Fri, 31 Mar 2023 11:46:49 +0000 (GMT) X-AuditID: b6c32a46-a4bff70000006d16-9a-6426c82ad749 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 1E.06.18071.928C6246; Fri, 31 Mar 2023 20:46:49 +0900 (KST) Received: from dell-Precision-7920-Tower.dsn.sec.samsung.com (unknown [10.229.83.99]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230331114649epsmtip15e76f833be6302e8058f00f8dafc8eaf~RfnyV6-m91642716427epsmtip1x; Fri, 31 Mar 2023 11:46:49 +0000 (GMT) From: Kyungsan Kim To: dragan@stancevic.com Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-cxl@vger.kernel.org, a.manzanares@samsung.com, viacheslav.dubeyko@bytedance.com, dan.j.williams@intel.com, seungjun.ha@samsung.com, wj28.lee@samsung.com Subject: Re: Re: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL Date: Fri, 31 Mar 2023 20:46:49 +0900 Message-Id: <20230331114649.400453-1-ks0204.kim@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrIJsWRmVeSWpSXmKPExsWy7bCmqa7WCbUUg223OS2mH1a0mD71AqPF obk32S3OzzrFYrFn70kWi3tr/rNa7Hu9l9niRedxJouODW8YLTbef8fmwOXx78QaNo/Fe14y eWz6NIndY/KN5YwefVtWMXosXmrj8XmTXAB7VLZNRmpiSmqRQmpecn5KZl66rZJ3cLxzvKmZ gaGuoaWFuZJCXmJuqq2Si0+ArltmDtB1SgpliTmlQKGAxOJiJX07m6L80pJUhYz84hJbpdSC lJwC8wK94sTc4tK8dL281BIrQwMDI1OgwoTsjNNbZ7IUrJOs2POgiaWBcZpoFyMnh4SAicSR E8tZuxi5OIQEdjBKnP/7gg0kISTwiVHi2E8tiMQ3RomJnV8ZYToOHjnBBpHYyyjx7c4BVoiO LiaJY1+qQGw2AW2JP1fOg00SEZCQ2LdmEVgzs8A/Rok9lyVBbGEBF4ndN/ewg9gsAqoSC2fM BrN5BWwkdr56zQaxTF5i5qXvYHFOAUeJtr5trBA1ghInZz5hgZgpL9G8dTYzyEESAp0cEm+6 30Fd6iLRsXg9C4QtLPHq+BZ2CFtK4vO7vVALiiUev/4HFS+ROLzkN1S9scS7m8+BlnEALdCU WL9LH8SUEFCWOHILai2fRMfhv+wQYV6JjjYhiEYVie3/ljPDLDq9fxPUcA+JtY2rGCHBNoVR 4vDkX+wTGBVmIflmFpJvZiEsXsDIvIpRLLWgODc9tdiowAgev8n5uZsYwWlVy20H45S3H/QO MTJxMB5ilOBgVhLhLTRWTRHiTUmsrEotyo8vKs1JLT7EaAoM64nMUqLJ+cDEnlcSb2hiaWBi ZmZobmRqYK4kzittezJZSCA9sSQ1OzW1ILUIpo+Jg1OqgaleiuPclIIC7rkrCxJFj3R+PeHd sInf/JVBkaHwbcna06dkbTPeyEYZ+JyM7nb6GVp9IHnRXEd5Q0/rdXuNxNKmqRsY+6xIniR0 7daNkzbdbLHFiwp6+yLDNbd/WjHpj7nhx/KJn3aU+h5WDmmNO/lp5zm2O/sLQw3/XffdyPva jufhtek8Jxbt/LDZaF39lvVPmsydVdbIhCWfUux8/lfb/+GXqLlfHVW6thav7nWaqB4Vc7d/ yhuO9Y3qE8u8PvEZPKzR6+WaGlGa8vBWycugYxdXlkznFloRsafn3bv3lcx3uBU0Z2xtqio7 F1wQq2hu0W+mstK9saNR1DdkxYforSZt07L4L/X3fl0Rr8RSnJFoqMVcVJwIALJBKA40BAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrPLMWRmVeSWpSXmKPExsWy7bCSnK7mCbUUg9er1C2mH1a0mD71AqPF obk32S3OzzrFYrFn70kWi3tr/rNa7Hu9l9niRedxJouODW8YLTbef8fmwOXx78QaNo/Fe14y eWz6NIndY/KN5YwefVtWMXosXmrj8XmTXAB7FJdNSmpOZllqkb5dAlfG6a0zWQrWSVbsedDE 0sA4TbSLkZNDQsBE4uCRE2xdjFwcQgK7GSXuT/jNDJGQknh/uo0dwhaWuN9yhBWiqINJ4tnq BSwgCTYBbYk/V86zgdgiAhIS+9YsYgQpYgYp2np5OlhCWMBFYvfNPWCTWARUJRbOmA1m8wrY SOx89ZoNYoO8xMxL38HinAKOEm1921hBbCEBB4npyxZD1QtKnJz5BGwxM1B989bZzBMYBWYh Sc1CklrAyLSKUTK1oDg3PbfYsMAwL7Vcrzgxt7g0L10vOT93EyM4DrQ0dzBuX/VB7xAjEwfj IUYJDmYlEd5CY9UUId6UxMqq1KL8+KLSnNTiQ4zSHCxK4rwXuk7GCwmkJ5akZqemFqQWwWSZ ODilGpgCr9epx1/nEl6gLzRXSrRpRgOf6XSzVIs1HjnTenvWz0voM9A/JX3DfA9vsnO0XlL6 xEXmVrYJYhsPKv7m2a2/7vukP0duOR70Pi1g+/hD0N9rU2NDEj54NtzktVfx19q9fc9xg3Dl 3/dkP/xvz/x+LsDz49GZRlHX8yc/eev+X+C+h8irrCO35f4vftP3rkl/8vY9sp82sq8JdExQ K3BkKUmJSvt1RNjO2u++e5CSruSyy2xXDR3mHK/UMb4mUOahstn9qvW6k00c9evkt6g7S56b mb7sPePOui/SqnPehW/TFf52bsXyJoU17wLfWwqJOy4U/8PE6ueftdh1/smc2pIX3QWRW2vj JHMKNiixFGckGmoxFxUnAgDcHQgw8gIAAA== X-CMS-MailID: 20230331114649epcas2p23d52cd1d224085e6192a0aaf22948e3e X-Msg-Generator: CA Content-Type: text/plain; charset="utf-8" X-Sendblock-Type: AUTO_CONFIDENTIAL CMS-TYPE: 102P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230331114649epcas2p23d52cd1d224085e6192a0aaf22948e3e References: X-Rspam-User: X-Rspamd-Queue-Id: 4DBB11C0022 X-Rspamd-Server: rspam01 X-Stat-Signature: ethq1j4quka8gjhgtutaqr4ec61uyyz6 X-HE-Tag: 1680263213-557405 X-HE-Meta: U2FsdGVkX1/WaaXbyOf/VbCQpJcl8iXcvoBlXyHz5Z0XexwkqyJNVxz/z8zwKZzKyiDv67r8YUanXJzGGPkfS+iVjnG9SV8eN98T2pKEgP+Sa795afnFYoBwUsDYistebQi2C5hntiSXQSNh37emBRHPzr+9pGexf9CNuHMqDIgW1porcj3x0Y+9m8uwtnAYjPvsg5u7gR4wratoZWLBGtA6zysoQtBcw64Ln5jJ0FKj2efK48MWoOXHKVPaJ3Lx+XyNRvlwdjafh8rnp3sI/xX3isoci3ohG53XiXQfWYfJv1k3u3YYiQqo3sRzLr5PRALsljuybIjxauZGUpARYRcQFyGZcUxqxOCzHdd5glD6W5wHK1tHnSbrrDGppFRsraLg3RWYjYgE6F+wyKuGGK2+g4Glv9j/aw7apkPCC8h4KXkjCi7MYDiarTglEijA35CUexYGowmW94DTCUguosmUln0JvqhAqCRWlddTUFYUqFCcaP7wATbwgXpyP51YCadtGeVXjMh0XiJE2iwgAPcF2IkB2IVeYMJ7P4LZ2dMhe3LMi7e/hQU4dpdiS6b7zgpVbArfXYsJi9nt2rFSnH3abNjz+m0CvEqTCAl0eBm0d0Q8BSCofdQAioX/GM8koHULpbP1xRt7fQ5D34MvMoanqUIRjYM4A6RyYg6Hmrr+P4VpIMwMP4VHu5+8t+mcb7r+ygKXNzeTn5CpqLQZU6bn2zZ2jwRwhmZFg4vplVg9a2kZjW3mMvEf4V8GH29MLD57QC12PLobw4WP8OHVEWLUXD2Y+IxlVWgxslXJqUyMNd+IoXE4YBKrlpYx8C7HNTlv/QCVIK2vm64T8L4/Bq6EH7h8W8iUlKGJ0Vd4X1a2BJl0daTUip6tFgBnejOInW6RcD6E+yVaPkAQpbeA9J4+HYJyUd+VjL44r5pmnyxIXFWvelrqUQcoXj0rlGkX5z9rYnl/hOfiYeT1pr5 Zxub1rId hCZ+1NUt/GXJutnVIhZn5+pk1VQBog1fvoFUMzhJjcTryDeNGfEoHmn4fI7ZoyegYN1hEam8+KR0p0j9VjdB1sHnTTyVV+9iK8CnVgesuNt4vB40= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Dragan Stancevic. Thank you for your interests and joning the discussion. >On 2/20/23 19:41, Kyungsan Kim wrote: >> CXL is a promising technology that leads to fundamental changes in computing architecture. >> To facilitate adoption and widespread of CXL memory, we are developing a memory tiering solution, called SMDK[1][2]. >> Using SMDK and CXL RAM device, our team has been working with industry and academic partners over last year. >> Also, thanks to many researcher's effort, CXL adoption stage is gradually moving forward from basic enablement to real-world composite usecases. >> At this moment, based on the researches and experiences gained working on SMDK, we would like to suggest a session at LSF/MM/BFP this year >> to propose possible Linux MM changes with a brief of SMDK. >> >> Adam Manzanares kindly adviced me that it is preferred to discuss implementation details on given problem and consensus at LSF/MM/BFP. >> Considering the adoption stage of CXL technology, however, let me suggest a design level discussion on the two MM expansions of SMDK this year. >> When we have design consensus with participants, we want to continue follow-up discussions with additional implementation details, hopefully. >> >> >> 1. A new zone, ZONE_EXMEM >> We added ZONE_EXMEM to manage CXL RAM device(s), separated from ZONE_NORMAL for usual DRAM due to the three reasons below. > >Hi Kyungsan- > >I read through your links and I am very interested in this >talk/discussion from the perspective of cloud/virtualization hypervisor >loads. > >The problem that I am starting to tackle is clustering of hypervisors >over cxl.mem for high availability of virtual machines. Or live >migration of virtual machines between hypervisors using cxl.mem [1]. > > >So I was wondering, with regards to the ZONE_XMEM, has any thought been >given to the shared memory across virtual hierarchies [2], where you >have cxl.mem access over cxl switches by multiple VH connections. It >seems to me that there might be a need for differentiation of direct >cxl.mem and switched cxl.mem. At least from the point of view where you >have multiple hypervisors sharing the memory over a switch. Where they >would potentially have to synchronize state/metadata about the memory. At first, in general we have thought that more SW layers(baremetal, virtualization, orchestration) would be related along with the progress of CXL topology(direct attached, switch/multilevel switch, rackscale/inter-rackscale with fabric). We think ZONE_EXMEM can be used as a static CXL identifier between hypervisor and host OS interaction for memory inflation/deflation, transcendent memory interface(frontswap/cleancache)[1], and isolation. [1] https://lwn.net/Articles/454795 > >[1] A high-level explanation is at https://protect2.fireeye.com/v1/url?k=6962eb99-098076c4-696360d6-000babd9f1ba-f4ae8300c44044a7&q=1&e=fca5fea0-6b57-4874-8ec1-637a6c1019b6&u=http%3A%2F%2Fnil-migration.org%2F >[2] Compute Express Link Specification r3.0, v1.0 8/1/22, Page 51, >figure 1-4, black color scheme circle(3) and bars. > > >-- >Peace can only come as a natural consequence >of universal enlightenment -Dr. Nikola Tesla