From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 490FAC761AF for ; Fri, 31 Mar 2023 11:45:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BBABE6B0071; Fri, 31 Mar 2023 07:45:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B6AF46B0072; Fri, 31 Mar 2023 07:45:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E4226B0075; Fri, 31 Mar 2023 07:45:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 871A36B0071 for ; Fri, 31 Mar 2023 07:45:34 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5BB0B14038F for ; Fri, 31 Mar 2023 11:45:34 +0000 (UTC) X-FDA: 80629013388.14.15EA91D Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by imf21.hostedemail.com (Postfix) with ESMTP id 300F61C001E for ; Fri, 31 Mar 2023 11:45:30 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=XfVbUM0R; dmarc=pass (policy=none) header.from=samsung.com; spf=pass (imf21.hostedemail.com: domain of ks0204.kim@samsung.com designates 203.254.224.25 as permitted sender) smtp.mailfrom=ks0204.kim@samsung.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680263131; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xe7BtmtLnH4DXm8w21CW7Kud50GrprJxmJ107O9JGIg=; b=JWTipVISkGAFvYShWQ2zwBJXGXHBcVY3VKuFLE1VrrCoe3u40khexm28GdDfq1Qkgo2PKl lujPCb6As4CwM5EXhY9wM5UfF4BfDkBgkJzAfS0JlDQofK0UJvWyE2PbSovDb/mWrf4jR7 uxW6S/MFPcOSyIV2dbtzvcrCWxNnGf0= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=XfVbUM0R; dmarc=pass (policy=none) header.from=samsung.com; spf=pass (imf21.hostedemail.com: domain of ks0204.kim@samsung.com designates 203.254.224.25 as permitted sender) smtp.mailfrom=ks0204.kim@samsung.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680263131; a=rsa-sha256; cv=none; b=ypHbrj04mm02CinfpuL5gyeOgh7rqLEvm4JWl34oIPx7XAOMVgpxHegRapRURwyxxn5xZm ha1OWQSKyq0S/Mr0E7YRpf+yhHCIfo15t73XHsKMj71FBji1V7/M3q8Iv7W7pknkOyfUim vKRp08NvuRQSg/cPG4bkrfyYrTh9rNU= Received: from epcas2p1.samsung.com (unknown [182.195.41.53]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230331114527epoutp02f0f9d769c87c8f48d28f2759e6272d91~RfmmNXCN10363803638epoutp029 for ; Fri, 31 Mar 2023 11:45:27 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230331114527epoutp02f0f9d769c87c8f48d28f2759e6272d91~RfmmNXCN10363803638epoutp029 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680263127; bh=xe7BtmtLnH4DXm8w21CW7Kud50GrprJxmJ107O9JGIg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XfVbUM0RSvyRvxOg9QN5SibBXJ8cg5eyYAI/QwmIa+rIQVVhtE0Oswx9xVblPg8RT rLMdkadi6sDxDnD0XrMFlSYUhkUKEhMB3mIt0XUPz5mbtMrpGa64qRO2hx3zwGZNoU mEs1iQEToBwppEJeZCFVQI2+g91PefkeiXaBhm04= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas2p1.samsung.com (KnoxPortal) with ESMTP id 20230331114527epcas2p1f49419ce25b034fae0c57aa9147d6a6a~RfmlqHd_l1988319883epcas2p1O; Fri, 31 Mar 2023 11:45:27 +0000 (GMT) Received: from epsmges2p4.samsung.com (unknown [182.195.36.92]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Pnz4Q41Zwz4x9Pt; Fri, 31 Mar 2023 11:45:26 +0000 (GMT) Received: from epcas2p4.samsung.com ( [182.195.41.56]) by epsmges2p4.samsung.com (Symantec Messaging Gateway) with SMTP id 75.EE.35469.6D7C6246; Fri, 31 Mar 2023 20:45:26 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas2p2.samsung.com (KnoxPortal) with ESMTPA id 20230331114526epcas2p2b6f1d4c8c1c0b2e3c12a425b6e48c0d8~RfmkqTeCu0937109371epcas2p2e; Fri, 31 Mar 2023 11:45:26 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230331114526epsmtrp2891df4bff90a96ef8670d98fec8cf1fd~Rfmkpg6oW0911809118epsmtrp2x; Fri, 31 Mar 2023 11:45:26 +0000 (GMT) X-AuditID: b6c32a48-791ff70000008a8d-b9-6426c7d65967 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id D4.62.31821.5D7C6246; Fri, 31 Mar 2023 20:45:26 +0900 (KST) Received: from dell-Precision-7920-Tower.dsn.sec.samsung.com (unknown [10.229.83.99]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230331114525epsmtip1e121ff5f214bbf1e6620326f323a66c3~RfmkecJAs1642716427epsmtip1o; Fri, 31 Mar 2023 11:45:25 +0000 (GMT) From: Kyungsan Kim To: rppt@kernel.org Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-cxl@vger.kernel.org, a.manzanares@samsung.com, viacheslav.dubeyko@bytedance.com, dan.j.williams@intel.com, seungjun.ha@samsung.com, wj28.lee@samsung.com Subject: Re: RE: RE(2): FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL Date: Fri, 31 Mar 2023 20:45:25 +0900 Message-Id: <20230331114525.400375-1-ks0204.kim@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrIJsWRmVeSWpSXmKPExsWy7bCmhe6142opBm8WSVtMP6xoMX3qBUaL 87NOsVjs2XuSxeLemv+sFvte72W2OLJ+O5PFi87jTBYdG94wWmy8/47Ngcvj34k1bB6L97xk 8ti0qpPNY9OnSewek28sZ/To27KK0ePzJrkA9qhsm4zUxJTUIoXUvOT8lMy8dFsl7+B453hT MwNDXUNLC3MlhbzE3FRbJRefAF23zByg85QUyhJzSoFCAYnFxUr6djZF+aUlqQoZ+cUltkqp BSk5BeYFesWJucWleel6eaklVoYGBkamQIUJ2Rnvfh1lLOiVqmg+uJ6tgXGSaBcjJ4eEgInE /jMTWLsYuTiEBHYwShy6v4AJwvnEKHF68nxmkCohgc+MErfnmcF0vJu5mh2iaBejxKTmtSwQ TheTxPrv71hAqtgEtCX+XDnPBmKLCAhLPPo5DyzOLPCPUWLPZUkQW1ggSOLwkU+MIDaLgKrE 99d3WEFsXgEbiXs3prBCbJOXmHnpO9A2Dg5OAS2JA2f5IEoEJU7OfAI1Ul6ieetsZpAbJAR6 OSRefD7PDNHrInFkYi8jhC0s8er4FnYIW0ri87u9bBB2scTj1/+g4iUSh5f8ZoGwjSXe3XzO CrKXWUBTYv0ufRBTQkBZ4sgtqLV8Eh2H/7JDhHklOtqEIBpVJLb/W84Ms+j0/k1Qwz0kTp9f yA4JzlqJHTc7WScwKsxC8swsJM/MQti7gJF5FaNYakFxbnpqsVGBCTx6k/NzNzGC06qWxw7G 2W8/6B1iZOJgPMQowcGsJMJbaKyaIsSbklhZlVqUH19UmpNafIjRFBjSE5mlRJPzgYk9ryTe 0MTSwMTMzNDcyNTAXEmc92OHcoqQQHpiSWp2ampBahFMHxMHp1QDU0PDbGuvSa3hq87cDHx+ Uf+vv8D83s86bJ6ly3o0/q7IvRy8IZT5n/K0lD1Ldl6JNJf6kLrviiJnBDuf+Srl37Er7woo ral5afw1dt2MqSFdecIXWBXe/tniu/h1kuxa27//81oYM3ucJ7YpfzNQNppn+afFJUl20XPP 3ccWRuk1F7T8NlRW/3RNb/rtLZMqzXYfjxM8XxB0OJ9F7k57dCp/9bJjASafcz27T51afu5v tdLjbmeNjMbECat4NcVKhB08GVqT6qxTZdKP/OXiuKA7y2Hb7oN5nReK94q8/LCrMuPMirYr Z19sD1id6HY3KyA55lEy4yL3H5teL1jKE3JzNl954N/osCM7fPLeKbEUZyQaajEXFScCAMTp xEU0BAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrHLMWRmVeSWpSXmKPExsWy7bCSnO6142opBktOMFlMP6xoMX3qBUaL 87NOsVjs2XuSxeLemv+sFvte72W2OLJ+O5PFi87jTBYdG94wWmy8/47Ngcvj34k1bB6L97xk 8ti0qpPNY9OnSewek28sZ/To27KK0ePzJrkA9igum5TUnMyy1CJ9uwSujHe/jjIW9EpVNB9c z9bAOEm0i5GTQ0LAROLdzNXsXYxcHEICOxgl7l5ezgyRkJJ4f7qNHcIWlrjfcoQVoqiDSaJh 3TwWkASbgLbEnyvn2UBsEaCiRz9B4lwczCBFWy9PB0pwcAgLBEicPgs2iEVAVeL76zusIDav gI3EvRtTWCEWyEvMvPSdHaScU0BL4sBZPpCwkICmxO5L79ghygUlTs58AraWGai8eets5gmM ArOQpGYhSS1gZFrFKJlaUJybnltsWGCUl1quV5yYW1yal66XnJ+7iREcBVpaOxj3rPqgd4iR iYPxEKMEB7OSCG+hsWqKEG9KYmVValF+fFFpTmrxIUZpDhYlcd4LXSfjhQTSE0tSs1NTC1KL YLJMHJxSDUxa0SXLTvnV3+mTry5TEykUCZyfouQnktL9OtpguWYkw7a45Y+3s91ZKST1epHG m9k/8+YomjJEvvUVmLnz3/9uvhP/bOyPvti97nVP2dwXN9RS1s2Pcvr//1KvwtyPT5flbz8w f9c8pyKexn2W5V3Pnu7Z13A2fV7Ypv3SH/7+MK3dvCv51foVT8/cdLgnk36p6ravZFNPfuvs V+/s38at8mMo/n5VXobx1C2xB8Z+B+ojF+Tt2/ddv0lWuselcK8CW1vWsiLFpJqU9OD9R34b MkyyDmO/ZzTxlX1lfZSuUHzO40kuh15IRCzpT8/ZtNlz3s5pkmrnGy3jU3br/csq/yKgGfWd N2KCjsC/9lQlluKMREMt5qLiRABqK+oa8QIAAA== X-CMS-MailID: 20230331114526epcas2p2b6f1d4c8c1c0b2e3c12a425b6e48c0d8 X-Msg-Generator: CA Content-Type: text/plain; charset="utf-8" X-Sendblock-Type: AUTO_CONFIDENTIAL CMS-TYPE: 102P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230331114526epcas2p2b6f1d4c8c1c0b2e3c12a425b6e48c0d8 References: X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 300F61C001E X-Stat-Signature: zpxq77p43784k5nqqs6gtpp36yfhuyb5 X-HE-Tag: 1680263130-660036 X-HE-Meta: U2FsdGVkX1/ghkakLreWm8HXCsDMStPyOP9gT2MsWsn07rqdsk1ow0BR2CXRkHQa4A/u+hi1cB7rZueK22OhfIywsyF5Sbi8UdlHtgXM69m29tlaWpq9a+KCb2hmCAnWjKSGJ37TgMIltdgb6ZLsEq0khGexD8/EBl5eiJ8NCyFCLYtyI12Ha7r8XO4yxwFLFL0A++mIARNIoUSoj+u2Pco4iEQPaEIPUo1PkkKZGsyawnfpsig3VNWMyFyaRzoJp/GSQ25H89KDoQs/v1ZiftWBjIt1ZJRjHseo/giyid7KhshbDn/ZAmnav88NeOebl4KzhHhfIvPcTyToR/t+2inqh+cYUbSiNnVOjDo4R2YyVPJ6Y3iROQsU9+mcQvdSCxVpCsaWbSoxr7ucopTisRChXCOmx2INb5UsCZOrHR3t67LVRfS6dhd4RJK4QYcE3+pqcTfI1Rcwkk9AZH+n96Yppy4WoQCsO/XXHYpoMgSuwDv840OvQ1a2NwTbCve2glaamXvcGzANc3YkBDoOXqOVog9/Z1CuC9epzsIRmy4kS4IvVUFO/olf8aIIOn05gYZuW/J/IlP+KqFSN3D570YtP57Zx53Y8g8IA1Nwmi32/AaQuzoBYVerHnQb1sTfpXUxbbMO3UVacXo24fwPFgVUESPt2yx+y4CL1xEGfhMQ0EcjMtLUL998GdWz/4lkNhzkKhvn3CqBL4Pst0Us/IuucYYM9uH2sk4m8MmUItvjMegtixTfwiABsXpDOoHFUk84lmIchXxuB6gMInggA0RnuQoBPXXk9AbL9mzkWuXwGm11d3Wcb3tHsnT3GV2KdY+Z3GNnYKnzWTCPtOgMX1XG8IjKaY1F4qcIF7UpSCL0u/Hlc6i2u6CjvCO8rRm4AAEf9vZEgVC51KtqkwM7U7TDRrMzcPlmxqrdhCj+J8p9sRAx/ynKyeZa4wXZCM4q1GY4TA5r/ij5Y58mHVd WPq2YFhv hINFHmHebFCKAoBQ81W29Sx2romfw5x6FUaZIonS6m+SULmLPwT/iCOQHPjQwRnhX2sKw4Dya3n1cpL1IumctZ/FBVIE6k+OVzjn1PK5I95wdrROLvH0ohOgdGV8XaLnixngUTXngP7hFe6Y/chvWyRAoSQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Thank you Mike Rapoport for participating discussion and adding your thought. >Hi, > >On Thu, Mar 23, 2023 at 07:51:05PM +0900, Kyungsan Kim wrote: >> I appreciate dan for the careful advice. >> >> >Kyungsan Kim wrote: >> >[..] >> >> >In addition to CXL memory, we may have other kind of memory in the >> >> >system, for example, HBM (High Bandwidth Memory), memory in FPGA card, >> >> >memory in GPU card, etc. I guess that we need to consider them >> >> >together. Do we need to add one zone type for each kind of memory? >> >> >> >> We also don't think a new zone is needed for every single memory >> >> device. Our viewpoint is the sole ZONE_NORMAL becomes not enough to >> >> manage multiple volatile memory devices due to the increased device >> >> types. Including CXL DRAM, we think the ZONE_EXMEM can be used to >> >> represent extended volatile memories that have different HW >> >> characteristics. >> > >> >Some advice for the LSF/MM discussion, the rationale will need to be >> >more than "we think the ZONE_EXMEM can be used to represent extended >> >volatile memories that have different HW characteristics". It needs to >> >be along the lines of "yes, to date Linux has been able to describe DDR >> >with NUMA effects, PMEM with high write overhead, and HBM with improved >> >bandwidth not necessarily latency, all without adding a new ZONE, but a >> >new ZONE is absolutely required now to enable use case FOO, or address >> >unfixable NUMA problem BAR." Without FOO and BAR to discuss the code >> >maintainability concern of "fewer degress of freedom in the ZONE >> >dimension" starts to dominate. >> >> One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases. >> ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time. >> ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation. >> Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag. >> In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped. >> We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases. >> As you well know, among heterogeneous DRAM devices, CXL DRAM is the first PCIe basis device, which allows hot-pluggability, different RAS, and extended connectivity. >> So, we thought it could be a graceful approach adding a new zone and separately manage the new features. > >This still does not describe what are the use cases that require having >kernel allocations on CXL.mem. > >I believe it's important to start with explanation *why* it is important to >have kernel allocations on removable devices. > In general, a memory system with DDR/CXL DRAM will have near/far memory. And, we think kernel already includes memory tiering solutions - Meta TPP, zswap, and pagecache. Some kernel contexts would prefer fast memory. For example, a hot data with time locality or a data for fast processing such as metadata or indexing. Others would enough with slow memory. For example, a zswap page which is being used while swapping. >> Kindly let me know any advice or comment on our thoughts. >> >> > >-- >Sincerely yours, >Mike.