From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 445A9C6FD1C for ; Thu, 23 Mar 2023 10:51:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B77356B0072; Thu, 23 Mar 2023 06:51:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B265F6B0074; Thu, 23 Mar 2023 06:51:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C9546B0078; Thu, 23 Mar 2023 06:51:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 840276B0072 for ; Thu, 23 Mar 2023 06:51:15 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 38D99160675 for ; Thu, 23 Mar 2023 10:51:15 +0000 (UTC) X-FDA: 80599846110.05.361C423 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by imf11.hostedemail.com (Postfix) with ESMTP id 04DAD4001A for ; Thu, 23 Mar 2023 10:51:11 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=f9gjPKBy; spf=pass (imf11.hostedemail.com: domain of ks0204.kim@samsung.com designates 203.254.224.34 as permitted sender) smtp.mailfrom=ks0204.kim@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679568672; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MxviwYlwt0W/gci07Q9zodQuj98HPISdycPrqXMA2vw=; b=3+Qx5x6vTnXL3m+di/v6aefdTXYtAsJx21k4bIrJt4HXW9kiiMLgPlRX9Y1tw3kKg0xxr9 AQrbzSxoIBv3gxjotjkMGitkBCfcjavtlGBnVGHgpWhnIJf3BPTrRv3Rl5GUDfRdKJEnpE 5mV4TO7OaqdIMwZHmJo3OoOy+2CGAng= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=f9gjPKBy; spf=pass (imf11.hostedemail.com: domain of ks0204.kim@samsung.com designates 203.254.224.34 as permitted sender) smtp.mailfrom=ks0204.kim@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679568672; a=rsa-sha256; cv=none; b=d+LhgaROg6CeL1vFUPEHZ0PrkKueTDCxNlTKTVehwsWsc5A8KnHrrN2j9ysvBqpQ6LmnHn 5KABR+IlxucTs884u7GieoJniCTg68oITKLbAoPrggx3oxFQZrJW7EvwDhD+Ea4xImORof skTrSFyMM4/gm80dF2hU8oUEYF3qdv4= Received: from epcas2p1.samsung.com (unknown [182.195.41.53]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20230323105108epoutp04fce61ae39360fadfa52394a35fd32e72~PBs4gWCA11162411624epoutp04z for ; Thu, 23 Mar 2023 10:51:08 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20230323105108epoutp04fce61ae39360fadfa52394a35fd32e72~PBs4gWCA11162411624epoutp04z DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1679568668; bh=MxviwYlwt0W/gci07Q9zodQuj98HPISdycPrqXMA2vw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f9gjPKByFouRjU2cSjkkIrSxeHS8HTpS4nyod0/BnBPUDs++fqV8DKm9uxO6m6tPV kRDwac6ldY+KcaldLweIOElJBmq9lPDmRLZPfBoBwV4c8R4qs84Zbj0dF92Ebj52dg 4yEBxOFdHlvW0KjjO96vHTRrJ0omfQKt/hVIpG68= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas2p2.samsung.com (KnoxPortal) with ESMTP id 20230323105107epcas2p205cefa0fdeb0ef47cb44455c90a09507~PBs37wD4p0502205022epcas2p2S; Thu, 23 Mar 2023 10:51:07 +0000 (GMT) Received: from epsmges2p3.samsung.com (unknown [182.195.36.90]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4Pj2FR0hYGz4x9Py; Thu, 23 Mar 2023 10:51:07 +0000 (GMT) Received: from epcas2p2.samsung.com ( [182.195.41.54]) by epsmges2p3.samsung.com (Symantec Messaging Gateway) with SMTP id 2F.8B.08750.A1F2C146; Thu, 23 Mar 2023 19:51:07 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas2p3.samsung.com (KnoxPortal) with ESMTPA id 20230323105106epcas2p39ea8de619622376a4698db425c6a6fb3~PBs2btIaT0693906939epcas2p33; Thu, 23 Mar 2023 10:51:06 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230323105106epsmtrp1c28a31c8ae7fd6b73844c8806ab1086d~PBs2bC69B1681716817epsmtrp1o; Thu, 23 Mar 2023 10:51:06 +0000 (GMT) X-AuditID: b6c32a47-777ff7000000222e-82-641c2f1a0328 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id E2.B2.31821.A1F2C146; Thu, 23 Mar 2023 19:51:06 +0900 (KST) Received: from dell-Precision-7920-Tower.dsn.sec.samsung.com (unknown [10.229.83.99]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230323105105epsmtip2082c66a9db596e5a9db97cdac0f0dbaa~PBs2PkLfp1816818168epsmtip2r; Thu, 23 Mar 2023 10:51:05 +0000 (GMT) From: Kyungsan Kim To: dan.j.williams@intel.com Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-cxl@vger.kernel.org, a.manzanares@samsung.com, viacheslav.dubeyko@bytedance.com, ying.huang@intel.com Subject: RE(2): FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL Date: Thu, 23 Mar 2023 19:51:05 +0900 Message-Id: <20230323105105.145783-1-ks0204.kim@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <641b7b2117d02_1b98bb294cb@dwillia2-xfh.jf.intel.com.notmuch> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprCJsWRmVeSWpSXmKPExsWy7bCmma60vkyKwdVEi+mHFS2mT73AaHF+ 1ikWiz17T7JY3Fvzn9Vi3+u9zBYdG94wWpycNZnFgcPj34k1bB6L97xk8tj0aRK7x+Qbyxk9 +rasYvT4vEkugC0q2yYjNTEltUghNS85PyUzL91WyTs43jne1MzAUNfQ0sJcSSEvMTfVVsnF J0DXLTMH6BwlhbLEnFKgUEBicbGSvp1NUX5pSapCRn5xia1SakFKToF5gV5xYm5xaV66Xl5q iZWhgYGRKVBhQnbG9Q8PmAr2CFXsOa3VwNjH38XIySEhYCIxaepeli5GLg4hgR2MEnMmv2OE cD4xSqy+ewDK+cwosWn6axaYlsvnVjNBJHYxSnxt7WaDcLqYJObf7GIHqWIT0Jb4c+U8G4gt IiAjseH/dmaQImaB04wSJ8+/BysSFnCX2L6khxXEZhFQlfi+9AQjiM0rYCOx6PwjVoh18hIz L30Hq+cU8JLYePw7C0SNoMTJmU/AbGagmuats8EWSAj8ZJf4eWAOE0Szi0TDps9QtrDEq+Nb 2CFsKYmX/W1QdrHE49f/oOwSicNLfkP9aSzx7uZzoCM4gBZoSqzfpQ9iSggoSxy5BbWWT6Lj 8F92iDCvREebEESjisT2f8uZYRad3r8JariHxPoPj6AhOpdRYnf/PvYJjAqzkHwzC8k3sxAW L2BkXsUollpQnJueWmxUYAyP4eT83E2M4OSp5b6DccbbD3qHGJk4GA8xSnAwK4nwujFLpAjx piRWVqUW5ccXleakFh9iNAWG9URmKdHkfGD6ziuJNzSxNDAxMzM0NzI1MFcS55W2PZksJJCe WJKanZpakFoE08fEwSnVwMTLO1+/vHz6E8O72zK+8YmcNe7rcFP/nxZkIPHNwuPX5yVfTvRu bt8aWJXFf6mxZMHXWZM28ogKh/mvv39y2hOdL/dWqjrKqywTSz5SxXxsUrD7jh9FZ9axfeuO 87nTdy637JKaq++1l/N/n5gkVGzJs4uzqeh5T3rj+qAXd81eXpv5cWrumscfWPV/qF21PLul Oi9KerHu2uI3sYLl7DfSy6X1uf5Fn88s7J+4XZf5v5PfwTl2qUsP/QrYt+D3xHWRIUvW7Oh5 uUjUP+atoUb1FN3zb98a5MY6rBMJORah+2BlTNmHqqiMQu5eHbkejhUb8xamn690ntfyQ/zu 4T3xhxYzpZ7LbxL66ve2uV6JpTgj0VCLuag4EQCff14bJwQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrPLMWRmVeSWpSXmKPExsWy7bCSvK6UvkyKQc9/FovphxUtpk+9wGhx ftYpFos9e0+yWNxb85/VYt/rvcwWHRveMFqcnDWZxYHD49+JNWwei/e8ZPLY9GkSu8fkG8sZ Pfq2rGL0+LxJLoAtissmJTUnsyy1SN8ugSvj+ocHTAV7hCr2nNZqYOzj72Lk5JAQMJG4fG41 UxcjF4eQwA5GiZs7fzBBJKQk3p9uY4ewhSXutxxhhSjqYJI4+GYJI0iCTUBb4s+V82wgtoiA jMSG/9uZQWxmgYuMEk9uR4PYwgLuEtuX9LCC2CwCqhLfl54A6+UVsJFYdP4RK8QCeYmZl76D LeMU8JLYePw7C4gtJOAp8e/vb3aIekGJkzOfsEDMl5do3jqbeQKjwCwkqVlIUgsYmVYxSqYW FOem5xYbFhjlpZbrFSfmFpfmpesl5+duYgSHuZbWDsY9qz7oHWJk4mA8xCjBwawkwuvGLJEi xJuSWFmVWpQfX1Sak1p8iFGag0VJnPdC18l4IYH0xJLU7NTUgtQimCwTB6dUA1P1lY2sl7Lb bI2jVqiZz2Wqu5RsK7fMSvuYFov8v717mW3ufFpy91mwoOGtB556dx4+L5bZfTRxxdRp9z75 vztx/0fY2s/uqioHmaa5e3lvC5BQPJewsImJ4+PbU/zHL55cOjlyR9bVk2k6qyaF5L1z6D2i 8j0rpj/uSoncubPtiR61c/KrQkQ3/Xc+XsFmn6zd9SLWjveQdt3MlEtXmNcIzb5xRkhk+uXZ yyZdKb79Trs3/vKX+z8qQ0QczLqWf7VfOLdomsoiwXvyy27c2nJYv/rLj6zHB45mdt7PtWJb JHjkzd41u7u3CwW8ibk+02WpSIvB089fr9QIF+YpCywrXsE2M/qUwJIOTo4PGjvilFiKMxIN tZiLihMBGOBuweICAAA= X-CMS-MailID: 20230323105106epcas2p39ea8de619622376a4698db425c6a6fb3 X-Msg-Generator: CA Content-Type: text/plain; charset="utf-8" X-Sendblock-Type: AUTO_CONFIDENTIAL CMS-TYPE: 102P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230323105106epcas2p39ea8de619622376a4698db425c6a6fb3 References: <641b7b2117d02_1b98bb294cb@dwillia2-xfh.jf.intel.com.notmuch> X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: a76yptbikmmezoc61fdxoe15wrkhew8b X-Rspamd-Queue-Id: 04DAD4001A X-HE-Tag: 1679568671-383664 X-HE-Meta: U2FsdGVkX1+ZcHoxyyoOa2eBtJq0Z3kX7E8tAH+adfGFP2NKE51CT+JrCoQggJ9mmM+NrKKNPGqpTnhtgD+ftwjjOodQANyuy4xYlKK3dOlxiUHkvcf/RvSrl0I6bjDK4pKhHp/PH1nfHQuXUC00mmHhkl1Q6f1u1xRM/8CIN1XMcLo0y3KVzdsy1Qwoqa4dYeb92FNJvjcMTA5ni7Nq6ao0Mf4FACYY/z8AAb2rTXS0rVdrC8mP6TX4ZQdmFOZmscRxfdorf5AJXr3sM+9RA8aFFJEw2uyF+JMJtADvowfKZJDe3t9kjzc8epGzPBSgJyxkPoUGvn2jZDiRyaFr3JXfYk04d+6azM6qlu/03W3NKlX2QhyxRsthBbzqAPi95PKG3BzNO22De2col8f5d/J0iETd841+wYAWbEETcoPC5xIDwYPmTWAoWNeL9L+5NEOSwSSW+W1t9fpN+zhP7/UDtcFDtIWwam8fmgSvy/GrS7/5I2jeg83au0a6+qlpwbuGbCtfh7cp7g6fUKXAAvKPxXfOTSq1UzmDn3B/QZ6WCBgFEZB95Gt6n0dnRns2c8tI8jWOLh1I1JGOLuzt398l0EmsLZpc8q0PK33bIwGV1/TZMBUNXZAQ0h/7VkW7zzDNg4eR6l22tJcceern/tNqNBiyk4i/jMXMYfYZ4arGLx1uisGag64BxsHGh0dDIhQPVtY/ig5Fw2e5PqPYse+I0M0rUL3+7gj4c0I3jM3/BLmSo4sY03weFS/n8nAhZSgFmlYK/L8D0v3T8qU/InviDW6v56p4yhVFBD6TCehjXYNgtoMm0Nc+dz6sP7I5YGdaEiUKHVdBsJhK5WNmlQGLzRDjYR5/i0eCzwv8Fb3vUSMzjStpcyr0PhXpHPG0CjjQWrgH4jJ4YS5NTH/pnQwiqD48nCfe6G/fXxurqtkFqUA4S2zzSMeegtpSt35b/U/tkdICgxCGcpXikgD Ignv/RW/ zCKRMq+z7rCvvjDwzaWuopkmnetDewwf3OWyUXZYS/FauHh3hhIPtRvkuJ1iudhh+pSfrgHpep45cudJwjk1/h2hQzJku8Nhx8ABkD88SLtm82hU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: I appreciate dan for the careful advice. >Kyungsan Kim wrote: >[..] >> >In addition to CXL memory, we may have other kind of memory in the >> >system, for example, HBM (High Bandwidth Memory), memory in FPGA card, >> >memory in GPU card, etc. I guess that we need to consider them >> >together. Do we need to add one zone type for each kind of memory? >> >> We also don't think a new zone is needed for every single memory >> device. Our viewpoint is the sole ZONE_NORMAL becomes not enough to >> manage multiple volatile memory devices due to the increased device >> types. Including CXL DRAM, we think the ZONE_EXMEM can be used to >> represent extended volatile memories that have different HW >> characteristics. > >Some advice for the LSF/MM discussion, the rationale will need to be >more than "we think the ZONE_EXMEM can be used to represent extended >volatile memories that have different HW characteristics". It needs to >be along the lines of "yes, to date Linux has been able to describe DDR >with NUMA effects, PMEM with high write overhead, and HBM with improved >bandwidth not necessarily latency, all without adding a new ZONE, but a >new ZONE is absolutely required now to enable use case FOO, or address >unfixable NUMA problem BAR." Without FOO and BAR to discuss the code >maintainability concern of "fewer degress of freedom in the ZONE >dimension" starts to dominate. One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases. ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time. ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation. Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag. In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped. We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases. As you well know, among heterogeneous DRAM devices, CXL DRAM is the first PCIe basis device, which allows hot-pluggability, different RAS, and extended connectivity. So, we thought it could be a graceful approach adding a new zone and separately manage the new features. Kindly let me know any advice or comment on our thoughts.