From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F05EC6FD1C for ; Fri, 24 Mar 2023 09:50:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2A756B0075; Fri, 24 Mar 2023 05:50:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B8C2E6B0078; Fri, 24 Mar 2023 05:50:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9DF086B007B; Fri, 24 Mar 2023 05:50:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 877526B0075 for ; Fri, 24 Mar 2023 05:50:40 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 56D5B807F1 for ; Fri, 24 Mar 2023 09:50:40 +0000 (UTC) X-FDA: 80603322240.15.ED21E9E Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by imf06.hostedemail.com (Postfix) with ESMTP id 2EC25180007 for ; Fri, 24 Mar 2023 09:50:36 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=nWTzx3uV; dmarc=pass (policy=none) header.from=samsung.com; spf=pass (imf06.hostedemail.com: domain of ks0204.kim@samsung.com designates 203.254.224.33 as permitted sender) smtp.mailfrom=ks0204.kim@samsung.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679651437; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fNG+NfokszMqErj+zrj3B6XcYcvLtdCG61STw+G5JU4=; b=eqRyD1919Hz4mK+a7Xmd01aNQIRp7ZExORjDB4S8cro+6u5J2KVsHLrL8STG5ZsAn2anEI cKql/l7w6qJNE7dJCinhwIW5x7Rqr22dRdrsRsVhNzmcUdXWqJ9Y18KhVgzGqV87OOrX4n oqpqEDqbnGu/IbUoLcfHjg2MVkvzHfU= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=nWTzx3uV; dmarc=pass (policy=none) header.from=samsung.com; spf=pass (imf06.hostedemail.com: domain of ks0204.kim@samsung.com designates 203.254.224.33 as permitted sender) smtp.mailfrom=ks0204.kim@samsung.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679651437; a=rsa-sha256; cv=none; b=PCfIM9y72XB7G499vZ3gkJxUyN/SvVqTXf0JstO6BHqQiACQCxWWL4pAAu6tgFdNnbGbuO upGcElgJ5YPIwz3EpIgXu7vyaB9Q2YafU44Vnxz39Igk8+7q7Igbdid75pgDvfVol1Nr8u riDvpeUCDkJCfnGMT78q0owsUg8AizI= Received: from epcas2p1.samsung.com (unknown [182.195.41.53]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230324095033epoutp03af0e8d6f14d65cf06dafc5af0f048020~PUhR8vjNN1696716967epoutp03g for ; Fri, 24 Mar 2023 09:50:33 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230324095033epoutp03af0e8d6f14d65cf06dafc5af0f048020~PUhR8vjNN1696716967epoutp03g DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1679651434; bh=fNG+NfokszMqErj+zrj3B6XcYcvLtdCG61STw+G5JU4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nWTzx3uVT9cgBxrsiALFsFAyjESDNLcbRhVcYCyHZbOU+AJqU+bfsluSe3nm+G7bZ DJomcqGGP3E8ZKA7b4/pXtc4KMwqW1ast6Udvzt1eQvRAqQoE66ZlJJBkgOHqYK+g5 QierlvqsQ84g4b78uOE5anqD3DWcKea4SKR//oHg= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas2p1.samsung.com (KnoxPortal) with ESMTP id 20230324095032epcas2p1b30f92cd47185a6755a9e4a72d0d7bdb~PUhRCMlby1512315123epcas2p1_; Fri, 24 Mar 2023 09:50:32 +0000 (GMT) Received: from epsmges2p1.samsung.com (unknown [182.195.36.92]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Pjcs42hQcz4x9Pp; Fri, 24 Mar 2023 09:50:32 +0000 (GMT) Received: from epcas2p1.samsung.com ( [182.195.41.53]) by epsmges2p1.samsung.com (Symantec Messaging Gateway) with SMTP id EF.36.61927.8627D146; Fri, 24 Mar 2023 18:50:32 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas2p2.samsung.com (KnoxPortal) with ESMTPA id 20230324095031epcas2p284095ae90b25a47360b5098478dffdaa~PUhQB-x5k3258732587epcas2p2I; Fri, 24 Mar 2023 09:50:31 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230324095031epsmtrp2c7c6e363505c55083162df20215d8800~PUhP9kMnd1597815978epsmtrp2g; Fri, 24 Mar 2023 09:50:31 +0000 (GMT) X-AuditID: b6c32a45-8bdf87000001f1e7-2d-641d72688487 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 87.A0.31821.7627D146; Fri, 24 Mar 2023 18:50:31 +0900 (KST) Received: from dell-Precision-7920-Tower.dsn.sec.samsung.com (unknown [10.229.83.99]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230324095031epsmtip1499ebbb0c866ca8ff809195238195cb1~PUhPwacid0993709937epsmtip1a; Fri, 24 Mar 2023 09:50:31 +0000 (GMT) From: Kyungsan Kim To: david@redhat.com Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-cxl@vger.kernel.org, a.manzanares@samsung.com, viacheslav.dubeyko@bytedance.com, dan.j.williams@intel.com Subject: RE(3): FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL Date: Fri, 24 Mar 2023 18:50:31 +0900 Message-Id: <20230324095031.148164-1-ks0204.kim@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <91d02705-1c3f-5f55-158a-1a68120df2f4@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprFJsWRmVeSWpSXmKPExsWy7bCmqW5GkWyKwYZvQhbTDytaTJ96gdHi 6/pfzBbnZ51isdiz9ySLxb01/1kt9r3ey2zRseENowOHx78Ta9g8Fu95yeSx6dMkdo/JN5Yz erzfd5XNo2/LKkaPz5vkAtijsm0yUhNTUosUUvOS81My89JtlbyD453jTc0MDHUNLS3MlRTy EnNTbZVcfAJ03TJzgI5SUihLzCkFCgUkFhcr6dvZFOWXlqQqZOQXl9gqpRak5BSYF+gVJ+YW l+al6+WlllgZGhgYmQIVJmRnrLv3halggkLFt+/fWRoY10h1MXJySAiYSLx9+Iuti5GLQ0hg B6PEvy/dTBDOJ0aJ+VcbWCGcz4wSyzY+YIRp6br7ghkisYtRYl/PeRYIp4tJ4taSD0wgVWwC 2hJ/rpxnA7FFBEQkfjx8yQhSxCxwnlFi24vf7CAJYQF3idmnHjGD2CwCqhI7H/8Ai/MK2Eh8 PnKIBWKdvMTMS9/B4pwCdhKdL26zQdQISpyc+QSshhmopnnrbLCTJAQaOSSaXsxmhWh2kTiz rpkJwhaWeHV8CzuELSXx+d1eNgi7WOLx639Q8RKJw0t+Qy02lnh38znQHA6gBZoS63fpg5gS AsoSR25BreWT6Dj8lx0izCvR0SYE0agisf3fcmaYRaf3b4Iq8ZDY9D4MElQTGSVOf7zLOIFR YRaSZ2YheWYWwt4FjMyrGMVSC4pz01OLjQoM4TGcnJ+7iRGcSLVcdzBOfvtB7xAjEwfjIUYJ DmYlEd53IbIpQrwpiZVVqUX58UWlOanFhxhNgUE9kVlKNDkfmMrzSuINTSwNTMzMDM2NTA3M lcR5pW1PJgsJpCeWpGanphakFsH0MXFwSjUwnXE/fkrkdsq5yrT76WWyn4RnxPR8PfDFv8JU oXVtcmL8qZ4pIgWS+37wWGiGzKh9pmfbV9v4Ydd3017tEE/P9s0PvU/esZ/mcmO2d50oq+J8 hbrcbw9EZ7FU19VtnNFeveUR34PLarfdFyzI9EphXVfsXTXL4lOANOfkD44TLB/or9J63hTd LV88u0yDgWGqm7XB7zyjywts5nr2cW9+tH7y1vf3tgVe/sWkuevlg8YXCQu/zVxaF/110vW0 I8x2DFNEf7vMEtnl5e52vv940vl0zv171m6VevvswBqJfa9iS6Y7qRemWR3ds/dKcujWR9FF C0KuNK8Umm+gVnbn8sbVrusNGc8oyN7l/REXrsRSnJFoqMVcVJwIAGN5xrgtBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrKLMWRmVeSWpSXmKPExsWy7bCSnG56kWyKwb/tOhbTDytaTJ96gdHi 6/pfzBbnZ51isdiz9ySLxb01/1kt9r3ey2zRseENowOHx78Ta9g8Fu95yeSx6dMkdo/JN5Yz erzfd5XNo2/LKkaPz5vkAtijuGxSUnMyy1KL9O0SuDLW3fvCVDBBoeLb9+8sDYxrpLoYOTkk BEwkuu6+YO5i5OIQEtjBKHFmwU12iISUxPvTbVC2sMT9liOsEEUdTBKnHl5hBEmwCWhL/Lly ng3EFhEQkfjx8CUjSBGzwFVGiY8vVjKDJIQF3CVmn3oEZrMIqErsfPwDbCqvgI3E5yOHWCA2 yEvMvPQdLM4pYCfR+eI20FAOoG22EjuP50KUC0qcnPkErJwZqLx562zmCYwCs5CkZiFJLWBk WsUomVpQnJueW2xYYJSXWq5XnJhbXJqXrpecn7uJERzyWlo7GPes+qB3iJGJg/EQowQHs5II 77sQ2RQh3pTEyqrUovz4otKc1OJDjNIcLErivBe6TsYLCaQnlqRmp6YWpBbBZJk4OKUamBJn lnhMMu+4lvO2Loll4cRHyyVSkzinrF/brK9utcBx5xxGbQfD9ebnNW7N6Fn7s0W0PP/QiQau 1gqFzktMPk0GV5Mn8Hvejt+0yf3blOtb/hZ47b1h6z1/VttWu3eMMfNb8rdbdRaxzWnbtab5 dutCywNpN0+Glv75dkOd6ceMCatY5Bx+zYzY76TBumnhg2rGrLTb128YXcp6cztCV9w6JdVW pGye5bzf375techbFr6Xozj/qMfis9LpmxsEF/FPCb65XuDDCbn1BxQErcPf1B3mv3nsnrPo N7Oo1wJHEp/mv56UL9jddzr8/smXbsJztZbKnvTg0Fx5+U/mE/01TNOfJZQqZpTYBGusn6bE UpyRaKjFXFScCAC7bWl76AIAAA== X-CMS-MailID: 20230324095031epcas2p284095ae90b25a47360b5098478dffdaa X-Msg-Generator: CA Content-Type: text/plain; charset="utf-8" X-Sendblock-Type: AUTO_CONFIDENTIAL CMS-TYPE: 102P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230324095031epcas2p284095ae90b25a47360b5098478dffdaa References: <91d02705-1c3f-5f55-158a-1a68120df2f4@redhat.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 2EC25180007 X-Stat-Signature: aus7a61k6wynfk345wa8q88z7eybzc6a X-HE-Tag: 1679651436-49528 X-HE-Meta: U2FsdGVkX18EKMbed2sDE5y9imNILg3Vm6qjH5WaqzvYwiKk6Rb2Zv7rorqYdwnRW78jTzydxR8NZcjrbySAGNGc1JXDe065ZHLOBL/tEkPRkq7j1Ux1JJAFBbIW1gmMtXVGuvNRFZz9uTzgs4gPWxo8t9ih6GVNHKsnS0C2ZKUY6ZhswEK6+BqMBmRAvzvFt45IuUaZWJ1Jcfiz7rq1FDgl5/N5JLD6cN1pfHcCSPO5hJx2yBbKHyXQH1eP4T6LVKKNwGp/ElUSAihH4n3JYVnIhmpNmjo/6DOz5gdoxKs8ezoMMjnyCSk56qAB8TxNowEKMiqPM90FcVpOu0D6sS9vrn0aMZN6ME0qXpJPbnB2dPjz18kRa5MIGs7N6n5Gg/6G1JWsLQMD1Zsx7eri4mMPLAgGeW63XIpCjLmYlsR7quywrLOyLNWDlAv9uonckr33xg4jtMBVqqMLS+b0r4xQ5bpwgznB9NGSWT/vYdLa+3A7UKt1aC/1fD3mqb3LM0AO5pYuBYSs/silzcgWy2w9PcWsHMI6LtfZw7KXkzIFFKAQy8vHMRZoLLT7tgo7/m12Px4kmVopwkFp2jD7/5+TVxLxnnfYZ5+7S94xehwSB8P6kM9ZgoW7mX8RA+DFOKnMbvBtYSIJWt5V1Zjaj8V5hFPdO2kZ2AukQatDYGrH9oYtVeEH0jEXDH7ZuvffEM3/syXWWocD9xIIM1MsCwsDldh0Wav1kIbJJZtt9+ggiO7wF7byOOWwMEbHGbe8VpNIgAcxBkZcH45KJwdRUQ0mejyfv7gmCqU49ORYrUi3zq9WMRMoCerZLRs7r6DU064MKJxDiWlNb80O957EbmloT2iWKzr5kGytmoRBrQIZEjC2qUd/v31QSjtNxbVNxRE+Q4p31wTU7MkMEPFx8RNrZMHM2o9V6L2EF3ykjIDssJ+U2Z4EW333ylHATa2EUFsvVdesT/OC/UhYTpT 2VVGahVg QXBij8lz2tMivjJyWT/9RJy5LBQCN+GeT+/PNoJ24Z3JOD0Qcp0rBYwoCxMv6ut9WXA7zLjcmtqyaN3MCwU7Z+9KcFgrK/fz2Y4MG3gLegnb6PE0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: >On 24.03.23 10:27, Kyungsan Kim wrote: >>> On 24.03.23 10:09, Kyungsan Kim wrote: >>>> Thank you David Hinderbrand for your interest on this topic. >>>> >>>>>> >>>>>>> Kyungsan Kim wrote: >>>>>>> [..] >>>>>>>>> In addition to CXL memory, we may have other kind of memory in the >>>>>>>>> system, for example, HBM (High Bandwidth Memory), memory in FPGA card, >>>>>>>>> memory in GPU card, etc. I guess that we need to consider them >>>>>>>>> together. Do we need to add one zone type for each kind of memory? >>>>>>>> >>>>>>>> We also don't think a new zone is needed for every single memory >>>>>>>> device. Our viewpoint is the sole ZONE_NORMAL becomes not enough to >>>>>>>> manage multiple volatile memory devices due to the increased device >>>>>>>> types. Including CXL DRAM, we think the ZONE_EXMEM can be used to >>>>>>>> represent extended volatile memories that have different HW >>>>>>>> characteristics. >>>>>>> >>>>>>> Some advice for the LSF/MM discussion, the rationale will need to be >>>>>>> more than "we think the ZONE_EXMEM can be used to represent extended >>>>>>> volatile memories that have different HW characteristics". It needs to >>>>>>> be along the lines of "yes, to date Linux has been able to describe DDR >>>>>>> with NUMA effects, PMEM with high write overhead, and HBM with improved >>>>>>> bandwidth not necessarily latency, all without adding a new ZONE, but a >>>>>>> new ZONE is absolutely required now to enable use case FOO, or address >>>>>>> unfixable NUMA problem BAR." Without FOO and BAR to discuss the code >>>>>>> maintainability concern of "fewer degress of freedom in the ZONE >>>>>>> dimension" starts to dominate. >>>>>> >>>>>> One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases. >>>>>> ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time. >>>>>> ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation. >>>>>> Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag. >>>> >>>>> That sounds like a bad hack :) . >>>> I consent you. >>>> >>>>>> In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped. >>>>>> We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases. >>>> >>>>> I once raised the idea of a ZONE_PREFER_MOVABLE [1], maybe that's >>>>> similar to what you have in mind here. In general, adding new zones is >>>>> frowned upon. >>>> >>>> Actually, we have already studied your idea and thought it is similar with us in 2 aspects. >>>> 1. ZONE_PREFER_MOVABLE allows a kernelspace allocation using a new zone >>>> 2. ZONE_PREFER_MOVABLE helps less fragmentation by splitting zones, and ordering allocation requests from the zones. >>>> >>>> We think ZONE_EXMEM also helps less fragmentation. >>>> Because it is a separated zone and handles a page allocation as movable by default. >>> >>> So how is it different that it would justify a different (more confusing >>> IMHO) name? :) Of course, names don't matter that much, but I'd be >>> interested in which other aspect that zone would be "special". >> >> FYI for the first time I named it as ZONE_CXLMEM, but we thought it would be needed to cover other extended memory types as well. >> So I changed it as ZONE_EXMEM. >> We also would like to point out a "special" zone aspeact, which is different from ZONE_NORMAL for tranditional DDR DRAM. >> Of course, a symbol naming is important more or less to represent it very nicely, though. >> Do you prefer ZONE_SPECIAL? :) > >I called it ZONE_PREFER_MOVABLE. If you studied that approach there must >be a good reason to name it differently? > The intention of ZONE_EXMEM is a separated logical management dimension originated from the HW diffrences of extended memory devices. Althought the ZONE_EXMEM considers the movable and frementation aspect, it is not all what ZONE_EXMEM considers. So it is named as it.