From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE4A3C77B62 for ; Fri, 31 Mar 2023 11:34:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0EEB46B0075; Fri, 31 Mar 2023 07:34:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 077E46B007D; Fri, 31 Mar 2023 07:34:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0BBC6B007E; Fri, 31 Mar 2023 07:34:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CACC26B0075 for ; Fri, 31 Mar 2023 07:34:26 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7DB4580398 for ; Fri, 31 Mar 2023 11:34:26 +0000 (UTC) X-FDA: 80628985332.18.39EEDFA Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by imf05.hostedemail.com (Postfix) with ESMTP id 517C1100010 for ; Fri, 31 Mar 2023 11:34:22 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=Zqov9iCy; spf=pass (imf05.hostedemail.com: domain of ks0204.kim@samsung.com designates 203.254.224.24 as permitted sender) smtp.mailfrom=ks0204.kim@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680262463; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pTAoPsTYMf5uRNFA7Dctv3mKoae3M7AjuQn6oIpo9Lw=; b=JK8jxsCiCdbYCYAQ6cZNDdYeBJfmUphyc3dl/kxv/w0jLEkF6R2C9XPXefmt6Qdm7DYF9q EhIoks9/Vh7RgbXUsZPD/NT/kGBjHuQXmUdqIN8gH7uAIlyrl6I8ddF3XMbAKZyvDMZZxI eqHf1w+LLpPmnhFflp2Bwpu1ybcJQnQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=Zqov9iCy; spf=pass (imf05.hostedemail.com: domain of ks0204.kim@samsung.com designates 203.254.224.24 as permitted sender) smtp.mailfrom=ks0204.kim@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680262463; a=rsa-sha256; cv=none; b=QbvQJzwfKWSjT6CwdBaxUleWjjahaHiZWJBTBSOeehQIFsQi9Hlxgng+D/HAE6F2NSNV76 ZPshHBnw3cIhigBMTv9Q/2UnLzib1QwChl8sf8NQqzAiZLzQEejFONSrb1O86sq7kMSWEY QOqMSJ8QJr4htkwfNxQ8yvMG9J48gUw= Received: from epcas2p2.samsung.com (unknown [182.195.41.54]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230331113420epoutp014ffa77898ba56cd1d754bd137b175555~Rfc4bVuJr0825708257epoutp015 for ; Fri, 31 Mar 2023 11:34:20 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230331113420epoutp014ffa77898ba56cd1d754bd137b175555~Rfc4bVuJr0825708257epoutp015 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680262460; bh=pTAoPsTYMf5uRNFA7Dctv3mKoae3M7AjuQn6oIpo9Lw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zqov9iCynZ0guXnPUOb005JNCXZ4MVVt6ZJQAb+J7uO0d181uXJpXKFo6i8b+MBhG FNTpPPM6ufr0qV2kfMQd9wpOmN4CR5XX5yx21hqNOEfFLmDs1T7xTo4NRBH7sF/Ryd PBrdNZyRC+LFTjf9+xBFX999hTxtolQSDcwz/WTI= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas2p4.samsung.com (KnoxPortal) with ESMTP id 20230331113419epcas2p42081172994e81f630bae550425c951ae~Rfc3YUC2j1265512655epcas2p4P; Fri, 31 Mar 2023 11:34:19 +0000 (GMT) Received: from epsmges2p1.samsung.com (unknown [182.195.36.89]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4PnyqZ2vQ1z4x9Pw; Fri, 31 Mar 2023 11:34:18 +0000 (GMT) Received: from epcas2p1.samsung.com ( [182.195.41.53]) by epsmges2p1.samsung.com (Symantec Messaging Gateway) with SMTP id 24.BB.61927.A35C6246; Fri, 31 Mar 2023 20:34:18 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas2p2.samsung.com (KnoxPortal) with ESMTPA id 20230331113417epcas2p20a886e1712dbdb1f8eec03a2ac0a47e2~Rfc2RA8Oa2604426044epcas2p2J; Fri, 31 Mar 2023 11:34:17 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230331113417epsmtrp1a363905f732f20f4e2e51d34acbe777d~Rfc2QQr-R0759407594epsmtrp1O; Fri, 31 Mar 2023 11:34:17 +0000 (GMT) X-AuditID: b6c32a45-8bdf87000001f1e7-f0-6426c53a8b87 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id B5.D1.31821.935C6246; Fri, 31 Mar 2023 20:34:17 +0900 (KST) Received: from dell-Precision-7920-Tower.dsn.sec.samsung.com (unknown [10.229.83.99]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230331113417epsmtip212672f647a7582cd7394d08215b4ca55~Rfc2Gu3GE2862228622epsmtip2C; Fri, 31 Mar 2023 11:34:17 +0000 (GMT) From: Kyungsan Kim To: gregory.price@memverge.com Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-cxl@vger.kernel.org, a.manzanares@samsung.com, viacheslav.dubeyko@bytedance.com, dan.j.williams@intel.com, seungjun.ha@samsung.com, wj28.lee@samsung.com Subject: RE: RE: RE(4): FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL Date: Fri, 31 Mar 2023 20:34:17 +0900 Message-Id: <20230331113417.400072-1-ks0204.kim@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrMJsWRmVeSWpSXmKPExsWy7bCmqa7VUbUUg8NdIhbTDytaTJ96gdGi oekRi8X5WadYLPbsPclicW/Nf1aLfa/3Mlu86DzOZNGx4Q2jxcb779gcuDz+nVjD5rF4z0sm j02fJrF7TL6xnNFj48f/7B59W1YxenzeJBfAHpVtk5GamJJapJCal5yfkpmXbqvkHRzvHG9q ZmCoa2hpYa6kkJeYm2qr5OIToOuWmQN0npJCWWJOKVAoILG4WEnfzqYov7QkVSEjv7jEVim1 ICWnwLxArzgxt7g0L10vL7XEytDAwMgUqDAhO+P951fMBdelKp5++M7ewHhdtIuRk0NCwERi 87bPbF2MXBxCAjsYJU50H2KEcD4xSsxf28oM4XxmlJix/TIrTMvTU/+YIBK7GCV2/VgD1dLF JHFw9XoWkCo2AW2JP1fOs4HYIgJyEhf2XQOzmQX+MUrsuSwJYgsLBEksefoHrJ5FQFXi0Jx+ RhCbV8BGYu6re0wQ2+QlZl76zg5icwroSNxcPIUZokZQ4uTMJywQM+UlmrfOBjtVQqCVQ+LE vLlsEM0uEr0dO5ghbGGJV8e3sEPYUhIv+9ug7GKJx6//QdklEoeX/GaBsI0l3t18DvQyB9AC TYn1u/RBTAkBZYkjt6DW8kl0HP7LDhHmlehoE4JoVJHY/m85M8yi0/s3QQ33kDi89gU4DIUE 6iUmnvjFNoFRYRaSZ2YheWYWwt4FjMyrGMVSC4pz01OLjQoM4RGcnJ+7iRGcWrVcdzBOfvtB 7xAjEwfjIUYJDmYlEd5CY9UUId6UxMqq1KL8+KLSnNTiQ4ymwKCeyCwlmpwPTO55JfGGJpYG JmZmhuZGpgbmSuK80rYnk4UE0hNLUrNTUwtSi2D6mDg4pRqYFlpt7UnPLlLs2T53+XmOq40N UZxa5Rk9+onFTwTXe884/mp6cPra6K6TIu3c7u/Wlf9el/Vmoepy79XXur5OSbr2w2+vRcs2 ddlja0Mnbpy/WG4fT/dUl8X/j/3fNePAkoml0XuOZ0/OW7u3dZPn+zcX5GXz3K1ET5YuWCnK 0qL5TiPpzybJjvPLV3TdDW5gkE123Rr/99/z+t6j4n8uvq37EDp7sZ+wzj7jaRwbOZ/eXPxd zzvZNonjcN/Ffyfar3lmTPm0p1GnL8L0zpn/LP94mW6KbFLbncTfNm/bsc9lyx+4HJu8xq/H 0Ne7znuFafENadv5dcKTS6wLFkXoZKstXcv7onbDOcGPZw6ePKfEUpyRaKjFXFScCABt1HeD NgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrHLMWRmVeSWpSXmKPExsWy7bCSvK7lUbUUg0Mv9CymH1a0mD71AqNF Q9MjFovzs06xWOzZe5LF4t6a/6wW+17vZbZ40XmcyaJjwxtGi43337E5cHn8O7GGzWPxnpdM Hps+TWL3mHxjOaPHxo//2T36tqxi9Pi8SS6APYrLJiU1J7MstUjfLoEr4/3nV8wF16Uqnn74 zt7AeF20i5GTQ0LAROLpqX9MILaQwA5GiV0f0yHiUhLvT7exQ9jCEvdbjrBC1HQwSXz+Jgli swloS/y5cp4NxBYRkJO4sO8akM3FwQxSs/XydLCEsECAxLaDL8CaWQRUJQ7N6WcEsXkFbCTm vrrHBLFAXmLmpe9gyzgFdCRuLp7CDLFMW+Ll90XMEPWCEidnPmEBsZmB6pu3zmaewCgwC0lq FpLUAkamVYySqQXFuem5xYYFRnmp5XrFibnFpXnpesn5uZsYwVGgpbWDcc+qD3qHGJk4GA8x SnAwK4nwFhqrpgjxpiRWVqUW5ccXleakFh9ilOZgURLnvdB1Ml5IID2xJDU7NbUgtQgmy8TB KdXAVOp34c/puTJ1Xe06vmEyshOV2hiWvVq/s/Nrrk76VXGrg5pXUrdYzrxaN1exPCuYoVOO 4+qUU3rnalksrgXfE2hKjjP4IXxCR2mdlluA5aoY2aMX3VYH2V4x+Wvkd33NIokZtlOucpy7 z78n5vGimM6mxqi/j++xzdDO9rm5pHXu/eWnpktufB8/T+88y3xB3j0SrxP5jrskqh3U3PxK 4NnFQ68dlGRjbKJ/XgjKFNp/Nude5Z7gijMKK7xSriywE+62Xd1ieotb4nHP9xT5yJ7y9cuP 6UlZ/jFqMT8RsPX/HJ+ZLd3SLHu0l5XLsk9/+PtL/a6w2dEumTe+alw6FBx9qoCL/7dUYUTV +lnLlFiKMxINtZiLihMBNLZBL/ECAAA= X-CMS-MailID: 20230331113417epcas2p20a886e1712dbdb1f8eec03a2ac0a47e2 X-Msg-Generator: CA Content-Type: text/plain; charset="utf-8" X-Sendblock-Type: AUTO_CONFIDENTIAL CMS-TYPE: 102P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230331113417epcas2p20a886e1712dbdb1f8eec03a2ac0a47e2 References: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 517C1100010 X-Stat-Signature: 4zgsotutnupjyu9k1k7ims4474embq6p X-Rspam-User: X-HE-Tag: 1680262462-952113 X-HE-Meta: U2FsdGVkX18Z8W7laY5VdbCxk0g20JTnK9bfwaRVTcSyx2eQNP3gNHcFgHaCYtztQ5hgknYA8C2vQRdIw+1D9fziPos5quqB5RwAlvYjMjH/6qjO/Q58K4Hxw37YpTuBHAdILXLCVqdVGOfaTTk7OuEJIKFuWc+b1cJpFAiS4P/7ukYMmUQszpWMcVxz892vX+9Rc6iIYgzzA0FPsUFH6zhMrmQgo//5H2BnOPQw9MKKatk5ao2Es8OdrAq6umVlasmORZ0CSEwbg3/rRViecL/6vVRwBT/KVwh6BahgK63TVXRqiifACvTW+niptFpjAUk45tVnP84FpeyfNZKMBUOysHXVe/AIEO+XnA1qSYszgbYeNRcUR3V9FszBRjKMIWTbO62WH/XBklceds8O9cBKu5VKoJ83nXxAIs/swYT50rk1vK8o1Ej5fumUPi14mS+gFY3ZI2FhG/yIS81w0m5z6mCYdtdZmsBhwrPOCzXpZcjeFy1YZnsyz0cEEan/cQH9itdWLGwXp7VMRhKL7BzbHaj+yIop9596imm5lduOEvk1nA3YzXDQMX5qKdsy3Mw7CVHU5ynHKmsRBXGzK2GxFUFx+eFvh56LykRrL+RoFB6XCPXnoFKCtawzhI5mQX5dYtpCBoAcECkedmyPGMIAaU3TPfDPzoVQvhJwCEwTJQRhynfHWp4peKTSeOJcwm9yhZFNbGQleiLk5zhsGdBiZEYs33mkQ1Xw2C9WAvF9K7DV6gL+9DbVLKnacxeTPKXXXcTwb1i9+YuMIskByYkagFhFClwJExsPSHx5zgV6NLz+hNTAHYYCsm6dASFO05B8XwDLwJExeBzTmNtLaC+AroXcvEVljJwUrN5ty6wXnA8Z4xBU+WBgsH8t9Kpxh60l2Vo3Bh8ykE+Y3gTkCQTycJ33D7upoL4lI8GYotxU8V+Iy8l9pXvelIuZLvb4QnJUVE02+Tc/KkdAOBz a7s2eG0n jYBm96cGiMSJ0sf85tOVavarp+HxNPDz/Fs3pjscIwpA4dRLdO9+xcXLVkwna4xO8JI7Yfev1wq06ms3uXeTbyKHPpdopCGQJMSNtIsXy6lnUUPd86tlCHnCKKotw3i0opZ1q1y5AGG2uClv74ZqImuCQSg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Gregory Price. Thank you for joining this topic and share your viewpoint. I'm sorry for late reply due to some major tasks of our team this week. >On Fri, Mar 24, 2023 at 05:48:08PM +0900, Kyungsan Kim wrote: >> >> Indeed, we tried the approach. It was able to allocate a kernel context from ZONE_MOVABLE using GFP_MOVABLE. >> However, we think it would be a bad practice for the 2 reasons. >> 1. It causes oops and system hang occasionally due to kernel page migration while swap or compaction. >> 2. Literally, the design intention of ZONE_MOVABLE is to a page movable. So, we thought allocating a kernel context from the zone hurts the intention. >> >> Allocating a kernel context out of ZONE_EXMEM is unmovable. >> a kernel context - alloc_pages(GFP_EXMEM,) > >What is the specific use case of this? If the answer is flexibility in >low-memory situations, why wouldn't the kernel simply change to free up >ZONE_NORMAL (swapping user memory, migrating user memory, etc) and >allocate as needed? > >I could see allocating kernel memory from local memory expanders >(directly attached to local CXL port), but I can't think of a case where >it would be preferable for kernel resources to live on remote memory. We have thought kernelspace memory tiering cases. What memory tiering we assumes is to locate a hot data in fast memory and a cold data in slow memory. We think zswap, pagecache, and Meta TPP(page promotion/demotion among nodes) as the kernelspace memory tiering cases. >Since local memory expanders are static devices, there shouldn't be a >great need for hotplug, which means the memory could be mapped >ZONE_NORMAL without issue. > IMHO, we think hot-add/remove is one of the key feature of CXL due to the composability aspect. Right now, CXL device and system connection is limited. But industry is preparing a CXL capable system that allows more than 10 CXL channels at backplane, pluggable with EDSFF. Not only that, along with the progress of CXL topology - from direct-attached to switch, multi-level switch, and fabric connection - I think the hot-add/remove usecase would become more important. >> Allocating a user context out of ZONE_EXMEM is movable. >> a user context - mmap(,,MAP_EXMEM,) - syscall - alloc_pages(GFP_EXMEM | GFP_MOVABLE,) >> This is how ZONE_EXMEM supports the two cases. >> > >Is it intended for a user to explicitly request MAP_EXMEM for it to get >used at all? As in, if i simply mmap() without MAP_EXMEM, will it >remain unutilized? Our intention is to allow below 3 cases 1. Explicit DDR allocation - mmap(,,MAP_NORMAL,) : allocation from ZONE_NORMAL or ZONE_MOVABLE, or allocation fails. 2. Explicit CXL allocation - mmap(,,MAP_EXMEM,) : allocation from ZONE_EXMEM, of allocation fails. 3. Implicit Memory allocation - mmap(,,,) : allocation from ZONE_NORMAL, ZONE_MOVABLE, or ZONE_EXMEM. In other words, no matter where DDR or CXL DRAM. Among that, 3 is similar with vanilla kernel operation in that the allocation request traverses among multiple zones or nodes. We think it would be good or bad for the mmap caller point of view. It is good because memory is allocated, while it could be bad because the caller does not have idea of allocated memory type. The later would hurt QoS metrics or userspace memory tiering operation, which expects near/far memory. > >~Gregory