From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D01D3C6FD1D for ; Thu, 30 Mar 2023 22:03:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 612DE6B0075; Thu, 30 Mar 2023 18:03:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 59B8F6B0078; Thu, 30 Mar 2023 18:03:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41564900002; Thu, 30 Mar 2023 18:03:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2A8276B0075 for ; Thu, 30 Mar 2023 18:03:33 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 092811C5F01 for ; Thu, 30 Mar 2023 22:03:33 +0000 (UTC) X-FDA: 80626941906.30.14F52E1 Received: from bird.elm.relay.mailchannels.net (bird.elm.relay.mailchannels.net [23.83.212.17]) by imf17.hostedemail.com (Postfix) with ESMTP id 6844140012 for ; Thu, 30 Mar 2023 22:03:30 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=stancevic.com header.s=dreamhost header.b=hk403yWl; spf=pass (imf17.hostedemail.com: domain of dragan@stancevic.com designates 23.83.212.17 as permitted sender) smtp.mailfrom=dragan@stancevic.com; dmarc=none; arc=pass ("mailchannels.net:s=arc-2022:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1680213810; a=rsa-sha256; cv=pass; b=OmJ2MZrA11ovZzbqrSBgA/Leo4MDqv25KmGSTYpqckN241kifIM9RIAVHD/jO/cXGUMmNA vqVstI5rh1o0pSGTGtzi/i/x+2iMtVOV4CQr39vdo+JsZeJGHRikpsce1adsYc6aHwvcCX m92hTVdqAsSBRNzbSupAXzOaIbHHu1Q= ARC-Authentication-Results: i=2; imf17.hostedemail.com; dkim=pass header.d=stancevic.com header.s=dreamhost header.b=hk403yWl; spf=pass (imf17.hostedemail.com: domain of dragan@stancevic.com designates 23.83.212.17 as permitted sender) smtp.mailfrom=dragan@stancevic.com; dmarc=none; arc=pass ("mailchannels.net:s=arc-2022:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680213810; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yPCzU6F6owMOhqlBeNEFKP1Ymsp3uChE2AkQ2JDym7k=; b=bYZCBNXHmLHgZIwS7ldhGQWPmNPriFPyJTAt53PUFUQFNnuARyyuSSqDBYh9H3RNTaPE/N YxuYv1ZgeUK+cH70iMyt05bwZXQjuW7tWETKiKaNfhzMlJVklitzKkCnTVfmmqo4h9oDzP K1Ca/yJNz6N14iKufLm5uKO2I+hBJ4g= X-Sender-Id: dreamhost|x-authsender|dragan@stancevic.com Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id E28485C10EC; Thu, 30 Mar 2023 22:03:28 +0000 (UTC) Received: from pdx1-sub0-mail-a294.dreamhost.com (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 1DB9A5C0FBE; Thu, 30 Mar 2023 22:03:28 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1680213808; a=rsa-sha256; cv=none; b=XMaUq83XK1kmiBrVQCBLGELpfGgEqlBZ4POvnXd6Ic9+Js2rKrQc3JQHOgxDHzMMHdHZjX IXkjGVQaJmfdt7UFBG8WX29gnsttJ7DNDYXln+SwfKnnOdnqQcC8gxMEQ4SOHANZq7r47O nwZRfd1yzSIZ7crc/iOAg3oTcYfGePJTK5jYw/cmk6QCUHFfrkTn5aBy8BS2bzP34COw6x dZJ/o6ocv79eaDc/C/sd9MjPdKaqBp5ShwvUeT8XoUWKSf3YKRhdWkN4XrKjuD3Y4juYmp zsMdTvjxEM50ZgfD2OZ/9zrayPxlryo7esfT/43qDYioBchn9CkV0XxnYsoq1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1680213808; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yPCzU6F6owMOhqlBeNEFKP1Ymsp3uChE2AkQ2JDym7k=; b=NWs1za+wNjVq6VIWH/eoSppAKw8d4ZBSK6dEk/aghERKTYPh9ypBV9q+JYMxZxlLRcfqCR pqqfDDB6+G1pNcGuFzN6Q8qArtQdgUlbXOidp67tDHAPAvT19OIO5NxCnGFCqzIOYpkVuc 6+CKAbMGTgNxX3DwhfVk7RhgdSttTjRl+eRS9Szq/VaYntF4iDSiJlmbjAkpTkyF5Q5N7g dxx1uKXLK8xskkjdpaypCGZRreoJTHrbyEFCylc9/KtYjrZ/K1ognh45NCyOgSC2j+foLq UNXaR+OtH7yC/vfk7eVD9Q2iF91EZBjQxSI47BxFVxI0W2IAOm+cMUCkRcRJkg== ARC-Authentication-Results: i=1; rspamd-786cb55f77-rg9g4; auth=pass smtp.auth=dreamhost smtp.mailfrom=dragan@stancevic.com X-Sender-Id: dreamhost|x-authsender|dragan@stancevic.com X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dragan@stancevic.com X-MailChannels-Auth-Id: dreamhost X-Print-Spicy: 279d1eea155a1243_1680213808495_2461694109 X-MC-Loop-Signature: 1680213808495:2329669094 X-MC-Ingress-Time: 1680213808495 Received: from pdx1-sub0-mail-a294.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.101.8.122 (trex/6.7.2); Thu, 30 Mar 2023 22:03:28 +0000 Received: from [192.168.1.31] (99-160-136-52.lightspeed.nsvltn.sbcglobal.net [99.160.136.52]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) (Authenticated sender: dragan@stancevic.com) by pdx1-sub0-mail-a294.dreamhost.com (Postfix) with ESMTPSA id 4Pncqx3tpMzJJ; Thu, 30 Mar 2023 15:03:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stancevic.com; s=dreamhost; t=1680213807; bh=yPCzU6F6owMOhqlBeNEFKP1Ymsp3uChE2AkQ2JDym7k=; h=Date:Subject:To:Cc:From:Content-Type:Content-Transfer-Encoding; b=hk403yWltsydDPdv3hVya3gI5wErNS+3eFchygO3ZCDDhUtXuy64l4J4TN6pAIzzp 7JjTOwoeSQ3o+vMrVSWUD5d2vfd0KpHqIOgqh5RlitqZN9ZBq0qtguqIpWoDSaRBQw w6dbIsv4NUDDyoY5bU1TwJ0Jaq1hZ/D7JuS35V+EC51zKC2cZFC43K6bTP0bZyLaXz 3ybyVSQ8y1xfKIEgBlFGf1Jiv+MboNE1jDrhF2T008wRHZSmsVfRdapWu7XQVW/yaG CYepfy9XIhzyS+RtjxIKyBHyhPNYmNx1SsBJ8eilgHjsRZrXHu00KoarGx135BkbNq miT3qQaFr40Ow== Message-ID: <362a9e19-fea5-e45a-3c22-3aa47e851aea@stancevic.com> Date: Thu, 30 Mar 2023 17:03:24 -0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Subject: Re: FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL Content-Language: en-US To: Mike Rapoport , Kyungsan Kim Cc: dan.j.williams@intel.com, lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-cxl@vger.kernel.org, a.manzanares@samsung.com, viacheslav.dubeyko@bytedance.com, ying.huang@intel.com, nil-migration@lists.linux.dev References: <641b7b2117d02_1b98bb294cb@dwillia2-xfh.jf.intel.com.notmuch> <20230323105105.145783-1-ks0204.kim@samsung.com> From: Dragan Stancevic In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Queue-Id: 6844140012 X-Rspamd-Server: rspam01 X-Stat-Signature: hjncxajm1tmtbs4ose1nr9h36oeiu798 X-HE-Tag: 1680213810-573108 X-HE-Meta: U2FsdGVkX1+7Nx+6lLfMwtblQSDv+DoneUyq6qAwkWogu/OvOGyfuwNK7MsYTySyAfdcda2VFTa+Qp4m/tNziXh3ZibYnIxrzneCBfENbeJVxJYkizChtQUjqBT74rJuqpmdUKAnJvntWcOyZnKXCaavVYcdv4TZbrZ88+itgbPHE3ig7yqIdqhW7Sdem/AdzAFmnV42tY14PcZldK4SuII3gEce3xucHJajEaUS9dWMW7FNyfMUBXVrH4BC20hcY9eYADvdu1hNZdhPqlWFmG6RFXq2fFa6eI64rP62sL/Emm/ul/FP1LG/B74uKjpOSiUXDCSM2r4ISXmdHqIZddUa4Xd/0rs7PE14tIjM4bVPyvgIHIbBKfp1EB3Z3pVQf/JBGSfHIUim1TDQc+QxuRtbywK/kjGFjC59daGvJkr7IQf2oWgdYWZfVCo5ANuEOMXGc4VZsa7gdAHFKO0kSr0ZiVYgkTwI8THtk63HRxtlyPoGED980devazbzccVezw0XNde6PWRN7uIo+y3s1cxM5tvo6PtIH3ZVZZuoYq4101GJ0Kxk+xFAGukcS2JJcwCsbBzCM7suiBa1XFmRpu/RmuceqhfCHN65B8FgyYBVf8gcXiMoCNX6QSdVquAV7X813kvSt51xUvySlEEowPhX9dQM67NsryzkKbdY+jWQnqPT62KJHhogYzUTMblXhyFOvq2JPm1v3AckfNV8m8ysjOmRbbdqm8VNrZXFGysNzD/ctTyJzgdYh1LN1qf2zcENUNoZmgLO11fUrKhpfWwHsyP4EwgCaPPtKK4uwY1wTga9f4AO5oGTcFVX+gU1yGloWBAvZgzKppBotivk2G6A/EVlcVQLslTID3de0t59x/Dfgh5D5Ui5QizFh5dqjX+wJAm9RjA4czMYy8+YxJhCnJe+IXmUNf8lm88SO3PksHzFYenK55h+XsiCgfmZwe34jAK7lHJ7xrPH8K/ zcpaGSC8 bwa3ut+EWzrfFCYXZF8ZnIzuM7lWEA7tOtZKSI92Qsd2chXsNeV0w/SEdy1bIrr17J2u4B4DMsYbkXykE0xMf9Pg3fHWI8UCRVFh0P+PRwCV2ICfnbfCEXKW9FAfgtSoSnrauq3Ku536HjDPWvH5na2su4zNdZJcj7dkJ/cvq53191uDTqr2LMeG3BEl2pgMXD24lmKEtyIUTy3uWQ1NJeKedLHAAjOJxX2vTTXYwjE1gQGbyZE5qWjhDfHYYi8hpDMM+guDh490Si2OTiATphCiBMyOHM9Gp5AfKDcMQZ3+JROT2KkEBVv863oTSTfLxgF7C/6djxJmK76A/2Fko+zLsBV/PEERwggvJvzPutEB9+c/Tenmzt9dYKA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 3/26/23 02:21, Mike Rapoport wrote: > Hi, > > [..] >> One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases. >> ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time. >> ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation. >> Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag. >> In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped. >> We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases. >> As you well know, among heterogeneous DRAM devices, CXL DRAM is the first PCIe basis device, which allows hot-pluggability, different RAS, and extended connectivity. >> So, we thought it could be a graceful approach adding a new zone and separately manage the new features. > > This still does not describe what are the use cases that require having > kernel allocations on CXL.mem. > > I believe it's important to start with explanation *why* it is important to > have kernel allocations on removable devices. Hi Mike, not speaking for Kyungsan here, but I am starting to tackle hypervisor clustering and VM migration over cxl.mem [1]. And in my mind, at least one reason that I can think of having kernel allocations from cxl.mem devices is where you have multiple VH connections sharing the memory [2]. Where for example you have a user space application stored in cxl.mem, and then you want the metadata about this process/application that the kernel keeps on one hypervisor be "passed on" to another hypervisor. So basically the same way processors in a single hypervisors cooperate on memory, you extend that across processors that span over physical hypervisors. If that makes sense... [1] A high-level explanation is at http://nil-migration.org [2] Compute Express Link Specification r3.0, v1.0 8/1/22, Page 51, figure 1-4, black color scheme circle(3) and bars. -- Peace can only come as a natural consequence of universal enlightenment -Dr. Nikola Tesla