From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C27EC77B73 for ; Wed, 12 Apr 2023 15:15:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF96B900003; Wed, 12 Apr 2023 11:15:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DA9BF6B0078; Wed, 12 Apr 2023 11:15:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C72BA900003; Wed, 12 Apr 2023 11:15:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B80D06B0074 for ; Wed, 12 Apr 2023 11:15:54 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8600BC0132 for ; Wed, 12 Apr 2023 15:15:54 +0000 (UTC) X-FDA: 80673089028.26.0F597C1 Received: from bedivere.hansenpartnership.com (bedivere.hansenpartnership.com [96.44.175.130]) by imf27.hostedemail.com (Postfix) with ESMTP id ADE6D4001E for ; Wed, 12 Apr 2023 15:15:50 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=hansenpartnership.com header.s=20151216 header.b=Se5GfN5k; dkim=pass header.d=hansenpartnership.com header.s=20151216 header.b=Se5GfN5k; spf=pass (imf27.hostedemail.com: domain of James.Bottomley@HansenPartnership.com designates 96.44.175.130 as permitted sender) smtp.mailfrom=James.Bottomley@HansenPartnership.com; dmarc=pass (policy=none) header.from=hansenpartnership.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681312551; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MKgHWqiP/+4kBn51/YgeSgMZCRhDVkbKV/qjRgLijyk=; b=J9jDWa/5o7lknZxLKd4DhSx96o43poEVsXeXNMQF1xLpyZzl40gQKmAmAvJlbdsln7o28u WgXvYnQezZlAwkQnG03tfuUJmnYvh7T7g3+RpDuFQEh7Y2Ep3MZE/1kQhfTH/1UBG6Hk7Z 65WUoTt0dQ1fXR/I/wt9E5krFbaZrEQ= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=hansenpartnership.com header.s=20151216 header.b=Se5GfN5k; dkim=pass header.d=hansenpartnership.com header.s=20151216 header.b=Se5GfN5k; spf=pass (imf27.hostedemail.com: domain of James.Bottomley@HansenPartnership.com designates 96.44.175.130 as permitted sender) smtp.mailfrom=James.Bottomley@HansenPartnership.com; dmarc=pass (policy=none) header.from=hansenpartnership.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681312551; a=rsa-sha256; cv=none; b=5r4NZ3mygGXaSGpDJqf+E3yzht7nXWf3D55aFWjScLoZFnBr2cVqcWHWVp9x5E9J6vfLlq cJViloNvSil/rXq4V0YNUtYgkaQGg5fr7KQj5faoQkNNioP2cFJD6zf4liRSUQiDk2ZDiB 0goq27tVD6Yq+YRIMWIeA7cucTy6R0w= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hansenpartnership.com; s=20151216; t=1681312548; bh=xkBLGR4ioaavPN9y7RmcGfEbICJNVyNyign6g2KnlmU=; h=Message-ID:Subject:From:To:Date:In-Reply-To:References:From; b=Se5GfN5k9BcViXcT63nTs8ICsnL80rS1wV6Ub2uCGrDHNRf2qJALnYXUFhMO73t5o Dm3ylkMEYXDn+M9IrnIjzcwH+ijL3B7dfES1SH2EuECdPI+milyGUCNbwqmgsobWYH GUOpAHujyxvbLLnEAzfnljgKnMvKbThfSeGn9Uro= Received: from localhost (localhost [127.0.0.1]) by bedivere.hansenpartnership.com (Postfix) with ESMTP id 533F21286CBD; Wed, 12 Apr 2023 11:15:48 -0400 (EDT) Received: from bedivere.hansenpartnership.com ([127.0.0.1]) by localhost (bedivere.hansenpartnership.com [127.0.0.1]) (amavis, port 10024) with ESMTP id l8ayoMz3Dxcj; Wed, 12 Apr 2023 11:15:48 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hansenpartnership.com; s=20151216; t=1681312548; bh=xkBLGR4ioaavPN9y7RmcGfEbICJNVyNyign6g2KnlmU=; h=Message-ID:Subject:From:To:Date:In-Reply-To:References:From; b=Se5GfN5k9BcViXcT63nTs8ICsnL80rS1wV6Ub2uCGrDHNRf2qJALnYXUFhMO73t5o Dm3ylkMEYXDn+M9IrnIjzcwH+ijL3B7dfES1SH2EuECdPI+milyGUCNbwqmgsobWYH GUOpAHujyxvbLLnEAzfnljgKnMvKbThfSeGn9Uro= Received: from [IPv6:2601:5c4:4302:c21::a774] (unknown [IPv6:2601:5c4:4302:c21::a774]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by bedivere.hansenpartnership.com (Postfix) with ESMTPSA id 66B1B1286CB1; Wed, 12 Apr 2023 11:15:47 -0400 (EDT) Message-ID: Subject: Re: [LSF/MM/BPF TOPIC] BoF VM live migration over CXL =?UTF-8?Q?memory=E2=80=8B?= From: James Bottomley To: David Hildenbrand , "Huang, Ying" , Gregory Price Cc: Dragan Stancevic , lsf-pc@lists.linux-foundation.org, nil-migration@lists.linux.dev, linux-cxl@vger.kernel.org, linux-mm@kvack.org Date: Wed, 12 Apr 2023 11:15:45 -0400 In-Reply-To: References: <5d1156eb-02ae-a6cc-54bb-db3df3ca0597@stancevic.com> <87v8i22abl.fsf@yhuang6-desk2.ccr.corp.intel.com> <87bkjtzu7e.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.42.4 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: ADE6D4001E X-Stat-Signature: jwkwfkw9wja5wbtk97rfy9cdsbuqc79k X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1681312550-8033 X-HE-Meta: U2FsdGVkX1+KFguelEwsONiXT1DBbMktcUd/SjgTOcoWn48E6tJ2SRkM82KoNQEqWApiKk0vtfmtLL+unYZXCG+0k9pFlYNXE02Y73/E8jq4VdKHKZJIgrEZj7V1AFUpzE5LEC7Jl5VJ7X7mNmNn0qnb5CVozIQnTey3eAIUNT2x+cvDrY9d3J9u4XX3dxnVt5xpXJc74XMyrAvEek64+/wH9bErzDdF9Nj4Qp1ymjuv3ychzoYEGivu42auDgYsHUdBRAqVTPYH1tsnOaxava4OHI6cc44S50amdmT1VY2W5LLk0NYW283xPn+Fm9xhbb3k+Ktsv9YrLjJQ/l0QJzIZQI3DvKhfFFxijx/zALRxp5bESVXtxVTPPP7VvK0CtH9YuYqa0q3Z3YKuxxi+2DlNp4sS8BtugKvCvolPua9B19oiEkF9gxsX+lUtYeRBz6oOBmFFCf3xc3GCjvMxwFE8t6lu5VAgASae4quwAGZrTDuwn4+OE2TLRcETxFLdidlL4rzibxMZzBz7cG7A1FRnRpU393G14a+cYo05Jamiff6hreN3f3opU0dfs0vtPVEWFxCQOlAwEDQCUFzy5QQQGygcaLhNtmaA9VzZ4T0ArAqVFQynMGRdyh9HxKgdiD2v83mGF3cZyXOtSaVGrFKepgEi5YIiTV6XpiW+fjpKZwyXeRPgXVy5/3lPJ8ZwOfkrHZ1odOYsuLQy5eAxGoVWIrgg7xW8prlDXmYmFPA3rw+3iRMWbnqTdhpEC45ESW9oscbPRRhPPsE3ATf9LkHI2eSBzkSDZEdEVilc2OyD31UKQ4QBwEuLOAdIBY4Jy+HsXocc3/zB9yOq4wvRqnLIsDVZ+uThpsqpDQZ6q8an1NdQiwjaPsgDWTVAGXfRaabru/x3S3bYS/MeCrouDD0RrMW4FXGkhrSkuJ3yAdCkZid+epMNu7EYMoUKJJgDwf8AZWUKYF/7flgdJXg IxmqyuA3 W7TIMHaj+hCK6ruLh0UQfG1nBGA12/fgndFiX6UaXfYyG0Itk+FIDUzgaRvx1HKYb+7s9vrNmae/0OoBvmwQeQHg8Hd6SRqOah3UzuhEBk/TddfOK8TX4kuvngS6DDmFOkkazUNXGcGxVZv9FXIhZ1BRmyL/2yI3R1SsRoL1BY6oKDmHEKxOcluJS6qIfM9OBKzxKS5jA+ycXeUXZTDp1AlAbdlKC2tFsFkBo9JmsCFSMQmj25Z9SOe3eoG5q60BPonAti1ClRkh4FkT8CDR8SQefN5euE6ZtfsIZxA/5klLA3ou6USgWbaWc8Pg0nnV351QA467F/UK0rxzsqQcrMn/gpJc7t8BWwsKTT28EG0XnjfLSw0IWE5BtWnsBWSxmru1Tpfb1wFL0LvF3LcFOv2tUr4uEESCFce5d+l4do1l74PqyuOEah76j6L6gkwxU+zCC X-Bogosity: Ham, tests=bogofilter, spamicity=0.357107, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 2023-04-12 at 10:38 +0200, David Hildenbrand wrote: > On 12.04.23 04:54, Huang, Ying wrote: > > Gregory Price writes: [...] > > > That feels like a hack/bodge rather than a proper solution to me. > > > > > > Maybe this is an affirmative argument for the creation of an > > > EXMEM zone. > > > > Let's start with requirements.  What is the requirements for a new > > zone type? > > I'm stills scratching my head regarding this. I keep hearing all > different kind of statements that just add more confusions "we want > it to be hotunpluggable" "we want to allow for long-term pinning > memory" "but we still want it to be movable" "we want to place some > unmovable allocations on it". Huh? This is the essential question about CXL memory itself: what would its killer app be? The CXL people (or at least the ones I've talked to) don't exactly know. Within IBM I've seen lots of ideas but no actual concrete applications. Given the rates at which memory density in systems is increasing, I'm a bit dubious of the extensible system pool argument. Providing extensible memory to VMs sounds a bit more plausible, particularly as it solves a big part of the local overcommit problem (although you still have a global one). I'm not really sure I buy the VM migration use case: iterative transfer works fine with small down times so transferring memory seems to be the least of problems with the VM migration use case (it's mostly about problems with attached devices). CXL 3.0 is adding sharing primitives for memory so now we have to ask if there are any multi-node shared memory use cases for this, but most of us have already been burned by multi-node shared clusters once in our career and are a bit leery of a second go around. Is there a use case I left out (or needs expanding)? James