From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24B53C6379F for ; Tue, 14 Feb 2023 21:18:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69AB96B0078; Tue, 14 Feb 2023 16:18:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 623DC6B007B; Tue, 14 Feb 2023 16:18:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C4766B007D; Tue, 14 Feb 2023 16:18:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 354DE6B0078 for ; Tue, 14 Feb 2023 16:18:37 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0DA08805B6 for ; Tue, 14 Feb 2023 21:18:37 +0000 (UTC) X-FDA: 80467161474.05.71EDEAE Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf03.hostedemail.com (Postfix) with ESMTP id D71F22001C for ; Tue, 14 Feb 2023 21:18:33 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf03.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676409515; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O3jAGJWBwFZw1spOOKhN+UeU0BNJur7Oq/Ssg+oMu4A=; b=T2Bmjn03SM31A7j25WD9hLyFALuGHOtZE/PCCyOeH72vE1eyFqDQywvBtsgvnNsAtRNVat +liL77yamifev6cFrqKBqzGWe5wLZg7pyOt+ExCtyjQkqO28GJj1d30bsS5RP3ml5namHm IKIcWGAzZWLUXo4qpddyFQPFPETJUNc= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf03.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676409515; a=rsa-sha256; cv=none; b=JB7V0aPBSAGymDivTgYceEvO94njjlyfZQeeQf1oxlap7YoWh99mrlIkRnDctENCVG8mQW YBH4H65Lg/CWxu7COLCedB9yE7f9To+ddJmn+eOTB2aLaoX7JMN7wBTvERQLLbvou9YG/P qoR9mWuj3fiK2jnqO2fn1bivXZprKNY= Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4PGYq41C2vz6J7QQ; Wed, 15 Feb 2023 05:13:52 +0800 (CST) Received: from localhost (10.81.204.253) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Tue, 14 Feb 2023 21:18:26 +0000 Date: Tue, 14 Feb 2023 21:18:24 +0000 From: Jonathan Cameron To: Gregory Price CC: Dan Williams , , David Hildenbrand , Kees Cook , , Dave Hansen , "Michal Hocko" , , Subject: Re: [PATCH 00/18] CXL RAM and the 'Soft Reserved' => 'System RAM' default Message-ID: <20230214211824.00007f91@Huawei.com> In-Reply-To: References: <167564534874.847146.5222419648551436750.stgit@dwillia2-xfh.jf.intel.com> <63ebd56e639e9_32d61294f4@dwillia2-xfh.jf.intel.com.notmuch> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.1.0 (GTK 3.24.33; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.81.204.253] X-ClientProxiedBy: lhrpeml500003.china.huawei.com (7.191.162.67) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D71F22001C X-Stat-Signature: fn6g6gctpm443gwbfmksyphrrnktam3j X-HE-Tag: 1676409513-229742 X-HE-Meta: U2FsdGVkX19skJnvfS/y85IPpvXI06JlN8siIFUdDHvmXfucTAOGYI34lGuZFvt69WTbR3aeRPb3b9Djr4FCLIkjp/GO/6u/rWje/qnegUTJNhTzr81r128hTk1OG/Ea6UDwyUszn1wur+421+d5YThvblVst2Kur436UDH0BgNHBQMpu78V1gPhtBoQvd2/Bgm4OtTDeIs8HOTbfIHOmRJRIEduXb8maEZr9TMcQT3jCpBemZP5ZAS5FtLcij64pWxkA9D9Vf87vN8iAESx+rT3w4sRBL8vhZvQh7puvEiG/MxB3n91HU678DlsO2XlqobY5ZTgZbcn2G/TbuSUghLK/7YGg1/saJSS0pefgI3mqJJeZM1Ok7ABhZm32lT00PXrlDWPYELcqoydqSZCsAyRTxK/KI8yeE1dlDYbPkA4lj73iNVbIhFB+i6uUHRgyE+Ht7JK+7IlyGDzIeIBiyOdnG1EzTRSY7rC+W0gsL7757DaAWAcGz0DpJE6HgHtCJtH5hVYmbMfZjYTPhEbRNtvfrQAy/sp5ZuI5FgWPXtEH8LyNQ1pEb4FEnnh/ruH41tcrenbzXTats+6XvqSSRR59juQbnNBFh2kffmRTrOuHxvmtqjSsqFScYfR1CgT0ZBBWqky2ievyfpbhnYDMdJ8a6c2/+S3WIE2iPxQKh1wvtxOQRu5u6w7bsdxcWbPk1aiE9FaOJPB/QTvZmNGGpPCMnwGsYrYgdcwQYCkQnLkeTUXxRgjnzmfki0ZmonOo7kllqxEBjNutdmSeELQAZtAuDTPWLPrvvxnehRhI5VysXYSpc6CjFjP+i3tq40ZnOv/48ku73d5/dttb5dWtq4gHcr16WMG/tcVTw+4gPe4mrcXc7XJz3WzycFL6i1svwNrzS6vrZq9XrqWZVn06N3elk53gXdZj091wsH0kVA4fFDZc//Nsfj4WNRU9cgRoSpfzLGzYGYw2N2CWWq G4GYx3Ev NpUSJ9vMaUXmZ9/yfcpGF4ySLlJJXuWxxXsmQ73N1lcs+vImdOlULte9YfHZt9aWIOOQkZ8Hs/lioWiBqMrVpLaokff+bmfeHI29jpjpc26LRZryfBD+9CiQyUQox399vm0IJbtZyg5vKl/TnnikMH5kJrm4kfNKvq0v87/ZI5+c6GlJ9zSv6H5UyHJjR9NTNai7HmWIWsukScsHIBtqHOBvRpTCkxGQCzsJBiH39fyIRZy0r+m4Z9/M5WZfupNqhZuMJswDwEhhpeIz1euSBdqrkaj+ASILBfcPOYpOAsatmDQXfUFnOua6zSpf7I+cNxrsQoAfRwvl0AVLB5y3Rp4SSjLfocFCJTRYmld82hDWTphOeXh0CJXJhDbd5dc4JVzk3Y8PMk9RhFyQR0PnEWWVH6nQl+0Kh+BJ5RY2CCUuovMZF4pvuxZ+C/7uJcWHgIksAaGvTxAI5y4UEP2Hz5Aqr3i33mDoXFT6x+Y5ljho7Ec0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, 14 Feb 2023 14:01:23 -0500 Gregory Price wrote: > On Tue, Feb 14, 2023 at 10:39:42AM -0800, Dan Williams wrote: > > Gregory Price wrote: > > > On Sun, Feb 05, 2023 at 05:02:29PM -0800, Dan Williams wrote: > > > > Summary: > > > > -------- > > > > > > > > CXL RAM support allows for the dynamic provisioning of new CXL RAM > > > > regions, and more routinely, assembling a region from an existing > > > > configuration established by platform-firmware. The latter is motivated > > > > by CXL memory RAS (Reliability, Availability and Serviceability) > > > > support, that requires associating device events with System Physical > > > > Address ranges and vice versa. > > > > > > > > > > Ok, I simplified down my tests and reverted a bunch of stuff, figured i > > > should report this before I dive further in. > > > > > > Earlier i was carrying the DOE patches and others, I've dropped most of > > > that to make sure i could replicate on the base kernel and qemu images > > > > > > QEMU branch: > > > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-01-26 > > > this is a little out of date at this point i think? but it shouldn't > > > matter, the results are the same regardless of what else i pull in. > > > > > > Kernel branch: > > > https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-6.3/cxl-ram-region > > > > Note that I acted on this feedback from Greg to break out a fix and > > merge it for v6.2-final > > > > http://lore.kernel.org/r/Y+CSOeHVLKudN0A6@kroah.com > > > > ...i.e. you are missing at least the passthrough decoder fix, but that > > would show up as a region creation failure not a QEMU crash. > > > > So I would move to testing cxl/next. > > > > I just noticed this, already spinning a new kernel. Will report back > > > Not ruling out the driver yet, but Fan's tests with hardware has me > > leaning more towards QEMU. > > Same, not much has changed and I haven't tested with hardware yet. Was > planning to install it on our local boxes sometime later this week. > > Was just so close to setting up a virtual memory pool in the lab, was > getting antsy :] Could you test it with TCG (just drop --enable-kvm)? We have a known limitation with x86 instructions running out of CXL emulated memory (side effect of emulating the interleave). You'll need a fix even on TCG for the corner case of an instruction bridging from normal ram to cxl memory. https://lore.kernel.org/qemu-devel/20230206193809.1153124-1-richard.henderson@linaro.org/ Performance will be bad, but so far this is only way we can do it correctly. Jonathan