From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 737DAC636CC for ; Wed, 15 Feb 2023 10:03:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 010E56B0080; Wed, 15 Feb 2023 05:03:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F033A6B0081; Wed, 15 Feb 2023 05:03:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCB976B0082; Wed, 15 Feb 2023 05:03:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CEE506B0080 for ; Wed, 15 Feb 2023 05:03:38 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 82BD31C6DE1 for ; Wed, 15 Feb 2023 10:03:38 +0000 (UTC) X-FDA: 80469089316.18.C5B4BCD Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf26.hostedemail.com (Postfix) with ESMTP id E9E6F140011 for ; Wed, 15 Feb 2023 10:03:34 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf26.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676455415; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l2Wd+NEZqMkxOAY9TdZMvpfAhNa52qmjDRX18OBFauM=; b=6PkTfhgxbWcUvyH29kfD2vzP/OAjpd8E3xhwnmqOPP5TTaX1/m69hY4Hiki+m+X2RLBA5g 1pToJ0tXU6emC9mEgZeHgxP7px2ZpBDElj0Iq3/PH8ar1ARJYaSJuIKK9a0D4HGZcCqH9H MT9uNCcLnzw0Dye9gZikUJmhcUHftI4= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf26.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676455415; a=rsa-sha256; cv=none; b=mmRQal9AaPxrtXO/0wL4fw462s8ODaH00dUZ2JSQdfGbRCBaMIcQob8tQrGJX8RdP062y1 +vx0TerJW3blwRxorsEvr9X4r5lGMdYp9RejctVmxKCJ68PkC7Cmct2KbTHaFg83sc0DL6 gEmEqxWT2fF2d1ZNvFZo60I5XTt9DLQ= Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4PGtp0322Dz6J67P; Wed, 15 Feb 2023 17:59:04 +0800 (CST) Received: from localhost (10.202.227.76) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Wed, 15 Feb 2023 10:03:28 +0000 Date: Wed, 15 Feb 2023 10:03:27 +0000 From: Jonathan Cameron To: Gregory Price CC: Dan Williams , , David Hildenbrand , Kees Cook , , Dave Hansen , Michal Hocko , , , Subject: Re: [PATCH 00/18] CXL RAM and the 'Soft Reserved' => 'System RAM' default Message-ID: <20230215100327.0000728f@Huawei.com> In-Reply-To: References: <167564534874.847146.5222419648551436750.stgit@dwillia2-xfh.jf.intel.com> <63ebd56e639e9_32d61294f4@dwillia2-xfh.jf.intel.com.notmuch> <20230214211824.00007f91@Huawei.com> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.1.0 (GTK 3.24.33; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.202.227.76] X-ClientProxiedBy: lhrpeml500005.china.huawei.com (7.191.163.240) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: E9E6F140011 X-Stat-Signature: dx4kip1567ipaem5wshhwkptjarx865n X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1676455414-37863 X-HE-Meta: U2FsdGVkX19TG4KOF4vXgXjqsppeO8WVkiC7y1T6RjiB7llrfakpM2++gNN6e+9PffycLKu+zpVNzbY8VwQ+j48wEUogh3SlL502pGRn/+9zSrY/AF/uCDQxzXonThfEEYs6TXSlStAnZEzMk0bN3UL+PwVCfNp4yyEntt8YbckltDociA1UX4LegDc/+8y9bX2IpL7GE4X9Rb8XrXpWw0V+eKCG/ERtjfQ6tcBjRvnMyPn2wbF5i2F9gFHf4rLclZrQYCSvQELpwBLd51VJZdsKglvvg85XbaHSytJkR3JG7HlFcRAAfxLr7s/eDLdGfpu1sw/ZHjio9EeSDM1Hhr85BDFUvnF9WpwJ1o0l67+1x80F1D/qaJDo4eW8NJf4gLoM+KSHVQdV/IjlcVh59IGZS1BSyo0sjkWhGE0sEAudY2tJNGAHg76xjdlIpxhaWqFignGqOhliHHeh9enmnxH2CMvI52qjR+FnMfGczwSWJO9L1MmnHLe+z40w25q+sZZRTXrSa3q16nRkbu3ZAHMBGi7mx7JEpw9rIztRHVcI5W3j7gCBQ0ljGz6BMOm6mwRZVhOeNMImvn8wUDuCC1jzixPvVc2i4X8RaZlHusZi1fVSTehNP74IxRZR1Eqp2v3FkdFDAiYbcxPcMS7pDhkUmvpscUbLi3QdiQ9AhKWestd/ycXU3WMIDoQHGywPjiZYZTQwd1lgKPdpNNIRsC9CnvquiZ8R+T6kXux//QRyhMYe9sL5SGrJSVxcXF684/vbt99kjSS/h+pPscd4cxwYjc5Bkar8fV1allyaqngjWoHFqYc+lO+H9hleMQjo3ziR6uzfnvqrF4OxfICo0L3MFuTHANSrNRjcTJxUgOGkRwLjeiykKbqyh7FxpVXpHnUITg1xyi4+I7saw3vgzCcxjQWVF6qQgI1BXesjwriwODm+fM3H+jidpsBdD8TVdMaOxzMocH52bAJeoTS /8GA0TL5 au6x046HNo4QWyF1XUgGl8bWFY7D09HVWvxzg2asqijq3h+8fyAGjXYo91XGFZP9QyUao+zIgGFiRw1M8f/5V0xC4aU7UlFaXMYT0NUJ0iOn3CcBHbaekUSS0zrOxxQTWm78FyVWKRQo75/8vYVsusAnDiuPJPMBrfXjEUauLgGVFXqn4t9iuyKIwQb5WwWRjAgSV5KJUFRPPscX21TFtvOgYrCqZIhlu5rfqB0qbrRpczBTkqTAtbTDdm1n6CUCz0dwASoaTQ9xgei78kQ4N3lcbsxXDDNN8HdeDLUZn41vN5Q1X/JRaZUtyjKAoEcLTKv8NRReSZNDP4jk1cnlO7DHVnYFgXntYefVcIpxiJW3cs/0t1TWzjSFAN5K3fFBvnYEpHj/m0hChn8sR+BSophzv3A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, 14 Feb 2023 16:54:02 -0500 Gregory Price wrote: > On Tue, Feb 14, 2023 at 04:51:53PM -0500, Gregory Price wrote: > > On Tue, Feb 14, 2023 at 09:18:24PM +0000, Jonathan Cameron wrote: > > > On Tue, 14 Feb 2023 14:01:23 -0500 > > > Gregory Price wrote: > > > > > > Could you test it with TCG (just drop --enable-kvm)? We have a known > > > limitation with x86 instructions running out of CXL emulated memory > > > (side effect of emulating the interleave). You'll need a fix even on TCG > > > for the corner case of an instruction bridging from normal ram to cxl memory. > > > https://lore.kernel.org/qemu-devel/20230206193809.1153124-1-richard.henderson@linaro.org/ > > > > > > Performance will be bad, but so far this is only way we can do it correctly. > > > > > > Jonathan > > > > > > > Siiiggghh... i had this patch and dropped --enable-kvm, but forgot to > > drop "accel=kvm" from the -machine line > > > > This was the issue. > > > > And let me tell you, if you numactl --membind=1 python, it is > > IMPRESSIVELY slow. I wonder if it's even hitting a few 100k > > instructions a second. > > > > > > This appears to be the issue. When I get a bit more time, try to dive > > into the deep dark depths of qemu memory regions to see how difficult > > a non-mmio fork might be, unless someone else is already looking at it. > > > > ~Gregory > > Just clarifying one thing: Even with the patch, KVM blows up. > Disabling KVM fixes this entirely. I haven't tested without KVM but > with the patch, i will do that now. yup. The patch only fixes TCG so that's expected behavior. Fingers crossed on this 'working'. I'm open to suggestions on how to work around the problem with KVM or indeed allow TCG to cache the instructions (right not it has to fetch and emulate each instruction on it's own). I can envision how we might do it for KVM with userspace page fault handling used to get a fault up to QEMU which can then stitch in a cache of the underlying memory as a stage 2 translation to the page (a little bit like how post migration copy works) though I've not prototyped anything... I think it would be complex code that would be little used so we may just have to cope with the emulation being slow. Intent is very much to be able to test the kernel code etc, not test it quickly :) Jonathan