From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0CCDC64EC7 for ; Tue, 14 Feb 2023 21:54:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BA406B0074; Tue, 14 Feb 2023 16:54:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 742DA6B0075; Tue, 14 Feb 2023 16:54:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BC386B0078; Tue, 14 Feb 2023 16:54:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 471B56B0074 for ; Tue, 14 Feb 2023 16:54:13 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 19D1F1A0DE6 for ; Tue, 14 Feb 2023 21:54:13 +0000 (UTC) X-FDA: 80467251186.01.A5B73B9 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2062.outbound.protection.outlook.com [40.107.243.62]) by imf24.hostedemail.com (Postfix) with ESMTP id F0BCE180002 for ; Tue, 14 Feb 2023 21:54:09 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=gMSlEO+p; spf=none (imf24.hostedemail.com: domain of gregory.price@memverge.com has no SPF policy when checking 40.107.243.62) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=memverge.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676411650; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hre9to8ddE2488MjnCmGPJYTu+zLTh7rPdL5cQWBgtc=; b=AuWyv6t4pPuiYaMzKFQ6EejIxc/V6ZdYCe40J9YzAtuyyTDwMhSrejwopM24Ejklqf/9/X 74etSTih1iKqi/Q2Fq7ZvgunIhfW6XCLH3hXz3TUpBkiDgWVCTnmgXiejXceVW7I8jVBnx ksUmSsVNmxNLYcSHPmYKqUwn60pdioY= ARC-Authentication-Results: i=2; imf24.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=gMSlEO+p; spf=none (imf24.hostedemail.com: domain of gregory.price@memverge.com has no SPF policy when checking 40.107.243.62) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=memverge.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1676411650; a=rsa-sha256; cv=pass; b=cd2MyqrW1O0CImVf3LLZZQ5Mx8jStq5nvitxcv2MDqqNEpkVHbd5Kt7N7PUblDQrqePg/5 +G81595FmmRURRzt0S2PBSonyWCRJl4vK41Zq/yQBLpzQuqkwEbZElV4ENg3MxqaEqOQYq ovrAfkqbjKhvW/pOB2z8xnxrRGKDeS8= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bTXMP9XiOkPnEAotjky1MNX/weNF6jcgPGHGMjTj8CldMzcZfVj/nlLYHPQ6TUfz6SPbAOZEwhQK/I+BzIrKePEgyJY7/4wsxdkHmOF0DYQQgF5Tats8hA80v0iMi0L3mGf2HmzQZ/Nz1spDJ3um/fD51QKYlmwj3JyRZe0Ql8sK57zJlK5QINHQTMw/RJD7GEj8l0c1LTqX5J0pVTFtEK1xkbw+lx6sFrzY38AjVFj4vz/Ma2GN6y1TBIzkL6w+vuNvrSLVYV2mXW5beylDEVpaA+egRmgP/r6RbxsizLQRSIFFn3W5UOPt8UMyAk01AmvqsNOzrPJVSe42wxuBwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hre9to8ddE2488MjnCmGPJYTu+zLTh7rPdL5cQWBgtc=; b=M4MTmmFCo2RmV1VWB9bHCxnk3HHDYV36EnSnQ5MAjhpxc/yx/1VOkyE3i4rDpFIiYfingA3ZDqh+5fK1vfboGPUkqkD8+IG59cE/1avnxVCoIqBJI4x6U3KV0ojY9OoYsoCqwn7MwYxNrKorxnCzXppynHXgqp8QGO1Tg5kM3lwdUoklM+seIhNEJYvjm9rdIJvD6l3nRFZ0x3qLUXVV0tcsOThPu+Y6BbBuF1IBF3igFrJO6CgymeGbMpLnZmacUvUeyZvixc1eGMimciEfczLGCPcu2B56R5cRV8kGlQrBmHJ6dQdkTsV3aaSk6a26ISMcbF3cTFVwWDdNs3fpXQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hre9to8ddE2488MjnCmGPJYTu+zLTh7rPdL5cQWBgtc=; b=gMSlEO+pWPAywxJTW8p4Wo7S9NYXbo92pE2iDIF9Hfr/p0nnuAcANO5WLqlOGi8Tr4GRRqqjY+7hdBnxDwzAVwDRuR4lUEB2JxgDDiG70A7V6OyB3ap+Loc5UkIuKZpkT5UaOidIjfdYe0hnNqS9XymAHJGlRr52nHOihkSicR4= Received: from BN6PR17MB3121.namprd17.prod.outlook.com (2603:10b6:405:7c::19) by CO6PR17MB4979.namprd17.prod.outlook.com (2603:10b6:5:355::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.24; Tue, 14 Feb 2023 21:54:08 +0000 Received: from BN6PR17MB3121.namprd17.prod.outlook.com ([fe80::d253:1eb3:9347:c660]) by BN6PR17MB3121.namprd17.prod.outlook.com ([fe80::d253:1eb3:9347:c660%3]) with mapi id 15.20.6086.024; Tue, 14 Feb 2023 21:54:07 +0000 Date: Tue, 14 Feb 2023 16:54:02 -0500 From: Gregory Price To: Jonathan Cameron Cc: Dan Williams , linux-cxl@vger.kernel.org, David Hildenbrand , Kees Cook , stable@vger.kernel.org, Dave Hansen , Michal Hocko , linux-mm@kvack.org, linux-acpi@vger.kernel.org Subject: Re: [PATCH 00/18] CXL RAM and the 'Soft Reserved' => 'System RAM' default Message-ID: References: <167564534874.847146.5222419648551436750.stgit@dwillia2-xfh.jf.intel.com> <63ebd56e639e9_32d61294f4@dwillia2-xfh.jf.intel.com.notmuch> <20230214211824.00007f91@Huawei.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: BYAPR11CA0066.namprd11.prod.outlook.com (2603:10b6:a03:80::43) To BN6PR17MB3121.namprd17.prod.outlook.com (2603:10b6:405:7c::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN6PR17MB3121:EE_|CO6PR17MB4979:EE_ X-MS-Office365-Filtering-Correlation-Id: d94d1261-0c15-41ee-0901-08db0ed5ff56 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: k/v7jYLDrvLz4lGXwXPRvOocsrQG+b5jbler6O9235Y+wafNWkThydVf/S2veD1PxSY98AglHkV9tvw0ixXt8Uf+MVALbQzp9M2yvVOnOHbvrfEREwUrnp/dOQS5ivZO4WTp8r5Tk5STBNk/HX1PU5d1JnVZ+pVUZ3DG9kpBtHmNFGp1s/Ln9Du3zZv20SyPpJVCuTCaKJIyd/aW/ZXpHaUnPDcSCNGqlqQzkQNfnjtLnvKGP5uZN7VoFR3LKo0UysIPL6D+eNlSW/KC44/GKgfre23gyw7e68t6SkzUJ6VNHrG+m7wfSHtQR2mNwWRXftvjlFgec3FqgTuHoR9EhcgjrMm1vh5BrCVVjM4ZYT0lidGXHVA8Spc398tQSzeFT59EUiSpNL32j9VNMRhQjYiyNrd9qj3DV9MqQdyXJMR4EN/OX+zHKqUcKBPi82Sa6yobS7Pa3x5Njs68T2PktwXWJQ4OqRVh5zP8swvImmOyrhhINvS7JwEC0GRJbgnRurve6keGyFrlbPENaAnAcTRdNebO0JqtwjNa0l+41hZpIUwKqiNM0fSbMDBGD3l1Tlngum6Eufy7pZ6TZqiaVxVD0cRKtUa41Vsfi8BV09UXBNTZDTDyt5qpgIfYD3db X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BN6PR17MB3121.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(136003)(366004)(376002)(346002)(396003)(39840400004)(451199018)(66946007)(38100700002)(6666004)(66476007)(5660300002)(478600001)(2616005)(6486002)(966005)(41300700001)(36756003)(6916009)(4326008)(66556008)(8936002)(8676002)(2906002)(54906003)(316002)(83380400001)(26005)(86362001)(186003)(6512007)(6506007)(7416002)(44832011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?NjZLD0rfL3iAVaV0Uh4zVXIvZwbiyThlXBCrlS4YcEAUDaNt8zGPGnKeugLn?= =?us-ascii?Q?8XypxYbAUedod+4+x2KSXKW2wz1TpG3W2jrn8q1OsXzHbgdd3DdGSGsksiu9?= =?us-ascii?Q?KAVVa40VnzjsOCX6L2XfZWnRZpFot64hgyz8DNAL5v7nR3g0AAwAmSENrgAX?= =?us-ascii?Q?y4hC9Io+4lehpjFTQ267LYhvLLficfYMaOW/cFcc/WnTuw6No/E93khY//oU?= =?us-ascii?Q?kHU7R5e5tqOBrXcNhNQNyWViKaY2rRadS5svilW77JXJXG+YcN9NgpHK2j2d?= =?us-ascii?Q?jz5CFahpRAf0GFsJkIrnxIR8fAh8FtOT4HA9Y+iBA+eHFV5D0L0Mc+GcfcqI?= =?us-ascii?Q?2ja/QpSaA6E5QOGwd6WIxNJgIvnqeOEVDAiQgGkhNrjojz7vJqFc4RuHEozz?= =?us-ascii?Q?LZHjEr4S//ZLF4yLXIHF88ZJE3KVUppvA6wcHn9hGoOR6TcwZ/Tsgb8ZBp+V?= =?us-ascii?Q?zVgp143zlMSFTOHJYwEQguLWRMCJyLirf6ty2X6RKB7Kaw4Eppg8OLXiV0x+?= =?us-ascii?Q?o2OaO44tv12+q9KXIpbC22Wh+a8QXaN85iQ+6hxCLi+cEV+Lf+odifdRN8mJ?= =?us-ascii?Q?4KNg7C7BcWoi45u8Nh999/AtWimqMLizwLT8maZ8x23dTs8CuHmRoAvfBhat?= =?us-ascii?Q?e3Kbwmrp/ime1RCO8a7TGcXEK5KBbBLf8obgO8Ymy8EAYKo9StvkPM/dwh8l?= =?us-ascii?Q?snOPHkT3MyUNxLJjEJvGbxKBnCOgYRf22PcLkC5FmNgKe3ZzbkwIY/pngX19?= =?us-ascii?Q?SoCWf4leFFhKpFz76tH0hLfDxops6OHtN2AjyJ2OTzZvbiuVhFfR2yR5RWXQ?= =?us-ascii?Q?1SmWrn6JiI9Q7y9KyHpoKYS40Zi7tq5Vr1buUul/OdiT+Ecn//ftPIL8dlkN?= =?us-ascii?Q?EH+BwwY5Y4agPHPSh1eKWet+qCmr+RXDnsK7BNdGPeV0Wkf6h1812t5/WJs2?= =?us-ascii?Q?Reg6IiSiXhvMGnMWDrU9LylHUY+WtS5mMQomquSkpJt5skCL7s/NloO2ePXt?= =?us-ascii?Q?HNkw/KC8BmHWfPtYkCEf78JSOE4YMupuhvf0qMB9HdSubpRJw53my6LsY6iU?= =?us-ascii?Q?96fZgdBbchz1jpoG7b0XBWvPkAtZmJMoW0KqlaBoeHEQpPV7SsW3X3zNaFtK?= =?us-ascii?Q?EaRWKVFMgONu9sq5FBn4FaF/bfNWDDL2x/Fo3kL/AB32ZNDCb1l4Ji1Tn80U?= =?us-ascii?Q?CH4pssQuZ6hpkqCdhCt3KIKhbVC6LzXqHTsoJY7TyyB+6q3+dJCWQfjyohxI?= =?us-ascii?Q?UiqwbxGeJwjBZqopq8M9rwUIgAoqEgxftsUSzTp0c3o/SrsdNPveu+xdakdY?= =?us-ascii?Q?lRVMLoN1J7G7pXnMxB37DQpfRmmsrbwEZjIVAou8niYjlsU1FQkHTQZ0vISD?= =?us-ascii?Q?mXVvn9VOoO5k607Y4Tk15MvP/AM7Lze9iHTc/R4yuKQOEIqrntJcMw2Edm7K?= =?us-ascii?Q?e+E1Ec4zJF3OMenVYCSSFYuehwrHsoAdklbQ8JaGDOVzKxSbJX0oemAN/VUF?= =?us-ascii?Q?dKuRCdkrM3g3V+FxTMweGAhjVXZFjNcB7pXXAr7nG53CHEElgvaFMY3WYYU5?= =?us-ascii?Q?kO+5FySurNLBUp7dx22VoDOIWf8cLJ9TEO7au+BKhhD0BN8Ac0xUJysNAJrU?= =?us-ascii?Q?EQ=3D=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: d94d1261-0c15-41ee-0901-08db0ed5ff56 X-MS-Exchange-CrossTenant-AuthSource: BN6PR17MB3121.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2023 21:54:07.8958 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Ug74ZcGSSnvI+f8AEbBlXmaapXONhTWMZugKiNV/7kjFZUBc8DZdY2umHtINV6mL9wQePLyhYum/PKxxJBna9im8K5aYQNnJs57bBqOqZvE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO6PR17MB4979 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: F0BCE180002 X-Stat-Signature: r3c1wmttw9y9k1onsbattnjzdch8gexh X-HE-Tag: 1676411649-567885 X-HE-Meta: U2FsdGVkX1/ynAkRhuoZbWVOF/ZraOMtFsQGi0u4PmTs9aNADkBijupabl0zENhRLdIh6577/ow/lWBD7kZwzmfIHeqUu6LmNFlN+SsQYb/B8ATJJR/Xn+hCJEdOeMRTa28m6kNpGhCAz/9iPqBUZEaHrK9+sPqfwVCuWx4BsHHoGZNWK7yOA9e8Or6C2FxQLoBtbjE9HyDZPdcuem5KJAFfZHs4jwJ6xmae4oLSlZbZ1mZI5rOVUeBO0Cp/t8eMMOu+23XOat6QdD2BM+oPzgECuTJpsoBCBwZg/plobntVIvOm/9wm+bjlCDQFO0sUZ7J0RW/MbQ26+mIPpR3w0nJKH/8FOFl8ZN5YCJ1uq+W/T8b9mmSh6QJjCjDe7l0MOoMvVA5bnU59QcWFAyVgJU4SV3UKEvj7fF6akVjf7pVvBTZKkc+tuJxPbEqTP2LPZDSA7DZ0BHDkUQ6MSkle8DPrT+bRvwkCSBYH8SzqcuDdMlb7e+IYR40M4nD4J2pBjXLzOXGzxaM2lVw2Ddv8bnvESwTUKrI1aLAkXYuhMlgO02YZb94nipdTU98fmvnDpc4saYtwfkYcHQghFtPaqQ6K99m1SoPiZAWY7+qkLB5hVCMkowgKJ1GH2CHqec21rhz2AysfgdT1RTEQUx3AnhI9YAqOyGqRmW3AWJ/vC37kzTN63wWSSaVQVL6HDmhFNUsT2yE5kQ/8dTkXgSeMx6bP1KHD5aoyHaSkDl3qnzdnT0UAsoWa9Lp853Ev+pu/3pqgzWdGxWnqvQ53C4sM779JS8Mtczu3ytnmgDx0aYQBoMRd0xzNt0jzJGyALiewmyLKXcXBXkUI4frJ8dvQ8pCUayy1auEZeX624T3PbSS/Uet8IC9IZiZPgCvKmokCYRt0bJz4Rg5uii9pOWwfHhC2Moxtc/FIpgtFTuPOsjboivqZHHjU/6BEgy/PiPujU8GuCunR0HZl75HET+N lGuOAjsJ 1rHXOtw3MpX36xQYdM+rtvnjTctkPS+L4yPNP5l9FasRe0gPJVpFSMYMhwTfq4+sQawixctjat5ye+C2uH6USWWKlkxXhGVSpP1nbNIKGuUtmNAutHh3Ed2/DZKseUQuRPrpVe1Sy/6NrFjr6p7RyYC8R8tqOT8mVGBbSM15jd0a0EFHH2umuwbyDb53beBvX63Mtpe+VKKLGq9AY4mLkxSL1i0a3IKaNGOL43If1b8xj9PIHYfIGQRJGJrYrzjcBFm9Ypjqr0qVVX5ztu1Qw9wh8GmmAjdtyLYSYO5eOIJnDxHtypZeeOQ5PGTnMjcLd/SANF6JIuN4QrydpdVgs6sYdCOyRC406I6BwRC3lqhsuBLs9lheXaZqvePrqCRRh/zrcSGTA87mUl03Nm7bVNmdiBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 14, 2023 at 04:51:53PM -0500, Gregory Price wrote: > On Tue, Feb 14, 2023 at 09:18:24PM +0000, Jonathan Cameron wrote: > > On Tue, 14 Feb 2023 14:01:23 -0500 > > Gregory Price wrote: > > > > Could you test it with TCG (just drop --enable-kvm)? We have a known > > limitation with x86 instructions running out of CXL emulated memory > > (side effect of emulating the interleave). You'll need a fix even on TCG > > for the corner case of an instruction bridging from normal ram to cxl memory. > > https://lore.kernel.org/qemu-devel/20230206193809.1153124-1-richard.henderson@linaro.org/ > > > > Performance will be bad, but so far this is only way we can do it correctly. > > > > Jonathan > > > > Siiiggghh... i had this patch and dropped --enable-kvm, but forgot to > drop "accel=kvm" from the -machine line > > This was the issue. > > And let me tell you, if you numactl --membind=1 python, it is > IMPRESSIVELY slow. I wonder if it's even hitting a few 100k > instructions a second. > > > This appears to be the issue. When I get a bit more time, try to dive > into the deep dark depths of qemu memory regions to see how difficult > a non-mmio fork might be, unless someone else is already looking at it. > > ~Gregory Just clarifying one thing: Even with the patch, KVM blows up. Disabling KVM fixes this entirely. I haven't tested without KVM but with the patch, i will do that now.