From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FE3CC282EC for ; Fri, 14 Mar 2025 11:09:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0AC9A280002; Fri, 14 Mar 2025 07:09:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 05B40280001; Fri, 14 Mar 2025 07:09:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E64C2280002; Fri, 14 Mar 2025 07:09:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C782C280001 for ; Fri, 14 Mar 2025 07:09:51 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7D162160F2E for ; Fri, 14 Mar 2025 11:09:51 +0000 (UTC) X-FDA: 83219886582.17.35339B7 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf15.hostedemail.com (Postfix) with ESMTP id 3584BA0003 for ; Fri, 14 Mar 2025 11:09:48 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741950589; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yk3lIk39SAre4g2jNzkSvTrq7PNYsx26HQ+SiHkU3Jo=; b=X0ktHf8ZlZ10uC30iCj/9KqSgnFYxj+bSpa0rlAXZ52Fh9MBaCdIb2b1Pt2hz7q+56QNft BCZJmr3qFeqEIln51ERmZu5xnIaOVukNHDs/QIgdHe+HJ1Qv5E6sqya5kv7XHJw2ksQC2L fnsWWxOjTVk3vhrkrD1Chlg5CoL2iPc= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741950589; a=rsa-sha256; cv=none; b=Ordc1FQImlM+IK7nE2lv8cCPGo72H8AA/+DsKKrHZKy6EpSdVyLQBXwuUSgNnQfNHqItdg 8qmD2lIjo7UNCQbjvw00fkWbK5GuJ7kXo3VJXxqTqWqXU8CSyoYUkHVllnEOnRozLVWueT e4lSGbvkVCreamQ09dIrkDhfY5eVbzE= Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4ZDhQ10LXDz6J7rd; Fri, 14 Mar 2025 19:06:33 +0800 (CST) Received: from frapeml500008.china.huawei.com (unknown [7.182.85.71]) by mail.maildlp.com (Postfix) with ESMTPS id 29C72140680; Fri, 14 Mar 2025 19:09:44 +0800 (CST) Received: from localhost (10.203.177.66) by frapeml500008.china.huawei.com (7.182.85.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Mar 2025 12:09:43 +0100 Date: Fri, 14 Mar 2025 11:09:42 +0000 From: Jonathan Cameron To: Gregory Price CC: , , , Subject: Re: [LSF/MM] CXL Boot to Bash - Section 0a: CFMWS and NUMA Flexiblity Message-ID: <20250314110942.0000476a@huawei.com> In-Reply-To: References: <20250313172004.00002236@huawei.com> X-Mailer: Claws Mail 4.3.0 (GTK 3.24.42; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.203.177.66] X-ClientProxiedBy: lhrpeml100004.china.huawei.com (7.191.162.219) To frapeml500008.china.huawei.com (7.182.85.71) X-Rspam-User: X-Rspamd-Queue-Id: 3584BA0003 X-Rspamd-Server: rspam03 X-Stat-Signature: pze6hrb9i97gnojd9edi7pt3jsenscx6 X-HE-Tag: 1741950588-335082 X-HE-Meta: U2FsdGVkX19mImyvxrc+LmrFKSokHkSwVw5Q2NiOyP4DpRFoWmkf/QQgv48sLrdhfjas80L2KsPcUoGtykSIaiw9xw3ThqvsNfAM+tS5MSfizC1GsUgnXRldojalawSm1CYwCHi5uxjBOokQ+2IRtL/FEyIMNbtZm2LfMdzykEc1qO8NhoEZ6eeQcQ2TB2qfD+azMFD4898bF+lR3IgU4aT34dPNKLhvggl10PF+MKGefHFg/XbLBbCfQJJe04KBeMT4d/WwqlHLaDqenpJazWPURNCvLZBdwPzOvDuLxA6za2CFjnhY0By8CBZmUM7It3J75Diq+ulbRr5sAn1MG7Yn3Ynni2m2KbpL+KsWXnwMlGV/6o3RrcnAXWmpZQ5fbU2etCPyMHGFtv0lVNIZn/h3iXPI3Z0EDXRsGfnqtYWykaqaQpF0PB0DxzpA73Tk1ZgXQiIAMhs6pjMwGihS5cw3fybD/b+lN0tYup0zEIQGoPfbXnHE4FSlj45eUPjtAcFzDsy2+y8fYRt1Io4kN4z0DyqO1HmUukw4Zj3L+onHHLpiB5psp9AhR1a/KqZbmtXX2dKJ4PPSdB6QU1O4a/RdCyeOKczqv0PgcsJ1CqXIFiTMWmmdz04n6jcKPH2ESnvmSpWAc7kLFsng4sjT+kKZvJ+gUYEfb22TPA8omY69Zex0NxQ8tnOfOHoPqo+AcBo0Beqcv7wtA4RYKPTCeJZfP31MH68OB+KvDs5OPmEdtv5hoIkhNJDrlA2f8SBWD787gTX9xiwD2ODO9OT37g7O2qhMJNrUCbe2dm/D66TENM2bq3YF9KF4nopYvOf7oZXgKmpYEwMTVxSXlqOmTe+zajTz/10heAoYquR71ycN9khmqLA/4htWAvE0dcnGMEfN2gdTgj5z7MTuLQFgW8wJhZWiggLO+G+TTXvYPZdsQFRKGHP+TNcI/dRJGblmsFr8ykWx5/J3EjrOyrg cebS2nKf m1yz37TEIyFRiCTnN+MDROxHd4e6tbjOO70ITbu6/cUdznDm4tlVmSGC2xZaWqNeMx24JMW2mNwnYvdfMppM+M/3SbtLpIPGV5DlDbwJYPSPJyXCLdsyR8vQ/McdM6j1OorycUrT3JtjnNjqsQpBB3Y9vvMJnoF4MnzYcgaSzY3DLKTYK0cmqj8msQ6PjyfNIylx5ODGDck5+y2ynCKpBHEnD1M4sTk14DwKds4d0ksys3l4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, 13 Mar 2025 14:17:57 -0400 Gregory Price wrote: > On Thu, Mar 13, 2025 at 05:20:04PM +0000, Jonathan Cameron wrote: > > Gregory Price wrote: > > > > > ------------------------------- > > > One 2GB Device, Multiple CFMWS. > > > ------------------------------- > > > Lets imagine we have one 2GB device attached to a host bridge. > > > > > > In this example, the device hosts 2GB of persistent memory - but we > > > might want the flexibility to map capacity as volatile or persistent. > > > > Fairly sure we block persistent in a volatile CFMWS in the kernel. > > Any bios actually does this? > > > > You might have a variable partition device but I thought in kernel at > > least we decided that no one was building that crazy? > > > > This was an example I pulled from Dan's notes elsewhere (i think). > > I was unaware that we blocked mapping persistent as volatile. I was > working off the assumption that could be flexible mapped similar to... > er... older, non-cxl hardware... cough. You can use it as volatile, but that doesn't mean we allow it in a CFMWS that says the host PA range is not suitable for persistent. A BIOS might though I think. > > > Maybe a QoS split is a better example to motivate one range, two places? > > > > That probably makes sense? > > > > ------------------------------------------------------------- > > > Two Devices On One Host Bridge - With and Without Interleave. > > > ------------------------------------------------------------- > > > What if we wanted some capacity on each endpoint hosted on its own NUMA > > > node, and wanted to interleave a portion of each device capacity? > > > > If anyone hits the lock on commit (i.e. annoying BIOS) the ordering > > checks on HPA kick in here and restrict flexibility a lot > > (assuming I understand them correctly that is) > > > > This is a good illustration of why we should at some point revisit > > multiple NUMA nodes per CFMWS. We have to burn SPA space just > > to get nodes. From a spec point of view all that is needed here > > is a single CFMWS. > > > > Along with the above note, and as mentioned on discord, I think this > whole section naturally evolves into a library of "Sane configurations" > and "We promise nothing for `reasons`" configurations. :) Snag is that as Dan pointed out on discord we assume this applies even without the lock. So it is possible to have device and host hardware combinations where things are forced to be very non-intuitive. > > Maybe that turns into a kernel doc section that requires updating if > a platform disagrees / comes up with new sane configurations. This is > certainly the most difficult area to lock down because we have no idea > who is going to `innovate` and how. Yup. It gets much more 'fun' once DCD partitions/ regions enter the game as there are many more types of memory. Jonathan > > ~Gregory