From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70C12C3600C for ; Tue, 8 Apr 2025 04:14:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F2DB6B002C; Tue, 8 Apr 2025 00:14:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6A0276B002D; Tue, 8 Apr 2025 00:14:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 567896B002E; Tue, 8 Apr 2025 00:14:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 388666B002C for ; Tue, 8 Apr 2025 00:14:40 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 01952B49BF for ; Tue, 8 Apr 2025 04:14:40 +0000 (UTC) X-FDA: 83309560362.19.0E79674 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf16.hostedemail.com (Postfix) with ESMTP id 0C6BE180004 for ; Tue, 8 Apr 2025 04:14:38 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=Yi87N2bz; dmarc=none; spf=pass (imf16.hostedemail.com: domain of gourry@gourry.net designates 209.85.222.179 as permitted sender) smtp.mailfrom=gourry@gourry.net ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744085679; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YxylarJR1WfRQR4vudg24nlR2rF7SNaVZ/wz82/8qPI=; b=D/2cqncFLE8nhsI9qn9ez0MjhrRXpnI6I977sldOEAJnHpHv1UiDnYpSyxIjyWdlOqGSLG ub5gf4VA5edUMAcvN6a/PQdlB/U3bANouY2Mzuzoa4Qv1RQ2gVnACOqY3C/YiDsjD7alfA 4Mc+rz+vkJP0djNpKplQXy9xQW3G24o= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=Yi87N2bz; dmarc=none; spf=pass (imf16.hostedemail.com: domain of gourry@gourry.net designates 209.85.222.179 as permitted sender) smtp.mailfrom=gourry@gourry.net ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744085679; a=rsa-sha256; cv=none; b=K8jNL7A0SQJKm64Oq3EPizT+kG1+obkq7FKAn+cjNcj1L5brEL7w061vRfhymLr3Fyoyxu ApDfs9lw/0gccwpbAbPGxcVobDzK1EKFrzEWak7WAE8+aDmDIJREatbJs0a8s80QsN7IpJ rDgUrXmwiSxQvHGq+JgLvmEIg8Eu6o0= Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-7c08fc20194so981749685a.2 for ; Mon, 07 Apr 2025 21:14:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1744085678; x=1744690478; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=YxylarJR1WfRQR4vudg24nlR2rF7SNaVZ/wz82/8qPI=; b=Yi87N2bzLbBJ6ajLJkxX8aJ381nBkjfR4CXWAyAwi3jyMMPrOSt1T6Lqvpwy8kIKKG oEhBTKVLbC/p66pfQ0TB0bL/kJSg6IiBT1dUZKra4frZ0+EYjzYPmJWiwGK1ullfH0wC qsuAgZ2HNaVpTH2WyTkUMjG5+cGUo/v9dsGT8J1et6IWVtUN1o7VtXC6dbvaSABdak7e MMqcJrKMuzTOqogz8eYNF4oIhMuj4mu3/qYZWX0X2DfSDvb3dUezQ8QrHqFduZg0UYAd PZBDA7ZxooSdgtItE0IoLiVCllAcsE2AXEXC2t6Ou8FsEBQT173FbBbehkTEvb4KiMT7 zbig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744085678; x=1744690478; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=YxylarJR1WfRQR4vudg24nlR2rF7SNaVZ/wz82/8qPI=; b=uGH48sEWFG+85SRwaGXmH1qJu1tYE9dgAc+VCivPOdKyO88JdAeYxqcLOORO0IAQug BHO8/HWdmkTMO4G7tjnjdcxu04pjqgBW9ZzJK8gZ7fWeU/tRGC66uPi0FPMouOo89aBG Zm7l0dfqLrqZjuY9+XpLEQjqUJhzCgOWD4K0A0o9onnRQx2t9dJVrGrervCbEfu2zSWm KbdrN2eyDHNBwh7a0TxgLZusYYG0B8CE9voQe0RAviccVUmDD2ujxrq5r1KX0S4Feau2 bIJi/nN+sknmnWw9dKUGx0xtUezu0Ow9KOwt5QJR6C4QQXHlM7J5wuU6jBOVcyxrgOdi jR8w== X-Forwarded-Encrypted: i=1; AJvYcCVHzsVx8AOPbXVdXpEcYWHy5gd11lUwSyNVgtdbKT0QXxsAj8WIHslwWybGNFkqbB5yLVBn/Jlm8Q==@kvack.org X-Gm-Message-State: AOJu0YxqAF5HM859t60RnNr5Z/LrbOer9XfVBv3a0Z1fNvOY8R84jKtp R2hNwHPY5GM1C+6YkL8lHyPRGZtzPLmfoKKMIeVsMq8zy1xC7T84Ww+XnItgJSw= X-Gm-Gg: ASbGncs+lAKHIVr72kaT7OmQm+7JEQMo6WZdkIiRXS70SzLJpoIhwVy9AILjc76bmoh /2BHVBkWdz7+MoSz9zAbpsea4hoPXeedb1HRF9PzNWVuoJT9+qWUfEFjwK6i3rNgLIC7TWHYopM Pc9T1j5ke7UJ5W43Zaisxj4P5elHW5HFMAEIFspI/CaxMz9yJXfjYkrLMxzdjJSd7CxBUOf7K1t 7jQVhF00Cwbg1SWA4vcKqCIHCC0JFrDs6cHiCc09tzSeiBsuTfExcj/VJRdcWeLUk+vNKHTtMLc kwxkbP4MHWcc2J9cchD53LShJwW55z7q+ss4QuP7ypkDx5dpg6Jo8P12K/Lfsv16slA5xuWncWj DC4yUoCfCNhv2yoUlMOIk40bm2PV5+GTJI07nkg== X-Google-Smtp-Source: AGHT+IG1fG6kfVEFdWk0CSWBtlYn13aGmNvut/72B2/Nbw/wJRvYa17H6rgmfkWBfGCZtZESEG8UjA== X-Received: by 2002:a05:620a:2942:b0:7c5:5f58:9158 with SMTP id af79cd13be357-7c774d14899mr2510471785a.9.1744085678001; Mon, 07 Apr 2025 21:14:38 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c76ea59cd8sm699226085a.69.2025.04.07.21.14.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Apr 2025 21:14:37 -0700 (PDT) Date: Tue, 8 Apr 2025 00:14:35 -0400 From: Gregory Price To: "Zhijian Li (Fujitsu)" Cc: "lsf-pc@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-cxl@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: Re: CXL Boot to Bash - Section 2a (Drivers): CXL Decoder Programming Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 0C6BE180004 X-Stat-Signature: 5959qobd1p7wjo4ammy85ooeej7w3dzh X-Rspam-User: X-HE-Tag: 1744085678-612151 X-HE-Meta: U2FsdGVkX19p/0b2QBJhONy6iOKIi+CEsq5Kw2HMpYb5RJol1mdebMcSOD7bwI6ubE1MQDk/kEbHMuWWYTP6NPfLPepgn8EuUi46Y2CJqBFGZJ6qogbVAD5Whh84QmGIEu9au+I0evOzVlAJ5npjzy6SS4idN62PC0P5zd/lqb4zcGOy/addENAj0TX1jfGC0I1RD2eThishXf6MizD4pXboPoxOrsOB45Y45R2FNHrLWpNOkUq/0bSjOCG3hK4Q/EwdkiflYVNyRSxeCqku/ycEeTT0wQPWajIybtDCEnIOiZ7IZU884zgmVZ5US+RISJM80Kph7QgcidO0NssF1QhBeQgWEcmN/gHAZGivIu5Zq4MPGtoMORtSk1ZynY20hCJPU2zJSxsu77kiTe0YPfwob5sXmgvJxk7q4BDFIq73Rzs9eKdlDRZsweF6DUHXk6BQ3WipFgKrp8zDXs2jDsTnxE0VLgTNQwtOl7ex2wL7F3pvdHCn4Hs+7iOnUN3wuMNz9HJCTnG179KyuBiHjzRnGXgyTdV/Jl1gt1PIFVh3Xxoj0NSsjX/YIhT+CDAcs8MP+FhUF2e6EJp07D/dZU6WyPR6gZ8lhgCRCWxwCnzJNapvBzNF5+E5wG4gsxvSFO5Uaka7LFKXNp10YGA7shVWIE7qkMMSr3POKQhN50SSQHSE1dTZ/E8TLfKh5f7ASLH0PGp82nXpRuJKSqdlsYcI6NtZxjpoP0X1Fc2wWIQ0hCYzKN1sTqvCBlZJhICAxCDWN6oUroEVkFdCShV9TjeOqx4HNviuhrukp9NgV3PwjrBV61pfXbMpKoV8rnVVQEztcKGxNZPEVgFJoO/3JFXSor78fLC5pY4ck1bFBkl+A/RFfYG7veouxl4YS/kl5yRetM77H33/jOecb033IZNWJBd4Fx+25JHk//RlUfAdsVnJSfvxW96SsfHg2N1t+vIuwmCA/ftqpFE5/n/ 4z11MrxU DVlQA7Fv5RxvHrbJOSSgE1KaqDvg4zPIWm4p0rTjsetWr9x5AlYjQpwuK6w0mFExUC0iFL4HXJ/I4VJHYK6n/uIy+ZF4r24NYZVHJdL7WDuhXENdjS/EDr9O2TZO8nQP8Z3xApD63ybSFKi4tApsg6nd2hTNk3lAbxSlnyYxbr/SSpE3Jutd60PrwtAibEseC1nm6lhoxBOb2nQd43pN5G1Hm6jRBUMQr/VfbznmeDWMLqkOa8gLw0qfbTgDx7jHuXngkXf/Z0iQWdyVfPS7t4mPEszf2EkqERorFw5jsixAM0dNUCGRw+aPeUOq/lVE1rB22llNICvEs0YKSS4LXrDU9s0HhevpaWN9kslREcuz3PEHcSH3JmviJ8/W0FNehajJEsdwreSFSA874bM2EgW18e9Tub4/Q+2i3qy02vgfH1KMGNoOrZo2xYXQBWMEOnvwygShGRpBlL8cCFVSoWGkXNBvpkLl+R29X X-Bogosity: Ham, tests=bogofilter, spamicity=0.001372, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Apr 08, 2025 at 03:10:24AM +0000, Zhijian Li (Fujitsu) wrote: > >> On 07/03/2025 07:56, Gregory Price wrote: > >>> What if instead, we had two 256MB endpoints on the same host bridge? > >>> > >>> ``` > >>> CEDT > >>> Subtable Type : 01 [CXL Fixed Memory Window Structure] > >>> Reserved : 00 > >>> Length : 002C > >>> Reserved : 00000000 > >>> Window base address : 0000000100000000 <- Memory Region > >>> Window size : 0000000020000000 <- 512MB > >>> Interleave Members (2^n) : 00 <- Not interleaved > >>> > >>> Memory Map: > >>> [mem 0x0000000100000000-0x0000000120000000] usable <- SPA > >>> > >>> Decoders > >>> decoder0.0 > >>> range=[0x100000000, 0x120000000] > >>> | > >>> decoder1.0 > >>> range=[0x100000000, 0x120000000] > >>> / \ > >>> decoded2.0 decoder3.0 > >>> range=[0x100000000, 0x110000000] range=[0x110000000, 0x120000000] > >>> ``` > >> > >> It reminds me that during construct_region(), it requires decoder range in the > >> switch/host-bridge is exact same with the endpoint decoder. see > >> match_switch_decoder_by_range() > > > From the code, we can infer this point. However, is this just a solution implemented in software, > or is it explicitly mandated by the CXL SPEC or elsewhere? If you are aware, please let me know. > The description you've quoted here is incorrect, as I didn't fully understand the correct interleave configuration. I plan on re-writing this portion with correct configurations over the next month. Linux does expect all decoders from root to endpoint to be programmed with the same range*[2]. please keep an eye on [1] for updates, i won't be updating this thread with further edits. > I have been trying for days to find documentary evidence to persuade our firmware team that, > during device provisioning, the programming of the HDM decoder should adhere to this principle: > The range in the HDM decoder should be exactly the same between the device and its upstream switch. > In general, everything included in this guide does not care about what the spec says is possible - it only concerns itself with what linux supports. If there is a mechanism described in the spec that isn't supported, it is expected that an interested vendor will come along to help support it. However, the current Linux driver absolutely expects the range in the HDM decoders should be exactly the same from root to endpoint*. My reading of the 3.1 spec suggests this is also defined by implication of the "Implementation Notes" at the end of section 8.2.4.20 CXL HDM Decoder Capability Structure IMPLEMENTATION NOTE CXL Host Bridge and Upstream Switch Port Decode Flow IMPLEMENTATION NOTE Device Decode Logic The host bridge/USP implementation note describes extracting bits for routing, while the device decode logic describes active translation from HPA to DPA. ~Gregory [1] https://gourryinverse.github.io/cxl-boot-to-bash/ ^ with the exception of Zen5 [2], which I don't recommend you replicate [2] https://lore.kernel.org/linux-cxl/20250218132356.1809075-1-rrichter@amd.com/T/#t