From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06270C36008 for ; Wed, 26 Mar 2025 09:28:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7EC8A28006F; Wed, 26 Mar 2025 05:28:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 79DAD280069; Wed, 26 Mar 2025 05:28:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 665B628006F; Wed, 26 Mar 2025 05:28:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4544B280069 for ; Wed, 26 Mar 2025 05:28:23 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A022E141313 for ; Wed, 26 Mar 2025 09:28:24 +0000 (UTC) X-FDA: 83263176528.14.5A3A45C Received: from zg8tmja5ljk3lje4ms43mwaa.icoremail.net (zg8tmja5ljk3lje4ms43mwaa.icoremail.net [209.97.181.73]) by imf26.hostedemail.com (Postfix) with ESMTP id E8863140008 for ; Wed, 26 Mar 2025 09:28:21 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf26.hostedemail.com: domain of wangyuquan1236@phytium.com.cn designates 209.97.181.73 as permitted sender) smtp.mailfrom=wangyuquan1236@phytium.com.cn ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742981303; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BoSGTyIsRcpvwYANrxx7DRV17Z/VJW1+ZcN4jHZ2JwY=; b=6M0LNOHFKaCaaRB38ZtU6/2a1O2J9a+rmtXATaQTjbP1ms4PjHVJOUDfdoLiXWxrKbs16+ JzWLaW0e5Dr1nVQ+JvcOG5NFnqaZ0marzpjVfPE0D+5gXfCs7K2XutcyEo/BZD423yXH6E z19SnYQ+mJj6CIMspKYeeyGj+N87gOc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742981303; a=rsa-sha256; cv=none; b=h4/h0AvzrFTaXcEvrzSXi0PPoaRyah/BwfMnlc+pRQGckGcNP0+2YTdJiboJkNVbuf9j0V Zz48bpyMHvaLLwkJDDczaMr+vYNeY5MkRN9XdUtvKpJw5RcNmBdXJfkJKuxScJLZnCKhbM hGbhUBx0vQ02c07fqJZVlzRHvEm2tmM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf26.hostedemail.com: domain of wangyuquan1236@phytium.com.cn designates 209.97.181.73 as permitted sender) smtp.mailfrom=wangyuquan1236@phytium.com.cn Received: from prodtpl.icoremail.net (unknown [10.12.1.20]) by hzbj-icmmx-6 (Coremail) with SMTP id AQAAfwCnrSWxyONnft0IDg--.16989S2; Wed, 26 Mar 2025 17:28:17 +0800 (CST) Received: from localhost (unknown [123.150.8.50]) by mail (Coremail) with SMTP id AQAAfwA3PYmvyONnhX1VAA--.4059S2; Wed, 26 Mar 2025 17:28:17 +0800 (CST) Date: Wed, 26 Mar 2025 17:28:00 +0800 From: Yuquan Wang To: Gregory Price Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [LSF/MM] CXL Boot to Bash - Section 4: Interleave Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-CM-TRANSID:AQAAfwA3PYmvyONnhX1VAA--.4059S2 X-CM-SenderInfo: 5zdqw5pxtxt0arstlqxsk13x1xpou0fpof0/1tbiAQABAWfjC7gD6QAAsc X-Coremail-Antispam: 1Uk129KBjvJXoW7Cw47Wr17WF1rCF4kAF45GFg_yoW8KFWxpr 43tryfKr4rJrWxJr17J3WUXry5Xry8AF45GF9Fgr4xuF1rAr18Xr18Gr18Zry8Ary8XF1j qa1UJ340qryUCaUanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUj1kv1TuYvTs0mT0YCTnIWj DUYxn0WfASr-VFAU7a7-sFnT9fnUUIcSsGvfJ3UbIYCTnIWIevJa73UjIFyTuYvj4RJUUU UUUUU X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E8863140008 X-Stat-Signature: amsxzi8zky784sccn7dm1kpyrnq87wjw X-Rspam-User: X-HE-Tag: 1742981301-520615 X-HE-Meta: U2FsdGVkX1/cflLiNdoYDoAu10rEg5Szo5PgonHtQU/kySGDFg2kx8gPN4jEWXWSQ75gSrfEIqbxTh9xYW9RuHbqPOw+XDiMBbBaIIO3ZVXWkKyDfM9EMaitrGfqOqIIGGri7N2FemfIeGdo6zaftfWJrDB7j8181OaNhZxAtWt8wihktNcaT2IV1PgYMVPEyi5N1SKwqjci6CT8/aEpQbVPni0kNHIsqrPS1wkEjuE08NbE+a4ip5BoCHYNIfZN58Uphu+6nWp96DeHXPOLdgYgJXnUEsAMCTJMzMn0lzoDsOP2OEb797s/P4pO92Qupn8Q1GUMZ0sSL73q4Ncz8zJz5snHVJrWlxqOJWE7L1Af+5P11Qh6Nu8kLciSThjUhewrmvodoXAVaIzHClA6CBqnpfhkbSJ47r03k2OFvNbhMNFSy9JTbSmkY3/gqp0bDELtncQFI9KyQTzR6Y/EKWsFlWOTlSNWCZdr5EmaeF+fmRQ1z1e22ywQ7K56dDoJXOXZ8VjvCQFLabohRMHj7udjtY9cYo1+RNcGFyaLCBwMXnvX3JxtMiwiXT+thDBu5RWLahWjV+rTOgYk0QKyVmYnCLbieI0iXsYlsD6F+HZFBXcSFcZOFX8HiGSd2h5XC3LYCjf4FGFcOk/aKMxfjMqqG1kLyaLuE5HLvoQYtaGlLM6xKXn+V/msjhVuGaY5T12OLcd3yRHWTrJe0JihIQY9W00GW1u1VOnI0n47nUTH9qUHNHp5AcNqgoZ27OfN6KUHYXf9NGqyMd7js1ta+BYC0JGpDOcukufFFWJaHPzre7y5A8M8FHjHD8HC/wCbDCKA+WwQocuzy2ZOqN72AchWLGEXz301LQYOqLtic3PbsbIYSidwRpsh03sCPepwvCx8WEOIe3/y8DM3TYb7Chv1Gj89oV0t8XiLFPa8N2E/9XOma8bDaXwBjLJ3Fj8Gxcf6xl3VXHIqaxuvuzB qH2txeWv VtDSa69rvZIw731Bzb5scI2AVOhj/qKktyzAjThGHzHkvQlmBszc+NRuf0CiwoJ3JHlhTIbCYmnzqrYuUY/c6m0sav69+kMSfDRBmFiaxjhQM8tOYFEiV/luqLNp+9WUnYPaJ0WbUqzvIYoc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.009653, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Mar 11, 2025 at 08:09:02PM -0400, Gregory Price wrote: > ----------------------- > Combination Interleave. > ----------------------- > Lets consider now a system where 2 Host Bridges have 2 CXL devices each, > and we want to interleave the entire set. This requires us to make use > of both inter and intra host bridge interleave. > > First, we can interleave this with the a single CEDT entry, the same as > the first inter-host-bridge CEDT (now assuming 1GB per device). > > ``` > Subtable Type : 01 [CXL Fixed Memory Window Structure] > Reserved : 00 > Length : 002C > Reserved : 00000000 > Window base address : 0000000300000000 <- Memory Region > Window size : 0000000100000000 <- 4GB > Interleave Members (2^n) : 01 <- 2-way interleave > Interleave Arithmetic : 00 > Reserved : 0000 > Granularity : 00000000 > Restrictions : 0006 <- Bit(2) - Volatile > QtgId : 0001 > First Target : 00000007 <- Host Bridge _UID > Next Target : 00000006 <- Host Bridge _UID > ``` > > This gives us a NUMA structure as follows: > ``` > NUMA Structure: > > ---------- -------- | ---------- > | cpu0 |-----| DRAM |----|---| Node 0 | > ---------- -------- | ---------- > / \ | > ------- ------- | ---------- > | HB0 |-----| HB1 |-------------|---| Node 1 | > ------- ------- | ---------- > / \ / \ | > CXL0 CXL1 CXL2 CXL3 | > ``` > > And the respective decoder programming looks as follows > ``` > Decoders: > CXL Root > | > decoder0.0 > IW:2 IG:256 > [0x300000000, 0x3FFFFFFFF] > / \ > Host Bridge 7 Host Bridge 6 > / \ > decoder1.0 decoder2.0 > IW:2 IG:512 IW:2 IG:512 > [0x300000000, 0x3FFFFFFFFF] [0x300000000, 0x3FFFFFFFF] > / \ / \ > endpoint0 endpoint1 endpoint2 endpoint3 > | | | | > decoder3.0 decoder4.0 decoder5.0 decoder6.0 > IW:4 IG:256 IW:4 IG:256 > [0x300000000, 0x3FFFFFFFF] [0x300000000, 0x3FFFFFFFF] > ``` > > Notice at both the root and the host bridge, the Interleave Ways is 2. > There are two targets at each level. The host bridge has a granularity > of 512 to capture its parent's ways and granularity (`2*256`). > > Each decoder is programmed with the total number of targets (4) and the > overall granularity (256B). > Sorry, I tried to set this topology on Qemu Virt and used: "cxl create-region -d decoder0.0 -t ram -m mem0,mem1,mem2,mem3" but it failed with: "cxl region: validate_ways: Interleave ways 2 is less than number of memdevs specified: 4" It seems like the CFMWs IW should be 4? Yuquan