Re: [LSF/MM] CXL Boot to Bash - Section 4: Interleave

[Gregory Price <gourry@gourry.net>]

On Thu, Mar 13, 2025 at 04:31:31PM +0800, Yuquan Wang wrote:
> On Tue, Mar 11, 2025 at 08:09:02PM -0400, Gregory Price wrote:
> >              Window size : 0000000080000000   <- 2GB
> 
> I think is "Window size : 0000000100000000   <- 4GB" here.
>

Quite right. I am planning to migrate this all to a github somewhere
after LSF for edits, so i'll take all the feedback and incorporate it
then.

> > There are two targets at each level.  The host bridge has a granularity
> > of 512 to capture its parent's ways and granularity (`2*256`).
> > 
> > Each decoder is programmed with the total number of targets (4) and the
> > overall granularity (256B).
> 
> Is there any relationship between endpoints'decoder setup(IW&IG) and
> others decoder?
>

I'm sure there's a mathematical relationship that dictates this up the
heirarchy, but each endpoint decoder needs to be programmed with the
same interleave weight and granularity of all other endpoints.

Technically unbalanced configurations are possible, but Linux does not
support them.

> Hi, Gregory
> 
> Seeing this, I have an assumption to discuss.
> 
> If the same system uses tables like below:
> 
> CFMWS:
>          Subtable Type : 01 [CXL Fixed Memory Window Structure]
>      Window base address : 0000000300000000   <- Memory Region
>              Window size : 0000000080000000   <- 2GB
>             First Target : 00000007           <- Host Bridge 7
> 
>          Subtable Type : 01 [CXL Fixed Memory Window Structure]
>      Window base address : 0000000400000000   <- Memory Region
>              Window size : 0000000080000000   <- 2GB
>             First Target : 00000007           <- Host Bridge 7
> 
> SRAT:
>         Subtable Type : 01 [Memory Affinity]
>      Proximity Domain : 00000000          <- NUMA Node 0
>          Base Address : 0000000300000000  <- Physical Memory Region
> 
>         Subtable Type : 01 [Memory Affinity]
>      Proximity Domain : 00000001          <- NUMA Node 1
>          Base Address : 0000000400000000  <- Physical Memory Region
> 
> 
> The first 2GB cxl memory region would locate at node0 with DRAM.
> 
> NUMA Structure:
> 
>         ---------     --------   |            ----------
>         | cpu0  |-----| DRAM |---|------------| Node 0 |
>         ---------     --------   |   /        ----------
>             |                    |  /first 2GB
>          -------                 | /          ----------
>          | HB0 |-----------------|------------| Node 1 |
>          -------                 |second 2GB  ----------
>         /       \                |
>    CXL Dev     CXL Dev           |
> ```
> 
> Is above configuration and structure valid?
>

This is correct, the association between memory and numa node is pretty
much purely logical.

I'm not sure WHY you'd want to do this, but yeah you can do this
(assuming you can get the BIOS to produce that SRAT).

~Gregory