Re: [LSF/MM] CXL Boot to Bash - Section 4: Interleave

[Yuquan Wang <wangyuquan1236@phytium.com.cn>]

On Tue, Mar 11, 2025 at 08:09:02PM -0400, Gregory Price wrote:
> 
> -----------------------------
> Intra-Host-Bridge Interleave.
> -----------------------------
> Now lets consider a system where we've placed 2 CXL devices on the same
> Host Bridge.  Maybe each CXL device is only capable of x8 PCIE, and we
> want to make full use of a single x16 link.
> 
> This setup only requires the BIOS to create a CEDT CFMWS which reports
> the entire capacity of all devices under the host bridge, but does not
> need to set up any interleaving.
> 
> In the follow case, the BIOS has configured as single 4GB memory region
> which only targets the single host bridge, but maps the entire memory
> capacity of both devices (2GB).
> 
> ```
> CFMWS:
>           Subtable Type : 01 [CXL Fixed Memory Window Structure]
>                 Reserved : 00
>                   Length : 002C
>                 Reserved : 00000000
>      Window base address : 0000000300000000   <- Memory Region
>              Window size : 0000000080000000   <- 2GB

I think is "Window size : 0000000100000000   <- 4GB" here.

> Interleave Members (2^n) : 00                 <- No host bridge interleave
>    Interleave Arithmetic : 00
>                 Reserved : 0000
>              Granularity : 00000000
>             Restrictions : 0006               <- Bit(2) - Volatile
>                    QtgId : 0001
>             First Target : 00000007           <- Host Bridge _UID
> ```
> 
> Assuming no other CEDT or SRAT entries exist, this will result in linux
> creating the following NUMA topology, where all CXL memory is in Node 1.
> 
> ```
> NUMA Structure:
>         ---------     --------   |    ----------
>         | cpu0  |-----| DRAM |---|----| Node 0 |
>         ---------     --------   |    ----------
>             |                    |
>          -------                 |    ----------
>          | HB0 |-----------------|----| Node 1 |
>          -------                 |    ----------
>         /       \                |
>    CXL Dev     CXL Dev           |
> ```
> 
> In this scenario, we program the decoders like so:
> ```
> Decoders
>                            CXL Root
>                               |
>                           decoder0.0
>                          IW:1  IG:256
>                   [0x300000000, 0x3FFFFFFFF]
>                               |
>                           Host Bridge
>                               |
>                           decoder1.0
>                          IW:2  IG:256
>                    [0x300000000, 0x3FFFFFFFF]
>                              /   \
>                    Endpoint 0     Endpoint 1
>                        |              |
>                    decoder2.0     decoder3.0
>                  IW:2  IG:256     IW:2  IG:256
>     [0x300000000, 0x3FFFFFFFF]    [0x300000000, 0x3FFFFFFFF]
> ```
> 
> The root decoder in this scenario does not participate in interleave,
> it simply forwards all accesses in this range to the host bridge.
> 
> The host bridge then applies the interleave across its connected devices
> and the decodes apply translation accordingly.
> 
> -----------------------
> Combination Interleave.
> -----------------------
> Lets consider now a system where 2 Host Bridges have 2 CXL devices each,
> and we want to interleave the entire set.  This requires us to make use
> of both inter and intra host bridge interleave.
> 
> First, we can interleave this with the a single CEDT entry, the same as
> the first inter-host-bridge CEDT (now assuming 1GB per device).
> 
> ```
>            Subtable Type : 01 [CXL Fixed Memory Window Structure]
>                 Reserved : 00
>                   Length : 002C
>                 Reserved : 00000000
>      Window base address : 0000000300000000   <- Memory Region
>              Window size : 0000000100000000   <- 4GB
> Interleave Members (2^n) : 01                 <- 2-way interleave
>    Interleave Arithmetic : 00
>                 Reserved : 0000
>              Granularity : 00000000
>             Restrictions : 0006               <- Bit(2) - Volatile
>                    QtgId : 0001
>             First Target : 00000007           <- Host Bridge _UID
>              Next Target : 00000006           <- Host Bridge _UID
> ```
> 
> This gives us a NUMA structure as follows:
> ```
> NUMA Structure:
> 
>          ----------     --------    |   ----------
>          |  cpu0  |-----| DRAM |----|---| Node 0 |
>          ----------     --------    |   ----------
>         /         \                 |
>     -------     -------             |   ----------
>     | HB0 |-----| HB1 |-------------|---| Node 1 |
>     -------     -------             |   ----------
>       / \         / \               |
>   CXL0   CXL1  CXL2  CXL3           |
> ```
> 
> And the respective decoder programming looks as follows
> ```
> Decoders:
>                              CXL  Root
>                                  |
>                              decoder0.0
>                             IW:2   IG:256
>                       [0x300000000, 0x3FFFFFFFF]
>                              /         \
>                 Host Bridge 7           Host Bridge 6
>                 /                                    \
>            decoder1.0                             decoder2.0
>           IW:2   IG:512                          IW:2   IG:512
>   [0x300000000, 0x3FFFFFFFFF]             [0x300000000, 0x3FFFFFFFF]
>             /    \                                  /    \
>    endpoint0      endpoint1                endpoint2      endpoint3
>       |               |                       |               |
>   decoder3.0      decoder4.0              decoder5.0      decoder6.0
>           IW:4  IG:256                            IW:4  IG:256
>   [0x300000000, 0x3FFFFFFFF]              [0x300000000, 0x3FFFFFFFF]
> ```
> 
> Notice at both the root and the host bridge, the Interleave Ways is 2.
> There are two targets at each level.  The host bridge has a granularity
> of 512 to capture its parent's ways and granularity (`2*256`).
> 
> Each decoder is programmed with the total number of targets (4) and the
> overall granularity (256B).

Is there any relationship between endpoints'decoder setup(IW&IG) and
others decoder?

> 
> We might use this setup if each CXL device is capable of x8 PCIE, and
> we have 2 Host Bridges capable of full x16 - utilizing all bandwidth
> available.
> 
> ---------------------------------------------
> Nuance: Hardware Interleave and Memory Holes.
> ---------------------------------------------
> You may encounter a system which cannot place the entire memory capacity
> into a single contiguous System Physical Address range.  That's ok,
> because we can just use multiple decoders to capture this nuance.
> 
> Most CXL devices allow for multiple decoders.
> 
> This may require an SRAT entry to keep these regions on the same node.
> (Obviously the relies on your platform vendor's BIOS)
> 
> ```
> CFMWS:
>          Subtable Type : 01 [CXL Fixed Memory Window Structure]
>                 Reserved : 00
>                   Length : 002C
>                 Reserved : 00000000
>      Window base address : 0000000300000000   <- Memory Region
>              Window size : 0000000080000000   <- 2GB
> Interleave Members (2^n) : 00                 <- No host bridge interleave
>    Interleave Arithmetic : 00
>                 Reserved : 0000
>              Granularity : 00000000
>             Restrictions : 0006               <- Bit(2) - Volatile
>                    QtgId : 0001
>             First Target : 00000007           <- Host Bridge 7
> 
>          Subtable Type : 01 [CXL Fixed Memory Window Structure]
>                 Reserved : 00
>                   Length : 002C
>                 Reserved : 00000000
>      Window base address : 0000000400000000   <- Memory Region
>              Window size : 0000000080000000   <- 2GB
> Interleave Members (2^n) : 00                 <- No host bridge interleave
>    Interleave Arithmetic : 00
>                 Reserved : 0000
>              Granularity : 00000000
>             Restrictions : 0006               <- Bit(2) - Volatile
>                    QtgId : 0001
>             First Target : 00000007           <- Host Bridge 7
> 
> SRAT:
>         Subtable Type : 01 [Memory Affinity]
>                Length : 28
>      Proximity Domain : 00000001          <- NUMA Node 1
>             Reserved1 : 0000
>          Base Address : 0000000300000000  <- Physical Memory Region
>        Address Length : 0000000080000000  <- first 2GB
> 
>         Subtable Type : 01 [Memory Affinity]
>                Length : 28
>      Proximity Domain : 00000001          <- NUMA Node 1
>             Reserved1 : 0000
>          Base Address : 0000000400000000  <- Physical Memory Region
>        Address Length : 0000000080000000  <- second 2GB
> ```
> 
> The SRAT entries allow us to keep the regions attached to the same node.
> ```
> 
> NUMA Structure:
>         ---------     --------   |    ----------
>         | cpu0  |-----| DRAM |---|----| Node 0 |
>         ---------     --------   |    ----------
>             |                    |
>          -------                 |    ----------
>          | HB0 |-----------------|----| Node 1 |
>          -------                 |    ----------
>         /       \                |
>    CXL Dev     CXL Dev           |
> ```
>
Hi, Gregory

Seeing this, I have an assumption to discuss.

If the same system uses tables like below:

CFMWS:
         Subtable Type : 01 [CXL Fixed Memory Window Structure]
                Reserved : 00
                  Length : 002C
                Reserved : 00000000
     Window base address : 0000000300000000   <- Memory Region
             Window size : 0000000080000000   <- 2GB
Interleave Members (2^n) : 00                 <- No host bridge interleave
   Interleave Arithmetic : 00
                Reserved : 0000
             Granularity : 00000000
            Restrictions : 0006               <- Bit(2) - Volatile
                   QtgId : 0001
            First Target : 00000007           <- Host Bridge 7

         Subtable Type : 01 [CXL Fixed Memory Window Structure]
                Reserved : 00
                  Length : 002C
                Reserved : 00000000
     Window base address : 0000000400000000   <- Memory Region
             Window size : 0000000080000000   <- 2GB
Interleave Members (2^n) : 00                 <- No host bridge interleave
   Interleave Arithmetic : 00
                Reserved : 0000
             Granularity : 00000000
            Restrictions : 0006               <- Bit(2) - Volatile
                   QtgId : 0001
            First Target : 00000007           <- Host Bridge 7

SRAT:
        Subtable Type : 01 [Memory Affinity]
               Length : 28
     Proximity Domain : 00000000          <- NUMA Node 0
            Reserved1 : 0000
         Base Address : 0000000300000000  <- Physical Memory Region
       Address Length : 0000000080000000  <- first 2GB

        Subtable Type : 01 [Memory Affinity]
               Length : 28
     Proximity Domain : 00000001          <- NUMA Node 1
            Reserved1 : 0000
         Base Address : 0000000400000000  <- Physical Memory Region
       Address Length : 0000000080000000  <- second 2GB


The first 2GB cxl memory region would locate at node0 with DRAM.

NUMA Structure:

        ---------     --------   |            ----------
        | cpu0  |-----| DRAM |---|------------| Node 0 |
        ---------     --------   |   /        ----------
            |                    |  /first 2GB
         -------                 | /          ----------
         | HB0 |-----------------|------------| Node 1 |
         -------                 |second 2GB  ----------
        /       \                |
   CXL Dev     CXL Dev           |
```

Is above configuration and structure valid?

Yuquan
> And the decoder programming would look like so
> ```
> Decoders:
>                                CXL  Root
>                              /           \
>                     decoder0.0           decoder0.1
>                   IW:1  IG:256           IW:1  IG:256
>     [0x300000000, 0x37FFFFFFF]           [0x400000000, 0x47FFFFFFF]
>                              \           /
>                               Host Bridge
>                              /           \
>                     decoder1.0           decoder1.1
>                   IW:2  IG:256           IW:2  IG:256
>     [0x300000000, 0x37FFFFFFF]           [0x400000000, 0x47FFFFFFF]
>               /   \                                /   \
>     Endpoint 0     Endpoint 1            Endpoint 0     Endpoint 1
>         |              |                     |              |
>     decoder2.0     decoder3.0            decoder2.1     decoder3.1
>             IW:2 IG:256                          IW:2 IG:256
>     [0x300000000, 0x37FFFFFFF]           [0x400000000, 0x47FFFFFFF]
> ```
> 
> Linux manages decoders in relation to the associated component, so
> decoders are N.M where N is the component and M is the decoder number.
> 
> If you look, you'll see each side of this tree looks individually
> equivalent to the intra-host-bridge interleave example, just with one
> half of the total memory each (matching the CFMWS ranges).
> 
> Each of the root decoders still has an interleave width of 1 because
> they both only target one host bridge (despite it being the same one).
> 
> 
> --------------------------------
> Software Interleave (Mempolicy).
> --------------------------------
> Linux provides a software mechanism to allow tasks to to interleave its
> memory across NUMA nodes - which may have different performance
> characteristics.  This component is called `mempolicy`, and is primarily
> operated on with the `set_mempolicy()` and `mbind()` syscalls.
> 
> These syscalls take a nodemask (bitmask representing NUMA node ids) as
> an argument to describe the intended allocation policy of the task.
> 
> The following policies are presently supported (as of v6.13)
> ```
> enum {
>         MPOL_DEFAULT,
>         MPOL_PREFERRED,
>         MPOL_BIND,
>         MPOL_INTERLEAVE,
>         MPOL_LOCAL,
>         MPOL_PREFERRED_MANY,
>         MPOL_WEIGHTED_INTERLEAVE,
> };
> ```
> 
> Let's look at `MPOL_INTERLEAVE` and `MPOL_WEIGHTED_INTERLEAVE`.
> 
> To quote the man page:
> ```
> MPOL_INTERLEAVE
>     This  mode  interleaves  page  allocations  across the nodes specified
>     in nodemask in numeric node ID order.  This optimizes for bandwidth
>     instead of latency by spreading out pages and memory accesses to those
>     pages across multiple nodes.  However, accesses to a single page will
>     still be limited to the memory bandwidth of a single node.
> 
> MPOL_WEIGHTED_INTERLEAVE (since Linux 6.9)
>     This mode interleaves page allocations across the nodes specified in
>     nodemask according to the weights in
>         /sys/kernel/mm/mempolicy/weighted_interleave
>     For example, if bits 0, 2, and 5 are set in nodemask and the contents of
>         /sys/kernel/mm/mempolicy/weighted_interleave/node0
>         /sys/ ... /node2
>         /sys/ ... /node5
>     are 4, 7, and 9, respectively, then pages in this region will be
>     allocated on nodes 0, 2, and 5 in a 4:7:9 ratio.
> ```
> 
> To put it simply, MPOL_INTERLEAVE will interleave allocations at a page
> granularity (4KB, 2MB, etc) across nodes in a 1:1 ratio, while
> MPOL_WEIGHTED_INTERLEAVE takes into account weights - which presumably
> map to the bandwidth of each respective node.
> 
> Or more concretely:
> 
> MPOL_INTERLEAVE
>     1:1 Interleave between two nodes.
>     malloc(4096)  ->  node0
>     malloc(4096)  ->  node1
>     malloc(4096)  ->  node0
>     malloc(4096)  ->  node1
>     ... and so on ...
> 
> MPOL_WEIGHTED_INTERLEAVE
>     2:1 Interleave between two nodes.
>     malloc(4096)  ->  node0
>     malloc(4096)  ->  node0
>     malloc(4096)  ->  node1
>     malloc(4096)  ->  node0
>     malloc(4096)  ->  node0
>     malloc(4096)  ->  node1
>     ... and so on ...
> 
> This is the preferred mechanism for *heterogeneous interleave* on Linux,
> as it allows for predictable performance based on the explicit (and
> visible) placement of memory.
> 
> It also allows for memory ZONE restrictions to enable better performance
> predictability (e.g. keeping kernel locks out of CXL while allowing
> workloads to leverage it for expansion or bandwidth).
> 
> ======================
> Mempolicy Limitations.
> ======================
> Mempolicy is a *per-task* allocation policy that is inherited by
> child-tasks on clone/fork. It can only be changed by the task itself,
> though cgroups may affect the effective nodemask via cpusets.
> 
> This means once a task has been launched, and external actor cannot
> change the policy of a running task - except possibly by migrating that
> task between cgroups or changing the cpusets.mems value of the cgroup
> the task lives in.
> 
> Additionally, If capacity on a given node is not available, allocations
> will fall back to another node in the node mask - which may cause
> interleave to become unbalanced.
> 
> ================================
> Hardware Interleave Limitations.
> ================================
> Granularities:
>    granularities are limited on hardware
>    (typically 256B up to 16KB by power of 2)
> 
> Ways:
>    Ways are limited by the CXL configuration to:
>    2,4,8,16,3,6,12
> 
> Balance:
>    Linux does not allow imbalanced interleave configurations
>    (e.g. 3-way interleave where 2 targets are on 1 HB and 1 on another)
> 
> Depending on your platform vendor and type of interleave, you may not
> be able to deconstruct an interleave region at all (decoders may be
> locked).  In this case, you may not have the flexiblity to convert
> operation from interleaved to non-interleave via the driver interface.
> 
> In the scenario where your interleave configuration is entirely driver
> managed, you cannot adjust the size of an interleave set without
> deconstructing the entire set.
> 
> ------------------------------------------------------------------------
> 
> Next we'll discuss how memory allocations occur in a CXL-enabled system,
> which may be affected by things like Reclaim and Tiering systems.
> 
> ~Gregory