Re: Should bios always mark CXL DRAM as EFI_MEMORY_SP?

Re: Should bios always mark CXL DRAM as EFI_MEMORY_SP?

[Dan Williams]

[ add linux-mm since my opinion is not the only one that matters here ]

Responses inline below with only my Linux kernel developer hat on,
i.e. not necessarily the view of $current_employer:

On Thu, Jan 27, 2022 at 8:18 AM John Groves <john@jagalactic.com> wrote:
>
> I’d like to seek some feedback and see whether a consensus exists or can be developed regarding how system firmware (bios/efi/etc) should present CXL DRAM to a system in a pre-fabric world (CXL 1.1/2.0).
>
>
>
> The CXL spec, along with the Intel documentation are pretty specific and useful, but one open issue seems not to be outright specified: should the system mark CXL-attached DRAM as “specific purpose” (EFI_MEMORY_SP)? Consistency across platforms is certainly desirable. If this behavior is not prescribed, we could end up with inconsistent behavior across server and bios vendors.
>
>
>
> If this is already specified, no need to read on (but please point me to where it’s specified).

There is no specification for how an OS handles EFI_MEMORY_SP.
Everything below is only a Linux perspective and likely any other OS
you ask will give a different perspective.

> Objective: I think everyone will likely agree that it should be possible to use CXL DRAM as either general-purpose memory, or via DAX, or a mix.
[..]
>
> ·      Currently cannot be online-converted to DAX-managed (can this work? Is it intended to be working?)
>

There is no guaranteed way to un-online memory, especially ZONE_NORMAL
memory. There are heuristics to make it fail less often, but in
general it's not reliable so online-conversion to DAX-managed is not
being attempted for the general case.

> If online conversion from general-purpose to DAX is not going to work, it seems that the default should preserve the ability to use it either way: mark the memory as EFI_MEMORY_SP.

Yes, unfortunately that requires a paradigm shift for end users to
make a policy decision about memory where they did not need to make
one before. My hope is that distributions would set a default daxctl
policy to just online soft-reserved (Linux term for EFI_MEMORY_SP)
memory. That way savvy users have a control point to change the policy
to varying degrees of exclusive access through a DAX-device instance /
instances, and other users, that don't even know what EFI_MEMORY_SP
is, will see just another NUMA node by default.

> Is there a right and wrong answer re:EFI_MEMORY_SP? How important is it to have consistency across platforms?

The Principle of Least Surprise applies, and the vast bulk of users
simply don't know that they need to care about memory types and memory
performance classes. The ones that do know and care are also likely
the ones to be surprised if they can not guarantee 100% exclusive
access, i.e. machines purpose built to run a workload where the
application gets 100% of the high performance memory.

The distro gets to decide the CONFIG_EFI_SOFT_RESERVE policy, and if
it chooses CONFIG_EFI_SOFT_RESERVE=y I think it should go further to
ship daxctl and a policy that onlines it by default.

https://github.com/pmem/ndctl/blob/main/Documentation/daxctl/daxctl-reconfigure-device.txt#L244

> If there is a consensus, the next question is who should express it. Perhaps the CXL consortium. I’m a part of that, but it seemed like the Linux dev community was the right place to start.

EFI_MEMORY_SP is defined as a hint, so to me that effectively kicks
all the policy questions over to OS specific / Distro specific
solution space.