Re: [PATCH] cxl: Update Soft Reserved resources upon region creation

[Gregory Price <gourry@gourry.net>]

On Thu, Dec 12, 2024 at 05:01:53PM -0800, Alison Schofield wrote:
> BIOS labels a resource Soft Reserved and programs a region using
> that range. Later, the existing cxl path to destroy that region
> does not free up that Soft Reserved range. Users cannot create
> another region in it's place. Resource lost. We considered simply
> removing soft reserved resources on region teardown, and you can
> probably find a patches on lore doing just that.
> 
> But - the problem grew. Sometimes BIOS creates an SR that is not
> aligned with the region they go on to program. Stranded resources.
> That's where the trim and give to DAX path originated.
> 
> But - the problem grew. Sometimes the CXL driver fails to enumerate
> that BIOS defined region. More stranded resources. Let's find those
> too and give them to DAX. This is something we are seeing in the
> wild now and why Dan raised its priority.
> 

Hm, this makes me concerned for what happens on "full hotplug" (literal
physical removal/addition) of CXL devices - kind of like we've seen
proposed with E3.S form factor devices from a variety of vendors.

Like what happens in the following scenario (rhetorical question, I want
to test this with QEMU - but i'm on a plane right now and want to get
the experiment process down).

Boot: No CXL device is present

Post-boot: CXL device is physically hot-plugged
 - there won't be a resource registered, so I would presume the ACPI
   / EFI / CXL drivers would register one.

Event 1: CXL device is shutdown and removed
   - Is the resource deleted?  I would presume yes.
   - Is this true if the CXL device *was* present at boot time?

     If i'm following correctly ^ this is the present scenario? 

Lets assume the device was present at boot, and the resource is not
deleted.  Now we have a "stale resource"?

Event 2A: A new CXL device is added
   - Possibility 1: Same capacity - resource is reused?
   - Possibility 2: Lower capacity - resource is chopped up?
   - Possibility 3: Higher capacity - resource is... lost forever?
                    Fails to map? ???

Event 2B: A new CXL device is added on a different PCI dev id, then
          Event 2A occurs.
   - Is the "stale resource" reused here, or is a new one created?

I hadn't really considered the impact of hotplug on the iomem resource
blocks (soft) reserved at boot, but this is concerning.

I remember ~1.5 years ago I was prototyping with hotplug behavior in
QEMU and saw that it was possible to do runtime ACPI/PCI add/remove of
CXL devices - this worked.  But I didn't look at the effects on iomem
resources - now i'm wondering what happens if I try to hot-unplug a CXL
device that was present at boot.

This won't affect me for the immediate future, but if we're mucking
around in this space, might as well ask the question.  I presume we'll
find even worse corner cases here :D :| :[ :<

I do know servers with front-facing E3.S CXL devices intended for
hot-replace exist and are a real use-case. I have no idea how that
is supposed to work the presence of stale iomem resources.

> Dan is also suggesting that at that last event - failure to enumerate
> a BIOS defined region, we tear down the entire ACPI0017 toplogy
> and give everything to DAX.
> 
> What Dan called, "the minimum requirement": all Soft Reserved ranges
> end up as dax-devices sounds like the right guideline moving forward.
>

I guess devils in the details here.  I sense an implication that it's
possible for two distinct pieces of SR-providing hardware (HBM and CXL)
could end up concatonated into a single SR range? That would obviously
necessitate the need for chopping up an SR.  So this all makes sense.

But I don't disagree with the need for this, just concerned that we have
CXL-specific logic landing in mm/ and e820 code.

~Gregory