Re: [External] [LSF/MM/BPF TOPIC] CXL Fabric Manager (FM) architecture

[Adam Manzanares <a.manzanares@samsung.com>]

On Wed, Feb 08, 2023 at 10:03:57AM -0800, Viacheslav A.Dubeyko wrote:
> 
> 
> > On Feb 8, 2023, at 8:38 AM, Adam Manzanares <a.manzanares@samsung.com> wrote:
> > 
> > On Thu, Feb 02, 2023 at 09:54:02AM +0000, Jonathan Cameron wrote:
> >> On Wed, 1 Feb 2023 12:04:56 -0800
> >> "Viacheslav A.Dubeyko" <viacheslav.dubeyko@bytedance.com> wrote:
> >> 
> >>>> 
> 
> <skipped>
> 
> >>> 
> >>> Most probably, we will have multiple FM implementations in firmware.
> >>> Yes, FM on host could be important for debug and to verify correctness
> >>> firmware-based implementations. But FM daemon on host could be important
> >>> to receive notifications and react somehow on these events. Also, journalling
> >>> of events/messages/events could be important responsibility of FM daemon
> >>> on host. 
> >> 
> >> I agree with an FM daemon somewhere (potentially running on the BMC type chip
> >> that also has the lower level FM-API access).  I think it is somewhat
> >> separate from the rest of this on basis it may well just be talking redfish
> >> to the FM and there are lots of tools for that sort of handling already.
> >> 
> > 
> > I would be interested in particpating in a BOF about this topic. I wonder what
> > happens when we have multiple switches with multiple FMs each on a separate BMC.
> > In this case, does it make more sense to have an owner of the global FM state 
> > be a user space application. Is this the job of the orchestrator?
> > 
> > The BMC based FM seems to have scalability issues, but will we hit them in
> > practice any time soon.
> 
> I had discussion recently and it looks like there are interesting points:
> (1) If we have multiple CXL switches (especially with complex hierarchy), then it is
> very compute-intensive activity. So, potentially, FM on firmware side could be not
> capable to digest and executes all responsibilities without potential performance
> degradation.
> (2) However, if we have FM on host side, then there is security concerns because
> FM sees everything and all details of multiple hosts and subsystems.
> (3) Technically speaking, there is one potential capability that user-space FM daemon
> can run as on host side as on CXL switch side. I mean here that if we implement
> user-space FM daemon, then it could be used to execute FM functionality on CXL
> switch side (maybe????). :)
> 
> <skipped>
> 
> >>>>>   - Manage surprise removal of devices  
> >>>> 
> >>>> Likewise, beyond reporting I wouldn't expect the FM daemon to have any idea
> >>>> what to do in the way of managing this.  Scream loudly?
> >>>> 
> >>> 
> >>> Maybe, it could require application(s) notification. Let’s imagine that application
> >>> uses some resources from removed device. Maybe, FM can manage kernel-space
> >>> metadata correction and helping to manage application requests to not existing
> >>> entities.
> >> 
> >> Notifications for the host are likely to come via inband means - so type3 driver
> >> handling rather than related to FM.  As far as the host is concerned this is the
> >> same as case where there is no FM and someone ripped a device out.
> >> 
> >> There might indeed be meta data to manage, but doubt it will have anything to
> >> do with kernel.
> >> 
> > 
> > I've also had similar thoughts, I think the OS responds to notifications that
> > are generated in-band after changes to the state of the FM are made through 
> > OOB means.
> > 
> > I envision the host sends REDFISH requests to a switch BMC that has an FM
> > implementation. Once the changes are implemented by the FM it would show up
> > as changes to the PCIe hierarchy on a host, which is capable of responding to
> > such changes.
> > 
> 
> I think I am not completely follow your point. :) First of all, I assume that if host
> sends REDFISH request, then it will be expected the confirmation of request execution.
> It means for me that host needs to receive some packet that informs that request
> executed successfully or failed. It means that some subsystem or application requested
> this change and only after receiving the confirmation requested capabilities can be used.
> And if FM is on CXL switch side, then how FM will show up the changes? It sounds for me
> that some FM subsystem should be on the host side to receive confirmation/notification
> and to execute the real changes in PCIe hierarchy. Am missing something here?

Hopefully I have a point ;). I do expect a host to receive a response  for a
given REDFISH request, but the request/response would be OOB. I would go back
to the example of hot plugging in a PCIe based devices. For example if an nvme
SSD is hot plugged, then the OS notified by HW that a new PCIe device has been
added. Going back to changes made by the FM, if the changes impact the CXL
hiearchy that is visible to a host, it is my expectation that the host OS will
be informed of the changes requested of the FM when the host HW becomes aware
of the changes (the in-band change).

> 
> Thanks,
> Slava.
>