Re: [RFC v1 1/3] luo: Live Update Orchestrator

[Jason Gunthorpe <jgg@nvidia.com>]

On Thu, Mar 20, 2025 at 03:00:31PM -0400, Pasha Tatashin wrote:

> > I also think we should give up on the sysfs. If fdbox is going forward
> > in a char dev direction then I think we should have two char devs
> > /dev/kho/serialize and /dev/kho/deserialize and run the whole thing
> 
> KHO is a mechanism to preserve kernel memory across reboots. It can be
> used independently of live update, for example, to preserve kexec
> reboot telemetry, traces, and for other purposes. The LUO utilizes KHO
> for memory preservation but also orchestrates specifically a live
> update process, provides a generic way for subsystems and devices to
> participate, handles error recovery, unclaimed devices, and other live
> update-specific steps.
> 
> That said, I can transition the LUO interface from sysfs to a character device.

Sure, I mean pick whatever name makes sense for this whole bundle..

> > through that. The concepts shown in the fdbox patches should be merged
> > into the kho/serialize char dev as just a general architecture of open
> > the char dev, put stuff into it, then finalize and do the kexec.
> 
> Some participating subsystems, such as interrupts, do not have a way
> to export a file descriptor. 

Interrupts that need to be preserved are owned by VFIO. Why do we need
to preserve interrupts? I thought the model was to halt all interrupts
and then re-inject a spurious one?

> It is unclear why we would require this
> for kernel-internal state that needs to be preserved for live update,
> which should instead register with internally.

Because there is almost no kernel state which is machine global and
unconditionally should be included. eg Interrupts for devices that are
not doing preservation should not be serialized. Only userspace knows
what should be preserved so you must always need a mechanism to tell
the kernel.

> IMO, the current API and state machine are quite simple (I plan to
> present and go through them at one of the Hypervisor Live Update
> meetings). However, I am open to changing to a different API, and we
> can expose it through a character device.

Everything seems simple before you actually try to use it :)

> > Also agree with Greg, I think this needs more thoughtful patch staging
> > with actual complete solutions. I think focusing on a progression of
> > demonstrable kexec preservation:
> >  - A simple KVM and the VM's backing memory in a memfd is perserved
> >  - A simple vfio-noiommu doing DMA to a preserved memfd, including not
> >    resetting the device (but with no iommu driver)
> >  - iommufd
> 
> We are working on this. However, each component builds upon the
> previous one, so it makes sense to discuss the lower layers early to
> get early feedback.

I think part of the problem is there are lots of people working on
pieces as though they are seperate components, and I'm not sure this
is entirely wise, or the components are actually seperate.  I see
fdbox and this luo patch series as effectively being the same
component, just different aspects of it.

I'm not entirely sure that Mike's work is actually really
separate. Yes you might use it with a crash kernel too, that mechanism
is going to trigger for a crash kernel scenario without something
triggering the serialization steps. It kind of makes sense to me that
the same uapi could both setup the crash scenario and choose what gets
pass to crash and also support kexec.

Jason