Re: [LSF/MM/BPF TOPIC] memory persistence over kexec

[David Rientjes <rientjes@google.com>]

On Mon, 20 Jan 2025, Jason Gunthorpe wrote:

> On Mon, Jan 20, 2025 at 09:54:15AM +0200, Mike Rapoport wrote:
> > Hi,
> > 
> > I'd like to discuss memory persistence across kexec.
> > 
> > Currently there is ongoing work on Kexec HandOver (KHO) [1] that allows
> > serialization and deserialization of kernel data as well as preserving
> > arbitrary memory ranges across kexec.
> > 
> > In addition, KHO keeps a physically contiguous memory regions that are
> > guaranteed to not have any memory that KHO would preserve, but still can be
> > used by the system. The kexeced kernel bootstraps itself using those
> > regions and sets all handed over memory as in use. KHO users then can
> > recover their state from the preserved data. This includes memory
> > reservations, where the user can either discard or claim reservations.
> > 
> > KHO can be used as the base layer for implementation of persistence-aware
> > memory allocator and persistent in-memory filesystem.
> > 
> > Aside from status update on KHO progress there are a few topics that I would
> > like to discuss:
> > * Is it feasible and desirable to enable KHO support in tmpfs and hugetlbfs?

This is a very timely discussion since the last Linux MM Alignment Session 
on the topic since some use cases, at least for tmpfs, have emerged.  Not 
necessarily a requirement, but more out of convenience.

> > * Or is it better to implement yet another in-memory filesystem dedicated
> >   for persistence?
> > * What is the best way to ensure that the memory we want to persist is not
> >   scattered all over the place?
> 
> There is alot of talk about taking *drivers* and having them survive
> kexec, meaning the driver has to put alot of its state into KHO and
> then get it back out again.
> 
> I've been hoping for a model where a driver can be told to "go to KHO"
> and the KHO code can be largely contained in the driver and regulated
> to recording the driver state. This implies the state may be
> fragmented all over memory.
> 

This sounds fantastic if it is doable!

> The other direction is that the driver has to start up in some special
> KHO mode and KHO becomes invasive on all driver paths to use special
> KHO allocations. This seems like a PITA.
> 
> You can see this difference just in the discussion around the iommu
> serialization where one idea was to have KHO be an integral (and
> invasive!) part of the page table operations from time zero vs some
> later serialization at kexec time.
> 
> Regardless, I'm interested in this discussion to bring some
> concreteness about how drivers work..
> 

+1, I'm also interested in this discussion.

As previously mentioned[1], we'll also start a biweekly on hypervisor live 
update to accelerate progress.  The first instance of that meeting will be 
next week, Monday, January 27 at 8am PST (UTC-8).  Calendar invites will 
go out later today for everybody on that email thread, if anybody else is 
interested in attending on a regular basis please email me.  Hoping this 
can be leveraged as well to build up to LSF/MM/BPF.

[1] 
https://lore.kernel.org/kexec/2908e4ab-abc4-ddd0-b191-fe820856cfb4@google.com/T/#u