Re: [LSF/MM/BPF TOPIC] memory persistence over kexec

[Mike Rapoport <rppt@kernel.org>]

Hi Pasha,

On Wed, Jan 22, 2025 at 06:30:22PM -0500, Pasha Tatashin wrote:
> > > On Mon, Jan 20, 2025 at 09:54:15AM +0200, Mike Rapoport wrote:
> > > > Hi,
> > > >
> > > > I'd like to discuss memory persistence across kexec.
> > > >
> 
> Hi Mike,
> 
> I'm very interested in this topic and can contribute both presenting
> and implementing changes upstream. We're planning on using KHO in our
> kernel at Google but there are some limitations for our use case that
> I believe can be addressed.
> 
> Limitations:
> 
> 1. Serialization callbacks are called by KHO during the activation
> phase in series. In most cases different device drivers are
> independent, the serialization can be parallelized.
> 
> 2. Once the serialization callbacks are done, the device tree data
> cannot be altered and drivers cannot add more data into the device
> tree (except limited modification where drivers can remember the exact
> node that was created and modify some properties, but that is too
> limited).
> This is bad because we have use cases where we need to save buffered
> data (not memory locations) into the device tree at some late stage
> before jumping to the new kernel.

The device tree data cannot be altered because at kexec load time it is
appended to kexec image and that image cannot be altered without a new
kexec load.
 
> 3. KHO requires devices to be serialized before
> kexec_file_load()/kexec_load(), which means that load becomes part of
> the VM blackout window, if KHO is used for hypervisor live update
> scenarios this is a very bad limitation.

KHO data has to be a part of kexec image and the way kexec works now there
is no way to add anything to kexec image after kexec load.
To be able to serialize the state closer to kexec reboot we'd need to
change the way kexec images are created, regardless of what data format
we use to pass the data between kernels.
 
> 4. KHO activation should not really be needed, there should be two
> phases: old KHO tree passed from the old kernel, and once it is fully
> consumed, new KHO tree that can be updated at any time by devices that
> is going to be passed to the next kernel during next reboot (kexec or
> firmware that is aware of KHO...), instead of activation there should
> be a user driver phase shift from old tree to new tree, once that is
> done drivers can start serialize at will.

If I understand you correctly, it's up driver to decide when to update the
data that should be passed to the new kernel?
Again, for now it's kexec limitation that kexec image cannot be altered
between load and exec. 
Still, it's not clear to me how drivers could decide when they need to do
the updates.

> As David mentioned there is going to be a hypervisor live update
> bi-weekly meeting, where we can discuss this.

Sure :)
 
> Pasha

-- 
Sincerely yours,
Mike.