Re: [LSF/MM/BPF TOPIC] BoF VM live migration over CXL memory​

[Gregory Price <gregory.price@memverge.com>]

On Wed, Apr 12, 2023 at 05:50:55PM +0200, David Hildenbrand wrote:
> 
> long-term: possibly forever, controlled by user space. In practice, anything
> longer than ~10 seconds ( best guess :) ). There can be long-term pinnings
> that are of very short duration, we just don't know what user space is up to
> and when it will decide to unpin.
> 
> Assume user space requests to trigger read/write of a user space page to a
> file: the page is pinned, DMA is started, once DMA completes the page is
> unpinned. Short-term. User space does not control how long the page remains
> pinned.
> 
> In contrast:
> 
> Example #1: mapping VM guest memory into an IOMMU using vfio for PCI
> passthrough requires pinning the pages. Until user space decides to unmap
> the pages from the IOMMU, the pages will remain pinned. -> long-term
> 
> Example #2: mapping a user space address range into an IOMMU to repeatedly
> perform RDMA using that address range requires pinning the pages. Until user
> space decides to unregister that range, the pages remain pinned. ->
> long-term
> 
> Example #3: registering a user space address range with io_uring as a fixed
> buffer, such that io_uring OPS can avoid the page table walks by simply
> using the pinned pages that were looked up once. As long as the fixed buffer
> remains registered, the pages stay pinned. -> long-term
> 
> -- 
> Thanks,
> 
> David / dhildenb
> 

That pretty much precludes live migration from using CXL as a transport
mechanism, since live migration would be a user-initiated process, you
would need what amounts to an atomic move between hosts to ensure pages
are not left pinned.

The more i'm reading the more i'm somewhat convinced CXL memory should
not allow pinning at all.


I suppose you could implement a new RDMA feature where the remote host's
CXL memory is temporarily mapped, data is migrated, and then that area
is unmapped. Basically the exact same RDMA mechanism, but using memory
instead of network. This would make the operation a kernel-controlled
if pin/unpin is required.


Lots to talk about.

~Gregory