Re: [PATCH v4 13/15] KVM: guest_memfd: implement userfaultfd operations

[Nikita Kalyazin <kalyazin@amazon.com>]

On 02/04/2026 23:05, Sean Christopherson wrote:
> On Thu, Apr 02, 2026, Mike Rapoport wrote:
>> From: Nikita Kalyazin <kalyazin@amazon.com>
>> 
>> userfaultfd notifications about page faults used for live
>> migration and snapshotting of VMs.
>> 
>> MISSING mode allows post-copy live migration and MINOR mode allows 
>> optimization for post-copy live migration for VMs backed with
>> shared hugetlbfs or tmpfs mappings as described in detail in
>> commit 7677f7fd8be7 ("userfaultfd: add minor fault registration
>> mode").
>> 
>> To use the same mechanisms for VMs that use guest_memfd to map
>> their memory, guest_memfd should support userfaultfd operations.
>> 
>> Add implementation of vm_uffd_ops to guest_memfd.
>> 
>> Signed-off-by: Nikita Kalyazin <kalyazin@amazon.com> Co-developed-
>> by: Mike Rapoport (Microsoft) <rppt@kernel.org> Signed-off-by:
>> Mike Rapoport (Microsoft) <rppt@kernel.org> --- mm/
>> filemap.c           |  1 + virt/kvm/guest_memfd.c | 84 ++++++++++++
>> +++++++++++++++++++++++++++++- 2 files changed, 83 insertions(+),
>> 2 deletions(-)
>> 
>> diff --git a/mm/filemap.c b/mm/filemap.c index
>> 406cef06b684..a91582293118 100644 --- a/mm/filemap.c +++ b/mm/
>> filemap.c @@ -262,6 +262,7 @@ void filemap_remove_folio(struct
>> folio *folio)
>> 
>> filemap_free_folio(mapping, folio); } 
>> +EXPORT_SYMBOL_FOR_MODULES(filemap_remove_folio, "kvm");
> 
> This can be EXPORT_SYMBOL_FOR_KVM so that the symbol is exported if
> and only if KVM is built as a module.
> 
>> /* * page_cache_delete_batch - delete several folios from page
>> cache diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c 
>> index 017d84a7adf3..46582feeed75 100644 --- a/virt/kvm/
>> guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -7,6 +7,7 @@ 
>> #include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include
>> <linux/pagemap.h> +#include <linux/userfaultfd_k.h>
>> 
>> #include "kvm_mm.h"
>> 
>> @@ -107,6 +108,12 @@ static int kvm_gmem_prepare_folio(struct kvm
>> *kvm, struct kvm_memory_slot *slot, return
>> __kvm_gmem_prepare_folio(kvm, slot, index, folio); }
>> 
>> +static struct folio *kvm_gmem_get_folio_noalloc(struct inode
>> *inode, pgoff_t pgoff) +{ +     return __filemap_get_folio(inode-
>> >i_mapping, pgoff, +                                FGP_LOCK |
>> FGP_ACCESSED, 0);
> 
> Note, this will conflict with commit 6dad5447c7bf ("KVM:
> guest_memfd: Don't set FGP_ACCESSED when getting folios") sitting in
> 
> https://github.com/kvm-x86/linux.git gmem
> 
> I think the resolution is to just end up with:
> 
> static struct folio *kvm_gmem_get_folio_noalloc(struct inode *inode,
> pgoff_t pgoff) { return filemap_lock_folio(inode->i_mapping, pgoff); 
> }
> 
> However, I think that'll be a moot point in the end (the conflict
> will be avoided). More below.
> 
>> +} + /* * Returns a locked folio on success.  The caller is
>> responsible for * setting the up-to-date flag before the memory is
>> mapped into the guest. @@ -126,8 +133,7 @@ static struct folio
>> *kvm_gmem_get_folio(struct inode *inode, pgoff_t index) * Fast-
>> path: See if folio is already present in mapping to avoid *
>> policy_lookup. */ -     folio = __filemap_get_folio(inode-
>> >i_mapping, index, -                                 FGP_LOCK |
>> FGP_ACCESSED, 0); +     folio = kvm_gmem_get_folio_noalloc(inode,
>> index); if (!IS_ERR(folio)) return folio;
>> 
>> @@ -457,12 +463,86 @@ static struct mempolicy
>> *kvm_gmem_get_policy(struct vm_area_struct *vma, } #endif /*
>> CONFIG_NUMA */
>> 
>> +#ifdef CONFIG_USERFAULTFD +static bool
>> kvm_gmem_can_userfault(struct vm_area_struct *vma, vm_flags_t
>> vm_flags) +{ +     struct inode *inode = file_inode(vma->vm_file); 
>> + +     /* +      * Only support userfaultfd for guest_memfd with
>> INIT_SHARED flag. +      * This ensures the memory can be mapped
>> to userspace. +      */ +     if (!(GMEM_I(inode)->flags &
>> GUEST_MEMFD_FLAG_INIT_SHARED)) +             return false;
> 
> I'm not comfortable with this change.  It works for now, but it's
> going to be wildly wrong when in-place conversion comes along.
> While I agree with the "Let's solve each problem in it's
> time :)"[*], the time for in-place conversion is now. In-place
> conversion isn't landing this cycle or next, but it's been in
> development for longer than UFFD support, and I'm not willing to
> punt solvable problems to that series, because it's plenty fat as
> is.
> 
> Happily, IIUC, this is an easy problem to solve, and will have a
> nice side effect for the common UFFD code.
> 
> My objection to an early, global "can_userfault()" check is that
> it's guaranteed to cause TOCTOU issues.  E.g. for VM_UFFD_MISSING
> and VM_UFFD_MINOR, the check on whether or not a given address can
> be faulted in needs to happen in __do_userfault(), not broadly when
> VM_UFFD_MINOR is added to a VMA.  Conceptually, that also better 
> aligns the code with the "normal" user fault path in
> kvm_gmem_fault_user_mapping().
> 
> I'm definitely not asking to fully prep for in-place conversion, I
> just want to set us up for success and also to not have to churn a
> pile of code.  Concretely, again IIUC, I think we just need to move
> the INIT_SHARED check to ->alloc_folio() and ->get_folio_noalloc().
> And if we extract kvm_gmem_is_shared_mem() now instead of waiting
> for in-place conversion, then we'll avoid a small amount of churn
> when in-place conversion comes along.
> 
> The bonus side effect is that dropping guest_memfd's more "complex" 
> can_userfault means the only remaining check is constant based on
> the backing memory vs. the UFFD flags.  If we want, the indirect
> call to a function can be replace with a constant vm_flags_t
> variable that enumerates the supported (or unsupported if we're
> feeling negative) flags, e.g.

Thanks Sean.  Checking for GUEST_MEMFD_FLAG_INIT_SHARED at the time of
use and adding uffd_ prefixes to the callbacks make sense to me.  I
tested your changes in my local setup and they are functional with minor 
tweaks:
  - remove the no-longer-used anon_can_userfault
  - fix for the vm_flags check


diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 100aeadd7180..df91c40c6281 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -32,14 +32,6 @@ struct mfill_state {
  	pmd_t *pmd;
  };

-static bool anon_can_userfault(struct vm_area_struct *vma, vm_flags_t 
vm_flags)
-{
-	/* anonymous memory does not support MINOR mode */
-	if (vm_flags & VM_UFFD_MINOR)
-		return false;
-	return true;
-}
-
  static struct folio *anon_alloc_folio(struct vm_area_struct *vma,
  				      unsigned long addr)
  {
@@ -2051,7 +2043,7 @@ bool vma_can_userfault(struct vm_area_struct *vma, 
vm_flags_t vm_flags,
  	    !ops->get_folio_noalloc)
  		return false;

-	return ops->supported_uffd_flags & vm_flags;
+	return (ops->supported_uffd_flags & vm_flags) == vm_flags;
  }

  static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,

> 
> diff --git a/include/linux/userfaultfd_k.h b/include/linux/
> userfaultfd_k.h index 6f33307c2780..8a2d0625ffa3 100644 --- a/
> include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@
> -82,8 +82,8 @@ extern vm_fault_t handle_userfault(struct vm_fault
> *vmf, unsigned long reason);
> 
> /* VMA userfaultfd operations */ struct vm_uffd_ops { -       /*
> Checks if a VMA can support userfaultfd */ -       bool
> (*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags); 
> +       /* What UFFD flags/modes are supported. */ +       const
> vm_flags_t supported_uffd_flags; /* * Called to resolve
> UFFDIO_CONTINUE request. * Should return the folio found at pgoff in
> the VMA's pagecache if it
> 
> with usage like:
> 
> static const struct vm_uffd_ops shmem_uffd_ops = { 
> .supported_uffd_flags   = __VM_UFFD_FLAGS, .get_folio_noalloc      =
> shmem_get_folio_noalloc, .alloc_folio            =
> shmem_mfill_folio_alloc, .filemap_add            =
> shmem_mfill_filemap_add, .filemap_remove         =
> shmem_mfill_filemap_remove, };
> 
> [*] https://lore.kernel.org/all/acZuW7_7yBdVsJqK@kernel.org
> 
>> +     return true; +}
> 
> ...
> 
>> +static const struct vm_uffd_ops kvm_gmem_uffd_ops = { 
>> +     .can_userfault     = kvm_gmem_can_userfault, 
>> +     .get_folio_noalloc = kvm_gmem_get_folio_noalloc, 
>> +     .alloc_folio       = kvm_gmem_folio_alloc, 
>> +     .filemap_add       = kvm_gmem_filemap_add, 
>> +     .filemap_remove    = kvm_gmem_filemap_remove,
> 
> Please use kvm_gmem_uffd_xxx().  The names are a bit verbose, but
> these are waaay to generic of names as-is, e.g.
> kvm_gmem_folio_alloc() has implications and restrictions far beyond
> just allocating a folio.
> 
> All in all, somelike like so (completely untested):
> 
> --- include/linux/userfaultfd_k.h |  4 +- mm/
> filemap.c                  |  1 + mm/hugetlb.c                  |  8
> +--- mm/shmem.c                    |  7 +-- mm/
> userfaultfd.c              |  6 +-- virt/kvm/guest_memfd.c        |
> 80 ++++++++++++++++++++++++++++++++++- 6 files changed, 87
> insertions(+), 19 deletions(-)
> 
> diff --git a/include/linux/userfaultfd_k.h b/include/linux/
> userfaultfd_k.h index 6f33307c2780..8a2d0625ffa3 100644 --- a/
> include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@
> -82,8 +82,8 @@ extern vm_fault_t handle_userfault(struct vm_fault
> *vmf, unsigned long reason);
> 
> /* VMA userfaultfd operations */ struct vm_uffd_ops { -       /*
> Checks if a VMA can support userfaultfd */ -       bool
> (*can_userfault)(struct vm_area_struct *vma, vm_flags_t vm_flags); 
> +       /* What UFFD flags/modes are supported. */ +       const
> vm_flags_t supported_uffd_flags; /* * Called to resolve
> UFFDIO_CONTINUE request. * Should return the folio found at pgoff in
> the VMA's pagecache if it diff --git a/mm/filemap.c b/mm/filemap.c 
> index 6cd7974d4ada..19dfcebcd23f 100644 --- a/mm/filemap.c +++ b/mm/
> filemap.c @@ -262,6 +262,7 @@ void filemap_remove_folio(struct folio
> *folio)
> 
> filemap_free_folio(mapping, folio); } 
> +EXPORT_SYMBOL_FOR_MODULES(filemap_remove_folio, "kvm");
> 
> /* * page_cache_delete_batch - delete several folios from page cache 
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c index
> 077968a8a69a..f55857961adb 100644 --- a/mm/hugetlb.c +++ b/mm/
> hugetlb.c @@ -4819,14 +4819,8 @@ static vm_fault_t
> hugetlb_vm_op_fault(struct vm_fault *vmf) }
> 
> #ifdef CONFIG_USERFAULTFD -static bool hugetlb_can_userfault(struct
> vm_area_struct *vma, -                                 vm_flags_t
> vm_flags) -{ -       return true; -} - static const struct
> vm_uffd_ops hugetlb_uffd_ops = { -       .can_userfault =
> hugetlb_can_userfault, +       .supported_uffd_flags =
> __VM_UFFD_FLAGS, }; #endif
> 
> diff --git a/mm/shmem.c b/mm/shmem.c index
> 239545352cd2..76d8488b9450 100644 --- a/mm/shmem.c +++ b/mm/shmem.c 
> @@ -3250,13 +3250,8 @@ static struct folio
> *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff) return
> folio; }
> 
> -static bool shmem_can_userfault(struct vm_area_struct *vma,
> vm_flags_t vm_flags) -{ -       return true; -} - static const
> struct vm_uffd_ops shmem_uffd_ops = { 
> -       .can_userfault          = shmem_can_userfault, 
> +       .supported_uffd_flags   = __VM_UFFD_FLAGS, 
> .get_folio_noalloc      = shmem_get_folio_noalloc, 
> .alloc_folio            = shmem_mfill_folio_alloc, 
> .filemap_add            = shmem_mfill_filemap_add, diff --git a/mm/
> userfaultfd.c b/mm/userfaultfd.c index 9ba6ec8c0781..ccbd7bb334c2
> 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -58,8 +58,8
> @@ static struct folio *anon_alloc_folio(struct vm_area_struct *vma, 
> }
> 
> static const struct vm_uffd_ops anon_uffd_ops = { 
> -       .can_userfault  = anon_can_userfault, 
> -       .alloc_folio    = anon_alloc_folio, 
> +       .supported_uffd_flags   = __VM_UFFD_FLAGS & ~VM_UFFD_MINOR, 
> +       .alloc_folio            = anon_alloc_folio, };
> 
> static const struct vm_uffd_ops *vma_uffd_ops(struct vm_area_struct
> *vma) @@ -2055,7 +2055,7 @@ bool vma_can_userfault(struct
> vm_area_struct *vma, vm_flags_t vm_flags, !ops->get_folio_noalloc) 
> return false;
> 
> -       return ops->can_userfault(vma, vm_flags); +       return ops-
> >supported_uffd_flags & vm_flags; }
> 
> static void userfaultfd_set_vm_flags(struct vm_area_struct *vma, 
> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index
> 462c5c5cb602..e634bf671d12 100644 --- a/virt/kvm/guest_memfd.c +++ b/
> virt/kvm/guest_memfd.c @@ -7,6 +7,7 @@ #include <linux/mempolicy.h> 
> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include
> <linux/userfaultfd_k.h>
> 
> #include "kvm_mm.h"
> 
> @@ -59,6 +60,11 @@ static pgoff_t kvm_gmem_get_index(struct
> kvm_memory_slot *slot, gfn_t gfn) return gfn - slot->base_gfn + slot-
> >gmem.pgoff; }
> 
> +static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t
> index) +{ +       return GMEM_I(inode)->flags &
> GUEST_MEMFD_FLAG_INIT_SHARED; +} + static int
> __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot
> *slot, pgoff_t index, struct folio *folio) { @@ -396,7 +402,7 @@
> static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) 
> if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode)) return
> VM_FAULT_SIGBUS;
> 
> -       if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)) 
> +       if (!kvm_gmem_is_shared_mem(inode, vmf->pgoff)) return
> VM_FAULT_SIGBUS;
> 
> folio = kvm_gmem_get_folio(inode, vmf->pgoff); @@ -456,12 +462,84 @@
> static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct
> *vma, } #endif /* CONFIG_NUMA */
> 
> +#ifdef CONFIG_USERFAULTFD +static struct folio
> *kvm_gmem_uffd_get_folio_noalloc(struct inode *inode, 
> +                                                    pgoff_t pgoff) 
> +{ +       if (!kvm_gmem_is_shared_mem(inode, pgoff)) 
> +               return NULL; + +       return
> filemap_lock_folio(inode->i_mapping, pgoff); +} + +static struct
> folio *kvm_gmem_uffd_folio_alloc(struct vm_area_struct *vma, 
> +                                              unsigned long addr) 
> +{ +       struct inode *inode = file_inode(vma->vm_file); +
> pgoff_t pgoff = linear_page_index(vma, addr); +       struct
> mempolicy *mpol; +       struct folio *folio; +       gfp_t gfp; + 
> +       if (unlikely(pgoff >= (i_size_read(inode) >> PAGE_SHIFT))) 
> +               return NULL; + +       if (!
> kvm_gmem_is_shared_mem(inode, pgoff)) +               return NULL; + 
> +       gfp = mapping_gfp_mask(inode->i_mapping); +       mpol =
> mpol_shared_policy_lookup(&GMEM_I(inode)->policy, pgoff); +
> mpol = mpol ?: get_task_policy(current); +       folio =
> filemap_alloc_folio(gfp, 0, mpol); +       mpol_cond_put(mpol); + 
> +       return folio; +} + +static int
> kvm_gmem_uffd_filemap_add(struct folio *folio, 
> +                                    struct vm_area_struct *vma, 
> +                                    unsigned long addr) +{ +
> struct inode *inode = file_inode(vma->vm_file); +       struct
> address_space *mapping = inode->i_mapping; +       pgoff_t pgoff =
> linear_page_index(vma, addr); +       int err; + +
> __folio_set_locked(folio); +       err = filemap_add_folio(mapping,
> folio, pgoff, GFP_KERNEL); +       if (err) { +
> folio_unlock(folio); +               return err; +       } + +
> return 0; +} + +static void kvm_gmem_uffd_filemap_remove(struct
> folio *folio, +                                        struct
> vm_area_struct *vma) +{ +       filemap_remove_folio(folio); +
> folio_unlock(folio); +} + +static const struct vm_uffd_ops
> kvm_gmem_uffd_ops = { +       .supported_uffd_flags   =
> __VM_UFFD_FLAGS, +       .get_folio_noalloc      =
> kvm_gmem_uffd_get_folio_noalloc, +       .alloc_folio            =
> kvm_gmem_uffd_folio_alloc, +       .filemap_add            =
> kvm_gmem_uffd_filemap_add, +       .filemap_remove         =
> kvm_gmem_uffd_filemap_remove, +}; +#endif /* CONFIG_USERFAULTFD */ + 
> static const struct vm_operations_struct kvm_gmem_vm_ops = { 
> .fault          = kvm_gmem_fault_user_mapping, #ifdef CONFIG_NUMA 
> .get_policy     = kvm_gmem_get_policy, .set_policy     =
> kvm_gmem_set_policy, #endif +#ifdef CONFIG_USERFAULTFD 
> +       .uffd_ops       = &kvm_gmem_uffd_ops, +#endif };
> 
> static int kvm_gmem_mmap(struct file *file, struct vm_area_struct
> *vma)
> 
> base-commit: d63beb006dba56d5fa219f106c7a97eb128c356f --