> On Sat, Sep 27, 2025 at 11:45 AM David P. Reed <dpreed@deepplum.com>
> wrote:
> >
> > OK - responses below.
>
> I think Peter will be able to help you the most, but I want to give my
> two cents anyway.
>
> >
> > I'm still unclear what my role is vs. the others cc'ed on this problem report
> is.
> >
> > Is anyone here (other than Andrew) a decision maker on what userfaultfd is
> supposed to do? I can see what the current code DOES - and honestly, it's
> seriously whacked semantically. (see the ExtMem paper for a reasonable use case
> that it cannot serve, my use case is quite similar). So is anyone here wanting to
> improve the functionality? I'm sure its current functions are used by some folks
> here - Google employees presumably focused on ChromeOS or Android, I suppose,
> suggest that there's a use case there.
>
> I think all of us want userfaultfd to be as useful as possible. :)
> Peter, Axel, and I are quite familiar with userfaultfd's use as a tool
> for enabling post-copy live migration for virtual machines.
> Userfaultfd minor faults were created expressly for this purpose. Axel
> wrote the userfaultfd minor fault support; I wrote the corresponding
> userspace code to use it in Google Cloud.

 

Excellent clarification. And congratulations on making that work.

>
> Peter is quite a bit more familiar with userfaultfd than me (and I
> think Axel, but I don't want to speak for him), so please excuse our
> mistakes. (mm is complicated!)
>
> There are a few others who care about userfaultfd who might jump in as
> soon as patches get sent. I think these folks (so on top of Peter and
> Andrew, people like Suren, Lorenzo, David Hildenbrand) will be the
> folks who Ack or Nak the patches.

 

That's good to know.

 

>
> >
> >
> >
> > My role started out by reporting that the documentation is both incomplete
> and confusing, both in the man pages and the "kernel documentation". And the
> rationale presented in the documentation doesn't make sense. Some of you guys
> admit that you really don't understand how "swap" is different from "file-backed
> paging" (except for the corner cases of hugetlbfs [sort of "file backed"],
> "file-backed by /dev/zero" [which ends up using "swap"], and tmpfs [also "file
> backed" but using "swap"]. And yet "anonymous, private" uses "swap" and the "swap
> cache", not the "page cache".
>
> The documentation is confusing; I agreed with you originally that it
> should be updated. (Do you want to send a patch? Perhaps I could give
> it a go when I find the time.)
>
> I spent some time writing out how I define the various terms being
> used here, I'll leave it at the end of this email in case it is
> helpful, but otherwise please just ignore it. I wouldn't say that the
> rationale in the documentation doesn't make sense. Userfaultfd exists
> to solve specific problems.

I thought it was a general purpose interface. My mistake. But I think it can be more general, at least encompassing my goal of having a userspace "interface" that monitors processes' page faults.

>
> >
> > Now, after digging into the question, I feel like there was never, ever a
> coherent architectural design for userfaultfd as a function. It's apparently just
> a "hack", not a "feature".
>
> Userfaultfd certainly isn't perfect, but it is critical for things
> like VM live migration, Android GC, CRIU, etc..
>
> >
> > I'd be happy to propose a much more coherent design (in my opinion as an
> operating systems designer for the past more than 20 years, starting with Multics
> in 1970 - you guys may not be interested in my input, which is fair. Is Linus
> interested? That would be a bunch of work for me, because I would do a thorough
> job, not just a bunch of random patches. But I'm not proposing to join the
> maintainer-club - I'm retired from that space, and I find the Linux kernel
> contributors poorly organized and chaotic.
> >
> > Or, I can just drop this interaction - concluding that userfaultfd is kind of
> useless as is, and really badly documented to boot.
>
> I am interested to hear your ideas for how you think userfaultfd
> should work and how it solves your problem. :) At the end of the day,
> I'm just trying (though clearly failing miserably) to help you solve
> your problem.
>
> Your characterization of userfaultfd as a "useless" "bunch of random
> patches" that is just a "hack" is wrong. I understand; it doesn't
> support your needs. I think what Peter, Axel, and I have been trying
> to understand is what exactly you're trying to do and how userfaultfd
> could (or may not) help you get there. You've shared some[1]
> details[2] about what you're looking for, so thank you for that, but I
> am still struggling to understand how the flexibility that you're
> asking for is actually the right tool for the problem(s) you're trying
> to solve.
>
> [1]: https://lore.kernel.org/linux-mm/1758037039.08578612@apps.rackspace.com/
> [2]: https://lore.kernel.org/linux-mm/1758042583.108320755@apps.rackspace.com/
>
> > There is no sensible way to respond to a "missing event" when "missing" means
> the page is swapped out (to SWAP) by UFFDIO_COPY or UFFDIO_ZEROPAGE. That's just
> weird, and you continue to insist on it. Where is the page that was swapped out?
> Well, one could look at the PTE in /proc/pid/maps, and you find that its "swap
> entry" is there as an index into a block device. (so, maybe you can open the swap
> device using some file descriptor and mmap() it into the manager process, then
> UFFDIO_COPY, but what if the swap page is actually in the "swap cache", you can't
> mmap any swap cache page via any userspace API - do you know a way to do that?)
>
> (Please see the terms that I use at the bottom of this email; let me
> reply using those terms.)
>
> UFFDIO_COPY has quite well-defined semantics (albeit, perhaps not
> *documented* well):
>
> * For anonymous VMAs: UFFDIO_COPY will allocate page(s), copy some
> user memory into the page(s) and map those pages at the specified VAs.
> * For hugetlbfs and shmem/tmpfs VMAs, UFFDIO_COPY will fill holes in
> the file's page cache with new pages, copy the user memory in, and map
> those pages. UFFDIO_CONTINUE is additionally supported; it skips the
> hole-filling step and requires the page cache to be populated.
>
> For UFFDIO_COPY, if a page at a to-be-populated VA has already been
> allocated (including if it has been reclaimed), the call will be
> rejected. It would effectively be overwriting the contents of the
> page; this is not supported today.
>
> If "missing" includes swapped out pages, UFFDIO_COPY and
> UFFDIO_ZEROPAGE would need to be allowed to overwrite the existing
> contents. "Sensible" or not, there has been no need for this yet.
>
> > Now I reported a bug in UFFIO_REGISTER [...]
>
> The bug you reported is in the documentation only.
>
> > [...] which you keep saying is the same as UFFDIO_CONTINUE. Well, it isn't! I
> can register a minor handler (which allows continue) if I use
> MAP_ANONYMOUS|MAP_SHARED. The same "swap cache" mechanics exactly apply. The only
> "sharing" is potential future sharing after that process forks, in which case, the
> same "swap page" is shared until a Copy on Write forces the page to be unshared -
> it is a writeable page, just sharing the same physical block. It can be swapped
> out to the swap cache and the swap device, which sets the PTE to be a "swap entry"
> that causes a page fault.
>
> (Using the terms at the bottom of this email.)
>
> For UFFDIO_CONTINUE, the swap cache mechanics are like:
>
> 1. For anonymous pages in the VMA: swap-outs will not clear the PTEs,
> touching the page will swap it back in again, UFFDIO_CONTINUE on it is
> disallowed.
> 2. For page cache pages in the VMA (i.e., not-yet-written-to pages for
> MAP_PRIVATE, any page for MAP_SHARED): swap-outs will clear the PTEs,
> and touching the page will trigger a minor fault, and UFFDIO_CONTINUE
> will swap it back in.
>
> For MAP_ANONYMOUS|MAP_PRIVATE, all pages in the VMA will be anonymous
> pages, so UFFDIO_CONTINUE will never be allowed, therefore
> registration in the first place is disallowed.
>
> (IMHO, it was dubious to have even allowed registering userfaultfd
> minor faults with *any* MAP_PRIVATE VMA.)
>
> > The swap device doesn't know where the pages are mapped. You need to look at
> the PTEs of all the processes to find the translation to swap cache entry, and if
> you want to go backward from swap entry to pages, you need to use a special XArray
> that finds VMAs given swap entry.
> >
> > But the point here I keep making is that UFFDIO_REGISTER rejects only
> MAP_ANONYMOUS that are MAP_PRIVATE and also not huge pages. To me that's weird.
>
> I hope my above explanation (of sorts) makes it a little less weird.
>
> > If it is the CoW case that doesn't work (I doubt it), well, you have to read
> the swapped out page into memory before copying it anyway. Then you copy on write,
> from the page read or found in the swap cache.
> >
> > Now, as you say, that may require allocating a new page, also in the swap
> cache. Is that a "missing" page in the weird userfaultfd terminology? If so, to
> handle it can't be done with UFFIO_COPY, because you can't access the contents
> from userspace. And it's not "write protected" from the perspective of WP.
>
> No it isn't a missing userfault. Data exists at the VA for which a
> userfault would be generated, therefore it cannot be "missing".
>
> >
> >
> >
> > > The only exception I can
> > > think of is swap faults, I could see anon swap faults (perhaps
> > > specifically when the page is in the swap cache?) being considered
> > > UFFD minor faults, but I would be curious to know what the use case is
> > > for that / why you would want to do that. The original use case for
> > > UFFD minor fault support was demand paging for VMs, where you have
> > > some kind of shared memory (shmem or hugetlb) where one side of the
> > > mapping is given to the VM, and the other side of the shared mapping
> > > is used by the hypervisor to populate guest memory on-demand in
> > > response to userfaultfd events.
> >
> >
> >
> > I think I've just answered this. userfaultfd doesn't support the "swap out"
> part of anonymous swapping at all. So, how could a manager get the page contents
> as of the instant it is put in the swap cache for writing out to the swap device?
> There's no "swap out" event mechanism, and no way to treat the swap device cached
> into the swap cache as a page source. (not to mention the zswap mechanism, which
> compresses some of the pages into an invisible piece of memory).
> >
> >
> > >
> > > To me it's not intended userfaultfd minor events are generated for
> > > writeprotect faults, to me that's the domain of userfaultfd-wp, not
> > > minor faults. James might be right that these unintentionally trigger
> > > minor faults today, I would need to do some more reading of the code
> > > to be certain though.
> >
> > I don't particulary care about writeprotect faults, but CoW probably
> shouldn't be considered the same as a writeprotect fault, because CoW is triggered
> by a write into a writeable area, ONLY in one of the mappings, whichever is
> written first. The process doesn't think of it as a "write" - it just is a kernel
> optimization of a common case where fork is followed by non-use, so the actual
> copy could have been done at fork time, semantically. It's a deferred read and
> allocation.
> >
> >
> >
> > I hope this helps clarify my concerns.
> >
> > There are several reasonable outcomes -
> >
> > 1. Much better documentation of what the code actually does (and why).
>
> Agreed.
>
> > 2. Fix the "bug" that prevents REGISTER of "minor" handler on private,
> anonymous mappings (obviously, you can REGISTER missing handlers as well), then
> document actually what happens during the life cycle of swapping of pages in
> detail, including MAP_PRIVATE|MAP_ANONYMOUS VMAs.
>
> Not a bug.
>
> > 3. Do a thorough analysis of what userfaultfd really should do, if the goal
> is to provide the ability of a "manager process" to get to handle all cases of
> page fault behavior on a case-by-case basis for regions of user addressable
> pages.
>
> What userfaultfd "should do" is up to the problems we need it to solve.
>
> > I'd be happy to contribute to (but not manage) whichever outcome - and I have
> what I think is a reasonable use case. (and I'm aware that this API accidentally
> created a serious hacker exploit earlier in its life, by creating a way to hang
> one process from another. I think that's no longer so easy.)
>
> I would be glad to hear what changes you think should be made to
> userfaultfd to better suit your needs.
>
> Sorry if this reply is somewhat incoherent; I've gone back and forth a
> few times on how to respond to your points in the most helpful way I
> can. I've tried to be as clear as possible without being too verbose.
>
> - James
>
> --
>
> Alrighty here are the terms/definitions I use, as I mentioned above.
> Again feel, free to ignore them if they are unhelpful:
>
> A "file-backed VMA" will load pages into the page cache. For most
> filesystems, the page is loaded from a disk (or a proper device), but
> for special filesystems like tmpfs, hugetlbfs, and ramfs, the page
> cache is populated with zeroed pages initially.
>
> tmpfs is kind of like a filesystem API for shmem, but they are so
> interconnected that many people use the terms interchangeably. (To
> clarify, I don't think of "shmem" as shorthand for "shared memory"; to
> me, it is the name of an mm subsystem.) Every MAP_ANONYMOUS|MAP_SHARED
> VMA is a shmem VMA; it is as if there is a tmpfs file backing VMAs
> like these, so they are in some contexts considered "file-backed". See
> shmem_zero_setup(). As far as I'm concerned, vma->vm_file is set, so
> the VMA is file-backed (even though the mmap flags included
> MAP_ANONYMOUS). I assume this is what you are referring to when you
> say "file-backed by /dev/zero".
>
> For any MAP_PRIVATE VMA, some pages may be "anonymous", in that no
> page cache is holding a reference to it (i.e., generally speaking, the
> only references on the page are the ones taken by the PTEs mapping the
> page). Reclaim of pages like these will put them in a swap cache.
>
> For pages where a reference is held in a page cache, if the page is
> dirty, it can be written out to disk. shmem implements "writeout" by
> swapping just like anonymous pages, but other filesystems implement it
> how you would expect.
>