[RFC] Huge remap_pfn_range for vfio-pci

[Axel Rasmussen <axelrasmussen@google.com>]

Hi,

I'm interested in extending remap_pfn_range to allow it to map the
range hugely (using PUDs or PMDs). The initial user I have in mind is
vfio-pci; I'm thinking when we're mapping large ranges for GPUs, we
can get both a performance and host overhead win by doing this hugely.

Another thing I have in the back of my mind is adding something KVM
can re-use to simplify its whole host_pfn_mapping_level /
hva_to_pfn_remapped / get_user_page_fast_only thing.

I know Peter and David are working on some related things (hugetlbfs
unification and follow_pte et al improvements, respectively). Although
I have a hacky proof of concept that works, I thought it best to get
some consensus on the design before I post something, so I don't
conflict with this existing / upcoming work.

Changing remap_pfn_range to install PUDs or PMDs is straightforward.
The hairy part is the fault / follow side of things:

1. follow_pte clearly doesn't work for this, since the leaf might be a
PUD or PMD instead. Most callers don't care about the PTE itself, they
care about the pgprot or flags it has set, so my idea was to add a new
interface which just yields those bits, instead of the actual PTE.

Peter, I think hugetlbfs unification may run into similar issues, do
you have some plan already to deal with PUD/PMD/PTE being different
types?

2. vfio-pci relies on vm_ops->fault. This is a problem because the
normal fault handler path doesn't call this until after it has walked
down to the PTE level, installing PUDs/PMDs along the way. I have only
gross ideas for how to deal with this:

- Add a VM_HUGEPFNMAP VMA flag indicating vm_ops->fault should be
called earlier in __handle_mm_fault
- Add a vm_ops->hugepfn_fault (name not important) which should be
called earlier in __handle_mm_fault
- Go ahead and let remap_pfn_range overwrite existing PUDs/PMDS

I wonder which of these folks find least offensive? Or is there a
better way I haven't thought of?

3. That's also an issue for CoW faults, but I don't know of any real
use case for CoW huge pfn mappings, so I thought we can just keep the
existing small mapping behavior for CoW VMAs. Any objections?

Thanks!