How to mmap any address space using huge pages?

How to mmap any address space using huge pages?

[Adam Turowski]

Hello all,
I have a device and I need to mmap it into a user space using 1GB huge
pages. AFAIK I cannot use HugeTLB nor THP because they're backed by the
memory. The remap_pfn_range doesn't use huge pages neither. So I used a
huge_fault handler to set up the pud descriptor myself (arm64 here, so no
transparent puds) and that works. The problem is that the kernel warns
about a bad pud when the vma mapping is removed, the zap_pud_range function
doesn't accept block puds. The vunmap_pud_range function works with block
puds and checks for their presence and calls the pud_clear_huge function.
If I add this function call to the zap_pud_range function, the warnings are
gone.

Am I missing something? What is the proper way to mmap (and munmap) any
address space using huge pages (esp. 1GB pages)?

Thanks

--
Regards,
Adam Turowski

Re: How to mmap any address space using huge pages?

[Jason Gunthorpe]

On Sat, Oct 16, 2021 at 11:46:59AM +0200, Adam Turowski wrote:
> Hello all,
> I have a device and I need to mmap it into a user space using 1GB huge
> pages. AFAIK I cannot use HugeTLB nor THP because they're backed by the
> memory. The remap_pfn_range doesn't use huge pages neither. So I used a
> huge_fault handler to set up the pud descriptor myself (arm64 here, so no
> transparent puds) and that works. The problem is that the kernel warns
> about a bad pud when the vma mapping is removed, the zap_pud_range function
> doesn't accept block puds. The vunmap_pud_range function works with block
> puds and checks for their presence and calls the pud_clear_huge function.
> If I add this function call to the zap_pud_range function, the warnings are
> gone.
> 
> Am I missing something? What is the proper way to mmap (and munmap) any
> address space using huge pages (esp. 1GB pages)?

AFAIK it can't be done today.

Certainly non-struct page memory cannot be installed in a PUD

If you create a ZONE_DEVICE struct page using memremap_pages() then
they can be inserted using vmf_insert_pfn_pud() - however that
requires arch support for pud_devmap() which only x86 does right now.

Jason

Re: How to mmap any address space using huge pages?

[Adam Turowski]

> On Sat, Oct 16, 2021 at 11:46:59AM +0200, Adam Turowski wrote:
> > Hello all,
> > I have a device and I need to mmap it into a user space using 1GB huge
> > pages. AFAIK I cannot use HugeTLB nor THP because they're backed by the
> > memory. The remap_pfn_range doesn't use huge pages neither. So I used a
> > huge_fault handler to set up the pud descriptor myself (arm64 here, so no
> > transparent puds) and that works. The problem is that the kernel warns
> > about a bad pud when the vma mapping is removed, the zap_pud_range function
> > doesn't accept block puds. The vunmap_pud_range function works with block
> > puds and checks for their presence and calls the pud_clear_huge function.
> > If I add this function call to the zap_pud_range function, the warnings are
> > gone.
> >
> > Am I missing something? What is the proper way to mmap (and munmap) any
> > address space using huge pages (esp. 1GB pages)?
>
> AFAIK it can't be done today.
>
> Certainly non-struct page memory cannot be installed in a PUD
>
> If you create a ZONE_DEVICE struct page using memremap_pages() then
> they can be inserted using vmf_insert_pfn_pud() - however that
> requires arch support for pud_devmap() which only x86 does right now.
>

OK, I can live with "bad pud" warnings for now.
Would it be a bad idea to make the zap_pud_range function to look like
vunmap_pud_range function by adding a pud_clear_huge(pud) call?

--
Regards,
Adam Turowski