Re: find_get_page() VS pin_user_pages()

[Alistair Popple <apopple@nvidia.com>]

"Teterevkov, Ivan" <Ivan.Teterevkov@amd.com> writes:

> Hello folks,
>
> I work with an application which aims to share memory in the userspace and
> interact with the NIC DMA. The memory allocation workflow begins in the
> userspace, which creates a new file backed by 2MiB hugepages with
> memfd_create(MFD_HUGETLB, MFD_HUGE_2MB) and fallocate(). Then the userspace
> makes an IOCTL to the kernel module with the file descriptor and size so that
> the kernel module can get the struct page with find_get_page(). Then the kernel
> module calls dma_map_single(page_address(page)) for NIC, which concludes the
> datapath. The allocated memory may (significantly) outlive the originating
> userspace application. The hugepages stay mapped with NIC, and the kernel
> module wants to continue using them and map to other applications that come and
> go with vm_mmap().
>
> I am studying the pin_user_pages*() family of functions, and I wonder if the
> outlined workflow requires it. The hugepages do not page out, but they can move
> as they may be allocated with GFP_HIGHUSER_MOVABLE. However, find_get_page()
> must increment the page reference counter without mapping and prevent it from
> moving. In particular, https://docs.kernel.org/mm/page_migration.html:

I'm not super familiar with the memfd_create()/find_get_page() workflow
but is there some reason you're not using pin_user_pages*(FOLL_LONGTERM)
to get the struct page initially? You're description above sounds
exactly the use case pin_user_pages() was designed for because it marks
the page as being writen to by DMA, makes sure it's not in a movable
zone, etc.

>> How migrate_pages() works
>> ...
>> Steps:
>> ...
>> 4. All the page table references to the page are converted to migration
>>    entries. This decreases the mapcount of a page. If the resulting mapcount
>>    is not zero then we do not migrate the page.
>
> Does find_get_page() achieve that condition or does the outlined workflow
> still requires pin_user_pages*() for safe DMA?

Yes. The extra page reference will prevent the migration regardless of
mapcount being zero or not. See folio_expected_refs() for how the extra
reference is detected.

> Thanks in advance,
> Ivan