Re: [LSF/MM/BPF TOPIC] The future of ZONE_DEVICE pages

[Balbir Singh <balbirs@nvidia.com>]

On 1/31/25 13:59, Alistair Popple wrote:
> I have a few topics that I would like to discuss around ZONE_DEVICE pages
> and their current and future usage in the kernel. Generally these pages are
> used to represent various forms of device memory (PCIe BAR space, coherent
> accelerator memory, persistent memory, unaddressable device memory). All
> of these require special treatment by the core MM so many features must be
> implemented specifically for ZONE_DEVICE pages.
> 
> I would like to get feedback on several ideas I've had for a while:
> 
> Large page migration for ZONE_DEVICE pages
> ==========================================
> 
> Currently large ZONE_DEVICE pages only exist for persistent memory use cases
> (DAX, FS DAX). This involves a special reference counting scheme which I hope to
> have fixed[1] by the time of the LSF/MM/BPF. Fixing this allows for other higher
> order ZONE_DEVICE folios.
> 
> Specifically I would like to introduce the possiblity of migrating large CPU
> folios to unaddressable (DEVICE_PRIVATE) or coherent (DEVICE_COHERENT) memory.
> The current interfaces (migrate_vma) don't allow that as they require all folios
> to be split.
> 
> Some of the issues are:
> 
> 1. What should the interface look like?
> 
> These are non-lru pages, so likely there is overlap with "non-lru page migration
> in a memdesc world"[2]
> 
> 2. How do we allow merging/splitting of pages during migration?
> 
> This is neccessary because when migrating back from device memory there may not
> be enough large CPU pages available.
> 
> 3. Any other issues?

I'd definitely be interested in the above topic. In general, I see a lot of overlap
of folio and struct page code, I think folio is a good abstraction and I am hoping some
day we just have folio's to abstract the size of the page.


> 
> [1] - https://lore.kernel.org/linux-mm/cover.11189864684e31260d1408779fac9db80122047b.1736488799.git-series.apopple@nvidia.com/
> [2] - https://lore.kernel.org/linux-mm/2612ac8a-d0a9-452b-a53d-75ffc6166224@redhat.com/
> 
> File-backed DEVICE_PRIVATE/COHERENT pages
> =========================================
> 
> Currently DEVICE_PRVIATE and DEVICE_COHERENT pages are only supported for
> private anonymous memory. This prevents devices from having local access to
> shared or file-backed mappings instead relying on remote DMA access which limits
> performance.
> 
> I have been prototyping allowing ZONE_DEVICE pages in the page cache with
> a callback when the CPU requires access. This approach seems promising and
> relatively straight-forward but I would like some early feedback on either this
> or alternate approaches that I should investigate.
> 

I assume this is for mapped page cache pages?

> Combining P2PDMA and DEVICE_PRIVATE pages
> =========================================
> 
> Currently device memory that cannot be directly accessed via the CPU can be
> represented by DEVICE_PRIVATE pages allowing it to be mapped and treated like
> a normal virtual page by userpsace. Many devices also support accessing device
> memory directly from the CPU via a PCIe BAR.
> 
> This access requires a P2PDMA page, meaning there are potentially two pages
> tracking the same piece of physical memory. This not only seems wasteful but
> fraught - for example device drivers need to keep page lifetimes in sync. I
> would like to discuss ways of solving this.

+1 for the topics

Balbir Singh