Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

[Alistair Popple <apopple@nvidia.com>]

On Fri, Aug 01, 2025 at 01:57:49PM -0300, Jason Gunthorpe wrote:
> On Fri, Aug 01, 2025 at 06:50:18PM +0200, David Hildenbrand wrote:
> > On 01.08.25 18:40, Jason Gunthorpe wrote:
> > > On Fri, Jul 25, 2025 at 10:31:25AM +1000, Alistair Popple wrote:
> > > 
> > > > The only issue would be if there were generic code paths that somehow have a
> > > > raw pfn obtained from neither a page-table walk or struct page. My assumption
> > > > (yet to be proven/tested) is that these paths don't exist.
> > > 
> > > hmm does it, it encodes the device private into a pfn and expects the
> > > caller to do pfn to page.

What callers need to do pfn to page when finding a device private pfn via
hmm_range_fault()? GPU drivers don't, they tend just to use the pfn as an offset
from the start of the pgmap to find whatever data structure they are using to
track device memory allocations.

The migrate_vma_*() calls do, but they could easily be changed to whatever
index scheme we use so long as we can encode that this is a device entry in the
MIGRATE_PFN flags.

So other than adding a HMM_PFN flag to say this is really a device index I don't
see too many issues here.

> > > This isn't set in stone and could be changed..
> > > 
> > > But broadly, you'd want to entirely eliminate the ability to go from
> > > pfn to device private or from device private to pfn.
> > > 
> > > Instead you'd want to work on some (space #, space index) tuple, maybe
> > > encoded in a pfn_t, but absolutely and typesafely distinct. Each
> > > driver gets its own 0 based space for device private information, the
> > > space is effectively the pgmap.
> > > 
> > > And if you do this, maybe we don't need struct page (I mean the type!)
> > > backing device memory at all.... Which would be a very worthwhile
> > > project.

Exactly! Although we still need enough of a struct page or something else to
still be able to map them in normal anonymous VMAs. Short term the motivation
for this project is that the current scheme of "stealing" pfns for the device
doesn't actually work in a lot of cases.

> > > Do we ever even use anything in the device private struct page? Do we
> > > refcount it?
> > 
> > ref-counted and map-counted ...
> 
> Hm, so it would turn into another struct page split up where we get
> ourselves a struct device_private and change all the places touching
> its refcount and mapcount to use the new type.
> 
> If we could use some index scheme we could then divorce from struct
> page and strink the struct size sooner.

Right, that is roughly along the lines of what I was thinking.

> Jason