Re: [PATCH v2 11/11] mm: Remove device private pages from the physical address space

[Zi Yan <ziy@nvidia.com>]

On 20 Jan 2026, at 18:34, Jordan Niethe wrote:

> Hi,
>
> On 21/1/26 10:06, Zi Yan wrote:
>> On 20 Jan 2026, at 18:02, Jordan Niethe wrote:
>>
>>> Hi,
>>>
>>> On 21/1/26 09:53, Zi Yan wrote:
>>>> On 20 Jan 2026, at 17:33, Jordan Niethe wrote:
>>>>
>>>>> On 14/1/26 07:04, Zi Yan wrote:
>>>>>> On 7 Jan 2026, at 4:18, Jordan Niethe wrote:
>>>>>>
>>>>>>> Currently when creating these device private struct pages, the first
>>>>>>> step is to use request_free_mem_region() to get a range of physical
>>>>>>> address space large enough to represent the devices memory. This
>>>>>>> allocated physical address range is then remapped as device private
>>>>>>> memory using memremap_pages().
>>>>>>>
>>>>>>> Needing allocation of physical address space has some problems:
>>>>>>>
>>>>>>>      1) There may be insufficient physical address space to represent the
>>>>>>>         device memory. KASLR reducing the physical address space and VM
>>>>>>>         configurations with limited physical address space increase the
>>>>>>>         likelihood of hitting this especially as device memory increases. This
>>>>>>>         has been observed to prevent device private from being initialized.
>>>>>>>
>>>>>>>      2) Attempting to add the device private pages to the linear map at
>>>>>>>         addresses beyond the actual physical memory causes issues on
>>>>>>>         architectures like aarch64 meaning the feature does not work there.
>>>>>>>
>>>>>>> Instead of using the physical address space, introduce a device private
>>>>>>> address space and allocate devices regions from there to represent the
>>>>>>> device private pages.
>>>>>>>
>>>>>>> Introduce a new interface memremap_device_private_pagemap() that
>>>>>>> allocates a requested amount of device private address space and creates
>>>>>>> the necessary device private pages.
>>>>>>>
>>>>>>> To support this new interface, struct dev_pagemap needs some changes:
>>>>>>>
>>>>>>>      - Add a new dev_pagemap::nr_pages field as an input parameter.
>>>>>>>      - Add a new dev_pagemap::pages array to store the device
>>>>>>>        private pages.
>>>>>>>
>>>>>>> When using memremap_device_private_pagemap(), rather then passing in
>>>>>>> dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to
>>>>>>> be remapped, dev_pagemap::nr_ranges will always be 1, and the device
>>>>>>> private range that is reserved is returned in dev_pagemap::range.
>>>>>>>
>>>>>>> Forbid calling memremap_pages() with dev_pagemap::ranges::type =
>>>>>>> MEMORY_DEVICE_PRIVATE.
>>>>>>>
>>>>>>> Represent this device private address space using a new
>>>>>>> device_private_pgmap_tree maple tree. This tree maps a given device
>>>>>>> private address to a struct dev_pagemap, where a specific device private
>>>>>>> page may then be looked up in that dev_pagemap::pages array.
>>>>>>>
>>>>>>> Device private address space can be reclaimed and the assoicated device
>>>>>>> private pages freed using the corresponding new
>>>>>>> memunmap_device_private_pagemap() interface.
>>>>>>>
>>>>>>> Because the device private pages now live outside the physical address
>>>>>>> space, they no longer have a normal PFN. This means that page_to_pfn(),
>>>>>>> et al. are no longer meaningful.
>>>>>>>
>>>>>>> Introduce helpers:
>>>>>>>
>>>>>>>      - device_private_page_to_offset()
>>>>>>>      - device_private_folio_to_offset()
>>>>>>>
>>>>>>> to take a given device private page / folio and return its offset within
>>>>>>> the device private address space.
>>>>>>>
>>>>>>> Update the places where we previously converted a device private page to
>>>>>>> a PFN to use these new helpers. When we encounter a device private
>>>>>>> offset, instead of looking up its page within the pagemap use
>>>>>>> device_private_offset_to_page() instead.
>>>>>>>
>>>>>>> Update the existing users:
>>>>>>>
>>>>>>>     - lib/test_hmm.c
>>>>>>>     - ppc ultravisor
>>>>>>>     - drm/amd/amdkfd
>>>>>>>     - gpu/drm/xe
>>>>>>>     - gpu/drm/nouveau
>>>>>>>
>>>>>>> to use the new memremap_device_private_pagemap() interface.
>>>>>>>
>>>>>>> Signed-off-by: Jordan Niethe <jniethe@nvidia.com>
>>>>>>> Signed-off-by: Alistair Popple <apopple@nvidia.com>
>>>>>>>
>>>>>>> ---
>>>>>>>
>>>>>>> NOTE: The updates to the existing drivers have only been compile tested.
>>>>>>> I'll need some help in testing these drivers.
>>>>>>>
>>>>>>> v1:
>>>>>>> - Include NUMA node paramater for memremap_device_private_pagemap()
>>>>>>> - Add devm_memremap_device_private_pagemap() and friends
>>>>>>> - Update existing users of memremap_pages():
>>>>>>>        - ppc ultravisor
>>>>>>>        - drm/amd/amdkfd
>>>>>>>        - gpu/drm/xe
>>>>>>>        - gpu/drm/nouveau
>>>>>>> - Update for HMM huge page support
>>>>>>> - Guard device_private_offset_to_page and friends with CONFIG_ZONE_DEVICE
>>>>>>>
>>>>>>> v2:
>>>>>>> - Make sure last member of struct dev_pagemap remains DECLARE_FLEX_ARRAY(struct range, ranges);
>>>>>>> ---
>>>>>>>     Documentation/mm/hmm.rst                 |  11 +-
>>>>>>>     arch/powerpc/kvm/book3s_hv_uvmem.c       |  41 ++---
>>>>>>>     drivers/gpu/drm/amd/amdkfd/kfd_migrate.c |  23 +--
>>>>>>>     drivers/gpu/drm/nouveau/nouveau_dmem.c   |  35 ++--
>>>>>>>     drivers/gpu/drm/xe/xe_svm.c              |  28 +---
>>>>>>>     include/linux/hmm.h                      |   3 +
>>>>>>>     include/linux/leafops.h                  |  16 +-
>>>>>>>     include/linux/memremap.h                 |  64 +++++++-
>>>>>>>     include/linux/migrate.h                  |   6 +-
>>>>>>>     include/linux/mm.h                       |   2 +
>>>>>>>     include/linux/rmap.h                     |   5 +-
>>>>>>>     include/linux/swapops.h                  |  10 +-
>>>>>>>     lib/test_hmm.c                           |  69 ++++----
>>>>>>>     mm/debug.c                               |   9 +-
>>>>>>>     mm/memremap.c                            | 193 ++++++++++++++++++-----
>>>>>>>     mm/mm_init.c                             |   8 +-
>>>>>>>     mm/page_vma_mapped.c                     |  19 ++-
>>>>>>>     mm/rmap.c                                |  43 +++--
>>>>>>>     mm/util.c                                |   5 +-
>>>>>>>     19 files changed, 391 insertions(+), 199 deletions(-)
>>>>>>>
>>>>>> <snip>
>>>>>>
>>>>>>> diff --git a/include/linux/mm.h b/include/linux/mm.h
>>>>>>> index e65329e1969f..b36599ab41ba 100644
>>>>>>> --- a/include/linux/mm.h
>>>>>>> +++ b/include/linux/mm.h
>>>>>>> @@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)
>>>>>>>      */
>>>>>>>     static inline unsigned long folio_pfn(const struct folio *folio)
>>>>>>>     {
>>>>>>> +	VM_BUG_ON(folio_is_device_private(folio));
>>>>>>
>>>>>> Please use VM_WARN_ON instead.
>>>>>
>>>>> ack.
>>>>>
>>>>>>
>>>>>>> +
>>>>>>>     	return page_to_pfn(&folio->page);
>>>>>>>     }
>>>>>>>
>>>>>>> diff --git a/include/linux/rmap.h b/include/linux/rmap.h
>>>>>>> index 57c63b6a8f65..c1561a92864f 100644
>>>>>>> --- a/include/linux/rmap.h
>>>>>>> +++ b/include/linux/rmap.h
>>>>>>> @@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)
>>>>>>>     static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)
>>>>>>>     {
>>>>>>>     	if (folio_is_device_private(folio))
>>>>>>> -		return page_vma_walk_pfn(folio_pfn(folio)) |
>>>>>>> +		return page_vma_walk_pfn(device_private_folio_to_offset(folio)) |
>>>>>>>     		       PVMW_PFN_DEVICE_PRIVATE;
>>>>>>>
>>>>>>>     	return page_vma_walk_pfn(folio_pfn(folio));
>>>>>>> @@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)
>>>>>>>
>>>>>>>     static inline struct page *page_vma_walk_pfn_to_page(unsigned long pvmw_pfn)
>>>>>>>     {
>>>>>>> +	if (pvmw_pfn & PVMW_PFN_DEVICE_PRIVATE)
>>>>>>> +		return device_private_offset_to_page(pvmw_pfn >> PVMW_PFN_SHIFT);
>>>>>>> +
>>>>>>>     	return pfn_to_page(pvmw_pfn >> PVMW_PFN_SHIFT);
>>>>>>>     }
>>>>>>
>>>>>> <snip>
>>>>>>
>>>>>>> diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c
>>>>>>> index 96c525785d78..141fe5abd33f 100644
>>>>>>> --- a/mm/page_vma_mapped.c
>>>>>>> +++ b/mm/page_vma_mapped.c
>>>>>>> @@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,
>>>>>>>     static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)
>>>>>>>     {
>>>>>>>     	unsigned long pfn;
>>>>>>> +	bool device_private = false;
>>>>>>>     	pte_t ptent = ptep_get(pvmw->pte);
>>>>>>>
>>>>>>>     	if (pvmw->flags & PVMW_MIGRATION) {
>>>>>>> @@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)
>>>>>>>     		if (!softleaf_is_migration(entry))
>>>>>>>     			return false;
>>>>>>>
>>>>>>> +		if (softleaf_is_migration_device_private(entry))
>>>>>>> +			device_private = true;
>>>>>>> +
>>>>>>>     		pfn = softleaf_to_pfn(entry);
>>>>>>>     	} else if (pte_present(ptent)) {
>>>>>>>     		pfn = pte_pfn(ptent);
>>>>>>> @@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)
>>>>>>>     			return false;
>>>>>>>
>>>>>>>     		pfn = softleaf_to_pfn(entry);
>>>>>>> +
>>>>>>> +		if (softleaf_is_device_private(entry))
>>>>>>> +			device_private = true;
>>>>>>>     	}
>>>>>>>
>>>>>>> +	if ((device_private) ^ !!(pvmw->pfn & PVMW_PFN_DEVICE_PRIVATE))
>>>>>>> +		return false;
>>>>>>> +
>>>>>>>     	if ((pfn + pte_nr - 1) < (pvmw->pfn >> PVMW_PFN_SHIFT))
>>>>>>>     		return false;
>>>>>>>     	if (pfn > ((pvmw->pfn >> PVMW_PFN_SHIFT) + pvmw->nr_pages - 1))
>>>>>>> @@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)
>>>>>>>     }
>>>>>>>
>>>>>>>     /* Returns true if the two ranges overlap.  Careful to not overflow. */
>>>>>>> -static bool check_pmd(unsigned long pfn, struct page_vma_mapped_walk *pvmw)
>>>>>>> +static bool check_pmd(unsigned long pfn, bool device_private, struct page_vma_mapped_walk *pvmw)
>>>>>>>     {
>>>>>>> +	if ((device_private) ^ !!(pvmw->pfn & PVMW_PFN_DEVICE_PRIVATE))
>>>>>>> +		return false;
>>>>>>> +
>>>>>>>     	if ((pfn + HPAGE_PMD_NR - 1) < (pvmw->pfn >> PVMW_PFN_SHIFT))
>>>>>>>     		return false;
>>>>>>>     	if (pfn > (pvmw->pfn >> PVMW_PFN_SHIFT) + pvmw->nr_pages - 1)
>>>>>>> @@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
>>>>>>>
>>>>>>>     				if (!softleaf_is_migration(entry) ||
>>>>>>>     				    !check_pmd(softleaf_to_pfn(entry),
>>>>>>> +					       softleaf_is_device_private(entry) ||
>>>>>>> +					       softleaf_is_migration_device_private(entry),
>>>>>>>     					       pvmw))
>>>>>>>     					return not_found(pvmw);
>>>>>>>     				return true;
>>>>>>> @@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
>>>>>>>     			if (likely(pmd_trans_huge(pmde))) {
>>>>>>>     				if (pvmw->flags & PVMW_MIGRATION)
>>>>>>>     					return not_found(pvmw);
>>>>>>> -				if (!check_pmd(pmd_pfn(pmde), pvmw))
>>>>>>> +				if (!check_pmd(pmd_pfn(pmde), false, pvmw))
>>>>>>>     					return not_found(pvmw);
>>>>>>>     				return true;
>>>>>>>     			}
>>>>>>
>>>>>> It seems to me that you can add a new flag like “bool is_device_private” to
>>>>>> indicate whether pfn is a device private index instead of pfn without
>>>>>> manipulating pvmw->pfn itself.
>>>>>
>>>>> We could do it like that, however my concern with using a new param was that
>>>>> storing this info seperately might make it easier to misuse a device private
>>>>> index as a regular pfn.
>>>>>
>>>>> It seemed like it could be easy to overlook both when creating the pvmw and
>>>>> then when accessing the pfn.
>>>>
>>>> That is why I asked for a helper function like page_vma_walk_pfn(pvmw) to
>>>> return the converted pfn instead of pvmw->pfn directly. You can add a comment
>>>> to ask people to use helper function and even mark pvmw->pfn /* do not use
>>>> directly */.
>>>
>>> Yeah I agree that is a good idea.
>>>
>>>>
>>>> In addition, your patch manipulates pfn by left shifting it by 1. Are you sure
>>>> there is no weird arch having pfns with bit 63 being 1? Your change could
>>>> break it, right?
>>>
>>> Currently for migrate pfns we left shift by pfns by MIGRATE_PFN_SHIFT (6), so I
>>> thought doing something similiar here should be safe.
>>
>> Yeah, but that limits to archs supporting HMM. page_vma_mapped_walk is used
>> by almost every arch, so it has a broader impact.
>
> That is a good point.
>
> I see a few options:
>
> - On every arch we can assume SWP_PFN_BITS? I could add a sanity check that we
>   have an extra bit on top of SWP_PFN_BITS within an unsigned long.

Yes, but if there is no extra bit, are you going to disable device private
pages?

> - We could define PVMW_PFN_SHIFT as 0 if !CONFIG_MIGRATION as the flag is not
>   required.

Sure, or !CONFIG_DEVICE_MIGRATION

> - Instead of modifying pvmw->pfn we could use pvmw->flags but that has the
>   issues of separating the offset type and offset.

It seems that I was not clear on my proposal. Here is the patch on top of
your patchset and it compiles.

Basically, pvmw->pfn stores either PFN or device private offset without
additional shift. Caller interprets pvmw->pfn based on
pvmw->flags & PVMW_DEVICE_PRIVATE. And you can ignore my helper function
of pvmw->pfn suggestion, since my patch below can use pvmw->pfn directly.

Let me know if my patch works. Thanks.

diff --git a/include/linux/rmap.h b/include/linux/rmap.h
index c1561a92864f..4423f0e886aa 100644
--- a/include/linux/rmap.h
+++ b/include/linux/rmap.h
@@ -921,6 +921,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr,
 #define PVMW_SYNC		(1 << 0)
 /* Look for migration entries rather than present PTEs */
 #define PVMW_MIGRATION		(1 << 1)
+#define PVMW_DEVICE_PRIVATE	(1 << 2)

 /* Result flags */

@@ -943,6 +944,13 @@ struct page_vma_mapped_walk {
 #define PVMW_PFN_DEVICE_PRIVATE	(1UL << 0)
 #define PVMW_PFN_SHIFT		1

+static inline unsigned long page_vma_walk_flags(struct folio *folio, unsigned long flags)
+{
+	if (folio_is_device_private(folio))
+		return flags | PVMW_DEVICE_PRIVATE;
+	return flags;
+}
+
 static inline unsigned long page_vma_walk_pfn(unsigned long pfn)
 {
 	return (pfn << PVMW_PFN_SHIFT);
@@ -951,23 +959,16 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)
 static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)
 {
 	if (folio_is_device_private(folio))
-		return page_vma_walk_pfn(device_private_folio_to_offset(folio)) |
-		       PVMW_PFN_DEVICE_PRIVATE;
-
-	return page_vma_walk_pfn(folio_pfn(folio));
-}
-
-static inline struct page *page_vma_walk_pfn_to_page(unsigned long pvmw_pfn)
-{
-	if (pvmw_pfn & PVMW_PFN_DEVICE_PRIVATE)
-		return device_private_offset_to_page(pvmw_pfn >> PVMW_PFN_SHIFT);
+		return device_private_folio_to_offset(folio);

-	return pfn_to_page(pvmw_pfn >> PVMW_PFN_SHIFT);
+	return (folio_pfn(folio));
 }

-static inline struct folio *page_vma_walk_pfn_to_folio(unsigned long pvmw_pfn)
+static inline struct folio *page_vma_walk_pfn_to_folio(struct page_vma_mapped_walk *pvmw)
 {
-	return page_folio(page_vma_walk_pfn_to_page(pvmw_pfn));
+	if (pvmw->flags & PVMW_DEVICE_PRIVATE)
+		return page_folio(device_private_offset_to_page(pvmw->pfn));
+	return pfn_folio(pvmw->pfn);
 }

 #define DEFINE_FOLIO_VMA_WALK(name, _folio, _vma, _address, _flags)	\
@@ -977,7 +978,7 @@ static inline struct folio *page_vma_walk_pfn_to_folio(unsigned long pvmw_pfn)
 		.pgoff = folio_pgoff(_folio),				\
 		.vma = _vma,						\
 		.address = _address,					\
-		.flags = _flags,					\
+		.flags = page_vma_walk_flags(_folio, _flags),		\
 	}

 static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw)
diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c
index 141fe5abd33f..e61a0e49a7c9 100644
--- a/mm/page_vma_mapped.c
+++ b/mm/page_vma_mapped.c
@@ -136,12 +136,12 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)
 			device_private = true;
 	}

-	if ((device_private) ^ !!(pvmw->pfn & PVMW_PFN_DEVICE_PRIVATE))
+	if ((device_private) ^ !!(pvmw->flags & PVMW_DEVICE_PRIVATE))
 		return false;

-	if ((pfn + pte_nr - 1) < (pvmw->pfn >> PVMW_PFN_SHIFT))
+	if ((pfn + pte_nr - 1) < pvmw->pfn)
 		return false;
-	if (pfn > ((pvmw->pfn >> PVMW_PFN_SHIFT) + pvmw->nr_pages - 1))
+	if (pfn > (pvmw->pfn + pvmw->nr_pages - 1))
 		return false;
 	return true;
 }
@@ -149,12 +149,12 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)
 /* Returns true if the two ranges overlap.  Careful to not overflow. */
 static bool check_pmd(unsigned long pfn, bool device_private, struct page_vma_mapped_walk *pvmw)
 {
-	if ((device_private) ^ !!(pvmw->pfn & PVMW_PFN_DEVICE_PRIVATE))
+	if ((device_private) ^ !!(pvmw->flags & PVMW_DEVICE_PRIVATE))
 		return false;

-	if ((pfn + HPAGE_PMD_NR - 1) < (pvmw->pfn >> PVMW_PFN_SHIFT))
+	if ((pfn + HPAGE_PMD_NR - 1) < pvmw->pfn)
 		return false;
-	if (pfn > (pvmw->pfn >> PVMW_PFN_SHIFT) + pvmw->nr_pages - 1)
+	if (pfn > pvmw->pfn + pvmw->nr_pages - 1)
 		return false;
 	return true;
 }
@@ -369,7 +369,7 @@ unsigned long page_mapped_in_vma(const struct page *page,
 		.pfn = folio_page_vma_walk_pfn(folio),
 		.nr_pages = 1,
 		.vma = vma,
-		.flags = PVMW_SYNC,
+		.flags = page_vma_walk_flags(folio, PVMW_SYNC),
 	};

 	pvmw.address = vma_address(vma, page_pgoff(folio, page), 1);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index be5682d345b5..5d81939bf12a 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4203,7 +4203,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 	pte_t *pte = pvmw->pte;
 	unsigned long addr = pvmw->address;
 	struct vm_area_struct *vma = pvmw->vma;
-	struct folio *folio = page_vma_walk_pfn_to_folio(pvmw->pfn);
+	struct folio *folio = page_vma_walk_pfn_to_folio(pvmw);
 	struct mem_cgroup *memcg = folio_memcg(folio);
 	struct pglist_data *pgdat = folio_pgdat(folio);
 	struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);





Best Regards,
Yan, Zi