[PATCH mm-hotfixes] mm/huge_memory: fix memory corruption on huge zero page move

[PATCH mm-hotfixes] mm/huge_memory: fix memory corruption on huge zero page move

[Lorenzo Stoakes]

In commit eb1521dad8f3 ("userfaultfd: handle zeropage moves by
UFFDIO_MOVE"), handling was added to enable the moving of huge zero pages
in move_pages_huge_pmd().

This achieves this by setting src_folio to NULL, and adding subsequent
checks for src_folio being NULL to determine whether to perform the usual
move operations or to simply establish the huge zero page in the
destination.

As part of this change, when installing the destination huge zero page it
invoked mk_huge_pmd() on src_page, correctly.

However, commit e3981db444a0 ("mm: add folio_mk_pmd()") updated the code in
the huge zero page branch from mk_huge_pmd(src_page, ...) to
folio_mk_pmd(src_folio, ...), where src_folio is guaranteed to be NULL at
this point.

This resulted in an invocation of folio_mk_pmd(NULL, ...) in effect, which
causes an invocation of page_to_pfn(0) and results in the installation of a
corrupted PMD entry and undefined behaviour.

This patch fixes the issue by obtaining the zero folio via
page_folio(src_page) and feeding this into folio_mk_pmd(). This retains the
use of folio_mk_pmd() whilst avoiding the memory corruption.

Additionally, this code path was not updated to reflect the changes
introduced by commit d82d09e48219 ("mm/huge_memory: mark PMD mappings of
the huge zero folio special"), meaning a zero huge folio was installed but
not marked special in this case.

This patch additionally fixes that issue by invoking pmd_mkspecial().

With thanks to Chris Down who exposed this bug by adding an explicit test
for UFFDIO_MOVE in commit f07254dce67d ("selftests/mm: add UFFDIO_MOVE huge
zeropage PMD regression test").

Fixes: e3981db444a0 ("mm: add folio_mk_pmd()")
Cc: <stable@vger.kernel.org>
Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
---
 mm/huge_memory.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index c17965f5682f..de2a775590f1 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2796,8 +2796,12 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm
 		/* Follow mremap() behavior and treat the entry dirty after the move */
 		_dst_pmd = pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
 	} else {
+		struct folio *zero_folio = page_folio(src_page);
+
+		VM_WARN_ON_ONCE_FOLIO(!is_huge_zero_folio(zero_folio), zero_folio);
 		src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd);
-		_dst_pmd = folio_mk_pmd(src_folio, dst_vma->vm_page_prot);
+		_dst_pmd = folio_mk_pmd(zero_folio, dst_vma->vm_page_prot);
+		_dst_pmd = pmd_mkspecial(_dst_pmd);
 	}
 	set_pmd_at(mm, dst_addr, dst_pmd, _dst_pmd);

--
2.53.0

Re: [PATCH mm-hotfixes] mm/huge_memory: fix memory corruption on huge zero page move

[David Hildenbrand (Arm)]

On 3/2/26 18:06, Lorenzo Stoakes wrote:
> In commit eb1521dad8f3 ("userfaultfd: handle zeropage moves by
> UFFDIO_MOVE"), handling was added to enable the moving of huge zero pages
> in move_pages_huge_pmd().
> 
> This achieves this by setting src_folio to NULL, and adding subsequent
> checks for src_folio being NULL to determine whether to perform the usual
> move operations or to simply establish the huge zero page in the
> destination.
> 
> As part of this change, when installing the destination huge zero page it
> invoked mk_huge_pmd() on src_page, correctly.
> 
> However, commit e3981db444a0 ("mm: add folio_mk_pmd()") updated the code in
> the huge zero page branch from mk_huge_pmd(src_page, ...) to
> folio_mk_pmd(src_folio, ...), where src_folio is guaranteed to be NULL at
> this point.
> 
> This resulted in an invocation of folio_mk_pmd(NULL, ...) in effect, which
> causes an invocation of page_to_pfn(0) and results in the installation of a
> corrupted PMD entry and undefined behaviour.
> 
> This patch fixes the issue by obtaining the zero folio via
> page_folio(src_page) and feeding this into folio_mk_pmd(). This retains the
> use of folio_mk_pmd() whilst avoiding the memory corruption.
> 
> Additionally, this code path was not updated to reflect the changes
> introduced by commit d82d09e48219 ("mm/huge_memory: mark PMD mappings of
> the huge zero folio special"), meaning a zero huge folio was installed but
> not marked special in this case.
> 
> This patch additionally fixes that issue by invoking pmd_mkspecial().
> 
> With thanks to Chris Down who exposed this bug by adding an explicit test
> for UFFDIO_MOVE in commit f07254dce67d ("selftests/mm: add UFFDIO_MOVE huge
> zeropage PMD regression test").
> 
> Fixes: e3981db444a0 ("mm: add folio_mk_pmd()")
> Cc: <stable@vger.kernel.org>
> Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
> ---
>  mm/huge_memory.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index c17965f5682f..de2a775590f1 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -2796,8 +2796,12 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm
>  		/* Follow mremap() behavior and treat the entry dirty after the move */
>  		_dst_pmd = pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
>  	} else {
> +		struct folio *zero_folio = page_folio(src_page);
> +
> +		VM_WARN_ON_ONCE_FOLIO(!is_huge_zero_folio(zero_folio), zero_folio);
>  		src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd);
> -		_dst_pmd = folio_mk_pmd(src_folio, dst_vma->vm_page_prot);
> +		_dst_pmd = folio_mk_pmd(zero_folio, dst_vma->vm_page_prot);
> +		_dst_pmd = pmd_mkspecial(_dst_pmd);
>  	}
>  	set_pmd_at(mm, dst_addr, dst_pmd, _dst_pmd);

There are already patches in flight:

https://lore.kernel.org/r/aaBVaHs8rIkNcwM0@chrisdown.name

:)

-- 
Cheers,

David

Re: [PATCH mm-hotfixes] mm/huge_memory: fix memory corruption on huge zero page move

[Lorenzo Stoakes]

On Mon, Mar 02, 2026 at 06:10:06PM +0100, David Hildenbrand (Arm) wrote:
> There are already patches in flight:
>
> https://lore.kernel.org/r/aaBVaHs8rIkNcwM0@chrisdown.name

Yup, if people would actually cc the right people I'd not have wasted my
afternoon.

Replying there.

>
> --
> Cheers,
>
> David

Thanks, Lorenzo

Re: [PATCH mm-hotfixes] mm/huge_memory: fix memory corruption on huge zero page move

[Andrew Morton]

On Mon, 2 Mar 2026 17:19:15 +0000 Lorenzo Stoakes <lorenzo.stoakes@oracle.com> wrote:

> On Mon, Mar 02, 2026 at 06:10:06PM +0100, David Hildenbrand (Arm) wrote:
> > There are already patches in flight:
> >
> > https://lore.kernel.org/r/aaBVaHs8rIkNcwM0@chrisdown.name
> 
> Yup, if people would actually cc the right people I'd not have wasted my
> afternoon.
> 
> Replying there.
> 

I'm not sure where this is headed but I'll queue this patch for now, to 
get it some testing, to lessen duplicated bug reports and to generally track things.

Re: [PATCH mm-hotfixes] mm/huge_memory: fix memory corruption on huge zero page move

[Lorenzo Stoakes]

On Mon, Mar 02, 2026 at 12:50:32PM -0800, Andrew Morton wrote:
> On Mon, 2 Mar 2026 17:19:15 +0000 Lorenzo Stoakes <lorenzo.stoakes@oracle.com> wrote:
>
> > On Mon, Mar 02, 2026 at 06:10:06PM +0100, David Hildenbrand (Arm) wrote:
> > > There are already patches in flight:
> > >
> > > https://lore.kernel.org/r/aaBVaHs8rIkNcwM0@chrisdown.name
> >
> > Yup, if people would actually cc the right people I'd not have wasted my
> > afternoon.
> >
> > Replying there.
> >
>
> I'm not sure where this is headed but I'll queue this patch for now, to
> get it some testing, to lessen duplicated bug reports and to generally track things.

Hi Andrew -

We should drop my patch and take Chris's.

However, life isn't so simple :) as there's been some confusion with his series
since his email client seemed to mess up somehow and a couple patches need to be
squashed on one another.

So to make life easier, I enclose a squashed patch (combining the commit
message from both), with my R-b, T-b tags attached, please replace my patch
with this one.

Also, could you make this into a 2 patch series with
https://lore.kernel.org/linux-mm/aaBWG4fajXXbjpVN@chrisdown.name/ _after_ this
one?

mm-unstable (I'm not sure why we're rushing changes there so quick...) had
_only_ this regression test patch in it, so we were left with a kernel that
splatted on running mm selftests, which is not what we want.

(It seems to me this is what mm-new is for, and it seems we have some holes in
our testing still since this got thorugh).

Please drop the patch you have for this already in mm-unstable and make sure we
only have it _after_ the fix is applied :)

Thanks, Lorenzo

----8<----
From fbabbc38c000308d2c621a4b3ec1f4bca1d71c1d Mon Sep 17 00:00:00 2001
From: Chris Down <chris@chrisdown.name>
Date: Tue, 3 Mar 2026 07:21:21 +0000
Subject: [PATCH] mm/huge_memory: Fix use of NULL folio in
 move_pages_huge_pmd()

move_pages_huge_pmd() handles UFFDIO_MOVE for both normal THPs and huge
zero pages. For the huge zero page path, src_folio is explicitly set to
NULL, and is used as a sentinel to skip folio operations like lock and
rmap.

In the huge zero page branch, src_folio is NULL, so folio_mk_pmd(NULL,
pgprot) passes NULL through folio_pfn() and page_to_pfn(). With
SPARSEMEM_VMEMMAP this silently produces a bogus PFN, installing a PMD
pointing to non-existent physical memory. On other memory models it is a
NULL dereference.

Use page_folio(src_page) to obtain the valid huge zero folio from the
page, which was obtained from pmd_page() and remains valid throughout.

After commit d82d09e48219 ("mm/huge_memory: mark PMD mappings of the
huge zero folio special"), moved huge zero PMDs must remain special so
vm_normal_page_pmd() continues to treat them as special mappings.

move_pages_huge_pmd() currently reconstructs the destination PMD in the
huge zero page branch, which drops PMD state such as pmd_special() on
architectures with CONFIG_ARCH_HAS_PTE_SPECIAL. As a result,
vm_normal_page_pmd() can treat the moved huge zero PMD as a normal page
and corrupt its refcount.

Instead of reconstructing the PMD from the folio, derive the destination
entry from src_pmdval after pmdp_huge_clear_flush(), then handle the PMD
metadata the same way move_huge_pmd() does for moved entries by marking
it soft-dirty and clearing uffd-wp.

Fixes: e3981db444a0 ("mm: add folio_mk_pmd()")
Cc: stable@vger.kernel.org
Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Tested-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Signed-off-by: Chris Down <chris@chrisdown.name>
---
 mm/huge_memory.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 8e2746ea74ad..c5d3e7bf6414 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2797,7 +2797,8 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm
 		_dst_pmd = pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
 	} else {
 		src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd);
-		_dst_pmd = folio_mk_pmd(src_folio, dst_vma->vm_page_prot);
+		_dst_pmd = move_soft_dirty_pmd(src_pmdval);
+		_dst_pmd = clear_uffd_wp_pmd(_dst_pmd);
 	}
 	set_pmd_at(mm, dst_addr, dst_pmd, _dst_pmd);

--
2.53.0

Re: [PATCH mm-hotfixes] mm/huge_memory: fix memory corruption on huge zero page move

[David Hildenbrand (Arm)]

On 3/3/26 08:25, Lorenzo Stoakes wrote:
> On Mon, Mar 02, 2026 at 12:50:32PM -0800, Andrew Morton wrote:
>> On Mon, 2 Mar 2026 17:19:15 +0000 Lorenzo Stoakes <lorenzo.stoakes@oracle.com> wrote:
>>
>>>
>>> Yup, if people would actually cc the right people I'd not have wasted my
>>> afternoon.
>>>
>>> Replying there.
>>>
>>
>> I'm not sure where this is headed but I'll queue this patch for now, to
>> get it some testing, to lessen duplicated bug reports and to generally track things.
> 
> Hi Andrew -
> 
> We should drop my patch and take Chris's.
> 
> However, life isn't so simple :) as there's been some confusion with his series
> since his email client seemed to mess up somehow and a couple patches need to be
> squashed on one another.
> 
> So to make life easier, I enclose a squashed patch (combining the commit
> message from both), with my R-b, T-b tags attached, please replace my patch
> with this one.
> 
> Also, could you make this into a 2 patch series with
> https://lore.kernel.org/linux-mm/aaBWG4fajXXbjpVN@chrisdown.name/ _after_ this
> one?
> 
> mm-unstable (I'm not sure why we're rushing changes there so quick...) had
> _only_ this regression test patch in it, so we were left with a kernel that
> splatted on running mm selftests, which is not what we want.
> 
> (It seems to me this is what mm-new is for, and it seems we have some holes in
> our testing still since this got thorugh).
> 
> Please drop the patch you have for this already in mm-unstable and make sure we
> only have it _after_ the fix is applied :)
> 
> Thanks, Lorenzo
> 
> ----8<----
> From fbabbc38c000308d2c621a4b3ec1f4bca1d71c1d Mon Sep 17 00:00:00 2001
> From: Chris Down <chris@chrisdown.name>
> Date: Tue, 3 Mar 2026 07:21:21 +0000
> Subject: [PATCH] mm/huge_memory: Fix use of NULL folio in
>  move_pages_huge_pmd()
> 
> move_pages_huge_pmd() handles UFFDIO_MOVE for both normal THPs and huge
> zero pages. For the huge zero page path, src_folio is explicitly set to
> NULL, and is used as a sentinel to skip folio operations like lock and
> rmap.
> 
> In the huge zero page branch, src_folio is NULL, so folio_mk_pmd(NULL,
> pgprot) passes NULL through folio_pfn() and page_to_pfn(). With
> SPARSEMEM_VMEMMAP this silently produces a bogus PFN, installing a PMD
> pointing to non-existent physical memory. On other memory models it is a
> NULL dereference.
> 
> Use page_folio(src_page) to obtain the valid huge zero folio from the
> page, which was obtained from pmd_page() and remains valid throughout.
> 
> After commit d82d09e48219 ("mm/huge_memory: mark PMD mappings of the
> huge zero folio special"), moved huge zero PMDs must remain special so
> vm_normal_page_pmd() continues to treat them as special mappings.
> 
> move_pages_huge_pmd() currently reconstructs the destination PMD in the
> huge zero page branch, which drops PMD state such as pmd_special() on
> architectures with CONFIG_ARCH_HAS_PTE_SPECIAL. As a result,
> vm_normal_page_pmd() can treat the moved huge zero PMD as a normal page
> and corrupt its refcount.
> 
> Instead of reconstructing the PMD from the folio, derive the destination
> entry from src_pmdval after pmdp_huge_clear_flush(), then handle the PMD
> metadata the same way move_huge_pmd() does for moved entries by marking
> it soft-dirty and clearing uffd-wp.
> 
> Fixes: e3981db444a0 ("mm: add folio_mk_pmd()")
> Cc: stable@vger.kernel.org
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
> Tested-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
> Signed-off-by: Chris Down <chris@chrisdown.name>
> ---

Acked-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David