Re: [PATCHv2] mm/khugepaged: Do not fail collapse_pte_mapped_thp() on SCAN_PMD_NULL

["Zach O'Keefe" <zokeefe@google.com>]

On Tue, Sep 16, 2025 at 2:54 AM Lorenzo Stoakes
<lorenzo.stoakes@oracle.com> wrote:
>
> On Mon, Sep 15, 2025 at 02:52:53PM +0100, Kiryl Shutsemau wrote:
> > From: Kiryl Shutsemau <kas@kernel.org>
> >
> > MADV_COLLAPSE on a file mapping behaves inconsistently depending on if
> > PMD page table is installed or not.
> >
> > Consider following example:
> >
> >       p = mmap(NULL, 2UL << 20, PROT_READ | PROT_WRITE,
> >                MAP_SHARED, fd, 0);
> >       err = madvise(p, 2UL << 20, MADV_COLLAPSE);
> >
> > fd is a populated tmpfs file.
> >
> > The result depends on the address that the kernel returns on mmap().
> > If it is located in an existing PMD table, the madvise() will succeed.
> > However, if the table does not exist, it will fail with -EINVAL.
> >
> > This occurs because find_pmd_or_thp_or_none() returns SCAN_PMD_NULL when
> > a page table is missing, which causes collapse_pte_mapped_thp() to fail.
> >
> > SCAN_PMD_NULL and SCAN_PMD_NONE should be treated the same in
> > collapse_pte_mapped_thp(): install the PMD leaf entry and allocate page
> > tables as needed.
> >
> > Signed-off-by: Kiryl Shutsemau <kas@kernel.org>

So, since we are trying to aim for consistency here, I think we ought
to also support the anonymous case.

I don't have a patch, but can spot at least two things we'd need to adjust:

First, we are defeated by the check in __thp_vma_allowable_orders();

        /*
         * THPeligible bit of smaps should show 1 for proper VMAs even
         * though anon_vma is not initialized yet.
         *
         * Allow page fault since anon_vma may be not initialized until
         * the first page fault.
         */
        if (!vma->anon_vma)
                return (smaps || in_pf) ? orders : 0;

I think we can probably just delete that check, but would need to confirm.

And second, madvise_collapse() doesn't route SCAN_PMD_NULL to
collapse_pte_mapped_thp(). I think we just need to audit places where
we return this code, to make sure it's faithfully describing a
situation where we can go ahead and install a new pmd. As a hasty
check, the return codes in check_pmd_state() don't look to follow
that, with !present and pmd_bad() returning SCAN_PMD_NULL. Likewise,
there are many underlying failure reasons for
pte_offset_map_ro_nolock()=>___pte_offset_map() that aren't "no PMD
entry".

WDYT?

> There was a v1 with tags, you've not propagated any of them? Did you feel
> the change was enough to remove them?
>
> Anyway, LGTM so:
>
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
>
> > ---
> >
> > v2:
> >  - Modify set_huge_pmd() instead of introducing install_huge_pmd();
> >
> > ---
> >  mm/khugepaged.c | 20 +++++++++++++++++++-
> >  1 file changed, 19 insertions(+), 1 deletion(-)
> >
> > diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> > index b486c1d19b2d..986718599355 100644
> > --- a/mm/khugepaged.c
> > +++ b/mm/khugepaged.c
> > @@ -1472,15 +1472,32 @@ static void collect_mm_slot(struct khugepaged_mm_slot *mm_slot)
> >  static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,
> >                       pmd_t *pmdp, struct folio *folio, struct page *page)
> >  {
> > +     struct mm_struct *mm = vma->vm_mm;
> >       struct vm_fault vmf = {
> >               .vma = vma,
> >               .address = addr,
> >               .flags = 0,
> > -             .pmd = pmdp,
> >       };
> > +     pgd_t *pgdp;
> > +     p4d_t *p4dp;
> > +     pud_t *pudp;
> >
> >       mmap_assert_locked(vma->vm_mm);
>
> NIT: you have mm as a local var should use here too. Not a big deal though
> obviously...
>
> >
> > +     if (!pmdp) {
> > +             pgdp = pgd_offset(mm, addr);
> > +             p4dp = p4d_alloc(mm, pgdp, addr);
> > +             if (!p4dp)
> > +                     return SCAN_FAIL;
> > +             pudp = pud_alloc(mm, p4dp, addr);
> > +             if (!pudp)
> > +                     return SCAN_FAIL;
> > +             pmdp = pmd_alloc(mm, pudp, addr);
> > +             if (!pmdp)
> > +                     return SCAN_FAIL;
> > +     }
> > +
> > +     vmf.pmd = pmdp;
> >       if (do_set_pmd(&vmf, folio, page))
> >               return SCAN_FAIL;
> >
> > @@ -1556,6 +1573,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
> >       switch (result) {
> >       case SCAN_SUCCEED:
> >               break;
> > +     case SCAN_PMD_NULL:
> >       case SCAN_PMD_NONE:
> >               /*
> >                * All pte entries have been removed and pmd cleared.
> > --
> > 2.50.1
> >
>