LPA2 on non-LPA2 hardware broken with 16K pages

[Asahi Lina <lina@asahilina.net>]

Hi,

I ran into this with the Asahi Linux downstream kernel, based on v6.9.9,
but I believe the problem is also still upstream. The issue seems to be
an interaction between folding one page table level at compile time and
another one at runtime.

With this config, we have:

CONFIG_PGTABLE_LEVELS=4
PAGE_SHIFT=14
PMD_SHIFT=25
PUD_SHIFT=36
PGDIR_SHIFT=47
pgtable_l5_enabled() == false (compile time)
pgtable_l4_enabled() == false (runtime, due to no LPA2)

With p4d folded at compile-time, and pud folded at runtime when LPA2 is
not supported.

With this setup, pgd_offset() is broken since the pgd is actually
supposed to become a pud but the shift is wrong, as it is set at compile
time:

#define pgd_index(a)  (((a) >> PGDIR_SHIFT) & (PTRS_PER_PGD - 1))

static inline pgd_t *pgd_offset_pgd(pgd_t *pgd, unsigned long address)
{
        return (pgd + pgd_index(address));
};

Then we follow the gup logic (abbreviated):

gup_pgd_range:
    pgdp = pgd_offset(current->mm, addr);
    pgd_t pgd = READ_ONCE(*pgdp);

At this point, pgd is just the 0th entry of the top level page table
(since those extra address bits will always be 0 for valid 47-bit user
addresses).

p4d then gets folded via pgtable-nop4d.h:

gup_p4d_range:
    p4dp = p4d_offset_lockless(pgdp, pgd, addr);
         = p4d_offset(&(pgd), address)
         = &pgd
    p4d_t p4d = READ_ONCE(*p4dp);

Now we have p4dp = stack address of pgd, and p4d = pgd.

gup_pud_range:
    pudp = pud_offset_lockless(p4dp, p4d, addr);
         -> if (!pgtable_l4_enabled())
           = p4d_to_folded_pud(p4dp, addr);
           = (pud_t *)PTR_ALIGN_DOWN(p4dp, PAGE_SIZE) + pud_index(addr);
    pud_t pud = READ_ONCE(*pudp);

Which is bad pointer math because it only works if p4dp points to a real
page table entry inside a page table, not a single u64 stack address.

This causes random oopses in internal_get_user_pages_fast and related
codepaths.

~~ Lina