Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kalesh Singh <kaleshsingh@google.com>]

On Thu, Feb 19, 2026 at 7:39 AM David Hildenbrand (Arm)
<david@kernel.org> wrote:
>
> On 2/19/26 16:08, Kiryl Shutsemau wrote:
> > No, there's no new hardware (that I know of). I want to explore what page size
> > means.
> >
> > The kernel uses the same value - PAGE_SIZE - for two things:
> >
> >    - the order-0 buddy allocation size;
> >
> >    - the granularity of virtual address space mapping;
> >
> > I think we can benefit from separating these two meanings and allowing
> > order-0 allocations to be larger than the virtual address space covered by a
> > PTE entry.
> >
> > The main motivation is scalability. Managing memory on multi-terabyte
> > machines in 4k is suboptimal, to say the least.
> >
> > Potential benefits of the approach (assuming 64k pages):
> >
> >    - The order-0 page size cuts struct page overhead by a factor of 16. From
> >      ~1.6% of RAM to ~0.1%;
> >
> >    - TLB wins on machines with TLB coalescing as long as mapping is naturally
> >      aligned;
> >
> >    - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
> >
> >    - 1G pages are within possibility for the buddy allocator - order-14
> >      allocation. It can open the road to 1G THPs.
> >
> >    - As with THP, fewer pages - less pressure on the LRU lock;
> >
> >    - ...
> >
> > The trade-off is memory waste (similar to what we have on architectures with
> > native 64k pages today) and complexity, mostly in the core-MM code.
> >
> > == Design considerations ==
> >
> > I want to split PAGE_SIZE into two distinct values:
> >
> >    - PTE_SIZE defines the virtual address space granularity;
> >
> >    - PG_SIZE defines the size of the order-0 buddy allocation;
> >
> > PAGE_SIZE is only defined if PTE_SIZE == PG_SIZE. It will flag which code
> > requires conversion, and keep existing code working while conversion is in
> > progress.
> >
> > The same split happens for other page-related macros: mask, shift,
> > alignment helpers, etc.
> >
> > PFNs are in PTE_SIZE units.
> >
> > The buddy allocator and page cache (as well as all I/O) operate in PG_SIZE
> > units.
> >
> > Userspace mappings are maintained with PTE_SIZE granularity. No ABI changes
> > for userspace. But we might want to communicate PG_SIZE to userspace to
> > get the optimal results for userspace that cares.
> >
> > PTE_SIZE granularity requires a substantial rework of page fault and VMA
> > handling:
> >
> >    - A struct page pointer and pgprot_t are not enough to create a PTE entry.
> >      We also need the offset within the page we are creating the PTE for.
> >
> >    - Since the VMA start can be aligned arbitrarily with respect to the
> >      underlying page, vma->vm_pgoff has to be changed to vma->vm_pteoff,
> >      which is in PTE_SIZE units.
> >
> >    - The page fault handler needs to handle PTE_SIZE < PG_SIZE, including
> >      misaligned cases;
> >
> > Page faults into file mappings are relatively simple to handle as we
> > always have the page cache to refer to. So you can map only the part of the
> > page that fits in the page table, similarly to fault-around.
> >
> > Anonymous and file-CoW faults should also be simple as long as the VMA is
> > aligned to PG_SIZE in both the virtual address space and with respect to
> > vm_pgoff. We might waste some memory on the ends of the VMA, but it is
> > tolerable.
> >
> > Misaligned anonymous and file-CoW faults are a pain. Specifically, mapping
> > pages across a page table boundary. In the worst case, a page is mapped across
> > a PGD entry boundary and PTEs for the page have to be put in two separate
> > subtrees of page tables.
> >
> > A naive implementation would map different pages on different sides of a
> > page table boundary and accept the waste of one page per page table crossing.
> > The hope is that misaligned mappings are rare, but this is suboptimal.
> >
> > mremap(2) is the ultimate stress test for the design.
> >
> > On x86, page tables are allocated from the buddy allocator and if PG_SIZE
> > is greater than 4 KB, we need a way to pack multiple page tables into a
> > single page. We could use the slab allocator for this, but it would
> > require relocating the page-table metadata out of struct page.
>
> When discussing per-process page sizes with Ryan and Dev, I mentioned
> that having a larger emulated page size could be interesting for other
> architectures as well.
>
> That is, we would emulate a 64K page size on Intel for user space as
> well, but let the OS work with 4K pages.
>
> We'd only allocate+map large folios into user space + pagecache, but
> still allow for page tables etc. to not waste memory.
>
> So "most" of your allocations in the system would actually be at least
> 64k, reducing zone lock contention etc.
>
>
> It doesn't solve all the problems you wanted to tackle on your list
> (e.g., "struct page" overhead, which will be sorted out by memdescs).

Hi Kiryl,

I'd be interested to discuss this at LSFMM.

On Android, we have a separate but related use case: we emulate the
userspace page size on x86, primarily to enable app developers to
conduct compatibility testing of their apps for 16KB Android devices.
[1]

It mainly works by enforcing a larger granularity on the VMAs to
emulate a userspace page size, somewhat similar to what David
mentioned, while the underlying kernel still operates on a 4KB
granularity. [2]

IIUC the current design would not enfore the larger granularity /
alignment for VMAs to avoid breaking ABI. However, I'd be interest to
discuss whether it can be extended to cover this usecase as well.

[1]  https://developer.android.com/guide/practices/page-sizes#16kb-emulator
[2] https://source.android.com/docs/core/architecture/16kb-page-size/getting-started-cf-x86-64-pgagnostic

Thanks,
Kalesh




>
> --
> Cheers,
>
> David
>