[LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau <kas@kernel.org>]

No, there's no new hardware (that I know of). I want to explore what page size
means.

The kernel uses the same value - PAGE_SIZE - for two things:

  - the order-0 buddy allocation size;

  - the granularity of virtual address space mapping;

I think we can benefit from separating these two meanings and allowing
order-0 allocations to be larger than the virtual address space covered by a
PTE entry.

The main motivation is scalability. Managing memory on multi-terabyte
machines in 4k is suboptimal, to say the least.

Potential benefits of the approach (assuming 64k pages):

  - The order-0 page size cuts struct page overhead by a factor of 16. From
    ~1.6% of RAM to ~0.1%;

  - TLB wins on machines with TLB coalescing as long as mapping is naturally
    aligned;

  - Order-5 allocation is 2M, resulting in less pressure on the zone lock;

  - 1G pages are within possibility for the buddy allocator - order-14
    allocation. It can open the road to 1G THPs.

  - As with THP, fewer pages - less pressure on the LRU lock;

  - ...

The trade-off is memory waste (similar to what we have on architectures with
native 64k pages today) and complexity, mostly in the core-MM code.

== Design considerations ==

I want to split PAGE_SIZE into two distinct values:

  - PTE_SIZE defines the virtual address space granularity;

  - PG_SIZE defines the size of the order-0 buddy allocation;

PAGE_SIZE is only defined if PTE_SIZE == PG_SIZE. It will flag which code
requires conversion, and keep existing code working while conversion is in
progress.

The same split happens for other page-related macros: mask, shift,
alignment helpers, etc.

PFNs are in PTE_SIZE units.

The buddy allocator and page cache (as well as all I/O) operate in PG_SIZE
units.

Userspace mappings are maintained with PTE_SIZE granularity. No ABI changes
for userspace. But we might want to communicate PG_SIZE to userspace to
get the optimal results for userspace that cares.

PTE_SIZE granularity requires a substantial rework of page fault and VMA
handling:

  - A struct page pointer and pgprot_t are not enough to create a PTE entry.
    We also need the offset within the page we are creating the PTE for.

  - Since the VMA start can be aligned arbitrarily with respect to the
    underlying page, vma->vm_pgoff has to be changed to vma->vm_pteoff,
    which is in PTE_SIZE units.

  - The page fault handler needs to handle PTE_SIZE < PG_SIZE, including
    misaligned cases;

Page faults into file mappings are relatively simple to handle as we
always have the page cache to refer to. So you can map only the part of the
page that fits in the page table, similarly to fault-around.

Anonymous and file-CoW faults should also be simple as long as the VMA is
aligned to PG_SIZE in both the virtual address space and with respect to
vm_pgoff. We might waste some memory on the ends of the VMA, but it is
tolerable.

Misaligned anonymous and file-CoW faults are a pain. Specifically, mapping
pages across a page table boundary. In the worst case, a page is mapped across
a PGD entry boundary and PTEs for the page have to be put in two separate
subtrees of page tables.

A naive implementation would map different pages on different sides of a
page table boundary and accept the waste of one page per page table crossing.
The hope is that misaligned mappings are rare, but this is suboptimal.

mremap(2) is the ultimate stress test for the design.

On x86, page tables are allocated from the buddy allocator and if PG_SIZE
is greater than 4 KB, we need a way to pack multiple page tables into a
single page. We could use the slab allocator for this, but it would
require relocating the page-table metadata out of struct page.

Things I have not thought much about yet:

  - Accounting for wasted memory;

  - rmap;

  - mapcount;

  - A lot of arch-specific code;

  - <insert my blind spot here>;

== Status ==

I have a POC implementation on top of v6.17:

git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git pte_size

It is WIP and full of hacks I am trying to make sense of now.

It compiles with my minimalistic kernel config and can boot to a shell with
both 16k and 64k base page sizes. The shell doesn't crash immediately, but
sometimes I wonder why :P

The patchset is large:

 378 files changed, 3348 insertions(+), 3102 deletions(-)

and it is far from being complete.

== Goals ==

I want to get feedback for the overall design and possible ways to
upstream.

My plan is to submit an RFC-quality patchset before the summit.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov