Re: [PATCH 00/11] mm/hugetlb: Eliminate fake head pages from vmemmap optimization

[Kiryl Shutsemau <kas@kernel.org>]

On Tue, Dec 09, 2025 at 10:20:14AM -0800, Frank van der Linden wrote:
> On Fri, Dec 5, 2025 at 11:44 AM Kiryl Shutsemau <kas@kernel.org> wrote:
> >
> > This series removes "fake head pages" from the HugeTLB vmemmap
> > optimization (HVO) by changing how tail pages encode their relationship
> > to the head page.
> >
> > It simplifies compound_head() and page_ref_add_unless(). Both are in the
> > hot path.
> >
> > Background
> > ==========
> >
> > HVO reduces memory overhead by freeing vmemmap pages for HugeTLB pages
> > and remapping the freed virtual addresses to a single physical page.
> > Previously, all tail page vmemmap entries were remapped to the first
> > vmemmap page (containing the head struct page), creating "fake heads" -
> > tail pages that appear to have PG_head set when accessed through the
> > deduplicated vmemmap.
> >
> > This required special handling in compound_head() to detect and work
> > around fake heads, adding complexity and overhead to a very hot path.
> >
> > New Approach
> > ============
> >
> > For architectures/configs where sizeof(struct page) is a power of 2 (the
> > common case), this series changes how position of the head page is encoded
> > in the tail pages.
> >
> > Instead of storing a pointer to the head page, the ->compound_info
> > (renamed from ->compound_head) now stores a mask.
> >
> > The mask can be applied to any tail page's virtual address to compute
> > the head page address. Critically, all tail pages of the same order now
> > have identical compound_info values, regardless of which compound page
> > they belong to.
> >
> > This enables a key optimization: instead of remapping tail vmemmap
> > entries to the head page (creating fake heads), we remap them to a
> > shared, pre-initialized vmemmap_tail page per hstate. The head page
> > gets its own dedicated vmemmap page, eliminating fake heads entirely.
> >
> > Benefits
> > ========
> >
> > 1. Smaller generated code. On defconfig, I see ~15K reduction of text
> >    in vmlinux:
> >
> >    add/remove: 6/33 grow/shrink: 54/262 up/down: 6130/-21922 (-15792)
> >
> > 2. Simplified compound_head(): No fake head detection needed. The
> >    function is now branchless for power-of-2 struct page sizes.
> >
> > 3. Eliminated race condition: The old scheme required synchronize_rcu()
> >    to coordinate between HVO remapping and speculative PFN walkers that
> >    might write to fake heads. With the head page always in writable
> >    memory, this synchronization is unnecessary.
> >
> > 4. Removed static key: hugetlb_optimize_vmemmap_key is no longer needed
> >    since compound_head() no longer has HVO-specific branches.
> >
> > 5. Cleaner architecture: The vmemmap layout is now straightforward -
> >    head page has its own vmemmap, tails share a read-only template.
> >
> > I had hoped to see performance improvement, but my testing thus far has
> > shown either no change or only a slight improvement within the noise.
> >
> > Series Organization
> > ===================
> >
> > Patches 1-3: Preparatory refactoring
> >   - Change prep_compound_tail() interface to take order
> >   - Rename compound_head field to compound_info
> >   - Move set/clear_compound_head() near compound_head()
> >
> > Patch 4: Core encoding change
> >   - Implement mask-based encoding for power-of-2 struct page
> >
> > Patches 5-6: HVO restructuring
> >   - Refactor vmemmap_walk to support separate head/tail pages
> >   - Introduce per-hstate vmemmap_tail, eliminate fake heads
> >
> > Patches 7-9: Cleanup
> >   - Remove fake head checks from compound_head(), PageTail(), etc.
> >   - Remove VMEMMAP_SYNCHRONIZE_RCU and synchronize_rcu() calls
> >   - Remove hugetlb_optimize_vmemmap_key static key
> >
> > Patch 10: Optimization
> >   - Implement branchless compound_head() for power-of-2 case
> >
> > Patch 11: Documentation
> >   - Update vmemmap_dedup.rst to reflect new architecture
> >
> > Kiryl Shutsemau (11):
> >   mm: Change the interface of prep_compound_tail()
> >   mm: Rename the 'compound_head' field in the 'struct page' to
> >     'compound_info'
> >   mm: Move set/clear_compound_head() to compound_head()
> >   mm: Rework compound_head() for power-of-2 sizeof(struct page)
> >   mm/hugetlb: Refactor code around vmemmap_walk
> >   mm/hugetlb: Remove fake head pages
> >   mm: Drop fake head checks and fix a race condition
> >   hugetlb: Remove VMEMMAP_SYNCHRONIZE_RCU
> >   mm/hugetlb: Remove hugetlb_optimize_vmemmap_key static key
> >   mm: Remove the branch from compound_head()
> >   hugetlb: Update vmemmap_dedup.rst
> >
> >  .../admin-guide/kdump/vmcoreinfo.rst          |   2 +-
> >  Documentation/mm/vmemmap_dedup.rst            |  62 ++---
> >  include/linux/hugetlb.h                       |   3 +
> >  include/linux/mm_types.h                      |  20 +-
> >  include/linux/page-flags.h                    | 163 +++++-------
> >  include/linux/page_ref.h                      |   8 +-
> >  include/linux/types.h                         |   2 +-
> >  kernel/vmcore_info.c                          |   2 +-
> >  mm/hugetlb.c                                  |   8 +-
> >  mm/hugetlb_vmemmap.c                          | 245 ++++++++----------
> >  mm/hugetlb_vmemmap.h                          |   4 +-
> >  mm/internal.h                                 |  11 +-
> >  mm/mm_init.c                                  |   2 +-
> >  mm/page_alloc.c                               |   4 +-
> >  mm/slab.h                                     |   2 +-
> >  mm/util.c                                     |  15 +-
> >  16 files changed, 242 insertions(+), 311 deletions(-)
> >
> > --
> > 2.51.2
> >
> >
> 
> I love this in general - I've always disliked the fake head
> construction (though I understand the reason behind it).
> 
> However, it seems like you didn't add support to vmemmap_populate_hvo,
> as far as I can tell. That's the function that is used to do HVO early
> on bootmem (memblock) allocated 'gigantic' pages. So I think that
> would break with this patch.

Ouch. Good catch. Will fix.

> Could you add support there too? I don't think it would be hard to.
> While at it, you could also do it for vmemmap_populate_hugepages to
> support devdax :-)

Yeah, DAX was on my radar. I will see if it makes sense to make part of
this patchset or make an follow up.

Other thing I want to change is that we probably want to make
vmemmap_tails per node, so each node would use local memory for it.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov