[PATCH 00/49] mm: Generalize vmemmap optimization for DAX and HugeTLB

[Muchun Song <songmuchun@bytedance.com>]

Overview:
This patch series generalizes the HugeTLB Vmemmap Optimization (HVO)
into a generic vmemmap optimization framework that can be used by both
HugeTLB and DAX.

Background:
Currently, the vmemmap optimization feature is highly coupled with
HugeTLB. However, DAX also has similar requirements for optimizing vmemmap
pages to save memory. The current implementation has separate vmemmap
optimization paths for HugeTLB and DAX, leading to duplicated logic,
complex initialization sequences, and architecture-specific flags.

Implementation:
This series breaks down the optimization into a generic framework:
- Patch 1-6: Fix bugs related to sparse vmemmap initialization and DAX.
- Patch 7-13: Refactor the existing sparse vmemmap initialization.
- Patch 14-26: Decouple the vmemmap optimization from HugeTLB and
  introduce generic optimization macros and functions.
- Patch 27-39: Switch HugeTLB and DAX to use the generic framework.
- Patch 40-49: Clean up the old HVO-specific code and simplify it.

Benifit:
- When CONFIG_DEFERRED_STRUCT_PAGE_INIT is disabled, all struct pages
  utilizing HVO (HugeTLB Vmemmap Optimization) skip initialization in
  memmap_init, significantly accelerating boot times.
- All architectures supporting HVO benefit from the optimizations 
  provided by SPARSEMEM_VMEMMAP_PREINIT without requiring 
  architecture-specific adaptations.
- Device DAX struct page savings are further improved, saving an 
  additional 4KB of struct page memory for every 2MB huge page.
- Vmemmap tail pages used for Device DAX shared mappings are changed 
  from read-write to read-only, enhancing system security.
- HugeTLB and Device DAX now share a unified vmemmap optimization 
  framework, reducing long-term maintenance overhead.

Testing:
- Verification: Compiled and tested on x86 architecture.


Chengkaitao (1):
  mm: Convert vmemmap_p?d_populate() to static functions

Muchun Song (48):
  mm/sparse: fix vmemmap accounting imbalance on memory hotplug error
  mm/sparse: add a @pgmap argument to memory deactivation paths
  mm/sparse: fix vmemmap page accounting for HVOed DAX
  mm/sparse: add a @pgmap parameter to arch vmemmap_populate()
  mm/sparse: fix missing architecture-specific page table sync for HVO
    DAX
  mm/mm_init: fix uninitialized pageblock migratetype for ZONE_DEVICE
    compound pages
  mm/mm_init: use pageblock_migratetype_init_range() in
    deferred_free_pages()
  mm: panic on memory allocation failure in sparse_init_nid()
  mm: move subsection_map_init() into sparse_init()
  mm: defer sparse_init() until after zone initialization
  mm: make set_pageblock_order() static
  mm: integrate sparse_vmemmap_init_nid_late() into sparse_init_nid()
  mm/cma: validate hugetlb CMA range by zone at reserve time
  mm/hugetlb: free cross-zone bootmem gigantic pages after allocation
  mm/hugetlb: initialize vmemmap optimization in early stage
  mm: remove sparse_vmemmap_init_nid_late()
  mm/mm_init: make __init_page_from_nid() static
  mm/sparse-vmemmap: remove the VMEMMAP_POPULATE_PAGEREF flag
  mm: rename vmemmap optimization macros to generic names
  mm/sparse: drop power-of-2 size requirement for struct mem_section
  mm/sparse: introduce compound page order to mem_section
  mm/mm_init: skip initializing shared tail pages for compound pages
  mm/sparse-vmemmap: initialize shared tail vmemmap page upon allocation
  mm/sparse-vmemmap: support vmemmap-optimizable compound page
    population
  mm/hugetlb: use generic vmemmap optimization macros
  mm: call memblocks_present() before HugeTLB initialization
  mm/hugetlb: switch HugeTLB to use generic vmemmap optimization
  mm: extract pfn_to_zone() helper
  mm/sparse-vmemmap: remove unused SPARSEMEM_VMEMMAP_PREINIT feature
  mm/hugetlb: remove HUGE_BOOTMEM_HVO flag and simplify pre-HVO logic
  mm/sparse-vmemmap: consolidate shared tail page allocation
  mm: introduce CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION
  mm/sparse-vmemmap: switch DAX to use generic vmemmap optimization
  mm/sparse-vmemmap: introduce section zone to struct mem_section
  powerpc/mm: use generic vmemmap_shared_tail_page() in compound vmemmap
  mm/sparse-vmemmap: unify DAX and HugeTLB vmemmap optimization
  mm/sparse-vmemmap: remap the shared tail pages as read-only
  mm/sparse-vmemmap: remove unused ptpfn argument
  mm/hugetlb_vmemmap: remove vmemmap_wrprotect_hvo() and related code
  mm/sparse: simplify section_vmemmap_pages()
  mm/sparse-vmemmap: introduce section_vmemmap_page_structs()
  powerpc/mm: rely on generic vmemmap_can_optimize() to simplify code
  mm/sparse-vmemmap: drop ARCH_WANT_OPTIMIZE_DAX_VMEMMAP and simplify
    checks
  mm/sparse-vmemmap: drop @pgmap parameter from vmemmap populate APIs
  mm/sparse: replace pgmap with order and zone in sparse_add_section()
  mm: redefine HVO as Hugepage Vmemmap Optimization
  Documentation/mm: restructure vmemmap_dedup.rst to reflect generalized
    HVO
  mm: consolidate struct page power-of-2 size checks for HVO

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/admin-guide/sysctl/vm.rst       |   2 +-
 Documentation/mm/vmemmap_dedup.rst            | 218 ++--------
 arch/powerpc/Kconfig                          |   1 -
 arch/powerpc/include/asm/book3s/64/radix.h    |  12 -
 arch/powerpc/mm/book3s64/radix_pgtable.c      | 114 +----
 arch/powerpc/mm/init_64.c                     |   1 +
 arch/riscv/Kconfig                            |   1 -
 arch/x86/Kconfig                              |   2 -
 fs/Kconfig                                    |   6 +-
 include/linux/hugetlb.h                       |   7 +-
 include/linux/memory_hotplug.h                |   2 +-
 include/linux/mm.h                            |  50 +--
 include/linux/mm_types.h                      |   2 +
 include/linux/mm_types_task.h                 |   4 +
 include/linux/mmzone.h                        | 143 ++++---
 include/linux/page-flags.h                    |  31 +-
 kernel/bounds.c                               |   2 +
 mm/Kconfig                                    |  20 +-
 mm/bootmem_info.c                             |   5 +-
 mm/cma.c                                      |   3 +-
 mm/hugetlb.c                                  | 143 +++----
 mm/hugetlb_cma.c                              |   3 +-
 mm/hugetlb_vmemmap.c                          | 237 +----------
 mm/hugetlb_vmemmap.h                          |  35 +-
 mm/internal.h                                 |  26 +-
 mm/memory_hotplug.c                           |  15 +-
 mm/mm_init.c                                  | 138 +++---
 mm/sparse-vmemmap.c                           | 392 +++++-------------
 mm/sparse.c                                   |  85 ++--
 mm/util.c                                     |   2 +-
 scripts/gdb/linux/mm.py                       |   6 +-
 32 files changed, 513 insertions(+), 1197 deletions(-)

-- 
2.20.1