[RFC PATCH v5 00/10] mm: Hot page tracking and promotion infrastructure

[Bharata B Rao <bharata@amd.com>]

Hi,

This is v5 of pghot, a hot-page tracking and promotion subsystem.
The major change in v5 is reducing the default hotness record size
to 1 byte per PFN and adding an optional precision mode
(CONFIG_PGHOT_PRECISE) that uses 4 bytes per PFN.

This patchset introduces a new subsystem for hot page tracking and
promotion (pghot) with the following goals:

- Unify hot page detection from multiple sources like hint faults,
  page table scans, hardware hints (AMD IBS).
- Decouple detection from migration.
- Centralize promotion logic via per-lower-tier-node kmigrated kernel
  thread.
- Move promotion rate‑limiting and related logic used by
  numa_balancing=2 (current NUMA balancing–based promotion) from
  the scheduler to pghot for broader reuse.
  
Currently, multiple kernel subsystems detect page accesses independently.
This patchset consolidates accesses from these mechanisms by providing:

- A common API for reporting page accesses.
- Shared infrastructure for tracking hotness at PFN granularity.
- Per-lower-tier-node kernel threads for promoting pages.

Here is a brief summary of how this subsystem works:

- Tracks frequency and last access time.
- Additionally, the accessing NUMA node ID (NID) for each recorded
  access is also tracked in the precision mode.
- These hotness parameters are maintained in a per-PFN hotness record
  within the existing mem_section data structure.
  - In default mode, one byte (u8) is used for hotness record. 5 bits are
    used to store time and bucketing scheme is used to represent a total
    access time up to 4s with HZ=1000. Default toptier NID (0) is used as
    the target for promotion which can be changed via debugfs tunable.
  - In precision mode, 4 bytes (u32) are used for each hotness record.
    14 bits are used to store time which can represent around 16s
    with HZ=1000.
- Classifies pages as hot based on configurable thresholds.
- Pages classified as hot are marked as ready for migration using the
  ready bit. Both modes use MSB of the hotness record as ready bit.
- Per-lower-tier-node kmigrated threads periodically scan the PFNs of
  lower-tier nodes, checking for the migration-ready bit to perform
  batched migrations. Interval between successive scans and batching
  value are configurable via debugfs tunables.

Memory overhead
---------------
Default mode: 1 byte per lower-tier PFN. For a 1TB lower-tier memory
this amounts to 256MB overhead (assuming 4K pages)

Precision mode: 4 bytes per lower-tier PFN. For a 1TB of lower memory
this amounts to 1G overhead.

Bit layout of hotness record
----------------------------
Default mode
- Bits 0-1: Frequency (2bits, 4 access samples)
- Bits 2-6: Bucketed time (5bits, up to 4s with HZ=1000)
- Bit 7: Migration ready bit

Precision mode
- Bits 0-9: Target NID (10 bits)
- Bits 10-12: Frequency (3bits, 8 access samples)
- Bits 13-26: Time (14bits, up to 16s with HZ=1000)
- Bits 27-30: Reserved
- Bit 31: Migration ready bit

Integrated sources
------------------
1. IBS - Instruction Based Sampling, hardware based sampling
   mechanism present on AMD CPUs.
2. klruscand - PTE‑A bit scanning built on MGLRU’s walk helpers.
3. NUMA Balancing (Tiering mode)
4. folio_mark_accessed() - Page cache access tracking (unmapped
   page cache pages)

Changes in v5
=============
- Significant reduction in memory overhead for storing per-PFN
  hotness data
- Two modes of operation (default and precision mode). The code
  which is specific to each implementation is moved to its own
  individual file.
- Many bug fixes, code cleanups and code reorganization.

Results
=======
TODO: Will post benchmark nubmers as reply to this patchset soon.

This v5 patchset applies on top of upstream commit 4941a17751c9 and
can be fetched from:

https://github.com/AMDESE/linux-mm/tree/bharata/pghot-rfcv5

v4: https://lore.kernel.org/linux-mm/20251206101423.5004-1-bharata@amd.com/
v3: https://lore.kernel.org/linux-mm/20251110052343.208768-1-bharata@amd.com/
v2: https://lore.kernel.org/linux-mm/20250910144653.212066-1-bharata@amd.com/
v1: https://lore.kernel.org/linux-mm/20250814134826.154003-1-bharata@amd.com/
v0: https://lore.kernel.org/linux-mm/20250306054532.221138-1-bharata@amd.com/

Bharata B Rao (7):
  mm: migrate: Allow misplaced migration without VMA
  mm: Hot page tracking and promotion
  mm: pghot: Precision mode for pghot
  mm: sched: move NUMA balancing tiering promotion to pghot
  x86: ibs: In-kernel IBS driver for memory access profiling
  x86: ibs: Enable IBS profiling for memory accesses
  mm: pghot: Add folio_mark_accessed() as hotness source

Gregory Price (1):
  migrate: Add migrate_misplaced_folios_batch()

Kinsey Ho (2):
  mm: mglru: generalize page table walk
  mm: klruscand: use mglru scanning for page promotion

 Documentation/admin-guide/mm/pghot.txt |  89 +++++
 arch/x86/events/amd/ibs.c              |  10 +
 arch/x86/include/asm/entry-common.h    |   3 +
 arch/x86/include/asm/hardirq.h         |   2 +
 arch/x86/include/asm/msr-index.h       |  16 +
 arch/x86/mm/Makefile                   |   1 +
 arch/x86/mm/ibs.c                      | 349 +++++++++++++++++
 include/linux/migrate.h                |   6 +
 include/linux/mmzone.h                 |  26 ++
 include/linux/pghot.h                  | 142 +++++++
 include/linux/vm_event_item.h          |  26 ++
 kernel/sched/debug.c                   |   1 -
 kernel/sched/fair.c                    | 152 +-------
 mm/Kconfig                             |  46 +++
 mm/Makefile                            |   7 +
 mm/huge_memory.c                       |  26 +-
 mm/internal.h                          |   4 +
 mm/klruscand.c                         | 110 ++++++
 mm/memory.c                            |  31 +-
 mm/migrate.c                           |  41 +-
 mm/mm_init.c                           |  10 +
 mm/pghot-default.c                     |  73 ++++
 mm/pghot-precise.c                     |  70 ++++
 mm/pghot-tunables.c                    | 196 ++++++++++
 mm/pghot.c                             | 505 +++++++++++++++++++++++++
 mm/swap.c                              |   8 +
 mm/vmscan.c                            | 181 ++++++---
 mm/vmstat.c                            |  26 ++
 28 files changed, 1917 insertions(+), 240 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/pghot.txt
 create mode 100644 arch/x86/mm/ibs.c
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/klruscand.c
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot-precise.c
 create mode 100644 mm/pghot-tunables.c
 create mode 100644 mm/pghot.c

-- 
2.34.1