[RFC PATCH V3 00/17] mm: slowtier page promotion based on PTE A bit

[Raghavendra K T <raghavendra.kt@amd.com>]

The current series has additional enhancements and comments' incorporation on top of
RFC V2.

This is an additional source of hot page generator to NUMAB, IBS [4], KMGLRUD [5].

Introduction:
=============
In the current hot page promotion, all the activities including the
process address space scanning, NUMA hint fault handling and page
migration is performed in the process context. i.e., scanning overhead is
borne by applications.

This RFC V2 patch series does slow-tier page promotion by using PTE Accessed
bit scanning. Scanning is done by a global kernel thread which routinely
scans all the processes' address spaces and checks for accesses by reading
the PTE A bit.

A separate migration thread migrates/promotes the pages to the top-tier
node based on a simple heuristic that uses top-tier scan/access information
of the mm.

Additionally based on the feedback, a prctl knob with a scalar value is
provided to control per task scanning.

Changes Since RFC V2:
===================
 - Enhanced logic to migrate on second access.

 - Using prctl scalar value to further tune the scanning efficiency.

 - Using of PFN instead of folio to record hot pages for easy integration
with kpromoted/kmigrated [4].

 - Rebasing on top of fork/exec changes in v6.16.

 - Revisiting mm_walk logic and folio validation based on Harry's comments.

 - Feedback from migration system to slowdown scanning when more migration failures
 happen.

 - Masami's comment on trace patch.

 - Bug fix to overnight idle system  crash due to incorrect kmemcache usage.

 - Enhanced target node finding logic to further obtain fallback nodes to migrate.
(TBD: This needs followup patch that actually does migration to fallback target nodes)

Changes since RFC V1:
=====================
- Addressing the review comments by Jonathan (Thank you for your closer
 reviews).

- Per mm migration list with separate lock to resolve race conditions/softlockups
reported by Davidlohr.

- Add one more filter before migration for LRU_GEN case to check whether
 folio is still hot.

- Rename kmmscand ==> kscand kmmmigrated ==> kmigrated (hopefully this
 gets merged into Bharat's upcoming migration thread)

Changes since RFC V0:
======================
- A separate migration thread is used for migration, thus alleviating need for
  multi-threaded scanning (at least as per tracing).

- A simple heuristic for target node calculation is added.

- prctl (David R) interface with scalar value is added to control per task scanning.

- Steve's comment on tracing incorporated.

- Davidlohr's reported bugfix.

- Initial scan delay similar to NUMAB1 mode added.

- Got rid of migration lock during mm_walk.

A note on per mm migration list using mm_slot:
=============================================
Using per mm migration list (mm_slot) has helped to reduce contention
 and thus easing mm teardown during process exit.

It also helps to tie PFN/folio with mm to make heuristics work better
 and further it would help to throttle migration per mm (OR process) (TBD).

A note on PTE A bit scanning:
============================
Major positive: Current patchset is able to cover all the process address
 space scanning effectively with simple algorithms to tune scan_size and
 scan_period.

Thanks to Jonathan, Davidlohr, David, Harry, Masami Steve for review feedback on RFCs.

Future plans:
================
Evaluate how integration with hotness monitoring subsystem works, OR
as a standalone integration with kmigrated API* of [4]

Results:
=======
Benchmark Cbench (by Bharata) to evaluate performance promotion in
slowtier system.

Benchmark allocates memory on both regular NUMA node and  slowtier node,
then does continuous access.
Goal: Finishing fixed numaber of access in less time

SUT: Genoa+ EPYC system

base 6.16 NUMAB2 (because this has the best perf)
patched 6.16 + current series

Time taken in sec (lower is better)
               base           patched
8GB            228            206
32GB           547            534
128GB          1100           920

Links:
[1] RFC V0: https://lore.kernel.org/all/20241201153818.2633616-1-raghavendra.kt@amd.com/
[2] RFC V1: https://lore.kernel.org/linux-mm/20250319193028.29514-1-raghavendra.kt@amd.com/
[3] RFC V2: https://lore.kernel.org/linux-mm/20250624055617.1291159-1-raghavendra.kt@amd.com/
[4] Hotpage detection and promotion: https://lore.kernel.org/linux-mm/20250814134826.154003-1-bharata@amd.com/T/#t
[5] MGLRU: https://lkml.org/lkml/2025/3/24/1458

Patch organization:
patch 1-5 initial skeleton for scanning and migration
patch 6: migration
patch 7-9: scanning optimizations
patch 10: target_node heuristic
patch 11: Migration failure feedback
patch 12-14: sysfs, vmstat and tracing
patch 15-16: prctl implementation and enhancements to scanning.
patch17: Fallback target node finding

Raghavendra K T (17):
  mm: Add kscand kthread for PTE A bit scan
  mm: Maintain mm_struct list in the system
  mm: Scan the mm and create a migration list
  mm/kscand: Add only hot pages to migration list
  mm: Create a separate kthread for migration
  mm/migration: migrate accessed folios to toptier node
  mm: Add throttling of mm scanning using scan_period
  mm: Add throttling of mm scanning using scan_size
  mm: Add initial scan delay
  mm: Add a heuristic to calculate target node
  mm/kscand: Implement migration failure feedback
  sysfs: Add sysfs support to tune scanning
  mm/vmstat: Add vmstat counters
  trace/kscand: Add tracing of scanning and migration
  prctl: Introduce new prctl to control scanning
  prctl: Fine tune scan_period with prctl scale param
  mm: Create a list of fallback target nodes

 Documentation/filesystems/proc.rst |    2 +
 fs/proc/task_mmu.c                 |    4 +
 include/linux/kscand.h             |   30 +
 include/linux/migrate.h            |    2 +
 include/linux/mm.h                 |   13 +
 include/linux/mm_types.h           |    7 +
 include/linux/vm_event_item.h      |   12 +
 include/trace/events/kmem.h        |   99 ++
 include/uapi/linux/prctl.h         |    7 +
 kernel/fork.c                      |    6 +
 kernel/sys.c                       |   25 +
 mm/Kconfig                         |    8 +
 mm/Makefile                        |    1 +
 mm/internal.h                      |    1 +
 mm/kscand.c                        | 1754 ++++++++++++++++++++++++++++
 mm/migrate.c                       |    2 +-
 mm/mmap.c                          |    2 +
 mm/vma_exec.c                      |    3 +
 mm/vmstat.c                        |   12 +
 19 files changed, 1989 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/kscand.h
 create mode 100644 mm/kscand.c


base-commit: 038d61fd642278bab63ee8ef722c50d10ab01e8f
-- 
2.34.1