[RFC PATCH 0/4] mm, bpf: BPF based THP adjustment

[Yafang Shao <laoar.shao@gmail.com>]

In our container environment, we aim to enable THP selectively—allowing
specific services to use it while restricting others. This approach is
driven by the following considerations:

1. Memory Fragmentation
   THP can lead to increased memory fragmentation, so we want to limit its
   use across services.
2. Performance Impact
   Some services see no benefit from THP, making its usage unnecessary.
3. Performance Gains
   Certain workloads, such as machine learning services, experience
   significant performance improvements with THP, so we enable it for them
   specifically. 

Since multiple services run on a single host in a containerized environment,
enabling THP globally is not ideal. Previously, we set THP to madvise,
allowing selected services to opt in via MADV_HUGEPAGE. However, this
approach had limitation:

- Some services inadvertently used madvise(MADV_HUGEPAGE) through
  third-party libraries, bypassing our restrictions.

To address this issue, we initially hooked the __x64_sys_madvise() syscall,
which is error-injectable, to blacklist unwanted services. While this
worked, it was error-prone and ineffective for services needing always mode,
as modifying their code to use madvise was impractical.

To achieve finer-grained control, we introduced an fmod_ret-based solution.
Now, we dynamically adjust THP settings per service by hooking
hugepage_global_{enabled,always}() via BPF. This allows us to set THP to
enable or disable on a per-service basis without global impact.

The hugepage_global_{enabled,always}() functions currently share the same
BPF hook, which limits THP configuration to either always or never. While
this suffices for our specific use cases, full support for all three modes
(always, madvise, and never) would require splitting them into separate
hooks.

This is the initial RFC patch—feedback is welcome!

Yafang Shao (4):
  mm: move hugepage_global_{enabled,always}() to internal.h
  mm: pass VMA parameter to hugepage_global_{enabled,always}()
  mm: add BPF hook for THP adjustment
  selftests/bpf: Add selftest for THP adjustment

 include/linux/huge_mm.h                       |  54 +-----
 mm/Makefile                                   |   3 +
 mm/bpf.c                                      |  36 ++++
 mm/bpf.h                                      |  21 +++
 mm/huge_memory.c                              |  50 ++++-
 mm/internal.h                                 |  21 +++
 mm/khugepaged.c                               |  18 +-
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/thp_adjust.c     | 176 ++++++++++++++++++
 .../selftests/bpf/progs/test_thp_adjust.c     |  32 ++++
 10 files changed, 344 insertions(+), 68 deletions(-)
 create mode 100644 mm/bpf.c
 create mode 100644 mm/bpf.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c

-- 
2.43.5