[RFC PATCH 0/4] memory tiering fairness by per-cgroup control of promotion and demotion

[kaiyang2@cs.cmu.edu]

From: Kaiyang Zhao <kaiyang2@cs.cmu.edu>

Currently in Linux, there is no concept of fairness in memory tiering. Depending
on the memory usage and access patterns of other colocated applications, an
application cannot be sure of how much memory in which tier it will get, and how
much its performance will suffer or benefit.

Fairness is, however, important in a multi-tenant system. For example, an
application may need to meet a certain tail latency requirement, which can be
difficult to satisfy without x amount of frequently accessed pages in top-tier
memory. Similarly, an application may want to declare a minimum throughput when
running on a system for capacity planning purposes, but without fairness
controls in memory tiering its throughput can fluctuate wildly as other
applications come and go on the system.

In this proposal, we amend the memory.low control in memcg to protect a cgroup’s
memory usage in top-tier memory. A low protection for top-tier memory is scaled
proportionally to the ratio of top-tier memory and total memory on the system.
The protection is then applied to reclaim for top-tier memory. Promotion by NUMA
balancing is also throttled through reduced scanning window when top-tier memory
is contended and the cgroup is over its protection.

Experiments we did with microbenchmarks exhibiting a range of memory access
patterns and memory size confirmed that when top-tier memory is contended, the
system moves towards a stable memory distribution where each cgroup’s memory
usage in local DRAM converges to the protected amounts.

One notable missing part in the patches is determining which NUMA nodes have
top-tier memory; currently they use hardcoded node 0 for top-tier memory and
node 1 for a CPU-less node backed by CXL memory. We’re working on removing
this artifact and correctly applying to top-tier nodes in the system.

Your feedback is greatly appreciated!

Kaiyang Zhao (4):
  Add get_cgroup_local_usage for estimating the top-tier memory usage
  calculate memory.low for the local node and track its usage
  use memory.low local node protection for local node reclaim
  reduce NUMA balancing scan size of cgroups over their local memory.low

 include/linux/memcontrol.h   | 25 ++++++++-----
 include/linux/page_counter.h | 16 ++++++---
 kernel/sched/fair.c          | 54 +++++++++++++++++++++++++---
 mm/hugetlb_cgroup.c          |  4 +--
 mm/memcontrol.c              | 68 ++++++++++++++++++++++++++++++------
 mm/page_counter.c            | 52 +++++++++++++++++++++------
 mm/vmscan.c                  | 19 +++++++---
 7 files changed, 192 insertions(+), 46 deletions(-)

-- 
2.43.0