Re: [RFC PATCH 0/6] mm/memcontrol: Make memcg limits tier-aware

Donet Tom
2026-03-24T10:31:13Z

Hi Josua

On 2/24/26 4:08 AM, Joshua Hahn wrote:
> Memory cgroups provide an interface that allow multiple workloads on a
> host to co-exist, and establish both weak and strong memory isolation
> guarantees. For large servers and small embedded systems alike, memcgs
> provide an effective way to provide a baseline quality of service for
> protected workloads.
>
> This works, because for the most part, all memory is equal (except for
> zram / zswap). Restricting a cgroup's memory footprint restricts how
> much it can hurt other workloads competing for memory. Likewise, setting
> memory.low or memory.min limits can provide weak and strong guarantees
> to the performance of a cgroup.
>
> However, on systems with tiered memory (e.g. CXL / compressed memory),
> the quality of service guarantees that memcg limits enforced become less
> effective, as memcg has no awareness of the physical location of its
> charged memory. In other words, a workload that is well-behaved within
> its memcg limits may still be hurting the performance of other
> well-behaving workloads on the system by hogging more than its
> "fair share" of toptier memory.
>
> Introduce tier-aware memcg limits, which scale memory.low/high to
> reflect the ratio of toptier:total memory the cgroup has access.
>
> Take the following scenario as an example:
> On a host with 3:1 toptier:lowtier, say 150G toptier, and 50Glowtier,
> setting a cgroup's limits to:
> memory.min: 15G
> memory.low: 20G
> memory.high: 40G
> memory.max: 50G
>
> Will be enforced at the toptier as:
> memory.min: 15G
> memory.toptier_low: 15G (20 * 150/200)
> memory.toptier_high: 30G (40 * 150/200)
> memory.max: 50G

Currently, the high and low thresholds are adjusted based on the ratio
of top-tier to total memory. One concern I see is that if the working
set size exceeds the top-tier high threshold, it could lead to frequent
demotions and promotions. Instead, would it make sense to introduce a
tunable knob to configure the top-tier high threshold?

Another concern is that if the lower-tier memory size is very large, the
cgroup may end up getting only a small portion of higher-tier memory.

>
> Let's say that there are 4 such cgroups on the host. Previously, it would
> be possible for 3 hosts to completely take over all of DRAM, while one
> cgroup could only access the lowtier memory. In the perspective of a
> tier-agnostic memcg limit enforcement, the three cgroups are all
> well-behaved, consuming within their memory limits.
>
> This is not to say that the scenario above is incorrect. In fact, for
> letting the hottest cgroups run in DRAM while pushing out colder cgroups
> to lowtier memory lets the system perform the most aggregate work total.
>
> But for other scenarios, the target might not be maximizing aggregate
> work, but maximizing the minimum performance guarantee for each
> individual workload (think hosts shared across different users, such as
> VM hosting services).
>
> To reflect these two scenarios, introduce a sysctl tier_aware_memcg,
> which allows the host to toggle between enforcing and overlooking
> toptier memcg limit breaches.
>
> This work is inspired & based off of Kaiyang Zhao's work from 2024 [1],
> where he referred to this concept as "memory tiering fairness".
> The biggest difference in the implementations lie in how toptier memory
> is tracked; in his implementation, an lruvec stat aggregation is done on
> each usage check, while in this implementation, a new cacheline is
> introduced in page_coutner to keep track of toptier usage (Kaiyang also
> introduces a new cachline in page_counter, but only uses it to cache
> capacity and thresholds). This implementation also extends the memory
> limit enforcement to memory.high as well.
>
> [1] https://lore.kernel.org/linux-mm/20240920221202.1734227-1-kaiyang2@cs.cmu.edu/
>
> ---
> Joshua Hahn (6):
> mm/memory-tiers: Introduce tier-aware memcg limit sysfs
> mm/page_counter: Introduce tiered memory awareness to page_counter
> mm/memory-tiers, memcontrol: Introduce toptier capacity updates
> mm/memcontrol: Charge and uncharge from toptier
> mm/memcontrol, page_counter: Make memory.low tier-aware
> mm/memcontrol: Make memory.high tier-aware
>
> include/linux/memcontrol.h | 21 ++++-
> include/linux/memory-tiers.h | 30 +++++++
> include/linux/page_counter.h | 31 ++++++-
> include/linux/swap.h | 3 +-
> kernel/cgroup/cpuset.c | 2 +-
> kernel/cgroup/dmem.c | 2 +-
> mm/memcontrol-v1.c | 6 +-
> mm/memcontrol.c | 155 +++++++++++++++++++++++++++++++----
> mm/memory-tiers.c | 63 ++++++++++++++
> mm/page_counter.c | 77 ++++++++++++++++-
> mm/vmscan.c | 24 ++++--
> 11 files changed, 376 insertions(+), 38 deletions(-)
>