Re: [RFC PATCH 0/6] mm/memcontrol: Make memcg limits tier-aware

[Donet Tom <donettom@linux.ibm.com>]

Hi Josua

On 2/24/26 4:08 AM, Joshua Hahn wrote:
> Memory cgroups provide an interface that allow multiple workloads on a
> host to co-exist, and establish both weak and strong memory isolation
> guarantees. For large servers and small embedded systems alike, memcgs
> provide an effective way to provide a baseline quality of service for
> protected workloads.
>
> This works, because for the most part, all memory is equal (except for
> zram / zswap). Restricting a cgroup's memory footprint restricts how
> much it can hurt other workloads competing for memory. Likewise, setting
> memory.low or memory.min limits can provide weak and strong guarantees
> to the performance of a cgroup.
>
> However, on systems with tiered memory (e.g. CXL / compressed memory),
> the quality of service guarantees that memcg limits enforced become less
> effective, as memcg has no awareness of the physical location of its
> charged memory. In other words, a workload that is well-behaved within
> its memcg limits may still be hurting the performance of other
> well-behaving workloads on the system by hogging more than its
> "fair share" of toptier memory.
>
> Introduce tier-aware memcg limits, which scale memory.low/high to
> reflect the ratio of toptier:total memory the cgroup has access.
>
> Take the following scenario as an example:
> On a host with 3:1 toptier:lowtier, say 150G toptier, and 50Glowtier,
> setting a cgroup's limits to:
> 	memory.min:  15G
> 	memory.low:  20G
> 	memory.high: 40G
> 	memory.max:  50G
>
> Will be enforced at the toptier as:
> 	memory.min:          15G
> 	memory.toptier_low:  15G (20 * 150/200)
> 	memory.toptier_high: 30G (40 * 150/200)
> 	memory.max:          50G



Currently, the high and low thresholds are adjusted based on the ratio 
of top-tier to total memory. One concern I see is that if the working 
set size exceeds the top-tier high threshold, it could lead to frequent 
demotions and promotions. Instead, would it make sense to introduce a 
tunable knob to configure the top-tier high threshold?

Another concern is that if the lower-tier memory size is very large, the 
cgroup may end up getting only a small portion of higher-tier memory.


>
> Let's say that there are 4 such cgroups on the host. Previously, it would
> be possible for 3 hosts to completely take over all of DRAM, while one
> cgroup could only access the lowtier memory. In the perspective of a
> tier-agnostic memcg limit enforcement, the three cgroups are all
> well-behaved, consuming within their memory limits.
>
> This is not to say that the scenario above is incorrect. In fact, for
> letting the hottest cgroups run in DRAM while pushing out colder cgroups
> to lowtier memory lets the system perform the most aggregate work total.
>
> But for other scenarios, the target might not be maximizing aggregate
> work, but maximizing the minimum performance guarantee for each
> individual workload (think hosts shared across different users, such as
> VM hosting services).
>
> To reflect these two scenarios, introduce a sysctl tier_aware_memcg,
> which allows the host to toggle between enforcing and overlooking
> toptier memcg limit breaches.
>
> This work is inspired & based off of Kaiyang Zhao's work from 2024 [1],
> where he referred to this concept as "memory tiering fairness".
> The biggest difference in the implementations lie in how toptier memory
> is tracked; in his implementation, an lruvec stat aggregation is done on
> each usage check, while in this implementation, a new cacheline is
> introduced in page_coutner to keep track of toptier usage (Kaiyang also
> introduces a new cachline in page_counter, but only uses it to cache
> capacity and thresholds). This implementation also extends the memory
> limit enforcement to memory.high as well.
>
> [1] https://lore.kernel.org/linux-mm/20240920221202.1734227-1-kaiyang2@cs.cmu.edu/
>
> ---
> Joshua Hahn (6):
>    mm/memory-tiers: Introduce tier-aware memcg limit sysfs
>    mm/page_counter: Introduce tiered memory awareness to page_counter
>    mm/memory-tiers, memcontrol: Introduce toptier capacity updates
>    mm/memcontrol: Charge and uncharge from toptier
>    mm/memcontrol, page_counter: Make memory.low tier-aware
>    mm/memcontrol: Make memory.high tier-aware
>
>   include/linux/memcontrol.h   |  21 ++++-
>   include/linux/memory-tiers.h |  30 +++++++
>   include/linux/page_counter.h |  31 ++++++-
>   include/linux/swap.h         |   3 +-
>   kernel/cgroup/cpuset.c       |   2 +-
>   kernel/cgroup/dmem.c         |   2 +-
>   mm/memcontrol-v1.c           |   6 +-
>   mm/memcontrol.c              | 155 +++++++++++++++++++++++++++++++----
>   mm/memory-tiers.c            |  63 ++++++++++++++
>   mm/page_counter.c            |  77 ++++++++++++++++-
>   mm/vmscan.c                  |  24 ++++--
>   11 files changed, 376 insertions(+), 38 deletions(-)
>