Re: [LSF/MM/BPF TOPIC] Unifying sources of page temperature information - what info is actually wanted?

["Huang, Ying" <ying.huang@linux.alibaba.com>]

Hi, Jonathan,

Sorry for late reply.

Jonathan Cameron <Jonathan.Cameron@huawei.com> writes:

> On Fri, 31 Jan 2025 12:28:03 +0000
> Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>
>> > Here is the list of potential discussion points:  
>> ...
>> 
>> > 2. Possibility of maintaining single source of truth for page hotness that would
>> > maintain hot page information from multiple sources and let other sub-systems
>> > use that info.  
>> Hi,
>> 
>> I was thinking of proposing a separate topic on a single source of hotness,
>> but this question covers it so I'll add some thoughts here instead.
>> I think we are very early, but sharing some experience and thoughts in a
>> session may be useful.
>
> Thinking more on this over lunch, I think it is worth calling this out as a
> potential session topic in it's own right rather than trying to find
> time within other sessions.  Hence the title change.
>
> I think a session would start with a brief listing of the temperature sources
> we have and those on the horizon to motivate what we are unifying, then
> discussion to focus on need for such a unification + requirements 
> (maybe with a straw man).
>
>> 
>> What do the other subsystems that want to use a single source of page hotness
>> want to be able to find out? (subject to filters like memory range, process etc)
>> 
>> A) How hot is page X?  
>> - Is this useful, or too much data? What would use it?
>>   * Application optimization maybe. Very handy for developing algorithms
>>     to do the rest of the options here as an Oracle!
>> - Provides both the cold and hot end of the scale, but maybe measurement
>>   techniques vary and can not be easily combined. Hard in general to combine
>>   multiple sources of truth if aiming for an absolute number.
>> 
>> B) Which pages are super hot?
>> - Probably these that make the most difference if they are in a slower memory tier.
>> 
>> C) Some pages are hot enough to consider moving?
>> - This may be good enough to get the key data into the fast memory over time.
>> - Can combine sources of info as being able to compare precise numbers doesn't matter.
>> 
>> D) Which pages are fairly cold?
>> - Likewise maybe good enough over time.
>> 
>> E) Which pages are very cold?
>> - Ideal case for tiering. Swap these with the super hot ones.
>> - Maybe extra signal for swap / zswap etc
>> 
>> F) Did these hot pages remain hot (and same for cold)
>> - This is needed to know when to back off doing things as we have unstable
>>   hotness (two phase applications are a pain for this), sampling a few
>>   pages may be fine.
>> 
>> Messy corners:
>> 
>> Temporal aspects.
>> - If only providing lists of hottest / coldest in last second, very hard
>>   to find those that are of a stable temperature. We end up moving
>>   very hot data (which is disruptive) and it doesn't stay hot.
>> - Can reduce that affect by long sampling windows on some measurement approaches
>>   (on hardware trackers that can trash accuracy due to resource exhaustion
>>    and other subtle effects).
>> - bistable / phase based applications are a pain but perhaps up to higher
>>   levels to back off.
>> 
>> My main interest is migrating in tiered systems but good to look at what
>> else would use a common layer.
>> 
>> Mostly I want to know something that is useful to move, and assume convergence
>> over the long term with the best things to move so to me the ideal layer has
>> following interface (strawman so shoot holes in it!):
>> 
>> 1) Give me up to X hotish pages from a slow tier (greater than a specific measure
>> of temperature)

Because the hot pages may be available upon page accessing (such PROT_NONE
page fault), the interface may be "push" style instead of "pull" style,
e.g.,

int register_hot_page_handler(void (*handler)(struct page *hot_page, int temperature));

>> 2) Give me X coldish pages a faster tier.
>> 3) I expect to ask again in X seconds so please have some info ready for me!
>> 4) (a path to get an idea of 'unhelpful moves' from earlier iterations - this
>>     is bleeding the tiering application into a shared interface though).

In addition to get a list hot/cold pages, it's also useful to get
hot/cold statistics of a memory device (NUMA node), e.g., something like
below,

Access frequency        percent
   > 1000 HZ            10%
 600-1000 HZ            20%
 200- 600 HZ            50%
   1- 200 HZ            15%
      < 1 HZ             5%

Compared with hot/cold pages list, this may be gotten with lower
overhead and can be useful to tune the promotion/demotion alrogithm.  At
the same time, a sampled (incomplete) list of hot/cold page list may be
available too.

>> If we have multiple subsystems using the data we will need to resolve their
>> conflicting demands to generate good enough data with appropriate overhead.
>> 
>> I'd also like a virtualized solution for case of hardware PA trackers (what
>> I have with CXL Hotness Monitoring Units) and classic memory pool / stranding
>> avoidance case where the VM is the right entity to make migration decisions.
>> Making that interface convey what the kernel is going to use would be an
>> efficient option. I'd like to hide how the sausage was made from the VM.

---
Best Regards,
Huang, Ying