Re: [LSF/MM/BPF TOPIC] Overhauling hot page detection and promotion based on PTE A bit scanning

[Jonathan Cameron <Jonathan.Cameron@huawei.com>]

On Fri, 24 Jan 2025 14:24:40 +0530
Raghavendra K T <raghavendra.kt@amd.com> wrote:

> On 1/23/2025 11:50 PM, SeongJae Park wrote:
> > Hi Raghavendra,
> > 
> > On Thu, 23 Jan 2025 10:57:21 +0000 Raghavendra K T <raghavendra.kt@amd.com> wrote:
> >   
> >> Bharata and I would like to propose the following topic for LSFMM.
> >>
> >> Topic: Overhauling hot page detection and promotion based on PTE A bit scanning.  
> > 
> > Thank you for proposing this.  I'm interested in this!
> >   
> 
> Thank you.
> 
> [...]
> 
> >> virtual vs physical address based scanning,  
> > 
> > DAMON supports both virtual and physical address spaces monitoring.  DAMON's
> > pages migration is currently not supported for virtual address spaces, though I
> > believe adding the support is not difficult.
> >   
> 
> Will check this.
> 
> [...]
> 
> >>
> >> 3. Discuss how hardware provided hotness info (like AMD IBS) can further aid
> >> promotion. Bharata had posted an RFC [4] on this a while back.  
> > 
> > Maybe CXL Hotness Monitoring Unit could also be an interesting thing to discuss
> > together.
> >   
> 
> Definitely.

Thanks for the shout out SJ.  I'm definitely interested in this topic from the
angle of the hardware hotness monitoring units (roughly speaking ones that give
you a list of hot pages - typically by PA).  Making sure any solution works
for those is perhaps key for the longer term.  Not entirely clear to me that
we are ready yet for data aggregation solution, or mixed techniques
but definitely interesting to brainstorm.

Until now my main focus has been on getting infrastructure in place to work out the
lower levels of using a hardware hotness monitoring unit (using QEMU for now
with TCG plugins to get the access data).  In general, not stuff I suspect
anyone will want to discuss at LSF/MM, but perhaps providing insights into how
good data we might get could be.

Unless the hardware units people build are very capable (and expensive)
the chances are we will have to deal with accuracy limitations that I
suspect the users of this data for migration etc do not want to explicitly
deal with.  If our tracking is coming from multiple sources we need to
deal with differences in, and potentially estimation of accuracy.
Anything efficient is going to have some accuracy issues (regions for Damon,
access bit scanning frequency for your technique, sampling for page fault
techniques, data in wrong place - access bit's will tell you to promote
stuff that is always in cache - arguably a waste of time etc)
I've no idea yet how painful this is going to be.  Using the different
sources to overcome limitations on each one is interesting but likely
to be complex and tricky to generalize.  Maybe access bit scanning
to detect hotish large scale regions, then a hardware tracker to separate
out 'hot' from 'warm'.  Sounds fun, but far form general!

Lots of problems to solve in this space. And when we have done that, there
is paravirtualizing hardware trackers / other methods, application
specific usage of the data (some apps will know better than the kernel and
will want this data, security / side channels etc).

For stretch goals there is even the fun question of hotness monitoring down
stream of interleave, particularly when it's scrambled and not a power of
2 ways.  Again, maybe not a general problem but will affect data biases.
How much of that we want to hide down in implementations below some general
'give me hot stuff' is an open question (I'm guessing hide almost everything
beyond controls on data bandwidth).

Jonathan

> 
> >>
> >> 4. Overlap with DAMON and potential reuse.  
> > 
> > I confess that it seems some of the works might overlap with DAMON to my biased
> > eyes.  I'm looking forward to attend this session, to make it less biased and
> > more aligned with people :)
> >   
> 
> Yes. Agree.
> 
> >>   
> >> Links:
> >>
> >> [1] https://lore.kernel.org/all/20241201153818.2633616-1-raghavendra.kt@amd.com/
> >> [2] https://lore.kernel.org/linux-mm/20241226012833.rmmbkws4wdhzdht6@ed.ac.uk/T/
> >> [3] https://lore.kernel.org/lkml/Z4XUoWlU-UgRik18@gourry-fedora-PF4VCD3F/T/
> >> [4] https://lore.kernel.org/lkml/20230208073533.715-2-bharata@amd.com/  
> > 
> > Again, thank you for proposing this topic, and I wish to see you at Montreal!
> >   
> 
> Same here .. Thank you :)
> 
> - Raghu
> 
>