Re: [LSF/MM/BPF TOPIC] DAMON Requirements for Access-aware MM of Future

[Jonathan Cameron <Jonathan.Cameron@huawei.com>]

On Wed, 29 Jan 2025 19:47:49 -0800
SeongJae Park <sj@kernel.org> wrote:

> Hi Yuanchu,
> 
> On Wed, 29 Jan 2025 18:15:08 -0800 Yuanchu Xie <yuanchu@google.com> wrote:
> 
> > On Mon, Jan 13, 2025 at 7:06 PM Gregory Price <gourry@gourry.net> wrote:  
> > > 5) Scarce resources
> > >
> > >    We need to be careful not to consume excessive amounts of resources
> > >    in an attempt to track all these identifying mechanisms.  Even 1 byte
> > >    per folio is 256MB on a 1TB machine.  This gets out of hand quick.
> > >
> > >    With task-work, I was able to add no additional resource consumption,
> > >    but deferring to a fully async scenario and needing to track things
> > >    like last-accessing CPU, timestamps, and etc.
> > >
> > >    We'll need to examine this closely if we decide to aggregate either
> > >    of these mechanisms.  
> > My concern with physical address space monitoring is fragmentation. I
> > ran some numbers on a few prod machines. Grouping by regions with the
> > same memcg and ignoring any unmapped memory to be generous, machines
> > with higher utilization can have a region/total pages ratio of ~40%,
> > and even those with lower utilization (<50%) can also reach 20%.
> > Accurately tracking these regions would require quite the region
> > metadata, on the order of GBs.  

I'd second this. Some cases are reasonably well behaved and
regions 'kind of work' for PA based tracking some very much not.
Add anything like overcommitted VMs on top and contiguity of 'hotness'
beyond very small regions goes out the window very quickly (unfortunately
I'm not able to share specific data).

So there are definitely cases where I'd expect something else to be needed.

There are a plenty of approximate tracking methods in the literature
that might be good enough with much lower overhead than precise tracking
(sketches etc) if we can feed them the right data.

Typically we don't need the answer on how hot all memory is, just some info
on 'this lot are particularly hot' and 'this lot are reasonably' cold.

Damon (as it currently stands) can sometimes give this info so to me
it's a possible producer of data for another layer that focuses on
abstracting the data to what we want only.  Hopefully we can make
that work for all the forms of tracking temperature that people are
looking at. I'm biased in favor of hardware units but no everyone will
have those toys available for a while yet :)

> 
> You're right, if we need page level accuracy access monitoring and want to use
> DAMON with its regions based mechanism for that, the memory overhead of
> damon_region could be high.  That's mainly because DAMON's regions-based
> mechanism has not designed for such usage.  It is more for a best-effort
> tradeoff between the overhead and the accuracy.
> 
> Regions-based mechanism is not necessarily the only mechanism of future DAMON,
> though.  If there are use cases that regions-based best-effort accuracy cannot
> be used while exactly the page level accuracy is really required, we can think
> about optimizing regions based mechanism or developing new one.
> 
> But, IMHO, the page level accurate access pattern is not always essential.  In
> many cases, being able to distinguish some amount of regions agains others
> based on access pattern is practical enough.  Indeed, DAMON has been used on
> real-world products with physical address based moitoring mode for years with
> no significant problem.  Also I think physical address space based monitoring
> results[1] on a real server workload that I shared recently seems not very bad.
> 
> Of course your use case could be different from what I have experienced so far.
> I'm curious if and why you really need page level accuracy.
> 
> [1] https://lore.kernel.org/20250110185232.54907-3-sj@kernel.org
> 
> 
> Thanks,
> SJ