Re: [LSF/MM/BPF TOPIC] Locally attached memory tiering

[Davidlohr Bueso <dave@stgolabs.net>]

On Thu, 09 May 2024, Huang, Ying wrote:

>With the default configuration, current NUMA balancing based promotion
>solution will almost try to promote any faulting pages.  To select hot
>pages to promote and control thrashing between NUMA nodes, the promote
>rate limit needs to be configured.  For example, via,
>
>echo 200 > /proc/sys/kernel/numa_balancing_promote_rate_limit_MBps
>
>200MB hot pages will be selected and promoted every second.  Can you try it?

Yes, I've played with this tunnable and, just like the LRU approach, it
shows nice micro wins (less amount of promotions/demotions) but little for
actual benchmark improvements at a higher level, merely noise level or
very sublte wins. In fact, the actual data from that series for this
parameter was a ~2% pmbench win with the rate limiting, but a 69% promotion
rate descrease.

And this is really my point, how much effort do we want to put in optimizing
software mechanisms for hot page detection? Are there other benchmarks we
should be using? And perhaps doing the async promotion and not incurring in
the numa balancing overhead and comparing the cost of migration before
promoting would yield some better numbers, but that also might be easy to
get wrong when compared to the relative hotness of the page.

Thanks,
Davidlohr