Re: Slow-tier Page Promotion discussion recap and open questions

["Zi Yan" <ziy@nvidia.com>]

On Mon Dec 30, 2024 at 12:30 AM EST, David Rientjes wrote:
> On Thu, 19 Dec 2024, Shivank Garg wrote:
>
> > On 12/18/2024 8:20 PM, Zi Yan wrote:
> > > On 17 Dec 2024, at 23:19, David Rientjes wrote:
> > > 
> > >> Hi everybody,
> > >>
> > >> We had a very interactive discussion last week led by RaghavendraKT on
> > >> slow-tier page promotion intended for memory tiering platforms, thank
> > >> you!  Thanks as well to everybody who attended and provided great
> > >> questions, suggestions, and feedback.
> > >>
> > >> The RFC patch series "mm: slowtier page promotion based on PTE A bit"[1]
> > >> is a proposal to allow for asynchronous page promotion based on memory
> > >> accesses as an alternative to NUMA Balancing based promotions.  There was
> > >> widespread interest in this topic and the discussion surfaced multiple
> > >> use cases and requirements, very focused on CXL use cases.
> > >>
> > > <snip>
> > >> ----->o-----
> > >> I asked about offloading the migration to a data mover, such as the PSP
> > >> for AMD, DMA engine, etc and whether that should be treated entirely
> > >> separately as a topic.  Bharata said there was a proof-of-concept
> > >> available from AMD that does just that but the initial results were not
> > >> that encouraging.
> > >>
> > >> Zi asked if the DMA engine saturated the link between the slow and fast
> > >> tiers.  If we want to offload to a copy engine, we need to verify that
> > >> the throughput is sufficient or we may be better off using idle cpus to
> > >> perform the migration for us.
> > > 
> > > <snip>
> > >>
> > >>  - we likely want to reconsider the single threaded nature of the kthread
> > >>    even if only for NUMA purposes
> > >>
> > > 
> > > Related to using DMA engine and/or multi threads for page migration, I had
> > > a patchset accelerating page migration[1] back in 2019. It showed good
> > > throughput speedup, ~4x using 16 threads to copy multiple 2MB THP. I think
> > > it is time to revisit the topic.
> > > 
> > > 
> > > [1] https://lore.kernel.org/linux-mm/20190404020046.32741-1-zi.yan@sent.com/
> > 
> > Hi All,
> > 
> > I wanted to provide some additional context regarding the AMD DMA offloading
> > POC mentioned by Bharata:
> > https://lore.kernel.org/linux-mm/20240614221525.19170-1-shivankg@amd.com
> > 
> > While the initial results weren't as encouraging as hoped, I plan to improve this
> > in next versions of the patchset.
> > 
> > The core idea in my RFC patchset is restructuring the folio move operation
> > to better leverage DMA hardware. Instead of the current folio-by-folio approach:
> > 
> > for_each_folio() {
> >     copy metadata + content + update PTEs
> > }
> > 
> > We batch the operations to minimize overhead:
> > 
> > for_each_folio() {
> >     copy metadata
> > }
> > DMA batch copy all content
> > for_each_folio() {
> >     update PTEs
> > }
> > 
> > My experiment showed that folio copy can consume up to 26.6% of total migration
> > cost when moving data between NUMA nodes. This suggests significant room for
> > improvement through DMA offloading, particularly for the larger transfers expected
> > in CXL scenarios.
> > 
> > It would be interesting work on combining these approaches for optimized page
> > promotion.
> > 
>
> This is very exciting, thanks Shivank and Zi!  The reason I brought this 
> topic up during the session on asynchronous page promotion for memory 
> tiering was because page migration is likely going to become *much* more 
> popular and will be in the critical path under system-wide memory 
> pressure.  Hardware assist and any software optimizations that can go 
> along with it would certainly be very interesting to discuss.
>
> Shivank, do you have an estimated timeline for when that patch series will 
> be refreshed?  Any planned integration with TMPM?
>
> Zi, are you looking to refresh your series and continue discussing page 
> migration offload?  We could set up another Linux MM Alignment Session 
> topic focused exactly on this and get representatives from the vendors 
> involved.

Sure. I am redoing the experiments with multithreads recently
and see more throughput increase (up to 10x througput with 32 threads)
on NVIDIA Grace CPUs.

Shivank's approach, using MIGRATE_SYNC_NO_COPY, looks simpler
than what I have done, splitting migrate_folio() into two parts[1]. I
am planning to rebuild my multithreaded folio copy patches on top of
Shivank's patches with some modifications. One thing to note is that
MIGRATE_SYNC_NO_COPY is removed by Kefeng (cc'd) recently[2], so I will
need to bring it back.


[1] https://github.com/x-y-z/linux-dev/tree/batched_page_migration_copy-v6.12
[2] https://lore.kernel.org/all/20240524052843.182275-6-wangkefeng.wang@huawei.com/

-- 
Best Regards,
Yan, Zi