haoxin writes: > On my arm64 server with 128 cores, 2 numa nodes. > > I used memhog as benchmark : > >   numactl -m -C 5 memhog -r100000 1G > > The test result as below: > > With this patch: > >   #time migratepages 8490 0 1 > >   real 0m1.161s > >   user 0m0.000s > >   sys 0m1.161s > > without this patch: > >   #time migratepages 8460 0 1 > >   real 0m2.068s > >   user 0m0.001s > >   sys 0m2.068s > > So you can see the migration performance improvement about *+78%* > > This is the perf record info. > > w/o > +  51.07%   0.09% migratepages > [kernel.kallsyms] [k] migrate_folio_extra > +  42.43%   0.04% migratepages [kernel.kallsyms] [k] folio_copy > +  42.34%  42.34% migratepages [kernel.kallsyms] [k] __pi_copy_page > +  33.99%   0.09% migratepages [kernel.kallsyms] [k] rmap_walk_anon > +  32.35%   0.04% migratepages [kernel.kallsyms] [k] try_to_migrate > *+  27.78%  27.78% migratepages > [kernel.kallsyms] [k] ptep_clear_flush * > +  8.19%   6.64% migratepages > [kernel.kallsyms] [k] folio_migrate_flagsmigrati_tlb_flush > > w/ this patch > +  18.57%   0.13% > migratepages   [kernel.kallsyms]  [k] > migrate_pages > +  18.23%   0.07% > migratepages   [kernel.kallsyms]  [k] > migrate_pages_batch > +  16.29%   0.13% > migratepages   [kernel.kallsyms]  [k] > migrate_folio_move > +  12.73%   0.10% > migratepages   [kernel.kallsyms]  [k] > move_to_new_folio > +  12.52%   0.06% > migratepages   [kernel.kallsyms]  [k] > migrate_folio_extra > > Therefore, this patch helps improve performance in page migration > > > So, you can add Tested-by: Xin Hao Thank you very much! Best Regards, Huang, Ying > > ( 2023/2/6 H2:33, Huang Ying S: >> From: "Huang, Ying" >> >> Now, migrate_pages() migrate folios one by one, like the fake code as >> follows, >> >> for each folio >> unmap >> flush TLB >> copy >> restore map >> >> If multiple folios are passed to migrate_pages(), there are >> opportunities to batch the TLB flushing and copying. That is, we can >> change the code to something as follows, >> >> for each folio >> unmap >> for each folio >> flush TLB >> for each folio >> copy >> for each folio >> restore map >> >> The total number of TLB flushing IPI can be reduced considerably. And >> we may use some hardware accelerator such as DSA to accelerate the >> folio copying. >> >> So in this patch, we refactor the migrate_pages() implementation and >> implement the TLB flushing batching. Base on this, hardware >> accelerated folio copying can be implemented. >> >> If too many folios are passed to migrate_pages(), in the naive batched >> implementation, we may unmap too many folios at the same time. The >> possibility for a task to wait for the migrated folios to be mapped >> again increases. So the latency may be hurt. To deal with this >> issue, the max number of folios be unmapped in batch is restricted to >> no more than HPAGE_PMD_NR in the unit of page. That is, the influence >> is at the same level of THP migration. >> >> We use the following test to measure the performance impact of the >> patchset, >> >> On a 2-socket Intel server, >> >> - Run pmbench memory accessing benchmark >> >> - Run `migratepages` to migrate pages of pmbench between node 0 and >> node 1 back and forth. >> >> With the patch, the TLB flushing IPI reduces 99.1% during the test and >> the number of pages migrated successfully per second increases 291.7%. >> >> This patchset is based on v6.2-rc4. >> >> Changes: >> >> v4: >> >> - Fixed another bug about non-LRU folio migration. Thanks Hyeonggon! >> >> v3: >> >> - Rebased on v6.2-rc4 >> >> - Fixed a bug about non-LRU folio migration. Thanks Mike! >> >> - Fixed some comments. Thanks Baolin! >> >> - Collected reviewed-by. >> >> v2: >> >> - Rebased on v6.2-rc3 >> >> - Fixed type force cast warning. Thanks Kees! >> >> - Added more comments and cleaned up the code. Thanks Andrew, Zi, Alistair, Dan! >> >> - Collected reviewed-by. >> >> from rfc to v1: >> >> - Rebased on v6.2-rc1 >> >> - Fix the deadlock issue caused by locking multiple pages synchronously >> per Alistair's comments. Thanks! >> >> - Fix the autonumabench panic per Rao's comments and fix. Thanks! >> >> - Other minor fixes per comments. Thanks! >> >> Best Regards, >> Huang, Ying