Re: [Question] About memory.c: process_huge_page

[Zhu Haoran <zhr1502@sjtu.edu.cn>]

"Huang, Ying" <ying.huang@linux.alibaba.com> writes:
>Hi, Haoran,
>
>Zhu Haoran <zhr1502@sjtu.edu.cn> writes:
>
>> Hi!
>>
>> I recently noticed the process_huge_page function in memory.c, which was
>> intended to keep the cache hotness of target page after processing. I compared
>> the vm-scalability anon-cow-seq-hugetlb microbench using the default
>> process_huge_page and sequential processing (code posted below).
>>
>> I ran test on epyc-7T83 with 36vCPUs and 64GB memory. Using default
>> process_huge_page, the avg bandwidth is 1148 mb/s. However sequential
>> processing yielded a better bandwidth of about 1255 mb/s and only
>> one-third cache-miss rate compared with default one.
>>
>> The same test was run on epyc-9654 with 36vCPU and 64GB mem. The
>> bandwidth result was similar but the difference was smaller: 1170mb/s
>> for default and 1230 mb/s for sequential. Although we did find the cache
>> miss rate here did the reverse, since the sequential processing seen 3
>> times miss more than the default.
>>
>> These result seem really inconsitent with the what described in your
>> patchset [1]. What factors might explain these behaviors?
>
>One possible difference is cache topology.  Can you try to bind the test
>process to the CPUs in one CCX (that is, share one LLC).  This make it
>possible to hit the local cache.

Thank you for the suggestion.

I reduced the test to 16 vCPUs and bound them to one CCX on the epyc-9654. The
rerun results are:

                     sequential    process_huge_page
BW (MB/s)                523.88               531.60   ( + 1.47%)
user cachemiss           0.318%               0.446%   ( +40.25%)
kernel cachemiss         1.405%              18.406%   ( + 1310%)
usertime                  26.72                18.76   ( -29.79%)
systime                   35.97                42.64   ( +18.54%)

I was able to reproduce the much lower user time, but the bw gap is still not
that significant as in your patch. It was bottlenecked by kernel cache-misses
and execution time. One possible explanation is that AMD has less aggressive
cache prefetcher, which fails to predict the access pattern of current
process_huge_page in kernel. To verify that I ran a microbench that iterates
through 4K pages in sequential/reverse order and access each page in seq/rev
order (4 combinations in total).

cachemiss rate
                  seq-seq  seq-rev  rev-seq  rev-rev
epyc-9654           0.08%    1.71%    1.98%    0.09%
epyc-7T83           1.07%   13.64%    6.23%    1.12%
i5-13500H          27.08%   28.87%   29.57%   25.35%

I also ran the anon-cow-seq on my laptop i5-13500H and all metrics aligned well
with your patch. So I guess this could be the root cause why AMD won't benefit
from the patch?

>> Thanks for your time.
>>
>> [1] https://lkml.org/lkml/2018/5/23/1072
>>
>> ---
>> Sincere,
>> Zhu Haoran
>>
>> ---
>>
>> static int process_huge_page(
>>         unsigned long addr_hint, unsigned int nr_pages,
>>         int (*process_subpage)(unsigned long addr, int idx, void *arg),
>>         void *arg)
>> {
>>     int i, ret;
>>     unsigned long addr = addr_hint &
>>         ~(((unsigned long)nr_pages << PAGE_SHIFT) - 1);
>>
>>     might_sleep();
>>     for (i = 0; i < nr_pages; i++) {
>>             cond_resched();
>>             ret = process_subpage(addr + i * PAGE_SIZE, i, arg);
>>             if (ret)
>>                     return ret;
>>     }
>>
>>     return 0;
>> }
>
>---
>Best Regards,
>Huang, Ying

---
Sincere,
Zhu Haoran