Re: [Question] About memory.c: process_huge_page

["Huang, Ying" <ying.huang@linux.alibaba.com>]

Zhu Haoran <zhr1502@sjtu.edu.cn> writes:

> "Huang, Ying" <ying.huang@linux.alibaba.com> writes:
>>Hi, Haoran,
>>
>>Zhu Haoran <zhr1502@sjtu.edu.cn> writes:
>>
>>> Hi!
>>>
>>> I recently noticed the process_huge_page function in memory.c, which was
>>> intended to keep the cache hotness of target page after processing. I compared
>>> the vm-scalability anon-cow-seq-hugetlb microbench using the default
>>> process_huge_page and sequential processing (code posted below).
>>>
>>> I ran test on epyc-7T83 with 36vCPUs and 64GB memory. Using default
>>> process_huge_page, the avg bandwidth is 1148 mb/s. However sequential
>>> processing yielded a better bandwidth of about 1255 mb/s and only
>>> one-third cache-miss rate compared with default one.
>>>
>>> The same test was run on epyc-9654 with 36vCPU and 64GB mem. The
>>> bandwidth result was similar but the difference was smaller: 1170mb/s
>>> for default and 1230 mb/s for sequential. Although we did find the cache
>>> miss rate here did the reverse, since the sequential processing seen 3
>>> times miss more than the default.
>>>
>>> These result seem really inconsitent with the what described in your
>>> patchset [1]. What factors might explain these behaviors?
>>
>>One possible difference is cache topology.  Can you try to bind the test
>>process to the CPUs in one CCX (that is, share one LLC).  This make it
>>possible to hit the local cache.
>
> Thank you for the suggestion.
>
> I reduced the test to 16 vCPUs and bound them to one CCX on the epyc-9654. The
> rerun results are:
>
>                      sequential    process_huge_page
> BW (MB/s)                523.88               531.60   ( + 1.47%)
> user cachemiss           0.318%               0.446%   ( +40.25%)
> kernel cachemiss         1.405%              18.406%   ( + 1310%)
> usertime                  26.72                18.76   ( -29.79%)
> systime                   35.97                42.64   ( +18.54%)
>
> I was able to reproduce the much lower user time, but the bw gap is still not
> that significant as in your patch. It was bottlenecked by kernel cache-misses
> and execution time. One possible explanation is that AMD has less aggressive
> cache prefetcher, which fails to predict the access pattern of current
> process_huge_page in kernel. To verify that I ran a microbench that iterates
> through 4K pages in sequential/reverse order and access each page in seq/rev
> order (4 combinations in total).
>
> cachemiss rate
>                   seq-seq  seq-rev  rev-seq  rev-rev
> epyc-9654           0.08%    1.71%    1.98%    0.09%
> epyc-7T83           1.07%   13.64%    6.23%    1.12%
> i5-13500H          27.08%   28.87%   29.57%   25.35%
>
> I also ran the anon-cow-seq on my laptop i5-13500H and all metrics aligned well
> with your patch. So I guess this could be the root cause why AMD won't benefit
> from the patch?

The cache size per process needs to be checked too.  The smaller the
cache size per process, the more the benefit.

>>> Thanks for your time.
>>>
>>> [1] https://lkml.org/lkml/2018/5/23/1072
>>>
>>> ---
>>> Sincere,
>>> Zhu Haoran
>>>
>>> ---
>>>
>>> static int process_huge_page(
>>>         unsigned long addr_hint, unsigned int nr_pages,
>>>         int (*process_subpage)(unsigned long addr, int idx, void *arg),
>>>         void *arg)
>>> {
>>>     int i, ret;
>>>     unsigned long addr = addr_hint &
>>>         ~(((unsigned long)nr_pages << PAGE_SHIFT) - 1);
>>>
>>>     might_sleep();
>>>     for (i = 0; i < nr_pages; i++) {
>>>             cond_resched();
>>>             ret = process_subpage(addr + i * PAGE_SIZE, i, arg);
>>>             if (ret)
>>>                     return ret;
>>>     }
>>>
>>>     return 0;
>>> }

---
Best Regards,
Huang, Ying