Re: Transparent Hugepage impact on memcpy

[Hitoshi Mitake <mitake@dcl.info.waseda.ac.jp>]

Hi Jianguo,

On Wed, Jun 5, 2013 at 12:26 PM, Jianguo Wu <wujianguo@huawei.com> wrote:
> Hi,
> One more question, I wrote a memcpy test program, mostly the same as with perf bench memcpy.
> But test result isn't consistent with perf bench when THP is off.
>
>         my program                              perf bench
> THP:    3.628368 GB/Sec (with prefault)         3.672879 GB/Sec (with prefault)
> NO-THP: 3.612743 GB/Sec (with prefault)         6.190187 GB/Sec (with prefault)
>
> Below is my code:
>         src = calloc(1, len);
>         dst = calloc(1, len);
>
>         if (prefault)
>                 memcpy(dst, src, len);
>         gettimeofday(&tv_start, NULL);
>         memcpy(dst, src, len);
>         gettimeofday(&tv_end, NULL);
>
>         timersub(&tv_end, &tv_start, &tv_diff);
>         free(src);
>         free(dst);
>
>         speed = (double)((double)len / timeval2double(&tv_diff));
>         print_bps(speed);
>
> This is weird, is it possible that perf bench do some build optimize?
>
> Thansk,
> Jianguo Wu.

perf bench mem memcpy is build with -O6. This is the compile command
line (you can get this with make V=1):
gcc -o bench/mem-memcpy-x86-64-asm.o -c -fno-omit-frame-pointer -ggdb3
-funwind-tables -Wall -Wextra -std=gnu99 -Werror -O6 .... # ommited

Can I see your compile option for your test program and the actual
command line executing perf bench mem memcpy?

Thanks,
Hitoshi

>
> On 2013/6/4 16:57, Jianguo Wu wrote:
>
>> Hi all,
>>
>> I tested memcpy with perf bench, and found that in prefault case, When Transparent Hugepage is on,
>> memcpy has worse performance.
>>
>> When THP on is 3.672879 GB/Sec (with prefault), while THP off is 6.190187 GB/Sec (with prefault).
>>
>> I think THP will improve performance, but the test result obviously not the case.
>> Andrea mentioned THP cause "clear_page/copy_page less cache friendly" in
>> http://events.linuxfoundation.org/slides/2011/lfcs/lfcs2011_hpc_arcangeli.pdf.
>>
>> I am not quite understand this, could you please give me some comments, Thanks!
>>
>> I test in Linux-3.4-stable, and my machine info is:
>> Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
>>
>> available: 2 nodes (0-1)
>> node 0 cpus: 0 1 2 3 8 9 10 11
>> node 0 size: 24567 MB
>> node 0 free: 23550 MB
>> node 1 cpus: 4 5 6 7 12 13 14 15
>> node 1 size: 24576 MB
>> node 1 free: 23767 MB
>> node distances:
>> node   0   1
>>   0:  10  20
>>   1:  20  10
>>
>> Below is test result:
>> ---with THP---
>> #cat /sys/kernel/mm/transparent_hugepage/enabled
>> [always] madvise never
>> #./perf bench mem memcpy -l 1gb -o
>> # Running mem/memcpy benchmark...
>> # Copying 1gb Bytes ...
>>
>>        3.672879 GB/Sec (with prefault)
>>
>> #./perf stat ...
>> Performance counter stats for './perf bench mem memcpy -l 1gb -o':
>>
>>           35455940 cache-misses              #   53.504 % of all cache refs     [49.45%]
>>           66267785 cache-references                                             [49.78%]
>>               2409 page-faults
>>          450768651 dTLB-loads
>>                                                   [50.78%]
>>              24580 dTLB-misses
>>               #    0.01% of all dTLB cache hits  [51.01%]
>>         1338974202 dTLB-stores
>>                                                  [50.63%]
>>              77943 dTLB-misses
>>                                                  [50.24%]
>>          697404997 iTLB-loads
>>                                                   [49.77%]
>>                274 iTLB-misses
>>               #    0.00% of all iTLB cache hits  [49.30%]
>>
>>        0.855041819 seconds time elapsed
>>
>> ---no THP---
>> #cat /sys/kernel/mm/transparent_hugepage/enabled
>> always madvise [never]
>>
>> #./perf bench mem memcpy -l 1gb -o
>> # Running mem/memcpy benchmark...
>> # Copying 1gb Bytes ...
>>
>>        6.190187 GB/Sec (with prefault)
>>
>> #./perf stat ...
>> Performance counter stats for './perf bench mem memcpy -l 1gb -o':
>>
>>           16920763 cache-misses              #   98.377 % of all cache refs     [50.01%]
>>           17200000 cache-references                                             [50.04%]
>>             524652 page-faults
>>          734365659 dTLB-loads
>>                                                   [50.04%]
>>            4986387 dTLB-misses
>>               #    0.68% of all dTLB cache hits  [50.04%]
>>         1013408298 dTLB-stores
>>                                                  [50.04%]
>>            8180817 dTLB-misses
>>                                                  [49.97%]
>>         1526642351 iTLB-loads
>>                                                   [50.41%]
>>                 56 iTLB-misses
>>               #    0.00% of all iTLB cache hits  [50.21%]
>>
>>        1.025425847 seconds time elapsed
>>
>> Thanks,
>> Jianguo Wu.
>
>
>
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>