[QUESTION] “place pages to tail” regress memory read bandwidth about 10% under our test cases

[QUESTION] “place pages to tail” regress memory read bandwidth about 10% under our test cases

[liupeng (DM)]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

Hi David,

We met a performance regression(lmbench bw_mmap_rd/bw_mem) on our server with
patch 7fef431be9c9(mm/page_alloc: place pages to tail in __free_pages_core()),

Case A.  ./bw_mmap_rd -P 1 512m mmap_only out.file
Case B.  ./bw_mem -P 1 512m frd

mode0: IMC Interleave = auto, Channel Interleave = auto, Rank Interleave = auto
                     without                        with        regression
case A           10535.14                       9645.02      8.4%
case B           10526.88                       9797.63      6.9%
also we found different memory interleaving have different results

mode1: IMC Interleave = 1-way, Channel Interleave = 2-way, Rank Interleave =2-way
                      without                        with          regression
case A           10543.01                       10643.12       -0.9%
case B           10971.33                       10853.99       1%

The 7fef431be9c9 changes the different memory layout,  it seems that the issue
is related with memory interleave. But the patch does lead to performance regression.

Any suggestion about this regression?
thanks.

Peng Liu


[-- Attachment #2: Type: text/html, Size: 6035 bytes --]

Re: [QUESTION] “place pages to tail” regress memory read bandwidth about 10% under our test cases

[David Hildenbrand]

On 30.11.21 12:51, liupeng (DM) wrote:
> Hi David,
> 
>  

Hi!

> 
> We met a performance regression(lmbench bw_mmap_rd/bw_mem) on our server
> with
> 
> patch 7fef431be9c9(mm/page_alloc: place pages to tail in
> __free_pages_core()),
> 
>  
> 
> Case A.  ./bw_mmap_rd -P 1 512m mmap_only out.file
> 
> Case B.  ./bw_mem -P 1 512m frd
> 
>  
> 
> mode0: IMC Interleave = auto, Channel Interleave = auto, Rank Interleave
> = auto
> 
>                      without                        with        regression
> 
> case A           10535.14                       9645.02      8.4%
> 
> case B           10526.88                       9797.63      6.9%
> 
> also we found different memory interleaving have different results
> 
>  
> 
> mode1: IMC Interleave = 1-way, Channel Interleave = 2-way, Rank
> Interleave =2-way
> 
>                       without                        with         
> regression
> 
> case A           10543.01                       10643.12       -0.9%
> 
> case B           10971.33                       10853.99       1%
> 
>  
> 
> The 7fef431be9c9 changes the different memory layout,  it seems that the
> issue

It doesn't change the physical memory layout. It changes the order in
which physical memory will get allocated.

> 
> is related with memory interleave. But the patch does lead to
> performance regression.
> 
>  
> 
> Any suggestion about this regression?

Does it happen only on specific HW?

Commit 7fef431be9c9 discovered a bunch of BUGs already, including
physical memory that has different performance characteristics due to
firmware bugs, which indicated some physical memory e.g., as uncached to
the OS.

That commit triggers allocation of physical memory in a different order,
which sometimes makes "bad physical memory" getting allocated earlier
than later in some benchmarks. Such effects are usually only visible on
a freshly started system; once the system was busy for some time and the
free page lists are nondeterministic, you might observe similar results.

I wrote a kernel module that was able to identify bad physical memory
regions in some affected setups:
	https://github.com/davidhildenbrand/kstream/blob/main/README.md

You can try running it after a freshly booted system to eventually
identify some "bad" physical memory areas. But it might not be the root
cause. Maybe it's just some caching effects?

-- 
Thanks,

David / dhildenb