Re: [linus:master] [shmem] a2e459555c: aim9.disk_src.ops_per_sec -19.0% regression

[Chuck Lever III <chuck.lever@oracle.com>]

> On Sep 13, 2023, at 2:47 AM, Feng Tang <feng.tang@intel.com> wrote:
> 
> On Tue, Sep 12, 2023 at 11:14:42PM +0800, Tang, Feng wrote:
> [...]
>>> 
>>> Well that's the problem. Since I can't run the reproducer, there's
>>> nothing I can do to troubleshoot the problem myself.
>> 
>> We dug more into the perf and other profiling data from 0Day server
>> running this case, and it seems that the new simple_offset_add()
>> called by shmem_mknod() brings extra cost related with slab,
>> specifically the 'radix_tree_node', which cause the regression.
>> 
>> Here is some slabinfo diff for commit a2e459555c5f and its parent:
>> 
>> 23a31d87645c6527 a2e459555c5f9da3e619b7e47a6 
>> ---------------- --------------------------- 
>> 
>>     26363           +40.2%      36956        slabinfo.radix_tree_node.active_objs
>>    941.00           +40.4%       1321        slabinfo.radix_tree_node.active_slabs
>>     26363           +40.3%      37001        slabinfo.radix_tree_node.num_objs
>>    941.00           +40.4%       1321        slabinfo.radix_tree_node.num_slabs
>> 
>> Also the perf profile show some difference
>> 
>>      0.01 ±223%      +0.1        0.10 ± 28%  pp.self.shuffle_freelist
>>      0.00            +0.1        0.11 ± 40%  pp.self.xas_create
>>      0.00            +0.1        0.12 ± 27%  pp.self.xas_find_marked
>>      0.00            +0.1        0.14 ± 18%  pp.self.xas_alloc
>>      0.03 ±103%      +0.1        0.17 ± 29%  pp.self.xas_descend
>>      0.00            +0.2        0.16 ± 23%  pp.self.xas_expand
>>      0.10 ± 22%      +0.2        0.27 ± 16%  pp.self.rcu_segcblist_enqueue
>>      0.92 ± 35%      +0.3        1.22 ± 11%  pp.self.kmem_cache_free
>>      0.00            +0.4        0.36 ± 16%  pp.self.xas_store
>>      0.32 ± 30%      +0.4        0.71 ± 12%  pp.self.__call_rcu_common
>>      0.18 ± 27%      +0.5        0.65 ±  8%  pp.self.kmem_cache_alloc_lru
>>      0.36 ± 79%      +0.6        0.96 ± 15%  pp.self.__slab_free
>>      0.00            +0.8        0.80 ± 14%  pp.self.radix_tree_node_rcu_free
>>      0.00            +1.0        1.01 ± 16%  pp.self.radix_tree_node_ctor
>> 
>> Some perf profile from a2e459555c5f is: 
>> 
>> -   17.09%     0.09%  singleuser       [kernel.kallsyms]            [k] path_openat   
>>   - 16.99% path_openat                
>>      - 12.23% open_last_lookups      
>>         - 11.33% lookup_open.isra.0
>>            - 9.05% shmem_mknod
>>               - 5.11% simple_offset_add
>>                  - 4.95% __xa_alloc_cyclic 
>>                     - 4.88% __xa_alloc
>>                        - 4.76% xas_store 
>>                           - xas_create
>>                              - 2.40% xas_expand.constprop.0
>>                                 - 2.01% xas_alloc
>>                                    - kmem_cache_alloc_lru
>>                                       - 1.28% ___slab_alloc
>>                                          - 1.22% allocate_slab 
>>                                             - 1.19% shuffle_freelist 
>>                                                - 1.04% setup_object
>>                                                     radix_tree_node_ctor
>> 
>> Please let me know if you need more info.
>> 
>>> 
>>> Is there any hope in getting this reproducer to run on Fedora?
>> 
>> Myself haven't succeeded to reproduce it locally, will keep trying
>> it tomorrow.
> 
> It can be reproduced on a local machien with CentOS 9 (similar to
> Fedora ?),

Excellent, I appreciate this!


> and some steps as:
> 
> * download source code
>  $ wget https://sourceforge.net/projects/aimbench/files/aim-suite9/Initial%20release/s9110.tar.Z
> 
> * untar the file and run 'make'
> 
> * create 2 files 's9workfile' and 'test.config'
> 
> $ cat s9workfile
> # @(#) s9workfile:1.2 1/22/96 00:00:00
> # AIM Independent Resource Benchmark - Suite IX Workfile
> FILESIZE: 5M
> disk_src
> 
> $ cat test.config
> test
> disk_src
> 200s
> /dev/shm/
> 
> * run the test with cmd "./singleuser -nl < test.config" 
> 
> The test case here is 'disk_src', so I picked one file 'disk_src.c'
> and attached it for quick reference.
> 
> The kernel config of my CentOS is different from what 0Day used, so 
> the perf-profile and peformance score are a little different, but
> the regression trend is the same, that commit a2e459555c5f has about
> 20% drop.
> 
> Also the test platform doesn't matter, I tried on several generation 
> of Xeon servers which can all reproduce it.


--
Chuck Lever