On Tue, Sep 12, 2023 at 11:14:42PM +0800, Tang, Feng wrote:
[...]
> > 
> > Well that's the problem. Since I can't run the reproducer, there's
> > nothing I can do to troubleshoot the problem myself.
> 
> We dug more into the perf and other profiling data from 0Day server
> running this case, and it seems that the new simple_offset_add()
> called by shmem_mknod() brings extra cost related with slab,
> specifically the 'radix_tree_node', which cause the regression.
> 
> Here is some slabinfo diff for commit a2e459555c5f and its parent:
> 
> 	23a31d87645c6527 a2e459555c5f9da3e619b7e47a6 
> 	---------------- --------------------------- 
>  
>      26363           +40.2%      36956        slabinfo.radix_tree_node.active_objs
>     941.00           +40.4%       1321        slabinfo.radix_tree_node.active_slabs
>      26363           +40.3%      37001        slabinfo.radix_tree_node.num_objs
>     941.00           +40.4%       1321        slabinfo.radix_tree_node.num_slabs
> 
> Also the perf profile show some difference
> 
>       0.01 ±223%      +0.1        0.10 ± 28%  pp.self.shuffle_freelist
>       0.00            +0.1        0.11 ± 40%  pp.self.xas_create
>       0.00            +0.1        0.12 ± 27%  pp.self.xas_find_marked
>       0.00            +0.1        0.14 ± 18%  pp.self.xas_alloc
>       0.03 ±103%      +0.1        0.17 ± 29%  pp.self.xas_descend
>       0.00            +0.2        0.16 ± 23%  pp.self.xas_expand
>       0.10 ± 22%      +0.2        0.27 ± 16%  pp.self.rcu_segcblist_enqueue
>       0.92 ± 35%      +0.3        1.22 ± 11%  pp.self.kmem_cache_free
>       0.00            +0.4        0.36 ± 16%  pp.self.xas_store
>       0.32 ± 30%      +0.4        0.71 ± 12%  pp.self.__call_rcu_common
>       0.18 ± 27%      +0.5        0.65 ±  8%  pp.self.kmem_cache_alloc_lru
>       0.36 ± 79%      +0.6        0.96 ± 15%  pp.self.__slab_free
>       0.00            +0.8        0.80 ± 14%  pp.self.radix_tree_node_rcu_free
>       0.00            +1.0        1.01 ± 16%  pp.self.radix_tree_node_ctor
> 
> Some perf profile from a2e459555c5f is: 
> 
> -   17.09%     0.09%  singleuser       [kernel.kallsyms]            [k] path_openat   
>    - 16.99% path_openat                
>       - 12.23% open_last_lookups      
>          - 11.33% lookup_open.isra.0
>             - 9.05% shmem_mknod
>                - 5.11% simple_offset_add
>                   - 4.95% __xa_alloc_cyclic 
>                      - 4.88% __xa_alloc
>                         - 4.76% xas_store 
>                            - xas_create
>                               - 2.40% xas_expand.constprop.0
>                                  - 2.01% xas_alloc
>                                     - kmem_cache_alloc_lru
>                                        - 1.28% ___slab_alloc
>                                           - 1.22% allocate_slab 
>                                              - 1.19% shuffle_freelist 
>                                                 - 1.04% setup_object
>                                                      radix_tree_node_ctor
> 
> Please let me know if you need more info.
> 
> > 
> > Is there any hope in getting this reproducer to run on Fedora?
> 
> Myself haven't succeeded to reproduce it locally, will keep trying
> it tomorrow.

It can be reproduced on a local machien with CentOS 9 (similar to
Fedora ?), and some steps as:

* download source code
  $ wget https://sourceforge.net/projects/aimbench/files/aim-suite9/Initial%20release/s9110.tar.Z

* untar the file and run 'make'

* create 2 files 's9workfile' and 'test.config'

	$ cat s9workfile
	# @(#) s9workfile:1.2 1/22/96 00:00:00
	# AIM Independent Resource Benchmark - Suite IX Workfile
	FILESIZE: 5M
	disk_src

	$ cat test.config
	test
	disk_src
	200s
	/dev/shm/

* run the test with cmd "./singleuser -nl < test.config" 

The test case here is 'disk_src', so I picked one file 'disk_src.c'
and attached it for quick reference.

The kernel config of my CentOS is different from what 0Day used, so 
the perf-profile and peformance score are a little different, but
the regression trend is the same, that commit a2e459555c5f has about
20% drop.

Also the test platform doesn't matter, I tried on several generation 
of Xeon servers which can all reproduce it. 

Thanks,
Feng