squashfs performance regression and readahea

["Song, Xiongwei" <Xiongwei.Song@windriver.com>]

Hello,

I am facing a performance regression on squashfs.  There are many squashfs 
partitions on our board. I am doing the operations below on 90 squashfs
partitions:

"for cnt in $(seq 0 9); do echo 3 > /proc/sys/vm/drop_caches; echo "Loop ${cnt}:"; time -v find /squashfs/part[0-9][0-9] | xargs -P 24 -i cat {} > /dev/null 2>/dev/null; echo ""; done"

On linux 4.18, I got the elapsed time statistics below with command above(run 
find/xargs/cat commands 10 times):
1:22.80 (1m + 22.80s)
0:59.76
1:01.43
1:02.48
1:03.03
1:02.92
1:03.19
1:03.22
1:03.26
1:03.14

On linux 5.10, huge performance regression:
5:48.69 (5m + 48.69s)
5:52.99
6:06.30
6:01.43
5:50.08
6:26.59
6:09.98
6:04.72
6:05.21
6:21.49

By "git bisect", I found this regression is related to readahead.  After reverting 
c1f6925e1091 ("mm: put readahead pages in cache earlier")  and
8151b4c8bee4 ("mm: add readahead address space operation") on linux 5.10,
the performance is improved:
1:37.16 (1m + 37.16s)
1:04.18
1:05.28
1:06.07
1:06.31
1:06.58
1:06.80
1:06.79
1:06.95
1:06.61

Also, I found disabling readahead is helpful with 9eec1d897139 ("squashfs: provide 
backing_dev_info in order to disable read-ahead"):
1:06.18 (1m + 6.18s)
1:05.65
1:06.34
1:06.88
1:06.52
1:06.78
1:06.61
1:06.99
1:06.60
1:06.79

I have also tired with the upstream linux 5.18, see the results below:
1:12.82 (1m + 12.82s)
1:07.68
1:08.94
1:09.65
1:09.87
1:10.32
1:10.47
1:10.34
1:10.24
1:10.34
 
As we can see that even if the readahead disabled, there is still extra 2 ~ 3s overhead
than linux 4.18. BTW, the reverted two commits above are from " Change readahead API "
series, see the following link:
https://lore.kernel.org/all/20200414150233.24495-11-willy@infradead.org/T/#m22d6de881c24057b776758ae8e7f5d54e2db8026
.

I would appreciate your comments and inputs.

Regards,
Xiognwei