Re: regression/bisected commit 773688a6cb24b0b3c2ba40354d883348a2befa38 make my system completely unusable under high load

[Marco Elver <elver@google.com>]

On Fri, 2 Feb 2024 at 17:35, Mikhail Gavrilov
<mikhail.v.gavrilov@gmail.com> wrote:
>
> On Fri, Feb 2, 2024 at 2:00 PM Marco Elver <elver@google.com> wrote:
> >
> > > Maybe we can try something else?
> >
> > That's strange - the patches at [1] definitely revert the change you
> > bisected to. It's possible there is some other strange side-effect. (I
> > assume that you are still running all this with a KASAN kernel.)
>
> Yes. build .config not changed between kernel builds.
>
> > Just so I understand it right:
> > You say before commit cc478e0b6bdffd20561e1a07941a65f6c8962cab the
> > game's FPS were good. But that is strange, because at that point we're
> > already doing stackdepot refcounting, i.e. after commit
> > 773688a6cb24b0b3c2ba40354d883348a2befa38 which you reported as the
> > initial performance regression. The patches at [2] fixed that problem.
> >
> > So now it's unclear to me how the simple change in
> > cc478e0b6bdffd20561e1a07941a65f6c8962cab causes the performance
> > problem, when in fact this is already with KASAN stackdepot
> > refcounting enabled but without the performance fixes from [1] and
> > [2].
> >
> > [2] https://lore.kernel.org/all/20240118110216.2539519-2-elver@google.com/
> >
> > My questions now would be:
> > - What was the game's FPS in the last stable kernel (v6.7)?
>
> [6.7] - 83 FPS - 13060 frames during benchmark.
>
> > - Can you collect another set of performance profiles between good and
> > bad? Maybe it would show where the time in the kernel is spent.
>
> Yes,
> please look at [aaa2c9a97c22 perf] and [cc478e0b6bdf perf]
>
> > perf diff perf-git-aaa2c9a97c22af5bf011f6dd8e0538219b45af88.data perf-git-cc478e0b6bdffd20561e1a07941a65f6c8962cab.data
> No kallsyms or vmlinux with build-id
> de2a040f828394c5ce34802389239c2a0668fcc7 was found
> No kallsyms or vmlinux with build-id
> 33ab1cd545f96f5ffc2a402a4c4cfa647fd727a0 was found
> # Event 'cycles:P'
> #
> # Baseline  Delta Abs  Shared Object
> Symbol
> # ........  .........  ..............................................
> .....................................................................................................................................................................................
> #
>     48.48%    +21.75%  [kernel.kallsyms]
> [k] 0xffffffff860065c0
>     36.13%    -16.49%  ShadowOfTheTombRaider
> [.] 0x00000000001d7f5e
>      4.43%     -2.10%  libvulkan_radeon.so
> [.] 0x000000000006b870
>      3.28%     -0.63%  libcef.so
> [.] 0x00000000021720e0
>      1.11%     -0.53%  libc.so.6
> [.] syscall
>      0.65%     -0.24%  libc.so.6
> [.] __memmove_avx512_unaligned_erms
>      0.31%     -0.14%  libc.so.6
> [.] __memset_avx512_unaligned_erms
>      0.26%     -0.13%  libm.so.6
> [.] __powf_fma
>      0.20%     -0.10%  [amdgpu]
> [k] amdgpu_bo_placement_from_domain
>      0.22%     -0.09%  [amdgpu]
> [k] amdgpu_vram_mgr_compatible
>      0.67%     -0.09%  armada-drm_dri.so
> [.] 0x00000000000192b4
>      0.15%     -0.08%  libc.so.6
> [.] sem_post@GLIBC_2.2.5
>      0.16%     -0.07%  [amdgpu]
> [k] amdgpu_vm_bo_update
>      0.14%     -0.07%  [amdgpu]
> [k] amdgpu_bo_list_entry_cmp
>      0.13%     -0.06%  libm.so.6
> [.] powf@GLIBC_2.2.5
>      0.14%     -0.06%  libMangoHud.so
> [.] 0x000000000001c4c0
>      0.10%     -0.06%  libc.so.6
> [.] __futex_abstimed_wait_common
>      0.19%     -0.05%  libGLESv2.so
> [.] 0x0000000000160a11
>      0.07%     -0.04%  libc.so.6
> [.] __new_sem_wait_slow64.constprop.0
>      0.10%     -0.04%  radeonsi_dri.so
> [.] 0x0000000000019454
>      0.05%     -0.03%  [amdgpu]
> [k] optc1_get_position
>      0.05%     -0.03%  libc.so.6
> [.] sem_wait@@GLIBC_2.34
>      0.22%     -0.02%  [vdso]
> [.] 0x00000000000005a0
>      0.10%     -0.02%  libc.so.6
> [.] __memcmp_evex_movbe
>                +0.02%  [JIT] tid 8383
> [.] 0x00007f2de0052823
>
>
> > - Could it be an inconclusive bisection?
>
> I checked twice:
> [6.7] - 83 FPS
> [aaa2c9a97c22] - 111 FPS
> [cc478e0b6bdf] - 64 FPS
> [6.8-rc2 with patches] - 82 FPS
>
>
> [6.7] https://i.postimg.cc/15yyzZBr/v6-7.png
> [6.7 perf] https://mega.nz/file/QwJ3hbob#RslLFVYgz1SWMcPR3eF9uEpFuqxdgkwXSatWts-1wVA
>
> [aaa2c9a97c22] https://i.postimg.cc/Sxv4VYhg/git-aaa2c9a97c22af5bf011f6dd8e0538219b45af88.png
> [aaa2c9a97c22 perf]
> https://mega.nz/file/dwQxha4J#2_nBF6uNzY11VX-T-Lr_-60WIMrbl1YEvPgY4CuXqEc
>
> [cc478e0b6bdf] https://i.postimg.cc/W3cQfMfw/git-cc478e0b6bdffd20561e1a07941a65f6c8962cab.png
> [cc478e0b6bdf perf]
> https://mega.nz/file/hl5kwLTC#_4Fg1KBXCnQ-8OElY7EYmPOoDG6ZeZYnKFjamWpklWw
>
> [6.8-rc2 with patches] https://i.postimg.cc/26dPpVsR/v6-8-rc2-with-patches.png
> [6.8-rc2 with patches perf]
> https://mega.nz/file/NxgTAb4L#0KO_WU-svpDw60Y3148RZhELPcUtFg3_VCDzJqSyz34

Thanks a lot for these results. There's definitely something strange
going - I'll try to have a detailed look some time next week.

In the meantime, this is clear: there does not seem to be a regression
between 6.7 and 6.8-rc with the patches, which is what I was
expecting. The fact that aaa2c9a97c22 is so much better could indicate
that until cc478e0b6bdf there was either a bug which turned something
into a no-op - or, the memsets() were acting as some kind of
prefetching hint to the CPU, which in turn caused a significant
reduction in cache misses. I think at this point we're not trying to
fix a regression, because we're on par with 6.7, but trying to make
sense of this information to optimize the code properly without luck
(but not sure if feasible). Hrm....