Re: [PATCH v7 1/7] Add AutoFDO support for Clang build

[Rong Xu <xur@google.com>]

Hi Peter, thanks for verifying and filing the bug report!

-Rong

On Tue, Nov 5, 2024 at 9:19 AM Peter Jung <ptr1337@cachyos.org> wrote:
>
> Here the bugreport, in case someone wants to track it:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=32340
>
> On 05.11.24 15:56, Peter Jung wrote:
> > You were right - reverting commit:
> > https://github.com/bminor/binutils-gdb/commit/
> > b20ab53f81db7eefa0db00d14f06c04527ac324c from the 2.43 branch does fix
> > the packaging.
> >
> > I will forward this to an issue at their bugzilla.
> >
> > On 05.11.24 15:33, Peter Jung wrote:
> >> Hi Rong,
> >>
> >> Glad that you were able to reproduce the issue!
> >> Thanks for finding the root cause as well as the part of the code.
> >> This really helps.
> >>
> >> I was able to do a successful packaging with binutils 2.42.
> >> Lets forward this to the binutils tracker and hope this will be soon
> >> solved. 🙂
> >>
> >> I have tested this also on the latest commit
> >> (e1e4078ac59740a79cd709d61872abe15aba0087) and the issue is also
> >> reproducible there.
> >>
> >> Thanks for your time! I dont see this as blocker. 🙂
> >> It gets time to get this series merged :P
> >>
> >> Best regards,
> >>
> >> Peter
> >>
> >>
> >>
> >> On 05.11.24 08:25, Rong Xu wrote:
> >>> We debugged this issue and we found the failure seems to only happen
> >>> with strip (version 2.43) in binutil.
> >>>
> >>> For a profile-use compilation, either with -fprofile-use (PGO or
> >>> iFDO), or -fprofile-sample-use (AutoFDO),
> >>> an ELF section of .llvm.call-graph-profile is created for the object.
> >>> For some reasons (like to save space?),
> >>> the relocations in this section are of type "rel', rather the more
> >>> common "rela" type.
> >>>
> >>> In this case,
> >>> $ readelf -r kvm.ko |grep llvm.call-graph-profile
> >>> Relocation section '.rel.llvm.call-graph-profile' at offset 0xf62a00
> >>> contains 4 entries:
> >>>
> >>> strip (v2.43.0) has difficulty handling the relocations in
> >>> .rel.llvm.call-graph-profile -- it silently failed with --strip-debug.
> >>> But strip (v.2.42) has no issue with kvm.ko. The strip in llvm (i.e.
> >>> llvm-strip) also passes with kvm.ko
> >>>
> >>> I compared binutil/strip source code for version v2.43.0 and v2.42.
> >>> The different is around here:
> >>> In v2.42 of bfd/elfcode.h
> >>>     1618       if ((entsize == sizeof (Elf_External_Rela)
> >>>     1619            && ebd->elf_info_to_howto != NULL)
> >>>     1620           || ebd->elf_info_to_howto_rel == NULL)
> >>>     1621         res = ebd->elf_info_to_howto (abfd, relent, &rela);
> >>>     1622       else
> >>>     1623         res = ebd->elf_info_to_howto_rel (abfd, relent, &rela);
> >>>
> >>> In v2.43.0 of bfd/elfcode.h
> >>>     1618       if (entsize == sizeof (Elf_External_Rela)
> >>>     1619           && ebd->elf_info_to_howto != NULL)
> >>>     1620         res = ebd->elf_info_to_howto (abfd, relent, &rela);
> >>>     1621       else if (ebd->elf_info_to_howto_rel != NULL)
> >>>     1622         res = ebd->elf_info_to_howto_rel (abfd, relent, &rela);
> >>>
> >>> In the 2.43 strip, line 1618 is false and line 1621 is also false.
> >>> "res" is returned as false and the program exits with -1.
> >>>
> >>> While in 2.42, line 1620 is true and we get "res" from line 1621 and
> >>> program functions correctly.
> >>>
> >>> I'm not familiar with binutil code base and don't know the reason for
> >>> removing line 1620.
> >>> I can file a bug for binutil for people to further investigate this.
> >>>
> >>> It seems to me that this issue should not be a blocker for our patch.
> >>>
> >>> Regards,
> >>>
> >>> -Rong
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Nov 4, 2024 at 12:24 PM Han Shen<shenhan@google.com> wrote:
> >>>> Hi Peter,
> >>>> Thanks for providing the detailed reproduce.
> >>>> Now I can see the error (after I synced to 6.12.0-rc6, I was using
> >>>> rc5).
> >>>> I'll look into that and report back.
> >>>>
> >>>>> I have tested your provided method, but the AutoFDO profile (lld does
> >>>> not get lto-sample-profile=$pathtoprofile passed)
> >>>>
> >>>> I see. You also turned on ThinLTO, which I didn't, so the profile was
> >>>> only used during compilation, not passed to lld.
> >>>>
> >>>> Thanks,
> >>>> Han
> >>>>
> >>>> On Mon, Nov 4, 2024 at 9:31 AM Peter Jung<ptr1337@cachyos.org> wrote:
> >>>>> Hi Han,
> >>>>>
> >>>>> I have tested your provided method, but the AutoFDO profile (lld does
> >>>>> not get lto-sample-profile=$pathtoprofile passed)  nor Clang as
> >>>>> compiler
> >>>>> gets used.
> >>>>> Please replace following PKGBUILD and config from linux-mainline with
> >>>>> the provided one in the gist. The patch is also included there.
> >>>>>
> >>>>> https://gist.github.com/ptr1337/c92728bb273f7dbc2817db75eedec9ed
> >>>>>
> >>>>> The main change I am doing here, is passing following to the build
> >>>>> array
> >>>>> and replacing "make all":
> >>>>>
> >>>>> make LLVM=1 LLVM_IAS=1 CLANG_AUTOFDO_PROFILE=${srcdir}/perf.afdo all
> >>>>>
> >>>>> When compiling the kernel with makepkg, this results at the
> >>>>> packaging to
> >>>>> following issue and can be reliable reproduced.
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Peter
> >>>>>
> >>>>>
> >>>>> On 04.11.24 05:50, Han Shen wrote:
> >>>>>> Hi Peter, thanks for reporting the issue. I am trying to reproduce it
> >>>>>> in the up-to-date archlinux environment. Below is what I have:
> >>>>>>     0. pacman -Syu
> >>>>>>     1. cloned archlinux build files from
> >>>>>> https://aur.archlinux.org/linux-mainline.git the newest mainline
> >>>>>> version is 6.12rc5-1.
> >>>>>>     2. changed the PKGBUILD file to include the patches series
> >>>>>>     3. changed the "config" to turn on clang autofdo
> >>>>>>     4. collected afdo profiles
> >>>>>>     5. MAKEFLAGS="-j48 V=1 LLVM=1 CLANG_AUTOFDO_PROFILE=$(pwd)/
> >>>>>> perf.afdo" \
> >>>>>>           makepkg -s --skipinteg --skippgp
> >>>>>>     6. install and reboot
> >>>>>> The above steps succeeded.
> >>>>>> You mentioned the error happens at "module_install", can you instruct
> >>>>>> me how to execute the "module_install" step?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Han
> >>>>>>
> >>>>>> On Sat, Nov 2, 2024 at 12:53 PM Peter Jung<ptr1337@cachyos.org>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> On 02.11.24 20:46, Peter Jung wrote:
> >>>>>>>> On 02.11.24 18:51, Rong Xu wrote:
> >>>>>>>>> Add the build support for using Clang's AutoFDO. Building the
> >>>>>>>>> kernel
> >>>>>>>>> with AutoFDO does not reduce the optimization level from the
> >>>>>>>>> compiler. AutoFDO uses hardware sampling to gather information
> >>>>>>>>> about
> >>>>>>>>> the frequency of execution of different code paths within a
> >>>>>>>>> binary.
> >>>>>>>>> This information is then used to guide the compiler's optimization
> >>>>>>>>> decisions, resulting in a more efficient binary. Experiments
> >>>>>>>>> showed that the kernel can improve up to 10% in latency.
> >>>>>>>>>
> >>>>>>>>> The support requires a Clang compiler after LLVM 17. This
> >>>>>>>>> submission
> >>>>>>>>> is limited to x86 platforms that support PMU features like LBR on
> >>>>>>>>> Intel machines and AMD Zen3 BRS. Support for SPE on ARM 1,
> >>>>>>>>>     and BRBE on ARM 1 is part of planned future work.
> >>>>>>>>>
> >>>>>>>>> Here is an example workflow for AutoFDO kernel:
> >>>>>>>>>
> >>>>>>>>> 1) Build the kernel on the host machine with LLVM enabled, for
> >>>>>>>>> example,
> >>>>>>>>>           $ make menuconfig LLVM=1
> >>>>>>>>>        Turn on AutoFDO build config:
> >>>>>>>>>          CONFIG_AUTOFDO_CLANG=y
> >>>>>>>>>        With a configuration that has LLVM enabled, use the
> >>>>>>>>> following
> >>>>>>>>>        command:
> >>>>>>>>>           scripts/config -e AUTOFDO_CLANG
> >>>>>>>>>        After getting the config, build with
> >>>>>>>>>          $ make LLVM=1
> >>>>>>>>>
> >>>>>>>>> 2) Install the kernel on the test machine.
> >>>>>>>>>
> >>>>>>>>> 3) Run the load tests. The '-c' option in perf specifies the
> >>>>>>>>> sample
> >>>>>>>>>       event period. We suggest     using a suitable prime number,
> >>>>>>>>>       like 500009, for this purpose.
> >>>>>>>>>       For Intel platforms:
> >>>>>>>>>          $ perf record -e BR_INST_RETIRED.NEAR_TAKEN:k -a -N -b -c
> >>>>>>>>> <count> \
> >>>>>>>>>            -o <perf_file> -- <loadtest>
> >>>>>>>>>       For AMD platforms:
> >>>>>>>>>          The supported system are: Zen3 with BRS, or Zen4 with
> >>>>>>>>> amd_lbr_v2
> >>>>>>>>>         For Zen3:
> >>>>>>>>>          $ cat proc/cpuinfo | grep " brs"
> >>>>>>>>>          For Zen4:
> >>>>>>>>>          $ cat proc/cpuinfo | grep amd_lbr_v2
> >>>>>>>>>          $ perf record --pfm-events
> >>>>>>>>> RETIRED_TAKEN_BRANCH_INSTRUCTIONS:k
> >>>>>>>>> -a \
> >>>>>>>>>            -N -b -c <count> -o <perf_file> -- <loadtest>
> >>>>>>>>>
> >>>>>>>>> 4) (Optional) Download the raw perf file to the host machine.
> >>>>>>>>>
> >>>>>>>>> 5) To generate an AutoFDO profile, two offline tools are
> >>>>>>>>> available:
> >>>>>>>>>       create_llvm_prof and llvm_profgen. The create_llvm_prof
> >>>>>>>>> tool is part
> >>>>>>>>>       of the AutoFDO project and can be found on GitHub
> >>>>>>>>>       (https://github.com/google/autofdo), version v0.30.1 or
> >>>>>>>>> later. The
> >>>>>>>>>       llvm_profgen tool is included in the LLVM compiler
> >>>>>>>>> itself. It's
> >>>>>>>>>       important to note that the version of llvm_profgen
> >>>>>>>>> doesn't need to
> >>>>>>>>>       match the version of Clang. It needs to be the LLVM 19
> >>>>>>>>> release or
> >>>>>>>>>       later, or from the LLVM trunk.
> >>>>>>>>>          $ llvm-profgen --kernel --binary=<vmlinux> --
> >>>>>>>>> perfdata=<perf_file> \
> >>>>>>>>>            -o <profile_file>
> >>>>>>>>>       or
> >>>>>>>>>          $ create_llvm_prof --binary=<vmlinux> --
> >>>>>>>>> profile=<perf_file> \
> >>>>>>>>>            --format=extbinary --out=<profile_file>
> >>>>>>>>>
> >>>>>>>>>       Note that multiple AutoFDO profile files can be merged
> >>>>>>>>> into one via:
> >>>>>>>>>          $ llvm-profdata merge -o <profile_file>  <profile_1> ...
> >>>>>>>>> <profile_n>
> >>>>>>>>>
> >>>>>>>>> 6) Rebuild the kernel using the AutoFDO profile file with the
> >>>>>>>>> same config
> >>>>>>>>>       as step 1, (Note CONFIG_AUTOFDO_CLANG needs to be enabled):
> >>>>>>>>>          $ make LLVM=1 CLANG_AUTOFDO_PROFILE=<profile_file>
> >>>>>>>>>
> >>>>>>>>> Co-developed-by: Han Shen<shenhan@google.com>
> >>>>>>>>> Signed-off-by: Han Shen<shenhan@google.com>
> >>>>>>>>> Signed-off-by: Rong Xu<xur@google.com>
> >>>>>>>>> Suggested-by: Sriraman Tallam<tmsriram@google.com>
> >>>>>>>>> Suggested-by: Krzysztof Pszeniczny<kpszeniczny@google.com>
> >>>>>>>>> Suggested-by: Nick Desaulniers<ndesaulniers@google.com>
> >>>>>>>>> Suggested-by: Stephane Eranian<eranian@google.com>
> >>>>>>>>> Tested-by: Yonghong Song<yonghong.song@linux.dev>
> >>>>>>>>> Tested-by: Yabin Cui<yabinc@google.com>
> >>>>>>>>> Tested-by: Nathan Chancellor<nathan@kernel.org>
> >>>>>>>>> Reviewed-by: Kees Cook<kees@kernel.org>
> >>>>>>>> Tested-by: Peter Jung<ptr1337@cachyos.org>
> >>>>>>>>
> >>>>>>> The compilations and testing with the "make pacman-pkg" function
> >>>>>>> from
> >>>>>>> the kernel worked fine.
> >>>>>>>
> >>>>>>> One problem I do face:
> >>>>>>> When I apply a AutoFDO profile together with the PKGBUILD [1] from
> >>>>>>> archlinux im running into issues at "module_install" at the
> >>>>>>> packaging.
> >>>>>>>
> >>>>>>> See following log:
> >>>>>>> ```
> >>>>>>> make[2]: *** [scripts/Makefile.modinst:125:
> >>>>>>> /tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-
> >>>>>>> autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/
> >>>>>>> arch/x86/kvm/kvm.ko]
> >>>>>>> Error 1
> >>>>>>> make[2]: *** Deleting file
> >>>>>>> '/tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-
> >>>>>>> autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/
> >>>>>>> arch/x86/kvm/kvm.ko'
> >>>>>>>      INSTALL
> >>>>>>> /tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-
> >>>>>>> autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/
> >>>>>>> crypto/cryptd.ko
> >>>>>>> make[2]: *** Waiting for unfinished jobs....
> >>>>>>> ```
> >>>>>>>
> >>>>>>>
> >>>>>>> This can be fixed with removed "INSTALL_MOD_STRIP=1" to the passed
> >>>>>>> parameters of module_install.
> >>>>>>>
> >>>>>>> This explicitly only happens, if a profile is passed - otherwise the
> >>>>>>> packaging works without problems.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Peter Jung
> >>>>>>>
> >>
> >
>