Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing

["Jarkko Sakkinen" <jarkko@kernel.org>]

On Thu Feb 29, 2024 at 12:55 AM EET, Helen Koike wrote:
> Dear Kernel Community,
>
> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a
> basic test pipeline triggered by code pushes to a GitLab-CI instance. This
> initial version includes static checks (checkpatch and smatch for now) and build
> tests across various architectures and configurations. It leverages an
> integrated cache for efficient build times and introduces a flexible 'scenarios'
> mechanism for subsystem-specific extensions.
>
> tl;dr: check this video to see a quick demo: https://youtu.be/TWiTjhjOuzg,
> but don't forget to check the "Motivation for this work" below. Your feedback,
> whether a simple thumbs up or down, is crucial to determine if it is worthwhile
> to pursue this initiative.
>
> GitLab is an Open Source platform that includes integrated CI/CD. The pipeline
> provided in this patch is designed to work out-of-the-box with any GitLab
> instance, including the gitlab.com Free Tier. If you reach the limits of the
> Free Tier, consider using community instances like https://gitlab.freedesktop.org/.
> Alternatively, you can set up a local runner for more flexibility. The
> bootstrap-gitlab-runner.sh script included with this patch simplifies this
> process, enabling you to run tests on your preferred infrastructure, including
> your own machine.
>
> For detailed information, please refer to the documentation included in the
> patch, or check the rendered version here: https://koike.pages.collabora.com/-/linux/-/jobs/298498/artifacts/artifacts/Documentation-output/ci/gitlab-ci/gitlab-ci.html .
>
>
> Motivation for this Work
> ========================
>
> We all know tests are a major topic in the community, so let's mention the
> specificities of this approach:
>
> 1. **Built-in User Interface:** GitLab CI/CD is growing in popularity and has an
> user-friendly interface. Our experience with the upstream DRM-CI in the kernel
> tree (see this blog post [https://www.collabora.com/news-and-blog/blog/2024/02/08/drm-ci-a-gitlab-ci-pipeline-for-linux-kernel-testing/] )
> has provided insights into how such a system can benefit the wider community.
>
> 2. **Distributed Infrastructure:**
> The proposed GitLab-CI pipeline is designed with a distributed infrastructure
> model, being possible to run in any gitlab instance. 
>
> 3. **Reduce regressions:** Fostering a culture where people habitually run
> validated tests and post their results can prevent many issues in post-merge
> tests.
>
> 4. **Collaborative Testing Environment:** The kernel community is already
> engaged in numerous testing efforts, including various GitLab-CI pipelines such
> as DRM-CI, which I maintain, along with other solutions like KernelCI and
> BPF-CI. This proposal is designed to further stimulate contributions to the
> evolving testing landscape. Our goal is to establish a comprehensive suite of
> common tools and files.
>
> 5. **Ownership of QA:** 
> Discrepancies between kernel code and outdated tests often lead to misattributed
> failures, complicating regression tracking. This issue, often arising from
> neglected or deprioritized test updates, creates uncertainty about the source of
> failures. Adopting an "always green pipeline" approach, as detailed in this
> patch's documentation, encourages timely maintenance and validation of tests.
> This ensures that testing accurately reflects the current state of the kernel,
> thereby improving the effectiveness of our QA processes.
>
> Additionally, if we discover that this method isn't working for us, we can
> easily remove it from the codebase, as it is primarily contained within the ci/
> folder.

Not to criticize but I can do  tests I ever want with either Github
or Gitlab simply by bootstrapping BuildRoot on top of whatever the CI
runner has. So I essentially need just enough deps to make a BR build,
and that's it. And e.g. could run x86 tests on ARM ISA runner with zero
issues. And can even have emulated TPM chip in the QEMU VM by building
swtpm.

I had this for some time running actually Gitlab runner. It does not
currently build QEMU but then it also did that:

https://gitlab.com/jarkkojs/linux-tpmdd-test

Essentially just executing this sequence:

git clone https://gitlab.com/jarkkojs/linux-tpmdd-test.git
cd linux-tpmdd-test
cmake -Bbuild && make -Cbuild buildroot-prepare
make -Cbuild/buildroot/build
build/buildroot/build/images/run-tests.sh

I use TCL's "expect" to make conclusions from the output :-)

I'm assuming that this has a bigger point that I can understand right
now but makes me a bit puzzled given that it is quite trivial problem
to my understanding (if you want to pursue to it). Like one work 
week maybe but not more than that...

Especially it feels weird that it needs kernel to be patched at all and
when I did read the motivation but it has sort of whitepaperish stuff
that does not really explain me the edge of this compared to e.g. to my
ad-hoc but still very usable solution (which is agnostic to runner's CPU
architecture, can emulated hardware and works in any possible hosting
with CI).

So maybe my review comment it this: do not assume that this would be
entirely new thing. It is not, and I'm sure some other people have
done this too in the past rather than just me. Instead this should
explain why this is so great that even kernel tree needs to be patched?

BR, Jarkko