kernel.org tooling update

kernel.org tooling update

[Konstantin Ryabitsev]

Hi, all:

These are the topics that were touched on at the maintainer summit when
discussing tooling on the kernel.org side of things.

--

# What is the state of tooling?

## b4 development update

Past year:

- No major new features in b4 over the past year, excepting `b4 dig`
- Seeing lots of adoption and use across subsystem, with a lot of maintainers
  recommending b4 as the preferred mechanism to submit patches
- Starting to see adoption by several non-kernel projects (openembedded, u-boot, others)
- Significant behind-the-scenes move of codebase to stricter typed code
- Continued work on `b4 review` that got shelved temporarily for other
  priorities.

### LETS PUT MOAR AI INTO IT!!1

I spent a lot of time on trying to integrate AI into b4 workflows, but with
little to show for it in the end due to lackluster results.

- Used local ollama as opposed to proprietary services, with the goal to avoid
  introducing hard dependencies on third-party commercial tooling. This is
  probably the main reason why my results were not so exciting as what others
  see with much more powerful models.

- Focused on thread/series summarization features as opposed to code analysis:

    - Summarize follow-ups (trailers, acks/nacks received), though this is
      already fairly well-handed with non-AI tooling.

    - Gauge "temperature" of the discussion to highlight controversial series.

    - Gauge quality of the submission; help decide "is this series worth
      looking at" before maintainers spend their effort looking at it, using
      maintainer-tailored prompts. This may be better done via CI/patchwork
      integration, than with b4.

    - Use LLM to prepare a merge commit message using the cover letter and
      summarizing the patches.

I did not end up releasing any features based on that work, because:

    - LLM was not fantastic at following discussions and keeping a clear
      picture of who said what, which is kind of crucial for maintainer
      decision making.

    - Very large series and huge threads run out fo context window, which
      causes the LLM to get even worse at "who said what" (and it's
      already not that great at it).

    - Thread analysis requires lots of VRAM and a modern graphics card, and is
      still fairly slow there (I used a fairly powerful GeForce RTX).

    - Actual code review is best if it happens post-apply in a temporary
      workdir or a temporary branch, so the agent can see the change in the
      context of the git tree and the entire codebase, not just the context
      lines of the patch itself.

I did have much better success when I worked to represent a thread not as
multiple messages, but as a single document with all interleaved follow-up
conversations collated together. However, this was done manually --
representing emails from arbitrary threads as such collated documents is a
separate challenge.

Using proprietary models and remote services will probably show better
results, but I did not have the funds or the inkling to do it (plus see the
concern for third-party commercial tooling). I may need to collaborate more
closely with the maintainers already doing it on their own instead of
continuing my separate work on it.

### AI crawler scourge

While working on LLM integration, it was certainly ironic that one of the
top challenges for us was to try to keep AI crawlers from overwhelming
kernel.org infrastructure. While we've put several mitigations in place, it's
a temporary relief at best.

## Continuous degradation of SMTP

We're increasingly having to deal with the degradation of the SMTP support by
all commercial companies:

    - major hosts are increasingly not interested in getting mail from anyone
      who isn't also a major mail service provider

    - their "bulk sender" guidelines are no good for us (e.g. requiring that
      we add one-click unsubscribe footers to all email)

    - their "spam filters" are increasingly based on training data, which
      means that "looks different from what most of our users receive" is
      enough to have patches and code discussions put into the "Junk" folder

    - they apply arbitrary throttling ("too many deliveries for the same
      message-id", "too many messages from the DKIM domain foobar.com")

    - anti-phishing services at commercial IT companies do horrible things to
      incoming messages

## Are we finally moving away from patches sent over email?

There are still important concerns when we consider moving away from "patches
sent via email":

    - SMTP is still the only widely used protocol we have for decentralized
      communication; everything else is experimental or has important
      drawbacks, such as:

        - it relies on single-point-of-failure services (e.g. Signal), or
        - it requires standing up esoteric software (which then become
          single-point-of-failure services), or
        - it requires an "everyone-must-switch-now" flag day

    - RFC-5322, with all its warts, is a well-defined standard for internet
      messages:

        - robust, capable of dealing with change while preserving legacy
        - easy to parse with libraries for almost any framework
        - easy to archive and query
        - has lots of tooling built around it

With lore and public-inbox, we *are* in the process of moving away from
relying on the increasingly unreliable SMTP layer. Lore can already let you do
the following things:

    - lets anyone submit patches via the web endpoint
    - lets anyone subscribe to lists via several protocols (NNTP, POP, IMAP)
    - lets anyone use lei to receive arbitrary feeds
    - can aggregate any number of sources, as long as they are RFC-5322
      messages (or can be converted to them)

Lore and public-inbox is becoming a kind of a distributed, replicating
messaging bus with a robust query and retrieval interface on top of it, and I
believe it's a fairly powerful framework we can build upon.

## Work on "local lore"

One downside of lore.kernel.org is that it's a central service, which runs
counter to our goal of limiting how many single points of failure we have.
There is fault-tolerance built into the system (lore.kernel.org is actually 4
different nodes in various parts of the world), but an adversary would have no
difficulty knocking out all nodes at once, which would impact the project
significantly.

The "local lore" projects it the attempt to provide a kind of "maintainer
container" that can be run locally or in any public cloud:

    - comes with a 6-month constantly-updating mirror of lore, using a
      failover set of replication URLs (including tor/onion)
    - comes with a pre-configured mirror of git repositories that are kept
      up-to-date in the same fashion
    - lets the maintainer set up lei queries that can push into their
      inbox, supporting Gmail+OAuth, JMAP, IMAP
    - provides a web submission endpoint and an SMTP service that can
      integrate with other SMTP relays
    - publishes a public-inbox feed of maintainer activity that central
      lore can pick up and integrate

There is a parallel goal here, which is to make it easier for devs to assume
maintainer duties without having to spend a week setting up their tooling.
In theory, all they would be need to do is to set up their maintainer
container and then use the web menu to choose which feeds they want to pull
and where they want messages delivered.

This project is still early in development, but I hope to be able to provide
test containers soon that people can set up and run.

## Other tools

### Bugzilla

It may be time to kill bugzilla:

    - despite periodic "we're not dead yet" emails, it doesn't appear very
      active
    - the upgrade path to 6.0 is broken for us due to bugzilla abandoning the
      5.2 development branch and continuing with 5.1
    - question remains with what to replace bugzilla, but it's a longer
      discussion topic that I don't want to raise here; it may be a job for
      the bugspray bot that can extend the two-way bridge functionality to
      multiple bug tracker frameworks

### Patchwork

Patchwork continues to be used widely:

    - we've introduced query-based patchworks, where instead of consuming the
      entire mailing list, we feed it the results of lei queries
    - I'm hoping to work with upstream to add a couple of features that would
      be of benefit to us, such as:

        - support for annotating patches and series (e.g. with LLM summaries)
        - an API endpoint to submit patches, so maintainers could add
          arbitrary series to their patchwork project, integrating with b4

## Web of Trust work

There is an ongoing work to replace our home-grown web of trust solution (that
does work but has important bottlenecks and scaling limitations) with
something both more distributed and easier to maintain. We're working with
OpenSSF to design the framework and I hope to present it to the community in
the next few months.

## Questions?

Send away!

-K

Re: kernel.org tooling update

[Mauro Carvalho Chehab]

Hi Konstantin,

Em Tue, 9 Dec 2025 23:48:24 -0500
Konstantin Ryabitsev <konstantin@linuxfoundation.org> escreveu:

> I spent a lot of time on trying to integrate AI into b4 workflows, but with
> little to show for it in the end due to lackluster results.
> 
> - Used local ollama as opposed to proprietary services, with the goal to avoid
>   introducing hard dependencies on third-party commercial tooling. This is
>   probably the main reason why my results were not so exciting as what others
>   see with much more powerful models.
> 
> - Focused on thread/series summarization features as opposed to code analysis:
> 
>     - Summarize follow-ups (trailers, acks/nacks received), though this is
>       already fairly well-handed with non-AI tooling.
> 
>     - Gauge "temperature" of the discussion to highlight controversial series.
> 
>     - Gauge quality of the submission; help decide "is this series worth
>       looking at" before maintainers spend their effort looking at it, using
>       maintainer-tailored prompts. This may be better done via CI/patchwork
>       integration, than with b4.
> 
>     - Use LLM to prepare a merge commit message using the cover letter and
>       summarizing the patches.
> 
> I did not end up releasing any features based on that work, because:
> 
>     - LLM was not fantastic at following discussions and keeping a clear
>       picture of who said what, which is kind of crucial for maintainer
>       decision making.
> 
>     - Very large series and huge threads run out fo context window, which
>       causes the LLM to get even worse at "who said what" (and it's
>       already not that great at it).
> 
>     - Thread analysis requires lots of VRAM and a modern graphics card, and is
>       still fairly slow there (I used a fairly powerful GeForce RTX).
> 
>     - Actual code review is best if it happens post-apply in a temporary
>       workdir or a temporary branch, so the agent can see the change in the
>       context of the git tree and the entire codebase, not just the context
>       lines of the patch itself.
> 
> I did have much better success when I worked to represent a thread not as
> multiple messages, but as a single document with all interleaved follow-up
> conversations collated together. However, this was done manually --
> representing emails from arbitrary threads as such collated documents is a
> separate challenge.

I would love to see what you got there. I tried to an experiment similar
to it, also with ollama, writing some code Python code from scratch, aiming
to run locally on my GPU (with has only 16GB VRAM but it is a brand new
RDNA4 GPU), using a prompt similar to this:

            You are an expert at summarizing email threads and discussion forums. 
	    Your task is to analyze the following text, which is a chunk of an
	    email thread with nested replies, and provide a concise, structured summary.

            **Instructions:**
            1.  **Reconstruct the Chronology:** Carefully analyze the indentation levels (e.g., `>>>`, `>`, `>>`) 
		and timestamps to determine the correct order of messages. The oldest message is likely the most indented.
            2.  **Identify Speakers:** For each message, extract the first name from the "From:" field (e.g., "From: John Doe" becomes "John").
            3.  **Consolidate by Topic and Speaker:** Group the main discussion points by topic. 
		For each topic, summarize what each person contributed, consolidating their points 
		even if they appear in multiple messages.
            4.  **Focus on New Information:** Ignore salutations (e.g., "Hi Mike,") and email
		signature blocks. Focus on the substantive content of each message.
            5.  **Output Format:** Provide the summary in the following structure:
                -   **Main Topic(s) of Discussion:** [List 1-3 main topics]
                -   **Summary by Participant:**
                    -   **[First Name 1]:** [Concise summary of their stance, questions,
			or information provided, in chronological order if important.]
                    -   **[First Name 2]:** [Concise summary of their stance, questions,
			or information provided.]
                -   **Outcome/Next Steps:** [Note any conclusions, decisions, or action items agreed upon.]

            **Text to Summarize:**
            {chunk}

Yet, grouping e-mails per thread is a challenge, specially since
I was planning to ask it to summarize it in short time intervals,
so, picking only the newer emails, and re-using already parsed data.

My goal is not to handle patches, as I doubt this would give anything
relevant. Instead, I wanted to keep in track with LKML and other high
traffic mailing lists to pick most relevant threads.

Btw, I got some success summarizing patch series from a given Kernel author
along an entire month using just the e-mail subject, with mistral-small3.2
LLM model, and a somewhat complex prompt. Goal was to summarize how many
patches were submitted, grouping them by threads and different open source
projects. Output were far a way from being perfect, and, if the number of
patches is too big, it starts forgetting about the context - with is one of
the current challenges with current LLM technology - even on proprietary
models.

It sounds to me that, with the current technology, the best approach
would be to ask AI to summarize each e-mail individually, then group 
the results using a non-AI approach (or mixing AI with normal programming).

> Using proprietary models and remote services will probably show better
> results, but I did not have the funds or the inkling to do it (plus see the
> concern for third-party commercial tooling). I may need to collaborate more
> closely with the maintainers already doing it on their own instead of
> continuing my separate work on it.

Yeah, the best is to have this not dependent on proprietary
models or on external GPU farms. I wonder if a DSX Spark would be
reasonably good with its 128GB unified RAM for something like that.
Its price is still too high, but maybe we'll end having similar
models next year to allow local tests with bigger models.

Thanks,
Mauro

Re: kernel.org tooling update

[Thorsten Leemhuis]

Lo! Thx for the update, much appreciated!

On 12/10/25 05:48, Konstantin Ryabitsev wrote:

> ### Bugzilla
> 
> It may be time to kill bugzilla:

Thx for bringing this up, as I a few months ago again looked somewhat
closer at the state of how well our bugzilla is working for the core
kernel. I didn't post the analysis in the end, but to me it looked like
the state of things was round the same as it was three years ago -- when
it wasn't working well, which was among the reasons why we came close to
abandoning bugzilla for kernel bugs[1].

[1] for those that don't remember, see https://lwn.net/Articles/910740/
and
https://lore.kernel.org/all/aa876027-1038-3e4a-b16a-c144f674c0b0@leemhuis.info/



>     - despite periodic "we're not dead yet" emails, it doesn't appear very
>       active
>     - the upgrade path to 6.0 is broken for us due to bugzilla abandoning the
>       5.2 development branch and continuing with 5.1
>     - question remains with what to replace bugzilla,

To me it looks like most subsystems don't care much or at all about
bugzilla.kernel.org. This made me wonder (and maybe you could gather
some opinions on this in Tokyo):

* How many kernel subsystems have a strong interest in a bug tracking
solution at all[2]? And how many of those might be happy by using some
external issue tracker, like those in github (like Rust for Linux,
thesofproject, and a few others do), gitlab (either directly, like
apparmor, or self-hosted, like the DRM subsystem)?

* Does the kernel as a whole need a bug tracking solution at all to
receive reports? We for now require email for patches, so why not for
bugs as well, unless a subsystem really wants something (see above)?

[2] Some numbers:
$ for i in "" mailto bugzilla github gitlab; do echo -n "Searching for
'^B:.*${i}': "; grep -c -E "^B:.*${i}" MAINTAINERS; done
Searching for '^B:.*': 70
Searching for '^B:.*mailto': 12
Searching for '^B:.*bugzilla': 23
Searching for '^B:.*github': 17
Searching for '^B:.*gitlab': 11

> but it's a longer discussion topic that I don't want to raise here;

Would like to be involved there.

> it may be a job for
>       the bugspray bot that can extend the two-way bridge functionality to
>       multiple bug tracker frameworks

FWIW, development of my regression tracker (regzbot) and me using it to
track regressions nearly stalled but is slowly restarting. Would be good
if we could work together here, as there is some overlap -- and
regression tracking afaics is something that a lot of people want and
consider important. And regzbot is already capable of monitoring reports
in various places (lore, gitlab, github, bugzilla); so if we decide that
we don't need a tracker for the kernel as a whole, it might already do
nearly everything for the bugs where tracking really helps a lot.

Ciao, Thorsten

Re: kernel.org tooling update

[Theodore Tso]

On Wed, Dec 10, 2025 at 02:30:37PM +0100, Thorsten Leemhuis wrote:
> * How many kernel subsystems have a strong interest in a bug tracking
> solution at all[2]? And how many of those might be happy by using some
> external issue tracker, like those in github (like Rust for Linux,
> thesofproject, and a few others do), gitlab (either directly, like
> apparmor, or self-hosted, like the DRM subsystem)?

One of the discussions we had (both during and after Konstantin's
tools session) was that all we really need is some kind of way of
associating state with a set of URL's to lore --- this could be used
to indicate "this is a bug report" and a set of flags "this bug has
been resolved / needs more information / needs triaging, etc".  A
different set of states is also what we would need as a replacement
for patchwork --- this patch series is a not applicable for a
subsystem, has been applied, has been rejected, etc.

This is quite similar to what you've done with your regzbot dashboard,
actually.

> FWIW, development of my regression tracker (regzbot) and me using it to
> track regressions nearly stalled but is slowly restarting. Would be good
> if we could work together here, as there is some overlap -- and
> regression tracking afaics is something that a lot of people want and
> consider important. And regzbot is already capable of monitoring reports
> in various places (lore, gitlab, github, bugzilla); so if we decide that
> we don't need a tracker for the kernel as a whole, it might already do
> nearly everything for the bugs where tracking really helps a lot.

  	 	    	    	       		- Ted

Re: kernel.org tooling update

[Stephen Hemminger]

On Wed, 10 Dec 2025 14:30:37 +0100
Thorsten Leemhuis <linux@leemhuis.info> wrote:

> he update, much appreciated!
> 
> On 12/10/25 05:48, Konstantin Ryabitsev wrote:
> 
> > ### Bugzilla
> > 
> > It may be time to kill bugzilla:  
> 
> Thx for bringing this up, as I a few months ago again looked somewhat
> closer at the state of how well our bugzilla is working for the core
> kernel. I didn't post the analysis in the end, but to me it looked like
> the state of things was round the same as it was three years ago -- when
> it wasn't working well, which was among the reasons why we came close to
> abandoning bugzilla for kernel bugs[1].
> 
> [1] for those that don't remember, see https://lwn.net/Articles/910740/
> and
> https://lore.kernel.org/all/aa876027-1038-3e4a-b16a-c144f674c0b0@leemhuis.info/
> 
> 
> 
> >     - despite periodic "we're not dead yet" emails, it doesn't appear very
> >       active
> >     - the upgrade path to 6.0 is broken for us due to bugzilla abandoning the
> >       5.2 development branch and continuing with 5.1
> >     - question remains with what to replace bugzilla,  
> 
> To me it looks like most subsystems don't care much or at all about
> bugzilla.kernel.org. This made me wonder (and maybe you could gather
> some opinions on this in Tokyo):
> 
> * How many kernel subsystems have a strong interest in a bug tracking
> solution at all[2]? And how many of those might be happy by using some
> external issue tracker, like those in github (like Rust for Linux,
> thesofproject, and a few others do), gitlab (either directly, like
> apparmor, or self-hosted, like the DRM subsystem)?
> 
> * Does the kernel as a whole need a bug tracking solution at all to
> receive reports? We for now require email for patches, so why not for
> bugs as well, unless a subsystem really wants something (see above)?

I am the default target for all networking bugzilla submissions.
Would be very happy to just see bugzilla die.
Right now, all I do is do a quick scan and respond to the junk submissions
and forward the rest to the netdev mailing list with a note on the bug
to go there in the future.

Issue tracking is not in the workflow for the community.

Re: kernel.org tooling update

[Randy Dunlap]

On 12/12/25 3:48 PM, Stephen Hemminger wrote:
> On Wed, 10 Dec 2025 14:30:37 +0100
> Thorsten Leemhuis <linux@leemhuis.info> wrote:
> 
>> he update, much appreciated!
>>
>> On 12/10/25 05:48, Konstantin Ryabitsev wrote:
>>
>>> ### Bugzilla
>>>
>>> It may be time to kill bugzilla:  
>>
>> Thx for bringing this up, as I a few months ago again looked somewhat
>> closer at the state of how well our bugzilla is working for the core
>> kernel. I didn't post the analysis in the end, but to me it looked like
>> the state of things was round the same as it was three years ago -- when
>> it wasn't working well, which was among the reasons why we came close to
>> abandoning bugzilla for kernel bugs[1].
>>
>> [1] for those that don't remember, see https://lwn.net/Articles/910740/
>> and
>> https://lore.kernel.org/all/aa876027-1038-3e4a-b16a-c144f674c0b0@leemhuis.info/
>>
>>
>>
>>>     - despite periodic "we're not dead yet" emails, it doesn't appear very
>>>       active
>>>     - the upgrade path to 6.0 is broken for us due to bugzilla abandoning the
>>>       5.2 development branch and continuing with 5.1
>>>     - question remains with what to replace bugzilla,  
>>
>> To me it looks like most subsystems don't care much or at all about
>> bugzilla.kernel.org. This made me wonder (and maybe you could gather
>> some opinions on this in Tokyo):
>>
>> * How many kernel subsystems have a strong interest in a bug tracking
>> solution at all[2]? And how many of those might be happy by using some
>> external issue tracker, like those in github (like Rust for Linux,
>> thesofproject, and a few others do), gitlab (either directly, like
>> apparmor, or self-hosted, like the DRM subsystem)?
>>
>> * Does the kernel as a whole need a bug tracking solution at all to
>> receive reports? We for now require email for patches, so why not for
>> bugs as well, unless a subsystem really wants something (see above)?
> 
> I am the default target for all networking bugzilla submissions.
> Would be very happy to just see bugzilla die.
> Right now, all I do is do a quick scan and respond to the junk submissions
> and forward the rest to the netdev mailing list with a note on the bug
> to go there in the future.
> 
> Issue tracking is not in the workflow for the community.

which can be observed by the number of Categories/Components that don't
have an assignee -- even a mailing list.  Which should be "fixed" IMO.
I.e., we are actively helping bugzilla entries to be ignored.

I don't think it's the tool itself. More likely something else...

-- 
~Randy

Re: kernel.org tooling update

[Lukas Wunner]

[cc += Bjorn, start of thread is here:
https://lore.kernel.org/ksummit/20251209-roaring-hidden-alligator-068eea@lemur/
]

On Tue, Dec 09, 2025 at 11:48:24PM -0500, Konstantin Ryabitsev wrote:
> ### Bugzilla
> 
> It may be time to kill bugzilla:
> 
>     - despite periodic "we're not dead yet" emails, it doesn't appear very
>       active
>     - the upgrade path to 6.0 is broken for us due to bugzilla abandoning the
>       5.2 development branch and continuing with 5.1
>     - question remains with what to replace bugzilla, but it's a longer
>       discussion topic that I don't want to raise here; it may be a job for
>       the bugspray bot that can extend the two-way bridge functionality to
>       multiple bug tracker frameworks

The PCI subsystem relies heavily on bugzilla to track issues,
collect dmesg/lspci output from reporters and furnish them with
debug or test patches.

The SOP when issues are reported on the mailing list without
sufficient information is to ask the reporter to open a bugzilla
issue and attach full dmesg and lspci -vvv output for analysis.

If bugzilla is deprecated, we'll need at least a way to exchange
files with reporters.  Preferably on kernel.org infrastructure
to be independent from 3rd parties.  A way to communicate with
reporters outside the mailing list is also useful to prevent
spamming linux-pci@vger.kernel.org with messages relevant only
to a single issue or system.

All the information now recorded in bugzilla should continue
to be available indefinitely so that Link: tags in commits
continue to work.  It's not uncommon to have to dig in old
bugzilla entries in order to understand the motivation for
a particular code section that was introduced years earlier.

Thanks,

Lukas

Re: kernel.org tooling update

[Jeff Johnson]

On 12/16/2025 8:21 AM, Lukas Wunner wrote:
> [cc += Bjorn, start of thread is here:
> https://lore.kernel.org/ksummit/20251209-roaring-hidden-alligator-068eea@lemur/
> ]
> 
> On Tue, Dec 09, 2025 at 11:48:24PM -0500, Konstantin Ryabitsev wrote:
>> ### Bugzilla
>>
>> It may be time to kill bugzilla:
>>
>>     - despite periodic "we're not dead yet" emails, it doesn't appear very
>>       active
>>     - the upgrade path to 6.0 is broken for us due to bugzilla abandoning the
>>       5.2 development branch and continuing with 5.1
>>     - question remains with what to replace bugzilla, but it's a longer
>>       discussion topic that I don't want to raise here; it may be a job for
>>       the bugspray bot that can extend the two-way bridge functionality to
>>       multiple bug tracker frameworks
> 
> The PCI subsystem relies heavily on bugzilla to track issues,
> collect dmesg/lspci output from reporters and furnish them with
> debug or test patches.
> 
> The SOP when issues are reported on the mailing list without
> sufficient information is to ask the reporter to open a bugzilla
> issue and attach full dmesg and lspci -vvv output for analysis.
> 
> If bugzilla is deprecated, we'll need at least a way to exchange
> files with reporters.  Preferably on kernel.org infrastructure
> to be independent from 3rd parties.  A way to communicate with
> reporters outside the mailing list is also useful to prevent
> spamming linux-pci@vger.kernel.org with messages relevant only
> to a single issue or system.
> 
> All the information now recorded in bugzilla should continue
> to be available indefinitely so that Link: tags in commits
> continue to work.  It's not uncommon to have to dig in old
> bugzilla entries in order to understand the motivation for
> a particular code section that was introduced years earlier.

At least some of the wireless maintainers also use bugzilla.
The ath11k & ath12k drivers have guidance in the wireless wiki:
https://wireless.docs.kernel.org/en/latest/en/users/drivers/ath11k/bugreport.html
https://wireless.docs.kernel.org/en/latest/en/users/drivers/ath12k/bugreport.html

So we would also want this or a similar service to be maintained.

/jeff

Re: kernel.org tooling update

[Mario Limonciello]

On 12/16/25 2:33 PM, Jeff Johnson wrote:
> On 12/16/2025 8:21 AM, Lukas Wunner wrote:
>> [cc += Bjorn, start of thread is here:
>> https://lore.kernel.org/ksummit/20251209-roaring-hidden-alligator-068eea@lemur/
>> ]
>>
>> On Tue, Dec 09, 2025 at 11:48:24PM -0500, Konstantin Ryabitsev wrote:
>>> ### Bugzilla
>>>
>>> It may be time to kill bugzilla:
>>>
>>>      - despite periodic "we're not dead yet" emails, it doesn't appear very
>>>        active
>>>      - the upgrade path to 6.0 is broken for us due to bugzilla abandoning the
>>>        5.2 development branch and continuing with 5.1
>>>      - question remains with what to replace bugzilla, but it's a longer
>>>        discussion topic that I don't want to raise here; it may be a job for
>>>        the bugspray bot that can extend the two-way bridge functionality to
>>>        multiple bug tracker frameworks
>>
>> The PCI subsystem relies heavily on bugzilla to track issues,
>> collect dmesg/lspci output from reporters and furnish them with
>> debug or test patches.
>>
>> The SOP when issues are reported on the mailing list without
>> sufficient information is to ask the reporter to open a bugzilla
>> issue and attach full dmesg and lspci -vvv output for analysis.
>>
>> If bugzilla is deprecated, we'll need at least a way to exchange
>> files with reporters.  Preferably on kernel.org infrastructure
>> to be independent from 3rd parties.  A way to communicate with
>> reporters outside the mailing list is also useful to prevent
>> spamming linux-pci@vger.kernel.org with messages relevant only
>> to a single issue or system.
>>
>> All the information now recorded in bugzilla should continue
>> to be available indefinitely so that Link: tags in commits
>> continue to work.  It's not uncommon to have to dig in old
>> bugzilla entries in order to understand the motivation for
>> a particular code section that was introduced years earlier.
> 
> At least some of the wireless maintainers also use bugzilla.
> The ath11k & ath12k drivers have guidance in the wireless wiki:
> https://wireless.docs.kernel.org/en/latest/en/users/drivers/ath11k/bugreport.html
> https://wireless.docs.kernel.org/en/latest/en/users/drivers/ath12k/bugreport.html
> 
> So we would also want this or a similar service to be maintained.
> 
> /jeff

I know that there was a mention of "external" Gitlab instances earlier 
in the thread.  How about standing up an LF Gitlab instance?

Subsystems that want to use it for issue tracking can have projects 
there specifically for that.

For example we could have a gitlab.kernel.org and then a project PCI for 
all PCI subsystem related issues.

This also "potentially" opens up the possibility of subsystems that want 
to engage in a forge PR/MR workflow with contributors to do so.

Re: kernel.org tooling update

[Jani Nikula]

On Tue, 16 Dec 2025, Mario Limonciello <superm1@kernel.org> wrote:
> On 12/16/25 2:33 PM, Jeff Johnson wrote:
>> On 12/16/2025 8:21 AM, Lukas Wunner wrote:
>>> [cc += Bjorn, start of thread is here:
>>> https://lore.kernel.org/ksummit/20251209-roaring-hidden-alligator-068eea@lemur/
>>> ]
>>>
>>> On Tue, Dec 09, 2025 at 11:48:24PM -0500, Konstantin Ryabitsev wrote:
>>>> ### Bugzilla
>>>>
>>>> It may be time to kill bugzilla:
>>>>
>>>>      - despite periodic "we're not dead yet" emails, it doesn't appear very
>>>>        active
>>>>      - the upgrade path to 6.0 is broken for us due to bugzilla abandoning the
>>>>        5.2 development branch and continuing with 5.1
>>>>      - question remains with what to replace bugzilla, but it's a longer
>>>>        discussion topic that I don't want to raise here; it may be a job for
>>>>        the bugspray bot that can extend the two-way bridge functionality to
>>>>        multiple bug tracker frameworks
>>>
>>> The PCI subsystem relies heavily on bugzilla to track issues,
>>> collect dmesg/lspci output from reporters and furnish them with
>>> debug or test patches.
>>>
>>> The SOP when issues are reported on the mailing list without
>>> sufficient information is to ask the reporter to open a bugzilla
>>> issue and attach full dmesg and lspci -vvv output for analysis.
>>>
>>> If bugzilla is deprecated, we'll need at least a way to exchange
>>> files with reporters.  Preferably on kernel.org infrastructure
>>> to be independent from 3rd parties.  A way to communicate with
>>> reporters outside the mailing list is also useful to prevent
>>> spamming linux-pci@vger.kernel.org with messages relevant only
>>> to a single issue or system.
>>>
>>> All the information now recorded in bugzilla should continue
>>> to be available indefinitely so that Link: tags in commits
>>> continue to work.  It's not uncommon to have to dig in old
>>> bugzilla entries in order to understand the motivation for
>>> a particular code section that was introduced years earlier.
>> 
>> At least some of the wireless maintainers also use bugzilla.
>> The ath11k & ath12k drivers have guidance in the wireless wiki:
>> https://wireless.docs.kernel.org/en/latest/en/users/drivers/ath11k/bugreport.html
>> https://wireless.docs.kernel.org/en/latest/en/users/drivers/ath12k/bugreport.html
>> 
>> So we would also want this or a similar service to be maintained.
>> 
>> /jeff
>
> I know that there was a mention of "external" Gitlab instances earlier 
> in the thread.  How about standing up an LF Gitlab instance?

FWIW, I've been rather discouraged about the free tier GitLab issues
experience. Feature wise, it's a step down from Bugzilla, even if the UI
is more modern. The best stuff is always going into the paid tier. For
this reason alone, I'm partial to something completely community driven
like Forgejo. There's at least the possibility of getting the new
features.


BR,
Jani.


>
> Subsystems that want to use it for issue tracking can have projects 
> there specifically for that.
>
> For example we could have a gitlab.kernel.org and then a project PCI for 
> all PCI subsystem related issues.
>
> This also "potentially" opens up the possibility of subsystems that want 
> to engage in a forge PR/MR workflow with contributors to do so.
>

-- 
Jani Nikula, Intel

Re: kernel.org tooling update

[Mario Limonciello]

On 12/18/25 7:37 AM, Jani Nikula wrote:
> On Tue, 16 Dec 2025, Mario Limonciello <superm1@kernel.org> wrote:
>> On 12/16/25 2:33 PM, Jeff Johnson wrote:
>>> On 12/16/2025 8:21 AM, Lukas Wunner wrote:
>>>> [cc += Bjorn, start of thread is here:
>>>> https://lore.kernel.org/ksummit/20251209-roaring-hidden-alligator-068eea@lemur/
>>>> ]
>>>>
>>>> On Tue, Dec 09, 2025 at 11:48:24PM -0500, Konstantin Ryabitsev wrote:
>>>>> ### Bugzilla
>>>>>
>>>>> It may be time to kill bugzilla:
>>>>>
>>>>>       - despite periodic "we're not dead yet" emails, it doesn't appear very
>>>>>         active
>>>>>       - the upgrade path to 6.0 is broken for us due to bugzilla abandoning the
>>>>>         5.2 development branch and continuing with 5.1
>>>>>       - question remains with what to replace bugzilla, but it's a longer
>>>>>         discussion topic that I don't want to raise here; it may be a job for
>>>>>         the bugspray bot that can extend the two-way bridge functionality to
>>>>>         multiple bug tracker frameworks
>>>>
>>>> The PCI subsystem relies heavily on bugzilla to track issues,
>>>> collect dmesg/lspci output from reporters and furnish them with
>>>> debug or test patches.
>>>>
>>>> The SOP when issues are reported on the mailing list without
>>>> sufficient information is to ask the reporter to open a bugzilla
>>>> issue and attach full dmesg and lspci -vvv output for analysis.
>>>>
>>>> If bugzilla is deprecated, we'll need at least a way to exchange
>>>> files with reporters.  Preferably on kernel.org infrastructure
>>>> to be independent from 3rd parties.  A way to communicate with
>>>> reporters outside the mailing list is also useful to prevent
>>>> spamming linux-pci@vger.kernel.org with messages relevant only
>>>> to a single issue or system.
>>>>
>>>> All the information now recorded in bugzilla should continue
>>>> to be available indefinitely so that Link: tags in commits
>>>> continue to work.  It's not uncommon to have to dig in old
>>>> bugzilla entries in order to understand the motivation for
>>>> a particular code section that was introduced years earlier.
>>>
>>> At least some of the wireless maintainers also use bugzilla.
>>> The ath11k & ath12k drivers have guidance in the wireless wiki:
>>> https://wireless.docs.kernel.org/en/latest/en/users/drivers/ath11k/bugreport.html
>>> https://wireless.docs.kernel.org/en/latest/en/users/drivers/ath12k/bugreport.html
>>>
>>> So we would also want this or a similar service to be maintained.
>>>
>>> /jeff
>>
>> I know that there was a mention of "external" Gitlab instances earlier
>> in the thread.  How about standing up an LF Gitlab instance?
> 
> FWIW, I've been rather discouraged about the free tier GitLab issues
> experience. Feature wise, it's a step down from Bugzilla, even if the UI
> is more modern. The best stuff is always going into the paid tier. For
> this reason alone, I'm partial to something completely community driven
> like Forgejo. There's at least the possibility of getting the new
> features.

Sure - totally.

> 
> 
> BR,
> Jani.
> 
> 
>>
>> Subsystems that want to use it for issue tracking can have projects
>> there specifically for that.
>>
>> For example we could have a gitlab.kernel.org and then a project PCI for
>> all PCI subsystem related issues.
>>
>> This also "potentially" opens up the possibility of subsystems that want
>> to engage in a forge PR/MR workflow with contributors to do so.
>>
>