Re: kernel.org tooling update

[Randy Dunlap <rdunlap@infradead.org>]

Hi,

On 12/9/25 8:48 PM, Konstantin Ryabitsev wrote:
> Hi, all:
> 
> These are the topics that were touched on at the maintainer summit when
> discussing tooling on the kernel.org side of things.
> 
> --
> 
> # What is the state of tooling?
> 
> ## b4 development update
> 
> Past year:
> 
> - No major new features in b4 over the past year, excepting `b4 dig`
> - Seeing lots of adoption and use across subsystem, with a lot of maintainers
>   recommending b4 as the preferred mechanism to submit patches
> - Starting to see adoption by several non-kernel projects (openembedded, u-boot, others)
> - Significant behind-the-scenes move of codebase to stricter typed code
> - Continued work on `b4 review` that got shelved temporarily for other
>   priorities.
> 
> ### LETS PUT MOAR AI INTO IT!!1
> 
> I spent a lot of time on trying to integrate AI into b4 workflows, but with
> little to show for it in the end due to lackluster results.
> 
> - Used local ollama as opposed to proprietary services, with the goal to avoid
>   introducing hard dependencies on third-party commercial tooling. This is
>   probably the main reason why my results were not so exciting as what others
>   see with much more powerful models.
> 
> - Focused on thread/series summarization features as opposed to code analysis:
> 
>     - Summarize follow-ups (trailers, acks/nacks received), though this is
>       already fairly well-handed with non-AI tooling.
> 
>     - Gauge "temperature" of the discussion to highlight controversial series.
> 
>     - Gauge quality of the submission; help decide "is this series worth
>       looking at" before maintainers spend their effort looking at it, using
>       maintainer-tailored prompts. This may be better done via CI/patchwork
>       integration, than with b4.
> 
>     - Use LLM to prepare a merge commit message using the cover letter and
>       summarizing the patches.
> 
> I did not end up releasing any features based on that work, because:
> 
>     - LLM was not fantastic at following discussions and keeping a clear
>       picture of who said what, which is kind of crucial for maintainer
>       decision making.
> 
>     - Very large series and huge threads run out fo context window, which
>       causes the LLM to get even worse at "who said what" (and it's
>       already not that great at it).
> 
>     - Thread analysis requires lots of VRAM and a modern graphics card, and is
>       still fairly slow there (I used a fairly powerful GeForce RTX).
> 
>     - Actual code review is best if it happens post-apply in a temporary
>       workdir or a temporary branch, so the agent can see the change in the
>       context of the git tree and the entire codebase, not just the context
>       lines of the patch itself.
> 
> I did have much better success when I worked to represent a thread not as
> multiple messages, but as a single document with all interleaved follow-up
> conversations collated together. However, this was done manually --
> representing emails from arbitrary threads as such collated documents is a
> separate challenge.
> 
> Using proprietary models and remote services will probably show better
> results, but I did not have the funds or the inkling to do it (plus see the
> concern for third-party commercial tooling). I may need to collaborate more
> closely with the maintainers already doing it on their own instead of
> continuing my separate work on it.
> 
> ### AI crawler scourge
> 
> While working on LLM integration, it was certainly ironic that one of the
> top challenges for us was to try to keep AI crawlers from overwhelming
> kernel.org infrastructure. While we've put several mitigations in place, it's
> a temporary relief at best.
> 
> ## Continuous degradation of SMTP
> 
> We're increasingly having to deal with the degradation of the SMTP support by
> all commercial companies:
> 
>     - major hosts are increasingly not interested in getting mail from anyone
>       who isn't also a major mail service provider
> 
>     - their "bulk sender" guidelines are no good for us (e.g. requiring that
>       we add one-click unsubscribe footers to all email)
> 
>     - their "spam filters" are increasingly based on training data, which
>       means that "looks different from what most of our users receive" is
>       enough to have patches and code discussions put into the "Junk" folder
> 
>     - they apply arbitrary throttling ("too many deliveries for the same
>       message-id", "too many messages from the DKIM domain foobar.com")
> 
>     - anti-phishing services at commercial IT companies do horrible things to
>       incoming messages
> 
> ## Are we finally moving away from patches sent over email?
> 
> There are still important concerns when we consider moving away from "patches
> sent via email":
> 
>     - SMTP is still the only widely used protocol we have for decentralized
>       communication; everything else is experimental or has important
>       drawbacks, such as:
> 
>         - it relies on single-point-of-failure services (e.g. Signal), or
>         - it requires standing up esoteric software (which then become
>           single-point-of-failure services), or
>         - it requires an "everyone-must-switch-now" flag day
> 
>     - RFC-5322, with all its warts, is a well-defined standard for internet
>       messages:
> 
>         - robust, capable of dealing with change while preserving legacy
>         - easy to parse with libraries for almost any framework
>         - easy to archive and query
>         - has lots of tooling built around it
> 
> With lore and public-inbox, we *are* in the process of moving away from
> relying on the increasingly unreliable SMTP layer. Lore can already let you do
> the following things:
> 
>     - lets anyone submit patches via the web endpoint
>     - lets anyone subscribe to lists via several protocols (NNTP, POP, IMAP)
>     - lets anyone use lei to receive arbitrary feeds
>     - can aggregate any number of sources, as long as they are RFC-5322
>       messages (or can be converted to them)
> 
> Lore and public-inbox is becoming a kind of a distributed, replicating
> messaging bus with a robust query and retrieval interface on top of it, and I
> believe it's a fairly powerful framework we can build upon.
> 
> ## Work on "local lore"
> 
> One downside of lore.kernel.org is that it's a central service, which runs
> counter to our goal of limiting how many single points of failure we have.
> There is fault-tolerance built into the system (lore.kernel.org is actually 4
> different nodes in various parts of the world), but an adversary would have no
> difficulty knocking out all nodes at once, which would impact the project
> significantly.
> 
> The "local lore" projects it the attempt to provide a kind of "maintainer
> container" that can be run locally or in any public cloud:
> 
>     - comes with a 6-month constantly-updating mirror of lore, using a
>       failover set of replication URLs (including tor/onion)
>     - comes with a pre-configured mirror of git repositories that are kept
>       up-to-date in the same fashion
>     - lets the maintainer set up lei queries that can push into their
>       inbox, supporting Gmail+OAuth, JMAP, IMAP
>     - provides a web submission endpoint and an SMTP service that can
>       integrate with other SMTP relays
>     - publishes a public-inbox feed of maintainer activity that central
>       lore can pick up and integrate
> 
> There is a parallel goal here, which is to make it easier for devs to assume
> maintainer duties without having to spend a week setting up their tooling.
> In theory, all they would be need to do is to set up their maintainer
> container and then use the web menu to choose which feeds they want to pull
> and where they want messages delivered.
> 
> This project is still early in development, but I hope to be able to provide
> test containers soon that people can set up and run.
> 
> ## Other tools
> 
> ### Bugzilla
> 
> It may be time to kill bugzilla:
> 
>     - despite periodic "we're not dead yet" emails, it doesn't appear very
>       active
>     - the upgrade path to 6.0 is broken for us due to bugzilla abandoning the
>       5.2 development branch and continuing with 5.1
>     - question remains with what to replace bugzilla, but it's a longer
>       discussion topic that I don't want to raise here; it may be a job for
>       the bugspray bot that can extend the two-way bridge functionality to
>       multiple bug tracker frameworks
> 
> ### Patchwork
> 
> Patchwork continues to be used widely:
> 
>     - we've introduced query-based patchworks, where instead of consuming the
>       entire mailing list, we feed it the results of lei queries
>     - I'm hoping to work with upstream to add a couple of features that would
>       be of benefit to us, such as:
> 
>         - support for annotating patches and series (e.g. with LLM summaries)
>         - an API endpoint to submit patches, so maintainers could add
>           arbitrary series to their patchwork project, integrating with b4
> 
> ## Web of Trust work
> 
> There is an ongoing work to replace our home-grown web of trust solution (that
> does work but has important bottlenecks and scaling limitations) with
> something both more distributed and easier to maintain. We're working with
> OpenSSF to design the framework and I hope to present it to the community in
> the next few months.
> 
> ## Questions?
> 
> Send away!

Was any of this discussed at the December summits?
Were there any decisions or conclusions?
Are they summarized and shared somewhere?

Thanks.
-- 
~Randy