Re: slowly decommission bugzilla?

[Sasha Levin <sashal@kernel.org>]

On Fri, Mar 06, 2026 at 04:01:24PM +0100, Laurent Pinchart wrote:
>I seem to have failed to convey my opinion properly, I'll try better.

Thanks, I appreciate that. Let me try too.

>> >> can you propose a different
>> >> deterministic solution that would have worked for
>> >> https://lore.kernel.org/all/DGRCO9SL0T5U.JTINSHJQ9KPK@imlonghao.com/ ?
>> >
>> > Not as a procedure to extract line numbers that will work retroactively
>> > of course, but I don't think that's the point.
>> >
>> > As we're discussing new developments to replace bugzilla, working with
>> > distributions to streamline bug reporting would be more deterministic.
>> > Running the stack trace decode in the server side is a good idea, and
>> > the server should receive in addition to the stack trace either the
>> > debuginfo or, when running a distro kernel, an identifier of the exact
>> > kernel version to download the debuginfo from an authoritative source.
>>
>> Look at the bug reports we discussed in the thread: one is a custom built
>> kernel, and one is a kernel without available debug symbols.
>
>Note that for custom-built kernels I think it's reasonable to ask the
>submitter for debuginfo (probably in the form of a decoded stack trace).
>If someone is able to build a kernel manually, I expect them to be able
>to run a stack trace decoding tool.
>
>> Even if we had the
>> best relationships with all our distro friends, had a central DB with all debug
>> symbols, we would still not able to tackle this.
>
>Sure, nothing we do will be able to increase our bug report handling to
>100% success. I even agree that there are cases where heuristics could
>be helpful. This being said, I haven't checked personally, but I would
>be very surprised if the majority of the bug reports were related to
>kernels for which debuginfo can't be accessed one way or another.
>
>What concerned me in my initial reply, and still concerns me, is
>applying a non-deterministic method to the problem when the vast
>majority of cases can be solved deterministically. It would mean we
>would consume more resources to get a suboptimal results, which is
>guaranteed to sometimes produce wrong but plausible-looking results.
>It's becoming an increasingly common pattern to throw LLMs as a magic
>solution to all problems, despite all the drawbacks, and I'm really
>getting tired of that.

I don't think this is an either/or thing though. We can do both - have
deterministic tooling *and* use LLMs where they help. They solve
different parts of the problem.

If I'm arguing for using an LLM as a solution for something, it doesn't mean
that I object to any other approach.

>My initial thought, before seeing your proposal, was to handle the
>majority case with a DB of debuginfo. Yes, it will require coordination
>with distributions, and some development, but what is being discussed in
>this mail thread is development of a new service anyway. Maybe I fail to
>see where this would be difficult or require more development effort
>than we can afford ?
>
>I now see you've made a proposal in this thread to embed file and line
>info in kallsyms. It seems better than what I had in mind, I will review
>it.

Thanks, I'd appreciate that.

But I also don't think either kallsyms or debuginfo DB can do this alone.
Decoding the stack trace is just step one - someone still has to figure out
what actually went wrong. The LLM approach I've been hacking on goes a lot
further than just running decode_stacktrace.sh: it reads the relevant source,
digs through git history to find the commit that likely broke things, and
searches lore for prior reports and related discussions.

Take the CC list as a concrete example. get_maintainer.pl gives you the
subsystem maintainers for the affected files, which is fine as a starting
point. But what you really want for a bug is to reach the person who wrote the
offending commit, or the developer who posted a related fix that never got
merged, or someone who reported a similar crash six months ago on a different
list. The bot finds those people by searching lore and git history and CCs
them. You can't really do that with a deterministic tool - it needs to
understand what's actually relevant in context.

>Let's see if you proved yourself wrong with the patch you posted in the
>mail thread :-)

I really really am not trying to religiously argue for or against LLMs: we can
keep throwing ideas at the problem and see what sticks. Tools that work well
will live on, the ones that fail will disappear.

There's no one way door decision we're making by giving LLMs a spin. If they
suck, we just drop that tool.

-- 
Thanks,
Sasha