Re: slowly decommission bugzilla?

[Laurent Pinchart <laurent.pinchart@ideasonboard.com>]

Hi Sasha,

On Sun, Mar 01, 2026 at 11:27:02AM -0500, Sasha Levin wrote:
> On Sun, Mar 01, 2026 at 05:13:17PM +0100, Laurent Pinchart wrote:
> > On Sun, Mar 01, 2026 at 10:42:57AM -0500, Sasha Levin wrote:
> >> On Sun, Mar 01, 2026 at 04:35:26PM +0100, Laurent Pinchart wrote:
> >> > On Sun, Mar 01, 2026 at 10:23:00AM -0500, Sasha Levin wrote:
> >> >> On Sat, Feb 28, 2026 at 03:56:11PM -0500, Steven Rostedt wrote:
> >> >> > On Sat, 28 Feb 2026 21:28:57 +0100 (CET) Richard Weinberger wrote:
> >> >> >
> >> >> >> Wouldn't that only work if the report is able to upload the kernel debug
> >> >> >> info too?
> >> >> >
> >> >> > Yes, this would be nice if we had the help from the distros that could
> >> >> > automate this process.
> >> >>
> >> >> So I've been poking at using LLMs for this.
> >> >>
> >> >> decode_stacktrace.sh is great when you have a debug build handy, but
> >> >> asking a random bug reporter to obtain debuginfo, set up the tooling,
> >> >> and run the script is quite the hurdle.
> >> >>
> >> >> The debuginfo problem is solvable on the server side though. Given a
> >> >> kernel version string from the oops, an LLM can figure out which distro
> >> >> and package version the reporter is running, pull the right debuginfo
> >> >> (or build from the matching config/tag if no debuginfo package exists),
> >> >> and run decode_stacktrace.sh itself.
> >> >
> >> > Do we really have to use non-deterministic tools that will inevitably
> >> > produce correct-looking but otherwise useless backtraces from time to
> >> > time, confusing developers and wasting time for everybody, when we can
> >> > instead easily develop tools that will work in a deterministic fashion ?
> >> > I'm getting *really* sick of people pushing for LLM usage when it's
> >> > clearly the wrong tool. Please stop.
> >>
> >> The LLM actually runs scripts/decode_stacktrace.sh. The non-deterministic part
> >> is being able to get our hands on various debug symbols from
> >> distros/vendors/etc.
> >
> > That was my understanding, yes.
> >
> >> Instead of just crapping on everything,
> >
> > Please tone the insults down.
> 
> Happily. Please stop telling me to shut up without offering an alternative :)

I seem to have failed to convey my opinion properly, I'll try better.

> >> can you propose a different
> >> deterministic solution that would have worked for
> >> https://lore.kernel.org/all/DGRCO9SL0T5U.JTINSHJQ9KPK@imlonghao.com/ ?
> >
> > Not as a procedure to extract line numbers that will work retroactively
> > of course, but I don't think that's the point.
> >
> > As we're discussing new developments to replace bugzilla, working with
> > distributions to streamline bug reporting would be more deterministic.
> > Running the stack trace decode in the server side is a good idea, and
> > the server should receive in addition to the stack trace either the
> > debuginfo or, when running a distro kernel, an identifier of the exact
> > kernel version to download the debuginfo from an authoritative source.
> 
> Look at the bug reports we discussed in the thread: one is a custom built
> kernel, and one is a kernel without available debug symbols.

Note that for custom-built kernels I think it's reasonable to ask the
submitter for debuginfo (probably in the form of a decoded stack trace).
If someone is able to build a kernel manually, I expect them to be able
to run a stack trace decoding tool.

> Even if we had the
> best relationships with all our distro friends, had a central DB with all debug
> symbols, we would still not able to tackle this.

Sure, nothing we do will be able to increase our bug report handling to
100% success. I even agree that there are cases where heuristics could
be helpful. This being said, I haven't checked personally, but I would
be very surprised if the majority of the bug reports were related to
kernels for which debuginfo can't be accessed one way or another.

What concerned me in my initial reply, and still concerns me, is
applying a non-deterministic method to the problem when the vast
majority of cases can be solved deterministically. It would mean we
would consume more resources to get a suboptimal results, which is
guaranteed to sometimes produce wrong but plausible-looking results.
It's becoming an increasingly common pattern to throw LLMs as a magic
solution to all problems, despite all the drawbacks, and I'm really
getting tired of that.

My initial thought, before seeing your proposal, was to handle the
majority case with a DB of debuginfo. Yes, it will require coordination
with distributions, and some development, but what is being discussed in
this mail thread is development of a new service anyway. Maybe I fail to
see where this would be difficult or require more development effort
than we can afford ?

I now see you've made a proposal in this thread to embed file and line
info in kallsyms. It seems better than what I had in mind, I will review
it.

> >> Or should we just let that bug report rot as it usually happens because you have
> >> some dislike for LLMs?
> >
> > Let's not resort to straw man arguments, you know that's not what I
> > said.
> 
> I know it's not your intent, but this is what happens in practice. Look at how
> many bug reports show up on our mailing lists and are either just ignored, or
> end up with a response or two that makes no progress.

I honestly don't know how much line numbers in stack traces would help
there. I feel the issue is more related to lack of maintainer bandwidth.

> > I will stand by my original claim, that using the community's
> > development resources to create non-deterministic tools when a
> > deterministic solution isn't more difficult is a waste of time and
> > resources. I've always disliked suboptimal solutions, it's not limited
> > to LLMs.
> 
> I don't disagree that a deterministic solution would be better. My argument is
> that I don't think we can have one.

Let's see if you proved yourself wrong with the patch you posted in the
mail thread :-)

> Again, feel free to prove me wrong, just don't tell me to shut up without
> offering a better alternative.

There are at least two ideas now (debuginfo DB and file:line in
kallsyms). Let's see where those leads.

-- 
Regards,

Laurent Pinchart