Re: [MAINTAINERS SUMMIT] The role of AI and LLMs in the kernel process

[Lorenzo Stoakes <lorenzo.stoakes@oracle.com>]

+cc Steven

Apologies, I meant to +cc you here as I mention an issue you encountered,
and forgot of course :>)

Cheers, Lorenzo

On Tue, Aug 05, 2025 at 06:55:29PM +0100, Lorenzo Stoakes wrote:
> On Tue, Aug 05, 2025 at 12:43:38PM -0400, James Bottomley wrote:
> > On Tue, 2025-08-05 at 17:03 +0100, Lorenzo Stoakes wrote:
> > > Unavoidably, LLMs are the hot topic in tech right now, and are here
> > > to stay.
> > >
> > > This poses unique problems:
> > >
> > > * Never before have people been able to generate as much content that
> > > may, on a surface reading, seem valid whilst in reality being quite
> > > the opposite.
> > >
> > > * Equally, LLM's can introduce very subtle mistakes that humans find
> > > difficult to pick up upon - humans implicitly assume that the classes
> > > of errors they will encounter are the kinds other humans would make -
> > > AI defeats that instinct.
> >
> > Do you have any examples of this?  I've found the opposite to be true:
>
> Sure - Steven encountered this in [1].
>
> As he says there:
>
> "If I had known, I would have examined the patch a little more thoroughly,
>  and would have discovered a very minor mistake in the patch."
>
> The algorithm is determining likely output based on statistics, and
> therefore density of input. Since in reality one can write infinite
> programs, it's mathematically inevitable that an LLM will have to 'infer'
> answers.
>
> That inference has no basis in dynamics, that is a model of reality that it
> can use to determine answers, rather it will, in essence, provide a random
> result.
>
> If there is a great deal of input (e.g. C programs), then that inference is
> likely to manifest in very subtle errors. See [2] for a thoughtful
> exploration from an AI expert on the topic of statistics vs. dynamics, and
> [3] for a broader exploration of the topic from the same author.
>
> [1]:https://lore.kernel.org/workflows/20250724194556.105803db@gandalf.local.home/
> [2]:https://blog.piekniewski.info/2016/11/01/statistics-and-dynamics/
> [3]:https://blog.piekniewski.info/2023/04/09/ai-reflections/
>
>
> > AI is capable of really big stupid mistakes when it hasn't seen enough
> > of the pattern, but I can't recall seeing it make something you'd
> > classify as a subtle mistake (I assume it could copy subtle mistakes
> > from wrong training data, so I'm not saying it can't, just that I
> > haven't seen any).
>
> It's not from incorrect training data, it's fundamental to how LLMs
> work.
>
> >
> > I think the big mistakes could possibly be avoided by asking people who
> > submit patches to also append the AI confidence score:
> >
> > https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/concept/accuracy-confidence?view=doc-intel-4.0.0
>
> That's interesting, though I don't know how reliable this might be.
>
> However it's for exactly this kind of input that I proposed the topic :)
>
> >
> > So we know how much similar training the model has seen before coming
> > to any conclusion about the value of the output.
> >
> > > * The kernel is uniquely sensitive to erroneous (especially subtly
> > > erroneous) code - even small errors can be highly consequential. We
> > > use a programming language that can almost be defined by its lack of
> > > any kind   of safety, and in some subsystems patches are simply taken
> > > if no obvious problems exist, making us rather vulnerable to this.
> >
> > I think that's really overlooking the fact that if properly trained (a
> > somewhat big *if* depending on the model) AI should be very good at
> > writing safe code in unsafe languages.  However it takes C specific
>
> I fundamentally disagree.
>
> The consequences of even extremely small mistakes can be very serious in C,
> as the language does little to nothing for you.
>
> No matter how much data it absorbs it cannot span the entire space of all
> possible programs or even anywhere close.
>
> I mean again, I apply the arguments above as to why I feel this is
> _fundamental_ to the approach.
>
> Kernel code is also very specific and has characteristics that render it
> different from userland. We must consider a great many more things that
> would be handled for us were we userland - interrupts, the context we are
> in, locks of all varieties, etc. etc.
>
> While there's a lot of kernel code (~10's of millions of line), for an LLM
> that is very small, and we simply cannot generate more.
>
> Yes it can eat up all the C it can, but that isn't quite the same.
>
> > training to do this, so any LLM that's absorbed a load of rust, python
> > and javascript from the internet will be correspondingly bad at writing
> > safe C code.  Hence the origin of the LLM and its training corpus would
> > be a key factor in deciding to trust it.
> >
> > > * On the other hand, there are use cases which are useful - test
> > > data/code generation, summarisation, smart auto-complete - so it'd
> > > perhaps be foolish to entirely dismiss AI.
> >
> > Patch backporting is another such nice use.
>
> As long as carefully checked :)
>
> >
> > > A very important non-technical point we must consider is that, the
> > > second we even appear to be open to AI submission of _any_ kind, the
> > > press will inevitably report on it gleefully, likely with
> > > oversimplified headlines like 'Linux accepts AI patches'.
> >
> > Oh, I think simply accepting AI patches is old news:
> >
> > https://www.cnbc.com/2025/04/29/satya-nadella-says-as-much-as-30percent-of-microsoft-code-is-written-by-ai.html
>
> That doesn't pertain to the kernel specifically.
>
> Of course code being written by AI is old news, but there's no doubt that
> tech publications would JUMP on anything even suggesting we are open in
> some broad way to AI submissions.
>
> Given Linus's rather neutral public position on AI, it'd certainly mark
> what _would be perceived_, in my view, as a sea change on this.
>
> >
> > > The moment that happens, we are likely to see a significant uptick in
> > > AI submissions whether we like it or not.
> > >
> > > I propose that we establish the broad rules as they pertain to the
> > > kernel, and would like to bring the discussion to the Maintainer's
> > > Summit so we can determine what those should be.
> > >
> > > It's important to get a sense of how maintainers feel about this -
> > > whether what is proposed is opt-in or opt-out - and how we actually
> > > implement this.
> > >
> > > There has been discussion on-list about this (see [0]), with many
> > > suggestions made including a 'traffic light' system per-subsystem,
> > > however many open questions remain - the devil is in the details.
> > >
> > > [0]:
> > > https://lore.kernel.org/all/20250727195802.2222764-1-sashal@kernel.or
> > > g/
> >
> > We're already getting AI generated bug reports from what I can tell.
> > It would be really helpful to see the AI confidence score for them as
> > well.
>
> That is definitely an interesting additional data point that could
> potentially be helpful here! I wasn't aware of this so thanks for that :)
>
> >
> > Regards,
> >
> > James
> >
> >
>
> Cheers, Lorenzo