Re: [MAINTAINERS SUMMIT] The role of AI and LLMs in the kernel process

[James Bottomley <James.Bottomley@HansenPartnership.com>]

On Tue, 2025-08-05 at 18:55 +0100, Lorenzo Stoakes wrote:
> On Tue, Aug 05, 2025 at 12:43:38PM -0400, James Bottomley wrote:
> > On Tue, 2025-08-05 at 17:03 +0100, Lorenzo Stoakes wrote:
> > > Unavoidably, LLMs are the hot topic in tech right now, and are
> > > here to stay.
> > > 
> > > This poses unique problems:
> > > 
> > > * Never before have people been able to generate as much content
> > > that may, on a surface reading, seem valid whilst in reality
> > > being quite the opposite.
> > > 
> > > * Equally, LLM's can introduce very subtle mistakes that humans
> > > find difficult to pick up upon - humans implicitly assume that
> > > the classes of errors they will encounter are the kinds other
> > > humans would make - AI defeats that instinct.
> > 
> > Do you have any examples of this?  I've found the opposite to be
> > true:
> 
> Sure - Steven encountered this in [1].
> 
> As he says there:
> 
> "If I had known, I would have examined the patch a little more
> thoroughly,  and would have discovered a very minor mistake in the
> patch."

Heh, well now you make me look it seems that the minor mistake is
adding at tail instead of head?  That seems to be because the hash list
API doesn't have a head add ...

I wouldn't really call that a subtle problem because the LLM would have
picked up the head to tail conversion if we'd had an at head API for it
to learn from.

> The algorithm is determining likely output based on statistics, and
> therefore density of input. Since in reality one can write infinite
> programs, it's mathematically inevitable that an LLM will have to
> 'infer' answers.
> 
> That inference has no basis in dynamics, that is a model of reality
> that it can use to determine answers, rather it will, in essence,
> provide a random result.
> 
> If there is a great deal of input (e.g. C programs), then that
> inference is
> likely to manifest in very subtle errors. See [2] for a thoughtful
> exploration from an AI expert on the topic of statistics vs.
> dynamics, and [3] for a broader exploration of the topic from the
> same author.

Amazingly enough when you're trying to sell a new thing, you become
very down on what you see as the old thing (bcachefs vs btrfs ...?)

> 
> [1]:
> https://lore.kernel.org/workflows/20250724194556.105803db@gandalf.loc
> al.home/
> [2]:https://blog.piekniewski.info/2016/11/01/statistics-and-dynamics/
> [3]:https://blog.piekniewski.info/2023/04/09/ai-reflections/
> 
[...]
> > > * The kernel is uniquely sensitive to erroneous (especially
> > > subtly erroneous) code - even small errors can be highly
> > > consequential. We use a programming language that can almost be
> > > defined by its lack of any kind   of safety, and in some
> > > subsystems patches are simply taken if no obvious problems exist,
> > > making us rather vulnerable to this.
> > 
> > I think that's really overlooking the fact that if properly trained
> > (a somewhat big *if* depending on the model) AI should be very good
> > at writing safe code in unsafe languages.  However it takes C
> > specific
> 
> I fundamentally disagree.
> 
> The consequences of even extremely small mistakes can be very serious
> in C, as the language does little to nothing for you.
> 
> No matter how much data it absorbs it cannot span the entire space of
> all possible programs or even anywhere close.

Neither can a human and we get by on mostly pattern matching ourselves
...

> I mean again, I apply the arguments above as to why I feel this is
> _fundamental_ to the approach.
> 
> Kernel code is also very specific and has characteristics that render
> it different from userland. We must consider a great many more things
> that would be handled for us were we userland - interrupts, the
> context we are in, locks of all varieties, etc. etc.
> 
> While there's a lot of kernel code (~10's of millions of line), for
> an LLM that is very small, and we simply cannot generate more.
> 
> Yes it can eat up all the C it can, but that isn't quite the same.

You seem to be assuming training is simply dump the data corpus and let
the model fend for itself.  It isn't it's a more painstaking process
that finds the mistakes in the output and gets the model to improve
itself ... it is more like human teaching.

I'm not saying current AI is perfect, but I am saying that most of the
issues with current AI can be traced to training problems which can be
corrected in the model if anyone cares enough to do it.  The useful
signal is that in all badly trained models I've seen the AI confidence
score is really low because of the multiple matches in different areas
that proper training would separate.  THat's why I think AI confidence
score should be the first thing we ask for.

Regards,

James