Re: Toy/demo: using ChatGPT to summarize lengthy LKML threads (b4 integration)

[Luis Chamberlain <mcgrof@kernel.org>]

On Tue, Feb 27, 2024 at 05:32:34PM -0500, Konstantin Ryabitsev wrote:
> Hi, all:
> 
> I was playing with shell-gpt and wrote a quickie integration that would allow
> retrieving (slimmed-down) threads from lore, feeding them to ChatGPT, and
> asking it to provide some basic analysis of the thread contents. Here's a
> recorded demo session:
> 
> https://asciinema.org/a/643435
> 
> A few notes:
> 
> 1. This is obviously not a replacement for actually reading email, but can
>    potentially be a useful asset for a busy maintainer who just wants a quick
>    summary of a lengthy thread before they look at it in detail.
> 2. This is not free or cheap! To digest a lengthy thread, you can expect
>    ChatGPT to generate enough tokens to cost you $1 or more in API usage fees.
>    I know it's nothing compared to how expensive some of y'all's time is, and
>    you can probably easily get that expensed by your employers, but for many
>    others it's a pretty expensive toy. I managed to make it a bit cheaper by
>    doing some surgery on the threads before feeding them to chatgpt (like
>    removing most of the message headers and throwing out some of the quoted
>    content), but there's a limit to how much we can throw out before the
>    analysis becomes dramatically less useful.
> 3. This only works with ChatGPT-4, as most threads are too long for
>    ChatGPT-3.5 to even process.
> 
> So, the question is -- is this useful at all? Am I wasting time poking in this
> direction, or is this something that would be of benefit to any of you? If the
> latter, I will document how to set this up and commit the thread minimization
> code I hacked together to make it cheaper.

While I probably wouldn't use it day to day, I expect younger
generations might use this more than us older generations to be more
productive, even if they get halluciations.

An LLM trained with more data relevant to patches might be more
suitable, and it is why I wanted the tooling for stable candidate patches
to be opened up, so to enable more exploring in areas like this.

A use case example might be training for identifying subsystems with
more memory safety issues.

Another might be to help to summarize further pull requests in one or
two sentences, or optionally few bullets. So for instance, I try to
document major bullet list changes for modules here:

https://kernelnewbies.org/KernelProjects/modules

So it is easier to track / go down memory lane. Doing this automatically
would allow me to use a tool to do this.

  Luis