Re: Toy/demo: using ChatGPT to summarize lengthy LKML threads (b4 integration)

[Hannes Reinecke <hare@suse.de>]

On 2/28/24 19:55, Bart Van Assche wrote:
> On 2/27/24 14:32, Konstantin Ryabitsev wrote:
>> I was playing with shell-gpt and wrote a quickie integration that 
>> would allow
>> retrieving (slimmed-down) threads from lore, feeding them to ChatGPT, and
>> asking it to provide some basic analysis of the thread contents. Here's a
>> recorded demo session:
>>
>> https://asciinema.org/a/643435
>>
>> A few notes:
>>
>> 1. This is obviously not a replacement for actually reading email, but 
>> can
>>     potentially be a useful asset for a busy maintainer who just wants 
>> a quick
>>     summary of a lengthy thread before they look at it in detail.
>> 2. This is not free or cheap! To digest a lengthy thread, you can expect
>>     ChatGPT to generate enough tokens to cost you $1 or more in API 
>> usage fees.
>>     I know it's nothing compared to how expensive some of y'all's time 
>> is, and
>>     you can probably easily get that expensed by your employers, but 
>> for many
>>     others it's a pretty expensive toy. I managed to make it a bit 
>> cheaper by
>>     doing some surgery on the threads before feeding them to chatgpt 
>> (like
>>     removing most of the message headers and throwing out some of the 
>> quoted
>>     content), but there's a limit to how much we can throw out before the
>>     analysis becomes dramatically less useful.
>> 3. This only works with ChatGPT-4, as most threads are too long for
>>     ChatGPT-3.5 to even process.
>>
>> So, the question is -- is this useful at all? Am I wasting time poking 
>> in this
>> direction, or is this something that would be of benefit to any of 
>> you? If the
>> latter, I will document how to set this up and commit the thread 
>> minimization
>> code I hacked together to make it cheaper.
> 
> Please do not publish the summaries generated by ChatGPT on the web. If
> these summaries would be published on the world wide web, ChatGPT or
> other LLMs probably would use these summaries as input data. If there
> would be any mistakes in these summaries, then these mistakes would end
> up being used as input data by multiple LLMs.
> 
Now there's a thought. Maybe we should do exactly the opposite, and 
posting _more_ ChatGPT generated content on the web?
Sending them into a deadly self-enforcing feedback loop?

But that's probably beside the point.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich