On Tue, Aug 05, 2025 at 07:01:01PM +0100, Lorenzo Stoakes wrote:

> However the point is that we can:

> - Generate scenarios relatively quickly that way.
> - Resulatant mistakes are at least _less_ impactful. And you'd hope, would come
>   out in the wash more easily.

> However, I wonder whether it might be better simply to restrict LLMs to test
> data generation or the like.

Yeah, test data seems much safer than test code since it's much more
constrained.  I'm definitely not 100% against trying things with code,
I'm more worried about people throwing something out there without
really looking at it properly since it's just test code.  With the
scenario generation for example the question is always if we'd be
happier with the test program being a scenario generator - sometimes the
answer is no (eg, because it's too expensive to run and you really need
to pick scenarios), sometimes the answer is yes even though the
resulting test program is more involved.

> These are again all exactly the kinds of things that make this an important
> topic to discuss I believe :)

Indeed, and I'm going to guess that it's going to be as much about
establishing taste as firm rules.