On Fri, Sep 07, 2018 at 11:13:20AM +0200, Daniel Vetter wrote:

> Of course this is much harder for anything that needs physical
> hardware, but even for driver subsystems there's lots you can do with
> test-drivers, selftests and a pile of emulation, to at least catch
> bugs in generic code. And for reasonably sized teams like drm/i915
> building a proper CI is a very obvious investement that will pay off.

It does depend what the primary focus of the QA people is of course -
one of the challenges that comes from people shipping stable in products
is that this tends to be where all the QA investment from vendors goes.
Vendors who are taking a longer term view of their investment in
upstream (and especially those who realistically still have to ship an
out of tree patch stack on top of whatever's upstream) often work on the
basis that they'll figure things out when they pick up the release for
production.  It's a cost, they know it's a cost but there's also costs
in having QA tracking upstream.

> > But other trees seem to be much more loosey-goosey about what they
> > will push to linux-next, since they want to let the 'bots catch
> > problems.  With the net result that they scare users away from wanting
> > to use linux-next.

> Yeah, if maintainers see linux-next as their personal testing ground
> then it becomes useless for actual integration testing. But it's not
> quite as bleak as it was 2 years ago I think, at least from what I'm
> seeing when in our linux-next runs for drm/i915. It still does need to
> get a lot better though.

My experience has been a lot better here working on embedded - yes,
things do occasionally collapse horribly but for the most part it's a
reasonably solid basis to work from for as long as I can remember and
when things do break they normally get fixed fast enough.  Things were a
bit worse before KernelCI and Olof's boot farm, over time (and with the
breakage the DT transition introduced winding down) those have helped
quite a bit.  I think some of that's down to differences in the
underlying hardware with more being directly visible it's easier to unit
test, don't know if there's anything else going on.