On Fri, Jul 29, 2016 at 11:20:19AM -0400, Steven Rostedt wrote:
> Mark Brown <broonie@kernel.org> wrote:

> > There's also the risk that we make it harder for a random user to pick
> > up the tests and predict what the expected results should be - one of
> > the things that can really hurt a testsuite is if users don't find it
> > consistent and stable.

> I believe the ideal solution would be that a test would check if the
> hardware it wants to test is available or not. If it is not, it simple
> returns "Unsupported", and not success or failure.

> The kselftests should be run by anyone. If the user doesn't have a
> kernel with the right configs, or the right hardware for the test, the
> test should exit politely, saying that it could not run due to not
> having the proper environment. But if the configs and HW are available,
> do you envision having any other type of inconsistent result?

Right, that's one good strategy - but that's still unpredictable for the
user and there's a reasonable class of bugs that don't get flagged up
when breakage causes something to fail to instantiate so catching
regressions is that bit harder.  It does also mean we're restricted to
things which don't require any test environment beyond the simple
existence of the hardware which can be a bit restrictive for some
classes of hardware.