Re: [Ksummit-discuss] [CORE TOPIC] kernel testing standard

["Justin M. Forbes" <jforbes@fedoraproject.org>]

On Fri, 2014-05-23 at 20:47 +0900, Masami Hiramatsu wrote:
> Hi,
> 
> As I discussed with Greg K.H. at LinuxCon Japan yesterday,
> I'd like to propose kernel testing standard as a separated topic.
> 
> Issue:
> There are many ways to test the kernel but it's neither well documented
> nor standardized/organized.
> 
> As you may know, testing kernel is important on each phase of kernel
> life-cycle. For example, even at the designing phase, actual test-case
> shows us what the new feature/design does, how is will work, and how
> to use it. This can improve the quality of the discussion.
> 
> Through the previous discussion I realized there are many different methods/
> tools/functions for testing kernel, LTP, trinity, tools/testing/selftest,
> in-kernel selftest etc. Each has good points and bad points.
> 
> So, I'd like to discuss how we can standardize them for each subsystem
> at this kernel summit.
> 
This would be a great discussion to have. Trying to build a regression
testing framework for kernel builds has been one of my big projects
recently.  Getting decent test coverage is kind of a nightmare at the
moment.

> My suggestion are,
> - Organizing existing in-tree kernel test frameworks (as "make test")
> - Documenting the standard testing method, including how to run,
>   how to add test-cases, and how to report.
> - Commenting standard testing for each subsystem, maybe by adding
>   UT: or TS: tags to MAINTAINERS, which describes the URL of
>   out-of-tree tests or the directory of the selftest.
> 
> Note that I don't tend to change the ways to test for subsystems which
> already have own tests, but organize it for who wants to get involved in
> and/or to evaluate it. :-)
> 
All good suggestions. As nice as it would be if tests were in tree, this
might be unmanageable. But even out of tree tests could be automatically
brought in provided they are listed somewhere in tree.  Ideally you
would be able to "make tests" and get all in tree tests run, or "make
alltests" and have it grab/build/run out of tree tests with git urls as
well. 

> I think we can strongly request developers to add test-cases for new features
> if we standardize the testing method.

Not just new features, bug fixes as well. Though writing a test can
sometimes be more difficult than the actual bug fix.  Without some sort
of framework, it is harder to ask for developer participation. We need a
framework that makes it easy.

> Suggested participants: greg k.h., Li Zefan, test-tool maintainers and
>                        subsystem maintainers.
I would love to be a part of this discussion, and creating a working
solution.

Justin