Re: [Ksummit-discuss] [CORE TOPIC] kernel testing standard

[chrubis@suse.cz]

Hi!
> > > >> As LTP is reported to
> > > >> be sane these days for some tests, it could conceivably be wrapped by "make
> > > >> test" to avoid duplicating effort there. I think that would be worthwhile
> > > >> if someone had the time to push it because it would be an unconditional win.
> > > >
> > > > That is what I have been asking for for _years_.  Hopefully someday
> > > > someone does this...
> > > 
> > > Does this read "let's pull LTP into the kernel source tree"?
> > 
> > Everyone always asks this, and I say, "Sure, why not?"
> > 
> > Actually, if people do complain about "why", then let's take the
> > "useful" subset of LTP for kernel developers.  It's a great place to
> > start, don't you agree?

This sure seems to be recurring topic. My opinion is that this wouldn't
be huge win.

There are a few downsides to this approach as well:

LTP is backward compatible and it makes sense to use the latest version
even for a few years old kernel/distro, because that brings you more
coverage and fixes. If it was part of the kernel tree there would be a
danger of people using outdated version packed with the kernel sources.

Splitting LTP it into two parts is in my opinion wrong thing to do,
because you will end up with two places that have to share some files
which will impose overhead in maintenance and confusion. Actually I've
looked over the testcases LTP has and it seems that there is a very
little testcases that are not kernel related (there are a few tests for
userspace tools (such as tar, unzip, su, at...) and some network tests).

> Cyril Hrubis is an LTP expert who has spend a considerable amount of time
> cleaning it up but is not often seen in kernel development circles so I added
> him to the cc. He's stated before that there is a large subset of LTP that
> is considerably cleaner than it was a few years ago. Cyril, you are probably
> not subscribed but the list archives for the "kernel testing standard" thread
> can be seen at http://lists.linuxfoundation.org/pipermail/ksummit-discuss/
> if you dig around a bit.

I've read this thread and what it's largely about is integration and
documentation both can be easily done even without pulling LTP into
kernel git tree.

Integrating LTP into 'make test' would be reasonably easy task. You just
need to download latest sources, build them, install and run it, then
look into single file for list of failures.

The more complicated part, as Ted said, is to figure out which tests to
run and on what occasion. For that you have to keep a list of testcases
per a group of interest which is time consuming and prone to error.

> There is a hazard that someone bisecting the tree would need to be careful
> to not bisect LTP instead. Otherwise, in your opinion how feasible
> would it be to import the parts of LTP you trust into the kernel tree
> under tools/testing/ ?

That depends on how closely you want to integrate with the rest of the
source tree. Copying all needed pieces from LTP source there would be
fairly easy.

However as I wrote earlier I want to avoid splitting the LTP source tree
in half. That would overly complicate the way LTP community works now.
What we do, for quite some time now, is to fix/cleanup testcase by
testcase and enable the fixed ones in default testrun.

-- 
Cyril Hrubis
chrubis@suse.cz