Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Alexey Dobriyan <adobriyan@gmail.com>]

On Thu, Jul 9, 2015 at 1:24 PM, Masami Hiramatsu
<masami.hiramatsu.pt@hitachi.com> wrote:

> This may be an off-topic, but I'd like to ask the selftest for tools.
> Currently tools/testing/selftests tests the kernel itself, but
> there are many tools under tools/, like perf too.
>
> Those are not configured by the kconfig, but selftests are also needed
> for tools. I have a runtests script which is just a bit modified
> ftracetest for perf-probe. I'd like to integrate it to selftests
> but I'm not sure that is a scope of kselftests.

This confusion is partially created by peculiar place where people
who wrote testsuite put it.

Gentlemen,
testsuite should be first class citizen in toplevel test/ directory,
command to run it should be "make test" not "make kselftest".
Only placing it in very visible place and using names which are intuitive
and familiar from userspace (git's t/ directory, glibc "make test") will give
hope that other developers will notice it and start using and improving it.
Excuse me, but tools/testing/selftests is hopeless.

>> Perhaps we should have a central location that each test needs to add
>> the required configuration for it to be properly tested. Then if users
>> want to test various subsystems, they would look in this location for
>> the proper configs (be it a directory that has files of the tests they
>> represent, and contain the configs needed). Then there should be no
>> real barrier for people to run these tests.
>
> /proc/kconfig[.gz]? I think we can add a list of required kconfigs
> for each testcase and indicate it. Moreover, we can include it as
> a part of kconfig and introduce CONFIG_KSELFTEST to enable those
> configs :)

I think primary use case is this:
* user builds and reboots into kernel with his custom config,
* user runs "make test" from fresh build directory,
* test harness runs everything runnable and maybe reports necessary
  config options to run more

/proc/kconfig.gz should be kept strictly for runtime config.