Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Shuah Khan <shuahkh@osg.samsung.com>]

On 07/10/2015 04:39 AM, Alexey Dobriyan wrote:
> On Thu, Jul 9, 2015 at 1:24 PM, Masami Hiramatsu
> <masami.hiramatsu.pt@hitachi.com> wrote:
> 
>> This may be an off-topic, but I'd like to ask the selftest for tools.
>> Currently tools/testing/selftests tests the kernel itself, but
>> there are many tools under tools/, like perf too.
>>
>> Those are not configured by the kconfig, but selftests are also needed
>> for tools. I have a runtests script which is just a bit modified
>> ftracetest for perf-probe. I'd like to integrate it to selftests
>> but I'm not sure that is a scope of kselftests.
> 
> This confusion is partially created by peculiar place where people
> who wrote testsuite put it.
> 
> Gentlemen,
> testsuite should be first class citizen in toplevel test/ directory,
> command to run it should be "make test" not "make kselftest".
> Only placing it in very visible place and using names which are intuitive
> and familiar from userspace (git's t/ directory, glibc "make test") will give
> hope that other developers will notice it and start using and improving it.
> Excuse me, but tools/testing/selftests is hopeless.

selftests are intended for kernel developers primarily. If developers
and users don't want to figure out what are the ways to test, then it
doesn't matter what the option is named. I would like to hear some
concrete data on why naming test would make it lot more usable.

> 
>>> Perhaps we should have a central location that each test needs to add
>>> the required configuration for it to be properly tested. Then if users
>>> want to test various subsystems, they would look in this location for
>>> the proper configs (be it a directory that has files of the tests they
>>> represent, and contain the configs needed). Then there should be no
>>> real barrier for people to run these tests.
>>
>> /proc/kconfig[.gz]? I think we can add a list of required kconfigs
>> for each testcase and indicate it. Moreover, we can include it as
>> a part of kconfig and introduce CONFIG_KSELFTEST to enable those
>> configs :)

I don't believe adding yet another kernel config option especially
for KSELFTEST is a good idea.

> 
> I think primary use case is this:
> * user builds and reboots into kernel with his custom config,
> * user runs "make test" from fresh build directory,
> * test harness runs everything runnable and maybe reports necessary
>   config options to run more
> 
> /proc/kconfig.gz should be kept strictly for runtime config.
> 

We do have ktest for that. Maybe ktest could include kselftest
run in its default boot test.

thanks,
-- Shuah

-- 
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978