Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Shuah Khan <shuahkh@osg.samsung.com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/07/2015 11:18 AM, Mark Brown wrote:
> On Tue, Jul 07, 2015 at 08:25:21AM -0700, Guenter Roeck wrote:
>> On 07/07/2015 02:24 AM, Mark Brown wrote:
> 
>>> The main things I'm aware of that are happening at the minute
>>> are kselftest development, the 0day tester, plus kernelci.org
>>> and the other build and boot/test bots that are running against
>>> various trees.
> 
>> Maybe list all known ones as a start ?
> 
> Off the top of my head the automated ones I'm aware of are Olof's
> build & boot test, Dan running smatch and I think some other static
> analysis stuff, someone (not sure who?) running some coccinelle
> stuff, Coverity and I've got a builder too.
> 
>>> In terms of discussion topics some of the issues I'm seeing
>>> are:
> 
>>> - Can we pool resources to share the workload of running things
>>> and interpreting results, ideally also providing some central
>>> way for people to discover what results are out there for them
>>> to look at for a given kernel in the different systems?
> 
>> That might be quite useful. However, I have seen that it doesn't
>> really help to just provide the test results. kissb test results
>> have been available for ages, and people just don't look at it.
>> Even the regular "Build regression" e-mails sent out by Geert
>> seem to be widely ignored.
> 
>> What I really found to help is to bisect new problems and send an
>> e-mail to the responsible maintainer and to the submitter of the
>> patch which introduced it. I'd like to automate that with my test
>> system, but unfortunately I just don't have the time to do it.
> 
> Yes, that's the "and interpreting" bit in the above - this only
> really works with people actively pushing.  You do start to get
> people checking themselves once things are perceived as something
> people care about but it does take active work to establish and
> maintain that.
> 
> It also really helps if things are delivered promptly, and against
> trees people are actively developing for.  But even with clear
> reports and sometimes patches not everyone shows an interest.  As
> we get more and more actual testing running that's going to start
> to become more serious, breaking the build or boot will also mean
> that automated tests don't get to run.
> 
> This is one of the things 0day gets really right, when it kicks in
> it'll e-mail people directly and promptly.
> 
>>> - Should we start carrying config fragments upstream designed
>>> to support testing, things like the distro config fragments
>>> that keep getting discussed are one example here but there's
>>> other things like collections of debug options we could be
>>> looking at.  Should we be more generally slimming defconfigs
>>> and moving things into fragments?
> 
>>> and there's always the the perennial ones about what people
>>> would like to see testing for.
> 
>> Sharing as many test bot configuration scripts and relevant
>> configurations as possible would be quite helpful. For example, I
>> am building various configurations for all architectures, but I
>> don't really know if they are relevant. Also, I would like to run
>> more qemu configurations, but it is really hard to find working
>> ones.
> 
> Grant (just CCed) was working intermittently on the qemu bit.  I
> think the last plan was to enhance the scripts Kevin has for
> driving his build farm.
> 

Thanks for starting this discussion. Now that Kselftest install is
in place and several cross-compile problems are fixed, I would like
to gauge interest in being able to include kselftest in qemu boot
tests. I added quicktest option in 4.2 to meet qemu environment time
constraints.

thanks,
- -- Shuah

- -- 
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJVnAtBAAoJEAsCRMQNDUMca5wP/jrsBO++J4IsOnaJYwChWRp1
XHNEtrVwnvuR9zGo295lQUZL0cihug8FiTkfHX3P6M+9kEJQaUKlMyJymh9y4dzU
cmDLhh8kXwdtPq1qM8jzaTqBb6sqXvzXxRoMi3X7AYPd2yA4uLMluAgj+dg8n1x/
W55fIi4ghypNQvtLcC92zyp5MKtBd5zr75HUZPxsqPwo7AHtqy0KoV+1fx+qQ/Lt
GnZxIlWQMmA0yubiAGZHl+nECfiwcx8iS302y30d3+8/YxAztUy9l/7VfgBBgyOp
9Ouw6xC8Xmyb8GfFUUBy4MIAvS0QrBcYQp/z9Qlmj5MioOBydPaS3+Y4AIKLEaeI
Vu+/OVqZO9At+HML6TeDBvM44PWcjfC0plm8n8PLvFtruVnLEnxKeeaXCUrj9AG7
8GUTscjkOA5p8r3qqtKny3OftKvUINXZUL+zbEnV7kUlhlDuOBSN5SDZk6VkpTSP
lTJOTew+At0JH0x8YEA5k6L94JNT4zfksS/4N7z6bGcIdBIITrn3MGYkoi1fCqh2
GLHqKV8fLMv5aEK4tt9phDgmJSxITsMbfwuwoPYK0KRY4hUfri73uuBB1Igg9noI
d3H7yVYpL7Td3b98g7VuzIfOEdMbv9jFGgPJb8A0MiGYeu87iu56jJEXaj2JIAEo
SxNLh6SDkNkuH+6l2FAJ
=jD4c
-----END PGP SIGNATURE-----