From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 8700F1E41 for ; Mon, 14 May 2018 09:25:53 +0000 (UTC) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 7F558196 for ; Mon, 14 May 2018 09:25:52 +0000 (UTC) Date: Mon, 14 May 2018 17:25:48 +0800 From: Fengguang Wu To: Boris Brezillon Message-ID: <20180514092548.qwv4ycsixduqd6q3@wfg-t540p.sh.intel.com> References: <20180510164722.GH8514@sasha-vm> <20180514101237.5df1e0d7@bbrezillon> <20180514103428.55285a70@bbrezillon> <20180514104803.23d2a8a3@bbrezillon> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20180514104803.23d2a8a3@bbrezillon> Cc: "ksummit-discuss@lists.linuxfoundation.org" , Greg KH , "linux-kernel@vger.kernel.org" , "w@1wt.eu" Subject: Re: [Ksummit-discuss] bug-introducing patches List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On Mon, May 14, 2018 at 10:48:03AM +0200, Boris Brezillon wrote: >+Fengguang > >On Mon, 14 May 2018 10:40:10 +0200 >Geert Uytterhoeven wrote: > >> Hi Boris, >> >> On Mon, May 14, 2018 at 10:34 AM, Boris Brezillon >> wrote: >> > On Mon, 14 May 2018 10:29:04 +0200 >> > Geert Uytterhoeven wrote: >> >> On Mon, May 14, 2018 at 10:12 AM, Boris Brezillon >> >> wrote: >> >> > On Mon, 14 May 2018 10:00:30 +0200 >> >> > Geert Uytterhoeven wrote: >> >> >> On Tue, May 1, 2018 at 10:00 PM, Sasha Levin >> >> >> wrote: >> >> >> > On Tue, May 01, 2018 at 03:44:50PM -0400, Theodore Y. Ts'o wrote: >> >> >> >>On Tue, May 01, 2018 at 04:38:21PM +0000, Sasha Levin wrote: >> >> >> > What's worse is that that commit is tagged for stable, which means >> >> >> > that (given Greg's schedule) it may find it's way to -stable users >> >> >> > even before some -next users/bots had a chance to test it out. >> >> >> >> >> >> I just noticed a case where a commit was picked up for stable, while a >> >> >> bot had flagged it as a build regression 18 hours earlier (with a CC to >> >> >> lkml). >> >> > >> >> > Also, this patch has been on a tree that I know is tested by Fengguang's >> >> > robots for more than a week (and in linux-next for 2 days, which, I >> >> > agree, is probably not enough), and still, I only received the bug >> >> > report when the patch reached mainline. Are there tests that are only >> >> > run on Linus' tree? >> >> >> >> Have your received a success report from Fengguang's bot, listing all >> >> configs tested (the broken one should be included; it is included in the >> >> configs tested on my branches)? >> > >> > Yes I did (see below). >> > >> > -->8-- >> > From: kbuild test robot >> > To: Boris Brezillon >> > Subject: [bbrezillon-0day:mtd/fixes] BUILD SUCCESS fc3a9e15b492eef707afd56b7478001fdecfe53f >> > Date: Mon, 07 May 2018 20:05:52 +0800 >> > User-Agent: Heirloom mailx 12.5 6/20/10 >> > >> > tree/branch: https://github.com/bbrezillon/linux-0day mtd/fixes >> > branch HEAD: fc3a9e15b492eef707afd56b7478001fdecfe53f mtd: rawnand: Make sure we wait tWB before polling the STATUS reg >> > >> > elapsed time: 49m >> > >> > configs tested: 142 >> >> But the failed config (m68k/allmodconfig) is not listed? > >Yes, that's my point. It seems that some configs are only rarely >(never?) tested on my linux-0day tree (probably because they take longer >to build), and I should only take kbuild robot results as an indication >not a guarantee. Yeah sorry, there is no 100% guarantee. There are 2 main aspects to this problem. - Response time vs coverage. Most build errors can be caught within 1 day. The build success notification email is typically sent within half day (a reasonable feedback time). At this time, it can only be a rough indication not a guarantee. After sending the 0day build success notification, the build tests will actually continue for about 1 week to increase test coverage. - Merge-test-bisect based workflow. If one branch is hard to merge with others, especially if it's based on old kernel, it'll receive much less test coverage. Branches with known build/boot errors will be excluded from further merges, too. Thanks, Fengguang