From: Benjamin Gilbert <bgilbert@redhat.com>
To: ksummit-discuss@lists.linuxfoundation.org
Subject: Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Stable trees and release time
Date: Wed, 5 Sep 2018 23:56:38 -0400 [thread overview]
Message-ID: <20180906035638.GA8506@trogon.sfo.coreos.systems> (raw)
In-Reply-To: <61221bc0-7610-36c6-496e-57b1a0d20eec@redhat.com>
On Tue, Sep 04, 2018 at 06:17:46PM -0700, Laura Abbott wrote:
> On 09/04/2018 04:43 PM, Guenter Roeck wrote:
> > On 09/04/2018 04:14 PM, Sasha Levin wrote:
> >> Maybe some concrete numbers will help here. Do you maybe know how many
> >> commits in the past year snuck past the -rc cycle into a stable release
> >> and found as buggy by Fedora's testing pipeline?
> >
> > ... and how many bugs were found during the existing test cycle ?
> >
> > The next question would be how many regressions were reported by users
> > after a release was published.
> >
> > The statistics I carried until early this year suggested a regression rate
> > of around 0.15% for stable releases, where regression means that a bug was
> > found post-release and had to be fixed later. It would indeed be interesting
> > to know how many of those were found by (automated ?) testing and how many
> > were found by users.
>
> I'd have to do some digging through bugzilla to get numbers. Some
> of this is also motivated by discussions with the CoreOS team who
> have also tried to use the stable kernels and ran into problems.
> I'll see if I can get some numbers.
We've shipped 11 different 4.14.x kernels on the CoreOS Container Linux
stable channel, all in this calendar year. Five of them had user-impacting
regressions that had to be fixed via OS updates:
4.14.30 - vxlan panic
https://github.com/coreos/bugs/issues/2382
4.14.42 - Failure to set MTU in xen-netfront
https://github.com/coreos/bugs/issues/2443
4.14.44 - Failure to bring up hv_netvsc interface after it was brought down
https://github.com/coreos/bugs/issues/2454
4.14.48 - Integer overflow causing tiny TCP receive windows
https://github.com/coreos/bugs/issues/2457
4.14.55 - Broken CIFS client
https://github.com/coreos/bugs/issues/2480
4.14.55 - Failure to mount ext4 filesystems 3 TB or larger
https://github.com/coreos/bugs/issues/2485
The TCP window bug was particularly exciting, since affected machines would
have downloaded the fixed OS image at ~300 bytes/sec. To avoid that, we had
to implement several workarounds in our update infrastructure, only the
second time we've had to do that in the history of Container Linux.
--Benjamin Gilbert
next prev parent reply other threads:[~2018-09-06 3:56 UTC|newest]
Thread overview: 74+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-09-04 20:58 Laura Abbott
2018-09-04 21:12 ` Jiri Kosina
2018-09-05 14:31 ` Greg KH
2018-09-04 21:22 ` Justin Forbes
2018-09-05 14:42 ` Greg KH
2018-09-05 15:10 ` Mark Brown
2018-09-05 15:10 ` Sasha Levin
2018-09-05 16:19 ` Guenter Roeck
2018-09-05 18:31 ` Laura Abbott
2018-09-05 21:23 ` Justin Forbes
2018-09-06 2:17 ` Eduardo Valentin
2018-09-04 21:33 ` Sasha Levin
2018-09-04 21:55 ` Guenter Roeck
2018-09-04 22:03 ` Laura Abbott
2018-09-04 23:14 ` Sasha Levin
2018-09-04 23:43 ` Guenter Roeck
2018-09-05 1:17 ` Laura Abbott
2018-09-06 3:56 ` Benjamin Gilbert [this message]
2018-09-04 21:58 ` Laura Abbott
2018-09-05 4:53 ` Sasha Levin
2018-09-05 6:48 ` Jiri Kosina
2018-09-05 8:16 ` Jan Kara
2018-09-05 8:32 ` Jiri Kosina
2018-09-05 8:56 ` Greg KH
2018-09-05 9:13 ` Geert Uytterhoeven
2018-09-05 9:33 ` Greg KH
2018-09-05 10:11 ` Mark Brown
2018-09-05 14:44 ` Steven Rostedt
2018-09-05 9:58 ` James Bottomley
2018-09-05 10:47 ` Mark Brown
2018-09-05 12:24 ` James Bottomley
2018-09-05 12:53 ` Jiri Kosina
2018-09-05 13:05 ` Greg KH
2018-09-05 13:15 ` Jiri Kosina
2018-09-05 14:00 ` Greg KH
2018-09-05 14:06 ` Sasha Levin
2018-09-05 21:02 ` Jiri Kosina
2018-09-05 16:39 ` James Bottomley
2018-09-05 17:06 ` Dmitry Torokhov
2018-09-05 17:33 ` Steven Rostedt
2018-09-05 13:03 ` Takashi Iwai
2018-09-05 13:27 ` Daniel Vetter
2018-09-05 14:05 ` Greg KH
2018-09-05 15:54 ` Daniel Vetter
2018-09-05 16:19 ` Sasha Levin
2018-09-05 16:26 ` Daniel Vetter
2018-09-05 19:09 ` Sasha Levin
2018-09-05 20:18 ` Sasha Levin
2018-09-05 20:33 ` Daniel Vetter
2018-09-05 14:20 ` Sasha Levin
2018-09-05 14:30 ` Takashi Iwai
2018-09-05 14:41 ` Sasha Levin
2018-09-05 14:46 ` Takashi Iwai
2018-09-05 14:54 ` Sasha Levin
2018-09-05 15:12 ` Takashi Iwai
2018-09-05 15:19 ` Thomas Gleixner
2018-09-05 15:29 ` Sasha Levin
2018-09-05 13:16 ` Mark Brown
2018-09-05 14:27 ` Sasha Levin
2018-09-05 14:50 ` Mark Brown
2018-09-05 15:00 ` Sasha Levin
2018-09-05 10:28 ` Thomas Gleixner
2018-09-05 11:20 ` Jiri Kosina
2018-09-05 14:41 ` Thomas Gleixner
2018-09-05 15:18 ` Steven Rostedt
2018-09-06 8:48 ` Thomas Gleixner
2018-09-06 12:47 ` Thomas Gleixner
2018-09-04 21:49 ` Guenter Roeck
2018-09-04 22:06 ` Laura Abbott
2018-09-04 23:35 ` Guenter Roeck
2018-09-05 1:45 ` Laura Abbott
2018-09-05 2:54 ` Guenter Roeck
2018-09-05 8:31 ` Jan Kara
2018-09-05 3:44 ` Eduardo Valentin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180906035638.GA8506@trogon.sfo.coreos.systems \
--to=bgilbert@redhat.com \
--cc=ksummit-discuss@lists.linuxfoundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox