From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTP id 54B7CAAC for ; Fri, 9 May 2014 17:58:05 +0000 (UTC) Received: from mail-yk0-f174.google.com (mail-yk0-f174.google.com [209.85.160.174]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id A7A36201AE for ; Fri, 9 May 2014 17:58:04 +0000 (UTC) Received: by mail-yk0-f174.google.com with SMTP id 9so3726306ykp.19 for ; Fri, 09 May 2014 10:58:04 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: <1399552623.17118.22.camel@i7.infradead.org> Date: Fri, 9 May 2014 13:58:03 -0400 Message-ID: From: Matthew Wilcox To: Roland Dreier Content-Type: multipart/alternative; boundary=20cf303ea712d2ff7204f8fb562f Cc: linux-nvme@lists.infradead.org, "ksummit-discuss@lists.linuxfoundation.org" Subject: Re: [Ksummit-discuss] [CORE TOPIC] Device error handling / reporting / isolation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , --20cf303ea712d2ff7204f8fb562f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I'm hearing a bunch of FUD around NVMe hotplug but precious little in the way of bug reports! Keith Busch has been doing a stellar job of fixing up the bugs that he's found, but I have seen precisely zero hotplug bugs reported to the NVMe mailing list. So put up or shut up. On 2014-05-09 1:49 PM, "Roland Dreier" wrote: > On Thu, May 8, 2014 at 5:37 AM, David Woodhouse > wrote: > > I'd like to have a discussion about handling device errors. > > > > IOMMUs are becoming more common, and we've seen some failure modes wher= e > > we just end up with an endless stream of fault reports from a given > > device, and the kernel can do nothing else. > > > > We may have various options for shutting it up =E2=80=94 a PCI function= level > > reset, power cycling the offending device, or maybe just configuring th= e > > IOMMU to *ignore* further errors from it, which would at least let the > > system get on with doing something useful (and if we do, when do we > > re-enable reporting?). > > I think there's a more general problem that's worth talking about > here. In addition to IOMMU faults, there are lots of other PCI errors > that can happen, and we have some small number of drivers that have > been "hardened" to try and recover from these errors. However even > for these "hardened" drivers it seems pretty easy to hit deadlocks > when the driver tries to tear down and reinitialize things. > > So I wonder if we can do better without proliferating error handling > tentacles into all sorts of low-level drivers ("did we just read > 0xffffffff here? how about here? are we in the middle of error > recovery? how about now?"). > > One context where this is becoming a real concern is with NVMe drives. > These are SSDs that (may) look like normal 2.5" drives, but use PCIe > rather than SATA or SAS to connect to the host. Since they look like > normal drives, it's natural to put them into hot-pluggable JBODs, but > it turns out we react much worse to PCIe surprise removal than, say, > SAS hotplug. > > - R. > _______________________________________________ > Ksummit-discuss mailing list > Ksummit-discuss@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss > --20cf303ea712d2ff7204f8fb562f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

I'm hearing a bunch of FUD around NVMe hotplug but precious little i= n the way of bug reports! Keith Busch has been doing a stellar job of fixin= g up the bugs that he's found, but I have seen precisely zero hotplug b= ugs reported to the NVMe mailing list. So put up or shut up.

On 2014-05-09 1:49 PM, "Roland Dreier"= <roland@kernel.org> wrote:<= br type=3D"attribution">
On Thu, May 8, 2014 at 5:37 AM, David Woodhouse <dwmw2@infradead.org> wrote:
> I'd like to have a discussion about handling device errors.
>
> IOMMUs are becoming more common, and we've seen some failure modes= where
> we just end up with an endless stream of fault reports from a given > device, and the kernel can do nothing else.
>
> We may have various options for shutting it up =E2=80=94 a PCI functio= n level
> reset, power cycling the offending device, or maybe just configuring t= he
> IOMMU to *ignore* further errors from it, which would at least let the=
> system get on with doing something useful (and if we do, when do we > re-enable reporting?).

I think there's a more general problem that's worth talking about here. =C2=A0In addition to IOMMU faults, there are lots of other PCI errors=
that can happen, and we have some small number of drivers that have
been "hardened" to try and recover from these errors. =C2=A0Howev= er even
for these "hardened" drivers it seems pretty easy to hit deadlock= s
when the driver tries to tear down and reinitialize things.

So I wonder if we can do better without proliferating error handling
tentacles into all sorts of low-level drivers ("did we just read
0xffffffff here? =C2=A0how about here? =C2=A0are we in the middle of error<= br> recovery? =C2=A0how about now?").

One context where this is becoming a real concern is with NVMe drives.
=C2=A0These are SSDs that (may) look like normal 2.5" drives, but use = PCIe
rather than SATA or SAS to connect to the host. =C2=A0Since they look like<= br> normal drives, it's natural to put them into hot-pluggable JBODs, but it turns out we react much worse to PCIe surprise removal than, say,
SAS hotplug.

=C2=A0- R.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discus= s@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ks= ummit-discuss
--20cf303ea712d2ff7204f8fb562f--