[Ksummit-discuss] [TECH TOPIC] (PCI) driver rebinding

[Ksummit-discuss] [TECH TOPIC] (PCI) driver rebinding

[Hannes Reinecke]

Hi all,

here's a thing which came up recently on the SCSI ml:

How should we handle devices with _several_ possible drivers?

Case in point are some HBAs, for which the plan is to have distinct
drivers for either target or initiator mode.

There are several options how this could be handled:
- Have a default driver, and manually rebind the 'real' driver once the
configuration could be read and acted upon.
That would have the drawback that we do an initialisation in the 'wrong'
mode, and later have to switch configuration. Which at best will 'just'
induce some pointless initialisation, at worst confuse attached devices
to no end. Not to mention introducing lots of interesting races with
udev and systemd
- Inhibit default binding, and load the correct drivers after the
configuration has been read.
Would be a deviation from the original scenario (where the driver are
bound/loaded as soon as the device becomes available). Has the drawback
that one would need to inhibit automatic bindings on the _bus_ level (ie
PCI), which will be even more interesting.
- Add some bus-specific (or even general?) device configuration syntax,
which would allow to pass in the required configuration from the
command-line (or reasonably early, anyway). For which we need a syntax
first, anyway. And need to figure out if we can have an abstract syntax
or would need to have a bus- (or even driver-) specific configuration.

So no easy way out here, and it might be worth having input from other
parties. And some might have similar problems (just thinking of
usb-modeswitch ...), so there might be some synergies to be had.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.com			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [Ksummit-discuss] [TECH TOPIC] (PCI) driver rebinding

[Matthew Wilcox]

[-- Attachment #1: Type: text/plain, Size: 2469 bytes --]

Is udev the solution here?

Have ->probe do common initialisation and return success. Then udev says
something to the driver to make it choose one or the other modes.

On 25 Jun 2017 11:40 am, "Hannes Reinecke" <hare@suse.com> wrote:

> Hi all,
>
> here's a thing which came up recently on the SCSI ml:
>
> How should we handle devices with _several_ possible drivers?
>
> Case in point are some HBAs, for which the plan is to have distinct
> drivers for either target or initiator mode.
>
> There are several options how this could be handled:
> - Have a default driver, and manually rebind the 'real' driver once the
> configuration could be read and acted upon.
> That would have the drawback that we do an initialisation in the 'wrong'
> mode, and later have to switch configuration. Which at best will 'just'
> induce some pointless initialisation, at worst confuse attached devices
> to no end. Not to mention introducing lots of interesting races with
> udev and systemd
> - Inhibit default binding, and load the correct drivers after the
> configuration has been read.
> Would be a deviation from the original scenario (where the driver are
> bound/loaded as soon as the device becomes available). Has the drawback
> that one would need to inhibit automatic bindings on the _bus_ level (ie
> PCI), which will be even more interesting.
> - Add some bus-specific (or even general?) device configuration syntax,
> which would allow to pass in the required configuration from the
> command-line (or reasonably early, anyway). For which we need a syntax
> first, anyway. And need to figure out if we can have an abstract syntax
> or would need to have a bus- (or even driver-) specific configuration.
>
> So no easy way out here, and it might be worth having input from other
> parties. And some might have similar problems (just thinking of
> usb-modeswitch ...), so there might be some synergies to be had.
>
> Cheers,
>
> Hannes
> --
> Dr. Hannes Reinecke                Teamlead Storage & Networking
> hare@suse.com                                  +49 911 74053 688
> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
> HRB 21284 (AG Nürnberg)
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
>

[-- Attachment #2: Type: text/html, Size: 3238 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] (PCI) driver rebinding

[Greg KH]

On Sun, Jun 25, 2017 at 05:39:34PM +0200, Hannes Reinecke wrote:
> Hi all,
> 
> here's a thing which came up recently on the SCSI ml:
> 
> How should we handle devices with _several_ possible drivers?

Go kick the hardware developers for trying to come up with such complex
beasts :)

> Case in point are some HBAs, for which the plan is to have distinct
> drivers for either target or initiator mode.
> 
> There are several options how this could be handled:
> - Have a default driver, and manually rebind the 'real' driver once the
> configuration could be read and acted upon.

Yes, do this.

> That would have the drawback that we do an initialisation in the 'wrong'
> mode, and later have to switch configuration. Which at best will 'just'
> induce some pointless initialisation, at worst confuse attached devices
> to no end. Not to mention introducing lots of interesting races with
> udev and systemd

How will udev and systemd have "races"?  We do this today all the time
in USB with the usb-modeswitch userspace code, as you point out.

> - Inhibit default binding, and load the correct drivers after the
> configuration has been read.

You can do that too vfio does it from what I recall.  It's easy in that
you can just not do a MODULE_DEVICE_TABLE() export and no driver will
automatically be loaded and userspace is required to load what it wants.

> Would be a deviation from the original scenario (where the driver are
> bound/loaded as soon as the device becomes available). Has the drawback
> that one would need to inhibit automatic bindings on the _bus_ level (ie
> PCI), which will be even more interesting.
> - Add some bus-specific (or even general?) device configuration syntax,
> which would allow to pass in the required configuration from the
> command-line (or reasonably early, anyway). For which we need a syntax
> first, anyway. And need to figure out if we can have an abstract syntax
> or would need to have a bus- (or even driver-) specific configuration.

Hah, good luck with that :)

greg k-h

Re: [Ksummit-discuss] [TECH TOPIC] (PCI) driver rebinding

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2564 bytes --]

On Sun, Jun 25, 2017 at 05:39:34PM +0200, Hannes Reinecke wrote:
> Hi all,
>
> here's a thing which came up recently on the SCSI ml:
>
> How should we handle devices with _several_ possible drivers?
>
> Case in point are some HBAs, for which the plan is to have distinct
> drivers for either target or initiator mode.
>
> There are several options how this could be handled:
> - Have a default driver, and manually rebind the 'real' driver once the
> configuration could be read and acted upon.
> That would have the drawback that we do an initialisation in the 'wrong'
> mode, and later have to switch configuration. Which at best will 'just'
> induce some pointless initialisation, at worst confuse attached devices
> to no end. Not to mention introducing lots of interesting races with
> udev and systemd
> - Inhibit default binding, and load the correct drivers after the
> configuration has been read.
> Would be a deviation from the original scenario (where the driver are
> bound/loaded as soon as the device becomes available). Has the drawback
> that one would need to inhibit automatic bindings on the _bus_ level (ie
> PCI), which will be even more interesting.
> - Add some bus-specific (or even general?) device configuration syntax,
> which would allow to pass in the required configuration from the
> command-line (or reasonably early, anyway). For which we need a syntax
> first, anyway. And need to figure out if we can have an abstract syntax
> or would need to have a bus- (or even driver-) specific configuration.

It sounds very similar to devlink, despite that tool was originally introduced
to net and intended for switches, it works on PCI level with BUS_NAME/BUS_ADDRESS
as an handle.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bfcd3a46617209454cfc0947ab093e37fd1e84ef

>
> So no easy way out here, and it might be worth having input from other
> parties. And some might have similar problems (just thinking of
> usb-modeswitch ...), so there might be some synergies to be had.
>
> Cheers,
>
> Hannes
> --
> Dr. Hannes Reinecke		   Teamlead Storage & Networking
> hare@suse.com			               +49 911 74053 688
> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
> HRB 21284 (AG Nürnberg)
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] (PCI) driver rebinding

[Bart Van Assche]

On Sun, 2017-06-25 at 17:39 +0200, Hannes Reinecke wrote:
> How should we handle devices with _several_ possible drivers?
> 
> Case in point are some HBAs, for which the plan is to have distinct
> drivers for either target or initiator mode.
> 
> There are several options how this could be handled:
> - Have a default driver, and manually rebind the 'real' driver once the
> configuration could be read and acted upon.
> That would have the drawback that we do an initialisation in the 'wrong'
> mode, and later have to switch configuration. Which at best will 'just'
> induce some pointless initialisation, at worst confuse attached devices
> to no end. Not to mention introducing lots of interesting races with
> udev and systemd
> - Inhibit default binding, and load the correct drivers after the
> configuration has been read.
> Would be a deviation from the original scenario (where the driver are
> bound/loaded as soon as the device becomes available). Has the drawback
> that one would need to inhibit automatic bindings on the _bus_ level (ie
> PCI), which will be even more interesting.
> - Add some bus-specific (or even general?) device configuration syntax,
> which would allow to pass in the required configuration from the
> command-line (or reasonably early, anyway). For which we need a syntax
> first, anyway. And need to figure out if we can have an abstract syntax
> or would need to have a bus- (or even driver-) specific configuration.
> 
> So no easy way out here, and it might be worth having input from other
> parties. And some might have similar problems (just thinking of
> usb-modeswitch ...), so there might be some synergies to be had.

(resending from an e-mail address that is subscribed to ksummit-discuss)

Hello Hannes,

Thanks for having brought up this topic. I think this is an important
topic and something we need to solve. I am interested to join the
discussion about this topic.

My preference is to solve this by following the same approach as for the
iSCSI, SRP and several other protocols: design the protocol and HBA such
that initiator and target mode can coexist on the same HBA port, make one
driver responsible for controlling the HBA, a second driver responsible
for providing initiator functionality and make a third driver responsible
for providing target functionality. Whether initiator and/or target mode
are activated depends on which drivers are loaded and also on which
drivers the user chooses to activate.  With this approach no PCI rebinding
is necessary.

Bart.

Re: [Ksummit-discuss] [TECH TOPIC] (PCI) driver rebinding

[Hannes Reinecke]

On 06/25/2017 11:38 PM, Matthew Wilcox wrote:
> Is udev the solution here?
> 
(Top posting; tsk. Probably the new job environment :-)

> Have ->probe do common initialisation and return success. Then udev says
> something to the driver to make it choose one or the other modes.
> 
Which would be option 2 from my proposal.
(And, incidentally, is what zSeries does.)
Plus you essentially spanning off your own bus from the PCI device,
which would solve the PCI binding problem.
But you just delegated the problem one level higher, as you then have to
configure your private bus. So I don't think it solves the underlying
problem, namely how to get the configuration to the driver.

Another problem here is that you rely on udev to do the binding, ie you
have to configure the device first before it can used. Which requires
the use of an initrd, and I'm not sure if everyone can agree to that.

Or come up with a commandline parameter instructing the driver on which
devices to bind.

So it's far from being clear-cut; hence this proposal.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.com			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [Ksummit-discuss] [TECH TOPIC] (PCI) driver rebinding

[Hannes Reinecke]

On 06/26/2017 06:59 AM, Greg KH wrote:
> On Sun, Jun 25, 2017 at 05:39:34PM +0200, Hannes Reinecke wrote:
>> Hi all,
>>
>> here's a thing which came up recently on the SCSI ml:
>>
>> How should we handle devices with _several_ possible drivers?
> 
> Go kick the hardware developers for trying to come up with such complex
> beasts :)
> 
>> Case in point are some HBAs, for which the plan is to have distinct
>> drivers for either target or initiator mode.
>>
>> There are several options how this could be handled:
>> - Have a default driver, and manually rebind the 'real' driver once the
>> configuration could be read and acted upon.
> 
> Yes, do this.
> 
>> That would have the drawback that we do an initialisation in the 'wrong'
>> mode, and later have to switch configuration. Which at best will 'just'
>> induce some pointless initialisation, at worst confuse attached devices
>> to no end. Not to mention introducing lots of interesting races with
>> udev and systemd
> 
> How will udev and systemd have "races"?  We do this today all the time
> in USB with the usb-modeswitch userspace code, as you point out.
> 
The problem here is that we will be generating lots of invalid devices
(ie devices which are only present in one mode, but not the other).
So if we start off with a default mode the driver might be creating lots
of devices, which will be torn down once the device is switched to the
configured mode.
_If_ for some reason one of these invalid devices has the same
identifier as one of the 'real' devices (think of eg multipathing),
udev/systemd will start processing this device.
But then we're killing that device shortly afterwards, making systemd
very unhappy and the user left with a stalled system.

So I'd rather not go down that way; ideally I would have the driver
start off in the 'correct' mode so that the above issue doesn't occur.

>> - Inhibit default binding, and load the correct drivers after the
>> configuration has been read.
> 
> You can do that too vfio does it from what I recall.  It's easy in that
> you can just not do a MODULE_DEVICE_TABLE() export and no driver will
> automatically be loaded and userspace is required to load what it wants.
> 
Yeah; however, if we do this, we either have to kill the module device
table from the driver (but how would the driver then know which device
to bind to?) or we have to kill the autoloading from the bus (but then
_none_ of the drivers will load, impacting the entire system).

While I would prefer this option I'm not sure how we can achieve this
sanely.
vfio doesn't really count here, as this is a generic PCI driver which
would work with any PCI device (at least in theory).

>> Would be a deviation from the original scenario (where the driver are
>> bound/loaded as soon as the device becomes available). Has the drawback
>> that one would need to inhibit automatic bindings on the _bus_ level (ie
>> PCI), which will be even more interesting.
>> - Add some bus-specific (or even general?) device configuration syntax,
>> which would allow to pass in the required configuration from the
>> command-line (or reasonably early, anyway). For which we need a syntax
>> first, anyway. And need to figure out if we can have an abstract syntax
>> or would need to have a bus- (or even driver-) specific configuration.
> 
> Hah, good luck with that :)
> 
That's why I put is last :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.com			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [Ksummit-discuss] [TECH TOPIC] (PCI) driver rebinding

[Greg KH]

On Mon, Jun 26, 2017 at 08:05:53AM +0200, Hannes Reinecke wrote:
> On 06/26/2017 06:59 AM, Greg KH wrote:
> > How will udev and systemd have "races"?  We do this today all the time
> > in USB with the usb-modeswitch userspace code, as you point out.
> > 
> The problem here is that we will be generating lots of invalid devices
> (ie devices which are only present in one mode, but not the other).
> So if we start off with a default mode the driver might be creating lots
> of devices, which will be torn down once the device is switched to the
> configured mode.
> _If_ for some reason one of these invalid devices has the same
> identifier as one of the 'real' devices (think of eg multipathing),
> udev/systemd will start processing this device.
> But then we're killing that device shortly afterwards, making systemd
> very unhappy and the user left with a stalled system.

If you have someone adding/removing a device on a pcie link (or
thunderbolt) really quickly, you are going to have this same "issue", so
it's something you are going to have to handle anyway.  Odds are it's
not going to be all that complex, userspace, if it is sane, should be
fine.

But you are talking "multipath", which is not sane, good luck!  :)

> So I'd rather not go down that way; ideally I would have the driver
> start off in the 'correct' mode so that the above issue doesn't occur.

I agree.

> >> - Inhibit default binding, and load the correct drivers after the
> >> configuration has been read.
> > 
> > You can do that too vfio does it from what I recall.  It's easy in that
> > you can just not do a MODULE_DEVICE_TABLE() export and no driver will
> > automatically be loaded and userspace is required to load what it wants.
> > 
> Yeah; however, if we do this, we either have to kill the module device
> table from the driver (but how would the driver then know which device
> to bind to?) or we have to kill the autoloading from the bus (but then
> _none_ of the drivers will load, impacting the entire system).

No, MODULE_DEVICE_TABLE is only needed for auto-loading of modules, it
does not affect what drivers bind to what devices at all.  Once the
module is loaded, all should "just work", unless you have multiple
drivers that say they bind to the same devices, then you have the same
issue as before.

> While I would prefer this option I'm not sure how we can achieve this
> sanely.
> vfio doesn't really count here, as this is a generic PCI driver which
> would work with any PCI device (at least in theory).

That sounds exactly what vfio does, they handle this all in userspace,
and do crazy stuff with the bind/unbind sysfs attributes.  That might
work here too if you have to do it in the end.

thanks,

greg k-h

Re: [Ksummit-discuss] [TECH TOPIC] (PCI) driver rebinding

[Lee Duncan]

On 06/25/2017 08:39 AM, Hannes Reinecke wrote:
> Hi all,
> 
> here's a thing which came up recently on the SCSI ml:
> 
> How should we handle devices with _several_ possible drivers?
> 
> Case in point are some HBAs, for which the plan is to have distinct
> drivers for either target or initiator mode.
> 
> There are several options how this could be handled:
> - Have a default driver, and manually rebind the 'real' driver once the
> configuration could be read and acted upon.
> That would have the drawback that we do an initialisation in the 'wrong'
> mode, and later have to switch configuration. Which at best will 'just'
> induce some pointless initialisation, at worst confuse attached devices
> to no end. Not to mention introducing lots of interesting races with
> udev and systemd
> - Inhibit default binding, and load the correct drivers after the
> configuration has been read.
> Would be a deviation from the original scenario (where the driver are
> bound/loaded as soon as the device becomes available). Has the drawback
> that one would need to inhibit automatic bindings on the _bus_ level (ie
> PCI), which will be even more interesting.
> - Add some bus-specific (or even general?) device configuration syntax,
> which would allow to pass in the required configuration from the
> command-line (or reasonably early, anyway). For which we need a syntax
> first, anyway. And need to figure out if we can have an abstract syntax
> or would need to have a bus- (or even driver-) specific configuration.
> 
> So no easy way out here, and it might be worth having input from other
> parties. And some might have similar problems (just thinking of
> usb-modeswitch ...), so there might be some synergies to be had.
> 
> Cheers,
> 
> Hannes
> 

I would very much like to be part of this discussion, as these drivers
are are often iscsi, and that's where I live. :)

Along similar lines, I'll suggest (in a separate email) discussing
iSCSI-boot.
-- 
Lee Duncan