[Ksummit-discuss] [TOPIC] userspace infrastructure services

[Ksummit-discuss] [TOPIC] userspace infrastructure services

[Stephen Hemminger]

I expected someone else to bring this topic up...

One thing that is happening is that there is lots of activity in moving core features
of the kernel into userspace (networking, storage, security). I don't want to get into
an argument over whether that is good or bad; but the problem from the kernel
point of view is that the existing API's are getting stretched to the boundary.

The area I am most familiar with is the DPDK which has to have its own UIO drivers
to work in all environments (get device into userspace). And then has to have its
own drivers to simulate network device (put device back into kernel).  This leads
to unmanageable ABI and development technical debt. I have heard that same thing
is planned for storage as well.

The question is should the kernel accommodate this by absorbing/integrating/fixing
this; or just let them suffer?

Re: [Ksummit-discuss] [TOPIC] userspace infrastructure services

[Christoph Hellwig]

On Wed, Aug 12, 2015 at 11:15:52AM -0700, Stephen Hemminger wrote:
> I expected someone else to bring this topic up...
> 
> One thing that is happening is that there is lots of activity in moving core features
> of the kernel into userspace (networking, storage, security). I don't want to get into
> an argument over whether that is good or bad;

Which I think is the most important question of them all.  From all
the pain caused and the little gains I'm a believer it's mostly bad,
especially for cases like:

> The area I am most familiar with is the DPDK which has to have its own UIO drivers
> to work in all environments (get device into userspace). And then has to have its
> own drivers to simulate network device (put device back into kernel).  This leads
> to unmanageable ABI and development technical debt.

Which interacts on both sides.  In that case "let them suffer for their
stupidity" is the only valid answer.

For a case where it's more reasonable there might be a different answer.

Re: [Ksummit-discuss] [TOPIC] userspace infrastructure services

[James Bottomley]

On Wed, 2015-08-12 at 11:21 -0700, Christoph Hellwig wrote:
> On Wed, Aug 12, 2015 at 11:15:52AM -0700, Stephen Hemminger wrote:
> > I expected someone else to bring this topic up...
> > 
> > One thing that is happening is that there is lots of activity in moving core features
> > of the kernel into userspace (networking, storage, security). I don't want to get into
> > an argument over whether that is good or bad;
> 
> Which I think is the most important question of them all.  From all
> the pain caused and the little gains I'm a believer it's mostly bad,
> especially for cases like:
> 
> > The area I am most familiar with is the DPDK which has to have its own UIO drivers
> > to work in all environments (get device into userspace). And then has to have its
> > own drivers to simulate network device (put device back into kernel).  This leads
> > to unmanageable ABI and development technical debt.
> 
> Which interacts on both sides.  In that case "let them suffer for their
> stupidity" is the only valid answer.
> 
> For a case where it's more reasonable there might be a different answer.

What about something like CRIU:

http://www.criu.org/Main_Page

?

in-kernel checkpoint/restore had been proposed many times before and
always rejected as being hugely complex (most publicly at the Kernel
Summit in Cambridge, where Oren Laadan proposed it).  CRIU took a
completely different approach where we broke the problem down into
sections and extended kernel interfaces to export state where it wasn't
available.  The net result is that we now have checkpoint/restore (and
thus process migration) for Linux at a significant reduction in the
complexity over the original code and realistically not a huge technical
debt.

What I like to think CRIU shows us is that if we look hard enough, there
is a way to separate the problem up so that a userspace becomes workable
without a massive increase in the kernel ABI surface.

The big problem is finding the way to do the split.  You can tell if you
get it wrong because the userspace component has a huge amount of state
shared with the kernel and the ABI balloons to cope with this (cough,
DPDK, cough)... however, what's not clear is how to get it right in the
first place.

Perhaps we should look first at how CRIU got it right and then discuss
how we might use those same principles to fix other projects, like DPDK.

James

Re: [Ksummit-discuss] [TOPIC] userspace infrastructure services

[Christoph Lameter]

On Wed, 12 Aug 2015, James Bottomley wrote:

> Perhaps we should look first at how CRIU got it right and then discuss
> how we might use those same principles to fix other projects, like DPDK.

Well one thing is to look at the existing implementation of the
APIs that are already available in the IBverbs/RMDA and cover both
networking as well as storage. They already separate the control and the
data transfer areas. Kernel provides control but gets out of the way of
data transfers. Arguably this could be cleaner but here is an existing way
of doing this that we already support and that has a decade long history
of use in the industry.

This is not about minimal gains but being able to get more than
just a fraction of the possible performance out of high end devices.

Re: [Ksummit-discuss] [TOPIC] userspace infrastructure services

[Stephen Hemminger]

On Wed, 12 Aug 2015 11:21:03 -0700
Christoph Hellwig <hch@infradead.org> wrote:

> On Wed, Aug 12, 2015 at 11:15:52AM -0700, Stephen Hemminger wrote:
> > I expected someone else to bring this topic up...
> > 
> > One thing that is happening is that there is lots of activity in moving core features
> > of the kernel into userspace (networking, storage, security). I don't want to get into
> > an argument over whether that is good or bad;
> 
> Which I think is the most important question of them all.  From all
> the pain caused and the little gains I'm a believer it's mostly bad,
> especially for cases like:
> 
> > The area I am most familiar with is the DPDK which has to have its own UIO drivers
> > to work in all environments (get device into userspace). And then has to have its
> > own drivers to simulate network device (put device back into kernel).  This leads
> > to unmanageable ABI and development technical debt.
> 
> Which interacts on both sides.  In that case "let them suffer for their
> stupidity" is the only valid answer.
> 
> For a case where it's more reasonable there might be a different answer.

I think this is like earlier work with embedded. If the community does
not engage, then the pain only increases on both sides over time.
Overall, the only long term solution is something like:
 "Yes, I see what you want, but how you are doing is stupid.
  Let's work together and see if there is a better way to do this"
In most cases, the kernel has already exposed a bunch of API's that make
the whole effort possible (uio, tun/tap, ...); either we say
"ignore those API's, they are toys you are not supposed to actually
use for real products"
or be professional and make them into first class citizens.