zero copy IO project

zero copy IO project

[Fabio Riccardi]

Hi,

after having analyzed a bit the various souces of inefficiency in Linux
(wrt a high end server), and having discarded what other people already
seem to be working on,  I'd like to enroll myself into a zero copy IO
project, much like that of UVM, genie, io-lite, fbufs, etc.

Is anybody aready working on this? Does anybody have ideas about it?
Anybody interested in a discussion of pros and cons of such an
architectural change to Linux?

TIA, ciao,

 - Fabio


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: zero copy IO project

[Rik van Riel]

On Mon, 4 Sep 2000, Fabio Riccardi wrote:

> after having analyzed a bit the various souces of inefficiency
> in Linux (wrt a high end server), and having discarded what
> other people already seem to be working on, I'd like to enroll
> myself into a zero copy IO project, much like that of UVM,
> genie, io-lite, fbufs, etc.
> 
> Is anybody aready working on this? Does anybody have ideas about
> it? Anybody interested in a discussion of pros and cons of such
> an architectural change to Linux?

The project (and data structure used) is called KIOBUF.

IIRC Stephen Tweedie and Ben LaHaise are working on it
and it will be a more generic zero-copy IO infrastructure
than io-lite and others.

Ben LaHaise made some documentation (or at least, some
slides) on KIOBUFs for the Ottawa Linux Symposium. You may
be able to get some documentation from him, and you can go
to the OLS ftp site to download an mp3 of the VM lecture...

Also, I'm sure they must have some TODO items for you ;))
(if you're interested in helping out)

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: zero copy IO project

[Ben LaHaise]

On Mon, 4 Sep 2000, Rik van Riel wrote:

> On Mon, 4 Sep 2000, Fabio Riccardi wrote:
...
> > Is anybody aready working on this? Does anybody have ideas about
> > it? Anybody interested in a discussion of pros and cons of such
> > an architectural change to Linux?
> 
> The project (and data structure used) is called KIOBUF.
> 
> IIRC Stephen Tweedie and Ben LaHaise are working on it
> and it will be a more generic zero-copy IO infrastructure
> than io-lite and others.
...

First off, the current discussion on linux-kernel seems to be hitting upon
a lot of the relevant points, at least related to the network stack.

In terms of actual ToDos at the moment, most of the work is in the design
of kiobuf based APIs.  The rw_kiovec operation in my async io patches is
along the lines of what we want to do, but I need to write up actual docs
describing how it should be used and what the ideas are behind it (along
with going back to Stephen and making sure it fits in with his
ideas ;-).  A number of infrastructure patches from Stephen need to be
merged (probably 2.5 stuff), and then some of the more interesting things
like the kick-ass pipe code can, too.

Hmmm, there is the fewer copy code for packet fragments (read: NFS) that
needs to be kiobufified.  And TUX's zero copy TX code should accept
kiobufs.  Stephen mentioned a kiovec container file that needs to be
written.  The block layer rototiling is being thought about by a number of
people (SGI's XFS has a number of good ideas for the filesystem space that
will be useful)...  And the list goes on...

What exactly do you wish to accomplish zero copy io for?  The kiovec
container is probably the most useful part of the plan for generic zero
copy io, since it allows you to have a handle for a buffer in kernel space
that does not have to cross over into user space, but that can have
relevant parts of data added to it.  Ie, a userspace web server could
write() the header for an HTTP request into the container, then make use
of sendfile operations to transmit it and then the data directly from the
file onto the wire.

This isn't making much of a scratch into the ideas floating around, but
given some feedback perhaps we can narrow things down to a more useful
scope. =)  Cheers,

		-ben

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: zero copy IO project

[Fabio Riccardi]

Hi, thanks for the pointers & explainations!

What I want is a server capable of handling high bandwidth communication and
the kiobuf mechanisms seem to be able to do the right thing, provided that
one rewrites the user applications accordingly...

If I understand correctly the kiobuf interface allows a user process to map a
piece of kernel memory in its own addressing space to use as an IO buffer.
What I originally had in mind was more something like netbsd's
UVM: _transparent_ zero-copy IO.

With the UVM  user applications just invokes the plain old fwrite (buff,
...) and the system grabs the buffer from the user space into kernel space
without the application noticing it (the original buffer becomes TCOW in the
application space).

Measurements (Brustoloni-Genie) show that explicit interfaces don't offer
significative performance advantages wrt emulated copy, while the latter
allows to run legacy (apache...) software without modification. Moreover it
keeps the number of OS interfaces to a bareable minimum.

I understand that a UVM like thing would imply major changes in the
VM architecture, since a page loanout mechanism has to be added to allow user
applications to export pieces of their addressing space to the kernel or to
some other user app.

Thoughts?

 - Fabio

Ben LaHaise wrote:

> On Mon, 4 Sep 2000, Rik van Riel wrote:
>
> > On Mon, 4 Sep 2000, Fabio Riccardi wrote:
> ...
> > > Is anybody aready working on this? Does anybody have ideas about
> > > it? Anybody interested in a discussion of pros and cons of such
> > > an architectural change to Linux?
> >
> > The project (and data structure used) is called KIOBUF.
> >
> > IIRC Stephen Tweedie and Ben LaHaise are working on it
> > and it will be a more generic zero-copy IO infrastructure
> > than io-lite and others.
> ...
>
> First off, the current discussion on linux-kernel seems to be hitting upon
> a lot of the relevant points, at least related to the network stack.
>
> In terms of actual ToDos at the moment, most of the work is in the design
> of kiobuf based APIs.  The rw_kiovec operation in my async io patches is
> along the lines of what we want to do, but I need to write up actual docs
> describing how it should be used and what the ideas are behind it (along
> with going back to Stephen and making sure it fits in with his
> ideas ;-).  A number of infrastructure patches from Stephen need to be
> merged (probably 2.5 stuff), and then some of the more interesting things
> like the kick-ass pipe code can, too.
>
> Hmmm, there is the fewer copy code for packet fragments (read: NFS) that
> needs to be kiobufified.  And TUX's zero copy TX code should accept
> kiobufs.  Stephen mentioned a kiovec container file that needs to be
> written.  The block layer rototiling is being thought about by a number of
> people (SGI's XFS has a number of good ideas for the filesystem space that
> will be useful)...  And the list goes on...
>
> What exactly do you wish to accomplish zero copy io for?  The kiovec
> container is probably the most useful part of the plan for generic zero
> copy io, since it allows you to have a handle for a buffer in kernel space
> that does not have to cross over into user space, but that can have
> relevant parts of data added to it.  Ie, a userspace web server could
> write() the header for an HTTP request into the container, then make use
> of sendfile operations to transmit it and then the data directly from the
> file onto the wire.
>
> This isn't making much of a scratch into the ideas floating around, but
> given some feedback perhaps we can narrow things down to a more useful
> scope. =)  Cheers,
>
>                 -ben

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: zero copy IO project

[Ben LaHaise]

On Mon, 4 Sep 2000, Fabio Riccardi wrote:

> Hi, thanks for the pointers & explainations!
> 
> What I want is a server capable of handling high bandwidth communication and
> the kiobuf mechanisms seem to be able to do the right thing, provided that
> one rewrites the user applications accordingly...
> 
> If I understand correctly the kiobuf interface allows a user process to map a
> piece of kernel memory in its own addressing space to use as an IO buffer.
> What I originally had in mind was more something like netbsd's
> UVM: _transparent_ zero-copy IO.

> With the UVM  user applications just invokes the plain old fwrite (buff,
> ...) and the system grabs the buffer from the user space into kernel space
> without the application noticing it (the original buffer becomes TCOW in the
> application space).

Anything that requires playing VM tricks is not something you'll find a
great deal of support for amongst developers -- see the posting Linus made
against exactly this.

It comes down to complexity and the amount of gain in generic
applications.  Take apache for example.  Enabling "zero copy" through VM
tricks will buy you no benefit when it comes to the http header sent out
on a request.  But the act of transmitting a file is already well handled
by the sendfile() model.

Fwiw, there are lots of libc optimisations that are worth *more* than
"zero copy" for typical applications.  Like pre-linking libraries.  stdio
could make use of mmap for fread.

		-ben

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: zero copy IO project

[Fabio Riccardi]

ok, I can see the point, I'll stick to the current framework then. I will postpone
anything radical to when I will have a better understanding of Linux' internals
and philosophy.

Two questions:

How can I contribute? What would interest me in the short term is to build an
optimized web server, using all the possible tricks available in the current
architecture, my company needs that and is happy that I spend my time to get
things going

Where can I find some docs? Rick mentioned that you have some design papers on the
kiobuf architecture. What is available about the sendfile interface?

 - Fabio

Ben LaHaise wrote:

> Anything that requires playing VM tricks is not something you'll find a
> great deal of support for amongst developers -- see the posting Linus made
> against exactly this.
>
> It comes down to complexity and the amount of gain in generic
> applications.  Take apache for example.  Enabling "zero copy" through VM
> tricks will buy you no benefit when it comes to the http header sent out
> on a request.  But the act of transmitting a file is already well handled
> by the sendfile() model.
>
> Fwiw, there are lots of libc optimisations that are worth *more* than
> "zero copy" for typical applications.  Like pre-linking libraries.  stdio
> could make use of mmap for fread.
>
>                 -ben
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux.eu.org/Linux-MM/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/