Re: [Ksummit-discuss] [TECH TOPIC] Netlink engine issues, and ways to fix those

[Alexei Starovoitov <alexei.starovoitov@gmail.com>]

On Fri, Sep 2, 2016 at 10:20 PM, Andrei Vagin <avagin@virtuozzo.com> wrote:
> The netlink interface proved itself as a great way to perform
> descriptor-based kernel/userspace communication. It is especially useful
> for cases involving a big amount of data to transfer. The netlink
> communication protocol is simple and elegant; it also allows to extend
> the message format without breaking backward compatibility.
>
> One big problem of netlink is credentials. When a user-space process is
> opening a new file descriptor, kernel saves the opener's credentials to
> f_cred field of the file struct. After that, every access to that fd are
> checked against the saved credentials. In essence, this allows for a
> process to open a file descriptor as root and then drop capabilities.
> With netlink socket, it is not possible to implement this access scheme.
>
> Currently netlink is widely used in the network subsystem, but there are
> also a few users outside of networking, such as audit and taskstats.
> Developers who used netlink for anything except the networking know
> there are some issues. For example, taskstats code has broken user and
> pid namespace support.
>
> Another potential user of netlink socket is task_diag, a faster
> /proc/PID-like interface proposed some time ago
> (https://lkml.org/lkml/2015/7/6/142). It makes sense to use the netlink
> interface for it, too, but the whole feature is currently blocked by the
> netlink discussion.
>
> A few months ago Andy Lutomirski suggested to rework the netlink
> interface in order to solve the known issues. We suggest discussing his
> idea:
>
> ----- snip --- snip --- snip -----
> (taken from http://lists.openwall.net/netdev/2016/05/05/51)
>
> The tl;dr is that Andrey wants to add an interface to ask a pidns some
> questions, and netlink looks natural, except that using netlink sockets
> to interrogate a pidns seems rather problematic.  I would also love to
> see a decent interface for interrogating user namespaces, and again,
> netlink would be great, except that it's a socket and makes no sense in
> this context.
>
> Netlink had, and possibly still has, tons of serious security bugs
> involving code checking send() callers' creds.  I found and fixed a few
> a couple years ago.  To reiterate once again, send() CANNOT use caller
> creds safely.  (I feel like I say this once every few weeks. It's
> getting old.)
>
> I realize that it's convenient to use a socket as a context to keep
> state between syscalls, but it has some annoying side effects:
>
>  - It makes people want to rely on send()'s caller's creds.
>  - It's miserable in combination with seccomp.
>  - It doesn't play nicely with namespaces.
>  - It makes me wonder why things like task_diag, which have nothing
>    to do with networking, seem to get tangled up with networking.
>
>
> Would it be worth considering adding a parallel interface, using it for
> new things, and slowly migrating old use cases over?
>
> int issue_kernel_command(int ns, int command, const struct iovec *iov, int iovcnt, int flags);
>
> ns is an actual namespace fd or:
>
> KERNEL_COMMAND_CURRENT_NETNS
> KERNEL_COMMAND_CURRENT_PIDNS
> etc, or a special one:
> KERNEL_COMMAND_GLOBAL.  KERNEL_COMMAND_GLOBAL can't be used in a
> non-root namespace.
>
> KERNEL_COMMAND_GLOBAL works even for namespaced things, if the
> relevant current ns is the init namespace.  (This feature is optional,
> but it would allow gradually namespacing global things.)
>
> command is an enumerated command.  Each command implies a namespace
> type, and, if you feed this thing the wrong namespace type, you get
> EINVAL.  The high bit of command indicates whether it's read-only
> command.
>
> iov gives a command in the format expected, which, for the most part,
> would be a netlink message.
>
> The return value is an fd that you can call read/readv on to read the
> response.  It's not a socket (or at least you can't do normal socket
> operations on it if it is a socket behind the scenes).  The
> implementation of read() promises *not* to look at caller creds.  The
> returned fd is unconditionally cloexec -- it's 2016 already.  Sheesh.
>
> When you've read all the data, all you can do is close the fd.  You
> can't issue another command on the same fd.  You also can't call write()
> or send() on the fd unless someone has a good reason why you should be
> able to and why it's safe.  You can't issue another command on the same
> fd.
>
> I imagine that the implementation could re-use a bunch of netlink code
> under the hood.

I'm very interested in this discussion.
Adding few folks as well.