Re: [MAINTAINERS/KERNEL SUMMIT] Trust and maintenance of file systems

[Dave Chinner <david@fromorbit.com>]

On Wed, Aug 30, 2023 at 04:07:39PM +0200, Christoph Hellwig wrote:
> Hi all,
> 
> we have a lot of on-disk file system drivers in Linux, which I consider
> a good thing as it allows a lot of interoperability.  At the same time
> maintaining them is a burden, and there is a lot expectation on how
> they are maintained.
> 
> Part 1: untrusted file systems
> 
> There has been a lot of syzbot fuzzing using generated file system
> images, which I again consider a very good thing as syzbot is good
> a finding bugs.  Unfortunately it also finds a lot of bugs that no
> one is interested in fixing.   The reason for that is that file system
> maintainers only consider a tiny subset of the file system drivers,
> and for some of them a subset of the format options to be trusted vs
> untrusted input.  It thus is not just a waste of time for syzbot itself,
> but even more so for the maintainers to report fuzzing bugs in other
> implementations.
> 
> What can we do to only mark certain file systems (and format options)
> as trusted on untrusted input and remove a lot of the current tension
> and make everyone work more efficiently?  Note that this isn't even
> getting into really trusted on-disk formats, which is a security
> discussion on it's own, but just into formats where the maintainers
> are interested in dealing with fuzzed images.

I think this completely misses the point of contention of the larger
syzbot vs filesystem discussion: the assertion that "testing via
syzbot means the subsystem is secure" where "secure" means "can be
used safely for operations that involve trust model violations".

Fundamentally, syzbot does nothing to actually validate the
filesystem is "secure". Fuzzing can only find existing bugs by
simulating an attacker, but it does nothing to address the
underlying issues that allow that attack channel to exist.

All "syzbot doesn't find bugs" means is that -random bit
manipulation- of the filesystem's metadata *hasn't found issues*.

Even though the XFS V5 format is pretty robust against random bit
manipulation, it's certainly not invulnerable and cannot detect
coordinated, multiple object corruptions (cross linked blocks,
cycles in trees, etc) without a full filesystem scan. These sorts of
corruptions are almost never going to be exercised by random bit
manipulation fuzzers like syzbot, but they are exactly the sort of
thing a malicious attacker with some knowledge of how the filesystem
works would look at....

Let's also address the elephant in the room: malicious attackers
don't need to to exploit flaws in the filesystem metadata structure
to trojan an unsuspecting user.

i.e. We cannot detect changes to metadata that are within valid
bounds and may be security sensitive - things like UIDs and GIDs,
inode permissions, inode flags, link counts, symbolic links, etc. We
also can't determine if the file data is unchanged, so it's easy to
trojan the contents of an executable file on a filesystem image.

IOWs, all the attacker needs to do is trojan an installer script on
an application or device driver disk/image, and the user will run it
as root themselves....

There are whole classes of malicious modifications that syzbot
doesn't exercise and we cannot detect nor defend against at the
filesystem level without changing the trust model the filesystem
operates under. And if we change the trust model, we are now talking
about on-disk format changes and using robust crypto for all the
data and metadata in the filesystem. At which point, we may as well
require a full disk encryption layer via dm-crypt....

If we say "filesystem is secure against untrusted input" then that
is what users will expect us to provide. It will also means that
every bug that syzbot might find will result in a high priority CVE,
because any issue arising from untrusted input is a now a major
system security issue.

As such, I just don't see how "tested with syzbot" equates with
"safe for untrusted use cases" whilst also reducing the impact of
the problems that syzbot finds and reports...

> Part 2: unmaintained file systems
> 
> A lot of our file system drivers are either de facto or formally
> unmaintained.  If we want to move the kernel forward by finishing
> API transitions (new mount API, buffer_head removal for the I/O path,
> ->writepage removal, etc) these file systems need to change as well
> and need some kind of testing.  The easiest way forward would be
> to remove everything that is not fully maintained, but that would
> remove a lot of useful features.

Linus has explicitly NACKed that approach.

https://lore.kernel.org/linux-fsdevel/CAHk-=wg7DSNsHY6tWc=WLeqDBYtXges_12fFk1c+-No+fZ0xYQ@mail.gmail.com/

Which is a problem, because historically we've taken code into
the kernel without requiring a maintainer, or the people who
maintained the code have moved on, yet we don't have a policy for
removing code that is slowly bit-rotting to uselessness.

> E.g. the hfsplus driver is unmaintained despite collecting odd fixes.
> It collects odd fixes because it is really useful for interoperating
> with MacOS and it would be a pity to remove it.  At the same time
> it is impossible to test changes to hfsplus sanely as there is no
> mkfs.hfsplus or fsck.hfsplus available for Linux.  We used to have
> one that was ported from the open source Darwin code drops, and
> I managed to get xfstests to run on hfsplus with them, but this
> old version doesn't compile on any modern Linux distribution and
> new versions of the code aren't trivially portable to Linux.
> 
> Do we have volunteers with old enough distros that we can list as
> testers for this code?  Do we have any other way to proceed?
>
> If we don't, are we just going to untested API changes to these
> code bases, or keep the old APIs around forever?

We do slowly remove device drivers and platforms as the hardware,
developers and users disappear. We do also just change driver APIs
in device drivers for hardware that no-one is actually able to test.
The assumption is that if it gets broken during API changes,
someone who needs it to work will fix it and send patches.

That seems to be the historical model for removing unused/obsolete
code from the kernel, so why should we treat unmaintained/obsolete
filesystems any differently?  i.e. Just change the API, mark it
CONFIG_BROKEN until someone comes along and starts fixing it...

-Dave.
-- 
Dave Chinner
david@fromorbit.com