Re: [RFC PATCH 7/7] tools: add skeleton code for userland testing of VMA logic

[Lorenzo Stoakes <lstoakes@gmail.com>]

On Thu, Jun 27, 2024 at 02:25:36PM -0400, Liam R. Howlett wrote:
> * Kees Cook <kees@kernel.org> [240627 12:58]:
> > On Thu, Jun 27, 2024 at 11:39:32AM +0100, Lorenzo Stoakes wrote:
> > > Establish a new userland VMA unit testing implementation under
> > > tools/testing which utilises existing logic providing maple tree support in
> > > userland utilising the now-shared code previously exclusive to radix tree
> > > testing.
> > >
> > > This provides fundamental VMA operations whose API is defined in mm/vma.h,
> > > while stubbing out superfluous functionality.
> > >
> > > This exists as a proof-of-concept, with the test implementation functional
> > > and sufficient to allow userland compilation of vma.c, but containing only
> > > cursory tests to demonstrate basic functionality.
> >
> > Interesting! Why do you want to have this in userspace instead of just
> > wiring up what you have here to KUnit so testing can be performed by
> > existing CI systems that are running all the KUnit tests?
>
> The primary reason we did the maple tree testing in userspace was for
> speed of testing.  We don't need to build the kernel, but a subset of
> APIs.  Debugging problems is also much quicker since we can instrument
> and rebuild, iterate down faster.  Tracing every call to the maple tree
> on boot alone is massive.
>
> It is also difficult to verify the vma correctness without exposing APIs
> we don't want exported (or, I guess, parse proc files..).  On my side, I
> have a module for testing the overall interface while I have more tests
> on the userspace side that poke and prod on internal states, and
> userspace rcu testing is possible.  I expect the same issues on the vma
> side.
>
> Adding tests can also be made very efficient with tracepoints dumping
> something to add to an array, for example.
>
> Finally, you have ultimate control on what other functions return (or
> do) - so you can fail allocations to test error paths, for example.  Or
> set the external function to fail after N allocations.  This comes in
> handy when a syzbot reports a failed allocation at line X caused a
> crash.
>
> This has worked out phenomenally on the maple tree side.  I've been able
> to record boot failures and import them, syzbot tests, and fuzzer tests.
> The result is a huge list of tests that allowed me to rewrite my node
> replacement algorithm and have it just work, once it passed the
> collected tests.
>
> I haven't used kunit as much as I have userspace testing, so I cannot
> say if all of these points are not possible, but I didn't see a way to
> test races like I do with rcu in userspace.
>
> Thanks,
> Liam

Liam's response is excellent, and obviously I agree
wholeheartedly. Additionally, I'm not really experienced with kunit, but
surely it's implemented as a kernel module somehow? If so, these interfaces
are largely not exported so it wouldn't be functional as a unit test.

And also as Liam says, it'd be very difficult to test this stuff _in_ the
kernel without unwanted side-effects triggering and it'd be very difficult
to isolate or mock components we don't want to play a role (for instance -
rlimits that we might not be able to control).

But overall (again as Liam says) the performance benefit, flexibility and
ability to recreate things at a whim are huge.

And the fact maple tree (which forms a HUGE part of these VMA operations)
and related radix tree and other shims/stubs already exist means that it
wasn't anywhere near as huge a task to implement this as it would be
otherwise.