Re: [RFC PATCH v2 0/7] Make core VMA operations internal and testable

[Lorenzo Stoakes <lorenzo.stoakes@oracle.com>]

On Fri, Jun 28, 2024 at 03:35:21PM GMT, Lorenzo Stoakes wrote:
> There are a number of "core" VMA manipulation functions implemented in
> mm/mmap.c, notably those concerning VMA merging, splitting, modifying,
> expanding and shrinking, which logically don't belong there.
>
> More importantly this functionality represents an internal implementation
> detail of memory management and should not be exposed outside of mm/
> itself.
>
> This patch series isolates core VMA manipulation functionality into its own
> file, mm/vma.c, and provides an API to the rest of the mm code in mm/vma.h.
>
> Importantly, it also carefully implements mm/vma_internal.h, which
> specifies which headers need to be imported by vma.c, leading to the very
> useful property that vma.c depends only on mm/vma.h and mm/vma_internal.h.
>
> This is useful, because we can then re-implement vma_internal.h in
> userland, stubbing out and adding shims for kernel mechanisms as required,
> and then can directly and very easily unit test internal VMA functionality.
>
> This patch series takes advantage of existing shim logic and full userland
> maple tree support contained in tools/testing/radix-tree/ and
> tools/include/linux/, separating out shared components of the radix tree
> implementation to provide this testing.
>
> Kernel functionality is stubbed and shimmed as needed in tools/testing/vma/
> which contains a fully functional userland vma_internal.h file and which
> imports mm/vma.c and mm/vma.h to be directly tested from userland.
>
> A simple, skeleton testing implementation is provided in
> tools/testing/vma/vma.c as a proof-of-concept, asserting that simple VMA
> merge, modify (testing split), expand and shrink functionality work
> correctly.
>
> v2:
> * Reword commit messages.
> * Replace vma_expand() / vma_shrink() wrappers with relocate_vma().
> * Make move_page_tables() internal too.
> * Have internal.h import vma.h.
> * Use header guards to more cleanly implement userland testing code.
> * Rename main.c to vma.c.
> * Update mm/vma_internal.h to have fewer superfluous comments.
> * Rework testing logic so we count test failures, and output test results.
> * Correct some SPDX license prefixes.
> * Make VM_xxx_ON() debug asserts forward to xxx_ON() macros.
> * Update VMA tests to correctly free memory, and re-enable ASAN leak
>   detection.
>
> v1:
> https://lore.kernel.org/all/cover.1719481836.git.lstoakes@gmail.com/
>
> Lorenzo Stoakes (7):
>   userfaultfd: move core VMA manipulation logic to mm/userfaultfd.c
>   mm: move vma_modify() and helpers to internal header
>   mm: move vma_shrink(), vma_expand() to internal header
>   mm: move internal core VMA manipulation functions to own file
>   MAINTAINERS: Add entry for new VMA files
>   tools: separate out shared radix-tree components
>   tools: add skeleton code for userland testing of VMA logic
>

As this is stabilising and there seems to be no major objections, the next
re-spin will be sent un-RFC'd.

I have a few small fixes on the userland testing code I will include, in
addition to the stuff commented on here.


>  MAINTAINERS                                   |   14 +
>  fs/exec.c                                     |   68 +-
>  fs/userfaultfd.c                              |  160 +-
>  include/linux/atomic.h                        |    2 +-
>  include/linux/mm.h                            |  112 +-
>  include/linux/mmzone.h                        |    3 +-
>  include/linux/userfaultfd_k.h                 |   19 +
>  mm/Makefile                                   |    2 +-
>  mm/internal.h                                 |  167 +-
>  mm/mmap.c                                     | 2070 ++---------------
>  mm/mmu_notifier.c                             |    2 +
>  mm/userfaultfd.c                              |  168 ++
>  mm/vma.c                                      | 1766 ++++++++++++++
>  mm/vma.h                                      |  362 +++
>  mm/vma_internal.h                             |   52 +
>  tools/testing/radix-tree/Makefile             |   68 +-
>  tools/testing/radix-tree/maple.c              |   14 +-
>  tools/testing/radix-tree/xarray.c             |    9 +-
>  tools/testing/shared/autoconf.h               |    2 +
>  tools/testing/{radix-tree => shared}/bitmap.c |    0
>  tools/testing/{radix-tree => shared}/linux.c  |    0
>  .../{radix-tree => shared}/linux/bug.h        |    0
>  .../{radix-tree => shared}/linux/cpu.h        |    0
>  .../{radix-tree => shared}/linux/idr.h        |    0
>  .../{radix-tree => shared}/linux/init.h       |    0
>  .../{radix-tree => shared}/linux/kconfig.h    |    0
>  .../{radix-tree => shared}/linux/kernel.h     |    0
>  .../{radix-tree => shared}/linux/kmemleak.h   |    0
>  .../{radix-tree => shared}/linux/local_lock.h |    0
>  .../{radix-tree => shared}/linux/lockdep.h    |    0
>  .../{radix-tree => shared}/linux/maple_tree.h |    0
>  .../{radix-tree => shared}/linux/percpu.h     |    0
>  .../{radix-tree => shared}/linux/preempt.h    |    0
>  .../{radix-tree => shared}/linux/radix-tree.h |    0
>  .../{radix-tree => shared}/linux/rcupdate.h   |    0
>  .../{radix-tree => shared}/linux/xarray.h     |    0
>  tools/testing/shared/maple-shared.h           |    9 +
>  tools/testing/shared/maple-shim.c             |    7 +
>  tools/testing/shared/shared.h                 |   34 +
>  tools/testing/shared/shared.mk                |   68 +
>  .../testing/shared/trace/events/maple_tree.h  |    5 +
>  tools/testing/shared/xarray-shared.c          |    5 +
>  tools/testing/shared/xarray-shared.h          |    4 +
>  tools/testing/vma/.gitignore                  |    6 +
>  tools/testing/vma/Makefile                    |   15 +
>  tools/testing/vma/errors.txt                  |    0
>  tools/testing/vma/generated/autoconf.h        |    2 +
>  tools/testing/vma/linux/atomic.h              |   12 +
>  tools/testing/vma/linux/mmzone.h              |   38 +
>  tools/testing/vma/vma.c                       |  207 ++
>  tools/testing/vma/vma_internal.h              |  882 +++++++
>  51 files changed, 3910 insertions(+), 2444 deletions(-)
>  create mode 100644 mm/vma.c
>  create mode 100644 mm/vma.h
>  create mode 100644 mm/vma_internal.h
>  create mode 100644 tools/testing/shared/autoconf.h
>  rename tools/testing/{radix-tree => shared}/bitmap.c (100%)
>  rename tools/testing/{radix-tree => shared}/linux.c (100%)
>  rename tools/testing/{radix-tree => shared}/linux/bug.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/cpu.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/idr.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/init.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/kconfig.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/kernel.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/kmemleak.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/local_lock.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/lockdep.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/maple_tree.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/percpu.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/preempt.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/radix-tree.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/rcupdate.h (100%)
>  rename tools/testing/{radix-tree => shared}/linux/xarray.h (100%)
>  create mode 100644 tools/testing/shared/maple-shared.h
>  create mode 100644 tools/testing/shared/maple-shim.c
>  create mode 100644 tools/testing/shared/shared.h
>  create mode 100644 tools/testing/shared/shared.mk
>  create mode 100644 tools/testing/shared/trace/events/maple_tree.h
>  create mode 100644 tools/testing/shared/xarray-shared.c
>  create mode 100644 tools/testing/shared/xarray-shared.h
>  create mode 100644 tools/testing/vma/.gitignore
>  create mode 100644 tools/testing/vma/Makefile
>  create mode 100644 tools/testing/vma/errors.txt
>  create mode 100644 tools/testing/vma/generated/autoconf.h
>  create mode 100644 tools/testing/vma/linux/atomic.h
>  create mode 100644 tools/testing/vma/linux/mmzone.h
>  create mode 100644 tools/testing/vma/vma.c
>  create mode 100644 tools/testing/vma/vma_internal.h
>
> --
> 2.45.1