[PATCH hotfix 6.12 v3 0/5] fix error handling in mmap_region() and refactor (hotfixes)

[Lorenzo Stoakes <lorenzo.stoakes@oracle.com>]

NOTE: This should be applied on mm-hotfixes-unstable in Andrew's mm tree as
      it relies on other pending hotfixes.

The mmap_region() function is somewhat terrifying, with spaghetti-like
control flow and numerous means by which issues can arise and incomplete
state, memory leaks and other unpleasantness can occur.

A large amount of the complexity arises from trying to handle errors late
in the process of mapping a VMA, which forms the basis of recently observed
issues with resource leaks and observable inconsistent state.

This series goes to great lengths to simplify how mmap_region() works and
to avoid unwinding errors late on in the process of setting up the VMA for
the new mapping, and equally avoids such operations occurring while the VMA
is in an inconsistent state.

The patches in this series comprise the minimal changes required to resolve
existing issues in mmap_region() error handling, in order that they can be
hotfixed and backported. There is additionally a follow up series which
goes further, separated out from the v1 series and sent and updated
separately.

v3:
* Added correct handling for arm64 MTE which was otherwise broken, as
  reported by Mark Brown.

v2:
* Marked first 4 patches as hotfixes, the rest as not.
* Improved comment in vma_close() as per Vlastimil.
* Updated hole byte count as per Jann.
* Updated comment in map_deny_write_exec() as per Jann.
* Dropped unnecessary vma_iter_free() as per Vlastimil, Liam.
* Corrected vms_abort_munmap_vmas() mistaken assumption about nr_pages as
  per Vlastimil.
* Changed order of initial checks in mmap_region() to avoid user-visible
  side effects as per Vlastimil, Liam.
* Corrected silly incorrect use of vma field.
* Various style corrects as per Liam.
* Fix horrid mistake with merge VMA, reworked the logic to avoid that
  nonsense altogether.
* Add fields to map state rather than using vmg fields to avoid
  confusion/risk of vmg state changing breaking things.
* Replaced last commit removing merge retry with one that retries the
  merge, only sanely.
https://lore.kernel.org/all/cover.1729715266.git.lorenzo.stoakes@oracle.com/

v1:
https://lore.kernel.org/all/cover.1729628198.git.lorenzo.stoakes@oracle.com/


Lorenzo Stoakes (5):
  mm: avoid unsafe VMA hook invocation when error arises on mmap hook
  mm: unconditionally close VMAs on error
  mm: refactor map_deny_write_exec()
  mm: refactor arch_validate_flags() and arm64 MTE handling
  mm: resolve faulty mmap_region() error path behaviour

 arch/arm64/include/asm/mman.h |  29 ++++++--
 arch/sparc/include/asm/mman.h |   5 +-
 include/linux/mman.h          |  23 ++++--
 mm/internal.h                 |  45 ++++++++++++
 mm/mmap.c                     | 128 ++++++++++++++++++----------------
 mm/mprotect.c                 |   4 +-
 mm/nommu.c                    |   7 +-
 mm/shmem.c                    |   3 -
 mm/vma.c                      |  14 ++--
 mm/vma.h                      |   6 +-
 10 files changed, 173 insertions(+), 91 deletions(-)

--
2.47.0