Re: [LSF/MM/BPF TOPIC] Mapping text with large folios

[Barry Song <21cnbao@gmail.com>]

On Thu, Mar 20, 2025 at 4:38 AM Ryan Roberts <ryan.roberts@arm.com> wrote:
>
> Hi All,
>
> I know this is very last minute, but I was hoping that it might be possible to
> squeeze in a session to discuss the following?
>
> Summary/Background:
>
> On arm64, physically contiguous and naturally aligned regions can take advantage
> of contpte mappings (e.g. 64 KB) to reduce iTLB pressure. However, for file
> regions containing text, current readahead behaviour often yields small,
> misaligned folios, preventing this optimization. This proposal introduces a
> special-case path for executable mappings, performing synchronous reads of an
> architecture-chosen size into large folios (64 KB on arm64). Early performance
> tests on real-world workloads (e.g. nginx, redis, kernel compilation) show ~2-9%
> gains.
>
> I’ve previously posted attempts to enable this performance improvement ([1],
> [2]), but there were objections and conversation fizzled out. Now that I have
> more compelling performance data, I’m hoping there is now stronger
> justification, and we can find a path forwards.
>
> What I’d Like to Cover:
>
>  - Describe how text memory should ideally be mapped and why it benefits
>    performance.
>
>  - Brief review of performance data.
>
>  - Discuss options for the best way to encourage text into large folios:
>      - Let the architecture request a preferred size
>      - Extend VMA attributes to include preferred THP size hint

We might need this for a couple of other cases.

1. The native heap—for example, a native heap like jemalloc—can configure
the base "granularity" and then use MADV_DONTNEED/FREE at that granularity
to manage memory. Currently, the default granularity is PAGE_SIZE, which can
lead to excessive folio splitting. For instance, if we set jemalloc's
granularity to
16KB while sysfs supports 16KB, 32KB, 64KB, etc., splitting can still occur.
Therefore, in some cases, I believe the kernel should be aware of how
userspace is managing memory.

2. Java heap GC compaction -  userfaultfd_move() things.
I am considering adding support for batched PTE/folios moves in
userfaultfd_move().
If sysfs enables 16KB, 32KB, 64KB, 128KB, etc., but the userspace Java
heap moves
memory at a 16KB granularity, it could lead to excessive folio splitting.

For exec, it seems we need a userspace-transparent approach. Asking each
application to modify its code to madvise the kernel on its preferred exec folio
size seems cumbersome.

I mean, we could whitelist all execs by default unless an application explicitly
requests to disable it?

>      - Provide a sysfs knob
>      - Plug into the “mapping min folio order” infrastructure
>      - Other approaches?
>
> [1] https://lore.kernel.org/all/20240215154059.2863126-1-ryan.roberts@arm.com/
> [2] https://lore.kernel.org/all/20240717071257.4141363-1-ryan.roberts@arm.com/
>
> Thanks,
> Ryan

Thanks
Barry