Re: [LSF/MM/BPF TOPIC] Flexible orders for anonymous folios

[Yang Shi <shy828301@gmail.com>]

On Wed, Feb 22, 2023 at 8:01 PM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Wed, Feb 22, 2023 at 04:55:20PM -0700, Yu Zhao wrote:
> > Currently anonymous folios only support two orders: 0 and the PMD
> > order. Flexible orders for anonymous folios can resist both internal
> > or external fragmentations when the PMD order is too underutilized or
> > costly to allocate. Flexible orders can also leverage the TLB
> > coalescing feature, .e.g., order 3 for AMD and order 4 for ARM CPUs.
> >
> > Discussion points:
> > 1. The page fault path: determining the best order and the fallback policy.
> > 2. The reclaim path: detecting the utilization and the splitting policy.
> > 3. The refcount and the mapcount models, e.g., reuse the PMD-mapped
> > THP model or not.
> > 4. The splitting, and the collapsing if needed.
> > 5. Other paths: COW, GUP, madvise(), mprotect(), page migration, etc.
>
> 6. Swap out an entire folio instead of splitting it before swap.

Yeah, the THP swap should be extended to support arbitrary order.
Other than swap out, swap in large folios should be considered as well
at some point

> 7. At some point we're going to want Zi Yan's patches to split a folio
>    into arbitrary order folios instead of just to order-0.

I agree.

> 8. For file folios, folio->index % folio->nr_pages is 0.  Do we want
>    to maintain that invariant for anon folios?  It helps tile the
>    folios so we don't end up with say, order-0, order-4, order-2 folios
>    in a sequence.

I think so. The TLB has alignment requirements anyway.

> 9. How do we ensure that two page faults do not overwrite each others
>    folios, eg PF1 decides to allocate an order-0 folio at index 6 and
>    PF2 decides to allocate an order-2 folio at index 4?

Good question. I don't have a good answer right now, but anyway the
kernel should make sure all the PTEs in the region covered by the
folio should be none off the top of my head.

>
> Probably some other things left to decide.
>
> > Now it seems to be a good time to revisit. Yang Shi and I have been
> > looking at some details, trying to scope out the work that would be
> > required. We don't have anything definitive yet, but we should have
> > enough to share by the time of the LSF/MM/BPF.
>
> Excellent.  There are other people also interested in this.  eg:
>
> Ryan Roberts: https://lore.kernel.org/linux-mm/4c991dcb-c5bb-86bb-5a29-05df24429607@arm.com/
> Fengwei Yin: https://lore.kernel.org/linux-mm/20230109072232.2398464-1-fengwei.yin@intel.com/
>
> Obviously I'm keenly interested.
> https://lore.kernel.org/linux-mm/Y%2FU8bQd15aUO97vS@casper.infradead.org/
>
> This is definitely worth its own session in the MM track.

Absolutely.