Re: [LSF/MM/BPF TOPIC] HGM for hugetlbfs

[David Hildenbrand <david@redhat.com>]

On 13.06.23 16:59, Jason Gunthorpe wrote:
> On Thu, Jun 08, 2023 at 09:10:15PM +0100, Matthew Wilcox wrote:
>> On Thu, Jun 08, 2023 at 08:34:10AM +0200, David Hildenbrand wrote:
>>> On 08.06.23 02:02, David Rientjes wrote:
>>>> While people have proposed 1GB THP support in the past, it was nacked, in
>>>> part, because of the suggestion to just use existing 1GB support in
>>>> hugetlb instead :)
>>>
>>> Yes, because I still think that the use for "transparent" (for the user)
>>> nowadays is very limited and not worth the complexity.
>>>
>>> IMHO, what you really want is a pool of large pages that (guarantees about
>>> availability and nodes) and fine control about who gets these pages. That's
>>> what hugetlb provides.
>>>
>>> In contrast to THP, you don't want to allow for
>>> * Partially mmap, mremap, munmap, mprotect them
>>> * Partially sharing then / COW'ing them
>>> * Partially mixing them with other anon pages (MADV_DONTNEED + refault)
>>> * Exclude them from some features KSM/swap
>>> * (swap them out and eventually split them for that)
>>>
>>> Because you don't want to get these pages PTE-mapped by the system *unless*
>>> there is a real reason (HGM, hwpoison) -- you want guarantees. Once such a
>>> page is PTE-mapped, you only want to collapse in place.
>>>
>>> But you don't want special-HGM, you simply want the core to PTE-map them
>>> like a (file) THP.
>>>
>>> IMHO, getting that realized much easier would be if we wouldn't have to care
>>> about some of the hugetlb complexity I raised (MAP_PRIVATE, PMD sharing),
>>> but maybe there is a way ...
>>
>> I favour a more evolutionary than revolutionary approach.  That is,
>> I think it's acceptable to add new features to hugetlbfs _if_ they're
>> combined with cleanup work that gets hugetlbfs closer to the main mm.
>> This is why I harp on things like pagewalk that currently need special
>> handling for hugetlb -- that's pointless; they should just be treated as
>> large folios.  GUP handles hugetlb separately too, and I'm not sure why.
> 
> Yes, this echo's my feelings too.
> 
> Making all the special core-mm cases around hugetlb even more
> complicated with HGM seems like a non-starter.
> 
> We need to get to a point where the core-mm handles all the PTE
> programming and supports arbitary order folios in the page tables
> uniformly for everyone.
> 
> hugetlb is just a special high order folio provider.
> 
> Get rid of all the special PTE formats, unique arch code, and special
> code in gup.c/pagewalkers/etc that supports hugetlbfs.
> 
> I think the general path to do that is to make the core-mm and all the
> hugetlb supporting arches support a core-code path for working with
> high order folios in page tables.
> 
> Maybe this is demo'd & tested with a temporary/simplified hugetlbfs
> uAPI. When the core MM and all the arches are ready you switch
> hugetlbfs to use the new core API and deleted all the page walk
> special cases.
> 
>  From there you can then teach the core code to do all the splitting
> and whatever that you want.

Yes, that's my hope.

As I said, some existing oddities like PMD sharing (VM use-cases don't 
really require that) and MAP_PRIVATE handling (again, VMs don't really 
require that) could make the conversion more problematic ... IMHO

So maybe we should really factor out the core hugetlb pooling logic and 
write a simplified v2 implementation that integrates nicely with the VM 
without all of these oddities.

We can then either port some of these oddities step by step from v1 to 
v2 or replace them by something better (for example: if we really want 
MAP_PRIVATE, then just do it like with any other file and use ordinary 
anon (THP) ).

One day, we can then just switch to v2 and remove v1. If we manage 
without any uABI changes, great.

Doing all the conversion in-place could turn out extremely painful and 
take much longer ... but I might be just taught otherwise.

As you say, hugetlb should just be a special folio provider ...

We can discuss tomorrow.

-- 
Cheers,

David / dhildenb