Re: [LSF/MM/BPF TOPIC] HGM for hugetlbfs

[David Rientjes <rientjes@google.com>]

On Thu, 8 Jun 2023, Matthew Wilcox wrote:

> On Thu, Jun 08, 2023 at 08:34:10AM +0200, David Hildenbrand wrote:
> > On 08.06.23 02:02, David Rientjes wrote:
> > > While people have proposed 1GB THP support in the past, it was nacked, in
> > > part, because of the suggestion to just use existing 1GB support in
> > > hugetlb instead :)
> > 
> > Yes, because I still think that the use for "transparent" (for the user)
> > nowadays is very limited and not worth the complexity.
> > 
> > IMHO, what you really want is a pool of large pages that (guarantees about
> > availability and nodes) and fine control about who gets these pages. That's
> > what hugetlb provides.
> > 
> > In contrast to THP, you don't want to allow for
> > * Partially mmap, mremap, munmap, mprotect them
> > * Partially sharing then / COW'ing them
> > * Partially mixing them with other anon pages (MADV_DONTNEED + refault)
> > * Exclude them from some features KSM/swap
> > * (swap them out and eventually split them for that)
> > 
> > Because you don't want to get these pages PTE-mapped by the system *unless*
> > there is a real reason (HGM, hwpoison) -- you want guarantees. Once such a
> > page is PTE-mapped, you only want to collapse in place.
> > 
> > But you don't want special-HGM, you simply want the core to PTE-map them
> > like a (file) THP.
> > 
> > IMHO, getting that realized much easier would be if we wouldn't have to care
> > about some of the hugetlb complexity I raised (MAP_PRIVATE, PMD sharing),
> > but maybe there is a way ...
> 
> I favour a more evolutionary than revolutionary approach.  That is,
> I think it's acceptable to add new features to hugetlbfs _if_ they're
> combined with cleanup work that gets hugetlbfs closer to the main mm.
> This is why I harp on things like pagewalk that currently need special
> handling for hugetlb -- that's pointless; they should just be treated as
> large folios.  GUP handles hugetlb separately too, and I'm not sure why.
> 
> That's not to be confused with "hugetlb must change to be more like
> the regular mm".  Sometimes both are bad, stupid and wrong, and need to
> be changed.  The MM has never had to handle 1GB pages before and, eg,
> handling mapcount by iterating over each struct page is not sensible
> because that's 16MB of data just to answer folio_mapcount().
> 

Ok, so I'll latch onto this feedback because I think it's (1) a concrete 
path forward to solve existing real-world pain by adding support to 
hugetlb (to address hwpoison and postcopy live migration latency) and (2) 
an overall and long-awaited improvement in maintainability for the MM 
subsystem.

Nobody on this thread is interested in substantially increasing the 
complexity of hugetlb.  That's true from the standpoint of sheer 
maintainability, but also reliability.  We've been bitten time and time 
again by hugetlb-only reliability issues, which are their own class of 
customer complaints.  These have not only been in hugetlb's reservation 
code.

In fact, from my POV, hugetlb *reliability* is the most important topic 
discussed so far in this thread and that can be substantially improved by 
this evolutionary approach that reduces the "special casing" that is the 
hugetlb subsystem today.  We would very much want a unified way of 
handling page walks, for example.

I don't think anybody here is advocating for making hugetlb more of a 
snowflake :)  Improving hugetlb maintainability *and* reliability is of 
critical importance to us, as is solving memory poisoning and live 
migration latency.  I don't think that one needs to block the other.

So the work to improve hugetlb reliability and maintainability is 
something that can be tractable and we'd definitely like feedback on so 
that we can contribute to it.

I'd very much prefer that this does not get in the way of solving 
real-world problems that HGM addresses just because it's an active source 
of real customer issues today.  Rest assured, making forward progress on 
HGM will not reduce our interest in improving hugetlb maintainability :)

I know that James is very eager to receive code review for the HGM series 
itself from anybody who would be willing to review it.  Is there a way to 
make forward progress on deciding whether HGM (with any code review 
comments addressed) has a path forward?