Re: [LSF/MM/BPF TOPIC] HGM for hugetlbfs

[David Rientjes <rientjes@google.com>]

On Fri, 2 Jun 2023, Mike Kravetz wrote:

> The benefit of HGM in the case of memory errors is fairly obvious.  As
> mentioned above, when a memory error is encountered on a hugetlb page,
> that entire hugetlb page becomes inaccessible to the application.  Losing,
> 1G or even 2M of data is often catastrophic for an application.  There
> is often no way to recover.  It just makes sense that recovering from
> the loss of 4K of data would generally be easier and more likely to be
> possible.  Today, when Oracle DB encounters a hard memory error on a
> hugetlb page it will shutdown.  Plans are currently in place repair and
> recover from such errors if possible.  Isolating the area of data loss
> to a single 4K page significantly increases the likelihood of repair and
> recovery.
> 
> Today, when a memory error is encountered on a hugetlb page an
> application is 'notified' of the error by a SIGBUS, as well as the
> virtual address of the hugetlb page and it's size.  This makes sense as
> hugetlb pages are accessed by a single page table entry, so you get all
> or nothing.  As mentioned by James above, this is catastrophic for VMs
> as the hypervisor has just been told that 2M or 1G is now inaccessible.
> With HGM, we can isolate such errors to 4K.
> 
> Backing VMs with hugetlb pages is a real use case today.  We are seeing
> memory errors on such hugetlb pages with the result being VM failures.
> One of the advantages of backing VMs with THPs is that they are split in
> the case of memory errors.  HGM would allow similar functionality.

Thanks for this context, Mike, it's very useful.

I think everybody is aligned on the desire to map memory at smaller 
granularities for multiple use cases and it's fairly clear that these use 
cases are critically important to multiple stakeholders.

I think the open question is whether this functionality is supported in 
hugetlbfs (like with HGM) or that there is a hard requirement that we must 
use THP for this support.

I don't think that hugetlbfs is feature frozen, but if there's a strong 
bias toward not merging additional complexity into the subsystem that 
would useful to know.  I personally think the critical use cases described 
above justify the added complexity of HGM to hugetlb and we wouldn't be 
blocked by the long standing (15+ years) desire to mesh hugetlb into the 
core MM subsystem before we can stop the pain associated with memory 
poisoning and live migration.

Are there strong objections to extending hugetlb for this support?