Re: [PATCH v2 2/4] hugetlb: add HPageCma flag and code to free non-gigantic pages in CMA

[Mike Kravetz <mike.kravetz@oracle.com>]

On 9/29/21 12:42 PM, Mike Kravetz wrote:
> On 9/24/21 2:36 AM, David Hildenbrand wrote:
>> On 23.09.21 19:53, Mike Kravetz wrote:
>>> When huge page demotion is fully implemented, gigantic pages can be
>>> demoted to a smaller huge page size.  For example, on x86 a 1G page
>>> can be demoted to 512 2M pages.  However, gigantic pages can potentially
>>> be allocated from CMA.  If a gigantic page which was allocated from CMA
>>> is demoted, the corresponding demoted pages needs to be returned to CMA.
>>>
>>> In order to track hugetlb pages that need to be returned to CMA, add the
>>> hugetlb specific flag HPageCma.  Flag is set when a huge page is
>>> allocated from CMA and transferred to any demoted pages.  Non-gigantic
>>> huge page freeing code checks for the flag and takes appropriate action.
>>
>> Do we really need that flag or couldn't we simply always try cma_release() and fallback to out ordinary freeing-path?
>>
>> IIRC, cma knows exactly if something was allocated via a CMA are and can be free via it. No need for additional tracking usually.
>>
> 
> Yes, I think this is possible.
> Initially, I thought the check for whether pages were part of CMA
> involved a mutex or some type of locking.  But, it really is
> lightweight.  So, should not be in issue calling in every case.

When modifying the code, I did come across one issue.  Sorry I did not
immediately remember this.

Gigantic pages are allocated as a 'set of pages' and turned into a compound
page by the hugetlb code.  They must be restored to a 'set of pages' before
calling cma_release.  You can not pass a compound page to cma_release.
Non-gigantic page are allocated from the buddy directly as compound pages.
They are returned to buddy as a compound page.

So, the issue comes about when freeing a non-gigantic page.  We would
need to convert to a 'set of pages' before calling cma_release just to
see if cma_release succeeds.  Then, if it fails convert back to a
compound page to call __free_pages.  Conversion is somewhat expensive as
we must modify every tail page struct.

Some possible solutions:
- Create a cma_pages_valid() routine that checks whether the pages
  belong to a cma region.  Only convert to 'set of pages' if cma_pages_valid
  and we know subsequent cma_release will succeed.
- Always convert non-gigantic pages to a 'set of pages' before freeing.
  Alternatively, don't allocate compound pages from buddy and just use
  the hugetlb gigantic page prep and destroy routines for all hugetlb
  page sizes.
- Use some kind of flag as in proposed patch.

Having hugetlb just allocate a set of pages from buddy is interesting.
This would make the allocate/free code paths for gigantic and
non-gigantic pages align more closely.  It may in overall code simplification,
not just for demote.
-- 
Mike Kravetz