Re: hugetlb page migration vs. overcommit

[Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>]

On 11/23/2017 04:11 AM, Mike Kravetz wrote:
> On 11/22/2017 07:28 AM, Michal Hocko wrote:
>> Hi,
>> is there any reason why we enforce the overcommit limit during hugetlb
>> pages migration? It's in alloc_huge_page_node->__alloc_buddy_huge_page
>> path. I am wondering whether this is really an intentional behavior.
> 
> I do not think it was intentional.  But, I was not around when that
> code was added.
> 
>> The page migration allocates a page just temporarily so we should be
>> able to go over the overcommit limit for the migration duration. The
>> reason I am asking is that hugetlb pages tend to be utilized usually
>> (otherwise the memory would be just wasted and pool shrunk) but then
>> the migration simply fails which breaks memory hotplug and other
>> migration dependent functionality which is quite suboptimal. You can
>> workaround that by increasing the overcommit limit.
> 
> Yes.  In an environment making optimal use of huge pages, you are unlikely
> to have 'spare pages' set aside for a potential migration operation.  So
> I agree that it would make sense to try and allocate overcommit pages for
> this purpose.

Thank you for pointing this out, Michal, Mike.
Doing overcommitting in hugepage migration is totally right to me,
I just didn't notice it when I wrote the code.

> 
>> Why don't we simply migrate as long as we are able to allocate the
>> target hugetlb page? I have a half baked patch to remove this
>> restriction, would there be an opposition to do something like that?
> 
> I would not be opposed and would help with this effort.  My concern would
> be any subtle hugetlb accounting issues once you start messing with
> additional overcommit pages.

Yes, hugetlb accounting always needs care when touching related code.
I can help testing.

Thanks,
Naoya Horiguchi