Re: [PATCH v3 1/2] mm: huge_memory: disallow hugepages if the system-wide THP sysfs settings are disabled

[Baolin Wang <baolin.wang@linux.alibaba.com>]

On 2025/6/23 19:08, Barry Song wrote:
> On Mon, Jun 23, 2025 at 8:28 PM Baolin Wang
> <baolin.wang@linux.alibaba.com> wrote:
>>
>> When invoking thp_vma_allowable_orders(), the TVA_ENFORCE_SYSFS flag is not
>> specified, we will ignore the THP sysfs settings. Whilst it makes sense for the
>> callers who do not specify this flag, it creates a odd and surprising situation
>> where a sysadmin specifying 'never' for all THP sizes still observing THP pages
>> being allocated and used on the system.
>>
>> The motivating case for this is MADV_COLLAPSE. The MADV_COLLAPSE will ignore
>> the system-wide Anon THP sysfs settings, which means that even though we have
>> disabled the Anon THP configuration, MADV_COLLAPSE will still attempt to collapse
>> into a Anon THP. This violates the rule we have agreed upon: never means never.
>>
> 
> Should we update the man page for madv_collapse ?
> https://man7.org/linux/man-pages/man2/madvise.2.html
> 
>                MADV_COLLAPSE is independent of any sysfs (see sysfs(5))
>                setting under /sys/kernel/mm/transparent_hugepage, both in
>                terms of determining THP eligibility, and allocation
>                semantics.  See Linux kernel source file
>                Documentation/admin-guide/mm/transhuge.rst for more
>                information.  MADV_COLLAPSE also ignores huge= tmpfs mount
>                when operating on tmpfs files.  Allocation for the new
>                hugepage may enter direct reclaim and/or compaction,
>                regardless of VMA flags (though VM_NOHUGEPAGE is still
>                respected).
> 
> So this effectively changes the uABI, right?

Good point. Will update the man page.

>> Currently, besides MADV_COLLAPSE not setting TVA_ENFORCE_SYSFS, there is only
>> one other instance where TVA_ENFORCE_SYSFS is not set, which is in the
>> collapse_pte_mapped_thp() function, but I believe this is reasonable from its
>> comments:
>>
>> "
>> /*
>>   * If we are here, we've succeeded in replacing all the native pages
>>   * in the page cache with a single hugepage. If a mm were to fault-in
>>   * this memory (mapped by a suitably aligned VMA), we'd get the hugepage
>>   * and map it by a PMD, regardless of sysfs THP settings. As such, let's
>>   * analogously elide sysfs THP settings here.
>>   */
>> if (!thp_vma_allowable_order(vma, vma->vm_flags, 0, PMD_ORDER))
>> "
>>
>> Another rule for madvise, referring to David's suggestion: “allowing for
>> collapsing in a VM without VM_HUGEPAGE in the "madvise" mode would be fine".
>>
>> To address this issue, the current strategy should be:
>>
>> If no hugepage modes are enabled for the desired orders, nor can we enable them
>> by inheriting from a 'global' enabled setting - then it must be the case that
>> all desired orders either specify or inherit 'NEVER' - and we must abort.
>>
>> Meanwhile, we should fix the khugepaged selftest for MADV_COLLAPSE by enabling
>> THP.
> 
> It’s a bit odd that the old test case expects collapsing to succeed
> even when we’ve set it
> to ‘never’.
> Setting it to ‘always’ doesn’t seem to test anything as a counterpart.
> 
> I assume the goal is to test that setting it to ‘never’ prevents collapsing?

The original logic will prevent khugepaged by setting THP_NEVER, 
allowing only madvise_collapse() to perform THP collapse. And this is 
the logic this patchset tries to fix, which is to also prevent 
madvise_collapse() from performing THP collapse when system-wide THP 
sysfs settings are disabled.

Therefore, it should be changed to THP_ALWAYS here to allow 
madvise_collapse() to perform THP collapse.

Of course, the current logic cannot completely disable khugepaged, but I 
haven't found a better way to modify it. As David suggested, changing to 
MADVISE mode would cause some test cases to fail because some tests 
previously set MADV_NOHUGEPAGE, and now there is no other way to clear 
the MADV_NOHUGEPAGE flag except for setting MADV_HUGEPAGE. As a result, 
khugepaged cannot be completely disabled either.

So I think we should introduce a new method to clear MADV_NOHUGEPAGE 
flag without setting MADV_HUGEPAGE in the future.