Re: [RFC v2 5/5] mm: document mTHP defer setting

[Nico Pache <npache@redhat.com>]

On Mon, Feb 17, 2025 at 8:14 AM Usama Arif <usamaarif642@gmail.com> wrote:
>
>
>
> On 11/02/2025 00:40, Nico Pache wrote:
> > Now that we have mTHP support in khugepaged, lets add it to the
> > transhuge admin guide to provide proper guidance.
> >
>
> I think you should move this patch to the mTHP khugepaged series, and just send
> THP=defer separately from mTHP khguepaged.
>
> > Signed-off-by: Nico Pache <npache@redhat.com>
> > ---
> >  Documentation/admin-guide/mm/transhuge.rst | 22 ++++++++++++++++------
> >  1 file changed, 16 insertions(+), 6 deletions(-)
> >
> > diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
> > index b3b18573bbb4..99ba3763c1c4 100644
> > --- a/Documentation/admin-guide/mm/transhuge.rst
> > +++ b/Documentation/admin-guide/mm/transhuge.rst
> > @@ -63,7 +63,7 @@ often.
> >  THP can be enabled system wide or restricted to certain tasks or even
> >  memory ranges inside task's address space. Unless THP is completely
> >  disabled, there is ``khugepaged`` daemon that scans memory and
> > -collapses sequences of basic pages into PMD-sized huge pages.
> > +collapses sequences of basic pages into huge pages.
> >
> >  The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
> >  interface and using madvise(2) and prctl(2) system calls.
> > @@ -103,8 +103,8 @@ madvise(MADV_HUGEPAGE) on their critical mmapped regions.
> >  Applications that would like to benefit from THPs but would still like a
> >  more memory conservative approach can choose 'defer'. This avoids
> >  inserting THPs at the page fault handler unless they are MADV_HUGEPAGE.
> > -Khugepaged will then scan the mappings for potential collapses into PMD
> > -sized pages. Admins using this the 'defer' setting should consider
> > +Khugepaged will then scan the mappings for potential collapses into (m)THP
> > +pages. Admins using this the 'defer' setting should consider
> >  tweaking khugepaged/max_ptes_none. The current default of 511 may
> >  aggressively collapse your PTEs into PMDs. Lower this value to conserve
> >  more memory (ie. max_ptes_none=64).
> > @@ -119,11 +119,14 @@ Global THP controls
> >
> >  Transparent Hugepage Support for anonymous memory can be entirely disabled
> >  (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE
> > -regions (to avoid the risk of consuming more memory resources) or enabled
> > -system wide. This can be achieved per-supported-THP-size with one of::
> > +regions (to avoid the risk of consuming more memory resources), defered to
> > +khugepaged, or enabled system wide.
> > +
> > +This can be achieved per-supported-THP-size with one of::
> >
> >       echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
> >       echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
> > +     echo defer >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
> >       echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
> >
> >  where <size> is the hugepage size being addressed, the available sizes
> > @@ -155,6 +158,13 @@ hugepage sizes have enabled="never". If enabling multiple hugepage
> >  sizes, the kernel will select the most appropriate enabled size for a
> >  given allocation.
> >
> > +khugepaged use max_ptes_none scaled to the order of the enabled mTHP size to
> > +determine collapses. When using mTHPs its recommended to set max_ptes_none low.
> > +Ideally less than HPAGE_PMD_NR / 2 (255 on 4k page size). This will prevent
> > +undesired "creep" behavior that leads to continously collapsing to a larger
> > +mTHP size. max_ptes_shared and max_ptes_swap have no effect when collapsing to a
> > +mTHP, and mTHP collapse will fail on shared or swapped out pages.
> > +
>
> This paragraph definitely belongs in the khugepaged series, as it doesn't have anything
> to do with THP=defer.
>
> re "Ideally less than HPAGE_PMD_NR / 2",
> what if you are running on amd, and using 16K and 2M THP=always only as, thats where
> the most TLB benefit is. Than this recommendation doesnt make sense?

That may be correct, I believe the creep requires two adjacent mTHP
levels ( ie 512kb, 1024kb) to be enabled for the issue to really
present. Although with max_ptes_none=511, you will almost always
satisfy the collapse request, so your 16Kb mTHPs will be promoted to
PMDs. I dont believe 511 is a good default if using mTHPs.
>
> Also even if you have all mTHP sizes as always, shouldnt you start by collapsing to
> the largest THP size first? (I haven't reviewed the khugepaged series yet, so might
> be have been discussed there, I will try and review it).

We do start at the largest first. The creep happens on a second pass
of the PMD, not immediately in the same collapse.

>
> Did you see the creep behavior you mentioned in your experiments?

Yes, I provided an example of how it happens here.
https://lore.kernel.org/lkml/CAA1CXcDiGLD=dZpFRyAuz4TLrVZZYGp=u7=Z9Q+g9RXbf-s2nA@mail.gmail.com/

>
>
> >  It's also possible to limit defrag efforts in the VM to generate
> >  anonymous hugepages in case they're not immediately free to madvise
> >  regions or to never try to defrag memory and simply fallback to regular
> > @@ -318,7 +328,7 @@ Alternatively, each supported anonymous THP size can be controlled by
> >  passing ``thp_anon=<size>[KMG],<size>[KMG]:<state>;<size>[KMG]-<size>[KMG]:<state>``,
> >  where ``<size>`` is the THP size (must be a power of 2 of PAGE_SIZE and
> >  supported anonymous THP)  and ``<state>`` is one of ``always``, ``madvise``,
> > -``never`` or ``inherit``.
> > +``defer``, ``never`` or ``inherit``.
> >
> >  For example, the following will set 16K, 32K, 64K THP to ``always``,
> >  set 128K, 512K to ``inherit``, set 256K to ``madvise`` and 1M, 2M
>