Re: [RFC PATCH v2 0/5] mm, bpf: BPF based THP adjustment

[Yafang Shao <laoar.shao@gmail.com>]

On Mon, May 26, 2025 at 6:49 PM David Hildenbrand <david@redhat.com> wrote:
>
> On 26.05.25 11:37, Yafang Shao wrote:
> > On Mon, May 26, 2025 at 4:14 PM David Hildenbrand <david@redhat.com> wrote:
> >>
> >>> Hi all,
> >>>
> >>> Let’s summarize the current state of the discussion and identify how
> >>> to move forward.
> >>>
> >>> - Global-Only Control is Not Viable
> >>> We all seem to agree that a global-only control for THP is unwise. In
> >>> practice, some workloads benefit from THP while others do not, so a
> >>> one-size-fits-all approach doesn’t work.
> >>>
> >>> - Should We Use "Always" or "Madvise"?
> >>> I suspect no one would choose 'always' in its current state. ;)
> >>
> >> IIRC, RHEL9 has the default set to "always" for a long time.
> >
> > good to know.
> >
> >>
> >> I guess it really depends on how different the workloads are that you
> >> are running on the same machine.
> >
> > Correct. If we want to enable THP for specific workloads without
> > modifying the kernel, we must isolate them on dedicated servers.
> > However, this approach wastes resources and is not an acceptable
> > solution.
> >
> >>
> >>   > Both Lorenzo and David propose relying on the madvise mode. However,>
> >> since madvise is an unprivileged userspace mechanism, any user can
> >>> freely adjust their THP policy. This makes fine-grained control
> >>> impossible without breaking userspace compatibility—an undesirable
> >>> tradeoff.
> >>
> >> If required, we could look into a "sealing" mechanism, that would
> >> essentially lock modification attempts performed by the process (i.e.,
> >> MADV_HUGEPAGE).
> >
> > If we don’t introduce a new THP mode and instead rely solely on
> > madvise, the "sealing" mechanism could either violate the intended
> > semantics of madvise(), or simply break madvise() entirely, right?
>
> We would have to be a bit careful, yes.
>
> Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because
> these options also fail with -EINVAL on kernels without THP support.
>
> Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.
>
> What you likely really want to do is seal when you configured
> MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.
>
> >>
> >> The could be added on top of the current proposals that are flying
> >> around, and could be done e.g., per-process.
> >
> > How about introducing a dedicated "process" mode? This would allow
> > each process to use different THP modes—some in "always," others in
> > "madvise," and the rest in "never." Future THP modes could also be
> > added to this framework.
>
> We have to be really careful about not creating even more mess with more
> modes.
>
> How would that design look like in detail (how would we set it per
> process etc?)?

I have a preliminary idea to implement this using BPF. We could define
the API as follows:

struct bpf_thp_ops {
       /**
        * @task_thp_mode: Get the THP mode for a specific task
        *
        * Return:
        * - TASK_THP_ALWAYS: "always" mode
        * - TASK_THP_MADVISE: "madvise" mode
        * - TASK_THP_NEVER: "never" mode
        * Future modes can also be added.
        */
       int (*task_thp_mode)(struct task_struct *p);
};

For observability, we could add a "THP mode" field to
/proc/[pid]/status. For example:

$ grep "THP mode" /proc/123/status
always
$ grep "THP mode" /proc/456/status
madvise
$ grep "THP mode" /proc/789/status
never

The THP mode for each task would be determined by the attached BPF
program based on the task's attributes. We would place the BPF hook in
appropriate kernel functions. Note that this setting wouldn't be
inherited during fork/exec - the BPF program would make the decision
dynamically for each task.
This approach also enables runtime adjustments to THP modes based on
system-wide conditions, such as memory fragmentation or other
performance overheads. The BPF program could adapt policies
dynamically, optimizing THP behavior in response to changing
workloads.

As Liam pointed out in another thread, naming is challenging here -
"process" might not be the most accurate term for this context.

-- 
Regards
Yafang