Re: [PATCH 0/1] prctl: allow overriding system THP policy to always

[Usama Arif <usamaarif642@gmail.com>]

On 07/05/2025 16:57, Zi Yan wrote:
> On 7 May 2025, at 11:12, Usama Arif wrote:
> 
>> On 07/05/2025 15:57, Zi Yan wrote:
>>> +Yafang, who is also looking at changing THP config at cgroup/container level.
>>>
>>> On 7 May 2025, at 10:00, Usama Arif wrote:
>>>
>>>> Allowing override of global THP policy per process allows workloads
>>>> that have shown to benefit from hugepages to do so, without regressing
>>>> workloads that wouldn't benefit. This will allow such types of
>>>> workloads to be run/stacked on the same machine.
>>>>
>>>> It also helps in rolling out hugepages in hyperscaler configurations
>>>> for workloads that benefit from them, where a single THP policy is
>>>> likely to be used across the entire fleet, and prctl will help override it.
>>>>
>>>> An advantage of doing it via prctl vs creating a cgroup specific
>>>> option (like /sys/fs/cgroup/test/memory.transparent_hugepage.enabled) is
>>>> that this will work even when there are no cgroups present, and my
>>>> understanding is there is a strong preference of cgroups controls being
>>>> hierarchical which usually means them having a numerical value.
>>>
>>> Hi Usama,
>>>
>>> Do you mind giving an example on how to change THP policy for a set of
>>> processes running in a container (under a cgroup)?
>>
>> Hi Zi,
>>
>> In our case, we create the processes in the cgroup via systemd. The way we will enable THP=always
>> for processes in a cgroup is in the same way we enable KSM for the cgroup.
>> The change in systemd would be very similar to the line in [1], where we would set prctl PR_SET_THP_ALWAYS
>> in exec-invoke.
>> This is at the start of the process, but you would already know at the start of the process
>> whether you want THP=always for it or not.
>>
>> [1] https://github.com/systemd/systemd/blob/2e72d3efafa88c1cb4d9b28dd4ade7c6ab7be29a/src/core/exec-invoke.c#L5045
> 
> You also need to add a new systemd.directives, e.g., MemoryTHP, to
> pass the THP enablement or disablement info from a systemd config file.
> And if you find those processes do not benefit from using THPs,
> you can just change the new "MemoryTHP" config and restart the processes.
> 
> Am I getting it? Thanks.
> 

Yes, thats right. They would exactly the same as what we (Meta) do
for KSM. So have MemoryTHP similar to MemroryKSM [1] and if MemoryTHP is set,
the ExecContext->memory_thp would be set similar to memory_ksm [2], and when
that is set, the prctl will be called at exec_invoke of the process [3].

The systemd changes should be quite simple to do.

[1] https://github.com/systemd/systemd/blob/2e72d3efafa88c1cb4d9b28dd4ade7c6ab7be29a/man/systemd.exec.xml#L1978 
[2] https://github.com/systemd/systemd/blob/2e72d3efafa88c1cb4d9b28dd4ade7c6ab7be29a/src/core/dbus-execute.c#L2151
[3] https://github.com/systemd/systemd/blob/2e72d3efafa88c1cb4d9b28dd4ade7c6ab7be29a/src/core/exec-invoke.c#L5045
>>>
>>> Yafang mentioned that the prctl approach would require restarting all running
>>> services[1] and other inflexiblities, so he proposed to use BPF to change THP
>>> policy[2]. I wonder if Yafang's issues also apply to your case and if you
>>> have a solution to them.
>>>
>>> Thanks.
>>>
>>> [1] https://lore.kernel.org/linux-mm/CALOAHbCXMi2GaZdHJaNLXxGsJf-hkDTrztsQiceaBcJ8d8p3cA@mail.gmail.com/
>>> [2] https://lore.kernel.org/linux-mm/20250429024139.34365-1-laoar.shao@gmail.com/
>>>>
>>>>
>>>> The output and code of test program is below:
>>>>
>>>> [root@vm4 vmuser]# echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
>>>> [root@vm4 vmuser]# echo inherit > /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled
>>>> [root@vm4 vmuser]# ./a.out
>>>> Default THP setting:
>>>> THP is not set to 'always'.
>>>> PR_SET_THP_ALWAYS = 1
>>>> THP is set to 'always'.
>>>> PR_SET_THP_ALWAYS = 0
>>>> THP is not set to 'always'.
>>>>
>>>>
>>>> #include <stdio.h>
>>>> #include <stdlib.h>
>>>> #include <string.h>
>>>> #include <unistd.h>
>>>> #include <sys/mman.h>
>>>> #include <sys/prctl.h>
>>>>
>>>> #define PR_SET_THP_ALWAYS 78
>>>> #define SIZE 12 * (2 * 1024 * 1024) // 24 MB
>>>>
>>>> void check_smaps(void) {
>>>>     FILE *file = fopen("/proc/self/smaps", "r");
>>>>     if (!file) {
>>>>         perror("fopen");
>>>>         return;
>>>>     }
>>>>
>>>>     char line[256];
>>>>     int is_hugepage = 0;
>>>>     while (fgets(line, sizeof(line), file)) {
>>>>         // if (strstr(line, "AnonHugePages:"))
>>>>         //     printf("%s\n", line);
>>>>         if (strstr(line, "AnonHugePages:") && strstr(line, "24576 kB"))
>>>> {
>>>>             // printf("%s\n", line);
>>>>             is_hugepage = 1;
>>>>             break;
>>>>         }
>>>>     }
>>>>     fclose(file);
>>>>     if (is_hugepage) {
>>>>         printf("THP is set to 'always'.\n");
>>>>     } else {
>>>>         printf("THP is not set to 'always'.\n");
>>>>     }
>>>> }
>>>>
>>>> void test_mmap_thp(void) {
>>>>     char *buffer = (char *)mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
>>>>                                 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>>>>     if (buffer == MAP_FAILED) {
>>>>         perror("mmap");
>>>>         return;
>>>>     }
>>>>     // Touch the memory to ensure it's allocated
>>>>     memset(buffer, 0, SIZE);
>>>>     check_smaps();
>>>>     munmap(buffer, SIZE);
>>>> }
>>>>
>>>> int main() {
>>>>     printf("Default THP setting: \n");
>>>>     test_mmap_thp();
>>>>     printf("PR_SET_THP_ALWAYS = 1 \n");
>>>>     prctl(PR_SET_THP_ALWAYS, 1, NULL, NULL, NULL);
>>>>     test_mmap_thp();
>>>>     printf("PR_SET_THP_ALWAYS = 0 \n");
>>>>     prctl(PR_SET_THP_ALWAYS, 0, NULL, NULL, NULL);
>>>>     test_mmap_thp();
>>>>
>>>>     return 0;
>>>> }
>>>>
>>>>
>>>> Usama Arif (1):
>>>>   prctl: allow overriding system THP policy to always per process
>>>>
>>>>  include/linux/huge_mm.h                          |  3 ++-
>>>>  include/linux/mm_types.h                         |  7 ++-----
>>>>  include/uapi/linux/prctl.h                       |  3 +++
>>>>  kernel/sys.c                                     | 16 ++++++++++++++++
>>>>  tools/include/uapi/linux/prctl.h                 |  3 +++
>>>>  .../perf/trace/beauty/include/uapi/linux/prctl.h |  3 +++
>>>>  6 files changed, 29 insertions(+), 6 deletions(-)
>>>>
>>>> -- 
>>>> 2.47.1
>>>
>>>
>>> --
>>> Best Regards,
>>> Yan, Zi
> 
> 
> --
> Best Regards,
> Yan, Zi