[PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Wenchao Hao]

Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
each VMA, but it does not include large pages smaller than PMD size.

This patch adds the statistics of anonymous huge pages allocated by
mTHP which is smaller than PMD size to AnonHugePages field in smaps.

Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
---
 fs/proc/task_mmu.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 38a5a3e9cba2..b655011627d8 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
 		if (!folio_test_swapbacked(folio) && !dirty &&
 		    !folio_test_dirty(folio))
 			mss->lazyfree += size;
+
+		/*
+		 * Count large pages smaller than PMD size to anonymous_thp
+		 */
+		if (!compound && PageHead(page) && folio_order(folio))
+			mss->anonymous_thp += folio_size(folio);
 	}
 
 	if (folio_test_ksm(folio))
-- 
2.45.0

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[David Hildenbrand]

On 03.12.24 14:49, Wenchao Hao wrote:
> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
> each VMA, but it does not include large pages smaller than PMD size.
> 
> This patch adds the statistics of anonymous huge pages allocated by
> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
> 
> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
> ---
>   fs/proc/task_mmu.c | 6 ++++++
>   1 file changed, 6 insertions(+)
> 
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 38a5a3e9cba2..b655011627d8 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
>   		if (!folio_test_swapbacked(folio) && !dirty &&
>   		    !folio_test_dirty(folio))
>   			mss->lazyfree += size;
> +
> +		/*
> +		 * Count large pages smaller than PMD size to anonymous_thp
> +		 */
> +		if (!compound && PageHead(page) && folio_order(folio))
> +			mss->anonymous_thp += folio_size(folio);
>   	}
>   
>   	if (folio_test_ksm(folio))


I think we decided to leave this (and /proc/meminfo) be one of the last
interfaces where this is only concerned with PMD-sized ones:

Documentation/admin-guide/mm/transhuge.rst:

The number of PMD-sized anonymous transparent huge pages currently used by the
system is available by reading the AnonHugePages field in ``/proc/meminfo``.
To identify what applications are using PMD-sized anonymous transparent huge
pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
fields for each mapping. (Note that AnonHugePages only applies to traditional
PMD-sized THP for historical reasons and should have been called
AnonHugePmdMapped).



-- 
Cheers,

David / dhildenb

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Ryan Roberts]

On 03/12/2024 14:17, David Hildenbrand wrote:
> On 03.12.24 14:49, Wenchao Hao wrote:
>> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
>> each VMA, but it does not include large pages smaller than PMD size.
>>
>> This patch adds the statistics of anonymous huge pages allocated by
>> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
>>
>> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
>> ---
>>   fs/proc/task_mmu.c | 6 ++++++
>>   1 file changed, 6 insertions(+)
>>
>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>> index 38a5a3e9cba2..b655011627d8 100644
>> --- a/fs/proc/task_mmu.c
>> +++ b/fs/proc/task_mmu.c
>> @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss,
>> struct page *page,
>>           if (!folio_test_swapbacked(folio) && !dirty &&
>>               !folio_test_dirty(folio))
>>               mss->lazyfree += size;
>> +
>> +        /*
>> +         * Count large pages smaller than PMD size to anonymous_thp
>> +         */
>> +        if (!compound && PageHead(page) && folio_order(folio))
>> +            mss->anonymous_thp += folio_size(folio);
>>       }
>>         if (folio_test_ksm(folio))
> 
> 
> I think we decided to leave this (and /proc/meminfo) be one of the last
> interfaces where this is only concerned with PMD-sized ones:
> 
> Documentation/admin-guide/mm/transhuge.rst:
> 
> The number of PMD-sized anonymous transparent huge pages currently used by the
> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
> To identify what applications are using PMD-sized anonymous transparent huge
> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
> fields for each mapping. (Note that AnonHugePages only applies to traditional
> PMD-sized THP for historical reasons and should have been called
> AnonHugePmdMapped).
> 

Agreed. If you need per-process metrics for mTHP, we have a python script at
tools/mm/thpmaps which does a fairly good job of parsing pagemap. --help gives
you all the options.

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Wenchao Hao]

On 2024/12/3 22:42, Ryan Roberts wrote:
> On 03/12/2024 14:17, David Hildenbrand wrote:
>> On 03.12.24 14:49, Wenchao Hao wrote:
>>> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
>>> each VMA, but it does not include large pages smaller than PMD size.
>>>
>>> This patch adds the statistics of anonymous huge pages allocated by
>>> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
>>>
>>> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
>>> ---
>>>   fs/proc/task_mmu.c | 6 ++++++
>>>   1 file changed, 6 insertions(+)
>>>
>>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>>> index 38a5a3e9cba2..b655011627d8 100644
>>> --- a/fs/proc/task_mmu.c
>>> +++ b/fs/proc/task_mmu.c
>>> @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss,
>>> struct page *page,
>>>           if (!folio_test_swapbacked(folio) && !dirty &&
>>>               !folio_test_dirty(folio))
>>>               mss->lazyfree += size;
>>> +
>>> +        /*
>>> +         * Count large pages smaller than PMD size to anonymous_thp
>>> +         */
>>> +        if (!compound && PageHead(page) && folio_order(folio))
>>> +            mss->anonymous_thp += folio_size(folio);
>>>       }
>>>         if (folio_test_ksm(folio))
>>
>>
>> I think we decided to leave this (and /proc/meminfo) be one of the last
>> interfaces where this is only concerned with PMD-sized ones:
>>
>> Documentation/admin-guide/mm/transhuge.rst:
>>
>> The number of PMD-sized anonymous transparent huge pages currently used by the
>> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
>> To identify what applications are using PMD-sized anonymous transparent huge
>> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
>> fields for each mapping. (Note that AnonHugePages only applies to traditional
>> PMD-sized THP for historical reasons and should have been called
>> AnonHugePmdMapped).
>>
> 
> Agreed. If you need per-process metrics for mTHP, we have a python script at
> tools/mm/thpmaps which does a fairly good job of parsing pagemap. --help gives
> you all the options.
> 

I tried this tool, and it is very powerful and practical IMO.
However, thereare two disadvantages:

- This tool is heavily dependent on Python and Python libraries.
  After installing several libraries with the pip command, I was able to
  get it running.
  In practice, the environment we need to analyze may be a mobile or
  embedded environment, where it is very difficult to deploy these
  libraries.
- It seems that this tool only counts file-backed large pages? During
  the actual test, I mapped a region of anonymous pages and mapped it
  as large pages, but the tool did not display those large pages.
  Below is my test file(mTHP related sysfs interface is set to "always"
  to make sure using large pages):

int main()
{
        int i;
        char *c;
        unsigned long *p;

        p = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
        if (!p) {
                perror("fail to get memory");
                exit(-1);
        }

        c = (unsigned char *)p;

        for (i = 0; i < SIZE / 8; i += 8)
                *(p + i) = 0xffff + i;

        while (1)
                sleep(10);

        return 0;
}

Thanks,
wenchao

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Ryan Roberts]

On 04/12/2024 14:40, Wenchao Hao wrote:
> On 2024/12/3 22:42, Ryan Roberts wrote:
>> On 03/12/2024 14:17, David Hildenbrand wrote:
>>> On 03.12.24 14:49, Wenchao Hao wrote:
>>>> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
>>>> each VMA, but it does not include large pages smaller than PMD size.
>>>>
>>>> This patch adds the statistics of anonymous huge pages allocated by
>>>> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
>>>>
>>>> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
>>>> ---
>>>>   fs/proc/task_mmu.c | 6 ++++++
>>>>   1 file changed, 6 insertions(+)
>>>>
>>>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>>>> index 38a5a3e9cba2..b655011627d8 100644
>>>> --- a/fs/proc/task_mmu.c
>>>> +++ b/fs/proc/task_mmu.c
>>>> @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss,
>>>> struct page *page,
>>>>           if (!folio_test_swapbacked(folio) && !dirty &&
>>>>               !folio_test_dirty(folio))
>>>>               mss->lazyfree += size;
>>>> +
>>>> +        /*
>>>> +         * Count large pages smaller than PMD size to anonymous_thp
>>>> +         */
>>>> +        if (!compound && PageHead(page) && folio_order(folio))
>>>> +            mss->anonymous_thp += folio_size(folio);
>>>>       }
>>>>         if (folio_test_ksm(folio))
>>>
>>>
>>> I think we decided to leave this (and /proc/meminfo) be one of the last
>>> interfaces where this is only concerned with PMD-sized ones:
>>>
>>> Documentation/admin-guide/mm/transhuge.rst:
>>>
>>> The number of PMD-sized anonymous transparent huge pages currently used by the
>>> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
>>> To identify what applications are using PMD-sized anonymous transparent huge
>>> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
>>> fields for each mapping. (Note that AnonHugePages only applies to traditional
>>> PMD-sized THP for historical reasons and should have been called
>>> AnonHugePmdMapped).
>>>
>>
>> Agreed. If you need per-process metrics for mTHP, we have a python script at
>> tools/mm/thpmaps which does a fairly good job of parsing pagemap. --help gives
>> you all the options.
>>
> 
> I tried this tool, and it is very powerful and practical IMO.
> However, thereare two disadvantages:
> 
> - This tool is heavily dependent on Python and Python libraries.
>   After installing several libraries with the pip command, I was able to
>   get it running.

I think numpy is the only package it uses which is not in the standard library?
What other libraries did you need to install?

>   In practice, the environment we need to analyze may be a mobile or
>   embedded environment, where it is very difficult to deploy these
>   libraries.

Yes, I agree that's a problem, especially for Android. The script has proven
useful to me for debugging in a traditional Linux distro environment though.

> - It seems that this tool only counts file-backed large pages? During

No; the tool counts file-backed and anon memory. But it reports it in separate
counters. See `thpmaps --help` for full details.

>   the actual test, I mapped a region of anonymous pages and mapped it
>   as large pages, but the tool did not display those large pages.
>   Below is my test file(mTHP related sysfs interface is set to "always"
>   to make sure using large pages):

Which mTHP sizes did you enable? Depending on your value of SIZE and which mTHP
sizes are enabled, you may not have a correctly aligned region in p. So mTHP
would not be allocated. Best to over-allocate then explicitly align p to the
mTHP size, then fault it in.

> 
> int main()
> {
>         int i;
>         char *c;
>         unsigned long *p;
> 
>         p = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

What is SIZE here?

>         if (!p) {
>                 perror("fail to get memory");
>                 exit(-1);
>         }
> 
>         c = (unsigned char *)p;
> 
>         for (i = 0; i < SIZE / 8; i += 8)
>                 *(p + i) = 0xffff + i;

Err... what's your intent here? I think you're writting to 1 in every 8 longs?
Probably just write to the first byte of every page.

Thanks,
Ryan

> 
>         while (1)
>                 sleep(10);
> 
>         return 0;
> }
> 
> Thanks,
> wenchao
> 

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Wenchao Hao]

On 2024/12/5 1:05, Ryan Roberts wrote:
> On 04/12/2024 14:40, Wenchao Hao wrote:
>> On 2024/12/3 22:42, Ryan Roberts wrote:
>>> On 03/12/2024 14:17, David Hildenbrand wrote:
>>>> On 03.12.24 14:49, Wenchao Hao wrote:
>>>>> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
>>>>> each VMA, but it does not include large pages smaller than PMD size.
>>>>>
>>>>> This patch adds the statistics of anonymous huge pages allocated by
>>>>> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
>>>>>
>>>>> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
>>>>> ---
>>>>>   fs/proc/task_mmu.c | 6 ++++++
>>>>>   1 file changed, 6 insertions(+)
>>>>>
>>>>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>>>>> index 38a5a3e9cba2..b655011627d8 100644
>>>>> --- a/fs/proc/task_mmu.c
>>>>> +++ b/fs/proc/task_mmu.c
>>>>> @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss,
>>>>> struct page *page,
>>>>>           if (!folio_test_swapbacked(folio) && !dirty &&
>>>>>               !folio_test_dirty(folio))
>>>>>               mss->lazyfree += size;
>>>>> +
>>>>> +        /*
>>>>> +         * Count large pages smaller than PMD size to anonymous_thp
>>>>> +         */
>>>>> +        if (!compound && PageHead(page) && folio_order(folio))
>>>>> +            mss->anonymous_thp += folio_size(folio);
>>>>>       }
>>>>>         if (folio_test_ksm(folio))
>>>>
>>>>
>>>> I think we decided to leave this (and /proc/meminfo) be one of the last
>>>> interfaces where this is only concerned with PMD-sized ones:
>>>>
>>>> Documentation/admin-guide/mm/transhuge.rst:
>>>>
>>>> The number of PMD-sized anonymous transparent huge pages currently used by the
>>>> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
>>>> To identify what applications are using PMD-sized anonymous transparent huge
>>>> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
>>>> fields for each mapping. (Note that AnonHugePages only applies to traditional
>>>> PMD-sized THP for historical reasons and should have been called
>>>> AnonHugePmdMapped).
>>>>
>>>
>>> Agreed. If you need per-process metrics for mTHP, we have a python script at
>>> tools/mm/thpmaps which does a fairly good job of parsing pagemap. --help gives
>>> you all the options.
>>>
>>
>> I tried this tool, and it is very powerful and practical IMO.
>> However, thereare two disadvantages:
>>
>> - This tool is heavily dependent on Python and Python libraries.
>>   After installing several libraries with the pip command, I was able to
>>   get it running.
> 
> I think numpy is the only package it uses which is not in the standard library?
> What other libraries did you need to install?
> 

Yes, I just tested it on the standard version (Fedora), and that is indeed the case.
Previously, I needed to install additional packages is because I removed some unused
software from the old environment.

Recently, I revisited and started using your tool again. It’s very useful, meeting
my needs and even exceeding them. I am now testing with qemu to run a fedora, so
it's easy to run it.

>>   In practice, the environment we need to analyze may be a mobile or
>>   embedded environment, where it is very difficult to deploy these
>>   libraries.
> 
> Yes, I agree that's a problem, especially for Android. The script has proven
> useful to me for debugging in a traditional Linux distro environment though.
> 
>> - It seems that this tool only counts file-backed large pages? During
> 
> No; the tool counts file-backed and anon memory. But it reports it in separate
> counters. See `thpmaps --help` for full details.
> 
>>   the actual test, I mapped a region of anonymous pages and mapped it
>>   as large pages, but the tool did not display those large pages.
>>   Below is my test file(mTHP related sysfs interface is set to "always"
>>   to make sure using large pages):
> 
> Which mTHP sizes did you enable? Depending on your value of SIZE and which mTHP
> sizes are enabled, you may not have a correctly aligned region in p. So mTHP
> would not be allocated. Best to over-allocate then explicitly align p to the
> mTHP size, then fault it in.
> 

I enabled 64k/128k/256k MTHP and have been studying, debugging, and changing
parts of the khugepaged code to try merging standard pages into mTHP large
pages. So, I wanted to use smap to observe the large page sizes in a process.

>>
>> int main()
>> {
>>         int i;
>>         char *c;
>>         unsigned long *p;
>>
>>         p = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> 
> What is SIZE here?
> 
>>         if (!p) {
>>                 perror("fail to get memory");
>>                 exit(-1);
>>         }
>>
>>         c = (unsigned char *)p;
>>
>>         for (i = 0; i < SIZE / 8; i += 8)
>>                 *(p + i) = 0xffff + i;
> 
> Err... what's your intent here? I think you're writting to 1 in every 8 longs?
> Probably just write to the first byte of every page.
> 

The data is fixed for the purpose of analyzing zram compression, so I filled
some data here.

> Thanks,
> Ryan
> 
>>
>>         while (1)
>>                 sleep(10);
>>
>>         return 0;
>> }
>>
>> Thanks,
>> wenchao
>>
> 

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Dev Jain]

On 16/12/24 9:28 pm, Wenchao Hao wrote:
> On 2024/12/5 1:05, Ryan Roberts wrote:
>> On 04/12/2024 14:40, Wenchao Hao wrote:
>>> On 2024/12/3 22:42, Ryan Roberts wrote:
>>>> On 03/12/2024 14:17, David Hildenbrand wrote:
>>>>> On 03.12.24 14:49, Wenchao Hao wrote:
>>>>>> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
>>>>>> each VMA, but it does not include large pages smaller than PMD size.
>>>>>>
>>>>>> This patch adds the statistics of anonymous huge pages allocated by
>>>>>> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
>>>>>>
>>>>>> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
>>>>>> ---
>>>>>>    fs/proc/task_mmu.c | 6 ++++++
>>>>>>    1 file changed, 6 insertions(+)
>>>>>>
>>>>>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>>>>>> index 38a5a3e9cba2..b655011627d8 100644
>>>>>> --- a/fs/proc/task_mmu.c
>>>>>> +++ b/fs/proc/task_mmu.c
>>>>>> @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss,
>>>>>> struct page *page,
>>>>>>            if (!folio_test_swapbacked(folio) && !dirty &&
>>>>>>                !folio_test_dirty(folio))
>>>>>>                mss->lazyfree += size;
>>>>>> +
>>>>>> +        /*
>>>>>> +         * Count large pages smaller than PMD size to anonymous_thp
>>>>>> +         */
>>>>>> +        if (!compound && PageHead(page) && folio_order(folio))
>>>>>> +            mss->anonymous_thp += folio_size(folio);
>>>>>>        }
>>>>>>          if (folio_test_ksm(folio))
>>>>>
>>>>> I think we decided to leave this (and /proc/meminfo) be one of the last
>>>>> interfaces where this is only concerned with PMD-sized ones:
>>>>>
>>>>> Documentation/admin-guide/mm/transhuge.rst:
>>>>>
>>>>> The number of PMD-sized anonymous transparent huge pages currently used by the
>>>>> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
>>>>> To identify what applications are using PMD-sized anonymous transparent huge
>>>>> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
>>>>> fields for each mapping. (Note that AnonHugePages only applies to traditional
>>>>> PMD-sized THP for historical reasons and should have been called
>>>>> AnonHugePmdMapped).
>>>>>
>>>> Agreed. If you need per-process metrics for mTHP, we have a python script at
>>>> tools/mm/thpmaps which does a fairly good job of parsing pagemap. --help gives
>>>> you all the options.
>>>>
>>> I tried this tool, and it is very powerful and practical IMO.
>>> However, thereare two disadvantages:
>>>
>>> - This tool is heavily dependent on Python and Python libraries.
>>>    After installing several libraries with the pip command, I was able to
>>>    get it running.
>> I think numpy is the only package it uses which is not in the standard library?
>> What other libraries did you need to install?
>>
> Yes, I just tested it on the standard version (Fedora), and that is indeed the case.
> Previously, I needed to install additional packages is because I removed some unused
> software from the old environment.
>
> Recently, I revisited and started using your tool again. It’s very useful, meeting
> my needs and even exceeding them. I am now testing with qemu to run a fedora, so
> it's easy to run it.
>
>>>    In practice, the environment we need to analyze may be a mobile or
>>>    embedded environment, where it is very difficult to deploy these
>>>    libraries.
>> Yes, I agree that's a problem, especially for Android. The script has proven
>> useful to me for debugging in a traditional Linux distro environment though.
>>
>>> - It seems that this tool only counts file-backed large pages? During
>> No; the tool counts file-backed and anon memory. But it reports it in separate
>> counters. See `thpmaps --help` for full details.
>>
>>>    the actual test, I mapped a region of anonymous pages and mapped it
>>>    as large pages, but the tool did not display those large pages.
>>>    Below is my test file(mTHP related sysfs interface is set to "always"
>>>    to make sure using large pages):
>> Which mTHP sizes did you enable? Depending on your value of SIZE and which mTHP
>> sizes are enabled, you may not have a correctly aligned region in p. So mTHP
>> would not be allocated. Best to over-allocate then explicitly align p to the
>> mTHP size, then fault it in.
>>
> I enabled 64k/128k/256k MTHP and have been studying, debugging, and changing
> parts of the khugepaged code to try merging standard pages into mTHP large
> pages. So, I wanted to use smap to observe the large page sizes in a process.

You can try my RFC khugepaged series for that ;)

>
>>> int main()
>>> {
>>>          int i;
>>>          char *c;
>>>          unsigned long *p;
>>>
>>>          p = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>> What is SIZE here?
>>
>>>          if (!p) {
>>>                  perror("fail to get memory");
>>>                  exit(-1);
>>>          }
>>>
>>>          c = (unsigned char *)p;
>>>
>>>          for (i = 0; i < SIZE / 8; i += 8)
>>>                  *(p + i) = 0xffff + i;
>> Err... what's your intent here? I think you're writting to 1 in every 8 longs?
>> Probably just write to the first byte of every page.
>>
> The data is fixed for the purpose of analyzing zram compression, so I filled
> some data here.
>
>> Thanks,
>> Ryan
>>
>>>          while (1)
>>>                  sleep(10);
>>>
>>>          return 0;
>>> }
>>>
>>> Thanks,
>>> wenchao
>>>
>

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Wenchao Hao]

On 2024/12/3 22:17, David Hildenbrand wrote:
> On 03.12.24 14:49, Wenchao Hao wrote:
>> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
>> each VMA, but it does not include large pages smaller than PMD size.
>>
>> This patch adds the statistics of anonymous huge pages allocated by
>> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
>>
>> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
>> ---
>>   fs/proc/task_mmu.c | 6 ++++++
>>   1 file changed, 6 insertions(+)
>>
>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>> index 38a5a3e9cba2..b655011627d8 100644
>> --- a/fs/proc/task_mmu.c
>> +++ b/fs/proc/task_mmu.c
>> @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
>>           if (!folio_test_swapbacked(folio) && !dirty &&
>>               !folio_test_dirty(folio))
>>               mss->lazyfree += size;
>> +
>> +        /*
>> +         * Count large pages smaller than PMD size to anonymous_thp
>> +         */
>> +        if (!compound && PageHead(page) && folio_order(folio))
>> +            mss->anonymous_thp += folio_size(folio);
>>       }
>>         if (folio_test_ksm(folio))
> 
> 
> I think we decided to leave this (and /proc/meminfo) be one of the last
> interfaces where this is only concerned with PMD-sized ones:
> 

Could you explain why?

When analyzing the impact of mTHP on performance, we need to understand
how many pages in the process are actually present as large pages.
By comparing this value with the actual memory usage of the process,
we can analyze the large page allocation success rate of the process,
and further investigate the situation of khugepaged. If the actual
proportion of large pages is low, the performance of the process may
be affected, which could be directly reflected in the high number of
TLB misses and page faults.

However, currently, only PMD-sized large pages are being counted, 
which is insufficient.

> Documentation/admin-guide/mm/transhuge.rst:
> 
> The number of PMD-sized anonymous transparent huge pages currently used by the
> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
> To identify what applications are using PMD-sized anonymous transparent huge
> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
> fields for each mapping. (Note that AnonHugePages only applies to traditional
> PMD-sized THP for historical reasons and should have been called
> AnonHugePmdMapped).
> 

Maybe rename this field, then AnonHugePages contains huge page of mTHP?

Thanks,
wenchao

> 
> 

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[David Hildenbrand]

On 04.12.24 15:30, Wenchao Hao wrote:
> On 2024/12/3 22:17, David Hildenbrand wrote:
>> On 03.12.24 14:49, Wenchao Hao wrote:
>>> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
>>> each VMA, but it does not include large pages smaller than PMD size.
>>>
>>> This patch adds the statistics of anonymous huge pages allocated by
>>> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
>>>
>>> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
>>> ---
>>>    fs/proc/task_mmu.c | 6 ++++++
>>>    1 file changed, 6 insertions(+)
>>>
>>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>>> index 38a5a3e9cba2..b655011627d8 100644
>>> --- a/fs/proc/task_mmu.c
>>> +++ b/vim 
>>> @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
>>>            if (!folio_test_swapbacked(folio) && !dirty &&
>>>                !folio_test_dirty(folio))
>>>                mss->lazyfree += size;
>>> +
>>> +        /*
>>> +         * Count large pages smaller than PMD size to anonymous_thp
>>> +         */
>>> +        if (!compound && PageHead(page) && folio_order(folio))
>>> +            mss->anonymous_thp += folio_size(folio);
>>>        }
>>>          if (folio_test_ksm(folio))
>>
>>
>> I think we decided to leave this (and /proc/meminfo) be one of the last
>> interfaces where this is only concerned with PMD-sized ones:
>>
> 
> Could you explain why?
> 
> When analyzing the impact of mTHP on performance, we need to understand
> how many pages in the process are actually present as large pages.
> By comparing this value with the actual memory usage of the process,
> we can analyze the large page allocation success rate of the process,
> and further investigate the situation of khugepaged. If the actual
> proportion of large pages is low, the performance of the process may
> be affected, which could be directly reflected in the high number of
> TLB misses and page faults.
> 
> However, currently, only PMD-sized large pages are being counted,
> which is insufficient.

As Ryan said, we have scripts to analyze that. We did not come to a 
conclusion yet how to handle smaps stats differently -- and whether we 
want to at all.

> 
>> Documentation/admin-guide/mm/transhuge.rst:
>>
>> The number of PMD-sized anonymous transparent huge pages currently used by the
>> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
>> To identify what applications are using PMD-sized anonymous transparent huge
>> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
>> fields for each mapping. (Note that AnonHugePages only applies to traditional
>> PMD-sized THP for historical reasons and should have been called
>> AnonHugePmdMapped).
>>
> 
> Maybe rename this field, then AnonHugePages contains huge page of mTHP?

It has the potential of breaking existing user space, which is why we 
didn't look into that yet.

AnonHugePmdMapped would be a lot cleaner, and could be added 
independently. It would be required as a first step.

-- 
Cheers,

David / dhildenb

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Wenchao Hao]

On 2024/12/4 22:37, David Hildenbrand wrote:
> On 04.12.24 15:30, Wenchao Hao wrote:
>> On 2024/12/3 22:17, David Hildenbrand wrote:
>>> On 03.12.24 14:49, Wenchao Hao wrote:
>>>> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
>>>> each VMA, but it does not include large pages smaller than PMD size.
>>>>
>>>> This patch adds the statistics of anonymous huge pages allocated by
>>>> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
>>>>
>>>> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
>>>> ---
>>>>    fs/proc/task_mmu.c | 6 ++++++
>>>>    1 file changed, 6 insertions(+)
>>>>
>>>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>>>> index 38a5a3e9cba2..b655011627d8 100644
>>>> --- a/fs/proc/task_mmu.c
>>>> +++ b/vim @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
>>>>            if (!folio_test_swapbacked(folio) && !dirty &&
>>>>                !folio_test_dirty(folio))
>>>>                mss->lazyfree += size;
>>>> +
>>>> +        /*
>>>> +         * Count large pages smaller than PMD size to anonymous_thp
>>>> +         */
>>>> +        if (!compound && PageHead(page) && folio_order(folio))
>>>> +            mss->anonymous_thp += folio_size(folio);
>>>>        }
>>>>          if (folio_test_ksm(folio))
>>>
>>>
>>> I think we decided to leave this (and /proc/meminfo) be one of the last
>>> interfaces where this is only concerned with PMD-sized ones:
>>>
>>
>> Could you explain why?
>>
>> When analyzing the impact of mTHP on performance, we need to understand
>> how many pages in the process are actually present as large pages.
>> By comparing this value with the actual memory usage of the process,
>> we can analyze the large page allocation success rate of the process,
>> and further investigate the situation of khugepaged. If the actual
>> proportion of large pages is low, the performance of the process may
>> be affected, which could be directly reflected in the high number of
>> TLB misses and page faults.
>>
>> However, currently, only PMD-sized large pages are being counted,
>> which is insufficient.
> 
> As Ryan said, we have scripts to analyze that. We did not come to a conclusion yet how to handle smaps stats differently -- and whether we want to at all.
> 

Hi David,

I replied Ryan about few disadvantages of the scripts. The scripts
is not helpful for my scenario.

>>
>>> Documentation/admin-guide/mm/transhuge.rst:
>>>
>>> The number of PMD-sized anonymous transparent huge pages currently used by the
>>> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
>>> To identify what applications are using PMD-sized anonymous transparent huge
>>> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
>>> fields for each mapping. (Note that AnonHugePages only applies to traditional
>>> PMD-sized THP for historical reasons and should have been called
>>> AnonHugePmdMapped).
>>>
>>
>> Maybe rename this field, then AnonHugePages contains huge page of mTHP?
> 
> It has the potential of breaking existing user space, which is why we didn't look into that yet.
> 

Got it.

> AnonHugePmdMapped would be a lot cleaner, and could be added independently. It would be required as a first step.
> 

While, if the meaning of AnonHugePages remains unchanged, simply adding a new field doesn't
seem to have any practical significance.

Thanks

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Ryan Roberts]

+ Dev Jain

On 04/12/2024 14:30, Wenchao Hao wrote:
> On 2024/12/3 22:17, David Hildenbrand wrote:
>> On 03.12.24 14:49, Wenchao Hao wrote:
>>> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
>>> each VMA, but it does not include large pages smaller than PMD size.
>>>
>>> This patch adds the statistics of anonymous huge pages allocated by
>>> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
>>>
>>> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
>>> ---
>>>   fs/proc/task_mmu.c | 6 ++++++
>>>   1 file changed, 6 insertions(+)
>>>
>>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>>> index 38a5a3e9cba2..b655011627d8 100644
>>> --- a/fs/proc/task_mmu.c
>>> +++ b/fs/proc/task_mmu.c
>>> @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
>>>           if (!folio_test_swapbacked(folio) && !dirty &&
>>>               !folio_test_dirty(folio))
>>>               mss->lazyfree += size;
>>> +
>>> +        /*
>>> +         * Count large pages smaller than PMD size to anonymous_thp
>>> +         */
>>> +        if (!compound && PageHead(page) && folio_order(folio))
>>> +            mss->anonymous_thp += folio_size(folio);
>>>       }
>>>         if (folio_test_ksm(folio))
>>
>>
>> I think we decided to leave this (and /proc/meminfo) be one of the last
>> interfaces where this is only concerned with PMD-sized ones:
>>
> 
> Could you explain why?
> 
> When analyzing the impact of mTHP on performance, we need to understand
> how many pages in the process are actually present as large pages.
> By comparing this value with the actual memory usage of the process,
> we can analyze the large page allocation success rate of the process,
> and further investigate the situation of khugepaged. If the actual

Note that khugepaged does not yet support collapse to mTHP sizes. Dev Jain
(CCed) is working on an implementation for that now. If you are planning to look
at this area, you might want to chat first.

> proportion of large pages is low, the performance of the process may
> be affected, which could be directly reflected in the high number of
> TLB misses and page faults.
> 
> However, currently, only PMD-sized large pages are being counted, 
> which is insufficient.
> 
>> Documentation/admin-guide/mm/transhuge.rst:
>>
>> The number of PMD-sized anonymous transparent huge pages currently used by the
>> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
>> To identify what applications are using PMD-sized anonymous transparent huge
>> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
>> fields for each mapping. (Note that AnonHugePages only applies to traditional
>> PMD-sized THP for historical reasons and should have been called
>> AnonHugePmdMapped).
>>
> 
> Maybe rename this field, then AnonHugePages contains huge page of mTHP?
> 
> Thanks,
> wenchao
> 
>>
>>
> 

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Lance Yang]

On Tue, Dec 3, 2024 at 10:17 PM David Hildenbrand <david@redhat.com> wrote:
>
> On 03.12.24 14:49, Wenchao Hao wrote:
> > Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
> > each VMA, but it does not include large pages smaller than PMD size.
> >
> > This patch adds the statistics of anonymous huge pages allocated by
> > mTHP which is smaller than PMD size to AnonHugePages field in smaps.
> >
> > Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
> > ---
> >   fs/proc/task_mmu.c | 6 ++++++
> >   1 file changed, 6 insertions(+)
> >
> > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > index 38a5a3e9cba2..b655011627d8 100644
> > --- a/fs/proc/task_mmu.c
> > +++ b/fs/proc/task_mmu.c
> > @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
> >               if (!folio_test_swapbacked(folio) && !dirty &&
> >                   !folio_test_dirty(folio))
> >                       mss->lazyfree += size;
> > +
> > +             /*
> > +              * Count large pages smaller than PMD size to anonymous_thp
> > +              */
> > +             if (!compound && PageHead(page) && folio_order(folio))
> > +                     mss->anonymous_thp += folio_size(folio);
> >       }
> >
> >       if (folio_test_ksm(folio))
>
>
> I think we decided to leave this (and /proc/meminfo) be one of the last
> interfaces where this is only concerned with PMD-sized ones:
>
> Documentation/admin-guide/mm/transhuge.rst:
>
> The number of PMD-sized anonymous transparent huge pages currently used by the
> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
> To identify what applications are using PMD-sized anonymous transparent huge
> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
> fields for each mapping. (Note that AnonHugePages only applies to traditional
> PMD-sized THP for historical reasons and should have been called
> AnonHugePmdMapped).

Yeah, I think we need to keep AnonHugePages unchanged within these interfaces
due to historical reasons ;)

Perhaps, there might be another way to count all THP allocated for each process.

Thanks,
Lance


>
>
>
> --
> Cheers,
>
> David / dhildenb
>
>

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Barry Song]

On Fri, Dec 6, 2024 at 7:16 PM Lance Yang <ioworker0@gmail.com> wrote:
>
> On Tue, Dec 3, 2024 at 10:17 PM David Hildenbrand <david@redhat.com> wrote:
> >
> > On 03.12.24 14:49, Wenchao Hao wrote:
> > > Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
> > > each VMA, but it does not include large pages smaller than PMD size.
> > >
> > > This patch adds the statistics of anonymous huge pages allocated by
> > > mTHP which is smaller than PMD size to AnonHugePages field in smaps.
> > >
> > > Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
> > > ---
> > >   fs/proc/task_mmu.c | 6 ++++++
> > >   1 file changed, 6 insertions(+)
> > >
> > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> > > index 38a5a3e9cba2..b655011627d8 100644
> > > --- a/fs/proc/task_mmu.c
> > > +++ b/fs/proc/task_mmu.c
> > > @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
> > >               if (!folio_test_swapbacked(folio) && !dirty &&
> > >                   !folio_test_dirty(folio))
> > >                       mss->lazyfree += size;
> > > +
> > > +             /*
> > > +              * Count large pages smaller than PMD size to anonymous_thp
> > > +              */
> > > +             if (!compound && PageHead(page) && folio_order(folio))
> > > +                     mss->anonymous_thp += folio_size(folio);
> > >       }
> > >
> > >       if (folio_test_ksm(folio))
> >
> >
> > I think we decided to leave this (and /proc/meminfo) be one of the last
> > interfaces where this is only concerned with PMD-sized ones:
> >
> > Documentation/admin-guide/mm/transhuge.rst:
> >
> > The number of PMD-sized anonymous transparent huge pages currently used by the
> > system is available by reading the AnonHugePages field in ``/proc/meminfo``.
> > To identify what applications are using PMD-sized anonymous transparent huge
> > pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
> > fields for each mapping. (Note that AnonHugePages only applies to traditional
> > PMD-sized THP for historical reasons and should have been called
> > AnonHugePmdMapped).
>
> Yeah, I think we need to keep AnonHugePages unchanged within these interfaces
> due to historical reasons ;)
>
> Perhaps, there might be another way to count all THP allocated for each process.

My point is that counting the THP allocations per process doesn't seem
as important
when compared to the overall system's status. We already have
interfaces to track
the following:

* The number of mTHPs allocated or fallback events;
* The total number of anonymous mTHP folios in the system.
* The total number of partially unmapped mTHP folios in the system.

To me, knowing the details for each process doesn’t seem particularly
critical for
profiling.  To be honest, I don't see a need for this at all, except perhaps for
debugging to verify if mTHP is present.

If feasible, we could explore converting Ryan's Python script into a native
C program. I believe this would be more than sufficient for embedded systems
and Android.

>
> Thanks,
> Lance
>
>
> >
> >
> >
> > --
> > Cheers,
> >
> > David / dhildenb

Thanks
Barry

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Ryan Roberts]

On 08/12/2024 06:06, Barry Song wrote:
> On Fri, Dec 6, 2024 at 7:16 PM Lance Yang <ioworker0@gmail.com> wrote:
>>
>> On Tue, Dec 3, 2024 at 10:17 PM David Hildenbrand <david@redhat.com> wrote:
>>>
>>> On 03.12.24 14:49, Wenchao Hao wrote:
>>>> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
>>>> each VMA, but it does not include large pages smaller than PMD size.
>>>>
>>>> This patch adds the statistics of anonymous huge pages allocated by
>>>> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
>>>>
>>>> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
>>>> ---
>>>>   fs/proc/task_mmu.c | 6 ++++++
>>>>   1 file changed, 6 insertions(+)
>>>>
>>>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>>>> index 38a5a3e9cba2..b655011627d8 100644
>>>> --- a/fs/proc/task_mmu.c
>>>> +++ b/fs/proc/task_mmu.c
>>>> @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
>>>>               if (!folio_test_swapbacked(folio) && !dirty &&
>>>>                   !folio_test_dirty(folio))
>>>>                       mss->lazyfree += size;
>>>> +
>>>> +             /*
>>>> +              * Count large pages smaller than PMD size to anonymous_thp
>>>> +              */
>>>> +             if (!compound && PageHead(page) && folio_order(folio))
>>>> +                     mss->anonymous_thp += folio_size(folio);
>>>>       }
>>>>
>>>>       if (folio_test_ksm(folio))
>>>
>>>
>>> I think we decided to leave this (and /proc/meminfo) be one of the last
>>> interfaces where this is only concerned with PMD-sized ones:
>>>
>>> Documentation/admin-guide/mm/transhuge.rst:
>>>
>>> The number of PMD-sized anonymous transparent huge pages currently used by the
>>> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
>>> To identify what applications are using PMD-sized anonymous transparent huge
>>> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
>>> fields for each mapping. (Note that AnonHugePages only applies to traditional
>>> PMD-sized THP for historical reasons and should have been called
>>> AnonHugePmdMapped).
>>
>> Yeah, I think we need to keep AnonHugePages unchanged within these interfaces
>> due to historical reasons ;)
>>
>> Perhaps, there might be another way to count all THP allocated for each process.
> 
> My point is that counting the THP allocations per process doesn't seem
> as important
> when compared to the overall system's status. We already have
> interfaces to track
> the following:
> 
> * The number of mTHPs allocated or fallback events;
> * The total number of anonymous mTHP folios in the system.
> * The total number of partially unmapped mTHP folios in the system.

I think an important missing piece here is "what percentage of memory that could
be mTHP is allocated as mTHP?" The script gives you that, which I think is useful.

> 
> To me, knowing the details for each process doesn’t seem particularly
> critical for
> profiling.  To be honest, I don't see a need for this at all, except perhaps for
> debugging to verify if mTHP is present.
> 
> If feasible, we could explore converting Ryan's Python script into a native
> C program. I believe this would be more than sufficient for embedded systems
> and Android.

Agreed. The kernel already fundamentally provides all the required info via
pagemap, kpageflags and smaps. So don't think we need to add anything new to the
kernel.

Thanks,
Ryan


> 
>>
>> Thanks,
>> Lance
>>
>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>>
>>> David / dhildenb
> 
> Thanks
> Barry

Re: [PATCH] smaps: count large pages smaller than PMD size to anonymous_thp

[Wenchao Hao]

On 2024/12/8 14:06, Barry Song wrote:
> On Fri, Dec 6, 2024 at 7:16 PM Lance Yang <ioworker0@gmail.com> wrote:
>>
>> On Tue, Dec 3, 2024 at 10:17 PM David Hildenbrand <david@redhat.com> wrote:
>>>
>>> On 03.12.24 14:49, Wenchao Hao wrote:
>>>> Currently, /proc/xxx/smaps reports the size of anonymous huge pages for
>>>> each VMA, but it does not include large pages smaller than PMD size.
>>>>
>>>> This patch adds the statistics of anonymous huge pages allocated by
>>>> mTHP which is smaller than PMD size to AnonHugePages field in smaps.
>>>>
>>>> Signed-off-by: Wenchao Hao <haowenchao22@gmail.com>
>>>> ---
>>>>   fs/proc/task_mmu.c | 6 ++++++
>>>>   1 file changed, 6 insertions(+)
>>>>
>>>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>>>> index 38a5a3e9cba2..b655011627d8 100644
>>>> --- a/fs/proc/task_mmu.c
>>>> +++ b/fs/proc/task_mmu.c
>>>> @@ -717,6 +717,12 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
>>>>               if (!folio_test_swapbacked(folio) && !dirty &&
>>>>                   !folio_test_dirty(folio))
>>>>                       mss->lazyfree += size;
>>>> +
>>>> +             /*
>>>> +              * Count large pages smaller than PMD size to anonymous_thp
>>>> +              */
>>>> +             if (!compound && PageHead(page) && folio_order(folio))
>>>> +                     mss->anonymous_thp += folio_size(folio);
>>>>       }
>>>>
>>>>       if (folio_test_ksm(folio))
>>>
>>>
>>> I think we decided to leave this (and /proc/meminfo) be one of the last
>>> interfaces where this is only concerned with PMD-sized ones:
>>>
>>> Documentation/admin-guide/mm/transhuge.rst:
>>>
>>> The number of PMD-sized anonymous transparent huge pages currently used by the
>>> system is available by reading the AnonHugePages field in ``/proc/meminfo``.
>>> To identify what applications are using PMD-sized anonymous transparent huge
>>> pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
>>> fields for each mapping. (Note that AnonHugePages only applies to traditional
>>> PMD-sized THP for historical reasons and should have been called
>>> AnonHugePmdMapped).
>>
>> Yeah, I think we need to keep AnonHugePages unchanged within these interfaces
>> due to historical reasons ;)
>>
>> Perhaps, there might be another way to count all THP allocated for each process.
> 
> My point is that counting the THP allocations per process doesn't seem
> as important
> when compared to the overall system's status. We already have
> interfaces to track
> the following:
> 
> * The number of mTHPs allocated or fallback events;
> * The total number of anonymous mTHP folios in the system.
> * The total number of partially unmapped mTHP folios in the system.
> 
> To me, knowing the details for each process doesn’t seem particularly
> critical for
> profiling.  To be honest, I don't see a need for this at all, except perhaps for
> debugging to verify if mTHP is present.
> 
> If feasible, we could explore converting Ryan's Python script into a native
> C program. I believe this would be more than sufficient for embedded systems
> and Android.
> 

Hi Barry,

Yes, the reason I want to use smap to collect this data is that I wasn’t familiar
with this tool before. When analyzing the performance impact of enabling mTHP, I want
to understand the actual memory usage of the process being analyzed, including the
proportions of anonymous pages, swap pages, large pages and so on.
This helps determine whether the test results align with expectations.

Indeed, the main purpose of adding this is to make debugging more convenient.
For now, I’ll perform the analysis and testing on the Fedora distribution, so I
can use the pyrhon tool directly.

If it becomes unavoidable to run this tool on embedded devices in the future, I may
take the time to create a simplified version of the analysis tool in C based on this
script.

>>
>> Thanks,
>> Lance
>>
>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>>
>>> David / dhildenb
> 
> Thanks
> Barry