[PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios

[PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios

[Baolin Wang]

When I tested the mincore() syscall, I observed that it takes longer with
64K mTHP enabled on my Arm64 server. The reason is the mincore_pte_range()
still checks each PTE individually, even when the PTEs are contiguous,
which is not efficient.

Thus we can use pte_batch_hint() to get the batch number of the present
contiguous PTEs, which can improve the performance. I tested the mincore()
syscall with 1G anonymous memory populated with 64K mTHP, and observed an
obvious performance improvement:

w/o patch		w/ patch		changes
6022us			549us			+91%

Moreover, I also tested mincore() with disabling mTHP/THP, and did not
see any obvious regression for base pages.

Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
---
Changes from v2:
- Re-calculate the max_nr, per Barry.
Changes from v1:
- Change to use pte_batch_hint() to get the batch number, per Ryan.

Note: I observed the min_t() can introduce a slight performance regression
for base pages, so I change to add a batch size check for base pages,
which can resolve the performance regression issue.
---
 mm/mincore.c | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/mm/mincore.c b/mm/mincore.c
index 832f29f46767..42d6c9c8da86 100644
--- a/mm/mincore.c
+++ b/mm/mincore.c
@@ -21,6 +21,7 @@
 
 #include <linux/uaccess.h>
 #include "swap.h"
+#include "internal.h"
 
 static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr,
 			unsigned long end, struct mm_walk *walk)
@@ -105,6 +106,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 	pte_t *ptep;
 	unsigned char *vec = walk->private;
 	int nr = (end - addr) >> PAGE_SHIFT;
+	int step, i;
 
 	ptl = pmd_trans_huge_lock(pmd, vma);
 	if (ptl) {
@@ -118,16 +120,26 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 		walk->action = ACTION_AGAIN;
 		return 0;
 	}
-	for (; addr != end; ptep++, addr += PAGE_SIZE) {
+	for (; addr != end; ptep += step, addr += step * PAGE_SIZE) {
 		pte_t pte = ptep_get(ptep);
 
+		step = 1;
 		/* We need to do cache lookup too for pte markers */
 		if (pte_none_mostly(pte))
 			__mincore_unmapped_range(addr, addr + PAGE_SIZE,
 						 vma, vec);
-		else if (pte_present(pte))
-			*vec = 1;
-		else { /* pte is a swap entry */
+		else if (pte_present(pte)) {
+			unsigned int batch = pte_batch_hint(ptep, pte);
+
+			if (batch > 1) {
+				unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
+
+				step = min_t(unsigned int, batch, max_nr);
+			}
+
+			for (i = 0; i < step; i++)
+				vec[i] = 1;
+		} else { /* pte is a swap entry */
 			swp_entry_t entry = pte_to_swp_entry(pte);
 
 			if (non_swap_entry(entry)) {
@@ -146,7 +158,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 #endif
 			}
 		}
-		vec++;
+		vec += step;
 	}
 	pte_unmap_unlock(ptep - 1, ptl);
 out:
-- 
2.43.5

Re: [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios

[Barry Song]

On Fri, May 9, 2025 at 12:45 PM Baolin Wang
<baolin.wang@linux.alibaba.com> wrote:
>
> When I tested the mincore() syscall, I observed that it takes longer with
> 64K mTHP enabled on my Arm64 server. The reason is the mincore_pte_range()
> still checks each PTE individually, even when the PTEs are contiguous,
> which is not efficient.
>
> Thus we can use pte_batch_hint() to get the batch number of the present
> contiguous PTEs, which can improve the performance. I tested the mincore()
> syscall with 1G anonymous memory populated with 64K mTHP, and observed an
> obvious performance improvement:
>
> w/o patch               w/ patch                changes
> 6022us                  549us                   +91%
>
> Moreover, I also tested mincore() with disabling mTHP/THP, and did not
> see any obvious regression for base pages.
>
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>

Reviewed-by: Barry Song <baohua@kernel.org>

> ---
> Changes from v2:
> - Re-calculate the max_nr, per Barry.
> Changes from v1:
> - Change to use pte_batch_hint() to get the batch number, per Ryan.
>
> Note: I observed the min_t() can introduce a slight performance regression
> for base pages, so I change to add a batch size check for base pages,
> which can resolve the performance regression issue.
> ---
>  mm/mincore.c | 22 +++++++++++++++++-----
>  1 file changed, 17 insertions(+), 5 deletions(-)
>
> diff --git a/mm/mincore.c b/mm/mincore.c
> index 832f29f46767..42d6c9c8da86 100644
> --- a/mm/mincore.c
> +++ b/mm/mincore.c
> @@ -21,6 +21,7 @@
>
>  #include <linux/uaccess.h>
>  #include "swap.h"
> +#include "internal.h"
>
>  static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr,
>                         unsigned long end, struct mm_walk *walk)
> @@ -105,6 +106,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>         pte_t *ptep;
>         unsigned char *vec = walk->private;
>         int nr = (end - addr) >> PAGE_SHIFT;
> +       int step, i;
>
>         ptl = pmd_trans_huge_lock(pmd, vma);
>         if (ptl) {
> @@ -118,16 +120,26 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>                 walk->action = ACTION_AGAIN;
>                 return 0;
>         }
> -       for (; addr != end; ptep++, addr += PAGE_SIZE) {
> +       for (; addr != end; ptep += step, addr += step * PAGE_SIZE) {
>                 pte_t pte = ptep_get(ptep);
>
> +               step = 1;
>                 /* We need to do cache lookup too for pte markers */
>                 if (pte_none_mostly(pte))
>                         __mincore_unmapped_range(addr, addr + PAGE_SIZE,
>                                                  vma, vec);
> -               else if (pte_present(pte))
> -                       *vec = 1;
> -               else { /* pte is a swap entry */
> +               else if (pte_present(pte)) {
> +                       unsigned int batch = pte_batch_hint(ptep, pte);
> +
> +                       if (batch > 1) {
> +                               unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
> +
> +                               step = min_t(unsigned int, batch, max_nr);
> +                       }
> +
> +                       for (i = 0; i < step; i++)
> +                               vec[i] = 1;
> +               } else { /* pte is a swap entry */
>                         swp_entry_t entry = pte_to_swp_entry(pte);
>
>                         if (non_swap_entry(entry)) {
> @@ -146,7 +158,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>  #endif
>                         }
>                 }
> -               vec++;
> +               vec += step;
>         }
>         pte_unmap_unlock(ptep - 1, ptl);
>  out:
> --
> 2.43.5
>

Re: [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios

[Dev Jain]

On 09/05/25 6:15 am, Baolin Wang wrote:
> When I tested the mincore() syscall, I observed that it takes longer with
> 64K mTHP enabled on my Arm64 server. The reason is the mincore_pte_range()
> still checks each PTE individually, even when the PTEs are contiguous,
> which is not efficient.
> 
> Thus we can use pte_batch_hint() to get the batch number of the present
> contiguous PTEs, which can improve the performance. I tested the mincore()
> syscall with 1G anonymous memory populated with 64K mTHP, and observed an
> obvious performance improvement:
> 
> w/o patch		w/ patch		changes
> 6022us			549us			+91%
> 
> Moreover, I also tested mincore() with disabling mTHP/THP, and did not
> see any obvious regression for base pages.
> 
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>

Nit: The subject line - s/pte_batch_bint()/pte_batch_hint()
Otherwise LGTM

Reviewed-by: Dev Jain <dev.jain@arm.com>

> ---
> Changes from v2:
> - Re-calculate the max_nr, per Barry.
> Changes from v1:
> - Change to use pte_batch_hint() to get the batch number, per Ryan.
> 
> Note: I observed the min_t() can introduce a slight performance regression
> for base pages, so I change to add a batch size check for base pages,
> which can resolve the performance regression issue.
> ---
>   mm/mincore.c | 22 +++++++++++++++++-----
>   1 file changed, 17 insertions(+), 5 deletions(-)
> 
> diff --git a/mm/mincore.c b/mm/mincore.c
> index 832f29f46767..42d6c9c8da86 100644
> --- a/mm/mincore.c
> +++ b/mm/mincore.c
> @@ -21,6 +21,7 @@
>   
>   #include <linux/uaccess.h>
>   #include "swap.h"
> +#include "internal.h"
>   
>   static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr,
>   			unsigned long end, struct mm_walk *walk)
> @@ -105,6 +106,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>   	pte_t *ptep;
>   	unsigned char *vec = walk->private;
>   	int nr = (end - addr) >> PAGE_SHIFT;
> +	int step, i;
>   
>   	ptl = pmd_trans_huge_lock(pmd, vma);
>   	if (ptl) {
> @@ -118,16 +120,26 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>   		walk->action = ACTION_AGAIN;
>   		return 0;
>   	}
> -	for (; addr != end; ptep++, addr += PAGE_SIZE) {
> +	for (; addr != end; ptep += step, addr += step * PAGE_SIZE) {
>   		pte_t pte = ptep_get(ptep);
>   
> +		step = 1;
>   		/* We need to do cache lookup too for pte markers */
>   		if (pte_none_mostly(pte))
>   			__mincore_unmapped_range(addr, addr + PAGE_SIZE,
>   						 vma, vec);
> -		else if (pte_present(pte))
> -			*vec = 1;
> -		else { /* pte is a swap entry */
> +		else if (pte_present(pte)) {
> +			unsigned int batch = pte_batch_hint(ptep, pte);
> +
> +			if (batch > 1) {
> +				unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
> +
> +				step = min_t(unsigned int, batch, max_nr);
> +			}
> +
> +			for (i = 0; i < step; i++)
> +				vec[i] = 1;
> +		} else { /* pte is a swap entry */
>   			swp_entry_t entry = pte_to_swp_entry(pte);
>   
>   			if (non_swap_entry(entry)) {
> @@ -146,7 +158,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>   #endif
>   			}
>   		}
> -		vec++;
> +		vec += step;
>   	}
>   	pte_unmap_unlock(ptep - 1, ptl);
>   out:

Re: [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios

[Baolin Wang]

On 2025/5/9 15:30, Dev Jain wrote:
> 
> 
> On 09/05/25 6:15 am, Baolin Wang wrote:
>> When I tested the mincore() syscall, I observed that it takes longer with
>> 64K mTHP enabled on my Arm64 server. The reason is the 
>> mincore_pte_range()
>> still checks each PTE individually, even when the PTEs are contiguous,
>> which is not efficient.
>>
>> Thus we can use pte_batch_hint() to get the batch number of the present
>> contiguous PTEs, which can improve the performance. I tested the 
>> mincore()
>> syscall with 1G anonymous memory populated with 64K mTHP, and observed an
>> obvious performance improvement:
>>
>> w/o patch        w/ patch        changes
>> 6022us            549us            +91%
>>
>> Moreover, I also tested mincore() with disabling mTHP/THP, and did not
>> see any obvious regression for base pages.
>>
>> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> 
> Nit: The subject line - s/pte_batch_bint()/pte_batch_hint()

Ah, fat finger. Hope Andrew can help to fix it:)

> Otherwise LGTM
> 
> Reviewed-by: Dev Jain <dev.jain@arm.com>

Thanks.

Re: [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios

[David Hildenbrand]

On 09.05.25 02:45, Baolin Wang wrote:
> When I tested the mincore() syscall, I observed that it takes longer with
> 64K mTHP enabled on my Arm64 server. The reason is the mincore_pte_range()
> still checks each PTE individually, even when the PTEs are contiguous,
> which is not efficient.
> 
> Thus we can use pte_batch_hint() to get the batch number of the present
> contiguous PTEs, which can improve the performance. I tested the mincore()
> syscall with 1G anonymous memory populated with 64K mTHP, and observed an
> obvious performance improvement:
> 
> w/o patch		w/ patch		changes
> 6022us			549us			+91%
> 
> Moreover, I also tested mincore() with disabling mTHP/THP, and did not
> see any obvious regression for base pages.
> 
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> ---
> Changes from v2:
> - Re-calculate the max_nr, per Barry.
> Changes from v1:
> - Change to use pte_batch_hint() to get the batch number, per Ryan.
> 
> Note: I observed the min_t() can introduce a slight performance regression
> for base pages, so I change to add a batch size check for base pages,
> which can resolve the performance regression issue.
> ---
>   mm/mincore.c | 22 +++++++++++++++++-----
>   1 file changed, 17 insertions(+), 5 deletions(-)
> 
> diff --git a/mm/mincore.c b/mm/mincore.c
> index 832f29f46767..42d6c9c8da86 100644
> --- a/mm/mincore.c
> +++ b/mm/mincore.c
> @@ -21,6 +21,7 @@
>   
>   #include <linux/uaccess.h>
>   #include "swap.h"
> +#include "internal.h"
>   
>   static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr,
>   			unsigned long end, struct mm_walk *walk)
> @@ -105,6 +106,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>   	pte_t *ptep;
>   	unsigned char *vec = walk->private;
>   	int nr = (end - addr) >> PAGE_SHIFT;
> +	int step, i;
>   
>   	ptl = pmd_trans_huge_lock(pmd, vma);
>   	if (ptl) {
> @@ -118,16 +120,26 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
>   		walk->action = ACTION_AGAIN;
>   		return 0;
>   	}
> -	for (; addr != end; ptep++, addr += PAGE_SIZE) {
> +	for (; addr != end; ptep += step, addr += step * PAGE_SIZE) {
>   		pte_t pte = ptep_get(ptep);
>   
> +		step = 1;
>   		/* We need to do cache lookup too for pte markers */
>   		if (pte_none_mostly(pte))
>   			__mincore_unmapped_range(addr, addr + PAGE_SIZE,
>   						 vma, vec);
> -		else if (pte_present(pte))
> -			*vec = 1;
> -		else { /* pte is a swap entry */
> +		else if (pte_present(pte)) {
> +			unsigned int batch = pte_batch_hint(ptep, pte);
> +
> +			if (batch > 1) {
> +				unsigned int max_nr = (end - addr) >> PAGE_SHIFT;

Nit: probably would have called this max_step to match step.

> +
> +				step = min_t(unsigned int, batch, max_nr);
> +			}
> +
> +			for (i = 0; i < step; i++)
> +				vec[i] = 1;

I'm surprised this micro-optimization matters that much. Probably the compiler
defers the calculation of max_nr. I am not convinced we need that level of
micro-optimization in this code ...


But if we're already micro-optimizing, you could have optimized out the loop as
well for order-0:

	unsigned int batch = pte_batch_hint(ptep, pte);
	
	if (batch > 1) {
		unsigned int max_nr = (end - addr) >> PAGE_SHIFT;

		step = min_t(unsigned int, batch, max_nr);
		for (i = 0; i < step; i++)
			vec[i] = 1;
	} else {
		*vec = 1;
	}


In any case

Acked-by: David Hildenbrand <david@redhat.com>

-- 
Cheers,

David / dhildenb

Re: [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios

[Baolin Wang]

On 2025/5/9 15:51, David Hildenbrand wrote:
> On 09.05.25 02:45, Baolin Wang wrote:
>> When I tested the mincore() syscall, I observed that it takes longer with
>> 64K mTHP enabled on my Arm64 server. The reason is the 
>> mincore_pte_range()
>> still checks each PTE individually, even when the PTEs are contiguous,
>> which is not efficient.
>>
>> Thus we can use pte_batch_hint() to get the batch number of the present
>> contiguous PTEs, which can improve the performance. I tested the 
>> mincore()
>> syscall with 1G anonymous memory populated with 64K mTHP, and observed an
>> obvious performance improvement:
>>
>> w/o patch        w/ patch        changes
>> 6022us            549us            +91%
>>
>> Moreover, I also tested mincore() with disabling mTHP/THP, and did not
>> see any obvious regression for base pages.
>>
>> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
>> ---
>> Changes from v2:
>> - Re-calculate the max_nr, per Barry.
>> Changes from v1:
>> - Change to use pte_batch_hint() to get the batch number, per Ryan.
>>
>> Note: I observed the min_t() can introduce a slight performance 
>> regression
>> for base pages, so I change to add a batch size check for base pages,
>> which can resolve the performance regression issue.
>> ---
>>   mm/mincore.c | 22 +++++++++++++++++-----
>>   1 file changed, 17 insertions(+), 5 deletions(-)
>>
>> diff --git a/mm/mincore.c b/mm/mincore.c
>> index 832f29f46767..42d6c9c8da86 100644
>> --- a/mm/mincore.c
>> +++ b/mm/mincore.c
>> @@ -21,6 +21,7 @@
>>   #include <linux/uaccess.h>
>>   #include "swap.h"
>> +#include "internal.h"
>>   static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned 
>> long addr,
>>               unsigned long end, struct mm_walk *walk)
>> @@ -105,6 +106,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned 
>> long addr, unsigned long end,
>>       pte_t *ptep;
>>       unsigned char *vec = walk->private;
>>       int nr = (end - addr) >> PAGE_SHIFT;
>> +    int step, i;
>>       ptl = pmd_trans_huge_lock(pmd, vma);
>>       if (ptl) {
>> @@ -118,16 +120,26 @@ static int mincore_pte_range(pmd_t *pmd, 
>> unsigned long addr, unsigned long end,
>>           walk->action = ACTION_AGAIN;
>>           return 0;
>>       }
>> -    for (; addr != end; ptep++, addr += PAGE_SIZE) {
>> +    for (; addr != end; ptep += step, addr += step * PAGE_SIZE) {
>>           pte_t pte = ptep_get(ptep);
>> +        step = 1;
>>           /* We need to do cache lookup too for pte markers */
>>           if (pte_none_mostly(pte))
>>               __mincore_unmapped_range(addr, addr + PAGE_SIZE,
>>                            vma, vec);
>> -        else if (pte_present(pte))
>> -            *vec = 1;
>> -        else { /* pte is a swap entry */
>> +        else if (pte_present(pte)) {
>> +            unsigned int batch = pte_batch_hint(ptep, pte);
>> +
>> +            if (batch > 1) {
>> +                unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
> 
> Nit: probably would have called this max_step to match step.

OK. If need respin the patch, I'll rename it.

>> +
>> +                step = min_t(unsigned int, batch, max_nr);
>> +            }
>> +
>> +            for (i = 0; i < step; i++)
>> +                vec[i] = 1;
> 
> I'm surprised this micro-optimization matters that much. Probably the 

Me too.

> compiler
> defers the calculation of max_nr. I am not convinced we need that level of
> micro-optimization in this code ...
> 
> 
> But if we're already micro-optimizing, you could have optimized out the 
> loop as
> well for order-0:
> 
>      unsigned int batch = pte_batch_hint(ptep, pte);
> 
>      if (batch > 1) {
>          unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
> 
>          step = min_t(unsigned int, batch, max_nr);
>          for (i = 0; i < step; i++)
>              vec[i] = 1;
>      } else {
>          *vec = 1;
>      }

I tried this method, and it had no impact on performance.

> In any case
> 
> Acked-by: David Hildenbrand <david@redhat.com>

Thanks.

Re: [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios

[David Hildenbrand]

> 
> I tried this method, and it had no impact on performance.
>

Might be that the compiler already did that for us.

-- 
Cheers,

David / dhildenb