* Re: [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios
2025-05-09 0:45 [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios Baolin Wang
@ 2025-05-09 1:49 ` Barry Song
2025-05-09 7:30 ` Dev Jain
2025-05-09 7:51 ` David Hildenbrand
2 siblings, 0 replies; 7+ messages in thread
From: Barry Song @ 2025-05-09 1:49 UTC (permalink / raw)
To: Baolin Wang
Cc: akpm, david, ryan.roberts, dev.jain, ziy, linux-mm, linux-kernel
On Fri, May 9, 2025 at 12:45 PM Baolin Wang
<baolin.wang@linux.alibaba.com> wrote:
>
> When I tested the mincore() syscall, I observed that it takes longer with
> 64K mTHP enabled on my Arm64 server. The reason is the mincore_pte_range()
> still checks each PTE individually, even when the PTEs are contiguous,
> which is not efficient.
>
> Thus we can use pte_batch_hint() to get the batch number of the present
> contiguous PTEs, which can improve the performance. I tested the mincore()
> syscall with 1G anonymous memory populated with 64K mTHP, and observed an
> obvious performance improvement:
>
> w/o patch w/ patch changes
> 6022us 549us +91%
>
> Moreover, I also tested mincore() with disabling mTHP/THP, and did not
> see any obvious regression for base pages.
>
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Barry Song <baohua@kernel.org>
> ---
> Changes from v2:
> - Re-calculate the max_nr, per Barry.
> Changes from v1:
> - Change to use pte_batch_hint() to get the batch number, per Ryan.
>
> Note: I observed the min_t() can introduce a slight performance regression
> for base pages, so I change to add a batch size check for base pages,
> which can resolve the performance regression issue.
> ---
> mm/mincore.c | 22 +++++++++++++++++-----
> 1 file changed, 17 insertions(+), 5 deletions(-)
>
> diff --git a/mm/mincore.c b/mm/mincore.c
> index 832f29f46767..42d6c9c8da86 100644
> --- a/mm/mincore.c
> +++ b/mm/mincore.c
> @@ -21,6 +21,7 @@
>
> #include <linux/uaccess.h>
> #include "swap.h"
> +#include "internal.h"
>
> static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr,
> unsigned long end, struct mm_walk *walk)
> @@ -105,6 +106,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
> pte_t *ptep;
> unsigned char *vec = walk->private;
> int nr = (end - addr) >> PAGE_SHIFT;
> + int step, i;
>
> ptl = pmd_trans_huge_lock(pmd, vma);
> if (ptl) {
> @@ -118,16 +120,26 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
> walk->action = ACTION_AGAIN;
> return 0;
> }
> - for (; addr != end; ptep++, addr += PAGE_SIZE) {
> + for (; addr != end; ptep += step, addr += step * PAGE_SIZE) {
> pte_t pte = ptep_get(ptep);
>
> + step = 1;
> /* We need to do cache lookup too for pte markers */
> if (pte_none_mostly(pte))
> __mincore_unmapped_range(addr, addr + PAGE_SIZE,
> vma, vec);
> - else if (pte_present(pte))
> - *vec = 1;
> - else { /* pte is a swap entry */
> + else if (pte_present(pte)) {
> + unsigned int batch = pte_batch_hint(ptep, pte);
> +
> + if (batch > 1) {
> + unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
> +
> + step = min_t(unsigned int, batch, max_nr);
> + }
> +
> + for (i = 0; i < step; i++)
> + vec[i] = 1;
> + } else { /* pte is a swap entry */
> swp_entry_t entry = pte_to_swp_entry(pte);
>
> if (non_swap_entry(entry)) {
> @@ -146,7 +158,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
> #endif
> }
> }
> - vec++;
> + vec += step;
> }
> pte_unmap_unlock(ptep - 1, ptl);
> out:
> --
> 2.43.5
>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios
2025-05-09 0:45 [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios Baolin Wang
2025-05-09 1:49 ` Barry Song
@ 2025-05-09 7:30 ` Dev Jain
2025-05-09 7:38 ` Baolin Wang
2025-05-09 7:51 ` David Hildenbrand
2 siblings, 1 reply; 7+ messages in thread
From: Dev Jain @ 2025-05-09 7:30 UTC (permalink / raw)
To: Baolin Wang, akpm, david
Cc: 21cnbao, ryan.roberts, ziy, linux-mm, linux-kernel
On 09/05/25 6:15 am, Baolin Wang wrote:
> When I tested the mincore() syscall, I observed that it takes longer with
> 64K mTHP enabled on my Arm64 server. The reason is the mincore_pte_range()
> still checks each PTE individually, even when the PTEs are contiguous,
> which is not efficient.
>
> Thus we can use pte_batch_hint() to get the batch number of the present
> contiguous PTEs, which can improve the performance. I tested the mincore()
> syscall with 1G anonymous memory populated with 64K mTHP, and observed an
> obvious performance improvement:
>
> w/o patch w/ patch changes
> 6022us 549us +91%
>
> Moreover, I also tested mincore() with disabling mTHP/THP, and did not
> see any obvious regression for base pages.
>
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Nit: The subject line - s/pte_batch_bint()/pte_batch_hint()
Otherwise LGTM
Reviewed-by: Dev Jain <dev.jain@arm.com>
> ---
> Changes from v2:
> - Re-calculate the max_nr, per Barry.
> Changes from v1:
> - Change to use pte_batch_hint() to get the batch number, per Ryan.
>
> Note: I observed the min_t() can introduce a slight performance regression
> for base pages, so I change to add a batch size check for base pages,
> which can resolve the performance regression issue.
> ---
> mm/mincore.c | 22 +++++++++++++++++-----
> 1 file changed, 17 insertions(+), 5 deletions(-)
>
> diff --git a/mm/mincore.c b/mm/mincore.c
> index 832f29f46767..42d6c9c8da86 100644
> --- a/mm/mincore.c
> +++ b/mm/mincore.c
> @@ -21,6 +21,7 @@
>
> #include <linux/uaccess.h>
> #include "swap.h"
> +#include "internal.h"
>
> static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr,
> unsigned long end, struct mm_walk *walk)
> @@ -105,6 +106,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
> pte_t *ptep;
> unsigned char *vec = walk->private;
> int nr = (end - addr) >> PAGE_SHIFT;
> + int step, i;
>
> ptl = pmd_trans_huge_lock(pmd, vma);
> if (ptl) {
> @@ -118,16 +120,26 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
> walk->action = ACTION_AGAIN;
> return 0;
> }
> - for (; addr != end; ptep++, addr += PAGE_SIZE) {
> + for (; addr != end; ptep += step, addr += step * PAGE_SIZE) {
> pte_t pte = ptep_get(ptep);
>
> + step = 1;
> /* We need to do cache lookup too for pte markers */
> if (pte_none_mostly(pte))
> __mincore_unmapped_range(addr, addr + PAGE_SIZE,
> vma, vec);
> - else if (pte_present(pte))
> - *vec = 1;
> - else { /* pte is a swap entry */
> + else if (pte_present(pte)) {
> + unsigned int batch = pte_batch_hint(ptep, pte);
> +
> + if (batch > 1) {
> + unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
> +
> + step = min_t(unsigned int, batch, max_nr);
> + }
> +
> + for (i = 0; i < step; i++)
> + vec[i] = 1;
> + } else { /* pte is a swap entry */
> swp_entry_t entry = pte_to_swp_entry(pte);
>
> if (non_swap_entry(entry)) {
> @@ -146,7 +158,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
> #endif
> }
> }
> - vec++;
> + vec += step;
> }
> pte_unmap_unlock(ptep - 1, ptl);
> out:
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios
2025-05-09 7:30 ` Dev Jain
@ 2025-05-09 7:38 ` Baolin Wang
0 siblings, 0 replies; 7+ messages in thread
From: Baolin Wang @ 2025-05-09 7:38 UTC (permalink / raw)
To: Dev Jain, akpm, david; +Cc: 21cnbao, ryan.roberts, ziy, linux-mm, linux-kernel
On 2025/5/9 15:30, Dev Jain wrote:
>
>
> On 09/05/25 6:15 am, Baolin Wang wrote:
>> When I tested the mincore() syscall, I observed that it takes longer with
>> 64K mTHP enabled on my Arm64 server. The reason is the
>> mincore_pte_range()
>> still checks each PTE individually, even when the PTEs are contiguous,
>> which is not efficient.
>>
>> Thus we can use pte_batch_hint() to get the batch number of the present
>> contiguous PTEs, which can improve the performance. I tested the
>> mincore()
>> syscall with 1G anonymous memory populated with 64K mTHP, and observed an
>> obvious performance improvement:
>>
>> w/o patch w/ patch changes
>> 6022us 549us +91%
>>
>> Moreover, I also tested mincore() with disabling mTHP/THP, and did not
>> see any obvious regression for base pages.
>>
>> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
>
> Nit: The subject line - s/pte_batch_bint()/pte_batch_hint()
Ah, fat finger. Hope Andrew can help to fix it:)
> Otherwise LGTM
>
> Reviewed-by: Dev Jain <dev.jain@arm.com>
Thanks.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios
2025-05-09 0:45 [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios Baolin Wang
2025-05-09 1:49 ` Barry Song
2025-05-09 7:30 ` Dev Jain
@ 2025-05-09 7:51 ` David Hildenbrand
2025-05-09 12:25 ` Baolin Wang
2 siblings, 1 reply; 7+ messages in thread
From: David Hildenbrand @ 2025-05-09 7:51 UTC (permalink / raw)
To: Baolin Wang, akpm
Cc: 21cnbao, ryan.roberts, dev.jain, ziy, linux-mm, linux-kernel
On 09.05.25 02:45, Baolin Wang wrote:
> When I tested the mincore() syscall, I observed that it takes longer with
> 64K mTHP enabled on my Arm64 server. The reason is the mincore_pte_range()
> still checks each PTE individually, even when the PTEs are contiguous,
> which is not efficient.
>
> Thus we can use pte_batch_hint() to get the batch number of the present
> contiguous PTEs, which can improve the performance. I tested the mincore()
> syscall with 1G anonymous memory populated with 64K mTHP, and observed an
> obvious performance improvement:
>
> w/o patch w/ patch changes
> 6022us 549us +91%
>
> Moreover, I also tested mincore() with disabling mTHP/THP, and did not
> see any obvious regression for base pages.
>
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> ---
> Changes from v2:
> - Re-calculate the max_nr, per Barry.
> Changes from v1:
> - Change to use pte_batch_hint() to get the batch number, per Ryan.
>
> Note: I observed the min_t() can introduce a slight performance regression
> for base pages, so I change to add a batch size check for base pages,
> which can resolve the performance regression issue.
> ---
> mm/mincore.c | 22 +++++++++++++++++-----
> 1 file changed, 17 insertions(+), 5 deletions(-)
>
> diff --git a/mm/mincore.c b/mm/mincore.c
> index 832f29f46767..42d6c9c8da86 100644
> --- a/mm/mincore.c
> +++ b/mm/mincore.c
> @@ -21,6 +21,7 @@
>
> #include <linux/uaccess.h>
> #include "swap.h"
> +#include "internal.h"
>
> static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr,
> unsigned long end, struct mm_walk *walk)
> @@ -105,6 +106,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
> pte_t *ptep;
> unsigned char *vec = walk->private;
> int nr = (end - addr) >> PAGE_SHIFT;
> + int step, i;
>
> ptl = pmd_trans_huge_lock(pmd, vma);
> if (ptl) {
> @@ -118,16 +120,26 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
> walk->action = ACTION_AGAIN;
> return 0;
> }
> - for (; addr != end; ptep++, addr += PAGE_SIZE) {
> + for (; addr != end; ptep += step, addr += step * PAGE_SIZE) {
> pte_t pte = ptep_get(ptep);
>
> + step = 1;
> /* We need to do cache lookup too for pte markers */
> if (pte_none_mostly(pte))
> __mincore_unmapped_range(addr, addr + PAGE_SIZE,
> vma, vec);
> - else if (pte_present(pte))
> - *vec = 1;
> - else { /* pte is a swap entry */
> + else if (pte_present(pte)) {
> + unsigned int batch = pte_batch_hint(ptep, pte);
> +
> + if (batch > 1) {
> + unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
Nit: probably would have called this max_step to match step.
> +
> + step = min_t(unsigned int, batch, max_nr);
> + }
> +
> + for (i = 0; i < step; i++)
> + vec[i] = 1;
I'm surprised this micro-optimization matters that much. Probably the compiler
defers the calculation of max_nr. I am not convinced we need that level of
micro-optimization in this code ...
But if we're already micro-optimizing, you could have optimized out the loop as
well for order-0:
unsigned int batch = pte_batch_hint(ptep, pte);
if (batch > 1) {
unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
step = min_t(unsigned int, batch, max_nr);
for (i = 0; i < step; i++)
vec[i] = 1;
} else {
*vec = 1;
}
In any case
Acked-by: David Hildenbrand <david@redhat.com>
--
Cheers,
David / dhildenb
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH v3] mm: mincore: use pte_batch_bint() to batch process large folios
2025-05-09 7:51 ` David Hildenbrand
@ 2025-05-09 12:25 ` Baolin Wang
2025-05-09 12:42 ` David Hildenbrand
0 siblings, 1 reply; 7+ messages in thread
From: Baolin Wang @ 2025-05-09 12:25 UTC (permalink / raw)
To: David Hildenbrand, akpm
Cc: 21cnbao, ryan.roberts, dev.jain, ziy, linux-mm, linux-kernel
On 2025/5/9 15:51, David Hildenbrand wrote:
> On 09.05.25 02:45, Baolin Wang wrote:
>> When I tested the mincore() syscall, I observed that it takes longer with
>> 64K mTHP enabled on my Arm64 server. The reason is the
>> mincore_pte_range()
>> still checks each PTE individually, even when the PTEs are contiguous,
>> which is not efficient.
>>
>> Thus we can use pte_batch_hint() to get the batch number of the present
>> contiguous PTEs, which can improve the performance. I tested the
>> mincore()
>> syscall with 1G anonymous memory populated with 64K mTHP, and observed an
>> obvious performance improvement:
>>
>> w/o patch w/ patch changes
>> 6022us 549us +91%
>>
>> Moreover, I also tested mincore() with disabling mTHP/THP, and did not
>> see any obvious regression for base pages.
>>
>> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
>> ---
>> Changes from v2:
>> - Re-calculate the max_nr, per Barry.
>> Changes from v1:
>> - Change to use pte_batch_hint() to get the batch number, per Ryan.
>>
>> Note: I observed the min_t() can introduce a slight performance
>> regression
>> for base pages, so I change to add a batch size check for base pages,
>> which can resolve the performance regression issue.
>> ---
>> mm/mincore.c | 22 +++++++++++++++++-----
>> 1 file changed, 17 insertions(+), 5 deletions(-)
>>
>> diff --git a/mm/mincore.c b/mm/mincore.c
>> index 832f29f46767..42d6c9c8da86 100644
>> --- a/mm/mincore.c
>> +++ b/mm/mincore.c
>> @@ -21,6 +21,7 @@
>> #include <linux/uaccess.h>
>> #include "swap.h"
>> +#include "internal.h"
>> static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned
>> long addr,
>> unsigned long end, struct mm_walk *walk)
>> @@ -105,6 +106,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned
>> long addr, unsigned long end,
>> pte_t *ptep;
>> unsigned char *vec = walk->private;
>> int nr = (end - addr) >> PAGE_SHIFT;
>> + int step, i;
>> ptl = pmd_trans_huge_lock(pmd, vma);
>> if (ptl) {
>> @@ -118,16 +120,26 @@ static int mincore_pte_range(pmd_t *pmd,
>> unsigned long addr, unsigned long end,
>> walk->action = ACTION_AGAIN;
>> return 0;
>> }
>> - for (; addr != end; ptep++, addr += PAGE_SIZE) {
>> + for (; addr != end; ptep += step, addr += step * PAGE_SIZE) {
>> pte_t pte = ptep_get(ptep);
>> + step = 1;
>> /* We need to do cache lookup too for pte markers */
>> if (pte_none_mostly(pte))
>> __mincore_unmapped_range(addr, addr + PAGE_SIZE,
>> vma, vec);
>> - else if (pte_present(pte))
>> - *vec = 1;
>> - else { /* pte is a swap entry */
>> + else if (pte_present(pte)) {
>> + unsigned int batch = pte_batch_hint(ptep, pte);
>> +
>> + if (batch > 1) {
>> + unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
>
> Nit: probably would have called this max_step to match step.
OK. If need respin the patch, I'll rename it.
>> +
>> + step = min_t(unsigned int, batch, max_nr);
>> + }
>> +
>> + for (i = 0; i < step; i++)
>> + vec[i] = 1;
>
> I'm surprised this micro-optimization matters that much. Probably the
Me too.
> compiler
> defers the calculation of max_nr. I am not convinced we need that level of
> micro-optimization in this code ...
>
>
> But if we're already micro-optimizing, you could have optimized out the
> loop as
> well for order-0:
>
> unsigned int batch = pte_batch_hint(ptep, pte);
>
> if (batch > 1) {
> unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
>
> step = min_t(unsigned int, batch, max_nr);
> for (i = 0; i < step; i++)
> vec[i] = 1;
> } else {
> *vec = 1;
> }
I tried this method, and it had no impact on performance.
> In any case
>
> Acked-by: David Hildenbrand <david@redhat.com>
Thanks.
^ permalink raw reply [flat|nested] 7+ messages in thread