[PATCH] mm/page_alloc: Consider PCP pages as part of pfmemalloc_reserve

[PATCH] mm/page_alloc: Consider PCP pages as part of pfmemalloc_reserve

[zhongjinji]

When free_pages becomes critically low, the kernel prevents other tasks
from entering the slow path to ensure that reclaiming tasks can
successfully allocate memory.

This blocking is important to avoid memory contention with reclaiming
tasks. However, in some cases it is unnecessary because the PCP list may
already contain sufficient pages, as freed pages are first placed there
and are not immediately visible to the buddy system.

By accounting PCP pages as part of pfmemalloc_reserve, we can reduce
unnecessary blocking and improve system responsiveness under low-memory
conditions.

Signed-off-by: zhongjinji <zhongjinji@honor.com>
---
 mm/internal.h   |  1 +
 mm/page_alloc.c | 14 ++++++++++++++
 mm/vmscan.c     |  1 +
 3 files changed, 16 insertions(+)

diff --git a/mm/internal.h b/mm/internal.h
index 45b725c3dc03..c8fcee51d662 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -842,6 +842,7 @@ static inline struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned int ord
 #define alloc_frozen_pages(...) \
 	alloc_hooks(alloc_frozen_pages_noprof(__VA_ARGS__))
 
+extern int zone_pcp_pages_count(struct zone *zone);
 extern void zone_pcp_reset(struct zone *zone);
 extern void zone_pcp_disable(struct zone *zone);
 extern void zone_pcp_enable(struct zone *zone);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 1999eb7e7c14..e34031946adb 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7134,6 +7134,20 @@ void zone_pcp_reset(struct zone *zone)
 	}
 }
 
+int zone_pcp_pages_count(struct zone *zone)
+{
+	struct per_cpu_pages *pcp;
+	int total_pcp_pages = 0;
+	int cpu;
+
+	for_each_online_cpu(cpu) {
+		pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu);
+		total_pcp_pages += pcp->count;
+	}
+
+	return total_pcp_pages;
+}
+
 #ifdef CONFIG_MEMORY_HOTREMOVE
 /*
  * All pages in the range must be in a single zone, must not contain holes,
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 674999999cd0..148f452d9cf5 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -6498,6 +6498,7 @@ static bool allow_direct_reclaim(pg_data_t *pgdat)
 
 		pfmemalloc_reserve += min_wmark_pages(zone);
 		free_pages += zone_page_state_snapshot(zone, NR_FREE_PAGES);
+		free_pages += zone_pcp_pages_count(zone);
 	}
 
 	/* If there are no reserves (unexpected config) then do not throttle */
-- 
2.17.1

Re: [PATCH] mm/page_alloc: Consider PCP pages as part of pfmemalloc_reserve

[Joshua Hahn]

On Tue, 21 Oct 2025 17:50:04 +0800 zhongjinji <zhongjinji@honor.com> wrote:

Hello Zhongjinji, thank you for your patch!

> When free_pages becomes critically low, the kernel prevents other tasks
> from entering the slow path to ensure that reclaiming tasks can
> successfully allocate memory.
> 
> This blocking is important to avoid memory contention with reclaiming
> tasks. However, in some cases it is unnecessary because the PCP list may
> already contain sufficient pages, as freed pages are first placed there
> and are not immediately visible to the buddy system.

Based on my limiting understanding of pcp free pages, I had a concern here
on whether this would really provide the desired effect. That is, the pages
in the pcp are not available to the buddy allocator unless we drain the pcp
lists (and this operation is not free), I was unsure if there was a clear
benefit to allowing the system to go unblocked.

If we are already at the point where we need the pcp pages to have enough
free pages to go over the watermark, perhaps it makes sense to just block
tasks for now, and enter direct reclaim? Allowing more allocations might
lead the system to be in a worse state than before, and will have to
go through direct reclaim anyways.

Please let me know if this makes sense! 
 
> By accounting PCP pages as part of pfmemalloc_reserve, we can reduce
> unnecessary blocking and improve system responsiveness under low-memory
> conditions.
> 
> Signed-off-by: zhongjinji <zhongjinji@honor.com>

[...snip...]

> +int zone_pcp_pages_count(struct zone *zone)
> +{
> +	struct per_cpu_pages *pcp;
> +	int total_pcp_pages = 0;
> +	int cpu;
> +
> +	for_each_online_cpu(cpu) {
> +		pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu);
> +		total_pcp_pages += pcp->count;

Could this be racy? What is stopping the pcp count from decreasing while we
are iterating over each online cpu, over each managed zone? Under the
memory pressure conditions that this patch is aiming to fix, I think that
there is a good chance the numer we get here will be very outdated by the time
we try to take action based on it, and we may require the system to be
further stalled since we don't take action to reclaim memory.

[...snip...]

Please feel free to let me know if I am missing something obvious. Again,
I am not very familiar with the pcp code, so there is a good chance that
you are seeing something that I am not : -)

Thank you for the patch, I hope you have a great day!
Joshua

Re: [PATCH] mm/page_alloc: Consider PCP pages as part of

[jinji zhong]

> Zhongjinji, thank you for your patch!
>
>> When free_pages becomes critically low, the kernel prevents other tasks
>> from entering the slow path to ensure that reclaiming tasks can
>> successfully allocate memory.
>> 
>> This blocking is important to avoid memory contention with reclaiming
>> tasks. However, in some cases it is unnecessary because the PCP list may
>> already contain sufficient pages, as freed pages are first placed there
>> and are not immediately visible to the buddy system.
>
>Based on my limiting understanding of pcp free pages, I had a concern here
>on whether this would really provide the desired effect. That is, the pages
>in the pcp are not available to the buddy allocator unless we drain the pcp
>lists (and this operation is not free), I was unsure if there was a clear
>benefit to allowing the system to go unblocked.

The purpose of this patch is to account for the memory in the pcp list
within memalloc_reserve, which would allow more tasks to enter the slow
path. For critical tasks, entering the slow path represents a better
alternative than being throttled until kswapd wakes up.

>If we are already at the point where we need the pcp pages to have enough
>free pages to go over the watermark, perhaps it makes sense to just block
>tasks for now, and enter direct reclaim? Allowing more allocations might
>lead the system to be in a worse state than before, and will have to
>go through direct reclaim anyways.
>
>Please let me know if this makes sense!
> 
>> By accounting PCP pages as part of pfmemalloc_reserve, we can reduce
>> unnecessary blocking and improve system responsiveness under low-memory
>> conditions.
>> 
>> Signed-off-by: zhongjinji <zhongjinji@honor.com>
>
>[...snip...]
>
>> +int zone_pcp_pages_count(struct zone *zone)
>> +{
>> +	struct per_cpu_pages *pcp;
>> +	int total_pcp_pages = 0;
>> +	int cpu;
>> +
>> +	for_each_online_cpu(cpu) {
>> +		pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu);
>> +		total_pcp_pages += pcp->count;
>
>Could this be racy? What is stopping the pcp count from decreasing while we
>are iterating over each online cpu, over each managed zone? Under the
>memory pressure conditions that this patch is aiming to fix, I think that
>there is a good chance the numer we get here will be very outdated by the time
>we try to take action based on it, and we may require the system to be
>further stalled since we don't take action to reclaim memory.

Thank you, Joshua. Indeed, the pcp->count might be outdated by the time. 
And kswapd will fail to allocate memory if the pages in the pcp lists are
on other CPUs. While drain_all_pages() may be triggered by direct reclaim
on other CPUs, some hard-to-predict scenarios might still exist.

Perhaps performing drain_all_pages() before actually calling
throttle_direct_reclaim() would be better.

Like the following code.

@@ -6535,6 +6535,7 @@ static bool allow_direct_reclaim(pg_data_t *pgdat)
 static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
 					nodemask_t *nodemask)
 {
+	bool drained = false;
 	struct zoneref *z;
 	struct zone *zone;
 	pg_data_t *pgdat = NULL;
@@ -6570,6 +6571,7 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
 	 * for remote pfmemalloc reserves and processes on different nodes
 	 * should make reasonable progress.
 	 */
+retry:
 	for_each_zone_zonelist_nodemask(zone, z, zonelist,
 					gfp_zone(gfp_mask), nodemask) {
 		if (zone_idx(zone) > ZONE_NORMAL)
@@ -6586,6 +6588,12 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
 	if (!pgdat)
 		goto out;
 
+	if (!drained) {
+		drained = true;
+		drain_all_pages(NULL);
+		goto retry;
+	}
+
 	/* Account for the throttling */
 	count_vm_event(PGSCAN_DIRECT_THROTTLE);

>[...snip...]
>
>Please feel free to let me know if I am missing something obvious. Again,
>I am not very familiar with the pcp code, so there is a good chance that
>you are seeing something that I am not : -)
>
>Thank you for the patch, I hope you have a great day!
>Joshua
>

Re: [PATCH] mm/page_alloc: Consider PCP pages as part of

[Vlastimil Babka]

On 10/21/25 20:04, jinji zhong wrote:
>> Zhongjinji, thank you for your patch!

Hi,

>>> When free_pages becomes critically low, the kernel prevents other tasks
>>> from entering the slow path to ensure that reclaiming tasks can
>>> successfully allocate memory.
>>> 
>>> This blocking is important to avoid memory contention with reclaiming
>>> tasks. However, in some cases it is unnecessary because the PCP list may
>>> already contain sufficient pages, as freed pages are first placed there
>>> and are not immediately visible to the buddy system.
>>
>>Based on my limiting understanding of pcp free pages, I had a concern here
>>on whether this would really provide the desired effect. That is, the pages
>>in the pcp are not available to the buddy allocator unless we drain the pcp
>>lists (and this operation is not free), I was unsure if there was a clear
>>benefit to allowing the system to go unblocked.
> 
> The purpose of this patch is to account for the memory in the pcp list
> within memalloc_reserve, which would allow more tasks to enter the slow
> path. For critical tasks, entering the slow path represents a better
> alternative than being throttled until kswapd wakes up.

Were you able to measure the benefits of the patch? What are the "critical
tasks"? Usually such task use mlockall() to avoid allocating memory during
operation. Maybe we should instead extempt some task priority classes from
being throttled?

>>If we are already at the point where we need the pcp pages to have enough
>>free pages to go over the watermark, perhaps it makes sense to just block
>>tasks for now, and enter direct reclaim? Allowing more allocations might
>>lead the system to be in a worse state than before, and will have to
>>go through direct reclaim anyways.
>>
>>Please let me know if this makes sense!
>> 
>>> By accounting PCP pages as part of pfmemalloc_reserve, we can reduce
>>> unnecessary blocking and improve system responsiveness under low-memory
>>> conditions.
>>> 
>>> Signed-off-by: zhongjinji <zhongjinji@honor.com>
>>
>>[...snip...]
>>
>>> +int zone_pcp_pages_count(struct zone *zone)
>>> +{
>>> +	struct per_cpu_pages *pcp;
>>> +	int total_pcp_pages = 0;
>>> +	int cpu;
>>> +
>>> +	for_each_online_cpu(cpu) {
>>> +		pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu);
>>> +		total_pcp_pages += pcp->count;
>>
>>Could this be racy? What is stopping the pcp count from decreasing while we
>>are iterating over each online cpu, over each managed zone? Under the
>>memory pressure conditions that this patch is aiming to fix, I think that
>>there is a good chance the numer we get here will be very outdated by the time
>>we try to take action based on it, and we may require the system to be
>>further stalled since we don't take action to reclaim memory.
> 
> Thank you, Joshua. Indeed, the pcp->count might be outdated by the time. 
> And kswapd will fail to allocate memory if the pages in the pcp lists are
> on other CPUs. While drain_all_pages() may be triggered by direct reclaim
> on other CPUs, some hard-to-predict scenarios might still exist.
> 
> Perhaps performing drain_all_pages() before actually calling
> throttle_direct_reclaim() would be better.

Draining makes more sense indeed. But it should be backed by data showing
improvement. As you say, draining is probably already happening via direct
reclaim.

> Like the following code.
> 
> @@ -6535,6 +6535,7 @@ static bool allow_direct_reclaim(pg_data_t *pgdat)
>  static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
>  					nodemask_t *nodemask)
>  {
> +	bool drained = false;
>  	struct zoneref *z;
>  	struct zone *zone;
>  	pg_data_t *pgdat = NULL;
> @@ -6570,6 +6571,7 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
>  	 * for remote pfmemalloc reserves and processes on different nodes
>  	 * should make reasonable progress.
>  	 */
> +retry:
>  	for_each_zone_zonelist_nodemask(zone, z, zonelist,
>  					gfp_zone(gfp_mask), nodemask) {
>  		if (zone_idx(zone) > ZONE_NORMAL)
> @@ -6586,6 +6588,12 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
>  	if (!pgdat)
>  		goto out;
>  
> +	if (!drained) {
> +		drained = true;
> +		drain_all_pages(NULL);
> +		goto retry;
> +	}
> +
>  	/* Account for the throttling */
>  	count_vm_event(PGSCAN_DIRECT_THROTTLE);
> 
>>[...snip...]
>>
>>Please feel free to let me know if I am missing something obvious. Again,
>>I am not very familiar with the pcp code, so there is a good chance that
>>you are seeing something that I am not : -)
>>
>>Thank you for the patch, I hope you have a great day!
>>Joshua
>>