[PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[Lance Yang]

From: Lance Yang <lance.yang@linux.dev>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
it does not make sense to try to detect whether it is underused, because
try_to_map_unused_to_zeropage(), called while splitting the folio, will not
actually replace any zeroed pages by the shared zeropage.

Splitting the folio in that case does not make any sense, so let's not even
scan to check if the folio is underused.

Suggested-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Lance Yang <lance.yang@linux.dev>
---
v1 -> v2:
 - Update subject and changelog (per David, thanks!)
 - https://lore.kernel.org/linux-mm/20250908040713.42960-1-lance.yang@linux.dev

 mm/huge_memory.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 77f0c3417973..6d99893c18e8 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4183,6 +4183,13 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
 		bool underused = false;
 
 		if (!folio_test_partially_mapped(folio)) {
+			/*
+			 * See try_to_map_unused_to_zeropage(): we cannot
+			 * optimize zero-filled pages after splitting an
+			 * mlocked folio.
+			 */
+			if (folio_test_mlocked(folio))
+				goto next;
 			underused = thp_underused(folio);
 			if (!underused)
 				goto next;
-- 
2.49.0

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[David Hildenbrand]

On 08.09.25 11:07, Lance Yang wrote:
> From: Lance Yang <lance.yang@linux.dev>
> 
> When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
> it does not make sense to try to detect whether it is underused, because
> try_to_map_unused_to_zeropage(), called while splitting the folio, will not
> actually replace any zeroed pages by the shared zeropage.
> 
> Splitting the folio in that case does not make any sense, so let's not even
> scan to check if the folio is underused.
> 
> Suggested-by: David Hildenbrand <david@redhat.com>

Feel free to drop that, I only tweaked you description :)

Acked-by: David Hildenbrand <david@redhat.com>

-- 
Cheers

David / dhildenb

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[Lance Yang]

On 2025/9/8 17:12, David Hildenbrand wrote:
> On 08.09.25 11:07, Lance Yang wrote:
>> From: Lance Yang <lance.yang@linux.dev>
>>
>> When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
>> it does not make sense to try to detect whether it is underused, because
>> try_to_map_unused_to_zeropage(), called while splitting the folio, 
>> will not
>> actually replace any zeroed pages by the shared zeropage.
>>
>> Splitting the folio in that case does not make any sense, so let's not 
>> even
>> scan to check if the folio is underused.
>>
>> Suggested-by: David Hildenbrand <david@redhat.com>
> 
> Feel free to drop that, I only tweaked you description :)

Well, I'd actually prefer to keep it. Thanks for setting me straight on
try_to_map_unused_to_zeropage() ;p


> 
> Acked-by: David Hildenbrand <david@redhat.com>

Cheers,
Lance

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[Kiryl Shutsemau]

On Mon, Sep 08, 2025 at 05:07:41PM +0800, Lance Yang wrote:
> From: Lance Yang <lance.yang@linux.dev>
> 
> When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
> it does not make sense to try to detect whether it is underused, because
> try_to_map_unused_to_zeropage(), called while splitting the folio, will not
> actually replace any zeroed pages by the shared zeropage.

It makes me think, does KSM follows the same logic as
try_to_map_unused_to_zeropage()?

I cannot immediately find what prevents KSM from replacing zeroed mlocked
folio with ZERO_PAGE().

Hm?

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[David Hildenbrand]

On 08.09.25 12:38, Kiryl Shutsemau wrote:
> On Mon, Sep 08, 2025 at 05:07:41PM +0800, Lance Yang wrote:
>> From: Lance Yang <lance.yang@linux.dev>
>>
>> When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
>> it does not make sense to try to detect whether it is underused, because
>> try_to_map_unused_to_zeropage(), called while splitting the folio, will not
>> actually replace any zeroed pages by the shared zeropage.
> 
> It makes me think, does KSM follows the same logic as
> try_to_map_unused_to_zeropage()?
> 
> I cannot immediately find what prevents KSM from replacing zeroed mlocked
> folio with ZERO_PAGE().
> 
> Hm?

I assume if you're using mlock and at the same time enable KSM for a 
process/VMA, you're doing something wrong.

In contrast, THP is supposed to be transparent (yeah, I know ...).

-- 
Cheers

David / dhildenb

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[Kiryl Shutsemau]

On Mon, Sep 08, 2025 at 01:32:05PM +0200, David Hildenbrand wrote:
> On 08.09.25 12:38, Kiryl Shutsemau wrote:
> > On Mon, Sep 08, 2025 at 05:07:41PM +0800, Lance Yang wrote:
> > > From: Lance Yang <lance.yang@linux.dev>
> > > 
> > > When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
> > > it does not make sense to try to detect whether it is underused, because
> > > try_to_map_unused_to_zeropage(), called while splitting the folio, will not
> > > actually replace any zeroed pages by the shared zeropage.
> > 
> > It makes me think, does KSM follows the same logic as
> > try_to_map_unused_to_zeropage()?
> > 
> > I cannot immediately find what prevents KSM from replacing zeroed mlocked
> > folio with ZERO_PAGE().
> > 
> > Hm?
> 
> I assume if you're using mlock and at the same time enable KSM for a
> process/VMA, you're doing something wrong.
> 
> In contrast, THP is supposed to be transparent (yeah, I know ...).

Yeah, I guess it is user error.

Maybe we should make ksm_compatible() return false for VM_LOCKED?
KSM breaks mlock() contract.

But it can be risky if someone already relies on this broken behaviour.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[David Hildenbrand]

On 08.09.25 13:44, Kiryl Shutsemau wrote:
> On Mon, Sep 08, 2025 at 01:32:05PM +0200, David Hildenbrand wrote:
>> On 08.09.25 12:38, Kiryl Shutsemau wrote:
>>> On Mon, Sep 08, 2025 at 05:07:41PM +0800, Lance Yang wrote:
>>>> From: Lance Yang <lance.yang@linux.dev>
>>>>
>>>> When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
>>>> it does not make sense to try to detect whether it is underused, because
>>>> try_to_map_unused_to_zeropage(), called while splitting the folio, will not
>>>> actually replace any zeroed pages by the shared zeropage.
>>>
>>> It makes me think, does KSM follows the same logic as
>>> try_to_map_unused_to_zeropage()?
>>>
>>> I cannot immediately find what prevents KSM from replacing zeroed mlocked
>>> folio with ZERO_PAGE().
>>>
>>> Hm?
>>
>> I assume if you're using mlock and at the same time enable KSM for a
>> process/VMA, you're doing something wrong.
>>
>> In contrast, THP is supposed to be transparent (yeah, I know ...).
> 
> Yeah, I guess it is user error.
> 
> Maybe we should make ksm_compatible() return false for VM_LOCKED?
> KSM breaks mlock() contract.

I was thinking the same and falsely remembered that we would already be 
checking for that.

> 
> But it can be risky if someone already relies on this broken behaviour.

Could be.

Staring at QEMU, we have the following parameters:

	mem-merge=on|off

     	Enables or disables memory merge support. This feature, when 	
         supported by the host, de-duplicates identical memory pages
         among VMs instances (enabled by default).

And

	-overcommit mem-lock=on|off|on-fault

	"Run qemu with hints about host resource overcommit. The default
	 is to assume that host overcommits all resources."


Now, I would assume that anybody who sets "-overcommit mem-lock=on" either

(a) Has KSM disabled on that machine.

(b) Sets mem-merge=off

as well. But QEMU would allow for configuring it.


Interestingly, mm_populate()->populate_vma_page_range() wants to break 
COW. [*]

But if the app later calls fork(), we still allow for cow-sharing pages 
with the child. (another case of "don't do it", like KSM I guess)


[*] it doesn't do it for mappings that start out R/O. I think we might 
end up with sharedzero pages in that case, but not sure if worth fixing.

-- 
Cheers

David / dhildenb

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[Kiryl Shutsemau]

On Mon, Sep 08, 2025 at 02:04:05PM +0200, David Hildenbrand wrote:
> On 08.09.25 13:44, Kiryl Shutsemau wrote:
> > On Mon, Sep 08, 2025 at 01:32:05PM +0200, David Hildenbrand wrote:
> > > On 08.09.25 12:38, Kiryl Shutsemau wrote:
> > > > On Mon, Sep 08, 2025 at 05:07:41PM +0800, Lance Yang wrote:
> > > > > From: Lance Yang <lance.yang@linux.dev>
> > > > > 
> > > > > When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
> > > > > it does not make sense to try to detect whether it is underused, because
> > > > > try_to_map_unused_to_zeropage(), called while splitting the folio, will not
> > > > > actually replace any zeroed pages by the shared zeropage.
> > > > 
> > > > It makes me think, does KSM follows the same logic as
> > > > try_to_map_unused_to_zeropage()?
> > > > 
> > > > I cannot immediately find what prevents KSM from replacing zeroed mlocked
> > > > folio with ZERO_PAGE().
> > > > 
> > > > Hm?
> > > 
> > > I assume if you're using mlock and at the same time enable KSM for a
> > > process/VMA, you're doing something wrong.
> > > 
> > > In contrast, THP is supposed to be transparent (yeah, I know ...).
> > 
> > Yeah, I guess it is user error.
> > 
> > Maybe we should make ksm_compatible() return false for VM_LOCKED?
> > KSM breaks mlock() contract.
> 
> I was thinking the same and falsely remembered that we would already be
> checking for that.
> 
> > 
> > But it can be risky if someone already relies on this broken behaviour.
> 
> Could be.
> 
> Staring at QEMU, we have the following parameters:
> 
> 	mem-merge=on|off
> 
>     	Enables or disables memory merge support. This feature, when 	
>         supported by the host, de-duplicates identical memory pages
>         among VMs instances (enabled by default).
> 
> And
> 
> 	-overcommit mem-lock=on|off|on-fault
> 
> 	"Run qemu with hints about host resource overcommit. The default
> 	 is to assume that host overcommits all resources."
> 
> 
> Now, I would assume that anybody who sets "-overcommit mem-lock=on" either
> 
> (a) Has KSM disabled on that machine.
> 
> (b) Sets mem-merge=off
> 
> as well. But QEMU would allow for configuring it.

ksm_madvise(MADV_MERGEABLE) succeeds on !vma_ksm_compatible(), so it
wouldn't be functional breakage, but may result in unexpected increase
of memory consumption.

> Interestingly, mm_populate()->populate_vma_page_range() wants to break COW.
> [*]
> 
> But if the app later calls fork(), we still allow for cow-sharing pages with
> the child. (another case of "don't do it", like KSM I guess)

CoW has bunch of these "don't do it". :P

> [*] it doesn't do it for mappings that start out R/O. I think we might end
> up with sharedzero pages in that case, but not sure if worth fixing.
> 
> -- 
> Cheers
> 
> David / dhildenb
> 

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[Lance Yang]

On 2025/9/8 20:45, Kiryl Shutsemau wrote:
> On Mon, Sep 08, 2025 at 02:04:05PM +0200, David Hildenbrand wrote:
>> On 08.09.25 13:44, Kiryl Shutsemau wrote:
>>> On Mon, Sep 08, 2025 at 01:32:05PM +0200, David Hildenbrand wrote:
>>>> On 08.09.25 12:38, Kiryl Shutsemau wrote:
>>>>> On Mon, Sep 08, 2025 at 05:07:41PM +0800, Lance Yang wrote:
>>>>>> From: Lance Yang <lance.yang@linux.dev>
>>>>>>
>>>>>> When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
>>>>>> it does not make sense to try to detect whether it is underused, because
>>>>>> try_to_map_unused_to_zeropage(), called while splitting the folio, will not
>>>>>> actually replace any zeroed pages by the shared zeropage.
>>>>>
>>>>> It makes me think, does KSM follows the same logic as
>>>>> try_to_map_unused_to_zeropage()?
>>>>>
>>>>> I cannot immediately find what prevents KSM from replacing zeroed mlocked
>>>>> folio with ZERO_PAGE().
>>>>>
>>>>> Hm?
>>>>
>>>> I assume if you're using mlock and at the same time enable KSM for a
>>>> process/VMA, you're doing something wrong.
>>>>
>>>> In contrast, THP is supposed to be transparent (yeah, I know ...).
>>>
>>> Yeah, I guess it is user error.
>>>
>>> Maybe we should make ksm_compatible() return false for VM_LOCKED?
>>> KSM breaks mlock() contract.
>>
>> I was thinking the same and falsely remembered that we would already be
>> checking for that.
>>
>>>
>>> But it can be risky if someone already relies on this broken behaviour.
>>
>> Could be.
>>
>> Staring at QEMU, we have the following parameters:
>>
>> 	mem-merge=on|off
>>
>>      	Enables or disables memory merge support. This feature, when 	
>>          supported by the host, de-duplicates identical memory pages
>>          among VMs instances (enabled by default).
>>
>> And
>>
>> 	-overcommit mem-lock=on|off|on-fault
>>
>> 	"Run qemu with hints about host resource overcommit. The default
>> 	 is to assume that host overcommits all resources."
>>
>>
>> Now, I would assume that anybody who sets "-overcommit mem-lock=on" either
>>
>> (a) Has KSM disabled on that machine.
>>
>> (b) Sets mem-merge=off
>>
>> as well. But QEMU would allow for configuring it.


That's really interesting ;)

I guess it's likely that people are already relying on that behavior,
even if it's flawed.

> 
> ksm_madvise(MADV_MERGEABLE) succeeds on !vma_ksm_compatible(), so it
> wouldn't be functional breakage, but may result in unexpected increase
> of memory consumption.

Hmm... it's hard to argue that nothing is broken.

An application losing the memory savings from KSM might not fit in
memory at all and could be taken down by the OOM killer ;p

> 
>> Interestingly, mm_populate()->populate_vma_page_range() wants to break COW.
>> [*]
>>
>> But if the app later calls fork(), we still allow for cow-sharing pages with
>> the child. (another case of "don't do it", like KSM I guess)
> 
> CoW has bunch of these "don't do it". :P
> 
>> [*] it doesn't do it for mappings that start out R/O. I think we might end
>> up with sharedzero pages in that case, but not sure if worth fixing.
>>
>> -- 
>> Cheers
>>
>> David / dhildenb
>>
> 

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[Lance Yang]

On 2025/9/8 18:38, Kiryl Shutsemau wrote:
> On Mon, Sep 08, 2025 at 05:07:41PM +0800, Lance Yang wrote:
>> From: Lance Yang <lance.yang@linux.dev>
>>
>> When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
>> it does not make sense to try to detect whether it is underused, because
>> try_to_map_unused_to_zeropage(), called while splitting the folio, will not
>> actually replace any zeroed pages by the shared zeropage.
> 
> It makes me think, does KSM follows the same logic as
> try_to_map_unused_to_zeropage()?
> 
> I cannot immediately find what prevents KSM from replacing zeroed mlocked
> folio with ZERO_PAGE().
> 
> Hm?

Good catch!

Surprisingly, my test shows that KSM is indeed able to replace a zeroed
mlocked THP with ZERO_PAGE(). And MADV_MERGEABLE allows a VMA to be marked
as mergeable even if it's already locked(VM_LOCKED) ...

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[Usama Arif]

On 08/09/2025 10:07, Lance Yang wrote:
> From: Lance Yang <lance.yang@linux.dev>
> 
> When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
> it does not make sense to try to detect whether it is underused, because
> try_to_map_unused_to_zeropage(), called while splitting the folio, will not
> actually replace any zeroed pages by the shared zeropage.
> 
> Splitting the folio in that case does not make any sense, so let's not even
> scan to check if the folio is underused.
> 
> Suggested-by: David Hildenbrand <david@redhat.com>
> Signed-off-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Usama Arif <usamaarif642@gmail.com>

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[Lorenzo Stoakes]

On Mon, Sep 08, 2025 at 05:07:41PM +0800, Lance Yang wrote:
> From: Lance Yang <lance.yang@linux.dev>
>
> When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
> it does not make sense to try to detect whether it is underused, because
> try_to_map_unused_to_zeropage(), called while splitting the folio, will not
> actually replace any zeroed pages by the shared zeropage.
>

I see it's along this lovely code path: :)

deferred_split_scan()
(thp_underused() returns true)
-> split_folio()
-> split_folio_to_list()
-> split_huge_page_to_list_to_order()
-> __folio_split()
-> remap_page()
-> remove_migration_ptes()
-> remove_migration_pte()
-> try_to_map_unused_to_zeropage()
(oops mlock'd)

> Splitting the folio in that case does not make any sense, so let's not even
> scan to check if the folio is underused.
>
> Suggested-by: David Hildenbrand <david@redhat.com>
> Signed-off-by: Lance Yang <lance.yang@linux.dev>

Anyway this does make a lot of sense! So:

Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>

> ---
> v1 -> v2:
>  - Update subject and changelog (per David, thanks!)
>  - https://lore.kernel.org/linux-mm/20250908040713.42960-1-lance.yang@linux.dev
>
>  mm/huge_memory.c | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 77f0c3417973..6d99893c18e8 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -4183,6 +4183,13 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
>  		bool underused = false;
>
>  		if (!folio_test_partially_mapped(folio)) {
> +			/*
> +			 * See try_to_map_unused_to_zeropage(): we cannot
> +			 * optimize zero-filled pages after splitting an
> +			 * mlocked folio.
> +			 */
> +			if (folio_test_mlocked(folio))
> +				goto next;
>  			underused = thp_underused(folio);
>  			if (!underused)
>  				goto next;
> --
> 2.49.0
>

Re: [PATCH v2 1/1] mm: skip mlocked THPs that are underused early in deferred_split_scan()

[Baolin Wang]

On 2025/9/8 17:07, Lance Yang wrote:
> From: Lance Yang <lance.yang@linux.dev>
> 
> When we stumble over a fully-mapped mlocked THP in the deferred shrinker,
> it does not make sense to try to detect whether it is underused, because
> try_to_map_unused_to_zeropage(), called while splitting the folio, will not
> actually replace any zeroed pages by the shared zeropage.
> 
> Splitting the folio in that case does not make any sense, so let's not even
> scan to check if the folio is underused.
> 
> Suggested-by: David Hildenbrand <david@redhat.com>
> Signed-off-by: Lance Yang <lance.yang@linux.dev>
> ---

LGTM.
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>

> v1 -> v2:
>   - Update subject and changelog (per David, thanks!)
>   - https://lore.kernel.org/linux-mm/20250908040713.42960-1-lance.yang@linux.dev
> 
>   mm/huge_memory.c | 7 +++++++
>   1 file changed, 7 insertions(+)
> 
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 77f0c3417973..6d99893c18e8 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -4183,6 +4183,13 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
>   		bool underused = false;
>   
>   		if (!folio_test_partially_mapped(folio)) {
> +			/*
> +			 * See try_to_map_unused_to_zeropage(): we cannot
> +			 * optimize zero-filled pages after splitting an
> +			 * mlocked folio.
> +			 */
> +			if (folio_test_mlocked(folio))
> +				goto next;
>   			underused = thp_underused(folio);
>   			if (!underused)
>   				goto next;