Re: [RFC] mm: stress-ng --mremap triggers severe lruvec lock contention in populate/unmap paths

[Pedro Falcato <pfalcato@suse.de>]

Hi,

On Tue, Apr 07, 2026 at 04:09:20PM -0400, Joseph Salisbury wrote:
> Hello,
> 
> I would like to ask for feedback on an MM performance issue triggered by
> stress-ng's mremap stressor:
> 
> stress-ng --mremap 8192 --mremap-bytes 4K --timeout 30 --metrics-brief
> 
> This was first investigated as a possible regression from 0ca0c24e3211 ("mm:
> store zero pages to be swapped out in a bitmap"), but the current evidence
> suggests that commit is mostly exposing an older problem for this workload
> rather than directly causing it.
> 
> 
> Observed behavior:
> 
> The metrics below are in this format:
>     stressor       bogo ops real time  usr time  sys time   bogo ops/s   
>  bogo ops/s
>                              (secs)    (secs)    (secs)   (real time)
> (usr+sys time)
> 
> On a 5.15-based kernel, the workload behaves much worse when swapping is
> disabled:
> 
>     swap enabled:
>       mremap 1660980 31.08 64.78 84.63 53437.09 11116.73
> 
>     swap disabled:
>       mremap 40786258 27.94 15.41 15354.79 1459749.43 2653.59
> 
> On a 6.12-based kernel with swap enabled, the same high-system-time behavior
> is also observed:
> 
>     mremap 77087729 21.50 29.95 30558.08 3584738.22 2520.19
> 
> A recent 7.0-rc5-based mainline build still behaves similarly:
> 
>     mremap 39208813 28.12 12.34 15318.39 1394408.50 2557.53
> 
> So this does not appear to be already fixed upstream.
> 
> 
> 
> The current theory is that 0ca0c24e3211 exposes this specific
> zero-page-heavy workload.  Before that change, swap-enabled runs actually
> swapped pages.  After that change, zero pages are stored in the swap bitmap
> instead, so the workload behaves much more like the swap-disabled case.
> 
> Perf data supports the idea that the expensive behavior is global LRU lock
> contention caused by short-lived populate/unmap churn.
> 
> The dominant stacks on the bad cases include:
> 
>     vm_mmap_pgoff
>       __mm_populate
>         populate_vma_page_range
>           lru_add_drain
>             folio_batch_move_lru
>               folio_lruvec_lock_irqsave
>                 native_queued_spin_lock_slowpath
> 
> and:
> 
>     __x64_sys_munmap
>       __vm_munmap
>         ...
>           release_pages
>             folios_put_refs
>               __page_cache_release
>                 folio_lruvec_relock_irqsave
>                   native_queued_spin_lock_slowpath
> 


Yes, this is known problematic. The lruvec locks are gigantic and, despite
the LRU cache in front, they are still problematic. It might be argued that the
current cache is downright useless for populate as it's too small to contain
a significant number of folios. Perhaps worth thinking about, but not trivial
to change given the way things are structured + the way folio batches work.

You should be able to see this on any workload that does lots of page faulting
or population (not dependent on mremap at all, etc)

> 
> 
> It was also found that adding '--mremap-numa' changes the behavior
> substantially:

"assign memory mapped pages to randomly selected NUMA nodes. This is
disabled for systems that do not support NUMA."

so this is just sharding your lock contention across your NUMA nodes (you
have an lruvec per node).

> 
> stress-ng --mremap 8192 --mremap-bytes 4K --timeout 30 --mremap-numa
> --metrics-brief
> 
> mremap 2570798 29.39 8.06 106.23 87466.50 22494.74
> 
> So it's possible that either actual swapping, or the mbind(...,
> MPOL_MF_MOVE) path used by '--mremap-numa', removes most of the excessive
> system time.
> 
> Does this look like a known MM scalability issue around short-lived
> MAP_POPULATE / munmap churn?

Yes. Is this an actual issue on some workload?

-- 
Pedro