* [PATCH mm-new v8 0/4] Improve khugepaged scan logic
@ 2026-02-21 9:39 Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 1/4] mm: khugepaged: add trace_mm_khugepaged_scan event Vernon Yang
` (3 more replies)
0 siblings, 4 replies; 7+ messages in thread
From: Vernon Yang @ 2026-02-21 9:39 UTC (permalink / raw)
To: akpm, david
Cc: lorenzo.stoakes, ziy, dev.jain, baohua, lance.yang, linux-mm,
linux-kernel, Vernon Yang
From: Vernon Yang <yanglincheng@kylinos.cn>
Hi all,
This series is improve the khugepaged scan logic, reduce CPU consumption,
prioritize scanning task that access memory frequently.
The following data is traced by bpftrace[1] on a desktop system. After
the system has been left idle for 10 minutes upon booting, a lot of
SCAN_PMD_MAPPED or SCAN_NO_PTE_TABLE are observed during a full scan by
khugepaged.
@scan_pmd_status[1]: 1 ## SCAN_SUCCEED
@scan_pmd_status[6]: 2 ## SCAN_EXCEED_SHARED_PTE
@scan_pmd_status[3]: 142 ## SCAN_PMD_MAPPED
@scan_pmd_status[2]: 178 ## SCAN_NO_PTE_TABLE
total progress size: 674 MB
Total time : 419 seconds ## include khugepaged_scan_sleep_millisecs
The khugepaged has below phenomenon: the khugepaged list is scanned in a
FIFO manner, as long as the task is not destroyed,
1. the task no longer has memory that can be collapsed into hugepage,
continues scan it always.
2. the task at the front of the khugepaged scan list is cold, they are
still scanned first.
3. everyone scan at intervals of khugepaged_scan_sleep_millisecs
(default 10s). If we always scan the above two cases first, the valid
scan will have to wait for a long time.
For the first case, when the memory is either SCAN_PMD_MAPPED or
SCAN_NO_PTE_TABLE or SCAN_PTE_MAPPED_HUGEPAGE [5], just skip it.
For the second case, if the user has explicitly informed us via
MADV_FREE that these folios will be freed, just skip it only.
The below is some performance test results.
kernbench results (testing on x86_64 machine):
baseline w/o patches test w/ patches
Amean user-32 18522.51 ( 0.00%) 18333.64 * 1.02%*
Amean syst-32 1137.96 ( 0.00%) 1113.79 * 2.12%*
Amean elsp-32 666.04 ( 0.00%) 659.44 * 0.99%*
BAmean-95 user-32 18520.01 ( 0.00%) 18323.57 ( 1.06%)
BAmean-95 syst-32 1137.68 ( 0.00%) 1110.50 ( 2.39%)
BAmean-95 elsp-32 665.92 ( 0.00%) 659.06 ( 1.03%)
BAmean-99 user-32 18520.01 ( 0.00%) 18323.57 ( 1.06%)
BAmean-99 syst-32 1137.68 ( 0.00%) 1110.50 ( 2.39%)
BAmean-99 elsp-32 665.92 ( 0.00%) 659.06 ( 1.03%)
Create three task[2]: hot1 -> cold -> hot2. After all three task are
created, each allocate memory 128MB. the hot1/hot2 task continuously
access 128 MB memory, while the cold task only accesses its memory
briefly andthen call madvise(MADV_FREE). Here are the performance test
results:
(Throughput bigger is better, other smaller is better)
Testing on x86_64 machine:
| task hot2 | without patch | with patch | delta |
|---------------------|---------------|---------------|---------|
| total accesses time | 3.14 sec | 2.93 sec | -6.69% |
| cycles per access | 4.96 | 2.21 | -55.44% |
| Throughput | 104.38 M/sec | 111.89 M/sec | +7.19% |
| dTLB-load-misses | 284814532 | 69597236 | -75.56% |
Testing on qemu-system-x86_64 -enable-kvm:
| task hot2 | without patch | with patch | delta |
|---------------------|---------------|---------------|---------|
| total accesses time | 3.35 sec | 2.96 sec | -11.64% |
| cycles per access | 7.29 | 2.07 | -71.60% |
| Throughput | 97.67 M/sec | 110.77 M/sec | +13.41% |
| dTLB-load-misses | 241600871 | 3216108 | -98.67% |
This series is based on mm-new.
Thank you very much for your comments and discussions.
V7 -> V8:
- Just not skip collapse for lazyfree folios in VM_DROPPABLE mappings.
- "cur_progress" equal to 1 when SCAN_PTE_MAPPED_HUGEPAGE in file case.
- V7 PATCH #5 has been merged into the mm-new branch.
- Some cleaning, more detail commit log, and pickup Reviewed-by.
V6 -> V7:
- Use "*cur_progress += 1" at the beginning of the loop in anon case.
- Always "cur_progress" equal to HPAGE_PMD_NR in file case.
- Some cleaning, and pickup Acked-by and Reviewed-by.
V5 -> V6:
- Simplify hpage_collapse_scan_file() [3] and hpage_collapse_scan_pmd().
- Skip lazy-free folios in the khugepaged only [4].
- pickup Reviewed-by.
V4 -> V5:
- Patch #3 are squashed to Patch #2
- File patch utilize "xas->xa_index" to fix issue.
- folio_is_lazyfree() to folio_test_lazyfree()
- Just skip lazyfree folio simply.
- Again test kernbench in the performance mode to improve stability.
- pickup Acked-by and Reviewed-by.
V3 -> V4:
- Rebase on mm-new.
- Make Patch #2 cleaner
- Fix the lazyfree folio continue to be collapsed when skipped ahead.
V2 -> V3:
- Refine scan progress number, add folio_is_lazyfree helper
- Fix warnings at SCAN_PTE_MAPPED_HUGEPAGE.
- For MADV_FREE, we will skip the lazy-free folios instead.
- For MADV_COLD, remove it.
- Used hpage_collapse_test_exit_or_disable() instead of vma = NULL.
- pickup Reviewed-by.
V1 -> V2:
- Rename full to full_scan_finished, pickup Acked-by.
- Just skip SCAN_PMD_MAPPED/NO_PTE_TABLE memory, not remove mm.
- Set VM_NOHUGEPAGE flag when MADV_COLD/MADV_FREE to just skip, not move mm.
- Again test performance at the v6.19-rc2.
V7 : https://lore.kernel.org/linux-mm/20260207081613.588598-1-vernon2gm@gmail.com
V6 : https://lore.kernel.org/linux-mm/20260201122554.1470071-1-vernon2gm@gmail.com
V5 : https://lore.kernel.org/linux-mm/20260123082232.16413-1-vernon2gm@gmail.com
V4 : https://lore.kernel.org/linux-mm/20260111121909.8410-1-yanglincheng@kylinos.cn
V3 : https://lore.kernel.org/linux-mm/20260104054112.4541-1-yanglincheng@kylinos.cn
V2 : https://lore.kernel.org/linux-mm/20251229055151.54887-1-yanglincheng@kylinos.cn
V1 : https://lore.kernel.org/linux-mm/20251215090419.174418-1-yanglincheng@kylinos.cn
[1] https://github.com/vernon2gh/app_and_module/blob/main/khugepaged/khugepaged_mm.bt
[2] https://github.com/vernon2gh/app_and_module/blob/main/khugepaged/app.c
[3] https://lore.kernel.org/linux-mm/4c35391e-a944-4e62-9103-4a1c4961f62a@arm.com
[4] https://lore.kernel.org/linux-mm/CACZaFFNY8+UKLzBGnmB3ij9amzBdKJgytcSNtA8fLCake8Ua=A@mail.gmail.com
[5] https://lore.kernel.org/linux-mm/4qdu7owpmxfh3ugsue775fxarw5g2gcggbxdf5psj75nnu7z2u@cv2uu2yocaxq
Vernon Yang (4):
mm: khugepaged: add trace_mm_khugepaged_scan event
mm: khugepaged: refine scan progress number
mm: add folio_test_lazyfree helper
mm: khugepaged: skip lazy-free folios
include/linux/page-flags.h | 5 +++
include/trace/events/huge_memory.h | 26 ++++++++++++++
mm/khugepaged.c | 57 ++++++++++++++++++++++++------
mm/rmap.c | 2 +-
mm/vmscan.c | 5 ++-
5 files changed, 81 insertions(+), 14 deletions(-)
base-commit: a6fdc327de4678e54b5122441c970371014117b0
--
2.51.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH mm-new v8 1/4] mm: khugepaged: add trace_mm_khugepaged_scan event
2026-02-21 9:39 [PATCH mm-new v8 0/4] Improve khugepaged scan logic Vernon Yang
@ 2026-02-21 9:39 ` Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 2/4] mm: khugepaged: refine scan progress number Vernon Yang
` (2 subsequent siblings)
3 siblings, 0 replies; 7+ messages in thread
From: Vernon Yang @ 2026-02-21 9:39 UTC (permalink / raw)
To: akpm, david
Cc: lorenzo.stoakes, ziy, dev.jain, baohua, lance.yang, linux-mm,
linux-kernel, Vernon Yang
From: Vernon Yang <yanglincheng@kylinos.cn>
Add mm_khugepaged_scan event to track the total time for full scan
and the total number of pages scanned of khugepaged.
Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
Acked-by: David Hildenbrand (Red Hat) <david@kernel.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Dev Jain <dev.jain@arm.com>
---
include/trace/events/huge_memory.h | 25 +++++++++++++++++++++++++
mm/khugepaged.c | 2 ++
2 files changed, 27 insertions(+)
diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h
index 4e41bff31888..384e29f6bef0 100644
--- a/include/trace/events/huge_memory.h
+++ b/include/trace/events/huge_memory.h
@@ -237,5 +237,30 @@ TRACE_EVENT(mm_khugepaged_collapse_file,
__print_symbolic(__entry->result, SCAN_STATUS))
);
+TRACE_EVENT(mm_khugepaged_scan,
+
+ TP_PROTO(struct mm_struct *mm, unsigned int progress,
+ bool full_scan_finished),
+
+ TP_ARGS(mm, progress, full_scan_finished),
+
+ TP_STRUCT__entry(
+ __field(struct mm_struct *, mm)
+ __field(unsigned int, progress)
+ __field(bool, full_scan_finished)
+ ),
+
+ TP_fast_assign(
+ __entry->mm = mm;
+ __entry->progress = progress;
+ __entry->full_scan_finished = full_scan_finished;
+ ),
+
+ TP_printk("mm=%p, progress=%u, full_scan_finished=%d",
+ __entry->mm,
+ __entry->progress,
+ __entry->full_scan_finished)
+);
+
#endif /* __HUGE_MEMORY_H */
#include <trace/define_trace.h>
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index c0f893bebcff..e2f6b68a0011 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -2527,6 +2527,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, enum scan_result
collect_mm_slot(slot);
}
+ trace_mm_khugepaged_scan(mm, progress, khugepaged_scan.mm_slot == NULL);
+
return progress;
}
--
2.51.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH mm-new v8 2/4] mm: khugepaged: refine scan progress number
2026-02-21 9:39 [PATCH mm-new v8 0/4] Improve khugepaged scan logic Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 1/4] mm: khugepaged: add trace_mm_khugepaged_scan event Vernon Yang
@ 2026-02-21 9:39 ` Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 3/4] mm: add folio_test_lazyfree helper Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 4/4] mm: khugepaged: skip lazy-free folios Vernon Yang
3 siblings, 0 replies; 7+ messages in thread
From: Vernon Yang @ 2026-02-21 9:39 UTC (permalink / raw)
To: akpm, david
Cc: lorenzo.stoakes, ziy, dev.jain, baohua, lance.yang, linux-mm,
linux-kernel, Vernon Yang
From: Vernon Yang <yanglincheng@kylinos.cn>
Currently, each scan always increases "progress" by HPAGE_PMD_NR,
even if only scanning a single PTE/PMD entry.
- When only scanning a sigle PTE entry, let me provide a detailed
example:
static int hpage_collapse_scan_pmd()
{
for (addr = start_addr, _pte = pte; _pte < pte + HPAGE_PMD_NR;
_pte++, addr += PAGE_SIZE) {
pte_t pteval = ptep_get(_pte);
...
if (pte_uffd_wp(pteval)) { <-- first scan hit
result = SCAN_PTE_UFFD_WP;
goto out_unmap;
}
}
}
During the first scan, if pte_uffd_wp(pteval) is true, the loop exits
directly. In practice, only one PTE is scanned before termination.
Here, "progress += 1" reflects the actual number of PTEs scanned, but
previously "progress += HPAGE_PMD_NR" always.
- When the memory has been collapsed to PMD, let me provide a detailed
example:
The following data is traced by bpftrace on a desktop system. After
the system has been left idle for 10 minutes upon booting, a lot of
SCAN_PMD_MAPPED or SCAN_NO_PTE_TABLE are observed during a full scan
by khugepaged.
From trace_mm_khugepaged_scan_pmd and trace_mm_khugepaged_scan_file, the
following statuses were observed, with frequency mentioned next to them:
SCAN_SUCCEED : 1
SCAN_EXCEED_SHARED_PTE: 2
SCAN_PMD_MAPPED : 142
SCAN_NO_PTE_TABLE : 178
total progress size : 674 MB
Total time : 419 seconds, include khugepaged_scan_sleep_millisecs
The khugepaged_scan list save all task that support collapse into hugepage,
as long as the task is not destroyed, khugepaged will not remove it from
the khugepaged_scan list. This exist a phenomenon where task has already
collapsed all memory regions into hugepage, but khugepaged continues to
scan it, which wastes CPU time and invalid, and due to
khugepaged_scan_sleep_millisecs (default 10s) causes a long wait for
scanning a large number of invalid task, so scanning really valid task
is later.
After applying this patch, when the memory is either SCAN_PMD_MAPPED or
SCAN_NO_PTE_TABLE, just skip it, as follow:
SCAN_EXCEED_SHARED_PTE: 2
SCAN_PMD_MAPPED : 147
SCAN_NO_PTE_TABLE : 173
total progress size : 45 MB
Total time : 20 seconds
SCAN_PTE_MAPPED_HUGEPAGE is the same, for detailed data, refer to
https://lore.kernel.org/linux-mm/4qdu7owpmxfh3ugsue775fxarw5g2gcggbxdf5psj75nnu7z2u@cv2uu2yocaxq
Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
Reviewed-by: Dev Jain <dev.jain@arm.com>
---
mm/khugepaged.c | 42 ++++++++++++++++++++++++++++++++----------
1 file changed, 32 insertions(+), 10 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index e2f6b68a0011..61e25cf5424b 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -68,7 +68,10 @@ enum scan_result {
static struct task_struct *khugepaged_thread __read_mostly;
static DEFINE_MUTEX(khugepaged_mutex);
-/* default scan 8*HPAGE_PMD_NR ptes (or vmas) every 10 second */
+/*
+ * default scan 8*HPAGE_PMD_NR ptes, pmd_mapped, no_pte_table or vmas
+ * every 10 second.
+ */
static unsigned int khugepaged_pages_to_scan __read_mostly;
static unsigned int khugepaged_pages_collapsed;
static unsigned int khugepaged_full_scans;
@@ -1231,7 +1234,8 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long a
}
static enum scan_result hpage_collapse_scan_pmd(struct mm_struct *mm,
- struct vm_area_struct *vma, unsigned long start_addr, bool *mmap_locked,
+ struct vm_area_struct *vma, unsigned long start_addr,
+ bool *mmap_locked, unsigned int *cur_progress,
struct collapse_control *cc)
{
pmd_t *pmd;
@@ -1247,19 +1251,27 @@ static enum scan_result hpage_collapse_scan_pmd(struct mm_struct *mm,
VM_BUG_ON(start_addr & ~HPAGE_PMD_MASK);
result = find_pmd_or_thp_or_none(mm, start_addr, &pmd);
- if (result != SCAN_SUCCEED)
+ if (result != SCAN_SUCCEED) {
+ if (cur_progress)
+ *cur_progress = 1;
goto out;
+ }
memset(cc->node_load, 0, sizeof(cc->node_load));
nodes_clear(cc->alloc_nmask);
pte = pte_offset_map_lock(mm, pmd, start_addr, &ptl);
if (!pte) {
+ if (cur_progress)
+ *cur_progress = 1;
result = SCAN_NO_PTE_TABLE;
goto out;
}
for (addr = start_addr, _pte = pte; _pte < pte + HPAGE_PMD_NR;
_pte++, addr += PAGE_SIZE) {
+ if (cur_progress)
+ *cur_progress += 1;
+
pte_t pteval = ptep_get(_pte);
if (pte_none_or_zero(pteval)) {
++none_or_zero;
@@ -2279,8 +2291,9 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
return result;
}
-static enum scan_result hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,
- struct file *file, pgoff_t start, struct collapse_control *cc)
+static enum scan_result hpage_collapse_scan_file(struct mm_struct *mm,
+ unsigned long addr, struct file *file, pgoff_t start,
+ unsigned int *cur_progress, struct collapse_control *cc)
{
struct folio *folio = NULL;
struct address_space *mapping = file->f_mapping;
@@ -2370,6 +2383,12 @@ static enum scan_result hpage_collapse_scan_file(struct mm_struct *mm, unsigned
}
}
rcu_read_unlock();
+ if (cur_progress) {
+ if (result == SCAN_PTE_MAPPED_HUGEPAGE)
+ *cur_progress = 1;
+ else
+ *cur_progress = HPAGE_PMD_NR;
+ }
if (result == SCAN_SUCCEED) {
if (cc->is_khugepaged &&
@@ -2448,6 +2467,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, enum scan_result
while (khugepaged_scan.address < hend) {
bool mmap_locked = true;
+ unsigned int cur_progress = 0;
cond_resched();
if (unlikely(hpage_collapse_test_exit_or_disable(mm)))
@@ -2464,7 +2484,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, enum scan_result
mmap_read_unlock(mm);
mmap_locked = false;
*result = hpage_collapse_scan_file(mm,
- khugepaged_scan.address, file, pgoff, cc);
+ khugepaged_scan.address, file, pgoff,
+ &cur_progress, cc);
fput(file);
if (*result == SCAN_PTE_MAPPED_HUGEPAGE) {
mmap_read_lock(mm);
@@ -2478,7 +2499,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, enum scan_result
}
} else {
*result = hpage_collapse_scan_pmd(mm, vma,
- khugepaged_scan.address, &mmap_locked, cc);
+ khugepaged_scan.address, &mmap_locked,
+ &cur_progress, cc);
}
if (*result == SCAN_SUCCEED)
@@ -2486,7 +2508,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, enum scan_result
/* move to next address */
khugepaged_scan.address += HPAGE_PMD_SIZE;
- progress += HPAGE_PMD_NR;
+ progress += cur_progress;
if (!mmap_locked)
/*
* We released mmap_lock so break loop. Note
@@ -2809,7 +2831,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
mmap_locked = false;
*lock_dropped = true;
result = hpage_collapse_scan_file(mm, addr, file, pgoff,
- cc);
+ NULL, cc);
if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
mapping_can_writeback(file->f_mapping)) {
@@ -2824,7 +2846,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
fput(file);
} else {
result = hpage_collapse_scan_pmd(mm, vma, addr,
- &mmap_locked, cc);
+ &mmap_locked, NULL, cc);
}
if (!mmap_locked)
*lock_dropped = true;
--
2.51.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH mm-new v8 3/4] mm: add folio_test_lazyfree helper
2026-02-21 9:39 [PATCH mm-new v8 0/4] Improve khugepaged scan logic Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 1/4] mm: khugepaged: add trace_mm_khugepaged_scan event Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 2/4] mm: khugepaged: refine scan progress number Vernon Yang
@ 2026-02-21 9:39 ` Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 4/4] mm: khugepaged: skip lazy-free folios Vernon Yang
3 siblings, 0 replies; 7+ messages in thread
From: Vernon Yang @ 2026-02-21 9:39 UTC (permalink / raw)
To: akpm, david
Cc: lorenzo.stoakes, ziy, dev.jain, baohua, lance.yang, linux-mm,
linux-kernel, Vernon Yang
From: Vernon Yang <yanglincheng@kylinos.cn>
Add folio_test_lazyfree() function to identify lazy-free folios to improve
code readability.
Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
Acked-by: David Hildenbrand (Red Hat) <david@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Dev Jain <dev.jain@arm.com>
Reviewed-by: Barry Song <baohua@kernel.org>
---
include/linux/page-flags.h | 5 +++++
mm/rmap.c | 2 +-
mm/vmscan.c | 5 ++---
3 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index fb6a83fe88b0..0426cac91c0b 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -724,6 +724,11 @@ static __always_inline bool folio_test_anon(const struct folio *folio)
return ((unsigned long)folio->mapping & FOLIO_MAPPING_ANON) != 0;
}
+static __always_inline bool folio_test_lazyfree(const struct folio *folio)
+{
+ return folio_test_anon(folio) && !folio_test_swapbacked(folio);
+}
+
static __always_inline bool PageAnonNotKsm(const struct page *page)
{
unsigned long flags = (unsigned long)page_folio(page)->mapping;
diff --git a/mm/rmap.c b/mm/rmap.c
index 0f00570d1b9e..bff8f222004e 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -2046,7 +2046,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
}
if (!pvmw.pte) {
- if (folio_test_anon(folio) && !folio_test_swapbacked(folio)) {
+ if (folio_test_lazyfree(folio)) {
if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, folio))
goto walk_done;
/*
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 6a87ac7be43c..9ce3f54f43b8 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -963,8 +963,7 @@ static void folio_check_dirty_writeback(struct folio *folio,
* They could be mistakenly treated as file lru. So further anon
* test is needed.
*/
- if (!folio_is_file_lru(folio) ||
- (folio_test_anon(folio) && !folio_test_swapbacked(folio))) {
+ if (!folio_is_file_lru(folio) || folio_test_lazyfree(folio)) {
*dirty = false;
*writeback = false;
return;
@@ -1508,7 +1507,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
}
}
- if (folio_test_anon(folio) && !folio_test_swapbacked(folio)) {
+ if (folio_test_lazyfree(folio)) {
/* follow __remove_mapping for reference */
if (!folio_ref_freeze(folio, 1))
goto keep_locked;
--
2.51.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH mm-new v8 4/4] mm: khugepaged: skip lazy-free folios
2026-02-21 9:39 [PATCH mm-new v8 0/4] Improve khugepaged scan logic Vernon Yang
` (2 preceding siblings ...)
2026-02-21 9:39 ` [PATCH mm-new v8 3/4] mm: add folio_test_lazyfree helper Vernon Yang
@ 2026-02-21 9:39 ` Vernon Yang
2026-02-21 10:27 ` Barry Song
3 siblings, 1 reply; 7+ messages in thread
From: Vernon Yang @ 2026-02-21 9:39 UTC (permalink / raw)
To: akpm, david
Cc: lorenzo.stoakes, ziy, dev.jain, baohua, lance.yang, linux-mm,
linux-kernel, Vernon Yang
From: Vernon Yang <yanglincheng@kylinos.cn>
For example, create three task: hot1 -> cold -> hot2. After all three
task are created, each allocate memory 128MB. the hot1/hot2 task
continuously access 128 MB memory, while the cold task only accesses
its memory briefly and then call madvise(MADV_FREE). However, khugepaged
still prioritizes scanning the cold task and only scans the hot2 task
after completing the scan of the cold task.
And if all folios in VM_DROPPABLE are lazyfree, Collapsing maintains
that property, so we can just collapse and memory pressure in the future
will free it up. In contrast, collapsing in !VM_DROPPABLE does not
maintain that property, the collapsed folio will not be lazyfree and
memory pressure in the future will not be able to free it up.
So if the user has explicitly informed us via MADV_FREE that this memory
will be freed, and this vma does not have VM_DROPPABLE flags, it is
appropriate for khugepaged to skip it only, thereby avoiding unnecessary
scan and collapse operations to reducing CPU wastage.
Here are the performance test results:
(Throughput bigger is better, other smaller is better)
Testing on x86_64 machine:
| task hot2 | without patch | with patch | delta |
|---------------------|---------------|---------------|---------|
| total accesses time | 3.14 sec | 2.93 sec | -6.69% |
| cycles per access | 4.96 | 2.21 | -55.44% |
| Throughput | 104.38 M/sec | 111.89 M/sec | +7.19% |
| dTLB-load-misses | 284814532 | 69597236 | -75.56% |
Testing on qemu-system-x86_64 -enable-kvm:
| task hot2 | without patch | with patch | delta |
|---------------------|---------------|---------------|---------|
| total accesses time | 3.35 sec | 2.96 sec | -11.64% |
| cycles per access | 7.29 | 2.07 | -71.60% |
| Throughput | 97.67 M/sec | 110.77 M/sec | +13.41% |
| dTLB-load-misses | 241600871 | 3216108 | -98.67% |
Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
Acked-by: David Hildenbrand (arm) <david@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
---
include/trace/events/huge_memory.h | 1 +
mm/khugepaged.c | 13 +++++++++++++
2 files changed, 14 insertions(+)
diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h
index 384e29f6bef0..bcdc57eea270 100644
--- a/include/trace/events/huge_memory.h
+++ b/include/trace/events/huge_memory.h
@@ -25,6 +25,7 @@
EM( SCAN_PAGE_LRU, "page_not_in_lru") \
EM( SCAN_PAGE_LOCK, "page_locked") \
EM( SCAN_PAGE_ANON, "page_not_anon") \
+ EM( SCAN_PAGE_LAZYFREE, "page_lazyfree") \
EM( SCAN_PAGE_COMPOUND, "page_compound") \
EM( SCAN_ANY_PROCESS, "no_process_for_page") \
EM( SCAN_VMA_NULL, "vma_null") \
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 61e25cf5424b..e792e9074b48 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -46,6 +46,7 @@ enum scan_result {
SCAN_PAGE_LRU,
SCAN_PAGE_LOCK,
SCAN_PAGE_ANON,
+ SCAN_PAGE_LAZYFREE,
SCAN_PAGE_COMPOUND,
SCAN_ANY_PROCESS,
SCAN_VMA_NULL,
@@ -574,6 +575,12 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
folio = page_folio(page);
VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
+ if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
+ folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
+ result = SCAN_PAGE_LAZYFREE;
+ goto out;
+ }
+
/* See hpage_collapse_scan_pmd(). */
if (folio_maybe_mapped_shared(folio)) {
++shared;
@@ -1326,6 +1333,12 @@ static enum scan_result hpage_collapse_scan_pmd(struct mm_struct *mm,
}
folio = page_folio(page);
+ if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
+ folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
+ result = SCAN_PAGE_LAZYFREE;
+ goto out_unmap;
+ }
+
if (!folio_test_anon(folio)) {
result = SCAN_PAGE_ANON;
goto out_unmap;
--
2.51.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH mm-new v8 4/4] mm: khugepaged: skip lazy-free folios
2026-02-21 9:39 ` [PATCH mm-new v8 4/4] mm: khugepaged: skip lazy-free folios Vernon Yang
@ 2026-02-21 10:27 ` Barry Song
2026-02-21 13:38 ` Vernon Yang
0 siblings, 1 reply; 7+ messages in thread
From: Barry Song @ 2026-02-21 10:27 UTC (permalink / raw)
To: Vernon Yang
Cc: akpm, david, lorenzo.stoakes, ziy, dev.jain, lance.yang,
linux-mm, linux-kernel, Vernon Yang
On Sat, Feb 21, 2026 at 5:40 PM Vernon Yang <vernon2gm@gmail.com> wrote:
>
> From: Vernon Yang <yanglincheng@kylinos.cn>
>
> For example, create three task: hot1 -> cold -> hot2. After all three
> task are created, each allocate memory 128MB. the hot1/hot2 task
> continuously access 128 MB memory, while the cold task only accesses
> its memory briefly and then call madvise(MADV_FREE). However, khugepaged
> still prioritizes scanning the cold task and only scans the hot2 task
> after completing the scan of the cold task.
>
> And if all folios in VM_DROPPABLE are lazyfree, Collapsing maintains
> that property, so we can just collapse and memory pressure in the future
I don’t think this is accurate. A VMA without VM_DROPPABLE
can still have all folios marked as lazyfree. Therefore, having
all folios lazyfree is not the reason why collapsing preserves
the property.
This raises a question: if a VMA without VM_DROPPABLE has
many contiguous lazyfree folios that can be collapsed, and
none of those folios are non-lazyfree, should we collapse
them and pass the lazyfree state to the new folio?
Currently, our approach skips the collapse, which also feels
a bit inconsistent.
> will free it up. In contrast, collapsing in !VM_DROPPABLE does not
> maintain that property, the collapsed folio will not be lazyfree and
> memory pressure in the future will not be able to free it up.
>
> So if the user has explicitly informed us via MADV_FREE that this memory
> will be freed, and this vma does not have VM_DROPPABLE flags, it is
> appropriate for khugepaged to skip it only, thereby avoiding unnecessary
> scan and collapse operations to reducing CPU wastage.
>
> Here are the performance test results:
> (Throughput bigger is better, other smaller is better)
>
> Testing on x86_64 machine:
>
> | task hot2 | without patch | with patch | delta |
> |---------------------|---------------|---------------|---------|
> | total accesses time | 3.14 sec | 2.93 sec | -6.69% |
> | cycles per access | 4.96 | 2.21 | -55.44% |
> | Throughput | 104.38 M/sec | 111.89 M/sec | +7.19% |
> | dTLB-load-misses | 284814532 | 69597236 | -75.56% |
>
> Testing on qemu-system-x86_64 -enable-kvm:
>
> | task hot2 | without patch | with patch | delta |
> |---------------------|---------------|---------------|---------|
> | total accesses time | 3.35 sec | 2.96 sec | -11.64% |
> | cycles per access | 7.29 | 2.07 | -71.60% |
> | Throughput | 97.67 M/sec | 110.77 M/sec | +13.41% |
> | dTLB-load-misses | 241600871 | 3216108 | -98.67% |
>
> Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
> Acked-by: David Hildenbrand (arm) <david@kernel.org>
> Reviewed-by: Lance Yang <lance.yang@linux.dev>
> ---
Overall, LGTM,
Reviewed-by: Barry Song <baohua@kernel.org>
> include/trace/events/huge_memory.h | 1 +
> mm/khugepaged.c | 13 +++++++++++++
> 2 files changed, 14 insertions(+)
>
> diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h
> index 384e29f6bef0..bcdc57eea270 100644
> --- a/include/trace/events/huge_memory.h
> +++ b/include/trace/events/huge_memory.h
> @@ -25,6 +25,7 @@
> EM( SCAN_PAGE_LRU, "page_not_in_lru") \
> EM( SCAN_PAGE_LOCK, "page_locked") \
> EM( SCAN_PAGE_ANON, "page_not_anon") \
> + EM( SCAN_PAGE_LAZYFREE, "page_lazyfree") \
> EM( SCAN_PAGE_COMPOUND, "page_compound") \
> EM( SCAN_ANY_PROCESS, "no_process_for_page") \
> EM( SCAN_VMA_NULL, "vma_null") \
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 61e25cf5424b..e792e9074b48 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -46,6 +46,7 @@ enum scan_result {
> SCAN_PAGE_LRU,
> SCAN_PAGE_LOCK,
> SCAN_PAGE_ANON,
> + SCAN_PAGE_LAZYFREE,
> SCAN_PAGE_COMPOUND,
> SCAN_ANY_PROCESS,
> SCAN_VMA_NULL,
> @@ -574,6 +575,12 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
> folio = page_folio(page);
> VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
>
> + if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
> + folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
I would prefer to add a comment about VM_DROPPABLE here
rather than only mentioning it in the changelog.
> + result = SCAN_PAGE_LAZYFREE;
> + goto out;
> + }
> +
> /* See hpage_collapse_scan_pmd(). */
> if (folio_maybe_mapped_shared(folio)) {
> ++shared;
> @@ -1326,6 +1333,12 @@ static enum scan_result hpage_collapse_scan_pmd(struct mm_struct *mm,
> }
> folio = page_folio(page);
>
> + if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
> + folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
> + result = SCAN_PAGE_LAZYFREE;
> + goto out_unmap;
> + }
As above.
> +
> if (!folio_test_anon(folio)) {
> result = SCAN_PAGE_ANON;
> goto out_unmap;
> --
> 2.51.0
>
Thanks
Barry
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH mm-new v8 4/4] mm: khugepaged: skip lazy-free folios
2026-02-21 10:27 ` Barry Song
@ 2026-02-21 13:38 ` Vernon Yang
0 siblings, 0 replies; 7+ messages in thread
From: Vernon Yang @ 2026-02-21 13:38 UTC (permalink / raw)
To: Barry Song
Cc: akpm, david, lorenzo.stoakes, ziy, dev.jain, lance.yang,
linux-mm, linux-kernel, Vernon Yang
On Sat, Feb 21, 2026 at 06:27:36PM +0800, Barry Song wrote:
> On Sat, Feb 21, 2026 at 5:40 PM Vernon Yang <vernon2gm@gmail.com> wrote:
> >
> > From: Vernon Yang <yanglincheng@kylinos.cn>
> >
> > For example, create three task: hot1 -> cold -> hot2. After all three
> > task are created, each allocate memory 128MB. the hot1/hot2 task
> > continuously access 128 MB memory, while the cold task only accesses
> > its memory briefly and then call madvise(MADV_FREE). However, khugepaged
> > still prioritizes scanning the cold task and only scans the hot2 task
> > after completing the scan of the cold task.
> >
> > And if all folios in VM_DROPPABLE are lazyfree, Collapsing maintains
> > that property, so we can just collapse and memory pressure in the future
>
> I don’t think this is accurate. A VMA without VM_DROPPABLE
> can still have all folios marked as lazyfree. Therefore, having
> all folios lazyfree is not the reason why collapsing preserves
> the property.
In folio_add_new_anon_rmap(), we know that the vma has the VM_DROPPABLE
attribute, which is the root reason why Collapsing maintains that property.
The above commit log clearly states "all folios in VM_DROPPABLE are lazyfree"
^^^^^^^^^^^^^^^
(the "if" is redundant and should be removed), not "all folios are lazyfree".
> This raises a question: if a VMA without VM_DROPPABLE has
> many contiguous lazyfree folios that can be collapsed, and
> none of those folios are non-lazyfree, should we collapse
> them and pass the lazyfree state to the new folio?
>
> Currently, our approach skips the collapse, which also feels
> a bit inconsistent.
Yes, they are inconsistent, because this question need to scan all folios
to make a decision, and it cannot solve the hot1->cold->hot2 scenario.
> > will free it up. In contrast, collapsing in !VM_DROPPABLE does not
> > maintain that property, the collapsed folio will not be lazyfree and
> > memory pressure in the future will not be able to free it up.
> >
> > So if the user has explicitly informed us via MADV_FREE that this memory
> > will be freed, and this vma does not have VM_DROPPABLE flags, it is
> > appropriate for khugepaged to skip it only, thereby avoiding unnecessary
> > scan and collapse operations to reducing CPU wastage.
> >
> > Here are the performance test results:
> > (Throughput bigger is better, other smaller is better)
> >
> > Testing on x86_64 machine:
> >
> > | task hot2 | without patch | with patch | delta |
> > |---------------------|---------------|---------------|---------|
> > | total accesses time | 3.14 sec | 2.93 sec | -6.69% |
> > | cycles per access | 4.96 | 2.21 | -55.44% |
> > | Throughput | 104.38 M/sec | 111.89 M/sec | +7.19% |
> > | dTLB-load-misses | 284814532 | 69597236 | -75.56% |
> >
> > Testing on qemu-system-x86_64 -enable-kvm:
> >
> > | task hot2 | without patch | with patch | delta |
> > |---------------------|---------------|---------------|---------|
> > | total accesses time | 3.35 sec | 2.96 sec | -11.64% |
> > | cycles per access | 7.29 | 2.07 | -71.60% |
> > | Throughput | 97.67 M/sec | 110.77 M/sec | +13.41% |
> > | dTLB-load-misses | 241600871 | 3216108 | -98.67% |
> >
> > Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
> > Acked-by: David Hildenbrand (arm) <david@kernel.org>
> > Reviewed-by: Lance Yang <lance.yang@linux.dev>
> > ---
>
> Overall, LGTM,
>
> Reviewed-by: Barry Song <baohua@kernel.org>
Thank you for review.
> > include/trace/events/huge_memory.h | 1 +
> > mm/khugepaged.c | 13 +++++++++++++
> > 2 files changed, 14 insertions(+)
> >
> > diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h
> > index 384e29f6bef0..bcdc57eea270 100644
> > --- a/include/trace/events/huge_memory.h
> > +++ b/include/trace/events/huge_memory.h
> > @@ -25,6 +25,7 @@
> > EM( SCAN_PAGE_LRU, "page_not_in_lru") \
> > EM( SCAN_PAGE_LOCK, "page_locked") \
> > EM( SCAN_PAGE_ANON, "page_not_anon") \
> > + EM( SCAN_PAGE_LAZYFREE, "page_lazyfree") \
> > EM( SCAN_PAGE_COMPOUND, "page_compound") \
> > EM( SCAN_ANY_PROCESS, "no_process_for_page") \
> > EM( SCAN_VMA_NULL, "vma_null") \
> > diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> > index 61e25cf5424b..e792e9074b48 100644
> > --- a/mm/khugepaged.c
> > +++ b/mm/khugepaged.c
> > @@ -46,6 +46,7 @@ enum scan_result {
> > SCAN_PAGE_LRU,
> > SCAN_PAGE_LOCK,
> > SCAN_PAGE_ANON,
> > + SCAN_PAGE_LAZYFREE,
> > SCAN_PAGE_COMPOUND,
> > SCAN_ANY_PROCESS,
> > SCAN_VMA_NULL,
> > @@ -574,6 +575,12 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
> > folio = page_folio(page);
> > VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
> >
> > + if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
> > + folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
>
> I would prefer to add a comment about VM_DROPPABLE here
> rather than only mentioning it in the changelog.
Is the following comment clear?
/*
* If the vma has the VM_DROPPABLE flag, the collapse will
* preserve the lazyfree property without needing to skip.
*/
> > + result = SCAN_PAGE_LAZYFREE;
> > + goto out;
> > + }
> > +
> > /* See hpage_collapse_scan_pmd(). */
> > if (folio_maybe_mapped_shared(folio)) {
> > ++shared;
> > @@ -1326,6 +1333,12 @@ static enum scan_result hpage_collapse_scan_pmd(struct mm_struct *mm,
> > }
> > folio = page_folio(page);
> >
> > + if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
> > + folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
> > + result = SCAN_PAGE_LAZYFREE;
> > + goto out_unmap;
> > + }
>
> As above.
>
> > +
> > if (!folio_test_anon(folio)) {
> > result = SCAN_PAGE_ANON;
> > goto out_unmap;
> > --
> > 2.51.0
> >
>
> Thanks
> Barry
>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-02-21 13:39 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-02-21 9:39 [PATCH mm-new v8 0/4] Improve khugepaged scan logic Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 1/4] mm: khugepaged: add trace_mm_khugepaged_scan event Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 2/4] mm: khugepaged: refine scan progress number Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 3/4] mm: add folio_test_lazyfree helper Vernon Yang
2026-02-21 9:39 ` [PATCH mm-new v8 4/4] mm: khugepaged: skip lazy-free folios Vernon Yang
2026-02-21 10:27 ` Barry Song
2026-02-21 13:38 ` Vernon Yang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox