From: Muchun Song <songmuchun@bytedance.com>
To: Andrew Morton <akpm@linux-foundation.org>,
David Hildenbrand <david@kernel.org>,
Muchun Song <muchun.song@linux.dev>,
Oscar Salvador <osalvador@suse.de>,
Michael Ellerman <mpe@ellerman.id.au>,
Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>,
"Liam R . Howlett" <Liam.Howlett@oracle.com>,
Vlastimil Babka <vbabka@kernel.org>,
Mike Rapoport <rppt@kernel.org>,
Suren Baghdasaryan <surenb@google.com>,
Michal Hocko <mhocko@suse.com>,
Nicholas Piggin <npiggin@gmail.com>,
Christophe Leroy <chleroy@kernel.org>,
aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com,
linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org,
linux-kernel@vger.kernel.org,
Muchun Song <songmuchun@bytedance.com>
Subject: [PATCH v2 4/6] mm/sparse-vmemmap: Pass @pgmap argument to arch vmemmap_populate()
Date: Wed, 15 Apr 2026 19:14:10 +0800 [thread overview]
Message-ID: <20260415111412.1003526-5-songmuchun@bytedance.com> (raw)
In-Reply-To: <20260415111412.1003526-1-songmuchun@bytedance.com>
Add the struct dev_pagemap pointer as a parameter to the architecture
specific vmemmap_populate(), vmemmap_populate_hugepages() and
vmemmap_populate_basepages() functions.
Currently, the vmemmap optimization for DAX is handled mostly in an
architecture-agnostic way via vmemmap_populate_compound_pages().
However, this approach skips crucial architecture-specific initialization
steps. For example, the x86 path must call sync_global_pgds() after
populating the vmemmap, which is currently being bypassed.
To lay the groundwork for fixing the vmemmap optimization in the arch
level, we need to pass the @pgmap pointer down to the arch specific
vmemmap_populate() location. Plumb the @pgmap argument through the APIs
of vmemmap_populate(), vmemmap_populate_hugepages() and
vmemmap_populate_basepages().
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
arch/arm64/mm/mmu.c | 6 +++---
arch/loongarch/mm/init.c | 7 ++++---
arch/powerpc/include/asm/book3s/64/radix.h | 3 ++-
arch/powerpc/mm/book3s64/radix_pgtable.c | 2 +-
arch/powerpc/mm/init_64.c | 4 ++--
arch/riscv/mm/init.c | 4 ++--
arch/s390/mm/vmem.c | 2 +-
arch/sparc/mm/init_64.c | 5 +++--
arch/x86/mm/init_64.c | 8 ++++----
include/linux/mm.h | 8 +++++---
mm/hugetlb_vmemmap.c | 4 ++--
mm/sparse-vmemmap.c | 10 ++++++----
12 files changed, 35 insertions(+), 28 deletions(-)
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index e5a42b7a0160..11227e104c48 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -1790,7 +1790,7 @@ int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
}
int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
- struct vmem_altmap *altmap)
+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
{
WARN_ON((start < VMEMMAP_START) || (end > VMEMMAP_END));
/* [start, end] should be within one section */
@@ -1798,9 +1798,9 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
if (!IS_ENABLED(CONFIG_ARM64_4K_PAGES) ||
(end - start < PAGES_PER_SECTION * sizeof(struct page)))
- return vmemmap_populate_basepages(start, end, node, altmap);
+ return vmemmap_populate_basepages(start, end, node, altmap, pgmap);
else
- return vmemmap_populate_hugepages(start, end, node, altmap);
+ return vmemmap_populate_hugepages(start, end, node, altmap, pgmap);
}
#ifdef CONFIG_MEMORY_HOTPLUG
diff --git a/arch/loongarch/mm/init.c b/arch/loongarch/mm/init.c
index c9c57f08fa2c..d61c2e09caae 100644
--- a/arch/loongarch/mm/init.c
+++ b/arch/loongarch/mm/init.c
@@ -123,12 +123,13 @@ int __meminit vmemmap_check_pmd(pmd_t *pmd, int node,
}
int __meminit vmemmap_populate(unsigned long start, unsigned long end,
- int node, struct vmem_altmap *altmap)
+ int node, struct vmem_altmap *altmap,
+ struct dev_pagemap *pgmap)
{
#if CONFIG_PGTABLE_LEVELS == 2
- return vmemmap_populate_basepages(start, end, node, NULL);
+ return vmemmap_populate_basepages(start, end, node, NULL, pgmap);
#else
- return vmemmap_populate_hugepages(start, end, node, NULL);
+ return vmemmap_populate_hugepages(start, end, node, NULL, pgmap);
#endif
}
diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h
index da954e779744..bde07c6f900f 100644
--- a/arch/powerpc/include/asm/book3s/64/radix.h
+++ b/arch/powerpc/include/asm/book3s/64/radix.h
@@ -321,7 +321,8 @@ extern int __meminit radix__vmemmap_create_mapping(unsigned long start,
unsigned long page_size,
unsigned long phys);
int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end,
- int node, struct vmem_altmap *altmap);
+ int node, struct vmem_altmap *altmap,
+ struct dev_pagemap *pgmap);
void __ref radix__vmemmap_free(unsigned long start, unsigned long end,
struct vmem_altmap *altmap);
extern void radix__vmemmap_remove_mapping(unsigned long start,
diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c
index 10aced261cff..568500343e5f 100644
--- a/arch/powerpc/mm/book3s64/radix_pgtable.c
+++ b/arch/powerpc/mm/book3s64/radix_pgtable.c
@@ -1112,7 +1112,7 @@ static inline pte_t *vmemmap_pte_alloc(pmd_t *pmdp, int node,
int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, int node,
- struct vmem_altmap *altmap)
+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
{
unsigned long addr;
unsigned long next;
diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c
index b6f3ae03ca9e..8f4aa5b32186 100644
--- a/arch/powerpc/mm/init_64.c
+++ b/arch/powerpc/mm/init_64.c
@@ -275,12 +275,12 @@ static int __meminit __vmemmap_populate(unsigned long start, unsigned long end,
}
int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
- struct vmem_altmap *altmap)
+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
{
#ifdef CONFIG_PPC_BOOK3S_64
if (radix_enabled())
- return radix__vmemmap_populate(start, end, node, altmap);
+ return radix__vmemmap_populate(start, end, node, altmap, pgmap);
#endif
return __vmemmap_populate(start, end, node, altmap);
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index b0092fb842a3..a04ae9727cbe 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -1348,7 +1348,7 @@ int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
}
int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
- struct vmem_altmap *altmap)
+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
{
WARN_ON((start < VMEMMAP_START) || (end > VMEMMAP_END));
@@ -1358,7 +1358,7 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
* memory hotplug, we are not able to update all the page tables with
* the new PMDs.
*/
- return vmemmap_populate_hugepages(start, end, node, altmap);
+ return vmemmap_populate_hugepages(start, end, node, altmap, pgmap);
}
#endif
diff --git a/arch/s390/mm/vmem.c b/arch/s390/mm/vmem.c
index eeadff45e0e1..a7bf8d3d5601 100644
--- a/arch/s390/mm/vmem.c
+++ b/arch/s390/mm/vmem.c
@@ -506,7 +506,7 @@ static void vmem_remove_range(unsigned long start, unsigned long size)
* Add a backed mem_map array to the virtual mem_map array.
*/
int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
- struct vmem_altmap *altmap)
+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
{
int ret;
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 367c269305e5..f870ca330f9e 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2591,9 +2591,10 @@ int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
}
int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend,
- int node, struct vmem_altmap *altmap)
+ int node, struct vmem_altmap *altmap,
+ struct dev_pagemap *pgmap)
{
- return vmemmap_populate_hugepages(vstart, vend, node, NULL);
+ return vmemmap_populate_hugepages(vstart, vend, node, NULL, pgmap);
}
#endif /* CONFIG_SPARSEMEM_VMEMMAP */
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 77b889b71cf3..e18cc81a30b4 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1557,7 +1557,7 @@ int __meminit vmemmap_check_pmd(pmd_t *pmd, int node,
}
int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
- struct vmem_altmap *altmap)
+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
{
int err;
@@ -1565,15 +1565,15 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
VM_BUG_ON(!PAGE_ALIGNED(end));
if (end - start < PAGES_PER_SECTION * sizeof(struct page))
- err = vmemmap_populate_basepages(start, end, node, NULL);
+ err = vmemmap_populate_basepages(start, end, node, NULL, pgmap);
else if (boot_cpu_has(X86_FEATURE_PSE))
- err = vmemmap_populate_hugepages(start, end, node, altmap);
+ err = vmemmap_populate_hugepages(start, end, node, altmap, pgmap);
else if (altmap) {
pr_err_once("%s: no cpu support for altmap allocations\n",
__func__);
err = -ENOMEM;
} else
- err = vmemmap_populate_basepages(start, end, node, NULL);
+ err = vmemmap_populate_basepages(start, end, node, NULL, pgmap);
if (!err)
sync_global_pgds(start, end - 1);
return err;
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0b776907152e..bebc5f892f81 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -4877,11 +4877,13 @@ void vmemmap_set_pmd(pmd_t *pmd, void *p, int node,
int vmemmap_check_pmd(pmd_t *pmd, int node,
unsigned long addr, unsigned long next);
int vmemmap_populate_basepages(unsigned long start, unsigned long end,
- int node, struct vmem_altmap *altmap);
+ int node, struct vmem_altmap *altmap,
+ struct dev_pagemap *pgmap);
int vmemmap_populate_hugepages(unsigned long start, unsigned long end,
- int node, struct vmem_altmap *altmap);
+ int node, struct vmem_altmap *altmap,
+ struct dev_pagemap *pgmap);
int vmemmap_populate(unsigned long start, unsigned long end, int node,
- struct vmem_altmap *altmap);
+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap);
int vmemmap_populate_hvo(unsigned long start, unsigned long end,
unsigned int order, struct zone *zone,
unsigned long headsize);
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 4a077d231d3a..50b7123f3bdd 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -829,7 +829,7 @@ void __init hugetlb_vmemmap_init_late(int nid)
*/
list_del(&m->list);
- vmemmap_populate(start, end, nid, NULL);
+ vmemmap_populate(start, end, nid, NULL, NULL);
nr_mmap = end - start;
memmap_boot_pages_add(DIV_ROUND_UP(nr_mmap, PAGE_SIZE));
@@ -845,7 +845,7 @@ void __init hugetlb_vmemmap_init_late(int nid)
if (vmemmap_populate_hvo(start, end, huge_page_order(h), zone,
HUGETLB_VMEMMAP_RESERVE_SIZE) < 0) {
/* Fallback if HVO population fails */
- vmemmap_populate(start, end, nid, NULL);
+ vmemmap_populate(start, end, nid, NULL, NULL);
nr_mmap = end - start;
} else {
m->flags |= HUGE_BOOTMEM_ZONES_VALID;
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index 05e3e2b94e32..f5245647afee 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -297,7 +297,8 @@ static int __meminit vmemmap_populate_range(unsigned long start,
}
int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end,
- int node, struct vmem_altmap *altmap)
+ int node, struct vmem_altmap *altmap,
+ struct dev_pagemap *pgmap)
{
return vmemmap_populate_range(start, end, node, altmap, -1, 0);
}
@@ -400,7 +401,8 @@ int __weak __meminit vmemmap_check_pmd(pmd_t *pmd, int node,
}
int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end,
- int node, struct vmem_altmap *altmap)
+ int node, struct vmem_altmap *altmap,
+ struct dev_pagemap *pgmap)
{
unsigned long addr;
unsigned long next;
@@ -445,7 +447,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end,
}
} else if (vmemmap_check_pmd(pmd, node, addr, next))
continue;
- if (vmemmap_populate_basepages(addr, next, node, altmap))
+ if (vmemmap_populate_basepages(addr, next, node, altmap, pgmap))
return -ENOMEM;
}
return 0;
@@ -559,7 +561,7 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn,
if (vmemmap_can_optimize(altmap, pgmap))
r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap);
else
- r = vmemmap_populate(start, end, nid, altmap);
+ r = vmemmap_populate(start, end, nid, altmap, pgmap);
if (r < 0)
return NULL;
--
2.20.1
next prev parent reply other threads:[~2026-04-15 11:14 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-15 11:14 [PATCH v2 0/6] mm: Fix vmemmap optimization accounting and initialization Muchun Song
2026-04-15 11:14 ` [PATCH v2 1/6] mm/sparse-vmemmap: Fix vmemmap accounting underflow Muchun Song
2026-04-15 11:26 ` Muchun Song
2026-04-15 15:53 ` Mike Rapoport
2026-04-15 11:14 ` [PATCH v2 2/6] mm/sparse-vmemmap: Pass @pgmap argument to memory deactivation paths Muchun Song
2026-04-15 15:55 ` Mike Rapoport
2026-04-15 11:14 ` [PATCH v2 3/6] mm/sparse-vmemmap: Fix DAX vmemmap accounting with optimization Muchun Song
2026-04-15 15:58 ` Mike Rapoport
2026-04-15 11:14 ` Muchun Song [this message]
2026-04-15 12:13 ` [PATCH v2 4/6] mm/sparse-vmemmap: Pass @pgmap argument to arch vmemmap_populate() Joao Martins
2026-04-15 12:21 ` Muchun Song
2026-04-15 11:14 ` [PATCH v2 5/6] mm/sparse-vmemmap: Fix missing architecture-specific page table sync Muchun Song
2026-04-15 11:14 ` [PATCH v2 6/6] mm/mm_init: Fix pageblock migratetype for ZONE_DEVICE compound pages Muchun Song
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260415111412.1003526-5-songmuchun@bytedance.com \
--to=songmuchun@bytedance.com \
--cc=Liam.Howlett@oracle.com \
--cc=akpm@linux-foundation.org \
--cc=aneesh.kumar@linux.ibm.com \
--cc=chleroy@kernel.org \
--cc=david@kernel.org \
--cc=joao.m.martins@oracle.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=ljs@kernel.org \
--cc=maddy@linux.ibm.com \
--cc=mhocko@suse.com \
--cc=mpe@ellerman.id.au \
--cc=muchun.song@linux.dev \
--cc=npiggin@gmail.com \
--cc=osalvador@suse.de \
--cc=rppt@kernel.org \
--cc=surenb@google.com \
--cc=vbabka@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox