From: "Matthew Wilcox (Oracle)" <willy@infradead.org>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>,
linux-arch@vger.kernel.org, linux-mm@kvack.org,
linux-kernel@vger.kernel.org
Subject: [PATCH v6 00/38] New page table range API
Date: Wed, 2 Aug 2023 16:13:28 +0100 [thread overview]
Message-ID: <20230802151406.3735276-1-willy@infradead.org> (raw)
For some reason, I didn't get an email when v5 was dropped from linux-next
(I got an email from Andrew saying he was going to, but when I didn't get
the automated emails, I assumed he'd changed his mind). So here's v6,
far later than I would have resent it.
This patchset changes the API used by the MM to set up page table entries.
The four APIs are:
set_ptes(mm, addr, ptep, pte, nr)
update_mmu_cache_range(vma, addr, ptep, nr)
flush_dcache_folio(folio)
flush_icache_pages(vma, page, nr)
flush_dcache_folio() isn't technically new, but no architecture
implemented it, so I've done that for them. The old APIs remain around
but are mostly implemented by calling the new interfaces.
The new APIs are based around setting up N page table entries at once.
The N entries belong to the same PMD, the same folio and the same VMA,
so ptep++ is a legitimate operation, and locking is taken care of for
you. Some architectures can do a better job of it than just a loop,
but I have hesitated to make too deep a change to architectures I don't
understand well.
One thing I have changed in every architecture is that PG_arch_1 is
now a per-folio bit instead of a per-page bit when used for dcache
clean/dirty tracking. This was something that would have to happen
eventually, and it makes sense to do it now rather than iterate over
every page involved in a cache flush and figure out if it needs to happen.
The point of all this is better performance, and Fengwei Yin has
measured improvement on x86. I suspect you'll see improvement on
your architecture too. Try the new will-it-scale test mentioned here:
https://lore.kernel.org/linux-mm/20230206140639.538867-5-fengwei.yin@intel.com/
You'll need to run it on an XFS filesystem and have
CONFIG_TRANSPARENT_HUGEPAGE set.
This patchset is the basis for much of the anonymous large folio work
being done by Ryan, so it's received quite a lot of testing over the
last few months.
v6:
- Fix other definitions of in_range(); either renaming them to not conflict
or removing them and using the new common definition.
- Use arch_enter_lazy_mmu_mode() and arch_leave_lazy_mmu_mode()
- Fix a compiler warning in the riscv implementation
- Update to next-20230731
v5:
- Add in_range() macro
- Fix numerous compilation problems on minority architectures (thanks LKP!)
- Add the 'vmf' argument to update_mmu_cache_range() to help MIPS
and other architectures that insert TLB entries in software,
rather than using a hardware page table walker
- Get rid of first_map_page() and next_map_page(); use
next_uptodate_folio() directly
- Actually move the mmap_miss accounting in filemap.c
- Add kernel-doc for set_pte_range()
- Correct determination of prefaulting in set_pte_range()
- More Acked & Reviewed tags
v4:
- Fix a few compile errors (mostly Mike Rapoport)
- Incorporate Mike's suggestion to avoid having to define set_ptes()
or set_pte_at() on the majority of architectures
- Optimise m68k's __flush_pages_to_ram (Geert Uytterhoeven)
- Fix sun3 (me)
- Fix sparc32 (me)
- Pick up a few more Ack/Reviewed tags
v3:
- Reinstate flush_dcache_icache_phys() on PowerPC
- Fix folio_flush_mapping(). The documentation was correct and the
implementation was completely wrong
- Change the flush_dcache_page() documentation to describe
flush_dcache_folio() instead
- Split ARM from ARC. I messed up my git commands
- Remove page_mapping_file()
- Rationalise how flush_icache_pages() and flush_icache_page() are defined
- Use flush_icache_pages() in do_set_pmd()
- Pick up Guo Ren's Ack for csky
Matthew Wilcox (Oracle) (34):
minmax: Add in_range() macro
mm: Convert page_table_check_pte_set() to page_table_check_ptes_set()
mm: Add generic flush_icache_pages() and documentation
mm: Add folio_flush_mapping()
mm: Remove ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO
mm: Add default definition of set_ptes()
alpha: Implement the new page table range API
arc: Implement the new page table range API
arm: Implement the new page table range API
arm64: Implement the new page table range API
csky: Implement the new page table range API
hexagon: Implement the new page table range API
ia64: Implement the new page table range API
loongarch: Implement the new page table range API
m68k: Implement the new page table range API
microblaze: Implement the new page table range API
mips: Implement the new page table range API
nios2: Implement the new page table range API
openrisc: Implement the new page table range API
parisc: Implement the new page table range API
powerpc: Implement the new page table range API
riscv: Implement the new page table range API
s390: Implement the new page table range API
sh: Implement the new page table range API
sparc32: Implement the new page table range API
sparc64: Implement the new page table range API
um: Implement the new page table range API
x86: Implement the new page table range API
xtensa: Implement the new page table range API
mm: Remove page_mapping_file()
mm: Rationalise flush_icache_pages() and flush_icache_page()
mm: Tidy up set_ptes definition
mm: Use flush_icache_pages() in do_set_pmd()
mm: Call update_mmu_cache_range() in more page fault handling paths
Yin Fengwei (4):
filemap: Add filemap_map_folio_range()
rmap: add folio_add_file_rmap_range()
mm: Convert do_set_pte() to set_pte_range()
filemap: Batch PTE mappings
Documentation/core-api/cachetlb.rst | 55 ++++----
Documentation/filesystems/locking.rst | 2 +-
arch/alpha/include/asm/cacheflush.h | 13 +-
arch/alpha/include/asm/pgtable.h | 10 +-
arch/arc/include/asm/cacheflush.h | 14 +-
arch/arc/include/asm/pgtable-bits-arcv2.h | 12 +-
arch/arc/include/asm/pgtable-levels.h | 1 +
arch/arc/mm/cache.c | 61 +++++----
arch/arc/mm/tlb.c | 18 ++-
arch/arm/include/asm/cacheflush.h | 29 ++---
arch/arm/include/asm/pgtable.h | 5 +-
arch/arm/include/asm/tlbflush.h | 14 +-
arch/arm/mm/copypage-v4mc.c | 5 +-
arch/arm/mm/copypage-v6.c | 5 +-
arch/arm/mm/copypage-xscale.c | 5 +-
arch/arm/mm/dma-mapping.c | 24 ++--
arch/arm/mm/fault-armv.c | 16 +--
arch/arm/mm/flush.c | 99 ++++++++------
arch/arm/mm/mm.h | 2 +-
arch/arm/mm/mmu.c | 14 +-
arch/arm/mm/nommu.c | 6 +
arch/arm/mm/pageattr.c | 6 +-
arch/arm64/include/asm/cacheflush.h | 4 +-
arch/arm64/include/asm/pgtable.h | 26 +++-
arch/arm64/mm/flush.c | 36 ++---
arch/csky/abiv1/cacheflush.c | 32 +++--
arch/csky/abiv1/inc/abi/cacheflush.h | 3 +-
arch/csky/abiv2/cacheflush.c | 32 ++---
arch/csky/abiv2/inc/abi/cacheflush.h | 11 +-
arch/csky/include/asm/pgtable.h | 8 +-
arch/hexagon/include/asm/cacheflush.h | 10 +-
arch/hexagon/include/asm/pgtable.h | 9 +-
arch/ia64/hp/common/sba_iommu.c | 26 ++--
arch/ia64/include/asm/cacheflush.h | 14 +-
arch/ia64/include/asm/pgtable.h | 4 +-
arch/ia64/mm/init.c | 28 ++--
arch/loongarch/include/asm/cacheflush.h | 1 -
arch/loongarch/include/asm/pgtable-bits.h | 4 +-
arch/loongarch/include/asm/pgtable.h | 33 ++---
arch/loongarch/mm/pgtable.c | 2 +-
arch/loongarch/mm/tlb.c | 2 +-
arch/m68k/include/asm/cacheflush_mm.h | 26 ++--
arch/m68k/include/asm/mcf_pgtable.h | 1 +
arch/m68k/include/asm/motorola_pgtable.h | 1 +
arch/m68k/include/asm/pgtable_mm.h | 10 +-
arch/m68k/include/asm/sun3_pgtable.h | 1 +
arch/m68k/mm/motorola.c | 2 +-
arch/microblaze/include/asm/cacheflush.h | 8 ++
arch/microblaze/include/asm/pgtable.h | 15 +--
arch/microblaze/include/asm/tlbflush.h | 4 +-
arch/mips/bcm47xx/prom.c | 2 +-
arch/mips/include/asm/cacheflush.h | 32 +++--
arch/mips/include/asm/pgtable-32.h | 10 +-
arch/mips/include/asm/pgtable-64.h | 6 +-
arch/mips/include/asm/pgtable-bits.h | 6 +-
arch/mips/include/asm/pgtable.h | 63 +++++----
arch/mips/mm/c-r4k.c | 5 +-
arch/mips/mm/cache.c | 56 ++++----
arch/mips/mm/init.c | 21 +--
arch/mips/mm/pgtable-32.c | 2 +-
arch/mips/mm/pgtable-64.c | 2 +-
arch/mips/mm/tlbex.c | 2 +-
arch/nios2/include/asm/cacheflush.h | 6 +-
arch/nios2/include/asm/pgtable.h | 28 ++--
arch/nios2/mm/cacheflush.c | 79 ++++++-----
arch/openrisc/include/asm/cacheflush.h | 8 +-
arch/openrisc/include/asm/pgtable.h | 15 ++-
arch/openrisc/mm/cache.c | 12 +-
arch/parisc/include/asm/cacheflush.h | 14 +-
arch/parisc/include/asm/pgtable.h | 37 ++++--
arch/parisc/kernel/cache.c | 107 ++++++++++-----
arch/powerpc/include/asm/book3s/32/pgtable.h | 5 -
arch/powerpc/include/asm/book3s/64/pgtable.h | 6 +-
arch/powerpc/include/asm/book3s/pgtable.h | 11 +-
arch/powerpc/include/asm/cacheflush.h | 14 +-
arch/powerpc/include/asm/kvm_ppc.h | 10 +-
arch/powerpc/include/asm/nohash/pgtable.h | 16 +--
arch/powerpc/include/asm/pgtable.h | 12 ++
arch/powerpc/mm/book3s64/hash_utils.c | 11 +-
arch/powerpc/mm/cacheflush.c | 40 ++----
arch/powerpc/mm/nohash/e500_hugetlbpage.c | 3 +-
arch/powerpc/mm/pgtable.c | 53 ++++----
arch/riscv/include/asm/cacheflush.h | 19 ++-
arch/riscv/include/asm/pgtable.h | 37 ++++--
arch/riscv/mm/cacheflush.c | 13 +-
arch/s390/include/asm/pgtable.h | 33 +++--
arch/sh/include/asm/cacheflush.h | 21 ++-
arch/sh/include/asm/pgtable.h | 7 +-
arch/sh/include/asm/pgtable_32.h | 5 +-
arch/sh/mm/cache-j2.c | 4 +-
arch/sh/mm/cache-sh4.c | 26 ++--
arch/sh/mm/cache-sh7705.c | 26 ++--
arch/sh/mm/cache.c | 52 ++++----
arch/sh/mm/kmap.c | 3 +-
arch/sparc/include/asm/cacheflush_32.h | 10 +-
arch/sparc/include/asm/cacheflush_64.h | 19 +--
arch/sparc/include/asm/pgtable_32.h | 8 +-
arch/sparc/include/asm/pgtable_64.h | 29 ++++-
arch/sparc/kernel/smp_64.c | 56 +++++---
arch/sparc/mm/init_32.c | 13 +-
arch/sparc/mm/init_64.c | 78 ++++++-----
arch/sparc/mm/tlb.c | 5 +-
arch/um/include/asm/pgtable.h | 7 +-
arch/x86/include/asm/pgtable.h | 14 +-
arch/xtensa/include/asm/cacheflush.h | 11 +-
arch/xtensa/include/asm/pgtable.h | 18 ++-
arch/xtensa/mm/cache.c | 83 +++++++-----
.../drm/arm/display/include/malidp_utils.h | 2 +-
.../display/komeda/komeda_pipeline_state.c | 24 ++--
drivers/gpu/drm/msm/adreno/a6xx_gmu.c | 6 -
.../net/ethernet/chelsio/cxgb3/cxgb3_main.c | 18 +--
drivers/virt/acrn/ioreq.c | 4 +-
fs/btrfs/misc.h | 2 -
fs/ext2/balloc.c | 2 -
fs/ext4/ext4.h | 2 -
fs/ufs/util.h | 6 -
include/asm-generic/cacheflush.h | 7 -
include/linux/cacheflush.h | 13 +-
include/linux/minmax.h | 27 ++++
include/linux/mm.h | 3 +-
include/linux/page_table_check.h | 13 +-
include/linux/pagemap.h | 28 ++--
include/linux/pgtable.h | 75 ++++++++---
include/linux/rmap.h | 2 +
lib/logic_pio.c | 3 -
mm/filemap.c | 123 ++++++++++--------
mm/memory.c | 56 ++++----
mm/page_table_check.c | 16 ++-
mm/rmap.c | 60 +++++++--
mm/util.c | 2 +-
net/netfilter/nf_nat_core.c | 6 +-
net/tipc/core.h | 2 +-
net/tipc/link.c | 10 +-
.../selftests/bpf/progs/get_branch_snapshot.c | 4 +-
134 files changed, 1520 insertions(+), 1056 deletions(-)
--
2.40.1
next reply other threads:[~2023-08-02 15:15 UTC|newest]
Thread overview: 52+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-02 15:13 Matthew Wilcox (Oracle) [this message]
2023-08-02 15:13 ` [PATCH v6 01/38] minmax: Add in_range() macro Matthew Wilcox (Oracle)
2023-08-03 13:00 ` Phi Nguyen
2023-08-03 13:22 ` Matthew Wilcox
2023-08-03 19:11 ` Phi Nguyen
2023-08-02 15:13 ` [PATCH v6 02/38] mm: Convert page_table_check_pte_set() to page_table_check_ptes_set() Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 03/38] mm: Add generic flush_icache_pages() and documentation Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 04/38] mm: Add folio_flush_mapping() Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 05/38] mm: Remove ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 06/38] mm: Add default definition of set_ptes() Matthew Wilcox (Oracle)
2023-10-12 13:53 ` David Woodhouse
2023-10-12 14:05 ` Matthew Wilcox
2023-10-12 14:43 ` David Woodhouse
2023-08-02 15:13 ` [PATCH v6 07/38] alpha: Implement the new page table range API Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 08/38] arc: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 09/38] arm: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 10/38] arm64: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 11/38] csky: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 12/38] hexagon: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 13/38] ia64: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 14/38] loongarch: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 15/38] m68k: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 16/38] microblaze: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 17/38] mips: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 18/38] nios2: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 19/38] openrisc: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 20/38] parisc: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 21/38] powerpc: " Matthew Wilcox (Oracle)
2023-08-03 23:38 ` Nathan Chancellor
2023-08-04 3:50 ` Matthew Wilcox
2023-08-02 15:13 ` [PATCH v6 22/38] riscv: " Matthew Wilcox (Oracle)
2023-09-01 16:25 ` patchwork-bot+linux-riscv
2023-08-02 15:13 ` [PATCH v6 23/38] s390: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 24/38] sh: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 25/38] sparc32: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 26/38] sparc64: " Matthew Wilcox (Oracle)
2023-09-04 15:36 ` Guenter Roeck
2023-09-04 17:43 ` Mike Rapoport
2023-09-04 19:37 ` Guenter Roeck
2023-08-02 15:13 ` [PATCH v6 27/38] um: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 28/38] x86: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 29/38] xtensa: " Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 30/38] mm: Remove page_mapping_file() Matthew Wilcox (Oracle)
2023-08-02 15:13 ` [PATCH v6 31/38] mm: Rationalise flush_icache_pages() and flush_icache_page() Matthew Wilcox (Oracle)
2023-08-02 15:14 ` [PATCH v6 32/38] mm: Tidy up set_ptes definition Matthew Wilcox (Oracle)
2023-08-02 15:14 ` [PATCH v6 33/38] mm: Use flush_icache_pages() in do_set_pmd() Matthew Wilcox (Oracle)
2023-08-02 15:14 ` [PATCH v6 34/38] filemap: Add filemap_map_folio_range() Matthew Wilcox (Oracle)
2023-08-02 15:14 ` [PATCH v6 35/38] rmap: add folio_add_file_rmap_range() Matthew Wilcox (Oracle)
2023-08-02 15:14 ` [PATCH v6 36/38] mm: Convert do_set_pte() to set_pte_range() Matthew Wilcox (Oracle)
2023-08-02 15:14 ` [PATCH v6 37/38] filemap: Batch PTE mappings Matthew Wilcox (Oracle)
2023-08-02 15:14 ` [PATCH v6 38/38] mm: Call update_mmu_cache_range() in more page fault handling paths Matthew Wilcox (Oracle)
2023-08-02 18:43 ` [PATCH v6 00/38] New page table range API Andrew Morton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230802151406.3735276-1-willy@infradead.org \
--to=willy@infradead.org \
--cc=akpm@linux-foundation.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox