From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D27EC432C0 for ; Sun, 24 Nov 2019 08:57:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 80AB520706 for ; Sun, 24 Nov 2019 08:57:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="q6VzoeuY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 80AB520706 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F0D096B0550; Sun, 24 Nov 2019 03:57:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EBD6B6B0551; Sun, 24 Nov 2019 03:57:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAC786B0552; Sun, 24 Nov 2019 03:57:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id C193C6B0550 for ; Sun, 24 Nov 2019 03:57:28 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id 2810B8249980 for ; Sun, 24 Nov 2019 08:57:28 +0000 (UTC) X-FDA: 76190567376.20.part66_7b5b08acba50c X-HE-Tag: part66_7b5b08acba50c X-Filterd-Recvd-Size: 12912 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Nov 2019 08:57:27 +0000 (UTC) Received: from aquarius.haifa.ibm.com (nesher1.haifa.il.ibm.com [195.110.40.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 62E6320706; Sun, 24 Nov 2019 08:57:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1574585846; bh=cMkCvQAGz3bA5X1biLYMWcwJA3SH3fllDC7ywe5VyFc=; h=From:To:Cc:Subject:Date:From; b=q6VzoeuYrYqyHpqrjmQ57nAWIZYFr26I9nf0e8Snt29o35sBPckCUoXLcNaPipoRc MYJxMomkRsl7Squ5CYTr0PIm5gry8Y+X5LxfSWVYpEgp45DWjXU4WSnpTF3zuCNr5/ FE+oHVoaOMXqUT9gAKAyxlptuIT6w90pt9gibU1g= From: Mike Rapoport To: "David S. Miller" Cc: sparclinux@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Mike Rapoport , Mike Rapoport Subject: [PATCH] sparc64: add support for folded p4d page tables Date: Sun, 24 Nov 2019 10:57:20 +0200 Message-Id: <20191124085720.6201-1-rppt@kernel.org> X-Mailer: git-send-email 2.24.0 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mike Rapoport Implement primitives necessary for the 4th level folding, add walks of p4= d level where appropriate and replace 5leve-fixup.h with pgtable-nop4d.h. Signed-off-by: Mike Rapoport --- arch/sparc/include/asm/pgalloc_64.h | 6 +++--- arch/sparc/include/asm/pgtable_64.h | 24 ++++++++++----------- arch/sparc/kernel/signal32.c | 6 +++++- arch/sparc/kernel/smp_64.c | 13 +++++++++++- arch/sparc/mm/fault_64.c | 6 +++++- arch/sparc/mm/hugetlbpage.c | 28 +++++++++++++++--------- arch/sparc/mm/init_64.c | 33 +++++++++++++++++++++++++---- 7 files changed, 84 insertions(+), 32 deletions(-) diff --git a/arch/sparc/include/asm/pgalloc_64.h b/arch/sparc/include/asm= /pgalloc_64.h index 9d3e5cc95bbb..264e76ceccf6 100644 --- a/arch/sparc/include/asm/pgalloc_64.h +++ b/arch/sparc/include/asm/pgalloc_64.h @@ -16,12 +16,12 @@ =20 extern struct kmem_cache *pgtable_cache; =20 -static inline void __pgd_populate(pgd_t *pgd, pud_t *pud) +static inline void __p4d_populate(p4d_t *p4d, pud_t *pud) { - pgd_set(pgd, pud); + p4d_set(p4d, pud); } =20 -#define pgd_populate(MM, PGD, PUD) __pgd_populate(PGD, PUD) +#define p4d_populate(MM, P4D, PUD) __p4d_populate(P4D, PUD) =20 static inline pgd_t *pgd_alloc(struct mm_struct *mm) { diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm= /pgtable_64.h index 6ae8016ef4ec..34ff3b43afbb 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -13,7 +13,7 @@ * the SpitFire page tables. */ =20 -#include +#include #include #include #include @@ -810,9 +810,9 @@ static inline int pmd_present(pmd_t pmd) =20 #define pud_bad(pud) (pud_val(pud) & ~PAGE_MASK) =20 -#define pgd_none(pgd) (!pgd_val(pgd)) +#define p4d_none(p4d) (!p4d_val(p4d)) =20 -#define pgd_bad(pgd) (pgd_val(pgd) & ~PAGE_MASK) +#define p4d_bad(p4d) (p4d_val(p4d) & ~PAGE_MASK) =20 #ifdef CONFIG_TRANSPARENT_HUGEPAGE void set_pmd_at(struct mm_struct *mm, unsigned long addr, @@ -859,13 +859,13 @@ static inline unsigned long pud_page_vaddr(pud_t pu= d) #define pmd_clear(pmdp) (pmd_val(*(pmdp)) =3D 0UL) #define pud_present(pud) (pud_val(pud) !=3D 0U) #define pud_clear(pudp) (pud_val(*(pudp)) =3D 0UL) -#define pgd_page_vaddr(pgd) \ - ((unsigned long) __va(pgd_val(pgd))) -#define pgd_present(pgd) (pgd_val(pgd) !=3D 0U) -#define pgd_clear(pgdp) (pgd_val(*(pgdp)) =3D 0UL) +#define p4d_page_vaddr(p4d) \ + ((unsigned long) __va(p4d_val(p4d))) +#define p4d_present(p4d) (p4d_val(p4d) !=3D 0U) +#define p4d_clear(p4dp) (p4d_val(*(p4dp)) =3D 0UL) =20 /* only used by the stubbed out hugetlb gup code, should never be called= */ -#define pgd_page(pgd) NULL +#define p4d_page(p4d) NULL =20 static inline unsigned long pud_large(pud_t pud) { @@ -884,8 +884,8 @@ static inline unsigned long pud_pfn(pud_t pud) /* Same in both SUN4V and SUN4U. */ #define pte_none(pte) (!pte_val(pte)) =20 -#define pgd_set(pgdp, pudp) \ - (pgd_val(*(pgdp)) =3D (__pa((unsigned long) (pudp)))) +#define p4d_set(p4dp, pudp) \ + (p4d_val(*(p4dp)) =3D (__pa((unsigned long) (pudp)))) =20 /* to find an entry in a page-table-directory. */ #define pgd_index(address) (((address) >> PGDIR_SHIFT) & (PTRS_PER_PGD -= 1)) @@ -896,8 +896,8 @@ static inline unsigned long pud_pfn(pud_t pud) =20 /* Find an entry in the third-level page table.. */ #define pud_index(address) (((address) >> PUD_SHIFT) & (PTRS_PER_PUD - 1= )) -#define pud_offset(pgdp, address) \ - ((pud_t *) pgd_page_vaddr(*(pgdp)) + pud_index(address)) +#define pud_offset(p4dp, address) \ + ((pud_t *) p4d_page_vaddr(*(p4dp)) + pud_index(address)) =20 /* Find an entry in the second-level page table.. */ #define pmd_offset(pudp, address) \ diff --git a/arch/sparc/kernel/signal32.c b/arch/sparc/kernel/signal32.c index a237810aa9f4..2a734ecd0a40 100644 --- a/arch/sparc/kernel/signal32.c +++ b/arch/sparc/kernel/signal32.c @@ -299,6 +299,7 @@ static void flush_signal_insns(unsigned long address) unsigned long pstate, paddr; pte_t *ptep, pte; pgd_t *pgdp; + p4d_t *p4dp; pud_t *pudp; pmd_t *pmdp; =20 @@ -318,7 +319,10 @@ static void flush_signal_insns(unsigned long address= ) pgdp =3D pgd_offset(current->mm, address); if (pgd_none(*pgdp)) goto out_irqs_on; - pudp =3D pud_offset(pgdp, address); + p4dp =3D p4d_offset(pgdp, address); + if (p4d_none(*p4dp)) + goto out_irqs_on; + pudp =3D pud_offset(p4dp, address); if (pud_none(*pudp)) goto out_irqs_on; pmdp =3D pmd_offset(pudp, address); diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c index a8275fea4b70..eb9455798338 100644 --- a/arch/sparc/kernel/smp_64.c +++ b/arch/sparc/kernel/smp_64.c @@ -1621,6 +1621,7 @@ static int __init pcpu_cpu_distance(unsigned int fr= om, unsigned int to) static void __init pcpu_populate_pte(unsigned long addr) { pgd_t *pgd =3D pgd_offset_k(addr); + p4d_t *p4d; pud_t *pud; pmd_t *pmd; =20 @@ -1633,7 +1634,17 @@ static void __init pcpu_populate_pte(unsigned long= addr) pgd_populate(&init_mm, pgd, new); } =20 - pud =3D pud_offset(pgd, addr); + p4d =3D p4d_offset(pgd, addr); + if (p4d_none(*p4d)) { + pud_t *new; + + new =3D memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); + if (!new) + goto err_alloc; + p4d_populate(&init_mm, p4d, new); + } + + pud =3D pud_offset(p4d, addr); if (pud_none(*pud)) { pmd_t *new; =20 diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c index 2371fb6b97e4..8b7ddbd14b65 100644 --- a/arch/sparc/mm/fault_64.c +++ b/arch/sparc/mm/fault_64.c @@ -80,6 +80,7 @@ static void __kprobes bad_kernel_pc(struct pt_regs *reg= s, unsigned long vaddr) static unsigned int get_user_insn(unsigned long tpc) { pgd_t *pgdp =3D pgd_offset(current->mm, tpc); + p4d_t *p4dp; pud_t *pudp; pmd_t *pmdp; pte_t *ptep, pte; @@ -88,7 +89,10 @@ static unsigned int get_user_insn(unsigned long tpc) =20 if (pgd_none(*pgdp) || unlikely(pgd_bad(*pgdp))) goto out; - pudp =3D pud_offset(pgdp, tpc); + p4dp =3D p4d_offset(pgdp, tpc); + if (p4d_none(*p4dp) || unlikely(p4d_bad(*p4dp))) + goto out; + pudp =3D pud_offset(p4dp, tpc); if (pud_none(*pudp) || unlikely(pud_bad(*pudp))) goto out; =20 diff --git a/arch/sparc/mm/hugetlbpage.c b/arch/sparc/mm/hugetlbpage.c index f78793a06bbd..7b9fa861b67c 100644 --- a/arch/sparc/mm/hugetlbpage.c +++ b/arch/sparc/mm/hugetlbpage.c @@ -277,11 +277,13 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz) { pgd_t *pgd; + p4d_t *p4d; pud_t *pud; pmd_t *pmd; =20 pgd =3D pgd_offset(mm, addr); - pud =3D pud_alloc(mm, pgd, addr); + p4d =3D p4d_offset(pgd, addr); + pud =3D pud_alloc(mm, p4d, addr); if (!pud) return NULL; if (sz >=3D PUD_SIZE) @@ -298,13 +300,17 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz) { pgd_t *pgd; + p4d_t *p4d; pud_t *pud; pmd_t *pmd; =20 pgd =3D pgd_offset(mm, addr); if (pgd_none(*pgd)) return NULL; - pud =3D pud_offset(pgd, addr); + p4d =3D p4d_offset(pgd, addr); + if (p4d_none(*p4d)) + return NULL; + pud =3D pud_offset(p4d, addr); if (pud_none(*pud)) return NULL; if (is_hugetlb_pud(*pud)) @@ -449,7 +455,7 @@ static void hugetlb_free_pmd_range(struct mmu_gather = *tlb, pud_t *pud, mm_dec_nr_pmds(tlb->mm); } =20 -static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd, +static void hugetlb_free_pud_range(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling) { @@ -458,7 +464,7 @@ static void hugetlb_free_pud_range(struct mmu_gather = *tlb, pgd_t *pgd, unsigned long start; =20 start =3D addr; - pud =3D pud_offset(pgd, addr); + pud =3D pud_offset(p4d, addr); do { next =3D pud_addr_end(addr, end); if (pud_none_or_clear_bad(pud)) @@ -481,8 +487,8 @@ static void hugetlb_free_pud_range(struct mmu_gather = *tlb, pgd_t *pgd, if (end - 1 > ceiling - 1) return; =20 - pud =3D pud_offset(pgd, start); - pgd_clear(pgd); + pud =3D pud_offset(p4d, start); + p4d_clear(p4d); pud_free_tlb(tlb, pud, start); mm_dec_nr_puds(tlb->mm); } @@ -492,6 +498,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long floor, unsigned long ceiling) { pgd_t *pgd; + p4d_t *p4d; unsigned long next; =20 addr &=3D PMD_MASK; @@ -511,10 +518,11 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, return; =20 pgd =3D pgd_offset(tlb->mm, addr); + p4d =3D p4d_offset(pgd, addr); do { - next =3D pgd_addr_end(addr, end); - if (pgd_none_or_clear_bad(pgd)) + next =3D p4d_addr_end(addr, end); + if (p4d_none_or_clear_bad(p4d)) continue; - hugetlb_free_pud_range(tlb, pgd, addr, next, floor, ceiling); - } while (pgd++, addr =3D next, addr !=3D end); + hugetlb_free_pud_range(tlb, p4d, addr, next, floor, ceiling); + } while (p4d++, addr =3D next, addr !=3D end); } diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index e6d91819da92..1cf0d666dea3 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -530,7 +530,8 @@ void __kprobes flush_icache_range(unsigned long start= , unsigned long end) paddr =3D kaddr & mask; else { pgd_t *pgdp =3D pgd_offset_k(kaddr); - pud_t *pudp =3D pud_offset(pgdp, kaddr); + p4d_t *p4dp =3D p4d_offset(pgdp, kaddr); + pud_t *pudp =3D pud_offset(p4dp, kaddr); pmd_t *pmdp =3D pmd_offset(pudp, kaddr); pte_t *ptep =3D pte_offset_kernel(pmdp, kaddr); =20 @@ -1653,6 +1654,7 @@ static unsigned long max_phys_bits =3D 40; bool kern_addr_valid(unsigned long addr) { pgd_t *pgd; + p4d_t *p4d; pud_t *pud; pmd_t *pmd; pte_t *pte; @@ -1674,7 +1676,11 @@ bool kern_addr_valid(unsigned long addr) if (pgd_none(*pgd)) return 0; =20 - pud =3D pud_offset(pgd, addr); + p4d =3D p4d_offset(pgd, addr); + if (p4d_none(*p4d)) + return 0; + + pud =3D pud_offset(p4d, addr); if (pud_none(*pud)) return 0; =20 @@ -1800,6 +1806,7 @@ static unsigned long __ref kernel_map_range(unsigne= d long pstart, while (vstart < vend) { unsigned long this_end, paddr =3D __pa(vstart); pgd_t *pgd =3D pgd_offset_k(vstart); + p4d_t *p4d; pud_t *pud; pmd_t *pmd; pte_t *pte; @@ -1814,7 +1821,20 @@ static unsigned long __ref kernel_map_range(unsign= ed long pstart, alloc_bytes +=3D PAGE_SIZE; pgd_populate(&init_mm, pgd, new); } - pud =3D pud_offset(pgd, vstart); + + p4d =3D p4d_offset(pgd, vstart); + if (p4d_none(*p4d)) { + pud_t *new; + + new =3D memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, + PAGE_SIZE); + if (!new) + goto err_alloc; + alloc_bytes +=3D PAGE_SIZE; + p4d_populate(&init_mm, p4d, new); + } + + pud =3D pud_offset(p4d, vstart); if (pud_none(*pud)) { pmd_t *new; =20 @@ -2612,13 +2632,18 @@ int __meminit vmemmap_populate(unsigned long vsta= rt, unsigned long vend, for (; vstart < vend; vstart +=3D PMD_SIZE) { pgd_t *pgd =3D vmemmap_pgd_populate(vstart, node); unsigned long pte; + p4d_t *p4d; pud_t *pud; pmd_t *pmd; =20 if (!pgd) return -ENOMEM; =20 - pud =3D vmemmap_pud_populate(pgd, vstart, node); + p4d =3D vmemmap_p4d_populate(pgd, vstart, node); + if (!p4d) + return -ENOMEM; + + pud =3D vmemmap_pud_populate(p4d, vstart, node); if (!pud) return -ENOMEM; =20 --=20 2.24.0