From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DF82C25B10 for ; Fri, 10 May 2024 12:20:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA8D06B00DA; Fri, 10 May 2024 08:20:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E589B6B00DB; Fri, 10 May 2024 08:20:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D47FF6B00DC; Fri, 10 May 2024 08:20:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B6C686B00DA for ; Fri, 10 May 2024 08:20:54 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 44D154179B for ; Fri, 10 May 2024 12:20:54 +0000 (UTC) X-FDA: 82102395228.03.23958C8 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf26.hostedemail.com (Postfix) with ESMTP id 8AA1D140015 for ; Fri, 10 May 2024 12:20:51 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715343651; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oMyG1wEcxtMni94RUo+70MMuWUn0q4HPYJp0UGSakGY=; b=7of4/z0bVVvlsYP63JgeIIKfdsg6GnYzZHf0V0isexY23hwZuYS4ySl7zeW6eBcvI4D72t 6ngZSUqPmkM4g3Ozo/+L3eVQ1x/CYLBN0aJBMXFQhxUOyfNsOO2x6R6f+qCNDJhOGwS6T7 tyTkgzFGEeqFnmr7KfyqcBMVv9fOyus= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715343651; a=rsa-sha256; cv=none; b=lO6OfAQuXOVsUgzYlAn+h4zuKDG/+fHklQHwLm1jslj2H+KtDkCMjEsIoE4d2QIsfXRRCV gD2SWC4M9vqJYFOSIiNu/4USjcFTgLVXpmb2UNuA0cZhKzdYsaCnyaaUBPvd3+yZwhcSlB Ii1mgT+JhnjUYdwQd1YrFAJSj043THw= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C921D106F; Fri, 10 May 2024 05:21:15 -0700 (PDT) Received: from [10.57.65.1] (unknown [10.57.65.1]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BF89A3F6A8; Fri, 10 May 2024 05:20:48 -0700 (PDT) Message-ID: <8aa0a606-f47c-4826-96ad-121ac1f7eb96@arm.com> Date: Fri, 10 May 2024 13:20:47 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH RESEND v2 1/9] riscv: Restore the pfn in a NAPOT pte when manipulated by core mm code Content-Language: en-GB To: Alexandre Ghiti , Catalin Marinas , Will Deacon , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org References: <20240508113419.18620-1-alexghiti@rivosinc.com> <20240508113419.18620-2-alexghiti@rivosinc.com> From: Ryan Roberts In-Reply-To: <20240508113419.18620-2-alexghiti@rivosinc.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Queue-Id: 8AA1D140015 X-Rspamd-Server: rspam06 X-Stat-Signature: zgw8hgi5qi9zqkg8rtwdbeyyo7j9b9t8 X-HE-Tag: 1715343651-723150 X-HE-Meta: U2FsdGVkX1+4FxS8wCQ3vqZwbDEUlVAuT1x6qJIAkYh++1abp7XhwB/Y3dHOpmfq+v2Tl8hdmdhTF3CIfUeG63a3rJIOc2Lh8F6mabm6URZzmEhQQOKyrbUsuVftHdpf3ssKN41QdnLyL1fU5Rb1bnAB9gGyu7DBS68b+L7vc5T+PcCD90gzCbWYA6T9EgDFix/OB7LsPl9TLMGzVTBDPgJX25O54b6lAr4gXmI0zaGQGWykOTtvR4qbC6z2s4Vdiy4tWvhPh1e3LjXTgZsi/fLXN0y/kMPSrxqAG/73uRfkDph2P0lranDIShP6bPBUPDx8t50vmYQUVW92TJNDp5fYqkIbZ0BI8LxTmvkwSpfra7ZyQTFftjmHGhEgdIv7l45NNTRGbYYSFGxJRK5kOstJIdwWamdH82fJdnZ3r51wswn94s0M//ydVR6qwUaV7hvhANZuqgyb0ONTmuIqKAUQTDxFxEqXEfxXDYh4kFGjDsI3Swta6ylgqgGHVrAFW6DPxlyo+hlfDw6aWEx3fBfqL2quPoOPMvKTHYgDlE6Mub4D/t68UZ0pmMjmlWd6RUH3LEBvgCEOHEiVddHbh77C9MhkhFu7wLs6WLEVK7LrBErt4yczoudAAJT1WFry/XOKAF0yJcKp4iScB4LL54sCz8u6lp/zFrn7NLM11UN6Hh+EHUhxv2Zmz8lwBRUJb/DE8x8AuDgyd86+OBncRJ/uPkLlEzBexVUOjztR8nnPcyGpSM0dLZYeMGeC54BPrOwnAoNfwqBe4UvD3NtR0PhnFTIg+mTq8sbzp5lJfPJNnY84MNnP4v9InXVAyF6rk7mkXiEAT5HokMPKnUVZOchm4mmE46GA1CRsyA4LnuvYehGr9y7RZC85iyLA/WR7ohIH34IKe8jFgRz/g7/fV8r6UKKP40qUXzV33KvzqPsC/T9rwThDRGdKy6GzmXm4KZ10b/dJOsdkLJ4a5Y8 Ltf5bOiW K1qyZTviLHmX9LQv58HcJnB+H+OV9zaxPkAwdcl6yBpttmY0GZQGypfQxk2iTdsG5kn3sjfvalpXz1yHBMu6SNykREP6rTwEQCYlX+xltxbe5UaBFJU9MNKfxNJaLQGhqYd2y6hRfdL1ANklYqyOUubqvNwAAe452W0kNSdsy7pWA7f3ku0c+PK/uRc0aJVE+lL4itq217M1/trc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 08/05/2024 12:34, Alexandre Ghiti wrote: > The core mm code expects to be able to extract the pfn from a pte. NAPOT > mappings work differently since its ptes actually point to the first pfn > of the mapping, the other bits being used to encode the size of the > mapping. > > So modify ptep_get() so that it returns a pte value that contains the > *real* pfn (which is then different from what the HW expects) and right > before storing the ptes to the page table, reset the pfn LSBs to the > size of the mapping. Did you consider leaving the pte as is and instead modifying your pte_pfn() implementation? For arm64 at least, it is beneficial to keep the pte marked as contiguous when passing it up to core-mm because there are other helpers which need to parse the contiguous bit (e.g. pte_leaf_size()). If we were to clear the cont bit in ptep_get() that info would be lost and perf_get_pgtable_size() would always conclude the leaf size is 4K even when it is actually 64K. > > And make sure that all NAPOT mappings are set using set_ptes(). > > Signed-off-by: Alexandre Ghiti > --- > arch/riscv/include/asm/pgtable-64.h | 11 +++ > arch/riscv/include/asm/pgtable.h | 105 ++++++++++++++++++++++++++-- > arch/riscv/mm/hugetlbpage.c | 38 +++++----- > 3 files changed, 128 insertions(+), 26 deletions(-) > > diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h > index 221a5c1ee287..9fe076fc503e 100644 > --- a/arch/riscv/include/asm/pgtable-64.h > +++ b/arch/riscv/include/asm/pgtable-64.h > @@ -106,6 +106,17 @@ enum napot_cont_order { > #define napot_cont_mask(order) (~(napot_cont_size(order) - 1UL)) > #define napot_pte_num(order) BIT(order) > > +static inline bool is_napot_order(unsigned int order) > +{ > + unsigned int napot_order; > + > + for_each_napot_order(napot_order) > + if (order == napot_order) > + return true; > + > + return false; > +} > + > #ifdef CONFIG_RISCV_ISA_SVNAPOT > #define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_CONT_ORDER_BASE)) > #else > diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h > index 9f8ea0e33eb1..268c828f5152 100644 > --- a/arch/riscv/include/asm/pgtable.h > +++ b/arch/riscv/include/asm/pgtable.h > @@ -297,6 +297,8 @@ static inline unsigned long pte_napot(pte_t pte) > return pte_val(pte) & _PAGE_NAPOT; > } > > +#define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) > + > static inline pte_t pte_mknapot(pte_t pte, unsigned int order) > { > int pos = order - 1 + _PAGE_PFN_SHIFT; > @@ -306,6 +308,12 @@ static inline pte_t pte_mknapot(pte_t pte, unsigned int order) > return __pte((pte_val(pte) & napot_mask) | napot_bit | _PAGE_NAPOT); > } > > +/* pte at entry must *not* encode the mapping size in the pfn LSBs. */ > +static inline pte_t pte_clear_napot(pte_t pte) > +{ > + return __pte(pte_val(pte) & ~_PAGE_NAPOT); > +} > + > #else > > static __always_inline bool has_svnapot(void) { return false; } > @@ -315,17 +323,14 @@ static inline unsigned long pte_napot(pte_t pte) > return 0; > } > > +#define pte_valid_napot(pte) false > + > #endif /* CONFIG_RISCV_ISA_SVNAPOT */ > > /* Yields the page frame number (PFN) of a page table entry */ > static inline unsigned long pte_pfn(pte_t pte) > { > - unsigned long res = __page_val_to_pfn(pte_val(pte)); > - > - if (has_svnapot() && pte_napot(pte)) > - res = res & (res - 1UL); > - > - return res; > + return __page_val_to_pfn(pte_val(pte)); > } > > #define pte_page(x) pfn_to_page(pte_pfn(x)) > @@ -525,9 +530,91 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) > > #define PFN_PTE_SHIFT _PAGE_PFN_SHIFT > > +#ifdef CONFIG_RISCV_ISA_SVNAPOT > +static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, > + size_t *pgsize) > +{ > + pte_t __pte; > + > + /* We must read the raw value of the pte to get the size of the mapping */ > + __pte = READ_ONCE(*ptep); > + > + if (pgsize) { > + if (size >= PGDIR_SIZE) > + *pgsize = PGDIR_SIZE; > + else if (size >= P4D_SIZE) > + *pgsize = P4D_SIZE; > + else if (size >= PUD_SIZE) > + *pgsize = PUD_SIZE; > + else if (size >= PMD_SIZE) > + *pgsize = PMD_SIZE; > + else > + *pgsize = PAGE_SIZE; > + } > + > + /* Make sure __pte is not a swap entry */ > + if (pte_valid_napot(__pte)) > + return napot_pte_num(napot_cont_order(__pte)); > + > + return 1; > +} > +#endif > + > +static inline pte_t ptep_get(pte_t *ptep) > +{ > + pte_t pte = READ_ONCE(*ptep); > + > +#ifdef CONFIG_RISCV_ISA_SVNAPOT > + /* > + * The pte we load has the N bit set and the size of the mapping in > + * the pfn LSBs: keep the N bit and replace the mapping size with > + * the *real* pfn since the core mm code expects to find it there. > + * The mapping size will be reset just before being written to the > + * page table in set_ptes(). > + */ > + if (unlikely(pte_valid_napot(pte))) { > + unsigned int order = napot_cont_order(pte); > + int pos = order - 1 + _PAGE_PFN_SHIFT; > + unsigned long napot_mask = ~GENMASK(pos, _PAGE_PFN_SHIFT); > + pte_t *orig_ptep = PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * napot_pte_num(order)); > + > + pte = __pte((pte_val(pte) & napot_mask) + ((ptep - orig_ptep) << _PAGE_PFN_SHIFT)); > + } > +#endif > + > + return pte; > +} > +#define ptep_get ptep_get > + > static inline void set_ptes(struct mm_struct *mm, unsigned long addr, > pte_t *ptep, pte_t pteval, unsigned int nr) > { > +#ifdef CONFIG_RISCV_ISA_SVNAPOT > + if (unlikely(pte_valid_napot(pteval))) { > + unsigned int order = ilog2(nr); > + > + if (!is_napot_order(order)) { > + /* > + * Something's weird, we are given a NAPOT pte but the > + * size of the mapping is not a known NAPOT mapping > + * size, so clear the NAPOT bit and map this without > + * NAPOT support: core mm only manipulates pte with the > + * real pfn so we know the pte is valid without the N > + * bit. > + */ > + pr_err("Incorrect NAPOT mapping, resetting.\n"); > + pteval = pte_clear_napot(pteval); > + } else { > + /* > + * NAPOT ptes that arrive here only have the N bit set > + * and their pfn does not contain the mapping size, so > + * set that here. > + */ > + pteval = pte_mknapot(pteval, order); > + } > + } > +#endif I think all this complexity comes along due to using this function both as a public interface that the core-mm uses (which never sets napot) and also using it as an internal interface that riscv-hugetlb uses (which does set napot)? It might be more understandable if you layer it into a lower level/internal API and a higher level/public API (similar to arm64)? > + > page_table_check_ptes_set(mm, ptep, pteval, nr); > > for (;;) { > @@ -535,6 +622,12 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, > if (--nr == 0) > break; > ptep++; > + > +#ifdef CONFIG_RISCV_ISA_SVNAPOT > + if (unlikely(pte_valid_napot(pteval))) > + continue; > +#endif > + > pte_val(pteval) += 1 << _PAGE_PFN_SHIFT; > } > } > diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c > index 5ef2a6891158..fe8067ee71b4 100644 > --- a/arch/riscv/mm/hugetlbpage.c > +++ b/arch/riscv/mm/hugetlbpage.c > @@ -256,8 +256,7 @@ void set_huge_pte_at(struct mm_struct *mm, > > clear_flush(mm, addr, ptep, pgsize, pte_num); > > - for (i = 0; i < pte_num; i++, ptep++, addr += pgsize) > - set_pte_at(mm, addr, ptep, pte); > + set_ptes(mm, addr, ptep, pte, pte_num); > } > > int huge_ptep_set_access_flags(struct vm_area_struct *vma, > @@ -267,16 +266,16 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, > int dirty) > { > struct mm_struct *mm = vma->vm_mm; > - unsigned long order; > + size_t pgsize; > pte_t orig_pte; > - int i, pte_num; > + int pte_num; > > if (!pte_napot(pte)) > return ptep_set_access_flags(vma, addr, ptep, pte, dirty); > > - order = napot_cont_order(pte); > - pte_num = napot_pte_num(order); > - ptep = huge_pte_offset(mm, addr, napot_cont_size(order)); > + pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); > + ptep = huge_pte_offset(mm, addr, pte_num * pgsize); > + > orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); > > if (pte_dirty(orig_pte)) > @@ -285,8 +284,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, > if (pte_young(orig_pte)) > pte = pte_mkyoung(pte); > > - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) > - set_pte_at(mm, addr, ptep, pte); > + set_ptes(mm, addr, ptep, pte, pte_num); > > return true; > } > @@ -301,7 +299,7 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, > if (!pte_napot(orig_pte)) > return ptep_get_and_clear(mm, addr, ptep); > > - pte_num = napot_pte_num(napot_cont_order(orig_pte)); > + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); > > return get_clear_contig(mm, addr, ptep, pte_num); > } > @@ -311,24 +309,23 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, > pte_t *ptep) > { > pte_t pte = ptep_get(ptep); > - unsigned long order; > + size_t pgsize; > pte_t orig_pte; > - int i, pte_num; > + int pte_num; > > if (!pte_napot(pte)) { > ptep_set_wrprotect(mm, addr, ptep); > return; > } > > - order = napot_cont_order(pte); > - pte_num = napot_pte_num(order); > - ptep = huge_pte_offset(mm, addr, napot_cont_size(order)); > + pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); > + ptep = huge_pte_offset(mm, addr, pte_num * pgsize); > + > orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); > > orig_pte = pte_wrprotect(orig_pte); > > - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) > - set_pte_at(mm, addr, ptep, orig_pte); > + set_ptes(mm, addr, ptep, orig_pte, pte_num); > } > > pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, > @@ -341,7 +338,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, > if (!pte_napot(pte)) > return ptep_clear_flush(vma, addr, ptep); > > - pte_num = napot_pte_num(napot_cont_order(pte)); > + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); > > return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); > } > @@ -351,6 +348,7 @@ void huge_pte_clear(struct mm_struct *mm, > pte_t *ptep, > unsigned long sz) > { > + size_t pgsize; > pte_t pte = ptep_get(ptep); > int i, pte_num; > > @@ -359,8 +357,8 @@ void huge_pte_clear(struct mm_struct *mm, > return; > } > > - pte_num = napot_pte_num(napot_cont_order(pte)); > - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) > + pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); > + for (i = 0; i < pte_num; i++, addr += pgsize, ptep++) > pte_clear(mm, addr, ptep); > } >