From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 800B3C48BEB for ; Thu, 15 Feb 2024 11:28:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10C996B00A7; Thu, 15 Feb 2024 06:28:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 096548D0013; Thu, 15 Feb 2024 06:28:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E78FE8D0001; Thu, 15 Feb 2024 06:28:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D1F036B00A7 for ; Thu, 15 Feb 2024 06:28:29 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6EC6AC0FDD for ; Thu, 15 Feb 2024 11:28:29 +0000 (UTC) X-FDA: 81793815138.01.E2C66BE Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf06.hostedemail.com (Postfix) with ESMTP id B1BE4180008 for ; Thu, 15 Feb 2024 11:28:27 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707996507; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YYG4qqZ87z3zRtWvZEpJsw+DeHVIlNICBjZSc8J3JrU=; b=KxGyKRpNW9XeG+GseY9z01sbu50I2inVSCBQbRH8ly2KCigmhq20smGl/JbkcrpWd30NYx bQIJBCu24klDDgqSoNEgplUT02MHcssbOSlLSQnAYJ1Vpm7pApISQkBTSmFozehxheKojB 1aiB4EqGEbB6I8KlEpup2i6ckC/GDQU= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707996507; a=rsa-sha256; cv=none; b=mdOJgvwXRZIHeO8UVkDb68pTWbkNkgl/alrm41+vOMGtwXGkUbp4Y+nHxGAfo7sm9idUG5 Aj8NE/2t0eTvUOYRpNZTTmhufvgQ8F+Ipd2SVfeseiMGS3aC3wM9mDiXpmIQno241vDOGL xT4Y/8ovOWti+GG5ieEZoOhPhYnYgPo= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D2B3ADA7; Thu, 15 Feb 2024 03:29:07 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.68.11]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 522743F7B4; Thu, 15 Feb 2024 03:28:23 -0800 (PST) Date: Thu, 15 Feb 2024 11:28:20 +0000 From: Mark Rutland To: Ryan Roberts Cc: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v6 13/18] arm64/mm: Implement new wrprotect_ptes() batch API Message-ID: References: <20240215103205.2607016-1-ryan.roberts@arm.com> <20240215103205.2607016-14-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240215103205.2607016-14-ryan.roberts@arm.com> X-Rspamd-Queue-Id: B1BE4180008 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: yrxcdca5uy5rmn5jn37nbpmq5b5j6hy9 X-HE-Tag: 1707996507-93821 X-HE-Meta: U2FsdGVkX1/0D92E/+pj5r49u5y0eYFeanHNPrVkWBDEDrpVWvvgaNByaHk8FoEaTrA8LWH5rtVpyRpcPoqeV34p37ZjFfQXpw45Uh76BKn2LUK5+lj3IgZ08K2wxyUuQiS82nRsSunOhanJTzbAJuwmZrHUVNEib3M9d9WiupO3/QngAJ03CqzBVlEp4dlccSvv68CGEQfJFisOnMI65RSpzX42755PqKzF/xVbpVaaEP9hDBlp8CVUw0QKgBmqpuTOrDQfJHUni2W77oMoBEe7x49MiWNRdmAu8Wl3UwQPXxzkTnf6vXX201HLRf0mmU2XCc9vuiAka/vbpU1FGf/azc6dyCqxvTWYtD9ZpuBqDaopgyS2Q/4IM7JTnVhBSCPnURbP/6MKy5UHO2Q/s3K7n4OLCbJavhiawmfNgq6oc6QdJJfCtKh4IqzgY/+mxjbzSFucp/AknshKDy9FEW85MPWeMjeI5sT1G/EvbybJN6bkDXX3zdm851cnpDg6Y0wlDiEOJw/gY0rv+SARKoLnbXNQP/3c6FeiTmyAyMcSFta/8KwDQRsE6+NQ3Qvgo4qkVyeQ/XNses2z7N1A8qWa6s8VSzmAC+knzGpvlwu1xYm82upP9COuo6xSXO40pFN7+4h8V6A+az8GmgkkGDcQ43rRhgSBM3nQk5ejBnZ4CM2j4tysFSD58OZtJsqR8XkZ0btrxjYn1vsw1O/oZBZBgBau+uboJdcDWeRGO/pCzdBy7SFpko3SGGSfVCyw/zzmqts0uE6GtR2rgUKiLrorwsNv57i5A5VCmEa2yiVqE5wkIL6fBfaTmMEWcGXt9wrCJLB7bOQafgWnmCvdSnZhJa45kmPmN0coQY6B2XzAphc1YTAmErPC4K7xOZ4xCM+p+XTkStL8TpyCiQ3DLslOG5RJWE9xIp7qvD753YKa1KKN4eLbhMQL1CXZvEMFs2s/tVodhMWpYx6ReSz yqq318Xh 87ZBufyOci6k5CNoS2CyXRYE8RxpsaTfgGzXYAzSlgOQeivcS0RT3j6hVb4Mt6lzSMDhonzENZYtmAUe5iqoEb6GeaxZny/QZ0z4mFjdkF8ng5De8DE59L7G2uAWzlvKfhZAkMoV48M1fpsiGoazdTEiV6X8k+DG48w+dDD5bIEpN7foBLVQZYNci0dvN88+cKh/wE2MRMmxu0gp/tV4JMNA5gZON64BMkJBdAQbKhsycQGpAY/Lr+I6ksMl9hczlp5HctjCeYxdIGfL0fLsrn+BBu7OC7TWU8lzEiQ6NzfrFwRv4jkyPHOaYaQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Feb 15, 2024 at 10:32:00AM +0000, Ryan Roberts wrote: > Optimize the contpte implementation to fix some of the fork performance > regression introduced by the initial contpte commit. Subsequent patches > will solve it entirely. > > During fork(), any private memory in the parent must be write-protected. > Previously this was done 1 PTE at a time. But the core-mm supports > batched wrprotect via the new wrprotect_ptes() API. So let's implement > that API and for fully covered contpte mappings, we no longer need to > unfold the contpte. This has 2 benefits: > > - reduced unfolding, reduces the number of tlbis that must be issued. > - The memory remains contpte-mapped ("folded") in the parent, so it > continues to benefit from the more efficient use of the TLB after > the fork. > > The optimization to wrprotect a whole contpte block without unfolding is > possible thanks to the tightening of the Arm ARM in respect to the > definition and behaviour when 'Misprogramming the Contiguous bit'. See > section D21194 at https://developer.arm.com/documentation/102105/ja-07/ > > Tested-by: John Hubbard > Signed-off-by: Ryan Roberts Acked-by: Mark Rutland Mark. > --- > arch/arm64/include/asm/pgtable.h | 61 ++++++++++++++++++++++++++------ > arch/arm64/mm/contpte.c | 38 ++++++++++++++++++++ > 2 files changed, 89 insertions(+), 10 deletions(-) > > diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h > index 831099cfc96b..8643227c318b 100644 > --- a/arch/arm64/include/asm/pgtable.h > +++ b/arch/arm64/include/asm/pgtable.h > @@ -978,16 +978,12 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, > } > #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ > > -/* > - * __ptep_set_wrprotect - mark read-only while trasferring potential hardware > - * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. > - */ > -static inline void __ptep_set_wrprotect(struct mm_struct *mm, > - unsigned long address, pte_t *ptep) > +static inline void ___ptep_set_wrprotect(struct mm_struct *mm, > + unsigned long address, pte_t *ptep, > + pte_t pte) > { > - pte_t old_pte, pte; > + pte_t old_pte; > > - pte = __ptep_get(ptep); > do { > old_pte = pte; > pte = pte_wrprotect(pte); > @@ -996,6 +992,25 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, > } while (pte_val(pte) != pte_val(old_pte)); > } > > +/* > + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware > + * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. > + */ > +static inline void __ptep_set_wrprotect(struct mm_struct *mm, > + unsigned long address, pte_t *ptep) > +{ > + ___ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); > +} > + > +static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, > + pte_t *ptep, unsigned int nr) > +{ > + unsigned int i; > + > + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) > + __ptep_set_wrprotect(mm, address, ptep); > +} > + > #ifdef CONFIG_TRANSPARENT_HUGEPAGE > #define __HAVE_ARCH_PMDP_SET_WRPROTECT > static inline void pmdp_set_wrprotect(struct mm_struct *mm, > @@ -1149,6 +1164,8 @@ extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, > unsigned long addr, pte_t *ptep); > extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, > unsigned long addr, pte_t *ptep); > +extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, > + pte_t *ptep, unsigned int nr); > extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, > unsigned long addr, pte_t *ptep, > pte_t entry, int dirty); > @@ -1268,12 +1285,35 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, > return contpte_ptep_clear_flush_young(vma, addr, ptep); > } > > +#define wrprotect_ptes wrprotect_ptes > +static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, > + pte_t *ptep, unsigned int nr) > +{ > + if (likely(nr == 1)) { > + /* > + * Optimization: wrprotect_ptes() can only be called for present > + * ptes so we only need to check contig bit as condition for > + * unfold, and we can remove the contig bit from the pte we read > + * to avoid re-reading. This speeds up fork() which is sensitive > + * for order-0 folios. Equivalent to contpte_try_unfold(). > + */ > + pte_t orig_pte = __ptep_get(ptep); > + > + if (unlikely(pte_cont(orig_pte))) { > + __contpte_try_unfold(mm, addr, ptep, orig_pte); > + orig_pte = pte_mknoncont(orig_pte); > + } > + ___ptep_set_wrprotect(mm, addr, ptep, orig_pte); > + } else { > + contpte_wrprotect_ptes(mm, addr, ptep, nr); > + } > +} > + > #define __HAVE_ARCH_PTEP_SET_WRPROTECT > static inline void ptep_set_wrprotect(struct mm_struct *mm, > unsigned long addr, pte_t *ptep) > { > - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); > - __ptep_set_wrprotect(mm, addr, ptep); > + wrprotect_ptes(mm, addr, ptep, 1); > } > > #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS > @@ -1305,6 +1345,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, > #define ptep_clear_flush_young __ptep_clear_flush_young > #define __HAVE_ARCH_PTEP_SET_WRPROTECT > #define ptep_set_wrprotect __ptep_set_wrprotect > +#define wrprotect_ptes __wrprotect_ptes > #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS > #define ptep_set_access_flags __ptep_set_access_flags > > diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c > index 6d7f40667fa2..bedb58524535 100644 > --- a/arch/arm64/mm/contpte.c > +++ b/arch/arm64/mm/contpte.c > @@ -26,6 +26,26 @@ static inline pte_t *contpte_align_down(pte_t *ptep) > return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); > } > > +static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, > + pte_t *ptep, unsigned int nr) > +{ > + /* > + * Unfold any partially covered contpte block at the beginning and end > + * of the range. > + */ > + > + if (ptep != contpte_align_down(ptep) || nr < CONT_PTES) > + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); > + > + if (ptep + nr != contpte_align_down(ptep + nr)) { > + unsigned long last_addr = addr + PAGE_SIZE * (nr - 1); > + pte_t *last_ptep = ptep + nr - 1; > + > + contpte_try_unfold(mm, last_addr, last_ptep, > + __ptep_get(last_ptep)); > + } > +} > + > static void contpte_convert(struct mm_struct *mm, unsigned long addr, > pte_t *ptep, pte_t pte) > { > @@ -238,6 +258,24 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, > } > EXPORT_SYMBOL(contpte_ptep_clear_flush_young); > > +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, > + pte_t *ptep, unsigned int nr) > +{ > + /* > + * If wrprotecting an entire contig range, we can avoid unfolding. Just > + * set wrprotect and wait for the later mmu_gather flush to invalidate > + * the tlb. Until the flush, the page may or may not be wrprotected. > + * After the flush, it is guaranteed wrprotected. If it's a partial > + * range though, we must unfold, because we can't have a case where > + * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this > + * would cause it to continue to be unpredictable after the flush. > + */ > + > + contpte_try_unfold_partial(mm, addr, ptep, nr); > + __wrprotect_ptes(mm, addr, ptep, nr); > +} > +EXPORT_SYMBOL(contpte_wrprotect_ptes); > + > int contpte_ptep_set_access_flags(struct vm_area_struct *vma, > unsigned long addr, pte_t *ptep, > pte_t entry, int dirty) > -- > 2.25.1 >