From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1793FC4332F for ; Tue, 12 Dec 2023 11:59:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A20706B02B6; Tue, 12 Dec 2023 06:59:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D0586B02B7; Tue, 12 Dec 2023 06:59:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 897626B02B8; Tue, 12 Dec 2023 06:59:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 76D7B6B02B6 for ; Tue, 12 Dec 2023 06:59:54 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 36F55120926 for ; Tue, 12 Dec 2023 11:59:54 +0000 (UTC) X-FDA: 81558022308.04.A79A954 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf25.hostedemail.com (Postfix) with ESMTP id 6A621A000C for ; Tue, 12 Dec 2023 11:59:52 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702382392; a=rsa-sha256; cv=none; b=WbzynSnGhRv6mvBnSLA2X2GVzbbReXKuWqBvdHImiFAXqFRmQ5u4Ej99vYLnDSimPxL72O kQjEjJQfGVT+ngU1r5yIO6FzmDMr8nSgXvjFgamyXxFf2u0MwdurTzq5sYktteppzURLwg P42tQ1KoWovOP5tTJOWHrm6wlu2+kfk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702382392; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/7jA2vD71DJ4qis/6PulCo8pvVKHpheYJKJinlPkXqw=; b=Y7e3yJDA3ueCAeo6IGBCITmhU5mNXk8jxEGG5z4SHsvzhVWvgwJFhrfUchFsmjfTK/+ZMt aoEvY2NoJBWWpRrTdqIvHqinpYkl/ke8C9qww9gfnBmHfaMSLXlA0fVVnwn7mjJOkth5za 77h/+WEgKuPKxejKUb6ASno+fUoN0ss= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7C688143D; Tue, 12 Dec 2023 04:00:34 -0800 (PST) Received: from [10.1.39.183] (XHFQ2J9959.cambridge.arm.com [10.1.39.183]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C02CE3F762; Tue, 12 Dec 2023 03:59:44 -0800 (PST) Message-ID: Date: Tue, 12 Dec 2023 11:59:43 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 14/15] arm64/mm: Implement ptep_set_wrprotects() to optimize fork() Content-Language: en-GB To: Alistair Popple Cc: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Yang Shi , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20231204105440.61448-1-ryan.roberts@arm.com> <20231204105440.61448-15-ryan.roberts@arm.com> <87cyvha2xd.fsf@nvdebian.thelocal> From: Ryan Roberts In-Reply-To: <87cyvha2xd.fsf@nvdebian.thelocal> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6A621A000C X-Stat-Signature: depk3z45ndg7cmu7nwendtnntixcmxd1 X-HE-Tag: 1702382392-236809 X-HE-Meta: U2FsdGVkX19tNlSmjlf9F0kiLVmsxC+FwzjplYOZYJ1umcLOkZdjYateXAWqI9lJN0MAn0ZbD3/6OqN8u5XYWFU2wxgXNuPMbJJ2h1eIe1YUAwAnT65MGILvqAzIiP06wAjZOZQtKOqrqcqkMciyAsxs/nKFh92grc7WwhfI1qDbANqLGU9r11lupqUqcBRYvdJodpehD79CYvoZEzYVf6TjzawqiV/s/r6vbZdgLwpqLtsg6kGtFAgWOKAVysLO6Vs0WDm9pzpV+i84smxt19nXQmuwuXytZRHL/RnjqcTbJWdQtuw6kxnVOCBiFws+IQxDv/PXzxwYx7nhsfXJkuLHxjh0EFAheIT2zM7iRbcsxm+1lDqJBxJmjAFVkpyHUQX3E2w8pPpkMX59DEKbXAHCULNzQs3LofAKjUvhEvmEWbAgb+aqStvSHZQWOoeXDA+wZ6xa7nRUzYuLkuOnqZ/Kpv3N2us0ne0ElqL2wMyC/1sG35fxEHld/trgrtIWvBgwGr5EJgtw0ZhKEcF5f+BCgWrCUNFeg8RfJ3LKoOs5+9jyqfv5Ck3LvQPW/5qgiOTZtaD2M0Uaj8o4R+w2m9OmizXCrTN+DHcFy9KbwIjwbTyiBhQUR/0X7arUuhswFYLTAImN9qt2o75LcsgEfOZYf/NXMtj/IFbDouiGqrAh2icFrURlLuyi5V8ZNmV/a9+ta5Ckk7gZcZ4tzhjRWgOjxhVRqXkt8xJPCbidGKX0idwdEdErNWgNaKyO03XOPKH0KKYFVRsSNbEdvM0A69SsG7JjXTLhuF4BYGLEciIkWx5+znHdM8j9JPxR6ZabvFMIU53TaQaRfTJ/ZhgABs7cs9eNj0LWY4cnApWCuP8/REU0VQaVR3PAf4uiog8KThseqo7b9l7gzMQWsl1cCfUhSgYFq6E5THbGwZoD04oRHz7Kc3Or+aVftTx/iWDUru5urBxjVuCeUCGwjKh NWrC9leZ UbhhNq7TTklZwidzDW5vHmULbyx5StpFUvJsA6cGR3ISRKqX2cW6BAvwBozpfwcd0oUFp7u0NENnVb3O8V7HG9EBemEhGQbsC/ivEnjxR/X6XxeKxUHCohBleGE2LknabaFSGDsxL37xqHWitrISFAFZ5L3K4B5mRrjt5K5X44uwDSF3fyYa1q1A7i5UEpDmZxYqdBU7vbiWIL3b60l3S/CFjZfSbqeEryj4chI65I8Hp/K+9nSEksXRLdDG580f5HxeGJSldVEyOKDKpz8Gy/2WfanpsZ4Tn6VbtFnGvE3Gu0OwEDGsNq9zOHnuMz0/zYDfP61bYBjIsO7s= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 08/12/2023 01:37, Alistair Popple wrote: > > Ryan Roberts writes: > >> With the core-mm changes in place to batch-copy ptes during fork, we can >> take advantage of this in arm64 to greatly reduce the number of tlbis we >> have to issue, and recover the lost fork performance incured when adding >> support for transparent contiguous ptes. >> >> If we are write-protecting a whole contig range, we can apply the >> write-protection to the whole range and know that it won't change >> whether the range should have the contiguous bit set or not. For ranges >> smaller than the contig range, we will still have to unfold, apply the >> write-protection, then fold if the change now means the range is >> foldable. >> >> This optimization is possible thanks to the tightening of the Arm ARM in >> respect to the definition and behaviour when 'Misprogramming the >> Contiguous bit'. See section D21194 at >> https://developer.arm.com/documentation/102105/latest/ >> >> Performance tested with the following test written for the will-it-scale >> framework: >> >> ------- >> >> char *testcase_description = "fork and exit"; >> >> void testcase(unsigned long long *iterations, unsigned long nr) >> { >> int pid; >> char *mem; >> >> mem = malloc(SZ_128M); >> assert(mem); >> memset(mem, 1, SZ_128M); >> >> while (1) { >> pid = fork(); >> assert(pid >= 0); >> >> if (!pid) >> exit(0); >> >> waitpid(pid, NULL, 0); >> >> (*iterations)++; >> } >> } >> >> ------- >> >> I see huge performance regression when PTE_CONT support was added, then >> the regression is mostly fixed with the addition of this change. The >> following shows regression relative to before PTE_CONT was enabled >> (bigger negative value is bigger regression): >> >> | cpus | before opt | after opt | >> |-------:|-------------:|------------:| >> | 1 | -10.4% | -5.2% | >> | 8 | -15.4% | -3.5% | >> | 16 | -38.7% | -3.7% | >> | 24 | -57.0% | -4.4% | >> | 32 | -65.8% | -5.4% | >> >> Signed-off-by: Ryan Roberts >> --- >> arch/arm64/include/asm/pgtable.h | 30 ++++++++++++++++++++--- >> arch/arm64/mm/contpte.c | 42 ++++++++++++++++++++++++++++++++ >> 2 files changed, 69 insertions(+), 3 deletions(-) >> >> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h >> index 15bc9cf1eef4..9bd2f57a9e11 100644 >> --- a/arch/arm64/include/asm/pgtable.h >> +++ b/arch/arm64/include/asm/pgtable.h >> @@ -984,6 +984,16 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, >> } while (pte_val(pte) != pte_val(old_pte)); >> } >> >> +static inline void __ptep_set_wrprotects(struct mm_struct *mm, >> + unsigned long address, pte_t *ptep, >> + unsigned int nr) >> +{ >> + unsigned int i; >> + >> + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) >> + __ptep_set_wrprotect(mm, address, ptep); >> +} >> + >> #ifdef CONFIG_TRANSPARENT_HUGEPAGE >> #define __HAVE_ARCH_PMDP_SET_WRPROTECT >> static inline void pmdp_set_wrprotect(struct mm_struct *mm, >> @@ -1139,6 +1149,8 @@ extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, >> unsigned long addr, pte_t *ptep); >> extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, >> unsigned long addr, pte_t *ptep); >> +extern void contpte_set_wrprotects(struct mm_struct *mm, unsigned long addr, >> + pte_t *ptep, unsigned int nr); >> extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, >> unsigned long addr, pte_t *ptep, >> pte_t entry, int dirty); >> @@ -1290,13 +1302,25 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, >> return contpte_ptep_clear_flush_young(vma, addr, ptep); >> } >> >> +#define ptep_set_wrprotects ptep_set_wrprotects >> +static inline void ptep_set_wrprotects(struct mm_struct *mm, unsigned long addr, >> + pte_t *ptep, unsigned int nr) >> +{ >> + if (!contpte_is_enabled(mm)) >> + __ptep_set_wrprotects(mm, addr, ptep, nr); >> + else if (nr == 1) { > > Why do we need the special case here? Couldn't we just call > contpte_set_wrprotects() with nr == 1? My intention is for this to be a fast path for ptep_set_wrprotect(). I'm having to work hard to prevent regressing the order-0 folios case. > >> + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); >> + __ptep_set_wrprotects(mm, addr, ptep, 1); >> + contpte_try_fold(mm, addr, ptep, __ptep_get(ptep)); >> + } else >> + contpte_set_wrprotects(mm, addr, ptep, nr); >> +} >> + >> #define __HAVE_ARCH_PTEP_SET_WRPROTECT >> static inline void ptep_set_wrprotect(struct mm_struct *mm, >> unsigned long addr, pte_t *ptep) >> { >> - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); >> - __ptep_set_wrprotect(mm, addr, ptep); >> - contpte_try_fold(mm, addr, ptep, __ptep_get(ptep)); >> + ptep_set_wrprotects(mm, addr, ptep, 1); >> } >> >> #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS >> diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c >> index e079ec61d7d1..2a57df16bf58 100644 >> --- a/arch/arm64/mm/contpte.c >> +++ b/arch/arm64/mm/contpte.c >> @@ -303,6 +303,48 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, >> } >> EXPORT_SYMBOL(contpte_ptep_clear_flush_young); >> >> +void contpte_set_wrprotects(struct mm_struct *mm, unsigned long addr, >> + pte_t *ptep, unsigned int nr) >> +{ >> + unsigned long next; >> + unsigned long end = addr + (nr << PAGE_SHIFT); >> + >> + do { >> + next = pte_cont_addr_end(addr, end); >> + nr = (next - addr) >> PAGE_SHIFT; >> + >> + /* >> + * If wrprotecting an entire contig range, we can avoid >> + * unfolding. Just set wrprotect and wait for the later >> + * mmu_gather flush to invalidate the tlb. Until the flush, the >> + * page may or may not be wrprotected. After the flush, it is >> + * guarranteed wrprotected. If its a partial range though, we >> + * must unfold, because we can't have a case where CONT_PTE is >> + * set but wrprotect applies to a subset of the PTEs; this would >> + * cause it to continue to be unpredictable after the flush. >> + */ >> + if (nr != CONT_PTES) >> + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); >> + >> + __ptep_set_wrprotects(mm, addr, ptep, nr); >> + >> + addr = next; >> + ptep += nr; >> + >> + /* >> + * If applying to a partial contig range, the change could have >> + * made the range foldable. Use the last pte in the range we >> + * just set for comparison, since contpte_try_fold() only >> + * triggers when acting on the last pte in the contig range. >> + */ >> + if (nr != CONT_PTES) >> + contpte_try_fold(mm, addr - PAGE_SIZE, ptep - 1, >> + __ptep_get(ptep - 1)); >> + >> + } while (addr != end); >> +} >> +EXPORT_SYMBOL(contpte_set_wrprotects); >> + >> int contpte_ptep_set_access_flags(struct vm_area_struct *vma, >> unsigned long addr, pte_t *ptep, >> pte_t entry, int dirty) >