From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B24CC369CB for ; Tue, 22 Apr 2025 08:19:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 89D496B0088; Tue, 22 Apr 2025 04:19:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 81FDC6B0089; Tue, 22 Apr 2025 04:19:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E76E6B008A; Tue, 22 Apr 2025 04:19:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 510196B0088 for ; Tue, 22 Apr 2025 04:19:01 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8F99F5FDC7 for ; Tue, 22 Apr 2025 08:19:02 +0000 (UTC) X-FDA: 83360979324.27.37C8380 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf03.hostedemail.com (Postfix) with ESMTP id E80F320006 for ; Tue, 22 Apr 2025 08:19:00 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf03.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745309941; a=rsa-sha256; cv=none; b=uCMzHll4bfIlmcDc9lucpKCydyDjD5K7eEyLB51/iy/Dh1YCzpDUC2jx7WBY22yUteBCd5 n8xtLGXAnUkE3LJs7Ine4aPeGmwhk3x08v+eRZMozp6LMGZlidzdi3LKG1BAyiHzGHshT4 Fw24R8wDYjl9+CkRS+MaCoEatl0f/mc= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf03.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745309941; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iDD/NagJFqFOcFy7661wD3VBPTljIbzoZMhUEK+uKA0=; b=Z94ImoaOy4Jl0xQsDwq7uZPJO/YV/xvxwwylcc+hFs1JVRKwdljt3dVIDoAFZEk4ILvWp1 ZOCtIF6rBRPLs56jTjPXRaQ57Y/IjgLqMRd4EwYDI7iSuXTFESPCVkb9TMotjRYrUl21XU hmfuQbUau1anRpG22vPJBcsTkq26p90= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 18458176A; Tue, 22 Apr 2025 01:18:56 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 64C2F3F66E; Tue, 22 Apr 2025 01:18:58 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , David Hildenbrand , "Matthew Wilcox (Oracle)" , Mark Rutland , Anshuman Khandual , Alexandre Ghiti , Kevin Brodsky Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 09/11] arm64/mm: Support huge pte-mapped pages in vmap Date: Tue, 22 Apr 2025 09:18:17 +0100 Message-ID: <20250422081822.1836315-10-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250422081822.1836315-1-ryan.roberts@arm.com> References: <20250422081822.1836315-1-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: E80F320006 X-Stat-Signature: u17bypn1aht8w161ipnpyoowphjjwirp X-HE-Tag: 1745309940-912678 X-HE-Meta: U2FsdGVkX19/rkde6L2EQ1wcr4MsydMSSo+flpdr3CIqm9eBuE85XQelFByUUAXBV9lIY3Pl8fihEZ1xqrwxY8MVV3IwA63RIBQbdcZIpqdPCcXmoucZ7CJgpVQT5T+lRWvseI8go/wjRN8aaPgVXc5rxn8u//nfUdLgGI5bBjCxI01QSay4HUmdwq7mVZn8m/SIuB7q3h219G278dKReo1y4dVJyvd6l97pEYdb8GHcfHXBxHVzi4GylnVUMV72gsFxXZSQKbpCewmsXpF+XxgHAYt/0+e0Fr9hf9wxsezwQ5ye5vm9Pxl+V9suGsI2EDCzg22KbMsqZvsrSPTHkyj5Ht6bQf+FNiiOz2+VbLlG82J0E3de0eLZzQpaEvesX9nXTgs0RkKov7y1KeCPute0saaiWegWQlM8Z8+1db+ECWpw47XW208F2haNTeJKN4qo6DsQeJ2+Rihn6oBugpzVJT41VCvMiX0SHl9xP5p3vnQF7GRiljNxtYL4708p5cdc8b3zCJoP43zIf/0Mx/ABzoKzZF4sNAJot4fgzUFLxZYjDjac0d7b5VZgmj1FDYMvLImHZB47N/zLdc2xcjJbiDbzilL3jgAkyn37nbDkFV/VE2EkcXQW2r2dWQ7fD1s6imyeyjULMuz+h706WxwhZ3dSiKuAAYbiOqP2zRyOZK96jv/Gp0HJdhpJ0uZzlKvQu3oFAt9+FVqaPJS14t8B2V1LDZAB8LPadS6cJSkt2+lcqdmlVYCmPIafanvnjFmK86udLyubuLIL/z5UD3jonxR6CxF3v14b39DpumnnWrt0A/26LAV3V/9DHIRz0uw7baXPgT3meg2DCKooNrecakeYiBPKOY36Bw1naNUglY1xPbkCXk7P+gfuwHcqIDjXVivUJ08Xe2RAuL+AX5bNATpLnG/FnXilCF2NjMZg2KMqT+nE/HyfMnzCdpMzIJNniGgKpymsVdJ7vCx y3vKOA0z aWw6HjVg6NdC4BzyfVA01cOb0aSF2VWYhALycofy6mFco4EeJOXczB3lXvXmSm0TyVaIpD4BTcn0ne2heSvgqMGEo1nUCyTtnUCWNRX9V1QU2+PXZY7lwy6Gr4fQvdRl9xMmFFbPcQrb/bunmAWNRxnG9Ww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement the required arch functions to enable use of contpte in the vmap when VM_ALLOW_HUGE_VMAP is specified. This speeds up vmap operations due to only having to issue a DSB and ISB per contpte block instead of per pte. But it also means that the TLB pressure reduces due to only needing a single TLB entry for the whole contpte block. Since vmap uses set_huge_pte_at() to set the contpte, that API is now used for kernel mappings for the first time. Although in the vmap case we never expect it to be called to modify a valid mapping so clear_flush() should never be called, it's still wise to make it robust for the kernel case, so amend the tlb flush function if the mm is for kernel space. Tested with vmalloc performance selftests: # kself/mm/test_vmalloc.sh \ run_test_mask=1 test_repeat_count=5 nr_pages=256 test_loop_count=100000 use_huge=1 Duration reduced from 1274243 usec to 1083553 usec on Apple M2 for 15% reduction in time taken. Reviewed-by: Anshuman Khandual Reviewed-by: Catalin Marinas Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/vmalloc.h | 45 ++++++++++++++++++++++++++++++++ arch/arm64/mm/hugetlbpage.c | 5 +++- 2 files changed, 49 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h index 38fafffe699f..12f534e8f3ed 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -23,6 +23,51 @@ static inline bool arch_vmap_pmd_supported(pgprot_t prot) return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } +#define arch_vmap_pte_range_map_size arch_vmap_pte_range_map_size +static inline unsigned long arch_vmap_pte_range_map_size(unsigned long addr, + unsigned long end, u64 pfn, + unsigned int max_page_shift) +{ + /* + * If the block is at least CONT_PTE_SIZE in size, and is naturally + * aligned in both virtual and physical space, then we can pte-map the + * block using the PTE_CONT bit for more efficient use of the TLB. + */ + if (max_page_shift < CONT_PTE_SHIFT) + return PAGE_SIZE; + + if (end - addr < CONT_PTE_SIZE) + return PAGE_SIZE; + + if (!IS_ALIGNED(addr, CONT_PTE_SIZE)) + return PAGE_SIZE; + + if (!IS_ALIGNED(PFN_PHYS(pfn), CONT_PTE_SIZE)) + return PAGE_SIZE; + + return CONT_PTE_SIZE; +} + +#define arch_vmap_pte_range_unmap_size arch_vmap_pte_range_unmap_size +static inline unsigned long arch_vmap_pte_range_unmap_size(unsigned long addr, + pte_t *ptep) +{ + /* + * The caller handles alignment so it's sufficient just to check + * PTE_CONT. + */ + return pte_valid_cont(__ptep_get(ptep)) ? CONT_PTE_SIZE : PAGE_SIZE; +} + +#define arch_vmap_pte_supported_shift arch_vmap_pte_supported_shift +static inline int arch_vmap_pte_supported_shift(unsigned long size) +{ + if (size >= CONT_PTE_SIZE) + return CONT_PTE_SHIFT; + + return PAGE_SHIFT; +} + #endif #define arch_vmap_pgprot_tagged arch_vmap_pgprot_tagged diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index d34703846ef4..0c8737f4f2ce 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -209,7 +209,10 @@ static void clear_flush(struct mm_struct *mm, for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) __ptep_get_and_clear_anysz(mm, ptep, pgsize); - __flush_hugetlb_tlb_range(&vma, saddr, addr, pgsize, true); + if (mm == &init_mm) + flush_tlb_kernel_range(saddr, addr); + else + __flush_hugetlb_tlb_range(&vma, saddr, addr, pgsize, true); } void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, -- 2.43.0