From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CB9BC36010 for ; Fri, 4 Apr 2025 03:50:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2326E6B0008; Thu, 3 Apr 2025 23:50:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BBC16B000A; Thu, 3 Apr 2025 23:50:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F29EA6B000C; Thu, 3 Apr 2025 23:50:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D03456B0008 for ; Thu, 3 Apr 2025 23:50:48 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D8C815924D for ; Fri, 4 Apr 2025 03:50:48 +0000 (UTC) X-FDA: 83294984976.07.1B34E7B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf03.hostedemail.com (Postfix) with ESMTP id E809820009 for ; Fri, 4 Apr 2025 03:50:46 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf03.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1743738647; a=rsa-sha256; cv=none; b=Rmfs/XFBlgq3/airtHU5eyl0wMGRVG0Vez+my8wJM/9/kPXV931zuyVe/80tgQRaTAGJly 3C5QYFljUajfQ04j2Qew+kaMFv6tizTTca1oYRlrUEiI0s4GgOVII/Mo+oCDb1QMRpIs2Q /b77DF497psUgXBtFLL1gtIda128xPs= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf03.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1743738647; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hbQX6Ypw2dmGdM9e6YkgLrIE+DUQlsu7fY3rshtNkcc=; b=C+/K1ufn2D9+wJb2K2JhZExG8vNBCRvqWgkXiTHgExuNpnQ+pG2j3WAwu4tYWmEu3jZIgq KXbVFuA6R7J9WqThGXcbt131UIhI5CvbYnGOYiPyMs5TbjA1xyk/7XtE7oX2NurjBxTTmV EFSQPMZ+5IpZgLoRVQr5bDZyzm5muRo= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5DE041516; Thu, 3 Apr 2025 20:50:48 -0700 (PDT) Received: from [10.162.40.17] (a077893.blr.arm.com [10.162.40.17]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 10FBC3F694; Thu, 3 Apr 2025 20:50:40 -0700 (PDT) Message-ID: <6e4e11ef-d0d2-48e8-9ecd-4ba6575c8e52@arm.com> Date: Fri, 4 Apr 2025 09:20:37 +0530 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 02/11] arm64: hugetlb: Refine tlb maintenance scope To: Ryan Roberts , Catalin Marinas , Will Deacon , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , David Hildenbrand , "Matthew Wilcox (Oracle)" , Mark Rutland , Alexandre Ghiti , Kevin Brodsky Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20250304150444.3788920-1-ryan.roberts@arm.com> <20250304150444.3788920-3-ryan.roberts@arm.com> Content-Language: en-US From: Anshuman Khandual In-Reply-To: <20250304150444.3788920-3-ryan.roberts@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E809820009 X-Stat-Signature: yygro7ci76qtny7dh81xnorwk5fmdyi1 X-Rspam-User: X-HE-Tag: 1743738646-947115 X-HE-Meta: U2FsdGVkX1+x+D4WViaDKfXLmkIDLcjPyOVed3kt8B8ikEErH0i2nVsmeFwLhiBvfTRqTN/vt1Xyf+PeYXlAMAXjua21JZDFGQHIq3PbHPXcLfRoIkG63Gj25kSE3qgmhv7GKK2dO31ATtZP4QTHPZUeAWKkE+48zXPkSQGe0p3bnyD/k2JQF4AveU8t0YvucPP5u+27xoDVbSA0npbDtdld1yTSlEHiSfnZcwA6mVTmCsOAed1n6Aa7hOYdxLC5AtLduvF70dE77n2xpBwXThpF5JRZ9lkzx2ANpeLt9Tc0/U16Yi+72xRdpd2aael5//mzkTAIsfP4Mvc7c/KZACnziKhvtYU1TRTBMA8/1HdC8tb3gFxny0AzaoOh4tQJQ82QMa0thaepChHInPw9tTzed26vri+RhNQ8QoEKEdWn4FLtKjTBkP9wpJyU8jJyPbCpNDqzb4rQ2/e15OLfWePQPnRxonf3Vy7D4ymgSQrmne4d1ckLcpHJQoTPbPsuATjbxHM0IUQfIZPaF+VBgcUJ9ADLy4dg+xffZ0KixVjPW4NqrWF1gOtzvD2g5cpk7pOj5UrhIySyjt15U+FttlRhxGhn/IBGgC8qAVt/uI/SUdFz9eAwF1a9aftV7gXtB1r2QoqofilMabirfMaLP6VuwJwYTY+kA5MEzmJQiCMRkwS1HTdyqQYecpi9p80w16TkbmGkNm+rnd/Yj7+H922NBvQXxiNTFF8DqqaeK2e209MlTb/1boc/72O5i8gE56+0zriwZNWBWhWblKlg5B6amw1kNMHZdifmL3ph/J6gPhO7zHDdlQJVwuU8pXF4SYN9TeLH8p9h/+jzX8UpPVpsmwDqWkOZrykqS8zodzCRRnWghBkRmD4516jX7SVpDsLLlnOTx78oPxeljOhRluMM7J5dkOszCqws66jdTdrhn3CeDZ7MG+NK0DyhKHlJJjmrSiP/+xbaay+9i3C 1+6+OcGT ha4UGmhP8VL+Nk0rdQh4acMZZRpW9KvKqb8gaN+bBD6zYPjX9pqkcOr63Qr5nL8Z8Kh9vVWnZCajfRjJIUoyHytSkBHRPM22tq9EaMrpsqnzpc2kb5RLht+0ZP0YRwKrNXvaOdnJ0915/CjuGIpMbM4UIZ2AO61izzF8I6e13LxiutLVz4zzgdTZroquqNyXJglmHjoaRovrWjcTHLfkotzxYqxOytDd8zssBgeODm2fMahSID0BkErZq7g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 3/4/25 20:34, Ryan Roberts wrote: > When operating on contiguous blocks of ptes (or pmds) for some hugetlb > sizes, we must honour break-before-make requirements and clear down the > block to invalid state in the pgtable then invalidate the relevant tlb > entries before making the pgtable entries valid again. > > However, the tlb maintenance is currently always done assuming the worst > case stride (PAGE_SIZE), last_level (false) and tlb_level > (TLBI_TTL_UNKNOWN). We can do much better with the hinting; In reality, > we know the stride from the huge_pte pgsize, we are always operating > only on the last level, and we always know the tlb_level, again based on > pgsize. So let's start providing these hints. > > Additionally, avoid tlb maintenace in set_huge_pte_at(). > Break-before-make is only required if we are transitioning the > contiguous pte block from valid -> valid. So let's elide the > clear-and-flush ("break") if the pte range was previously invalid. > > Signed-off-by: Ryan Roberts > --- > arch/arm64/include/asm/hugetlb.h | 29 +++++++++++++++++++---------- > arch/arm64/mm/hugetlbpage.c | 9 ++++++--- > 2 files changed, 25 insertions(+), 13 deletions(-) > > diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h > index 07fbf5bf85a7..2a8155c4a882 100644 > --- a/arch/arm64/include/asm/hugetlb.h > +++ b/arch/arm64/include/asm/hugetlb.h > @@ -69,29 +69,38 @@ extern void huge_ptep_modify_prot_commit(struct vm_area_struct *vma, > > #include > > -#define __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE > -static inline void flush_hugetlb_tlb_range(struct vm_area_struct *vma, > - unsigned long start, > - unsigned long end) > +static inline void __flush_hugetlb_tlb_range(struct vm_area_struct *vma, > + unsigned long start, > + unsigned long end, > + unsigned long stride, > + bool last_level) > { > - unsigned long stride = huge_page_size(hstate_vma(vma)); > - > switch (stride) { > #ifndef __PAGETABLE_PMD_FOLDED > case PUD_SIZE: > - __flush_tlb_range(vma, start, end, PUD_SIZE, false, 1); > + __flush_tlb_range(vma, start, end, PUD_SIZE, last_level, 1); > break; > #endif > case CONT_PMD_SIZE: > case PMD_SIZE: > - __flush_tlb_range(vma, start, end, PMD_SIZE, false, 2); > + __flush_tlb_range(vma, start, end, PMD_SIZE, last_level, 2); > break; > case CONT_PTE_SIZE: > - __flush_tlb_range(vma, start, end, PAGE_SIZE, false, 3); > + __flush_tlb_range(vma, start, end, PAGE_SIZE, last_level, 3); > break; > default: > - __flush_tlb_range(vma, start, end, PAGE_SIZE, false, TLBI_TTL_UNKNOWN); > + __flush_tlb_range(vma, start, end, PAGE_SIZE, last_level, TLBI_TTL_UNKNOWN); > } > } > > +#define __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE > +static inline void flush_hugetlb_tlb_range(struct vm_area_struct *vma, > + unsigned long start, > + unsigned long end) > +{ > + unsigned long stride = huge_page_size(hstate_vma(vma)); > + > + __flush_hugetlb_tlb_range(vma, start, end, stride, false); > +} > + > #endif /* __ASM_HUGETLB_H */ > diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c > index 6a2af9fb2566..065be8650aa5 100644 > --- a/arch/arm64/mm/hugetlbpage.c > +++ b/arch/arm64/mm/hugetlbpage.c > @@ -183,8 +183,9 @@ static pte_t get_clear_contig_flush(struct mm_struct *mm, > { > pte_t orig_pte = get_clear_contig(mm, addr, ptep, pgsize, ncontig); > struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); > + unsigned long end = addr + (pgsize * ncontig); > > - flush_tlb_range(&vma, addr, addr + (pgsize * ncontig)); > + __flush_hugetlb_tlb_range(&vma, addr, end, pgsize, true); > return orig_pte; > } > > @@ -209,7 +210,7 @@ static void clear_flush(struct mm_struct *mm, > for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) > __ptep_get_and_clear(mm, addr, ptep); > > - flush_tlb_range(&vma, saddr, addr); > + __flush_hugetlb_tlb_range(&vma, saddr, addr, pgsize, true); > } > > void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, > @@ -238,7 +239,9 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, > dpfn = pgsize >> PAGE_SHIFT; > hugeprot = pte_pgprot(pte); > > - clear_flush(mm, addr, ptep, pgsize, ncontig); > + /* Only need to "break" if transitioning valid -> valid. */ > + if (pte_valid(__ptep_get(ptep))) > + clear_flush(mm, addr, ptep, pgsize, ncontig); > > for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) > __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); LGTM Reviewed-by: Anshuman Khandual