From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0739EC072A2 for ; Sun, 19 Nov 2023 16:59:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 968766B038F; Sun, 19 Nov 2023 11:59:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 918996B0395; Sun, 19 Nov 2023 11:59:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E0876B0397; Sun, 19 Nov 2023 11:59:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6ED296B038F for ; Sun, 19 Nov 2023 11:59:27 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4DF74160439 for ; Sun, 19 Nov 2023 16:59:27 +0000 (UTC) X-FDA: 81475314774.16.115E81C Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf12.hostedemail.com (Postfix) with ESMTP id 995C94000B for ; Sun, 19 Nov 2023 16:59:25 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700413165; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=32v9hmaEbNfcXhLF5H8H4hbGt87SQQUXGvgE7D2n01o=; b=Cl+LHGa9E+JlVfdpRHRQayTRgV2nQOVGB+K1vgktDxHj8fdYSaU2dshhbQD45EmRyMBvHZ /O+Zf7wY+ya3dod8xfVUXeDOBVIgSbZCFg6QqiU/r16t1i3p8+/bBEF7BNoLBWAQe7Y4Gu IlIP8kCnatY0b44qRH9hR/P/Ev4+I7M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700413165; a=rsa-sha256; cv=none; b=BpbaMwgOFrvvcbgggIOLTZwkefA2RXzbfEJ/vdH2atuME/Dobpui7V80FpHhmFob9eC0Fk nl6PPVUEPTUuJFkAllw49SofWa1bv2MBwFFS4nYRBQfWGOfkSjpeGzZK/PAhnACrdFN0D6 nijFxAlO1/6sdfzljDSGh0jOfMZmplY= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 182CE1477; Sun, 19 Nov 2023 09:00:11 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D31763F6C4; Sun, 19 Nov 2023 08:59:19 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 20/27] mm: hugepage: Handle huge page fault on access Date: Sun, 19 Nov 2023 16:57:14 +0000 Message-Id: <20231119165721.9849-21-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 995C94000B X-Rspam-User: X-Stat-Signature: 3i9jec3hj7pingdcb39nc5uoif1mjr34 X-Rspamd-Server: rspam03 X-HE-Tag: 1700413165-273045 X-HE-Meta: U2FsdGVkX1+jYv0ijON6cEFX3YtL/vBh3o9dQOw4qor7EwEOwPh314TS6LmULdXePXCLS9p7Pt2/qj3kypNOwDKJMohttgVxGrHYtIH9kEUBKyLxlZHgZNeh/8MtovPhdkDhzK/8PfoU6NF2eKmxGQBP3lCls8wmjVNBC1d8vje0H6eKmE3KsUPxADrEXkbjT4Yxqs6auw5U5l1CCgipS3HpHqv/yEYM4EuyXbsFQx9PJFuRjVr9YEXUVcHAwea3FYDEdV6zjCg03Lx7IFJh/NRElEj3A6gkZp8cHFOzWIRdGniZ8xHcVDbv3TjFQdKfQ7mPPkYRlMGk6gDmRg0htSPNKbT+CiHsndXZW/94Vu8QR6Ph2OOJJXKtXAKh8nczWUOBfSakOlADsmTaydYUshKnYixGalgwJIxR5Evt6LPwoMPvU7WUrQSj/qfmFI9dCxN84TCCd/gP4SM+YltSJEh9dqLOssE/ALr6FzCUpj7lZ8FFILjWSOCHSqHN1xos1QI0ExM9AZLPHGPdUZbH9IUnlv6NRXp2bn3j3lp0pmG7wt8RCWcOFNpYbCH2ltwSa28z5eIsXm2ywaAWBdGCfqtrsYHumNSmU/0TJJJH0T7CT86koYXcBCFoIuTawAfHUSMBBMGo7PZ1bhZlytB4HtOqo/cFx53YmzGi4tyapxesKPq1CuhWQq9cy88xTZwDwRBDrhXVEBww/4JX8YGR/Bw/veQ9Q604CX847IOj1vR/w26DZeb9m2HWUeY/+aUnezALTLgSuZ6FYg289E+G4LkSHwGV0TAk6GUYOPqPutmnZA26n9q9VlWpw2ajjncH4bzbA4Ure71BHP37d8jPrI66W2MjjO3IwHcbyjACeXQycqEx/VWZK63buLGgC77Us040mCb+QsoSem9PvUwTnI8BfZ4kNKvq4CVU2Gw1ea1D3vVeMPvZhwVll+thGhSBmfoV9DpsrGTpfnLuQ6z ckkmtuAk JoIRtsuwzPNz/gRdtjZAtzWZLqjxCLpIffSGpbxrbPaK40hqB/qmJHVyPrWH61VX1SU373B1XSHCNKItIdDOWHuAKoZbk+lBfdDKldlJ1ZXETNa2xXZ8R+DJJ3DdzaEAyuSuHpoiJYjrUzusF1okIZbL7Oqnr/VlUTepo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Handle PAGE_FAULT_ON_ACCESS faults for huge pages in a similar way to regular pages. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/mte_tag_storage.h | 1 + arch/arm64/include/asm/pgtable.h | 7 ++ arch/arm64/mm/fault.c | 81 ++++++++++++++++++++++++ include/linux/huge_mm.h | 2 + include/linux/pgtable.h | 5 ++ mm/huge_memory.c | 4 +- mm/memory.c | 3 + 7 files changed, 101 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index c70ced60a0cd..b97406d369ce 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -35,6 +35,7 @@ void free_tag_storage(struct page *page, int order); bool page_tag_storage_reserved(struct page *page); vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf); +vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf); #else static inline bool tag_storage_enabled(void) { diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 8cc135f1c112..1704411c096d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -477,6 +477,13 @@ static inline vm_fault_t arch_do_page_fault_on_access(struct vm_fault *vmf) return handle_page_missing_tag_storage(vmf); return VM_FAULT_SIGBUS; } + +static inline vm_fault_t arch_do_huge_page_fault_on_access(struct vm_fault *vmf) +{ + if (tag_storage_enabled()) + return handle_huge_page_missing_tag_storage(vmf); + return VM_FAULT_SIGBUS; +} #endif /* CONFIG_ARCH_HAS_FAULT_ON_ACCESS */ #define pmd_present_invalid(pmd) (!!(pmd_val(pmd) & PMD_PRESENT_INVALID)) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f5fa583acf18..6730a0812a24 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -1041,6 +1041,87 @@ vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf) return 0; +out_retry: + put_page(page); + if (vmf->flags & FAULT_FLAG_VMA_LOCK) + vma_end_read(vma); + if (fault_flag_allow_retry_first(vmf->flags)) { + err = VM_FAULT_RETRY; + } else { + /* Replay the fault. */ + err = 0; + } + return err; +} + +vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf) +{ + unsigned long haddr = vmf->address & HPAGE_PMD_MASK; + struct vm_area_struct *vma = vmf->vma; + pmd_t old_pmd, new_pmd; + bool writable = false; + struct page *page; + vm_fault_t err; + int ret; + + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); + if (unlikely(!pmd_same(vmf->orig_pmd, *vmf->pmd))) { + spin_unlock(vmf->ptl); + return 0; + } + + old_pmd = vmf->orig_pmd; + new_pmd = pmd_modify(old_pmd, vma->vm_page_prot); + + /* + * Detect now whether the PMD could be writable; this information + * is only valid while holding the PT lock. + */ + writable = pmd_write(new_pmd); + if (!writable && vma_wants_manual_pte_write_upgrade(vma) && + can_change_pmd_writable(vma, vmf->address, new_pmd)) + writable = true; + + page = vm_normal_page_pmd(vma, haddr, new_pmd); + if (!page) + goto out_map; + + if (!(vma->vm_flags & VM_MTE)) + goto out_map; + + get_page(page); + vma_set_access_pid_bit(vma); + + spin_unlock(vmf->ptl); + writable = false; + + if (unlikely(is_migrate_isolate_page(page))) + goto out_retry; + + ret = reserve_tag_storage(page, HPAGE_PMD_ORDER, GFP_HIGHUSER_MOVABLE); + if (ret) + goto out_retry; + + put_page(page); + + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); + if (unlikely(!pmd_same(old_pmd, *vmf->pmd))) { + spin_unlock(vmf->ptl); + return 0; + } + +out_map: + /* Restore the PMD */ + new_pmd = pmd_modify(old_pmd, vma->vm_page_prot); + new_pmd = pmd_mkyoung(new_pmd); + if (writable) + new_pmd = pmd_mkwrite(new_pmd, vma); + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, new_pmd); + update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); + spin_unlock(vmf->ptl); + + return 0; + out_retry: put_page(page); if (vmf->flags & FAULT_FLAG_VMA_LOCK) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index fa0350b0812a..bb84291f9231 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -36,6 +36,8 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, pgprot_t newprot, unsigned long cp_flags); +bool can_change_pmd_writable(struct vm_area_struct *vma, unsigned long addr, + pmd_t pmd); vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write); vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e2c761dd6c41..de45f475bf8d 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1473,6 +1473,11 @@ static inline vm_fault_t arch_do_page_fault_on_access(struct vm_fault *vmf) { return VM_FAULT_SIGBUS; } + +static inline vm_fault_t arch_do_huge_page_fault_on_access(struct vm_fault *vmf) +{ + return VM_FAULT_SIGBUS; +} #endif #endif /* CONFIG_MMU */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9beead961a65..d1402b43ea39 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1406,8 +1406,8 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) return VM_FAULT_FALLBACK; } -static inline bool can_change_pmd_writable(struct vm_area_struct *vma, - unsigned long addr, pmd_t pmd) +inline bool can_change_pmd_writable(struct vm_area_struct *vma, + unsigned long addr, pmd_t pmd) { struct page *page; diff --git a/mm/memory.c b/mm/memory.c index a04a971200b9..46b926625503 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5168,6 +5168,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, return 0; } if (pmd_trans_huge(vmf.orig_pmd) || pmd_devmap(vmf.orig_pmd)) { + if (fault_on_access_pmd(vmf.orig_pmd) && vma_is_accessible(vma)) + return arch_do_huge_page_fault_on_access(&vmf); + if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma)) return do_huge_pmd_numa_page(&vmf); -- 2.42.1