From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 721B5C072A2 for ; Sun, 19 Nov 2023 16:59:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 090E86B0399; Sun, 19 Nov 2023 11:59:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 03F5F6B039E; Sun, 19 Nov 2023 11:59:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E497C6B039F; Sun, 19 Nov 2023 11:59:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CF9326B0399 for ; Sun, 19 Nov 2023 11:59:37 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id ACF7E1CA748 for ; Sun, 19 Nov 2023 16:59:37 +0000 (UTC) X-FDA: 81475315194.06.008E690 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf03.hostedemail.com (Postfix) with ESMTP id 12A8E20003 for ; Sun, 19 Nov 2023 16:59:35 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf03.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700413176; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iK9Uf1jMUUazqXSY67y98N6NjP4gpcfMkd425wXO8kw=; b=FtatSOUqdWfAii/Z0hzwPzgSBKBdL+y2QhWJ36PHm71ItIkS9gneQtDwrZzvfPeXd4eS9C 83hr5Dy9hIrEKCS0/EUBvKF0LIrbbpFb3QFb6WtJASNrRPfmoMW6aUOUJk0qcL4ORqMKa3 yzHSNT/PzeVkS4MTxMUFryCTjWU/vfA= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf03.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700413176; a=rsa-sha256; cv=none; b=CBMl7KaazUo/I2Mg+G63uvTqsJdrGNM1byWW8mFgGC01IE1nPuuVbZNoiHBNS41o+apUBS y8+MO2WUukjmoQDZjaLGeGLYkSpTCdtLG5ym9YdBo91pqm0IUp/Oo/FwqG/NEc6r7nl0ck idLKvuuZCRMmYq9iVJJTp6ZLbu/TZuc= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A8837FEC; Sun, 19 Nov 2023 09:00:21 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 77E993F6C4; Sun, 19 Nov 2023 08:59:30 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 22/27] arm64: mte: swap: Handle tag restoring when missing tag storage Date: Sun, 19 Nov 2023 16:57:16 +0000 Message-Id: <20231119165721.9849-23-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 12A8E20003 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: pnk5uf6wzkqs51ggq77t4qtg4cxa1gh7 X-HE-Tag: 1700413175-322336 X-HE-Meta: U2FsdGVkX18ePERiJFq1mBkn0Tc+J9JdvkzgzCc4P+seKAacZKc/tQSBWfFf5HH0hQGqLQO90ypQU5ZK+aGJ1o3H6d/cTo0oSjnLKPKd3UKNru/WkPA4jukaO2rMFBzFen14Y8cznrTE11Jvk14lSP5hV76CmlkCdU3UyfWCHHBVsPnwQx5EQINV4h82xh60YFvI59NutYqpbiv8qh51GcwksIAFqjCikrHZqBxDI8HGVJh4ecyUwdKIj3WTCWwNV0cg/65yFXgtQ0M/juGIId5fdxosLJU2KFUZDbjN8P06GkGN9HoN7Mu1vLqQ2dxxRBRxW8FWOrQ+0Fh5hIWnXUmuY7UlzMclVxIMwq2ZWXjqA45mXt0wSfhQQBH646PzDcDoPVwsCY/XeGh4ZLTsWEzdG92QpM1au8R947+45PPqyZczb2k6VmbXdkaeW1ks+GgetBwGCx3lWQ6fv6BBchlvdCJpLMyLxS8+UytGotcngw6T5kW2w+4CxQaJwdfgwCIOXlOILdsqx3evAh9DwAWu2VOd7LULHjCU1O45wtftQnSb/mj+hwZMhm2DDUao0gXqVfMyhoZUkk/uPPZoCA/fLXcD2hYULEzkUua/5vst63f3odBTtve0+BoIHrLJAM4sUSgn0NBVZPq0mLMZB8+RcCSKtYV7q1rivPKhhvtD5+VOkJ+xdKBS+mK58vNbFKlNob4IxGAW5a9pTUTehiP9CjL98wkQ8scMop8N3ewRgnKsr/pvAfwHB07QxneWGQB3WMzrhtI/7zbqHT9LpBhVRnQrKfeY314gDj8Z77s41xHj9B/SgzyY4o8jnRx53fPnOTIrrox7hWFlEnBCsYJu2e7CmZCBM8lhEAIMggmn+XEkFQBiYGaYSFjEbj/0cKNuFHeRg9c2tdVW5Bfvr0npmZv5dl6mnA7Q3pkWRL7Fk03zbHJHT+HOdUywIg/Qqo93nf4LydXAhhID+y1 ZFY5sD6s i0ya930ylfEEvFU7OVbkc9aV/TE07I5zhp6ufRrAVjm+civib1lPGaAasaEevMnbzp3FIaV7T2NhmDBzo6D0jrY4a09lOIcMC1B3YaMpFCUfjS232ivpQSJioBn6Mn55u8ZK0Ew6ABnNd099BdF5aAvLXnemvSXoI8hgV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Linux restores tags when a page is swapped in and there are tags associated with the swap entry which the new page will replace. The saved tags are restored even if the page will not be mapped as tagged, to protect against cases where the page is shared between different VMAs, and is tagged in some, but untagged in others. By using this approach, the process can still access the correct tags following an mprotect(PROT_MTE) on the non-MTE enabled VMA. But this poses a challenge for managing tag storage: in the scenario above, when a new page is allocated to be swapped in for the process where it will be mapped as untagged, the corresponding tag storage block is not reserved. mte_restore_page_tags_by_swp_entry(), when it restores the saved tags, will overwrite data in the tag storage block associated with the new page, leading to data corruption if the block is in use by a process. Get around this issue by saving the tags in a new xarray, this time indexed by the page pfn, and then restoring them when tag storage is reserved for the page. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/mte_tag_storage.h | 9 ++ arch/arm64/include/asm/pgtable.h | 11 +++ arch/arm64/kernel/mte_tag_storage.c | 20 +++- arch/arm64/mm/mteswap.c | 112 +++++++++++++++++++++++ 4 files changed, 148 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index 6a8b19a6a758..a3c38099fe1a 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -37,6 +37,15 @@ bool page_is_tag_storage(struct page *page); vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf); vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf); + +void tags_by_pfn_lock(void); +void tags_by_pfn_unlock(void); + +void *mte_erase_tags_for_pfn(unsigned long pfn); +bool mte_save_tags_for_pfn(void *tags, unsigned long pfn); +void mte_restore_tags_for_pfn(unsigned long start_pfn, int order); + +vm_fault_t mte_try_transfer_swap_tags(swp_entry_t entry, struct page *page); #else static inline bool tag_storage_enabled(void) { diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 1704411c096d..1a25b7d601c2 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1084,6 +1084,17 @@ static inline void arch_swap_invalidate_area(int type) mte_invalidate_tags_area_by_swp_entry(type); } +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE +#define __HAVE_ARCH_SWAP_PREPARE_TO_RESTORE +static inline vm_fault_t arch_swap_prepare_to_restore(swp_entry_t entry, + struct folio *folio) +{ + if (tag_storage_enabled()) + return mte_try_transfer_swap_tags(entry, &folio->page); + return 0; +} +#endif + #define __HAVE_ARCH_SWAP_RESTORE static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) { diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 5096ce859136..6b11bb408b51 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -547,8 +547,10 @@ int reserve_tag_storage(struct page *page, int order, gfp_t gfp) mutex_lock(&tag_blocks_lock); /* Check again, this time with the lock held. */ - if (page_tag_storage_reserved(page)) - goto out_unlock; + if (page_tag_storage_reserved(page)) { + mutex_unlock(&tag_blocks_lock); + return 0; + } /* Make sure existing entries are not freed from out under out feet. */ xa_lock_irqsave(&tag_blocks_reserved, flags); @@ -583,9 +585,10 @@ int reserve_tag_storage(struct page *page, int order, gfp_t gfp) } page_set_tag_storage_reserved(page, order); -out_unlock: mutex_unlock(&tag_blocks_lock); + mte_restore_tags_for_pfn(page_to_pfn(page), order); + return 0; out_error: @@ -612,7 +615,8 @@ void free_tag_storage(struct page *page, int order) struct tag_region *region; unsigned long page_va; unsigned long flags; - int ret; + void *tags; + int i, ret; ret = tag_storage_find_block(page, &start_block, ®ion); if (WARN_ONCE(ret, "Missing tag storage block for pfn 0x%lx", page_to_pfn(page))) @@ -622,6 +626,14 @@ void free_tag_storage(struct page *page, int order) /* Avoid writeback of dirty tag cache lines corrupting data. */ dcache_inval_tags_poc(page_va, page_va + (PAGE_SIZE << order)); + tags_by_pfn_lock(); + for (i = 0; i < (1 << order); i++) { + tags = mte_erase_tags_for_pfn(page_to_pfn(page + i)); + if (unlikely(tags)) + mte_free_tag_buf(tags); + } + tags_by_pfn_unlock(); + end_block = start_block + order_to_num_blocks(order) * region->block_size; xa_lock_irqsave(&tag_blocks_reserved, flags); diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index 2a43746b803f..20d718a514af 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -20,6 +20,114 @@ void mte_free_tag_buf(void *buf) kfree(buf); } +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE +static DEFINE_XARRAY(tags_by_pfn); + +void tags_by_pfn_lock(void) +{ + xa_lock(&tags_by_pfn); +} + +void tags_by_pfn_unlock(void) +{ + xa_unlock(&tags_by_pfn); +} + +void *mte_erase_tags_for_pfn(unsigned long pfn) +{ + return __xa_erase(&tags_by_pfn, pfn); +} + +bool mte_save_tags_for_pfn(void *tags, unsigned long pfn) +{ + void *entry; + int ret; + + ret = xa_reserve(&tags_by_pfn, pfn, GFP_KERNEL); + if (ret) + return true; + + tags_by_pfn_lock(); + + if (page_tag_storage_reserved(pfn_to_page(pfn))) { + tags_by_pfn_unlock(); + return false; + } + + entry = __xa_store(&tags_by_pfn, pfn, tags, GFP_ATOMIC); + if (xa_is_err(entry)) { + xa_release(&tags_by_pfn, pfn); + goto out_unlock; + } else if (entry) { + mte_free_tag_buf(entry); + } + +out_unlock: + tags_by_pfn_unlock(); + return true; +} + +void mte_restore_tags_for_pfn(unsigned long start_pfn, int order) +{ + struct page *page = pfn_to_page(start_pfn); + unsigned long pfn; + void *tags; + + tags_by_pfn_lock(); + + for (pfn = start_pfn; pfn < start_pfn + (1 << order); pfn++, page++) { + if (WARN_ON_ONCE(!page_tag_storage_reserved(page))) + continue; + + tags = mte_erase_tags_for_pfn(pfn); + if (unlikely(tags)) { + /* + * Mark the page as tagged so mte_sync_tags() doesn't + * clear the tags. + */ + WARN_ON_ONCE(!try_page_mte_tagging(page)); + mte_copy_page_tags_from_buf(page_address(page), tags); + set_page_mte_tagged(page); + mte_free_tag_buf(tags); + } + } + + tags_by_pfn_unlock(); +} + +/* + * Note on locking: swap in/out is done with the folio locked, which eliminates + * races with mte_save/restore_page_tags_by_swp_entry. + */ +vm_fault_t mte_try_transfer_swap_tags(swp_entry_t entry, struct page *page) +{ + void *swap_tags, *pfn_tags; + bool saved; + + /* + * mte_restore_page_tags_by_swp_entry() will take care of copying the + * tags over. + */ + if (likely(page_mte_tagged(page) || page_tag_storage_reserved(page))) + return 0; + + swap_tags = xa_load(&tags_by_swp_entry, entry.val); + if (!swap_tags) + return 0; + + pfn_tags = mte_allocate_tag_buf(); + if (!pfn_tags) + return VM_FAULT_OOM; + + memcpy(pfn_tags, swap_tags, MTE_PAGE_TAG_STORAGE_SIZE); + saved = mte_save_tags_for_pfn(pfn_tags, page_to_pfn(page)); + if (!saved) + mte_free_tag_buf(pfn_tags); + + return 0; +} +#endif + int mte_save_page_tags_by_swp_entry(struct page *page) { void *tags, *ret; @@ -54,6 +162,10 @@ void mte_restore_page_tags_by_swp_entry(swp_entry_t entry, struct page *page) if (!tags) return; + /* Tags will be restored when tag storage is reserved. */ + if (tag_storage_enabled() && unlikely(!page_tag_storage_reserved(page))) + return; + if (try_page_mte_tagging(page)) { mte_copy_page_tags_from_buf(page_address(page), tags); set_page_mte_tagged(page); -- 2.42.1