From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78AA1C072A2 for ; Sun, 19 Nov 2023 16:59:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 179266B0395; Sun, 19 Nov 2023 11:59:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 129B46B0399; Sun, 19 Nov 2023 11:59:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F335B6B039B; Sun, 19 Nov 2023 11:59:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E46006B0395 for ; Sun, 19 Nov 2023 11:59:32 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C13894014E for ; Sun, 19 Nov 2023 16:59:32 +0000 (UTC) X-FDA: 81475314984.05.0ADA2AD Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf19.hostedemail.com (Postfix) with ESMTP id EAF691A000E for ; Sun, 19 Nov 2023 16:59:30 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700413171; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5ORLzcwN9LX8c3TlnGGtQGRS/MzysJfZU2qd8FHh5bY=; b=h4n3h99BIJgCi3AJkVupd2cVyYW9QmXeByx12XvQLDYe/1CyZ325ICmpQ34nwrNDFzpBxl eR5iEKvobV/AMmG3Fp9QNfSZ2Ar2tj/7E+i3lV51f8pOa5vWOWhKLY7a3mXEOsEP1WdDyf 8GtJd1zY17dyovgNF+DhkprgcpvI9iM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700413171; a=rsa-sha256; cv=none; b=dkpiEVq+oOmLK/hH0Ba/iWiC7Nm779z4YTp4EelcnSt7Eso2ByiOwgv1K6euuvJadHCLhl qifxzZKm3JH+6JqcbYxUWUTARI4SffzosjS8uCS/p3YgXniA/bm/jyvXdOMPp9PmLNTek2 kRaDWkY7S9QsQV070rC/TVyLWtJPMcc= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6BE0DDA7; Sun, 19 Nov 2023 09:00:16 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 21D8F3F6C4; Sun, 19 Nov 2023 08:59:25 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 21/27] mm: arm64: Handle tag storage pages mapped before mprotect(PROT_MTE) Date: Sun, 19 Nov 2023 16:57:15 +0000 Message-Id: <20231119165721.9849-22-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: EAF691A000E X-Rspam-User: X-Stat-Signature: q3pcss7bp9rtgqkqn8bigyzt1nme1wjb X-Rspamd-Server: rspam03 X-HE-Tag: 1700413170-18481 X-HE-Meta: U2FsdGVkX1+dRDUth7zifAAr1xqzUyjk+3UzQJ+LZSamXU1o8Wy0+6pB/ABJJ1tn0VhUH4McjyW1LbbjBo5AbwmvPyxfU3d/t2Mg3UHvG9X0LUDPPXE60aQ+tPl1usXCn+TfLhZzTJrKo1izKj3giPXEcDs8CHCI16Q5vVhmAT+sNJxN5vMAbivYlLfz6ngQKPM/vi56xRgOmZelpl5xwgzaiGX2MdZERU+87RDkAt9f/4REH8tM4UTya84d2ph0dKY0YQ28cwv8y1v9rjg/L4N79l8ocGC4Iho0KlFAYRjMBEosW5hBF/5v5EjQvlXqlZqIGxKtNe3PZPGHEfZQT6u6iG7K02/oZIAmvB14B7Ri5ZN0FEgr78R/4/lvFQbddR3fPDaUEntwex/TLeHdGsQLhCSCQfNJlhWwadmaUyqyRp/ljFBRwEt6lnjGUHHk7/Wmvc9+9yYlxOvNFvaaxXDn8mk6BztHg08JHW+mTMZfc6VrNy0di4pnZzeqSVntcXiVbGaOh5xQv+YHVE7cwpsV+IhSS/0AbGZKb1p2j0t+2UftpDFxVww7PcEV+2MvTW3m3+5XydefZ6eGBLZT3hHxp1LJNnwWrgwgSphNmIZfzoFKos5hatY2k/xxLy9d+196m+KX07G6OTCoOkPWzZ2xLVxweBHobWeJjf5mZlSlYdfJ6k4FAO+uhss8bo0zjARah452497aVu7w1eM1MQ6eifq5WaMJvBCsLBpVbu64fUH/y5Xu+CcdmIS94lrksXh+1oFVMiEthmqeAuTu+xuFZD5rVv5R+rT1flH2ZT8RcJaR4Hho5l0FeinfznJYzquNttSWBwzze6UhtnhXM+88JScFSqJP22eJ6WUiMwtVgrssqayg/W01FI1UrDaHSILf87sfBXdpuCS6oxjb0CGTTGCbrV8w0YgXAjaBuVIFu4b8Nak7iyY+5RpJj8IgZN7ITN1y6TGGkZ7DKUZ 8nn7k2zX Zugt1fDc/IVCloHqzqMbgXBgTg/ZDC2UGQKftz1cKSOslFx4D5gTNAZ89+QJ0gfYa5z3YjknatBjTyPpBqH6V1yOPM4szURRDcyxS23wcK2B3VkaaJXN3pCsfiOYAIMM6WzK+MwmYG3ITDVN2ii6TrNu0z0y5BR5dc926/GldmFrEZe9T13ioz7eVcNvbMH+PuUI8aCC4kNQYkNgH/41/rz0s2w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/mte_tag_storage.h | 1 + arch/arm64/kernel/mte_tag_storage.c | 15 +++++++ arch/arm64/mm/fault.c | 55 ++++++++++++++++++++++++ include/linux/migrate.h | 8 +++- include/linux/migrate_mode.h | 1 + mm/internal.h | 6 --- 6 files changed, 78 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index b97406d369ce..6a8b19a6a758 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -33,6 +33,7 @@ int reserve_tag_storage(struct page *page, int order, gfp_t gfp); void free_tag_storage(struct page *page, int order); bool page_tag_storage_reserved(struct page *page); +bool page_is_tag_storage(struct page *page); vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf); vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf); diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index a1cc239f7211..5096ce859136 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -500,6 +500,21 @@ bool page_tag_storage_reserved(struct page *page) return test_bit(PG_tag_storage_reserved, &page->flags); } +bool page_is_tag_storage(struct page *page) +{ + unsigned long pfn = page_to_pfn(page); + struct range *tag_range; + int i; + + for (i = 0; i < num_tag_regions; i++) { + tag_range = &tag_regions[i].tag_range; + if (tag_range->start <= pfn && pfn <= tag_range->end) + return true; + } + + return false; +} + int reserve_tag_storage(struct page *page, int order, gfp_t gfp) { unsigned long start_block, end_block; diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 6730a0812a24..964c5ae161a3 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -956,6 +957,50 @@ void tag_clear_highpage(struct page *page) } #ifdef CONFIG_ARM64_MTE_TAG_STORAGE + +#define MR_TAGGED_TAG_STORAGE MR_ARCH_1 + +extern bool isolate_lru_page(struct page *page); +extern void putback_movable_pages(struct list_head *l); + +/* Returns with the page reference dropped. */ +static void migrate_tag_storage_page(struct page *page) +{ + struct migration_target_control mtc = { + .nid = NUMA_NO_NODE, + .gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_TAGGED, + }; + unsigned long i, nr_pages = compound_nr(page); + LIST_HEAD(pagelist); + int ret, tries; + + lru_cache_disable(); + + for (i = 0; i < nr_pages; i++) { + if (!isolate_lru_page(page + i)) { + ret = -EAGAIN; + goto out; + } + /* Isolate just grabbed another reference, drop ours. */ + put_page(page + i); + list_add_tail(&(page + i)->lru, &pagelist); + } + + tries = 5; + while (tries--) { + ret = migrate_pages(&pagelist, alloc_migration_target, NULL, (unsigned long)&mtc, + MIGRATE_SYNC, MR_TAGGED_TAG_STORAGE, NULL); + if (ret == 0 || ret != -EBUSY) + break; + } + +out: + if (ret != 0) + putback_movable_pages(&pagelist); + + lru_cache_enable(); +} + vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; @@ -1013,6 +1058,11 @@ vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf) if (unlikely(is_migrate_isolate_page(page))) goto out_retry; + if (unlikely(page_is_tag_storage(page))) { + migrate_tag_storage_page(page); + return 0; + } + ret = reserve_tag_storage(page, 0, GFP_HIGHUSER_MOVABLE); if (ret) goto out_retry; @@ -1098,6 +1148,11 @@ vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf) if (unlikely(is_migrate_isolate_page(page))) goto out_retry; + if (unlikely(page_is_tag_storage(page))) { + migrate_tag_storage_page(page); + return 0; + } + ret = reserve_tag_storage(page, HPAGE_PMD_ORDER, GFP_HIGHUSER_MOVABLE); if (ret) goto out_retry; diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 0acef592043c..afca42ace735 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -10,8 +10,6 @@ typedef struct folio *new_folio_t(struct folio *folio, unsigned long private); typedef void free_folio_t(struct folio *folio, unsigned long private); -struct migration_target_control; - /* * Return values from addresss_space_operations.migratepage(): * - negative errno on page migration failure; @@ -57,6 +55,12 @@ struct movable_operations { void (*putback_page)(struct page *); }; +struct migration_target_control { + int nid; /* preferred node id */ + nodemask_t *nmask; + gfp_t gfp_mask; +}; + /* Defined in mm/debug.c: */ extern const char *migrate_reason_names[MR_TYPES]; diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h index f37cc03f9369..c6c5c7726d26 100644 --- a/include/linux/migrate_mode.h +++ b/include/linux/migrate_mode.h @@ -29,6 +29,7 @@ enum migrate_reason { MR_CONTIG_RANGE, MR_LONGTERM_PIN, MR_DEMOTION, + MR_ARCH_1, MR_TYPES }; diff --git a/mm/internal.h b/mm/internal.h index ddf6bb6c6308..96fff5dfc041 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -949,12 +949,6 @@ static inline bool is_migrate_highatomic_page(struct page *page) void setup_zone_pageset(struct zone *zone); -struct migration_target_control { - int nid; /* preferred node id */ - nodemask_t *nmask; - gfp_t gfp_mask; -}; - /* * mm/filemap.c */ -- 2.42.1