From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8232FC47DA9 for ; Tue, 30 Jan 2024 00:04:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D114E6B0074; Mon, 29 Jan 2024 19:04:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CC1346B0075; Mon, 29 Jan 2024 19:04:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B3B376B007B; Mon, 29 Jan 2024 19:04:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A1D936B0074 for ; Mon, 29 Jan 2024 19:04:35 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2E32814058D for ; Tue, 30 Jan 2024 00:04:35 +0000 (UTC) X-FDA: 81734030910.06.B5F9822 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf30.hostedemail.com (Postfix) with ESMTP id 59A3880017 for ; Tue, 30 Jan 2024 00:04:33 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zEwHtYL6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of pcc@google.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=pcc@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706573073; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XsMqf9pWxijCVSf9NKW4Fy1rCmOsTFbCNLYH99qokC8=; b=S72bmwOTQxtHNrAOy4DmOYaywPlbbHvA0URuF2C4piEiU1P6EkTkycu+pCBECtH7RQa73d iMjF4hcYt7KWXQB3gQ+6lg+o6p+lC4Ch4ghAbsBx+u6nmqCEabEkEAzfRCzRZXN9C7LfiW TsR6vKLu/DIS/GTkCAuC1lJCPPPPGaQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zEwHtYL6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of pcc@google.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=pcc@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706573073; a=rsa-sha256; cv=none; b=E4Qc8WerkkBo0mBzfEHAZAeBht/CFfa/Pc8i19qAX6zEuLH51QB6Fearj2yNd2Tkqyl6jx W5kxAPbWCWTXcv3C6luZExhQk0wez6tvcC/kh6072T+Xam3js23dtns072X4yZjau9oejZ VknmD01hLFFRULyr7jm15jMVSO98+Hk= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-1d5ce88b51cso83105ad.0 for ; Mon, 29 Jan 2024 16:04:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706573072; x=1707177872; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=XsMqf9pWxijCVSf9NKW4Fy1rCmOsTFbCNLYH99qokC8=; b=zEwHtYL6e8XmECiXQ9Il4+hyqtf6yZCLNvpoj2vIZP5IYCVNw7Xv+hu3H8MQZQgRAc moUl1Pfyt7V3wkZ+ajncMkIl5A+d7O6YGxfEnfkTfmfnw8LxoKySo359C3vesoz9BSK5 wTYjCyf4r3ndQEUAQpq72Ms9B3f71LCc2Xiif0OdG4nuuNknOjRLaZplhc67ADH1qLwA ySeCr0rUnWMebRXHVv0ZnZvsq1KIGbULTT+wPa+VtJiDqpVpig2SABvEIH6evYnOJUTR HNjiKzqz/CIxvkNVDZS88lUaWgTxTt+bqAj18wL86YwW+MfHorWYVrxegL4G9VdCyjXt GdZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706573072; x=1707177872; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XsMqf9pWxijCVSf9NKW4Fy1rCmOsTFbCNLYH99qokC8=; b=X/DDiKFJKGY3LmekOQ+jB/dcG302ZnC63WJMffT8bEqP58FoOWA7MxwK9ycBRdSyPs G6gTmJ5b9RXlNJN/ZLn2xA6JgSNDVLNlMzLzsPbGAYinepP8kv9V4uIvDfNBVw1sfUjt f+AO8EHhNS7e7FXs9M2WU0ydEElpGhd6jbeISGz6uBsQhIwgubArKpHi7hMn/yMIfPoE hMR2WJAzGxZ+lFvjDcLEVogTzyF3PcTZDOo3feGowRwoPLvFmrv/rGceMAxYKYGU2IeF nF9lslLXOxZ27b46JDWGc2XeeK7xbdcJas7wAcGnjz9V7I9ZLMBBOleLyzgXlzwg3Pmz etIA== X-Gm-Message-State: AOJu0YyisLypIGzwdkgNtJsqwuJ1uU+OOEb35AP2e7qwoNqMFLWhynxV FFAFwNEzSd4wWQ1F247QCSeXwG/XUO6S/AydKhq9IuFT1Ym7BQKthY+RYsUu9mATNHNmGlL+aoi /yG9HNX9Z19hLl0Hkgqy4L+TUn4Miom1Pzulu X-Google-Smtp-Source: AGHT+IFyM5oCFBirmOpOeGwpkpTYbv6MnkTqUfPBLo5XRceX1ihVflhMS0aXC9nYBTQv8DrmkqFPshTcxb3N5LOfyxY= X-Received: by 2002:a17:902:e547:b0:1d7:806a:a13e with SMTP id n7-20020a170902e54700b001d7806aa13emr100755plf.28.1706573071712; Mon, 29 Jan 2024 16:04:31 -0800 (PST) MIME-Version: 1.0 References: <20240125164256.4147-1-alexandru.elisei@arm.com> <20240125164256.4147-24-alexandru.elisei@arm.com> In-Reply-To: <20240125164256.4147-24-alexandru.elisei@arm.com> From: Peter Collingbourne Date: Mon, 29 Jan 2024 16:04:18 -0800 Message-ID: Subject: Re: [PATCH RFC v3 23/35] arm64: mte: Try to reserve tag storage in arch_alloc_page() To: Alexandru Elisei Cc: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 59A3880017 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: x5t44qpqqbdjqyjhumh4ymthay7f4nup X-HE-Tag: 1706573073-51149 X-HE-Meta: U2FsdGVkX19sNmEz4vIomAXKgUSdSu4zjdG9MON3i1617nK7klrq/dJsBX/bw+ulH64z4FwWCZo0SOX53hGSe6mGwPgrOPj4120PBVy79j1qOpejNOy5PVPEQ0CkqIm7RZqbFQY3EyX40IQhtFY79TD4jjsZG488F7LqmzcZEDpKMSMTXqad1PP/P9beU+sFao3KAR16Oa4E7bn3sB4qJ5TEsb0oqRl72H23T4b3U3KRrJR2Ms4bLKGApgUb03kOQGq6h8K+bIUDQ3QKV6hR2S56fT5sLK12OStS/4uglPEyHLAJijpX3l9kK+TwTyqo1eotvo39anRVcOQGLmpN/jmRvMU5ECbhNwTCDFxTJX7KgsOpLJuPM7uTHhWgFmPTj/yOEJdkXfvOfUiK4VEQFLOJfc9NtIirDzTYNEn79HxEqjSX65KuI3ezbKqm4IKrrxZ7+GketYMHRr6e6S4N/RtAKp9Vho0Gsg/T4NC4sVcQUlNpdrceUWtLGBCpLBqz7Wg9s9zJ/TWPCF4p9tF4ukSPmATWuGcPccQNRPRcrtmc3u9YvngiNy47lV/OQQq6XDicAzB70d/jIlrdVDSB9fiYHdzlUv5HdFoHQIv7A5ZXSkqX5VFNvA5dMzNtZ4KGNP9Std1m5z8zGKuaS4zctR6hISYmSQVrI2b6LX6YXO2CQORRKXOoCIjIuhZCbAL5b3Z6koXg71cYZ2IBqHcvVFiaT8CnIe7+1u/7sPaC0aeVbaiidUL8zFubVJZRYv5Zl9vZKd5b9F7HLPh+5wT1x8n+45Jh3Zfmm9gvH6KhBWbEYhyMYoXxb75U7j06wCUJrlPfTtugPmGTpRYhLpUM4aRiQo2aTcDmgLcszIHDeEMhaMV8WqNrM73esmt6lmH9NJ92OeudrMW3w3d8JAn3lUiUIppAI//WDk0fTV153+TX1YfOY0XQdtL8HftanG/r/hnxbJcco+rD9A6noLY O6eadICx 80ltVpnu0drYGYnu2jD7c9pZLBhAmEmO1qj+wYrSeK8Vx1GA1KTTj5WGMZTrZ9edJDs7Pfa4n+kU7iU5fC2suNY1DqvKRbm4XdIyqCyWTax5JfcNtVpanJ7di3aPic1inUOKMFjM0rriyqxGBTULGyovFhWlmc5ziBCfZAlc/l38KmZngZ7/v37Zmw6jaPbJAEQarHQn2c3ejsVSWWQ9A/C4edQ4gAdCJV4LihA8DP3XWysmVM7cMvZZSkVfVVO/FszbWICi/48Sb7RyP7ICxaWCH4rTQE9jaFo6QkMzzHzH8t9XYm9i6p4ynK/KgH69lGYsIODDNL4fblUkkpbecf47ikA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Jan 25, 2024 at 8:45=E2=80=AFAM Alexandru Elisei wrote: > > Reserve tag storage for a page that is being allocated as tagged. This > is a best effort approach, and failing to reserve tag storage is > allowed. > > When all the associated tagged pages have been freed, return the tag > storage pages back to the page allocator, where they can be used again fo= r > data allocations. > > Signed-off-by: Alexandru Elisei > --- > > Changes since rfc v2: > > * Based on rfc v2 patch #16 ("arm64: mte: Manage tag storage on page > allocation"). > * Fixed calculation of the number of associated tag storage blocks (Hyeso= o > Yu). > * Tag storage is reserved in arch_alloc_page() instead of > arch_prep_new_page(). > > arch/arm64/include/asm/mte.h | 16 +- > arch/arm64/include/asm/mte_tag_storage.h | 31 +++ > arch/arm64/include/asm/page.h | 5 + > arch/arm64/include/asm/pgtable.h | 19 ++ > arch/arm64/kernel/mte_tag_storage.c | 234 +++++++++++++++++++++++ > arch/arm64/mm/fault.c | 7 + > fs/proc/page.c | 1 + > include/linux/kernel-page-flags.h | 1 + > include/linux/page-flags.h | 1 + > include/trace/events/mmflags.h | 3 +- > mm/huge_memory.c | 1 + > 11 files changed, 316 insertions(+), 3 deletions(-) > > diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h > index 8034695b3dd7..6457b7899207 100644 > --- a/arch/arm64/include/asm/mte.h > +++ b/arch/arm64/include/asm/mte.h > @@ -40,12 +40,24 @@ void mte_free_tag_buf(void *buf); > #ifdef CONFIG_ARM64_MTE > > /* track which pages have valid allocation tags */ > -#define PG_mte_tagged PG_arch_2 > +#define PG_mte_tagged PG_arch_2 > /* simple lock to avoid multiple threads tagging the same page */ > -#define PG_mte_lock PG_arch_3 > +#define PG_mte_lock PG_arch_3 > +/* Track if a tagged page has tag storage reserved */ > +#define PG_tag_storage_reserved PG_arch_4 > + > +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE > +DECLARE_STATIC_KEY_FALSE(tag_storage_enabled_key); > +extern bool page_tag_storage_reserved(struct page *page); > +#endif > > static inline void set_page_mte_tagged(struct page *page) > { > +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE > + /* Open code mte_tag_storage_enabled() */ > + WARN_ON_ONCE(static_branch_likely(&tag_storage_enabled_key) && > + !page_tag_storage_reserved(page)); > +#endif > /* > * Ensure that the tags written prior to this function are visibl= e > * before the page flags update. > diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/includ= e/asm/mte_tag_storage.h > index 7b3f6bff8e6f..09f1318d924e 100644 > --- a/arch/arm64/include/asm/mte_tag_storage.h > +++ b/arch/arm64/include/asm/mte_tag_storage.h > @@ -5,6 +5,12 @@ > #ifndef __ASM_MTE_TAG_STORAGE_H > #define __ASM_MTE_TAG_STORAGE_H > > +#ifndef __ASSEMBLY__ > + > +#include > + > +#include > + > #ifdef CONFIG_ARM64_MTE_TAG_STORAGE > > DECLARE_STATIC_KEY_FALSE(tag_storage_enabled_key); > @@ -15,6 +21,15 @@ static inline bool tag_storage_enabled(void) > } > > void mte_init_tag_storage(void); > + > +static inline bool alloc_requires_tag_storage(gfp_t gfp) > +{ > + return gfp & __GFP_TAGGED; > +} > +int reserve_tag_storage(struct page *page, int order, gfp_t gfp); > +void free_tag_storage(struct page *page, int order); > + > +bool page_tag_storage_reserved(struct page *page); > #else > static inline bool tag_storage_enabled(void) > { > @@ -23,6 +38,22 @@ static inline bool tag_storage_enabled(void) > static inline void mte_init_tag_storage(void) > { > } > +static inline bool alloc_requires_tag_storage(struct page *page) This function should take a gfp_t to match the CONFIG_ARM64_MTE_TAG_STORAGE case. Peter > +{ > + return false; > +} > +static inline int reserve_tag_storage(struct page *page, int order, gfp_= t gfp) > +{ > + return 0; > +} > +static inline void free_tag_storage(struct page *page, int order) > +{ > +} > +static inline bool page_tag_storage_reserved(struct page *page) > +{ > + return true; > +} > #endif /* CONFIG_ARM64_MTE_TAG_STORAGE */ > > +#endif /* !__ASSEMBLY__ */ > #endif /* __ASM_MTE_TAG_STORAGE_H */ > diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.= h > index 88bab032a493..3a656492f34a 100644 > --- a/arch/arm64/include/asm/page.h > +++ b/arch/arm64/include/asm/page.h > @@ -35,6 +35,11 @@ void copy_highpage(struct page *to, struct page *from)= ; > void tag_clear_highpage(struct page *to); > #define __HAVE_ARCH_TAG_CLEAR_HIGHPAGE > > +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE > +void arch_alloc_page(struct page *, int order, gfp_t gfp); > +#define HAVE_ARCH_ALLOC_PAGE > +#endif > + > #define clear_user_page(page, vaddr, pg) clear_page(page) > #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) > > diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pg= table.h > index 2499cc4fa4f2..f30466199a9b 100644 > --- a/arch/arm64/include/asm/pgtable.h > +++ b/arch/arm64/include/asm/pgtable.h > @@ -10,6 +10,7 @@ > > #include > #include > +#include > #include > #include > #include > @@ -1069,6 +1070,24 @@ static inline void arch_swap_restore(swp_entry_t e= ntry, struct folio *folio) > mte_restore_page_tags_by_swp_entry(entry, &folio->page); > } > > +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE > + > +#define __HAVE_ARCH_FREE_PAGES_PREPARE > +static inline void arch_free_pages_prepare(struct page *page, int order) > +{ > + if (tag_storage_enabled() && page_mte_tagged(page)) > + free_tag_storage(page, order); > +} > + > +#define __HAVE_ARCH_ALLOC_CMA > +static inline bool arch_alloc_cma(gfp_t gfp_mask) > +{ > + if (tag_storage_enabled() && alloc_requires_tag_storage(gfp_mask)= ) > + return false; > + return true; > +} > + > +#endif /* CONFIG_ARM64_MTE_TAG_STORAGE */ > #endif /* CONFIG_ARM64_MTE */ > > #define __HAVE_ARCH_CALC_VMA_GFP > diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_= tag_storage.c > index d58c68b4a849..762c7c803a70 100644 > --- a/arch/arm64/kernel/mte_tag_storage.c > +++ b/arch/arm64/kernel/mte_tag_storage.c > @@ -34,6 +34,31 @@ struct tag_region { > static struct tag_region tag_regions[MAX_TAG_REGIONS]; > static int num_tag_regions; > > +/* > + * A note on locking. Reserving tag storage takes the tag_blocks_lock mu= tex, > + * because alloc_contig_range() might sleep. > + * > + * Freeing tag storage takes the xa_lock spinlock with interrupts disabl= ed > + * because pages can be freed from non-preemptible contexts, including f= rom an > + * interrupt handler. > + * > + * Because tag storage can be freed from interrupt contexts, the xarray = is > + * defined with the XA_FLAGS_LOCK_IRQ flag to disable interrupts when ca= lling > + * xa_store(). This is done to prevent a deadlock with free_tag_storage(= ) being > + * called from an interrupt raised before xa_store() releases the xa_loc= k. > + * > + * All of the above means that reserve_tag_storage() cannot run concurre= ntly > + * with itself (no concurrent insertions), but it can run at the same ti= me as > + * free_tag_storage(). The first thing that reserve_tag_storage() does a= fter > + * taking the mutex is increase the refcount on all present tag storage = blocks > + * with the xa_lock held, to serialize against freeing the blocks. This = is an > + * optimization to avoid taking and releasing the xa_lock after each ite= ration > + * if the refcount operation was moved inside the loop, where it would h= ave had > + * to be executed for each block. > + */ > +static DEFINE_XARRAY_FLAGS(tag_blocks_reserved, XA_FLAGS_LOCK_IRQ); > +static DEFINE_MUTEX(tag_blocks_lock); > + > static u32 __init get_block_size_pages(u32 block_size_bytes) > { > u32 a =3D PAGE_SIZE; > @@ -364,3 +389,212 @@ static int __init mte_enable_tag_storage(void) > return -EINVAL; > } > arch_initcall(mte_enable_tag_storage); > + > +static void page_set_tag_storage_reserved(struct page *page, int order) > +{ > + int i; > + > + for (i =3D 0; i < (1 << order); i++) > + set_bit(PG_tag_storage_reserved, &(page + i)->flags); > +} > + > +static void block_ref_add(unsigned long block, struct tag_region *region= , int order) > +{ > + int count; > + > + count =3D min(1u << order, 32 * region->block_size_pages); > + page_ref_add(pfn_to_page(block), count); > +} > + > +static int block_ref_sub_return(unsigned long block, struct tag_region *= region, int order) > +{ > + int count; > + > + count =3D min(1u << order, 32 * region->block_size_pages); > + return page_ref_sub_return(pfn_to_page(block), count); > +} > + > +static bool tag_storage_block_is_reserved(unsigned long block) > +{ > + return xa_load(&tag_blocks_reserved, block) !=3D NULL; > +} > + > +static int tag_storage_reserve_block(unsigned long block, struct tag_reg= ion *region, int order) > +{ > + int ret; > + > + ret =3D xa_err(xa_store(&tag_blocks_reserved, block, pfn_to_page(= block), GFP_KERNEL)); > + if (!ret) > + block_ref_add(block, region, order); > + > + return ret; > +} > + > +static int order_to_num_blocks(int order, u32 block_size_pages) > +{ > + int num_tag_storage_pages =3D max((1 << order) / 32, 1); > + > + return DIV_ROUND_UP(num_tag_storage_pages, block_size_pages); > +} > + > +static int tag_storage_find_block_in_region(struct page *page, unsigned = long *blockp, > + struct tag_region *region) > +{ > + struct range *tag_range =3D ®ion->tag_range; > + struct range *mem_range =3D ®ion->mem_range; > + u64 page_pfn =3D page_to_pfn(page); > + u64 block, block_offset; > + > + if (!(mem_range->start <=3D page_pfn && page_pfn <=3D mem_range->= end)) > + return -ERANGE; > + > + block_offset =3D (page_pfn - mem_range->start) / 32; > + block =3D tag_range->start + rounddown(block_offset, region->bloc= k_size_pages); > + > + if (block + region->block_size_pages - 1 > tag_range->end) { > + pr_err("Block 0x%llx-0x%llx is outside tag region 0x%llx-= 0x%llx\n", > + PFN_PHYS(block), PFN_PHYS(block + region->block_s= ize_pages + 1) - 1, > + PFN_PHYS(tag_range->start), PFN_PHYS(tag_range->e= nd + 1) - 1); > + return -ERANGE; > + } > + *blockp =3D block; > + > + return 0; > + > +} > + > +static int tag_storage_find_block(struct page *page, unsigned long *bloc= k, > + struct tag_region **region) > +{ > + int i, ret; > + > + for (i =3D 0; i < num_tag_regions; i++) { > + ret =3D tag_storage_find_block_in_region(page, block, &ta= g_regions[i]); > + if (ret =3D=3D 0) { > + *region =3D &tag_regions[i]; > + return 0; > + } > + } > + > + return -EINVAL; > +} > + > +bool page_tag_storage_reserved(struct page *page) > +{ > + return test_bit(PG_tag_storage_reserved, &page->flags); > +} > + > +int reserve_tag_storage(struct page *page, int order, gfp_t gfp) > +{ > + unsigned long start_block, end_block; > + struct tag_region *region; > + unsigned long block; > + unsigned long flags; > + int ret =3D 0; > + > + VM_WARN_ON_ONCE(!preemptible()); > + > + if (page_tag_storage_reserved(page)) > + return 0; > + > + /* > + * __alloc_contig_migrate_range() ignores gfp when allocating the > + * destination page for migration. Regardless, massage gfp flags = and > + * remove __GFP_TAGGED to avoid recursion in case gfp stops being > + * ignored. > + */ > + gfp &=3D ~__GFP_TAGGED; > + if (!(gfp & __GFP_NORETRY)) > + gfp |=3D __GFP_RETRY_MAYFAIL; > + > + ret =3D tag_storage_find_block(page, &start_block, ®ion); > + if (WARN_ONCE(ret, "Missing tag storage block for pfn 0x%lx", pag= e_to_pfn(page))) > + return -EINVAL; > + end_block =3D start_block + order_to_num_blocks(order, region->bl= ock_size_pages); > + > + mutex_lock(&tag_blocks_lock); > + > + /* Check again, this time with the lock held. */ > + if (page_tag_storage_reserved(page)) > + goto out_unlock; > + > + /* Make sure existing entries are not freed from out under out fe= et. */ > + xa_lock_irqsave(&tag_blocks_reserved, flags); > + for (block =3D start_block; block < end_block; block +=3D region-= >block_size_pages) { > + if (tag_storage_block_is_reserved(block)) > + block_ref_add(block, region, order); > + } > + xa_unlock_irqrestore(&tag_blocks_reserved, flags); > + > + for (block =3D start_block; block < end_block; block +=3D region-= >block_size_pages) { > + /* Refcount incremented above. */ > + if (tag_storage_block_is_reserved(block)) > + continue; > + > + ret =3D cma_alloc_range(region->cma, block, region->block= _size_pages, 3, gfp); > + /* Should never happen. */ > + VM_WARN_ON_ONCE(ret =3D=3D -EEXIST); > + if (ret) > + goto out_error; > + > + ret =3D tag_storage_reserve_block(block, region, order); > + if (ret) { > + cma_release(region->cma, pfn_to_page(block), regi= on->block_size_pages); > + goto out_error; > + } > + } > + > + page_set_tag_storage_reserved(page, order); > +out_unlock: > + mutex_unlock(&tag_blocks_lock); > + > + return 0; > + > +out_error: > + xa_lock_irqsave(&tag_blocks_reserved, flags); > + for (block =3D start_block; block < end_block; block +=3D region-= >block_size_pages) { > + if (tag_storage_block_is_reserved(block) && > + block_ref_sub_return(block, region, order) =3D=3D 1) = { > + __xa_erase(&tag_blocks_reserved, block); > + cma_release(region->cma, pfn_to_page(block), regi= on->block_size_pages); > + } > + } > + xa_unlock_irqrestore(&tag_blocks_reserved, flags); > + > + mutex_unlock(&tag_blocks_lock); > + > + return ret; > +} > + > +void free_tag_storage(struct page *page, int order) > +{ > + unsigned long block, start_block, end_block; > + struct tag_region *region; > + unsigned long flags; > + int ret; > + > + ret =3D tag_storage_find_block(page, &start_block, ®ion); > + if (WARN_ONCE(ret, "Missing tag storage block for pfn 0x%lx", pag= e_to_pfn(page))) > + return; > + > + end_block =3D start_block + order_to_num_blocks(order, region->bl= ock_size_pages); > + > + xa_lock_irqsave(&tag_blocks_reserved, flags); > + for (block =3D start_block; block < end_block; block +=3D region-= >block_size_pages) { > + if (WARN_ONCE(!tag_storage_block_is_reserved(block), > + "Block 0x%lx is not reserved for pfn 0x%lx", block, p= age_to_pfn(page))) > + continue; > + > + if (block_ref_sub_return(block, region, order) =3D=3D 1) = { > + __xa_erase(&tag_blocks_reserved, block); > + cma_release(region->cma, pfn_to_page(block), regi= on->block_size_pages); > + } > + } > + xa_unlock_irqrestore(&tag_blocks_reserved, flags); > +} > + > +void arch_alloc_page(struct page *page, int order, gfp_t gfp) > +{ > + if (tag_storage_enabled() && alloc_requires_tag_storage(gfp)) > + reserve_tag_storage(page, order, gfp); > +} > diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c > index c022e473c17c..1ffaeccecda2 100644 > --- a/arch/arm64/mm/fault.c > +++ b/arch/arm64/mm/fault.c > @@ -37,6 +37,7 @@ > #include > #include > #include > +#include > #include > #include > #include > @@ -950,6 +951,12 @@ gfp_t arch_calc_vma_gfp(struct vm_area_struct *vma, = gfp_t gfp) > > void tag_clear_highpage(struct page *page) > { > + if (tag_storage_enabled() && !page_tag_storage_reserved(page)) { > + /* Don't zero the tags if tag storage is not reserved */ > + clear_page(page_address(page)); > + return; > + } > + > /* Newly allocated page, shouldn't have been tagged yet */ > WARN_ON_ONCE(!try_page_mte_tagging(page)); > mte_zero_clear_page_tags(page_address(page)); > diff --git a/fs/proc/page.c b/fs/proc/page.c > index 195b077c0fac..e7eb584a9234 100644 > --- a/fs/proc/page.c > +++ b/fs/proc/page.c > @@ -221,6 +221,7 @@ u64 stable_page_flags(struct page *page) > #ifdef CONFIG_ARCH_USES_PG_ARCH_X > u |=3D kpf_copy_bit(k, KPF_ARCH_2, PG_arch_2); > u |=3D kpf_copy_bit(k, KPF_ARCH_3, PG_arch_3); > + u |=3D kpf_copy_bit(k, KPF_ARCH_4, PG_arch_4); > #endif > > return u; > diff --git a/include/linux/kernel-page-flags.h b/include/linux/kernel-pag= e-flags.h > index 859f4b0c1b2b..4a0d719ffdd4 100644 > --- a/include/linux/kernel-page-flags.h > +++ b/include/linux/kernel-page-flags.h > @@ -19,5 +19,6 @@ > #define KPF_SOFTDIRTY 40 > #define KPF_ARCH_2 41 > #define KPF_ARCH_3 42 > +#define KPF_ARCH_4 43 > > #endif /* LINUX_KERNEL_PAGE_FLAGS_H */ > diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h > index b7237bce7446..03f03e6d735e 100644 > --- a/include/linux/page-flags.h > +++ b/include/linux/page-flags.h > @@ -135,6 +135,7 @@ enum pageflags { > #ifdef CONFIG_ARCH_USES_PG_ARCH_X > PG_arch_2, > PG_arch_3, > + PG_arch_4, > #endif > __NR_PAGEFLAGS, > > diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflag= s.h > index 6ca0d5ed46c0..ba962fd10a2c 100644 > --- a/include/trace/events/mmflags.h > +++ b/include/trace/events/mmflags.h > @@ -125,7 +125,8 @@ IF_HAVE_PG_HWPOISON(hwpoison) = \ > IF_HAVE_PG_IDLE(idle) \ > IF_HAVE_PG_IDLE(young) \ > IF_HAVE_PG_ARCH_X(arch_2) \ > -IF_HAVE_PG_ARCH_X(arch_3) > +IF_HAVE_PG_ARCH_X(arch_3) \ > +IF_HAVE_PG_ARCH_X(arch_4) > > #define show_page_flags(flags) \ > (flags) ? __print_flags(flags, "|", \ > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > index 2bad63a7ec16..47932539cc50 100644 > --- a/mm/huge_memory.c > +++ b/mm/huge_memory.c > @@ -2804,6 +2804,7 @@ static void __split_huge_page_tail(struct folio *fo= lio, int tail, > #ifdef CONFIG_ARCH_USES_PG_ARCH_X > (1L << PG_arch_2) | > (1L << PG_arch_3) | > + (1L << PG_arch_4) | > #endif > (1L << PG_dirty) | > LRU_GEN_MASK | LRU_REFS_MASK)); > -- > 2.43.0 >