* [PATCH v5 2/2] arm64: mte: avoid clearing PSTATE.TCO on entry unless necessary
2022-02-19 1:29 [PATCH v5 1/2] kasan: split kasan_*enabled() functions into a separate header Peter Collingbourne
@ 2022-02-19 1:29 ` Peter Collingbourne
2022-02-22 16:38 ` [PATCH v5 1/2] kasan: split kasan_*enabled() functions into a separate header Andrey Konovalov
2022-02-22 22:38 ` Will Deacon
2 siblings, 0 replies; 4+ messages in thread
From: Peter Collingbourne @ 2022-02-19 1:29 UTC (permalink / raw)
To: Catalin Marinas, Vincenzo Frascino, Will Deacon,
Andrey Konovalov, Mark Rutland
Cc: Peter Collingbourne, Evgenii Stepanov, linux-arm-kernel, linux-mm
On some microarchitectures, clearing PSTATE.TCO is expensive. Clearing
TCO is only necessary if in-kernel MTE is enabled, or if MTE is
enabled in the userspace process in synchronous (or, soon, asymmetric)
mode, because we do not report uaccess faults to userspace in none
or asynchronous modes. Therefore, adjust the kernel entry code to
clear TCO only if necessary.
Because it is now possible to switch to a task in which TCO needs to
be clear from a task in which TCO is set, we also need to do the same
thing on task switch.
Signed-off-by: Peter Collingbourne <pcc@google.com>
Link: https://linux-review.googlesource.com/id/I52d82a580bd0500d420be501af2c35fa8c90729e
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
---
v5:
- fix header circular dependency with KASAN_GENERIC/KASAN_SW_TAGS
enabled with a dependent patch
v4:
- some changes suggested by Catalin
v3:
- switch to a C implementation
v2:
- do the same thing in cpu_switch_to()
arch/arm64/include/asm/mte.h | 22 ++++++++++++++++++++++
arch/arm64/kernel/entry-common.c | 3 +++
arch/arm64/kernel/entry.S | 7 -------
arch/arm64/kernel/mte.c | 3 +++
4 files changed, 28 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
index 075539f5f1c8..adcb937342f1 100644
--- a/arch/arm64/include/asm/mte.h
+++ b/arch/arm64/include/asm/mte.h
@@ -11,7 +11,9 @@
#ifndef __ASSEMBLY__
#include <linux/bitfield.h>
+#include <linux/kasan-enabled.h>
#include <linux/page-flags.h>
+#include <linux/sched.h>
#include <linux/types.h>
#include <asm/pgtable-types.h>
@@ -86,6 +88,26 @@ static inline int mte_ptrace_copy_tags(struct task_struct *child,
#endif /* CONFIG_ARM64_MTE */
+static inline void mte_disable_tco_entry(struct task_struct *task)
+{
+ if (!system_supports_mte())
+ return;
+
+ /*
+ * Re-enable tag checking (TCO set on exception entry). This is only
+ * necessary if MTE is enabled in either the kernel or the userspace
+ * task in synchronous or asymmetric mode (SCTLR_EL1.TCF0 bit 0 is set
+ * for both). With MTE disabled in the kernel and disabled or
+ * asynchronous in userspace, tag check faults (including in uaccesses)
+ * are not reported, therefore there is no need to re-enable checking.
+ * This is beneficial on microarchitectures where re-enabling TCO is
+ * expensive.
+ */
+ if (kasan_hw_tags_enabled() ||
+ (task->thread.sctlr_user & (1UL << SCTLR_EL1_TCF0_SHIFT)))
+ asm volatile(SET_PSTATE_TCO(0));
+}
+
#ifdef CONFIG_KASAN_HW_TAGS
/* Whether the MTE asynchronous mode is enabled. */
DECLARE_STATIC_KEY_FALSE(mte_async_or_asymm_mode);
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index ef7fcefb96bd..7093b578e325 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -6,6 +6,7 @@
*/
#include <linux/context_tracking.h>
+#include <linux/kasan.h>
#include <linux/linkage.h>
#include <linux/lockdep.h>
#include <linux/ptrace.h>
@@ -56,6 +57,7 @@ static void noinstr enter_from_kernel_mode(struct pt_regs *regs)
{
__enter_from_kernel_mode(regs);
mte_check_tfsr_entry();
+ mte_disable_tco_entry(current);
}
/*
@@ -103,6 +105,7 @@ static __always_inline void __enter_from_user_mode(void)
CT_WARN_ON(ct_state() != CONTEXT_USER);
user_exit_irqoff();
trace_hardirqs_off_finish();
+ mte_disable_tco_entry(current);
}
static __always_inline void enter_from_user_mode(struct pt_regs *regs)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 772ec2ecf488..e1013a83d4f0 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -308,13 +308,6 @@ alternative_if ARM64_HAS_IRQ_PRIO_MASKING
msr_s SYS_ICC_PMR_EL1, x20
alternative_else_nop_endif
- /* Re-enable tag checking (TCO set on exception entry) */
-#ifdef CONFIG_ARM64_MTE
-alternative_if ARM64_MTE
- SET_PSTATE_TCO(0)
-alternative_else_nop_endif
-#endif
-
/*
* Registers that may be useful after this macro is invoked:
*
diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
index f418ebc65f95..f983795b5eda 100644
--- a/arch/arm64/kernel/mte.c
+++ b/arch/arm64/kernel/mte.c
@@ -253,6 +253,9 @@ void mte_thread_switch(struct task_struct *next)
mte_update_sctlr_user(next);
mte_update_gcr_excl(next);
+ /* TCO may not have been disabled on exception entry for the current task. */
+ mte_disable_tco_entry(next);
+
/*
* Check if an async tag exception occurred at EL1.
*
--
2.35.1.473.g83b2b277ed-goog
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH v5 1/2] kasan: split kasan_*enabled() functions into a separate header
2022-02-19 1:29 [PATCH v5 1/2] kasan: split kasan_*enabled() functions into a separate header Peter Collingbourne
2022-02-19 1:29 ` [PATCH v5 2/2] arm64: mte: avoid clearing PSTATE.TCO on entry unless necessary Peter Collingbourne
@ 2022-02-22 16:38 ` Andrey Konovalov
2022-02-22 22:38 ` Will Deacon
2 siblings, 0 replies; 4+ messages in thread
From: Andrey Konovalov @ 2022-02-22 16:38 UTC (permalink / raw)
To: Peter Collingbourne
Cc: Catalin Marinas, Vincenzo Frascino, Will Deacon, Mark Rutland,
Evgenii Stepanov, Linux ARM, Linux Memory Management List
On Sat, Feb 19, 2022 at 2:30 AM Peter Collingbourne <pcc@google.com> wrote:
>
> In an upcoming commit we are going to need to call
> kasan_hw_tags_enabled() from arch/arm64/include/asm/mte.h. This
> would create a circular dependency between headers if KASAN_GENERIC
> or KASAN_SW_TAGS is enabled: linux/kasan.h -> linux/pgtable.h ->
> asm/pgtable.h -> asm/mte.h -> linux/kasan.h. Break the cycle
> by introducing a new header linux/kasan-enabled.h with the
> kasan_*enabled() functions that can be included from asm/mte.h.
>
> Link: https://linux-review.googlesource.com/id/I5b0d96c6ed0026fc790899e14d42b2fac6ab568e
> Signed-off-by: Peter Collingbourne <pcc@google.com>
> ---
> include/linux/kasan-enabled.h | 33 +++++++++++++++++++++++++++++++++
> include/linux/kasan.h | 23 +----------------------
> 2 files changed, 34 insertions(+), 22 deletions(-)
> create mode 100644 include/linux/kasan-enabled.h
>
> diff --git a/include/linux/kasan-enabled.h b/include/linux/kasan-enabled.h
> new file mode 100644
> index 000000000000..4b6615375022
> --- /dev/null
> +++ b/include/linux/kasan-enabled.h
> @@ -0,0 +1,33 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef _LINUX_KASAN_ENABLED_H
> +#define _LINUX_KASAN_ENABLED_H
> +
> +#ifdef CONFIG_KASAN_HW_TAGS
> +
> +DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled);
> +
> +static __always_inline bool kasan_enabled(void)
> +{
> + return static_branch_likely(&kasan_flag_enabled);
> +}
> +
> +static inline bool kasan_hw_tags_enabled(void)
> +{
> + return kasan_enabled();
> +}
> +
> +#else /* CONFIG_KASAN_HW_TAGS */
> +
> +static inline bool kasan_enabled(void)
> +{
> + return IS_ENABLED(CONFIG_KASAN);
> +}
> +
> +static inline bool kasan_hw_tags_enabled(void)
> +{
> + return false;
> +}
> +
> +#endif /* CONFIG_KASAN_HW_TAGS */
> +
> +#endif /* LINUX_KASAN_ENABLED_H */
> diff --git a/include/linux/kasan.h b/include/linux/kasan.h
> index 4a45562d8893..b6a93261c92a 100644
> --- a/include/linux/kasan.h
> +++ b/include/linux/kasan.h
> @@ -3,6 +3,7 @@
> #define _LINUX_KASAN_H
>
> #include <linux/bug.h>
> +#include <linux/kasan-enabled.h>
> #include <linux/kernel.h>
> #include <linux/static_key.h>
> #include <linux/types.h>
> @@ -83,33 +84,11 @@ static inline void kasan_disable_current(void) {}
>
> #ifdef CONFIG_KASAN_HW_TAGS
>
> -DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled);
> -
> -static __always_inline bool kasan_enabled(void)
> -{
> - return static_branch_likely(&kasan_flag_enabled);
> -}
> -
> -static inline bool kasan_hw_tags_enabled(void)
> -{
> - return kasan_enabled();
> -}
> -
> void kasan_alloc_pages(struct page *page, unsigned int order, gfp_t flags);
> void kasan_free_pages(struct page *page, unsigned int order);
>
> #else /* CONFIG_KASAN_HW_TAGS */
>
> -static inline bool kasan_enabled(void)
> -{
> - return IS_ENABLED(CONFIG_KASAN);
> -}
> -
> -static inline bool kasan_hw_tags_enabled(void)
> -{
> - return false;
> -}
> -
> static __always_inline void kasan_alloc_pages(struct page *page,
> unsigned int order, gfp_t flags)
> {
> --
> 2.35.1.473.g83b2b277ed-goog
>
Reviewed-by: Andrey Konovalov <andreyknvl@gmail.com>
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH v5 1/2] kasan: split kasan_*enabled() functions into a separate header
2022-02-19 1:29 [PATCH v5 1/2] kasan: split kasan_*enabled() functions into a separate header Peter Collingbourne
2022-02-19 1:29 ` [PATCH v5 2/2] arm64: mte: avoid clearing PSTATE.TCO on entry unless necessary Peter Collingbourne
2022-02-22 16:38 ` [PATCH v5 1/2] kasan: split kasan_*enabled() functions into a separate header Andrey Konovalov
@ 2022-02-22 22:38 ` Will Deacon
2 siblings, 0 replies; 4+ messages in thread
From: Will Deacon @ 2022-02-22 22:38 UTC (permalink / raw)
To: Andrey Konovalov, Catalin Marinas, Mark Rutland,
Vincenzo Frascino, Peter Collingbourne
Cc: kernel-team, Will Deacon, linux-arm-kernel, Evgenii Stepanov, linux-mm
On Fri, 18 Feb 2022 17:29:44 -0800, Peter Collingbourne wrote:
> In an upcoming commit we are going to need to call
> kasan_hw_tags_enabled() from arch/arm64/include/asm/mte.h. This
> would create a circular dependency between headers if KASAN_GENERIC
> or KASAN_SW_TAGS is enabled: linux/kasan.h -> linux/pgtable.h ->
> asm/pgtable.h -> asm/mte.h -> linux/kasan.h. Break the cycle
> by introducing a new header linux/kasan-enabled.h with the
> kasan_*enabled() functions that can be included from asm/mte.h.
>
> [...]
Applied to arm64 (for-next/mte), thanks!
[1/2] kasan: split kasan_*enabled() functions into a separate header
https://git.kernel.org/arm64/c/f9b5e46f4097
[2/2] arm64: mte: avoid clearing PSTATE.TCO on entry unless necessary
https://git.kernel.org/arm64/c/38ddf7dafaea
Cheers,
--
Will
https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev
^ permalink raw reply [flat|nested] 4+ messages in thread