[PATCH RFC/RFT v2 0/2] Converge common flows for cpu assisted shadow stack

[PATCH RFC/RFT v2 0/2] Converge common flows for cpu assisted shadow stack

[Deepak Gupta]

x86, arm64 and risc-v support cpu assisted shadow stack. x86 was first
one and most of the shadow stack related code is in x86 arch directory.
arm64 guarded control stack (GCS) patches from Mark Brown are in -next.

This led to obvious discussion many how to merge certain common flows in
generic code. Recent one being [1]. Goes without saying having generic
code helps with bug management as well (not having to fix same bug for 3
different arches).

High level common flow between x86, riscv and arm64:

- Enabling is via prctl.
  Enabling and book keeping per task_struct in thread data strutures
  differ on each architecture. This version of patchset doesn't
  try to merge those flows.

- Managing virtual memory for shadow stack handled similarly.
  From kernel's perspective shadow stack writeable memory which can be
  written by only certain selected store operations (depending on arch)
  This patch converges this notion between different architecture to
  allocate, map and free shadow stack.

- Virtual memory management of shadow stack on clone/fork is similar.
  Treatment of copy-on-write (COW) or using parent's stack (CLONE_VFORK)
  or allocating new shadow stack (CLONE_VM) are similar in all arch.
  Thus logic to setup shadow stack should be similar on clone/fork

Mark brown introduced `ARCH_HAS_SHADOW_STACK` as part of arm64 gcs series
[2] and this patch set depends on it. This patchset uses same config to
move as much as possible common code in generic kernel. Additionaly this
patchset introduces wrapper abstractions where arch specific handling is
required.

Generic code and arch specific code for shadow stack are independent
modules and can call into each other. This is by design because each
architecture's enabling mechanisms are different but at the same time
from kernel's perspective it's a special memory which is writeable from
certain selected store operations.

I've not tested this. Only compiled for x86 with shadow stack enable. Thus
this is a RFC and possible looking for some help to test as well on x86.

[1] - https://lore.kernel.org/all/20241008-v5_user_cfi_series-v6-0-60d9fe073f37@rivosinc.com/T/#m98d14237663150778a3f8df59a76a3fe6318624a
[2] - https://lore.kernel.org/linux-arm-kernel/20241001-arm64-gcs-v13-0-222b78d87eee@kernel.org/T/#m1ff65a49873b0e770e71de7af178f581c72be7ad

To: Thomas Gleixner <tglx@linutronix.de>
To: Ingo Molnar <mingo@redhat.com>
To: Borislav Petkov <bp@alien8.de>
To: Dave Hansen <dave.hansen@linux.intel.com>
To: x86@kernel.org
To: H. Peter Anvin <hpa@zytor.com>
To: Andrew Morton <akpm@linux-foundation.org>
To: Liam R. Howlett <Liam.Howlett@oracle.com>
To: Vlastimil Babka <vbabka@suse.cz>
To: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
To: Arnd Bergmann <arnd@arndb.de>
Cc: linux-kernel@vger.kernel.org
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-mm@kvack.org
Cc: linux-arch@vger.kernel.org
Cc: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Mark Brown <broonie@kernel.org>

Signed-off-by: Deepak Gupta <debug@rivosinc.com>
---
Changes in v2:
- Doesn't carry patch which introduces `ARCH_HAS_SHADOW_STACK`. Most likely
  it'll be merged as part of arm64 gcs patch series.
- moves shstk_setup back into x86 portion. Primary reason is that entire arch
  specific prctl specific handling can't be made generic easily due to arch
  differences.
- Due to prctl handling code remaining arch specific, removed generic wrappers
  to set thread status and shstk enabling
- Removed x86 specific comment
- Added `SHADOW_STACK_SET_MARKER`
- Link to v1: https://lore.kernel.org/r/20241010-shstk_converge-v1-0-631beca676e7@rivosinc.com

---
Deepak Gupta (2):
      mm: helper `is_shadow_stack_vma` to check shadow stack vma
      kernel: converge common shadow stack flow agnostic to arch

 arch/x86/include/asm/shstk.h           |   7 +
 arch/x86/include/uapi/asm/mman.h       |   3 -
 arch/x86/kernel/shstk.c                | 223 +++++---------------------------
 include/linux/usershstk.h              |  22 ++++
 include/uapi/asm-generic/mman-common.h |   5 +
 kernel/Makefile                        |   2 +
 kernel/usershstk.c                     | 230 +++++++++++++++++++++++++++++++++
 mm/gup.c                               |   2 +-
 mm/mmap.c                              |   2 +-
 mm/vma.h                               |  10 +-
 10 files changed, 305 insertions(+), 201 deletions(-)
---
base-commit: 4e0105ad0161b4262b51f034a757c4899c647487
change-id: 20241010-shstk_converge-aefbcbef5d71
--
- debug

[PATCH RFC/RFT v2 1/2] mm: helper `is_shadow_stack_vma` to check shadow stack vma

[Deepak Gupta]

VM_SHADOW_STACK (alias to VM_HIGH_ARCH_5) is used to encode shadow stack
VMA on three architectures (x86 shadow stack, arm GCS and RISC-V shadow
stack). In case architecture doesn't implement shadow stack, it's VM_NONE
Introducing a helper `is_shadow_stack_vma` to determine shadow stack vma
or not.

Signed-off-by: Deepak Gupta <debug@rivosinc.com>
---
 mm/gup.c  |  2 +-
 mm/mmap.c |  2 +-
 mm/vma.h  | 10 +++++++---
 3 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/mm/gup.c b/mm/gup.c
index a82890b46a36..8e6e14179f6c 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1282,7 +1282,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
 		    !writable_file_mapping_allowed(vma, gup_flags))
 			return -EFAULT;
 
-		if (!(vm_flags & VM_WRITE) || (vm_flags & VM_SHADOW_STACK)) {
+		if (!(vm_flags & VM_WRITE) || is_shadow_stack_vma(vm_flags)) {
 			if (!(gup_flags & FOLL_FORCE))
 				return -EFAULT;
 			/* hugetlb does not support FOLL_FORCE|FOLL_WRITE. */
diff --git a/mm/mmap.c b/mm/mmap.c
index dd4b35a25aeb..0853e6784069 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -708,7 +708,7 @@ static unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info)
  */
 static inline unsigned long stack_guard_placement(vm_flags_t vm_flags)
 {
-	if (vm_flags & VM_SHADOW_STACK)
+	if (is_shadow_stack_vma(vm_flags))
 		return PAGE_SIZE;
 
 	return 0;
diff --git a/mm/vma.h b/mm/vma.h
index 819f994cf727..0f238dc37231 100644
--- a/mm/vma.h
+++ b/mm/vma.h
@@ -357,7 +357,7 @@ static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi,
 }
 
 /*
- * These three helpers classifies VMAs for virtual memory accounting.
+ * These four helpers classifies VMAs for virtual memory accounting.
  */
 
 /*
@@ -368,6 +368,11 @@ static inline bool is_exec_mapping(vm_flags_t flags)
 	return (flags & (VM_EXEC | VM_WRITE | VM_STACK)) == VM_EXEC;
 }
 
+static inline bool is_shadow_stack_vma(vm_flags_t vm_flags)
+{
+	return !!(vm_flags & VM_SHADOW_STACK);
+}
+
 /*
  * Stack area (including shadow stacks)
  *
@@ -376,7 +381,7 @@ static inline bool is_exec_mapping(vm_flags_t flags)
  */
 static inline bool is_stack_mapping(vm_flags_t flags)
 {
-	return ((flags & VM_STACK) == VM_STACK) || (flags & VM_SHADOW_STACK);
+	return ((flags & VM_STACK) == VM_STACK) || is_shadow_stack_vma(flags);
 }
 
 /*
@@ -387,7 +392,6 @@ static inline bool is_data_mapping(vm_flags_t flags)
 	return (flags & (VM_WRITE | VM_SHARED | VM_STACK)) == VM_WRITE;
 }
 
-
 static inline void vma_iter_config(struct vma_iterator *vmi,
 		unsigned long index, unsigned long last)
 {

-- 
2.34.1

Re: [PATCH RFC/RFT v2 1/2] mm: helper `is_shadow_stack_vma` to check shadow stack vma

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

On Wed, Oct 16, 2024 at 02:57:33PM -0700, Deepak Gupta wrote:
> VM_SHADOW_STACK (alias to VM_HIGH_ARCH_5) is used to encode shadow stack
> VMA on three architectures (x86 shadow stack, arm GCS and RISC-V shadow
> stack). In case architecture doesn't implement shadow stack, it's VM_NONE
> Introducing a helper `is_shadow_stack_vma` to determine shadow stack vma
> or not.

Reviewed-by: Mark Brown <broonie@kernel.org>

though

> @@ -387,7 +392,6 @@ static inline bool is_data_mapping(vm_flags_t flags)
>  	return (flags & (VM_WRITE | VM_SHARED | VM_STACK)) == VM_WRITE;
>  }
>  
> -
>  static inline void vma_iter_config(struct vma_iterator *vmi,
>  		unsigned long index, unsigned long last)
>  {
> 

Unrelated (but reasonable) whitespace change.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH RFC/RFT v2 1/2] mm: helper `is_shadow_stack_vma` to check shadow stack vma

[Deepak Gupta]

On Thu, Oct 17, 2024 at 12:22:02PM +0100, Mark Brown wrote:
>On Wed, Oct 16, 2024 at 02:57:33PM -0700, Deepak Gupta wrote:
>> VM_SHADOW_STACK (alias to VM_HIGH_ARCH_5) is used to encode shadow stack
>> VMA on three architectures (x86 shadow stack, arm GCS and RISC-V shadow
>> stack). In case architecture doesn't implement shadow stack, it's VM_NONE
>> Introducing a helper `is_shadow_stack_vma` to determine shadow stack vma
>> or not.
>
>Reviewed-by: Mark Brown <broonie@kernel.org>
>
>though
>
>> @@ -387,7 +392,6 @@ static inline bool is_data_mapping(vm_flags_t flags)
>>  	return (flags & (VM_WRITE | VM_SHARED | VM_STACK)) == VM_WRITE;
>>  }
>>
>> -
>>  static inline void vma_iter_config(struct vma_iterator *vmi,
>>  		unsigned long index, unsigned long last)
>>  {
>>
>
>Unrelated (but reasonable) whitespace change.

Yeah I'll put that in commit message indicating this clean up in next version.

Re: [PATCH RFC/RFT v2 1/2] mm: helper `is_shadow_stack_vma` to check shadow stack vma

[Edgecombe, Rick P]

On Wed, 2024-10-16 at 14:57 -0700, Deepak Gupta wrote:
> VM_SHADOW_STACK (alias to VM_HIGH_ARCH_5) is used to encode shadow stack
> VMA on three architectures (x86 shadow stack, arm GCS and RISC-V shadow
> stack). In case architecture doesn't implement shadow stack, it's VM_NONE
> Introducing a helper `is_shadow_stack_vma` to determine shadow stack vma
> or not.

I don't understand why we need this. IIRC was some discussion about wanting to
abstract different ways of testing for shadow stack VMAs for different
architectures since risc-v was going to do it differently. But now this says
they are all the same, so what's wrong with the open coded check?

> 
> Signed-off-by: Deepak Gupta <debug@rivosinc.com>
> ---
>  mm/gup.c  |  2 +-
>  mm/mmap.c |  2 +-
>  mm/vma.h  | 10 +++++++---
>  3 files changed, 9 insertions(+), 5 deletions(-)
> 
> diff --git a/mm/gup.c b/mm/gup.c
> index a82890b46a36..8e6e14179f6c 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -1282,7 +1282,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
>  		    !writable_file_mapping_allowed(vma, gup_flags))
>  			return -EFAULT;
>  
> -		if (!(vm_flags & VM_WRITE) || (vm_flags & VM_SHADOW_STACK)) {
> +		if (!(vm_flags & VM_WRITE) || is_shadow_stack_vma(vm_flags)) {
>  			if (!(gup_flags & FOLL_FORCE))
>  				return -EFAULT;
>  			/* hugetlb does not support FOLL_FORCE|FOLL_WRITE. */
> diff --git a/mm/mmap.c b/mm/mmap.c
> index dd4b35a25aeb..0853e6784069 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -708,7 +708,7 @@ static unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info)
>   */
>  static inline unsigned long stack_guard_placement(vm_flags_t vm_flags)
>  {
> -	if (vm_flags & VM_SHADOW_STACK)
> +	if (is_shadow_stack_vma(vm_flags))
>  		return PAGE_SIZE;
>  
>  	return 0;
> diff --git a/mm/vma.h b/mm/vma.h
> index 819f994cf727..0f238dc37231 100644
> --- a/mm/vma.h
> +++ b/mm/vma.h
> @@ -357,7 +357,7 @@ static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi,
>  }
>  
>  /*
> - * These three helpers classifies VMAs for virtual memory accounting.
> + * These four helpers classifies VMAs for virtual memory accounting.
>   */
>  
>  /*
> @@ -368,6 +368,11 @@ static inline bool is_exec_mapping(vm_flags_t flags)
>  	return (flags & (VM_EXEC | VM_WRITE | VM_STACK)) == VM_EXEC;
>  }
>  
> +static inline bool is_shadow_stack_vma(vm_flags_t vm_flags)
> +{
> +	return !!(vm_flags & VM_SHADOW_STACK);
> +}
> +
>  /*
>   * Stack area (including shadow stacks)
>   *
> @@ -376,7 +381,7 @@ static inline bool is_exec_mapping(vm_flags_t flags)
>   */
>  static inline bool is_stack_mapping(vm_flags_t flags)
>  {
> -	return ((flags & VM_STACK) == VM_STACK) || (flags & VM_SHADOW_STACK);
> +	return ((flags & VM_STACK) == VM_STACK) || is_shadow_stack_vma(flags);
>  }
>  
>  /*
> @@ -387,7 +392,6 @@ static inline bool is_data_mapping(vm_flags_t flags)
>  	return (flags & (VM_WRITE | VM_SHARED | VM_STACK)) == VM_WRITE;
>  }
>  
> -
>  static inline void vma_iter_config(struct vma_iterator *vmi,
>  		unsigned long index, unsigned long last)
>  {
> 

Re: [PATCH RFC/RFT v2 1/2] mm: helper `is_shadow_stack_vma` to check shadow stack vma

[Deepak Gupta]

On Fri, Nov 01, 2024 at 09:47:25PM +0000, Edgecombe, Rick P wrote:
>On Wed, 2024-10-16 at 14:57 -0700, Deepak Gupta wrote:
>> VM_SHADOW_STACK (alias to VM_HIGH_ARCH_5) is used to encode shadow stack
>> VMA on three architectures (x86 shadow stack, arm GCS and RISC-V shadow
>> stack). In case architecture doesn't implement shadow stack, it's VM_NONE
>> Introducing a helper `is_shadow_stack_vma` to determine shadow stack vma
>> or not.
>
>I don't understand why we need this. IIRC was some discussion about wanting to
>abstract different ways of testing for shadow stack VMAs for different
>architectures since risc-v was going to do it differently. But now this says
>they are all the same, so what's wrong with the open coded check?
>

Yeah it was born out of the need due to risc-v overloading `VM_WRITE`. Given
that's out of the door, this helper is not useful for that need anymore.

However during rebasing I saw certain vma helpers were re-organized in a new
file `vma.h` and there was helper for stack as well. So I kept it. 

I don't know if it's helpful or harmful. Let me know, I can let it go as well.

[PATCH RFC/RFT v2 2/2] kernel: converge common shadow stack flow agnostic to arch

[Deepak Gupta]

CPU assisted shadow stack are supported by x86, arm64 and risc-v. In
terms of enabling shadow stack feature for usermode code in kernel,
they have following commonalities

- Expose a user ABI (via a prctl) to allow user mode to explicitly
  ask for enabling shadow stack instead of by default enabling it.
  x86 series pre-dates arm64 or risc-v announcment of support, so it
  ended up doing a arch specific prctl instead of generic one. arm64
  and risc-v have converged on using generic prctl and each of them
  can handle it appropriatley.

- On fork or clone, shadow stack has to be COWed or not COWed depending
  on CLONE_VM was passed or not. Additionally if CLONE_VFORK was passed
  then same (parent one) shadow stack should be used.

- To create shadow stack mappings, implement `map_shadow_stack` system
  call.

This patch picks up Mark Brown's `ARCH_HAS_USER_SHADOW_STACK` config
introduction and incorproate most of the common flows between different
architectures.

On a high level, shadow stack allocation and shadow stack de-allocation
are base operations on virtual memory and common between architectures.
Similarly shadow stack setup on prctl (arch specific or otherwise) is a
common flow. Treatment of shadow stack virtual memory on `clone/fork` and
implementaiton of `map_shadow_stack` is also converged into common flow.

To implement these common flows, each architecture have arch-specific
enabling mechanism as well as arch-specific data structures in task/
thread struct. So additionally this patch tries to abstract certain
operation/helpers and allowing each architecture to have their arch_*
implementation to implement the abstractions.

Signed-off-by: Deepak Gupta <debug@rivosinc.com>
---
 arch/x86/include/asm/shstk.h           |   7 +
 arch/x86/include/uapi/asm/mman.h       |   3 -
 arch/x86/kernel/shstk.c                | 223 +++++---------------------------
 include/linux/usershstk.h              |  22 ++++
 include/uapi/asm-generic/mman-common.h |   5 +
 kernel/Makefile                        |   2 +
 kernel/usershstk.c                     | 230 +++++++++++++++++++++++++++++++++
 7 files changed, 296 insertions(+), 196 deletions(-)

diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h
index 4cb77e004615..b40c3d91538b 100644
--- a/arch/x86/include/asm/shstk.h
+++ b/arch/x86/include/asm/shstk.h
@@ -37,6 +37,13 @@ static inline int shstk_update_last_frame(unsigned long val) { return 0; }
 static inline bool shstk_is_enabled(void) { return false; }
 #endif /* CONFIG_X86_USER_SHADOW_STACK */
 
+int arch_create_shstk_token(unsigned long ssp, unsigned long *token_addr);
+bool arch_user_shstk_supported(void);
+bool arch_is_shstk_enabled(struct task_struct *task);
+void arch_set_shstk_base_size(struct task_struct *task, unsigned long base,
+			unsigned long size);
+void arch_get_shstk_base_size(struct task_struct *task, unsigned long *base,
+			unsigned long *size);
 #endif /* __ASSEMBLY__ */
 
 #endif /* _ASM_X86_SHSTK_H */
diff --git a/arch/x86/include/uapi/asm/mman.h b/arch/x86/include/uapi/asm/mman.h
index 46cdc941f958..ac1e6277212b 100644
--- a/arch/x86/include/uapi/asm/mman.h
+++ b/arch/x86/include/uapi/asm/mman.h
@@ -5,9 +5,6 @@
 #define MAP_32BIT	0x40		/* only give out 32bit addresses */
 #define MAP_ABOVE4G	0x80		/* only map above 4GB */
 
-/* Flags for map_shadow_stack(2) */
-#define SHADOW_STACK_SET_TOKEN	(1ULL << 0)	/* Set up a restore token in the shadow stack */
-
 #include <asm-generic/mman.h>
 
 #endif /* _ASM_X86_MMAN_H */
diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c
index 059685612362..d53a7efd70b5 100644
--- a/arch/x86/kernel/shstk.c
+++ b/arch/x86/kernel/shstk.c
@@ -25,6 +25,7 @@
 #include <asm/special_insns.h>
 #include <asm/fpu/api.h>
 #include <asm/prctl.h>
+#include <linux/usershstk.h>
 
 #define SS_FRAME_SIZE 8
 
@@ -43,11 +44,39 @@ static void features_clr(unsigned long features)
 	current->thread.features &= ~features;
 }
 
+bool arch_user_shstk_supported(void)
+{
+	return cpu_feature_enabled(X86_FEATURE_USER_SHSTK);
+}
+
+bool arch_is_shstk_enabled(struct task_struct *task)
+{
+	return features_enabled(ARCH_SHSTK_SHSTK);
+}
+
+void arch_set_shstk_base_size(struct task_struct *task, unsigned long base,
+			unsigned long size)
+{
+	struct thread_shstk *shstk = &task->thread.shstk;
+
+	shstk->base = base;
+	shstk->size = size;
+}
+
+void arch_get_shstk_base_size(struct task_struct *task, unsigned long *base,
+			unsigned long *size)
+{
+	struct thread_shstk *shstk = &task->thread.shstk;
+
+	*base = shstk->base;
+	*size = shstk->size;
+}
+
 /*
  * Create a restore token on the shadow stack.  A token is always 8-byte
  * and aligned to 8.
  */
-static int create_rstor_token(unsigned long ssp, unsigned long *token_addr)
+int arch_create_shstk_token(unsigned long ssp, unsigned long *token_addr)
 {
 	unsigned long addr;
 
@@ -72,88 +101,6 @@ static int create_rstor_token(unsigned long ssp, unsigned long *token_addr)
 	return 0;
 }
 
-/*
- * VM_SHADOW_STACK will have a guard page. This helps userspace protect
- * itself from attacks. The reasoning is as follows:
- *
- * The shadow stack pointer(SSP) is moved by CALL, RET, and INCSSPQ. The
- * INCSSP instruction can increment the shadow stack pointer. It is the
- * shadow stack analog of an instruction like:
- *
- *   addq $0x80, %rsp
- *
- * However, there is one important difference between an ADD on %rsp
- * and INCSSP. In addition to modifying SSP, INCSSP also reads from the
- * memory of the first and last elements that were "popped". It can be
- * thought of as acting like this:
- *
- * READ_ONCE(ssp);       // read+discard top element on stack
- * ssp += nr_to_pop * 8; // move the shadow stack
- * READ_ONCE(ssp-8);     // read+discard last popped stack element
- *
- * The maximum distance INCSSP can move the SSP is 2040 bytes, before
- * it would read the memory. Therefore a single page gap will be enough
- * to prevent any operation from shifting the SSP to an adjacent stack,
- * since it would have to land in the gap at least once, causing a
- * fault.
- */
-static unsigned long alloc_shstk(unsigned long addr, unsigned long size,
-				 unsigned long token_offset, bool set_res_tok)
-{
-	int flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G;
-	struct mm_struct *mm = current->mm;
-	unsigned long mapped_addr, unused;
-
-	if (addr)
-		flags |= MAP_FIXED_NOREPLACE;
-
-	mmap_write_lock(mm);
-	mapped_addr = do_mmap(NULL, addr, size, PROT_READ, flags,
-			      VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL);
-	mmap_write_unlock(mm);
-
-	if (!set_res_tok || IS_ERR_VALUE(mapped_addr))
-		goto out;
-
-	if (create_rstor_token(mapped_addr + token_offset, NULL)) {
-		vm_munmap(mapped_addr, size);
-		return -EINVAL;
-	}
-
-out:
-	return mapped_addr;
-}
-
-static unsigned long adjust_shstk_size(unsigned long size)
-{
-	if (size)
-		return PAGE_ALIGN(size);
-
-	return PAGE_ALIGN(min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G));
-}
-
-static void unmap_shadow_stack(u64 base, u64 size)
-{
-	int r;
-
-	r = vm_munmap(base, size);
-
-	/*
-	 * mmap_write_lock_killable() failed with -EINTR. This means
-	 * the process is about to die and have it's MM cleaned up.
-	 * This task shouldn't ever make it back to userspace. In this
-	 * case it is ok to leak a shadow stack, so just exit out.
-	 */
-	if (r == -EINTR)
-		return;
-
-	/*
-	 * For all other types of vm_munmap() failure, either the
-	 * system is out of memory or there is bug.
-	 */
-	WARN_ON_ONCE(r);
-}
-
 static int shstk_setup(void)
 {
 	struct thread_shstk *shstk = &current->thread.shstk;
@@ -191,48 +138,6 @@ void reset_thread_features(void)
 	current->thread.features_locked = 0;
 }
 
-unsigned long shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags,
-				       unsigned long stack_size)
-{
-	struct thread_shstk *shstk = &tsk->thread.shstk;
-	unsigned long addr, size;
-
-	/*
-	 * If shadow stack is not enabled on the new thread, skip any
-	 * switch to a new shadow stack.
-	 */
-	if (!features_enabled(ARCH_SHSTK_SHSTK))
-		return 0;
-
-	/*
-	 * For CLONE_VFORK the child will share the parents shadow stack.
-	 * Make sure to clear the internal tracking of the thread shadow
-	 * stack so the freeing logic run for child knows to leave it alone.
-	 */
-	if (clone_flags & CLONE_VFORK) {
-		shstk->base = 0;
-		shstk->size = 0;
-		return 0;
-	}
-
-	/*
-	 * For !CLONE_VM the child will use a copy of the parents shadow
-	 * stack.
-	 */
-	if (!(clone_flags & CLONE_VM))
-		return 0;
-
-	size = adjust_shstk_size(stack_size);
-	addr = alloc_shstk(0, size, 0, false);
-	if (IS_ERR_VALUE(addr))
-		return addr;
-
-	shstk->base = addr;
-	shstk->size = size;
-
-	return addr + size;
-}
-
 static unsigned long get_user_shstk_addr(void)
 {
 	unsigned long long ssp;
@@ -402,44 +307,6 @@ int restore_signal_shadow_stack(void)
 	return 0;
 }
 
-void shstk_free(struct task_struct *tsk)
-{
-	struct thread_shstk *shstk = &tsk->thread.shstk;
-
-	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) ||
-	    !features_enabled(ARCH_SHSTK_SHSTK))
-		return;
-
-	/*
-	 * When fork() with CLONE_VM fails, the child (tsk) already has a
-	 * shadow stack allocated, and exit_thread() calls this function to
-	 * free it.  In this case the parent (current) and the child share
-	 * the same mm struct.
-	 */
-	if (!tsk->mm || tsk->mm != current->mm)
-		return;
-
-	/*
-	 * If shstk->base is NULL, then this task is not managing its
-	 * own shadow stack (CLONE_VFORK). So skip freeing it.
-	 */
-	if (!shstk->base)
-		return;
-
-	/*
-	 * shstk->base is NULL for CLONE_VFORK child tasks, and so is
-	 * normal. But size = 0 on a shstk->base is not normal and
-	 * indicated an attempt to free the thread shadow stack twice.
-	 * Warn about it.
-	 */
-	if (WARN_ON(!shstk->size))
-		return;
-
-	unmap_shadow_stack(shstk->base, shstk->size);
-
-	shstk->size = 0;
-}
-
 static int wrss_control(bool enable)
 {
 	u64 msrval;
@@ -502,36 +369,6 @@ static int shstk_disable(void)
 	return 0;
 }
 
-SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsigned int, flags)
-{
-	bool set_tok = flags & SHADOW_STACK_SET_TOKEN;
-	unsigned long aligned_size;
-
-	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
-		return -EOPNOTSUPP;
-
-	if (flags & ~SHADOW_STACK_SET_TOKEN)
-		return -EINVAL;
-
-	/* If there isn't space for a token */
-	if (set_tok && size < 8)
-		return -ENOSPC;
-
-	if (addr && addr < SZ_4G)
-		return -ERANGE;
-
-	/*
-	 * An overflow would result in attempting to write the restore token
-	 * to the wrong location. Not catastrophic, but just return the right
-	 * error code and block it.
-	 */
-	aligned_size = PAGE_ALIGN(size);
-	if (aligned_size < size)
-		return -EOVERFLOW;
-
-	return alloc_shstk(addr, aligned_size, size, set_tok);
-}
-
 long shstk_prctl(struct task_struct *task, int option, unsigned long arg2)
 {
 	unsigned long features = arg2;
diff --git a/include/linux/usershstk.h b/include/linux/usershstk.h
new file mode 100644
index 000000000000..4ab27a1ab3f8
--- /dev/null
+++ b/include/linux/usershstk.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _SHSTK_H
+#define _SHSTK_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+
+unsigned long alloc_shstk(unsigned long addr, unsigned long size,
+				 unsigned long token_offset, bool set_res_tok);
+int create_shstk_token(unsigned long ssp, unsigned long *token_addr);
+bool user_shstk_supported(void);
+bool is_shstk_enabled(struct task_struct *task);
+void set_shstk_base_size(struct task_struct *task, unsigned long base,
+			unsigned long size);
+void get_shstk_base_size(struct task_struct *task, unsigned long *base,
+			unsigned long *size);
+unsigned long adjust_shstk_size(unsigned long size);
+void unmap_shadow_stack(u64 base, u64 size);
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _SHSTK_H */
diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
index 6ce1f1ceb432..5d6fb32fda95 100644
--- a/include/uapi/asm-generic/mman-common.h
+++ b/include/uapi/asm-generic/mman-common.h
@@ -87,4 +87,9 @@
 #define PKEY_ACCESS_MASK	(PKEY_DISABLE_ACCESS |\
 				 PKEY_DISABLE_WRITE)
 
+/* Flags for map_shadow_stack(2) */
+#define SHADOW_STACK_SET_TOKEN	(1ULL << 0)	/* Set up a restore token in the shadow stack */
+#define SHADOW_STACK_SET_MARKER (1ULL << 1) /* Set up a top of stack marker in the shadow stack */
+
+#define SHADOW_STACK_SET_MASK	(SHADOW_STACK_SET_TOKEN | SHADOW_STACK_SET_MARKER)
 #endif /* __ASM_GENERIC_MMAN_COMMON_H */
diff --git a/kernel/Makefile b/kernel/Makefile
index 87866b037fbe..1922c456b954 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -140,6 +140,8 @@ KCOV_INSTRUMENT_stackleak.o := n
 
 obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o
 
+obj-$(CONFIG_ARCH_HAS_USER_SHADOW_STACK) += usershstk.o
+
 $(obj)/configs.o: $(obj)/config_data.gz
 
 targets += config_data config_data.gz
diff --git a/kernel/usershstk.c b/kernel/usershstk.c
new file mode 100644
index 000000000000..1ebce6b768aa
--- /dev/null
+++ b/kernel/usershstk.c
@@ -0,0 +1,230 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * shstk.c - Intel shadow stack support
+ *
+ * Copyright (c) 2021, Intel Corporation.
+ * Yu-cheng Yu <yu-cheng.yu@intel.com>
+ */
+
+#include <linux/sched.h>
+#include <linux/bitops.h>
+#include <linux/types.h>
+#include <linux/mm.h>
+#include <linux/mman.h>
+#include <linux/slab.h>
+#include <linux/uaccess.h>
+#include <linux/sched/signal.h>
+#include <linux/compat.h>
+#include <linux/sizes.h>
+#include <linux/user.h>
+#include <linux/syscalls.h>
+#include <asm/shstk.h>
+#include <linux/usershstk.h>
+
+#define SHSTK_ENTRY_SIZE sizeof(void *)
+
+bool user_shstk_supported(void)
+{
+	return arch_user_shstk_supported();
+}
+
+bool is_shstk_enabled(struct task_struct *task)
+{
+	return arch_is_shstk_enabled(task);
+}
+
+void set_shstk_base_size(struct task_struct *task, unsigned long base,
+			unsigned long size)
+{
+	arch_set_shstk_base_size(task, base, size);
+}
+
+void get_shstk_base_size(struct task_struct *task, unsigned long *base,
+			unsigned long *size)
+{
+	arch_get_shstk_base_size(task, base, size);
+}
+
+int create_shstk_token(unsigned long ssp, unsigned long *token_addr)
+{
+	return arch_create_shstk_token(ssp, token_addr);
+}
+
+unsigned long adjust_shstk_size(unsigned long size)
+{
+	if (size)
+		return PAGE_ALIGN(size);
+
+	return PAGE_ALIGN(min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G));
+}
+
+void unmap_shadow_stack(u64 base, u64 size)
+{
+	int r;
+
+	r = vm_munmap(base, size);
+
+	/*
+	 * mmap_write_lock_killable() failed with -EINTR. This means
+	 * the process is about to die and have it's MM cleaned up.
+	 * This task shouldn't ever make it back to userspace. In this
+	 * case it is ok to leak a shadow stack, so just exit out.
+	 */
+	if (r == -EINTR)
+		return;
+
+	/*
+	 * For all other types of vm_munmap() failure, either the
+	 * system is out of memory or there is bug.
+	 */
+	WARN_ON_ONCE(r);
+}
+
+/*
+ * allocates a fresh shadow stack mapping and if required place a shadow
+ * stack token at base
+ */
+unsigned long alloc_shstk(unsigned long addr, unsigned long size,
+				 unsigned long token_offset, bool set_res_tok)
+{
+	int flags = MAP_ANONYMOUS | MAP_PRIVATE;
+
+	flags |= IS_ENABLED(CONFIG_X86_64) ? MAP_ABOVE4G : 0;
+
+	struct mm_struct *mm = current->mm;
+	unsigned long mapped_addr, unused;
+
+	if (addr)
+		flags |= MAP_FIXED_NOREPLACE;
+
+	mmap_write_lock(mm);
+	mapped_addr = do_mmap(NULL, addr, size, PROT_READ, flags,
+			      VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL);
+	mmap_write_unlock(mm);
+
+	if (!set_res_tok || IS_ERR_VALUE(mapped_addr))
+		goto out;
+
+	if (create_shstk_token(mapped_addr + token_offset, NULL)) {
+		vm_munmap(mapped_addr, size);
+		return -EINVAL;
+	}
+
+out:
+	return mapped_addr;
+}
+
+void shstk_free(struct task_struct *tsk)
+{
+	unsigned long base, size;
+
+	if (!user_shstk_supported() ||
+	    !is_shstk_enabled(current))
+		return;
+
+	/*
+	 * When fork() with CLONE_VM fails, the child (tsk) already has a
+	 * shadow stack allocated, and exit_thread() calls this function to
+	 * free it.  In this case the parent (current) and the child share
+	 * the same mm struct.
+	 */
+	if (!tsk->mm || tsk->mm != current->mm)
+		return;
+
+	get_shstk_base_size(tsk, &base, &size);
+	/*
+	 * If shstk->base is NULL, then this task is not managing its
+	 * own shadow stack (CLONE_VFORK). So skip freeing it.
+	 */
+	if (!base)
+		return;
+
+	/*
+	 * shstk->base is NULL for CLONE_VFORK child tasks, and so is
+	 * normal. But size = 0 on a shstk->base is not normal and
+	 * indicated an attempt to free the thread shadow stack twice.
+	 * Warn about it.
+	 */
+	if (WARN_ON(!size))
+		return;
+
+	unmap_shadow_stack(base, size);
+
+	set_shstk_base_size(tsk, 0, 0);
+}
+
+SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsigned int, flags)
+{
+	bool set_tok = flags & SHADOW_STACK_SET_TOKEN;
+	unsigned long aligned_size;
+
+	if (!user_shstk_supported())
+		return -EOPNOTSUPP;
+
+	if (flags & ~SHADOW_STACK_SET_MASK)
+		return -EINVAL;
+
+	/* If there isn't space for a token */
+	if (set_tok && size < SHSTK_ENTRY_SIZE)
+		return -ENOSPC;
+
+	if (addr && (addr & (PAGE_SIZE - 1)))
+		return -EINVAL;
+
+	if (IS_ENABLED(CONFIG_X86_64) &&
+		addr && addr < SZ_4G)
+		return -ERANGE;
+
+	/*
+	 * An overflow would result in attempting to write the restore token
+	 * to the wrong location. Not catastrophic, but just return the right
+	 * error code and block it.
+	 */
+	aligned_size = PAGE_ALIGN(size);
+	if (aligned_size < size)
+		return -EOVERFLOW;
+
+	return alloc_shstk(addr, aligned_size, size, set_tok);
+}
+
+unsigned long shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags,
+				       unsigned long stack_size)
+{
+	unsigned long addr, size;
+
+	if (!user_shstk_supported())
+		return -EOPNOTSUPP;
+
+	/*
+	 * If shadow stack is not enabled on the new thread, skip any
+	 * switch to a new shadow stack.
+	 */
+	if (!is_shstk_enabled(tsk))
+		return 0;
+
+	/*
+	 * For CLONE_VFORK the child will share the parents shadow stack.
+	 * Make sure to clear the internal tracking of the thread shadow
+	 * stack so the freeing logic run for child knows to leave it alone.
+	 */
+	if (clone_flags & CLONE_VFORK) {
+		set_shstk_base_size(tsk, 0, 0);
+		return 0;
+	}
+
+	/*
+	 * For !CLONE_VM the child will use a copy of the parents shadow
+	 * stack.
+	 */
+	if (!(clone_flags & CLONE_VM))
+		return 0;
+
+	size = adjust_shstk_size(stack_size);
+	addr = alloc_shstk(0, size, 0, false);
+	if (IS_ERR_VALUE(addr))
+		return addr;
+
+	set_shstk_base_size(tsk, addr, size);
+
+	return addr + size;
+}

-- 
2.34.1

Re: [PATCH RFC/RFT v2 2/2] kernel: converge common shadow stack flow agnostic to arch

[Edgecombe, Rick P]

On Wed, 2024-10-16 at 14:57 -0700, Deepak Gupta wrote:
> -/*
> - * VM_SHADOW_STACK will have a guard page. This helps userspace protect
> - * itself from attacks. The reasoning is as follows:
> - *
> - * The shadow stack pointer(SSP) is moved by CALL, RET, and INCSSPQ. The
> - * INCSSP instruction can increment the shadow stack pointer. It is the
> - * shadow stack analog of an instruction like:
> - *
> - *   addq $0x80, %rsp
> - *
> - * However, there is one important difference between an ADD on %rsp
> - * and INCSSP. In addition to modifying SSP, INCSSP also reads from the
> - * memory of the first and last elements that were "popped". It can be
> - * thought of as acting like this:
> - *
> - * READ_ONCE(ssp);       // read+discard top element on stack
> - * ssp += nr_to_pop * 8; // move the shadow stack
> - * READ_ONCE(ssp-8);     // read+discard last popped stack element
> - *
> - * The maximum distance INCSSP can move the SSP is 2040 bytes, before
> - * it would read the memory. Therefore a single page gap will be enough
> - * to prevent any operation from shifting the SSP to an adjacent stack,
> - * since it would have to land in the gap at least once, causing a
> - * fault.
> - */

I want to take a deeper look at this series once I can apply and test it, but
can we maybe make this comment more generic and keep it? I think it is similar
reasoning for arm (?), is there anything situation like this for risc-v? Or
rather, why does risc-v have the guard gaps?

Re: [PATCH RFC/RFT v2 2/2] kernel: converge common shadow stack flow agnostic to arch

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

On Fri, Nov 01, 2024 at 09:50:27PM +0000, Edgecombe, Rick P wrote:
> On Wed, 2024-10-16 at 14:57 -0700, Deepak Gupta wrote:

> > - * The maximum distance INCSSP can move the SSP is 2040 bytes, before
> > - * it would read the memory. Therefore a single page gap will be enough
> > - * to prevent any operation from shifting the SSP to an adjacent stack,
> > - * since it would have to land in the gap at least once, causing a
> > - * fault.

> I want to take a deeper look at this series once I can apply and test it, but
> can we maybe make this comment more generic and keep it? I think it is similar
> reasoning for arm (?), is there anything situation like this for risc-v? Or
> rather, why does risc-v have the guard gaps?

Yes, for arm64 you can only move the pointer in single frames so a
single page is enough.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH RFC/RFT v2 2/2] kernel: converge common shadow stack flow agnostic to arch

[Deepak Gupta]

On Fri, Nov 01, 2024 at 10:39:15PM +0000, Mark Brown wrote:
>On Fri, Nov 01, 2024 at 09:50:27PM +0000, Edgecombe, Rick P wrote:
>> On Wed, 2024-10-16 at 14:57 -0700, Deepak Gupta wrote:
>
>> > - * The maximum distance INCSSP can move the SSP is 2040 bytes, before
>> > - * it would read the memory. Therefore a single page gap will be enough
>> > - * to prevent any operation from shifting the SSP to an adjacent stack,
>> > - * since it would have to land in the gap at least once, causing a
>> > - * fault.
>
>> I want to take a deeper look at this series once I can apply and test it, but
>> can we maybe make this comment more generic and keep it? I think it is similar
>> reasoning for arm (?), is there anything situation like this for risc-v? Or
>> rather, why does risc-v have the guard gaps?
>
>Yes, for arm64 you can only move the pointer in single frames so a
>single page is enough.

Yeah on risc-v as well guard gap is expected and single page is enough.

I removed this comment from here because of x86 specifics. I can make it
generic, do you think it belongs here or the place where we define
VM_SHADOW_STACK?

Re: [PATCH RFC/RFT v2 2/2] kernel: converge common shadow stack flow agnostic to arch

[Edgecombe, Rick P]

On Thu, 2024-11-14 at 15:30 -0800, Deepak Gupta wrote:
> On Fri, Nov 01, 2024 at 10:39:15PM +0000, Mark Brown wrote:
> > On Fri, Nov 01, 2024 at 09:50:27PM +0000, Edgecombe, Rick P wrote:
> > > On Wed, 2024-10-16 at 14:57 -0700, Deepak Gupta wrote:
> > 
> > > > - * The maximum distance INCSSP can move the SSP is 2040 bytes, before
> > > > - * it would read the memory. Therefore a single page gap will be enough
> > > > - * to prevent any operation from shifting the SSP to an adjacent stack,
> > > > - * since it would have to land in the gap at least once, causing a
> > > > - * fault.
> > 
> > > I want to take a deeper look at this series once I can apply and test it, but
> > > can we maybe make this comment more generic and keep it? I think it is similar
> > > reasoning for arm (?), is there anything situation like this for risc-v? Or
> > > rather, why does risc-v have the guard gaps?
> > 
> > Yes, for arm64 you can only move the pointer in single frames so a
> > single page is enough.
> 
> Yeah on risc-v as well guard gap is expected and single page is enough.
> 
> I removed this comment from here because of x86 specifics. I can make it
> generic, do you think it belongs here or the place where we define
> VM_SHADOW_STACK?

I think near VM_SHADOW_STACK actually, good idea. IIRC it got moved from
VM_SHADOW_STACK because it was too x86 specific. So if it's generic I think that
would fit.

Re: [PATCH RFC/RFT v2 0/2] Converge common flows for cpu assisted shadow stack

[Edgecombe, Rick P]

On Wed, 2024-10-16 at 14:57 -0700, Deepak Gupta wrote:
> ---
> base-commit: 4e0105ad0161b4262b51f034a757c4899c647487
> change-id: 20241010-shstk_converge-aefbcbef5d71

Where can I find this base commit?

Re: [PATCH RFC/RFT v2 0/2] Converge common flows for cpu assisted shadow stack

[Deepak Gupta]

On Fri, Nov 01, 2024 at 09:47:31PM +0000, Edgecombe, Rick P wrote:
>On Wed, 2024-10-16 at 14:57 -0700, Deepak Gupta wrote:
>> ---
>> base-commit: 4e0105ad0161b4262b51f034a757c4899c647487
>> change-id: 20241010-shstk_converge-aefbcbef5d71
>
>Where can I find this base commit?

I am sorry. I picked up Mark's "mm: Introduce ARCH_HAS_USER_SHADOW_STACK"
locally and then created patches.

Should have rebased with arm64/for-next.
But for that as well base commit will be on arm64/for-next.

You can apply "mm: Introduce ARCH_HAS_USER_SHADOW_STACK" on "v6.12-rc1"
and then these patches.

Alternatively I can send a v3 with above patch.

Re: [PATCH RFC/RFT v2 0/2] Converge common flows for cpu assisted shadow stack

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 316 bytes --]

On Thu, Nov 14, 2024 at 03:15:41PM -0800, Deepak Gupta wrote:

> Alternatively I can send a v3 with above patch.

I guess at this point it's probably as well to just rebase onto
v6.13-rc1 when that appears, that should have the GCS series in it and
it's probably worth rebasing/resending when that comes out anyway.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]