From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D1BFC7EE43 for ; Tue, 13 Jun 2023 00:13:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CD4A8E001D; Mon, 12 Jun 2023 20:12:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 757968E000B; Mon, 12 Jun 2023 20:12:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50C038E001E; Mon, 12 Jun 2023 20:12:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 385718E001D for ; Mon, 12 Jun 2023 20:12:36 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0D073A035D for ; Tue, 13 Jun 2023 00:12:36 +0000 (UTC) X-FDA: 80895798312.21.C6B54AA Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf01.hostedemail.com (Postfix) with ESMTP id D403340011 for ; Tue, 13 Jun 2023 00:12:33 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=nUkU1rNg; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf01.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686615154; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p3jyjKsRiT5U7d0D/fXsaKNoPvKBDHZ161THvSejxrY=; b=FWph9zpaWwrMZLnYVogN3AYlra/L7LEpD6DO26b96Cc156AhOmb1okKDHwgMPQIHgArTjs 9rZoKlT57TCbf5y7kCP/Up032UUfoqcJxU8WFmC8ZoXrtn9inqkE/226pLQpXUvEcj+eB0 YAB0DNnSW2uYGXSjqwEknVE7l0QQQ30= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=nUkU1rNg; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf01.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686615154; a=rsa-sha256; cv=none; b=Qseso594HVIUGxY3HiBP53gwKCAO1BEN8bs2iIJo0a3Prb3Y5TIh88K29mYt/gTFTUXzGU BdrXsGwfzpt2t7kCKcOBAiVS9fyCyHtQ/j37PA+dzBJsUKmR3d7/fxwULwMZlqAqoqTvbo rkyZgoSSYHuJ5Lpe948zUsjN5T0nSVo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686615154; x=1718151154; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BugAl8HBv2gDjezo8BApRaTr6a0cYuyYBztBA+16s0k=; b=nUkU1rNgXwNiW5Mom6Xq9Dm0YRHaa9aSa3/XDmV1LxCAwE8SnapZg2mS MRZrorD7WfvUr+otvmZpJ0PRtz9AMiU+2qTxdDZIzIKgZB3kfRn26AqK6 LImrhF6KAC+0Fw4fsGyde5Z6mB0PR16lCRORhTMxTe7KZUK7+7V09ifM8 MDZf2nxoRGoEZv3u1q0Eh6F4Iy5lB8ozcx+r6bVcaOT/qCaqzc1X3KVof 3SE+m9KxVk7Ny1pioDuKLOd0asMyojiCOlz6l8A7WHt7asLjVGGrFspjq rROb8qEbHEorg/68eJmt8crmsMardpwPAvxbd6KTYgDQ4fHa6gbiwQ2Zw Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10739"; a="361557326" X-IronPort-AV: E=Sophos;i="6.00,238,1681196400"; d="scan'208";a="361557326" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jun 2023 17:12:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10739"; a="835671093" X-IronPort-AV: E=Sophos;i="6.00,238,1681196400"; d="scan'208";a="835671093" Received: from almeisch-mobl1.amr.corp.intel.com (HELO rpedgeco-desk4.amr.corp.intel.com) ([10.209.42.242]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jun 2023 17:12:31 -0700 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com, szabolcs.nagy@arm.com, torvalds@linux-foundation.org, broonie@kernel.org Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu , Pengfei Xu Subject: [PATCH v9 28/42] x86/shstk: Add user-mode shadow stack support Date: Mon, 12 Jun 2023 17:10:54 -0700 Message-Id: <20230613001108.3040476-29-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230613001108.3040476-1-rick.p.edgecombe@intel.com> References: <20230613001108.3040476-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: D403340011 X-Stat-Signature: updpszdiq4c3or1y36fkjat7dgbu73s6 X-Rspam-User: X-HE-Tag: 1686615153-455673 X-HE-Meta: U2FsdGVkX1/eSKWBug5NehUbV0sd2AWumGE+B/eeSnvV0OUd/chrZDf/ABHOLe2K8YgsAK26AVVEe77J+wIz9/7noE+h8SSSK0GmV2/AKcFPNc0wCceCYQXBK/cHn1IEcGl/nhhUJ19aKWetZcqmzXfMLGmAyuro3ENU+3PezeTcykQNkzVKaqApdRAtUR1xRpOY0gf57erzIgHU6bMaEwieCt2dfbw+WJSS/xBTDiPQ44ideb7iyT4ig8DsFOAYaI6rsLsLBrXr/LR2qfM7jmOzGtmAJXychxSSsNrBjXP5AfRztYF2Ei3g/qEtpnQkC7/cOBNT7yiTERB8FllwDmTvrzQCUeEAPJdOHHXWwUYnDMG3Nv1Dde7kIS3uiwqZ7Hqas03/c/QATt93f2YB5q+ruU3yEX54ZXXIrRtOZ0D4pZ3aAC8695wvhFDe9WuSDCFvT7x6cjYyvtok5m8BKH7RChZT0/TIojDzqPE0WsWHb//Zpqg88e/vdo64EEQcifS+nKDLjjQ+kHmBPpkiTaq/0Z5kilQ0yl/oOCI+iE5vDBsEWXdvsThznQcYBl0WiT9fJx2HoG4zkmgZUjPpRgQNnJ7xlJ3yC4U6BXHt/w4O0VLO3z8z7gPzO53CYpF0cwdYDOyTOWeMlf7/ONbdAk4s6YgMLiCwkdV9wjdSfOjSMFBx9j75v3leRGUyu2JQ1gQAf5YG/A9p7FXwcZfvAX9DxIdGlmKpCLhBcWjfommz33mbxKRY1mmCl/5+Xggyqt/HbU3XysdWW7X0OGTJendS5hz6Fb9PAIzpxrK0sEi/UJIrVvqxbZANIP9tc+1Svg/L+dpiPHMHTz/Tb0Hw7DAev3h35Qay7Of8gZKhV/SExlbL1LTGP37DuXa6BrC8u8bQU4Nu0oSld8YNX0AzpSOQvKBGSvKVi9DA++Xlx7PfywjYFbfOUYR6LVvj32ZsPVIedLmUyQjdS7D0EcW Ctn4mD7S b5I680aIQWjA5QK4xbHoH5qaytAmip997c6O3ULrFEaCssjTKku4zWLMEoxi2lDHnOnunSo+FZ6YF+bZvBkrV2sHfd4xOtkRoAmPtN2RcpZhOYoOwvk+K+GymIchqo8jKcL9bPdnSJ8n0ue112ueJWKqmbF7PVgh59e/OoWzBjVkqB6pOwe9gOKwRLIC+kJGuEic+r4KhUh+flPPTqeyKbKj1qd6Fi9icJWHjtvzSFLaezEDmqDoqPHmwnUAeEtnava5H6Fd8C5hYsn2GR3jCGVtOIw+jEgP3mKYfalG5nEZpX+DtP6UkpDuGCWSX3OggtA4Zz0RaSx/qkNNTZDVTXV3Td0WzbOtOIMRQLvZvCoVjqAArOV3TdJURLA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag and has a fixed size of min(RLIMIT_STACK, 4GB). Keep the task's shadow stack address and size in thread_struct. This will be copied when cloning new threads, but needs to be cleared during exec, so add a function to do this. 32 bit shadow stack is not expected to have many users and it will complicate the signal implementation. So do not support IA32 emulation or x32. Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe Reviewed-by: Borislav Petkov (AMD) Reviewed-by: Kees Cook Acked-by: Mike Rapoport (IBM) Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook --- arch/x86/include/asm/processor.h | 2 + arch/x86/include/asm/shstk.h | 7 ++ arch/x86/include/uapi/asm/prctl.h | 3 + arch/x86/kernel/shstk.c | 145 ++++++++++++++++++++++++++++++ 4 files changed, 157 insertions(+) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 407d5551b6a7..2a5ec5750ba7 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -479,6 +479,8 @@ struct thread_struct { #ifdef CONFIG_X86_USER_SHADOW_STACK unsigned long features; unsigned long features_locked; + + struct thread_shstk shstk; #endif /* Floating point and extended processor state */ diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h index ec753809f074..2b1f7c9b9995 100644 --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -8,12 +8,19 @@ struct task_struct; #ifdef CONFIG_X86_USER_SHADOW_STACK +struct thread_shstk { + u64 base; + u64 size; +}; + long shstk_prctl(struct task_struct *task, int option, unsigned long features); void reset_thread_features(void); +void shstk_free(struct task_struct *p); #else static inline long shstk_prctl(struct task_struct *task, int option, unsigned long arg2) { return -EINVAL; } static inline void reset_thread_features(void) {} +static inline void shstk_free(struct task_struct *p) {} #endif /* CONFIG_X86_USER_SHADOW_STACK */ #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 1cd44ecc9ce0..6a8e0e1bff4a 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -34,4 +34,7 @@ #define ARCH_SHSTK_DISABLE 0x5002 #define ARCH_SHSTK_LOCK 0x5003 +/* ARCH_SHSTK_ features bits */ +#define ARCH_SHSTK_SHSTK (1ULL << 0) + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 41ed6552e0a5..3cb85224d856 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -8,14 +8,159 @@ #include #include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include +static bool features_enabled(unsigned long features) +{ + return current->thread.features & features; +} + +static void features_set(unsigned long features) +{ + current->thread.features |= features; +} + +static void features_clr(unsigned long features) +{ + current->thread.features &= ~features; +} + +static unsigned long alloc_shstk(unsigned long size) +{ + int flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G; + struct mm_struct *mm = current->mm; + unsigned long addr, unused; + + mmap_write_lock(mm); + addr = do_mmap(NULL, addr, size, PROT_READ, flags, + VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); + + mmap_write_unlock(mm); + + return addr; +} + +static unsigned long adjust_shstk_size(unsigned long size) +{ + if (size) + return PAGE_ALIGN(size); + + return PAGE_ALIGN(min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G)); +} + +static void unmap_shadow_stack(u64 base, u64 size) +{ + while (1) { + int r; + + r = vm_munmap(base, size); + + /* + * vm_munmap() returns -EINTR when mmap_lock is held by + * something else, and that lock should not be held for a + * long time. Retry it for the case. + */ + if (r == -EINTR) { + cond_resched(); + continue; + } + + /* + * For all other types of vm_munmap() failure, either the + * system is out of memory or there is bug. + */ + WARN_ON_ONCE(r); + break; + } +} + +static int shstk_setup(void) +{ + struct thread_shstk *shstk = ¤t->thread.shstk; + unsigned long addr, size; + + /* Already enabled */ + if (features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + /* Also not supported for 32 bit and x32 */ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || in_32bit_syscall()) + return -EOPNOTSUPP; + + size = adjust_shstk_size(0); + addr = alloc_shstk(size); + if (IS_ERR_VALUE(addr)) + return PTR_ERR((void *)addr); + + fpregs_lock_and_load(); + wrmsrl(MSR_IA32_PL3_SSP, addr + size); + wrmsrl(MSR_IA32_U_CET, CET_SHSTK_EN); + fpregs_unlock(); + + shstk->base = addr; + shstk->size = size; + features_set(ARCH_SHSTK_SHSTK); + + return 0; +} + void reset_thread_features(void) { + memset(¤t->thread.shstk, 0, sizeof(struct thread_shstk)); current->thread.features = 0; current->thread.features_locked = 0; } +void shstk_free(struct task_struct *tsk) +{ + struct thread_shstk *shstk = &tsk->thread.shstk; + + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || + !features_enabled(ARCH_SHSTK_SHSTK)) + return; + + if (!tsk->mm) + return; + + unmap_shadow_stack(shstk->base, shstk->size); +} + +static int shstk_disable(void) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return -EOPNOTSUPP; + + /* Already disabled? */ + if (!features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + fpregs_lock_and_load(); + /* Disable WRSS too when disabling shadow stack */ + wrmsrl(MSR_IA32_U_CET, 0); + wrmsrl(MSR_IA32_PL3_SSP, 0); + fpregs_unlock(); + + shstk_free(current); + features_clr(ARCH_SHSTK_SHSTK); + + return 0; +} + long shstk_prctl(struct task_struct *task, int option, unsigned long features) { if (option == ARCH_SHSTK_LOCK) { -- 2.34.1