From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 542E4C46467 for ; Thu, 19 Jan 2023 21:24:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F4C4280002; Thu, 19 Jan 2023 16:24:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A532280001; Thu, 19 Jan 2023 16:24:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1408280002; Thu, 19 Jan 2023 16:24:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D9C4F280001 for ; Thu, 19 Jan 2023 16:24:07 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C3922A033A for ; Thu, 19 Jan 2023 21:24:07 +0000 (UTC) X-FDA: 80372826534.05.7792231 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf27.hostedemail.com (Postfix) with ESMTP id BD0AF40007 for ; Thu, 19 Jan 2023 21:24:05 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="hSM//sq/"; spf=pass (imf27.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674163446; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=XA8Iioxyvl6Xy5Wa5VtmFxF3mszcNtBcOHPIPzhymEA=; b=nmymhWdO69s1deRXSLJQIqmiSKOwaX9caw+wJ7a7mGxeNXfvclhMheLFhAk2elKXr5u+rQ V/mIflYZMTqiJUlrE9FdvBH8PdNaFteRKUUP4dqX4ti9CtR4UZFA7szQ2X7JrCMeoZFBGg hEoYjiHn5NJ5WpzhVQL1+vfQNL19geI= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="hSM//sq/"; spf=pass (imf27.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674163446; a=rsa-sha256; cv=none; b=JcQU5il6NqNzQexrGeLqy5y1MsuvpNnFLpTB5N/OB898E++LlB8relb051H7b126qPNyXW dmFXAtH2Q8WpT/a5frktLc80cefBUN9lp7TToF1Izx/HkRSRniCGZzkqcgFBonZqCD3it1 F14hFfDn+gGlmYVfgcyzgw/HFABB7aM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163445; x=1705699445; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=7r4JBhnxBM6qBcyW6ndYNR0sg/Hfe6OTwhDLlSBw690=; b=hSM//sq/0NC/6Isd1k30Cjr+uCueMcWxGLH4qq83yhAeT8Qi3Q2VclHw DpA37VqRG4wadKd8VC7bktLhrnOfKvLaI1FyRfz6Yzqp8dIamthzP6+DN Gy3DbhEN5PPizo+OQwEMrvxWZwYWLFCZU5o5VCjOXjz41vPWsaHlMEs1z kQMENXuYAQAoGmx7kERojpP6nrOssXcvvkoBIaR8jAQFV3xIKOVInIkn+ 1lUSs+Pxeurc0PI5HOi4nDheNG1cLm7YW1wY01Z1YRTz9+wf5bqfRd+qd FEh9DjckH4tNhV6n082jUWAD0nLyN+a3hEwSY5g6XDUDK0Hy/mgfcrNwL g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119776" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119776" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:04 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139124" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139124" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:03 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 25/39] mm: Warn on shadow stack memory in wrong vma Date: Thu, 19 Jan 2023 13:23:03 -0800 Message-Id: <20230119212317.8324-26-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: BD0AF40007 X-Stat-Signature: oyywbeu5widcur8pfz6nb8ykmfi6ti89 X-Rspam-User: X-HE-Tag: 1674163445-809673 X-HE-Meta: U2FsdGVkX19UMPyoGHW/HFdbLodNYrqe1tYLvdi1t900YYKXlmaWwmvPawmdeHuh1GtscH64MQeKbm6CqfJOeDfyRU9cjkkW91X6PxSRHyCq07OYpbZPLkAwK56Iwako2IMq5+qIBzZFZUV0MhAU1uR6Ng5ChEPcpC5rIAydC7mm5m+0BV5XNC+sp+xS7GIXPpC1IfiZfYxOkgm2JflRl8gpJgSCcvs4f/kTCmzfw0qFoR/R0B//6mFQMdi28F4yKxZIeHDhr3rG8wKM6H874YXL+VB+IVRB3/McXRNZW6yvP5sWQcYCbgHqSqtddpnfCdodRSxtkfk3a/RoEYHNKnFWYbwK62HxxzVbkFPxFlT3Oqch5WWAIWVFYs5u5jkcXQfW9mN0djuC5y5axOrMLP5jZ4DMxILq68PFdi2Bd+fV05PmJZzgCqAAqw9SqXPplOGPr+XwJWiKVWLdOHEkSn3gNP0Ho5gH6I3OuM+RVz4OSQ1iCiqsyY6tT+b1Kj9oMFYWMgHkZUV97bqE6HlqcfwDTBpuPzdyBPppvNkt5CKgTu4rxbxAvearTyrLsWvtvvx+RxZWC1aYhUfh/2hOpvtOYwgxVK11V+qRrFB1CS6y/PGLd+nKRF9jVu+NC8clm5nD+EEojxcm8o5Y+aZ4M0ABI6ZsNNfvNUueZLWHpKqkeEMAYnSrHE6TJMjrbElYIrCIPSDNklxYzaeb2MdffFmi+mM0zuaG/Hq6Wo0ruSHW4ijQB6HydcaILCR4IG7QcmFoZsIe+FlAPIyMiNm9T7cxQOsAxR6wMD3y2Eqw/Y50VV8IY1wAmP1G6LsWFY/dZoPHt1SR4E30qdIQSd2GkWxkvymOf9C4YuVzcOZRbQtJgaUuBNWP66C+53ximv9pRKABTpXysuGTrun6hz2GX6omIGsREHrMF75wQQPmZXhV0A2/2cc1x/+cd3cOQRTJp3uzxfSAE7GqcnIncVq Sw638ES2 8m3/pk9h0iVgpdr08JmfhWrFX9drEeT7c0Ori754i/BUHDyXZjwJ6M045Q4eEc5Ilii7jvdh0K83T3IxUFmO6COt5ZLAQvxMjZsw9xl0wy97rLXv0J2RCGVZD43lz1WzYjuUjFqglHLm75GZDpnACY+XiB0dE0TvA99FPION9CeRc06/lYIrRgmI9AHtxZbc8cM9ibneLhuKl4DgC4CBzvXlW1HbhYGNjxCzt6YuYX2Q1GR8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by the kernel on x86. Previous patches have changed the kernel to now avoid creating these PTEs unless they are for shadow stack memory. In case any missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic, warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow stack VMAs when they are being zapped. This won't catch transient cases but should have decent coverage. It will be compiled out when shadow stack is not configured. In order to check if a pte is shadow stack in core mm code, add default implementations for pte_shstk() and pmd_shstk(). Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Rick Edgecombe --- v5: - Fix typo in commit log v3: - New patch arch/x86/include/asm/pgtable.h | 2 ++ include/linux/pgtable.h | 14 ++++++++++++++ mm/huge_memory.c | 2 ++ mm/memory.c | 2 ++ 4 files changed, 20 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 425ded5dd6ec..356f1d43e403 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -129,6 +129,7 @@ static inline bool pte_dirty(pte_t pte) return pte_flags(pte) & _PAGE_DIRTY_BITS; } +#define pte_shstk pte_shstk static inline bool pte_shstk(pte_t pte) { if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) @@ -147,6 +148,7 @@ static inline bool pmd_dirty(pmd_t pmd) return pmd_flags(pmd) & _PAGE_DIRTY_BITS; } +#define pmd_shstk pmd_shstk static inline bool pmd_shstk(pmd_t pmd) { if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 49ce1f055242..04d0bc466e43 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -539,6 +539,20 @@ static inline pte_t pte_mkwrite_shstk(pte_t pte) } #endif +#ifndef pte_shstk +static inline bool pte_shstk(pte_t pte) +{ + return false; +} +#endif + +#ifndef pmd_shstk +static inline bool pmd_shstk(pmd_t pte) +{ + return false; +} +#endif + #ifndef pmd_mkwrite_shstk static inline pmd_t pmd_mkwrite_shstk(pmd_t pmd) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index fbb8beb9265e..5bd71da75dec 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1700,6 +1700,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, */ orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd, tlb->fullmm); + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pmd_shstk(orig_pmd)); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); if (vma_is_special_huge(vma)) { if (arch_needs_pgtable_deposit()) diff --git a/mm/memory.c b/mm/memory.c index 5e5107232a26..c4cc38baffc5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1381,6 +1381,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pte_shstk(ptent)); tlb_remove_tlb_entry(tlb, pte, addr); zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); -- 2.17.1