From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7197DC54EED for ; Mon, 23 Jan 2023 22:05:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 824AC6B0098; Mon, 23 Jan 2023 17:05:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C8DF6B009B; Mon, 23 Jan 2023 17:05:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BC866B0096; Mon, 23 Jan 2023 17:05:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 27D806B0098 for ; Mon, 23 Jan 2023 17:05:38 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EB15816071C for ; Mon, 23 Jan 2023 22:05:37 +0000 (UTC) X-FDA: 80387446314.05.5EF7D84 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf29.hostedemail.com (Postfix) with ESMTP id D6BDC12000C for ; Mon, 23 Jan 2023 22:05:34 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=DUILHKFl; spf=none (imf29.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.115) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674511535; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CTJZWXrll6yca84fCeKN5uQPJXa8IESHCD+UdxuT9eY=; b=bRMzWu56seHhxp3B7o1WAnpmbImp6oW5O1n4HXoH5ID4Cd4q7oiu8DtzxgdyMaIn4cD8aV K0py0Wy5q5byVxAt4Ue2QRmH86oUCdKXupC7KNFKedLWdLrUi9z407QE73ut4RBe8G8ENq oPtQ+Sh+HiKClw+H5fZALIjAagyd+9U= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=DUILHKFl; spf=none (imf29.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.115) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674511535; a=rsa-sha256; cv=none; b=fnem6EDdI24wwGA9DunUDMcBA69vimCN9nivdHWYSt30TrnHeQnj6VBzHWQijxJWiHPaug K/4rn2FiK2h4MVaya995/26ym2C5kC4Z5z62YodA8/P1OZf/3Oj8uB5lkdiYXnrvXc8FTD O8+WgwCglD1Jfc+LypFf+QxOoLUrPao= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511534; x=1706047534; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Xffq36fUS8+sdsJCnvn6O/XkT/sivjc0AK1M915RtWI=; b=DUILHKFl/MZV2O+yaZPCph6ND4O3omtrS6qc3Lu97anMacCfVBmm22w5 ElH7iw0FxYnTUZz2EFTkB6tKDHrWNdttbppLMLn2zOHsX/VVPiLEmz1Rr sR945HRfcDHnJrDsr+LFThf1HkdKvXQCI3XJeoM8qzdpgVBSbApMx4SJO TFhMUWQ6q/EO4qunSDHwH8aAwBzxrZplmbXVbR/KIY0ykukWOwqUv7MJs BGJNwwmwJUUi8q0OgL1Cnc17uTpcmI5H1Z5dYi0Jzsz7nkSakjBfoaDx6 y4A2WZGo0yFGGXUG7QaYslgqMyhMM2lg2guMFi+njU/MjfYGSxINr/s7T g==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="326198138" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="326198138" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:14 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="694103353" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="694103353" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:07 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 21F3A10942D; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 01/17] x86/mm: Rework address range check in get_user() and put_user() Date: Tue, 24 Jan 2023 01:04:44 +0300 Message-Id: <20230123220500.21077-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D6BDC12000C X-Stat-Signature: pfbkmyuqhu5zhmhpo4bd5n1o1agzhmah X-Rspam-User: X-HE-Tag: 1674511534-783261 X-HE-Meta: U2FsdGVkX18U/khydIloJ/WnhHYXvPWBCmE2Qin0Yel0CrHR0ivJOYeCGRQCaWu59y80+gEPdS1iUJ6vnwXr+ecyL4NNgoq0rhfg3zXe3lMEJE9OBwND+lca/zjy3zbz0V8buZeQs5CKvLRVsaUrPZkkGpZVcYn8Xb0tOwSPbw9udfcK82IrHOXuG9kzj0CJhSBNlP1M5fBct6J8zqCAxsTU6fvRnoimfrSAR3s6yXu5ewnRp1/Sh7ynvVdb81RhUlmrG4v4Ba2A2Tg8iRn2N1u1aZbWyO9ffwsOaK6jxjTiwIzT1atUuszwxHYu21DK+grTgBzVpLoIoz6fr+gudKHertQcVcA8T4f8ZcLbiooYti3oD8Jz9J8QOY+X7yKDW+w4CuJKuEsKQsKuKbh0aAyI6ySpk5uYcyj1gjTexNfY5ophc+lp0j3gpXPkHm/DHfJGktll71/zl9UJ+SE+DxHp/wKXJoejYQo3BSNg70OX0m5dxO7RbyoesVAhb31wVkItXy0GwnFP/j+bbQJQHYdqXrC/dEQ8K9bjl7SCHpgpquWsBgV1UIics2vfxLUa/wAYf/fuCHJNEP9OJx3WrhK/JAPAFtdM3+RdtyklMC0kJo4ZX2SRa95fOS2tJ/jPObffRaCp0aR8AiIfN+80ZVG5j/Gwb74tjSbJ3N51hDmQkrWXOYPAWPcnx2XbI3oe/xkjO4pK1YzgiAUqAqxK8lM5bihgXixX/gmaEHmjmQfZ9u63MnoijboTDX+XnUP4ApVUAo9E3jn6Hyp+OkevYMYRkRncQo0OhOviFGR4wR3VeRNLnBjCYejk1N9X/ypPVtGXRdR05WhhMY0WoTGDODN6j9O1w2dBE4XZr6azvdeQRB1v/IkYVyUV1TOUmYzEbM4UK8cFstiytWDEST0EpJ31ijK9xk3nRtXIHEfRaZZkzC3wGmeNblPmonGRl3o9IsAfN56qXf+1h4Cpl0f zu20w8N1 4nEL1qzuh1UYsFlvFnHkdFgXnyJ2IgSgQ2RLy4p7HKCi2RyK3lx5xgCucTDH5/h2+P9eCn1JgcjZXYJb/2wVlrMuMzAgLAxv5N1FjHJjeR0fMY1bOPitY0pemzd15XlyfUYbjpDI6btWBm3LJWXYUJ+4AJRRvoXgsFu0v5/qg8GCBssJqsO2E4suRvmYKLe4nUJyrXaVsAaqgdMlCXPX6qcVo/WaIIJw04C9G4jTWS9jSllRoFF3NwYAxTPN7UHdDV8JGfPr3y50SgWs3ptsohw0omGAC8E9s3zSFFozehgm4WPau3kZnq+DgpNMC7fRfnhwauJPBx4NAMygLtIV9dtOAaYc31aDJm66AjGwfRpVas/ennZ8Q/prmYulhWdB2El62nq9WX8hPF3w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The functions get_user() and put_user() check that the target address range resides in the user space portion of the virtual address space. In order to perform this check, the functions compare the end of the range against TASK_SIZE_MAX. For kernels compiled with CONFIG_X86_5LEVEL, this process requires some additional trickery using ALTERNATIVE, as TASK_SIZE_MAX depends on the paging mode in use. Linus suggested that this check could be simplified for 64-bit kernels. It is sufficient to check bit 63 of the address to ensure that the range belongs to user space. Additionally, the use of branches can be avoided by setting the target address to all ones if bit 63 is set. There's no need to check the end of the access range as there's huge gap between end of userspace range and start of the kernel range. The gap consists of canonical hole and unused ranges on both kernel and userspace sides. If an address with bit 63 set is passed down, it will trigger a #GP exception. _ASM_EXTABLE_UA() complains about this. Replace it with plain _ASM_EXTABLE() as it is expected behaviour now. The updated get_user() and put_user() checks are also compatible with Linear Address Masking, which allows user space to encode metadata in the upper bits of pointers and eliminates the need to untag the address before handling it. Signed-off-by: Kirill A. Shutemov Suggested-by: Linus Torvalds Acked-by: Peter Zijlstra (Intel) --- arch/x86/lib/getuser.S | 83 ++++++++++++++++-------------------------- arch/x86/lib/putuser.S | 54 ++++++++++++--------------- 2 files changed, 55 insertions(+), 82 deletions(-) diff --git a/arch/x86/lib/getuser.S b/arch/x86/lib/getuser.S index b70d98d79a9d..b64a2bd1a1ef 100644 --- a/arch/x86/lib/getuser.S +++ b/arch/x86/lib/getuser.S @@ -37,22 +37,22 @@ #define ASM_BARRIER_NOSPEC ALTERNATIVE "", "lfence", X86_FEATURE_LFENCE_RDTSC -#ifdef CONFIG_X86_5LEVEL -#define LOAD_TASK_SIZE_MINUS_N(n) \ - ALTERNATIVE __stringify(mov $((1 << 47) - 4096 - (n)),%rdx), \ - __stringify(mov $((1 << 56) - 4096 - (n)),%rdx), X86_FEATURE_LA57 -#else -#define LOAD_TASK_SIZE_MINUS_N(n) \ - mov $(TASK_SIZE_MAX - (n)),%_ASM_DX -#endif +.macro check_range size:req +.if IS_ENABLED(CONFIG_X86_64) + mov %rax, %rdx + sar $63, %rdx + or %rdx, %rax +.else + cmp $TASK_SIZE_MAX-\size+1, %eax + jae .Lbad_get_user + sbb %edx, %edx /* array_index_mask_nospec() */ + and %edx, %eax +.endif +.endm .text SYM_FUNC_START(__get_user_1) - LOAD_TASK_SIZE_MINUS_N(0) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=1 ASM_STAC 1: movzbl (%_ASM_AX),%edx xor %eax,%eax @@ -62,11 +62,7 @@ SYM_FUNC_END(__get_user_1) EXPORT_SYMBOL(__get_user_1) SYM_FUNC_START(__get_user_2) - LOAD_TASK_SIZE_MINUS_N(1) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=2 ASM_STAC 2: movzwl (%_ASM_AX),%edx xor %eax,%eax @@ -76,11 +72,7 @@ SYM_FUNC_END(__get_user_2) EXPORT_SYMBOL(__get_user_2) SYM_FUNC_START(__get_user_4) - LOAD_TASK_SIZE_MINUS_N(3) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=4 ASM_STAC 3: movl (%_ASM_AX),%edx xor %eax,%eax @@ -90,30 +82,17 @@ SYM_FUNC_END(__get_user_4) EXPORT_SYMBOL(__get_user_4) SYM_FUNC_START(__get_user_8) -#ifdef CONFIG_X86_64 - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=8 ASM_STAC +#ifdef CONFIG_X86_64 4: movq (%_ASM_AX),%rdx - xor %eax,%eax - ASM_CLAC - RET #else - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user_8 - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX - ASM_STAC 4: movl (%_ASM_AX),%edx 5: movl 4(%_ASM_AX),%ecx +#endif xor %eax,%eax ASM_CLAC RET -#endif SYM_FUNC_END(__get_user_8) EXPORT_SYMBOL(__get_user_8) @@ -166,7 +145,7 @@ EXPORT_SYMBOL(__get_user_nocheck_8) SYM_CODE_START_LOCAL(.Lbad_get_user_clac) ASM_CLAC -bad_get_user: +.Lbad_get_user: xor %edx,%edx mov $(-EFAULT),%_ASM_AX RET @@ -184,23 +163,23 @@ SYM_CODE_END(.Lbad_get_user_8_clac) #endif /* get_user */ - _ASM_EXTABLE_UA(1b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(2b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(3b, .Lbad_get_user_clac) + _ASM_EXTABLE(1b, .Lbad_get_user_clac) + _ASM_EXTABLE(2b, .Lbad_get_user_clac) + _ASM_EXTABLE(3b, .Lbad_get_user_clac) #ifdef CONFIG_X86_64 - _ASM_EXTABLE_UA(4b, .Lbad_get_user_clac) + _ASM_EXTABLE(4b, .Lbad_get_user_clac) #else - _ASM_EXTABLE_UA(4b, .Lbad_get_user_8_clac) - _ASM_EXTABLE_UA(5b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(4b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(5b, .Lbad_get_user_8_clac) #endif /* __get_user */ - _ASM_EXTABLE_UA(6b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(7b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(8b, .Lbad_get_user_clac) + _ASM_EXTABLE(6b, .Lbad_get_user_clac) + _ASM_EXTABLE(7b, .Lbad_get_user_clac) + _ASM_EXTABLE(8b, .Lbad_get_user_clac) #ifdef CONFIG_X86_64 - _ASM_EXTABLE_UA(9b, .Lbad_get_user_clac) + _ASM_EXTABLE(9b, .Lbad_get_user_clac) #else - _ASM_EXTABLE_UA(9b, .Lbad_get_user_8_clac) - _ASM_EXTABLE_UA(10b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(9b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(10b, .Lbad_get_user_8_clac) #endif diff --git a/arch/x86/lib/putuser.S b/arch/x86/lib/putuser.S index 32125224fcca..3062d09a776d 100644 --- a/arch/x86/lib/putuser.S +++ b/arch/x86/lib/putuser.S @@ -33,20 +33,20 @@ * as they get called from within inline assembly. */ -#ifdef CONFIG_X86_5LEVEL -#define LOAD_TASK_SIZE_MINUS_N(n) \ - ALTERNATIVE __stringify(mov $((1 << 47) - 4096 - (n)),%rbx), \ - __stringify(mov $((1 << 56) - 4096 - (n)),%rbx), X86_FEATURE_LA57 -#else -#define LOAD_TASK_SIZE_MINUS_N(n) \ - mov $(TASK_SIZE_MAX - (n)),%_ASM_BX -#endif +.macro check_range size:req +.if IS_ENABLED(CONFIG_X86_64) + mov %rcx, %rbx + sar $63, %rbx + or %rbx, %rcx +.else + cmp $TASK_SIZE_MAX-\size+1, %ecx + jae .Lbad_put_user +.endif +.endm .text SYM_FUNC_START(__put_user_1) - LOAD_TASK_SIZE_MINUS_N(0) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=1 ASM_STAC 1: movb %al,(%_ASM_CX) xor %ecx,%ecx @@ -66,9 +66,7 @@ SYM_FUNC_END(__put_user_nocheck_1) EXPORT_SYMBOL(__put_user_nocheck_1) SYM_FUNC_START(__put_user_2) - LOAD_TASK_SIZE_MINUS_N(1) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=2 ASM_STAC 3: movw %ax,(%_ASM_CX) xor %ecx,%ecx @@ -88,9 +86,7 @@ SYM_FUNC_END(__put_user_nocheck_2) EXPORT_SYMBOL(__put_user_nocheck_2) SYM_FUNC_START(__put_user_4) - LOAD_TASK_SIZE_MINUS_N(3) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=4 ASM_STAC 5: movl %eax,(%_ASM_CX) xor %ecx,%ecx @@ -110,9 +106,7 @@ SYM_FUNC_END(__put_user_nocheck_4) EXPORT_SYMBOL(__put_user_nocheck_4) SYM_FUNC_START(__put_user_8) - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=8 ASM_STAC 7: mov %_ASM_AX,(%_ASM_CX) #ifdef CONFIG_X86_32 @@ -144,15 +138,15 @@ SYM_CODE_START_LOCAL(.Lbad_put_user_clac) RET SYM_CODE_END(.Lbad_put_user_clac) - _ASM_EXTABLE_UA(1b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(2b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(3b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(4b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(5b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(6b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(7b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(9b, .Lbad_put_user_clac) + _ASM_EXTABLE(1b, .Lbad_put_user_clac) + _ASM_EXTABLE(2b, .Lbad_put_user_clac) + _ASM_EXTABLE(3b, .Lbad_put_user_clac) + _ASM_EXTABLE(4b, .Lbad_put_user_clac) + _ASM_EXTABLE(5b, .Lbad_put_user_clac) + _ASM_EXTABLE(6b, .Lbad_put_user_clac) + _ASM_EXTABLE(7b, .Lbad_put_user_clac) + _ASM_EXTABLE(9b, .Lbad_put_user_clac) #ifdef CONFIG_X86_32 - _ASM_EXTABLE_UA(8b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(10b, .Lbad_put_user_clac) + _ASM_EXTABLE(8b, .Lbad_put_user_clac) + _ASM_EXTABLE(10b, .Lbad_put_user_clac) #endif -- 2.39.1