From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08D08C004D4 for ; Thu, 19 Jan 2023 23:07:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C8136B007E; Thu, 19 Jan 2023 18:07:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 36D756B0082; Thu, 19 Jan 2023 18:07:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 171BD6B0080; Thu, 19 Jan 2023 18:07:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 05D1D6B0074 for ; Thu, 19 Jan 2023 18:07:07 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D1D1DC0DCA for ; Thu, 19 Jan 2023 23:07:06 +0000 (UTC) X-FDA: 80373086052.04.1A9FFE2 Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by imf01.hostedemail.com (Postfix) with ESMTP id 6AF564001B for ; Thu, 19 Jan 2023 23:07:04 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=shutemov.name header.s=fm1 header.b=wmotPjFA; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=Sz4qXjKs; spf=pass (imf01.hostedemail.com: domain of kirill@shutemov.name designates 66.111.4.28 as permitted sender) smtp.mailfrom=kirill@shutemov.name; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674169624; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r++m5npef9Z2Dvz6d7fU0ciFE3OnTgWcXxCyHr666VU=; b=XS3+uAODrVwPnHLVe72RQGGZ+GnTtwDvwGimm32vZIvpqDfGvfXIIpI2HO9ExP32MiLbbS y1/58rT1LPx28Fn9w50ZKmmlr8MrCfwtVt9GBxTDqJUcihjn0Ax0zXggOY4aOnGgXchq/R 4DtgH/f9zdyo4r6KJmnYoJ4nZoWHF5o= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=shutemov.name header.s=fm1 header.b=wmotPjFA; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=Sz4qXjKs; spf=pass (imf01.hostedemail.com: domain of kirill@shutemov.name designates 66.111.4.28 as permitted sender) smtp.mailfrom=kirill@shutemov.name; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674169624; a=rsa-sha256; cv=none; b=qGMP7tvNkFMghuBkiIOg58uEOcTth8N1d6HtYnIVC89vNn+slbwy8PWLcMoSahfpUi2h/V XpcvIq0ZV0puqhCMp73/O3nIadkD/yzGP2qvBdO/7y124hkq5d64rt1T1/Vzmu7cL9uerN ZbiL8OD4TuhJorMqC4JfN8fDlothzKw= Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id DBF795C00A5; Thu, 19 Jan 2023 18:07:03 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Thu, 19 Jan 2023 18:07:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm1; t=1674169623; x=1674256023; bh=r+ +m5npef9Z2Dvz6d7fU0ciFE3OnTgWcXxCyHr666VU=; b=wmotPjFAJWqkqA21x0 cAZp2OnMhLCCfxuYtNZwsqqRJnqVp8tLaSFJ5lHIKb9lX5NmFVjdna+XwwLPiSCS JKHpVvKfkDb6mWvwt8btGty/zakCbGGFPBVVA1LG2qRyV18GYs9a4AtuU/kj9lVl RNZQC2fNX36wZWuvNHDUd4KZsp+AEti9oVV0fipolvcOUxsvI8Z1Vqjw1BU+QBs6 zgt4sA8IYEbaSTopbvNqf1p34NXD4WI1S7BzZ9xwAZB0t5uXmuwFMR9AFLDOXfq4 2gnbIcRDfL011FrGhwMHdQt/WzXtPV/GcBWGSEpOooDQ1ZAP8EE0wSvXQ7zark0x fqvw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1674169623; x=1674256023; bh=r++m5npef9Z2Dvz6d7fU0ciFE3On TgWcXxCyHr666VU=; b=Sz4qXjKs3PmsXl4+Px/iD/KqEPtr7XnnAQTl0mypVQ9H zC7/3hNR7zcP+k68FtpMsIy8xnok/4tB6zyBINRkgiTSZ1gTN1Mi0Q4/ZHK/pLZv y2ZwZINGZtfDu1dZXK+PeH+v/YXB9a9vwebs1zsl0OTtZhG/TgUOkLuQ5RODs5+y KTJRcDFCbS9lemnnabSRUNqU1GoX390gBrQOhNAqRC8/PtmdTM2PNh+ngkTemImB JJC/OJPAs2HPtXPYM2IBwlbrtDj5cXcqbLWIw7JCef5oUG1dkvIcsVx8b+pvc3B8 Sjy1de+EDj/5IfK+nRkZBr+y1Dtm/SwTqMRvc6E4ag== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrudduuddgtdejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesthdttddttddtvdenucfhrhhomhepfdfmihhr ihhllhcutedrucfuhhhuthgvmhhovhdfuceokhhirhhilhhlsehshhhuthgvmhhovhdrnh grmhgvqeenucggtffrrghtthgvrhhnpefhieeghfdtfeehtdeftdehgfehuddtvdeuheet tddtheejueekjeegueeivdektdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmh epmhgrihhlfhhrohhmpehkihhrihhllhesshhhuhhtvghmohhvrdhnrghmvg X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 19 Jan 2023 18:07:02 -0500 (EST) Received: by box.shutemov.name (Postfix, from userid 1000) id AE4F0104925; Fri, 20 Jan 2023 02:06:59 +0300 (+03) Date: Fri, 20 Jan 2023 02:06:59 +0300 From: "Kirill A. Shutemov" To: Peter Zijlstra Cc: "Kirill A. Shutemov" , Dave Hansen , Andy Lutomirski , x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sami Tolvanen , ndesaulniers@google.com, joao@overdrivepizza.com, Andrew Cooper Subject: Re: [PATCHv14 08/17] x86/mm: Reduce untagged_addr() overhead until the first LAM user Message-ID: <20230119230659.pda5jigd5qxpnpq4@box.shutemov.name> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> <20230111123736.20025-9-kirill.shutemov@linux.intel.com> <20230117135703.voaumisreld7crfb@box> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6AF564001B X-Stat-Signature: njk1qgptpbnhuryqwnp1maf7tftnnng3 X-HE-Tag: 1674169624-994635 X-HE-Meta: U2FsdGVkX1/KTD9c0Yts7JgDS3AC4uqD/WGttwQ3xrkSlrEH67nlm16C/d/MG/htlC89BfHiLsABcIOfK+70P8IGpN0aQRlqiHXVWaHEBPb6/PsN1reQOeQsPtzBcbdU7ykSIXLRvbOAMfZwXluF9yPtsehRdorpVvI0C2EwF5oZQE0RXGrUOGeI4PUTBKgEIKxY66IP9FxMXd1xexDgYNoGznGAVWt7++MK8MX4ZcYULtCUy3xmnYAebJvRDZQaeEQP7DbT2vNKWVKUgHbiY+1VYsdb7CU1RWMIrFBK4HnVYD2ifULdGiiLFQsDKVn8DHCb+F4XxWXZKIgt8Xjza1FMUVnuCPweFJepqDFFr/561/YoTT+Lj45DETTzNsecUmfU8UCvwdSt+dQuUlSHR+Hh7RqbpMwfHZbYq9bEXa9y4gSvu1OujQCTtC5JSwfRVNAjIpGHpUvOif/fxdtYASwuSgiIdrOhfgHFp/9zeIWTaMUKdCCMO8LfSlEpoRvHd4Mw2/WvuDrSvWT+uK/qanUv9q5xpAGz1IpFJT8MUpE1VCuqOez83lbSHkNRO9XaAtfl5DyyHKoiEmgqbxz2itRrbT8ozvoin26dH0nVtP0q8ZtssgLvq2twmRVffIgXhAeSr7zcntiEGnVaJfEfn2xnYygZOfk2Vq7uPSM/Sr4BCp4jVJ3bMbwsOzi4R+Ag7/iPe8cuZEQ+feYuAD7aCzjQ6diYGVSOgFSfpafi+nLfzdYc0ajunvVzNt8U5NU4HCnVOBYRkxzORS5/XCk2QiuXZkydKgzzDxFzPosUjjH+WNI55oDeYoYLhH4GwLqrmn06fQy4xhCwZXTctBDnC7MY3aUW+LEyK9hPruKrdOk2oha8GP5a98jZtVEKSaWdMkl9ZY2LxPVJ/PcnUYdfp+oWKiJ8zjqv3DJF05ds2U/0eB/gA+BLHeTrtYON711Fd5BYtWUnanW4muAUSgD kV02Yzbf LO6QEYWtRRuDMQ2zCvBMq4eBEgrkAZ0iak2YBOIQCc/xXHFvuv132zTvgYH8+B7JMWqCmnsxh01OpDfzDLpuGxE6/ge1DR6IEiBPJwC1J9kLWT5SlqNM/tYcKuRs6f9AIXDlsHgoUvGcNt25h1MMCvzbglO5zh+YEFQchd1Q9+u4OI/TfJkYwyh9AlbdLcc0yWsYdkztKTTXE1TbZW2SrFcNGgrd1wL3dL1AckzwkQzQ5VkdKeFgzy/YP2T2977eh1F+fZ4kwR8QcyoXOXoDT+eaOUQmq9FBFErwLlwpeHSOpIt+UYPuZ9uutEmGF6kzwnhPk5UqgKbJZq3orIpMAD3LvmK2GVRLcBHi94PCbcCqjeFzStSp6OVyy/8mdeU3dmgDaoZAxEUjH3F8/HiZaSP3cj2XAa6YgdUMQfcCuRlPgpuZiLe0Xz/nrTg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Jan 17, 2023 at 04:02:06PM +0100, Peter Zijlstra wrote: > On Tue, Jan 17, 2023 at 04:57:03PM +0300, Kirill A. Shutemov wrote: > > On Tue, Jan 17, 2023 at 02:05:22PM +0100, Peter Zijlstra wrote: > > > On Wed, Jan 11, 2023 at 03:37:27PM +0300, Kirill A. Shutemov wrote: > > > > > > > #define __untagged_addr(untag_mask, addr) > > > > u64 __addr = (__force u64)(addr); \ > > > > - s64 sign = (s64)__addr >> 63; \ > > > > - __addr &= untag_mask | sign; \ > > > > + if (static_branch_likely(&tagged_addr_key)) { \ > > > > + s64 sign = (s64)__addr >> 63; \ > > > > + __addr &= untag_mask | sign; \ > > > > + } \ > > > > (__force __typeof__(addr))__addr; \ > > > > }) > > > > > > > > #define untagged_addr(addr) __untagged_addr(current_untag_mask(), addr) > > > > > > Is the compiler clever enough to put the memop inside the branch? > > > > Hm. You mean current_untag_mask() inside static_branch_likely()? > > > > But it is preprocessor who does this, not compiler. So, yes, the memop is > > inside the branch. > > > > Or I didn't understand your question. > > Nah, call it a pre-lunch dip, I overlooked the whole CPP angle -- d'0h. > > That said, I did just put it through a compiler to see wth it did and it > is pretty gross: I tried to replace static branch with alternative. It kinda works, but required few hack. Thanks to Andrew Cooper for helping to untangle them. I am not sure if it worth the effort. I don't have any evidence that it helps. untagged_addr() overhead is rather small and hides in noise of syscall cost. I only made alternative for untagged_addr(), but not for untagged_addr_remote(). _remote() case has very few users. BTW, it would be nice to be able to apply alternative later, delaying it until the first user of LAM, like I did with static_branch. We don't have a way to do this right? Any opinions? I am okay dropping the patch altogether. diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index c44b56f7ffba..3f0c31044f02 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -75,6 +75,12 @@ # define DISABLE_CALL_DEPTH_TRACKING (1 << (X86_FEATURE_CALL_DEPTH & 31)) #endif +#ifdef CONFIG_ADDRESS_MASKING +# define DISABLE_LAM 0 +#else +# define DISABLE_LAM (1 << (X86_FEATURE_LAM & 31)) +#endif + #ifdef CONFIG_INTEL_IOMMU_SVM # define DISABLE_ENQCMD 0 #else @@ -115,7 +121,7 @@ #define DISABLED_MASK10 0 #define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING) -#define DISABLED_MASK12 0 +#define DISABLED_MASK12 (DISABLE_LAM) #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index f9f85d596581..57ccb91fcccf 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -24,28 +25,48 @@ static inline bool pagefault_disabled(void); #endif #ifdef CONFIG_ADDRESS_MASKING -DECLARE_STATIC_KEY_FALSE(tagged_addr_key); +static inline unsigned long __untagged_addr(unsigned long addr) +{ + /* + * Magic with the 'sign' allows to untag userspace pointer without + * any branches while leaving kernel addresses intact. + */ + long sign; + + /* + * Refer tlbstate_untag_mask directly to avoid RIP-relative relocation + * in alternative instructions. The relocation gets wrong when gets + * copied to the target place. + */ + asm (ALTERNATIVE("", + "sar $63, %[sign]\n\t" /* user_ptr ? 0 : -1UL */ + "or %%gs:tlbstate_untag_mask, %[sign]\n\t" + "and %[sign], %[addr]\n\t", X86_FEATURE_LAM) + : [addr] "+r" (addr), [sign] "=r" (sign) + : "m" (tlbstate_untag_mask), "[sign]" (addr)); + + return addr; +} -/* - * Mask out tag bits from the address. - * - * Magic with the 'sign' allows to untag userspace pointer without any branches - * while leaving kernel addresses intact. - */ -#define __untagged_addr(untag_mask, addr) ({ \ - u64 __addr = (__force u64)(addr); \ - if (static_branch_likely(&tagged_addr_key)) { \ - s64 sign = (s64)__addr >> 63; \ - __addr &= untag_mask | sign; \ - } \ - (__force __typeof__(addr))__addr; \ +#define untagged_addr(addr) ({ \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr(__addr); \ }) -#define untagged_addr(addr) __untagged_addr(current_untag_mask(), addr) +static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, + unsigned long addr) +{ + long sign = addr >> 63; + + mmap_assert_locked(mm); + addr &= (mm)->context.untag_mask | sign; + + return addr; +} #define untagged_addr_remote(mm, addr) ({ \ - mmap_assert_locked(mm); \ - __untagged_addr((mm)->context.untag_mask, addr); \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr_remote(mm, __addr); \ }) #else diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 0831d2be190f..e006725afdf1 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -745,9 +745,6 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) #ifdef CONFIG_ADDRESS_MASKING -DEFINE_STATIC_KEY_FALSE(tagged_addr_key); -EXPORT_SYMBOL_GPL(tagged_addr_key); - #define LAM_U57_BITS 6 static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) @@ -787,8 +784,6 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags); mmap_write_unlock(mm); - - static_branch_enable(&tagged_addr_key); return 0; } #endif -- Kiryl Shutsemau / Kirill A. Shutemov