From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DE60C53210 for ; Tue, 27 Dec 2022 03:08:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 59F678E0007; Mon, 26 Dec 2022 22:08:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B49E8E0005; Mon, 26 Dec 2022 22:08:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2685F8E0007; Mon, 26 Dec 2022 22:08:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 11ED58E0005 for ; Mon, 26 Dec 2022 22:08:54 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D9D4C406CB for ; Tue, 27 Dec 2022 03:08:53 +0000 (UTC) X-FDA: 80286604146.11.D9915DD Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf13.hostedemail.com (Postfix) with ESMTP id EA82F20006 for ; Tue, 27 Dec 2022 03:08:51 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=cFiXzpj7; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf13.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.151) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672110532; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JaHQeXh+L8VDr3v14jKHr2jtqcMM5uGvvy+DcgWw4zw=; b=E3S2cC2Wc8XKwPFL/W3OqwdSwKsnduWUAJ+yC3cGXL8xqR+IQJ4ocKP4Qs2F12EJwT4NdU 071BC4l3KUXOlJt8UuNeu32B7/md+0qcNcq4xXdXDPIbUHaPuBA+Il9uxzLALwZwPKOYd1 hJCG6m6hMY9LyV9Q9voueczOb53EsXQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=cFiXzpj7; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf13.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.151) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672110532; a=rsa-sha256; cv=none; b=ZYpAiMIB+1GvMNIQb+cpnaCaELlCClSS3Nr95HxplwUWdX37He2UYLhTCBIg0KtVGBYY2l 9YOxR/OY4L2Qpj1jh/RXSma7xUpu/jo/kq5JlreJTFZRIzJeWIlbFml3GyVjgJLToYjMHc 2Fy57eznZhtxZ+N3cX2iTeVK2slIJZQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110532; x=1703646532; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cWbMXq+BulH9gIMaAKVyjGjPeJ6XOcZ2RXQodh7xORI=; b=cFiXzpj7nKQO3QhZiHiv3juqbPE4qhL29p1AajyAloVQtPBxFSNjfsTg UpOkPjacLoJ3Uq9nQ5jRtRB80bS+uZBIExqCe9u6uiBYePwWJPG7Ta8MP MnF+GJEXpxPPFRagrE2/ioNSO+OW06P3ZYUfzGY40inD8Qe1dIGaBychC pl+XSFSC1IqIQ31uTHCkBouREyxJO3+nBxp+AXMkuqv3xDMAUqmAtdzfs /pdhpFpGRAFJtU6TdJyJUlHVWcXToSFy6jVq5XpDF0Y9A0Okrdf4mMzey MtgrRbm+Rwa6i7pblo/GfwVC06CUr3KH+S4HySRSy8hWv9X7QTljr78Su Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="300994698" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="300994698" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:50 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="741605217" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="741605217" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:46 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id BEEAE10BBB4; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 07/16] x86/mm: Reduce untagged_addr() overhead until the first LAM user Date: Tue, 27 Dec 2022 06:08:20 +0300 Message-Id: <20221227030829.12508-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: EA82F20006 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: ibhqorworzsysj98dhc4wq9q41w9skcc X-HE-Tag: 1672110531-19131 X-HE-Meta: U2FsdGVkX19Yda4dGtHR1HA0jfwT8QeV4Aq/pV0/vxP0cYjlLEZauuSYCnDrUewpL2+p0RdRM5GPOg90GxQayruNCq1kyhysOaSqbL8WoiCeogjIF4YJVBRqFL+U68tiX/p54H66o2Etru0hYg0+gChlSd9/elJg1qo0pE0EkcSQiePBnuc8jwgXHDuAmBlKN4BOOVpiba4+2Pse3ZKGcMvoOMECSe5KaPuyNyriCPfehGzbatEgCuFG/Ozz6teGrctrcZzp240oAkg2OG2ZYuMwKQUdPLz0emiPseP0fOpbD+NUJPzFNW50neVM1cnM0yV1tRf5n2UU9uJ5DJNlZNaRy+p9xjHobhYfpotHvCwp18/j3jABij1zJxTveuN4axbBYJcBgxO4OR/IM6jTMSLKGdP3jA4AFhX7KgdDCk+DEruJcuPfzC1LXH7zC1MFW/uwA5Fuptmtda9+IxltlbyZkZ8MS3qIW+v08w90Pfp1cUVu/A3BJPSgjG48MPL69Yww/dae2qD6jt1mahIK34k3lfUEx5E+A4kPe6yloitgkQ9Fp0/BQF52ktWregWY2TXJFN9gqi++M6LozC/7d8yjV1XE61FvS+/+1MGpNabN+VrXFPcWlObtwlQw0p4ItZAiyd/dwHzJKGBPy0cyWJRwg8KpT1DUQ+Th2bf7qeMdrAUl1FE8P9PCr/bOd4nsGETiho18DzL5vI8T7OmtOuq0M2rQn5/cnM88YujeTDHChlwiAC7ekDguFCKY6tRnU5MUI5tdqMaKBYAbDC8myK56YHJ7hXhnFDyePGrKRkoKYXeRNfu3L6xE/4YYXDZXCWcKdsidp3sZF2iHPltVc2fLl5HZ247sVTS/rpHELFMJy3MO/5g8kKLkEin42MZBdVWw6iHf0CVS5BUUFfUlTct2khEVvSQOgFihdsPH4hNs4OjTrqXsbXTjo/1n+Fzlz541kfCXQEz/6Ueobtv GLPxwYjm rVdxcAPD0dr3fUvG4Qk6XsnYX67sMCE4EJS6u X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use static key to reduce untagged_addr() overhead. The key only gets enabled when the first process enables LAM. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/uaccess.h | 9 +++++++-- arch/x86/kernel/process_64.c | 4 ++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index cbb463e9344f..1d931c7f6741 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -23,6 +23,8 @@ static inline bool pagefault_disabled(void); #endif #ifdef CONFIG_X86_64 +DECLARE_STATIC_KEY_FALSE(tagged_addr_key); + /* * Mask out tag bits from the address. * @@ -31,8 +33,11 @@ static inline bool pagefault_disabled(void); */ #define __untagged_addr(mm, addr) ({ \ u64 __addr = (__force u64)(addr); \ - s64 sign = (s64)__addr >> 63; \ - __addr &= READ_ONCE((mm)->context.untag_mask) | sign; \ + if (static_branch_likely(&tagged_addr_key)) { \ + s64 sign = (s64)__addr >> 63; \ + u64 mask = READ_ONCE((mm)->context.untag_mask); \ + __addr &= mask | sign; \ + } \ (__force __typeof__(addr))__addr; \ }) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index fef127ed79b6..09e7f3d3fb5c 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -743,6 +743,9 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) } #endif +DEFINE_STATIC_KEY_FALSE(tagged_addr_key); +EXPORT_SYMBOL_GPL(tagged_addr_key); + #define LAM_U57_BITS 6 static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) @@ -773,6 +776,7 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) mmap_write_unlock(mm); + static_branch_enable(&tagged_addr_key); return 0; } -- 2.38.2