From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A746DE77194 for ; Mon, 30 Dec 2024 17:57:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 38CB06B00A8; Mon, 30 Dec 2024 12:57:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F7116B00AE; Mon, 30 Dec 2024 12:57:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C7846B00AB; Mon, 30 Dec 2024 12:57:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 246BC6B00A1 for ; Mon, 30 Dec 2024 12:57:35 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DA394454EE for ; Mon, 30 Dec 2024 17:57:34 +0000 (UTC) X-FDA: 82952381778.05.3230846 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf16.hostedemail.com (Postfix) with ESMTP id 9650E180005 for ; Mon, 30 Dec 2024 17:56:45 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf16.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735581432; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NgX1sHYsFGUW87qNeidZuY+X0xFEZh+rGS/R4qqusWc=; b=A1f+Kuopsw9uUxbUBX6y7ZomiK9IqQjAw/IpjMXlLUlwGOc9hGRuz/TvkzaA75qhVf/7ZQ 2awuR8J9EFTU90jT1AtNAjjUJWclVRvvKL+Sytv1xCcSfCWB0Zme7rz8nJD/fccDDdfYhU mRx044emDYXdMt8dv94bljZMU1qstM4= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf16.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735581432; a=rsa-sha256; cv=none; b=zu7fblzo4o0TDjdqSFcGIGY8ez5WlHzgWc2MBQlKFzmsGziHFsOY/Wts3DYTPrypaSbWFg MDHEUrTXet+zYl9nl/3opGkZzPMbdajcmwnx8CFFykF3vm9j641PdjbwWAC7MGrtYp7oV8 MCUJs7lOAsjqPOpBgqNzlXeJ6TQrDaA= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tSJze-000000008Lf-1jF1; Mon, 30 Dec 2024 12:55:54 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, akpm@linux-foundation.org, nadav.amit@gmail.com, zhengqi.arch@bytedance.com, linux-mm@kvack.org, Rik van Riel Subject: [PATCH 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing Date: Mon, 30 Dec 2024 12:53:09 -0500 Message-ID: <20241230175550.4046587-9-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230175550.4046587-1-riel@surriel.com> References: <20241230175550.4046587-1-riel@surriel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 9650E180005 X-Stat-Signature: xtwbi99c8s18xxxtttcrkofxheykn3fm X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1735581405-667409 X-HE-Meta: U2FsdGVkX19OheMkKB9GFlYlTLyI2suZE1Q70PyShddu0AU/IiX/ZxnOqaTjG4t8VlFoE798pLge/BlZDzdNKjgQOzZVMLYB3fHxgTUFd8r3Gsfh+p59bOiixTyXVRVGhr5eVyUvsVyVMKRz8CH5CPnxKfy1H9HZra27E2k6p1/Lnm4d7FGRXZO4htA1+Ynk4Wgc9F5Mmfqtp90KI8RWA/nBsr4P9fnPOoDzJQpG4Zd+KDlHMEt0TKhJJaqI3DD2bSXIXq349DBYFM4BSo4cS9p16Aqcjoe8WrYikIrBJhkJvHTWT+UuJ1LKUGu2hUCg5gzQBq0N/3zXz45jW+7iGw4tF3P1SUDosiWnWcwvIPAoYRoVzkTeA+3bHkmvKctjJTSSIFI7mWcjuAvl8vN0wKmLpoweTpBPfjbcUhRuXcn7lD/7GqOp6+QProYGkQ5Tpqr2Eo9qWuEhiW7Wsdg+A1snRDhQI0A2YN6cgHgsM8y4ugf1dG3DMMnTrL1PkiEq+At2m8vaxD2mFJlIavBTLPLiUty7jOJZkQWrJ7xp7sTk6xFtUsZ3rz9/cShltzk2n/HSbK83igr+mpcqIQiIFKvNqoMs5PHgcEpRaX2zB28PZp38KyPLssQ1TIG0lLS2iN0v6nQ3P5e/aPbQnlYlUzvNHSchDsKStEmIcXmsz1FY9CgNPq1M4Fz0SMsTUbkZQyT7bG8f5nVipJukqgp52zavhWZg3JhRT0ehnNsk4u1ryevAVzB7t/cLGWKxUe6WjYwOWwmxgtGYJce/1xCkAVVH65EyjzZLHHrNSTUQedQQUSdg6Ryz8uiPeYoaEs9RBt0zRu8iTDnz/q6HZ9CVOk7E9vGZtEjtqxSD3bn0yP609pz22NYDy5wT+JpIt6BSUBg/Ha/AZ/TIbbwYhq3Ag6PHkmOynzB73nuCnaOxWcZ7z5OsMM3/xD8rdjVNGo/hTJjBtFOiFYBCPTFqeNu BHOZdYmj fzwVTmmCUlUbbz6H5L4A5jZjWRy8P478Fsjl4IpXExO7x4l0/rBoIqJDFk5kvPHT/gZFpR8q0qR4Uygw2OHwE2VSsPUwhrnlmHktn3sZkxQNeF5THN2GS/kmZm3R+1Paa/FNsmdVxawSOrqGHLFN6bZ4sFkeRFHtMNa8yr0LyUp7IZh5BQVIlxx0c1EXxfg04SNYXQhHbqXgaWtC74lOdf1+eZQJPbJwu8VWb0jSJC4a7bI4wJPP1LYX9w4mIhxMm/6T2hy1969KngA6huFX2U/6dEc9S2KmasKZ9mCaVvB0Nh2Mtdch1dB7bcrPPvTAf7DFIGjmDon+UdwWfBYDBeWjGFzjRsmDg8PecOI3jAXxOwLu9aYiaYH5rpK0l4aPL7OZT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated. Use broadcast TLB flushing when that is available. Signed-off-by: Rik van Riel --- arch/x86/mm/tlb.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 266d5174fc7b..64f1679c37e1 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1310,8 +1310,16 @@ EXPORT_SYMBOL_GPL(__flush_tlb_all); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) { struct flush_tlb_info *info; + int cpu; + + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + guard(preempt)(); + invlpgb_flush_all_nonglobals(); + tlbsync(); + return; + } - int cpu = get_cpu(); + cpu = get_cpu(); info = get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false, TLB_GENERATION_INVALID); -- 2.47.1