From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21C65E77188 for ; Fri, 3 Jan 2025 12:40:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5E3586B007B; Fri, 3 Jan 2025 07:40:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 56C786B0082; Fri, 3 Jan 2025 07:40:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 40CB96B0083; Fri, 3 Jan 2025 07:40:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 209316B007B for ; Fri, 3 Jan 2025 07:40:06 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BE76E16079B for ; Fri, 3 Jan 2025 12:40:05 +0000 (UTC) X-FDA: 82966097382.28.756ADF8 Received: from mail.alien8.de (mail.alien8.de [65.109.113.108]) by imf05.hostedemail.com (Postfix) with ESMTP id B544A10000C for ; Fri, 3 Jan 2025 12:38:24 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=alien8.de header.s=alien8 header.b=ZoF4ZNKC; spf=pass (imf05.hostedemail.com: domain of bp@alien8.de designates 65.109.113.108 as permitted sender) smtp.mailfrom=bp@alien8.de; dmarc=pass (policy=none) header.from=alien8.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735907969; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uSAkp86v8WrK3u8tMQk38EidNnwWzrpudINQvSd8W3M=; b=tG/EEdMxzhi4+QtA/pkmdebsQ+zdiwylHu2sNEoAt8rDLvgi8Ebt5CF6R8xcuRGmjzJaaU qIk1uzDSgn/lfADtqoWDdMn+9fXfYdtNt2Ox6VsaP9i/D/phYZ2Yq5JNkuIvZb0ibs/KTh qsfonhDAYdBgHqteqhGEM526bDOV4sM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735907969; a=rsa-sha256; cv=none; b=UkQwjY/u0b8fcVyo7WIA8klFVdAykudhCjvdP26Axkm35xdDPdzuBocPCDjFxYJ+vQZ14s I6lBrCvDoy1aaZpYW15f30TlAkNfFpCjktHhfDrxbFQSB4Mc8qjCSORN8Nv3uZJM25CetD 36RMGerWuIPkt7gb8hgo/iPk0iiiuHo= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=alien8.de header.s=alien8 header.b=ZoF4ZNKC; spf=pass (imf05.hostedemail.com: domain of bp@alien8.de designates 65.109.113.108 as permitted sender) smtp.mailfrom=bp@alien8.de; dmarc=pass (policy=none) header.from=alien8.de Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTP id 68D2A40E021D; Fri, 3 Jan 2025 12:40:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at mail.alien8.de Received: from mail.alien8.de ([127.0.0.1]) by localhost (mail.alien8.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id k7B4Nw9S4TSq; Fri, 3 Jan 2025 12:39:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alien8.de; s=alien8; t=1735907997; bh=uSAkp86v8WrK3u8tMQk38EidNnwWzrpudINQvSd8W3M=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=ZoF4ZNKCsRhGPWXcXkFaJCIN0/o03/DTV76hGfT2vvU2VcqbDznk01uYytQQzwhog DCi1D5Y4XIWb9Y13e1OocHSHwypiCXkBhWFDoGH6g24ND0tOmEN+JkpmwNWGH3BDGw cUUfCwfKrbdNNteTqT/uSvCph2ERVdUsBFPT3nmpbMGYeJIpU6srdFMlAiCM1APTDe snExbvQqSi9nU0I18RsrRj6DwbZwNa948KwGsszbDOqRJu1NZrQuFRVLW2Dw0eFxIk gIG0fG4lVrqjSJAYKmIsuINstFluloHYzK63Ug9E0n7XMXZTls8kjy7IKmejoavHhP 0Fsn9d9PntKesHCHqgWvqzgmX8GDtw+C4QFFhkFvCTNV3bWLW9cd9yT5x0OkvLLT9t 3rbVqpY7t/sf50Zp1G7AIn5pv8ueMos/HBNE95f2nHDgA8ytXoJlLz06L5Jw2zkwYA xg+ZKcF0zCYeA9bq4dOShHgZ4j4rGZyLQKxXl9IXZrm7Is3EEPbATBPu76l5e7bXK6 fYwIOmcChQ+Zjfx51PcTeRfobitdUwDE8V7nV3+hPfDWgxLPhE5094NkSj3wDDQXal cHJ0UrsA8DVj8GVvYbmWGPVCwZFn8PuL6qdCpykx1zHPWtW1RygaboFR5axOaq8IY/ PSstCizRtyjyGnz5ZemyBL5I= Received: from zn.tnic (p200300Ea971F93bA329c23Fffea6a903.dip0.t-ipconnect.de [IPv6:2003:ea:971f:93ba:329c:23ff:fea6:a903]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTPSA id 4052F40E01F9; Fri, 3 Jan 2025 12:39:43 +0000 (UTC) Date: Fri, 3 Jan 2025 13:39:37 +0100 From: Borislav Petkov To: Rik van Riel Cc: x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com, akpm@linux-foundation.org, nadav.amit@gmail.com, zhengqi.arch@bytedance.com, linux-mm@kvack.org Subject: Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes Message-ID: <20250103123937.GAZ3faiaokjxVQM1pk@fat_crate.local> References: <20241230175550.4046587-1-riel@surriel.com> <20241230175550.4046587-7-riel@surriel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20241230175550.4046587-7-riel@surriel.com> X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B544A10000C X-Stat-Signature: hepi819im5o4paq7rr1riqeyog4diyjc X-Rspam-User: X-HE-Tag: 1735907904-569392 X-HE-Meta: U2FsdGVkX1+fLGyQWdQX47qKgekn+13oRMpGCohilBbedt/eUxtn154HY2wC6XDjcgKe5sj0qLabsjLGUQ0yXWVcfl1+2FPapABMk6C/6oXtgTlMba7O9BQD8jHJekIiuqJEJJQhBkivg7gCViPC2SqzD7Qp0q0+1d/TRV0zRaOLdRWwXNPk7UQ/XMi5nZeGrm1RmPmmuHKnzUdFGFssuikzuUrTtpiM8Za2SGtaS65RteJH1IHt0OiB4T+V+iCQwDB31IfyS298pKPUx34lqZJ8Aw6ryMzD1qm7ISZluaeT4tQJJsAhoWt/zfgt5ODh0mEb0Kv6J2TJE4SlyzM7BE+fjEDnsvnEzoQUzOYcw7ZCkbyLua4rZHxQau/RKQCGHzCSxHAob5CBbBkgPbhtVRmrscvnlXAZonqSj49+OXMe6so6wMXiUCMlKg9dNVXzvTmQbSZ1+AgENEL3GVPVnXw3CTbaeDdqlS3FoerEkV16ZYPLcuyuwepLXr73rc8HRAQtJtANugw+xJoEJekc0GTLbp8VeHeKzxXKOXw67xDjlweQt+geuxXQwgGNgE6BBQvj2DnR3Zht2k4xSeljHHWyRd5eoRfq0FUD7jKQqkD5WYKV/RLRoyx50vZK2pwv4W+cYmVEc59dsCo/fTbEccXqgtEiAorS0ihwZXjMZeRDhZqlu6ca3SVG6FFLOSoB7uN/6dUGOAh4OvZa3u0aR9YtXevlLMaMxiB9ZScxdkaGwM2f13V7FYw/4FVUDaNfDGfOkItr4kc6rt+Gh0CyUwcRLpfMagIJ14Phsu5Xu1MH7ORizdJ6ENTq5B0K/nDqY39+6N9Sn5iLi7NWcQ0dDINLZU76gJDbSbMyMg8EhnIPjug2wcEN/HxV9nxpIN2qH6M9W64+YB0W0WHDsAE4wyn5+mZ48s48cdoPmunpApNb6rVV5s7h9fYO2e0hx4P3TcgEqrflTmbDUZzYO3E npBXvjIz 4/S9rQByt8AO8hyfijlrbeJyFFLUhpqFkWnfEpaE4iYe+65YSbh64RP6kx2yPlnEO/jLYnQRtog3CZy2z1fqHuq3QQH0wfBSGYGv3UMVh6i7rhE4Be3pBN/JYI6GzaxEQFMUh5jQYQzRrOBHvgabdhYF9gLKtPEVCpP8CV8pJ7hn4eN0ZaHGamuiKZgQRsEpb9oMGZo5Ql5MEghbp69PerMJehewSwWuBqX14kk/4i7eKpRN3pfpmYL7QkEJrEg84MBWPVS7M+4/Jv0ECTTMjXqJyOcUWmAeN7FskWySd4utVgxS5r1ZmuCwLriQrQKo2do0MZhjYxa/floA6/0L5ELiY0rKYQ+x6ZvYR25hevfpwAnPT4xzW7rgOW9ROgMv/EXNo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Dec 30, 2024 at 12:53:07PM -0500, Rik van Riel wrote: > Use broadcast TLB invalidation for kernel addresses when available. > > This stops us from having to send IPIs for kernel TLB flushes. > > Signed-off-by: Rik van Riel > --- > arch/x86/mm/tlb.c | 31 +++++++++++++++++++++++++++++++ > 1 file changed, 31 insertions(+) > > diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c > index 6cf881a942bb..29207dc5b807 100644 > --- a/arch/x86/mm/tlb.c > +++ b/arch/x86/mm/tlb.c > @@ -1077,6 +1077,32 @@ void flush_tlb_all(void) > on_each_cpu(do_flush_tlb_all, NULL, 1); > } > > +static void broadcast_kernel_range_flush(unsigned long start, unsigned long end) > +{ > + unsigned long addr; > + unsigned long maxnr = invlpgb_count_max; > + unsigned long threshold = tlb_single_page_flush_ceiling * maxnr; The tip-tree preferred ordering of variable declarations at the beginning of a function is reverse fir tree order:: struct long_struct_name *descriptive_name; unsigned long foo, bar; unsigned int tmp; int ret; The above is faster to parse than the reverse ordering:: int ret; unsigned int tmp; unsigned long foo, bar; struct long_struct_name *descriptive_name; And even more so than random ordering:: unsigned long foo, bar; int ret; struct long_struct_name *descriptive_name; unsigned int tmp; And you can get rid of maxnr and get the reversed xmas tree order: diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 29207dc5b807..8a85acd9483d 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1079,9 +1079,8 @@ void flush_tlb_all(void) static void broadcast_kernel_range_flush(unsigned long start, unsigned long end) { + unsigned long threshold = tlb_single_page_flush_ceiling * invlpgb_count_max; unsigned long addr; - unsigned long maxnr = invlpgb_count_max; - unsigned long threshold = tlb_single_page_flush_ceiling * maxnr; /* * TLBSYNC only waits for flushes originating on the same CPU. @@ -1095,7 +1094,7 @@ static void broadcast_kernel_range_flush(unsigned long start, unsigned long end) } else { unsigned long nr; for (addr = start; addr < end; addr += nr << PAGE_SHIFT) { - nr = min((end - addr) >> PAGE_SHIFT, maxnr); + nr = min((end - addr) >> PAGE_SHIFT, invlpgb_count_max); invlpgb_flush_addr(addr, nr); } } > + /* > + * TLBSYNC only waits for flushes originating on the same CPU. > + * Disabling migration allows us to wait on all flushes. > + */ > + guard(preempt)(); Migration? Why not migrate_disable() then? Although there's a big, thorny comment in include/linux/preempt.h about its influence on sched. > + > + if (end == TLB_FLUSH_ALL || > + (end - start) > threshold << PAGE_SHIFT) { > + invlpgb_flush_all(); > + } else { > + unsigned long nr; > + for (addr = start; addr < end; addr += nr << PAGE_SHIFT) { > + nr = min((end - addr) >> PAGE_SHIFT, maxnr); > + invlpgb_flush_addr(addr, nr); > + } > + } > + > + tlbsync(); > +} -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette