From: Rik van Riel <riel@surriel.com>
To: x86@kernel.org
Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com,
dave.hansen@linux.intel.com, luto@kernel.org,
peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com,
bp@alien8.de, hpa@zytor.com, akpm@linux-foundation.org,
linux-mm@kvack.org
Subject: [RFC PATCH v2 00/11] AMD broadcast TLB invalidation
Date: Sun, 22 Dec 2024 21:55:06 -0500 [thread overview]
Message-ID: <20241223025751.3268975-1-riel@surriel.com> (raw)
Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.
This allows the kernel to invalidate TLB entries on remote CPUs without
needing to send IPIs, without having to wait for remote CPUs to handle
those interrupts, and with less interruption to what was running on
those CPUs.
Because x86 PCID space is limited, and there are some very large
systems out there, broadcast TLB invalidation is only used for
processes that are active on 3 or more CPUs, with the threshold
being gradually increased the more the PCID space gets exhausted.
Combined with the removal of unnecessary lru_add_drain calls
(see https://lkml.org/lkml/2024/12/19/1388) this results in a
nice performance boost for the will-it-scale tlb_flush2_threads
test on an AMD Milan system with 36 cores:
- vanilla kernel: 527k loops/second
- lru_add_drain removal: 731k loops/second
- only INVLPGB: 527k loops/second
- lru_add_drain + INVLPGB: 1157k loops/second
Profiling with only the INVLPGB changes showed while
TLB invalidation went down from 40% of the total CPU
time to only around 4% of CPU time, the contention
simply moved to the LRU lock.
Fixing both at the same time about doubles the
number of iterations per second from this case.
v2:
- Apply suggestions by Peter and Borislav (thank you!)
- Fix bug in arch_tlbbatch_flush, where we need to do both
the TLBSYNC, and flush the CPUs that are in the cpumask.
- Some updates to comments and changelogs based on questions.
next reply other threads:[~2024-12-23 3:06 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-12-23 2:55 Rik van Riel [this message]
2024-12-23 2:55 ` [PATCH 01/11] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional Rik van Riel
2024-12-23 6:01 ` Qi Zheng
2024-12-23 20:20 ` Rik van Riel
2024-12-24 18:26 ` Peter Zijlstra
2024-12-23 2:55 ` [PATCH 02/11] x86/mm: add X86_FEATURE_INVLPGB definition Rik van Riel
2024-12-23 2:55 ` [PATCH 03/11] x86/mm: get INVLPGB count max from CPUID Rik van Riel
2024-12-25 23:42 ` Nadav Amit
2024-12-23 2:55 ` [PATCH 04/11] x86/mm: add INVLPGB support code Rik van Riel
2024-12-23 2:55 ` [PATCH 05/11] x86/mm: use INVLPGB for kernel TLB flushes Rik van Riel
2024-12-23 2:55 ` [PATCH 06/11] x86/tlb: use INVLPGB in flush_tlb_all Rik van Riel
2024-12-23 2:55 ` [PATCH 07/11] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing Rik van Riel
2024-12-23 2:55 ` [PATCH 08/11] x86/mm: enable broadcast TLB invalidation for multi-threaded processes Rik van Riel
2024-12-25 23:22 ` Nadav Amit
2024-12-25 23:32 ` Nadav Amit
2024-12-23 2:55 ` [PATCH 09/11] x86,tlb: do targeted broadcast flushing from tlbbatch code Rik van Riel
2024-12-23 2:55 ` [PATCH 10/11] x86/mm: enable AMD translation cache extensions Rik van Riel
2024-12-23 2:55 ` [PATCH 11/11] x86/mm: only invalidate final translations with INVLPGB Rik van Riel
2024-12-24 18:08 ` [RFC PATCH v2 00/11] AMD broadcast TLB invalidation Michael Kelley
2024-12-25 14:48 ` Rik van Riel
2025-01-10 19:29 ` Tom Lendacky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20241223025751.3268975-1-riel@surriel.com \
--to=riel@surriel.com \
--cc=akpm@linux-foundation.org \
--cc=bp@alien8.de \
--cc=dave.hansen@linux.intel.com \
--cc=hpa@zytor.com \
--cc=kernel-team@meta.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=luto@kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox