From: Rik van Riel <riel@surriel.com>
To: x86@kernel.org
Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org,
dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com,
nadav.amit@gmail.com, thomas.lendacky@amd.com,
kernel-team@meta.com, linux-mm@kvack.org,
akpm@linux-foundation.org, jannh@google.com,
mhklinux@outlook.com, andrew.cooper3@citrix.com,
Rik van Riel <riel@surriel.com>,
Manali Shukla <Manali.Shukla@amd.com>
Subject: [PATCH v8 12/12] x86/mm: only invalidate final translations with INVLPGB
Date: Tue, 4 Feb 2025 20:40:01 -0500 [thread overview]
Message-ID: <20250205014033.3626204-13-riel@surriel.com> (raw)
In-Reply-To: <20250205014033.3626204-1-riel@surriel.com>
Use the INVLPGB_FINAL_ONLY flag when invalidating mappings with INVPLGB.
This way only leaf mappings get removed from the TLB, leaving intermediate
translations cached.
On the (rare) occasions where we free page tables we do a full flush,
ensuring intermediate translations get flushed from the TLB.
Signed-off-by: Rik van Riel <riel@surriel.com>
Tested-by: Manali Shukla <Manali.Shukla@amd.com>
---
arch/x86/include/asm/invlpgb.h | 10 ++++++++--
arch/x86/mm/tlb.c | 8 ++++----
2 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/arch/x86/include/asm/invlpgb.h b/arch/x86/include/asm/invlpgb.h
index a1d5dedd5217..5fba41671a6d 100644
--- a/arch/x86/include/asm/invlpgb.h
+++ b/arch/x86/include/asm/invlpgb.h
@@ -67,9 +67,15 @@ static inline void invlpgb_flush_user(unsigned long pcid,
static inline void invlpgb_flush_user_nr_nosync(unsigned long pcid,
unsigned long addr,
u16 nr,
- bool pmd_stride)
+ bool pmd_stride,
+ bool freed_tables)
{
- __invlpgb(0, pcid, addr, nr - 1, pmd_stride, INVLPGB_PCID | INVLPGB_VA);
+ unsigned long flags = INVLPGB_PCID | INVLPGB_VA;
+
+ if (!freed_tables)
+ flags |= INVLPGB_FINAL_ONLY;
+
+ __invlpgb(0, pcid, addr, nr - 1, pmd_stride, flags);
}
/* Flush all mappings for a given PCID, not including globals. */
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index c064e27df1f3..7b116898c547 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -518,10 +518,10 @@ static void broadcast_tlb_flush(struct flush_tlb_info *info)
nr = min(maxnr, (info->end - addr) >> info->stride_shift);
nr = max(nr, 1);
- invlpgb_flush_user_nr_nosync(kern_pcid(asid), addr, nr, pmd);
+ invlpgb_flush_user_nr_nosync(kern_pcid(asid), addr, nr, pmd, info->freed_tables);
/* Do any CPUs supporting INVLPGB need PTI? */
if (static_cpu_has(X86_FEATURE_PTI))
- invlpgb_flush_user_nr_nosync(user_pcid(asid), addr, nr, pmd);
+ invlpgb_flush_user_nr_nosync(user_pcid(asid), addr, nr, pmd, info->freed_tables);
addr += nr << info->stride_shift;
} while (addr < info->end);
@@ -1688,10 +1688,10 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch,
batch->used_invlpgb = true;
migrate_disable();
}
- invlpgb_flush_user_nr_nosync(kern_pcid(asid), uaddr, 1, false);
+ invlpgb_flush_user_nr_nosync(kern_pcid(asid), uaddr, 1, false, false);
/* Do any CPUs supporting INVLPGB need PTI? */
if (static_cpu_has(X86_FEATURE_PTI))
- invlpgb_flush_user_nr_nosync(user_pcid(asid), uaddr, 1, false);
+ invlpgb_flush_user_nr_nosync(user_pcid(asid), uaddr, 1, false, false);
/*
* Some CPUs might still be using a local ASID for this
--
2.47.1
prev parent reply other threads:[~2025-02-05 1:42 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-02-05 1:39 [PATCH v8 00/12] AMD broadcast TLB invalidation Rik van Riel
2025-02-05 1:39 ` [PATCH v8 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional Rik van Riel
2025-02-05 1:39 ` [PATCH v8 02/12] x86/mm: remove pv_ops.mmu.tlb_remove_table call Rik van Riel
2025-02-05 1:39 ` [PATCH v8 03/12] x86/mm: consolidate full flush threshold decision Rik van Riel
2025-02-05 12:20 ` Peter Zijlstra
2025-02-05 13:00 ` Peter Zijlstra
2025-02-05 13:52 ` Rik van Riel
2025-02-05 1:39 ` [PATCH v8 04/12] x86/mm: get INVLPGB count max from CPUID Rik van Riel
2025-02-05 1:39 ` [PATCH v8 05/12] x86/mm: add INVLPGB support code Rik van Riel
2025-02-05 1:39 ` [PATCH v8 06/12] x86/mm: use INVLPGB for kernel TLB flushes Rik van Riel
2025-02-05 1:39 ` [PATCH v8 07/12] x86/mm: use INVLPGB in flush_tlb_all Rik van Riel
2025-02-05 1:39 ` [PATCH v8 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing Rik van Riel
2025-02-05 1:39 ` [PATCH v8 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes Rik van Riel
2025-02-05 1:39 ` [PATCH v8 10/12] x86/mm: do targeted broadcast flushing from tlbbatch code Rik van Riel
2025-02-05 13:51 ` Peter Zijlstra
2025-02-05 14:52 ` Rik van Riel
2025-02-05 1:40 ` [PATCH v8 11/12] x86/mm: enable AMD translation cache extensions Rik van Riel
2025-02-05 1:40 ` Rik van Riel [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250205014033.3626204-13-riel@surriel.com \
--to=riel@surriel.com \
--cc=Manali.Shukla@amd.com \
--cc=akpm@linux-foundation.org \
--cc=andrew.cooper3@citrix.com \
--cc=bp@alien8.de \
--cc=dave.hansen@linux.intel.com \
--cc=jannh@google.com \
--cc=kernel-team@meta.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhklinux@outlook.com \
--cc=nadav.amit@gmail.com \
--cc=peterz@infradead.org \
--cc=thomas.lendacky@amd.com \
--cc=x86@kernel.org \
--cc=zhengqi.arch@bytedance.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox