Re: [PATCH v10 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

[Peter Zijlstra <peterz@infradead.org>]

On Tue, Feb 11, 2025 at 04:08:04PM -0500, Rik van Riel wrote:

I poked around at this function a little, and ended up with the below.

As to your question if any INVLPGB capable hardware needs PTI; the
answer is no. No AMD machine needs PTI; but it should still work, just
in case someone needs to test something.

static void broadcast_tlb_flush(struct flush_tlb_info *info)
{
	unsigned long asid = info->mm->context.global_asid;
	bool pmd = info->stride_shift == PMD_SHIFT;
	unsigned long addr = info->start;

	/*
	 * TLB flushes with INVLPGB are kicked off asynchronously.
	 * The inc_mm_tlb_gen() guarantees page table updates are done
	 * before these TLB flushes happen.
	 */
	if (info->end == TLB_FLUSH_ALL) {
		invlpgb_flush_single_pcid_nosync(kern_pcid(asid));
		if (static_cpu_has(X86_FEATURE_PTI))
			invlpgb_flush_single_pcid_nosync(user_pcid(asid));

	} else do {
		unsigned long nr = 1;

		if (info->stride_shift <= PMD_SHIFT) {
			/*
			 * Calculate how many pages can be flushed at once; if the
			 * remainder of the range is less than one page, flush one.
			 */
			nr = (info->end - addr) >> info->stride_shift);
			nr = clamp_val(nr, 1, invlpgb_count_max);
		}

		invlpgb_flush_user_nr_nosync(kern_pcid(asid), addr, nr, pmd);
		if (static_cpu_has(X86_FEATURE_PTI))
			invlpgb_flush_user_nr_nosync(user_pcid(asid), addr, nr, pmd);

		addr += nr << info->stride_shift;
	} while (addr < info->end);

	finish_asid_transition(info);

	/* Wait for the INVLPGBs kicked off above to finish. */
	tlbsync();
}