[RESEND PATCH v3 0/3] x86/mm: LAM fixups and cleanups

[RESEND PATCH v3 0/3] x86/mm: LAM fixups and cleanups

[Yosry Ahmed]

This series has fixups and cleanups for LAM. Most importantly, patch 1
fixes a sycnhronization issue that may cause crashes of userspace
applications. This is a resend of v3, rebased on top of v6.10-rc6.

v3: https://lore.kernel.org/lkml/20240418012835.3360429-1-yosryahmed@google.com/
v2: https://lore.kernel.org/lkml/20240312155641.4003683-1-yosryahmed@google.com/
v1: https://lore.kernel.org/lkml/20240312035951.3535980-1-yosryahmed@google.com/
RFC: https://lore.kernel.org/lkml/20240307133916.3782068-1-yosryahmed@google.com/

Yosry Ahmed (3):
  x86/mm: Use IPIs to synchronize LAM enablement
  x86/mm: Fix LAM inconsistency during context switch
  x86/mm: Cleanup prctl_enable_tagged_addr() nr_bits error checking

 arch/x86/include/asm/mmu_context.h |  8 +++++++-
 arch/x86/include/asm/tlbflush.h    |  9 ++++-----
 arch/x86/kernel/process_64.c       | 25 ++++++++++++++++---------
 arch/x86/mm/tlb.c                  | 15 ++++++++-------
 4 files changed, 35 insertions(+), 22 deletions(-)

-- 
2.45.2.803.g4e1b14247a-goog

[RESEND PATCH v3 1/3] x86/mm: Use IPIs to synchronize LAM enablement

[Yosry Ahmed]

LAM can only be enabled when a process is single-threaded.  But _kernel_
threads can temporarily use a single-threaded process's mm.

If LAM is enabled by a userspace process while a kthread is using its
mm, the kthread will not observe LAM enablement (i.e.  LAM will be
disabled in CR3). This could be fine for the kthread itself, as LAM only
affects userspace addresses. However, if the kthread context switches to
a thread in the same userspace process, CR3 may or may not be updated
because the mm_struct doesn't change (based on pending TLB flushes). If
CR3 is not updated, the userspace thread will run incorrectly with LAM
disabled, which may cause page faults when using tagged addresses.
Example scenario:

CPU 1                                   CPU 2
/* kthread */
kthread_use_mm()
                                        /* user thread */
                                        prctl_enable_tagged_addr()
                                        /* LAM enabled on CPU 2 */
/* LAM disabled on CPU 1 */
                                        context_switch() /* to CPU 1 */
/* Switching to user thread */
switch_mm_irqs_off()
/* CR3 not updated */
/* LAM is still disabled on CPU 1 */

Synchronize LAM enablement by sending an IPI from
prctl_enable_tagged_addr() to all CPUs running with the mm_struct to
enable LAM. This makes sure LAM is enabled on CPU 1 in the above
scenario before prctl_enable_tagged_addr() returns and userspace starts
using tagged addresses, and before it's possible to run the userspace
process on CPU 1.

In switch_mm_irqs_off(), move reading the LAM mask until after
mm_cpumask() is updated. This ensures that if an outdated LAM mask is
written to CR3, an IPI is received to update it right after IRQs are
re-enabled.

Fixes: 82721d8b25d7 ("x86/mm: Handle LAM on context switch")
Suggested-by: Andy Lutomirski <luto@kernel.org>
Reviewed-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Change-Id: I7fd573a9db5fe5284d69bc46f9b3758f1f9fb467
Signed-off-by: Yosry Ahmed <yosryahmed@google.com>
---
 arch/x86/kernel/process_64.c | 13 +++++++++++--
 arch/x86/mm/tlb.c            |  7 +++----
 2 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c
index 6d3d20e3e43a9..e1ce0dfd24258 100644
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -798,6 +798,16 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr)
 
 #define LAM_U57_BITS 6
 
+static void enable_lam_func(void *__mm)
+{
+	struct mm_struct *mm = __mm;
+
+	if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm) {
+		write_cr3(__read_cr3() | mm->context.lam_cr3_mask);
+		set_tlbstate_lam_mode(mm);
+	}
+}
+
 static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits)
 {
 	if (!cpu_feature_enabled(X86_FEATURE_LAM))
@@ -830,8 +840,7 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits)
 		return -EINVAL;
 	}
 
-	write_cr3(__read_cr3() | mm->context.lam_cr3_mask);
-	set_tlbstate_lam_mode(mm);
+	on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true);
 	set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags);
 
 	mmap_write_unlock(mm);
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 44ac64f3a047c..a041d2ecd8380 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -503,9 +503,9 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,
 {
 	struct mm_struct *prev = this_cpu_read(cpu_tlbstate.loaded_mm);
 	u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid);
-	unsigned long new_lam = mm_lam_cr3_mask(next);
 	bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy);
 	unsigned cpu = smp_processor_id();
+	unsigned long new_lam;
 	u64 next_tlb_gen;
 	bool need_flush;
 	u16 new_asid;
@@ -619,9 +619,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,
 			cpumask_clear_cpu(cpu, mm_cpumask(prev));
 		}
 
-		/*
-		 * Start remote flushes and then read tlb_gen.
-		 */
+		/* Start receiving IPIs and then read tlb_gen (and LAM below) */
 		if (next != &init_mm)
 			cpumask_set_cpu(cpu, mm_cpumask(next));
 		next_tlb_gen = atomic64_read(&next->context.tlb_gen);
@@ -633,6 +631,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,
 		barrier();
 	}
 
+	new_lam = mm_lam_cr3_mask(next);
 	set_tlbstate_lam_mode(next);
 	if (need_flush) {
 		this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id);
-- 
2.45.2.803.g4e1b14247a-goog

Re: [RESEND PATCH v3 1/3] x86/mm: Use IPIs to synchronize LAM enablement

[Borislav Petkov]

On Tue, Jul 02, 2024 at 01:21:37PM +0000, Yosry Ahmed wrote:
> Change-Id: I7fd573a9db5fe5284d69bc46f9b3758f1f9fb467

What do those tags mean?

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [RESEND PATCH v3 1/3] x86/mm: Use IPIs to synchronize LAM enablement

[Yosry Ahmed]

On Tue, Jul 2, 2024 at 11:58 AM Borislav Petkov <bp@alien8.de> wrote:
>
> On Tue, Jul 02, 2024 at 01:21:37PM +0000, Yosry Ahmed wrote:
> > Change-Id: I7fd573a9db5fe5284d69bc46f9b3758f1f9fb467
>
> What do those tags mean?

Those are gerrit tags (from internal tooling) that I mistakenly
included, please ignore them. I can resend the patches without them if
needed. Sorry about that.

[RESEND PATCH v3 2/3] x86/mm: Fix LAM inconsistency during context switch

[Yosry Ahmed]

LAM can only be enabled when a process is single-threaded.  But _kernel_
threads can temporarily use a single-threaded process's mm.  That means
that a context-switching kernel thread can race and observe the mm's LAM
metadata (mm->context.lam_cr3_mask) change.

The context switch code does two logical things with that metadata:
populate CR3 and populate 'cpu_tlbstate.lam'.  If it hits this race,
'cpu_tlbstate.lam' and CR3 can end up out of sync.

This de-synchronization is currently harmless.  But it is confusing and
might lead to warnings or real bugs.

Update set_tlbstate_lam_mode() to take in the LAM mask and untag mask
instead of an mm_struct pointer, and while we are at it, rename it to
cpu_tlbstate_update_lam(). This should also make it clearer that we are
updating cpu_tlbstate. In switch_mm_irqs_off(), read the LAM mask once
and use it for both the cpu_tlbstate update and the CR3 update.

Reviewed-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Change-Id: I8bcf94bbf28ebdbbe75e3939e712246a029f84b6
Signed-off-by: Yosry Ahmed <yosryahmed@google.com>
---
 arch/x86/include/asm/mmu_context.h | 8 +++++++-
 arch/x86/include/asm/tlbflush.h    | 9 ++++-----
 arch/x86/kernel/process_64.c       | 6 ++++--
 arch/x86/mm/tlb.c                  | 8 +++++---
 4 files changed, 20 insertions(+), 11 deletions(-)

diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h
index 8dac45a2c7fcf..19091ebb86338 100644
--- a/arch/x86/include/asm/mmu_context.h
+++ b/arch/x86/include/asm/mmu_context.h
@@ -88,7 +88,13 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next)
 #ifdef CONFIG_ADDRESS_MASKING
 static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm)
 {
-	return mm->context.lam_cr3_mask;
+	/*
+	 * When switch_mm_irqs_off() is called for a kthread, it may race with
+	 * LAM enablement. switch_mm_irqs_off() uses the LAM mask to do two
+	 * things: populate CR3 and populate 'cpu_tlbstate.lam'. Make sure it
+	 * reads a single value for both.
+	 */
+	return READ_ONCE(mm->context.lam_cr3_mask);
 }
 
 static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm)
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
index 25726893c6f4d..69e79fff41b80 100644
--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -399,11 +399,10 @@ static inline  u64 tlbstate_lam_cr3_mask(void)
 	return lam << X86_CR3_LAM_U57_BIT;
 }
 
-static inline void set_tlbstate_lam_mode(struct mm_struct *mm)
+static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask)
 {
-	this_cpu_write(cpu_tlbstate.lam,
-		       mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT);
-	this_cpu_write(tlbstate_untag_mask, mm->context.untag_mask);
+	this_cpu_write(cpu_tlbstate.lam, lam >> X86_CR3_LAM_U57_BIT);
+	this_cpu_write(tlbstate_untag_mask, untag_mask);
 }
 
 #else
@@ -413,7 +412,7 @@ static inline u64 tlbstate_lam_cr3_mask(void)
 	return 0;
 }
 
-static inline void set_tlbstate_lam_mode(struct mm_struct *mm)
+static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask)
 {
 }
 #endif
diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c
index e1ce0dfd24258..26a853328f2d4 100644
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -801,10 +801,12 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr)
 static void enable_lam_func(void *__mm)
 {
 	struct mm_struct *mm = __mm;
+	unsigned long lam;
 
 	if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm) {
-		write_cr3(__read_cr3() | mm->context.lam_cr3_mask);
-		set_tlbstate_lam_mode(mm);
+		lam = mm_lam_cr3_mask(mm);
+		write_cr3(__read_cr3() | lam);
+		cpu_tlbstate_update_lam(lam, mm_untag_mask(mm));
 	}
 }
 
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index a041d2ecd8380..1fe9ba33c5805 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -11,6 +11,7 @@
 #include <linux/sched/smt.h>
 #include <linux/task_work.h>
 #include <linux/mmu_notifier.h>
+#include <linux/mmu_context.h>
 
 #include <asm/tlbflush.h>
 #include <asm/mmu_context.h>
@@ -632,7 +633,6 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,
 	}
 
 	new_lam = mm_lam_cr3_mask(next);
-	set_tlbstate_lam_mode(next);
 	if (need_flush) {
 		this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id);
 		this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen);
@@ -651,6 +651,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,
 
 	this_cpu_write(cpu_tlbstate.loaded_mm, next);
 	this_cpu_write(cpu_tlbstate.loaded_mm_asid, new_asid);
+	cpu_tlbstate_update_lam(new_lam, mm_untag_mask(next));
 
 	if (next != prev) {
 		cr4_update_pce_mm(next);
@@ -697,6 +698,7 @@ void initialize_tlbstate_and_flush(void)
 	int i;
 	struct mm_struct *mm = this_cpu_read(cpu_tlbstate.loaded_mm);
 	u64 tlb_gen = atomic64_read(&init_mm.context.tlb_gen);
+	unsigned long lam = mm_lam_cr3_mask(mm);
 	unsigned long cr3 = __read_cr3();
 
 	/* Assert that CR3 already references the right mm. */
@@ -704,7 +706,7 @@ void initialize_tlbstate_and_flush(void)
 
 	/* LAM expected to be disabled */
 	WARN_ON(cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57));
-	WARN_ON(mm_lam_cr3_mask(mm));
+	WARN_ON(lam);
 
 	/*
 	 * Assert that CR4.PCIDE is set if needed.  (CR4.PCIDE initialization
@@ -723,7 +725,7 @@ void initialize_tlbstate_and_flush(void)
 	this_cpu_write(cpu_tlbstate.next_asid, 1);
 	this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id);
 	this_cpu_write(cpu_tlbstate.ctxs[0].tlb_gen, tlb_gen);
-	set_tlbstate_lam_mode(mm);
+	cpu_tlbstate_update_lam(lam, mm_untag_mask(mm));
 
 	for (i = 1; i < TLB_NR_DYN_ASIDS; i++)
 		this_cpu_write(cpu_tlbstate.ctxs[i].ctx_id, 0);
-- 
2.45.2.803.g4e1b14247a-goog

[RESEND PATCH v3 3/3] x86/mm: Cleanup prctl_enable_tagged_addr() nr_bits error checking

[Yosry Ahmed]

There are two separate checks in prctl_enable_tagged_addr() that nr_bits
is in the correct range. The checks are arranged such the correct case
is sandwiched between both error cases, which do exactly the same thing.

Simplify the if condition and pull the correct case outside with the
rest of the success code path.

Reviewed-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Change-Id: Ia4f78de99d02f8ce28787346404d59ef13ca3d92
Signed-off-by: Yosry Ahmed <yosryahmed@google.com>
---
 arch/x86/kernel/process_64.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c
index 26a853328f2d4..c62098332f05c 100644
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -831,17 +831,13 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits)
 		return -EBUSY;
 	}
 
-	if (!nr_bits) {
-		mmap_write_unlock(mm);
-		return -EINVAL;
-	} else if (nr_bits <= LAM_U57_BITS) {
-		mm->context.lam_cr3_mask = X86_CR3_LAM_U57;
-		mm->context.untag_mask =  ~GENMASK(62, 57);
-	} else {
+	if (!nr_bits || nr_bits > LAM_U57_BITS) {
 		mmap_write_unlock(mm);
 		return -EINVAL;
 	}
 
+	mm->context.lam_cr3_mask = X86_CR3_LAM_U57;
+	mm->context.untag_mask =  ~GENMASK(62, 57);
 	on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true);
 	set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags);
 
-- 
2.45.2.803.g4e1b14247a-goog

Re: [RESEND PATCH v3 0/3] x86/mm: LAM fixups and cleanups

[Andrew Morton]

On Tue,  2 Jul 2024 13:21:36 +0000 Yosry Ahmed <yosryahmed@google.com> wrote:

> This series has fixups and cleanups for LAM. Most importantly, patch 1
> fixes a sycnhronization issue that may cause crashes of userspace
> applications. This is a resend of v3, rebased on top of v6.10-rc6.

"Crashes of userspace applications" is bad.  Yet the patchset has been
floating about for four months.

It's unclear (to me) how serious this is.  Can you please explain how
common this is, what the userspace application needs to do to trigger
this, etc?

Re: [RESEND PATCH v3 0/3] x86/mm: LAM fixups and cleanups

[Yosry Ahmed]

On Tue, Jul 2, 2024 at 10:36 AM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Tue,  2 Jul 2024 13:21:36 +0000 Yosry Ahmed <yosryahmed@google.com> wrote:
>
> > This series has fixups and cleanups for LAM. Most importantly, patch 1
> > fixes a sycnhronization issue that may cause crashes of userspace
> > applications. This is a resend of v3, rebased on top of v6.10-rc6.
>
> "Crashes of userspace applications" is bad.  Yet the patchset has been
> floating about for four months.
>
> It's unclear (to me) how serious this is.  Can you please explain how
> common this is, what the userspace application needs to do to trigger
> this, etc?

I don't think it would be common. The bug only happens on new hardware
supporting LAM, and it happens in a specific scenario where a
userspace task enables LAM while a kthread is using (borrowing) its
mm_struct on another CPU.

So it is possible but I certainly wouldn't call it common or easily triggerable.

Re: [RESEND PATCH v3 0/3] x86/mm: LAM fixups and cleanups

[Andrew Morton]

On Tue, 2 Jul 2024 10:39:03 -0700 Yosry Ahmed <yosryahmed@google.com> wrote:

> On Tue, Jul 2, 2024 at 10:36 AM Andrew Morton <akpm@linux-foundation.org> wrote:
> >
> > On Tue,  2 Jul 2024 13:21:36 +0000 Yosry Ahmed <yosryahmed@google.com> wrote:
> >
> > > This series has fixups and cleanups for LAM. Most importantly, patch 1
> > > fixes a sycnhronization issue that may cause crashes of userspace
> > > applications. This is a resend of v3, rebased on top of v6.10-rc6.
> >
> > "Crashes of userspace applications" is bad.  Yet the patchset has been
> > floating about for four months.
> >
> > It's unclear (to me) how serious this is.  Can you please explain how
> > common this is, what the userspace application needs to do to trigger
> > this, etc?
> 
> I don't think it would be common. The bug only happens on new hardware
> supporting LAM, and it happens in a specific scenario where a
> userspace task enables LAM while a kthread is using (borrowing) its
> mm_struct on another CPU.
> 
> So it is possible but I certainly wouldn't call it common or easily triggerable.

But when people run older (or current) kernels on newer hardware, they
will hit this.  So a backport to cover 82721d8b25d7 ("x86/mm: Handle
LAM on context switch") is needed.

The series doesn't seem to be getting much traction so I can add it to
mm.git's mm-unstable branch for wider testing, but it's clearly an x86
tree thing.

Re: [RESEND PATCH v3 0/3] x86/mm: LAM fixups and cleanups

[Dave Hansen]

On 7/2/24 11:35, Andrew Morton wrote:
> But when people run older (or current) kernels on newer hardware, they
> will hit this.  So a backport to cover 82721d8b25d7 ("x86/mm: Handle
> LAM on context switch") is needed.
> 
> The series doesn't seem to be getting much traction so I can add it to
> mm.git's mm-unstable branch for wider testing, but it's clearly an x86
> tree thing.

I was really hoping Andy L would look at this since he suggested this
whole thing really.

I completely agree that this needs some wider testing.  How about I pull
it into x86/mm so it gets some linux-next testing instead of having it
in mm-unstable?  Maybe it'll also attract Andy's attention once it's in
there.

Re: [RESEND PATCH v3 0/3] x86/mm: LAM fixups and cleanups

[Yosry Ahmed]

On Tue, Jul 2, 2024 at 11:38 AM Dave Hansen <dave.hansen@intel.com> wrote:
>
> On 7/2/24 11:35, Andrew Morton wrote:
> > But when people run older (or current) kernels on newer hardware, they
> > will hit this.  So a backport to cover 82721d8b25d7 ("x86/mm: Handle
> > LAM on context switch") is needed.
> >
> > The series doesn't seem to be getting much traction so I can add it to
> > mm.git's mm-unstable branch for wider testing, but it's clearly an x86
> > tree thing.
>
> I was really hoping Andy L would look at this since he suggested this
> whole thing really.
>
> I completely agree that this needs some wider testing.  How about I pull
> it into x86/mm so it gets some linux-next testing instead of having it
> in mm-unstable?  Maybe it'll also attract Andy's attention once it's in
> there.

That would be great. Thanks Dave!