Re: [LSF/MM/BPF TOPIC] Improve this_cpu_ops performance for ARM64 (and potentially other architectures)

[Catalin Marinas <catalin.marinas@arm.com>]

On Thu, Feb 12, 2026 at 09:12:55PM +0000, Ryan Roberts wrote:
> On 12/02/2026 19:36, Catalin Marinas wrote:
> > On Thu, Feb 12, 2026 at 06:45:19PM +0000, Ryan Roberts wrote:
> >> On 12/02/2026 17:54, Catalin Marinas wrote:
> >>> On Wed, Feb 11, 2026 at 03:58:50PM -0800, Yang Shi wrote:
> >>>> On Wed, Feb 11, 2026 at 3:29 PM Tejun Heo <tj@kernel.org> wrote:
> >>>>> On Wed, Feb 11, 2026 at 03:14:57PM -0800, Yang Shi wrote:
> >>>>> ...
> >>>>>> Overhead
> >>>>>> ========
> >>>>>> 1. Some extra virtual memory space. But it shouldn’t be too much. I
> >>>>>> saw 960K with Fedora default kernel config. Given terabytes virtual
> >>>>>> memory space on 64 bit machine, 960K is negligible.
> >>>>>> 2. Some extra physical memory for percpu kernel page table. 4K *
> >>>>>> (nr_cpus – 1) for PGD pages, plus the page tables used by percpu local
> >>>>>> mapping area. A couple of megabytes with Fedora default kernel config
> >>>>>> on AmpereOne with 160 cores.
> >>>>>> 3. Percpu allocation and free will be slower due to extra virtual
> >>>>>> memory allocation and page table manipulation. However, percpu is
> >>>>>> allocated by chunk. One chunk typically holds a lot percpu variables.
> >>>>>> So the slowdown should be negligible. The test result below also
> >>>>>> proved it.
> >>> [...]
> >>>>> One property that this breaks is per_cpu_ptr() of a given CPU disagreeing
> >>>>> with this_cpu_ptr(). e.g. If there are users that take this_cpu_ptr() and
> >>>>> uses that outside preempt disable block (which is a bit odd but allowed),
> >>>>> the end result would be surprising. Hmm... I wonder whether it'd be
> >>>>> worthwhile to keep this_cpu_ptr() returning the global address - ie. make it
> >>>>> access global offset from local mapping and then return the computed global
> >>>>> address. This should still be pretty cheap and gets rid of surprising and
> >>>>> potentially extremely subtle corner cases.
> >>>>
> >>>> Yes, this is going to be a problem. So we don't change how
> >>>> this_cpu_ptr() works and keep it returning the global address. Because
> >>>> I noticed this may cause confusion for list APIs too. For example,
> >>>> when initializing a list embedded into a percpu variable, the ->next
> >>>> and ->prev will be initialized to global addresses by using
> >>>> per_cpu_ptr(), but if the list is accessed via this_cpu_ptr(), list
> >>>> head will be dereferenced by using local address, then list_empty()
> >>>> will complain, which compare the list head pointer and ->next pointer.
> >>>> This will cause some problems.
> >>>>
> >>>> So we just use the local address for this_cpu_add/sub/inc/dec and so
> >>>> on, which just manipulate a scalar counter.
> >>>
> >>> I wonder how much overhead is caused by calling into the scheduler on
> >>> preempt_enable(). It would be good to get some numbers for something
> >>> like the patch below (also removing the preempt disabling for
> >>> this_cpu_read() as I don't think it matters - a thread cannot
> >>> distinguish whether it was preempted between TPIDR read and variable
> >>> read or immediately after the variable read; we can't do this for writes
> >>> as other threads may notice unexpected updates).
> >>>
> >>> Another wild hack could be to read the kernel instruction at
> >>> (current_pt_regs()->pc - 4) in arch_irqentry_exit_need_resched() and
> >>> return false if it's a read from TPIDR_EL1/2, together with removing the
> >>> preempt disabling. Or some other lighter way of detecting this_cpu_*
> >>> constructs without full preemption disabling.
> >>
> >> Could a sort of kernel version of restartable sequences help? i.e. detect
> >> preemption instead of preventing it?
> > 
> > Yes, in principle that's what we'd need but it's too expensive to check,
> > especially as those accessors are inlined.
> 
> Could we use bit 63 of tpidr_el[12] to indicate "don't preempt"? a sort of
> arch-specifc preemption disable mechanism that doesn't require load/store...

As long as it doesn't nest with interrupts, in which case some refcount
would be needed.

But I need to check Yang's emails to see whether the actual TPIDR access
is problematic.

> > For the write variants with LL/SC, we can check the TPIDR_EL2 again
> > between the LDXR and STXR and bail out if it's different from the one
> > read outside the loop. An interrupt would clear the exclusive monitor
> > anyway and STXR fail. This won't work for the theoretical
> > this_cpu_read() case.
> 
> Could you clarify that last sentence? - we don't need it to work for
> this_cpu_read() because we don't need to disable preemption for that case, right?

Mostly right but there can be some theoretical scenario where a thread
expects to be the only one running on a CPU and any sequence of
modifications to a per-cpu variable to be atomic:

https://lore.kernel.org/r/aY4fQOgyx3meku3b@arm.com

-- 
Catalin