Re: [LSF/MM/BPF TOPIC] Improve this_cpu_ops performance for ARM64 (and potentially other architectures)

[Catalin Marinas <catalin.marinas@arm.com>]

On Thu, Feb 12, 2026 at 06:45:19PM +0000, Ryan Roberts wrote:
> On 12/02/2026 17:54, Catalin Marinas wrote:
> > On Wed, Feb 11, 2026 at 03:58:50PM -0800, Yang Shi wrote:
> >> On Wed, Feb 11, 2026 at 3:29 PM Tejun Heo <tj@kernel.org> wrote:
> >>> On Wed, Feb 11, 2026 at 03:14:57PM -0800, Yang Shi wrote:
> >>> ...
> >>>> Overhead
> >>>> ========
> >>>> 1. Some extra virtual memory space. But it shouldn’t be too much. I
> >>>> saw 960K with Fedora default kernel config. Given terabytes virtual
> >>>> memory space on 64 bit machine, 960K is negligible.
> >>>> 2. Some extra physical memory for percpu kernel page table. 4K *
> >>>> (nr_cpus – 1) for PGD pages, plus the page tables used by percpu local
> >>>> mapping area. A couple of megabytes with Fedora default kernel config
> >>>> on AmpereOne with 160 cores.
> >>>> 3. Percpu allocation and free will be slower due to extra virtual
> >>>> memory allocation and page table manipulation. However, percpu is
> >>>> allocated by chunk. One chunk typically holds a lot percpu variables.
> >>>> So the slowdown should be negligible. The test result below also
> >>>> proved it.
> > [...]
> >>> One property that this breaks is per_cpu_ptr() of a given CPU disagreeing
> >>> with this_cpu_ptr(). e.g. If there are users that take this_cpu_ptr() and
> >>> uses that outside preempt disable block (which is a bit odd but allowed),
> >>> the end result would be surprising. Hmm... I wonder whether it'd be
> >>> worthwhile to keep this_cpu_ptr() returning the global address - ie. make it
> >>> access global offset from local mapping and then return the computed global
> >>> address. This should still be pretty cheap and gets rid of surprising and
> >>> potentially extremely subtle corner cases.
> >>
> >> Yes, this is going to be a problem. So we don't change how
> >> this_cpu_ptr() works and keep it returning the global address. Because
> >> I noticed this may cause confusion for list APIs too. For example,
> >> when initializing a list embedded into a percpu variable, the ->next
> >> and ->prev will be initialized to global addresses by using
> >> per_cpu_ptr(), but if the list is accessed via this_cpu_ptr(), list
> >> head will be dereferenced by using local address, then list_empty()
> >> will complain, which compare the list head pointer and ->next pointer.
> >> This will cause some problems.
> >>
> >> So we just use the local address for this_cpu_add/sub/inc/dec and so
> >> on, which just manipulate a scalar counter.
> > 
> > I wonder how much overhead is caused by calling into the scheduler on
> > preempt_enable(). It would be good to get some numbers for something
> > like the patch below (also removing the preempt disabling for
> > this_cpu_read() as I don't think it matters - a thread cannot
> > distinguish whether it was preempted between TPIDR read and variable
> > read or immediately after the variable read; we can't do this for writes
> > as other threads may notice unexpected updates).
> > 
> > Another wild hack could be to read the kernel instruction at
> > (current_pt_regs()->pc - 4) in arch_irqentry_exit_need_resched() and
> > return false if it's a read from TPIDR_EL1/2, together with removing the
> > preempt disabling. Or some other lighter way of detecting this_cpu_*
> > constructs without full preemption disabling.
> 
> Could a sort of kernel version of restartable sequences help? i.e. detect
> preemption instead of preventing it?

Yes, in principle that's what we'd need but it's too expensive to check,
especially as those accessors are inlined.

For the write variants with LL/SC, we can check the TPIDR_EL2 again
between the LDXR and STXR and bail out if it's different from the one
read outside the loop. An interrupt would clear the exclusive monitor
anyway and STXR fail. This won't work for the theoretical
this_cpu_read() case.

-- 
Catalin