Re: [LSF/MM/BPF TOPIC] Improve this_cpu_ops performance for ARM64 (and potentially other architectures)

[Yang Shi <shy828301@gmail.com>]

On Thu, Feb 12, 2026 at 10:43 AM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> More thoughts...
>
> On Thu, Feb 12, 2026 at 05:54:19PM +0000, Catalin Marinas wrote:
> > On Wed, Feb 11, 2026 at 03:58:50PM -0800, Yang Shi wrote:
> > > So we just use the local address for this_cpu_add/sub/inc/dec and so
> > > on, which just manipulate a scalar counter.
> >
> > I wonder how much overhead is caused by calling into the scheduler on
> > preempt_enable(). It would be good to get some numbers for something
> > like the patch below
>
> In case it wasn't obvious, the patch messes up the scheduling, so I
> don't propose it as such, only to get some idea of where the bottleneck
> is. Maybe it could be made to work with some need_resched() checks.

Yeah, I was wondering whether it would make something wrong or not
because I noticed the comment right before _pcp_protect().

And I saw some confusing results by running kernel build workload with
the suggested patch, it should be caused by the messed up scheduler. I
can got much more stable result with "page_fault3_processes -s 20 -t
1" from will-it-scale. The test just launches one process, so it can
minimize the impact from messed up scheduler. The baseline is mainline
kernel.

systime improvement:
baseline           no schedule              no preemption
1                         0.96                             0.92

profiling diff (perf diff)
baseline vs no schedule
5.48%     -1.40%  [kernel.kallsyms]      [k] mod_memcg_lruvec_state

baseline vs no preemption
5.48%     -2.21%  [kernel.kallsyms]                   [k] mod_memcg_lruvec_state

>
> > (also removing the preempt disabling for
> > this_cpu_read() as I don't think it matters - a thread cannot
> > distinguish whether it was preempted between TPIDR read and variable
> > read or immediately after the variable read; we can't do this for writes
> > as other threads may notice unexpected updates).
>
> There's a theoretical case where even this_cpu_read() needs preemption
> disabling, e.g.:
>
> thread0:
>         preempt_disable();
>         this_cpu_write(var, unique_val);
>         // check that no-one has seen unique_value;
>         this_cpu_write(var, other_val);
>         preempt_enable();
>
> thread1:
>         this_cpu_read(var);
>
> thread1 is not supposed to see the unique_val but it would if it was
> preempted in the middle of the per-cpu op and migrated to another CPU.

I'm not sure whether kernel may make some decision by using the
counter read from this_cpu_read() or not. If kernel does so, it may
mess up something if the wrong counter is read.

Thanks,
Yang

>
> > Another wild hack could be to read the kernel instruction at
> > (current_pt_regs()->pc - 4) in arch_irqentry_exit_need_resched() and
> > return false if it's a read from TPIDR_EL1/2, together with removing the
> > preempt disabling.
>
> This one also breaks the kernel scheduling just like using
> preempt_enable_no_resched(). It might be possible but in combination
> with additional need_resched() checks.
>
> --
> Catalin