Re: [LSF/MM/BPF TOPIC] Improve this_cpu_ops performance for ARM64 (and potentially other architectures)

[Ryan Roberts <ryan.roberts@arm.com>]

On 12/02/2026 17:54, Catalin Marinas wrote:
> On Wed, Feb 11, 2026 at 03:58:50PM -0800, Yang Shi wrote:
>> On Wed, Feb 11, 2026 at 3:29 PM Tejun Heo <tj@kernel.org> wrote:
>>> On Wed, Feb 11, 2026 at 03:14:57PM -0800, Yang Shi wrote:
>>> ...
>>>> Overhead
>>>> ========
>>>> 1. Some extra virtual memory space. But it shouldn’t be too much. I
>>>> saw 960K with Fedora default kernel config. Given terabytes virtual
>>>> memory space on 64 bit machine, 960K is negligible.
>>>> 2. Some extra physical memory for percpu kernel page table. 4K *
>>>> (nr_cpus – 1) for PGD pages, plus the page tables used by percpu local
>>>> mapping area. A couple of megabytes with Fedora default kernel config
>>>> on AmpereOne with 160 cores.
>>>> 3. Percpu allocation and free will be slower due to extra virtual
>>>> memory allocation and page table manipulation. However, percpu is
>>>> allocated by chunk. One chunk typically holds a lot percpu variables.
>>>> So the slowdown should be negligible. The test result below also
>>>> proved it.
> [...]
>>> One property that this breaks is per_cpu_ptr() of a given CPU disagreeing
>>> with this_cpu_ptr(). e.g. If there are users that take this_cpu_ptr() and
>>> uses that outside preempt disable block (which is a bit odd but allowed),
>>> the end result would be surprising. Hmm... I wonder whether it'd be
>>> worthwhile to keep this_cpu_ptr() returning the global address - ie. make it
>>> access global offset from local mapping and then return the computed global
>>> address. This should still be pretty cheap and gets rid of surprising and
>>> potentially extremely subtle corner cases.
>>
>> Yes, this is going to be a problem. So we don't change how
>> this_cpu_ptr() works and keep it returning the global address. Because
>> I noticed this may cause confusion for list APIs too. For example,
>> when initializing a list embedded into a percpu variable, the ->next
>> and ->prev will be initialized to global addresses by using
>> per_cpu_ptr(), but if the list is accessed via this_cpu_ptr(), list
>> head will be dereferenced by using local address, then list_empty()
>> will complain, which compare the list head pointer and ->next pointer.
>> This will cause some problems.
>>
>> So we just use the local address for this_cpu_add/sub/inc/dec and so
>> on, which just manipulate a scalar counter.
> 
> I wonder how much overhead is caused by calling into the scheduler on
> preempt_enable(). It would be good to get some numbers for something
> like the patch below (also removing the preempt disabling for
> this_cpu_read() as I don't think it matters - a thread cannot
> distinguish whether it was preempted between TPIDR read and variable
> read or immediately after the variable read; we can't do this for writes
> as other threads may notice unexpected updates).
> 
> Another wild hack could be to read the kernel instruction at
> (current_pt_regs()->pc - 4) in arch_irqentry_exit_need_resched() and
> return false if it's a read from TPIDR_EL1/2, together with removing the
> preempt disabling. Or some other lighter way of detecting this_cpu_*
> constructs without full preemption disabling.

Could a sort of kernel version of restartable sequences help? i.e. detect
preemption instead of preventing it?