Re: [LSF/MM/BPF TOPIC] Improve this_cpu_ops performance for ARM64 (and potentially other architectures)

[Ryan Roberts <ryan.roberts@arm.com>]

On 16/02/2026 10:37, Catalin Marinas wrote:
> On Thu, Feb 12, 2026 at 09:12:55PM +0000, Ryan Roberts wrote:
>> On 12/02/2026 19:36, Catalin Marinas wrote:
>>> On Thu, Feb 12, 2026 at 06:45:19PM +0000, Ryan Roberts wrote:
>>>> On 12/02/2026 17:54, Catalin Marinas wrote:
>>>>> On Wed, Feb 11, 2026 at 03:58:50PM -0800, Yang Shi wrote:
>>>>>> On Wed, Feb 11, 2026 at 3:29 PM Tejun Heo <tj@kernel.org> wrote:
>>>>>>> On Wed, Feb 11, 2026 at 03:14:57PM -0800, Yang Shi wrote:
>>>>>>> ...
>>>>>>>> Overhead
>>>>>>>> ========
>>>>>>>> 1. Some extra virtual memory space. But it shouldn’t be too much. I
>>>>>>>> saw 960K with Fedora default kernel config. Given terabytes virtual
>>>>>>>> memory space on 64 bit machine, 960K is negligible.
>>>>>>>> 2. Some extra physical memory for percpu kernel page table. 4K *
>>>>>>>> (nr_cpus – 1) for PGD pages, plus the page tables used by percpu local
>>>>>>>> mapping area. A couple of megabytes with Fedora default kernel config
>>>>>>>> on AmpereOne with 160 cores.
>>>>>>>> 3. Percpu allocation and free will be slower due to extra virtual
>>>>>>>> memory allocation and page table manipulation. However, percpu is
>>>>>>>> allocated by chunk. One chunk typically holds a lot percpu variables.
>>>>>>>> So the slowdown should be negligible. The test result below also
>>>>>>>> proved it.
>>>>> [...]
>>>>>>> One property that this breaks is per_cpu_ptr() of a given CPU disagreeing
>>>>>>> with this_cpu_ptr(). e.g. If there are users that take this_cpu_ptr() and
>>>>>>> uses that outside preempt disable block (which is a bit odd but allowed),
>>>>>>> the end result would be surprising. Hmm... I wonder whether it'd be
>>>>>>> worthwhile to keep this_cpu_ptr() returning the global address - ie. make it
>>>>>>> access global offset from local mapping and then return the computed global
>>>>>>> address. This should still be pretty cheap and gets rid of surprising and
>>>>>>> potentially extremely subtle corner cases.
>>>>>>
>>>>>> Yes, this is going to be a problem. So we don't change how
>>>>>> this_cpu_ptr() works and keep it returning the global address. Because
>>>>>> I noticed this may cause confusion for list APIs too. For example,
>>>>>> when initializing a list embedded into a percpu variable, the ->next
>>>>>> and ->prev will be initialized to global addresses by using
>>>>>> per_cpu_ptr(), but if the list is accessed via this_cpu_ptr(), list
>>>>>> head will be dereferenced by using local address, then list_empty()
>>>>>> will complain, which compare the list head pointer and ->next pointer.
>>>>>> This will cause some problems.
>>>>>>
>>>>>> So we just use the local address for this_cpu_add/sub/inc/dec and so
>>>>>> on, which just manipulate a scalar counter.
>>>>>
>>>>> I wonder how much overhead is caused by calling into the scheduler on
>>>>> preempt_enable(). It would be good to get some numbers for something
>>>>> like the patch below (also removing the preempt disabling for
>>>>> this_cpu_read() as I don't think it matters - a thread cannot
>>>>> distinguish whether it was preempted between TPIDR read and variable
>>>>> read or immediately after the variable read; we can't do this for writes
>>>>> as other threads may notice unexpected updates).
>>>>>
>>>>> Another wild hack could be to read the kernel instruction at
>>>>> (current_pt_regs()->pc - 4) in arch_irqentry_exit_need_resched() and
>>>>> return false if it's a read from TPIDR_EL1/2, together with removing the
>>>>> preempt disabling. Or some other lighter way of detecting this_cpu_*
>>>>> constructs without full preemption disabling.
>>>>
>>>> Could a sort of kernel version of restartable sequences help? i.e. detect
>>>> preemption instead of preventing it?
>>>
>>> Yes, in principle that's what we'd need but it's too expensive to check,
>>> especially as those accessors are inlined.
>>
>> Could we use bit 63 of tpidr_el[12] to indicate "don't preempt"? a sort of
>> arch-specifc preemption disable mechanism that doesn't require load/store...
> 
> As long as it doesn't nest with interrupts, in which case some refcount
> would be needed.
> 
> But I need to check Yang's emails to see whether the actual TPIDR access
> is problematic.

We can't set the bit atomically so we could still be prempted between the read
and write back. So this is no good. Ignore my rambling...

> 
>>> For the write variants with LL/SC, we can check the TPIDR_EL2 again
>>> between the LDXR and STXR and bail out if it's different from the one
>>> read outside the loop. An interrupt would clear the exclusive monitor
>>> anyway and STXR fail. This won't work for the theoretical
>>> this_cpu_read() case.
>>
>> Could you clarify that last sentence? - we don't need it to work for
>> this_cpu_read() because we don't need to disable preemption for that case, right?
> 
> Mostly right but there can be some theoretical scenario where a thread
> expects to be the only one running on a CPU and any sequence of
> modifications to a per-cpu variable to be atomic:
> 
> https://lore.kernel.org/r/aY4fQOgyx3meku3b@arm.com

Yeah, I saw that after posting this. Thanks for the education :)