Re: [LSF/MM/BPF TOPIC] Improve this_cpu_ops performance for ARM64 (and potentially other architectures)

[Ryan Roberts <ryan.roberts@arm.com>]

On 12/02/2026 19:36, Catalin Marinas wrote:
> On Thu, Feb 12, 2026 at 06:45:19PM +0000, Ryan Roberts wrote:
>> On 12/02/2026 17:54, Catalin Marinas wrote:
>>> On Wed, Feb 11, 2026 at 03:58:50PM -0800, Yang Shi wrote:
>>>> On Wed, Feb 11, 2026 at 3:29 PM Tejun Heo <tj@kernel.org> wrote:
>>>>> On Wed, Feb 11, 2026 at 03:14:57PM -0800, Yang Shi wrote:
>>>>> ...
>>>>>> Overhead
>>>>>> ========
>>>>>> 1. Some extra virtual memory space. But it shouldn’t be too much. I
>>>>>> saw 960K with Fedora default kernel config. Given terabytes virtual
>>>>>> memory space on 64 bit machine, 960K is negligible.
>>>>>> 2. Some extra physical memory for percpu kernel page table. 4K *
>>>>>> (nr_cpus – 1) for PGD pages, plus the page tables used by percpu local
>>>>>> mapping area. A couple of megabytes with Fedora default kernel config
>>>>>> on AmpereOne with 160 cores.
>>>>>> 3. Percpu allocation and free will be slower due to extra virtual
>>>>>> memory allocation and page table manipulation. However, percpu is
>>>>>> allocated by chunk. One chunk typically holds a lot percpu variables.
>>>>>> So the slowdown should be negligible. The test result below also
>>>>>> proved it.
>>> [...]
>>>>> One property that this breaks is per_cpu_ptr() of a given CPU disagreeing
>>>>> with this_cpu_ptr(). e.g. If there are users that take this_cpu_ptr() and
>>>>> uses that outside preempt disable block (which is a bit odd but allowed),
>>>>> the end result would be surprising. Hmm... I wonder whether it'd be
>>>>> worthwhile to keep this_cpu_ptr() returning the global address - ie. make it
>>>>> access global offset from local mapping and then return the computed global
>>>>> address. This should still be pretty cheap and gets rid of surprising and
>>>>> potentially extremely subtle corner cases.
>>>>
>>>> Yes, this is going to be a problem. So we don't change how
>>>> this_cpu_ptr() works and keep it returning the global address. Because
>>>> I noticed this may cause confusion for list APIs too. For example,
>>>> when initializing a list embedded into a percpu variable, the ->next
>>>> and ->prev will be initialized to global addresses by using
>>>> per_cpu_ptr(), but if the list is accessed via this_cpu_ptr(), list
>>>> head will be dereferenced by using local address, then list_empty()
>>>> will complain, which compare the list head pointer and ->next pointer.
>>>> This will cause some problems.
>>>>
>>>> So we just use the local address for this_cpu_add/sub/inc/dec and so
>>>> on, which just manipulate a scalar counter.
>>>
>>> I wonder how much overhead is caused by calling into the scheduler on
>>> preempt_enable(). It would be good to get some numbers for something
>>> like the patch below (also removing the preempt disabling for
>>> this_cpu_read() as I don't think it matters - a thread cannot
>>> distinguish whether it was preempted between TPIDR read and variable
>>> read or immediately after the variable read; we can't do this for writes
>>> as other threads may notice unexpected updates).
>>>
>>> Another wild hack could be to read the kernel instruction at
>>> (current_pt_regs()->pc - 4) in arch_irqentry_exit_need_resched() and
>>> return false if it's a read from TPIDR_EL1/2, together with removing the
>>> preempt disabling. Or some other lighter way of detecting this_cpu_*
>>> constructs without full preemption disabling.
>>
>> Could a sort of kernel version of restartable sequences help? i.e. detect
>> preemption instead of preventing it?
> 
> Yes, in principle that's what we'd need but it's too expensive to check,
> especially as those accessors are inlined.

Could we use bit 63 of tpidr_el[12] to indicate "don't preempt"? a sort of
arch-specifc preemption disable mechanism that doesn't require load/store...

> 
> For the write variants with LL/SC, we can check the TPIDR_EL2 again
> between the LDXR and STXR and bail out if it's different from the one
> read outside the loop. An interrupt would clear the exclusive monitor
> anyway and STXR fail. This won't work for the theoretical
> this_cpu_read() case.

Could you clarify that last sentence? - we don't need it to work for
this_cpu_read() because we don't need to disable preemption for that case, right?

Thanks,
Ryan