Re: [PATCH v2 bpf-next 0/2] enforce W^X for trampoline and dispatcher

Re: [PATCH v2 bpf-next 0/2] enforce W^X for trampoline and dispatcher

[Jesper Dangaard Brouer]

On 26/09/2022 20.47, Song Liu wrote:
> Changes v1 => v2:
> 1. Update arch_prepare_bpf_dispatcher to use a RO image and a RW buffer.
>     (Alexei) Note: I haven't found an existing test to cover this part, so
>     this part was tested manually (comparing the generated dispatcher is
>     the same).
> 
> Jeff Layton reported CPA W^X warning linux-next [1]. It turns out to be
> W^X issue with bpf trampoline and bpf dispatcher. Fix these by:
> 
> 1. Use bpf_prog_pack for bpf_dispatcher;
> 2. Set memory permission properly with bpf trampoline.

Indirectly related to your patchset[0].
  - TL;DR calling set_memory_x() have side-effects

We are getting reports that loading BPF-progs (jit stage) cause issues 
for RT in the form of triggering work on isolated CPUs.  It looks like 
BTF JIT stage cause a TLB flush on all CPUs, including isolated CPUs.

The triggering function is set_memory_x() (see call-stack[2]).

We have noticed (and appreciate) you have previously improved the 
situation in this patchset[3]:
  [3] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=80123f0ac4a6

Is this patchset also part of improving the situation, or does it 
introduce more calls to set_memory_x() ?


> [1] https://lore.kernel.org/lkml/c84cc27c1a5031a003039748c3c099732a718aec.camel@kernel.org/


[2] Call stack triggering issue:

         smp_call_function_many_cond+0x1
         smp_call_function+0x39
         on_each_cpu+0x2a
         cpa_flush+0x11a
         change_page_attr_set_clr+0x129
         set_memory_x+0x37
         bpf_int_jit_compile+0x36f
         bpf_prog_select_runtime+0xc6
         bpf_prepare_filter+0x523
         sk_attach_filter+0x13
         sock_setsockopt+0x920
         __sys_setsockopt+0x16a
         __x64_sys_setsockopt+0x20
         do_syscall_64+0x87
         entry_SYSCALL_64_after_hwframe+0x65


[0] https://lore.kernel.org/all/20220926184739.3512547-1-song@kernel.org/#r

--Jesper

Re: [PATCH v2 bpf-next 0/2] enforce W^X for trampoline and dispatcher

[Song Liu]

> On Sep 28, 2022, at 2:42 AM, Jesper Dangaard Brouer <jbrouer@redhat.com> wrote:
> 
> 
> On 26/09/2022 20.47, Song Liu wrote:
>> Changes v1 => v2:
>> 1. Update arch_prepare_bpf_dispatcher to use a RO image and a RW buffer.
>>    (Alexei) Note: I haven't found an existing test to cover this part, so
>>    this part was tested manually (comparing the generated dispatcher is
>>    the same).
>> Jeff Layton reported CPA W^X warning linux-next [1]. It turns out to be
>> W^X issue with bpf trampoline and bpf dispatcher. Fix these by:
>> 1. Use bpf_prog_pack for bpf_dispatcher;
>> 2. Set memory permission properly with bpf trampoline.
> 
> Indirectly related to your patchset[0].
> - TL;DR calling set_memory_x() have side-effects
> 
> We are getting reports that loading BPF-progs (jit stage) cause issues for RT in the form of triggering work on isolated CPUs.  It looks like BTF JIT stage cause a TLB flush on all CPUs, including isolated CPUs.
> 
> The triggering function is set_memory_x() (see call-stack[2]).
> 
> We have noticed (and appreciate) you have previously improved the situation in this patchset[3]:
> [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=80123f0ac4a6
> 
> Is this patchset also part of improving the situation, or does it introduce more calls to set_memory_x() ?

This set doesn't change numbers of set_memory_x() calls for trampolines. 
We plan to move trampolines to use bpf_prog_pack (or the new vmalloc_exec, 
if I am lucky) in 6.2. We will see fewer set_memory_x() calls after that.

Thanks,
Song

> 
> 
>> [1] https://lore.kernel.org/lkml/c84cc27c1a5031a003039748c3c099732a718aec.camel@kernel.org/
> 
> 
> [2] Call stack triggering issue:
> 
>        smp_call_function_many_cond+0x1
>        smp_call_function+0x39
>        on_each_cpu+0x2a
>        cpa_flush+0x11a
>        change_page_attr_set_clr+0x129
>        set_memory_x+0x37
>        bpf_int_jit_compile+0x36f
>        bpf_prog_select_runtime+0xc6
>        bpf_prepare_filter+0x523
>        sk_attach_filter+0x13
>        sock_setsockopt+0x920
>        __sys_setsockopt+0x16a
>        __x64_sys_setsockopt+0x20
>        do_syscall_64+0x87
>        entry_SYSCALL_64_after_hwframe+0x65
> 
> 
> [0] https://lore.kernel.org/all/20220926184739.3512547-1-song@kernel.org/#r
> 
> --Jesper
>