Re: [PATCH] percpu: km: ensure it is used with NOMMU (either UP or SMP)

[Dennis Zhou <dennis@kernel.org>]

On Tue, Dec 14, 2021 at 09:12:06PM +0100, Geert Uytterhoeven wrote:
> Hi Dennis,
> 
> On Tue, Dec 14, 2021 at 8:18 PM Dennis Zhou <dennis@kernel.org> wrote:
> > On Tue, Dec 14, 2021 at 08:02:58PM +0100, Geert Uytterhoeven wrote:
> > > On Tue, Dec 14, 2021 at 6:26 PM Dennis Zhou <dennis@kernel.org> wrote:
> > > > On Tue, Dec 14, 2021 at 05:29:22PM +0100, Geert Uytterhoeven wrote:
> > > > > On Wed, Dec 1, 2021 at 12:53 PM Vladimir Murzin <vladimir.murzin@arm.com> wrote:
> > > > > > On 11/30/21 5:41 PM, Dennis Zhou wrote:
> > > > > > > On Tue, Nov 30, 2021 at 05:29:54PM +0000, Vladimir Murzin wrote:
> > > > > > >> Currently, NOMMU pull km allocator via !SMP dependency because most of
> > > > > > >> them are UP, yet for SMP+NOMMU vm allocator gets pulled which:
> > > > > > >>
> > > > > > >> * may lead to broken build [1]
> > > > > > >> * ...or not working runtime due to [2]
> > > > > > >>
> > > > > > >> It looks like SMP+NOMMU case was overlooked in bbddff054587 ("percpu:
> > > > > > >> use percpu allocator on UP too") so restore that.
> > > > > > >>
> > > > > > >> [1]
> > > > > > >> For ARM SMP+NOMMU (R-class cores)
> > > > > > >>
> > > > > > >> arm-none-linux-gnueabihf-ld: mm/percpu.o: in function `pcpu_post_unmap_tlb_flush':
> > > > > > >> mm/percpu-vm.c:188: undefined reference to `flush_tlb_kernel_range'
> > > > > > >>
> > > > > > >> [2]
> > > > > > >> static inline
> > > > > > >> int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
> > > > > > >>                 pgprot_t prot, struct page **pages, unsigned int page_shift)
> > > > > > >> {
> > > > > > >>        return -EINVAL;
> > > > > > >> }
> > > > > > >>
> > > > > > >> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> 
> > > > > > IIRC, RISC-V also have SMP+NOMMU, so adding them as well.
> > > > >
> > > > > I had seen the j-Core thread, but completely forgot about
> > > > > Canaan K210 (RV64 SMP+NOMMU).
> > > > >
> > > > > This became commit 3583521aabac76e5 ("percpu: km: ensure it is used
> > > > > with NOMMU (either UP or SMP)").  And now booting K210 prints:
> > > > >
> > > > >     percpu: wasting 10 pages per chunk
> > > > >
> > > > > a) Is this bad?
> > > >
> > > > It's not great.. Can you share the line on boot with the following
> > > > prefix: pcpu-alloc [1].
> > >
> > > There are no such lines.
> > > "make mm/percpu.i mm/percpu.s" and inspecting the generated files,
> > > and vmlinux, proves the code is there. But apparently it's not called.
> > >
> > > So there may be no issue on my system?
> >
> > I might be missing something, but that can't be right. Percpu calls
> > pcpu_dump_alloc_info() from pcpu_setup_first_chunk() which is called by
> > both embed/page first chunk code.
> >
> > Ummm. That can't be right. Percpu call pcpu_dump_alloc_info() from
> > pcpu_setup_first_chunk() which everyone should call. On my machine:
> >
> > $ dmesg | grep "pcpu-alloc"
> > [    0.065118] pcpu-alloc: s184320 r8192 d28672 u262144 alloc=1*2097152
> 
> Doh, it wasn't printed to the console, due to KERN_DEBUG. Dmesg
> does have it:
> 
> <7>[    0.000000] pcpu-alloc: s15520 r0 d29536 u45056 alloc=11*4096
> <7>[    0.000000] pcpu-alloc: [0] 0 [0] 1
> 

I see, so what's happening is we're allocating 11 pages * 2, and due to
percpu-km we round up to a contiguous 32 pages for backing pages. This
results in the warning of wasting 10 pages. Given the size of the static
region, I'm not too worried for now. I can't imagine the config would
use that much percpu memory.

We can massage the discrepancy for-v5.17. Basically in percpu-km, we
align to 4k even though our allocation gets rounded up to the next power
of 2. I don't have a lot of bandwidth right now, but I might be able to
think about it over the next few weeks.

Thanks,
Dennis