Re: [OFFLIST PATCH 2/2] cgroup: use subsystem-specific rstat locks to avoid contention

[Klara Modin <klarasmodin@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1330 bytes --]

Hi,

On 2025-04-28 23:15:58 -0700, Shakeel Butt wrote:
> Please ignore this patch as it was sent by mistake.

This seems to have made it into next:

748922dcfabd ("cgroup: use subsystem-specific rstat locks to avoid contention")

It causes a BUG and eventually a panic on my Raspberry Pi 1:

WARNING: CPU: 0 PID: 0 at mm/percpu.c:1766 pcpu_alloc_noprof (mm/percpu.c:1766 (discriminator 2)) 
illegal size (0) or align (4) for percpu allocation
CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 6.15.0-rc7-next-20250521-00086-ga9fb18e56aad #263 NONE
Hardware name: BCM2835
Call trace:
unwind_backtrace from show_stack (arch/arm/kernel/traps.c:259) 
show_stack from dump_stack_lvl (lib/dump_stack.c:122) 
dump_stack_lvl from __warn (kernel/panic.c:729 kernel/panic.c:784) 
__warn from warn_slowpath_fmt (kernel/panic.c:815) 
warn_slowpath_fmt from pcpu_alloc_noprof (mm/percpu.c:1766 (discriminator 2)) 
pcpu_alloc_noprof from ss_rstat_init (kernel/cgroup/rstat.c:515) 
ss_rstat_init from cgroup_init_subsys (kernel/cgroup/cgroup.c:6134 (discriminator 2)) 
cgroup_init_subsys from cgroup_init (kernel/cgroup/cgroup.c:6240) 
cgroup_init from start_kernel (init/main.c:1093) 
 start_kernel from 0x0
...
kernel BUG at kernel/cgroup/cgroup.c:6134!
Internal error: Oops - BUG: 0 [#1] ARM

Reverting resolved it for me.

Regards,
Klara Modin

[-- Attachment #2: bisect-log --]
[-- Type: text/plain, Size: 2923 bytes --]

# bad: [7bac2c97af4078d7a627500c9bcdd5b033f97718] Add linux-next specific files for 20250521
# good: [b36ddb9210e6812eb1c86ad46b66cc46aa193487] Merge tag 'for-linus-6.15-ofs2' of git://git.kernel.org/pub/scm/linux/kernel/git/hubcap/linux
git bisect start 'next/master' 'next/stable'
# good: [d86bd5895658c9c4317ecf66d43bd16995e18483] Merge branch 'spi-nor/next' of git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux.git
git bisect good d86bd5895658c9c4317ecf66d43bd16995e18483
# good: [bf65a86fb3f72e1da64f68c29fe6d51de870e463] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux-dt.git
git bisect good bf65a86fb3f72e1da64f68c29fe6d51de870e463
# good: [56baa9ffcd362c315ee1d01bb089b8c2d9d068d3] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/westeri/thunderbolt.git
git bisect good 56baa9ffcd362c315ee1d01bb089b8c2d9d068d3
# bad: [2c0faaaca998913917026967c2b5adeac3168855] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git
git bisect bad 2c0faaaca998913917026967c2b5adeac3168855
# good: [be4e0b332d0f11bd70e2db7a9bccfdfd95f58dec] Merge branch 'togreg' of git://git.kernel.org/pub/scm/linux/kernel/git/jic23/iio.git
git bisect good be4e0b332d0f11bd70e2db7a9bccfdfd95f58dec
# good: [427ab512c2c8784686738ade287e5eb52bd8292a] staging: gpib: Change error code for no listener
git bisect good 427ab512c2c8784686738ade287e5eb52bd8292a
# good: [3139fde502884920b33404600ae918a08d683703] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/soundwire.git
git bisect good 3139fde502884920b33404600ae918a08d683703
# good: [86d2a97e8b5851cfeafc08c9efd7846e4f72adf3] Merge branch 'counter-next' of git://git.kernel.org/pub/scm/linux/kernel/git/wbg/counter.git
git bisect good 86d2a97e8b5851cfeafc08c9efd7846e4f72adf3
# good: [9002b75aa8e6f034ffbd1c1ccac46927a1cf0f12] irqchip/renesas-rzv2h: Add rzv2h_icu_register_dma_req()
git bisect good 9002b75aa8e6f034ffbd1c1ccac46927a1cf0f12
# good: [541a4219bd66bef56d93dbd306dc64a4d70ae99e] cgroup: compare css to cgroup::self in helper for distingushing css
git bisect good 541a4219bd66bef56d93dbd306dc64a4d70ae99e
# bad: [86aadd4d2347b11a239e755d5eb540488bbf5b8c] Merge branch 'for-6.16' into for-next
git bisect bad 86aadd4d2347b11a239e755d5eb540488bbf5b8c
# bad: [93b35663f2018ff2accf4336a909081883eda76b] cgroup: helper for checking rstat participation of css
git bisect bad 93b35663f2018ff2accf4336a909081883eda76b
# bad: [748922dcfabdd655d25fb6dd09a60e694a3d35e6] cgroup: use subsystem-specific rstat locks to avoid contention
git bisect bad 748922dcfabdd655d25fb6dd09a60e694a3d35e6
# good: [5da3bfa029d6809e192d112f39fca4dbe0137aaf] cgroup: use separate rstat trees for each subsystem
git bisect good 5da3bfa029d6809e192d112f39fca4dbe0137aaf
# first bad commit: [748922dcfabdd655d25fb6dd09a60e694a3d35e6] cgroup: use subsystem-specific rstat locks to avoid contention

[-- Attachment #3: boot-log --]
[-- Type: text/plain, Size: 7746 bytes --]

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 6.15.0-rc7-next-20250521-00086-ga9fb18e56aad (klara@soda.int.kasm.eu) (armv6j-unknown-linux-gnueabihf-gcc (Gentoo 15.1.0 p55) 15.1.0, GNU ld (Gentoo 2.44 p1) 2.44.0) #263 Wed May 21 23:48:00 CEST 2025
[    0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[    0.000000] OF: fdt: Machine model: Raspberry Pi Model B Rev 2
[    0.000000] earlycon: pl11 at MMIO32 0x20201000 (options '')
[    0.000000] printk: legacy bootconsole [pl11] enabled
[    0.000000] Memory policy: Data cache writeback
[    0.000000] Reserved memory: created CMA memory pool at 0x1b000000, size 64 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x1b000000..0x1effffff (65536 KiB) map reusable linux,cma
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000001effffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000001effffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000001effffff]
[    0.000000] CPU: All CPU(s) started in SVC mode.
[    0.000000] Kernel command line: bootargs_pre=coherent_pool=6M smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 console=tty0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 root=/dev/mmcblk0p2 rw rootfstype=nilfs2 rootflags=discard earlycon=pl011,mmio32,0x20201000
[    0.000000] Unknown kernel command line parameters "bootargs_pre=coherent_pool=6M", will be passed to user space.
[    0.000000] printk: log buffer data + meta data: 16384 + 51200 = 67584 bytes
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 126976
[    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[    0.000000] SLUB: HWalign=32, Order=0-1, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] ftrace: allocating 33191 entries in 65 pages
[    0.000000] ftrace: allocated 65 pages with 2 groups
[    0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.
[    0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000013] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[    0.008502] clocksource: timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[    0.017997] bcm2835: system timer (irq = 27)
[    0.023325] kfence: initialized - using 2097152 bytes for 255 objects at 0x(ptrval)-0x(ptrval)
[    0.032532] Console: colour dummy device 80x30
[    0.037075] printk: legacy console [tty0] enabled
[    0.043066] Calibrating delay loop... 697.95 BogoMIPS (lpj=3489792)
[    0.108806] CPU: Testing write buffer coherency: ok
[    0.113907] pid_max: default: 4096 minimum: 301
[    0.118984] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.126496] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.135751] ------------[ cut here ]------------
[    0.140554] WARNING: CPU: 0 PID: 0 at mm/percpu.c:1766 pcpu_alloc_noprof (mm/percpu.c:1766 (discriminator 2)) 
[    0.148469] illegal size (0) or align (4) for percpu allocation
[    0.154511] CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 6.15.0-rc7-next-20250521-00086-ga9fb18e56aad #263 NONE
[    0.164999] Hardware name: BCM2835
[    0.168456] Call trace:
[    0.168485] unwind_backtrace from show_stack (arch/arm/kernel/traps.c:259) 
[    0.176410] show_stack from dump_stack_lvl (lib/dump_stack.c:122) 
[    0.181574] dump_stack_lvl from __warn (kernel/panic.c:729 kernel/panic.c:784) 
[    0.186379] __warn from warn_slowpath_fmt (kernel/panic.c:815) 
[    0.191448] warn_slowpath_fmt from pcpu_alloc_noprof (mm/percpu.c:1766 (discriminator 2)) 
[    0.197674] pcpu_alloc_noprof from ss_rstat_init (kernel/cgroup/rstat.c:515) 
[    0.203393] ss_rstat_init from cgroup_init_subsys (kernel/cgroup/cgroup.c:6134 (discriminator 2)) 
[    0.209267] cgroup_init_subsys from cgroup_init (kernel/cgroup/cgroup.c:6240) 
[    0.215049] cgroup_init from start_kernel (init/main.c:1093) 
[    0.220315]  start_kernel from 0x0
[    0.223837] ---[ end trace 0000000000000000 ]---
[    0.228555] ------------[ cut here ]------------
[    0.233239] kernel BUG at kernel/cgroup/cgroup.c:6134!
[    0.238450] Internal error: Oops - BUG: 0 [#1] ARM
[    0.243323] CPU: 0 UID: 0 PID: 0 Comm: swapper Tainted: G        W           6.15.0-rc7-next-20250521-00086-ga9fb18e56aad #263 NONE
[    0.255389] Tainted: [W]=WARN
[    0.258403] Hardware name: BCM2835
[    0.261856] PC is at cgroup_init_subsys (kernel/cgroup/cgroup.c:6134 (discriminator 3)) 
[    0.266747] LR is at pcpu_alloc_noprof (mm/percpu.c:1766 (discriminator 2)) 
[    0.271640] pc : lr : psr: a0000053 
[    0.277988] sp : c0d01f70  ip : 00000000  fp : c0d296fc
[    0.283283] r10: c0d29704  r9 : c0d2a2e8  r8 : c0d29788
[    0.288579] r7 : 00000000  r6 : c0d2a2d8  r5 : c0ddc590  r4 : c0d6d894
[    0.295189] r3 : c0d08c80  r2 : 00000000  r1 : 00000000  r0 : fffffff4
[    0.301800] Flags: NzCv  IRQs on  FIQs off  Mode SVC_32  ISA ARM  Segment none
[    0.309121] Control: 00c5387d  Table: 00004008  DAC: 00000051
[    0.314940] Register r0 information: non-paged memory
[    0.320077] Register r1 information: NULL pointer
[    0.324858] Register r2 information: NULL pointer
[    0.329637] Register r3 information: non-slab/vmalloc memory
[    0.335384] Register r4 information: non-slab/vmalloc memory
[    0.341130] Register r5 information: non-slab/vmalloc memory
[    0.346875] Register r6 information: non-slab/vmalloc memory
[    0.352620] Register r7 information: NULL pointer
[    0.357399] Register r8 information: non-slab/vmalloc memory
[    0.363141] Register r9 information: non-slab/vmalloc memory
[    0.368885] Register r10 information: non-slab/vmalloc memory
[    0.374716] Register r11 information: non-slab/vmalloc memory
[    0.380549] Register r12 information: NULL pointer
[    0.385416] Process swapper (pid: 0, stack limit = 0x(ptrval))
[    0.391330] Stack: (0xc0d01f70 to 0xc0d02000)
[    0.395761] 1f60:                                     c0d6d894 00000002 c0d296f4 c0dc2278
[    0.404053] 1f80: c0d29788 c0c0ff40 dafff19e c0c1953c c0d2b618 c0d12e18 c0d088e0 c0dba048
[    0.412344] 1fa0: dafff180 00000000 c0dba024 c0d088e0 c0d08960 c0dba010 dafff19e c0c016bc
[    0.420635] 1fc0: ffffffff ffffffff 00000000 c0c00694 00000000 00000000 c0c34f28 00000000
[    0.428921] 1fe0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    0.437197] Call trace:
[    0.437219] cgroup_init_subsys from cgroup_init (kernel/cgroup/cgroup.c:6240) 
[    0.445566] cgroup_init from start_kernel (init/main.c:1093) 
[    0.450828]  start_kernel from 0x0
[ 0.454326] Code: e1a00004 eb0001bb e3500000 0a000001 (e7f001f2)
All code
========
   0:	e1a00004 	mov	r0, r4
   4:	eb0001bb 	bl	0x6f8
   8:	e3500000 	cmp	r0, #0
   c:	0a000001 	beq	0x18
  10:*	e7f001f2 	udf	#18		<-- trapping instruction

Code starting with the faulting instruction
===========================================
   0:	e7f001f2 	udf	#18
[    0.460550] ---[ end trace 0000000000000000 ]---
[    0.465252] Kernel panic - not syncing: Attempted to kill the idle task!
[    0.472066] Rebooting in 30 seconds..

[-- Attachment #4: config.gz --]
[-- Type: application/gzip, Size: 24847 bytes --]