Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 86895e1e85b3a8220936304841b20d01a6239976 ("[PATCH 3/4] mm/memcg: Add a local_lock_t for IRQ and TASK object.") url: https://github.com/0day-ci/linux/commits/Sebastian-Andrzej-Siewior/mm-memcg-Address-PREEMPT_RT-problems-instead-of-disabling-it/20220126-004438 patch link: https://lore.kernel.org/linux-mm/20220125164337.2071854-4-bigeasy@linutronix.de in testcase: kernel-selftests version: kernel-selftests-x86_64-f050cde9-1_20220127 with following parameters: group: cgroup ucode: 0xe2 test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel. test-url: https://www.kernel.org/doc/Documentation/kselftest.txt on test machine: 8 threads Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz with 16G memory caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot [ 698.190282][ T6895] WARNING: possible circular locking dependency detected [ 698.197143][ T6895] 5.17.0-rc1-00003-g86895e1e85b3 #1 Not tainted [ 698.203222][ T6895] ------------------------------------------------------ [ 698.210077][ T6895] with_stress.sh/6895 is trying to acquire lock: [ 698.216247][ T6895] ffff888100b42428 (&mm->mmap_lock#2){++++}-{3:3}, at: mpol_rebind_mm (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:212 include/linux/mmap_lock.h:35 include/linux/mmap_lock.h:72 mm/mempolicy.c:384) [ 698.225194][ T6895] [ 698.225194][ T6895] but task is already holding lock: [ 698.232394][ T6895] ffffffff8569b1f0 (&cpuset_rwsem){++++}-{0:0}, at: cpuset_attach (kernel/cgroup/cpuset.c:2257) [ 698.240991][ T6895] [ 698.240991][ T6895] which lock already depends on the new lock. [ 698.240991][ T6895] [ 698.251226][ T6895] [ 698.251226][ T6895] the existing dependency chain (in reverse order) is: [ 698.260076][ T6895] [ 698.260076][ T6895] -> #2 (&cpuset_rwsem){++++}-{0:0}: [ 698.267369][ T6895] lock_acquire (kernel/locking/lockdep.c:438 kernel/locking/lockdep.c:5641 kernel/locking/lockdep.c:5604) [ 698.272241][ T6895] percpu_down_write (kernel/locking/percpu-rwsem.c:220) [ 698.277455][ T6895] cpuset_css_online (include/linux/instrumented.h:86 include/asm-generic/bitops/instrumented-atomic.h:28 kernel/cgroup/cpuset.c:2820) [ 698.282667][ T6895] online_css (kernel/cgroup/cgroup.c:5250) [ 698.287277][ T6895] cgroup_apply_control_enable (kernel/cgroup/cgroup.c:5321 kernel/cgroup/cgroup.c:3124) [ 698.293448][ T6895] cgroup_apply_control (kernel/cgroup/cgroup.c:3207) [ 698.298830][ T6895] cgroup_subtree_control_write (kernel/cgroup/cgroup.c:3232 kernel/cgroup/cgroup.c:3365) [ 698.305080][ T6895] cgroup_file_write (kernel/cgroup/cgroup.c:3855) [ 698.310378][ T6895] kernfs_fop_write_iter (fs/kernfs/file.c:300) [ 698.316020][ T6895] new_sync_write (fs/read_write.c:504 (discriminator 1)) [ 698.321059][ T6895] vfs_write (fs/read_write.c:590) [ 698.325667][ T6895] ksys_write (fs/read_write.c:643) [ 698.330276][ T6895] do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80) [ 698.335065][ T6895] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:113) [ 698.341314][ T6895] [ 698.341314][ T6895] -> #1 (cpu_hotplug_lock){++++}-{0:0}: [ 698.348869][ T6895] lock_acquire (kernel/locking/lockdep.c:438 kernel/locking/lockdep.c:5641 kernel/locking/lockdep.c:5604) [ 698.353733][ T6895] cpus_read_lock (arch/x86/include/asm/preempt.h:80 include/linux/percpu-rwsem.h:53 kernel/cpu.c:309) [ 698.358600][ T6895] drain_all_stock (mm/memcontrol.c:2207 mm/memcontrol.c:2193) [ 698.363641][ T6895] try_charge_memcg (mm/memcontrol.c:2563) [ 698.368941][ T6895] charge_memcg (mm/memcontrol.c:6766) [ 698.373724][ T6895] __mem_cgroup_charge (include/linux/cgroup.h:403 mm/memcontrol.c:6787) [ 698.379116][ T6895] do_anonymous_page (mm/memory.c:3783) [ 698.384503][ T6895] __handle_mm_fault (mm/memory.c:4568 mm/memory.c:4705) [ 698.389976][ T6895] handle_mm_fault (mm/memory.c:4803) [ 698.395102][ T6895] do_user_addr_fault (arch/x86/mm/fault.c:1397) [ 698.400486][ T6895] exc_page_fault (arch/x86/include/asm/irqflags.h:29 arch/x86/include/asm/irqflags.h:70 arch/x86/include/asm/irqflags.h:130 arch/x86/mm/fault.c:1492 arch/x86/mm/fault.c:1540) [ 698.405440][ T6895] asm_exc_page_fault (arch/x86/include/asm/idtentry.h:568) [ 698.410651][ T6895] [ 698.410651][ T6895] -> #0 (&mm->mmap_lock#2){++++}-{3:3}: [ 698.418210][ T6895] check_prev_add (kernel/locking/lockdep.c:3064) [ 698.423341][ T6895] __lock_acquire (kernel/locking/lockdep.c:3187 kernel/locking/lockdep.c:3801 kernel/locking/lockdep.c:5027) [ 698.428552][ T6895] lock_acquire (kernel/locking/lockdep.c:438 kernel/locking/lockdep.c:5641 kernel/locking/lockdep.c:5604) [ 698.433419][ T6895] down_write (include/linux/instrumented.h:101 include/linux/atomic/atomic-instrumented.h:1779 kernel/locking/rwsem.c:254 kernel/locking/rwsem.c:1258 kernel/locking/rwsem.c:1268 kernel/locking/rwsem.c:1515) [ 698.438033][ T6895] mpol_rebind_mm (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:212 include/linux/mmap_lock.h:35 include/linux/mmap_lock.h:72 mm/mempolicy.c:384) [ 698.442995][ T6895] cpuset_attach (include/linux/instrumented.h:71 include/asm-generic/bitops/instrumented-non-atomic.h:134 kernel/cgroup/cpuset.c:258 kernel/cgroup/cpuset.c:2293) [ 698.447947][ T6895] cgroup_migrate_execute (kernel/cgroup/cgroup.c:2513) [ 698.453764][ T6895] cgroup_update_dfl_csses (kernel/cgroup/cgroup.c:2975) [ 698.459586][ T6895] cgroup_subtree_control_write (kernel/cgroup/cgroup.c:3232 kernel/cgroup/cgroup.c:3365) [ 698.465835][ T6895] cgroup_file_write (kernel/cgroup/cgroup.c:3855) [ 698.471130][ T6895] kernfs_fop_write_iter (fs/kernfs/file.c:300) [ 698.476775][ T6895] new_sync_write (fs/read_write.c:504 (discriminator 1)) [ 698.481811][ T6895] vfs_write (fs/read_write.c:590) [ 698.486417][ T6895] ksys_write (fs/read_write.c:643) [ 698.491022][ T6895] do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80) [ 698.495802][ T6895] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:113) [ 698.502050][ T6895] [ 698.502050][ T6895] other info that might help us debug this: [ 698.502050][ T6895] [ 698.512106][ T6895] Chain exists of: [ 698.512106][ T6895] &mm->mmap_lock#2 --> cpu_hotplug_lock --> &cpuset_rwsem [ 698.512106][ T6895] [ 698.524934][ T6895] Possible unsafe locking scenario: [ 698.524934][ T6895] [ 698.532226][ T6895] CPU0 CPU1 [ 698.537436][ T6895] ---- ---- [ 698.542648][ T6895] lock(&cpuset_rwsem); [ 698.546735][ T6895] lock(cpu_hotplug_lock); [ 698.553593][ T6895] lock(&cpuset_rwsem); [ 698.560197][ T6895] lock(&mm->mmap_lock#2); [ 698.564550][ T6895] [ 698.564550][ T6895] *** DEADLOCK *** [ 698.564550][ T6895] [ 698.572532][ T6895] 5 locks held by with_stress.sh/6895: [ 698.577839][ T6895] #0: ffff8881013f2448 (sb_writers#7){.+.+}-{0:0}, at: ksys_write (fs/read_write.c:643) [ 698.586519][ T6895] #1: ffff88817c726088 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter (fs/kernfs/file.c:288) [ 698.596064][ T6895] #2: ffffffff8568d788 (cgroup_mutex){+.+.}-{3:3}, at: cgroup_lock_and_drain_offline (kernel/cgroup/cgroup.c:3001) [ 698.606385][ T6895] #3: ffffffff8568d550 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_update_dfl_csses (include/linux/spinlock.h:374 (discriminator 9) kernel/cgroup/cgroup.c:2951 (discriminator 9)) [ 698.617312][ T6895] #4: ffffffff8569b1f0 (&cpuset_rwsem){++++}-{0:0}, at: cpuset_attach (kernel/cgroup/cpuset.c:2257) [ 698.626335][ T6895] [ 698.626335][ T6895] stack backtrace: [ 698.632074][ T6895] CPU: 1 PID: 6895 Comm: with_stress.sh Not tainted 5.17.0-rc1-00003-g86895e1e85b3 #1 [ 698.641447][ T6895] Hardware name: HP HP Z240 SFF Workstation/802E, BIOS N51 Ver. 01.63 10/05/2017 [ 698.650384][ T6895] Call Trace: [ 698.653526][ T6895] [ 698.656319][ T6895] dump_stack_lvl (lib/dump_stack.c:107) [ 698.660669][ T6895] check_noncircular (kernel/locking/lockdep.c:2143) [ 698.665449][ T6895] ? print_circular_bug+0x480/0x480 [ 698.671103][ T6895] ? mark_lock+0xca/0x13c0 [ 698.675976][ T6895] ? mark_lock_irq (kernel/locking/lockdep.c:4564) [ 698.680765][ T6895] check_prev_add (kernel/locking/lockdep.c:3064) [ 698.685379][ T6895] ? __lock_acquire (arch/x86/include/asm/bitops.h:214 include/asm-generic/bitops/instrumented-non-atomic.h:135 kernel/locking/lockdep.c:199 kernel/locking/lockdep.c:3713 kernel/locking/lockdep.c:3769 kernel/locking/lockdep.c:5027) [ 698.690247][ T6895] __lock_acquire (kernel/locking/lockdep.c:3187 kernel/locking/lockdep.c:3801 kernel/locking/lockdep.c:5027) [ 698.694949][ T6895] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4885) [ 698.700772][ T6895] ? lock_is_held_type (kernel/locking/lockdep.c:5380 kernel/locking/lockdep.c:5682) [ 698.705645][ T6895] ? rcu_read_lock_sched_held (include/linux/lockdep.h:283 kernel/rcu/update.c:125) [ 698.711128][ T6895] lock_acquire (kernel/locking/lockdep.c:438 kernel/locking/lockdep.c:5641 kernel/locking/lockdep.c:5604) [ 698.715481][ T6895] ? mpol_rebind_mm (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:212 include/linux/mmap_lock.h:35 include/linux/mmap_lock.h:72 mm/mempolicy.c:384) [ 698.720096][ T6895] ? rcu_read_unlock (include/linux/rcupdate.h:723 (discriminator 5)) [ 698.724709][ T6895] ? get_task_mm (kernel/fork.c:1311) [ 698.728972][ T6895] ? rcu_read_lock_sched_held (include/linux/lockdep.h:283 kernel/rcu/update.c:125) [ 698.734445][ T6895] ? rcu_read_lock_bh_held (kernel/rcu/update.c:120) [ 698.739578][ T6895] ? lock_acquire (kernel/locking/lockdep.c:438 kernel/locking/lockdep.c:5641 kernel/locking/lockdep.c:5604) [ 698.744098][ T6895] ? lock_is_held_type (kernel/locking/lockdep.c:5380 kernel/locking/lockdep.c:5682) [ 698.748968][ T6895] down_write (include/linux/instrumented.h:101 include/linux/atomic/atomic-instrumented.h:1779 kernel/locking/rwsem.c:254 kernel/locking/rwsem.c:1258 kernel/locking/rwsem.c:1268 kernel/locking/rwsem.c:1515) [ 698.753063][ T6895] ? mpol_rebind_mm (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:212 include/linux/mmap_lock.h:35 include/linux/mmap_lock.h:72 mm/mempolicy.c:384) [ 698.757668][ T6895] ? down_write_killable (kernel/locking/rwsem.c:1512) [ 698.762797][ T6895] ? rwlock_bug+0xc0/0xc0 [ 698.767581][ T6895] mpol_rebind_mm (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:212 include/linux/mmap_lock.h:35 include/linux/mmap_lock.h:72 mm/mempolicy.c:384) [ 698.772023][ T6895] cpuset_attach (include/linux/instrumented.h:71 include/asm-generic/bitops/instrumented-non-atomic.h:134 kernel/cgroup/cpuset.c:258 kernel/cgroup/cpuset.c:2293) [ 698.776463][ T6895] ? guarantee_online_cpus (kernel/cgroup/cpuset.c:2243) [ 698.781763][ T6895] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4232 kernel/locking/lockdep.c:4292 kernel/locking/lockdep.c:4244) [ 698.787587][ T6895] cgroup_migrate_execute (kernel/cgroup/cgroup.c:2513) [ 698.792895][ T6895] ? _raw_spin_unlock_irq (arch/x86/include/asm/irqflags.h:45 arch/x86/include/asm/irqflags.h:80 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 698.797942][ T6895] cgroup_update_dfl_csses (kernel/cgroup/cgroup.c:2975) [ 698.803249][ T6895] ? cgroup_apply_control_enable (kernel/cgroup/cgroup.c:2939) [ 698.809075][ T6895] ? css_next_descendant_pre (kernel/cgroup/cgroup.c:4394 kernel/cgroup/cgroup.c:4375) [ 698.814468][ T6895] cgroup_subtree_control_write (kernel/cgroup/cgroup.c:3232 kernel/cgroup/cgroup.c:3365) [ 698.820203][ T6895] ? cgroup_max_descendants_write (kernel/cgroup/cgroup.c:3283) [ 698.826116][ T6895] cgroup_file_write (kernel/cgroup/cgroup.c:3855) [ 698.830903][ T6895] ? css_release_work_fn (kernel/cgroup/cgroup.c:3855) [ 698.836028][ T6895] ? __kasan_slab_alloc (mm/kasan/common.c:431 mm/kasan/common.c:469) [ 698.840896][ T6895] ? lock_is_held_type (kernel/locking/lockdep.c:5380 kernel/locking/lockdep.c:5682) [ 698.845762][ T6895] ? css_release_work_fn (kernel/cgroup/cgroup.c:3855) [ 698.850886][ T6895] kernfs_fop_write_iter (fs/kernfs/file.c:300) [ 698.856015][ T6895] new_sync_write (fs/read_write.c:504 (discriminator 1)) [ 698.860544][ T6895] ? new_sync_read (fs/read_write.c:493) [ 698.865151][ T6895] ? ksys_write (fs/read_write.c:643) [ 698.869413][ T6895] ? rcu_read_unlock (include/linux/rcupdate.h:723 (discriminator 5)) [ 698.874027][ T6895] ? lock_is_held_type (kernel/locking/lockdep.c:5380 kernel/locking/lockdep.c:5682) [ 698.878895][ T6895] ? rcu_read_lock_held (kernel/rcu/update.c:341) [ 698.883772][ T6895] vfs_write (fs/read_write.c:590) [ 698.887867][ T6895] ksys_write (fs/read_write.c:643) [ 698.891963][ T6895] ? __ia32_sys_read (fs/read_write.c:633) [ 698.896570][ T6895] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4232 kernel/locking/lockdep.c:4292 kernel/locking/lockdep.c:4244) [ 698.902386][ T6895] ? syscall_enter_from_user_mode (arch/x86/include/asm/irqflags.h:45 arch/x86/include/asm/irqflags.h:80 kernel/entry/common.c:107) [ 698.908117][ T6895] do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80) [ 698.912384][ T6895] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4232 kernel/locking/lockdep.c:4292 kernel/locking/lockdep.c:4244) [ 698.918203][ T6895] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:113) [ 698.923935][ T6895] RIP: 0033:0x7fe2d6264504 [ 698.928205][ T6895] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 f9 61 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53 All code ======== 0: 00 f7 add %dh,%bh 2: d8 64 89 02 fsubs 0x2(%rcx,%rcx,4) 6: 48 c7 c0 ff ff ff ff mov $0xffffffffffffffff,%rax To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests sudo bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run sudo bin/lkp run generated-yaml-file # if come across any failure that blocks the test, # please remove ~/.lkp and /lkp dir to run from a clean state. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang