Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git superblock-scaling commit 40ceea09e84d1b9319236b27ad3162422310e5d0 Author: Dave Chinner AuthorDate: Wed Mar 4 14:36:27 2015 -0500 Commit: Josef Bacik CommitDate: Wed Mar 4 14:39:28 2015 -0500 bdi: add a new writeback list for sync wait_sb_inodes() current does a walk of all inodes in the filesystem to find dirty one to wait on during sync. This is highly inefficient and wastes a lot of CPU when there are lots of clean cached inodes that we don't need to wait on. To avoid this "all inode" walk, we need to track inodes that are currently under writeback that we need to wait for. We do this by adding inodes to a writeback list on the bdi when the mapping is first tagged as having pages under writeback. wait_sb_inodes() can then walk this list of "inodes under IO" and wait specifically just for the inodes that the current sync(2) needs to wait for. To avoid needing all the realted locking to be safe against interrupts, Jan Kara suggested that we be lazy about removal from the writeback list. That is, we don't remove inodes from the writeback list on IO completion, but do it directly during a wait_sb_inodes() walk. This means that the a rare sync(2) call will have some work to do skipping clean inodes However, in the current problem case of concurrent sync workloads, concurrent wait_sb_inodes() calls only walk the very recently dispatched inodes and hence should have very little work to do. This also means that we have to remove the inodes from the writeback list during eviction. Do this in inode_wait_for_writeback() once all writeback on the inode is complete. Signed-off-by: Dave Chinner +------------------------------------------+------------+------------+------------+ | | d2ee191143 | 40ceea09e8 | 45b8e7be56 | +------------------------------------------+------------+------------+------------+ | boot_successes | 72 | 0 | 0 | | boot_failures | 8 | 20 | 12 | | BUG:kernel_boot_hang | 8 | | | | BUG:unable_to_handle_kernel | 0 | 20 | 12 | | Oops | 0 | 20 | 12 | | RIP:blk_get_backing_dev_info | 0 | 20 | 12 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 20 | 12 | | backtrace:add_disk | 0 | 20 | 12 | | backtrace:brd_init | 0 | 20 | 12 | | backtrace:kernel_init_freeable | 0 | 20 | 12 | +------------------------------------------+------------+------------+------------+ [ 0.699779] Linux agpgart interface v0.103 [ 0.700296] Hangcheck: starting hangcheck timer 0.9.1 (tick is 180 seconds, margin is 60 seconds). [ 0.701079] [drm] Initialized drm 1.1.0 20060810 [ 0.702193] BUG: unable to handle kernel NULL pointer dereference at 0000000000000550 [ 0.702883] IP: [] blk_get_backing_dev_info+0xb/0x1a [ 0.703443] PGD 0 [ 0.703632] Oops: 0000 [#1] PREEMPT SMP [ 0.704009] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.0.0-rc2-00135-g40ceea0 #1 [ 0.704142] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014 [ 0.704142] task: ffff880012860000 ti: ffff880012868000 task.ti: ffff880012868000 [ 0.704142] RIP: 0010:[] [] blk_get_backing_dev_info+0xb/0x1a [ 0.704142] RSP: 0000:ffff88001286bcd8 EFLAGS: 00010202 [ 0.704142] RAX: 0000000000000000 RBX: ffff8800124147a0 RCX: 0000000000000000 [ 0.704142] RDX: ffff880012860720 RSI: 0000000000000000 RDI: ffff8800124145c0 [ 0.704142] RBP: ffff88001286bcd8 R08: 0000000000000001 R09: 0000000000000000 [ 0.704142] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88001282a000 [ 0.704142] R13: ffffffff81c1c600 R14: ffffffff81c1c600 R15: ffff8800124145d8 [ 0.704142] FS: 0000000000000000(0000) GS:ffff880013a00000(0000) knlGS:0000000000000000 [ 0.704142] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 0.704142] CR2: 0000000000000550 CR3: 0000000002211000 CR4: 00000000000006f0 [ 0.704142] Stack: [ 0.704142] ffff88001286bcf8 ffffffff81155424 ffff8800124147a0 ffff880012414820 [ 0.704142] ffff88001286bd28 ffffffff8115558d ffff880012860720 ffff8800124147a0 [ 0.704142] ffff880012414958 ffffffff81c1c600 ffff88001286bd58 ffffffff8114a2e6 [ 0.704142] Call Trace: [ 0.704142] [] inode_to_bdi+0x36/0x45 [ 0.704142] [] inode_wait_for_writeback+0x3f/0xc2 [ 0.704142] [] evict+0xa2/0x15e [ 0.704142] [] iput+0x160/0x16d [ 0.704142] [] bdput+0xd/0xf [ 0.704142] [] __blkdev_put+0x166/0x18a [ 0.704142] [] blkdev_put+0x114/0x11d [ 0.704142] [] add_disk+0x44d/0x461 [ 0.704142] [] brd_init+0x95/0x160 [ 0.704142] [] ? ramdisk_size+0x1a/0x1a [ 0.704142] [] do_one_initcall+0xe8/0x175 [ 0.704142] [] kernel_init_freeable+0x1d0/0x258 [ 0.704142] [] ? rest_init+0xbc/0xbc [ 0.704142] [] kernel_init+0x9/0xd5 [ 0.704142] [] ret_from_fork+0x7c/0xb0 [ 0.704142] [] ? rest_init+0xbc/0xbc [ 0.704142] Code: ca 48 c1 ea 04 29 d0 ba 01 00 00 00 89 8f 80 08 00 00 ff c8 85 c0 0f 4e c2 89 87 84 08 00 00 c3 48 8b 87 10 01 00 00 55 48 89 e5 <48> 8b 80 50 05 00 00 5d 48 05 58 02 00 00 c3 48 89 fa 31 c0 b9 [ 0.704142] RIP [] blk_get_backing_dev_info+0xb/0x1a [ 0.704142] RSP [ 0.704142] CR2: 0000000000000550 [ 0.704142] ---[ end trace 5c64cf25111d3d67 ]--- [ 0.704142] Kernel panic - not syncing: Fatal exception git bisect start 45b8e7be563c57fc42d69d5239b4829b5586620d 13a7a6ac0a11197edcd0f756a035f472b42cdf8b -- git bisect bad 980171ac3db20fc792b9b1298067344725a5a285 # 19:07 0- 20 Merge 'luto/x86/entry' into devel-xian-x86_64-201503051818 git bisect bad 7a2a5fad21b95990713cbdfaccc9eeba4e98f9b8 # 19:13 0- 20 Merge 'kees/format-security' into devel-xian-x86_64-201503051818 git bisect good cadb5884edc7353ecb245cf0874ead1f9565f2a7 # 19:29 20+ 0 Merge 'trace/ftrace/urgent' into devel-xian-x86_64-201503051818 git bisect good 30abe812fb9b18b25ebb9d2d214a70013a191ccb # 19:34 20+ 0 Merge 'paulburton/wip-ci20-v4.0' into devel-xian-x86_64-201503051818 git bisect good 0d0fc17147f433ffe27f8d2fcd3b29e109694fe3 # 19:40 20+ 0 Merge 'arm-soc/next/drivers' into devel-xian-x86_64-201503051818 git bisect bad caca114c0271d4df06e2ff1acee68dd62be43d66 # 20:03 0- 20 Merge 'josef-btrfs/superblock-scaling' into devel-xian-x86_64-201503051818 git bisect good d2ee19114357bdf21c59a3ac61eb053ef1c0dc4e # 20:15 20+ 8 inode: rename i_wb_list to i_io_list git bisect bad 63738525a6ebdf74bb3eb1c3dba16c0bb6895d97 # 20:28 0- 20 inode: convert per-sb inode list to a list_lru git bisect bad a05899067cddc24276e43e0d440da791738cf967 # 20:42 0- 20 writeback: periodically trim the writeback list git bisect bad 40ceea09e84d1b9319236b27ad3162422310e5d0 # 21:12 0- 20 bdi: add a new writeback list for sync # first bad commit: [40ceea09e84d1b9319236b27ad3162422310e5d0] bdi: add a new writeback list for sync git bisect good d2ee19114357bdf21c59a3ac61eb053ef1c0dc4e # 21:14 60+ 8 inode: rename i_wb_list to i_io_list # extra tests with DEBUG_INFO git bisect bad 40ceea09e84d1b9319236b27ad3162422310e5d0 # 22:55 0- 22 bdi: add a new writeback list for sync # extra tests on HEAD of linux-devel/devel-xian-x86_64-201503051818 git bisect bad 45b8e7be563c57fc42d69d5239b4829b5586620d # 22:55 0- 12 0day head guard for 'devel-xian-x86_64-201503051818' # extra tests on tree/branch josef-btrfs/superblock-scaling git bisect bad d119f33d7f868e92c2d7fd21da1aade94584994d # 23:13 0- 60 inode: don't softlockup when evicting inodes # extra tests on tree/branch linus/master git bisect good 6587457b4b3d663b237a0f95ddf6e67d1828c8ea # 23:41 60+ 2 Merge tag 'dma-buf-for-4.0-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/sumits/dma-buf # extra tests on tree/branch next/master git bisect good cbbf783608bd1f177fd8b1f6498bb2481116beed # 23:53 60+ 0 Add linux-next specific files for 20150305 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 initrd=yocto-minimal-x86_64.cgz wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd kvm=( qemu-system-x86_64 -cpu kvm64 -enable-kvm -kernel $kernel -initrd $initrd -m 320 -smp 1 -net nic,vlan=1,model=e1000 -net user,vlan=1 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 rd.udev.log-priority=err systemd.log_target=journal systemd.log_level=warning debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- Thanks, Fengguang