Greeting, FYI, we noticed a 139.4% improvement of stress-ng.loop.ops_per_sec due to commit: commit: 6cc8e7430801fa238bd7d3acae1eb406c6e02fe1 ("loop: scale loop device by introducing per device lock") https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master in testcase: stress-ng on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory with following parameters: nr_threads: 10% disk: 1HDD testtime: 60s fs: ext4 class: os test: loop cpufreq_governor: performance ucode: 0x5003003 In addition to that, the commit also has significant impact on the following tests: +------------------+----------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.loop.ops_per_sec 140.6% improvement | | test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory | | test parameters | class=device | | | cpufreq_governor=performance | | | disk=1HDD | | | nr_threads=100% | | | test=loop | | | testtime=60s | | | ucode=0x5003003 | +------------------+----------------------------------------------------------------------+ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml bin/lkp run compatible-job.yaml ========================================================================================= class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode: os/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/loop/stress-ng/60s/0x5003003 commit: 767630c63b ("bdev: Do not return EBUSY if bdev discard races with write") 6cc8e74308 ("loop: scale loop device by introducing per device lock") 767630c63bb23acf 6cc8e7430801fa238bd7d3acae1 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 0:6 -4% 0:6 perf-profile.children.cycles-pp.error_entry 0:6 -4% 0:6 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 647.50 ± 5% +138.9% 1547 ± 3% stress-ng.loop.ops 10.74 ± 5% +139.4% 25.72 ± 3% stress-ng.loop.ops_per_sec 2243 ± 6% +75.4% 3934 ± 3% stress-ng.time.involuntary_context_switches 22.00 ± 4% +65.2% 36.33 ± 3% stress-ng.time.percent_of_cpu_this_job_got 14.02 ± 3% +63.2% 22.89 ± 3% stress-ng.time.system_time 20775 ± 5% +125.8% 46901 ± 16% stress-ng.time.voluntary_context_switches 195323 ± 7% +22.5% 239317 ± 10% numa-numastat.node1.numa_hit 1.30 ± 68% -61.2% 0.50 ± 44% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork 7274 ± 2% +28.4% 9340 ± 2% meminfo.Active 4226 ± 3% +44.9% 6126 ± 2% meminfo.Active(file) 3174 ± 3% +60.1% 5082 ± 2% meminfo.Buffers 0.27 ± 10% +0.3 0.56 ± 8% mpstat.cpu.all.iowait% 0.42 ± 3% +0.2 0.60 ± 4% mpstat.cpu.all.sys% 0.04 ± 3% +0.0 0.06 ± 6% mpstat.cpu.all.usr% 1070 ± 3% +40.6% 1504 vmstat.io.bi 3208 ± 3% +60.1% 5137 ± 2% vmstat.memory.buff 5446 ± 2% +50.3% 8184 ± 3% vmstat.system.cs 4417358 ± 3% +39.9% 6181053 ± 23% cpuidle.C1.time 45013 ± 6% +72.1% 77477 ± 53% cpuidle.C1.usage 196417 ± 4% +47.2% 289159 ± 12% cpuidle.POLL.time 29151 ± 5% +43.1% 41711 ± 9% cpuidle.POLL.usage 761.67 +5.5% 803.33 ± 2% proc-vmstat.nr_active_anon 1058 ± 3% +44.8% 1532 ± 2% proc-vmstat.nr_active_file 20307 +1.4% 20591 proc-vmstat.nr_slab_reclaimable 42850 +1.6% 43553 proc-vmstat.nr_slab_unreclaimable 761.67 +5.5% 803.33 ± 2% proc-vmstat.nr_zone_active_anon 1058 ± 3% +44.8% 1532 ± 2% proc-vmstat.nr_zone_active_file 454102 +12.4% 510568 proc-vmstat.numa_hit 365903 +15.3% 421780 proc-vmstat.numa_local 4689 ± 3% +70.6% 7997 ± 3% proc-vmstat.pgactivate 503369 +19.6% 602248 proc-vmstat.pgalloc_normal 370729 +26.0% 467222 proc-vmstat.pgfree 69989 ± 3% +40.1% 98054 proc-vmstat.pgpgin 676.83 ± 6% +17.5% 795.17 ± 2% slabinfo.bdev_cache.active_objs 676.83 ± 6% +17.5% 795.17 ± 2% slabinfo.bdev_cache.num_objs 1503 ± 11% +43.8% 2162 ± 2% slabinfo.buffer_head.active_objs 1509 ± 11% +44.4% 2180 ± 2% slabinfo.buffer_head.num_objs 1960 ± 6% +27.0% 2490 ± 2% slabinfo.dquot.active_objs 1960 ± 6% +27.0% 2490 ± 2% slabinfo.dquot.num_objs 5779 ± 4% +13.1% 6534 ± 3% slabinfo.kmalloc-rcl-64.active_objs 5779 ± 4% +13.1% 6534 ± 3% slabinfo.kmalloc-rcl-64.num_objs 2543 ± 3% +14.1% 2903 ± 3% slabinfo.kmalloc-rcl-96.active_objs 2543 ± 3% +14.1% 2903 ± 3% slabinfo.kmalloc-rcl-96.num_objs 1068 ± 6% +18.6% 1266 slabinfo.request_queue.active_objs 1068 ± 6% +18.6% 1266 slabinfo.request_queue.num_objs 3.011e+08 +9.1% 3.286e+08 perf-stat.i.branch-instructions 5403 ± 2% +51.9% 8206 ± 3% perf-stat.i.context-switches 144.06 ± 2% +14.0% 164.24 ± 2% perf-stat.i.cpu-migrations 1707 ± 12% +22.1% 2085 ± 10% perf-stat.i.cycles-between-cache-misses 3.644e+08 +11.1% 4.048e+08 perf-stat.i.dTLB-loads 1.782e+08 +16.1% 2.068e+08 perf-stat.i.dTLB-stores 1.437e+09 +10.5% 1.588e+09 perf-stat.i.instructions 1.22 ± 8% +23.8% 1.52 ± 8% perf-stat.i.metric.K/sec 9.02 ± 2% +11.5% 10.06 ± 2% perf-stat.i.metric.M/sec 358554 ± 4% +30.1% 466359 ± 5% perf-stat.i.node-load-misses 68252 ± 7% +19.2% 81323 ± 4% perf-stat.i.node-loads 169904 ± 11% +63.7% 278159 ± 6% perf-stat.i.node-store-misses 1690 ± 11% +20.9% 2043 ± 10% perf-stat.overall.cycles-between-cache-misses 2.964e+08 +9.1% 3.233e+08 perf-stat.ps.branch-instructions 5317 ± 2% +51.9% 8076 ± 3% perf-stat.ps.context-switches 141.76 ± 2% +14.0% 161.59 ± 2% perf-stat.ps.cpu-migrations 3.587e+08 +11.1% 3.984e+08 perf-stat.ps.dTLB-loads 1.754e+08 +16.0% 2.035e+08 perf-stat.ps.dTLB-stores 1.415e+09 +10.5% 1.563e+09 perf-stat.ps.instructions 11.19 ± 2% -2.2% 10.94 perf-stat.ps.major-faults 352880 ± 4% +30.1% 458953 ± 5% perf-stat.ps.node-load-misses 67186 ± 7% +19.1% 80032 ± 4% perf-stat.ps.node-loads 167186 ± 11% +63.7% 273686 ± 6% perf-stat.ps.node-store-misses 8.979e+10 +10.7% 9.939e+10 perf-stat.total.instructions 58913 +10.2% 64920 interrupts.CAL:Function_call_interrupts 543.33 ± 31% +70.7% 927.33 ± 7% interrupts.CPU0.CAL:Function_call_interrupts 614.50 ± 5% +11.6% 685.83 ± 5% interrupts.CPU10.CAL:Function_call_interrupts 572.50 ± 6% +16.9% 669.17 ± 10% interrupts.CPU11.CAL:Function_call_interrupts 140.00 ± 36% +82.0% 254.83 ± 33% interrupts.CPU2.NMI:Non-maskable_interrupts 140.00 ± 36% +82.0% 254.83 ± 33% interrupts.CPU2.PMI:Performance_monitoring_interrupts 127.00 ± 44% +87.8% 238.50 ± 12% interrupts.CPU20.NMI:Non-maskable_interrupts 127.00 ± 44% +87.8% 238.50 ± 12% interrupts.CPU20.PMI:Performance_monitoring_interrupts 152.17 ± 26% +62.8% 247.67 ± 16% interrupts.CPU21.NMI:Non-maskable_interrupts 152.17 ± 26% +62.8% 247.67 ± 16% interrupts.CPU21.PMI:Performance_monitoring_interrupts 194.33 ± 28% -39.5% 117.50 ± 37% interrupts.CPU29.NMI:Non-maskable_interrupts 194.33 ± 28% -39.5% 117.50 ± 37% interrupts.CPU29.PMI:Performance_monitoring_interrupts 207.50 ± 26% -40.2% 124.17 ± 25% interrupts.CPU32.NMI:Non-maskable_interrupts 207.50 ± 26% -40.2% 124.17 ± 25% interrupts.CPU32.PMI:Performance_monitoring_interrupts 206.50 ± 17% -46.5% 110.50 ± 25% interrupts.CPU33.NMI:Non-maskable_interrupts 206.50 ± 17% -46.5% 110.50 ± 25% interrupts.CPU33.PMI:Performance_monitoring_interrupts 578.33 ± 3% +15.9% 670.50 ± 10% interrupts.CPU43.CAL:Function_call_interrupts 586.33 ± 4% +17.3% 687.50 ± 7% interrupts.CPU45.CAL:Function_call_interrupts 148.33 ± 35% +68.1% 249.33 ± 19% interrupts.CPU49.NMI:Non-maskable_interrupts 148.33 ± 35% +68.1% 249.33 ± 19% interrupts.CPU49.PMI:Performance_monitoring_interrupts 573.00 ± 5% +17.8% 675.17 ± 9% interrupts.CPU57.CAL:Function_call_interrupts 559.83 ± 6% +17.1% 655.50 ± 6% interrupts.CPU67.CAL:Function_call_interrupts 588.67 ± 5% +17.1% 689.17 ± 7% interrupts.CPU74.CAL:Function_call_interrupts 558.33 ± 7% +30.2% 727.00 ± 18% interrupts.CPU79.CAL:Function_call_interrupts 193.83 ± 20% -43.0% 110.50 ± 38% interrupts.CPU81.NMI:Non-maskable_interrupts 193.83 ± 20% -43.0% 110.50 ± 38% interrupts.CPU81.PMI:Performance_monitoring_interrupts 591.17 ± 8% +18.5% 700.50 ± 10% interrupts.CPU87.CAL:Function_call_interrupts 536.00 ± 6% +32.4% 709.83 ± 13% interrupts.CPU88.CAL:Function_call_interrupts 15028 ± 5% +16.6% 17516 ± 8% interrupts.NMI:Non-maskable_interrupts 15028 ± 5% +16.6% 17516 ± 8% interrupts.PMI:Performance_monitoring_interrupts 24148 +21.8% 29419 softirqs.BLOCK 12043 ± 15% -17.3% 9961 ± 6% softirqs.CPU1.RCU 10618 ± 5% -10.4% 9516 ± 5% softirqs.CPU12.RCU 10634 ± 6% -11.8% 9380 ± 4% softirqs.CPU13.RCU 10374 ± 5% -10.7% 9262 ± 4% softirqs.CPU14.RCU 10161 ± 3% -11.8% 8961 ± 5% softirqs.CPU16.RCU 10238 ± 3% -12.7% 8935 ± 4% softirqs.CPU17.RCU 10241 ± 3% -10.3% 9184 ± 4% softirqs.CPU18.RCU 10171 ± 3% -9.4% 9212 ± 5% softirqs.CPU19.RCU 10008 ± 4% -11.2% 8883 ± 6% softirqs.CPU20.RCU 10115 ± 3% -11.3% 8970 ± 4% softirqs.CPU21.RCU 10237 ± 2% -12.4% 8968 ± 3% softirqs.CPU22.RCU 10617 ± 5% -12.7% 9274 ± 6% softirqs.CPU24.RCU 9909 ± 2% -10.7% 8846 ± 5% softirqs.CPU25.RCU 9800 ± 2% -12.5% 8571 ± 3% softirqs.CPU26.RCU 10069 ± 5% -11.6% 8903 ± 5% softirqs.CPU27.RCU 10846 ± 10% -17.3% 8965 ± 5% softirqs.CPU28.RCU 10036 ± 5% -11.9% 8836 ± 6% softirqs.CPU30.RCU 10172 ± 7% -12.7% 8883 ± 4% softirqs.CPU31.RCU 10594 ± 6% -14.1% 9105 ± 6% softirqs.CPU36.RCU 10281 ± 5% -9.5% 9303 ± 4% softirqs.CPU49.RCU 10510 ± 4% -15.0% 8937 ± 6% softirqs.CPU55.RCU 10904 ± 5% -13.5% 9434 ± 7% softirqs.CPU57.RCU 2200 ±206% -100.0% 0.50 ±152% softirqs.CPU58.NET_RX 9863 ± 2% -11.2% 8755 ± 3% softirqs.CPU64.RCU 10247 ± 4% -11.8% 9033 ± 3% softirqs.CPU65.RCU 10101 ± 4% -10.5% 9042 ± 2% softirqs.CPU68.RCU 10155 ± 7% -10.9% 9052 ± 3% softirqs.CPU70.RCU 10153 ± 4% -12.5% 8888 ± 4% softirqs.CPU71.RCU 10158 ± 7% -13.0% 8834 ± 4% softirqs.CPU72.RCU 9909 ± 2% -13.9% 8528 ± 4% softirqs.CPU73.RCU 10625 ± 15% -19.9% 8514 ± 4% softirqs.CPU74.RCU 9998 ± 6% -11.7% 8829 ± 5% softirqs.CPU76.RCU 10191 ± 7% -13.0% 8866 ± 5% softirqs.CPU77.RCU 10061 ± 7% -12.0% 8852 ± 5% softirqs.CPU79.RCU 10730 ± 6% -11.7% 9478 ± 4% softirqs.CPU8.RCU 10626 ± 7% -11.2% 9431 ± 4% softirqs.CPU9.RCU 75.32 ± 3% -10.9 64.46 ± 3% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 74.52 ± 3% -10.8 63.68 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 74.56 ± 3% -10.8 63.72 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 74.56 ± 3% -10.8 63.72 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 65.18 ± 4% -7.5 57.66 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 22.43 ± 2% -4.8 17.63 ± 9% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 16.91 ± 2% -3.4 13.52 ± 7% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 10.84 ± 2% -2.1 8.74 ± 6% perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 10.76 ± 2% -2.1 8.69 ± 6% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 10.56 ± 2% -2.0 8.53 ± 6% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 1.66 ± 30% -1.1 0.52 ± 47% perf-profile.calltrace.cycles-pp.__mutex_lock.lo_release.__blkdev_put.blkdev_put.blkdev_close 1.66 ± 30% -1.1 0.52 ± 47% perf-profile.calltrace.cycles-pp.lo_release.__blkdev_put.blkdev_put.blkdev_close.__fput 2.57 ± 13% -0.8 1.77 ± 9% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 1.77 ± 13% -0.7 1.05 ± 6% perf-profile.calltrace.cycles-pp.mutex_spin_on_owner.__mutex_lock.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64 3.50 ± 6% -0.7 2.84 ± 8% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack 3.05 ± 5% -0.6 2.47 ± 7% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 2.50 ± 5% -0.6 1.95 ± 7% perf-profile.calltrace.cycles-pp.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 2.49 ± 5% -0.5 1.94 ± 7% perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 2.48 ± 5% -0.5 1.94 ± 7% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt 2.73 ± 6% -0.5 2.24 ± 9% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 1.14 ± 11% -0.4 0.77 ± 7% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu 1.04 ± 8% -0.2 0.80 ± 15% perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt 0.47 ± 45% +0.4 0.87 ± 10% perf-profile.calltrace.cycles-pp.loop_remove.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.26 ±100% +0.5 0.76 ± 12% perf-profile.calltrace.cycles-pp.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue.loop_remove 0.26 ±100% +0.5 0.76 ± 12% perf-profile.calltrace.cycles-pp.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue 0.26 ±100% +0.5 0.76 ± 12% perf-profile.calltrace.cycles-pp.blk_mq_exit_queue.blk_cleanup_queue.loop_remove.loop_control_ioctl.__x64_sys_ioctl 0.26 ±100% +0.5 0.76 ± 12% perf-profile.calltrace.cycles-pp.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue.loop_remove.loop_control_ioctl 0.27 ±100% +0.5 0.78 ± 11% perf-profile.calltrace.cycles-pp.blk_cleanup_queue.loop_remove.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64 0.00 +0.7 0.69 ± 14% perf-profile.calltrace.cycles-pp.smp_call_function_many_cond.on_each_cpu_cond_mask.kill_bdev.__blkdev_put.blkdev_put 0.00 +0.7 0.69 ± 14% perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.kill_bdev.__blkdev_put.blkdev_put.blkdev_close 0.00 +0.7 0.69 ± 14% perf-profile.calltrace.cycles-pp.kill_bdev.__blkdev_put.blkdev_put.blkdev_close.__fput 2.46 ± 12% +2.8 5.30 ± 19% perf-profile.calltrace.cycles-pp.set_capacity_and_notify.cold.loop_set_size.lo_ioctl.blkdev_ioctl.block_ioctl 2.44 ± 11% +2.8 5.29 ± 19% perf-profile.calltrace.cycles-pp.vprintk_emit.printk.set_capacity_and_notify.cold.loop_set_size.lo_ioctl 2.46 ± 12% +2.8 5.30 ± 19% perf-profile.calltrace.cycles-pp.loop_set_size.lo_ioctl.blkdev_ioctl.block_ioctl.__x64_sys_ioctl 2.44 ± 11% +2.8 5.29 ± 19% perf-profile.calltrace.cycles-pp.printk.set_capacity_and_notify.cold.loop_set_size.lo_ioctl.blkdev_ioctl 3.21 ± 12% +3.3 6.48 ± 14% perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write 2.50 ± 20% +3.6 6.09 ± 11% perf-profile.calltrace.cycles-pp.loop_configure.lo_ioctl.blkdev_ioctl.block_ioctl.__x64_sys_ioctl 2.37 ± 21% +3.6 6.01 ± 11% perf-profile.calltrace.cycles-pp.printk.set_capacity_and_notify.cold.loop_set_size.loop_configure.lo_ioctl 2.37 ± 21% +3.6 6.01 ± 11% perf-profile.calltrace.cycles-pp.vprintk_emit.printk.set_capacity_and_notify.cold.loop_set_size.loop_configure 2.37 ± 21% +3.6 6.01 ± 11% perf-profile.calltrace.cycles-pp.loop_set_size.loop_configure.lo_ioctl.blkdev_ioctl.block_ioctl 2.37 ± 21% +3.6 6.01 ± 11% perf-profile.calltrace.cycles-pp.set_capacity_and_notify.cold.loop_set_size.loop_configure.lo_ioctl.blkdev_ioctl 4.12 ± 14% +3.8 7.93 ± 16% perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock 4.12 ± 14% +3.8 7.93 ± 16% perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit 4.33 ± 13% +4.2 8.52 ± 16% perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.printk 14.71 ± 14% +4.3 18.99 ± 10% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 4.40 ± 16% +4.4 8.84 ± 15% perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.printk.set_capacity_and_notify.cold 7.00 ± 13% +4.6 11.64 ± 12% perf-profile.calltrace.cycles-pp.block_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 7.00 ± 13% +4.6 11.64 ± 12% perf-profile.calltrace.cycles-pp.blkdev_ioctl.block_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.81 ± 16% +4.7 9.50 ± 15% perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.printk.set_capacity_and_notify.cold.loop_set_size 6.92 ± 13% +4.7 11.62 ± 12% perf-profile.calltrace.cycles-pp.lo_ioctl.blkdev_ioctl.block_ioctl.__x64_sys_ioctl.do_syscall_64 10.51 ± 14% +4.7 15.26 ± 10% perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 12.76 ± 13% +4.8 17.57 ± 10% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.52 ± 4% +5.7 12.19 ± 3% perf-profile.calltrace.cycles-pp.memcpy_toio.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread 6.75 ± 4% +5.8 12.59 ± 3% perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork 6.83 ± 4% +6.4 13.19 ± 3% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 6.88 ± 4% +6.4 13.24 ± 3% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 7.29 ± 5% +6.5 13.74 ± 3% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 7.29 ± 5% +6.5 13.75 ± 3% perf-profile.calltrace.cycles-pp.ret_from_fork 75.32 ± 3% -10.9 64.46 ± 3% perf-profile.children.cycles-pp.do_idle 75.32 ± 3% -10.9 64.46 ± 3% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 75.32 ± 3% -10.9 64.46 ± 3% perf-profile.children.cycles-pp.cpu_startup_entry 74.56 ± 3% -10.8 63.72 ± 3% perf-profile.children.cycles-pp.start_secondary 65.82 ± 4% -7.5 58.28 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state 65.87 ± 4% -7.5 58.34 ± 3% perf-profile.children.cycles-pp.cpuidle_enter 20.41 -4.0 16.42 ± 7% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 8.69 ± 14% -3.9 4.77 ± 15% perf-profile.children.cycles-pp.__mutex_lock 17.24 ± 2% -3.4 13.80 ± 7% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 14.15 ± 3% -2.7 11.47 ± 5% perf-profile.children.cycles-pp.asm_call_sysvec_on_stack 4.94 ± 7% -2.2 2.73 ± 14% perf-profile.children.cycles-pp.mutex_spin_on_owner 10.97 ± 2% -2.1 8.88 ± 6% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 10.77 ± 2% -2.0 8.72 ± 6% perf-profile.children.cycles-pp.hrtimer_interrupt 3.69 ± 22% -1.7 2.00 ± 27% perf-profile.children.cycles-pp.osq_lock 1.89 ± 22% -1.3 0.63 ± 23% perf-profile.children.cycles-pp.lo_release 2.60 ± 13% -0.8 1.81 ± 9% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 3.59 ± 5% -0.7 2.93 ± 8% perf-profile.children.cycles-pp.tick_sched_timer 3.14 ± 5% -0.6 2.54 ± 7% perf-profile.children.cycles-pp.irq_exit_rcu 2.55 ± 5% -0.6 2.00 ± 7% perf-profile.children.cycles-pp.do_softirq_own_stack 2.57 ± 5% -0.5 2.05 ± 7% perf-profile.children.cycles-pp.__softirqentry_text_start 2.81 ± 6% -0.5 2.31 ± 9% perf-profile.children.cycles-pp.update_process_times 1.15 ± 10% -0.4 0.79 ± 7% perf-profile.children.cycles-pp.rebalance_domains 1.07 ± 9% -0.2 0.83 ± 15% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 0.90 ± 8% -0.2 0.69 ± 15% perf-profile.children.cycles-pp.irqtime_account_irq 1.44 ± 7% -0.2 1.23 ± 11% perf-profile.children.cycles-pp.scheduler_tick 0.89 ± 8% -0.2 0.74 ± 11% perf-profile.children.cycles-pp.lapic_next_deadline 0.59 ± 9% -0.1 0.45 ± 9% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.84 ± 5% -0.1 0.72 ± 9% perf-profile.children.cycles-pp.read_tsc 0.29 ± 11% -0.1 0.22 ± 14% perf-profile.children.cycles-pp.note_gp_changes 0.23 ± 8% -0.1 0.17 ± 15% perf-profile.children.cycles-pp.arch_cpu_idle_enter 0.08 ± 23% -0.1 0.03 ±101% perf-profile.children.cycles-pp.vm_mmap_pgoff 0.08 ± 22% -0.1 0.03 ±100% perf-profile.children.cycles-pp.do_mmap 0.21 ± 9% -0.0 0.17 ± 14% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust 0.12 ± 19% -0.0 0.09 ± 8% perf-profile.children.cycles-pp.mmput 0.12 ± 17% -0.0 0.09 ± 8% perf-profile.children.cycles-pp.exit_mmap 0.06 ± 17% +0.0 0.11 ± 18% perf-profile.children.cycles-pp.d_invalidate 0.02 ±142% +0.0 0.07 ± 21% perf-profile.children.cycles-pp.strncpy_from_user 0.02 ±141% +0.0 0.07 ± 19% perf-profile.children.cycles-pp.do_faccessat 0.03 ±103% +0.0 0.08 ± 20% perf-profile.children.cycles-pp.___slab_alloc 0.03 ±103% +0.1 0.09 ± 22% perf-profile.children.cycles-pp.__slab_alloc 0.03 ±100% +0.1 0.08 ± 19% perf-profile.children.cycles-pp.del_gendisk 0.01 ±223% +0.1 0.07 ± 15% perf-profile.children.cycles-pp.pcpu_alloc 0.04 ± 72% +0.1 0.10 ± 15% perf-profile.children.cycles-pp.getname_flags 0.01 ±223% +0.1 0.07 ± 31% perf-profile.children.cycles-pp.ahci_single_level_irq_intr 0.09 ± 11% +0.1 0.16 ± 13% perf-profile.children.cycles-pp.cpumask_next 0.01 ±223% +0.1 0.08 ± 33% perf-profile.children.cycles-pp.__handle_irq_event_percpu 0.00 +0.1 0.07 ± 10% perf-profile.children.cycles-pp.blk_mq_init_queue_data 0.01 ±223% +0.1 0.08 ± 31% perf-profile.children.cycles-pp.handle_irq_event_percpu 0.03 ±102% +0.1 0.10 ± 25% perf-profile.children.cycles-pp.asm_common_interrupt 0.03 ±102% +0.1 0.10 ± 25% perf-profile.children.cycles-pp.common_interrupt 0.09 ± 21% +0.1 0.16 ± 17% perf-profile.children.cycles-pp.d_alloc 0.01 ±223% +0.1 0.09 ± 29% perf-profile.children.cycles-pp.handle_edge_irq 0.01 ±223% +0.1 0.09 ± 32% perf-profile.children.cycles-pp.handle_irq_event 0.09 ± 21% +0.1 0.16 ± 28% perf-profile.children.cycles-pp.new_inode 0.08 ± 26% +0.1 0.16 ± 28% perf-profile.children.cycles-pp.new_inode_pseudo 0.09 ± 22% +0.1 0.16 ± 28% perf-profile.children.cycles-pp.debugfs_get_inode 0.17 ± 22% +0.1 0.24 ± 20% perf-profile.children.cycles-pp.__dentry_kill 0.06 ± 55% +0.1 0.14 ± 22% perf-profile.children.cycles-pp.alloc_inode 0.11 ± 20% +0.1 0.19 ± 7% perf-profile.children.cycles-pp.d_alloc_parallel 0.33 ± 14% +0.1 0.42 ± 6% perf-profile.children.cycles-pp.schedule 0.21 ± 15% +0.1 0.31 ± 14% perf-profile.children.cycles-pp.pick_next_task_fair 0.18 ± 15% +0.1 0.27 ± 12% perf-profile.children.cycles-pp.kmem_cache_free 0.17 ± 24% +0.1 0.28 ± 13% perf-profile.children.cycles-pp.walk_component 0.17 ± 14% +0.1 0.28 ± 13% perf-profile.children.cycles-pp.rcu_do_batch 0.12 ± 25% +0.1 0.22 ± 8% perf-profile.children.cycles-pp.lookup_one_len 0.15 ± 15% +0.1 0.26 ± 15% perf-profile.children.cycles-pp.__lookup_slow 0.41 ± 9% +0.1 0.52 ± 6% perf-profile.children.cycles-pp.__schedule 0.12 ± 26% +0.1 0.24 ± 9% perf-profile.children.cycles-pp.start_creating 0.12 ± 26% +0.1 0.24 ± 34% perf-profile.children.cycles-pp.link_path_walk 0.23 ± 8% +0.1 0.35 ± 11% perf-profile.children.cycles-pp.mnt_get_count 0.23 ± 8% +0.1 0.37 ± 10% perf-profile.children.cycles-pp.mntput_no_expire 0.25 ± 17% +0.1 0.40 ± 12% perf-profile.children.cycles-pp.kmem_cache_alloc 0.17 ± 27% +0.1 0.32 ± 33% perf-profile.children.cycles-pp.path_lookupat 0.19 ± 20% +0.2 0.37 ± 11% perf-profile.children.cycles-pp.debugfs_create_files 0.19 ± 19% +0.2 0.37 ± 11% perf-profile.children.cycles-pp.__debugfs_create_file 0.22 ± 23% +0.2 0.42 ± 11% perf-profile.children.cycles-pp.blk_mq_debugfs_register 0.21 ± 23% +0.2 0.41 ± 10% perf-profile.children.cycles-pp.blk_mq_debugfs_register_hctx 0.26 ± 21% +0.2 0.48 ± 14% perf-profile.children.cycles-pp.blk_register_queue 0.28 ± 21% +0.2 0.52 ± 14% perf-profile.children.cycles-pp.__device_add_disk 0.49 ± 8% +0.3 0.76 ± 12% perf-profile.children.cycles-pp.blk_mq_exit_queue 0.49 ± 8% +0.3 0.76 ± 12% perf-profile.children.cycles-pp.blk_mq_debugfs_unregister_hctx 0.50 ± 8% +0.3 0.78 ± 11% perf-profile.children.cycles-pp.blk_cleanup_queue 0.51 ± 10% +0.3 0.80 ± 11% perf-profile.children.cycles-pp.simple_recursive_removal 0.32 ± 17% +0.3 0.60 ± 13% perf-profile.children.cycles-pp.loop_add 0.51 ± 9% +0.3 0.80 ± 11% perf-profile.children.cycles-pp.debugfs_remove 0.55 ± 8% +0.3 0.87 ± 10% perf-profile.children.cycles-pp.loop_remove 0.24 ± 25% +0.4 0.63 ± 17% perf-profile.children.cycles-pp.io_serial_out 0.01 ±223% +0.5 0.51 ± 17% perf-profile.children.cycles-pp.sync_rcu_exp_select_node_cpus 0.00 +0.5 0.52 ± 16% perf-profile.children.cycles-pp.smp_call_function_single 0.20 ± 29% +0.6 0.79 ± 10% perf-profile.children.cycles-pp.kill_bdev 0.20 ± 28% +0.7 0.86 ± 9% perf-profile.children.cycles-pp.on_each_cpu_cond_mask 0.20 ± 28% +0.7 0.86 ± 9% perf-profile.children.cycles-pp.smp_call_function_many_cond 3.70 ± 12% +3.4 7.09 ± 13% perf-profile.children.cycles-pp.io_serial_in 2.50 ± 20% +3.6 6.09 ± 11% perf-profile.children.cycles-pp.loop_configure 4.54 ± 12% +3.7 8.22 ± 14% perf-profile.children.cycles-pp.serial8250_console_putchar 4.75 ± 12% +3.9 8.63 ± 14% perf-profile.children.cycles-pp.wait_for_xmitr 4.77 ± 12% +4.1 8.84 ± 14% perf-profile.children.cycles-pp.uart_console_write 5.00 ± 12% +4.3 9.27 ± 14% perf-profile.children.cycles-pp.serial8250_console_write 16.64 ± 11% +4.3 20.98 ± 9% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 5.44 ± 12% +4.5 9.95 ± 14% perf-profile.children.cycles-pp.console_unlock 7.05 ± 13% +4.6 11.65 ± 12% perf-profile.children.cycles-pp.block_ioctl 7.05 ± 13% +4.6 11.65 ± 12% perf-profile.children.cycles-pp.blkdev_ioctl 6.97 ± 13% +4.7 11.63 ± 12% perf-profile.children.cycles-pp.lo_ioctl 10.57 ± 14% +4.7 15.26 ± 10% perf-profile.children.cycles-pp.__x64_sys_ioctl 14.30 ± 11% +4.9 19.20 ± 9% perf-profile.children.cycles-pp.do_syscall_64 6.74 ± 4% +5.8 12.58 ± 3% perf-profile.children.cycles-pp.memcpy_toio 6.75 ± 4% +5.8 12.59 ± 3% perf-profile.children.cycles-pp.drm_fb_helper_damage_work 5.48 ± 11% +6.3 11.77 ± 12% perf-profile.children.cycles-pp.vprintk_emit 5.48 ± 11% +6.3 11.77 ± 12% perf-profile.children.cycles-pp.printk 6.83 ± 4% +6.4 13.19 ± 3% perf-profile.children.cycles-pp.process_one_work 6.88 ± 4% +6.4 13.25 ± 3% perf-profile.children.cycles-pp.worker_thread 7.29 ± 5% +6.5 13.74 ± 3% perf-profile.children.cycles-pp.kthread 7.30 ± 5% +6.5 13.76 ± 3% perf-profile.children.cycles-pp.ret_from_fork 4.83 ± 16% +6.5 11.31 ± 13% perf-profile.children.cycles-pp.loop_set_size 4.83 ± 16% +6.5 11.31 ± 13% perf-profile.children.cycles-pp.set_capacity_and_notify.cold 4.91 ± 8% -2.2 2.71 ± 14% perf-profile.self.cycles-pp.mutex_spin_on_owner 3.63 ± 21% -1.6 1.99 ± 27% perf-profile.self.cycles-pp.osq_lock 0.58 ± 20% -0.2 0.40 ± 8% perf-profile.self.cycles-pp.update_process_times 0.89 ± 8% -0.2 0.73 ± 12% perf-profile.self.cycles-pp.lapic_next_deadline 0.57 ± 9% -0.1 0.43 ± 8% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.81 ± 5% -0.1 0.70 ± 10% perf-profile.self.cycles-pp.read_tsc 0.20 ± 11% -0.0 0.15 ± 16% perf-profile.self.cycles-pp.tsc_verify_tsc_adjust 0.08 ± 11% +0.0 0.13 ± 26% perf-profile.self.cycles-pp.kmem_cache_free 0.08 ± 22% +0.1 0.14 ± 18% perf-profile.self.cycles-pp.kmem_cache_alloc 0.15 ± 14% +0.1 0.21 ± 14% perf-profile.self.cycles-pp.mnt_get_count 0.24 ± 25% +0.4 0.63 ± 17% perf-profile.self.cycles-pp.io_serial_out 0.00 +0.5 0.51 ± 17% perf-profile.self.cycles-pp.smp_call_function_single 0.20 ± 29% +0.6 0.82 ± 4% perf-profile.self.cycles-pp.smp_call_function_many_cond 0.04 ±141% +1.8 1.83 ± 16% perf-profile.self.cycles-pp.vprintk_emit 3.70 ± 12% +3.4 7.09 ± 13% perf-profile.self.cycles-pp.io_serial_in 6.69 ± 4% +5.8 12.51 ± 3% perf-profile.self.cycles-pp.memcpy_toio stress-ng.time.system_time 30 +----------------------------------------------------------------------+ | | 25 |-+ O O | |O O O OO OO OO O OO OO OO OOO OO OO | | O O O OOO O O O O O O OO O O O O | 20 |-+ | | | 15 |-+.+ +.+++ +. +++.+++ .++ .+++ .+++ .++ + +. +. ++ ++. | |+ ++ + + ++ ++ + ++ ++.+ ++ ++ : ++ +.++ ++| 10 |-+ : : | | : : | | : | 5 |-+ : | | : | 0 +----------------------------------------------------------------------+ stress-ng.time.percent_of_cpu_this_job_got 40 +----------------------------------------------------------------------+ |O O OO OOO OO OOOO O O OO OO OO O OO | 35 |-O O O OO O O O O O O O O | 30 |-+ | | | 25 |-+ | | +.+ +.+++ +. +++.+++ .++ .+++ .+++ .++ ++ +.++ +.+ ++ .+ ++. +| 20 |++ ++ + + ++ ++ + ++ ++.+ + : : + + + + | | : : | 15 |-+ : : | 10 |-+ :: | | : | 5 |-+ : | | : | 0 +----------------------------------------------------------------------+ stress-ng.time.involuntary_context_switches 4500 +--------------------------------------------------------------------+ |O O O O O O O OOO O | 4000 |-O O O OOOO OO OO OOOO OO OO OO OO O O OO O | 3500 |-+ O OO | | | 3000 |-+ | 2500 |-+ ++ +. + +. | |++.+ ++.++++++.+++++.++ : +++ .+++++ +++++.+++++ +++ +.++++ ++| 2000 |-+ ++ +.+ + : : | 1500 |-+ : : | | : : | 1000 |-+ :: | 500 |-+ :: | | : | 0 +--------------------------------------------------------------------+ stress-ng.loop.ops 1800 +--------------------------------------------------------------------+ | O O O O O O OO | 1600 |OO O OOOO OO OO OOOOO OO OO OO OO OO O OOOO | 1400 |-+ O O OO | | | 1200 |-+ | 1000 |-+ | | | 800 |-+ + +. + +. | 600 |++.+++++.++++++.+++++.+++ +.+++++.+++++ +++++.+++++ +++ +.++++ ++| | : : | 400 |-+ : : | 200 |-+ :: | | : | 0 +--------------------------------------------------------------------+ stress-ng.loop.ops_per_sec 30 +----------------------------------------------------------------------+ | O O O O O O O O | 25 |OO O OO OOOO OOOO OOOO O OO OO O OOO OOOO O | | O O OO | | | 20 |-+ | | | 15 |-+ | | + .+ + +. | 10 |++.++++.+++++.++++.+++++.+ ++.++++.+++++ +++.+++++.++ +.++ ++.+++ ++| | : : | | : : | 5 |-+ : | | : | 0 +----------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-csl-2sp7: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory ========================================================================================= class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode: device/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/loop/stress-ng/60s/0x5003003 commit: 767630c63b ("bdev: Do not return EBUSY if bdev discard races with write") 6cc8e74308 ("loop: scale loop device by introducing per device lock") 767630c63bb23acf 6cc8e7430801fa238bd7d3acae1 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:6 -17% :6 kmsg.blk_update_request:I/O_error,dev_loop#,sector#op#:(READ)flags#phys_seg#prio_class :6 33% 2:6 kmsg.debugfs:Directory'loop#'with_parent'block'already_present 0:6 -3% 0:6 perf-profile.children.cycles-pp.error_entry 0:6 -2% 0:6 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 1070 ± 5% +137.5% 2542 ± 4% stress-ng.loop.ops 17.42 ± 5% +140.6% 41.91 ± 4% stress-ng.loop.ops_per_sec 64.37 -2.1% 63.02 stress-ng.time.elapsed_time 64.37 -2.1% 63.02 stress-ng.time.elapsed_time.max 2075 ± 4% +75.2% 3635 ± 4% stress-ng.time.involuntary_context_switches 26.83 ± 3% +59.6% 42.83 ± 4% stress-ng.time.percent_of_cpu_this_job_got 17.58 ± 3% +53.9% 27.06 ± 3% stress-ng.time.system_time 27069 ± 7% +31.1% 35492 ± 4% stress-ng.time.voluntary_context_switches 0.44 ± 5% +0.2 0.62 ± 3% mpstat.cpu.all.sys% 0.16 ± 2% +0.0 0.19 ± 3% mpstat.cpu.all.usr% 598.17 ± 16% -73.4% 159.00 ± 9% vmstat.io.bi 361.00 ± 14% -95.9% 14.67 ± 19% vmstat.memory.buff 4581 ± 5% +13.3% 5189 ± 2% vmstat.system.cs 1161 ± 2% +13.7% 1321 slabinfo.request_queue.active_objs 1161 ± 2% +13.7% 1321 slabinfo.request_queue.num_objs 3470 +16.0% 4027 ± 3% slabinfo.shmem_inode_cache.active_objs 3474 +16.0% 4028 ± 3% slabinfo.shmem_inode_cache.num_objs 782.67 ± 3% +6.4% 833.00 ± 2% proc-vmstat.nr_active_anon 94.50 ± 4% -27.9% 68.17 proc-vmstat.nr_active_file 95.50 ± 8% -59.9% 38.33 proc-vmstat.nr_inactive_file 3743 +1.3% 3791 proc-vmstat.nr_shmem 25198 ± 2% -4.3% 24124 proc-vmstat.nr_slab_reclaimable 782.67 ± 3% +6.4% 833.00 ± 2% proc-vmstat.nr_zone_active_anon 94.50 ± 4% -27.9% 68.17 proc-vmstat.nr_zone_active_file 95.50 ± 8% -59.9% 38.33 proc-vmstat.nr_zone_inactive_file 481597 ± 2% +5.1% 506266 proc-vmstat.numa_hit 392250 ± 2% +6.0% 415881 proc-vmstat.numa_local 89347 +1.2% 90385 proc-vmstat.numa_other 3740 ± 7% +45.5% 5443 ± 4% proc-vmstat.pgactivate 548576 ± 3% +16.1% 636943 proc-vmstat.pgalloc_normal 398656 ± 4% +22.5% 488228 ± 2% proc-vmstat.pgfree 40416 ± 16% -73.9% 10544 ± 9% proc-vmstat.pgpgin 5115 ± 9% -54.9% 2307 ± 7% proc-vmstat.unevictable_pgs_culled 3.809e+08 ± 3% +13.6% 4.326e+08 perf-stat.i.branch-instructions 2358502 ± 10% +27.9% 3017632 ± 9% perf-stat.i.cache-misses 4439 ± 5% +15.6% 5131 ± 2% perf-stat.i.context-switches 4.894e+09 ± 6% +22.3% 5.984e+09 ± 6% perf-stat.i.cpu-cycles 160.49 ± 3% +10.3% 177.08 perf-stat.i.cpu-migrations 4.319e+08 ± 3% +14.9% 4.961e+08 perf-stat.i.dTLB-loads 1.899e+08 ± 4% +22.2% 2.321e+08 ± 2% perf-stat.i.dTLB-stores 1.828e+09 ± 3% +15.1% 2.104e+09 perf-stat.i.instructions 0.05 ± 6% +22.3% 0.06 ± 6% perf-stat.i.metric.GHz 10.69 ± 2% +16.0% 12.40 perf-stat.i.metric.M/sec 364538 ± 3% +41.4% 515603 ± 4% perf-stat.i.node-load-misses 112390 ± 5% +52.2% 171079 ± 7% perf-stat.i.node-loads 186665 ± 4% +75.9% 328357 ± 3% perf-stat.i.node-store-misses 83230 ± 19% +58.1% 131582 ± 17% perf-stat.i.node-stores 3.755e+08 ± 3% +13.5% 4.261e+08 perf-stat.ps.branch-instructions 2322845 ± 10% +27.9% 2970407 ± 9% perf-stat.ps.cache-misses 4370 ± 5% +15.5% 5049 ± 2% perf-stat.ps.context-switches 4.82e+09 ± 6% +22.2% 5.89e+09 ± 6% perf-stat.ps.cpu-cycles 158.00 ± 3% +10.3% 174.27 perf-stat.ps.cpu-migrations 4.256e+08 ± 3% +14.8% 4.885e+08 perf-stat.ps.dTLB-loads 1.871e+08 ± 4% +22.1% 2.285e+08 ± 2% perf-stat.ps.dTLB-stores 1.802e+09 ± 3% +15.0% 2.072e+09 perf-stat.ps.instructions 359034 ± 3% +41.4% 507579 ± 4% perf-stat.ps.node-load-misses 110677 ± 5% +52.1% 168386 ± 7% perf-stat.ps.node-loads 183795 ± 4% +75.8% 323141 ± 3% perf-stat.ps.node-store-misses 82030 ± 19% +57.9% 129529 ± 17% perf-stat.ps.node-stores 1.177e+11 ± 3% +12.5% 1.323e+11 perf-stat.total.instructions 2168 ± 38% -60.6% 854.17 ± 44% interrupts.CPU0.CAL:Function_call_interrupts 126.17 ± 34% +103.6% 256.83 ± 25% interrupts.CPU1.NMI:Non-maskable_interrupts 126.17 ± 34% +103.6% 256.83 ± 25% interrupts.CPU1.PMI:Performance_monitoring_interrupts 552.33 ± 4% +32.5% 731.83 ± 33% interrupts.CPU14.CAL:Function_call_interrupts 157.17 ± 16% +58.2% 248.67 ± 24% interrupts.CPU19.NMI:Non-maskable_interrupts 157.17 ± 16% +58.2% 248.67 ± 24% interrupts.CPU19.PMI:Performance_monitoring_interrupts 149.33 ± 19% +48.4% 221.67 ± 13% interrupts.CPU20.NMI:Non-maskable_interrupts 149.33 ± 19% +48.4% 221.67 ± 13% interrupts.CPU20.PMI:Performance_monitoring_interrupts 150.67 ± 46% +50.8% 227.17 ± 17% interrupts.CPU21.NMI:Non-maskable_interrupts 150.67 ± 46% +50.8% 227.17 ± 17% interrupts.CPU21.PMI:Performance_monitoring_interrupts 126.50 ± 22% +84.7% 233.67 ± 18% interrupts.CPU23.NMI:Non-maskable_interrupts 126.50 ± 22% +84.7% 233.67 ± 18% interrupts.CPU23.PMI:Performance_monitoring_interrupts 139.67 ± 26% +36.3% 190.33 ± 15% interrupts.CPU25.NMI:Non-maskable_interrupts 139.67 ± 26% +36.3% 190.33 ± 15% interrupts.CPU25.PMI:Performance_monitoring_interrupts 108.33 ± 29% +108.5% 225.83 ± 19% interrupts.CPU29.NMI:Non-maskable_interrupts 108.33 ± 29% +108.5% 225.83 ± 19% interrupts.CPU29.PMI:Performance_monitoring_interrupts 557.33 ± 3% +32.6% 739.17 ± 22% interrupts.CPU36.CAL:Function_call_interrupts 93.33 ± 29% +95.7% 182.67 ± 26% interrupts.CPU39.NMI:Non-maskable_interrupts 93.33 ± 29% +95.7% 182.67 ± 26% interrupts.CPU39.PMI:Performance_monitoring_interrupts 146.17 ± 18% +62.7% 237.83 ± 45% interrupts.CPU48.NMI:Non-maskable_interrupts 146.17 ± 18% +62.7% 237.83 ± 45% interrupts.CPU48.PMI:Performance_monitoring_interrupts 551.83 ± 4% +11.4% 614.67 ± 3% interrupts.CPU58.CAL:Function_call_interrupts 540.33 ± 2% +14.9% 620.67 ± 7% interrupts.CPU59.CAL:Function_call_interrupts 132.83 ± 25% +58.1% 210.00 ± 28% interrupts.CPU61.NMI:Non-maskable_interrupts 132.83 ± 25% +58.1% 210.00 ± 28% interrupts.CPU61.PMI:Performance_monitoring_interrupts 538.50 +14.1% 614.33 ± 4% interrupts.CPU64.CAL:Function_call_interrupts 118.67 ± 33% +84.3% 218.67 ± 30% interrupts.CPU64.NMI:Non-maskable_interrupts 118.67 ± 33% +84.3% 218.67 ± 30% interrupts.CPU64.PMI:Performance_monitoring_interrupts 544.00 ± 4% +12.5% 612.17 ± 3% interrupts.CPU65.CAL:Function_call_interrupts 114.50 ± 28% +139.9% 274.67 ± 30% interrupts.CPU66.NMI:Non-maskable_interrupts 114.50 ± 28% +139.9% 274.67 ± 30% interrupts.CPU66.PMI:Performance_monitoring_interrupts 601.17 ± 9% +15.7% 695.83 ± 7% interrupts.CPU7.CAL:Function_call_interrupts 554.33 ± 2% +14.9% 637.00 ± 6% interrupts.CPU71.CAL:Function_call_interrupts 133.50 ± 24% +67.3% 223.33 ± 17% interrupts.CPU71.NMI:Non-maskable_interrupts 133.50 ± 24% +67.3% 223.33 ± 17% interrupts.CPU71.PMI:Performance_monitoring_interrupts 564.67 ± 10% +20.4% 680.00 ± 20% interrupts.CPU74.CAL:Function_call_interrupts 566.00 ± 8% +17.2% 663.50 ± 12% interrupts.CPU90.CAL:Function_call_interrupts 550.00 ± 3% +16.7% 642.00 ± 13% interrupts.CPU93.CAL:Function_call_interrupts 554.67 ± 5% +18.1% 655.17 ± 16% interrupts.CPU95.CAL:Function_call_interrupts 13344 ± 6% +37.3% 18317 ± 6% interrupts.NMI:Non-maskable_interrupts 13344 ± 6% +37.3% 18317 ± 6% interrupts.PMI:Performance_monitoring_interrupts 58.74 ± 25% +1561.4% 975.89 ± 48% sched_debug.cfs_rq:/.load_avg.avg 998.33 ± 4% +3862.1% 39555 ± 15% sched_debug.cfs_rq:/.load_avg.max 202.08 ± 17% +2535.1% 5324 ± 27% sched_debug.cfs_rq:/.load_avg.stddev 0.15 ± 20% -42.2% 0.09 ± 9% sched_debug.cfs_rq:/.nr_running.avg 0.36 ± 8% -25.7% 0.27 ± 5% sched_debug.cfs_rq:/.nr_running.stddev 31.58 ± 48% -82.4% 5.57 ± 81% sched_debug.cfs_rq:/.removed.load_avg.avg 165.98 ± 29% -71.3% 47.65 ± 80% sched_debug.cfs_rq:/.removed.load_avg.stddev 10.40 ± 51% -76.2% 2.47 ± 87% sched_debug.cfs_rq:/.removed.runnable_avg.avg 423.17 ± 19% -57.0% 182.17 ± 81% sched_debug.cfs_rq:/.removed.runnable_avg.max 58.41 ± 31% -64.5% 20.76 ± 81% sched_debug.cfs_rq:/.removed.runnable_avg.stddev 10.39 ± 51% -76.2% 2.47 ± 87% sched_debug.cfs_rq:/.removed.util_avg.avg 422.67 ± 19% -56.9% 182.17 ± 81% sched_debug.cfs_rq:/.removed.util_avg.max 58.37 ± 31% -64.4% 20.76 ± 81% sched_debug.cfs_rq:/.removed.util_avg.stddev 326.47 ± 3% -46.7% 173.98 ± 7% sched_debug.cfs_rq:/.runnable_avg.avg 1189 ± 17% -24.8% 894.75 ± 9% sched_debug.cfs_rq:/.runnable_avg.max 300.15 ± 6% -30.8% 207.57 ± 3% sched_debug.cfs_rq:/.runnable_avg.stddev 325.98 ± 3% -46.7% 173.64 ± 7% sched_debug.cfs_rq:/.util_avg.avg 1189 ± 17% -24.9% 893.17 ± 10% sched_debug.cfs_rq:/.util_avg.max 299.75 ± 6% -30.8% 207.34 ± 3% sched_debug.cfs_rq:/.util_avg.stddev 36.35 ± 16% -56.5% 15.81 ± 21% sched_debug.cfs_rq:/.util_est_enqueued.avg 824.33 -41.3% 483.50 ± 5% sched_debug.cfs_rq:/.util_est_enqueued.max 134.88 ± 6% -48.8% 69.12 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.stddev 742830 ± 2% +22.7% 911266 ± 11% sched_debug.cpu.avg_idle.avg 6798 ±154% +2699.7% 190334 ± 60% sched_debug.cpu.avg_idle.min 30009 +100.2% 60091 sched_debug.cpu.clock.avg 30014 +100.2% 60096 sched_debug.cpu.clock.max 30004 +100.3% 60085 sched_debug.cpu.clock.min 29877 +99.2% 59522 sched_debug.cpu.clock_task.avg 29996 +99.0% 59684 sched_debug.cpu.clock_task.max 24935 ± 2% +118.5% 54491 sched_debug.cpu.clock_task.min 299.29 ± 23% -28.7% 213.34 ± 7% sched_debug.cpu.curr->pid.avg 2546 +61.9% 4122 sched_debug.cpu.curr->pid.max 0.14 ± 20% -37.8% 0.08 ± 10% sched_debug.cpu.nr_running.avg 0.34 ± 8% -22.2% 0.27 ± 6% sched_debug.cpu.nr_running.stddev 1483 +111.8% 3142 sched_debug.cpu.nr_switches.avg 8532 ± 20% +50.3% 12820 ± 11% sched_debug.cpu.nr_switches.max 458.00 ± 10% +172.5% 1248 ± 13% sched_debug.cpu.nr_switches.min 1240 ± 14% +47.5% 1830 ± 5% sched_debug.cpu.nr_switches.stddev 0.01 ± 57% +6158.3% 0.65 ± 8% sched_debug.cpu.nr_uninterruptible.avg -13.67 +51.2% -20.67 sched_debug.cpu.nr_uninterruptible.min 7.01 ± 7% +25.3% 8.78 ± 8% sched_debug.cpu.nr_uninterruptible.stddev 30006 +100.2% 60087 sched_debug.cpu_clk 29495 +101.9% 59559 sched_debug.ktime 30339 +99.2% 60436 sched_debug.sched_clk 12341 ± 5% -11.2% 10959 ± 7% softirqs.CPU0.SCHED 10516 ± 4% -15.9% 8848 ± 6% softirqs.CPU1.RCU 10656 ± 3% -17.0% 8848 ± 6% softirqs.CPU10.RCU 10582 ± 4% -19.2% 8553 ± 5% softirqs.CPU11.RCU 10427 ± 3% -14.7% 8898 ± 9% softirqs.CPU12.RCU 10361 ± 2% -17.1% 8594 ± 7% softirqs.CPU13.RCU 10465 ± 4% -15.6% 8834 ± 5% softirqs.CPU14.RCU 10493 ± 3% -16.6% 8750 ± 4% softirqs.CPU15.RCU 10044 ± 5% -16.9% 8350 ± 4% softirqs.CPU16.RCU 9997 ± 6% -18.1% 8187 ± 2% softirqs.CPU17.RCU 9862 ± 7% -18.4% 8048 ± 3% softirqs.CPU18.RCU 9446 ± 5% -12.7% 8249 ± 7% softirqs.CPU18.SCHED 10033 ± 5% -19.5% 8072 softirqs.CPU19.RCU 10869 ± 10% -16.9% 9030 ± 4% softirqs.CPU2.RCU 9861 ± 5% -16.0% 8286 ± 2% softirqs.CPU20.RCU 10127 ± 7% -16.0% 8508 ± 7% softirqs.CPU21.RCU 10064 ± 7% -18.3% 8226 ± 4% softirqs.CPU22.RCU 10065 ± 5% -21.0% 7955 ± 2% softirqs.CPU23.RCU 9651 ± 6% -13.9% 8314 ± 8% softirqs.CPU27.RCU 10519 ± 10% -23.4% 8055 ± 7% softirqs.CPU28.RCU 10166 ± 9% -20.9% 8044 ± 7% softirqs.CPU29.RCU 10692 ± 3% -16.8% 8896 ± 5% softirqs.CPU3.RCU 9779 ± 9% -14.5% 8358 ± 5% softirqs.CPU30.RCU 9860 ± 4% -17.9% 8092 ± 4% softirqs.CPU31.RCU 10565 ± 6% -13.6% 9132 ± 5% softirqs.CPU33.RCU 10572 ± 8% -13.5% 9145 ± 6% softirqs.CPU35.RCU 10560 ± 8% -15.8% 8890 ± 4% softirqs.CPU37.RCU 10667 ± 4% -21.8% 8339 ± 9% softirqs.CPU4.RCU 9938 ± 4% -12.3% 8712 ± 7% softirqs.CPU4.SCHED 10227 ± 7% -15.4% 8655 ± 4% softirqs.CPU42.RCU 10465 ± 6% -13.1% 9096 ± 8% softirqs.CPU43.RCU 11087 ± 13% -15.0% 9429 ± 8% softirqs.CPU45.RCU 9642 ± 2% -16.6% 8038 ± 8% softirqs.CPU46.SCHED 10998 ± 15% -25.7% 8172 ± 9% softirqs.CPU48.RCU 9964 ± 5% -18.5% 8116 ± 11% softirqs.CPU49.RCU 10461 ± 4% -16.4% 8746 ± 4% softirqs.CPU5.RCU 10071 ± 6% -14.0% 8664 ± 4% softirqs.CPU50.RCU 10389 ± 5% -16.0% 8726 ± 6% softirqs.CPU51.RCU 10872 ± 8% -18.2% 8890 ± 7% softirqs.CPU52.RCU 10574 ± 4% -15.0% 8984 ± 6% softirqs.CPU53.RCU 10505 ± 6% -14.3% 9003 ± 5% softirqs.CPU54.RCU 10362 ± 5% -16.6% 8645 ± 6% softirqs.CPU55.RCU 10612 ± 8% -17.3% 8779 ± 7% softirqs.CPU56.RCU 10255 ± 3% -16.3% 8580 ± 4% softirqs.CPU57.RCU 10390 ± 5% -15.2% 8813 ± 7% softirqs.CPU58.RCU 10671 ± 3% -16.9% 8873 ± 5% softirqs.CPU59.RCU 10631 ± 4% -15.3% 9008 ± 6% softirqs.CPU6.RCU 10206 ± 4% -17.7% 8403 ± 3% softirqs.CPU60.RCU 10376 ± 4% -18.7% 8432 ± 5% softirqs.CPU61.RCU 10258 ± 3% -16.1% 8605 ± 4% softirqs.CPU62.RCU 10352 ± 4% -17.0% 8591 ± 6% softirqs.CPU63.RCU 9925 ± 8% -17.2% 8214 ± 3% softirqs.CPU64.RCU 9913 ± 6% -19.3% 8002 ± 2% softirqs.CPU65.RCU 9628 ± 8% -19.6% 7736 ± 3% softirqs.CPU66.RCU 9923 ± 6% -19.4% 7999 ± 2% softirqs.CPU67.RCU 9700 ± 5% -16.5% 8095 ± 4% softirqs.CPU68.RCU 9727 ± 5% -17.2% 8050 ± 2% softirqs.CPU69.RCU 10720 ± 4% -19.6% 8618 ± 6% softirqs.CPU7.RCU 9775 ± 6% -16.6% 8149 ± 5% softirqs.CPU70.RCU 10177 ± 5% -20.8% 8059 softirqs.CPU71.RCU 9844 ± 5% -17.5% 8117 ± 5% softirqs.CPU72.RCU 9480 ± 6% -17.0% 7871 ± 2% softirqs.CPU73.RCU 9917 ± 8% -17.6% 8170 ± 6% softirqs.CPU75.RCU 9730 ± 6% -17.1% 8068 ± 5% softirqs.CPU76.RCU 9739 ± 7% -17.9% 7994 ± 4% softirqs.CPU77.RCU 10010 ± 19% -18.1% 8193 ± 5% softirqs.CPU78.RCU 10620 ± 5% -16.4% 8883 ± 7% softirqs.CPU8.RCU 10466 ± 6% -14.6% 8935 ± 8% softirqs.CPU80.RCU 10111 ± 4% -12.3% 8870 ± 7% softirqs.CPU81.RCU 10515 ± 7% -16.5% 8785 ± 5% softirqs.CPU85.RCU 10184 ± 9% -13.3% 8832 ± 5% softirqs.CPU87.RCU 10385 ± 4% -16.8% 8645 ± 3% softirqs.CPU9.RCU 10170 ± 5% -10.8% 9072 ± 6% softirqs.CPU95.RCU 987064 ± 4% -15.5% 833756 ± 4% softirqs.RCU 898196 -9.6% 811795 ± 2% softirqs.SCHED 15824 ± 3% +41.4% 22371 softirqs.TIMER 72.44 -8.8 63.63 ± 3% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 71.66 -8.8 62.89 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 71.66 -8.8 62.89 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 71.60 -8.7 62.86 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 23.80 ± 8% -6.5 17.35 ± 9% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 62.45 ± 4% -5.6 56.85 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 18.01 ± 4% -4.4 13.60 ± 5% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 11.56 ± 3% -2.8 8.78 ± 5% perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 11.51 ± 3% -2.8 8.74 ± 5% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 11.25 ± 4% -2.7 8.56 ± 5% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 4.45 ± 14% -1.9 2.57 ± 8% perf-profile.calltrace.cycles-pp.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.88 ± 14% -1.4 4.49 ± 9% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt 3.88 ± 11% -1.0 2.85 ± 8% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack 3.42 ± 8% -1.0 2.44 ± 7% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 2.79 ± 9% -0.9 1.93 ± 9% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt 2.80 ± 8% -0.9 1.94 ± 9% perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 2.80 ± 8% -0.8 1.96 ± 9% perf-profile.calltrace.cycles-pp.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 3.19 ± 12% -0.8 2.42 ± 10% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 3.03 ± 9% -0.7 2.29 ± 9% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 1.26 ± 9% -0.5 0.72 ± 5% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu 1.59 ± 7% -0.4 1.19 ± 8% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 1.43 ± 15% -0.4 1.06 ± 15% perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 1.39 ± 15% -0.4 1.03 ± 15% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 1.07 ± 7% -0.3 0.80 ± 19% perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt 0.68 ± 8% +0.4 1.12 ± 13% perf-profile.calltrace.cycles-pp.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue.loop_remove 0.68 ± 8% +0.4 1.12 ± 13% perf-profile.calltrace.cycles-pp.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue 0.68 ± 8% +0.4 1.12 ± 13% perf-profile.calltrace.cycles-pp.blk_mq_exit_queue.blk_cleanup_queue.loop_remove.loop_control_ioctl.__x64_sys_ioctl 0.68 ± 8% +0.4 1.12 ± 13% perf-profile.calltrace.cycles-pp.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue.loop_remove.loop_control_ioctl 0.69 ± 7% +0.5 1.15 ± 12% perf-profile.calltrace.cycles-pp.blk_cleanup_queue.loop_remove.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64 0.32 ±104% +0.5 0.84 ± 7% perf-profile.calltrace.cycles-pp.loop_add.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.79 ± 6% +0.5 1.34 ± 11% perf-profile.calltrace.cycles-pp.loop_remove.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.9 0.90 ± 16% perf-profile.calltrace.cycles-pp.smp_call_function_single.sync_rcu_exp_select_node_cpus.process_one_work.worker_thread.kthread 0.00 +0.9 0.90 ± 16% perf-profile.calltrace.cycles-pp.sync_rcu_exp_select_node_cpus.process_one_work.worker_thread.kthread.ret_from_fork 3.75 ± 13% +3.6 7.37 ± 14% perf-profile.calltrace.cycles-pp.loop_configure.lo_ioctl.blkdev_ioctl.block_ioctl.__x64_sys_ioctl 3.55 ± 14% +3.8 7.33 ± 14% perf-profile.calltrace.cycles-pp.printk.set_capacity_and_notify.cold.loop_set_size.loop_configure.lo_ioctl 3.55 ± 14% +3.8 7.33 ± 14% perf-profile.calltrace.cycles-pp.vprintk_emit.printk.set_capacity_and_notify.cold.loop_set_size.loop_configure 3.55 ± 14% +3.8 7.33 ± 14% perf-profile.calltrace.cycles-pp.loop_set_size.loop_configure.lo_ioctl.blkdev_ioctl.block_ioctl 3.55 ± 14% +3.8 7.33 ± 14% perf-profile.calltrace.cycles-pp.set_capacity_and_notify.cold.loop_set_size.loop_configure.lo_ioctl.blkdev_ioctl 2.72 ± 13% +4.1 6.80 ± 9% perf-profile.calltrace.cycles-pp.printk.set_capacity_and_notify.cold.loop_set_size.lo_ioctl.blkdev_ioctl 2.72 ± 13% +4.1 6.80 ± 9% perf-profile.calltrace.cycles-pp.vprintk_emit.printk.set_capacity_and_notify.cold.loop_set_size.lo_ioctl 2.72 ± 13% +4.1 6.81 ± 9% perf-profile.calltrace.cycles-pp.loop_set_size.lo_ioctl.blkdev_ioctl.block_ioctl.__x64_sys_ioctl 2.72 ± 13% +4.1 6.81 ± 9% perf-profile.calltrace.cycles-pp.set_capacity_and_notify.cold.loop_set_size.lo_ioctl.blkdev_ioctl.block_ioctl 3.98 ± 9% +4.1 8.11 ± 11% perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write 5.41 ± 4% +4.5 9.95 ± 12% perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock 5.41 ± 4% +4.5 9.95 ± 12% perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit 9.42 ± 10% +4.8 14.25 ± 10% perf-profile.calltrace.cycles-pp.blkdev_ioctl.block_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 9.42 ± 10% +4.8 14.25 ± 10% perf-profile.calltrace.cycles-pp.block_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 9.37 ± 10% +4.9 14.23 ± 10% perf-profile.calltrace.cycles-pp.lo_ioctl.blkdev_ioctl.block_ioctl.__x64_sys_ioctl.do_syscall_64 5.68 ± 5% +5.1 10.78 ± 12% perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.printk 5.83 ± 7% +5.5 11.29 ± 12% perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.printk.set_capacity_and_notify.cold 6.26 ± 7% +5.8 12.07 ± 12% perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.printk.set_capacity_and_notify.cold.loop_set_size 8.32 ± 3% +6.7 15.03 ± 5% perf-profile.calltrace.cycles-pp.memcpy_toio.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread 8.53 ± 3% +7.0 15.49 ± 5% perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork 8.97 ± 3% +7.7 16.71 ± 4% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 8.99 ± 3% +7.7 16.73 ± 4% perf-profile.calltrace.cycles-pp.ret_from_fork 8.62 ± 3% +7.9 16.50 ± 4% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 8.58 ± 3% +7.9 16.47 ± 4% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 72.44 -8.8 63.63 ± 4% perf-profile.children.cycles-pp.do_idle 72.44 -8.8 63.63 ± 3% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 72.44 -8.8 63.63 ± 3% perf-profile.children.cycles-pp.cpu_startup_entry 71.66 -8.8 62.89 ± 3% perf-profile.children.cycles-pp.start_secondary 8.16 ± 11% -7.3 0.87 ± 27% perf-profile.children.cycles-pp.__mutex_lock 6.08 ± 9% -5.8 0.31 ± 23% perf-profile.children.cycles-pp.mutex_spin_on_owner 63.12 ± 4% -5.6 57.55 ± 4% perf-profile.children.cycles-pp.cpuidle_enter 63.05 ± 4% -5.6 57.49 ± 4% perf-profile.children.cycles-pp.cpuidle_enter_state 21.81 ± 5% -5.4 16.42 ± 6% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 18.48 ± 4% -4.5 13.94 ± 5% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 15.08 ± 3% -3.8 11.31 ± 4% perf-profile.children.cycles-pp.asm_call_sysvec_on_stack 11.78 ± 3% -2.8 8.94 ± 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 11.54 ± 3% -2.8 8.77 ± 5% perf-profile.children.cycles-pp.hrtimer_interrupt 4.45 ± 14% -1.9 2.57 ± 8% perf-profile.children.cycles-pp.loop_control_ioctl 2.01 ± 28% -1.5 0.52 ± 35% perf-profile.children.cycles-pp.osq_lock 6.07 ± 15% -1.4 4.63 ± 8% perf-profile.children.cycles-pp.__hrtimer_run_queues 1.25 ± 30% -1.2 0.10 ± 54% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 1.23 ± 30% -1.1 0.09 ± 57% perf-profile.children.cycles-pp.task_work_run 1.31 ± 32% -1.1 0.18 ± 41% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 1.19 ± 29% -1.1 0.07 ± 77% perf-profile.children.cycles-pp.__fput 1.13 ± 31% -1.1 0.04 ±105% perf-profile.children.cycles-pp.blkdev_close 1.13 ± 31% -1.1 0.04 ±105% perf-profile.children.cycles-pp.blkdev_put 1.10 ± 32% -1.1 0.04 ±105% perf-profile.children.cycles-pp.__blkdev_put 3.96 ± 11% -1.0 2.92 ± 8% perf-profile.children.cycles-pp.tick_sched_timer 3.55 ± 8% -1.0 2.55 ± 8% perf-profile.children.cycles-pp.irq_exit_rcu 0.99 ± 37% -1.0 0.03 ±105% perf-profile.children.cycles-pp.lo_release 2.93 ± 9% -0.9 2.04 ± 11% perf-profile.children.cycles-pp.__softirqentry_text_start 2.92 ± 9% -0.9 2.05 ± 12% perf-profile.children.cycles-pp.do_softirq_own_stack 3.27 ± 12% -0.8 2.48 ± 9% perf-profile.children.cycles-pp.tick_sched_handle 3.12 ± 9% -0.8 2.36 ± 9% perf-profile.children.cycles-pp.update_process_times 0.77 ±172% -0.7 0.05 ± 46% perf-profile.children.cycles-pp.poll_idle 1.28 ± 9% -0.5 0.75 ± 12% perf-profile.children.cycles-pp.rebalance_domains 1.31 ± 15% -0.5 0.78 ± 15% perf-profile.children.cycles-pp.do_sys_open 1.31 ± 15% -0.5 0.78 ± 15% perf-profile.children.cycles-pp.do_sys_openat2 0.67 ± 27% -0.5 0.17 ± 30% perf-profile.children.cycles-pp.__blkdev_get 0.67 ± 27% -0.5 0.17 ± 30% perf-profile.children.cycles-pp.lo_open 1.21 ± 18% -0.5 0.73 ± 15% perf-profile.children.cycles-pp.do_filp_open 1.20 ± 18% -0.5 0.73 ± 15% perf-profile.children.cycles-pp.path_openat 0.70 ± 24% -0.4 0.26 ± 26% perf-profile.children.cycles-pp.blkdev_get_by_dev 0.70 ± 24% -0.4 0.26 ± 25% perf-profile.children.cycles-pp.blkdev_open 0.72 ± 24% -0.4 0.29 ± 23% perf-profile.children.cycles-pp.do_dentry_open 1.64 ± 8% -0.4 1.23 ± 8% perf-profile.children.cycles-pp.scheduler_tick 1.48 ± 17% -0.4 1.08 ± 15% perf-profile.children.cycles-pp.irq_enter_rcu 1.44 ± 16% -0.4 1.05 ± 15% perf-profile.children.cycles-pp.tick_irq_enter 1.10 ± 6% -0.3 0.82 ± 18% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 0.33 ± 17% -0.3 0.05 ± 80% perf-profile.children.cycles-pp.unit_file_find_dropin_paths 0.93 ± 12% -0.2 0.69 ± 9% perf-profile.children.cycles-pp.read_tsc 0.88 ± 13% -0.2 0.64 ± 15% perf-profile.children.cycles-pp.load_balance 0.24 ± 12% -0.2 0.05 ± 79% perf-profile.children.cycles-pp.openat64 0.51 ± 29% -0.2 0.34 ± 10% perf-profile.children.cycles-pp.run_rebalance_domains 0.61 ± 20% -0.1 0.46 ± 9% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.41 ± 12% -0.1 0.27 ± 12% perf-profile.children.cycles-pp.arch_scale_freq_tick 0.43 ± 8% -0.1 0.30 ± 19% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.12 ± 17% -0.1 0.05 ± 47% perf-profile.children.cycles-pp.open64 0.21 ± 17% -0.1 0.13 ± 17% perf-profile.children.cycles-pp.new_sync_read 0.20 ± 9% -0.1 0.13 ± 27% perf-profile.children.cycles-pp.call_cpuidle 0.14 ± 14% -0.1 0.08 ± 23% perf-profile.children.cycles-pp.cpumask_next_and 0.12 ± 20% -0.1 0.07 ± 47% perf-profile.children.cycles-pp.mmput 0.11 ± 17% -0.0 0.07 ± 47% perf-profile.children.cycles-pp.exit_mmap 0.06 ± 54% +0.1 0.11 ± 20% perf-profile.children.cycles-pp.evict 0.04 ± 80% +0.1 0.10 ± 27% perf-profile.children.cycles-pp.blk_mq_init_queue_data 0.11 ± 33% +0.1 0.18 ± 21% perf-profile.children.cycles-pp.new_inode_pseudo 0.12 ± 31% +0.1 0.19 ± 19% perf-profile.children.cycles-pp.new_inode 0.07 ± 53% +0.1 0.15 ± 15% perf-profile.children.cycles-pp.d_invalidate 0.11 ± 32% +0.1 0.19 ± 16% perf-profile.children.cycles-pp.debugfs_get_inode 0.08 ± 24% +0.1 0.17 ± 51% perf-profile.children.cycles-pp.del_gendisk 0.13 ± 26% +0.1 0.25 ± 11% perf-profile.children.cycles-pp.cpumask_next 0.22 ± 15% +0.1 0.35 ± 7% perf-profile.children.cycles-pp.__dentry_kill 0.29 ± 10% +0.1 0.42 ± 10% perf-profile.children.cycles-pp.dput 0.30 ± 9% +0.2 0.50 ± 20% perf-profile.children.cycles-pp.mnt_get_count 0.31 ± 8% +0.2 0.55 ± 22% perf-profile.children.cycles-pp.mntput_no_expire 0.37 ± 15% +0.3 0.64 ± 13% perf-profile.children.cycles-pp.sys_imageblit 0.37 ± 15% +0.3 0.64 ± 13% perf-profile.children.cycles-pp.drm_fb_helper_sys_imageblit 0.54 ± 26% +0.3 0.85 ± 7% perf-profile.children.cycles-pp.loop_add 0.41 ± 14% +0.3 0.72 ± 13% perf-profile.children.cycles-pp.bit_putcs 0.41 ± 14% +0.3 0.73 ± 13% perf-profile.children.cycles-pp.fbcon_putcs 0.41 ± 15% +0.3 0.74 ± 12% perf-profile.children.cycles-pp.fbcon_redraw 0.42 ± 14% +0.3 0.75 ± 13% perf-profile.children.cycles-pp.lf 0.42 ± 14% +0.3 0.75 ± 13% perf-profile.children.cycles-pp.con_scroll 0.42 ± 14% +0.3 0.75 ± 13% perf-profile.children.cycles-pp.fbcon_scroll 0.45 ± 13% +0.3 0.79 ± 12% perf-profile.children.cycles-pp.vt_console_print 0.68 ± 8% +0.4 1.12 ± 13% perf-profile.children.cycles-pp.blk_mq_exit_queue 0.68 ± 8% +0.4 1.12 ± 13% perf-profile.children.cycles-pp.blk_mq_debugfs_unregister_hctx 0.69 ± 7% +0.5 1.15 ± 12% perf-profile.children.cycles-pp.blk_cleanup_queue 0.70 ± 8% +0.5 1.21 ± 14% perf-profile.children.cycles-pp.debugfs_remove 0.70 ± 8% +0.5 1.21 ± 14% perf-profile.children.cycles-pp.simple_recursive_removal 0.79 ± 6% +0.5 1.34 ± 11% perf-profile.children.cycles-pp.loop_remove 0.28 ± 20% +0.6 0.87 ± 17% perf-profile.children.cycles-pp.io_serial_out 0.00 +0.9 0.94 ± 18% perf-profile.children.cycles-pp.sync_rcu_exp_select_node_cpus 0.00 +0.9 0.94 ± 18% perf-profile.children.cycles-pp.smp_call_function_single 3.75 ± 13% +3.6 7.37 ± 14% perf-profile.children.cycles-pp.loop_configure 4.51 ± 9% +4.3 8.79 ± 11% perf-profile.children.cycles-pp.io_serial_in 5.74 ± 6% +4.6 10.32 ± 12% perf-profile.children.cycles-pp.serial8250_console_putchar 6.00 ± 6% +4.8 10.81 ± 12% perf-profile.children.cycles-pp.wait_for_xmitr 9.43 ± 10% +4.8 14.25 ± 10% perf-profile.children.cycles-pp.blkdev_ioctl 9.43 ± 10% +4.8 14.25 ± 10% perf-profile.children.cycles-pp.block_ioctl 9.38 ± 10% +4.9 14.23 ± 10% perf-profile.children.cycles-pp.lo_ioctl 6.02 ± 6% +5.2 11.17 ± 12% perf-profile.children.cycles-pp.uart_console_write 6.29 ± 6% +5.4 11.69 ± 12% perf-profile.children.cycles-pp.serial8250_console_write 6.74 ± 6% +5.8 12.49 ± 12% perf-profile.children.cycles-pp.console_unlock 8.52 ± 3% +6.9 15.47 ± 5% perf-profile.children.cycles-pp.memcpy_toio 8.53 ± 3% +7.0 15.49 ± 5% perf-profile.children.cycles-pp.drm_fb_helper_damage_work 8.97 ± 3% +7.7 16.71 ± 4% perf-profile.children.cycles-pp.kthread 8.99 ± 3% +7.8 16.74 ± 4% perf-profile.children.cycles-pp.ret_from_fork 6.74 ± 6% +7.8 14.57 ± 10% perf-profile.children.cycles-pp.printk 6.74 ± 6% +7.8 14.57 ± 10% perf-profile.children.cycles-pp.vprintk_emit 6.27 ± 7% +7.9 14.14 ± 10% perf-profile.children.cycles-pp.loop_set_size 6.27 ± 7% +7.9 14.14 ± 10% perf-profile.children.cycles-pp.set_capacity_and_notify.cold 8.62 ± 3% +7.9 16.50 ± 4% perf-profile.children.cycles-pp.worker_thread 8.58 ± 3% +7.9 16.47 ± 4% perf-profile.children.cycles-pp.process_one_work 6.07 ± 9% -5.8 0.30 ± 23% perf-profile.self.cycles-pp.mutex_spin_on_owner 1.99 ± 28% -1.5 0.52 ± 36% perf-profile.self.cycles-pp.osq_lock 0.68 ±172% -0.6 0.05 ± 46% perf-profile.self.cycles-pp.poll_idle 0.94 ± 6% -0.3 0.68 ± 21% perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.91 ± 13% -0.2 0.67 ± 9% perf-profile.self.cycles-pp.read_tsc 0.52 ± 13% -0.1 0.38 ± 10% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.40 ± 12% -0.1 0.27 ± 12% perf-profile.self.cycles-pp.arch_scale_freq_tick 0.52 ± 18% -0.1 0.40 ± 9% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.45 ± 7% -0.1 0.33 ± 20% perf-profile.self.cycles-pp.do_idle 0.23 ± 8% -0.1 0.15 ± 25% perf-profile.self.cycles-pp.__softirqentry_text_start 0.19 ± 12% -0.1 0.12 ± 26% perf-profile.self.cycles-pp.call_cpuidle 0.22 ± 12% -0.1 0.16 ± 14% perf-profile.self.cycles-pp.hrtimer_interrupt 0.04 ± 72% +0.1 0.09 ± 13% perf-profile.self.cycles-pp.cpumask_next 0.19 ± 19% +0.1 0.29 ± 27% perf-profile.self.cycles-pp.mnt_get_count 0.37 ± 15% +0.3 0.64 ± 13% perf-profile.self.cycles-pp.sys_imageblit 0.28 ± 20% +0.6 0.87 ± 17% perf-profile.self.cycles-pp.io_serial_out 0.00 +0.9 0.94 ± 18% perf-profile.self.cycles-pp.smp_call_function_single 0.00 +2.1 2.07 ± 11% perf-profile.self.cycles-pp.vprintk_emit 4.51 ± 9% +4.3 8.79 ± 11% perf-profile.self.cycles-pp.io_serial_in 8.48 ± 3% +6.9 15.39 ± 5% perf-profile.self.cycles-pp.memcpy_toio Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Oliver Sang