Greeting, FYI, we noticed a -9.5% regression of stress-ng.lockbus.ops_per_sec due to commit: commit: 8901ecc2315b850f35a7b8c1b73b12388b72aa78 ("clocksource: Retry clock read if long delays detected") https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master in testcase: stress-ng on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory with following parameters: nr_threads: 100% disk: 1HDD testtime: 60s class: memory test: lockbus cpufreq_governor: performance ucode: 0x5003006 please be noted below in dmesg.xz (attached) [ 28.110351] [ 28.302357] hrtimer: interrupt took 1878423 ns [ 29.690760] clocksource: timekeeping watchdog on CPU53: hpet read-back delay of 169583ns, attempt 4, marking unstable [ 29.860306] tsc: Marking TSC unstable due to clocksource watchdog [ 30.559390] TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'. [ 30.726282] sched_clock: Marking unstable (30052964508, 499342225)<-(30915547410, -363240730) [ 31.620401] clocksource: Switched to clocksource hpet If you fix the issue, kindly add following tag Reported-by: kernel test robot Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run bin/lkp run generated-yaml-file ========================================================================================= class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode: memory/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/lockbus/stress-ng/60s/0x5003006 commit: v5.13-rc1 8901ecc231 ("clocksource: Retry clock read if long delays detected") v5.13-rc1 8901ecc2315b850f35a7b8c1b73 ---------------- --------------------------- %stddev %change %stddev \ | \ 248755 -9.5% 225157 stress-ng.lockbus.ops 4136 -9.5% 3742 stress-ng.lockbus.ops_per_sec 6580 ± 5% +28.7% 8470 stress-ng.time.percent_of_cpu_this_job_got 200.39 ± 7% +61.5% 323.71 ± 7% stress-ng.time.system_time 3892 ± 5% +27.1% 4946 stress-ng.time.user_time 13.38 -6.7% 12.48 ± 5% boot-time.dhcp 208168 ± 14% +676.6% 1616642 ± 24% cpuidle.POLL.time 674.50 ± 12% -17.4% 557.33 ± 10% interrupts.CPU95.CAL:Function_call_interrupts 170270 ± 7% +22.1% 207860 ± 6% softirqs.RCU 225.06 +1.3% 227.89 turbostat.PkgWatt 61.83 ± 6% +28.0% 79.17 vmstat.cpu.us 28.52 ± 13% -61.1% 11.10 ± 6% iostat.cpu.system 63.82 ± 5% +27.6% 81.41 iostat.cpu.user 16.30 ± 21% -15.5 0.83 ± 42% mpstat.cpu.all.irq% 57.65 ± 7% +17.1 74.73 mpstat.cpu.all.usr% 0.00 ± 75% +286.7% 0.01 ± 35% perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.01 ± 14% +140.0% 0.02 ± 13% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 0.00 ± 72% +255.0% 0.01 ± 5% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.01 ± 6% +180.0% 0.02 ± 60% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 9.00 ± 30% -48.1% 4.67 ± 45% perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 550561 +5.5% 580618 proc-vmstat.nr_file_pages 307417 +10.0% 338160 ± 2% proc-vmstat.nr_mapped 2993 +8.4% 3246 proc-vmstat.nr_page_table_pages 303607 +9.9% 333614 ± 3% proc-vmstat.nr_shmem 694035 +3.0% 714653 proc-vmstat.numa_hit 607455 +3.4% 628114 proc-vmstat.numa_local 513499 +5.4% 541036 proc-vmstat.pgfault 4823 ± 4% +47.7% 7124 ± 4% proc-vmstat.pgreuse 839921 ± 8% +13.1% 950002 ± 4% numa-meminfo.node0.Inactive 839800 ± 8% +13.1% 949852 ± 4% numa-meminfo.node0.Inactive(anon) 53189 ± 8% +9.2% 58066 ± 6% numa-meminfo.node0.KReclaimable 629217 ± 3% +12.2% 705752 ± 2% numa-meminfo.node0.Mapped 7941 ± 24% +21.9% 9680 ± 3% numa-meminfo.node0.PageTables 53189 ± 8% +9.2% 58066 ± 6% numa-meminfo.node0.SReclaimable 599179 ± 4% +12.7% 675027 ± 3% numa-meminfo.node0.Shmem 605295 ± 3% +12.5% 680806 ± 2% numa-meminfo.node1.Mapped 622631 ± 3% +11.1% 691846 ± 2% numa-meminfo.node1.Shmem 193696 ± 8% +19.0% 230483 ± 3% numa-vmstat.node0.nr_inactive_anon 142337 ± 2% +19.5% 170125 ± 2% numa-vmstat.node0.nr_mapped 1868 ± 25% +25.7% 2348 ± 3% numa-vmstat.node0.nr_page_table_pages 136818 ± 2% +18.9% 162611 ± 2% numa-vmstat.node0.nr_shmem 13219 ± 8% +9.5% 14473 ± 6% numa-vmstat.node0.nr_slab_reclaimable 193666 ± 8% +19.0% 230467 ± 3% numa-vmstat.node0.nr_zone_inactive_anon 1132156 ± 12% +11.2% 1258864 ± 5% numa-vmstat.node0.numa_hit 143857 +15.9% 166725 numa-vmstat.node1.nr_file_pages 135723 ± 3% +20.1% 163039 numa-vmstat.node1.nr_mapped 142136 +17.3% 166704 numa-vmstat.node1.nr_shmem 5.12 ± 20% -68.1% 1.63 ± 24% perf-stat.i.MPKI 7.441e+08 ± 11% +61.5% 1.202e+09 ± 8% perf-stat.i.branch-instructions 2.27 ± 7% -1.3 0.94 ± 8% perf-stat.i.branch-miss-rate% 26460369 ± 13% -46.8% 14082434 ± 6% perf-stat.i.branch-misses 2567959 ± 10% -27.7% 1857035 ± 5% perf-stat.i.cache-misses 2820 ± 7% -14.3% 2416 ± 2% perf-stat.i.context-switches 303.17 ± 9% -83.0% 51.67 ± 11% perf-stat.i.cpi 129563 ± 5% -16.9% 107705 perf-stat.i.cpu-clock 190.21 ± 4% -18.4% 155.23 ± 2% perf-stat.i.cpu-migrations 298667 ± 8% -29.0% 212034 ± 4% perf-stat.i.cycles-between-cache-misses 0.20 ± 9% +0.1 0.26 ± 11% perf-stat.i.dTLB-store-miss-rate% 3.276e+08 ± 11% -41.6% 1.913e+08 ± 3% perf-stat.i.dTLB-stores 77.27 ± 3% +7.3 84.53 perf-stat.i.iTLB-load-miss-rate% 560579 ± 19% -46.9% 297416 ± 10% perf-stat.i.iTLB-loads 3.622e+09 ± 11% +52.4% 5.52e+09 ± 8% perf-stat.i.instructions 3174 ± 8% +134.9% 7456 ± 4% perf-stat.i.instructions-per-iTLB-miss 0.26 ± 11% -49.4% 0.13 ± 7% perf-stat.i.ipc 89.57 ± 12% -53.7% 41.43 ± 7% perf-stat.i.major-faults 1.77 ± 6% +32.7% 2.35 perf-stat.i.metric.GHz 14.33 ± 11% +45.7% 20.88 ± 8% perf-stat.i.metric.M/sec 5011 ± 6% -34.8% 3265 ± 4% perf-stat.i.minor-faults 73.21 ± 3% +9.3 82.54 perf-stat.i.node-load-miss-rate% 121230 ± 8% -34.7% 79105 ± 3% perf-stat.i.node-loads 50.05 ± 11% +25.9 75.94 ± 3% perf-stat.i.node-store-miss-rate% 365780 ± 12% -50.4% 181434 ± 5% perf-stat.i.node-stores 5100 ± 6% -35.2% 3306 ± 4% perf-stat.i.page-faults 129586 ± 5% -16.9% 107712 perf-stat.i.task-clock 4.80 ± 24% -72.4% 1.33 ± 20% perf-stat.overall.MPKI 2.74 ± 8% -2.2 0.57 ± 8% perf-stat.overall.branch-miss-rate% 213.44 ± 11% -75.3% 52.81 ± 12% perf-stat.overall.cpi 259389 ± 11% -21.7% 203189 ± 3% perf-stat.overall.cycles-between-cache-misses 80.09 ± 2% +5.0 85.10 perf-stat.overall.iTLB-load-miss-rate% 2124 ± 7% +238.2% 7184 ± 4% perf-stat.overall.instructions-per-iTLB-miss 0.00 ± 11% +304.5% 0.02 ± 11% perf-stat.overall.ipc 72.33 ± 3% +10.3 82.64 perf-stat.overall.node-load-miss-rate% 35.64 ± 16% +29.0 64.68 ± 2% perf-stat.overall.node-store-miss-rate% 1.982e+08 ± 11% +393.9% 9.789e+08 ± 10% perf-stat.ps.branch-instructions 783739 ± 6% +45.5% 1140674 ± 6% perf-stat.ps.cache-misses 2.27e+08 ± 10% +170.4% 6.138e+08 ± 9% perf-stat.ps.dTLB-loads 451286 ± 10% +36.6% 616231 ± 8% perf-stat.ps.iTLB-load-misses 9.571e+08 ± 11% +363.0% 4.432e+09 ± 10% perf-stat.ps.instructions 17.47 ± 14% -29.4% 12.33 perf-stat.ps.major-faults 1452 ± 5% +24.8% 1812 ± 5% perf-stat.ps.minor-faults 94781 ± 7% +130.4% 218412 ± 7% perf-stat.ps.node-load-misses 36077 ± 5% +26.9% 45791 ± 5% perf-stat.ps.node-loads 45982 ± 21% +163.9% 121369 ± 7% perf-stat.ps.node-store-misses 82536 ± 12% -20.0% 66048 perf-stat.ps.node-stores 1469 ± 5% +24.2% 1824 ± 5% perf-stat.ps.page-faults 4.04e+10 ± 8% +567.9% 2.698e+11 ± 9% perf-stat.total.instructions 39.23 ± 6% -23.6 15.65 ± 27% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 38.89 ± 6% -23.3 15.56 ± 27% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 40.52 ± 6% -20.9 19.65 ± 26% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 47.38 ± 4% -17.4 29.94 ± 9% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 30.70 ± 7% -16.7 14.02 ± 27% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 21.38 ± 7% -14.2 7.16 ± 30% perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 5.98 ± 28% -5.3 0.63 ± 12% perf-profile.calltrace.cycles-pp.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 6.22 ± 12% -3.9 2.27 ± 31% perf-profile.calltrace.cycles-pp.update_load_avg.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle 5.58 ± 14% -3.9 1.71 ± 28% perf-profile.calltrace.cycles-pp.update_curr.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle 4.50 ± 10% -3.0 1.51 ± 31% perf-profile.calltrace.cycles-pp.hrtimer_active.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle 2.50 ± 14% -2.2 0.31 ±100% perf-profile.calltrace.cycles-pp.trigger_load_balance.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 2.43 ± 8% -2.1 0.35 ±107% perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 1.97 ± 8% -1.4 0.59 ± 54% perf-profile.calltrace.cycles-pp.update_cfs_group.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle 1.78 ± 13% -1.3 0.47 ± 80% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.update_load_avg.task_tick_fair.scheduler_tick.update_process_times 1.77 ± 15% -1.1 0.69 ± 55% perf-profile.calltrace.cycles-pp.__update_load_avg_se.update_load_avg.task_tick_fair.scheduler_tick.update_process_times 2.21 ± 12% -0.9 1.33 ± 14% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 2.21 ± 12% -0.9 1.33 ± 14% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.85 ± 7% -0.8 2.00 ± 10% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt 3.54 ± 6% -0.8 2.70 ± 9% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 1.60 ± 10% -0.6 0.96 ± 12% perf-profile.calltrace.cycles-pp.execve 1.59 ± 10% -0.6 0.96 ± 12% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve 1.59 ± 10% -0.6 0.96 ± 12% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 1.95 ± 8% -0.6 1.31 ± 9% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu 1.59 ± 11% -0.6 0.96 ± 12% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 1.59 ± 10% -0.6 0.96 ± 12% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 1.63 ± 13% -0.6 1.02 ± 16% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start 1.31 ± 10% -0.5 0.79 ± 13% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.81 ± 24% -0.5 0.29 ±100% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 1.19 ± 10% -0.5 0.70 ± 13% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64 1.15 ± 10% -0.5 0.67 ± 13% perf-profile.calltrace.cycles-pp.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve 0.86 ± 9% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 1.13 ± 8% -0.4 0.77 ± 12% perf-profile.calltrace.cycles-pp.asm_exc_page_fault 1.20 ± 5% -0.4 0.85 ± 16% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork 0.74 ± 8% -0.3 0.39 ± 72% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.74 ± 8% -0.3 0.39 ± 72% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.74 ± 8% -0.3 0.39 ± 72% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.08 ± 8% -0.3 0.74 ± 13% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault 1.07 ± 9% -0.3 0.73 ± 13% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 0.91 ± 9% -0.3 0.61 ± 14% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 0.92 ± 11% -0.3 0.62 ± 13% perf-profile.calltrace.cycles-pp.__libc_fork 0.78 ± 16% -0.3 0.51 ± 46% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains 1.11 ± 15% +0.5 1.64 ± 11% perf-profile.calltrace.cycles-pp.read 1.09 ± 15% +0.5 1.62 ± 11% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 1.09 ± 15% +0.5 1.62 ± 11% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.07 ± 15% +0.5 1.60 ± 11% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.08 ± 15% +0.5 1.61 ± 11% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.97 ± 18% +0.6 1.54 ± 12% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.17 ±141% +0.6 0.77 ± 21% perf-profile.calltrace.cycles-pp.__irqentry_text_end 0.59 ± 45% +0.7 1.33 ± 11% perf-profile.calltrace.cycles-pp.seq_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64 0.50 ± 45% +0.8 1.25 ± 12% perf-profile.calltrace.cycles-pp.show_stat.seq_read_iter.new_sync_read.vfs_read.ksys_read 0.77 ± 19% +0.8 1.56 ± 31% perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +3.4 3.37 ± 20% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.00 +3.7 3.67 ± 21% perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 76.25 ± 2% +4.6 80.88 ± 2% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt 0.00 +4.7 4.65 ± 23% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times 0.00 +5.2 5.18 ± 25% perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle 0.00 +5.3 5.29 ± 25% perf-profile.calltrace.cycles-pp.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 3.85 ± 19% +5.4 9.24 ± 26% perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 1.11 ± 11% +7.7 8.77 ± 25% perf-profile.calltrace.cycles-pp.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.00 +8.2 8.16 ± 25% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt 66.60 ± 3% +11.5 78.13 ± 3% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 3.32 ± 8% +12.4 15.75 ± 20% perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 59.52 ± 3% +12.8 72.32 ± 5% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 59.20 ± 3% +13.0 72.23 ± 5% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +14.6 14.64 ± 21% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 6.36 ± 12% +19.2 25.58 ± 6% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +20.6 20.55 ± 6% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt 1.27 ± 25% +21.2 22.46 ± 5% perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 44.12 ± 5% -26.6 17.57 ± 25% perf-profile.children.cycles-pp.tick_sched_handle 43.72 ± 5% -26.3 17.47 ± 25% perf-profile.children.cycles-pp.update_process_times 45.65 ± 6% -23.5 22.13 ± 23% perf-profile.children.cycles-pp.tick_sched_timer 53.67 ± 3% -20.2 33.48 ± 8% perf-profile.children.cycles-pp.__hrtimer_run_queues 34.46 ± 7% -18.8 15.70 ± 25% perf-profile.children.cycles-pp.scheduler_tick 23.76 ± 6% -15.8 7.95 ± 28% perf-profile.children.cycles-pp.task_tick_fair 11.74 ± 42% -7.5 4.26 ± 77% perf-profile.children.cycles-pp.asm_exc_nmi 6.25 ± 21% -5.5 0.72 ± 11% perf-profile.children.cycles-pp.perf_event_task_tick 7.19 ± 11% -4.4 2.77 ± 25% perf-profile.children.cycles-pp.update_load_avg 6.06 ± 9% -4.0 2.11 ± 23% perf-profile.children.cycles-pp.update_curr 4.70 ± 16% -3.8 0.86 ± 44% perf-profile.children.cycles-pp.ghes_notify_nmi 4.97 ± 9% -3.3 1.66 ± 29% perf-profile.children.cycles-pp.hrtimer_active 3.57 ± 21% -3.3 0.28 ± 32% perf-profile.children.cycles-pp.__intel_pmu_enable_all 2.84 ± 14% -2.3 0.58 ± 20% perf-profile.children.cycles-pp.trigger_load_balance 2.79 ± 8% -2.1 0.65 ± 34% perf-profile.children.cycles-pp.rcu_sched_clock_irq 3.22 ± 7% -2.1 1.12 ± 50% perf-profile.children.cycles-pp.native_sched_clock 8.93 ± 7% -1.9 7.03 ± 11% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 8.92 ± 7% -1.9 7.02 ± 11% perf-profile.children.cycles-pp.do_syscall_64 2.35 ± 8% -1.7 0.69 ± 21% perf-profile.children.cycles-pp.irqtime_account_irq 2.19 ± 8% -1.5 0.73 ± 31% perf-profile.children.cycles-pp.update_cfs_group 1.78 ± 6% -1.3 0.44 ± 9% perf-profile.children.cycles-pp._raw_spin_lock 1.59 ± 15% -1.3 0.28 ± 17% perf-profile.children.cycles-pp.update_rq_clock 2.09 ± 12% -1.3 0.82 ± 25% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 1.35 ± 25% -1.3 0.09 ± 73% perf-profile.children.cycles-pp.nmi_handle 4.19 ± 5% -1.2 3.02 ± 9% perf-profile.children.cycles-pp.rebalance_domains 2.08 ± 14% -1.1 0.95 ± 26% perf-profile.children.cycles-pp.__update_load_avg_se 3.36 ± 7% -1.1 2.27 ± 9% perf-profile.children.cycles-pp.load_balance 1.61 ± 8% -1.0 0.56 ± 50% perf-profile.children.cycles-pp.native_flush_tlb_one_user 1.26 ± 23% -1.0 0.24 ± 30% perf-profile.children.cycles-pp.arch_scale_freq_tick 1.36 ± 41% -1.0 0.40 ± 67% perf-profile.children.cycles-pp.paranoid_entry 1.18 ± 10% -0.9 0.25 ± 17% perf-profile.children.cycles-pp.sched_clock_cpu 1.07 ± 10% -0.9 0.19 ± 60% perf-profile.children.cycles-pp.exc_nmi 1.13 ± 34% -0.8 0.29 ± 70% perf-profile.children.cycles-pp.repeat_nmi 2.29 ± 8% -0.8 1.47 ± 10% perf-profile.children.cycles-pp.find_busiest_group 2.16 ± 9% -0.8 1.37 ± 9% perf-profile.children.cycles-pp.update_sd_lb_stats 0.92 ± 24% -0.7 0.18 ± 30% perf-profile.children.cycles-pp.sync_regs 1.09 ± 10% -0.7 0.35 ± 7% perf-profile.children.cycles-pp.hrtimer_update_next_event 1.70 ± 11% -0.7 0.97 ± 12% perf-profile.children.cycles-pp.__x64_sys_execve 1.70 ± 11% -0.7 0.97 ± 12% perf-profile.children.cycles-pp.do_execveat_common 0.93 ± 7% -0.7 0.26 ± 8% perf-profile.children.cycles-pp.__remove_hrtimer 1.60 ± 10% -0.6 0.96 ± 12% perf-profile.children.cycles-pp.execve 0.77 ± 15% -0.6 0.15 ± 30% perf-profile.children.cycles-pp.update_irq_load_avg 0.93 ± 11% -0.6 0.32 ± 31% perf-profile.children.cycles-pp.acpi_os_read_memory 1.40 ± 10% -0.6 0.80 ± 13% perf-profile.children.cycles-pp.bprm_execve 2.13 ± 4% -0.6 1.57 ± 10% perf-profile.children.cycles-pp.asm_exc_page_fault 1.27 ± 10% -0.6 0.71 ± 13% perf-profile.children.cycles-pp.exec_binprm 2.00 ± 4% -0.5 1.46 ± 10% perf-profile.children.cycles-pp.exc_page_fault 1.22 ± 10% -0.5 0.68 ± 13% perf-profile.children.cycles-pp.load_elf_binary 2.00 ± 4% -0.5 1.45 ± 9% perf-profile.children.cycles-pp.do_user_addr_fault 1.72 ± 4% -0.5 1.20 ± 10% perf-profile.children.cycles-pp.handle_mm_fault 0.87 ± 12% -0.5 0.36 ± 31% perf-profile.children.cycles-pp.reweight_entity 1.62 ± 4% -0.5 1.12 ± 10% perf-profile.children.cycles-pp.__handle_mm_fault 0.69 ± 29% -0.5 0.19 ± 10% perf-profile.children.cycles-pp.native_irq_return_iret 0.79 ± 8% -0.5 0.29 ± 10% perf-profile.children.cycles-pp.enqueue_hrtimer 0.74 ± 13% -0.5 0.25 ± 31% perf-profile.children.cycles-pp.update_min_vruntime 0.68 ± 8% -0.5 0.19 ± 6% perf-profile.children.cycles-pp.timerqueue_del 0.83 ± 7% -0.5 0.38 ± 16% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.71 ± 43% -0.4 0.27 ± 76% perf-profile.children.cycles-pp.first_nmi 1.17 ± 9% -0.4 0.73 ± 14% perf-profile.children.cycles-pp.exit_mmap 1.17 ± 9% -0.4 0.74 ± 15% perf-profile.children.cycles-pp.mmput 0.68 ± 9% -0.4 0.24 ± 9% perf-profile.children.cycles-pp.timerqueue_add 0.55 ± 17% -0.4 0.14 ± 28% perf-profile.children.cycles-pp.account_user_time 1.05 ± 4% -0.4 0.67 ± 19% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.83 ± 6% -0.4 0.48 ± 11% perf-profile.children.cycles-pp.do_filp_open 0.56 ± 11% -0.4 0.20 ± 8% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.82 ± 6% -0.3 0.47 ± 10% perf-profile.children.cycles-pp.path_openat 1.20 ± 4% -0.3 0.85 ± 16% perf-profile.children.cycles-pp.smpboot_thread_fn 0.84 ± 5% -0.3 0.50 ± 11% perf-profile.children.cycles-pp.do_sys_open 0.83 ± 5% -0.3 0.49 ± 11% perf-profile.children.cycles-pp.do_sys_openat2 0.49 ± 12% -0.3 0.18 ± 8% perf-profile.children.cycles-pp.__hrtimer_get_next_event 0.92 ± 10% -0.3 0.63 ± 14% perf-profile.children.cycles-pp.__libc_fork 1.08 ± 7% -0.3 0.78 ± 12% perf-profile.children.cycles-pp.finish_task_switch 0.65 ± 12% -0.3 0.35 ± 12% perf-profile.children.cycles-pp.begin_new_exec 0.81 ± 23% -0.3 0.52 ± 12% perf-profile.children.cycles-pp.worker_thread 0.37 ± 9% -0.3 0.09 ± 49% perf-profile.children.cycles-pp.default_do_nmi 0.59 ± 14% -0.3 0.31 ± 12% perf-profile.children.cycles-pp.vm_mmap_pgoff 0.56 ± 15% -0.3 0.29 ± 12% perf-profile.children.cycles-pp.do_mmap 0.68 ± 11% -0.3 0.43 ± 16% perf-profile.children.cycles-pp.copy_process 0.33 ± 15% -0.3 0.08 ± 38% perf-profile.children.cycles-pp.ghes_copy_tofrom_phys 0.71 ± 26% -0.3 0.46 ± 13% perf-profile.children.cycles-pp.process_one_work 0.71 ± 11% -0.2 0.46 ± 15% perf-profile.children.cycles-pp.__do_sys_clone 0.71 ± 11% -0.2 0.46 ± 15% perf-profile.children.cycles-pp.kernel_clone 0.83 ± 5% -0.2 0.59 ± 10% perf-profile.children.cycles-pp.do_fault 0.64 ± 10% -0.2 0.40 ± 20% perf-profile.children.cycles-pp.try_to_wake_up 0.28 ± 8% -0.2 0.03 ±100% perf-profile.children.cycles-pp.rb_erase 0.50 ± 15% -0.2 0.27 ± 13% perf-profile.children.cycles-pp.mmap_region 0.30 ± 10% -0.2 0.07 ± 23% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.58 ± 9% -0.2 0.36 ± 15% perf-profile.children.cycles-pp.unmap_vmas 0.30 ± 8% -0.2 0.08 ± 16% perf-profile.children.cycles-pp.tick_program_event 0.33 ± 6% -0.2 0.12 ± 16% perf-profile.children.cycles-pp.hrtimer_forward 0.25 ± 20% -0.2 0.03 ±100% perf-profile.children.cycles-pp.account_process_tick 0.46 ± 15% -0.2 0.24 ± 7% perf-profile.children.cycles-pp.ksys_mmap_pgoff 0.46 ± 8% -0.2 0.25 ± 12% perf-profile.children.cycles-pp.kmem_cache_alloc 0.75 ± 8% -0.2 0.55 ± 16% perf-profile.children.cycles-pp.__x64_sys_exit_group 0.75 ± 8% -0.2 0.55 ± 16% perf-profile.children.cycles-pp.do_group_exit 0.75 ± 8% -0.2 0.55 ± 16% perf-profile.children.cycles-pp.do_exit 0.33 ± 16% -0.2 0.14 ± 66% perf-profile.children.cycles-pp.intel_pmu_handle_irq 0.28 ± 7% -0.2 0.09 ± 51% perf-profile.children.cycles-pp.__native_set_fixmap 0.49 ± 9% -0.2 0.29 ± 8% perf-profile.children.cycles-pp.__alloc_pages 0.47 ± 10% -0.2 0.28 ± 20% perf-profile.children.cycles-pp.dup_mm 0.42 ± 9% -0.2 0.23 ± 20% perf-profile.children.cycles-pp.walk_component 0.51 ± 10% -0.2 0.33 ± 15% perf-profile.children.cycles-pp.unmap_page_range 0.27 ± 12% -0.2 0.09 ± 54% perf-profile.children.cycles-pp.calc_global_load_tick 0.44 ± 11% -0.2 0.26 ± 20% perf-profile.children.cycles-pp.dup_mmap 0.41 ± 9% -0.2 0.24 ± 20% perf-profile.children.cycles-pp.link_path_walk 0.23 ± 38% -0.2 0.06 ±118% perf-profile.children.cycles-pp.intel_bts_disable_local 0.61 ± 6% -0.2 0.45 ± 16% perf-profile.children.cycles-pp.run_ksoftirqd 0.47 ± 9% -0.2 0.30 ± 15% perf-profile.children.cycles-pp.zap_pte_range 0.33 ± 14% -0.2 0.16 ± 22% perf-profile.children.cycles-pp.__calc_delta 0.22 ± 36% -0.2 0.06 ±119% perf-profile.children.cycles-pp.__intel_pmu_disable_all 0.55 ± 6% -0.2 0.39 ± 13% perf-profile.children.cycles-pp.filemap_map_pages 0.31 ± 12% -0.2 0.15 ± 28% perf-profile.children.cycles-pp.__accumulate_pelt_segments 0.40 ± 9% -0.1 0.26 ± 12% perf-profile.children.cycles-pp.setlocale 0.20 ± 16% -0.1 0.06 ± 46% perf-profile.children.cycles-pp.sched_slice 0.17 ± 7% -0.1 0.04 ± 75% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime 0.33 ± 7% -0.1 0.20 ± 11% perf-profile.children.cycles-pp.get_page_from_freelist 0.16 ± 16% -0.1 0.04 ±104% perf-profile.children.cycles-pp.native_set_fixmap 0.33 ± 14% -0.1 0.20 ± 15% perf-profile.children.cycles-pp.free_pgtables 0.16 ± 21% -0.1 0.04 ± 72% perf-profile.children.cycles-pp.schedule_idle 0.19 ± 38% -0.1 0.07 ± 18% perf-profile.children.cycles-pp.__poll 0.32 ± 16% -0.1 0.20 ± 15% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 0.30 ± 15% -0.1 0.19 ± 19% perf-profile.children.cycles-pp.call_timer_fn 0.19 ± 9% -0.1 0.08 ± 6% perf-profile.children.cycles-pp.rb_insert_color 0.29 ± 9% -0.1 0.18 ± 21% perf-profile.children.cycles-pp.alloc_pages_vma 0.22 ± 6% -0.1 0.11 ± 27% perf-profile.children.cycles-pp.filename_lookup 0.24 ± 19% -0.1 0.13 ± 10% perf-profile.children.cycles-pp.__do_munmap 0.27 ± 10% -0.1 0.17 ± 11% perf-profile.children.cycles-pp.__open64_nocancel 0.14 ± 34% -0.1 0.03 ±108% perf-profile.children.cycles-pp.__evlist__enable 0.21 ± 7% -0.1 0.11 ± 24% perf-profile.children.cycles-pp.path_lookupat 0.30 ± 4% -0.1 0.21 ± 12% perf-profile.children.cycles-pp.next_uptodate_page 0.18 ± 7% -0.1 0.08 ± 24% perf-profile.children.cycles-pp.__do_sys_newstat 0.19 ± 4% -0.1 0.10 ± 16% perf-profile.children.cycles-pp.step_into 0.31 ± 4% -0.1 0.21 ± 19% perf-profile.children.cycles-pp.wp_page_copy 0.17 ± 6% -0.1 0.08 ± 24% perf-profile.children.cycles-pp.vfs_statx 0.19 ± 12% -0.1 0.10 ± 18% perf-profile.children.cycles-pp.elf_map 0.14 ± 34% -0.1 0.06 ± 46% perf-profile.children.cycles-pp.do_sys_poll 0.17 ± 8% -0.1 0.09 ± 12% perf-profile.children.cycles-pp.dput 0.20 ± 15% -0.1 0.12 ± 11% perf-profile.children.cycles-pp.__split_vma 0.14 ± 34% -0.1 0.06 ± 47% perf-profile.children.cycles-pp.__x64_sys_poll 0.15 ± 13% -0.1 0.07 ± 19% perf-profile.children.cycles-pp.vma_interval_tree_insert 0.17 ± 20% -0.1 0.09 ± 31% perf-profile.children.cycles-pp.enqueue_task_fair 0.17 ± 23% -0.1 0.09 ± 31% perf-profile.children.cycles-pp.ttwu_do_activate 0.17 ± 30% -0.1 0.09 ± 15% perf-profile.children.cycles-pp.__pagevec_lru_add 0.15 ± 20% -0.1 0.07 ± 21% perf-profile.children.cycles-pp.vma_link 0.21 ± 15% -0.1 0.13 ± 14% perf-profile.children.cycles-pp.do_anonymous_page 0.18 ± 23% -0.1 0.10 ± 25% perf-profile.children.cycles-pp.proc_reg_read_iter 0.17 ± 13% -0.1 0.09 ± 15% perf-profile.children.cycles-pp.__alloc_file 0.17 ± 11% -0.1 0.10 ± 15% perf-profile.children.cycles-pp.alloc_empty_file 0.20 ± 12% -0.1 0.13 ± 16% perf-profile.children.cycles-pp.tlb_finish_mmu 0.19 ± 12% -0.1 0.12 ± 17% perf-profile.children.cycles-pp.tlb_flush_mmu 0.17 ± 14% -0.1 0.09 ± 15% perf-profile.children.cycles-pp.__mmap 0.26 ± 8% -0.1 0.19 ± 18% perf-profile.children.cycles-pp.write 0.32 ± 8% -0.1 0.24 ± 14% perf-profile.children.cycles-pp.kmem_cache_free 0.20 ± 22% -0.1 0.13 ± 12% perf-profile.children.cycles-pp.release_pages 0.16 ± 17% -0.1 0.09 ± 16% perf-profile.children.cycles-pp.copy_strings 0.17 ± 14% -0.1 0.09 ± 14% perf-profile.children.cycles-pp.__vma_adjust 0.31 ± 10% -0.1 0.24 ± 16% perf-profile.children.cycles-pp.cpumask_next_and 0.16 ± 15% -0.1 0.09 ± 22% perf-profile.children.cycles-pp.do_mprotect_pkey 0.10 ± 9% -0.1 0.03 ±101% perf-profile.children.cycles-pp.get_obj_cgroup_from_current 0.09 ± 14% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__xstat64 0.17 ± 13% -0.1 0.10 ± 21% perf-profile.children.cycles-pp.__x64_sys_mprotect 0.13 ± 10% -0.1 0.07 ± 18% perf-profile.children.cycles-pp.rmqueue 0.14 ± 11% -0.1 0.08 ± 16% perf-profile.children.cycles-pp.do_open 0.16 ± 6% -0.1 0.10 ± 23% perf-profile.children.cycles-pp.__lookup_slow 0.16 ± 7% -0.1 0.10 ± 11% perf-profile.children.cycles-pp.prep_new_page 0.15 ± 13% -0.1 0.09 ± 15% perf-profile.children.cycles-pp.unlink_anon_vmas 0.14 ± 12% -0.1 0.09 ± 14% perf-profile.children.cycles-pp.lookup_fast 0.15 ± 15% -0.1 0.09 ± 21% perf-profile.children.cycles-pp.mprotect_fixup 0.11 ± 19% -0.1 0.06 ± 50% perf-profile.children.cycles-pp.__get_user_pages_remote 0.10 ± 11% -0.1 0.04 ± 75% perf-profile.children.cycles-pp.anon_vma_fork 0.09 ± 9% -0.1 0.03 ± 70% perf-profile.children.cycles-pp.__get_free_pages 0.08 ± 31% -0.1 0.03 ± 99% perf-profile.children.cycles-pp.strnlen_user 0.08 ± 12% -0.1 0.03 ± 99% perf-profile.children.cycles-pp.page_counter_try_charge 0.14 ± 21% -0.1 0.08 ± 21% perf-profile.children.cycles-pp.unlink_file_vma 0.11 ± 17% -0.1 0.06 ± 50% perf-profile.children.cycles-pp.__get_user_pages 0.17 ± 7% -0.1 0.11 ± 12% perf-profile.children.cycles-pp.___might_sleep 0.10 ± 15% -0.1 0.05 ± 46% perf-profile.children.cycles-pp.remove_vma 0.09 ± 18% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.mutex_lock 0.18 ± 11% -0.1 0.12 ± 15% perf-profile.children.cycles-pp.page_remove_rmap 0.17 ± 10% -0.1 0.12 ± 13% perf-profile.children.cycles-pp.select_task_rq_fair 0.08 ± 14% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__check_object_size 0.10 ± 22% -0.0 0.05 ± 46% perf-profile.children.cycles-pp.unmap_region 0.09 ± 5% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.getname_flags 0.13 ± 5% -0.0 0.08 ± 20% perf-profile.children.cycles-pp.d_alloc_parallel 0.10 ± 14% -0.0 0.06 ± 19% perf-profile.children.cycles-pp.do_open_execat 0.09 ± 8% -0.0 0.04 ± 73% perf-profile.children.cycles-pp.__pte_alloc 0.15 ± 12% -0.0 0.10 ± 19% perf-profile.children.cycles-pp.vmstat_update 0.10 ± 10% -0.0 0.05 ± 46% perf-profile.children.cycles-pp.do_dentry_open 0.13 ± 14% -0.0 0.08 ± 17% perf-profile.children.cycles-pp.select_idle_sibling 0.12 ± 22% -0.0 0.08 ± 21% perf-profile.children.cycles-pp.__cgroup_account_cputime_field 0.08 ± 8% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.__dentry_kill 0.07 ± 12% -0.0 0.03 ±100% perf-profile.children.cycles-pp.__memcg_kmem_charge_page 0.10 ± 17% -0.0 0.05 ± 48% perf-profile.children.cycles-pp.lru_add_drain 0.11 ± 14% -0.0 0.07 ± 14% perf-profile.children.cycles-pp.open64 0.08 ± 14% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state 0.13 ± 8% -0.0 0.09 ± 10% perf-profile.children.cycles-pp.clear_page_erms 0.11 ± 13% -0.0 0.07 ± 21% perf-profile.children.cycles-pp.get_arg_page 0.13 ± 8% -0.0 0.08 ± 24% perf-profile.children.cycles-pp.__wake_up_common_lock 0.08 ± 13% -0.0 0.04 ± 72% perf-profile.children.cycles-pp.lru_add_drain_cpu 0.12 ± 12% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.__fput 0.11 ± 15% -0.0 0.07 ± 16% perf-profile.children.cycles-pp.__clear_user 0.07 ± 10% -0.0 0.03 ±100% perf-profile.children.cycles-pp.__d_alloc 0.09 ± 12% -0.0 0.05 ± 47% perf-profile.children.cycles-pp.obj_cgroup_charge_pages 0.13 ± 11% -0.0 0.09 ± 20% perf-profile.children.cycles-pp._dl_addr 0.10 ± 7% -0.0 0.06 ± 17% perf-profile.children.cycles-pp.__mod_lruvec_page_state 0.13 ± 16% -0.0 0.09 ± 22% perf-profile.children.cycles-pp.perf_event_mmap 0.09 ± 6% -0.0 0.05 ± 50% perf-profile.children.cycles-pp.___perf_sw_event 0.10 ± 9% -0.0 0.07 ± 14% perf-profile.children.cycles-pp.pte_alloc_one 0.09 ± 27% -0.0 0.05 ± 50% perf-profile.children.cycles-pp.vm_area_dup 0.09 ± 20% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.down_write 0.09 ± 14% -0.0 0.06 ± 18% perf-profile.children.cycles-pp.malloc 0.09 ± 10% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.sum_zone_numa_state 0.10 ± 13% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.sysfs_kf_seq_show 0.10 ± 15% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.dev_attr_show 0.08 ± 10% -0.0 0.06 ± 8% perf-profile.children.cycles-pp.__might_sleep 0.09 ± 11% -0.0 0.06 ± 17% perf-profile.children.cycles-pp.__perf_sw_event 0.07 ± 45% +0.0 0.11 ± 13% perf-profile.children.cycles-pp.seq_put_decimal_ull_width 0.04 ± 71% +0.1 0.09 ± 25% perf-profile.children.cycles-pp.ksoftirqd_running 0.00 +0.1 0.06 ± 13% perf-profile.children.cycles-pp.num_to_str 0.48 ± 8% +0.3 0.77 ± 21% perf-profile.children.cycles-pp.__irqentry_text_end 0.07 ± 14% +0.3 0.39 ± 10% perf-profile.children.cycles-pp.get_cpu_idle_time_us 0.07 ± 14% +0.3 0.39 ± 11% perf-profile.children.cycles-pp.get_idle_time 0.00 +0.4 0.42 ± 11% perf-profile.children.cycles-pp.get_cpu_iowait_time_us 0.00 +0.4 0.42 ± 11% perf-profile.children.cycles-pp.get_iowait_time 0.00 +0.5 0.47 ± 21% perf-profile.children.cycles-pp.sched_clock_local 1.39 ± 17% +0.5 1.87 ± 11% perf-profile.children.cycles-pp.ksys_read 1.33 ± 16% +0.5 1.82 ± 11% perf-profile.children.cycles-pp.vfs_read 1.11 ± 15% +0.5 1.64 ± 11% perf-profile.children.cycles-pp.read 1.03 ± 18% +0.5 1.57 ± 12% perf-profile.children.cycles-pp.new_sync_read 0.93 ± 17% +0.6 1.49 ± 11% perf-profile.children.cycles-pp.seq_read_iter 0.55 ± 23% +0.7 1.26 ± 11% perf-profile.children.cycles-pp.show_stat 1.00 ± 16% +0.9 1.86 ± 29% perf-profile.children.cycles-pp.rcu_core 4.49 ± 21% +5.7 10.15 ± 26% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.00 +6.0 5.98 ± 22% perf-profile.children.cycles-pp.sched_clock_tick 85.42 ± 2% +6.4 91.85 ± 3% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 75.45 ± 2% +11.3 86.74 ± 2% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 67.36 ± 2% +13.1 80.42 ± 4% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 67.01 ± 2% +13.3 80.31 ± 4% perf-profile.children.cycles-pp.hrtimer_interrupt 3.66 ± 8% +13.8 17.51 ± 19% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 6.67 ± 10% +21.5 28.15 ± 6% perf-profile.children.cycles-pp.clockevents_program_event 3.16 ± 11% +42.0 45.11 ± 3% perf-profile.children.cycles-pp.ktime_get 0.00 +57.3 57.25 ± 5% perf-profile.children.cycles-pp.read_hpet 11.64 ± 42% -7.4 4.24 ± 77% perf-profile.self.cycles-pp.asm_exc_nmi 4.70 ± 16% -3.8 0.86 ± 44% perf-profile.self.cycles-pp.ghes_notify_nmi 3.68 ± 14% -3.1 0.57 ± 8% perf-profile.self.cycles-pp.perf_event_task_tick 2.82 ± 22% -2.6 0.25 ± 27% perf-profile.self.cycles-pp.__intel_pmu_enable_all 4.08 ± 16% -2.5 1.55 ± 30% perf-profile.self.cycles-pp.hrtimer_active 3.57 ± 9% -2.2 1.41 ± 24% perf-profile.self.cycles-pp.update_curr 3.10 ± 8% -1.9 1.16 ± 19% perf-profile.self.cycles-pp.ktime_get_update_offsets_now 2.93 ± 5% -1.8 1.09 ± 49% perf-profile.self.cycles-pp.native_sched_clock 2.35 ± 13% -1.6 0.71 ± 30% perf-profile.self.cycles-pp.task_tick_fair 2.12 ± 18% -1.6 0.52 ± 21% perf-profile.self.cycles-pp.trigger_load_balance 2.16 ± 11% -1.6 0.59 ± 33% perf-profile.self.cycles-pp.rcu_sched_clock_irq 1.71 ± 21% -1.5 0.21 ± 22% perf-profile.self.cycles-pp.update_process_times 2.31 ± 10% -1.3 0.98 ± 23% perf-profile.self.cycles-pp.update_load_avg 1.35 ± 25% -1.3 0.09 ± 73% perf-profile.self.cycles-pp.nmi_handle 1.75 ± 15% -1.1 0.68 ± 33% perf-profile.self.cycles-pp.update_cfs_group 1.61 ± 8% -1.0 0.56 ± 50% perf-profile.self.cycles-pp.native_flush_tlb_one_user 1.36 ± 41% -1.0 0.40 ± 67% perf-profile.self.cycles-pp.paranoid_entry 1.05 ± 10% -0.9 0.19 ± 60% perf-profile.self.cycles-pp.exc_nmi 1.18 ± 8% -0.8 0.34 ± 11% perf-profile.self.cycles-pp._raw_spin_lock 1.13 ± 34% -0.8 0.29 ± 70% perf-profile.self.cycles-pp.repeat_nmi 1.25 ± 14% -0.8 0.49 ± 16% perf-profile.self.cycles-pp.irqtime_account_irq 0.89 ± 25% -0.7 0.18 ± 29% perf-profile.self.cycles-pp.sync_regs 1.38 ± 14% -0.7 0.68 ± 29% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.83 ± 20% -0.6 0.21 ± 31% perf-profile.self.cycles-pp.arch_scale_freq_tick 0.93 ± 11% -0.6 0.32 ± 31% perf-profile.self.cycles-pp.acpi_os_read_memory 1.34 ± 15% -0.6 0.77 ± 29% perf-profile.self.cycles-pp.__update_load_avg_se 0.68 ± 29% -0.5 0.15 ± 10% perf-profile.self.cycles-pp.native_irq_return_iret 0.71 ± 43% -0.4 0.27 ± 76% perf-profile.self.cycles-pp.first_nmi 0.58 ± 8% -0.4 0.15 ± 21% perf-profile.self.cycles-pp.scheduler_tick 0.62 ± 15% -0.4 0.23 ± 10% perf-profile.self.cycles-pp.__hrtimer_run_queues 0.51 ± 16% -0.4 0.12 ± 12% perf-profile.self.cycles-pp.update_irq_load_avg 0.56 ± 13% -0.3 0.22 ± 32% perf-profile.self.cycles-pp.update_min_vruntime 0.68 ± 21% -0.3 0.37 ± 8% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.37 ± 9% -0.3 0.09 ± 49% perf-profile.self.cycles-pp.default_do_nmi 0.89 ± 8% -0.3 0.61 ± 8% perf-profile.self.cycles-pp.update_sd_lb_stats 0.36 ± 15% -0.3 0.10 ± 21% perf-profile.self.cycles-pp.tick_sched_timer 0.33 ± 15% -0.3 0.08 ± 38% perf-profile.self.cycles-pp.ghes_copy_tofrom_phys 0.53 ± 11% -0.2 0.28 ± 34% perf-profile.self.cycles-pp.reweight_entity 0.54 ± 9% -0.2 0.32 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.26 ± 18% -0.2 0.05 ± 76% perf-profile.self.cycles-pp.account_user_time 0.33 ± 15% -0.2 0.12 ± 21% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.33 ± 16% -0.2 0.14 ± 66% perf-profile.self.cycles-pp.intel_pmu_handle_irq 0.28 ± 7% -0.2 0.09 ± 51% perf-profile.self.cycles-pp.__native_set_fixmap 0.36 ± 13% -0.2 0.17 ± 7% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.33 ± 9% -0.2 0.15 ± 4% perf-profile.self.cycles-pp.timerqueue_add 0.23 ± 38% -0.2 0.06 ±118% perf-profile.self.cycles-pp.intel_bts_disable_local 0.26 ± 23% -0.2 0.09 ± 11% perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt 0.19 ± 21% -0.2 0.03 ±101% perf-profile.self.cycles-pp.account_process_tick 0.21 ± 35% -0.2 0.06 ±119% perf-profile.self.cycles-pp.__intel_pmu_disable_all 0.22 ± 12% -0.1 0.09 ± 55% perf-profile.self.cycles-pp.calc_global_load_tick 0.23 ± 10% -0.1 0.10 ± 7% perf-profile.self.cycles-pp.hrtimer_forward 0.16 ± 16% -0.1 0.04 ±104% perf-profile.self.cycles-pp.native_set_fixmap 0.19 ± 14% -0.1 0.07 ± 21% perf-profile.self.cycles-pp.tick_program_event 0.18 ± 8% -0.1 0.07 ± 25% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.18 ± 13% -0.1 0.07 ± 15% perf-profile.self.cycles-pp.__remove_hrtimer 0.20 ± 15% -0.1 0.09 ± 13% perf-profile.self.cycles-pp.__hrtimer_get_next_event 0.26 ± 13% -0.1 0.16 ± 22% perf-profile.self.cycles-pp.__calc_delta 0.13 ± 15% -0.1 0.03 ±100% perf-profile.self.cycles-pp.sched_slice 0.21 ± 13% -0.1 0.12 ± 5% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.12 ± 16% -0.1 0.04 ± 75% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime 0.17 ± 9% -0.1 0.10 ± 11% perf-profile.self.cycles-pp.timerqueue_del 0.19 ± 14% -0.1 0.13 ± 32% perf-profile.self.cycles-pp.__accumulate_pelt_segments 0.12 ± 15% -0.1 0.07 ± 11% perf-profile.self.cycles-pp.rb_insert_color 0.09 ± 11% -0.0 0.05 ± 47% perf-profile.self.cycles-pp.kmem_cache_alloc 0.04 ± 71% +0.0 0.07 ± 10% perf-profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup 0.03 ± 70% +0.0 0.07 ± 26% perf-profile.self.cycles-pp.update_group_capacity 0.00 +0.1 0.07 ± 18% perf-profile.self.cycles-pp.ksoftirqd_running 0.14 ± 13% +0.1 0.22 ± 10% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.05 ± 45% +0.1 0.14 ± 9% perf-profile.self.cycles-pp.run_rebalance_domains 0.17 ± 19% +0.1 0.29 ± 10% perf-profile.self.cycles-pp.irq_exit_rcu 0.02 ±141% +0.2 0.17 ± 32% perf-profile.self.cycles-pp.rcu_core 0.24 ± 20% +0.2 0.45 ± 14% perf-profile.self.cycles-pp.idle_cpu 0.00 +0.2 0.25 ± 21% perf-profile.self.cycles-pp.sched_clock_local 0.32 ± 7% +0.3 0.63 ± 10% perf-profile.self.cycles-pp.__irqentry_text_end 0.14 ± 16% +0.3 0.49 ± 14% perf-profile.self.cycles-pp.__softirqentry_text_start 2.15 ± 12% +1.5 3.64 ± 5% perf-profile.self.cycles-pp.ktime_get 0.00 +54.2 54.23 ± 8% perf-profile.self.cycles-pp.read_hpet stress-ng.time.user_time 5200 +--------------------------------------------------------------------+ | OO O O O O O O O | 5000 |-+OO OO OO O O O O O O OO O OO O O O O OO O | 4800 |O+ O O O O O O | | | 4600 |-+ | 4400 |-+ | | | 4200 |-+ + + + + + + +. + | 4000 |-+ +. + ++ :: :+ :: : + + :.+ : : + : | | + +. : + : : : :: : : :: : :+ ++ + : : : : : : : | 3800 |:.+ + :.+ + .+ : : :: : + :: : + :: + :.+ :: :.+ ::| 3600 |++ + + + +.+ + + : + : + : + :| | + + + +| 3400 +--------------------------------------------------------------------+ stress-ng.time.percent_of_cpu_this_job_got 9000 +--------------------------------------------------------------------+ | | 8500 |-+ OO O O OOO OO O O O OO O O OO O | | OO OO O O OO O O OO O O O OO O | |O O O | 8000 |-+ | | | 7500 |-+ | | | 7000 |-+ + + + + + + + | | +. + :: :: :: : .+ : :+ : | | + +. : + : ++ :: : + :: : :.+++ +.+ : : : + : : | 6500 |:.+ + :.+ + + : ::+ + + :: + + :: + :.+ :: :.+ ::| |+ + + :+ +.: + + : + : + : + :| 6000 +--------------------------------------------------------------------+ stress-ng.lockbus.ops 250000 +------------------------------------------------------------------+ | ++ +.++.+++.++.+ +++ + ++.+ ++ ++ + +.++ + ++.+| 245000 |-+ | | | | | 240000 |-+ | | | 235000 |-+ | | O | 230000 |-+ O O O O O | |O O O O O O O O O | | O O O O O | 225000 |-+O OO O O O OO O O O O OO | | O O O O O O | 220000 +------------------------------------------------------------------+ stress-ng.lockbus.ops_per_sec 4150 +--------------------------------------------------------------------+ 4100 |-+ +.+ ++ ++ + +.+ +.+ +.+ | | | 4050 |-+ | 4000 |-+ | | | 3950 |-+ | 3900 |-+ | 3850 |-+ O O | |O O O O O O | 3800 |-+ O O O O O O O | 3750 |-+ O OO O O O | | O OO O O O O O OO O O | 3700 |-+ OO O O O O O | 3650 +--------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang