Greeting, FYI, we noticed a -8.5% regression of stress-ng.lockbus.ops_per_sec due to commit: commit: 8c4b798316c77d18b351a36e3660e412de762d79 ("clocksource: Retry clock read if long delays detected") https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master in testcase: stress-ng on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory with following parameters: nr_threads: 100% disk: 1HDD testtime: 60s class: cpu-cache test: lockbus cpufreq_governor: performance ucode: 0x5003006 If you fix the issue, kindly add following tag Reported-by: kernel test robot Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run bin/lkp run generated-yaml-file ========================================================================================= class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode: cpu-cache/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/lockbus/stress-ng/60s/0x5003006 commit: 5005eca245 ("clocksource: Provide module parameters to inject delays in watchdog") 8c4b798316 ("clocksource: Retry clock read if long delays detected") 5005eca245d49ac4 8c4b798316c77d18b351a36e366 ---------------- --------------------------- %stddev %change %stddev \ | \ 4115 -8.5% 3766 stress-ng.lockbus.ops_per_sec 6784 ± 2% +23.0% 8344 stress-ng.time.percent_of_cpu_this_job_got 208.45 ± 5% +67.4% 348.85 ± 11% stress-ng.time.system_time 4019 ± 2% +22.2% 4910 stress-ng.time.user_time 42341 ± 48% +330.6% 182325 ± 37% cpuidle.POLL.time 618938 ± 3% +10.7% 685156 ± 2% numa-meminfo.node1.Mapped 64.00 ± 2% +20.6% 77.17 vmstat.cpu.us 26.34 ± 7% -54.6% 11.96 ± 7% iostat.cpu.system 65.83 ± 3% +21.4% 79.90 iostat.cpu.user 13.77 ± 12% -12.9 0.82 ± 38% mpstat.cpu.all.irq% 59.54 ± 4% +13.9 73.47 mpstat.cpu.all.usr% 142187 ± 5% +13.1% 160881 ± 4% numa-vmstat.node1.nr_mapped 141628 ± 4% +14.7% 162466 ± 4% numa-vmstat.node1.nr_shmem 8265 ± 58% -86.7% 1097 ± 37% softirqs.NET_RX 175377 ± 8% +15.2% 201967 ± 5% softirqs.RCU 51992 +2.4% 53264 interrupts.CAL:Function_call_interrupts 56247 ± 6% +32.5% 74499 ± 10% interrupts.CPU0.LOC:Local_timer_interrupts 92.33 ± 20% +50.4% 138.83 ± 7% interrupts.CPU0.RES:Rescheduling_interrupts 5377 ± 28% -42.3% 3100 ± 30% interrupts.CPU10.NMI:Non-maskable_interrupts 5377 ± 28% -42.3% 3100 ± 30% interrupts.CPU10.PMI:Performance_monitoring_interrupts 291.50 ± 10% +29.4% 377.33 ± 5% interrupts.CPU47.RES:Rescheduling_interrupts 1967 ± 3% +20.3% 2366 interrupts.RES:Rescheduling_interrupts 842.17 ± 66% +446.0% 4597 ± 65% proc-vmstat.nr_active_anon 2978 +9.4% 3257 proc-vmstat.nr_page_table_pages 20810 +1.7% 21165 proc-vmstat.nr_slab_reclaimable 842.17 ± 66% +446.0% 4597 ± 65% proc-vmstat.nr_zone_active_anon 131948 ±140% +136.1% 311596 ± 38% proc-vmstat.pgactivate 518116 ± 2% +4.6% 541978 proc-vmstat.pgfault 4733 ± 6% +50.7% 7131 ± 5% proc-vmstat.pgreuse 0.00 ± 55% +587.5% 0.01 ± 25% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.00 ± 47% +295.8% 0.02 ± 9% perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read 0.01 ± 19% +310.8% 0.03 ± 56% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 0.00 ± 46% +540.0% 0.03 ± 36% perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.01 ± 28% +694.1% 0.05 ± 75% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown] 0.01 ± 11% +541.0% 0.04 ± 70% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 2.80 ± 14% +65.7% 4.63 ± 12% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork 2.72 ± 48% -90.9% 0.25 ±223% perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 4.33 ± 68% -92.3% 0.33 ±223% perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64 6.73 ± 64% -93.1% 0.46 ±223% perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 2516450 ± 2% +19.5% 3007360 sched_debug.cfs_rq:/.min_vruntime.avg 2786568 ± 3% +14.7% 3195433 sched_debug.cfs_rq:/.min_vruntime.max 1341933 ± 6% +25.8% 1687931 ± 5% sched_debug.cfs_rq:/.min_vruntime.min 1.13 ± 97% -99.1% 0.01 ±223% sched_debug.cfs_rq:/.spread.avg 108.61 ± 97% -99.1% 1.00 ±223% sched_debug.cfs_rq:/.spread.max 11.03 ± 97% -99.1% 0.10 ±223% sched_debug.cfs_rq:/.spread.stddev 1108101 ± 13% -84.4% 172708 ± 41% sched_debug.cfs_rq:/.spread0.avg 1377563 ± 11% -74.7% 348242 ± 17% sched_debug.cfs_rq:/.spread0.max -65171 +1722.9% -1188034 sched_debug.cfs_rq:/.spread0.min 3820707 ± 9% -36.2% 2436693 ± 14% sched_debug.cpu.avg_idle.avg 13038027 ± 12% -40.3% 7789745 ± 14% sched_debug.cpu.avg_idle.max 2506387 ± 12% -35.0% 1628581 ± 21% sched_debug.cpu.avg_idle.stddev 20.52 ± 36% +998.3% 225.34 ± 13% sched_debug.cpu.clock.stddev 42166 ± 3% +29.9% 54759 sched_debug.cpu.clock_task.min 1867 ± 11% -60.3% 741.00 ± 3% sched_debug.cpu.clock_task.stddev 3607599 ± 3% -57.6% 1531356 ± 15% sched_debug.cpu.max_idle_balance_cost.avg 9226194 ± 11% -47.4% 4857518 ± 11% sched_debug.cpu.max_idle_balance_cost.max 1892220 ± 8% -56.3% 826153 ± 23% sched_debug.cpu.max_idle_balance_cost.stddev 1.00 -100.0% 0.00 sched_debug.sched_clock_stable() 4.56 ± 4% -64.0% 1.64 ± 11% perf-stat.i.MPKI 1.042e+09 ± 7% +19.1% 1.242e+09 ± 10% perf-stat.i.branch-instructions 2.63 ± 4% -1.6 1.05 ± 6% perf-stat.i.branch-miss-rate% 41155786 ± 8% -50.2% 20492728 ± 8% perf-stat.i.branch-misses 3790635 ± 13% -39.3% 2299993 ± 6% perf-stat.i.cache-misses 16736018 ± 6% -34.3% 10987918 ± 5% perf-stat.i.cache-references 3040 ± 4% -9.5% 2751 ± 3% perf-stat.i.context-switches 298.11 ± 8% -80.7% 57.40 ± 22% perf-stat.i.cpi 125682 ± 2% -14.8% 107100 ± 3% perf-stat.i.cpu-clock 191.72 ± 3% -16.2% 160.71 ± 2% perf-stat.i.cpu-migrations 294407 ± 9% -27.7% 212990 ± 5% perf-stat.i.cycles-between-cache-misses 0.02 ± 10% -0.0 0.01 ± 12% perf-stat.i.dTLB-load-miss-rate% 121512 ± 5% -34.4% 79730 ± 11% perf-stat.i.dTLB-load-misses 0.20 ± 8% +0.0 0.24 ± 10% perf-stat.i.dTLB-store-miss-rate% 4.246e+08 ± 7% -44.8% 2.341e+08 ± 8% perf-stat.i.dTLB-stores 77.32 ± 2% +8.8 86.11 perf-stat.i.iTLB-load-miss-rate% 1414999 ± 3% -17.9% 1162074 ± 6% perf-stat.i.iTLB-load-misses 851842 ± 8% -50.2% 423983 ± 8% perf-stat.i.iTLB-loads 2808 ± 7% +83.6% 5156 ± 10% perf-stat.i.instructions-per-iTLB-miss 0.31 ± 9% -54.3% 0.14 ± 11% perf-stat.i.ipc 93.97 ± 7% -54.3% 42.90 ± 10% perf-stat.i.major-faults 1.78 ± 5% +31.6% 2.34 perf-stat.i.metric.GHz 1.26 ± 14% -30.8% 0.87 ± 4% perf-stat.i.metric.K/sec 5445 ± 10% -37.7% 3394 ± 7% perf-stat.i.minor-faults 74.27 +9.3 83.52 perf-stat.i.node-load-miss-rate% 146539 ± 14% -38.8% 89694 ± 7% perf-stat.i.node-loads 49.40 ± 4% +28.1 77.46 perf-stat.i.node-store-miss-rate% 390280 ± 6% -56.4% 170291 ± 7% perf-stat.i.node-stores 5539 ± 10% -38.0% 3437 ± 7% perf-stat.i.page-faults 125706 ± 2% -14.8% 107108 ± 3% perf-stat.i.task-clock 3.86 ± 3% -63.2% 1.42 ± 13% perf-stat.overall.MPKI 3.34 ± 2% -2.6 0.77 ± 10% perf-stat.overall.branch-miss-rate% 162.35 ± 10% -66.2% 54.82 ± 16% perf-stat.overall.cpi 0.01 ± 8% -0.0 0.01 ± 10% perf-stat.overall.dTLB-load-miss-rate% 80.16 ± 2% +5.1 85.23 perf-stat.overall.iTLB-load-miss-rate% 1771 ± 10% +180.8% 4973 ± 12% perf-stat.overall.instructions-per-iTLB-miss 0.01 ± 11% +199.8% 0.02 ± 15% perf-stat.overall.ipc 73.89 +9.2 83.10 perf-stat.overall.node-load-miss-rate% 34.75 ± 5% +32.7 67.46 ± 2% perf-stat.overall.node-store-miss-rate% 2.203e+08 ± 18% +348.7% 9.885e+08 ± 15% perf-stat.ps.branch-instructions 872384 ± 13% +47.2% 1284050 ± 4% perf-stat.ps.cache-misses 4106664 ± 14% +51.6% 6227454 ± 6% perf-stat.ps.cache-references 1871 ± 10% +31.4% 2459 perf-stat.ps.context-switches 71293 ± 9% +28.9% 91877 perf-stat.ps.cpu-clock 1.704e+11 ± 7% +40.1% 2.386e+11 perf-stat.ps.cpu-cycles 114.07 ± 7% +21.1% 138.18 ± 3% perf-stat.ps.cpu-migrations 32365 ± 10% +42.9% 46255 ± 8% perf-stat.ps.dTLB-load-misses 2.409e+08 ± 17% +165.6% 6.397e+08 ± 13% perf-stat.ps.dTLB-loads 602712 ± 12% +48.6% 895483 ± 5% perf-stat.ps.iTLB-load-misses 1.07e+09 ± 18% +317.2% 4.464e+09 ± 15% perf-stat.ps.instructions 16.30 ± 16% -24.1% 12.37 ± 7% perf-stat.ps.major-faults 1353 ± 9% +38.3% 1871 ± 6% perf-stat.ps.minor-faults 101821 ± 18% +138.8% 243173 ± 4% perf-stat.ps.node-load-misses 35793 ± 15% +38.0% 49397 ± 4% perf-stat.ps.node-loads 43679 ± 17% +201.9% 131871 ± 7% perf-stat.ps.node-store-misses 82413 ± 20% -23.0% 63446 ± 4% perf-stat.ps.node-stores 1369 ± 9% +37.5% 1883 ± 6% perf-stat.ps.page-faults 71298 ± 9% +28.9% 91878 perf-stat.ps.task-clock 4.783e+10 ± 3% +472.2% 2.737e+11 ± 13% perf-stat.total.instructions 36.27 ± 5% -14.1 22.21 ± 27% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 35.94 ± 5% -13.9 22.05 ± 27% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 37.43 ± 5% -10.6 26.84 ± 26% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 42.76 ± 4% -9.8 32.92 ± 14% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 29.39 ± 7% -8.9 20.48 ± 28% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 19.99 ± 7% -8.5 11.52 ± 32% perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 6.31 ± 32% -5.6 0.71 ± 19% perf-profile.calltrace.cycles-pp.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 5.16 ± 13% -2.5 2.64 ± 31% perf-profile.calltrace.cycles-pp.update_curr.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle 5.78 ± 11% -2.1 3.71 ± 36% perf-profile.calltrace.cycles-pp.update_load_avg.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle 4.04 ± 13% -1.7 2.34 ± 29% perf-profile.calltrace.cycles-pp.hrtimer_active.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle 2.22 ± 16% -1.7 0.53 ± 45% perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 2.72 ± 8% -1.1 1.62 ± 13% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 2.62 ± 7% -1.1 1.56 ± 13% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.83 ± 11% -0.8 1.03 ± 47% perf-profile.calltrace.cycles-pp.update_cfs_group.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle 1.23 ± 2% -0.7 0.53 ± 48% perf-profile.calltrace.cycles-pp.hrtimer_update_next_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 3.04 ± 6% -0.7 2.37 ± 3% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt 3.74 ± 6% -0.6 3.14 ± 3% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 1.54 ± 8% -0.6 0.98 ± 50% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.update_load_avg.task_tick_fair.scheduler_tick.update_process_times 1.68 ± 10% -0.6 1.13 ± 10% perf-profile.calltrace.cycles-pp.execve 1.68 ± 10% -0.6 1.13 ± 10% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve 1.68 ± 10% -0.6 1.13 ± 10% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 1.68 ± 10% -0.6 1.13 ± 10% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 1.67 ± 10% -0.5 1.12 ± 10% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 2.05 ± 6% -0.5 1.51 ± 4% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu 1.39 ± 9% -0.5 0.92 ± 11% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.66 ± 8% -0.4 1.22 ± 9% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start 1.26 ± 9% -0.4 0.83 ± 12% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64 0.71 ± 8% -0.4 0.28 ±100% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork 0.71 ± 8% -0.4 0.28 ±100% perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork 0.71 ± 8% -0.4 0.28 ±100% perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork 1.22 ± 9% -0.4 0.80 ± 12% perf-profile.calltrace.cycles-pp.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve 1.44 ± 22% -0.4 1.07 ± 17% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork 0.71 ± 8% -0.3 0.37 ± 71% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork 0.70 ± 12% -0.3 0.38 ± 72% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 0.70 ± 12% -0.3 0.38 ± 72% perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.87 ± 20% -0.3 0.56 ± 49% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains 0.86 ± 11% -0.3 0.55 ± 47% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.86 ± 11% -0.3 0.55 ± 47% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.86 ± 11% -0.3 0.55 ± 47% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.36 ± 8% -0.3 1.08 ± 13% perf-profile.calltrace.cycles-pp.asm_exc_page_fault 1.30 ± 8% -0.3 1.03 ± 13% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault 1.30 ± 8% -0.3 1.03 ± 13% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 1.12 ± 9% -0.2 0.88 ± 14% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 1.05 ± 9% -0.2 0.82 ± 15% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 0.92 ± 8% -0.2 0.72 ± 9% perf-profile.calltrace.cycles-pp.__libc_fork 1.33 ± 8% +0.4 1.72 ± 8% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.82 ± 11% +0.7 1.51 ± 8% perf-profile.calltrace.cycles-pp.seq_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64 0.70 ± 11% +0.7 1.42 ± 9% perf-profile.calltrace.cycles-pp.show_stat.seq_read_iter.new_sync_read.vfs_read.ksys_read 0.31 ±104% +0.8 1.09 ± 44% perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 3.00 ± 11% +2.2 5.17 ± 45% perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 0.00 +3.8 3.82 ± 19% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.85 ± 7% +3.9 4.79 ± 44% perf-profile.calltrace.cycles-pp.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.00 +4.2 4.23 ± 20% perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.00 +4.5 4.46 ± 44% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt 0.00 +5.7 5.73 ± 21% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times 0.00 +6.6 6.55 ± 22% perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle 0.00 +6.7 6.72 ± 22% perf-profile.calltrace.cycles-pp.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 3.02 ± 13% +10.9 13.94 ± 22% perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 64.95 ± 2% +11.4 76.36 ± 3% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +12.8 12.83 ± 23% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 55.01 ± 2% +14.8 69.85 ± 4% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 54.57 ± 2% +15.2 69.76 ± 4% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 6.40 ± 6% +15.2 21.64 ± 13% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +17.3 17.27 ± 13% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt 1.54 ± 2% +17.6 19.16 ± 12% perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 41.14 ± 5% -16.3 24.84 ± 26% perf-profile.children.cycles-pp.tick_sched_handle 40.76 ± 5% -16.1 24.67 ± 26% perf-profile.children.cycles-pp.update_process_times 42.58 ± 5% -12.5 30.12 ± 25% perf-profile.children.cycles-pp.tick_sched_timer 48.92 ± 4% -11.9 36.99 ± 13% perf-profile.children.cycles-pp.__hrtimer_run_queues 33.34 ± 7% -10.5 22.86 ± 27% perf-profile.children.cycles-pp.scheduler_tick 22.36 ± 6% -9.7 12.68 ± 31% perf-profile.children.cycles-pp.task_tick_fair 6.65 ± 26% -5.8 0.83 ± 18% perf-profile.children.cycles-pp.perf_event_task_tick 4.88 ± 11% -3.9 1.02 ± 35% perf-profile.children.cycles-pp.ghes_notify_nmi 5.18 ± 10% -3.6 1.53 ± 24% perf-profile.children.cycles-pp.irqtime_account_irq 5.79 ± 10% -2.6 3.21 ± 28% perf-profile.children.cycles-pp.update_curr 2.78 ± 7% -2.5 0.26 ± 25% perf-profile.children.cycles-pp.__intel_pmu_enable_all 6.78 ± 10% -2.4 4.41 ± 33% perf-profile.children.cycles-pp.update_load_avg 10.38 ± 7% -2.1 8.23 ± 10% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 4.48 ± 12% -1.9 2.56 ± 29% perf-profile.children.cycles-pp.hrtimer_active 9.92 ± 7% -1.9 8.00 ± 10% perf-profile.children.cycles-pp.do_syscall_64 2.54 ± 14% -1.8 0.69 ± 20% perf-profile.children.cycles-pp.rcu_sched_clock_irq 2.66 ± 14% -1.6 1.08 ± 35% perf-profile.children.cycles-pp.native_sched_clock 1.70 ± 2% -1.1 0.57 ± 13% perf-profile.children.cycles-pp._raw_spin_lock 1.69 ± 2% -1.1 0.58 ± 24% perf-profile.children.cycles-pp.update_rq_clock 1.74 ± 7% -1.1 0.67 ± 38% perf-profile.children.cycles-pp.native_flush_tlb_one_user 1.36 ± 7% -1.0 0.35 ± 17% perf-profile.children.cycles-pp.trigger_load_balance 1.15 ± 30% -1.0 0.17 ± 47% perf-profile.children.cycles-pp.nmi_handle 1.26 ± 5% -0.9 0.35 ± 29% perf-profile.children.cycles-pp.arch_scale_freq_tick 2.07 ± 10% -0.9 1.20 ± 31% perf-profile.children.cycles-pp.update_cfs_group 3.57 ± 6% -0.8 2.72 ± 4% perf-profile.children.cycles-pp.load_balance 0.99 ± 24% -0.8 0.16 ± 24% perf-profile.children.cycles-pp.sync_regs 1.22 ± 4% -0.8 0.39 ± 21% perf-profile.children.cycles-pp.sched_clock_cpu 4.41 ± 6% -0.8 3.58 ± 4% perf-profile.children.cycles-pp.rebalance_domains 1.45 ± 2% -0.8 0.67 ± 22% perf-profile.children.cycles-pp.hrtimer_update_next_event 2.38 ± 6% -0.7 1.72 ± 5% perf-profile.children.cycles-pp.find_busiest_group 1.78 ± 10% -0.6 1.15 ± 10% perf-profile.children.cycles-pp.do_execveat_common 1.79 ± 10% -0.6 1.16 ± 10% perf-profile.children.cycles-pp.__x64_sys_execve 0.85 ± 8% -0.6 0.22 ± 49% perf-profile.children.cycles-pp.exc_nmi 2.25 ± 6% -0.6 1.62 ± 5% perf-profile.children.cycles-pp.update_sd_lb_stats 1.01 ± 6% -0.6 0.39 ± 26% perf-profile.children.cycles-pp.sched_clock 1.68 ± 10% -0.6 1.13 ± 10% perf-profile.children.cycles-pp.execve 1.48 ± 9% -0.5 0.94 ± 10% perf-profile.children.cycles-pp.bprm_execve 1.34 ± 9% -0.5 0.85 ± 11% perf-profile.children.cycles-pp.exec_binprm 1.30 ± 9% -0.5 0.82 ± 12% perf-profile.children.cycles-pp.load_elf_binary 0.74 ± 3% -0.5 0.27 ± 20% perf-profile.children.cycles-pp.update_irq_load_avg 2.50 ± 8% -0.4 2.05 ± 9% perf-profile.children.cycles-pp.asm_exc_page_fault 1.33 ± 10% -0.4 0.88 ± 16% perf-profile.children.cycles-pp.mmput 0.56 ± 7% -0.4 0.11 ± 24% perf-profile.children.cycles-pp.account_process_tick 1.32 ± 10% -0.4 0.88 ± 16% perf-profile.children.cycles-pp.exit_mmap 0.68 ± 6% -0.4 0.24 ± 9% perf-profile.children.cycles-pp.enqueue_hrtimer 2.04 ± 8% -0.4 1.60 ± 9% perf-profile.children.cycles-pp.handle_mm_fault 2.35 ± 8% -0.4 1.92 ± 9% perf-profile.children.cycles-pp.exc_page_fault 2.35 ± 8% -0.4 1.92 ± 8% perf-profile.children.cycles-pp.do_user_addr_fault 1.90 ± 8% -0.4 1.47 ± 9% perf-profile.children.cycles-pp.__handle_mm_fault 0.99 ± 8% -0.4 0.58 ± 8% perf-profile.children.cycles-pp.do_sys_openat2 0.99 ± 9% -0.4 0.58 ± 8% perf-profile.children.cycles-pp.do_sys_open 0.96 ± 9% -0.4 0.56 ± 6% perf-profile.children.cycles-pp.do_filp_open 0.95 ± 9% -0.4 0.55 ± 6% perf-profile.children.cycles-pp.path_openat 0.60 ± 6% -0.4 0.20 ± 12% perf-profile.children.cycles-pp.timerqueue_add 0.47 ± 18% -0.4 0.09 ± 17% perf-profile.children.cycles-pp.native_irq_return_iret 0.59 ± 41% -0.4 0.21 ± 11% perf-profile.children.cycles-pp.write 0.76 ± 7% -0.4 0.39 ± 9% perf-profile.children.cycles-pp._raw_spin_lock_irq 1.45 ± 22% -0.4 1.08 ± 17% perf-profile.children.cycles-pp.smpboot_thread_fn 0.69 ± 9% -0.4 0.33 ± 28% perf-profile.children.cycles-pp.acpi_os_read_memory 0.58 ± 4% -0.4 0.22 ± 29% perf-profile.children.cycles-pp.__remove_hrtimer 0.63 ± 4% -0.4 0.28 ± 20% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.69 ± 11% -0.3 0.35 ± 25% perf-profile.children.cycles-pp.update_min_vruntime 0.68 ± 2% -0.3 0.38 ± 24% perf-profile.children.cycles-pp.__hrtimer_get_next_event 0.46 ± 4% -0.3 0.16 ± 28% perf-profile.children.cycles-pp.timerqueue_del 0.70 ± 13% -0.3 0.41 ± 13% perf-profile.children.cycles-pp.begin_new_exec 0.49 ± 41% -0.3 0.21 ± 58% perf-profile.children.cycles-pp.intel_pmu_handle_irq 0.46 ± 2% -0.3 0.19 ± 28% perf-profile.children.cycles-pp.account_user_time 1.04 ± 8% -0.3 0.76 ± 8% perf-profile.children.cycles-pp.do_fault 0.36 ± 10% -0.3 0.10 ± 29% perf-profile.children.cycles-pp.ghes_copy_tofrom_phys 0.39 ± 5% -0.3 0.14 ± 37% perf-profile.children.cycles-pp.__native_set_fixmap 0.34 ± 6% -0.3 0.09 ± 59% perf-profile.children.cycles-pp.default_do_nmi 0.66 ± 10% -0.3 0.41 ± 16% perf-profile.children.cycles-pp.vm_mmap_pgoff 0.62 ± 10% -0.2 0.38 ± 17% perf-profile.children.cycles-pp.do_mmap 0.67 ± 6% -0.2 0.43 ± 14% perf-profile.children.cycles-pp.try_to_wake_up 0.88 ± 11% -0.2 0.64 ± 17% perf-profile.children.cycles-pp.do_exit 0.65 ± 6% -0.2 0.42 ± 14% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.88 ± 11% -0.2 0.64 ± 17% perf-profile.children.cycles-pp.__x64_sys_exit_group 0.88 ± 11% -0.2 0.64 ± 17% perf-profile.children.cycles-pp.do_group_exit 0.35 ± 34% -0.2 0.12 ± 22% perf-profile.children.cycles-pp.pipe_write 0.45 ± 25% -0.2 0.22 ± 23% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 0.83 ± 5% -0.2 0.61 ± 17% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.51 ± 8% -0.2 0.29 ± 12% perf-profile.children.cycles-pp.kmem_cache_alloc 0.56 ± 10% -0.2 0.34 ± 16% perf-profile.children.cycles-pp.mmap_region 0.49 ± 8% -0.2 0.29 ± 7% perf-profile.children.cycles-pp.link_path_walk 0.67 ± 11% -0.2 0.46 ± 16% perf-profile.children.cycles-pp.unmap_vmas 0.47 ± 8% -0.2 0.27 ± 8% perf-profile.children.cycles-pp.walk_component 0.53 ± 12% -0.2 0.32 ± 17% perf-profile.children.cycles-pp.ksys_mmap_pgoff 0.72 ± 10% -0.2 0.52 ± 10% perf-profile.children.cycles-pp.filemap_map_pages 0.92 ± 9% -0.2 0.72 ± 9% perf-profile.children.cycles-pp.__libc_fork 0.30 ± 31% -0.2 0.11 ± 18% perf-profile.children.cycles-pp.pipe_read 0.28 ± 31% -0.2 0.09 ± 23% perf-profile.children.cycles-pp.__wake_up_common_lock 0.71 ± 8% -0.2 0.53 ± 10% perf-profile.children.cycles-pp.__do_sys_clone 0.71 ± 7% -0.2 0.53 ± 10% perf-profile.children.cycles-pp.kernel_clone 0.68 ± 8% -0.2 0.50 ± 10% perf-profile.children.cycles-pp.copy_process 0.60 ± 12% -0.2 0.42 ± 16% perf-profile.children.cycles-pp.unmap_page_range 0.55 ± 13% -0.2 0.39 ± 18% perf-profile.children.cycles-pp.zap_pte_range 0.25 ± 5% -0.2 0.09 ± 7% perf-profile.children.cycles-pp.hrtimer_forward 0.46 ± 10% -0.2 0.30 ± 11% perf-profile.children.cycles-pp.next_uptodate_page 0.47 ± 8% -0.2 0.32 ± 11% perf-profile.children.cycles-pp.dup_mm 0.46 ± 12% -0.2 0.31 ± 14% perf-profile.children.cycles-pp.setlocale 0.45 ± 9% -0.1 0.30 ± 12% perf-profile.children.cycles-pp.dup_mmap 0.20 ± 18% -0.1 0.06 ± 72% perf-profile.children.cycles-pp.native_set_fixmap 0.23 ± 19% -0.1 0.09 ± 5% perf-profile.children.cycles-pp.proc_reg_read_iter 0.24 ± 15% -0.1 0.11 ± 36% perf-profile.children.cycles-pp.sched_slice 0.36 ± 9% -0.1 0.24 ± 20% perf-profile.children.cycles-pp.free_pgtables 0.26 ± 3% -0.1 0.14 ± 28% perf-profile.children.cycles-pp.tick_program_event 0.27 ± 10% -0.1 0.15 ± 29% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.31 ± 12% -0.1 0.20 ± 18% perf-profile.children.cycles-pp.__open64_nocancel 0.21 ± 8% -0.1 0.11 ± 6% perf-profile.children.cycles-pp.alloc_empty_file 0.25 ± 7% -0.1 0.14 ± 45% perf-profile.children.cycles-pp.calc_global_load_tick 0.24 ± 6% -0.1 0.14 ± 17% perf-profile.children.cycles-pp.__split_vma 0.18 ± 10% -0.1 0.07 ± 16% perf-profile.children.cycles-pp.rb_insert_color 0.21 ± 9% -0.1 0.10 ± 7% perf-profile.children.cycles-pp.__alloc_file 0.22 ± 9% -0.1 0.12 ± 15% perf-profile.children.cycles-pp.filename_lookup 0.16 ± 55% -0.1 0.06 ± 48% perf-profile.children.cycles-pp.__poll 0.22 ± 10% -0.1 0.12 ± 17% perf-profile.children.cycles-pp.path_lookupat 0.27 ± 11% -0.1 0.17 ± 21% perf-profile.children.cycles-pp.__do_munmap 0.47 ± 4% -0.1 0.38 ± 9% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.20 ± 6% -0.1 0.11 ± 18% perf-profile.children.cycles-pp.__vma_adjust 0.19 ± 11% -0.1 0.11 ± 15% perf-profile.children.cycles-pp.__lookup_slow 0.17 ± 11% -0.1 0.09 ± 15% perf-profile.children.cycles-pp.vma_interval_tree_insert 0.18 ± 15% -0.1 0.10 ± 20% perf-profile.children.cycles-pp.enqueue_task_fair 0.17 ± 13% -0.1 0.09 ± 16% perf-profile.children.cycles-pp.__do_sys_newstat 0.18 ± 15% -0.1 0.10 ± 18% perf-profile.children.cycles-pp.ttwu_do_activate 0.34 ± 5% -0.1 0.27 ± 14% perf-profile.children.cycles-pp.kmem_cache_free 0.17 ± 14% -0.1 0.09 ± 16% perf-profile.children.cycles-pp.vfs_statx 0.28 ± 12% -0.1 0.20 ± 7% perf-profile.children.cycles-pp.call_timer_fn 0.20 ± 5% -0.1 0.13 ± 17% perf-profile.children.cycles-pp.elf_map 0.14 ± 11% -0.1 0.07 ± 18% perf-profile.children.cycles-pp.open64 0.23 ± 9% -0.1 0.15 ± 19% perf-profile.children.cycles-pp.tlb_finish_mmu 0.18 ± 9% -0.1 0.10 ± 10% perf-profile.children.cycles-pp.lookup_fast 0.17 ± 9% -0.1 0.10 ± 8% perf-profile.children.cycles-pp.dput 0.22 ± 9% -0.1 0.15 ± 20% perf-profile.children.cycles-pp.tlb_flush_mmu 0.17 ± 9% -0.1 0.10 ± 15% perf-profile.children.cycles-pp.step_into 0.19 ± 5% -0.1 0.12 ± 10% perf-profile.children.cycles-pp.__x64_sys_mprotect 0.19 ± 5% -0.1 0.12 ± 10% perf-profile.children.cycles-pp.do_mprotect_pkey 0.14 ± 9% -0.1 0.07 ± 49% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime 0.17 ± 19% -0.1 0.10 ± 13% perf-profile.children.cycles-pp.select_task_rq_fair 0.33 ± 4% -0.1 0.27 ± 9% perf-profile.children.cycles-pp.get_page_from_freelist 0.20 ± 8% -0.1 0.13 ± 22% perf-profile.children.cycles-pp.cpuacct_charge 0.13 ± 17% -0.1 0.06 ± 17% perf-profile.children.cycles-pp.kstat_irqs_usr 0.18 ± 5% -0.1 0.11 ± 10% perf-profile.children.cycles-pp.mprotect_fixup 0.17 ± 11% -0.1 0.10 ± 23% perf-profile.children.cycles-pp.vma_link 0.12 ± 22% -0.1 0.06 ± 15% perf-profile.children.cycles-pp.touch_atime 0.15 ± 15% -0.1 0.09 ± 13% perf-profile.children.cycles-pp.d_alloc_parallel 0.09 ± 16% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__xstat64 0.14 ± 7% -0.1 0.08 ± 32% perf-profile.children.cycles-pp.__fput 0.31 ± 9% -0.1 0.26 ± 9% perf-profile.children.cycles-pp.wp_page_copy 0.15 ± 12% -0.1 0.09 ± 14% perf-profile.children.cycles-pp.do_open 0.18 ± 16% -0.1 0.12 ± 20% perf-profile.children.cycles-pp.__mmap 0.38 ± 6% -0.1 0.32 ± 13% perf-profile.children.cycles-pp.__calc_delta 0.16 ± 8% -0.1 0.11 ± 22% perf-profile.children.cycles-pp.unlink_anon_vmas 0.13 ± 17% -0.1 0.07 ± 18% perf-profile.children.cycles-pp.d_alloc 0.11 ± 11% -0.1 0.06 ± 46% perf-profile.children.cycles-pp.remove_vma 0.15 ± 17% -0.1 0.10 ± 20% perf-profile.children.cycles-pp.unlink_file_vma 0.15 ± 8% -0.1 0.09 ± 15% perf-profile.children.cycles-pp.dequeue_task_fair 0.12 ± 10% -0.1 0.07 ± 21% perf-profile.children.cycles-pp.vma_interval_tree_remove 0.17 ± 8% -0.1 0.11 ± 14% perf-profile.children.cycles-pp.vmstat_update 0.16 ± 8% -0.1 0.11 ± 26% perf-profile.children.cycles-pp.__cgroup_account_cputime_field 0.09 ± 10% -0.0 0.04 ± 73% perf-profile.children.cycles-pp.__d_alloc 0.14 ± 2% -0.0 0.09 ± 15% perf-profile.children.cycles-pp.load_elf_interp 0.14 ± 13% -0.0 0.10 ± 11% perf-profile.children.cycles-pp._dl_addr 0.16 ± 8% -0.0 0.11 ± 14% perf-profile.children.cycles-pp.refresh_cpu_vm_stats 0.09 ± 5% -0.0 0.04 ± 72% perf-profile.children.cycles-pp.irq_work_run_list 0.11 ± 9% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.get_obj_cgroup_from_current 0.11 ± 18% -0.0 0.06 ± 48% perf-profile.children.cycles-pp.__x64_sys_poll 0.11 ± 18% -0.0 0.06 ± 48% perf-profile.children.cycles-pp.do_sys_poll 0.15 ± 11% -0.0 0.11 ± 12% perf-profile.children.cycles-pp.copy_page_range 0.10 ± 16% -0.0 0.05 ± 46% perf-profile.children.cycles-pp.do_dentry_open 0.10 ± 6% -0.0 0.06 ± 50% perf-profile.children.cycles-pp.__cgroup_account_cputime 0.16 ± 22% -0.0 0.12 ± 17% perf-profile.children.cycles-pp.update_rt_rq_load_avg 0.07 ± 15% -0.0 0.03 ±100% perf-profile.children.cycles-pp.__dentry_kill 0.09 ± 11% -0.0 0.05 ± 46% perf-profile.children.cycles-pp.__d_lookup_rcu 0.11 ± 16% -0.0 0.07 ± 20% perf-profile.children.cycles-pp.enqueue_entity 0.08 ± 20% -0.0 0.04 ± 72% perf-profile.children.cycles-pp.__list_del_entry_valid 0.16 ± 11% -0.0 0.12 ± 12% perf-profile.children.cycles-pp.kcpustat_cpu_fetch 0.10 ± 6% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.vm_area_dup 0.10 ± 15% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.__might_sleep 0.09 ± 10% -0.0 0.06 ± 46% perf-profile.children.cycles-pp.anon_vma_fork 0.12 ± 8% -0.0 0.08 ± 7% perf-profile.children.cycles-pp.pte_alloc_one 0.10 ± 18% -0.0 0.07 ± 16% perf-profile.children.cycles-pp.get_arg_page 0.14 ± 8% -0.0 0.10 ± 20% perf-profile.children.cycles-pp.perf_event_mmap 0.08 ± 17% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.strnlen_user 0.10 ± 17% -0.0 0.07 ± 16% perf-profile.children.cycles-pp.__get_user_pages_remote 0.10 ± 18% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.__get_user_pages 0.08 ± 14% -0.0 0.05 ± 45% perf-profile.children.cycles-pp.create_elf_tables 0.10 ± 10% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.do_open_execat 0.07 ± 7% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.refill_obj_stock 0.11 ± 10% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.__clear_user 0.09 ± 8% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.lru_add_drain 0.08 ± 13% -0.0 0.06 ± 8% perf-profile.children.cycles-pp.lru_add_drain_cpu 0.08 ± 14% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.__check_object_size 0.15 ± 20% +0.1 0.22 ± 17% perf-profile.children.cycles-pp.memcpy_toio 0.15 ± 20% +0.1 0.24 ± 19% perf-profile.children.cycles-pp.drm_fb_helper_damage_work 0.18 ± 8% +0.2 0.36 ± 57% perf-profile.children.cycles-pp.do_idle 0.18 ± 8% +0.2 0.37 ± 57% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 0.18 ± 8% +0.2 0.37 ± 57% perf-profile.children.cycles-pp.cpu_startup_entry 0.17 ± 10% +0.2 0.36 ± 54% perf-profile.children.cycles-pp.start_secondary 0.01 ±223% +0.2 0.21 ± 63% perf-profile.children.cycles-pp.cpuidle_enter 0.01 ±223% +0.2 0.21 ± 63% perf-profile.children.cycles-pp.cpuidle_enter_state 1.46 ± 9% +0.3 1.74 ± 8% perf-profile.children.cycles-pp.new_sync_read 0.13 ±100% +0.3 0.45 ± 16% perf-profile.children.cycles-pp.collapse_huge_page 0.10 ± 14% +0.3 0.43 ± 7% perf-profile.children.cycles-pp.get_cpu_idle_time_us 0.29 ± 44% +0.3 0.63 ± 34% perf-profile.children.cycles-pp.process_interval 0.10 ± 17% +0.3 0.43 ± 7% perf-profile.children.cycles-pp.get_idle_time 0.13 ±103% +0.3 0.48 ± 16% perf-profile.children.cycles-pp.khugepaged_scan_mm_slot 0.13 ±103% +0.3 0.48 ± 16% perf-profile.children.cycles-pp.khugepaged_scan_pmd 0.14 ±100% +0.3 0.49 ± 16% perf-profile.children.cycles-pp.khugepaged 0.04 ± 71% +0.4 0.47 ± 10% perf-profile.children.cycles-pp.get_iowait_time 0.02 ±142% +0.4 0.46 ± 11% perf-profile.children.cycles-pp.get_cpu_iowait_time_us 1.14 ± 9% +0.5 1.67 ± 7% perf-profile.children.cycles-pp.seq_read_iter 0.80 ± 17% +0.6 1.36 ± 38% perf-profile.children.cycles-pp.rcu_core 0.00 +0.7 0.67 ± 25% perf-profile.children.cycles-pp.sched_clock_local 0.71 ± 11% +0.7 1.43 ± 9% perf-profile.children.cycles-pp.show_stat 3.58 ± 9% +2.2 5.82 ± 43% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.00 +7.6 7.59 ± 21% perf-profile.children.cycles-pp.sched_clock_tick 74.19 ± 2% +11.5 85.71 ± 2% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 3.43 ± 11% +12.4 15.79 ± 21% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 62.93 ± 2% +15.6 78.51 ± 3% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 62.43 ± 2% +16.0 78.42 ± 3% perf-profile.children.cycles-pp.hrtimer_interrupt 6.89 ± 6% +17.4 24.27 ± 13% perf-profile.children.cycles-pp.clockevents_program_event 3.42 ± 2% +36.6 40.06 ± 6% perf-profile.children.cycles-pp.ktime_get 0.00 +50.2 50.16 ± 10% perf-profile.children.cycles-pp.read_hpet 4.88 ± 11% -3.9 1.02 ± 35% perf-profile.self.cycles-pp.ghes_notify_nmi 3.79 ± 19% -3.2 0.62 ± 10% perf-profile.self.cycles-pp.perf_event_task_tick 3.77 ± 16% -2.5 1.30 ± 26% perf-profile.self.cycles-pp.irqtime_account_irq 2.27 ± 12% -2.0 0.24 ± 21% perf-profile.self.cycles-pp.__intel_pmu_enable_all 2.98 ± 14% -1.8 1.19 ± 15% perf-profile.self.cycles-pp.ktime_get_update_offsets_now 3.90 ± 16% -1.7 2.21 ± 26% perf-profile.self.cycles-pp.hrtimer_active 3.49 ± 13% -1.6 1.89 ± 26% perf-profile.self.cycles-pp.update_curr 2.06 ± 20% -1.5 0.59 ± 18% perf-profile.self.cycles-pp.rcu_sched_clock_irq 2.42 ± 12% -1.4 1.03 ± 34% perf-profile.self.cycles-pp.native_sched_clock 2.44 ± 13% -1.4 1.07 ± 27% perf-profile.self.cycles-pp.task_tick_fair 1.61 ± 18% -1.3 0.28 ± 23% perf-profile.self.cycles-pp.update_process_times 2.48 ± 22% -1.2 1.27 ± 30% perf-profile.self.cycles-pp.clockevents_program_event 1.74 ± 7% -1.1 0.67 ± 38% perf-profile.self.cycles-pp.native_flush_tlb_one_user 1.15 ± 30% -1.0 0.17 ± 47% perf-profile.self.cycles-pp.nmi_handle 2.27 ± 11% -0.9 1.36 ± 28% perf-profile.self.cycles-pp.update_load_avg 0.95 ± 25% -0.8 0.16 ± 25% perf-profile.self.cycles-pp.sync_regs 1.04 ± 11% -0.8 0.28 ± 18% perf-profile.self.cycles-pp.trigger_load_balance 1.15 ± 8% -0.8 0.39 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 1.74 ± 14% -0.7 1.02 ± 27% perf-profile.self.cycles-pp.update_cfs_group 0.89 ± 8% -0.6 0.26 ± 24% perf-profile.self.cycles-pp.arch_scale_freq_tick 0.84 ± 8% -0.6 0.22 ± 49% perf-profile.self.cycles-pp.exc_nmi 0.49 ± 8% -0.4 0.10 ± 21% perf-profile.self.cycles-pp.account_process_tick 0.72 ± 17% -0.4 0.33 ± 20% perf-profile.self.cycles-pp.hrtimer_interrupt 0.46 ± 19% -0.4 0.07 ± 10% perf-profile.self.cycles-pp.native_irq_return_iret 1.30 ± 6% -0.4 0.92 ± 23% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.69 ± 9% -0.4 0.33 ± 28% perf-profile.self.cycles-pp.acpi_os_read_memory 0.53 ± 15% -0.3 0.20 ± 10% perf-profile.self.cycles-pp.update_irq_load_avg 0.51 ± 10% -0.3 0.19 ± 13% perf-profile.self.cycles-pp.__hrtimer_run_queues 0.48 ± 7% -0.3 0.17 ± 22% perf-profile.self.cycles-pp.update_rq_clock 0.50 ± 5% -0.3 0.20 ± 27% perf-profile.self.cycles-pp.scheduler_tick 1.04 ± 14% -0.3 0.74 ± 12% perf-profile.self.cycles-pp.update_sd_lb_stats 0.37 ± 29% -0.3 0.08 ± 18% perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt 0.39 ± 12% -0.3 0.11 ± 38% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.49 ± 41% -0.3 0.21 ± 58% perf-profile.self.cycles-pp.intel_pmu_handle_irq 0.34 ± 2% -0.3 0.07 ± 47% perf-profile.self.cycles-pp.tick_sched_timer 0.36 ± 10% -0.3 0.10 ± 29% perf-profile.self.cycles-pp.ghes_copy_tofrom_phys 0.39 ± 5% -0.3 0.14 ± 37% perf-profile.self.cycles-pp.__native_set_fixmap 0.34 ± 6% -0.3 0.09 ± 59% perf-profile.self.cycles-pp.default_do_nmi 0.54 ± 7% -0.2 0.29 ± 18% perf-profile.self.cycles-pp.update_min_vruntime 0.55 ± 14% -0.2 0.34 ± 10% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.42 ± 9% -0.2 0.22 ± 17% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.51 ± 4% -0.2 0.31 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.31 ± 12% -0.2 0.11 ± 12% perf-profile.self.cycles-pp.timerqueue_add 0.48 ± 6% -0.2 0.33 ± 13% perf-profile.self.cycles-pp.update_blocked_averages 0.20 ± 18% -0.1 0.06 ± 72% perf-profile.self.cycles-pp.native_set_fixmap 0.20 ± 9% -0.1 0.06 ± 47% perf-profile.self.cycles-pp.account_user_time 0.34 ± 8% -0.1 0.21 ± 16% perf-profile.self.cycles-pp.__hrtimer_get_next_event 0.24 ± 12% -0.1 0.11 ± 18% perf-profile.self.cycles-pp.hrtimer_update_next_event 0.22 ± 10% -0.1 0.10 ± 11% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.18 ± 8% -0.1 0.08 ± 12% perf-profile.self.cycles-pp.hrtimer_forward 0.22 ± 9% -0.1 0.13 ± 46% perf-profile.self.cycles-pp.calc_global_load_tick 0.13 ± 12% -0.1 0.05 ± 49% perf-profile.self.cycles-pp.sched_slice 0.12 ± 15% -0.1 0.06 ± 8% perf-profile.self.cycles-pp.rb_insert_color 0.17 ± 13% -0.1 0.11 ± 25% perf-profile.self.cycles-pp.tick_program_event 0.15 ± 17% -0.1 0.10 ± 24% perf-profile.self.cycles-pp.cpuacct_charge 0.17 ± 2% -0.1 0.12 ± 22% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.09 ± 13% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.finish_task_switch 0.11 ± 10% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.kmem_cache_alloc 0.09 ± 15% -0.0 0.05 ± 48% perf-profile.self.cycles-pp.__cgroup_account_cputime_field 0.09 ± 11% -0.0 0.05 ± 49% perf-profile.self.cycles-pp.__remove_hrtimer 0.10 ± 12% +0.1 0.18 ± 33% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.02 ±141% +0.1 0.10 ± 11% perf-profile.self.cycles-pp.memcpy_toio 0.07 ± 19% +0.1 0.19 perf-profile.self.cycles-pp.run_rebalance_domains 0.00 +0.1 0.12 ± 35% perf-profile.self.cycles-pp.rcu_core 0.00 +0.3 0.33 ± 19% perf-profile.self.cycles-pp.sched_clock_local 0.17 ± 6% +0.3 0.51 ± 11% perf-profile.self.cycles-pp.__softirqentry_text_start 2.38 ± 6% +1.4 3.80 ± 7% perf-profile.self.cycles-pp.ktime_get 0.00 +47.1 47.13 ± 12% perf-profile.self.cycles-pp.read_hpet stress-ng.time.user_time 5200 +--------------------------------------------------------------------+ | OO OO OO O OO O O OO | 5000 |-+ O OO O O OO O OO O OO | 4800 |-+ O O | | | 4600 |-+ | 4400 |-+ | | | 4200 |-++. .+ + | 4000 |-+: ++ + .++ + : .+ .+ + .+| |.: + +.+ : : + + ++ + +. : +. + + ++ | 3800 |-+ + : : +. + +.++.+ :: + + + ++ : | 3600 |-+ +.++ ++ :.+ : + +: | | + + + | 3400 +--------------------------------------------------------------------+ stress-ng.time.percent_of_cpu_this_job_got 9000 +--------------------------------------------------------------------+ | | 8500 |-OO OO OO O OO OO OO OO O OO O O OO | | O O O OO | 8000 |-+ O | | | 7500 |-+ | | | 7000 |-++.+ .+ + | |. : +. +. + .++ + + + +.++.+ + .+| 6500 |-+ + + + : .++. +.+ + .++ + :+ : + : ++ | | +.++ +.++ + :.+ + + + + ++.: | 6000 |-+ + + + | | | 5500 +--------------------------------------------------------------------+ stress-ng.lockbus.ops 260000 +------------------------------------------------------------------+ | + | 255000 |-+ : : +. | 250000 |-.+ +. +.+ + .+ : +. .++. .+ .+ + .+ +.++.++.+ +. | |+ +.+ + +.+ + + ++ ++ ++.++.++ + + +.+ +| 245000 |-+ | 240000 |-+ | | | 235000 |-+ O | 230000 |-+ O O O O | | OO O O O O OO | 225000 |-+ O O O O O O O | 220000 |O+O O OO O O | | O | 215000 +------------------------------------------------------------------+ stress-ng.lockbus.ops_per_sec 4300 +--------------------------------------------------------------------+ | + | 4200 |-+ : : + | |. +. .++.++. .++.+ : +. .++.+ .++. + +. +.++.++.++.+ +.+| 4100 |-+ ++ ++ + ++ + ++.++.++ + +.+ | | | 4000 |-+ | | | 3900 |-+ | | O | 3800 |-+ O O O O OO O O | | O O O O | 3700 |-OO O O O O O O O | | O OO O O O | 3600 +--------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang