Greeting, FYI, we noticed a -10.1% regression of reaim.jobs_per_min due to commit: commit: cda0d205259a7e7c1f8e296130ac091557134a66 ("[PATCH v5 2/2] memcg: infrastructure to flush memcg stats") url: https://github.com/0day-ci/linux/commits/Shakeel-Butt/memcg-switch-lruvec-stats-to-rstat/20210718-103230 base: https://git.kernel.org/cgit/linux/kernel/git/tj/cgroup.git for-next in testcase: reaim on test machine: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory with following parameters: runtime: 300s nr_task: 100% test: new_dbase cpufreq_governor: performance ucode: 0xde test-description: REAIM is an updated and improved version of AIM 7 benchmark. test-url: https://sourceforge.net/projects/re-aim-7/ In addition to that, the commit also has significant impact on the following tests: +------------------+-------------------------------------------------------------------------------------+ | testcase: change | reaim: reaim.jobs_per_min -9.5% regression | | test machine | 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory | | test parameters | cpufreq_governor=performance | | | nr_task=100% | | | runtime=300s | | | test=new_fserver | | | ucode=0xde | +------------------+-------------------------------------------------------------------------------------+ | testcase: change | fsmark: fsmark.files_per_sec -6.9% regression | | test machine | 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory | | test parameters | cpufreq_governor=performance | | | disk=1BRD_48G | | | filesize=4M | | | fs=btrfs | | | iterations=1x | | | nr_threads=1t | | | sync_method=fsyncBeforeClose | | | test_size=24G | | | ucode=0x5003006 | +------------------+-------------------------------------------------------------------------------------+ | testcase: change | reaim: reaim.jobs_per_min -10.7% regression | | test machine | 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory | | test parameters | cpufreq_governor=performance | | | nr_task=100% | | | runtime=300s | | | test=five_sec | | | ucode=0xde | +------------------+-------------------------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run bin/lkp run generated-yaml-file ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/300s/lkp-cfl-e1/new_dbase/reaim/0xde commit: 0c7c5da68b ("memcg: switch lruvec stats to rstat") cda0d20525 ("memcg: infrastructure to flush memcg stats") 0c7c5da68b4a09ff cda0d205259a7e7c1f8e296130a ---------------- --------------------------- %stddev %change %stddev \ | \ 5.09 +5.8% 5.39 reaim.child_systime 5.44 -1.3% 5.37 reaim.child_utime 136122 -10.1% 122315 reaim.jobs_per_min 8507 -10.1% 7644 reaim.jobs_per_min_child 140593 -9.5% 127245 reaim.max_jobs_per_min 0.73 +11.3% 0.81 reaim.parent_time 1.68 +54.9% 2.61 ± 2% reaim.std_dev_percent 0.01 +82.8% 0.02 reaim.std_dev_time 62961 +1456.4% 979901 reaim.time.involuntary_context_switches 15091 -4.8% 14367 reaim.time.major_page_faults 30304357 -3.1% 29368128 reaim.time.minor_page_faults 557.91 +2.5% 571.83 reaim.time.system_time 589.92 -4.3% 564.56 reaim.time.user_time 194873 +2.7% 200206 reaim.time.voluntary_context_switches 173333 -3.1% 168000 reaim.workload 4225548 ± 3% +58.6% 6703765 ± 25% cpuidle..usage 12.09 +1.7 13.74 mpstat.cpu.all.sys% 2487 +700.9% 19922 vmstat.system.cs 21607 +28.8% 27835 ± 17% vmstat.system.in 36.54 +4.2% 38.08 boot-time.boot 11.56 +2.9% 11.90 ± 3% boot-time.dhcp 487.32 +4.7% 510.04 boot-time.idle 20013 +143.0% 48643 ± 3% meminfo.Active 19783 +145.2% 48504 ± 3% meminfo.Active(anon) 28739 +105.6% 59076 ± 2% meminfo.Shmem 1875 ± 6% -10.3% 1683 ± 3% slabinfo.kmalloc-rcl-256.active_objs 1916 ± 4% -9.5% 1734 ± 2% slabinfo.kmalloc-rcl-256.num_objs 1089 ± 4% -13.1% 946.14 ± 5% slabinfo.pid.active_objs 1089 ± 4% -13.1% 946.14 ± 5% slabinfo.pid.num_objs 1127 ± 5% -7.9% 1038 ± 2% slabinfo.task_delay_info.active_objs 1127 ± 5% -7.9% 1038 ± 2% slabinfo.task_delay_info.num_objs 32528 ± 74% +4337.7% 1443515 ± 17% turbostat.C1 0.02 ± 44% +1.0 1.02 ± 41% turbostat.C1% 358034 ± 38% +272.8% 1334615 ± 90% turbostat.C1E 0.06 ± 54% +694.8% 0.44 ± 98% turbostat.CPU%c3 35.84 +7.3% 38.46 turbostat.CorWatt 6582740 +28.6% 8467818 ± 17% turbostat.IRQ 11.71 ± 4% -11.2% 10.40 ± 8% turbostat.Pkg%pc2 36.27 +7.2% 38.90 turbostat.PkgWatt 4948 +145.1% 12127 ± 3% proc-vmstat.nr_active_anon 595570 +1.2% 602681 proc-vmstat.nr_file_pages 7890 +5.5% 8324 proc-vmstat.nr_mapped 7184 +105.7% 14775 ± 2% proc-vmstat.nr_shmem 4948 +145.1% 12127 ± 3% proc-vmstat.nr_zone_active_anon 29299018 -3.0% 28427445 proc-vmstat.numa_hit 29299018 -3.0% 28427445 proc-vmstat.numa_local 58688 +30.2% 76398 proc-vmstat.pgactivate 29590587 -3.0% 28698285 proc-vmstat.pgalloc_normal 30793164 -3.0% 29863963 proc-vmstat.pgfault 29571246 -3.1% 28659712 proc-vmstat.pgfree 1958161 -3.2% 1896449 proc-vmstat.pgreuse 34.22 -2.6% 33.34 perf-stat.i.MPKI 4.04e+09 +6.6% 4.306e+09 perf-stat.i.branch-instructions 1.34 -0.0 1.30 perf-stat.i.branch-miss-rate% 31862409 +8.4% 34539247 perf-stat.i.branch-misses 1.39 ± 4% -0.1 1.27 perf-stat.i.cache-miss-rate% 6.763e+08 +4.6% 7.073e+08 perf-stat.i.cache-references 2474 +707.5% 19982 perf-stat.i.context-switches 1.19 -2.4% 1.16 perf-stat.i.cpi 1.794e+10 +5.8% 1.897e+10 perf-stat.i.cpu-cycles 312.57 +85.3% 579.14 perf-stat.i.cpu-migrations 3386 ± 3% +8.3% 3665 perf-stat.i.cycles-between-cache-misses 0.07 -0.0 0.06 ± 2% perf-stat.i.dTLB-load-miss-rate% 3.903e+09 +9.1% 4.257e+09 perf-stat.i.dTLB-loads 0.02 -0.0 0.02 perf-stat.i.dTLB-store-miss-rate% 467914 -2.7% 455349 perf-stat.i.dTLB-store-misses 2.337e+09 +5.4% 2.463e+09 perf-stat.i.dTLB-stores 69.23 -3.8 65.41 ± 4% perf-stat.i.iTLB-load-miss-rate% 641802 +6.6% 683980 ± 4% perf-stat.i.iTLB-load-misses 344005 +26.1% 433937 ± 8% perf-stat.i.iTLB-loads 1.964e+10 +5.5% 2.073e+10 perf-stat.i.instructions 0.92 +1.3% 0.93 perf-stat.i.ipc 50.15 -4.7% 47.78 perf-stat.i.major-faults 1.12 +5.7% 1.19 perf-stat.i.metric.GHz 154.74 +2.7% 158.85 perf-stat.i.metric.K/sec 684.89 +7.1% 733.44 perf-stat.i.metric.M/sec 101097 -2.9% 98143 perf-stat.i.minor-faults 1266584 -1.7% 1245496 perf-stat.i.node-stores 101148 -2.9% 98191 perf-stat.i.page-faults 0.79 +0.0 0.80 perf-stat.overall.branch-miss-rate% 0.68 ± 2% -0.0 0.64 perf-stat.overall.cache-miss-rate% 3905 ± 2% +6.8% 4172 perf-stat.overall.cycles-between-cache-misses 0.02 -0.0 0.01 ± 4% perf-stat.overall.dTLB-load-miss-rate% 0.02 -0.0 0.02 perf-stat.overall.dTLB-store-miss-rate% 65.10 -3.9 61.22 perf-stat.overall.iTLB-load-miss-rate% 34168578 +8.8% 37185877 perf-stat.overall.path-length 4.028e+09 +6.6% 4.293e+09 perf-stat.ps.branch-instructions 31761900 +8.4% 34426923 perf-stat.ps.branch-misses 6.742e+08 +4.6% 7.05e+08 perf-stat.ps.cache-references 2467 +707.3% 19922 perf-stat.ps.context-switches 1.789e+10 +5.7% 1.891e+10 perf-stat.ps.cpu-cycles 311.76 +85.2% 577.51 perf-stat.ps.cpu-migrations 3.891e+09 +9.1% 4.244e+09 perf-stat.ps.dTLB-loads 466586 -2.7% 454015 perf-stat.ps.dTLB-store-misses 2.33e+09 +5.4% 2.456e+09 perf-stat.ps.dTLB-stores 639845 +6.6% 681843 ± 4% perf-stat.ps.iTLB-load-misses 343006 +26.1% 432660 ± 8% perf-stat.ps.iTLB-loads 1.958e+10 +5.5% 2.067e+10 perf-stat.ps.instructions 50.01 -4.7% 47.64 perf-stat.ps.major-faults 100816 -2.9% 97861 perf-stat.ps.minor-faults 1262741 -1.7% 1241553 perf-stat.ps.node-stores 100866 -2.9% 97908 perf-stat.ps.page-faults 5.923e+12 +5.5% 6.247e+12 perf-stat.total.instructions 20646 ± 5% +638.2% 152404 interrupts.CAL:Function_call_interrupts 1553 ± 65% +524.0% 9695 ± 8% interrupts.CPU0.CAL:Function_call_interrupts 1455 ± 2% +3654.2% 54636 ± 6% interrupts.CPU0.RES:Rescheduling_interrupts 4.00 ± 77% +5085.7% 207.43 ± 10% interrupts.CPU0.TLB:TLB_shootdowns 1429 ± 33% +535.0% 9077 ± 4% interrupts.CPU1.CAL:Function_call_interrupts 3201 ± 5% +23.3% 3948 ± 2% interrupts.CPU1.NMI:Non-maskable_interrupts 3201 ± 5% +23.3% 3948 ± 2% interrupts.CPU1.PMI:Performance_monitoring_interrupts 1427 ± 2% +3468.7% 50931 ± 6% interrupts.CPU1.RES:Rescheduling_interrupts 4.67 ± 52% +4488.8% 214.14 ± 10% interrupts.CPU1.TLB:TLB_shootdowns 843.83 ± 53% +1007.6% 9346 ± 9% interrupts.CPU10.CAL:Function_call_interrupts 1476 ± 3% +3642.0% 55231 ± 5% interrupts.CPU10.RES:Rescheduling_interrupts 4.00 ± 40% +5232.1% 213.29 ± 8% interrupts.CPU10.TLB:TLB_shootdowns 1094 ± 58% +744.2% 9242 ± 7% interrupts.CPU11.CAL:Function_call_interrupts 1452 ± 3% +3495.2% 52214 ± 3% interrupts.CPU11.RES:Rescheduling_interrupts 5.17 ± 47% +3978.3% 210.71 ± 9% interrupts.CPU11.TLB:TLB_shootdowns 1155 ± 68% +769.0% 10036 ± 11% interrupts.CPU12.CAL:Function_call_interrupts 1502 ± 4% +3506.4% 54174 ± 8% interrupts.CPU12.RES:Rescheduling_interrupts 4.00 ± 57% +5350.0% 218.00 ± 9% interrupts.CPU12.TLB:TLB_shootdowns 1530 ± 58% +544.1% 9857 ± 11% interrupts.CPU13.CAL:Function_call_interrupts 1484 ± 2% +3436.4% 52509 ± 2% interrupts.CPU13.RES:Rescheduling_interrupts 5.17 ± 26% +4069.6% 215.43 ± 8% interrupts.CPU13.TLB:TLB_shootdowns 1181 ± 46% +808.1% 10732 ± 10% interrupts.CPU14.CAL:Function_call_interrupts 1454 ± 3% +3479.2% 52065 ± 6% interrupts.CPU14.RES:Rescheduling_interrupts 4.17 ± 59% +5073.7% 215.57 ± 8% interrupts.CPU14.TLB:TLB_shootdowns 1553 ± 91% +507.4% 9435 ± 6% interrupts.CPU15.CAL:Function_call_interrupts 1462 ± 3% +3482.7% 52385 ± 5% interrupts.CPU15.RES:Rescheduling_interrupts 5.00 ± 25% +4048.6% 207.43 ± 5% interrupts.CPU15.TLB:TLB_shootdowns 907.33 ± 41% +891.7% 8997 ± 6% interrupts.CPU2.CAL:Function_call_interrupts 1453 ± 2% +3546.2% 53003 ± 8% interrupts.CPU2.RES:Rescheduling_interrupts 6.83 ± 39% +2847.7% 201.43 ± 7% interrupts.CPU2.TLB:TLB_shootdowns 835.50 ± 26% +1012.7% 9296 ± 8% interrupts.CPU3.CAL:Function_call_interrupts 2370 ± 35% +64.6% 3901 interrupts.CPU3.NMI:Non-maskable_interrupts 2370 ± 35% +64.6% 3901 interrupts.CPU3.PMI:Performance_monitoring_interrupts 1461 ± 3% +3520.2% 52921 ± 4% interrupts.CPU3.RES:Rescheduling_interrupts 5.00 ± 40% +3997.1% 204.86 ± 7% interrupts.CPU3.TLB:TLB_shootdowns 1083 ± 59% +770.8% 9437 ± 6% interrupts.CPU4.CAL:Function_call_interrupts 1438 ± 2% +3735.7% 55182 ± 4% interrupts.CPU4.RES:Rescheduling_interrupts 4.83 ± 46% +4191.6% 207.43 ± 4% interrupts.CPU4.TLB:TLB_shootdowns 1532 ±100% +510.2% 9350 ± 4% interrupts.CPU5.CAL:Function_call_interrupts 1455 ± 2% +3475.4% 52052 ± 7% interrupts.CPU5.RES:Rescheduling_interrupts 5.17 ± 36% +3809.7% 202.00 ± 6% interrupts.CPU5.TLB:TLB_shootdowns 822.17 ± 34% +1052.6% 9476 ± 3% interrupts.CPU6.CAL:Function_call_interrupts 1398 +3703.0% 53172 ± 3% interrupts.CPU6.RES:Rescheduling_interrupts 3.50 ± 48% +6242.9% 222.00 ± 8% interrupts.CPU6.TLB:TLB_shootdowns 1197 ± 14% +710.7% 9706 ± 4% interrupts.CPU7.CAL:Function_call_interrupts 1605 ± 3% +3303.7% 54657 ± 6% interrupts.CPU7.RES:Rescheduling_interrupts 5.17 ± 46% +4088.9% 216.43 ± 3% interrupts.CPU7.TLB:TLB_shootdowns 2116 ± 85% +354.4% 9617 ± 6% interrupts.CPU8.CAL:Function_call_interrupts 1453 ± 3% +3569.5% 53318 ± 8% interrupts.CPU8.RES:Rescheduling_interrupts 4.67 ± 44% +4396.9% 209.86 ± 9% interrupts.CPU8.TLB:TLB_shootdowns 1808 ± 70% +403.0% 9098 ± 4% interrupts.CPU9.CAL:Function_call_interrupts 1452 +3589.8% 53581 ± 7% interrupts.CPU9.RES:Rescheduling_interrupts 5.00 ± 69% +4020.0% 206.00 ± 3% interrupts.CPU9.TLB:TLB_shootdowns 46658 ± 7% +22.5% 57148 ± 2% interrupts.NMI:Non-maskable_interrupts 46658 ± 7% +22.5% 57148 ± 2% interrupts.PMI:Performance_monitoring_interrupts 23433 +3535.9% 852037 interrupts.RES:Rescheduling_interrupts 76.33 ± 10% +4317.5% 3372 interrupts.TLB:TLB_shootdowns 0.03 ± 5% -13.8% 0.03 ± 5% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.08 ± 8% +65.6% 0.13 ± 13% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.10 ± 9% +104.2% 0.20 ± 6% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.03 ±144% -86.5% 0.00 ± 59% perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 0.11 ± 68% -87.6% 0.01 ±147% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__dentry_kill.dput 0.20 ± 31% -45.1% 0.11 ± 19% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.filemap_map_pages.do_fault 0.18 ± 37% -87.1% 0.02 ± 34% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 0.00 ±223% +4738.1% 0.15 ± 76% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 0.01 ± 52% -88.3% 0.00 ±119% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_fork.copy_process 0.01 ± 45% +1315.8% 0.20 ± 28% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_write_unlock_irq.release_task.wait_task_zombie 1.43 ± 15% -29.3% 1.01 ± 16% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.schedule_tail.ret_from_fork.[unknown] 0.15 ±139% -94.7% 0.01 ± 81% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.fs_reclaim_acquire 0.25 ±110% -99.3% 0.00 ±198% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.kmem_cache_alloc 0.02 ± 46% -65.2% 0.01 ±121% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.memcg_slab_free_hook 0.03 ±139% -95.8% 0.00 ±148% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.kmem_cache_alloc 0.04 ±146% -96.5% 0.00 ± 68% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.get_obj_cgroup_from_current.kmem_cache_alloc 0.03 ± 98% -96.5% 0.00 ±106% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.__alloc_file 0.00 ±223% +4308.2% 0.05 ±131% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__alloc_file 0.01 ±187% +2343.6% 0.22 ±118% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__d_alloc 0.01 ±196% +900.3% 0.14 ± 95% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.vm_area_alloc 0.00 ±223% +1361.7% 0.05 ± 86% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.vm_area_dup 0.01 ± 29% -63.5% 0.00 ± 39% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 0.42 ± 73% -77.3% 0.10 ± 96% perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra 0.02 ± 18% -23.9% 0.01 ± 3% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork 0.33 ±215% +1242.2% 4.44 ±109% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_read_unlock.wait_task_zombie.do_wait 0.37 ±203% +1154.6% 4.62 ± 79% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.do_wp_page.__handle_mm_fault 1.77 ± 72% +456.3% 9.84 ± 52% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.wp_page_copy.__handle_mm_fault 0.00 ±223% +74295.2% 2.23 ± 97% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 0.01 ± 45% +7769.2% 1.15 ± 83% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.delete_from_page_cache.truncate_inode_page 0.03 ± 76% -75.0% 0.01 ± 98% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_fork.copy_process 0.01 ± 46% +54592.7% 7.84 ± 41% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_write_unlock_irq.release_task.wait_task_zombie 0.04 ± 13% +4067.1% 1.85 ± 70% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.unmap_region.__do_munmap 0.00 ±142% +19789.1% 0.83 ±223% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__alloc_pages.alloc_pages_vma 0.01 ± 88% +22526.4% 1.47 ±165% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__handle_mm_fault.handle_mm_fault 3.84 ±115% +289.2% 14.94 ± 43% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.do_user_addr_fault.exc_page_fault 0.02 ±127% +12104.3% 2.01 ±133% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.29 ±184% +1499.9% 4.61 ± 27% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.handle_mm_fault.do_user_addr_fault 0.00 ±223% +4871.4% 0.06 ±123% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.get_mem_cgroup_from_mm 1.75 ±126% -88.3% 0.20 ±208% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.kmem_cache_alloc 0.00 ±141% +5568.6% 0.14 ±141% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release._copy_from_user 0.02 ±116% +7531.1% 1.74 ±116% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.rmqueue.get_page_from_freelist 0.01 ±223% +63806.9% 3.30 ±157% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.kmem_cache_free.___pmd_free_tlb.free_p4d_range 0.00 ±141% +1968.6% 0.05 ± 79% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.filemap_map_pages.do_fault 0.00 ±223% +1e+05% 1.22 ±129% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__alloc_file 0.01 ±187% +63923.9% 5.87 ±110% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__d_alloc 0.01 ±102% +3193.6% 0.26 ±165% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_fork 0.01 ±196% +24750.1% 3.52 ± 84% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.vm_area_alloc 0.00 ±223% +78405.3% 2.49 ± 99% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.vm_area_dup 0.42 ±217% +1802.0% 8.00 ± 67% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.memcg_slab_free_hook.kmem_cache_free 12.74 ± 72% -81.7% 2.34 ±140% perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra 2.94 +99.6% 5.88 ± 31% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork 0.09 ± 3% -60.2% 0.04 ± 4% perf-sched.total_sch_delay.average.ms 50.98 -85.8% 7.23 ± 3% perf-sched.total_wait_and_delay.average.ms 16846 +625.7% 122252 ± 3% perf-sched.total_wait_and_delay.count.ms 7245 -53.2% 3392 ± 28% perf-sched.total_wait_and_delay.max.ms 50.90 -85.9% 7.20 ± 3% perf-sched.total_wait_time.average.ms 7245 -53.2% 3392 ± 28% perf-sched.total_wait_time.max.ms 15.26 ± 7% -24.8% 11.47 ± 9% perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 11.02 ±110% -92.6% 0.82 ± 42% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 681.03 ± 3% -73.2% 182.59 ± 11% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 231.38 -16.0% 194.27 perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 132.26 ± 2% -98.4% 2.08 ± 4% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork 128.00 ± 12% +3429.0% 4517 ± 16% perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 139.00 ± 6% +848.6% 1318 ± 4% perf-sched.wait_and_delay.count.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.filemap_map_pages.do_fault 175.83 ± 7% +1452.4% 2729 ± 8% perf-sched.wait_and_delay.count.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 0.83 ±223% +15140.0% 127.00 ± 33% perf-sched.wait_and_delay.count.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.shmem_add_to_page_cache.shmem_getpage_gfp 26.33 ± 4% +139.8% 63.14 ± 5% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 1299 +21.2% 1573 ± 2% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork 1194 ± 2% +6800.0% 82420 ± 3% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork 15.11 ± 45% +4686.8% 723.15 ± 61% perf-sched.wait_and_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 69.89 ±223% +2717.9% 1969 ± 32% perf-sched.wait_and_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.shmem_add_to_page_cache.shmem_getpage_gfp 7245 -80.2% 1432 ± 32% perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 15.16 ± 7% -25.7% 11.27 ± 9% perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 10.99 ±110% -92.6% 0.82 ± 42% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 3.02 ± 46% +216.9% 9.58 ± 79% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_read_unlock.wait_task_zombie.do_wait 0.01 ±112% +242.9% 0.02 ± 56% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__d_lookup.lookup_fast 0.00 ±223% +3292.9% 0.07 ±133% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__pmd_alloc.__handle_mm_fault 0.00 ±223% +8.2e+05% 24.46 ±238% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__pmd_alloc.copy_page_range 0.01 ±223% +15653.9% 1.16 ± 95% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.do_sigaction.__x64_sys_rt_sigaction 0.48 ±223% +1187.2% 6.13 ± 20% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 0.04 ± 58% +428.4% 0.20 ± 73% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.__wake_up_common_lock.pipe_write 0.00 ±223% +2.6e+05% 7.31 ±131% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.release_pages.__pagevec_release 2.11 ± 54% -81.6% 0.39 ±149% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_fork.copy_process 0.54 ±223% +809.0% 4.87 ± 34% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.thread_group_cputime_adjusted.wait_task_zombie 0.00 ±223% +17141.8% 0.37 ±234% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.unix_stream_read_generic.unix_stream_recvmsg 2.45 ± 53% -94.2% 0.14 ± 66% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.kernel_init_free_pages.prep_new_page.get_page_from_freelist 2.71 ± 80% -97.4% 0.07 ± 88% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.perf_iterate_sb.perf_event_mmap.do_brk_flags 0.00 ±223% +13726.4% 0.30 ±207% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.shmem_write_end.generic_perform_write.__generic_file_write_iter 0.01 ± 92% +10275.6% 0.71 ±145% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__d_lookup_rcu.lookup_fast 0.00 ±223% +892.9% 0.02 ± 38% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__get_user_8.get_user_arg_ptr 0.00 ±223% +1422.3% 0.06 ±115% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__memcg_kmem_charge_page.__alloc_pages 0.00 ±144% +4191.6% 0.21 ± 81% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.__might_fault 0.00 ±223% +1490.5% 0.02 ± 64% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__rcu_read_unlock.memcg_slab_free_hook 0.00 ±223% +15559.3% 0.34 ±212% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.alloc_pages_vma.do_anonymous_page 0.00 ±223% +2156.0% 0.09 ± 96% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.call_rcu.task_work_run 0.00 ±223% +895.9% 0.03 ± 56% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.cgroup_throttle_swaprate.do_anonymous_page 0.00 ±223% +538.1% 0.02 ± 22% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.d_path.perf_event_mmap 0.01 ±223% +1628.2% 0.23 ± 83% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.debug_lockdep_rcu_enabled.___might_sleep 0.00 ±223% +9085.7% 0.18 ±139% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.down_read_trylock.do_user_addr_fault 0.03 ±199% +843.3% 0.28 ± 58% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.exit_to_user_mode_prepare 0.00 ±223% +836.7% 0.02 ± 24% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.fs_reclaim_release.__alloc_pages 0.01 ±223% +2452.4% 0.15 ±150% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.__d_alloc 0.02 ±223% +9193.2% 2.25 ±187% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_free.do_sys_openat2 0.00 ±150% +518.6% 0.02 ± 23% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_free.unlink_anon_vmas 0.00 ±223% +15682.1% 0.21 ±210% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.link_path_walk.part 0.00 ±223% +1210.9% 0.04 ± 79% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.load_elf_binary.exec_binprm 0.00 ±223% +1509.5% 0.02 ± 62% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__memcg_kmem_charge_page 0.00 ±223% +19029.4% 0.54 ± 70% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.aa_file_perm 0.00 ±106% +2440.8% 0.06 ±121% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.memcg_slab_free_hook 0.00 ±223% +7689.3% 0.10 ±181% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.mntput_no_expire 0.04 ±223% +4078.3% 1.57 ±158% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.xas_start 0.00 ±223% +5391.4% 0.14 ±165% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.handle_mm_fault 0.00 ±223% +31278.0% 0.68 ± 90% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.refill_obj_stock.obj_cgroup_charge 0.04 ±148% +1296.7% 0.54 ± 42% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.set_root.nd_jump_root 0.00 ±223% +91500.0% 1.83 ±117% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.step_into.walk_component 0.00 ±223% +660.0% 0.02 ± 40% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.uncharge_page.mem_cgroup_uncharge_list 0.00 ±223% +1937.4% 0.04 ± 81% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.vm_area_dup.__split_vma 9.63 ±179% -97.1% 0.28 ± 34% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.__perf_sw_event.handle_mm_fault.do_user_addr_fault 3.30 ± 44% -96.7% 0.11 ± 98% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.__alloc_pages.alloc_pages_vma 3.10 ± 78% -93.5% 0.20 ± 93% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.get_mem_cgroup_from_mm.mem_cgroup_charge 0.00 ±223% +33366.7% 0.50 ±138% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.security_file_alloc 0.00 ±223% +602.9% 0.01 ± 45% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.mntput_no_expire.terminate_walk 0.01 ±146% +1014.3% 0.08 ±120% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.prepend_path.isra 0.00 ±223% +7828.6% 0.13 ±215% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.uncharge_page.mem_cgroup_uncharge_list 0.00 ±223% +1366.7% 0.04 ±142% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.unmap_vmas.unmap_region 0.00 ±223% +29440.8% 0.69 ±154% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.up_write.anon_vma_clone 0.00 ±223% +47480.0% 0.79 ±199% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.memcg_slab_free_hook.kmem_cache_free 681.02 ± 3% -73.2% 182.59 ± 11% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 231.35 -16.0% 194.25 perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 132.25 ± 2% -98.4% 2.07 ± 4% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork 1.08 ±222% +633.8% 7.96 ± 66% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown] 0.21 ±191% +72062.8% 152.74 ±227% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.50 ± 59% +4625.6% 165.16 ±206% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_read_unlock.wait_task_zombie.do_wait 0.01 ±104% +619.5% 0.05 ± 98% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__d_lookup.lookup_fast 0.00 ±223% +17007.1% 0.68 ±166% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__pmd_alloc.__handle_mm_fault 0.00 ±223% +4.9e+06% 147.41 ±237% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__pmd_alloc.copy_page_range 16.77 ± 40% +5900.4% 1006 perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.do_anonymous_page.__handle_mm_fault 5.04 ± 66% +227.0% 16.48 ± 33% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.do_wp_page.__handle_mm_fault 0.08 ±223% +63355.5% 48.44 ±174% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.finish_fault.do_fault 0.84 ± 75% +68794.5% 577.34 ± 85% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.wp_page_copy.__handle_mm_fault 14.63 ± 47% +4838.6% 722.46 ± 61% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 0.01 ±223% +1.3e+05% 9.24 ± 62% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.do_sigaction.__x64_sys_rt_sigaction 74.80 ±206% +2533.1% 1969 ± 32% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.shmem_add_to_page_cache.shmem_getpage_gfp 0.48 ±223% +330.0% 2.04 ± 83% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.signal_setup_done.arch_do_signal_or_restart 1.09 ±223% +876.2% 10.59 ± 74% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.sigprocmask.__x64_sys_rt_sigprocmask 0.48 ±223% +4611.7% 22.44 ± 27% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 6.55 ± 55% +2355.2% 160.79 ±213% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.delete_from_page_cache.truncate_inode_page 1.40 ±223% +781.2% 12.31 ± 60% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.do_wait.kernel_wait4 2.33 ±197% +592.2% 16.16 ± 26% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.rcu_preempt_deferred_qs_irqrestore.__rcu_read_unlock 0.00 ±223% +1.1e+07% 297.91 ±149% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.release_pages.__pagevec_release 5.53 ± 50% +10419.9% 582.14 ± 83% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.release_pages.tlb_flush_mmu 0.54 ±223% +1856.2% 10.48 ± 51% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.thread_group_cputime_adjusted.wait_task_zombie 0.00 ±223% +51275.8% 1.11 ±235% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.unix_stream_read_generic.unix_stream_recvmsg 6.84 ± 45% +288.2% 26.57 ± 15% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_write_unlock_irq.release_task.wait_task_zombie 4.43 ±132% +236.2% 14.88 ± 37% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.stop_one_cpu.sched_exec 1.13 ±108% +1162.3% 14.32 ± 44% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.exit_mmap.mmput 4.43 ± 70% +21666.4% 964.76 ± 61% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.lru_cache_add.shmem_getpage_gfp.shmem_write_begin 0.01 ±143% +22411.0% 1.31 ± 84% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.schedule_tail.ret_from_fork.[unknown] 0.00 ±223% +51236.3% 1.11 ±224% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.shmem_write_end.generic_perform_write.__generic_file_write_iter 5.58 ± 74% +2113.2% 123.51 ±210% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.handle_mm_fault.do_user_addr_fault 0.00 ±223% +2078.6% 0.09 ± 71% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__get_user_8.get_user_arg_ptr 1.22 ±223% +12415.0% 152.35 ±228% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__handle_mm_fault.handle_mm_fault 0.11 ±175% +3.5e+05% 400.77 ±163% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__mem_cgroup_charge.mem_cgroup_charge 0.00 ±223% +10645.1% 0.45 ±160% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__memcg_kmem_charge_page.__alloc_pages 0.01 ±144% +64806.6% 4.44 ± 96% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.__might_fault 0.00 ±223% +2061.9% 0.06 ± 87% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__rcu_read_unlock.memcg_slab_free_hook 0.00 ±223% +31586.9% 1.32 ±217% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.alloc_pages_vma.do_anonymous_page 0.01 ±223% +12325.1% 1.01 ±111% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.call_rcu.task_work_run 0.00 ±223% +3996.4% 0.16 ±119% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.cgroup_throttle_swaprate.do_anonymous_page 0.00 ±223% +6485.7% 0.26 ±102% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.d_alloc_parallel.__lookup_slow 0.00 ±223% +2323.8% 0.07 ± 41% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.d_path.perf_event_mmap 0.02 ±223% +24760.7% 6.05 ± 84% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.debug_lockdep_rcu_enabled.___might_sleep 5.79 ± 45% +2650.3% 159.25 ±215% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.do_user_addr_fault.exc_page_fault 0.00 ±223% +38532.1% 1.55 ±163% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.down_read_trylock.do_user_addr_fault 0.08 ±214% +7364.1% 5.98 ± 65% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.exit_to_user_mode_prepare 4.99 ± 59% +2975.2% 153.54 ±226% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.exit_to_user_mode_prepare.irqentry_exit_to_user_mode 1.13 ±195% +1365.8% 16.53 ± 49% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.01 ±223% +1592.0% 0.14 ±125% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.fs_reclaim_acquire.kmem_cache_alloc 0.00 ±223% +5018.4% 0.12 ± 52% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.fs_reclaim_release.__alloc_pages 1.39 ±184% +1210.9% 18.17 ± 47% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.handle_mm_fault.do_user_addr_fault 0.00 ±223% +3631.1% 0.11 ±104% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.__anon_vma_prepare 0.02 ±223% +13668.3% 2.48 ±158% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.__d_alloc 0.01 ±186% +18694.4% 2.69 ±150% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.anon_vma_fork 0.07 ±223% +6497.8% 4.77 ± 93% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_free.do_sys_openat2 0.01 ±147% +3742.1% 0.26 ± 68% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_free.unlink_anon_vmas 2.35 ±121% +6566.9% 156.56 ±219% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.ktime_get_coarse_real_ts64.current_time 0.00 ±223% +28316.0% 1.18 ±225% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.link_path_walk.part 0.00 ±223% +4271.4% 0.12 ± 80% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.load_elf_binary.exec_binprm 0.00 ±223% +5105.5% 0.23 ±135% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__memcg_kmem_charge_page 0.79 ±175% +15699.4% 124.95 ±209% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__might_fault 0.00 ±223% +2.5e+05% 7.17 ± 71% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.aa_file_perm 0.17 ±223% +3438.4% 6.19 ± 48% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.down_read_trylock 1.37 ±170% +371.7% 6.46 ± 45% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.down_write 0.01 ±188% +27098.9% 3.54 ±121% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.get_mem_cgroup_from_objcg 1.22 ±220% +947.0% 12.73 ± 97% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.handle_mm_fault 0.01 ±117% +37573.5% 3.96 ±178% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.memcg_slab_free_hook 0.00 ±223% +55391.4% 1.39 ±206% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.mntput_no_expire 1.15 ±223% +892.5% 11.38 ± 78% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.pagecache_get_page 2.42 ±166% +452.4% 13.35 ± 60% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.thread_group_cputime 1.74 ±139% +863.3% 16.77 ± 46% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.___might_sleep 0.07 ±223% +25515.2% 19.17 ±154% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.xas_start 0.00 ±223% +1.4e+05% 3.51 ±203% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.handle_mm_fault 0.21 ±112% +3319.5% 7.11 ± 70% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.kmem_cache_alloc 0.05 ±223% +2412.6% 1.37 ±136% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.memcg_slab_free_hook 0.08 ±223% +3681.8% 3.08 ±136% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.mem_cgroup_from_task.handle_mm_fault 0.01 ±223% +1502.6% 0.15 ± 85% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.next_uptodate_page.filemap_map_pages 0.17 ±149% +3739.4% 6.53 ±153% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.refill_obj_stock.memcg_slab_free_hook 0.00 ±223% +1.9e+05% 7.75 ±101% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.refill_obj_stock.obj_cgroup_charge 0.08 ±182% +14050.5% 11.93 ± 46% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.set_root.nd_jump_root 0.00 ±223% +1.5e+05% 6.19 ±152% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.step_into.walk_component 2.32 ±120% +6531.1% 154.11 ±224% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.syscall_enter_from_user_mode.do_syscall_64 1.00 ±223% +365.8% 4.66 ± 93% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.uncharge_batch.mem_cgroup_uncharge_list 0.00 ±223% +2414.3% 0.06 ± 88% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.uncharge_page.mem_cgroup_uncharge_list 0.02 ±223% +3580.9% 0.76 ±144% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.unlink_anon_vmas.free_pgtables 0.00 ±223% +6717.6% 0.15 ±102% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.vm_area_dup.__split_vma 1.30 ±209% +559.4% 8.60 ± 73% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.vmacache_find.find_vma 0.01 ±162% +3435.1% 0.41 ±120% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.kfree.perf_event_mmap.do_brk_flags 0.00 ±223% +620.0% 0.03 ± 48% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.get_mem_cgroup_from_objcg.uncharge_page 0.01 ±141% +45293.8% 2.42 ±237% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.get_mem_cgroup_from_objcg.obj_cgroup_charge_pages 2.50 ±142% +219.9% 8.00 ± 71% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.handle_mm_fault.do_user_addr_fault 0.08 ±223% +2018.2% 1.68 ±135% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.__d_alloc 0.00 ±223% +84204.8% 2.53 ±126% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.security_file_alloc 0.00 ±223% +4519.0% 0.21 ±101% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.vm_area_alloc 0.00 ±223% +784.2% 0.03 ± 61% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.mntput_no_expire.terminate_walk 0.02 ±180% +2149.5% 0.34 ±134% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.prepend_path.isra 0.00 ±223% +29252.9% 0.98 ±232% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.uncharge_page.mem_cgroup_uncharge_list 0.00 ±223% +3247.6% 0.10 ±174% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.unmap_vmas.unmap_region 0.00 ±223% +1.3e+05% 2.98 ±163% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.up_write.anon_vma_clone 1.24 ±205% +12646.0% 157.82 ±219% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_clone 3.42 ± 93% +8491.7% 293.49 ±152% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_fork 1.01 ±222% +29628.1% 300.90 ±149% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.vm_area_dup 0.01 ±223% +3e+06% 153.73 ±225% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.memcg_slab_free_hook.kmem_cache_free 7245 -80.2% 1432 ± 32% perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 17.95 -4.6 13.35 perf-profile.calltrace.cycles-pp.sieve 6.02 ± 2% -1.6 4.38 ± 2% perf-profile.calltrace.cycles-pp.string_rtns_1 9.11 -1.2 7.86 perf-profile.calltrace.cycles-pp.page_test 8.75 -1.2 7.55 perf-profile.calltrace.cycles-pp.asm_exc_page_fault.page_test 8.58 -1.1 7.44 perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.page_test 8.44 -1.1 7.34 perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.page_test 7.42 -0.9 6.55 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.page_test 3.10 ± 3% -0.8 2.30 ± 2% perf-profile.calltrace.cycles-pp.add_int.add_int 6.73 ± 2% -0.8 5.93 perf-profile.calltrace.cycles-pp.brk 3.16 ± 4% -0.8 2.36 ± 2% perf-profile.calltrace.cycles-pp.__strncat_chk 3.05 ± 3% -0.8 2.25 ± 3% perf-profile.calltrace.cycles-pp.add_long.add_long 6.36 ± 2% -0.7 5.62 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.brk 6.34 ± 2% -0.7 5.60 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk 5.95 ± 2% -0.6 5.31 perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk 2.23 ± 2% -0.6 1.61 ± 2% perf-profile.calltrace.cycles-pp.div_long 2.50 ± 7% -0.6 1.91 ± 9% perf-profile.calltrace.cycles-pp.write 2.49 ± 7% -0.6 1.90 ± 9% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 2.48 ± 7% -0.6 1.90 ± 9% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 2.39 ± 7% -0.6 1.83 ± 9% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 2.38 ± 8% -0.6 1.82 ± 9% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 2.08 ± 9% -0.5 1.59 ± 10% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.85 ± 3% -0.5 1.37 ± 3% perf-profile.calltrace.cycles-pp.__strcat_chk 1.80 ± 5% -0.5 1.32 ± 3% perf-profile.calltrace.cycles-pp.add_short.add_short 1.15 ± 14% -0.4 0.73 ± 13% perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork 1.12 ± 14% -0.4 0.72 ± 13% perf-profile.calltrace.cycles-pp.memcpy_toio.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread 2.62 ± 9% -0.4 2.24 ± 10% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 2.72 ± 9% -0.4 2.35 ± 10% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 6.14 ± 2% -0.4 5.79 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve 6.14 ± 2% -0.4 5.79 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 6.14 ± 2% -0.3 5.79 perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 6.18 ± 2% -0.3 5.84 perf-profile.calltrace.cycles-pp.execve 6.08 ± 2% -0.3 5.74 perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 1.42 ± 10% -0.3 1.10 ± 5% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 1.64 ± 5% -0.3 1.32 ± 3% perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk 1.86 ± 10% -0.3 1.55 ± 9% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 1.77 ± 10% -0.3 1.46 ± 9% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 3.24 ± 2% -0.3 2.96 ± 9% perf-profile.calltrace.cycles-pp.lock_page_memcg.page_remove_rmap.zap_pte_range.unmap_page_range.unmap_vmas 0.74 ± 8% -0.3 0.48 ± 41% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 0.65 ± 7% -0.3 0.39 ± 63% perf-profile.calltrace.cycles-pp.shmem_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.new_sync_write 0.57 ± 5% -0.3 0.31 ± 86% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.elf_map.load_elf_binary.exec_binprm.bprm_execve 0.65 ± 7% -0.3 0.39 ± 63% perf-profile.calltrace.cycles-pp.shmem_getpage_gfp.shmem_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter 0.92 ± 10% -0.3 0.67 ± 8% perf-profile.calltrace.cycles-pp.div_int 1.00 ± 4% -0.2 0.77 ± 9% perf-profile.calltrace.cycles-pp.div_short 3.74 ± 2% -0.2 3.53 perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk 0.98 ± 6% -0.2 0.77 ± 5% perf-profile.calltrace.cycles-pp.generic_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64 0.95 ± 4% -0.2 0.75 ± 5% perf-profile.calltrace.cycles-pp.get_mem_cgroup_from_mm.mem_cgroup_charge.do_anonymous_page.__handle_mm_fault.handle_mm_fault 0.87 ± 6% -0.2 0.68 ± 6% perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.new_sync_write.vfs_write.ksys_write 1.50 ± 3% -0.2 1.33 ± 3% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.begin_new_exec 1.51 ± 3% -0.2 1.35 ± 3% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.begin_new_exec.load_elf_binary 0.92 ± 6% -0.2 0.76 ± 4% perf-profile.calltrace.cycles-pp.__xstat64 0.88 ± 6% -0.2 0.73 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__xstat64 0.88 ± 6% -0.2 0.73 ± 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64 0.85 ± 6% -0.2 0.70 ± 3% perf-profile.calltrace.cycles-pp.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64 0.82 ± 5% -0.2 0.67 ± 5% perf-profile.calltrace.cycles-pp.perf_event_mmap.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.74 ± 7% -0.1 0.60 ± 5% perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.new_sync_write.vfs_write 0.82 ± 7% -0.1 0.67 ± 4% perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64 1.52 ± 2% -0.1 1.37 ± 2% perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64 1.49 ± 2% -0.1 1.35 ± 2% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.__do_munmap.__x64_sys_brk 1.46 ± 2% -0.1 1.34 ± 2% perf-profile.calltrace.cycles-pp.zap_pte_range.unmap_page_range.unmap_vmas.unmap_region.__do_munmap 1.20 ± 2% -0.1 1.12 ± 2% perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.unmap_page_range.unmap_vmas.unmap_region 0.63 ± 3% +0.1 0.70 ± 7% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.94 ± 3% +0.1 2.07 ± 3% perf-profile.calltrace.cycles-pp.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 2.55 ± 2% +0.1 2.69 perf-profile.calltrace.cycles-pp.dup_mmap.dup_mm.copy_process.kernel_clone.__do_sys_clone 2.10 ± 3% +0.1 2.24 ± 2% perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault 0.68 ± 2% +0.1 0.82 ± 6% perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.60 ± 2% +0.2 2.75 ± 2% perf-profile.calltrace.cycles-pp.dup_mm.copy_process.kernel_clone.__do_sys_clone.do_syscall_64 0.68 ± 2% +0.2 0.83 ± 6% perf-profile.calltrace.cycles-pp.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.15 +0.2 3.31 ± 2% perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.11 ± 2% +0.2 2.31 ± 6% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit 2.12 ± 2% +0.2 2.32 ± 6% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit 2.81 +0.4 3.24 ± 4% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 2.81 +0.4 3.25 ± 4% perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 3.20 ± 2% +0.4 3.64 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.20 ± 2% +0.4 3.64 ± 4% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.20 ± 2% +0.4 3.64 ± 4% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.09 ± 4% +0.5 2.57 ± 3% perf-profile.calltrace.cycles-pp.mem_cgroup_charge.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 0.00 +0.5 0.54 ± 5% perf-profile.calltrace.cycles-pp.page_add_new_anon_rmap.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 0.00 +0.6 0.57 ± 8% perf-profile.calltrace.cycles-pp.lock_acquire._raw_spin_lock.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats 4.68 ± 2% +0.6 5.26 perf-profile.calltrace.cycles-pp.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve 0.00 +0.6 0.59 ± 7% perf-profile.calltrace.cycles-pp._raw_spin_lock.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work 4.91 ± 2% +0.6 5.51 perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.71 ± 2% +0.6 5.32 perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64 0.00 +0.7 0.66 ± 6% perf-profile.calltrace.cycles-pp.__schedule.schedule.worker_thread.kthread.ret_from_fork 0.00 +0.7 0.68 ± 6% perf-profile.calltrace.cycles-pp.schedule.worker_thread.kthread.ret_from_fork 0.00 +0.8 0.76 ± 6% perf-profile.calltrace.cycles-pp.lock_acquire.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work 3.15 ± 3% +0.9 4.01 perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.98 +1.0 7.96 ± 2% perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault 0.00 +1.0 0.98 ± 3% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +1.0 1.00 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +1.1 1.14 ± 4% perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 7.31 ± 4% +1.2 8.46 ± 7% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 7.36 ± 4% +1.2 8.53 ± 7% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 9.76 +1.2 10.93 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 8.08 ± 5% +1.5 9.59 ± 8% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 8.09 ± 5% +1.5 9.60 ± 8% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 8.09 ± 5% +1.5 9.60 ± 8% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 0.00 +1.6 1.57 ± 6% perf-profile.calltrace.cycles-pp.mem_cgroup_css_rstat_flush.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work 4.27 ± 2% +1.6 5.90 ± 4% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 8.61 ± 3% +1.7 10.27 ± 7% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 0.00 +1.7 1.69 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.10 ± 3% +2.4 6.54 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 4.55 ± 3% +2.7 7.26 ± 2% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 4.60 ± 3% +2.8 7.35 ± 2% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault 4.68 ± 3% +2.8 7.47 ± 2% perf-profile.calltrace.cycles-pp.asm_exc_page_fault 0.00 +4.1 4.07 ± 3% perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work.worker_thread 0.00 +4.1 4.12 ± 3% perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work.worker_thread.kthread 1.18 ± 14% +4.2 5.38 ± 2% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 0.00 +4.3 4.31 ± 3% perf-profile.calltrace.cycles-pp.mem_cgroup_flush_stats.process_one_work.worker_thread.kthread.ret_from_fork 9.40 +4.6 13.98 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 9.41 +4.6 13.99 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 1.19 ± 14% +5.0 6.15 ± 2% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 1.44 ± 11% +5.0 6.45 ± 2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 1.45 ± 11% +5.0 6.46 ± 2% perf-profile.calltrace.cycles-pp.ret_from_fork 17.96 -4.6 13.36 perf-profile.children.cycles-pp.sieve 6.05 ± 2% -1.6 4.40 ± 2% perf-profile.children.cycles-pp.string_rtns_1 9.36 -1.3 8.06 perf-profile.children.cycles-pp.page_test 6.75 ± 2% -0.8 5.94 perf-profile.children.cycles-pp.brk 3.10 ± 3% -0.8 2.30 ± 2% perf-profile.children.cycles-pp.add_int 3.17 ± 4% -0.8 2.36 ± 2% perf-profile.children.cycles-pp.__strncat_chk 3.05 ± 3% -0.8 2.26 ± 3% perf-profile.children.cycles-pp.add_long 2.23 ± 2% -0.6 1.61 ± 2% perf-profile.children.cycles-pp.div_long 2.51 ± 7% -0.6 1.92 ± 9% perf-profile.children.cycles-pp.write 1.86 ± 3% -0.5 1.37 ± 2% perf-profile.children.cycles-pp.__strcat_chk 1.80 ± 5% -0.5 1.32 ± 3% perf-profile.children.cycles-pp.add_short 1.15 ± 14% -0.4 0.73 ± 13% perf-profile.children.cycles-pp.drm_fb_helper_damage_work 1.14 ± 14% -0.4 0.73 ± 13% perf-profile.children.cycles-pp.memcpy_toio 6.19 ± 2% -0.3 5.84 perf-profile.children.cycles-pp.execve 0.93 ± 10% -0.3 0.67 ± 8% perf-profile.children.cycles-pp.div_int 2.19 ± 5% -0.2 1.96 ± 4% perf-profile.children.cycles-pp.__hrtimer_run_queues 1.00 ± 4% -0.2 0.77 ± 9% perf-profile.children.cycles-pp.div_short 2.93 ± 6% -0.2 2.72 ± 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 2.78 ± 5% -0.2 2.58 ± 4% perf-profile.children.cycles-pp.hrtimer_interrupt 0.93 ± 5% -0.2 0.76 ± 4% perf-profile.children.cycles-pp.__xstat64 0.44 ± 9% -0.1 0.33 ± 8% perf-profile.children.cycles-pp.semop 0.57 ± 8% -0.1 0.46 ± 4% perf-profile.children.cycles-pp.mem_rtns_1 0.24 ± 5% -0.1 0.19 ± 11% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.18 ± 11% -0.0 0.13 ± 17% perf-profile.children.cycles-pp.strnlen_user 0.21 ± 4% -0.0 0.17 ± 8% perf-profile.children.cycles-pp.unlink 0.08 ± 14% -0.0 0.04 ± 64% perf-profile.children.cycles-pp.load_balance 0.30 ± 5% -0.0 0.26 ± 4% perf-profile.children.cycles-pp.count 0.06 ± 8% +0.0 0.07 ± 6% perf-profile.children.cycles-pp.__delete_from_page_cache 0.08 ± 5% +0.0 0.10 ± 9% perf-profile.children.cycles-pp.truncate_inode_page 0.08 ± 10% +0.0 0.10 ± 6% perf-profile.children.cycles-pp.delete_from_page_cache 0.07 ± 17% +0.0 0.09 ± 9% perf-profile.children.cycles-pp.wait_task_zombie 0.23 ± 5% +0.0 0.26 ± 7% perf-profile.children.cycles-pp.__get_user_pages_remote 0.06 ± 11% +0.0 0.09 ± 10% perf-profile.children.cycles-pp.__pagevec_release 0.06 ± 11% +0.0 0.09 ± 16% perf-profile.children.cycles-pp.__tlb_remove_page_size 0.23 ± 5% +0.0 0.26 ± 6% perf-profile.children.cycles-pp.__get_user_pages 0.08 ± 16% +0.0 0.11 ± 11% perf-profile.children.cycles-pp.alloc_bprm 0.20 ± 5% +0.0 0.24 ± 4% perf-profile.children.cycles-pp.shmem_truncate_range 0.23 ± 4% +0.0 0.27 ± 6% perf-profile.children.cycles-pp.get_arg_page 0.24 ± 6% +0.0 0.27 ± 5% perf-profile.children.cycles-pp.shmem_add_to_page_cache 0.21 ± 4% +0.0 0.24 ± 5% perf-profile.children.cycles-pp.shmem_evict_inode 0.10 ± 13% +0.0 0.14 ± 7% perf-profile.children.cycles-pp.free_p4d_range 0.21 ± 4% +0.0 0.25 ± 6% perf-profile.children.cycles-pp.evict 0.20 ± 4% +0.0 0.24 ± 5% perf-profile.children.cycles-pp.shmem_undo_range 0.11 ± 14% +0.0 0.15 ± 11% perf-profile.children.cycles-pp.update_rq_clock 0.04 ± 71% +0.0 0.07 ± 9% perf-profile.children.cycles-pp._raw_spin_trylock 0.10 ± 13% +0.0 0.14 ± 7% perf-profile.children.cycles-pp.free_pgd_range 0.08 ± 20% +0.0 0.12 ± 16% perf-profile.children.cycles-pp.lock_pin_lock 0.12 ± 18% +0.0 0.16 ± 5% perf-profile.children.cycles-pp.shift_arg_pages 0.12 ± 14% +0.0 0.16 ± 13% perf-profile.children.cycles-pp.uncharge_batch 0.24 ± 5% +0.0 0.28 ± 7% perf-profile.children.cycles-pp.smpboot_thread_fn 0.02 ±141% +0.0 0.06 ± 8% perf-profile.children.cycles-pp.unaccount_page_cache_page 0.14 ± 12% +0.0 0.19 ± 6% perf-profile.children.cycles-pp.setup_arg_pages 0.19 ± 8% +0.0 0.24 ± 3% perf-profile.children.cycles-pp.__put_anon_vma 0.20 ± 10% +0.0 0.24 ± 10% perf-profile.children.cycles-pp.preempt_count_add 0.30 ± 6% +0.0 0.35 ± 6% perf-profile.children.cycles-pp.debug_smp_processor_id 0.05 ± 45% +0.1 0.10 ± 11% perf-profile.children.cycles-pp.do_raw_spin_unlock 0.02 ±141% +0.1 0.07 ± 8% perf-profile.children.cycles-pp.newidle_balance 0.00 +0.1 0.06 ± 12% perf-profile.children.cycles-pp.select_idle_sibling 0.20 ± 12% +0.1 0.26 ± 7% perf-profile.children.cycles-pp.shmem_file_read_iter 0.25 ± 4% +0.1 0.31 ± 11% perf-profile.children.cycles-pp.create_elf_tables 0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.check_preempt_curr 0.10 ± 14% +0.1 0.17 ± 7% perf-profile.children.cycles-pp.update_load_avg 0.58 ± 4% +0.1 0.66 ± 5% perf-profile.children.cycles-pp.__d_alloc 0.48 ± 5% +0.1 0.55 ± 7% perf-profile.children.cycles-pp.vm_area_alloc 0.00 +0.1 0.08 ± 8% perf-profile.children.cycles-pp.rcu_note_context_switch 0.00 +0.1 0.09 ± 21% perf-profile.children.cycles-pp.enqueue_entity 0.08 ± 9% +0.1 0.17 ± 7% perf-profile.children.cycles-pp.irqentry_exit 0.00 +0.1 0.09 ± 6% perf-profile.children.cycles-pp.preempt_schedule_irq 0.00 +0.1 0.09 ± 10% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.00 +0.1 0.09 ± 13% perf-profile.children.cycles-pp.perf_tp_event 0.00 +0.1 0.10 ± 14% perf-profile.children.cycles-pp.put_prev_entity 0.05 ± 47% +0.1 0.15 ± 12% perf-profile.children.cycles-pp.cpuacct_charge 0.00 +0.1 0.10 ± 13% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template 0.71 +0.1 0.82 ± 5% perf-profile.children.cycles-pp.rcu_do_batch 0.79 ± 2% +0.1 0.89 ± 4% perf-profile.children.cycles-pp.rcu_core 1.49 ± 3% +0.1 1.59 ± 2% perf-profile.children.cycles-pp.fs_reclaim_acquire 1.45 ± 4% +0.1 1.56 ± 2% perf-profile.children.cycles-pp.mmap_region 0.82 ± 4% +0.1 0.94 ± 3% perf-profile.children.cycles-pp.mem_cgroup_uncharge_list 1.48 ± 4% +0.1 1.60 ± 2% perf-profile.children.cycles-pp.do_mmap 1.08 ± 7% +0.1 1.20 perf-profile.children.cycles-pp.anon_vma_fork 1.62 ± 4% +0.1 1.74 ± 3% perf-profile.children.cycles-pp.vm_mmap_pgoff 0.00 +0.1 0.12 ± 12% perf-profile.children.cycles-pp.preempt_schedule_common 0.00 +0.1 0.13 ± 11% perf-profile.children.cycles-pp.preempt_schedule_thunk 1.42 ± 2% +0.1 1.55 ± 4% perf-profile.children.cycles-pp.___perf_sw_event 2.14 ± 2% +0.1 2.27 perf-profile.children.cycles-pp.mark_lock 0.00 +0.1 0.13 ± 15% perf-profile.children.cycles-pp.enqueue_task_fair 0.00 +0.1 0.14 ± 11% perf-profile.children.cycles-pp._find_next_bit 2.55 ± 2% +0.1 2.70 ± 2% perf-profile.children.cycles-pp.dup_mmap 2.60 ± 2% +0.1 2.75 ± 2% perf-profile.children.cycles-pp.dup_mm 0.81 ± 7% +0.2 0.96 ± 4% perf-profile.children.cycles-pp.unlink_anon_vmas 2.23 +0.2 2.38 ± 2% perf-profile.children.cycles-pp.trace_hardirqs_on 0.00 +0.2 0.16 ± 12% perf-profile.children.cycles-pp.ttwu_do_activate 3.16 +0.2 3.32 ± 2% perf-profile.children.cycles-pp.copy_process 3.25 +0.2 3.41 ± 2% perf-profile.children.cycles-pp.__do_sys_clone 3.25 +0.2 3.41 ± 2% perf-profile.children.cycles-pp.kernel_clone 0.10 ± 14% +0.2 0.26 ± 9% perf-profile.children.cycles-pp._raw_spin_unlock_irq 0.05 ± 46% +0.2 0.22 ± 7% perf-profile.children.cycles-pp.mem_cgroup_charge_statistics 0.07 ± 17% +0.2 0.24 ± 9% perf-profile.children.cycles-pp.pick_next_task_fair 0.00 +0.2 0.17 ± 6% perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi 1.16 ± 3% +0.2 1.33 ± 3% perf-profile.children.cycles-pp.validate_chain 1.06 ± 4% +0.2 1.24 ± 4% perf-profile.children.cycles-pp.free_pgtables 0.58 ± 9% +0.2 0.76 ± 4% perf-profile.children.cycles-pp.remove_vma 0.01 ±223% +0.2 0.20 ± 8% perf-profile.children.cycles-pp.select_task_rq_fair 0.12 ± 30% +0.2 0.31 ± 4% perf-profile.children.cycles-pp.finish_task_switch 1.74 ± 3% +0.2 1.94 ± 3% perf-profile.children.cycles-pp.rcu_read_lock_held 1.09 ± 4% +0.2 1.29 ± 2% perf-profile.children.cycles-pp.__this_cpu_preempt_check 0.36 ± 7% +0.2 0.57 ± 6% perf-profile.children.cycles-pp.lru_add_drain_cpu 0.04 ± 72% +0.2 0.25 ± 9% perf-profile.children.cycles-pp.dequeue_entity 0.06 ± 17% +0.2 0.27 ± 7% perf-profile.children.cycles-pp.dequeue_task_fair 0.52 ± 5% +0.2 0.76 ± 6% perf-profile.children.cycles-pp.lru_add_drain 0.94 ± 5% +0.2 1.19 ± 3% perf-profile.children.cycles-pp.__mem_cgroup_charge 0.07 ± 12% +0.2 0.32 ± 5% perf-profile.children.cycles-pp.schedule_idle 0.18 ± 12% +0.3 0.47 ± 8% perf-profile.children.cycles-pp.update_curr 0.78 ± 5% +0.3 1.07 ± 4% perf-profile.children.cycles-pp.__pagevec_lru_add_fn 2.67 ± 3% +0.3 2.97 ± 3% perf-profile.children.cycles-pp.mem_cgroup_charge 0.56 ± 2% +0.3 0.89 ± 4% perf-profile.children.cycles-pp.page_add_new_anon_rmap 1.43 ± 3% +0.3 1.77 ± 2% perf-profile.children.cycles-pp.lru_cache_add 0.41 ± 12% +0.3 0.75 ± 5% perf-profile.children.cycles-pp.lock_page_lruvec_irqsave 2.69 ± 2% +0.4 3.06 ± 2% perf-profile.children.cycles-pp.rcu_is_watching 2.78 ± 2% +0.4 3.16 ± 2% perf-profile.children.cycles-pp.begin_new_exec 2.59 ± 2% +0.4 2.98 ± 3% perf-profile.children.cycles-pp.rcu_lockdep_current_cpu_online 1.44 ± 4% +0.4 1.83 ± 3% perf-profile.children.cycles-pp.match_held_lock 0.45 ± 7% +0.4 0.88 ± 5% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 1.54 ± 3% +0.4 1.97 ± 2% perf-profile.children.cycles-pp.memcg_slab_free_hook 3.23 ± 3% +0.4 3.67 ± 4% perf-profile.children.cycles-pp.__x64_sys_exit_group 3.23 ± 3% +0.4 3.67 ± 4% perf-profile.children.cycles-pp.do_group_exit 3.23 ± 3% +0.4 3.67 ± 4% perf-profile.children.cycles-pp.do_exit 1.01 ± 5% +0.4 1.46 ± 3% perf-profile.children.cycles-pp._raw_spin_unlock 1.90 ± 2% +0.4 2.35 ± 2% perf-profile.children.cycles-pp.kmem_cache_free 1.62 ± 3% +0.5 2.10 ± 2% perf-profile.children.cycles-pp.release_pages 1.79 ± 2% +0.5 2.28 ± 2% perf-profile.children.cycles-pp.tlb_finish_mmu 4.90 ± 2% +0.5 5.40 perf-profile.children.cycles-pp.kmem_cache_alloc 1.76 ± 2% +0.5 2.26 ± 2% perf-profile.children.cycles-pp.tlb_flush_mmu 4.05 +0.5 4.59 ± 2% perf-profile.children.cycles-pp.page_remove_rmap 1.20 ± 3% +0.6 1.75 ± 3% perf-profile.children.cycles-pp.__pagevec_lru_add 4.68 ± 2% +0.6 5.26 perf-profile.children.cycles-pp.load_elf_binary 4.91 ± 2% +0.6 5.51 perf-profile.children.cycles-pp.bprm_execve 4.72 ± 2% +0.6 5.32 perf-profile.children.cycles-pp.exec_binprm 5.13 +0.6 5.76 ± 3% perf-profile.children.cycles-pp.zap_pte_range 5.30 +0.6 5.93 ± 3% perf-profile.children.cycles-pp.unmap_vmas 5.24 +0.6 5.87 ± 3% perf-profile.children.cycles-pp.unmap_page_range 6.09 ± 2% +0.6 6.73 perf-profile.children.cycles-pp.do_execveat_common 6.15 ± 2% +0.6 6.80 perf-profile.children.cycles-pp.__x64_sys_execve 6.59 ± 2% +0.7 7.25 perf-profile.children.cycles-pp.check_preemption_disabled 0.17 ± 21% +0.7 0.86 ± 4% perf-profile.children.cycles-pp.schedule 0.09 ± 10% +0.7 0.82 ± 4% perf-profile.children.cycles-pp.try_to_wake_up 1.76 ± 3% +0.8 2.51 ± 2% perf-profile.children.cycles-pp._raw_spin_lock 5.65 ± 2% +0.8 6.41 ± 2% perf-profile.children.cycles-pp.rcu_read_lock_held_common 5.49 +0.8 6.27 ± 2% perf-profile.children.cycles-pp.exit_mmap 5.50 +0.8 6.28 ± 2% perf-profile.children.cycles-pp.mmput 1.00 ± 3% +0.8 1.81 ± 2% perf-profile.children.cycles-pp.__mod_lruvec_page_state 3.51 ± 2% +0.9 4.41 perf-profile.children.cycles-pp.unmap_region 0.29 ± 6% +1.0 1.25 ± 4% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state 4.35 ± 3% +1.0 5.33 perf-profile.children.cycles-pp.__do_munmap 0.00 +1.0 0.98 ± 5% perf-profile.children.cycles-pp.__queue_work 7.13 +1.0 8.14 ± 2% perf-profile.children.cycles-pp.do_anonymous_page 0.00 +1.0 1.03 ± 5% perf-profile.children.cycles-pp.queue_work_on 5.95 ± 2% +1.0 7.00 perf-profile.children.cycles-pp.__x64_sys_brk 0.32 ± 9% +1.1 1.43 ± 2% perf-profile.children.cycles-pp.__schedule 0.32 ± 17% +1.1 1.43 ± 4% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 12.31 +1.2 13.53 perf-profile.children.cycles-pp.__handle_mm_fault 0.46 ± 10% +1.2 1.71 ± 3% perf-profile.children.cycles-pp.do_raw_spin_lock 0.04 ±100% +1.3 1.29 ± 4% perf-profile.children.cycles-pp.cgroup_rstat_updated 7.84 ± 3% +1.3 9.12 ± 5% perf-profile.children.cycles-pp.cpuidle_enter 7.84 ± 3% +1.3 9.12 ± 6% perf-profile.children.cycles-pp.cpuidle_enter_state 8.89 +1.4 10.25 perf-profile.children.cycles-pp.__lock_acquire 9.58 ± 2% +1.5 11.08 perf-profile.children.cycles-pp.lock_release 8.09 ± 5% +1.5 9.60 ± 8% perf-profile.children.cycles-pp.start_secondary 0.81 ± 7% +1.5 2.35 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.00 +1.6 1.58 ± 6% perf-profile.children.cycles-pp.mem_cgroup_css_rstat_flush 16.58 +1.6 18.18 perf-profile.children.cycles-pp.asm_exc_page_fault 15.80 +1.7 17.46 perf-profile.children.cycles-pp.do_user_addr_fault 8.61 ± 3% +1.7 10.27 ± 7% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 8.61 ± 3% +1.7 10.27 ± 7% perf-profile.children.cycles-pp.cpu_startup_entry 8.61 ± 3% +1.7 10.27 ± 7% perf-profile.children.cycles-pp.do_idle 14.31 +1.7 15.98 perf-profile.children.cycles-pp.handle_mm_fault 16.01 +1.7 17.68 perf-profile.children.cycles-pp.exc_page_fault 4.52 +1.7 6.22 ± 4% perf-profile.children.cycles-pp.intel_idle 9.80 +1.7 11.53 perf-profile.children.cycles-pp.lock_is_held_type 12.58 ± 2% +2.1 14.70 perf-profile.children.cycles-pp.rcu_read_lock_sched_held 30.34 +2.5 32.85 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 30.30 +2.5 32.82 perf-profile.children.cycles-pp.do_syscall_64 17.41 +2.9 20.27 perf-profile.children.cycles-pp.lock_acquire 0.00 +4.1 4.08 ± 3% perf-profile.children.cycles-pp.cgroup_rstat_flush_locked 0.00 +4.1 4.12 ± 3% perf-profile.children.cycles-pp.cgroup_rstat_flush_irqsafe 1.18 ± 14% +4.2 5.38 ± 2% perf-profile.children.cycles-pp.process_one_work 0.00 +4.3 4.31 ± 3% perf-profile.children.cycles-pp.mem_cgroup_flush_stats 1.19 ± 14% +5.0 6.15 ± 2% perf-profile.children.cycles-pp.worker_thread 1.44 ± 11% +5.0 6.45 ± 2% perf-profile.children.cycles-pp.kthread 1.49 ± 11% +5.0 6.51 ± 2% perf-profile.children.cycles-pp.ret_from_fork 17.68 -4.5 13.13 perf-profile.self.cycles-pp.sieve 5.91 ± 2% -1.6 4.31 perf-profile.self.cycles-pp.string_rtns_1 3.06 ± 3% -0.8 2.26 ± 2% perf-profile.self.cycles-pp.add_int 3.11 ± 4% -0.8 2.31 ± 2% perf-profile.self.cycles-pp.__strncat_chk 3.00 ± 3% -0.8 2.21 ± 3% perf-profile.self.cycles-pp.add_long 2.20 ± 2% -0.6 1.58 ± 2% perf-profile.self.cycles-pp.div_long 1.82 ± 3% -0.5 1.34 ± 2% perf-profile.self.cycles-pp.__strcat_chk 1.76 ± 4% -0.5 1.30 ± 3% perf-profile.self.cycles-pp.add_short 1.12 ± 14% -0.4 0.72 ± 13% perf-profile.self.cycles-pp.memcpy_toio 0.91 ± 10% -0.3 0.66 ± 8% perf-profile.self.cycles-pp.div_int 0.98 ± 4% -0.2 0.75 ± 10% perf-profile.self.cycles-pp.div_short 0.56 ± 9% -0.1 0.45 ± 5% perf-profile.self.cycles-pp.mem_rtns_1 0.32 ± 3% -0.1 0.25 ± 6% perf-profile.self.cycles-pp.page_test 0.19 ± 8% -0.0 0.15 ± 6% perf-profile.self.cycles-pp.malloc 0.06 ± 11% +0.0 0.08 ± 14% perf-profile.self.cycles-pp.__mod_lruvec_page_state 0.07 ± 18% +0.0 0.11 ± 13% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state 0.04 ± 71% +0.1 0.09 ± 14% perf-profile.self.cycles-pp.do_raw_spin_unlock 0.47 ± 6% +0.1 0.54 ± 5% perf-profile.self.cycles-pp.lockdep_hardirqs_on_prepare 0.00 +0.1 0.07 ± 12% perf-profile.self.cycles-pp.__schedule 0.04 ±100% +0.1 0.16 ± 12% perf-profile.self.cycles-pp.cgroup_rstat_updated 0.14 ± 9% +0.1 0.27 ± 5% perf-profile.self.cycles-pp.do_raw_spin_lock 2.06 ± 2% +0.1 2.19 ± 2% perf-profile.self.cycles-pp.mark_lock 0.00 +0.1 0.14 ± 9% perf-profile.self.cycles-pp._find_next_bit 1.08 ± 3% +0.1 1.22 ± 3% perf-profile.self.cycles-pp.rcu_read_lock_sched_held 1.12 ± 3% +0.2 1.30 ± 3% perf-profile.self.cycles-pp.validate_chain 1.61 ± 2% +0.2 1.83 ± 3% perf-profile.self.cycles-pp.rcu_is_watching 0.00 +0.3 0.27 ± 5% perf-profile.self.cycles-pp.cgroup_rstat_flush_locked 1.83 ± 3% +0.3 2.10 ± 4% perf-profile.self.cycles-pp.rcu_lockdep_current_cpu_online 1.16 ± 3% +0.3 1.46 ± 2% perf-profile.self.cycles-pp.match_held_lock 2.24 ± 2% +0.3 2.56 ± 2% perf-profile.self.cycles-pp.lock_acquire 2.32 ± 3% +0.5 2.86 ± 2% perf-profile.self.cycles-pp.lock_release 6.17 ± 2% +0.6 6.77 perf-profile.self.cycles-pp.check_preemption_disabled 6.33 +1.1 7.41 perf-profile.self.cycles-pp.__lock_acquire 0.32 ± 17% +1.1 1.43 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 6.20 +1.2 7.39 perf-profile.self.cycles-pp.lock_is_held_type 0.00 +1.5 1.45 ± 7% perf-profile.self.cycles-pp.mem_cgroup_css_rstat_flush 4.52 +1.7 6.22 ± 4% perf-profile.self.cycles-pp.intel_idle reaim.time.user_time 615 +---------------------------------------------------------------------+ 610 |-+ + | | : | 605 |-+ : : | 600 |-+ : : | 595 |-+ : : .+.+.+.+. .+ + + + | 590 |.+.: ++.+.+ + :.+. .+. + + :+ .+. .+. +. + +| | + + +.+ +.+.+ + +.+ + +.+.+ +.+ | 585 |-+ | 580 |-+ | 575 |-O O O OO O O O O O O O O | 570 |-+ | | O O O O | 565 |-+ O O O O O OO O O O | 560 +---------------------------------------------------------------------+ reaim.time.involuntary_context_switches 1e+06 +------------------------------------------------------------------+ 900000 |-O OO O O O OO O O O O O | | | 800000 |-+ | 700000 |-+ | | | 600000 |-+ | 500000 |-+ | 400000 |-+ | | | 300000 |-+ | 200000 |-+ | | | 100000 |.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.| 0 +------------------------------------------------------------------+ reaim.jobs_per_min 142000 +------------------------------------------------------------------+ 140000 |.+.+ .+.+. +.+. .+.+.+ | | :.+ + + : | 138000 |-+ + : .| 136000 |-+ +.+.+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+ | 134000 |-+ | 132000 |-+ | | | 130000 |-+ OO O O O OO O O | 128000 |-O O O O | 126000 |-+ | 124000 |-+ | | O O O OO O | 122000 |-+ O O O O O O O O | 120000 +------------------------------------------------------------------+ reaim.jobs_per_min_child 8800 +--------------------------------------------------------------------+ | + : + + .+.+.+.+ + | 8600 |-+ : : + : | | + : .+. .+ .+. .+. .+. .+. .+ .+. .| | + +.+ + +.+.+.+.++ + + + + + | 8400 |-+ | | | 8200 |-+ | | O | 8000 |-O O O O O O O O O OO O | | | | | 7800 |-+ | | O O OO O O OO | 7600 +--------------------------------------------------------------------+ reaim.std_dev_time 0.022 +-------------------------------------------------------------------+ | | 0.02 |-+ O O O O O O | | O O O O O O O O O O O O O O O O O O O | | O O | 0.018 |-+ | | | 0.016 |-+ | | | 0.014 |-+ | | | | | 0.012 |-+.+.+ .+.+. +.+. .+. .+. | |.+ +.+ +.+ +.+.+.++.+.+.+.+.++.+ +.+.++.+ +.+.++.+.+.| 0.01 +-------------------------------------------------------------------+ reaim.std_dev_percent 2.8 +---------------------------------------------------------------------+ | O O | 2.6 |-O O O O O O O OO O O O O O O O | | O O O O O O O O | | O | 2.4 |-+ | | | 2.2 |-+ | | | 2 |-+ | | | | .+. .+.+. .+.+.+. | 1.8 |.+ +.++ + +.+ .+. | | +.+.+.+.+.+.+.+.+.++.+.+.+.+.+ +.++.+. .+.| 1.6 +---------------------------------------------------------------------+ reaim.max_jobs_per_min 150000 +------------------------------------------------------------------+ | | | | 145000 |.+.+ +.+.+. .+.+. .+. | | :+ ++ + + | | + :.+.+. .++.+. .+. .+.+.+.+ .+.+.+. +.+.| 140000 |-+ + + + ++.+ + + | | | 135000 |-+ | | O OO O O O OO O O O O O | | | 130000 |-+ | | O O O OO O | | O O O O O O | 125000 +------------------------------------------------------------------+ perf-sched.total_wait_time.average.ms 70 +----------------------------------------------------------------------+ | .+ + | 60 |-+ + .+ : +.+.+ : + | |+ + + .+ :+ + : +. | 50 |-+ + + + +.+.+.+.++.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.| | | 40 |-+ | | | 30 |-+ | | | 20 |-+ | | | 10 |-O O O O O OO O O O O O O O O O O | | O O O O O O O O O O | 0 +----------------------------------------------------------------------+ perf-sched.total_sch_delay.average.ms 0.1 +--------------------------------------------------------------------+ | + +. .+. + + + + + | 0.09 |-+ + + : +.+ ++ + .+. .++.+ +.+. .+. :+ | |. .+.+ .+ +.+. .+. + : + + + + +.+.| 0.08 |-+ + + + : : | | :: | 0.07 |-+ + | | | 0.06 |-+ | | | 0.05 |-+ | | | 0.04 |-+ O | | O O O O O O O O O OO O O O O OO O O O O O OO O | 0.03 +--------------------------------------------------------------------+ perf-sched.total_wait_and_delay.count.ms 140000 +------------------------------------------------------------------+ | O O O O O O | 120000 |-+ O O O O O O O OO O | | O O OO O O O O O | 100000 |-+ O O | | | 80000 |-+ | | | 60000 |-+ | | | 40000 |-+ | | | 20000 |.+.++. .++. .+. .++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.| | +.+.+ +.+ + | 0 +------------------------------------------------------------------+ perf-sched.total_wait_and_delay.average.ms 70 +----------------------------------------------------------------------+ | .+ + | 60 |-+ + .+ : +.+.+ : + | |+ + + .+ :+ + : +. .+. | 50 |-+ + + + +.+.+.+.++.+.+.+ +.+.+.+.+.+.++.+.+.+.+.+.| | | 40 |-+ | | | 30 |-+ | | | 20 |-+ | | | 10 |-O O O O O OO O O O O O O O O O O | | O O O O O O O O O O | 0 +----------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork 180 +---------------------------------------------------------------------+ | .+ .+. .+. | 160 |.+ : ++.+. .+.+ + + | 140 |-+ : + + : .+. .+. .+. .| | + +.+.+.+.+.+ +.+.++.+.+ +.+.+.+.++.+ + | 120 |-+ | 100 |-+ | | | 80 |-+ | 60 |-+ | | | 40 |-+ | 20 |-+ | | | 0 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork 90000 +-------------------------------------------------------------------+ | O OO O O OO O O O | 80000 |-+ O O O O O | 70000 |-O O O O O OO O O O | | O O | 60000 |-+ | 50000 |-+ | | | 40000 |-+ | 30000 |-+ | | | 20000 |-+ | 10000 |-+ | | | 0 +-------------------------------------------------------------------+ perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork 180 +---------------------------------------------------------------------+ | .+ .+. .+. | 160 |.+ : ++.+. .+.+ + + | 140 |-+ : + + : .+. .+. .+. .| | + +.+.+.+.+.+ +.+.++.+.+ +.+.+.+.++.+ + | 120 |-+ | 100 |-+ | | | 80 |-+ | 60 |-+ | | | 40 |-+ | 20 |-+ | | | 0 +---------------------------------------------------------------------+ 6000 +--------------------------------------------------------------------+ | O O O | 5000 |-+ O O | | O O O O OO O O | | O O O O O | 4000 |-+ O | | O O O O O O | 3000 |-+ O | | O | 2000 |-+ | | | | | 1000 |-+ | | | 0 +--------------------------------------------------------------------+ 3500 +--------------------------------------------------------------------+ | O O O O O OO | 3000 |-+ O O O O O O O O O O | | O O | 2500 |-+ O O O O | | O O | 2000 |-+ | | | 1500 |-+ | | | 1000 |-+ | | | 500 |-+ | |.+.+.++.+.+.+.+.+. .++.+.+.+.+.+.++.+.+.+.+.+.++.+.+.+.+.+.+.++.+.+.| 0 +--------------------------------------------------------------------+ 1800 +--------------------------------------------------------------------+ | O | 1600 |-+ O O O | 1400 |-O O O O O O O O O O O | | O O O O OO O O O OO | 1200 |-+ O | 1000 |-+ | | | 800 |-+ | 600 |-+ | | | 400 |-+ | 200 |-+ | |.+.+.++.+.+.+.+.+.+.++.+.+.+.+.+.++.+.+.+.+.+.++.+.+.+.+.+.+.++.+.+.| 0 +--------------------------------------------------------------------+ 70 +----------------------------------------------------------------------+ 65 |-+ O O O O | | O O O O O O O O O O | 60 |-O O O O | 55 |-+ O O O O O O O O O | | | 50 |-+ | 45 |-+ | 40 |-+ | | | 35 |-+ | 30 |-+ .+. +. | |.+. .+. +. .+. .+.+. +.+. .+.+.+.+. .+ +.+ +.+. .+. .| 25 |-+ + +.+.+ +.+.+.+.+.+ + + + + + + | 20 +----------------------------------------------------------------------+ 1.4 +---------------------------------------------------------------------+ | | 1.2 |-+ | | | 1 |-+ | | | 0.8 |-+ | | | 0.6 |-+ | | | 0.4 |-+ | | | 0.2 |-+ | | O .+ O O O O | 0 +---------------------------------------------------------------------+ 1.2 +---------------------------------------------------------------------+ | O | 1 |-+ | | | | | 0.8 |-+ O | | | 0.6 |-+ | | O O | 0.4 |-+ | | | | | 0.2 |-+ | | O | 0 +---------------------------------------------------------------------+ 0.14 +--------------------------------------------------------------------+ | | 0.12 |-+ | | | 0.1 |-+ | | | 0.08 |-+ :| | :| 0.06 |-+ :| | :| 0.04 |-O :| | O O O O :| 0.02 |-+ O O O O O O O O O : | | O O O O O O O O O O O : | 0 +--------------------------------------------------------------------+ 1.2 +---------------------------------------------------------------------+ | O | 1 |-+ | | | | | 0.8 |-+ O | | | 0.6 |-+ | | | 0.4 |-+ | | | | | 0.2 |-+ | | | 0 +---------------------------------------------------------------------+ 3 +---------------------------------------------------------------------+ | O | 2.5 |-+ | | | | | 2 |-+ | | | 1.5 |-+ | | | 1 |-+ | | | | | 0.5 |-+ | | | 0 +---------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-cfl-e1: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/300s/lkp-cfl-e1/new_fserver/reaim/0xde commit: 0c7c5da68b ("memcg: switch lruvec stats to rstat") cda0d20525 ("memcg: infrastructure to flush memcg stats") 0c7c5da68b4a09ff cda0d205259a7e7c1f8e296130a ---------------- --------------------------- %stddev %change %stddev \ | \ 7.99 +1.2% 8.09 reaim.child_systime 3.26 -1.7% 3.20 reaim.child_utime 121091 -9.5% 109580 reaim.jobs_per_min 7568 -9.5% 6848 reaim.jobs_per_min_child 123789 -8.4% 113349 reaim.max_jobs_per_min 0.78 +10.6% 0.86 reaim.parent_time 1.04 +83.8% 1.91 reaim.std_dev_percent 0.01 +65.8% 0.02 reaim.std_dev_time 133153 +527.8% 835973 reaim.time.involuntary_context_switches 8602 -12.8% 7502 reaim.time.major_page_faults 16988615 -2.9% 16491812 reaim.time.minor_page_faults 398.33 -2.5% 388.29 reaim.time.percent_of_cpu_this_job_got 858.75 -1.7% 843.93 reaim.time.system_time 348.24 -4.5% 332.44 reaim.time.user_time 387134 +4.7% 405213 reaim.time.voluntary_context_switches 170666 -2.9% 165714 reaim.workload 4180544 ± 3% +75.6% 7340133 ± 24% cpuidle..usage 0.00 ± 42% +0.0 0.02 ±109% mpstat.cpu.all.iowait% 55207 ± 3% -7.3% 51204 ± 3% softirqs.CPU8.NET_RX 36.52 +3.6% 37.86 boot-time.boot 11.58 +1.5% 11.75 boot-time.dhcp 3985 +384.8% 19324 vmstat.system.cs 21914 ± 2% +33.7% 29299 ± 16% vmstat.system.in 28464 ± 2% +85.3% 52734 ± 7% meminfo.Active 28304 ± 2% +84.7% 52277 ± 8% meminfo.Active(anon) 37567 +67.6% 62971 ± 6% meminfo.Shmem 2909 ± 3% -14.5% 2489 ± 3% slabinfo.cred_jar.active_objs 2909 ± 3% -14.5% 2489 ± 3% slabinfo.cred_jar.num_objs 3454 ± 4% -10.0% 3110 ± 3% slabinfo.kmalloc-256.active_objs 3458 ± 4% -10.1% 3110 ± 3% slabinfo.kmalloc-256.num_objs 2242 ± 5% -17.4% 1853 ± 5% slabinfo.kmalloc-rcl-256.active_objs 2309 ± 5% -17.0% 1918 ± 5% slabinfo.kmalloc-rcl-256.num_objs 1264 ± 6% -18.5% 1030 ± 3% slabinfo.pid.active_objs 1264 ± 6% -18.5% 1030 ± 3% slabinfo.pid.num_objs 1323 ± 5% -18.6% 1077 ± 3% slabinfo.task_delay_info.active_objs 1323 ± 5% -18.6% 1077 ± 3% slabinfo.task_delay_info.num_objs 7105 ± 2% +84.0% 13071 ± 8% proc-vmstat.nr_active_anon 597622 +1.1% 604120 proc-vmstat.nr_file_pages 8026 +3.7% 8325 proc-vmstat.nr_mapped 9395 +67.6% 15749 ± 6% proc-vmstat.nr_shmem 7105 ± 2% +84.0% 13071 ± 8% proc-vmstat.nr_zone_active_anon 13394184 -2.6% 13042821 proc-vmstat.numa_hit 13394184 -2.6% 13042821 proc-vmstat.numa_local 63852 +24.3% 79392 ± 3% proc-vmstat.pgactivate 14095860 -2.9% 13688343 proc-vmstat.pgalloc_normal 17475839 -2.8% 16985166 proc-vmstat.pgfault 14070144 -3.0% 13646066 proc-vmstat.pgfree 1782101 -3.0% 1729290 proc-vmstat.pgreuse 77684 ± 98% +2574.2% 2077436 ± 48% turbostat.C1 0.05 ± 40% +1.7 1.75 ± 56% turbostat.C1% 341592 ± 41% +252.7% 1204679 ± 36% turbostat.C1E 0.58 ± 37% +1.2 1.75 ± 40% turbostat.C1E% 134873 ± 72% -74.8% 34048 ±131% turbostat.C8 2.65 ± 73% -2.0 0.64 ±131% turbostat.C8% 0.05 ± 20% +989.8% 0.51 ± 80% turbostat.CPU%c3 1.60 ± 80% -79.2% 0.33 ±150% turbostat.CPU%c7 37.60 +7.9% 40.56 ± 2% turbostat.CorWatt 6681937 +34.0% 8953217 ± 16% turbostat.IRQ 11.48 ± 4% -13.2% 9.96 ± 10% turbostat.Pkg%pc2 36.67 -11.0% 32.65 ± 11% turbostat.Pkg%pc3 38.03 +7.8% 41.00 ± 2% turbostat.PkgWatt 4.34e+09 +6.1% 4.604e+09 perf-stat.i.branch-instructions 26075142 +13.8% 29666013 perf-stat.i.branch-misses 2.63 ± 2% -0.4 2.24 perf-stat.i.cache-miss-rate% 1.968e+08 +24.3% 2.445e+08 perf-stat.i.cache-references 3986 +386.3% 19385 perf-stat.i.context-switches 1.18 -2.9% 1.15 perf-stat.i.cpi 1.917e+10 +4.2% 1.997e+10 perf-stat.i.cpu-cycles 533.29 +99.9% 1066 perf-stat.i.cpu-migrations 3085 ± 2% +5.1% 3243 perf-stat.i.cycles-between-cache-misses 0.06 ± 2% -0.0 0.06 perf-stat.i.dTLB-load-miss-rate% 5.264e+09 +6.1% 5.584e+09 perf-stat.i.dTLB-loads 2.789e+09 +4.0% 2.901e+09 perf-stat.i.dTLB-stores 72.39 -6.3 66.12 ± 3% perf-stat.i.iTLB-load-miss-rate% 378101 +30.7% 494297 ± 6% perf-stat.i.iTLB-loads 2.105e+10 +5.1% 2.212e+10 perf-stat.i.instructions 0.92 +2.1% 0.94 perf-stat.i.ipc 28.63 -12.8% 24.97 perf-stat.i.major-faults 1.20 +4.2% 1.25 perf-stat.i.metric.GHz 139.31 +6.4% 148.22 perf-stat.i.metric.K/sec 787.04 +5.2% 828.12 perf-stat.i.metric.M/sec 56273 -2.8% 54676 perf-stat.i.minor-faults 56302 -2.8% 54701 perf-stat.i.page-faults 9.35 +18.3% 11.06 perf-stat.overall.MPKI 0.60 +0.0 0.64 perf-stat.overall.branch-miss-rate% 2.79 -0.5 2.27 perf-stat.overall.cache-miss-rate% 3489 +3.2% 3600 perf-stat.overall.cycles-between-cache-misses 0.01 -0.0 0.01 ± 3% perf-stat.overall.dTLB-load-miss-rate% 0.01 -0.0 0.00 perf-stat.overall.dTLB-store-miss-rate% 70.11 -5.7 64.38 perf-stat.overall.iTLB-load-miss-rate% 23728 +4.6% 24813 ± 3% perf-stat.overall.instructions-per-iTLB-miss 37316899 +8.3% 40412939 perf-stat.overall.path-length 4.328e+09 +6.1% 4.591e+09 perf-stat.ps.branch-instructions 26003309 +13.8% 29583403 perf-stat.ps.branch-misses 1.962e+08 +24.3% 2.438e+08 perf-stat.ps.cache-references 3974 +386.3% 19331 perf-stat.ps.context-switches 1.911e+10 +4.2% 1.991e+10 perf-stat.ps.cpu-cycles 532.01 +99.9% 1063 perf-stat.ps.cpu-migrations 5.25e+09 +6.1% 5.568e+09 perf-stat.ps.dTLB-loads 2.781e+09 +4.0% 2.893e+09 perf-stat.ps.dTLB-stores 377084 +30.7% 492966 ± 6% perf-stat.ps.iTLB-loads 2.099e+10 +5.1% 2.206e+10 perf-stat.ps.instructions 28.55 -12.8% 24.91 perf-stat.ps.major-faults 56125 -2.8% 54533 perf-stat.ps.minor-faults 56153 -2.8% 54558 perf-stat.ps.page-faults 6.369e+12 +5.2% 6.697e+12 perf-stat.total.instructions 33001 ± 5% +333.4% 143034 ± 2% interrupts.CAL:Function_call_interrupts 2974 ± 54% +207.0% 9132 ± 7% interrupts.CPU0.CAL:Function_call_interrupts 2969 ± 2% +1363.9% 43462 ± 3% interrupts.CPU0.RES:Rescheduling_interrupts 21.83 ± 22% +1446.1% 337.57 ± 7% interrupts.CPU0.TLB:TLB_shootdowns 2239 ± 44% +300.7% 8974 ± 3% interrupts.CPU1.CAL:Function_call_interrupts 2965 ± 2% +1378.5% 43847 ± 4% interrupts.CPU1.RES:Rescheduling_interrupts 25.67 ± 14% +1200.2% 333.71 ± 5% interrupts.CPU1.TLB:TLB_shootdowns 2677 ± 53% +224.5% 8687 ± 13% interrupts.CPU10.CAL:Function_call_interrupts 2960 +1369.3% 43490 ± 3% interrupts.CPU10.RES:Rescheduling_interrupts 21.17 ± 21% +1474.6% 333.29 ± 3% interrupts.CPU10.TLB:TLB_shootdowns 2899 ± 41% +192.4% 8479 ± 9% interrupts.CPU11.CAL:Function_call_interrupts 2895 ± 29% +34.8% 3902 ± 4% interrupts.CPU11.NMI:Non-maskable_interrupts 2895 ± 29% +34.8% 3902 ± 4% interrupts.CPU11.PMI:Performance_monitoring_interrupts 2955 ± 2% +1382.3% 43803 ± 2% interrupts.CPU11.RES:Rescheduling_interrupts 23.50 ± 17% +1331.6% 336.43 ± 5% interrupts.CPU11.TLB:TLB_shootdowns 1597 ± 30% +456.0% 8882 ± 7% interrupts.CPU12.CAL:Function_call_interrupts 2949 +1365.1% 43212 ± 2% interrupts.CPU12.RES:Rescheduling_interrupts 23.33 ± 14% +1335.1% 334.86 ± 4% interrupts.CPU12.TLB:TLB_shootdowns 2284 ± 35% +316.9% 9522 ± 10% interrupts.CPU13.CAL:Function_call_interrupts 2865 ± 26% +34.1% 3841 ± 5% interrupts.CPU13.NMI:Non-maskable_interrupts 2865 ± 26% +34.1% 3841 ± 5% interrupts.CPU13.PMI:Performance_monitoring_interrupts 2947 +1376.6% 43523 ± 2% interrupts.CPU13.RES:Rescheduling_interrupts 24.50 ± 21% +1326.2% 349.43 ± 5% interrupts.CPU13.TLB:TLB_shootdowns 1620 ± 17% +433.1% 8639 ± 5% interrupts.CPU14.CAL:Function_call_interrupts 2946 ± 2% +1385.4% 43772 ± 2% interrupts.CPU14.RES:Rescheduling_interrupts 23.33 ± 18% +1385.3% 346.57 ± 7% interrupts.CPU14.TLB:TLB_shootdowns 1576 ± 20% +486.0% 9235 ± 9% interrupts.CPU15.CAL:Function_call_interrupts 2950 ± 2% +1360.2% 43077 ± 4% interrupts.CPU15.RES:Rescheduling_interrupts 22.83 ± 9% +1421.0% 347.29 ± 3% interrupts.CPU15.TLB:TLB_shootdowns 1797 ± 45% +392.3% 8846 ± 4% interrupts.CPU2.CAL:Function_call_interrupts 2909 ± 3% +1414.7% 44071 interrupts.CPU2.RES:Rescheduling_interrupts 25.17 ± 10% +1177.8% 321.57 ± 4% interrupts.CPU2.TLB:TLB_shootdowns 1569 ± 24% +473.5% 8998 ± 6% interrupts.CPU3.CAL:Function_call_interrupts 3174 ± 20% +23.1% 3908 ± 4% interrupts.CPU3.NMI:Non-maskable_interrupts 3174 ± 20% +23.1% 3908 ± 4% interrupts.CPU3.PMI:Performance_monitoring_interrupts 3013 +1343.3% 43492 interrupts.CPU3.RES:Rescheduling_interrupts 26.00 ± 14% +1180.8% 333.00 ± 6% interrupts.CPU3.TLB:TLB_shootdowns 1620 ± 13% +429.6% 8581 ± 5% interrupts.CPU4.CAL:Function_call_interrupts 2878 ± 27% +33.5% 3842 ± 4% interrupts.CPU4.NMI:Non-maskable_interrupts 2878 ± 27% +33.5% 3842 ± 4% interrupts.CPU4.PMI:Performance_monitoring_interrupts 2933 ± 2% +1398.8% 43966 ± 2% interrupts.CPU4.RES:Rescheduling_interrupts 20.67 ± 24% +1498.8% 330.43 ± 5% interrupts.CPU4.TLB:TLB_shootdowns 2286 ± 56% +280.6% 8703 ± 5% interrupts.CPU5.CAL:Function_call_interrupts 2975 ± 2% +1373.5% 43848 ± 3% interrupts.CPU5.RES:Rescheduling_interrupts 23.50 ± 8% +1334.7% 337.14 ± 8% interrupts.CPU5.TLB:TLB_shootdowns 1447 ± 15% +523.4% 9022 ± 4% interrupts.CPU6.CAL:Function_call_interrupts 2917 ± 3% +1401.5% 43802 ± 2% interrupts.CPU6.RES:Rescheduling_interrupts 22.17 ± 10% +1459.0% 345.57 ± 5% interrupts.CPU6.TLB:TLB_shootdowns 2405 ± 17% +291.3% 9412 ± 4% interrupts.CPU7.CAL:Function_call_interrupts 3094 ± 2% +1319.7% 43934 ± 2% interrupts.CPU7.RES:Rescheduling_interrupts 23.00 ± 17% +1322.4% 327.14 ± 3% interrupts.CPU7.TLB:TLB_shootdowns 1933 ± 39% +369.4% 9074 ± 7% interrupts.CPU8.CAL:Function_call_interrupts 3010 ± 2% +1374.6% 44387 ± 3% interrupts.CPU8.RES:Rescheduling_interrupts 24.17 ± 20% +1241.3% 324.14 ± 3% interrupts.CPU8.TLB:TLB_shootdowns 2072 ± 15% +326.6% 8840 ± 6% interrupts.CPU9.CAL:Function_call_interrupts 2955 +1383.1% 43828 ± 2% interrupts.CPU9.RES:Rescheduling_interrupts 25.33 ± 19% +1222.9% 335.14 ± 4% interrupts.CPU9.TLB:TLB_shootdowns 48547 ± 8% +17.3% 56951 ± 7% interrupts.NMI:Non-maskable_interrupts 48547 ± 8% +17.3% 56951 ± 7% interrupts.PMI:Performance_monitoring_interrupts 47453 +1374.1% 699520 interrupts.RES:Rescheduling_interrupts 376.17 ± 7% +1328.4% 5373 interrupts.TLB:TLB_shootdowns 5.54 ± 3% -0.9 4.59 ± 5% perf-profile.calltrace.cycles-pp.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.53 ± 3% -0.9 4.58 ± 5% perf-profile.calltrace.cycles-pp.iterate_dir.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.44 ± 3% -0.9 4.52 ± 5% perf-profile.calltrace.cycles-pp.dcache_readdir.iterate_dir.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.23 ± 3% -0.9 4.34 ± 5% perf-profile.calltrace.cycles-pp.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64.do_syscall_64 3.16 ± 10% -0.8 2.37 ± 11% perf-profile.calltrace.cycles-pp.write 3.13 ± 10% -0.8 2.35 ± 11% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 3.14 ± 10% -0.8 2.35 ± 11% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 3.04 ± 10% -0.8 2.27 ± 12% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 3.02 ± 10% -0.8 2.26 ± 12% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 2.88 ± 4% -0.6 2.30 ± 9% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.03 ± 10% -0.5 1.51 ± 6% perf-profile.calltrace.cycles-pp.div_long 2.11 ± 8% -0.5 1.63 ± 4% perf-profile.calltrace.cycles-pp.add_int.add_int 0.90 ± 13% -0.4 0.45 ± 67% perf-profile.calltrace.cycles-pp.devkmsg_write.cold.new_sync_write.vfs_write.ksys_write.do_syscall_64 0.90 ± 13% -0.4 0.45 ± 67% perf-profile.calltrace.cycles-pp.devkmsg_emit.devkmsg_write.cold.new_sync_write.vfs_write.ksys_write 0.75 ± 21% -0.4 0.31 ± 86% perf-profile.calltrace.cycles-pp.terminate_walk.path_openat.do_filp_open.do_sys_openat2.do_sys_open 0.90 ± 13% -0.4 0.45 ± 67% perf-profile.calltrace.cycles-pp.vprintk_emit.devkmsg_emit.devkmsg_write.cold.new_sync_write.vfs_write 1.77 ± 7% -0.4 1.34 ± 6% perf-profile.calltrace.cycles-pp.kill 1.76 ± 7% -0.4 1.33 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 1.76 ± 7% -0.4 1.33 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.kill 5.86 ± 3% -0.4 5.47 ± 2% perf-profile.calltrace.cycles-pp.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap.mmput 9.51 ± 2% -0.4 9.15 ± 3% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64 9.52 ± 2% -0.4 9.16 ± 3% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.68 ± 12% -0.4 0.33 ± 88% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.59 ± 7% -0.3 1.25 ± 8% perf-profile.calltrace.cycles-pp.__strcat_chk 4.13 ± 2% -0.3 3.79 ± 2% perf-profile.calltrace.cycles-pp.lock_page_memcg.page_remove_rmap.zap_pte_range.unmap_page_range.unmap_vmas 2.10 ± 5% -0.3 1.78 ± 2% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.begin_new_exec 2.11 ± 5% -0.3 1.79 ± 2% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.begin_new_exec.load_elf_binary 2.12 ± 3% -0.3 1.81 ± 6% perf-profile.calltrace.cycles-pp.dput.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64 1.19 ± 8% -0.3 0.89 ± 4% perf-profile.calltrace.cycles-pp.do_open.path_openat.do_filp_open.do_sys_openat2.do_sys_open 0.59 ± 7% -0.3 0.30 ± 86% perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 1.03 ± 12% -0.3 0.75 ± 10% perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork 1.15 ± 7% -0.3 0.87 ± 5% perf-profile.calltrace.cycles-pp.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 1.00 ± 12% -0.3 0.73 ± 10% perf-profile.calltrace.cycles-pp.memcpy_toio.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread 4.40 ± 3% -0.3 4.15 ± 2% perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap 1.06 ± 7% -0.2 0.81 ± 4% perf-profile.calltrace.cycles-pp.kill_proc_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 1.16 ± 9% -0.2 0.91 ± 3% perf-profile.calltrace.cycles-pp.add_short.add_short 1.20 ± 8% -0.2 0.95 ± 7% perf-profile.calltrace.cycles-pp.__close 1.16 ± 8% -0.2 0.92 ± 7% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 1.16 ± 8% -0.2 0.92 ± 7% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__close 0.99 ± 7% -0.2 0.76 ± 4% perf-profile.calltrace.cycles-pp.kill_pid_info.kill_proc_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.84 ± 11% -0.2 0.62 ± 10% perf-profile.calltrace.cycles-pp.div_int 0.94 ± 8% -0.2 0.75 ± 7% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 0.65 ± 5% -0.2 0.46 ± 41% perf-profile.calltrace.cycles-pp.lock_release._raw_spin_unlock.scan_positives.dcache_readdir.iterate_dir 1.21 ± 3% -0.2 1.03 ± 6% perf-profile.calltrace.cycles-pp.elf_map.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common 1.18 ± 3% -0.2 1.01 ± 8% perf-profile.calltrace.cycles-pp.lockref_put_or_lock.dput.scan_positives.dcache_readdir.iterate_dir 1.18 ± 6% -0.2 1.01 ± 5% perf-profile.calltrace.cycles-pp.copy_strings.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.40 ± 2% -0.2 2.24 ± 2% perf-profile.calltrace.cycles-pp.lock_acquire.lock_page_memcg.page_remove_rmap.zap_pte_range.unmap_page_range 0.63 ± 5% -0.2 0.48 ± 41% perf-profile.calltrace.cycles-pp.lock_acquire._raw_spin_lock_nested.scan_positives.dcache_readdir.iterate_dir 0.80 ± 10% -0.1 0.65 ± 9% perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 0.99 ± 4% -0.1 0.85 ± 6% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.elf_map.load_elf_binary.exec_binprm.bprm_execve 0.78 ± 4% -0.1 0.66 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_unlock.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64 0.89 ± 5% -0.1 0.76 ± 6% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.elf_map.load_elf_binary.exec_binprm 0.88 ± 5% -0.1 0.75 ± 6% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.elf_map.load_elf_binary 2.04 ± 2% -0.1 1.93 ± 3% perf-profile.calltrace.cycles-pp.lock_page_memcg.page_add_file_rmap.do_set_pte.filemap_map_pages.do_fault 0.73 ± 4% -0.1 0.62 ± 7% perf-profile.calltrace.cycles-pp._raw_spin_lock.lockref_put_or_lock.dput.scan_positives.dcache_readdir 0.66 ± 5% -0.1 0.56 ± 6% perf-profile.calltrace.cycles-pp.lock_acquire._raw_spin_lock.lockref_put_or_lock.dput.scan_positives 0.66 ± 5% -0.1 0.57 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_lock_nested.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64 0.69 ± 2% -0.1 0.61 ± 9% perf-profile.calltrace.cycles-pp._raw_spin_lock.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64 0.00 +0.5 0.53 ± 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.worker_thread.kthread.ret_from_fork 0.00 +0.6 0.55 ± 3% perf-profile.calltrace.cycles-pp.schedule.worker_thread.kthread.ret_from_fork 0.00 +0.7 0.74 ± 4% perf-profile.calltrace.cycles-pp.lock_acquire.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work 0.00 +1.5 1.50 ± 3% perf-profile.calltrace.cycles-pp.mem_cgroup_css_rstat_flush.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work 6.91 +1.9 8.80 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 6.96 +1.9 8.86 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 4.22 ± 2% +2.0 6.18 ± 4% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 7.66 +2.4 10.02 ± 5% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 7.66 +2.4 10.03 ± 5% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 7.66 +2.4 10.03 ± 5% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 8.19 +2.5 10.65 ± 5% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 0.00 +3.8 3.80 perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work.worker_thread 0.00 +3.8 3.85 perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work.worker_thread.kthread 1.06 ± 12% +4.0 5.08 perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 0.00 +4.1 4.06 perf-profile.calltrace.cycles-pp.mem_cgroup_flush_stats.process_one_work.worker_thread.kthread.ret_from_fork 2.17 ± 6% +4.6 6.73 ± 2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 2.18 ± 6% +4.6 6.75 ± 2% perf-profile.calltrace.cycles-pp.ret_from_fork 1.09 ± 12% +4.6 5.71 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 57.01 ± 2% -2.6 54.44 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 56.96 ± 2% -2.6 54.40 perf-profile.children.cycles-pp.do_syscall_64 5.54 ± 3% -0.9 4.59 ± 5% perf-profile.children.cycles-pp.__x64_sys_getdents64 5.53 ± 3% -0.9 4.58 ± 5% perf-profile.children.cycles-pp.iterate_dir 5.44 ± 3% -0.9 4.52 ± 5% perf-profile.children.cycles-pp.dcache_readdir 5.24 ± 3% -0.9 4.34 ± 5% perf-profile.children.cycles-pp.scan_positives 3.16 ± 10% -0.8 2.37 ± 11% perf-profile.children.cycles-pp.write 2.03 ± 10% -0.5 1.51 ± 6% perf-profile.children.cycles-pp.div_long 11.54 -0.5 11.02 ± 2% perf-profile.children.cycles-pp.do_sys_open 11.53 -0.5 11.02 ± 2% perf-profile.children.cycles-pp.do_sys_openat2 8.26 ± 2% -0.5 7.77 perf-profile.children.cycles-pp.lock_page_memcg 2.12 ± 8% -0.5 1.64 ± 4% perf-profile.children.cycles-pp.add_int 4.96 ± 2% -0.4 4.52 ± 3% perf-profile.children.cycles-pp.dput 3.23 ± 3% -0.4 2.80 ± 7% perf-profile.children.cycles-pp.vfs_write 1.77 ± 7% -0.4 1.34 ± 6% perf-profile.children.cycles-pp.kill 3.24 ± 3% -0.4 2.81 ± 7% perf-profile.children.cycles-pp.ksys_write 10.26 ± 2% -0.4 9.84 ± 2% perf-profile.children.cycles-pp.path_openat 10.28 ± 2% -0.4 9.86 ± 2% perf-profile.children.cycles-pp.do_filp_open 2.96 ± 4% -0.4 2.54 ± 8% perf-profile.children.cycles-pp.new_sync_write 6.88 ± 2% -0.4 6.50 perf-profile.children.cycles-pp.filemap_map_pages 0.90 ± 13% -0.3 0.55 ± 33% perf-profile.children.cycles-pp.devkmsg_write.cold 0.90 ± 13% -0.3 0.55 ± 33% perf-profile.children.cycles-pp.devkmsg_emit 1.59 ± 7% -0.3 1.26 ± 8% perf-profile.children.cycles-pp.__strcat_chk 7.76 ± 2% -0.3 7.46 perf-profile.children.cycles-pp.do_fault 1.03 ± 12% -0.3 0.75 ± 10% perf-profile.children.cycles-pp.drm_fb_helper_damage_work 1.02 ± 12% -0.3 0.74 ± 10% perf-profile.children.cycles-pp.memcpy_toio 4.86 ± 2% -0.3 4.58 ± 2% perf-profile.children.cycles-pp.do_set_pte 4.00 ± 2% -0.3 3.75 ± 2% perf-profile.children.cycles-pp.page_add_file_rmap 1.20 ± 8% -0.2 0.95 ± 7% perf-profile.children.cycles-pp.__close 1.16 ± 9% -0.2 0.91 ± 3% perf-profile.children.cycles-pp.add_short 2.77 ± 4% -0.2 2.53 ± 2% perf-profile.children.cycles-pp.debug_lockdep_rcu_enabled 0.84 ± 11% -0.2 0.62 ± 10% perf-profile.children.cycles-pp.div_int 1.98 ± 2% -0.2 1.78 ± 5% perf-profile.children.cycles-pp.lockref_put_or_lock 1.50 ± 6% -0.2 1.32 ± 8% perf-profile.children.cycles-pp.do_open 1.20 ± 4% -0.2 1.02 ± 10% perf-profile.children.cycles-pp.__x64_sys_kill 1.11 ± 4% -0.2 0.94 ± 10% perf-profile.children.cycles-pp.kill_proc_info 2.28 ± 3% -0.2 2.13 ± 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 1.04 ± 4% -0.2 0.88 ± 10% perf-profile.children.cycles-pp.kill_pid_info 2.29 ± 3% -0.2 2.14 ± 3% perf-profile.children.cycles-pp.fs_reclaim_acquire 0.88 ± 4% -0.1 0.73 ± 4% perf-profile.children.cycles-pp.ip_finish_output2 0.96 ± 4% -0.1 0.81 ± 4% perf-profile.children.cycles-pp.ip_output 1.87 ± 5% -0.1 1.72 ± 3% perf-profile.children.cycles-pp.__might_fault 1.14 ± 5% -0.1 1.00 ± 3% perf-profile.children.cycles-pp.sock_write_iter 1.13 ± 5% -0.1 1.00 ± 4% perf-profile.children.cycles-pp.sock_sendmsg 3.18 ± 2% -0.1 3.06 ± 2% perf-profile.children.cycles-pp.mark_lock 1.63 ± 3% -0.1 1.52 ± 5% perf-profile.children.cycles-pp.step_into 0.72 ± 4% -0.1 0.62 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_nested 0.68 ± 2% -0.1 0.58 ± 3% perf-profile.children.cycles-pp.__local_bh_enable_ip 0.58 ± 3% -0.1 0.48 ± 8% perf-profile.children.cycles-pp.do_send_sig_info 0.46 ± 11% -0.1 0.36 ± 8% perf-profile.children.cycles-pp.__x64_sys_rt_sigreturn 1.22 ± 5% -0.1 1.12 ± 3% perf-profile.children.cycles-pp.trace_hardirqs_off 0.32 ± 6% -0.1 0.23 ± 3% perf-profile.children.cycles-pp.strnlen_user 0.94 ± 3% -0.1 0.85 ± 4% perf-profile.children.cycles-pp.lockdep_hardirqs_off 1.22 ± 4% -0.1 1.14 ± 3% perf-profile.children.cycles-pp.copy_strings 0.55 ± 2% -0.1 0.46 ± 6% perf-profile.children.cycles-pp.do_softirq 0.33 ± 9% -0.1 0.24 ± 10% perf-profile.children.cycles-pp.aa_get_task_label 0.50 ± 7% -0.1 0.41 ± 6% perf-profile.children.cycles-pp.net_rx_action 0.48 ± 6% -0.1 0.40 ± 6% perf-profile.children.cycles-pp.__napi_poll 0.61 ± 5% -0.1 0.53 ± 9% perf-profile.children.cycles-pp.do_truncate 0.46 ± 5% -0.1 0.39 ± 6% perf-profile.children.cycles-pp.process_backlog 1.44 ± 4% -0.1 1.37 ± 2% perf-profile.children.cycles-pp.terminate_walk 1.50 ± 3% -0.1 1.43 ± 3% perf-profile.children.cycles-pp.__open64_nocancel 0.80 ± 3% -0.1 0.73 ± 3% perf-profile.children.cycles-pp.get_user_arg_ptr 0.36 ± 9% -0.1 0.29 ± 6% perf-profile.children.cycles-pp.kernel_wait4 0.37 ± 9% -0.1 0.30 ± 8% perf-profile.children.cycles-pp.inet_recvmsg 0.18 ± 12% -0.1 0.11 ± 15% perf-profile.children.cycles-pp.__waitpid 0.54 ± 5% -0.1 0.47 ± 6% perf-profile.children.cycles-pp.count 0.38 ± 9% -0.1 0.31 ± 9% perf-profile.children.cycles-pp.sock_read_iter 0.36 ± 9% -0.1 0.29 ± 5% perf-profile.children.cycles-pp.__do_sys_wait4 0.34 ± 9% -0.1 0.28 ± 6% perf-profile.children.cycles-pp.do_wait 0.16 ± 13% -0.1 0.10 ± 16% perf-profile.children.cycles-pp.io_serial_out 0.49 ± 5% -0.1 0.43 ± 6% perf-profile.children.cycles-pp.arch_do_signal_or_restart 0.24 ± 12% -0.1 0.18 ± 13% perf-profile.children.cycles-pp.apparmor_task_getsecid 0.24 ± 14% -0.1 0.18 ± 15% perf-profile.children.cycles-pp.tcp_recvmsg 0.31 ± 8% -0.1 0.26 ± 6% perf-profile.children.cycles-pp.ip_local_deliver 0.31 ± 9% -0.1 0.26 ± 6% perf-profile.children.cycles-pp.ip_local_deliver_finish 0.21 ± 15% -0.1 0.15 ± 14% perf-profile.children.cycles-pp.tcp_recvmsg_locked 0.34 ± 6% -0.1 0.29 ± 6% perf-profile.children.cycles-pp.__netif_receive_skb_one_core 0.22 ± 14% -0.1 0.16 ± 9% perf-profile.children.cycles-pp.sighandler 0.46 ± 4% -0.1 0.41 perf-profile.children.cycles-pp.syscall_enter_from_user_mode 0.32 ± 7% -0.1 0.27 ± 5% perf-profile.children.cycles-pp.ip_rcv 0.24 ± 6% -0.0 0.20 ± 11% perf-profile.children.cycles-pp.send_signal 0.07 ± 10% -0.0 0.04 ± 65% perf-profile.children.cycles-pp.__libc_calloc 0.17 ± 10% -0.0 0.13 ± 7% perf-profile.children.cycles-pp.dentry_needs_remove_privs 0.12 ± 13% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.compar1 0.13 ± 10% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.simple_xattr_get 0.19 ± 8% -0.0 0.16 ± 11% perf-profile.children.cycles-pp.__setup_rt_frame 0.16 ± 10% -0.0 0.13 ± 7% perf-profile.children.cycles-pp.security_inode_need_killpriv 0.16 ± 8% -0.0 0.13 ± 6% perf-profile.children.cycles-pp.cap_inode_need_killpriv 0.16 ± 8% -0.0 0.13 ± 6% perf-profile.children.cycles-pp.__vfs_getxattr 0.12 ± 7% -0.0 0.10 ± 16% perf-profile.children.cycles-pp.security_task_kill 0.12 ± 8% -0.0 0.09 ± 15% perf-profile.children.cycles-pp.apparmor_task_kill 0.16 ± 11% -0.0 0.13 ± 8% perf-profile.children.cycles-pp.__alloc_skb 0.08 ± 10% -0.0 0.06 ± 41% perf-profile.children.cycles-pp.sort_rtns_1 0.08 ± 11% -0.0 0.06 ± 5% perf-profile.children.cycles-pp.kmalloc_reserve 0.08 ± 8% -0.0 0.06 ± 5% perf-profile.children.cycles-pp.__kmalloc_node_track_caller 0.06 ± 14% +0.0 0.09 ± 11% perf-profile.children.cycles-pp.prandom_u32 0.09 ± 12% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.lock_pin_lock 0.24 ± 8% +0.0 0.27 ± 9% perf-profile.children.cycles-pp.shift_arg_pages 0.09 ± 19% +0.0 0.13 ± 12% perf-profile.children.cycles-pp.cpuacct_charge 0.01 ±223% +0.1 0.06 ± 12% perf-profile.children.cycles-pp.__pagevec_release 0.00 +0.1 0.06 ± 8% perf-profile.children.cycles-pp.select_idle_sibling 0.05 ± 46% +0.1 0.10 ± 19% perf-profile.children.cycles-pp.enqueue_entity 0.40 ± 3% +0.1 0.46 ± 4% perf-profile.children.cycles-pp.__pmd_alloc 0.35 ± 9% +0.1 0.41 ± 7% perf-profile.children.cycles-pp.__pagevec_lru_add 0.21 ± 6% +0.1 0.28 ± 5% perf-profile.children.cycles-pp.__pagevec_lru_add_fn 0.04 ± 71% +0.1 0.11 ± 10% perf-profile.children.cycles-pp.preempt_schedule_thunk 0.04 ± 71% +0.1 0.11 ± 10% perf-profile.children.cycles-pp.preempt_schedule_common 0.09 ± 10% +0.1 0.16 ± 11% perf-profile.children.cycles-pp.irqentry_exit 0.14 ± 7% +0.1 0.22 ± 10% perf-profile.children.cycles-pp.page_add_new_anon_rmap 0.11 ± 9% +0.1 0.19 ± 11% perf-profile.children.cycles-pp.update_load_avg 0.00 +0.1 0.07 ± 12% perf-profile.children.cycles-pp.rcu_note_context_switch 1.08 ± 5% +0.1 1.16 ± 4% perf-profile.children.cycles-pp.wp_page_copy 0.00 +0.1 0.08 ± 23% perf-profile.children.cycles-pp.put_prev_entity 0.00 +0.1 0.08 ± 17% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.06 ± 11% +0.1 0.14 ± 17% perf-profile.children.cycles-pp.enqueue_task_fair 0.83 ± 3% +0.1 0.92 ± 4% perf-profile.children.cycles-pp.tlb_finish_mmu 0.82 ± 3% +0.1 0.91 ± 4% perf-profile.children.cycles-pp.tlb_flush_mmu 0.00 +0.1 0.09 ± 14% perf-profile.children.cycles-pp.perf_tp_event 0.79 ± 3% +0.1 0.88 ± 4% perf-profile.children.cycles-pp.release_pages 0.00 +0.1 0.09 ± 15% perf-profile.children.cycles-pp.preempt_schedule_irq 0.38 ± 7% +0.1 0.48 ± 7% perf-profile.children.cycles-pp._raw_spin_unlock_irq 0.00 +0.1 0.11 ± 14% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template 0.04 ± 71% +0.1 0.14 ± 17% perf-profile.children.cycles-pp.ttwu_do_activate 0.12 ± 12% +0.1 0.23 ± 12% perf-profile.children.cycles-pp.pick_next_task_fair 0.09 ± 9% +0.1 0.22 ± 8% perf-profile.children.cycles-pp.dequeue_entity 0.00 +0.1 0.14 ± 15% perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi 0.11 ± 11% +0.1 0.25 ± 7% perf-profile.children.cycles-pp.dequeue_task_fair 0.06 ± 48% +0.1 0.20 ± 8% perf-profile.children.cycles-pp.select_task_rq_fair 0.01 ±223% +0.1 0.15 ± 11% perf-profile.children.cycles-pp._find_next_bit 0.22 ± 7% +0.1 0.37 ± 5% perf-profile.children.cycles-pp.finish_task_switch 1.19 ± 7% +0.2 1.34 perf-profile.children.cycles-pp.unlink_anon_vmas 0.27 ± 6% +0.2 0.43 ± 9% perf-profile.children.cycles-pp.update_curr 0.99 ± 6% +0.2 1.15 ± 3% perf-profile.children.cycles-pp.remove_vma 0.12 ± 12% +0.2 0.35 ± 5% perf-profile.children.cycles-pp.schedule_idle 0.48 ± 21% +0.3 0.81 ± 6% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.49 ± 8% +0.4 0.84 ± 4% perf-profile.children.cycles-pp.__mod_lruvec_page_state 0.99 ± 9% +0.4 1.37 ± 3% perf-profile.children.cycles-pp.do_raw_spin_lock 2.95 ± 4% +0.4 3.33 ± 3% perf-profile.children.cycles-pp.memcg_slab_free_hook 0.50 ± 11% +0.4 0.88 ± 6% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.28 ± 7% +0.5 0.76 ± 3% perf-profile.children.cycles-pp.schedule 0.16 ± 10% +0.6 0.76 ± 2% perf-profile.children.cycles-pp.try_to_wake_up 0.00 +0.8 0.81 ± 4% perf-profile.children.cycles-pp.__queue_work 0.55 ± 4% +0.8 1.37 ± 3% perf-profile.children.cycles-pp.__schedule 0.25 ± 9% +0.8 1.10 ± 4% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state 0.00 +0.8 0.85 ± 4% perf-profile.children.cycles-pp.queue_work_on 0.01 ±223% +0.9 0.90 ± 4% perf-profile.children.cycles-pp.cgroup_rstat_updated 0.57 ± 8% +0.9 1.51 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.00 +1.5 1.50 ± 3% perf-profile.children.cycles-pp.mem_cgroup_css_rstat_flush 4.51 +1.9 6.41 perf-profile.children.cycles-pp.intel_idle 7.44 +2.0 9.41 ± 3% perf-profile.children.cycles-pp.cpuidle_enter 7.44 +2.0 9.41 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state 7.66 +2.4 10.03 ± 5% perf-profile.children.cycles-pp.start_secondary 8.19 +2.5 10.65 ± 5% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 8.19 +2.5 10.65 ± 5% perf-profile.children.cycles-pp.cpu_startup_entry 8.19 +2.5 10.65 ± 5% perf-profile.children.cycles-pp.do_idle 0.00 +3.8 3.81 perf-profile.children.cycles-pp.cgroup_rstat_flush_locked 0.00 +3.8 3.85 perf-profile.children.cycles-pp.cgroup_rstat_flush_irqsafe 1.06 ± 12% +4.0 5.08 perf-profile.children.cycles-pp.process_one_work 0.00 +4.1 4.06 perf-profile.children.cycles-pp.mem_cgroup_flush_stats 2.17 ± 6% +4.6 6.73 ± 2% perf-profile.children.cycles-pp.kthread 2.27 ± 6% +4.6 6.85 ± 2% perf-profile.children.cycles-pp.ret_from_fork 1.09 ± 12% +4.6 5.71 perf-profile.children.cycles-pp.worker_thread 1.99 ± 10% -0.5 1.48 ± 6% perf-profile.self.cycles-pp.div_long 2.06 ± 8% -0.5 1.59 ± 4% perf-profile.self.cycles-pp.add_int 1.55 ± 7% -0.3 1.22 ± 8% perf-profile.self.cycles-pp.__strcat_chk 1.00 ± 12% -0.3 0.73 ± 10% perf-profile.self.cycles-pp.memcpy_toio 1.12 ± 8% -0.2 0.89 ± 3% perf-profile.self.cycles-pp.add_short 2.45 ± 4% -0.2 2.24 ± 3% perf-profile.self.cycles-pp.debug_lockdep_rcu_enabled 0.82 ± 11% -0.2 0.61 ± 10% perf-profile.self.cycles-pp.div_int 0.16 ± 13% -0.1 0.10 ± 16% perf-profile.self.cycles-pp.io_serial_out 0.11 ± 14% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.compar1 0.08 ± 11% -0.0 0.06 ± 41% perf-profile.self.cycles-pp.sort_rtns_1 0.00 +0.1 0.06 ± 16% perf-profile.self.cycles-pp.__schedule 0.01 ±223% +0.1 0.11 ± 10% perf-profile.self.cycles-pp.cgroup_rstat_updated 0.00 +0.1 0.14 ± 9% perf-profile.self.cycles-pp._find_next_bit 0.00 +0.2 0.23 ± 8% perf-profile.self.cycles-pp.cgroup_rstat_flush_locked 3.62 ± 3% +0.2 3.86 ± 2% perf-profile.self.cycles-pp.lock_release 0.47 ± 22% +0.3 0.80 ± 5% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.00 +1.4 1.39 ± 3% perf-profile.self.cycles-pp.mem_cgroup_css_rstat_flush 4.51 +1.9 6.41 perf-profile.self.cycles-pp.intel_idle 0.05 ± 7% +112.1% 0.11 ± 8% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.06 ± 7% +212.1% 0.18 ± 65% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.02 ± 22% +198.9% 0.06 ± 14% perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 0.03 ± 80% -62.9% 0.01 ± 39% perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 0.11 ± 39% +105.5% 0.23 ± 28% perf-sched.sch_delay.avg.ms.io_schedule.__lock_page_killable.filemap_fault.__do_fault 0.05 ± 48% -65.6% 0.02 ± 74% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__dentry_kill.dput 0.05 ± 65% -79.6% 0.01 ±103% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.lockref_put_or_lock.dput 0.02 ±134% -92.5% 0.00 ± 67% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.pick_file.close_fd 0.10 ± 23% -64.9% 0.04 ± 26% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 0.00 ±152% +3058.7% 0.14 ±138% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.release_task.wait_task_zombie 0.00 ±151% +852.7% 0.04 ± 45% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 0.05 ±178% -90.8% 0.00 ± 83% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.do_send_sig_info.kill_pid_info 0.00 ±223% +11900.0% 0.20 ±183% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.thread_group_cputime_adjusted.wait_task_zombie 0.02 ± 10% +708.3% 0.13 ± 28% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_write_unlock_irq.release_task.wait_task_zombie 0.22 ±172% -98.8% 0.00 ± 80% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.fd_install.do_sys_openat2.do_sys_open 0.14 ±178% -97.4% 0.00 ± 94% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.perf_iterate_sb.perf_event_mmap.__install_special_mapping 0.05 ±171% -92.6% 0.00 ± 89% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_acquire.mntput_no_expire 0.01 ± 49% -80.1% 0.00 ±158% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.d_alloc_parallel.__lookup_slow 0.06 ± 56% -98.0% 0.00 ± 98% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_free.remove_vma 0.09 ±131% -94.1% 0.01 ±108% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.ktime_get_coarse_real_ts64.current_time 0.01 ± 72% +1774.7% 0.12 ± 81% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.down_read_trylock 0.11 ±150% -92.3% 0.01 ±120% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.kmem_cache_alloc 0.18 ±137% -99.0% 0.00 ± 72% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.memcg_slab_free_hook 0.02 ±111% -95.3% 0.00 ± 75% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.obj_cgroup_charge_pages 0.11 ±116% -82.0% 0.02 ± 79% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.___might_sleep 0.02 ± 47% -86.1% 0.00 ± 81% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.refill_obj_stock.memcg_slab_free_hook 0.01 ± 48% -91.1% 0.00 ±115% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.kernel_clone.__do_sys_clone.do_syscall_64 0.14 ±203% -97.8% 0.00 ± 66% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.kmem_cache_alloc.__alloc_file 0.03 ±136% -93.7% 0.00 ± 67% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.kmem_cache_alloc.anon_vma_clone 0.34 ±213% -99.1% 0.00 ±106% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.mntput_no_expire.terminate_walk 0.21 ±136% -98.7% 0.00 ± 47% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release._copy_from_user.copy_strings 0.03 ±122% -89.4% 0.00 ±115% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.get_mem_cgroup_from_objcg.obj_cgroup_charge_pages 0.01 ± 37% -89.7% 0.00 ± 54% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.anon_vma_clone 0.05 ±184% -97.4% 0.00 ± 99% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.anon_vma_fork 0.02 ± 84% -87.8% 0.00 ± 79% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.getname_flags 0.03 ± 87% -90.9% 0.00 ± 95% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.mem_cgroup_uncharge_list.release_pages 0.14 ± 69% -93.8% 0.01 ±148% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.memcg_slab_free_hook.kmem_cache_free 0.01 ±201% +1003.1% 0.08 ± 91% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__anon_vma_prepare 0.01 ±119% +29990.8% 1.71 ±236% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_fork 0.01 ± 82% +615.4% 0.09 ±107% perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.filename_create.do_linkat.__x64_sys_link 0.02 ± 23% +2589.1% 0.46 ±211% perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2 11.95 ± 21% +48.1% 17.70 ± 18% perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 2.05 ± 44% +307.3% 8.34 ± 29% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 1.44 ± 91% +369.6% 6.79 ± 78% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 0.05 ± 94% +4699.5% 2.54 ±162% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.01 ±223% +23956.2% 1.28 ±213% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.__local_bh_enable_ip.copy_fpstate_to_sigframe.__setup_rt_frame 0.57 ± 96% +933.8% 5.91 ± 41% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_read_unlock.wait_task_zombie.do_wait 5.55 ± 26% +138.7% 13.24 ± 22% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.filemap_map_pages.do_fault 0.02 ± 56% +13714.0% 2.39 ±154% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.get_signal.arch_do_signal_or_restart 0.00 ±152% +33881.0% 1.53 ±161% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.release_task.wait_task_zombie 0.00 ±100% +22738.6% 0.76 ±183% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.ret_from_fork.[unknown] 0.02 ± 22% +15927.9% 3.31 ± 98% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.shmem_add_to_page_cache.shmem_getpage_gfp 0.01 ±141% +7238.1% 0.44 ± 78% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 0.08 ±159% +2093.7% 1.84 ±186% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.delete_from_page_cache.truncate_inode_page 0.01 ±100% +13227.7% 1.09 ±223% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.release_pages.__pagevec_release 10.04 ± 34% +75.6% 17.63 ± 24% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_move_task.do_exit 0.00 ±223% +96714.3% 1.61 ±215% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.thread_group_cputime_adjusted.wait_task_zombie 0.02 ± 14% +33245.7% 6.72 ± 69% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_write_unlock_irq.release_task.wait_task_zombie 0.00 ±223% +6237.1% 0.16 ±209% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.__pagevec_release.shmem_undo_range 0.34 ±138% +781.6% 3.03 ± 81% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.exit_mmap.mmput 0.26 ±182% -91.0% 0.02 ± 89% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.perf_iterate_sb.perf_event_mmap.__install_special_mapping 0.35 ±209% +967.7% 3.74 ± 52% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.try_to_wake_up.wake_up_q.rwsem_wake 0.04 ± 12% -14.6% 0.04 ± 9% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.vprintk_emit.devkmsg_emit.constprop 0.33 ± 97% +1104.9% 3.94 ±146% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.memcg_slab_free_hook.kmem_cache_free 0.02 ± 98% +1101.6% 0.21 ±151% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__mem_cgroup_charge.mem_cgroup_charge 0.00 ±223% +47660.0% 0.40 ±233% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.count.isra 0.07 ±208% +4562.7% 3.28 ±130% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.find_vma.do_user_addr_fault 0.11 ±164% +3912.2% 4.51 ± 72% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.handle_mm_fault.do_user_addr_fault 0.20 ±102% -90.8% 0.02 ±101% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_free.remove_vma 0.46 ± 85% +857.7% 4.38 ± 91% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__might_fault 0.01 ± 92% +35773.7% 4.42 ± 89% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.down_read_trylock 0.03 ± 94% +5361.6% 1.58 ±112% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.get_mem_cgroup_from_objcg 0.12 ±136% +638.2% 0.90 ± 83% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.mem_cgroup_from_task 0.00 ±142% +816.5% 0.02 ± 89% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.mem_cgroup_charge.wp_page_copy 0.00 ±141% +414.3% 0.02 ± 32% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.pagecache_get_page.shmem_getpage_gfp 0.19 ±209% +3391.3% 6.52 ± 79% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__alloc_file 0.01 ±187% +32718.1% 2.68 ±109% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__anon_vma_prepare 0.05 ±201% +2.7e+05% 144.31 ±242% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__d_alloc 0.00 ±141% +71174.5% 3.33 ±149% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_clone 0.01 ±108% +2.4e+06% 146.00 ±239% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_fork 0.02 ±141% +14953.1% 2.31 ±110% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.vm_area_alloc 0.01 ± 83% +27325.7% 2.74 ±103% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.vm_area_dup 0.02 ± 63% +17882.6% 4.44 ± 37% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.memcg_slab_free_hook.kmem_cache_free 0.15 ±183% +3099.4% 4.67 ±102% perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.anon_vma_clone.anon_vma_fork.dup_mmap 0.34 ±117% +1079.3% 4.02 ± 99% perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.filename_create.do_linkat.__x64_sys_link 2.37 ± 46% +6268.4% 151.16 ±229% perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2 0.63 ±209% +1026.8% 7.12 ± 82% perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.unlink_anon_vmas.free_pgtables.exit_mmap 3.11 ± 8% +1463.8% 48.56 ±217% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork 17.70 ± 20% +826.4% 163.98 ±209% perf-sched.total_sch_delay.max.ms 29.22 -73.2% 7.84 ± 8% perf-sched.total_wait_and_delay.average.ms 30290 +285.8% 116849 ± 9% perf-sched.total_wait_and_delay.count.ms 6631 -55.2% 2969 perf-sched.total_wait_and_delay.max.ms 29.17 -73.3% 7.78 ± 8% perf-sched.total_wait_time.average.ms 6631 -55.2% 2968 perf-sched.total_wait_time.max.ms 59.63 ± 13% -17.3% 49.31 ± 6% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.affine_move_task.__set_cpus_allowed_ptr 310.64 ± 21% -33.9% 205.45 ± 9% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 71.02 ± 4% -96.6% 2.42 ± 9% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork 134.33 ± 44% +1034.5% 1524 ± 12% perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 374.00 ± 3% +381.6% 1801 ± 12% perf-sched.wait_and_delay.count.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.filemap_map_pages.do_fault 382.67 ± 6% +469.6% 2179 ± 12% perf-sched.wait_and_delay.count.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 36.33 ± 5% +76.1% 64.00 ± 10% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 2283 ± 4% +3039.1% 71671 ± 10% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork 14.07 ± 30% +5035.2% 722.29 ± 60% perf-sched.wait_and_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 0.38 ± 80% +165.3% 1.02 ± 28% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.93 ±193% +8638.6% 81.15 ±153% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.__mutex_lock.perf_poll.do_sys_poll 0.00 ±223% +3392.9% 0.05 ±108% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__pte_alloc.do_anonymous_page 0.00 ±223% +2573.2% 0.07 ±125% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__pte_alloc.finish_fault 2.90 ± 23% -88.9% 0.32 ± 67% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.copy_pte_range.copy_page_range 0.33 ±157% +6283.7% 21.35 ± 34% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.do_anonymous_page.__handle_mm_fault 0.08 ±223% +1416.0% 1.15 ± 56% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.do_exit.do_group_exit 0.00 ±141% +617.3% 0.02 ± 36% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.follow_page_pte.__get_user_pages 1.23 ± 35% -83.1% 0.21 ± 46% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.pick_file.close_fd 5.03 ± 30% -75.4% 1.24 ± 83% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.__x64_sys_rt_sigreturn.do_syscall_64 2.90 ± 43% -68.5% 0.91 ± 44% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.do_sigaction.__x64_sys_rt_sigaction 6.23 ± 29% +199.3% 18.65 ± 28% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.shmem_add_to_page_cache.shmem_getpage_gfp 1.24 ±141% +367.8% 5.79 ± 18% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 6.47 ± 13% -68.7% 2.02 ± 13% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.delete_from_page_cache.truncate_inode_page 0.51 ± 96% +384.9% 2.49 ± 40% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.rcu_preempt_deferred_qs_irqrestore.__rcu_read_unlock 2.08 ± 55% -50.8% 1.02 ± 87% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_fork.copy_process 0.84 ± 16% +96.2% 1.65 ± 18% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_move_task.do_exit 1.04 ±222% +455.7% 5.80 ± 18% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.thread_group_cputime_adjusted.wait_task_zombie 2.40 ± 35% -61.2% 0.93 ± 50% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.wake_up_new_task.kernel_clone 59.62 ± 13% -17.3% 49.31 ± 6% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.affine_move_task.__set_cpus_allowed_ptr 0.00 ±223% +20410.7% 0.82 ±158% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.d_alloc_parallel.lookup_open.path_openat 0.28 ± 86% +644.7% 2.07 ±132% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.exit_mmap.mmput 0.00 ±223% +1.8e+05% 3.69 ±216% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.touch_atime.pick_link.step_into 6.29 ± 21% -82.8% 1.08 ± 53% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.vfs_write.ksys_write.do_syscall_64 4.32 ±106% -99.9% 0.00 ±244% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.__local_bh_enable_ip.arch_do_signal_or_restart 19.54 ±122% -89.4% 2.08 ± 33% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.ktime_get_coarse_real_ts64.current_time 5.89 ±176% -89.6% 0.61 ± 45% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_acquire.__might_fault 11.38 ±109% -89.2% 1.23 ± 33% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_acquire.kmem_cache_alloc 2.46 ±105% -80.6% 0.48 ±244% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_release.strncpy_from_user 0.00 ±223% +11214.3% 0.13 ±186% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.___might_sleep.kmem_cache_alloc 0.00 ±223% +2287.8% 0.03 ±151% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.___might_sleep.task_work_run 0.00 ±223% +1e+06% 22.50 ±239% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__alloc_pages.alloc_pages_vma 0.02 ±214% +557.3% 0.15 ± 91% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__mem_cgroup_charge.mem_cgroup_charge 0.06 ±167% +521.5% 0.40 ± 93% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.do_user_addr_fault 0.01 ±223% +1226.2% 0.08 ±104% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.kmem_cache_alloc 0.00 ±223% +7641.1% 0.21 ±199% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__pmd_alloc.copy_page_range 0.00 ±223% +1957.1% 0.04 ± 93% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__rcu_read_lock.get_obj_cgroup_from_current 0.00 ±150% +435.7% 0.02 ± 46% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__rcu_read_unlock.memcg_slab_free_hook 0.00 ±223% +4576.2% 0.07 ±133% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__vma_link_rb.vma_link 0.00 ±223% +2139.3% 0.03 ± 74% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi._copy_from_user.copy_strings 0.04 ±211% +2642.2% 1.19 ± 83% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.account_kernel_stack.dup_task_struct 0.00 ±223% +31985.7% 1.44 ± 85% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.acct_collect.do_exit 0.00 ±223% +3475.5% 0.04 ± 99% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.call_rcu.__dentry_kill 0.01 ±223% +9379.0% 0.54 ±111% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.check_stack_object.__check_object_size 0.00 ±158% +2526.9% 0.07 ±164% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.copy_strings.isra 0.08 ±223% +1204.5% 1.07 ± 94% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.copy_user_enhanced_fast_string.copyout 0.00 ±223% +1082.9% 0.02 ± 67% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.debug_lockdep_rcu_enabled.get_obj_cgroup_from_current 0.08 ±138% +744.4% 0.64 ± 55% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.debug_lockdep_rcu_enabled.rcu_read_lock_held_common 0.07 ± 67% +498.8% 0.45 ± 30% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.do_user_addr_fault.exc_page_fault 0.17 ±150% +399.5% 0.87 ± 36% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.exit_to_user_mode_prepare 0.11 ±140% +886.9% 1.08 ± 41% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.find_vma.do_user_addr_fault 0.00 ±223% +4042.9% 0.04 ±154% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.free_unref_page_list.release_pages 0.00 ±223% +45705.7% 1.15 ±118% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.generic_perform_write.__generic_file_write_iter 0.00 ±223% +3517.1% 0.03 ±113% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.get_mem_cgroup_from_objcg.obj_cgroup_charge_pages 0.10 ± 99% +270.0% 0.36 ± 33% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.handle_mm_fault.do_user_addr_fault 0.00 ±223% +14471.4% 0.36 ±122% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_free.do_sys_openat2 0.02 ±223% +3189.9% 0.56 ±161% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.link_path_walk.part 0.00 ±223% +4738.1% 0.07 ±180% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__task_pid_nr_ns 0.00 ±223% +34475.0% 0.46 ± 70% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.aa_file_perm 0.00 ±223% +7828.6% 0.16 ±127% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.cgroup_throttle_swaprate 0.08 ± 97% +1132.5% 0.94 ± 19% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.handle_mm_fault 0.00 ±223% +17814.3% 0.42 ± 89% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.mnt_want_write 0.00 ±223% +7185.7% 0.08 ±101% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.mntput_no_expire 0.09 ±168% +8240.6% 7.37 ±225% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.pagecache_get_page 0.06 ±198% +656.0% 0.47 ±102% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.xas_start 0.00 ±223% +20115.7% 0.67 ± 84% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.aa_file_perm 0.00 ±223% +4428.6% 0.05 ± 99% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.get_user_arg_ptr 0.03 ±156% +893.6% 0.34 ± 87% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.handle_mm_fault 0.63 ±102% -90.6% 0.06 ± 73% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.memcg_slab_free_hook.kmem_cache_free 0.00 ±223% +66634.7% 0.78 ±156% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.path_init.path_lookupat 0.01 ±223% +5980.4% 0.49 ±110% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.rcu_read_lock_held_common.rcu_read_lock_held 0.01 ±104% +4944.9% 0.29 ± 76% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.step_into.walk_component 0.11 ±223% +760.9% 0.93 ± 99% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.strncpy_from_user.getname_flags 1.57 ±141% +269.0% 5.78 ± 19% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.thread_group_cputime.thread_group_cputime_adjusted 0.07 ±223% +915.4% 0.67 ± 67% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.unmap_page_range.unmap_vmas 0.01 ±223% +7943.2% 0.50 ±123% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.unmap_vmas.exit_mmap 2.91 ± 73% -87.7% 0.36 ± 95% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.kmem_cache_alloc.vm_area_dup 0.00 ±173% +1014.3% 0.04 ± 88% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.create_elf_tables.isra 1.94 ± 53% -83.8% 0.31 ± 39% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.get_obj_cgroup_from_current.kmem_cache_alloc 4.04 ± 46% -91.6% 0.34 ± 87% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.anon_vma_clone 2.23 ± 33% -80.9% 0.43 ± 61% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.anon_vma_fork 2.36 ± 42% -81.8% 0.43 ± 73% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.vm_area_dup 0.15 ±164% +417.2% 0.78 ± 64% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.up_read.do_user_addr_fault 0.41 ±166% +1891.6% 8.25 ± 60% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__alloc_file 0.04 ±203% +39096.9% 15.81 ± 44% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__d_alloc 1.17 ±222% +1833.9% 22.67 ± 43% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_clone 2.06 ±139% +618.3% 14.83 ± 92% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_fork 2.16 ± 34% -72.0% 0.60 ± 52% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_lockdep_current_cpu_online.rcu_read_lock_held_common.rcu_read_lock_held 310.63 ± 21% -33.9% 205.44 ± 9% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 71.01 ± 4% -96.6% 2.40 ± 9% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork 1.50 ±134% +782.0% 13.23 ± 58% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.93 ±193% +32546.6% 303.18 ±147% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.__mutex_lock.perf_poll.do_sys_poll 0.57 ±222% +771.6% 4.94 ± 78% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.__mutex_lock.uprobe_clear_state.mmput 7.07 ± 48% +176.2% 19.53 ± 14% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_read_unlock.wait_task_zombie.do_wait 0.00 ±223% +7686.3% 0.32 ±155% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__pte_alloc.do_anonymous_page 0.00 ±223% +21741.1% 0.58 ±155% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__pte_alloc.finish_fault 2.40 ±124% +41821.9% 1004 perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.do_anonymous_page.__handle_mm_fault 0.08 ±223% +4129.1% 3.21 ± 41% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.do_exit.do_group_exit 0.02 ±158% +824.4% 0.18 ± 91% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.follow_page_pte.__get_user_pages 2.25 ±141% +139.2% 5.38 ± 68% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.shmem_mknod.lookup_open 3.11 ± 84% +23006.2% 718.99 ± 62% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.wp_page_copy.__handle_mm_fault 11.01 ± 26% +6440.1% 720.17 ± 61% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 1.51 ±150% +538.7% 9.66 ± 21% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.release_task.wait_task_zombie 9.42 ± 30% +13230.7% 1255 ± 21% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.shmem_add_to_page_cache.shmem_getpage_gfp 1.81 ±152% +830.7% 16.83 ± 35% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 8.13 ± 35% +140.9% 19.60 ± 39% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.delete_from_page_cache.truncate_inode_page 1.67 ±155% +419.5% 8.66 ± 42% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.do_wait.kernel_wait4 3.27 ±114% +325.8% 13.93 ± 33% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.rcu_preempt_deferred_qs_irqrestore.__rcu_read_unlock 3.40 ±100% +322.3% 14.35 ± 26% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.release_pages.__pagevec_release 2.40 ± 53% +232.2% 7.98 ± 51% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.release_pages.tlb_flush_mmu 12.67 ± 30% +53.0% 19.38 ± 15% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_move_task.do_exit 1.04 ±222% +1216.2% 13.75 ± 30% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.thread_group_cputime_adjusted.wait_task_zombie 6.34 ± 23% +2406.8% 158.86 ±216% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_write_unlock_irq.release_task.wait_task_zombie 0.00 ±223% +48803.6% 1.96 ±155% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.d_alloc_parallel.lookup_open.path_openat 1.16 ±223% +979.8% 12.49 ± 46% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.__pagevec_release.shmem_undo_range 2.32 ± 75% +6569.2% 154.86 ±222% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.exit_mmap.mmput 7.51 ± 49% +13964.7% 1056 ± 76% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.lru_cache_add.shmem_getpage_gfp.shmem_write_begin 0.00 ±223% +1.9e+05% 3.86 ±205% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.touch_atime.pick_link.step_into 1.89 ±120% +217.2% 6.01 ± 34% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.try_to_wake_up.wake_up_q.rwsem_wake 5.55 ± 80% -100.0% 0.00 ±244% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.__local_bh_enable_ip.arch_do_signal_or_restart 0.00 ±223% +62691.8% 1.47 ±194% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.___might_sleep.kmem_cache_alloc 0.00 ±223% +7173.5% 0.08 ±209% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.___might_sleep.task_work_run 0.00 ±223% +6.5e+06% 269.83 ±239% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__alloc_pages.alloc_pages_vma 0.69 ±116% +1181.7% 8.87 ± 44% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__handle_mm_fault.handle_mm_fault 0.07 ±213% +6062.5% 4.12 ± 92% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__mem_cgroup_charge.mem_cgroup_charge 0.07 ±164% +5287.8% 3.52 ± 90% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.do_user_addr_fault 0.01 ±223% +7326.2% 0.45 ±108% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.kmem_cache_alloc 1.13 ±223% +548.7% 7.35 ±110% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.pagecache_get_page 0.00 ±223% +56926.8% 1.52 ±214% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__pmd_alloc.copy_page_range 0.00 ±223% +2255.3% 0.09 ± 73% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__rcu_read_lock.get_obj_cgroup_from_current 0.01 ±149% +673.6% 0.05 ± 73% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__rcu_read_unlock.memcg_slab_free_hook 0.01 ±223% +1334.2% 0.13 ± 78% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__virt_addr_valid.__check_object_size 0.00 ±223% +19252.4% 0.29 ±163% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__vma_link_rb.vma_link 0.00 ±223% +2439.3% 0.10 ±113% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi._copy_from_user.copy_strings 0.05 ±200% +10677.3% 4.89 ± 79% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.account_kernel_stack.dup_task_struct 0.00 ±223% +1.1e+05% 4.92 ±132% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.acct_collect.do_exit 0.00 ±223% +24379.4% 1.10 ±219% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.anon_vma_interval_tree_insert.anon_vma_clone 0.00 ±223% +14153.1% 0.33 ±122% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.call_rcu.__dentry_kill 0.02 ±223% +27738.5% 4.69 ±122% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.check_stack_object.__check_object_size 0.01 ±141% +15807.9% 1.30 ±197% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.copy_strings.isra 0.08 ±223% +5409.9% 4.52 ± 87% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.copy_user_enhanced_fast_string.copyout 0.33 ±134% +1612.6% 5.58 ± 54% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.debug_lockdep_rcu_enabled.___might_sleep 0.00 ±223% +5000.0% 0.08 ±126% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.debug_lockdep_rcu_enabled.get_obj_cgroup_from_current 0.15 ±138% +3202.9% 4.97 ± 56% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.debug_lockdep_rcu_enabled.rcu_read_lock_held_common 0.53 ± 42% +2440.7% 13.42 ± 36% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.do_user_addr_fault.exc_page_fault 0.52 ±109% +1741.8% 9.54 ± 19% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.exit_to_user_mode_prepare 1.47 ±180% +779.0% 12.90 ± 39% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.34 ±163% +3235.6% 11.44 ± 25% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.find_vma.do_user_addr_fault 0.00 ±223% +13706.9% 0.58 ±215% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.free_unref_page_list.release_pages 0.00 ±223% +1.6e+05% 3.95 ±123% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.generic_perform_write.__generic_file_write_iter 0.00 ±223% +15517.1% 0.13 ±185% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.get_mem_cgroup_from_objcg.obj_cgroup_charge_pages 0.11 ±161% +1.4e+05% 145.15 ±240% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.get_obj_cgroup_from_current.kmem_cache_alloc 0.00 ±223% +8128.6% 0.19 ±135% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.handle_mm_fault.__get_user_pages 0.31 ±105% +3112.1% 9.98 ± 16% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.handle_mm_fault.do_user_addr_fault 0.22 ±135% +2566.3% 5.80 ± 55% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.__alloc_file 0.01 ±109% +7095.6% 0.71 ±151% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.security_file_alloc 0.03 ±186% +5975.7% 1.64 ±212% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.vm_area_alloc 0.17 ±181% +3002.3% 5.41 ± 62% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.vm_area_dup 0.00 ±223% +1.2e+05% 3.07 ±122% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_free.do_sys_openat2 1.52 ±180% +28630.2% 437.90 ±112% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.ktime_get_coarse_real_ts64.current_time 0.07 ±223% +10748.6% 7.36 ±154% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.link_path_walk.part 0.02 ±179% +15642.6% 2.49 ±141% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__memcg_kmem_charge_page 1.72 ±121% +740.8% 14.50 ± 80% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__might_fault 0.00 ±223% +13528.6% 0.20 ±192% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__task_pid_nr_ns 0.00 ±223% +1.6e+05% 6.69 ± 64% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.aa_file_perm 0.00 ±223% +59020.5% 2.27 ±131% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.cgroup_throttle_swaprate 0.09 ±207% +1812.0% 1.69 ± 94% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.down_read 1.48 ±156% +642.8% 11.02 ± 50% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.down_write 0.00 ±223% +20134.9% 0.91 ±232% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.filemap_get_read_batch 5.13 ± 61% +189.8% 14.88 ± 46% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.fs_reclaim_acquire 0.59 ±203% +1013.4% 6.55 ± 49% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.get_mem_cgroup_from_objcg 0.27 ± 89% +3684.7% 10.33 ± 26% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.handle_mm_fault 0.00 ±223% +3e+05% 7.02 ± 89% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.mnt_want_write 0.00 ±223% +68600.0% 1.60 ±127% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.mntput_no_expire 0.23 ±152% +63872.1% 147.14 ±236% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.pagecache_get_page 0.00 ±223% +17083.2% 0.49 ±172% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.perf_iterate_sb 5.92 ± 38% +104.5% 12.11 ± 29% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.___might_sleep 1.24 ±205% +369.3% 5.81 ± 42% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.mem_cgroup_from_task 2.56 ±108% +5832.7% 151.68 ±228% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.rcu_read_lock_held 0.08 ±204% +6275.1% 5.39 ± 97% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.xas_start 0.18 ±141% +1652.7% 3.18 ±127% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.__alloc_pages 0.00 ±223% +1.5e+05% 4.96 ± 71% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.aa_file_perm 0.13 ±223% +2434.5% 3.35 ± 96% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.get_mem_cgroup_from_mm 0.06 ±173% +4990.9% 2.84 ±128% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.get_mem_cgroup_from_objcg 0.00 ±223% +29529.7% 1.23 ±165% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.get_user_arg_ptr 0.09 ±167% +3688.7% 3.57 ± 89% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.handle_mm_fault 0.10 ±166% +1.4e+05% 144.52 ±241% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.up_read 0.30 ±140% +2334.1% 7.41 ±114% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.page_counter_try_charge.obj_cgroup_charge_pages 0.01 ±223% +18087.1% 2.15 ±148% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.path_openat.do_filp_open 0.01 ±223% +49341.1% 3.96 ±110% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.rcu_read_lock_held_common.rcu_read_lock_held 0.22 ±152% +3582.4% 8.22 ± 81% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.refill_obj_stock.obj_cgroup_charge 0.01 ±104% +94192.9% 5.66 ± 82% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.step_into.walk_component 0.11 ±223% +4235.2% 4.67 ± 86% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.strncpy_from_user.getname_flags 1.63 ±147% +693.9% 12.96 ± 31% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.syscall_enter_from_user_mode.do_syscall_64 0.49 ± 76% +1354.1% 7.06 ± 40% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.syscall_exit_to_user_mode_prepare.syscall_exit_to_user_mode 2.36 ±153% +532.7% 14.90 ± 45% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.thread_group_cputime.thread_group_cputime_adjusted 0.02 ±192% +7757.4% 1.62 ±191% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.try_charge.__mem_cgroup_charge 0.01 ±223% +5863.0% 0.51 ±216% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.uncharge_batch.mem_cgroup_uncharge_list 0.13 ±223% +2934.5% 4.02 ± 52% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.unmap_page_range.unmap_vmas 0.01 ±223% +39090.0% 2.42 ±134% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.unmap_vmas.exit_mmap 5.55 ± 36% +4888.5% 276.68 ±232% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.__perf_sw_event.handle_mm_fault.do_user_addr_fault 0.01 ±160% +4986.6% 0.54 ±201% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.kmem_cache_free.___pte_free_tlb.free_p4d_range 7.37 ± 9% +2047.6% 158.29 ±217% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.fs_reclaim_acquire.kmem_cache_alloc 1.90 ±142% +320.8% 8.00 ± 10% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.pagecache_get_page.shmem_getpage_gfp 0.00 ±150% +7446.6% 0.29 ±107% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.create_elf_tables.isra 0.02 ±120% +7077.1% 1.08 ±215% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.try_to_unlazy.path_openat 0.34 ±149% +2030.2% 7.21 ± 37% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.up_read.do_user_addr_fault 1.38 ±205% +62408.7% 862.20 ± 40% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__alloc_file 0.11 ±215% +8e+05% 861.59 ± 40% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__d_alloc 1.35 ±222% +63833.9% 863.32 ± 39% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_clone 2.09 ±136% +34318.0% 719.28 ± 62% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_fork 0.10 ±223% +1755.1% 1.94 ±162% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.vm_area_alloc 2.24 ±137% +6776.7% 153.76 ±225% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.vm_area_dup 1.02 ±139% +14761.3% 151.86 ±228% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.memcg_slab_free_hook.kmem_cache_free 0.21 ± 66% +703.0% 1.70 ±114% perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.__vma_adjust.__split_vma.mprotect_fixup 8.58 ± 26% +3400.3% 300.47 ±147% perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.anon_vma_clone.anon_vma_fork.dup_mmap *************************************************************************************************** lkp-csl-2ap2: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory ========================================================================================= compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode: gcc-9/performance/1BRD_48G/4M/btrfs/1x/x86_64-rhel-8.3/1t/debian-10.4-x86_64-20200603.cgz/fsyncBeforeClose/lkp-csl-2ap2/24G/fsmark/0x5003006 commit: 0c7c5da68b ("memcg: switch lruvec stats to rstat") cda0d20525 ("memcg: infrastructure to flush memcg stats") 0c7c5da68b4a09ff cda0d205259a7e7c1f8e296130a ---------------- --------------------------- %stddev %change %stddev \ | \ 55.18 -6.9% 51.40 fsmark.files_per_sec 120.73 +7.8% 130.20 fsmark.time.elapsed_time 120.73 +7.8% 130.20 fsmark.time.elapsed_time.max 116.29 +7.5% 125.06 fsmark.time.system_time 0.62 +0.3 0.91 mpstat.cpu.all.sys% 9117 ± 4% +154.2% 23180 ± 5% numa-meminfo.node3.Active(anon) 12157 ± 4% +118.1% 26513 ± 5% meminfo.Active(anon) 31896 +59.5% 50869 ± 2% meminfo.Shmem 2274 ± 4% +152.8% 5749 ± 5% numa-vmstat.node3.nr_active_anon 5404 ± 30% +58.1% 8546 ± 18% numa-vmstat.node3.nr_shmem 2274 ± 4% +152.8% 5749 ± 5% numa-vmstat.node3.nr_zone_active_anon 207967 -7.3% 192752 vmstat.io.bo 1.00 +100.0% 2.00 vmstat.procs.r 2604 +872.7% 25334 vmstat.system.cs 19899 ± 5% +12.2% 22332 ± 5% softirqs.CPU1.SCHED 17317 ± 5% +15.3% 19971 ± 5% softirqs.CPU100.SCHED 18092 ± 6% +14.0% 20631 ± 4% softirqs.CPU108.SCHED 32264 ± 14% -13.8% 27817 ± 17% softirqs.CPU109.RCU 18441 ± 3% +11.2% 20507 ± 4% softirqs.CPU118.SCHED 18586 ± 2% +13.6% 21117 ± 7% softirqs.CPU9.SCHED 9915 ± 57% +99.8% 19814 ± 4% softirqs.CPU96.SCHED 3039 ± 4% +118.1% 6628 ± 5% proc-vmstat.nr_active_anon 33813 -1.1% 33430 proc-vmstat.nr_active_file 9198 +9.8% 10100 proc-vmstat.nr_mapped 7974 +59.5% 12722 ± 2% proc-vmstat.nr_shmem 3039 ± 4% +118.1% 6628 ± 5% proc-vmstat.nr_zone_active_anon 33813 -1.1% 33430 proc-vmstat.nr_zone_active_file 4926 ± 5% -55.8% 2176 ± 31% proc-vmstat.numa_hint_faults 3990 ± 22% -63.0% 1475 ± 43% proc-vmstat.numa_hint_faults_local 27189 +74.6% 47479 proc-vmstat.pgactivate 666580 +4.9% 699366 proc-vmstat.pgfault 799.00 ± 21% +87.1% 1495 ± 12% slabinfo.btrfs_trans_handle.active_objs 799.00 ± 21% +87.1% 1495 ± 12% slabinfo.btrfs_trans_handle.num_objs 2044 ± 22% +118.4% 4466 ± 12% slabinfo.ext4_io_end_vec.active_objs 2044 ± 22% +118.4% 4466 ± 12% slabinfo.ext4_io_end_vec.num_objs 15776 ± 5% -7.0% 14666 ± 2% slabinfo.kmalloc-128.num_objs 3212 ± 20% +27.2% 4087 ± 6% slabinfo.posix_timers_cache.active_objs 3226 ± 19% +26.9% 4094 ± 6% slabinfo.posix_timers_cache.num_objs 1151 ± 5% -8.7% 1050 ± 3% slabinfo.task_group.active_objs 1151 ± 5% -8.7% 1050 ± 3% slabinfo.task_group.num_objs 2777 ± 6% +14.6% 3183 ± 5% slabinfo.xfs_efd_item.active_objs 2779 ± 6% +14.6% 3184 ± 5% slabinfo.xfs_efd_item.num_objs 3.698e+09 ± 5% +23.1% 4.551e+09 ± 6% perf-stat.i.branch-instructions 2464 +929.5% 25372 perf-stat.i.context-switches 1.64e+10 ± 4% +18.0% 1.936e+10 ± 8% perf-stat.i.cpu-cycles 5.176e+09 ± 5% +21.1% 6.266e+09 ± 5% perf-stat.i.dTLB-loads 1.736e+10 ± 5% +22.1% 2.121e+10 ± 6% perf-stat.i.instructions 5079 ± 10% +40.6% 7140 ± 11% perf-stat.i.instructions-per-iTLB-miss 3.83 ± 15% -15.3% 3.25 ± 6% perf-stat.i.major-faults 0.09 ± 4% +18.1% 0.10 ± 8% perf-stat.i.metric.GHz 61.30 ± 5% +21.0% 74.16 ± 5% perf-stat.i.metric.M/sec 4956 -2.5% 4835 perf-stat.i.minor-faults 4960 -2.5% 4838 perf-stat.i.page-faults 5096 ± 10% +40.5% 7158 ± 11% perf-stat.overall.instructions-per-iTLB-miss 3.666e+09 ± 5% +23.1% 4.515e+09 ± 6% perf-stat.ps.branch-instructions 2442 +930.3% 25167 perf-stat.ps.context-switches 1.626e+10 ± 4% +18.1% 1.921e+10 ± 8% perf-stat.ps.cpu-cycles 5.132e+09 ± 5% +21.1% 6.217e+09 ± 5% perf-stat.ps.dTLB-loads 1.721e+10 ± 5% +22.2% 2.104e+10 ± 6% perf-stat.ps.instructions 3.80 ± 15% -15.4% 3.22 ± 6% perf-stat.ps.major-faults 4909 -2.4% 4793 perf-stat.ps.minor-faults 4913 -2.4% 4796 perf-stat.ps.page-faults 2.086e+12 ± 4% +31.9% 2.751e+12 ± 5% perf-stat.total.instructions 525504 ± 4% +9.9% 577408 ± 2% interrupts.CAL:Function_call_interrupts 2667 ± 4% +25.3% 3341 ± 25% interrupts.CPU0.CAL:Function_call_interrupts 2533 ± 4% +128.4% 5785 ± 57% interrupts.CPU1.CAL:Function_call_interrupts 2204 ± 26% +111.5% 4663 ± 87% interrupts.CPU10.CAL:Function_call_interrupts 6.50 ± 90% +3190.1% 213.86 ±144% interrupts.CPU102.RES:Rescheduling_interrupts 215.33 ± 33% +117.8% 469.00 ± 42% interrupts.CPU108.NMI:Non-maskable_interrupts 215.33 ± 33% +117.8% 469.00 ± 42% interrupts.CPU108.PMI:Performance_monitoring_interrupts 202.00 ± 29% +56.4% 316.00 ± 32% interrupts.CPU117.NMI:Non-maskable_interrupts 202.00 ± 29% +56.4% 316.00 ± 32% interrupts.CPU117.PMI:Performance_monitoring_interrupts 222.17 ± 34% +114.0% 475.43 ± 40% interrupts.CPU12.NMI:Non-maskable_interrupts 222.17 ± 34% +114.0% 475.43 ± 40% interrupts.CPU12.PMI:Performance_monitoring_interrupts 2452 +65.9% 4069 ± 62% interrupts.CPU122.CAL:Function_call_interrupts 193.50 ± 32% +93.1% 373.57 ± 36% interrupts.CPU14.NMI:Non-maskable_interrupts 193.50 ± 32% +93.1% 373.57 ± 36% interrupts.CPU14.PMI:Performance_monitoring_interrupts 192.17 ± 38% +127.9% 438.00 ± 45% interrupts.CPU15.NMI:Non-maskable_interrupts 192.17 ± 38% +127.9% 438.00 ± 45% interrupts.CPU15.PMI:Performance_monitoring_interrupts 7.17 ±130% +6177.1% 449.86 ±129% interrupts.CPU17.RES:Rescheduling_interrupts 2472 ± 2% +56.4% 3868 ± 64% interrupts.CPU26.CAL:Function_call_interrupts 9.67 ±155% +2832.0% 283.43 ±138% interrupts.CPU3.RES:Rescheduling_interrupts 3.83 ± 96% +7223.0% 280.71 ±106% interrupts.CPU4.RES:Rescheduling_interrupts 7.50 ± 99% +4263.8% 327.29 ±105% interrupts.CPU5.RES:Rescheduling_interrupts 161.17 ± 20% +46.3% 235.86 ± 15% interrupts.CPU81.NMI:Non-maskable_interrupts 161.17 ± 20% +46.3% 235.86 ± 15% interrupts.CPU81.PMI:Performance_monitoring_interrupts 51316 ± 6% +20.1% 61616 ± 8% interrupts.NMI:Non-maskable_interrupts 51316 ± 6% +20.1% 61616 ± 8% interrupts.PMI:Performance_monitoring_interrupts 1298 ± 7% +1608.1% 22176 ± 9% interrupts.RES:Rescheduling_interrupts 0.01 ± 19% -72.9% 0.00 ± 26% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 0.02 ± 29% -68.9% 0.01 perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork 0.01 ± 20% -31.4% 0.01 ± 6% perf-sched.total_sch_delay.average.ms 220.68 -89.1% 24.00 ± 2% perf-sched.total_wait_and_delay.average.ms 11823 +833.5% 110368 perf-sched.total_wait_and_delay.count.ms 9308 ± 6% -26.6% 6835 ± 15% perf-sched.total_wait_and_delay.max.ms 220.66 -89.1% 23.99 ± 2% perf-sched.total_wait_time.average.ms 9307 ± 6% -26.6% 6835 ± 15% perf-sched.total_wait_time.max.ms 0.07 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.btrfs_start_ordered_extent.btrfs_wait_ordered_range.btrfs_sync_file.do_fsync 2.74 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 6.82 ± 28% -36.0% 4.36 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.affine_move_task.__set_cpus_allowed_ptr 4.20 ± 9% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork 108.93 ±141% +176.8% 301.52 ± 36% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 622.38 ± 23% -49.8% 312.43 ± 2% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 0.01 ± 26% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra 300.44 ± 5% -98.4% 4.68 ± 9% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork 541.33 -100.0% 0.00 perf-sched.wait_and_delay.count.btrfs_start_ordered_extent.btrfs_wait_ordered_range.btrfs_sync_file.do_fsync 306.00 -100.0% 0.00 perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 397.67 ± 10% -100.0% 0.00 perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork 21.67 ± 4% +167.7% 58.00 ± 3% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 72.00 -100.0% 0.00 perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra 1328 +7408.2% 99708 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork 0.24 ± 9% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.btrfs_start_ordered_extent.btrfs_wait_ordered_range.btrfs_sync_file.do_fsync 41.50 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 5.03 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork 0.09 ± 65% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra 8288 ± 10% -18.9% 6725 ± 15% perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork 0.06 ± 2% +18.6% 0.07 ± 2% perf-sched.wait_time.avg.ms.btrfs_start_ordered_extent.btrfs_wait_ordered_range.btrfs_sync_file.do_fsync 6.82 ± 28% -36.1% 4.36 perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.affine_move_task.__set_cpus_allowed_ptr 0.06 ± 41% -77.1% 0.01 ±158% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.stop_one_cpu.migrate_task_to 131.30 ±105% +129.6% 301.49 ± 36% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 622.37 ± 23% -49.8% 312.42 ± 2% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 300.41 ± 5% -98.4% 4.67 ± 9% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork 0.23 ± 9% +119.2% 0.50 ± 36% perf-sched.wait_time.max.ms.btrfs_start_ordered_extent.btrfs_wait_ordered_range.btrfs_sync_file.do_fsync 0.03 ±100% +297.4% 0.11 ± 24% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.__set_page_dirty_nobuffers.btrfs_dirty_pages 0.06 ± 41% -77.1% 0.01 ±158% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.stop_one_cpu.migrate_task_to 14.70 ± 5% +16.6% 17.14 ± 4% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 8288 ± 10% -18.9% 6725 ± 15% perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork 78.84 ± 2% -5.7 73.09 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 78.84 ± 2% -5.7 73.09 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 78.80 ± 2% -5.7 73.05 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 79.28 ± 2% -5.5 73.80 ± 2% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 6.75 ± 6% -0.7 6.05 ± 4% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.86 ± 19% -0.3 0.52 ± 42% perf-profile.calltrace.cycles-pp.__lock_acquire.lock_acquire.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.66 ± 8% -0.2 0.41 ± 63% perf-profile.calltrace.cycles-pp.lock_release._raw_spin_unlock_irqrestore.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.74 ± 11% -0.2 0.52 ± 41% perf-profile.calltrace.cycles-pp.btrfs_set_extent_delalloc.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 0.73 ± 10% -0.2 0.52 ± 41% perf-profile.calltrace.cycles-pp.set_extent_bit.btrfs_set_extent_delalloc.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter 1.92 ± 5% -0.2 1.73 ± 7% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.23 ± 9% -0.2 1.05 ± 9% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 1.00 ± 8% -0.2 0.83 ± 8% perf-profile.calltrace.cycles-pp.rcu_read_lock_sched_held.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 1.15 ± 8% -0.2 0.98 ± 9% perf-profile.calltrace.cycles-pp.lock_acquire._raw_spin_lock_irq.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.66 ± 10% -0.1 0.58 ± 7% perf-profile.calltrace.cycles-pp.lock_release.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt 0.00 +1.0 0.97 ± 11% perf-profile.calltrace.cycles-pp.rcu_read_lock_sched_held.lock_acquire._raw_spin_lock.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe 0.00 +1.0 1.01 ± 14% perf-profile.calltrace.cycles-pp.rcu_read_lock_sched_held.lock_release._raw_spin_unlock.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe 0.00 +1.6 1.60 ± 13% perf-profile.calltrace.cycles-pp.lock_release._raw_spin_unlock.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats 0.00 +1.8 1.85 ± 13% perf-profile.calltrace.cycles-pp._raw_spin_unlock.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work 0.00 +2.3 2.26 ± 10% perf-profile.calltrace.cycles-pp.__lock_acquire.lock_acquire._raw_spin_lock.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe 0.00 +3.7 3.74 ± 11% perf-profile.calltrace.cycles-pp.lock_acquire._raw_spin_lock.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats 0.00 +3.9 3.86 ± 11% perf-profile.calltrace.cycles-pp._raw_spin_lock.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work 0.00 +6.8 6.79 ± 10% perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work.worker_thread 0.00 +6.8 6.83 ± 10% perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work.worker_thread.kthread 0.00 +7.0 6.96 ± 10% perf-profile.calltrace.cycles-pp.mem_cgroup_flush_stats.process_one_work.worker_thread.kthread.ret_from_fork 0.00 +7.3 7.30 ± 10% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 0.00 +7.8 7.78 ± 10% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 0.00 +8.0 8.00 ± 10% perf-profile.calltrace.cycles-pp.ret_from_fork 0.00 +8.0 8.00 ± 10% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 78.84 ± 2% -5.7 73.09 ± 3% perf-profile.children.cycles-pp.start_secondary 79.28 ± 2% -5.5 73.80 ± 2% perf-profile.children.cycles-pp.do_idle 79.28 ± 2% -5.5 73.80 ± 2% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 79.28 ± 2% -5.5 73.80 ± 2% perf-profile.children.cycles-pp.cpu_startup_entry 1.16 ± 5% -0.2 0.98 ± 11% perf-profile.children.cycles-pp.lockdep_hardirqs_on_prepare 1.19 ± 8% -0.2 1.03 ± 6% perf-profile.children.cycles-pp.trace_hardirqs_on 0.74 ± 11% -0.2 0.58 ± 12% perf-profile.children.cycles-pp.btrfs_set_extent_delalloc 0.78 ± 11% -0.2 0.62 ± 16% perf-profile.children.cycles-pp.alloc_extent_state 1.41 ± 7% -0.2 1.26 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.76 ± 10% -0.2 0.61 ± 12% perf-profile.children.cycles-pp.set_extent_bit 0.66 ± 9% -0.1 0.52 ± 17% perf-profile.children.cycles-pp.lockdep_init_map_type 0.65 ± 9% -0.1 0.52 ± 17% perf-profile.children.cycles-pp.static_obj 0.60 ± 10% -0.1 0.47 ± 17% perf-profile.children.cycles-pp.__is_kernel_percpu_address 0.61 ± 11% -0.1 0.49 ± 18% perf-profile.children.cycles-pp.__init_waitqueue_head 0.34 ± 11% -0.1 0.27 ± 11% perf-profile.children.cycles-pp.hrtimer_update_next_event 0.24 ± 10% -0.1 0.17 ± 15% perf-profile.children.cycles-pp.__remove_hrtimer 0.19 ± 10% -0.1 0.13 ± 15% perf-profile.children.cycles-pp.timerqueue_del 0.46 ± 7% -0.1 0.39 ± 9% perf-profile.children.cycles-pp.trace_hardirqs_on_prepare 0.69 ± 3% -0.1 0.63 ± 4% perf-profile.children.cycles-pp.debug_smp_processor_id 0.13 ± 16% -0.0 0.09 ± 13% perf-profile.children.cycles-pp.menu_reflect 0.39 ± 5% -0.0 0.35 ± 6% perf-profile.children.cycles-pp.lockdep_hardirqs_on 0.16 ± 9% -0.0 0.12 ± 17% perf-profile.children.cycles-pp.btrfs_set_delalloc_extent 0.15 ± 7% -0.0 0.12 ± 11% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 0.10 ± 11% -0.0 0.08 ± 11% perf-profile.children.cycles-pp.rcu_nmi_enter 0.08 ± 14% +0.0 0.10 ± 11% perf-profile.children.cycles-pp.remove_vma 0.06 ± 13% +0.0 0.09 ± 16% perf-profile.children.cycles-pp.newidle_balance 0.09 ± 9% +0.1 0.14 ± 7% perf-profile.children.cycles-pp.___perf_sw_event 0.20 ± 9% +0.1 0.26 ± 8% perf-profile.children.cycles-pp.do_raw_spin_unlock 0.27 ± 12% +0.1 0.34 ± 6% perf-profile.children.cycles-pp.memcg_slab_free_hook 0.02 ±142% +0.1 0.10 ± 12% perf-profile.children.cycles-pp.select_task_rq_fair 0.00 +0.1 0.08 ± 18% perf-profile.children.cycles-pp.enqueue_task_fair 0.01 ±223% +0.1 0.10 ± 18% perf-profile.children.cycles-pp.pick_next_task_fair 0.00 +0.1 0.10 ± 22% perf-profile.children.cycles-pp.ttwu_do_activate 0.07 ± 50% +0.1 0.17 ± 14% perf-profile.children.cycles-pp.update_curr 0.01 ±223% +0.1 0.13 ± 18% perf-profile.children.cycles-pp.finish_task_switch 0.01 ±223% +0.1 0.13 ± 16% perf-profile.children.cycles-pp.dequeue_entity 0.01 ±223% +0.1 0.14 ± 16% perf-profile.children.cycles-pp.dequeue_task_fair 0.00 +0.2 0.22 ± 15% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.10 ± 2% +0.2 1.33 ± 4% perf-profile.children.cycles-pp.match_held_lock 0.06 ± 59% +0.2 0.29 ± 13% perf-profile.children.cycles-pp.schedule_idle 0.10 ± 15% +0.3 0.36 ± 13% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state 0.00 +0.3 0.28 ± 17% perf-profile.children.cycles-pp.cgroup_rstat_updated 0.00 +0.4 0.35 ± 9% perf-profile.children.cycles-pp.mem_cgroup_css_rstat_flush 0.11 ± 19% +0.4 0.48 ± 15% perf-profile.children.cycles-pp.schedule 0.56 ± 9% +0.4 0.95 ± 7% perf-profile.children.cycles-pp.do_raw_spin_lock 0.07 ± 17% +0.4 0.49 ± 12% perf-profile.children.cycles-pp.try_to_wake_up 0.73 ± 11% +0.5 1.24 ± 14% perf-profile.children.cycles-pp.__mod_lruvec_page_state 0.00 +0.6 0.58 ± 13% perf-profile.children.cycles-pp.__queue_work 0.20 ± 14% +0.6 0.78 ± 13% perf-profile.children.cycles-pp.__schedule 0.00 +0.6 0.60 ± 14% perf-profile.children.cycles-pp.queue_work_on 2.35 ± 4% +1.6 3.96 ± 7% perf-profile.children.cycles-pp._raw_spin_unlock 3.97 ± 3% +3.5 7.48 ± 6% perf-profile.children.cycles-pp._raw_spin_lock 0.00 +6.8 6.81 ± 10% perf-profile.children.cycles-pp.cgroup_rstat_flush_locked 0.00 +6.8 6.83 ± 10% perf-profile.children.cycles-pp.cgroup_rstat_flush_irqsafe 0.00 +7.0 6.96 ± 10% perf-profile.children.cycles-pp.mem_cgroup_flush_stats 0.18 ± 10% +7.1 7.31 ± 10% perf-profile.children.cycles-pp.process_one_work 0.45 ± 6% +7.6 8.02 ± 10% perf-profile.children.cycles-pp.ret_from_fork 0.43 ± 8% +7.6 8.00 ± 10% perf-profile.children.cycles-pp.kthread 0.20 ± 10% +7.6 7.78 ± 10% perf-profile.children.cycles-pp.worker_thread 1.67 ± 9% -0.2 1.46 ± 2% perf-profile.self.cycles-pp.cpuidle_enter_state 0.47 ± 6% -0.1 0.40 ± 7% perf-profile.self.cycles-pp.lockdep_hardirqs_on_prepare 0.28 ± 10% -0.0 0.23 ± 7% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.25 ± 14% -0.0 0.21 ± 17% perf-profile.self.cycles-pp.__is_kernel_percpu_address 0.19 ± 5% -0.0 0.14 ± 21% perf-profile.self.cycles-pp.brd_do_bvec 0.07 ± 16% -0.0 0.03 ± 86% perf-profile.self.cycles-pp.timerqueue_del 0.16 ± 14% -0.0 0.12 ± 20% perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt 0.08 ± 6% -0.0 0.05 ± 44% perf-profile.self.cycles-pp.rb_erase 0.18 ± 8% +0.0 0.23 ± 8% perf-profile.self.cycles-pp.do_raw_spin_unlock 0.06 ± 9% +0.1 0.12 ± 11% perf-profile.self.cycles-pp._raw_spin_lock 0.10 ± 9% +0.1 0.18 ± 14% perf-profile.self.cycles-pp._raw_spin_unlock 0.88 ± 3% +0.2 1.04 ± 3% perf-profile.self.cycles-pp.match_held_lock 0.54 ± 7% +0.2 0.72 ± 6% perf-profile.self.cycles-pp.do_raw_spin_lock 0.00 +0.2 0.22 ± 15% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.00 +0.2 0.22 ± 6% perf-profile.self.cycles-pp.cgroup_rstat_flush_locked 0.00 +0.3 0.33 ± 10% perf-profile.self.cycles-pp.mem_cgroup_css_rstat_flush *************************************************************************************************** lkp-cfl-e1: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/300s/lkp-cfl-e1/five_sec/reaim/0xde commit: 0c7c5da68b ("memcg: switch lruvec stats to rstat") cda0d20525 ("memcg: infrastructure to flush memcg stats") 0c7c5da68b4a09ff cda0d205259a7e7c1f8e296130a ---------------- --------------------------- %stddev %change %stddev \ | \ 16.84 +1.2% 17.05 reaim.child_systime 1.17 -1.7% 1.15 reaim.child_utime 77332 -10.7% 69064 reaim.jobs_per_min 4833 -10.7% 4316 reaim.jobs_per_min_child 79421 -10.3% 71230 reaim.max_jobs_per_min 1.23 +12.0% 1.38 reaim.parent_time 1.00 +61.7% 1.61 ± 2% reaim.std_dev_percent 0.01 +84.7% 0.02 ± 2% reaim.std_dev_time 186593 +620.2% 1343882 reaim.time.involuntary_context_switches 15277 -15.4% 12918 reaim.time.major_page_faults 45920952 -4.7% 43779707 reaim.time.minor_page_faults 549.17 -3.4% 530.57 reaim.time.percent_of_cpu_this_job_got 1554 -3.5% 1500 reaim.time.system_time 108.32 -6.3% 101.53 reaim.time.user_time 147200 -4.7% 140342 reaim.workload 4260000 ± 24% +85.2% 7890750 ± 16% cpuidle..usage 0.00 ± 94% +0.0 0.01 ±125% mpstat.cpu.all.iowait% 37619 +10.2% 41448 softirqs.CPU5.SCHED 4236 ± 3% -20.4% 3371 ± 4% slabinfo.cred_jar.active_objs 4236 ± 3% -20.4% 3371 ± 4% slabinfo.cred_jar.num_objs 5335 +425.1% 28014 vmstat.system.cs 25198 ± 13% +34.2% 33810 ± 10% vmstat.system.in 36327 +93.6% 70343 ± 3% meminfo.Active 36170 +93.4% 69943 ± 4% meminfo.Active(anon) 374203 +9.7% 410350 meminfo.Committed_AS 45925 +77.1% 81324 ± 4% meminfo.Shmem 60038 ± 58% +4479.1% 2749217 ± 35% turbostat.C1 0.06 ± 34% +1.9 1.96 ± 41% turbostat.C1% 374823 ± 34% +283.9% 1438762 ± 52% turbostat.C1E 0.66 ± 33% +1.4 2.10 ± 55% turbostat.C1E% 92340 ± 84% -99.3% 670.00 ±204% turbostat.C8 48.13 +7.4% 51.70 turbostat.CorWatt 7689266 ± 13% +33.9% 10296075 ± 9% turbostat.IRQ 11.56 ± 34% -29.5% 8.15 ± 9% turbostat.Pkg%pc2 48.58 +7.4% 52.15 turbostat.PkgWatt 9048 +92.7% 17436 ± 4% proc-vmstat.nr_active_anon 599695 +1.5% 608725 proc-vmstat.nr_file_pages 8070 +4.9% 8467 proc-vmstat.nr_mapped 11469 +77.0% 20298 ± 4% proc-vmstat.nr_shmem 9048 +92.7% 17436 ± 4% proc-vmstat.nr_zone_active_anon 42656780 -4.6% 40700689 proc-vmstat.numa_hit 42656780 -4.6% 40700689 proc-vmstat.numa_local 92407 +22.4% 113128 proc-vmstat.pgactivate 43569042 -4.7% 41530882 proc-vmstat.pgalloc_normal 46430133 -4.6% 44284838 proc-vmstat.pgfault 43538315 -4.7% 41477247 proc-vmstat.pgfree 1629898 -4.2% 1561322 proc-vmstat.pgreuse 5.801e+09 +6.4% 6.174e+09 perf-stat.i.branch-instructions 30752328 +16.4% 35783086 perf-stat.i.branch-misses 2.16 ± 2% -0.3 1.81 perf-stat.i.cache-miss-rate% 2.765e+08 +25.7% 3.475e+08 perf-stat.i.cache-references 5341 +426.1% 28100 perf-stat.i.context-switches 1.15 -2.8% 1.11 perf-stat.i.cpi 2.523e+10 +4.4% 2.633e+10 perf-stat.i.cpu-cycles 747.05 +84.5% 1378 perf-stat.i.cpu-migrations 3729 +7.5% 4008 perf-stat.i.cycles-between-cache-misses 7.878e+09 +5.2% 8.284e+09 perf-stat.i.dTLB-loads 0.01 ± 9% -0.0 0.01 perf-stat.i.dTLB-store-miss-rate% 560872 -4.9% 533422 perf-stat.i.dTLB-store-misses 4.383e+09 +2.4% 4.487e+09 perf-stat.i.dTLB-stores 68.79 ± 2% -5.0 63.79 ± 2% perf-stat.i.iTLB-load-miss-rate% 1070659 ± 3% +6.7% 1142823 ± 2% perf-stat.i.iTLB-load-misses 578652 +21.6% 703855 ± 3% perf-stat.i.iTLB-loads 2.76e+10 +5.4% 2.908e+10 perf-stat.i.instructions 0.94 +2.0% 0.96 perf-stat.i.ipc 50.65 -15.2% 42.96 perf-stat.i.major-faults 0.24 ± 3% +12.4% 0.27 ± 3% perf-stat.i.metric.G/sec 1.58 +4.4% 1.65 perf-stat.i.metric.GHz 199.04 +3.3% 205.70 perf-stat.i.metric.K/sec 904.24 +3.3% 933.92 perf-stat.i.metric.M/sec 150989 -4.4% 144377 perf-stat.i.minor-faults 151039 -4.4% 144420 perf-stat.i.page-faults 36548 ± 4% +356.1% 166686 interrupts.CAL:Function_call_interrupts 2202 ± 12% +376.0% 10483 ± 4% interrupts.CPU0.CAL:Function_call_interrupts 4568 +1471.3% 71781 ± 2% interrupts.CPU0.RES:Rescheduling_interrupts 25.83 ± 21% +1613.2% 442.57 ± 6% interrupts.CPU0.TLB:TLB_shootdowns 2257 ± 18% +353.1% 10231 ± 3% interrupts.CPU1.CAL:Function_call_interrupts 4591 +1491.3% 73058 ± 2% interrupts.CPU1.RES:Rescheduling_interrupts 30.33 ± 13% +1384.9% 450.43 ± 3% interrupts.CPU1.TLB:TLB_shootdowns 2310 ± 20% +330.9% 9957 ± 4% interrupts.CPU10.CAL:Function_call_interrupts 4562 ± 2% +1496.4% 72842 ± 2% interrupts.CPU10.RES:Rescheduling_interrupts 31.33 ± 19% +1291.0% 435.86 ± 4% interrupts.CPU10.TLB:TLB_shootdowns 2043 ± 19% +414.7% 10517 ± 8% interrupts.CPU11.CAL:Function_call_interrupts 4444 ± 2% +1532.5% 72561 interrupts.CPU11.RES:Rescheduling_interrupts 26.17 ± 17% +1588.6% 441.86 ± 4% interrupts.CPU11.TLB:TLB_shootdowns 2257 ± 24% +363.1% 10452 ± 6% interrupts.CPU12.CAL:Function_call_interrupts 4576 +1474.2% 72041 ± 2% interrupts.CPU12.RES:Rescheduling_interrupts 27.17 ± 15% +1606.4% 463.57 ± 5% interrupts.CPU12.TLB:TLB_shootdowns 1948 ± 18% +449.5% 10706 ± 7% interrupts.CPU13.CAL:Function_call_interrupts 4551 +1487.9% 72280 ± 3% interrupts.CPU13.RES:Rescheduling_interrupts 28.67 ± 17% +1446.8% 443.43 ± 4% interrupts.CPU13.TLB:TLB_shootdowns 2363 ± 23% +353.4% 10714 ± 7% interrupts.CPU14.CAL:Function_call_interrupts 4521 ± 2% +1497.2% 72218 ± 2% interrupts.CPU14.RES:Rescheduling_interrupts 27.00 ± 19% +1561.4% 448.57 ± 4% interrupts.CPU14.TLB:TLB_shootdowns 2111 ± 20% +381.6% 10167 ± 5% interrupts.CPU15.CAL:Function_call_interrupts 4594 +1460.4% 71685 interrupts.CPU15.RES:Rescheduling_interrupts 26.67 ± 21% +1553.2% 440.86 ± 5% interrupts.CPU15.TLB:TLB_shootdowns 2477 ± 23% +319.6% 10397 ± 4% interrupts.CPU2.CAL:Function_call_interrupts 4551 ± 2% +1496.0% 72641 ± 2% interrupts.CPU2.RES:Rescheduling_interrupts 29.50 ± 15% +1381.4% 437.00 ± 4% interrupts.CPU2.TLB:TLB_shootdowns 2220 ± 17% +348.6% 9962 ± 4% interrupts.CPU3.CAL:Function_call_interrupts 4577 +1483.3% 72469 ± 2% interrupts.CPU3.RES:Rescheduling_interrupts 27.83 ± 18% +1437.2% 427.86 ± 3% interrupts.CPU3.TLB:TLB_shootdowns 1926 ± 13% +457.0% 10731 ± 7% interrupts.CPU4.CAL:Function_call_interrupts 4532 +1490.4% 72080 ± 3% interrupts.CPU4.RES:Rescheduling_interrupts 29.00 ± 18% +1472.4% 456.00 ± 5% interrupts.CPU4.TLB:TLB_shootdowns 2441 ± 22% +313.7% 10101 ± 4% interrupts.CPU5.CAL:Function_call_interrupts 4613 +1479.5% 72866 interrupts.CPU5.RES:Rescheduling_interrupts 25.33 ± 25% +1684.8% 452.14 ± 5% interrupts.CPU5.TLB:TLB_shootdowns 2400 ± 16% +337.5% 10502 ± 6% interrupts.CPU6.CAL:Function_call_interrupts 4527 +1480.1% 71535 interrupts.CPU6.RES:Rescheduling_interrupts 26.50 ± 22% +1557.1% 439.14 ± 5% interrupts.CPU6.TLB:TLB_shootdowns 2799 ± 22% +294.2% 11034 ± 6% interrupts.CPU7.CAL:Function_call_interrupts 4623 ± 2% +1435.4% 70979 ± 2% interrupts.CPU7.RES:Rescheduling_interrupts 27.83 ± 21% +1465.4% 435.71 ± 8% interrupts.CPU7.TLB:TLB_shootdowns 2487 ± 12% +303.6% 10039 ± 5% interrupts.CPU8.CAL:Function_call_interrupts 4534 +1504.6% 72761 ± 3% interrupts.CPU8.RES:Rescheduling_interrupts 28.17 ± 21% +1515.4% 455.00 ± 5% interrupts.CPU8.TLB:TLB_shootdowns 2300 ± 13% +364.6% 10685 ± 7% interrupts.CPU9.CAL:Function_call_interrupts 4563 +1458.2% 71100 interrupts.CPU9.RES:Rescheduling_interrupts 25.00 ± 24% +1686.9% 446.71 ± 7% interrupts.CPU9.TLB:TLB_shootdowns 72933 +1483.5% 1154904 interrupts.RES:Rescheduling_interrupts 442.33 ± 4% +1508.9% 7116 interrupts.TLB:TLB_shootdowns 0.04 ± 4% +138.7% 0.11 ± 6% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.05 ± 5% +158.0% 0.13 ± 4% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.11 ± 39% +127.2% 0.26 ± 32% perf-sched.sch_delay.avg.ms.io_schedule.__lock_page_killable.filemap_fault.__do_fault 0.02 ± 27% -46.9% 0.01 ± 26% perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read 0.03 ± 7% -15.7% 0.02 ± 15% perf-sched.sch_delay.avg.ms.pipe_write.new_sync_write.vfs_write.ksys_write 0.02 ± 31% +659.1% 0.13 ± 55% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_read_unlock.wait_task_zombie.do_wait 0.12 ± 78% -77.2% 0.03 ±121% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__dentry_kill.dput 0.20 ±142% -99.6% 0.00 ±115% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.begin_new_exec.load_elf_binary 0.02 ± 57% -78.0% 0.00 ±112% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.change_p4d_range.change_protection 0.02 ± 72% -79.4% 0.01 ± 91% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.do_sys_openat2.do_sys_open 0.04 ±150% -84.4% 0.01 ± 73% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.finish_fault.do_fault 0.07 ± 8% -62.4% 0.03 ± 33% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 0.00 ±223% +3094.3% 0.08 ±179% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.release_task.wait_task_zombie 0.12 ±108% -84.9% 0.02 ± 94% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.task_work_run.exit_to_user_mode_prepare 0.02 ± 46% +282.4% 0.07 ± 68% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 0.10 ±115% -88.0% 0.01 ±111% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.release_pages.tlb_flush_mmu 0.20 ± 70% +182.4% 0.55 ± 14% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_move_task.do_exit 0.13 ±113% -93.8% 0.01 ± 98% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.fd_install.do_sys_openat2.do_sys_open 0.21 ±215% -99.7% 0.00 ± 86% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu 0.12 ± 68% -76.3% 0.03 ± 40% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.exit_mmap.mmput 0.03 ±143% -94.6% 0.00 ±105% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.free_pages_and_swap_cache.tlb_flush_mmu 0.03 ±105% -94.8% 0.00 ± 66% perf-sched.sch_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk.lru_cache_add.shmem_getpage_gfp.shmem_write_begin 0.02 ± 30% -79.6% 0.00 ± 90% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.free_unref_page_list.release_pages 0.05 ±118% +288.1% 0.18 ± 55% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.handle_mm_fault.do_user_addr_fault 0.02 ± 47% -70.1% 0.01 ±130% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.kmem_cache_free.__put_anon_vma 0.03 ± 94% -80.9% 0.01 ± 92% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_acquire.__lock_task_sighand 0.03 ± 40% -83.3% 0.01 ±155% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_acquire.aa_file_perm 0.06 ± 60% +150.6% 0.15 ± 43% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_is_held_type.___might_sleep 0.02 ±107% -93.1% 0.00 ±197% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.vm_area_dup.dup_mmap 0.04 ±118% -93.0% 0.00 ±116% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.vma_interval_tree_insert.__vma_adjust 0.04 ± 66% -89.1% 0.00 ±183% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.___might_sleep.unmap_page_range 0.01 ±106% -85.2% 0.00 ±101% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.dput 0.00 ±223% +610.2% 0.01 ± 47% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.acct_collect.do_exit 0.03 ±100% -85.6% 0.00 ±100% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.d_alloc_parallel.__lookup_slow 0.05 ± 89% -96.1% 0.00 ± 58% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.d_path.perf_event_mmap 0.11 ± 74% -86.4% 0.01 ± 80% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__might_fault 0.05 ± 55% -78.8% 0.01 ± 23% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.fs_reclaim_acquire 0.11 ±118% -86.2% 0.02 ± 93% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.kmem_cache_alloc 0.09 ± 43% -86.4% 0.01 ±121% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.memcg_slab_free_hook 0.09 ± 56% -86.6% 0.01 ± 68% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.___might_sleep 0.17 ± 97% -96.3% 0.01 ± 86% perf-sched.sch_delay.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.kmem_cache_alloc 0.12 ±196% -97.3% 0.00 ±138% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.__mark_inode_dirty.generic_update_time.touch_atime 0.02 ± 26% -95.8% 0.00 ± 97% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.kmem_cache_free.unlink_anon_vmas.free_pgtables 0.02 ±138% -94.4% 0.00 ±118% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.kmem_cache_alloc.__anon_vma_prepare 0.04 ± 74% -95.4% 0.00 ± 86% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.kmem_cache_alloc.vm_area_dup 0.06 ± 40% -70.2% 0.02 ± 97% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.memcg_slab_free_hook.kmem_cache_free 0.01 ± 68% -86.9% 0.00 ±112% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.perf_iterate_sb.perf_event_mmap 0.05 ± 72% -71.1% 0.01 ±142% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.uncharge_page.mem_cgroup_uncharge_list 0.07 ±103% -88.2% 0.01 ± 99% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.get_obj_cgroup_from_current.kmem_cache_alloc 0.09 ± 79% -97.7% 0.00 ± 84% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.__anon_vma_prepare 0.02 ± 53% -85.6% 0.00 ±102% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.__pmd_alloc 0.03 ± 72% -79.2% 0.01 ±125% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc_trace.perf_event_mmap 0.03 ±142% -98.4% 0.00 ±169% perf-sched.sch_delay.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.mem_cgroup_charge.do_anonymous_page 0.01 ± 25% +243.9% 0.05 ± 67% perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.down_read.walk_component.path_lookupat.isra 0.01 ± 13% +475.0% 0.09 ±132% perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.anon_vma_clone.anon_vma_fork.dup_mmap 0.00 ±223% +4900.0% 0.05 ±140% perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.anon_vma_fork.dup_mmap.dup_mm 0.01 ± 21% +307.1% 0.05 ± 58% perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.01 ± 34% +397.9% 0.06 ± 61% perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.filename_create.do_linkat.__x64_sys_link 12.48 ± 31% +76.0% 21.98 ± 9% perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.02 ± 27% +32984.9% 6.18 ± 74% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_read_unlock.wait_task_zombie.do_wait 0.48 ±179% -96.2% 0.02 ±116% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.begin_new_exec.load_elf_binary 0.99 ±103% +507.9% 6.01 ± 53% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.do_anonymous_page.__handle_mm_fault 0.06 ± 16% +4772.3% 2.90 ± 87% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.do_wp_page.__handle_mm_fault 5.12 ± 33% +186.5% 14.66 ± 22% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.filemap_map_pages.do_fault 1.66 ± 73% +491.1% 9.84 ± 29% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.wp_page_copy.__handle_mm_fault 5.83 ± 36% +131.2% 13.47 ± 30% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 0.06 ±127% +852.8% 0.53 ±146% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.get_signal.arch_do_signal_or_restart 0.00 ±223% +42871.4% 1.07 ±209% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.release_task.wait_task_zombie 0.02 ±128% +7349.0% 1.30 ±172% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.ret_from_fork.[unknown] 0.00 ±223% +3033.9% 0.08 ±113% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.sigprocmask.__x64_sys_rt_sigprocmask 0.02 ± 53% +9229.0% 2.02 ± 81% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 0.02 ± 8% +20160.8% 3.68 ±124% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.delete_from_page_cache.truncate_inode_page 0.01 ±101% +1939.1% 0.19 ±113% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.do_wait.kernel_wait4 6.45 ± 74% +198.2% 19.24 ± 21% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_move_task.do_exit 0.01 ± 50% +9448.0% 1.42 ±143% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.thread_group_cputime_adjusted.wait_task_zombie 0.19 ±198% +2910.9% 5.70 ± 55% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_write_unlock_irq.release_task.wait_task_zombie 7.16 ± 66% +231.6% 23.74 ± 13% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.stop_one_cpu.sched_exec 0.00 ±223% +1.1e+05% 3.91 ±157% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.exit_signals.do_exit.do_group_exit 0.39 ±116% +1113.2% 4.70 ±139% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.unmap_region.__do_munmap 8.76 ± 23% +77.5% 15.54 ± 22% perf-sched.sch_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk.schedule_tail.ret_from_fork.[unknown] 0.33 ±173% +1025.4% 3.72 ± 81% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.handle_mm_fault.do_user_addr_fault 1.95 ± 72% +325.3% 8.30 ± 52% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_is_held_type.___might_sleep 0.47 ± 86% +1401.6% 6.99 ± 94% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.memcg_slab_free_hook.kmem_cache_free 0.07 ±134% -95.8% 0.00 ±115% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.vma_interval_tree_insert.__vma_adjust 0.03 ± 92% +10097.9% 2.72 ± 83% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.do_user_addr_fault 0.00 ±223% +1504.1% 0.02 ± 36% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.acct_collect.do_exit 2.34 ± 61% +468.7% 13.30 ± 33% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.do_user_addr_fault.exc_page_fault 0.00 ±223% +702.0% 0.02 ± 44% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.down_read_trylock.do_user_addr_fault 0.43 ±203% +1077.3% 5.11 ± 40% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.handle_mm_fault.do_user_addr_fault 0.02 ±109% +5174.6% 0.84 ±205% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.down_write_killable 0.06 ± 30% +4203.6% 2.75 ± 93% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.memcg_slab_free_hook.kmem_cache_free 0.05 ±181% +723.6% 0.42 ± 91% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.syscall_exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.00 ±223% +8372.5% 0.18 ±101% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.up_read.do_user_addr_fault 0.00 ±223% +1.4e+05% 1.81 ±234% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.xas_load.xas_find 0.00 ±141% +3282.9% 0.08 ±194% perf-sched.sch_delay.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.xas_start.xas_load 0.13 ±175% -87.4% 0.02 ±123% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.__mark_inode_dirty.generic_update_time.touch_atime 1.66 ±113% +364.7% 7.71 ± 42% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.__perf_sw_event.do_user_addr_fault.exc_page_fault 0.05 ± 39% -77.3% 0.01 ± 68% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.kmem_cache_free.unlink_anon_vmas.free_pgtables 0.12 ±223% +1385.4% 1.78 ±105% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.thread_group_cputime.thread_group_cputime_adjusted 0.33 ± 90% -92.5% 0.02 ± 80% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.__anon_vma_prepare 0.12 ±174% -95.2% 0.01 ±160% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.mem_cgroup_charge.do_anonymous_page 0.02 ±141% +22936.6% 3.69 ±153% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__anon_vma_prepare 0.02 ±108% +13620.0% 2.74 ±129% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__d_alloc 0.00 ±146% +29728.6% 1.29 ±136% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_clone 0.05 ±223% +1812.4% 0.96 ± 99% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_fork 0.02 ±111% +9575.2% 1.84 ±111% perf-sched.sch_delay.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.vm_area_alloc 0.04 ± 52% +6245.5% 2.70 ±157% perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.anon_vma_clone.anon_vma_fork.dup_mmap 0.00 ±223% +13314.3% 0.27 ±143% perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.anon_vma_fork.dup_mmap.dup_mm 0.93 ± 33% +377.4% 4.42 ± 58% perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.52 ± 6% +48.2% 5.22 ± 57% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork 0.05 ± 2% -20.3% 0.04 perf-sched.total_sch_delay.average.ms 24.34 ± 2% -77.8% 5.41 ± 10% perf-sched.total_wait_and_delay.average.ms 37615 +339.2% 165218 ± 5% perf-sched.total_wait_and_delay.count.ms 24.29 ± 2% -77.9% 5.37 ± 10% perf-sched.total_wait_time.average.ms 7.69 ± 6% -16.0% 6.46 ± 12% perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 6.52 ± 58% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 4.93 ± 44% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.scan_positives.dcache_readdir 294.26 ± 28% -50.5% 145.74 ± 16% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 87.51 ± 5% -16.8% 72.80 ± 11% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 57.89 ± 3% -97.1% 1.70 ± 10% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork 268.33 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 315.17 ± 4% -73.3% 84.00 ±158% perf-sched.wait_and_delay.count.io_schedule.__lock_page_killable.filemap_fault.__do_fault 491.17 ± 4% +502.5% 2959 ± 4% perf-sched.wait_and_delay.count.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.filemap_map_pages.do_fault 235.00 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.scan_positives.dcache_readdir 547.67 ± 4% +584.9% 3751 ± 8% perf-sched.wait_and_delay.count.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 38.83 ± 9% +113.0% 82.71 ± 6% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 3532 +15.4% 4076 ± 5% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork 2873 ± 3% +3435.5% 101580 ± 5% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork 509.76 ± 96% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 511.70 ± 95% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.scan_positives.dcache_readdir 177.28 ±207% +308.9% 724.85 ± 60% perf-sched.wait_and_delay.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 7.64 ± 6% -17.2% 6.33 ± 13% perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.04 ±130% +10932.2% 4.74 ±212% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown] 6.47 ± 58% -61.2% 2.51 ± 8% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 0.00 ±141% +2257.1% 0.06 ± 47% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_read_unlock.exec_binprm.bprm_execve 0.00 ±223% +1689.3% 0.02 ±139% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__pte_alloc.finish_fault 18.18 ±188% -97.1% 0.53 ± 41% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.copy_pte_range.copy_page_range 1.42 ±107% -86.1% 0.20 ±128% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.filemap_map_pages.do_fault 0.00 ±223% +31642.9% 0.16 ±192% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.pgd_alloc.mm_init 4.91 ± 44% -56.2% 2.15 ± 38% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.scan_positives.dcache_readdir 4.45 ± 16% -48.3% 2.30 ± 34% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.__x64_sys_rt_sigreturn.do_syscall_64 3.01 ± 49% +284.5% 11.58 ± 82% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 0.05 ± 45% +9984.3% 4.76 ±131% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.__wake_up_common_lock.pipe_write 6.77 ± 4% -85.3% 1.00 ± 27% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.delete_from_page_cache.truncate_inode_page 3.45 ± 10% -42.3% 1.99 ± 32% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.do_send_sig_info.kill_pid_info 0.00 ±223% +2182.1% 0.03 ±100% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.get_random_u32.arch_align_stack 2.99 ± 33% -53.1% 1.40 ± 31% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_fork.copy_process 0.76 ± 11% +106.1% 1.57 ± 6% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_move_task.do_exit 1.79 ± 23% -52.2% 0.86 ± 43% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.wake_up_new_task.kernel_clone 1.62 ± 35% -80.6% 0.32 ± 85% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.free_pages_and_swap_cache.tlb_flush_mmu 0.91 ± 33% -78.7% 0.19 ± 55% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.unmap_region.__do_munmap 0.00 ±223% +21276.2% 0.64 ±200% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.mnt_drop_write.do_unlinkat.do_syscall_64 2.76 ± 42% -90.9% 0.25 ± 64% perf-sched.wait_time.avg.ms.preempt_schedule_common.preempt_schedule_thunk.perf_iterate_sb.perf_event_mmap.do_brk_flags 2.72 ± 50% -74.3% 0.70 ±107% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.free_unref_page_list.release_pages 1.47 ± 57% -92.6% 0.11 ±183% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.kmem_cache_free.remove_vma 0.00 ±223% +1271.4% 0.01 ± 53% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.___might_sleep.copy_strings 0.04 ±156% +1105.5% 0.47 ± 40% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__handle_mm_fault.handle_mm_fault 0.03 ±161% +5970.7% 1.90 ± 76% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__local_bh_enable_ip.arch_do_signal_or_restart 0.00 ±223% +2400.0% 0.02 ±131% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.down_write_killable 0.00 ±223% +47861.0% 0.88 ±164% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.aa_get_task_label.apparmor_task_getsecid 0.00 ±223% +11202.9% 0.28 ± 84% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.common_file_perm.security_file_permission 0.00 ±223% +6298.9% 0.28 ±139% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.copy_user_enhanced_fast_string._copy_from_user 0.00 ±223% +12987.5% 0.35 ±106% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.copy_user_enhanced_fast_string.copyout 0.00 ±223% +65085.7% 0.22 ±224% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.do_dentry_open.do_open 0.01 ±223% +8642.9% 1.12 ± 67% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.42 ± 38% -47.4% 0.75 ± 26% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.00 ±146% +12813.0% 0.50 ± 96% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.find_vma.do_user_addr_fault 0.08 ±134% +184.1% 0.22 ± 59% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.__alloc_file 0.03 ±223% +20360.2% 6.41 ± 89% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__fget_files 0.84 ±112% -89.9% 0.08 ± 73% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.get_mem_cgroup_from_mm 0.05 ± 75% +469.9% 0.31 ± 55% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.handle_mm_fault 0.00 ±223% +12399.2% 0.35 ±110% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.path_init 0.00 ±223% +26716.3% 0.31 ± 60% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.vfs_write 0.34 ± 25% -44.9% 0.19 ± 18% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.___might_sleep 0.00 ±223% +2968.6% 0.03 ± 68% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.__memcg_kmem_charge_page 0.00 ±223% +45528.6% 0.23 ± 70% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.xas_descend 0.04 ±223% +563.2% 0.27 ± 92% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.xas_start 0.04 ±203% +943.6% 0.46 ±110% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.aa_file_perm 0.01 ±165% +1435.2% 0.14 ±159% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.cgroup_throttle_swaprate 0.07 ±152% +838.6% 0.70 ± 65% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.handle_mm_fault 0.00 ±223% +23621.4% 0.16 ±152% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.page_counter_try_charge.obj_cgroup_charge_pages 0.01 ±154% +2000.5% 0.31 ± 71% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.refill_obj_stock.obj_cgroup_charge 0.04 ±188% +529.3% 0.27 ± 82% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.unmap_page_range.unmap_vmas 0.02 ±183% +3708.7% 0.63 ± 76% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.up_read.do_user_addr_fault 0.00 ±141% +1782.4% 0.08 ±180% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.vm_unmapped_area.arch_get_unmapped_area_topdown 0.00 ±223% +25295.9% 0.30 ± 64% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.vmacache_find.find_vma 0.00 ±223% +9764.3% 0.20 ±209% perf-sched.wait_time.avg.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.walk_component.link_path_walk 1.11 ± 25% -51.6% 0.54 ± 19% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.__perf_sw_event.handle_mm_fault.do_user_addr_fault 0.00 ±152% +7242.9% 0.15 ±224% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.kmem_cache_free.__put_anon_vma.unlink_anon_vmas 0.00 ± 76% +2532.7% 0.03 ±122% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.__might_fault.get_user_arg_ptr 2.73 ± 33% -48.9% 1.39 ± 32% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.dput.scan_positives 1.85 ± 42% -78.8% 0.39 ± 93% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.fs_reclaim_acquire.__alloc_pages 1.58 ± 12% -76.8% 0.37 ± 65% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.get_mem_cgroup_from_mm.mem_cgroup_charge 1.30 ± 39% -71.3% 0.37 ± 53% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.get_obj_cgroup_from_current.kmem_cache_alloc 0.01 ± 79% +318.4% 0.03 ± 68% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.prepend_path.isra 0.76 ±223% +600.6% 5.33 ± 11% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.thread_group_cputime.thread_group_cputime_adjusted 0.00 ±223% +1287.0% 0.03 ±121% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.__alloc_pages.__get_free_pages 105.94 ±219% -99.5% 0.53 ±108% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.__alloc_pages.alloc_pages_vma 20.60 ±200% -98.2% 0.37 ± 56% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.get_mem_cgroup_from_mm.mem_cgroup_charge 2.78 ± 41% -88.2% 0.33 ± 62% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.kmem_cache_alloc.vm_area_dup 1.85 ± 27% -88.8% 0.21 ± 88% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.mem_cgroup_charge.do_anonymous_page 1.55 ± 36% -78.4% 0.33 ±108% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.mem_cgroup_uncharge_list.release_pages 0.34 ± 32% -74.1% 0.09 ± 31% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.memcg_slab_free_hook.kmem_cache_free 2.16 ± 74% -92.9% 0.15 ±145% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.perf_event_mmap.do_brk_flags 0.58 ±223% +781.1% 5.15 ± 14% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.thread_group_cputime.thread_group_cputime_adjusted 0.01 ±141% +445.5% 0.03 ±111% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.wp_page_copy.__handle_mm_fault 2.04 ± 35% -78.3% 0.44 ±112% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.get_obj_cgroup_from_current.kmem_cache_alloc 0.04 ±223% +9636.3% 4.12 ±223% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__d_alloc 0.95 ±149% +917.3% 9.62 ± 79% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_clone 1.39 ± 37% -61.0% 0.54 ± 42% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.rcu_read_lock_held_common.rcu_read_lock_held 1.95 ± 60% -74.3% 0.50 ± 38% perf-sched.wait_time.avg.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_lockdep_current_cpu_online.rcu_read_lock_held_common.rcu_read_lock_held 0.04 ±141% +69206.9% 30.26 ±135% perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.anon_vma_fork.dup_mmap.dup_mm 294.22 ± 28% -50.5% 145.72 ± 16% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 87.48 ± 5% -16.8% 72.78 ± 11% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 57.88 ± 3% -97.1% 1.69 ± 10% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork 0.14 ±122% +1.1e+05% 152.64 ±227% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown] 6.21 ± 42% +143.8% 15.13 ± 35% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.02 ± 43% -68.6% 1.58 ±133% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_read_unlock.do_wait.kernel_wait4 0.01 ±142% +15034.7% 0.88 ± 28% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_read_unlock.exec_binprm.bprm_execve 6.53 ± 24% +7474.9% 494.92 ±113% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_read_unlock.wait_task_zombie.do_wait 2.34 ±141% +6328.5% 150.26 ±232% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.__pmd_alloc.copy_page_range 9.61 ± 13% +8868.5% 862.08 ± 39% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.do_anonymous_page.__handle_mm_fault 0.00 ±223% +42044.6% 1.12 ±186% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.pgd_alloc.mm_init 177.02 ±207% +308.8% 723.69 ± 60% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock.zap_pte_range.unmap_page_range 7.76 ± 15% +1879.9% 153.69 ±225% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.do_sigaction.__x64_sys_rt_sigaction 0.61 ±202% +511.4% 3.70 ± 80% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.futex_exit_release.exit_mm_release 8.75 ± 17% +92.0% 16.80 ± 39% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.get_signal.arch_do_signal_or_restart 1.84 ±223% +630.6% 13.43 ± 33% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.release_task.wait_task_zombie 3.27 ± 58% +9126.8% 302.10 ±146% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irq.wait_task_zombie.do_wait 3.17 ± 66% +13581.8% 433.83 ±112% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.__wake_up_common_lock.pipe_write 7.42 ± 8% +157.9% 19.12 ± 37% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.delete_from_page_cache.truncate_inode_page 3.27 ±113% +228.9% 10.75 ± 21% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.do_wait.kernel_wait4 3.72 ± 83% +283.0% 14.24 ± 8% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.rcu_preempt_deferred_qs_irqrestore.__rcu_read_unlock 6.55 ± 20% +10910.3% 721.27 ± 62% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.release_pages.tlb_flush_mmu 0.02 ±212% +2925.0% 0.60 ±113% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.rwsem_wake.isra 7.20 ± 8% +60.4% 11.54 ± 26% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_fork.copy_process 10.13 ± 32% +118.2% 22.10 ± 14% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.sched_move_task.do_exit 4.88 ± 53% +180.5% 13.68 ± 29% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_spin_unlock_irqrestore.thread_group_cputime_adjusted.wait_task_zombie 4.92 ± 56% +8906.5% 442.97 ±109% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk._raw_write_unlock_irq.release_task.wait_task_zombie 3.40 ± 83% +294.7% 13.41 ± 24% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.cpu_stop_queue_work.stop_one_cpu.sched_exec 0.52 ±223% +1456.8% 8.11 ± 71% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.exit_signals.do_exit.do_group_exit 4.60 ± 42% +6321.5% 295.15 ±151% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.lru_add_drain.exit_mmap.mmput 0.00 ±223% +36990.3% 1.55 ±170% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.mnt_drop_write.do_unlinkat.do_syscall_64 0.00 ±223% +1495.2% 0.05 ±140% perf-sched.wait_time.max.ms.preempt_schedule_common.preempt_schedule_thunk.rwsem_down_write_slowpath.unlink_anon_vmas.free_pgtables 1.14 ±217% +347.5% 5.10 ± 75% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.kmem_cache_alloc.__d_alloc 7.78 ± 19% +59.5% 12.41 ± 25% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.kmem_cache_alloc.vm_area_dup 3.67 ± 50% -83.4% 0.61 ±197% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.kmem_cache_free.remove_vma 8.33 ± 10% +96.5% 16.37 ± 30% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.lock_acquire.get_obj_cgroup_from_current 0.01 ±141% +11672.2% 0.69 ±172% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.___pmd_free_tlb.free_p4d_range 0.12 ±156% +8195.3% 9.69 ± 38% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__handle_mm_fault.handle_mm_fault 0.11 ±223% +2602.9% 2.85 ±111% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__local_bh_enable_ip.__fpu__restore_sig 0.03 ±161% +16380.9% 5.16 ± 66% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__local_bh_enable_ip.arch_do_signal_or_restart 2.82 ± 97% +5262.6% 151.09 ±230% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__mem_cgroup_charge.mem_cgroup_charge 0.60 ±206% +25004.9% 149.42 ±232% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.__might_fault 0.19 ±146% +1714.2% 3.48 ±102% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__might_sleep.down_write 0.01 ±223% +21307.5% 2.46 ±139% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.__rcu_read_unlock.dput 0.00 ±223% +5.8e+06% 144.37 ±241% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.aa_file_perm.common_file_perm 0.00 ±223% +71167.5% 2.61 ±151% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.aa_get_task_label.apparmor_task_getsecid 0.00 ±223% +1.6e+05% 3.98 ± 82% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.common_file_perm.security_file_permission 0.00 ±223% +11128.6% 0.30 ±121% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.copy_strings.isra 0.01 ±223% +29050.6% 3.74 ±150% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.copy_user_enhanced_fast_string._copy_from_user 0.00 ±223% +1.5e+05% 4.01 ± 86% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.copy_user_enhanced_fast_string.copyout 0.03 ±213% +4267.9% 1.40 ±186% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.d_lookup.lookup_dcache 0.00 ±223% +1.1e+05% 1.25 ±233% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.do_dentry_open.do_open 0.02 ±223% +22138.8% 4.74 ± 68% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.16 ± 64% +3667.2% 156.55 ±220% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.do_user_addr_fault.exc_page_fault 0.08 ±223% +6720.7% 5.66 ±112% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.down_read_trylock.do_user_addr_fault 1.52 ±145% +705.3% 12.27 ± 32% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.exit_to_user_mode_prepare 7.16 ± 11% +121.6% 15.86 ± 32% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.01 ±141% +84292.1% 7.60 ± 66% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.find_vma.do_user_addr_fault 0.04 ±205% +1176.4% 0.45 ±108% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.free_pages_and_swap_cache.tlb_flush_mmu 1.92 ±127% +8073.6% 157.00 ±219% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.handle_mm_fault.do_user_addr_fault 0.22 ±145% +3590.2% 8.30 ± 73% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.__alloc_file 0.10 ±223% +3801.0% 4.06 ±103% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.getname_flags 0.08 ±146% +2656.7% 2.34 ±122% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kmem_cache_alloc.vm_area_alloc 0.05 ±187% +1790.1% 0.99 ±120% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.kthread_blkcg.cgroup_throttle_swaprate 2.44 ±122% +449.2% 13.40 ± 32% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.ktime_get_coarse_real_ts64.current_time 0.00 ±223% +10871.4% 0.26 ±212% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__d_lookup 0.08 ±223% +36479.6% 30.48 ± 82% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__fget_files 5.50 ± 59% +97.5% 10.86 ± 31% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.__might_fault 2.10 ±113% +391.9% 10.35 ± 22% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.aa_file_perm 0.03 ±177% +3898.4% 1.29 ±211% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.cgroup_throttle_swaprate 1.30 ±176% +739.9% 10.96 ± 34% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.down_read_trylock 2.52 ±114% +344.4% 11.21 ± 46% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.down_write 3.22 ± 89% +161.8% 8.44 ± 27% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.get_obj_cgroup_from_current 0.29 ± 93% +2767.1% 8.21 ± 20% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.handle_mm_fault 0.14 ±223% +5523.7% 8.05 ± 43% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.pagecache_get_page 0.00 ±223% +1.2e+05% 3.34 ±110% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.path_init 0.00 ±223% +6.1e+05% 7.18 ± 50% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_acquire.vfs_write 6.77 ± 23% +239.8% 23.00 ± 53% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.___might_sleep 0.00 ±223% +8857.1% 0.15 ±104% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.__memcg_kmem_charge_page 0.19 ±137% +1638.9% 3.34 ± 95% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.cgroup_throttle_swaprate 0.06 ±175% +2742.1% 1.71 ±168% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.get_mem_cgroup_from_mm 2.16 ±124% +214.1% 6.78 ± 22% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.mem_cgroup_from_task 0.00 ±223% +2.9e+05% 5.25 ± 63% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.xas_descend 0.14 ±223% +3535.7% 4.96 ± 75% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_is_held_type.xas_start 1.06 ±223% +527.8% 6.68 ± 40% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release._copy_from_user 0.08 ±212% +6635.0% 5.47 ±102% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.aa_file_perm 0.01 ±177% +15285.1% 1.87 ±162% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.cgroup_throttle_swaprate 0.14 ±162% +6587.0% 9.46 ± 50% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.handle_mm_fault 1.77 ±125% +8487.3% 151.72 ±228% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.kmem_cache_alloc 0.05 ±223% +2843.3% 1.42 ±150% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lock_release.perf_event_mmap 0.00 ±223% +5081.8% 0.10 ±170% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.lockdep_init_map_type.__raw_spin_lock_init 1.17 ±187% +2440.6% 29.72 ±189% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.mem_cgroup_from_task.handle_mm_fault 1.26 ±105% +253.9% 4.46 ± 52% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.memcg_slab_free_hook.kmem_cache_free 1.74 ±132% +395.9% 8.64 ± 35% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.obj_cgroup_charge.kmem_cache_alloc 0.00 ±223% +2.4e+05% 3.15 ±165% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.page_counter_try_charge.obj_cgroup_charge_pages 0.00 ±223% +1987.8% 0.05 ±106% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.perf_lock_task_context.perf_pin_task_context 0.00 ±223% +8128.6% 0.18 ±113% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.pte_alloc_one.do_fault 0.00 ±223% +28185.7% 1.04 ±229% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.rcu_is_watching.dput 0.81 ± 86% +582.0% 5.49 ± 51% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.refill_obj_stock.memcg_slab_free_hook 0.04 ±168% +16709.0% 6.95 ± 44% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.refill_obj_stock.obj_cgroup_charge 0.22 ±146% +2102.2% 4.91 ±136% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.static_obj.lockdep_init_map_type 0.00 ±223% +50820.0% 1.27 ±185% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.strlen.perf_event_mmap 5.01 ± 52% +229.3% 16.51 ± 31% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.syscall_enter_from_user_mode.do_syscall_64 1.63 ±223% +951.0% 17.10 ± 16% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.thread_group_cputime.thread_group_cputime_adjusted 0.05 ±151% +5056.5% 2.71 ± 85% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.unmap_page_range.unmap_vmas 0.03 ±200% +19804.7% 6.00 ± 41% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.up_read.do_user_addr_fault 0.01 ±141% +2797.1% 0.24 ±192% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.vm_unmapped_area.arch_get_unmapped_area_topdown 0.00 ±223% +2e+05% 5.26 ± 46% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.vmacache_find.find_vma 0.00 ±223% +24977.0% 0.96 ±215% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.walk_component.link_path_walk 0.00 ±223% +70171.4% 1.41 ±238% perf-sched.wait_time.max.ms.preempt_schedule_irq.irqentry_exit.asm_sysvec_reschedule_ipi.xas_start.xas_load 8.36 ± 19% +2637.0% 228.71 ±158% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.__perf_sw_event.do_user_addr_fault.exc_page_fault 0.01 ±223% +19788.2% 1.16 ±214% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.kmem_cache_alloc.__alloc_file.alloc_empty_file 0.01 ±146% +8216.4% 0.57 ±231% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.kmem_cache_free.__put_anon_vma.unlink_anon_vmas 5.94 ± 42% +85.7% 11.02 ± 29% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.__might_fault._copy_from_user 0.01 ± 75% +11643.9% 1.60 ±186% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.__might_fault.get_user_arg_ptr 4.10 ± 75% +121.1% 9.07 ± 18% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.down_read_trylock.do_user_addr_fault 2.49 ±130% +402.6% 12.53 ± 55% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.handle_mm_fault.do_user_addr_fault 0.76 ±223% +1327.6% 10.87 ± 20% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_acquire.thread_group_cputime.thread_group_cputime_adjusted 0.00 ±223% +4832.5% 0.09 ±174% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.__alloc_pages.__get_free_pages 0.00 ±223% +71889.3% 1.92 ±156% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.copy_page_to_iter.shmem_file_read_iter 3.56 ± 10% +46.6% 5.22 ± 28% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.memcg_slab_free_hook.kmem_cache_free 0.58 ±223% +722.0% 4.78 ± 69% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.proc_invalidate_siblings_dcache.release_task 0.58 ±223% +1111.7% 7.08 ± 34% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.thread_group_cputime.thread_group_cputime_adjusted 0.01 ±168% +1055.9% 0.13 ±121% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.lock_release.wp_page_copy.__handle_mm_fault 1.13 ±215% +38575.8% 436.26 ±112% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__alloc_file 0.08 ±213% +3170.2% 2.58 ±203% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__anon_vma_prepare 0.08 ±223% +1.8e+05% 151.13 ±230% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.__d_alloc 2.20 ±142% +26189.6% 578.11 ± 84% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_clone 1.39 ±223% +31241.3% 436.58 ±112% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.kmem_cache_alloc.anon_vma_fork 6.93 ± 6% +82.5% 12.64 ± 58% perf-sched.wait_time.max.ms.preempt_schedule_notrace.preempt_schedule_notrace_thunk.rcu_is_watching.rcu_read_lock_held_common.rcu_read_lock_held 9.37 ± 14% +1609.7% 160.15 ±214% perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.anon_vma_clone.anon_vma_fork.dup_mmap 0.05 ±141% +6.5e+05% 295.63 ±151% perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.anon_vma_fork.dup_mmap.dup_mm 8.11 ± 19% +66.7% 13.52 ± 43% perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.dup_mmap.dup_mm.copy_process 0.30 ± 61% +298.5% 1.20 ± 26% perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.vma_link.mmap_region.do_mmap 12.00 -2.2 9.77 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.84 -2.2 9.62 ± 4% perf-profile.calltrace.cycles-pp.dcache_readdir.iterate_dir.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.97 -2.2 9.75 ± 4% perf-profile.calltrace.cycles-pp.iterate_dir.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.42 -2.2 9.24 ± 4% perf-profile.calltrace.cycles-pp.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64.do_syscall_64 11.56 ± 3% -1.2 10.36 ± 7% perf-profile.calltrace.cycles-pp.execve 11.44 ± 3% -1.2 10.26 ± 7% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 11.44 ± 3% -1.2 10.26 ± 7% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 11.44 ± 3% -1.2 10.26 ± 7% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve 11.31 ± 3% -1.2 10.13 ± 7% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 12.64 ± 2% -1.1 11.57 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 2.84 ± 8% -1.1 1.78 ± 20% perf-profile.calltrace.cycles-pp.do_raw_spin_lock.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64 2.45 ± 9% -1.0 1.45 ± 24% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.do_raw_spin_lock.scan_positives.dcache_readdir.iterate_dir 4.78 ± 3% -0.9 3.88 ± 17% perf-profile.calltrace.cycles-pp.kill 4.76 ± 3% -0.9 3.85 ± 17% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.kill 4.75 ± 3% -0.9 3.85 ± 17% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 5.38 ± 4% -0.9 4.48 ± 3% perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault 5.01 ± 4% -0.8 4.21 ± 8% perf-profile.calltrace.cycles-pp.lock_page_memcg.page_remove_rmap.zap_pte_range.unmap_page_range.unmap_vmas 6.32 ± 4% -0.6 5.68 ± 3% perf-profile.calltrace.cycles-pp.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap.mmput 3.14 ± 3% -0.6 2.52 ± 16% perf-profile.calltrace.cycles-pp.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 2.90 ± 4% -0.6 2.34 ± 16% perf-profile.calltrace.cycles-pp.kill_proc_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 9.05 -0.5 8.56 ± 2% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.26 ± 5% -0.5 3.77 ± 3% perf-profile.calltrace.cycles-pp.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 1.60 ± 15% -0.5 1.13 ± 3% perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.vfs_statx.__do_sys_newstat.do_syscall_64 3.86 -0.5 3.39 perf-profile.calltrace.cycles-pp.dput.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64 8.61 -0.5 8.14 ± 3% perf-profile.calltrace.cycles-pp.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve 4.76 ± 4% -0.5 4.29 ± 2% perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap 8.70 -0.5 8.23 ± 3% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64 1.72 ± 12% -0.4 1.30 ± 19% perf-profile.calltrace.cycles-pp.filename_lookup.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.98 ± 3% -0.4 4.59 ± 2% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.96 ± 3% -0.4 4.58 ± 2% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64 6.33 -0.3 5.99 ± 2% perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.54 ± 5% -0.3 1.23 ± 17% perf-profile.calltrace.cycles-pp.do_send_sig_info.kill_pid_info.kill_proc_info.__x64_sys_kill.do_syscall_64 2.61 ± 5% -0.3 2.30 ± 2% perf-profile.calltrace.cycles-pp.lock_acquire.lock_page_memcg.page_remove_rmap.zap_pte_range.unmap_page_range 2.74 ± 4% -0.3 2.43 ± 3% perf-profile.calltrace.cycles-pp.kill_pid_info.kill_proc_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.30 -0.3 5.01 ± 2% perf-profile.calltrace.cycles-pp.dup_mm.copy_process.kernel_clone.__do_sys_clone.do_syscall_64 5.19 -0.3 4.90 ± 2% perf-profile.calltrace.cycles-pp.dup_mmap.dup_mm.copy_process.kernel_clone.__do_sys_clone 2.89 ± 2% -0.3 2.61 ± 3% perf-profile.calltrace.cycles-pp.do_set_pte.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault 0.73 ± 6% -0.3 0.46 ± 63% perf-profile.calltrace.cycles-pp.get_mem_cgroup_from_mm.mem_cgroup_charge.do_anonymous_page.__handle_mm_fault.handle_mm_fault 2.01 ± 2% -0.3 1.74 ± 8% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.begin_new_exec.load_elf_binary 1.99 ± 2% -0.3 1.73 ± 8% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.begin_new_exec 2.52 -0.2 2.27 ± 2% perf-profile.calltrace.cycles-pp.setlocale 5.87 -0.2 5.62 ± 3% perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault 1.46 ± 5% -0.2 1.21 ± 6% perf-profile.calltrace.cycles-pp.elf_map.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common 2.30 -0.2 2.06 ± 3% perf-profile.calltrace.cycles-pp.lock_page_memcg.page_add_file_rmap.do_set_pte.filemap_map_pages.do_fault 0.63 ± 7% -0.2 0.40 ± 63% perf-profile.calltrace.cycles-pp.perf_event_mmap.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.21 ± 5% -0.2 0.98 ± 15% perf-profile.calltrace.cycles-pp.unlink 1.20 ± 5% -0.2 0.97 ± 15% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink 1.20 ± 6% -0.2 0.97 ± 15% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 2.40 -0.2 2.17 ± 3% perf-profile.calltrace.cycles-pp.page_add_file_rmap.do_set_pte.filemap_map_pages.do_fault.__handle_mm_fault 1.26 ± 4% -0.2 1.04 ± 14% perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk 2.10 -0.2 1.88 ± 2% perf-profile.calltrace.cycles-pp.lockref_put_or_lock.dput.scan_positives.dcache_readdir.iterate_dir 1.06 ± 24% -0.2 0.83 ± 3% perf-profile.calltrace.cycles-pp.kmem_cache_alloc.vm_area_dup.dup_mmap.dup_mm.copy_process 1.19 ± 6% -0.2 0.98 ± 7% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.elf_map.load_elf_binary.exec_binprm.bprm_execve 1.09 ± 6% -0.2 0.89 ± 15% perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 0.80 ± 5% -0.2 0.60 ± 43% perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.96 ± 4% -0.2 0.77 ± 16% perf-profile.calltrace.cycles-pp.group_send_sig_info.kill_pid_info.kill_proc_info.__x64_sys_kill.do_syscall_64 1.08 ± 5% -0.2 0.88 ± 8% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.elf_map.load_elf_binary.exec_binprm 1.06 ± 6% -0.2 0.87 ± 7% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.elf_map.load_elf_binary 1.40 ± 2% -0.2 1.21 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_unlock.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64 1.29 ± 2% -0.2 1.11 ± 4% perf-profile.calltrace.cycles-pp._raw_spin_lock_nested.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64 1.32 ± 3% -0.2 1.14 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64 1.24 ± 2% -0.2 1.06 ± 4% perf-profile.calltrace.cycles-pp.lock_acquire._raw_spin_lock_nested.scan_positives.dcache_readdir.iterate_dir 1.02 ± 3% -0.2 0.84 ± 15% perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 0.99 ± 3% -0.2 0.82 ± 15% perf-profile.calltrace.cycles-pp.__alloc_pages.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault 2.02 -0.2 1.85 perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.96 ± 2% -0.2 1.80 perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 0.92 ± 5% -0.2 0.76 ± 17% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 1.22 ± 3% -0.2 1.06 ± 3% perf-profile.calltrace.cycles-pp.lock_acquire._raw_spin_lock.scan_positives.dcache_readdir.iterate_dir 1.13 ± 3% -0.2 0.97 ± 3% perf-profile.calltrace.cycles-pp.lock_release._raw_spin_unlock.scan_positives.dcache_readdir.iterate_dir 1.32 ± 2% -0.1 1.18 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock.lockref_put_or_lock.dput.scan_positives.dcache_readdir 0.59 ± 6% -0.1 0.45 ± 40% perf-profile.calltrace.cycles-pp.rcu_read_lock_sched_held.lock_acquire.lock_page_memcg.page_remove_rmap.zap_pte_range 1.35 ± 2% -0.1 1.21 ± 3% perf-profile.calltrace.cycles-pp.lock_acquire.lock_page_memcg.page_add_file_rmap.do_set_pte.filemap_map_pages 2.85 ± 2% -0.1 2.72 perf-profile.calltrace.cycles-pp.__libc_fork 0.87 ± 3% -0.1 0.74 ± 3% perf-profile.calltrace.cycles-pp.sync 0.86 ± 3% -0.1 0.74 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sync 0.86 ± 3% -0.1 0.74 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sync 0.97 ± 5% -0.1 0.84 ± 4% perf-profile.calltrace.cycles-pp.lock_acquire.dput.scan_positives.dcache_readdir.iterate_dir 0.86 ± 3% -0.1 0.74 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_sync.do_syscall_64.entry_SYSCALL_64_after_hwframe.sync 0.86 ± 3% -0.1 0.74 ± 3% perf-profile.calltrace.cycles-pp.ksys_sync.__x64_sys_sync.do_syscall_64.entry_SYSCALL_64_after_hwframe.sync 1.39 ± 2% -0.1 1.27 ± 3% perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.40 ± 2% -0.1 1.28 ± 4% perf-profile.calltrace.cycles-pp.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.44 ± 2% -0.1 2.32 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork 2.44 ± 2% -0.1 2.32 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork 0.80 ± 3% -0.1 0.69 ± 4% perf-profile.calltrace.cycles-pp.iterate_supers.ksys_sync.__x64_sys_sync.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.94 ± 4% -0.1 0.82 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap 0.94 ± 4% -0.1 0.82 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap 2.44 ± 2% -0.1 2.32 ± 2% perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork 2.44 ± 2% -0.1 2.32 ± 2% perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork 0.94 ± 4% -0.1 0.83 ± 3% perf-profile.calltrace.cycles-pp.__mmap 1.19 ± 2% -0.1 1.08 ± 4% perf-profile.calltrace.cycles-pp.lock_acquire._raw_spin_lock.lockref_put_or_lock.dput.scan_positives 0.82 ± 3% -0.1 0.71 ± 6% perf-profile.calltrace.cycles-pp.__xstat64 0.68 ± 3% -0.1 0.57 ± 4% perf-profile.calltrace.cycles-pp.__lock_acquire.lock_acquire._raw_spin_lock_nested.scan_positives.dcache_readdir 0.91 ± 4% -0.1 0.80 ± 3% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap 0.94 ± 4% -0.1 0.84 ± 8% perf-profile.calltrace.cycles-pp.__clear_user.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common 0.73 ± 3% -0.1 0.63 ± 6% perf-profile.calltrace.cycles-pp.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64 0.88 ± 4% -0.1 0.78 ± 3% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap 0.75 ± 3% -0.1 0.65 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__xstat64 0.75 ± 2% -0.1 0.65 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64 2.33 -0.1 2.23 ± 2% perf-profile.calltrace.cycles-pp.anon_vma_fork.dup_mmap.dup_mm.copy_process.kernel_clone 0.70 ± 3% -0.1 0.61 ± 6% perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64 1.14 ± 3% -0.1 1.05 ± 3% perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64_nocancel 1.18 ± 3% -0.1 1.09 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__open64_nocancel.setlocale 1.18 ± 3% -0.1 1.09 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64_nocancel.setlocale 1.19 ± 3% -0.1 1.11 ± 3% perf-profile.calltrace.cycles-pp.__open64_nocancel.setlocale 1.14 ± 3% -0.1 1.06 ± 3% perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64_nocancel.setlocale 0.69 ± 4% -0.1 0.61 ± 4% perf-profile.calltrace.cycles-pp.__lock_acquire.lock_acquire._raw_spin_lock.scan_positives.dcache_readdir 0.69 ± 4% -0.1 0.61 ± 2% perf-profile.calltrace.cycles-pp.asm_exc_page_fault.setlocale 0.67 ± 5% -0.1 0.59 ± 5% perf-profile.calltrace.cycles-pp.rcu_read_lock_sched_held.lock_release._raw_spin_unlock.scan_positives.dcache_readdir 0.67 ± 4% -0.1 0.59 ± 2% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.setlocale 0.68 ± 5% -0.1 0.60 ± 2% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.setlocale 0.79 ± 4% -0.1 0.72 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.79 ± 4% -0.1 0.72 ± 4% perf-profile.calltrace.cycles-pp.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.70 ± 3% -0.1 0.63 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_unlock.lockref_put_or_lock.dput.scan_positives.dcache_readdir 0.66 ± 3% -0.1 0.60 ± 5% perf-profile.calltrace.cycles-pp.__lock_acquire.lock_acquire._raw_spin_lock.lockref_put_or_lock.dput 0.70 ± 5% -0.1 0.63 ± 4% perf-profile.calltrace.cycles-pp.__lock_acquire.lock_acquire.lock_page_memcg.page_add_file_rmap.do_set_pte 0.74 ± 3% -0.1 0.68 ± 4% perf-profile.calltrace.cycles-pp.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.58 ± 5% -0.1 0.52 ± 2% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.setlocale 0.00 +0.6 0.56 ± 5% perf-profile.calltrace.cycles-pp.__schedule.schedule.worker_thread.kthread.ret_from_fork 0.00 +0.6 0.58 ± 4% perf-profile.calltrace.cycles-pp.schedule.worker_thread.kthread.ret_from_fork 0.00 +0.9 0.92 ± 2% perf-profile.calltrace.cycles-pp.lock_acquire.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work 3.56 ± 8% +1.6 5.19 ± 3% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 5.41 ± 8% +1.8 7.24 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 5.46 ± 8% +1.8 7.30 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 0.00 +1.9 1.95 ± 2% perf-profile.calltrace.cycles-pp.mem_cgroup_css_rstat_flush.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work 6.01 ± 9% +2.3 8.32 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 6.01 ± 9% +2.3 8.32 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 6.00 ± 9% +2.3 8.31 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 6.40 ± 9% +2.4 8.83 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 0.00 +4.7 4.70 ± 2% perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_locked.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work.worker_thread 0.00 +4.7 4.75 ± 2% perf-profile.calltrace.cycles-pp.cgroup_rstat_flush_irqsafe.mem_cgroup_flush_stats.process_one_work.worker_thread.kthread 0.00 +5.0 4.97 ± 2% perf-profile.calltrace.cycles-pp.mem_cgroup_flush_stats.process_one_work.worker_thread.kthread.ret_from_fork 0.51 ± 44% +5.3 5.83 perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 1.45 ± 3% +5.8 7.26 perf-profile.calltrace.cycles-pp.ret_from_fork 1.43 ± 3% +5.8 7.24 perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 0.62 ± 7% +5.9 6.49 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 68.58 -6.3 62.23 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 68.53 -6.3 62.18 perf-profile.children.cycles-pp.do_syscall_64 12.00 -2.2 9.77 ± 4% perf-profile.children.cycles-pp.__x64_sys_getdents64 11.84 -2.2 9.62 ± 4% perf-profile.children.cycles-pp.dcache_readdir 11.98 -2.2 9.75 ± 4% perf-profile.children.cycles-pp.iterate_dir 11.43 -2.2 9.26 ± 4% perf-profile.children.cycles-pp.scan_positives 21.24 -1.5 19.78 perf-profile.children.cycles-pp.asm_exc_page_fault 20.66 -1.3 19.32 perf-profile.children.cycles-pp.exc_page_fault 20.44 -1.3 19.12 perf-profile.children.cycles-pp.do_user_addr_fault 33.66 -1.2 32.41 perf-profile.children.cycles-pp.lock_acquire 16.79 -1.2 15.57 perf-profile.children.cycles-pp.__handle_mm_fault 11.56 ± 3% -1.2 10.37 ± 7% perf-profile.children.cycles-pp.execve 19.02 -1.1 17.89 perf-profile.children.cycles-pp.handle_mm_fault 9.69 ± 2% -1.1 8.61 perf-profile.children.cycles-pp.lock_page_memcg 11.75 -0.9 10.80 perf-profile.children.cycles-pp.check_preemption_disabled 23.40 -0.9 22.48 perf-profile.children.cycles-pp.rcu_read_lock_sched_held 8.81 -0.9 7.90 perf-profile.children.cycles-pp.do_fault 4.80 ± 3% -0.9 3.88 ± 17% perf-profile.children.cycles-pp.kill 7.80 -0.8 6.96 ± 2% perf-profile.children.cycles-pp.filemap_map_pages 11.79 -0.8 11.00 perf-profile.children.cycles-pp.__x64_sys_execve 11.65 -0.8 10.86 perf-profile.children.cycles-pp.do_execveat_common 6.16 -0.7 5.46 ± 2% perf-profile.children.cycles-pp.dput 18.41 -0.7 17.71 perf-profile.children.cycles-pp.lock_release 10.40 -0.6 9.78 perf-profile.children.cycles-pp.rcu_read_lock_held_common 5.50 ± 2% -0.6 4.88 ± 2% perf-profile.children.cycles-pp.do_set_pte 8.02 ± 3% -0.6 7.42 perf-profile.children.cycles-pp.unmap_vmas 7.92 ± 3% -0.6 7.33 perf-profile.children.cycles-pp.unmap_page_range 10.56 ± 2% -0.6 9.99 perf-profile.children.cycles-pp.mmput 17.03 -0.6 16.46 perf-profile.children.cycles-pp.__lock_acquire 10.54 ± 2% -0.6 9.97 perf-profile.children.cycles-pp.exit_mmap 7.72 ± 3% -0.6 7.16 perf-profile.children.cycles-pp.zap_pte_range 9.20 -0.5 8.66 perf-profile.children.cycles-pp.bprm_execve 17.96 -0.5 17.42 perf-profile.children.cycles-pp.lock_is_held_type 3.54 ± 2% -0.5 3.01 ± 2% perf-profile.children.cycles-pp.debug_lockdep_rcu_enabled 8.75 -0.5 8.23 ± 2% perf-profile.children.cycles-pp.load_elf_binary 8.85 -0.5 8.33 ± 2% perf-profile.children.cycles-pp.exec_binprm 4.53 -0.5 4.02 ± 2% perf-profile.children.cycles-pp.page_add_file_rmap 6.34 ± 2% -0.5 5.88 perf-profile.children.cycles-pp.do_sys_open 6.34 ± 2% -0.5 5.87 perf-profile.children.cycles-pp.do_sys_openat2 10.06 -0.4 9.63 perf-profile.children.cycles-pp.kmem_cache_alloc 3.29 -0.4 2.87 ± 2% perf-profile.children.cycles-pp.__x64_sys_kill 5.76 ± 3% -0.4 5.36 perf-profile.children.cycles-pp.do_filp_open 5.75 ± 3% -0.4 5.35 perf-profile.children.cycles-pp.path_openat 6.01 ± 3% -0.4 5.63 perf-profile.children.cycles-pp.page_remove_rmap 3.04 -0.4 2.66 ± 2% perf-profile.children.cycles-pp.kill_proc_info 2.86 -0.4 2.49 ± 2% perf-profile.children.cycles-pp.kill_pid_info 3.93 -0.4 3.56 perf-profile.children.cycles-pp.trace_hardirqs_on 3.40 -0.4 3.03 ± 2% perf-profile.children.cycles-pp.vm_mmap_pgoff 6.52 -0.4 6.17 ± 2% perf-profile.children.cycles-pp.__do_sys_clone 6.52 -0.4 6.17 ± 2% perf-profile.children.cycles-pp.kernel_clone 3.56 ± 2% -0.4 3.20 ± 2% perf-profile.children.cycles-pp.lockdep_hardirqs_on_prepare 6.34 -0.3 5.99 ± 2% perf-profile.children.cycles-pp.copy_process 3.13 -0.3 2.79 ± 2% perf-profile.children.cycles-pp.do_mmap 3.05 -0.3 2.72 ± 2% perf-profile.children.cycles-pp.mmap_region 3.31 ± 2% -0.3 2.98 ± 3% perf-profile.children.cycles-pp.___might_sleep 3.99 -0.3 3.67 perf-profile.children.cycles-pp.mark_lock 2.22 -0.3 1.91 perf-profile.children.cycles-pp.perf_event_mmap 2.51 -0.3 2.20 ± 2% perf-profile.children.cycles-pp.fs_reclaim_acquire 4.88 -0.3 4.57 perf-profile.children.cycles-pp.rcu_is_watching 2.86 ± 2% -0.3 2.55 perf-profile.children.cycles-pp.__alloc_pages 2.75 ± 3% -0.3 2.45 ± 2% perf-profile.children.cycles-pp.lockref_put_or_lock 5.31 -0.3 5.02 ± 2% perf-profile.children.cycles-pp.dup_mm 5.20 -0.3 4.91 ± 2% perf-profile.children.cycles-pp.dup_mmap 2.65 ± 3% -0.3 2.37 ± 2% perf-profile.children.cycles-pp.get_obj_cgroup_from_current 2.45 ± 2% -0.3 2.18 ± 4% perf-profile.children.cycles-pp.__might_fault 3.55 ± 2% -0.3 3.28 perf-profile.children.cycles-pp.walk_component 4.79 -0.3 4.54 perf-profile.children.cycles-pp.rcu_lockdep_current_cpu_online 2.52 -0.2 2.27 ± 2% perf-profile.children.cycles-pp.setlocale 3.08 ± 3% -0.2 2.83 perf-profile.children.cycles-pp.link_path_walk 2.58 ± 2% -0.2 2.33 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 1.21 ± 5% -0.2 0.98 ± 15% perf-profile.children.cycles-pp.unlink 6.20 -0.2 5.97 ± 2% perf-profile.children.cycles-pp.do_anonymous_page 1.61 ± 2% -0.2 1.39 ± 3% perf-profile.children.cycles-pp.do_send_sig_info 2.16 -0.2 1.95 ± 2% perf-profile.children.cycles-pp.vfs_statx 2.12 -0.2 1.91 ± 2% perf-profile.children.cycles-pp.__do_sys_newstat 2.88 ± 3% -0.2 2.67 ± 3% perf-profile.children.cycles-pp.rcu_read_lock_held 1.50 ± 4% -0.2 1.29 ± 3% perf-profile.children.cycles-pp.elf_map 2.14 ± 2% -0.2 1.94 ± 3% perf-profile.children.cycles-pp.filename_lookup 2.11 ± 2% -0.2 1.91 perf-profile.children.cycles-pp.mark_held_locks 2.24 -0.2 2.04 perf-profile.children.cycles-pp.ksys_mmap_pgoff 2.09 ± 2% -0.2 1.90 ± 3% perf-profile.children.cycles-pp.path_lookupat 1.35 ± 2% -0.2 1.16 ± 5% perf-profile.children.cycles-pp._raw_spin_lock_nested 1.58 -0.2 1.41 perf-profile.children.cycles-pp.alloc_pages_vma 2.00 ± 2% -0.2 1.84 perf-profile.children.cycles-pp.exit_to_user_mode_prepare 1.62 -0.2 1.45 ± 3% perf-profile.children.cycles-pp.__perf_sw_event 1.48 ± 5% -0.2 1.32 ± 4% perf-profile.children.cycles-pp.trace_hardirqs_off 1.17 ± 4% -0.2 1.01 ± 3% perf-profile.children.cycles-pp.lockdep_hardirqs_off 1.41 -0.2 1.25 ± 2% perf-profile.children.cycles-pp.do_brk_flags 1.16 ± 5% -0.1 1.02 ± 6% perf-profile.children.cycles-pp.do_unlinkat 4.74 -0.1 4.60 perf-profile.children.cycles-pp.__x64_sys_brk 1.20 ± 3% -0.1 1.06 ± 2% perf-profile.children.cycles-pp.down_write 1.64 ± 2% -0.1 1.51 perf-profile.children.cycles-pp.__open64_nocancel 2.86 ± 2% -0.1 2.73 ± 2% perf-profile.children.cycles-pp.__libc_fork 1.01 ± 3% -0.1 0.88 ± 5% perf-profile.children.cycles-pp.group_send_sig_info 1.22 ± 5% -0.1 1.09 ± 3% perf-profile.children.cycles-pp.step_into 0.89 ± 3% -0.1 0.76 ± 2% perf-profile.children.cycles-pp.get_user_arg_ptr 1.33 ± 4% -0.1 1.21 ± 2% perf-profile.children.cycles-pp.copy_strings 0.87 ± 3% -0.1 0.74 ± 3% perf-profile.children.cycles-pp.sync 1.16 ± 2% -0.1 1.04 ± 4% perf-profile.children.cycles-pp.get_mem_cgroup_from_mm 0.86 ± 3% -0.1 0.74 ± 3% perf-profile.children.cycles-pp.__x64_sys_sync 0.86 ± 3% -0.1 0.74 ± 3% perf-profile.children.cycles-pp.ksys_sync 1.26 ± 4% -0.1 1.14 ± 3% perf-profile.children.cycles-pp.arch_do_signal_or_restart 2.50 ± 2% -0.1 2.38 ± 3% perf-profile.children.cycles-pp.mem_cgroup_charge 0.77 ± 4% -0.1 0.65 ± 4% perf-profile.children.cycles-pp.d_path 0.81 ± 3% -0.1 0.69 ± 4% perf-profile.children.cycles-pp.iterate_supers 0.59 ± 4% -0.1 0.47 ± 5% perf-profile.children.cycles-pp.count 0.95 ± 5% -0.1 0.83 ± 3% perf-profile.children.cycles-pp.__mmap 1.10 ± 4% -0.1 0.99 ± 4% perf-profile.children.cycles-pp.getname_flags 0.87 -0.1 0.76 ± 3% perf-profile.children.cycles-pp.terminate_walk 2.02 ± 2% -0.1 1.91 perf-profile.children.cycles-pp.vm_area_dup 0.94 ± 3% -0.1 0.83 ± 4% perf-profile.children.cycles-pp.next_uptodate_page 0.82 ± 3% -0.1 0.71 ± 6% perf-profile.children.cycles-pp.__xstat64 1.79 ± 3% -0.1 1.68 ± 2% perf-profile.children.cycles-pp.d_alloc_parallel 0.93 ± 3% -0.1 0.82 ± 4% perf-profile.children.cycles-pp.perf_iterate_sb 0.66 ± 5% -0.1 0.56 ± 4% perf-profile.children.cycles-pp.__task_pid_nr_ns 1.12 ± 3% -0.1 1.02 ± 5% perf-profile.children.cycles-pp.__x64_sys_rt_sigreturn 1.91 ± 3% -0.1 1.80 ± 2% perf-profile.children.cycles-pp.__lookup_slow 0.66 ± 4% -0.1 0.55 ± 4% perf-profile.children.cycles-pp.syscall_enter_from_user_mode 0.84 ± 3% -0.1 0.74 ± 2% perf-profile.children.cycles-pp.obj_cgroup_charge 0.56 ± 4% -0.1 0.46 ± 8% perf-profile.children.cycles-pp.prepend_path 0.56 ± 6% -0.1 0.46 ± 18% perf-profile.children.cycles-pp.sighandler 2.33 -0.1 2.24 ± 2% perf-profile.children.cycles-pp.anon_vma_fork 0.88 ± 5% -0.1 0.79 ± 3% perf-profile.children.cycles-pp.__memcg_kmem_charge_page 0.68 ± 5% -0.1 0.59 ± 5% perf-profile.children.cycles-pp.send_signal 0.31 ± 13% -0.1 0.22 ± 12% perf-profile.children.cycles-pp.strnlen_user 0.98 ± 4% -0.1 0.90 ± 3% perf-profile.children.cycles-pp.__clear_user 0.85 ± 4% -0.1 0.77 perf-profile.children.cycles-pp.get_mem_cgroup_from_objcg 0.80 ± 5% -0.1 0.72 ± 4% perf-profile.children.cycles-pp.obj_cgroup_charge_pages 0.64 ± 4% -0.1 0.56 ± 3% perf-profile.children.cycles-pp.up_write 0.66 ± 3% -0.1 0.58 ± 5% perf-profile.children.cycles-pp.restore_sigcontext 0.85 ± 3% -0.1 0.77 ± 2% perf-profile.children.cycles-pp.lockdep_hardirqs_on 0.73 ± 4% -0.1 0.65 ± 4% perf-profile.children.cycles-pp.mem_cgroup_from_task 0.79 ± 4% -0.1 0.72 ± 4% perf-profile.children.cycles-pp.__x64_sys_mprotect 0.79 ± 4% -0.1 0.72 ± 4% perf-profile.children.cycles-pp.do_mprotect_pkey 0.54 ± 5% -0.1 0.47 ± 3% perf-profile.children.cycles-pp.lookup_fast 0.80 ± 4% -0.1 0.73 ± 7% perf-profile.children.cycles-pp.__x64_sys_link 0.54 ± 4% -0.1 0.47 ± 6% perf-profile.children.cycles-pp.try_to_unlazy 0.50 ± 6% -0.1 0.43 ± 7% perf-profile.children.cycles-pp.check_kill_permission 1.08 ± 2% -0.1 1.01 ± 3% perf-profile.children.cycles-pp.__split_vma 0.64 ± 5% -0.1 0.57 ± 4% perf-profile.children.cycles-pp.cgroup_throttle_swaprate 0.50 ± 4% -0.1 0.43 ± 3% perf-profile.children.cycles-pp.vma_merge 0.65 ± 2% -0.1 0.59 ± 6% perf-profile.children.cycles-pp.get_page_from_freelist 0.43 ± 5% -0.1 0.37 ± 6% perf-profile.children.cycles-pp.__legitimize_path 0.36 ± 2% -0.1 0.29 ± 3% perf-profile.children.cycles-pp._dl_addr 0.55 ± 4% -0.1 0.49 ± 5% perf-profile.children.cycles-pp.time 0.74 ± 3% -0.1 0.68 ± 4% perf-profile.children.cycles-pp.mprotect_fixup 0.46 ± 3% -0.1 0.40 ± 3% perf-profile.children.cycles-pp.__lock_task_sighand 0.45 ± 5% -0.1 0.39 ± 7% perf-profile.children.cycles-pp.copy_fpstate_to_sigframe 0.25 ± 6% -0.1 0.19 ± 8% perf-profile.children.cycles-pp.irqentry_exit_to_user_mode 0.38 ± 5% -0.1 0.32 ± 7% perf-profile.children.cycles-pp.do_wait 0.43 ± 2% -0.1 0.37 ± 4% perf-profile.children.cycles-pp.down_read 0.39 ± 5% -0.1 0.34 ± 7% perf-profile.children.cycles-pp.__do_sys_wait4 0.39 ± 5% -0.1 0.34 ± 7% perf-profile.children.cycles-pp.kernel_wait4 0.41 ± 6% -0.1 0.35 ± 6% perf-profile.children.cycles-pp.lockref_get_not_dead 0.46 ± 4% -0.1 0.40 ± 7% perf-profile.children.cycles-pp.trace_hardirqs_off_finish 0.08 ± 10% -0.1 0.03 ± 87% perf-profile.children.cycles-pp.load_balance 0.36 ± 4% -0.0 0.32 ± 4% perf-profile.children.cycles-pp.free_unref_page_list 0.33 ± 5% -0.0 0.29 ± 5% perf-profile.children.cycles-pp.__rcu_read_lock 0.50 ± 3% -0.0 0.45 ± 5% perf-profile.children.cycles-pp.preempt_count_add 0.61 ± 4% -0.0 0.57 ± 4% perf-profile.children.cycles-pp.lookup_open 0.40 ± 5% -0.0 0.35 ± 2% perf-profile.children.cycles-pp.__get_free_pages 0.25 ± 5% -0.0 0.21 ± 11% perf-profile.children.cycles-pp.audit_signal_info_syscall 0.30 ± 4% -0.0 0.27 ± 3% perf-profile.children.cycles-pp.tracer_hardirqs_on 0.26 ± 4% -0.0 0.23 ± 5% perf-profile.children.cycles-pp.kmem_cache_alloc_trace 0.24 ± 5% -0.0 0.21 ± 8% perf-profile.children.cycles-pp.wait4 0.27 ± 4% -0.0 0.24 ± 5% perf-profile.children.cycles-pp.__pud_alloc 0.09 ± 17% -0.0 0.06 ± 12% perf-profile.children.cycles-pp.do_task_dead 0.25 ± 4% -0.0 0.22 ± 9% perf-profile.children.cycles-pp.__cxa_atexit 0.17 ± 6% -0.0 0.14 ± 10% perf-profile.children.cycles-pp.__might_sleep 0.09 ± 11% -0.0 0.07 ± 13% perf-profile.children.cycles-pp.__x64_sys_unlink 0.06 ± 7% +0.0 0.09 ± 12% perf-profile.children.cycles-pp.delete_from_page_cache 0.07 ± 13% +0.0 0.10 ± 14% perf-profile.children.cycles-pp.truncate_inode_page 0.08 ± 12% +0.0 0.11 ± 22% perf-profile.children.cycles-pp.run_rebalance_domains 0.04 ± 71% +0.0 0.07 ± 15% perf-profile.children.cycles-pp.prandom_u32 0.08 ± 12% +0.1 0.13 ± 12% perf-profile.children.cycles-pp.update_rq_clock 0.06 ± 13% +0.1 0.11 ± 7% perf-profile.children.cycles-pp.enqueue_entity 0.05 ± 48% +0.1 0.10 ± 14% perf-profile.children.cycles-pp.lock_pin_lock 0.00 +0.1 0.06 ± 13% perf-profile.children.cycles-pp.select_idle_sibling 0.00 +0.1 0.06 ± 8% perf-profile.children.cycles-pp.unaccount_page_cache_page 0.02 ±141% +0.1 0.07 ± 15% perf-profile.children.cycles-pp.__delete_from_page_cache 0.00 +0.1 0.06 ± 12% perf-profile.children.cycles-pp.check_preempt_curr 0.08 ± 13% +0.1 0.14 ± 10% perf-profile.children.cycles-pp.cpuacct_charge 0.12 ± 7% +0.1 0.18 ± 13% perf-profile.children.cycles-pp.update_load_avg 0.07 ± 14% +0.1 0.15 ± 6% perf-profile.children.cycles-pp.enqueue_task_fair 0.00 +0.1 0.08 ± 12% perf-profile.children.cycles-pp.rcu_note_context_switch 0.06 ± 13% +0.1 0.14 ± 10% perf-profile.children.cycles-pp.preempt_schedule_common 0.00 +0.1 0.09 ± 8% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.06 ± 11% +0.1 0.15 ± 10% perf-profile.children.cycles-pp.preempt_schedule_thunk 0.00 +0.1 0.09 ± 12% perf-profile.children.cycles-pp.put_prev_entity 0.08 ± 14% +0.1 0.17 ± 6% perf-profile.children.cycles-pp.irqentry_exit 0.35 ± 7% +0.1 0.44 ± 6% perf-profile.children.cycles-pp.lock_page_lruvec_irqsave 0.58 ± 4% +0.1 0.67 ± 7% perf-profile.children.cycles-pp._raw_spin_unlock_irq 0.67 ± 2% +0.1 0.77 ± 3% perf-profile.children.cycles-pp.lru_add_drain 0.00 +0.1 0.11 ± 11% perf-profile.children.cycles-pp.perf_tp_event 0.00 +0.1 0.11 ± 10% perf-profile.children.cycles-pp.preempt_schedule_irq 0.22 ± 26% +0.1 0.33 ± 21% perf-profile.children.cycles-pp.tick_nohz_next_event 0.00 +0.1 0.11 ± 10% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template 0.13 ± 10% +0.1 0.24 ± 9% perf-profile.children.cycles-pp.pick_next_task_fair 0.44 ± 4% +0.1 0.55 ± 3% perf-profile.children.cycles-pp.lru_add_drain_cpu 0.28 ± 26% +0.1 0.41 ± 21% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 0.73 ± 3% +0.1 0.86 perf-profile.children.cycles-pp.__pagevec_lru_add_fn 0.26 ± 10% +0.1 0.40 ± 10% perf-profile.children.cycles-pp.finish_task_switch 0.02 ±142% +0.1 0.16 ± 6% perf-profile.children.cycles-pp.ttwu_do_activate 0.09 ± 18% +0.1 0.23 ± 6% perf-profile.children.cycles-pp.select_task_rq_fair 0.05 ± 45% +0.1 0.19 ± 8% perf-profile.children.cycles-pp.mem_cgroup_charge_statistics 0.00 +0.1 0.15 ± 14% perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi 0.09 ± 15% +0.1 0.24 ± 8% perf-profile.children.cycles-pp.dequeue_entity 0.52 ± 4% +0.2 0.68 ± 3% perf-profile.children.cycles-pp.page_add_new_anon_rmap 0.10 ± 13% +0.2 0.27 ± 8% perf-profile.children.cycles-pp.dequeue_task_fair 0.37 ± 27% +0.2 0.54 ± 21% perf-profile.children.cycles-pp.menu_select 0.00 +0.2 0.18 ± 4% perf-profile.children.cycles-pp._find_next_bit 0.26 ± 7% +0.2 0.45 ± 5% perf-profile.children.cycles-pp.update_curr 1.12 ± 2% +0.2 1.32 perf-profile.children.cycles-pp.__pagevec_lru_add 0.14 ± 16% +0.2 0.34 ± 12% perf-profile.children.cycles-pp.schedule_idle 0.64 ± 4% +0.5 1.09 ± 4% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.28 ± 7% +0.5 0.78 ± 3% perf-profile.children.cycles-pp.schedule 1.06 ± 2% +0.6 1.61 ± 3% perf-profile.children.cycles-pp.__mod_lruvec_page_state 0.16 ± 4% +0.7 0.83 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 0.59 ± 8% +0.9 1.45 ± 3% perf-profile.children.cycles-pp.__schedule 0.00 +0.9 0.90 ± 2% perf-profile.children.cycles-pp.__queue_work 0.00 +1.0 0.97 ± 2% perf-profile.children.cycles-pp.queue_work_on 0.37 ± 4% +1.1 1.46 ± 3% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state 0.07 ± 18% +1.3 1.37 ± 4% perf-profile.children.cycles-pp.cgroup_rstat_updated 0.92 ± 2% +1.4 2.30 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 3.78 ± 8% +1.7 5.50 ± 3% perf-profile.children.cycles-pp.intel_idle 5.82 ± 9% +1.9 7.75 ± 4% perf-profile.children.cycles-pp.cpuidle_enter 5.81 ± 9% +1.9 7.75 ± 4% perf-profile.children.cycles-pp.cpuidle_enter_state 0.00 +2.0 1.96 ± 2% perf-profile.children.cycles-pp.mem_cgroup_css_rstat_flush 6.01 ± 9% +2.3 8.32 ± 3% perf-profile.children.cycles-pp.start_secondary 6.40 ± 9% +2.4 8.83 ± 4% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 6.40 ± 9% +2.4 8.83 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry 6.40 ± 9% +2.4 8.83 ± 4% perf-profile.children.cycles-pp.do_idle 0.00 +4.7 4.71 ± 2% perf-profile.children.cycles-pp.cgroup_rstat_flush_locked 0.00 +4.7 4.75 ± 2% perf-profile.children.cycles-pp.cgroup_rstat_flush_irqsafe 0.00 +5.0 4.97 ± 2% perf-profile.children.cycles-pp.mem_cgroup_flush_stats 0.59 ± 8% +5.2 5.83 perf-profile.children.cycles-pp.process_one_work 1.54 ± 3% +5.8 7.35 perf-profile.children.cycles-pp.ret_from_fork 1.43 ± 3% +5.8 7.24 perf-profile.children.cycles-pp.kthread 0.62 ± 7% +5.9 6.49 perf-profile.children.cycles-pp.worker_thread 10.94 -0.9 10.05 perf-profile.self.cycles-pp.check_preemption_disabled 3.13 ± 2% -0.5 2.67 ± 2% perf-profile.self.cycles-pp.debug_lockdep_rcu_enabled 3.82 -0.3 3.51 perf-profile.self.cycles-pp.mark_lock 11.35 -0.2 11.11 perf-profile.self.cycles-pp.lock_is_held_type 4.30 -0.2 4.13 ± 2% perf-profile.self.cycles-pp.lock_acquire 2.89 -0.1 2.75 perf-profile.self.cycles-pp.rcu_is_watching 2.00 ± 2% -0.1 1.90 ± 3% perf-profile.self.cycles-pp.rcu_read_lock_sched_held 0.90 -0.1 0.80 ± 3% perf-profile.self.cycles-pp.lockdep_hardirqs_on_prepare 0.60 ± 6% -0.1 0.53 ± 4% perf-profile.self.cycles-pp.lockdep_hardirqs_off 0.93 ± 2% -0.1 0.86 ± 3% perf-profile.self.cycles-pp.mark_held_locks 0.58 ± 5% -0.1 0.52 ± 4% perf-profile.self.cycles-pp.lockdep_hardirqs_on 0.64 -0.1 0.58 ± 4% perf-profile.self.cycles-pp.___might_sleep 0.33 ± 3% -0.1 0.27 ± 4% perf-profile.self.cycles-pp._dl_addr 0.30 ± 5% -0.0 0.25 ± 4% perf-profile.self.cycles-pp.__rcu_read_lock 0.30 ± 7% -0.0 0.26 ± 8% perf-profile.self.cycles-pp.next_uptodate_page 0.07 ± 5% -0.0 0.04 ± 63% perf-profile.self.cycles-pp.down_write 0.23 ± 7% -0.0 0.19 ± 8% perf-profile.self.cycles-pp.lock_page_memcg 0.25 ± 4% -0.0 0.22 ± 3% perf-profile.self.cycles-pp.tracer_hardirqs_on 0.16 ± 7% -0.0 0.13 ± 12% perf-profile.self.cycles-pp.page_remove_rmap 0.10 ± 10% -0.0 0.08 ± 15% perf-profile.self.cycles-pp.trace_hardirqs_on_prepare 0.06 ± 7% -0.0 0.05 ± 9% perf-profile.self.cycles-pp.__rb_insert_augmented 0.10 ± 11% +0.0 0.12 ± 6% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state 0.00 +0.1 0.07 ± 16% perf-profile.self.cycles-pp.__schedule 0.07 ± 18% +0.1 0.18 ± 15% perf-profile.self.cycles-pp.cgroup_rstat_updated 0.00 +0.2 0.17 ± 4% perf-profile.self.cycles-pp._find_next_bit 4.45 +0.2 4.67 perf-profile.self.cycles-pp.lock_release 0.00 +0.3 0.30 ± 6% perf-profile.self.cycles-pp.cgroup_rstat_flush_locked 3.78 ± 8% +1.7 5.50 ± 3% perf-profile.self.cycles-pp.intel_idle 0.00 +1.8 1.80 ± 2% perf-profile.self.cycles-pp.mem_cgroup_css_rstat_flush Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang