Greeting, FYI, we noticed a -99.9% regression of stress-ng.loop.ops_per_sec due to commit: commit: c76f48eb5c084b1e15c931ae8cc1826cd771d70d ("block: take bd_mutex around delete_partitions in del_gendisk") https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master in testcase: stress-ng on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory with following parameters: nr_threads: 10% disk: 1HDD testtime: 60s fs: ext4 class: os test: loop cpufreq_governor: performance ucode: 0x5003006 If you fix the issue, kindly add following tag Reported-by: kernel test robot Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml bin/lkp run compatible-job.yaml ========================================================================================= class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode: os/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/loop/stress-ng/60s/0x5003006 commit: d3c4a43d92 ("block: refactor blk_drop_partitions") c76f48eb5c ("block: take bd_mutex around delete_partitions in del_gendisk") d3c4a43d9291279c c76f48eb5c084b1e15c931ae8cc ---------------- --------------------------- %stddev %change %stddev \ | \ 1464 -99.2% 12.00 ± 33% stress-ng.loop.ops 24.35 -99.9% 0.03 ± 41% stress-ng.loop.ops_per_sec 62.28 +789.4% 553.94 ± 13% stress-ng.time.elapsed_time 62.28 +789.4% 553.94 ± 13% stress-ng.time.elapsed_time.max 3671 -99.1% 32.20 ± 27% stress-ng.time.involuntary_context_switches 9792 -1.6% 9636 stress-ng.time.minor_page_faults 36.17 -100.0% 0.00 stress-ng.time.percent_of_cpu_this_job_got 22.65 ± 2% -98.9% 0.26 ± 35% stress-ng.time.system_time 59875 ± 14% -98.6% 851.60 ±110% stress-ng.time.voluntary_context_switches 2189 ± 2% -2.9% 2125 boot-time.idle 102.01 +482.0% 593.67 ± 12% uptime.boot 9211 +501.5% 55405 ± 11% uptime.idle 0.60 ± 3% -0.6 0.02 ± 12% mpstat.cpu.all.iowait% 1.27 +0.6 1.86 ± 27% mpstat.cpu.all.irq% 0.59 -0.6 0.03 ± 2% mpstat.cpu.all.sys% 0.07 ± 2% -0.1 0.01 ± 2% mpstat.cpu.all.usr% 179096 ± 16% +272.4% 666980 ± 17% numa-numastat.node0.local_node 224078 ± 5% +220.9% 718975 ± 19% numa-numastat.node0.numa_hit 155998 ± 18% +281.6% 595329 ± 33% numa-numastat.node1.local_node 199503 ± 4% +215.8% 630069 ± 31% numa-numastat.node1.numa_hit 1598 ± 3% -99.9% 1.20 ±122% vmstat.io.bi 5044 -66.3% 1700 ± 2% vmstat.memory.buff 1.00 -100.0% 0.00 vmstat.procs.r 8439 ± 4% -83.8% 1367 ± 6% vmstat.system.cs 5636242 -49.3% 2860359 ± 6% cpuidle.C1.time 60648 ± 2% -25.4% 45246 ± 7% cpuidle.C1.usage 5.204e+09 ± 22% +607.9% 3.684e+10 ± 41% cpuidle.C1E.time 11235694 ± 14% +684.5% 88145411 ± 16% cpuidle.C1E.usage 41714 ± 6% +162.2% 109376 ± 11% cpuidle.POLL.usage 9288 +58.8% 14748 ± 4% meminfo.Active 3197 +296.2% 12670 ± 5% meminfo.Active(anon) 6089 -65.9% 2076 ± 5% meminfo.Active(file) 86058 ± 5% +107.8% 178815 meminfo.AnonHugePages 4981 -65.9% 1698 ± 2% meminfo.Buffers 825613 -12.4% 723628 ± 3% meminfo.Committed_AS 16237013 ± 3% -11.0% 14454104 ± 8% meminfo.DirectMap2M 18587 -12.6% 16252 meminfo.KernelStack 2850592 -17.0% 2366313 meminfo.Memused 7.00 +26105.7% 1834 ± 24% meminfo.Mlocked 5522 -14.7% 4711 meminfo.PageTables 23075 +42.7% 32929 ± 2% meminfo.Shmem 915.33 ± 33% -73.1% 245.80 ± 23% numa-vmstat.node0.nr_active_file 40463 ± 35% -39.9% 24330 ± 69% numa-vmstat.node0.nr_anon_pages 42808 ± 31% -36.5% 27171 ± 57% numa-vmstat.node0.nr_inactive_anon 0.00 +2.5e+104% 247.20 ± 26% numa-vmstat.node0.nr_mlock 797.33 ± 28% -31.1% 549.60 ± 24% numa-vmstat.node0.nr_page_table_pages 12722 ± 3% -13.8% 10971 ± 9% numa-vmstat.node0.nr_slab_reclaimable 27196 ± 8% -13.5% 23535 ± 8% numa-vmstat.node0.nr_slab_unreclaimable 915.33 ± 33% -73.1% 245.80 ± 23% numa-vmstat.node0.nr_zone_active_file 42808 ± 31% -36.5% 27171 ± 57% numa-vmstat.node0.nr_zone_inactive_anon 849837 ± 15% +26.7% 1076564 ± 9% numa-vmstat.node0.numa_local 457.67 ± 20% +399.8% 2287 ± 17% numa-vmstat.node1.nr_active_anon 8770 ± 8% -15.2% 7437 ± 3% numa-vmstat.node1.nr_kernel_stack 0.00 +2.1e+104% 210.40 ± 29% numa-vmstat.node1.nr_mlock 457.67 ± 20% +399.8% 2287 ± 17% numa-vmstat.node1.nr_zone_active_anon 3665 ± 33% -73.1% 985.00 ± 23% numa-meminfo.node0.Active(file) 161847 ± 35% -39.9% 97309 ± 69% numa-meminfo.node0.AnonPages 171672 ± 31% -36.4% 109247 ± 57% numa-meminfo.node0.Inactive 171227 ± 31% -36.5% 108673 ± 57% numa-meminfo.node0.Inactive(anon) 50893 ± 3% -13.8% 43888 ± 9% numa-meminfo.node0.KReclaimable 1526012 ± 4% -26.1% 1127392 ± 12% numa-meminfo.node0.MemUsed 3.00 +32893.3% 989.80 ± 26% numa-meminfo.node0.Mlocked 3195 ± 28% -31.4% 2193 ± 24% numa-meminfo.node0.PageTables 50893 ± 3% -13.8% 43888 ± 9% numa-meminfo.node0.SReclaimable 108773 ± 8% -13.5% 94141 ± 8% numa-meminfo.node0.SUnreclaim 159668 ± 6% -13.6% 138030 ± 6% numa-meminfo.node0.Slab 4255 ± 29% +140.8% 10246 ± 14% numa-meminfo.node1.Active 1829 ± 20% +400.4% 9154 ± 17% numa-meminfo.node1.Active(anon) 8775 ± 8% -15.2% 7439 ± 3% numa-meminfo.node1.KernelStack 3.00 +28053.3% 844.60 ± 29% numa-meminfo.node1.Mlocked 799.67 +296.1% 3167 ± 5% proc-vmstat.nr_active_anon 1522 -65.9% 518.80 ± 5% proc-vmstat.nr_active_file 63795 -8.3% 58521 proc-vmstat.nr_anon_pages 87.00 ± 28% +78.9% 155.60 ± 28% proc-vmstat.nr_dirtied 68723 -7.6% 63478 proc-vmstat.nr_inactive_anon 18586 -12.6% 16247 proc-vmstat.nr_kernel_stack 10029 -4.4% 9587 proc-vmstat.nr_mapped 1.00 +45720.0% 458.20 ± 24% proc-vmstat.nr_mlock 1380 -14.9% 1174 proc-vmstat.nr_page_table_pages 5765 +42.8% 8231 ± 2% proc-vmstat.nr_shmem 24003 -5.8% 22600 proc-vmstat.nr_slab_reclaimable 49519 -7.8% 45650 proc-vmstat.nr_slab_unreclaimable 77.50 ± 31% +95.1% 151.20 ± 29% proc-vmstat.nr_written 799.67 +296.1% 3167 ± 5% proc-vmstat.nr_zone_active_anon 1522 -65.9% 518.80 ± 5% proc-vmstat.nr_zone_active_file 68723 -7.6% 63478 proc-vmstat.nr_zone_inactive_anon 971.50 ± 30% -58.3% 405.00 ± 51% proc-vmstat.numa_hint_faults 457086 +202.7% 1383638 ± 10% proc-vmstat.numa_hit 368542 +251.9% 1296853 ± 11% proc-vmstat.numa_local 88543 -2.0% 86784 proc-vmstat.numa_other 9561 ± 2% -60.7% 3757 ± 5% proc-vmstat.pgactivate 553335 +163.0% 1455536 ± 10% proc-vmstat.pgalloc_normal 220882 +623.5% 1598121 ± 12% proc-vmstat.pgfault 415665 +247.1% 1442871 ± 10% proc-vmstat.pgfree 104075 ± 3% -99.1% 916.80 ± 69% proc-vmstat.pgpgin 451476 ± 7% +734.6% 3767973 ± 15% proc-vmstat.pgpgout 14210 +649.7% 106533 ± 12% proc-vmstat.pgreuse 8304 ± 82% +301.2% 33317 ± 9% sched_debug.cfs_rq:/.load.avg 46081 ±116% +188.8% 133105 ± 7% sched_debug.cfs_rq:/.load.stddev 178.19 ± 31% -73.7% 46.80 ± 10% sched_debug.cfs_rq:/.load_avg.avg 6121 ± 14% -76.4% 1442 ± 11% sched_debug.cfs_rq:/.load_avg.max 751.44 ± 17% -72.0% 210.71 ± 9% sched_debug.cfs_rq:/.load_avg.stddev 0.17 ± 17% -58.6% 0.07 ± 8% sched_debug.cfs_rq:/.nr_running.avg 0.37 ± 7% -36.2% 0.24 ± 3% sched_debug.cfs_rq:/.nr_running.stddev 393.19 ± 8% -83.9% 63.44 ± 6% sched_debug.cfs_rq:/.runnable_avg.avg 1333 ± 17% -51.9% 641.22 ± 3% sched_debug.cfs_rq:/.runnable_avg.max 317.55 ± 10% -61.0% 123.72 ± 4% sched_debug.cfs_rq:/.runnable_avg.stddev 392.44 ± 8% -83.9% 63.32 ± 6% sched_debug.cfs_rq:/.util_avg.avg 1330 ± 17% -51.9% 640.31 ± 3% sched_debug.cfs_rq:/.util_avg.max 316.96 ± 10% -61.0% 123.61 ± 4% sched_debug.cfs_rq:/.util_avg.stddev 37.95 ± 33% -86.6% 5.08 ± 34% sched_debug.cfs_rq:/.util_est_enqueued.avg 518.00 ± 5% -79.3% 107.10 ± 25% sched_debug.cfs_rq:/.util_est_enqueued.max 121.83 ± 18% -83.3% 20.33 ± 23% sched_debug.cfs_rq:/.util_est_enqueued.stddev 740981 ± 2% +26.7% 938799 sched_debug.cpu.avg_idle.avg 1996 ± 23% +6329.5% 128386 ± 56% sched_debug.cpu.avg_idle.min 306945 ± 5% -53.4% 143028 ± 13% sched_debug.cpu.avg_idle.stddev 38618 +637.1% 284649 ± 15% sched_debug.cpu.clock.avg 38623 +637.0% 284653 ± 15% sched_debug.cpu.clock.max 38613 +637.2% 284645 ± 15% sched_debug.cpu.clock.min 38490 +626.0% 279434 ± 14% sched_debug.cpu.clock_task.avg 38609 +624.6% 279743 ± 14% sched_debug.cpu.clock_task.max 33619 +716.4% 274485 ± 15% sched_debug.cpu.clock_task.min 286.92 ± 19% -51.0% 140.52 ± 7% sched_debug.cpu.curr->pid.avg 2432 +254.8% 8630 ± 12% sched_debug.cpu.curr->pid.max 755.23 ± 8% +28.1% 967.57 ± 10% sched_debug.cpu.curr->pid.stddev 897805 ± 31% -40.4% 535075 sched_debug.cpu.max_idle_balance_cost.max 55398 ± 60% -89.2% 5981 ± 39% sched_debug.cpu.max_idle_balance_cost.stddev 0.00 ± 13% -48.9% 0.00 ± 8% sched_debug.cpu.next_balance.stddev 0.13 ± 18% -73.1% 0.03 ± 11% sched_debug.cpu.nr_running.avg 0.33 ± 7% -49.8% 0.17 ± 3% sched_debug.cpu.nr_running.stddev 1577 +225.4% 5133 ± 10% sched_debug.cpu.nr_switches.avg 10235 ± 13% +526.1% 64083 ± 22% sched_debug.cpu.nr_switches.max 471.33 ± 17% +200.5% 1416 ± 16% sched_debug.cpu.nr_switches.min 1438 ± 6% +476.3% 8287 ± 14% sched_debug.cpu.nr_switches.stddev 0.01 ± 31% +854.0% 0.12 ± 4% sched_debug.cpu.nr_uninterruptible.avg 38615 +637.1% 284646 ± 15% sched_debug.cpu_clk 38121 +645.4% 284151 ± 15% sched_debug.ktime 38979 +631.2% 285008 ± 15% sched_debug.sched_clk 11.12 ± 2% +175.9% 30.68 ±104% perf-stat.i.MPKI 3.418e+08 -49.6% 1.723e+08 ± 7% perf-stat.i.branch-instructions 1.34 ± 4% +2.2 3.49 ±107% perf-stat.i.branch-miss-rate% 8485 ± 4% -84.2% 1344 ± 6% perf-stat.i.context-switches 3.44 ± 3% +52.6% 5.25 ± 28% perf-stat.i.cpi 160.47 -39.2% 97.62 perf-stat.i.cpu-migrations 4.201e+08 -46.4% 2.254e+08 ± 3% perf-stat.i.dTLB-loads 2.097e+08 -41.9% 1.217e+08 ± 2% perf-stat.i.dTLB-stores 2039329 -16.7% 1698838 ± 21% perf-stat.i.iTLB-loads 1.625e+09 -49.1% 8.277e+08 ± 7% perf-stat.i.instructions 1029 ± 4% -41.2% 605.47 ± 22% perf-stat.i.instructions-per-iTLB-miss 0.30 ± 3% -31.8% 0.21 ± 22% perf-stat.i.ipc 11.18 -85.5% 1.62 ± 20% perf-stat.i.major-faults 1.64 ± 9% -40.8% 0.97 ± 17% perf-stat.i.metric.K/sec 10.33 -45.0% 5.69 ± 5% perf-stat.i.metric.M/sec 3162 -11.6% 2796 perf-stat.i.minor-faults 500094 ± 3% -59.2% 203915 ± 34% perf-stat.i.node-load-misses 86458 ± 5% -65.8% 29541 ± 55% perf-stat.i.node-loads 273470 -86.8% 36116 ± 14% perf-stat.i.node-store-misses 60952 ± 9% -86.2% 8425 ±101% perf-stat.i.node-stores 3205 -12.7% 2797 perf-stat.i.page-faults 10.58 ± 2% +182.6% 29.91 ±104% perf-stat.overall.MPKI 1.37 ± 3% +2.1 3.48 ±104% perf-stat.overall.branch-miss-rate% 3.29 ± 3% +55.9% 5.13 ± 28% perf-stat.overall.cpi 1034 ± 4% -41.2% 608.25 ± 22% perf-stat.overall.instructions-per-iTLB-miss 0.30 ± 3% -31.6% 0.21 ± 22% perf-stat.overall.ipc 3.363e+08 -48.9% 1.72e+08 ± 7% perf-stat.ps.branch-instructions 8351 ± 4% -83.9% 1342 ± 6% perf-stat.ps.context-switches 94471 +1.4% 95822 perf-stat.ps.cpu-clock 157.89 -38.3% 97.44 perf-stat.ps.cpu-migrations 4.134e+08 -45.6% 2.25e+08 ± 3% perf-stat.ps.dTLB-loads 2.063e+08 -41.1% 1.215e+08 ± 2% perf-stat.ps.dTLB-stores 2006862 -15.5% 1695650 ± 21% perf-stat.ps.iTLB-loads 1.6e+09 -48.3% 8.263e+08 ± 7% perf-stat.ps.instructions 11.01 -85.2% 1.63 ± 20% perf-stat.ps.major-faults 3112 -10.3% 2791 perf-stat.ps.minor-faults 492165 ± 3% -58.6% 203545 ± 33% perf-stat.ps.node-load-misses 85077 ± 5% -65.3% 29492 ± 55% perf-stat.ps.node-loads 269158 -86.6% 36060 ± 14% perf-stat.ps.node-store-misses 59987 ± 9% -86.0% 8420 ±101% perf-stat.ps.node-stores 3154 -11.5% 2793 perf-stat.ps.page-faults 94471 +1.4% 95822 perf-stat.ps.task-clock 1.016e+11 +350.5% 4.579e+11 ± 13% perf-stat.total.instructions 3935 ± 4% -57.2% 1684 ± 6% slabinfo.Acpi-Parse.active_objs 3935 ± 4% -57.2% 1684 ± 6% slabinfo.Acpi-Parse.num_objs 1575 -84.5% 244.20 ± 6% slabinfo.bdev_cache.active_objs 1575 -84.5% 244.20 ± 6% slabinfo.bdev_cache.num_objs 323.00 ± 11% -33.7% 214.00 ± 22% slabinfo.biovec-128.num_objs 2514 ± 4% -42.9% 1435 ± 4% slabinfo.buffer_head.active_objs 2535 ± 4% -43.4% 1435 ± 4% slabinfo.buffer_head.num_objs 4061 ± 5% -20.4% 3232 ± 7% slabinfo.dmaengine-unmap-16.active_objs 4072 ± 5% -20.6% 3233 ± 7% slabinfo.dmaengine-unmap-16.num_objs 2479 -90.3% 240.20 ± 28% slabinfo.dquot.active_objs 2479 -90.3% 240.20 ± 28% slabinfo.dquot.num_objs 5213 ± 13% -68.9% 1620 ± 23% slabinfo.ext4_extent_status.active_objs 5213 ± 13% -68.9% 1620 ± 23% slabinfo.ext4_extent_status.num_objs 4.67 ±127% +4400.0% 210.00 ± 36% slabinfo.ext4_pending_reservation.active_objs 4.67 ±127% +4400.0% 210.00 ± 36% slabinfo.ext4_pending_reservation.num_objs 2360 ± 2% -49.0% 1204 ± 5% slabinfo.file_lock_cache.active_objs 2360 ± 2% -49.0% 1204 ± 5% slabinfo.file_lock_cache.num_objs 7031 ± 5% -47.0% 3729 ± 14% slabinfo.fsnotify_mark_connector.active_objs 7031 ± 5% -47.0% 3729 ± 14% slabinfo.fsnotify_mark_connector.num_objs 1417 -9.7% 1280 slabinfo.inode_cache.active_slabs 76560 -9.7% 69145 slabinfo.inode_cache.num_objs 1417 -9.7% 1280 slabinfo.inode_cache.num_slabs 99699 -14.6% 85176 slabinfo.kernfs_node_cache.active_objs 3119 -14.7% 2661 slabinfo.kernfs_node_cache.active_slabs 99815 -14.7% 85176 slabinfo.kernfs_node_cache.num_objs 3119 -14.7% 2661 slabinfo.kernfs_node_cache.num_slabs 44972 -18.9% 36462 slabinfo.kmalloc-16.num_objs 7059 ± 2% -18.4% 5763 ± 2% slabinfo.kmalloc-192.num_objs 6610 ± 2% -12.7% 5771 ± 2% slabinfo.kmalloc-1k.active_objs 7524 ± 2% -21.1% 5937 slabinfo.kmalloc-1k.num_objs 9426 ± 2% -8.0% 8676 ± 3% slabinfo.kmalloc-256.active_objs 9582 ± 2% -8.8% 8737 ± 3% slabinfo.kmalloc-256.num_objs 7902 ± 2% -9.3% 7170 ± 3% slabinfo.kmalloc-2k.active_objs 8225 ± 2% -11.7% 7265 ± 3% slabinfo.kmalloc-2k.num_objs 2221 -12.4% 1945 slabinfo.kmalloc-4k.num_objs 6092 +29.2% 7873 ± 3% slabinfo.kmalloc-96.active_objs 7021 ± 2% +13.9% 7994 ± 3% slabinfo.kmalloc-96.num_objs 3282 -84.4% 513.20 ± 6% slabinfo.kmalloc-rcl-192.active_objs 3282 -84.4% 513.20 ± 6% slabinfo.kmalloc-rcl-192.num_objs 5514 ± 2% -11.6% 4875 slabinfo.kmalloc-rcl-512.active_objs 5518 ± 2% -11.6% 4876 slabinfo.kmalloc-rcl-512.num_objs 6565 ± 3% -21.8% 5131 ± 8% slabinfo.kmalloc-rcl-64.active_objs 6565 ± 3% -21.8% 5131 ± 8% slabinfo.kmalloc-rcl-64.num_objs 2880 ± 6% -19.0% 2332 ± 11% slabinfo.kmalloc-rcl-96.active_objs 2880 ± 6% -19.0% 2332 ± 11% slabinfo.kmalloc-rcl-96.num_objs 6217 -77.1% 1424 ± 21% slabinfo.mbcache.active_objs 6217 -77.1% 1424 ± 21% slabinfo.mbcache.num_objs 13404 +13.2% 15170 slabinfo.proc_inode_cache.active_objs 13413 +13.3% 15197 slabinfo.proc_inode_cache.num_objs 1246 -88.5% 142.80 ± 13% slabinfo.request_queue.active_objs 1246 -88.5% 142.80 ± 13% slabinfo.request_queue.num_objs 528.17 ± 7% -33.5% 351.00 ± 13% slabinfo.skbuff_fclone_cache.active_objs 528.17 ± 7% -33.5% 351.00 ± 13% slabinfo.skbuff_fclone_cache.num_objs 1399 -9.4% 1268 slabinfo.task_struct.active_objs 1403 -9.2% 1274 slabinfo.task_struct.active_slabs 1403 -9.2% 1274 slabinfo.task_struct.num_objs 1403 -9.2% 1274 slabinfo.task_struct.num_slabs 5015 ± 3% -23.2% 3852 ± 3% slabinfo.trace_event_file.active_objs 5015 ± 3% -23.2% 3852 ± 3% slabinfo.trace_event_file.num_objs 11008 +54.0% 16950 ± 8% slabinfo.vmap_area.active_objs 11022 +55.3% 17115 ± 8% slabinfo.vmap_area.num_objs 0.00 ±223% +1820.0% 0.01 ± 34% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64 0.00 ± 56% +232.3% 0.01 ± 26% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.00 ±152% +428.0% 0.01 ± 30% perf-sched.sch_delay.avg.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread 0.00 ±141% +644.0% 0.01 ± 31% perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io.submit_bio_wait 0.00 ±223% +753.3% 0.01 ± 32% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.00 ± 22% +268.3% 0.02 ± 98% perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 0.00 ±223% +2930.0% 0.02 ± 28% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64 0.01 ± 63% +550.0% 0.04 ± 61% perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.00 ±152% +756.0% 0.02 ± 35% perf-sched.sch_delay.max.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread 0.00 ±141% +920.0% 0.02 ± 22% perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io.submit_bio_wait 0.00 ±223% +1126.7% 0.02 ± 25% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.01 ± 30% +25140.0% 2.02 ±197% perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 0.18 ± 25% +1.2e+05% 219.77 ± 10% perf-sched.total_wait_and_delay.average.ms 332.83 ± 18% +1933.6% 6768 ± 10% perf-sched.total_wait_and_delay.count.ms 14.17 ± 33% +70409.9% 9992 perf-sched.total_wait_and_delay.max.ms 0.16 ± 31% +1.3e+05% 219.43 ± 10% perf-sched.total_wait_time.average.ms 13.18 ± 43% +75740.1% 9992 perf-sched.total_wait_time.max.ms 0.00 ±223% +1.2e+08% 818.24 perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64 0.25 ±123% +1.1e+05% 272.84 ± 5% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.02 ±146% +3853.7% 0.75 ± 2% perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.04 ± 43% +2.2e+05% 76.70 ± 5% perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read 0.00 ±141% +1.7e+07% 289.31 ± 12% perf-sched.wait_and_delay.avg.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io.submit_bio_wait 0.00 ±223% +3.2e+07% 478.42 perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 2.71 ± 35% +583.0% 18.48 ± 61% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 0.06 ±132% +1.3e+06% 723.00 ± 3% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 1.48 ± 92% +30586.5% 453.90 ± 7% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork 0.50 ±100% +2100.0% 11.00 perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64 5.67 ± 13% +4223.5% 245.00 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.83 ±107% +29108.0% 243.40 perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 104.83 ± 58% +1907.8% 2104 ± 5% perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read 96.33 +956.3% 1017 ± 4% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr 0.33 ±141% +9920.0% 33.40 ± 14% perf-sched.wait_and_delay.count.schedule_timeout.io_schedule_timeout.wait_for_completion_io.submit_bio_wait 0.17 ±223% +23540.0% 39.40 ± 2% perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork 4.33 ± 31% +20036.9% 872.60 ± 79% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 97.00 +1149.9% 1212 ± 3% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork 5.50 ± 20% +10583.6% 587.60 ± 2% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork 0.00 ±223% +1.5e+08% 1000 perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64 0.97 ±137% +3.9e+05% 3771 ± 65% perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.03 ±148% +40301.4% 13.40 perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.44 ±101% +2.3e+05% 1006 perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read 0.00 ±223% +2.1e+08% 9992 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr 0.00 ±141% +1.8e+07% 301.75 ± 12% perf-sched.wait_and_delay.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io.submit_bio_wait 0.00 ±223% +3.4e+07% 505.03 perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 4.46 ± 25% +6607.2% 299.40 ± 48% perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 4.80 ±147% +31635.7% 1522 ± 67% perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork 7.15 ±103% +1.3e+05% 8972 perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork 0.25 ±124% +1.1e+05% 272.84 ± 5% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.02 ±149% +4656.6% 0.75 ± 2% perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.03 ± 51% +2.6e+05% 76.70 ± 5% perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read 2.08 ± 21% +2272.6% 49.41 ±143% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 2.70 ± 35% +583.6% 18.46 ± 61% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 0.03 ±222% +2.4e+06% 722.99 ± 3% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 0.93 ±150% +48396.0% 450.37 ± 6% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork 0.97 ±137% +3.9e+05% 3771 ± 65% perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.03 ±149% +42879.3% 13.40 perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.43 ±106% +2.3e+05% 1006 perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read 0.00 ±223% +2.1e+08% 9992 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr 9.79 ± 29% +4253.2% 426.40 ± 19% perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 2.42 ±141% +281.7% 9.22 ± 2% perf-sched.wait_time.max.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread 4.46 ± 25% +6613.9% 299.37 ± 48% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 2.96 ±222% +51425.8% 1522 ± 67% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork 5.46 ±154% +1.6e+05% 8972 perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork 128.00 +768.1% 1111 ± 12% interrupts.9:IO-APIC.9-fasteoi.acpi 64630 +21.1% 78254 ± 3% interrupts.CAL:Function_call_interrupts 127590 +770.8% 1111103 ± 13% interrupts.CPU0.LOC:Local_timer_interrupts 128.00 +768.1% 1111 ± 12% interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi 127666 +770.3% 1111040 ± 13% interrupts.CPU1.LOC:Local_timer_interrupts 127702 +770.1% 1111139 ± 13% interrupts.CPU10.LOC:Local_timer_interrupts 127701 +770.1% 1111121 ± 13% interrupts.CPU11.LOC:Local_timer_interrupts 127505 +771.5% 1111167 ± 13% interrupts.CPU12.LOC:Local_timer_interrupts 127697 +770.1% 1111136 ± 13% interrupts.CPU13.LOC:Local_timer_interrupts 127617 +770.7% 1111119 ± 13% interrupts.CPU14.LOC:Local_timer_interrupts 127415 +772.2% 1111307 ± 13% interrupts.CPU15.LOC:Local_timer_interrupts 127685 +770.3% 1111238 ± 13% interrupts.CPU16.LOC:Local_timer_interrupts 127875 +768.9% 1111078 ± 13% interrupts.CPU17.LOC:Local_timer_interrupts 127440 +771.9% 1111101 ± 13% interrupts.CPU18.LOC:Local_timer_interrupts 127667 +770.4% 1111256 ± 13% interrupts.CPU19.LOC:Local_timer_interrupts 837.00 ± 18% -26.5% 615.20 ± 2% interrupts.CPU2.CAL:Function_call_interrupts 127702 +770.1% 1111115 ± 13% interrupts.CPU2.LOC:Local_timer_interrupts 263.00 ± 43% -53.1% 123.40 ± 40% interrupts.CPU2.NMI:Non-maskable_interrupts 263.00 ± 43% -53.1% 123.40 ± 40% interrupts.CPU2.PMI:Performance_monitoring_interrupts 127644 +770.4% 1111076 ± 13% interrupts.CPU20.LOC:Local_timer_interrupts 127463 +771.7% 1111091 ± 13% interrupts.CPU21.LOC:Local_timer_interrupts 291.67 ± 15% -60.8% 114.20 ± 16% interrupts.CPU21.NMI:Non-maskable_interrupts 291.67 ± 15% -60.8% 114.20 ± 16% interrupts.CPU21.PMI:Performance_monitoring_interrupts 127839 +769.1% 1111087 ± 13% interrupts.CPU22.LOC:Local_timer_interrupts 127883 +769.1% 1111402 ± 13% interrupts.CPU23.LOC:Local_timer_interrupts 665.50 ± 7% -9.8% 600.40 interrupts.CPU24.CAL:Function_call_interrupts 127511 +771.4% 1111102 ± 13% interrupts.CPU24.LOC:Local_timer_interrupts 127638 +770.5% 1111094 ± 13% interrupts.CPU25.LOC:Local_timer_interrupts 127979 +768.3% 1111283 ± 13% interrupts.CPU26.LOC:Local_timer_interrupts 127715 +770.1% 1111187 ± 13% interrupts.CPU27.LOC:Local_timer_interrupts 242.00 ± 37% -44.0% 135.60 ± 27% interrupts.CPU27.NMI:Non-maskable_interrupts 242.00 ± 37% -44.0% 135.60 ± 27% interrupts.CPU27.PMI:Performance_monitoring_interrupts 127644 +770.5% 1111182 ± 13% interrupts.CPU28.LOC:Local_timer_interrupts 127860 +769.0% 1111086 ± 13% interrupts.CPU29.LOC:Local_timer_interrupts 127959 +768.4% 1111218 ± 13% interrupts.CPU3.LOC:Local_timer_interrupts 232.50 ± 19% -46.3% 124.80 ± 38% interrupts.CPU3.NMI:Non-maskable_interrupts 232.50 ± 19% -46.3% 124.80 ± 38% interrupts.CPU3.PMI:Performance_monitoring_interrupts 127875 +768.8% 1111027 ± 13% interrupts.CPU30.LOC:Local_timer_interrupts 248.83 ± 33% -44.7% 137.60 ± 29% interrupts.CPU30.NMI:Non-maskable_interrupts 248.83 ± 33% -44.7% 137.60 ± 29% interrupts.CPU30.PMI:Performance_monitoring_interrupts 128053 +767.6% 1111029 ± 13% interrupts.CPU31.LOC:Local_timer_interrupts 206.50 ± 21% -44.3% 115.00 ± 16% interrupts.CPU31.NMI:Non-maskable_interrupts 206.50 ± 21% -44.3% 115.00 ± 16% interrupts.CPU31.PMI:Performance_monitoring_interrupts 127400 +772.1% 1111026 ± 13% interrupts.CPU32.LOC:Local_timer_interrupts 127540 +771.2% 1111145 ± 13% interrupts.CPU33.LOC:Local_timer_interrupts 127782 +769.6% 1111188 ± 13% interrupts.CPU34.LOC:Local_timer_interrupts 127628 +770.5% 1111058 ± 13% interrupts.CPU35.LOC:Local_timer_interrupts 187.83 ± 30% -37.3% 117.80 ± 14% interrupts.CPU35.NMI:Non-maskable_interrupts 187.83 ± 30% -37.3% 117.80 ± 14% interrupts.CPU35.PMI:Performance_monitoring_interrupts 127515 +771.4% 1111140 ± 13% interrupts.CPU36.LOC:Local_timer_interrupts 127613 +770.7% 1111074 ± 13% interrupts.CPU37.LOC:Local_timer_interrupts 127622 +770.6% 1111045 ± 13% interrupts.CPU38.LOC:Local_timer_interrupts 127671 +770.2% 1111029 ± 13% interrupts.CPU39.LOC:Local_timer_interrupts 127590 +770.8% 1111091 ± 13% interrupts.CPU4.LOC:Local_timer_interrupts 280.67 ± 37% -50.0% 140.40 ± 24% interrupts.CPU4.NMI:Non-maskable_interrupts 280.67 ± 37% -50.0% 140.40 ± 24% interrupts.CPU4.PMI:Performance_monitoring_interrupts 127627 +770.5% 1111031 ± 13% interrupts.CPU40.LOC:Local_timer_interrupts 127492 +771.5% 1111110 ± 13% interrupts.CPU41.LOC:Local_timer_interrupts 127525 +771.2% 1110964 ± 13% interrupts.CPU42.LOC:Local_timer_interrupts 127450 +771.7% 1111040 ± 13% interrupts.CPU43.LOC:Local_timer_interrupts 127883 +768.8% 1111113 ± 13% interrupts.CPU44.LOC:Local_timer_interrupts 127463 +771.6% 1111027 ± 13% interrupts.CPU45.LOC:Local_timer_interrupts 127432 +771.9% 1111029 ± 13% interrupts.CPU46.LOC:Local_timer_interrupts 127545 +771.1% 1111016 ± 13% interrupts.CPU47.LOC:Local_timer_interrupts 128036 +767.8% 1111089 ± 13% interrupts.CPU48.LOC:Local_timer_interrupts 203.33 ± 15% -47.4% 107.00 ± 31% interrupts.CPU48.NMI:Non-maskable_interrupts 203.33 ± 15% -47.4% 107.00 ± 31% interrupts.CPU48.PMI:Performance_monitoring_interrupts 127859 +769.0% 1111058 ± 13% interrupts.CPU49.LOC:Local_timer_interrupts 246.33 ± 18% -59.0% 101.00 ± 31% interrupts.CPU49.NMI:Non-maskable_interrupts 246.33 ± 18% -59.0% 101.00 ± 31% interrupts.CPU49.PMI:Performance_monitoring_interrupts 127782 +769.6% 1111137 ± 13% interrupts.CPU5.LOC:Local_timer_interrupts 216.00 ± 24% -34.6% 141.20 ± 29% interrupts.CPU5.NMI:Non-maskable_interrupts 216.00 ± 24% -34.6% 141.20 ± 29% interrupts.CPU5.PMI:Performance_monitoring_interrupts 687.17 ± 7% -12.0% 604.60 interrupts.CPU50.CAL:Function_call_interrupts 127612 +770.7% 1111107 ± 13% interrupts.CPU50.LOC:Local_timer_interrupts 260.00 ± 46% -59.4% 105.60 ± 44% interrupts.CPU50.NMI:Non-maskable_interrupts 260.00 ± 46% -59.4% 105.60 ± 44% interrupts.CPU50.PMI:Performance_monitoring_interrupts 127753 +769.7% 1111107 ± 13% interrupts.CPU51.LOC:Local_timer_interrupts 127741 +769.8% 1111151 ± 13% interrupts.CPU52.LOC:Local_timer_interrupts 220.50 ± 25% -42.9% 125.80 ± 37% interrupts.CPU52.NMI:Non-maskable_interrupts 220.50 ± 25% -42.9% 125.80 ± 37% interrupts.CPU52.PMI:Performance_monitoring_interrupts 127472 +771.8% 1111284 ± 13% interrupts.CPU53.LOC:Local_timer_interrupts 127618 +770.6% 1111088 ± 13% interrupts.CPU54.LOC:Local_timer_interrupts 127806 +769.5% 1111287 ± 13% interrupts.CPU55.LOC:Local_timer_interrupts 127518 +771.4% 1111184 ± 13% interrupts.CPU56.LOC:Local_timer_interrupts 127480 +771.8% 1111368 ± 13% interrupts.CPU57.LOC:Local_timer_interrupts 127654 +770.4% 1111081 ± 13% interrupts.CPU58.LOC:Local_timer_interrupts 756.00 ± 18% -22.7% 584.40 ± 5% interrupts.CPU59.CAL:Function_call_interrupts 127593 +770.9% 1111192 ± 13% interrupts.CPU59.LOC:Local_timer_interrupts 127625 +770.6% 1111090 ± 13% interrupts.CPU6.LOC:Local_timer_interrupts 127575 +770.9% 1111088 ± 13% interrupts.CPU60.LOC:Local_timer_interrupts 127992 +768.1% 1111073 ± 13% interrupts.CPU61.LOC:Local_timer_interrupts 127785 +769.5% 1111137 ± 13% interrupts.CPU62.LOC:Local_timer_interrupts 188.33 ± 27% -53.2% 88.20 ± 25% interrupts.CPU62.NMI:Non-maskable_interrupts 188.33 ± 27% -53.2% 88.20 ± 25% interrupts.CPU62.PMI:Performance_monitoring_interrupts 127774 +769.6% 1111109 ± 13% interrupts.CPU63.LOC:Local_timer_interrupts 127792 +769.5% 1111184 ± 13% interrupts.CPU64.LOC:Local_timer_interrupts 217.67 ± 42% -52.0% 104.40 ± 31% interrupts.CPU64.NMI:Non-maskable_interrupts 217.67 ± 42% -52.0% 104.40 ± 31% interrupts.CPU64.PMI:Performance_monitoring_interrupts 127921 +768.6% 1111130 ± 13% interrupts.CPU65.LOC:Local_timer_interrupts 211.50 ± 32% -44.8% 116.80 ± 38% interrupts.CPU65.NMI:Non-maskable_interrupts 211.50 ± 32% -44.8% 116.80 ± 38% interrupts.CPU65.PMI:Performance_monitoring_interrupts 127828 +769.3% 1111230 ± 13% interrupts.CPU66.LOC:Local_timer_interrupts 201.33 ± 22% -46.1% 108.60 ± 30% interrupts.CPU66.NMI:Non-maskable_interrupts 201.33 ± 22% -46.1% 108.60 ± 30% interrupts.CPU66.PMI:Performance_monitoring_interrupts 127662 +770.4% 1111223 ± 13% interrupts.CPU67.LOC:Local_timer_interrupts 280.00 ± 29% -62.6% 104.60 ± 26% interrupts.CPU67.NMI:Non-maskable_interrupts 280.00 ± 29% -62.6% 104.60 ± 26% interrupts.CPU67.PMI:Performance_monitoring_interrupts 127887 +769.1% 1111510 ± 13% interrupts.CPU68.LOC:Local_timer_interrupts 231.17 ± 24% -52.2% 110.60 ± 33% interrupts.CPU68.NMI:Non-maskable_interrupts 231.17 ± 24% -52.2% 110.60 ± 33% interrupts.CPU68.PMI:Performance_monitoring_interrupts 127704 +770.1% 1111152 ± 13% interrupts.CPU69.LOC:Local_timer_interrupts 241.83 ± 29% -49.6% 121.80 ± 39% interrupts.CPU69.NMI:Non-maskable_interrupts 241.83 ± 29% -49.6% 121.80 ± 39% interrupts.CPU69.PMI:Performance_monitoring_interrupts 127479 +771.6% 1111091 ± 13% interrupts.CPU7.LOC:Local_timer_interrupts 127637 +770.5% 1111111 ± 13% interrupts.CPU70.LOC:Local_timer_interrupts 127568 +771.0% 1111167 ± 13% interrupts.CPU71.LOC:Local_timer_interrupts 232.00 ± 24% -49.2% 117.80 ± 43% interrupts.CPU71.NMI:Non-maskable_interrupts 232.00 ± 24% -49.2% 117.80 ± 43% interrupts.CPU71.PMI:Performance_monitoring_interrupts 127584 +770.8% 1111048 ± 13% interrupts.CPU72.LOC:Local_timer_interrupts 127752 +769.9% 1111272 ± 13% interrupts.CPU73.LOC:Local_timer_interrupts 127705 +770.1% 1111229 ± 13% interrupts.CPU74.LOC:Local_timer_interrupts 187.17 ± 30% -34.1% 123.40 ± 43% interrupts.CPU74.NMI:Non-maskable_interrupts 187.17 ± 30% -34.1% 123.40 ± 43% interrupts.CPU74.PMI:Performance_monitoring_interrupts 127802 +769.4% 1111144 ± 13% interrupts.CPU75.LOC:Local_timer_interrupts 253.83 ± 24% -54.3% 116.00 ± 22% interrupts.CPU75.NMI:Non-maskable_interrupts 253.83 ± 24% -54.3% 116.00 ± 22% interrupts.CPU75.PMI:Performance_monitoring_interrupts 127785 +769.6% 1111238 ± 13% interrupts.CPU76.LOC:Local_timer_interrupts 202.00 ± 37% -43.3% 114.60 ± 17% interrupts.CPU76.NMI:Non-maskable_interrupts 202.00 ± 37% -43.3% 114.60 ± 17% interrupts.CPU76.PMI:Performance_monitoring_interrupts 127789 +769.6% 1111269 ± 13% interrupts.CPU77.LOC:Local_timer_interrupts 203.50 ± 35% -42.4% 117.20 ± 17% interrupts.CPU77.NMI:Non-maskable_interrupts 203.50 ± 35% -42.4% 117.20 ± 17% interrupts.CPU77.PMI:Performance_monitoring_interrupts 127542 +771.2% 1111187 ± 13% interrupts.CPU78.LOC:Local_timer_interrupts 127678 +770.2% 1111113 ± 13% interrupts.CPU79.LOC:Local_timer_interrupts 200.17 ± 19% -48.1% 103.80 ± 31% interrupts.CPU79.NMI:Non-maskable_interrupts 200.17 ± 19% -48.1% 103.80 ± 31% interrupts.CPU79.PMI:Performance_monitoring_interrupts 127650 +770.6% 1111283 ± 13% interrupts.CPU8.LOC:Local_timer_interrupts 127841 +769.2% 1111160 ± 13% interrupts.CPU80.LOC:Local_timer_interrupts 127489 +771.6% 1111176 ± 13% interrupts.CPU81.LOC:Local_timer_interrupts 127668 +770.5% 1111340 ± 13% interrupts.CPU82.LOC:Local_timer_interrupts 127892 +768.8% 1111175 ± 13% interrupts.CPU83.LOC:Local_timer_interrupts 127640 +770.5% 1111123 ± 13% interrupts.CPU84.LOC:Local_timer_interrupts 127693 +770.2% 1111192 ± 13% interrupts.CPU85.LOC:Local_timer_interrupts 127458 +771.7% 1111105 ± 13% interrupts.CPU86.LOC:Local_timer_interrupts 127492 +771.5% 1111079 ± 13% interrupts.CPU87.LOC:Local_timer_interrupts 127699 +770.1% 1111103 ± 13% interrupts.CPU88.LOC:Local_timer_interrupts 127963 +768.3% 1111112 ± 13% interrupts.CPU89.LOC:Local_timer_interrupts 165.17 ± 22% -33.6% 109.60 ± 27% interrupts.CPU89.NMI:Non-maskable_interrupts 165.17 ± 22% -33.6% 109.60 ± 27% interrupts.CPU89.PMI:Performance_monitoring_interrupts 127532 +771.2% 1111094 ± 13% interrupts.CPU9.LOC:Local_timer_interrupts 127736 +769.9% 1111139 ± 13% interrupts.CPU90.LOC:Local_timer_interrupts 127585 +770.9% 1111141 ± 13% interrupts.CPU91.LOC:Local_timer_interrupts 127793 +769.6% 1111251 ± 13% interrupts.CPU92.LOC:Local_timer_interrupts 127737 +770.0% 1111275 ± 13% interrupts.CPU93.LOC:Local_timer_interrupts 207.67 ± 41% -38.3% 128.20 ± 43% interrupts.CPU93.NMI:Non-maskable_interrupts 207.67 ± 41% -38.3% 128.20 ± 43% interrupts.CPU93.PMI:Performance_monitoring_interrupts 127478 +771.7% 1111196 ± 13% interrupts.CPU94.LOC:Local_timer_interrupts 127809 +769.5% 1111269 ± 13% interrupts.CPU95.LOC:Local_timer_interrupts 12257088 +770.3% 1.067e+08 ± 13% interrupts.LOC:Local_timer_interrupts 0.00 +1.2e+104% 115.20 ± 33% interrupts.MCP:Machine_check_polls 18319 ± 2% -36.9% 11562 ± 20% interrupts.NMI:Non-maskable_interrupts 18319 ± 2% -36.9% 11562 ± 20% interrupts.PMI:Performance_monitoring_interrupts 19041 ± 2% -66.9% 6302 ± 8% softirqs.BLOCK 12376 ± 16% +188.6% 35722 ± 27% softirqs.CPU0.RCU 11554 ± 8% +416.4% 59667 ± 32% softirqs.CPU0.SCHED 9451 ± 4% +250.7% 33142 ± 37% softirqs.CPU1.RCU 9894 ± 15% +601.5% 69406 ± 14% softirqs.CPU1.SCHED 8818 ± 3% +274.3% 33010 ± 38% softirqs.CPU10.RCU 7845 ± 17% +767.1% 68029 ± 16% softirqs.CPU10.SCHED 8758 ± 2% +269.8% 32390 ± 35% softirqs.CPU11.RCU 8268 ± 12% +564.8% 54965 ± 19% softirqs.CPU11.SCHED 8777 ± 3% +283.0% 33620 ± 37% softirqs.CPU12.RCU 8562 ± 6% +635.3% 62958 ± 27% softirqs.CPU12.SCHED 8618 ± 2% +291.8% 33766 ± 39% softirqs.CPU13.RCU 8896 ± 5% +671.5% 68637 ± 15% softirqs.CPU13.SCHED 8822 ± 3% +274.8% 33071 ± 39% softirqs.CPU14.RCU 8765 ± 7% +617.6% 62905 ± 27% softirqs.CPU14.SCHED 8892 ± 5% +266.9% 32623 ± 38% softirqs.CPU15.RCU 8770 ± 13% +690.8% 69353 ± 15% softirqs.CPU15.SCHED 8633 ± 5% +261.4% 31202 ± 35% softirqs.CPU16.RCU 8236 ± 12% +728.8% 68262 ± 15% softirqs.CPU16.SCHED 8631 ± 3% +286.5% 33363 ± 35% softirqs.CPU17.RCU 8590 ± 7% +702.7% 68958 ± 15% softirqs.CPU17.SCHED 8570 ± 4% +292.6% 33648 ± 38% softirqs.CPU18.RCU 9000 ± 3% +665.7% 68922 ± 16% softirqs.CPU18.SCHED 8654 ± 4% +276.0% 32543 ± 35% softirqs.CPU19.RCU 8259 ± 7% +663.8% 63080 ± 27% softirqs.CPU19.SCHED 9003 ± 3% +278.0% 34036 ± 38% softirqs.CPU2.RCU 8969 ± 9% +664.7% 68583 ± 15% softirqs.CPU2.SCHED 8553 ± 2% +285.0% 32935 ± 37% softirqs.CPU20.RCU 8481 ± 14% +707.0% 68448 ± 16% softirqs.CPU20.SCHED 8347 ± 2% +294.6% 32937 ± 36% softirqs.CPU21.RCU 8053 ± 16% +685.5% 63253 ± 27% softirqs.CPU21.SCHED 8486 ± 2% +292.4% 33301 ± 36% softirqs.CPU22.RCU 8747 ± 7% +624.9% 63411 ± 27% softirqs.CPU22.SCHED 8466 ± 3% +288.6% 32904 ± 36% softirqs.CPU23.RCU 8409 ± 12% +715.9% 68618 ± 15% softirqs.CPU23.SCHED 8609 ± 2% +273.7% 32175 ± 35% softirqs.CPU24.RCU 7874 ± 10% +644.0% 58587 ± 36% softirqs.CPU24.SCHED 8457 ± 3% +251.9% 29760 ± 35% softirqs.CPU25.RCU 7892 ± 11% +770.0% 68665 ± 17% softirqs.CPU25.SCHED 8221 ± 3% +270.5% 30462 ± 36% softirqs.CPU26.RCU 8702 ± 5% +688.0% 68574 ± 17% softirqs.CPU26.SCHED 8603 ± 6% +269.9% 31822 ± 37% softirqs.CPU27.RCU 8671 ± 6% +692.2% 68696 ± 17% softirqs.CPU27.SCHED 8324 ± 3% +278.8% 31527 ± 35% softirqs.CPU28.RCU 8693 ± 9% +688.6% 68556 ± 17% softirqs.CPU28.SCHED 8367 ± 2% +267.4% 30742 ± 38% softirqs.CPU29.RCU 8451 ± 5% +711.8% 68609 ± 17% softirqs.CPU29.SCHED 9104 ± 7% +268.9% 33581 ± 38% softirqs.CPU3.RCU 8759 ± 4% +683.0% 68583 ± 16% softirqs.CPU3.SCHED 8364 ± 2% +276.4% 31485 ± 38% softirqs.CPU30.RCU 8018 ± 11% +646.0% 59812 ± 9% softirqs.CPU30.SCHED 8229 ± 3% +278.4% 31143 ± 35% softirqs.CPU31.RCU 8340 ± 12% +627.0% 60629 ± 30% softirqs.CPU31.SCHED 8608 +287.8% 33384 ± 36% softirqs.CPU32.RCU 8322 ± 8% +720.5% 68286 ± 17% softirqs.CPU32.SCHED 9482 ± 17% +244.0% 32616 ± 35% softirqs.CPU33.RCU 8092 ± 10% +744.1% 68312 ± 16% softirqs.CPU33.SCHED 8629 ± 3% +286.6% 33363 ± 35% softirqs.CPU34.RCU 8400 ± 11% +715.7% 68515 ± 17% softirqs.CPU34.SCHED 8834 ± 2% +279.4% 33515 ± 35% softirqs.CPU35.RCU 7691 ± 18% +683.2% 60233 ± 7% softirqs.CPU35.SCHED 9121 ± 13% +254.2% 32306 ± 35% softirqs.CPU36.RCU 8076 ± 17% +747.2% 68419 ± 17% softirqs.CPU36.SCHED 8649 +278.8% 32764 ± 35% softirqs.CPU37.RCU 8334 ± 9% +719.0% 68259 ± 17% softirqs.CPU37.SCHED 9588 ± 17% +242.1% 32807 ± 35% softirqs.CPU38.RCU 7864 ± 19% +780.2% 69217 ± 16% softirqs.CPU38.SCHED 8701 ± 8% +279.0% 32980 ± 35% softirqs.CPU39.RCU 8315 ± 15% +720.1% 68199 ± 17% softirqs.CPU39.SCHED 9298 ± 3% +260.1% 33479 ± 38% softirqs.CPU4.RCU 7896 ± 16% +768.0% 68540 ± 16% softirqs.CPU4.SCHED 8717 ± 2% +284.6% 33526 ± 36% softirqs.CPU40.RCU 8084 ± 13% +747.4% 68504 ± 16% softirqs.CPU40.SCHED 9221 ± 11% +265.7% 33718 ± 35% softirqs.CPU41.RCU 8288 ± 17% +663.6% 63284 ± 29% softirqs.CPU41.SCHED 8549 ± 5% +264.7% 31178 ± 36% softirqs.CPU42.RCU 8585 ± 15% +693.0% 68084 ± 17% softirqs.CPU42.SCHED 8447 ± 5% +289.6% 32916 ± 35% softirqs.CPU43.RCU 8799 ± 4% +674.1% 68118 ± 17% softirqs.CPU43.SCHED 9261 ± 18% +259.0% 33248 ± 36% softirqs.CPU44.RCU 8813 ± 5% +674.3% 68242 ± 17% softirqs.CPU44.SCHED 8566 ± 2% +284.7% 32959 ± 36% softirqs.CPU45.RCU 8649 ± 5% +688.2% 68177 ± 17% softirqs.CPU45.SCHED 8503 ± 2% +284.8% 32717 ± 35% softirqs.CPU46.RCU 8827 ± 4% +677.6% 68649 ± 16% softirqs.CPU46.SCHED 8764 ± 2% +288.3% 34030 ± 35% softirqs.CPU47.RCU 7976 ± 11% +465.8% 45132 ± 32% softirqs.CPU47.SCHED 8597 ± 2% +254.6% 30487 ± 34% softirqs.CPU48.RCU 6897 ± 28% +705.5% 55560 ± 42% softirqs.CPU48.SCHED 8707 ± 2% +262.2% 31540 ± 36% softirqs.CPU49.RCU 7951 ± 15% +771.6% 69308 ± 14% softirqs.CPU49.SCHED 8812 ± 4% +257.0% 31464 ± 48% softirqs.CPU5.RCU 8683 ± 7% +693.4% 68897 ± 15% softirqs.CPU5.SCHED 8667 ± 3% +284.5% 33326 ± 38% softirqs.CPU50.RCU 8629 ± 5% +639.1% 63781 ± 26% softirqs.CPU50.SCHED 8612 ± 2% +285.5% 33205 ± 39% softirqs.CPU51.RCU 8038 ± 14% +769.7% 69915 ± 15% softirqs.CPU51.SCHED 9342 ± 18% +258.4% 33480 ± 37% softirqs.CPU52.RCU 8941 ± 8% +686.8% 70350 ± 14% softirqs.CPU52.SCHED 8657 ± 4% +268.4% 31894 ± 36% softirqs.CPU53.RCU 9130 ± 5% +670.6% 70360 ± 14% softirqs.CPU53.SCHED 8567 ± 3% +294.4% 33793 ± 40% softirqs.CPU54.RCU 8566 ± 6% +720.0% 70248 ± 15% softirqs.CPU54.SCHED 8452 ± 3% +240.4% 28775 ± 35% softirqs.CPU55.RCU 8596 ± 8% +703.8% 69098 ± 15% softirqs.CPU55.SCHED 8433 ± 5% +295.7% 33366 ± 38% softirqs.CPU56.RCU 8981 ± 10% +673.9% 69506 ± 15% softirqs.CPU56.SCHED 8646 ± 4% +287.2% 33478 ± 38% softirqs.CPU57.RCU 7896 ± 14% +788.8% 70183 ± 14% softirqs.CPU57.SCHED 8628 ± 5% +281.9% 32951 ± 37% softirqs.CPU58.RCU 8711 ± 6% +701.7% 69841 ± 14% softirqs.CPU58.SCHED 9119 ± 6% +271.0% 33834 ± 40% softirqs.CPU59.RCU 7599 ± 22% +831.7% 70802 ± 14% softirqs.CPU59.SCHED 8794 ± 3% +257.0% 31390 ± 50% softirqs.CPU6.RCU 7587 ± 15% +806.2% 68752 ± 15% softirqs.CPU6.SCHED 8794 ± 7% +280.6% 33472 ± 38% softirqs.CPU60.RCU 8915 ± 12% +683.9% 69888 ± 14% softirqs.CPU60.SCHED 8736 ± 6% +272.0% 32497 ± 39% softirqs.CPU61.RCU 7830 ± 20% +789.0% 69619 ± 14% softirqs.CPU61.SCHED 8438 ± 2% +289.9% 32903 ± 38% softirqs.CPU62.RCU 8923 ± 3% +682.3% 69806 ± 14% softirqs.CPU62.SCHED 8514 ± 4% +236.4% 28638 ± 37% softirqs.CPU63.RCU 8927 ± 4% +683.3% 69927 ± 14% softirqs.CPU63.SCHED 8684 ± 5% +260.8% 31333 ± 34% softirqs.CPU64.RCU 8740 ± 9% +700.0% 69919 ± 14% softirqs.CPU64.SCHED 8657 ± 9% +285.0% 33334 ± 35% softirqs.CPU65.RCU 8775 ± 7% +643.6% 65250 ± 25% softirqs.CPU65.SCHED 8601 ± 7% +284.4% 33061 ± 36% softirqs.CPU66.RCU 7824 ± 25% +796.5% 70146 ± 14% softirqs.CPU66.SCHED 8926 ± 6% +261.6% 32278 ± 35% softirqs.CPU67.RCU 7495 ± 17% +817.6% 68774 ± 12% softirqs.CPU67.SCHED 8306 ± 4% +298.9% 33131 ± 37% softirqs.CPU68.RCU 9084 ± 4% +674.4% 70346 ± 14% softirqs.CPU68.SCHED 8712 ± 4% +276.1% 32765 ± 36% softirqs.CPU69.RCU 7531 ± 19% +829.7% 70023 ± 15% softirqs.CPU69.SCHED 8949 ± 6% +273.5% 33428 ± 37% softirqs.CPU7.RCU 8804 ± 15% +676.2% 68346 ± 16% softirqs.CPU7.SCHED 8292 ± 4% +301.4% 33290 ± 37% softirqs.CPU70.RCU 8676 ± 9% +710.8% 70353 ± 14% softirqs.CPU70.SCHED 8563 +282.7% 32769 ± 35% softirqs.CPU71.RCU 8678 ± 8% +707.5% 70081 ± 14% softirqs.CPU71.SCHED 8123 ± 3% +279.7% 30844 ± 37% softirqs.CPU72.RCU 8626 ± 10% +717.3% 70498 ± 16% softirqs.CPU72.SCHED 7931 ± 2% +277.6% 29949 ± 38% softirqs.CPU73.RCU 8313 ± 9% +668.6% 63900 ± 27% softirqs.CPU73.SCHED 7884 ± 6% +282.8% 30180 ± 36% softirqs.CPU74.RCU 8014 ± 16% +778.3% 70389 ± 16% softirqs.CPU74.SCHED 8094 ± 5% +282.6% 30968 ± 36% softirqs.CPU75.RCU 8563 ± 9% +721.0% 70309 ± 16% softirqs.CPU75.SCHED 7935 ± 2% +292.7% 31159 ± 35% softirqs.CPU76.RCU 8914 ± 5% +694.1% 70791 ± 15% softirqs.CPU76.SCHED 8066 ± 3% +296.0% 31941 ± 41% softirqs.CPU77.RCU 8786 ± 6% +665.6% 67270 ± 26% softirqs.CPU77.SCHED 8060 ± 2% +283.3% 30894 ± 37% softirqs.CPU78.RCU 8524 ± 9% +728.1% 70589 ± 16% softirqs.CPU78.SCHED 7895 ± 3% +288.4% 30666 ± 36% softirqs.CPU79.RCU 9026 ± 2% +677.7% 70198 ± 16% softirqs.CPU79.SCHED 8299 ± 9% +305.4% 33649 ± 38% softirqs.CPU8.RCU 8638 ± 8% +628.8% 62951 ± 27% softirqs.CPU8.SCHED 8113 ± 3% +295.0% 32048 ± 36% softirqs.CPU80.RCU 8343 ± 8% +744.1% 70426 ± 16% softirqs.CPU80.SCHED 8513 ± 5% +267.0% 31241 ± 36% softirqs.CPU81.RCU 8074 ± 14% +767.3% 70027 ± 16% softirqs.CPU81.SCHED 8183 ± 3% +291.7% 32057 ± 37% softirqs.CPU82.RCU 8472 ± 11% +733.9% 70651 ± 16% softirqs.CPU82.SCHED 8534 ± 3% +249.6% 29836 ± 44% softirqs.CPU83.RCU 8262 ± 7% +750.6% 70283 ± 16% softirqs.CPU83.SCHED 8470 ± 7% +267.7% 31146 ± 35% softirqs.CPU84.RCU 7999 ± 16% +775.9% 70062 ± 16% softirqs.CPU84.SCHED 8495 ± 5% +272.5% 31646 ± 36% softirqs.CPU85.RCU 7740 ± 16% +805.9% 70119 ± 16% softirqs.CPU85.SCHED 8248 ± 4% +269.8% 30507 ± 33% softirqs.CPU86.RCU 9214 ± 4% +662.6% 70274 ± 16% softirqs.CPU86.SCHED 8240 ± 2% +278.6% 31196 ± 36% softirqs.CPU87.RCU 8462 ± 10% +727.6% 70035 ± 16% softirqs.CPU87.SCHED 8201 ± 4% +295.1% 32406 ± 37% softirqs.CPU88.RCU 8727 ± 11% +708.8% 70592 ± 16% softirqs.CPU88.SCHED 8665 ± 3% +275.1% 32503 ± 36% softirqs.CPU89.RCU 8501 ± 5% +732.1% 70743 ± 16% softirqs.CPU89.SCHED 8602 ± 7% +287.1% 33299 ± 38% softirqs.CPU9.RCU 8684 ± 8% +688.1% 68440 ± 16% softirqs.CPU9.SCHED 8018 ± 3% +268.1% 29515 ± 34% softirqs.CPU90.RCU 8549 ± 8% +710.1% 69266 ± 17% softirqs.CPU90.SCHED 8327 ± 3% +280.5% 31684 ± 37% softirqs.CPU91.RCU 7884 ± 18% +787.3% 69959 ± 16% softirqs.CPU91.SCHED 8110 +291.6% 31763 ± 36% softirqs.CPU92.RCU 9117 ± 3% +667.6% 69986 ± 16% softirqs.CPU92.SCHED 8112 ± 2% +295.0% 32044 ± 37% softirqs.CPU93.RCU 8179 ± 12% +759.1% 70263 ± 16% softirqs.CPU93.SCHED 8804 ± 9% +267.0% 32313 ± 36% softirqs.CPU94.RCU 7897 ± 14% +793.6% 70567 ± 16% softirqs.CPU94.SCHED 8547 ± 5% +284.9% 32900 ± 36% softirqs.CPU95.RCU 7890 ± 16% +633.5% 57881 ± 24% softirqs.CPU95.SCHED 5318 ±154% +353.4% 24111 ± 60% softirqs.NET_RX 827812 +274.8% 3102713 ± 36% softirqs.RCU 811580 +699.7% 6489945 ± 16% softirqs.SCHED 17742 ± 4% +117.8% 38644 ± 17% softirqs.TIMER 18.86 ± 2% -18.9 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 16.79 ± 3% -16.8 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.83 ± 5% -13.8 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 12.17 ± 3% -12.1 0.11 ±200% perf-profile.calltrace.cycles-pp.ret_from_fork 12.15 ± 3% -12.0 0.11 ±200% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 11.77 ± 3% -11.8 0.00 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 11.69 ± 3% -11.7 0.00 perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 11.10 ± 3% -11.1 0.00 perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork 10.71 ± 3% -10.7 0.00 perf-profile.calltrace.cycles-pp.memcpy_toio.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread 10.30 ± 8% -10.3 0.00 perf-profile.calltrace.cycles-pp.block_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.30 ± 8% -10.3 0.00 perf-profile.calltrace.cycles-pp.blkdev_ioctl.block_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.29 ± 8% -10.3 0.00 perf-profile.calltrace.cycles-pp.lo_ioctl.blkdev_ioctl.block_ioctl.__x64_sys_ioctl.do_syscall_64 8.22 ± 8% -8.2 0.00 perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.printk.set_capacity_and_notify.cold.loop_set_size 7.64 ± 8% -7.6 0.00 perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.printk.set_capacity_and_notify.cold 7.29 ± 8% -7.0 0.33 ±124% perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.printk 6.73 ± 8% -6.4 0.31 ±123% perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock 6.74 ± 8% -6.4 0.31 ±123% perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit 5.37 ± 5% -5.4 0.00 perf-profile.calltrace.cycles-pp.loop_configure.lo_ioctl.blkdev_ioctl.block_ioctl.__x64_sys_ioctl 5.32 ± 5% -5.3 0.00 perf-profile.calltrace.cycles-pp.printk.set_capacity_and_notify.cold.loop_set_size.loop_configure.lo_ioctl 5.32 ± 5% -5.3 0.00 perf-profile.calltrace.cycles-pp.vprintk_emit.printk.set_capacity_and_notify.cold.loop_set_size.loop_configure 5.32 ± 5% -5.3 0.00 perf-profile.calltrace.cycles-pp.loop_set_size.loop_configure.lo_ioctl.blkdev_ioctl.block_ioctl 5.32 ± 5% -5.3 0.00 perf-profile.calltrace.cycles-pp.set_capacity_and_notify.cold.loop_set_size.loop_configure.lo_ioctl.blkdev_ioctl 5.57 ± 10% -5.3 0.26 ±123% perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write 0.66 ± 7% +0.5 1.19 ± 15% perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 0.57 ± 4% +0.5 1.11 ± 17% perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.72 ± 18% +0.6 1.30 ± 13% perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.86 ± 7% +0.6 1.45 ± 22% perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 0.00 +0.7 0.72 ± 11% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt 0.75 ± 5% +0.8 1.55 ± 9% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.96 ± 14% +0.8 1.78 ± 13% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 1.02 ± 13% +0.8 1.87 ± 14% perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 1.17 ± 7% +0.9 2.03 ± 11% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 1.71 ± 5% +1.1 2.84 ± 15% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 2.19 ± 5% +1.6 3.75 ± 16% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 2.16 ± 5% +1.7 3.82 ± 16% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 2.22 ± 5% +1.8 4.07 ± 19% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 2.08 ± 7% +2.0 4.04 ± 26% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 2.56 ± 5% +2.2 4.74 ± 15% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 3.80 ± 3% +3.4 7.16 ± 18% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 7.75 ± 2% +6.2 13.98 ± 10% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 7.89 ± 2% +6.4 14.24 ± 10% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 12.29 ± 2% +9.7 22.01 ± 10% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 5.35 ± 15% +10.1 15.48 ± 30% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 57.16 +16.8 73.99 ± 8% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 17.95 ± 4% +18.3 36.22 ± 18% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 59.09 +20.6 79.73 ± 5% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 65.46 +31.4 96.82 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 65.49 +31.4 96.89 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 65.49 +31.4 96.89 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 66.20 +31.9 98.14 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 20.79 ± 2% -19.8 1.00 ± 21% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 18.43 ± 3% -17.5 0.97 ± 21% perf-profile.children.cycles-pp.do_syscall_64 13.86 ± 5% -13.9 0.00 perf-profile.children.cycles-pp.__x64_sys_ioctl 12.17 ± 3% -11.8 0.40 ± 31% perf-profile.children.cycles-pp.ret_from_fork 12.15 ± 3% -11.8 0.40 ± 32% perf-profile.children.cycles-pp.kthread 11.77 ± 3% -11.5 0.22 ± 40% perf-profile.children.cycles-pp.worker_thread 11.69 ± 3% -11.5 0.21 ± 44% perf-profile.children.cycles-pp.process_one_work 11.10 ± 3% -10.9 0.19 ± 51% perf-profile.children.cycles-pp.drm_fb_helper_damage_work 11.08 ± 3% -10.9 0.19 ± 51% perf-profile.children.cycles-pp.memcpy_toio 10.33 ± 8% -10.3 0.00 perf-profile.children.cycles-pp.block_ioctl 10.33 ± 8% -10.3 0.00 perf-profile.children.cycles-pp.blkdev_ioctl 10.33 ± 8% -10.3 0.00 perf-profile.children.cycles-pp.lo_ioctl 9.80 ± 9% -9.8 0.00 perf-profile.children.cycles-pp.loop_set_size 9.80 ± 9% -9.8 0.00 perf-profile.children.cycles-pp.set_capacity_and_notify.cold 10.12 ± 9% -9.3 0.77 ± 47% perf-profile.children.cycles-pp.printk 10.12 ± 9% -9.3 0.77 ± 47% perf-profile.children.cycles-pp.vprintk_emit 8.50 ± 8% -7.7 0.77 ± 47% perf-profile.children.cycles-pp.console_unlock 7.91 ± 8% -7.2 0.74 ± 47% perf-profile.children.cycles-pp.serial8250_console_write 7.55 ± 8% -6.8 0.71 ± 47% perf-profile.children.cycles-pp.uart_console_write 7.33 ± 8% -6.6 0.69 ± 47% perf-profile.children.cycles-pp.wait_for_xmitr 6.52 ± 12% -6.5 0.00 perf-profile.children.cycles-pp.__mutex_lock 6.99 ± 8% -6.3 0.67 ± 47% perf-profile.children.cycles-pp.serial8250_console_putchar 6.03 ± 9% -5.4 0.59 ± 43% perf-profile.children.cycles-pp.io_serial_in 5.37 ± 5% -5.4 0.00 perf-profile.children.cycles-pp.loop_configure 3.14 ± 16% -3.0 0.13 ± 31% perf-profile.children.cycles-pp.do_sys_open 3.13 ± 16% -3.0 0.13 ± 31% perf-profile.children.cycles-pp.do_sys_openat2 2.99 ± 16% -2.9 0.11 ± 27% perf-profile.children.cycles-pp.do_filp_open 2.98 ± 16% -2.9 0.11 ± 27% perf-profile.children.cycles-pp.path_openat 1.29 ± 16% -1.2 0.10 ± 76% perf-profile.children.cycles-pp.delay_tsc 0.58 ± 16% -0.5 0.04 ± 85% perf-profile.children.cycles-pp.io_serial_out 0.47 ± 16% -0.3 0.16 ± 45% perf-profile.children.cycles-pp.__schedule 0.37 ± 17% -0.3 0.11 ± 52% perf-profile.children.cycles-pp.schedule 0.30 ± 16% -0.2 0.06 ± 52% perf-profile.children.cycles-pp.walk_component 0.26 ± 15% -0.2 0.07 ± 50% perf-profile.children.cycles-pp.link_path_walk 0.23 ± 14% -0.2 0.07 ± 77% perf-profile.children.cycles-pp.newidle_balance 0.30 ± 13% -0.2 0.15 ± 30% perf-profile.children.cycles-pp.ksys_read 0.30 ± 15% -0.2 0.14 ± 29% perf-profile.children.cycles-pp.vfs_read 0.20 ± 12% -0.1 0.07 ± 55% perf-profile.children.cycles-pp.new_sync_read 0.01 ±223% +0.1 0.06 ± 18% perf-profile.children.cycles-pp.unmap_page_range 0.03 ±102% +0.1 0.09 ± 27% perf-profile.children.cycles-pp.__do_sys_clone 0.01 ±223% +0.1 0.07 ± 11% perf-profile.children.cycles-pp.unmap_vmas 0.05 ± 75% +0.1 0.11 ± 22% perf-profile.children.cycles-pp.__libc_fork 0.02 ±141% +0.1 0.08 ± 23% perf-profile.children.cycles-pp.rcu_needs_cpu 0.14 ± 19% +0.1 0.21 ± 14% perf-profile.children.cycles-pp.hrtimer_get_next_event 0.04 ± 77% +0.1 0.11 ± 29% perf-profile.children.cycles-pp.do_fault 0.08 ± 22% +0.1 0.15 ± 23% perf-profile.children.cycles-pp.rcu_dynticks_eqs_enter 0.02 ±144% +0.1 0.09 ± 38% perf-profile.children.cycles-pp.begin_new_exec 0.02 ±143% +0.1 0.10 ± 27% perf-profile.children.cycles-pp.filemap_map_pages 0.07 ± 47% +0.1 0.14 ± 28% perf-profile.children.cycles-pp.mmput 0.07 ± 47% +0.1 0.14 ± 28% perf-profile.children.cycles-pp.exit_mmap 0.09 ± 17% +0.1 0.17 ± 34% perf-profile.children.cycles-pp.trigger_load_balance 0.11 ± 23% +0.1 0.20 ± 21% perf-profile.children.cycles-pp.__handle_mm_fault 0.12 ± 23% +0.1 0.21 ± 19% perf-profile.children.cycles-pp.handle_mm_fault 0.13 ± 21% +0.1 0.22 ± 19% perf-profile.children.cycles-pp.exc_page_fault 0.13 ± 21% +0.1 0.22 ± 19% perf-profile.children.cycles-pp.do_user_addr_fault 0.08 ± 23% +0.1 0.17 ± 37% perf-profile.children.cycles-pp.exec_binprm 0.07 ± 26% +0.1 0.17 ± 37% perf-profile.children.cycles-pp.load_elf_binary 0.14 ± 21% +0.1 0.25 ± 23% perf-profile.children.cycles-pp.asm_exc_page_fault 0.13 ± 12% +0.1 0.24 ± 22% perf-profile.children.cycles-pp.rcu_eqs_enter 0.10 ± 14% +0.1 0.21 ± 61% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.09 ± 25% +0.1 0.21 ± 28% perf-profile.children.cycles-pp.bprm_execve 0.13 ± 22% +0.1 0.25 ± 48% perf-profile.children.cycles-pp.__hrtimer_get_next_event 0.12 ± 18% +0.1 0.24 ± 27% perf-profile.children.cycles-pp.execve 0.12 ± 18% +0.1 0.24 ± 27% perf-profile.children.cycles-pp.__x64_sys_execve 0.12 ± 18% +0.1 0.24 ± 27% perf-profile.children.cycles-pp.do_execveat_common 0.14 ± 9% +0.2 0.31 ± 40% perf-profile.children.cycles-pp.timerqueue_add 0.13 ± 6% +0.2 0.30 ± 65% perf-profile.children.cycles-pp.timerqueue_del 0.00 +0.2 0.19 ± 91% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.61 ± 7% +0.2 0.80 ± 16% perf-profile.children.cycles-pp.load_balance 0.20 ± 12% +0.2 0.39 ± 28% perf-profile.children.cycles-pp.update_irq_load_avg 0.17 ± 12% +0.2 0.36 ± 33% perf-profile.children.cycles-pp.enqueue_hrtimer 0.17 ± 12% +0.2 0.37 ± 64% perf-profile.children.cycles-pp.__remove_hrtimer 0.30 ± 13% +0.2 0.51 ± 13% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.19 ± 22% +0.2 0.42 ± 23% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust 0.19 ± 21% +0.2 0.43 ± 25% perf-profile.children.cycles-pp.arch_cpu_idle_enter 0.24 ± 18% +0.2 0.48 ± 34% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.23 ± 10% +0.2 0.48 ± 40% perf-profile.children.cycles-pp.hrtimer_update_next_event 0.34 ± 4% +0.2 0.58 ± 19% perf-profile.children.cycles-pp.update_rq_clock 0.32 ± 14% +0.3 0.57 ± 9% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.52 ± 6% +0.4 0.91 ± 14% perf-profile.children.cycles-pp.native_sched_clock 0.54 ± 5% +0.4 0.96 ± 17% perf-profile.children.cycles-pp.sched_clock 0.32 ± 29% +0.4 0.77 ± 47% perf-profile.children.cycles-pp.irq_work_single 0.61 ± 7% +0.4 1.06 ± 11% perf-profile.children.cycles-pp.read_tsc 0.32 ± 31% +0.5 0.77 ± 47% perf-profile.children.cycles-pp.asm_sysvec_irq_work 0.32 ± 31% +0.5 0.77 ± 47% perf-profile.children.cycles-pp.sysvec_irq_work 0.32 ± 31% +0.5 0.77 ± 47% perf-profile.children.cycles-pp.__sysvec_irq_work 0.32 ± 30% +0.5 0.77 ± 47% perf-profile.children.cycles-pp.irq_work_run 0.34 ± 28% +0.5 0.81 ± 46% perf-profile.children.cycles-pp.irq_work_run_list 0.61 ± 9% +0.5 1.12 ± 16% perf-profile.children.cycles-pp.sched_clock_cpu 0.60 ± 4% +0.5 1.13 ± 17% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 0.69 ± 8% +0.5 1.23 ± 14% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.72 ± 8% +0.6 1.34 ± 24% perf-profile.children.cycles-pp.irqtime_account_irq 0.89 ± 7% +0.7 1.54 ± 23% perf-profile.children.cycles-pp.lapic_next_deadline 0.77 ± 4% +0.8 1.58 ± 8% perf-profile.children.cycles-pp.rebalance_domains 0.98 ± 14% +0.9 1.84 ± 14% perf-profile.children.cycles-pp.tick_irq_enter 1.23 ± 6% +0.9 2.09 ± 11% perf-profile.children.cycles-pp.scheduler_tick 1.02 ± 13% +0.9 1.91 ± 14% perf-profile.children.cycles-pp.irq_enter_rcu 1.81 ± 3% +1.1 2.90 ± 15% perf-profile.children.cycles-pp.__softirqentry_text_start 2.24 ± 4% +1.6 3.82 ± 16% perf-profile.children.cycles-pp.irq_exit_rcu 2.25 ± 4% +1.6 3.89 ± 16% perf-profile.children.cycles-pp.update_process_times 2.29 ± 4% +1.8 4.11 ± 19% perf-profile.children.cycles-pp.tick_sched_handle 2.12 ± 7% +2.0 4.10 ± 25% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 2.65 ± 5% +2.2 4.82 ± 15% perf-profile.children.cycles-pp.tick_sched_timer 3.94 ± 3% +3.3 7.26 ± 18% perf-profile.children.cycles-pp.__hrtimer_run_queues 7.97 ± 2% +6.2 14.18 ± 10% perf-profile.children.cycles-pp.hrtimer_interrupt 8.09 ± 2% +6.3 14.43 ± 10% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 12.58 ± 2% +9.7 22.32 ± 10% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 5.43 ± 16% +10.3 15.69 ± 29% perf-profile.children.cycles-pp.menu_select 15.95 ± 3% +13.7 29.69 ± 13% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 59.71 +21.0 80.76 ± 5% perf-profile.children.cycles-pp.cpuidle_enter_state 59.73 +21.1 80.81 ± 5% perf-profile.children.cycles-pp.cpuidle_enter 65.49 +31.4 96.89 perf-profile.children.cycles-pp.start_secondary 66.20 +31.9 98.14 perf-profile.children.cycles-pp.secondary_startup_64_no_verify 66.20 +31.9 98.14 perf-profile.children.cycles-pp.cpu_startup_entry 66.20 +31.9 98.14 perf-profile.children.cycles-pp.do_idle 10.98 ± 3% -10.8 0.19 ± 51% perf-profile.self.cycles-pp.memcpy_toio 6.03 ± 9% -5.4 0.59 ± 43% perf-profile.self.cycles-pp.io_serial_in 1.29 ± 16% -1.2 0.10 ± 76% perf-profile.self.cycles-pp.delay_tsc 0.58 ± 16% -0.5 0.04 ± 85% perf-profile.self.cycles-pp.io_serial_out 0.06 ± 52% +0.1 0.11 ± 29% perf-profile.self.cycles-pp.tick_sched_timer 0.02 ±141% +0.1 0.08 ± 23% perf-profile.self.cycles-pp.rcu_needs_cpu 0.08 ± 12% +0.1 0.15 ± 21% perf-profile.self.cycles-pp.load_balance 0.08 ± 22% +0.1 0.15 ± 23% perf-profile.self.cycles-pp.rcu_dynticks_eqs_enter 0.08 ± 13% +0.1 0.15 ± 32% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.00 +0.1 0.09 ± 23% perf-profile.self.cycles-pp.sched_clock 0.07 ± 14% +0.1 0.16 ± 13% perf-profile.self.cycles-pp.tick_irq_enter 0.07 ± 23% +0.1 0.17 ± 63% perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt 0.16 ± 19% +0.1 0.26 ± 18% perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt 0.08 ± 14% +0.1 0.19 ± 33% perf-profile.self.cycles-pp.timerqueue_add 0.07 ± 59% +0.1 0.19 ± 49% perf-profile.self.cycles-pp.scheduler_tick 0.14 ± 20% +0.1 0.27 ± 9% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.15 ± 12% +0.1 0.29 ± 39% perf-profile.self.cycles-pp.__hrtimer_run_queues 0.17 ± 16% +0.2 0.33 ± 14% perf-profile.self.cycles-pp.hrtimer_interrupt 0.09 ± 23% +0.2 0.25 ± 20% perf-profile.self.cycles-pp.rebalance_domains 0.00 +0.2 0.19 ± 91% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.20 ± 12% +0.2 0.39 ± 29% perf-profile.self.cycles-pp.update_irq_load_avg 0.19 ± 22% +0.2 0.40 ± 27% perf-profile.self.cycles-pp.tsc_verify_tsc_adjust 0.32 ± 13% +0.2 0.53 ± 8% perf-profile.self.cycles-pp.do_idle 0.19 ± 20% +0.2 0.42 ± 33% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.42 ± 10% +0.3 0.71 ± 29% perf-profile.self.cycles-pp.tick_nohz_next_event 0.50 ± 5% +0.3 0.85 ± 16% perf-profile.self.cycles-pp.native_sched_clock 0.44 ± 12% +0.4 0.82 ± 26% perf-profile.self.cycles-pp.irqtime_account_irq 0.42 ± 13% +0.4 0.81 ± 13% perf-profile.self.cycles-pp.update_process_times 0.60 ± 7% +0.4 1.02 ± 12% perf-profile.self.cycles-pp.read_tsc 0.50 ± 7% +0.5 0.96 ± 20% perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.89 ± 7% +0.6 1.54 ± 23% perf-profile.self.cycles-pp.lapic_next_deadline 3.09 ± 27% +8.1 11.17 ± 35% perf-profile.self.cycles-pp.menu_select perf-sched.total_wait_time.max.ms 10000 +-------------------------------------------------------------------+ 9000 |-+ | | | 8000 |-+ | 7000 |-+ | | | 6000 |-+ | 5000 |-+ | 4000 |-+ | | | 3000 |-+ | 2000 |-+ | | | 1000 |-+ | 0 +-------------------------------------------------------------------+ perf-sched.total_wait_time.average.ms 300 +---------------------------------------------------------------------+ | | 250 |-+ O O O O OOOO | | O OO O OO OO OO O O OO OO O O | |OO O O O | 200 |-+ O O O | | | 150 |-+ | | | 100 |-+ | | | | | 50 |-+ | | | 0 +---------------------------------------------------------------------+ perf-sched.total_wait_and_delay.count.ms 8000 +--------------------------------------------------------------------+ | O | 7000 |OO O O OOO | 6000 |-+OOOOO OO OO OO OO OO O OO O | | O O O O O | 5000 |-+ | | | 4000 |-+ | | | 3000 |-+ | 2000 |-+ | | | 1000 |-+ | |+ +++ ++.+++ +++ ++++++.+++ +++ +++ +++ .++++ ++++++++ +.++++++++| 0 +--------------------------------------------------------------------+ perf-sched.total_wait_and_delay.max.ms 10000 +-------------------------------------------------------------------+ 9000 |-+ | | | 8000 |-+ | 7000 |-+ | | | 6000 |-+ | 5000 |-+ | 4000 |-+ | | | 3000 |-+ | 2000 |-+ | | | 1000 |-+ | 0 +-------------------------------------------------------------------+ perf-sched.total_wait_and_delay.average.ms 300 +---------------------------------------------------------------------+ | | 250 |-+ O O O O O O OOOO | | O OO O O O OO O O OO OO O O | |OO O O O | 200 |-+ O O O | | | 150 |-+ | | | 100 |-+ | | | | | 50 |-+ | | | 0 +---------------------------------------------------------------------+ perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read 1200 +--------------------------------------------------------------------+ | | 1000 |OOOOOO OOOOOOOOOOOOO OOOOOOOOOOOO OOO | | | | | 800 |-+ | | | 600 |-+ | | | 400 |-+ | | | | | 200 |-+ | | | 0 +--------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read 90 +----------------------------------------------------------------------+ | O OOOOOO O O O O O O | 80 |-+ O OO O O O O O | 70 |-+ OO O O O O O | | | 60 |-+ | 50 |-+ | | | 40 |-+ | 30 |-+ | | | 20 |-+ | 10 |-+ | | | 0 +----------------------------------------------------------------------+ perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read 2500 +--------------------------------------------------------------------+ | O O O | | O O OO OO OO OOOO | 2000 |-O O OOOOO O O O O | |O OO O O O OO | | | 1500 |-+ | | | 1000 |-+ | | | | | 500 |-+ | | + + +++ + ++ | | +++ ++.+++ + ++ ::+.++ +++ ++ +++ .+ :+++ ++ + :+ : + +++| 0 +--------------------------------------------------------------------+ perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read 1200 +--------------------------------------------------------------------+ | | 1000 |OOOOOOOO OOOOOOOOOOOOOOOO OOOOOOOOOO | | | | | 800 |-+ | | | 600 |-+ | | | 400 |-+ | | | | | 200 |-+ | | | 0 +--------------------------------------------------------------------+ perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read 90 +----------------------------------------------------------------------+ | O OOOOOO O O O O O O | 80 |-+ O OO O O OO O | 70 |-+ O O O O O O O | | | 60 |-+ | 50 |-+ | | | 40 |-+ | 30 |-+ | | | 20 |-+ | 10 |-+ | | | 0 +----------------------------------------------------------------------+ perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork 6000 +--------------------------------------------------------------------+ | O | 5000 |-+ O | | | | | 4000 |-+ | | O | 3000 |-+ | | | 2000 |-+ | | | | | 1000 |OO OOO OOOOOOOOOOOOO OOOOOOO OOO OOO | | | 0 +--------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 800 +---------------------------------------------------------------------+ |OOOOO OOOOOOOO O OOOOOOOOOOO O OO OO | 700 |-+ O O O O | 600 |-+ | | | 500 |-+ | | | 400 |-+ | | | 300 |-+ | 200 |-+ | | | 100 |-+ | | | 0 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork 1400 +--------------------------------------------------------------------+ |OOOO OO O OO OOOOO O O OOOOO O | 1200 |-+ OO O O OO OO OO OO | | | 1000 |-+ | | | 800 |-+ | | | 600 |-+ | | | 400 |-+ | | | 200 |-+ + | |++++++++.++++++++++++++++.++++++++++++++++.+++ ++++++++++++.++++++++| 0 +--------------------------------------------------------------------+ perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork 6000 +--------------------------------------------------------------------+ | O | 5000 |-+ O | | | | | 4000 |-+ | | O | 3000 |-+ | | | 2000 |-+ | | | | | 1000 |OO OOOOO OOOOOOOOOOOOOOOO OO OOO OOO | | | 0 +--------------------------------------------------------------------+ perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 800 +---------------------------------------------------------------------+ |OOOOOO OOOOOOO OOOOO OOOOOOOO OO O O | 700 |-+ O O O O | 600 |-+ | | | 500 |-+ | | | 400 |-+ | | | 300 |-+ | 200 |-+ | | | 100 |-+ | | | 0 +---------------------------------------------------------------------+ 10000 +-------------------------------------------------------------------+ 9000 |-+ | | | 8000 |-+ | 7000 |-+ | | | 6000 |-+ | 5000 |-+ | 4000 |-+ | | | 3000 |-+ | 2000 |-+ | | | 1000 |-+ | 0 +-------------------------------------------------------------------+ 20 +----------------------------------------------------------------------+ 18 |O+OOO O O OOO OOO OOO OOO OOO OOO OO OOO OOO O | | | 16 |-+ | 14 |-+ | | | 12 |-+ | 10 |-+ | 8 |-+ | | | 6 |-+ | 4 |-+ | | | 2 |-+ | 0 +----------------------------------------------------------------------+ 1100 +--------------------------------------------------------------------+ 1000 |OOOO OO OOOOOOOOOO O O OOOOO O | | OO OO OO OO OO | 900 |-+ | 800 |-+ | 700 |-+ | 600 |-+ | | | 500 |-+ | 400 |-+ | 300 |-+ | 200 |-+ | | | 100 |++++++++.++++++++++++++++.++++++++++++++++.++++++++++++++++.++++++++| 0 +--------------------------------------------------------------------+ 10000 +-------------------------------------------------------------------+ 9000 |-+ | | | 8000 |-+ | 7000 |-+ | | | 6000 |-+ | 5000 |-+ | 4000 |-+ | | | 3000 |-+ | 2000 |-+ | | | 1000 |-+ | 0 +-------------------------------------------------------------------+ 20 +----------------------------------------------------------------------+ 18 |O+OOO O O OOO OOO OOO OOO OOO OOO OO OOO OOO O | | | 16 |-+ | 14 |-+ | | | 12 |-+ | 10 |-+ | 8 |-+ | | | 6 |-+ | 4 |-+ | | | 2 |-+ | 0 +----------------------------------------------------------------------+ perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork 10000 +-------------------------------------------------------------------+ 9000 |O+ OOO O O O | | OO O O O OO OOOOOOOOOO OOO O OO | 8000 |-O O O O | 7000 |-+ | | | 6000 |-+ | 5000 |-+ | 4000 |-+ | | | 3000 |-+ | 2000 |-+ | | | 1000 |-+ | 0 +-------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork 600 +---------------------------------------------------------------------+ | | 500 |-+ O O O | | O O O O O O OO O O O O | |O O O O OO OO O O OO OOO O O | 400 |-+ O O | | | 300 |-+ | | | 200 |-+ | | | | | 100 |-+ | | | 0 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork 700 +---------------------------------------------------------------------+ | O O | 600 |OOOOOO OO OOOOOOOO OO OOOOOOO O O | | OO O O O | 500 |-+ | | | 400 |-+ | | | 300 |-+ | | | 200 |-+ | | | 100 |-+ | | | 0 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork 10000 +-------------------------------------------------------------------+ 9000 |O+ OOO O O O | | OO O O O OO OOOOOOOOOO OOO O OO | 8000 |-O O O O | 7000 |-+ | | | 6000 |-+ | 5000 |-+ | 4000 |-+ | | | 3000 |-+ | 2000 |-+ | | | 1000 |-+ | 0 +-------------------------------------------------------------------+ perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork 600 +---------------------------------------------------------------------+ | | 500 |-+ O O O O | | O O O O O O OO O O O | |O O O O OO OO O O OO OOO O O | 400 |-+ O O | | | 300 |-+ | | | 200 |-+ | | | | | 100 |-+ | | | 0 +---------------------------------------------------------------------+ 2500 +--------------------------------------------------------------------+ | | | O | 2000 |-+ | | | | | 1500 |-+ | | O O | 1000 |-+ O | |O | | O O O O | 500 |-+ O OO | | OO O OO OO OOOOOOO OOOOOOO O | | | 0 +--------------------------------------------------------------------+ 300 +---------------------------------------------------------------------+ |O O O O OOOO O O OOOO O OO | 250 |-O OOO O OO OO OO OO OO O O | | | | | 200 |-+ | | | 150 |-+ | | | 100 |-+ | | | | | 50 |-+ | | | 0 +---------------------------------------------------------------------+ 250 +---------------------------------------------------------------------+ | | | | 200 |-+ | | | | | 150 |-+ | | | 100 |-+ | | | | | 50 |-+ | | | | + + + + + + + + | 0 +---------------------------------------------------------------------+ 300 +---------------------------------------------------------------------+ |O O O O OOOO O O OOOO O OO | 250 |-O OOO O OO OO OO OO OO O O | | | | | 200 |-+ | | | 150 |-+ | | | 100 |-+ | | | | | 50 |-+ | | | 0 +---------------------------------------------------------------------+ perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 16 +----------------------------------------------------------------------+ | O O | 14 |OOOOO O OO OOOO OOOOO OO O O OOO O | 12 |-+ OO O O O O O | | | 10 |-+ | | | 8 |-+ | | | 6 |-+ | 4 |-+ | | | 2 |-+ | | | 0 +----------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.9 +---------------------------------------------------------------------+ | O O OO OO OO O O O OOO | 0.8 |O+O O O O O OO OO OO O | 0.7 |-+ OO O O O O O | | | 0.6 |-+ | 0.5 |-+ | | | 0.4 |-+ | 0.3 |-+ | | | 0.2 |-+ | 0.1 |-+ | | + + + + + | 0 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_6 250 +---------------------------------------------------------------------+ |O O OOOOOO O O O | | O O O O O O O | 200 |-+ | | | | | 150 |-+ | | | 100 |-+ | | | | | 50 |-+ | | | | | 0 +---------------------------------------------------------------------+ 16 +----------------------------------------------------------------------+ | O O | 14 |OOOOO O OO OOOO OOOOO OO O O OOO O | 12 |-+ OO O O O O O | | | 10 |-+ | | | 8 |-+ | | | 6 |-+ | 4 |-+ | | | 2 |-+ | | | 0 +----------------------------------------------------------------------+ 0.9 +---------------------------------------------------------------------+ |OOO O OO OO O O O | 0.8 |-+ O O O OO OO OO O | 0.7 |-+ OO O O O O O | | | 0.6 |-+ | 0.5 |-+ | | | 0.4 |-+ | 0.3 |-+ | | | 0.2 |-+ | 0.1 |-+ | | + + + + + + | 0 +---------------------------------------------------------------------+ perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.05 +-------------------------------------------------------------------+ 0.045 |-+ O O | | O | 0.04 |-+ O O | 0.035 |-+ O O | | | 0.03 |-+ | 0.025 |OOOO OO OO O O O | 0.02 |-+ O O O | | O O | 0.015 |-+ OO O O OO OO O OO | 0.01 |-+ | | | 0.005 |-+ :| 0 +-------------------------------------------------------------------+ 40 +----------------------------------------------------------------------+ | O O O O O O | 35 |-+ | 30 |-+ | | | 25 |-+ | | | 20 |-+ | | | 15 |-+ | 10 |-+ | | | 5 |-+ | | | 0 +----------------------------------------------------------------------+ perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_for 600 +---------------------------------------------------------------------+ | | 500 |OOOOO OOOOOOOOOO OOOOOOOOOOO OOOOOOOO | | | | | 400 |-+ | | | 300 |-+ | | | 200 |-+ | | | | | 100 |-+ | | | 0 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_for 500 +---------------------------------------------------------------------+ 450 |OOOOO OOOOOOOOOO OOOOOOOOOOO OOOOOOOO | | | 400 |-+ | 350 |-+ | | | 300 |-+ | 250 |-+ | 200 |-+ | | | 150 |-+ | 100 |-+ | | | 50 |-+ | 0 +---------------------------------------------------------------------+ 50 +----------------------------------------------------------------------+ 45 |-+ O O O O O O O | | O | 40 |-+ OO O O O | 35 |-+ O O O O O | | OO OO O O O OO | 30 |O+ O O O O | 25 |-+ | 20 |-O O | | | 15 |-+ | 10 |-+ | | | 5 |-+ | 0 +----------------------------------------------------------------------+ 450 +---------------------------------------------------------------------+ | O | 400 |-+ | 350 |O+ O | | O OO O O O OOOO | 300 |-+ O O O O O O | 250 |-+ O O O O O | | O O O O O O | 200 |-+ O O O | 150 |-+ | | | 100 |-+ | 50 |-+ | | | 0 +---------------------------------------------------------------------+ 450 +---------------------------------------------------------------------+ | O | 400 |-+ O | 350 |-+ | |O O O O OO | 300 |-+OO OOO OO O OO | 250 |-+ O O O O OO O | | O O O | 200 |-+ O O O O O O | 150 |-+ | | | 100 |-+ | 50 |-+ | | | 0 +---------------------------------------------------------------------+ 20 +----------------------------------------------------------------------+ | | | | 15 |-+ | | | | | | | 10 |-+ | | | | | 5 |-+ | | | | | | .+.+ | 0 +----------------------------------------------------------------------+ 1000 +--------------------------------------------------------------------+ 900 |-+ | | | 800 |-+ | 700 |-+ | | | 600 |-+ | 500 |-+ | 400 |-+ | | | 300 |-+ | 200 |-+ | | | 100 |-+ | 0 +--------------------------------------------------------------------+ 900 +---------------------------------------------------------------------+ | | 800 |-+ | 700 |-+ | | | 600 |-+ | 500 |-+ | | | 400 |-+ | 300 |-+ | | | 200 |-+ | 100 |-+ | | | 0 +---------------------------------------------------------------------+ 14 +----------------------------------------------------------------------+ |O | 12 |-+ O O OO | | OO O O OOOOOO O OOOOOOOOOOO OOOOOOO | 10 |-+ | | | 8 |-+ | | | 6 |-+ | | | 4 |-+ | | | 2 |-+ | | + +. + .+ + + + +.+ ++ ++ | 0 +----------------------------------------------------------------------+ 1200 +--------------------------------------------------------------------+ | | 1000 |OOOO OOOOOOOOO OOOOOOOOO OOOOOOOO OOOO | | | | | 800 |-+ | | | 600 |-+ | | | 400 |-+ | | | | | 200 |-+ | | | 0 +--------------------------------------------------------------------+ 900 +---------------------------------------------------------------------+ | OO OO OOOO OO OOOO OOOOOOOO OOOOOOO | 800 |-+ O O OO | 700 |O+ | | | 600 |-+ | 500 |-+ | | | 400 |-+ | 300 |-+ | | | 200 |-+ | 100 |-+ | | | 0 +---------------------------------------------------------------------+ stress-ng.time.system_time 30 +----------------------------------------------------------------------+ | | 25 |-++ + + ++++ + + + + + ++++ + | | ::+.+ ++ +++++ + + : ::: ::++ : :+ ++ +++ + + :++++++ .++ +| |++ + + + + + ++.+ + +++ + + + + ++ | 20 |-+ | | | 15 |-+ | | | 10 |-+ | | | | | 5 |-+ | | | 0 +----------------------------------------------------------------------+ stress-ng.time.elapsed_time 900 +---------------------------------------------------------------------+ | | 800 |-+ | 700 |-+O O OOOOOO O O O | | | 600 |-+ | 500 |-O O O O O O OO OO | | | 400 |-+ O O | 300 |O+ O O | | | 200 |-+ | 100 |-+ | |++++++.+++++++++++++.+++++++++++++.+++++++++++++.+++++++++++++.++++++| 0 +---------------------------------------------------------------------+ stress-ng.time.elapsed_time.max 900 +---------------------------------------------------------------------+ | | 800 |-+ | 700 |-+O O OOOOOO O O O | | | 600 |-+ | 500 |-O O O O O O OO OO | | | 400 |-+ O O | 300 |O+ O O | | | 200 |-+ | 100 |-+ | |++++++.+++++++++++++.+++++++++++++.+++++++++++++.+++++++++++++.++++++| 0 +---------------------------------------------------------------------+ stress-ng.time.voluntary_context_switches 90000 +-------------------------------------------------------------------+ | + | 80000 |-+ :+ + | 70000 |+++ .+ + : + + + + + :: + + + +| |: :++ + : + : : : : :++ : :++ : + :: + + +: + :: :| 60000 |-:: + : : :+ : ++: +: :: +. : :: + + : + :: ::+ :+: :++ | 50000 |-: ++: ++ + : + + : ++ : + + +: +: + + : | | + + + + + + + + | 40000 |-+ | 30000 |-+ | | | 20000 |-+ | 10000 |-+ | | O O O O | 0 +-------------------------------------------------------------------+ stress-ng.time.involuntary_context_switches 4500 +--------------------------------------------------------------------+ | + + +++ + ++ + + + + .+ ++++ + | 4000 |-+:: + +. +++ :+ :+ :+ : :: + : :: + : : :+ :++ +. + +| 3500 |++ ++ ++ + + + +++ ++ +++ ++ +++ + + + ++++ ++ | | | 3000 |-+ | 2500 |-+ | | | 2000 |-+ | 1500 |-+ | | | 1000 |-+ | 500 |-+ | | | 0 +--------------------------------------------------------------------+ stress-ng.loop.ops 1800 +--------------------------------------------------------------------+ | + + ++ + + + + + .+ + + | 1600 |-+:: + : + + :+ :+++ + + :: + + :: + : ++++ :+ :+ +.+ + | 1400 |++ ++ +.++ ++ + + + ++ ++ ++ ++ ++ + + ++ +++++ +| | | 1200 |-+ | 1000 |-+ | | | 800 |-+ | 600 |-+ | | | 400 |-+ | 200 |-+ | | | 0 +--------------------------------------------------------------------+ stress-ng.loop.ops_per_sec 30 +----------------------------------------------------------------------+ | + + ++ + + + + + + + + | 25 |-+:: .+ : + + + :+++ :: :: + : :+ + : ++++ + + :+ ++ + | |++ ++ +++ ++ +++ + ++.+ ++ +++ ++ ++ + + ++ ++.+++ +| | | 20 |-+ | | | 15 |-+ | | | 10 |-+ | | | | | 5 |-+ | | | 0 +----------------------------------------------------------------------+ stress-ng.time.percent_of_cpu_this_job_got 42 +----------------------------------------------------------------------+ | : | 41 |-++ + : + + + + + | 40 |-+: : : : : ++ + : : : : + ++ + | | : : : : :::: : : : :: : :::: : | 39 |-+: :: ++ : :+ : : : :: :: :: :+ : +: + + | | :: : : : : : : : :: :: :: : : : : :: : : | 38 |-: : + : : : : : : : : + : : : + : : : :: :: : | | : : : : : : : : : : : : : : : : : : : :: ::: : | 37 |-: : : : ++: : : : : : :: : : :: :+: : : ::+: : + +| 36 |-+ ++ ++ : + ++: : : + ++ : + ++ + + :: : + ++ +: + | | : : : : : + : : :: : :+ :: | 35 |++ + + + ++ ++ : : + + | | : : | 34 +----------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang