From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65870C83F1A for ; Thu, 10 Jul 2025 05:51:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0C936B00A8; Thu, 10 Jul 2025 01:51:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DBD976B00A9; Thu, 10 Jul 2025 01:51:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD3496B00AA; Thu, 10 Jul 2025 01:51:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AAD606B00A8 for ; Thu, 10 Jul 2025 01:51:53 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 21E331D955B for ; Thu, 10 Jul 2025 05:51:53 +0000 (UTC) X-FDA: 83647283706.11.BADF0D2 Received: from mxhk.zte.com.cn (mxhk.zte.com.cn [160.30.148.34]) by imf01.hostedemail.com (Postfix) with ESMTP id 351284000D for ; Thu, 10 Jul 2025 05:51:49 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of jiang.kun2@zte.com.cn designates 160.30.148.34 as permitted sender) smtp.mailfrom=jiang.kun2@zte.com.cn; dmarc=pass (policy=none) header.from=zte.com.cn ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752126711; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references; bh=edTAgnkrcXDZuE6Fe5vC9Asg09tJWesHEOM5So3H7FU=; b=m7OXjqI/fxHzB4F2E6b4gMllryEAUCVITYHDJzF+KQ6o4+amxDU7BH4woOeWewqlqJa8bu 3uX0LVFYwguPDPX227D28mPncJoqkO2rAamlRhnmbWGibplIh0NMkCajqPXkFcNhx0LjVj bN7rYS1WwBQ2mXLsZfw1Yw/2K3UmKnU= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of jiang.kun2@zte.com.cn designates 160.30.148.34 as permitted sender) smtp.mailfrom=jiang.kun2@zte.com.cn; dmarc=pass (policy=none) header.from=zte.com.cn ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752126711; a=rsa-sha256; cv=none; b=nNavSyazmphShnxzxjicCg23VrYB8grAV8Lva2xnGQjTuXEO9eKG721rJxIdyGvsrBeKjD ehCZ0Y3x4RZ6v56Sa63gcv03pmV0wQQyiWOLGkscaUZPqq6HOTToj83472Uy9pJWx2T4dx ghwl29bJ9sik8fxJirq5ByhXa+sYGm4= Received: from mse-fl2.zte.com.cn (unknown [10.5.228.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mxhk.zte.com.cn (FangMail) with ESMTPS id 4bd3rL3f6Cz5F2lt; Thu, 10 Jul 2025 13:51:46 +0800 (CST) Received: from njy2app02.zte.com.cn ([10.40.13.116]) by mse-fl2.zte.com.cn with SMTP id 56A5pen3017791; Thu, 10 Jul 2025 13:51:40 +0800 (+08) (envelope-from jiang.kun2@zte.com.cn) Received: from mapi (njy2app01[null]) by mapi (Zmail) with MAPI id mid204; Thu, 10 Jul 2025 13:51:41 +0800 (CST) Date: Thu, 10 Jul 2025 13:51:41 +0800 (CST) X-Zmail-TransId: 2af9686f54ed565-26977 X-Mailer: Zmail v1.0 Message-ID: <2025071013514177028RdjISjqeIOnTCRvGAwy@zte.com.cn> Mime-Version: 1.0 From: To: , , , Cc: , , , , , , , , , , Subject: =?UTF-8?B?W1BBVENIIGxpbnV4IG5leHRdIGRvY3M6IHVwZGF0ZSBkb2NzIGFmdGVyIGludHJvZHVjaW5nIGRlbGF5dG9w?= Content-Type: text/plain; charset="UTF-8" X-MAIL:mse-fl2.zte.com.cn 56A5pen3017791 X-Fangmail-Anti-Spam-Filtered: true X-Fangmail-MID-QID: 686F54F2.000/4bd3rL3f6Cz5F2lt X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 351284000D X-Stat-Signature: tc9f6enepo9r4mpmw79cdp9g4jxysmsw X-Rspam-User: X-HE-Tag: 1752126709-675794 X-HE-Meta: U2FsdGVkX1/QqAO9cxhXevCEcKeLs/80GybAOZyaKBRVRUXV9so1cA4XGebmdrwpTDoTVqyHXW6bdlXcMZaLCb6kKHa32L0FiQsr05Yidd27iIxpbGMC7W/luzFba10BliFxZN/hcfyOEBRF2/DxhfAqhjZ7GzPyJIE83tcagUb/0DvsYrJzKQBdelqGz4pUh0uncMnI6KfVG8nDEdvRv3cXHqXzxvBI1nlWPsQjAsig4egrlB5iOjfmvtlYVKLrEKJvs3Sp5cJWQjypd1frUx2ZPmnOMqpSNAjOUMhv24xWNpKbg8ahL3+IAFxAJL5Oad/ATJ2Y+fPxBmYjAxRNDXQKuMMm3IJNX6G/TMb3bPCNhgkmKgmehww/mp9qV8LMdaGZWbMi6qA9IPbXPQi0ILQGnupXN5ZMVM9H4cwEGqU50kSRoQSPBgWOOWtO5zNN/+950nJF1ChoR4aUZWulH1pwmhBQVlsCI5GKRAv0foj1aOtrYWwKw9RvkmjsOEu3nmcDRGLm8OW7vtcolZS09s0gALIW2u/BxxeVwiM8qe123A2GqNbEegPkIQOeMu3k+UNVounas5TD78cyxbYjS/ty56ixbZNURzmzFYeGStSk1Q4o/xsaxSUP6JcTLHBo86NIyCwX+DvigFzlVNvYzRiRcX2rJEfKbaOEFiOQXdI2wYKD+K/IazQ1M+kQuDmbQ1+HMRcLWX7cKwN6XQ0SirHKmrd9d8/5mMFA5jYDX+LxY0f1oF0KHeRgrXQnpgMLP7KHr5OaJb/MffHUYlGVfS6GgzVKQiYIFh3c+m0d+tr8Dl2l8cgX6cTm5lgAzettU4i6yGfFWTGgn5dSNjgxBJ5IBAVMR8IlmrwM3MXbgxAh+k0DzTKtpNz/nH1o6DwiPtBxqebFcvd+LefOC7jKue/LK6rUGHS7u4ubCT1BvgKQeZktiNCp3TWPOsH9BgWtPt47i5f3gBUrskuG8VW PGOicEoV mqYsINAk6lsJ1VurwAkQWz9szZdy8uecoOz5mYrlm+3+RHgm8NE5JSC8qGEr4Y75pagS67h8LNuurMwiEQf1Vn9ZveI0Ilch3JQ+4xt/excLUmKuYTqFE8QoH/cfxf53Kn4YnpcNxqjrfzU6cPAFG2qFwbHn3r+RwrZC/3M6FrRW7XLhpIXh6Z6nTGHv+bCcQ9sAEmhUT0S98lORshaPZftaJgQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Wang Yaxin The "getdelays" can only display the latency of a single task by specifying a PID, we introduce the "delaytop" with the following capabilities: 1. system view: monitors latency metrics (CPU, I/O, memory, IRQ, etc.) for all system processes 2. supports field-based sorting (e.g., default sort by CPU latency in descending order) 3. dynamic interactive interface: focus on specific processes with --pid; limit displayed entries with --processes 20; control monitoring duration with --iterations; Signed-off-by: Fan Yu Signed-off-by: Wang Yaxin Signed-off-by: Jiang Kun --- Documentation/accounting/delay-accounting.rst | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/Documentation/accounting/delay-accounting.rst b/Documentation/accounting/delay-accounting.rst index 210c194d4a7b..664950328fb7 100644 --- a/Documentation/accounting/delay-accounting.rst +++ b/Documentation/accounting/delay-accounting.rst @@ -131,3 +131,50 @@ Get IO accounting for pid 1, it works only with -p:: linuxrc: read=65536, write=0, cancelled_write=0 The above command can be used with -v to get more debug information. + +After the system starts, use `delaytop` to get the Top-N high-latency tasks. +this tool supports sorting by CPU latency in descending order by default, +displays the top 20 high-latency tasks by default, and refreshes the latency +data every 2 seconds by default. + +Get Top-N tasks delay, since system boot:: + + bash# ./delaytop + Top 20 processes (sorted by CPU delay): + + PID TGID COMMAND CPU(ms) IO(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms) IRQ(ms) + --------------------------------------------------------------------------------------------- + 32 32 kworker/2:0H-sy 23.65 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 497 497 kworker/R-scsi_ 1.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 495 495 kworker/R-scsi_ 1.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 494 494 scsi_eh_0 1.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 485 485 kworker/R-ata_s 0.90 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 574 574 kworker/R-kdmfl 0.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 34 34 idle_inject/3 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 1123 1123 nde-netfilter 0.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 60 60 ksoftirqd/7 0.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 114 114 kworker/0:2-cgr 0.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 496 496 scsi_eh_1 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 51 51 cpuhp/6 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 1667 1667 atd 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 45 45 cpuhp/5 0.23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 1102 1102 nde-backupservi 0.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 1098 1098 systemsettings 0.21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 1100 1100 audit-monitor 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 53 53 migration/6 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 1482 1482 sshd 0.19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + 39 39 cpuhp/4 0.19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 + +Dynamic interactive interface of delaytop:: + + # ./delaytop -p pid + Print delayacct stats + + # ./delaytop -P num + Display the top N tasks + + # ./delaytop -n num + Set delaytop refresh frequency (num times) + + # ./delaytop -d secs + Specify refresh interval as secs -- 2.25.1