| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Enhanced display logic to conditionally show PSI information only when
successfully read, with helpful guidance for users to enable PSI support
(psi=1 cmdline parameter).
Link: https://lkml.kernel.org/r/20250907001417537vSx6nUsb3ILqI0iQ-WnGp@zte.com.cn
Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
Reviewed-by: xu xin <xu.xin16@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The original delaytop only supported static output with limited
interaction. Users had to restart the tool with different command-line
options to change sorting or display modes, which disrupted continuous
monitoring and reduced productivity during performance investigations.
Adds real-time interactive controls through keyboard input:
1) Add interactive menu system with visual prompts
2) Support dynamic sorting changes without restarting
3) Enable toggle of memory verbose mode with 'M' key
The interactive mode transforms delaytop from a static monitoring tool
into a dynamic investigation platform, allowing users to adapt the view in
real-time based on observed performance patterns.
Link: https://lkml.kernel.org/r/20250907001338580EURha20BxWFmBSrUpS8D1@zte.com.cn
Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
Reviewed-by: xu xin <xu.xin16@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The original delaytop tool always displayed detailed memory subsystem
breakdown, which could be overwhelming for users who only need high-level
overview.
Add flexible display control allowing users to choose their preferred
information granularity.
The new flexibility provides:
1) For quick monitoring: use normal mode to reduce visual clutter
2) For deep analysis: use verbose mode to see all memory subsystem details
Link: https://lkml.kernel.org/r/202509070012527934u0ySb3teQ4gOYKnocyNO@zte.com.cn
Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
Reviewed-by: xu xin <xu.xin16@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "tools/delaytop: implement real-time keyboard interaction
support", v2.
Current Limitations
===================
The current delaytop implementation has two main limitations:
1) Static sorting only by CPU delay Forcing users to restart with
different parameters to analyze other resource bottlenecks.
2) Memory delay information is always expanded Causing information
overload when only high-level memory pressure monitoring is needed.
Improvements
============
1) Implemented dynamic sorting capability
- Interactive key 'o' triggers sort mode.
- Supports sorting by CPU/IO/Memory/IRQ delays.
- Memory subcategories available in verbose mode.
* c - CPU delay (default)
* i - IO delay
* m - Total memory delay
* q - IRQ delay
* s/r/t/p/w - Memory subcategories (in verbose mode)
2) Added memory display modes
- Compact view (default): shows aggregated memory delays.
- Verbose view ('M' key): breaks down into memory sub-delays.
* SWAP - swapin delays
* RCL - freepages reclaim delays
* THR - thrashing delays
* CMP - compaction delays
* WP - write-protect copy delays
Practical benefits
==================
1) Dynamic Sorting for Real-Time Bottleneck Detection System
administrators can now dynamically change sorting to identify different
types of resource bottlenecks without restarting.
2) Enhanced Usability with On-Screen Keybindings More intuitive
interactive usage with on-screen keybindings help. Reduced screen
clutter when only memory overview is needed.
Use Case
========
# ./delaytop
System Pressure Information: (avg10/avg60vg300/total)
CPU some: 0.0%/ 0.0%/ 0.0%/ 106817(ms)
CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms)
IO full: 0.0%/ 0.0%/ 0.0%/ 2245(ms)
IO some: 0.0%/ 0.0%/ 0.0%/ 2791(ms)
IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
[o]sort [M]memverbose [q]quit
Top 20 processes (sorted by cpu delay):
PID TGID COMMAND CPU(ms) IO(ms) IRQ(ms) MEM(ms)
------------------------------------------------------------------------
110 110 kworker/15:0H-s 27.91 0.00 0.00 0.00
57 57 cpuhp/7 3.18 0.00 0.00 0.00
99 99 cpuhp/14 2.97 0.00 0.00 0.00
51 51 cpuhp/6 0.90 0.00 0.00 0.00
44 44 kworker/4:0H-sy 0.80 0.00 0.00 0.00
76 76 idle_inject/10 0.31 0.00 0.00 0.00
100 100 idle_inject/14 0.30 0.00 0.00 0.00
1309 1309 systemsettings 0.29 0.00 0.00 0.00
60 60 ksoftirqd/7 0.28 0.00 0.00 0.00
45 45 cpuhp/5 0.22 0.00 0.00 0.00
63 63 cpuhp/8 0.20 0.00 0.00 0.00
87 87 cpuhp/12 0.18 0.00 0.00 0.00
93 93 cpuhp/13 0.17 0.00 0.00 0.00
1265 1265 acpid 0.17 0.00 0.00 0.00
1552 1552 sshd 0.17 0.00 0.00 0.00
2584 2584 sddm-helper 0.16 0.00 0.00 0.00
1284 1284 rtkit-daemon 0.15 0.00 0.00 0.00
1326 1326 nde-netfilter 0.14 0.00 0.00 0.00
27 27 cpuhp/2 0.13 0.00 0.00 0.00
631 631 kworker/11:2-rc 0.11 0.00 0.00 0.00
# ./delaytop -M
System Pressure Information: (avg10/avg60vg300/total)
CPU some: 0.0%/ 0.0%/ 0.0%/ 106827(ms)
CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms)
IO full: 0.0%/ 0.0%/ 0.0%/ 2245(ms)
IO some: 0.0%/ 0.0%/ 0.0%/ 2791(ms)
IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
[o]sort [M]memverbose [q]quit
Top 20 processes (sorted by mem delay):
PID TGID COMMAND MEM(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms)
------------------------------------------------------------------------------------------
121732 121732 delaytop 0.01 0.00 0.00 0.00 0.00 0.01
95876 95876 top 0.00 0.00 0.00 0.00 0.00 0.00
121641 121641 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00
121693 121693 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00
121661 121661 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00
1 1 systemd 0.00 0.00 0.00 0.00 0.00 0.00
2 2 kthreadd 0.00 0.00 0.00 0.00 0.00 0.00
3 3 pool_workqueue_ 0.00 0.00 0.00 0.00 0.00 0.00
4 4 kworker/R-rcu_g 0.00 0.00 0.00 0.00 0.00 0.00
5 5 kworker/R-rcu_p 0.00 0.00 0.00 0.00 0.00 0.00
6 6 kworker/R-slub_ 0.00 0.00 0.00 0.00 0.00 0.00
7 7 kworker/R-netns 0.00 0.00 0.00 0.00 0.00 0.00
9 9 kworker/0:0H-sy 0.00 0.00 0.00 0.00 0.00 0.00
11 11 kworker/u32:0-n 0.00 0.00 0.00 0.00 0.00 0.00
12 12 kworker/R-mm_pe 0.00 0.00 0.00 0.00 0.00 0.00
13 13 rcu_tasks_kthre 0.00 0.00 0.00 0.00 0.00 0.00
14 14 rcu_tasks_rude_ 0.00 0.00 0.00 0.00 0.00 0.00
15 15 rcu_tasks_trace 0.00 0.00 0.00 0.00 0.00 0.00
16 16 ksoftirqd/0 0.00 0.00 0.00 0.00 0.00 0.00
17 17 rcu_preempt 0.00 0.00 0.00 0.00 0.00 0.00
When psi is not enabled:
# ./delaytop
System Pressure Information: (avg10/avg60vg300/total)
PSI not found: check if psi=1 enabled in cmdline
This patch (of 5):
The delaytop tool only supported sorting by CPU delay, which limited its
usefulness when users needed to identify bottlenecks in other subsystems.
Users had no way to sort processes by IO, IRQ, or other delay types to
quickly pinpoint specific performance issues.
Add -s/--sort option to allow sorting by different delay types. Users can
now quickly identify bottlenecks in specific subsystems by sorting
processes by the relevant delay metric.
Link: https://lkml.kernel.org/r/20250907001101305vrTGnXaRNvtmsGkp-Ljk_@zte.com.cn
Link: https://lkml.kernel.org/r/20250907001205573L3XpsQMIQnLgDqiiKYd3H@zte.com.cn
Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
Reviewed-by: xu xin <xu.xin16@zte.com.cn>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Wang Yaxin <wang.yaxin@zte.com.cn>
Cc: Yang Yang <yang.yang29@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
This patch improves error diagnostics and documentation for delaytop:
1) Enhanced error logging:
- Added explicit error messages in critical failure paths
- Implemented BOOL_FPRINT macro for robust output handling
2) PSI feature documentation:
- Updated header comment to reflect PSI monitoring capability
- Improved output formatting for PSI information
System Pressure Information: (avg10/avg60/avg300/total)
CPU some: 0.0%/ 0.0%/ 0.0%/ 345(ms)
CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms)
IO full: 0.0%/ 0.0%/ 0.0%/ 65(ms)
IO some: 0.0%/ 0.0%/ 0.0%/ 79(ms)
IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Link: https://lkml.kernel.org/r/202507281628341752gMXCMN7S-Vz_LHYHum9r@zte.com.cn
Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn>
Acked-by: Yang Yang <yang.yang29@zte.com.cn>
Cc: Fan Yu <fan.yu9@zte.com.cn>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: xu xin <xu.xin16@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Support showing whole delay of system by reading PSI, just like the first
few lines of information output by the top command. the output of
delaytop includes both system-wide delay and delay of individual tasks,
providing a more comprehensive reflection of system latency status.
Use case
========
bash# ./delaytop
System Pressure Information: (avg10/avg60/avg300/total)
CPU: full: 0.0%/ 0.0%/ 0.0%/0 some: 0.1%/ 0.0%/ 0.0%/14216596
Memory: full: 0.0%/ 0.0%/ 0.0%/34010659 some: 0.0%/ 0.0%/ 0.0%/35406492
IO: full: 0.1%/ 0.0%/ 0.0%/51029453 some: 0.1%/ 0.0%/ 0.0%/55330465
IRQ: full: 0.0%/ 0.0%/ 0.0%/0
Top 20 processes (sorted by CPU delay):
PID TGID COMMAND CPU(ms) IO(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms) IRQ(ms)
---------------------------------------------------------------------------------------------
32 32 kworker/2:0H-sy 23.65 0.00 0.00 0.00 0.00 0.00 0.00 0.00
497 497 kworker/R-scsi_ 1.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
495 495 kworker/R-scsi_ 1.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
494 494 scsi_eh_0 1.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00
485 485 kworker/R-ata_s 0.90 0.00 0.00 0.00 0.00 0.00 0.00 0.00
574 574 kworker/R-kdmfl 0.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
34 34 idle_inject/3 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1123 1123 nde-netfilter 0.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00
60 60 ksoftirqd/7 0.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
114 114 kworker/0:2-cgr 0.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
496 496 scsi_eh_1 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00
51 51 cpuhp/6 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1667 1667 atd 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00
45 45 cpuhp/5 0.23 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1102 1102 nde-backupservi 0.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1098 1098 systemsettings 0.21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1100 1100 audit-monitor 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
53 53 migration/6 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1482 1482 sshd 0.19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
39 39 cpuhp/4 0.19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Link: https://lkml.kernel.org/r/20250710135451340_5pOgpIFi0M5AE7H44W1D@zte.com.cn
Co-developed-by: Fan Yu <fan.yu9@zte.com.cn>
Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn>
Signed-off-by: Jiang Kun <jiang.kun2@zte.com.cn>
Cc: Balbir Singh <bsingharora@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Peilin He <he.peilin@zte.com.cn>
Cc: Qiang Tu <tu.qiang35@zte.com.cn>
Cc: wangyong <wang.yong12@zte.com.cn>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Yang Yang <yang.yang29@zte.com.cn>
Cc: Yunkai Zhang <zhang.yunkai@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Problem
=======
The "getdelays" can only display the latency of a single task
by specifying a PID, but it has the following limitations:
1. single-task perspective: only supports querying the latency (CPU, I/O,
memory, etc.) of an individual task via PID and cannot provide a global
analysis of high-latency processes across the system.
2. lack of High-Latency process awareness: when the overall system
latency is high (e.g., a spike in CPU latency), there is no way to
quickly identify the top N processes contributing to the highest
latency.
3. poor interactivity: It lacks dynamic sorting and refresh
capabilities (similar to top), making it difficult to monitor latency
changes in real time.
Solution
========
To address these limitations, we introduce the "delaytop" with the
following capabilities:
1. system view: monitors latency metrics (CPU, I/O, memory, IRQ, etc.)
for all system processes
2. supports field-based sorting (e.g., default sort by CPU latency in
descending order)
3. dynamic interactive interface: focus on specific processes with
--pid; limit displayed entries with --processes 20; control monitoring
duration with --iterations;
Use case
========
bash# ./delaytop
Top 20 processes (sorted by CPU delay):
PID TGID COMMAND CPU(ms) IO(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms) IRQ(ms)
---------------------------------------------------------------------------------------------
26 26 kworker/1:0H 5.55 0.00 0.00 0.00 0.00 0.00 0.00 0.00
32 32 kworker/2:0H-kb 2.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00
38 38 kworker/3:0H-ev 2.88 0.00 0.00 0.00 0.00 0.00 0.00 0.00
84 84 kworker/R-vfio- 1.62 0.00 0.00 0.00 0.00 0.00 0.00 0.00
24 24 ksoftirqd/1 1.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00
19 19 idle_inject/0 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00
16 16 rcu_exp_par_gp_ 0.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11 11 kworker/0:1 0.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00
22 22 idle_inject/1 0.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00
3 3 pool_workqueue_ 0.74 0.00 0.00 0.00 0.00 0.00 0.00 0.00
81 81 scsi_eh_1 0.59 0.00 0.00 0.00 0.00 0.00 0.00 0.00
30 30 ksoftirqd/2 0.42 0.00 0.00 0.00 0.00 0.00 0.00 0.00
36 36 ksoftirqd/3 0.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
9 9 kworker/0:0-eve 0.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
8 8 kworker/R-netns 0.34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
76 76 kworker/1:1-pm 0.32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
21 21 cpuhp/1 0.30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
4 4 kworker/R-rcu_g 0.21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12 12 kworker/u16:0-i 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1 1 init 0.18 0.00 0.00 0.00 0.00 0.00 0.08 0.00
Link: https://lkml.kernel.org/r/20250619211843633h05gWrBDMFkEH6xAVm_5y@zte.com.cn
Co-developed-by: Fan Yu <fan.yu9@zte.com.cn>
Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
Signed-off-by: Yaxin Wang <wang.yaxin@zte.com.cn>
Cc: Balbir Singh <bsingharora@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Peilin He <he.peilin@zte.com.cn>
Cc: Qiang Tu <tu.qiang35@zte.com.cn>
Cc: wangyong <wang.yong12@zte.com.cn>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Yang Yang <yang.yang29@zte.com.cn>
Cc: ye xingchen <ye.xingchen@zte.com.cn>
Cc: Yunkai Zhang <zhang.yunkai@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
