linux/tools/tracing/rtla/src/timerlat_hist.c, branch v6.17

rtla/timerlat: Add action on end feature

2025-07-25T20:43:57Z

Implement actions on end next to actions on threshold. A new option, --on-end is added, parallel to --on-threshold. Instead of being executed whenever a latency threshold is reached, it is executed at the end of the measurement. For example: $ rtla timerlat hist -d 5s --on-end trace will save the trace output at the end. All actions supported by --on-threshold are also supported by --on-end, except for continue, which does nothing with --on-end. Cc: John Kacur Cc: Luis Goncalves Cc: Arnaldo Carvalho de Melo Cc: Chang Yin Cc: Costa Shulyupin Cc: Crystal Wood Cc: Gabriele Monaco Link: https://lore.kernel.org/20250626123405.1496931-6-tglozar@redhat.com Signed-off-by: Tomas Glozar Signed-off-by: Steven Rostedt (Google)

rtla/timerlat: Add continue action

2025-07-25T20:43:57Z

Introduce option to resume tracing after a latency threshold overflow. The option is implemented as an action named "continue". Example: $ rtla timerlat top -q -T 200 -d 1s --on-threshold \ exec,command="echo Threshold" --on-threshold continue Threshold Threshold Threshold Timer Latency ... The feature is supported for both hist and top. After the continue action is executed, processing of the list of actions is stopped and tracing is resumed. Cc: John Kacur Cc: Luis Goncalves Cc: Arnaldo Carvalho de Melo Cc: Chang Yin Cc: Costa Shulyupin Cc: Crystal Wood Cc: Gabriele Monaco Link: https://lore.kernel.org/20250626123405.1496931-5-tglozar@redhat.com Signed-off-by: Tomas Glozar Signed-off-by: Steven Rostedt (Google)

rtla/timerlat: Add action on threshold feature

2025-07-25T20:43:56Z

Extend the functionality provided by the -t/--trace option, which triggers saving the contents of a tracefs buffer after tracing is stopped, to support implementing arbitrary actions. A new option, --on-threshold, is added, taking an argument that further specifies the action. Actions added in this patch are: - trace[,file=]: Saves tracefs buffer, optionally taking a filename. - signal,num=,pid=: Sends signal to process. "parent" might be specified instead of number to send signal to parent process. - shell,command=: Execute shell command. Multiple actions may be specified and will be executed in order, including multiple actions of the same type. Trace output requested via -t and -a now adds a trace action to the end of the list. If an action fails, the following actions are not executed. For example, this command: $ rtla timerlat -T 20 --on-threshold trace \ --on-threshold shell,command="grep ipi_send timerlat_trace.txt" \ --on-threshold signal,num=2,pid=parent will send signal 2 (SIGINT) to parent process, but only if saved trace contains the text "ipi_send". This way, the feature can be used for flexible reactions on latency spikes, and allows combining rtla with other tooling like perf. Cc: John Kacur Cc: Luis Goncalves Cc: Arnaldo Carvalho de Melo Cc: Chang Yin Cc: Costa Shulyupin Cc: Crystal Wood Cc: Gabriele Monaco Link: https://lore.kernel.org/20250626123405.1496931-3-tglozar@redhat.com Signed-off-by: Tomas Glozar Signed-off-by: Steven Rostedt (Google)

rtla/timerlat: Introduce enum timerlat_tracing_mode

2025-07-25T20:43:56Z

After the introduction of BPF-based sample collection, rtla-timerlat effectively runs in one of three modes: - Pure BPF mode, with tracefs only being used to set up the timerlat tracer. Sample processing and stop on threshold are handled by BPF. - tracefs mode. BPF is unsupported or kernel is lacking the necessary trace event (osnoise:timerlat_sample). Stop on theshold is handled by timerlat tracer stopping tracing in all instances. - BPF/tracefs mixed mode - BPF is used for sample collection for top or histogram, tracefs is used for trace output and/or auto-analysis. Stop on threshold is handled both through BPF program, which stops sample collection for top/histogram and wakes up rtla, and by timerlat tracer, which stops tracing for trace output/auto-analysis instances. Add enum timerlat_tracing_mode, with three values: - TRACING_MODE_BPF - TRACING_MODE_TRACEFS - TRACING_MODE_MIXED Those represent the modes described above. A field of this type is added to struct timerlat_params, named "mode", replacing the no_bpf variable. params->mode is set in timerlat_{top,hist}_parse_args to TRACING_MODE_BPF or TRACING_MODE_MIXED based on whether trace output and/or auto-analysis is requested. timerlat_{top,hist}_main then checks if BPF is not unavailable or disabled, in that case, it sets params->mode to TRACING_MODE_TRACEFS. A condition is added to timerlat_apply_config that skips setting timerlat tracer thresholds if params->mode is TRACING_MODE_BPF (those are unnecessary, since they only turn off tracing, which is already turned off in that case, since BPF is used to collect samples). Cc: John Kacur Cc: Luis Goncalves Cc: Arnaldo Carvalho de Melo Cc: Chang Yin Cc: Costa Shulyupin Cc: Crystal Wood Cc: Gabriele Monaco Link: https://lore.kernel.org/20250626123405.1496931-2-tglozar@redhat.com Signed-off-by: Tomas Glozar Signed-off-by: Steven Rostedt (Google)

rtla: Set distinctive exit value for failed tests

2025-05-07T19:36:19Z

A test is considered failed when a sample trace exceeds the threshold. Failed tests return the same exit code as passed tests, requiring test frameworks to determine the result by searching for "hit stop tracing" in the output. Assign a distinct exit code for failed tests to enable the use of shell expressions and seamless integration with testing frameworks without the need to parse output. Add enum type for return value. Update `make check`. Cc: Daniel Bristot de Oliveira Cc: John Kacur Cc: "Luis Claudio R. Goncalves" Cc: Eder Zulian Cc: Dan Carpenter Cc: Jan Stancek Link: https://lore.kernel.org/20250417185757.2194541-1-costa.shul@redhat.com Signed-off-by: Costa Shulyupin Reviewed-by: Tomas Glozar Signed-off-by: Steven Rostedt (Google)

rtla: Unify apply_config between top and hist

2025-03-26T14:36:39Z

The functions osnoise_top_apply_config and osnoise_hist_apply_config, as well as timerlat_top_apply_config and timerlat_hist_apply_config, are mostly the same. Move common part from them into separate functions osnoise_apply_config and timerlat_apply_config. For rtla-timerlat, also unify params->user_hist and params->user_top into one field called params->user_data, and move several fields used only by timerlat-top into the top-only section of struct timerlat_params. Cc: Luis Goncalves Link: https://lore.kernel.org/20250320092500.101385-3-tglozar@redhat.com Signed-off-by: Tomas Glozar Reviewed-by: John Kacur Signed-off-by: Steven Rostedt (Google)

rtla: Fix segfault in save_trace_to_file call

2025-03-26T14:35:20Z

Running rtla with exit on threshold, but without saving trace leads to a segmenetation fault: $ rtla timerlat hist -T 10 ... Max timerlat IRQ latency from idle: 4.29 us in cpu 0 Segmentation fault This is caused by null pointer deference in the call of save_trace_to_file, which attempts to dereference an uninitialized osnoise_tool variable: save_trace_to_file(record->trace.inst, params->trace_output); ^ this is uninitialized if params->trace_output is not set Fix this by not attempting to dereference "record" if it is NULL and passing NULL instead. As a safety measure, the first field is also checked for NULL inside save_trace_to_file. Cc: John Kacur Cc: Luis Goncalves Cc: Costa Shulyupin Link: https://lore.kernel.org/20250313141034.299117-1-tglozar@redhat.com Fixes: dc4d4e7c72d1 ("rtla: Refactor save_trace_to_file") Signed-off-by: Tomas Glozar Signed-off-by: Steven Rostedt (Google)

rtla: Refactor save_trace_to_file

2025-03-04T19:17:19Z

The functions osnoise_hist_main(), osnoise_top_main(), timerlat_hist_main(), and timerlat_top_main() are lengthy and contain duplicated code. Refactor by consolidating the duplicate lines into the save_trace_to_file() function. Cc: Daniel Bristot de Oliveira Cc: John Kacur Cc: "Luis Claudio R. Goncalves" Cc: Eder Zulian Cc: Dan Carpenter Cc: Gabriele Monaco Link: https://lore.kernel.org/20250219115138.406075-1-costa.shul@redhat.com Signed-off-by: Costa Shulyupin Reviewed-by: Tomas Glozar Tested-by: Tomas Glozar Signed-off-by: Steven Rostedt (Google)

rtla/timerlat_hist: Use BPF to collect samples

2025-03-04T17:35:18Z

Collect samples using BPF program instead of pulling them from tracefs. If the osnoise:timerlat_sample tracepoint is unavailable or the BPF program fails to load for whatever reason, rtla falls back to the old implementation. The collection of samples using the BPF program is fully self-contained and requires no activity of the userspace part of rtla during the measurement. Thus, instead of waking up every second to collect samples, rtla simply sleeps until woken up by a signal or threshold overflow. Cc: John Kacur Cc: Luis Goncalves Cc: Gabriele Monaco Cc: Clark Williams Link: https://lore.kernel.org/20250218145859.27762-6-tglozar@redhat.com Signed-off-by: Tomas Glozar Signed-off-by: Steven Rostedt (Google)

rtla/timerlat: Unify params struct

2025-03-04T17:35:17Z

Instead of having separate structs timerlat_top_params and timerlat_hist_params, use one struct timerlat_params for both. This allows code using the structs to be shared between timerlat-top and timerlat-hist. Cc: John Kacur Cc: Luis Goncalves Cc: Gabriele Monaco Cc: Clark Williams Link: https://lore.kernel.org/20250218145859.27762-2-tglozar@redhat.com Signed-off-by: Tomas Glozar Signed-off-by: Steven Rostedt (Google)