8c98420987
Use BPF to collect statistics on the CPU usage based on perf BPF skeletons. Example usage: # perf kwork top -h Usage: perf kwork top [<options>] -b, --use-bpf Use BPF to measure task cpu usage -C, --cpu <cpu> list of cpus to profile -i, --input <file> input file name -n, --name <name> event name to profile -s, --sort <key[,key2...]> sort by key(s): rate, runtime, tid --time <str> Time span for analysis (start,stop) # # perf kwork -k sched top -b Starting trace, Hit <Ctrl+C> to stop and report ^C Total : 160702.425 ms, 8 cpus %Cpu(s): 36.00% id, 0.00% hi, 0.00% si %Cpu0 [|||||||||||||||||| 61.66%] %Cpu1 [|||||||||||||||||| 61.27%] %Cpu2 [||||||||||||||||||| 66.40%] %Cpu3 [|||||||||||||||||| 61.28%] %Cpu4 [|||||||||||||||||| 61.82%] %Cpu5 [||||||||||||||||||||||| 77.41%] %Cpu6 [|||||||||||||||||| 61.73%] %Cpu7 [|||||||||||||||||| 63.25%] PID SPID %CPU RUNTIME COMMMAND ------------------------------------------------------------- 0 0 38.72 8089.463 ms [swapper/1] 0 0 38.71 8084.547 ms [swapper/3] 0 0 38.33 8007.532 ms [swapper/0] 0 0 38.26 7992.985 ms [swapper/6] 0 0 38.17 7971.865 ms [swapper/4] 0 0 36.74 7447.765 ms [swapper/7] 0 0 33.59 6486.942 ms [swapper/2] 0 0 22.58 3771.268 ms [swapper/5] 9545 9351 2.48 447.136 ms sched-messaging 9574 9351 2.09 418.583 ms sched-messaging 9724 9351 2.05 372.407 ms sched-messaging 9531 9351 2.01 368.804 ms sched-messaging 9512 9351 2.00 362.250 ms sched-messaging 9514 9351 1.95 357.767 ms sched-messaging 9538 9351 1.86 384.476 ms sched-messaging 9712 9351 1.84 386.490 ms sched-messaging 9723 9351 1.83 380.021 ms sched-messaging 9722 9351 1.82 382.738 ms sched-messaging 9517 9351 1.81 354.794 ms sched-messaging 9559 9351 1.79 344.305 ms sched-messaging 9725 9351 1.77 365.315 ms sched-messaging <SNIP> # perf kwork -k sched top -b -n perf Starting trace, Hit <Ctrl+C> to stop and report ^C Total : 151563.332 ms, 8 cpus %Cpu(s): 26.49% id, 0.00% hi, 0.00% si %Cpu0 [ 0.01%] %Cpu1 [ 0.00%] %Cpu2 [ 0.00%] %Cpu3 [ 0.00%] %Cpu4 [ 0.00%] %Cpu5 [ 0.00%] %Cpu6 [ 0.00%] %Cpu7 [ 0.00%] PID SPID %CPU RUNTIME COMMMAND ------------------------------------------------------------- 9754 9754 0.01 2.303 ms perf # # perf kwork -k sched top -b -C 2,3,4 Starting trace, Hit <Ctrl+C> to stop and report ^C Total : 48016.721 ms, 3 cpus %Cpu(s): 27.82% id, 0.00% hi, 0.00% si %Cpu2 [|||||||||||||||||||||| 74.68%] %Cpu3 [||||||||||||||||||||| 71.06%] %Cpu4 [||||||||||||||||||||| 70.91%] PID SPID %CPU RUNTIME COMMMAND ------------------------------------------------------------- 0 0 29.08 4734.998 ms [swapper/4] 0 0 28.93 4710.029 ms [swapper/3] 0 0 25.31 3912.363 ms [swapper/2] 10248 10158 1.62 264.931 ms sched-messaging 10253 10158 1.62 265.136 ms sched-messaging 10158 10158 1.60 263.013 ms bash 10360 10158 1.49 243.639 ms sched-messaging 10413 10158 1.48 238.604 ms sched-messaging 10531 10158 1.47 234.067 ms sched-messaging 10400 10158 1.47 240.631 ms sched-messaging 10355 10158 1.47 230.586 ms sched-messaging 10377 10158 1.43 234.835 ms sched-messaging 10526 10158 1.42 232.045 ms sched-messaging 10298 10158 1.41 222.396 ms sched-messaging 10410 10158 1.38 221.853 ms sched-messaging 10364 10158 1.38 226.042 ms sched-messaging 10480 10158 1.36 213.633 ms sched-messaging 10370 10158 1.36 223.620 ms sched-messaging 10553 10158 1.34 217.169 ms sched-messaging 10291 10158 1.34 211.516 ms sched-messaging 10251 10158 1.34 218.813 ms sched-messaging 10522 10158 1.33 218.498 ms sched-messaging 10288 10158 1.33 216.787 ms sched-messaging <SNIP> Reviewed-by: Ian Rogers <irogers@google.com> Signed-off-by: Yang Jihong <yangjihong1@huawei.com> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Kan Liang <kan.liang@linux.intel.com> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Ravi Bangoria <ravi.bangoria@amd.com> Cc: Sandipan Das <sandipan.das@amd.com> Link: https://lore.kernel.org/r/20230812084917.169338-15-yangjihong1@huawei.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
215 lines
5.8 KiB
Plaintext
215 lines
5.8 KiB
Plaintext
perf-kowrk(1)
|
|
=============
|
|
|
|
NAME
|
|
----
|
|
perf-kwork - Tool to trace/measure kernel work properties (latencies)
|
|
|
|
SYNOPSIS
|
|
--------
|
|
[verse]
|
|
'perf kwork' {record|report|latency|timehist|top}
|
|
|
|
DESCRIPTION
|
|
-----------
|
|
There are several variants of 'perf kwork':
|
|
|
|
'perf kwork record <command>' to record the kernel work
|
|
of an arbitrary workload.
|
|
|
|
'perf kwork report' to report the per kwork runtime.
|
|
|
|
'perf kwork latency' to report the per kwork latencies.
|
|
|
|
'perf kwork timehist' provides an analysis of kernel work events.
|
|
|
|
'perf kwork top' to report the task cpu usage.
|
|
|
|
Example usage:
|
|
perf kwork record -- sleep 1
|
|
perf kwork report
|
|
perf kwork report -b
|
|
perf kwork latency
|
|
perf kwork latency -b
|
|
perf kwork timehist
|
|
perf kwork top
|
|
perf kwork top -b
|
|
|
|
By default it shows the individual work events such as irq, workqeueu,
|
|
including the run time and delay (time between raise and actually entry):
|
|
|
|
Runtime start Runtime end Cpu Kwork name Runtime Delaytime
|
|
(TYPE)NAME:NUM (msec) (msec)
|
|
----------------- ----------------- ------ ------------------------- ---------- ----------
|
|
1811186.976062 1811186.976327 [0000] (s)RCU:9 0.266 0.114
|
|
1811186.978452 1811186.978547 [0000] (s)SCHED:7 0.095 0.171
|
|
1811186.980327 1811186.980490 [0000] (s)SCHED:7 0.162 0.083
|
|
1811186.981221 1811186.981271 [0000] (s)SCHED:7 0.050 0.077
|
|
1811186.984267 1811186.984318 [0000] (s)SCHED:7 0.051 0.075
|
|
1811186.987252 1811186.987315 [0000] (s)SCHED:7 0.063 0.081
|
|
1811186.987785 1811186.987843 [0006] (s)RCU:9 0.058 0.645
|
|
1811186.988319 1811186.988383 [0000] (s)SCHED:7 0.064 0.143
|
|
1811186.989404 1811186.989607 [0002] (s)TIMER:1 0.203 0.111
|
|
1811186.989660 1811186.989732 [0002] (s)SCHED:7 0.072 0.310
|
|
1811186.991295 1811186.991407 [0002] eth0:10 0.112
|
|
1811186.991639 1811186.991734 [0002] (s)NET_RX:3 0.095 0.277
|
|
1811186.989860 1811186.991826 [0002] (w)vmstat_shepherd 1.966 0.345
|
|
...
|
|
|
|
Times are in msec.usec.
|
|
|
|
OPTIONS
|
|
-------
|
|
-D::
|
|
--dump-raw-trace=::
|
|
Display verbose dump of the sched data.
|
|
|
|
-f::
|
|
--force::
|
|
Don't complain, do it.
|
|
|
|
-k::
|
|
--kwork::
|
|
List of kwork to profile (irq, softirq, workqueue, sched, etc)
|
|
|
|
-v::
|
|
--verbose::
|
|
Be more verbose. (show symbol address, etc)
|
|
|
|
OPTIONS for 'perf kwork report'
|
|
----------------------------
|
|
|
|
-b::
|
|
--use-bpf::
|
|
Use BPF to measure kwork runtime
|
|
|
|
-C::
|
|
--cpu::
|
|
Only show events for the given CPU(s) (comma separated list).
|
|
|
|
-i::
|
|
--input::
|
|
Input file name. (default: perf.data unless stdin is a fifo)
|
|
|
|
-n::
|
|
--name::
|
|
Only show events for the given name.
|
|
|
|
-s::
|
|
--sort::
|
|
Sort by key(s): runtime, max, count
|
|
|
|
-S::
|
|
--with-summary::
|
|
Show summary with statistics
|
|
|
|
--time::
|
|
Only analyze samples within given time window: <start>,<stop>. Times
|
|
have the format seconds.microseconds. If start is not given (i.e., time
|
|
string is ',x.y') then analysis starts at the beginning of the file. If
|
|
stop time is not given (i.e, time string is 'x.y,') then analysis goes
|
|
to end of file.
|
|
|
|
OPTIONS for 'perf kwork latency'
|
|
----------------------------
|
|
|
|
-b::
|
|
--use-bpf::
|
|
Use BPF to measure kwork latency
|
|
|
|
-C::
|
|
--cpu::
|
|
Only show events for the given CPU(s) (comma separated list).
|
|
|
|
-i::
|
|
--input::
|
|
Input file name. (default: perf.data unless stdin is a fifo)
|
|
|
|
-n::
|
|
--name::
|
|
Only show events for the given name.
|
|
|
|
-s::
|
|
--sort::
|
|
Sort by key(s): avg, max, count
|
|
|
|
--time::
|
|
Only analyze samples within given time window: <start>,<stop>. Times
|
|
have the format seconds.microseconds. If start is not given (i.e., time
|
|
string is ',x.y') then analysis starts at the beginning of the file. If
|
|
stop time is not given (i.e, time string is 'x.y,') then analysis goes
|
|
to end of file.
|
|
|
|
OPTIONS for 'perf kwork timehist'
|
|
---------------------------------
|
|
|
|
-C::
|
|
--cpu::
|
|
Only show events for the given CPU(s) (comma separated list).
|
|
|
|
-g::
|
|
--call-graph::
|
|
Display call chains if present (default off).
|
|
|
|
-i::
|
|
--input::
|
|
Input file name. (default: perf.data unless stdin is a fifo)
|
|
|
|
-k::
|
|
--vmlinux=<file>::
|
|
Vmlinux pathname
|
|
|
|
-n::
|
|
--name::
|
|
Only show events for the given name.
|
|
|
|
--kallsyms=<file>::
|
|
Kallsyms pathname
|
|
|
|
--max-stack::
|
|
Maximum number of functions to display in backtrace, default 5.
|
|
|
|
--symfs=<directory>::
|
|
Look for files with symbols relative to this directory.
|
|
|
|
--time::
|
|
Only analyze samples within given time window: <start>,<stop>. Times
|
|
have the format seconds.microseconds. If start is not given (i.e., time
|
|
string is ',x.y') then analysis starts at the beginning of the file. If
|
|
stop time is not given (i.e, time string is 'x.y,') then analysis goes
|
|
to end of file.
|
|
|
|
OPTIONS for 'perf kwork top'
|
|
---------------------------------
|
|
|
|
-b::
|
|
--use-bpf::
|
|
Use BPF to measure task cpu usage.
|
|
|
|
-C::
|
|
--cpu::
|
|
Only show events for the given CPU(s) (comma separated list).
|
|
|
|
-i::
|
|
--input::
|
|
Input file name. (default: perf.data unless stdin is a fifo)
|
|
|
|
-n::
|
|
--name::
|
|
Only show events for the given name.
|
|
|
|
-s::
|
|
--sort::
|
|
Sort by key(s): rate, runtime, tid
|
|
|
|
--time::
|
|
Only analyze samples within given time window: <start>,<stop>. Times
|
|
have the format seconds.microseconds. If start is not given (i.e., time
|
|
string is ',x.y') then analysis starts at the beginning of the file. If
|
|
stop time is not given (i.e, time string is 'x.y,') then analysis goes
|
|
to end of file.
|
|
|
|
SEE ALSO
|
|
--------
|
|
linkperf:perf-record[1]
|