IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
This prototype is obtained indirectly, by luck, from some other header
in probe-event.c in most systems, but recently exploded on alpine:edge:
8 13.39 alpine:edge : FAIL gcc version 13.2.1 20240309 (Alpine 13.2.1_git20240309)
util/probe-event.c: In function 'convert_exec_to_group':
util/probe-event.c:225:16: error: implicit declaration of function 'basename' [-Werror=implicit-function-declaration]
225 | ptr1 = basename(exec_copy);
| ^~~~~~~~
util/probe-event.c:225:14: error: assignment to 'char *' from 'int' makes pointer from integer without a cast [-Werror=int-conversion]
225 | ptr1 = basename(exec_copy);
| ^
cc1: all warnings being treated as errors
make[3]: *** [/git/perf-6.8.0/tools/build/Makefile.build:158: util] Error 2
Fix it by adding the libgen.h header where basename() is prototyped.
Fixes: fb7345bbf7 ("perf probe: Support basic dwarf-based operations on uprobe events")
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: https://lore.kernel.org/lkml/
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Building perf with EXTRA_CFLAGS="-fsanitize=address" a leak is detect
when trying to add a probe to a non-existent function:
# perf probe -x ~/bin/perf dso__neW
Probe point 'dso__neW' not found.
Error: Failed to add events.
=================================================================
==296634==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 128 byte(s) in 1 object(s) allocated from:
#0 0x7f67642ba097 in calloc (/lib64/libasan.so.8+0xba097)
#1 0x7f67641a76f1 in allocate_cfi (/lib64/libdw.so.1+0x3f6f1)
Direct leak of 65 byte(s) in 1 object(s) allocated from:
#0 0x7f67642b95b5 in __interceptor_realloc.part.0 (/lib64/libasan.so.8+0xb95b5)
#1 0x6cac75 in strbuf_grow util/strbuf.c:64
#2 0x6ca934 in strbuf_init util/strbuf.c:25
#3 0x9337d2 in synthesize_perf_probe_point util/probe-event.c:2018
#4 0x92be51 in try_to_find_probe_trace_events util/probe-event.c:964
#5 0x93d5c6 in convert_to_probe_trace_events util/probe-event.c:3512
#6 0x93d6d5 in convert_perf_probe_events util/probe-event.c:3529
#7 0x56f37f in perf_add_probe_events /var/home/acme/git/perf-tools-next/tools/perf/builtin-probe.c:354
#8 0x572fbc in __cmd_probe /var/home/acme/git/perf-tools-next/tools/perf/builtin-probe.c:738
#9 0x5730f2 in cmd_probe /var/home/acme/git/perf-tools-next/tools/perf/builtin-probe.c:766
#10 0x635d81 in run_builtin /var/home/acme/git/perf-tools-next/tools/perf/perf.c:323
#11 0x6362c1 in handle_internal_command /var/home/acme/git/perf-tools-next/tools/perf/perf.c:377
#12 0x63667a in run_argv /var/home/acme/git/perf-tools-next/tools/perf/perf.c:421
#13 0x636b8d in main /var/home/acme/git/perf-tools-next/tools/perf/perf.c:537
#14 0x7f676302950f in __libc_start_call_main (/lib64/libc.so.6+0x2950f)
SUMMARY: AddressSanitizer: 193 byte(s) leaked in 2 allocation(s).
#
synthesize_perf_probe_point() returns a "detachec" strbuf, i.e. a
malloc'ed string that needs to be free'd.
An audit will be performed to find other such cases.
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: https://lore.kernel.org/lkml/ZM0l1Oxamr4SVjfY@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since @symbol variable access is not supported by uprobe event, it must be
correctly warn user instead of kernel version update.
Committer testing:
With/without the patch:
[root@quaco ~]# perf probe -x ~/bin/perf -L sigtrap_handler
<sigtrap_handler@/home/acme/git/perf-tools-next/tools/perf/tests/sigtrap.c:0>
0 sigtrap_handler(int signum __maybe_unused, siginfo_t *info, void *ucontext __maybe_unused)
1 {
2 if (!__atomic_fetch_add(&ctx.signal_count, 1, __ATOMIC_RELAXED))
3 ctx.first_siginfo = *info;
4 __atomic_fetch_sub(&ctx.tids_want_signal, syscall(SYS_gettid), __ATOMIC_RELAXED);
5 }
static void *test_thread(void *arg)
{
[root@quaco ~]# perf probe -x ~/bin/perf sigtrap_handler:4 "ctx.signal_count"
Without the patch:
[root@quaco ~]# perf probe -x ~/bin/perf sigtrap_handler:4 "ctx.signal_count"
Failed to write event: Invalid argument
Please upgrade your kernel to at least 3.14 to have access to feature @ctx
Error: Failed to add events.
[root@quaco ~]#
With the patch:
[root@quaco ~]#
Failed to write event: Invalid argument
@ctx accesses a variable by symbol name, but that is not supported for user application probe.
Error: Failed to add events.
[root@quaco ~]#
Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Closes: https://lore.kernel.org/all/ZLWDEjvFjrrEJODp@kernel.org/
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Link: https://lore.kernel.org/r/169055397023.67089.12693645664676964310.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The "address" member of "struct probe_trace_point" uses long data type.
If kernel is 64-bit and perf program is 32-bit, size of "address"
variable is 32 bits.
As a result, upper 32 bits of address read from kernel are truncated, an
error occurs during address comparison in kprobe_warn_out_range().
Before:
# perf probe -a schedule
schedule is out of .text, skip it.
Error: Failed to add events.
Solution:
Change data type of "address" variable to u64 and change corresponding
address printing and value assignment.
After:
# perf.new.new probe -a schedule
Added new event:
probe:schedule (on schedule)
You can now use it in all perf tools, such as:
perf record -e probe:schedule -aR sleep 1
# perf probe -l
probe:schedule (on schedule@kernel/sched/core.c)
# perf record -e probe:schedule -aR sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.156 MB perf.data (1366 samples) ]
# perf report --stdio
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 1K of event 'probe:schedule'
# Event count (approx.): 1366
#
# Overhead Command Shared Object Symbol
# ........ ............... ................. ............
#
6.22% migration/0 [kernel.kallsyms] [k] schedule
6.22% migration/1 [kernel.kallsyms] [k] schedule
6.22% migration/2 [kernel.kallsyms] [k] schedule
6.22% migration/3 [kernel.kallsyms] [k] schedule
6.15% migration/10 [kernel.kallsyms] [k] schedule
6.15% migration/11 [kernel.kallsyms] [k] schedule
6.15% migration/12 [kernel.kallsyms] [k] schedule
6.15% migration/13 [kernel.kallsyms] [k] schedule
6.15% migration/14 [kernel.kallsyms] [k] schedule
6.15% migration/15 [kernel.kallsyms] [k] schedule
6.15% migration/4 [kernel.kallsyms] [k] schedule
6.15% migration/5 [kernel.kallsyms] [k] schedule
6.15% migration/6 [kernel.kallsyms] [k] schedule
6.15% migration/7 [kernel.kallsyms] [k] schedule
6.15% migration/8 [kernel.kallsyms] [k] schedule
6.15% migration/9 [kernel.kallsyms] [k] schedule
0.22% rcu_sched [kernel.kallsyms] [k] schedule
...
#
# (Cannot load tips.txt file, please install perf!)
#
Signed-off-by: Yang Jihong <yangjihong1@huawei.com>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Frank Ch. Eigler <fche@redhat.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jianlin Lv <jianlin.lv@arm.com>
Cc: Jin Yao <yao.jin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Li Huafei <lihuafei1@huawei.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Link: http://lore.kernel.org/lkml/20210715063723.11926-1-yangjihong1@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Now the boot-time tracing supports kprobes events and that must be
written in bootconfig file in the following format.
ftrace.event.kprobes.<EVENT_NAME>.probes = <PROBE-DEF>
'perf probe' already supports --definition (-D) action to show probe
definitions, but the format is for tracefs:
[p|r][:EVENT_NAME] <PROBE-DEF>
This patch adds the --bootconfig option for -D action so that it outputs
the probe definitions in bootconfig format. E.g.
$ perf probe --bootconfig -D "path_lookupat:7 err:s32 s:string"
ftrace.event.kprobes.path_lookupat_L7.probe = 'path_lookupat.isra.0+309 err_s32=%ax:s32 s_string=+0(%r13):string'
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lore.kernel.org/lkml/162282412351.452340.14871995440005640114.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
'perf probe' internally checks the probe target is in the text area in
post-process (after analyzing debuginfo). But it fails if the probe
target is in the "inittext".
This is a good limitation for the online kernel because such functions
have gone after booting. However, for using it for boot-time tracing,
user may want to put a probe on init functions.
This skips the post checking process if the target is offline kenrel so
that user can get the probe definition on the init functions.
Without this patch:
$ perf probe -k ./build-x86_64/vmlinux -D do_mount_root:10
Probe point 'do_mount_root:10' not found.
Error: Failed to add events.
With this patch:
$ perf probe -k ./build-x86_64/vmlinux -D do_mount_root:10
p:probe/do_mount_root_L10 mount_block_root+300
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lore.kernel.org/lkml/162282410293.452340.13347006295826431632.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Report possible permission error including kptr_restrict setting
for map__load() failure. This can happen when non-superuser runs
perf probe.
With this patch, perf probe shows the following message.
$ perf probe vfs_read
Failed to load symbols from /proc/kallsyms
Please ensure you can read the /proc/kallsyms symbol addresses.
If the /proc/sys/kernel/kptr_restrict is '2', you can not read
kernel symbol address even if you are a superuser. Please change
it to '1'. If kptr_restrict is '1', the superuser can read the
symbol addresses.
In that case, please run this command again with sudo.
Error: Failed to add events.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Link: http://lore.kernel.org/lkml/162282065877.448336.10047912688119745151.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When run as normal user with default sysctl kernel.kptr_restrict=0
and kernel.perf_event_paranoid=2, perf probe fails with:
$ ./perf probe move_page_tables
Relocated base symbol is not found!
The warning message is not much informative. The reason perf fails
is because /proc/kallsyms is restricted by perf_event_paranoid=2
for normal user and thus perf fails to read relocated address of
the base symbol.
Tweaking kptr_restrict and perf_event_paranoid can change the
behavior of perf probe. Also, running as root or privileged user
works too. Add these details in the warning message.
Plus, kmap->ref_reloc_sym might not be always set even if
host_machine is initialized. Above is the example of the same.
Remove that comment.
Reported-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lore.kernel.org/lkml/20210525043744.193297-1-ravi.bangoria@linux.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Perf failed to add a kretprobe event with debuginfo of vmlinux which is
compiled by gcc with -fpatchable-function-entry option enabled. The
same issue with kernel module.
Issue:
# perf probe -v 'kernel_clone%return $retval'
......
Writing event: r:probe/kernel_clone__return _text+599624 $retval
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
# cat /sys/kernel/debug/tracing/error_log
[156.75] trace_kprobe: error: Retprobe address must be an function entry
Command: r:probe/kernel_clone__return _text+599624 $retval
^
# llvm-dwarfdump vmlinux |grep -A 10 -w 0x00df2c2b
0x00df2c2b: DW_TAG_subprogram
DW_AT_external (true)
DW_AT_name ("kernel_clone")
DW_AT_decl_file ("/home/code/linux-next/kernel/fork.c")
DW_AT_decl_line (2423)
DW_AT_decl_column (0x07)
DW_AT_prototyped (true)
DW_AT_type (0x00dcd492 "pid_t")
DW_AT_low_pc (0xffff800010092648)
DW_AT_high_pc (0xffff800010092b9c)
DW_AT_frame_base (DW_OP_call_frame_cfa)
# cat /proc/kallsyms |grep kernel_clone
ffff800010092640 T kernel_clone
# readelf -s vmlinux |grep -i kernel_clone
183173: ffff800010092640 1372 FUNC GLOBAL DEFAULT 2 kernel_clone
# objdump -d vmlinux |grep -A 10 -w \<kernel_clone\>:
ffff800010092640 <kernel_clone>:
ffff800010092640: d503201f nop
ffff800010092644: d503201f nop
ffff800010092648: d503233f paciasp
ffff80001009264c: a9b87bfd stp x29, x30, [sp, #-128]!
ffff800010092650: 910003fd mov x29, sp
ffff800010092654: a90153f3 stp x19, x20, [sp, #16]
The entry address of kernel_clone converted by debuginfo is _text+599624
(0x92648), which is consistent with the value of DW_AT_low_pc attribute.
But the symbolic address of kernel_clone from /proc/kallsyms is
ffff800010092640.
This issue is found on arm64, -fpatchable-function-entry=2 is enabled when
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y;
Just as objdump displayed the assembler contents of kernel_clone,
GCC generate 2 NOPs at the beginning of each function.
kprobe_on_func_entry detects that (_text+599624) is not the entry address
of the function, which leads to the failure of adding kretprobe event.
kprobe_on_func_entry
->_kprobe_addr
->kallsyms_lookup_size_offset
->arch_kprobe_on_func_entry // FALSE
The cause of the issue is that the first instruction in the compile unit
indicated by DW_AT_low_pc does not include NOPs.
This issue exists in all gcc versions that support
-fpatchable-function-entry option.
I have reported it to the GCC community:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98776
Currently arm64 and PA-RISC may enable fpatchable-function-entry option.
The kernel compiled with clang does not have this issue.
FIX:
This GCC issue only cause the registration failure of the kretprobe event
which doesn't need debuginfo. So, stop using debuginfo for retprobe.
map will be used to query the probe function address.
Signed-off-by: Jianlin Lv <Jianlin.Lv@arm.com>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: clang-built-linux@googlegroups.com
Cc: Frank Ch. Eigler <fche@redhat.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sumanth Korikkar <sumanthk@linux.ibm.com>
Link: http://lore.kernel.org/lkml/20210210062646.2377995-1-Jianlin.Lv@arm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Passing build_id object to build_id__sprintf function, so it can operate
with the proper size of build id.
This will create proper md5 build id readable names,
like following:
a50e350e97c43b4708d09bcd85ebfff7
instead of:
a50e350e97c43b4708d09bcd85ebfff700000000
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Ian Rogers <irogers@google.com>
Link: https://lore.kernel.org/r/20201013192441.1299447-5-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Replace build_id byte array with struct build_id object and all the code
that references it.
The objective is to carry size together with build id array, so it's
better to keep both together.
This is preparatory change for following patches, and there's no
functional change.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Ian Rogers <irogers@google.com>
Link: https://lore.kernel.org/r/20201013192441.1299447-2-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since 'perf probe' heavily depends on debuginfo, debuginfod gives us
many benefits on the 'perf probe' command on remote machine.
Especially, this will be helpful for the embedded devices which will not
have enough storage, or boot with a cross-build kernel whose source code
is in the host machine.
This will work as similar to commit c7a14fdcb3 ("perf build-ids:
Fall back to debuginfod query if debuginfo not found")
Tested with:
(host) $ cd PATH/TO/KBUILD/DIR/
(host) $ debuginfod -F .
...
(remote) # perf probe -L vfs_read
Failed to find the path for the kernel: No such file or directory
Error: Failed to show lines.
(remote) # export DEBUGINFOD_URLS="http://$HOST_IP:8002/"
(remote) # perf probe -L vfs_read
<vfs_read@...>
0 ssize_t vfs_read(struct file *file, char __user *buf, size_t count, loff_t *pos)
{
2 ssize_t ret;
if (!(file->f_mode & FMODE_READ))
return -EBADF;
6 if (!(file->f_mode & FMODE_CAN_READ))
return -EINVAL;
8 if (unlikely(!access_ok(buf, count)))
return -EFAULT;
11 ret = rw_verify_area(READ, file, pos, count);
12 if (ret)
return ret;
if (count > MAX_RW_COUNT)
...
(remote) # perf probe -a "vfs_read count"
Added new event:
probe:vfs_read (on vfs_read with count)
(remote) # perf probe -l
probe:vfs_read (on vfs_read@ksrc/linux/fs/read_write.c with count)
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Reviewed-by: Frank Ch. Eigler <fche@redhat.com>
Cc: Aaron Merey <amerey@redhat.com>
Cc: Daniel Thompson <daniel.thompson@linaro.org>
Link: http://lore.kernel.org/lkml/160041610083.912668.13659563860278615846.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
'perf probe' uses ref_reloc_sym to adjust symbol offset address from
debuginfo address or ref_reloc_sym based address, but that is misusing
reloc_sym->addr and reloc_sym->unrelocated_addr. If map is not
relocated (map->reloc == 0), we can use reloc_sym->addr as unrelocated
address instead of reloc_sym->unrelocated_addr.
This usually does not happen. If we have a non-stripped ELF binary, we
will use it for map and debuginfo, if not, we use only kallsyms without
debuginfo. Thus, the map is always relocated (ELF and DWARF binary) or
not relocated (kallsyms).
However, if we allow the combination of debuginfo and kallsyms based map
(like using debuginfod), we have to check the map->reloc and choose the
collect address of reloc_sym.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Reviewed-by: Frank Ch. Eigler <fche@redhat.com>
Cc: Aaron Merey <amerey@redhat.com>
Cc: Daniel Thompson <daniel.thompson@linaro.org>
Link: http://lore.kernel.org/lkml/160041609047.912668.14314639291419159274.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Warn if the probe target function is a GNU indirect function (GNU_IFUNC)
because it may not be what the user wants to probe.
The GNU indirect function ( https://sourceware.org/glibc/wiki/GNU_IFUNC )
is the dynamic symbol solved at runtime. An IFUNC function is a selector
which is invoked from the ELF loader, but the symbol address of the
function which will be modified by the IFUNC is the same as the IFUNC in
the symbol table. This can confuse users trying to probe such functions.
For example, memcpy is an IFUNC.
probe_libc:memcpy (on __new_memcpy_ifunc@x86_64/multiarch/memcpy.c in /usr/lib64/libc-2.30.so)
the probe is put on an IFUNC.
perf 1742 [000] 26201.715632: probe_libc:memcpy: (7fdaa53824c0)
7fdaa53824c0 __new_memcpy_ifunc+0x0 (inlined)
7fdaa5d4a980 elf_machine_rela+0x6c0 (inlined)
7fdaa5d4a980 elf_dynamic_do_Rela+0x6c0 (inlined)
7fdaa5d4a980 _dl_relocate_object+0x6c0 (/usr/lib64/ld-2.30.so)
7fdaa5d42155 dl_main+0x1cc5 (/usr/lib64/ld-2.30.so)
7fdaa5d5831a _dl_sysdep_start+0x54a (/usr/lib64/ld-2.30.so)
7fdaa5d3ffeb _dl_start_final+0x25b (inlined)
7fdaa5d3ffeb _dl_start+0x25b (/usr/lib64/ld-2.30.so)
7fdaa5d3f117 .annobin_rtld.c+0x7 (inlined)
And the event is invoked from the ELF loader instead of the target
program's main code.
Moreover, at this moment, we can not probe on the function which will
be selected by the IFUNC, because it is determined at runtime. But
uprobe will be prepared before running the target binary.
Thus, I decided to warn user when 'perf probe' detects that the probe
point is on an GNU IFUNC symbol. Someone who wants to probe an IFUNC
symbol to debug the IFUNC function can ignore this warning.
Committer notes:
I.e., this warning will be emitted if the probe point is an IFUNC:
"Warning: The probe function (%s) is a GNU indirect function.\n"
"Consider identifying the final function used at run time and set the probe directly on that.\n"
Complete set of steps:
# readelf -sW /lib64/libc-2.29.so | grep IFUNC | tail
22196: 0000000000109a80 183 IFUNC GLOBAL DEFAULT 14 __memcpy_chk
22214: 00000000000b7d90 191 IFUNC GLOBAL DEFAULT 14 __gettimeofday
22336: 000000000008b690 60 IFUNC GLOBAL DEFAULT 14 memchr
22350: 000000000008b9b0 89 IFUNC GLOBAL DEFAULT 14 __stpcpy
22420: 000000000008bb10 76 IFUNC GLOBAL DEFAULT 14 __strcasecmp_l
22582: 000000000008a970 60 IFUNC GLOBAL DEFAULT 14 strlen
22585: 00000000000a54d0 92 IFUNC WEAK DEFAULT 14 wmemset
22600: 000000000010b030 92 IFUNC GLOBAL DEFAULT 14 __wmemset_chk
22618: 000000000008b8a0 183 IFUNC GLOBAL DEFAULT 14 __mempcpy
22675: 000000000008ba70 76 IFUNC WEAK DEFAULT 14 strcasecmp
#
# perf probe -x /lib64/libc-2.29.so strlen
Warning: The probe function (strlen) is a GNU indirect function.
Consider identifying the final function used at run time and set the probe directly on that.
Added new event:
probe_libc:strlen (on strlen in /usr/lib64/libc-2.29.so)
You can now use it in all perf tools, such as:
perf record -e probe_libc:strlen -aR sleep 1
#
Reported-by: Andi Kleen <andi@firstfloor.org>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Link: http://lore.kernel.org/lkml/159438669349.62703.5978345670436126948.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
There is a case that several same-name symbols points to the same
address. In that case, 'perf probe' returns an error.
E.g.
# perf probe -x /lib64/libc-2.30.so -v -a "memcpy arg1=%di"
probe-definition(0): memcpy arg1=%di
symbol:memcpy file:(null) line:0 offset:0 return:0 lazy:(null)
parsing arg: arg1=%di into name:arg1 %di
1 arguments
symbol:setjmp file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:longjmp file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:longjmp_target file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:lll_lock_wait_private file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_arena_max file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_arena_test file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_tunable_tcache_max_bytes file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_tunable_tcache_count file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_tunable_tcache_unsorted_limit file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_trim_threshold file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_top_pad file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_mmap_threshold file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_mmap_max file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_perturb file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_mxfast file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_heap_new file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_arena_reuse_free_list file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_arena_reuse file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_arena_reuse_wait file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_arena_new file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_arena_retry file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_sbrk_less file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_heap_free file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_heap_less file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_tcache_double_free file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_heap_more file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_sbrk_more file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_malloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_memalign_retry file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_free_dyn_thresholds file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_realloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_calloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt file:(null) line:0 offset:0 return:0 lazy:(null)
Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.30.so.debug
Try to find probe point from debuginfo.
Opening /sys/kernel/debug/tracing//README write=0
Failed to find the location of the '%di' variable at this address.
Perhaps it has been optimized out.
Use -V with the --range option to show '%di' location range.
An error occurred in debuginfo analysis (-2).
Trying to use symbols.
Opening /sys/kernel/debug/tracing//uprobe_events write=1
Writing event: p:probe_libc/memcpy /usr/lib64/libc-2.30.so:0x914c0 arg1=%di
Writing event: p:probe_libc/memcpy /usr/lib64/libc-2.30.so:0x914c0 arg1=%di
Failed to write event: File exists
Error: Failed to add events. Reason: File exists (Code: -17)
You can see that perf tried to write completely the same probe
definition twice, which caused an error.
To fix this issue, check the symbol list and drop duplicated symbols
(which has the same symbol name and address) from it.
With this patch:
# perf probe -x /lib64/libc-2.30.so -a "memcpy arg1=%di"
Failed to find the location of the '%di' variable at this address.
Perhaps it has been optimized out.
Use -V with the --range option to show '%di' location range.
Added new events:
probe_libc:memcpy (on memcpy in /usr/lib64/libc-2.30.so with arg1=%di)
probe_libc:memcpy (on memcpy in /usr/lib64/libc-2.30.so with arg1=%di)
You can now use it in all perf tools, such as:
perf record -e probe_libc:memcpy -aR sleep 1
Committer notes:
Fix this build error on 32-bit arches by using PRIx64 for symbol->start,
that is an u64:
In file included from util/probe-event.c:27:
util/probe-event.c: In function 'find_probe_trace_events_from_map':
util/probe-event.c:2978:14: error: format '%lx' expects argument of type 'long unsigned int', but argument 5 has type 'u64' {aka 'long long unsigned int'} [-Werror=format=]
pr_debug("Found duplicated symbol %s @ %lx\n",
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
util/debug.h:17:21: note: in definition of macro 'pr_fmt'
#define pr_fmt(fmt) fmt
^~~
util/probe-event.c:2978:5: note: in expansion of macro 'pr_debug'
pr_debug("Found duplicated symbol %s @ %lx\n",
^~~~~~~~
Reported-by: Andi Kleen <andi@firstfloor.org>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Reviewed-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Link: http://lore.kernel.org/lkml/159438666401.62703.15196394835032087840.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since commit 03db8b583d ("perf tools: Fix
maps__find_symbol_by_name()") introduced map address range check in
maps__find_symbol_by_name(), we can not get "_etext" from kernel map
because _etext is placed on the edge of the kernel .text section (=
kernel map in perf.)
To fix this issue, this checks the address correctness by map address
range information (map->start and map->end) instead of using _etext
address.
This can cause an error if the target inlined function is embedded in
both __init function and normal function.
For exaample, request_resource() is a normal function but also embedded
in __init reserve_setup(). In this case, the probe point in
reserve_setup() must be skipped.
However, without this fix, it failes to setup all probe points:
# ./perf probe -v request_resource
probe-definition(0): request_resource
symbol:request_resource file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (8 entries long)
Using /usr/lib/debug/lib/modules/5.5.17-200.fc31.x86_64/vmlinux for symbols
Open Debuginfo file: /usr/lib/debug/lib/modules/5.5.17-200.fc31.x86_64/vmlinux
Try to find probe point from debuginfo.
Matched function: request_resource [15e29ad]
found inline addr: 0xffffffff82fbf892
Probe point found: reserve_setup+204
found inline addr: 0xffffffff810e9790
Probe point found: request_resource+0
Found 2 probe_trace_events.
Opening /sys/kernel/debug/tracing//kprobe_events write=1
Opening /sys/kernel/debug/tracing//README write=0
Writing event: p:probe/request_resource _text+33290386
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
#
With this fix,
# ./perf probe request_resource
reserve_setup is out of .text, skip it.
Added new events:
(null):(null) (on request_resource)
probe:request_resource (on request_resource)
You can now use it in all perf tools, such as:
perf record -e probe:request_resource -aR sleep 1
#
Fixes: 03db8b583d ("perf tools: Fix maps__find_symbol_by_name()")
Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: stable@vger.kernel.org
Link: http://lore.kernel.org/lkml/158763967332.30755.4922496724365529088.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix to check kprobe blacklist address correctly with relocated address
by adjusting debuginfo address.
Since the address in the debuginfo is same as objdump, it is different
from relocated kernel address with KASLR. Thus, 'perf probe' always
misses to catch the blacklisted addresses.
Without this patch, 'perf probe' can not detect the blacklist addresses
on a KASLR enabled kernel.
# perf probe kprobe_dispatcher
Failed to write event: Invalid argument
Error: Failed to add events.
#
With this patch, it correctly shows the error message.
# perf probe kprobe_dispatcher
kprobe_dispatcher is blacklisted function, skip it.
Probe point 'kprobe_dispatcher' not found.
Error: Failed to add events.
#
Fixes: 9aaf5a5f47 ("perf probe: Check kprobes blacklist when adding new events")
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: stable@vger.kernel.org
Link: http://lore.kernel.org/lkml/158763966411.30755.5882376357738273695.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since the commit 6a13a0d7b4 ("ftrace/kprobe: Show the maxactive number
on kprobe_events") introduced to show the instance number of kretprobe
events, the length of the 1st format of the kprobe event will not 1, but
it can be longer. This caused a parser error in perf-probe.
Skip the length check the 1st format of the kprobe event to accept this
instance number.
Without this fix:
# perf probe -a vfs_read%return
Added new event:
probe:vfs_read__return (on vfs_read%return)
You can now use it in all perf tools, such as:
perf record -e probe:vfs_read__return -aR sleep 1
# perf probe -l
Semantic error :Failed to parse event name: r16:probe/vfs_read__return
Error: Failed to show event list.
And with this fixes:
# perf probe -a vfs_read%return
...
# perf probe -l
probe:vfs_read__return (on vfs_read%return)
Fixes: 6a13a0d7b4 ("ftrace/kprobe: Show the maxactive number on kprobe_events")
Reported-by: Yuxuan Shui <yshuiv7@gmail.com>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Yuxuan Shui <yshuiv7@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: stable@vger.kernel.org
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=207587
Link: http://lore.kernel.org/lkml/158877535215.26469.1113127926699134067.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
And pick the shortest name: 'struct maps'.
The split existed because we used to have two groups of maps, one for
functions and one for variables, but that only complicated things,
sometimes we needed to figure out what was at some address and then had
to first try it on the functions group and if that failed, fall back to
the variables one.
That split is long gone, so for quite a while we had only one struct
maps per struct map_groups, simplify things by combining those structs.
First patch is the minimum needed to merge both, follow up patches will
rename 'thread->mg' to 'thread->maps', etc.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: https://lkml.kernel.org/n/tip-hom6639ro7020o708trhxh59@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Trace a magic number as immediate value if the target variable is not
found at some probe points which is based on one probe event.
This feature is good for the case if you trace a source code line with
some local variables, which is compiled into several instructions and
some of the variables are optimized out on some instructions.
Even if so, with this feature, perf probe trace a magic number instead
of such disappeared variables and fold those probes on one event.
E.g. without this patch:
# perf probe -D "pud_page_vaddr pud"
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
Failed to find 'pud' in this function.
p:probe/pud_page_vaddr _text+23480787 pud=%ax:x64
p:probe/pud_page_vaddr _text+23808453 pud=%bp:x64
p:probe/pud_page_vaddr _text+23558082 pud=%ax:x64
p:probe/pud_page_vaddr _text+328373 pud=%r8:x64
p:probe/pud_page_vaddr _text+348448 pud=%bx:x64
p:probe/pud_page_vaddr _text+23816818 pud=%bx:x64
With this patch:
# perf probe -D "pud_page_vaddr pud" | head
spurious_kernel_fault is blacklisted function, skip it.
vmalloc_fault is blacklisted function, skip it.
p:probe/pud_page_vaddr _text+23480787 pud=%ax:x64
p:probe/pud_page_vaddr _text+149051 pud=\deade12d:x64
p:probe/pud_page_vaddr _text+23808453 pud=%bp:x64
p:probe/pud_page_vaddr _text+315926 pud=\deade12d:x64
p:probe/pud_page_vaddr _text+23807209 pud=\deade12d:x64
p:probe/pud_page_vaddr _text+23557365 pud=%ax:x64
p:probe/pud_page_vaddr _text+314097 pud=%di:x64
p:probe/pud_page_vaddr _text+314015 pud=\deade12d:x64
p:probe/pud_page_vaddr _text+313893 pud=\deade12d:x64
p:probe/pud_page_vaddr _text+324083 pud=\deade12d:x64
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Cc: Tom Zanussi <tom.zanussi@linux.intel.com>
Link: http://lore.kernel.org/lkml/157406476931.24476.6261475888681844285.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Support multiprobe event if the event is based on function and lines and
kernel supports it. In this case, perf probe creates the first probe
with an event, and tries to append following probes on that event, since
those probes must be on the same source code line.
Before this patch;
# perf probe -a vfs_read:18
Added new events:
probe:vfs_read_L18 (on vfs_read:18)
probe:vfs_read_L18_1 (on vfs_read:18)
You can now use it in all perf tools, such as:
perf record -e probe:vfs_read_L18_1 -aR sleep 1
#
After this patch (on multiprobe supported kernel)
# perf probe -a vfs_read:18
Added new events:
probe:vfs_read_L18 (on vfs_read:18)
probe:vfs_read_L18 (on vfs_read:18)
You can now use it in all perf tools, such as:
perf record -e probe:vfs_read_L18 -aR sleep 1
#
Committer testing:
On a kernel that doesn't support multiprobe events, after this patch:
# uname -a
Linux quaco 5.3.8-200.fc30.x86_64 #1 SMP Tue Oct 29 14:46:22 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# grep append /sys/kernel/debug/tracing/README
be modified by appending '.descending' or '.ascending' to a
can be modified by appending any of the following modifiers
#
# perf probe -a vfs_read:18
Added new events:
probe:vfs_read_L18 (on vfs_read:18)
probe:vfs_read_L18_1 (on vfs_read:18)
You can now use it in all perf tools, such as:
perf record -e probe:vfs_read_L18_1 -aR sleep 1
# perf probe -l
probe:vfs_read_L18 (on vfs_read:18@fs/read_write.c)
probe:vfs_read_L18_1 (on vfs_read:18@fs/read_write.c)
#
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Cc: Tom Zanussi <tom.zanussi@linux.intel.com>
Link: http://lore.kernel.org/lkml/157406475010.24476.586290752591512351.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
