linux/kernel/trace
Steven Rostedt (VMware) 10464b4aa6 ring-buffer: Add rb_time_t 64 bit operations for speeding up 32 bit
After a discussion with the new time algorithm to have nested events still
have proper time keeping but required using local64_t atomic operations.
Mathieu was concerned about the performance this would have on 32 bit
machines, as in most cases, atomic 64 bit operations on them can be
expensive.

As the ring buffer's timing needs do not require full features of local64_t,
a wrapper is made to implement a new rb_time_t operation that uses two longs
on 32 bit machines but still uses the local64_t operations on 64 bit
machines. There's a switch that can be made in the file to force 64 bit to
use the 32 bit version just for testing purposes.

All reads do not need to succeed if a read happened while the stamp being
read is in the process of being updated. The requirement is that all reads
must succed that were done by an interrupting event (where this event was
interrupted by another event that did the write). Or if the event itself did
the write first. That is: rb_time_set(t, x) followed by rb_time_read(t) will
always succeed (even if it gets interrupted by another event that writes to
t. The result of the read will be either the previous set, or a set
performed by an interrupting event.

If the read is done by an event that interrupted another event that was in
the process of setting the time stamp, and no other event came along to
write to that time stamp, it will fail and the rb_time_read() will return
that it failed (the value to read will be undefined).

A set will always write to the time stamp and return with a valid time
stamp, such that any read after it will be valid.

A cmpxchg may fail if it interrupted an event that was in the process of
updating the time stamp just like the reads do. Other than that, it will act
like a normal cmpxchg.

The way this works is that the rb_time_t is made of of three fields. A cnt,
that gets updated atomically everyting a modification is made. A top that
represents the most significant 30 bits of the time, and a bottom to
represent the least significant 30 bits of the time. Notice, that the time
values is only 60 bits long (where the ring buffer only uses 59 bits, which
gives us 18 years of nanoseconds!).

The top two bits of both the top and bottom is a 2 bit counter that gets set
by the value of the least two significant bits of the cnt. A read of the top
and the bottom where both the top and bottom have the same most significant
top 2 bits, are considered a match and a valid 60 bit number can be created
from it. If they do not match, then the number is considered invalid, and
this must only happen if an event interrupted another event in the midst of
updating the time stamp.

This is only used for 32 bits machines as 64 bit machines can get better
performance out of the local64_t. This has been tested heavily by forcing 64
bit to use this logic.

Link: https://lore.kernel.org/r/20200625225345.18cf5881@oasis.local.home
Link: http://lkml.kernel.org/r/20200629025259.309232719@goodmis.org

Inspired-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
2020-06-30 17:18:51 -04:00
..
blktrace.c blktrace: Avoid sparse warnings when assigning q->blk_trace 2020-06-17 09:07:11 -06:00
bpf_trace.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-06-25 18:27:40 -07:00
fgraph.c tracing: Define MCOUNT_INSN_SIZE when not defined without direct calls 2020-01-02 21:56:44 -05:00
ftrace_internal.h x86/ftrace: Have ftrace trampolines turn read-only at the end of system boot up 2020-05-12 18:24:34 -04:00
ftrace.c x86/ftrace: Only have the builtin ftrace_regs_caller call direct hooks 2020-06-29 11:42:47 -04:00
Kconfig Tracing updates for 5.8: 2020-06-09 10:06:18 -07:00
kprobe_event_gen_test.c tracing: Add kprobe event command generation test module 2020-01-30 09:46:28 -05:00
Makefile Rebase locking/kcsan to locking/urgent 2020-06-11 20:02:46 +02:00
power-traces.c
preemptirq_delay_test.c tracing: Wait for preempt irq delay thread to execute 2020-05-11 17:00:34 -04:00
ring_buffer_benchmark.c tracing: Make struct ring_buffer less ambiguous 2020-01-13 13:19:38 -05:00
ring_buffer.c ring-buffer: Add rb_time_t 64 bit operations for speeding up 32 bit 2020-06-30 17:18:51 -04:00
rpm-traces.c
synth_event_gen_test.c tracing: Have synthetic event test use raw_smp_processor_id() 2020-02-20 17:43:41 -05:00
trace_benchmark.c trace: Use pr_warn instead of pr_warning 2019-10-18 15:01:57 +02:00
trace_benchmark.h
trace_boot.c tracing/boottime: Fix kprobe multiple events 2020-06-23 21:51:50 -04:00
trace_branch.c tracing: Make struct ring_buffer less ambiguous 2020-01-13 13:19:38 -05:00
trace_clock.c
trace_dynevent.c tracing: Use seq_buf for building dynevent_cmd string 2020-02-01 13:10:15 -05:00
trace_dynevent.h tracing: Remove check_arg() callbacks from dynevent args 2020-02-01 13:09:23 -05:00
trace_entries.h tracing: Make ftrace packed events have align of 1 2020-06-16 21:21:02 -04:00
trace_event_perf.c Merge branch 'perf/urgent' into perf/core, to pick up fixes 2019-10-28 12:38:26 +01:00
trace_events_filter_test.h
trace_events_filter.c tracing: Avoid memory leak in process_system_preds() 2019-12-19 18:24:17 -05:00
trace_events_hist.c tracing: Move synthetic events to a separate file 2020-06-01 08:23:22 -04:00
trace_events_inject.c tracing: Initialize val to zero in parse_entry of inject code 2020-01-02 19:04:57 -05:00
trace_events_synth.c tracing: Move synthetic events to a separate file 2020-06-01 08:23:22 -04:00
trace_events_trigger.c tracing: Fix event trigger to accept redundant spaces 2020-06-23 21:51:40 -04:00
trace_events.c tracing: Add hist_debug trace event files for histogram debugging 2020-06-01 08:22:30 -04:00
trace_export.c tracing: Make ftrace packed events have align of 1 2020-06-16 21:21:02 -04:00
trace_functions_graph.c ring-buffer: Rename ring_buffer_read() to read_buffer_iter_advance() 2020-03-19 19:11:19 -04:00
trace_functions.c trace: Fix typo in allocate_ftrace_ops()'s comment 2020-06-16 21:21:02 -04:00
trace_hwlat.c tracing: Have hwlat ts be first instance and record count of instances 2020-03-03 17:33:43 -05:00
trace_irqsoff.c tracing: Rename trace_buffer to array_buffer 2020-01-13 13:19:38 -05:00
trace_kdb.c tracing: Rename trace_buffer to array_buffer 2020-01-13 13:19:38 -05:00
trace_kprobe_selftest.c
trace_kprobe_selftest.h
trace_kprobe.c maccess: rename probe_user_{read,write} to copy_{from,to}_user_nofault 2020-06-17 10:57:41 -07:00
trace_mmiotrace.c tracing: Make struct ring_buffer less ambiguous 2020-01-13 13:19:38 -05:00
trace_nop.c
trace_output.c mmap locking API: use coccinelle to convert mmap_sem rwsem call sites 2020-06-09 09:39:14 -07:00
trace_output.h
trace_preemptirq.c x86/entry: Rename trace_hardirqs_off_prepare() 2020-06-11 15:15:24 +02:00
trace_printk.c tracing: Add locked_down checks to the open calls of files created for tracefs 2019-10-12 20:48:06 -04:00
trace_probe_tmpl.h tracing/probe: Support user-space dereference 2019-05-25 23:04:42 -04:00
trace_probe.c tracing/probe: Fix memleak in fetch_op_data operations 2020-06-16 21:21:02 -04:00
trace_probe.h tracing/probe: Replace zero-length array with flexible-array 2020-06-15 23:08:32 -05:00
trace_sched_switch.c tracing: Fix sched switch start/stop refcount racy updates 2020-01-30 09:46:10 -05:00
trace_sched_wakeup.c tracing: Make struct ring_buffer less ambiguous 2020-01-13 13:19:38 -05:00
trace_selftest_dynamic.c
trace_selftest.c tracing: Rename trace_buffer to array_buffer 2020-01-13 13:19:38 -05:00
trace_seq.c tracing: Remove unused TRACE_SEQ_BUF_USED 2020-01-21 18:39:54 -05:00
trace_stack.c trace: fix an incorrect __user annotation on stack_trace_sysctl 2020-06-08 10:13:56 -04:00
trace_stat.c tracing: Fix tracing_stat return values in error handling paths 2020-01-24 18:06:48 -05:00
trace_stat.h tracing: Use generic type for comparator function 2019-11-14 13:15:11 -05:00
trace_synth.h tracing: Move synthetic events to a separate file 2020-06-01 08:23:22 -04:00
trace_syscalls.c Tracing updates: 2020-02-06 07:12:11 +00:00
trace_uprobe.c tracing/probe: Fix bpf_task_fd_query() for kprobes and uprobes 2020-06-09 11:10:12 -07:00
trace.c tracing: Move pipe reference to trace array instead of current_tracer 2020-06-30 14:29:33 -04:00
trace.h tracing: Move pipe reference to trace array instead of current_tracer 2020-06-30 14:29:33 -04:00
tracing_map.c tracing: Convert local functions in tracing_map.c to static 2020-04-22 22:07:26 -04:00
tracing_map.h