linux/kernel/rcu
Paul E. McKenney d119357d07 rcu-tasks: Treat only synchronous grace periods urgently
The performance requirements on RCU Tasks, and in particular on RCU
Tasks Trace, have evolved over time as the workloads have evolved.
The current implementation is designed to provide low grace-period
latencies, and also to accommodate short-duration floods of callbacks.

However, current workloads can also provide a constant background
callback-queuing rate of a few hundred call_rcu_tasks_trace() invocations
per second.  This results in continuous back-to-back RCU Tasks Trace
grace periods, which in turn can consume the better part of 10% of a CPU.
One could take the attitude that there are several tens of other CPUs on
the systems running such workloads, but energy efficiency is a thing.
On these systems, although asynchronous grace-period requests happen
every few milliseconds, synchronous grace-period requests are quite rare.

This commit therefore arrranges for grace periods to be initiated
immediately in response to calls to synchronize_rcu_tasks*() and
also to calls to synchronize_rcu_mult() that are passed one of the
call_rcu_tasks*() functions.  These are recognized by the tell-tale
wakeme_after_rcu callback function.

In other cases, callbacks are gathered up for up to about 250 milliseconds
before a grace period is initiated.  This results in more than an order of
magnitude reduction in RCU Tasks Trace grace periods, with corresponding
reduction in consumption of CPU time.

Reported-by: Alexei Starovoitov <ast@kernel.org>
Reported-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2023-07-14 15:00:11 -07:00
..
Kconfig rcu: Employ jiffies-based backstop to callback time limit 2023-05-11 13:42:39 -07:00
Kconfig.debug rcu: Allow up to five minutes expedited RCU CPU stall-warning timeouts 2023-01-09 12:09:52 -08:00
Makefile rcuperf: Change rcuperf to rcuscale 2020-08-24 18:39:24 -07:00
rcu_segcblist.c rcu: Throttle callback invocation based on number of ready callbacks 2023-01-03 17:28:34 -08:00
rcu_segcblist.h rcu: Throttle callback invocation based on number of ready callbacks 2023-01-03 17:28:34 -08:00
rcu.h rcu-tasks: Stop rcu_tasks_invoke_cbs() from using never-onlined CPUs 2023-05-11 13:42:39 -07:00
rcuscale.c rcu/rcuscale: Stop kfree_scale_thread thread(s) after unloading rcuscale 2023-05-11 13:48:50 -07:00
rcutorture.c Merge branches 'rcu/staging-core', 'rcu/staging-docs' and 'rcu/staging-kfree', remote-tracking branches 'paul/srcu-cf.2023.04.04a', 'fbq/rcu/lockdep.2023.03.27a' and 'fbq/rcu/rcutorture.2023.03.20a' into rcu/staging 2023-04-05 13:50:37 +00:00
refscale.c refscale: Move shutdown from wait_event() to wait_event_idle() 2023-03-20 11:12:16 -07:00
srcutiny.c rcu: Annotate SRCU's update-side lockdep dependencies 2023-03-27 11:15:59 -07:00
srcutree.c Merge branches 'rcu/staging-core', 'rcu/staging-docs' and 'rcu/staging-kfree', remote-tracking branches 'paul/srcu-cf.2023.04.04a', 'fbq/rcu/lockdep.2023.03.27a' and 'fbq/rcu/rcutorture.2023.03.20a' into rcu/staging 2023-04-05 13:50:37 +00:00
sync.c rcu/sync: Use call_rcu_hurry() instead of call_rcu 2022-11-29 14:04:33 -08:00
tasks.h rcu-tasks: Treat only synchronous grace periods urgently 2023-07-14 15:00:11 -07:00
tiny.c rcu: Refactor kvfree_call_rcu() and high-level helpers 2023-01-03 17:48:40 -08:00
tree_exp.h rcu: Mark additional concurrent load from ->cpu_no_qs.b.exp 2023-05-11 13:42:39 -07:00
tree_nocb.h rcu/nocb: Make shrinker iterate only over NOCB CPUs 2023-05-11 13:44:50 -07:00
tree_plugin.h rcu: Mark additional concurrent load from ->cpu_no_qs.b.exp 2023-05-11 13:42:39 -07:00
tree_stall.h rcu: Allow up to five minutes expedited RCU CPU stall-warning timeouts 2023-01-09 12:09:52 -08:00
tree.c Merge branches 'doc.2023.05.10a', 'fixes.2023.05.11a', 'kvfree.2023.05.10a', 'nocb.2023.05.11a', 'rcu-tasks.2023.05.10a', 'torture.2023.05.15a' and 'rcu-urgent.2023.06.06a' into HEAD 2023-06-07 13:44:06 -07:00
tree.h rcu: Add RCU stall diagnosis information 2023-01-05 12:21:11 -08:00
update.c Merge branch 'stall.2023.01.09a' into HEAD 2023-02-02 16:40:07 -08:00