iv/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
b66c079aab
linux/kernel/scftorture.c

667 lines
20 KiB
C
Raw Normal View History

scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
// SPDX-License-Identifier: GPL-2.0+
//
// Torture test for smp_call_function() and friends.
//
// Copyright (C) Facebook, 2020.
//
// Author: Paul E. McKenney <paulmck@kernel.org>
#define pr_fmt(fmt) fmt
#include <linux/atomic.h>
#include <linux/bitops.h>
#include <linux/completion.h>
#include <linux/cpu.h>
#include <linux/delay.h>
#include <linux/err.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/kthread.h>
#include <linux/kernel.h>
#include <linux/mm.h>
#include <linux/module.h>
#include <linux/moduleparam.h>
#include <linux/notifier.h>
#include <linux/percpu.h>
#include <linux/rcupdate.h>
#include <linux/rcupdate_trace.h>
#include <linux/reboot.h>
#include <linux/sched.h>
#include <linux/spinlock.h>
#include <linux/smp.h>
#include <linux/stat.h>
#include <linux/srcu.h>
#include <linux/slab.h>
#include <linux/torture.h>
#include <linux/types.h>
#define SCFTORT_STRING "scftorture"
#define SCFTORT_FLAG SCFTORT_STRING ": "
#define VERBOSE_SCFTORTOUT(s, x...) \
scftorture: Add missing '\n' to flush message Add '\n' to macros to flush message for each call. Acked-by: Davidlohr Bueso <dbueso@suse.de> Signed-off-by: Li Zhijian <zhijianx.li@intel.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-10-29 12:40:25 +03:00
do { if (verbose) pr_alert(SCFTORT_FLAG s "\n", ## x); } while (0)
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
scftorture: Always log error message Unconditionally log messages corresponding to errors. Acked-by: Davidlohr Bueso <dbueso@suse.de> Signed-off-by: Li Zhijian <zhijianx.li@intel.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-11-03 11:30:27 +03:00
#define SCFTORTOUT_ERRSTRING(s, x...) pr_alert(SCFTORT_FLAG "!!! " s "\n", ## x)
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Paul E. McKenney <paulmck@kernel.org>");
// Wait until there are multiple CPUs before starting test.
torture_param(int, holdoff, IS_BUILTIN(CONFIG_SCF_TORTURE_TEST) ? 10 : 0,
"Holdoff time before test start (s)");
torture_param(int, longwait, 0, "Include ridiculously long waits? (seconds)");
torture_param(int, nthreads, -1, "# threads, defaults to -1 for all CPUs.");
torture_param(int, onoff_holdoff, 0, "Time after boot before CPU hotplugs (s)");
torture_param(int, onoff_interval, 0, "Time between CPU hotplugs (s), 0=disable");
torture_param(int, shutdown_secs, 0, "Shutdown time (ms), <= zero to disable.");
torture_param(int, stat_interval, 60, "Number of seconds between stats printk()s.");
scftorture: Add full-test stutter capability In virtual environments on systems with hardware assist, inter-processor interrupts must do very different things based on whether the target vCPU is running or not. This commit therefore enables torture-test stuttering to better test these running/not-running transitions. Suggested-by: Chris Mason <clm@fb.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-24 22:11:57 +03:00
torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable");
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
torture_param(bool, use_cpus_read_lock, 0, "Use cpus_read_lock() to exclude CPU hotplug.");
torture_param(int, verbose, 0, "Enable verbose debugging printk()s");
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
torture_param(int, weight_resched, -1, "Testing weight for resched_cpu() operations.");
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
torture_param(int, weight_single, -1, "Testing weight for single-CPU no-wait operations.");
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
torture_param(int, weight_single_rpc, -1, "Testing weight for single-CPU RPC operations.");
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
torture_param(int, weight_single_wait, -1, "Testing weight for single-CPU operations.");
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
torture_param(int, weight_many, -1, "Testing weight for multi-CPU no-wait operations.");
torture_param(int, weight_many_wait, -1, "Testing weight for multi-CPU operations.");
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
torture_param(int, weight_all, -1, "Testing weight for all-CPU no-wait operations.");
torture_param(int, weight_all_wait, -1, "Testing weight for all-CPU operations.");
char *torture_type = "";
#ifdef MODULE
# define SCFTORT_SHUTDOWN 0
#else
# define SCFTORT_SHUTDOWN 1
#endif
torture_param(bool, shutdown, SCFTORT_SHUTDOWN, "Shutdown at end of torture test.");
struct scf_statistics {
struct task_struct *task;
int cpu;
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
long long n_resched;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
long long n_single;
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
long long n_single_ofl;
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
long long n_single_rpc;
long long n_single_rpc_ofl;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
long long n_single_wait;
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
long long n_single_wait_ofl;
long long n_many;
long long n_many_wait;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
long long n_all;
long long n_all_wait;
};
static struct scf_statistics *scf_stats_p;
static struct task_struct *scf_torture_stats_task;
static DEFINE_PER_CPU(long long, scf_invoked_count);
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
// Data for random primitive selection
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
#define SCF_PRIM_RESCHED 0
#define SCF_PRIM_SINGLE 1
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
#define SCF_PRIM_SINGLE_RPC 2
#define SCF_PRIM_MANY 3
#define SCF_PRIM_ALL 4
#define SCF_NPRIMS 8 // Need wait and no-wait versions of each,
// except for SCF_PRIM_RESCHED and
// SCF_PRIM_SINGLE_RPC.
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
static char *scf_prim_name[] = {
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
"resched_cpu",
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
"smp_call_function_single",
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
"smp_call_function_single_rpc",
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
"smp_call_function_many",
"smp_call_function",
};
struct scf_selector {
unsigned long scfs_weight;
int scfs_prim;
bool scfs_wait;
};
static struct scf_selector scf_sel_array[SCF_NPRIMS];
static int scf_sel_array_len;
static unsigned long scf_sel_totweight;
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
// Communicate between caller and handler.
struct scf_check {
bool scfc_in;
bool scfc_out;
int scfc_cpu; // -1 for not _single().
bool scfc_wait;
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
bool scfc_rpc;
struct completion scfc_completion;
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
};
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
// Use to wait for all threads to start.
static atomic_t n_started;
static atomic_t n_errs;
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
static atomic_t n_mb_in_errs;
static atomic_t n_mb_out_errs;
static atomic_t n_alloc_errs;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
static bool scfdone;
scftorture: Flag errors in torture-compatible manner This commit prints error counts on the statistics line and also adds a "!!!" if any of the counters are non-zero. Allocation failures are (somewhat) forgiven, but all other errors result in a "FAILURE" print at the end of the test. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 02:06:22 +03:00
static char *bangstr = "";
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
scftorture: Make symbol 'scf_torture_rand' static The sparse tool complains as follows kernel/scftorture.c:124:1: warning: symbol '__pcpu_scope_scf_torture_rand' was not declared. Should it be static? And this per-CPU variable is not used outside of scftorture.c, so this commit marks it static. Reported-by: Hulk Robot <hulkci@huawei.com> Signed-off-by: Wei Yongjun <weiyongjun1@huawei.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 19:56:50 +03:00
static DEFINE_TORTURE_RANDOM_PERCPU(scf_torture_rand);
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
extern void resched_cpu(int cpu); // An alternative IPI vector.
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
// Print torture statistics. Caller must ensure serialization.
static void scf_torture_stats_print(void)
{
int cpu;
scftorture: Summarize per-thread statistics This commit summarizes the per-thread statistics, providing counts of the number of single, many, and all calls, both no-wait and wait, and, for the single case, the number where the target CPU was offline. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 02:13:37 +03:00
int i;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
long long invoked_count = 0;
bool isdone = READ_ONCE(scfdone);
scftorture: Summarize per-thread statistics This commit summarizes the per-thread statistics, providing counts of the number of single, many, and all calls, both no-wait and wait, and, for the single case, the number where the target CPU was offline. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 02:13:37 +03:00
struct scf_statistics scfs = {};
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
for_each_possible_cpu(cpu)
invoked_count += data_race(per_cpu(scf_invoked_count, cpu));
scftorture: Summarize per-thread statistics This commit summarizes the per-thread statistics, providing counts of the number of single, many, and all calls, both no-wait and wait, and, for the single case, the number where the target CPU was offline. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 02:13:37 +03:00
for (i = 0; i < nthreads; i++) {
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
scfs.n_resched += scf_stats_p[i].n_resched;
scftorture: Summarize per-thread statistics This commit summarizes the per-thread statistics, providing counts of the number of single, many, and all calls, both no-wait and wait, and, for the single case, the number where the target CPU was offline. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 02:13:37 +03:00
scfs.n_single += scf_stats_p[i].n_single;
scfs.n_single_ofl += scf_stats_p[i].n_single_ofl;
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
scfs.n_single_rpc += scf_stats_p[i].n_single_rpc;
scftorture: Summarize per-thread statistics This commit summarizes the per-thread statistics, providing counts of the number of single, many, and all calls, both no-wait and wait, and, for the single case, the number where the target CPU was offline. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 02:13:37 +03:00
scfs.n_single_wait += scf_stats_p[i].n_single_wait;
scfs.n_single_wait_ofl += scf_stats_p[i].n_single_wait_ofl;
scfs.n_many += scf_stats_p[i].n_many;
scfs.n_many_wait += scf_stats_p[i].n_many_wait;
scfs.n_all += scf_stats_p[i].n_all;
scfs.n_all_wait += scf_stats_p[i].n_all_wait;
}
scftorture: Flag errors in torture-compatible manner This commit prints error counts on the statistics line and also adds a "!!!" if any of the counters are non-zero. Allocation failures are (somewhat) forgiven, but all other errors result in a "FAILURE" print at the end of the test. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 02:06:22 +03:00
if (atomic_read(&n_errs) || atomic_read(&n_mb_in_errs) ||
scftorture: Forgive memory-allocation failure if KASAN Kernels built with CONFIG_KASAN=y quarantine newly freed memory in order to better detect use-after-free errors. However, this can exhaust memory more quickly in allocator-heavy tests, which can result in spurious scftorture failure. This commit therefore forgives memory-allocation failure in kernels built with CONFIG_KASAN=y, but continues counting the errors for use in detailed test-result analyses. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2023-05-16 05:00:10 +03:00
atomic_read(&n_mb_out_errs) ||
(!IS_ENABLED(CONFIG_KASAN) && atomic_read(&n_alloc_errs)))
scftorture: Flag errors in torture-compatible manner This commit prints error counts on the statistics line and also adds a "!!!" if any of the counters are non-zero. Allocation failures are (somewhat) forgiven, but all other errors result in a "FAILURE" print at the end of the test. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 02:06:22 +03:00
bangstr = "!!! ";
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
pr_alert("%s %sscf_invoked_count %s: %lld resched: %lld single: %lld/%lld single_ofl: %lld/%lld single_rpc: %lld single_rpc_ofl: %lld many: %lld/%lld all: %lld/%lld ",
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
SCFTORT_FLAG, bangstr, isdone ? "VER" : "ver", invoked_count, scfs.n_resched,
scftorture: Summarize per-thread statistics This commit summarizes the per-thread statistics, providing counts of the number of single, many, and all calls, both no-wait and wait, and, for the single case, the number where the target CPU was offline. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 02:13:37 +03:00
scfs.n_single, scfs.n_single_wait, scfs.n_single_ofl, scfs.n_single_wait_ofl,
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
scfs.n_single_rpc, scfs.n_single_rpc_ofl,
scftorture: Summarize per-thread statistics This commit summarizes the per-thread statistics, providing counts of the number of single, many, and all calls, both no-wait and wait, and, for the single case, the number where the target CPU was offline. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 02:13:37 +03:00
scfs.n_many, scfs.n_many_wait, scfs.n_all, scfs.n_all_wait);
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
torture_onoff_stats();
scftorture: Flag errors in torture-compatible manner This commit prints error counts on the statistics line and also adds a "!!!" if any of the counters are non-zero. Allocation failures are (somewhat) forgiven, but all other errors result in a "FAILURE" print at the end of the test. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 02:06:22 +03:00
pr_cont("ste: %d stnmie: %d stnmoe: %d staf: %d\n", atomic_read(&n_errs),
atomic_read(&n_mb_in_errs), atomic_read(&n_mb_out_errs),
atomic_read(&n_alloc_errs));
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
}
// Periodically prints torture statistics, if periodic statistics printing
// was specified via the stat_interval module parameter.
static int
scf_torture_stats(void *arg)
{
VERBOSE_TOROUT_STRING("scf_torture_stats task started");
do {
schedule_timeout_interruptible(stat_interval * HZ);
scf_torture_stats_print();
torture_shutdown_absorb("scf_torture_stats");
} while (!torture_must_stop());
torture_kthread_stopping("scf_torture_stats");
return 0;
}
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
// Add a primitive to the scf_sel_array[].
static void scf_sel_add(unsigned long weight, int prim, bool wait)
{
struct scf_selector *scfsp = &scf_sel_array[scf_sel_array_len];
// If no weight, if array would overflow, if computing three-place
// percentages would overflow, or if the scf_prim_name[] array would
// overflow, don't bother. In the last three two cases, complain.
if (!weight ||
WARN_ON_ONCE(scf_sel_array_len >= ARRAY_SIZE(scf_sel_array)) ||
WARN_ON_ONCE(0 - 100000 * weight <= 100000 * scf_sel_totweight) ||
WARN_ON_ONCE(prim >= ARRAY_SIZE(scf_prim_name)))
return;
scf_sel_totweight += weight;
scfsp->scfs_weight = scf_sel_totweight;
scfsp->scfs_prim = prim;
scfsp->scfs_wait = wait;
scf_sel_array_len++;
}
// Dump out weighting percentages for scf_prim_name[] array.
static void scf_sel_dump(void)
{
int i;
unsigned long oldw = 0;
struct scf_selector *scfsp;
unsigned long w;
for (i = 0; i < scf_sel_array_len; i++) {
scfsp = &scf_sel_array[i];
w = (scfsp->scfs_weight - oldw) * 100000 / scf_sel_totweight;
pr_info("%s: %3lu.%03lu %s(%s)\n", __func__, w / 1000, w % 1000,
scf_prim_name[scfsp->scfs_prim],
scfsp->scfs_wait ? "wait" : "nowait");
oldw = scfsp->scfs_weight;
}
}
// Randomly pick a primitive and wait/nowait, based on weightings.
static struct scf_selector *scf_sel_rand(struct torture_random_state *trsp)
{
int i;
unsigned long w = torture_random(trsp) % (scf_sel_totweight + 1);
for (i = 0; i < scf_sel_array_len; i++)
if (scf_sel_array[i].scfs_weight >= w)
return &scf_sel_array[i];
WARN_ON_ONCE(1);
return &scf_sel_array[0];
}
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
// Update statistics and occasionally burn up mass quantities of CPU time,
// if told to do so via scftorture.longwait. Otherwise, occasionally burn
// a little bit.
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
static void scf_handler(void *scfc_in)
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
{
int i;
int j;
unsigned long r = torture_random(this_cpu_ptr(&scf_torture_rand));
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
struct scf_check *scfcp = scfc_in;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
scftorture: Add smp_call_function_many() memory-ordering checks This commit adds checks for memory misordering across calls to and returns from smp_call_function_many() in the case where the caller waits. Misordering results in a splat. Note that in contrast to smp_call_function_single(), this code does not test memory ordering into the handler in the no-wait case because none of the handlers would be able to free the scf_check structure without introducing heavy synchronization to work out which was last. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 22:30:02 +03:00
if (likely(scfcp)) {
WRITE_ONCE(scfcp->scfc_out, false); // For multiple receivers.
if (WARN_ON_ONCE(unlikely(!READ_ONCE(scfcp->scfc_in))))
atomic_inc(&n_mb_in_errs);
}
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
this_cpu_inc(scf_invoked_count);
if (longwait <= 0) {
scftorture: Fix distribution of short handler delays The scftorture test module's scf_handler() function is supposed to provide three different distributions of short delays (including "no delay") and one distribution of long delays, if specified by the scftorture.longwait module parameter. However, the second of the two non-zero-wait short delays is disabled due to the first such delay's "goto out" not being enclosed in the "then" clause with the "udelay()". This commit therefore adjusts the code to provide the intended set of delays. Fixes: e9d338a0b179 ("scftorture: Add smp_call_function() torture test") Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2022-03-01 04:40:49 +03:00
if (!(r & 0xffc0)) {
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
udelay(r & 0x3f);
scftorture: Fix distribution of short handler delays The scftorture test module's scf_handler() function is supposed to provide three different distributions of short delays (including "no delay") and one distribution of long delays, if specified by the scftorture.longwait module parameter. However, the second of the two non-zero-wait short delays is disabled due to the first such delay's "goto out" not being enclosed in the "then" clause with the "udelay()". This commit therefore adjusts the code to provide the intended set of delays. Fixes: e9d338a0b179 ("scftorture: Add smp_call_function() torture test") Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2022-03-01 04:40:49 +03:00
goto out;
}
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
}
if (r & 0xfff)
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
goto out;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
r = (r >> 12);
if (longwait <= 0) {
udelay((r & 0xff) + 1);
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
goto out;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
}
r = r % longwait + 1;
for (i = 0; i < r; i++) {
for (j = 0; j < 1000; j++) {
udelay(1000);
cpu_relax();
}
}
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
out:
if (unlikely(!scfcp))
return;
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
if (scfcp->scfc_wait) {
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
WRITE_ONCE(scfcp->scfc_out, true);
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
if (scfcp->scfc_rpc)
complete(&scfcp->scfc_completion);
} else {
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
kfree(scfcp);
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
}
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
}
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
// As above, but check for correct CPU.
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
static void scf_handler_1(void *scfc_in)
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
{
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
struct scf_check *scfcp = scfc_in;
if (likely(scfcp) && WARN_ONCE(smp_processor_id() != scfcp->scfc_cpu, "%s: Wanted CPU %d got CPU %d\n", __func__, scfcp->scfc_cpu, smp_processor_id())) {
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
atomic_inc(&n_errs);
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
}
scf_handler(scfcp);
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
}
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
// Randomly do an smp_call_function*() invocation.
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_random_state *trsp)
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
{
scftorture: Pause testing after memory-allocation failure The scftorture test can quickly execute a large number of calls to no-wait smp_call_function(), each of which holds a block of memory until the corresponding handler is invoked. Especially when the longwait module parameter is specified, this can chew up an arbitrarily large amount of memory. This commit therefore blocks after each memory-allocation failure, with the duration a function of longwait. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2023-05-16 16:21:02 +03:00
bool allocfail = false;
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
uintptr_t cpu;
scftorture: Consolidate scftorture_invoke_one() check and kfree() This commit moves checking of the ->scfc_out field and the freeing of the scf_check structure down below the end of switch statement, thus saving a few lines of code. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 00:13:02 +03:00
int ret = 0;
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
struct scf_check *scfcp = NULL;
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
struct scf_selector *scfsp = scf_sel_rand(trsp);
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
if (use_cpus_read_lock)
cpus_read_lock();
else
preempt_disable();
scftorture: Add smp_call_function() memory-ordering checks This commit adds checks for memory misordering across calls to and returns from smp_call_function() in the case where the caller waits. Misordering results in a splat. Note that in contrast to smp_call_function_single(), this code does not test memory ordering into the handler in the no-wait case because none of the handlers would be able to free the scf_check structure without introducing heavy synchronization to work out which was last. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 23:49:06 +03:00
if (scfsp->scfs_prim == SCF_PRIM_SINGLE || scfsp->scfs_wait) {
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
scfcp = kmalloc(sizeof(*scfcp), GFP_ATOMIC);
scftorture: Forgive memory-allocation failure if KASAN Kernels built with CONFIG_KASAN=y quarantine newly freed memory in order to better detect use-after-free errors. However, this can exhaust memory more quickly in allocator-heavy tests, which can result in spurious scftorture failure. This commit therefore forgives memory-allocation failure in kernels built with CONFIG_KASAN=y, but continues counting the errors for use in detailed test-result analyses. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2023-05-16 05:00:10 +03:00
if (!scfcp) {
WARN_ON_ONCE(!IS_ENABLED(CONFIG_KASAN));
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
atomic_inc(&n_alloc_errs);
scftorture: Pause testing after memory-allocation failure The scftorture test can quickly execute a large number of calls to no-wait smp_call_function(), each of which holds a block of memory until the corresponding handler is invoked. Especially when the longwait module parameter is specified, this can chew up an arbitrarily large amount of memory. This commit therefore blocks after each memory-allocation failure, with the duration a function of longwait. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2023-05-16 16:21:02 +03:00
allocfail = true;
scftorture: Consolidate scftorture_invoke_one() scf_check initialization This commit hoists much of the initialization of the scf_check structure out of the switch statement, thus saving a few lines of code. The initialization of the ->scfc_in field remains in each leg of the switch statement in order to more heavily stress memory ordering. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-09 23:58:32 +03:00
} else {
scfcp->scfc_cpu = -1;
scfcp->scfc_wait = scfsp->scfs_wait;
scfcp->scfc_out = false;
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
scfcp->scfc_rpc = false;
scftorture: Consolidate scftorture_invoke_one() scf_check initialization This commit hoists much of the initialization of the scf_check structure out of the switch statement, thus saving a few lines of code. The initialization of the ->scfc_in field remains in each leg of the switch statement in order to more heavily stress memory ordering. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-09 23:58:32 +03:00
}
scftorture: Add smp_call_function() memory-ordering checks This commit adds checks for memory misordering across calls to and returns from smp_call_function() in the case where the caller waits. Misordering results in a splat. Note that in contrast to smp_call_function_single(), this code does not test memory ordering into the handler in the no-wait case because none of the handlers would be able to free the scf_check structure without introducing heavy synchronization to work out which was last. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 23:49:06 +03:00
}
switch (scfsp->scfs_prim) {
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
case SCF_PRIM_RESCHED:
if (IS_BUILTIN(CONFIG_SCF_TORTURE_TEST)) {
cpu = torture_random(trsp) % nr_cpu_ids;
scfp->n_resched++;
resched_cpu(cpu);
scftorture: Count reschedule IPIs Currently, only those IPIs that invoke scftorture's scf_handler() IPI handler function are counted. This means that runs exercising only scftorture.weight_resched will look like they have made no forward progress, resulting in "GP HANG" complaints from the rcutorture scripting. This commit therefore increments the scf_invoked_count per-CPU counter immediately after calling resched_cpu(). Fixes: 1ac78b49d61d4 ("scftorture: Add an alternative IPI vector") Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-07-14 16:53:51 +03:00
this_cpu_inc(scf_invoked_count);
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
}
break;
scftorture: Add smp_call_function() memory-ordering checks This commit adds checks for memory misordering across calls to and returns from smp_call_function() in the case where the caller waits. Misordering results in a splat. Note that in contrast to smp_call_function_single(), this code does not test memory ordering into the handler in the no-wait case because none of the handlers would be able to free the scf_check structure without introducing heavy synchronization to work out which was last. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 23:49:06 +03:00
case SCF_PRIM_SINGLE:
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
cpu = torture_random(trsp) % nr_cpu_ids;
if (scfsp->scfs_wait)
scfp->n_single_wait++;
else
scfp->n_single++;
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
if (scfcp) {
scfcp->scfc_cpu = cpu;
scftorture: Prevent compiler from reducing race probabilities Detecting smp_call_function() memory misordering requires close timing, so it is necessary to have the checks immediately before and after the call to the smp_call_function*() function under test. This commit therefore inserts barrier() calls to prevent the compiler from optimizing memory-misordering detection down into the zone of extreme improbability. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 02:38:16 +03:00
barrier(); // Prevent race-reduction compiler optimizations.
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
scfcp->scfc_in = true;
}
ret = smp_call_function_single(cpu, scf_handler_1, (void *)scfcp, scfsp->scfs_wait);
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
if (ret) {
if (scfsp->scfs_wait)
scfp->n_single_wait_ofl++;
else
scfp->n_single_ofl++;
scftorture: Add smp_call_function_single() memory-ordering checks This commit adds checks for memory misordering across calls to smp_call_function_single() and also across returns in the case where the caller waits. Misordering results in a splat. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 06:49:50 +03:00
kfree(scfcp);
scftorture: Consolidate scftorture_invoke_one() check and kfree() This commit moves checking of the ->scfc_out field and the freeing of the scf_check structure down below the end of switch statement, thus saving a few lines of code. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 00:13:02 +03:00
scfcp = NULL;
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
}
break;
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
case SCF_PRIM_SINGLE_RPC:
if (!scfcp)
break;
cpu = torture_random(trsp) % nr_cpu_ids;
scfp->n_single_rpc++;
scfcp->scfc_cpu = cpu;
scfcp->scfc_wait = true;
init_completion(&scfcp->scfc_completion);
scfcp->scfc_rpc = true;
barrier(); // Prevent race-reduction compiler optimizations.
scfcp->scfc_in = true;
ret = smp_call_function_single(cpu, scf_handler_1, (void *)scfcp, 0);
if (!ret) {
if (use_cpus_read_lock)
cpus_read_unlock();
else
preempt_enable();
wait_for_completion(&scfcp->scfc_completion);
if (use_cpus_read_lock)
cpus_read_lock();
else
preempt_disable();
} else {
scfp->n_single_rpc_ofl++;
kfree(scfcp);
scfcp = NULL;
}
break;
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
case SCF_PRIM_MANY:
if (scfsp->scfs_wait)
scfp->n_many_wait++;
else
scfp->n_many++;
scftorture: Prevent compiler from reducing race probabilities Detecting smp_call_function() memory misordering requires close timing, so it is necessary to have the checks immediately before and after the call to the smp_call_function*() function under test. This commit therefore inserts barrier() calls to prevent the compiler from optimizing memory-misordering detection down into the zone of extreme improbability. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 02:38:16 +03:00
if (scfcp) {
barrier(); // Prevent race-reduction compiler optimizations.
scftorture: Add smp_call_function_many() memory-ordering checks This commit adds checks for memory misordering across calls to and returns from smp_call_function_many() in the case where the caller waits. Misordering results in a splat. Note that in contrast to smp_call_function_single(), this code does not test memory ordering into the handler in the no-wait case because none of the handlers would be able to free the scf_check structure without introducing heavy synchronization to work out which was last. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 22:30:02 +03:00
scfcp->scfc_in = true;
scftorture: Prevent compiler from reducing race probabilities Detecting smp_call_function() memory misordering requires close timing, so it is necessary to have the checks immediately before and after the call to the smp_call_function*() function under test. This commit therefore inserts barrier() calls to prevent the compiler from optimizing memory-misordering detection down into the zone of extreme improbability. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 02:38:16 +03:00
}
scftorture: Add smp_call_function_many() memory-ordering checks This commit adds checks for memory misordering across calls to and returns from smp_call_function_many() in the case where the caller waits. Misordering results in a splat. Note that in contrast to smp_call_function_single(), this code does not test memory ordering into the handler in the no-wait case because none of the handlers would be able to free the scf_check structure without introducing heavy synchronization to work out which was last. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 22:30:02 +03:00
smp_call_function_many(cpu_online_mask, scf_handler, scfcp, scfsp->scfs_wait);
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
break;
case SCF_PRIM_ALL:
if (scfsp->scfs_wait)
scfp->n_all_wait++;
else
scfp->n_all++;
scftorture: Prevent compiler from reducing race probabilities Detecting smp_call_function() memory misordering requires close timing, so it is necessary to have the checks immediately before and after the call to the smp_call_function*() function under test. This commit therefore inserts barrier() calls to prevent the compiler from optimizing memory-misordering detection down into the zone of extreme improbability. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 02:38:16 +03:00
if (scfcp) {
barrier(); // Prevent race-reduction compiler optimizations.
scftorture: Add smp_call_function() memory-ordering checks This commit adds checks for memory misordering across calls to and returns from smp_call_function() in the case where the caller waits. Misordering results in a splat. Note that in contrast to smp_call_function_single(), this code does not test memory ordering into the handler in the no-wait case because none of the handlers would be able to free the scf_check structure without introducing heavy synchronization to work out which was last. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 23:49:06 +03:00
scfcp->scfc_in = true;
scftorture: Prevent compiler from reducing race probabilities Detecting smp_call_function() memory misordering requires close timing, so it is necessary to have the checks immediately before and after the call to the smp_call_function*() function under test. This commit therefore inserts barrier() calls to prevent the compiler from optimizing memory-misordering detection down into the zone of extreme improbability. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 02:38:16 +03:00
}
scftorture: Add smp_call_function() memory-ordering checks This commit adds checks for memory misordering across calls to and returns from smp_call_function() in the case where the caller waits. Misordering results in a splat. Note that in contrast to smp_call_function_single(), this code does not test memory ordering into the handler in the no-wait case because none of the handlers would be able to free the scf_check structure without introducing heavy synchronization to work out which was last. [ paulmck: s/GFP_KERNEL/GFP_ATOMIC/ per kernel test robot feedback. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 23:49:06 +03:00
smp_call_function(scf_handler, scfcp, scfsp->scfs_wait);
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
break;
scftorture: Check unexpected "switch" statement value This commit adds a "default" case to the switch statement in scftorture_invoke_one() which contains a WARN_ON_ONCE() and an assignment to ->scfc_out to suppress knock-on warnings. These knock-on warnings could otherwise cause the user to think that there was a memory-ordering problem in smp_call_function() instead of a bug in scftorture.c itself. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 22:15:37 +03:00
default:
WARN_ON_ONCE(1);
if (scfcp)
scfcp->scfc_out = true;
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
}
scftorture: Consolidate scftorture_invoke_one() check and kfree() This commit moves checking of the ->scfc_out field and the freeing of the scf_check structure down below the end of switch statement, thus saving a few lines of code. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 00:13:02 +03:00
if (scfcp && scfsp->scfs_wait) {
scftorture: Adapt memory-ordering test to UP operation On uniprocessor systems, smp_call_function() does nothing. This commit therefore avoids complaining about the lack of handler accesses in the single-CPU case where there is no handler. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-04 01:23:19 +03:00
if (WARN_ON_ONCE((num_online_cpus() > 1 || scfsp->scfs_prim == SCF_PRIM_SINGLE) &&
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
!scfcp->scfc_out)) {
pr_warn("%s: Memory-ordering failure, scfs_prim: %d.\n", __func__, scfsp->scfs_prim);
scftorture: Consolidate scftorture_invoke_one() check and kfree() This commit moves checking of the ->scfc_out field and the freeing of the scf_check structure down below the end of switch statement, thus saving a few lines of code. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 00:13:02 +03:00
atomic_inc(&n_mb_out_errs); // Leak rather than trash!
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
} else {
scftorture: Consolidate scftorture_invoke_one() check and kfree() This commit moves checking of the ->scfc_out field and the freeing of the scf_check structure down below the end of switch statement, thus saving a few lines of code. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 00:13:02 +03:00
kfree(scfcp);
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
}
scftorture: Prevent compiler from reducing race probabilities Detecting smp_call_function() memory misordering requires close timing, so it is necessary to have the checks immediately before and after the call to the smp_call_function*() function under test. This commit therefore inserts barrier() calls to prevent the compiler from optimizing memory-misordering detection down into the zone of extreme improbability. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 02:38:16 +03:00
barrier(); // Prevent race-reduction compiler optimizations.
scftorture: Consolidate scftorture_invoke_one() check and kfree() This commit moves checking of the ->scfc_out field and the freeing of the scf_check structure down below the end of switch statement, thus saving a few lines of code. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 00:13:02 +03:00
}
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
if (use_cpus_read_lock)
cpus_read_unlock();
else
preempt_enable();
scftorture: Pause testing after memory-allocation failure The scftorture test can quickly execute a large number of calls to no-wait smp_call_function(), each of which holds a block of memory until the corresponding handler is invoked. Especially when the longwait module parameter is specified, this can chew up an arbitrarily large amount of memory. This commit therefore blocks after each memory-allocation failure, with the duration a function of longwait. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2023-05-16 16:21:02 +03:00
if (allocfail)
schedule_timeout_idle((1 + longwait) * HZ); // Let no-wait handlers complete.
else if (!(torture_random(trsp) & 0xfff))
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
schedule_timeout_uninterruptible(1);
}
// SCF test kthread. Repeatedly does calls to members of the
// smp_call_function() family of functions.
static int scftorture_invoker(void *arg)
{
scftorture: Block scftorture_invoker() kthreads for offline CPUs Currently, CPU-hotplug operations might result in all but two of (say) 100 CPUs being offline, which in turn might result in false-positive diagnostics due to overload. This commit therefore causes scftorture_invoker() kthreads for offline CPUs to loop blocking for 200 milliseconds at a time, thus continuously adjusting the number of threads to match the number of online CPUs. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-03 00:15:33 +03:00
int cpu;
scftorture: Add debug output for wrong-CPU warning This commit adds the desired CPU, the actual CPU, and nr_cpu_ids to the wrong-CPU warning in scftorture_invoker(), the better to help with debugging. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-11-11 21:12:05 +03:00
int curcpu;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
DEFINE_TORTURE_RANDOM(rand);
struct scf_statistics *scfp = (struct scf_statistics *)arg;
scftorture: Block scftorture_invoker() kthreads for offline CPUs Currently, CPU-hotplug operations might result in all but two of (say) 100 CPUs being offline, which in turn might result in false-positive diagnostics due to overload. This commit therefore causes scftorture_invoker() kthreads for offline CPUs to loop blocking for 200 milliseconds at a time, thus continuously adjusting the number of threads to match the number of online CPUs. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-03 00:15:33 +03:00
bool was_offline = false;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
VERBOSE_SCFTORTOUT("scftorture_invoker %d: task started", scfp->cpu);
scftorture: Block scftorture_invoker() kthreads for offline CPUs Currently, CPU-hotplug operations might result in all but two of (say) 100 CPUs being offline, which in turn might result in false-positive diagnostics due to overload. This commit therefore causes scftorture_invoker() kthreads for offline CPUs to loop blocking for 200 milliseconds at a time, thus continuously adjusting the number of threads to match the number of online CPUs. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-03 00:15:33 +03:00
cpu = scfp->cpu % nr_cpu_ids;
scftorture: Avoid false-positive warnings in scftorture_invoker() If the call to set_cpus_allowed_ptr() in scftorture_invoker() fails, a later WARN_ONCE() complains. But with the advent of 570a752b7a9b ("lib/smp_processor_id: Use is_percpu_thread() instead of nr_cpus_allowed"), this complaint can be drowned out by complaints from smp_processor_id(). The rationale for this change is that scftorture's kthreads are not marked with PF_NO_SETAFFINITY, which means that a system administrator could change affinity at any time. However, scftorture is a torture test, and the system administrator might well have a valid test-the-test reason for changing affinity. This commit therefore changes to raw_smp_processor_id() in order to avoid the noise, and also adds a WARN_ON_ONCE() to the call to set_cpus_allowed_ptr() in order to directly detect immediate failure. There is no WARN_ON_ONCE() within the test loop, allowing human-reflex-based affinity resetting, if desired. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-04 22:37:44 +03:00
WARN_ON_ONCE(set_cpus_allowed_ptr(current, cpumask_of(cpu)));
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
set_user_nice(current, MAX_NICE);
if (holdoff)
schedule_timeout_interruptible(holdoff * HZ);
scftorture: Avoid false-positive warnings in scftorture_invoker() If the call to set_cpus_allowed_ptr() in scftorture_invoker() fails, a later WARN_ONCE() complains. But with the advent of 570a752b7a9b ("lib/smp_processor_id: Use is_percpu_thread() instead of nr_cpus_allowed"), this complaint can be drowned out by complaints from smp_processor_id(). The rationale for this change is that scftorture's kthreads are not marked with PF_NO_SETAFFINITY, which means that a system administrator could change affinity at any time. However, scftorture is a torture test, and the system administrator might well have a valid test-the-test reason for changing affinity. This commit therefore changes to raw_smp_processor_id() in order to avoid the noise, and also adds a WARN_ON_ONCE() to the call to set_cpus_allowed_ptr() in order to directly detect immediate failure. There is no WARN_ON_ONCE() within the test loop, allowing human-reflex-based affinity resetting, if desired. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-04 22:37:44 +03:00
VERBOSE_SCFTORTOUT("scftorture_invoker %d: Waiting for all SCF torturers from cpu %d", scfp->cpu, raw_smp_processor_id());
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
// Make sure that the CPU is affinitized appropriately during testing.
scftorture: Avoid false-positive warnings in scftorture_invoker() If the call to set_cpus_allowed_ptr() in scftorture_invoker() fails, a later WARN_ONCE() complains. But with the advent of 570a752b7a9b ("lib/smp_processor_id: Use is_percpu_thread() instead of nr_cpus_allowed"), this complaint can be drowned out by complaints from smp_processor_id(). The rationale for this change is that scftorture's kthreads are not marked with PF_NO_SETAFFINITY, which means that a system administrator could change affinity at any time. However, scftorture is a torture test, and the system administrator might well have a valid test-the-test reason for changing affinity. This commit therefore changes to raw_smp_processor_id() in order to avoid the noise, and also adds a WARN_ON_ONCE() to the call to set_cpus_allowed_ptr() in order to directly detect immediate failure. There is no WARN_ON_ONCE() within the test loop, allowing human-reflex-based affinity resetting, if desired. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-04 22:37:44 +03:00
curcpu = raw_smp_processor_id();
scftorture: Add debug output for wrong-CPU warning This commit adds the desired CPU, the actual CPU, and nr_cpu_ids to the wrong-CPU warning in scftorture_invoker(), the better to help with debugging. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-11-11 21:12:05 +03:00
WARN_ONCE(curcpu != scfp->cpu % nr_cpu_ids,
"%s: Wanted CPU %d, running on %d, nr_cpu_ids = %d\n",
__func__, scfp->cpu, curcpu, nr_cpu_ids);
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
if (!atomic_dec_return(&n_started))
while (atomic_read_acquire(&n_started)) {
if (torture_must_stop()) {
VERBOSE_SCFTORTOUT("scftorture_invoker %d ended before starting", scfp->cpu);
goto end;
}
schedule_timeout_uninterruptible(1);
}
VERBOSE_SCFTORTOUT("scftorture_invoker %d started", scfp->cpu);
do {
scftorture_invoke_one(scfp, &rand);
scftorture: Block scftorture_invoker() kthreads for offline CPUs Currently, CPU-hotplug operations might result in all but two of (say) 100 CPUs being offline, which in turn might result in false-positive diagnostics due to overload. This commit therefore causes scftorture_invoker() kthreads for offline CPUs to loop blocking for 200 milliseconds at a time, thus continuously adjusting the number of threads to match the number of online CPUs. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-03 00:15:33 +03:00
while (cpu_is_offline(cpu) && !torture_must_stop()) {
schedule_timeout_interruptible(HZ / 5);
was_offline = true;
}
if (was_offline) {
set_cpus_allowed_ptr(current, cpumask_of(cpu));
was_offline = false;
}
scftorture: Add cond_resched() to test loop Although the test loop does randomly delay, which would provide quiescent states and so forth, it is possible for there to be a series of long smp_call_function*() handler runtimes with no delays, which results in softlockup and RCU CPU stall warning messages. This commit therefore inserts a cond_resched() into the main test loop. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 01:53:02 +03:00
cond_resched();
scftorture: Add full-test stutter capability In virtual environments on systems with hardware assist, inter-processor interrupts must do very different things based on whether the target vCPU is running or not. This commit therefore enables torture-test stuttering to better test these running/not-running transitions. Suggested-by: Chris Mason <clm@fb.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-24 22:11:57 +03:00
stutter_wait("scftorture_invoker");
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
} while (!torture_must_stop());
VERBOSE_SCFTORTOUT("scftorture_invoker %d ended", scfp->cpu);
end:
torture_kthread_stopping("scftorture_invoker");
return 0;
}
static void
scftorture_print_module_parms(const char *tag)
{
pr_alert(SCFTORT_FLAG
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
"--- %s: verbose=%d holdoff=%d longwait=%d nthreads=%d onoff_holdoff=%d onoff_interval=%d shutdown_secs=%d stat_interval=%d stutter=%d use_cpus_read_lock=%d, weight_resched=%d, weight_single=%d, weight_single_rpc=%d, weight_single_wait=%d, weight_many=%d, weight_many_wait=%d, weight_all=%d, weight_all_wait=%d\n", tag,
verbose, holdoff, longwait, nthreads, onoff_holdoff, onoff_interval, shutdown, stat_interval, stutter, use_cpus_read_lock, weight_resched, weight_single, weight_single_rpc, weight_single_wait, weight_many, weight_many_wait, weight_all, weight_all_wait);
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
}
static void scf_cleanup_handler(void *unused)
{
}
static void scf_torture_cleanup(void)
{
int i;
if (torture_cleanup_begin())
return;
WRITE_ONCE(scfdone, true);
scftorture: Avoid NULL pointer exception on early exit When scftorture finds an error in the module parameters controlling the relative frequencies of smp_call_function*() variants, it takes an early exit. So early that it has not allocated memory to track the kthreads running the test, which results in a segfault. This commit therefore checks for the existence of the memory before attempting to stop the kthreads that would otherwise have been recorded in that non-existent memory. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-07-10 03:53:27 +03:00
if (nthreads && scf_stats_p)
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
for (i = 0; i < nthreads; i++)
torture_stop_kthread("scftorture_invoker", scf_stats_p[i].task);
else
goto end;
smp_call_function(scf_cleanup_handler, NULL, 0);
torture_stop_kthread(scf_torture_stats, scf_torture_stats_task);
scf_torture_stats_print(); // -After- the stats thread is stopped!
scftorture: Summarize per-thread statistics This commit summarizes the per-thread statistics, providing counts of the number of single, many, and all calls, both no-wait and wait, and, for the single case, the number where the target CPU was offline. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-01 02:13:37 +03:00
kfree(scf_stats_p); // -After- the last stats print has completed!
scf_stats_p = NULL;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
scftorture: Flag errors in torture-compatible manner This commit prints error counts on the statistics line and also adds a "!!!" if any of the counters are non-zero. Allocation failures are (somewhat) forgiven, but all other errors result in a "FAILURE" print at the end of the test. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-02 02:06:22 +03:00
if (atomic_read(&n_errs) || atomic_read(&n_mb_in_errs) || atomic_read(&n_mb_out_errs))
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
scftorture_print_module_parms("End of test: FAILURE");
else if (torture_onoff_failures())
scftorture_print_module_parms("End of test: LOCK_HOTPLUG");
else
scftorture_print_module_parms("End of test: SUCCESS");
end:
torture_cleanup_end();
}
static int __init scf_torture_init(void)
{
long i;
int firsterr = 0;
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
unsigned long weight_resched1 = weight_resched;
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
unsigned long weight_single1 = weight_single;
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
unsigned long weight_single_rpc1 = weight_single_rpc;
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
unsigned long weight_single_wait1 = weight_single_wait;
unsigned long weight_many1 = weight_many;
unsigned long weight_many_wait1 = weight_many_wait;
unsigned long weight_all1 = weight_all;
unsigned long weight_all_wait1 = weight_all_wait;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
if (!torture_init_begin(SCFTORT_STRING, verbose))
return -EBUSY;
scftorture_print_module_parms("Start of test");
scftorture: Allow zero weight to exclude an smp_call_function*() category This commit reworks the weighting calculations to allow zero to be specified to disable a given weight. For example, specifying the scftorture.weight_resched=0 kernel boot parameter without specifying a non-zero value for any of the other scftorture.weight_* parameters would provide the default weights for the others, but would refrain from doing any resched-based IPIs. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-07-14 00:12:54 +03:00
if (weight_resched <= 0 &&
weight_single <= 0 && weight_single_rpc <= 0 && weight_single_wait <= 0 &&
weight_many <= 0 && weight_many_wait <= 0 &&
weight_all <= 0 && weight_all_wait <= 0) {
weight_resched1 = weight_resched == 0 ? 0 : 2 * nr_cpu_ids;
weight_single1 = weight_single == 0 ? 0 : 2 * nr_cpu_ids;
weight_single_rpc1 = weight_single_rpc == 0 ? 0 : 2 * nr_cpu_ids;
weight_single_wait1 = weight_single_wait == 0 ? 0 : 2 * nr_cpu_ids;
weight_many1 = weight_many == 0 ? 0 : 2;
weight_many_wait1 = weight_many_wait == 0 ? 0 : 2;
weight_all1 = weight_all == 0 ? 0 : 1;
weight_all_wait1 = weight_all_wait == 0 ? 0 : 1;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
} else {
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
if (weight_resched == -1)
weight_resched1 = 0;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
if (weight_single == -1)
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
weight_single1 = 0;
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
if (weight_single_rpc == -1)
weight_single_rpc1 = 0;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
if (weight_single_wait == -1)
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
weight_single_wait1 = 0;
if (weight_many == -1)
weight_many1 = 0;
if (weight_many_wait == -1)
weight_many_wait1 = 0;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
if (weight_all == -1)
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
weight_all1 = 0;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
if (weight_all_wait == -1)
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
weight_all_wait1 = 0;
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
}
scftorture: Account for weight_resched when checking for all zeroes The "all zero weights makes no sense" error is emitted even when scftorture.weight_resched is non-zero because it was left out of the enclosing "if" condition. This commit adds it in. Fixes: 1ac78b49d61d4 ("scftorture: Add an alternative IPI vector") Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-07-14 01:13:56 +03:00
if (weight_resched1 == 0 && weight_single1 == 0 && weight_single_rpc1 == 0 &&
weight_single_wait1 == 0 && weight_many1 == 0 && weight_many_wait1 == 0 &&
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
weight_all1 == 0 && weight_all_wait1 == 0) {
scftorture: Always log error message Unconditionally log messages corresponding to errors. Acked-by: Davidlohr Bueso <dbueso@suse.de> Signed-off-by: Li Zhijian <zhijianx.li@intel.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-11-03 11:30:27 +03:00
SCFTORTOUT_ERRSTRING("all zero weights makes no sense");
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
firsterr = -EINVAL;
goto unwind;
}
scftorture: Add an alternative IPI vector The scftorture tests currently use only smp_call_function() and friends, which means that these tests cannot locate bugs caused by interactions between different IPI vectors. This commit therefore adds the rescheduling IPI to the mix. Note that this commit permits resched_cpus() only when scftorture is built in. This is a workaround. Longer term, this will use real wakeups rather than resched_cpu(). Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-03 23:09:47 +03:00
if (IS_BUILTIN(CONFIG_SCF_TORTURE_TEST))
scf_sel_add(weight_resched1, SCF_PRIM_RESCHED, false);
else if (weight_resched1)
scftorture: Always log error message Unconditionally log messages corresponding to errors. Acked-by: Davidlohr Bueso <dbueso@suse.de> Signed-off-by: Li Zhijian <zhijianx.li@intel.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-11-03 11:30:27 +03:00
SCFTORTOUT_ERRSTRING("built as module, weight_resched ignored");
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
scf_sel_add(weight_single1, SCF_PRIM_SINGLE, false);
scftorture: Add RPC-like IPI tests This commit adds the single_weight_rpc module parameter, which causes the IPI handler to awaken the IPI sender. In many scheduler configurations, this will result in an IPI back to the sender that is likely to be received at a time when the sender CPU is idle. The intent is to stress IPI reception during CPU busy-to-idle transitions. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-06-25 03:53:43 +03:00
scf_sel_add(weight_single_rpc1, SCF_PRIM_SINGLE_RPC, true);
scftorture: Implement weighted primitive selection This commit uses the scftorture.weight* kernel parameters to randomly chooses between smp_call_function_single(), smp_call_function_many(), and smp_call_function(). For each variant, it also randomly chooses whether to invoke it synchronously (wait=1) or asynchronously (wait=0). The percentage weighting for each option are dumped to the console log (search for "scf_sel_dump"). This accumulates statistics, which a later commit will dump out at the end of the run. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-26 03:05:58 +03:00
scf_sel_add(weight_single_wait1, SCF_PRIM_SINGLE, true);
scf_sel_add(weight_many1, SCF_PRIM_MANY, false);
scf_sel_add(weight_many_wait1, SCF_PRIM_MANY, true);
scf_sel_add(weight_all1, SCF_PRIM_ALL, false);
scf_sel_add(weight_all_wait1, SCF_PRIM_ALL, true);
scf_sel_dump();
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
if (onoff_interval > 0) {
firsterr = torture_onoff_init(onoff_holdoff * HZ, onoff_interval, NULL);
scftorture: Warn on individual scf_torture_init() error conditions When running scftorture as a module, any scf_torture_init() issues will be reflected in the error code from modprobe or insmod, as the case may be. However, these error codes are not available when running scftorture built-in, for example, when using the kvm.sh script. This commit therefore adds WARN_ON_ONCE() to allow distinguishing scf_torture_init() errors when running scftorture built-in. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-08-06 02:01:51 +03:00
if (torture_init_error(firsterr))
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
goto unwind;
}
if (shutdown_secs > 0) {
firsterr = torture_shutdown_init(shutdown_secs, scf_torture_cleanup);
scftorture: Warn on individual scf_torture_init() error conditions When running scftorture as a module, any scf_torture_init() issues will be reflected in the error code from modprobe or insmod, as the case may be. However, these error codes are not available when running scftorture built-in, for example, when using the kvm.sh script. This commit therefore adds WARN_ON_ONCE() to allow distinguishing scf_torture_init() errors when running scftorture built-in. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-08-06 02:01:51 +03:00
if (torture_init_error(firsterr))
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
goto unwind;
}
scftorture: Add full-test stutter capability In virtual environments on systems with hardware assist, inter-processor interrupts must do very different things based on whether the target vCPU is running or not. This commit therefore enables torture-test stuttering to better test these running/not-running transitions. Suggested-by: Chris Mason <clm@fb.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-24 22:11:57 +03:00
if (stutter > 0) {
firsterr = torture_stutter_init(stutter, stutter);
scftorture: Warn on individual scf_torture_init() error conditions When running scftorture as a module, any scf_torture_init() issues will be reflected in the error code from modprobe or insmod, as the case may be. However, these error codes are not available when running scftorture built-in, for example, when using the kvm.sh script. This commit therefore adds WARN_ON_ONCE() to allow distinguishing scf_torture_init() errors when running scftorture built-in. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-08-06 02:01:51 +03:00
if (torture_init_error(firsterr))
scftorture: Add full-test stutter capability In virtual environments on systems with hardware assist, inter-processor interrupts must do very different things based on whether the target vCPU is running or not. This commit therefore enables torture-test stuttering to better test these running/not-running transitions. Suggested-by: Chris Mason <clm@fb.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-09-24 22:11:57 +03:00
goto unwind;
}
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
// Worker tasks invoking smp_call_function().
if (nthreads < 0)
nthreads = num_online_cpus();
scf_stats_p = kcalloc(nthreads, sizeof(scf_stats_p[0]), GFP_KERNEL);
if (!scf_stats_p) {
scftorture: Always log error message Unconditionally log messages corresponding to errors. Acked-by: Davidlohr Bueso <dbueso@suse.de> Signed-off-by: Li Zhijian <zhijianx.li@intel.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-11-03 11:30:27 +03:00
SCFTORTOUT_ERRSTRING("out of memory");
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
firsterr = -ENOMEM;
goto unwind;
}
scftorture: Add missing '\n' to flush message Add '\n' to macros to flush message for each call. Acked-by: Davidlohr Bueso <dbueso@suse.de> Signed-off-by: Li Zhijian <zhijianx.li@intel.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-10-29 12:40:25 +03:00
VERBOSE_SCFTORTOUT("Starting %d smp_call_function() threads", nthreads);
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
atomic_set(&n_started, nthreads);
for (i = 0; i < nthreads; i++) {
scf_stats_p[i].cpu = i;
firsterr = torture_create_kthread(scftorture_invoker, (void *)&scf_stats_p[i],
scf_stats_p[i].task);
scftorture: Warn on individual scf_torture_init() error conditions When running scftorture as a module, any scf_torture_init() issues will be reflected in the error code from modprobe or insmod, as the case may be. However, these error codes are not available when running scftorture built-in, for example, when using the kvm.sh script. This commit therefore adds WARN_ON_ONCE() to allow distinguishing scf_torture_init() errors when running scftorture built-in. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-08-06 02:01:51 +03:00
if (torture_init_error(firsterr))
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
goto unwind;
}
if (stat_interval > 0) {
firsterr = torture_create_kthread(scf_torture_stats, NULL, scf_torture_stats_task);
scftorture: Warn on individual scf_torture_init() error conditions When running scftorture as a module, any scf_torture_init() issues will be reflected in the error code from modprobe or insmod, as the case may be. However, these error codes are not available when running scftorture built-in, for example, when using the kvm.sh script. This commit therefore adds WARN_ON_ONCE() to allow distinguishing scf_torture_init() errors when running scftorture built-in. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-08-06 02:01:51 +03:00
if (torture_init_error(firsterr))
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
goto unwind;
}
torture_init_end();
return 0;
unwind:
torture_init_end();
scf_torture_cleanup();
scftorture: Shut down if nonsensical arguments given If (say) a 10-hour scftorture run is started, but the module parameters are so nonsensical that the run doesn't even start, then scftorture will wait the full ten hours when run built into a guest OS. This commit therefore shuts down the system in this case so that the error is reported immediately instead of ten hours hence. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2021-07-14 00:20:35 +03:00
if (shutdown_secs) {
WARN_ON(!IS_MODULE(CONFIG_SCF_TORTURE_TEST));
kernel_power_off();
}
scftorture: Add smp_call_function() torture test This commit adds an smp_call_function() torture test that repeatedly invokes this function and complains if things go badly awry. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-06-25 01:59:59 +03:00
return firsterr;
}
module_init(scf_torture_init);
module_exit(scf_torture_cleanup);
Reference in New Issue Copy Permalink