Introduction
------------

For a developer, the simplest way of running most tests on a local
machine from within the git repository is:

  make test

This runs all unit tests (onnode, takeover, tool, eventscripts) and
the tests against local daemons (simple) using the script
tests/run_tests.sh.

When running tests against a real or virtual cluster the script
tests/run_cluster_tests.sh can be used.  This runs all integration
tests (simple, complex).

Both of these scripts can also take a list of tests to run.  You can
also pass options, which are then passed to run_tests.  However, if
you just try to pass options to run_tests then you lose the default
list of tests that are run.  You can't have everything...

scripts/run_tests
-----------------

The above scripts invoke tests/scripts/run_tests.  This script has a
lot of command-line switches.  Some of the more useful options
include:

  -s  Print a summary of tests results after running all tests

  -l  Use local daemons for integration tests

      This allows the tests in "simple" to be run against local
      daemons.

      All integration tests communicate with cluster nodes using
      onnode or the ctdb tool, which both have some test hooks to
      support local daemons.

      By default 3 daemons are used.  If you want to use a different
      number of daemons then do not use this option but set
      TEST_LOCAL_DAEMONS to the desired number of daemons instead.
      The -l option just sets TEST_LOCAL_DAEMONS to 3...  :-)

  -e  Exit on the first test failure

  -C  Clean up - kill daemons and remove $TEST_VAR_DIR when done

      Tests uses a temporary/var directory for test state.  By default,
      this directory is not removed when tests are complete, so you
      can do forensics or, for integration tests, re-run tests that
      have failed against the same directory (with the same local
      daemons setup).  So this option cleans things up.

      Also kills local daemons associated with directory.

  -V  Use <dir> as $TEST_VAR_DIR

      Use the specified temporary temporary/var directory.

  -H  No headers - for running single test with other wrapper

      This allows tests to be embedded in some other test framework
      and executed one-by-one with all the required
      environment/infrastructure.

      This replaces the old ctdb_test_env script.

How do the tests find remote test programs?
-------------------------------------------

If the all of the cluster nodes have the CTDB git tree in the same
location as on the test client then no special action is necessary.
The simplest way of doing this is to share the tree to cluster nodes
and test clients via NFS.

If cluster nodes do not have the CTDB git tree then
CTDB_TEST_REMOTE_DIR can be set to a directory that, on each cluster
node, contains the contents of tests/scripts/ and tests/bin/.

In the future this will hopefully (also) be supported via a ctdb-test
package.

Running the ctdb tool under valgrind
------------------------------------

The easiest way of doing this is something like:

  VALGRIND="valgrind -q" scripts/run_tests ...

This can be used to cause all invocations of the ctdb client (and,
with local daemons, the ctdbd daemons themselves) to occur under
valgrind.

NOTE: Some libc calls seem to do weird things and perhaps cause
spurious output from ctdbd at start time.  Please read valgrind output
carefully before reporting bugs.  :-)

How is the ctdb tool invoked?
-----------------------------

$CTDB determines how to invoke the ctdb client.  If not already set
and if $VALGRIND is set, this is set to "$VALGRIND ctdb".  If this is
not already set but $VALGRIND is not set, this is simply set to "ctdb"