1
0
mirror of https://github.com/samba-team/samba.git synced 2025-01-06 13:18:07 +03:00
Commit Graph

54 Commits

Author SHA1 Message Date
Martin Schwenke
b20ee18031 ctdb-tests: Fix a cut and paste error in a comment
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue May 31 05:56:43 UTC 2022 on sn-devel-184
2022-05-31 05:56:43 +00:00
Martin Schwenke
90a96f06a9 ctdb-recoverd: Do not ban on unknown error when taking cluster lock
If the cluster filesystem is unavailable then I/O errors may occur.
This is no worse than contention, so don't ban.  This avoids having
services unavailable for longer than necessary.

Update the associated test to simply confirm that this results in a
leaderless cluster, and leadership is restored when the lock can once
again be taken.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-05-31 05:06:29 +00:00
Martin Schwenke
331c435ce5 ctdb-tests: Add a test for stalled node triggering election
A stalled node probably continues to hold the cluster lock, so confirm
elections work in this case.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=14958

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Feb 14 02:46:01 UTC 2022 on sn-devel-184
2022-02-14 02:46:01 +00:00
Martin Schwenke
265e44abc4 ctdb-tests: Factor out functions to detect when generation changes
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14958

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-02-14 01:47:31 +00:00
Martin Schwenke
01313ea243 ctdb-tests: Improve test coverage for leader role yield and elections
Rename test, clean up node selection.  Duplicate for for banning and
removing leader capability cases.  Repeat all 3 tests without cluster
lock.

All of the standard election triggers are now tested, with and without
cluster lock.  Due to test cluster configuration limitations, the
tests without cluster lock are skipped on a real cluster.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-01-17 10:21:33 +00:00
Martin Schwenke
73555e8248 ctdb-recoverd: Use race for cluster lock as election when lock is enabled
If the cluster is partitioned then nodes in one partition can not take
the lock anyway, so election is pointless.  It just introduces
unnecessary corner cases.

Instead just race for the lock.

When a node notices a lack of leader and notifies other nodes of an
election via an unknown leader broadcast, the cluster lock election is
hooked into this broadcast.

The test needs to be updated because losing the cluster lock can now
result in a leadership change.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-01-17 10:21:33 +00:00
Martin Schwenke
f02e097485 ctdb-tools: recovery master -> leader
The following command names are changed:

  recmaster -> leader
  setrecmasterrole -> setleaderrole

Command output changed for the following commands:

  status
  getcapabilities

Documentation and tests are updated to reflect these changes.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-01-17 10:21:33 +00:00
Martin Schwenke
403db5b528 ctdb-tests: Factor out getting leader and waiting for leader change
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-01-17 10:21:33 +00:00
Martin Schwenke
037abf8620 ctdb-tests: Avoid a race
See the comment in the code for details.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-01-17 10:21:32 +00:00
Martin Schwenke
ef7e3265f7 ctdb-tests: Setup cluster with expected arguments
ctdb_test_init() doesn't actually pass arguments to local_daemons.sh.
This needs to be done using ctdb_nodes_start_custom().

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-01-17 10:21:32 +00:00
Martin Schwenke
6a81f43177 ctdb-tests: Actually wait for record to migrate to lmaster node
This test has been failing with:

  Wait until record is migrated to lmaster node 0
  <30|BAD: node 0 is not dmaster
  dmaster: 1
  rsn: 8
  flags: 0x00010000 MIGRATED_WITH_DATA
  data(6) = "value1"
  *** TEST COMPLETED (RC=1) AT 2021-02-02 06:18:48, CLEANING UP...

This should never happen.  If this really fails then the wait should
time out.

The problem is that wait_until() does:

  "$@" || _rc=$?

and vacuum_test_key_dmaster() currently calls ctdb_test_fail() on
failure, which causes the shell to exit.  Instead, pass a variant to
wait_until() that simply returns the correct status instead of
exiting.

An alternative would be to change the statement in wait_until() to do:

  ("$@") || _rc=$?

so it captures the exit.  However, this is a global change and
requires more thought.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Jeremy Allison <jra@samba.org>
2021-02-08 22:33:14 +00:00
Martin Schwenke
b68105b8f7 ctdb-tests: Strengthen node state checking in ctdb disable/enable test
Check that the desired state is set on all nodes instead of just the
test node.  This ensures that node flags have correctly propagated
across the cluster.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=14513
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Oct  6 04:32:06 UTC 2020 on sn-devel-184
2020-10-06 04:32:06 +00:00
Martin Schwenke
d9d8bf8c54 ctdb-tests: Simplify comment in large database recovery test
The older style controls mentioned are being removed.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-09-11 05:06:42 +00:00
Martin Schwenke
971c20e9dc ctdb-tools: Drop "ctdb isnotrecmaster" command
This isn't used anywhere and can easily be checked via "ctdb pnn" and
"ctdb recmaster" commands.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-08-17 04:51:32 +00:00
Martin Schwenke
484a764e83 ctdb-tests: Improve test portability/quality
Avoid use of non-portable md5sum by constructing database names using
index.  Improve indentation, use more modern commands, code
improvements (shellcheck).

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Wed Jul 22 09:14:35 UTC 2020 on sn-devel-184
2020-07-22 09:14:35 +00:00
Martin Schwenke
f4c2c77ff7 ctdb-tests: Improve test quality
Simplify code, use more modern commands, code improvements (shellcheck).

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 07:53:36 +00:00
Martin Schwenke
c6c81ea287 ctdb-tests: Improve test portability
"wc -l" on some platforms (e.g. FreeBSD) contains leading spaces, so
strip them.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 07:53:36 +00:00
Martin Schwenke
244eaad76a ctdb-tests: Improve test quality
Select test node with IPs instead of using a fixed node.  Remove
unnecessary code, use more modern commands, code
improvements (shellcheck).

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 07:53:36 +00:00
Martin Schwenke
760c3039b0 ctdb-tests: Improve test portability
"wc -l" on some platforms (e.g. FreeBSD) contains leading spaces and
stops "$num from being a number.  Create a more portable solution and
put it in a function instead of repeating the logic.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 07:53:36 +00:00
Martin Schwenke
41ff58338a ctdb-tests: Drop uses of "onnode any ..." in testcases
It would be nice to get rid of "onnode any".  There's no use making
tests nondeterministic.  If covering different cases matters then they
should be explicitly handled.

In most places "any" is replaced by "$test_node".  In some cases,
where $test_node is not set, a fixed node that is already used
elsewhere can be reused.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 07:53:36 +00:00
Martin Schwenke
37c26a9590 ctdb-tests: Separate custom cluster startup from test initialisation
Separate cluster startup from test initialisation for tests that start
the cluster with customised configuration.  In these cases the result
of the cluster startup is actually the point of the test.
Additionally, pubips.013.failover_noop.sh claims to have completed
test initialisation twice, which just seems wrong.

The result is:

* ctdb_test_init() takes one option (-n) to indicate when it should
  not configure/start the cluster

* New function ctdb_nodes_start_custom() accepts options for special
  cluster configuration, only operates on local daemons and triggers a
  test failure rather than a test error on failure.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 07:53:36 +00:00
Martin Schwenke
8b24cae630 ctdb-tests: Update preamble for INTEGRATION tests
* Use "#!/usr/bin/env bash" for improved portability

* Drop test_info() definition and replace it with a comment

  The use of test_info() is pointless.

* Drop call to cluster_is_healthy()

  This is a holdover from when the previous test would restart daemons
  to get things ready for a test.  There was also a bug where going
  into recovery during the restart would sometimes cause the cluster
  to become unhealthy.  If we really need something like this then we
  can add it to ctdb_test_init().

* Make order of preamble consistent

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 07:53:35 +00:00
Martin Schwenke
1565446508 ctdb-tests: Show hung script debugging output
The output in a test failure appears to contain no pstree output
because "00\.test\.script,.*" does not match.  However, this is just a
guess because the output is not shown.

Showing the output makes it easier to understand test failures.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 07:53:35 +00:00
Martin Schwenke
6a3372e895 Revert "ctdb-tests: Show hung script debugging output"
Fix missing Reviewed-by: tag.

This reverts commit c78de201f8.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 05:07:46 +00:00
Martin Schwenke
4438e44f88 Revert "ctdb-tests: Update preamble for INTEGRATION tests"
Fix missing Reviewed-by: tag.

This reverts commit 65f56505e2.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 05:07:46 +00:00
Martin Schwenke
e11526ad54 Revert "ctdb-tests: Separate custom cluster startup from test initialisation"
Fix missing Reviewed-by: tag.

This reverts commit e9df17b500.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 05:07:45 +00:00
Martin Schwenke
c9dfdeaddc Revert "ctdb-tests: Drop uses of "onnode any ..." in testcases"
Fix missing Reviewed-by: tag.

This reverts commit aa5b214eaa.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 05:07:45 +00:00
Martin Schwenke
635d5cfa31 Revert "ctdb-tests: Improve test portability"
Fix missing Reviewed-by: tag.

This reverts commit 1079d6e3ae.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 05:07:45 +00:00
Martin Schwenke
c83ece42e5 Revert "ctdb-tests: Improve test quality"
Fix missing Reviewed-by: tag.

This reverts commit ea1cbff624.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 05:07:45 +00:00
Martin Schwenke
cf3b1fb390 Revert "ctdb-tests: Improve test portability"
Fix missing Reviewed-by: tag.

This reverts commit 1f6556916e.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 05:07:45 +00:00
Martin Schwenke
979a6c8c5f Revert "ctdb-tests: Improve test quality"
Fix missing Reviewed-by: tag.

This reverts commit a308f2534d.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 05:07:45 +00:00
Martin Schwenke
d035b69b53 Revert "ctdb-tests: Improve test portability/quality"
Fix missing Reviewed-by: tag.

This reverts commit d2f8cd835d.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-07-22 05:07:45 +00:00
Martin Schwenke
d2f8cd835d ctdb-tests: Improve test portability/quality
Avoid use of non-portable md5sum by constructing database names using
index.  Improve indentation, use more modern commands, code
improvements (shellcheck).

Signed-off-by: Martin Schwenke <martin@meltin.net>
2020-07-22 02:42:38 +00:00
Martin Schwenke
a308f2534d ctdb-tests: Improve test quality
Simplify code, use more modern commands, code improvements (shellcheck).

Signed-off-by: Martin Schwenke <martin@meltin.net>
2020-07-22 02:42:38 +00:00
Martin Schwenke
1f6556916e ctdb-tests: Improve test portability
"wc -l" on some platforms (e.g. FreeBSD) contains leading spaces, so
strip them.

Signed-off-by: Martin Schwenke <martin@meltin.net>
2020-07-22 02:42:38 +00:00
Martin Schwenke
ea1cbff624 ctdb-tests: Improve test quality
Select test node with IPs instead of using a fixed node.  Remove
unnecessary code, use more modern commands, code
improvements (shellcheck).

Signed-off-by: Martin Schwenke <martin@meltin.net>
2020-07-22 02:42:38 +00:00
Martin Schwenke
1079d6e3ae ctdb-tests: Improve test portability
"wc -l" on some platforms (e.g. FreeBSD) contains leading spaces and
stops "$num from being a number.  Create a more portable solution and
put it in a function instead of repeating the logic.

Signed-off-by: Martin Schwenke <martin@meltin.net>
2020-07-22 02:42:38 +00:00
Martin Schwenke
aa5b214eaa ctdb-tests: Drop uses of "onnode any ..." in testcases
It would be nice to get rid of "onnode any".  There's no use making
tests nondeterministic.  If covering different cases matters then they
should be explicitly handled.

In most places "any" is replaced by "$test_node".  In some cases,
where $test_node is not set, a fixed node that is already used
elsewhere can be reused.

Signed-off-by: Martin Schwenke <martin@meltin.net>
2020-07-22 02:42:38 +00:00
Martin Schwenke
e9df17b500 ctdb-tests: Separate custom cluster startup from test initialisation
Separate cluster startup from test initialisation for tests that start
the cluster with customised configuration.  In these cases the result
of the cluster startup is actually the point of the test.
Additionally, pubips.013.failover_noop.sh claims to have completed
test initialisation twice, which just seems wrong.

The result is:

* ctdb_test_init() takes one option (-n) to indicate when it should
  not configure/start the cluster

* New function ctdb_nodes_start_custom() accepts options for special
  cluster configuration, only operates on local daemons and triggers a
  test failure rather than a test error on failure.

Signed-off-by: Martin Schwenke <martin@meltin.net>
2020-07-22 02:42:38 +00:00
Martin Schwenke
65f56505e2 ctdb-tests: Update preamble for INTEGRATION tests
* Use "#!/usr/bin/env bash" for improved portability

* Drop test_info() definition and replace it with a comment

  The use of test_info() is pointless.

* Drop call to cluster_is_healthy()

  This is a holdover from when the previous test would restart daemons
  to get things ready for a test.  There was also a bug where going
  into recovery during the restart would sometimes cause the cluster
  to become unhealthy.  If we really need something like this then we
  can add it to ctdb_test_init().

* Make order of preamble consistent

Signed-off-by: Martin Schwenke <martin@meltin.net>
2020-07-22 02:42:37 +00:00
Martin Schwenke
c78de201f8 ctdb-tests: Show hung script debugging output
The output in a test failure appears to contain no pstree output
because "00\.test\.script,.*" does not match.  However, this is just a
guess because the output is not shown.

Showing the output makes it easier to understand test failures.

Signed-off-by: Martin Schwenke <martin@meltin.net>
2020-07-22 02:42:37 +00:00
Martin Schwenke
1e55591bc5 ctdb-tests: Add a new fetch ring test that also checks hot keys
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri May 22 08:05:54 UTC 2020 on sn-devel-184
2020-05-22 08:05:54 +00:00
Martin Schwenke
fb38252677 ctdb-tests: Update fetch_ring to take database and key on command line
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-05-22 06:41:45 +00:00
Martin Schwenke
7cff3ed12c ctdb-tests: Use a local "ctdb shutdown" command to avoid a race
When "ctdb shutdown" is run with -n <N> it does not wait for the node
<N>'s ctdbd to go down but exits immediately.  This means that the
local_daemons.sh shutdown command can find the PID file still present
and then attempt the shutdown, but the daemon can have exited between
the check and the shutdown.  Although the test waits until the node is
disconnected, the transport is taken down just before the exit, so
this does not guarantee the daemon has exited.

A local shutdown command (no -n <N>) waits until the socket
disconnects and this happens *after* the PID file is gone, so this is
safe to use with the local_daemons.sh shutdown command.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Mar  2 10:39:28 UTC 2020 on sn-devel-184
2020-03-02 10:39:28 +00:00
Martin Schwenke
dc076b835f ctdb-tests: Rationalise node stop/start/restart
Separate functions are not needed for stopping/starting/restarting
individual nodes.  The stop and start functions essentially just use
onnode, though for local daemons this is embedded in local_daemons.sh.
So, just provide one stop and one start function that takes an
optional nodespec, defaulting to all nodes.

Restarting becomes common.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-02-18 02:56:38 +00:00
Martin Schwenke
bd279d3f98 ctdb-tests: Fix getdbmap test so that it actually works sanely
* Typo in variable name db_map_pattern
* Variable num_db_init used before set
* dbmap_pattern does not cover database flags

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Wed Feb 12 04:38:47 UTC 2020 on sn-devel-184
2020-02-12 04:38:47 +00:00
Martin Schwenke
a6d464aa2e ctdb-tests: Use a here document to improve readability
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-02-12 03:11:39 +00:00
Martin Schwenke
b9f23f5b49 ctdb-tests: Use select_test_node()
select_test_node_and_ips() is not required in these cases.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-02-12 03:11:39 +00:00
Martin Schwenke
14b1dffc27 ctdb-tests: Add some tests to check recovery from recovery lock issues
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2020-01-21 11:39:40 +00:00
Martin Schwenke
6de5706b4d ctdb-tests: Add vacuuming tests
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Thu Oct 24 05:28:21 UTC 2019 on sn-devel-184
2019-10-24 05:28:21 +00:00