IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
The logic currently in ctdb_ctrl_modflags() will be optimised so that
it no longer matches the pattern for a control function. So, remove
this function and squash its functionality into the only caller.
Although there are some superficial changes, the behaviour is
unchanged.
Flattening the 2 functions produces some seriously weird logic for
setting the new flags, to the point where using ctdb_ctrl_modflags()
for this purpose now looks very strange. The weirdness will be
cleaned up in a subsequent commit.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is clearer than using the MODFLAGS control directly.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This makes fields such as recmaster and nodemap easily available if
required.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
An unused argument needlessly extends the length of function calls. A
subsequent change will allow rec->nodemap to be used if necessary.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Avoid use of non-portable md5sum by constructing database names using
index. Improve indentation, use more modern commands, code
improvements (shellcheck).
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Wed Jul 22 09:14:35 UTC 2020 on sn-devel-184
Simplify code, use more modern commands, code improvements (shellcheck).
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
"wc -l" on some platforms (e.g. FreeBSD) contains leading spaces, so
strip them.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Select test node with IPs instead of using a fixed node. Remove
unnecessary code, use more modern commands, code
improvements (shellcheck).
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
"wc -l" on some platforms (e.g. FreeBSD) contains leading spaces and
stops "$num from being a number. Create a more portable solution and
put it in a function instead of repeating the logic.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
It would be nice to get rid of "onnode any". There's no use making
tests nondeterministic. If covering different cases matters then they
should be explicitly handled.
In most places "any" is replaced by "$test_node". In some cases,
where $test_node is not set, a fixed node that is already used
elsewhere can be reused.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Separate cluster startup from test initialisation for tests that start
the cluster with customised configuration. In these cases the result
of the cluster startup is actually the point of the test.
Additionally, pubips.013.failover_noop.sh claims to have completed
test initialisation twice, which just seems wrong.
The result is:
* ctdb_test_init() takes one option (-n) to indicate when it should
not configure/start the cluster
* New function ctdb_nodes_start_custom() accepts options for special
cluster configuration, only operates on local daemons and triggers a
test failure rather than a test error on failure.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The only caller calls ctdb_test_error() on failure and nesting this
calls can be confusing. A future change will make this even more
confusing.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Mostly avoidance of quoting warnings.
Silencing warnings about unquoted $CTDB_TEST_CAT_RESULTS_OPTS is
handled by passing '-' to cat when that variable's value is empty.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Apart from the non-constant sourcing of include files.
Mostly avoidance of quoting warnings.
One subtle change is to simply pass "120" to wait_until_ready() to
stop warnings that it expects arguments but none are passed (both
SC2119 and SC2120). There seems no way to indicate to structure
function argument handling so that shellcheck realises arguments are
optional. In later shellcheck versions, disabling SC2120 for a
function also silences complaints about its callers... but not all of
our testing uses "later" shellcheck versions.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
* Use "#!/usr/bin/env bash" for improved portability
* Drop test_info() definition and replace it with a comment
The use of test_info() is pointless.
* Drop call to cluster_is_healthy()
This is a holdover from when the previous test would restart daemons
to get things ready for a test. There was also a bug where going
into recovery during the restart would sometimes cause the cluster
to become unhealthy. If we really need something like this then we
can add it to ctdb_test_init().
* Make order of preamble consistent
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Avoid:
.../UNIT/shellcheck/scripts/local.sh: line 14: type: shellcheck: not found
The "type" command in dash prints the "not found" message to stdout
but the bash version prints to stderr, so redirect stderr too.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The output in a test failure appears to contain no pstree output
because "00\.test\.script,.*" does not match. However, this is just a
guess because the output is not shown.
Showing the output makes it easier to understand test failures.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This will allow local daemons to be used in more contexts, especially
in tests run by Jenkins where the directory names for some targets can
be very long.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The standalone build still includes tests, as does the top-level build
when --enable-selftest is used. The latter is consistent with the use
of --enable-selftest in the rest of the tree.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
In certain circumstance, which aren't obvious, cat(1) can fail when
attempting to write a lot of data. This is due to something (probably
write(2)) returning EAGAIN.
Given that the -v option should only really be used for test
debugging, ignore the failure instead of spending time debugging it.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14446
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
In certain circumstance, which aren't obvious, cat(1) can fail when
attempting to write a lot of data. This is due to something (probably
write(2)) returning EAGAIN.
Given that the -v option should only really be used for test
debugging, ignore the failure instead of spending time debugging it.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14446
Signed-off-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Wed Jul 22 04:10:47 UTC 2020 on sn-devel-184
Avoid use of non-portable md5sum by constructing database names using
index. Improve indentation, use more modern commands, code
improvements (shellcheck).
Signed-off-by: Martin Schwenke <martin@meltin.net>
Select test node with IPs instead of using a fixed node. Remove
unnecessary code, use more modern commands, code
improvements (shellcheck).
Signed-off-by: Martin Schwenke <martin@meltin.net>
"wc -l" on some platforms (e.g. FreeBSD) contains leading spaces and
stops "$num from being a number. Create a more portable solution and
put it in a function instead of repeating the logic.
Signed-off-by: Martin Schwenke <martin@meltin.net>
It would be nice to get rid of "onnode any". There's no use making
tests nondeterministic. If covering different cases matters then they
should be explicitly handled.
In most places "any" is replaced by "$test_node". In some cases,
where $test_node is not set, a fixed node that is already used
elsewhere can be reused.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Separate cluster startup from test initialisation for tests that start
the cluster with customised configuration. In these cases the result
of the cluster startup is actually the point of the test.
Additionally, pubips.013.failover_noop.sh claims to have completed
test initialisation twice, which just seems wrong.
The result is:
* ctdb_test_init() takes one option (-n) to indicate when it should
not configure/start the cluster
* New function ctdb_nodes_start_custom() accepts options for special
cluster configuration, only operates on local daemons and triggers a
test failure rather than a test error on failure.
Signed-off-by: Martin Schwenke <martin@meltin.net>
The only caller calls ctdb_test_error() on failure and nesting this
calls can be confusing. A future change will make this even more
confusing.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Mostly avoidance of quoting warnings.
Silencing warnings about unquoted $CTDB_TEST_CAT_RESULTS_OPTS is
handled by passing '-' to cat when that variable's value is empty.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Apart from the non-constant sourcing of include files.
Mostly avoidance of quoting warnings.
One subtle change is to simply pass "120" to wait_until_ready() to
stop warnings that it expects arguments but none are passed (both
SC2119 and SC2120). There seems no way to indicate to structure
function argument handling so that shellcheck realises arguments are
optional. In later shellcheck versions, disabling SC2120 for a
function also silences complaints about its callers... but not all of
our testing uses "later" shellcheck versions.
Signed-off-by: Martin Schwenke <martin@meltin.net>
* Use "#!/usr/bin/env bash" for improved portability
* Drop test_info() definition and replace it with a comment
The use of test_info() is pointless.
* Drop call to cluster_is_healthy()
This is a holdover from when the previous test would restart daemons
to get things ready for a test. There was also a bug where going
into recovery during the restart would sometimes cause the cluster
to become unhealthy. If we really need something like this then we
can add it to ctdb_test_init().
* Make order of preamble consistent
Signed-off-by: Martin Schwenke <martin@meltin.net>
Avoid:
.../UNIT/shellcheck/scripts/local.sh: line 14: type: shellcheck: not found
The "type" command in dash prints the "not found" message to stdout
but the bash version prints to stderr, so redirect stderr too.
Signed-off-by: Martin Schwenke <martin@meltin.net>
The output in a test failure appears to contain no pstree output
because "00\.test\.script,.*" does not match. However, this is just a
guess because the output is not shown.
Showing the output makes it easier to understand test failures.
Signed-off-by: Martin Schwenke <martin@meltin.net>
This will allow local daemons to be used in more contexts, especially
in tests run by Jenkins where the directory names for some targets can
be very long.
Signed-off-by: Martin Schwenke <martin@meltin.net>
The standalone build still includes tests, as does the top-level build
when --enable-selftest is used. The latter is consistent with the use
of --enable-selftest in the rest of the tree.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Drop some unnecessary whitespace and re-indent push().
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Options can be set in ONNODE_SSH, so this variable is unnecessary.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The current code only ever swaps with slot 0. This will only ever
happen with slots 0 and 1, so probably never sorts.
Replace with qsort().
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
ctdbd currently only logs when a new hot key is added. If a key gets
hotter then nothing new is logged.
Log hot key updates when the number of migrations has doubled since
the last time that key was logged.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This message indicates that a hot key was added, so say that. After
all the hot key slots have been filled the id will always be 0, so
don't bother logging it.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
These should be unsigned but luck is currently on our side.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
There are 2 reasons for this. Sorting of hot keys is broken and will
be changed to an implementation that needs a named (i.e. not
anonymous) structure. Also, at least one non-protocol field will be
added to facilitate more useful logging.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Testing control: 4 bytes msec delay plus a blob, return the request after the
delay. This is an enhanced "ping" which can be used to test asynchronous
clients.
Doesn't have the full protocol implementation yet
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
On debian buster, this variable doesn't exist anymore. Look at this PR
as a reference:
https://github.com/gluster/storhaug/pull/30
Signed-off-by: Renaud Fortier <renaud.fortier@fsaa.ulaval.ca>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Apr 23 08:07:51 UTC 2020 on sn-devel-184
The vacuuming integration tests set VacuumInterval to a very high
number to avoid vacuuming collisions. This is done after the cluster
is healthy, so Samba will have already been started and vacuuming will
already be scheduled *at the default interval* for databases attached
by Samba. This means that vacuuming controls used by vacuuming tests
can still collide with the scheduled vacuuming events.
Add some logic to reschedule a vacuuming event that has fired but
where VacuumInterval has increased since it was originally scheduled.
The increase in VacuumInterval is used as the time offset for
rescheduling the event.
Although this changes production behaviour for the convenience of
testing, the new behaviour is completely reasonable and obeys the
principle of least surprise.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Apr 7 03:04:57 UTC 2020 on sn-devel-184
No behaviour change. This is final staging to make the next change
completely obvious.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
No behaviour change. This just makes future changes clearer by
avoiding reformatting (or introducing local variables).
Clean up error handling while touching a relevant line.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Inside the nested event loop in ctdb_ctrl_getnodemap(), various
asynchronous handlers may dereference rec->nodemap, which will be
NULL.
One example is lost_reclock_handler(), which causes rec->nodemap to be
unconditionally dereferenced in list_of_nodes() via this call chain:
list_of_nodes()
list_of_active_nodes()
set_recovery_mode()
force_election()
lost_reclock_handler()
Instead of attempting to trace all of the cases, just avoid leaving
rec->nodemap set to NULL. Attempting to use an old value is generally
harmless, especially since it will be the same as the new value in
most cases.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14324
Reported-by: Volker Lendecke <vl@samba.org>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Tue Mar 24 01:22:45 UTC 2020 on sn-devel-184
Neither the recovery daemon nor the recovery helper should attach
databases outside of the recovery process.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14294
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This builds a complete list of databases across the cluster so it can
be used to create databases on the nodes where they are missing.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14294
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This will be used to build a merged list of databases from all nodes,
allowing the recovery helper to create missing databases.
It would be possible to also include the db_name field in this
structure but that would cause a lot of churn. This field is used
locally in the recovery of each database so can continue to live in
the relevant state structure(s).
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14294
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is currently only set by the recovery daemon when it attaches
missing databases, so there is no obvious behaviour change. However,
attaching missing databases can now be moved to the recovery helper as
long as it sets this flag.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14294
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
ctdb_ctrl_createdb() is only called by the recovery daemon, so this is
a safe, temporary change. This is temporary because
ctdb_ctrl_createdb(), create_missing_remote_databases() and
create_missing_local_databases() will all go away soon.
Note that this doesn't cause a change in behaviour. The main daemon
will still only defer attaches from non-recoverd processes during
recovery.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14294
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Commit 3cc230b5ee says:
Dont allow clients to connect to databases untile we are well past
and through the initial recovery phase
It is unclear what this commit was attempting to do. The commit
message implies that more attaches should be deferred but the code
change adds a conjunction that causes less attaches to be deferred.
In particular, no attaches will be deferred after startup is complete.
This seems wrong.
To implement what seems to be stated in the commit message an "or"
needs to be used so that non-recovery daemon attaches are deferred
either when in recovery or before startup is complete. Making this
change highlights that attaches need to be allowed during the
"startup" event because this is when smbd is started.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14294
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If a node is marked for banning, confirm that it's not become inactive
during the recovery. If yes, then don't ban the node.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14294
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
It's possible to have a node stopped, but recovery master not yet
updated flags on the local ctdb daemon when recovery is started. So do
not trust the list of active nodes obtained from the local node. Query
the connected nodes to calculate the list of active nodes.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14294
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
New vnnmap is constructed using the information from all the connected
nodes. So there is no need to fetch the vnnmap from recovery master.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14294
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
The only place the outgoing connection needs to be stopped is when
there is a timeout when waiting for the connection to become writable.
Add a new function ctdb_tcp_node_connect_timeout() to handle this
case.
All of the other cases are attempts to establish a new outgoing
connection (initial attempt, retry after an error or disconnect, ...)
so drop stopping the connection in those cases.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14295
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Mar 12 05:29:20 UTC 2020 on sn-devel-184
No change in behaviour. This makes the code self-documenting.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14295
Signed-off-by: Ralph Boehme <slow@samba.org>
Signed-off-by: Martin Schwenke <martin@meltin.net>
No change in behaviour. This makes the code self-documenting.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14295
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
The node dead upcall has already restarted the outgoing connection.
There's no need to repeat it.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14295
Signed-off-by: Ralph Boehme <slow@samba.org>
Signed-off-by: Martin Schwenke <martin@meltin.net>
ctdb_tcp_tnode_cb() is called when we receive data on the outgoing connection.
This can happen when we get an EOF on the connection because the other side as
closed. In this case data will be NULL.
It would also be called if we received data from the peer. In this case data
will not be NULL.
The latter case is a fatal error though and we already call
ctdb_tcp_stop_connection() for this case as well, which means even though the
node is not fully connected anymore, by not calling the node_dead() upcall
NODE_FLAGS_DISCONNECTED will not be set.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14295
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Since commit 77deaadca8, a nodeA which
had previously accepted a connection from nodeB (where nodeB dies
e.g. as as result of fencing) when nodeB attempts to connect again
after restarting is always rejected with
ctdb_listen_event: Incoming queue active, rejecting connection from w.x.y.z
messages.
Consolidate dead node handling in the TCP restart handling.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14295
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
If NODE_FLAGS_DISCONNECTED is set the node can be in half-connected state. With
this change we ensure to restart the transport for this case.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14295
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Better compatibility, since od output isn't consistent on FreeBSD.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Mar 10 09:17:12 UTC 2020 on sn-devel-184
When the tests are run interactively this will make it more noticeable
that shellcheck is not installed because the test summary will
indicate missing tests.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Skipped tests return a status that indicates failure. In combination
with the -e option this results in an exit with failure on the first
skipped test.
Convert skipped test status to success. The skip has already been
counted.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Christof Schmitt <cs@samba.org>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Mon Mar 9 02:27:18 UTC 2020 on sn-devel-184
When "ctdb shutdown" is run with -n <N> it does not wait for the node
<N>'s ctdbd to go down but exits immediately. This means that the
local_daemons.sh shutdown command can find the PID file still present
and then attempt the shutdown, but the daemon can have exited between
the check and the shutdown. Although the test waits until the node is
disconnected, the transport is taken down just before the exit, so
this does not guarantee the daemon has exited.
A local shutdown command (no -n <N>) waits until the socket
disconnects and this happens *after* the PID file is gone, so this is
safe to use with the local_daemons.sh shutdown command.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Mar 2 10:39:28 UTC 2020 on sn-devel-184
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Sat Feb 29 11:53:42 UTC 2020 on sn-devel-184
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Sat Feb 29 08:02:50 UTC 2020 on sn-devel-184
Signed-off-by: Anoop C S <anoopcs@redhat.com>
Reviewed-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Feb 27 07:34:53 UTC 2020 on sn-devel-184
Till now 50.samba script was based on RHEL versions <=6 where we didn't
have separate start up script for nmb and smbd used to start nmbd when
required. Now that nmbd has its own start up script named "nmb" it is
reasonable to have "nmb" as default value for CTDB_SERVICE_NMB inside
new 48.netbios ctdb script.
Signed-off-by: Anoop C S <anoopcs@redhat.com>
Reviewed-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This change basically moves out nmbd references from 50.samba script to
a new 48.netbios script. Accordingly ctdb test scripts are tweaked to
cope with newly added script.
Signed-off-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
The example output doesn't tell a user what command generated it.
Adding the command makes the examples much more useful.
Reported-by: Stefan Kania <stefan@kania-online.de>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Feb 18 04:22:56 UTC 2020 on sn-devel-184
This depends on a small amount of internal knowledge but is the
cleanest way of avoiding errors for nodes that have already been shut
down.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Separate functions are not needed for stopping/starting/restarting
individual nodes. The stop and start functions essentially just use
onnode, though for local daemons this is embedded in local_daemons.sh.
So, just provide one stop and one start function that takes an
optional nodespec, defaulting to all nodes.
Restarting becomes common.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
csbuild says:
ctdb/server/ctdb_lock.c: scope_hint: In function ‘ctdb_find_lock_context’
ctdb/server/ctdb_lock.c:671:33: warning: comparison of integer expressions of different signedness: ‘int’ and ‘uint32_t’ {aka ‘unsigned int’} [-Wsign-compare]
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
* Typo in variable name db_map_pattern
* Variable num_db_init used before set
* dbmap_pattern does not cover database flags
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Wed Feb 12 04:38:47 UTC 2020 on sn-devel-184
Shellcheck noticed that pnn was never referenced. Not sure this ever
worked or whether it got broken somewhere along the way.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
select_test_node_and_ips() is not required in these cases.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If we can't bind the local end of an outgoing connection then
something has gone wrong. Retrying is better than failing into a
zombie state. The interface might come back up and/or the address my
be reconfigured.
While here, do the same thing for the other (potentially transient)
failures.
The unknown address family failure is special but just handle it via a
retry. Technically it can't happen because the node address parsing
can only return values with address family AF_INET or AF_INET6.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14274
Reported-by: 耿纪超 <gengjichao@jd.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If ctdbd hangs when shutting down in post-test clean-up then killing
the process group can kill the test. When in test mode, create a
process group but only in the top-level ctdb tool - the natgw and lvs
helpers also run the ctdb tool.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
When running tests from a top-level build, a stale build in ctdb/bin/
will be preferred and may cause confusing results.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is used to set several variables so it might as well be cd-proof.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
There is no sane way of keeping stdin open when using the shell to
background ctdbd in local_daemons.sh. Instead, have ctdbd fork when
not interactive and when test mode is enabled. become_daemon() can't
be used for this: if it forks then it also closes stdin.
For the interactive case, become_daemon() wasn't doing anything
special, so do nothing instead.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
These don't need to depend on do_fork. Child logging should be set up
whenever the daemon is not interactive. The stdin handler should be
setup whenever test mode is enabled.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
No functional changes.
This is staging for a change that makes ctdbd fork when test mode is
enabled but interactive is not set.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
A subsequent file descriptor allocation may return 0 and unexpected
things may then happen.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Otherwise, if the test is run via ssh it will "unexpectedly" find
itself at the other end of a pipe.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This doesn't work when stdin is not a tty.
This reverts commit ea754bfdec.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Jan 30 13:53:22 UTC 2020 on sn-devel-184
POSIX says:
If job control is disabled (see set, -m), the standard input for an
asynchronous list, before any explicit redirections are performed,
shall be considered to be assigned to a file that has the same
properties as /dev/null. This shall not happen if job control is
enabled. In all cases, explicit redirection of standard input shall
override this activity.
ctdbd is backgrounded at startup, so the above causes stdin to be
redirected from /dev/null. Enable job control to work around this.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Tue Jan 28 11:24:35 UTC 2020 on sn-devel-184
This allows a test environment to simply close its end of a pipe to
cleanly shutdown ctdbd. Like in smbd, this is only done if stdin is a
pipe or a socket.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This avoids a crash if ctdb_shutdown_sequence() is called before
monitoring is initialised.
Switch to using TALLOC_FREE() while touching this function.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Testing against a commonly used cluster filesystem has shown no
performance impact, as expected.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This makes it more like the way it works with a cluster filesystem.
It also allows the subdirectory to be manipulated in tests.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
See the comments added to the tests.
It may be possible to rewrite these so they do something sane for
IPv6... some other time.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14227
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Jan 3 00:00:55 UTC 2020 on sn-devel-184
ss added square brackets around IPv6 addresses in versions > 4.12.0
via commit aba9c23a6e1cb134840c998df14888dca469a485. CentOS 7 added
this feature somewhere mid-release. So, backward compatibility is
obviously needed.
As per the comment protocol/protocol_util.c should probably print and
parse such square brackets. However, for backward compatibility the
brackets would have to be stripped in both places in
update_tickles()... or added to the ss output when missing. Best to
leave this until we have a connection tracking daemon.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14227
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Nov 14 12:03:46 UTC 2019 on sn-devel-184
If any of the option parsing or command parsing fails, generate usage
message.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
The type-checking is superfluous and gets in the way of readability.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Thu Nov 14 03:45:44 UTC 2019 on sn-devel-184
Commit c68b6f96f2 changed the talloc hierarchy such that outgoing TCP sockets
while sitting in the async connect() syscall are not freed via
ctdb_tcp_shutdown() anymore, they are hanging off a longer-running structure.
Free this structure as well.
If an outgoing TCP socket leaks into a long-running child process (possibly the
recovery daemon), this connection will never be closed as seen by the
destination node. Because with recent changes incoming connections will not be
accepted as long as any incoming connection is alive, with that socket leak
into the recovery daemon we will never again be able to successfully connect to
the node that is affected by this leak. Further attempts to connect will be
discarded by the destination as long as the recovery daemon keeps this socket
alive.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14175
RN: Avoid communication breakdown on node reconnect
Signed-off-by: Martin Schwenke <martin@meltin.net>
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Wed Nov 13 13:31:10 UTC 2019 on sn-devel-184
We have a macro for NULLing out the pointer
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Nov 8 01:35:11 UTC 2019 on sn-devel-184
This can fail as follows:
--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--
Running test ./tests/UNIT/tool/ctdb.process-exists.003.sh (02:26:30)
--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--
ctdb.process-exists.003 - ctdbd process with multiple connections on node 0
Setting up fake ctdbd
<10||0|
OK
<10|PID 26107 exists
|0|
OK
==================================================
Running "ctdb -d NOTICE process-exists 26107 0x1234567812345678"
PASSED
==================================================
Running "ctdb -d NOTICE process-exists 26107 0xaebbccdd12345678"
Registered SRVID 0xaebbccdd12345678
--------------------------------------------------
Output (Exit status: 1):
--------------------------------------------------
PID 26107 with SRVID 0xaebbccdd12345678 does not exist
--------------------------------------------------
Required output (Exit status: 0):
--------------------------------------------------
PID 26107 with SRVID 0xaebbccdd12345678 exists
FAILED
connection to daemon closed, exiting
==========================================================================
TEST FAILED: ./tests/UNIT/tool/ctdb.process-exists.003.sh (status 1) (duration: 0s)
==========================================================================
This happens when dummy_client has not registered the SRVID (for its
10th connection) before the 2nd simple_test.
Change the initial wait to ensure that the SRVID is registered.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Wed Nov 6 02:46:24 UTC 2019 on sn-devel-184
Improve quoting and indentation. Print a clear error if the cluster
goes back into recovery and doesn't come back out.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Retrying like this hides bugs. The cluster should come up first time,
every time.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This file descriptor is owned by the incoming queue. It will be
closed when the queue is torn down.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14175
RN: Avoid communication breakdown on node reconnect
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
CTDB's incoming queue handling does not check whether an existing
queue exists, so can overwrite the pointer to the queue. This used to
be harmless until commit c68b6f96f2
changed the read callback to use a parent structure as the callback
data. Instead of cleaning up an orphaned queue on disconnect, as
before, this will now free the new queue.
At first glance it doesn't seem possible that 2 incoming connections
from the same node could be processed before the intervening
disconnect. However, the incoming connections and disconnect occur on
different file descriptors. The queue can become orphaned on node A
when the following sequence occurs:
1. Node A comes up
2. Node A accepts an incoming connection from node B
3. Node B processes a timeout before noticing that outgoing the queue is writable
4. Node B tears down the outgoing connection to node A
5. Node B initiates a new connection to node A
6. Node A accepts an incoming connection from node B
Node A processes then the disconnect of the old incoming connection
from (2) but tears down the new incoming connection from (6). This
then occurs until the originally affected node is restarted.
However, due to the number of outgoing connection attempts and
associated teardowns, this induces the same behaviour on the
corresponding incoming queue on all nodes that node A attempts to
connect to. Therefore, other nodes become affected and need to be
restarted too.
As a result, the whole cluster probably needs to be restarted to
recover from this situation.
The problem can occur any time CTDB is started on a node.
The fix is to avoid accepting new incoming connections when a queue
for incoming connections is already present. The connecting node will
simply retry establishing its outgoing connection.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14175
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This makes it consistent with the reverse case. Also, in_fd will soon
be removed.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14175
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
