IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
As far as we know, nobody uses this and it just complicates the
logging subsystem.
Remove all ringbuffer code and documentation. Update the local
daemons startup code correspondingly.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Volker Lendecke <vl@samba.org>
When ctdb daemon starts up, it considers itself the recovery master
and tries to do first recovery. However, it's possible that there is
already a recovery master and the current node has not yet heard from it.
So do not ban ourselves immediately if ctdb_recovery_lock() fails when
doing first recovery.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This makes it consistent with Samba, to ease transition.
Update unit test code to link to with tdb_wrap instead of including
db_wrap.c.
There are some potential whitespace fixes in this commit that have
been ignored. CTDB's lib/tdb_wrap will be deleted after the
transition to Samba's lib/tdb_wrap, so there's no point polishing it
too much.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This makes it consistent with the rest of the code and avoids problems
when some variant of lib/util isn't in the include path.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Sometimes the recovery daemon fails to get the recovery lock on one
node so that node is banned. This seems to always happen during an
election. The recovery is triggered because other nodes are found to
have recovery mode enabled. They have recovery mode enabled because
an election has been forced.
The recovery daemon's main_loop() only does an initial check for an
election. After that, a node can force an election and, in the
process, set itself to be the current winner. In this situation,
verify_recmode() will always return MONITOR_RECOVERY_NEEDED so
do_recovery() is called. If the previous recovery master hasn't
admitted defeat and released the recovery lock, then do_recovery()
will rightly fail. However, it would be better if it failed a little
more gracefully, since this case is not that unusual.
Instead of trying to take the recovery lock, return early with an
error if there is an election in progress. Note that the race is
still there but it is now much narrower.
There are probably more subtle ways of avoiding this issue, including
something like this in main_loop():
- if (pnn != rec->recmaster) {
+ if (pnn != rec->recmaster || rec->election_timeout) {
return;
}
However, this check is done earlier so it leaves the race window open
a little wider.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Jul 21 06:57:07 CEST 2014 on sn-devel-104
Setting recovery mode to active is the only correct way to inform recovery
daemon to run database recovery. Only freezing databases without setting
recovery mode should not trigger database recovery, as this mechanism
is used in tool to implement wipedb/restoredb commands.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
That makes people think there's a problem (and report bugs) so say
something a bit less scary instead...
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
It is a non-trivial event and will make it easier to debug recovery
lock issues.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is unnecessary since ctdbd_pid is set very early in the code before
creating any other processes including recovery daemon.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Sat Jul 5 09:20:27 CEST 2014 on sn-devel-104
As part of vacuuming, recoverd attaches to databases to migrate records.
When detaching a database from main daemon, it should be removed from
recovery daemon also.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Wed Apr 23 17:05:45 CEST 2014 on sn-devel-104
Database priority is a global property and all the nodes should have the
priority set for the databases. Just setting priority on one node can
lead to problems in the recovery as a database can be frozen at wrong
priority and then freezing database would not succeed.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: David Disseldorp <ddiss@samba.org>
Autobuild-User(master): David Disseldorp <ddiss@samba.org>
Autobuild-Date(master): Mon Apr 7 14:06:26 CEST 2014 on sn-devel-104
Failure might be expected when disabling takeover runs on banned
nodes, since they might be suffering from performance problems or
similar. More broadly, administrators who reconfigure a cluster that
isn't in a happy state aren't necessarily doing something sensible.
However, allowing takeover runs to be disabled on inactive nodes stops
reconfiguration of stopped nodes. This is probaby an unreasonable
limitation, so drop it.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Otherwise new requests can come in during the latter parts of the
takeover run when the IP allocation algorithm has already run, and the
new requests will be dequeued even though they haven't really be
processed.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
Otherwise recovery ends up done by RSN when it is unnecessary.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
When running a mixed version cluster, compatibility with older
versions was was broken during recent refactorisation.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
This also happens earlier in do_recovery() and the nodemap is not
updated after that, so this update is redundant.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Michael Adam <obnox@samba.org>
Rebalance target nodes should be set even if a deferred rebalance is
not configured. The user can explicitly cause a takeover run.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit afd9b51644af074752d74c412cb4e7ec2eba2c69)
This is unnecessary due to 03e2e436db5cfd29a56d13f5d2101e42389bfc94.
Furthermore, if a node doesn't force an election but wins it then it
can fail to record that it is the new recovery master. This can lead
to a reverse split brain where there is no recovery master.
This reverts commit c5035657606283d2e35bea40992505e84ca8e7be.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Conflicts:
server/ctdb_recoverd.c
(This used to be ctdb commit c8b542e059a54b8d524bd430cad9d82e5edd864d)
5 minutes is too long to leave the cluster in limbo if the recovery
daemon dies during a takeover run, even though this is quite unlikely.
We need a new recover master to be able to do takeover runs fairly
quickly.
This reverts commit 71080676bb4acbd0d9b595a30cf7fe6dddbf426f.
(This used to be ctdb commit 3e41170c78fc7a2bf526129c9b7db3739b61c6bf)
It is a 1 line wrapper around ctdb_load_nodes_file(), so use that
instead. We need less code... :-)
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 4a5d5935f4410a93a3343d85a24dbcddae2c4c20)
This reverts commit 4b0f32047e8bece0a052bdbe2209afe91b7e8ce3.
This is not necessary. It just causes a memory leak.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 25fd05505f61dc595c0ef25bb6e332274d5530e8)
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-programmed-with: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 484c46eaae056480baf050fd91868f2fd0537985)
It does cope with node that don't have the lmaster capability.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 588172bcb6bf267339e2bd09e23d2c4904a27a41)
60 seconds might not be long enough to kill all connections and
release IPs.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 71080676bb4acbd0d9b595a30cf7fe6dddbf426f)
Takeover runs are currently silent when they succeed. However, they
are important, so log something by default.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit b39aa2e401fbb581207d986bac93778e9c01acdc)
It can change during a takeover run. If it does then don't free it.
There are potentially fancier solutions (e.g. check what PNNs are new
to the list) to this issue but this is the simplest.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit e81589b7084c661adf617e166cc2c25b4939f841)
The current implementation has a few flaws:
* A takeover run is called unconditionally when the timer goes even if
the recovery master role has moved. This means a node other than
the recovery master can incorrectly do a takeover run.
* The rebalancing target nodes are cleared in the setup for a takeover
run, regardless of whether the takeover run succeeds.
* The timer to force a rebalance isn't cleared if another takeover run
occurs before the deadline. Any forced rebalancing will happen in
the first takeover run and when the timer expires some time later
then an unnecessary takeover run will occur.
* If the recovery master role moves then the rebalancing data will
stay on the original node and affect the next takeover run to occur
if the recovery master role should come back to the original node.
Instead, store an array of rebalance target nodes in the recovery
master context. This is passed as an extra argument to
ctdb_takeover_run() each time it is called and is cleared when a
takeover run succeeds. The timer hangs off the array of rebalance
target nodes, which is cleared if the node isn't the recovery master.
This means that it is possible to lose rebalance data if the recovery
master role moves. However, that's a difficult problem to solve. The
best way of approaching it is probably to try to stop the recovery
master role from jumping around unnecesarily when inactive nodes join
the cluster.
The long term solution is to avoid this nonsense completely. The IP
allocation algorithm needs to cache state between runs so that it
knows which nodes have just become healthy. This also needs recovery
master stability.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit c51c1efe5fc7fa668597f2acd435dee16e410fc9)
The takeover run will fail anyway but deferring seems like a cleaner
option.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 428f800bcdf3dbfe19de8bb36099fbf01ebeaab4)
Use disable_takeover_runs_handler() instead of maintaining duplicate
logic.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 0a51a85915486b2a8fded7ba6444b18c6c1ee8e8)
This implements a superset of CTDB_SRVID_DISABLE_IP_CHECK. It stops
the IP checks but also causes any attempted takeover runs to fail and
be rescheduled.
This is meant to completely stop IP movements.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 00db4de53a0d86013e79e6577e7e6cf3ef864e56)
The code that handles IP reallocate requests can be reused.
This also changes the result back to a SRVID caller to the PNN on
success or a negative error code on failure. None of the callers
currently look at the result so this is harmless... but it will be
useful later.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit e4eae6e3291baa299a1d0f733ab11b138ee699a3)
No need for a separate one for each SRVID.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit d9c22b04d5aa7938a3965bd3144568664eb772ce)
Nested takeover runs should never happens so they should fail.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 8ed29c60c0a7dd29f2a6efdf694d38e94281e1c4)
It is set on every failure anyway.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit e5f94c7857405bdeac233069003c3769b3dc3616)
Factor the calling sequence for ctdb_takeover_run() into a new
function and call it instead. This changes rec->need_takeover_run to
false for each successful takeover run and that seems to be the right
thing to do.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 9a3f0c0e61ca5c17e020c6e0463d73c7cf4f7c09)
On rare occasions when a node that has been inactive it will trigger
an election when it becomes active again. If that node has been up
for the longest then it will win the election and the recovery master
role will spuriously move.
While a node remains inactive we reset the priority time to discourage
it from winning elections. The priority time will now reflect roughly
how long the node has been active rather than how long it has been up.
That means the most stable node is more likely to win elections.
Having a stable recovery master means that disabling takeover runs
while reloading IPs is more likely to succeed. It also improves the
chances of being able to cache information in the recovery master -
for example, between takeover runs.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit f0f48f22f45e4c82eba2582efae307e25385de81)
This reverts commit 10a057d8e15c8c18e540598a940d3548c731b0b4.
This approach would not work when creating local databases since currently
there is no control to receive TDB flags for remote databases.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit ca61eb776ab862bd269e45ee0f9f96e7e1e0e001)
When creating missing databases either locally or remotely, make sure
to use the correct tdb flags from other nodes. Without this, volatile
databases can get attached without TDB_INCOMPATIBLE_HASH flag.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 10a057d8e15c8c18e540598a940d3548c731b0b4)
Currently nodemap used by recovery master is the one obtained from the local
node. This information may have been updated while processing main loop.
Before comparing node flags on all the nodes, create up-to-date node flags
information based on the information received from all the nodes.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit fcf77dec5af973a0e32f3999bc012053a6f47a96)
