IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Instead of includes.h, include the required header files explicitly.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This groups function prototypes for system specific functions in
common/system.h and removes them from ctdb_private.h.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Just use ctdb_tcp_connection. It is the same. There are no external
users.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Volker Lendecke <vl@samba.org>
Some code involved in releasing IPs is not re-entrant. Memory
corruption can occur if, for example, overlapping attempts are made to
ban a node. We haven't been able to recreate the corruption but this
should protect against it.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This needs to be cleared to avoid stale data when a new recovery
master is elected.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If DisableIPFailover is set then something else may be managing public
IP addresses so CTDB should leave them alone.
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
There won't be an IP tree. It is only ever initialised during a
takeover run.
The alternate to this would be to avoid sending
CTDB_SRVID_RECD_UPDATE_IP in "ctdb moveip". This logic is probably
best kept out of the CLI tool.
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The potential for public IP addresses to shuffle around during node
initialisation disappeared a while ago because IP addresses can only
be assigned to a node that is in CTDB_RUNSTATE_RUNNING. This means
that interfaces might as well just be initialised as "up". If any
interfaces are actually "down" then this will be rectified by the
"startup" event in 10.interfaces.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
To support external failover of IP addresses if DisableIPFailover is
set. CTDB's idea of IP address assignment can be manipulated using
"ctdb moveip". Checking if the IP address is already held breaks
this in several places.
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
ctdb_sys_have_ip() should only be run if if do_publicipcheck is set.
This is clearer if written as 2 nested if-statements rather than as a
lazy conjuction.
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is done by 10.interace where the monitor event fails when there
is a missing interface. The in-daemon interface checking adds no
value.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If all nodes are still in, say, FIRST_RECOVERY runstate, then the logs
contain unfortunate noise like:
recoverd:Failed to find node to cover ip 10.0.2.131
This avoids that by adding an early exit that avoids running
takeover_run_core() when there are no nodes in the
CTDB_RUNSTATE_RUNNING.
To support this add the runstate to the ipflags structure. There are
clearly other ways of hacking this but this seems the simplest.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Due to usage of CTDB_NO_MEMORY macro,
some of the resources are not freed in failure cases.
Signed-off-by: Rajesh Joseph <rjoseph@redhat.com>
Reviewed-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Autobuild-User(master): Günther Deschner <gd@samba.org>
Autobuild-Date(master): Fri Apr 17 16:49:05 CEST 2015 on sn-devel-104
ctdb_sys_find_ifname() doesn't work for IPv6 addresses so don't use
it.
Trust the eventscript to do sanity checking on the interface. Current
warnings are replaced with equivalents generated by the eventscript.
The unlikely message:
Public IP %s is hosted on interface %s but we have no VNN
will be replaced by:
WARNING: Public IP %s hosted on interface %s but VNN says __none__
which is clear enough.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is part of a migration to Samba's lib/util. CTDB always passes 0
(i.e. no max_size) so use a simple assert() to enforce this, rather
than changing a lot of code that will be discarded anyway.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This was useful for debugging the race fixed by commit
4f79fa6c7c. It might be useful again.
Also fix a nearby comment typo.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Jun 20 02:07:48 CEST 2014 on sn-devel-104
It might as well be near where it is used. Add a comment explaining
it.
Also add/update comments at the top of the RELEASE_IP and TAKEOVER_IP
loops to explain what is happening.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon May 5 06:20:39 CEST 2014 on sn-devel-104
Signed-off-by: Gregor Beck <gbeck@sernet.de>
Reviewed-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Tue Apr 1 02:59:05 CEST 2014 on sn-devel-104
Previous commits maintained the ordering between
ctdb_remove_orphaned_ifaces() and ctdb_vnn_unassign_iface(). This
meant that ctdb_remove_orphaned_ifaces() needed to steal the orphaned
interfaces and they would be freed later.
Unassign the interface first and things get simpler.
ctdb_remove_orphaned_ifaces() is now self-contained.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Sun Mar 23 06:20:43 CET 2014 on sn-devel-104
reloadips really expects deleted IPs to be released before completing.
Otherwise the recovery daemon starts failing the local IP check. The
races that follow can cause a node to be banned.
To make the error handling simple, do the actual deletion in
release_ip_callback().
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Commit 0723fedced added a cheap
implemention of ctdb_control_startup() that simply flags the recipient
node as needing to send updates for each IP when the tickle update
loop next fires. Commit 026996550d
ensures that a node only sends tickle updates once being flagged to do
so.
CTDB_CONTROL_STARTUP is broadcast to all nodes, so this is a good
start. However, the tickle updates are only broadcast to connected
nodes. A recently started node may not yet be considered to be
connected because the keepalive monitoring loop may not yet have
marked the node as connected. This means that the tickle update loop
races with the keepalive monitoring loop. If the tickle update loop
wins then updates will not be sent to the recently started node.
The simplest improvement is to stop the tickle update from depending
on whether a node is connected or not. So instead of broadcasting
tickle updates to connected nodes, they are broadcast to all nodes.
Since no reply is expected, this should work just fine.
While looking at this code, ctdb_ctrl_set_tcp_tickles() is named like
a client function. It isn't a client function. Also, 2 of the
arguments are ignored. So rename this function to
ctdb_send_set_tcp_tickles_for_ip() and remove the ignored arguments.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
CTDB tracks connections to be able to send tickle ACKs and gratuitous
ARPs. When there are no public IPs, there is no need for tickle ACKs
and gratuitous ARPs.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Tue Mar 4 03:01:38 CET 2014 on sn-devel-104
Fix suggested by by Kevin Osborn <kosborn@overlandstorage.com>.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Feb 27 13:54:59 CET 2014 on sn-devel-104
tcp_update_flag is set to true whenever tickles are added or deleted.
This flag is used to determine whether or not to send tickles list to
other nodes. Once tickles list is sent to other nodes successfully,
set tcp_update_flag to false, so ctdbd does not keep sending same tickles
list every TickleUpdateInterval (20 seconds).
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This doesn't implement what was recommended. That would require
careful error handling, probably with a fallback to this code anyway.
This is simple and does no worse that the current code. That is, the
new node is updated on the next call to tdb_update_tcp_tickles().
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This fixes ctdb crash reported in bug #10366.
Fix suggested by Kevin Osborn <kosborn@overlandstorage.com>.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
* Remove unnecessary candimbl parameter.
This parameter can be cheaply calculated in
lcp2_failback_candidate(). The compiler will probably do an
excellent job optimising it. :-)
* Clarify a debug statement
This is much clearer than doing a complex recalculation of a known
value.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Currently this can be checked many times. However, there's no point
calling the rebalance/failback code at all if there are no rebalance
candidates.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
srcimbl gets changed on every iteration of the loop. The value that
should be stored for the new imbalance of the source node is
minsrcimbl.
To help diagnose this, added some extra debug that can be left in.
The extra debug changes the output of a couple of tests. Note that
the resulting IP allocations in those tests is unchanged - only the
debug output is changed.
Also add some new tests that illustrates the bug.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Currently timeouts for controls to inactive nodes can cause banning
credits to be applied. This should not happen.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This was added to support external monitoring using CTDB event scripts.
However, it was never used.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Also get rid of ctdb_set_event_script_dir(). It creates an
unnecessary copy of something that will be around for the lifetime of
the process.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 21b4d1aba00902f1eee0cbf4f082b0794fd5b738)
Otherwise, if existing IPs are added to extra nodes (that have,
perhaps, been disconnected) then those IPs will not be rebalanced
across the extra nodes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit ceb30432a9a550778aed0b422a654fc5287b82a3)
Deleting IPs can take a while because IPs are released and connections
are killed. This can take a while so do them in parallel. In fact,
since the set of IPs being added and deleted will be disjoint, send
all the adds/deletes at the same time and then wait.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 85a5b544ec032173e98c9cc3b5402a76b961aa3b)
The current implementation has a few flaws:
* A takeover run is called unconditionally when the timer goes even if
the recovery master role has moved. This means a node other than
the recovery master can incorrectly do a takeover run.
* The rebalancing target nodes are cleared in the setup for a takeover
run, regardless of whether the takeover run succeeds.
* The timer to force a rebalance isn't cleared if another takeover run
occurs before the deadline. Any forced rebalancing will happen in
the first takeover run and when the timer expires some time later
then an unnecessary takeover run will occur.
* If the recovery master role moves then the rebalancing data will
stay on the original node and affect the next takeover run to occur
if the recovery master role should come back to the original node.
Instead, store an array of rebalance target nodes in the recovery
master context. This is passed as an extra argument to
ctdb_takeover_run() each time it is called and is cleared when a
takeover run succeeds. The timer hangs off the array of rebalance
target nodes, which is cleared if the node isn't the recovery master.
This means that it is possible to lose rebalance data if the recovery
master role moves. However, that's a difficult problem to solve. The
best way of approaching it is probably to try to stop the recovery
master role from jumping around unnecesarily when inactive nodes join
the cluster.
The long term solution is to avoid this nonsense completely. The IP
allocation algorithm needs to cache state between runs so that it
knows which nodes have just become healthy. This also needs recovery
master stability.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit c51c1efe5fc7fa668597f2acd435dee16e410fc9)
Previously flagging a failure was probably avoided because of attempts
to run "ipreallocated" events on stopped and banned nodes, which would
fail because they are in recovery. Given the change to a new control
and that fallback only retries the old method on active nodes, this
should never fail in reasonable circumstances.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 53722430ad35f80935aabd12fa07654126443b8b)
They will reject it because they are in recovery. This can result in
extra banning credits being applied to banned nodes.
This corresponds to commit 9132e6814ed927fa317f333f03dedb18f75d0e5b
from the 1.2.40 branch.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 403938804caf1322f9773d63197e4303a7b2a788)
This should have been removed with the associated code in commit
14bd0b6961ef1294e9cba74ce875386b7dfbf446.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 36de63843de10a1f2a9ccdbbee24cc1d08542984)
This is an internal structure. It was moved into ctdb_private.h a
long time ago to allow unit testing. Unit test compilation was
changed shortly afterwards to make this unnecessary.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit db57261d7dc264e161659a8c547f44fbd9e88eeb)
Commit f73a4b1495830bcdd094a93732a89dd53b3c2f78 added a safety check
to ensure that CTDB never kills unrelated processes. However, client
processes are unrelated.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 782814288bb560099ee44b607bf35f3eddf37f82)
Currently the fail callback is called once per (takeip/releaseip) control
failure. This is overkill and can get a node banned much too quickly.
Instead, keep track of control failures per node and only call fail
callback once per failed node.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit bf4a7c1ad87e0e848296d15d63eb8cd901ca5335)
This helps distinguish processes in process list in top, perf, etc.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 2493f57ce268d6fe7e4c40a87852c347fd60d29e)
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-Programmed-With: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 41182623891d74a7e9e9c453183411a161201e67)
At the moment this is silent and it can be confusing to see IPs just
disappear.
Also, this message:
Been in recovery mode for too long. Dropping all IPS
can cause anxiety when all IPs should already have been dropped.
Adding a comforting message saying that 0 IPs were dropped relieves
such anxiety. :-)
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 4d0f26b306fc465d551d340b0e7dce4412eae3fd)
The log messages in verify_remote_ip_allocation() are confusing
because they don't include the PNN of the problem node, because it is
not known in this function.
Add the PNN of the node being verified as a function argument and then
shuffle the log messages around to make them clearer.
Also fold 3 nested if statements into just one.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit f0942fa01cd422133fc9398f56b4855397d7bc86)
At the moment there (at least) are 2 bugs that cause rogue IPs:
* A race where release_ip_callback() runs after a "subsequent" take IP
has completed. The IP is back on an interface but we unset
vnn->iface in the callback.
* A "releaseip" eventscript times out. We ignore the timeout and call
it success, deleting the VNN even if the IP is still hosted.
We could decide not to ignore the timeout and ban the node, but
killing TCP connections can take a long time and that might result
in a lot of manning. We probably won't reinstate banning on
"releaseip" until killing TCP connections has been optimised.
In both cases, a rogue IP can be avoided by leaving vnn->iface set and
simply failing the control.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit c5797f2942e83da24df548ea07196fbbac0eab20)
Previous code changes work around a potential problems but do not
provide useful information when the a problem occurs.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit f1f1b0c24b9b6cd24b83a4e4da16e179287ec6ac)
Consider the case of upgrading a cluster node by node, where some
nodes are still running older versions of CTDB without the
IPREALLOCATED control. If a "new" node takes over as recovery master
and a failover occurs, then it will attempt to send IPREALLOCATED
controls to all nodes. The "old" nodes will fail in a fairly
nondescript way (result == -1).
To try to handle this situation, fall back to the EVENTSCRIPT control
to handle "ipreallocated". Only do this on the failed nodes.
However, do not do this on nodes that timed out (they've probably
implemented the control and we should call the regular fail_callback
to get those nodes banned) or for stopped nodes (since they can't
actually run the "ipreallocated" event via the EVENTSCRIPT control).
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit b2654853ce9b7c18c5874b080bc94d3118078a5d)
Currently the order of the first IP allocation, including the first
"ipreallocated" event, and the "startup" event is undefined. Both of
these events can (re)start services.
This stops IPs being hosted before the "startup" event has completed.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit f15dd562fd8c08cafd957ce9509102db7eb49668)
If a tunable is not implemented on a remote node then this should not
be fatal. In this case the takeover run can continue using benign
defaults for the tunables.
However, timeouts and any unexpected errors should be fatal. These
should abort the takeover run because they can lead to unexpected IP
movements.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit c0c27762ea728ed86405b29c642ba9e43200f4ae)
Both of the current defaults are implicitly 0. It is better to make
the defaults obvious.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 1190bb0d9c14dc5889c2df56f6c8986db23d81a1)
This allows states, including startup and shutdown states, to be
clearly tracked. This doesn't include regular runtime "states", which
are handled by node flags.
Introduce new functions ctdb_set_runstate(), runstate_to_string() and
runstate_from_string().
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 8076773a9924dcf8aff16f7d96b2b9ac383ecc28)
Modifying the node flags with IP-allocation-only flags is not
necessary. It causes breakage if the flags are not cleared after use.
ctdb_takeover_run() no longer needs the general node flags - it only
needs the IP flags.
Instead of modifying the node flags in nodemap, construct a custom IP
flags list and have takeover_run_core() use that instead of node
flags. As well as being safer, this makes the IP allocation code more
self contained and a little bit clearer.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 14bd0b6961ef1294e9cba74ce875386b7dfbf446)
At the moment, when someone deletes all the IPs on a node, all we see
are the release IP messages and we have to guess why.
Some would argue that add/release are more significant than
take/release so they should be logged.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 3c3df1d6afec7e3e721f9bcd4e8b8e008fd6e50b)
These should never be seen outside the IP allocation code.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit e143abd16ccde2e0edfe103673d31a5fb06b6aef)
If these flags are left set they will confuse other recovery daemon
code.
Factor the clearing code into new function clear_ipflags().
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 45c776958017ea7001f061842c9e0f60e4a25f23)
This has been replaced by set_ipflags() and associated functionality.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit d0a3822573db296e73cc897835f783c8abc084b3)
This is really a check to see if a node is already hosting IPs. If
so, we assume it was previously healthy so it isn't considered as a
rebalance candidate. There's no need to limit this to healthy node,
since this is checked elsewhere.
Due to this the variable newly_healthy is renamed everywhere to
rebalance_candidates.
The mask argument is now completely unused.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 65e0ea6c2c0629e19349ba4b9affa221fde2b070)
This is a no-op and is in a separate commit to make the previous
commit less cumbersome.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 107e656bbe24f9d21fbaf886a3e9417da4effe5a)
This really needs to be per-node. The rename is because nodes with
this tunable switched on should drop IPs if they become unhealthy (or
disabled in some other way).
* Add new flag NODE_FLAGS_NOIPHOST, only used in recovery daemon.
* Enhance set_ipflags_internal() and set_ipflags() to setup
NODE_FLAGS_NOIPHOST depending on setting of NoIPHostOnAllDisabled
and/or whether nodes are disabled/inactive.
* Replace can_node_servce_ip() with functions can_node_host_ip() and
can_node_takeover_ip(). These functions are the only ones that need
to look at NODE_FLAGS_NOIPTAKEOVER and NODE_FLAGS_NOIPHOST. They
can make the decision without looking at any other flags due to
previous setup.
* Remove explicit flag checking in IP allocation functions (including
unassign_unsuitable_ips()) and just call can_node_host_ip() and
can_node_takeover_ip() as appropriate.
* Update test code to handle CTDB_SET_NoIPHostOnAllDisabled.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 1308a51f73f2e29ba4dbebb6111d9309a89732cc)
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 1fb5352d2b6918fcc6f630db49275d25a3eebe8d)
This means "ipreallocated" is now run on stopped nodes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 83b61f7414b1f7a3424497ac987ca0724fba9eaa)
This is an alternative to using ctdb_run_eventscripts() that can be
used when in recovery.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 27a44685f0d7a88804b61a1542bb42adc8f88cb1)
When collating IP information for IP layout, only trust the nodes that are
hosting an IP, to have correct information about that IP. Ignore what all the
other nodes think.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 1c7adbccc69ac276d2b957ad16c3802fdb8868ca)
This makes the code much more readable and maintainable.
As a side effect, fix a memory leak in LCP2.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 6a1d88a17321f7e1dc84b4823d5e7588516a6904)
Move the code into a new function so it can be called from a number of
places.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 8adb255e62dbe60d1e983047acd7b9c941231d11)
The retry loop is currently in ctdb_takeover_run_core(). Pushing it
into each function will make it possible to put each algorithm into a
separate top-level function. This will make the code much clearer and
more maintainable.
Also keep associated test code compatible.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit f6ce18d011dd9043b04256690d826deb2640cd89)
Neither basic_failback() nor lcp2_failback() unassign IPs anymore, so
there's no point looping back that far.
Also fix a unit test that now fails because looping back to handle
unassigned IPs is no longer logged.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit c09aeaecad7d3232b1c07bab826b96818756f5e0)
Instead of unassigning, looping back and depending on
basic_allocate_unassigned.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 4dc08e37dec464c8785a2ddae15c7c69d3c81ac3)
This seems to be the right thing to do instead of calling into the
failback code and continually skipping the release of an IP.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 4c87e7cb3fa2cf2e034fa8454364e0a7fe0c8f81)
If this is done earlier then some other logic can be improved. Also,
this should be a warning since no error condition is set.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit e06476e07197b7327b8bdac9c0b2e7281798ffec)
Add a new function ctdb_remove_orphaned_ifaces() and call it in
ctdb_control_del_public_address().
ctdb_remove_orphaned_ifaces() uses a naive implementation that does
things in a very obvious way. There are many ways to improve the
performance - some are mentioned in a comment in the code. However, I
doubt that this will be a bottleneck even with a large number of
public IPs. Running the eventscript is likely to outweigh the cost of
this cleanup.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit cc1a3ae911d3fee8b87fda5de5ab6d9499d7510a)
Neither up nor down is a good default value for the link status of a
new interface. Up means that IPs can be assigned to interfaces before
the true state is known and they can move away quickly if the interface
is actually down. Down means that IPs can't be assigned to an interface
for a variable amount of time - until a monitor cycle occurs - and this
can result in imbalanced IPs.
This is a neat compromise. Before the startup event completes, IPs
can't be assigned to interfaces because all interfaces begin in a down
state. As soon as the startup event completes, IPs can be allocated
to any interface that has been marked up by the eventscript. Later,
during normal operation, newly added IPs can be assigned to new
interfaces immediately. The IPs will still move away if an interface
is noticed to be down in the next monitor cycle, but that is the
exception rather than the rule.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 9275a69a414482f1053ae14528d5972575b9214e)
If any of the nodes fail takeover run (either due to timeout or failure
to complete within takeover_timeout interval) from main loop, recovery
master will give up trying takeover run with following message:
"Unable to setup public takeover addresses. Try again later"
And as a side-effect the monitoring is disabled on all the nodes. Before
ctdb_takeover_run() is called from main loop, monitoring get disabled via
startrecovery event. Since ctdb_takeover_run() fails, it never runs
recovered event and monitoring does not get re-enabled.
In main_loop, ctdb_takeover_run() is called with a takeover_fail_callback.
This callback will get called if any of the nodes fail in handling
takeip/releaseip/ipreallocated events in ctdb_takeover_run().
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit a5c6bb1fffb8dc3960af113957a1fd080cc7c245)
Disable for TakeoverTimeout seconds.
Otherwise the the recovery daemon can get overzealous and start trying
to add/delete addresses that it thinks are missing but where the
eventscript just hasn't finished. This didn't used to matter so much
but it is more important now that concurrent takeip/releaseip/updateip
generate error - we want to avoid spamming the log.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 56fcee3c7730cb12fa666072d5400949af6e5f7c)
There's a race here where release and takeover events for an IP can
run at the same time. For example, a "ctdb deleteip" and a takeover
initiated by the recovery daemon. The timeline is as follows:
1. The release code registers a callback to update the VNN. The
callback is executed *after* the eventscripts run the releaseip
event.
2. The release code calls the eventscripts for the releaseip event,
removing IP from its interface.
The takeover code "updates" the VNN saying that IP is on some
iface.... even if/though the address is already there.
3. The release callback runs, removing the iface associated with IP in
the VNN.
The takeover code calls the eventscripts for the takeip event,
adding IP to an interface.
As a result, CTDB doesn't think it should be hosting IP but IP is on
an interface. The recovery daemon fixes this later... but it
shouldn't happen.
This patch can cause some additional noise in the logs:
Release of IP 10.0.2.133/24 on interface eth2 node:2
recoverd:We are still serving a public address '10.0.2.133' that we should not be serving. Removing it.
Release of IP 10.0.2.133/24 rejected update for this IP already in flight
recoverd:client/ctdb_client.c:2455 ctdb_control for release_ip failed
recoverd:Failed to release local ip address
In this case the node has started releasing an IP when the recovery
daemon notices the addresses is still hosted and initiates another
release. This noise is harmless but annoying.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit bfe16cf69bf2eee93c0d831f76d88bba0c2b96c2)
Stops the behaviour where unhealthy nodes can host IPs when there are
no healthy nodes. Set this to 1 when an immediate complete outage is
preferred when all nodes are unhealthy. The alternative
(i.e. default) can lead to undefined behaviour when the shared
filesystem is unavailable.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit a555940fb5c914b7581667a05153256ad7d17774)
The existing code makes one fatally bad assumption:
vnn->iface->references can never be -1 (or max-unit32_t in this case).
Right now the reference counting is broken so a reference count of -1
is possible and causes a spurious updateip when vnn->iface is the same
as best_face. This can occur frequently because we get a lot of
redundant takeovers, especially when each IP can only be hosted on one
interface.
This makes the code much more defensive by noting that when best_iface
is the same as vnn->iface there is never a need for an updateip event.
This effectively neuters the updateip code path when IPs can only be
hosted by a single interface.
This should obsolete 6a74515f0a1e24d97cee3ba05d89133aac7ad2b7.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 7054e4ded59c6b8f254dcfefaef64da05f25aecd)
This reverts commit 4308935ba48ac7a29e7523315acf580019715f0f.
This fixes 16_ctdb_config_add_ip.sh test when run against local daemons. When
running against local daemons, if the interface is assigned as soon as an IP is
added, then takeover would never assign this IP address.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 06dfd13604d08910e07cbf927c338d7b9fce9a2f)
This message used to be correct because the ipreallocated event only
handled updating the NAT gateway. However, that has changed so the
message needs to be updated.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit cc9d96f4248e45ea99c5f00db1526426ac26fbc2)
When running on local daemons no IPs are actually assigned to
interfaces. Commit 9a806dec8687e2ec08a308853b61af6aed5e5d1e broke
ctdb_control_release_ip() for local daemons because it asks the system
which interface the given IP is on, instead of the old behaviour of
trusting CTDB's internal records.
For local deamons (i.e. !ctdb->do_checkpublicip) revert to the old
behaviour of looking up the interface internally. This is good
enough, given that the tests don't tend to misconfigure the addresses.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 38e8651b955afdbaf0ae87c24c55c052f8209290)
instead of using the interface where ctdb thinks the ip is hosted at.
The difference is that this now allows us to handle cases where we want to release an ip but ctdbd does not know which interface the ip is assigned on.
(user has used 'ip addr add...' and manually assigned an ip to the wrong interface)
(This used to be ctdb commit c6bf22ba5c01001b7febed73dd16a03bd3fd2bed)
Wrap all creation of child processes inside ctdb_fork() which is used to track all processes we have spawned.
Capture SIGCHLD to track also which child processes have terminated.
Wrap kill() inside ctdb_kill() and make sure that we never send a !0 signal to a child process pid that has already terminated (and might have been replaced with a
(This used to be ctdb commit f73a4b1495830bcdd094a93732a89dd53b3c2f78)
and also update the "read public address file" to not check if the address exists already locally when we read if from the child process, to stop it
from spamming the logs with "We already host ..."
messages
(This used to be ctdb commit 334ea830f1bf33419f4a1e78f23afd41a852d0f4)
Also add a method to use the recovery master/daemon to reload the public ips on all nodes in the cluster.
Reloading the public ips on all node sin the cluster is only suported if all nodes in the cluster are available and healthy.
(This used to be ctdb commit 05603e914f8c12618d7e06943c0f7df207f645b0)
Remove the old global setting for this unused tunable and add it as a new node flag. This node flag is only valid/defined within the takeover subsystem in the recovery daemon. Add async functions to collec the NoIPFailback settings for each node.
This will later e used to disqualify certain nodes from being takeover targets when we perform reallocation.
(This used to be ctdb commit 668f3e88a9e5f598706952b7140547640c85a5ed)
the referenced VNN structure is.
Also, remove the circular reference between the two objects KIPPCTP and VNN
(This used to be ctdb commit 02b62482164a3c69715949074feb7f191a29d534)
These were intentionally not static so they could be linked to in unit
test programs. However, using the CCAN-style unit tests where
relevant code is just included, this is no longer necessary.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit d0e9e8554614bd49ffb9ec3509feaa0e80d0f65d)
This patch changes the callback signature for traversal
functions to allow a client to abort a traverse before it finishes.
Updates to all callers and examples as well as rb-test tool.
(This used to be ctdb commit 8ab0c63ad36cfbbb1e5fed46a1f4c47b1fdb581f)
There's a bug in LCP2. Selecting the node with the highest imbalance
doesn't always work. Some nodes can have a high imbalance metric
because they have a lot of IPs. However, these nodes can be part of a
group that is perfectly balanced. Nodes in another group with less
IPs might actually be imbalanced.
Instead of just trying the source node with the highest imbalance this
tries them in descending order of imbalance until it finds one where
an IP can be moved to another node.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 574091d5aced5e87aefad52f8bc47aa75c25fbf6)
There's a bug in LCP2. Selecting the node with the highest imbalance
doesn't always work. Some nodes can have a high imbalance metric
because they have a lot of IPs. However, these nodes can be part of a
group that is perfectly balanced. Nodes in another group with less
IPs might actually be imbalanced.
Factor out the code from lcp2_failback() that actually takes a node
and decides which address should be moved to which node.
This is the first step in fixing the above bug.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 75718c5768b5bb5c0bcd7dd90e0327c6ed22a63d)
this triggered a check for "only run the eventscript if we host the address" to trigger and shortcir=cuit calling the eventscript.
An effect of this would be that 'ctdb delip' would remove the ip from ctdb, but fail to delete it from the interface.
S1028798
(This used to be ctdb commit b82524f240bf21769dd7624ca6026763d38b9396)
cant talloc off vnn since it is not yet initialized and might not always be NULL
(This used to be ctdb commit 3d37be3e2bfb61ede824028aeebaa18ba304faae)
This will make it much easier to root-cause problems such as
S1029023
when an external application deleted the interface while it is still is in use by ctdbd.
(This used to be ctdb commit 9abf9c919a7e6789695490e2c3de56c21b63fa57)
check that the actual interface exist, print error and fail startup if the interface does not exist.
(This used to be ctdb commit cd33bbe6454b7b0316bdfffbd06c67b29779e873)
sometimes we do want to try to set the linkstate for interfaces that are not in use by public addresses right now (but posisbly by other mechanisms) and these messages just spam the logs
S1026357
(This used to be ctdb commit f2fe0a090a9650910ebe49514b3ca01dc593bea3)
Move struct ctdb_public_ip_list to ctdb_private.h and put some
definitions for some functions from ctdb_takeover.c there. This
allows those functions to be called from unit tests.
Add ctdb_takeover_tests.c and the Makefile support to build it.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 9d34be0233edf3bc022345c0494c4b2a4d7f8480)
The current non-deterministic IP allocation algorithm balances IPs
across the whole cluster. It does not consider different
interfaces/VLANs/subnets, so these different groups of IPs aren't
generally well balanced.
This adds the LCP2 algorithm for IP allocation and allows it to be
enabled by setting the "LCP2PublicIPs" tunable to 1.
The LCP2 algorithm calculates the imbalance of a node by totalling the
squares of the distances between each IP on the node. The IP distance
is defined as the length longest common prefix (LCP) of bits that is
found when comparing 2 IPs. The imbalance of a cluster is the maximum
imbalance for any node. At each step the algorithm selects an
allocation to the IP/node combination that results in the choosing the
allocation that best reduces the imbalance of the cluster.
The implementation splits out the IP allocation part of
ctdb_takeover_run() into new function ctdb_takeover_run_core(), and
then extracts out the basic IP assignment code into new functions
basic_allocate_unassigned() and basic_failback(). 3 new functions
lcp2_init(), lcp2_allocate_unassigned() and lcp2_failback() implement
the LCP2 algorithm, and are hooked into ctdb_takeover_run_core().
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 61fc7fbd0235469df22deb6581c6bd47e30bc0be)
Dont talloc_free(vnn) immediately but postphone it until later when
the eventscript callback has completed.
CQ S1026664
(This used to be ctdb commit 0a99e8742a261b1d3a2c8830f5c19ea6c2c47cad)
nodes with many addresses to nodes with few addresses,
loop up to num_ips+5 times instead of only 5 times.
When we have very many public ips per node, we might need to loop more than
5 times or else we will exit without reaching optimal balance.
(This used to be ctdb commit aa8114a625a637277561a66c80bdece3c27e9e20)
Simplify the handling of setting the links in the 10.interface eventscript
and remove the optimization to only call setifacelink on state change
to make the code simpler to read.
If a take ip event fails, flag the node as unhealthy.
Add a check to the interface script to check if the interface exists
or if it has been deleted.
So that we can capture and become UNHELTHY if someone deletes an interface
we are using to host public addresses.
(This used to be ctdb commit 4ab63d2a7262aff30d5eced184c294c9c9dd4974)
by external services failing to start, or blocking CTDBD from finishing the startup phase,
we can encounter a situation where we have not yet fully initialized, but a
remote recovery master tries to release a certain ip clusterwide.
In this situation the node that is pinned down in init/startup phase
would fail to perform the release of the ip address since we are not yet fully operational and not yet host any valid interfaces.
In this situation, we just need to remain unhealthy, there is on need to
also ban the node.
Remove the autobanning for this condition and just let the node remain in
unhealthy mode.
Banning is overkill in this situation when the system is broken and just
draws attention to ctdbd instead of the root cause.
(This used to be ctdb commit d8af74e4c4961deb94c18dde8ba7fc07e944729c)
flag the interface as initially being "link ok"
so that we can add it and startup.
The eventscript can later drop the flag if required
(This used to be ctdb commit 720849b756c825fb8b285f09972a8c39f1888a99)
but thinks it is still unassigned (-1).
add code to the recovery daemon to detect this case and trigger a reallocation
so that the ip gets covered
and change the takeip code to allow for this condition, taking on an ip address that is
already hosted.
cq s1021073
(This used to be ctdb commit 9020baf27cab7821c9094cda185206fb7af0fee7)
since if they are the same for whatever reason this triggers the system
to go into an infinite loop and is unrobust
The scriptds have been changed instead to be able to cope with this
situation for enhanced robustness
During takeover_run and when merging all ip allocations across the cluster
try to kepe track of when and which node currently hosts an ip address
so that we avoid extra ip failovers between nodes
(This used to be ctdb commit cf778b5aaf6356401e3985acccc7df9e08ab6930)
This is called everytime a reallocation is performed.
While STARTRECOVERY/RECOVERED events are only called when
we do ipreallocation as part of a full database/cluster recovery,
this new event can be used to trigger on when we just do a light
failover due to a node becomming unhealthy.
I.e. situations where we do a failover but we do not perform a full
cluster recovery.
Use this to trigger for natgw so we select a new natgw master node
when failover happens and not just when cluster rebuilds happen.
(This used to be ctdb commit 7f4c591388adae20e98984001385cba26598ec67)
Add a new "ctdb deltickle" command to delete tickles from the database.
This can ONLY be used for tickles created by "ctdb addtickle".
Push any "addtickle/deltickle" updates to other nodes every TickleUpdateInterval seconds'
(This used to be ctdb commit acded034e2f0dcae4c2c9e54e16a001caf23caec)
After 5 attempts to send a RST to a client without any response, we free
"con"; this is done during a traverse. This frees the node we are walking
through (the node is made a child of "con" down in rb_tree.c's
trbt_create_node() (Valgrind would catch this, as Martin confirmed).
So, we create a temporary parent and reparent onto that; then we free
that parent after the traverse, thus deleting the unwanted nodes.
CQ:S1019041
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 08f7f85477610a4916c1ec866aa467b28f1bbec3)
In Samba this is now called "tevent", and while we use the backwards
compatibility wrappers they don't offer EVENT_FD_AUTOCLOSE: that is now
a separate tevent_fd_set_auto_close() function.
This is based on Samba version 7f29f817fa.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 85e5e760cc91eb3157d3a88996ce474491646726)
verify that all nodes agree on the most recent ip address assignments
broke "ctdb moveip ..." since that call would never trigger
a full takeover run and thus would immediately trigger an inconsistency.
Add a new message to the recovery daemon where we can tell the recovery daemon to update its assignments.
BZ62782
(This used to be ctdb commit e7069082e5f0380dcddee247db8754218ce18cab)
addresses and verify that the remote nodes have/keep a consistent view of
assigned addresses.
If a remote node has an inconsistent view of addresses visavi the recovery
master this will trigger a full ip reallocation.
(This used to be ctdb commit f3bf2ab61f8dbbc806ec23a68a87aaedd458e712)
This makes sure that we don't get public addresses assigned during the
initial recovery and remove them again in the startup event.
metze
(This used to be ctdb commit f872e8c63a2f8979e6a0d088630575bdd4d7b4f1)
This only marks the interface status and doesn't
generate any directly triggered action.
The actions is later taken by the recovery process
in verify_ip_allocation.
metze
(This used to be ctdb commit cff58b27c970e9252d131125941c372019fd6660)
We know ask for the known and available interfaces.
This means a node gets a RELEASE_IP event for all interfaces
it "knows", but doesn't serve and a node only gets a TAKE_IP event
for "available" interfaces.
metze
(This used to be ctdb commit a695a38e49e7c3e15a9706392dc920eeab1f11ba)
Currently the timeout handler in eventscript.c does the banning if a
timeout happens. However, because monitor events are different, it has
to special case them.
As we call the callback anyway in this case, we should make that handle
-ETIME as it sees fit: for everyone but the monitor event, we simply ban
ourselves. The more complicated monitor event banning logic is now in
ctdb_monitor.c where it belongs.
Note: I wrapped the other bans in "if (status == -ETIME)", though they
should probably ban themselves on any error. This change should be a
noop.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 9ecee127e19a9e7cae114a66f3514ee7a75276c5)
Rather than doing strcmp everywhere, pass an explicit enum around. This
also subtly documents what options are available. The "options" arg
is now used for extra arguments only.
Unfortunately, gcc complains on empty format strings, so we make
ctdb_event_script() take no varargs, and add ctdb_event_script_args(). We
leave ctdb_event_script_callback() taking varargs, which means callers
have to do "%s", "".
For the moment, we have CTDB_EVENT_UNKNOWN for handling forced scripts
from the ctdb tool.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 8001488be4f2beb25e943fe01b2afc2e8779930d)
Everyone uses the same timeout value, so just remove it from the API.
If we ever need variable timeouts, that might as well be central too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 533c3e053293941d2a9484b495e78d45f478bb08)
this is to better handle linux clients which often default to ignore grat arps that arrive within 1 second of eachother.
(This used to be ctdb commit 5664da36943b4901a807a9594b0f45e859aafbf3)
This is used to mark nodes as being DELETED internally in ctdb
so that nodes are not renumbered if / when they are removed from the nodes file.
This is used to be able to do "ctdb reloadnodes" at runtime without
causing nodes to be renumbered.
To do this, instead of deleting a node from the nodes file, just comment it out like
1.0.0.1
#1.0.0.2
1.0.0.3
After removing 1.0.0.2 from the cluster, the remaining nodes retain their
pnn's from prior to the deletion, namely 0 and 2
Any line in the nodes file that is commented out represents a DELETED pnn
(This used to be ctdb commit 6a5e4fd7fa391206b463bb4e976502f3ac5bd343)
allow clients to register either ipv4 or ipv6 client connections to the tickles list
(This used to be ctdb commit d9b44d7c3255b0fd7359b9afeb613e6ff4c4eaac)
described an ip address not known to be a public address.
This could happen if someone for genuine reasons accesses a share
through a static ip address.
It can also happen if non homogenous public address configurations are
used and when a tcp description is pushed out to a different node that
does not server/know the specific ip address.
(This used to be ctdb commit 9b1d089c99413f3681440f3cf33c293d118c9108)
older ipv4-only version of these controls.
We need this so that we are backwardcompatible with old versions of ctdb
and so that we can interoperate with a ipv4-only recmaster during a
rolling upgrade.
(This used to be ctdb commit 6b76c520f97127099bd9fbaa0fa7af1c61947fb7)
