IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Only create PID file when actually starting the daemon, rather than
when setting up the context. This will facilitate future changes.
Tweak test to confirm that PID file is no longer created during setup.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12978
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Aug 25 13:32:58 CEST 2017 on sn-devel-144
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Mon Aug 14 13:00:16 CEST 2017 on sn-devel-144
This causes any tracked connections for the IP address to be lost.
When doing a takeip, the server sends a tickle ACK to the client, the
client responds with a valid ACK and the server's TCP stack responds
with a reset because the connection does not exist. However, in the
updateip, case the connection *does* exist, so the tickle *does not*
cause the connection to be reset.
ctdb_announce_vnn_iface() clears the list of tracked TCP connections
while sending the tickle ACKs. So, if there are no reconnects as in
the takeip case, then the list of connections is simply lost.
The "updateip" event in the 10.interface event script already sends
gratuitous ARPs and tickles connections in both directions. This
ensures that traffic continues after packets may have been dropped
when the script temporarily blocks traffic to the IP address.
All of this means that the call to ctdb_announce_vnn_iface() can just
be deleted.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This allows to differentiate between the two database models.
ctdb_db_persistent() - replicated and permanent
ctdb_db_volatile() - distributed and temporary
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
There is no need for with_jenkinshash and with_mutexes flags, since the
tdb_flags are now calculated based on database type.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
The tdb open flags should be calculated based on the database type and
ctdb tunables.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12857
If we drop public IPs because CTDB is in recovery for too long, then
avoid spamming logs "Trigger takeoverrun" every second.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12857
This can be used later in the main_loop to avoid the local ip check.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12857
This simplifies the code and avoids complicated conditions.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12857
If the recovery mode is already set to the expected value, there is
nothing to do.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Mon Jun 19 19:56:22 CEST 2017 on sn-devel-144
Currently, every lock helper will log a message if it cannot get a lock.
This can spam the logs and overwhelm syslog if there are hundreds of
lock helpers waiting for contended record.
Instead keep track of the record for which we have already logged once
with specific timeout interval. If we get timeout interval larger than
the previously logged interval, then log again once. This will reduce
the amount of logs for contended records to a single log entry per 10
seconds per record.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Persistent databases are now always recovered by sequence number. So
there is no need to keep the empty records in the database since they
will never be recovered record-by-record using RSN.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Sat Jun 17 16:47:55 CEST 2017 on sn-devel-144
Persistent databases are now always recovered by sequence number. So
there is no need to keep the empty records in the database since they
will never be recovered record-by-record using RSN.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This avoids spamming the logs during recovery at NOTICE level.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Tue Jun 13 13:22:09 CEST 2017 on sn-devel-144
Signed-off-by: Richard Sharpe <realrichardsharpe@gmail.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Jun 7 09:22:29 CEST 2017 on sn-devel-144
This allows to pass data to a child process via stdin.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This allows to mix CTDB major versions in a single cluster.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Wed May 24 21:06:28 CEST 2017 on sn-devel-144
Extend CTDB_REQ_KEEPALIVE packet to include version and uptime. If CTDB
versions do not match shutdown ctdb.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12697
During revoking readonly delegations, if one of the nodes disappears,
then there is no point re-trying revoking readonly delegation immedately.
The database needs to be recovered before the revoke operation can
succeed.
However, if the revoke is successful, then all the write requests need
to be processed immediately before the read-only requests. This avoids
starving write requests, in case there are read-only requests coming
from other nodes.
In deferred_call_destructor, the result of revoke is not available and
deferred calls cannot be correctly ordered. To correctly order the
deferred calls, process them in revokechild_destructor where the result
of revoke is known.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12697
This reverts commit ad758cb869.
This is an incomplete fix and introduces a regression.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Instead of using hopcount as a metric for hot records, use the number
of migrations per second as a metric.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Wed Apr 5 08:35:45 CEST 2017 on sn-devel-144
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12697
During revoking readonly delegations, if one of the nodes disappears, then
there is no point re-trying revoking readonly delegation. The database
needs to be recovered before the revoke operation can succeed. So retry
only after a grace period.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Mar 17 14:05:57 CET 2017 on sn-devel-144
This avoids confusing log messages like:
ctdbd[21635]: releaseip called for an ip '10.1.1.1' that is not a public address
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Feb 24 11:50:36 CET 2017 on sn-devel-144
Redundant releases will be sent to all connected nodes anyway, so this
is no worse. This will facilitate an improvement to avoid sending
releases to nodes with no known IPs.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
As with other controls, processes the errors by hand instead of using
ctdb_client_control_multi_error(). This will make it easier to add
banning credits for failures.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
A simple optimisation to avoid unnecessary communication.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The current code will fetch IP from all connected and all active
nodes, so this can't happen. However, catch it anyway in case the
calling code changes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This simplifies error handling and make failures less likely after
send.
This also means that num_nodes is not required in the state.
Also quietly remove unused ev and client from state.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Chris Lamb <chris@chris-lamb.co.uk>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Chris Lamb <chris@chris-lamb.co.uk>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Chris Lamb <chris@chris-lamb.co.uk>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Feb 17 14:45:10 CET 2017 on sn-devel-144
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12513
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Tue Jan 17 15:00:15 CET 2017 on sn-devel-144
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12469
The code to lock multiple databases has been dropped from ctdb_lock.c.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: David Disseldorp <ddiss@samba.org>
Autobuild-User(master): David Disseldorp <ddiss@samba.org>
Autobuild-Date(master): Wed Dec 28 05:18:08 CET 2016 on sn-devel-144
Introduce a single new tunable IPAllocAlgorithm to set the IP
allocation algorithm. This defaults to 2 for LCP2 IP address
allocation.
Tunables LCP2PublicIPs and DeterministicIPs are obsolete.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Instead of gathering the value from all nodes, just use the value on
the recovery master and have it affect all nodes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Instead of gathering the value from all nodes, just use the value on
the recovery master and have it affect all nodes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Sun Dec 18 18:10:50 CET 2016 on sn-devel-144
Following controls are now implemented by event daemon
- RUN_EVENTSCRIPTS
- GET_EVENT_SCRIPT_STATUS
- ENABLE_SCRIPT
- DISABLE_SCRIPT
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
The recovery helper does it's own logging, so there is no need to
pass logfd.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Mon Dec 5 11:59:42 CET 2016 on sn-devel-144
This avoids the extra argument of logfd to ctdb_lock_helper. The log
messages from lock helper are captured by ctdbd.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
- Use fprintf() before logging is initialized
- replace DEBUG_ALERT with DEBUG_ERR
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Use a "bitmap" of available IPs for each IP address instead of walking
the list of available IP addresses.
For ctdb/tests/takeover/lcp2.030.sh, this improves the time taken on
my laptop from:
real 0m11.997s
user 0m11.960s
sys 0m0.000s
to
real 0m8.571s
user 0m8.544s
sys 0m0.000s
So, when assigning all 900 IP addresses the improvement is about 25%.
For the no-op case (where all IPs are already assigned to nodes), the
extra setup adds a small fraction of a second for 900 IPs.
Intermediate cases result in intermediate improvements.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This function is currently called twice each time a node is banned.
ctdb_local_node_got_banned() is already called from the banning code,
either due to a received banning control or a node banning itself.
Given that other nodes can't set a node's BANNED flag, a node can only
be banned via the above mechanisms, so drop the redundant call.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Persistent databases are now always recovered by sequence number, so
there is no need for this tunable.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Nov 25 08:13:59 CET 2016 on sn-devel-144
This inserts the code from ctdb_cmdline_init() function directly in
main(), so common/cmdline.[ch] can be removed.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12392
Earlier we were relying on SCHED_RESET_ON_FORK to reset the priority of lock
helper processes. Since SCHED_RESET_ON_FORK support has been removed, the
scheduling priority of child processes created using vfork() need to be reset
explicitly in the helper processes.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Nov 18 10:18:27 CET 2016 on sn-devel-144
This was a weak attempt at exclusivity. PID file creation now does
that properly.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12287
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
No use touching the socket if PID file creation fails.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12287
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is useful information if ctdb is unable to freeze any of the
databases on banning or stopping.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This is available in the IP allocation state.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12254
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Add an early return if there are no known IP addresses.
Also add an extra comment for clarification.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12254
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Merged IP list won't be available here...
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12254
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This was dropped because it wasn't used, but it will be needed again.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12254
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This function is called only once from force_election() and does not
require freezing of databases.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
If the interfaces have different names then they are different
interfaces.
Also, move assignment of new_name just above where is is first used.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Now takes a pointer to an interface structure and does direct pointer
comparisons.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
To keep this change small, this leaves behind some redundant calls to
ctdb_find_iface() and similar. They will be cleaned up later.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This will allow a change to the way interfaces are handled in a VNN.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
There's no point parsing the interfaces twice, especially since it
doesn't improve error handling.
This also removes a use of strdup(3)/free(3), which is not generally
used in our code.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Also add a missing out-of-memory check for vnn->ifaces.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This also moves the interface validation down, making more obvious
that it can be consolidated.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If anything should be used here it should be talloc functions.
However, this is a remnant from when ctdb_sys_find_ifname() was used
here and, for some reason, it used strdup(3).
In this case the interface string doesn't actually need to be copied.
The only use of it is when ctdb_event_script_callback_v() uses it with
the format string in a call to talloc_vasprintf(). In the same
context the IP address isn't copied.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If *async_reply isn't set then the calling code will reply to the
control and free the control structure. In some places the control
structure pointer is stolen onto state before a synchronous exit due
to an error condition. The error handling then frees state and
returns an error. The calling code will access-after-free when trying
to reply to the control.
To make this easier to understand, the convention is that any
(immediate) error results in a synchronous reply to the control via an
error return code AND *async_reply not being set. In this case the
control structure pointer should never be stolen onto state. State is
never used for a synchronous reply, it is only ever used by a
callback.
Also initialise state->c to NULL so that any premature call to a
callback (e.g. in an immediate error path) is more obvious.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12180
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The callback should never be called before an immediate return. The
callback might reply to a control and the caller of
ctdb_event_script_callback_v() may not have assigned/stolen the
pointer to control structure into the private data. Therefore,
calling the callback can dereference an uninitialised pointer to the
control structure when attempting to reply.
An event script isn't being run until the child has been forked. So
update relevant state and set the destructor after this.
If the child can't be forked then free the state and return with an
error. The callback will not be called and the caller will process
the error correctly.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12180
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The callback should never be called before an immediate return. The
callback might reply to a control and the caller of
ctdb_event_script_callback_v() may not have assigned/stolen the
pointer to control structure into the private data. Therefore,
calling the callback can dereference an uninitialised pointer to the
control structure when attempting to reply.
ctdb_event_script_callback_v() must succeed when there are no event
scripts. On success the caller will mark the call as asynchronous and
expect the callback to be called. Given that it can't be called
before return then it needs to be scheduled.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12180
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
When an error occurs so an IP address is not released then the PNN in
the VNN is currently incorrectly updated.
Instead, update the PNN in the callback when the release is
successful. Also, explicitly update the PNN on redundant releases.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12158
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Sun Aug 21 22:45:33 CEST 2016 on sn-devel-144
Many years ago takeover_callback_state was used for both IP takeover
and release. Now it is only used when releasing an IP so rename it to
improve clarity.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12158
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Commit c40fc62642 runs the IP allocation
algorithm after calculating the timeout offset. If the algorithm
takes a long time then there may be no attempt to release or take over
IPs.
Instead, reset the timeout just before the RELEASE_IP stage if an
early jump to IPREALLOCATED was not taken.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12161
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Thu Aug 18 12:36:37 CEST 2016 on sn-devel-144
This is inconsistent with the rest of the local IP verification. It
should notice problems but not try to fix them directly. Like other
cases, it should use an IP takeover run to try to fix the problem. In
this case the address might have just been added and an out-of-band
RELEASE_IP might cause conflicts (i.e. "another change is in flight")
with a scheduled IP takeover run.
This effectively reverts commit
694c1b269e. Not sure why this was
needed after c7e648c2d1. More recently
commit 6471541d6d moves responsibility
for determining interface/netmask to 10.interface so this should
continue to work just fine.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Cause an "updateip" instead of just logging a message.
This may reset existing connections. However, CTDB doesn't think the
address should already be hosted on the node so there should be no
connections.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This doesn't currently happen but it will in a subsequent commit.
That commit and this one could be squashed but then the functional
change gets lost in amongst this one.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This reverts commit 4136f27145.
If the IP address is on an interface then it won't help to pretend
that it isn't. This will simply cause a takeip event, which will fail
because the address can't be added. Note that the IP address isn't
necessarily new - something unexpected may have happened.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The address may already be assigned to another node, so this is wrong.
It also leaves the interface unknown.
This is better left to code that handles rogue IP addresses. A
takeover run should correctly takeover the address if it is assigned
to this node or release it if it is assigned to another node. Coming
soon...
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This has the advantage of using common code. Also, if there was
previously a failed attempt to release the IP address as part of a
delete, then this will finish processing the delete.
Extra care needs to be taken when a VNN is actually deleted.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12158
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This contains the cleanup that needs to be done after an IP address is
released from an interface.
state->vnn is set to the return value from release_ip_post(), which is
either the original VNN, or NULL if it was deleted. This allows
correct handling of the in-flight flag in the destructor for state.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12158
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If there's an allocation failure then the implicit early return in
CTDB_NO_MEMORY_VOID() means that no reply is sent to the control.
ctdb_daemon_send_message() makes a copy of the data, so don't copy it
here and remove an unnecessary chance of failure.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12158
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If RELEASE_IP fails then updating the VNN makes it inconsistent with
reality. Instead, log the failure and move on to the next IP
address.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12158
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The "releaseip" event in 10.interface will determine the interface and
do the right thing.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12158
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Interfaces going up or down are always interesting, so log these at
error level.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12157
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is related to an error, so repeatedly log at error level instead
of trying to avoid repetition.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12157
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Failures are already logged at alert/error level above, so just log
the summary at notice level.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12157
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The current message is broken:
Control SET_DB_PRIORITY is not implemented any more, use instead
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12126
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
These were used in serial recovery and for restoring databases using
older ctdb tool. New code uses database specific transaction controls.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This control was used by the older implementation of tool to restore a
database from backup. In the new implemenation of tool, it freezes and thaws
only the database being restored.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This drastically simplifies the code. "ctdb reloadips" behaves the
same, since it causes a takeover run immediately after IPs are
deleted. "ctdb delip" now needs to be followed with an explicit "ctdb
ipreallocate".
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
RELEASE_IP sometimes times out because killing TCP connections can
take a long time.
The aim of the takeover timeout is actually to limit the total amount
of time for an IP takeover run. So, calculate a combined timeout
offset once and use it for each of the RELEASE_IP, TAKEOVER_IP,
IPREALLOCATED stages. This gives RELEASE_IP more time to kill TCP
connections but still limits the total time.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
There are no database priorities anymore, so the function name does
not make any sense. Call the code in thaw_priority() directly from
ctdb_control_thaw().
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Parallel database recovery freezes databases in parallel and irrespective
of database priority. So drop priority from freeze/thaw code.
Database priority will be dropped completely soon.
Now FREEZE and THAW controls operate on all the databases.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
If the database is not frozen and recovery mode is not active, then
vacuuming can continue.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Logging that node has lost election is less useful than knowing which
node has won the election.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
No longer require CTDB context but pass in number of nodes, algorithm,
no_ip_failback and force_rebalance_nodes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
At the moment IP is short-circuited when there are no available IP
addresses. However, if some IP addresses are already allocated then
"no available IP addresses" means that all the addresses should
(probably) be released. The current short-circuit means that no
already hosted IP addresses will be released.
The short-circuit exists to avoid lots of messages saying that all IP
addresses can not be assigned at startup time. So, add a check to
ipalloc_can_host_ips() so that it succeeds if IP addresses are already
allocated to nodes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Abstracts out code involving internals of IP allocation state.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is never used in the allocation algorithms. It is only used when
building the merged IP list.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
How the existing IP layout is constructed and how the merged IP list is
sorted are important aspects of the IP allocation algorithm. Construct the
merged IP list when known and available IPs are assigned.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Use ctdb_fetch_remote_public_ips() inline to fetch each list. Assign
them into the IP allocation state separately.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Factor out new function ctdb_fetch_remote_public_ips() to fetch known
or available public IP addresses, according to flags.
This also drops the hack where the array from a
ctdb_public_ip_list_old was assigned to a pointer in a
ctdb_public_ip_list.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
It isn't used outside this function, so just use a local variable.
This makes create_merged_ip_list() independent of the CTDB context.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
It is only run during a takeover run and only logs errors. It doesn't
actually do anything to fix potential errors. The takeover run should
fix any inconsistencies anyway.
Instead, leave a comment in the recovery daemon's monitoring loop to
add proper remote IP verification later.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is unnecessary. IP allocation state already has a node count and
"i" is already a PNN.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Deleted (and other inactive) nodes will have an empty list of known
IP addresses.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This pointer is for an array that is always allocated. The check is
meant to skip a node that has no IP addresses. However, when there
are no IP addresses the loop below will not do anything anyway.
Add this as a check at the beginning of the function instead.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
In case of database recovery failure, if there are no banning credits
assigned, then the async computation is never terminated. The else
condition is missing in (max_credits >= NUM_RETRIES) check.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Jun 24 09:56:23 CEST 2016 on sn-devel-144
(showing what is the rule and what is the exception)
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Tue Jun 21 11:48:29 CEST 2016 on sn-devel-144
Once DB_PUSH_START is processed as part of recovery, push_started
flag tracks if there are multiple attempts to send DB_PUSH_START.
In DB_PUSH_CONFIRM, once the record count is confirmed, all information
related to DB_PUSH should be reset. However, The push_started flag was
not reset when the push_state was reset.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Wed Jun 8 14:31:52 CEST 2016 on sn-devel-144
BUG: https://bugzilla.samba.org/show_bug.cgi?id=11956
In do_recovery, after the recovery and takeover is complete, recoverd
event is triggered. When the parallel database recovery was separated,
ctdb_recovery_helper implemented sending END_RECOVERY control which
causes recoverd event to be triggered. So when there is parallel database
recovery, recoverd event is triggered twice.
Instead move the call to run_recovered_eventscript() explicitly in
the serial recovery code path. This avoids the duplication trigger of
recoverd event.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Wed Jun 8 04:52:18 CEST 2016 on sn-devel-144
The recovery lock helper must exit when it notices its parent is gone.
However, that can take a few seconds.
The usual way of terminating the recovery daemon is for the main ctdbd
to send it a SIGTERM. Installing a handler is nice and simple.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If the process holding the recovery lock terminates unexpectedly then
the recovery daemon needs to know that the lock is no longer held.
While here, rename hold_reclock_handler() to take_reclock_handler() so
there is a clear difference between the two handler names.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This makes the API more general. If they are needed in a handler then
they can be in the private data.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This will allow a simplification of the cluster mutex API, so the
private data can be registered when calling ctdb_cluster_mutex().
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
It won't be called more than once by the cluster mutex code.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
After the first activity on the file descriptor, ignore any subsequent
activity. Single-shot handlers are easier to write.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
It isn't necessarily a file.
Don't bother changing the control, since it doesn't pervade the code.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Setting the recovery lock file at startup can be done more simply.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Support for updating the recovery lock is being removed because it
isn't possible to recover from failure.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If the recovery lock setting is not consistent with that of the
recovery master then abort.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The timeout RecoverTimeout (default 120) is used for control messages
sent during the recovery. If any of the nodes does not respond to any
of the recovery control messages for RecoverTimeout seconds, then it
will cause a failure of recovery of a database. Recovery helper will
retry the recovery for a database 5 times.
In the worst case, if a database could not be recovered within 5 attempts,
a total of 600 seconds would have passed. During this time period other
timeouts will be triggered causing unnecessary failures as follows:
1. During the recovery, even though recoverd is processing events,
it does not send a ping message to ctdb daemon. If a ping message is
not received for RecdPingTimeout (default 60) seconds, then ctdb will
count it as unresponsive recovery daemon. If the recovery daemon
fails for RecdFailCount (default 10) times, then ctdb daemon will
restart recovery daemon. So after 600 seconds, ctdb daemon will
restart recovery daemon.
2. If ctdb daemon stays in recovery for RecoveryDropAllIPs (default 120),
then it will drop all the public addresses. This will cause all
SMB client to be disconnected unnecessarily. The released public
addresses will not be taken over till the recovery is complete.
To avoid dropping of IPs and restarting recovery daemon during a delayed
recovery, adjust RecoverTimeout to 30 seconds and limit number of
retries for recovering a database to 3. If we don't hear from a node
for more than 25 seconds, then the node is considered disconnected.
So 30 seconds is sufficient timeout for controls during recovery.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Mon Jun 6 08:49:15 CEST 2016 on sn-devel-144
If the node becomes stopped or banned after recovery is marked
active, then it will never freeze the databases, and hence the
node will keep banning itself indefinitely, until ctdbd is restarted.
This is a regression from 4.3, introduced with
b4357a79d9
and
d8f3b490bb
BUG: https://bugzilla.samba.org/show_bug.cgi?id=11945
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Wed Jun 1 17:36:12 CEST 2016 on sn-devel-144
This adapts the debug message in local_node_got_banned
to reflect what the function is currently doing.
This message was not adapted when the function was changed.
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If a recovery is going to be done then this will be followed by a
takeover run anyway. So, there's no use doing the takeover run
checks, potentially doing a takeover run and then doing a recovery.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The recovery daemon should be less involved in the service monitoring
logic.
The cases handled here are already handled elsewhere:
* When a node becomes unhealthy/healthy the monitoring code will
trigger a takeover run
* When a node is disabled/enabled the ctdb CLI tool will trigger a
takeover run
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
It will just become healthy again in the next monitor cycle.
Instead, let the recovery master ban it if the problem persists.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Banning is now handled by the takeover code sending banning credit
messages.
This commit makes a change in behaviour quite obvious. Takeover runs
were initiated from several locations in the code but banning was only
done from one of these locations. Now banning can be done from any
failed takeover run.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Post-process failues and only send banning credits to the node with
the most failures.
If there is a widespread problem or a problem on the recovery master
node then this should help avoid banning all the nodes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This will allow banning credits assignments to be limited according to
some criteria.
Note that this only matters when multiple controls are sent to each
node: RELEASE_IP and TAKEOVER_IP. This doesn't change the behaviour
for IPREALLOCATED.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Banning credits are now assigned by takeover runs called from all
locations in the recovery daemon. Previously this only happened from
one of the callers. When separating out the takeover run code the
behaviour should be consistent.
The callback (and corresponding data) passed to ctdb_takeover_run() is
now ignored. Dropping this will allow the interface between the
recovery daemon and IP takeover to be simplified.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Probably due to oversight, this is currently only used for the
"takeip" step.
This does consistent error handling and provides a layer of
indirection to the passed callback, so use it for "releaseip" and
"ipreallocated" steps too.
The callback data now needs to be initialised before the first
possible jump to "ipreallocated".
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Abstract out the initialisation of the callback data. Later, we'll
need to do it multiple times or move it.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The ipreallocated control has been in CTDB for a long time.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Now all the IP takeover code for non-master node is in this function.
The function can always be renamed to something more suitable.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri May 6 15:10:59 CEST 2016 on sn-devel-144
Update log levels and messages, comments and wrapping of long lines.
No functional changes.
Note that interfaces_have_changed() already does adequate logging.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
When public IP checking is disabled, verify_local_ip_allocation()
still retrieves known IP addresses and runs through a loop that does
nothing.
Instead, completely skip the retrieval and checking loop.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This makes verify_local_ip_allocation() self-contained and simplifies
main_loop().
Due to indentation changes, this commit is most easily read when
ignoring whitespace.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
There is no need to return one of several states and then trigger an
election for one of those return states. Have the recovery master
validation trigger the election directly and just return whether
monitoring should continue.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Change this to return just 0 or -1. It isn't monitoring anything.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
update_local_flags() never returns MONITOR_ELECTION_NEEDED, so drop
this entire if-statement.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Deferred attach processing is done unconditionally at this point. It
is then done again if recovery lock checking is done and completes
successfuly. If the recovery lock checking fails then it should not
be done at all.
Move this processing so it is done with the early exit when the
recovery lock is not being used.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The cluster mutex code already passes the latency and expects the
handler to update the statistics.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
ctdb_recovery_have_lock(), ctdb_recovery_lock(),
ctdb_recovery_unlock() are only used by recovery daemon, so move them
there.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This means that the cluster mutex handle can now be treated as opaque.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
All of the ctdb_cluster_mutex_* infrastucture can now handle an
arbitrary mutex.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The underlying change is to allow the cluster mutex argstring to
optionally contain a helper command. When the argument string starts
with '!' then the first word is the helper command to run. This is
now the standard way of changing the helper from the default.
CTDB_CLUSTER_MUTEX_HELPER show now only be used to change the location
of the default helper when testing.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is currently just treated as the name of a lock file. However,
it is really some arbitrary arguments to lock helper.
Therefore, it should be parsed and passed as separate arguments to the
lock helper.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Replace the file descriptor for the recovery lock in the CTDB context
with the cluster mutex handle, where non-NULL means locked.
Attempting to take the recovery lock is now asynchronous and no longer
blocks the recovery daemon.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Unlike fcntl(2), some other helper might need to explicitly take
action to release a mutex. This can be done by catching SIGTERM.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This implements the type of fcntl locking that the recovery lock uses.
The intent is to use it for multiple locks and allow the choice of
helper to be configured.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is currently only used to check whether the recovery lock can be
taken. However, name it more generally in anticipation of using it
for general cluster mutex taking and testing.
No functional changes. A couple of debug message simplifications and
code rearrangements.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is used to reply to the recmode control for all the different
cases. The callers can later be generalised to use a pointer, which
can then be used for recovery lock handling in different contexts.
Note that the handle is now freed in set_recmode_handler() rather than
the callbacks.
There is one difference in behaviour. Deferred attach calls are now
processed in the timeout case, where they weren't before. That's a
bug fix!
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
'0' = Child took the mutex
'1' = Unable to take mutex - contention
'2' = Unable to take mutex - timeout
'3' = Unable to take mutex - error
This is a straightforward API. When the child is generalised to an
external helper then this makes it easier for a helper to be, for
example, a simple script.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Use the more general name "cluster mutex", since we are likely to end
up with more than one cluster-wide lock. There will probably be a
dedicated recovery lock, held only during recovery, and also a second
lock that is held by the master node. Currently one lock is used for
both purposes.
At the moment the struct and functions are involved with setting the
recovery mode. However, they'll be abstracted out to more generally
deal with the cluster mutexes, so "recmode" -> "cluster_mutex". Drop
"set" from names, since this is used to test the lock. Also drop
"ctdb" prefix from functions, since they are local to this file. The
struct will eventually be a long-lived handle that will release the
mutex when freed, so name it accordingly.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
In particular, LVS won't work at all if there are no public IP
addresses.
This is a temporary solution until a generic reconfiguration hook is
implemented.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Why allocate all that memory and transfer all that data across the
socket?
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
These controls have never been used and also they do not use the server_id
structure defined in samba. In future, similar controls can be added to
register/unregister using proper server_id structure.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
ctdb->idr and ctdb->srv get initialized as part of ctdb_init() called
from ctdb_cmdline_init().
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This allows common.h and ctdb_private.h to be dropped.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This will be used in a standalone helper.
Don't worry that the API isn't clean and opaque. All of the code will
eventually move into the helper and will no longer be used by the
daemon.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This function knows nothing about CTDB contexts or VNNs, so it can be
used elsewhere.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The destructor used in this instances needs a CTDB context and a VNN.
However, destructors used in other cases may need different data.
For this instance create a local structure to hold the required data.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
We don't want this code to depend on a CTDB context, so don't go
looking there for an event context.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
If one or more nodes are misbehaving during recovery, keep track of
failures as ban_credits. If the node with the highest ban_credits exceeds
5 ban credits, then tell recovery daemon to assign banning credits.
This will ban only a single node at a time in case of recovery failure.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Mar 25 06:57:32 CET 2016 on sn-devel-144
This will be called from recovery helper to assign banning credits to
misbehaving node.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This abstraction uses capabilities of the remote nodes to either send
older PUSH_DB controls or newer DB_PUSH_START and DB_PUSH_CONFIRM
controls.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This abstraction depending on the capability of the remote node either
uses older PULL_DB control or newer DB_PULL control.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Also, rename traverse function and traverse state for recdb_records
consistently.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This variable is used to set the dmaster value for each record in
recdb_traverse().
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This will be used to limit the size of record buffer sent in newer
controls for recovery and existing controls for vacuuming.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Interface names that are too long will be truncated by strncpy(3)
later on. It is better to validate the length of each new interface
name to ensure it will be usable.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Jeremy Allison <jra@samba.org>
If set, this was used to setup an IP takeover run on a timer after
certain updates to the public IP address configuration (e.g. "ctdb
addip").
However, "ctdb reloadips" completely manages public IP reconfiguration
and avoids the anomalies that DeferredRebalanceOnNodeAdd was
introduced to work around.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is undocumented and is not needed. It was a workaround for
trying to ensure public IP addresses are properly rebalanced after
running "ctdb addip" on multiple nodes. "ctdb reloadips" is a better
solution.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
During the recovery process, the timeout value for sending all controls
is decided by RecoverTimeout tunable. So in the recovery process,
first get the tunables, so the control timeout gets set correctly.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Use real-time priority only for obtaining record and database locks.
Do not open databases with real-time priority as it can cause thundering
herd on fcntl lock while opening tdb database. Also relinquish real-time
priority after the lock is obtained.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Mon Mar 7 11:29:00 CET 2016 on sn-devel-144
CTDB releases all IPs in following cases: starting up, shutting down,
node gets banned, node does not come out of recovery for a long time.
Always inform samba when CTDB releases IP addresses.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
CTDB already notifies Samba with RELEASE_IP message. Samba can take
appropriate action based on that.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
It does not make sense to update this statistic for the timeout case,
since this could skew the statistic. To keep it simple, just update
it for the usual case where there is lock contention, since this is
the usual case. So the daemon statistic measures time to test the
lock and the corresponding recovery daemon statistic measures time to
take the lock.
Additionally, the recovery daemon will eventually use this code to
take the lock, and the method of updating the latency statistic will
need to be pushed further out to a configurable handler that depends
on the calling context.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Feb 23 10:32:06 CET 2016 on sn-devel-144
Have 0 indicate that the lock was taken. This allows non-zero values
to be used to indicate why the lock could not be taken. EACCES means
lock contention.
For now use just EACCES to cover all failures, since
ctdb_recovery_lock() returns a bool and details of other errors will
be lost. ctdb_recovery_lock() will undergo some big changes, so don't
try to fix this now.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This currently returns an incorrect error when the expected number of
bytes are not read. Separate out the different cases to clarify the
logic and avoid reporting the wrong error.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is already done before the destructor is assigned.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The callbacks that use this value are only ever called if recovery
mode is being set to NORMAL. So do not check if recmode is NORMAL
either.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The child process writes the status into the pipe before looping to
wait.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Where possible, this should no longer be used.
struct ctdb_public_ip_list is a fixed size structure and introduces an
extra level of indirection. This means one level of indirection can
be dropped for known_public_ips and available_public_ips.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Feb 12 08:40:21 CET 2016 on sn-devel-144
This matches the behaviour during serial database recovery.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Feb 11 08:01:14 CET 2016 on sn-devel-144
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Uri Simchoni <uri@samba.org>
Autobuild-User(master): Uri Simchoni <uri@samba.org>
Autobuild-Date(master): Tue Feb 9 22:28:08 CET 2016 on sn-devel-144
This reverts commit 0ff90f4fac.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=11707
The checks against database generation are not required since
the global generation is updated as part of updating vnnmap
before the actual database recovery. This change was done in
5aab31a39a.
Checking only against the database generation is incomplete. It can
cause CTDB to abort if the following sequence of events happen.
- CTDB gets REQ_DMASTER packet (gen1)
This packet processing gets deferred to get a record lock
- CTDB goes into recovery, marks RECOVERY_ACTIVE
CTDB recovery helper updates vnnmap (gen2)
- CTDB processes REQ_DMASTER packet (gen1)
The check against database generation (gen1) succeeds.
The check for lmaster is now invalid because VNNMAP has changed.
This will cause CTDB to abort due to protocol error.
Reverting the patch stops processing packets of older generation before
they get into call processing.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Tue Feb 9 12:39:24 CET 2016 on sn-devel-144
ctdb/server/ipalloc_lcp2.c:264:29: warning: 'minimbl' may be used uninitialized
in this function [-Wmaybe-uninitialized]
Guenther
Signed-off-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sun Feb 7 00:56:44 CET 2016 on sn-devel-144
Commit cfa0ffe780 introduced a memory
leak. Never assume...
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Michael Adam <obnox@samba.org>
The first element of these structures is a 32-bit PNN. On 64-bit
systems this field can be followed by 32-bits of padding. When the
structures are copied this can cause uninitialised memory to be
copied.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Michael Adam <obnox@samba.org>
Thousands of these can be generated each second, rendering INFO level
debugging useless.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Michael Adam <obnox@samba.org>
Shorter temporary variables for compactness/readability. "tmp_ip" is
5 characters longer than "t". In each for statement it is used 4
times, so costs 20 characters. Save those extra characters so that
future edits will avoid going over 80 columns.
Tweak whitespace for readability, rewrap some code.
No functional changes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
As per the comment:
If the IP address is hosted on this node then remove the connection.
Otherwise this function has been called because the server IP
address has been released to another node and the client has exited.
This means that we should not delete the connection information.
The takeover node processes connections too.
This doesn't matter at the moment, since the empty connection list for
an IP address that has been released will never be pushed to another
node. However, it matters if the connection information is stored in
a real replicated database.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
In a subsequent commit ctdb_takeover_client_destructor_hook() needs to
know the VNN. So just have both callers of
ctdb_remove_tcp_connection() do the lookup and pass in the VNN.
This should cause no change in behaviour.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Tickle list updates are broadcast to all connected nodes and are
accepted even when received on the same node that sent them. This
could actually lead to lost connection information when information
about new connections is received while an update is in-flight.
Instead, return early when the IP is hosted on the current node, since
it is the only one that could have sent the update.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
It hasn't worked since commit cda5f02c7c
in 2009, which reworked the banning code. Since then
ctdb_control_modflags() has contained a comment saying:
/* we don't let other nodes modify our BANNED status */
Unbanning all nodes originally occurred here when the recovery master
role moved to a new node. The logic could have been meant for the
case when the old recovery master was malfunctioning, so got banned.
If any other nodes had been banned by this recovery master then they
would be unbanned. However, this would also unban the old recovery
master, which is probably suboptimal. The logic would also trigger if
a node was banned for a good reason and then the recovery master was
stopped. So, apart from doing nothing, the logic is too simplistic so
might as well be removed.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The banning code caters for the case where the node specified in the
bantime data is not the node receiving the control. This never
happens. There are 2 places where ctdb_ctrl_set_ban() is called: the
ctdb CLI tool and the recovery daemon. Both pass the same node in the
bantime data that they are sending the control to. There are no plans
to do anything more elaborate, so just delete the handling of this
special case.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This can be easily decomposed into 2 separate arrays.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Nov 23 05:34:55 CET 2015 on sn-devel-104
This puts all of the memory allocation for ipalloc_state into its init
function. This also simplifies the code because
set_ipflags_internal() can no longer fail because it no longer
allocates memory.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is cleaner than returning ipflags and assigning them into
ipalloc_state afterwards.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Instead of local or passed temporary contexts.
This has the side effect of making ipalloc_state available inside the
modified functions, making future use of ipalloc_state simpler.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The only likely failure is out of memory, so just return boolean
value.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
As do the functions called below it. They no longer need a CTDB
context.
create_merged_ip_list() now takes both a CTDB context and an
ipalloc_state.
Drop ipalloc_state from CTDB context. So the substitution in the
code is:
ctdb->ipalloc_state -> ipalloc_state
Tweak the test code to match.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
For various reasons create_merged_ip_list() needs a CTDB context.
This is difficult to resolve now for a few reasons, including:
* The ip_tree needs somewhere to live.
It isn't very useful in its current form. However, in the future
real remote IP monitoring will probably be added back, so leave it
around.
* It uses node flags from the ctdb_node structure.
This could be changed by putting a node map into ipalloc_state
and referencing that.
For now, it is easier to move it out to where there will be a CTDB
context available for the forseeable future. ctdb_takeover_run() will
need one as long as the current client interface is used.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The IP allocation algorithms need the value of this tunable, so copy
it to avoid needing the CTDB context.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Algorithm-related tunables from the CTDB context no longer need to be
accessed in the allocation logic.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Most of the IP allocation code does not need a CTDB context. However,
temporarily hang this off the CTDB context and make only the changes
relating to known/available IP address. This makes those logic
changes obvious without burying them in function type changes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Starting to untangle cluster management, database recovery and public
IP allocation. This is a non-trivial subset of the cluster management
code that runs in the recovery daemon on all nodes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Nov 16 11:47:45 CET 2015 on sn-devel-104
Capabilities are used when computing an election result so having them
up-to-date seems like a good idea.
Also update several instances of an ambiguous comment.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The VNN map is only needed on the recovery master, so no need for all
recovery daemons to retrieve it.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is already handled in update_recovery_lock(), which is called
immediately before.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The only non-obvious part here is dropping the setting of the nodemap
local variable to NULL. If the following control succeeds then it is
set, otherwise return and it doesn't matter.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
That is, using CTDB_CURRENT_NODE makes this more obvious.
Also fix incorrect error messages.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Each recovery daemon knows who the recmaster is and is in sync with
its local daemon. The recovery master is running this check so do not
bother checking with its local daemon - both agree that it is the
recovery master.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The recovery daemon already knows which node is the master. This
relies on rec->recmaster being correctly initialised and correctly set
during elections.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Recovery should not do cluster management functions. Setting the
recovery master should only be done via an election.
Main loop will determine if recovery master is inconsistent across the
cluster and force an election if necessary.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The recovery daemon pushes knowledge of recovery master election
progress/result to local daemon. It then retrieves that information
again.
Instead, have the recovery daemon reliably track election
progress/result in rec->recmaster so it doesn't need to be retrieved.
Be careful to maintain consistency by only doing this when the local
daemon has been updated.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
There can be no holes in the nodemap. Even if a node has been deleted
it will take a slot in the nodemap. The only exception is that the
nodemap shrinks if nodes are deleted from the end. That should never
include the master because a node should be shutdown before being
deleted, and an election should already have take place.
To avoid walking off the end of the nodemap nodes array just confirm
that the master node's PNN is a valid index into the array. No need
to walk through the nodemap.
After this, in this section of the code j is now invalid. So use the
master's PNN to index into the nodemap. This is safe.
In the process, clean up some log messages to avoid saying "Force
reelection". It's just an "election".
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This only applies to monitor events so renaming clarifies this.
Note that this change is not backward compatible. Users with
CTDB_SET_EventScriptTimeoutCount=<n>
in their configuration will get failures when starting CTDB but the
cause will be clearly logged.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
It is only updated for monitor events, so it is meaningless here.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Checking runstates is unnecessary now that nodes that are not RUNNING
will return no available IP addresses. I have no idea why I didn't do
it this way originally.
Tweak the test code to cope with this.
Note that this is a backward-incompatible change. If new and old
versions of CTDB are running together in a cluster and a new node
takes over as recovery master then old nodes will be able to host
public IP addresses before they are in RUNNING runstate. This is
mitigated by the bias towards recovery master stability in elections.
If it is important that nodes do not host IPs until they are RUNNING
then do not restart nodes running the old version.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The available IPs list is now only non-empty for nodes that are in
RUNNING runstate. So, to avoid running the IP allocation algorithm
when there are no available available IPs, explicitly check for
available IPs rather than checking runstates.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This will allow wonderful simplification (i.e. removal) of some of the
runstate checking in the takeover run code.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This variable is used for adding a prefix to log entries from various
child processes.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Michael Adam <obnox@samba.org>
These functions are only used in the ctdb daemon code.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Michael Adam <obnox@samba.org>
This function can only called from ctdb daemon.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Michael Adam <obnox@samba.org>
This is currently done before each IP takeover run, so just factor it
in.
ctdb_reload_remote_public_ips() becomes static.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Thu Nov 12 09:28:45 CET 2015 on sn-devel-104
This will help to untangle known and available public IP lists from
the CTDB context.
verify_remote_ip_allocation() needs a forward declaration.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Remote IP validation is only called when a takeover run is about to
happen anyway, so don't bother flagging one. Given that a takeover
run isn't being triggered, also drop the test that checks if takeover
runs are disabled. These are the only uses of the rec argument, so
drop it.
One possible further simplification would be to remove this function
because it doesn't accomplish anything. However, it is worth leaving
it as a reminder that remote IP validation should be done properly at
some time in the future.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
It is only used by the caller to print a message that includes the
culprit. However, ctdb_reload_remote_public_ips() already prints
perfectly good messages and they include the culprit.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
No need to do it immediately. It will happen in less than a second.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This is not a recovery, so do not run "startrecovery and "recovered"
events. There are other IP takeover runs where these are not run.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
The recovery start/end times used in the checks at the top of
verify_local_ip_allocation() are set by the START_RECOVERY and
END_RECOVERY controls. A couple of takeover runs escape the checks
because they were added later and are not surrounded by these
controls.
Recovery and IP allocation need to be untangled from each other, so
recovery-related events should not be relied on for IP allocation.
This means the solution is not to add these where they are "missing".
The concern that the checks are addressing is to avoid local IP
verification when IP addresses are in a state of flux. Takeover runs
on non-master nodes are already disabled while a takeover run is in
progress, so local IP verification is already skipped in that case.
The other case is the master node, which will be busy with the
takeover run, rather than running main_loop().
The other issue is races. verify_local_ip_allocation() takes a
non-zero amount of time to fetch IP addresses from the local CTDB
daemon and during this time a recovery or takeover run can start, but
a takeover run can still be triggered. The current tests do not stop
this.
Apart from all of this, with most reasonable public IP address
configurations, an extra takeover run will be a no-op so is not a
cause for concern.
It is safe to drop these checks.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Nov 6 13:43:45 CET 2015 on sn-devel-104
This gets rid of the duplicate definitions from ctdb_protocol.h.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This is to avoid clash with samba structure server_id.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
When a node gets banned, it should go into recovery and freeze all
databases. We rely on the recovery daemon to detect the banned state
and put the node in recovery and freeze all databases.
Recent change in b4357a79d9 took explicit
freezing out of banning code but left the setting of recovery mode
to ACTIVE. Recovery daemon will freeze databases only if the recovery
mode is NORMAL. Recovery mode set to ACTIVE is an indication that the
freeze has started.
Do not set the recovery mode to ACTIVE in banning. Let recovery daemon
take care of it.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Oct 30 10:32:38 CET 2015 on sn-devel-104
This groups function prototypes for common client/server functions in
common/common.h and removes them from ctdb_private.h.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
When building standalone ctdb from git repo, samba_version_file correctly
includes git sha in VERSION string. When building standalone ctdb from
tarball, samba_version_file puts UNKNOWN in the VERSION string.
Use the packaged include/ctdb_version.h file to set the correct git sha.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Instead of includes.h, include the required header files explicitly.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
This groups function prototypes for system specific functions in
common/system.h and removes them from ctdb_private.h.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
While 0000 is possible from the UNIX/POSIX point of view,
these permissions create problems in an environment with
selinux enabled, which is more strict.
This aligns the perms of the read only tracking db with other
internal dbs.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=11577
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Wed Oct 28 06:13:09 CET 2015 on sn-devel-104
Centrally define all the default capabilities to make the defaults
crystal clear. Capability-related command-line options now have a
direct correspondence rather than a reverse correspondence.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Michael Adam <obnox@samba.org>
Once the databases are recovered, all the pending calls are resent.
If the vnnmap is not updated, then the nodes can redirect calls to nodes
that are not part of the new vnnmap.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Oct 16 09:31:34 CEST 2015 on sn-devel-104
Recovery daemon checks if it is the recovery master before performing
certain checks. During those checks it's possible that re-election can
change the recmaster. In such a case, the recovery daemon should never
do a database recovery.
This is not complete fix since the recovery master can still change
while the recovery is going on. The correct fix is to abort recovery
if the recovery master changes.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Wed Oct 7 17:55:05 CEST 2015 on sn-devel-104
The reason for favouring more connected node is to create a larger
cluster in case of a split brain. In split brain condition, the nodes
are not communicating across partitions and each partition will run its
own election. Among all the partitions, the node which holds the recovery
lock will eventually "win". All the other nodes which won election but
could not grab recovery lock will end up banning themselves.
This also prevents the recovery master role from bouncing between nodes
during startup when the entire cluster is restarted.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
If election occurs during SMB activity, then trying to freeze all the
databases can cause samba/ctdb deadlock which parallel database recovery
is trying to avoid.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
