1
0
mirror of https://github.com/samba-team/samba.git synced 2025-01-11 05:18:09 +03:00
Commit Graph

2591 Commits

Author SHA1 Message Date
Ronnie Sahlberg
7d39ac131b convert handling of gratious arps and their controls and helpers to
use the ctdb_sock_addr structure so tehy work for both ipv4 and ipv6

(This used to be ctdb commit 86d6f53512d358ff68b58dac737ffa7576c3cce6)
2008-06-04 15:13:00 +10:00
Ronnie Sahlberg
1c88f422d5 add a parameter for the tdb-flags to the client function
ctdb_attach()   so that we can pass TDB_NOSYNC when we attach to
a persistent database and want fast unsafe writes instead of
slow but safe tdb_transaction writes.

enhance the ctdb_persistent test suite to test both safe and unsafe writes

(This used to be ctdb commit 4948574f5a290434f3edd0c052cf13f3645deec4)
2008-06-04 10:46:20 +10:00
Ronnie Sahlberg
60a3fb926d dont bother casting to a void* private_data pointer,
just pass it as 'state' structure

(This used to be ctdb commit 1d7c3eb454e33cd17c74606c4ea011fd79959c80)
2008-05-28 13:40:12 +10:00
Ronnie Sahlberg
0b0f5bc5e6 remove another field we dont need in the childwrite_handle structure
(This used to be ctdb commit 70085523f4c35a20786023c489325554e2a6f9c1)
2008-05-28 13:31:58 +10:00
Ronnie Sahlberg
71ec7b25b0 remote a comment that is no longer relevant
remove a field in the childwrite_handle structure we dont need

(This used to be ctdb commit a53db1ec3f29f4418ff51e0f452026c12470bf93)
2008-05-28 13:30:22 +10:00
Ronnie Sahlberg
ceaf488f05 do persistent writes in a child process
(This used to be ctdb commit 2da3d1f876f5d654f849af8a3e588f5a61300c3d)
2008-05-28 13:04:25 +10:00
Ronnie Sahlberg
0941019cb7 restore a timeout value to the default settings instead of the hardcoded 3 second test value
(This used to be ctdb commit 437752d002a108bcbbf6dc8bfb5dbf16dc5f1c58)
2008-05-22 16:33:36 +10:00
Ronnie Sahlberg
dd6c9d5a78 fix some memory hierarchy bugs in allocation of the state structure for persistent writes.
since these two controls (UPDATE_RECORD and PERSISTENT_STORE) can respond
asynchronously to the control,   we can not allocate the state variable as a child off ctdb_req_control  instead we must allocate state as a child off ctdb itself
and steal ctdb_req_control so it becomes a child of state.

othervise both ctdb_req_control and also state will be released immediately after we have finished setting up the async reply and returned.

(This used to be ctdb commit 6f6de0becd179be9eb9a6bf70562b090205ce196)
2008-05-22 16:29:46 +10:00
Ronnie Sahlberg
d895f43504 cleanup of the previous patch.
With these patches, ctdbd will enforce and (by default) always use
tdb_transactions when updating/writing records to a persistent database.

This might come with a small performance degratation  since transactions
are slower than no transactions at all.

If a client, such as samba wants to use a persistent database but does NOT
want to pay the performance penalty, it can specify TDB_NOSYNC  as the
srvid parameter in the ctdb_control() for CTDB_CONTROL_DB_ATTACH_PERSISTENT.

In this case CTDBD will remember that "this database is not that important"
so I can use unsafe (no transaction) tdb_stores to write the updates.
It will be faster than the default (always use transaction) but less crash safe.

(This used to be ctdb commit 3d85d2cf669686f89cacdc481eaa97aef1ba62c0)
2008-05-22 13:12:53 +10:00
Ronnie Sahlberg
ed2cf0291d second try for safe transaction stores into persistend tdb databases
for stores into persistent databases, ALWAYS use a lockwait child take out the lock for the record and never the daemon itself.

(This used to be ctdb commit 7fb6cf549de1b5e9ac5a3e4483c7591850ea2464)
2008-05-22 12:47:33 +10:00
Ronnie Sahlberg
92a0c0fc13 lowe the loglevel for the warning that releaseip was called for a non-public address.
the address might be a public address on a different node so no need to fiull up the logs with thoise messages

(This used to be ctdb commit c8181476748395fe6ec5284c49e9d37b882d15ea)
2008-05-21 11:50:41 +10:00
Ronnie Sahlberg
9c23bf7776 lower the loglevel for when we have "tickles" for an ip address that is not a public address on the local node (it may be a public address on other nodes)
(This used to be ctdb commit 1360c2f08a463f288b344d02025e84113743026d)
2008-05-21 11:44:50 +10:00
Ronnie Sahlberg
f4fd4d0af8 dont disable/enable monitoring for each eventscript, instead
just disable the monitoring during the "startrecovery" event and enable it again once recovery has completed

(This used to be ctdb commit 68029894f80804c9f31fc90ed0c1b58f75812c3d)
2008-05-16 08:20:40 +10:00
Ronnie Sahlberg
37b681627e dont check whether the "recovered" event was successful or not
since  this event wont run unless the recovery mode is normal   but we
can not know what the recovery mode will be in the future on a remote node
so since we issue these commands   that will execute in the future at some other node
it is pointless to try to check if it worked or not

in particular if "failure to successfully run the eventscript" would then trigger a full new recovery which is disruptive and expensive.

(This used to be ctdb commit 2c292039a0139dcf5bb2bd964eb6f8902d094c50)
2008-05-15 15:01:01 +10:00
Ronnie Sahlberg
f2661ec859 remove some unnessecary tests if ->vnn is null or not
(This used to be ctdb commit f0169ac8166a19d65ce254496e21d095aed87c2f)
2008-05-15 13:28:19 +10:00
Ronnie Sahlberg
09cc3ccff5 Update some debug statements. Dont say that recovery failed if the failed function was invoked from outside of recovery
(This used to be ctdb commit 3038d0b74895b51af4f85f2f304508ed16d245f4)
2008-05-15 12:28:52 +10:00
Ronnie Sahlberg
3e14bbcce6 Merge git://git.samba.org/tridge/ctdb
(This used to be ctdb commit d5fb4489f83f1f956b2c083cfad1861c5ddde283)
2008-05-15 08:02:51 +10:00
Andrew Tridgell
8ec3665231 put the return in the right place
We were refusing the 'startrecovery' event

(This used to be ctdb commit 788d38812d73729f11d12e9812b16092c0ae4123)
2008-05-14 22:05:09 +10:00
Andrew Tridgell
e465110f95 Fix the chicken and egg problem with ctdb/samba and a registry smb.conf
This attempts to fix the problem of ctdb event scripts blocking due to
attempted access to the ctdb databases during recovery. The changes are:

  - now only the 'shutdown' and 'startrecovery' events can be called
    with the databases locked in recovery. The event scripts must ensure
    that for these two events no database access is attempted

  - the recovered, takeip and releaseip events could previously be called
    inside a recovery. The code now ensures that this doesn't happen, delaying
    the events till after recovery has finished

  - the 50.samba event script now avoids using testparm unless it is really
    needed

This needs extensive testing.

(This used to be ctdb commit e3cdb8f2be6a44ec877efcd75c7297edb008a80b)
2008-05-14 20:57:04 +10:00
Ronnie Sahlberg
909ff219e0 Start implementing support for ipv6.
This enhances the framework for sending tcp tickles to be able to send ipv6 tickles as well.

Since we can not use one single RAW socket to send both handcrafted ipv4 and ipv6 packets, instead of always opening TWO sockets, one ipv4 and one ipv6 we get rid of the helper ctdb_sys_open_sending_socket() and just open (and close)  a raw socket of the appropriate type inside ctdb_sys_send_tcp().
We know which type of socket v4/v6 to use based on the sin_family of the destination address.

Since ctdb_sys_send_tcp() opens its own socket  we no longer nede to pass a socket
descriptor as a parameter.  Get rid of this redundant parameter and fixup all callers.

(This used to be ctdb commit 406a2a1e364cf71eb15e5aeec3b87c62f825da92)
2008-05-14 15:47:47 +10:00
Ronnie Sahlberg
b8eb5925cf Try to use tdb transactions when updating a record and record header inside the ctdb daemon.
If a transaction could be started, do safe transaction store when updating the record inside the daemon.
If the transaction could not be started (maybe another samba process has a lock on the database?) then just do a normal store instead (instead of blocking the ctdb daemon).

The client can "signal" ctdb that updates to this database should, if possible, be done using safe transactions by specifying the TDB_NOSYNC flag when attaching to the database.
The TDB flags are passed to ctdb in the "srvid" field of the control header when attaching using the CTDB_CONTROL_DB_ATTACH_PERSISTENT.

Currently, samba3.2 does not yet tell ctdbd to handle any persistent databases using safe transactions.

If samba3.2 wants a particular persistent database to be handled using
safe transactions inside the ctdbd daemon, it should pass
TDB_NOSYNC as the flags to the call to attach to a persistent database
in ctdbd_db_attach()     it currently specifies 0 as the srvid

(This used to be ctdb commit 8d6ecf47318188448d934ab76e40da7e4cece67d)
2008-05-12 13:37:31 +10:00
Ronnie Sahlberg
adf40341a7 ctdb->methods becomes NULL when we shutdown the transport.
If we shutdown the transport   and CTDB later decides to send a command out
for queueing, the call to ctdb->methods->allocate_pkt() will SEGV.

This could trigger for example when we are in the process of shuttind down CTDBD and have already shutdown the transport but we are still waiting for the
"shutdown" eventscripts to finish.
If the event scripts now take much much longer to execute for some reason, this
race condition becomes much more probable.

Decorate all dereferencing of ctdb->methods->    with a check that ctdb->menthods is non-NULL

(This used to be ctdb commit c4c2c53918da6fb566d6e9cbd6b02e61ae2921e7)
2008-05-11 14:28:33 +10:00
Ronnie Sahlberg
f196afd58b fix a bug where the public ip addresses of the cluster would not be redistributed across the cluster after a recovery was performed.
Remove a bogus check inside the recovery daemon that ONLY redistributed public addresses IFF the local node had/served public addresses.
This was a valid optimization long ago when we enforced that all nodes must use the same public addresses file   but is invalid today where we can have different public addresses configs on all nodes  and even have some nodes that do NOT use public addresses at all.

(This used to be ctdb commit 5833e6b99d9afaf35dc8354df8676b9115418b23)
2008-05-09 13:41:31 +10:00
Andrew Tridgell
abe6d816bb fixed realloc bug
Should always use type safe talloc functions when possible. In this case we were allocating bytes instead of uint32_t

(This used to be ctdb commit cb14ee57dd0a589242da1ac2830bb7939df460a5)
2008-05-08 19:59:24 +10:00
Ronnie Sahlberg
92b61cd7d5 Expand the client async framework so that it can take a callback function.
This allows us to use the async framework also for controls that return
outdata.

Add a "capabilities" field to the ctdb_node structure. This field is
only initialized and kept valid inside the recovery daemon context and not
inside the main ctdb daemon.

change the GET_CAPABILITIES control to return the capabilities in outdata instead of in the res return variable.

When performing a recovery inside the recovery daemon, read the capabilities from all connected nodes and update the ctdb->nodes list of nodes.
when building the new vnnmap after the database rebuild in recovery, do not include any nodes which lack the LMASTER capability in the new vnnmap.
Unless there are no available connected node that sports the LMASTER capability in which case we let the local node (recmaster) take on the lmaster role temporarily (i.e. become a member of the vnnmap list)

(This used to be ctdb commit 0f1883c69c689b28b0c04148774840b2c4081df6)
2008-05-06 15:42:59 +10:00
Ronnie Sahlberg
2c23959616 make sure we lose all elections for recmaster role if we do not have the recmaster capability.
(unless there are no other node at all available with this capability)

(This used to be ctdb commit 8556e9dc897c6b9b9be0b52f391effb1f72fbd80)
2008-05-06 13:56:56 +10:00
Ronnie Sahlberg
6863c8f573 close and reopen the reclock pnn file at regular intervals.
handle failure to get/hold the reclock pnn file better and just
treat it as a transient backend filesystem error and try again later
instead of shutting down the recovery daemon

when we have lost the pnn file   and if we are recmaster
release the recmaster role so that someone else can become recmaster isntead

(This used to be ctdb commit e513277fb09b951427be8351d04c877e0a15359d)
2008-05-06 13:27:17 +10:00
Ronnie Sahlberg
80f85dc390 Monitor that the recovery daemon is still running from the main ctdb daemon
and if it has terminated, then we shut down the main daemon as well

(This used to be ctdb commit 7e587acaf8006254e89ff9b4bf48454821c85863)
2008-05-06 11:19:17 +10:00
Ronnie Sahlberg
d86e48d5ff Add ability to disable recmaster and lmaster roles through sysconfig file and
command line arguments

(This used to be ctdb commit 34b952e4adc53ee82345275a0e28231fa1b2533e)
2008-05-06 10:41:22 +10:00
Ronnie Sahlberg
a9c45f9513 Add a capabilities field to the ctdb structure
Define two capabilities :
can be recmaster
can be lmaster
Default both capabilities to YES

Update the ctdb tool to read capabilities off a node

(This used to be ctdb commit 50f1255ea9ed15bb8fa11cf838b29afa77e857fd)
2008-05-06 10:02:27 +10:00
Ronnie Sahlberg
073f4a7cb4 when a node disgrees with us re who is recmaster
make it mark that node as a lcuprit so it eventually gets banned

(This used to be ctdb commit eff3f326f8ce6070c9f3c430cd14d1b71a8db220)
2008-04-22 00:56:27 +10:00
Ronnie Sahlberg
0e1a20b603 Revert "Revert "Revert "- accept an optional set of tdb_flags from clients on open a database,"""
remove the transaction stuff and push   so that the git tree will work

This reverts commit 539bbdd9b0d0346b42e66ef2fcfb16f39bbe098b.

(This used to be ctdb commit 876d3aca18c27c2239116c8feb6582b3a68c6571)
2008-04-10 15:59:51 +10:00
Ronnie Sahlberg
39f119b42c Revert "Revert "- accept an optional set of tdb_flags from clients on open a database,""
This reverts commit 171d1d71ef9f2373620bd7da3adaecb405338603.

(This used to be ctdb commit 539bbdd9b0d0346b42e66ef2fcfb16f39bbe098b)
2008-04-10 14:57:41 +10:00
Ronnie Sahlberg
9684befa16 Revert "- accept an optional set of tdb_flags from clients on open a database,"
This reverts commit 49330f97c78ca0669615297ac3d8498651831214.

(This used to be ctdb commit 171d1d71ef9f2373620bd7da3adaecb405338603)
2008-04-10 14:45:45 +10:00
Andrew Tridgell
dc15a9c1f6 - accept an optional set of tdb_flags from clients on open a database,
thus allowing the client to pass through the TDB_NOSYNC flag

- ensure that tdb_store() operations on persistent databases that don't
  have TDB_NOSYNC set happen inside a transaction wrapper, thus making
  them crash safe

(This used to be ctdb commit 49330f97c78ca0669615297ac3d8498651831214)
2008-04-10 15:25:48 +10:00
Ronnie Sahlberg
cd1858d126 fix compiler warning during a fatal error failing to lock down the socket
(This used to be ctdb commit 0ad22de1a614dc2d1926546027be5f5eea3381ed)
2008-04-10 09:56:49 +10:00
Ronnie Sahlberg
2da3fe1b17 From Chris Cowan
secure the domain socket and set permissions properly

(This used to be ctdb commit ac6a362fc2fc4a56b4c310478a96eb12daace176)
2008-04-10 06:51:53 +10:00
Ronnie Sahlberg
6b797f148c From Chris Cowan
Add support in AIX to track the PID of a client that connects to the unix domain socket

(This used to be ctdb commit 4c006c675d577d4a45f4db2929af6d50bc28dd9e)
2008-04-03 10:58:51 +11:00
Ronnie Sahlberg
e8e67ef576 add a mechanism to force a node to run the eventscripts with arbitrary arguments
ctdb eventscript "command argument argument ..."

(This used to be ctdb commit 118a16e763d8332c6ce4d8b8e194775fb874c8c8)
2008-04-02 11:13:30 +11:00
Ronnie Sahlberg
03d30f405d decorate the memdump output with a nice field for ctdb_client structures to show the pid of the client that attached
(This used to be ctdb commit 0d9314302d0b988b6ab5d533deef40c5b343c249)
2008-04-01 17:17:21 +11:00
Ronnie Sahlberg
27a7f854f5 add improvements to tracking memory usage in ctdbd adn the recovery daemon
and a ctdb command to pull the talloc memory map from a recovery daemon
ctdb rddumpmemory

(This used to be ctdb commit d23950be7406cf288f48b660c0f57a9b8d7bdd05)
2008-04-01 15:34:54 +11:00
Ronnie Sahlberg
0d7b34c9e5 Add two new controls to add/delete public ip address from a node at runtime.
The controls only modify the runtime setting of which public addresses a node
can server and does not modify /etc/ctdb/public_addresses.
To make the change permanent you also need to edit /etc/ctdb/public_addresses
manually.

After ip addresses have been added/deleted you need to invoke a recovery
for the ip addresses to be redistributed.

(This used to be ctdb commit f8294d103fdd8a720d0b0c337d3973c7fdf76b5c)
2008-03-27 09:23:27 +11:00
Ronnie Sahlberg
26ec64a571 fix a memory leak
allocate the memory to the 'call' context and not off the 'ctdb' context

(This used to be ctdb commit be89005bd5d13409e377d425db2aad1c0d5b3826)
2008-03-25 11:11:13 +11:00
Ronnie Sahlberg
2863d2cfd1 From M Dietz,
Add back the controls to enable/disable monitoring we used to have for debugging but removed a while ago

(This used to be ctdb commit 8477f6a079e2beb8c09c19702733c4e17f5032fe)
2008-03-25 08:27:38 +11:00
Ronnie Sahlberg
d53424731f in ctdb_call_local() we can not talloc_steal() the returned data and hang it off ctdb.
This can cause a memory leak if the call is terminated before we have managed to respond to the client.
(and the call is talloc_free()d but the data is still hanging off ctdb)

instead we must talloc_steal() the data and hang it off the call structure to avoid the memory leak.

In order to do this we must also change the call structure that is passed into ctdb_call_local() to be allocated through talloc().

This structure was previously either a static variable, or an element of a larger talloc()ed structure (ctdb_call_state or ctdb_client_call_state) so
we must change all creations of a ctdb_call into explicitely creating it through talloc()

(This used to be ctdb commit 4becf32aea088a25686e8bc330eb47d85ae0ef8f)
2008-03-19 13:54:17 +11:00
Ronnie Sahlberg
e19264ea26 change the log level for the message when someone connects to a non-public ip
(This used to be ctdb commit bc9c4f0d52e9b06aceb08cea99ed3fd20b44616c)
2008-03-13 07:54:55 +11:00
Ronnie Sahlberg
74d57f8d51 Redo the vacukming process to mkake it scalable.
Vacumming used to delete one record at a time on all nodes, that was
m*n behaviour and would require a huge storm of ctdb->ctdb controls and just wouldnt scale at all.

The new vacuming process collects all records to be deleted locally and then only sends 1 control to the other nodes. This control contains a list of all records to be deleted.

(This used to be ctdb commit 9e625ece19a91f362c9539fa73b6b2108f0d9c53)
2008-03-13 07:53:29 +11:00
Ronnie Sahlberg
a89ed0fdc2 add a new tunable 'NoIPFailback'
when this tunable is set, ip addresses will only be failed over when a node
fails. And only those ip addresses held by the failed node will be reallocated
in the cluster.

When a node becomes active again, this will not lead to any failback of ip addresses.

This can reduce the number of "ip address movements" in the cluster since we dont automatically fail an ip address back, but can also lead to an unbalanced cluster since we no longer attempt to spread the ip addresses out evenly across the active nodes.

This tuneable can NOT be active at the same time as DeterministicIPs are used.

(This used to be ctdb commit d3b8a461b15bc584fa1785eb5922de6d49d8f6c4)
2008-03-03 12:52:16 +11:00
Ronnie Sahlberg
e08519b74d when we reallocate the ip addresses for nodes, we must make sure that
a node that has been allocated to server an ip actually CAN serve that ip
(if we use differing public_addresses files on each node)

(This used to be ctdb commit fdaf7cb2d7682507fbf4c6c2b833b327c93fac08)
2008-03-03 10:53:23 +11:00
Ronnie Sahlberg
57d29f1011 add a num_connected field to the rec structure that holds the number
of connected nodes

num_active only contains the number of active nodes and would thus not count
banned nodes

(This used to be ctdb commit 06d3ce470766ef0b60d68ccd84de5437146cc147)
2008-03-03 10:24:17 +11:00
Ronnie Sahlberg
f6f7f54bd6 add a new tunable : reclockpingperiod
once every such interval :
* the recovery master on each node will uppdate the "connected" count in the
reclock count file (ctdb getreclock)
* if the node thinks it is a recovery master but it detects another node
  that is DISCONNECTED but which still holds a lock to the reclock count file
  this may mean that we have a split cluster.
  if that other node that is DISCONNECTED but still holds the lock on hte reclock
  pnn count file, is MORE connected than the local node,
  yield the recmaster role and let the other half of the lcuster take over

this add a second, last chance mechanism to detect split clusters.
IF the cluster is split but GPFS is not yet split, this mechanism makes
the largest half of the cluster become the active half.

(This used to be ctdb commit 07af425f444531942cce8abff112c1524228d287)
2008-03-03 09:19:30 +11:00
Ronnie Sahlberg
cadd95263f change recmaster from being a local variable in monitor_cluster() to be a member of the ctdb_recoverd structure
(This used to be ctdb commit b7f955338f50c92374b4f559268fb3a1a516aefa)
2008-03-03 07:53:46 +11:00
Ronnie Sahlberg
814570f904 update the reclock pnn count for how many nodes are connected to the current node once every 60 seconds
(This used to be ctdb commit bf1863cc9e2539b2c3e53c664b493b459ebfcc8b)
2008-02-29 13:14:47 +11:00
Ronnie Sahlberg
efa29c6c98 store the num_active variable (number of connected/active nodes) inside the rec
structure and avoid passing this as an extra parameter to do_recovery()

(This used to be ctdb commit 8bb229aa3b4bd41e48d4e4e2e148d8680c8ba436)
2008-02-29 12:55:20 +11:00
Ronnie Sahlberg
e0036942bc add a new file <reclock>.pnn where each recovery daemon can lock that byte at offset==pnn to offer an alternative way to detect which nodes are active instead of relying on CONNECTED being accurate.
(This used to be ctdb commit 21d3319eaf463e2a00637d440ee2d4d15f53bf09)
2008-02-29 12:37:42 +11:00
Ronnie Sahlberg
4adeafef11 add a control to get the name of the reclock file from the daemon
(This used to be ctdb commit 9effb22cc1616d684352d7ebabb359e69adb0f52)
2008-02-29 10:03:39 +11:00
Ronnie Sahlberg
7bc8007f93 add a new tunable DisableWhenUnhealthy which when set will cause a node to automatically become DISABLED anytime monitoring fails and the node becomes UNHEALTHY.
Use with caution.

(This used to be ctdb commit c20293360db67f9876b0c84e5e9e12a5868964cb)
2008-02-22 10:33:09 +11:00
Ronnie Sahlberg
f3b474cffb Add debug output to indicate why a node starts up in DISABLED state
(This used to be ctdb commit 8df75775966ead36e1073896fedeff674a6e0587)
2008-02-22 09:52:57 +11:00
Ronnie Sahlberg
39539f6044 Add a new parameter to /etc/sysconfig/ctdb
CTDB_START_AS_DISABLED="yes"

and command line argument
--start-as-disabled

When set, this makes the ctdb node to always start in DISABLED mode and will thus not host any public ip addresses.
The administrator must manually "ctdb enable" the node after it has started when the administrator wants the node to start hosting public ip addresses.

Using this option it is possible to start ctdb on a node without causing any reallocation of ip addresses when it is starting. The node will still merge with the cluster and there will still be a recovery phase but the ip address allocations will not change in the cluster.

(This used to be ctdb commit b93d29f43f5306c244c887b54a77bca8a061daf2)
2008-02-22 09:42:52 +11:00
Ronnie Sahlberg
9f99b44fd1 to make it easier/less disruptive to add nodes to a running cluster
add a new control that causes the node to drop the current nodes list
and reread it from the nodes file.
During this operation, the node will also drop the tcp layer and restart it.

When we drop the tcp layer, by talloc_free()ing the ctcp structure
add a destructor to ctcp so that we also can clean up and remove the references in the ctdb structure to the transport layer

add two new commands for the ctdb tool.
one to list all nodes in the nodesfile and the second a command to trigger a node to drop the transport and reinitialize it with the nde nodes file

(This used to be ctdb commit 4bc20ac73e9fa94ffd43cccb6eeb438eeff9963c)
2008-02-19 14:44:48 +11:00
Ronnie Sahlberg
bef60e8200 read the current debuglevel in each loop in the recovery daemon so that we
pick up when they change in the parent daemon

(This used to be ctdb commit 792d5471ff0c2947b6e66183925860de27f30eaf)
2008-02-18 19:38:04 +11:00
Ronnie Sahlberg
3f56526037 Specify and print debuglevels by name and not by number
(This used to be ctdb commit 79ad830294b8b677fbd0c5ad7ed6fbde71f74f8d)
2008-02-05 10:26:23 +11:00
Andrew Tridgell
f6e53f433b merge from ronnie
(This used to be ctdb commit e7b57d38cf7255be823a223cf15b7526285b4f1c)
2008-02-04 20:07:15 +11:00
Andrew Tridgell
9d6ac0cf55 added debug constants to allow for better mapping to syslog levels
(This used to be ctdb commit 7ba8f1dde318eab03f4257e5a89fd23e7281e502)
2008-02-04 17:44:24 +11:00
Andrew Tridgell
feb7c05734 removed dependence on dprintf
(This used to be ctdb commit c156db449218bf9432e3a6cb3ce0f617197c9069)
2008-01-29 14:31:51 +11:00
Andrew Tridgell
146d4b0db7 merge async recovery changes from Ronnie
(This used to be ctdb commit 576e317640d25f8059114f15c6f1ebcee5e5b6e2)
2008-01-29 13:59:28 +11:00
Andrew Tridgell
eb044bb1d6 make ctdb dumpmemory work remotely, and dump the talloc
memory tree to stdout. This is much more useful than putting it in the log, and also fixes
a bug where the pipe would overflow internally and cause ctdbd to lockup

(This used to be ctdb commit e236979e2162d9bd7a495086342168a696cf76c5)
2008-01-22 14:22:41 +11:00
Andrew Tridgell
d945b1af03 merge from ronnie
(This used to be ctdb commit 5f6d59b9d18c694d82591238bc7a6bb98726a3ed)
2008-01-17 16:46:56 +11:00
Ronnie Sahlberg
9625483c2d add ctdb_uptime.c
(This used to be ctdb commit 4c7153681ed4d68d601720d043f9ff95ac7647a9)
2008-01-17 16:37:05 +11:00
Ronnie Sahlberg
9055978b46 add a ctdb uptime command that prints when ctdb was started and when the
last recovery occured

(This used to be ctdb commit b86e8ccbdac044bb949c4fc2ebb27635126272a9)
2008-01-17 11:33:23 +11:00
Andrew Tridgell
5683a8d1e1 cope better with large debug dumps
(This used to be ctdb commit fc3733f8e966376f50799fd1aa7b0a8e1cf66e0e)
2008-01-16 23:06:37 +11:00
Andrew Tridgell
be9594c156 fixed handling of \r from stdout of subprocesses
(This used to be ctdb commit f1acec5db4948d8e48412a8546bb181b08a2c5fd)
2008-01-16 22:40:01 +11:00
Andrew Tridgell
0080683da8 fixed two 64bit warnings
(This used to be ctdb commit c61fe240713ae2e917f69f827c6927405f02f5d4)
2008-01-16 22:16:15 +11:00
Andrew Tridgell
97ede94e40 The recovery daemon does not need to be a realtime task
(This used to be ctdb commit f552acf7c1f9dd37eb35d9716ea3fb02304aae8f)
2008-01-16 22:08:33 +11:00
Andrew Tridgell
b62b7fcde8 added syslog support, and use a pipe to catch logging from child processes to the ctdbd logging functions
(This used to be ctdb commit 1306b04cd01e996fd1aa1159a9521f2ff7b06165)
2008-01-16 22:03:01 +11:00
Ronnie Sahlberg
5b7838d768 ctdb_control_send() does not need to take an outdata parameter
remove the outdata parameter from the function and all callers

(This used to be ctdb commit e3951337f8df2ae19cce61c954036590c7a03582)
2008-01-16 10:23:26 +11:00
Andrew Tridgell
bf9e33d4cf - catch a case where the client disconnects during a call
- track all talloc memory, using NULL context

(This used to be ctdb commit bf89c56002f5311520e91cb367753bc46e5dddc9)
2008-01-16 09:44:48 +11:00
Andrew Tridgell
6c56e9d347 fixed a memory leak in the recovery daemon
(This used to be ctdb commit 73c27cf4c62cbe44b2b8fd00f907974d0808500c)
2008-01-15 20:11:44 +11:00
Ronnie Sahlberg
ba31feaec0 split node health monitoring and checking for connected/disconnected
nodes into two separate files.

move the monitoring of keepalives for detecting connected/disconnected 
remote nodes into ctdb_keepalive.c

(This used to be ctdb commit 23a57b20c314d5f11a433cf251eb9d9de743849a)
2008-01-15 08:42:12 +11:00
Andrew Tridgell
b866a147d2 get rid of monitor_retry as well
(This used to be ctdb commit c957cf9c1d99d5d3f4ca726f7a867c829660a2b7)
2008-01-10 14:49:43 +11:00
Andrew Tridgell
538f519dba exponential backoff in health monitoring for faster startup
(This used to be ctdb commit 1b04a1f675f73b48366ba98803a58c3d8df1b6e1)
2008-01-10 14:40:56 +11:00
Andrew Tridgell
3b3fceacbe block alarm signals during critical sections of vacuum
(This used to be ctdb commit cfb14ae76f00f10d27b56c034b2247ab12d63065)
2008-01-10 09:43:14 +11:00
Andrew Tridgell
59d69bb709 only match vacuum list if on the same database
(This used to be ctdb commit 27e56955e93027534780cc7549ddb224670d82b6)
2008-01-09 10:22:20 +11:00
Andrew Tridgell
9559249e15 ensure the main daemon doesn't use a blocking lock on the freelist
(This used to be ctdb commit 73f8257906b09e6516f675883d8e7a3c455ad869)
2008-01-08 22:31:48 +11:00
Andrew Tridgell
1c91398aef ensure the recovery daemon is not clagged up by vacuum calls
(This used to be ctdb commit ff7e80e247bf5a86adda0ef850d901478449675b)
2008-01-08 21:28:42 +11:00
Andrew Tridgell
96100fcae6 added two new ctdb commands:
ctdb vacuum   : vacuums all the databases, deleting any zero length
                 ctdb records

 ctdb repack   : repacks all the databases, resulting in a perfectly
                 packed database with no freelist entries

(This used to be ctdb commit 3532119c84ab3247051ed6ba21ba3243ae2f6bf4)
2008-01-08 17:23:27 +11:00
Andrew Tridgell
25bb60f112 show start/stop time of recovery on all nodes
(This used to be ctdb commit 9f7662279c367eb3e8a58e6f4aeca521e6f1f1d0)
2008-01-08 09:30:11 +11:00
Andrew Tridgell
37861932ce merge from ronnie
(This used to be ctdb commit 0aa6e04438aa5ec727815689baa19544df042cf7)
2008-01-07 16:17:22 +11:00
Andrew Tridgell
d38fbaa38b nicer onnode output
(This used to be ctdb commit ac5c1e090d007bc2e3965589731620b87c0217fb)
2008-01-07 14:31:13 +11:00
Andrew Tridgell
4258098e98 catch internal traversal errors
(This used to be ctdb commit 8caa85ad71be5d20a8d6f0cb3d52aff6905657a4)
2008-01-07 14:08:25 +11:00
Andrew Tridgell
528e4d7a2b more efficient traversal in pulldb control
(This used to be ctdb commit fe614b10868e63b70e081b5bbfb74bf16fdf5716)
2008-01-07 14:07:01 +11:00
Andrew Tridgell
748843a3c6 added paranoid transaction ids
(This used to be ctdb commit afc1da53873cdbd31fcc8c6b22fae262e344cf6e)
2008-01-06 13:24:55 +11:00
Andrew Tridgell
c08f2616cd new simpler and much faster recovery code based on tdb transactions
(This used to be ctdb commit 9ef2268a1674b01f60c58fed72af8ac982fe77a3)
2008-01-06 12:38:01 +11:00
Andrew Tridgell
4f5b717aa3 change default tunables to cope with larger dbs
(This used to be ctdb commit d91a2d43d1f0562cc3a12e6e1e2767f75d888f72)
2008-01-06 12:36:58 +11:00
Andrew Tridgell
108aafcdb2 non-persistent databases don't need sync transactions
(This used to be ctdb commit 52fd86addd23e4d6e0af2c716bd83d19675b1f5a)
2008-01-06 12:36:30 +11:00
Andrew Tridgell
9311f7fb7e fixed the bug that make "onnode N service ctdb start" hang
(This used to be ctdb commit b50dcb16f30a60abce42f491f9b0aae7948b8206)
2008-01-05 12:09:29 +11:00
Andrew Tridgell
e4aefbc66d a new tunable DatabaseMaxDead that enables the tdb max dead cache logic
(This used to be ctdb commit 01c519c3658a8fcb9545b507b597e723658e4c4e)
2008-01-05 09:36:53 +11:00
Andrew Tridgell
023a230d9c a useful hack for checking correct behaviour of recovery
(This used to be ctdb commit d88b95a5407b53ead47ca0638ee60653ea3d3d07)
2008-01-05 09:36:21 +11:00
Andrew Tridgell
f79dfd04c0 convert much of the recovery logic to be async and parallel across all nodes
(This used to be ctdb commit 8b72a02bf1045d8befb342a4111ca1316889262e)
2008-01-05 09:35:43 +11:00
Andrew Tridgell
9a625534c1 this fixes the non-dmaster bug that has plagued us for months
(This used to be ctdb commit 2acf6c6201862debfca054a09262f75c066d2deb)
2008-01-05 09:34:47 +11:00
Andrew Tridgell
fc21f78231 make some specific cases of the non-dmaster bug non-fatal
(This used to be ctdb commit 7b516ab06c7ba7ffe9ecf3f76720df5360176b2c)
2008-01-05 09:32:29 +11:00
Andrew Tridgell
e9987cf236 fixed a warning
(This used to be ctdb commit f34d0f9351c1cda3327efb14e173f249f7854570)
2008-01-05 09:30:49 +11:00
Andrew Tridgell
afc7275c16 fixed a warning
(This used to be ctdb commit d6255438d63943736b24a7a6da190b6933379a61)
2008-01-04 12:42:10 +11:00
Andrew Tridgell
2509821503 prevent a re-ban loop for single node clusters
(This used to be ctdb commit b20a3369655bcba274c99091157ba7466994e848)
2008-01-04 12:11:29 +11:00
Andrew Tridgell
41fb8e283b add randrec to Makefile
(This used to be ctdb commit ded1f7903e8a6525ab1888e8c4f50c71fa23cc19)
2008-01-04 09:19:06 +11:00
Andrew Tridgell
bb06e831a0 more optimisations to recovery
(This used to be ctdb commit 9a41ad0a842cd4f3792d6e84b5c809b7ff6f342e)
2008-01-02 22:44:46 +11:00
Andrew Tridgell
2a2f1e3d91 fixed segv on failed ctdb_ctrl_getnodemap
(This used to be ctdb commit 5daf9a72f0e60a9af7cf32ae6d759be7d94857ec)
2007-12-27 10:07:01 +11:00
Andrew Tridgell
6ef3bff4ed merge from ronnie
(This used to be ctdb commit 072ef744951d3aa59dd8be70578b99b18c37d988)
2007-12-04 15:20:40 +11:00
Andrew Tridgell
a55c3709ea make DeterministicIPs the default
(This used to be ctdb commit e7d077e98a40a62dbd6bfd174f29afba7b5529ef)
2007-12-04 15:18:27 +11:00
Ronnie Sahlberg
7cef33b40a rework banning/unbanning nodes
ctdb_recoverd.c
Always handle banning/unbanning locally on the node that is being 
banned/unbanned instead of on the recovery master.
This means that if a ban request comes in to the recovery master for a 
remote node, we pass the request on to the remote node instead of 
setting up the ban and ban timeouts locally.

ctdb.c
send ban/unban requests to the node being banned/unbanned instead of to 
the recmaster

(This used to be ctdb commit 880dd9f5fd0b91e450da93e195cc5c62cb1dcd6e)
2007-12-03 15:45:53 +11:00
Ronnie Sahlberg
64008e28bb for the banned status, we should allocate this structure as a child of
the banned_nodes array and not the rec structure so that  ban_state is 
destroyed when the banned_nodes array gets destroyed
(and so that when this struct is destroyed, that any pending 
ctdb_ban_timeout events are also destroyed.)

othervise we may end up with multiple ban_timeout timed events going in 
parallell since we destroy/recreate the banned_nodes structure during 
election   but we never destroy/recreate the rec structure.

(This used to be ctdb commit fbd663d56a2a4421a5c0e541962c87e2e9c7cd82)
2007-12-03 11:39:17 +11:00
Andrew Tridgell
7edb41692e merge from ronnie
(This used to be ctdb commit 6653a0b67381310236e548e5fc0a9e27209b44e0)
2007-12-03 10:19:24 +11:00
Ronnie Sahlberg
2f1baf34d3 up the loglevel for the enable/disable monitoring to level 1
(This used to be ctdb commit 5043a0afeedbd30c7f64c2733c8ae5bf75479a98)
2007-12-01 10:06:42 +11:00
Ronnie Sahlberg
07dd0f6ff0 log that monitoring has been "disabled" not that it has been "stopped"
when monitoring is disabled

(This used to be ctdb commit e7c92f661a523deae9544b679d412ae79cc0ede7)
2007-11-30 10:53:35 +11:00
Ronnie Sahlberg
975fbc8e22 always set up a new monitoring event regardless of whether monitoring is
enabled or not

(This used to be ctdb commit c3035f46d1a65d2d97c8be7e679d59e471c092c2)
2007-11-30 10:14:43 +11:00
Ronnie Sahlberg
50573c5391 add ctdb_disable/enable_monitoring() that only modifies the monitoring
flag.
change calling of the recovered/takeip/releaseip event scripts to use 
these enable/disable functions instead of stopping/starting monitoring.

when we disable monitoring we want all events to still be running
in particular the events to monitor for dead nodes  and we only want to 
supress running the monitor event scripts

(This used to be ctdb commit a006dcc4f75aba950dd701ad7d1a84e89df285e8)
2007-11-30 10:09:54 +11:00
Ronnie Sahlberg
0eb6c04dc1 get rid of the control to set the monitoring mode.
monitoring should always be enabled
(though a node may want to temporarily disable running the "monitor"
event scripts but can do so internally without the need for this 
control)

(This used to be ctdb commit e3a33618026823e6af845fd8513cddb08e6b5584)
2007-11-30 10:00:04 +11:00
Ronnie Sahlberg
192ba82b73 ->monitor_context is NULL when monitoring is disabled.
Check whether monitoring is enabled or not before creating new events
and log why the event is not set up othervise

(This used to be ctdb commit 2f352b2606c04a65ce461fc2e99e6d6251ac4f20)
2007-11-30 09:02:37 +11:00
Ronnie Sahlberg
8ac8cce487 dont manipulate ctdb->monitoring_mode directly from the SET_MON_MODE
control, instead call ctdb_start/stop_monitoring()

ctdb_stop_monitoring() dont allocate a new monitoring context, leave it 
NULL. Also set the monitoring_mode in this function so that 
ctdb_stop/start_monitoring() and ->monitoring_mode are kept in sync.
Add a debug message to log that we have stopped monitoring.

ctdb_start_monitoring()  check whether monitoring is already active and 
make the function idempotent.
Create the monitoring context when monitoring is started.
Update ->monitoring_mode once the monitoring has been started.
Add a debug message to log that we have started monitoring.

When we temporarily stop monitoring while running an event script,
restart monitoring after the event script wrapper returns instead of in 
the event script callback.

Let monitoring_mode start out as DISABLED and let it be enabled once we call ctdb_start_monitoring.

dont check for MONITORING_DISABLED in check_fore_dead_nodes(). If 
monitoring is disabled, this event handler will not be called.

(This used to be ctdb commit 3a93ae8bdcffb1adbd6243844f3058fc742f76aa)
2007-11-30 08:44:34 +11:00
Ronnie Sahlberg
5c3a270991 move ctdb_set_culprit higher up in the file
when we are the recmaster and we update the local flags for all the 
nodes, if one of the nodes fail to respond and give us his flags,
set that node as a "culprit"

as one of the first things to do in the monitor_cluster loop, check if 
the current culprit has caused too many (20) failures and if so ban that 
node.


this is for the situation where a remote node may still be CONNECTED but 
it fails to respond to the getnodemap control  causing the recovery 
master to loop in monitor_cluster   aborting the monitoring when the 
node fails to respond   but before anything will trigger a call to 
do_recovery().
If one or more of the databases or nodes are frozen at this stage, this 
would lead to smbd being blocked for potentially a longish time.

(This used to be ctdb commit 83b0261f2cb453195b86f547d360400103a8b795)
2007-11-28 15:04:20 +11:00
Ronnie Sahlberg
9e73dc87cc Add a --node-ip argument so that one can specify which ip address a
specific instance of ctdbd should bind to. This helps when running a
"virtual" cluster on a single machine where all instcances bind to 
different alias interfaces.

If --node-ip is specified, then we will only try to bind to this ip 
address only. Othervise we fall back to the original method trying the
ip addresses in /etc/ctdb/nodes one by one until we find one we can bind 
to.

No variable in /etc/sysconfig/ctdb added since this parameter only makes 
sense in a virtual test/debug cluster.

(This used to be ctdb commit d96cb02c2c24f9eabbc53d3d38e90dea49cff3e0)
2007-11-26 10:52:55 +11:00
Ronnie Sahlberg
0597be3386 when monitoring the node from the recovery daemon, check that the
recovery daemon and the ctdb daemon both agree on whether the node is 
banned or not   and if they disagree then reban the node again after 
logging an error to the debug log

(This used to be ctdb commit 6cd6e534493066edd4bb2c6ae5be0e9a9d495aa0)
2007-11-23 12:41:29 +11:00
Ronnie Sahlberg
a260145f9f check for recursive bans in ctdb_ban_node() and remove the previous ban
if this is an attempt to ban an already banned node

(This used to be ctdb commit 214f2d7b04d0a491d466fc85c8d016efde416f9e)
2007-11-23 12:38:37 +11:00
Ronnie Sahlberg
6b284e5905 add log output for when ctdb_ban_node() and ctdb_unban_node() are called
when these functions are called to ban or unban a node make sure we 
update the CTDB_NODE_BANNED flag in rec->node_flags since this field and
flag are checked during the election process

(This used to be ctdb commit 740c632ae96a2d34327d1b575780aaf079d93f4f)
2007-11-23 12:36:14 +11:00
Ronnie Sahlberg
b5e79fb06f If update_local_flags() finds that a node has changed its BANNED status
so it differs from what the local ctdb daemon on the recovery master 
thinks it should be  we should call for a re-election

(This used to be ctdb commit 21ad6039c31ef5cc0e40a35a41220f91943947cb)
2007-11-23 11:53:06 +11:00
Ronnie Sahlberg
b2a81fb6b1 when we as the recovery daemon on the recovery master detects that the
flags differ between the local ctdb daemon and the remote node
we can force a flags update on all nodes and not just the local daemon

(This used to be ctdb commit a924eb89c966ecbae029ca137e06cffd40cc70fd)
2007-11-23 11:31:42 +11:00
Ronnie Sahlberg
af5bc9b915 add an extra log if we get a modflags control but it doesnt change any
flags


in update_local_flags()
(this is only called if we are or we belive we are the recmaster)
when we detect that the flags of a remote node is different from what 
our local node thinks the flags should be for that remote node
we should send a node-flag-changed message to the local daemon so 
that it updates the flags for that node.

(This used to be ctdb commit 36225e4e271f7a4065398253747fb20054f99a53)
2007-11-23 10:52:29 +11:00
Ronnie Sahlberg
c36ce05d08 if we get a modflag control but the flags remain unchanged, log this
(This used to be ctdb commit 5a0cd9b37b21665054bd35facd87f0a6ff4dcd55)
2007-11-23 10:31:51 +11:00
Ronnie Sahlberg
e95a4b5cdb when we print "Remote node had flags xx local had flags xx
we swapped the flags when printing them to the log

(This used to be ctdb commit 9fc8831a7fcd34763567227d61cd525ec441ebf2)
2007-11-23 09:54:38 +11:00
Andrew Tridgell
45f0fdfc20 make election handling much more scalable
(This used to be ctdb commit 05938d462b92bd9ecb8e35f53651bded47c48675)
2007-11-13 10:27:44 +11:00
Andrew Tridgell
3427793f01 don't do the first startup event until we are out of recovery
(This used to be ctdb commit 689940eb6e23f16ee063331caf3986613a8963ea)
2007-11-12 13:10:15 +11:00
Andrew Tridgell
bde886988b prevent a deadly embrace between smbd and ctdbd by moving the calling
of the startup event scripts after the point where recovery has
started and the node is in normal operation

This makes the 'startup' script just a special type of the 'monitor'
script which is called first

(This used to be ctdb commit 7424c30a5fd04aea0137c466b4318c3f185280d8)
2007-11-12 10:53:11 +11:00
Ronnie Sahlberg
1d6a74f943 when shutting down, we should stop monitoring
(This used to be ctdb commit 325683ef8f326f0565a827ff2c493adcab6e0d64)
2007-10-22 12:34:51 +10:00
Ronnie Sahlberg
4a97876fb7 when we are shutting down, we should first shut down the recovery daemon
(This used to be ctdb commit 39ade6b329adcd3234124d6a8daaa6181abf739b)
2007-10-22 12:34:08 +10:00
Andrew Tridgell
f47f758fe8 merge from ronnie
(This used to be ctdb commit d444fdc7782496abe4b27003b647ac49fb52e6be)
2007-10-19 09:39:07 +10:00
Ronnie Sahlberg
d1ba047b7f add a new transport method so that when a node is marked as dead, we
shut down and restart the transport

othervise, if we use the tcp transport the tcp connection might try to 
retransmit the queued data during the time the node is unavailable.
this together with the exponential backoff for tcp means that the tcp 
connection quickly reaches the maximum backoff rto which is often 60 or 
120 seconds.   this would mean that it could take up to 60/120 seconds 
before the tcp layer detects that the connection is dead and it has to 
be reestablished.

(This used to be ctdb commit 0256db470879ce556b0f00070f7ebeaf37e529ab)
2007-10-19 08:58:30 +10:00
Ronnie Sahlberg
755511d28d set the flags explicitely isnstead of masking them in
(This used to be ctdb commit 27a5f9dead44890683f9dbc4f07cda11264aa03b)
2007-10-18 16:54:00 +10:00
Andrew Tridgell
b814462c38 added some debug lines to help track down a problem
(This used to be ctdb commit 2ca31e9de179f76e392a26cc8305e2473357c760)
2007-10-18 16:27:36 +10:00
Andrew Tridgell
d939a2901b merge from ronnie
(This used to be ctdb commit 75d4b386293e186a6bb8532515585ab72670d663)
2007-10-18 15:44:02 +10:00
Ronnie Sahlberg
ce7a054d20 add back the test inside the daemon that if someone asks us to drop
recovery mode back to NORMAL that we can not lock the reclock file   
since at this stage it MUST be locked by the recovery daemon.

in order to avoid a non-blocking fnctl() lock from blocking and cause 
"issues"  we move the 'test that we can not lock reclock file' into a 
child process.

(This used to be ctdb commit 3af994641ec2234e37da1fa1f693441586471a7e)
2007-10-16 15:27:07 +10:00
Ronnie Sahlberg
056aac6e0c add a new tunable : DeterministicIPs that makes the allocation of
public addresses to nodes deterministic.

Activate it by adding CTDB_SET_DeterministicIPs=1 in /etc/sysconfig/ctdb

When this is set,    the first entry in /etc/ctdb/public_addresses will 
always be hosted by node 0, when that node is available, the second 
entry by node1 and so on.

This tunable allows the allocation of addresses to become very 
unbalanced and is only for debugging/testing use.
Beware, this feature requires that /etc/ctdb/public_addresses are 
identical on all the nodes in the cluster.

(This used to be ctdb commit f0ca221f235731542090d8a6c86f2b7cd2ce2f96)
2007-10-16 12:15:02 +10:00
Ronnie Sahlberg
25d3a031d0 include system/network.h so we get the prototype for inet_aton()
(This used to be ctdb commit 7145764b2d217f88a723dcb0ffd4e5a1567d64cf)
2007-10-16 11:29:33 +10:00
Ronnie Sahlberg
7e2e1b14fb merge from tridge
(This used to be ctdb commit 9e6bc12c9be2dabcfb9c6aeef257ef4737287fab)
2007-10-16 11:26:22 +10:00
Ronnie Sahlberg
b3ff7d904d dont try to lock the file from inside the ctdb daemon.
eventhough we dont want a blocking lock it does appear that the fcntl()
call can block for a while if gpfs is in the process of rebuilding 
itself after a node arriving/leaving the cluster

(This used to be ctdb commit 6c0d206dea7116db71bccb4802a93dd7283249f6)
2007-10-16 09:50:31 +10:00
Andrew Tridgell
99bc0aca93 sync flags between nodes in monitor loop in recmaster
(This used to be ctdb commit 6eef86e06388fc53a1212f1e2783ae174c6cd210)
2007-10-15 14:28:51 +10:00
Andrew Tridgell
0e855c0772 merge from ronnie
(This used to be ctdb commit d18712caba11855010be52f90bac656683076676)
2007-10-15 14:17:49 +10:00
Andrew Tridgell
174879621e add config option for disabling bans
(This used to be ctdb commit 153b911f7f957d4c564b04f5aa878033a02da9e4)
2007-10-15 13:22:58 +10:00
Ronnie Sahlberg
1a4999076b first check that recovery master is connected (we know this from our own
flags)

then pull the flags off recovery master before checking if it is banned

(This used to be ctdb commit 94c1d234e57a40eda2d8b892dd9fbe1ffc4b3433)
2007-10-11 07:10:17 +10:00
Ronnie Sahlberg
167e100d4b simplify election handling
make sure we read and update the flags from all remote nodes before we 
reach the first codepath that can call do_recovery()
since during do_recovery() we need to know what the flags are.

(This used to be ctdb commit e85f3806483ea420559d449e0e4d81bec996740f)
2007-10-11 06:16:36 +10:00
Ronnie Sahlberg
33a6aa3c3f merge from tridge
(This used to be ctdb commit 4690a205fe4325b03ab044bdb5fbc9aa3e94db6e)
2007-10-10 10:49:55 +10:00
Andrew Tridgell
011a205b86 make sure reconnected nodes start off as unhealthy so they don't get a public IP
(This used to be ctdb commit c733ec6760cae01ce277f491caf1355e46de5cf7)
2007-10-10 10:45:22 +10:00
Ronnie Sahlberg
bdd67bba1e add a --single-public-ip argument to ctdbd to specify the ip address
used in single public ip address mode.
when using this argument, --public-interface must also be used.

add a vnn structure to the ctdb context to describe the single public ip 
address


update the killtcp control in the daemon that if a socketpair that is to 
be killed does not match a normal public address it checks if the 
destination address maches the single public ip address and if so uses 
that vnn structure from the ctdb context


this allows killtcp to kill also connections to the single public ip 
instead of only normal public addresses

(This used to be ctdb commit 5661ba17b91f62821dec1c76056c78b99752a90b)
2007-10-10 09:42:32 +10:00
Ronnie Sahlberg
7735957693 remove some debug outputs
(This used to be ctdb commit f29c0b52df1f455909ba133e3ad3bc462dc32929)
2007-10-09 13:45:42 +10:00
Ronnie Sahlberg
80cd82f8e4 add a control to send gratious arps from the ctdb daemon
(This used to be ctdb commit 563819dd1acb344f95aabb4bad990b36f7ea4520)
2007-10-09 11:56:09 +10:00
Ronnie Sahlberg
de6c5ed14d merge from tridge
(This used to be ctdb commit 02cda01c032804cb1c53593ceb98685c827e2d58)
2007-10-06 08:11:24 +10:00
Andrew Tridgell
50770008df fixed several places where we set the recovery culprit incorrectly
(This used to be ctdb commit d9da73395fa443801fc68ec53a42b548e832d58a)
2007-10-05 13:51:31 +10:00
Andrew Tridgell
4115492992 - catch ESTALE in the recovery lock by trying a read()
- priortise nodes that are unbanned and healthy in the election

(This used to be ctdb commit 929feb475dfdf7283f0e99b50b179e1c91d3a39f)
2007-10-05 13:28:21 +10:00
Andrew Tridgell
fb48f2d5a2 we are the culprit if we can't get the reclock
(This used to be ctdb commit 1d320e113c6134ff6822b985a47131d8204af35a)
2007-10-05 12:01:40 +10:00
Ronnie Sahlberg
72379ee3eb change async.private to async.private_data since private is a reserved
work in c++

(This used to be ctdb commit 79eb28f6cd5dcc30b04966d202a050eaf98a2552)
2007-09-26 14:25:32 +10:00
Ronnie Sahlberg
359448ff00 when we have a public ip address mismatch (i.e. we hold addresses we
shouldnt   or we are not holding addresses wqe should)
we must first freeze the local node before we set the recovery mode

(This used to be ctdb commit a77a77e8b5180f6a4a1f3d7d4ff03811f3b71b56)
2007-09-24 10:52:26 +10:00
Andrew Tridgell
e3d0ec8797 fixed a fd leak on the recovery lock
(This used to be ctdb commit 186f35c42ed4fcc9ed44390b0dd036ece475d45e)
2007-09-24 10:19:07 +10:00
Andrew Tridgell
80100c3573 run monitoring more quickly when unhealthy and at startup
(This used to be ctdb commit ff1c205928e3ef5bcc6bf4e4b2122a19fa38d8f4)
2007-09-24 10:12:18 +10:00
Andrew Tridgell
b87ddd9148 no longer wait at startup for services to become available, instead
set the node initially unhealthy and let the status monitoring bring the node online.
This fixes a problem with winbindd, where it refused to start because secrets.tdb was not populated
but we could not populate ctdbd, because the net command would not run while ctdbd was still doing startup
and thus frozen
(This used to be ctdb commit 3a001b793dd76fb96addf1e2ccb74da326fbcfbc)
2007-09-24 10:00:14 +10:00
Andrew Tridgell
4178cb98a1 fixed a valgrind error, and some warnings
(This used to be ctdb commit c0f52dbb385fa0748680adb7c40755c92e577551)
2007-09-24 09:57:14 +10:00
Andrew Tridgell
2607c222fc avoid using connected nodes that aren't in the vnn map yet
(This used to be ctdb commit 2b5ae133f5f6fa9ad1a8896fe4b4c542d4ca462d)
2007-09-21 15:44:13 +10:00
Ronnie Sahlberg
51d912063c in ctdb_control_persistent_store() we must talloc_steal() the pointer to
c   to prevent it from being immediately freed (and our persistent store 
state with it) if we need to wait asynchronously for other nodes before 
we can reply back to the client

(This used to be ctdb commit fa5915280933e4d2e7d4d07199829c9c2b87a335)
2007-09-21 15:19:33 +10:00
Ronnie Sahlberg
61e885d0b9 when ctdb attaches to a database it broadcasts the attach to all other
nodes so that the db is created on them as well

when we send this broadcast   we must use the correct control and not 
assume all databases created are of the temporary kind 

(This used to be ctdb commit 106f816d4a0814ca4418de051289d9fc62df7dd2)
2007-09-21 13:47:40 +10:00
Andrew Tridgell
c60988325d added support for persistent databases in ctdbd
(This used to be ctdb commit 3115090a0d882beca9d70761130b74bb0821f201)
2007-09-21 12:24:02 +10:00
Andrew Tridgell
81bfa58d58 make sure we set close on exec on any possibly inherited fds
(This used to be ctdb commit d9dec82076f14a348e7b67b4350180681ff86f32)
2007-09-19 11:46:37 +10:00
Andrew Tridgell
c62490569b cope with non-standard install dirs in event scripts
(This used to be ctdb commit 52fff5345873690a9cc86495f414343eaa3bd540)
2007-09-14 14:14:03 +10:00
Andrew Tridgell
955d4d8615 make sure all public IPs are removed at startup
(This used to be ctdb commit b16f33787f2a9471285037f4a6d470e826536570)
2007-09-14 11:56:40 +10:00
Ronnie Sahlberg
6052078b53 let each node verify that they have a correct assignment of public ip
addresses (i.e. htey hold those they should hold   and they dont hold 
any of those they shouldnt hold)

if an inconsistency is found, mark the local node as recovery mode 
active
and wait for the recovery master to trigger a full blown recovery

(This used to be ctdb commit 55a5bfc8244c5b9cdda3f11992f384f00566b5dc)
2007-09-14 10:16:36 +10:00
Andrew Tridgell
42fc00bda9 - merge from ronnie
- add a flag to check that recovery completed correctly. If not, re-trigger it in monitoring

(This used to be ctdb commit d5ed941d9bab4af30d8b5f9b77bdf43d9218d69b)
2007-09-14 09:49:12 +10:00
Ronnie Sahlberg
4186d8eaba when a ctdb_takeover_run has failed we must make sure that
need_takeover_run is set to true  or else we might forget to rerun it 
again during the next recovery


othervise,  need_takeover_run is only set to true IFF the node flags for 
a remote node and the local nodes differ.
It is possible that a takeover run fails  and thus the reassignment of 
ip addresses is incomplete  but before we get back to the test in    
monitor_cluster()  that all the node flags of all nodes have converged 
and they now match each others again.   and thus causing 
monitor_cluster() to fail to realize that a takeover run is needed.

(This used to be ctdb commit ae7e866787cebd14394983ce1834387c959d1022)
2007-09-13 14:51:37 +10:00
Andrew Tridgell
9d50595b8a prevent recursion in the calling of ctdb_takeover_run
(This used to be ctdb commit 0fbdeb7c91b965d9bc5ecc7b24e31070378d8f1d)
2007-09-13 14:08:18 +10:00
Andrew Tridgell
30de14fe79 force recovery if unable to tell a node to release an IP
(This used to be ctdb commit 6895788d2499344a03357e5c1103cb8383e9eaf7)
2007-09-13 11:19:49 +10:00
Andrew Tridgell
3c0f61cb92 we don't need the is_loopback logic in ctdb any more
(This used to be ctdb commit 4ecf29ade0099c7180932288191de9840c8d90a9)
2007-09-13 10:45:06 +10:00
Andrew Tridgell
67bd64ef35 - don't allow the registration of clients with IPs we don't hold
- change some debug levels to make tracking of IP release problems easier
(This used to be ctdb commit 5f9aed62adaf87750f953412c55b29c58e4bb6c0)
2007-09-12 13:22:31 +10:00
Andrew Tridgell
a478c78f03 changed some debug levels
(This used to be ctdb commit ed764533e1c2f8982e1577ca5e7f5f4482a15345)
2007-09-12 13:21:19 +10:00
Andrew Tridgell
5b65a6c7f0 get interface right
(This used to be ctdb commit e0edc38d7e897f7de2850eb2cfd17fea75c16fcc)
2007-09-10 20:45:27 +10:00
Andrew Tridgell
8cd7ca149e fixed a pointer cast warning
(This used to be ctdb commit df0e7a4aa13112d613702d8ea0fb0e18510d293c)
2007-09-10 15:16:17 +10:00
Andrew Tridgell
f3ae1cdb02 - use struct sockaddr_in more consistently instead of string addresses
- allow for public_address lines with a defaulting interface

(This used to be ctdb commit 29cb760f76e639a0f2ce1d553645a9dc26ee09e5)
2007-09-10 14:27:29 +10:00
Andrew Tridgell
70ec39b1b1 add back in --public-interface as a default
(This used to be ctdb commit cdf56daf69b2c8381ee673943e982ad20f19affd)
2007-09-10 14:26:35 +10:00
Andrew Tridgell
42168177ef merge from ronnie
(This used to be ctdb commit 1f21d4d563232926c35d03c4d69eb69190823dc6)
2007-09-10 13:21:11 +10:00
Ronnie Sahlberg
4ac749bfa4 change the signature to ctdb_sys_have_ip() to also return:
a bool that specifies whether the ip was held by a loopback adaptor or 
not
 the name of the interface where the ip was held

when we release an ip address from an interface, move the ip address 
over to the loopback interface

when we release an ip address  after we have move it onto loopback, 
use 60.nfs to kill off the server side (the local part) of the tcp 
connection   so that the tcp connections dont survive a 
failover/failback

61.nfstickle,   since we kill hte tcp connections when we release an ip 
address   we no longer need to restart the nfs service in 61.nfstickle

update ctdb_takeover to use the new signature for ctdb_sys_have_ip

when we add a tcp connection to kill in ctdb_killtcp_add_connection()
check if either the srouce or destination address match a known public 
address

(This used to be ctdb commit f9fd2a4719c50f6b8e01d0a1b3a74b76b52ecaf3)
2007-09-10 07:20:44 +10:00
Ronnie Sahlberg
e4eeceaf3a dont dereference vnn before we have assigned it a pointer value
(This used to be ctdb commit 2a8fc69aea8527b22a3fe57427677e4caff57338)
2007-09-05 14:29:44 +10:00
Ronnie Sahlberg
77ec4d5248 allow different nodes in the cluster to use different public_addresses
files
so that we can partition the cluster into different subsets of nodes 
which each serve a different subset of the public addresses

(This used to be ctdb commit 889e0fe69e4c88c6166282b12843b8d9727552d6)
2007-09-04 23:15:23 +10:00
Ronnie Sahlberg
8f819c6a0e get rid of the ctdb_vnn_list structure and just use a single list of
ctdb_vnn

(This used to be ctdb commit 7b9fd06321af17043136b1420b57284450ae7ba5)
2007-09-04 18:20:29 +10:00
Ronnie Sahlberg
cf45c5096c we cant have takeover_ctx hanging off ctdb since it is freed/recreated
everytime we release an ip.
this context is used to hold all resources needed when sending out 
gratious arps and tcp tickles during ip takeover.

we hang it off the vnn structure that manages that particular ip address 
instead   so that we can have multiple ones going in parallell

this bug (or the same bug in different shape) has probably been in ctdb 
for very very long   but is likely to be hard to trigger

(This used to be ctdb commit c58db1cadaba253b2659573673b28c235ef7db76)
2007-09-04 14:36:52 +10:00
Ronnie Sahlberg
3e6be59f61 fix typo in debug output
(This used to be ctdb commit 011a777c6e538ca79f104c7884a4f0e222997382)
2007-09-04 14:21:35 +10:00
Ronnie Sahlberg
784eac9079 dont just always return 0 from the killtcp control.
return 0 or -1 so that the ctdb tool knows whether the control succeeded 
or not

(This used to be ctdb commit cace8b40090be5529ec6b463d3839d0e22f4039d)
2007-09-04 14:19:18 +10:00
Ronnie Sahlberg
a50e83448c change vnn to pnn in the traverse structure
(This used to be ctdb commit d56ae0963b420edea6a2d5eeb408a9811af3f3f6)
2007-09-04 10:49:21 +10:00
Ronnie Sahlberg
f69321edc8 change debug output from vnn to pnn
(This used to be ctdb commit 93a7cf759ae3f9af6671b9f8589e1399a669b46f)
2007-09-04 10:47:02 +10:00
Ronnie Sahlberg
d66d9cdd22 change debug output from vnn to pnn
change ctdb_daemon_send_message to take pnn as parameter isntead of vnn

(This used to be ctdb commit e352a2bbf9bb9a0b2c4f8329e8a529cf02414097)
2007-09-04 10:45:41 +10:00
Ronnie Sahlberg
157be530dd change ctdb_ctrl_getvnn to ctdb_ctrl_getpnn
(This used to be ctdb commit ef47cc4cd416065c69382e4d9e76c30a0a34e42f)
2007-09-04 10:38:48 +10:00
Ronnie Sahlberg
211b497818 change ctdb_node_flags_change.vnn to ctdb_node_flags_changed.pnn
change ctdb_ban_info.vnn to ctdb_ban_info.pnn

(This used to be ctdb commit fcedd40e0493948829e1c921d4fe30e9196e398a)
2007-09-04 10:33:10 +10:00
Ronnie Sahlberg
6f693bbcbd change server_id.vnn to server_id.pnn
(This used to be ctdb commit 26f2ee2b754a9271454412f05111a19b3013c6eb)
2007-09-04 10:21:51 +10:00
Ronnie Sahlberg
583b6e6ba6 change ctdb_get_vnn to ctdb_get_pnn
(This used to be ctdb commit 1e19930198c2bcc7ccb755e0ee51555fb823029a)
2007-09-04 10:18:44 +10:00
Ronnie Sahlberg
fc9d39c3a6 change ctdb_validate_vnn to ctdb_validate_pnn
(This used to be ctdb commit a4a1f41b69475b9dc16d8fd7f8965c32e96c32f0)
2007-09-04 10:09:58 +10:00
Ronnie Sahlberg
eb4cf6a686 change ctdb->vnn to ctdb->pnn
(This used to be ctdb commit 8c776e5707e503ec6586aae39ac6b3ea5a2fd2bc)
2007-09-04 10:06:36 +10:00
Ronnie Sahlberg
12ebb74838 change how we do public addresses and takeover so that we can have
multiple public addresses spread across multiple interfaces on each 
node.

this is a massive patch since we have previously made the assumtion that 
we only have one public address per node.

get rid of the public_interface argument.  the public addresses file 
now explicitely lists which interface the address belongs to

(This used to be ctdb commit 462ebbc791e906a6b874c862defea43235597ca8)
2007-09-04 09:50:07 +10:00
Andrew Tridgell
8c94d4dc87 merge from ronnie
(This used to be ctdb commit ab11fd70cf4d2165a5b55930cbad6fddf5397f54)
2007-08-27 18:04:53 +10:00
Ronnie Sahlberg
794fb10634 add an extra debug statement when we send a SIGTERM to a process
(This used to be ctdb commit a9c1be9cf9efdc69bfc95657b70e9f8b8230cda8)
2007-08-27 17:33:46 +10:00
Andrew Tridgell
7f630b67f6 fixed segv when no public interface is set
(This used to be ctdb commit 55b415f87bd3cba13c73ccd2fe661720754a6af7)
2007-08-27 11:49:42 +10:00
Ronnie Sahlberg
7f02e16143 add async versions of the freeze node control and freeze all nodes in
parallell 

(This used to be ctdb commit f34e89f54d9f4380e76eb1b5b2385a4d8500b505)
2007-08-27 10:31:22 +10:00
Ronnie Sahlberg
a9c45b2562 change the monitoring of recmode in the recovery daemon to use a fully
async eventdriven api for controls

(This used to be ctdb commit 8d0e43428c507967a0d96e6a4c5c821ac269c546)
2007-08-27 09:40:10 +10:00
Ronnie Sahlberg
801bdbdc80 add a control to pull the server id list off a node
(This used to be ctdb commit 38aa759aa88a042c31b401551f6a713fb7bbe84e)
2007-08-26 10:57:02 +10:00
Ronnie Sahlberg
6681da31df add an initial implementation of a service_id structure and three
controls to  register/unregister/check a server id.

a server id consists of TYPE:VNN:ID    where type is specific to the 
application.  VNN is the node where the serverid was registered and ID 
might be a node unique identifier such as a pid or similar.


Clients can register a server id for themself at the local ctdb daemon.
When a client dissappears   or when the domain socket connection for the 
client drops  then any and all server ids registered across that domain 
socket will also be automatically removed from the store.

clients can register as many server_ids as they want at the same time    
but each TYPE:VNN:ID must be globally unique.

Clients have the option of explicitely unregister a server id by using 
the UNREGISTER control.


Registration and unregistration can only be done by clients to the local 
daemon. clients can not register their server id to a remote node.


clients can check if a server id does exist on any ctdb node in the 
network by using the check control

(This used to be ctdb commit d44798feec26147c5cc05922cb2186f0ef0307be)
2007-08-24 15:53:41 +10:00
Ronnie Sahlberg
495a6403da change the api for managing callbacks to controls so that isntead of
passing it as a parameter we set the callback function explicitely from 
the caller if the ..._send() function returned a valid state pointer.

(This used to be ctdb commit aa939570662786455f63299b62c99882cff29d42)
2007-08-24 10:42:06 +10:00
Ronnie Sahlberg
62a03ef9d5 get rid of the explicit global timeout used in the previous example and
try this time by relying on the timeouts for the individual controls

(This used to be ctdb commit 448a0eb4fd896dc545aa0b4bb2ba4628491578be)
2007-08-23 19:38:54 +10:00
Ronnie Sahlberg
f854b5f876 try out a slightly different api for controls where you provide a
callback function which is called upon completion (or timeout) of the 
control.

modify scanning of recmaster in the monitoring_cluster code to try the 
api out

(This used to be ctdb commit c37843f1d97b169afec910e7ddb4e5ac12c3015c)
2007-08-23 19:27:09 +10:00
Ronnie Sahlberg
4c13bf0c5f break checking that the recoverymode on all nodes are ok out into its
own function

(This used to be ctdb commit 813cf9a252af96da24122b80f24aabeed2911939)
2007-08-23 13:48:39 +10:00
Ronnie Sahlberg
8fd3df2553 hang the ctdb_req_control structure off the ctdb_client_control_state
struct  so that if we timeout a control we can print debug info such as 
what opcode failed and to which node

we dont need the *status parameter to ctdb_client_control_state

create async versions of the getrecmaster control

pass a memory context to getrecmaster

(This used to be ctdb commit 558b680c82f830fba82c283c78c2de8a0b150b75)
2007-08-23 13:00:10 +10:00
Andrew Tridgell
d95476fa38 merge from ronnie
(This used to be ctdb commit e0f1c1acb1188500674626d631e1a1b8726e72ad)
2007-08-22 17:31:29 +10:00
Ronnie Sahlberg
50c09b7465 when we receive a packet from the network, check explicitely that the
node is not banned it the call is for a database record. i.e a REQ/REPLY 
CALL/DMASTER

if we get such a call while banned, ignore the packet and write an entry 
in the logfile

(This used to be ctdb commit 79eb0863609fbb12e28ebf734101b1d3f359b330)
2007-08-22 12:53:24 +10:00
Ronnie Sahlberg
f6e0336b23 create a define to represent the 'invalid' generation id we used in two
places.

create a new helper function to generate new generation id values that 
know about the invalid id and avoids generating it.

update the ctdb status tool to know about the invalid generation id and 
print the string INVALID instead

(This used to be ctdb commit 4fbcd189543cb8a92227fdcd3d158472e558ccda)
2007-08-22 12:38:31 +10:00
Ronnie Sahlberg
e3b6d1e511 if the node is inactive i.e. banned or disconnected then that node is
not participating in the cluster

if a client tries to attach to a database while the node is inactive,  
return an error back to the client and fail the attach

(This used to be ctdb commit b26949f3c8e54f3bc60da04d7b4ac69f301068fc)
2007-08-22 11:34:48 +10:00
Ronnie Sahlberg
b47384d57a when a node becomes banned its databases are no longer part of ctdb
and it should thus no longer serve any database access calls until it 
has been reintroduced into the cluster.

when becoming banned,   reset the local generation id to 1   to prevent 
any further database access calls from other nodes from being processed.

(This used to be ctdb commit b531021db43ebaa5f5d0ace28c59913d359bd8a8)
2007-08-22 10:38:35 +10:00
Ronnie Sahlberg
5fef81a6f1 if lockwait takes an excessive time to complete. log the time it took to
complete and also the name of the database

(This used to be ctdb commit 221ef0348fd8113a017d229d8c2c7aa5c4dfb5c2)
2007-08-22 09:46:48 +10:00
Ronnie Sahlberg
8b06fc7284 change the structure used for node flag change messages so that we can
see both the old flags as well as the new flags (so we can tell which 
flags changed)

send the CTDB_SRVID_RECONFIGURE messages to connected nodes only, not to 
every node, connected or not, in the cluster.


in the handler inside the recovery daemon which is invoked for node flag 
change messages, only do a takeover_run() and redistribute the ip addresses IF it was the 
disabled or the unhealthy flags that changed. Also send out the cluster 
reconfigured message in this case.
If any of the other flags changed we dont need to do the takeover_run(0 
here since that will be done during recovery.

(This used to be ctdb commit 5549b2058e2c148a8ca9d419123acf3247bb8829)
2007-08-21 17:25:15 +10:00
Ronnie Sahlberg
4e4dd6b886 when we shutdown the service due to receiving a 'ctdb shutdown' command
from the administrator, log this as 'Received SHUTDOWN command. Stopping 
CTDB daemon.'   so that the administrator will know when looking at the 
log 'why' the ctdb service was terminated.

Previously the only thing logged was 'shutting down' which is not 
detailed enough.

(This used to be ctdb commit 5b818c1b72b6594a8d6e45e1865026e3ce33ae63)
2007-08-21 09:46:27 +10:00
Ronnie Sahlberg
5228abef64 add an atexit() that will print "CTDB daemon shutting down" in the log
when the main daemon exits

(This used to be ctdb commit f7422397be2e319bfbee5bf0670583c353eda86d)
2007-08-21 09:43:53 +10:00
Ronnie Sahlberg
a03c8d4954 setup the logfile much earlier in the startup procedure for ctdbd
change initial errors that cause ctdb to fail to start from printf to 
DEBUG(0

add a DEBUG(0 to log that the ctdb service is starting

(This used to be ctdb commit 680b4fbb283dd68567a62a83345f11a6cc1dd0e5)
2007-08-21 09:33:03 +10:00
Ronnie Sahlberg
b582e13cae make sure that the event script is executable and just ignore it
othervise

(This used to be ctdb commit 65eb7845c70489d654acaaf99cd2c8eac7df11dc)
2007-08-21 09:22:14 +10:00
Ronnie Sahlberg
aed2c58c64 dont pollute the log with 'Registered PID XXX for client YYY' at log
level 0.

change the log level to 3 for this information message

(This used to be ctdb commit f28d713d9cacd2312932b51175aa8402c96ef76b)
2007-08-21 08:42:42 +10:00
Ronnie Sahlberg
7e1f840c8d if a public address has already been taken over by a node, then let that
public address remain at that node until either the node becomes 
unhealthy or the original/primary node for that address becomes healthy 
again.


Othervise what will happen is 
1, if we ban a node,   the banning code immediately does a 
takeover_run() and reassigns the public address to a different node in 
the cluster.
2, a few seconds later (at most) the recovery daemon will detect that 
the number of nodes has shrunk and will initiate a recovery.
During the recovery  the public address would again be assigned to a 
node, this time a different node.

(This used to be ctdb commit 30a6b7a648e22873d8ce6289a3d6dc42c4b9e3b3)
2007-08-20 14:16:58 +10:00
Andrew Tridgell
405e123ffb removed redundent debug message
(This used to be ctdb commit 9ee742b7cc43be7da6b568308912a3f2cfe4f4d3)
2007-08-20 11:13:38 +10:00
Andrew Tridgell
46639ac19e merged new event script calling code from ronnnie
(This used to be ctdb commit bbacad61b3eee4276ffe44ed2a23949aca8152cf)
2007-08-20 11:10:30 +10:00
Ronnie Sahlberg
7322e82bcb add text to the event script timeout log on how to find out which script
timed out

(This used to be ctdb commit bd6db995fb00ed45c5f0a50bbe6cf5d0fe22a194)
2007-08-15 15:08:42 +10:00
Ronnie Sahlberg
3b9d50f3ee change the now rather small /etc/ctdb/events script into a service
specific script /etc/ctdb/events.d/00.ctdb

get rid of CTDB_EVENTS_SCRIPT and --event-script

(This used to be ctdb commit 81ccfaf838e5772d4a58eb6a70224b7b39aba9f3)
2007-08-15 15:01:31 +10:00
Ronnie Sahlberg
ff58f7c7ea add a comment that the talloc_free also removes the script from the tree
(This used to be ctdb commit ce71f6e9cf983cc4fe66935ad6c18d55dfed03a5)
2007-08-15 14:46:06 +10:00
Ronnie Sahlberg
4023576e50 call the service specific event scripts directly from the forked child
instead for from /etc/ctdb/events so that we can get better debugging 
output in the logs when something fails in the scripts

(This used to be ctdb commit 4ed96b768aea1611e8002f7095d3c4d12ccf77a3)
2007-08-15 14:44:03 +10:00
Ronnie Sahlberg
5a02262a06 comment that ctdb_event_script_v() is called from a forked childs
context and thus can make blocking calls

(This used to be ctdb commit b31d98281f15995ad340d2510e08e04ed46e271a)
2007-08-15 10:48:10 +10:00
Ronnie Sahlberg
56d5ef27b6 add a wrapper function to create the key used to insert/lookup a certain
tcp connection in the tree that stores the tcp connections to kill by 
sending an RST

add a define that specified the keylength instead of hardcoding it as 4

(This used to be ctdb commit 6a8322cbae10f2c78b2e286c75aeb25ece12ea7f)
2007-08-15 10:01:00 +10:00
Ronnie Sahlberg
adb49f02f0 change the mem hierarchy for trees. let the node be owned by the data
we store in the tree and use a node destructor so that when the data is 
talloc_free()d we also remove the node from the tree.

(This used to be ctdb commit b8dabd1811ebd85ee031563e95085f720a2fa04d)
2007-08-09 14:08:59 +10:00
Ronnie Sahlberg
9c216d0d76 when we want to kill a tcp connection we stored the connection
description (src + dst sockaddr_in) in a linked list.
everytime we receive a captured packet from the network we had to walk 
this list in linear time to see if the packet matched a connection we 
wanted to RST.
which wouldnt scale very well.


replace the linked list with a redblack tree that is indexed by
src address, src port,  dst address,   dst port
to make checking whether the packet belongs to a connection we want to 
RST very fast and scalable


the reason we need to capture packets when we want to kill a TCP 
connection is because we must wait for an ACK coming back from the 
remote host  so that we can learn which sequence number to use in the 
RST.
Most tcp today will ingore any and all RST segments unless the 
sequencenumber lies exactly on the right edge of the window to make 
spoofing RST a little bit more difficult.

(This used to be ctdb commit ced18caea8582af042287beb6333dd1f8ba3344d)
2007-08-08 15:09:19 +10:00
Ronnie Sahlberg
203306400e add helpers to traverse a tree where the key is an array of uint32
(This used to be ctdb commit d328c66827cafff6356e96df2a782930274fe139)
2007-08-08 13:50:18 +10:00
Ronnie Sahlberg
dd14afe6aa after we have checked dest address that it is a public address
update addr to the source address so the rpintout in the log matches
the client that attached to samba

(This used to be ctdb commit 72098b71c79469c86769ca82bbd484c81902d27c)
2007-07-30 16:10:14 +10:00
Ronnie Sahlberg
e666808f60 no need to have a separate assignment of the tcparray pointer followed
by a talloc_steal()
use the returned pointer in talloc_steal as the value to assign

(This used to be ctdb commit 5c6375ad3bbecfa725ec3b1477f259e5a8191866)
2007-07-25 08:03:58 +10:00
Ronnie Sahlberg
81294825e7 when we build the arp structure for sending gratious arp (and tcp
tickles) just talloc_steal the enture tcp_array into the arp 
structure instead of copying each of the entries into a linked list
and then releasing the tcparray.

(This used to be ctdb commit 468e237740cf37a65872ef700bbb1284ede8352a)
2007-07-24 07:46:51 +10:00
Ronnie Sahlberg
ea56d1d20e set the tcp tickle update flag to true once we have done a takeover and
tickled all connections
othervise the other nodes will still remember this list until next time 
we have had a connection/client closing.

(This used to be ctdb commit cb8e5d4bbee2f14f498735489f673ff3679dfd9d)
2007-07-20 19:11:45 +10:00
Ronnie Sahlberg
81767b2a7b when a client connects with TCP_CLIENT we should look at the
destination address to find the public address   not the source address

(This used to be ctdb commit d6d4a7f38a52c1c2579a54d14cb7a6981fb42f5b)
2007-07-20 17:04:08 +10:00
Ronnie Sahlberg
fca90ce3c3 updated ctdb tickle management
there is an array for each node/public address that contains tcp tickles

we send a TCP_ADD as a broadcast to all nodes when a client is added

if tcp tickles are removed, they are only removed immediately from the 
local node.
once every 20 seconds a node will push/broadcast out the tickle list for 
all public addresses it manages.   this will remove any deleted tickles 
from the remote nodes

(This used to be ctdb commit e3c432a915222e1392d91835bc7a73a96ab61ac9)
2007-07-20 15:05:55 +10:00
Ronnie Sahlberg
7b17afdfcd change the tickle list from one global list into an array per public
ip/node

once we have started sending all tickles for a specific ip   delete the 
entire array   so that the tickles dont remain forever in the ctdb 
server

add a control to send the full list of every tickle that is registered 
for a particular public ip/node

(This used to be ctdb commit d0eee33e44d3f8e26debbec21d41e2cbdbb520e6)
2007-07-20 10:06:41 +10:00
Andrew Tridgell
394190d3cc - log registering of tcp clients
- don\'t remove a tcp entry if we do not own the ip
(This used to be ctdb commit 400aa284b9785ce6409e7600df429f5849e3867d)
2007-07-19 15:04:54 +10:00
Andrew Tridgell
689195b455 make sure we still run events when waiting for ctdb_event_script()
(This used to be ctdb commit 05efbfe9ff9691c1d7441e7b9855aed25791faf0)
2007-07-19 13:36:00 +10:00
Andrew Tridgell
fb22d3bd2c merged from ronnie
(This used to be ctdb commit 765b07fa5d1af07c8c7212d19d8e9574060b3039)
2007-07-18 20:13:57 +10:00
Ronnie Sahlberg
4d1f3acc94 add a check if start_node is beyond the end of the nodemap and reset it
back to 0 if it is to prevent an infinite loop.

this could happen if in the future we add a mechanism to add/remove 
nodes to a cluster at runtime

(This used to be ctdb commit 217e80a468713fec86ccb0608460e3401046bb98)
2007-07-16 08:36:09 +10:00
Ronnie Sahlberg
49f98e79fd change the way we pick/find a new node to takeover for a failed node
to keep a static that controls at which noide to start searching the 
list for takeover candidates next time we need to find a node.

each time we find a node to takeover, reset the start variable to point 
to the next node in the list

this makes the distribution of takeover nodes much more even

(This used to be ctdb commit e9800df5a21079ea478d16f7dd2fd4707de85650)
2007-07-16 08:28:44 +10:00
Ronnie Sahlberg
f09566a81a add a private_data field to the killtcp structure and let the system
specific routines populate it as it see fit when creating a 
capture socket.
pass this structure to read_tcp and close capture socket as parameter

(This used to be ctdb commit 79bbfcfb2223889126fe307d5bbfd24917da07ee)
2007-07-13 17:07:10 +10:00
Andrew Tridgell
8f637e6317 ensure killtcp structure is initialised
(This used to be ctdb commit 2fe7d1ce87e55e125411e7406a9e00b8f55e3cb7)
2007-07-13 11:55:58 +10:00
Andrew Tridgell
1e14ecd176 - merge from ronnie
- cleaner handling of system capture socket

(This used to be ctdb commit d194a41a71b8466d0726dcbae3970a86386fcb3c)
2007-07-13 11:31:18 +10:00
Andrew Tridgell
d2a5af7eb8 fully save/restore scheduler parameters
(This used to be ctdb commit 59408eabe7515d49a6eef3b6fb2590a1cd1df956)
2007-07-13 09:35:46 +10:00
Andrew Tridgell
698a8bc909 fixed the sense of do_setsched
(This used to be ctdb commit 68bca2454ff43ce6d8aab2f87d669d33f5f2a10c)
2007-07-13 09:14:31 +10:00
Andrew Tridgell
fc73bc5c24 added --nosetsched option to ctdbd
(This used to be ctdb commit 4cbbb88c1735c7d112e751e22da1c1c69e09bf4a)
2007-07-13 08:47:02 +10:00
Ronnie Sahlberg
a650497680 as an optimization for when we want to send multiple tickles at a time
let the caller create the sending socket and use a single socket instead 
of one new one for each tickle.
pass a sending socket to ctdb_sys_send_tcp()

ctdb_sys_kill_tcp is not longer used so remove it

set the socketflags for close on exec and nonblocking in the helper that 
creates the sockets instead of in the caller

add a helper to create a sending socket to send tickles from

(This used to be ctdb commit 469f3fb238a0674a2b48fdf1a7e657e32428178a)
2007-07-12 09:22:06 +10:00
Ronnie Sahlberg
823b7d4a5f rename killtcp->fd to killtcp->capture_fd
we might want to have two sockets attached to the killtcp structure
one for capturing and a second one for sending  so we dont have to 
create a new socket for each tickle we want to send

(This used to be ctdb commit b3e82ec38047bbec1edfd88ade264077d4cbd2ee)
2007-07-12 08:52:24 +10:00
Ronnie Sahlberg
76ab80104a make the ctdb tool use the killtcp control in the daemon instead of
calling killtcp directly

(This used to be ctdb commit d21e3e9cf11bdcba6234302e033d6549c557dd69)
2007-07-12 08:30:04 +10:00
Ronnie Sahlberg
1ed0c3a9f7 add daemon code for the new kill_tcp control
(This used to be ctdb commit 8fe4ae62255ecb2db36bea736ff17409ba6614c5)
2007-07-11 18:24:25 +10:00
Ronnie Sahlberg
e4db03f7e6 add a ctdb_ prefix to two public functions
(This used to be ctdb commit 32adee5426aa75ddcd4d648ef326ed03d5ff5c46)
2007-07-11 18:13:03 +10:00
Ronnie Sahlberg
aa080f66d9 first cut at a better and more scalable socketkiller
that can kill multiple connections asynchronously using one listening 
socket

(This used to be ctdb commit 22bb44f3d745aa354becd75d30774992f6c40b3a)
2007-07-11 17:43:51 +10:00
Ronnie Sahlberg
0c44e0ad46 add a ctdb_kill_tcp_callback() that will perform a kill tcp using a
background process

(This used to be ctdb commit dcfcaacff56347d94c244512eb72219b05ef9c3d)
2007-07-11 12:33:14 +10:00
Ronnie Sahlberg
135a964220 pass the header to ctdb_become_dmaster instead of just the reqid
this allows us to print from which node Invalid or Dropped orphan become 
dmaster packets came from

(This used to be ctdb commit 88efd1bf4c796cd2b184156b72296587bc38bb40)
2007-07-11 09:44:52 +10:00
Ronnie Sahlberg
2eef287fab print the operation code in the debug message when we discard a packet
due to incorrect generation number

(This used to be ctdb commit 3151e3b2607291572fc6e7380fd60ef7ce438307)
2007-07-11 08:41:29 +10:00
Andrew Tridgell
32de198fd3 update lib/replace from samba4
(This used to be ctdb commit f0555484105668c01c21f56322992e752e831109)
2007-07-10 15:29:31 +10:00
Ronnie Sahlberg
a859723912 nicer handling of DISCONNECTED flag when we update the node flags from
a remote message

(This used to be ctdb commit 9a50ad22be61a09761ffda89de91ef3221917c84)
2007-07-09 17:40:15 +10:00
Ronnie Sahlberg
69f3a09e6f when a remote node has sent us a message to update the flags for a node,
dont let those messages modify the DISCONNECTED flag.

the DISCONNECTED flag must be managed locally since it describes whether 
the local node can communicate with the remote node or not

(This used to be ctdb commit 5650673205d335a32d4f27f66847ea66752a00f0)
2007-07-09 13:21:17 +10:00
Ronnie Sahlberg
b871c3e365 a better way to fix the DISCONNECT|BANNED vs DISCONNECT bug
(This used to be ctdb commit 5c638d7731c5a268de02d3a37828ac7aec9a12de)
2007-07-09 12:55:15 +10:00
Ronnie Sahlberg
3499c8c673 when checking the nodemap flags for consitency while monitoring the
cluster,   we cant check that both the BANNED and the DISCONNECTED flags 
are both set at the same time   since if a node becomes banned just 
before it is DISCONNECTED   there is no guarantee that all other nodes 
will have seen the BANNED flag.

So we must first check the DISCONNECTED flag only   and only if the 
DISCONNECTED flag is not set should we check the BANNED flag.


othervise this can cause a recovery loop while some nodes thing the 
disconnected node is DISCONNECTED|BANNED and other think it is just 
DISCONNECTED

(This used to be ctdb commit 0967b2fff376ead631d98e78b3a97253fc109c69)
2007-07-09 12:33:00 +10:00
Andrew Tridgell
f1db15ffe1 fixed sense of inet_aton test
(This used to be ctdb commit ed5cf9b43c49312d3736e85077863d23990acce8)
2007-07-08 21:09:09 +10:00
Andrew Tridgell
056d3c35a4 call kill_clients when releasing all IPs, as well as for individual IPs
(This used to be ctdb commit ad68904720eb69757601589b06726190321685ac)
2007-07-08 20:45:12 +10:00
Andrew Tridgell
af5ee9981e we do tell banned nodes to release IPs
(This used to be ctdb commit 381dc0421d4d825398c03dcff4e79e3f76c3c981)
2007-07-08 20:24:03 +10:00
Andrew Tridgell
a55c03b31b log the generation numbers to give a hint about this bug
(This used to be ctdb commit 12018494baa33c5f6c52e6eae94ac77a56d3e5a0)
2007-07-08 19:36:55 +10:00
Andrew Tridgell
006227e80a forgot to add this
(This used to be ctdb commit 30fc56b7489e42633532964096e53faee1319dde)
2007-07-04 17:45:46 +10:00
Andrew Tridgell
bdf01ed7c0 - neaten up the command line for killtcp
- split out the event script code into a separate module
- get rid of the separate takeover directory

(This used to be ctdb commit 8ea2c923a3e2464200ff79bf2c3f1f89e6a93ad4)
2007-07-04 16:51:13 +10:00
Ronnie Sahlberg
1cd8bc0c64 add a tuneable to control how long we wait after a successful recovery
before we alow another recovery to be initiated

(This used to be ctdb commit f3b43519423b7a73e6a2dd986bdf11203b8653cf)
2007-07-04 08:36:59 +10:00
Andrew Tridgell
6399cf9542 added code to kill registered clients on a IP release
(This used to be ctdb commit ca0243b544987ce0618a99ac87b4abf598991e93)
2007-06-19 03:54:06 +10:00
Andrew Tridgell
732353de5f - merged ctdb_store test from ronnie
- added DatabaseHashSize tunable
- added logging of events inside recovery (for timing)

(This used to be ctdb commit 3593cdb928b91e217faf1b3c537fa28dc82cdace)
2007-06-17 23:31:44 +10:00
Andrew Tridgell
97d5bea2eb on startup release all IPs, in case we have any left over from a previous run
(This used to be ctdb commit 5eb2f8f5f70f567c264d6929e95899b70f0e4ec0)
2007-06-12 19:44:54 +10:00
Andrew Tridgell
91362083a1 make sure we start the freeze process quickly on all nodes when we are going to do recovery - this prevents serialisation of freeze, which can take a long time
(This used to be ctdb commit 52675c19e420d83d9556a3e73d9a4b490078aa9c)
2007-06-11 23:03:23 +10:00
Andrew Tridgell
031e205832 raise the default keepalive limit
(This used to be ctdb commit 4776a187a183bd129ded70e9c018c197b3d618be)
2007-06-11 22:27:23 +10:00
Andrew Tridgell
a31ece536c more detail in recovery message
(This used to be ctdb commit bc18a39efcf1fa5edfadc4c2f842f7cf035e4fbd)
2007-06-11 21:37:09 +10:00
Andrew Tridgell
044a2e04c4 - send tcp info to all connected nodes, not just vnnmap nodes
- use a non-blocking freeze when banned
- release all IPs when banned

(This used to be ctdb commit 070e85e532b33b792f85c3e72eee205d906aaf85)
2007-06-10 08:46:33 +10:00
Andrew Tridgell
18ae6e56f0 propogate flag changes to all connected nodes
(This used to be ctdb commit 711d1f7e20f1e98caaf08a57df0b1825ff6e97a0)
2007-06-09 21:58:50 +10:00
Ronnie Sahlberg
40585aed37 should be sufficient to unban nodes when we unbecome recmaster
(This used to be ctdb commit 8a6c4e675b4b877a9d0a7a3701973573ff0b71e8)
2007-06-09 20:13:25 +10:00
Ronnie Sahlberg
5458196b3f unban all nodes when we release recmaster role or when we win an
election

(This used to be ctdb commit 48fb7483b3fe391e2d0b78718af29f69a641525e)
2007-06-09 20:11:51 +10:00
Ronnie Sahlberg
c873c7d4da remove rht unban code from when we take recmaster role. we can not
send control broadcasts yet

(This used to be ctdb commit 39a05dc1d74d49685e6daf929df169d936585208)
2007-06-09 19:49:28 +10:00
Ronnie Sahlberg
9a0d7a688f add code to unban when we become/unbecome recmaster
(This used to be ctdb commit a22cf9b8a6fd46128faca958f33a75cb3fc1ee12)
2007-06-09 19:42:41 +10:00
Andrew Tridgell
06a71762a4 some #include cleanups
(This used to be ctdb commit 1a07d87122d51a40cd8ad5fe13533298c26857cb)
2007-06-07 22:26:27 +10:00
Andrew Tridgell
b50096c835 more code rearrangement
(This used to be ctdb commit 2bcf3b16163041f03add2e5bf9f1f5fb3599ec24)
2007-06-07 22:16:48 +10:00
Andrew Tridgell
ae3d54094b start splitting the code into separate client and server pieces
(This used to be ctdb commit 603cd77988c181525946cd5eb0f4d0d646b58059)
2007-06-07 22:06:19 +10:00