1
0
mirror of https://github.com/samba-team/samba.git synced 2024-12-23 17:34:34 +03:00
Commit Graph

94 Commits

Author SHA1 Message Date
Ronnie Sahlberg
3f819741ad ctdb: hold transaction locks during freeze, mark during recover.
Make the ctdb parent "mark" the transaction lock once the child process
has frozen/locked the entire database.
This stops the ctdb daemon from using  a blocking fcntl() locking on the tdb during the
read traverse during recovery.

CQ 1021388

(This used to be ctdb commit 52ee2b3ce822344d0f55ac040fe25f6ec5c0d7c2)
2011-01-18 14:07:44 +11:00
Ronnie Sahlberg
5f76f3c0e2 Add a new tunable : DisableIPFailover that when set to non 0
will stopp any ip reallocations at all from happening.

(This used to be ctdb commit d8d37493478a26c5f1809a5f3df89ffd6e149281)
2010-11-10 14:55:24 +11:00
Ronnie Sahlberg
39c367a68f Create macros to update the statistics counters and use these macros
everywhere instead of manipulating the coutenrs directly.

(This used to be ctdb commit 2e648df890e5713bc575965d87937827b068d0d7)
2010-09-29 12:14:24 +10:00
Ronnie Sahlberg
7c682dda59 When memory allocations for recovery fails,
dont dereference a null pointer while trying to print the log message for the failure.

also shutdown ctdb with ctdb_fatal()

(This used to be ctdb commit f8642d0438c6bbb34a72c25d6a904b626e247410)
2010-09-03 12:00:48 +10:00
Ronnie Sahlberg
2e8aac6689 Merge commit 'rusty/ports-from-1.0.112' into foo
(This used to be ctdb commit 13e58d92f5f1723e850a82ae030d0ca57e89b1ee)
2010-08-19 13:17:56 +10:00
Rusty Russell
9fbb191b78 logging: give a unique logging name to each forked child.
This means we can distinguish which child is logging, esp. via syslog where we have no pid.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

(This used to be ctdb commit 68b3761a0874429b90731741f0531f76dcfbb081)
2010-08-18 11:46:32 +09:30
Rusty Russell
f93440c4b7 event: Update events to latest Samba version 0.9.8
In Samba this is now called "tevent", and while we use the backwards
compatibility wrappers they don't offer EVENT_FD_AUTOCLOSE: that is now
a separate tevent_fd_set_auto_close() function.

This is based on Samba version 7f29f817fa.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>


(This used to be ctdb commit 85e5e760cc91eb3157d3a88996ce474491646726)
2010-08-18 09:16:31 +09:30
Ronnie Sahlberg
6f1221e9e1 Add the number of performed recoveries to the "ctdb statistics" output.
(This used to be ctdb commit fa045733cb81412f0d02ab52d74eabc7efca8b3d)
2010-05-11 09:44:53 +10:00
Ronnie Sahlberg
3f226d0c8e Lower the loglevel for "Recovery lock successfully taken"
from ERR to NOTICE

BZ62086

(This used to be ctdb commit 7fa8486f9ffe2a039360b07423f734bdd884fe1d)
2010-04-07 10:45:03 +10:00
Ronnie Sahlberg
aea0a724d8 In ctdb_control_end_recovery,
We used to talloc_steal c (the command packet) and make it a child of the
"event script state context".
If we failed to create a eventscript child context for some reason,
this would have talloc freed state, but at the same time it would also
implicitely have freed c.
Once ctdb_control_end_recovery() returns the error back to the caller,
the caller would dereference both c, and also outdata which is a child of c
and we would either read garbage data or segv.

Change the ordering so we only talloc_steal c as a child of state IFF
we have successfully created a child context for the script.

BZ61068

(This used to be ctdb commit 259054c3632e42bbaa614ee7e888e6e850733d60)
2010-02-23 12:50:32 +11:00
Ronnie Sahlberg
d7c00d8d7e Drop the debug level for logging fd creation to DEBUG_DEBUG
(This used to be ctdb commit eae1d4f9e52e73b4d8769868fffdafa590d03784)
2010-02-04 06:37:41 +11:00
Stefan Metzmacher
94bc40307a server: Use tdb_check to verify persistent tdbs on startup
Depending on --max-persistent-check-errors we allow ctdb
to start with unhealthy persistent databases.

The default is 0 which means to reject a startup with
unhealthy dbs.

The health of the persistent databases is checked after each
recovery. Node monitoring and the "startup" is deferred
until all persistent databases are healthy.

Databases can become healthy automaticly by a completely
HEALTHY node joining the cluster. Or by an administrator
with "ctdb backupdb/restoredb" or "ctdb wipedb".

metze

(This used to be ctdb commit 15f133d5150ed1badb4fef7d644f10cd08a25cb5)
2009-12-16 08:06:10 +01:00
Rusty Russell
928b8dcb31 eventscript: handle banning within the callbacks
Currently the timeout handler in eventscript.c does the banning if a
timeout happens.  However, because monitor events are different, it has
to special case them.

As we call the callback anyway in this case, we should make that handle
-ETIME as it sees fit: for everyone but the monitor event, we simply ban
ourselves.  The more complicated monitor event banning logic is now in
ctdb_monitor.c where it belongs.

Note: I wrapped the other bans in "if (status == -ETIME)", though they
should probably ban themselves on any error.  This change should be a
noop.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>


(This used to be ctdb commit 9ecee127e19a9e7cae114a66f3514ee7a75276c5)
2009-12-07 23:48:57 +10:30
Martin Schwenke
a64ccf07c1 Add flag to ctdb_event_script_callback indicating when called by client.
Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit a1d654a982ca56fade82552f4e6b5586236d3233)
2009-11-26 15:49:49 +11:00
Rusty Russell
2d9254404d eventscript: introduce enum for different event script calls.
Rather than doing strcmp everywhere, pass an explicit enum around.  This
also subtly documents what options are available.  The "options" arg
is now used for extra arguments only.

Unfortunately, gcc complains on empty format strings, so we make
ctdb_event_script() take no varargs, and add ctdb_event_script_args().  We
leave ctdb_event_script_callback() taking varargs, which means callers
have to do "%s", "".

For the moment, we have CTDB_EVENT_UNKNOWN for handling forced scripts
from the ctdb tool.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>


(This used to be ctdb commit 8001488be4f2beb25e943fe01b2afc2e8779930d)
2009-11-24 11:16:49 +10:30
Rusty Russell
2763df22de eventscript: put timeout inside ctdb_event_script_callback_v
Everyone uses the same timeout value, so just remove it from the API.
If we ever need variable timeouts, that might as well be central too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>


(This used to be ctdb commit 533c3e053293941d2a9484b495e78d45f478bb08)
2009-11-24 11:09:46 +10:30
Ronnie Sahlberg
e07ca41886 change the eventscript handling to allow EventScriptTimeout for each individual script isntead of for the entire set of scripts
restructure the talloc hierarchy to allow this

(This used to be ctdb commit 64da4402c6ad485f1d0a604878a7b0c01a0ea5f0)
2009-10-28 16:11:54 +11:00
Ronnie Sahlberg
9de3652380 add logging everytime we create a filedescriptor in the main ctdb daemon
so we can spot if there are leaks.

plug two leaks for filedescriptors related to when sending ARP fail
and one leak when we can not parse the local address during tcp connection establish

(This used to be ctdb commit ddd089810a14efe4be6e1ff3eccaa604e4913c9e)
2009-10-15 11:24:54 +11:00
Ronnie Sahlberg
3ac5a52969 Port Volkers deadlock avoidance patch to HEAD.
This patch ensures that we lock all non-notify related databases first and
then the notify databases to avoiud a deadlock where samba needs to lock records on two databases at once (and notify being the second database).

Newer versions of samba would instead use the set-db-prio control to set this explicitely on a database per database basis instead of relying on  hardcoded database names. This patch will be reverted in the future when all updated versions of samba has been pushed out.

(This used to be ctdb commit 70e7781df1f118a0e2632a9c634f3fd388fa6c8c)
2009-10-14 08:17:49 +11:00
Ronnie Sahlberg
771802b212 allow setting the recmode even when not completely frozen.
we sometimes have to do this when we want to trigger a recovery

(This used to be ctdb commit 46194e87e189521375b39b4ef33da2b493429fd8)
2009-10-12 13:06:16 +11:00
Ronnie Sahlberg
73c0adb029 initial attempt at freezing databases in priority order
(This used to be ctdb commit e8d692590da1070c87a4144031e3306d190ebed2)
2009-10-12 12:08:39 +11:00
Ronnie Sahlberg
e5e9fc48b1 create a new event : stopped.
This event is called when a node is stopped and is used by eventscripts that need to do certain cleanup and removal of configuration or ip addresses or routing ...

Note that a STOPPED node is considered "inactive" and as such will not be running the "recovered" event when the rest of the cluster has recovered.

(This used to be ctdb commit 65e9309564611bf937ded3c74a79abff895d7c59)
2009-07-17 12:26:16 +10:00
Ronnie Sahlberg
b57811bee6 change the infolevel when logging stop/continue commands
(This used to be ctdb commit 1e007c833098b03dd81797c081da1ae1b10c971c)
2009-07-09 14:34:12 +10:00
Ronnie Sahlberg
88f3c40d9c add two new controls, CTOP_NODE and CONTINUE_NODE
that are used to stop/continue a node instead of using modflags messages

(This used to be ctdb commit 54b4a02053a0f98f8c424e7f658890254023d39a)
2009-07-09 12:22:46 +10:00
Ronnie Sahlberg
180a576f7b Dont access the reclock file at all if VerifyRecoveryLock is zero and also
make sure the reclock file is closed if the variable is cleared at runtime

(This used to be ctdb commit a25f4888689a0725971606163d87c39a41669292)
2009-06-25 11:41:18 +10:00
Ronnie Sahlberg
baead0fdcc dont leak file descriptors when set recmdoe timesout
(This used to be ctdb commit fc8a364eb095ec11ca01246a583bf1dc53510141)
2009-06-19 14:58:06 +10:00
Ronnie Sahlberg
1183b364f1 reduce the timeout we wait for the reclock child process to finish to 5 seconds
before we log an error and abort

(This used to be ctdb commit 6d1e4321b63973c2e53c63d386e8cc0bd9605cae)
2009-06-19 13:09:11 +10:00
Ronnie Sahlberg
e6170b5389 add a new node state : DELETED.
This is used to mark nodes as being DELETED internally in ctdb
so that nodes are not renumbered if / when they are removed from the nodes file.

This is used to be able to do "ctdb reloadnodes" at runtime without
causing nodes to be renumbered.
To do this, instead of deleting a node from the nodes file, just comment it out like

   1.0.0.1
   #1.0.0.2
   1.0.0.3

After removing 1.0.0.2 from the cluster,  the remaining nodes retain their
pnn's from prior to the deletion, namely 0 and 2

Any line in the nodes file that is commented out represents a DELETED pnn

(This used to be ctdb commit 6a5e4fd7fa391206b463bb4e976502f3ac5bd343)
2009-06-01 14:18:34 +10:00
Ronnie Sahlberg
98a54c4675 Track how long it takes to take out the recovery lock from both the main dameon and also from the recovery daemon.
Log this in "ctdb statistics".

Also add a varaible "RecLockLatencyMs" that will log an error everytime it takes longer than this to access the reclock file.

(This used to be ctdb commit 042377ed803bb8f7ca9d6ea1a387427b7b8ba45a)
2009-05-14 10:33:25 +10:00
Ronnie Sahlberg
42891227a4 add extra debug statements to the log to make it easier to see when a recovery dameon has hung due to the underlying filesystem hanging.
(This used to be ctdb commit 5b0067a4e335cbbf6e606646e612d4bfcfdb7441)
2009-05-12 18:39:34 +10:00
root
6793f077a8 Add a new variable VerifyRecoveryLock which can be used to disable the test that the recovery daemon holds the lock properly when performing a recovery
(This used to be ctdb commit 329df9e47e6ca8ab5143985a999e68f37c6d88a5)
2009-05-01 01:17:59 +10:00
Ronnie Sahlberg
38ea6708dd add a tuneable RecoveryDropAllIPs so it is possible to control after how long a node that has been stuck in recovery will wait until it will yield all public addresses.
this now defaults to 60 seconds

This is useful if a split brain occurs due to network partitioning since it will make sure that the "other half" of the cluster that does not contain the recovery master will eventually release all ips and thus avoiding a duplicate ip situation for the public addresses

(This used to be ctdb commit 70f21428c9eec96bcc787be191e7478ad68956dc)
2009-04-24 18:28:08 +10:00
Ronnie Sahlberg
ce3283f7cb increase the loglevel for the message we print when we automatically release all ips when we have been in recovery for too long
(This used to be ctdb commit 7af060ded5113a49832f6a08a942523a202586b3)
2009-04-24 18:11:10 +10:00
Ronnie Sahlberg
7265c713db we need to set the port properly in the parse_ip helper
(This used to be ctdb commit 43fe18d86995744ba61c7a6405b70edcb265930a)
2009-03-24 13:45:11 +11:00
Michael Adam
3cca0f75e4 Fix treatment of link local ipv6 addresses: set the scope id.
metze / Michael

Signed-off-by: Michael Adam <obnox@samba.org>

(This used to be ctdb commit 9d12de1ca6107801dada927729e755c0949d73bf)
2009-01-19 22:50:53 +01:00
Ronnie Sahlberg
edb7241c05 redesign how reloadnodes is implemented.
modify the transport methods to allow to restart individual connections
and set up destructors properly.

only tear down/set-up tcp connections to nodes removed from the cluster
or nodes added to the cluster.
Leave tcp connections to unchanged nodes connected.

make "ctdb reloadnodes" explicitely cause a recovery of the cluster once
the files have been realoaded

(This used to be ctdb commit d1057ed6de7de9f2a64d8fa012c52647e89b515b)
2008-12-02 13:26:30 +11:00
Ronnie Sahlberg
1e2831898c allow to change the recmaster even the database is not frozen
(This used to be ctdb commit 03e2e436db5cfd29a56d13f5d2101e42389bfc94)
2008-11-21 16:24:12 +11:00
Andrew Tridgell
59b6a9a9e6 fixed problem with looping ctdb recoveries
After a node failure, GPFS can get into a state where non-blocking
fcntl() locks can take a long time. This means to the ctdb set_recmode
test timing out, which leads to a recovery failure, and a new
recovery. The recovery loop can last a long time.

The fix is to consider a fcntl timeout as a success of this test. The
test is to see that we can't lock the shared reclock file, so a
timeout is fine for a success.

(This used to be ctdb commit 6579a6a2a7161214adedf0f67dce62f4a4ad1afe)
2008-11-21 10:24:13 +11:00
Ronnie Sahlberg
b9bd20ce55 add a context and a timed event so that once we have been in recovery
mode for too long we drop all public ip addresses

(This used to be ctdb commit 403c68f96e1380dd07217c688de2730464f77ea0)
2008-10-22 11:04:41 +11:00
Ronnie Sahlberg
cb300382b0 update TAKEIP/RELEASEIP/GETPUBLICIP/GETNODEMAP controls so we retain an
older ipv4-only version of these controls.

We need this so that we are backwardcompatible with old versions of ctdb
and so that we can interoperate with a ipv4-only recmaster during a
rolling upgrade.

(This used to be ctdb commit 6b76c520f97127099bd9fbaa0fa7af1c61947fb7)
2008-10-14 10:40:29 +11:00
Ronnie Sahlberg
46187433ca remove an unused variable
(This used to be ctdb commit 4237bd3753dcb024c17461e974414bef1b609416)
2008-10-07 18:14:44 +11:00
Ronnie Sahlberg
1778280d50 When we reload the nodes file
instead of shutting down/restarting the entire tcp layer
just bounce all outgoing connections and reconnect

(This used to be ctdb commit e701a531868149f16561011e65794a4a46ee6596)
2008-10-07 18:12:54 +11:00
Ronnie Sahlberg
3e274e5f8c use the correct tunable failcount not timeout
(This used to be ctdb commit 475cfada33b4c13aaaca773d5485bbe26bffbf46)
2008-09-17 14:24:12 +10:00
Ronnie Sahlberg
a3bbe238c9 The ctdb daemon keeps track of whether the recovery process is running
correctly by measuring how long it was since the last successful
communication with the recovery daemon was recorded.

After a certain timeout the ctdb daemon would deem the recovery daemon
as inoperable and shut down.

If the system clock is suddenly changed forward by many (60 or more)
seconds this could cause the timeout to trigger prematurely/immediately
where ctdb would incorrectly think that more than 60 seconds had passed
since last successful communications and thus abort.

Instead of cehcking for one timeout occuring, only deem the recovery
daemon to be "down" and trigger a shutdown if communications have
timedout for three intervals in a row.

(This used to be ctdb commit 196968c552e6ebcb57389d769a4b25f42fa8bc5d)
2008-09-17 14:17:41 +10:00
Ronnie Sahlberg
6474f3278d additional monitoring between the two daemons.
we currently only monitor that the dameons are running by kill(0, pid)
and verifying the the domain socket between them is ok.

this is not sufficient since we can have a situation where the recovery
daemon is hung.

this new code monitors that the recovery daemon is operating.
if the recovery hangs, we log this and shut down the main daemon

(This used to be ctdb commit cd69d292292eaab3aac0e9d9fc57cb621597c63c)
2008-09-09 13:44:46 +10:00
Ronnie Sahlberg
ef997d344f initial ipv6 patch
Signed-off-by: Ronnie Sahlberg <ronniesahlberg@gmail.com>

(This used to be ctdb commit 1f131f21386f428bbbbb29098d56c2f64596583b)
2008-08-19 14:58:29 +10:00
Ronnie Sahlberg
b9d8bb23af remove the reclock file we store pnn counts in.
This file creates additional locking stress on the backend filesystem and we may not need it anyway.

(This used to be ctdb commit 84236e03e40bcf46fa634d106903277c149a734f)
2008-08-06 11:52:26 +10:00
Andrew Tridgell
cf739ac892 renamed the pulldb structure to a ctdb_marshall_buffer
(This used to be ctdb commit bad53b2d342bb9760497e6f4a61e64ca50d6e771)
2008-07-30 19:59:18 +10:00
Andrew Tridgell
abe0232818 rename the structure we use for marshalling multiple records
(This used to be ctdb commit 4d205476d286570a6e1f52b59af42858ce051106)
2008-07-30 14:24:56 +10:00
Ronnie Sahlberg
334db8ccba proper waitpid() fix.
remove all waitpid() calls and use the event system to trap sigchld

(This used to be ctdb commit 77458b2b6b51b2970c12b0e5b097088d3fb9d358)
2008-07-09 14:02:54 +10:00