1
0
mirror of https://github.com/samba-team/samba.git synced 2024-12-30 13:18:05 +03:00
Commit Graph

1374 Commits

Author SHA1 Message Date
Martin Schwenke
44a0466ac1 ctdb-recoverd: Only respond to currently queued ipreallocated requests
Otherwise new requests can come in during the latter parts of the
takeover run when the IP allocation algorithm has already run, and the
new requests will be dequeued even though they haven't really be
processed.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
2013-11-27 18:46:16 +01:00
Martin Schwenke
efc77ba6ac ctdb-recoverd: For persistent databases a sequence number of 0 is valid
Otherwise recovery ends up done by RSN when it is unnecessary.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
2013-11-27 18:46:16 +01:00
Amitay Isaacs
4ea721b2c1 ctdb-locking: Use vfork instead of fork to exec helpers
There is a significant overhead using fork() over vfork(), specially
when the child process execs a helper.  The overhead is in memory space
and time.

    # strace -c ./test_fork 1024 200
    count=1024, size=204800, total=200M
    failed fork=0
    time for fork() = 4879.597000 us
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    100.00    4.543321        3304      1375       375 clone
      0.00    0.000071           0      1033           mmap
      0.00    0.000000           0         1           read
      0.00    0.000000           0         3           write
      0.00    0.000000           0         2           open
      0.00    0.000000           0         2           close
      0.00    0.000000           0         3           fstat
      0.00    0.000000           0         3           mprotect
      0.00    0.000000           0         1           munmap
      0.00    0.000000           0         3           brk
      0.00    0.000000           0         1         1 access
      0.00    0.000000           0         1           execve
      0.00    0.000000           0         1           arch_prctl
    ------ ----------- ----------- --------- --------- ----------------
    100.00    4.543392                  2429       376 total

    # strace -c ./test_vfork 1024 200
    count=1024, size=204800, total=200M
    failed fork=0
    time for fork() = 82.041000 us
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     96.47    0.001204           1      1000           vfork
      3.53    0.000044           0      1033           mmap
      0.00    0.000000           0         1           read
      0.00    0.000000           0         3           write
      0.00    0.000000           0         2           open
      0.00    0.000000           0         2           close
      0.00    0.000000           0         3           fstat
      0.00    0.000000           0         3           mprotect
      0.00    0.000000           0         1           munmap
      0.00    0.000000           0         3           brk
      0.00    0.000000           0         1         1 access
      0.00    0.000000           0         1           execve
      0.00    0.000000           0         1           arch_prctl
    ------ ----------- ----------- --------- --------- ----------------
    100.00    0.001248                  2054         1 total

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
2013-11-27 18:46:16 +01:00
Amitay Isaacs
0eeb73c187 ctdb-locking: Update current lock statistics when lock is scheduled
When a child process is created for a lock request, the current locks
statistics should be updated immediately.  This will provide accurate
information on number of active lock requests.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
2013-11-27 18:46:16 +01:00
Amitay Isaacs
3879e9991f ctdb-locking: Do not merge multiple lock requests to avoid unfair scheduling
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
2013-11-27 18:46:16 +01:00
Amitay Isaacs
094f34e9bf ctdb-locking: Implement active lock requests limit per database
This limit was currently a global limit and not per database.  This
prevents any database freeze lock requests from getting scheduled if
the global limit was reached.

Only individual record requests should be limited and database freeze
requests should always get scheduled.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
2013-11-27 18:46:16 +01:00
Martin Schwenke
028fe930b6 ctdb-recoverd: Fix backward compatibility for CTDB_SRVID_TAKEOVER_RUN
When running a mixed version cluster, compatibility with older
versions was was broken during recent refactorisation.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
2013-11-27 18:46:16 +01:00
Martin Schwenke
6fbf399191 ctdb-recoverd: A node refuses to play against itself
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
2013-11-27 18:46:16 +01:00
Martin Schwenke
2038d166ad ctdb-recoverd: Remove duplicate code to update flags during recovery
This also happens earlier in do_recovery() and the nodemap is not
updated after that, so this update is redundant.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Michael Adam <obnox@samba.org>
2013-11-27 18:46:16 +01:00
Amitay Isaacs
6d1b74f052 ctdb-server: Coverity fixes
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>
2013-11-19 17:13:03 +01:00
Amitay Isaacs
41d37058ca tunables: Remove obsolete tunables
Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit ca5fc3431573c44d55d09d987c715fb53756fc1f)
2013-10-30 15:37:11 +11:00
Martin Schwenke
62076d3089 recoverd: Rebalancing should be done regardless tunable
Rebalance target nodes should be set even if a deferred rebalance is
not configured.  The user can explicitly cause a takeover run.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit afd9b51644af074752d74c412cb4e7ec2eba2c69)
2013-10-30 12:19:49 +11:00
Martin Schwenke
6b42805717 recoverd: Improve an error message in the election code
Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 275ed9ebe287e39d891888c13810c70f347af8ac)
2013-10-30 11:34:56 +11:00
Martin Schwenke
5f80f4255c Revert "if a new node enters the cluster, that node will already be frozen at start"
This is unnecessary due to 03e2e436db5cfd29a56d13f5d2101e42389bfc94.
Furthermore, if a node doesn't force an election but wins it then it
can fail to record that it is the new recovery master.  This can lead
to a reverse split brain where there is no recovery master.

This reverts commit c5035657606283d2e35bea40992505e84ca8e7be.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

Conflicts:
	server/ctdb_recoverd.c

(This used to be ctdb commit c8b542e059a54b8d524bd430cad9d82e5edd864d)
2013-10-30 11:34:56 +11:00
Martin Schwenke
45b44a7155 ctdbd: When a node is connected, log at DEBUG NOTICE not DEBUG_INFO
This is important enough that we should see it when the log level is
DEBUG_NOTICE.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit eb8ec5681bfccb26c8ffae72952d54bb0ba46249)
2013-10-29 17:14:56 +11:00
Martin Schwenke
f88cf2d013 Revert "recoverd: Disable takeover runs on other nodes for 5 minutes"
5 minutes is too long to leave the cluster in limbo if the recovery
daemon dies during a takeover run, even though this is quite unlikely.
We need a new recover master to be able to do takeover runs fairly
quickly.

This reverts commit 71080676bb4acbd0d9b595a30cf7fe6dddbf426f.

(This used to be ctdb commit 3e41170c78fc7a2bf526129c9b7db3739b61c6bf)
2013-10-29 17:14:55 +11:00
Amitay Isaacs
fc7f335843 daemon: Change the default recovery method for persistent databases
Use sequence numbers to do recovery for persistent databases instead of
RSNs.  This fixes the problem of registry corruption during recovery.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 56486d1c01cc8ad0e4b8cee7a22429e72e50f03d)
2013-10-28 18:51:22 +11:00
Amitay Isaacs
4432aef6d1 packaging: Move ctdb/ directory from /var to /var/lib
Introduce CTDB_VARDIR variable that points to /var/lib/ctdb by default.
This makes CTDB_VARDIR consistent across C code and scripts.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 2c09aac71188f43cd592572b10ea30b7a2969678)
2013-10-25 12:06:07 +11:00
Martin Schwenke
b595712f25 ctdbd: Simplify database directory setting logic
No need to check if the options are set.  The options are always set
via static defaults.

No need to talloc_strdup() the values via wrapper functions.  The
options aren't going away.  Remove now unused ctdb_set_tdb_dir() and
similar functions.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 1fe82f3d7b610547ff4945887f15dd6c5798a49b)
2013-10-25 12:06:06 +11:00
Martin Schwenke
a604c3d945 ctdbd: Remove duplicate database directory setting logic
Defaults for ctdb->db_directory and similar variables are currently
set in 2 places.

Change this to set them in only 1 place and make the directories at
initialisation time instead of waiting until later.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit d73d84346488a2ed54e6a86f9d7ec641c8e33ace)
2013-10-25 12:06:06 +11:00
Martin Schwenke
e782b61732 ctdbd: Pass the public address file location in ctdb context
No need to pass it as an extra argument to ctdb_start_daemon.

Also ensure options.public_address_list gets a nice static default.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit a3d63a9db89d08bb284b3b3a6db773422f21b477)
2013-10-22 15:37:54 +11:00
Martin Schwenke
463a091a77 ctdbd: Debug locks by default with override from enviroment variable
Default is debug_locks.sh, relative to CTDB_BASE.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit c11803e3dcc905a45a08d743595e63f9ca445f0d)
2013-10-22 15:37:54 +11:00
Martin Schwenke
4adc8f4f09 ctdbd: Default for event_script_dir should use CTDB_BASE
Also get rid of ctdb_set_event_script_dir().  It creates an
unnecessary copy of something that will be around for the lifetime of
the process.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 21b4d1aba00902f1eee0cbf4f082b0794fd5b738)
2013-10-22 15:37:54 +11:00
Martin Schwenke
f9ce563135 ctdbd: Add nodes_file member to struct ctdb_context
This allows ctdb_load_nodes_file() to move to ctdb_server.c and
ctdb_set_nlist() to become static.

Setting ctdb->nodes_file needs to be done early, before the nodes file
is loaded.  It is now set from CTDB_BASE instead ETCDIR, so setting
CTDB_BASE also needs to be done earlier.

Unhack ctdbd_test.c - it no longer needs to define
ctdb_load_nodes_file().

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 20e705e63bd3b20837cc3ac92fdcf2a9650ccfc8)
2013-10-22 15:37:54 +11:00
Martin Schwenke
7c90395136 ctdbd: Don't check CTDB_BASE before setting it, just don't override
That's what the 3rd argument to setenv(3) is for...  :-)

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 30ca419aa1c78008f81839497921bbfba480e7fc)
2013-10-22 15:37:54 +11:00
Martin Schwenke
82e5effc40 ctdbd: Fix some errors in the popt configuration
That 4th argument isn't a default or similar, so consistently make it 0.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 1c0a627df1b510f49c65ffeb4474240c8856cdf2)
2013-10-22 14:34:05 +11:00
Martin Schwenke
fbd2617cb8 recoverd: Remove function reload_nodes_file()
It is a 1 line wrapper around ctdb_load_nodes_file(), so use that
instead.  We need less code...  :-)

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 4a5d5935f4410a93a3343d85a24dbcddae2c4c20)
2013-10-22 14:34:03 +11:00
Martin Schwenke
a93361fca2 Revert "null out the pointer before we reload the nodes file"
This reverts commit 4b0f32047e8bece0a052bdbe2209afe91b7e8ce3.

This is not necessary.  It just causes a memory leak.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 25fd05505f61dc595c0ef25bb6e332274d5530e8)
2013-10-22 14:34:03 +11:00
Amitay Isaacs
e63232e974 recoverd: Ignore failed flag updates on inactive nodes
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-programmed-with: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 484c46eaae056480baf050fd91868f2fd0537985)
2013-10-22 14:34:03 +11:00
Martin Schwenke
4812291ff8 recoverd: Fix the VNN lmaster consistency check
It does cope with node that don't have the lmaster capability.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 588172bcb6bf267339e2bd09e23d2c4904a27a41)
2013-10-22 11:49:54 +11:00
Amitay Isaacs
be33efa3e4 ctdbd: Remove transaction code related to TRANS2 commits
This removes data types and structure elements related to TRANS2
persistent transaction code.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 22a253b7ccf1ff854cddf0b67969dc84d7d6a654)
2013-10-04 15:20:25 +10:00
Amitay Isaacs
91d644325d ctdbd: Deprecate TRANS2 commit controls
Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 7d176352986317e63696d74252ff5d8eccb2fee5)
2013-10-04 15:20:25 +10:00
Amitay Isaacs
1ff9645865 ctdbd: Create a utility function to log error for "not implemented" controls
Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 3c892ea1b5aa42686adb82ce29b9fcfdf9d204a1)
2013-10-04 15:20:25 +10:00
Amitay Isaacs
a2d6bbe67a traverse: Send traverse end record from traverse child process
Traverse records are sent directly from traverse child process, but
the last empty record signalling end of traverse is sent from ctdbd.
This creates a race condition between ctdbd and traverse child.
There are two fds from traverse child to ctdbd - a pipe to track status
of the child process and unix socket connection for sending records.
It's possible that last few records are sitting in unix socket buffer
when ctdbd reads the status written from traverse child.  This will
be interpreted as end of traverse and ctdbd will send the last empty
record to originating node before it has processed the pending packets
in unix socket connection.

The race is avoided by sending the last empty record marking end of
traverse from the child process.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 37e22fc3ac3eb64732f2e67058f5b7b06c093fbf)
2013-09-25 14:59:45 +10:00
Amitay Isaacs
f1f1788f10 traverse: Wait till all data has been flushed from output queue
To improve the traverse performance, records are directly sent from
traverse child process to the originating node.  Make sure that all the
data is sent via socket, before informing ctdbd that traverse is complete.

Without waiting for all the packets to be flushed from the queue,
child process can incorrectly signal ctdbd that traverse has ended.
This will cause the pending records in the queue never to make it to
the originating node and traverse information will not be complete.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 482ac708cb79cb6378d814a79c2cf13f88435bc4)
2013-09-25 14:59:45 +10:00
Amitay Isaacs
1740cbb58c traverse: Use ctdb local variable for convenience
Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 25e9cf86328252f96215b54b94551dd7bbdd2db4)
2013-09-25 14:59:45 +10:00
Amitay Isaacs
c4f49a5342 traverse: Check if local traverse failed or succeeded
By passing the result of tdb_traverse_read() allows ctdbd to determine
if the local traverse succeeded or not.  In case of a problem with local
traverse, ctdbd can log an error.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit abd51a9f41ebb178c4ea4491bdedf9a9433e7232)
2013-09-25 14:59:45 +10:00
Amitay Isaacs
76d9d2e5e1 traverse: Log information when traverse starts and ends
Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit e4aba8598b00a810e721de64ac44dccc9af04ab6)
2013-09-25 14:59:45 +10:00
Martin Schwenke
430ae84877 recoverd: Disable takeover runs on other nodes for 5 minutes
60 seconds might not be long enough to kill all connections and
release IPs.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 71080676bb4acbd0d9b595a30cf7fe6dddbf426f)
2013-09-19 12:58:32 +10:00
Martin Schwenke
07d3a1b234 recoverd: Improve logging for takeover runs
Takeover runs are currently silent when they succeed.  However, they
are important, so log something by default.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit b39aa2e401fbb581207d986bac93778e9c01acdc)
2013-09-19 12:57:36 +10:00
Martin Schwenke
566d66e6ab recoverd: Be careful about freeing the list of IP rebalance target nodes
It can change during a takeover run.  If it does then don't free it.

There are potentially fancier solutions (e.g. check what PNNs are new
to the list) to this issue but this is the simplest.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit e81589b7084c661adf617e166cc2c25b4939f841)
2013-09-19 12:54:31 +10:00
Martin Schwenke
4fb0d4a301 recoverd: reloadips should rebalance target nodes for new IPs
Otherwise, if existing IPs are added to extra nodes (that have,
perhaps, been disconnected) then those IPs will not be rebalanced
across the extra nodes.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit ceb30432a9a550778aed0b422a654fc5287b82a3)
2013-09-19 12:54:31 +10:00
Martin Schwenke
950e23f664 ctdbd: Make ctdb_reloadips_child send controls asynchronously
Deleting IPs can take a while because IPs are released and connections
are killed.  This can take a while so do them in parallel.  In fact,
since the set of IPs being added and deleted will be disjoint, send
all the adds/deletes at the same time and then wait.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 85a5b544ec032173e98c9cc3b5402a76b961aa3b)
2013-09-19 12:54:31 +10:00
Martin Schwenke
b33ee7a2a4 recoverd: Fix the implementation of CTDB_SRVID_REBALANCE_NODE
The current implementation has a few flaws:

* A takeover run is called unconditionally when the timer goes even if
  the recovery master role has moved.  This means a node other than
  the recovery master can incorrectly do a takeover run.

* The rebalancing target nodes are cleared in the setup for a takeover
  run, regardless of whether the takeover run succeeds.

* The timer to force a rebalance isn't cleared if another takeover run
  occurs before the deadline.  Any forced rebalancing will happen in
  the first takeover run and when the timer expires some time later
  then an unnecessary takeover run will occur.

* If the recovery master role moves then the rebalancing data will
  stay on the original node and affect the next takeover run to occur
  if the recovery master role should come back to the original node.

Instead, store an array of rebalance target nodes in the recovery
master context.  This is passed as an extra argument to
ctdb_takeover_run() each time it is called and is cleared when a
takeover run succeeds.  The timer hangs off the array of rebalance
target nodes, which is cleared if the node isn't the recovery master.

This means that it is possible to lose rebalance data if the recovery
master role moves.  However, that's a difficult problem to solve.  The
best way of approaching it is probably to try to stop the recovery
master role from jumping around unnecesarily when inactive nodes join
the cluster.

The long term solution is to avoid this nonsense completely.  The IP
allocation algorithm needs to cache state between runs so that it
knows which nodes have just become healthy.  This also needs recovery
master stability.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit c51c1efe5fc7fa668597f2acd435dee16e410fc9)
2013-09-19 12:54:31 +10:00
Martin Schwenke
1793412de2 recoverd: Remove unused CTDB_SRVID_RELOAD_ALL_IPS and handler
Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 4cd727439a0824ebb8dbcf737d9888ffc3c41184)
2013-09-19 12:54:31 +10:00
Martin Schwenke
e7cc998570 recoverd: Defer ipreallocated requests when takeover runs are disabled
The takeover run will fail anyway but deferring seems like a cleaner
option.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 428f800bcdf3dbfe19de8bb36099fbf01ebeaab4)
2013-09-19 12:54:31 +10:00
Martin Schwenke
2f472b4573 recoverd: Reimplement CTDB_SRVID_DISABLE_IP_CHECK
Use disable_takeover_runs_handler() instead of maintaining duplicate
logic.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 0a51a85915486b2a8fded7ba6444b18c6c1ee8e8)
2013-09-19 12:54:31 +10:00
Martin Schwenke
5f0913d321 recoverd: New SRVID message CTDB_SRVID_DISABLE_TAKEOVER_RUNS
This implements a superset of CTDB_SRVID_DISABLE_IP_CHECK.  It stops
the IP checks but also causes any attempted takeover runs to fail and
be rescheduled.

This is meant to completely stop IP movements.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 00db4de53a0d86013e79e6577e7e6cf3ef864e56)
2013-09-19 12:54:31 +10:00
Martin Schwenke
0ba7e2ce31 recoverd: Factor out the SRVID handling code
The code that handles IP reallocate requests can be reused.

This also changes the result back to a SRVID caller to the PNN on
success or a negative error code on failure.  None of the callers
currently look at the result so this is harmless... but it will be
useful later.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit e4eae6e3291baa299a1d0f733ab11b138ee699a3)
2013-09-19 12:54:30 +10:00
Martin Schwenke
4c3f8dc3bb recoverd: Make the SRVID request structure generic
No need for a separate one for each SRVID.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit d9c22b04d5aa7938a3965bd3144568664eb772ce)
2013-09-19 12:54:30 +10:00