1
0
mirror of https://github.com/samba-team/samba.git synced 2025-02-01 05:47:28 +03:00

8057 Commits

Author SHA1 Message Date
Martin Schwenke
5419978537 ctdb-common: Fix memory leak
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13943

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 30bc6e2529cdd444d4ec7902844c3a6fb0858090)
2019-05-17 07:18:32 +00:00
Martin Schwenke
76c7302105 ctdb-recoverd: Fix memory leak
state is always freed before exiting this function, so allocate fde
off it instead of long-lived ctdb context.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13943

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 6a2941e2a9fd6ab2d5b8dbac042b61a7b1b0b914)
2019-05-17 07:18:32 +00:00
Andreas Schneider
925871f580 ctdb:common: Do not print NULL if we don't get a sockpath
sock_socket_start_recv() might not fill sockpath if we return early.

Found by GCC 9.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13937

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 830cb7e67568de5f3ce359cb6af3be8ab545c824)
2019-05-17 07:18:31 +00:00
Martin Schwenke
1c2c081f43 ctdb-daemon: Never use 0 as a client ID
ctdb_control_db_attach() and ctdb_control_db_detach() assume that any
control with client ID 0 comes from another daemon and treat it
specially.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13930

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 8663e0a64fbdb9ea16babbfe87d6f5d7a7b72bbd)
2019-05-17 07:18:30 +00:00
Martin Schwenke
24d70220b2 ctdb-tests: Fix logic error in simple ctdb reloadips test
There is a chance that restoring IP addresses to the test node will
result in different IP addresses being assigned to that node.
Removing a single IP address may then fail (or be a no-op) if it is
done after the restore.

So, swap the single IP address removal to happen first, then restore,
then remove all IP addresses.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit dc89db8ca6aadd4a9f7e8a85843c53709d04587c)
2019-05-17 07:18:30 +00:00
Martin Schwenke
9f679ba14d ctdb-tests: Make ctdb reloadips tests more reliable
ctdb reloadips will fail if it can't disable takover runs.  The most
likely reason for this is that there is already a takeover run in
progress.  We can't predict when this will happen, so retry if this
occurs.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 8be4ee1a28d5c037955832b6f827d40f28f02796)
2019-05-17 07:18:30 +00:00
Martin Schwenke
0ffba5145c ctdb-tests: Capture output in $out on failure as well
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit cf00db40355b49443263187f9d97934f91287e51)
2019-05-17 07:18:30 +00:00
Martin Schwenke
1eb5d2e4fc ctdb-tests: Don't clean up test var directory in autotest target
If the directory is always cleaned up then it is not possible to look
at daemon logs to debug test failures.

This target is only really used by autobuild.py, which (optionally)
cleans up the parent directory anyway.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue May  7 06:56:01 UTC 2019 on sn-devel-184

(cherry picked from commit 5a9e338330fe136908a3a17a5df81c054c5cc5b0)
2019-05-17 07:18:30 +00:00
Martin Schwenke
15e5d62b3d ctdb-tests: Fix usage message
Since commit 0e9ead8f28fced3ebfa888786a1dc5bb59e734a3 daemons have
been shut down after each test, so this option no longer has anything
to do with killing daemons.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit a2ab6485e027ebb13871c7d83b7626ac5c9b98c0)
2019-05-17 07:18:30 +00:00
Martin Schwenke
814471f46e ctdb-tests: Wait to allow database attach/detach to take effect
Sometimes the detach test fails:

  Check detaching single test database detach_test1.tdb
  BAD: database detach_test1.tdb is still attached
  Number of databases:4
  dbid:0x5ae995ee name:detach_test4.tdb path:tests/var/simple/node.0/db/volatile/detach_test4.tdb.0
  dbid:0xd84cc13c name:detach_test3.tdb path:tests/var/simple/node.0/db/volatile/detach_test3.tdb.0
  dbid:0x8e8e8cef name:detach_test2.tdb path:tests/var/simple/node.0/db/volatile/detach_test2.tdb.0
  dbid:0xc62491f4 name:detach_test1.tdb path:tests/var/simple/node.0/db/volatile/detach_test1.tdb.0
  Number of databases:3
  dbid:0x5ae995ee name:detach_test4.tdb path:tests/var/simple/node.1/db/volatile/detach_test4.tdb.1
  dbid:0xd84cc13c name:detach_test3.tdb path:tests/var/simple/node.1/db/volatile/detach_test3.tdb.1
  dbid:0x8e8e8cef name:detach_test2.tdb path:tests/var/simple/node.1/db/volatile/detach_test2.tdb.1
  Number of databases:4
  dbid:0x5ae995ee name:detach_test4.tdb path:tests/var/simple/node.2/db/volatile/detach_test4.tdb.2
  dbid:0xd84cc13c name:detach_test3.tdb path:tests/var/simple/node.2/db/volatile/detach_test3.tdb.2
  dbid:0x8e8e8cef name:detach_test2.tdb path:tests/var/simple/node.2/db/volatile/detach_test2.tdb.2
  dbid:0xc62491f4 name:detach_test1.tdb path:tests/var/simple/node.2/db/volatile/detach_test1.tdb.2
  *** TEST COMPLETED (RC=1) AT 2019-04-27 03:35:40, CLEANING UP...

When issued from a client, the detach control re-broadcasts itself
asynchronously to all nodes and then returns success.  The controls to
some nodes to do the actual detach may still be in flight when success
is returned to the client.  Therefore, the test should wait for a few
seconds to allow the asynchronous controls to complete.

The same is true for the attach control, so workaround the problem in
the attach test too.

An alternative is to make the attach and detach controls synchronous
by avoiding the broadcast and waiting for the results of the
individual controls sent to the nodes.  However, a simple
implementation would involve adding new nested event loops.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 3cb53a7a05409925024d6a67bcfaeb962d896e0b)
2019-05-17 07:18:30 +00:00
Martin Schwenke
3f104bd0db ctdb-tests: Avoid bulk output in $out, prefer $outfile
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 066cc5b0c561464ed08890d9aa1a1a55b545e9cc)
2019-05-17 07:18:30 +00:00
Martin Schwenke
b594f5161d ctdb-tests: Make try_command_on_node less error-prone
This sometimes fails, apparently due to a cat process in onnode
getting EAGAIN.  The conclusion is that tests that process large
amounts of output should not depend on a sub-shell delivering that
output into a shell variable.

Change try_command_on_node() to leave all of the output in file
$outfile and just put the first 1KB into $out.  $outfile is removed
after each test completes.

Change the implementation of sanity_check_output() to use $outfile
instead of $out.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 9d02452a24625df5f62fd6d45a16effe2fa45fbe)
2019-05-17 07:18:29 +00:00
Martin Schwenke
7c97bc8328 ctdb-tests: Change sanity_check_output() to internally use $out
All callers are currently passed $out.  Global variable $out is used
in many other places so use it here to simplify the interface and make
future changes simpler.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 7c3819d1ac264acf998f426e0cef7f6211e0ddee)
2019-05-17 07:18:29 +00:00
Martin Schwenke
30b5d837d5 ctdb-tests: Extend test to cover ctdb rddumpmemory
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13923

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 8108b3134c017c22d245fc5b2207a88d44ab0dd2)
2019-05-17 07:18:29 +00:00
Martin Schwenke
08e229df43 ctdb-tools: Fix ctdb dumpmemory to avoid printing trailing NUL
Fix ctdb rddumpmemory too.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13923

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit f78d9388fb459dc83fafb4da6e683e3137ad40e1)
2019-05-17 07:18:29 +00:00
Amitay Isaacs
945a41d384 ctdb-common: Avoid race between fd and signal events
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13895

In run_proc, there was an implicit assumption that when a process exits,
fd event (pipe between parent and child) would be processed first and
signal event (SIGCHLD for the child) would be processed later.

However, that is not the case.  SIGCHLD can be received asynchronously
any time even when the pipe data has not fully been read.  This causes
run_proc to miss some of the output from child process in tests.

When SIGCHLD is being processed, if the pipe between parent and child is
still open, then do an explict read from the pipe to ensure we read any
data still in the pipe before closing the pipe.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Apr 12 08:19:29 UTC 2019 on sn-devel-144

(cherry picked from commit 289201277cd983b27cdfd5376c607eab112b4082)

Autobuild-User(v4-9-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-9-test): Mon Apr 15 12:55:46 UTC 2019 on sn-devel-144
2019-04-15 12:55:46 +00:00
Martin Schwenke
d9c47cb86e ctdb-daemon: Revert "We can not assume that just because we could complete a TCP handshake"
We also can not assume that nodes can be marked as connected via only
the keepalive mechanism.  Keepalives are not sent to disconnected
nodes so, in the absence of other packets (e.g. broadcasts), 2 nodes
may never become marked as connected to each other.

Revert to marking nodes as connected in the TCP transport code.  If a
connection is to a non(-operational) ctdbd then it will revert to
disconnected after a short while and may actually flap.  This should
be rare.

This reverts commit 66919db3d7ab1e091223faf515b183af8bfddc83.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13888

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 38dc6d11a26c2e9a2cae7927321f2216ceb1c5ec)
2019-04-15 08:28:11 +00:00
Martin Schwenke
49fa08814e ctdb-scripts: Update statd-callout to try several configuration files
The alternative seems to be to try something via CTDB_NFS_CALLOUT.
That would be complicated and seems like overkill for something this
simple.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13860

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit a2bd4085896804ee2da811e17f18c78a5bf4e658)
2019-04-12 07:57:11 +00:00
Martin Schwenke
dae0e8ec96 ctdb-scripts: Allow load_system_config() to take multiple alternatives
The situation for NFS config has got more complicated and is probably
broken in statd-callout on Debian-like systems at the moment.  Allow
several alternative configuration names to be tried.  Stop after the
first that is found and loaded.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13860

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit 0d67ea5fcca766734ecc73ad6b0139f7c13a15c5)
2019-04-12 07:57:11 +00:00
Martin Schwenke
14069988a9 ctdb-tests: Update NFS test infrastructure to support systemd services
The tests are written around the default of sysvinit-redhat.  Add
support for systemd-redhat.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13860

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit 2833ddcfcb780497264e0f412a9ad6e26a9bc657)
2019-04-12 07:57:11 +00:00
Martin Schwenke
aee71ea686 ctdb-scripts: Add systemd services to NFS call-out
At least Red Hat and Debian appear to use (a variant of?) the upstream
systemd units for NFS, so adding support for these services is
relatively easy.  Distributions using Sys-V init can patch the
call-out to use the relevant Sys-V init services.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13860

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit a8fafd377ff0cb07ab161e437c5fe024704345eb)
2019-04-12 07:57:11 +00:00
Martin Schwenke
7932032de4 ctdb-scripts: Start NFS quota service if defined
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13860

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit 708c04071af8d6ddc3bf2bddbde4d5847f440c0e)
2019-04-12 07:57:11 +00:00
Martin Schwenke
5a97b7f00a ctdb-scripts: Stop/start mount/rquotad/status via NFS call-out
When an NFS check restarts a failed service by hand then systemd will
be unable to stop or start this service again because (at least) the
PID file will be wrong.  Do this via the NFS Linux kernel call-out
instead.  Allow the call-out to use the services instead of doing
manual restarts.  Add variables for mount, status and rquotad services
to support this.

Adding systemd NFS services to the call-out will follow.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13860

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit 42103b568698d8087d27f0848b402ccb7cfac86b)
2019-04-12 07:57:11 +00:00
Martin Schwenke
f00827672c ctdb-scripts: Factor out nfs_load_config()
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13860

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit 8de0a339b550e7363d265af04ad69f2179af75c6)
2019-04-12 07:57:10 +00:00
Martin Schwenke
022b9a6ca7 ctdb-scripts: Add test variable CTDB_NFS_DISTRO_STYLE
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13860

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit e72c3c800a50fe746164e319e21180c44d041619)
2019-04-12 07:57:10 +00:00
Martin Schwenke
117586288b ctdb-scripts: Rename variable nfslock_service to nfs_lock_service
There will be more of these variable for other services so, for
readability, it makes sense for them to start with "nfs_".

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13860

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit 9981353ab79dce81b698c535977be4a681119d1e)
2019-04-12 07:57:10 +00:00
Martin Schwenke
d415458f6f ctdb-scripts: Reindent some functions prior to making changes
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13860

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit d7e187c1a7046196ec96637bdc14cc6b042eafcc)
2019-04-12 07:57:10 +00:00
Martin Schwenke
5f94bc518c ctdb-tests: Add some testing for IPv4-mapped IPv6 address parsing
ctdb_sock_addr values are hashed in some contexts.  This means that
all of the memory used for the ctdb_sock_addr should be consistent
regardless of how parsing is done.  The first 2 cases are just sanity
checks but the 3rd case involving an IPv4-mapped IPv6 address is the
real target of this test addition.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13839

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit d9286701cd9253bf3b42cac3d850ae8c23743e6d)
2019-03-22 09:55:36 +00:00
Zhu Shangzhong
eb4e66cd0f ctdb: Initialize addr struct to zero before reparsing as IPV4
Failed to kill the tcp connection that using IPv4-mapped IPv6 address
(e.g. ctdb_killtcp eth0 ::ffff:192.168.200.44:2049
::ffff:192.168.200.45:863).

When the ctdb_killtcp is used to kill the tcp connection, the IPs and
ports in the connection will be parsed to conn.client and conn.server
(call stack: main->ctdb_sock_addr_from_string->ip_from_string). In
the ip_from_string, as we are using IPv4-mapped IPv6 addresses, the
ipv6_from_string will be used to parse ip to addr.ip6 first. The next
step the ipv4_from_string will be used to reparse ip to addr.ip.

As a result, the data that dump from conn.server is "2 0 8 1 192 168
200 44 0 0 0 0 0 0 0 0 0 0 255 255 192 168 200 44 0 0 0 0", the data
from conn.client is "2 0 3 95 192 168 200 45 0 0 0 0 0 0 0 0 0 0 255 255
192 168 200 45 0 0 0 0". The connection will be add to conn_list by
ctdb_connection_list_add. Then the reset_connections_send uses conn_list
as parameter to start to reset connections in the conn_list.

In the reset_connections_send, the database "connections" will be
created. The connections from conn_list will be written to the
database(call db_hash_add), and use the data that dump from conn_client
and conn_server as key.

In the reset_connections_capture_tcp_handler, the
ctdb_sys_read_tcp_packet will receive data on the raw socket. And
extract the IPs and ports from the tcp packet. when extracting IP and
port, the tcp4_extract OR tcp6_extract will be used. Then we got the
new conn.client and conn.server. the data that dump from the
conn.server is "2 0 8 1 192 168 200 44 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0", the data from conn.client is "2 0 3 95 192 168 200 45 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0". Finally, we use the data as key to check
if this connection is one being reset(call db_hash_delete). The
db_hash_delete will return ENOENT. Because the two key that being used
by db_hash_delete and db_hash_add are different.

So, the TCP RST will be NOT sent for the connection forever. We should
initialize addr struct to zero before reparsing as IPV4 in the
ip_from_string.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13839

Signed-off-by: Zhu Shangzhong <zhu.shangzhong@zte.com.cn>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit 539b5ff32b32b7c75dfaaa119e41f5af6ff1e6fc)
2019-03-22 09:55:36 +00:00
Martin Schwenke
9e4b17aaab ctdb-packaging: Test package requires tcpdump
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13838

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit 957c38b65ca060eabe1e676f8dfb54839d706155)
2019-03-22 09:55:36 +00:00
Martin Schwenke
6e76e884f5 ctdb-packaging: ctdb package should not own system library directory
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13838

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@samba.org>
(cherry picked from commit b2b8dce4fc56c27ef0131104b316346565369dd7)
2019-03-22 09:55:35 +00:00
Martin Schwenke
eb16d3b7bc ctdb-cluster-mutex: Separate out command and file handling
This code is difficult to read and there really is no common code
between the 2 cases.  For example, there is no need to split a
filename into words.  Separating each of the 2 cases into its own
function makes the logic much easier to understand.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Feb 25 03:40:16 CET 2019 on sn-devel-144

(cherry picked from commit c93430fe8fe530a55b9a04cf6cc660c3d420e333)
(cherry picked from commit d5131afc533102ed5adfb147bf1a316e51810729)
2019-03-04 10:37:54 +00:00
Martin Schwenke
65c3c5801f ctdb-recoverd: Time out attempt to take recovery lock after 120s
Currently this will wait forever.  It really needs a timeout in case
the cluster filesystem (or other lock mechanism) is completely wedged.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 13a1a4808935290dceb219daccd7aac3fda4e184)
2019-03-04 10:37:54 +00:00
Martin Schwenke
4c059e03ef ctdb-recoverd: Ban node on unknown error when taking recovery lock
We really shouldn't see unknown errors.  They probably represent a
misconfigured recovery lock or similar.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 45a77d65b2e39b4af94da4ab99575f4ee08a7ebd)
2019-03-04 10:37:54 +00:00
Martin Schwenke
fd9a02c0bb ctdb-recoverd: Make recoverd context available in recovery lock handle
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit c0fb62ed3954fc6e8667480aba92003fc270f257)
2019-03-04 10:37:54 +00:00
Martin Schwenke
f63f2a0ee3 ctdb-recoverd: Clean up logging on failure to take recovery lock
Add an explicit case for a timeout and clean up the other messages.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 7e4aae6943291c3144c8a3ff97537e8d4c7dc7c9)
2019-03-04 10:37:54 +00:00
Martin Schwenke
fb8c3bd899 ctdb-recoverd: Free cluster mutex handler on failure to take lock
If nested events occur while the file descriptor handler is still
active then chaos can ensue.  For example, if a node is banned and the
lock is explicitly cancelled (e.g. due to election loss) then
double-talloc-free()s abound.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 621658cbed5d91d7096fc208bac2ff93a1880e7d)
2019-03-04 10:37:53 +00:00
Martin Schwenke
592f02112b ctdb-config: Change example recovery lock setting to one that fails
ctdbd will start without a recovery lock configured.  It will log a
message saying that this is not optimal.  However, a careless user may
overlook both this message and the importance of setting a recovery
lock.  If the existing example configuration is uncommented then the
directory containing it will be created (by 01.reclock.script) and the
failure (i.e. multiple nodes able to take the lock) will be confusing.

Instead, change the example setting to one that will result in banned
nodes, encouraging users to consciously configure (or deconfigure) the
recovery lock.  Tweak the corresponding comment.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13790

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit 82e7f38214896c2c200132bc6dde3348cfac16cc)
2019-03-04 10:37:53 +00:00
Volker Lendecke
d1428435b5 ctdb: Print locks latency in machinereadable stats
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13742
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Wed Jan 16 05:34:17 CET 2019 on sn-devel-144

(cherry picked from commit 193a0d6f01372604b925d1972591062a0bb2400f)

Autobuild-User(v4-9-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-9-test): Fri Feb  1 15:18:15 CET 2019 on sn-devel-144
2019-02-01 15:18:15 +01:00
Martin Schwenke
6a549df241 ctdb-daemon: Exit with error if a database directory does not exist
Since 4.9.0, the log messages can be confusing if a required database
directory does not exist.  Explicitly check for database directories,
logging a clear error and exiting if one is missing.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13696

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Dec  3 06:56:41 CET 2018 on sn-devel-144

(cherry picked from commit dd7574afd1b2fb6a88defa154bc3d15e94f9ce0d)

Autobuild-User(v4-9-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-9-test): Wed Dec  5 13:01:52 CET 2018 on sn-devel-144
2018-12-05 13:01:52 +01:00
Martin Schwenke
7cd5db7a63 ctdb-tests: Make the debug hung script test cope with unreadable stacks
Ideally this would just involve using "test -r".  However, operating
system security features may mean that kernel stacks are not readable
even though they appear to be.

Instead, try reading that stack of a process on the test node.  If
that succeeds then so should reading the stack of the "stuck" sleep
process in the test.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13684

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Tim Beale <timbeale@catalyst.net.nz>

Autobuild-User(master): Tim Beale <timbeale@samba.org>
Autobuild-Date(master): Thu Nov 15 08:15:32 CET 2018 on sn-devel-144

(cherry picked from commit c1dd6382e3211792e313f7d559b943f55c9cb0e1)

Autobuild-User(v4-9-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-9-test): Tue Nov 20 15:50:33 CET 2018 on sn-devel-144
2018-11-20 15:50:33 +01:00
Martin Schwenke
53104cbc32 ctdb-recovery: Ban a node that causes recovery failure
... instead of applying banning credits.

There have been a couple of cases where recovery repeatedly takes just
over 2 minutes to fail.  Therefore, banning credits expire between
failures and a continuously problematic node is never banned,
resulting in endless recoveries.  This is because it takes 2
applications of banning credits before a node is banned, which
generally involves 2 recovery failures.

The recovery helper makes up to 3 attempts to recover each database
during a single run.  If a node causes 3 failures then this is really
equivalent to 3 recovery failures in the model that existed before the
recovery helper added retries.  In that case the node would have been
banned after 2 failures.

So, instead of applying banning credits to the "most failing" node,
simply ban it directly from the recovery helper.

If multiple nodes are causing recovery failures then this can cause a
node to be banned more quickly than it might otherwise have been, even
pre-recovery-helper.  However, 90 seconds (i.e. 3 failures) is a long
time to be in recovery, so banning earlier seems like the best
approach.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13670

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Nov  5 06:52:33 CET 2018 on sn-devel-144

(cherry picked from commit 27df4f002a594dbb2f2a38afaccf3e22f19818e1)
2018-11-06 09:11:22 +01:00
Martin Schwenke
6c44382e5f ctdb-daemon: Fix valgrind hit in event code
==25741== Syscall param write(buf) points to uninitialised byte(s)
==25741==    at 0x4939291: write (write.c:27)
==25741==    by 0x4868285: sys_write (sys_rw.c:68)
==25741==    by 0x13915D: sock_queue_trigger (sock_io.c:316)
==25741==    by 0x4DE6478: tevent_common_invoke_immediate_handler (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741==    by 0x4DE64A2: tevent_common_loop_immediate (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741==    by 0x4DEBE5A: ??? (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741==    by 0x4DEA2D6: ??? (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741==    by 0x4DE57E3: _tevent_loop_once (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741==    by 0x15D1BA: ctdb_event_script_args (eventscript.c:821)
==25741==    by 0x13B437: ctdb_start_daemon (ctdb_daemon.c:1315)
==25741==    by 0x110642: main (ctdbd.c:393)
==25741==  Address 0x57888a4 is 100 bytes inside a block of size 144 alloc'd
==25741==    at 0x48357BF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25741==    by 0x4B9B7C0: talloc_named_const (in /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.14)
==25741==    by 0x15CCC6: eventd_client_write (eventscript.c:430)
==25741==    by 0x15CCC6: eventd_client_run (eventscript.c:556)
==25741==    by 0x15CCC6: ctdb_event_script_run (eventscript.c:649)
==25741==    by 0x15D198: ctdb_event_script_args (eventscript.c:812)
==25741==    by 0x13B437: ctdb_start_daemon (ctdb_daemon.c:1315)
==25741==    by 0x110642: main (ctdbd.c:393)
==25741==

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659

Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Oct 22 09:27:15 CEST 2018 on sn-devel-144

(cherry picked from commit fbea9d36996f248ba2b077f12ad16c199b853134)

Autobuild-User(v4-9-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-9-test): Thu Oct 25 11:05:48 CEST 2018 on sn-devel-144
2018-10-25 11:05:48 +02:00
Amitay Isaacs
c3179a1db2 ctdb-event: Check the return status of sock_daemon_set_startup_fd
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(cherry picked from commit a1909603808b994b7822b697494e39e8da4aaa66)
2018-10-25 07:59:09 +02:00
Amitay Isaacs
ec028efc66 ctdb-common: Set close-on-exec for startup fd
The startup_fd should not be propagated to the child processes created
from a daemon.  It should only be used in the daemon code to return the
status of the startup.  Another use of startup_fd is to notify the
parent if the daemon process has exited.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(cherry picked from commit 80549927bc1741a4b8af8b8e830de4d37fa0c4a8)
2018-10-25 07:59:09 +02:00
Martin Schwenke
1616cbd522 ctdb-daemon: Exit if eventd goes away
ctdbd enters a broken state if eventd goes away.  A clean shutdown is
not possible because that involves running events.  Restarting eventd
is possible but this might mask a serious problem and it is possible
that eventd might keep on disappearing.  Just exit.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit c9e1603a5d0c1a216439d4a2b0e7cdc05181e898)
2018-10-25 07:59:09 +02:00
Martin Schwenke
a835e64f2e ctdb-daemon: Return early when refusing to run an event script
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit a3d12252fa8e0a7e900b819dec30bdb9da458254)
2018-10-25 07:59:09 +02:00
Amitay Isaacs
07c97238bc ctdb-tests: Drop code for RECEIVE_RECORDS control
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(cherry picked from commit 83b3c5670d85c607c1cf1ab8cfc2c967d4d16721)
2018-10-10 15:51:57 +02:00
Amitay Isaacs
0147504ff7 ctdb-protocol: Drop marshalling code for RECEIVE_RECORDS control
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(cherry picked from commit 2f89bd96fb6c5e50cfc09604ceb6b96a94cb4f56)
2018-10-10 15:51:57 +02:00
Amitay Isaacs
a1a679563b ctdb-protocol: Mark RECEIVE_RECORDS control obsolete
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(cherry picked from commit 81dae71fa74bfd83a5701e4841b5a0a13cbe87a1)
2018-10-10 15:51:57 +02:00