1
0
mirror of https://github.com/samba-team/samba.git synced 2025-01-26 10:04:02 +03:00

525 Commits

Author SHA1 Message Date
Martin Schwenke
143ec9e4c2 ctdb-scripts: Drop file removal involving $ctdb_managed_dir
This directory has been under $CTDB_VARDIR for a long time, so is
already removed.  The command is also potentially dangerous if
$ctdb_managed_dir is not set.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Jose A. Rivera <jarrpa@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
2015-09-15 08:56:20 +02:00
Martin Schwenke
8c2948eba5 ctdb-scripts: Drop 62.cnfs eventscript
To get a similar effect just do something like this:

  mmaddcallback ctdb-disable-on-quorumLoss \
    --command /usr/bin/ctdb \
    --event quorumLoss --parms "disable"

  mmaddcallback ctdb-enable-on-quorumReached \
    --command /usr/bin/ctdb \
    --event quorumReached --parms "enable"

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Jose A. Rivera <jarrpa@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
2015-09-15 08:56:20 +02:00
Martin Schwenke
9d49dfac5f ctdb-scripts: New internal variable CTDB_NFS_CALLOUT_STATE_DIR
60.nfs sets this to a directory where the NFS callout can store state.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Sep 14 08:36:34 CEST 2015 on sn-devel-104
2015-09-14 08:36:34 +02:00
Martin Schwenke
4164d7bf31 ctdb-scripts: Add default filesystem usage warnings
Always check filesystem usage for the database directories.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Sat Aug 29 20:08:48 CEST 2015 on sn-devel-104
2015-08-29 20:08:47 +02:00
Martin Schwenke
0f28ccf87a ctdb-scripts: Add default system memory usage warnings
CTDB should warn by default if too much system memory or swap is used.

The tests have also been tweaked.  In particular, the filesystem-only
tests need to initialise the memory information to avoid errors where
meminfo isn't set.

Document the defaults, warning against disabling them.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
2c601f1895 ctdb-scripts: Enable system monitoring eventscript by default
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
b18e4ae0c9 ctdb-scripts: Throttle system resource monitoring warnings
They are only printed when the percentage usage changes.  This should
stop the logs from being filled with warnings.

Add a test for the throttling.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
e6b5163bc1 ctdb-scripts: Don't shutdown CTDB when memory monitoring fails
Marking the node unhealthy should cause Samba processes to close,
possible freeing a stack of memory.  If not, then it is somebody
else's problem.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
b6a0e4b856 ctdb-scripts: New consistent system memory and swap monitoring
New variables CTDB_MONITOR_MEMORY_USAGE and CTDB_MONITOR_SWAP_USAGE.
Both take a pair of <warn_threshold>:<unhealthy_threshold> where each
theshold is specified as a percentage.

This adds a callout to check_thresholds() that is run when the
unhealthy threshold is reached.

Add some combination tests.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
02fa6c3d10 ctdb-scripts: Factor out new function check_thresholds()
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
b7b6e25b3e ctdb-scripts: Memory monitoring uses thresholds expressed as percentages
CTDB_MONITOR_FREE_MEMORY and CTDB_MONITOR_FREE_MEMORY_WARN are now
percentages that specify thresholds of acceptable memory usage.

Memory/swap usage in tests also specified as percentages.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
bd2845d7eb ctdb-scripts: Use MemAvailable if it is in /proc/meminfo
Otherwise calculate, as before.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
99b8ef5121 ctdb-scripts: Only use /proc/meminfo for memory checks, not "free"
No need to use 2 different sources of information for similar checks.
Also, output of free has been changed, whereas /proc/meminfo is a
kernel API, which will not change.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
ab58c7abd9 ctdb-scripts: Move system memory checking to 05.system
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
fa1050690b ctdb-scripts: New configuration variable CTDB_MONITOR_FILESYSTEM_USAGE
This allows both errors (i.e. unhealthy) and warnings for different
thresholds.  It replaces CTDB_CHECK_FS_USE.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
8f713c87c1 ctdb-scripts: Don't fail monitoring if sanity checks fail
Just log some warnings.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
6b4a46e574 ctdb-scripts: Move filesystem monitoring into a function, clean it up
Drop obvious comments.  Use die() for less lines of code.  Use a case
statement to avoid forking unnecessary processes for each filesystem
being checked.  Drop parentheses around percentages in messages.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
47f7d1b1c8 ctdb-scripts: Rename 40.fs_use to 05.system
Will put all the system monitoring in here, simplifying 00.ctdb.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-29 17:06:25 +02:00
Martin Schwenke
7d04778c82 ctdb-scripts: Improve error handling for 50.samba testparm failure
Also add tests.  Update testparm stub to fake error and timeout.  Add
timeout stub.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-08-07 05:33:29 +02:00
Martin Schwenke
dd88c2ab8c ctdb-scripts: Support RPC checks for tcp6 and udp6
This adds new configuration variable CTDB_RPCINFO_LOCALHOST6.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-07-21 07:54:09 +02:00
Martin Schwenke
fa6f22d7ae ctdb-scripts: Add registration for CTDB_NFS_CALLOUT operations
This is an optimisation to avoid forking the callout for operations
that are not implemented.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-07-14 09:57:18 +02:00
Martin Schwenke
a02bdb97f9 ctdb-scripts: Move NFS support functions to 60.nfs
Now that there is only a single NFS eventscript, other eventscripts no
longer need to load all of this.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-07-14 09:57:18 +02:00
Martin Schwenke
6586651508 ctdb-scripts: Remove 60.ganesha, replace with callout for 60.nfs
This isn't a straightforward move of code from 60.ganesha to the
callout.  Simplifications have been made to allow better
interoperation with the new NFS checking logic.

The following configuration variables have been removed:

  CTDB_GANESHA_REC_SUBDIR

    Edit NFS ganesha callout to change this location

  CTDB_NFS_SERVER_MODE, NFS_SERVER_MODE

    Use CTDB_NFS_CALLOUT instead

  CTDB_NFS_SKIP_KNFSD_ALIVE_CHECK, CTDB_SKIP_GANESHA_NFSD_CHECK

    Disable the corresponding .check file instead

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-07-14 09:57:18 +02:00
Martin Schwenke
9c87d1dd29 ctdb-scripts: Parameterise 60.nfs with $CTDB_NFS_CALLOUT
The goal is to have a single NFS eventscript.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-07-14 09:57:18 +02:00
Martin Schwenke
74428e5c14 ctdb-scripts: Switch NFS checks to new style
Note that the 60.ganesha RPC checks need to be identical to those in
the nfs-checks.d/ directory.  This is because the NFS unit test
infrastructure checks output against what should be produced by the
checks in nfs-checks.d/.  This is a minor issue, since one of the aims
of this work is to remove the need for a separate 60.ganesha.

In most cases configuration variable CTDB_NFS_DUMP_STUCK_THREADS is
now ignored.  This is now handled by passing the desired number of
threads to the command specified in the service_debug_cmd variable in
a .check file.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-07-14 09:57:18 +02:00
Martin Schwenke
6f3ee81d17 ctdb-scripts: NFS RPC checks should be simple and consistent
Change status, nlockmgr, mountd, rquotad to be unhealthy after 6
rpcinfo check failures and do a verbose restart after every 2
failures.  Change 60.ganesha for consistency, since 60.ganesha tests
are broken and depend on the consistency.

Apart from the consistency aspect, the check infrastructure will soon
be simplified so that it only allows the equivalent of "unhealthy" and
"verbose restart:b" actions.

Update tests to have a corresponding numbers of iterations.  Run 1
extra iteration in most tests to check there are no unexpected
behaviour changes after the designated number of iterations completes.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-07-14 09:57:17 +02:00
Martin Schwenke
bc71251433 ctdb-scripts: Support monitoring of interestingly named VLANs on bonds
VLAN interfaces on bonds with a name other than <iface>.<id>@<iface>
are not currently supported.  That is, where the VLAN name isn't based
on the underlying bond name.  Such VLAN interfaces can be created with
the "ip link" command, as opposed to the "vconfig" command, or by
renaming a VLAN interface.

This is improved by determining the underlying interface name for a
VLAN from the output of "ip link".

No serious attempt is made to support VLANs with '@' in their name,
although this seems to be legal.  Why would you do that?

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-07-14 09:57:16 +02:00
Martin Schwenke
0a65013b9d ctdb-scripts: Use an "if" statement instead of "&&"
If statd-callout is unwanted, so is removed, then this code fails.
Change to an "if" so that it succeeds as intended.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-07-09 06:23:21 +02:00
Martin Schwenke
27674c413d ctdb-scripts: Create the directory containing the recovery lock
This will handle the most obvious cases.  It won't handle the case
where the directory is missing and the recovery lock location is
updated at run-time.  However, this is a good improvement.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-07-01 04:18:28 +02:00
Jose A. Rivera
7266c68856 ctdb: Change use of 'which' to 'type' in scripts.
While 'which' is a very common tool, on many distros it is not a requirement
that it be installed. 'type' is a shell built-in specified by the Open Group,
and is found in shells like bash, dash, and ksh across multiple OSes.

Signed-off-by: Jose A. Rivera <jarrpa@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>

Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Fri Jun  5 20:39:47 CEST 2015 on sn-devel-104
2015-06-05 20:39:47 +02:00
Martin Schwenke
3b25face87 ctdb-scripts: New eventscript 10.external
This is an alternative to 10.interface and is installed as disabled by
default.  It should only be used with DisableIPFailover=yes and when
IP failover is being handled externally.  In this mode CTDB can be
informed of public IP address movements using "ctdb moveip".

During the "startup" event, this eventscript currently finds any
public IP addresses configured in $CTDB_PUBLIC_ADDRESSES and tells
CTDB which node they are on using "ctdb moveip".  This allows CTDB to
send ARPs and tickle-ACKs.

Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-05-13 06:42:13 +02:00
Martin Schwenke
0d0512cb65 ctdb-scripts: Drop all public IP addresses from 10.interface
00.ctdb should not know about public IP addresses.

Move related tests to operate on 10.interface.

Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-05-13 06:42:13 +02:00
Martin Schwenke
c927ec928c ctdb-scripts: Drop update of public address configuration from config.tdb
This isn't used or documented anywhere.

2 differing points of view:

* This is a very good idea but it should probably be generalised to
  cover more configuration items.  This would end up like the Samba
  registry configuration and would use a tool to support setting
  configuration values.

* If people really want to update configuration while a node is down
  then they should fix the configuration before bringing up that node.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-05-10 03:22:13 +02:00
Amitay Isaacs
f6af2d96c2 ctdb-scripts: Run tdb checker under timeout command
If tdb database file size grows beyond 4GB, tdbtool/tdbdump can hang
indefinitely.  This will prevent CTDB from starting up.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
2015-04-30 06:45:26 +02:00
Martin Schwenke
0ae57588eb ctdb-scripts: Simplify a command pipeline
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-04-27 03:32:11 +02:00
Martin Schwenke
1092f9755f ctdb-scripts: Replace uses of "ctdb pnn" with ctdb_get_pnn()
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-04-27 03:32:10 +02:00
Martin Schwenke
09b5e4978a ctdb-scripts: Changed uses of "ctdb xpnn" to ctdb_get_pnn()
"ctdb xpnn" does not work when sysctl net.ipv4.ip_nonlocal_bind=1,
since it determines the node by attempting to bind to each addres in
the nodes file.  The solution is to not use "ctdb xpnn".  After the
initial call, ctdb_get_pnn() will be more efficient that "ctdb xpnn".

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-04-27 03:32:10 +02:00
Amitay Isaacs
4f82ef4b38 ctdb-scripts: Simplify 00.ctdb event script
Avoid extra which commands.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2015-03-20 13:49:26 +01:00
Martin Schwenke
38279d7ec1 ctdb-eventscripts: Make 11.natgw stateful
IP addresses and routes are only changed if either the NAT gateway
configuration or the NAT gateway master node has changed.  If running
"ip monitor" this will minimise the amount of noise seen.  It should
also be more lightweight at the expense of managing a couple of state
files.

Add a test to check that configuration changes behave correctly.
Tweak the static route result generation code so that the required
output is sorted.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-03-16 06:41:06 +01:00
Martin Schwenke
500c6e194b ctdb-scripts: Change statd-callout to be more scalable
Updating ctdb.tdb on each add-client, del-client and each delete
during notify was too ambitious.  Persistent transactions do not
perform well enough to do this.

Revert to having add-client and del-client create touch files.  Each
monitor event calls "statd-callout update" to convert touch files into
ctdb.tdb records.

Update testcases to do the "update" and add an extra test.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-03-04 10:42:27 +01:00
Martin Schwenke
dc32f11b87 ctdb-scripts: Improve messages about invalid tunables during "setup"
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Wed Feb 18 08:03:33 CET 2015 on sn-devel-104
2015-02-18 08:03:33 +01:00
Martin Schwenke
39686f4505 ctdb-scripts: Fix tunable setup code by making it shell-agnostic
All tunables set in configuration are currently set to 0 on system
where /bin/sh is dash (and perhaps other non-bash shells).  dash puts
single quotes around all values in the output of the "set" builtin
command, whereas bash only puts them around values when something
needs to be quoted.  Tunables always have a simple integer value so
dash will quote them and bash won't.  The setup code currently passes
the raw value, including any quotes to "ctdb setvar ...".  This
command does no error checking on the input, so "'1'" is converted to
0.

Change the code so that the value is determined from the shell
variable and is independent of the "set" output.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-02-18 05:34:06 +01:00
Martin Schwenke
9b67c1fa37 ctdb-scripts: Error message, comment and whitespace cleanups
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-01-28 06:01:09 +01:00
Martin Schwenke
1a5414b6d2 ctdb-scripts: iSCSI eventscript should fail when PNN can't be determined
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-01-28 06:01:08 +01:00
Martin Schwenke
d1bd26e5eb ctdb-scripts: Make 70.iscsi IPv6-aware
Block iSCSI port for families of all address the node is configured to
host.

Could just unconditional add blocking using ip6tables instead.
However, this would produce errors when no IPv6 public addresses are
configured and ip6tables is not installed.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2015-01-28 06:01:08 +01:00
Martin Schwenke
6471541d6d ctdb-scripts: Make 10.interface IPv6-safe
Add checking to "releaseip" and "updateip" to ensure that the given IP
address is really on the given interface with the given netmask.  If
reality doesn't match the given arguments then believe reality.

Use new function iptables_wrapper() instead of calling iptables()
directly.

Use new function flush_route_cache() instead of doing IPv4-specific
/proc magic.

Remove setting of otherwise unused variable "failed".

Fix a test for which the error message has changed.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2014-12-05 21:02:40 +01:00
Martin Schwenke
4940f191d3 ctdb-scripts: Update eventscripts to use ctdb -X instead of ctdb -Y
Also update associated eventscript unit tests and ctdb stub.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2014-12-05 21:02:39 +01:00
Martin Schwenke
4cd5be87da ctdb-scripts: Try to restart statd after every 10 failures
Also add and update tests for statd stack dumps.  Update the existing
60.ganesha statd test to do more iterations.  Duplicate the result as
a new test for 60.nfs.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2014-11-18 04:17:10 +01:00
Martin Schwenke
bc59e508d3 ctdb-eventscripts: Remove special case for virtio_net
The current check is incorrect in 2 ways:

* Commit be71a84565e9e7532a77c175732b764d1f42c1cd contained a thinko
  that stops virtio_net interfaces from simply being marked up

* virtio_net interfaces can actually be down

virtio_net has supported ethtool since Linux 2.6.29, so just remove
the special case.  This means that testing CTDB on very old virtual
machines is not supported.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Thu Jul 31 13:08:47 CEST 2014 on sn-devel-104
2014-07-31 13:08:47 +02:00
Martin Schwenke
7c2c6748e3 ctdb-eventscripts: Remove unused argument to natgw_ensure_master()
This was used to limit damage in the "recovered" event.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Jul 29 10:03:16 CEST 2014 on sn-devel-104
2014-07-29 10:03:16 +02:00