1
0
mirror of https://github.com/samba-team/samba.git synced 2025-01-25 06:04:04 +03:00

1163 Commits

Author SHA1 Message Date
Martin Schwenke
cc74417341 eventscripts: Avoid using a temporary file in 62.cnfs
Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 81833052d7ee8f76b1e98376a0273448640cfa8e)
2013-08-22 17:00:20 +10:00
Martin Schwenke
bb974f150b scripts: Remove gdb_backtrace
This uses potentially insecure temporary files and is not referenced
anywhere else.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 4b914d7e217202f3d11a8e95f9f74bc17869475b)
2013-08-22 17:00:20 +10:00
Martin Schwenke
fec69034ee eventscripts: Become unhealthy faster on nfsd failure
Anecdotal evidence suggests that most nfsd RPC check failures are due
to cluster filesystem or storage problem.  Apparently these are rarely
helped by attempting to restart the NFS service because the restart
tends to hang.

Fail after 2 nfsd RPC check failures, instead of waiting for 6
failures.  Restart on every 10th failure to try to bring the node back
to good health.

Update unit tests to match.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit e9ef93f7b6dad59eabaa32124df81f3e74c651ef)
2013-08-14 16:10:30 +10:00
Martin Schwenke
e6ce2f55ef eventscripts: Improve message logged when a counter hits a limit
It should print the actual number of consecutive failures rather than
the limit.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit ff5f0d1e29af2b293e30cdc54bed03a644be7038)
2013-08-14 15:57:04 +10:00
Martin Schwenke
35d9631eda eventscripts: Print a message when waiting for TCP connections to be killed
This makes the gaps in the logs more obvious.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 11fbf4789d783dd0bac22754b374dd9ea4b03bad)
2013-08-14 15:57:04 +10:00
Martin Schwenke
b1f7337d2b eventscripts: New configuration variable $CTDB_RPCINFO_LOCALHOST
Passing "localhost" to the rpcinfo command causes overheads, like
reading /etc/services multiple times.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 1d61988af9e4fa3621a3e2d06a859bcb53df2d67)
2013-08-14 15:57:04 +10:00
Martin Schwenke
0ca046577f eventscripts: Add modulo (%) operator to ctdb_check_counter()
Also add it to the corresponding eventscript unit test infrastructure.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit f4ef83a256f59eeb00b9a5bc10c28347e1ad1031)
2013-08-14 15:57:03 +10:00
Martin Schwenke
bdbe37b24f eventscripts: Separate out RPC service restart code
While doing this:

* Explicitly assign RPC program and version information in
  _nfs_check_rpc_common().  This is more lines of code but is easier
  to read.

* Don't print the options when starting a service.  Trying to print it
  makes the code messy for little benefit.

  Update the eventscript unit testing code and a Ganesha test to
  reflect this.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit e8b531405665885196c95fe1608db33a255bf761)
2013-08-14 15:57:03 +10:00
Martin Schwenke
df539a66cb eventscripts: Remove support for RPC service 'q' and 's' restart flags
They're hard to maintain and provide very little benefit.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 1a1be43f8466d46913dcdfe6dcedb94316cd28ad)
2013-08-14 15:57:03 +10:00
Martin Schwenke
5459cdc8a6 eventscripts: When restarting the nfslock service only show output of start
That is, /dev/null the "stop" output.  This is consistent with the way
CTDB generally deals with the output when stopping a service.

It also makes updating the eventscript unit tests easier.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit c7332526b1b488abefeb4be78a7cd3f2f9abc451)
2013-08-14 15:57:03 +10:00
Martin Schwenke
98163e01a9 scripts: Do not run ctdb tool commands when debugging hung "init" event
CTDB daemon is not ready to accept clients in INIT runstate (init event).
CTDB daemon will start accepting connections in SETUP runstate (setup event)
and later.

Also, minor log formatting changes.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 81d7ce03b28d592a1337639e14d9ea141e20bfff)
2013-08-09 11:04:55 +10:00
Amitay Isaacs
f5ddb49e62 eventscripts: Use configured RECLOCK file instead of asking CTDB
On cluster where recovery lock file is not being used, asking CTDB daemon
is unnecessary overhead.  And if CTDB is using recovery file, then changing
configuration without restarting is *stupid*.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-Programmed-With: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 44eb86e6042adb6efe75d2a5528b82a0f21d496d)
2013-08-09 11:04:55 +10:00
Martin Schwenke
3c73949317 initscript: The wrapper script should export CTDB_SOCKET
This ensures that any invocation of the ctdb tool (within the wrapper)
gets the desired value.  This at least ensures that ctdbd will be
started.

If a non-standard value is set for CTDB_SOCKET then command-line users
will still need the variable in their environment.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 37ccc7c6cc43a80aaa92291aea7a438f4225488a)
2013-07-29 15:58:51 +10:00
Martin Schwenke
a8dd716146 eventscripts: kill_tcp_connections() should send connections to stdin
This avoids issuing multiple "ctdb killtcp" commands to terminate tcp
connections, one per connection.  This will considerably reduce the
time when there is a large number of tcp connections.  This also makes
it possible to avoid calling "ctdb killtcp" when there are no connections.

Add a couple of unit tests for killtcp and update eventscript unit
test infrastructure to support.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit a20d94717d2e4ab866d8a002cdf39c0669b74c6a)
2013-07-29 15:53:06 +10:00
Martin Schwenke
67b22b6e94 scripts: Run scriptstatus for hung event
The timeout information printed by ctdbd is less than useful because
it refers to the cumulative time taken by the eventscripts run so far.
Adding scriptstatus output indicates where time was actually spent.

Since there is now quite a bit of output, serialise the calls to this
script using flock.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 1b016b2dfc5d7d3f2a42ce4dfe569608e90eb714)
2013-07-29 14:02:13 +10:00
Martin Schwenke
1da757d91a eventscripts: A missing interface should cause monitoring to fail
A missing interface is at least as bad as an interface with a link
that is down so should have a similar effect.

This couldn't be done previously because orphaned interfaces used to
be listed for monitoring.  This was worked around in 10.interface in
commit 49b2d1bd9554461ed8edbfc21e777c0eca9e1443 and fixed in ctdbd in
commit cc1a3ae911d3fee8b87fda5de5ab6d9499d7510a.

If $CTDB_PARTIALLY_ONLINE_INTERFACES="yes" then monitoring won't
actually fail but the interface is still marked as down.

While we're touching this code, use "ip link" instead of "ip addr".
It is marginally cheaper but not enough for a separate patch.  ;-)

This effectively reverts d67955b42f7627be9dae995230c8fcbb8a948ec2.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 501f19b16fd6d67fbb754248868c38ee5bcf79ef)
2013-07-19 15:35:41 +10:00
Martin Schwenke
4b5c9c7991 eventscripts: Get list of configured interfaces using "ctdb ifaces"
This was previosuly changed because ctdbd didn't garbage collect
orphaned interfaces.  This was fixed in commit
cc1a3ae911d3fee8b87fda5de5ab6d9499d7510a.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit c6ab0f9405d5fa5b0b1693bc92e59da0d555a9d7)
2013-07-19 15:35:41 +10:00
Martin Schwenke
7610b6c009 scripts: ctdbd_wrapper logs a message to syslog if syslog is not being used
It can be very disconcerting when logging to syslog is expected but
nothing is being logged there.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 412bc0e20bef694d4e911dc9c984fd7716231f1f)
2013-07-11 15:18:06 +10:00
Martin Schwenke
e4d99cc899 packaging: Add systemd support
Based on an original patch by Sumit Bose <sbose@redhat.com>.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit e43a4b7b69a21c4cec2453dcac436b64bf5d7f06)
2013-07-10 18:14:33 +10:00
Martin Schwenke
adbee6ae4e initscript: Simpify initscript and control CTDB via new ctdbd_wrapper
Currently the initscript is very complex.  This makes it hard to read
and hard to add support for new init systems, such as systemd.

Create a wrapper called ctdbd_wrapper to be installed alongside ctdbd.
This is called by the initscript to start and stop ctdbd.  It does the
ctdbd option construct and waits until ctdbd is properly initialised
before it exits.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit e3abc7eebab5cceddc4ce7817890dd5db9be3450)
2013-07-10 15:19:27 +10:00
Amitay Isaacs
ae0afad8ee initscript: Export CTDB_DEBUG_LOCKS variable
Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit a415a1986900135f889efc25ecaf2761b1dae81a)
2013-07-10 14:33:18 +10:00
Amitay Isaacs
f46d0e783c scripts: Add an example debug_locks.sh script to debug locking issue
Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit c711ff4702c5f95b75e4bf030665fc2afffc2f9e)
2013-07-10 14:33:18 +10:00
Martin Schwenke
d6d1fb1f46 eventscripts: New configuration variable $CTDB_SKIP_GANESHA_NFSD_CHECK
This allows 60.ganesha to be unit tested, except for the core Ganesha
monitoring code.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit f606df4f2db754592e6d1a16c26e155cacb2beef)
2013-07-05 15:52:33 +10:00
Martin Schwenke
7f6169b207 eventscript: Move Ganesha nfsd monitoring to a function
Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit ceb5b2d37f7ab4894908ec26f3812b3bed991525)
2013-07-05 15:52:33 +10:00
Martin Schwenke
c3e83d4532 eventscripts: Drop RPC service version from nfs_check_rpc_service() calls
Support for this was removed in commit
77302dbfd85754e02559eccb2dd6c090db0b6b9f and I overlooked its use in
60.ganesha.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 520914e7ee1b879c1080e5857fda18ed5b973fd6)
2013-07-05 15:52:33 +10:00
Martin Schwenke
4e07c6c433 eventscripts: When replaying monitor status, don't log empty output
Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit ce04f1c107b4392ca955d9f29b93aaaae62439ce)
2013-07-05 15:52:33 +10:00
Martin Schwenke
01d879806b eventscripts: "setup" event doesn't need to wait for SETUP runstate
The "setup" event isn't called until ctdbd is in CTDB_RUNSTATE_SETUP
anyway...

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 9ea57af557028b1d2e5c560e7bcf4d014b9a8b1e)
2013-06-20 13:01:10 +10:00
Martin Schwenke
4eed91b54a eventscripts: 13.per_ip_routing should not try hard to find public_addresses
This essentially reverts d4621277240721e6d130a930b0100506b64467ea.
This was added for testing but the test code was actually broken.
CTDB itself will only process public IPs if $CTDB_PUBLIC_ADDRESSES is
set, so no code should try to be more flexible than that!

The test code has been fixed instead.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 3b11b27f3e22e99947bc2d6c49c4427bd7a0e332)
2013-06-20 13:01:10 +10:00
Martin Schwenke
6317285c4f scripts: Move TDB checking from initscript to "init" event
It makes sense to do this in the "init" event and make the initscript
less complicated.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 3bc93f312b8464fbfa2b2c44fffedc591fe5a3e0)
2013-06-20 13:01:10 +10:00
Martin Schwenke
961468146e scripts: Move dropping of all IPs from initscript to "init" event
It makes sense to do this in the "init" event and make the initscript
less complicated.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 0b77cceb49a30a181063adc7868d42d2851318e8)
2013-06-20 13:01:09 +10:00
Martin Schwenke
bee02e06e6 scripts: drop_ip() should use delete_ip_from_iface()
Otherwise secondary addresses that aren't owned by CTDB could be
dropped.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 5ffce65a1ad659b198ddf647622b899bdde45c72)
2013-06-20 13:01:09 +10:00
Martin Schwenke
a1eb516f0a scripts: drop_all_public_ips() now prints messages to stdout, not log
Change all callers to maintain current behaviour.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 0b67397ef5419c781a35916575151da7b7e7cc27)
2013-06-20 13:01:09 +10:00
Martin Schwenke
45878d4363 eventscripts: New configuration varable $CTDB_NFS_DUMP_STUCK_THREADS
If some nfsd threads are still alive after a shutdown during a restart
then this indicates the maximum number of threads for which a stack
trace should be dumped.  This can be useful for trying to determine
why nfsd is stuck.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 2503245db10d567af708a04edd3a3b488c24f401)
2013-06-14 15:15:06 +10:00
Martin Schwenke
f408caea2a eventscripts: Add new option $CTDB_MONITOR_NFS_THREAD_COUNT
Consider the following example:

1. There are 256 nfsd threads configured.
2. 200 threads are "stuck" in system calls, perhaps waiting for the
   underlying filesystem when an attempt is made to restart NFS.
3. 56 threads exit when NFS is stopped.
4. 56 new threads are started when NFS is started.
5. 200 "stuck" threads exit leaving only 56 threads running.

Setting this option to "yes" makes the 60.nfs monitor event look for
this situation and try to correct it.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 99b0d8b8ecc36dfc493775b9ebced54539c182d2)
2013-06-13 20:01:22 +10:00
Martin Schwenke
2e515f2306 eventscripts: Fix statd-callout update handling
60.nfs and 60.ganesha touch $statd_update_trigger every time they're
run.  This stops the statd-callout updates from ever being called.

Make this logic self-contained and move it to new function
nfs_statd_update() in the functions file.  Call this in 60.nfs and
60.ganesha with the appropriate update period as the only argument.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reported-by: Poornima Gupte <poornima.gupte@in.ibm.com>

(This used to be ctdb commit 1b5968f6be084590667f4f15ff3bef13ed9a2973)
2013-05-28 16:11:47 +10:00
Martin Schwenke
1eab9c898c eventscripts: Stop NAT gateway's delete_all() from polluting the log
Every time a node that wasn't the NAT gateway master gets reconfigured
something like this appears in the log:

  ctdbd: 11.natgw: Failed to del 10.0.1.139 on dev eth1

Since this usually fails it is better to mute the error than to have
it pollute the log.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 0ca7a98ffef50cbd06849cfbf65fb4a3d668b7bd)
2013-05-27 15:15:25 +10:00
Martin Schwenke
66019e3287 scripts: Provide mktemp function for platforms without mktemp command
This is needed for AIX and possibly others.

Also provide a cheaper mktemp function is needed in the run_tests
script.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit b2b572e9049c7138bd223226475bef8fe3e01f10)
2013-05-27 15:14:33 +10:00
Martin Schwenke
a989a299d1 eventscripts: 11.natgw should not call ctdb tool in "init" event
The current code calls "ctdb setnatgwstate ..." on every event.
However, calling the ctdb tool in the "init" event is not permitted.

Instead, update the capability when it is needed and at regular
intervals via the "monitor" event.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 39a43feae7c7de07ddaf2d6cb962f923d47d0c19)
2013-05-24 14:08:07 +10:00
Martin Schwenke
6d9667f01c ctdbd: Add new runstate CTDB_RUNSTATE_FIRST_RECOVERY
This adds more serialisation to the startup, ensuring that the
"startup" event runs after everything to do with the first recovery
(including the "recovered" event).

Given that it now takes longer to get to the "startup" state, the
initscript needs to wait until ctdbd gets to "first_recovery".

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit ed6814ff0a59ddbb1c1b3128b505380f60d7aeb7)
2013-05-24 14:08:07 +10:00
Martin Schwenke
b5ebff6931 tools/ctdb: "ctdb runstate" now accepts optional expected run state arguments
If one or more run states are specified then "ctdb runstate" succeeds
only if ctdbd is in one of those run states.

At the moment, if the "setup" event fails then the initscript succeeds
but ctdbd exits almost immediately.  This behaviour isn't very
friendly.

The initscript now waits until ctdbd is in "startup" or "running" run
state via the use of "ctdb runstate startup running", meaning that ctdbd
has successfully passed the "setup" event.

The "setup" event code in 00.ctdb now waits until ctdbd is in the
"setup" run state before proceeding via the use of "ctdb runstate setup".

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 4a2effcc455be67ff4a779a59ca81ba584312cd6)
2013-05-24 14:08:07 +10:00
Martin Schwenke
bb39f0a186 scripts: Rework notify.sh to use notify.d/ directory
This makes it easier to add notification handlers.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit d29e9a420b133088bf23a847c8d1dbce56c25eb0)
2013-05-23 16:18:23 +10:00
Martin Schwenke
51dbaecb54 eventscripts: Fix regression in _loadconfig()
fff88940f71058e4eefd65f50a6701389c005c17 introduced a regression.
Without $service_name set by default, the CTDB configuration is no
longer loaded when loadconfig() is called without any arguments.
That's bad.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit f1619a36c1beba11533052dc5728fa3adaa08870)
2013-05-22 14:24:21 +10:00
Martin Schwenke
ff9831f5b1 initscript: If CTDB doesn't become ready, print a message before killing
Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit e6b6b793f61556c21e8daf34abf89ee7b388ecfb)
2013-05-22 14:24:21 +10:00
Amitay Isaacs
84bcb95952 eventscripts: Do not use bashism for string comparison
Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit b0cae7d5a00ef3764bae187affc8e9a252f4b329)
2013-05-20 19:47:10 +10:00
Martin Schwenke
de84c1fd3c eventscripts: NFS RPC checks no longer support "knfsd"
No longer used, support removed from test infrastructure.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 0eb351ff4c7ee096de7c5e0a59561067091fa32e)
2013-05-07 12:55:09 +10:00
Martin Schwenke
434f9e8594 eventscripts: 60.nfs uses nfs_check_rpc_services() to check NFS RPC services
* New directory nfs-rpc-checks.d/ replaces hardcoded rules in 60.nfs

* Installation and packaging additions to handle nfs-rpc-checks.d/

* Unit test updates, including deleting 1 test that sanity checked
  test infrastructure

* Test infrastructure changes to use nfs-rpc-checks.d/

Note that this removes support for $CTDB_NFS_SKIP_KNFSD_ALIVE_CHECK in
60.nfs.  To get the equivalent behaviour, edit 20.nfsd.check and
remove/comment all lines.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 7e792d6768d9ca420ce3713cb122e63afd594b15)
2013-05-07 12:55:09 +10:00
Martin Schwenke
05b2edeec2 eventscripts: NFS RPC checks allows "nfsd" in addition to "knfsd"
Want nfs_check_rpc_services() to support filenames without the 'k'.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit d9775fcbd6e30eef8382bea68e2f9bad2309f2c1)
2013-05-06 20:40:58 +10:00
Martin Schwenke
c52183c055 eventscripts: New function nfs_check_rpc_services()
This is intended to replace nfs_check_rpc_service(), which builds
configuration into eventscripts.

nfs_check_rpc_services() uses a directory of configuration checks that
can be edited by an administrator.  The files have one limit check and
a set of actions per line.  The program name is extracted from the
file name.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 9bc8fbee6550ed2814fb35c70d57fab21ef1b8fd)
2013-05-06 20:40:58 +10:00
Martin Schwenke
167acd1cd5 eventscripts: nfs_check_rpc_action() should be _nfs_check_rpc_action()
Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 5a717fd495ba5a2bfd481d69f38b68fa4576716f)
2013-05-06 20:40:58 +10:00
Martin Schwenke
bdab9d1ea6 eventscripts: Factor out common code from nfs_check_rpc_service()
This creates new function _nfs_check_rpc_common().

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit cc3bb42e48bbdabd19187c231846b98589b4f4f3)
2013-05-06 20:40:58 +10:00