IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit c446579fc442955ecc74f5566eaa0635c3171498)
A monitor event following a "ctdb delip" might reconfigure services.
If the monitor event is cancelled then a service might be stopped but
not yet restarted and this could result in the subsequent monitor
events failing.
This obviously needs to be fixed in CTDB itself. This will happen by
making "ctdb reloadips" the supported way of reconfiguring IPs.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 618ea3660e36e7bd92b686e1ca8728cf63c3c068)
This uses potentially insecure temporary files and is not referenced
anywhere else.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 4b914d7e217202f3d11a8e95f9f74bc17869475b)
The useful cases are either CTDB_CURRENT_NODE, in which case
ctdb_get_pnn() does the job, or a PNN, which is... ummm... a PNN! :-)
This works because parse_nodestring() validates PNNs.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 7b3f7eea2465efb099a2faf3e42174bc97b13a16)
* ipreallocate is cluster-wide so should not be auto-all
* enablescript, disablescript, getreclock, setreclock, natgwlist can
all be auto-all without issues
* xpnn, ipiface a local-only so don't work with -n, so might as well
not be auto-all
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 123a4677528cb46bee1c6dad8a5162eba9880bc1)
This is an internal structure. It was moved into ctdb_private.h a
long time ago to allow unit testing. Unit test compilation was
changed shortly afterwards to make this unnecessary.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit db57261d7dc264e161659a8c547f44fbd9e88eeb)
This ensures that the result of eventscripts is updated and callback is
processed before debugging hung script. So "ctdb scriptstatus" output
will be useful from debug hung script.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-Programmed-With: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 4ed2efb838d2ac97746666f614ebef5fdf3cdd5e)
This should avoid memory bloat when a request bounces between nodes.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 7677fb263f06a97398e2c546e32273fb96edca69)
If we process all the data available in a socket buffer, CTDB can stay busy
processing lots of packets via immediate event mechanism in tevent. After
processing an immediate event, tevent returns without epoll_wait. So as long
as there are immediate events, tevent will never poll other FDs. CTDB will
report this as "Event handling took xx seconds" warning. This is misleading
since CTDB is very busy processing packets, but never gets to the point of
polling FDs.
The improvement in socket handling made it worse when handling traverse
control. There were lots of packets filled in the socket buffer quickly and
CTDB stayed busy processing those packets and not polling other FDs and timer
events. This can lead to controls timing out and in worse case other nodes
marking busy node as disconnected.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 92939c1178d04116d842708bc2d6a9c2950e36cc)
This reverts commit 5e9b1a7e24d058ff88aaa0563db36a804e866fa9.
This is not the best approach. Allowing queue buffer size to grow
indefinitely causes large number of CTDB packets to be queued up very
quickly which when processed via immediate events will block CTDB from
processing events from other FDs. If there are immediate events queued
up, tevent will never process any of the FDs till all immediate events
are processed.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit d8b094e804efc53fae9f44c6ef961b7b5797d290)
This reverts commit 035c0d981bde8c0eee8b3f24ba8e2dc817e5b504.
This is a premature optimization. Record can bounce between nodes
very quickly if it is a contended record. There is no need to hold a
record on a node unnecessarily. In case record contention becomes bad,
enabling sticky records on a database is a better idea.
Conflicts:
include/ctdb_private.h
server/ctdb_tunables.c
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit ac417b0003f0116f116834ad2ac51482d25cfa0d)
Empty record with rsn=0 should not be written on any other node other than
dmaster. This is however not true for persistent databases. So currently
apply the check only for volatile databases.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit df83ae7a047dab4803e0d94b1c11df48ae17ca96)
This has the side effect of making these commands more resilient to
control timeouts.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 0fe79662e20e347d9e1cb12a42cd356e33572402)
Now we will only have one set of bugs. :-)
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 444521c852749558f39dc6131acce9e47eefd489)
Having other functions call control_ipreallocate() suggests that the
it might look at the argv/argv arguments that are passed. This is not
the case. Change the callers so they call the new ipreallocate()
function instead.
Broadcast CTDB_SRVID_TAKEOVER_RUN to all connected nodes. Inactive
nodes will ignore it. This is safe since we only want 1 reply. If we
didn't get a response, we don't actually care if there's no active
recovery master - just fire, wait, retry, ...
Ignore some failures on the basis that they might be transient, so it
is probably worth retrying.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 4bf0b1c9d21986eecb7682f935bd6154c65533cc)
This has already been stored at connect time and can't fail.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit d8eb2e7fdd7645719370dad4f2faa5c3fffa8249)
This is caused by corruption of a record header such that the records
on two nodes point to each other as dmaster. This makes a request for
that record bounce between nodes endlessly.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit f0853013655ac3bedf1b793de128fb679c6db6c6)
This is caused by corruption of a record header such that the records
on two nodes point to each other as dmaster. This makes a request for
that record bounce between nodes endlessly.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit a610bc351f0754c84c78c27d02f9a695e60c5b0f)
Anecdotal evidence suggests that most nfsd RPC check failures are due
to cluster filesystem or storage problem. Apparently these are rarely
helped by attempting to restart the NFS service because the restart
tends to hang.
Fail after 2 nfsd RPC check failures, instead of waiting for 6
failures. Restart on every 10th failure to try to bring the node back
to good health.
Update unit tests to match.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit e9ef93f7b6dad59eabaa32124df81f3e74c651ef)
The current 3 second timeout is arbitrary and users trip over it
sometimes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit b49c4f39666d5b1596213bf41bcdc47ed3c327ae)
It should print the actual number of consecutive failures rather than
the limit.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit ff5f0d1e29af2b293e30cdc54bed03a644be7038)
This makes the gaps in the logs more obvious.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 11fbf4789d783dd0bac22754b374dd9ea4b03bad)
Passing "localhost" to the rpcinfo command causes overheads, like
reading /etc/services multiple times.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 1d61988af9e4fa3621a3e2d06a859bcb53df2d67)
Also add it to the corresponding eventscript unit test infrastructure.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit f4ef83a256f59eeb00b9a5bc10c28347e1ad1031)
While doing this:
* Explicitly assign RPC program and version information in
_nfs_check_rpc_common(). This is more lines of code but is easier
to read.
* Don't print the options when starting a service. Trying to print it
makes the code messy for little benefit.
Update the eventscript unit testing code and a Ganesha test to
reflect this.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit e8b531405665885196c95fe1608db33a255bf761)
That is, output that goes through background_with_logging() just gets
"&" prepended to each line. This is cleaner than having the tests
grovel through logs.
Update some 49.winbind/50.samba tests to deal with this.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 3ba933d806106d12bc48b83b22d0f314d9d1e5e5)
They're hard to maintain and provide very little benefit.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 1a1be43f8466d46913dcdfe6dcedb94316cd28ad)
That is, /dev/null the "stop" output. This is consistent with the way
CTDB generally deals with the output when stopping a service.
It also makes updating the eventscript unit tests easier.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit c7332526b1b488abefeb4be78a7cd3f2f9abc451)
This should minimise the chances of a control timing out.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 63be516673c5d9c0d543617bf1bb8bca919956a8)
Update the missing IP test to wait until restarts are complete.
Otherwise a service restart can collide with the following monitor
event and cause chaos.
Also, do not disable 10.interface until it matters. Disabling it too
early can cause even more chaos if something goes wrong with the
monitor step.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 4e3bd06916bd3adac213fb18c7c2a24854b02d45)
When creating missing databases either locally or remotely, recovery
master calls ctdb_ctrl_createdb(). Recovery master always passes 0
for tdb_flags. For volatile databases, if TDB_INCOMPATIBLE_HASH is not
specified, then they will be attached without using jenkins hash causing
database corruption.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 2fc6b6403707a292d134140fc0b9145b454992c5)
This reverts commit 10a057d8e15c8c18e540598a940d3548c731b0b4.
This approach would not work when creating local databases since currently
there is no control to receive TDB flags for remote databases.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit ca61eb776ab862bd269e45ee0f9f96e7e1e0e001)
Currently queue buffer size is realloc'd every time we need to extend the
buffer. Small increments can cause memory fragmentation. Instead always
extend buffer in multiples of 4K. This should reduce multiple talloc_realloc
calls when there are lots of packets in the socket buffer.
Also, if queue buffer has grown larger than 64K, throw away the buffer once
all the requests in the queue have been processed. That way queue does not
hold on to large buffers.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 5e9b1a7e24d058ff88aaa0563db36a804e866fa9)
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-Programmed-With: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 867afb247bd8cc86c8d738f051a44cc534cafacf)
Instead of logging from ctdb_request_call(), log the message from
ctdb_make_record_sticky(). That way if the record is already sticky, the
message is not repeated unnecessarily.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 44a64d1c388bfe3c3388b191edfaedecfb7bb831)
CTDB daemon is not ready to accept clients in INIT runstate (init event).
CTDB daemon will start accepting connections in SETUP runstate (setup event)
and later.
Also, minor log formatting changes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 81d7ce03b28d592a1337639e14d9ea141e20bfff)
Currently if the debug hung script takes long time to finish, the subsequent
monitor event can collide with the previous event which is not yet finished.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 9e99e0eb072e2b845914ee3896acbc66b96138d7)
On cluster where recovery lock file is not being used, asking CTDB daemon
is unnecessary overhead. And if CTDB is using recovery file, then changing
configuration without restarting is *stupid*.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-Programmed-With: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 44eb86e6042adb6efe75d2a5528b82a0f21d496d)
If there are multiple lock helper processes waiting for the same record, then
it will cause a thundering herd when that record has been unlocked. So avoid
scheduling lock contexts for the same record. This will also mean that
multiple requests will get queued up behind the same lock context and can be
processed quickly once the lock has been obtained.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit ebecc3a18f1cb397a78b56eaf8f752dd5495bcc9)
So that ctdb_lock_schedule() can call this function without requiring extra
prototype declaration.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 68af5405acc123b5a90decd2123e2a02961a8fcf)
When creating missing databases either locally or remotely, make sure
to use the correct tdb flags from other nodes. Without this, volatile
databases can get attached without TDB_INCOMPATIBLE_HASH flag.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 10a057d8e15c8c18e540598a940d3548c731b0b4)
Currently nodemap used by recovery master is the one obtained from the local
node. This information may have been updated while processing main loop.
Before comparing node flags on all the nodes, create up-to-date node flags
information based on the information received from all the nodes.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit fcf77dec5af973a0e32f3999bc012053a6f47a96)
Instead of maintaining another structure, add an element as place holder for
marshall buffer of hot keys. This avoids duplication of the structure.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit e73b2e12adc9db1dedb48d32bba3a8406a80f4cd)
The structure cannot be removed without adding support for marshalling keys
for hot records.
This reverts commit 26a4653df594d351ca0dc1bd5f5b2f5b0eb0a9a5.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 023ca2e84f5ed064a288526b9c2bc7e06674dd81)
This simplifies building since we don't use any of the Samba
extensions.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 57aa2dffea60abd73a95233f8b761cc676adebb6)
This ensures that any invocation of the ctdb tool (within the wrapper)
gets the desired value. This at least ensures that ctdbd will be
started.
If a non-standard value is set for CTDB_SOCKET then command-line users
will still need the variable in their environment.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 37ccc7c6cc43a80aaa92291aea7a438f4225488a)
Commit f73a4b1495830bcdd094a93732a89dd53b3c2f78 added a safety check
to ensure that CTDB never kills unrelated processes. However, client
processes are unrelated.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 782814288bb560099ee44b607bf35f3eddf37f82)
This avoids issuing multiple "ctdb killtcp" commands to terminate tcp
connections, one per connection. This will considerably reduce the
time when there is a large number of tcp connections. This also makes
it possible to avoid calling "ctdb killtcp" when there are no connections.
Add a couple of unit tests for killtcp and update eventscript unit
test infrastructure to support.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit a20d94717d2e4ab866d8a002cdf39c0669b74c6a)
This will allows eventscripts to send information about multiple tcp
connections to a single "ctdb killtcp" command, saving the overhead of
setting up a client connection per tcp connection.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit af5aa369c266430fe912df0c26116b68bac3572e)
Regardless of whether a summary is being printed!
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit a69e03a5e4671e998d45b4fef8611a421bbdb3e1)
Currently the fail callback is called once per (takeip/releaseip) control
failure. This is overkill and can get a node banned much too quickly.
Instead, keep track of control failures per node and only call fail
callback once per failed node.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit bf4a7c1ad87e0e848296d15d63eb8cd901ca5335)
The timeout information printed by ctdbd is less than useful because
it refers to the cumulative time taken by the eventscripts run so far.
Adding scriptstatus output indicates where time was actually spent.
Since there is now quite a bit of output, serialise the calls to this
script using flock.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 1b016b2dfc5d7d3f2a42ce4dfe569608e90eb714)
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit e0f3fa1020e13b84bdd672538168d148f1847d57)
Refactor the NFS test setup/cleanup code into new common functions.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 29e98017221326bdc9b1c4f7c05b3b495c1de29b)
Change the command from "true" to "hostname" since the former won't
produce any output when used in combination with "onnode -p". This
could just be changed to "echo" but the hostname might actually be
useful.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit ae3c03d80264e997b7da9f3279d7810e18b8a1df)
Register print_exit_message() earlier so that it covers most of the
early exits.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 90d792cf28d6a823141e4c417b6978f02a9cf596)
Don't blindly remove the socket.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 3dd5b925dcf0e9a5b877638e471c5ecf36b46c58)
A missing interface is at least as bad as an interface with a link
that is down so should have a similar effect.
This couldn't be done previously because orphaned interfaces used to
be listed for monitoring. This was worked around in 10.interface in
commit 49b2d1bd9554461ed8edbfc21e777c0eca9e1443 and fixed in ctdbd in
commit cc1a3ae911d3fee8b87fda5de5ab6d9499d7510a.
If $CTDB_PARTIALLY_ONLINE_INTERFACES="yes" then monitoring won't
actually fail but the interface is still marked as down.
While we're touching this code, use "ip link" instead of "ip addr".
It is marginally cheaper but not enough for a separate patch. ;-)
This effectively reverts d67955b42f7627be9dae995230c8fcbb8a948ec2.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 501f19b16fd6d67fbb754248868c38ee5bcf79ef)
This was previosuly changed because ctdbd didn't garbage collect
orphaned interfaces. This was fixed in commit
cc1a3ae911d3fee8b87fda5de5ab6d9499d7510a.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit c6ab0f9405d5fa5b0b1693bc92e59da0d555a9d7)
Commit 8076773a9924dcf8aff16f7d96b2b9ac383ecc28 introduced a potential
regression because a node may not have completed the "recovered" event
(so might still be in CTDB_RUNSTATE_FIRST_RECOVERY) when another node
becomes healthy.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 57ef5d3827ea3417a32703e259a53ce6fd10ac45)
It can be very disconcerting when logging to syslog is expected but
nothing is being logged there.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 412bc0e20bef694d4e911dc9c984fd7716231f1f)
Because of commit a779d83a6213e2ba
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit a4afe7af9c9391048d6f80135bbd5e15367770c7)
Commit 9119a568c2b4601318f7751f537dca2f92a7230b attempted to fix this.
However, this was wrong because old_flags and new_flags were confused.
The latter has since been fixed in commit
7eb2f89979360b6cc98ca9b17c48310277fa89fc so this can now be fixed
properly.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 40f2825d6e818dc8c745b6385a545969dfb45fbc)
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-programmed-with: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 14141b02b61d2783b750ee5b30f9520253e88f09)
Based on an original patch by Sumit Bose <sbose@redhat.com>.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit e43a4b7b69a21c4cec2453dcac436b64bf5d7f06)
The "‘tevent_loop_allow_nesting’ is deprecated" warnings will be
around for a while and are annoying.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 30a0040fbb7c4d97d107f0e55c600295c2603a68)
This reverts the last part of 788cdbddbc902a5b076d23473450065b551d274d
- the rest of this has been implicitly reverted via tevent syncs.
This is just leftover noise.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit b6bbfb4c464c39e322830cbbebcc51c225508584)
Currently the initscript is very complex. This makes it hard to read
and hard to add support for new init systems, such as systemd.
Create a wrapper called ctdbd_wrapper to be installed alongside ctdbd.
This is called by the initscript to start and stop ctdbd. It does the
ctdbd option construct and waits until ctdbd is properly initialised
before it exits.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit e3abc7eebab5cceddc4ce7817890dd5db9be3450)
This helps distinguish processes in process list in top, perf, etc.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 2493f57ce268d6fe7e4c40a87852c347fd60d29e)
Currently CTDB daemon reads records from a child process and then sends them to
srcnode via TRAVERSE_DATA control. This ties up main CTDB daemon and also
requires an extra copy of the record in the CTDB daemon. Instead send records
directly from traverse child process.
The control from child process still goes via local CTDB daemon as there
is no infrastructure currently to open a TCP socket to the srcnode.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 1a74192aa7d51ed99553e7292860027f06b6ef37)
So that traverse child process can directly send the TRAVERSE_DATA control to
the srcnode without first sending it to local node.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit faabce1b99fb3de9ff03bf54d303e7656538fee3)
The code for deadlock detection and killing smbd process causing deadlock
has been removed and replaced with external debug script.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 2211cd94bea266547d3e6f167d3160a6b23bec88)
Use an external script to parse /proc/locks and log useful debugging
information about locks rather than doing that in C code.
To use this feature, add configuration variable to /etc/sysconfig/ctdb:
CTDB_DEBUG_LOCKS=/etc/ctdb/debug_locks.sh
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 2bfb8499366d530f16515b08928056bbda40f781)
0 < 1 ms
1 < 10 ms
2 < 100 ms
3 < 1 s
4 < 2 s
5 < 4 s
6 < 8 s
7 < 16 s
8 < 32 s
9 < 64 s
10 >= 64 s
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 6fc36a7036933237d09151a0baf4d8ccd2bc2c99)
Send the ctdb_db_statistics directly instead of first copying it to
duplicate ctdb_db_statistics_wire structure. This simplifies the
implementation of the control to get database statistics.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 26a4653df594d351ca0dc1bd5f5b2f5b0eb0a9a5)
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-Programmed-With: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 41182623891d74a7e9e9c453183411a161201e67)
This allows 60.ganesha to be unit tested, except for the core Ganesha
monitoring code.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit f606df4f2db754592e6d1a16c26e155cacb2beef)
Support for this was removed in commit
77302dbfd85754e02559eccb2dd6c090db0b6b9f and I overlooked its use in
60.ganesha.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 520914e7ee1b879c1080e5857fda18ed5b973fd6)
At the moment this is silent and it can be confusing to see IPs just
disappear.
Also, this message:
Been in recovery mode for too long. Dropping all IPS
can cause anxiety when all IPs should already have been dropped.
Adding a comforting message saying that 0 IPs were dropped relieves
such anxiety. :-)
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 4d0f26b306fc465d551d340b0e7dce4412eae3fd)
* Add a variable to the loop to make the code more readable and have
it generally fit into 80 columns.
* Improve comments.
* Improve log messages.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 0a292fa8939a1343e44cadaa8ed9f3c0f18ca82f)
The log messages in verify_remote_ip_allocation() are confusing
because they don't include the PNN of the problem node, because it is
not known in this function.
Add the PNN of the node being verified as a function argument and then
shuffle the log messages around to make them clearer.
Also fold 3 nested if statements into just one.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit f0942fa01cd422133fc9398f56b4855397d7bc86)
When the recovery master notices a node in recovery mode it starts the
recovery process, it doesn't restart it.
Update documentation to match.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 298c4d2c3b4ea3d900c91f5a0a5aca2952a13d61)
This is slightly easier to read because it all fits on 1 line.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 035bf3eecf99337c84d4ad16cdbf297b1fa037db)
The "init" event only really fails in the scripts, which should log
something useful on failure. Therefore, a core dump isn't terribly
useful and sometimes attracts unwanted attention.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 3af2d833b63af9931792106db71797f3692669a8)
This is like ctdb_fatal() but exits cleanly without dumping core or
generating a backtrace.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit c0a9456692c88a7a5542cd893d8f326524d3f94e)
At the moment there (at least) are 2 bugs that cause rogue IPs:
* A race where release_ip_callback() runs after a "subsequent" take IP
has completed. The IP is back on an interface but we unset
vnn->iface in the callback.
* A "releaseip" eventscript times out. We ignore the timeout and call
it success, deleting the VNN even if the IP is still hosted.
We could decide not to ignore the timeout and ban the node, but
killing TCP connections can take a long time and that might result
in a lot of manning. We probably won't reinstate banning on
"releaseip" until killing TCP connections has been optimised.
In both cases, a rogue IP can be avoided by leaving vnn->iface set and
simply failing the control.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit c5797f2942e83da24df548ea07196fbbac0eab20)
Previous code changes work around a potential problems but do not
provide useful information when the a problem occurs.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit f1f1b0c24b9b6cd24b83a4e4da16e179287ec6ac)
This fixes the floating point error if num_locks = 0.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 16afe36de52561a62372c14b567683dc898369d5)
This fixes the segmentation error if any of the test code fails to
connect to CTDB daemon.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit d48eecd748830598f4f080952f2bf05d6f92738c)
The result has been sent before the child keeps waiting for parent
ctdbd process.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 9aa13bcedd83d463c871e3cf1f3a65da3cd83992)
This reverts d09570c70551aa40390ce9ceffe7bc234e1afafe.
... hoping the segv has been found in last 6 years. :-)
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 9b529189f8456fad7868fc154ae27a6fd87e93b3)
To build CTDB RPMs with system installed libraries, use following command:
./packaging/RPM/makerpms.sh \
--with system_talloc \
--with system_tdb \
--with system_tevent
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit bb54f3924ff19cd089b0a166fe8368db162ad709)
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-Programmed-With: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 53d34eb2f9e5434dea4e7182b6af566a3a96a368)
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-Programmed-With: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 6fe584d05543eebd24abd19bab502dc4da04e921)
It's bundled in ctdb-tests package.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 7e53fbf92b6dd5211d918ea0e23126b7dfa50c42)
There is no point in banning the node if init or shutdown event times
out since it's going to quit anyway.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit ef1c4e99ca66e7a990bc557f34abb624c315e6ba)
When we trigger an election because the recmaster considers itself inactive,
update our local nodemap with the recmaster's flags before calling
force_election(). This way, we don't send the inactive node freeze commands
(e.g.) that may fail and then lead to ourselves getting banned.
The theory is that this should help avoiding banning loops.
Signed-off-by: Michael Adam <obnox@samba.org>
(This used to be ctdb commit 932360992b08a5483d90c0590218ba0fd756119e)
Can not continue with recovery or monitoring cluster.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 14399de1dd0bd8dabf1f48b1457e3ccb37589d8a)
Since we have nodemap information, there is no need to hardcode the
limit of 20.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-Programmed-With: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit aea12dce83ef385e9fb3bc03ac7ace0874a0e3fe)
If a node gets banned first, then it should not ban other nodes.
This code was moved up in main_loop to avoid waiting for nodemap
from other nodes (commit 83b0261f2cb453195b86f547d360400103a8b795).
To prevent a banned node from banning other nodes, we need to first get
nodemap information from local node, so trying to ban other nodes can
fail if we are already banned.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit ae1693905036ecdbc4594fde1f12500faae4a554)
Since there is an early exit if a node is stopped or banned, we can wait till
the node becomes active to start initial election.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 593a17678fbd3109e118154b034d43b852659518)
Since we do an early return if a node is stopped or banned, move update
capabilities code below the early return and just before we check the
capabilities of current recovery master.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 93bcb6617e1024f810533e12390a572f51703ca0)
If a node is stopped or banned, it will cause early return from the
main_loop, so this check is redundent. The election will called by an
active node.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 815ddd3341b7e9db39e05a3a3fcd9a1420f053bc)
A stopped or banned node cannot do anything useful. So do not participate
in any cluster activity and do not cause any unnecessary network traffic.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 2396981c4bcf30530aeb7f4395093cc202105b50)
If the current node is banned or stopped, then it should not assign banning
credits to other nodes since the current node will not have up-to-date flags
of other nodes.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 38304f88e0c634e97d4687c25adef975f71537b8)
If the banned pnn is not the local node, the function returns early.
So no need for additional check.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 297d93cecc3c0655e72ecac38508e113bdbeab9c)
When this function is called, we are already committed to banning
and there is no point in failing this function. In case, freezing of
databases fails, it will be fixed from recovery daemon.
(This used to be ctdb commit bb178338658b4ae32382a1f62f7c21cee1d4878f)
If this function fails due to memory errors, there is no way to recover.
The best course of action is to abort.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 46efe7a886f8c4c56f19536adc98a73c22db906a)
ctdb_start_freeze() is called from ctdb_control_freeze() which fixes the
priority if it's 0 and return error if it's invalid. Other callers of
ctdb_start_freeze() are internal to CTDB. So if priority is invalid in
ctdb_start_freeze(), definitely something is seriously wrong.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 87716e8f504d659515d3dbcf93badbf106873bc8)
This ensures that whenever databases are frozen either via sending
control or by calling ctdb_start_freeze(), the action is logged.
Since ctdb_control_freeze() calls ctdb_start_freeze(), move logging of
message in early return condition if databases are already frozen.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 478e24bceda3fedfba54ccb48faa115df726b819)
This was broken by commit a9a1156ea4e10483a4bf4265b8e9203f0af033aa.
Instead of a SRVID_SET_NODE_FLAGS message to recovery daemon, a control
was sent to the local daemon which in turn informed the recovery daemon.
And while doing this change old flags were sent via CONTROL_MODIFY_FLAGS.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 7eb2f89979360b6cc98ca9b17c48310277fa89fc)
At the moment there is no easy way to force a recovery when attempting
to reproduce certain classes of bugs. This option is added without
documentation because it is dangerous until the bugs are fixed! :-)
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 4f87925a287f612a6ab3b5da1a387a31c7bea28f)
The example notification script is now much more useful.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 4ba7c73eeab98296c9168e0b0fed1f6bb9f32733)
Also check that we're not in recovery mode.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit b7aaa28b3a6a2de923417f3d143f8d516447711e)
No need for 2 recoveries after a restart.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit b953524185632d7f96a76d8f3bbed7ac1d143d40)
The runstate can't be set to SHUTDOWN twice, so the current naive code
causes a panic on the 2nd shutdown. This regression was introduced in
commit 8076773a9924dcf8aff16f7d96b2b9ac383ecc28.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit f1b7ca8dc3f34a59c7b3e55748f974ac9ed8f458)
The "setup" event isn't called until ctdbd is in CTDB_RUNSTATE_SETUP
anyway...
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 9ea57af557028b1d2e5c560e7bcf4d014b9a8b1e)
These test dropping of IPs and TDB checking.
New stubs for date, tdbdump, tdbtool.
Enhance ip stub to handle "ip addr show to ..."
Tweak some infrastructure.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit aabf0bf41cb8ec344f06b69492fb6c2a27f9e900)
This essentially reverts d4621277240721e6d130a930b0100506b64467ea.
This was added for testing but the test code was actually broken.
CTDB itself will only process public IPs if $CTDB_PUBLIC_ADDRESSES is
set, so no code should try to be more flexible than that!
The test code has been fixed instead.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 3b11b27f3e22e99947bc2d6c49c4427bd7a0e332)
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit c3e7a6e10d486ba0dbafdf110db540675b2317bc)
Messages are lost until it is really up because syslogd_is_started is
set too early. Adding a pipe to do the notification allows the parent
to wait and only set syslogd_is_started when the logging daemon is
actually ready.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit f3dd2eec200d6eeada2ea19cd7e76f1edfad6167)
It makes sense to do this in the "init" event and make the initscript
less complicated.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 3bc93f312b8464fbfa2b2c44fffedc591fe5a3e0)
It makes sense to do this in the "init" event and make the initscript
less complicated.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 0b77cceb49a30a181063adc7868d42d2851318e8)
Otherwise secondary addresses that aren't owned by CTDB could be
dropped.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 5ffce65a1ad659b198ddf647622b899bdde45c72)
Change all callers to maintain current behaviour.
Signed-off-by: Martin Schwenke <martin@meltin.net>
(This used to be ctdb commit 0b67397ef5419c781a35916575151da7b7e7cc27)
It should run before:
* the transport is started;
* databases are attached; and
* processing configuration files (e.g. nodes, public_addresses).
Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 0a0c8543f167e11b75a622513367b083e42cbd3f)
This avoids premature exits from "ctdb stop" and "ctdb continue" due to
intermittent control (e.g. getpnn, getnodemap) timeouts.
This needs a proper fix to distinguish between timeout and failure
conditions and take appropriate action.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit c48583fd238496a81ddc46a21892f0b49559036a)
