IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Rename CTDB_NFS_STATE_MNT to CTDB_NFS_SHARED_STATE_DIR. It doesn't
have to be a mount but can be any directory in a cluster filesystem.
CTDB_NFS_SHARED_STATE_DIR will soon be used in statd_callout_helper,
so the variable name might as well be better.
With this change, it will still only be used by nfs-ganesha-callout,
which isn't yet supported (i.e. it still lives in doc/examples). The
rest of the comments below refer to behaviour changes in that script.
CTDB_NFS_SHARED_STATE_DIR is now mandatory when GPFS is used. This is
much saner that choosing the first GPFS filesystem - if the state
directory changes then connection metadata can be lost.
Drop CTDB_NFS_STATE_FS_TYPE. The filesystem type is now determined
from CTDB_NFS_SHARED_STATE_DIR and it is now checked against supported
filesystems. This will catch the case when the filesystem for the
specified directory has not been mounted and the filesystem for the
mountpoint (e.g. ext4) is not a supported filesystem for shared state.
A side-effect is that the filesystem containing
CTDB_NFS_SHARED_STATE_DIR must be mounted when nfs-ganesha-callout is
first run.
While touching this file, my shfmt pre-commit hook wants to insert a
trailing ;; into a case statement. Let's sneak that in here too.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This allows CTDB to be configured to use "ss -K" to reset TCP
connections on "releaseip". This is only supported when the kernel is
configured with CONFIG_INET_DIAG_DESTROY enabled.
From the documentation:
ss -K has been supported in ss since iproute 4.5 in March 2016 and
in the Linux kernel since 4.4 in December 2015. However, the
required kernel configuration item CONFIG_INET_DIAG_DESTROY is
disabled by default. Although enabled in Debian kernels since
~2017 and in Ubuntu since at least 18.04,, this has only recently
been enabled in distributions such as RHEL. There seems to be no
way, including running ss -K, to determine if this is supported, so
use of this feature needs to be configurable. When available, it
should be the fastest, most reliable way of killing connections.
For RHEL and derivatives, this was enabled as follows:
* RHEL 8 via https://bugzilla.redhat.com/show_bug.cgi?id=2230213,
arriving in version kernel-4.18.0-513.5.1.el8_9
* RHEL 9 via https://issues.redhat.com/browse/RHEL-212, arriving in
kernel-5.14.0-360.el9
Enabling this option results in a small behaviour change because ss -K
always does a 2-way kill (i.e. it also sends a RST to the client).
Only a 1-way kill is done for SMB connections when ctdb_killtcp is
used - the reasons for this are shrouded in history and the 2-way kill
seems to work fine.
For the summary that is logged, when CTDB_KILLTCP_USE_SS_KILL is "yes"
or "try", always log the method used, even the fallback to
ctdb_killtcp. However, when set to "no", maintain the existing
output.
The decision to use -K rather than --kill is because short options are
trivial to implement in test stubs.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jerry Heyman <jheyman@ddn.com>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Nov 7 00:12:34 UTC 2024 on atb-devel-224
This will be used in a slightly different context in a subsequent
commit. In that case, the number of killed connections will be passed
instead of the total number of connections, so support this here via
different modes instead of churning later.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jerry Heyman <jheyman@ddn.com>
Currently TCP ports like NFS lock manager are not tracked. It is
easier to track all connections than to add a configuration system to
try to track specified ports, so do that.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jerry Heyman <jheyman@ddn.com>
Running ss to get current connections before running ctdb gettickles
means the ss output might be out of date when the 2 lists are
compared. Some tickles might have been added after ss was run by some
other means (e.g. SMB tickles, added internally) and they would be
deleted according to the stale ss output.
This isn't currently a problem because update_tickles() is currently
only called with port 2049, so all tickles are managed by this code.
That will change in a subsequent commit.
Changing the order means the reverse problem can occur, where
update_tickles() attempts to delete an already deleted tickle. That
may happen occasionally but is harmless because it doesn't result in
missing information. It (currently) just causes a message to be
logged at DEBUG level.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jerry Heyman <jheyman@ddn.com>
This should really be done for all connections to public IP addresses.
Leave the port number there for now - this is just the first step.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jerry Heyman <jheyman@ddn.com>
This option has been available since ~2018 and has been implemented in
the stub since then. I guess we didn't use it because CentOS 7?
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jerry Heyman <jheyman@ddn.com>
Since commit 224e99804e, square brackets
have been parsed by daemon and tool code, so drop the compatibility
code from here.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jerry Heyman <jheyman@ddn.com>
With an empty IP filter, all incoming connections to port 2049 will be
listed, not just those to public IP addresses. This causes error
messages like the following to be logged:
ctdb-eventd[...]: 60.nfs: Failed to add 1 tickles
since the connection being added seems to be for a random NFS mount
that doesn't use a public IP addresses.
This has been a problem for a long time (probably since commit
04fe9e2074 in 2015). It isn't currently
a huge deal because it only affects NFS connections. However, this
code will soon be used to track connections to public IP addresses on
all ports. This would result in a constant stream of log messages,
since there will always be some active connections.
The theory behind the fix is that if a node hosts no public IPs then
it should have no relevant connections and has no business changing
the list of registered tickles.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jerry Heyman <jheyman@ddn.com>
Massage a couple of lines manually so they're formatted sanely given
the new indentation. Re-run shfmt to ensure no further changes.
Best reviewed with "git show -w".
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jerry Heyman <jheyman@ddn.com>
That is, no longer set sysctl net.ipv4.conf.all.arp_filter=1 in
10.interface. Only do this in 13.per_ip_routing.
This effectively reverts commit
0ebd7beb4b by Ronnie Sahlberg from 2007.
I have discussed this with Ronnie. This setting was originally added
to force incoming traffic to the interface hosting each IP. This
would spread the load across multiple interfaces hosting the same
subnet. Without the setting, incoming traffic would go to the first
interface to answer an ARP request, so could be unbalanced if one
interface tended to answer more quickly.
However, networks are now faster and interface bonding/teaming works
well in Linux, so it is less likely that multiple interfaces will be
used in this way.
Also, problems are occurring in exactly the case this is meant to
help: when multiple interfaces host the same subnet.
The Linux kernel documentation for this option says:
arp_filter - BOOLEAN
- 1 - Allows you to have multiple network interfaces on the same
subnet, and have the ARPs for each interface be answered
based on whether or not the kernel would route a packet from
the ARP'd IP out that interface (therefore you must use source
based routing for this to work). In other words it allows control
of which cards (usually 1) will respond to an arp request.
- 0 - (default) The kernel can respond to arp requests with addresses
from other interfaces. This may seem wrong but it usually makes
sense, because it increases the chance of successful communication.
IP addresses are owned by the complete host on Linux, not by
particular interfaces. Only for more complex setups like load-
balancing, does this behaviour cause problems.
arp_filter for the interface will be enabled if at least one of
conf/{all,interface}/arp_filter is set to TRUE,
it will be disabled otherwise
Note the part for arp_filter=1 that says "you must use source based
routing for this to work". The problems are probably due to a lack of
source-based routing when this is only used with 10.interface. In
this case, outbound packets can come from a different
interface (corresponding to the first matching route), with a
different MAC address. There is clearly some infrastructure or packet
filtering out there that objects to such asymmetric packet flows.
So, drop this setting from 10.interface because it isn't working as
intended. Continue to enable it in 13.per_ip_routing, which exists to
set up the required source-based routing.
This change may affect balancing of packet flows when public IP
addresses can be hosted by multiple interfaces, but does not stop that
feature from working.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Anoop C S <anoopcs@samba.org>
Autobuild-User(master): Anoop C S <anoopcs@samba.org>
Autobuild-Date(master): Thu Oct 17 18:53:32 UTC 2024 on atb-devel-224
If an NFS service check is set to, say, unhealthy_after=2 then it will
always switch from the (default startup) unhealthy state to healthy,
even if there is a fatal problem. If all services/scripts appear OK
then the node will become healthy. When the counter hits the limit it
will return to unhealthy. This is misleading.
Instead, never use the counter at startup, until the service becomes
healthy. This stops services flapping unhealthy-healthy-unhealthy.
A side-effect is that a service that starts in a broken state will
never be restarted to try to fix the problem. This makes sense. The
counting and restarting really exist to deal with problems that might
occur under load. The first monitor events occur before public IPs
are hosted, so there can be no load. If a service doesn't start
reliably the first time then the admin probably wants to know about
it.
nfs_iterate_test() is updated to run an initial monitor event to mark
the services as healthy. This initialises the counter so it can be
used for the important part of the test. Passing the -i option avoids
running the extra monitor event, so the first iteration will be the
initial monitor event.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
This makes initial failure to retrieve statistics less likely to
result in a statistics change. To help with this, statistics
retrieval stderr now goes to the log - only stdout goes to the file.
This means that the test code for checking statistics changes needs to
be redone to actually run the statistics command and check. As with
rpcinfo output, this output needs to behave as deterministically in
the test code as it done in the event script.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Checking statistics is only really relevant to timeouts. That is, if
an rpcinfo times out it is worth checking if the service making
progress. If the RPC service is not registered then the statistics
don't need to be checked because they shouldn't be changing.
The 2 previously added tests added to check statistics progress now
behave identically and fail on all iterations. To support testing
with "timeouts", an optional TIMEOUT flag can now be added to the RPC
service passed to nfs_iterate_test(). 2 new tests are added to
exercise the new behaviour.
The 2 new "if" statements in nfs_iterate_test() could be combined.
However, a subsequent commit would split them and would be more
difficult to read.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
rpc.statd is single-threaded and runs its HA callout synchronously. If
it is too slow then latency accumulates and rpc.statd's backlog grows.
Running a pair of add-client/del-client events with the current code
averages ~0.030s in my test environment. This mean that 1000 clients
reclaiming locks after failover can easily cause 10s of latency. This
could cause rpc.statd to become unresponsive, resulting in a time out
for an rpcinfo-based health check of the status service.
Split the add-client/del-client events out to a standalone
statd_callout executable, written in C, to be used as the HA callout
for rpc.statd. All other functions move to statd_callout_helper.
Now, running a pair of add-client/del-client events in my test
environment averages only ~0.002s. This seems less likely to cause
latency problems.
The standalone statd_callout executable needs to read a configuration
file, which is generated by statd_callout_helper from the "startup"
event. It also needs access to a list of currently assigned public
IPs.
For backward compatibility, during installation a symlink is created
from $CTDB_BASE/statd-callout to the new statd_callout, which is
installed in the helper directory.
Testing this as part of the eventscript unit tests starts to become
even more of a hack than it used to be. However, the dependency on
stubs and the corresponding setup of fake state makes it hard to move
this elsewhere.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Tue Jun 25 04:24:57 UTC 2024 on atb-devel-224
When monitoring an RPC service, the rpcinfo command might time out
even though the service is making progress. In this case, it is just
slow, so counting the timeout as a failure and potentially restarting
the service will not help. The problem is determining if a service is
making progress.
Add a new NFS checks service_stats_command. This command is intended
to run a statistics command. The output is naively compared using
cmp(1). If the output changes then rpcinfo failures are converted to
successes.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Exports may be contained in an include file rather than the top-level
ganesha.conf.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
This simplifies and removes a bad hack. Also, in my test environment,
it also drops the average time take to run an add-client/del-client
pair from ~0.055s to ~0.030s.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Take advantage of new function find_statd_sm_dir() when clearing the
local system statd state directory, so it uses the correct directory
when running on a non-RH distro.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
For add-client and del-client, statd-callout is called by rpc.statd,
which runs as rpcuser, statd or some other non-root system user. This
means that add-client and del-client can't write in the statd-callout
state directory if it is only writable by root. rpc.statd must be
able to write to its own local system statd state directory, so find
this directory and use it as a reference to set the ownership of
CTDB's statd-callout state directory.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
rpc.statd runs statd-callout as a non-root user, which is currently
hacked around using some sudo logic that fails to work in some
contexts (e.g. in a container).
Use $CTDB_MY_PUBLIC_IPS_CACHE to access the node's currently assigned
public IPs, for add-client/del-client. This avoids connecting to
ctdbd when called from rpc.statd.
Also, use $CTDB_MY_PUBLIC_IPS_CACHE in other places where it makes
sense.
Connections to ctdbd are still made in the "notify" action, but this
is always run as root.
In the test code, set the PNN after public addresses setup so that the
cache of assigned IPs correctly initialised.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
This is way more complicated than I would like but, as per the
comment, this is due to complexities in the way public IPs work. The
main consumer will be statd-callout, which will then be able to run as
a non-root user.
Also generate the cache file in test code, whenever the PNN is set.
However, this can cause "ctdb ip" to generate a fake IP layout before
public IPs are setup. So, have the "ctdb ip" stub generate the IP
layout every time it is run to avoid it being stale.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Add new variables statd_callout_state_dir and statd_callout_queue_dir
- the latter is for files queued by add-client/del-client.
Use $statd_callout_queue_dir to avoid a global cd to the queue
directory near the top of the script.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
All of the other uses of ctdb.tdb are in statd-callout.
New variable statd_callout_db makes it easy to change the database
name in future, perhaps even allowing it to be configurable.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Tweak some lines to avoid overflowing 80 columns.
Best viewed with "git show -w".
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Using xargs with sort -u to de-duplicate this list was my idea and
causes a couple of things to go wrong. The use of xargs causes
double-quotes to be lost. The resulting $public_ifaces value also
contains newlines. The newlines could be removed with an additional
xargs at the end of the pipeline... but that would add an extra level
of quote stripping.
I have unsuccessfully tried to find an alternative, but still elegant,
command pipeline that de-duplicates the list, while maintaining
quoting.
So, just drop the de-duplication.
This might make interface_ifindex_exists_with_options() slightly less
efficient. However, that function walks the whole list, only
terminating early when a match is found on both interface and options,
so at least it will be correct.
Include an extra testcase.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Apr 18 09:08:34 UTC 2024 on atb-devel-224
get_all_interfaces() functions gets all names for all public interfaces.
However name is misleading. Thus renamed it to get_public_ifaces() and
moved it under functions.
Signed-off-by: Vinit Agnihotri <vagnihotri@ddn.com>
Reviewed-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
These were generated by 06.nfs.script.
Signed-off-by: Vinit Agnihotri <vagnihotri@ddn.com>
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Wed Mar 6 07:09:06 UTC 2024 on atb-devel-224
If someone wants to enable the witness service
samba-dcerpcd needs to be started as standalone service
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15577
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>
We can easily monitor if the service is running at all,
that better than no monitoring at all...
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15577
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>
/etc/os-release is quite universal. It can be found on most Linux
distros and on FreeBSD.
Attempt to use /etc/os-release to detect Red Hat, SUSE and Debian
based distros. If /etc/os-release exists but distro is unknown then
$ID is printed as the detected distro, which will probably result in
sub-optimal behaviour, but when tracing it will at least indicate that
a new distro needs to be handled.
The only way to handle missing /etc/os-release is to set
CTDB_INIT_STYLE - see ctdb.sysconfig(5) for details.
The event script unit tests are updated to use /etc/os-release so
the new logic is exercised.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Oct 30 09:19:11 UTC 2023 on atb-devel-224
This effectively provides simple testing for the threshold-based
approach.
Add new script option CTDB_VSFTPD_MONITOR_THRESHOLDS.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Oct 3 04:53:38 UTC 2023 on atb-devel-224
This can be used for simple failure counting, without restarts, as
used in the 40.vsftpd event script. That case will subsequently be
converted and this functionality can also be used elsewhere.
Add documentation to ctdb-script.options(5) to allow parameters that
use this to be more easily described.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Uninitialised counters are treated as 0, but still produce an error.
The redirect to stderr needs to come before the redirect for a missing
counter file.
The seemingly saner alternative of moving it outside the subshell
works when dash is /bin/sh (e.g. on Debian) but does not work when
bash is /bin/sh (e.g. on Fedora).
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Logging in statd-callout tests is currently useless. This will
provide a way of seeing errors in those tests.
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
