1
0
mirror of https://github.com/samba-team/samba.git synced 2024-12-22 13:34:15 +03:00
Commit Graph

9058 Commits

Author SHA1 Message Date
Andreas Schneider
6d7d82938b ctdb:client: Fix code spelling
Best reviewed with: `git show --word-diff`

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Martin Schwenke <mschwenke@ddn.com>
2023-03-24 07:01:31 +00:00
Martin Schwenke
238056e5aa ctdb-scripts: Avoid using testparm to process its own output
When testparm processes the output of "testparm -v" (which includes
default values) it appears to do global checks (or some other sort of
initialisation logic) for all specified values.  This includes a DNS
lookup for the node's hostname, as a side-effect of a libldap
ldap_set_option() call when processing "ldap debug level".  If DNS
servers are down then this can induce timeouts, possibly resulting in
monitor timeouts.

Avoid this by using sed to extract configuration values from the
testparm cache file.

This is already shown to work when retrieving share paths, where
testparm is basically used as cat.  Update the sed pattern to avoid
matching empty values on the right-hand side of the equals ('=') -
this avoids the default empty path value (and "smb ports" never has an
empty value).

Corresponding test changes:

* 50.samba.monitor.111.sh no longer expects a failure from being
  unable to set smb ports, since testparm is no longer used in that
  code path.

* smb ports needs to be set in fake smb.conf so it is in the default
  output and can be extracted using sed.

* Although testparm --parameter-name is no longer used in
  50.samba.script, update the stub implementation (in case it is ever
  used again) to extract from fake smb.conf, since "smb ports" is now
  set there.  The change from $parameter to $param allows a long line
  to stay below 80 columns.

Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Tue Feb 14 08:43:53 UTC 2023 on atb-devel-224
2023-02-14 08:43:53 +00:00
Martin Schwenke
9a04ca1e1c ctdb-scripts: Do not replace commas with spaces in "smb ports" list
The list changed back to space-separated in commit
93448f4be9, so simplify the code a
little.

Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
2023-02-14 07:44:30 +00:00
Martin Schwenke
029dddfb79 ctdb-scripts: Reformat script with "shfmt -w -p -i 0 -fn"
Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
2023-02-14 07:44:30 +00:00
Rob van der Linde
851127f5c9 Python: remove pydoctor
Removes:

* waf pydoctor
* waf wafdocs
* make pydoctor

There is no "make wafdocs" it only appears to be in wscript.

The reasoning being is these are broken and appear to not have been run for some time.

Signed-off-by: Rob van der Linde <rob@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu Feb  2 21:15:54 UTC 2023 on atb-devel-224
2023-02-02 21:15:54 +00:00
Michael Tokarev
96154a26fe spelling fixes for 4.18 (errror implemenation proces Controler)
One of changes is somewhat interesting, it is "tfork waiter proces"
process title in tfork.c. I wonder why no one noticed this before.
There's another similar process title in there, "tfork waiter process(%d)".
Hopefully no one does grep for "proces$" (and there's no reason to).

Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Rowland Penny <rpenny@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu Jan 26 20:46:11 UTC 2023 on atb-devel-224
2023-01-26 20:46:11 +00:00
Volker Lendecke
35ee3e0231 ctdb: Fix the build on FreeBSD
"basename" is define in libgen.h included from system/dir.h

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2023-01-18 11:49:38 +00:00
Volker Lendecke
688be0177b ctdb: Fix a use-after-free in run_proc
If you happen to talloc_free(run_ctx) before all the tevent_req's
hanging off it, you run into the following:

==495196== Invalid read of size 8
==495196==    at 0x10D757: run_proc_state_destructor (run_proc.c:413)
==495196==    by 0x488F736: _tc_free_internal (talloc.c:1158)
==495196==    by 0x488FBDD: _talloc_free_internal (talloc.c:1248)
==495196==    by 0x4890F41: _talloc_free (talloc.c:1792)
==495196==    by 0x48538B1: tevent_req_received (tevent_req.c:293)
==495196==    by 0x4853429: tevent_req_destructor (tevent_req.c:129)
==495196==    by 0x488F736: _tc_free_internal (talloc.c:1158)
==495196==    by 0x4890AF6: _tc_free_children_internal (talloc.c:1669)
==495196==    by 0x488F967: _tc_free_internal (talloc.c:1184)
==495196==    by 0x488FBDD: _talloc_free_internal (talloc.c:1248)
==495196==    by 0x4890F41: _talloc_free (talloc.c:1792)
==495196==    by 0x10DE62: main (run_proc_test.c:86)
==495196==  Address 0x55b77f8 is 152 bytes inside a block of size 160 free'd
==495196==    at 0x48399AB: free (vg_replace_malloc.c:538)
==495196==    by 0x488FB25: _tc_free_internal (talloc.c:1222)
==495196==    by 0x488FBDD: _talloc_free_internal (talloc.c:1248)
==495196==    by 0x4890F41: _talloc_free (talloc.c:1792)
==495196==    by 0x10D315: run_proc_context_destructor (run_proc.c:329)
==495196==    by 0x488F736: _tc_free_internal (talloc.c:1158)
==495196==    by 0x488FBDD: _talloc_free_internal (talloc.c:1248)
==495196==    by 0x4890F41: _talloc_free (talloc.c:1792)
==495196==    by 0x10DE62: main (run_proc_test.c:86)
==495196==  Block was alloc'd at
==495196==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==495196==    by 0x488EAD9: __talloc_with_prefix (talloc.c:783)
==495196==    by 0x488EC73: __talloc (talloc.c:825)
==495196==    by 0x488F0FC: _talloc_named_const (talloc.c:982)
==495196==    by 0x48925B1: _talloc_zero (talloc.c:2421)
==495196==    by 0x10C8F2: proc_new (run_proc.c:61)
==495196==    by 0x10D4C9: run_proc_send (run_proc.c:381)
==495196==    by 0x10DDF6: main (run_proc_test.c:79)

This happens because run_proc_context_destructor() directly does a
talloc_free() on the struct proc_context's and not the enclosing
tevent_req's. run_proc_kill() makes sure that we don't follow
proc->req, but it forgets the "state->proc", which is free()'ed, but
later dereferenced in run_proc_state_destructor().

This is an attempt at a quick fix, I believe we should convert
run_proc_context->plist into an array of tevent_req's, so that we can
properly TALLOC_FREE() according to the "natural" hierarchy and not
just pull an arbitrary thread out of that heap.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Thu Oct  6 15:10:20 UTC 2022 on sn-devel-184
2022-10-06 15:10:20 +00:00
Martin Schwenke
d9dda4b7af ctdb-scripts: Add debugging variable CTDB_KILLTCP_DEBUGLEVEL
To debug ctdb_killtcp failures, add

  CTDB_KILLTCP_DEBUGLEVEL=DEBUG

to script.options.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Sep 20 11:42:16 UTC 2022 on sn-devel-184
2022-09-20 11:42:16 +00:00
Martin Schwenke
9f7d69a05b ctdb-common: Support IB in pcap-based capture
Add simple support for IPoIB via DLT_LINUX_SLL and DLT_LINUX_SLL2.
This seems to work, even when an IB interface is specified.

If this is later found to be insufficient, support for DLT_IPOIB can
be implemented.  See https://www.tcpdump.org/linktypes.html for a
starting point.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
e5541a7e02 ctdb-common: Support "any" interface for pcap-based capture
This uses Linux cooked capture link-layer headers.  See:

  https://www.tcpdump.org/linktypes/LINKTYPE_LINUX_SLL.html
  https://www.tcpdump.org/linktypes/LINKTYPE_LINUX_SLL2.html

The header type needs to be checked to ensure the protocol
type (i.e. ether type, for the protocols we might be interested in) is
meaningful.  The size of the header needs to be known so it can be
skipped, allowing the IP header to be found and parsed.

It would be possible to define support for DLT_LINUX_SLL2 if it is
missing.  However, if a platform is missing support in the header file
then it is almost certainly missing in the run-time library too.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
3bf20300ac ctdb-common: Add packet type detection to pcap-based capture
The current code will almost certainly generate ENOMSG for
non-ethernet packets, even for ethernet packets when the "any"
interface is used.

pcap_datalink(3PCAP) says:

  Do NOT assume that the packets for a given capture or ``savefile``
  will have any given link-layer header type, such as DLT_EN10MB for
  Ethernet.  For example, the "any" device on Linux will have a
  link-layer header type of DLT_LINUX_SLL or DLT_LINUX_SLL2 even if
  all devices on the sys‐ tem at the time the "any" device is opened
  have some other data link type, such as DLT_EN10MB for Ethernet.

So, pcap_datalink() must be used.

Detect pcap packet types that are supported (currently only ethernet)
in the open code. There is no use continuing if the read code can't
parse packets.  The pattern of using switch statements supports future
addition of other packet types.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
5dd964aa02 ctdb-tools: Improve/add debug
In particular, knowing the reason fetching the packet fails can help
with debugging unsupported protocols in the pcap code.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
33a80c1d63 ctdb-common: Improve/add debug
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
075414dc05 ctdb-common: Use pcap_get_selectable_fd()
This is preferred because it will fail for devices that do not support
epoll_wait() and similar.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
40380a8042 ctdb-common: Stop a pcap-related crash on error
errbuf can't be NULL.  Might as well use it.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
8b54587b1a ctdb-common: Fix a warning in the pcap code
[173/416] Compiling ctdb/common/system_socket.c
../../common/system_socket.c: In function ‘ctdb_sys_read_tcp_packet’:
../../common/system_socket.c:1016:15: error: cast discards ‘const’ qualifier from pointer target type [-Werror=cast-qual]
 1016 |         eth = (struct ether_header *)buffer;
      |               ^

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
ad445abebd ctdb-common: Do not use raw socket when ENABLE_PCAP is defined
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
c522f4f604 ctdb-common: Move a misplaced comment
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
d1543d5c78 ctdb-build: Add --enable-pcap configure option
This forces the use pcap for packet capture on Linux.

It appears that using a raw socket for capture does not work with
infiniband - pcap support for that to come.

Don't (yet?) change the default capture method to pcap.  On some
platforms (e.g. my personal Intel NUC, running Debian testing), pcap
is much less reliable than the raw socket.  However, pcap seems fine
on most other platforms.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
a83e9ca696 ctdb-build: Use pcap-config when available
The build currently fails on AIX, which can't find the pcap headers
because they're installed in a non-standard place.  However, there is
a pcap-config script available.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-20 10:43:37 +00:00
Martin Schwenke
4f5b4bd9df ctdb-tests: Reformat remaining test stubs with "shfmt -w -p -i 0 -fn"
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Sep 16 04:35:09 UTC 2022 on sn-devel-184
2022-09-16 04:35:09 +00:00
Martin Schwenke
0e388a1994 ctdb-tests: Include eventscript stub commands in shellcheck test
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-16 03:36:32 +00:00
Martin Schwenke
4ee0abaece ctdb-tests: Avoid shellcheck warnings in remaining test stubs
A small amount of effort...

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-16 03:36:32 +00:00
Martin Schwenke
a31fb7e5ab ctdb-scripts: Simplify determination of real interface
This can now be made trivial.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-16 03:36:32 +00:00
Martin Schwenke
5abaec4992 ctdb-tests: Implement "ip -brief link show" in ip stub
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-16 03:36:32 +00:00
Martin Schwenke
ef921bdbdb ctdb-tests: Avoid ShellCheck warnings
Although this is a test stub, it is complicated enough to encourage
ShellCheck cleanliness.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-16 03:36:32 +00:00
Martin Schwenke
67e0ca5e01 ctdb-tests: Reformat script with "shfmt -w -p -i 0 -fn"
As per current Samba convention.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-16 03:36:32 +00:00
Martin Schwenke
517f09eb6f ctdb-scripts: Drop assumption that there are VLANs with no '@'
VLAN configuration on Linux often uses a convention of naming a VLAN
on <iface> with VLAN ID <tag> as <iface>.<tag>.  To be able to monitor
the underlying interface, the original 10.interface code naively
simply stripped off the '.' and everything after (i.e. ".*", as a glob
pattern).

Some users do not use the above convention.  A VLAN can be named
without including the underlying interface, but still with a
tag (e.g. vlan<tag> - the word "vlan" following by the tag) or, more
generally, perhaps without a tag (e.g. <vlan> - an arbitrary name).
The ip(8) command lists a VLAN as <vlan>@<iface>.  The underlying
interface can be found by stripping everything up to and including an
'@' (i.e. "*@").

Commit bc71251433 added support for
stripping "*@".  However, on suspicion, it kept support for the case
where there is no '@', falling back to stripping ".*".  If ip(8) ever
did this then it was a long time ago - it has been printing a format
including '@' since at least 2004.

Stripping ".*" interferes with interesting administrative decisions,
like having '.' in interface names.

So, drop the fallback to stripping ".*" because it appears to be
unnecessary and can cause inconvenience.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-09-16 03:36:32 +00:00
Michael Tokarev
3ce1d2fde5 Fix spelling mistakes.
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Mon Sep 12 02:29:32 UTC 2022 on sn-devel-184
2022-09-12 02:29:32 +00:00
Martin Schwenke
a0e0fde039 ctdb-tests: Avoid shellcheck warnings
Mostly

  SC2086: Double quote to prevent globbing and word splitting.

Use ctdb_onnode() where it simplifies code.  No behaviour changes
intended.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Thu Aug 25 16:15:45 UTC 2022 on sn-devel-184
2022-08-25 16:15:45 +00:00
Martin Schwenke
ff4935d180 ctdb-tests: Simplify IP address checking
Use a new function and wait_until() to simplify.

get_test_ip_mask_and_iface() not needed here because
select_test_node_and_ips() sets $test_ip, and neither $mask nor $iface
is used.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-08-25 15:22:36 +00:00
Martin Schwenke
42aedc62e3 ctdb-tests: Fix typos
These lines are just wrong:

  try_command_on_node -v $test_node "ip addr show to ${test_node}"
  if -n "$out"; then

The 2nd variable referenced should be $test_ip.  The 2nd line causes
"-n: command not found" because it is missing [] test command
brackets.

Both typos would probably make the test pass unconditionally.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-08-25 15:22:36 +00:00
Martin Schwenke
b88e7322d9 ctdb-tests: Reformat script using shfmt -w -p -i 0 -fn
Whitespace changes only.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-08-25 15:22:36 +00:00
Martin Schwenke
3aecd6e7b5 ctdb-common: CID 1507498: Control flow issues (DEADCODE)
Fix typo in error checking.  While here adjust the bottom of the
range, making errno 0 invalid.

Add corresponding test cases using an alternative syntax for errno packets
(#nnn[;] - trailing ';' is optional).

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Aug  1 09:19:55 UTC 2022 on sn-devel-184
2022-08-01 09:19:55 +00:00
Martin Schwenke
dde461868f ctdb-tests: Add tests for cluster mutex I/O timeout
Block the locker helper child by taking a lock on the 2nd byte of the
lock file.  This will cause a ping timeout if the process is blocked
for long enough.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Jul 28 11:10:54 UTC 2022 on sn-devel-184
2022-07-28 11:10:54 +00:00
Martin Schwenke
25d32ae97a ctdb-tests: Terminate event loop if lock is no longer held
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
061315cc79 ctdb-mutex: Test the lock by locking a 2nd byte range
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
97a1714ee9 ctdb-mutex: open() and fstat() when testing lock file
This makes a file descriptor available for other I/O.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
c07e81abf0 ctdb-mutex: Factor out function fcntl_lock_fd()
Allows blocking mode and start offset to be specified.  Always locks a
1-byte range.

Make the lock structure static to avoid initialising the whole
structure each time.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
9daf22a5c9 ctdb-mutex: Handle pings from lock checking child to parent
The ping timeout is specified by passing an extra argument to the
mutex helper, representing the ping timeout in seconds.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
b5db286791 ctdb-mutex: Do inode checks in a child process
In future this will allow extra I/O tests and a timeout in the parent
to (hopefully) release the lock if the child gets wedged.  For
simplicity, use tmon only to detect when either parent or child goes
away.  Plumbing a timeout for pings from child to parent will be done
later.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
2ecdbcb22c ctdb-mutex: Rename wait_for_lost to lock_io_check
This will be generalised to do more I/O-based checks.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
7ab2e8f127 ctdb-mutex: Rename recheck_time to recheck_interval
There will be more timeouts so clarify the intent of this one.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
c396b61504 ctdb-mutex: Consistently use progname in error messages
To avoid error messages having ridiculously long paths, set progname
to basename(argv[0]).

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
a8da8810f1 ctdb-tests: Add tests for trivial FD monitoring
tmon_ping_test covers complex 2-way interaction between processes
using tmon_ping_send(), including via a socketpair().  tmon_test
covers the more general functionality of tmon_send() but uses a
simpler 1-way harness with wide coverage.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
8d04235f46 ctdb-common: Add trivial FD monitoring abstraction
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
f9467cdf3b ctdb-build: Link in backtrace support for ctdb_util_tests
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
7a1c43fc74 ctdb-build: Separate test backtrace support into separate subsystem
A convention when testing members of ctdb-util is to include the .c
file so that static functions can potentially be tested.  This means
that such tests can't be linked against ctdb-util or duplicate symbols
will be encountered.

ctdb-tests-common depends on ctdb-client, which depends in turn on
ctdb-util, so this can't be used to pull in backtrace support.
Instead, make ctdb-tests-backtrace its own subsystem.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
b195e8c0d0 ctdb-build: Sort sources in ctdb-util and ctdb_unit_tests
Also, rename ctdb_unit_tests to ctdb_util_tests.  The sorting makes
it clear that only items from ctdb-util are tested here.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-28 10:09:34 +00:00
Martin Schwenke
3efa56aa61 ctdb-daemon: Fix printing of tickle ACKs
Commit f5a2037734 arguably got this
back-to-front:

  2022-07-27T09:50:01.985857+10:00 testn1 ctdbd[17820]: ../../ctdb/server/ctdb_takeover.c:514 sending TAKE_IP for '10.0.1.173'
  2022-07-27T09:50:01.990601+10:00 testn1 ctdbd[17820]: Send TCP tickle ACK: 10.0.1.77:33004 -> 10.0.1.173:2049
  2022-07-27T09:50:01.991323+10:00 testn1 ctdb-takeover[19758]: TAKEOVER_IP 10.0.1.173 succeeded on node 0

Unfortunately there is an inconsistency somewhere in the connection
tracking code used for tickle ACKs, making this less than obvious.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Thu Jul 28 09:02:08 UTC 2022 on sn-devel-184
2022-07-28 09:02:08 +00:00
Martin Schwenke
30c40046ef ctdb-build: Add missing dependency on talloc
The include isn't strictly necessary, since it is included via
common/reqid.c anyway.  However, it is a useful hint.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Jul 22 17:01:00 UTC 2022 on sn-devel-184
2022-07-22 17:01:00 +00:00
Martin Schwenke
e831af7b25 ctdb-tests: Work around unreadable file test failure when root
root can read files for which the mode prohibits reading, so this test
case fails when run as root.  Work around this when running as root.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
b20ccaa36d ctdb-scripts: Use "git config" as last resort to parse nfs.conf
Some versions of nfs-utils (e.g. recent CentOS 7) use /etc/nfs.conf
but do not include the nfsconf utility to extract values from the
file.  However, git has an excellent conf file parser, so use it as a
last resort.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
db37043bc5 ctdb-scripts: Avoid ShellCheck warning SC2295
For example:

In /home/martins/samba/samba/ctdb/tools/onnode line 304:
    [ "$nodes" != "${nodes%[ ${nl}]*}" ] && verbose=true
                             ^---^ SC2295 (info): Expansions inside ${..} need to be quoted separately, otherwise they match as patterns.

Did you mean:
    [ "$nodes" != "${nodes%[ "${nl}"]*}" ] && verbose=true

For more information:
  https://www.shellcheck.net/wiki/SC2295 -- Expansions inside ${..} need to b...

Who knew?  Thanks ShellCheck!

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
00f1d6d947 ctdb-common: Use POSIX if_nameindex() to check interface existence
This works as an unprivileged user, so avoids unnecessary errors when
running in test mode (and not as root):

  2022-02-18T12:21:12.436491+11:00 node.0 ctdbd[6958]: ctdb_sys_check_iface_exists: Failed to open raw socket
  2022-02-18T12:21:12.436534+11:00 node.0 ctdbd[6958]: ctdb_sys_check_iface_exists: Failed to open raw socket
  2022-02-18T12:21:12.436557+11:00 node.0 ctdbd[6958]: ctdb_sys_check_iface_exists: Failed to open raw socket
  2022-02-18T12:21:12.436577+11:00 node.0 ctdbd[6958]: ctdb_sys_check_iface_exists: Failed to open raw socket

The corresponding porting test would now become pointless because it
would just confirm that "fake" does not exist.  Attempt to make it
useful by using a less likely name than "fake" and attempting to
detect the loopback interface.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
c77a4fde7a ctdb-daemon: Modernise debug in ctdb_add_public_address()
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
d62fcba7dc ctdb-daemon: Avoid spurious error sending ARPs for released IP
A public IP address can be released in between (and probably before)
attempts to send ARPs.  One situation when this can occur is when a
cluster is shutting down: node A shuts down first, public IPs from
node A are taken over by node B, node B is shutdown.

Notice this when it occurs and cancel further attempts to send ARPs.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
f5a2037734 ctdb-daemon: Modernise debug in ctdb_control_send_arp()
For the tickle ACK logging, render the connection in a buffer.  This
produces more complete information.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
ec5f6425b7 ctdb-protocol: Add separator argument to ctdb_connection_to_buf()
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
440bd86a99 ctdb-daemon: Drop unused ban_state element from CTDB node structure
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
9898e7c555 ctdb-recoverd: Clean up banning culprit code
Make this fully self-contained in the recovery daemon and avoid
indexing by PNN.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
19fbc2da38 ctdb-recoverd: Add pnn field to banning state structure
This structure is now standalone, so indexing by PNN can be avoided
via a subsequent commit.  Index by culprit here to make this commit
simple.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
0b5dd07604 ctdb-recoverd: Add function node_flags() and use it in elections
Indexing a node map by PNN is suboptimal.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 16:09:31 +00:00
Martin Schwenke
e396eb9fbc ctdb-scripts: Only run unhealthy call-out when passing threshold
For memory usage, no need to dump all of this data on every failed
monitor event.  The first call will be enough to diagnose the problem.
The node will then go unhealthy, drop clients and memory usage should
then drop.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Jul 22 07:32:54 UTC 2022 on sn-devel-184
2022-07-22 07:32:54 +00:00
Martin Schwenke
36bd6fd01f ctdb-scripts: Always check memory usage
If filesystem usage exceeds the unhealthy threshold then checking
memory usage checking is not done.  Always do them both.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 06:38:32 +00:00
Martin Schwenke
5e7bbcb069 ctdb-scripts: Avoid ShellCheck info SC2162
SC2162 (info): read without -r will mangle backslashes.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 06:38:32 +00:00
Martin Schwenke
dc7aaca889 ctdb-scripts: Reduce length of very long lines
Use printf to allow easier line breaks and use some early returns.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 06:38:32 +00:00
Martin Schwenke
fc485feae8 ctdb-scripts: De-clutter validate_percentage()
It always takes 2 arguments.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 06:38:32 +00:00
Martin Schwenke
a832c8e273 ctdb-scripts: Reformat using shfmt -w -p -i 0 -fn
About to modify this file, so reformat first as per recent Samba
convention.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 06:38:32 +00:00
Martin Schwenke
3df39aa7fb ctdb-scripts: Avoid ShellCheck warning SC2164
SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

A problem can only occur if /etc/ctdb/ or an important subdirectory is
removed, which means the script itself would not be found.  Use && to
silence ShellCheck.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-07-22 06:38:32 +00:00
Martin Schwenke
be293a125f ctdb-tests: Add new tool unit tests to cover UNKNOWN state
Signed-off-by: Vinit Agnihotri <vagnihotri@ddn.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Jun 28 10:16:59 UTC 2022 on sn-devel-184
2022-06-28 10:16:59 +00:00
Vinit Agnihotri
794f125802 ctdb-tool: Add UNKNOWN pseudo state
When a node is starting, CTDB reports remote nodes as unhealthy by
default.  This can be misleading.

To hide this, report an "UNKNOWN" pseudo state when a remote node is
not disconnected and the runstate is less than or equal to
"FIRST_RECOVERY".

Signed-off-by: Vinit Agnihotri <vagnihotri@ddn.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-28 09:24:31 +00:00
Vinit Agnihotri
428bc71f98 ctdb-tests: Add runstate handling to fake ctdbd
Signed-off-by: Vinit Agnihotri <vagnihotri@ddn.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-28 09:24:31 +00:00
Martin Schwenke
05601cebc9 ctdb-tests: Return error on empty fake ctdbd configuration blocks
These would be unintended errors.  The block should be omitted to keep
the default value.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-28 09:24:31 +00:00
Martin Schwenke
80ba66013e ctdb-scripts: Drop use of eval in CTDB callout handling
eval is not required and causes the follow ShellCheck warning:

  SC2294 (warning): eval negates the benefit of arrays. Drop eval to
  preserve whitespace/symbols (or eval as string).

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Jun 24 10:40:50 UTC 2022 on sn-devel-184
2022-06-24 10:40:50 +00:00
Martin Schwenke
4cbb0b13ba ctdb-tests: Do not require eval tricks for faking NFS callout
The current code requires the use of eval in the NFS callout handling
to facilitate testing.  Improve the code to remove this need.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-24 09:49:33 +00:00
Martin Schwenke
0247fd8a02 ctdb-scripts: Avoid ShellCheck warning SC2162
SC2162 read without -r will mangle backslashes

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-24 09:49:33 +00:00
Martin Schwenke
7f799a8d6f ctdb-tests: Fix faking of program stack traces
The current code works in all current cases but is lazy and wrong.
Fix it to avoid breaking on code changes involving different thread
setups.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-24 09:49:33 +00:00
Martin Schwenke
0b728a4e8f ctdb-tests: Improve Debian-style event script unit testing
Tests can be run by hand using different distro styles, such as:

  CTDB_NFS_DISTRO_STYLE=systemd-debian \
    ./tests/run_tests.sh ./tests/UNIT/eventscripts/{06,60}.nfs.*

This fixes known problems for Debian styles, so the tests now pass for
the following values of CTDB_NFS_DISTRO_STYLE:

  systemd-redhat
  sysvinit-redhat
  systemd-debian
  sysvinit-debian

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-24 09:49:33 +00:00
Martin Schwenke
7f3a0c7e9c ctdb-scripts: Parameterise /etc directory to aid testing
At the moment test results can be influenced by real system
configuration files.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-24 09:49:32 +00:00
Martin Schwenke
337ef7c1b4 ctdb-scripts: Set NFS services to "AUTO" if started by another service
For example, in Sys-V init "rquotad" is started by the main "nfs"
service.  At the moment the call-out can't distinguish between this
case and "should never be run".  Services set to "AUTO" are
hand-stopped/started via service_stop()/service_start() on failure via
restart_after.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-24 09:49:32 +00:00
Martin Schwenke
8b8660d883 ctdb-scripts: Refactor the manual RPC service start/stop
This logic needs improving, so factor the decision making into new
functions service_or_manual_stop() and service_or_manual_start().

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-24 09:49:32 +00:00
Martin Schwenke
cd018d0ff5 ctdb-scripts: Simplify and rename basic_stop() and basic_start()
Drop the argument.  These now just stop/start the overall NFS service,
so rename them appropriately.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-24 09:49:32 +00:00
Martin Schwenke
09fd1e5579 ctdb-scripts: Move nfslock out of basic_stop() and basic_start()
These are only called in one place and should be done inline, since
that is less confusing.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-24 09:49:32 +00:00
Martin Schwenke
a43a1ebe51 ctdb-tests: Reformat script
Samba is reformatting shell scripts using

  shfmt -w -p -i 0 -fn

so update this one before editing.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-06-24 09:49:32 +00:00
Martin Schwenke
e752f841e6 ctdb-daemon: Use DEBUG() macro for child logging
Directly using dbgtext() with file logging results in a log entry with
no header, which is wrong.  This is a regression, introduced in commit
10d15c9e5d.  Prior to this, CTDB's
callback for file logging would always add a header.

Use DEBUG() instead dbgtext().  Note that DEBUG() effectively compares
the passed script_log_level with DEBUGLEVEL, so an explicit check is
no longer necessary.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=15090

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Thu Jun 16 13:33:10 UTC 2022 on sn-devel-184
2022-06-16 13:33:10 +00:00
Martin Schwenke
88f35cf862 ctdb-daemon: Drop unused prefix, logfn, logfn_private
These aren't set anywhere in the code.

Drop the log argument because it is also no longer used.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=15090

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Volker Lendecke <vl@samba.org>
2022-06-16 12:42:35 +00:00
Martin Schwenke
1596a3e84b ctdb-common: Tell file logging not to redirect stderr
This allows ctdb_set_child_logging() to work.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=15090

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Volker Lendecke <vl@samba.org>
2022-06-16 12:42:35 +00:00
Martin Schwenke
b20ee18031 ctdb-tests: Fix a cut and paste error in a comment
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue May 31 05:56:43 UTC 2022 on sn-devel-184
2022-05-31 05:56:43 +00:00
Martin Schwenke
90a96f06a9 ctdb-recoverd: Do not ban on unknown error when taking cluster lock
If the cluster filesystem is unavailable then I/O errors may occur.
This is no worse than contention, so don't ban.  This avoids having
services unavailable for longer than necessary.

Update the associated test to simply confirm that this results in a
leaderless cluster, and leadership is restored when the lock can once
again be taken.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-05-31 05:06:29 +00:00
Martin Schwenke
a400f4e7cc ctdb-doc: Fix typos in the policy routing documentation
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-05-31 05:06:29 +00:00
Martin Schwenke
da9decfc5e ctdb-daemon: Remove unused #includes of rb_tree.h
ctdb_takeover.c and eventscript.c no longer use this.
ipalloc_common.c has never used it.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-05-31 05:06:29 +00:00
Martin Schwenke
80de84d36e ctdb-daemon: Log per-database summary of resent calls
After a recovery that takes a significant amount of time the logs are
flooded with messages about every resent call.

Log a summary instead and demote per-call messages to INFO level.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-05-31 05:06:29 +00:00
Pavel Filipenský
8cb6565011 ctdb: Covscan: unchecked return value for trbt_traversearray32()
Signed-off-by: Pavel Filipenský <pfilipen@redhat.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2022-05-14 03:49:32 +00:00
Pavel Filipenský
91d1d0e4c8 ctdb: Fix trailing whitespace in rb_tree.c
Signed-off-by: Pavel Filipenský <pfilipen@redhat.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2022-05-14 03:49:32 +00:00
Martin Schwenke
64275fc1a2 ctdb-tests: Add backtrace on abort to some tests
These are easier to debug with a backtrace.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue May  3 10:13:23 UTC 2022 on sn-devel-184
2022-05-03 10:13:23 +00:00
Martin Schwenke
d39377d6fc ctdb-tests: Provide a method to dump the stack on abort
Some tests make generous use of assert() and it can be difficult to
guess the cause of failures without resorting to GDB.  This provides
some help.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-05-03 09:19:31 +00:00
Martin Schwenke
73b27def7b build: Add missing ctdb-client dependencies
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-05-03 09:19:31 +00:00
Martin Schwenke
d57d624a77 ctdb-build: Drop unnecessary uses of include/ sub-directory
None of these include any files from the include/ sub-directory.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2022-05-03 09:19:31 +00:00