IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
When sending messages to clients, ctdb checks for EAGAIN error code and
schedules next write in the subsequent event loop. Using sys_write in
these places causes ctdb to loop hard till a client is able to read from
the socket. With real time scheduling, ctdb daemon spins consuming 100%
of CPU trying to write to the client sockets. This can be quite harmful
when running under VMs or machines with single CPU.
This regression was introduced when all read/write calls were replaced to
use sys_read/sys_write wrappers (c1558adeaa).
The existing code backs off in case of EAGAIN failures and waits for an
event loop to process the write again. This should give ctdb clients
a chance to get scheduled and to process the ctdb socket.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Tue Feb 24 12:29:30 CET 2015 on sn-devel-104
If we have already partly written a packet, "data" and thus "pkt->data"
does not point to the start of the packet anymore. Assign "hdr" while
it still points at the start of the header.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Jul 22 06:09:50 CEST 2014 on sn-devel-104
If we process all the data available in a socket buffer, CTDB can stay busy
processing lots of packets via immediate event mechanism in tevent. After
processing an immediate event, tevent returns without epoll_wait. So as long
as there are immediate events, tevent will never poll other FDs. CTDB will
report this as "Event handling took xx seconds" warning. This is misleading
since CTDB is very busy processing packets, but never gets to the point of
polling FDs.
The improvement in socket handling made it worse when handling traverse
control. There were lots of packets filled in the socket buffer quickly and
CTDB stayed busy processing those packets and not polling other FDs and timer
events. This can lead to controls timing out and in worse case other nodes
marking busy node as disconnected.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 92939c1178d04116d842708bc2d6a9c2950e36cc)
This reverts commit 5e9b1a7e24d058ff88aaa0563db36a804e866fa9.
This is not the best approach. Allowing queue buffer size to grow
indefinitely causes large number of CTDB packets to be queued up very
quickly which when processed via immediate events will block CTDB from
processing events from other FDs. If there are immediate events queued
up, tevent will never process any of the FDs till all immediate events
are processed.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit d8b094e804efc53fae9f44c6ef961b7b5797d290)
Currently queue buffer size is realloc'd every time we need to extend the
buffer. Small increments can cause memory fragmentation. Instead always
extend buffer in multiples of 4K. This should reduce multiple talloc_realloc
calls when there are lots of packets in the socket buffer.
Also, if queue buffer has grown larger than 64K, throw away the buffer once
all the requests in the queue have been processed. That way queue does not
hold on to large buffers.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 5e9b1a7e24d058ff88aaa0563db36a804e866fa9)
tevent_schedule_immediate() is much more efficient at handling events that need
to be processed immediately rather than creating timed events with
timeval_zero().
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit 11734be353a1e246163eda631d35dfe55d1d6fb1)
This improves the processing of packets considerably. It has been
observed that there can be as many as 10 packets in the socket buffer and
the current code of reading a single packet from a socket at a time is
not very optimal. This change reads all the bytes from socket buffer and
then parses to extract multiple packets. If there are multiple packets,
set up a timed event to process next packet.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(This used to be ctdb commit d788bc8f7212b7dc1587ae592242dc8c876f4053)
queue_io_read() may be reentered via the queue callback, recoverd is
particularly guilty of this.
queue_io_read() is not safe for reentry if more than one packet is
received and partial chunks follow - data read off the pipe on re-entry
is assumed to be the start-of-packet four byte length. This leads to a
wrongly aligned stream and the notorious "Invalid packet of length 0"
errors.
This change fixes queue_io_read() to be safe under reentry, only a
single packet is processed per call.
https://bugzilla.samba.org/show_bug.cgi?id=8319
(This used to be ctdb commit 9ea41d2fab612772f861270c8a59c01c43bd3a4c)
In Samba this is now called "tevent", and while we use the backwards
compatibility wrappers they don't offer EVENT_FD_AUTOCLOSE: that is now
a separate tevent_fd_set_auto_close() function.
This is based on Samba version 7f29f817fa.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 85e5e760cc91eb3157d3a88996ce474491646726)
We've been seeing "Invalid packet of length 0" errors, but we don't know
what is sending them. Add a name for each queue, and print nread.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit e6cf0e8f14f4263fbd8b995418909199924827e9)
ctdb_client.h is the existing internal client interface (which was mainly
in ctdb.h), and ctdb_protocol.h is the information needed for the wire
protocol only.
ctdb.h will be the new, shiny, libctdb API.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 4bba6b8cd47b352f98d41f9f06258d5ac3c9adef)
packets, to avoid the queue to grow excessively if smbd has blocked.
This could cause traverse packets to become discarded in case the main
smbd daemon does a traverse of a database while there is a recovery
(sending a erconfigured message to smbd, causing an avalanche of unlock
messages to be sent across the cluster.)
This avalance of messages could cause also the tranversal message to be
discarded causing the main smbd process to hang indefinitely waiting
for the traversal message that will never arrive.
Bump the maximum queue length before starting to discard messages from
1000 to 1000000 and at the same time rework the queueing slightly so we
can append messages cheaply to the queue instead of walking the list
from head to tail every time.
(This used to be ctdb commit 59ba5d7f80e0465e5076533374fb9ee862ed7bb6)
Wolfgang saw a talloc complaint about using freed memory in ctdb_tcp_read_cb.
His fix was to remove the talloc_free() in that function, which causes
loops when a socket is closed (as it does not get removed from the event
system), eg:
netcat 192.168.1.2 4379 < /dev/null
The real bug is that when we have more than one pending packet in the
queue, we loop calling the callback without any safeguards should that
callback free the queue (as it tends to do on invalid packets). This
can be reproduced by sending more than one bogus packet at once:
# Length word at start: 4 == empty packet (assumed little endian)
/usr/bin/printf \\4\\0\\0\\0\\4\\0\\0\\0 > /tmp/pkt
netcat 192.168.1.2 4379 < /tmp/pkt
Using a destructor we can check if the callback frees us, and exit
immediately. Elsewhere, we return after the callback anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 4d0523dd94fb07e860b3e8118691f93d1ef8d0fa)
make ctdb_queue_length() cheaper by using a counter variable instead of counting the number of packets each time.
(This used to be ctdb commit 331c6e3afd96d8b5e191153a631efdbdabb6ea33)
Add a new tunable to control the maximum queue size we allow to a blocked client before we start discarding REQ_MESSAGES instead of queueing them for delivery.
This avoids having queued up very very large number of MESSAGES that samba semds
between eachother to nodes that are blocked/banned/stopped for extended periods
.
(This used to be ctdb commit f76d6fed8f9630450263b9fa4b5fdf3493fb1e11)
and a ctdb command to pull the talloc memory map from a recovery daemon
ctdb rddumpmemory
(This used to be ctdb commit d23950be7406cf288f48b660c0f57a9b8d7bdd05)
- use the tdb_chainlock_mark() call to allow us to guarantee forward progress in the ctdb_lockwait code
(This used to be ctdb commit e201e98aad0fef6a779a80f3b1ae7792953e2d6b)
for 2 days.
The main bug was in smbd, but there was a secondary (and more subtle)
bug in ctdb that the bug in smbd exposed. When we get send a dmaster
reply, we have to correctly update the dmaster in the recipient even
if the original requst has timed out, otherwise ctdbd can get into a
loop fighting over who will handle a key.
This patch also cleans up the packet allocation, and makes ctdbd
become a real daemon.
(This used to be ctdb commit 59405e59ef522b97d8e20e4b14310a217141ac7c)
- when we do a store_unlock the lock record becomes unlocked, so we
must destroy it (or we leak memory)
(This used to be ctdb commit d85955640e670dd580073da96b25fb8a10c08d18)
send the correct structure back to a client
assorted other cleanups
(tests/test1.sh now works in daemon mode)
(This used to be ctdb commit f4593754cab750dfdb9384884502e2e1b8fde1f0)
