1
0
mirror of https://github.com/samba-team/samba.git synced 2025-01-18 06:04:06 +03:00

749 Commits

Author SHA1 Message Date
Ronnie Sahlberg
17f0e0890c ReadOnly: Add a new flag to call request packet to indicate that the client wants a readonly delegation
(This used to be ctdb commit a3f54a556e97170eedf43708d58dd32446ca5840)
2011-08-23 10:29:40 +10:00
Ronnie Sahlberg
dda2616cf5 ReadOnly: Add a function to start a revoke of all delegations for a record.
This triggers a child process to be created to perform the actual potentially blocking calls that are required.

(This used to be ctdb commit 7d575ee92c95bc4aab78a33bc1aac7ff0811ab3a)
2011-08-23 10:27:31 +10:00
Ronnie Sahlberg
1bb855bd52 ReadOnly: Add functions to register CALLs to a context used to handle deferal of processing of CALL commands.
Once the contexts are freed, the deferred calls are re-issued to the input packet processing functions again.
This is needed when/if a CALL can not currently be processed by the main engine due to the record being locked down for revoking of all delegations.

The data is passed through several layers of callbacks, and finally a timed event callback to ensure that the processing of the packet will be restarted again at the topmost eventloop, avoinding event loop nesting.

(This used to be ctdb commit cc6f78efcfa3b8caeffbd68018e6dfbf81488dce)
2011-08-23 10:25:57 +10:00
Ronnie Sahlberg
3d495c48d2 ReadOnly: Add an extra flag to ctdb_call_local to specify whether we want to write the record and header back to the tdb (for example we do when performing dmaster migrations)
(This used to be ctdb commit b935e83255aeb3754b2fd37cf5611e02f7283514)
2011-08-23 10:25:05 +10:00
Ronnie Sahlberg
1441b77cce ReadOnly: Add "readonly" flag to the ctdb_db_context to indicate if this database supports readonly operations or not. Add a private lock-less tdb file to the ctdb_db_context to use for tracking delegarions for records
Assume all databases will support readonly mode for now and se thte flag for all databases. At later stage we will add support to control on a per database level whether delegations will be supported or not.

(This used to be ctdb commit 502f86f79944df4bac9094f716e54110c511dc24)
2011-08-23 10:24:26 +10:00
Ronnie Sahlberg
8f63a5dadd ReadOnly: Add 4 new record flags to handle read only delegation and revoking of delegations
(This used to be ctdb commit 875b0bede217547b51f02648b6a28a3c98b6b949)
2011-08-23 10:17:08 +10:00
Ronnie Sahlberg
e8127f0e0f ReadOnly: Add clientside functions to send the UPDATE_RECORD control
(This used to be ctdb commit 74a5b3d7bafd8827a4ee80095fde5798263821e4)
2011-08-23 10:11:38 +10:00
Ronnie Sahlberg
f924b3f40e ReadOnly: Add helper functions to manipulate a TDB_DATA as a bitmap for nodes that we are tracking as having a readonly delegation
(This used to be ctdb commit d10084e62d37674bb8d9e31d457fd23e050545be)
2011-08-23 10:09:42 +10:00
Ronnie Sahlberg
00a870f759 ReadOnly records: Add a new RPC function FETCH_WITH_HEADER.
This function differs from the old FETCH in that this function will also fetch the record header and not just the record data

(This used to be ctdb commit c7196d16e8e03bb2a64be164d15a7502300eae0e)
2011-08-23 10:06:59 +10:00
Volker Lendecke
21bb8abc93 libctdb: "ctdb_request_free" does not need the ctdb_connection parameter
Signed-off-by: Michael Adam <obnox@samba.org>

(This used to be ctdb commit 5a5ed2a43b76bec69494b6cdc6451527f5c472e5)
2011-08-22 17:11:07 +02:00
Martin Schwenke
5ac67504ca Tests: Initial test code for LCP2 IP allocation algorithm.
Move struct ctdb_public_ip_list to ctdb_private.h and put some
definitions for some functions from ctdb_takeover.c there.  This
allows those functions to be called from unit tests.

Add ctdb_takeover_tests.c and the Makefile support to build it.

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 9d34be0233edf3bc022345c0494c4b2a4d7f8480)
2011-07-29 09:01:36 +10:00
Martin Schwenke
ff1a81c872 IP allocation - add LCP2 algorithm.
The current non-deterministic IP allocation algorithm balances IPs
across the whole cluster.  It does not consider different
interfaces/VLANs/subnets, so these different groups of IPs aren't
generally well balanced.

This adds the LCP2 algorithm for IP allocation and allows it to be
enabled by setting the "LCP2PublicIPs" tunable to 1.

The LCP2 algorithm calculates the imbalance of a node by totalling the
squares of the distances between each IP on the node.  The IP distance
is defined as the length longest common prefix (LCP) of bits that is
found when comparing 2 IPs.  The imbalance of a cluster is the maximum
imbalance for any node.  At each step the algorithm selects an
allocation to the IP/node combination that results in the choosing the
allocation that best reduces the imbalance of the cluster.

The implementation splits out the IP allocation part of
ctdb_takeover_run() into new function ctdb_takeover_run_core(), and
then extracts out the basic IP assignment code into new functions
basic_allocate_unassigned() and basic_failback().  3 new functions
lcp2_init(), lcp2_allocate_unassigned() and lcp2_failback() implement
the LCP2 algorithm, and are hooked into ctdb_takeover_run_core().

Signed-off-by: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit 61fc7fbd0235469df22deb6581c6bd47e30bc0be)
2011-07-29 09:01:17 +10:00
Michael Adam
827e871ec4 ctdb_private.h: add record flag CTDB_REC_FLAG_AUTOMATIC
This is a flag that shall signa that a record has been automatically generated by ctdb
and not by an explicit client store operation. This will be used in the ctdb_ltdb_fetch
operation which stores an empty record with default initial header before trying to
migrate the record from the dmaster when the record does not exist in the local tdb.

(This used to be ctdb commit 46381a3cb58ccc11422af8f7798c80ea8d72294f)
2011-03-14 13:35:51 +01:00
Michael Adam
9e8d6b82b5 server: Use the ctdb_ltdb_store_server() in the ctdb daemon for non-persistent dbs
This is realized by adding a ctdb_ltdb_store_fn function pointer to the db
context and filling it in the attach procedure for non-persistent dbs.

(This used to be ctdb commit df49ec44de80affa5ccc637dec12a20a26e8706e)
2011-03-14 13:35:50 +01:00
Michael Adam
a6b13b21c1 client: add accessor function ctdb_header_from_record_handle().
(This used to be ctdb commit cf57efd440ccc3db381386f4749bfcbf8ac5ecae)
2011-03-14 13:35:50 +01:00
Michael Adam
50bd249990 vacuum: add ctdb_local_schedule_for_deletion()
(This used to be ctdb commit b70bc141d84f7355d2c6c901961b7366db566980)
2011-03-14 13:35:49 +01:00
Michael Adam
8569fcbc83 server: implement a new control SCHEDULE_FOR_DELETION to fill the delete_queue.
(This used to be ctdb commit 680223074e992b32ccf6f42cb80c3fa93074fee7)
2011-03-14 13:35:49 +01:00
Michael Adam
46a05397a4 control: add a new control opcode CTDB_CONTROL_SCHEDULE_FOR_DELETION
(This used to be ctdb commit 4cebfa33db3c7effa087f753530c52b2dd8550e6)
2011-03-14 13:35:49 +01:00
Michael Adam
77d4d156d3 control: add macro CHECK_CONTROL_MIN_DATA_SIZE.
This is for the control dispatcher to check whether the input data has
a required minimum size.

(This used to be ctdb commit 2038e745db33cc5c3b4e2db8a00a57ede03906a2)
2011-03-14 13:35:49 +01:00
Michael Adam
9d20f76052 Add a tunable VacuumFastPathCount.
This will control how many fast-path vacuuming runs wil have to
be done, before a full vacuuming will be triggered, i.e. one with
a db-traversal.

(This used to be ctdb commit 0d997ec7e61a7bee2cb05456f9c7d5e6f7a44797)
2011-03-14 13:35:47 +01:00
Michael Adam
cd061f3dee Add a delete_queue to the ctdb database context struct.
This list will be filled by the client using a new
delete control. The list will then be used to implement
a fast-path vacuuming that will traverse this list instead
of traversing the database.

(This used to be ctdb commit 9bbedf786b26bb074f668b31f29a9032af958673)
2011-03-14 13:35:45 +01:00
Michael Adam
f7eeb42219 add a new record flag CTDB_REC_FLAG_VACUUM_MIGRATED.
This is to be used internally. The purpose is to flag a record
as been migrated by a VACUUM_MIGRATION, which is triggered by
a VACUUM_FETCH message as part of the vacuuming. The local store
routine will base its decision whether to delete or to store
the record (among other things) upon the value of this flag.

This flag should never be stored in the local database copies.

(This used to be ctdb commit dd2449c422f323f9b5485e45107a9cc5acc09e08)
2011-03-14 13:35:44 +01:00
Michael Adam
f3fbd31d85 call: Move definition of call flags down to the definition of the flags field.
(This used to be ctdb commit 86c844fb08a7fd33e94f56b8d5e43278120e1162)
2011-03-14 13:35:44 +01:00
Michael Adam
a2c11d6edc call: add new call flag CTDB_CALL_FLAG_VACUUM_MIGRATION
This is to be used when the CTDB_SRVID_VACUUM_FETCH message
triggers the migration of deleted records to the lmaster.
The lmaster can then delete records that have not been
migrated with data instead of storing them.

(This used to be ctdb commit 455cc6616e10b7f09589f9b87cb60f591bb502b0)
2011-03-14 13:35:44 +01:00
Ronnie Sahlberg
8acb677c9c Deferred attach : at early startup, defer any db attach calls until we are out of recovery.
(This used to be ctdb commit eeaabd579841f60ab2c5b004cbbb1f5de2bfe685)
2011-03-01 12:13:34 +11:00
Michael Adam
2bd04f0ff8 persistent: add ctdb_persistent_finish_trans3_commits().
This function walks all databases and checks for running trans3 commits.
It sends replies to all of them (with error code) and ends them.
To be called when a recovery finishes.

(This used to be ctdb commit 70ba153b532528bdccea70c5ea28972257f384c1)
2011-02-24 10:35:26 +01:00
Michael Adam
ace1efb878 persistent: add a ctdb_persistent_state member to the ctdb_db context.
To be used for tracking running transaction commits through recoveries.

(This used to be ctdb commit 1237e15df4af58a3d220eea42a4b75e21e65029f)
2011-02-24 10:35:25 +01:00
Ronnie Sahlberg
65f44e159f Add two new flags for the ltdb header.
One of which signals that the record has never been migrated to/from a node
while containing data.
This property "has never been migrated while non-zero" is important later
to provide heuristics on which records we might be able to purge
from the tdb files cheaply, i.e. without having to rely on the full-blown
database vacuum.

These records are belived to be very common and the pattern would look like
this :
1, no record exists at all.
2, client opens a file
3, samba requests the record for this file
4, an empty record is created on the LMASTER
5, the empty record is migrated to the DMASTER
6, samba writes a <sharemode> to the record locally and the record grows
7, client finishes working the file and closes the file
8, samba removes the sharemode and the record becomes empty again.
9, much later : vacuuming will delete the record

At stage 8, since the record has never been migrated onto a node wile being
non-zero it would be safe, and much more efficient to just delete the record
completely from the database and hand it back to the LMASTER.

The flags occupy the same uint32_t as was previously used for laccessor/lacount
in the header. For now, make sure the flags only define/use the top 16 bits
of this field so that we are sure we dont collide with bits set to one
from previous generations of the ctdb cluster database prior to this
change in semantics of this word.

This is a rework of Michaels patch :
commit 2af1a47cbe1a608496c8caf3eb0c990eb7259a0d
Author: Michael Adam <obnox@samba.org>
Date:   Tue Nov 30 17:00:54 2010 +0100

    add a DEFAULT record flag and a MIGRATED_WITH_DATA record flag.

(This used to be ctdb commit e075670dee8e6ecaba54986f87a85be3d0528b6b)
2011-02-18 10:14:56 +11:00
Ronnie Sahlberg
b57bd0f896 Remove LACOUNT and LACCESSOR and migrate the records immediately.
This concept didnt work out and it is really just as expensive as a full migration
anyway, without the benefit of caching the data for subsequence accesses.

Now, migrate the records immediately on first access.
This will be combined with a "cheap vacuum-lite" for special empty records to
prevent growth of databases.

Later extensions to mimic read-only behaviour of records will include proper shared read-only locking of database records, making the laccessor/lacount read-only access to the data obsolete anyway.

By removing this special case and handling of lacount laccessor makes the codapath where shared read-only locking will be be implemented simpler, and frees up space in the ctdb_ltdb header for use by vacuuming flags as well as read-only locking flags.

(This used to be ctdb commit 155dd1f4885fe142c6f8bd09430f65daf8a17e51)
2011-02-18 10:08:32 +11:00
Ronnie Sahlberg
0f33605866 LockWait congestion.
Add a dlist to track all active lockwait child processes.
Everytime creating a new lockwait handle, check if there is already an
active lockwait process for this database/key and if so,
send the new request straight to the overflow queue.

This means we will only have one active lockwaic child process for a certain key,
even if there were thousands of fetch-lock requests for this key.

When the lockwait processing finishes for the original request, the processing in d_overflow() will automagically process all remaining keys as well.

Add back a --nosetsched argument to make it easier to run under gdb

(This used to be ctdb commit 3e9317a2e1f687b04bf51575d47fcd4faa6e6515)
2011-01-24 12:21:58 +11:00
Rusty Russell
e57362ecf4 ctdb_lockwait: create overflow queue.
Once we have more than 200 children waiting on a particular db, don't create
any more.  Just put them on an overflow queue, and when a child gets a lock
search that queue to see if others were after the same lock (they probably
were).

(This used to be ctdb commit 5e614e8cfd1e9a4b13035a0e400b7a60a745b510)
2011-01-24 12:21:50 +11:00
Ronnie Sahlberg
fcd98a7e59 LIBCTDB: add support for traverse
(This used to be ctdb commit 9463e04038ba36792583f83bd95c1af322dc283a)
2011-01-14 17:38:56 +11:00
Ronnie Sahlberg
c4006ce844 Add ctdb_fork(0 which will fork a child process and drop the real-time
scheduler for the child.

Use ctdb_fork() from callers where we dont want the child to be running
at real-time privilege.

(This used to be ctdb commit 58795a4c9e0624e20fa3e0023b65127053edd103)
2011-01-11 07:40:41 +11:00
Ronnie Sahlberg
ea0df6d882 Revert scheduling back to use real-time processes
Revert this patch:
commit 482c302d46e2162d0cf552f8456bc49573ae729d

We may need to use real-time processes for the main daemon and the recovery daemon to handle the cases where systems come under very high loads.

(This used to be ctdb commit 08bef9dcab6e4da15fc783f8624e5ed09aa060b5)
2011-01-11 07:40:35 +11:00
Ronnie Sahlberg
c69ada0090 add a new ctdb_ltdb function to delete a record in a normal database
(This used to be ctdb commit fe9070ec9be69e6a6fcbf9899e7ced24541c9c3a)
2010-12-07 15:32:30 +11:00
Ronnie Sahlberg
83e68b62dd delay loading the public ip address file until after we have started the transport and discovered ouw own pnn number
(This used to be ctdb commit 1b57fc866fc836b5dbd3ef7b646e5a0f4280e81e)
2010-11-10 14:55:24 +11:00
Ronnie Sahlberg
5f76f3c0e2 Add a new tunable : DisableIPFailover that when set to non 0
will stopp any ip reallocations at all from happening.

(This used to be ctdb commit d8d37493478a26c5f1809a5f3df89ffd6e149281)
2010-11-10 14:55:24 +11:00
Ronnie Sahlberg
5ef29f9f25 Update latency countes to show min/max and average
(This used to be ctdb commit 1919e949af4641ffe919123e44b02fb87c13ab9f)
2010-10-11 15:12:24 +11:00
Ronnie Sahlberg
3ba7ac13eb Create a tunable for how often to collect rolling statistics and initialize it to 1 second
(This used to be ctdb commit cb8c779bb5d9862abbe08919aa181a1a1b2bef18)
2010-09-30 15:00:57 +10:00
Ronnie Sahlberg
9f66a93f12 Add rolling statistics that are collected across 10 second intervals.
Add a new command "ctdb stats [num]" that prints the [num] most recent statistics intervals collected.

(This used to be ctdb commit e6e16fcd5a45ebd3739a8160c8fb5f44494edb9e)
2010-09-29 12:14:45 +10:00
Ronnie Sahlberg
41b6e09fb1 Add a new statistics structure to keep the current running statistics
(This used to be ctdb commit 09e5a2fb47c312f71f455cdbf8d9cabcca1041a4)
2010-09-29 12:14:35 +10:00
Ronnie Sahlberg
39c367a68f Create macros to update the statistics counters and use these macros
everywhere instead of manipulating the coutenrs directly.

(This used to be ctdb commit 2e648df890e5713bc575965d87937827b068d0d7)
2010-09-29 12:14:24 +10:00
Ronnie Sahlberg
c6e20a06c7 set up a handler to catch and log debug messages from the tevent layer
(This used to be ctdb commit fdb4c02f595fa207310a9a48da3fefd653fa9e4b)
2010-09-28 08:30:26 +10:00
Ronnie Sahlberg
22ea35f17d adda GETPUBLICIPS control to libctdb and use this in the test example
enhance the test example to show the new releaseip/takeip messages

(This used to be ctdb commit 21cc57883e6c02b0e037211b26d1d866d5d7f03d)
2010-09-15 14:58:11 +10:00
Stefan Metzmacher
0b5bd411ca server/banning: also release all ips if we're banning ourself
metze

(This used to be ctdb commit c386f2c62f06f1c60047b7d4b1ec7a9eec11873c)
2010-09-14 15:50:31 +10:00
Ronnie Sahlberg
d8d8b9e1d7 add a new serverid to send a message everytime an ip address is taken on the local node
(This used to be ctdb commit 1261f3d9702800a4e59550c881350daf479f00ef)
2010-09-13 15:43:19 +10:00
Ronnie Sahlberg
991a6ae2a0 Update the comment for the range reserved for SAMBA and
define a new symbol to represent this range similarly to NFSD and ISCSID

Keep the old symbol name to be backward compatible with software using
these headers.

(This used to be ctdb commit 2ce34e50d057ba95249117a581658a5ad7e8eb60)
2010-09-13 15:10:36 +10:00
Ronnie Sahlberg
09a08b0da3 define and reserve a range of ctdb message ports for use by nfs and iscsi servers
(This used to be ctdb commit 84a44ac8ee74dd7af15e378c6cafbedb95feec60)
2010-09-13 15:10:24 +10:00
Ronnie Sahlberg
65382a59d1 Add two new server types to the server_id structure.
NFSD and ISCSID for now.

(This used to be ctdb commit 4cd4bab68f0ba0305a585a2aabcb6871cdb11d96)
2010-09-13 15:10:12 +10:00
Ronnie Sahlberg
a2c874bd61 Implement a new function GETNODEMAP in libctdb.
This function returns a pointer to a nodemap structure.

The returned structure must later be freed by calling ctdb_free_nodemap().

Move the definition of ctdb_sock_addr from ctdb_client.h to ctdb_protocol.h

Move the definition of the node flags, ctdb_node_and_flags and ctdb_node_map from ctdb_private.h to ctdb_protocol.h

Add both sync and async example for ctdb_getnodemap to the test application libctdb/tst.c

(This used to be ctdb commit 31c10eb2b337fd7d8a97a1f9e69b0e7570fec71d)
2010-09-13 14:32:11 +10:00