1
0
mirror of https://github.com/samba-team/samba.git synced 2024-12-25 23:21:54 +03:00
Commit Graph

3523 Commits

Author SHA1 Message Date
Michael Adam
827e871ec4 ctdb_private.h: add record flag CTDB_REC_FLAG_AUTOMATIC
This is a flag that shall signa that a record has been automatically generated by ctdb
and not by an explicit client store operation. This will be used in the ctdb_ltdb_fetch
operation which stores an empty record with default initial header before trying to
migrate the record from the dmaster when the record does not exist in the local tdb.

(This used to be ctdb commit 46381a3cb58ccc11422af8f7798c80ea8d72294f)
2011-03-14 13:35:51 +01:00
Michael Adam
6506314c4a ctdb_ltdb_store_server: add ability to send SCHEDULE_FOR_DELETION control to ctdb_ltdb_store.
(This used to be ctdb commit ab2711701999a5ecc23a36b3d9ba8e94f92e4c87)
2011-03-14 13:35:51 +01:00
Michael Adam
1924d0d365 ctdb_ltdb_store_server: Improve debug message in ctdb_ltdb_store when store or delete fails.
(This used to be ctdb commit 2559b2a45eb11834da3b0e0963e24351c8b7477f)
2011-03-14 13:35:51 +01:00
Michael Adam
7088e2144f ctdb_ltdb_store_server: always store the data when ctdb_ltdb_store() is called from the client
This also fixes a segfault since ctdb_lmaster uses the vnn_map.

(This used to be ctdb commit e58c8f51f27e468897af5210b80e5f5f45c3c4bb)
2011-03-14 13:35:51 +01:00
Michael Adam
6384512eb7 ctdb_ltdb_store_server: implement fastpath vacuuming deletion based on VACUUM_MIGRATED flag.
When the record has been obtained by the lmaster as part of the vacuuming-fetch
handler and it is empty and never been migrated with data, then such records
are deleted instead of being stored. These records have automatically been
deleted when leaving the former dmaster, so that they vanish for good when
hitting the lmaster in this way. This will reduces the load on traditional
vacuuming.

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

(This used to be ctdb commit c9b65f3602f51bcbf0e6d82c12076c31e4aebe38)
2011-03-14 13:35:51 +01:00
Michael Adam
7602f9a9af ctdb_ltdb_store_server: delete an empty record that is safe to delete instead of storing locally.
When storing a record that is being migrated off to another node
and has never been migrated with data, then we can safely delete it
from the local tdb instead of storing the record with empty data.

Note: This record is not deleted if we are its lmaster or dmaster.

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

(This used to be ctdb commit 3cca0d4b48325d86de2cb0b44bb7811a30701352)
2011-03-14 13:35:50 +01:00
Michael Adam
9e8d6b82b5 server: Use the ctdb_ltdb_store_server() in the ctdb daemon for non-persistent dbs
This is realized by adding a ctdb_ltdb_store_fn function pointer to the db
context and filling it in the attach procedure for non-persistent dbs.

(This used to be ctdb commit df49ec44de80affa5ccc637dec12a20a26e8706e)
2011-03-14 13:35:50 +01:00
Michael Adam
7948be380c server: create a server variant ctdb_ltdb_store_server() of ctdb_ltdb_store().
This is supposed to contain logic for deleting records that are safe
to delete and scheduling records for deletion. It will be called in
server context for non-persistent databases instead of the standard
ctdb_ltdb_store() function.

(This used to be ctdb commit 23631ffc152486aed9ce5b69a391e52bc4947833)
2011-03-14 13:35:50 +01:00
Michael Adam
4cca8876e2 daemon: fill ctdb->ctdbd_pid early
(This used to be ctdb commit 3da1e2e30bf34622f08e6ecd5b8fe55684e5007a)
2011-03-14 13:35:50 +01:00
Michael Adam
670330b331 test: send SCHEDULE_FOR_DELETION control from randrec test.
(This used to be ctdb commit 30aa55b3efc6fbd4078f93da386b6aeb337c1a0c)
2011-03-14 13:35:50 +01:00
Michael Adam
a6b13b21c1 client: add accessor function ctdb_header_from_record_handle().
(This used to be ctdb commit cf57efd440ccc3db381386f4749bfcbf8ac5ecae)
2011-03-14 13:35:50 +01:00
Michael Adam
50bd249990 vacuum: add ctdb_local_schedule_for_deletion()
(This used to be ctdb commit b70bc141d84f7355d2c6c901961b7366db566980)
2011-03-14 13:35:49 +01:00
Michael Adam
8569fcbc83 server: implement a new control SCHEDULE_FOR_DELETION to fill the delete_queue.
(This used to be ctdb commit 680223074e992b32ccf6f42cb80c3fa93074fee7)
2011-03-14 13:35:49 +01:00
Michael Adam
46a05397a4 control: add a new control opcode CTDB_CONTROL_SCHEDULE_FOR_DELETION
(This used to be ctdb commit 4cebfa33db3c7effa087f753530c52b2dd8550e6)
2011-03-14 13:35:49 +01:00
Michael Adam
77d4d156d3 control: add macro CHECK_CONTROL_MIN_DATA_SIZE.
This is for the control dispatcher to check whether the input data has
a required minimum size.

(This used to be ctdb commit 2038e745db33cc5c3b4e2db8a00a57ede03906a2)
2011-03-14 13:35:49 +01:00
Michael Adam
a5cb537745 vacuum: lower level of hash collision debug message to INFO
(This used to be ctdb commit b9bdef46fedfbc543263b67cfee3e896773cd8e8)
2011-03-14 13:35:49 +01:00
Michael Adam
122682be63 vacuum: add statistics output to the fast and full traverse runs.
(This used to be ctdb commit 3addd28aa73883b3b05888e309d19db0eb67eab9)
2011-03-14 13:35:49 +01:00
Michael Adam
e8b08ce702 vacuum: refactor insert_delete_record_data_into_tree() out of add_record_to_delete_tree()
for reuse in filling the delete_queue.

(This used to be ctdb commit 7bbb12695c24da25671f1c39a411295d35870d2c)
2011-03-14 13:35:48 +01:00
Michael Adam
61af2b4a1d vacuum: change all Vacuum*Interval tunables to default to 10
So, by default we have a fastpath vacuuming every 10 seconds and
full blown db-traverse vacuuming once every 10 minutes.

(This used to be ctdb commit 4f0ace982dbb5b4f9c035dbf4cb0ae74cd18d81b)
2011-03-14 13:35:48 +01:00
Michael Adam
c95b4f841b vacuum: disable full db-traverse vacuuming runs when VacuumFastPathCount == 0
(This used to be ctdb commit 571683e7c48aeed8ce41c584d016ced7ff0d2e2d)
2011-03-14 13:35:48 +01:00
Michael Adam
28fdb00a22 vacuum: Only run full vacuumig (db traverse) every VacuumFastPathCount times.
(This used to be ctdb commit 23b8c8c5fc8604ee0bd6da1f4b5152277eb5f1c0)
2011-03-14 13:35:48 +01:00
Michael Adam
cab1f75db5 vacuum: reset the fast path count in the event handle if it exceeds the limit.
(This used to be ctdb commit 91e6d36a190b1c9e4c8b18f7833e51c5c9a67574)
2011-03-14 13:35:48 +01:00
Michael Adam
f502e5aa98 vacuum: bump the number of fast-path runs in the vacuum child destructor
(This used to be ctdb commit c0668bfe0bb4e69988ae34d875568d08539e6fb9)
2011-03-14 13:35:47 +01:00
Michael Adam
3c4c560019 vacuum: add a fast_path_count to the vacuum_handle.
(This used to be ctdb commit 53a39d0cc5ea251c2189ec8178ccb769fa046c43)
2011-03-14 13:35:47 +01:00
Michael Adam
9d20f76052 Add a tunable VacuumFastPathCount.
This will control how many fast-path vacuuming runs wil have to
be done, before a full vacuuming will be triggered, i.e. one with
a db-traversal.

(This used to be ctdb commit 0d997ec7e61a7bee2cb05456f9c7d5e6f7a44797)
2011-03-14 13:35:47 +01:00
Michael Adam
8395fcc079 vacuum: traverse the delete_queue befor traversing the database.
(This used to be ctdb commit 04c335f9195a5fd83c91a57d06b1e4eaa511844e)
2011-03-14 13:35:47 +01:00
Michael Adam
bb22135f28 vacuum: add delete_queue_traverse() for traversal of the delete_queue.
(This used to be ctdb commit 5eee05c4d256c08f4ee60a1a69efda6844e39729)
2011-03-14 13:35:47 +01:00
Michael Adam
08ab829f0b vacuum: reduce indentation in add_record_to_delete_tree()
This simplyfies the logical structure a bit by using early return.

(This used to be ctdb commit 4d32908fdcec120426536a761e1d0be60f076198)
2011-03-14 13:35:47 +01:00
Michael Adam
6a1857ee70 vacuum: refactor new add_record_to_delete_tree() out of vacuum_traverse().
This will be reused by the traversal of the delete_queue list.

(This used to be ctdb commit 4407e5a7fb045ce56b6d902f7116de663ea648cb)
2011-03-14 13:35:46 +01:00
Michael Adam
d331d7a7b3 vacuum: skip adding records to list of records to send to lmaster on lmaster
This list is skipped afterwards when the lists are processed.

(This used to be ctdb commit e99834c1a2eea60f7f974c0689ae0a65cfe178ff)
2011-03-14 13:35:46 +01:00
Michael Adam
3d7bfc7a3e vacuum: refactor new add_record_to_vacuum_fetch_list() out of vacuum_traverse().
This is the function that fills the list of records to send to each lmaster
with the VACUUM_FETCH message.

This function will be reused in the traverse function for the delete_queue.

(This used to be ctdb commit d4ab790c1f679e833eb97816762fcfcee15ccb10)
2011-03-14 13:35:46 +01:00
Michael Adam
ee593284d5 server: rename ctdb_repack_db() to ctdb_vacuum_and_repack_db()
(This used to be ctdb commit 6c603f85726d2efac9710af7c4875ded2ca7230e)
2011-03-14 13:35:46 +01:00
Michael Adam
a81f740f3d When wiping a database, clear the delete_queue.
(This used to be ctdb commit 731a6011ce4a1301f86eacb039955745f2b5d866)
2011-03-14 13:35:46 +01:00
Michael Adam
6a1cfe82f1 vaccum: clear the fast-path vacuuming delete_queue after creating the vacuuming child.
Maybe we should keep a copy for the case that the vacuuming fails?

(This used to be ctdb commit f19fe5b45748a6998c6950a5b1db7ec2c4468c1c)
2011-03-14 13:35:45 +01:00
Michael Adam
b9c9b989ce When attaching to a non-persistent DB, initialize the delete_queue.
(This used to be ctdb commit 0aff1b61dd1b683c6739478008a5b014b933df50)
2011-03-14 13:35:45 +01:00
Michael Adam
cd061f3dee Add a delete_queue to the ctdb database context struct.
This list will be filled by the client using a new
delete control. The list will then be used to implement
a fast-path vacuuming that will traverse this list instead
of traversing the database.

(This used to be ctdb commit 9bbedf786b26bb074f668b31f29a9032af958673)
2011-03-14 13:35:45 +01:00
Michael Adam
dbb520b6ad call: becoming dmaster in VACUUM_MIGRATION, set the VACUUM_MIGRATED record flag
This temporary flag is used for the local record storage function to
decide whether to delete an empty record which has never been migrated
with data as part of the fast-path vacuuming process or, or to store
the record.

(This used to be ctdb commit c11ca778ee90444c44dee0a629cd2eefa3a1f75e)
2011-03-14 13:35:45 +01:00
Michael Adam
73e6618a48 call: hand the submitted record_flags to local record storage function.
(This used to be ctdb commit 4079b8bf7a57a27a45d29784a1b0a414c778e552)
2011-03-14 13:35:45 +01:00
Michael Adam
eb1b7d1c05 call: transfer the record flags in the ctdb call packets.
This way, the MIGRATED_WITH_DATA information can be transported
along with the records. This is important for vacuuming to function
properly.

The record flags are appended to the data section of the ctdb_req_dmaster
and ctdb_reply_dmaster structs.

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

(This used to be ctdb commit 945187d64cfc7bd30a0c3b0d548cbe582d95dde3)
2011-03-14 13:35:44 +01:00
Michael Adam
2ad1c3f6c7 server: in the VACUUM_FETCH handler, add the VACUUM_MIGRAION to the call flags
This way, the records coming in via this handler, can be treated appropriately.
Namely, they can be deleted instead of being stored when the meet the fast-path
vacuuming criteria (empty, never migrated with data...)

(This used to be ctdb commit fb5d832104970320359b3e474eb291ca3d629380)
2011-03-14 13:35:44 +01:00
Michael Adam
f7eeb42219 add a new record flag CTDB_REC_FLAG_VACUUM_MIGRATED.
This is to be used internally. The purpose is to flag a record
as been migrated by a VACUUM_MIGRATION, which is triggered by
a VACUUM_FETCH message as part of the vacuuming. The local store
routine will base its decision whether to delete or to store
the record (among other things) upon the value of this flag.

This flag should never be stored in the local database copies.

(This used to be ctdb commit dd2449c422f323f9b5485e45107a9cc5acc09e08)
2011-03-14 13:35:44 +01:00
Michael Adam
f3fbd31d85 call: Move definition of call flags down to the definition of the flags field.
(This used to be ctdb commit 86c844fb08a7fd33e94f56b8d5e43278120e1162)
2011-03-14 13:35:44 +01:00
Michael Adam
a2c11d6edc call: add new call flag CTDB_CALL_FLAG_VACUUM_MIGRATION
This is to be used when the CTDB_SRVID_VACUUM_FETCH message
triggers the migration of deleted records to the lmaster.
The lmaster can then delete records that have not been
migrated with data instead of storing them.

(This used to be ctdb commit 455cc6616e10b7f09589f9b87cb60f591bb502b0)
2011-03-14 13:35:44 +01:00
Michael Adam
89f27f9424 recoverd: in a recovery, set the MIGRATED_WITH_DATA flag on all records
Those records that are kept after recovery, are non-empty, and
stored identically on all nodes. So this is as if they had been
migrated with data.

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

(This used to be ctdb commit 101be642e492a3a54231e2e3e6553a59380fe702)
2011-03-14 13:35:43 +01:00
Michael Adam
64fc05e562 server: when we migrate off a record with data, set the MIGRATED_WITH_DATA flag
(This used to be ctdb commit f5fb232117886186066ab3430fdd2307cba94960)
2011-03-14 13:35:43 +01:00
Michael Adam
3cb3620acf vacuum: check lmaster against num_nodes instead of vnn_map->size
When lmaster is bigger than the biggest recorded node number,
then exit the traverse with error.

(This used to be ctdb commit 3930c7796b72bbf275bbca8aaeceec3e705a964b)
2011-03-14 13:35:43 +01:00
Michael Adam
dd75ae8c15 vacuum: reduce indentation of the loop sending VACUUM_FETCH controls
This slightly improves the code structure in that loop.

(This used to be ctdb commit bc4990e600c53433a924a0d70e3488a5a6bdc1ff)
2011-03-14 13:35:43 +01:00
Michael Adam
70fa7394fa vacuum: correctly send TRY_DELETE_RECORDS ctrl to all active nodes
Originally, the control was sent to all records in the vnn_map, but
there was something still missing here:
When a node can not become lmaster (via CTDB_CAPABILITY_LMASTER=no)
then it will not be part of the vnn_map. So such a node would
be active but never receive the TRY_DELETE_RECORDS control from a
vacuuming run.

This is fixed in this change by correctly building the list of
active nodes first in the same way that the recovery process does it.

(This used to be ctdb commit 49247df4a47a8a107fa7dd7b187e69e243e6bdbe)
2011-03-14 13:35:43 +01:00
Michael Adam
33b5d9c1a1 vacuum: in ctdb_vacuum_db, fix the length of the array of vacuum fetch lists
This patch fixes segfaults in the vacuum child when at least one
node has been stopped or removed from the cluster:

The size of the vnn_map is only the number of active nodes
(that can be lmaster). But the node numbers that are referenced
by the vnn_map spread over all configured nodes.

Since the array of vacuum fetch lists is referenced by the
key's lmaster's node number later on, the array needs to
be of size num_nodes instad of vnn_map->size.

(This used to be ctdb commit 136508e3f4dd0acc210dde938ad59ef38b63d3a1)
2011-03-11 23:00:51 +01:00
Michael Adam
8d49f4878c Fix typos in a comment in vacuum_traverse.
(This used to be ctdb commit 6a4df8242ee4d095ff03229a168b83bcd84c8a7a)
2011-03-09 21:14:14 +01:00