samba-mirror/ctdb/doc/readonlyrecords.txt

Read-Only locks in CTDB
=======================

Problem
=======
CTDB currently only supports exclusive Read-Write locks for clients(samba) accessing the TDB databases.
This works well mostly, but when very mny number of clients are accessing the same file, at the same time,
this will cause the exclusive lock as well as the record itself to rapidly bounce between nodes, and acts as
a scalability limitation.

This primarily affects locking.tdb and brlock.tdb, two databases where record access is mostly read, and where a read request is magnitudes more common than read-write requests.

For the common case, if CTDB provided shared non-exlusive Read-Only lock semantincs, this would greatly improve scaling for these workloads.


Desired properties
==================
We can not make backward incompatible changes the ctdb/ltdb header for the records.

A Read-Only lock enabled ctdb demon must be able to interoperate with a non-Read-Only lock enbled daemon.

Getting a Read-Only look should not be slower than getting a Read-Write lock.

Requesting a Read-Only lock should never trigger a record migration.

When revoking Read-Only locks for a record, this should involve only those nodes that hold a Read-Only lock right now and should avoid broadcasting opportunistic revocations. (must track which nodes are delegated to)

When a Read-Write lock is requested, if there are Read-Only locks delegated to other nodes, the DMASTER will defer the record migration until all read-only locks are first revoked (synchronous revoke).

Due to the cost of revoking Read-Only locks has on getting a Read-Write lock, the implementation should try to avoid
creating Read-Only locks, unless it has indication that there is contention. This may mean that even if client requests a Read-Only lock we may still provide a full Read-Write lock in order to avoid the cost of revoking the locks in some cases.

Read-Only locks require additional state to be stored in a separate database, containing information about which nodes have have been delegated Read-Only locks. This database should be kept at minimal size.

Read-Only locks should not significantly complicate the normal record/create/migration/deletion cycle for normal records.

Read-Only locks should not complicate the recovery process.

Read-Only locks should not complicate the vacuuming process.

We should avoid forking new child processes as far as possible from the main daemon.

Client-side implementation, samba, libctdb, others, should have minimal impact when Read-Only locks are implemented.
Client-side implementation must be possible with only minor conditionals added to the existing lock-check-fetch-unlock loop that clients use today for Read-Write locks. So that clients only need one single loop that can handle both Read-Write locking as well as Read-Only locking. Clients should not need two nearly identical loops.


Implementation
==============

Four new flags are allocated in the ctdb/ltdb record header.
HAVE_DELEGATIONS, HAVE_READONLY_LOCK, REVOKING_READONLY and REVOKE_COMPLETE

HAVE_DELEGATIONS is a flag that can only be set on the node that is currently the DMASTER for the record. When set, this
flag indicates that there are Read-Only locks delegated to other nodes in the cluster for this record.

HAVE_READONLY is a flag that is only set on nodes that are NOT the DMASTER for the record. If set this flag
indicates that this record contains an up-to-date Read-Only version of this record.
A client that only needs to read, but not to write, the record can safely use the content of this record as is regardless of the value of the DMASTER field of the record.

REVOKING_READONLY is a flag that is used while a set of read only delegations are being revoked.
This flag is only set when HAVE_DELEGATIONS is also set, and is cleared at the same time as HAVE_DELEGATIONS is cleared.
Normal operations is that first the HAVE_DELEGATIONS flag is set when the first delegation is generated.
When the delegations are about to be revoked, the REVOKING_READONLY flag is set too.
Once all delegations are revoked, both flags are cleared at the same time.
While REVOKING_READONLY is set, any requests for the record, either normal request or request for readonly will be deferred.
Deferred requests are linked to a list of deferred requests for the hash of the record until the time that the revokation is completed.
This flags is set by the main ctdb daemon when it starts revoking this record.

REVOKE_COMPLETE
The actual revoke of records is done by a child process, spawned from the ctdb amin daemon when it starts the process to revoke the records.
Once the child process has finished revoking all delegations, it will set the flag REVOKE_COMPLETE for this record to signal to the master daemon that the record has been successfully revoked.
At this stage the child process will also trigger an event in the main daemon that revoke is complete, and that the main dameon should start re-processing all deferred calls.



Once the revoke process is completed. There will be at least one deferred call to access this record, the initical call to for an exclusive fetch_lock() that triggered the revoke process to be started.
In addition to nthis deferred call there may also be additional requests that have also become deferred while the revoke was in process. These can be either exclusive fetch_locks() or they can be other calls to request a new readonly lock on the record.
Once the revoke is completed, the main daemon will reprocess all exclusive fetch_lock() requests immediately and respond to the clients.
But any requests for readadonly locks will be deferred for an additional period of time before they are re-processed.
This is to allow the client that needs a fetch_lock() to update the record to get some time to access and work on the record without having to compete with the possibly very many requests to get new readonly delegations created.






The ctdbdb structure is expanded so that it contains one extra TDB database for each normal, non-persistent datbase.
This new database is used for tracking delegations for the records. A record in the normal database that has "HAVE_DELEGATION" set will always have a corresponding record at the same key. This record contains the set of all nodes that the record is delegated to.
This tracking database is lockless, using TDB_NOLOCK, and is only ever accessed by the main ctdbd daemon.
The lockless nature and the fact that no other process ever access this TDB means we are guranteed non-blocking access to records in the trcking database.

The ctdb_call PDU is allocated with two new flags WANT_READONLY and WITH_HEADER.
This first flag is used to explicitely requesting a read-only record from the DMASTER/LMASTER.
The second flag is used to request that the fetch operation will return not only the data for the record but also
the record header. 
If the record does not yet exist, this is a returned as an error to the client and the client will retry the request loop.

A new control is added to make remote nodes remove the HAVE_READONLY_LOCK from a record.



Client implementation
=====================
Clients today use a loop for record fetch lock that looks like this
    try_again:
        lock record in tdb

        if record does not exist in tdb,
            unlock record
            ask ctdb to migrate record onto the node
            goto try_again

        if record dmaster != this node pnn
            unlock record
            ask ctdb to migrate record onto the node
            goto try_again

    finished:

where we basically spin, until the record is migrated onto the node and we have managed pinning it down.

This will change to instead do

    try_again:
        lock record in tdb

        if record does not exist in tdb,
            unlock record
            ask ctdb to migrate record onto the node
            goto try_again

        if record dmaster == current node pnn
            goto finished

        if read-only lock
            if HAVE_READONLY_LOCK or HAVE_DELEGATIONS is set
                goto finished
            else
                unlock record 
                ask ctdb for read-only copy (WANT_READONLY|WITH_HEADER)
                if failed to get read-only copy (*A)
                    ask ctdb to migrate the record onto the node
                    goto try_again
                lock record in tdb
                goto finished

        unlock record
        ask ctdb to migrate record onto the node
        goto try_again

    finished:

If the record does not yet exist in the local TDB, we always perform a full fetch for a Read-Write lock, even if only a Read-Only lock ws requested.
This means that for first access, we use the the current grab a Read-Write lock, just like we do today.
This creates the record and migrates it onto the node. The local node becomes DMASTER for the record.
This did not yet trigger a Read-Only delegation to be created.
Future reference to this record by the local samba daemons will still access/lock the record locally without triggereing a Read-Only delegation to be created since the record is already hosted on the local node as DMASTER.

Only if the record is contended, i.e. it has been created but we are no longer the DMASTER for the record, only for this case will we create a Read-Only delegation for this record.

This heuristics provide a mechanism where we will not create Read-Only delegations until we have some indication that the record may be contended.

This avoids creating and revoking Read-Only delegations when only a single client is repeatedly accessing the same set of records.
This also aims to limit the size of the tracking tdb.


Server implementation
=====================
When receiving a ctdb_call with the WANT_READONLY flag:

If this is the LMASTER for the record and the record does not yet exist, LMASTER will return an error back to the client (*A above) and the client will try to recover. In particular, LMASTER will not create a new record for this case.

If this is the LMASTER for the record and the record exists, the PDU will be forwrded to the DMASTER for the record.

If this node is not the DMASTER for this record, we forward the PDU back to the LMASTER. Just as we always do today.

If this is the DMASTER for the record, we need to create a Read-Only delegation. This is done by
     lock record
     increase the RSN by one for this record
     set the HAVE_DELEGATIONS flag for the record
     write the updated record to the TDB
     create/update the tracking TDB nd add this new node to the set of delegations
     send a modified copy of the record back to the requesting client.
         modifications are that RSN is decremented by one, so delegated records are "older" than on the DMASTER,
         it has HAVE_DELEGATIONS flag stripped off, and has HAVE_READONLY_LOCK added.
     unlock record

Important to note is that this does not trigger a record migration.


When receiving a ctdb_call without the WANT_READONLY flag:

If this is the DMASTER for the this might trigger a migration. So,
IF the record has HAVE_DELEGATIONS set, we create a child process and defer processing of this PDU until the child process has completed.

From the child process we will call out to all nodes that have delegations for this record and tell them to invalidate this record by clearing the HAVE_READONLY_LOCK from the record.
Once all delegated nodes respond back, the child process signals back to the main daemon the revoke has completed.
(child process may not access the tracking tdb since it is lockless)

Main process is triggered to re-process the PDU once the child process has finished.
Main daemon deletes the corresponding record in the tracking database, clears the HAVE_DELEGATIONS flag for the record and then proceeds to perform the migration as usual.

When receiving a ctdb_call without the flag we want all delegations to be revoked, so we must take care that the delegations are revoked unconditionally before we even check if we are already the DMASTER (in which case thie ctdb_call would normally just be  no-op  (*B below))



Recovery process changes
========================
A recovery implicitely clears/revokes any read only records and delegations from all databases.

During delegations of Read-Only locks, this is done in such way that delegated records will have a RSN smaller than the DMASTER. This guarantees that read-only copies always have a RSN that is smaller than the DMASTER.

During recoveries we do not need to take any special action other than always picking the copy of the record that has the highest RSN, which is what we already do today.

During the recovery process, we strip all flags off all records while writing the new contnent of the database during the PUSH_DB control. 

During processing of the PUSH_DB control and once the new database has been written we then also wipe the tracking database.

This makes changes to the recovery process minimal and nonintrusive.



Vacuuming process
=================
Vacuuming needs only minimal changes.


When vacuuming runs, it will do a fetch_lock to migrate any remote records back onto the LMASTER before the record can be purged. This will automatically force all delegations for that record to be revoked before the migration is copied back onto the LMASTER.
This handles the case where LMASTER is not the DMASTER for the record that will be purged.
The migration here does force any delegations to be revoked before the migration takes place.

Missing is the case when delegations exist and the LMASTER is also the DMASTER.
For this case we need to change the vacuuming to unconditionally always try to do a fetch_lock when HAVE_DELEGATIONS is set, even if the record is already stored locally. (*B)
This fetch lock will not cause any migrations by the ctdb daemon, but since it does not have the WANT_READONLY
this will still force the delegations to be revoked but no migrations trigger.


Traversal process
=================
Traversal process is changed to ignore any records with the HAVE_READONLY_LOCK


Forward/Backward Compatibility
==============================
Non-readonly locking daemons must be able to interoperate with readonly locking enabled daemons.

Non-readonly enabled daemons fetching records from Readonly enabled daemons:
Non-readonly enabled daemons do not know, and never set the WANT_READONLY flag so these daemons will always request a full migration for a full fetch-lock for all records. Thus a request from a non-readonly enabled daemon will always cause any existing delegations to be immediately revoked. Access will work but performance may be harmed since there will be a lot of revoking of delegations.

Readonly enabled dameons fetching records with WANT_READONLY from non-readonly enabled daemons:
Non-readonly enabled daemons ingore the WANT_READONLY flag and never return delegations. They always return a full record migration.
Full record migration is allowed by the protocol, even if the originator only requests the 'hint' WANT_READONLY,
so this access also interoperates between daemons with different capabilities.