samba-mirror/ctdb/doc/readonlyrecords.txt

Read-Only locks in CTDB
=======================

Problem
=======
CTDB currently only supports exclusive Read-Write locks for clients(samba) accessing the
TDB databases.
This mostly works well but when very many clients are accessing the same file,
at the same time, this causes the exclusive lock as well as the record itself to
rapidly bounce between nodes and acts as a scalability limitation.

This primarily affects locking.tdb and brlock.tdb, two databases where record access is 
read-mostly and where writes are semi-rare.

For the common case, if CTDB provided shared non-exclusive Read-Only lock semantics
this would greatly improve scaling for these workloads.


Desired properties
==================
We can not make backward incompatible changes the ctdb_ltdb header for the records.

A Read-Only lock enabled ctdb demon must be able to interoperate with a non-Read-Only
lock enbled daemon.

Getting a Read-Only lock should not be slower than getting a Read-Write lock.

When revoking Read-Only locks for a record, this should involve only those nodes that
currently hold a Read-Only lock and should avoid broadcasting opportunistic revocations.
(must track which nodes are delegated to)

When a Read-Write lock is requested, if there are Read-Only locks delegated to other
nodes, the DMASTER will defer the record migration until all read-only locks are first
revoked (synchronous revoke).

Due to the cost of revoking Read-Only locks has on getting a Read-Write lock, the
implementation should try to avoid creating Read-Only locks unless it has indication
that there is contention. This may mean that even if client requests a Read-Only lock
we might still provide a full Read-Write lock in order to avoid the cost of revoking
the locks in some cases.

Read-Only locks require additional state to be stored in a separate database, containing
information about which nodes have have been delegated Read-Only locks.
This database should be kept at minimal size.

Read-Only locks should not significantly complicate the normal record
create/migration/deletion cycle for normal records.

Read-Only locks should not complicate the recovery process.

Read-Only locks should not complicate the vacuuming process.

We should avoid forking new child processes as far as possible from the main daemon.

Client-side implementation, samba, libctdb, others, should have minimal impact when
Read-Only locks are implemented.
Client-side implementation must be possible with only minor conditionals added to the
existing lock-check-fetch-unlock loop that clients use today for Read-Write locks. So
that clients only need one single loop that can handle both Read-Write locking as well
as Read-Only locking. Clients should not need two nearly identical loops.


Implementation
==============

Four new flags are allocated in the ctdb_ltdb record header.
HAVE_DELEGATIONS, HAVE_READONLY_LOCK, REVOKING_READONLY and REVOKE_COMPLETE

HAVE_DELEGATIONS is a flag that can only be set on the node that is currently the
DMASTER for the record. When set, this flag indicates that there are Read-Only locks
delegated to other nodes in the cluster for this record.

HAVE_READONLY is a flag that is only set on nodes that are NOT the DMASTER for the
record. If set this flag indicates that this record contains an up-to-date Read-Only
version of this record. A client that only needs to read, but not to write, the record
can safely use the content of this record as is regardless of the value of the DMASTER
field of the record.

REVOKING_READONLY is a flag that is used while a set of read only delegations are being
revoked.
This flag is only set when HAVE_DELEGATIONS is also set, and is cleared at the same time
as HAVE_DELEGATIONS is cleared.
Normal operations is that first the HAVE_DELEGATIONS flag is set when the first
delegation is generated. When the delegations are about to be revoked, the
REVOKING_READONLY flag is set too.
Once all delegations are revoked, both flags are cleared at the same time.
While REVOKING_READONLY is set, any requests for the record, either normal request or
request for readonly will be deferred.
Deferred requests are linked on a list for deferred requests until the time that the
revokation is completed.
This flags is set by the main ctdb daemon when it starts revoking this record.

REVOKE_COMPLETE
The actual revoke of records is done by a child process, spawned from the main ctdb
daemon when it starts the process to revoke the records.
Once the child process has finished revoking all delegations it will set the flag
REVOKE_COMPLETE for this record to signal to the main daemon that the record has been
successfully revoked.
At this stage the child process will also trigger an event in the main daemon that
revoke is complete and that the main daemon should start re-processing all deferred
requests.



Once the revoke process is completed there will be at least one deferred request to
access this record. That is the initical call to for an exclusive fetch_lock() that
triggered the revoke process to be started.
In addition to this deferred request there may also be additional requests that have
also become deferred while the revoke was in process. These can be either exclusive
fetch_locks() or they can be readonly lock requests.
Once the revoke is completed the main daemon will reprocess all exclusive fetch_lock()
requests immediately and respond to these clients.
Any requests for readadonly lock requests will be deferred for an additional period of
time before they are re-processed.
This is to allow the client that needs a fetch_lock() to update the record to get some
time to access and work on the record without having to compete with the possibly
very many readonly requests.





The ctdb_db structure is expanded so that it contains one extra TDB database for each
normal, non-persistent database.
This new database is used for tracking delegations for the records.
A record in the normal database that has "HAVE_DELEGATION" set will always have a
corresponding record at the same key. This record contains the set of all nodes that
the record is delegated to.
This tracking database is lockless, using TDB_NOLOCK, and is only ever accessed by
the main ctdbd daemon.
The lockless nature and the fact that no other process ever access this TDB means we
are guaranteed non-blocking access to records in the tracking database.

The ctdb_call PDU is allocated with a new flag WANT_READONLY and possibly also a new
callid: CTDB_FETCH_WITH_HEADER_FUNC.
This new function returns not only the record, as CTDB_FETCH_FUNC does, but also
returns the full ctdb_ltdb record HEADER prepended to the record.
This function is optional, clients that do not care what the header is can continue
using just CTDB_FETCH_FUNC


This flag is used to requesting a read-only record from the DMASTER/LMASTER.
If the record does not yet exist, this is a returned as an error to the client and the
client will retry the request loop.

A new control is added to make remote nodes remove the HAVE_READONLY_LOCK from a record
and to invalidate any deferred readonly copies from the databases.



Client implementation
=====================
Clients today use a loop for record fetch lock that looks like this
    try_again:
        lock record in tdb

        if record does not exist in tdb,
            unlock record
            ask ctdb to migrate record onto the node
            goto try_again

        if record dmaster != this node pnn
            unlock record
            ask ctdb to migrate record onto the node
            goto try_again

    finished:

where we basically spin, until the record is migrated onto the node and we have managed
to pin it down.

This will change to instead to something like

    try_again:
        lock record in tdb

        if record does not exist in tdb,
            unlock record
            ask ctdb to migrate record onto the node
            goto try_again

        if record dmaster == current node pnn
            goto finished

        if read-only lock
            if HAVE_READONLY_LOCK or HAVE_DELEGATIONS is set
                goto finished
            else
                unlock record 
                ask ctdb for read-only copy (WANT_READONLY[|WITH_HEADER])
                if failed to get read-only copy (*A)
                    ask ctdb to migrate the record onto the node
                    goto try_again
                lock record in tdb
                goto finished

        unlock record
        ask ctdb to migrate record onto the node
        goto try_again

    finished:

If the record does not yet exist in the local TDB, we always perform a full fetch for a
Read-Write lock even if only a Read-Only lock was requested.
This means that for first access we always grab a Read-Write lock and thus upgrade any
requests for Read-Only locks into a Read-Write request.
This creates the record, migrates it onto the node and makes the local node become
the DMASTER for the record.

Future reference to this same record by the local samba daemons will still access/lock
the record locally without triggereing a Read-Only delegation to be created since the
record is already hosted on the local node as DMASTER.

Only if the record is contended, i.e. it has been created an migrated onto the node but
we are no longer the DMASTER for this record, only for this case will we create a
Read-Only delegation.
This heuristics provide a mechanism where we will not create Read-Only delegations until
we have some indication that the record may be contended.

This avoids creating and revoking Read-Only delegations when only a single client is
repeatedly accessing the same set of records.
This also aims to limit the size of the tracking tdb.


Server implementation
=====================
When receiving a ctdb_call with the WANT_READONLY flag:

If this is the LMASTER for the record and the record does not yet exist, LMASTER will
return an error back to the client (*A above) and the client will try to recover.
In particular, LMASTER will not create a new record for this case.

If this is the LMASTER for the record and the record exists, the PDU will be forwarded to
the DMASTER for the record.

If this node is not the DMASTER for this record, we forward the PDU back to the
LMASTER. Just as we always do today.

If this is the DMASTER for the record, we need to create a Read-Only delegation.
This is done by
     lock record
     increase the RSN by one for this record
     set the HAVE_DELEGATIONS flag for the record
     write the updated record to the TDB
     create/update the tracking TDB nd add this new node to the set of delegations
     send a modified copy of the record back to the requesting client.
         modifications are that RSN is decremented by one, so delegated records are "older" than on the DMASTER,
         it has HAVE_DELEGATIONS flag stripped off, and has HAVE_READONLY_LOCK added.
     unlock record

Important to note is that this does not trigger a record migration.


When receiving a ctdb_call without the WANT_READONLY flag:

If this is the DMASTER for the this might trigger a migration. If there exists
delegations we must first revoke these before allowing the Read-Write request from
proceeding. So,
IF the record has HAVE_DELEGATIONS set, we create a child process and defer processing
of this PDU until the child process has completed.

From the child process we will call out to all nodes that have delegations for this
record and tell them to invalidate this record by clearing the HAVE_READONLY_LOCK from
the record.
Once all delegated nodes respond back, the child process signals back to the main daemon
the revoke has completed. (child process may not access the tracking tdb since it is
lockless)

Main process is triggered to re-process the PDU once the child process has finished.
Main daemon deletes the corresponding record in the tracking database, clears the
HAVE_DELEGATIONS flag for the record and then proceeds to perform the migration as usual.

When receiving a ctdb_call without the flag we want all delegations to be revoked,
so we must take care that the delegations are revoked unconditionally before we even
check if we are already the DMASTER (in which case the ctdb_call would normally just
be  no-op  (*B below))



Recovery process changes
========================
A recovery implicitly clears/revokes any read only records and delegations from all
databases.

During delegations of Read-Only locks, this is done in such way that delegated records
will have a RSN smaller than the DMASTER. This guarantees that read-only copies always
have a RSN that is smaller than the DMASTER.

During recoveries we do not need to take any special action other than always picking
the copy of the record that has the highest RSN, which is what we already do today.

During the recovery process, we strip all flags off all records while writing the new
content of the database during the PUSH_DB control. 

During processing of the PUSH_DB control and once the new database has been written we
then also wipe the tracking database.

This makes changes to the recovery process minimal and nonintrusive.



Vacuuming process
=================
Vacuuming needs only minimal changes.


When vacuuming runs, it will do a fetch_lock to migrate any remote records back onto the
LMASTER before the record can be purged. This will automatically force all delegations
for that record to be revoked before the migration is copied back onto the LMASTER.
This handles the case where LMASTER is not the DMASTER for the record that will be
purged.
The migration in this case does force any delegations to be revoked before the
vacuuming takes place.

Missing is the case when delegations exist and the LMASTER is also the DMASTER.
For this case we need to change the vacuuming to unconditionally always try to do a
fetch_lock when HAVE_DELEGATIONS is set, even if the record is already stored locally.
(*B)
This fetch lock will not cause any migrations by the ctdb daemon, but since it does
not have the WANT_READONLY this will still force the delegations to be revoked but no
migration will trigger.


Traversal process
=================
Traversal process is changed to ignore any records with the HAVE_READONLY_LOCK


Forward/Backward Compatibility
==============================
Non-readonly locking daemons must be able to interoperate with readonly locking enabled daemons.

Non-readonly enabled daemons fetching records from Readonly enabled daemons:
Non-readonly enabled daemons do not know, and never set the WANT_READONLY flag so these daemons will always request a full migration for a full fetch-lock for all records. Thus a request from a non-readonly enabled daemon will always cause any existing delegations to be immediately revoked. Access will work but performance may be harmed since there will be a lot of revoking of delegations.

Readonly enabled daemons fetching records with WANT_READONLY from non-readonly enabled daemons:
Non-readonly enabled daemons ignore the WANT_READONLY flag and never return delegations. They always return a full record migration.
Full record migration is allowed by the protocol, even if the originator only requests the 'hint' WANT_READONLY,
so this access also interoperates between daemons with different capabilities.