IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
So that we automatically defragment the free list when freelist_size is called
(unless the database is read only).
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
This is intended to be called to reduce the fragmentation in the
freelist. This is to make up the deficiency of the freelist
to be not doubly linked. If the freelist were doubly linked,
we could easily avoid the creation of adjacent freelist entries.
But with the current singly linked list, it is only possible
to cheaply merge a new free record into a freelist entry on the left,
not on the right...
This can be called periodically, e.g. in the vacuuming process
of a ctdb cluster.
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Variant of check_merge_with_left_record() that reads the record
itself if necessary.
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Check whether the record left of a given freelist record is
also a freelist record, and if so, merge the two records.
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
by using early returns and better variable names,
and reducing indentation.
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
This adds optional support for locking based on
shared robust mutexes.
The caller can use the TDB_MUTEX_LOCKING flag
together with TDB_CLEAR_IF_FIRST after verifying
with tdb_runtime_check_for_robust_mutexes() that
it's supported by the current system.
The caller should be aware that using TDB_MUTEX_LOCKING
implies some limitations, e.g. it's not possible to
have multiple read chainlocks on a given hash chain
from multiple processes.
Note: that this doesn't make tdb thread safe!
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Pair-Programmed-With: Michael Adam <obnox@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
This makes it possible to have some extra headers before
the real tdb content starts in the file.
This will be used used e.g. to implement locking based on robust mutexes.
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Pair-Programmed-With: Michael Adam <obnox@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
This will allow to store a feature mask in the tdb header on disk,
so that openers can check if they can handle the features
other openers are using.
Pair-Programmed-With: Volker Lendecke <vl@samba.org>
Pair-Programmed-With: Michael Adam <obnox@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Mon May 12 21:07:04 CEST 2014 on sn-devel-104
When in tdb_store we re-use a dead record reactivated from the
target hash chain itself, we currently leave it in its place in
the chain. When we re-use a dead record from a different chain or
from the freelist instead, we insert it at the beginning of the
target chain.
This patch changes the behaviour to always newly store a
record at the beginning of the hash chain. This removes
a special case and hence simplifies the allocation code.
On the other hand side, it introduces two additioal tdb_ofs_write
calls for the in-chain-case.
Note the subtelty of the patch that by moving the case of the candidate
record's chain as new case "i=0" into the for loop, we also reverse the
order of the two steps in the for-loop body (non blocking freelist alloc
and searching for dead record in a chain) in order to keep the overall
order of execution identical.
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Wed Apr 9 10:37:08 CEST 2014 on sn-devel-104
In a metadata-intensive benchmark we have seen the locking.tdb freelist to be
one of the central contention points. This patch removes most of the contention
on the freelist. Ages ago we already reduced freelist contention by using the
even much older DEAD records: If TDB_VOLATILE is set, don't directly put
deleted records on the freelist, but just mark a few of them just as DEAD. The
next new record can them re-use that space without consulting the freelist.
This patch builds upon the DEAD records: If we need space and the freelist is
busy, instead of doing a blocking wait on the freelist, start looking into
other chains for DEAD records and steal them from there. This way every hash
chain becomes a small freelist. Just wander around the hash chains as long as
the freelist is still busy.
With this patch and the tdb mutex patch (following hopefully some time soon)
you can see a heavily busy clustered smbd run without locking.tdb futex
syscalls.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Will be used soon to unlink a dead record from a chain
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
This aligns the tdb_find_dead API with the tdb_allocate API and thus makes it a
bit easier to understand, at least for me.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Hash chains are (or can be made) short enough that a full search for the
best-fitting dead record is feasible. The freelist can become much longer,
there we don't do the full search but accept records which are too large.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
If the freelist is heavily contended, we should avoid accessing it
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
tdb_purge_dead can change the next pointer of "rec" if we purge the record
right behind the current record to be deleted. Just overwrite the magic,
not the whole record with stale data.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
This makes them more efficient due to better distribution
of keys across hash chains.
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Feb 15 08:26:07 CET 2014 on sn-devel-104
by using the same variable as hash as in the lock.
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Feb 15 03:21:07 CET 2014 on sn-devel-104
Make the lock/unlock bracket more obvious by extracting
locking (and finding) from the special cases to the top
of the function. This also lets us take lock and find
the record outside the special case branches (use dead
records or not).
There is a small semantic change implied:
In the dead records case, the record to delete is looked
up before the current dead records are potentially purged.
Hence, if the record to delete is not found, the dead
records are also not purge. This does not make a big
difference though, because purging is only delayed until
directly befor the next record to delete is in fact found.
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
In normal operations we have at most 3 entries in this array. Don't
bother with shrinking.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Sat Dec 14 13:19:47 CET 2013 on sn-devel-104
Failing to do so will result in corrupt tdbs: We will overwrite
the hash chain pointers with 0x42424242.
Pair-Programmed-With: Volker Lendecke <vl@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Rusty Russell <rusty@rustcorp.com.au>
We round up at maximun to a new size of 4GB,
but still return at least the given size.
The caller has to deal with ENOSPC itself.
Pair-Programmed-With: Volker Lendecke <vl@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Rusty Russell <rusty@rustcorp.com.au>
We can have offsets > 2G, so use unsigned values. Fixes other prints to be
native types rather than casts, too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue May 28 11:22:14 CEST 2013 on sn-devel-104
makes it possible to easily determine if the tdb under examination
uses jenkins hash or not
Signed-off-by: Christian Ambach <ambi@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
The "else" keywords are not necessary here, we return in the preceding
if clause
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Mar 5 14:00:47 CET 2013 on sn-devel-104
realloc(NULL, ...) is equivalent to malloc. We are already using this
realloc property for tdb->lockrecs. It should not make any difference
in speed, it just makes for a little simpler code.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Feb 19 17:30:13 CET 2013 on sn-devel-104
The transaction code uses tdb_alrecord_lock/upgrade, so it should also
use the tdb_allrecord_unlock function just for symmetry reasons
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
For the mutex code we will have to lock the hashchain and the record
lock area independently. So we will have to call the loop twice. And,
it's a small refactoring for the better anyway I think.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
All arguments but the cmd are the same. To me this looks a bit better
and saves some bytes in the object code.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Simo Sorce <idra@samba.org>
Autobuild-Date(master): Sat Feb 16 17:13:32 CET 2013 on sn-devel-104
header.hash_size was the only thing we ever referenced outside of
tdb_open_ex and its direct callees. So this shrinks the tdb_context by
164 bytes.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Feb 5 13:18:28 CET 2013 on sn-devel-104
These functions are deliberately left without prototypes according to
3fdeaa399, but without prototypes we get warnings.
Reviewed-by: Rusty Russell <rusty@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Mon Jan 7 11:20:19 CET 2013 on sn-devel-104
I had to ask git blame to find why we have to do it here...
Reviewed-by: Rusty Russell <rusty@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Fri Dec 21 13:54:39 CET 2012 on sn-devel-104
We usually "goto fail" on every error and then in normal flow set the
return variable to success. This patch removes a comment which from my
point of view is now obsolete. It violates the {} rule from README.Coding
here in favor of the style used in this function.
Reviewed-by: Rusty Russell <rusty@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
methods->tdb_write expects data in on-disk format. For reading that
record, methods->tdb_read() has taken care of the on-disk to in-memory
representation according to the DOCONV() flag passed down. tdb_rec_write()
is a wrapper around methods->tdb_write just doing the CONVERT() on the
way to disk.
Reviewed-by: Rusty Russell <rusty@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
When winbind is restarted, there is a potential crash in tdb. Following
situation: We are in a cluster with ctdb. A winbind child hangs
in a request to the DC. Cluster monitoring decides the node has a
problem. Cluster monitoring decides to kill ctdbd. winbind child
still hangs in a RPC request. winbind parent figures that ctdb is
dead and immediately commits suicide. winbind parent is restarted by
cluster management, overwriting gencache.tdb with CLEAR_IF_FIRST. The
CLEAR_IF_FIRST logic as implemented now will not see that a child still
has the tdb open, only the parent holds the ACTIVE_LOCK due to performance
reasons. During the CLEAR_IF_FIRST logic is done, there is a very small
window where we ftruncate(tfd, 0) the file and re-write a proper header
without a lock. When during this small window the winbind child comes
back, wanting to store something into gencache.tdb, that winbind child
will crash with a SIGBUS.
Sounds unlikely? See:
[2012/09/29 07:02:31.871607, 0] lib/util.c:1183(smb_panic)
PANIC (pid 1814517): internal error
[2012/09/29 07:02:31.877596, 0] lib/util.c:1287(log_stack_trace)
BACKTRACE: 35 stack frames:
#0 winbindd(log_stack_trace+0x1a) [0x7feb7d4ca18a]
#1 winbindd(smb_panic+0x2b) [0x7feb7d4ca25b]
#2 winbindd(+0x1a3cc4) [0x7feb7d4bacc4]
#3 /lib64/libc.so.6(+0x32900) [0x7feb7a929900]
#4 /lib64/libc.so.6(memcpy+0x35) [0x7feb7a97f355]
#5 /usr/lib64/libtdb.so.1(+0x6e76) [0x7feb7b0b0e76]
#6 /usr/lib64/libtdb.so.1(+0x3d37) [0x7feb7b0add37]
#7 /usr/lib64/libtdb.so.1(+0x863d) [0x7feb7b0b263d]
#8 /usr/lib64/libtdb.so.1(+0x8700) [0x7feb7b0b2700]
#9 /usr/lib64/libtdb.so.1(+0x2505) [0x7feb7b0ac505]
#10 /usr/lib64/libtdb.so.1(+0x25b7) [0x7feb7b0ac5b7]
#11 /usr/lib64/libtdb.so.1(tdb_fetch+0x13) [0x7feb7b0ac633]
#12 winbindd(gencache_set_data_blob+0x259) [0x7feb7d4d8449]
#13 winbindd(gencache_set+0x53) [0x7feb7d4d85b3]
#14 winbindd(gencache_del+0x5e) [0x7feb7d4d879e]
#15 winbindd(saf_delete+0x93) [0x7feb7d54b693]
#16 winbindd(+0xe507e) [0x7feb7d3fc07e]
#17 winbindd(+0xe85e5) [0x7feb7d3ff5e5]
#18 winbindd(+0xe65be) [0x7feb7d3fd5be]
#19 winbindd(+0xe7562) [0x7feb7d3fe562]
#20 winbindd(init_dc_connection+0x2e) [0x7feb7d3fe5be]
#21 winbindd(+0xe75d9) [0x7feb7d3fe5d9]
#22 winbindd(cm_connect_netlogon+0x58) [0x7feb7d3fe658]
#23 winbindd(_wbint_PingDc+0x61) [0x7feb7d410991]
#24 winbindd(+0x103175) [0x7feb7d41a175]
#25 winbindd(winbindd_dual_ndrcmd+0xb7) [0x7feb7d4107d7]
#26 winbindd(+0xf8609) [0x7feb7d40f609]
#27 winbindd(+0xf9075) [0x7feb7d410075]
#28 winbindd(tevent_common_loop_immediate+0xe8) [0x7feb7d4db198]
#29 winbindd(run_events_poll+0x3c) [0x7feb7d4d93fc]
#30 winbindd(+0x1c2b52) [0x7feb7d4d9b52]
#31 winbindd(_tevent_loop_once+0x90) [0x7feb7d4d9f60]
#32 winbindd(main+0x7b3) [0x7feb7d3e7aa3]
#33 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7feb7a915cdd]
#34 winbindd(+0xce2a9) [0x7feb7d3e52a9]
This is in a winbind child, logfiles surrounding indicate the parent
was restarted.
This patch takes all chain locks around the CLEAR_IF_FIRST introduced
tdb_new_database.
When probing for a size change (eg. just before tdb_expand, tdb_check,
tdb_rescue) we call tdb_oob(tdb, tdb->map_size, 1, 1). Unfortunately
this does nothing if the tdb has actually shrunk, which as Volker
demonstrated, can actually happen if a "longlived" parent crashes.
So move the map/update size/remap before the limit check.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(As suggested by Stefan Metzmacher, based on the change to ntdb.)
Since commit ec96ea690e, we handle the case
where a process dies during a transaction commit. Unfortunately, TDB_NOSYNC
means this no longer works, as it disables the recovery area as well as the
actual msync/fsync. We should do everything except the syncs.
This also means we can do a complete test with $TDB_NO_FSYNC set; just
to get more complete coverage, we disable it explicitly for one test
(where we override the actual sync calls anyway).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This patch adds two lock functions used by CTDB to perform asynchronous
locking. These functions do not actually perform any fcntl operations,
but only increment internal counters.
- tdb_transaction_write_lock_mark()
- tdb_transaction_write_lock_unmark()
It also exposes two internal functions
- tdb_lock_nonblock()
- tdb_unlock()
These functions are NOT exposed in include/tdb.h to prevent any further
uses of these functions. If you ever need to use these functions, consider
using tdb2.
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
We unmap the tdb on expand, the remap. But when we have INCOHERENT_MMAP
(ie. OpenBSD) and we're inside a transaction, doing the expand can mean
we need to read from the database to partially fill a transaction block.
This fails, because if mmap is incoherent we never allow accessing the
database via read/write.
The solution is not to unmap and remap until we've actually written the
padding at the end of the file.
Reported-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Fri Mar 23 02:53:15 CET 2012 on sn-devel-104
This comment appears in two places in the code (commit
4c6a8273c6 from 2001):
/*
* We must ensure the file is unmapped before doing this
* to ensure consistency with systems like OpenBSD where
* writes and mmaps are not consistent.
*/
But this doesn't help, because if one process is using mmap and another
using pwrite, we get incoherent results. As demonstrated by OpenBSD's
failure on the tdb unit tests.
Rather than disable mmap on OpenBSD, we test for this issue and force mmap
to be enabled. This means that we will fail on very large TDBs on 32-bit
systems, but it's better than the horrendous performance penalty on every
OpenBSD system.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The most convenient way to write unit tests in C is to directly
#include the C files (CCAN uses this, for example). That works quite
well, but it means that tdb_private.h now needs to be protected
against multiple inclusions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We allocate a new recovery area by expanding the file. But if the
recovery area is already at the end of file (as shown in at least one
client case), we can simply expand the record, rather than freeing it
and creating a new one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Wed Dec 21 06:25:40 CET 2011 on sn-devel-104
If we're expanding because the current recovery area is too small, we
expand only the amount we need. This can quickly lead to exponential
growth when we have a slowly-expanding record (hence a
slowly-expanding transaction size).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I came across a tdb which had wrapped to 4G + 4K, and the contents had been
destroyed by processes which thought it only 4k long. Fix this by checking
on open, and making tdb_oob() check for wrap itself.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Mon Dec 19 07:52:01 CET 2011 on sn-devel-104
TDB2 testing revealed that tdb1 doesn't do this. It's minor, but fix it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Tue Aug 16 10:47:41 CEST 2011 on sn-devel-104
Andrew Bartlett complained that valgrind needs --partial-loads-ok=yes otherwise
the Jenkins hash makes it complain.
My benchmarking here revealed that at least with modern gcc (4.5) and CPU
(Intel i5 32 bit) there's no measurable performance penalty for the
"correct" code, so rip out the optimized one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Wed Jun 8 11:05:47 CEST 2011 on sn-devel-104
If it's really the recovery area, we can trust the rec_len field, and
don't have to go groping for bitpatterns.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Tue Apr 19 14:15:22 CEST 2011 on sn-devel-104
ldb can create huge records when saving indexes.
Limit the tdb expansion to avoid consuming a lot of memory for
no good reason if the record being saved is huge.
tdb_repack() is expensive and consumes memory, so we can spend some
effort to see if it's worthwhile. In particular, tdbbackup doesn't
need to repack: it started with an empty database!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
tdb_name() might be used within the given log function,
which might be called from within tdb_open_ex().
metze
Autobuild-User: Stefan Metzmacher <metze@samba.org>
Autobuild-Date: Fri Nov 12 11:22:21 UTC 2010 on sn-devel-104
This flag to tdb_open/tdb_open_ex effects creation of a new database:
1) Uses the Jenkins lookup3 hash instead of the old gdbm hash if none is
specified,
2) Places a non-zero field in header->rwlocks, so older versions of TDB will
refuse to open it.
This means that the caller (ie Samba) can set this flag to safely
change the hash function. Versions of TDB from this one on will either
use the correct hash or refuse to open (if a different hash is specified).
Older TDB versions will see the nonzero rwlocks field and refuse to open
it under any conditions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If the caller to tdb_open_ex() doesn't specify a hash, and tdb_old_hash
doesn't match, try tdb_jenkins_hash.
This was Metze's idea: it makes life simpler, especially with the upcoming
TDB_INCOMPATIBLE_HASH flag.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a better hash than the default: shipping it with tdb makes it easy
for callers to use it as the hash by passing it to tdb_open_ex().
This version taken from CCAN and modified, which took it from
http://www.burtleburtle.net/bob/c/lookup3.c.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is Stefan Metzmacher <metze@samba.org>'s patch with minor changes:
1) Use the TDB_MAGIC constant so both hashes aren't of strings.
2) Check the hash in tdb_check (paranoia, really).
3) Additional check in the (unlikely!) case where both examples hash to 0.
4) Cosmetic changes to var names and complaint message.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Commit bc1c82ea13 "Fix tdb_check() to work with read-only tdb databases."
claimed to do this, but tdb_lockall_read() fails on read-only databases.
Also make sure we can still do tdb_check() inside a transaction (weird,
but we previously allowed it so don't break the API).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can end up with dead areas when we die during transaction commit;
tdb_check() fails on such a (valid) database.
This is particularly noticable now we no longer truncate on recovery;
if the recovery area was at the end of the file we used to remove it
that way.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We saw tdb_lockall() take 71 seconds under heavy load; this is because Linux
(at least) doesn't prevent new small locks being obtained while we're waiting
for a big log.
The workaround is to do divide and conquer using non-blocking chainlocks: if
we get down to a single chain we block. Using a simple test program where
children did "hold lock for 100ms, sleep for 1 second" the time to do
tdb_lockall() dropped signifiantly. There are ln(hashsize) locks taken in
the contended case, but that's slow anyway.
More analysis is given in my blog at http://rusty.ozlabs.org/?p=120
This may also help transactions, though in that case it's the initial
read lock which uses this gradual locking routine; the update-to-write-lock
code is separate and still tries to update in one go.
Even though ABI doesn't change, minor version bumped so behavior change
can be easily detected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Commit 207a213c/24fed55d purported to fix the problem of signals during
tdb_new_database (which could cause a spurious short write, hence a failure).
However, the code is wrong: newdb+written is not correct.
Fix this by introducing a general tdb_write_all() and using it here and in
the tracing code.
Cc: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now use -fvisibilty=hidden to hide symbols from outside the tdb
shared library.
This also moved tdb_transaction_recover() into the tdb_private.h
header, as it should never have been a public API. For that reason we
are changing the version number. We're only doing a minor version
increment as it is extremely unlikely that anyone was actually using
tdb_transaction_recover() as its locking requirements were rather
unusual.
Pair-Programmed-With: Rusty Russell <rusty@samba.org>
tdb transactions were designed to be robust against the machine
powering off, but interestingly were never designed to handle the case
where an administrator kill -9's a process during commit. Because
recovery is only done on tdb_open, processes with the tdb already
mapped will simply use it despite it being corrupt and needing
recovery.
The solution to this is to check for recovery every time we grab a
data lock: we could have gained the lock because a process just died.
This has no measurable cost: here is the time for tdbtorture -s 0 -n 1
-l 10000:
Before:
2.75 2.50 2.81 3.19 2.91 2.53 2.72 2.50 2.78 2.77 = Avg 2.75
After:
2.81 2.57 3.42 2.49 3.02 2.49 2.84 2.48 2.80 2.43 = Avg 2.74
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The current recovery code truncates the tdb file on recovery. This is
fine if recovery is only done on first open, but is a really bad idea
as we move to allowing recovery on "live" databases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now the transaction code uses the standard allrecord lock, that stops
us from trying to grab any per-record locks anyway. We don't need to
have special noop lock ops for transactions.
This is a nice simplification: if you see brlock, you know it's really
going to grab a lock.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
tdb_release_extra_locks() is too general: it carefully skips over the
transaction lock, even though the only caller then drops it. Change
this, and rename it to show it's clearly transaction-specific.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now the transaction allrecord lock is the standard one, and thus is cleaned
in tdb_release_extra_locks(), _tdb_transaction_cancel() doesn't need to
know what type it is.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Centralize locking of all chains of the tdb; rename _tdb_lockall to
tdb_allrecord_lock and _tdb_unlockall to tdb_allrecord_unlock, and
tdb_brlock_upgrade to tdb_allrecord_upgrade.
Then we use this in the transaction code. Unfortunately, if the transaction
code records that it has grabbed the allrecord lock read-only, write locks
will fail, so we treat this upgradable lock as a write lock, and mark it
as upgradable using the otherwise-unused offset field.
One subtlety: now the transaction code is using the allrecord_lock, the
tdb_release_extra_locks() function drops it for us, so we no longer need
to do it manually in _tdb_transaction_cancel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Records themselves get (read) locked by the traversal code against delete.
Interestingly, this locking isn't done when the allrecord lock has been
taken, though the allrecord lock until recently didn't cover the actual
records (it now goes to end of file).
The write record lock, grabbed by the delete code, is not suppressed
by the allrecord lock. This is now bad: it causes us to punch a hole
in the allrecord lock when we release the write record lock. Make this
consistent: *no* record locks of any kind when the allrecord lock is
taken.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were previously inconsistent with our "global" lock: the
transaction code grabbed it from FREELIST_TOP to end of file, and the
rest of the code grabbed it from FREELIST_TOP to end of the hash
chains. Change it to always grab to end of file for simplicity and
so we can merge the two.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was redundant before this patch series: it mirrored num_lockrecs
exactly. It still does.
Also, skip useless branch when locks == 1: unconditional assignment is
cheaper anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is pure overhead, but it centralizes the locking. Realloc (esp. as
most implementations are lazy) is fast compared to the fnctl anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Use our newly-generic nested lock tracking for the active lock.
Note that the tdb_have_extra_locks() and tdb_release_extra_locks()
functions have to skip over this lock now it is tracked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This never nests, so it's overkill, but it centralizes the locking into
lock.c and removes the ugly flag in the transaction code to track whether
we have the lock or not.
Note that we have a temporary hack so this places a real lock, despite
the fact that we are in a transaction.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than a boutique lock and a separate nest count, use our
newly-generic nested lock tracking for the transaction lock.
Note that the tdb_have_extra_locks() and tdb_release_extra_locks()
functions have to skip over this lock now it is tracked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Factor out two loops which find locks; we are going to introduce a couple
more so a helper makes sense.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Move locking intelligence back into lock.c, rather than open-coding the
lock release in transaction.c.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In many places we check whether locks are held: add a helper to do this.
The _tdb_lockall() case has already checked for the allrecord lock, so
the extra work done by tdb_have_extra_locks() is merely redundant.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
tdb_transaction_lock() and tdb_transaction_unlock() do nothing if we
hold the allrecord lock. However, the two locks don't overlap, so
this is wrong.
This simplification makes the transaction lock a straight-forward nested
lock.
There are two callers for these functions:
1) The transaction code, which already makes sure the allrecord_lock
isn't held.
2) The traverse code, which wants to stop transactions whether it has the
allrecord lock or not. There have been deadlocks here before, however
this should not bring them back (I hope!)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Because fcntl locks don't nest, we track them in the tdb->lockrecs array
and only place/release them when the count goes to 1/0. We only do this
for record locks, so we simply place the list number (or -1 for the free
list) in the structure.
To generalize this:
1) Put the offset rather than list number in struct tdb_lock_type.
2) Rename _tdb_lock() to tdb_nest_lock, make it non-static and move the
allrecord check out to the callers (except the mark case which doesn't
care).
3) Rename _tdb_unlock() to tdb_nest_unlock(), make it non-static and
move the allrecord out to the callers (except mark again).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The word global is overloaded in tdb. The global_lock inside struct
tdb_context is used to indicate we hold a lock across all the chains.
Rename it to allrecord_lock.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The word global is overloaded in tdb. The GLOBAL_LOCK offset is used at
open time to serialize initialization (and by the transaction code to block
open).
Rename it to OPEN_LOCK.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now tdb_open() calls tdb_transaction_cancel() instead of
_tdb_transaction_cancel, we can make it static.
Signed-off-by: Rusty Russell<rusty@rustcorp.com.au>
This is taken from the CCAN code base: rather than using tdb_brlock for
locking and unlocking, we split it into brlock and brunlock functions.
For extra debugging information, brunlock says what kind of lock it is
unlocking (even though fnctl locks don't need this). This requires an
extra argument to tdb_transaction_unlock() so we know whether the
lock was upgraded to a write lock or not.
We also use a "flags" argument tdb_brlock:
1) TDB_LOCK_NOWAIT replaces lck_type = F_SETLK (vs F_SETLKW).
2) TDB_LOCK_MARK_ONLY replaces setting TDB_MARK_LOCK bit in ltype.
3) TDB_LOCK_PROBE replaces the "probe" argument.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If a process (or the machine) dies after just after writing the
recovery head (pointing at the end of file), the recovery record will filled
with 0x42. This will not invoke a recovery on open, since rec.magic
!= TDB_RECOVERY_MAGIC.
Unfortunately, the first transaction commit will happily reuse that
area: tdb_recovery_allocate() doesn't check the magic. The recovery
record has length 0x42424242, and it writes that back into the
now-valid-looking transaction header) for the next comer (which
happens to be tdb_wipe_all in my tests).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There was a bug in tdb where the
tdb_brlock(tdb, GLOBAL_LOCK, F_UNLCK, F_SETLKW, 0, 1);
(ending the transaction-"mutex") was done before the
/* remove the recovery marker */
This means that when a transaction is committed there is a window where another
opener of the file sees the transaction marker while the transaction committer
is still fully functional and working on it. This led to transaction being
rolled back by that second opener of the file while transaction_commit() gave
no error to the caller.
This patch moves the F_UNLCK to after the recovery marker was removed, closing
this window.
We need to keep TDB_ALLOW_NESTING as default behavior,
so that existing code continues to work.
However we may change the default together with a major version
number change in future.
metze
Make the default be that transaction is not allowed and any attempt to create a nested transaction will fail with TDB_ERR_NESTING.
If an application can cope with transaction nesting and the implicit
semantics of tdb_transaction_commit(), it can enable transaction nesting
by using the TDB_ALLOW_NESTING flag.
(cherry picked from ctdb commit 3e49e41c21eb8c53084aa8cc7fd3557bdd8eb7b6)
Signed-off-by: Stefan Metzmacher <metze@samba.org>
So that erroneous double tdb_close() calls do not try to close() same
fd again. This is like SAFE_FREE() but for fd.
Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
