1
0
mirror of https://github.com/samba-team/samba.git synced 2024-12-23 17:34:34 +03:00
Commit Graph

275 Commits

Author SHA1 Message Date
Volker Lendecke
c7211882a7 tdb: Truncate the file after expand failure
Without this it's very easy to create virtually huge files: ftruncate expands a
file, the pwrites fail with ENOSPC, thus the write fails. The next writer runs
into the same situation, and ftruncate-expands the file even further. tdb_check
will then spend ages reading the 4GB of zeros byte by byte.

Here we hold the freelist lock or are inside a transaction, so it is safe to
cut the file again. Nobody can have used the space that we have tried to
allocate, so we can't have any stray pointers corrupting the database.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-08-24 01:46:08 +02:00
Volker Lendecke
3e7efbfb36 tdb: Protect against EINTR
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-08-24 01:46:08 +02:00
Volker Lendecke
c66b214537 tdb: Fix a typo
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-08-24 01:46:08 +02:00
Ralph Boehme
a43affd4c5 tdb: fix tbdtool list freelist output
Due to the non-fixable bug in the BUCKET macro tdbtool list printed some
other hash chainlist, not the freelist.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=12888

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-08-22 23:32:13 +02:00
Ralph Boehme
d2c78695c6 tdb: document hashtable bucket offset calculation madness
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-08-22 23:32:13 +02:00
Ralph Boehme
f2b7bc23df tbd: BUCKET(-1) returns wrong offset because tdb->hash_size is an unsigned int
The following C program demonstrates the issue:

  #include <stdio.h>
  #include <stdlib.h>
  #include <stdarg.h>
  #include <stdbool.h>
  #include <string.h>
  #include <unistd.h>
  #include <errno.h>
  #include <dirent.h>
  #include <sys/types.h>

  int main(int argc, char **argv)
  {
      int hash = -1;
      int tsize_signed = 10;
      unsigned int tsize_unsigned = 10;
      int bucket;

  #define BUCKET(hash, tsize) ((hash) % (tsize))

      bucket = BUCKET(hash, tsize_unsigned);
      printf("hash [%d] tsize [%d] bucket [%d]\n", hash, tsize_unsigned, bucket);

      bucket = BUCKET(hash, tsize_signed);
      printf("hash [%d] tsize [%d] bucket [%d]\n", hash, tsize_signed, bucket);

      return 0;
  }

Output:

$ ./tmp
hash [-1] tsize [10] bucket [5]
hash [-1] tsize [10] bucket [-1]

The first version is what the current BUCKET() macro does. As a result
we lock the hashtable chain in bucket 5, NOT the freelist.

-1 is sign converted to an unsigned int 4294967295 and

  4294967295 % 10 = 5.

As all callers will lock the same wrong list consistently locking is
still consistent.

Stumpled across this when looking at the output of `tdbtool DB list`
which always printed some other hashchain and not the freelist.

The freelist bucket offset computation doesn't use the BUCKET macro in
freelist.c (directly or indirectly) when working on the freelist, it
just directly uses the FREELIST_TOP define, so this problem only affects
tdbtool list.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-08-22 23:32:13 +02:00
Ralph Boehme
a5a19fa5ef tdb: rename struct tdb_traverse_lock hash member to list
The variable stores the hashtable bucket, not the hash. No change in
behaviour.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-08-22 23:32:13 +02:00
Volker Lendecke
a90702fc5d tdb: Clarify the CLEAR_IF_FIRST locked logic
This is another level of indentation, but it took me a while staring at the
if-condition to find that "locked" was assigned the result of "==0", not the
return value of tdb_nest_lock().

Best viewed with "git show -b".

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-17 11:54:10 +02:00
Andreas Schneider
d7e60bc17e tdb: Do not allow to pass NULL as the buffer to transaction_write()
This fixes a GCC warning.

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Aug 10 02:26:09 CEST 2017 on sn-devel-144
2017-08-10 02:26:09 +02:00
Andreas Schneider
47bb27652e tdb: Write zero data using 8k buffer in transaction_expand_file()
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2017-08-09 22:34:17 +02:00
Volker Lendecke
f6a382fff8 tdb: Avoid NULL tdb_write
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-09 22:34:17 +02:00
Volker Lendecke
5c55c2563d tdb: Consistency check for tdb_storev
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-09 22:34:17 +02:00
Andrew Bartlett
32702a9745 tdb: Add new function tdb_transaction_active()
This will allow callers to avoid their own reference counting of transactions.

Additionally, this will always line up with the acutal transaction state, even
in the error cases where tdb can cancel the transaction

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
f80076fe43 tdb: Remove locking from tdb_traverse_read()
This restores the original intent of tdb_traverse_read() in
7dd31288a7

This is needed to avoid a deadlock with tdb_lockall() and the
transaction start, as ldb_tdb should take the allrecord lock during a
search (which calls tdb_traverse), and can otherwise deadlock against
a transaction starting in another process

We add a test to show that a transaction can now start while a read
traverse is in progress

This allows more operations to happen in parallel.  The blocking point
is moved to the prepare commit.

This in turn permits a roughly doubling of unindexed search
performance, because currently ldb_tdb omits to take the lock due to
an unrelated bug, but taking the allrecord lock triggers the
above-mentioned deadlock.

This behaviour was added in 251aaafe3a for
Solaris 10 in 2005. But the run-fcntl-deadlock test works also on Solaris 10,
see https://lists.samba.org/archive/samba-technical/2017-April/119876.html.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:18 +02:00
Andrew Bartlett
1fd8ec23c7 tdb: Improve debugging when the allrecord lock fails to upgrade
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-23 01:13:24 +02:00
Andrew Bartlett
3607826334 tdb: Improve debugging in _tdb_transaction_start
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-04-27 14:52:17 +02:00
Andrew Bartlett
1148e8f040 tdb: Improve debugging when the allrecord lock fails to upgrade
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-04-27 14:52:17 +02:00
Ralph Boehme
19b193ebc9 tdb: runtime check for robust mutexes may hang in threaded programs
The current runtime check for robust mutexes in
tdb_runtime_check_for_robust_mutexes() is not thread-safe.

When called in a multi-threaded program where any another thread doesn't
have SIGCHLD blocked, we may end up hung in sigsuspend() waiting for a
SIGCHLD of a child procecss and the signal was delivered to another
thread.

Revert to the previous behaviour of waiting for the child instead of
waiting for the SIGCHLD signal.

Ensure the pid we wait for is not reset to -1 in a toctou race with the
signal handler.

Check whether waitpid() returns ECHILD which can happen if the signal
handler is run by more then one thread in parallel (yes, this can
happen) or if tdb_robust_mutex_wait_for_child() and the signal handler
are racing.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=12593

Pair-programmed-with: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Ralph Boehme <slow@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-04-27 14:52:16 +02:00
Volker Lendecke
a6f1532d7f tdb: Do lock upgrades properly
When a process holds a readlock and wants to upgrade, this needs to be
reflected in the underlying lock. Without this, it is possible to cheat:
One process holds a readlock, and another process wants to write this
record. All the writer has to do is take a readonly lock on the key and
then do the store.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-04-10 20:44:21 +02:00
Volker Lendecke
97cafdcfaa tdb: Fix some signed/unsigned hickups
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-04-10 20:44:21 +02:00
Volker Lendecke
275d9fc7d9 tdb: Fix mutexes on FreeBSD
susv4 on mmap has the following snippet:

> The state of synchronization objects such as mutexes, semaphores,
> barriers, and conditional variables placed in shared memory mapped
> with MAP_SHARED becomes undefined when the last region in any process
> containing the synchronization object is unmapped.

This means we can't keep the mutex mmap area unmapped at any point
in time.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12455

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Nov 29 23:59:52 CET 2016 on sn-devel-144
2016-11-29 23:59:52 +01:00
Volker Lendecke
5ce95abf37 tdb: Only mmap the mutex area if not already mmap'ed
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12455

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2016-11-29 20:03:25 +01:00
Volker Lendecke
a2843cfd4d tdb: NULL out tdb->mutexes in tdb_mutex_munmap
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12455

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2016-11-29 20:03:25 +01:00
Volker Lendecke
c27c7d44fb tdb: Use tdb_storev in tdb_append
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2016-08-29 19:03:27 +02:00
Volker Lendecke
504b04b817 tdb: Add tdb_storev
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2016-08-29 19:03:27 +02:00
Volker Lendecke
647e61b0e8 tdb: Add tdb_trace_1plusn_rec_flag_ret
Needed for tdb_storev

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2016-08-29 19:03:27 +02:00
Volker Lendecke
37e644fc20 tdb: Vectorize _tdb_store
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2016-08-29 19:03:26 +02:00
Volker Lendecke
f6f4e5e6b5 tdb: Vectorize tdb_update_hash
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2016-08-29 19:03:26 +02:00
Volker Lendecke
6295080361 tdb: Allow _v variant in tdb_update_hash_cmp
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2016-08-29 19:03:26 +02:00
Volker Lendecke
babd5432ed tdb: Remove unnecessary checks
This has already been done in tdb_find()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2016-08-29 19:03:26 +02:00
Volker Lendecke
4913180aad tdb: Do an overflow check
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2016-08-29 19:03:26 +02:00
Volker Lendecke
6dc4e294a5 tdb: Fix a signed/unsigned hickup
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2016-08-29 19:03:26 +02:00
Volker Lendecke
91a17e5413 tdb: Don't malloc for every record in traverse
This gains a few percent in tdbbackup

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Fri Jul 15 00:52:00 CEST 2016 on sn-devel-144
2016-07-15 00:52:00 +02:00
Bob Campbell
7700ee810b tdb: avoid many fcntl calls when incrementing seqnum
Signed-off-by: Bob Campbell <bobcampbell@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sun Jul  3 18:11:30 CEST 2016 on sn-devel-144
2016-07-03 18:11:29 +02:00
Jeremy Allison
04967d6e88 s3: tdb: On some platforms pthread_mutex_trylock() returns EBUSY not EDEADLK.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Wed Jun 29 15:14:44 CEST 2016 on sn-devel-144
2016-06-29 15:14:44 +02:00
Volker Lendecke
e6ed803a3b tdb mutex check: Fix CID 1358473 Uninitialized scalar variable
This comes via a "goto cleanup" before suspend_mask is initialized

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Uri Simchoni <uri@samba.org>

Autobuild-User(master): Uri Simchoni <uri@samba.org>
Autobuild-Date(master): Tue Apr 12 11:39:35 CEST 2016 on sn-devel-144
2016-04-12 11:39:34 +02:00
Uri Simchoni
ef3d837040 tdb: rework cleanup logic in tdb_runtime_check_for_robust_mutexes()
The cleanup logic used six goto lables, at least I'm not able to make
sane modifications to such a beast.

By using state flags that track which objects are initialized and need
cleanup, we get rid of the goto labels. It comes at a cost though: you
have to be careful to correctly set the cleanup flags.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Uri Simchoni <uri@samba.org>
2016-04-11 15:22:26 +02:00
Ralph Boehme
a7a77e2f43 tdb: avoid a race condition when checking for robust mutexes
This fixes a race between calling waitpid() in two places (SIGCHLD the
signal handler and the rendezvous code when waiting for the child to
terminate), by

- blocking SIGCHLD before installing our signal handler

- in the rendezvous code call sigssuspend() which unblocks SIGCHLD and
  suspends the thread and waits for signal delivery

BUG: https://bugzilla.samba.org/show_bug.cgi?id=11808

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Uri Simchoni <uri@samba.org>

Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Tue Mar 29 16:04:19 CEST 2016 on sn-devel-144
2016-03-29 16:04:19 +02:00
Andrew Bartlett
dbd87b94aa tdb: Refuse to load a database with hash size 0
This just ensures we reject (rather than div-by-0) a corrupt
DB with a zero hash size.

Found with american fuzzy lop

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Dec 18 08:26:25 CET 2015 on sn-devel-144
2015-12-18 08:26:24 +01:00
Volker Lendecke
1061a9cafd tdb: Fix bug 11381, deadlock
This fixes a deadlock in tdb that is a bad interaction between tdb_lockall
and tdb_traverse. This deadlock condition has been around even before
tdb mutexes, it's just that the kernel fcntl EDEADLK detection protected
us from this ABBA lock condition to become a real deadlock stalling
processes. With tdb mutexes, this deadlock protection is gone, so we do
lock dead.

This patch glosses over this particular ABBA condition, making tdb with
mutexes behave the same as tdb without mutexes. Admittedly this is no
real fix, but it works around a real user's problem.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=11381
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2015-07-09 00:42:15 +02:00
Alexander Drozdov
e4fe0aff52 tdb: introduce tdb_chainlock_read_nonblock(), a nonblock variant of tdb_chainlock_read()
Signed-off-by: Alexander Drozdov <al.drozdov@gmail.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2015-04-28 21:28:18 +02:00
Jeremy Allison
05b61ea47d lib: tdb: Use sigaction when testing for robust mutexes.
Working fix that copes with oldact.sa_handler == NULL
if no handler initially set.

Fixes bug #11175 - Lots of winbindd zombie processes on Solaris platform.

https://bugzilla.samba.org/show_bug.cgi?id=11175

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>

Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Thu Mar 26 04:29:42 CET 2015 on sn-devel-104
2015-03-26 04:29:42 +01:00
Andreas Schneider
8c4d4f9e62 Revert "lib: tdb: Use sigaction when testing for robust mutexes."
This fails on Linux platforms with robust mutex support with the
following error:

tdb(/home/asn/workspace/projects/samba/git/st/nt4_dc/lockdir/gencache_notrans.tdb):
    tdb_mutex_open_ok[/home/asn/workspace/projects/samba/git/st/nt4_dc/lockdir/gencache_notrans.tdb]:
    Can use mutexes only with MUTEX_LOCKING or NOLOCK

We also see winbind is not able to start with this error message trying
to open the serverid.tdb.

This reverts commit d191436728.

Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Wed Mar 25 14:58:38 CET 2015 on sn-devel-104
2015-03-25 14:58:38 +01:00
Jeremy Allison
d191436728 lib: tdb: Use sigaction when testing for robust mutexes.
Fixes bug #11175 - Lots of winbindd zombie processes on Solaris platform.

https://bugzilla.samba.org/show_bug.cgi?id=11175

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Tue Mar 24 14:43:22 CET 2015 on sn-devel-104
2015-03-24 14:43:22 +01:00
Stefan Metzmacher
d0839af9d6 tdb: allow transactions on on tdb's with TDB_MUTEX_LOCKING
There's no real reason to disallow transactions as the
allrecord lock is also available with mutexes enabled.

E.g. ctdbd requires transactions also on non-persistent databases
opened with TDB_CLEAR_IF_FIRST and TDB_MUTEX_LOCKING.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=11004

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2014-12-19 09:20:06 +01:00
Volker Lendecke
42b2e5ca8c tdb: Fix tdb_runtime_check_for_robust_mutexes()
When using exit() instead of _exit(), the child will flush buffered stdout
(and other stdio) content that it inherited from the parent process. In
make test, this led to duplicate output from net registry which then
confused the blackbox selftest.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2014-11-26 16:43:04 +01:00
Stefan Metzmacher
c8d05e934e tdb: allow tdb_open_ex() with O_RDONLY of TDB_FEATURE_FLAG_MUTEX tdbs.
Bug: https://bugzilla.samba.org/show_bug.cgi?id=10781

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
2014-11-20 16:23:05 +01:00
Volker Lendecke
703ef59d00 tdb: Fix a comment
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Sep 30 23:08:07 CEST 2014 on sn-devel-104
2014-09-30 23:08:07 +02:00
Volker Lendecke
17998fc4cf tdb: Improve wording in a comment
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu Sep 25 00:59:16 CEST 2014 on sn-devel-104
2014-09-25 00:59:16 +02:00
Michael Adam
55ff3a3b91 tdb: defragment the freelist in tdb_allocate_from_freelist()
While we are traversing the freelist anyways, merge a record
with the left if it is also a free list record.

That partially makes up for the fragmentation introduced by
the lack of merging with right records in tdb_free().

Note there is a potential slight downside:
If the left record we merge the current record into was earlier
in the chain and has hence already been met in traverse,
then we can not use the enlarged record even if it might be
a new best fit.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Thu Jun 26 12:16:03 CEST 2014 on sn-devel-104
2014-06-26 12:16:03 +02:00
Michael Adam
56f9231c8e tdb: use tdb_freelist_merge_adjacent in tdb_freelist_size()
So that we automatically defragment the free list when freelist_size is called
(unless the database is read only).

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2014-06-26 10:00:11 +02:00
Michael Adam
843a8a5c7b tdb: add tdb_freelist_merge_adjacent()
This is intended to be called to reduce the fragmentation in the
freelist. This is to make up the deficiency of the freelist
to be not doubly linked. If the freelist were doubly linked,
we could easily avoid the creation of adjacent freelist entries.
But with the current singly linked list, it is only possible
to cheaply merge a new free record into a freelist entry on the left,
not on the right...

This can be called periodically, e.g. in the vacuuming process
of a ctdb cluster.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2014-06-26 10:00:11 +02:00
Michael Adam
73c439f581 tdb: add utility function check_merge_ptr_with_left_record()
Variant of check_merge_with_left_record() that reads the record
itself if necessary.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2014-06-26 10:00:11 +02:00
Michael Adam
4bec28bfa9 tdb: simplify tdb_free() using check_merge_with_left_record()
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2014-06-26 10:00:11 +02:00
Michael Adam
117807cd2d tdb: add utility function check_merge_with_left_record()
Check whether the record left of a given freelist record is
also a freelist record, and if so, merge the two records.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2014-06-26 10:00:11 +02:00
Michael Adam
66f3330be8 tdb: improve comments for tdb_free().
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2014-06-26 10:00:11 +02:00
Michael Adam
8be5c8a6db tdb: factor merge_with_left_record() out of tdb_free()
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2014-06-26 10:00:11 +02:00
Michael Adam
63673aea9f tdb: fix debug message in tdb_free()
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2014-06-26 10:00:11 +02:00
Michael Adam
08a76aabe9 tdb: reduce indentation in tdb_free() for merging left
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2014-06-26 10:00:11 +02:00
Michael Adam
87ac4ac523 tdb: increase readability of read_record_on_left()
by using early returns and better variable names,
and reducing indentation.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2014-06-26 10:00:11 +02:00
Michael Adam
f5a777a36c tdb: factor read_record_on_left() out of tdb_free()
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2014-06-26 10:00:11 +02:00
Volker Lendecke
db5bda56bf tdb: add TDB_MUTEX_LOCKING support
This adds optional support for locking based on
shared robust mutexes.

The caller can use the TDB_MUTEX_LOCKING flag
together with TDB_CLEAR_IF_FIRST after verifying
with tdb_runtime_check_for_robust_mutexes() that
it's supported by the current system.

The caller should be aware that using TDB_MUTEX_LOCKING
implies some limitations, e.g. it's not possible to
have multiple read chainlocks on a given hash chain
from multiple processes.

Note: that this doesn't make tdb thread safe!

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Pair-Programmed-With: Michael Adam <obnox@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2014-05-22 21:05:15 +02:00
Volker Lendecke
cbd73ba163 tdb: introduce tdb->hdr_ofs
This makes it possible to have some extra headers before
the real tdb content starts in the file.

This will be used used e.g. to implement locking based on robust mutexes.

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Pair-Programmed-With: Michael Adam <obnox@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2014-05-22 21:05:15 +02:00
Stefan Metzmacher
c29e64d97e tdb: introduce TDB_SUPPORTED_FEATURE_FLAGS
This will allow to store a feature mask in the tdb header on disk,
so that openers can check if they can handle the features
other openers are using.

Pair-Programmed-With: Volker Lendecke <vl@samba.org>
Pair-Programmed-With: Michael Adam <obnox@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2014-05-22 21:05:15 +02:00
Stefan Metzmacher
c0b0648555 tdb: use asprintf() to simplify tdb_summary()
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2014-05-22 21:05:15 +02:00
Stefan Metzmacher
e77cbe252f tdb: return ENOSYS if the tdb was created with spinlocks.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Mon May 12 21:07:04 CEST 2014 on sn-devel-104
2014-05-12 21:07:04 +02:00
Michael Adam
d9566085c6 tdb: consolidate tdb allocation code - re-use dead records at hash top.
When in tdb_store we re-use a dead record reactivated from the
target hash chain itself, we currently leave it in its place in
the chain. When we re-use a dead record from a different chain or
from the freelist instead, we insert it at the beginning of the
target chain.

This patch changes the behaviour to always newly store a
record at the beginning of the hash chain. This removes
a special case and hence simplifies the allocation code.
On the other hand side, it introduces two additioal tdb_ofs_write
calls for the in-chain-case.

Note the subtelty of the patch that by moving the case of the candidate
record's chain as new case "i=0" into the for loop, we also reverse the
order of the two steps in the for-loop body (non blocking freelist alloc
and searching for dead record in a chain) in order to keep the overall
order of execution identical.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Wed Apr  9 10:37:08 CEST 2014 on sn-devel-104
2014-04-09 10:37:08 +02:00
Stefan Metzmacher
80dff80ee9 tdb: don't alter errno on success of tdb_open_ex()
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2014-04-02 09:03:42 +02:00
Volker Lendecke
3034a5a62b tdb: Reduce freelist contention
In a metadata-intensive benchmark we have seen the locking.tdb freelist to be
one of the central contention points. This patch removes most of the contention
on the freelist. Ages ago we already reduced freelist contention by using the
even much older DEAD records: If TDB_VOLATILE is set, don't directly put
deleted records on the freelist, but just mark a few of them just as DEAD. The
next new record can them re-use that space without consulting the freelist.

This patch builds upon the DEAD records: If we need space and the freelist is
busy, instead of doing a blocking wait on the freelist, start looking into
other chains for DEAD records and steal them from there. This way every hash
chain becomes a small freelist. Just wander around the hash chains as long as
the freelist is still busy.

With this patch and the tdb mutex patch (following hopefully some time soon)
you can see a heavily busy clustered smbd run without locking.tdb futex
syscalls.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2014-03-18 13:42:10 +01:00
Volker Lendecke
1461362e93 tdb: Make "tdb_purge_dead" internally public
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2014-03-18 13:42:10 +01:00
Volker Lendecke
92ce9fd9af tdb: Make "tdb_find_dead" internally public
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2014-03-18 13:42:10 +01:00
Volker Lendecke
4ca018692f tdb: Add "last_ptr" to tdb_find_dead
Will be used soon to unlink a dead record from a chain

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2014-03-18 13:42:10 +01:00
Volker Lendecke
cb09d7937c tdb: Move adding tailer space to tdb_find_dead
This aligns the tdb_find_dead API with the tdb_allocate API and thus makes it a
bit easier to understand, at least for me.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2014-03-18 13:42:10 +01:00
Volker Lendecke
255edd1b41 tdb: Do a best fit search for dead records
Hash chains are (or can be made) short enough that a full search for the
best-fitting dead record is feasible. The freelist can become much longer,
there we don't do the full search but accept records which are too large.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2014-03-18 13:42:10 +01:00
Volker Lendecke
d1ce0110f0 tdb: Don't purge records to a blocked freelist
If the freelist is heavily contended, we should avoid accessing it

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2014-03-18 13:42:10 +01:00
Volker Lendecke
5f7b481349 tdb: Fix a tdb corruption
tdb_purge_dead can change the next pointer of "rec" if we purge the record
right behind the current record to be deleted. Just overwrite the magic,
not the whole record with stale data.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2014-03-18 13:42:10 +01:00
Michael Adam
001b9582cc tdb: always open internal databases with incompatible hash.
This makes them more efficient due to better distribution
of keys across hash chains.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Feb 15 08:26:07 CET 2014 on sn-devel-104
2014-02-15 08:26:06 +01:00
Michael Adam
41b7acacb3 tdb: in tdb_delete_hash, make lock/unlock bracket more obvious
by using the same variable as hash as in the lock.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Feb 15 03:21:07 CET 2014 on sn-devel-104
2014-02-15 03:21:07 +01:00
Michael Adam
cde8e290c9 tdb: simplify tdb_delete_hash() a bit
Make the lock/unlock bracket more obvious by extracting
locking (and finding) from the special cases to the top
of the function. This also lets us take lock and find
the record outside the special case branches (use dead
records or not).

There is a small semantic change implied:

In the dead records case, the record to delete is looked
up before the current dead records are potentially purged.
Hence, if the record to delete is not found, the dead
records are also not purge. This does not make a big
difference though, because purging is only delayed until
directly befor the next record to delete is in fact found.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2014-02-14 15:55:46 -08:00
Michael Adam
adb2cd1eee tdb: tdbtool: dump record magic with fixed number of 8 hex digits
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2014-02-14 15:53:25 -08:00
Michael Adam
057adfae47 tdb: tdbtool: dump record hash with fixed number of 8 hex digits
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2014-02-14 15:53:25 -08:00
Volker Lendecke
f3556bd03b tdb: Avoid reallocs for lockrecs
In normal operations we have at most 3 entries in this array. Don't
bother with shrinking.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Sat Dec 14 13:19:47 CET 2013 on sn-devel-104
2013-12-14 13:19:47 +01:00
Christian Ambach
6d88bfcab4 lib/tdb: fix compiler warnings
about a variable shadowing a global declaration

Signed-off-by: Christian Ambach <ambi@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2013-12-12 14:21:27 -08:00
Volker Lendecke
1f269fcc6e tdb: Add another overflow check to tdb_expand_adjust
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Mon Jun  3 14:08:54 CEST 2013 on sn-devel-104
2013-06-03 14:08:53 +02:00
Volker Lendecke
d9b4f19e73 tdb: Make tdb_recovery_allocate overflow-safe
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Rusty Russell <rusty@rustcorp.com.au>
2013-06-03 10:21:32 +02:00
Volker Lendecke
8b215df445 tdb: Make tdb_recovery_size overflow-safe
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Rusty Russell <rusty@rustcorp.com.au>
2013-06-03 10:21:31 +02:00
Stefan Metzmacher
7ae09a9695 tdb: add proper OOM/ENOSPC handling to tdb_expand()
Failing to do so will result in corrupt tdbs: We will overwrite
the hash chain pointers with 0x42424242.

Pair-Programmed-With: Volker Lendecke <vl@samba.org>

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Rusty Russell <rusty@rustcorp.com.au>
2013-06-03 10:21:30 +02:00
Stefan Metzmacher
854c5f0aac tdb: add overflow detection to tdb_expand_adjust()
We round up at maximun to a new size of 4GB,
but still return at least the given size.

The caller has to deal with ENOSPC itself.

Pair-Programmed-With: Volker Lendecke <vl@samba.org>

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Rusty Russell <rusty@rustcorp.com.au>
2013-06-03 10:21:28 +02:00
Stefan Metzmacher
e19d46f7e3 tdb: add overflow/ENOSPC handling to tdb_expand_file()
Pair-Programmed-With: Volker Lendecke <vl@samba.org>

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Rusty Russell <rusty@rustcorp.com.au>
2013-06-03 10:21:27 +02:00
Stefan Metzmacher
a07ba17e0c tdb: add a 'new_size' helper variable to tdb_expand_file()
Pair-Programmed-With: Volker Lendecke <vl@samba.org>

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Rusty Russell <rusty@rustcorp.com.au>
2013-06-03 10:21:22 +02:00
Volker Lendecke
4483bf143d tdb: Add overflow-checking tdb_add_off_t
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Rusty Russell <rusty@rustcorp.com.au>
2013-06-03 10:21:20 +02:00
Rusty Russell
3bd686c5ad tdb: fix logging of offets and lengths.
We can have offsets > 2G, so use unsigned values.  Fixes other prints to be
native types rather than casts, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue May 28 11:22:14 CEST 2013 on sn-devel-104
2013-05-28 11:22:14 +02:00
Christian Ambach
11f467d0bd tdb: include information about hash function being used in tdbtool info output
makes it possible to easily determine if the tdb under examination
uses jenkins hash or not

Signed-off-by: Christian Ambach <ambi@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2013-05-14 14:34:20 +02:00
Volker Lendecke
a92c08e18b tdb: Little format change
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2013-03-26 10:11:47 +01:00
Volker Lendecke
68698b4e64 tdb: Slightly simplify tdb_expand_file
The "else" keywords are not necessary here, we return in the preceding
if clause

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Mar  5 14:00:47 CET 2013 on sn-devel-104
2013-03-05 14:00:47 +01:00
Volker Lendecke
a7fdd4f7c2 tdb: Slightly simplify transaction_write
realloc(NULL, ...) is equivalent to malloc. We are already using this
realloc property for tdb->lockrecs. It should not make any difference
in speed, it just makes for a little simpler code.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Feb 19 17:30:13 CET 2013 on sn-devel-104
2013-02-19 17:30:13 +01:00
Volker Lendecke
fcb345f5d6 tdb: Make tdb_release_transaction_locks use tdb_allrecord_unlock
The transaction code uses tdb_alrecord_lock/upgrade, so it should also
use the tdb_allrecord_unlock function just for symmetry reasons

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2013-02-19 15:46:45 +01:00
Volker Lendecke
3534e4e8d5 tdb: Factor out the retry loop from tdb_allrecord_upgrade
For the mutex code we will have to lock the hashchain and the record
lock area independently. So we will have to call the loop twice. And,
it's a small refactoring for the better anyway I think.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2013-02-19 15:46:45 +01:00
Volker Lendecke
1f93f08364 tdb: Simplify fcntl_lock() a bit
All arguments but the cmd are the same. To me this looks a bit better
and saves some bytes in the object code.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2013-02-19 15:46:45 +01:00
Volker Lendecke
542400a966 tdb: Use tdb_null in freelistcheck
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2013-02-19 15:46:45 +01:00