1
0
mirror of https://github.com/samba-team/samba.git synced 2024-12-23 17:34:34 +03:00
Commit Graph

110 Commits

Author SHA1 Message Date
Volker Lendecke
d2b852d79b tdb: Fix a typo
Reviewed-by: Rusty Russell <rusty@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2012-12-21 11:54:40 +01:00
Volker Lendecke
2c3fd8a13e tdb: Fix a missing CONVERT
methods->tdb_write expects data in on-disk format. For reading that
record, methods->tdb_read() has taken care of the on-disk to in-memory
representation according to the DOCONV() flag passed down. tdb_rec_write()
is a wrapper around methods->tdb_write just doing the CONVERT() on the
way to disk.

Reviewed-by: Rusty Russell <rusty@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2012-12-21 11:54:33 +01:00
Volker Lendecke
c62f8baff8 tdb: Make tdb robust against improper CLEAR_IF_FIRST restart
When winbind is restarted, there is a potential crash in tdb. Following
situation: We are in a cluster with ctdb. A winbind child hangs
in a request to the DC. Cluster monitoring decides the node has a
problem. Cluster monitoring decides to kill ctdbd. winbind child
still hangs in a RPC request. winbind parent figures that ctdb is
dead and immediately commits suicide. winbind parent is restarted by
cluster management, overwriting gencache.tdb with CLEAR_IF_FIRST. The
CLEAR_IF_FIRST logic as implemented now will not see that a child still
has the tdb open, only the parent holds the ACTIVE_LOCK due to performance
reasons. During the CLEAR_IF_FIRST logic is done, there is a very small
window where we ftruncate(tfd, 0) the file and re-write a proper header
without a lock. When during this small window the winbind child comes
back, wanting to store something into gencache.tdb, that winbind child
will crash with a SIGBUS.

Sounds unlikely? See:

[2012/09/29 07:02:31.871607,  0] lib/util.c:1183(smb_panic)
  PANIC (pid 1814517): internal error
[2012/09/29 07:02:31.877596,  0] lib/util.c:1287(log_stack_trace)
  BACKTRACE: 35 stack frames:
   #0 winbindd(log_stack_trace+0x1a) [0x7feb7d4ca18a]
   #1 winbindd(smb_panic+0x2b) [0x7feb7d4ca25b]
   #2 winbindd(+0x1a3cc4) [0x7feb7d4bacc4]
   #3 /lib64/libc.so.6(+0x32900) [0x7feb7a929900]
   #4 /lib64/libc.so.6(memcpy+0x35) [0x7feb7a97f355]
   #5 /usr/lib64/libtdb.so.1(+0x6e76) [0x7feb7b0b0e76]
   #6 /usr/lib64/libtdb.so.1(+0x3d37) [0x7feb7b0add37]
   #7 /usr/lib64/libtdb.so.1(+0x863d) [0x7feb7b0b263d]
   #8 /usr/lib64/libtdb.so.1(+0x8700) [0x7feb7b0b2700]
   #9 /usr/lib64/libtdb.so.1(+0x2505) [0x7feb7b0ac505]
   #10 /usr/lib64/libtdb.so.1(+0x25b7) [0x7feb7b0ac5b7]
   #11 /usr/lib64/libtdb.so.1(tdb_fetch+0x13) [0x7feb7b0ac633]
   #12 winbindd(gencache_set_data_blob+0x259) [0x7feb7d4d8449]
   #13 winbindd(gencache_set+0x53) [0x7feb7d4d85b3]
   #14 winbindd(gencache_del+0x5e) [0x7feb7d4d879e]
   #15 winbindd(saf_delete+0x93) [0x7feb7d54b693]
   #16 winbindd(+0xe507e) [0x7feb7d3fc07e]
   #17 winbindd(+0xe85e5) [0x7feb7d3ff5e5]
   #18 winbindd(+0xe65be) [0x7feb7d3fd5be]
   #19 winbindd(+0xe7562) [0x7feb7d3fe562]
   #20 winbindd(init_dc_connection+0x2e) [0x7feb7d3fe5be]
   #21 winbindd(+0xe75d9) [0x7feb7d3fe5d9]
   #22 winbindd(cm_connect_netlogon+0x58) [0x7feb7d3fe658]
   #23 winbindd(_wbint_PingDc+0x61) [0x7feb7d410991]
   #24 winbindd(+0x103175) [0x7feb7d41a175]
   #25 winbindd(winbindd_dual_ndrcmd+0xb7) [0x7feb7d4107d7]
   #26 winbindd(+0xf8609) [0x7feb7d40f609]
   #27 winbindd(+0xf9075) [0x7feb7d410075]
   #28 winbindd(tevent_common_loop_immediate+0xe8) [0x7feb7d4db198]
   #29 winbindd(run_events_poll+0x3c) [0x7feb7d4d93fc]
   #30 winbindd(+0x1c2b52) [0x7feb7d4d9b52]
   #31 winbindd(_tevent_loop_once+0x90) [0x7feb7d4d9f60]
   #32 winbindd(main+0x7b3) [0x7feb7d3e7aa3]
   #33 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7feb7a915cdd]
   #34 winbindd(+0xce2a9) [0x7feb7d3e52a9]

This is in a winbind child, logfiles surrounding indicate the parent
was restarted.

This patch takes all chain locks around the CLEAR_IF_FIRST introduced
tdb_new_database.
2012-10-06 13:23:42 +02:00
Rusty Russell
37fd93194d tdb: Make robust against shrinking tdbs
When probing for a size change (eg. just before tdb_expand, tdb_check,
tdb_rescue) we call tdb_oob(tdb, tdb->map_size, 1, 1).  Unfortunately
this does nothing if the tdb has actually shrunk, which as Volker
demonstrated, can actually happen if a "longlived" parent crashes.

So move the map/update size/remap before the limit check.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-10-06 13:23:41 +02:00
Rusty Russell
90f463b25f tdb: add tdb_rescue()
This allows for an emergency best-effort dump.  It's a little better than
strings(1).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-10-04 09:04:19 +09:30
Volker Lendecke
a168a7c791 tdb: Fix a typo
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Tue Oct  2 19:52:16 CEST 2012 on sn-devel-104
2012-10-02 19:52:16 +02:00
Rusty Russell
1783fe3443 tdb: make TDB_NOSYNC merely disable sync.
(As suggested by Stefan Metzmacher, based on the change to ntdb.)

Since commit ec96ea690e, we handle the case
where a process dies during a transaction commit.  Unfortunately, TDB_NOSYNC
means this no longer works, as it disables the recovery area as well as the
actual msync/fsync.  We should do everything except the syncs.

This also means we can do a complete test with $TDB_NO_FSYNC set; just
to get more complete coverage, we disable it explicitly for one test
(where we override the actual sync calls anyway).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Amitay Isaacs
3fdeaa3992 lib/tdb: Add/expose lock functions to support CTDB
This patch adds two lock functions used by CTDB to perform asynchronous
locking. These functions do not actually perform any fcntl operations,
but only increment internal counters.

 - tdb_transaction_write_lock_mark()
 - tdb_transaction_write_lock_unmark()

It also exposes two internal functions
 - tdb_lock_nonblock()
 - tdb_unlock()

These functions are NOT exposed in include/tdb.h to prevent any further
uses of these functions. If you ever need to use these functions, consider
using tdb2.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
2012-03-29 20:07:03 +10:30
Rusty Russell
4442c0b2c9 lib/tdb: fix transaction issue for HAVE_INCOHERENT_MMAP.
We unmap the tdb on expand, the remap.  But when we have INCOHERENT_MMAP
(ie. OpenBSD) and we're inside a transaction, doing the expand can mean
we need to read from the database to partially fill a transaction block.
This fails, because if mmap is incoherent we never allow accessing the
database via read/write.

The solution is not to unmap and remap until we've actually written the
padding at the end of the file.

Reported-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Fri Mar 23 02:53:15 CET 2012 on sn-devel-104
2012-03-23 02:53:15 +01:00
Rusty Russell
330e3e1b91 lib/tdb: fix missing return 0 code.
fde694274e made tdb_mmap return an int,
but didn't put the return 0 on the "internal db" case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-03-23 10:41:55 +10:30
Rusty Russell
fde694274e lib/tdb: fix OpenBSD incoherent mmap.
This comment appears in two places in the code (commit
4c6a8273c6 from 2001):

	/*
	 * We must ensure the file is unmapped before doing this
	 * to ensure consistency with systems like OpenBSD where
	 * writes and mmaps are not consistent.
	 */

But this doesn't help, because if one process is using mmap and another
using pwrite, we get incoherent results.  As demonstrated by OpenBSD's
failure on the tdb unit tests.

Rather than disable mmap on OpenBSD, we test for this issue and force mmap
to be enabled.  This means that we will fail on very large TDBs on 32-bit
systems, but it's better than the horrendous performance penalty on every
OpenBSD system.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-03-22 01:57:37 +01:00
Rusty Russell
390b9a2dd8 tdb: make tdb_private.h idempotent.
The most convenient way to write unit tests in C is to directly
#include the C files (CCAN uses this, for example).  That works quite
well, but it means that tdb_private.h now needs to be protected
against multiple inclusions.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-02-14 04:04:43 +10:30
Ira Cooper
7b42ceb414 Fix compile when TDB_TRACE is enabled.
Autobuild-User: Jeremy Allison <jra@samba.org>
Autobuild-Date: Fri Jan  6 04:16:41 CET 2012 on sn-devel-104
2012-01-06 04:16:41 +01:00
Volker Lendecke
c1e9537ed0 tdb: Use tdb_parse_record in tdb_update_hash
This avoids a tdb_fetch, thus a malloc/memcpy/free in the tdb_store path
2011-12-25 13:31:58 +01:00
Rusty Russell
5767224b7f tdb: don't free old recovery area when expanding if already at EOF.
We allocate a new recovery area by expanding the file.  But if the
recovery area is already at the end of file (as shown in at least one
client case), we can simply expand the record, rather than freeing it
and creating a new one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Wed Dec 21 06:25:40 CET 2011 on sn-devel-104
2011-12-21 06:25:40 +01:00
Rusty Russell
3a2a755e33 tdb: use same expansion factor logic when expanding for new recovery area.
If we're expanding because the current recovery area is too small, we
expand only the amount we need.  This can quickly lead to exponential
growth when we have a slowly-expanding record (hence a
slowly-expanding transaction size).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2011-12-21 14:17:16 +10:30
Volker Lendecke
664add1775 tdb: Avoid a malloc/memcpy in _tdb_store 2011-12-19 15:18:08 +01:00
Rusty Russell
b64494535d tdb: be more careful on 4G files.
I came across a tdb which had wrapped to 4G + 4K, and the contents had been
destroyed by processes which thought it only 4k long.  Fix this by checking
on open, and making tdb_oob() check for wrap itself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Mon Dec 19 07:52:01 CET 2011 on sn-devel-104
2011-12-19 07:52:01 +01:00
Rusty Russell
ee720fc19c tdb: increment sequence number in tdb_wipe_all().
TDB2 testing revealed that tdb1 doesn't do this.  It's minor, but fix it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Tue Aug 16 10:47:41 CEST 2011 on sn-devel-104
2011-08-16 10:47:41 +02:00
Rusty Russell
4fa51257b2 tdb: enable VALGRIND to remove valgrind noise.
Andrew Bartlett complained that valgrind needs --partial-loads-ok=yes otherwise
the Jenkins hash makes it complain.

My benchmarking here revealed that at least with modern gcc (4.5) and CPU
(Intel i5 32 bit) there's no measurable performance penalty for the
"correct" code, so rip out the optimized one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Wed Jun  8 11:05:47 CEST 2011 on sn-devel-104
2011-06-08 11:05:47 +02:00
Rusty Russell
36cfa7b79e tdb: make sure we skip over recovery area correctly.
If it's really the recovery area, we can trust the rec_len field, and
don't have to go groping for bitpatterns.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Tue Apr 19 14:15:22 CEST 2011 on sn-devel-104
2011-04-19 14:15:22 +02:00
Simo Sorce
cb884186a5 tdb_expand: limit the expansion with huge records
ldb can create huge records when saving indexes.
Limit the tdb expansion to avoid consuming a lot of memory for
no good reason if the record being saved is huge.
2011-04-18 22:15:11 +09:30
Rusty Russell
094ab60053 tdb: tdb_repack() only when it's worthwhile.
tdb_repack() is expensive and consumes memory, so we can spend some
effort to see if it's worthwhile.  In particular, tdbbackup doesn't
need to repack: it started with an empty database!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2011-04-18 22:15:11 +09:30
Rusty Russell
6aa72dae8f tdb: fix transaction recovery area for converted tdbs.
This is why macros are dangerous; these were converting the pointers, not the
things pointed to!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2011-04-18 22:15:11 +09:30
Volker Lendecke
0080f944b4 tdb: Fix Coverity ID 2238: SECURE_CODING 2011-03-30 09:58:32 +02:00
Volker Lendecke
25397de589 tdb: Fix Coverity ID 2192: NO_EFFECT
(ret < 0) can never be true
2011-03-27 22:22:12 +02:00
Volker Lendecke
91cad71390 tdb: Fix a C++ warning
Autobuild-User: Volker Lendecke <vlendec@samba.org>
Autobuild-Date: Sat Feb 12 19:50:55 CET 2011 on sn-devel-104
2011-02-12 19:50:55 +01:00
Rusty Russell
cac57328a6 tdb: tdb_summary() support.
Autobuild-User: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date: Wed Dec 29 10:12:05 CET 2010 on sn-devel-104
2010-12-29 10:12:05 +01:00
Matthias Dieter Wallnöfer
989d8803f2 tdb:common/open.c - use "discard_const_p" for certain "tdb->name" assignments
In order to suppress compiler warnings.
2010-11-27 21:50:42 +01:00
Stefan Metzmacher
dedd064aa8 tdb: set tdb->name early, as it's needed for tdb_name()
tdb_name() might be used within the given log function,
which might be called from within tdb_open_ex().

metze

Autobuild-User: Stefan Metzmacher <metze@samba.org>
Autobuild-Date: Fri Nov 12 11:22:21 UTC 2010 on sn-devel-104
2010-11-12 11:22:21 +00:00
Jelmer Vernooij
62c4af9942 tdb: Set _PUBLIC_ in C file rather than header files (Debian bug 600898)
Autobuild-User: Jelmer Vernooij <jelmer@samba.org>
Autobuild-Date: Thu Oct 21 11:47:22 UTC 2010 on sn-devel-104
2010-10-21 11:47:22 +00:00
Rusty Russell
2dcf76c924 tdb: TDB_INCOMPATIBLE_HASH, to allow safe changing of default hash.
This flag to tdb_open/tdb_open_ex effects creation of a new database:
1) Uses the Jenkins lookup3 hash instead of the old gdbm hash if none is
   specified,
2) Places a non-zero field in header->rwlocks, so older versions of TDB will
   refuse to open it.

This means that the caller (ie Samba) can set this flag to safely
change the hash function.  Versions of TDB from this one on will either
use the correct hash or refuse to open (if a different hash is specified).
Older TDB versions will see the nonzero rwlocks field and refuse to open
it under any conditions.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-27 10:48:28 +09:30
Rusty Russell
ccac258d14 tdb: automatically identify Jenkins hash tdbs
If the caller to tdb_open_ex() doesn't specify a hash, and tdb_old_hash
doesn't match, try tdb_jenkins_hash.

This was Metze's idea: it makes life simpler, especially with the upcoming
TDB_INCOMPATIBLE_HASH flag.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-27 10:48:28 +09:30
Rusty Russell
3258cf3f11 tdb: add Bob Jenkins lookup3 hash as helper hash.
This is a better hash than the default: shipping it with tdb makes it easy
for callers to use it as the hash by passing it to tdb_open_ex().

This version taken from CCAN and modified, which took it from
http://www.burtleburtle.net/bob/c/lookup3.c.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-27 10:48:28 +09:30
Günther Deschner
1585c4df68 lib/tdb: fix c++ build warning in tdb_header_hash().
Guenther
2010-09-20 16:15:11 -07:00
Andrew Tridgell
ff515ff477 tdb: added TDB_NO_FSYNC env variable
this might help reduce test times and load on test machines
2010-09-16 21:09:17 +10:00
Rusty Russell
786b726300 tdb: put example hashes into header, so we notice incorrect hash_fn.
This is Stefan Metzmacher <metze@samba.org>'s patch with minor changes:
1) Use the TDB_MAGIC constant so both hashes aren't of strings.
2) Check the hash in tdb_check (paranoia, really).
3) Additional check in the (unlikely!) case where both examples hash to 0.
4) Cosmetic changes to var names and complaint message.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-13 20:05:59 +09:30
Rusty Russell
f77708e962 tdb: fix tdb_check() on other-endian tdbs.
We must not endian-convert the magic string, just the rest.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-13 19:59:18 +09:30
Rusty Russell
82e5644c9d tdb: fix tdb_check() on read-only TDBs to actually work.
Commit bc1c82ea13 "Fix tdb_check() to work with read-only tdb databases."
claimed to do this, but tdb_lockall_read() fails on read-only databases.

Also make sure we can still do tdb_check() inside a transaction (weird,
but we previously allowed it so don't break the API).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-13 19:58:23 +09:30
Rusty Russell
9e0deff904 tdb: make check more robust against recovery failures.
We can end up with dead areas when we die during transaction commit;
tdb_check() fails on such a (valid) database.

This is particularly noticable now we no longer truncate on recovery;
if the recovery area was at the end of the file we used to remove it
that way.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-13 19:55:26 +09:30
Rusty Russell
11ab43084b tdb: workaround starvation problem in locking entire database.
We saw tdb_lockall() take 71 seconds under heavy load; this is because Linux
(at least) doesn't prevent new small locks being obtained while we're waiting
for a big log.

The workaround is to do divide and conquer using non-blocking chainlocks: if
we get down to a single chain we block.  Using a simple test program where
children did "hold lock for 100ms, sleep for 1 second" the time to do
tdb_lockall() dropped signifiantly.  There are ln(hashsize) locks taken in
the contended case, but that's slow anyway.

More analysis is given in my blog at http://rusty.ozlabs.org/?p=120

This may also help transactions, though in that case it's the initial
read lock which uses this gradual locking routine; the update-to-write-lock
code is separate and still tries to update in one go.

Even though ABI doesn't change, minor version bumped so behavior change
can be easily detected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-08-14 02:31:22 +09:30
Jeremy Allison
bc1c82ea13 Fix tdb_check() to work with read-only tdb databases. The function tdb_lockall() uses F_WRLCK internally, which doesn't work on a fd opened with O_RDONLY. Use tdb_lockall_read() instead.
Jeremy.
2010-07-29 08:56:35 +09:30
Günther Deschner
f7a3bd4fa4 tdb: fix the build on mac os x 10.6.4.
Guenther
2010-07-01 23:14:57 +02:00
Günther Deschner
2eab1d7fdc tdb: remove unused variable in tdb_new_database().
Guenther
2010-05-11 13:41:17 +02:00
Rusty Russell
91e4a1760d tdb: fix short write logic in tdb_new_database
Commit 207a213c/24fed55d purported to fix the problem of signals during
tdb_new_database (which could cause a spurious short write, hence a failure).
However, the code is wrong: newdb+written is not correct.

Fix this by introducing a general tdb_write_all() and using it here and in
the tracing code.

Cc: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-05-05 15:37:18 +09:30
Andrew Tridgell
773a8afbba tdb: update tdb ABI to use hide_symbols=True
We now use -fvisibilty=hidden to hide symbols from outside the tdb
shared library.

This also moved tdb_transaction_recover() into the tdb_private.h
header, as it should never have been a public API. For that reason we
are changing the version number. We're only doing a minor version
increment as it is extremely unlikely that anyone was actually using
tdb_transaction_recover() as its locking requirements were rather
unusual.

Pair-Programmed-With: Rusty Russell <rusty@samba.org>
2010-04-20 15:50:27 +10:00
Volker Lendecke
261c3b4f1b tdb: Add a non-blocking version of tdb_transaction_start 2010-03-26 14:27:47 -04:00
Volker Lendecke
59315887a0 tdb: Fix indentation in tdb_new_database() 2010-03-25 10:30:10 +01:00
Volker Lendecke
ea8e0d5d54 Fix some nonempty blank lines 2010-03-25 10:24:45 +01:00
Volker Lendecke
fb98f60594 tdb: If tdb_parse_record does not find a record, return -1 instead of 0 2010-02-28 17:40:59 +01:00