IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu Jan 23 20:41:46 UTC 2020 on sn-devel-184
* Upgrade waf to version 2.0.18 to fix a cross-compilation issue.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13846
Signed-off-by: Uri Simchoni <uri@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* Build fixes
* Improve the performance by inlining the tdb_oob() checks
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Aug 20 14:45:41 UTC 2019 on sn-devel-184
When you set
in tdbtorture.c to make it more similar to locking.tdb use,
bin/tdbtorture -m -n 1 -l 100000 -s
becomes twice as fast. This is a pretty extreme case, but all other
tests that I did improve significantly as well.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
This is common between both implementations of tdb_oob(). It's
faster if we don't have to dereference function pointers.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
tdb_oob() will become a public function encapsulating the pointer
dereferences.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Fix the ubsan warning
lib/tdb/common/tdb.c:184:9: runtime error: null pointer passed as
argument 2, which is declared to never be null"
memcmp call now guarded by a length check.
memcmp returns zero when called with a zero length parameter.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Noel Power <npower@samba.org>
Autobuild-User(master): Noel Power <npower@samba.org>
Autobuild-Date(master): Mon Jul 1 14:50:54 UTC 2019 on sn-devel-184
Fixes:
lib/tdb/common/rescue.c:299:2: warning: Null pointer passed as an argument to a 'nonnull' parameter <--[clang]
qsort(found.arr, found.num, sizeof(found.arr[0]), cmp_key);
^ ~~~~~~~~~
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Fixes:
lib/tdb/common/transaction.c:354:2: warning: Null pointer passed as an argument to a 'nonnull' parameter <--[clang]
memcpy(tdb->transaction->blocks[blk] + off, buf, len);
&
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Fixes:
lib/tdb/common/lock.c:933:6: warning: Dereference of null pointer <--[clang]
if (tdb->allrecord_lock.count) {
^~~~~~~~~~~~~~~~~~~~~~~~~
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
We no longer need Samba to be py2/py3 compatible so we choose to return to the standard
function names.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <noel.power@suse.com>
We no longer need Samba to be py2/py3 compatible so we choose to return to the standard
function names.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <noel.power@suse.com>
Fixes
lib/tdb/common/freelistcheck.c:96:3: warning: Value stored to 'last_ptr' is never read <--[clang]
last_ptr = rec_ptr;
^ ~~~~~~~
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Gary Lockyer gary@catalyst.net.nz
Autobuild-User(master): Noel Power <npower@samba.org>
Autobuild-Date(master): Tue Jun 11 13:31:01 UTC 2019 on sn-devel-184
README is far from a complete, good and accurate document, but what's
in there should at least not have obvious errors.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
* Do not propogate any errors from tdb_repack() to to tdb_transaction_commit()
The repack may fail due to lock ordering or memory limits, but the original data has
been written so the call must succeed. (bug 13952)
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri May 17 08:21:52 UTC 2019 on sn-devel-184
The call to tdb_repack() inside tdb_transaction_commit()
is an optimization, not part of the transaction itself,
so failing due to lock or other errors isn't a fatal error
that should cause the caller to think the transaction was
a failure by returning -1.
The tdb transaction itself has finished and been committed
onto stable storage via fsync and all locks released at the
point tdb_repack() is called.
tdb_repack() is only called here as it's a convenient point
to attempt to reduce tdb fragmentation without having to add
a timer call to repack in all users of tdb.
This causes lock ordering issues in Samba, showing up as:
ldb: ltdb: tdb(../private/sam.ldb.d/DC=SAMBA2008R2,DC=EXAMPLE,DC=COM.ldb): tdb_transaction_prepare_commit: failed to upgrade hash locks: Locking error
This is because Samba has multiple tdb databases open, and the lock order between them
is important.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13952
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
squash 'cast between incompatible function types' warning
To avoid warning above produced by using
-Wcast-function-type we;
+ ensure PyCFunctions of type METH_NOARGS defined dummy arg
+ ensure PyCFunctions of type METH_KEYWORDS use PY_DISCARD_FUNC_SIG
macro
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
* Remove Python 2.x support except to build just the bare C library
* Remove --extra-python (a build time mode to produce Python2 and Python3
bindings at the same time)
* New minor version to allow Samba 4.10 to release a tdb if
required from that branch
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
We do this by removing the confusing mandatory option to
conf.SAMBA_CHECK_PYTHON{,_HEADERS}(), instead just use the value of
--disable-python internally
This follows the default minimum of Python 3.4 and keeps things consistent
with the main Samba build where --disable-python is required to skip building
python bindings.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
When calling make from the ldb, talloc, tdb, and
tevent bundles, we need to first find the
location of the waf script. Currently the build
fails since it can't find waf.
Fixes regression caused by a660b7f.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Traceback (most recent call last):
File "tdb-1.3.17/third_party/waf/waflib/Scripting.py", line 158, in waf_entry_point
run_commands()
File "tdb-1.3.17/third_party/waf/waflib/Scripting.py", line 251, in run_commands
ctx = run_command(cmd_name)
File "tdb-1.3.17/third_party/waf/waflib/Scripting.py", line 235, in run_command
ctx.execute()
File "tdb-1.3.17/third_party/waf/waflib/Context.py", line 204, in execute
self.recurse([os.path.dirname(g_module.root_path)])
File "tdb-1.3.17/third_party/waf/waflib/Context.py", line 286, in recurse
user_function(self)
File "tdb-1.3.17/wscript", line 225, in testonly
cmd = "BINDIR={} {}".format(blddir, sh_test)
ValueError: zero length field name in format
Signed-off-by: Lukas Slebodnik <lslebodn@fedoraproject.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sun Jan 20 03:49:59 CET 2019 on sn-devel-144
This avoids creating a mini-configure in the configure script.
Users wishing to use python2 to build need to specify PYTHON=
to both ./configure and make
After we merged the python3 change, it became clear that relying on systems prefixing
the correct python just causes trouble and make debugging harder, so only use $PYTHON
for the override, not the default case
This essentially reverts a660b7fb8e but
leaves the files more consistent.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <npower@samba.org>
This works around python3 having a new hash seed each time it starts to allow
a second "make" not to rebuild the world.
This should probably be reverted once we find the hash that is causing
the issue, but should reduce frustration for now.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Make sure default make and configure for all now defaults
to building with python3.
To build a samba (or sub component e.g. talloc etc.) with python3
./configure && make
To build a samba (or sub component e.g. talloc etc.) with python2
PYTHON=python ./configure && PYTHON=python make
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
tdb->max_dead_records is "int", as is the corresponding parameter to
tdb_set_max_dead(). Not that a signed variable makes any sense, but
this is old code and tdb_set_max_dead() is a public API which we
should not change for this.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Nov 6 21:52:32 CET 2018 on sn-devel-144
This is a prerequisite to allow gencache to run on a non-transactioned
database with mutexes.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
This is a lightweight readonly traverse of a single chain, see the
comment in the header file.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Before this patch we had 3 loops walking a hash chain to delete
records:
tdb_do_delete() to find the predecessor of the record that was to be
deleted. tdb_count_dead(), the name says it all and tdb_purge_dead()
to give back all dead records from a chain to the freelist.
This patch introduces tdb_trim_dead that walks a hash chain just
once. While it does so it counts the number of dead records, and all
records beyond tdb->max_dead_records are moved to the freelist.
Normal record deletion now works by always marking a record as dead in
step 1 and then calling tdb_trim_dead. This is made safe against
circular chains by doing the slow chain walk only in the case when we
did not delete a dead record during our walk.
It changes our dynamics a bit:
When deleting a record with non-zero max_dead_records, now we always
leave that number of records around when deleting, doing a blocking
lock on the freelist when we found too many dead records.
Previously when exceeding max_dead_records we wiped all dead records
to start accumulating them from scratch, assuming we could lock the
freelist in a nonblocking fashion.
The net effect for an uncontended freelist is the same: In
tdb_allocate() we still completely hand over all dead records to the
freelist when we could lock it, it just happens later than without
this patch.
This means for a lightly loaded system we will potentially leave more
dead records around in databases like locking.tdb. However, on a
heavily loaded system we become more predictable: If the freelist is
so heavily contended that across many deletes we can't get hold of it,
previously we accumulated more dead records than max_dead_records
would allow. This is a really lowlevel tradeoff that is likely hard to
measure, but to me becoming more deterministic without sacrificing too
much parallelism (we keep more dead records around) is worth trying.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Oct 30 02:48:38 CET 2018 on sn-devel-144
The next commit will change the handling of dead records, removing the
"tdb_do_delete" function. As traverses should not happen in normal
operations, dead records from them should be rare, and relying on
traverses to remove them is a very bad idea IMHO.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Right now we don't do transactions with mutexed tdbs. tdbtorture -m
locks up. I haven't really investigated why that is the case. The lockup
confused me quite a bit until I figured out it works fine as long as it
does not do transactions, which is all we need right now.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
This has not been activated by default for ages and can be very
inefficient. With check_merge_with_left_record() we have an
alternative that will merge freelist records while we walk it
anyway. This has reduced fragmentation significantly
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Try to store a record for which the (circular) freelist does not have
any entry.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
This just walks tdb_find by searching for a nonexistent record
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
We can't really do the full check while the freelist is modified on the
fly. As long as we don't merge any freelist entries, we should be good
to apply this check.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
This captures the tdb_rescue protection against circular hash chains
with a slow pointer updated only on every other record traverse
If a hash chain has a loop, eventually the next_ptr
will cycle around and be identical to the 'slow' pointer.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Simple bash test for readonly locks on tdbbackup:
1. Running tdbbackup on a database with and without readonly locks enabled.
2. Dump both backups and original.
3. Check all three dumps match.
A binary sample_tdb.tdb file is included for the test because the existing
sample tdbs in lib/tdb/test are either corrupt or empty.
Signed-off-by: Aaron Haslett <aaron.haslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
The netcmd 'domain backup offline' command will use the tdbbackup tool but
require readonly locking of tdb databases, otherwise all database access would
be blocked during a backup. This patch adds the option. A backup script
should use this tool with the readonly locks option after taking a transaction
lock on the target database.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
* Fix build on AIX
* Python3 compatibility fixes
* Use tdb_wipe_all in "erase" command
* Harden allocating the tdb recovery area
* Make sure the hash size fits
* Harden tdb_check_used_record against overflow
* Harden tdb_rec_read
* Handle TDB_NEXT_LOCK_ERR in tdb_traverse_internal
* Fix build warnings
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13493
Here is the build error on AIX 7.1.
../../lib/tdb/tools/tdbtool.c:39:12: error: 'disable_lock' redeclared as different kind of symbol
static int disable_lock;
^~~~~~~~~~~~
In file included from /usr/include/sys/gfs.h:24:0,
from /usr/include/sys/vfs.h:27,
from ../../lib/replace/system/filesys.h:48,
from ../../lib/tdb/tools/tdbtool.c:26:
/usr/include/sys/lock_def.h:314:5: note: previous declaration of 'disable_lock' was here
int disable_lock(int,simple_lock_t);
^~~~~~~~~~~~
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
In py3, iterxxx methods are removed.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This is a lot quicker on large, fragmented databases. tdb_delete can
leave the freelist in a fragmented mess.
Also, it's a lot more robust: I've got a 4GB tdb file that was affected
by the problem fixed with c7211882a7. These databases have large space
at the end that is not part of any record or freelist
entry. tdb_wipe_all converts this space into a freelist entry. One
downside is that with those broken databases (which should not happen
after c7211882a7) have unallocated blocks in their file range after
this operation.
I think the speed advantage outweighs this disadvantage.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Many of those warnings are difficult to fix, but this one was easy :-)
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu Mar 22 07:21:44 CET 2018 on sn-devel-144
* Add protection against EINTR.
* Truncate the file after expand failure, ENOSPC
* Use posix_fallocate() to expand the file
* Fix GCC compiler warnings
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Aug 24 21:17:48 CEST 2017 on sn-devel-144
This should be significantly faster than pwriting.
openbsd doesn't have posix_fallocate, so we do need the fallback. Also, it
might have weird failure modes, so we keep the old code in place except for
posix_fallocate returning success or ENOSPC.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu Aug 24 05:38:49 CEST 2017 on sn-devel-144
More README.Coding, but I need "ret" in the next commit as well :-)
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Without this it's very easy to create virtually huge files: ftruncate expands a
file, the pwrites fail with ENOSPC, thus the write fails. The next writer runs
into the same situation, and ftruncate-expands the file even further. tdb_check
will then spend ages reading the 4GB of zeros byte by byte.
Here we hold the freelist lock or are inside a transaction, so it is safe to
cut the file again. Nobody can have used the space that we have tried to
allocate, so we can't have any stray pointers corrupting the database.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Due to the non-fixable bug in the BUCKET macro tdbtool list printed some
other hash chainlist, not the freelist.
Bug: https://bugzilla.samba.org/show_bug.cgi?id=12888
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
The following C program demonstrates the issue:
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <stdbool.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <dirent.h>
#include <sys/types.h>
int main(int argc, char **argv)
{
int hash = -1;
int tsize_signed = 10;
unsigned int tsize_unsigned = 10;
int bucket;
#define BUCKET(hash, tsize) ((hash) % (tsize))
bucket = BUCKET(hash, tsize_unsigned);
printf("hash [%d] tsize [%d] bucket [%d]\n", hash, tsize_unsigned, bucket);
bucket = BUCKET(hash, tsize_signed);
printf("hash [%d] tsize [%d] bucket [%d]\n", hash, tsize_signed, bucket);
return 0;
}
Output:
$ ./tmp
hash [-1] tsize [10] bucket [5]
hash [-1] tsize [10] bucket [-1]
The first version is what the current BUCKET() macro does. As a result
we lock the hashtable chain in bucket 5, NOT the freelist.
-1 is sign converted to an unsigned int 4294967295 and
4294967295 % 10 = 5.
As all callers will lock the same wrong list consistently locking is
still consistent.
Stumpled across this when looking at the output of `tdbtool DB list`
which always printed some other hashchain and not the freelist.
The freelist bucket offset computation doesn't use the BUCKET macro in
freelist.c (directly or indirectly) when working on the freelist, it
just directly uses the FREELIST_TOP define, so this problem only affects
tdbtool list.
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
The variable stores the hashtable bucket, not the hash. No change in
behaviour.
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
This is another level of indentation, but it took me a while staring at the
if-condition to find that "locked" was assigned the result of "==0", not the
return value of tdb_nest_lock().
Best viewed with "git show -b".
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
This fixes a GCC warning.
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Aug 10 02:26:09 CEST 2017 on sn-devel-144
