IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
lib/util/safe_string.h is similar to source3/include/safe_string.h, but
the former has fewer checks. It is missing bcopy, strcasecmp, and
strncasecmp.
Add the missing elements to lib/util/safe_string.h remove the other
safe_string.h which is in the source3-specific path. To accomodate
existing uses of str(n?)casecmp, add #undef lines to source files where
they are used.
Signed-off-by: Matthew DeVore <matvore@google.com>
Reviewed-by: David Mulder <dmulder@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Aug 28 02:18:40 UTC 2020 on sn-devel-184
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14418
Signed-off-by: Mathieu Parent <math.parent@gmail.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Aug 25 04:23:19 UTC 2020 on sn-devel-184
Otherwise a malformed control with unexpected NULL data will segfault
ldb_control_to_string(), though this is not very likely to affect
anyone in practice as converting controls to strings is rarely
necessary. If it happens at all in Samba it is in Python code.
Found by Honggfuzz using fuzz_ldb_parse_control.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Wed Jul 29 04:43:23 UTC 2020 on sn-devel-184
Prevent use after free issues if ldb_lock_backend_callback is called
twice, usually due to ldb_module_done being called twice. This can happen if a
module ignores the return value from function a function that calls
ldb_module_done as part of it's error handling.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14364
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Fix regression introduced by commit ce2bf5c72b
lldb_add_msg_attr() calls ldb_msg_add_empty() which, in turn, calls
calls _ldb_msg_add_el() which already increments msg->num_elements by one.
As a result, msg->num_elements is bigger than the actual number of
elements and any iteration over elements would step over elements array
boundary.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14413
Signed-off-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Fri Jun 19 08:35:33 UTC 2020 on sn-devel-184
We won't get NULL data over ldap, but it can be set via 'local_oid:'.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Limit the number of nested conditionals allowed by ldb_parse tree to
128, to avoid potential stack overflow issues.
Credit Oss-Fuzz
REF: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=19508
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Gary Lockyer <gary@samba.org>
Autobuild-Date(master): Sun May 10 23:21:08 UTC 2020 on sn-devel-184
We want to keep going down the module stack, and not start from the top again.
ASQ is above the ACL modules, but below paged_results and we do not wish to
re-trigger that work.
Thanks to Andrei Popa <andrei.popa@next-gen.ro> for finding,
reporting and working with us to diagnose this issue!
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14331
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Initilalise "stale" to zero before passing a pointer to it to
mdb_reader_check.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Tue Apr 7 12:29:00 UTC 2020 on sn-devel-184
In use process failures and Bind9 shut downs leave stale entries in the
lmdb reader table. This can result in lmdb filling it's database file, as
the free list can not be reclaimed due to the stale reader.
In this fix we call mdb_reader_check at the start of each transaction,
to free any stale readers. As the default maximum number of readers is
127, this should not impact on performance to any great extent.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14330
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Mar 31 01:26:07 UTC 2020 on sn-devel-184
Fix indentation of list members and fix lines > 79 characters to remove
PEP8 warnings.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add cmocka tests to confirm lmdb's handling of the free list.
As a result of lmdb's MVCC (Multiversion Concurrency Control) long
running read transactions or stale readers (read transactions where the
process exited without ending the transaction) can cause the database to
run out of space.
Items in the free list are only reused when they would not be visible in
a read transaction. So long running read transactions prevent entries
in the free list being reused, and the database can run out of space.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
ldb* tools, when passed a raw filename assume tdb://
By default, ldb_tdb will call tdb with O_CREAT.
TDB, when passed O_CREAT and a not-tdb file, will wipe the file.
This means that if you run ldbedit <path to mdb-format-ldb file> the file
will be wiped, which is unexpected. I noticed this while trying to
corrupt a sam.ldb backend file (for testing), but instead I wiped it!
Ideally tdb would not do that, but the behaviour has been this way for
decades. Ideally ldb would have had a "create db" command, but this
has been the job of ldbadd for over a decade.
So this just blunts the knife for ldbedit.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14302
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
This avoids using "ldb" as the memory context in most cases, and may avoid
a long-term memory leak if future changes cause dn_key not to be freed.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14299
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
We avoid bare if conditions in Samba.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14299
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
We need to ensure we TALLOC_FREE(tmp_ctx) in every error exit.
Reported by Alex Kolesnik, ABISoft Ltd., www.abisoft.biz. Thanks!
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14299
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Wed Feb 26 23:11:39 UTC 2020 on sn-devel-184
* Bug 14270:
Samba 4.11 and later give incorrect results for SCOPE_ONE searches
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(master): Tue Feb 25 12:59:02 UTC 2020 on sn-devel-184
This changes the LDB behaviour in the combination of a SCOPE_ONE search and
an index returning less than 10 results.
After b6b5b5fe35 the list->strict flag
became set to false in all cases, rather than being left to the
value set by the caller.
This changes the ldb_kv_index_dn_one() code to force strict
mode on success instead.
Thanks to Marcus Granér, ICEYE Oy for reporting.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14270
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
TALLOC_FREE the ldb_control allocated in ldb_parse_control_from_string
when none of the cases match.
Credit to OSS-Fuzz
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: David Disseldorp <ddiss@samba.org>
This fix the following build failures:
[2466/3864] Linking bin/default/lib/ldb/ldbmodify
In file included from /home/buildroot/autobuild/instance-0/output-1/host/opt/ext-toolchain/lib/gcc/mips64el-buildroot-linux-uclibc/5.5.0/include/stdint.h:9:0,
from ../../lib/tevent/tevent.h:31,
from ../../lib/ldb/include/ldb.h:51,
from ../../lib/ldb/tests/test_ldb_dn.c:25:
/home/buildroot/autobuild/instance-0/output-1/host/mips64el-buildroot-linux-uclibc/sysroot/usr/include/stdint.h:122:27: error: conflicting types for 'uintptr_t'
typedef unsigned long int uintptr_t;
^
In file included from ../../lib/ldb/tests/test_ldb_dn.c:23:0:
/home/buildroot/autobuild/instance-0/output-1/host/mips64el-buildroot-linux-uclibc/sysroot/usr/include/cmocka.h:132:28: note: previous declaration of 'uintptr_t' was here
typedef unsigned int uintptr_t;
^
In file included from /home/buildroot/autobuild/instance-0/output-1/host/opt/ext-toolchain/lib/gcc/mips64el-buildroot-linux-uclibc/5.5.0/include/stdint.h:9:0,
from ../../lib/tevent/tevent.h:31,
from ../../lib/ldb/tests/ldb_key_value_test.c:48:
/home/buildroot/autobuild/instance-0/output-1/host/mips64el-buildroot-linux-uclibc/sysroot/usr/include/stdint.h:122:27: error: conflicting types for 'uintptr_t'
typedef unsigned long int uintptr_t;
^
In file included from ../../lib/ldb/tests/ldb_key_value_test.c:43:0:
/home/buildroot/autobuild/instance-0/output-1/host/mips64el-buildroot-linux-uclibc/sysroot/usr/include/cmocka.h:132:28: note: previous declaration of 'uintptr_t' was here
typedef unsigned int uintptr_t;
^
Fixes:
- http://autobuild.buildroot.org/results/9507739b3d5d51024ee9c60b74c2f85d5004e7e2
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14218
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Reviewed-by: Uri Simchoni <uri@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Fix ASAN detected use after free. No security implications as the
talloc_free is followed immediately by the print statement and the value
printed is an integer
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Gary Lockyer <gary@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Fri Dec 6 11:13:24 UTC 2019 on sn-devel-184
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Isaac Boukris <iboukris@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Dec 5 17:13:36 UTC 2019 on sn-devel-184
We are setting ldb_debug_string() as the ldb debug function. The context
we give it is only valid as long as we're in that function.
On failure, we jump to the teardown function. The pointer for
debug_string isn't valid anymore, but the ldb debug system still points
to that address, and when we store the location of the allocated string,
we overwrite memory, in that case something from talloc and a
talloc_free() then jumps into the nirvana.
Thanks to Florian Weimer who helped debugging this.
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Isaac Boukris <iboukris@samba.org>
The print functions used in Samba NULL terminate, but do not assume they will
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14049
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
This is beyond the normal level of clarity we expect in Samba, and is of course
rudundent, but this is a complex routine that has confusing tests, some of
pointers and some of boolean state values.
This tries to make the code as clear as possible pending a more comprehensive
rewrite.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14049
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14049
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14049
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Aug 27 01:16:33 UTC 2019 on sn-devel-184
This is a macro that sets the pointer to NULL after the talloc_free()
and is part of our standard coding practices.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Thankfully this only fails if the DB is corrupt and has a duplicate record.
The test was at the wrong end of the loop, and was for the
wrong boundary condition. A write after the end of the array would
occour before the condition was hit.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13695
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Fix the lmdb size test which ensures that databases > 4GiB can be
written by the lmdb backend. This test is not run as part of the normal
CI run as it exhausts the available disk on the test runners.
It was broken by changes to LDB allowing the lmdb map size to be
specified, and requiring GUID indexing by default.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The msg for each database record is allocated on the module context, but
never freed. The module seems like it could be a long-running context (as
the database would normally get repacked by the samba executable).
Even if it's not a proper leak, it shouldn't hurt to cleanup the memory.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14059
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Garming Sam <garming@samba.org>
Autobuild-Date(master): Tue Aug 20 04:57:10 UTC 2019 on sn-devel-184
Firstly, with Samba AD this looks a little weird because we log the same
message 5 times (once for every partition). If we log that we're doing
this to records in different partitions, hopefully someone with a little
Samba knowledge can figure out what's going on.
Secondly, the info about what partitions are actually changing might be
useful. E.g. if we hit a fatal error repacking the 3rd partition, and
the transaction doesn't abort properly, then it would be useful to know
what partitions were repacked and which ones weren't.
There doesn't appear to be a useful name for the partition
(ldb_kv->kv_ops->name() doesn't seem any more intelligible to a user),
so just log the first record that we update. We can use that to infer
the partition database).
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14059
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The "format 0x26011968" log confused me (and I'm a developer).
We can subtract the base offset from the pack format to get a more
user-friendly number, e.g. v0 (not actually used), v1, v2, etc.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14059
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The main reason is so that any future pack formats will continue
incrementing this number in a sequential fashion.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14059
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Store it on the repack context so that we can log a more informative
message "Repacking from format x to format y".
While this is not really a big deal currently, it could be worth
recording for potential future scenarios (i.e. supporting three or more
pack versions), where upgrades could potentially skip an intermediary
pack format version.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14059
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
LDB_DEBUG_WARNING gets logged by Samba as level 2, whereas the default
log level for Samba is 0. It's not really fair to the user to change the
format of their database on disk and potentially not tell them.
This patch adds a log with level zero (using a alias define, as this
technically isn't a fatal problem).
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14059
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
In case of a failing talloc_realloc(), the only reference
to the originally allocated memory is overwritten.
Instead use a temp var until success is verified.
Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Matthias Dieter Wallnöfer <mdw@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Use iterate_range kv op to skip the index section of the database when
running a full search. Quick local testing showed 18% improved throughput
on a full search with no results on a 50k database. With more results,
improvement is smaller but still noticeable.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Aug 2 02:29:42 UTC 2019 on sn-devel-184
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Wed Jul 10 05:48:52 UTC 2019 on sn-devel-184
There was no way to call ldb.open without evoking signal 11, so it is
unlikely anyone was using it.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Following the python/C convention for checking vs non-checking
convertors.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
In the Python/C API, conversion functions which check the types of their arguments
have names like:
double PyFloat_AsDouble(PyObject *pyfloat);
while conversion macros that don't check have names like:
PyFloat_AS_DOUBLE(pyfloat)
The pyldb_Ldb_AsLdbContext() macro looks like one of the checking functions
but it actually isn't. This has fooled us more than once. Here we fork
the macro into two -- one which performs checks and keeps the camel
case, and one with a shouty name that keeps the check-free behaviour.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
This will be used by pyldb_Ldb_AsLdbContext().
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
If you try to add a dn to itself, it expands as it goes. The resulting
loop cannot end well.
It looks like this in Python:
dn = ldb.Dn(ldb.Ldb(), 'CN=y,DC=x')
dn.add_base(dn)
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
* add ldb_options_get
add a function to get the options passed in ldb connect.
* add "batch_mode" option.
This options stops sub transactions being started for key value
operations. It is intended to improve the performance in batch
operations. As it bypasses the protections on operations if an
operation fails, the entire transaction will be aborted by a commit.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The previous code would override the caller with the DB size
estimate rather than allowing the caller to force the bigger size.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Test the the ldb "batch_mode" option sets batch mode operation.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
When performing a join the overhead of the sub transactions protecting
key value operations becomes significant. This commit adds a new
"batch_mode" option that disables the sub transactions around key value
operations.
The operation level index cache is also disabled, which means the
overall transaction level index cache can become inconsistent if an
operation fails. To protect against this and other possible on disk
inconsistencies, if any operation fails during a batch_mode
transaction the commit will fail and transaction will be rolled back.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This is needed for modules to access the ldb->options array, as this in in ldb_private.h
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
ldb_tdb can be a module, but the test is actually looking for ltdb_err_map() in
ldb_tdb_err_map.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Jul 4 03:51:58 UTC 2019 on sn-devel-184
Flag is used to enforce binary encoded attribute values per attribute.
Signed-off-by: Björn Baumbach <bb@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Allow the lmdb map size to be specified in the ldb option
"lmdb_env_size".
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* copy the ldb_options passed to ldb_connect onto the ldb_context,
making them more generally available.
* fix index buffering.
As a performance enhancement the indexes are cached in memory during a
transaction, and written to disk as part of the prepare commit. The
indexes could become corrupt in the event of a failed operation.
* fix read beyond buffer
Calling the "ldb_parse_tree" function with a filter consisting of
exactly a single space (" ") would trigger a read beyond the input
buffer.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13900
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Copy the options supplied to to ldb_connect, and place them on the
ldb_context. This allows backend options i.e. lmbd map size to be passed
cleanly from the callers.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Fixes:
lib/ldb/common/ldb.c:1091:3: warning: Value stored to 'ret' is never read <--[clang]
ret = 0;
^ ~
1 warning generated.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
This came about because in py2 we had to check for strings and unicode.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <noel.power@suse.com>
Autobuild-User(master): Noel Power <npower@samba.org>
Autobuild-Date(master): Mon Jun 24 18:48:53 UTC 2019 on sn-devel-184
We no longer need Samba to be py2/py3 compatible so we choose to return to the standard
function names.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <noel.power@suse.com>
We no longer need Samba to be py2/py3 compatible so we choose to return to the standard
function names.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <noel.power@suse.com>
We no longer need Samba to be py2/py3 compatible so we choose to return to the standard
function names.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <noel.power@suse.com>
We no longer need Samba to be py2/py3 compatible so we choose to return to the standard
function names.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <noel.power@suse.com>
We no longer need Samba to be py2/py3 compatible so we choose to return to the standard
function names.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <noel.power@suse.com>
We no longer need Samba to be py2/py3 compatible so we choose to return to the standard
function names.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <noel.power@suse.com>
We no longer need Samba to be py2/py3 compatible so we choose to return to the standard
function names.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <noel.power@suse.com>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Jun 21 05:45:02 UTC 2019 on sn-devel-184
As a performance enhancement the key value layer maintains a cache of
the index records, which is written to disk as part of a prepare commit.
This patch adds an extra cache at the operation layer to ensure that the
cached indexes remain consistent in the event of an operation failing.
Add test to test for index corruption in a failed modify.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
The callers will soon have two possible parents for this pointer, so
we need to remove this check, which was added out of caution given
the rather strange pattern of putting an active memory pointer into a
TDB (as a hash map).
That is, the only callers that did call this with "true" would
have to call this with "false", so just remove the complexity.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add test exercising the sub/nested transactions wrapping the key value
operations.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Use the nested transaction support added to the key value back ends to
make key value operations atomic. This will ensure that rename
operation failures, which delete the original record and add a new
record, leave the database in a consistent state.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add limited nested transaction support to the back ends to make the key value
operations atomic (for those back ends that support nested transactions).
Note: that only the lmdb backend currently supports nested transactions.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Remove the "DeprecationWarning: Please use assertEqual instead."
warnings from api.py
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add a test to document that ldb does not currently support nested
transactions.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Fixes csbuild error.
Error: COMPILER_WARNING:
lib/ldb/ldb_key_value/ldb_kv_search.c: scope_hint: In
function ‘search_func’ lib/ldb/ldb_key_value/ldb_kv_search.c:308:47:
warning: unused parameter ‘ldb_kv’ [-Wunused-parameter]
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Jun 13 08:27:12 UTC 2019 on sn-devel-184
Fixes csbuild error.
Error: COMPILER_WARNING:
lib/ldb/ldb_key_value/ldb_kv_index.c: scope_hint: In function
‘ldb_kv_index_add1’
lib/ldb/ldb_key_value/ldb_kv_index.c:2601:15: warning: comparison of
integer expressions of different signedness: ‘int’ and ‘unsigned int’
[-Wsign-compare]
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Fixes csbuild error.
Error: COMPILER_WARNING:
lib/ldb/ldb_key_value/ldb_kv_index.c: scope_hint: In function
‘ldb_kv_index_traverse_store’
lib/ldb/ldb_key_value/ldb_kv_index.c:779:60: warning: unused parameter
‘tdb’ [-Wunused-parameter]
By tagging the tdb_context parameter in ldb_kv_index_traverse_store as
_UNUSED_
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Fixes csbuild error.
Error: COMPILER_WARNING:
lib/ldb/ldb_key_value/ldb_kv_index.c: scope_hint: In function
‘ldb_kv_key_dn_from_idx’
lib/ldb/ldb_key_value/ldb_kv_index.c:545:15: warning: comparison of
integer expressions of different signedness: ‘int’ and ‘unsigned int’
[-Wsign-compare]
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Add lib/util/attr.h to the ldb distribution, to make the _UNUSED_ macro
available.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Fixes:
lib/tevent/tevent_wrapper.c:295:3: warning: Access to field 'next' results in a dereference of a null pointer (loaded from field 'prev') <--[clang]
Additionally fix similar instance of the same macro
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Gary Lockyer gary@catalyst.net.nz
Calling the "ldb_parse_tree" function with a filter consisting of
exactly a single space (" ") would trigger a read beyond the input
buffer. A unittest is included.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13900
Signed-off-by: Michael Hanselmann <public@hansmi.ch>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Tim Beale <timbeale@catalyst.net.nz>
* Default pack format changed to version 2 for GUID Indexed databases
(essentially Samba's sam.ldb). This means faster unpacking for records
with lots of attributes.
* Enforce link between all new database features, following toggling of
GUID indexing. If user toggles GUID indexing on/off, target pack format
is changed. Likewise the special ORDERED_INTEGER index format is used
(when GUID indexing on) or behaves like INTEGER (when GUID indexing off).
* Database repacked if packing format not as expected.
* In the case of MDB, since GUID indexing is mandatory, feature toggling
provided by pack_format_override LDB option.
* Check for errors from ldb_unpack_data() in ldb_tdb
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed May 29 05:56:59 UTC 2019 on sn-devel-184
For TDB databases, toggling GUID indexing mode will also toggle
pack format version 2. This provides a convenient downgrade path for
Samba databases, but the process doesn't work for MDB databases because
GUID indexing cannot be disabled when the MDB backend is used. This patch
addresses that corner case by providing support for a pack_format_override
option which will force the database to use pack format version 2.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>
Python test to confirm that after enabling or disabling GUID indexing,
the database is repacked on the next transaction with V1 if GUID
indexing was disabled, or V2 if it was enabled.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
VERY IMPORTANT PATCH
Now that we have a new packing format, we need to enable it by repacking
the database. We've decided to link all new database features together,
so once GUID indexing is enabled, the database will be repacked with
version 2 format. Repacking is done following the same iterate pattern as
reindexing.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Only enable ORDERED_INTEGER and index_format_fn functionality if GUID
indexing is enabled. This is in line with the rest of ldb_kv which binds
the new ORDERED_INTEGER to GUID indexed databases, and allows a practical
way to create the old index format (by disabling the GUID index).
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>
We need to check for the errors given by ldb_unpack() et al by preserving
the error code from kv_ctx->parser() called by tdb_parse_record() in
ltdb_parse_record().
Otherwise we will silently accept corrupt records and segfault later.
Likewise new pack formats will confuse the parser but not be
detected except by the incomplete struct ldb_message.
With this patch, the user will see a message like:
Invalid data for index DN=@BASEINFO
Failed to connect to 'st/ad_dc/private/sam.ldb' with backend 'tdb': Unable to load ltdb cache records for backend 'ldb_tdb backend'
Failed to connect to st/ad_dc/private/sam.ldb - Unable to load ltdb cache records for backend 'ldb_tdb backend'
This can be refined in the future by a specific check for
pack format versions in a higher caller, but this much is
needed regardless to detect corrupt records.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13959
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Parse errors aren't passed up correctly by the tdb backend. This
patch modifies a test to expose the issue, next patch will fix it.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13959
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Ensure that the referrals returned in a search request use the same
scheme as the request, i.e. referrals recieved via ldap are prefixed
with "ldap://" and those over ldaps are prefixed with "ldaps://"
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12478
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri May 24 05:12:14 UTC 2019 on sn-devel-184
For testing we need to know the actual KV level key of records and each
record's pack format version. This patch makes ldbdump add comments with
that info. We will parse it out in python tests.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed May 22 05:58:17 UTC 2019 on sn-devel-184
In future commits we'll be adding more logging to LDB, which breaks the
ldb_key_value_test suite. By removing the debug handler, a bug
involving an expired debug_string variable being written to is avoided.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
We want to reuse the reindex context struct for repacking, but it has an
unnecessary module pointer on it. Turns out the existing code doesn't
need it either, so this patch deletes the pointer.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
* Checking pack format is version 1 and erroring if not (will change soon)
* Pack format routines for unpack and pack version 2 (but not used)
* Test fixes for issues caused by upcoming repack functionality for upgrade
* Making ldbdump print out pack format info and keys so we have low level visibility for testing in python
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Pack function for new pack format with values separated from other data
so that while unpacking, the value section (which is probably large)
doesn't have to be loaded into cache/memory.
The new format is disabled for now.
Two tests are added that operate on a detailed binary breakdown of the
new format.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Unpack function for new pack format with values separated from other
data so that while unpacking, the value section (which is probably large)
doesn't have to be loaded into cache/memory.
Additionally, width of length field can now vary per-element to save space.
The old unpack routine is still present and is called if the old pack
format version number is found.
LDB torture suite is modified to run relevant tests on both old and new
pack format.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Since we're about to introduce a new packing format, it's a good time to
improve our code style and change some magic numbers into explicit
constants.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Replacing push and pull functions (which may cause issues with Undefined
Sanitizer) with Andreas Schneider's excellent macros which are a work in
progress and not yet merged into master. Once his work is upstream, I'll
rebase and change this code to import his headers.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
We will be adding a new packing format in forthcoming commits and there
may be more versions in the future. We need to make sure the database
contains records in a format we know how to read and write.
Done by fetching the @BASEINFO record and reading the first 4
bytes which contain the packing format version.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
To avoid warning above produced by using
-Wcast-function-type we;
+ ensure PyCFunctions of type METH_NOARGS defined dummy arg
+ ensure PyCFunctions of type METH_KEYWORDS use PY_DISCARD_FUNC_SIG
macro
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
* Rework pack routines to remove unused features:
- ldb_unpack_data_only_attr_list_flags() is removed
- LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC is now always implied
This improves the unpack performace significantly.
* Improve search performance via new internal flag
LDB_UNPACK_DATA_FLAG_READ_LOCKED which removes a memdup
* Improve search performance during full scan by ignoring
index records early.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Unpack functions currently take an attribute list to restrict the set of
attributes to be returned in the constructed message. This
functionality is never used and complicates implementation of
forthcoming new pack format. This patch removes that functionality.
Using the unpack function then filtering the result turns
out not to be any slower.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
This patch moves ldb_kv's filter code into the pack code to replace
'only attr list' functionality which will be removed in forthcoming
commit. Unpacking data then filtering the result is not any slower
than the removed 'only attr list' approach.
'only attr list' test repurposed to test unpack -> filter flow.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Optimising filter_attrs by removing msg and dn allocation/copying. The
caller can construct the msg and possibly steal the dn.
Also giving the function an ldb for future use.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Making unpack flag LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC required
behaviour, since allocating data during unpack is slow and unnecessary
in all current usages. In any future unpack usage, if editing of
returned memory is required, some function that duplicates the message
should be used, such as one of the filter_attrs functions, or msg_copy.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
This more modern routine allocates a nice talloc tree.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
The DN is now exploded so as to improve other aspects of the search handling.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Previously we half-heartedly checked one end.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
ldb_msg_new() is currently the same as talloc_zero(), but it might
not always be.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
It is unlikely to be NULL, since we're in the callback.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
We don't want to see this:
python3 -c "import sys
sys.path.insert(0, 'bin/python')
import ldb
m = ldb.Message()
e = ldb.MessageElement('q')
try:
m.add(e)
except ldb.LdbError:
pass
print(m)
"
Segmentation fault (core dumped)
instead we want this:
Traceback (most recent call last):
File "<string>", line 7, in <module>
ValueError: The element has no name
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <npower@samba.org>
Autobuild-User(master): Noel Power <npower@samba.org>
Autobuild-Date(master): Tue Apr 23 19:03:35 UTC 2019 on sn-devel-144
We have the ldb_kv in the caller, just fill it into the context and
so avoid the cost of the talloc_get_type().
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Apr 11 05:25:02 UTC 2019 on sn-devel-144
This makes LDAP bind significantly faster in the case of having many
members, due to large size of these records (with tens of thousands of
member links). During the nested group calculation, you are only
interested in memberOf not the member links.
(We add a bit-field to determine whether or not the backend actually
supports pointing into database memory. For some reason TDB pointers
aren't stable, so for now we set this option just on LMDB backends.)
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
@ records like @IDXLIST are only available via a base search on the specific name
but the method by which they were excluded was expensive, after the unpack the
DN is exploded and ldb_match_msg_error() would reject it for failing to match the
scope.
This uses the fact that @ records have the DN=@ prefix on their TDB/LMDB key
to quickly exclude them from consideration.
Based on analysis by Garming Sam.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13893
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Apr 10 06:23:39 UTC 2019 on sn-devel-144
* Version bump for adding index_format_fn to the schema syntax structure.
* Range index support added, allowing <= and >= operations to be indexed
* Improved reindex performance by setting the in-memory TDB hash size correctly
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
We need to make sure that duplicates are correctly returned (uSNChanged
for instance is UNIQUE but, we should be able to index on attributes
which are not unique).
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Testing max, min and negative values for indexed 32 and 64 bit types.
This has to be done in two different files because the 64 bit type is
LDB_SYNTAX_INTEGER which is implemented at the ldb level, while the 32
bit is added in the ldb-samba module. Schema syntax binding added for
ldb-samba.
We also need to make sure that full scans are not invoked for LMDB.
Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Adding ordered integer proto schema handling in kv index cache. This
allows ordered 64 bit integers to be used in cached fields like
@ATTRIBUTES
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
We had to devise our own scheme for writing integers in a human readable
format which also sorted correctly numerically. This might look a bit
confusing to outsiders, so here's a large comment as a peace offering.
Pair-programmed-with: Tim Beale <timbeale@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Activating <= and >= mdb indexing in samba for int32 and int64 attributes by:
1. Adding index_format_fn to LDB_SYNTAX_SAMBA_INT32 in ldb_samba
2. Cloning the 64bit LDB_SYNTAX_INTEGER type as LDB_SYNTAX_ORDERED_INTEGER
3. Adding index_format_fn to the new type
4. Modifying LargeInteger use the new type in samba schema
5. Bumping the index version to trigger reindexing
Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The reason we needed it in the first place was that the original
canonicalize is being used for non-index functions and it never produced
the right order originally (at least for integers).
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
It makes no difference in our standard case because \0 will always go
before any value for our index_format_fn, but this is better for
correctness (in case we do mess up our NUL terminations elsewhere).
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Full implementation of <= and >= indexed searching using iterate_range
backend operation. Adds index_format_fn to ldb_schema_syntax so
requires an ABI version bump. The function must be provided for any
type for which <= and >= indexing is required, and must return a
lexicographically ordered canonicalization of a value. This causes
index entries to be written in correct order to the database, so
iterate_range on the index DNs can be used.
ldb_kv_index_key is modified to return an index DN with attribute name
but without value if an empty value is provided. This is needed for
constructing keys that match the beginning or end of an index DN range.
Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Adding iterate_range to LDB API and implementing in LMDB. This
operation takes a start_key and end_key and returns all records between
the two, inclusive of both. This will be used to implementing indexing
for <= and >= expressions.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
If ldb_kv_filter_attrs() fails, we don't know that the dn of filtered_msg
is OK.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Apr 5 05:46:55 UTC 2019 on sn-devel-144
Set the default index cache from the passed option
"transaction_index_cache_size" on open. This allows the default cache
size to be overridden when processing large transactions i.e. joining a
large domain.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Set the index cache size to the number of records in the databse when
reindexing.
This significantly improves reindex performance. For a domain with
100,000 users the reindex times are reduced from 17 minutes to 45
seconds.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add the get_size method to the ldb_key_value layer, this will allow the
reindexing code to get an estimate of the number of records in the
database.
The lmdb backend returns an accurate count of the number of records in
the database withe the mdb_env_stat call.
The tdb backend does not provide a low cost method to determine the
number of records on the database. It does provide a tdb_summary call
however this this walks the entire database.
So for tdb we use the map size divided by 500, this over estimates the counts
for small domains, but the extra memory allocated for the cache should
not be significant.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Pass the index cache size to ldb_kv_index_transaction_start. This will
allow it to be set for reindex and join operations, where the current
defaults result in a significant performance penalty on large databases.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Remove the lazy initialisation of the index cache. This make setting
the size of the cache for re-indexing easier, which will be done in
later commits.
Performance testing shows that the removal of lazy initialisation makes
no appreciable difference to performance.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* Remove Python 2.x support except to build just the bare C library
* Remove --extra-python (a build time mode to produce Python2 and Python3
bindings at the same time)
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Mar 21 05:08:49 UTC 2019 on sn-devel-144
We do this by removing the confusing mandatory option to
conf.SAMBA_CHECK_PYTHON{,_HEADERS}(), instead just use the value of
--disable-python internally
This follows the default minimum of Python 3.4 and keeps things consistent
with the main Samba build where --disable-python is required to skip building
python bindings.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
These are no longer used by the build system so avoid
confusion by removing them from the tree.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This option is quite invasive in waf and was mainly for the python3 transition.
Testing with multiple python versions can be done by testing a full compile against
multiple versions, likewise multiple different binding versions can be created
the same way.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
