IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
With the changes to make samba python code Py2/Py3 compatible there
now are many instances where string content is decoded.
Decoded string variables in Py2 are returned as the unicode type. Many
Py2 c-module functions that take string arguments only check for the
string type. However now it's quite possibe the content formally passed
as a string argument is now passed as unicode after being decoded,
such arguments are rejected and code can fail subtly. This only affects
places where the type is directly checked e.g. via PyStr_Check etc.
arguments that are parsed by ParseTuple* functions generally already
accept both string and unicode (if 's', 'z', 's*' format specifiers
are used)
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Fri Apr 6 14:58:48 CEST 2018 on sn-devel-144
Add tests for the behaviour the ldb layer expects the key value layer to
provide. This should make it easier to add another KV store
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This avoids magic numbers and also is careful against overflow
from a long attr_for_dn.
This is done as a distinct commit to make the previous behaviour
change more clear, and to show that this does not change the
calculations, only improves the overflow check.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
These are tests are specifically for when the GUID index is not in use
which is always in with ldb_mdb.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13335
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Apr 5 07:53:10 CEST 2018 on sn-devel-144
The re-index traverse can abort part-way though and we need to ensure
that the transaction is never committed as that will leave an un-useable db.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13335
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Mar 8 14:14:37 CET 2018 on sn-devel-144
Without the destructor firing, this test used to pass, but now we show
that we must be able to open a new ldb handle.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Mar 7 04:38:02 CET 2018 on sn-devel-144
This allows sharing of the originally ldb_tdb operations to the new
ldb_mdb backend.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Mar 6 01:39:16 CET 2018 on sn-devel-144
This can handle both read-only and writable traverses.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This will allow us to change the backend from tdb to lmdb.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
With no schema syntax, this would occasionally crash as it dereferenced
some possibly NULL sequence of memory.
Note: Removing all tests except this one, made it crash reliably.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Tests for the index truncation code as well as the GUID index
format in general.
Covers truncation of both the DN and equality search keys.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Mar 3 09:58:40 CET 2018 on sn-devel-144
Fold together two identical cases to simplify the code.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Rather than add many special cases, over-long unique values are simply banned.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
In the truncation case a duplicate is perfectly expected.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Modify the indexing code to handle a maximum key length, index keys
greater than the maximum length will be truncated to the maximum length.
And the unuque index code has been altered to handle multiple records
for the same index key.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
It is not the job of the index code to enforce this, but do give a
a warning given it has been detected.
However, now that we do allow it, we must never return the same
object twice to the caller, so filter for it in ltdb_index_filter().
The GUID list is sorted, which makes this cheap to handle, thankfully.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Allow the setting of the maximum key length, this allows the testing of
index key truncation code. Index key truncation is required to allow
the samba indexing scheme to be used with backends that enforce a
maximum key length.
This will allow emulation of a length-limited key DB for testing.
This is a testing-only feature, as the index format changes
based on this value.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* Expose the SHOW_BINARY, ENABLE_TRACING and DONT_CREATE_DB flag constants
in the python api.
* Extend dn.is_child_of() test.
* Don't load LDB_MODULESDIR as a module file.
* Fix binary data in debug log (bug #13185).
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Wed Feb 28 04:54:21 CET 2018 on sn-devel-144
Tests to ensure:
When duplicate objects are added, the GUID was printed in the debug log
are passed through the escape function.
And that duplicate DN's do not generate debug log entries.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13185
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Feb 26 07:29:49 CET 2018 on sn-devel-144
When duplicate objects were added, the GUID was printed in the debug log
The GUID was not escaped and therefore displayed as binary content.
This patch splits out the duplicate DN creation error and the duplicate
GIUD error. Duplicate DN's are a normal event and don't require debug
logging.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13185
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Fix up tests that were performing a null check on the wrong variable
after a call to ldb_msg_new
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Feb 24 15:50:35 CET 2018 on sn-devel-144
We are setting modules directory here(LDB_MODULESDIR) so treat it this
way, no need to attempt to load it as a module file.
Signed-off-by: Timur I. Bakeyev <timur@iXsystems.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Uri Simchoni <uri@samba.org>
Add this test so ensure that this (unclear) behaviour does
not change silently.
Signed-off-by: Björn Baumbach <bb@sernet.de>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Expose the SHOW_BINARY, ENABLE_TRACING and DONT_CREATE_DB flag constants
in the python api.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* Intersect the index from SCOPE_ONELEVEL with the index for the search expression
(bug #13191)
* smaller/greater comparison tests
* Show the last successful DN when failing to parse LDIF
* ldb_index: Add an attriubute flag to require a unique value.
* silence some clang warnings in picky developer mode
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
This helps ensure we do not have to scan all objects at this level
which could be very many (one per DNS zone entry).
However, due to the O(n*m) behaviour in list_intersect() for older
databases, we only do this in the GUID index mode, leaving the behaviour
unchanged for existing callers that do not specify the GUID index mode.
NOTE WELL: the behaviour of disallowDNFilter is enforced
in the index code, so this fixes SCOPE_ONELEVEL to also
honour disallowDNFilter, hence the additional tests.
The change to select the SUBTREE index in the absense of
the ONELEVEL index enforces this.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13191
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The comparison result has been ignored, which is not good. Also remove
the "ldbsearch" command in the error branch which has not much sense.
The scripts needs to be run through test-tdb.sh, test-ldap.sh or
test-sqlite3.sh which I didn't realise before. Hence less changes are needed
and this is a reduced version of the patch published on the mailing list.
Signed-off-by: Matthias Dieter Wallnöfer <mdw@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Matthias Dieter Wallnöfer <mdw@samba.org>
Autobuild-Date(master): Tue Dec 19 03:09:12 CET 2017 on sn-devel-144
Add attribute flag LDB_FLAG_INTERNAL_UNIQUE_VALUE, to request that the
added attribute is unique on the index.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13004
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Avoid const in casting since it doesn't increase code
safety in this case and causes clang to generate const-qual
warning. Also initialize a pointer to NULL to silence clang
uninitialized variable warning.
Signed-off-by: Uri Simchoni <uri@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* GUID Index support.
NOTE: When activated by setting @IDXGUID in the @INDEXLIST dn, all
entries in the DB are re-keyed in a way that is NOT visible to
ldb 1.2.2 and earlier. To re-key back to the previous format, remove
the @IDXGUID attribute from @INDEXLIST using ldb 1.2.2 or later.
(ldb 1.2.2 can re-key, but not otherwise read, the new DB format).
* Give LDB_ERR_CONSTRAINT_VIOLATION, not LDB_ERR_ENTRY_ALREADY_EXISTS
when a duplicate value is detected in a unique index
* Print status information during a > 10,000 entry re-index
(as this can be slow)
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Sep 23 01:24:19 CEST 2017 on sn-devel-144
This avoids loading any second index for these cases.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
The python TestCase API will keep a reference to the test object until the end
of the tests, long after we need the actual LDB or the fd.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
An ldb context is valid without a backing file for tests of ldb.Message and ldb.MessageElement
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This is not actually a great test, as the filter would
fail to match these anyway, but it at least checks the
codepath is safe.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
A re-index of 10,000 entries is slow enough and rare enought that we can
justify the message being at LDB_DEBUG_WARNING as otherwise the administrator
will be sure the "lockup" was one.
The default for ldb is to print LDB_DEBUG_WARNING in comand-line tools
and the default for Samba is to log it at level 2.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The ltdb->cache->attribute_indexes test is not correct with the GUID index mode
so for consistency remove it. This will make re-index on a large un-indexed
database slower, but that is better than making the wrong choice on a large
GUID-indexed database.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This allows easy testing of our unique index code and behaivour from python
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This was (unintentionally) disabled by
6ef6182554 in 2006.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This is important with the GUID index, as a DN lookup is much more common now.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The casefolding of a DN is one of the more expensive and pointless things in LDB
operation. The ldb_dn abstraction works hard to avoid duplicating this work, but
we can work harder to save that information.
Here we copy in the DN, that has been casefolded already for the index,
and keep that as the returned DN, after stripping any extended components.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This ensures we cover the case where the DN does not match the DB exactly
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This should ensure that the upper or lower case the user chooses does not impact
on the filtering, at least for database that have checkBaseOnSearch set.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
We will never have more results than is in either list or list2.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This change ensures we walk the short list and look up into the longer of the two lists.
ltdb_dn_list_find_val() will do a binary search for the GUID case.
Before GUID indexes this was O(n*m), now it is O(n*log(m)).
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This only works if we have the GUID index format, as otherwise these are unsorted.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This allows the binary search to still operate on the list, even after
a or operator in the search expression
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This allows us to merge the lists finding common values.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The LDB_ERR_ENTRY_ALREADY_EXISTS error code is detected in repl_meta_data as indicating
that the DN exists, and that a conflict record should be created.
This is really a constraint violation, not a duplicate record.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This will give us errors earlier if the index code becomes broken
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This is unlikely, but when it happens it will be really painful to debug.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This allows all the previous patches to be enabled.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This connects the GUID based index records to GUID based TDB keys.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This avoids re-checking the fetched DN against the scope
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This will allow a common point to parse index records into a TDB key,
allowing them to be a GUID or DN in the future
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This would be pointless and we no longer query for it.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The objectGUID (or similar) is already the record key, there is
no need to index it to itself.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
When we have the full ldb_message we can read the objectGUID as the TDB key
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This allows the optional use of GUID based TDB key.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This follows modern Samba coding style where memory
returned is allocated on a supplied memory context.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This allows us to format the TDB key as DN=GUID=f7c953ee-cf9c-433f-b423-21ce04d09591
and so be compatible with an un-indexed search and a re-index with an old ldb.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This allows us to use the ldif_write function later to create a string GUID for the TDB key.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The @IDXGUID attribute in the @INDEXLIST will be objectGUID
in Samba.
The @IDX_DN_GUID attribute in the @INDEXLIST will be GUID
in Samba.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
With the GUID index option, the values in the index result list may
not be a DN but the objectGUID. We look up the @IDXDN index with the
case-folded DN to get that if required.
We re-use the code from the SCOPE_BASE search do avoid duplication
and for reliablity.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This will allow the code to be re-used for storing the DN->GUID index
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This will allow use of a GUID TDB key in the future. While ltdb_search_base()
might be marginally slower than tdb_exists(), no allocation is done for the
attributes or DN, and renmaes are not a very common operation.
This allows a check if the target DN exists even when the direct DN -> key
link is broken.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The GUID index case can not directly use ltdb_key_dn() and tdb_exists() to
show that a records exists.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This allows, when enabled, the index record to contain (say) the objectGUID, not the DN
of the record.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This function is used to find an existing index value and this
change allows it to find the value by GUID rather than by DN once
the GUID index is enabled.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This allows the objectGUID, rather than the DN, to be the index key
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This will make it easier to delete records with the GUID TDB key
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Nothing reads these currently, but we should refuse to load a mixed up index
in the future
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This packing should be more efficient to read than the ldb_pack format.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This replaces dn_list_cmp() with functions that do not attempt to
to care about string termination. All index values are case sensitive
and correctly length-bound already, even for a DN index
so just use a length check and memcmp()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This would totally break our index scheme if this could be modified.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This will be used to determine if we are in GUID index mode
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This will make it easier to switch the GUID index mode on and off
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This allows us to slowly split out the tdb key in the DB from being the DN
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This will allow changing to a GUID tdb key in the future
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This will in time allow us to generate a TDB key from
the msg, eg from an objectGUID.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This avoids an extra DB lookup for the base, when that is the only
record we will return, and avoids going into the index code for
a base search, as that won't work for special DNs once the GUID
index mode is enabled.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
* Bug #13033 LDB open with LDB_FLG_RDONLY can cause the database
to fail to open
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13033
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
As the kernel is no longer enforcing the read-only DB
add some tests.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13033
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
We support opening and LDB multiple times in a process, but do not support this in tdb.
As we can open the ldb with different flags, we must ensure a later read-write
open is possible.
Additionally, a read-only TDB will refuse the all-record lock, preventing
the ldb from even loading.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13033
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This is a serious condition, and should be logged.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13033
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
The ltdb_lock_read() routine did not return an LDB error code, but -1.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13033
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
* Bug #13017: Add ldb_ldif_message_redacted_string() to allow debug
of redacted log messages, avoiding showing secret values
* Bug #13015: Allow re-index of newer databases with binary GUID TDB keys
(this officially removes support for re-index of the original
pack format 0, rather than simply segfaulting).
* Avoid memory allocation and so make modify of records in ldb_tdb faster
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This is designed to be a drop in replacement for
ldb_ldif_message_string() while better protecting privacy.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13017
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
These are not found on any AD DC, and would segfault previous LDB
versions.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13015
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
If backported, this allows old ldb versions to full-search and re-index newer databases
and in current code allows the transition to and from a GUID or incrementing ID based index
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13016
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
We want to rename the objects, then scan looking for the index values.
This avoids a DB modify during the index scan traverse (the index values
are actually added to an in-memory TDB, written in prepare_commit()).
This allows us to remove the "this might already exist" case in the
index handling, we now know that the entry did not exist in the index
before we add it.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13015
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
The keys may not always be a null terminated string, they could well
be a binary GUID in a future revision, for efficiency..
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13016
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Aug 30 14:58:32 CEST 2017 on sn-devel-144
This avoids duplicate code and allows us to use the allocation-avoiding
LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC flag.
We can not use LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC as el2->values
is talloc_realloc()ed in the routine.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Aug 29 11:13:50 CEST 2017 on sn-devel-144
This will allow us to avoid a full unpack in situations where we just want to confirm
if the DN exists
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
ldb modules are not (yet) unloaded and are only loaded once (there is a check
that makes sure of this). Allocate off the NULL context. We never want this
to be freed until process shutdown. If eventually we add the ability to
unload ldb modules we can add a deregister function that walks and frees the list.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12932
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* Bug #12882: Do not install _ldb_text.py if we have system libldb
* Use libraries from build dir for testsuite
* Bug #12900: Fix index out of bound in ldb_msg_find_common_values
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Sat Jul 22 03:46:25 CEST 2017 on sn-devel-144
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Fri Jul 7 20:10:37 CEST 2017 on sn-devel-144
cmocka unit test failed on i386
[==========] Running 2 test(s).
[ RUN ] test_ldb_msg_find_duplicate_val
[ OK ] test_ldb_msg_find_duplicate_val
[ RUN ] test_ldb_msg_find_common_values
[ FAILED ] test_ldb_msg_find_common_values
[==========] 2 test(s) run.
[ ERROR ] --- 0x14 != 0
[ LINE ] --- ../tests/ldb_msg.c:266: error: Failure!
[ PASSED ] 1 test(s).
[ FAILED ] 1 test(s), listed below:
[ FAILED ] test_ldb_msg_find_common_values
1 FAILED TEST(S)
But we were just lucky on other platforms because there is
index out of bound according to valgrind error.
==3298== Invalid read of size 4
==3298== at 0x486FCF6: ldb_val_cmp (ldb_msg.c:95)
==3298== by 0x486FCF6: ldb_msg_find_common_values (ldb_msg.c:266)
==3298== by 0x109A3D: test_ldb_msg_find_common_values (ldb_msg.c:265)
==3298== by 0x48E7490: ??? (in /usr/lib/libcmocka.so.0.4.1)
==3298== by 0x48E7EB0: _cmocka_run_group_tests (in /usr/lib/libcmocka.so.0.4.1)
==3298== by 0x1089B7: main (ldb_msg.c:352)
==3298== Address 0x4b07734 is 4 bytes after a block of size 48 alloc'd
==3298== at 0x483223E: malloc (vg_replace_malloc.c:299)
==3298== by 0x4907AA7: _talloc_array (in /usr/lib/libtalloc.so.2.1.9)
==3298== by 0x486FBF8: ldb_msg_find_common_values (ldb_msg.c:245)
==3298== by 0x109A3D: test_ldb_msg_find_common_values (ldb_msg.c:265)
==3298== by 0x48E7490: ??? (in /usr/lib/libcmocka.so.0.4.1)
==3298== by 0x48E7EB0: _cmocka_run_group_tests (in /usr/lib/libcmocka.so.0.4.1)
==3298== by 0x1089B7: main (ldb_msg.c:352)
Signed-off-by: Lukas Slebodnik <lslebodn@redhat.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
_ldb_text.py is installed as part of the ldb package and also if you
compile Samba with the system ldb version. This way we have have the
file twice in the same location and run into file confilcts.
This has already been fixed some time ago:
60dc26bfe1
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12882
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
There was a failure when tests were executed after after extracting
ldb tarball.
sh$ make -j8 check
WAF_MAKE=1 PATH=buildtools/bin:../../buildtools/bin:$PATH waf test
ldbadd: error while loading shared libraries: libldb.so.1: cannot open shared object file: No such file or directory
cat: write error: Broken pipe
Traceback (most recent call last):
File "tests/python/api.py", line 10, in <module>
import ldb
ImportError: libldb.so.1: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "tests/python/api.py", line 10, in <module>
import ldb
ImportError: libpyldb-util.so.1: cannot open shared object file: No such file or directory
bin/ldb_tdb_mod_op_test: error while loading shared libraries: libldb.so.1: cannot open shared object file: No such file or directory
testsuite returned 1
Signed-off-by: Lukas Slebodnik <lslebodn@redhat.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Alexander Bokovoy <ab@samba.org>
* handle one more LDB_FLAG_INTERNAL_DISABLE_SINGLE_VALUE_CHECK
case in ldb_tdb
* fix ldb_tdb locking (performance) problems
* fix ldb_tdb search inconsistencies by adding
read_[un]lock() hooks to the module stack
(bug #12858)
* add cmocka based tests for the locking issues
* ldb_version.h provides LDB_VERSION_{MAJOR,MINOR,RELEASE} defines
* protect ldb_modules.h from being used by Samba < 4.7
Note: that this release (as well as 1.1.30 and 1.1.31)
may cause problems for older applications, e.g. Samba
See https://bugzilla.samba.org/show_bug.cgi?id=12859
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Samba versions before 4.7 are incompatible with the read_[un]lock()
behaviour introduced into ldb.
This makes sure older Samba versions fail to compile against
ldb >= 1.2.0.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12859
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This indicates what feature set Samba assumes from the used
libldb from the system.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12859
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The new ldb whole-db lock behaviour now allows this test
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
(ldb releases have been made while this patch set was in train)
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
We must hold locks not just for the duration of each search, but for the whole search
as our module stack may make multiple search requests to build up the whole result.
This is explains a number of replication and read corruption issues in Samba
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
This allows Samba to provide a consistent view of the DB
despite the use of multiple databases via the partitions module
and over multiple callbacks via a module stack.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
This will be used to implement read locking in ldb_tdb
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Currently, a lock is not held against modifications once the final
record is returned via a callback, so modifications can be made
during the DONE callback. This makes it hard to write modules
that interpert an ldb search result and do further processing
so will change in the future to allow the full search to be
atomic.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
A modify or rename during a search must not cause a search to change
output, and attributes having an index should in particular not see
any change in behaviour in this respect
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
This is required to pass the test in future, because
otherwise the clean up will fail because we hold locks.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
If we do not do this, then we never take the all record lock, and instead do a lock
for every record as we go, which is very slow during a large search
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
repl_meta_data knows whether linked attributes are appropriately
[un-]duplicated, and this is how it tells ldb_tdb that.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Pair-programmed-with: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* Add efficient function to find duplicate values in ldb messages
(this makes large multi-valued attributes in ldb_tdb more efficient)
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
ldb backends need to make sure they are not adding duplicate values to
multi-valued attributes in ADD and MODIFY operations. Until now they
have done this inefficiently using nested loops. Here we add common
functions that deal with large numbers of values in O(n log n) time,
but continue to use the simple methods for small numbers of values.
These functions take a struct ldb_context pointer and an options flag
arguments, although the ldb is not used, and only one bit of the
options has meaning. This is to allow further patches to switch on
schema-aware comparisons.
This entails an ABI jump to add the two new functions.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This helps make some future commits less confusing
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
