IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
This makes LDAP bind significantly faster in the case of having many
members, due to large size of these records (with tens of thousands of
member links). During the nested group calculation, you are only
interested in memberOf not the member links.
(We add a bit-field to determine whether or not the backend actually
supports pointing into database memory. For some reason TDB pointers
aren't stable, so for now we set this option just on LMDB backends.)
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
@ records like @IDXLIST are only available via a base search on the specific name
but the method by which they were excluded was expensive, after the unpack the
DN is exploded and ldb_match_msg_error() would reject it for failing to match the
scope.
This uses the fact that @ records have the DN=@ prefix on their TDB/LMDB key
to quickly exclude them from consideration.
Based on analysis by Garming Sam.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13893
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Apr 10 06:23:39 UTC 2019 on sn-devel-144
* Version bump for adding index_format_fn to the schema syntax structure.
* Range index support added, allowing <= and >= operations to be indexed
* Improved reindex performance by setting the in-memory TDB hash size correctly
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
We need to make sure that duplicates are correctly returned (uSNChanged
for instance is UNIQUE but, we should be able to index on attributes
which are not unique).
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Testing max, min and negative values for indexed 32 and 64 bit types.
This has to be done in two different files because the 64 bit type is
LDB_SYNTAX_INTEGER which is implemented at the ldb level, while the 32
bit is added in the ldb-samba module. Schema syntax binding added for
ldb-samba.
We also need to make sure that full scans are not invoked for LMDB.
Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Adding ordered integer proto schema handling in kv index cache. This
allows ordered 64 bit integers to be used in cached fields like
@ATTRIBUTES
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
We had to devise our own scheme for writing integers in a human readable
format which also sorted correctly numerically. This might look a bit
confusing to outsiders, so here's a large comment as a peace offering.
Pair-programmed-with: Tim Beale <timbeale@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Activating <= and >= mdb indexing in samba for int32 and int64 attributes by:
1. Adding index_format_fn to LDB_SYNTAX_SAMBA_INT32 in ldb_samba
2. Cloning the 64bit LDB_SYNTAX_INTEGER type as LDB_SYNTAX_ORDERED_INTEGER
3. Adding index_format_fn to the new type
4. Modifying LargeInteger use the new type in samba schema
5. Bumping the index version to trigger reindexing
Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The reason we needed it in the first place was that the original
canonicalize is being used for non-index functions and it never produced
the right order originally (at least for integers).
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
It makes no difference in our standard case because \0 will always go
before any value for our index_format_fn, but this is better for
correctness (in case we do mess up our NUL terminations elsewhere).
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Full implementation of <= and >= indexed searching using iterate_range
backend operation. Adds index_format_fn to ldb_schema_syntax so
requires an ABI version bump. The function must be provided for any
type for which <= and >= indexing is required, and must return a
lexicographically ordered canonicalization of a value. This causes
index entries to be written in correct order to the database, so
iterate_range on the index DNs can be used.
ldb_kv_index_key is modified to return an index DN with attribute name
but without value if an empty value is provided. This is needed for
constructing keys that match the beginning or end of an index DN range.
Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Adding iterate_range to LDB API and implementing in LMDB. This
operation takes a start_key and end_key and returns all records between
the two, inclusive of both. This will be used to implementing indexing
for <= and >= expressions.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
If ldb_kv_filter_attrs() fails, we don't know that the dn of filtered_msg
is OK.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Apr 5 05:46:55 UTC 2019 on sn-devel-144
Set the default index cache from the passed option
"transaction_index_cache_size" on open. This allows the default cache
size to be overridden when processing large transactions i.e. joining a
large domain.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Set the index cache size to the number of records in the databse when
reindexing.
This significantly improves reindex performance. For a domain with
100,000 users the reindex times are reduced from 17 minutes to 45
seconds.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add the get_size method to the ldb_key_value layer, this will allow the
reindexing code to get an estimate of the number of records in the
database.
The lmdb backend returns an accurate count of the number of records in
the database withe the mdb_env_stat call.
The tdb backend does not provide a low cost method to determine the
number of records on the database. It does provide a tdb_summary call
however this this walks the entire database.
So for tdb we use the map size divided by 500, this over estimates the counts
for small domains, but the extra memory allocated for the cache should
not be significant.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Pass the index cache size to ldb_kv_index_transaction_start. This will
allow it to be set for reindex and join operations, where the current
defaults result in a significant performance penalty on large databases.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Remove the lazy initialisation of the index cache. This make setting
the size of the cache for re-indexing easier, which will be done in
later commits.
Performance testing shows that the removal of lazy initialisation makes
no appreciable difference to performance.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* Remove Python 2.x support except to build just the bare C library
* Remove --extra-python (a build time mode to produce Python2 and Python3
bindings at the same time)
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Mar 21 05:08:49 UTC 2019 on sn-devel-144
We do this by removing the confusing mandatory option to
conf.SAMBA_CHECK_PYTHON{,_HEADERS}(), instead just use the value of
--disable-python internally
This follows the default minimum of Python 3.4 and keeps things consistent
with the main Samba build where --disable-python is required to skip building
python bindings.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
These are no longer used by the build system so avoid
confusion by removing them from the tree.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This option is quite invasive in waf and was mainly for the python3 transition.
Testing with multiple python versions can be done by testing a full compile against
multiple versions, likewise multiple different binding versions can be created
the same way.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
* Fix standalone build of ldb.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Feb 26 12:10:40 CET 2019 on sn-devel-144
When calling make from the ldb, talloc, tdb, and
tevent bundles, we need to first find the
location of the waf script. Currently the build
fails since it can't find waf.
Fixes regression caused by a660b7f.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* CVE-2019-3824 out of bounds read in wildcard compare (bug 13773)
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Gary Lockyer <gary@samba.org>
Autobuild-Date(master): Mon Feb 25 22:54:13 CET 2019 on sn-devel-144
Add cmocka tests for ldb_wildcard_match.
Running test_wildcard_match under valgrind reproduces
CVE-2019-3824 out of bounds read in wildcard compare (bug 13773)
valgrind --suppressions=lib/ldb/tests/ldb_match_test.valgrind\
bin/ldb_match_test
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
ldb_handler_copy and ldb_val_dup over allocate by one and add a trailing '\0'
to the data, to make them safe to use the C string functions on.
However testing for the trailing '\0' is not the correct way to test for
the end of a value, the length should be checked instead.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Check the operation type of the passed parse tree, and return
LDB_INAPPROPRIATE_MATCH if the operation is not LDB_OP_SUBSTRING.
A query of "attribute=*" gets parsed as LDB_OP_PRESENT, checking the
operation and failing ldb_wildcard_match should help prevent confusion
writing tests.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Initialise the created ldb_parse_tree with talloc_zero, this ensures
that it is correctly initialised if inadvertently passed to a function
expecting a different operation type.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
There is valgrind error in few tests tests/test-generic.sh
91 echo "Test wildcard match"
92 $VALGRIND ldbadd $LDBDIR/tests/test-wildcard.ldif || exit 1
93 $VALGRIND ldbsearch '(cn=test*multi)' || exit 1
95 $VALGRIND ldbsearch '(cn=*test_multi)' || exit 1
97 $VALGRIND ldbsearch '(cn=test*multi*test*multi)' || exit 1
e.g.
==3098== Memcheck, a memory error detector
==3098== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3098== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==3098== Command: ./bin/ldbsearch (cn=test*multi)
==3098==
==3098== Invalid read of size 1
==3098== at 0x483CEE7: memchr (vg_replace_strmem.c:890)
==3098== by 0x49A9073: memmem (in /usr/lib64/libc-2.28.9000.so)
==3098== by 0x485DFE9: ldb_wildcard_compare (ldb_match.c:313)
==3098== by 0x485DFE9: ldb_match_substring (ldb_match.c:360)
==3098== by 0x485DFE9: ldb_match_message (ldb_match.c:572)
==3098== by 0x558F8FA: search_func (ldb_kv_search.c:549)
==3098== by 0x48C78CA: ??? (in /usr/lib64/libtdb.so.1.3.17)
==3098== by 0x48C7A60: tdb_traverse_read (in /usr/lib64/libtdb.so.1.3.17)
==3098== by 0x557B7C4: ltdb_traverse_fn (ldb_tdb.c:274)
==3098== by 0x558FBFA: ldb_kv_search_full (ldb_kv_search.c:594)
==3098== by 0x558FBFA: ldb_kv_search (ldb_kv_search.c:854)
==3098== by 0x558E497: ldb_kv_callback (ldb_kv.c:1713)
==3098== by 0x48FCD58: tevent_common_invoke_timer_handler (in /usr/lib64/libtevent.so.0.9.38)
==3098== by 0x48FCEFD: tevent_common_loop_timer_delay (in /usr/lib64/libtevent.so.0.9.38)
==3098== by 0x48FE14A: ??? (in /usr/lib64/libtevent.so.0.9.38)
==3098== Address 0x4b4ab81 is 0 bytes after a block of size 129 alloc'd
==3098== at 0x483880B: malloc (vg_replace_malloc.c:309)
==3098== by 0x491048B: talloc_strndup (in /usr/lib64/libtalloc.so.2.1.15)
==3098== by 0x48593CA: ldb_casefold_default (ldb_utf8.c:59)
==3098== by 0x485F68D: ldb_handler_fold (attrib_handlers.c:64)
==3098== by 0x485DB88: ldb_wildcard_compare (ldb_match.c:257)
==3098== by 0x485DB88: ldb_match_substring (ldb_match.c:360)
==3098== by 0x485DB88: ldb_match_message (ldb_match.c:572)
==3098== by 0x558F8FA: search_func (ldb_kv_search.c:549)
==3098== by 0x48C78CA: ??? (in /usr/lib64/libtdb.so.1.3.17)
==3098== by 0x48C7A60: tdb_traverse_read (in /usr/lib64/libtdb.so.1.3.17)
==3098== by 0x557B7C4: ltdb_traverse_fn (ldb_tdb.c:274)
==3098== by 0x558FBFA: ldb_kv_search_full (ldb_kv_search.c:594)
==3098== by 0x558FBFA: ldb_kv_search (ldb_kv_search.c:854)
==3098== by 0x558E497: ldb_kv_callback (ldb_kv.c:1713)
==3098== by 0x48FCD58: tevent_common_invoke_timer_handler (in /usr/lib64/libtevent.so.0.9.38)
==3098==
# record 1
dn: cn=test_multi_test_multi_test_multi,o=University of Michigan,c=TEST
cn: test_multi_test_multi_test_multi
description: test multi wildcards matching
objectclass: person
sn: multi_test
name: test_multi_test_multi_test_multi
distinguishedName: cn=test_multi_test_multi_test_multi,o=University of Michiga
n,c=TEST
# returned 1 records
# 1 entries
# 0 referrals
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Lukas Slebodnik <lslebodn@fedoraproject.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett abartlet@samba.org
Autobuild-User(master): Noel Power <npower@samba.org>
Autobuild-Date(master): Wed Feb 13 14:51:12 CET 2019 on sn-devel-144
Every time I look at this file, I spend a few minutes wondering how
these bits of code are ever run. Never again.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
PyList_Append doesn't steal references, so if the item created is
a temp object, created just to be added to the list we need to
decref the item appended in order for it to be released.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Py_BuildValue when processing format 'O' will
'Pass a Python object untouched (except for its reference count,
which is incremented by one'
Basically this means if you are using a new reference to a PyObject
to pass to BuildValue (to be used with the 'O' format) the reference
*isn't* stolen so you really do need to DECREF it in order to ensure
it gets cleaned up.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
* pyldb: make ldb.connect() url mandatory
* New version number for master (Samba 4.11 eventually)
The 1.5.x series will be maintained in the v4-10-test branch
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Feb 1 07:02:56 CET 2019 on sn-devel-144
The call fails without it, so we might as well fail sooner
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The old name confused me because it's not really related to the
one-level index at all. It's the result from evaluating the indexed
search specified in the ac->tree.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13762
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Disclaimer: this is based on my limited understanding of what the code
is doing.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13762
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This comment was written before the GUID_index_attribute block of code
existed. So we now *do* load the index values and *do* check for a
strict intersect, so the comment is redundant.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13762
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Commit 88ae60ed18 introduced a problem that made one-level
searches inefficient if there were a lot of child objects in the same
level, and the requested object didn't exist. Basically, it ignored the
case where ldb_kv_index_dn() returned LDB_ERR_NO_SUCH_OBJECT, i.e. the
indexed lookup was successful, but didn't find a match. At which point,
there was no more processing we needed to do.
The behaviour after 88ae60ed18 was to fall-through and run the
ldb_kv_index_filter() function over *all* the children. This still
returned the correct result, but could be costly if there were a lot of
children.
The case 88ae60ed18 was trying to fix was where we could not do
an indexed search (e.g. trying to match on a 'attribute=*' filter). In
which case we want to ignore the LDB_ERR_OPERATIONS_ERROR and just run
ldb_kv_index_filter() over all the children. This is still more
efficient than the fallback of doing a full database scan.
This patch adds in a short-circuit for the NO_SUCH_OBJECT case, so we
can skip the unnecessary ldb_kv_index_filter() work.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13762
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Test failed on s390x but there is a simple reproducer for any
architecture.
The built-in function repr returns the canonical string representation
of the object. We needn't care about order attributes in string
representation. Therefore test should pass for any order.
for i in {1..30}; do
PYTHONHASHSEED=random \
python2 -c 'import ldb; msg = ldb.Message(); msg.dn = ldb.Dn(ldb.Ldb(), "dc=foo29"); msg["dc"] = b"foo"; print(repr(msg)) '
done
======================================================================
FAIL: test_repr (__main__.LdbMsgTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests/python/api.py", line 2322, in test_repr
"Message({'dn': Dn('dc=foo29'), 'dc': MessageElement(['foo'])})")
AssertionError: "Message({'dc': MessageElement(['foo']), 'dn': Dn('dc=foo29')})" != "Message({'dn': Dn('dc=foo29'), 'dc': MessageElement(['foo'])})"
----------------------------------------------------------------------
Ran 1025 tests in 29.146s
FAILED (failures=1)
Signed-off-by: Lukas Slebodnik <lslebodn@fedoraproject.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
* Build fixes
* dirsync: Allow arbitrary length cookies
(bug #13686)
* The build uses python3 by default:
* --extra-python would take python2 now
* To build with python2 only use:
PYTHON=python2 ./configure
PYTHON=python2 make
PYTHON=python2 make install
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Karolin Seeger <kseeger@samba.org>
While 'es' format works great for unicode (in python2) and
str (in python3) The behaviour with str (in python2) is unexpected.
In python2 the str type is (re-encoded) with the specified encoding.
In python2 the 'et' type would be a better match, that ensures 'str'
type is treated like it was with 's' (no reencoding) and unicode is
encoded with the specified encoding. However in python3 'et' allows
byte (or bytearray) params to be accepted (with no reencoding), we
don't want this. This patch adds a new PYARG_STR_UNI format code which
is a hybrid, in python2 it evaluates to 'et' and in python3 'es' and
so gives the desired behaviour for each python version.
Additionally remove the associated known fail.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13616
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Sun Jan 13 03:53:00 CET 2019 on sn-devel-144
This test should demonstrate an error with the 'es' format in python
where a 'str' byte-string is passed (containing utf8 encoded bytes)
with some characters that cannot be decoded as ascii. The same
code if run in python3 should generate an error (needs string not
bytes)
Also Add knownfail for ldb.Dn passed utf8 encoded byte string
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Replacing paged results module to use GUID list instead of storing
result list in memory, in order to improve memory performance.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This will require memset_s() because of a later commit moving
ZERO_STRUCT to use memset_s().
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This avoids creating a mini-configure in the configure script.
Users wishing to use python2 to build need to specify PYTHON=
to both ./configure and make
After we merged the python3 change, it became clear that relying on systems prefixing
the correct python just causes trouble and make debugging harder, so only use $PYTHON
for the override, not the default case
This essentially reverts a660b7fb8e but
leaves the files more consistent.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Noel Power <npower@samba.org>
This works around python3 having a new hash seed each time it starts to allow
a second "make" not to rebuild the world.
This should probably be reverted once we find the hash that is causing
the issue, but should reduce frustration for now.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Make sure default make and configure for all now defaults
to building with python3.
To build a samba (or sub component e.g. talloc etc.) with python3
./configure && make
To build a samba (or sub component e.g. talloc etc.) with python2
PYTHON=python ./configure && PYTHON=python make
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
waf code is making assumptions that main build is py2 and extra build
is py3, this results in wrong library names being used.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The length of the cookie is proportional to the number of DCs ever in
the domain (as it stores the uptodateness vector which has stale
invocationID).
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13686
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
../lib/ldb/pyldb.c: In function ‘PyDict_AsMessage’:
../lib/ldb/pyldb.c:90:22: error: initialization discards ‘const’ qualifier from pointer target type [-Werror=discarded-qualifiers]
#define PyStr_AsUTF8 PyUnicode_AsUTF8
^~~~~~~~~~~~~~~~
../lib/ldb/pyldb.c:1359:19: note: in expansion of macro ‘PyStr_AsUTF8’
char *key_str = PyStr_AsUTF8(key);
^~~~~~~~~~~~
../lib/ldb/pyldb.c: In function ‘py_ldb_msg_getitem_helper’:
../lib/ldb/pyldb.c:3336:7: error: assignment discards ‘const’ qualifier from pointer target type [-Werror=discarded-qualifiers]
name = PyStr_AsUTF8(py_name);
^
../lib/ldb/pyldb.c: In function ‘py_ldb_msg_setitem’:
../lib/ldb/pyldb.c:3502:12: error: assignment discards ‘const’ qualifier from pointer target type [-Werror=discarded-qualifiers]
attr_name = PyStr_AsUTF8(name);
^
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Uri Simchoni <uri@samba.org>
Just use NULL in test case. talloc_autofree_context() is deprecated.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Oct 19 03:43:58 CEST 2018 on sn-devel-144
perf said all the time was in strlen.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Thu Oct 18 13:17:30 CEST 2018 on sn-devel-144
Additionally remove the associated known fail.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13616
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
object dn format should be a utf8 encoded string
Note: Currently this fails in python2 as the c python binding for
the dn string param uses PyArg_ParseTupleAndKeywords() with 's'
format, this will accept str *or* unicode in the default encoding.
The default encoding in python2 is... ascii.
Also adding here a knownfail to squash the error produced by the test.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Sep 6 15:50:17 CEST 2018 on sn-devel-144
Fixes an error detected by buildroot autobuilders:
http://autobuild.buildroot.net/results/573/573e2268e205e10d1352fa81122d8f225fdb4575/build-end.log
/home/rclinux/rc-buildroot-test/scripts/instance-1/output/host/mips64el-buildroot-linux-uclibc/sysroot/usr/include/stdint.h:122:27:
error: conflicting types for 'uintptr_t'
typedef unsigned long int uintptr_t;
^
In file included from ../lib/ldb/tests/ldb_msg.c:17:0:
../third_party/cmocka/cmocka.h:126:28: note: previous declaration of 'uintptr_t' was here
typedef unsigned int uintptr_t;
The define __WORDSIZE is missing when cmocka.h decides how to
define uintptr_t, this patch includes stdint.h when needed.
Patch sent upstream:
https://lists.samba.org/archive/samba-technical/2018-January/125306.html
Signed-off-by: Bernd Kuhls <bernd.kuhls@t-online.de>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Fri Aug 24 17:22:10 CEST 2018 on sn-devel-144
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
* New API ldb_dn_add_child_val() avoids passing untrusted input to
ldb_dn_add_child_fmt() (bug 13466)
* Free memory nearer to the allocation in calls made by ldbsearch
* Do not overwrite ldb_transaction_commit failure error messages
with a pointless del_transaction()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
These additional API tests just check that an invalid base DN
is never accepted.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This is safer for untrusted input than ldb_dn_add_child_fmt()
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13466
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
If the DN is not valid the ltdb_search_dn1() will catch it with ldb_dn_validate() which
is the only safe way to check this. ldb_dn_is_valid() does not actually check, but instead
returns only the result of the previous checks, if there was one.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
No matter commit succeeded or failed, transation will be delete afterwards.
So there is no need to delete it here.
Aganst Samba this causes an `LDAP error 51 LDAP_BUSY` error when the transaction
fails, say while we try to add users to groups in large amount and
the original error is lost.
In Samba, the rootdse module fails early in the del part of the
start/end/del pattern, and in ldb_tdb and ldb_mdb a failed commit
always ends the transaction, even on failure.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
We are allocating msg02, but check in assertion msg01, which makes no
sense here.
Signed-off-by: Timur I. Bakeyev <timur@freebsd.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
This ensures we fail with a good error code before an eventual ldb_dn_get_casefold() which
would otherwise fail.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
ldb_dn_from_ldb_val() does not validate this untrusted input, so a later
call to ldb_dn_get_casefold() can fail if the input is not valid.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
Convert the key value functions to use ldb_val instead of TDB_DATA.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Jul 30 17:23:22 CEST 2018 on sn-devel-144
Rename all the LTDB_* constants to LDB_KV_* as they are key value level
constants and not tdb specific.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Move the key value code to a separate subdirectory.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Rename ltdb_parse_data_unpack_ctx to ldb_kv_parse_data_unpack_ctx, as
it's a key value level structure and not ltdb specific.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Rename ltdb_idxptr to ldb_kv_idxptr as it's key value level and not tdb
specific.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Rename tdb_key_ctx to key_ctx, as it's key value level and not tdb
specific.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Rename ltdb_cache to ldb_kv_cache as it's key value level and not tdb
specific
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Tidy up the code format after the rename of ltdb_private to
ldb_kv_private
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Rename ltdb_private to ldb_kv_private as it contains key value operation
context.
Note there is still some tdb specific context that can be refactored into a
separate structure along the lines of the lmdb context.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Fix up the code formatting after the rename of ltdb_req_spy to
ldb_kv_req_spy
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Rename ltdb_req_spy to ldb_kv_req_spy, as it is key value level and not
tdb specific.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Rename ltdb_context to ldb_kv_context as it is a key value level
structure and not tdb specific.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Fix up the formatting after the rename of ltdb_reindex_context to
ldb_kv_reindex_context.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Rename struct ltdb_reindex_context to ldb_kv_reindex_context, as this is
a key value level structure and not a tdb specific structure.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Clean up the code format after the rename in the previous commit.
Hopefully doing a rename commit followed by a reformat commit makes the
code easier to review.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Rename the ldb key value functions from ltdb_* to ldb_kv_*. The renaming
is preparation for the separation of the tdb specific code from the key
value code. This work is a follow on from the addition of the lmdb
backend.
Note that the next commit tidies up the code formatting.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
ldb attributes are either bytes (py3) or str (py2)
Some places in the code do str(res[0]['attribute'][0])
which results in
'result' (py2)
b'result' (py3)
or more commonly the attribute is used to construct a string e.g.
"blah=" + res[0]['attribute'][0] + ",foo,bar=...."
giving
"blah=result,foo,bar=...." (py2)
and very unhelpfully
"blah=b'result',foo,bar=...." (py3)
lots of code already constructs various strings for passing to other
api using the above. To avoid many excessive
res[0]['attribute'][0].decode('utf8')
code like 'res[0]['attribute'][0]'
will now return LdbBytes (a new object subclassing 'bytes') in py3
instead of bytes. This object has a custom '__str__' method which
attempts to return a string decoded to uft8. In Py2 this will behave as
it did previously (this is the safer option at the moment)
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
* add some missing return value checks
* Fix several mem leaks in ldb_index ldb_search ldb_tdb (bug#13475)
* ldb_tdb: Use mem_ctx and so avoid leak onto long-term memory
on duplicated add. (bug#13471)
* ldb: Fix memory leak on module context (bug#13459)
* Refused build of Samba 4.8 with ldb 1.4 (bug #13519)
* Prevent similar issues in the future at configure time (bug #13519)
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Jul 12 07:43:22 CEST 2018 on sn-devel-144
Samba is not compatible with new versions of ldb (except release versions)
Other users would not notice the breakages, but Samba makes many
more assuptions about the LDB internals than any other package.
(Specifically, LDB 1.2 and 1.4 broke builds against released
Samba versions)
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13519
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
there are also mem leaks in this function
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13475
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Jun 15 23:07:25 CEST 2018 on sn-devel-144
After a duplicated add a small amount of memory can be leaked onto a
long-term context.
Found by Andrej Gessel https://github.com/andigese8fb45125e (commitcomment-29334102)
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13471
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Jun 15 04:39:41 CEST 2018 on sn-devel-144
Introduced in e8cdacc509
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13459
Signed-off-by: Lukas Slebodnik <lslebodn@fedoraproject.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Jun 1 11:10:24 CEST 2018 on sn-devel-144
* New LMDB backend (experimental)
* Comprehensive tests for index behaviour
* Enforce transactions for writes
* Enforce read lock use for all reads
* Fix memory leak in paged_results module.
We hold at most 10 outstanding paged result cookies
(bug #13362)
* Fix compiler warnings
* Python3 improvements
* Restore --disable-python build
* Fix for performance regression on one-level searches
(bug #13448)
* Samba's subtree_rename could fail to rename some entries
(bug #13452)
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13448
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
When no search filter is specified, the code falls back to using
'(|(objectClass=*)(distinguishedName=*)'. ltdb_index_dn() then failed
because matching against '*' is not indexed. The error return then
caused the code to fallback to a full-scan of the DB, which could have a
considerable performance hit.
Instead, we want to continue on and do the ltdb_index_filter() over the
indexed results that were returned.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13448
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Otherwise Samba modules like subtree_rename can fail as they modify the
index during the callback.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13452
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This ensures that the error string returned to the caller reflects a failure in this call.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Timur I. Bakeyev <timur@iXsystems.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
This ensures we do not corrupt such an index by making changes to the
main database without knowing that the index values are now in a
sub-database.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The way to know if we are in a transaction is if there is a non-NULL
transaction handle.
This allows the ldb_mdb_kv_ops_test test to be run.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
ldb_mdb is now able to pass the full ldb_mod_op_test when compiled against lmdb.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Tests to ensure that the mdb_env wrapping code correctly handles
multiple ldb's point to the same physical database file.
The test_ldb_close_with_multiple_connections tests are in
ldb_mod_op_test due to the utility code it uses from
elsewhere in that test.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This ensures we leave the FD behind if we exec() in a child process.
This deliberatly the same as TDB, as we want the same behaviour as
we have come to expect with that backend.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
MDB_env's may not be reused accross forks. Check the pid that the lmdb
structure was created by, and return an error if it is being used by a
different process.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Under some circumstances mdb_env_open returns EBADE, we treat this as
indicating the file is not a valid lmdb format file.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Wrap mdb_env_open to ensure that we only have one MDB_env opened per
database in each process
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
We need to enforce the GUID index mode so end-users do not get a
supprise in mid-operation and we enforce a max key length of 511 so
that the index key trunctation is done correctly.
Otherwise the DB will appear to work until a very long key (DN or
index) is used, after which it will be sad.
Because the previous ldb_lmdb_test confirmed the key length by
creating a large DN, those tests are re-worked to use the GUID index
mode. In turn, new tests are written that create a special DN around
the maximum key length.
Finally a test is included that demonstrates that adding entries to
the LMDB DB without GUID index mode fails.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
This is used in selftest with 'ldb:nosync = true'.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13362
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue May 15 09:37:21 CEST 2018 on sn-devel-144
Based on work for an mdb-specific test by Gary Lockyer
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed May 9 07:27:24 CEST 2018 on sn-devel-144
We need to show that despite the internal cache of TDB pointers that it
is safe to open a ldb_tdb after a fork()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Otherwise we rely on the caller doing tdb_reopen_all() which should
not be their job.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
(Split from a larger commit by Andrew Bartlett)
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
We may relax this restriction in the future, but for now do not assume
that the caller has done a tdb_reopen_all() at the right time.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The LMDB backend requires the GUID index mode, so prepare for it
by setting a unique objectGUID on each record. Also prepare for the
index list to be optionally set as an attribute on the test object,
allowing the GUID index mode to be set later when LMDB is configured.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu May 3 11:08:12 CEST 2018 on sn-devel-144
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
This will ensure we match LMDB behaviour and avoid a repeat of the per-record locking
issues (compared with full DB locking) we had before Samba 4.7.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
This avoids keeping a counter, which can be error-prone.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
The ldb_cache startup code would previously not take a read lock
nor a sufficiently wide write transaction.
The new code takes a read lock, and if it needs to write takes a
write lock (transaction) and re-reads before continuing.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
We need to enforce the GUID index mode so end-users do not get a supprise
in mid-operation and we enforce a max key length of 511 so that the
index key trunctation is done correctly.
Otherwise the DB will appear to work until a very long key (DN or index)
is used, after which it will be sad.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
With the changes to make samba python code Py2/Py3 compatible there
now are many instances where string content is decoded.
Decoded string variables in Py2 are returned as the unicode type. Many
Py2 c-module functions that take string arguments only check for the
string type. However now it's quite possibe the content formally passed
as a string argument is now passed as unicode after being decoded,
such arguments are rejected and code can fail subtly. This only affects
places where the type is directly checked e.g. via PyStr_Check etc.
arguments that are parsed by ParseTuple* functions generally already
accept both string and unicode (if 's', 'z', 's*' format specifiers
are used)
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Fri Apr 6 14:58:48 CEST 2018 on sn-devel-144
Add tests for the behaviour the ldb layer expects the key value layer to
provide. This should make it easier to add another KV store
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This avoids magic numbers and also is careful against overflow
from a long attr_for_dn.
This is done as a distinct commit to make the previous behaviour
change more clear, and to show that this does not change the
calculations, only improves the overflow check.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
These are tests are specifically for when the GUID index is not in use
which is always in with ldb_mdb.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13335
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Apr 5 07:53:10 CEST 2018 on sn-devel-144
The re-index traverse can abort part-way though and we need to ensure
that the transaction is never committed as that will leave an un-useable db.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13335
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Mar 8 14:14:37 CET 2018 on sn-devel-144
Without the destructor firing, this test used to pass, but now we show
that we must be able to open a new ldb handle.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Mar 7 04:38:02 CET 2018 on sn-devel-144
This allows sharing of the originally ldb_tdb operations to the new
ldb_mdb backend.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Mar 6 01:39:16 CET 2018 on sn-devel-144
This can handle both read-only and writable traverses.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This will allow us to change the backend from tdb to lmdb.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
With no schema syntax, this would occasionally crash as it dereferenced
some possibly NULL sequence of memory.
Note: Removing all tests except this one, made it crash reliably.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Tests for the index truncation code as well as the GUID index
format in general.
Covers truncation of both the DN and equality search keys.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Mar 3 09:58:40 CET 2018 on sn-devel-144
Fold together two identical cases to simplify the code.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Rather than add many special cases, over-long unique values are simply banned.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
In the truncation case a duplicate is perfectly expected.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Modify the indexing code to handle a maximum key length, index keys
greater than the maximum length will be truncated to the maximum length.
And the unuque index code has been altered to handle multiple records
for the same index key.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
It is not the job of the index code to enforce this, but do give a
a warning given it has been detected.
However, now that we do allow it, we must never return the same
object twice to the caller, so filter for it in ltdb_index_filter().
The GUID list is sorted, which makes this cheap to handle, thankfully.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Allow the setting of the maximum key length, this allows the testing of
index key truncation code. Index key truncation is required to allow
the samba indexing scheme to be used with backends that enforce a
maximum key length.
This will allow emulation of a length-limited key DB for testing.
This is a testing-only feature, as the index format changes
based on this value.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* Expose the SHOW_BINARY, ENABLE_TRACING and DONT_CREATE_DB flag constants
in the python api.
* Extend dn.is_child_of() test.
* Don't load LDB_MODULESDIR as a module file.
* Fix binary data in debug log (bug #13185).
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Wed Feb 28 04:54:21 CET 2018 on sn-devel-144
Tests to ensure:
When duplicate objects are added, the GUID was printed in the debug log
are passed through the escape function.
And that duplicate DN's do not generate debug log entries.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13185
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Feb 26 07:29:49 CET 2018 on sn-devel-144
When duplicate objects were added, the GUID was printed in the debug log
The GUID was not escaped and therefore displayed as binary content.
This patch splits out the duplicate DN creation error and the duplicate
GIUD error. Duplicate DN's are a normal event and don't require debug
logging.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13185
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Fix up tests that were performing a null check on the wrong variable
after a call to ldb_msg_new
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Feb 24 15:50:35 CET 2018 on sn-devel-144
We are setting modules directory here(LDB_MODULESDIR) so treat it this
way, no need to attempt to load it as a module file.
Signed-off-by: Timur I. Bakeyev <timur@iXsystems.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Uri Simchoni <uri@samba.org>
Add this test so ensure that this (unclear) behaviour does
not change silently.
Signed-off-by: Björn Baumbach <bb@sernet.de>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Expose the SHOW_BINARY, ENABLE_TRACING and DONT_CREATE_DB flag constants
in the python api.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* Intersect the index from SCOPE_ONELEVEL with the index for the search expression
(bug #13191)
* smaller/greater comparison tests
* Show the last successful DN when failing to parse LDIF
* ldb_index: Add an attriubute flag to require a unique value.
* silence some clang warnings in picky developer mode
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
This helps ensure we do not have to scan all objects at this level
which could be very many (one per DNS zone entry).
However, due to the O(n*m) behaviour in list_intersect() for older
databases, we only do this in the GUID index mode, leaving the behaviour
unchanged for existing callers that do not specify the GUID index mode.
NOTE WELL: the behaviour of disallowDNFilter is enforced
in the index code, so this fixes SCOPE_ONELEVEL to also
honour disallowDNFilter, hence the additional tests.
The change to select the SUBTREE index in the absense of
the ONELEVEL index enforces this.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13191
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
The comparison result has been ignored, which is not good. Also remove
the "ldbsearch" command in the error branch which has not much sense.
The scripts needs to be run through test-tdb.sh, test-ldap.sh or
test-sqlite3.sh which I didn't realise before. Hence less changes are needed
and this is a reduced version of the patch published on the mailing list.
Signed-off-by: Matthias Dieter Wallnöfer <mdw@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Matthias Dieter Wallnöfer <mdw@samba.org>
Autobuild-Date(master): Tue Dec 19 03:09:12 CET 2017 on sn-devel-144
Add attribute flag LDB_FLAG_INTERNAL_UNIQUE_VALUE, to request that the
added attribute is unique on the index.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13004
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
