IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
We had to devise our own scheme for writing integers in a human readable
format which also sorted correctly numerically. This might look a bit
confusing to outsiders, so here's a large comment as a peace offering.
Pair-programmed-with: Tim Beale <timbeale@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Activating <= and >= mdb indexing in samba for int32 and int64 attributes by:
1. Adding index_format_fn to LDB_SYNTAX_SAMBA_INT32 in ldb_samba
2. Cloning the 64bit LDB_SYNTAX_INTEGER type as LDB_SYNTAX_ORDERED_INTEGER
3. Adding index_format_fn to the new type
4. Modifying LargeInteger use the new type in samba schema
5. Bumping the index version to trigger reindexing
Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The reason we needed it in the first place was that the original
canonicalize is being used for non-index functions and it never produced
the right order originally (at least for integers).
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
It makes no difference in our standard case because \0 will always go
before any value for our index_format_fn, but this is better for
correctness (in case we do mess up our NUL terminations elsewhere).
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Full implementation of <= and >= indexed searching using iterate_range
backend operation. Adds index_format_fn to ldb_schema_syntax so
requires an ABI version bump. The function must be provided for any
type for which <= and >= indexing is required, and must return a
lexicographically ordered canonicalization of a value. This causes
index entries to be written in correct order to the database, so
iterate_range on the index DNs can be used.
ldb_kv_index_key is modified to return an index DN with attribute name
but without value if an empty value is provided. This is needed for
constructing keys that match the beginning or end of an index DN range.
Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Adding iterate_range to LDB API and implementing in LMDB. This
operation takes a start_key and end_key and returns all records between
the two, inclusive of both. This will be used to implementing indexing
for <= and >= expressions.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
If ldb_kv_filter_attrs() fails, we don't know that the dn of filtered_msg
is OK.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Apr 5 05:46:55 UTC 2019 on sn-devel-144
Set the default index cache from the passed option
"transaction_index_cache_size" on open. This allows the default cache
size to be overridden when processing large transactions i.e. joining a
large domain.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Set the index cache size to the number of records in the databse when
reindexing.
This significantly improves reindex performance. For a domain with
100,000 users the reindex times are reduced from 17 minutes to 45
seconds.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add the get_size method to the ldb_key_value layer, this will allow the
reindexing code to get an estimate of the number of records in the
database.
The lmdb backend returns an accurate count of the number of records in
the database withe the mdb_env_stat call.
The tdb backend does not provide a low cost method to determine the
number of records on the database. It does provide a tdb_summary call
however this this walks the entire database.
So for tdb we use the map size divided by 500, this over estimates the counts
for small domains, but the extra memory allocated for the cache should
not be significant.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Pass the index cache size to ldb_kv_index_transaction_start. This will
allow it to be set for reindex and join operations, where the current
defaults result in a significant performance penalty on large databases.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Remove the lazy initialisation of the index cache. This make setting
the size of the cache for re-indexing easier, which will be done in
later commits.
Performance testing shows that the removal of lazy initialisation makes
no appreciable difference to performance.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* Remove Python 2.x support except to build just the bare C library
* Remove --extra-python (a build time mode to produce Python2 and Python3
bindings at the same time)
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Mar 21 05:08:49 UTC 2019 on sn-devel-144
We do this by removing the confusing mandatory option to
conf.SAMBA_CHECK_PYTHON{,_HEADERS}(), instead just use the value of
--disable-python internally
This follows the default minimum of Python 3.4 and keeps things consistent
with the main Samba build where --disable-python is required to skip building
python bindings.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
These are no longer used by the build system so avoid
confusion by removing them from the tree.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This option is quite invasive in waf and was mainly for the python3 transition.
Testing with multiple python versions can be done by testing a full compile against
multiple versions, likewise multiple different binding versions can be created
the same way.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
* Fix standalone build of ldb.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Feb 26 12:10:40 CET 2019 on sn-devel-144
When calling make from the ldb, talloc, tdb, and
tevent bundles, we need to first find the
location of the waf script. Currently the build
fails since it can't find waf.
Fixes regression caused by a660b7f.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* CVE-2019-3824 out of bounds read in wildcard compare (bug 13773)
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Gary Lockyer <gary@samba.org>
Autobuild-Date(master): Mon Feb 25 22:54:13 CET 2019 on sn-devel-144
Add cmocka tests for ldb_wildcard_match.
Running test_wildcard_match under valgrind reproduces
CVE-2019-3824 out of bounds read in wildcard compare (bug 13773)
valgrind --suppressions=lib/ldb/tests/ldb_match_test.valgrind\
bin/ldb_match_test
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
ldb_handler_copy and ldb_val_dup over allocate by one and add a trailing '\0'
to the data, to make them safe to use the C string functions on.
However testing for the trailing '\0' is not the correct way to test for
the end of a value, the length should be checked instead.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Check the operation type of the passed parse tree, and return
LDB_INAPPROPRIATE_MATCH if the operation is not LDB_OP_SUBSTRING.
A query of "attribute=*" gets parsed as LDB_OP_PRESENT, checking the
operation and failing ldb_wildcard_match should help prevent confusion
writing tests.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Initialise the created ldb_parse_tree with talloc_zero, this ensures
that it is correctly initialised if inadvertently passed to a function
expecting a different operation type.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
There is valgrind error in few tests tests/test-generic.sh
91 echo "Test wildcard match"
92 $VALGRIND ldbadd $LDBDIR/tests/test-wildcard.ldif || exit 1
93 $VALGRIND ldbsearch '(cn=test*multi)' || exit 1
95 $VALGRIND ldbsearch '(cn=*test_multi)' || exit 1
97 $VALGRIND ldbsearch '(cn=test*multi*test*multi)' || exit 1
e.g.
==3098== Memcheck, a memory error detector
==3098== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3098== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==3098== Command: ./bin/ldbsearch (cn=test*multi)
==3098==
==3098== Invalid read of size 1
==3098== at 0x483CEE7: memchr (vg_replace_strmem.c:890)
==3098== by 0x49A9073: memmem (in /usr/lib64/libc-2.28.9000.so)
==3098== by 0x485DFE9: ldb_wildcard_compare (ldb_match.c:313)
==3098== by 0x485DFE9: ldb_match_substring (ldb_match.c:360)
==3098== by 0x485DFE9: ldb_match_message (ldb_match.c:572)
==3098== by 0x558F8FA: search_func (ldb_kv_search.c:549)
==3098== by 0x48C78CA: ??? (in /usr/lib64/libtdb.so.1.3.17)
==3098== by 0x48C7A60: tdb_traverse_read (in /usr/lib64/libtdb.so.1.3.17)
==3098== by 0x557B7C4: ltdb_traverse_fn (ldb_tdb.c:274)
==3098== by 0x558FBFA: ldb_kv_search_full (ldb_kv_search.c:594)
==3098== by 0x558FBFA: ldb_kv_search (ldb_kv_search.c:854)
==3098== by 0x558E497: ldb_kv_callback (ldb_kv.c:1713)
==3098== by 0x48FCD58: tevent_common_invoke_timer_handler (in /usr/lib64/libtevent.so.0.9.38)
==3098== by 0x48FCEFD: tevent_common_loop_timer_delay (in /usr/lib64/libtevent.so.0.9.38)
==3098== by 0x48FE14A: ??? (in /usr/lib64/libtevent.so.0.9.38)
==3098== Address 0x4b4ab81 is 0 bytes after a block of size 129 alloc'd
==3098== at 0x483880B: malloc (vg_replace_malloc.c:309)
==3098== by 0x491048B: talloc_strndup (in /usr/lib64/libtalloc.so.2.1.15)
==3098== by 0x48593CA: ldb_casefold_default (ldb_utf8.c:59)
==3098== by 0x485F68D: ldb_handler_fold (attrib_handlers.c:64)
==3098== by 0x485DB88: ldb_wildcard_compare (ldb_match.c:257)
==3098== by 0x485DB88: ldb_match_substring (ldb_match.c:360)
==3098== by 0x485DB88: ldb_match_message (ldb_match.c:572)
==3098== by 0x558F8FA: search_func (ldb_kv_search.c:549)
==3098== by 0x48C78CA: ??? (in /usr/lib64/libtdb.so.1.3.17)
==3098== by 0x48C7A60: tdb_traverse_read (in /usr/lib64/libtdb.so.1.3.17)
==3098== by 0x557B7C4: ltdb_traverse_fn (ldb_tdb.c:274)
==3098== by 0x558FBFA: ldb_kv_search_full (ldb_kv_search.c:594)
==3098== by 0x558FBFA: ldb_kv_search (ldb_kv_search.c:854)
==3098== by 0x558E497: ldb_kv_callback (ldb_kv.c:1713)
==3098== by 0x48FCD58: tevent_common_invoke_timer_handler (in /usr/lib64/libtevent.so.0.9.38)
==3098==
# record 1
dn: cn=test_multi_test_multi_test_multi,o=University of Michigan,c=TEST
cn: test_multi_test_multi_test_multi
description: test multi wildcards matching
objectclass: person
sn: multi_test
name: test_multi_test_multi_test_multi
distinguishedName: cn=test_multi_test_multi_test_multi,o=University of Michiga
n,c=TEST
# returned 1 records
# 1 entries
# 0 referrals
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773
Signed-off-by: Lukas Slebodnik <lslebodn@fedoraproject.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett abartlet@samba.org
Autobuild-User(master): Noel Power <npower@samba.org>
Autobuild-Date(master): Wed Feb 13 14:51:12 CET 2019 on sn-devel-144
Every time I look at this file, I spend a few minutes wondering how
these bits of code are ever run. Never again.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
PyList_Append doesn't steal references, so if the item created is
a temp object, created just to be added to the list we need to
decref the item appended in order for it to be released.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Py_BuildValue when processing format 'O' will
'Pass a Python object untouched (except for its reference count,
which is incremented by one'
Basically this means if you are using a new reference to a PyObject
to pass to BuildValue (to be used with the 'O' format) the reference
*isn't* stolen so you really do need to DECREF it in order to ensure
it gets cleaned up.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
* pyldb: make ldb.connect() url mandatory
* New version number for master (Samba 4.11 eventually)
The 1.5.x series will be maintained in the v4-10-test branch
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Feb 1 07:02:56 CET 2019 on sn-devel-144
The call fails without it, so we might as well fail sooner
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The old name confused me because it's not really related to the
one-level index at all. It's the result from evaluating the indexed
search specified in the ac->tree.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13762
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Disclaimer: this is based on my limited understanding of what the code
is doing.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13762
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This comment was written before the GUID_index_attribute block of code
existed. So we now *do* load the index values and *do* check for a
strict intersect, so the comment is redundant.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13762
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Commit 88ae60ed186c9 introduced a problem that made one-level
searches inefficient if there were a lot of child objects in the same
level, and the requested object didn't exist. Basically, it ignored the
case where ldb_kv_index_dn() returned LDB_ERR_NO_SUCH_OBJECT, i.e. the
indexed lookup was successful, but didn't find a match. At which point,
there was no more processing we needed to do.
The behaviour after 88ae60ed186c9 was to fall-through and run the
ldb_kv_index_filter() function over *all* the children. This still
returned the correct result, but could be costly if there were a lot of
children.
The case 88ae60ed186c9 was trying to fix was where we could not do
an indexed search (e.g. trying to match on a 'attribute=*' filter). In
which case we want to ignore the LDB_ERR_OPERATIONS_ERROR and just run
ldb_kv_index_filter() over all the children. This is still more
efficient than the fallback of doing a full database scan.
This patch adds in a short-circuit for the NO_SUCH_OBJECT case, so we
can skip the unnecessary ldb_kv_index_filter() work.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13762
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Test failed on s390x but there is a simple reproducer for any
architecture.
The built-in function repr returns the canonical string representation
of the object. We needn't care about order attributes in string
representation. Therefore test should pass for any order.
for i in {1..30}; do
PYTHONHASHSEED=random \
python2 -c 'import ldb; msg = ldb.Message(); msg.dn = ldb.Dn(ldb.Ldb(), "dc=foo29"); msg["dc"] = b"foo"; print(repr(msg)) '
done
======================================================================
FAIL: test_repr (__main__.LdbMsgTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests/python/api.py", line 2322, in test_repr
"Message({'dn': Dn('dc=foo29'), 'dc': MessageElement(['foo'])})")
AssertionError: "Message({'dc': MessageElement(['foo']), 'dn': Dn('dc=foo29')})" != "Message({'dn': Dn('dc=foo29'), 'dc': MessageElement(['foo'])})"
----------------------------------------------------------------------
Ran 1025 tests in 29.146s
FAILED (failures=1)
Signed-off-by: Lukas Slebodnik <lslebodn@fedoraproject.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
