1
0
mirror of https://github.com/samba-team/samba.git synced 2025-01-12 09:18:10 +03:00
Commit Graph

980 Commits

Author SHA1 Message Date
Andrew Bartlett
9d46795b20 ldb: extend API tests
These additional API tests just check that an invalid base DN
is never accepted.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:24 +02:00
Andrew Bartlett
2dafbd3213 ldb: Add new function ldb_dn_add_child_val()
This is safer for untrusted input than ldb_dn_add_child_fmt()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13466

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:24 +02:00
Andrew Bartlett
542e7c1791 ldb_tdb: Remove pointless check of ldb_dn_is_valid()
If the DN is not valid the ltdb_search_dn1() will catch it with ldb_dn_validate() which
is the only safe way to check this.  ldb_dn_is_valid() does not actually check, but instead
returns only the result of the previous checks, if there was one.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:24 +02:00
Andrej Gessel
d71c655eec fix mem leak in ldbsearch
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:24 +02:00
Andrej Gessel
d645546aef fix mem leak in ltdb_index_dn_base_dn and ltdb_search_indexed
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:24 +02:00
Joe Guo
b9e2a2de24 ldb: no need to call del_transaction in ldb_transaction_commit
No matter commit succeeded or failed, transation will be delete afterwards.
So there is no need to delete it here.

Aganst Samba this causes an `LDAP error 51 LDAP_BUSY` error when the transaction
fails, say while we try to add users to groups in large amount and
the original error is lost.

In Samba, the rootdse module fails early in the del part of the
start/end/del pattern, and in ldb_tdb and ldb_mdb a failed commit
always ends the transaction, even on failure.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:23 +02:00
Timur I. Bakeyev
e4f38b0670 ldb tests: fix assertion on wrong pointer
We are allocating msg02, but check in assertion msg01, which makes no
sense here.

Signed-off-by: Timur I. Bakeyev <timur@freebsd.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
2018-08-14 19:08:25 +02:00
Andrew Bartlett
b7f0ee93f5 Release LDB 1.5.0 for CVE-2018-1140
* Security fix for CVE-2018-1140 (NULL pointer de-reference, bug 13374)
* Fix memory leaks and missing error checks (bug 13459, 13471, 13475)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-14 13:57:15 +02:00
Andrew Bartlett
b6b72d0063 CVE-2018-1140 ldb: Add tests for search add and rename with a bad dn= DN
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
2018-08-14 13:57:15 +02:00
Andrew Bartlett
3c1fbb1832 CVE-2018-1140 ldb_tdb: Check for DN validity in add, rename and search
This ensures we fail with a good error code before an eventual ldb_dn_get_casefold() which
would otherwise fail.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
2018-08-14 13:57:15 +02:00
Andrew Bartlett
b27d973341 CVE-2018-1140 ldb_tdb: Ensure the dn in distinguishedName= is valid before use
ldb_dn_from_ldb_val() does not validate this untrusted input, so a later
call to ldb_dn_get_casefold() can fail if the input is not valid.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
2018-08-14 13:57:15 +02:00
Andrew Bartlett
3f95957d6d CVE-2018-1140 ldb: Check for ldb_dn_get_casefold() failure in ldb_sqlite
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
2018-08-14 13:57:15 +02:00
Andrej Gessel
0998f2f1bc CVE-2018-1140 Add NULL check for ldb_dn_get_casefold() in ltdb_index_dn_attr()
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
2018-08-14 13:57:15 +02:00
Gary Lockyer
c891df4218 lib ldb key value: convert TDB_DATA structs to ldb_val
Convert the key value functions to use ldb_val instead of TDB_DATA.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Jul 30 17:23:22 CEST 2018 on sn-devel-144
2018-07-30 17:23:22 +02:00
Gary Lockyer
f2d5c2c5cc lib ldb: rename LTDB_* constants to LDB_KV_*
Rename all the LTDB_* constants to LDB_KV_* as they are key value level
constants and not tdb specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:54 +02:00
Gary Lockyer
19be0be232 lib ldb: move key value code to lib/ldb/ldb_key_value
Move the key value code to a separate subdirectory.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:54 +02:00
Gary Lockyer
67c055406c lib ldb: rename ltdb_parse_data_unpack_ctx
Rename ltdb_parse_data_unpack_ctx to ldb_kv_parse_data_unpack_ctx, as
it's a key value level structure and not ltdb specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:54 +02:00
Gary Lockyer
9191d3baf5 lib ldb: remove unused function prototypes
Remove unused function prototypes

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:54 +02:00
Gary Lockyer
d3bfd374a6 lib ldb: rename ltdb_idxptr to ldb_kv_idxptr
Rename ltdb_idxptr to ldb_kv_idxptr as it's key value level and not tdb
specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:54 +02:00
Gary Lockyer
f6d5cf5eb1 lib ldb: rename tdb_key_ctx to key_ctx
Rename tdb_key_ctx to key_ctx, as it's key value level and not tdb
specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
72724f75ce lib ldb: rename ltdb_cache to ldb_kv_cache
Rename ltdb_cache to ldb_kv_cache as it's key value level and not tdb
specific

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
b8c9c305b3 lib ldb: format rename ldb_kv_private
Tidy up the code format after the rename of ltdb_private to
ldb_kv_private

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
e0186d1f2d lib ldb: rename ltdb_private to ldb_kv_private
Rename ltdb_private to ldb_kv_private as it contains key value operation
context.

Note there is still some tdb specific context that can be refactored into a
separate structure along the lines of the lmdb context.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
641b38e458 lib ldb: format rename of ltdb_req_spy
Fix up the code formatting after the rename of ltdb_req_spy to
ldb_kv_req_spy

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
17a84d12e2 lib ldb: rename ltdb_req_spy to ldb_kv_req_spy
Rename ltdb_req_spy to ldb_kv_req_spy, as it is key value level and not
tdb specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
c782d710e9 lib ldb: rename ltdb_context to ldb_kv_context
Rename ltdb_context to ldb_kv_context as it is a key value level
structure and not tdb specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
e969de0797 lib ldb: reformat ltdb_reindex_context rename
Fix up the formatting after the rename of ltdb_reindex_context to
ldb_kv_reindex_context.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
e8aa764ea7 lib ldb: rename struct ltdb_reindex_context
Rename struct ltdb_reindex_context to ldb_kv_reindex_context, as this is
a key value level structure and not a tdb specific structure.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:52 +02:00
Gary Lockyer
9e6294668a lib ldb: fix formatting of ldb_kv rename.
Clean up the code format after the rename in the previous commit.
Hopefully doing a rename commit followed by a reformat commit makes the
code easier to review.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:52 +02:00
Gary Lockyer
1c8ea099ce lib ldb: Rename functions to ldb_kv
Rename the ldb key value functions from ltdb_* to ldb_kv_*. The renaming
is preparation for the separation of the tdb specific code from the key
value code.  This work is a follow on from the addition of the lmdb
backend.

Note that the next commit tidies up the code formatting.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:52 +02:00
Stefan Metzmacher
591d72f9c7 ldb_mdb: #ifdef EBADE as it is not portable
E.g. FreeBSD 11.2 doesn't have it.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-27 13:07:14 +02:00
Noel Power
7b170206b4 lib/ldb: Implement a bytes derived object for attributes py2/py3
ldb attributes are either bytes (py3) or str (py2)

Some places in the code do str(res[0]['attribute'][0])
which results in
   'result' (py2)
  b'result' (py3)

or more commonly the attribute is used to construct a string e.g.
   "blah=" + res[0]['attribute'][0] + ",foo,bar=...."

giving
   "blah=result,foo,bar=...." (py2)
and very unhelpfully
   "blah=b'result',foo,bar=...." (py3)

lots of code already constructs various strings for passing to other
api using the above. To avoid many excessive
    res[0]['attribute'][0].decode('utf8')

code like 'res[0]['attribute'][0]'

will now return LdbBytes (a new object subclassing 'bytes') in py3
instead of bytes. This object has a custom '__str__' method which
attempts to return a string decoded to uft8. In Py2 this will behave as
it did previously (this is the safer option at the moment)

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-07-13 01:12:24 +02:00
Stefan Metzmacher
3eecdbcc38 ldb: version 1.4.1
* add some missing return value checks
* Fix several mem leaks in ldb_index ldb_search ldb_tdb (bug#13475)
* ldb_tdb: Use mem_ctx and so avoid leak onto long-term memory
  on duplicated add. (bug#13471)
* ldb: Fix memory leak on module context (bug#13459)
* Refused build of Samba 4.8 with ldb 1.4 (bug #13519)
* Prevent similar issues in the future at configure time (bug #13519)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Jul 12 07:43:22 CEST 2018 on sn-devel-144
2018-07-12 07:43:22 +02:00
Andrew Bartlett
52efa79653 ldb: Refuse to build Samba against a newer minor version of ldb
Samba is not compatible with new versions of ldb (except release versions)

Other users would not notice the breakages, but Samba makes many
more assuptions about the LDB internals than any other package.

(Specifically, LDB 1.2 and 1.4 broke builds against released
Samba versions)

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13519

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-07-12 04:32:08 +02:00
Andrew Bartlett
1a559fd6a9 ldb: Ban ldb 1.4.x with Samba 4.8 and earlier
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13519

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-07-12 04:32:08 +02:00
Andrej Gessel
f75e8f58cd check return value before using key_values
there are also mem leaks in this function

Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2018-06-23 02:00:28 +02:00
Andrej Gessel
6b52d21e60 ldb: check return values
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2018-06-23 02:00:28 +02:00
Andrej Gessel
3ca1c09f68 Fix several mem leaks in ldb_index ldb_search ldb_tdb
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13475

Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Jun 15 23:07:25 CEST 2018 on sn-devel-144
2018-06-15 23:07:25 +02:00
Andrew Bartlett
f4f3abfa0e ldb_tdb: Use mem_ctx and so avoid leak onto long-term memory on duplicated add.
After a duplicated add a small amount of memory can be leaked onto a
long-term context.

Found by Andrej Gessel https://github.com/andiges

e8fb45125e (commitcomment-29334102)

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13471
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Jun 15 04:39:41 CEST 2018 on sn-devel-144
2018-06-15 04:39:41 +02:00
Lukas Slebodnik
d161aa3522 ldb: Fix memory leak on module context
Introduced in e8cdacc509

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13459

Signed-off-by: Lukas Slebodnik <lslebodn@fedoraproject.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Jun  1 11:10:24 CEST 2018 on sn-devel-144
2018-06-01 11:10:24 +02:00
Andrew Bartlett
4e2eb5660a ldb: Release ldb 1.4.0
* New LMDB backend (experimental)
* Comprehensive tests for index behaviour
* Enforce transactions for writes
* Enforce read lock use for all reads
* Fix memory leak in paged_results module.
  We hold at most 10 outstanding paged result cookies
  (bug #13362)
* Fix compiler warnings
* Python3 improvements
* Restore --disable-python build
* Fix for performance regression on one-level searches
  (bug #13448)
* Samba's subtree_rename could fail to rename some entries
  (bug #13452)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-30 04:23:28 +02:00
Andrew Bartlett
e99c199d81 ldb: Add tests for when we should expect a full scan
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13448

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-30 04:23:28 +02:00
Andrew Bartlett
88ae60ed18 ldb: One-level search was incorrectly falling back to full DB scan
When no search filter is specified, the code falls back to using
'(|(objectClass=*)(distinguishedName=*)'. ltdb_index_dn() then failed
because matching against '*' is not indexed. The error return then
caused the code to fallback to a full-scan of the DB, which could have a
considerable performance hit.

Instead, we want to continue on and do the ltdb_index_filter() over the
indexed results that were returned.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13448

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-30 04:23:27 +02:00
Andrew Bartlett
9e143ee9b9 ldb: Explain why an entry can vanish from the index
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-30 04:23:27 +02:00
Andrew Bartlett
3632775d7a ldb: Indicate that the ltdb_dn_list_sort() in list_union is a bit subtle.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-30 04:23:27 +02:00
Andrew Bartlett
d02cd236dc ldb: Save a copy of the index result before calling the callbacks.
Otherwise Samba modules like subtree_rename can fail as they modify the
index during the callback.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13452

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-30 04:23:27 +02:00
Andrew Bartlett
04e3c4bea2 ldb: Reset error string before running prepare_commit() hook
This ensures that the error string returned to the caller reflects a failure in this call.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-30 04:23:26 +02:00
Timur I. Bakeyev
95c117ff11 Make ldb configuration --disable-python work as intended
Signed-off-by: Timur I. Bakeyev <timur@iXsystems.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2018-05-25 10:01:22 +02:00
Andrew Bartlett
cb5da7a5c6 ldb: Reject a possible future ldb_mdb with the index in a sub-database
This ensures we do not corrupt such an index by making changes to the
main database without knowing that the index values are now in a
sub-database.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:12 +02:00
Gary Lockyer
866af3270d ldb: Add MDB support to ldb://
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2018-05-23 02:27:12 +02:00
Gary Lockyer
aeeab1753e ldb_mdb/tests: add tests for multiple opens across forks
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:12 +02:00
Gary Lockyer
04884a8012 ldb_mdb/tests: test large index key value
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:12 +02:00
Gary Lockyer
65f6ce7a6e ldb_mdb: Remove implicit read lock and remove transaction counter
The way to know if we are in a transaction is if there is a non-NULL
transaction handle.

This allows the ldb_mdb_kv_ops_test test to be run.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:11 +02:00
Andrew Bartlett
d8919d2a59 ldb_mdb: Run the ldb_mdb_mod_op_test
ldb_mdb is now able to pass the full ldb_mod_op_test when compiled against lmdb.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:11 +02:00
Gary Lockyer
be335f1fbc ldb_mdb/tests: Tests for wrap open
Tests to ensure that the mdb_env wrapping code correctly handles
multiple ldb's point to the same physical database file.

The test_ldb_close_with_multiple_connections tests are in
ldb_mod_op_test due to the utility code it uses from
elsewhere in that test.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:11 +02:00
Andrew Bartlett
4dc44659dd ldb_mdb: Use mdb_env_get_fd() to get the FD for fstat() and FD_CLOEXEC
This ensures we leave the FD behind if we exec() in a child process.

This deliberatly the same as TDB, as we want the same behaviour as
we have come to expect with that backend.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:11 +02:00
Gary Lockyer
322e42818b ldb_mdb: prevent MDB_env reuse across forks
MDB_env's may not be reused accross forks.  Check the pid that the lmdb
structure was created by, and return an error if it is being used by a
different process.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:11 +02:00
Gary Lockyer
14f5c7522c ldb_mdb: handle EBADE from mdb_env_open
Under some circumstances mdb_env_open returns EBADE, we treat this as
indicating the file is not a valid lmdb format file.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:11 +02:00
Gary Lockyer
eb1bc2ec09 ldb_mdb: Wrap mdb_env_open
Wrap mdb_env_open to ensure that we only have one MDB_env opened per
database in each process

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:11 +02:00
Gary Lockyer
f9a12b6433 ldb_mdb: Apply LMDB key length restrictions at key-value layer
We need to enforce the GUID index mode so end-users do not get a
supprise in mid-operation and we enforce a max key length of 511 so
that the index key trunctation is done correctly.

Otherwise the DB will appear to work until a very long key (DN or
index) is used, after which it will be sad.

Because the previous ldb_lmdb_test confirmed the key length by
creating a large DN, those tests are re-worked to use the GUID index
mode.  In turn, new tests are written that create a special DN around
the maximum key length.

Finally a test is included that demonstrates that adding entries to
the LMDB DB without GUID index mode fails.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:11 +02:00
Gary Lockyer
53d9d4974d ldb_mdb/tests: Run api and index test also on lmdb
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:11 +02:00
Gary Lockyer
a5a000b68e ldb_mdb/tests: Add tests to check for max key length and DB size
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:11 +02:00
Gary Lockyer
0d2d1e5bf0 ldb_mdb: Don't allow modify operations on a read only db
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-23 02:27:11 +02:00
Garming Sam
95d1e474cf ldb_mdb: Store pid to change destructor on fork
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-05-23 02:27:11 +02:00
Garming Sam
e4e6d794ee ldb_mdb: Enable LDB_FLG_NOSYNC in ldb_mdb
This is used in selftest with 'ldb:nosync = true'.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-05-23 02:27:11 +02:00
Garming Sam
5ec491040c ldb_mdb: Implement the lmdb backend for ldb
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-05-23 02:27:10 +02:00
Volker Lendecke
9fbd4672b0 lib: Hold at most 10 outstanding paged result cookies
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13362
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue May 15 09:37:21 CEST 2018 on sn-devel-144
2018-05-15 09:37:21 +02:00
Volker Lendecke
8063995a92 lib: Put "results_store" into a doubly linked list
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13362
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2018-05-15 06:32:40 +02:00
Noel Power
521bc6056e Bulk: enclose .keys() method with list where list (from python2) expected
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-05-12 21:38:17 +02:00
Andrew Bartlett
ba33d90ed6 ldb: Ensure we can open a new LDB after a fork()
Based on work for an mdb-specific test by Gary Lockyer

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed May  9 07:27:24 CEST 2018 on sn-devel-144
2018-05-09 07:27:24 +02:00
Andrew Bartlett
f891b8dc32 ldb: Add tests for ldb_tdb use after a fork()
We need to show that despite the internal cache of TDB pointers that it
is safe to open a ldb_tdb after a fork()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-09 04:29:48 +02:00
Andrew Bartlett
2136664941 ldb_tdb: Allow use of a TDB for ldb_tdb after as fork()
Otherwise we rely on the caller doing tdb_reopen_all() which should
not be their job.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-09 04:29:48 +02:00
Andrew Bartlett
3b06915663 ldb: Reset errno before checking it in ltdb_connect()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-09 04:29:48 +02:00
Gary Lockyer
daf79e5b35 ldb/tests: add tests for transaction_{start,commit}/lock_read across forks
(Split from a larger commit by Andrew Bartlett)

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-09 04:29:48 +02:00
Andrew Bartlett
1174b52b91 ldb_tdb: Prevent ldb_tdb reuse after a fork()
We may relax this restriction in the future, but for now do not assume
that the caller has done a tdb_reopen_all() at the right time.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2018-05-09 04:29:48 +02:00
Gary Lockyer
8a0c7f39d6 ldb: Prepare to allow tests to operate on ldb_mdb (by using the GUID index)
The LMDB backend requires the GUID index mode, so prepare for it
by setting a unique objectGUID on each record.  Also prepare for the
index list to be optionally set as an attribute on the test object,
allowing the GUID index mode to be set later when LMDB is configured.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu May  3 11:08:12 CEST 2018 on sn-devel-144
2018-05-03 11:08:12 +02:00
Gary Lockyer
1bff2ae128 ldb: Introduce new generic ldb:// prefix to allow backend autodetection
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2018-05-03 08:17:45 +02:00
Gary Lockyer
c8b45a3509 ldb tests: add cmocka tests of kv operation interactions with transactions
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-05-03 08:17:45 +02:00
Gary Lockyer
9ca34b9c4a ldb tests: api ensure database correctly populated
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-05-03 08:17:45 +02:00
Andrew Bartlett
25e1cfad3a ldb_tdb: Disallow reads without a transaction or read lock
This will ensure we match LMDB behaviour and avoid a repeat of the per-record locking
issues (compared with full DB locking) we had before Samba 4.7.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-05-03 08:17:44 +02:00
Gary Lockyer
f04bbd3ec3 ldb_tdb: Disallow TDB nested transactions and use tdb_transaction_active()
This avoids keeping a counter, which can be error-prone.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
2018-05-03 08:17:44 +02:00
Gary Lockyer
501b35f861 ldb_tdb: Do not make search or DB modifications without a lock
The ldb_cache startup code would previously not take a read lock
nor a sufficiently wide write transaction.

The new code takes a read lock, and if it needs to write takes a
write lock (transaction) and re-reads before continuing.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
2018-05-03 08:17:44 +02:00
Gary Lockyer
39e5faa77c ldb: make key/value backends expose if there is an active transaction
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
2018-05-03 08:17:44 +02:00
Gary Lockyer
705cf715f9 ldb_tdb: Make sure max_key_length != 0 requires a GUID index mode
We need to enforce the GUID index mode so end-users do not get a supprise
in mid-operation and we enforce a max key length of 511 so that the
index key trunctation is done correctly.

Otherwise the DB will appear to work until a very long key (DN or index)
is used, after which it will be sad.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2018-05-03 08:17:44 +02:00
Stefan Metzmacher
e48bd332dc ldb/tests: don't use TEVENT_DEPRECATED in ldb_kv_ops_test.c
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-05-03 08:17:44 +02:00
Noel Power
db8da077ec lib/ldb: Additionally accept unicode as string param in Py2
With the changes to make samba python code Py2/Py3 compatible there
now are many instances where string content is decoded.
Decoded string variables in Py2 are returned as the unicode type. Many
Py2 c-module functions that take string arguments only check for the
string type. However now it's quite possibe the content formally passed
as a string argument is now passed as unicode after being decoded,
such arguments are rejected and code can fail subtly. This only affects
places where the type is directly checked e.g. via PyStr_Check etc.
arguments that are parsed by ParseTuple* functions generally already
accept both string and unicode (if 's', 'z', 's*' format specifiers
are used)

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
2018-04-30 15:43:19 +02:00
Douglas Bagnall
0d56edb9bb ldb/pyldb: initialize optional parameter in ldb_connect()
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
2018-04-13 07:27:14 +02:00
Volker Lendecke
4c8faa732e ldb: Fix trailing whitespace
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Böhme <slow@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Fri Apr  6 17:57:04 CEST 2018 on sn-devel-144
2018-04-06 17:57:03 +02:00
Ralph Boehme
567fe36c36 ldb/tests: remove lmdb.h include from test
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Fri Apr  6 14:58:48 CEST 2018 on sn-devel-144
2018-04-06 14:58:48 +02:00
Gary Lockyer
0935b25bd2 ldb: Unwind transaction counter if start_transaction fails
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
2018-04-06 02:08:45 +02:00
Gary Lockyer
5ca90e758a ldb tests: add cmocka tests of kv operations
Add tests for the behaviour the ldb layer expects the key value layer to
provide.  This should make it easier to add another KV store

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-04-06 02:08:45 +02:00
Gary Lockyer
1fceb64dc2 ldb_tdb: ltdb_tdb_delete require active transaction
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-04-06 02:08:45 +02:00
Gary Lockyer
e001c5fb5f ldb_tdb: ltdb_tdb_store require active transaction
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-04-06 02:08:45 +02:00
Gary Lockyer
8d1b11aac9 ldb_tdb: ltdb_tdb_parse_record map tdb error codes
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-04-06 02:08:45 +02:00
Gary Lockyer
d206fcf5c7 ldb tests: ldb_mod_op_test use correct ldb to create dn
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-04-06 02:08:45 +02:00
Gary Lockyer
0f7d153495 ldb test: close pipes to stop forked tests failing on failure
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-04-06 02:08:44 +02:00
Gary Lockyer
412cdb1714 ldb index: Add tests for truncated base 64 index keys
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-04-06 02:08:44 +02:00
Andrew Bartlett
556466e7e3 ldb_tdb: A more robust check for if we can fit the index string in
This avoids magic numbers and also is careful against overflow
from a long attr_for_dn.

This is done as a distinct commit to make the previous behaviour
change more clear, and to show that this does not change the
calculations, only improves the overflow check.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-04-06 02:08:44 +02:00
Gary Lockyer
d161a6dcfe ldb index: Fix truncation key length calculation
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-04-06 02:08:44 +02:00
Andrew Bartlett
eb4205fc31 ldb: Allow GUID index mode to be tested on TDB
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-04-06 02:08:44 +02:00
Andrew Bartlett
68d709741a ldb: Ignore these tests in mdb test mode
These are tests are specifically for when the GUID index is not in use
which is always in with ldb_mdb.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-04-06 02:08:44 +02:00
Garming Sam
7bf853950d ldb: Change remaining fetch prototypes to remove TDB_DATA
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-04-06 02:08:44 +02:00
Garming Sam
7885181650 ldb: Change some prototypes to using ldb_val instead of TDB_DATA
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-04-06 02:08:44 +02:00
Andrew Bartlett
f8b368c9f0 ldb: Fix missing NULL terminator in ldb_mod_op_test testsuite
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-04-06 02:08:44 +02:00
Andrew Bartlett
653a0a1ba9 ldb: Add test to show a reindex failure must not leave the DB corrupt
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13335

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Apr  5 07:53:10 CEST 2018 on sn-devel-144
2018-04-05 07:53:10 +02:00
Gary Lockyer
06d9566ef7 lib ldb tests: Prepare to run api and index test on tdb and lmdb
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13335

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-04-05 04:50:16 +02:00
Andrew Bartlett
e481e4f30f ldb_tdb: Ensure we can not commit an index that is corrupt due to partial re-index
The re-index traverse can abort part-way though and we need to ensure
that the transaction is never committed as that will leave an un-useable db.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13335

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-04-05 04:50:16 +02:00
Andreas Schneider
889d1daf87 ldb: Fix overflow checks
This fixes compilation with -Wstrict-overflow=2.

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2018-04-03 20:20:10 +02:00
Andreas Schneider
be709e8cde ldb: Add test for ldb_qsort()
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2018-04-03 20:20:10 +02:00
Stefan Metzmacher
666dda907b ldb/tests: avoid 'return void_function();' which isn't portable
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13343

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Björn Jacke <bjacke@samba.org>
2018-04-03 16:41:09 +02:00
Andreas Schneider
dcd0a7f738 ldb: Fix size types in ldb_ldif functions
This fixes compilation with -Wstrict-overflow=2

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-20 23:16:16 +01:00
Garming Sam
f0bebcc4e0 ldb_tdb: Remove unnecessary call to tdb_get_seqnum
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Mar  8 14:14:37 CET 2018 on sn-devel-144
2018-03-08 14:14:37 +01:00
Gary Lockyer
7f625f9b27 ldb_mod_op_test: Make sure that closing the database frees locks
Without the destructor firing, this test used to pass, but now we show
that we must be able to open a new ldb handle.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Mar  7 04:38:02 CET 2018 on sn-devel-144
2018-03-07 04:38:02 +01:00
Gary Lockyer
87708c3f91 ldb_mod_op_test: Add new nested transactions test
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-06 23:30:26 +01:00
Gary Lockyer
0009a12b1f ldb: Remove python warning in tests/python/index.py
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-06 23:30:26 +01:00
Garming Sam
4014499b12 ldb_tdb: Build a key value operation library
This allows sharing of the originally ldb_tdb operations to the new
ldb_mdb backend.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Mar  6 01:39:16 CET 2018 on sn-devel-144
2018-03-06 01:39:16 +01:00
Garming Sam
4d5180e45c ldb_tdb: Implement a traversal function in key value ops
This can handle both read-only and writable traverses.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-05 20:50:15 +01:00
Garming Sam
c9f2ff275b ldb_tdb: Use key value ops for fetch command
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-05 20:50:15 +01:00
Garming Sam
68423e9f4a ldb_tdb: factor out the (to be) common init code
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-05 20:50:15 +01:00
Gary Lockyer
141148edd3 ldb_tdb: Add errorstr to the key value ops
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-05 20:50:15 +01:00
Garming Sam
c66a0054fb ldb_tdb: Remove tdb_get_seqnum and use a generic 'has_changed'
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-05 20:50:15 +01:00
Garming Sam
e21a476c27 ldb_tdb: Add lock_read and unlock_read to key value ops
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-05 20:50:15 +01:00
Garming Sam
448101a3fd ldb_tdb: Replace tdb transaction code with generic key value ones
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-05 20:50:15 +01:00
Garming Sam
e33fb2c633 ldb_tdb: Replace exists, name and error_map with key value ops
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-05 20:50:15 +01:00
Garming Sam
9c8f00aaba ldb_tdb: Begin abstracting out the base key value operations
This will allow us to change the backend from tdb to lmdb.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-05 20:50:15 +01:00
Gary Lockyer
c4c64ff51f ldb_mod_op_test: Fix core dump on ldb_case_attrs_index_test_teardown
With no schema syntax, this would occasionally crash as it dereferenced
some possibly NULL sequence of memory.

Note: Removing all tests except this one, made it crash reliably.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-05 20:50:14 +01:00
Andreas Schneider
db4d72d21c ldb: Directly return an error and do not fall through
Detected by -Wimplicit-fallthrough.

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2018-03-05 16:01:17 +01:00
Gary Lockyer
4c0c888b57 ldb_tdb: Add tests for truncated index keys
Tests for the index truncation code as well as the GUID index
format in general.

Covers truncation of both the DN and equality search keys.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Mar  3 09:58:40 CET 2018 on sn-devel-144
2018-03-03 09:58:40 +01:00
Gary Lockyer
9a4fd23407 ldb_tdb: Combine identical not GUID index and special DN cases
Fold together two identical cases to simplify the code.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-03 05:20:35 +01:00
Gary Lockyer
ee74d4bb80 ldb_tdb: Refuse to store a value in a unique index that is too long
Rather than add many special cases, over-long unique values are simply banned.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-03 05:20:35 +01:00
Gary Lockyer
1d86a0895a ldb_tdb: Do not give the warning of duplicate attributes in truncation
In the truncation case a duplicate is perfectly expected.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-03 05:20:35 +01:00
Gary Lockyer
0918068997 ldb_tdb: Cope with key truncation
Modify the indexing code to handle a maximum key length, index keys
greater than the maximum length will be truncated to the maximum length.
And the unuque index code has been altered to handle multiple records
for the same index key.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-03 05:20:35 +01:00
Gary Lockyer
5c1504b94d ldb_tdb: Do not fail in GUID index mode if there is a duplicate attribute
It is not the job of the index code to enforce this, but do give a
a warning given it has been detected.

However, now that we do allow it, we must never return the same
object twice to the caller, so filter for it in ltdb_index_filter().

The GUID list is sorted, which makes this cheap to handle, thankfully.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-03 05:20:35 +01:00
Gary Lockyer
0bfbcdb691 ldb_tdb: Add support for an option to restrict the key length
Allow the setting of the maximum key length, this allows the testing of
index key truncation code.  Index key truncation is required to allow
the samba indexing scheme to be used with backends that enforce a
maximum key length.

This will allow emulation of a length-limited key DB for testing.

This is a testing-only feature, as the index format changes
based on this value.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-03 05:20:35 +01:00
Andreas Schneider
7adf497443 lib:ldb: Add FALL_THROUGH statements in ldb_map/ldb_map_outbound.c
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-01 04:37:41 +01:00
Andreas Schneider
9d2296e070 lib:ldb: Add FALL_THROUGH statements in ldb_map/ldb_map.c 2018-03-01 04:37:41 +01:00
Andreas Schneider
3429e9d9e5 lib:ldb: Add FALL_THROUGH statements in ldb_map/ldb_map_inbound.c
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-01 04:37:41 +01:00
Andreas Schneider
2b01bd0475 lib:ldb: Add FALL_THROUGH statements in common/ldb_dn.c
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-03-01 04:37:41 +01:00
Stefan Metzmacher
cb58e18800 ldb: version 1.3.2
* Expose the SHOW_BINARY, ENABLE_TRACING and DONT_CREATE_DB flag constants
  in the python api.
* Extend dn.is_child_of() test.
* Don't load LDB_MODULESDIR as a module file.
* Fix binary data in debug log (bug #13185).

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Wed Feb 28 04:54:21 CET 2018 on sn-devel-144
2018-02-28 04:54:21 +01:00
Gary Lockyer
2a85bcb3f4 ldb_debug tests: Fix binary data in debug log
Tests to ensure:
    When duplicate objects are added, the GUID was printed in the debug log
    are passed through the escape function.
    And that duplicate DN's do not generate debug log entries.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13185

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Feb 26 07:29:49 CET 2018 on sn-devel-144
2018-02-26 07:29:49 +01:00
Andrew Bartlett
c5a14306c8 ldb_debug: Fix binary data in debug log
When duplicate objects were added, the GUID was printed in the debug log
The GUID was not escaped and therefore displayed as binary content.

This patch splits out the duplicate DN creation error and the duplicate
GIUD error.  Duplicate DN's are a normal event and don't require debug
logging.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13185

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-02-26 02:32:09 +01:00
Gary Lockyer
1ed693423d ldb tests: fix null test on incorrect variable
Fix up tests that were  performing a null check on the wrong variable
after a call to ldb_msg_new

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Feb 24 15:50:35 CET 2018 on sn-devel-144
2018-02-24 15:50:35 +01:00
Timur I. Bakeyev
3450dd6a50 Don't load LDB_MODULESDIR as a module file
We are setting modules directory here(LDB_MODULESDIR) so treat it this
way, no need to attempt to load it as a module file.

Signed-off-by: Timur I. Bakeyev <timur@iXsystems.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Uri Simchoni <uri@samba.org>
2018-02-19 07:36:23 +01:00
Björn Baumbach
cf338b8260 pyldb: extend dn.is_child_of() test: dn is child of itself
Add this test so ensure that this (unclear) behaviour does
not change silently.

Signed-off-by: Björn Baumbach <bb@sernet.de>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-02-08 04:58:09 +01:00
Gary Lockyer
736e438bf4 pyldb: Expose extra flags
Expose the SHOW_BINARY, ENABLE_TRACING and DONT_CREATE_DB flag constants
in the python api.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-02-07 23:45:23 +01:00
Stefan Metzmacher
bf19b6ccdc ldb: version 1.3.1
* Intersect the index from SCOPE_ONELEVEL with the index for the search expression
  (bug #13191)
* smaller/greater comparison tests
* Show the last successful DN when failing to parse LDIF
* ldb_index: Add an attriubute flag to require a unique value.
* silence some clang warnings in picky developer mode

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-01-12 22:42:22 +01:00
Andrew Bartlett
ef240aaca0 ldb: Intersect the index from SCOPE_ONELEVEL with the index for the search expression
This helps ensure we do not have to scan all objects at this level
which could be very many (one per DNS zone entry).

However, due to the O(n*m) behaviour in list_intersect() for older
databases, we only do this in the GUID index mode, leaving the behaviour
unchanged for existing callers that do not specify the GUID index mode.

NOTE WELL: the behaviour of disallowDNFilter is enforced
in the index code, so this fixes SCOPE_ONELEVEL to also
honour disallowDNFilter, hence the additional tests.

The change to select the SUBTREE index in the absense of
the ONELEVEL index enforces this.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13191

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-12-20 04:22:09 +01:00
Matthias Dieter Wallnöfer
e72a8793ae LDB:test-generic.sh - fix smaller/greater comparison tests
The comparison result has been ignored, which is not good. Also remove
the "ldbsearch" command in the error branch which has not much sense.

The scripts needs to be run through test-tdb.sh, test-ldap.sh or
test-sqlite3.sh which I didn't realise before. Hence less changes are needed
and this is a reduced version of the patch published on the mailing list.

Signed-off-by: Matthias Dieter Wallnöfer <mdw@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Matthias Dieter Wallnöfer <mdw@samba.org>
Autobuild-Date(master): Tue Dec 19 03:09:12 CET 2017 on sn-devel-144
2017-12-19 03:09:12 +01:00
Andrew Bartlett
4ea7aa9265 ldb: Show the last successful DN when failing to parse LDIF
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-12-14 08:20:14 +01:00
Gary Lockyer
2599f29144 ldb ldb_index: Add an attriubute flag to require a unique value.
Add attribute flag LDB_FLAG_INTERNAL_UNIQUE_VALUE, to request that the
added attribute is unique on the index.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13004

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
2017-12-10 00:47:29 +01:00
Uri Simchoni
3cca62a2ac ldb: silence some clang warnings in picky developer mode
Avoid const in casting since it doesn't increase code
safety in this case and causes clang to generate const-qual
warning. Also initialize a pointer to NULL to silence clang
uninitialized variable warning.

Signed-off-by: Uri Simchoni <uri@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-11-22 10:20:20 +01:00
Andrew Bartlett
c40531d9b9 ldb: Release ldb 1.3.0
* GUID Index support.

  NOTE: When activated by setting @IDXGUID in the @INDEXLIST dn, all
  entries in the DB are re-keyed in a way that is NOT visible to
  ldb 1.2.2 and earlier.  To re-key back to the previous format, remove
  the @IDXGUID attribute from @INDEXLIST using ldb 1.2.2 or later.

  (ldb 1.2.2 can re-key, but not otherwise read, the new DB format).

* Give LDB_ERR_CONSTRAINT_VIOLATION, not LDB_ERR_ENTRY_ALREADY_EXISTS
  when a duplicate value is detected in a unique index

* Print status information during a > 10,000 entry re-index
  (as this can be slow)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Sep 23 01:24:19 CEST 2017 on sn-devel-144
2017-09-23 01:24:19 +02:00
Andrew Bartlett
303739e801 ldb_tdb: Treat distinguishedName and objectGUID (in Samba) as unique
This avoids loading any second index for these cases.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-22 21:20:25 +02:00
Andrew Bartlett
644b3ffb14 ldb: Ensure we do not run out of File descriptors in autobuild
The python TestCase API will keep a reference to the test object until the end
of the tests, long after we need the actual LDB or the fd.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-22 21:20:25 +02:00
Andrew Bartlett
a0b46ecdd2 ldb: Do not make an ldb file for API tests
An ldb context is valid without a backing file for tests of ldb.Message and ldb.MessageElement

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-22 21:20:25 +02:00
Andrew Bartlett
8565ca1c6d ldb: Add test showing a search against the index is not possible
This is not actually a great test, as the filter would
fail to match these anyway, but it at least checks the
codepath is safe.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:25 +02:00
Andrew Bartlett
f0624d3890 ldb_tdb: Also ban a (indexed) search against like @IDXDN=foo
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:25 +02:00
Andrew Bartlett
f4f9acf781 ldb_tdb: Update comments for the delete_index() pass of ltdb_reindex()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:25 +02:00
Andrew Bartlett
61b66b8d0a ldb_tdb: Print progress messages on re-index
A re-index of 10,000 entries is slow enough and rare enought that we can
justify the message being at LDB_DEBUG_WARNING as otherwise the administrator
will be sure the "lockup" was one.

The default for ldb is to print LDB_DEBUG_WARNING in comand-line tools
and the default for Samba is to log it at level 2.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
c71ddab974 ldb_tdb: Remove incorrect early return from re-index
The ltdb->cache->attribute_indexes test is not correct with the GUID index mode
so for consistency remove it.  This will make re-index on a large un-indexed
database slower, but that is better than making the wrong choice on a large
GUID-indexed database.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
0503f0984a ldb: Add more tests covering in-transaction GUID index and unique index behaviour
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
00410ba99f ldb_tdb: Add UNIQUE_INDEX as a possible flag
This allows easy testing of our unique index code and behaivour from python

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
561ef20634 ldb_tdb: Remove LTDB_FLAG_HIDDEN and ignore "HIDDEN" in @ATTRIBUTES
This was (unintentionally) disabled by
6ef6182554 in 2006.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
29c303f21b ldb_tdb: Clean up index records on ltdb_index_add_new() failure.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
83bc607463 ldb_tdb: Describe index format and control points
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
3113d871bd ldb_tdb: Give a good error message on add without an objectGUID
(or whatever the @IDX_GUID value is)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
b9d561b023 ldb_tdb: Avoid canonicalise and base64 work for DN values, these are already OK
This is important with the GUID index, as a DN lookup is much more common now.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
3e60f02753 ldb_tdb: Clean up list in after use in ltdb_key_dn_from_idx()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
d198a43b55 ldb_tdb: Avoid allocation of a DN between the GUID index and the DB lookup
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
4e9978506a ldb_tdb: Move constants into ldb_tdb.h
This helps ensure we keep these all in sync.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
b0a6b5ab51 ldb_tdb: Optimise ltdb_search_and_return_base() to re-use casefolding
The casefolding of a DN is one of the more expensive and pointless things in LDB
operation.  The ldb_dn abstraction works hard to avoid duplicating this work, but
we can work harder to save that information.

Here we copy in the DN, that has been casefolded already for the index,
and keep that as the returned DN, after stripping any extended components.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
f14370d471 ldb: Add tests for base DN of a different case
This ensures we cover the case where the DN does not match the DB exactly

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
dfe85ecaca ldb_tdb: Use the DN extracted from the DB to filter the message later
This should ensure that the upper or lower case the user chooses does not impact
on the filtering, at least for database that have checkBaseOnSearch set.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:24 +02:00
Andrew Bartlett
23e0a553d4 ldb_tdb: Add tests for add/modify with the GUID index
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
24c71ed211 ldb: Also test the new GUID index mode
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
64b4fbc613 ldb: Add an unused objectGUID to each record in SearchTests
This will then be used by the GUID index tests.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
a2ada5844d ldb_tdb: Duplicate values are no longer permitted in the index
By removing the qsort() we avoid work.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
3c6977b17c ldb_tdb: Reduce memory consumption in list_intersect()
We will never have more results than is in either list or list2.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
97b026a73f ldb_tdb: Use the binary search more efficiently in list_intersect()
This change ensures we walk the short list and look up into the longer of the two lists.

ltdb_dn_list_find_val() will do a binary search for the GUID case.

Before GUID indexes this was O(n*m), now it is O(n*log(m)).

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
b6bf7e7b0b ldb_tdb: Use a binary search to speed up ltdb_dn_list_find_val()
This only works if we have the GUID index format, as otherwise these are unsorted.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
fdff9a7087 ldb_tdb: Rework list_union to not return duplicates, and keep sort order
This allows the binary search to still operate on the list, even after
a or operator in the search expression

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
b86a46df1d ldb_tdb: Sort inputs to list_union()
This allows us to merge the lists finding common values.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
1390e55bb5 ldb_tdb: sort GUID index list at add time by inserting in sorted order
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
cf8537068f ldb_tdb: Change error code on unique index violation
The LDB_ERR_ENTRY_ALREADY_EXISTS error code is detected in repl_meta_data as indicating
that the DN exists, and that a conflict record should be created.

This is really a constraint violation, not a duplicate record.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
e8fb45125e ldb_tdb: Re-add of both existing DN and GUID must gives LDB_ERR_ENTRY_ALREADY_EXISTS
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
e16c8aa5da ldb_tdb: Add unique index checking for @IDXDN
This will give us errors earlier if the index code becomes broken

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
b97d556731 ldb_tdb: Improve debugging in ltdb_modify_index_dn() on casefold failure
This is unlikely, but when it happens it will be really painful to debug.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
1f42ded463 ldb_tdb: Add improved error strings on ltdb_key_dn_from_idx() failure
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
50f36e4fb5 ldb_tdb: Read from @INDEXLIST or an override if we are using a GUID index
This allows all the previous patches to be enabled.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
2f8a8c765f ldb_tdb: Optionally use GUID index values a direct TDB keys
This connects the GUID based index records to GUID based TDB keys.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:23 +02:00
Andrew Bartlett
93b18984bd ldb_tdb: Trust the BASE and ONELEVEL index
This avoids re-checking the fetched DN against the scope

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
e4562e58f9 ldb_tdb: Add ltdb_idx_to_key() and use it in ltdb_index_filter()
This will allow a common point to parse index records into a TDB key,
allowing them to be a GUID or DN in the future

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
ba762fac81 ldb_tdb: Do not add an index for GUID_index_attribute
This would be pointless and we no longer query for it.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
c4f35462e9 ldb_tdb: Do not query an index on the GUID_index_attribute
The objectGUID (or similar) is already the record key, there is
no need to index it to itself.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
6008382191 ldb_tdb: Optionally use GUID index in ltdb_search_dn1()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
fa44c5262b ldb_tdb: Use the objectGUID (or similar) as the TDB key in ltdb_key_msg()
When we have the full ldb_message we can read the objectGUID as the TDB key

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
f253dcf8b9 ldb_tdb: Use ltdb_key_msg() in ltdb_delete_noindex()
This allows the optional use of GUID based TDB key.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
6ad4bdc9de ldb_tdb: Add mem_ctx to ltdb_key_dn() and ltdb_key_msg()
This follows modern Samba coding style where memory
returned is allocated on a supplied memory context.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
fb77ad4244 ldb_tdb: Check version number on index
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
e394b9a856 ldb_tdb: Add an index shortcut for a <GUID= DN
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
ccb9443664 ldb_tdb: Add a function to get the GUID key for a DN
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
f26d1a8c5d ldb_tdb: Add a function to take a GUID and make the TDB_DATA key
This allows us to format the TDB key as DN=GUID=f7c953ee-cf9c-433f-b423-21ce04d09591
and so be compatible with an un-indexed search and a re-index with an old ldb.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
aa68957f7f ldb_tdb: Load the syntax of the GUID index attr during ltdb_cache_load()
This allows us to use the ldif_write function later to create a string GUID for the TDB key.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
8555f6ccc2 ldb_tdb: add control points for the new GUID index mode
The @IDXGUID attribute in the @INDEXLIST will be objectGUID
in Samba.

The @IDX_DN_GUID attribute in the @INDEXLIST will be GUID
in Samba.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
3b0698e0b6 ldb_tdb: Do not directly assign DN into the index result list
With the GUID index option, the values in the index result list may
not be a DN but the objectGUID.  We look up the @IDXDN index with the
case-folded DN to get that if required.

We re-use the code from the SCOPE_BASE search do avoid duplication
and for reliablity.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
19ee0438ff ldb_tdb: Pass ltdb_private to ltdb_dn_list_load()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:22 +02:00
Andrew Bartlett
2bac00a7b9 ldb_tdb: Give LDB_ERR_CONSTRAINT_VIOLATION when a duplicate GUID index is detected
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
06e67c5894 ldb_tdb: Give LDB_ERR_ENTRY_ALREADY_EXISTS when a duplicate DN index is detected
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
ec30439c43 ldb_tdb: Add/remove a GUID index of the DN during ltdb_index_add_all()/ltdb_index_delete()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
98e82113ed ldb_tdb: Split ltdb_index_onelevel() into a helper function
This will allow the code to be re-used for storing the DN->GUID index

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
b2aff12366 ldb_tdb: Prepare to handle rename with GUID index by using ltdb_search_base()
This will allow use of a GUID TDB key in the future.  While ltdb_search_base()
might be marginally slower than tdb_exists(), no allocation is done for the
attributes or DN, and renmaes are not a very common operation.

This allows a check if the target DN exists even when the direct DN -> key
link is broken.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
c24df8e431 ldb_tdb: Implement ltdb_search_base() for a GUID index
The GUID index case can not directly use ltdb_key_dn() and tdb_exists() to
show that a records exists.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
754329a9d9 ldb_tdb: Optionally store a GUID as the index record
This allows, when enabled, the index record to contain (say) the objectGUID, not the DN
of the record.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
b154acb0c7 ldb_tdb: Optionally use a GUID index key in ltdb_dn_list_find_msg()
This function is used to find an existing index value and this
change allows it to find the value by GUID rather than by DN once
the GUID index is enabled.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
9d9ae1c005 ldb_tdb: Pass the full ldb_message to ldb index funtions
This allows the objectGUID, rather than the DN, to be the index key

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
4c01ccb47e ldb_tdb: Delete a successful tdb_store on index add fail in ltdb_add_internal()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
e8cdacc509 ldb_tdb: modify ltdb_delete_noindex() to take a struct ldb_message
This will make it easier to delete records with the GUID TDB key

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
de641db8b7 ldb_tdb: Write GUID index values as version 3
Nothing reads these currently, but we should refuse to load a mixed up index
in the future

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
91e6028c08 ldb_tdb: Refuse to load a GUID index that is not a multiple of 16 bytes
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
ad26b4e292 ldb_tdb: Read GUID index values as one packed LDB attribute
This packing should be more efficient to read than the ldb_pack format.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
f6e953d316 ldb_tdb: Store GUID index values in one packed ldb attribute
This should make them more memory efficient

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:21 +02:00
Andrew Bartlett
367b7ab748 ldb_tdb: Move constants into ldb_tdb.h
This helps ensure we keep these all in sync.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:20 +02:00
Andrew Bartlett
c17404826e ldb_tdb: replace strange dn_list_cmp() in index code
This replaces dn_list_cmp() with functions that do not attempt to
to care about string termination.  All index values are case sensitive
and correctly length-bound already, even for a DN index
so just use a length check and memcmp()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:20 +02:00
Andrew Bartlett
a44d3aedba ldb_tdb: Do not allow a modification of the GUID_index_attribute (objectGUID)
This would totally break our index scheme if this could be modified.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:20 +02:00
Andrew Bartlett
26dd6f6365 ldb_tdb: Add GUID_index_attribute to ltdb_private
This will be used to determine if we are in GUID index mode

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:20 +02:00
Andrew Bartlett
3d952157d7 ldb_tdb: Provide struct ltdb_private to index routines
This will make it easier to switch the GUID index mode on and off

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:20 +02:00
Andrew Bartlett
dded01598e ldb_tdb: Use a more complete error mapping in ltdb_search_key()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:20 +02:00
Andrew Bartlett
6db8095835 ldb_tdb: Add ltdb_search_key()
This allows us to slowly split out the tdb key in the DB from being the DN

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:20 +02:00
Andrew Bartlett
d4c1a600b1 ldb_tdb: Use ltdb_key_msg() in re_index()
This will allow changing to a GUID tdb key in the future

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:20 +02:00
Andrew Bartlett
1b310ad99c ldb_tdb: provide ldb_key_dn() and ldb_key_msg()
This will in time allow us to generate a TDB key from
the msg, eg from an objectGUID.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:20 +02:00
Andrew Bartlett
cd8ecb3332 ldb_tdb: Add helper function ltdb_search_and_return_base()
This avoids an extra DB lookup for the base, when that is the only
record we will return, and avoids going into the index code for
a base search, as that won't work for special DNs once the GUID
index mode is enabled.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:20 +02:00
Andrew Bartlett
d8a22884c1 ldb_tdb: Make ldb_match_message() available to ldb_tdb
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-22 21:20:20 +02:00
Andrew Bartlett
6c28abc249 ldb: Release 1.2.3
* Bug #13033 LDB open with LDB_FLG_RDONLY can cause the database
   to fail to open

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13033

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-20 02:25:30 +02:00
Gary Lockyer
f5f3657c48 ldb: Add tests for read only behaviour
As the kernel is no longer enforcing the read-only DB
add some tests.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13033

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-20 02:25:30 +02:00
Andrew Bartlett
22854f9b8d ldb_tdb: Change ltdb_connect() NOT to request a kernel-level read only TDB
We support opening and LDB multiple times in a process, but do not support this in tdb.

As we can open the ldb with different flags, we must ensure a later read-write
open is possible.

Additionally, a read-only TDB will refuse the all-record lock, preventing
the ldb from even loading.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13033

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-20 02:25:30 +02:00
Andrew Bartlett
13777d35d0 ldb_tdb: Give a debug message as well as setting the error string if prepare_commit() fails
This is a serious condition, and should be logged.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13033

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-20 02:25:30 +02:00
Andrew Bartlett
75e88e40a2 ldb_tdb: Map TDB error codes into LDB error codes in ltdb_lock_read()
The ltdb_lock_read() routine did not return an LDB error code, but -1.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13033

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-20 02:25:30 +02:00
Andrew Bartlett
ba54816875 ldb: version 1.2.2
* Bug #13017: Add ldb_ldif_message_redacted_string() to allow debug
              of redacted log messages, avoiding showing secret values

* Bug #13015: Allow re-index of newer databases with binary GUID TDB keys
              (this officially removes support for re-index of the original
              pack format 0, rather than simply segfaulting).
* Avoid memory allocation and so make modify of records in ldb_tdb faster

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:27 +02:00
Andrew Bartlett
37bb62990b ldb: Add new ldb_ldif_message_redacted_string() with tests
This is designed to be a drop in replacement for
ldb_ldif_message_string() while better protecting privacy.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13017

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:27 +02:00
Andrew Bartlett
01c49b1a35 ldb_tdb: Refuse to re-index very old database with no DN in the record
These are not found on any AD DC, and would segfault previous LDB
versions.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13015

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:27 +02:00
Andrew Bartlett
5463601122 ldb_tdb: Use braces in ltdb_dn_list_find_val()
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13015

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:26 +02:00
Andrew Bartlett
6246c326a7 ldb_tdb: Check for talloc_strdup() failure in ltdb_index_add1()
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13015

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:26 +02:00
Andrew Bartlett
ffc8023a0e ldb_tdb: Check for errors during tdb operations in ltdb_reindex()
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13015

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:26 +02:00
Andrew Bartlett
3ce80cfb60 ldb_tdb: Use memcmp rather than strncmp() in ltdb_key_is_record(), re_key() and re_index()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:26 +02:00
Andrew Bartlett
fec666b334 ldb_tdb: Create a common ltdb_key_is_record() allowing multiple key forms
If backported, this allows old ldb versions to full-search and re-index newer databases
and in current code allows the transition to and from a GUID or incrementing ID based index

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13016

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:26 +02:00
Andrew Bartlett
2d0007ee5a ldb_tdb: Do not trigger the unique index check during a re-index, use another pass
We want to rename the objects, then scan looking for the index values.

This avoids a DB modify during the index scan traverse (the index values
are actually added to an in-memory TDB, written in prepare_commit()).

This allows us to remove the "this might already exist" case in the
index handling, we now know that the entry did not exist in the index
before we add it.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13015

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:26 +02:00
Andrew Bartlett
da1e23418a ldb_tdb: Use memcmp() to compare TDB keys in re_index()
The keys may not always be a null terminated string, they could well
be a binary GUID in a future revision, for efficiency..

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13016

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:26 +02:00
Andrew Bartlett
a5a2243f07 ldb: Add tests for indexed and unindexed search expressions
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Aug 30 14:58:32 CEST 2017 on sn-devel-144
2017-08-30 14:58:32 +02:00
Andrew Bartlett
841e763a2b ldb: Fix tests to call the parent tearDown(), not setUp in tearDown
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-08-30 10:48:20 +02:00
Andrew Bartlett
3164c0ac54 ldb_tdb: Rework ltdb_modify_internal() to use ltdb_search_dn1() internally
This avoids duplicate code and allows us to use the allocation-avoiding
LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC flag.

We can not use LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC as el2->values
is talloc_realloc()ed in the routine.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Aug 29 11:13:50 CEST 2017 on sn-devel-144
2017-08-29 11:13:50 +02:00
Andrew Bartlett
bff81a2c9c ldb: Add LDB_UNPACK_DATA_FLAG_NO_ATTRS
This will allow us to avoid a full unpack in situations where we just want to confirm
if the DN exists

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:29 +02:00
Jeremy Allison
99e4bea89e lib: ldb: Python. Take care of freeing the passed in module description if ldb_register_module() fails.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12932

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-26 21:35:22 +02:00
Jeremy Allison
41b1f8a20c lib: ldb: Use NULL to allocate modules not talloc_autofree_context().
ldb modules are not (yet) unloaded and are only loaded once (there is a check
that makes sure of this). Allocate off the NULL context. We never want this
to be freed until process shutdown. If eventually we add the ability to
unload ldb modules we can add a deregister function that walks and frees the list.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12932

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-26 21:35:21 +02:00
Stefan Metzmacher
19dfccea74 ldb: version 1.2.1
* Bug #12882: Do not install _ldb_text.py if we have system libldb
* Use libraries from build dir for testsuite
* Bug #12900: Fix index out of bound in ldb_msg_find_common_values

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Sat Jul 22 03:46:25 CEST 2017 on sn-devel-144
2017-07-22 03:46:25 +02:00
Douglas Bagnall
4b3de61185 ldb/tests: more thoroughly test empty ldb_msg elements
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Fri Jul  7 20:10:37 CEST 2017 on sn-devel-144
2017-07-07 20:10:37 +02:00
Douglas Bagnall
282410fa24 ldb: avoid searching empty lists in ldb_msg_find_common_values
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-07-07 15:51:25 +02:00
Lukas Slebodnik
963d9f12f9 ldb: Fix index out of bound in ldb_msg_find_common_values
cmocka unit test failed on i386
[==========] Running 2 test(s).
[ RUN      ] test_ldb_msg_find_duplicate_val
[       OK ] test_ldb_msg_find_duplicate_val
[ RUN      ] test_ldb_msg_find_common_values
[  FAILED  ] test_ldb_msg_find_common_values
[==========] 2 test(s) run.
[  ERROR   ] --- 0x14 != 0
[   LINE   ] --- ../tests/ldb_msg.c:266: error: Failure!
[  PASSED  ] 1 test(s).
[  FAILED  ] 1 test(s), listed below:
[  FAILED  ] test_ldb_msg_find_common_values
 1 FAILED TEST(S)

But we were just lucky on other platforms because there is
index out of bound according to valgrind error.

==3298== Invalid read of size 4
==3298==    at 0x486FCF6: ldb_val_cmp (ldb_msg.c:95)
==3298==    by 0x486FCF6: ldb_msg_find_common_values (ldb_msg.c:266)
==3298==    by 0x109A3D: test_ldb_msg_find_common_values (ldb_msg.c:265)
==3298==    by 0x48E7490: ??? (in /usr/lib/libcmocka.so.0.4.1)
==3298==    by 0x48E7EB0: _cmocka_run_group_tests (in /usr/lib/libcmocka.so.0.4.1)
==3298==    by 0x1089B7: main (ldb_msg.c:352)
==3298==  Address 0x4b07734 is 4 bytes after a block of size 48 alloc'd
==3298==    at 0x483223E: malloc (vg_replace_malloc.c:299)
==3298==    by 0x4907AA7: _talloc_array (in /usr/lib/libtalloc.so.2.1.9)
==3298==    by 0x486FBF8: ldb_msg_find_common_values (ldb_msg.c:245)
==3298==    by 0x109A3D: test_ldb_msg_find_common_values (ldb_msg.c:265)
==3298==    by 0x48E7490: ??? (in /usr/lib/libcmocka.so.0.4.1)
==3298==    by 0x48E7EB0: _cmocka_run_group_tests (in /usr/lib/libcmocka.so.0.4.1)
==3298==    by 0x1089B7: main (ldb_msg.c:352)

Signed-off-by: Lukas Slebodnik <lslebodn@redhat.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-07-07 15:51:25 +02:00
Andreas Schneider
98e30cf491 waf: Do not install _ldb_text.py if we have system libldb
_ldb_text.py is installed as part of the ldb package and also if you
compile Samba with the system ldb version. This way we have have the
file twice in the same location and run into file confilcts.

This has already been fixed some time ago:
    60dc26bfe1

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12882

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
2017-07-07 09:14:26 +02:00
Lukas Slebodnik
d5d6d209de ldb: Use libraries from build dir for testsuite
There was a failure when tests were executed after after extracting
ldb tarball.

  sh$ make -j8 check
  WAF_MAKE=1 PATH=buildtools/bin:../../buildtools/bin:$PATH waf test
  ldbadd: error while loading shared libraries: libldb.so.1: cannot open shared object file: No such file or directory
  cat: write error: Broken pipe
  Traceback (most recent call last):
    File "tests/python/api.py", line 10, in <module>
      import ldb
  ImportError: libldb.so.1: cannot open shared object file: No such file or directory
  Traceback (most recent call last):
    File "tests/python/api.py", line 10, in <module>
      import ldb
  ImportError: libpyldb-util.so.1: cannot open shared object file: No such file or directory
  bin/ldb_tdb_mod_op_test: error while loading shared libraries: libldb.so.1: cannot open shared object file: No such file or directory
  testsuite returned 1

Signed-off-by: Lukas Slebodnik <lslebodn@redhat.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Alexander Bokovoy <ab@samba.org>
2017-07-04 21:39:22 +02:00
Stefan Metzmacher
3f0a2500ce ldb: version 1.2.0
* handle one more LDB_FLAG_INTERNAL_DISABLE_SINGLE_VALUE_CHECK
  case in ldb_tdb
* fix ldb_tdb locking (performance) problems
* fix ldb_tdb search inconsistencies by adding
  read_[un]lock() hooks to the module stack
  (bug #12858)
* add cmocka based tests for the locking issues
* ldb_version.h provides LDB_VERSION_{MAJOR,MINOR,RELEASE} defines
* protect ldb_modules.h from being used by Samba < 4.7
  Note: that this release (as well as 1.1.30 and 1.1.31)
  may cause problems for older applications, e.g. Samba
  See https://bugzilla.samba.org/show_bug.cgi?id=12859

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-02 17:35:20 +02:00
Stefan Metzmacher
0fac7e4d73 ldb:includes: protect ldb_modules.h from being used by Samba < 4.7
Samba versions before 4.7 are incompatible with the read_[un]lock()
behaviour introduced into ldb.

This makes sure older Samba versions fail to compile against
ldb >= 1.2.0.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12859

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-02 17:35:20 +02:00
Stefan Metzmacher
a1158d2237 ldb:wscript: define EXPECTED_SYSTEM_LDB_VERSION_{MAJOR,MINOR,RELEASE}
This indicates what feature set Samba assumes from the used
libldb from the system.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12859

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-02 17:35:19 +02:00
Stefan Metzmacher
774b2e4f5a ldb:wscript: provide LDB_VERSION_{MAJOR,MINOR,RELEASE} in ldb_version.h
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
b8b8439f1a ldb:tests: Extend api.py testsuite to show transaction_commit() blocks against the whole-db read lock
The new ldb whole-db lock behaviour now allows this test

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
6d7208c36f ldb:tests: Extend api.py testsuite to show transaction contents can not be seen outside the transaction
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
86e8425418 ldb:tests: Add test to show that locks are released on TALLOC_FREE(req)
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
4b7c274cc0 ldb:tests: Correct comment about version numbers
(ldb releases have been made while this patch set was in train)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
a8958515f0 ldb: Lock the whole backend database for the duration of a search
We must hold locks not just for the duration of each search, but for the whole search
as our module stack may make multiple search requests to build up the whole result.

This is explains a number of replication and read corruption issues in Samba

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
d3c0df35c9 ldb_tdb: Implement read_lock and read_unlock module operations
This allows Samba to provide a consistent view of the DB
despite the use of multiple databases via the partitions module
and over multiple callbacks via a module stack.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
6a3e77edc3 ldb: Add read_lock and read_unlock to ldb_module_ops
This will be used to implement read locking in ldb_tdb

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
31ef1d6211 ldb:tests: Add test encoding current locking behaviour during ldb_search()
Currently, a lock is not held against modifications once the final
record is returned via a callback, so modifications can be made
during the DONE callback.  This makes it hard to write modules
that interpert an ldb search result and do further processing
so will change in the future to allow the full search to be
atomic.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
0cd75c1fb1 ldb:tests: Show that writes do not appear during an ldb_search()
A modify or rename during a search must not cause a search to change
output, and attributes having an index should in particular not see
any change in behaviour in this respect

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
fd0beef179 ldb:tests: don't assert the results before doing the final search finished
This is required to pass the test in future, because
otherwise the clean up will fail because we hold locks.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Garming Sam
289d7d8fc5 ldb:tdb: Ensure we correctly decrement ltdb->read_lock_count
If we do not do this, then we never take the all record lock, and instead do a lock
for every record as we go, which is very slow during a large search

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Andrew Bartlett
25e580f86e ldb_pack: use ldb_dn_from_ldb_val() and avoid a duplicate strlen() call
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-07-02 17:35:19 +02:00
Volker Lendecke
b67262b40a ldb: Fix CID 1412926 Unchecked return value
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-06-20 18:50:06 +02:00
Garming Sam
990b23d7b6 replmd: check single values in replmd_add_fix_la
repl_meta_data knows whether linked attributes are appropriately
[un-]duplicated, and this is how it tells ldb_tdb that.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Pair-programmed-with: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-06-15 17:33:10 +02:00
Douglas Bagnall
6f956cce62 ldb: 1.1.31
* Add efficient function to find duplicate values in ldb messages
  (this makes large multi-valued attributes in ldb_tdb more efficient)

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-06-15 17:33:10 +02:00
Douglas Bagnall
7d6f36aae3 ldb: relatively efficient functions for finding duplicate values
ldb backends need to make sure they are not adding duplicate values to
multi-valued attributes in ADD and MODIFY operations. Until now they
have done this inefficiently using nested loops. Here we add common
functions that deal with large numbers of values in O(n log n) time,
but continue to use the simple methods for small numbers of values.

These functions take a struct ldb_context pointer and an options flag
arguments, although the ldb is not used, and only one bit of the
options has meaning. This is to allow further patches to switch on
schema-aware comparisons.

This entails an ABI jump to add the two new functions.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-06-15 17:33:10 +02:00
Douglas Bagnall
aa61a2212b ldb.h whitespace
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-06-15 17:33:10 +02:00
Douglas Bagnall
16d208acc4 ldb tests/ldb_mod_op_test: don't double include cmocka.h
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-06-15 17:33:10 +02:00
Douglas Bagnall
433d600c56 ldb: fix a typo
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-06-15 17:33:10 +02:00
Douglas Bagnall
44764ee33d ldb: fix whitespace in ldb_msg.c
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-06-15 17:33:10 +02:00
Andrew Bartlett
f587db7c89 ldb: Rename module -> next_module for clarity
This helps make some future commits less confusing

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-06-15 01:24:25 +02:00
Andrew Bartlett
2277301e46 ldb_tdb: Improve logging on unique index violation
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-06-15 01:24:25 +02:00
Andrew Bartlett
f6f73f7091 ldb_tdb: Remove the idxptr DB before we re-index
We do not want the cache or any of the values in it, we want to read the real DB
@INDEX: records.

This matters if a re-index is tiggered in the same transaction
as the modify of the values in the index.  Otherwise we won't see
the old index record (it will not show up in the tdb_traverse)
and so fail to remove it.

That in turn can cause a spurious unqiue index violation.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-06-15 01:24:25 +02:00
Andrew Bartlett
d8f3034c16 ldb_tdb: Check for memory allocation failure in ltdb_index_transaction_start()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-06-15 01:24:25 +02:00
Andrew Bartlett
13e09c72cd ldb: Version 1.1.30
* let ldbdump parse the -i option
* don't allow the reveal_internals control for ldbedit
* only allow --show-binary for ldbsearch
* don't let ldbsearch create non-existing files
* fix ldb_tdb search inconsistencies
* add cmocka based tests
* provide an interface for improved indexing for callers
  like Samba, which will allow much better performance.
* Makes ldb access to tdb:// databases use a private event context
  rather than the global event context passed in by the caller.
  This is because running other operations while locks are held
  or a search is being conducted is not safe.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:27 +02:00
Andrew Bartlett
e001ac9ef5 ldb: Add Doxygen documentation for ldb_handle_use_global_event_context()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-05-31 06:34:27 +02:00
Andrew Bartlett
850732ebbb ldb: Add Doxygen docs for ldb_set_require_private_event_context()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-05-31 06:34:27 +02:00
Andrew Bartlett
ba3eff930f ldb: Add Doxygen docs for ldb_handle_get_event_context()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-05-31 06:34:27 +02:00
Andrew Bartlett
d73d926458 ldb: Add Doxygen docs for ldb_schema_set_override_indexlist()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-05-31 06:34:27 +02:00
Andrew Bartlett
bcd891dc3f ldb: Add Doxygen docs for ldb_schema_attribute_set_override_handler
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-05-31 06:34:27 +02:00
Andrew Bartlett
cb03b12ac2 ldb: Add Doxygen comments for ldb_req_*trusted() functions
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-05-31 06:34:27 +02:00
Andrew Bartlett
02a4915b91 ldb: Add test for ldb_build_search_req()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:27 +02:00
Andrew Bartlett
6777770d35 ldb: Add tests for new ldb handle and event context behaviour
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:27 +02:00
Andrew Bartlett
c29201d6d3 ldb: Add ldb_handle_use_global_event_context()
This will allow the IRPC to be processed in the main event loop of the
server, not the private event context for this request

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:27 +02:00
Andrew Bartlett
e67d3568e5 ldb: Use the private event context in ldb_tdb and ldb_wait()
This enables the previous commits, and ensures that ldb_tdb is safe from operations while locks
are held

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:27 +02:00
Andrew Bartlett
25795c3da9 ldb: Force use of a private event context in ldb_tdb
ldb_tdb holds locks while making callbacks, so force the use of a per-request event context

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
cd97f45976 ldb: Create private event contexts in top level requests, chain to children
We must ensure that the ldb_request we call ldb_wait() will share an event context with all
the eventual request that the ldb backend creates events on

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
0f13243f5e ldb: Add ldb_set_require_private_event_context()
This will allow us to force use of the global event context for use when Samba
must make an IRPC call from within the ldb stack, to another part of the same
process

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
a83df55693 ldb: Add ldb_handle_get_event_context()
This will allow us to obtain a private event context for use while we hold
locks in ldb_tdb, that is not shared with the global state of the application.

This will ensure we do not perform other operations while we hold the lock

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
1ba6b9aae8 ldb: Add ldb_build_req_common() helper function
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
aa54ed0365 ldb: Add tests for the schema and index override hooks
Because this uses ldb_private.h we no longer build the
test binary if we are building against a system ldb

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
e903e08027 ldb: Move test_ldb_attrs_case_insensitive closer to setup/teardown functions
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
2335e57a89 ldb_tdb: Avoid reading the index list from the DB if we are already set to override it
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
e584fe85a5 ldb: Allow a caller (in particular Samba) to handle the list of attributes with an index
By doing that, Samba will use a binary search to locate the attributes
rather than an O(n) search, during every search or modify of the database.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
29dc93298a ldb_tdb: consistently use ltdb->cache->attribute_indexes to determine if we have indexes
This is instead of checking the number of elements via ltdb->cache->indexlist->num_elements

In turn, this allows us to avoid fetching ltdb->cache->indexlist in the future

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
e1bc43dca1 ldb_tdb: change the arguments to ldb_is_indexed() to provide the ltdb_private
By doing this, we can be more efficient in locating if we have an index in
the future.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
05e8dcb2f9 ldb_tdb: Split index load out into a sub-funciton: ltdb_index_load
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-31 06:34:26 +02:00
Andrew Bartlett
c23b191ac4 ldb_tdb: Call talloc_free(options_dn) as soon as we are done with options_dn
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-23 01:13:25 +02:00
Andrew Bartlett
496feccad4 ldb_tdb: Provide better debugging on end_trans failures
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-23 01:13:24 +02:00
Andrew Bartlett
a2a1962dc0 ldb_tdb: Provide better debugging on prepare_commit failures
ltdb_index_transaction_commit() does LDB operations, sets the ldb
error string and returns LDB errors so we should not check the tdb
error code.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-23 01:13:24 +02:00
Andrew Bartlett
47a8fb8e70 ldb: Do not use mktemp() nor leak files into /tmp during api.py test
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-06 12:03:16 +02:00
Andrew Bartlett
be87e9dfa3 ldb: Add test for transaction deadlock detected when waiting for a search
This was the original intent of 7dd31288a7
but was broken in 251aaafe3a and
hidden by 4bb2958f16.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-06 12:03:16 +02:00
Andrew Bartlett
e7f3b457d6 ldb: Add some tests to clarify the current iterator behaviour
search_iterator() is no more memory efficient than search() because all the results
come back at the first res.next() call

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-05-06 12:03:16 +02:00
Jakub Hrozek
3d049f2aad ldb:tests: Unit test the ldb_rename() operation
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:26 +02:00
Jakub Hrozek
a05fda2f3f ldb:tests: Add tests for case insensitive searches
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:26 +02:00
Jakub Hrozek
f44c0827eb ldb:tests: unit test for ldb_search()
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:26 +02:00
Jakub Hrozek
68a00330af ldb:tests: Add a modify test
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:26 +02:00
Jakub Hrozek
5113116a83 ldb:tests: Add a test for ldb transactions
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:26 +02:00
Jakub Hrozek
dddac9fc3c ldb:tests: Add a basic delete test
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:26 +02:00
Jakub Hrozek
b423c3e8d5 ldb:tests: Add a basic search test
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:26 +02:00
Jakub Hrozek
dbe1dd5fab ldb:tests: A rudimentary ldb_add() test
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:26 +02:00
Andreas Schneider
b49ee16b9c ldb:tests: Build a ldb test for the tdb backend
Pair-Programmed-With: Andrew Bartlet <abartlet@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:26 +02:00
Jakub Hrozek
15ad5de6aa ldb:tests: Add a simple cmocka test for ldb_connect()
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:26 +02:00
Jakub Hrozek
317f3c6f3d ldb: Clarify LDB_MODULES_PATH is used
Make it (hopefully more) clear where modules are loaded from.

Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:26 +02:00
Jakub Hrozek
1a90b55da6 ldb_tdb: Remove unused function parameter
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:25 +02:00
Jakub Hrozek
1c7c4b474c ldb_tdb: Remove unused function parameter
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:25 +02:00
Jakub Hrozek
9d4168e745 ldb_tdb: Remove unused function ltdb_add_attr_results
Signed-off-by: Jakub Hrozek <jakub.hrozek@posteo.se>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-04-19 15:46:25 +02:00
Garming Sam
ee04f96b69 ldb_tdb: Add better comments for duplicate attr values
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Pair-programmed-with: Andrew Bartlett <abartlet@samba.org>
2017-03-13 05:10:10 +01:00
Garming Sam
b562a90646 ldb_tdb: Do not check for duplicate values during a rename
This is not the time to be pretending to be dbcheck, and there are
exceptions to the single-value rules in Samba. This is needed for
the same reasons as the modify case.

(Note: this error was triggered with the demote of an RODC with links)

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Pair-programmed-with: Andrew Bartlett <abartlet@samba.org>
2017-03-13 05:10:10 +01:00
Garming Sam
3b5aeaba95 ldb_tdb: Do not care about duplicates if single value check disabled
This behaviour of ignoring duplicates with the flag
LDB_FLAG_INTERNAL_DISABLE_SINGLE_VALUE_CHECK is also used in the replace
case here.

When we add a forward DN+Binary link with a duplicate DN, this prevents
us from not being able to add the backlink because it appears to be a
duplicate here.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Pair-programmed-with: Bob Campbell <bobcampbell@catalyst.net.nz>
2017-03-13 05:10:10 +01:00
Ian Stakenvicius
ce956be075 waf: disable-python - align ldb's wscript
If samba is set to use a system copy of ldb, and ldb wasn't built with
python support, then no system pyldb-util will be found.  If samba is
being built without python support then pyldb-util isn not needed, so
do not bother to try and find it.

The system ldb check had to be duplicated due to the earlier commits
which changed order of ldb and pyldb-util checks, and by association
also added a dependency of pyldb-util onto ldb.  This seemed cleaner
than messing with variables.

The build configuration for pyldb-util needs to exist even if it's
not being built, so that dependency resolution can occur throughout
the rest of the samba build system -- this required dropping the higher
level conditional and using the enabled= parameter instead.

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-03-10 07:31:12 +01:00
Andrew Bartlett
583ff0a0e3 lib/ldb: Enable use of a python3 pyldb-util system library
To do this, we have to install a .pc file for the python3 pyldb-util

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Incorportaing fixes by Petr Viktorin <pviktori@redhat.com>

Signed-off-by: Petr Viktorin <pviktori@redhat.com>
2017-03-10 07:31:10 +01:00
Douglas Bagnall
bb4ad8dffe pyldb: p3k readiness: allow single unicode string in msg element
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-03-10 07:31:09 +01:00
Stefan Metzmacher
35f92b82b5 ldb: add LDB_FLG_DONT_CREATE_DB
This avoids creating an new tdb files on ldbsearch
or other callers which use LDB_FLG_DONT_CREATE_DB.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Mar  9 16:02:21 CET 2017 on sn-devel-144
2017-03-09 16:02:21 +01:00
Stefan Metzmacher
a2daa664cc ldb/tools: only use LDB_FLG_SHOW_BINARY for 'ldbsearch'
--show-binary is only useful for ldbseach in all other cases
it will destroy data.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Feb 24 03:59:01 CET 2017 on sn-devel-144
2017-02-24 03:59:01 +01:00
Garming Sam
bed19f3744 ldbedit: Prevent the use of the reveal internals control
This is almost certainly not what you want to do. Providing the output of reveal as the input of modify
will necessarily revivify all dead linked attributes (regardless of --extended-dn or not).

This is extremely unexpected behaviour, so we prevent this from happening.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12596

Autobuild-User(master): Garming Sam <garming@samba.org>
Autobuild-Date(master): Wed Feb 22 04:23:05 CET 2017 on sn-devel-144
2017-02-22 04:23:05 +01:00
Stefan Metzmacher
4b295b106c wscript: remove executable bits for all wscript* files
These files should not be executable.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Wed Jan 11 20:21:01 CET 2017 on sn-devel-144
2017-01-11 20:21:01 +01:00
Garming Sam
77b51ba2f2 ldb_tdb: avoid erroneous error messages
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Autobuild-User(master): Garming Sam <garming@samba.org>
Autobuild-Date(master): Fri Dec 23 02:28:54 CET 2016 on sn-devel-144
2016-12-23 02:28:54 +01:00
Garming Sam
a18e115ffe ldbdump: Parse the -i option
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2016-12-22 22:42:22 +01:00
Björn Jacke
c44e1916eb man pages: change http://samba.org to https://www.samba.org
Signed-off-by: Bjoern Jacke <bj@sernet.de>
Reviewed-by: Karolin Seeger <kseeger@samba.org>
2016-12-09 13:10:26 +01:00
Andrew Bartlett
aa63600afb ldb: new ldb version 1.1.29
- new OID LDB_CONTROL_RECALCULATE_RDN_OID
 - honour LDB_CONTROL_RECALCULATE_RDN_OID in rdn_modify
 - fix handling of @ATTRIBUTES containing * in rdn_modify
 - improve startup performance on AD DC databases
  - These have lots of index attributes and attributes specified

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2016-12-01 05:54:24 +01:00
Andrew Bartlett
6ca5e9ee1c ldb: Add test for behaviour of rdn_name
Cover a wildcard in @ATTRIBUTES and the normal case.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2016-12-01 05:54:23 +01:00
Andrew Bartlett
4dfe84a984 ldb: Cope with a->name being *
The default schema when loaded from the DB will have a name of *, not NULL.

This feature is rarely used, and was incompatible with the rdn_name module
until now.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2016-12-01 05:54:23 +01:00
Stefan Metzmacher
ea3c96647f ldb:rdn_name: add support for LDB_CONTROL_RECALCULATE_RDN_OID on ldb_modify()
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12399

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2016-12-01 05:54:23 +01:00
Stefan Metzmacher
25aa26178f ldb:rdn_name: normalize rdn_name in rdn_rename_callback()
We already do that on 'add'.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12399

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2016-12-01 05:54:23 +01:00
Andrew Bartlett
f599a2d244 ldb: load @ATTRIBUTES faster by sorting once, not at each insertion
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2016-12-01 05:54:23 +01:00
Andrew Bartlett
393b8f3c1d ldb: Add helper function ldb_schema_attribute_fill_with_syntax()
This will allow us to avoid calling ldb_schema_attribute_add_with_syntax()
in a tight loop.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2016-12-01 05:54:23 +01:00