1
0
mirror of https://github.com/samba-team/samba.git synced 2025-01-07 17:18:11 +03:00
Commit Graph

1290 Commits

Author SHA1 Message Date
Andreas Schneider
b3bc2c950f lib:ldb: Use tdb_traverse_read() in ldb_kv_index_sub_transaction_commit()
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Thu Sep 26 09:59:34 UTC 2024 on atb-devel-224
2024-09-26 09:59:34 +00:00
Andreas Schneider
03c2f3d1fd lib:ldb: Use tdb_parse_record() in ldb_kv_sub_transaction_traverse()
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2024-09-26 08:46:29 +00:00
Andreas Schneider
3582250538 lib:ldb: Use tdb_parse_record() in ldb_kv_dn_list_store()
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2024-09-26 08:46:29 +00:00
Andreas Schneider
090f8009b3 lib:ldb: Use tdb_parse_record() in ldb_kv_dn_list_load()
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2024-09-26 08:46:28 +00:00
Andreas Schneider
fdef894d79 ldb: Build lmdb backend also in non-AD case
We should build with lmdb support also if it is not in AD case. The lmdb
backend is also used e.g. by sssd.

If you don't want to build it, you can always specify --without-ldb-lmdb

BUG: https://bugzilla.samba.org/show_bug.cgi?id=15721

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Wed Sep 25 05:36:13 UTC 2024 on atb-devel-224
2024-09-25 05:36:13 +00:00
Douglas Bagnall
3d47fc749a ldb:test:api_search: shift remaining setUp adds to class add_index
Before:

ldb.python.api_search -> 52
ldb.python.api_search.tr -> 48

After:

ldb.python.api_search.tr -> 10
ldb.python.api_search -> 9

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Tue Sep 24 10:38:03 UTC 2024 on atb-devel-224
2024-09-24 10:38:03 +00:00
Douglas Bagnall
45137017af ldb:test:api_search: remove pass-through setUp()s
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:39 +00:00
Douglas Bagnall
1871eb9d1c ldb:test:api_search: use @unittest.skipIf for LMDB tests
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:39 +00:00
Douglas Bagnall
235c7d15ef ldb:test:api_search: skip LMDB tests at class level
LMDB is not going to turn up midway through the testsuite, so we might
as well skip the whole class

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:39 +00:00
Douglas Bagnall
d9225d2460 ldb:test:api_search: use class method to add index
The index attribute of these classes was a mutable object, which made
me nervous. If a subclass were to alter the class.index (rather than
replace it), it would affect tests in other subclasses.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:39 +00:00
Douglas Bagnall
f7269c2fa3 ldb:test:api_search: tidy up files in each subclass
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:39 +00:00
Douglas Bagnall
2029cd7a7b ldb:test:api_search: set up the database once and use copies
The api_search tests are really slow, in part because we populate the
database identically for all the individual tests. We can instead do
it once and use copies.

This makes the test several times faster.
Comparing "make test TESTS=ldb.python":

Before:

TOP 10 slowest tests
ldb.python.api_search.tr -> 415
ldb.python.api_search -> 413
ldb.python.api_add_modify -> 5
ldb.python.api_add_modify.tr -> 4
ldb.python.index.tr -> 4
ldb.python.index -> 4
ldb.python.api_simple.tr -> 2
ldb.python.api_simple -> 2
ldb.python.repack -> 2
ldb.python.api_misc.tr -> 1
'testonly' finished successfully (14m13.276s)

After:

TOP 10 slowest tests
ldb.python.api_search.tr -> 52
ldb.python.api_search -> 50
ldb.python.api_add_modify -> 7
ldb.python.api_add_modify.tr -> 7
ldb.python.index.tr -> 4
ldb.python.index -> 4
ldb.python.api_simple -> 4
ldb.python.api_simple.tr -> 2
ldb.python.repack -> 2
ldb.python.api_misc -> 1
'testonly' finished successfully (2m14.505s)

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
98d408be86 ldb:test:api_search: use test name as db name
This will help enusre we are using the right DB when (in the next
commit) we set up the database for each test via copy. It also helps
with debugging.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
146540a128 ldb:test:api_search: put config options on class, not instance.
This makes them available to class methods.

In a later commit we will convert the index attribute into an
add_index() class method, because having a mutable variable on a class
is a slightly bad idea.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
69a00fd0d7 ldb:test:api_search: improve attribute access tests
`list = res.msgs` tests nothing more than a bare `res.msgs`,
and we can instead assert that the result is a list.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
b08427d086 ldb:test:api_base: make flags method a class method
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
8d97ce958f ldb:test:api_base: simplify prefix selection
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
83f3afaac4 ldb:tests:index: use abi_base
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
0c968d374a ldb:tests:crash: rationalise imports
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
102660e7d1 ldb:tests:api_simple uses conventional 4 space indent
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
b932143c8d ldb:tests:repack: use common api_base variables
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
ccc732dae2 ldb:tests:api_misc: fix Control test
`self.assertRaises(TypeError, ldb.Control, ldb, 1234)` is raising a
TypeError because `ldb` is not an ldb.Ldb object, it is the ldb module.

What we want to test here is that the non-string `1234` raises a
TypeError -- the previous line tests the type of the ldb argument.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
c706021d55 ldb:tests: move api.py to api_misc
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
081125f3b3 ldb:tests: make api_simple module
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
565eba4120 ldb:tests: make api_add_modify module
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
e4410ada21 ldb:tests: make api_search module
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
6e947a20ee ldb:tests: make api_base module
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
16820c883d ldb:tests:api.py uses .disconnect before rmdir
super.tearDown() was removing the tmpdir, but because self.ldb had
the file open, the directory was not cleared.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
09d9680f5e ldb:pytest:api: remove unnecessary super() parameters
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
b1fd6e2b07 ldb:pytest:api: remove pass-though tearDowns
The result is exactly the same, unless we previously had the wrong
class name in the pass-through, in which case the result is *probably*
the same, only more correct.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
64e3c5cba9 ldb:py bindings: ldb.Ldb().__str__ prints connection URL
before:  "<ldb connection>"
after:   "<ldb connection tdb:///path/to/samdb.ldb>"

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
2024-09-24 09:14:38 +00:00
Douglas Bagnall
e2a74963fb ldb:kv_index: help static analysers to not worry (CID 1615192)
The point of this realloc is that we are not using this array, but
keeping it around to remain a node the talloc tree. We'd prefer to
reduce it to nothing.

Coverity rightly spotted that it was reallocing an array of `struct
ldb_val` to an array of `struct ldb_val *`, which has a different size
and all. But it doesn't matter in this case, because we will never use
it.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=15590

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Jennifer Sutton <josutton@catalyst.net.nz>
2024-08-28 04:24:39 +00:00
Jo Sutton
5851ae5554 ldb: Fix ldb public library header files being unusable
An accidental negation means that ldb_version.h is not installed when
ldb is built as a public library.

This is a regression introduced by commit
625fb48326.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=15690

Signed-off-by: Jo Sutton <josutton@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Sun Aug  4 01:35:55 UTC 2024 on atb-devel-224
2024-08-04 01:35:55 +00:00
Jule Anger
dbc5b73d86 ldb: change the version to 2.11.0 for Samba 4.22
Signed-off-by: Jule Anger <janger@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Jule Anger <janger@samba.org>
Autobuild-Date(master): Mon Jul 29 10:06:23 UTC 2024 on atb-devel-224
2024-07-29 10:06:23 +00:00
Douglas Bagnall
e58e4a5aa9 ldb:kv_index: use subtransaction_cancel in transaction_cancel
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Sat Jul 27 23:51:44 UTC 2024 on atb-devel-224
2024-07-27 23:51:44 +00:00
Douglas Bagnall
18131aebf8 ldb:kv_index: subtransaction_cancel: check for nested tdb
Just in case, but also so ldb_kv_index_transaction_cancel() can use
this and retain the same logic.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2024-07-27 22:47:39 +00:00
Douglas Bagnall
9ecf96859f ldb:kv_index: don't recalculate a length
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2024-07-27 22:47:39 +00:00
Douglas Bagnall
1bf9ede94f ldb:kv_index: realloc away old dn list
We can't just free it, because has the GUID index list as a child, and
these are shared by the new dn list (from the subtransaction we are
committing). But if the dn list is long and the main transaction is
long-lived, we can save a lot of memory by turning this dn list into
an almost empty node in the talloc tree. This returns us to roughly
the situation we had prior to the last commit.

For example, with the repro.sh script on bug 15590 in indexes mode
with 10000 rules, The last 3 commits use this much memory at the end
of an unusually large transaction:

full talloc report on 'struct ldb_context' (total 4012222 bytes in 90058 blocks)
full talloc report on 'struct ldb_context' (total 2405482219 bytes in 90058 blocks)
full talloc report on 'struct ldb_context' (total 4282195 bytes in 90058 blocks)

That is, the last commit increased usage 500 fold, and this commit
brings it back to normal.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=15590

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2024-07-27 22:47:39 +00:00
Douglas Bagnall
5f0198d698 ldb_kv_index: dn_list load sub transaction can re-use keys
We don't want to modify the original list, but we can reuse the keys
if we treat them as immutable and don't free them. That makes it a lot
quicker if there are many keys (i.e. where an index is useful) and may
sub-transactions. In particular, it avoids O(n²) talloc_memdups.

A removed comment that says "We have to free the top level index
memory otherwise we would leak", and this will be addressed in the
next commit.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=15590

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2024-07-27 22:47:39 +00:00
Douglas Bagnall
ed7bc50b00 ldb:ldb_kv_dn_list_find_val: check for int overflow
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2024-07-27 22:47:39 +00:00
Douglas Bagnall
d8c7768fed ldb_kv_cache: always initialise dn_list.strict
The strict flag is only read in list intersection, so most of the time
it doesn't matter whether it is set because that path is not used.
Nevertheless seeing it set to all kinds of values is distracting.

The undefined behaviour has likely been hidden from static analysis
because the structure is passed through the in-memory tdb before use.

Incorrect true values will have disabled an optimisation but not
caused the wrong result.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2024-07-27 22:47:39 +00:00
Jo Sutton
4e8ca6140a ldb: Attach appropriate ldb context to returned result
This is done by adding a new API that avoids the problems of
ldb_dn_copy() and makes it clear that a struct ldb_context *
pointer will be stored in the new copy.

Signed-off-by: Jo Sutton <josutton@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2024-06-11 04:32:30 +00:00
Andrew Bartlett
e178f6b0e9 ldb_wrap: Provide a way to avoid Samba using ldb_wrap()
ldb_wrap is a caching mechansim, and it should probably be removed
but for now provide a way to avoid it in specific cases where we
know it is harmful.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jo Sutton <josutton@catalyst.net.nz>
2024-06-10 04:27:30 +00:00
Jo Sutton
38cfdb6623 ldb: Fix typo
Signed-off-by: Jo Sutton <josutton@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2024-06-04 19:49:36 +00:00
Douglas Bagnall
d6581d213d ldb: move struct ldb_debug_ops to ldb_private.h
Only accessed through struct ldb_context -> debug_ops, which is already private.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu May 23 00:19:30 UTC 2024 on atb-devel-224
2024-05-23 00:19:30 +00:00
Douglas Bagnall
6dd68d8978 ldb: move struct ldb_utf8_fns to ldb_private.h
It is only accessed via ldb functions that find it on the already-private
struct ldb_context.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2024-05-22 23:12:32 +00:00
Douglas Bagnall
b22e1d3207 ldb: don't cast to unsigned for ldb_ascii_toupper()
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2024-05-22 23:12:32 +00:00
Douglas Bagnall
e33a0dd70f ldb: ldb_set_utf8_functions follows README.Coding
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2024-05-22 23:12:32 +00:00
Douglas Bagnall
4a6a1d1f0a ldb: deprecate ldb_set_utf8_fns
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2024-05-22 23:12:32 +00:00
Douglas Bagnall
42ae85d70a ldb: remove old ldb_comparison_fold_utf8_broken()
There are no callers.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2024-05-22 23:12:32 +00:00