IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Change all uses of ldb_kv_filter_attrs() to use
ldb_filter_attrs_in_place() instead. This function does less work than
its predecessor, and no longer requires the allocation of a second ldb
message. Some of the work is able to be split out into separate
functions that each accomplish a single task, with a purpose to make the
code clearer.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15270
Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The LDB filter processing is where the time is spent in the LDB stack
but the timeout event will not get run while this is ongoing, so we
must confirm we have not yet timed out manually.
RN: Ensure that the LDB request has not timed out during filter processing
as the LDAP server MaxQueryDuration is otherwise not honoured.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14694
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Store it on the repack context so that we can log a more informative
message "Repacking from format x to format y".
While this is not really a big deal currently, it could be worth
recording for potential future scenarios (i.e. supporting three or more
pack versions), where upgrades could potentially skip an intermediary
pack format version.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14059
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
When performing a join the overhead of the sub transactions protecting
key value operations becomes significant. This commit adds a new
"batch_mode" option that disables the sub transactions around key value
operations.
The operation level index cache is also disabled, which means the
overall transaction level index cache can become inconsistent if an
operation fails. To protect against this and other possible on disk
inconsistencies, if any operation fails during a batch_mode
transaction the commit will fail and transaction will be rolled back.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
As a performance enhancement the key value layer maintains a cache of
the index records, which is written to disk as part of a prepare commit.
This patch adds an extra cache at the operation layer to ensure that the
cached indexes remain consistent in the event of an operation failing.
Add test to test for index corruption in a failed modify.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Use the nested transaction support added to the key value back ends to
make key value operations atomic. This will ensure that rename
operation failures, which delete the original record and add a new
record, leave the database in a consistent state.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add limited nested transaction support to the back ends to make the key value
operations atomic (for those back ends that support nested transactions).
Note: that only the lmdb backend currently supports nested transactions.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
For TDB databases, toggling GUID indexing mode will also toggle
pack format version 2. This provides a convenient downgrade path for
Samba databases, but the process doesn't work for MDB databases because
GUID indexing cannot be disabled when the MDB backend is used. This patch
addresses that corner case by providing support for a pack_format_override
option which will force the database to use pack format version 2.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>
VERY IMPORTANT PATCH
Now that we have a new packing format, we need to enable it by repacking
the database. We've decided to link all new database features together,
so once GUID indexing is enabled, the database will be repacked with
version 2 format. Repacking is done following the same iterate pattern as
reindexing.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
We want to reuse the reindex context struct for repacking, but it has an
unnecessary module pointer on it. Turns out the existing code doesn't
need it either, so this patch deletes the pointer.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
We will be adding a new packing format in forthcoming commits and there
may be more versions in the future. We need to make sure the database
contains records in a format we know how to read and write.
Done by fetching the @BASEINFO record and reading the first 4
bytes which contain the packing format version.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Optimising filter_attrs by removing msg and dn allocation/copying. The
caller can construct the msg and possibly steal the dn.
Also giving the function an ldb for future use.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
This makes LDAP bind significantly faster in the case of having many
members, due to large size of these records (with tens of thousands of
member links). During the nested group calculation, you are only
interested in memberOf not the member links.
(We add a bit-field to determine whether or not the backend actually
supports pointing into database memory. For some reason TDB pointers
aren't stable, so for now we set this option just on LMDB backends.)
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
@ records like @IDXLIST are only available via a base search on the specific name
but the method by which they were excluded was expensive, after the unpack the
DN is exploded and ldb_match_msg_error() would reject it for failing to match the
scope.
This uses the fact that @ records have the DN=@ prefix on their TDB/LMDB key
to quickly exclude them from consideration.
Based on analysis by Garming Sam.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13893
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Apr 10 06:23:39 UTC 2019 on sn-devel-144
Adding iterate_range to LDB API and implementing in LMDB. This
operation takes a start_key and end_key and returns all records between
the two, inclusive of both. This will be used to implementing indexing
for <= and >= expressions.
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Set the default index cache from the passed option
"transaction_index_cache_size" on open. This allows the default cache
size to be overridden when processing large transactions i.e. joining a
large domain.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add the get_size method to the ldb_key_value layer, this will allow the
reindexing code to get an estimate of the number of records in the
database.
The lmdb backend returns an accurate count of the number of records in
the database withe the mdb_env_stat call.
The tdb backend does not provide a low cost method to determine the
number of records on the database. It does provide a tdb_summary call
however this this walks the entire database.
So for tdb we use the map size divided by 500, this over estimates the counts
for small domains, but the extra memory allocated for the cache should
not be significant.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Pass the index cache size to ldb_kv_index_transaction_start. This will
allow it to be set for reindex and join operations, where the current
defaults result in a significant performance penalty on large databases.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Convert the key value functions to use ldb_val instead of TDB_DATA.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Jul 30 17:23:22 CEST 2018 on sn-devel-144
Rename all the LTDB_* constants to LDB_KV_* as they are key value level
constants and not tdb specific.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Move the key value code to a separate subdirectory.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
