IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Add limited nested transaction support to the back ends to make the key value
operations atomic (for those back ends that support nested transactions).
Note: that only the lmdb backend currently supports nested transactions.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Remove the "DeprecationWarning: Please use assertEqual instead."
warnings from api.py
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add a test to document that ldb does not currently support nested
transactions.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Fixes csbuild error.
Error: COMPILER_WARNING:
lib/ldb/ldb_key_value/ldb_kv_search.c: scope_hint: In
function ‘search_func’ lib/ldb/ldb_key_value/ldb_kv_search.c:308:47:
warning: unused parameter ‘ldb_kv’ [-Wunused-parameter]
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Jun 13 08:27:12 UTC 2019 on sn-devel-184
Fixes csbuild error.
Error: COMPILER_WARNING:
lib/ldb/ldb_key_value/ldb_kv_index.c: scope_hint: In function
‘ldb_kv_index_add1’
lib/ldb/ldb_key_value/ldb_kv_index.c:2601:15: warning: comparison of
integer expressions of different signedness: ‘int’ and ‘unsigned int’
[-Wsign-compare]
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Fixes csbuild error.
Error: COMPILER_WARNING:
lib/ldb/ldb_key_value/ldb_kv_index.c: scope_hint: In function
‘ldb_kv_index_traverse_store’
lib/ldb/ldb_key_value/ldb_kv_index.c:779:60: warning: unused parameter
‘tdb’ [-Wunused-parameter]
By tagging the tdb_context parameter in ldb_kv_index_traverse_store as
_UNUSED_
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Fixes csbuild error.
Error: COMPILER_WARNING:
lib/ldb/ldb_key_value/ldb_kv_index.c: scope_hint: In function
‘ldb_kv_key_dn_from_idx’
lib/ldb/ldb_key_value/ldb_kv_index.c:545:15: warning: comparison of
integer expressions of different signedness: ‘int’ and ‘unsigned int’
[-Wsign-compare]
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Add lib/util/attr.h to the ldb distribution, to make the _UNUSED_ macro
available.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Fixes:
lib/tevent/tevent_wrapper.c:295:3: warning: Access to field 'next' results in a dereference of a null pointer (loaded from field 'prev') <--[clang]
Additionally fix similar instance of the same macro
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Gary Lockyer gary@catalyst.net.nz
Calling the "ldb_parse_tree" function with a filter consisting of
exactly a single space (" ") would trigger a read beyond the input
buffer. A unittest is included.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13900
Signed-off-by: Michael Hanselmann <public@hansmi.ch>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Tim Beale <timbeale@catalyst.net.nz>
* Default pack format changed to version 2 for GUID Indexed databases
(essentially Samba's sam.ldb). This means faster unpacking for records
with lots of attributes.
* Enforce link between all new database features, following toggling of
GUID indexing. If user toggles GUID indexing on/off, target pack format
is changed. Likewise the special ORDERED_INTEGER index format is used
(when GUID indexing on) or behaves like INTEGER (when GUID indexing off).
* Database repacked if packing format not as expected.
* In the case of MDB, since GUID indexing is mandatory, feature toggling
provided by pack_format_override LDB option.
* Check for errors from ldb_unpack_data() in ldb_tdb
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed May 29 05:56:59 UTC 2019 on sn-devel-184
For TDB databases, toggling GUID indexing mode will also toggle
pack format version 2. This provides a convenient downgrade path for
Samba databases, but the process doesn't work for MDB databases because
GUID indexing cannot be disabled when the MDB backend is used. This patch
addresses that corner case by providing support for a pack_format_override
option which will force the database to use pack format version 2.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>
Python test to confirm that after enabling or disabling GUID indexing,
the database is repacked on the next transaction with V1 if GUID
indexing was disabled, or V2 if it was enabled.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
VERY IMPORTANT PATCH
Now that we have a new packing format, we need to enable it by repacking
the database. We've decided to link all new database features together,
so once GUID indexing is enabled, the database will be repacked with
version 2 format. Repacking is done following the same iterate pattern as
reindexing.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Only enable ORDERED_INTEGER and index_format_fn functionality if GUID
indexing is enabled. This is in line with the rest of ldb_kv which binds
the new ORDERED_INTEGER to GUID indexed databases, and allows a practical
way to create the old index format (by disabling the GUID index).
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>
We need to check for the errors given by ldb_unpack() et al by preserving
the error code from kv_ctx->parser() called by tdb_parse_record() in
ltdb_parse_record().
Otherwise we will silently accept corrupt records and segfault later.
Likewise new pack formats will confuse the parser but not be
detected except by the incomplete struct ldb_message.
With this patch, the user will see a message like:
Invalid data for index DN=@BASEINFO
Failed to connect to 'st/ad_dc/private/sam.ldb' with backend 'tdb': Unable to load ltdb cache records for backend 'ldb_tdb backend'
Failed to connect to st/ad_dc/private/sam.ldb - Unable to load ltdb cache records for backend 'ldb_tdb backend'
This can be refined in the future by a specific check for
pack format versions in a higher caller, but this much is
needed regardless to detect corrupt records.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13959
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Parse errors aren't passed up correctly by the tdb backend. This
patch modifies a test to expose the issue, next patch will fix it.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13959
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Ensure that the referrals returned in a search request use the same
scheme as the request, i.e. referrals recieved via ldap are prefixed
with "ldap://" and those over ldaps are prefixed with "ldaps://"
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12478
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri May 24 05:12:14 UTC 2019 on sn-devel-184
For testing we need to know the actual KV level key of records and each
record's pack format version. This patch makes ldbdump add comments with
that info. We will parse it out in python tests.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed May 22 05:58:17 UTC 2019 on sn-devel-184
In future commits we'll be adding more logging to LDB, which breaks the
ldb_key_value_test suite. By removing the debug handler, a bug
involving an expired debug_string variable being written to is avoided.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
We want to reuse the reindex context struct for repacking, but it has an
unnecessary module pointer on it. Turns out the existing code doesn't
need it either, so this patch deletes the pointer.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
* Checking pack format is version 1 and erroring if not (will change soon)
* Pack format routines for unpack and pack version 2 (but not used)
* Test fixes for issues caused by upcoming repack functionality for upgrade
* Making ldbdump print out pack format info and keys so we have low level visibility for testing in python
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Pack function for new pack format with values separated from other data
so that while unpacking, the value section (which is probably large)
doesn't have to be loaded into cache/memory.
The new format is disabled for now.
Two tests are added that operate on a detailed binary breakdown of the
new format.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Unpack function for new pack format with values separated from other
data so that while unpacking, the value section (which is probably large)
doesn't have to be loaded into cache/memory.
Additionally, width of length field can now vary per-element to save space.
The old unpack routine is still present and is called if the old pack
format version number is found.
LDB torture suite is modified to run relevant tests on both old and new
pack format.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Since we're about to introduce a new packing format, it's a good time to
improve our code style and change some magic numbers into explicit
constants.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Replacing push and pull functions (which may cause issues with Undefined
Sanitizer) with Andreas Schneider's excellent macros which are a work in
progress and not yet merged into master. Once his work is upstream, I'll
rebase and change this code to import his headers.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
We will be adding a new packing format in forthcoming commits and there
may be more versions in the future. We need to make sure the database
contains records in a format we know how to read and write.
Done by fetching the @BASEINFO record and reading the first 4
bytes which contain the packing format version.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
To avoid warning above produced by using
-Wcast-function-type we;
+ ensure PyCFunctions of type METH_NOARGS defined dummy arg
+ ensure PyCFunctions of type METH_KEYWORDS use PY_DISCARD_FUNC_SIG
macro
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
* Rework pack routines to remove unused features:
- ldb_unpack_data_only_attr_list_flags() is removed
- LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC is now always implied
This improves the unpack performace significantly.
* Improve search performance via new internal flag
LDB_UNPACK_DATA_FLAG_READ_LOCKED which removes a memdup
* Improve search performance during full scan by ignoring
index records early.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Unpack functions currently take an attribute list to restrict the set of
attributes to be returned in the constructed message. This
functionality is never used and complicates implementation of
forthcoming new pack format. This patch removes that functionality.
Using the unpack function then filtering the result turns
out not to be any slower.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
This patch moves ldb_kv's filter code into the pack code to replace
'only attr list' functionality which will be removed in forthcoming
commit. Unpacking data then filtering the result is not any slower
than the removed 'only attr list' approach.
'only attr list' test repurposed to test unpack -> filter flow.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Optimising filter_attrs by removing msg and dn allocation/copying. The
caller can construct the msg and possibly steal the dn.
Also giving the function an ldb for future use.
NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Making unpack flag LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC required
behaviour, since allocating data during unpack is slow and unnecessary
in all current usages. In any future unpack usage, if editing of
returned memory is required, some function that duplicates the message
should be used, such as one of the filter_attrs functions, or msg_copy.
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
This more modern routine allocates a nice talloc tree.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
The DN is now exploded so as to improve other aspects of the search handling.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Previously we half-heartedly checked one end.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
