1
0
mirror of https://github.com/samba-team/samba.git synced 2024-12-25 23:21:54 +03:00
Commit Graph

3158 Commits

Author SHA1 Message Date
Andrew Bartlett
0f7aa3db52 lib/krb5_wrap: Bring list of all enc types into krb5_wrap 2012-08-28 07:57:28 +10:00
Stefan Metzmacher
95f3662bbd lib/util: don't SMB_ASSERT() in process_exists_by_pid()
Just return false...

metze
2012-08-24 13:43:32 +02:00
Stefan Metzmacher
f83521a4ca lib/param: fix usage of 'write list = +Group'
metze

Autobuild-User(master): Björn Jacke <bj@sernet.de>
Autobuild-Date(master): Fri Aug 24 11:28:17 CEST 2012 on sn-devel-104
2012-08-24 11:28:17 +02:00
Björn Jacke
b4252f8fa6 crypto/aes_ccm_128: fix compile warning on openindiana 2012-08-23 16:43:48 +02:00
Matthias Dieter Wallnöfer
32cd618e6c LDB:ldb_tdb.c - deny multi-valued attributes manipulation with doublets
This refers to LDB add operations as well, we have only to be careful on
"@ATTRIBUTES" entries.

E.g.

dn: cn=testperson,cn=users,dc=...,dc=...
objectClass: person
url: www.example.com
url: www.example.com

should not work.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
2012-08-22 01:31:57 +02:00
Matthias Dieter Wallnöfer
cb63b34b05 LDB:ldbsearch - add search filter tests
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
2012-08-22 01:31:57 +02:00
Matthias Dieter Wallnöfer
6a8c697954 LDB:ldbsearch - search filters do not only contain "="
Also "<=", ">=", "~"... are allowed as well. Enumeration taken from
ldb_parse_filtertype().
This was the cause of not identifying the search filter as described in bug
https://bugzilla.samba.org/show_bug.cgi?id=8647.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
2012-08-22 01:31:55 +02:00
Matthias Dieter Wallnöfer
536c082505 LDB:ldif_handlers.c - LDB_OP_GREATER/LDB_OP_LESS are thought as ">=" or "<="
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
2012-08-22 01:31:55 +02:00
Andrew Bartlett
4d5471f1c6 build: Remove special case for the build farm
Except in the formatting of the selftest output, this removes the special case
of the build farm, so that an autobuild, a manual make test and the build farm
are more similar.

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Aug 21 06:39:04 CEST 2012 on sn-devel-104
2012-08-21 06:39:04 +02:00
Michael Adam
471a853e34 lib/util: add server_id_set_disconnected() and server_id_is_disconnected()
Utility functions for handling the special placeholder server-id value
for disconnected clients (to be used for durable handles).

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
2012-08-17 20:07:07 +02:00
Stefan Metzmacher
3cdf441da1 lib/util: let server_id_str() skip the task_id if 0 in the cluster case too
server_id_from_string() already handles that case.

metze
2012-08-17 20:07:07 +02:00
Stefan Metzmacher
d4a0aeb49a lib/util: add server_id_from_string()
metze
2012-08-17 20:07:07 +02:00
Stefan Metzmacher
8149623ad4 lib/util: add server_id_equal()
metze
2012-08-17 20:07:07 +02:00
Stefan Metzmacher
d7d8646e63 tevent: change version to 0.9.17 after adding the "poll_mt" backend
metze
2012-08-16 21:10:32 +02:00
Volker Lendecke
d7af2c842a tevent: Add threaded poll_mt testcase
Signed-off-by: Stefan Metzmacher <metze@samba.org>
2012-08-16 20:52:17 +02:00
Volker Lendecke
fa71f32411 lib/tevent: Add a thread-safe tevent backend
Signed-off-by: Stefan Metzmacher <metze@samba.org>
2012-08-16 20:49:11 +02:00
Volker Lendecke
d860aa2cac tevent_poll: Decouple poll_ev->fds handling from adding/removing fds
Step 1 in a python backend for multiple threads

Signed-off-by: Stefan Metzmacher <metze@samba.org>
2012-08-16 20:48:32 +02:00
Björn Jacke
1f50b6c3ae tdb/test: fix build on OSF/1
Autobuild-User(master): Björn Jacke <bj@sernet.de>
Autobuild-Date(master): Fri Aug 10 23:33:20 CEST 2012 on sn-devel-104
2012-08-10 23:33:20 +02:00
Björn Jacke
13f8674a15 build: rename security → samba-security
there is a libsecurity on OSF1 which clasheѕ with our security lib. see bug #9023.

Signed-off-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Björn Jacke <bj@sernet.de>
Autobuild-Date(master): Fri Aug 10 14:22:21 CEST 2012 on sn-devel-104
2012-08-10 14:22:20 +02:00
Andrew Bartlett
1d1bdc315b lib/ldb: Use tdb_parse_record and a callback rather than tdb_fetch()
This avoid allocation at the tdb layer as we will allocate this
with talloc right away anyway.

Andrew Bartlett
2012-08-10 10:18:27 +10:00
Andrew Bartlett
11d60d13dc s4-ldb_wrap: Do not vasprintf() the ldb debug messages that will not be shown
This malloc() and free() actually shows up quite high on a call profile of
provision of the AD DC.

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Aug  9 13:51:06 CEST 2012 on sn-devel-104
2012-08-09 13:51:06 +02:00
Andrew Bartlett
73f0cb5278 lib/ldb: Do not vasprintf() the tevent debug messages that will not be shown
This malloc() and free() actually shows up quite high on a call profile of
provision of the AD DC.

This allows the debug handler to decide if the argument list should be
printed.

Andrew Bartlett
2012-08-09 20:14:23 +10:00
Andrew Bartlett
299fc75228 lib/ldb: Use tdb_exists() rather than tdb_fetch()/talloc_free()
This avoids pulling the record and doing an allocation when we just
want to know if it exists.

Andrew Bartlett
2012-08-09 20:14:22 +10:00
Christian Ambach
0dfc33026e lib/socket_wrapper: writev returns ssize_t, not int 2012-08-06 18:07:52 +02:00
Christian Ambach
18c152f8aa lib/param: move enum dns_update_settings to lib/param 2012-08-06 18:07:52 +02:00
Andrew Bartlett
4e0fb8b4b1 Revert "ldb: Add parameter to avoid NULL format string flagged by -Werror=format"
This reverts commit cddcaf7bd2.

-Werror=format is no longer uses when it would cause this to fail.
 This is a legitimate use of the ldb_search API.

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Aug  6 14:47:27 CEST 2012 on sn-devel-104
2012-08-06 14:47:26 +02:00
Andrew Bartlett
49b2720317 lib/param: Also enable vlp when --enable-selftest is specified 2012-08-06 20:51:55 +10:00
Andreas Schneider
41cffa3c8b doc: Remove build/ from doxygen config or it will not work in brew.
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Sat Aug  4 16:31:22 CEST 2012 on sn-devel-104
2012-08-04 16:31:22 +02:00
Gregor Beck
cff3ad4089 lib/dbwrap: rewrite lock order check to ease debugging
Signed-off-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Fri Aug  3 12:52:38 CEST 2012 on sn-devel-104
2012-08-03 12:52:38 +02:00
Andrew Bartlett
8208d7b2c2 lib/ldb: Ensure rename target does not exist before deleting old record
This is all in a transaction, but when we are handling rename errors in the repl_meta_data
module, we key off the error, and do not close the transaction.  We found that the
old record was gone and so could not try renaming it again to a conflict DN.

Andrew Bartlett
2012-07-31 14:15:29 +02:00
Volker Lendecke
c03e9769cd tevent: Refer to the correct header in testsuite.c
Signed-off-by: Stefan Metzmacher <metze@samba.org>
2012-07-31 11:40:22 +02:00
Andrew Bartlett
cddcaf7bd2 ldb: Add parameter to avoid NULL format string flagged by -Werror=format
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Jul 30 15:48:53 CEST 2012 on sn-devel-104
2012-07-30 15:48:53 +02:00
Andrew Bartlett
ea6b8ee026 lib/tdb: Fix format string errors found by -Werror=format in tdb tests 2012-07-30 14:25:10 +10:00
Andrew Bartlett
17bddd15de lib/ntdb: Fix format string errors found by -Werror=format in ntdb tests 2012-07-30 14:25:10 +10:00
Andrew Bartlett
5b4792d9d2 lib/ccan: Fix format string errors found by -Werror=format in ccan failtest 2012-07-30 14:25:09 +10:00
Andrew Bartlett
dc00df2add lib/param: Rename "socket address" to "nbt client socket address" to clarify role
This parameter is only used in our NBT client code and in nmbd as a
fallback when we fail to select a better interface from "interfaces"
to use directly.

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Jul 27 12:16:25 CEST 2012 on sn-devel-104
2012-07-27 12:16:25 +02:00
Andrew Bartlett
ec64f7907d lib/param: Mark lpcfg_cachedir and lpcfg_statedir as const char *
This matters as we sync up the declarations with the source3/param code.

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Jul 27 05:26:45 CEST 2012 on sn-devel-104
2012-07-27 05:26:45 +02:00
Andrew Bartlett
ca24820736 lib/param: Remove unused "paranoid server security"
This became unused when security=server went away.

Andrew Bartlett
2012-07-27 03:34:23 +02:00
Andrew Bartlett
ecb89cebd6 docs: Mark "socket address" as deprecated 2012-07-27 03:34:23 +02:00
Andrew Bartlett
42e405a422 param: Make socket_address common, Revert 611ef42053
This essentially reverts the commit 611ef42053
Author: Yasuma Takeda <yasuma@osstech.co.jp>
Date:   Fri Dec 5 13:37:51 2008 -0800

    Fix bug #5944 - nmbd does not boot if socket adress = "" is defined in smb.conf

Intead, the documentation is fixed so that the correct default is
recorded.  Removing the special case handling here allows this to be
dealt with in the same way as all other parameters.

Andrew Bartlett
2012-07-27 03:34:23 +02:00
Andrew Bartlett
bc1826abd6 lib/param: Merge parameter tables into a common file
This file is then included into both loadparm systems.

Andrew Bartlett

Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>
2012-07-27 03:34:23 +02:00
Andrew Bartlett
c4eeda91b0 lib/param: Merge in source3 parameters into parmeter table
This finishes the series to remove the various warnings from smb.conf
loading when a source3 parameter is used.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-27 03:34:22 +02:00
Andrew Bartlett
081a010f8c lib/param: Merge "Security Options" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-27 03:34:22 +02:00
Andrew Bartlett
8cb47eff7e lib/param: Merge "Logging Options" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-27 03:34:22 +02:00
Andrew Bartlett
08691f7eaf lib/param: Merge "Protocol Options" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-27 03:34:22 +02:00
Andrew Bartlett
97a6447c06 lib/param: Merge "Tuning Options" section from source3/param
This will reduced the weight of the eventual merge of all of source3/param.

Andrew Bartlett
2012-07-27 03:34:22 +02:00
Andrew Bartlett
c5713acf5a lib/param: Merge "Printing Options" section from source3/param
This will reduced the weight of the eventual merge of all of source3/param.

Andrew Bartlett
2012-07-27 03:34:22 +02:00
Björn Jacke
4f0b8f30c9 s3/aio_fork: fix build on irix
IRIX needs _XOPEN_SOURCE defined for SCM_RIGHTS to be available
2012-07-26 18:27:40 +02:00
Andrew Bartlett
115a88e18f lib/param: Merge "Filename Handling" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Jul 26 14:16:02 CEST 2012 on sn-devel-104
2012-07-26 14:16:01 +02:00
Andrew Bartlett
deb9a05a59 lib/param: Merge "Domain Options" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-26 12:23:20 +02:00
Andrew Bartlett
932f0a6af4 lib/param: Merge "Logon Options" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-26 12:23:20 +02:00
Andrew Bartlett
b727e706c3 lib/param: Merge "Browse Options" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-26 12:23:20 +02:00
Andrew Bartlett
3f1dfdd8b4 lib/param: Merge "WINS Options" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-26 12:23:20 +02:00
Andrew Bartlett
ebc2cc7f8a lib/param: Merge "Locking Options" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-26 12:23:20 +02:00
Andrew Bartlett
3e9c00112e lib/param: Merge "Ldap Options" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-26 12:23:20 +02:00
Andrew Bartlett
8d2ba95c90 lib/param: Merge "EventLog Options" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-26 12:23:20 +02:00
Andrew Bartlett
a0e08acbe0 lib/param: Merge "Miscellaneous Options" section from source3/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-26 12:23:20 +02:00
Andrew Bartlett
221a430e22 lib/param: Merge VFS and MSDFS parameters from source3 into lib/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Jul 24 12:53:17 CEST 2012 on sn-devel-104
2012-07-24 12:53:17 +02:00
Andrew Bartlett
c7df664448 lib/param: Merge Winbind parameters from source3 into lib/param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-24 11:01:18 +02:00
Andrew Bartlett
19a886724a lib/param: Merge DNS parameters with source3 param
This will make the merge of the whole table smoother.

Based on an earlier patch
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-24 11:01:18 +02:00
Andrew Bartlett
9ee084fd39 lib/param: Rearrange AD DC options to make a merge with the source3 table easier
Andrew Bartlett
2012-07-24 11:01:18 +02:00
Andrew Bartlett
222850bf80 lib/param: Re-arrange TLS parameters into their own section
This makes the merge with the source3 table less disruptive

Based on an earlier patch:
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Andrew Bartlett
2012-07-24 11:01:17 +02:00
Andrew Bartlett
c1065ff073 lib/param: Add handler overrides
These #defines allow us to merge the parameter table without providing the
handler functions quite yet.  This helps us do this task in stages.

Andrew Bartlett
2012-07-24 11:01:17 +02:00
Andrew Bartlett
3367fd01d4 lib/param: Merge handling of security/domain master/domain logons/server role
This ensures that the same input parameters always gives the same
output values in both loadparm systems.

Andrew Bartlett
2012-07-24 11:01:17 +02:00
Andrew Bartlett
b62b031815 lib/param: Make lp_usershare_max_shares() a common parameter
Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>
2012-07-24 11:01:17 +02:00
Andrew Bartlett
f3562424b6 lib/param: Move all enum declarations to lib/param
This is in preperation for the parameter table being made common.

Andrew Bartlett

Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>
2012-07-24 11:01:17 +02:00
Andrew Bartlett
280d1877fd lib/param: Rename param_enums.c to param_table.c
This is in preperation for the parameter table being merged into this file.

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Jul 24 09:16:44 CEST 2012 on sn-devel-104
2012-07-24 09:16:44 +02:00
Andrew Bartlett
b94ab3c608 lib/param: Remove 'case insensitive filesystem'
We use the slightly confusing 'case sensitive = yes' option for the same behaviour.

This avoids adding even more confusing documentation for the ntvfs-only option.

Andrew Bartlett
2012-07-24 10:46:12 +10:00
Andrew Bartlett
8b1a9f3ebd lib/param: Remove "ntptr providor" and hard-code in s4 spoolss server
This stub codebase does not justify a merged parameter.

Andrew Bartlett

Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>
2012-07-24 10:46:12 +10:00
Andrew Bartlett
80d3208830 param: Make internal handler for 'server role' common 2012-07-24 10:46:12 +10:00
Andrew Bartlett
9521cd0409 param: Make the 'unicode' parameter common
This parameter is most useful for interop testing, rather than production use.

Andrew Bartlett
2012-07-24 10:46:12 +10:00
Stefan Metzmacher
5adf63fe30 lib/crypto: add aes_ccm_128
metze
2012-07-23 18:19:36 +02:00
Jeremy Allison
3e476e184d Add debugs to functions. Add pidfile_unlink(). 2012-07-19 16:08:16 -07:00
Jeremy Allison
2922fdaaf0 Move source4/smbd/pidfile into lib/util in preparation for making it in common. 2012-07-19 15:41:52 -07:00
Andrew Bartlett
03a6137001 s3-param: Remove special case for lp_ctdbd_socket(), set CTDB_PATH as default
This changes the default based on the #ifdef rather than an override
on a parameter value of ""

The less special override functions we have the easier it is to merge
the loadparm tables.

Andrew Bartlett

Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Jul 19 09:56:01 CEST 2012 on sn-devel-104
2012-07-19 09:56:01 +02:00
Andrew Bartlett
0f57da57e3 lib/param: bring lp_smb_ports() into common by making it a list everywhere 2012-07-19 08:02:32 +02:00
Andrew Bartlett
cf9bd1d9ed s3-param: Make lp_name_resolve_order() return a list
This allows this parameter, one of the few with differing declarations
between the loadparm systems, to be brought into common.

Andrew Bartlett

Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>
2012-07-19 08:02:32 +02:00
Andrew Bartlett
8822b3b662 s4-param: Remove unused "idmap trusted only"
When we revamp the idmap layer, we will end up just following the s3
options, and this option is not used there either.

Andrew Bartlett

Pair-Programmed-With: Andrew Tridgell <tridge@samba.org>
2012-07-19 08:02:32 +02:00
Andrew Bartlett
d2ae8179ea lib/param: Add my copyright 2012-07-19 08:02:32 +02:00
Andrew Bartlett
1cbbd2d980 lib/param: bring lp_time_server() into common 2012-07-19 08:02:32 +02:00
Rusty Russell
1f8b574adb talloc_stack: abort in developer me if no stackframe on talloc_tos()
Don't tolerate leaks in developer mode.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-07-18 15:07:27 +09:30
Rusty Russell
fe72740e82 loadparm: make the source3/ lp_ functions take an explicit TALLOC_CTX *.
They use talloc_tos() internally: hoist that up to the callers, some
of whom don't want to us talloc_tos().

A simple patch, but hits a lot of files.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-07-18 15:07:23 +09:30
Rusty Russell
a620fc0372 lib/util/modules.c: fix stackframe leak.
do_smb_load_module() doesn't free its stackframe on success.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-07-18 05:02:31 +09:30
Rusty Russell
84fb37fe37 talloc_stack: report lazy freeing (panic if DEVELOPER).
talloc_stackframe() stacks, so if you forget to free one, the outer
one will free it.  However, it's not a good idea to rely too heavily
on this behaviour: it can lead to delays in the release of memory or
destructors.

I had an elaborate hack to make sure every talloc_stackframe() was
freed in the exact same function it was allocated, however all bugs it
caught were simply lazy freeing, so this patch just checks for that.

This doesn't check for stackframes we don't free up on exit: that would
be nice, but uncovers some uncomfortable (but probably harmless) cases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-07-18 04:57:31 +09:30
Rusty Russell
f9b51ff33e talloc_stack: always include the location when creating a talloc_stackframe().
Much better for debugging.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-07-18 04:56:31 +09:30
Rusty Russell
311281c2c5 talloc_stack: handle more than one talloc_stackframe_pool()
The only reason we make one stackframe parent of the next is so we use
our parent's pool.  That doesn't make sense if we're a new pool, and
wouldn't work anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-07-18 04:55:31 +09:30
Rusty Russell
4f331872bc talloc: don't allow a talloc_pool inside a talloc_pool.
We explicitly call free() on a pool which falls to zero, assuming it's
not inside another pool (we crash).  Check on creation and explicitly
document this case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-07-18 04:54:31 +09:30
Rusty Russell
8893215aaf talloc: use a struct for pool headers.
This neatens the code a bit (we should do a similar thing for all the
TALLOC_CHUNK macros).

Two subtler changes:
(1) As a result of the struct, we actually pack object_count into the 
    talloc header on 32-bit platforms (since the header is 40 bytes, but
    needs to be 16-byte aligned).
(2) I avoid VALGRIND_MAKE_MEM_UNDEFINED on memmove when we resize the
    only entry in a pool; that's done later anyway.

With -O2 on my 11.04 Ubuntu 32-bit x86 laptop, the talloc_pool speed as
measured by testsuite.c actually increases 10%.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-07-18 04:53:31 +09:30
Andrew Bartlett
c92a5670e3 pytdb: Check if the database is closed before we touch it
If .close() has already been called, we have to play dead - the
self->ctx is just not valid any more, as we have been shut down to
allow some other part of Samba to open the tdb.

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Jul 16 13:51:52 CEST 2012 on sn-devel-104
2012-07-16 13:51:51 +02:00
Andrew Bartlett
a8e88332a3 pytdb: Check for errors parsing strings into TDB_DATA
The call to PyStringAsString() can raise an exception, and we
want to return that rather than following a NULL pointer later.

Andrew Bartlett
2012-07-16 11:58:21 +02:00
Andrew Bartlett
1013fab5f8 lib/util: Allocate enough space to reference blob->data[len]
Found by Thomas Hood <jdthood@gmail.com> using valgrind.

Thanks!

Andrew Bartlett
2012-07-13 08:12:17 +02:00
Andrew Bartlett
7b1fb36ad0 lib/ldb: Bump ldb release due to pyldb changes
We strictly need these changes to pass make test, and the concat change is
backwards incompatible, so we really want to use the right version.

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Jul  9 04:34:06 CEST 2012 on sn-devel-104
2012-07-09 04:34:06 +02:00
Andrew Bartlett
979215ad59 pyldb: Add bindings for ldb_dn_remove_base_components 2012-07-06 22:55:49 +10:00
Andrew Bartlett
8d99b398d9 pyldb: Fix dn concat operation to be the other way around
This now concatonates Dn(ldb, "cn=config") + Dn(ldb, "dc=samba,dc=org") as "cn=config,dc=samba,dc=org"

Andrew Bartlett
2012-07-06 22:45:33 +10:00
Andrew Bartlett
2908bbe06a s4-selftest: Test samba-tool domain dcpromo
This needs a new environment to test it properly.  This requires a raise in the
number of socket wrapper interfaces.

Andrew Bartlett
2012-07-06 08:10:18 +02:00
Björn Jacke
7abe51f840 talloc: remove unused variables
found by the IRIX compiler

Autobuild-User(master): Björn Jacke <bj@sernet.de>
Autobuild-Date(master): Thu Jul  5 23:50:54 CEST 2012 on sn-devel-104
2012-07-05 23:50:54 +02:00
Björn Jacke
a79496d90c lib/addns: remove defines we don't need or have for sure via libreplace 2012-07-05 22:00:52 +02:00
Björn Jacke
4b83d61e7d lib/addns: remove use of uint8 uint16 and uint32 in favour of C99 types 2012-07-05 22:00:52 +02:00
Björn Jacke
fa98ef112b replace: make the INT64_MAX define more portable 2012-07-04 12:01:20 +02:00
Rusty Russell
74bf0c6ec6 ntdb: make --disable-ntdb work properly.
As per bug #9024, make --disable-ntdb work again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date(master): Wed Jul  4 08:11:33 CEST 2012 on sn-devel-104
2012-07-04 08:11:33 +02:00
Andrew Bartlett
a8acaeebb6 ldb: bump version due to header and internal implementation changes
We need this version, not the previous release, for Samba.

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Jul  3 17:20:32 CEST 2012 on sn-devel-104
2012-07-03 17:20:32 +02:00
Jeremy Allison
ec9aae6251 Ensure we select the correct syscall numbers on a 32-bit Linux system. 2012-07-02 18:38:14 -07:00
Rusty Russell
8150f69dc4 ccan: make it a grouping library.
Andrew Bartlett pointed out that making CCAN a non-library will break
the build in a different way in future: when two separate private
libraries start using the same CCAN module, the symbol duplicate
detection will fire (since private libaries don't use any symbol
hiding).  That doesn't happen yet, but it will surely happen
eventually.

So, for now at least, we build as a private library again.  This
unfortunately means the top-level build creates a libccan.so, which
contains all the ccan modules whether you need them or not.  Given the
size of the library, I don't think this is a win.  But it's simple.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date(master): Sat Jun 30 11:19:04 CEST 2012 on sn-devel-104
2012-06-30 11:19:03 +02:00
Jeremy Allison
f81fd1ccc3 Try and fix the autoconf build on Solaris/Nexenta/etc.
Their AC_TRY_RUN doesn't include any current CPPFLAGS. Make
the set[res]uid checks independent of this. Needs a small
change to the waf build in order to code with the change.

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Jun 30 00:32:36 CEST 2012 on sn-devel-104
2012-06-30 00:32:36 +02:00
Stefan Metzmacher
19941a9054 lib/ccan: readd execinfo dependency to failtest
metze

Signed-off-by: Michael Adam <obnox@samba.org>
2012-06-29 15:21:15 +02:00
Stefan Metzmacher
3c3ed70690 lib/util: fix fd leak in anonymous_shared_allocate() if MAP_ANON is not available
metze
2012-06-29 15:21:10 +02:00
Rusty Russell
f3862b9171 ccan: we're subsystems, not a library.
Don't expose a libccan.so; it would produce clashes if someone else
does the same thing.  Unfortunately, if we just change it from a
SAMBA_LIBRARY to a SAMBA_SUBSYSTEM, it doesn't create a static library
as we'd like, but links all the object files in.  This means we get
many duplicates (eg. everyone gets a copy of tally, even though only
ntdb wants it).

So, the solution is twofold:
1) Make the ccan modules separate.
2) Make the ccan modules SAMBA_SUBSYSTEMs not SAMBA_LIBRARYs so we don't
   build shared libraries which we can't share.
3) Make the places which uses ccan explicit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date(master): Fri Jun 29 06:22:44 CEST 2012 on sn-devel-104
2012-06-29 06:22:44 +02:00
Jeremy Allison
485787f0df Move back to using per-thread credentials on Linux. Fixes the glibc native AIO lost wakeup problem.
See this post:

https://lists.samba.org/archive/samba-technical/2012-June/085101.html

for details.

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Jun 29 03:57:45 CEST 2012 on sn-devel-104
2012-06-29 03:57:45 +02:00
Jeremy Allison
821bd95156 Replace all uses of setXX[ug]id() and setgroups with samba_setXX[ug]id() calls.
Will allow thread-specific credentials to be added by modifying
the central definitions. Deliberately left the setXX[ug]id()
call in popt as this is not used in Samba.
2012-06-28 17:15:16 -07:00
Jeremy Allison
7630fe50bd Add missing setresgid() wrapper. 2012-06-28 17:15:16 -07:00
Stefan Metzmacher
a5b62a60cc lib/param: let "SMB2" select PROTOCOL_SMB2_10
metze
2012-06-29 00:14:57 +02:00
Björn Jacke
0a1aaca1bd replace: define INT64_MAX when not defined
Tru64 doesn't have any stdint.h

Autobuild-User(master): Björn Jacke <bj@sernet.de>
Autobuild-Date(master): Thu Jun 28 00:45:58 CEST 2012 on sn-devel-104
2012-06-28 00:45:58 +02:00
Andrew Bartlett
666dba3353 s3-param: Rename loadparm_s3_context -> loadparm_s3_helpers
This helps clarify the role of this structure and wrapper function.

The purpose here is to provide helper functions to the lib/param
loadparm_context that point back at the s3 lp_ functions.  This allows
a struct loadparm_context to be passed to any point in the code, and
always refer to the correct loadparm system.  If this has not been
set, the variables loaded in the lib/param code will be returned.

As requested by Michael Adam.

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Jun 27 17:11:16 CEST 2012 on sn-devel-104
2012-06-27 17:11:16 +02:00
Andrew Bartlett
46db466372 lib/ldb: Print trace messages for modify correctly 2012-06-27 11:29:17 +02:00
Rusty Russell
66f59f0409 tdb: finish weaning off err.h.
Commit 3c4263e758 said it removed err.h
from tdb, unfortuntely it didn't: tap-interface.h still included it.

This finishes it properly!

Reported-by:Stefan Metzmacher <metze@samba.org>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date(master): Tue Jun 26 10:22:03 CEST 2012 on sn-devel-104
2012-06-26 10:22:03 +02:00
Stefan Metzmacher
97981491eb lib/param: add "SMB3" as alias for PROTOCOL_SMB3_00
metze
2012-06-25 20:55:06 +02:00
Matthieu Patou
1c850b2f17 ldb: lay foundation for proper utc/generalized time handling
We use to handle UTCtime and generalized time the same way. The thing is
that it's not the case, they are different in the way they are set (most
of the time) with different format and also stored and return in
different format too.
2012-06-22 23:22:03 -07:00
Matthieu Patou
9ebb081cce ldb: add the VERIFY_NAME control 2012-06-22 23:22:02 -07:00
Stefan Metzmacher
4676e7f65d dbwrap: don't ignore the result of dbwrap_parse_record in dbwrap_fetch_int32()
metze

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Fri Jun 22 17:10:52 CEST 2012 on sn-devel-104
2012-06-22 17:10:52 +02:00
Stefan Metzmacher
0f25bd78ca dbwrap: intialize state.status in dbwrap_fetch_int32()
This might not be needed, but it makes it more clear that
we won't use uninitialized memory, it the callback was not triggered.

metze
2012-06-22 15:16:53 +02:00
Volker Lendecke
d7b9e0d19a dbwrap: Convert fetch_int32 to dbwrap_parse_record
Now dbwrap_fetch_int32 is used in smbd/locking/posix.c is used a
lot more often than before.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
2012-06-22 15:16:52 +02:00
Volker Lendecke
e0abd1938a dbwrap: Add dbwrap_fetch_int32
Signed-off-by: Stefan Metzmacher <metze@samba.org>
2012-06-22 15:16:47 +02:00
Volker Lendecke
ce7b0b9906 dbwrap: Add dbwrap_change_int32_atomic
Signed-off-by: Stefan Metzmacher <metze@samba.org>
2012-06-22 15:16:45 +02:00
Rusty Russell
3c4263e758 tdb: don't use err.h in tests.
It's not portable.  While we could use ccan/err, it seems overkill since
we actually only use it in one test (I obviously cut & paste the #include).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date(master): Fri Jun 22 09:22:28 CEST 2012 on sn-devel-104
2012-06-22 09:22:28 +02:00
Rusty Russell
1783fe3443 tdb: make TDB_NOSYNC merely disable sync.
(As suggested by Stefan Metzmacher, based on the change to ntdb.)

Since commit ec96ea690e, we handle the case
where a process dies during a transaction commit.  Unfortunately, TDB_NOSYNC
means this no longer works, as it disables the recovery area as well as the
actual msync/fsync.  We should do everything except the syncs.

This also means we can do a complete test with $TDB_NO_FSYNC set; just
to get more complete coverage, we disable it explicitly for one test
(where we override the actual sync calls anyway).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
945473aac0 dbwrap: dbwrap_hash_size().
Implemented for ntdb and tdb; falls back to 0 for others.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
e92cb556fd dbwrap: dbwrap_name().
Useful for debug messages: particularly once we start switching between .tdb
and .ntdb files.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
41f799d877 dbwrap: dbwrap_transaction_start_nonblock().
Implemented for ntdb and tdb; falls back to the blocking variant
for others.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
f6eb187fda dbwrap: dbwrap_fetch_locked_timeout().
Implemented for ntdb and tdb; falls back to the non-timeout variant
for others.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
431667b47c dbwrap: add dbwrap_check() function.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
9d97bf3f47 dbwrap: dbwrap_local_open()
This simply opens a tdb: it will eventually switch depending on the
extension.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
1acf548eb7 dbwrap: remove get_flags().
The flags returned were TDB-specific: this was only used for detecting
the endianness of obsolete databases (the conversion code was put in in
2003, with reference to Samba 2.3).

It's easier to remove it than to translate the NTDB flags to TDB flags,
and it's a really weird thing to ask for anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
fc9b298761 util_tdb: move timeout chainlock variants from source3/lib/util/util_tdb.c
We're about to use them for dbwrap.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
02bacf1f95 util: util_ntdb ntdb_fetch_int32/ntdb_store_int32 and ntdb_add_int32_atomic
Similar to the util_tdb versions, but return the error code.

ntdb_add_int32_atomic seems a clearer name than tdb_change_int32_atomic.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
348159d1e4 util: util_ntdb.c gets NTDB_ERROR => NTSTATUS map.
Very similar to the tdb version.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
88ad365caf util: util_ntdb.c gains bystring functions.
Very similar to the util_tdb versions, but these return the error.
I've only implemented those functions actually used.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
8113d53bb4 util: ntdb_new() supports NTDB_CLEAR_IF_FIRST.
There are various issues with NTDB_CLEAR_IF_FIRST which makes it
better if we don't have to use it, but much of the code does, so
we fake up support here.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
735290f474 util: util_ntdb.c
The first function is ntdb_new: this is preferred over ntdb_open, as
it makes the ntdb_context returned (and all NTDB_DATA returned from
ntdb_fetch) valid talloc pointers.

The API is very similar to tdb_wrap_open().

Note that we handle $TDB_NO_FSYNC here, since ntdb doesn't do that
hack (and it's great for speeding up testing!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
7c1d9fb3c1 ntdb: take advantage of direct access across expand.
This means we no longer have to unmap if we want to compare a record.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
4c51ee1116 ntdb: test arbitrary operations during ntdb_parse_record().
In particular, this tests that we can store enough records to make the
database expand while we map the given record.  We use a global lock for
this, but it could happen in theory with another process.

It also tests the that we can recurse inside ntdb_parse_record().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
01ec4a72de ntdb: make database read-only during ntdb_parse() callback.
Since we have a readlock, any write will grab a write lock: if it happens
to be on the same bucket, we'll fail.

For that reason, enforce read-only so every write operation fails
(even for NTDB_NOLOCK or NTDB_INTERNAL dbs), and document it!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
bd5c061932 ntdb: allow direct access for NTDB_INTERNAL dbs during expansion.
NTDB_INTERNAL databases need to malloc and copy to keep old versions
around if we expand, in a similar way to the manner in which keep old
mmaps around.

Of course, it only works for read-only accesses, since the two copies
are not synced.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
0a34f342c3 ntdb: enhancement to allow direct access to the ntdb map during expansion.
This means keeping the old mmap around when we expand the database.
We could revert to read/write, except for platforms with incoherent
mmap (ie. OpenBSD), where we need to use mmap for all accesses.

Thus we keep a linked list of old maps, and unmap them when the last access
finally goes away.

This is required if we want ntdb_parse_record() callbacks to be able
to expand the database.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
66d151d689 ntdb: don't munmap the database on every close.
Since we can have multiple openers, we should leave the mmap in place
for the other openers to use.  Enhance the test to check the bug (it
still works, because without mmap we fall back to read/write, but
performance would be terrible!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:17 +02:00
Rusty Russell
406bd2d121 ntdb: hand correct error code when alloc_read allocation fails.
-ECUTNPASTE.  This is not a usage error!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:16 +02:00
Rusty Russell
f7f6992c1e autobuild: always set TDB_NO_FSYNC.
Then we unset it inside the tdb test target itself.  This means that
new code can't accidently forget it, and we can set it in the
'buildnice' script on sn-devel, for example.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:16 +02:00
Rusty Russell
0265837ee8 ntdb: respect TDB_NO_FSYNC flag for 'make test'
This reduces test time from 31 seconds to 6, on my laptop.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-22 07:35:16 +02:00
Jelmer Vernooij
7b43053ad6 Add --disable-ntdb option for building.
Autobuild-User(master): Jelmer Vernooij <jelmer@samba.org>
Autobuild-Date(master): Thu Jun 21 19:59:57 CEST 2012 on sn-devel-104
2012-06-21 19:59:57 +02:00
Rusty Russell
5ec4305510 ntdb: fix occasional abort in testing.
Occasionally, the capability test inserts multiple used records and they
clash, but our primitive test layout engine doesn't handle hash clashes
and aborts.

Force a seed value which we know doesn't clash.

Reported-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date(master): Wed Jun 20 16:50:20 CEST 2012 on sn-devel-104
2012-06-20 16:50:20 +02:00
Rusty Russell
7e80b89d48 ntdb: add autoconf support.
This is copied from tdb; we build the utilities, but as nothing else
links against it, we shouldn't be adding anything to the normal samba
binary sizes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date(master): Tue Jun 19 07:31:06 CEST 2012 on sn-devel-104
2012-06-19 07:31:06 +02:00
Rusty Russell
316e5e376c lib/tdb_wrap: use tdb directly, not tdb_compat.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:07 +02:00
Rusty Russell
df4a6e8228 ldb: use tdb directly, not tdb_compat.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:07 +02:00
Rusty Russell
6dc02e832a lib/dbwrap: depend directly on tdb, not tdb_compat.
Simple change, as we get rid of tdb_compat in favour of either ntdb directly
or dbwrap.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:07 +02:00
Rusty Russell
2fc3265873 lib/util_tdb: depend directly on tdb, not tdb_compat.
Simple change, as we get rid of tdb_compat in favour of tdb directly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:07 +02:00
Rusty Russell
5ff92d8f7d ntdb: update documentation.
Update the design.lyx file with the latest status and the change in hashing.
Also, refresh and add examples to the TDB_porting.txt file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:07 +02:00
Rusty Russell
b888bc4316 ntdb: optimize ntdb_fetch.
We access the key on lookup, then access the data in the caller.  It
makes more sense to access both at once.  We also put in a likely()
for the case where the hash is not chained.

Before:
Adding 1000 records: 3644-3724(3675) ns (129656 bytes)
Finding 1000 records: 1596-1696(1622) ns (129656 bytes)
Missing 1000 records: 1409-1525(1452) ns (129656 bytes)
Traversing 1000 records: 1636-1747(1668) ns (129656 bytes)
Deleting 1000 records: 3138-3223(3175) ns (129656 bytes)
Re-adding 1000 records: 3278-3414(3329) ns (129656 bytes)
Appending 1000 records: 5396-5529(5426) ns (253312 bytes)
Churning 1000 records: 9451-10095(9584) ns (253312 bytes)
smbtorture results (--entries=1000)
ntdb speed 183881-191112(188223) ops/sec

After:
Adding 1000 records: 3590-3701(3640) ns (129656 bytes)
Finding 1000 records: 1539-1605(1566) ns (129656 bytes)
Missing 1000 records: 1398-1440(1413) ns (129656 bytes)
Traversing 1000 records: 1629-2015(1710) ns (129656 bytes)
Deleting 1000 records: 3118-3236(3163) ns (129656 bytes)
Re-adding 1000 records: 3235-3355(3275) ns (129656 bytes)
Appending 1000 records: 5335-5444(5385) ns (253312 bytes)
Churning 1000 records: 9350-9955(9494) ns (253312 bytes)
smbtorture results (--entries=1000)
ntdb speed 180559-199981(195106) ops/sec
2012-06-19 05:38:07 +02:00
Rusty Russell
8fdd20b22f ntdb: add -h arg to ntdbrestore
Since our default hashsize is 8192 not 131, we look fat when we convert
near-empty TDBs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:07 +02:00
Rusty Russell
a941b19e5d ntdb: reduce default hashsize on ntdbtorture.
Just like tdbtorture, having a hashsize of 2 stresses us much more!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:07 +02:00
Rusty Russell
87f871aae3 ntdb: add NTDB_ATTRIBUTE_HASHSIZE
Since we've given up on expansion, let them frob the hashsize again.
We have attributes, so we should use them for optional stuff like
this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:07 +02:00
Rusty Russell
dd42962878 ntdb: remove hash table trees.
TDB2 started with a top-level hash of 1024 entries, divided into 128
groups of 8 buckets.  When a bucket filled, the 8 bucket group
expanded into pointers into 8 new 64-entry hash tables.  When these
filled, they expanded in turn, etc.

It's a nice idea to automatically expand the hash tables, but it
doesn't pay off.  Remove it for NTDB.

1) It only beats TDB performance when the database is huge and the
   TDB hashsize is small.  We are about 20% slower on medium-size
   databases (1000 to 10000 records), worse on really small ones.
2) Since we're 64 bits, our hash tables are already twice as expensive
   as TDB.
3) Since our hash function is good, it means that all groups tend to
   fill at the same time, meaning the hash enlarges by a factor of 128
   all at once, leading to a very large database at that point.
4) Our efficiency would improve if we enlarged the top level, but
   that makes our minimum db size even worse: it's already over 8k,
   and jumps to 1M after about 1000 entries!
5) Making the sub group size larger gives a shallower tree, which
   performs better, but makes the "hash explosion" problem worse.
6) The code is complicated, having to handle delete and reshuffling
   groups of hash buckets, and expansion of buckets.
7) We have to handle the case where all the records somehow end up with
   the same hash value, which requires special code to chain records for
   that case.

On the other hand, it would be nice if we didn't degrade as badly as
TDB does when the hash chains get long.

This patch removes the hash-growing code, but instead of chaining like
TDB does when a bucket fills, we point the bucket to an array of
record pointers.  Since each on-disk NTDB pointer contains some hash
bits from the record (we steal the upper 8 bits of the offset), 99.5%
of the time we don't need to load the record to determine if it
matches.  This makes an array of offsets much more cache-friendly than
a linked list.

Here are the times (in ns) for tdb_store of N records, tdb_store of N
records the second time, and a fetch of all N records.  I've also
included the final database size and the smbtorture local.[n]tdb_speed
results.

Benchmark details:
1) Compiled with -O2.
2) assert() was disabled in TDB2 and NTDB.
3) The "optimize fetch" patch was applied to NTDB.

10 runs, using tmpfs (otherwise massive swapping as db hits ~30M,
despite plenty of RAM).

				Insert	Re-ins	Fetch	Size	dbspeed
				(nsec)	(nsec)	(nsec)	(Kb)	(ops/sec)
TDB (10000 hashsize):	
	100 records:		 3882	 3320	1609	   53	203204
	1000 records:		 3651	 3281	1571	  115	218021
	10000 records:		 3404	 3326	1595	  880	202874
	100000 records:		 4317	 3825	2097	 8262	126811
	1000000 records:	11568	11578	9320	77005	 25046

TDB2 (1024 hashsize, expandable):
	100 records:		 3867	 3329	1699	   17	187100	
	1000 records:		 4040	 3249	1639	  154	186255
	10000 records:		 4143	 3300	1695	 1226	185110
	100000 records:		 4481	 3425	1800	17848	163483
	1000000 records:	 4055	 3534	1878   106386	160774

NTDB (8192 hashsize)
	100 records:		 4259	 3376	1692	   82	190852
	1000 records:		 3640	 3275	1566	  130	195106
	10000 records:		 4337	 3438	1614	  773	188362
	100000 records:		 4750	 5165	1746	 9001	169197
	1000000 records:	 4897	 5180	2341	83838	121901

Analysis:
	1) TDB wins on small databases, beating TDB2 by ~15%, NTDB by ~10%.
	2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
	   than TDB2/NTDB).
	3) TDB does horribly when hash chains get 100 long (fetch 4x slower
	   than NTDB, 5x slower than TDB2, insert about 2-3x slower).
	4) TDB2 databases are 40% larger than TDB1.  NTDB is about 15% larger
	   than TDB1
2012-06-19 05:38:07 +02:00
Rusty Russell
f986554b1e ntdb: special accessor functions for read/write of an offset.
We also split off the NTDB_CONVERT case (where the ntdb is of a
different endian) into its own io function.

NTDB speed:
Adding 10000 records: 3894-9951(8553) ns (815528 bytes)
Finding 10000 records: 1644-4294(3580) ns (815528 bytes)
Missing 10000 records: 1497-4018(3303) ns (815528 bytes)
Traversing 10000 records: 1585-4225(3505) ns (815528 bytes)
Deleting 10000 records: 3088-8154(6927) ns (815528 bytes)
Re-adding 10000 records: 3192-8308(7089) ns (815528 bytes)
Appending 10000 records: 5187-13307(11365) ns (1274312 bytes)
Churning 10000 records: 6772-17567(15078) ns (1274312 bytes)
NTDB speed in transaction:
Adding 10000 records: 1602-2404(2214) ns (815528 bytes)
Finding 10000 records: 456-871(778) ns (815528 bytes)
Missing 10000 records: 393-522(503) ns (815528 bytes)
Traversing 10000 records: 729-1015(945) ns (815528 bytes)
Deleting 10000 records: 1065-1476(1374) ns (815528 bytes)
Re-adding 10000 records: 1397-1930(1819) ns (815528 bytes)
Appending 10000 records: 2927-3351(3184) ns (1274312 bytes)
Churning 10000 records: 3921-4697(4378) ns (1274312 bytes)
smbtorture results:
ntdb speed 86581-191518(175666) ops/sec
Applying patch..increase-top-level.patch
2012-06-19 05:38:07 +02:00
Rusty Russell
9133a98c44 ntdb: inline oob check
The simple "is it in range" check can be inline; complex cases can be
handed through to the normal or transaction handler.

NTDB speed:
Adding 10000 records: 4111-9983(9149) ns (815528 bytes)
Finding 10000 records: 1667-4464(3810) ns (815528 bytes)
Missing 10000 records: 1511-3992(3546) ns (815528 bytes)
Traversing 10000 records: 1698-4254(3724) ns (815528 bytes)
Deleting 10000 records: 3608-7998(7358) ns (815528 bytes)
Re-adding 10000 records: 3259-8504(7805) ns (815528 bytes)
Appending 10000 records: 5393-13579(12356) ns (1274312 bytes)
Churning 10000 records: 6966-17813(16136) ns (1274312 bytes)
NTDB speed in transaction:
Adding 10000 records: 916-2230(2004) ns (815528 bytes)
Finding 10000 records: 330-866(770) ns (815528 bytes)
Missing 10000 records: 196-520(471) ns (815528 bytes)
Traversing 10000 records: 356-879(800) ns (815528 bytes)
Deleting 10000 records: 505-1267(1108) ns (815528 bytes)
Re-adding 10000 records: 658-1681(1477) ns (815528 bytes)
Appending 10000 records: 1088-2827(2498) ns (1274312 bytes)
Churning 10000 records: 1636-4267(3785) ns (1274312 bytes)
smbtorture results:
ntdb speed 85588-189430(157110) ops/sec
2012-06-19 05:38:07 +02:00
Rusty Russell
d938c0b591 ntdb: allocator attribute.
This is designed to allow us to make ntdb_context (and NTDB_DATA returned
from ntdb_fetch) a talloc pointer.  But it can also be used for any other
alternate allocator.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:07 +02:00
Rusty Russell
6d5a3e1602 ntdb: still prepare recovery area with NTDB_NOSYNC.
NTDB_NOSYNC now just prevents the fsync/msync calls, which speeds
testing while still providing full coverage.  It also provides safety
against processes dying during transaction commit (though obviously,
not against the machine dying).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
89b0d5ac6c ntdb: simply disallow NULL names.
TDB allows this for internal databases, but it's a bad idea, since the
name is useful for logging.

They're a hassle to deal with, and we'd just end up putting "unnamed"
in there, so let the user deal with it.  If they don't, they get an
informative core dump.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
7fae6c44e2 ntdb: reduce transaction pagesize from 64k to 16k.
The performance numbers for transaction pagesize are indeterminate:
larger pagesizes means a smaller transaction array, and a better
chance of having a contiguous record (more efficient for
ntdb_parse_record and some internal operations inside a transaction).

On the other hand, large pagesize means more I/O even if we change a
few bytes.

But it also controls the multiple by which we will enlarge the file,
and hence the minimum db size.  It's 4k for tdb1, but 16k seems
reasonable in these modern times.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
35381cad1f ntdb: remove last block transactoin logic.
Now our database is always a multiple of NTDB_PGSIZE, we can remove the
special handling for the last block.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
db2508840d ntdb: create initial database to be multiple of NTDB_PGSIZE.
As copied from tdb1, there is logic in the transaction code to handle
a non-PGSIZE multiple db, but in fact this only happens for a
completely unused database: as soon as we add anything to it, it is
expanded to a NTDB_PGSIZE multiple.

If we create the database with a free record which pads it out to
NTDB_PGSIZE, we can remove this last-page-is-different logic.

Of course, the fake ntdbs we create in our tests now also need to be
multiples of NTDB_PGSIZE, so we change some numbers there too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
9396757676 ntdb: make sure file is always a multiple of PAGESIZE (now NTDB_PGSIZE)
ntdb uses tdb's transaction code, and it has an undocumented but implicit
assumption: that the transaction recovery area is always aligned to the
transaction pagesize.  This means that no block will overlap the recovery
area.

This is maintained by rounding the size of the database up, so do the same
for ntdb.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
dd4eed4759 ntdb: fix recovery data write.
We were missing the last few bytes.  Found by 100 runs of ntdbtorture
-t -k.

The transaction test code didn't catch this, because usually those
last few bytes are irrelevant to the actual contents of the database.
We fix the test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
40cf08823d ntdb: enhance external-helper test code.
Our external test helper is a bit primitive when it comes to doing STORE or
FETCH commands: let us specify the data we expect, instead of assuming it's
the same as the key.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
3bccb610c1 ntdb: use NTDB_LOG_WARNING level for failed open() without O_CREAT.
This is a fairly common pattern in Samba, and if we log an error on
every open it spams the logs.  On the other hand, other errors are
potentially more serious, so we still use NTDB_LOG_ERROR on them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
c7273629a2 ccan: remove bogus debug print.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
8a7c535db1 ntdb: make fork test more thorough.
We document that the child of a fork() can do a brunlock() if the parent
does a brlock: we should not log an error when they do this.

Also, test the case where we fork() and return inside a parse function
(which is allowed).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
d48f6f884b ntdb: print \n at end of log messages in tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
5027f9cd02 ntdb: reduce race between creating file and getting open lock.
In tdb, we grab the open lock immediately after we open the file.  In
ntdb, we usually did some work first.  tdbtorture managed to get in
before the creator grabbed the lock:

	testing with 3 processes, 5000 loops, seed=1338246020
	ntdb:torture.ntdb:IO Error:ntdb_open: torture.ntdb is not a ntdb file
	29023:torture.ntdb:db open failed

At cost of a little duplicated code, we can reduce the race.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
1765c0f9ba ntdb: catch any valgrind errors in test
Make --valgrind and --valgrind-log options work!

Amitay figured this out!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
fc9b8ee790 ntdb: catch any valgrind errors in test
We need --error-exitcode=, otherwise valgrind errors don't cause the
test to fail.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
f5e9ed1ea9 ntdb: remove ntdb_error()
It was a hack to make compatibility easier.  Since we're not doing that,
it can go away: all callers must use the return value now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Rusty Russell
16cc345d4f TDB2: Goodbye TDB2, Hello NTDB.
This renames everything from tdb2 to ntdb: importantly, we no longer
use the tdb_ namespace, so you can link against both ntdb and tdb if
you want to.

This also enables building of standalone ntdb by the autobuild script.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:06 +02:00
Kirill Smelkov
76758b9767 tdb2: Fix typo in TDB1_porting.txt
Judging by code it's tdb1, where you needed to free old key's dptr
manually.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:05 +02:00
Rusty Russell
c3dcdf08f3 TDB2: more internal cleanups after TDB1 compatibility removal.
This eliminates the separate tdb2 substructure, and makes some
tdb1-required functions static.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:05 +02:00
Rusty Russell
cab6e11678 TDB2: remove TDB1 compatibility.
This rips out all the TDB1 compatibility from tdb2.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:05 +02:00
Rusty Russell
6244f668a3 TDB2: make SAMBA use tdb1 again for the moment.
Otherwise the following surgery will break the SAMBA build and testsuite.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:05 +02:00
Rusty Russell
5bad913938 ccan: check for err.h ourselves
Heimdal does this, but that doesn't help the autoconf build or the standalone
libntdb build.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-06-19 05:38:05 +02:00
Jelmer Vernooij
bf5934ca1b tdb/wscript: Remove unecessary semicolons. 2012-06-19 02:43:23 +02:00
Stefan Metzmacher
da76cda93f lib/param: add missing prototype of lpcfg_parm_long()
metze
2012-06-18 15:26:42 +02:00
Jelmer Vernooij
8283d9ec4a attr: Look for attr/attributes.h too.
Fixes finding of ATTR_ROOT on GNU/kFreeBSD.

Autobuild-User(master): Jelmer Vernooij <jelmer@samba.org>
Autobuild-Date(master): Sat Jun 16 18:54:27 CEST 2012 on sn-devel-104
2012-06-16 18:54:26 +02:00
Andrew Bartlett
39766b75a4 s4-lib/param: FLAG DAY for the default FILE SERVER
This commit changes the default file server to be s3fs.  Existing
installs wishing to keep the ntvfs file server need to set this in
their smb.conf:

server services = +smb -s3fs
dcerpc endpoint services = +winreg +srvsvc

Andrew Bartlett
2012-06-16 08:18:10 +02:00
Volker Lendecke
442cb66c16 dbwrap: Remove an unnecessary ZERO_STRUCT
We assign the only struct member one line down

Signed-off-by: Stefan Metzmacher <metze@samba.org>
2012-06-15 18:32:21 +02:00
Volker Lendecke
1d9ff7d3bc dbwrap: dbwrap_trans_store_uint32->dbwrap_trans_store_uint32_bystring
Signed-off-by: Michael Adam <obnox@samba.org>

Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Fri Jun 15 14:20:04 CEST 2012 on sn-devel-104
2012-06-15 14:19:57 +02:00
Volker Lendecke
64fec465c1 dbwrap: dbwrap_trans_store_int32->dbwrap_trans_store_int32_bystring
Signed-off-by: Michael Adam <obnox@samba.org>
2012-06-15 12:14:29 +02:00
Volker Lendecke
07d6c25525 dbwrap: dbwrap_trans_change_int32_atomic->dbwrap_trans_change_int32_atomic_bystring
Signed-off-by: Michael Adam <obnox@samba.org>
2012-06-15 12:14:28 +02:00
Volker Lendecke
8b99d40520 dbwrap: dbwrap_change_int32_atomic->dbwrap_change_int32_atomic_bystring
Signed-off-by: Michael Adam <obnox@samba.org>
2012-06-15 12:14:28 +02:00
Volker Lendecke
9275d571d9 dbwrap: dbwrap_trans_change_uint32_atomic->dbwrap_trans_change_uint32_atomic_bystring
Signed-off-by: Michael Adam <obnox@samba.org>
2012-06-15 12:14:28 +02:00
Volker Lendecke
37ad03f91d dbwrap: dbwrap_change_uint32_atomic->dbwrap_change_uint32_atomic_bystring
Signed-off-by: Michael Adam <obnox@samba.org>
2012-06-15 12:14:28 +02:00
Volker Lendecke
8f94ecbb90 dbwrap: dbwrap_store_uint32->dbwrap_store_uint32_bystring
Signed-off-by: Michael Adam <obnox@samba.org>
2012-06-15 12:14:28 +02:00
Volker Lendecke
7f19a5ab52 dbwrap: dbwrap_fetch_uint32->dbwrap_fetch_uint32_bystring
Signed-off-by: Michael Adam <obnox@samba.org>
2012-06-15 12:14:28 +02:00
Volker Lendecke
749314fcf9 dbwrap: dbwrap_store_int32->dbwrap_store_int32_bystring
Signed-off-by: Michael Adam <obnox@samba.org>
2012-06-15 12:14:27 +02:00
Volker Lendecke
737c0a5473 dbwrap: dbwrap_fetch_int32->dbwrap_fetch_int32_bystring
Signed-off-by: Michael Adam <obnox@samba.org>
2012-06-15 12:14:27 +02:00