1
0
mirror of https://github.com/samba-team/samba.git synced 2024-12-22 13:34:15 +03:00
Commit Graph

6577 Commits

Author SHA1 Message Date
Alexander Bokovoy
b0a12c81b1 lib/crypto/wscript_configure: update to handle waf 2.0.4
Signed-off-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-09-05 06:37:22 +02:00
Alexander Bokovoy
b999fd3c25 lib/crypto/wscript: update to handle waf 2.0.4
Signed-off-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-09-05 06:37:22 +02:00
Thomas Nagy
8077f462c9 build:wafsamba: Build on waf 1.9
Signed-off-by: Thomas Nagy <tnagy@waf.io>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-09-05 06:37:21 +02:00
Alexander Bokovoy
f3e349bebc krb5-samba: interdomain trust uses different salt principal
Salt principal for the interdomain trust is krbtgt/DOMAIN@REALM where
DOMAIN is the sAMAccountName without the dollar sign ($)

The salt principal for the BLA$ user object was generated wrong.

dn: CN=bla.base,CN=System,DC=w4edom-l4,DC=base
securityIdentifier: S-1-5-21-4053568372-2049667917-3384589010
trustDirection: 3
trustPartner: bla.base
trustPosixOffset: -2147483648
trustType: 2
trustAttributes: 8
flatName: BLA

dn: CN=BLA$,CN=Users,DC=w4edom-l4,DC=base
userAccountControl: 2080
primaryGroupID: 513
objectSid: S-1-5-21-278041429-3399921908-1452754838-1597
accountExpires: 9223372036854775807
sAMAccountName: BLA$
sAMAccountType: 805306370
pwdLastSet: 131485652467995000

The salt stored by Windows in the package_PrimaryKerberosBlob
(within supplementalCredentials) seems to be
'W4EDOM-L4.BASEkrbtgtBLA' for the above trust
and Samba stores 'W4EDOM-L4.BASEBLA$'.

While the salt used when building the keys from
trustAuthOutgoing/trustAuthIncoming is
'W4EDOM-L4.BASEkrbtgtBLA.BASE', which we handle correct.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13539

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Alexander Bokovoy <ab@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Sep  5 03:57:22 CEST 2018 on sn-devel-144
2018-09-05 03:57:22 +02:00
Volker Lendecke
b7d77ce4b3 lib: Fix the build on FreeBSD
FreeBSD needs to explicitly #include <unistd.h> for geteuid() and close()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Fri Aug 31 18:42:31 CEST 2018 on sn-devel-144
2018-08-31 18:42:31 +02:00
Christof Schmitt
83f0ac5c6d torture: Add test for directory_create_or_exist
Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

Autobuild-User(master): Christof Schmitt <cs@samba.org>
Autobuild-Date(master): Thu Aug 30 21:19:31 CEST 2018 on sn-devel-144
2018-08-30 21:19:31 +02:00
Christof Schmitt
9f60a77e0b lib: Fix lstat check in directory_create_or_exist
The lstat check in directory_create_or_exist did not verify whether an
existing object is actually a directory. Also move the check to only
apply when mkdir returns EEXIST; this fixes CID 241930 Time of check
time of use.

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2018-08-30 18:07:14 +02:00
Bernd Kuhls
7c89edfe54 Fix uClibc build on 64bit platforms by including stdint.h
Fixes an error detected by buildroot autobuilders:
http://autobuild.buildroot.net/results/573/573e2268e205e10d1352fa81122d8f225fdb4575/build-end.log

/home/rclinux/rc-buildroot-test/scripts/instance-1/output/host/mips64el-buildroot-linux-uclibc/sysroot/usr/include/stdint.h:122:27:
error: conflicting types for 'uintptr_t'
 typedef unsigned long int uintptr_t;
                           ^
In file included from ../lib/ldb/tests/ldb_msg.c:17:0:
../third_party/cmocka/cmocka.h:126:28: note: previous declaration of 'uintptr_t' was here
       typedef unsigned int uintptr_t;

The define __WORDSIZE is missing when cmocka.h decides how to
define uintptr_t, this patch includes stdint.h when needed.

Patch sent upstream:
https://lists.samba.org/archive/samba-technical/2018-January/125306.html

Signed-off-by: Bernd Kuhls <bernd.kuhls@t-online.de>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Fri Aug 24 17:22:10 CEST 2018 on sn-devel-144
2018-08-24 17:22:09 +02:00
Douglas Bagnall
5d0dc38cfe ldb tests: remove unused code from match_rules.py
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-08-24 07:49:31 +02:00
Joe Guo
9f5bbcc10a PEP8: fix E713: test for membership should be 'not in'
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:31 +02:00
Joe Guo
7b031b01a1 PEP8: fix E502: the backslash is redundant between brackets
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:30 +02:00
Joe Guo
4d529c400b PEP8: fix E306: expected 1 blank line before a nested definition, found 0
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:30 +02:00
Joe Guo
115f2a71b8 PEP8: fix E305: expected 2 blank lines after class or function definition, found 1
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:30 +02:00
Joe Guo
fd6b2086cb PEP8: fix E303: too many blank lines (2)
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:30 +02:00
Joe Guo
211c9a5f85 PEP8: fix E302: expected 2 blank lines, found 1
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:29 +02:00
Joe Guo
542e91ef92 PEP8: fix E301: expected 1 blank line, found 0
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:29 +02:00
Joe Guo
12d3fbe15c PEP8: fix E231: missing whitespace after ','
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:28 +02:00
Joe Guo
87bbc2df97 PEP8: fix E226: missing whitespace around arithmetic operator
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:28 +02:00
Joe Guo
ba0827b560 PEP8: fix E203: whitespace before ':'
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:27 +02:00
Joe Guo
562411bd95 PEP8: fix E202: whitespace before ')'
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:27 +02:00
Joe Guo
5d532543ab PEP8: fix E128: continuation line under-indented for visual indent
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:27 +02:00
Joe Guo
bbb9f57603 PEP8: fix E127: continuation line over-indented for visual indent
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:26 +02:00
Joe Guo
dba0c7ebe5 PEP8: fix E124: closing bracket does not match visual indentation
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:26 +02:00
Joe Guo
484ce0634b PEP8: fix E123: closing bracket does not match indentation of opening bracket's line
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:26 +02:00
Joe Guo
086daf5f36 PEP8: fix E101: indentation contains mixed spaces and tabs
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:25 +02:00
Paulo Alcantara
41aa55f492 s3: util: Do not take over stderr when there is no log file
In case we don't have either a /var/log/samba directory, or pass a
non-existent log directory through '-l' option, all commands that are
daemonized with '-D' option hang when executed within a subshell.

An example on how to trigger that:

  # rm -r /var/log/samba
  # s=$(nmbd -D -s /etc/samba/smb.conf -l /foo123)
  (never returns)

So, when the above command is executed within a subshell the following
happens:

  (a) Parent shell creates a pipe, sets write side of it to fd 1
    (stdout), call read() on read-side fd, forks off a new child process
    and then executes nmbd in it.
  (b) nmbd sets up initial logging to go through fd 1 (stdout) by
    calling setup_logging(..., DEBUG_DEFAULT_STDOUT). 'state.fd' is now
    set to 1.
  (c) reopen_logs() is called by the first time which then calls
    reopen_logs_internal()
  (d) in reopen_logs_internal(), it attempts to create log.nmbd file in
    /foo123 directory and fails because directory doesn't exist.
  (e) Regardless whether the log file was created or not, it calls
    dup2(state.fd, 2) which dups fd 1 into fd 2.
  (f) At some point, fd 0 and 1 are closed and set to /dev/null

The problem with that is because parent shell in (a) is still blocked in
read() call and the new write side of the pipe is now fd 2 -- after
dup2() in (e) -- and remains unclosed.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13578

Signed-off-by: Paulo Alcantara <palcantara@suse.de>
Reviewed-by: Jim McDonough <jmcd@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Aug 18 01:32:25 CEST 2018 on sn-devel-144
2018-08-18 01:32:25 +02:00
Volker Lendecke
0d38e9e848 dbwrap: Fix a typo
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Rowland Penny <rpenny@samba.org>
2018-08-17 18:25:08 +02:00
Andrew Bartlett
bdbb9422c0 ldb: Release LDB 1.5.1
* New API ldb_dn_add_child_val() avoids passing untrusted input to
  ldb_dn_add_child_fmt() (bug 13466)
* Free memory nearer to the allocation in calls made by ldbsearch
* Do not overwrite ldb_transaction_commit failure error messages
  with a pointless del_transaction()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:24 +02:00
Andrew Bartlett
9d46795b20 ldb: extend API tests
These additional API tests just check that an invalid base DN
is never accepted.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:24 +02:00
Andrew Bartlett
2dafbd3213 ldb: Add new function ldb_dn_add_child_val()
This is safer for untrusted input than ldb_dn_add_child_fmt()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13466

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:24 +02:00
Andrew Bartlett
542e7c1791 ldb_tdb: Remove pointless check of ldb_dn_is_valid()
If the DN is not valid the ltdb_search_dn1() will catch it with ldb_dn_validate() which
is the only safe way to check this.  ldb_dn_is_valid() does not actually check, but instead
returns only the result of the previous checks, if there was one.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:24 +02:00
Andrej Gessel
d71c655eec fix mem leak in ldbsearch
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:24 +02:00
Andrej Gessel
d645546aef fix mem leak in ltdb_index_dn_base_dn and ltdb_search_indexed
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:24 +02:00
Joe Guo
b9e2a2de24 ldb: no need to call del_transaction in ldb_transaction_commit
No matter commit succeeded or failed, transation will be delete afterwards.
So there is no need to delete it here.

Aganst Samba this causes an `LDAP error 51 LDAP_BUSY` error when the transaction
fails, say while we try to add users to groups in large amount and
the original error is lost.

In Samba, the rootdse module fails early in the del part of the
start/end/del pattern, and in ldb_tdb and ldb_mdb a failed commit
always ends the transaction, even on failure.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-15 07:08:23 +02:00
Timur I. Bakeyev
e4f38b0670 ldb tests: fix assertion on wrong pointer
We are allocating msg02, but check in assertion msg01, which makes no
sense here.

Signed-off-by: Timur I. Bakeyev <timur@freebsd.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
2018-08-14 19:08:25 +02:00
Andrew Bartlett
b7f0ee93f5 Release LDB 1.5.0 for CVE-2018-1140
* Security fix for CVE-2018-1140 (NULL pointer de-reference, bug 13374)
* Fix memory leaks and missing error checks (bug 13459, 13471, 13475)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-14 13:57:15 +02:00
Andrew Bartlett
b6b72d0063 CVE-2018-1140 ldb: Add tests for search add and rename with a bad dn= DN
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
2018-08-14 13:57:15 +02:00
Andrew Bartlett
3c1fbb1832 CVE-2018-1140 ldb_tdb: Check for DN validity in add, rename and search
This ensures we fail with a good error code before an eventual ldb_dn_get_casefold() which
would otherwise fail.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
2018-08-14 13:57:15 +02:00
Andrew Bartlett
b27d973341 CVE-2018-1140 ldb_tdb: Ensure the dn in distinguishedName= is valid before use
ldb_dn_from_ldb_val() does not validate this untrusted input, so a later
call to ldb_dn_get_casefold() can fail if the input is not valid.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
2018-08-14 13:57:15 +02:00
Andrew Bartlett
3f95957d6d CVE-2018-1140 ldb: Check for ldb_dn_get_casefold() failure in ldb_sqlite
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
2018-08-14 13:57:15 +02:00
Andrej Gessel
0998f2f1bc CVE-2018-1140 Add NULL check for ldb_dn_get_casefold() in ltdb_index_dn_attr()
Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374
2018-08-14 13:57:15 +02:00
Andreas Schneider
8479401b02 lib: Add support to parse MS Catalog files
Signed-off-by: Andreas Schneider <asn@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Aug  9 19:57:02 CEST 2018 on sn-devel-144
2018-08-09 19:57:02 +02:00
Richard Sharpe
d76c7b204e Minor, really small, documentation fix.
Signed-off-by: Richard Sharpe <realrichardsharpe@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
2018-08-06 08:46:41 +02:00
Aaron Haslett
04217372f4 tdb: test for readonly locks mode on tdbbackup command
Simple bash test for readonly locks on tdbbackup:
1. Running tdbbackup on a database with and without readonly locks enabled.
2. Dump both backups and original.
3. Check all three dumps match.

A binary sample_tdb.tdb file is included for the test because the existing
sample tdbs in lib/tdb/test are either corrupt or empty.

Signed-off-by: Aaron Haslett <aaron.haslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-08-06 05:37:42 +02:00
Aaron Haslett
8f83933f6e tdb: adding readonly locks mode to tdbbackup tool
The netcmd 'domain backup offline' command will use the tdbbackup tool but
require readonly locking of tdb databases, otherwise all database access would
be blocked during a backup.  This patch adds the option.  A backup script
should use this tool with the readonly locks option after taking a transaction
lock on the target database.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-08-06 05:37:42 +02:00
Stefan Metzmacher
f68b5ee75f pthreadpool: reset monitor_fd after calling tevent_fd_set_auto_close()
This tries to convince Coverity that we don't have a resource leak:

CID 1438157:    (RESOURCE_LEAK)
Handle variable "monitor_fd" going out of scope leaks the handle.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2018-07-31 11:26:16 +02:00
Stefan Metzmacher
6da0d68f49 pthreadpool: ignore the return value of poll(NULL, 0UL, 1)
Otherwise Coverity reports this:

CID 1438160:    (CHECKED_RETURN)
Calling "poll(NULL, 0UL, 1)" without checking return value. This
library function may fail and return an error code.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2018-07-31 11:26:16 +02:00
Gary Lockyer
c891df4218 lib ldb key value: convert TDB_DATA structs to ldb_val
Convert the key value functions to use ldb_val instead of TDB_DATA.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Jul 30 17:23:22 CEST 2018 on sn-devel-144
2018-07-30 17:23:22 +02:00
Gary Lockyer
f2d5c2c5cc lib ldb: rename LTDB_* constants to LDB_KV_*
Rename all the LTDB_* constants to LDB_KV_* as they are key value level
constants and not tdb specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:54 +02:00
Gary Lockyer
19be0be232 lib ldb: move key value code to lib/ldb/ldb_key_value
Move the key value code to a separate subdirectory.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:54 +02:00
Gary Lockyer
67c055406c lib ldb: rename ltdb_parse_data_unpack_ctx
Rename ltdb_parse_data_unpack_ctx to ldb_kv_parse_data_unpack_ctx, as
it's a key value level structure and not ltdb specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:54 +02:00
Gary Lockyer
9191d3baf5 lib ldb: remove unused function prototypes
Remove unused function prototypes

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:54 +02:00
Gary Lockyer
d3bfd374a6 lib ldb: rename ltdb_idxptr to ldb_kv_idxptr
Rename ltdb_idxptr to ldb_kv_idxptr as it's key value level and not tdb
specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:54 +02:00
Gary Lockyer
f6d5cf5eb1 lib ldb: rename tdb_key_ctx to key_ctx
Rename tdb_key_ctx to key_ctx, as it's key value level and not tdb
specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
72724f75ce lib ldb: rename ltdb_cache to ldb_kv_cache
Rename ltdb_cache to ldb_kv_cache as it's key value level and not tdb
specific

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
b8c9c305b3 lib ldb: format rename ldb_kv_private
Tidy up the code format after the rename of ltdb_private to
ldb_kv_private

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
e0186d1f2d lib ldb: rename ltdb_private to ldb_kv_private
Rename ltdb_private to ldb_kv_private as it contains key value operation
context.

Note there is still some tdb specific context that can be refactored into a
separate structure along the lines of the lmdb context.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
641b38e458 lib ldb: format rename of ltdb_req_spy
Fix up the code formatting after the rename of ltdb_req_spy to
ldb_kv_req_spy

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
17a84d12e2 lib ldb: rename ltdb_req_spy to ldb_kv_req_spy
Rename ltdb_req_spy to ldb_kv_req_spy, as it is key value level and not
tdb specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
c782d710e9 lib ldb: rename ltdb_context to ldb_kv_context
Rename ltdb_context to ldb_kv_context as it is a key value level
structure and not tdb specific.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
e969de0797 lib ldb: reformat ltdb_reindex_context rename
Fix up the formatting after the rename of ltdb_reindex_context to
ldb_kv_reindex_context.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:53 +02:00
Gary Lockyer
e8aa764ea7 lib ldb: rename struct ltdb_reindex_context
Rename struct ltdb_reindex_context to ldb_kv_reindex_context, as this is
a key value level structure and not a tdb specific structure.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:52 +02:00
Gary Lockyer
9e6294668a lib ldb: fix formatting of ldb_kv rename.
Clean up the code format after the rename in the previous commit.
Hopefully doing a rename commit followed by a reformat commit makes the
code easier to review.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:52 +02:00
Gary Lockyer
1c8ea099ce lib ldb: Rename functions to ldb_kv
Rename the ldb key value functions from ltdb_* to ldb_kv_*. The renaming
is preparation for the separation of the tdb specific code from the key
value code.  This work is a follow on from the addition of the lmdb
backend.

Note that the next commit tidies up the code formatting.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-30 14:31:52 +02:00
Stefan Metzmacher
ff863f2d98 pthreadpool: we need to use pthreadpool_tevent_per_thread_cwd() on the callers pool
In pthreadpool_tevent_job_send() we remember if the job will be chdir
safe. It means we means we need to ask the callers pool when calling
pthreadpool_tevent_per_thread_cwd(), as the callers pool might
be a wrapper using pthreadpool_tevent_force_per_thread_cwd().

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-27 13:07:14 +02:00
Stefan Metzmacher
591d72f9c7 ldb_mdb: #ifdef EBADE as it is not portable
E.g. FreeBSD 11.2 doesn't have it.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-27 13:07:14 +02:00
Ralph Boehme
3fd1a41f68 pthreadpool: add a missing include
Reported-by: David Disseldorp <ddiss@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2018-07-25 17:49:05 +02:00
Gary Lockyer
a5e02f7264 lib audit_logging: add _WARN_UNUSED_RESULT_
Have the compiler issue a warning when the return code from the API is
ignored.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Gary Lockyer <gary@samba.org>
Autobuild-Date(master): Wed Jul 25 09:28:31 CEST 2018 on sn-devel-144
2018-07-25 09:28:31 +02:00
Gary Lockyer
6f4f8c51e0 json: Add unit tests for error handling
Add cmocka unit tests to exercise the error handling in the JSON
routines.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-25 06:29:50 +02:00
Gary Lockyer
79f494e51e json: Modify API to use return codes
Modify the auditing JSON API to return a response code, as the consensus
was that the existing error handling was aesthetically displeasing.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-07-25 06:29:50 +02:00
Ralph Boehme
0d2eeb9422 lib/util: rename USE_LINUX_THREAD_CREDENTIALS to HAVE_LINUX_THREAD_CREDENTIALS
The define reflects the results of a feature test, not a configure
option.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2018-07-24 17:38:28 +02:00
Ralph Boehme
fb6b6cf3e4 pthreadpool: test cancelling and freeing jobs of a wrapped pthreadpool_tevent
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:28 +02:00
Stefan Metzmacher
f9745d8b52 pthreadpool: implement pthreadpool_tevent_wrapper_create() infrastructure
This can be used implement a generic per thread impersonation
for thread pools.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:28 +02:00
Stefan Metzmacher
3c4cdb2907 pthreadpool: add pthreadpool_restart_check[_monitor_{fd,drain}]()
This makes it possible to monitor the pthreadpool for exited worker
threads and may restart new threads from the main thread again.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:28 +02:00
Stefan Metzmacher
fbafdc99ef pthreadpool: add tests for pthreadpool_tevent_[current_job_]per_thread_cwd()
Note this currently this doesn't enforce the support for
unshare(CLONE_FS) as some contraint container environment
(e.g. docker) reject the whole unshare() system call.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:28 +02:00
Stefan Metzmacher
12a45ee1a6 pthreadpool: add pthreadpool_tevent_[current_job_]per_thread_cwd()
This can be used to check if worker threads run with
unshare(CLONE_FS).

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:27 +02:00
Ralph Boehme
65e4742d16 pthreadpool: call unshare(CLONE_FS) if available
This paves the way for pthreadpool jobs that are path based.

Callers can use pthreadpool_per_thread_cwd() to check if
the current pool supports it.

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:27 +02:00
Ralph Boehme
40d15260d2 pthreadpool: test cancelling and freeing pending pthreadpool_tevent jobs/pools
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Ralph Boehme <slow@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
2018-07-24 17:38:27 +02:00
Stefan Metzmacher
f23cac39b3 pthreadpool: add a comment about a further optimization in pthreadpool_tevent_job_destructor()
This seems to be a really rare race, it's likely that the immediate
event will still trigger and cleanup.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:27 +02:00
Stefan Metzmacher
aa9b64eccf pthreadpool: maintain a list of job_states on each pthreadpool_tevent_glue
We should avoid traversing a linked list within a thread without holding
a mutex!

Using a mutex would be very tricky as we'll likely deadlock with
the mutexes at the raw pthreadpool layer.

So we use somekind of spinlock using atomic_thread_fence in order to
protect the access to job->state->glue->{tctx,ev} in
pthreadpool_tevent_job_signal().

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:27 +02:00
Stefan Metzmacher
9b73fda926 pthreadpool: add helgrind magic to PTHREAD_TEVENT_JOB_THREAD_FENCE_*()
This avoids the expected helgrind/drd warnings on the job states which
are protected by the thread fence.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:27 +02:00
Stefan Metzmacher
66aaa22f0f lib/replace: also check for valgrind/helgrind.h
This will be used in lib/pthreadpool/pthreadpool_tevent.c
in order to avoid extected helgrind/drd warnings.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:27 +02:00
Stefan Metzmacher
9656b8d8ee pthreadpool: add some lockless coordination between the main and job threads
In the direction from the main process to the job thread, we have:

- 'maycancel', which is set when tevent_req_cancel() is called,
- 'orphaned' is the job request, tevent_context or pthreadpool_tevent
  was talloc_free'ed.

The job function can consume these by using:

   /*
    * return true - if tevent_req_cancel() was called.
    */
   bool pthreadpool_tevent_current_job_canceled(void);

   /*
    * return true - if talloc_free() was called on the job request,
    * tevent_context or pthreadpool_tevent.
    */
   bool pthreadpool_tevent_current_job_orphaned(void);

   /*
    * return true if canceled and orphaned are both false.
    */
   bool pthreadpool_tevent_current_job_continue(void);

In the other direction we remember the following points
in the job execution:

- 'started'  - set when the job is picked up by a worker thread
- 'executed' - set once the job function returned.
- 'finished' - set when pthreadpool_tevent_job_signal() is entered
- 'dropped'  - set when pthreadpool_tevent_job_signal() leaves with orphaned
- 'signaled' - set when pthreadpool_tevent_job_signal() leaves normal

There're only one side writing each element,
either the main process or the job thread.

This means we can do the coordination with a full memory
barrier using atomic_thread_fence(memory_order_seq_cst).
lib/replace provides fallbacks if C11 stdatomic.h is not available.

A real pthreadpool requires pthread and atomic_thread_fence() (or an
replacement) to be available, otherwise we only have pthreadpool_sync.c.
But this should not make a real difference, as at least
__sync_synchronize() is availabe since 2005 in gcc.
We also require __thread which is available since 2002.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:26 +02:00
Stefan Metzmacher
5fa5764f30 replace: add checks for atomic_thread_fence(memory_order_seq_cst) and add possible fallbacks
This implements a full memory barrier.
On ubuntu amd64 with results in an 'mfence' instruction.

This is required to syncronization between threads, where
there's typically only one write of a memory that should be
synced between all threads with the barrier.

Much more details can be found here:
https://gcc.gnu.org/onlinedocs/gcc-7.3.0/gcc/_005f_005fatomic-Builtins.html#g_t_005f_005fatomic-Builtins
https://gcc.gnu.org/onlinedocs/gcc-7.3.0/gcc/_005f_005fsync-Builtins.html#g_t_005f_005fsync-Builtins

The main one we use seems to be in C11 via stdatomic.h,
the oldest fallback is __sync_synchronize(), which is available
since 2005 in gcc.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:26 +02:00
Stefan Metzmacher
2ebb5847de lib/replace: check for __thread support
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:26 +02:00
Stefan Metzmacher
c51cae583f pthreadpool: replace assert_return_code(ret, 0); with assert_int_equal(ret, 0);
We need to assert the exact value!

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:26 +02:00
Stefan Metzmacher
617d9c877d pthreadpool: test pthreadpool_tevent_max_threads() returns the expected result
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:26 +02:00
Stefan Metzmacher
26b35cb20c pthreadpool: make sure a pthreadpool is marked as stopped in child processes
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:26 +02:00
Stefan Metzmacher
9d31bbf979 tevent: use talloc_zero_size() for the private state in tevent_context_wrapper_create()
This is watch tevent_req_create() uses and what callers of
tevent_context_wrapper_create() would therefore also expect.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-24 17:38:25 +02:00
Ralph Boehme
ba991da434 tevent: fix CID 1437974 dereference after null check
Probably a copy/paste error from the tevent_debug() statement a few
lines above as at this place we want to pass main_ev directly to
tevent_debug() anyway.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Tue Jul 17 16:21:21 CEST 2018 on sn-devel-144
2018-07-17 16:21:21 +02:00
Ralph Boehme
3761d42e4f tevent: fix CID 1437976 dereference before null check
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2018-07-17 13:33:06 +02:00
Noel Power
7b170206b4 lib/ldb: Implement a bytes derived object for attributes py2/py3
ldb attributes are either bytes (py3) or str (py2)

Some places in the code do str(res[0]['attribute'][0])
which results in
   'result' (py2)
  b'result' (py3)

or more commonly the attribute is used to construct a string e.g.
   "blah=" + res[0]['attribute'][0] + ",foo,bar=...."

giving
   "blah=result,foo,bar=...." (py2)
and very unhelpfully
   "blah=b'result',foo,bar=...." (py3)

lots of code already constructs various strings for passing to other
api using the above. To avoid many excessive
    res[0]['attribute'][0].decode('utf8')

code like 'res[0]['attribute'][0]'

will now return LdbBytes (a new object subclassing 'bytes') in py3
instead of bytes. This object has a custom '__str__' method which
attempts to return a string decoded to uft8. In Py2 this will behave as
it did previously (this is the safer option at the moment)

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-07-13 01:12:24 +02:00
Stefan Metzmacher
3eee52b44d pthreadpool: allocate glue->tctx on glue as memory context.
This means it will go aways together with glue and thte event context.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Jul 12 17:18:01 CEST 2018 on sn-devel-144
2018-07-12 17:18:00 +02:00
Stefan Metzmacher
25756425aa pthreadpool: maintain a global list of orphaned pthreadpool_tevent_jobs
Instead of leaking the memory forever, we retry the cleanup,
if other pthreadpool_tevent_*() functions are used.

pthreadpool_tevent_cleanup_orphaned_jobs() could also be called
by external callers.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-12 14:25:19 +02:00
Stefan Metzmacher
fa070d9007 pthreadpool: make use of pthreadpool_stop() in pthreadpool_tevent_destructor()
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-12 14:25:19 +02:00
Stefan Metzmacher
791c05144e pthreadpool: add pthreadpool_tevent_job_cancel()
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-12 14:25:19 +02:00
Stefan Metzmacher
245d684d28 pthreadpool: split out pthreadpool_tevent_job from pthreadpool_tevent_job_state
This makes it much easier to handle orphaned jobs,
we either wait for the immediate tevent to trigger
or we just keep leaking the memory.

The next commits will improve this further.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-12 14:25:19 +02:00
Stefan Metzmacher
cdbad9041b pthreadpool: let pthreadpool_tevent_job_send() fail with an invalid pool
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-12 14:25:19 +02:00
Stefan Metzmacher
f19552e239 pthreadpool: split out a pthreadpool_stop() from pthreadpool_destroy()
This can be used in combination with pthreadpool_cancel_job() to
implement a multi step shutdown of the pool.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-12 14:25:19 +02:00
Stefan Metzmacher
5976841614 pthreadpool: don't process further jobs when shutting down
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2018-07-12 14:25:19 +02:00