1
0
mirror of https://github.com/samba-team/samba.git synced 2025-01-12 09:18:10 +03:00
Commit Graph

34393 Commits

Author SHA1 Message Date
Gary Lockyer
17d3a9d431 process_standard: Use the new process_context
Use the new process_context to control the from_parent_fd
This avoids the use of global variables, and will in the next patch
allow process_standard to run as what was known as single without
over-stamping a different process model.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-10-19 05:33:10 +02:00
Gary Lockyer
b852ad044b source4/smbd: refactor the process model for prefork
Refactor the process model code to allow the addition of a prefork
    process model.

    - Add a process context to contain process model specific state
    - Add a service details structure to allow service to indicate which
      process model options they can support.

    In the new code the services advertise the features they support to the
    process model.  The process model context is plumbed through to allow the
    process model to keep track of the supported options, and any state
    the process model may require.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-10-19 05:33:09 +02:00
Gary Lockyer
6d7a8d80cd tests: Add a blackbox test for smbcontrol
Add tests to check that samba processes have started and that they can be
pinged.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-10-19 05:33:09 +02:00
Lumir Balhar
87154bcfa9 python: Port samba.registry module to Python 3 compatible form
Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-10-18 10:20:26 +02:00
Tim Beale
4c9b380527 selftest: Add sanity-check RODC can't use cache to reveal secrets
Bug 12977 highlighted that Samba only checks exop GetNcChanges requests
once, when they're first received. This makes sense because valid exop
requests should only ever involve a single request. For regular
(non-exop) GetNcChanges requests, the server stores a cache of the
object GUIDs to return.

What we don't want to happen is for a malicious/compromised RODC to use
this cache to circumvent privilege checks, and receive secrets that it's
normally not permitted to access (e.g. the administrator's password).

The specific scenario we're concerned about is:
- The RODC sends a regular GetNcChanges request for all objects (without
  secrets). (This causes the server to build its GUID array cache).
- The RODC then sends a follow-on request for the next chunk, but sets
  the REPL_SECRET exop this time.

The only thing inadvertently preventing Samba from leaking secrets in
this case is updating msDS-RevealedUsers for auditing. It's possible
that a future code change may alter the codepath and open up a
security-hole without realizing. This patch adds a test case so if that
ever did happen, the selftests would detect the problem.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12977

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-10-14 07:28:18 +02:00
Ralph Boehme
5f52a0fbe8 s4/torture: vfs_fruit: test xattr unpacking
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13076

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Fri Oct 13 21:44:02 CEST 2017 on sn-devel-144
2017-10-13 21:44:01 +02:00
Ralph Boehme
c5b25d40e1 s4/torture: vfs_fruit: replace AppleDouble data blob with xattr data
The osx_adouble_w_xattr datablob is used to test conversion from sidecar
._ file metdata to Samba compatible ._ file.

The previous data blob didn't contain xattr data, the new one does.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13076

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
2017-10-13 17:40:07 +02:00
Stefan Metzmacher
2717263c14 s4:heimdal_build: there's no need to define HAVE_KRB5_ADDRESSES twice
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Björn Jacke <bjacke@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Wed Oct 11 12:33:42 CEST 2017 on sn-devel-144
2017-10-11 12:33:42 +02:00
Samuel Cabrero
015bdbd10b s3: spoolss: Extend publish_toggle test to check returned GUID string format
Extend the rpc.spoolss.printer.addprinter.publish_toggle test to
check the format of the returned GUID string in GetPrinter info
level 7 structure.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=12993

Signed-off-by: Samuel Cabrero <scabrero@suse.de>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Oct 11 06:39:00 CEST 2017 on sn-devel-144
2017-10-11 06:39:00 +02:00
Gary Lockyer
099df25f56 source4 smbd: remove global control pipe from process_standard.
The standard model uses a pipe to signal the worker processes spawned on
accept that the controlling process has terminated and that they should
shut down.  This pipe is currently a static global variable in
process_standard.c.

This patch replaces that global pipe with a file descriptor passed into
the process model init functions, giving  a single mechanism across all process
models.  This paves the way for the addition of a pre-fork process model.

Ensuring that the correct file descriptors are closed, is difficult so
it is best do this only once rather than require the process models to
do this individually.

Notes on debugging pipe ownership:

Add code to log the process id and the file descriptor of the writeable
pipe.

run:
   lsof | grep FIFO | grep samba | grep <process id>
   this will produce lines like:

   samba 25624 him 4w FIFO 0,10 0t0 472206 pipe

   where: 4w is the file descriptor and mode and the number to the left
          of "pipe" is the pipe id.
then:
   lsof | grep FIFO | grep samba | grep <pipe id>

   This will display all the processes with the pipe open and the mode
   only the smbd master process should have it open in write mode.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Thu Sep 28 02:08:34 CEST 2017 on sn-devel-144
2017-09-28 02:08:34 +02:00
Tim Beale
0d4c3e5e3f replmd: RMD_VERSION incorrectly incremented for link conflicts
This problem was noticed when 2 DCs added the same linked attribute at
roughly the same time. One DC would have a later timestamp than the
other, so it would re-apply the same link information. However, when it
did this, replmd_update_la_val() would incorrectly increment the
RMD_VERSION for the attribute. We then end up with one DC having a
higher RMD_VERSION than the others (and it doesn't replicate the new
RMD_VERSION out).

During replication RMD_VERSION is used to determine whether a linked
attribute is old (and should be ignored), or whether the information is
new and should be applied to the DB. This RMD_VERSION discrepancy could
potentially cause a subsequent linked attribute update to be ignored.

Normally when a local DB operation is performed, we just pass in a
version of zero and get replmd_update_la_val() to increment what's
already in the DB. However, we *never* want this to happen during
replication - we should always use the version we receive from the peer
DC.

This patch fixes the problem by separating the API into two:
- replmd_update_la_val(): we're updating a linked attribute in the DB,
  and so as part of this operation we always want to increment the
  version number (the version no longer need to be passed in because
  we can work it out from the existing DB entry).
- replmd_set_la_val(): we want to set a linked attribute to use the
  exact values we're telling it, including the version. This is what
  replication needs to use.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13038
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Sep 26 09:36:48 CEST 2017 on sn-devel-144
2017-09-26 09:36:48 +02:00
Tim Beale
5f93ac1f6f selftest: Add test for a re-animated object conflict
Added a test to simulate a user accidentally being deleted and 2
different admins trying to resolve the problem simultaneously - one by
re-animating the object and one by just creating a new object with
the same name.

Currently this test fails on Samba because it chooses the higher
version
number as the winner instead of the latest change.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13039

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-26 05:33:17 +02:00
Tim Beale
d278f5ea99 selftest: Windows resolves object conflicts differently to Samba
While testing link conflicts I noticed that Windows resolves conflicts
differently to Samba. Samba considers the version number first when
resolving the conflict, whereas Windows always takes the latest change.

The existing object conflict test cases didn't detect this problem
because they were both modifying the object the same number of times (so
they had the same version number).

I've added new tests that highlight the problem. They are basically the
same as the existing rename tests, except that only one DC does the
rename. Samba will always pick the renamed object as the winner, whereas
Windows picks the most recent change.

I've marked this test as a known fail for now.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13039

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-26 05:33:17 +02:00
Tim Beale
1b395f488a selftest: replica_sync did not fully cleanup if test failed
Normally the replica_sync tests do the cleanup at the end of the test
case, rather than in the tearDown(). However, if the tests don't run to
completion (because they fail), then the objects may not get cleaned up
properly, which causes the tests to fail on the 2nd test-env.

The problem is the object deletion only occurs on DC2 and it relies on
replication to propagate the deletion to DC1. Presumably this
propagation could be missed because the tests are repeatedly turning off
inbound replication on both DCs.

This patch changes the tearDown() so it tries to delete the objects off
both DCs, which appears to fix the problem.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-26 05:33:17 +02:00
Gary Lockyer
4aa1c09dbc s4:smbd: set samba root process title
Set the process title in the samba root process to clearly identify it
in ps output.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2017-09-26 00:41:16 +02:00
Gary Lockyer
4f84b6dfa8 s4:smbd: call setproctitle_init
Call setproctitle_init() in main which suppresses the
  "samba: setproctitle not initialized, please either call
   setproctitle_init() or link against libbsd-ctor."
messages, but more importantly it displays meaningful details in ps
output.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=9816

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-26 00:41:16 +02:00
Tim Beale
8c56aa2c91 selftest: Rename ntlmauth tests to ntlmdisabled
There are already some existing ntlm_auth tests, so the new tests I've
added make things a bit confusing. Also, ntlmdisabled probably better
reflects the specific case we're trying to test.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-26 00:41:16 +02:00
Tim Beale
1a1c4ad71c selftest: Add new AD DC testenv with NTLM disabled
This is so that we test the source4 case as well. Currently the only
testenv with NTLM disabled is ktest, and that only exercises the source3
code.

I've tried to support the new test environment with minimal changes to the
Samba4.pm setup code.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-26 00:41:16 +02:00
Andrew Bartlett
7ddf47951b scripting: Add script (backportable) to undo a GUID index
This script allows the DB to be read, and re-indexed, by an earlier Samba version,
most likely 4.7 with some backported patches.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Sep 23 09:16:31 CEST 2017 on sn-devel-144
2017-09-23 09:16:31 +02:00
Andrew Bartlett
5d404eaeab Do not re-use the attribute @IDXVERSION for SAMDB_INDEXING_VERSION
Confusing these two concepts is not a good idea, SAMDB_INDEXING_VERSION refers to
a change in a Samba rule to canonicalise one of our attributes, not the
in-DB index format.

As we already change @INDEXLIST in this version, this commit
is at no extra cost.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-23 05:26:15 +02:00
Andrew Bartlett
effac54893 dsdb: Set that Samba uses the GUID index in LDB
This is optional, but only to aid the downgrade script (and in case
there is some major issue found with it).  We don't support that mode,
as that would require us to test and maintain multiple code paths and
not optimise queries to be GUID centric.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-23 05:26:15 +02:00
Andrew Bartlett
a205369697 provision: Add a fixed GUID to the samba4top objectclass definition
This is only used in the OpenLDAP backend and will certainly be removed before this becomes production.

(a production backend will use the real AD top objectclass)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-23 05:26:15 +02:00
Christof Schmitt
508aebf40a torture: Add testcase for compound CREATE-WRITE-CLOSE request
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13047

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-09-22 05:45:21 +02:00
Richard Sharpe
2d97c8a4a5 Make sure smbtorture tests can run if someone has set their min protocol above NT1.
This code is SMB1 only, and already modifies
maxprotocol, so this change is appropriate.

Signed-off-by: Richard Sharpe <richard.sharpe@primarydata.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-09-20 22:48:15 +02:00
Andrew Bartlett
ee4418e73f dsdb: Only trigger a re-index once per @INDEXLIST modification
A modify of both @INDEXLIST and @ATTRIBUTES will still trigger two re-index passes
but that is a task for later.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=9527

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Sep 20 12:29:49 CEST 2017 on sn-devel-144
2017-09-20 12:29:49 +02:00
Andrew Bartlett
9e9a8d8f88 s4-dnsserver: Check for too many DNS results
If we had this check in when the wildcard DNS tests were written, we would have
noticed that the name needed to be escaped (see previous commit).

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12994
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-20 08:15:21 +02:00
Andrew Bartlett
c174702107 s4-dnsserver: Always encode user-supplied names when looking up DNS records
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12994

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-20 08:15:21 +02:00
Garming Sam
3e1870c26c kcc: Remove unused, untested KCC code
This code tries to implement the full KCC algorithm, but never
actually worked correctly.

Removing this doesn't affect the full-mesh KCC. This code only
attempted to calculate a graph using the "proper" algorithm, though it
neglected to write its results back into the database. The full-mesh
calculation occurs elsewhere.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Wed Sep 20 06:28:07 CEST 2017 on sn-devel-144
2017-09-20 06:28:07 +02:00
Andrew Bartlett
dd53be2756 ldap_server: Plumb ldb error string from a failed connect to ldapsrv_terminate_connection()
However, do not plumb it to the client-seen error string, as it could contain server paths.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-20 02:25:30 +02:00
Andrew Bartlett
c1e41d489d samdb: Rework samdb_connect_url() to return LDB error code and an error string
This allows debugging of why the LDB failed to start up.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-20 02:25:30 +02:00
Günther Deschner
2c745cfac5 s4-torture: move lease break handler outside the lease testsuite.
Guenther

Signed-off-by: Guenther Deschner <gd@samba.org>
Pair-Programmed-With: Jose A. Rivera <jarrpa@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Tue Sep 19 09:36:40 CEST 2017 on sn-devel-144
2017-09-19 09:36:40 +02:00
Tim Beale
1541c50b37 selftest: Add some tests for linked attribute conflicts
Currently we have tests that check we can resolve object conflicts, but
these don't test anything related to conflicting linked attributes.
This patch adds some basic tests that checks that Samba can resolve
conflicting linked attributes.

This highlights some problems with Samba, as the following tests
currently fail:
- test_conflict_single_valued_link: Samba currently can't resolve a
  conflicting targets for a single-valued linked attribute - the
  replication exits with an error.
- test_link_deletion_conflict: If 2 DCs add the same linked attribute,
  currently when they resolve this conflict the RMD_VERSION for the
  linked attribute incorrectly gets incremented. This means the version
  numbers get out of step and subsequent changes to the linked attribute
  can be dropped/ignored.
- test_full_sync_link_conflict: fails for the same reason as above.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

Autobuild-User(master): Garming Sam <garming@samba.org>
Autobuild-Date(master): Mon Sep 18 09:56:41 CEST 2017 on sn-devel-144
2017-09-18 09:56:41 +02:00
Tim Beale
46c1f7bdee getncchanges.c: max_links calculation didn't work well in some cases
The max_links calculation didn't work particularly well if max_links was
set to a value lower than max_objects.

As soon as repl_chunk->object_count exceeded repl_chunk->max_links, the
chunk would be deemed full, even if there was only one link to send (or
even worse, no links to send). For example, if max_objects=100 and
max_links=10, then it would send back chunks of 10 objects (or 9 objects
and 1 link).

I believe the historic reason this logic exists is to avoid overfilling
the response message. It's hard to tell what the appropriate limit would
be because the total message size would depend on how many attributes
each object has.

I couldn't think of logic that would be suitable for all cases. I toyed
with the idea of working out a percentage of how full the message is.
However, adjusting the max_links doesn't really make sense when the
settings are small enough, e.g. max_objects=100 and max_links=100 is
never going to overfill the message, so there's no reason to alter the
values.

In the end I went with:
- If the user is using non-default values, just use those.
- In the default value case, just use the historic calculation

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:25 +02:00
Tim Beale
3c0d80d0c3 replmd: Avoid duplicated debug/warnings
We display warnings if a target object is missing but it's still OK to
continue the replication. Currently we need to check the target twice -
once to verify it when we first receive it, and once when we actually
commit it (we can't skip the 2nd check altogether because in the join
case, they could occur quite far apart).

One annoying side-effect is we get the same warning message coming out
twice in these special cases.

In the cases where we're checking the dsdb_repl_flags, we can actually
just bypass the verification checks for the target object (if it doesn't
exist we still continue anyway). This may save us a tiny bit of
unnecessary work.

For cross-partition links, we can limit logging these warnings to when
the objects are actually being committed. This avoids spurious warnings
in the join case (i.e. we receive the link before we receive the target
object's partition, but we have received all partitions by the time we
actually commit the objects).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:25 +02:00
Tim Beale
44ca84166e replmd: Allow missing targets if GET_TGT has already been set
While running the selftests, I noticed a case where DC replication
unexpectedly sends a linked attribute for a deleted object (created in
the drs.ridalloc_exop tests). The problem is due to the
msDS-NC-Replica-Locations attribute, which is a (known) one-way link.
Because it is a one-way link, when the test demotes the DC and deletes
the link target, there is no backlink to delete the link from the source
object.

After much debate and head-scratching, we decided that there wasn't an
ideal way to resolve this problem. Any automated intervention could
potentially do the wrong thing, especially if the link spans partitions.
Running dbcheck will find this problem and is able to fix it (providing
the deleted object is still a tombstone). So the recommendation is to
run dbcheck on your DCs every 6 months (or more frequently if using a
lower tombstone lifetime setting).

However, it does highlight a problem with the current GET_TGT
implementation. If the tombstone object had been expunged and you
upgraded to 4.8, then you would be stuck - replication would fail
because the target object can't be resolved, even with GET_TGT, and
dbcheck would not be able to fix the hanging link. The solution is to
not fail the replication for an unknown target if GET_TGT has already
been set (i.e. the dsdb_repl_flags contains
DSDB_REPL_FLAG_TARGETS_UPTODATE).

It's debatable whether we should add a hanging link in this case or
ignore/drop the link. Some cases to consider:
- If you're talking to a DC that still sends all the links last, you
  could still get object deletion between processing the source object's
  links and sending the target (GET_TGT just restarts the replication
  cycle from scratch). Adding a hanging link in this case would be
  incorrect and would add spurious information to the DB.
- Suppose there's a bug in Samba that incorrectly results in an object
  disappearing. If other DCs then remove any links that pointed to that
  object, it makes recovering from the problem harder. However, simply
  ignoring the link shouldn't result in data loss, i.e. replication won't
  remove the existing link information from other DCs. Data loss in this
  case would only occur if a new DC were brought online, or if it were a
  new link that was affected.
Based on this, I think ignoring the link does the least harm.

This problem also highlights that we should really be using the same
logic in both the unknown target and the deleted target cases.
Combining the logic and moving it into a common
replmd_allow_missing_target() function fixes the problem. (This also has
the side-effect of fixing another logic flaw - in the deleted object
case we would unnecessarily retry with GET_TGT if the target object was
in another partition. This is pointless work, because GET_TGT won't
resolve the target).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:25 +02:00
Tim Beale
278039ff78 getncchanges.c: Support GET_TGT better with large numbers of links
A source object can potentially link to thousands of target objects.
We have to be careful not to overfill the GetNCChanges response message
with more data than it's possible to send. We also don't want the client
to timeout while we're busy checking the linked attributes. The GET_TGT
support added so far is fairly dumb - this patch extends it to better
handle larger numbers of links.

To do so, this extends the repl_chunk usage so that it also works out if
the current chunk is full of links. Now as soon as the chunk is full of
either links or objects, we stop and send it back.

These changes now mean that we need to also check:
- that all the links for the last source object in the previous chunk
  have been sent, before we move on and send the next object. This only
  takes effect when immediate_link_sync is configured. It also means
  that a chunk in the middle of the replication cycle can now contain
  only links, and no objects.
- when GET_TGT is used, we only send back the links that we've verified
  the target object for. i.e. if we stop checking links because we timed
  out, we only send back the links whose targets were checked.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:25 +02:00
Garming Sam
7ba10844d1 getncchanges.c: Refactor to track more state using repl_chunk
To prepare GET_TGT to deal with a large number of links better, there
is now a 'repl_chunk' struct to help keep track of all the factors
relating to the current chunk of replication data (i.e. how many
objects/links we can send and how many we've already processed). This
means we can have a consistent way of working out whether the current
chunk is full (whether that be due to objects, links, or just too much
time taken).

This patch should not alter functionality. This is just a refactor to
add the basic framework, which will be used in the next patch.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-09-18 05:51:25 +02:00
Tim Beale
10df9f6bfd getncchanges.py: Add a multi-valued linked attribute test
Add a test where a source object links to multiple different targets.
First we do the replication without GET_TGT and check that the server
can handle sending a chunk containing only links (in the middle of the
replication). Then we repeat the replication forcing GET_TGT to be used.

To avoid having to create 1500 objects/links, I've lowered the 'max
link sync' setting on the vampire_dc testenv to 250.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:25 +02:00
Tim Beale
693e3adc1a getncchanges.py: Add a test for dropped cross-partition links
Samba would drop linked attributes that span partitions if it didn't
know about the target object. This patch adds a test that exposes the
problem.

I've re-used the code from the previous re-animation test to do this.
I've also added a very basic DcConnection helper class that basically
stores the connection state information the drs_base.py uses for
replication. This allows us to switch the DC we want to replicate from
easily. This approach could potentially be retro-fitted to some of the
existing test cases, as it allows us to test both the DRS client code
and server code at the same time.

Note this test case relates to the code change for commit
fae5df891c.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:24 +02:00
Tim Beale
9697190600 getncchanges.py: Add test for replicating reanimated objects
Reading between the lines, this scenario seems to be the main reason
that Microsoft added the GET_TGT flag. MS AD can handle getting links
for unknown targets OK, but if it receives links for a deleted/recycled
target then it would tend to drop the received links. Samba client also
used to drop the links if talking to a Microsoft DC (or a Samba server
with GET_TGT support).

The specific scenario is the client side already knows about a deleted
object. That object is then re-animated and used as the target for a
linked attribute. *Then* the target object gets updated again so it gets
sent in a later replication chunk to the linked attribute, i.e. the
client receives the link before it learns that the target object has
been re-animated.

In this test we're interested in particular at how the client behaves
when it receives a linked attribute for a deleted object. (It *should*
retry with GET_TGT to make sure the target is up-to-date. However, it
was just dropping the linked attribute).

To exercise the client-side, we disable replication, setup the
links/objects on one DC the way we want them, then force a replication
to the second DC. We then check that when we query each DC, they both
tell us about the links/objects we're expecting (i.e. no links got
lost).

Note that this wasn't a problem with older versions of Samba-to-Samba
because sending the links last guaranteed that the target objects were
always up-to-date.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:24 +02:00
Tim Beale
ed2fc52243 drs: Add basic GET_TGT support
This adds basic DRS_GET_TGT support. If the GET_TGT flag is specified
then the server will use the object cache to store the objects it sends
back. If the target object for a linked attribute is not in the cache
(i.e. it has not been sent already), then it is added to the response
message.

Note that large numbers of linked attributes will not be handled well
yet - the server could potentially try to send more than will fit in a
single repsonse message.

Also note that the client can sometimes set the GET_TGT flag even if the
server is still sending the links last. In this case, we know the client
supports GET_TGT so it's safe to send the links interleaved with the
source objects (the alternative of fetching the target objects but not
sending the links until last doesn't really make any sense).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:24 +02:00
Tim Beale
821094d50b getncchanges.py: Add tests for object deletion during replication
Add tests that delete the source and target objects for linked
attributes in the middle of a replication cycle.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:24 +02:00
Tim Beale
469aed088f getnc_exop.py: Extend EXOP_REPL_OBJ test case to use GET_TGT
We already check that when we use GET_ANC that we still only receive a
single object when EXOP_REPL_OBJ is used. This extends the test to also
check that only a single object is returned when GET_TGT is used.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:24 +02:00
Tim Beale
00b20c825c getncchanges.py: Add test for GET_ANC and GET_TGT combined
The code has to handle needing GET_ANC and GET_TGT in combination, i.e.
where we fetch the target object for the linked attribute and the target
object's parent is unknown as well. This patch adds a test case to
exercise this code path.

The second part of this test exercises GET_ANC/GET_TGT for an
incremental replication, where the objects are getting filtered by an
uptodateness-vector/HWM.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:24 +02:00
Tim Beale
6ec9ef2beb getncchanges.py: Add test for adding links during replication
We have identified a case where the Samba server can send linked
attributes but not the target object. In this case, the Samba DRS client
would hit the "Failed to re-resolve GUID" case in replmd and silently
discard the linked attribute.

However, Samba will resend the linked attribute in the next cycle
(because its USN is still higher than the committed HWM), so it should
recover OK. On older releases, this may have caused problems if the
first error resulting in a hanging link (which might mean the second
time it's processed it still fails to be added).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:24 +02:00
Tim Beale
af82bdefcc getncchanges.py: Add some GET_TGT test cases
test_repl_get_tgt:
- Adds 2 sets of objects
- Links one set to the other
- Changes the order so the target object comes last in the
  replication (which means the client has to use GET_TGT)
- Checks that when GET_TGT is used that we have received all target
  objects we need to resolve the linked attibutes
- Checks that we expect to receive the linked attributes *before*
  the last chunk is sent (by default, Samba sends all the links at
  the end, so this fails)
- Checks that we eventually receive all expected objects, and all
  links we receive match what is expected

test_repl_get_tgt_chain:
  This adds the linked attributes in a more complicated chain. We add
  300 objects, but the links for 100 objects will point to a linked
  chain of 200 objects.
  This was mainly to determine whether or not Windows follows the
  target object (i.e. whether it sends all the links for the target
  object as well). It turns out Windows maintains its own linked
  attribute DB, so it sends the links based on USN.

Note that the 2 testenvs fail for different reasons. promoted_dc fails
because it is sending all the linked attributes last. vampire_dc fails
because it doesn't support GET_TGT yet, so it sends the link before the
peer knows about the target object.

Note that to test against vampire_dc (rather than the ad_dc_ntvfs DC),
we need to send the GetNCChanges requests to DC2 instead of DC1.
I've left the DC numbering scheme as is, but I've addeed a test_ldb_dc
handle to drs_base.py - it defaults to DC1, but tests can override it
easily and still have everything work.

While running the new tests through autobuild, I noticed an intermittent
LDAP_ENTRY_ALREADY_EXISTS failure in the test setup(). This appears to
be due to a timing issue in the background replication between the
multiple testenvs. Adding some randomness so that the test base OU is
unique seems to avoid the problem.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:24 +02:00
Tim Beale
172eedc076 getnc_exop.py: Fix GET_TGT behaviour in DRS tests
The existing code never passed the more_flags parameter into the
actual getNCChanges request, i.e. _getnc_req10(). This meant the
existing GET_TGT tests effectively did nothing.

Passing the flag through properly means we have to now change the tests
as the DNs returned by Windows now include any target objects in the
linked attributes. These tests now fail against Samba (because it
doesn't support GET_TGT yet).

Also added comments to the tests to help explain what they are actually
doing.

Note that Samba and Windows can return the objects in different orders,
due to significant differences in their underlying DB implementations
(Windows stores links in a separate DB, so sends links ordered strictly
by USN, whereas Samba sends links based on the USN of the source
object). To make the test a fair comparison between Windows and Samba,
we need to use dn_ordered=False.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-18 05:51:24 +02:00
Gary Lockyer
af38d73b06 s4/smbd: set the process group.
Set the process group in the samba daemon, the --no-process-group option
allows this to be disabled.  The no-process-group option needs to be
disabled in self test.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Mon Sep 18 04:39:50 CEST 2017 on sn-devel-144
2017-09-18 04:39:50 +02:00
Andreas Schneider
05169a6047 samba_upgradedns: When we setup the internal dns cleanup bind-dns dir
Make sure to remove everything from the bind-dns directory to avoid
possible security issues with the named group having write access to all
AD partions

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-09-16 00:57:25 +02:00
Andreas Schneider
8cf5c5f0fa samba_upgradedns: Print better hints after we migrated the config
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-09-16 00:57:25 +02:00
Andreas Schneider
aef2b915a2 samba_upgradedns: Change the group of the 'binddns dir' too
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-09-16 00:57:24 +02:00
Andreas Schneider
bf64939d22 s4:bind_dlz: Try the 'binddns dir' first
The directory is normally empty if you did not provision or call
samba_upgradedns for the bind_dlz module.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2017-09-16 00:57:24 +02:00
Tim Beale
e115a42318 getncchanges.c: Send linked attributes in each chunk
Instead of sending all the linked attributes at the end, add a
configurable option to send the links in each replication chunk.

The benefits of this approach are:
- it can reduce memory overhead, as we don't have to keep all the links
in memory over the entire replication cycle.
- the client should never end up knowing about objects but not their
links. (Although we're not sure that this has actually resulted in
replication problems, i.e. missing links).

Note that until we support GET_TGT, this approach can mean we now send
a link where the client doesn't know about the target object, causing
the client to siliently drop that linked attribute. Hence, this option
is switched off by default.

Implementation-wise, this code works fairly the same as before. Instead
of sorting the entire getnc_state->la_sorted array at the end and then
splitting it up over chunks, we now split the links up over chunks and
then sort them when we copy them into the message. This should be OK, as
I believe the MS-DRSR Doc says the links in the message should be sorted
(rather than sorting *all* the links overall). Windows behaviour seems
to chunk the links based on USN and then sort them.

getnc_state->la_idx now tracks which links in getnc_state->la_list[]
have already been sent (instead of tracking getnc_state->la_sorted).
This means the la_sorted array no longer needs to be stored in
getnc_state and we can free the array's memory once we've copied the
links into the message. Unfortunately, the link_given/link_total debug
no longer reports the correct information, so I've moved these into
getncchanges_state struct (and now free the struct a bit later so it's
safe to reference in the debug).

The vampire_dc testenv has been updated to use this new behaviour.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Fri Sep 15 10:07:33 CEST 2017 on sn-devel-144
2017-09-15 10:07:33 +02:00
Tim Beale
c15c53817c getnchanges.c: Avoid unnecessary continue
There's not really much after the continue that we're skipping now. We
can just flip the logic and avoid the continue.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:13 +02:00
Tim Beale
3a8dfcc6a1 getncchanges.c: Split out code to get an object for a response
Basically, everytime we try to add an object to the response, we want
to:
- Build it (i.e. pack it into an RPC message format)
- Add it to our object-cache if we're keeping one
- Add any ancestors needed for the client to resolve it (if GET_ANC)

GET_TGT is going to use the exact same code, so split this out into a
separate function, rather than duplicating it.

The GET_ANC case also uses almost identical code, but it differs in a
couple of minor aspects. I've left this as is for now, as I'm not sure
if this is by accident or by design.

Because all the memory was talloc'd off the 'obj' variable, we now need
to replace it with a tmp TALLOC_CTX.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:13 +02:00
Tim Beale
ec8a7fa22e getncchanges.c: Reduce the parameters to get_nc_changes_build_object()
Fifteen parameters seems a bit excessive. Instead, pass it the structs
containing the information it cares about.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:13 +02:00
Tim Beale
1d70375a10 getncchanges.c: Remove unused ncRoot_dn parameter
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:13 +02:00
Tim Beale
85898a6e2f getncchanges.c: Remove a really old TODO
This TODO was added in 2009 (before Samba supported linked_attributes
in getNCChanges())

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:12 +02:00
Tim Beale
b4b843b710 getncchanges.c: Replace hard-coded numbers with a define
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:12 +02:00
Tim Beale
63818f022f getncchanges.c: Refactor how objects get added to the response
Adding GET_TGT support is going to make things more complicated, and I
think we are going to struggle to do this without refactoring things a
bit.

This patch adds a helper struct to store state related to a single
GetNCChanges chunk. I plan to add to this with things like max_links,
max_objects, etc, which will cutdown on the number of variables/
parameters we pass around.

I found the double-pointer logic where we add objects to the response
confusing - hopefully this refactor simplifies things slightly, and it
allows us to reuse the code for the GET_TGT case.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:12 +02:00
Tim Beale
f77d55f1be getncchanges.c: Refactor how we add ancestor links
If the current object had already been sent as an ancestor, we were
duplicating the code that added its links and updated the HWM mark.
We want these to occur when we reach the place where the object's USN
naturally occurs.

Instead of duplicating this code, we can just skip the call to
get_nc_changes_build_object() if the object has already been sent.
There is already an existing 'nothing to send'/continue case after we've
updated the highwater mark.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:12 +02:00
Tim Beale
4057438e3b getncchanges.c: Add ancestor links when the object normally gets sent
Currently we add links each time we send an object, but we don't
actually send these links until the end of the replication cycle.

In subsequent patches we want the links to be sent in the same chunk as
their source object, ideally in as close to USN order as possible.
Processing ancestors complicates this a bit, as the ancestor will have a
higher USN than what we're currently up to, and so potentially will the
ancestor's links.

This patch moves where the ancestor's links get added to the
getnc_state->la_list. The ancestor's links now get added when the object
would normally get sent based purely on its USN (we update the highwater
mark at this point too).

This should not affect functionality, i.e. because we send all the links
at the end, it should make no difference at what point they get added to
the list.

This duplicates a tiny bit of code, but this will be cleaned up in the
next patch.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:12 +02:00
Tim Beale
2abdd09aa2 getncchanges.c: Split GET_ANC block out into its own function
When we add GET_TGT support, it's going to need to reuse all this code
(i.e. to add any ancestors of the link target). This also trims down
the rather large dcesrv_drsuapi_DsGetNCChanges() function a bit.

Note also fixed a compiler warning in the WERR_DS_DRA_INCONSISTENT_DIT
error block which may have caused issues previously (statement was
terminated by a ',' rather than a ';').

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:12 +02:00
Tim Beale
46b3aab514 getncchanges.c: Split sorting linked attributes into separate function
Longer-term we want to split up the links so that they're sent over
multiple GetNCChanges response messages. So it makes sense to split this
code out into its own function. In the short-term, this removes some of
the complexity from dcesrv_drsuapi_DsGetNCChanges() so that the function
is not quite so big.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:12 +02:00
Tim Beale
240e433611 getncchanges.c: Rename anc_cache to obj_cache
When we add GET_TGT support we will reuse the ancestor cache and it
should work the same way - if we've already sent an object because it
was needed for resolving a child object or a link target, then there's
no point sending it again.

This just renames anc_cache --> obj_cache.

An extra is_get_anc flag has been added to getnc_state - once GET_TGT
support is added, we can't assume GET_ANC based solely on the existence
of the obj_cache.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-15 06:18:12 +02:00
Alexander Bokovoy
3f0938b938 Install dcerpc/__init__.py for all Python environments
Also fix whitespace. We use tabs, not spaces in Python/waf code.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13030

Signed-off-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Sep 14 22:29:39 CEST 2017 on sn-devel-144
2017-09-14 22:29:39 +02:00
Andrew Bartlett
24600e8e91 repl_meta_data: Show failing replicated entry in error code
This re-work of our LDIF printing avoids some of the privacy issue from
printing the full LDIF at level 4, while showing the entry that actually fails.

Instead, with e3988f8f74 we now print the DN
only at level 4, then the full message at 8.

With this patch on failure, we print the redacted failing message at 5.

While all of the DRS replication data is potentially sensitive
the passwords are most sensitive, and are now not printed unencrypted.

This discourages users from sending the full failing trace, as the
last entry is much more likely the issue.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-14 18:43:16 +02:00
Andrew Bartlett
5d9bb80a02 schema: Rework dsdb_schema_set_indices_and_attributes() db operations
Commit ec9b1e881c did not fully fix this.

There is no value in using dsdb_replace(), we are under the read lock
and replace just confuses things further.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13025

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-09-14 18:43:16 +02:00
Andrew Bartlett
8d8d31eb2b dsdb: Add missing \n to debug
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:27 +02:00
Andrew Bartlett
51289a6f9b debug: Add new debug class "drs_repl" for DRS replication processing
This is used in the client and in the server

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:27 +02:00
Andrew Bartlett
e3988f8f74 repl_meta_data: Re-work printing of replicated entries
This re-work of our LDIF printing avoids some of the privacy issue from
printing the full LDIF at level 4, while showing the entry that actually fails.

Instead, we print the DN only at level 4, then the full message at 8.

While all of the DRS replication data is potentially sensitive
the passwords are most sensitive, and are now not printed unencrypted.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:27 +02:00
Andrew Bartlett
7cfaf70694 linked_attributes: Use ldb_ldif_message_redacted_string() for consistency
This avoids printing un-encrypted secret values in logs, and while links are not likely
secret, this avoids a future copy and paste using ldb_ldif_message_string() again.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:27 +02:00
Andrew Bartlett
cc78de5581 repl_meta_data: Use ldb_ldif_message_redacted_string() to avoid printing secrets in logs
This avoids printing un-encrypted secret values in logs

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-09-07 06:56:27 +02:00
Lumir Balhar
c81aff362f python: Fix bad type in conversion of NTSTATUS.
More info: https://lists.samba.org/archive/samba-technical/2017-August/122574.html

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-09-06 11:35:18 +02:00
Andreas Schneider
2d0e13837d python:samba: Add code to remove obsolete files in the private dir
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Sep  6 03:54:19 CEST 2017 on sn-devel-144
2017-09-06 03:54:19 +02:00
Andreas Schneider
8f2dee256e python:samba: Use 'binddns dir' in samba-tool and samba_upgradedns
This provisions the bind_dlz files in the 'binddns dir'. If you want to
migrate to the new files strcuture you can run samba_upgradedns!

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-09-05 23:58:20 +02:00
Andreas Schneider
3fa7c43ef7 s4:bind_dlz: Use the 'binddns dir' if possible
The code makes sure we are backwards compatible. It will first check if
we still have files in the private directory, if yes it will use those.

If the the file is not in the private directory it will try the binddns
dir.

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-09-05 23:58:20 +02:00
Andrew Bartlett
a5dbcbeeed password_hash: Make a common failure with "password hash gpg key ids" clearer
This drove me to strace before I understood what it really meant.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-08-30 10:48:20 +02:00
Andrew Bartlett
7fdeea0f30 dsdb: Add comment showing where the normal password rules are applied
This looks like a footnote, but is actually where the default password rules are applied.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-08-30 10:48:19 +02:00
Peter Somogyi
f0b85c1da2 s4/torture: make --unclist active with smb2 testcases
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12985

Signed-off-by: Peter Somogyi <psomogyi@hu.ibm.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Wed Aug 30 00:16:51 CEST 2017 on sn-devel-144
2017-08-30 00:16:51 +02:00
Tim Beale
b6e6379514 selftest: Use a unique(ish) OU for every run of getnc_unpriv
An intermittent problem I noticed with tests in the past is that the
setup can fail to create the base OU because it already exists.
I believe this is because the previous testenv DC has replicated out the
test object, but not its deletion at the point that the next testenv DC
starts running the test.

This only seemed to happen very occassionally (I haven't seen it
happen with getnc_unpriv yet, but I also haven't run it through the
autobuild yet).

Using same randomness in the test OU should help avoid this sort of
problem, and it matches what some other replication tests do.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Tim Beale
d6a384b24b s4-drsuapi/selftest: Add extra tests for invalid DNs
Add some test cases to check for requests for invalid/non-existent DNs.
This exercises the first return case added in commit:
  s4-drsuapi: Refuse to replicate an NC is that not actually an NC

I've also updated the error code returned here to match Windows.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Tim Beale
37ed946c75 selftest: Update getnc_unpriv tests to pass against Samba
In general Windows seems to return BAD_DN rather than ACCESS_DENIED for
an unprivileged user. In the the long-term, it's unrealistic to think
that Samba and Windows will agree exactly on every error code returned.
So for the tests to be maintainable and pass against Windows and Samba,
they need to handle differences in expected errors. To get around this
problem, I've changed the expected_error to be a set, so that multiple
error codes (one for Microsoft, one for Samba) can be specified for each
test case. This approach also highlights the cases where Microsoft and
Samba currently differ.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Tim Beale
2d0766a48b s4-drsuapi: Set getnc_state *after* we've checked request is valid
We were creating the getnc_state (and storing it on the connection)
before we had done some basic checks that the request was valid. If the
request was not valid and we returned early with an error, then the
partially-initialized getnc_state was left hanging on the connection.
The next request that got sent on the connection would try to use this,
rather than creating a new getnc_state from scratch.

The main side-effect of this was if you sent an invalid GetNCChanges
request twice, then it could be rejected the first time and accepted the
second time.

Note that although an invalid request was accepted, it would typically
not return any objects, so it would not actually leak any secure
information.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Tim Beale
6158f1839f selftest: GetNCChanges can 'accept' a repeated bad request
In theory, if we send the exact same rejected request again, we should
get the same response back from the DC. However, we don't - the request
is accepted if we send it a second time.

This patch updates the repl_rodc test to demonstrate the problem (which
now causes the test to fail).

Note that although the bad GetNCChanges request is not rejected outright,
the response that gets sent back is empty - it has no objects in it, so
it's not an actual security hole. It is annoying problem for writing
self-tests though.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Tim Beale
3c8fa7b27f s4-drsuapi: Change REPL_SECRET error code to match Windows
The existing SOURCE_DISABLED error code doesn't seem to make a lot of
sense. Window sends back an ACCESS_DENIED error in the same situation,
which seems more appropriate.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Tim Beale
122c8e1fa2 selftest: Extend further getnc_unpriv tests to pass against windows 2012R2
An important change in this patch is changing the ACE type from
 A (Allow)
to
 AO (Object Allow)

as that will then respect the supplied GUID, which we also make use
the constant from the security.idl.

This reworks the tests to check replication with users with the
following rights:
- only GET_CHANGES
- only GET_ALL_CHANGES
- both GET_CHANGES and GET_ALL_CHANGES
- no rights

We basically want to test various different GetNCChanges requests
against each type of user rights, and the only difference is the
error/success value we get back. I've structured the tests this way, so
that we have 4 test_repl_xyz_userpriv() functions (to cover each of the
above user rights cases), and each test sends the same series of
GetNCChanges requests of varying validity.

Currently all these tests fail against Samba because Samba sends
different error codes to Windows.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Andrew Bartlett
87bc8d8f16 selftest: Confirm privileged replication of an OU is not permitted
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Andrew Bartlett
cdb8c4ae80 selftest: Move get_partial_attribute_set() to DrsBaseTestCase
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Andrew Bartlett
83f2338009 selftest: encrypt the LDAP connection in drs_base.py
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Andrew Bartlett
607ba1a203 s4-drsuapi: Refuse to replicate an NC is that not actually an NC
This prevents replication of an OU, you must replicate a whole NC per Windows 2012R2

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Andrew Bartlett
53512529be selftest: Make dirsync test use symobolic name and OA not A
A is for Allow, OA is for Object Allow, which means check the GUID.

The previous ACE allowed all access, which was not the intention.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Andrew Bartlett
2feea24061 dsdb: Use samba.generate_random_password() in dirsync test
We do not like fixed passwords

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12946

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Tim Beale
f8a30d3167 s4-drsuapi: Use sam_ctx consistently in dcesrv_drsuapi_DsGetNCChanges()
Trying to use bstate->sam_ctx_system by mistake can cause crashes if
non-admin users replicate. To avoid this problem we use the sam_ctx
variable, however it wasn't used consistently everywhere. Replace the
remaining references to b_state->sam_ctx to avoid potential confusion.

This change was made based on review feedback from Metze.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Andrew Bartlett
dd863b6049 s4-drsuapi: Avoid segfault when replicating as a non-admin with GUID_DRS_GET_CHANGES
Users who are not administrator do not get b_state->sam_ctx_system filled in.

We should probably use the 'sam_ctx' variable in all cases (instead of
b_state->sam_ctx*), but I'll make this change in a separate patch, so
that the bug fix remains independent from other tidy-ups.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12946

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-29 07:23:28 +02:00
Love Hornquist Astrand
dd3e06f14e HEIMDAL: don't bother seeing q if not sent
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12986

Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from heimdal commit 19f9fdbcea11013cf13ac72c416f161ee55dee2b)

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Aug 28 15:10:54 CEST 2017 on sn-devel-144
2017-08-28 15:10:54 +02:00
Love Hornquist Astrand
9f245aafdc HEIMDAL: allow optional q in DH DomainParameters
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12986

Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from heimdal commit e8317b955f5a390c4f296871ba6987ad05478c95)
2017-08-28 11:18:15 +02:00
Ralph Boehme
51f40a0e1d s4/torture: add a test for rename change notification with inotify enabled
This is already fixed in master by
5eccc2fd0072409f166c63e6876266f926411423~10..5eccc2fd0072409f166c63e6876266f926411423.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=12903

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Aug 26 05:05:08 CEST 2017 on sn-devel-144
2017-08-26 05:05:08 +02:00
Andreas Schneider
ec8ec35bd7 s4:torture: Do not overrun arrays in test_displayshares()
If we do not 'break', we overrun the array access size.

Found by Coverity.

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2017-08-25 16:15:08 +02:00
Andreas Schneider
1e002db765 s4:torture: Fix talloc_array in test_EnumValue()
Found by Coverity.

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2017-08-25 16:15:08 +02:00
Andreas Schneider
c90069b264 s4:torture: The teardown function should just return
The teardown functions should not return on error but finish cleaning
up!

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12984

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Aug 24 13:23:22 CEST 2017 on sn-devel-144
2017-08-24 13:23:21 +02:00
Andreas Schneider
bd44e435fa s4:torture: Delete printer before we remove the driver
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12984

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>
2017-08-24 09:30:33 +02:00
Andreas Schneider
731fe596ac s4:torture: Use a different driver name for add_driver tests
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12984

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>
2017-08-24 09:30:33 +02:00
Lumir Balhar
0ffe030c0d python: Make generated modules samba.ntstatus and samba.werror Python 3 compatible.
Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Tue Aug 22 17:38:17 CEST 2017 on sn-devel-144
2017-08-22 17:38:17 +02:00
Lumir Balhar
051a3ff6eb python: scripting: Port ntstatus and werror generators to Python 3 compatible form.
Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlet <abartlet@samba.org>
2017-08-22 13:47:15 +02:00
Tim Beale
0bb4d28272 getncchanges.py: Add test for GET_ANC and linked attributes
Add a basic test that when we use GET_ANC and the parents have linked
attributes, then we receive all the expected links and all the expected
objects by the end of the test.

This extends the test code to track what linked attributes get received
and check whether they match what's present on the DC.

Also made some minor cleanups to store the received objects/links each
time we successfully receive a GETNCChanges response (this saves the
test case having to repeat this code every time).

Note that although this test involves linked attributes, it shouldn't
exercise the GET_TGT case at all.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:12 +02:00
Tim Beale
9f0ae6e44d getncchanges.py: Add GET_ANC replication test case
This test:
- creates blocks of parent/child objects
- modifies the parents, so the child gets received first in the
  replication (which means the client has to use GET_ANC)
- checks that we always receive the parent before the child (if not, it
  either retries with GET_ANC, or asserts if GET_ANC is already set)
- modifies the parent objects to change their USN while the
  replication is in progress
- checks that all expected objects are received by the end of the
  test

I've added a repl_get_next() function to help simulate a client's
behaviour - if it encounters an object it doesn't know the parent of,
then it retries with GET_ANC.

Also added some debug to drs_base.py that developers can turn on to make
it easier to see what objects we're actually receiving in the
responses.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:12 +02:00
Tim Beale
4cfc296885 getncchanges.py: Add a new test for replication
This adds a new test to check that if objects are modified during a
replication, then those objects don't wind up missing from the
replication data.

Note that when this scenario occurs, samba returns the objects in a
different order to Windows. This test doesn't care what order the
replicated objects get returned in, so long as they all have been
received by the end of the test.

As part of this, I've refactored _check_replication() in drs_base.py so
it can be reused in new tests. In these cases, the objects are split up
over multiple different chunks. So asserting that the objects are returned
in a specific order makes it difficult to run the same test on both Samba
and Windows.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:12 +02:00
Tim Beale
fae5df891c replmd: Try to add forward-link for unknown cross-partition links
Previously Samba would just drop cross-partition links where the link
target object is unknown. Instead, what we want to do is try to add the
forward link for the GUID specified. We can't add the backlink because
we don't know the target, however, dbcheck should be able to fix any
missing backlinks.

The new behaviour should now mean dbcheck will detect the problem and be
able to fix it. It's still not ideal, but it's better than dropping the
link completely.

I've updated the log so that it has higher severity and tells the user
what they need to do to fix it.

These changes now mean that the selftests now detect an error - instead
of completely dropping the serverReference, we now have a missing
backlink. I've updated the selftests to fix up any missing
serverReference backlinks before running dbcheck.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:12 +02:00
Tim Beale
89cf5c3f76 replmd: Don't fail cycle if we get link for deleted object with GET_TGT
We are going to end up supporting 2 different server schemes:
A. the old/default behaviour of sending all the linked attributes last,
   at the end of the replication cycle.
B. the new/Microsoft way of sending the linked attributes interleaved
   with the source/target objects.

Normally if we're talking to a server using the old scheme-A, we won't
ever use the GET_TGT flag. However, there are a couple of cases where
it can happen:
- A link to a new object was added during the replication cycle.
- An object was deleted while the replication was in progress (and
the linked attribute got queued before the object was deleted).

Talking to an Samba DC running the old scheme will just cause it to
start the replication cycle from scratch again, which is fairly
harmless. However, there is a chance that the same thing can happen
again, in which case the replication cycle will fail (because GET_TGT
was already set).

Even if we're using the new scheme (B), we could still potentially hit
this case, as we can still queue up linked attributes between requests
(group memberships can be larger than what can fit into a single
replication chunk).

If GET_TGT is set in the GetNcChanges request, then the local copy of
the target object should always be up-to-date when we process the linked
attribute. So if we still think the target object is deleted/recycled at
this point, then it's safe to ignore the linked attribute (because we
know our local copy is up-to-date). This logic matches the MS spec logic
in ProcessLinkValue().

Not failing the replication cycle may be beneficial if we're trying to
do a full-sync of a large database. Otherwise it might be time-consuming
and frustrating to repeat the sync unnecessarily.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:12 +02:00
Tim Beale
ab12aed7e1 replmd: Avoid dropping links if link target is deleted
The server-side can potentially send the linked attribute before the
target-object. This happens on Microsoft, and will happen on Samba once
server-side GET_TGT support is added. In these cases there is a hole
where the Samba client can silently drop the linked attribute.

If the old copy of the target object was deleted/recycled, then the
client can receive the new linked attribute before it realizes the target
has now been reincarnated. It silently ignores the linked attribute,
thinking its receiving out of date information, when really it's the
client's copy of the target object that's out of date.

In this case we want to retry with the GET_TGT flag set, which will
force the updated version of the target object to be sent along with the
linked attribute. This deleted/recycled target case is the main reason
that Windows added the GET_TGT flag.

If the server sends all the links at the end, instead of along with the
source object, then this case can still be hit. If so, it will cause the
server to restart the replication from the beginning again. This is
probably preferential to silently dropping links.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:12 +02:00
Tim Beale
18ea167ade replmd: Move where we store linked attributes
There was a bug in my previous patch where the code would verify
*all* links in the list, rather than just the ones that are new. And it
would do this for every replication chunk it received, regardless of
whether there were actually any links in that chunk.

The problem is by the time we want to verify the attributes, we don't
actually know which attributes are new. We can fix this by moving where
we store the linked attributes from the start of processing the
replication chunk to the end of processing the chunk. We can then verify
the new linked attributes at the same time we store them.

Longer-term we may want to try to apply the linked attribute at this
point. This would save looking up the source/target objects twice, but
it makes things a bit more complicated (attributes will usually apply at
this point *most* of the time, but not *all* the time).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:12 +02:00
Tim Beale
f87332eb35 replmd: Set GET_ANC if Windows sends a link with unknown source object
Windows replication can send the linked attribute before it sends the
source object. The MS-DRSR spec says that in this case the client should
resend the GetNCChanges request with the GET_ANC flag set. In my testing
this resolves the problem - Windows will include the source object for the
linked attribute in the same replication chunk.

This problem doesn't happen with Samba-to-Samba replication, because the
source object for the linked attribute is guaranteed to have already been
sent.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:12 +02:00
Tim Beale
cc201c2c4f drepl: Support GET_TGT on periodic replication client
- Update IDL comments to include Microsoft reference doc
- Add support for sending v10 GetNCChanges request (needed for the
  GET_TGT flag, which is in the new 'more_flags' field)
- Update to also set the GET_TGT flag in the same place we were setting
  GET_ANC (I split this logic out into a separate function).
- The state struct now needs to hold a 'more_flags' field as well (this
  flag is different to the GET_ANC replica flag)

Note that using the GET_TGT when replicating from a Windows DC could be
highly inefficient. Because Samba keeps the GET_TGT flag set throughout
the replication cycle, it will basically receive a repeated object from
Windows for every single linked attribute that it receives.

I believe Windows behaviour only expects the client to set the GET_TGT
flag when it actually needs to (i.e. when it receives a target object it
doesn't know about), rather than throughout the replication cycle.
However, this approach won't work with Samba-to-Samba replication,
because when the server receives the GET_TGT flag it restarts the
replication cycle from scratch. So if we only set the GET_TGT flag when
the client encountered an unknown target then Samba-to-Samba could
potentially get into an endless replication loop.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:12 +02:00
Tim Beale
67617d4700 drs: Check target object is known after applying objects
Currently we only check that the target object is known at the end of
the transaction (i.e. the .prepare_commit hook). It's too late at this
point to resend the request with GET_TGT. Move this processing earlier
on, after we've applied all the objects (i.e. off the .extended hook).

In reality, we need to perform the checks at both points. I've
split the common code that gets the source/target details out of the
la_entry into a helper function. It's not the greatest function ever,
but seemed to make more sense than duplicating the code.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:12 +02:00
Tim Beale
f69596cd21 drs: Fail replication transaction instead of dropping links
If the DRS client received a linked attribute that it couldn't resolve
the target for, then it would just ignore that link and keep going. That
link would then be lost forever (although a full-sync would resolve
this). Instead of silently ignoring the link, fail the transaction.

This *can* happen on Samba, but it is unusual. The target object and
linked-attribute would need to be added while a replication is still in
progress. It can also happen fairly easily when talking to a Windows DC.

There are two import exceptions to this:

1). Linked attributes that span partitions. We can never guarantee that
we will have received the target object, because it may be in a partition
we haven't replicated yet. Samba doesn't have a great way of handling
this currently, but we shouldn't fail the replication (because that breaks
basic join tests). Just skip that linked attribute and hope that a
subsequent full-sync will fix it.
(I queried Microsoft and they said resolving cross-partition linked
attributes is a implementation-specific problem to solve. GET_TGT won't
resolve it)

2). When the replication involves a subset of objects, e.g.
critical-only. In these cases, we don't increase the highwater-mark, so
it is probably not such a dire problem if we don't add the link. In the
case of critical-only, we will do a subsequent full sync which will then
add the links.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:11 +02:00
Tim Beale
04ce638a1f replmd: Split checking link attr target into new function
We want to re-use this code to check that the linked attribute's target
object exists *before* we try to commit the transaction. This will allow
us to re-request the block with the GET_TGT flag set.

This splits checking the target object exists into a separate function.

Minor changes of note:
- the 'parent' argument was passed to replmd_process_linked_attribute()
  as NULL, so I've just replaced where it was used in the refactored code
  with NULL.
- I've tweaked the "Failed to find GUID" error message slightly to display
  the attribute ID rather than the attribute name (saves repeating
  lookups and/or passing extra arguments).
- Tweaked the replmd_deletion_state() logic - it only made sense to call
  it in the code block where we actually found the target

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
2017-08-18 06:07:11 +02:00
Gary Lockyer
beeec1ff7c tests: replace traffic_summary test with python blackbox test
Replace the shell subunit test for script/traffic_summary.pl with a
python black box test.

This involves moving the test files to more standard locations.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Thu Aug 17 07:59:38 CEST 2017 on sn-devel-144
2017-08-17 07:59:38 +02:00
Gary Lockyer
7057abcfcd scripts: Scripts to replay and generate samba traffic
Scripts to generate representative network traffic and replay this to a
samba instance.  For load testing, performance profiling and capacity
planning.

traffic_learner  process a file generated by traffic_summary and
                 generate a model that can be used by traffic_replay to
                 generate samba network traffic.

traffic_replay   Replay a summary file generated by traffic_summary, or
                 use a model created by traffic_learner to generate
                 network traffic.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>
Pair-programmed-with: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>
Pair-Programmed-With: Tim Beale <timbeale@catalyst.net.nz>
2017-08-17 04:06:06 +02:00
Jeremy Allison
5473a277e3 lib: talloc: Use the system <talloc.h> include.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu Aug 17 00:53:48 CEST 2017 on sn-devel-144
2017-08-17 00:53:48 +02:00
Jeremy Allison
bee22f187c lib: tevent: Use system <tevent.h>, not internal header path (except in self-test).
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-16 20:58:13 +02:00
Volker Lendecke
652bf0ca7e libhttp: Remove an unneeded include
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Aug 16 04:11:47 CEST 2017 on sn-devel-144
2017-08-16 04:11:47 +02:00
Andrew Bartlett
0dbcd6a4ea py-librpc: Strictly check the type of the incoming sid pointer
This avoids casting another type of object to a void* and then to a SID

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

Autobuild-User(master): Garming Sam <garming@samba.org>
Autobuild-Date(master): Tue Aug 15 12:00:58 CEST 2017 on sn-devel-144
2017-08-15 12:00:58 +02:00
Andrew Bartlett
5bb341fb9c s4/lib/tls: Use SHA256 to sign the TLS certificates
The use of SHA-1 has been on the "do not" list for a while now, so make our
self-signed certificates use SHA256 using the new
gnutls_x509_crt_sign2 provided since GNUTLS 1.2.0

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12953
2017-08-15 08:07:10 +02:00
Gary Lockyer
edcbc99125 dnsserver: Add support for dns wildcards
Add support for dns wildcard records. i.e. if the following records
exist

  exact.samba.example.com 3600 A 1.1.1.1
  *.samba.example.com     3600 A 1.1.1.2

look up on exact.samba.example.com will return 1.1.1.1
look up on *.samba.example.com     will return 1.1.1.2
look up on other.samba.example.com will return 1.1.1.2

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12952
2017-08-15 08:07:10 +02:00
Gary Lockyer
34acf5a992 dnsserver: Tighten DNS name checking
Add checks for the maximum permitted length, maximum number of labels
and the maximum label length.  These extra checks will be used by the
DNS wild card handling.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12952
2017-08-15 08:07:10 +02:00
Gary Lockyer
1184770a76 dnsserver: Tests for dns wildcard entries
Add tests for dns wildcards.
Tests validated against Windows Server 2012 R2

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12952
2017-08-15 08:07:10 +02:00
Jeremy Allison
f816de5636 s4: com: Replace erroneous inclusion of internal talloc.h header with external.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12932

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Aug 15 08:06:40 CEST 2017 on sn-devel-144
2017-08-15 08:06:40 +02:00
Andreas Schneider
4791c152b0 s4:rpc_server:backupkey: Move variable into scope
CID: #1415510

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12959

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2017-08-11 18:21:22 +02:00
Andreas Schneider
abd74c3ba5 heimdal: Fix printing a short int into a string
The size of portstr is too small to print an integer and we should print
a short anyway.

This fixes building with GCC 7.1

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12930

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Fri Aug 11 18:08:04 CEST 2017 on sn-devel-144
2017-08-11 18:08:04 +02:00
Ralph Boehme
bbc225de83 s4/torture: additional tests for kernel-oplocks
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Richard Sharpe <realrichardsharpe@gmail.com>
2017-08-09 18:41:06 +02:00
Ralph Boehme
a334fff8a8 s4/torture: reproducer for kernel oplocks issue with streams
test_smb2_kernel_oplocks3() wouldn't have failed without the patches,
I'm just adding it to have at least one test that tests with 2
clients. All other tests use just one client.

test_smb2_kernel_oplocks4() is the reproducer.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=12791

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Richard Sharpe <realrichardsharpe@gmail.com>
2017-08-09 18:41:06 +02:00
Andreas Schneider
1c3b678e7d s3:torture: Fix spoolss test to build with -O3
Initialize variables so that we do not get a build warning that they
might be used uninitilized.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12930

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2017-08-09 09:49:08 +02:00
Andreas Schneider
b5283c70e3 s4:samdb: Fix building Samba with -O3
gcc error: ‘result’ may be used uninitialized

This wont happen, because ldb will return and error, but the compiler
doesn't understand this.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12930

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2017-08-09 09:49:08 +02:00
Stefan Metzmacher
b88f9384b0 s4:auth/ntlmssp: add support for using "winbind" as DC
This adds support for trusted domains to the auth stack on AD DCs.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:03 +02:00
Stefan Metzmacher
a6ad56cfa9 s4:auth: use "sam winbind" for the netlogon server
This adds authentication support for trusted domains to the
netlogon server.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:03 +02:00
Stefan Metzmacher
690c5e6f36 s4:auth/ntlm: provide auth_check_password_wrapper_send/recv to auth4_context
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:03 +02:00
Stefan Metzmacher
79b891a746 s4:auth_winbind: implement async authentication via IRPC
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:03 +02:00
Stefan Metzmacher
8022b63f6c s4:rpc_server/netlogon: make use of auth_check_password_send/recv()
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:03 +02:00
Stefan Metzmacher
023bd2d15d s4:rpc_server/netlogon: make use of async kdc_check_generic_kerberos_send/recv()
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:03 +02:00
Stefan Metzmacher
373da95b0b s4:rpc_server/netlogon: prepare dcesrv_netr_LogonSamLogon_base for async processing
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:03 +02:00
Stefan Metzmacher
b10d01d14a s4:rpc_server/netlogon: check auth_level for validation level 6 already in dcesrv_netr_LogonSamLogon_check()
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:03 +02:00
Stefan Metzmacher
c8d14a554e s4:librpc/rpc: add support for HttpAuthOption=negotiate
Note that rpcproxy.dll on Windows doesn't support kerberos,
it allways downgrades the connection to NTLMSSP.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:03 +02:00
Stefan Metzmacher
77b44fbe36 s4:lib/http: pass down the target service/hostname to gensec
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:02 +02:00
Stefan Metzmacher
d1f479e73b s4:lib/http: add HTTP_AUTH_NEGOTIATE which maps to the "http_negotiate" gensec backend
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:02 +02:00
Stefan Metzmacher
9fd27d7a47 s4:http/gensec: implement "http_negotiate" using GENSEC_OID_SPNEGO
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:02 +02:00
Stefan Metzmacher
8813295e1f s4:http/gensec: make the "NTLM" base64 wrapping more generic
We only need to know the prefix "NTLM" and the submech oid GENSEC_OID_NTLMSSP
everything else can be generic.

This should allow us to implement "Negotiate" with GENSEC_OID_SPNEGO
trivial.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:02 +02:00
Stefan Metzmacher
a219c359b9 s4:http/gensec: rename ntlm.c to generic.c
Check with git show -C

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:02 +02:00
Stefan Metzmacher
b5854ecc98 s4:librpc/rpc: pass down HTTP_AUTH_* values directly to dcerpc_pipe_open_roh_send()
They get passed to http_send_auth_request_send() unmodified.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:02 +02:00
Stefan Metzmacher
8d36bbe187 s4:librpc/rpc: remember the target_hostname on ncacn_http connections
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:02 +02:00
Stefan Metzmacher
a6ae9da98d s4:lib/http: pass a generic prefix blob to http_parse_auth_response()
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:02 +02:00
Stefan Metzmacher
bdece1df9c s4:lib/http: use strcasecmp(h->key, "WWW-Authenticate") instead of strncasecmp()
The key is already normalized and should match completely.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:02 +02:00
Stefan Metzmacher
56ab5cdbe0 s4:lib/http: remove indentation level from http_parse_auth_response()
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:02 +02:00
Stefan Metzmacher
e42f12c6be s4:lib/http: let http_read_response_send/recv() also consume the body if it fits into a max value
We need to consume full HTTP responses from the socket during the
authentication exchanges, otherwise our HTTP parser gets out of sync for
the next requests.

This will be important for gensec mechs which use an even number
for authentication packets.

I guess this should be done just based on the Content-Length value and
not based on the response code.

So far I saw bodies with 200 and 401 codes.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:01 +02:00
Stefan Metzmacher
7b86da08ea s4:lib/http: lower HTTP_MAX_HEADER_SIZE from UINT_MAX to 0x1FFFF
We don't need very large headers, the largest ones are
"Authorization" or "WWW-Authenticate", but 128k should be
more than enough for all headers.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-08-07 15:20:01 +02:00
Andrew Bartlett
d5750f0163 dsdb: Fix dsdb_next_callback to correctly use ldb_module_done() etc
If we do not call ldb_module_done() then we do not know that up_req->callback()
has been called, and ldb_next_request() will call the callback again.

If called twice, the new ldb_lock_backend_callback() in ldb 1.2.0 will segfault.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12904

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Aug  1 07:52:38 CEST 2017 on sn-devel-144
2017-08-01 07:52:38 +02:00
Andrew Bartlett
39c6274084 dsdb: Do not force a re-index of sam.ldb on upgrade to 4.7
This means that no compatibleFeatures or incompatibleFeatures will be honoured
until a re-index, but that can be triggered when these features are set.

New databases will still get this support.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12855
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-08-01 03:59:24 +02:00
Richard Sharpe
2186d4d317 Add a test for unsetting Delete-on-Close before the close by DoC opener.
Windows semantics says that any unset of Delete-on-Close before the client
that opened for Delete-on-Close closes the file is silently ignored and the file
is still deleted on the last close. This test tests that in a single open case.

Signed-off-by: Richard Sharpe <realrichardsharpe@gmail.com>
Reviewed-by: Volker Lendecke <vl@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Fri Jul 28 11:47:06 CEST 2017 on sn-devel-144
2017-07-28 11:47:06 +02:00
Richard Sharpe
886c26238a Fix incorrect error messages:unsetting not setting.
The error messages are wrong and could give testers the wrong idea.

Signed-off-by: Richard Sharpe <realrichardsharpe@gmail.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
2017-07-28 07:53:25 +02:00
Andrew Bartlett
1ea6b5168f s4-rpc_server: Improve debug of new endpoints
This helps us know what process model is required and what one is in use.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12939

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Jul 28 04:12:08 CEST 2017 on sn-devel-144
2017-07-28 04:12:08 +02:00
Andrew Bartlett
bc48c4b54b s4-rpc_server: ensure we get a new endpoint for netlogon
If we share the single process RPC servers with the multi-process RPC servers
on the same endpoint, they will default to running in an single process

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12939

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:15 +02:00
Andrew Bartlett
0554bc237f s4-cldap/netlogon: Match Windows 2012R2 and return NETLOGON_NT_VERSION_5 when version unspecified
The previous patch set this incorrectly to NETLOGON_NT_VERSION_1

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:15 +02:00
Arvid Requate
88db634ed8 s4-dsdb/netlogon: allow missing ntver in cldap ping
BUG: https://bugzilla.samba.org/show_bug.cgi?id=11392

Signed-off-by: Arvid Requate <requate@univention.de>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-28 00:25:15 +02:00
Arvid Requate
22a94b728b s4:torture/ldap: Test netlogon without NtVer
BUG: https://bugzilla.samba.org/show_bug.cgi?id=11392

Signed-off-by: Arvid Requate <requate@univention.de>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-28 00:25:15 +02:00
Tim Beale
69f593ec5a repl: Remove old TODO
This TODO was added in 2007 before we supported linked attributes.
It's no longer relevant.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-28 00:25:15 +02:00
Tim Beale
50b638d15c getncchanges.c: Remove unused null_scope variable
This was added in 4cc6b5a69b but the very next commit
(f1c6bab60e) removed where it was set, which meant the variable
was always false and seemingly pointless.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-28 00:25:15 +02:00
Tim Beale
dddcf80660 getnc_exop.py: Fix typo in function name
This drove me crazy when I tried to search for it.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-28 00:25:15 +02:00
Tim Beale
475a320646 libnet: Initialize req_level in become_dc tests
The net.api.become.dc tests would always pass the request into
libnet_vampire_cb_store_chunk() with req_level=0, which meant that
storing the chunk didn't use the correct replica_flags/exop.

I noticed this problem when working on client-side support for GET_TGT.
My changes relied on the critical-only request flag being passed down
into replmd, but because the request flags weren't passed correctly, my
changes caused the become_dc tests to fail.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-28 00:25:15 +02:00
Andrew Bartlett
47a90dcc57 selftest: Use get_creds_ccache_name() in fsmo.py
This avoids a new kinit for every role transfer

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:14 +02:00
Andrew Bartlett
e91782541e selftest: Add and use new helper function get_creds_ccache_name()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:14 +02:00
Andrew Bartlett
7ad34d1206 selftest: Use new --krb5-ccache in drs_base.py
This means that instead of doing a new kinit, the process-wide ccache
is re-used, which is much faster.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:14 +02:00
Andrew Bartlett
4cc5ceb297 selftest: Port DrsBaseTestCase._{en,dis}able_all_repl() to self.runsubcmd()
This avoids forking a subprocess with self.check_run()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:14 +02:00
Andrew Bartlett
f7c46ed56c selftest: Port DrsBaseTestCase._disable_inbound_repl() to self.runsubcmd()
This avoids forking a subprocess with self.check_run()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:14 +02:00
Andrew Bartlett
cc3d83677b selftest: Port DrsBaseTestCase._enable_inbound_repl() to self.runsubcmd()
This avoids forking a subprocess with self.check_run()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:14 +02:00
Andrew Bartlett
09ce35ecf3 selftest: Port DrsBaseTestCase._net_drs_replicate() to self.runsubcmd()
This avoids forking a subprocess with self.check_run()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:14 +02:00
Andrew Bartlett
24de78e16a selftest: Remove unused import in ridalloc_exop.py
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-07-28 00:25:14 +02:00
Andrew Bartlett
f9bd16d8f3 selftest: Use self.runsubcmd() in DrsReplicaSyncTestCase
This will allow catching the correct error messages and failure when _net_drs_replicate()
is reworked to not use a subprocess.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:14 +02:00
Andrew Bartlett
6a75d4a8cf selftest: Use self.runsubcmd() to run samba-tool for _test_force_demote in ridalloc_exop.py
This is the standard way to run samba-tool from in the test scripts and allows
assertion that the command ran as expected

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:14 +02:00
Andrew Bartlett
37cf29ef7d selftest: Use self.runsubcmd() to run samba-tool for _test_join in ridalloc_exop.py
This is the standard way to run samba-tool from in the test scripts and allows
assertion that the command ran as expected

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:14 +02:00
Andrew Bartlett
a5f62958cc selftest: Add tests for credentials.get_named_ccache()
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2017-07-28 00:25:13 +02:00
Jeremy Allison
c27a6438cb s4: schema: Allocate global_schema off the NULL context, not the talloc_autofree_context().
The ldb context keeps a talloc_reference to this also, so the long-live allocation
context can be NULL.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12932

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-26 21:35:22 +02:00
Jeremy Allison
fe2ac3e304 s4: COM: Remove talloc_autofree_context() from (unused) COM code.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12932

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-26 21:35:21 +02:00
Jeremy Allison
34ca1b3af4 s4: modules. Fix missing TALLOC_CTX in module init function.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12932

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2017-07-26 21:35:21 +02:00
Marc Muehlfeld
4b56f80379 Set log level for "Not authorative for" from 2 to 5
Signed-off-by: Marc Muehlfeld <mmuehlfeld@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Marc Muehlfeld <mmuehlfeld@samba.org>
Autobuild-Date(master): Wed Jul 26 21:34:48 CEST 2017 on sn-devel-144
2017-07-26 21:34:48 +02:00
Christof Schmitt
ffee37c243 torture: Add sharemode tests for SMB2
There are two pieces: Test access with different sharemodes through SMB
and verify access, and also provide tests that can be used with file
systems enforcing share modes outside of Samba.

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Jul 26 09:30:31 CEST 2017 on sn-devel-144
2017-07-26 09:30:31 +02:00
Stefan Metzmacher
8268374c83 s4:lib/http: rewrite http_send_auth_request_*() using gensec_update_send/recv
The new logic makes it much clearer that we have a loop of

gensec_update_send()
gensec_update_recv()
http_send_request_send()
http_send_request_recv()
http_read_response_send()
http_read_response_recv()

Until the local gensec and the server are ready.

I've tested this against Windows 2008R2 like this:

bin/smbtorture \
  -W BLA --realm=BLA.BASE \
  -s /dev/null -Uadministrator%A1b2C3d4 \
  ncacn_http:w2k8r2-219[593,RpcProxy=w2k8r2-219.bla.base,HttpUseTls=false,HttpAuthOption=basic] \
  rpc.epmapper.epmapper.Lookup_simple \

and:

bin/smbtorture \
  -W BLA --realm=BLA.BASE \
  -s /dev/null -Uadministrator%A1b2C3d4 \
  ncacn_http:w2k8r2-219[593,RpcProxy=w2k8r2-219.bla.base,HttpUseTls=false,HttpAuthOption=ntlm] \
  rpc.epmapper.epmapper.Lookup_simple \

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2017-07-25 13:51:11 +02:00
Andrew Bartlett
427a11b812 s4-netlogon: Use log_escape to protect against un-validated strings
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
2017-07-24 23:29:23 +02:00
Andrew Bartlett
abd821b76b s4-netlogon: Extend ServerAuthenticate3 logging to split up username forms
This splits out the username into the input, mapped and obtained
just as we do elsewhere.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
2017-07-24 23:29:23 +02:00
Gary Lockyer
efc335a030 source4 netlogon: Add authentication logging for ServerAuthenticate3
Log NETLOGON authentication activity by instrumenting the
netr_ServerAuthenticate3 processing.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12865

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
2017-07-24 23:29:23 +02:00
Gary Lockyer
f3d3e6da5a tests auth_log: Add new tests for NETLOGON
Tests for the logging of NETLOGON authentications in the
netr_ServerAuthenticate3 message processing

Test code based on the existing auth_log tests.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12865

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
2017-07-24 23:29:23 +02:00
Gary Lockyer
1898096c7e rpc: use symbolic constant to replace /root/ncalrpc_as_system
Modified to use constant AS_SYSTEM_MAGIC_PATH_TOKEN instead of string literal
"/root/ncalrpc_as_system"

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12865

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
2017-07-24 23:29:22 +02:00
Andreas Schneider
9b64b11c2f mit-kdb: Fix NULL pointer check after malloc
This fixes building with GCC 7.1.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12930

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Mon Jul 24 18:45:34 CEST 2017 on sn-devel-144
2017-07-24 18:45:33 +02:00
Andreas Schneider
314cf60893 s4:kcc: Add a NULL check before qsort()
This fixes building with GCC 7.1.1

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12930

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2017-07-24 14:50:11 +02:00
Bob Campbell
eb2e77970e samdb/cracknames: support user and service principal as desired format
This adds support for DRSUAPI_DS_NAME_FORMAT_USER_PRINCIPAL and
DRSUAPI_DS_NAME_FORMAT_SERVICE_PRINCIPAL as desired formats.

This also causes the test in cracknames.py to no longer fail.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12842

Signed-off-by: Bob Campbell <bobcampbell@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Jul 24 11:10:26 CEST 2017 on sn-devel-144
2017-07-24 11:10:26 +02:00
Bob Campbell
c186e02b40 samdb/cracknames: do not show recycled when a guid is desired
Previously, when a GUID was desired to
cracknames, it would include recycled objects as well. This would
sometimes result in two objects being returned from a query which is
supposed to return a unique GUID. For example, if a deleted user had
the same sAMAccountName as a non-deleted user and cracknames was used to
find the GUID of this account, it would return two GUIDs, and so would
fail with DRSUAPI_DS_NAME_STATUS_NOT_UNIQUE.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12842

Signed-off-by: Bob Campbell <bobcampbell@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-07-24 07:14:10 +02:00
Bob Campbell
4779afe0d2 python/tests: add python test for cracknames
This fails due the bug, which causes the related test in
drsuapi_cracknames.c to flap. It also fails due to us not yet supporting
DRSUAPI_DS_NAME_FORMAT_USER_PRINCIPAL or
DRSUAPI_DS_NAME_FORMAT_SERVICE_PRINCIPAL.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12842

Signed-off-by: Bob Campbell <bobcampbell@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-07-24 07:14:10 +02:00
Andrew Bartlett
f1e4ae9971 s4-drepl: Block GetNCChanges during a DsReplicaSync
If we do not block these, we can get RPC faults
(DCERPC_NCA_S_PROTO_ERROR) which gives WERR_WRITE_FAULT back to the
DsReplicaSync call as there are two outstanding requests on the wire
at the one time.

We will get to the next operation as soon as this is finished
when we call run_pending_ops().

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12926

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Sun Jul 23 12:32:49 CEST 2017 on sn-devel-144
2017-07-23 12:32:49 +02:00
Andrew Bartlett
6c45db6779 s4-drepl: Use tevent_schedule_immediate() in DsReplicaSync handler
When we are sent a DsReplicaSync() we should work on inbound replication
(ideally from the requested source, but so far we just start the whole queue)
right away, not after 1 second.

We should also target inbound replication, not any outbound replication
notification that may happen to be due.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12921

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Jul 22 07:45:31 CEST 2017 on sn-devel-144
2017-07-22 07:45:31 +02:00
Stefan Metzmacher
13f91927e0 s4:http/gensec: add missing tevent_req_done() to gensec_http_ntlm_update_done()
This was missing in commit d718e92d5e.

Sadly we can't have automated tests for this as we only implement
the client side for this protocol.

I've tested with using:

bin/smbtorture \
  -W BLA --realm=BLA.BASE \
  -s /dev/null -Uadministrator%A1b2C3d4 \
  ncacn_http:w2k8r2-219[593,RpcProxy=w2k8r2-219.bla.base,HttpUseTls=false,HttpAuthOption=basic] \
  rpc.epmapper.epmapper.Lookup_simple \

and:

bin/smbtorture \
  -W BLA --realm=BLA.BASE \
  -s /dev/null -Uadministrator%A1b2C3d4 \
  ncacn_http:w2k8r2-219[593,RpcProxy=w2k8r2-219.bla.base,HttpUseTls=false,HttpAuthOption=ntlm] \
  rpc.epmapper.epmapper.Lookup_simple \

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12919

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Jul 21 23:29:39 CEST 2017 on sn-devel-144
2017-07-21 23:29:39 +02:00
Andrew Bartlett
4031b303e4 getncchanges: Do not segfault if somehow we get 0 results from an ldb_search with scope BASE
This should not happen, but we have seen this happen in autobuild
before the whole-DB locking issues were resolved by
https://bugzilla.samba.org/show_bug.cgi?id=12858

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2017-07-21 09:30:25 +02:00