IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
self.create_backup() uses self.run_cmd(), which is a wrapper around
self.check_output(). Rework the code to call the underlying
check_output() function directly instead.
The reason we're doing this is we want run_cmd() to catch exceptions and
fail the test (i.e. in the next patch). However, we can't do that because
this test case relies on receiving the exceptions.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13676
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Tag prefork work processes with "(worker 0)", and sort the process list
on server name to get a consistent order.
Service: PID
--------------------------------------
cldap_server 15588
...
ldap_server 15584
ldap_server(worker 0) 15627
ldap_server(worker 1) 15630
ldap_server(worker 2) 15632
ldap_server(worker 3) 15634
nbt_server 15576
notify-daemon 15638
...
samba 0
...
wrepl_server 15580
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add tests for the restarting of failed/terminated process, by the
pre-fork process model.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Nov 21 10:46:20 CET 2018 on sn-devel-144
Rather than unstable hash order. Ideally we'd do them in proper DN order.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Then we can reuse the re obj.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This will simplify the logic and improve performance.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Two mistakes here:
- res[:-1] will copy but lost the last char
- string is immutable in python, there is no need to copy it explicitly
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This option has default value False, and was actually not passed down from cli
to LDAPBase. However, LDAPBase.__init__ has default value True for it.
After the change, a few tests using ldapcmp are affected.
Add --skip-missing-dn explicitly to keep the behavior consistent,
otherwise test will fail.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Just define another dict for return value, seems no need to modify
original dict.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This simplify the logic and improve performance a lot.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The list comprehension will repeat for each item.
For large database, this make the command freeze.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
So we don't need to validate ourselves.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
So we don't need to validate ourselves.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This method actually changed both objects and print info.
__eq__ is not a proper name and is not designed for this case.
Rename to diff.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Bundel -> Bundle
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add extra tests to test the content returned by samr_EnumDomainUsers,
and tests for the result caching added in the following commit.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add extra tests to test the content returned by samr_EnumDomainGroups,
and tests for the result caching added in the following commit.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add extra tests to test the content returned by samr_QueryDisplayInfo,
which is not tested for the ADDC. Also adds tests for the result
caching added in the following commit.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Under normal operation, users shouldn't see giant cookies in their logs.
We still log the initial cookie retrieved from the cache database, which
should still be helpful for identifying corrupt cookies.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13686
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The replUpToDateVector could be incorrect after an offline backup was
restored. This means replication propagation dampening doesn't work
properly. In the worst case, a singleton DC would have no
replUpToDateVector at all, and so *all* objects created on that DC get
replicated every time a new DRS connection is established between 2 DCs.
This becomes a real problem if you used that singleton DC to create 100K
objects...
This patch flushes the replUpToDateVector when an offline backup gets
restored. We need to do this before we add in the new DC and remove the
old DCs.
Note that this is only a problem for offline backups. The online/rename
backups are received over DRS, and as part of the replication they
receive the latest replUpToDateVector from the DC being backed up.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
It will be easy to forget that the backupType marker doesn't exist on
v4.9. However, this seems like a dumb reason not to support v4.9
backup-files. Add a wrapper function to avoid potential problems
cropping up in future.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
We are starting to hit restore cases that are only applicable to a
particular type of backup. We already had a marker to differentiate
renames, but differentiating offline backups would also be useful.
Note that this raises a slight compatibility issue for backups created
on v4.9, as the marker won't exist. However, it's only offline backups
we will use this marker for (at the moment), and this option doesn't
exist on v4.9, so there's no problem.
Removing the markers has been refactored out into a separate function to
handle the optional presence of the new marker.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
We noticed that offline backups were missing a replUpToDateVector for
the original DC, if the backup was taken on a singleton DC. This patch
adds an assertion to the existing test-cases to highlight the problem.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
We need to examine the contents of PYTHON env variable which should defined the
python version to be used when running tests.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Python 3.4 seems to need a string
parsed = json.loads (out_jsobj)
File "/usr/lib/python3.4/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
however Python 3.5 seems to be happy to consume bytes (or string)
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Testing a string against an int value is illegal and
is not necessary in this case, this patch removes the
problematic test.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Make sure either the output of tests and/or the item we are searching match
in type. Output of cmd in python3 is bytes, depending on the was the test is
written it may be easier just to convert all output or just a single string
that is used in the test
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Change all instance where python scripts are called so that the
correct python version as specified by $PYTHON is used
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
mdb_copy() was dutifully checking the PATH for the mdb_copy executable,
then, if it didn't find it, blindly proceeding anyway and trying to run
a non-existent executable. This resulted in a cryptic error:
ERROR(<type 'exceptions.OSError'>): uncaught exception - [Errno 2] No
such file or directory
Add in an extra check that we actually find the executable and raise a
better human-readable exception if we don't.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Fri Nov 9 21:07:47 CET 2018 on sn-devel-144
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Nov 8 11:03:11 CET 2018 on sn-devel-144
generate_users_and_groups() now generates the machine acounts as well as
the user accounts, so it seems there's no need to also have
generate_traffic_accounts(), which does the same job.
Instead, we can just pass through the number of machine acounts to
generate_users_and_groups() and delete the other function.
Also updated generate_users_and_groups() so that machine_accounts is
no longer optional (we want to create machine accounts in all cases).
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Generate separate machine accounts for populating a large DB vs
replaying network traffic.
We want to use different userAccountControl flags in each of the above
cases (i.e. commit 3338a3e257). However, this means that once you
use the --generate-users-only option, you can't replay network packets
against the machine accounts.
We can avoid this problem by creating separate machine accounts for each
of 2 different cases, e.g. STGM-0-x machines for traffic-replay, and
PC-0-x machines for padding out the database.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The traffic_replay group/user/machine account names follow a standard
format. This adds a function to generate the machine-name. It also makes
sure the existing user_name() function gets called in all applicable
places.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
We create machine accounts for 2 different purposes:
1). For traffic generation, i.e. testing realistic network packets.
2). For generating a realistic large DB.
Unfortunately, we want to use different userAccountControl flags for
the 2 different cases. Commit 3338a3e257 changed the flags used
for case #2, but this breaks case #1.
The problem is generate_users_and_groups() is called in both cases,
so we want the 'traffic account' flag passed into that function.
This ensures that the machine accounts get created with the appropriate
userAccountControl flags for the particular case you want to test.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
I was assuming that generate_users_and_groups() only gets called in the
--generate-users-only case. However, it also gets called in the default
traffic replay case.
This patch reworks the code so that the number of machine accounts to
create gets passed in, and the 'create 25% more computers than users'
assumption only applies to the --generate-users-only case.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
A few of the gpo commands use an identical temporary directory structure
that can be constructed using shared code.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
req.more_flags only exists for v10 requests, so we throw an exception if
we try to dereference that field on a v8 (or v5) request. Unfortunately,
we were checking that we support v10 *after* we had tried to access the
more_flags. This patch fixes up the order of the checks.
This may be a problem trying to replicate with an older Windows DC
(pre-2008R2), and was reported on the samba mailing-list at one point:
https://lists.samba.org/archive/samba/2018-June/216541.html
Unfortunately this patch doesn't help the overall situation at all (the
join will fail because we can't resolve the link target and we can't use
GET_TGT). But it now gives you a more meaningful error, i.e.
ERROR(runtime): uncaught exception - (8639, "Failed to process 'chunk'
of DRS replicated objects: DOS code 0x000021bf"
instead of:
ERROR(<type 'exceptions.AttributeError'>): uncaught exception -
'drsuapi.DsGetNCChangesRequest8' object has no attribute 'more_flags'
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Tim Beale <timbeale@samba.org>
Autobuild-Date(master): Tue Nov 6 07:15:33 CET 2018 on sn-devel-144
Tweak the code slightly to avoid some 80+ character lines.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The LDAP connection can also timeout when trying to join a Windows DC
with a very large database. However, in this case Windows gives a
slightly different error message (NT_STATUS_CONNECTION_RESET instead of
NT_STATUS_CONNECTION_DISCONNECTED).
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13612
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
In python2 we decode str types in load_xml, in python3 these are
str class(s) which we cannot decode.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
In python2 as far as I can see GptTmplInfParser.write_binary more
or less works by accident.
write_binary creates a writer for the 'utf8' codec, such a writer
should consume unicode and emit utf8 encoded bytes. This writer
is passed to each of the sections managed by GptTmplInfParser as
follows
def write_binary(self, filename):
with codecs.open(filename, 'wb+',
self.encoding) as f:
for s in self.sections:
self.sections[s].write_section(s, f)
And each section type itself is encoding its result to 'utf-16-le'
e.g.
class UnicodeParam(AbstractParam):
def write_section(self, header, fp):
fp.write(u'[Unicode]\r\nUnicode=yes\r\n'.encode(self.encoding)
But this makes little sense, it seems like sections are encoded to one
encoding but the total file is supposed to be encoded as ut8??? Also
having an encoding per ParamType doesn't seem correct.
Bizarely in PY2 this works and it actually encodes the whole file as utf-16le
In PY3 you can't do this as the writer wants to deal with strings not bytes
(after the extra encode phase in 'write_section'.
So, changes here are to remove the unnecessary encoding in each 'write_section'
method, additionally in GptTmplInfParser.write_binary the
codecs.open call now uses the correct codec (e.g. 'utf-16-le') to write
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Fixes:
1) various ldb.bytes that should be displayed as strings in PY3
2) sorting of lists of xml Element in PY3
3) various 'open' need to be opened in binary mode (to accept binary
data)
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
The previous version here was using UnicodeReader which was
wrapping the UTF8Recoder class and passing that to csv.reader.
It looks like the intention was to read a bytestream in a
certain encoding and then reencode it to a different encoding.
And then UnicodeReader creates unicode from the newly encoded stream.
This is unnecssary, we know the encoding of the bytesstream and
codec.getreader will happily consume the bytstream and give back
unicode. The unicode can be fed directly into csv.writer.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Fixes
1) use compat versions of ConfigParser and StringIO
2) fix sort list of XML Elements
3) open file needs to be opened in binary mode as write_pretty_xml
routing uses BytesIO() object.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Although this is unintuitive it's because we are writing unicode
not bytes (both in PY2 & PY3). using the 'b' mode causes an error in
PY3.
In PY3 we can define the encoding, but not in PY2.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
'file' no longer exists in PY3 replace with 'open'
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Fixes.
1) sorting of xml.etree.ElementTree.Element, in PY2 sort
seems to sort lists of these. In PY3 this no longer works.
Choosing tag as the sort key for py3 so at least in python3
there is a consistent sort (probably won't match how it is
sorted in PY2 but nothing seems to depend on that)
2) md5 requires bytes
3) tostring returns bytes in PY3, adjust code for that
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Due to the userAccountControl flags we were specifying, the machine
accounts were all created as critical objects. When trying to populate
1000s of machine accounts in a DB, this makes replication unnecessarily
slow (because it has to replicate them all twice).
This patch changes it so when we're just creating machine accounts for
the purpose of populating a semi-realistic DB, we jsut use the default
WORKSTATION_TRUST_ACCOUNT flag.
Note that for the accounts used for traffic-replay, we apparently need
the existing flags in order for the DC to accept certain requests.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Tim Beale <timbeale@samba.org>
Autobuild-Date(master): Mon Nov 5 03:43:24 CET 2018 on sn-devel-144
Currently the tool only generates the machine accounts needed for
traffic generation. However, this isn't realistic if we're trying to use
the tool to generate users to simulate a large network.
This patch generates machine accoutns along with the user accounts.
Note we assume there will be more computer accounts than users in a real
network (e.g. work laptops, servers, etc), so generate slightly more
computer accounts.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
When creating 1000s of users you currently get a lot of debug, but at
the same time you have no idea how far through creating the users you
actually are.
Instead of logging every single user account that's created, log every
50th (as well as how far through the overall generation we are).
Logger already includes timestamps, so we can remove generating the
timestamp diff manually. User creation is the slowest operation - adding
groups/memberships is much faster, so we don't need to log as
frequently.
Note that there is a usability trade-off on how frequently we log
depending on whether the user is using the slower (but more common)
method of going via LDAP, vs the much faster (but more obscure) method
of writing directly to sam.ldb with ldb:nosync=true. In my tests, we end
up logging every ~30-ish secs with LDAP, and every ~3 seconds with
direct file writes.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Using logger is more helpful here because it includes timestamps, so we
can see how long things are taking. It's also more consistent with the
rest of the traffic_replay logging.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Each user-group membership was being written to the DB in a single
operation. With large numbers of users (e.g. 10,000 in average 15 groups
each), this becomes a lot of operations (e.g. 150,000). This patch
reworks the code so that we write the memberships for a group in
one operation. E.g. instead of 150,000 DB operations, we might make
1,500. This makes writing the group memberships several times
faster.
Note that rthere is a performance vs memory tradeoff. When we hit
10,000+ members in a group, memory-usage in the underlying DB modify
operation becomes very inefficient/costly. So we avoid potential memory
usage problems by writing no more than 1,000 users to a group at once.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
We can speed up writing the group memberships by adding multiple users
to a group in a single DB modify operation.
To do this, we first need to reorganize the assignments so instead
of being a set of tuples, it's a dictionary where key=group and
value=list-of-users-in-group.
add_users_to_groups() now iterates through the users/groups slightly
differently, but mostly it's just indentation changes. We haven't
changed the number of DB operations yet - we'll do that in the next
patch.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
When adding 10,000 users, one user would end up in over 1000 groups.
With 100,000 users, it would be more like 10,000 groups. While it makes
sense to have groups with large numbers of users, having a single user
in 1000s of groups is probably less realistic.
This patch changes the shape of the Pareto distribution that we use to
assign users to groups. The aim is to cap users at belonging to at most
~500 groups. Increasing the shape of the Pareto distribution pushes the
user assignments so they're closer to the average, and the tail (with
users in lots of groups) is not so large).
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
The current probability we were assigning to users roughly approximates
the Pareto Distribution (with shape=1.0). This means the code now uses a
documented algorithm (i.e. explanation on Wikipedia). It also allows us
to vary the distribution by changing the shape parameter.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
When assigning 10,000 users to 15 groups each (on average),
assign_groups() would take over 30 seconds. This did not include any DB
operations whatsoever. This patch improves things, so that it takes less
than a second in the same situation.
The problem was the code was looping ~23 million times where the
'random.random() < probability * 10000' condition was not met. The
problem is individual group/user probabilities get lower as the number
of groups/users increases. And so with large numbers of users, most of
the time the calculated probability was very small and didn't meet the
threshold.
This patch changes it so we can select a user/group in one go, avoiding
the need to loop multiple times.
Basically we distribute the users (or groups) between 0.0 and 1.0, so
that each user has their own 'slice', and this slice is proporational to
their weighted probability. random.random() generates a value between
0.0 and 1.0, so we can use this to pick a 'slice' (or rather, we use
this as an index into the list, using .bisect()). Users/groups with
larger probabilities end up with larger slices, so are more likely to
get picked.
The end result is roughly the same distribution as before, although the
first 10 or so user/groups seem to get picked more frequently, so the
weighted-probability calculations may need tweaking some more.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This doesn't change functionality at all. It just moves the probability
calculations out into separate functions.
We want to tweak the logic/implementation behind this code, but the
rest of assign_groups() doesn't really care how the underlying
probabilities are worked out, so long as it gets a suitably random
user/group membership each time round the loop.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Wrap up the group assignment calculations in a helper class. We're going
to tweak the internals a bit in subsequent patches, but the rest of the
code doesn't really need to know about these changes.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
In this code
def f(a, b=[]):
b.append(a)
return b
all single argument calls to f() will affect the same copy of b.
In the controls case, controls=None has the same effect as
controls=[].
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
When given a list, it will use the list directly as an argument list,
avoiding shell-expansion and the intermediatory process.
This removes shell expansion trouble, and saves the machine a little
bit of work.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Thu Nov 1 09:40:02 CET 2018 on sn-devel-144
This was unused and broken. e.g. here:
- def init(self):
- # Check to see that this 'existing' LDAP backend in fact exists
- ldapi_db = Ldb(self.ldapi_uri)
there is no attribute self.ldapi_uri, so this would always raise an
exception.
It was being left around in case it became useful, but that doesn't
seem to be happening.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
There were 2 % formats and 3 arguments.
Also reformat for line length
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
previously these would have raised an exception
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
but str(e) is the same as str(e.message), so we can use that
on 2 and 3.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
We were skipping a level in the inheritance chain, which had no effect
in this case (no .setUps or .tearDowns were missed) but it would be
confusing if the parents ever changed.
Note: in python 3, you just call super() with no args, and it works
out the right thing.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
With large domains it's hard to get an idea of how many groups there
are, and how many users are in each group, on average. However, this
could have a big impact on whether a problem can be reproduced or not.
This patch dumps out some summary information so that you can get a
quick idea of how big the groups are.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Wed Oct 31 03:40:41 CET 2018 on sn-devel-144
This adds an easy way for users to see (via samba-tool) how many members
are in various groups, without querying the members for each individual
group.
For example, you could pipe this output to grep to check for groups with
zero or one members (i.e. historic groups that may no longer make
sense).
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Currently the online/rename backup files always use the default backend
(TDB) and there is no way to change this.
This patch adds the backend-store option to the backup commands so that
you can create a backup with an MDB backend, if needed.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This reduces noise, so the messages only come out if you specify
--debug.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
This makes it easier to debug dbcheck problems.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
This is not always going to work, and is not guaranteed to be
consistent even between minor versions.
Here is a simple counterexample:
>>> a = 'hello'
>>> a is 'hello'
True
>>> a is 'hello'.lower()
False
>>> a == a.lower()
True
Possibly it always works for the empty string, but we cannot rely
on that.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Mon Oct 29 23:13:36 CET 2018 on sn-devel-144
Python's property() function works like this:
property([getter[, setter[, delete[, doc]]]])
but we have been forgetting the delete function, or rather setting it
to be a string. A string is not callable and is unlikely to succeed at
deleting the property.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <npower@samba.org>
Autobuild-User(master): Noel Power <npower@samba.org>
Autobuild-Date(master): Fri Oct 26 00:50:37 CEST 2018 on sn-devel-144
This test will be revealed to the world in the next commit.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <npower@samba.org>
Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <npower@samba.org>
The 'from_xml()' definition is replaced a few lines down
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <npower@samba.org>
The error is in the value, and StandardError is not in Python 3
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <npower@samba.org>
We had our own special version with very few entries, but only
used it in one place.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <npower@samba.org>
Strangely the test was failing on CI only, looks like there is an
issue with order of elements returned from dict.items() with python3.4
(version of python in CI docker instance) and python3.6 (version on my
development machine). Changed code to sort the keys so order of help
printed out should be the same for each invocation.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Enclose filter with list as filter object has no 'len' method
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
part of PY3 port samba4.blackbox.upgrade.samba3-upgrade*
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
part of PY3 port samba4.blackbox.upgrade.samba3-upgrade*
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Fix attributes that need to be treated as str not bytes
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Getting Exception: must be str, not ldb.bytes error in scheme_upgrade
phase of test
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
o Fix various ldb attribute that need to be converted to string
o dict has no 'has_key' method
o ndr_unpack needs bytes not string
o b64encode needs bytes (so open file with binary mode)
o StandardError was removed in python3 use Exception instead
o fix octal literals
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* Various string related fixed py2/py3
* Fix strange double decode followed by encode code (see comments
in change)
* Added dump_attr_values, simply printing attribute values (from
sequence) doesn't work (when using string '%s' format codes in
existing string). We need to print out string from bytes in PY3
and fallback to repr(which will print b'blah') if we get a decode error
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Really strange bug caused by map being updated while being iterated.
This caused keys to be skipped and inconsistent and incorrect
results from ldapcmp.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Fixes
1) use compat versions of ConfigParser and StringIO
2) open file needs to be opened in binary mode as write_pretty_xml
routine uses BytesIO() object.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
ConfigParser module changed name to configParser in PY3, additionally
the behaviour regarding interpolation has changed. ConfigParser now
has a default interpolation param whose behaviour demands that '%' is
escaped. To maintain behaviour with the python2 version this default
param needs to be changed. Add some alias(s) and 'shim' Configparser
symbol in samba.compat to cater for this.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Previously the py2 api for strcasecmp_m/strstr_m required strings/unicode
but couldn't actually handle unicode with anything other than the default
encoding (e.g. ascii). The c-api as been fixed and the encoding steps
(which were unnecessary and causing errors in PY3) have been removed
py_strcasecmp_m & py_strstr_m use PyArg_ParseTuple() with 's' which
in Py2 tries to decode string with the default (e.g. ascii) encoding
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Python3 no longer has a long type so the 'L' postfix is no
longer valid. Additionally in python2 an int that exceeds will
be transparently converted into a long when necessary
Ensure python scripts are called with the python version that
is defined by $PYTHON
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Can't sort a dict view, create a list from view then use list.sort
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
map in python3 returns an iterable, in python2 it returned
a list. Iterable has no sort method, use sort function instead or
construct a list from the iterable so you can use list.sort
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
When creating a new user and specifying the local file path of the
sam.ldb DB, it's possible to create an account that you can't actually
login with.
This commit contains tests to verify the bug.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13653
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add the random_password method to the BlackboxTestCase class and remove
duplicated copies from other test cases. Also use SystemRandom so that
the generated passwords are more cryptographically sound.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Oct 18 10:02:19 CEST 2018 on sn-devel-144
Add cmd to print uptodateness summary with json support.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Get utdv summary from distances matrix and support attr filters.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
We need to reuse it in drs cmd.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
To avoid returning 2 values from get_utdv_distances.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Extract function to reuse later.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Move methods from cmd_uptodateness to new module.
Will reuse in drs cmd later.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Both visualize and drs cmd will have uptodateness functions.
Create a new module to reuse code.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Joining a large DB can take so long that the LDAP connection times out.
The previous patch fixed the 'happy case' where the join succeeds.
However, if the commit or replication fails (throwing an exception),
then the cleanup code can also fail when it tries to delete objects from
the remote DC. This then gives you an error pointing to
cleanup_old_accounts() rather than what actually went wrong.
This patch adds a sanity-check that if the join fails, that the LDB
connection to the remote DC is still alive, before we start deleting
objects.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13612
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
We check for the default base DN as this does require authentication, but
we do not need to search for more than just that (so use SCOPE_BASE) and
we need no attributes, so ask for none
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
When joining a very large domain (e.g. 100K users), the replication can
take so long that the LDAP connection to the remote DC times out.
This patch avoids the problem by adding in a sanity-check after the
replication finishes that the LDB connection is still alive. If not,
then we reconnect.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13612
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
drs_Replicate.replicate() could consume a large amount of memory when
replicating a large DB. This is not a leak - the memory gets freed when
the function returns (i.e. once the partition is fully replicated).
However, while the partition is in the process of being replicated, it
accumulates memory for each replication chunk it receives. This can have
considerable overhead with 1000s of objects/links in the partition.
This was exhausting memory when joining a VM with 1Gb RAM to a DC with
25K users (average ~15 group memberships per user).
It seems that by storing a reference to something that's on the ctr's
talloc tree, it doesn't free up the memory for each ctr message (until
the function actually returns and req is destroyed).
With 10K users (and average 15 group memberships per user), .replicate()
consumed 211Mb of memory, according to talloc.report_full(). With this
patch, it goes down to just the current ctr message (1-2Mb).
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Oct 17 08:56:42 CEST 2018 on sn-devel-144
This should not happen, but it does sometimes in an autobuild
environment. Rather than reporting this by crashing, we report it by
showing there is no DNS name.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Fri Oct 12 15:27:07 CEST 2018 on sn-devel-144
The code to expunge tombstones uses long(), which is not Python3
compatible. Python3 uses int() instead, and works out how big it needs
to be.
As long as we don't run the samba-tool command on a 32-bit machine
after the year 2038, then we should avoid any integer overflow on
Python 2.x.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
A filled session info is needed by some vfs modules, e.g. full_audit.
Signed-off-by: Björn Baumbach <bb@sernet.de>
Reviewed-by: Volker Lendecke <vl@samba.org>
This is an example of how to use the new logger.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
By doing this, we don't need to repeat the log level convert code any more.
Also, logs have colors now.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
These print are actually progress infomation, should use logger to
print to stderr, other than stdout.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
We need a consitent way for logging in Samba Python code.
This module provides a factory method `get_samba_logger` to create logger,
with a reasonable default format and optional color.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
When we run `traffic_replay --generate-users-only`, if we cancel it or
it breaks in middle, it won't do anything when we try to run it again.
This is because the code will check the first user/group to create. If
it's already there, then it thought task already done, and break the loop.
This commit change the behavior:
We search existing users/groups first, skip existing ones, and
create non-existing ones. So we can run it multi-times to make sure the
expected users and groups are actually created.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
Autobuild-User(master): Aurélien Aptel <aaptel@samba.org>
Autobuild-Date(master): Mon Oct 8 21:25:59 CEST 2018 on sn-devel-144
This option forces the reapplication of policy,
and works the same as MS 'gpupdate /force'
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
This returns a list of guids for gpos applied
plus settings applied and their previous values.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
The apply_map function should not be a requirement
to implement the gp_ext class, since only the
gp_sec_ext uses it now.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
We don't get this far if we are not a DC, and if somehow we do the
errors will be no more informative due to this special case.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
Policy specific setting application should be
handled by the group policy extension, not the
read/parse handler.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
MS spec describes the policy callback as a
function called ProcessGroupPolicy which accepts
a pDeletedGPOList and a pChangedGPOList param.
The Group Policy Client Side Extension then
iterates over the deleted, then the changed gpo
lists and applies/unapplies policy. We should do
this also.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
These functions were added by Luke, but have
never actually done anything. If/when we
read from these *.pol files, we won't need these
separate functions to do it.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
Initialize variables for the gp_ext in the
constructor instead of passing them via the parse
function.
This is a dependency of the "gpo: Implement
process_group_policy() gp_ext func" patch, since
the parse() function is now called by the ext,
instead of by gpupdate within apply_gp(). The
parse() function should only take the path
variable, to simplify writing Client Side
Extensions.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
Whether an extension should apply should be
determined by the extension, not by the
calling script.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
The SamDB is only used by the gp_sec_ext, and
isn't needed elsewhere, so initialize it where
we need it and avoid passing it around
everywhere.
It makes the most sense to put this in the setter
class that uses it, so pass our creds down so we
have access to it, then initialize it there.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
These methods don't need redefined in the child
class.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
Pair-Programmed-With: Andreas Schneider <asn@samba.org>
Signed-off-by: Mathieu Parent <math.parent@gmail.com>
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Tue Oct 2 17:30:29 CEST 2018 on sn-devel-144
That test was already marked as flapping in commit
5a73f904e1.
However the test generates an UNEXPECTED(error)
instead of an UNEXPECTED(failure).
[67(594)/75 at 36m28s] samba.tests.kcc.python3(vampire_dc)
repsFrom source DSA guid (62f01940-a784-4e60-947a-a661c787c8cc) not found
repsFrom source DSA guid (62f01940-a784-4e60-947a-a661c787c8cc) not found
repsFrom source DSA guid (62f01940-a784-4e60-947a-a661c787c8cc) not found
repsFrom source DSA guid (62f01940-a784-4e60-947a-a661c787c8cc) not found
repsFrom source DSA guid (62f01940-a784-4e60-947a-a661c787c8cc) not found
UNEXPECTED(error): samba.tests.kcc.python3.samba.tests.kcc.KCCTests.test_verify(vampire_dc)
REASON: Exception: Exception: Traceback (most recent call last):
File "bin/python/samba/tests/kcc/__init__.py", line 80, in test_verify
attempt_live_connections=False)
File "bin/python/samba/kcc/__init__.py", line 2659, in run
('connected',))
File "bin/python/samba/kcc/__init__.py", line 2513, in plot_all_connections
vertex_colors=vertex_colours)
File "bin/python/samba/kcc/graph_utils.py", line 334, in verify_and_dot
for p, e, doc in errors)))
samba.kcc.graph_utils.GraphError: The 'dsa_final CN=NTDS Settings,CN=LOCALVAMPIREDC,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=samba,DC=example,DC=com' graph lacks the following properties:
connected: the graph is not connected, as the following vertices are unreachable:
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
i.e. protect the client against man-in-the-middle attacks by default.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Ignoring autogenerated .sock files and directories during file system based
offline domain backup.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13604
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Note: Unlike other libraries this library is been only built for the
configured python version. It depends on availability of 'swat' python
module.
The swat module is hosted externally, it seems not to have been modified
for a number of years, I don't think swat is python3 compatabile. These
changes are enough to get allow the samba binary to launch a web server
(which will just display a placeholder page announcing you need to install
swat). It maybe that removing this functionality is what we should do, but
that is a decision that can be made at a later time.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Sep 26 10:40:49 CEST 2018 on sn-devel-144
When a new DC is joined to the domain, samba-tool would automatically
detect an appropriate site for the new DC. However, it only did this if
the --server option wasn't specified. The new DC's site got
automatically updated as part of the finddc() work, however, this step
gets skipped if we already know the server DC to join to.
In other words, if Default-First-Site-Name doesn't exist and you specify
--server in the join, then you have to also specify --site manually,
otherwise the command fails. This is precisely what's happening in the
join_ldapcmp.sh test, now that the backupfromdc testenv no longer has the
Default-First-Site-Name present.
This patch adds a new find_dc_site() function which uses the same
net.finddc() API (except based on the server-address rather than
domain-name). Assigning DEFAULTSITE has been moved so that it only
gets done if finddc() can't determine the site.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
The provision code already defines "Default-First-Site-Name" so we might
as well reuse it.
The join.py already uses a suitable default, so assigning the default in
the domain netcmd code is unnecessary.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Normally when a new DC joins a domain, samba-tool works out the new
DC's site automatically. However, it does this by querying the existing
DC using CLDAP. In the restore case, there is no DC running. We could
still query the DB on disk and work out the correct site based on the
new DC's IP, however:
- comparing between the CN=Subnet DNs and an IP-address string seems
like it'd be non-trivial to write, and
- in the lab-domain rename case, chances are the user will want a
completely different subnet to what's already in the DB.
The restore command now has a --site option so the user can specify an
appropriate site for the restored DC. This patch makes the restore
command work by default (i.e. without a --site option) even if the
default Default-First-Site-Name doesn't exist. Basically the solution is
to just check Default-First-Site-Name exists and create it if it
doesn't. As the recommended workflow is to use the restored DC as a
temporary seed that you'll later throw away, this approach seems
acceptable. Subsequent DCs will then be joined to the running restored
DC, so an appropriate site will be determined using CLDAP. The only
side-effect is potentially an extra Site object.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Add a test-case that exercises the new '--site' restore option and
ensures the restored DC gets added to the correct site.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Restoring a backup only worked if the Default-First-Site-Name site was
still present. When the new restored DC account is created, it was
trying to add the new server's DN under CN=Default-First-Site-Name.
However, if the original domain was setup using a different site, then
the restore would fail because the DN didn't exist.
When running the restore command, you should be able to specify the
site that you want the new/restored DC to be in (same as during a
DC 'join'). Passing the correct --site argument is one way to avoid
this problem. (A subsequent patch will further improve the tool so it
can work around non-default sites automatically).
Note we also need to pass the site through to where the new DNS entries
get registered (in the rename case).
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
