1
0
mirror of https://github.com/samba-team/samba.git synced 2024-12-22 13:34:15 +03:00
Commit Graph

246 Commits

Author SHA1 Message Date
Joe Guo
af0e0c544b script/autobuild.py: rename sdir to test_source_dir
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-05-10 08:19:16 +00:00
Andreas Schneider
f1bf02c78a autobuild: Build also Samba AD with MIT Kerberos
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2019-04-25 10:47:16 +00:00
Lutz Justen
92f30f91e4 waf: build: Respect --disable-python for third_party modules
Skips installation of samba/third_party stuff into the python directory if
--disable-python is set.

Added test after install that confirms no python modules installed.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13905

Signed-off-by: Lutz Justen <ljusten@google.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Apr 24 07:32:31 UTC 2019 on sn-devel-184
2019-04-24 07:32:31 +00:00
Garming Sam
bfd762b53a selftest: rename schemaupgrade_dc (+pair) to schema_dc
This is needed because the name of the autobuild job and
the name of the selftest env end up in the socket path
for ncalrpc sockets.

The challenge is that (for example)
/memdisk/autobuild/fl/b2424063/samba-schemaupgrade/bin/ab/schemaupgrade_pair_dc/ncalrpc/np/protected_storage
does not fit in a struct sockaddr_un.

Signed-off-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Apr 12 05:41:36 UTC 2019 on sn-devel-144
2019-04-12 05:41:36 +00:00
Aaron Haslett
59ee3c864c selftest: split schemaupgrade testenv out
Schemaupgrade tests are particularly resource intensive and are causing
runners to hit their memory and CPU limits, so we need to split them
out.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2019-04-11 04:17:11 +00:00
Douglas Bagnall
dad98d0334 autobuild: attempt authenticated email if environment suggests it
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-04-05 04:41:25 +00:00
Stefan Metzmacher
be6cf83c01 autobuild: try to distribute the tasks a bit more
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2019-02-27 10:16:11 +00:00
Stefan Metzmacher
e0bd12e054 autobuild: spread ad-dc tests over 6 autobuild/ci separate tasks/jobs
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2019-02-27 10:16:11 +00:00
Stefan Metzmacher
54278049e2 autobuild: add samba-ad-member task
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2019-02-27 10:16:11 +00:00
Stefan Metzmacher
2d576c3afc autobuild: run ad_dc_backup tests in samba-ad-dc-backup
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2019-02-27 10:16:10 +00:00
Stefan Metzmacher
1bc2456b87 autobuild: move maptoguest and simpleserver to 'samba-fileserver'
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2019-02-27 10:16:10 +00:00
Stefan Metzmacher
3cf317c9b8 autobuild: move nt4_dc_schannel out of 'samba'
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2019-02-27 10:16:10 +00:00
Stefan Metzmacher
13fe139fb2 selftest:Samba4: add ad_dc_backup alias to ad_dc
This will allow us to run really most tests in an isolated
autobuild/ci task later.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2019-02-27 10:16:10 +00:00
Stefan Metzmacher
c217a15a2c selftest:Samba4: add ad_dc_default alias to ad_dc_ntvfs
This will allow us to run really most tests in an isolated
autobuild/ci task later.

This will apply to tests, which may not rely on the ntvfs backend, so
the ad_dc_default alias can point to another environment in future.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2019-02-27 10:16:10 +00:00
Stefan Metzmacher
c82b60c827 selftest:Samba4: add ad_dc_slowtests alias to ad_dc_ntvfs
This will allow us to run really slow tests in an isolated
autobuild/ci task later.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2019-02-27 10:16:10 +00:00
Stefan Metzmacher
62eeab8f6c selftest:Samba4: add fl2008dc as alias to ad_dc_ntvfs
Using aliases it will be possible to split the large amount
of tests which use ad_dc_ntvfs into multiple autobuild/ci
tasks/jobs later.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2019-02-27 10:16:10 +00:00
Tim Beale
96472306bf selftest: Add basic sanity-check tests for nopython target
Previously we were only checking samba compiled OK with
--disable-python, not that it actually ran.

The main problem is all the make test framework is based around
subunit/smbtorture, neither of which we seem to build with
disable-python. However, for just a simple sanity-check, we can just
bypass all the subunit-filter work and just call the Perl code directly.
This works OK as long as it's just simple shell script tests that we're
running, as we can check the script's exit code directly.

The main thing that we really want to test is that we can start up the
smbd testenv and connect to it (i.e. a simple smbclient test).

This patch adds a new 'make test-nopython' target. This disables the
subunit filtering, and runs a small test-list that was generated manually.

Note that currently this has the limitation that it doesn't support known
failures or flapping tests. However, just checking that smd starts up OK
is probably OK for now.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Feb 20 02:10:00 CET 2019 on sn-devel-144
2019-02-20 02:10:00 +01:00
Tim Beale
63ea86804f autobuild: Drop 'py2' flag
This isn't used any more. It was only being set, never referenced.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-02-15 04:35:23 +01:00
Tim Beale
689a1ee747 autobuild: Replace samba-buildpy2-only with samba-nopython-py2
For Samba 4.11, the minimum python2 functionality we will support (for
now, at least - we may change our minds) is for the --disable-python
target, i.e. if you're excluding all the python functionality from
samba, then WAF should still support being built with python2.

The use case here is old unix platforms that want to use smbd, but don't
have python3 support.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-02-15 04:35:23 +01:00
Tim Beale
e11969def1 autobuild: Remove ${EXTRA_PYTHON} variable
We no longer build the python2 bindings, only python3. So we can get rid
of this variable now.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-02-15 04:35:23 +01:00
Tim Beale
9124e44d71 autobuild: Tidy up unnecessary line-breaks in 'TESTS='
Now that we've dropped the {PY3_ONLY} variable, there's no need for
line-breaks in some of the 'TESTS=' values. We can tidy this up a bit.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-02-15 04:35:23 +01:00
Tim Beale
8c5a5a58c4 autobuild: Remove the PY3_ONLY variable
This variable is no longer needed as all the tests run using python3
now.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-02-15 04:35:22 +01:00
Tim Beale
6ba99c3672 autobuild: Update variable name to make more sense
When we switched from python2 being the default to python3, we didn't
update this variable name. It's now handling the python2 case, but it's
a boolean flag named 'py3', which is rather confusing.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-02-15 04:35:22 +01:00
Tim Beale
5f3beb6e41 autobuild: Drop py2 autobuild jobs
Samba v4.11 will no longer support python2, so let's drop the autobuild
jobs. This will save some gitlab/sn-devel time and money, as it's less
work for CI to do.

Note that this highlights some previous inconsistencies:
- samba-none-env-py2 was being built for gitlab but not sn-devel.
- samba-nt4-py2 was being built for sn-devel but not gitlab

I've left samba-buildpy2-only for now, which will be addressed in a
subsequent patch.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-02-15 04:35:22 +01:00
Stefan Metzmacher
35d66610ca script/autobuild.py: add 'lsb_release -a' and 'mount' to system-info.txt
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2019-02-14 02:18:29 +01:00
Ralph Boehme
b1740f3baf CI: split out "samba-ad-dc-ntvfs[-py2]" test targets
Many AD tests currently use the "samba" target. Split out a new target
"samba-ad-dc-ntvfs" and have all tests that use the "ad_dc_ntvfs" env
use the new target. This should greatly speed up the runtime for the "samba"
target and avoid swapping.

This reduces the total CI time by ~ 55%, I got an autobuild and a gitlab
pipeline finished in just ~ 100 mins!

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Ralph Boehme <slow@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Mon Feb 11 14:10:12 CET 2019 on sn-devel-144
2019-02-11 14:10:12 +01:00
Tim Beale
95b2c9d775 autobuild: Split backup/restore testenvs out into separate job
The samba-ad-dc-2 job was reaching its limits with the number of
testenvs and what the resource-limited CI machines can handle.
Samba processes were getting swapped out of memory, causing CI runs
to fail.

This patch splits the backup/restore testenv targets into a separate
autobuild job: samba-ad-dc-backup.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Feb  5 12:23:31 CET 2019 on sn-devel-144
2019-02-05 12:23:31 +01:00
Douglas Bagnall
35e1a7989d autobuild: convert top_commit_msg to str for Py 3
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-12-12 04:38:13 +01:00
Douglas Bagnall
25ab429371 autobuild: py3: cope with bytes when compiling system-info.txt
The command output looks like b'foo\nbar' in string-space.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-12-12 04:38:13 +01:00
Noel Power
bc92277a79 autobuild: We should run autobuild with python3 by default
sn-devel autobuild runing autobuild.py (via git hooks I suppose) but
if run directly (e.g. depending on script shebang) then 'python' aka
python2 will run. This will cause an error when building some targets
because the autobuild script itself sometimes builds paths based
on the version of python executing the script e.g ${PYTHON_PREFIX}.

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-12-10 10:38:26 +01:00
Noel Power
b0b9c62e1c autobuild: Convert old py3 tasks to py2
Now that we are building with python3 by default we need to
convert the old python3 test tasks to python2 (e.g. reverse how
we used do it)

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-12-10 10:38:25 +01:00
Noel Power
dfe5787c25 autobuild: Adjust autobuild for PY2/PY3 get_python_lib behaviour
The resuls of get_python_lib are different between python2 & python3
and this results in autobuild generating the wrong PYTHONPATH with
python3.

python2
=======
print ("%s" % get_python_lib(standard_lib=1, prefix='/my/prefix'))
/my/prefix/lib64/python2.7

python3
print ("%s" % get_python_lib(standard_lib=1, prefix='/my/prefix'))
/my/prefix/lib/python3.6

But with addition of plat_specific param the results are the same

python2
=======
print ("%s" % get_python_lib(plat_specific=1, standard_lib=0, prefix='/my/prefix'))
/my/prefix/lib64/python2.7/site-packages

python3
=======
print ("%s" % get_python_lib(plat_specific=1, standard_lib=0, prefix='/my/prefix'))
/my/prefix/lib64/python3.6/site-packages

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-12-10 10:38:25 +01:00
Noel Power
ae5dd18162 autobuild: Modify old samba_buildpy3_only job to python2
Since autobuild now builds python3 by default we need to change
the previously buildpy3 only job to python2

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-12-10 10:38:25 +01:00
Noel Power
9bdeb95f2c autobuild: Remove temporary purepy3 tasks
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-12-10 10:38:24 +01:00
Noel Power
61adb75b9f CI: Add new (TEMP) pure python3 autobuild jobs for samba-build & samba-nt4
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-12-10 10:38:24 +01:00
Noel Power
fd87c89c64 script: Add new (temporary) pure python3 ad-dc test
Ideally we want all the tests to run under python3 by default (no
special task for this) and then convert the existing '-py3' tasks
to run the python tests with python3.
However at the moment the convertion process is not ready to do this,
for a while we need to run separate autobuild tasks for this.

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-12-10 10:38:22 +01:00
Andreas Schneider
a9477eb3bd autobuild: Add _FORTIFY_SOURCE=2 to the -O3 build
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-11-28 23:19:21 +01:00
Noel Power
2ca3b3ebd8 script: Add new (temporary) pure python3 ad-dc-2 test
Ideally we want all the tests to run under python3 by default (no
special task for this) and then convert the existing '-py3' tasks
to run the python tests with python3.
However at the moment the convertion process is not ready to do this,
for a while we need to run separate autobuild tasks for this.

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-11-14 05:07:17 +01:00
Douglas Bagnall
5e79630822 autobuild: remove unused variables
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
2018-11-01 05:08:15 +01:00
Noel Power
d357141546 CI/Autobuild: Remove samba-none-env-py3 test
We now run a purepython3 none-env test, later when the whole
build is running under python3 we will resurrect this job
but as (samba-none-env-py2) for python2

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Oct 23 09:10:22 CEST 2018 on sn-devel-144
2018-10-23 09:10:22 +02:00
Noel Power
f8b5dd9544 script: Add new (temporary) autobuild task for none-env tests.
Ideally we want all the tests to run under python3 by default (no
special task for this) and then convert the existing '-py3' tasks
to run the python tests with python3.
However at the moment the convertion process is not ready to do this,
for a while we need to run separate autobuild tasks for this.

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-10-23 05:50:29 +02:00
Noel Power
de1e6d7856 script: add new autobuild task for building pure python3
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-09-28 01:01:23 +02:00
Philipp Gesang
c98f997192 turn --with-json-audit into global --with-json
Fold the build option --with-json-audit into the toplevel wscript
to reflect the fact that JSON support is no longer local to the
audit subsystem.

Signed-off-by: Philipp Gesang <philipp.gesang@intra2net.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-09-26 20:41:07 +02:00
Alexander Bokovoy
8f022a0be8 script/autobuild: Fix formatting in send_email
Commit cb40e2bbc8 introduced a print
statement with a broken formatting. Reported by pylint.

Signed-off-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-09-05 06:37:28 +02:00
Alexander Bokovoy
67ed1ea6c4 script/autobuild: re-use CACHE_SUFFIX from waflib
Signed-off-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-09-05 06:37:28 +02:00
Thomas Nagy
5967335387 script/autobuild: use --out instead of -b when calling configure
Signed-off-by: Thomas Nagy <tnagy@waf.io>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-09-05 06:37:21 +02:00
Thomas Nagy
9aa8f2badd build:wafsamba: detail where we are processing the autobuild
Signed-off-by: Thomas Nagy <tnagy@waf.io>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-09-05 06:37:21 +02:00
Andrew Bartlett
825d67fd35 autobuild: Avoid subshell for tail -f invocation
This should mean one less process in the process tree, and less places to hold
FDs open.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13591

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Aug 29 08:20:55 CEST 2018 on sn-devel-144
2018-08-29 08:20:55 +02:00
Andrew Bartlett
47c14ef64e autobuild: use close_fds=True to avoid *.stderr and *.stdout inheriting into every process
This closes fds other than 0, 1, 2.

This ensures only the correct *.stderr and *.stdout is attached, via
the stdout/stderr parameter to Popen(), but not every other FD
currently open in python at the time Popen is called.

For the tail invocation and other calls to Popen(), because fds 0, 1,
2 are still attached, these function as before.

Per https://docs.python.org/2.6/library/subprocess.html:

"If close_fds is true, all file descriptors except 0, 1 and
2 will be closed before the child process is executed. (Unix only)."

And regarding the passed in parameters:

"stdin, stdout and stderr specify the executed programs’ standard
input,
standard output and standard error file handles, respectively.  "
...

"With None (the default), no redirection will occur;
the child’s file handles will be inherited from the parent. "

(The unwanted inherited files would be on a random high FD, where the
program wouldn't know what to do with them, but counting towards the
process FD limit).

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13591

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-08-29 05:28:17 +02:00
Andrew Bartlett
6b11643617 autobuild: Move backup/restore environments into to samba-ad-dc-2 autobuild job
This tries to to split up the tasks more evenly and may help with the python3
work by isolating them from the long samba job.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
2018-08-29 05:28:16 +02:00