sin/samba-mirror
1
0
Fork 0
mirror of https://github.com/samba-team/samba.git synced 2024-12-22 13:34:15 +03:00
Code

selftest: Catch error codes from failing testsuites

Browse Source 
Testsuites declared with functions such as plantestsuite() are not run
directly, but are piped through filter-subunit. The overall exit code of
the executed test command is that returned by the last command in the
pipeline (that is, filter-subunit), and thus the actual testsuite return
code is lost.

A real consequence of this is that an error in setUpClass() in a Python
testsuite causes the whole testsuite to be skipped silently.

The --fail-on-empty option partially addressed this, but didn't help if
the testsuite contained multiple test classes, only one of which
contained an error.

We now use bash with the pipefail option, which makes the return code of
the last failing command into the return code of the entire pipeline.
That means that we properly fail if a testsuite returns a non-zero exit
code, but doesn't otherwise exhibit any failures in its output.

This doesn't help for cases where a testsuite has other failing tests
that become xfails due to knownfail entries. In that case, the overall
'testsuite-failure' will be turned into 'testsuite-xfail' by
filter-subunit and the silent failures will remain unheeded. Still, this
is better than the existing situation.

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Wed Apr 12 14:57:55 UTC 2023 on atb-devel-224
This commit is contained in:
Joseph Sutton 2023-04-06 16:31:09 +12:00 committed by Andreas Schneider
parent 62893486c6
commit 2ff55b3da7
1 changed files with 5 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
selftest/selftest.pl
View File

@ -129,7 +129,11 @@ sub run_testsuite($$$$$)
Subunit::start_testsuite($name);
Subunit::progress_push();
Subunit::report_time();
system($cmd);
# Enable pipefail so that we catch failing testsuites that are part of a
# pipeline (typically, piped through filter-subunit). This won't catch
# any testsuite failures that are turned into testsuite-xfails by
# filter-subunit.
system("bash", "-o", "pipefail", "-c", $cmd);
Subunit::report_time();
Subunit::progress_pop();