1
0
mirror of https://github.com/samba-team/samba.git synced 2024-12-23 17:34:34 +03:00
Commit Graph

79 Commits

Author SHA1 Message Date
Joseph Sutton
421dc7fc4d python:subunit: Avoid misleading "Test was never started" error message
subunithelper.py keeps track of tests that have been started, and
displays an error message if a test reports an outcome without having
previously been started. However, it makes the assumption that a test
has finished once it has reported a single outcome. This means that a
misleading error message will be displayed if it receives multiple
outcomes from the same test (which can happen if a test using the Python
unittest framework does not complete successfully, and the cleanup
subsequently fails), and any actual errors from the cleanup remain
undisplayed.

This commit ensures that only a single outcome is reported for each
test, and only after the test has finished. Outcomes are buffered up
until the stopTest() function is called, when a single outcome is
determined and all errors received for that test are output.

FilterOps still needs to output test outcomes immediately rather than
buffering them, otherwise they are never picked up and passed on to the
remote test case by subunithelper.parse_results(). This would result in
an error as the test would be considered to have never finished.

    Example subunitrun output before the change:

time: 2021-04-28 01:28:49.862123Z
test: samba.tests.example.ExampleTests.test
time: 2021-04-28 01:28:49.862215Z
failure: samba.tests.example.ExampleTests.test [
Traceback (most recent call last):
  File "bin/python/samba/tests/example.py", line 28, in test
    self.fail()
AssertionError: None
]
time: 2021-04-28 01:28:49.862407Z
failure: samba.tests.example.ExampleTests.test [
Traceback (most recent call last):
  File "bin/python/samba/tests/example.py", line 31, in tearDown
    self.fail()
AssertionError: None
]
time: 2021-04-28 01:28:49.862467Z
time: 2021-04-28 01:28:49.862510Z

    and after:

time: 2021-04-28 01:29:19.949347Z
test: samba.tests.example.ExampleTests.test
time: 2021-04-28 01:29:19.949440Z
time: 2021-04-28 01:29:19.949590Z
time: 2021-04-28 01:29:19.949640Z
failure: samba.tests.example.ExampleTests.test [
Traceback (most recent call last):
  File "bin/python/samba/tests/example.py", line 28, in test
    self.fail()
AssertionError: None
Traceback (most recent call last):
  File "bin/python/samba/tests/example.py", line 31, in tearDown
    self.fail()
AssertionError: None
]
time: 2021-04-28 01:29:19.949702Z

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2021-06-11 08:38:34 +00:00
Douglas Bagnall
aecb2b779b python: remove all 'from __future__ import print_function'
This made Python 2's print behave like Python 3's print().

In some cases, where we had:

   from __future__ import print_function
   """Intended module documentation..."""

this will have the side effect of making the intended module documentation
work as the actual module documentation (i.e. becoming __doc__), because
it is once again the first statement in the module.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2021-04-28 03:43:34 +00:00
Andreas Schneider
35459b753a selftest: Add support for python-dateutil >= 2.7.1
This uses the more widespread python-dateutil instead of python-iso8601.

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>

Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Tue Feb  2 19:53:35 UTC 2021 on sn-devel-184
2021-02-02 19:53:35 +00:00
Andreas Schneider
273a3c089d selftest: Directly import python-iso8601
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
2021-02-02 18:48:35 +00:00
Douglas Bagnall
9938a9f7db selftest/subunit: python file modernisation
Python idioms for iterating over a line and closing it have improved,
and we should keep up.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <npower@samba.org>
2020-12-09 16:00:39 +00:00
Stefan Metzmacher
dcd5a64ca7 selftest/subunithelper: also output as much of unterminated DETAILS
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Wed Nov 11 22:43:46 UTC 2020 on sn-devel-184
2020-11-11 22:43:46 +00:00
Stefan Metzmacher
3f556d9770 selftest/subunithelper: only let ']\n' lines to terminate
It should not be enough that a line ends with ']\n' is accident,
subunit DETAILS are terminated with '\n]\n'!

This gives a much higher chance to see the actual problem
without having them filtered by various 'filter-subunit' invocations.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2020-11-11 21:14:32 +00:00
Douglas Bagnall
ace5038031 python compat: remove binary_type
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <npower@samba.org>
2020-08-11 16:37:35 +00:00
Noel Power
ade47b3df0 PY3: Only decode when necessary 2018-10-23 05:50:25 +02:00
Joe Guo
9f5bbcc10a PEP8: fix E713: test for membership should be 'not in'
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:31 +02:00
Joe Guo
211c9a5f85 PEP8: fix E302: expected 2 blank lines, found 1
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:29 +02:00
Joe Guo
c809a86023 PEP8: fix E261: at least two spaces before inline comment
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:29 +02:00
Joe Guo
12d3fbe15c PEP8: fix E231: missing whitespace after ','
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:28 +02:00
Joe Guo
87bbc2df97 PEP8: fix E226: missing whitespace around arithmetic operator
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:28 +02:00
Joe Guo
32266d2d48 PEP8: fix E225: missing whitespace around operator
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:28 +02:00
Joe Guo
5d532543ab PEP8: fix E128: continuation line under-indented for visual indent
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:27 +02:00
Joe Guo
b43408b383 PEP8: fix E121: continuation line under-indented for hanging indent
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:26 +02:00
Joe Guo
4fc08d8f14 PEP8: fix E111: indentation is not a multiple of four
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-08-24 07:49:25 +02:00
Joe Guo
f3b5287538 python: bulk replace dict.iteritems to items for py3
In py3, iterxxx methods are removed.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-04-13 07:27:12 +02:00
Noel Power
40e7d57999 selftest: convert print func to be py2/py3 compatible
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2018-03-23 07:28:24 +01:00
Douglas Bagnall
7d79575de8 selftest: subunithelper needs to follow the subunit spec more closely
In particular allow ]\n without \n]\n as used by cmocka

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-02-22 01:04:18 +01:00
Douglas Bagnall
16e173ad2e selftest and autrobuild: convert 'except X, e' to 'except X as e'
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2018-02-15 00:18:30 +01:00
Douglas Bagnall
5b60600b32 selftest: use an additional directory of knownfail/flapping files
This makes it easier to add a temporary knownfail to cover a patch
series.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Jun  3 13:55:41 CEST 2017 on sn-devel-144
2017-06-03 13:55:41 +02:00
Douglas Bagnall
e908873757 make perftest: for performance testing
This runs a selection of subunit tests and reduces the output to only
the time it takes to run each test.

The tests are listed in selftest/perf_tests.py.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2016-08-31 07:09:26 +02:00
Douglas Bagnall
85b4a3ea61 filter-subunit: default to empty affixes, saving verbose checks
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2016-08-31 07:09:26 +02:00
Douglas Bagnall
809f4c718a subunithelper: use set for efficient inclusion test
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2016-08-31 07:09:26 +02:00
Jelmer Vernooij
02da1b3798 Remove another call to addUnexpectedSuccess with too many arguments.
Change-Id: I5f1917e74ffd57b5ae228231c27fbdf70de2992f
Signed-off-by: Jelmer Vernooij <jelmer@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2015-03-06 04:41:49 +01:00
Jelmer Vernooij
5d4a938c06 TestCase.addUnexpectedSuccess doesn't take an error.
Change-Id: I90f7dd225d9ed3cbc515292de9a37a816ac0639f
Signed-off-by: Jelmer Vernooij <jelmer@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2015-03-06 04:41:49 +01:00
Jelmer Vernooij
b0cba7950a Drop support for failfast mode, rather than adding support for it everywhere.
Change-Id: I4d6070a0e3b89d5e390f84754dddba9ec17ddf21
Signed-off-by: Jelmer Vernooij <jelmer@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2015-03-06 04:41:48 +01:00
Jelmer Vernooij
b2af53db74 Set failfast property for test reporters that need it.
Change-Id: Ibd632b9f569c23e60bcd13bcff805e367dd2e71c
Signed-off-by: Jelmer Vernooij <jelmer@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2015-03-06 04:41:48 +01:00
Jelmer Vernooij
850b3938e1 Support using third party iso8601 module if system doesn't provide one.
Change-Id: I5d035738d244d66d33788636c8ee8b322c227a0e
Signed-off-by: Jelmer Vernooij <jelmer@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2015-03-06 04:41:48 +01:00
Jelmer Vernooij
8c6d85b817 subunithelper: Fix progress support.
Change-Id: I5dd2ca0f3bc02821d5c9e1dc878bba577667d162
Signed-off-by: Jelmer Vernooij <jelmer@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2015-03-06 04:41:47 +01:00
Jelmer Vernooij
80e387adfa Use iso8601 from the system, rather than the one bundled with subunit.
Change-Id: I681bf79eb9ebe45b6972b3783c8e799eb612400b
Signed-Off-By: Jelmer Vernooij <jelmer@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2015-03-06 04:41:47 +01:00
Jelmer Vernooij
ee0b548e9d Fix handling of unexpected failures in subunithelper.
Change-Id: I0da9fcec4a54c43c171b76bb9015ea84389c9bc6
Signed-off-by: Jelmer Vernooij <jelmer@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2015-03-06 04:41:47 +01:00
Jelmer Vernooij
3f88e978a3 Use samba.subunit in selftest.subunithelper, except for iso8601.
Change-Id: Iec43e0368db91dbe981b39ac8388b50c1eafee0c
Signed-off-by: Jelmer Vernooij <jelmer@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2015-03-06 04:41:47 +01:00
Jelmer Vernooij
fb4b0ac611 Display total number of tests in format-subunit, not just number of test suites.
Change-Id: Iae17df773f380b13d2b3f69e83593139c01c4a6d
Signed-Off-By: Jelmer Vernooij <jelmer@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2014-11-19 02:46:03 +01:00
Michael Adam
5340be1740 subunit: report [X/Y at Zs] instead of [X/Y in Zs]
when running test X out of Y after Z secons have passed

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Matthieu Patou <mat@matws.net>
2014-10-23 15:56:04 +02:00
Stefan Metzmacher
3632c59e25 selftest/subunithelper.py: correctly pass testsuite-uxsuccess to end_testsuite()
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu Mar 13 23:49:36 CET 2014 on sn-devel-104
2014-03-13 23:49:35 +01:00
Stefan Metzmacher
fee156f5b9 selftest/subunithelper.py: correctly handle fail_immediately in end_testsuite of FilterOps
This way --fail-immediately also works if a command segfaults.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2014-03-13 21:45:13 +01:00
Stefan Metzmacher
14246953b2 selftest/subunithelper.py: correctly handle unexpected success in FilterOps
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2014-03-13 21:45:12 +01:00
Andrew Bartlett
0cb0427f13 selftest: Place the test environment at the end of the failure lines
This allows a knownfail entry to be added for only one environment, rather than
all environments.

Andrew Bartlett
2012-04-27 14:51:05 +02:00
Jelmer Vernooij
0fee2c01b8 selftest: Display unexpected successes and expected failures. 2011-12-05 23:11:04 +01:00
Jelmer Vernooij
e62654578b selftest: Add --flapping argument to filter-subunit. 2011-12-05 23:11:04 +01:00
Jelmer Vernooij
bbd77cbbac subunithelper: Exit with 1 if more than zero testsuites failed or
errorred.

Autobuild-User: Jelmer Vernooij <jelmer@samba.org>
Autobuild-Date: Sun Nov 21 21:13:00 CET 2010 on sn-devel-104
2010-11-21 21:13:00 +01:00
Jelmer Vernooij
4f87f8f33e filter-subunit: Add --fail-on-empty argument. 2010-11-03 17:05:19 +00:00
Kamen Mazdrashki
27f72fb19b subunithelper: Don't access 'message' attribute directly as it deprecated
Signed-off-by: Jelmer Vernooij <jelmer@samba.org>

Autobuild-User: Jelmer Vernooij <jelmer@samba.org>
Autobuild-Date: Sun Oct 10 01:50:09 UTC 2010 on sn-devel-104
2010-10-10 01:50:09 +00:00
Jelmer Vernooij
5cdef70823 subunithelper: Remove accidentally added line. 2010-10-02 23:32:12 +02:00
Jelmer Vernooij
14efa95593 subunithelper: Fix format time. 2010-10-02 22:31:31 +02:00
Jelmer Vernooij
72119de02b subunithelper: Make filter options optional. 2010-10-02 18:41:14 +02:00
Jelmer Vernooij
4ece674a99 selftest: Move plain text formatter to subunithelper. 2010-10-02 17:02:02 +02:00