1
0
mirror of https://github.com/samba-team/samba.git synced 2024-12-22 13:34:15 +03:00
samba-mirror/selftest
Michael Adam 77ba781c7a selftest: mark the samba4.blackbox.dbcheck test as flapping.
This causes frequent autobuild failures recently as follows:

[1730(16550)/1735 at 2h7m43s] samba4.blackbox.dbcheck(ad_dc_ntvfs)
UNEXPECTED(failure): samba4.blackbox.dbcheck(ad_dc_ntvfs).dbcheck(ad_dc_ntvfs:local)
REASON: Exception: Exception: Checking 6799 objects
ERROR: wrong dn[CN=DrsDelObjUser_1426756150,CN=Deleted Objects,DC=samba,DC=example,DC=com] name='DrsDelObjUser_1426756150
DEL:77ec48cb-06cc-4157-a565-770720c30929' new_dn[CN=DrsDelObjUser_1426756150

FAILED (1 failures, 0 errors and 0 unexpected successes in 0 testsuites)

Has been seen flapping since quite some time:

https://git.samba.org/autobuild.flakey/2012-06-22-0340/samba.stdout
https://git.samba.org/autobuild.flakey/2012-06-25-2339/samba.stdout
https://git.samba.org/autobuild.flakey/2012-07-02-1741/samba.stdout
https://git.samba.org/autobuild.flakey/2012-07-31-1344/samba.stdout
https://git.samba.org/autobuild.flakey/2012-09-24-1331/samba.stdout
https://git.samba.org/autobuild.flakey/2012-11-21-0933/samba.stdout
https://git.samba.org/autobuild.flakey/2012-12-02-0140/samba.stdout
https://git.samba.org/autobuild.flakey/2012-12-11-1933/samba.stdout
...
https://git.samba.org/autobuild.flakey/2014-08-20-1715/samba.stdout
https://git.samba.org/autobuild.flakey/2014-09-29-0222/samba.stdout
https://git.samba.org/autobuild.flakey/2014-12-13-1119/samba.stdout
https://git.samba.org/autobuild.flakey/2015-03-19-0529/samba.stdout
https://git.samba.org/autobuild.flakey/2015-03-19-0828/samba.stdout
https://git.samba.org/autobuild.flakey/2015-03-19-1128/samba.stdout

Possibly related to the failure of tombstone_ranimation,
drs.delete_object, and blackbox.samba_tool_demote.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>

Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Thu Mar 19 20:43:32 CET 2015 on sn-devel-104
2015-03-19 20:43:31 +01:00
..
target autobuild: Remove ntdb target 2015-03-17 11:30:51 +01:00
tests Use samba TestCase so we get all compatibility functions on Python < 2.7. 2015-03-06 04:41:48 +01:00
__init__.py selftest.py: Add dummy init file (required to run tests). 2012-03-04 18:02:07 +01:00
client.py selfclient.client: Factor out write_clientconf. 2012-03-05 03:45:57 +01:00
devel_env.sh selftest: Add a bash env file you can source. 2014-04-17 14:56:05 +02:00
filter-subunit filter-subunit: Remove import of unnecessary third party modules testtools and subunit. 2015-03-06 04:41:47 +01:00
flapping selftest: mark the samba4.blackbox.dbcheck test as flapping. 2015-03-19 20:43:31 +01:00
format-subunit format-subunit: Remove import of unnecessary third party modules testtools and subunit. 2015-03-06 04:41:47 +01:00
gdb_backtrace selftest: Cope with no binary argument being supplied to gdb_backtrace 2011-06-24 13:17:19 +10:00
gdb_backtrace_test.c selftest: Fix typo in comment. 2010-06-01 09:45:19 +02:00
gdb_run selftest: move gdb_* and valgrind_* scripts to selftest/ 2008-10-28 12:20:59 +01:00
in_screen selftest: Add support for tmate. 2014-06-06 02:25:52 +02:00
knownfail selftest: rename env s3dc to nt4_dc 2015-03-16 23:04:47 +01:00
quick selftest: Have only one set of selftest knownfail and skip files 2011-10-28 13:10:27 +02:00
README Remove documentation for testsuite-count subunit extension, which is no longer used. 2015-02-17 15:41:10 +01:00
run.py Fix use of iso8601.Utc. 2015-03-06 04:41:48 +01:00
selftest.pl selftest: rename env member to nt4_member 2015-03-17 01:46:18 +01:00
selftest.pl.1 selftest: Move manual page into a separate file. 2012-03-02 03:49:09 +01:00
selftest.py selftest: rename env dc to ad_dc_ntvfs 2015-03-16 23:04:46 +01:00
selftesthelpers.py Add basic tap2subunit converter, rather than relying on the one from subunit-tools. 2015-03-06 04:41:47 +01:00
skip selftest: rename env dc to ad_dc_ntvfs 2015-03-16 23:04:46 +01:00
slow selftest: Have only one set of selftest knownfail and skip files 2011-10-28 13:10:27 +02:00
socket_wrapper.py selftest.socket_wrapper: interface is an integer. 2012-03-04 18:02:06 +01:00
SocketWrapper.pm selftest: Add copyright headers 2009-03-25 15:40:39 +01:00
Subunit.pm selftest: Let tests add prefix to tests by themselve. 2010-09-22 17:48:23 -07:00
subunithelper.py Remove another call to addUnexpectedSuccess with too many arguments. 2015-03-06 04:41:49 +01:00
tap2subunit Add basic tap2subunit converter, rather than relying on the one from subunit-tools. 2015-03-06 04:41:47 +01:00
testlist.py selftest: Add separate command line for listing tests, allowing us of subunit-filter (which doesn't support subunit v2). 2014-11-30 14:22:05 +01:00
tests.py selftest: rename env dc to ad_dc_ntvfs 2015-03-16 23:04:46 +01:00
TODO Move selftest code to top-level. 2008-09-18 19:51:03 +02:00
valgrind_run selftest: set valgrind options 2013-12-11 17:14:21 +01:00
wscript selftest: use env.SELFTEST_PREFIX to define subunit_cache 2015-01-19 13:54:03 +01:00

# vim: ft=rst

This directory contains test scripts that are useful for running a
bunch of tests all at once.

There are two parts to this:

 * The test runner (selftest/selftest.pl)
 * The test formatter

selftest.pl simply outputs subunit, which can then be formatted or analyzed
by tools that understand the subunit protocol. One of these tools is
format-subunit, which is used by default as part of "make test".

Available testsuites
====================
The available testsuites are obtained from a script, usually
source{3,4}/selftest/tests.py. This script should for each testsuite output
the name of the test, the command to run and the environment that should be
provided. Use the included "plantest" function to generate the required output.

Testsuite behaviour
===================

Exit code
------------
The testsuites should exit with a non-zero exit code if at least one
test failed. Skipped tests should not influence the exit code.

Output format
-------------
Testsuites can simply use the exit code to indicate whether all of their
tests have succeeded or one or more have failed. It is also possible to
provide more granular information using the Subunit protocol.

This protocol works by writing simple messages to standard output. Any
messages that can not be interpreted by this protocol are considered comments
for the last announced test.

For a full description of the subunit protocol, see the README file in the subunit
repository at http://github.com/testing-cabal/subunit.

The following commands are Samba extensions to Subunit:

start-testsuite
~~~~~~~~~~~~~~~
start-testsuite: name

The testsuite name is used as prefix for all containing tests.

skip-testsuite
~~~~~~~~~~~~~~
skip-testsuite: name

Mark the testsuite with the specified name as skipped.

testsuite-success
~~~~~~~~~~~~~~~~~
testsuite-success: name

Indicate that the testsuite has succeeded successfully.

testsuite-fail
~~~~~~~~~~~~~~
testsuite-fail: name

Indicate that a testsuite has failed.

Environments
============
Tests often need to run against a server with particular things set up,
a "environment". This environment is provided by the test "target": Samba 3,
Samba 4 or Windows.

The environments are currently available include

 - none: No server set up, no variables set.
 - dc,s3dc: Domain controller set up. The following environment variables will
   be set:

     * USERNAME: Administrator user name
     * PASSWORD: Administrator password
     * DOMAIN: Domain name
     * REALM: Realm name
     * SERVER: DC host name
     * SERVER_IP: DC IPv4 address
     * SERVER_IPV6: DC IPv6 address
     * NETBIOSNAME: DC NetBIOS name
     * NETIOSALIAS: DC NetBIOS alias

 - member,s4member,s3member: Domain controller and member server that is joined to it set up. The
   following environment variables will be set:

     * USERNAME: Domain administrator user name
     * PASSWORD: Domain administrator password
     * DOMAIN: Domain name
     * REALM: Realm name
     * SERVER: Name of the member server

See Samba.pm, Samba3.pm and Samba4.pm for the full list.

Running tests
=============

To run all the tests use::

   make test

To run a quicker subset run::

   make quicktest

To run a specific test, use this syntax::

   make test TESTS=testname

for example::

   make test TESTS=samba4.BASE-DELETE