7dfbe5b64f
As part of documentation, supply some very simple test cases
to illustrate how test cases work. One test case shows
commands in the setup, command, verify and teardown stages.
Other test cases show how to have a working test case that
does not have commands in the setup, verify and/or teardown
stages.
Specifically, the command lists for setup and teardown can
be empty. And the verify command must have a command, but
it can be /bin/true. The regex must have a string, we
recommend a single space, and the count of matches must be
zero if you do not want to use the match feature of verify.
Verify will always look for a return code of success (0)
so we give /bin/true when we do not want to make a check
there.
Also, update the documentation for testcases to be more
specific in the cases of:
- accepting non-success return codes in setup and
teardown stages
- how to write the test when no setup, teardown
and/or verify are desired.
To run the example test cases:
$ sudo -E ./tdc.py -f creating-testcases/example.json -l
1f: (example) simple test to test framework
2f: (example) simple test, no need for verify
3f: (example) simple test, no need for setup or teardown (or verify)
$ sudo -E ./tdc.py -f creating-testcases/example.json
Test 1f: simple test to test framework
Test 2f: simple test, no need for verify
Test 3f: simple test, no need for setup or teardown (or verify)
All test results:
1..3
ok 1 1f simple test to test framework
ok 2 2f simple test, no need for verify
ok 3 3f simple test, no need for setup or teardown (or verify)
$
Signed-off-by: Brenda J. Butler <bjb@mojatatu.com>
Acked-by: Lucas Bates <lucasb@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
82 lines
3.8 KiB
Plaintext
82 lines
3.8 KiB
Plaintext
tdc - Adding test cases for tdc
|
|
|
|
Author: Lucas Bates - lucasb@mojatatu.com
|
|
|
|
ADDING TEST CASES
|
|
-----------------
|
|
|
|
User-defined tests should be added by defining a separate JSON file. This
|
|
will help prevent conflicts when updating the repository. Refer to
|
|
template.json for the required JSON format for test cases.
|
|
|
|
Include the 'id' field, but do not assign a value. Running tdc with the -i
|
|
option will generate a unique ID for that test case.
|
|
|
|
tdc will recursively search the 'tc' subdirectory for .json files. Any
|
|
test case files you create in these directories will automatically be included.
|
|
If you wish to store your custom test cases elsewhere, be sure to run tdc
|
|
with the -f argument and the path to your file.
|
|
|
|
Be aware of required escape characters in the JSON data - particularly when
|
|
defining the match pattern. Refer to the tctests.json file for examples when
|
|
in doubt.
|
|
|
|
|
|
TEST CASE STRUCTURE
|
|
-------------------
|
|
|
|
Each test case has required data:
|
|
|
|
id: A unique alphanumeric value to identify a particular test case
|
|
name: Descriptive name that explains the command under test
|
|
category: A list of single-word descriptions covering what the command
|
|
under test is testing. Example: filter, actions, u32, gact, etc.
|
|
setup: The list of commands required to ensure the command under test
|
|
succeeds. For example: if testing a filter, the command to create
|
|
the qdisc would appear here.
|
|
This list can be empty.
|
|
Each command can be a string to be executed, or a list consisting
|
|
of a string which is a command to be executed, followed by 1 or
|
|
more acceptable exit codes for this command.
|
|
If only a string is given for the command, then an exit code of 0
|
|
will be expected.
|
|
cmdUnderTest: The tc command being tested itself.
|
|
expExitCode: The code returned by the command under test upon its termination.
|
|
tdc will compare this value against the actual returned value.
|
|
verifyCmd: The tc command to be run to verify successful execution.
|
|
For example: if the command under test creates a gact action,
|
|
verifyCmd should be "$TC actions show action gact"
|
|
matchPattern: A regular expression to be applied against the output of the
|
|
verifyCmd to prove the command under test succeeded. This pattern
|
|
should be as specific as possible so that a false positive is not
|
|
matched.
|
|
matchCount: How many times the regex in matchPattern should match. A value
|
|
of 0 is acceptable.
|
|
teardown: The list of commands to clean up after the test is completed.
|
|
The environment should be returned to the same state as when
|
|
this test was started: qdiscs deleted, actions flushed, etc.
|
|
This list can be empty.
|
|
Each command can be a string to be executed, or a list consisting
|
|
of a string which is a command to be executed, followed by 1 or
|
|
more acceptable exit codes for this command.
|
|
If only a string is given for the command, then an exit code of 0
|
|
will be expected.
|
|
|
|
|
|
SETUP/TEARDOWN ERRORS
|
|
---------------------
|
|
|
|
If an error is detected during the setup/teardown process, execution of the
|
|
tests will immediately stop with an error message and the namespace in which
|
|
the tests are run will be destroyed. This is to prevent inaccurate results
|
|
in the test cases.
|
|
|
|
Repeated failures of the setup/teardown may indicate a problem with the test
|
|
case, or possibly even a bug in one of the commands that are not being tested.
|
|
|
|
It's possible to include acceptable exit codes with the setup/teardown command
|
|
so that it doesn't halt the script for an error that doesn't matter. Turn the
|
|
individual command into a list, with the command being first, followed by all
|
|
acceptable exit codes for the command.
|
|
|