1
1
mirror of https://github.com/systemd/systemd-stable.git synced 2024-12-28 07:21:32 +03:00
systemd-stable/test/run-unit-tests.py

78 lines
2.3 KiB
Python
Raw Normal View History

tests: add a runner for installed tests We have "installed tests", but don't provide an easy way to run them. The protocol is very simple: each test must return 0 for success, 77 means "skipped", anything else is an error. In addition, we want to print test output only if the test failed. I wrote this simple script. It is pretty basic, but implements the functions listed above. Since it is written in python it should be easy to add option parsing (like running only specific tests, or running unsafe tests, etc.) I looked at the following alternatives: - Ubuntu root-unittests: this works, but just dumps all output to the terminal, has no coloring. - @ssahani's test runner [2] It uses the unittest library and the test suite was implented as a class, and doesn't implement any of the functions listed above. - cram [3,4] cram runs our tests, but does not understand the "ignore the output" part, has not support for our magic skip code (it uses hardcoded 80 instead), and seems dead upstream. - meson test Here the idea would be to provide an almost-empty meson.build file under /usr/lib/systemd/tests/ that would just define all the tests. This would allow us to reuse the test runner we use normally. Unfortunately meson requires a build directory and configuration to be done before running tests. This would be possible, but seems a lot of effort to just run a few binaries. [1] https://salsa.debian.org/systemd-team/systemd/blob/242c96addb06480ec9cd75248a5660f37a17b4b9/debian/tests/root-unittests [2] https://github.com/systemd/systemd-fedora-ci/blob/master/upstream/systemd-upstream-tests.py [3] https://bitheap.org/cram/ [4] https://pypi.org/project/pytest-cram/ Fixes #10069.
2018-09-20 17:34:14 +03:00
#!/usr/bin/env python3
# SPDX-License-Identifier: LGPL-2.1-or-later
tests: add a runner for installed tests We have "installed tests", but don't provide an easy way to run them. The protocol is very simple: each test must return 0 for success, 77 means "skipped", anything else is an error. In addition, we want to print test output only if the test failed. I wrote this simple script. It is pretty basic, but implements the functions listed above. Since it is written in python it should be easy to add option parsing (like running only specific tests, or running unsafe tests, etc.) I looked at the following alternatives: - Ubuntu root-unittests: this works, but just dumps all output to the terminal, has no coloring. - @ssahani's test runner [2] It uses the unittest library and the test suite was implented as a class, and doesn't implement any of the functions listed above. - cram [3,4] cram runs our tests, but does not understand the "ignore the output" part, has not support for our magic skip code (it uses hardcoded 80 instead), and seems dead upstream. - meson test Here the idea would be to provide an almost-empty meson.build file under /usr/lib/systemd/tests/ that would just define all the tests. This would allow us to reuse the test runner we use normally. Unfortunately meson requires a build directory and configuration to be done before running tests. This would be possible, but seems a lot of effort to just run a few binaries. [1] https://salsa.debian.org/systemd-team/systemd/blob/242c96addb06480ec9cd75248a5660f37a17b4b9/debian/tests/root-unittests [2] https://github.com/systemd/systemd-fedora-ci/blob/master/upstream/systemd-upstream-tests.py [3] https://bitheap.org/cram/ [4] https://pypi.org/project/pytest-cram/ Fixes #10069.
2018-09-20 17:34:14 +03:00
import argparse
tests: add a runner for installed tests We have "installed tests", but don't provide an easy way to run them. The protocol is very simple: each test must return 0 for success, 77 means "skipped", anything else is an error. In addition, we want to print test output only if the test failed. I wrote this simple script. It is pretty basic, but implements the functions listed above. Since it is written in python it should be easy to add option parsing (like running only specific tests, or running unsafe tests, etc.) I looked at the following alternatives: - Ubuntu root-unittests: this works, but just dumps all output to the terminal, has no coloring. - @ssahani's test runner [2] It uses the unittest library and the test suite was implented as a class, and doesn't implement any of the functions listed above. - cram [3,4] cram runs our tests, but does not understand the "ignore the output" part, has not support for our magic skip code (it uses hardcoded 80 instead), and seems dead upstream. - meson test Here the idea would be to provide an almost-empty meson.build file under /usr/lib/systemd/tests/ that would just define all the tests. This would allow us to reuse the test runner we use normally. Unfortunately meson requires a build directory and configuration to be done before running tests. This would be possible, but seems a lot of effort to just run a few binaries. [1] https://salsa.debian.org/systemd-team/systemd/blob/242c96addb06480ec9cd75248a5660f37a17b4b9/debian/tests/root-unittests [2] https://github.com/systemd/systemd-fedora-ci/blob/master/upstream/systemd-upstream-tests.py [3] https://bitheap.org/cram/ [4] https://pypi.org/project/pytest-cram/ Fixes #10069.
2018-09-20 17:34:14 +03:00
import glob
import os
import pathlib
tests: add a runner for installed tests We have "installed tests", but don't provide an easy way to run them. The protocol is very simple: each test must return 0 for success, 77 means "skipped", anything else is an error. In addition, we want to print test output only if the test failed. I wrote this simple script. It is pretty basic, but implements the functions listed above. Since it is written in python it should be easy to add option parsing (like running only specific tests, or running unsafe tests, etc.) I looked at the following alternatives: - Ubuntu root-unittests: this works, but just dumps all output to the terminal, has no coloring. - @ssahani's test runner [2] It uses the unittest library and the test suite was implented as a class, and doesn't implement any of the functions listed above. - cram [3,4] cram runs our tests, but does not understand the "ignore the output" part, has not support for our magic skip code (it uses hardcoded 80 instead), and seems dead upstream. - meson test Here the idea would be to provide an almost-empty meson.build file under /usr/lib/systemd/tests/ that would just define all the tests. This would allow us to reuse the test runner we use normally. Unfortunately meson requires a build directory and configuration to be done before running tests. This would be possible, but seems a lot of effort to just run a few binaries. [1] https://salsa.debian.org/systemd-team/systemd/blob/242c96addb06480ec9cd75248a5660f37a17b4b9/debian/tests/root-unittests [2] https://github.com/systemd/systemd-fedora-ci/blob/master/upstream/systemd-upstream-tests.py [3] https://bitheap.org/cram/ [4] https://pypi.org/project/pytest-cram/ Fixes #10069.
2018-09-20 17:34:14 +03:00
import subprocess
import sys
try:
import colorama as c
GREEN = c.Fore.GREEN
YELLOW = c.Fore.YELLOW
RED = c.Fore.RED
RESET_ALL = c.Style.RESET_ALL
BRIGHT = c.Style.BRIGHT
except ImportError:
GREEN = YELLOW = RED = RESET_ALL = BRIGHT = ''
class total:
total = None
good = 0
skip = 0
fail = 0
tests: add a runner for installed tests We have "installed tests", but don't provide an easy way to run them. The protocol is very simple: each test must return 0 for success, 77 means "skipped", anything else is an error. In addition, we want to print test output only if the test failed. I wrote this simple script. It is pretty basic, but implements the functions listed above. Since it is written in python it should be easy to add option parsing (like running only specific tests, or running unsafe tests, etc.) I looked at the following alternatives: - Ubuntu root-unittests: this works, but just dumps all output to the terminal, has no coloring. - @ssahani's test runner [2] It uses the unittest library and the test suite was implented as a class, and doesn't implement any of the functions listed above. - cram [3,4] cram runs our tests, but does not understand the "ignore the output" part, has not support for our magic skip code (it uses hardcoded 80 instead), and seems dead upstream. - meson test Here the idea would be to provide an almost-empty meson.build file under /usr/lib/systemd/tests/ that would just define all the tests. This would allow us to reuse the test runner we use normally. Unfortunately meson requires a build directory and configuration to be done before running tests. This would be possible, but seems a lot of effort to just run a few binaries. [1] https://salsa.debian.org/systemd-team/systemd/blob/242c96addb06480ec9cd75248a5660f37a17b4b9/debian/tests/root-unittests [2] https://github.com/systemd/systemd-fedora-ci/blob/master/upstream/systemd-upstream-tests.py [3] https://bitheap.org/cram/ [4] https://pypi.org/project/pytest-cram/ Fixes #10069.
2018-09-20 17:34:14 +03:00
def argument_parser():
p = argparse.ArgumentParser()
p.add_argument('-u', '--unsafe', action='store_true',
help='run "unsafe" tests too')
p.add_argument('-A', '--artifact_directory',
help='store output from failed tests in this dir')
return p
opts = argument_parser().parse_args()
tests: add a runner for installed tests We have "installed tests", but don't provide an easy way to run them. The protocol is very simple: each test must return 0 for success, 77 means "skipped", anything else is an error. In addition, we want to print test output only if the test failed. I wrote this simple script. It is pretty basic, but implements the functions listed above. Since it is written in python it should be easy to add option parsing (like running only specific tests, or running unsafe tests, etc.) I looked at the following alternatives: - Ubuntu root-unittests: this works, but just dumps all output to the terminal, has no coloring. - @ssahani's test runner [2] It uses the unittest library and the test suite was implented as a class, and doesn't implement any of the functions listed above. - cram [3,4] cram runs our tests, but does not understand the "ignore the output" part, has not support for our magic skip code (it uses hardcoded 80 instead), and seems dead upstream. - meson test Here the idea would be to provide an almost-empty meson.build file under /usr/lib/systemd/tests/ that would just define all the tests. This would allow us to reuse the test runner we use normally. Unfortunately meson requires a build directory and configuration to be done before running tests. This would be possible, but seems a lot of effort to just run a few binaries. [1] https://salsa.debian.org/systemd-team/systemd/blob/242c96addb06480ec9cd75248a5660f37a17b4b9/debian/tests/root-unittests [2] https://github.com/systemd/systemd-fedora-ci/blob/master/upstream/systemd-upstream-tests.py [3] https://bitheap.org/cram/ [4] https://pypi.org/project/pytest-cram/ Fixes #10069.
2018-09-20 17:34:14 +03:00
tests = glob.glob('/usr/lib/systemd/tests/test-*')
if opts.unsafe:
tests += glob.glob('/usr/lib/systemd/tests/unsafe/test-*')
if not opts.artifact_directory and os.getenv('ARTIFACT_DIRECTORY'):
opts.artifact_directory = os.getenv('ARTIFACT_DIRECTORY')
total.total = len(tests)
tests: add a runner for installed tests We have "installed tests", but don't provide an easy way to run them. The protocol is very simple: each test must return 0 for success, 77 means "skipped", anything else is an error. In addition, we want to print test output only if the test failed. I wrote this simple script. It is pretty basic, but implements the functions listed above. Since it is written in python it should be easy to add option parsing (like running only specific tests, or running unsafe tests, etc.) I looked at the following alternatives: - Ubuntu root-unittests: this works, but just dumps all output to the terminal, has no coloring. - @ssahani's test runner [2] It uses the unittest library and the test suite was implented as a class, and doesn't implement any of the functions listed above. - cram [3,4] cram runs our tests, but does not understand the "ignore the output" part, has not support for our magic skip code (it uses hardcoded 80 instead), and seems dead upstream. - meson test Here the idea would be to provide an almost-empty meson.build file under /usr/lib/systemd/tests/ that would just define all the tests. This would allow us to reuse the test runner we use normally. Unfortunately meson requires a build directory and configuration to be done before running tests. This would be possible, but seems a lot of effort to just run a few binaries. [1] https://salsa.debian.org/systemd-team/systemd/blob/242c96addb06480ec9cd75248a5660f37a17b4b9/debian/tests/root-unittests [2] https://github.com/systemd/systemd-fedora-ci/blob/master/upstream/systemd-upstream-tests.py [3] https://bitheap.org/cram/ [4] https://pypi.org/project/pytest-cram/ Fixes #10069.
2018-09-20 17:34:14 +03:00
for test in tests:
name = os.path.basename(test)
ex = subprocess.run(test, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
tests: add a runner for installed tests We have "installed tests", but don't provide an easy way to run them. The protocol is very simple: each test must return 0 for success, 77 means "skipped", anything else is an error. In addition, we want to print test output only if the test failed. I wrote this simple script. It is pretty basic, but implements the functions listed above. Since it is written in python it should be easy to add option parsing (like running only specific tests, or running unsafe tests, etc.) I looked at the following alternatives: - Ubuntu root-unittests: this works, but just dumps all output to the terminal, has no coloring. - @ssahani's test runner [2] It uses the unittest library and the test suite was implented as a class, and doesn't implement any of the functions listed above. - cram [3,4] cram runs our tests, but does not understand the "ignore the output" part, has not support for our magic skip code (it uses hardcoded 80 instead), and seems dead upstream. - meson test Here the idea would be to provide an almost-empty meson.build file under /usr/lib/systemd/tests/ that would just define all the tests. This would allow us to reuse the test runner we use normally. Unfortunately meson requires a build directory and configuration to be done before running tests. This would be possible, but seems a lot of effort to just run a few binaries. [1] https://salsa.debian.org/systemd-team/systemd/blob/242c96addb06480ec9cd75248a5660f37a17b4b9/debian/tests/root-unittests [2] https://github.com/systemd/systemd-fedora-ci/blob/master/upstream/systemd-upstream-tests.py [3] https://bitheap.org/cram/ [4] https://pypi.org/project/pytest-cram/ Fixes #10069.
2018-09-20 17:34:14 +03:00
if ex.returncode == 0:
print(f'{GREEN}PASS: {name}{RESET_ALL}')
total.good += 1
elif ex.returncode == 77:
print(f'{YELLOW}SKIP: {name}{RESET_ALL}')
total.skip += 1
else:
print(f'{RED}FAIL: {name}{RESET_ALL}')
total.fail += 1
output_file = None
if opts.artifact_directory:
output_dir = pathlib.Path(opts.artifact_directory) / 'unit-tests'
output_dir.mkdir(parents=True, exist_ok=True)
output_file = output_dir / name
output_file.write_bytes(ex.stdout)
try:
print(ex.stdout.decode('utf-8'))
except UnicodeDecodeError:
print(f'{BRIGHT}Note, some test output shown here is not UTF-8')
if output_file:
print(f'For actual test output see artifact file {output_file}')
print(f'{RESET_ALL}')
print(ex.stdout.decode('utf-8', errors='replace'))
sys.stdout.flush()
tests: add a runner for installed tests We have "installed tests", but don't provide an easy way to run them. The protocol is very simple: each test must return 0 for success, 77 means "skipped", anything else is an error. In addition, we want to print test output only if the test failed. I wrote this simple script. It is pretty basic, but implements the functions listed above. Since it is written in python it should be easy to add option parsing (like running only specific tests, or running unsafe tests, etc.) I looked at the following alternatives: - Ubuntu root-unittests: this works, but just dumps all output to the terminal, has no coloring. - @ssahani's test runner [2] It uses the unittest library and the test suite was implented as a class, and doesn't implement any of the functions listed above. - cram [3,4] cram runs our tests, but does not understand the "ignore the output" part, has not support for our magic skip code (it uses hardcoded 80 instead), and seems dead upstream. - meson test Here the idea would be to provide an almost-empty meson.build file under /usr/lib/systemd/tests/ that would just define all the tests. This would allow us to reuse the test runner we use normally. Unfortunately meson requires a build directory and configuration to be done before running tests. This would be possible, but seems a lot of effort to just run a few binaries. [1] https://salsa.debian.org/systemd-team/systemd/blob/242c96addb06480ec9cd75248a5660f37a17b4b9/debian/tests/root-unittests [2] https://github.com/systemd/systemd-fedora-ci/blob/master/upstream/systemd-upstream-tests.py [3] https://bitheap.org/cram/ [4] https://pypi.org/project/pytest-cram/ Fixes #10069.
2018-09-20 17:34:14 +03:00
print(f'{BRIGHT}OK: {total.good} SKIP: {total.skip} FAIL: {total.fail}{RESET_ALL}')
sys.exit(total.fail > 0)