1
0
mirror of https://github.com/systemd/systemd.git synced 2024-10-30 14:55:37 +03:00
systemd/test
2022-08-31 23:33:13 +09:00
..
dmidecode-dumps memory-id: Work-around incorrect "Number of slots" 2022-03-07 15:33:26 +00:00
fuzz network: NetLabel integration 2022-08-29 14:23:17 +09:00
hwdb.d
journal-data
knot-data test: Introduce systemd-resolved test suite 2022-07-04 12:21:55 +02:00
TEST-01-BASIC test: make masking of supplementary services configurable 2022-06-19 11:27:41 +02:00
TEST-02-UNITTESTS test: ignore missing coverage in TEST-02 2022-04-07 12:45:13 +02:00
TEST-03-JOBS test: drop redundant IMAGE_NAME= 2022-06-10 12:31:14 +09:00
TEST-04-JOURNAL
TEST-05-RLIMITS
TEST-06-SELINUX
TEST-07-ISSUE-1981
TEST-08-ISSUE-2730 test: respect the global $QEMU_TIMEOUT 2022-08-26 19:16:45 +02:00
TEST-09-ISSUE-2691 test: respect the global $QEMU_TIMEOUT 2022-08-26 19:16:45 +02:00
TEST-10-ISSUE-2467
TEST-11-ISSUE-3166 test: respect the global $QEMU_TIMEOUT 2022-08-26 19:16:45 +02:00
TEST-12-ISSUE-3171
TEST-13-NSPAWN-SMOKE
TEST-14-MACHINE-ID
TEST-15-DROPIN
TEST-16-EXTEND-TIMEOUT
TEST-17-UDEV test: respect the global $QEMU_TIMEOUT 2022-08-26 19:16:45 +02:00
TEST-18-FAILUREACTION test: respect the global $QEMU_TIMEOUT 2022-08-26 19:16:45 +02:00
TEST-19-DELEGATE test: respect the global $QEMU_TIMEOUT 2022-08-26 19:16:45 +02:00
TEST-20-MAINPIDGAMES
TEST-21-DFUZZER test: run TEST-21 in nspawn only if $TEST_PREFER_NSPAWN is set 2022-06-26 22:06:02 +02:00
TEST-22-TMPFILES
TEST-23-TYPE-EXEC
TEST-24-CRYPTSETUP test: store the key on a separate device 2022-05-26 15:06:41 +02:00
TEST-25-IMPORT
TEST-26-SETENV
TEST-27-STDOUTFILE
TEST-28-PERCENTJ-WANTEDBY
TEST-29-PORTABLE test: install losetup by default 2021-12-20 23:34:27 +09:00
TEST-30-ONCLOCKCHANGE
TEST-31-DEVICE-ENUMERATION test: respect the global $QEMU_TIMEOUT 2022-08-26 19:16:45 +02:00
TEST-32-OOMPOLICY
TEST-33-CLEAN-UNIT
TEST-34-DYNAMICUSERMIGRATE
TEST-35-LOGIN Revert "tests: add test for handling of background sessions" 2022-07-14 11:07:13 +02:00
TEST-36-NUMAPOLICY test: bump the base VM memory to 768M 2022-08-26 15:40:34 +02:00
TEST-37-RUNTIMEDIRECTORYPRESERVE
TEST-38-FREEZER
TEST-39-EXECRELOAD
TEST-40-EXEC-COMMAND-EX
TEST-41-ONESHOT-RESTART
TEST-42-EXECSTOPPOST
TEST-43-PRIVATEUSER-UNPRIV core: support ExtensionDirectories in user manager 2022-03-10 20:38:10 +00:00
TEST-44-LOG-NAMESPACE
TEST-45-TIMEDATE test: import timedated test from debian/ubuntu test suite 2022-06-10 16:04:52 +09:00
TEST-46-HOMED test: make homed test run in qemu 2021-11-16 12:33:44 +01:00
TEST-47-ISSUE-14566
TEST-48-START-STOP-NO-RELOAD
TEST-49-RUNTIME-BIND-PATHS
TEST-50-DISSECT test: respect the global $QEMU_TIMEOUT 2022-08-26 19:16:45 +02:00
TEST-51-ISSUE-16115
TEST-52-HONORFIRSTSHUTDOWN test: check journal directly instead of capturing console output 2022-06-17 17:32:41 +02:00
TEST-53-ISSUE-16347
TEST-54-CREDS tmpfiles: automatically provision /etc/issue.d/ + /etc/motd.d/ + /etc/hosts from credentials 2022-07-21 00:06:22 +02:00
TEST-55-OOMD
TEST-56-EXIT-TYPE test: require unified cgroup hierarchy for TEST-56 2022-02-01 22:13:48 +01:00
TEST-57-ONSUCCESS-UPHOLD
TEST-58-REPART test: make TEST-58-REPART support nspawn 2022-08-22 17:00:38 +02:00
TEST-59-RELOADING-RESTART
TEST-60-MOUNT-RATELIMIT
TEST-61-UNITTESTS-QEMU
TEST-62-RESTRICT-IFACES
TEST-63-PATH test: make TEST-63 more reliable on slower machines 2022-08-25 16:21:39 +00:00
TEST-64-UDEV-STORAGE test-64: run one more subtest on non-KVM environment with relaxed condition 2022-08-28 09:18:18 +00:00
TEST-65-ANALYZE
TEST-66-DEVICE-ISOLATION
TEST-67-INTEGRITY test: respect the global $QEMU_TIMEOUT 2022-08-26 19:16:45 +02:00
TEST-68-PROPAGATE-EXIT-STATUS tests: add test to cover service exit status propagation 2021-12-13 14:44:06 +00:00
TEST-69-SHUTDOWN test: use a dropin to override the end.service unit 2022-03-21 16:41:49 +01:00
TEST-70-TPM2 test: install libgcc_s.so.1 explicitly if available 2022-08-02 18:28:35 +02:00
TEST-71-HOSTNAME test: import hostnamed tests from debian/ubuntu test suite 2022-06-14 20:56:18 +09:00
TEST-72-SYSUPDATE test: add integration test for sysupdate 2022-03-19 00:13:55 +01:00
TEST-73-LOCALE test: import localed tests from debian/ubuntu test suite 2022-06-14 20:59:36 +09:00
TEST-75-RESOLVED test: install /etc/default/knot if available as well 2022-08-23 11:26:35 +01:00
TEST-76-SYSCTL sysctl: add --strict option to fail if sysctl does not exists 2022-07-25 10:15:43 +02:00
test-bcd test: Add BCD unit test 2021-12-11 21:32:29 +01:00
test-execute various: try to use DEFAULT_USER_SHELL for root too 2022-08-24 10:02:46 +02:00
test-fstab-generator test: store empty files rather than symlinks for test-fstab-generator 2022-01-09 18:17:40 +01:00
test-keymap-util
test-network test: testing for networkd NetLabel feature 2022-08-29 14:23:17 +09:00
test-network-generator-conversion
test-path
test-path-util
test-resolve resolved: Test for DnsStream (plain TCP DNS and DoT) 2022-01-27 14:33:56 +09:00
test-sysusers sysusers: use log_syntax (changes prefix from '[%s:%u]' to '%s:%u:') 2022-08-22 12:52:25 +02:00
test-umount
testsuite-04.units
testsuite-06.units
testsuite-08.units TEST-08: don't force ext4 for / 2021-11-16 16:58:49 +01:00
testsuite-10.units
testsuite-11.units
testsuite-16.units
testsuite-28.units
testsuite-30.units/systemd-timedated.service.d
testsuite-52.units
testsuite-63.units test: make TEST-63 more reliable on slower machines 2022-08-25 16:21:39 +00:00
units test-29-portable: enable debugging logs of udevd 2022-08-31 23:33:13 +09:00
.gitignore
create-busybox-container
create-sys-script.py
hwdb-test.sh hwdb-test: don't rely on --root= quirk 2022-08-22 14:07:45 +02:00
meson.build test-compare-versions: basic test for systemd-analyze compare-versions 2022-05-19 11:07:28 +02:00
mkosi-check-and-shutdown.service ci: check for failed services after boot 2021-12-10 10:25:43 +01:00
mkosi-check-and-shutdown.sh ci: check for failed services after boot 2021-12-10 10:25:43 +01:00
mkosi.build.networkd-test
mkosi.default.networkd-test mkosi: Remove usage of deprecated option names/sections 2022-07-18 16:54:58 +02:00
mkosi.nspawn.networkd-test
networkd-test.py network/bridge: fix UseBPDU= and AllowPortToBeRoot= 2022-08-14 06:56:05 +09:00
README.testsuite docs: use lowercase "qemu" and other minor tweaks to test docs 2022-05-04 12:58:43 +02:00
rule-syntax-check.py
run-integration-tests.sh test runner: print time before/after tests 2021-11-22 09:05:00 +00:00
run-unit-tests.py
sd-script.py
sys-script.py
sysv-generator-test.py
test-compare-versions.sh analyze: use '' instead of the empty string when showing versions 2022-05-25 16:01:14 +02:00
test-exec-deserialization.py
test-fstab-generator.sh test: store empty files rather than symlinks for test-fstab-generator 2022-01-09 18:17:40 +01:00
test-functions test: make the virt detection quiet 2022-08-26 19:18:20 +02:00
test-network-generator-conversion.sh
test-rpm-macros.sh tree-wide: drop manually-crafted message for missing variables 2022-05-04 20:53:46 +09:00
test-shutdown.py test: bump the reboot timeout to 60 seconds 2022-08-26 19:16:45 +02:00
test-systemctl-enable.sh Revert "shared/install: create relative symlinks for enablement and aliasing" 2022-05-05 19:02:26 +02:00
test-systemd-tmpfiles.py tmpfiles: optionally, decode string to write to files with base64 2022-07-15 11:55:03 +02:00
test-sysusers.sh.in sysusers: use log_syntax (changes prefix from '[%s:%u]' to '%s:%u:') 2022-08-22 12:52:25 +02:00
testdata
udev-dmi-memory-id-test.sh
udev-test.pl test: add testcase for #23288 2022-05-07 02:51:31 +09:00

The extended testsuite only works with UID=0. It consists of the subdirectories
named "test/TEST-??-*", each of which contains a description of an OS image and
a test which consists of systemd units and scripts to execute in this image.
The same image is used for execution under `systemd-nspawn` and `qemu`.

To run the extended testsuite do the following:

$ ninja -C build  # Avoid building anything as root later
$ sudo test/run-integration-tests.sh
ninja: Entering directory `/home/zbyszek/src/systemd/build'
ninja: no work to do.
--x-- Running TEST-01-BASIC --x--
+ make -C TEST-01-BASIC clean setup run
make: Entering directory '/home/zbyszek/src/systemd/test/TEST-01-BASIC'
TEST-01-BASIC CLEANUP: Basic systemd setup
TEST-01-BASIC SETUP: Basic systemd setup
...
TEST-01-BASIC RUN: Basic systemd setup [OK]
make: Leaving directory '/home/zbyszek/src/systemd/test/TEST-01-BASIC'
--x-- Result of TEST-01-BASIC: 0 --x--
--x-- Running TEST-02-CRYPTSETUP --x--
+ make -C TEST-02-CRYPTSETUP clean setup run

If one of the tests fails, then $subdir/test.log contains the log file of
the test.

To run just one of the cases:

$ sudo make -C test/TEST-01-BASIC clean setup run

Specifying the build directory
==============================

If the build directory is not detected automatically, it can be specified
with BUILD_DIR=:

$ sudo BUILD_DIR=some-other-build/ test/run-integration-tests

or

$ sudo make -C test/TEST-01-BASIC BUILD_DIR=../../some-other-build/ ...

Note that in the second case, the path is relative to the test case directory.
An absolute path may also be used in both cases.

Testing installed binaries instead of built
===========================================

To run the extended testsuite using the systemd installed on the system instead
of the systemd from a build, use the NO_BUILD=1:

$ sudo NO_BUILD=1 test/run-integration-tests

Configuration variables
=======================

TEST_NO_QEMU=1
    Don't run tests under qemu

TEST_QEMU_ONLY=1
    Run only tests that require qemu

TEST_NO_NSPAWN=1
    Don't run tests under systemd-nspawn

TEST_PREFER_NSPAWN=1
    Run all tests that do not require qemu under systemd-nspawn

TEST_NO_KVM=1
    Disable qemu KVM auto-detection (may be necessary when you're trying to run the
    *vanilla* qemu and have both qemu and qemu-kvm installed)

TEST_NESTED_KVM=1
    Allow tests to run with nested KVM. By default, the testsuite disables
    nested KVM if the host machine already runs under KVM. Setting this
    variable disables such checks

QEMU_MEM=512M
    Configure amount of memory for qemu VMs (defaults to 512M)

QEMU_SMP=1
    Configure number of CPUs for qemu VMs (defaults to 1)

KERNEL_APPEND='...'
    Append additional parameters to the kernel command line

NSPAWN_ARGUMENTS='...'
    Specify additional arguments for systemd-nspawn

QEMU_TIMEOUT=infinity
    Set a timeout for tests under qemu (defaults to infinity)

NSPAWN_TIMEOUT=infinity
    Set a timeout for tests under systemd-nspawn (defaults to infinity)

INTERACTIVE_DEBUG=1
    Configure the machine to be more *user-friendly* for interactive debuggung
    (e.g. by setting a usable default terminal, suppressing the shutdown after
    the test, etc.)

The kernel and initramfs can be specified with $KERNEL_BIN and $INITRD.
(Fedora's or Debian's default kernel path and initramfs are used by default)

A script will try to find your qemu binary. If you want to specify a different
one with $QEMU_BIN.

Debugging the qemu image
========================

If you want to log in the testsuite virtual machine, you can specify additional
kernel command line parameter with $KERNEL_APPEND and then log in as root.

$ sudo make -C test/TEST-01-BASIC KERNEL_APPEND="systemd.unit=multi-user.target" run

Root password is empty.

Ubuntu CI
=========

New PR submitted to the project are run through regression tests, and one set
of those is the 'autopkgtest' runs for several different architectures, called
'Ubuntu CI'.  Part of that testing is to run all these tests.  Sometimes these
tests are temporarily deny-listed from running in the 'autopkgtest' tests while
debugging a flaky test; that is done by creating a file in the test directory
named 'deny-list-ubuntu-ci', for example to prevent the TEST-01-BASIC test from
running in the 'autopkgtest' runs, create the file
'TEST-01-BASIC/deny-list-ubuntu-ci'.

The tests may be disabled only for specific archs, by creating a deny-list file
with the arch name at the end, e.g.
'TEST-01-BASIC/deny-list-ubuntu-ci-arm64' to disable the TEST-01-BASIC test
only on test runs for the 'arm64' architecture.

Note the arch naming is not from 'uname -m', it is Debian arch names:
https://wiki.debian.org/ArchitectureSpecificsMemo

For PRs that fix a currently deny-listed test, the PR should include removal
of the deny-list file.

In case a test fails, the full set of artifacts, including the journal of the
failed run, can be downloaded from the artifacts.tar.gz archive which will be
reachable in the same URL parent directory as the logs.gz that gets linked on
the Github CI status.

To add new dependencies or new binaries to the packages used during the tests,
a merge request can be sent to: https://salsa.debian.org/systemd-team/systemd
targeting the 'upstream-ci' branch.

The cloud-side infrastructure, that is hooked into the Github interface, is
located at:

https://git.launchpad.net/autopkgtest-cloud/

In case of infrastructure issues with this CI, things might go wrong in two
places:

- starting a job: this is done via a Github webhook, so check if the HTTP POST
  are failing on https://github.com/systemd/systemd/settings/hooks
- running a job: all currently running jobs are listed at
  https://autopkgtest.ubuntu.com/running#pkg-systemd-upstream in case the PR
  does not show the status for some reason
- reporting the job result: this is done on Canonical's cloud infrastructure,
  if jobs are started and running but no status is visible on the PR, then it is
  likely that reporting back is not working

For infrastructure help, reaching out to Canonical via the #ubuntu-devel channel
on libera.chat is an effective way to receive support in general.

Manually running a part of the Ubuntu CI test suite
===================================================

In some situations one may want/need to run one of the tests run by Ubuntu CI
locally for debugging purposes. For this, you need a machine (or a VM) with
the same Ubuntu release as is used by Ubuntu CI (Focal ATTOW).

First of all, clone the Debian systemd repository and sync it with the code of
the PR (set by the $UPSTREAM_PULL_REQUEST env variable) you'd like to debug:

# git clone https://salsa.debian.org/systemd-team/systemd.git
# cd systemd
# git checkout upstream-ci
# TEST_UPSTREAM=1 UPSTREAM_PULL_REQUEST=12345 ./debian/extra/checkout-upstream

Now install necessary build & test dependencies:

## PPA with some newer Ubuntu packages required by upstream systemd
# add-apt-repository -y ppa:upstream-systemd-ci/systemd-ci
# apt build-dep -y systemd
# apt install -y autopkgtest debhelper genisoimage git qemu-system-x86 \
                 libzstd-dev libfdisk-dev libtss2-dev libfido2-dev libssl-dev \
                 python3-jinja2 zstd

Build systemd deb packages with debug info:

# DEB_BUILD_OPTIONS="nocheck nostrip" dpkg-buildpackage -us -uc
# cd ..

Prepare a testbed image for autopkgtest (tweak the release as necessary):

# autopkgtest-buildvm-ubuntu-cloud -v -a amd64 -r focal

And finally run the autopkgtest itself:

# autopkgtest -o logs *.deb systemd/ \
              --timeout-factor=3 \
              --test-name=boot-and-services \
              --shell-fail \
              -- autopkgtest-virt-qemu autopkgtest-focal-amd64.img

where --test-name= is the name of the test you want to run/debug. The
--shell-fail option will pause the execution in case the test fails and shows
you the information how to connect to the testbed for further debugging.

Manually running LGTM/CodeQL analysis
=====================================

This is mostly useful for debugging various CodeQL/LGTM quirks.

Download the CodeQL Bundle from https://github.com/github/codeql-action/releases
and unpack it somewhere. From now the 'tutorial' assumes you have the `codeql`
binary from the unpacked archive in $PATH for brevity.

Switch to the systemd repository if not already:

$ cd <systemd-repo>

Create an initial CodeQL database:

$ CCACHE_DISABLE=1 codeql database create codeqldb --language=cpp -vvv

Disabling ccache is important, otherwise you might see CodeQL complaining:

No source code was seen and extracted to /home/mrc0mmand/repos/@ci-incubator/systemd/codeqldb.
This can occur if the specified build commands failed to compile or process any code.
 - Confirm that there is some source code for the specified language in the project.
 - For codebases written in Go, JavaScript, TypeScript, and Python, do not specify
   an explicit --command.
 - For other languages, the --command must specify a "clean" build which compiles
   all the source code files without reusing existing build artefacts.

If you want to run all queries systemd uses in LGTM/CodeQL, run:

$ codeql database analyze codeqldb/ --format csv --output results.csv .github/codeql-custom.qls .lgtm/cpp-queries/*.ql -vvv

Note: this will take a while.

If you're interested in a specific check, the easiest way (without hunting down
the specific CodeQL query file) is to create a custom query suite. For example:

$ cat >test.qls <<EOF
- queries: .
  from: codeql/cpp-queries
- include:
    id:
        - cpp/missing-return
EOF

And then execute it in the same way as above:

$ codeql database analyze codeqldb/ --format csv --output results.csv test.qls -vvv

More about query suites here: https://codeql.github.com/docs/codeql-cli/creating-codeql-query-suites/

The results are then located in the `results.csv` file as a comma separated
values list (obviously), which is the most human-friendly output format the
CodeQL utility provides (so far).