test: Make it possible to run the integration tests standalone (#36868)

Currently, to run the integration tests, it's still necessary to
install various other build tools besides meson: A compiler, gperf,
libcap, ... which we want to avoid in CI systems where we receive
prebuilt systemd packages and only want to test them. Examples are
Debian's autopkgtest CI and Fedora CI. Let's make it possible for
these systems to run the integration tests without having to install
any other build dependency besides meson by extracting the logic
required to run the integration tests with meson into a separate
subdirectory and adding a standalone top-level meson.build file which
can be used to configure a meson tree with as its only purpose running
the integration tests.

Practically, we do the following:
- all the integration test directories and integration-test-wrapper.py
  are moved from test/ to test/integration-tests/.
- All the installation logic is kept out of test/integration-tests/ or
  any of its subdirectories and moved into test/meson.build instead.
- We add test/integration-tests/standalone/meson.build to run the
  integration tests standalone. This meson file includes
  test/integration-tests via a cute symlink hack to trick meson into
  including a parent directory with subdir().
- Documentation is included on how to use the new standalone mode.

.github/workflows/coverage.yml

@@ -13,7 +13,7 @@ on:
       - v[0-9]+-stable
     paths:
       - .github/workflows/coverage.yml
-      - test/integration-test-wrapper.py
+      - test/integration-tests/integration-test-wrapper.py
 
 permissions:
   contents: read

.github/workflows/linter.yml

@@ -37,7 +37,7 @@ jobs:
           VALIDATE_GITHUB_ACTIONS: true
 
       - name: Check that tabs are not used in Python code
-        run: sh -c '! git grep -P "\\t" -- src/boot/generate-hwids-section.py src/ukify/ukify.py test/integration-test-wrapper.py'
+        run: sh -c '! git grep -P "\\t" -- src/boot/generate-hwids-section.py src/ukify/ukify.py test/integration-tests/integration-test-wrapper.py'
 
       - name: Install ruff and mypy
         run: |
@@ -47,18 +47,18 @@ jobs:
       - name: Run mypy
         run: |
           python3 -m mypy --version
-          python3 -m mypy src/boot/generate-hwids-section.py src/test/generate-sym-test.py src/ukify/ukify.py test/integration-test-wrapper.py
+          python3 -m mypy src/boot/generate-hwids-section.py src/test/generate-sym-test.py src/ukify/ukify.py test/integration-tests/integration-test-wrapper.py
 
       - name: Run ruff check
         run: |
           ruff --version
-          ruff check src/boot/generate-hwids-section.py src/test/generate-sym-test.py src/ukify/ukify.py test/integration-test-wrapper.py
+          ruff check src/boot/generate-hwids-section.py src/test/generate-sym-test.py src/ukify/ukify.py test/integration-tests/integration-test-wrapper.py
 
       - name: Run ruff format
         run: |
           ruff --version
-          if ! ruff format --check src/boot/generate-hwids-section.py src/test/generate-sym-test.py src/ukify/ukify.py test/integration-test-wrapper.py
+          if ! ruff format --check src/boot/generate-hwids-section.py src/test/generate-sym-test.py src/ukify/ukify.py test/integration-tests/integration-test-wrapper.py
           then
               echo "Please run 'ruff format' on the above files or apply the diffs below manually"
-              ruff format --check --quiet --diff  src/boot/generate-hwids-section.py src/test/generate-sym-test.py src/ukify/ukify.py test/integration-test-wrapper.py
+              ruff format --check --quiet --diff  src/boot/generate-hwids-section.py src/test/generate-sym-test.py src/ukify/ukify.py test/integration-tests/integration-test-wrapper.py
           fi

.gitignore

@@ -29,8 +29,9 @@ __pycache__/
 /mkosi.installdir/
 /mkosi.key
 /mkosi.crt
-/mkosi.tools
+/mkosi.tools/
 /mkosi.tools.manifest
+/.mkosi-private/
 mkosi.local.conf
 /tags
 .dir-locals-2.el

README

@@ -313,14 +313,6 @@ REQUIREMENTS:
         https://systemd.io/SEPARATE_USR_IS_BROKEN
         https://systemd.io/THE_CASE_FOR_THE_USR_MERGE
 
-        Additional packages are necessary to run some tests:
-        - nc                 (used by test/TEST-12-ISSUE-3171)
-        - python             (test-udev which is installed is in python)
-        - python-pyparsing
-        - python-evdev       (used by hwdb parsing tests)
-        - strace             (used by test/test-functions)
-        - capsh              (optional, used by test-execute)
-
 POLICY FOR SUPPORT OF DISTRIBUTIONS AND ARCHITECTURES:
         systemd main branch and latest major or stable releases are generally
         expected to compile on current versions of popular distributions (at

docs/ARCHITECTURE.md

@@ -122,7 +122,7 @@ Sources in `test/TEST-*` implement system-level testing for executables, librari
 Most of those tests should be able to run via `systemd-nspawn`, which is orders-of-magnitude faster than `qemu`, but some tests require privileged operations like using `dm-crypt` or `loopdev`.
 They are clearly marked if that is the case.
 
-See [`test/README.md`](https://github.com/systemd/systemd/blob/main/test/README.md) for more specific details.
+See [`test/integration-tests/README.md`](https://github.com/systemd/systemd/blob/main/test/integration-tests/README.md) for more specific details.
 
 ## hwdb
 

docs/CODE_QUALITY.md

@@ -21,9 +21,10 @@ available functionality:
    change these flags for an already set up build tree, too, with "meson
    configure -C build -D…".)
 
-2. Use `./test/run-integration-tests.sh` to run the full integration test
-   suite. This will build OS images with a number of integration tests and run
-   them using `systemd-nspawn` and `qemu`. Requires root.
+2. Run the full integration test suite as described in
+   [test/integration-tests/README.md](/../test/integration-tests/README.md).
+   This will build OS images with a number of integration tests and run them
+   using `systemd-nspawn` and `qemu`. Requires root.
 
 3. Use `./coccinelle/run-coccinelle.sh` to run all
    [Coccinelle](http://coccinelle.lip6.fr/) semantic patch scripts we ship. The

docs/TESTING_WITH_SANITIZERS.md

@@ -102,5 +102,4 @@ make things work as expected in most cases. This will, obviously, not work with
 statically linked sanitizer libraries.
 
 These shenanigans are performed automatically when running the integration test
-suite (i.e. `test/TEST-??-*`) and are located in `test/test-functions` (mainly,
-but not only, in the `create_asan_wrapper` function).
+suite.

hwdb.d/meson.build

@@ -80,4 +80,4 @@ run_target(
 
 run_target(
         'update-hwdb-autosuspend',
-        command : [update_hwdb_autosuspend_sh, project_source_root])
+        command : [update_hwdb_autosuspend_sh, meson.project_source_root()])

man/meson.build

@@ -212,7 +212,7 @@ endif
 ############################################################
 
 buildroot_substs = configuration_data()
-buildroot_substs.set_quoted('BUILD_ROOT', project_build_root)
+buildroot_substs.set_quoted('BUILD_ROOT', meson.project_build_root())
 
 configure_file(
         input : 'man.in',
@@ -229,7 +229,7 @@ configure_file(
 update_dbus_docs = custom_target(
         'update-dbus-docs-impl',
         output : 'update-dbus-docs',
-        command : [update_dbus_docs_py, '--build-dir', project_build_root, '@INPUT@'],
+        command : [update_dbus_docs_py, '--build-dir', meson.project_build_root(), '@INPUT@'],
         input : dbus_docs,
         depends : dbus_programs)
 
@@ -237,7 +237,7 @@ if conf.get('BUILD_MODE_DEVELOPER') == 1
         test('dbus-docs-fresh',
              update_dbus_docs_py,
              suite : 'dist',
-             args : ['--build-dir', project_build_root, '--test', dbus_docs],
+             args : ['--build-dir', meson.project_build_root(), '--test', dbus_docs],
              depends : dbus_programs)
 
         test('check-version-history',
@@ -250,7 +250,7 @@ update_man_rules = custom_target(
         'update-man-rules-impl',
         output : 'update-man-rules',
         command : [update_man_rules_py,
-                   '@0@/man/*.xml'.format(project_source_root),
+                   '@0@/man/*.xml'.format(meson.project_source_root()),
                    '@0@/rules/meson.build'.format(meson.current_source_dir())],
         depends : man_page_depends)
 

meson.build

@@ -29,13 +29,9 @@ conf.set('PROJECT_VERSION', project_major_version,
          description : 'Numerical project version (used where a simple number is expected)')
 conf.set_quoted('PROJECT_VERSION_FULL', meson.project_version(), description : 'Full project version')
 
-# This is to be used instead of meson.source_root(), as the latter will return
-# the wrong result when systemd is being built as a meson subproject
-project_source_root = meson.current_source_dir()
-project_build_root = meson.current_build_dir()
 relative_source_path = run_command('realpath',
-                                   '--relative-to=@0@'.format(project_build_root),
-                                   project_source_root,
+                                   '--relative-to=@0@'.format(meson.project_build_root()),
+                                   meson.project_source_root(),
                                    check : true).stdout().strip()
 conf.set_quoted('RELATIVE_SOURCE_PATH', relative_source_path)
 
@@ -2173,8 +2169,8 @@ runtest_env = custom_target(
         output : 'systemd-runtest.env',
         command : [sh, '-c',
                    '{ echo SYSTEMD_TEST_DATA=@0@; echo SYSTEMD_CATALOG_DIR=@1@; } >@OUTPUT@'.format(
-                           project_source_root / 'test',
-                           project_build_root / 'catalog')],
+                           meson.project_source_root() / 'test',
+                           meson.project_build_root() / 'catalog')],
         depends : catalogs,
         build_by_default : true)
 
@@ -2747,8 +2743,8 @@ if get_option('mode') == 'developer' and want_tests != 'false' and jekyll.found(
              jekyll,
              suite : 'dist',
              args : ['build',
-                     '--source', project_source_root / 'docs',
-                     '--destination', project_build_root / '_site'])
+                     '--source', meson.project_source_root() / 'docs',
+                     '--destination', meson.project_build_root() / '_site'])
 endif
 
 #####################################################################
@@ -2825,7 +2821,7 @@ endforeach
 if git.found()
         all_files = run_command(
                 env, '-u', 'GIT_WORK_TREE',
-                git, '--git-dir=@0@/.git'.format(project_source_root),
+                git, '--git-dir=@0@/.git'.format(meson.project_source_root()),
                      'ls-files', ':/*.[ch]', ':/*.cc',
                 check : false)
         if all_files.returncode() == 0
@@ -2834,10 +2830,10 @@ if git.found()
                 custom_target(
                         'tags',
                         output : 'tags',
-                        command : [env, 'etags', '-o', '@0@/TAGS'.format(project_source_root)] + all_files)
+                        command : [env, 'etags', '-o', '@0@/TAGS'.format(meson.project_source_root())] + all_files)
                 run_target(
                         'ctags',
-                        command : [env, 'ctags', '--tag-relative=never', '-o', '@0@/tags'.format(project_source_root)] + all_files)
+                        command : [env, 'ctags', '--tag-relative=never', '-o', '@0@/tags'.format(meson.project_source_root())] + all_files)
 
                 ############################################
 
@@ -2845,7 +2841,7 @@ if git.found()
                         test('check-includes',
                              files('tools/check-includes.py'),
                              args: all_files,
-                             env : ['PROJECT_SOURCE_ROOT=@0@'.format(project_source_root)],
+                             env : ['PROJECT_SOURCE_ROOT=@0@'.format(meson.project_source_root())],
                              suite : 'headers')
                 endif
         endif
@@ -2860,18 +2856,18 @@ if git.found()
         ####################################################
 
         git_head = run_command(
-                git, '--git-dir=@0@/.git'.format(project_source_root),
+                git, '--git-dir=@0@/.git'.format(meson.project_source_root()),
                      'rev-parse', 'HEAD',
                 check : false).stdout().strip()
         git_head_short = run_command(
-                git, '--git-dir=@0@/.git'.format(project_source_root),
+                git, '--git-dir=@0@/.git'.format(meson.project_source_root()),
                      'rev-parse', '--short=7', 'HEAD',
                 check : false).stdout().strip()
 
         run_target(
                 'git-snapshot',
                 command : [git, 'archive',
-                           '-o', '@0@/systemd-@1@.tar.gz'.format(project_source_root,
+                           '-o', '@0@/systemd-@1@.tar.gz'.format(meson.project_source_root(),
                                                                  git_head_short),
                            '--prefix', 'systemd-@0@/'.format(git_head),
                            'HEAD'])
@@ -2903,7 +2899,7 @@ custom_target('installed-unit-files.txt',
               capture : true,
               install : want_tests != 'no' and install_tests,
               install_dir : testdata_dir,
-              command : [meson_extract_unit_files, project_build_root])
+              command : [meson_extract_unit_files, meson.project_build_root()])
 
 #####################################################################
 

mkosi.images/build/mkosi.conf.d/opensuse/mkosi.build.chroot

@@ -64,6 +64,12 @@ if ((COVERAGE)); then
     MKOSI_CFLAGS="$MKOSI_CFLAGS -fprofile-dir=/coverage"
 fi
 
+# The opensuse spec tars up stuff in test/ and unpacks it in test/integration-tests, which we now use for our
+# own purposes, so let's get rid of that specific bit of logic in the opensuse spec until they've had a chance
+# to adapt.
+sed "pkg/$PKG_SUBDIR${GIT_SUBDIR:+/$GIT_SUBDIR}/systemd.spec" -e '1062,1075d' -e '/integration-tests\/README/d' >/tmp/systemd.spec
+mount --bind /tmp/systemd.spec "pkg/$PKG_SUBDIR${GIT_SUBDIR:+/$GIT_SUBDIR}/systemd.spec"
+
 build() {
     IFS=
     # shellcheck disable=SC2046

mkosi.postinst.chroot

@@ -58,8 +58,8 @@ SYSTEMD_REPART_MKFS_OPTIONS_EXT4="" \
     --dry-run=no \
     --size=auto \
     --offline=true \
-    --root test/TEST-24-CRYPTSETUP \
-    --definitions test/TEST-24-CRYPTSETUP/keydev.repart \
+    --root test/integration-tests/TEST-24-CRYPTSETUP \
+    --definitions test/integration-tests/TEST-24-CRYPTSETUP/keydev.repart \
     "$OUTPUTDIR/keydev.raw"
 
 can_test_pkcs11() {
@@ -132,7 +132,7 @@ EOF
     certtool --generate-self-signed \
       --load-privkey="pkcs11:token=TestToken;object=RSATestKey;type=private" \
       --load-pubkey="pkcs11:token=TestToken;object=RSATestKey;type=public" \
-      --template "test/TEST-24-CRYPTSETUP/template.cfg" \
+      --template "test/integration-tests/TEST-24-CRYPTSETUP/template.cfg" \
       --outder --outfile "/tmp/rsa_test.crt"
 
     pkcs11-tool --module "$SOFTHSM_MODULE" --token-label "TestToken" --pin "env:GNUTLS_PIN" --so-pin "env:GNUTLS_SO_PIN" --write-object "/tmp/rsa_test.crt" --type cert --label "RSATestKey"
@@ -144,7 +144,7 @@ EOF
     certtool --generate-self-signed \
       --load-privkey="pkcs11:token=TestToken;object=ECTestKey;type=private" \
       --load-pubkey="pkcs11:token=TestToken;object=ECTestKey;type=public" \
-      --template "test/TEST-24-CRYPTSETUP/template.cfg" \
+      --template "test/integration-tests/TEST-24-CRYPTSETUP/template.cfg" \
       --outder --outfile "/tmp/ec_test.crt"
 
     pkcs11-tool --module "$SOFTHSM_MODULE" --token-label "TestToken" --pin "env:GNUTLS_PIN" --so-pin "env:GNUTLS_SO_PIN" --write-object "/tmp/ec_test.crt" --type cert --label "ECTestKey"

src/test/meson.build

@@ -12,8 +12,8 @@ test_hashmap_ordered_c = custom_target(
 path = run_command(sh, '-c', 'echo "$PATH"', check: true).stdout().strip()
 test_env = environment()
 test_env.set('SYSTEMD_LANGUAGE_FALLBACK_MAP', language_fallback_map)
-test_env.set('PATH', project_build_root + ':' + path)
-test_env.set('PROJECT_BUILD_ROOT', project_build_root)
+test_env.set('PATH', meson.project_build_root() + ':' + path)
+test_env.set('PROJECT_BUILD_ROOT', meson.project_build_root())
 test_env.set('SYSTEMD_SLOW_TESTS', want_slow_tests ? '1' : '0')
 
 if efi_addon != ''

test/TEST-52-HONORFIRSTSHUTDOWN/meson.build

@@ -1,9 +0,0 @@
-# SPDX-License-Identifier: LGPL-2.1-or-later
-
-integration_tests += [
-        integration_test_template + {
-                'name' : fs.name(meson.current_source_dir()),
-        },
-]
-
-testdata_subdirs += [meson.current_source_dir() / 'TEST-52-HONORFIRSTSHUTDOWN.units']

test/TEST-85-NETWORK/meson.build

@@ -1,31 +0,0 @@
-# SPDX-License-Identifier: LGPL-2.1-or-later
-
-name = fs.name(meson.current_source_dir())
-unit = configure_file(
-        input : files('../test.service.in'),
-        output : '@0@.service'.format(name),
-        configuration : integration_test_template['configuration'] + {
-                'command' : '@0@ --no-journal'.format(testdata_dir / 'test-network/systemd-networkd-tests.py')
-        },
-)
-
-systemd_networkd_tests_py = files('../test-network/systemd-networkd-tests.py')
-network_testcases = run_command('sed',
-                                '-ne',
-                                '/^class .*Tests/ { s/^class *//; s/(.*$//; p}',
-                                systemd_networkd_tests_py,
-                                check : true).stdout().split()
-
-foreach testcase : network_testcases
-        integration_tests += [
-                integration_test_template + {
-                        'name' : '@0@-@1@'.format(name, testcase),
-                        'unit' : unit,
-                        'cmdline' : integration_test_template['cmdline'] + [
-                                'systemd.setenv=TEST_MATCH_TESTCASE=@0@'.format(testcase)
-                        ],
-                        'priority' : 10,
-                        'vm' : true,
-                },
-        ]
-endforeach

test/fuzz/meson.build

@@ -55,7 +55,7 @@ sanitize_address_undefined = custom_target(
         'sanitize-address-undefined-fuzzers',
         output : 'sanitize-address-undefined-fuzzers',
         command : [meson_build_sh,
-                   project_source_root,
+                   meson.project_source_root(),
                    '@OUTPUT@',
                    'fuzzers',
                    ' '.join(fuzz_c_args + '-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION'),
@@ -71,13 +71,13 @@ sanitize_address_undefined = custom_target(
 fuzz_sanitizers = [['address,undefined', sanitize_address_undefined]]
 fuzz_testsdir = 'test/fuzz'
 
-if git.found() and fs.is_dir(project_source_root / '.git')
+if git.found() and fs.is_dir(meson.project_source_root() / '.git')
         out = run_command(env, '-u', 'GIT_WORK_TREE',
-                          git, '--git-dir=@0@/.git'.format(project_source_root),
+                          git, '--git-dir=@0@/.git'.format(meson.project_source_root()),
                           'ls-files', ':/@0@/*/*'.format(fuzz_testsdir),
                           check: true)
 else
-        out = run_command(sh, '-c', 'cd "@0@"; echo @1@/*/*'.format(project_source_root, fuzz_testsdir), check: true)
+        out = run_command(sh, '-c', 'cd "@0@"; echo @1@/*/*'.format(meson.project_source_root(), fuzz_testsdir), check: true)
 endif
 
 # Add crafted fuzz inputs we have in the repo

test/integration-tests/README.md

@@ -151,6 +151,31 @@ options to provide mkosi with a directory containing the systemd packages or a
 repository file that points to a repository with the systemd packages that
 should be installed.
 
+If the rpms are installed on the host system that the tests are running on,
+you'll probably want to disable usage of the tools tree so that the tools from
+the host system are used to build the image by adding the following to
+`mkosi.local.conf`:
+
+```conf
+[Build]
+ToolsTree=
+```
+
+On the other hand, if the rpms are available but not installed on the host
+system, you'll want to make sure they're installed into the mkosi tools tree so
+that they're used to build the image by using either
+`ToolsTreePackageDirectories=` or `ToolsTreeSandboxTrees=` similarly to
+`PackageDirectories=` or `SandboxTrees=` mentioned above.
+
+Finally, we'll make use of the standalone mode of running the integration tests
+to avoid having to install any build dependencies.
+
+```sh
+$ mkosi -f sandbox -- meson setup testsuite test/integration-tests/standalone
+$ mkosi -f
+$ mkosi sandbox -- meson test -C testsuite --num-processes "$(($(nproc) / 4))"
+```
+
 ### SELinux AVCs
 
 To have `TEST-06-SELINUX` check for SELinux denials, write the following to

test/integration-tests/TEST-04-JOURNAL/meson.build

@@ -8,5 +8,3 @@ integration_tests += [
                 'storage' : 'persistent',
         },
 ]
-
-testdata_subdirs += [meson.current_source_dir() / 'TEST-04-JOURNAL.units']

test/integration-tests/TEST-06-SELINUX/meson.build

@@ -13,5 +13,3 @@ integration_tests += [
                 'mkosi-args' : integration_test_template['mkosi-args'] + ['--runtime-build-sources=no'],
         },
 ]
-
-testdata_subdirs += [meson.current_source_dir() / 'TEST-06-SELINUX.units']

test/integration-tests/TEST-07-PID1/meson.build

@@ -6,5 +6,3 @@ integration_tests += [
                 'unit' : files('TEST-07-PID1.service'),
         },
 ]
-
-testdata_subdirs += [meson.current_source_dir() / 'TEST-07-PID1.units']

test/integration-tests/TEST-16-EXTEND-TIMEOUT/meson.build

@@ -7,5 +7,3 @@ integration_tests += [
                 'coredump-exclude-regex' : '/(bash|sleep)$',
         },
 ]
-
-testdata_subdirs += [meson.current_source_dir() / 'TEST-16-EXTEND-TIMEOUT.units']