# Automation Environment The automation env is serving the tests of Nmstate. It may be used both locally and through CI. ## Components - Container specifications to be used for the tests are in the `packaging` directory. The images are published on quay: https://quay.io/organization/nmstate - run-tests.sh: Execute the tests in a container using 'nmstate/fedora-nmstate-dev' container image. The following steps are executed: - Run the container (defined in the Dockerfile) as a daemon. - Stop NetworkManager before adding additional networks (ifaces). - Add additonal networks (ifaces) to the container. - Start NetworkManager. - Execute all tests in the container. It also handles the cleanup of the container and nets (stop,rm). ## Running the Tests Assuming *podman* is installed on the host, just run: `./automation/run-tests.sh` By default, `./automation/run-tests.sh` will run all tests in the container using 'nmstate/fedora-nmstate-dev' container image. You may change the test type by specifying the `--test-type` flag, for example: * `./automation/run-tests.sh --test-type integ --el8`: Integration tests (without slow test cases) using 'nmstate/c8s-nmstate-dev' container image. * `./automation/run-tests.sh --test-type integ`: Integration tests (without slow test cases) using 'nmstate/fedora-nmstate-dev' container image. * `./automation/run-tests.sh --test-type integ_tier1`: Integration tier1 test cases using `nmstate/fedora-nmstate-dev` container image. * `./automation/run-tests.sh --test-type integ_tier2`: Integration tier2 test cases using `nmstate/fedora-nmstate-dev` container image. * `./automation/run-tests.sh --test-type integ_slow`: Integration slow test cases using `nmstate/fedora-nmstate-dev` container image. For a full list of command-line flags, run `./automation/run-tests.sh --help`. ## Development ### Run the tests manually in the container For debugging, it is convenient to run the container and then connect to it in order to run the tests. Setting the environment variable `debug_exit_shell` will make the script start a shell instead of exiting the script after an error or running the scripts: `debug_exit_shell=1 ./automation/run-tests.sh` After closing the shell, the container will be removed. Alternatively it is possible to provide the `--debug-shell` command-line option. To specify a different container image for the tests, specify it with the `CONTAINER_IMAGE` variable: `CONTAINER_IMAGE=local/c8s-nmstate-dev debug_exit_shell=1 ./automation/run-tests.sh` It is also possible to pass extra arguments to PDB using the `nmstate_pytest_extra_args` variable or via `--pytest-args` command-line option, for example: `nmstate_pytest_extra_args="--pdb -x" ./automation/run-tests.sh` or: `./automation/run-tests.sh --pytest-args "--pdb -x"` ### Build a new container image ``` ../packaging/build-container.sh local/c8s-nmstate-dev ../packaging/build-container.sh local/fedora-nmstate-dev ``` To test the image, either specify it manually as described above or tag it locally: ``` podman tag local/c8s-nmstate-dev quay.io/nmstate/c8s-nmstate-dev:latest podman tag local/fedora-nmstate-dev quay.io/nmstate/fedora-nmstate-dev:latest ``` ### Push local image to the docker hub The container images are automatically rebuilt for new commits to the base branch or new tags. Therefore updates to the quay images should always happen with a pull request that is merged to ensure that the change is persistent. If this is not feasible, a new build could be pushed as follow to the Docker Hub: ```shell podman login quay.io podman tag local/c8s-nmstate-dev nmstate/c8s-nmstate-dev:latest podman push nmstate/c8s-nmstate-dev:latest \ quay.io/nmstate/c8s-nmstate-dev:latest podman tag local/fedora-nmstate-dev nmstate/fedora-nmstate-dev:latest podman push nmstate/fedora-nmstate-dev:latest \ quay.io/nmstate/fedora-nmstate-dev:latest ``` It will be overwritten after the next commit to base, though. ### Test in bare-metal OS for InfiniBand In order to perform integration test cases against InfiniBand feature, running test in bare-metal OS is required. Assuming the InfiniBand card is listed in `ip link` as `mlx5_ib0` and been configured as `datagram` mode. ```shell sudo dnf install `./packaging/make_rpm.sh|tail -1` sudo ip netns add tmp sudo ip link add eth1 type veth peer name eth1peer sudo ip link add eth2 type veth peer name eth2peer sudo ip link set eth1 up sudo ip link set eth2 up sudo ip link set eth1peer netns tmp sudo ip link set eth2peer netns tmp sudo ip netns exec tmp ip link set eth1peer up sudo ip netns exec tmp ip link set eth2peer up sudo nmcli device set eth1 managed yes sudo nmcli device set eth2 managed yes cd tests/integration # Set TEST_IB_CONNECTED_MODE=1 when cards are configured as connected mode sudo env TEST_REAL_NIC=mlx5_ib0 pytest-3 -vv ./infiniband_test.py ``` ### Run tests at a Kubernetes cluster By default if `automation/run-tests.sh` is call with `--k8s` option it will startup an internal k8s cluster running in top of qemu with one node, it will install a pod that wraps the nmstate-dev image selected and will run the selected tests inside of the pod. The pod mounts the NetworkManager and ovs sockets from nodes so nmstate tests are running over node network configuration. For external clusters the env vars KUBEVIRT_PROVIDER=external and KUBECONFIG need to be set, the rest works the same.