2018-10-17 17:55:02 +03:00
# Automation Environment
2019-02-28 21:40:56 +08:00
The automation env is serving the tests of Nmstate.
2018-10-17 17:55:02 +03:00
It may be used both locally and through CI.
## Components
2019-11-12 11:53:54 +01:00
- Container specifications to be used for the tests are in the `packaging`
2021-11-30 16:25:17 +08:00
directory. The images are published on quay:
https://quay.io/organization/nmstate
2019-01-17 20:03:54 +08:00
2019-02-28 21:40:56 +08:00
- run-tests.sh: Execute the tests in a container using
2020-02-20 23:56:55 +08:00
'nmstate/fedora-nmstate-dev' container image.
2018-10-17 17:55:02 +03:00
The following steps are executed:
- Run the container (defined in the Dockerfile) as a daemon.
- Stop NetworkManager before adding additional networks (ifaces).
- Add additonal networks (ifaces) to the container.
- Start NetworkManager.
2019-02-28 21:40:56 +08:00
- Execute all tests in the container.
2018-10-17 17:55:02 +03:00
It also handles the cleanup of the container and nets (stop,rm).
## Running the Tests
2020-02-20 23:56:55 +08:00
Assuming *podman* is installed on the host, just run:
2019-02-28 21:40:56 +08:00
`./automation/run-tests.sh`
By default, `./automation/run-tests.sh` will run all tests in the container
2020-02-20 23:56:55 +08:00
using 'nmstate/fedora-nmstate-dev' container image.
2020-01-07 16:05:51 +01:00
You may change the test type by specifying the `--test-type` flag, for example:
2019-02-28 21:40:56 +08:00
2019-11-12 11:53:54 +01:00
* `./automation/run-tests.sh --test-type integ --el8` :
2020-02-13 15:09:11 +01:00
Integration tests (without slow test cases) using
2021-11-30 16:25:17 +08:00
'nmstate/c8s-nmstate-dev' container image.
2019-02-28 21:40:56 +08:00
* `./automation/run-tests.sh --test-type integ` :
2020-02-13 15:09:11 +01:00
Integration tests (without slow test cases) using
2020-02-20 23:56:55 +08:00
'nmstate/fedora-nmstate-dev' container image.
2020-02-13 15:09:11 +01:00
2020-03-17 23:32:36 +01:00
* `./automation/run-tests.sh --test-type integ_tier1` :
Integration tier1 test cases using `nmstate/fedora-nmstate-dev` container
image.
* `./automation/run-tests.sh --test-type integ_tier2` :
Integration tier2 test cases using `nmstate/fedora-nmstate-dev` container
image.
2020-02-13 15:09:11 +01:00
* `./automation/run-tests.sh --test-type integ_slow` :
2020-02-20 23:56:55 +08:00
Integration slow test cases using `nmstate/fedora-nmstate-dev` container
image.
2018-10-17 17:55:02 +03:00
2020-01-07 16:05:51 +01:00
For a full list of command-line flags, run `./automation/run-tests.sh --help` .
2018-10-17 17:55:02 +03:00
## Development
### Run the tests manually in the container
2018-10-31 09:08:00 +01:00
For debugging, it is convenient to run the container and then connect to it in
order to run the tests. Setting the environment variable `debug_exit_shell`
will make the script start a shell instead of exiting the script after an error
or running the scripts:
2019-02-28 21:40:56 +08:00
`debug_exit_shell=1 ./automation/run-tests.sh`
2018-10-31 09:08:00 +01:00
2019-02-06 19:31:19 +01:00
After closing the shell, the container will be removed. Alternatively it is
possible to provide the `--debug-shell` command-line option.
2018-11-27 12:30:39 +01:00
To specify a different container image for the tests, specify it with the
2020-01-07 15:58:04 +01:00
`CONTAINER_IMAGE` variable:
2018-11-27 12:30:39 +01:00
2021-11-30 16:25:17 +08:00
`CONTAINER_IMAGE=local/c8s-nmstate-dev debug_exit_shell=1 ./automation/run-tests.sh`
2018-11-27 12:30:55 +01:00
It is also possible to pass extra arguments to PDB using the
2019-02-06 19:31:19 +01:00
`nmstate_pytest_extra_args` variable or via `--pytest-args` command-line
option, for example:
2019-02-28 21:40:56 +08:00
`nmstate_pytest_extra_args="--pdb -x" ./automation/run-tests.sh`
2018-11-27 12:30:39 +01:00
2019-02-06 19:31:19 +01:00
or:
2019-04-26 20:39:13 +02:00
`./automation/run-tests.sh --pytest-args "--pdb -x"`
2019-02-06 19:31:19 +01:00
2018-11-27 12:33:08 +01:00
### Build a new container image
2018-10-17 17:55:02 +03:00
```
2021-11-30 16:25:17 +08:00
../packaging/build-container.sh local/c8s-nmstate-dev
2020-02-26 11:31:11 +01:00
../packaging/build-container.sh local/fedora-nmstate-dev
2018-11-27 12:33:08 +01:00
```
To test the image, either specify it manually as described above or tag it locally:
```
2021-11-30 16:25:17 +08:00
podman tag local/c8s-nmstate-dev quay.io/nmstate/c8s-nmstate-dev:latest
podman tag local/fedora-nmstate-dev quay.io/nmstate/fedora-nmstate-dev:latest
2018-11-27 12:33:08 +01:00
```
2020-03-17 16:59:47 +08:00
### Push local image to the docker hub
2020-08-22 13:24:30 +02:00
The container images are automatically rebuilt for new commits to the base
2021-11-30 16:25:17 +08:00
branch or new tags. Therefore updates to the quay images should always
2019-02-19 16:44:46 +01:00
happen with a pull request that is merged to ensure that the change is
persistent. If this is not feasible, a new build could be pushed as follow to
the Docker Hub:
```shell
2021-11-30 16:25:17 +08:00
podman login quay.io
podman tag local/c8s-nmstate-dev nmstate/c8s-nmstate-dev:latest
podman push nmstate/c8s-nmstate-dev:latest \
quay.io/nmstate/c8s-nmstate-dev:latest
2020-02-20 23:56:55 +08:00
2020-02-26 11:31:11 +01:00
podman tag local/fedora-nmstate-dev nmstate/fedora-nmstate-dev:latest
podman push nmstate/fedora-nmstate-dev:latest \
2021-11-30 16:25:17 +08:00
quay.io/nmstate/fedora-nmstate-dev:latest
2018-10-17 17:55:02 +03:00
```
2019-02-19 16:44:46 +01:00
2020-08-22 13:24:30 +02:00
It will be overwritten after the next commit to base, though.
2020-09-30 11:28:12 +08:00
### Test in bare-metal OS for InfiniBand
In order to perform integration test cases against InfiniBand feature,
running test in bare-metal OS is required.
Assuming the InfiniBand card is listed in `ip link` as `mlx5_ib0` and been
configured as `datagram` mode.
```shell
sudo dnf install `./packaging/make_rpm.sh|tail -1`
sudo ip netns add tmp
sudo ip link add eth1 type veth peer name eth1peer
sudo ip link add eth2 type veth peer name eth2peer
sudo ip link set eth1 up
sudo ip link set eth2 up
sudo ip link set eth1peer netns tmp
sudo ip link set eth2peer netns tmp
sudo ip netns exec tmp ip link set eth1peer up
sudo ip netns exec tmp ip link set eth2peer up
sudo nmcli device set eth1 managed yes
sudo nmcli device set eth2 managed yes
cd tests/integration
# Set TEST_IB_CONNECTED_MODE=1 when cards are configured as connected mode
sudo env TEST_REAL_NIC=mlx5_ib0 pytest-3 -vv ./infiniband_test.py
```
2021-03-24 14:52:11 +01:00
### Run tests at a Kubernetes cluster
By default if `automation/run-tests.sh` is call with `--k8s` option it will
startup an internal k8s cluster running in top of qemu with one node, it
will install a pod that wraps the nmstate-dev image selected and will run the
selected tests inside of the pod.
The pod mounts the NetworkManager and ovs sockets from nodes so nmstate tests
are running over node network configuration.
For external clusters the env vars KUBEVIRT_PROVIDER=external and KUBECONFIG
need to be set, the rest works the same.