Support unit test in CI container

Changed the test automation script to run the all tests on Fedora CI
docker image.

To change the test type and docker image, you may:
    * Use `--test-type <TYPE>` argument or environment variable `TEST_TYPE`.
      Current test types are: lint, unit_py27, unit_py36, unit_py37,
      and integ.
    * Use `--el7` to use 'nmstate/centos7-nmstate-dev:latest' docker
      image.
    * Use environment variable `DOCKER_IMAGE` to specify docker image.

The `run-tests.environment.yaml` file is added to instruct stdci to
run integration test on CentOS only as it was.

The test automation script has been renamed
`./automation/run-integration-tests.sh` to `./automation/run-tests.sh`.

Signed-off-by: Gris Ge <fge@redhat.com>

.stdci.yml

@@ -5,4 +5,4 @@ Architectures:
       runtime-requirements:
         host-distro: same
 script:
-    from-file: automation/run-integration-tests.sh
+    from-file: automation/run-tests.sh

automation/README.md

@@ -1,5 +1,5 @@
 # Automation Environment
-The automation env is serving the integration tests of Nmstate.
+The automation env is serving the tests of Nmstate.
 It may be used both locally and through CI.
 
 ## Components
@@ -15,28 +15,51 @@ It may be used both locally and through CI.
   The image can be found at:
   https://hub.docker.com/r/nmstate/fedora-nmstate-dev/
 
-- run-integration-tests.sh: Execute the integration tests in a
-  container using docker.
+- run-tests.sh: Execute the tests in a container using
+  'nmstate/fedora-nmstate-dev' docker image.
 
   The following steps are executed:
   - Run the container (defined in the Dockerfile) as a daemon.
   - Stop NetworkManager before adding additional networks (ifaces).
   - Add additonal networks (ifaces) to the container.
   - Start NetworkManager.
-  - Execute the integration tests (using tox) in the container.
+  - Execute all tests in the container.
 
   It also handles the cleanup of the container and nets (stop,rm).
 
-- run-integration-tests.mounts: Includes mounts to be used by the
-  oVirt CI (STDCI) worker.
+- run-tests.mounts: Includes mounts to be used by the oVirt CI (STDCI) worker.
 
-- run-integration-tests.packages: Includes the packages needed by
-  the oVirt CI (STDCI) worker.
+- run-tests.packages: Includes the packages needed by the oVirt CI (STDCI)
+  worker.
+
+- run-tests.environment.yaml: Instruct the oVirt CI (STDCI) to run
+  integration test.
 
 ## Running the Tests
-Assuming *docker* is installed on the host,
-just run:
-`./automation/run-integration-tests.sh`
+Assuming *docker* is installed on the host, just run:
+`./automation/run-tests.sh`
+
+By default, `./automation/run-tests.sh` will run all tests in the container
+using 'nmstate/fedora-nmstate-dev' docker image.
+You may change the test type by using:
+
+ * `./automation/run-tests.sh --test-type lint`:
+   Static analysis of code using 'nmstate/fedora-nmstate-dev' docker image.
+
+ * `./automation/run-tests.sh --test-type unit_py27`:
+   Unit tests in Python 2.7 using 'nmstate/fedora-nmstate-dev' docker image.
+
+ * `./automation/run-tests.sh --test-type unit_py36`:
+   Unit tests in Python 3.6 using 'nmstate/fedora-nmstate-dev' docker image.
+
+ * `./automation/run-tests.sh --test-type unit_py37`:
+   Unit tests in Python 3.7 using 'nmstate/fedora-nmstate-dev' docker image.
+
+ * `./automation/run-tests.sh --test-type integ --el7`:
+   Integration tests using 'nmstate/centos7-nmstate-dev' docker image.
+
+ * `./automation/run-tests.sh --test-type integ`:
+   Integration tests using 'nmstate/fedora-nmstate-dev' docker image.
 
 ## Development
 
@@ -46,7 +69,7 @@ order to run the tests. Setting the environment variable `debug_exit_shell`
 will make the script start a shell instead of exiting the script after an error
 or running the scripts:
 
-`debug_exit_shell=1 ./automation/run-integration-tests.sh`
+`debug_exit_shell=1 ./automation/run-tests.sh`
 
 After closing the shell, the container will be removed. Alternatively it is
 possible to provide the `--debug-shell` command-line option.
@@ -54,13 +77,13 @@ possible to provide the `--debug-shell` command-line option.
 To specify a different container image for the tests, specify it with the
 `DOCKER_IMAGE` variable:
 
-`DOCKER_IMAGE=local/centos7-nmstate-dev debug_exit_shell=1 ./automation/run-integration-tests.sh`
+`DOCKER_IMAGE=local/centos7-nmstate-dev debug_exit_shell=1 ./automation/run-tests.sh`
 
 It is also possible to pass extra arguments to PDB using the
 `nmstate_pytest_extra_args` variable or via `--pytest-args` command-line
 option, for example:
 
-`nmstate_pytest_extra_args="--pdb -x" ./automation/run-integration-tests.sh`
+`nmstate_pytest_extra_args="--pdb -x" ./automation/run-tests.sh`
 
 or:
 
@@ -70,7 +93,7 @@ or:
 Alternatively, the following commands start the container manually:
 
 ```
-DOCKER_IMAGE="nmstate/centos7-nmstate-dev"
+DOCKER_IMAGE="nmstate/fedora-nmstate-dev"
 NET0="nmstate-net0"
 NET1="nmstate-net1"
 

automation/run-integration-tests.sh

@@ -1,148 +0,0 @@
-#!/bin/sh -ex
-
-EXEC_PATH=$(dirname "$(realpath "$0")")
-PROJECT_PATH="$(dirname $EXEC_PATH)"
-: ${DOCKER_IMAGE:=nmstate/centos7-nmstate-dev}
-EXPORT_DIR="$PWD/exported-artifacts"
-CONT_EXPORT_DIR="/exported-artifacts"
-if [[ $DOCKER_IMAGE == *"fedora"* ]];then
-    PYTHON_SITE_PATH_CMD="rpm -E %{python3_sitelib}"
-else
-    PYTHON_SITE_PATH_CMD="rpm -E %{python_sitelib}"
-fi
-
-NET0="nmstate-net0"
-NET1="nmstate-net1"
-
-test -t 1 && USE_TTY="-t"
-
-function remove_container {
-    res=$?
-    [ "$res" -ne 0 ] && echo "*** ERROR: $res"
-    docker_exec 'rm -rf /workspace/nmstate/*nmstate*.rpm'
-    docker rm $CONTAINER_ID -f
-    docker network rm $NET0
-    docker network rm $NET1
-}
-
-function pyclean {
-        find . -type f -name "*.py[co]" -delete
-        find . -type d -name "__pycache__" -delete
-}
-
-function docker_exec {
-    docker exec $USE_TTY -i $CONTAINER_ID /bin/bash -c "$1"
-}
-
-function add_extra_networks {
-    docker network create $NET0 || true
-    docker network create $NET1 || true
-    docker network connect $NET0 $CONTAINER_ID
-    docker network connect $NET1 $CONTAINER_ID
-    docker_exec '
-      ip addr flush eth1 && \
-      ip addr flush eth2
-    '
-}
-
-function dump_network_info {
-    docker_exec '
-      nmcli dev; \
-      nmcli con; \
-      ip addr; \
-      ip route; \
-      cat /etc/resolv.conf; \
-      head /proc/sys/net/ipv6/conf/*/disable_ipv6; \
-    '
-}
-
-function install_nmstate {
-    docker_exec '
-      cd /workspace/nmstate &&
-      rpm -ivh `./packaging/make_rpm.sh|tail -1 || exit 1`
-    '
-}
-
-function run_tests {
-    docker_exec "
-      cd /workspace/nmstate &&
-      pytest \
-        --verbose --verbose \
-        --log-level=DEBUG \
-        --log-date-format='%Y-%m-%d %H:%M:%S' \
-        --log-format='%(asctime)s %(filename)s:%(lineno)d %(levelname)s %(message)s' \
-        --durations=5 \
-        --cov=\$($PYTHON_SITE_PATH_CMD)/libnmstate \
-        --cov=\$($PYTHON_SITE_PATH_CMD)/nmstatectl \
-        --cov-report=html:htmlcov-integ \
-        --cov-report=term \
-        tests/integration \
-    ${nmstate_pytest_extra_args}"
-}
-
-function collect_artifacts {
-    docker_exec "
-      journalctl > "$CONT_EXPORT_DIR/journal.log" && \
-      cd /workspace/nmstate && cp core* "$CONT_EXPORT_DIR/" || true
-    "
-}
-
-function run_exit {
-    dump_network_info
-    collect_artifacts
-    remove_container
-}
-
-function open_shell {
-    res=$?
-    [ "$res" -ne 0 ] && echo "*** ERROR: $res"
-    set +o errexit
-    docker_exec 'echo "pytest tests/integration --pdb" >> ~/.bash_history'
-    docker_exec 'cd /workspace/nmstate; exec /bin/bash'
-    run_exit
-}
-
-options=$(getopt --options "" --long pytest-args:,help,debug-shell -- "${@}")
-eval set -- "$options"
-while true; do
-    case "$1" in
-    --pytest-args)
-        shift
-        nmstate_pytest_extra_args="$1"
-        ;;
-    --debug-shell)
-        debug_exit_shell="1"
-        ;;
-    --help)
-        echo "$0 [--pytest-args=...] [--help] [--debug-shell]"
-        exit
-        ;;
-    --)
-        shift
-        break
-        ;;
-    esac
-    shift
-done
-
-cd $EXEC_PATH
-docker --version && cat /etc/resolv.conf
-
-mkdir -p $EXPORT_DIR
-
-lsmod | grep -q ^openvswitch || modprobe openvswitch || { echo 1>&2 "Please run 'modprobe openvswitch' as root"; exit 1; }
-
-CONTAINER_ID="$(docker run --privileged -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v $PROJECT_PATH:/workspace/nmstate -v $EXPORT_DIR:$CONT_EXPORT_DIR $DOCKER_IMAGE)"
-[ -n "$debug_exit_shell" ] && trap open_shell EXIT || trap run_exit EXIT
-docker_exec 'while ! systemctl is-active dbus; do sleep 1; done'
-docker_exec '
-    systemctl start NetworkManager
-    while ! systemctl is-active NetworkManager; do sleep 1; done
-'
-pyclean
-
-dump_network_info
-install_nmstate
-add_extra_networks
-dump_network_info
-run_tests

automation/run-tests.environment.yaml

@@ -0,0 +1,5 @@
+---
+- name: 'TEST_TYPE'
+  value: 'integ'
+- name: 'DOCKER_IMAGE'
+  value: 'nmstate/centos7-nmstate-dev'

automation/run-tests.sh

@@ -0,0 +1,220 @@
+#!/bin/sh -ex
+
+EXEC_PATH=$(dirname "$(realpath "$0")")
+PROJECT_PATH="$(dirname $EXEC_PATH)"
+EXPORT_DIR="$PWD/exported-artifacts"
+CONT_EXPORT_DIR="/exported-artifacts"
+
+NET0="nmstate-net0"
+NET1="nmstate-net1"
+
+TEST_TYPE_ALL="all"
+TEST_TYPE_LINT="lint"
+TEST_TYPE_UNIT_PY27="unit_py27"
+TEST_TYPE_UNIT_PY36="unit_py36"
+TEST_TYPE_UNIT_PY37="unit_py37"
+TEST_TYPE_INTEG="integ"
+
+test -t 1 && USE_TTY="-t"
+
+function remove_container {
+    res=$?
+    [ "$res" -ne 0 ] && echo "*** ERROR: $res"
+    docker_exec 'rm -rf /workspace/nmstate/*nmstate*.rpm'
+    docker rm $CONTAINER_ID -f
+    docker network rm $NET0
+    docker network rm $NET1
+}
+
+function pyclean {
+        find . -type f -name "*.py[co]" -delete
+        find . -type d -name "__pycache__" -delete
+}
+
+function docker_exec {
+    docker exec $USE_TTY -i $CONTAINER_ID \
+        /bin/bash -c "cd /workspace/nmstate && $1"
+}
+
+function add_extra_networks {
+    docker network create $NET0 || true
+    docker network create $NET1 || true
+    docker network connect $NET0 $CONTAINER_ID
+    docker network connect $NET1 $CONTAINER_ID
+    docker_exec '
+      ip addr flush eth1 && \
+      ip addr flush eth2
+    '
+}
+
+function dump_network_info {
+    docker_exec '
+      nmcli dev; \
+      nmcli con; \
+      ip addr; \
+      ip route; \
+      cat /etc/resolv.conf; \
+      head /proc/sys/net/ipv6/conf/*/disable_ipv6; \
+    '
+}
+
+function install_nmstate {
+    docker_exec '
+      rpm -ivh `./packaging/make_rpm.sh|tail -1 || exit 1`
+    '
+}
+
+function run_tests {
+    if [ $TEST_TYPE == $TEST_TYPE_ALL ] || \
+       [ $TEST_TYPE == $TEST_TYPE_LINT ];then
+        if [[ $DOCKER_IMAGE == *"centos"* ]]; then
+            echo "Running unit test in $DOCKER_IMAGE container is not " \
+                 "support yet"
+        else
+            docker_exec 'tox -e flake8,pylint'
+        fi
+    fi
+
+    if [ $TEST_TYPE == $TEST_TYPE_ALL ] || \
+       [ $TEST_TYPE == $TEST_TYPE_UNIT_PY27 ];then
+        if [[ $DOCKER_IMAGE == *"centos"* ]]; then
+            echo "Running unit test in $DOCKER_IMAGE container is not " \
+                 "support yet"
+        else
+            docker_exec 'tox -e check-py27'
+        fi
+    fi
+
+    if [ $TEST_TYPE == $TEST_TYPE_ALL ] || \
+       [ $TEST_TYPE == $TEST_TYPE_UNIT_PY36 ];then
+        if [[ $DOCKER_IMAGE == *"centos"* ]]; then
+            echo "Running unit test in $DOCKER_IMAGE container is not " \
+                 "support yet"
+        else
+            docker_exec 'tox -e check-py36'
+        fi
+    fi
+
+    if [ $TEST_TYPE == $TEST_TYPE_ALL ] || \
+       [ $TEST_TYPE == $TEST_TYPE_UNIT_PY37 ];then
+        if [[ $DOCKER_IMAGE == *"centos"* ]]; then
+            echo "Running unit test in $DOCKER_IMAGE container is not " \
+                 "support yet"
+        else
+            docker_exec 'tox -e check-py37'
+        fi
+    fi
+
+    if [ $TEST_TYPE == $TEST_TYPE_ALL ] || \
+       [ $TEST_TYPE == $TEST_TYPE_INTEG ];then
+        docker_exec "
+          cd /workspace/nmstate &&
+          pytest \
+            --verbose --verbose \
+            --log-level=DEBUG \
+            --log-date-format='%Y-%m-%d %H:%M:%S' \
+            --log-format='%(asctime)s %(filename)s:%(lineno)d %(levelname)s %(message)s' \
+            --durations=5 \
+            --cov=\$($PYTHON_SITE_PATH_CMD)/libnmstate \
+            --cov=\$($PYTHON_SITE_PATH_CMD)/nmstatectl \
+            --cov-report=html:htmlcov-integ \
+            --cov-report=term \
+            tests/integration \
+            ${nmstate_pytest_extra_args}"
+    fi
+}
+
+function collect_artifacts {
+    docker_exec "
+      journalctl > "$CONT_EXPORT_DIR/journal.log" && \
+      cp core* "$CONT_EXPORT_DIR/" || true
+    "
+}
+
+function run_exit {
+    dump_network_info
+    collect_artifacts
+    remove_container
+}
+
+function open_shell {
+    res=$?
+    [ "$res" -ne 0 ] && echo "*** ERROR: $res"
+    set +o errexit
+    docker_exec 'echo "pytest tests/integration --pdb" >> ~/.bash_history'
+    docker_exec 'exec /bin/bash'
+    run_exit
+}
+
+options=$(getopt --options "" \
+    --long pytest-args:,help,debug-shell,test-type:,el7\
+    -- "${@}")
+eval set -- "$options"
+while true; do
+    case "$1" in
+    --pytest-args)
+        shift
+        nmstate_pytest_extra_args="$1"
+        ;;
+    --debug-shell)
+        debug_exit_shell="1"
+        ;;
+    --test-type)
+        shift
+        TEST_TYPE="$1"
+        ;;
+    --el7)
+        DOCKER_IMAGE="nmstate/centos7-nmstate-dev"
+        ;;
+    --help)
+        echo -n "$0 [--pytest-args=...] [--help] [--debug-shell] "
+        echo -n "[--el7] "
+        echo "[--test-type=<TEST_TYPE>]"
+        echo "    Valid TEST_TYPE are:"
+        echo "     * $TEST_TYPE_ALL (default)"
+        echo "     * $TEST_TYPE_LINT"
+        echo "     * $TEST_TYPE_INTEG"
+        echo "     * $TEST_TYPE_UNIT_PY27"
+        echo "     * $TEST_TYPE_UNIT_PY36"
+        echo "     * $TEST_TYPE_UNIT_PY37"
+        exit
+        ;;
+    --)
+        shift
+        break
+        ;;
+    esac
+    shift
+done
+
+#Valid TEST_TYPE are: all, lint, unit_py27, unit_py36, unit_py37, integ.
+: ${TEST_TYPE:=$TEST_TYPE_ALL}
+: ${DOCKER_IMAGE:=nmstate/fedora-nmstate-dev}
+
+if [[ $DOCKER_IMAGE == *"fedora"* ]];then
+    PYTHON_SITE_PATH_CMD="rpm -E %{python3_sitelib}"
+else
+    PYTHON_SITE_PATH_CMD="rpm -E %{python_sitelib}"
+fi
+
+cd $EXEC_PATH
+docker --version && cat /etc/resolv.conf
+
+mkdir -p $EXPORT_DIR
+
+lsmod | grep -q ^openvswitch || modprobe openvswitch || { echo 1>&2 "Please run 'modprobe openvswitch' as root"; exit 1; }
+
+CONTAINER_ID="$(docker run --privileged -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v $PROJECT_PATH:/workspace/nmstate -v $EXPORT_DIR:$CONT_EXPORT_DIR $DOCKER_IMAGE)"
+[ -n "$debug_exit_shell" ] && trap open_shell EXIT || trap run_exit EXIT
+docker_exec 'while ! systemctl is-active dbus; do sleep 1; done'
+docker_exec '
+    systemctl start NetworkManager
+    while ! systemctl is-active NetworkManager; do sleep 1; done
+'
+pyclean
+
+dump_network_info
+install_nmstate
+add_extra_networks
+dump_network_info
+run_tests