keremet/rpm-ostree
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
456a3ec7c2
rpm-ostree/tests/vmcheck/overlay.sh

34 lines
973 B
Bash
Raw Normal View History

Redo `vmcheck` to abstract over ssh-config, drop all building My development environment is now using "pet" docker containers. I use VMs for testing things that require that (like rpm-ostree). This patch builds on work from @jlebon in https://github.com/projectatomic/rpm-ostree/pull/509 to rework `vmcheck` such that it can work on any `ssh-config`. By default we expect this to be Vagrant. However, I go a lot farther and delete the `vmbuild` code that was trying to do builds in a container on the target VM. I think this is still worth pursuing at some point, but for now I think it's reasonable to assume that the rpm-ostree developer audience uses Linux as their host workstation and hence has containers. (There's another important point here in that for developing lower level things like rpm-ostree, there's a strong push to make the VM disposable and not a pet) Closes: #516 Approved by: jlebon
2016-11-14 20:07:43 +03:00
#!/bin/bash
Rework vmcheck to use `kola spawn`, move off of PAPR There's a lot going on here, but essentially: 1. We change the `vmcheck` model so that it always operates on an immutable base image. It takes that image and dynamically launches a separate VM for each test using `kola spawn`. This means we can drop a lot of hacks around re-using the same VMs. 2. Following from 1., `vmoverlay` now takes as input a base image, overlays the built rpm-ostree bits, then creates a new base image. Of course, we don't have to do this in CI, because we build FCOS with the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay` then will be more for the developer case where one doesn't want to iterate via `cosa build` to test rpm-ostree changes. I say "will" because the functionality doesn't exist yet; I'd like to enhance `cosa dev-overlay` to do this. (Note `vmsync` should still works just as before too.) 3. `vmcheck` can be run without building the tree first, as `tests/vmcheck.sh`. The `make vmcheck` target still exists though for finger compatibility and better meshing with `vmoverlay` in the developer case. What's really nice about using kola spawn is that it takes care of a lot of things for us, such as the qemu command, journal and console gathering, and SSH. Similarly to the compose testsuites, we're using parallel here to run multiple vmcheck tests at once. (On developer laptops, we cap parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
set -euo pipefail
Redo `vmcheck` to abstract over ssh-config, drop all building My development environment is now using "pet" docker containers. I use VMs for testing things that require that (like rpm-ostree). This patch builds on work from @jlebon in https://github.com/projectatomic/rpm-ostree/pull/509 to rework `vmcheck` such that it can work on any `ssh-config`. By default we expect this to be Vagrant. However, I go a lot farther and delete the `vmbuild` code that was trying to do builds in a container on the target VM. I think this is still worth pursuing at some point, but for now I think it's reasonable to assume that the rpm-ostree developer audience uses Linux as their host workstation and hence has containers. (There's another important point here in that for developing lower level things like rpm-ostree, there's a strong push to make the VM disposable and not a pet) Closes: #516 Approved by: jlebon
2016-11-14 20:07:43 +03:00
Rework vmcheck to use `kola spawn`, move off of PAPR There's a lot going on here, but essentially: 1. We change the `vmcheck` model so that it always operates on an immutable base image. It takes that image and dynamically launches a separate VM for each test using `kola spawn`. This means we can drop a lot of hacks around re-using the same VMs. 2. Following from 1., `vmoverlay` now takes as input a base image, overlays the built rpm-ostree bits, then creates a new base image. Of course, we don't have to do this in CI, because we build FCOS with the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay` then will be more for the developer case where one doesn't want to iterate via `cosa build` to test rpm-ostree changes. I say "will" because the functionality doesn't exist yet; I'd like to enhance `cosa dev-overlay` to do this. (Note `vmsync` should still works just as before too.) 3. `vmcheck` can be run without building the tree first, as `tests/vmcheck.sh`. The `make vmcheck` target still exists though for finger compatibility and better meshing with `vmoverlay` in the developer case. What's really nice about using kola spawn is that it takes care of a lot of things for us, such as the qemu command, journal and console gathering, and SSH. Similarly to the compose testsuites, we're using parallel here to run multiple vmcheck tests at once. (On developer laptops, we cap parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
. ${commondir}/libvm.sh
vmcheck: adapt for non-vagrant hosts This is just the final bit required to make sure the vagrant and non-vagrant paths can work happily together. It's mostly minor fixes, though the most major change which also affects vagrant is that we now sync to the root home dir, rather than ~vagrant. Closes: #524 Approved by: cgwalters
2016-11-22 06:22:51 +03:00
Rework vmcheck to use `kola spawn`, move off of PAPR There's a lot going on here, but essentially: 1. We change the `vmcheck` model so that it always operates on an immutable base image. It takes that image and dynamically launches a separate VM for each test using `kola spawn`. This means we can drop a lot of hacks around re-using the same VMs. 2. Following from 1., `vmoverlay` now takes as input a base image, overlays the built rpm-ostree bits, then creates a new base image. Of course, we don't have to do this in CI, because we build FCOS with the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay` then will be more for the developer case where one doesn't want to iterate via `cosa build` to test rpm-ostree changes. I say "will" because the functionality doesn't exist yet; I'd like to enhance `cosa dev-overlay` to do this. (Note `vmsync` should still works just as before too.) 3. `vmcheck` can be run without building the tree first, as `tests/vmcheck.sh`. The `make vmcheck` target still exists though for finger compatibility and better meshing with `vmoverlay` in the developer case. What's really nice about using kola spawn is that it takes care of a lot of things for us, such as the qemu command, journal and console gathering, and SSH. Similarly to the compose testsuites, we're using parallel here to run multiple vmcheck tests at once. (On developer laptops, we cap parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
# Thin wrapper around `cosa dev-overlay`.
vmcheck: adapt for non-vagrant hosts This is just the final bit required to make sure the vagrant and non-vagrant paths can work happily together. It's mostly minor fixes, though the most major change which also affects vagrant is that we now sync to the root home dir, rather than ~vagrant. Closes: #524 Approved by: cgwalters
2016-11-22 06:22:51 +03:00
Rework vmcheck to use `kola spawn`, move off of PAPR There's a lot going on here, but essentially: 1. We change the `vmcheck` model so that it always operates on an immutable base image. It takes that image and dynamically launches a separate VM for each test using `kola spawn`. This means we can drop a lot of hacks around re-using the same VMs. 2. Following from 1., `vmoverlay` now takes as input a base image, overlays the built rpm-ostree bits, then creates a new base image. Of course, we don't have to do this in CI, because we build FCOS with the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay` then will be more for the developer case where one doesn't want to iterate via `cosa build` to test rpm-ostree changes. I say "will" because the functionality doesn't exist yet; I'd like to enhance `cosa dev-overlay` to do this. (Note `vmsync` should still works just as before too.) 3. `vmcheck` can be run without building the tree first, as `tests/vmcheck.sh`. The `make vmcheck` target still exists though for finger compatibility and better meshing with `vmoverlay` in the developer case. What's really nice about using kola spawn is that it takes care of a lot of things for us, such as the qemu command, journal and console gathering, and SSH. Similarly to the compose testsuites, we're using parallel here to run multiple vmcheck tests at once. (On developer laptops, we cap parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
# First, we need to find the image to operate on.
if [ -n "${VMIMAGE:-}" ]; then
src_img=${VMIMAGE}
tests: Add RPMOSTREE_TEST_NO_OVERLAY to skip overlay The use case here is to run our tests against the shipped tree; this could be used by a-h-t for example: https://github.com/projectatomic/atomic-host-tests/issues/74 I tried this with just `TESTS=basic` and it failed for `usroverlay`. So we'd have to start adding feature detection to the test suite to make this truly useful, but let's at least start with the basic bits now to play with it. Closes: #1251 Approved by: jlebon
2018-02-14 17:55:39 +03:00
else
Rework vmcheck to use `kola spawn`, move off of PAPR There's a lot going on here, but essentially: 1. We change the `vmcheck` model so that it always operates on an immutable base image. It takes that image and dynamically launches a separate VM for each test using `kola spawn`. This means we can drop a lot of hacks around re-using the same VMs. 2. Following from 1., `vmoverlay` now takes as input a base image, overlays the built rpm-ostree bits, then creates a new base image. Of course, we don't have to do this in CI, because we build FCOS with the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay` then will be more for the developer case where one doesn't want to iterate via `cosa build` to test rpm-ostree changes. I say "will" because the functionality doesn't exist yet; I'd like to enhance `cosa dev-overlay` to do this. (Note `vmsync` should still works just as before too.) 3. `vmcheck` can be run without building the tree first, as `tests/vmcheck.sh`. The `make vmcheck` target still exists though for finger compatibility and better meshing with `vmoverlay` in the developer case. What's really nice about using kola spawn is that it takes care of a lot of things for us, such as the qemu command, journal and console gathering, and SSH. Similarly to the compose testsuites, we're using parallel here to run multiple vmcheck tests at once. (On developer laptops, we cap parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
basearch=$(cosa basearch)
cosa_builds=${COSA_BUILDS:-cosa-builds}
cosa_buildid=${COSA_BUILDID:-latest}
cosa_builddir=${cosa_builds}/${cosa_buildid}/${basearch}
if [ ! -e "${cosa_builddir}/meta.json" ]; then
tests: Misc tweaks - Have libvm.sh inherit libtest, otherwise we don't have `fatal` - Add `error: ` prefix to `fatal` messages for clarity - Add missing plural
2019-12-14 19:21:03 +03:00
fatal "No image provided (use VMIMAGE, or cosa-builds/ or COSA_BUILDS)"
Rework vmcheck to use `kola spawn`, move off of PAPR There's a lot going on here, but essentially: 1. We change the `vmcheck` model so that it always operates on an immutable base image. It takes that image and dynamically launches a separate VM for each test using `kola spawn`. This means we can drop a lot of hacks around re-using the same VMs. 2. Following from 1., `vmoverlay` now takes as input a base image, overlays the built rpm-ostree bits, then creates a new base image. Of course, we don't have to do this in CI, because we build FCOS with the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay` then will be more for the developer case where one doesn't want to iterate via `cosa build` to test rpm-ostree changes. I say "will" because the functionality doesn't exist yet; I'd like to enhance `cosa dev-overlay` to do this. (Note `vmsync` should still works just as before too.) 3. `vmcheck` can be run without building the tree first, as `tests/vmcheck.sh`. The `make vmcheck` target still exists though for finger compatibility and better meshing with `vmoverlay` in the developer case. What's really nice about using kola spawn is that it takes care of a lot of things for us, such as the qemu command, journal and console gathering, and SSH. Similarly to the compose testsuites, we're using parallel here to run multiple vmcheck tests at once. (On developer laptops, we cap parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
fi
vmcheck/overlay: use --keep-metadata for source-title One issue right now is that repeated applications of `make vmoverlay` will result in losing the source title. Let's use the new `--keep-metadata` in subsequent `make vmoverlay` invocations to make sure it gets transferred over correctly. Closes: #1205 Approved by: cgwalters
2018-01-12 23:55:34 +03:00
Rework vmcheck to use `kola spawn`, move off of PAPR There's a lot going on here, but essentially: 1. We change the `vmcheck` model so that it always operates on an immutable base image. It takes that image and dynamically launches a separate VM for each test using `kola spawn`. This means we can drop a lot of hacks around re-using the same VMs. 2. Following from 1., `vmoverlay` now takes as input a base image, overlays the built rpm-ostree bits, then creates a new base image. Of course, we don't have to do this in CI, because we build FCOS with the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay` then will be more for the developer case where one doesn't want to iterate via `cosa build` to test rpm-ostree changes. I say "will" because the functionality doesn't exist yet; I'd like to enhance `cosa dev-overlay` to do this. (Note `vmsync` should still works just as before too.) 3. `vmcheck` can be run without building the tree first, as `tests/vmcheck.sh`. The `make vmcheck` target still exists though for finger compatibility and better meshing with `vmoverlay` in the developer case. What's really nice about using kola spawn is that it takes care of a lot of things for us, such as the qemu command, journal and console gathering, and SSH. Similarly to the compose testsuites, we're using parallel here to run multiple vmcheck tests at once. (On developer laptops, we cap parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
cosa_qemu_path=$(jq -er '.images.qemu.path' "${cosa_builddir}/meta.json")
src_img=${cosa_builddir}/${cosa_qemu_path}
Initial support for automatic updates This patch introduces a new `AutomaticUpdatePolicy` configuration. This was a long time coming for rpm-ostree, given that its update model makes it extremely apt for such a feature. The config supports a `check` mode, which should be very useful to Atomic Workstation users, as well as a `reboot` mode, which could be used in its present form in simple single node Atomic Host situations. There is still a lot of work to be done, including integrating advisories, and supporting a `deploy` mode. This feature hopefully will be leveraged as well by higher-level projects like GNOME Software and Cockpit. Closes: #1147 Approved by: cgwalters
2017-12-15 20:01:46 +03:00
fi
Rework vmcheck to use `kola spawn`, move off of PAPR There's a lot going on here, but essentially: 1. We change the `vmcheck` model so that it always operates on an immutable base image. It takes that image and dynamically launches a separate VM for each test using `kola spawn`. This means we can drop a lot of hacks around re-using the same VMs. 2. Following from 1., `vmoverlay` now takes as input a base image, overlays the built rpm-ostree bits, then creates a new base image. Of course, we don't have to do this in CI, because we build FCOS with the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay` then will be more for the developer case where one doesn't want to iterate via `cosa build` to test rpm-ostree changes. I say "will" because the functionality doesn't exist yet; I'd like to enhance `cosa dev-overlay` to do this. (Note `vmsync` should still works just as before too.) 3. `vmcheck` can be run without building the tree first, as `tests/vmcheck.sh`. The `make vmcheck` target still exists though for finger compatibility and better meshing with `vmoverlay` in the developer case. What's really nice about using kola spawn is that it takes care of a lot of things for us, such as the qemu command, journal and console gathering, and SSH. Similarly to the compose testsuites, we're using parallel here to run multiple vmcheck tests at once. (On developer laptops, we cap parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
if [ -z "${SKIP_VMOVERLAY:-}" ]; then
# XXX: to develop
cosa dev-overlay --src-image "${src_img}" --add-tree insttree/ \
--output-dir vmoverlay/ --output-ref vmcheck
target_img=vmoverlay/$(jq -er '.images.qemu.path' "vmoverlay/meta.json")
vmcheck/overlay: add version and timestamp in source title This makes it much easier to know how old the base we're working from is and whether it's time to upgrade before hacking some more. Closes: #1205 Approved by: cgwalters
2018-01-12 23:55:34 +03:00
else
Rework vmcheck to use `kola spawn`, move off of PAPR There's a lot going on here, but essentially: 1. We change the `vmcheck` model so that it always operates on an immutable base image. It takes that image and dynamically launches a separate VM for each test using `kola spawn`. This means we can drop a lot of hacks around re-using the same VMs. 2. Following from 1., `vmoverlay` now takes as input a base image, overlays the built rpm-ostree bits, then creates a new base image. Of course, we don't have to do this in CI, because we build FCOS with the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay` then will be more for the developer case where one doesn't want to iterate via `cosa build` to test rpm-ostree changes. I say "will" because the functionality doesn't exist yet; I'd like to enhance `cosa dev-overlay` to do this. (Note `vmsync` should still works just as before too.) 3. `vmcheck` can be run without building the tree first, as `tests/vmcheck.sh`. The `make vmcheck` target still exists though for finger compatibility and better meshing with `vmoverlay` in the developer case. What's really nice about using kola spawn is that it takes care of a lot of things for us, such as the qemu command, journal and console gathering, and SSH. Similarly to the compose testsuites, we're using parallel here to run multiple vmcheck tests at once. (On developer laptops, we cap parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
target_img=${src_img}
vmcheck/overlay: add version and timestamp in source title This makes it much easier to know how old the base we're working from is and whether it's time to upgrade before hacking some more. Closes: #1205 Approved by: cgwalters
2018-01-12 23:55:34 +03:00
fi
Rework vmcheck to use `kola spawn`, move off of PAPR There's a lot going on here, but essentially: 1. We change the `vmcheck` model so that it always operates on an immutable base image. It takes that image and dynamically launches a separate VM for each test using `kola spawn`. This means we can drop a lot of hacks around re-using the same VMs. 2. Following from 1., `vmoverlay` now takes as input a base image, overlays the built rpm-ostree bits, then creates a new base image. Of course, we don't have to do this in CI, because we build FCOS with the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay` then will be more for the developer case where one doesn't want to iterate via `cosa build` to test rpm-ostree changes. I say "will" because the functionality doesn't exist yet; I'd like to enhance `cosa dev-overlay` to do this. (Note `vmsync` should still works just as before too.) 3. `vmcheck` can be run without building the tree first, as `tests/vmcheck.sh`. The `make vmcheck` target still exists though for finger compatibility and better meshing with `vmoverlay` in the developer case. What's really nice about using kola spawn is that it takes care of a lot of things for us, such as the qemu command, journal and console gathering, and SSH. Similarly to the compose testsuites, we're using parallel here to run multiple vmcheck tests at once. (On developer laptops, we cap parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
ln -sf "$(realpath ${target_img})" tests/vmcheck/image.qcow2
Reference in New Issue Copy Permalink