2019-07-22 19:50:18 +03:00
@Library('github.com/coreos/coreos-ci-lib@master') _
2019-12-13 20:12:53 +03:00
COSA_IMAGE = 'quay.io/coreos-assembler/coreos-assembler:latest'
COSA_BUILDROOT_IMAGE = 'registry.svc.ci.openshift.org/coreos/cosa-buildroot:latest'
2019-07-22 19:50:18 +03:00
stage("Build") {
parallel rpms: {
2019-12-13 20:12:53 +03:00
coreos.pod(image: COSA_BUILDROOT_IMAGE, runAsUser: 0) {
2019-07-22 19:50:18 +03:00
checkout scm
sh """
set -euo pipefail
2019-12-13 20:14:32 +03:00
# fetch tags so `git describe` gives a nice NEVRA when building the RPM
git fetch origin --tags
2019-07-22 19:50:18 +03:00
ci/installdeps.sh
git submodule update --init
# We lose sanitizers (all the *san) here by building straight to RPMs, but we can
# restore those through a build opt later on. Being able to stash RPMs directly is
# super nice (and archiving later on will make it easy for anyone to download
# binaries from PRs in the future) and meshes well with the following stages.
export PATH="/root/.cargo/bin:\$PATH"
cargo install cbindgen
cbindgen -c rust/cbindgen.toml -o rpmostree-rust.h rust
cd packaging
make -f Makefile.dist-packaging rpm
"""
2019-12-13 20:14:32 +03:00
// make it easy for anyone to download the RPMs
archiveArtifacts 'packaging/**/*.rpm'
2019-07-22 19:50:18 +03:00
stash includes: 'packaging/**/*.rpm', name: 'rpms'
}
},
codestyle: {
2019-12-13 20:12:53 +03:00
coreos.pod(image: COSA_IMAGE) {
2019-10-03 22:14:11 +03:00
checkout scm
2019-07-22 19:50:18 +03:00
sh """
set -euo pipefail
# Jenkins by default only fetches the branch it's testing. Explicitly fetch master
# for ci-commitmessage-submodules.sh
git fetch origin +refs/heads/master:refs/remotes/origin/master
2019-10-03 22:14:11 +03:00
ci/ci-commitmessage-submodules.sh
2019-07-22 19:50:18 +03:00
ci/codestyle.sh
"""
}
},
2019-12-20 22:16:24 +03:00
rust: {
2019-12-13 20:12:53 +03:00
coreos.pod(image: COSA_BUILDROOT_IMAGE, runAsUser: 0) {
2019-07-22 19:50:18 +03:00
checkout scm
sh """
set -euo pipefail
2019-09-27 18:06:01 +03:00
ci/msrv.sh
2019-12-20 22:16:24 +03:00
cd rust && cargo test
2019-07-22 19:50:18 +03:00
"""
}
}}
stage("Build FCOS") {
2019-12-13 20:12:53 +03:00
coreos.pod(image: COSA_IMAGE, runAsUser: 0, kvm: true) {
2019-07-22 19:50:18 +03:00
unstash 'rpms'
sh """
set -euo pipefail
2019-12-13 20:16:40 +03:00
rpms=\$(find packaging/ ! -name '*.src.rpm' -name '*.rpm')
2019-07-22 19:50:18 +03:00
# install our built rpm-ostree
2019-12-13 20:16:40 +03:00
dnf install -y \${rpms}
2019-07-22 19:50:18 +03:00
2019-12-13 20:16:40 +03:00
# and build FCOS with our built rpm-ostree inside of it
2019-07-22 19:50:18 +03:00
coreos-assembler init --force https://github.com/coreos/fedora-coreos-config
2019-12-13 20:16:40 +03:00
mkdir -p overrides/rpm
mv \${rpms} overrides/rpm
rm -rf packaging
2019-07-22 19:50:18 +03:00
coreos-assembler build
"""
stash includes: 'builds/latest/*/*.qcow2', name: 'fcos'
}
}
Rework vmcheck to use `kola spawn`, move off of PAPR
There's a lot going on here, but essentially:
1. We change the `vmcheck` model so that it always operates on an
immutable base image. It takes that image and dynamically launches a
separate VM for each test using `kola spawn`. This means we can drop
a lot of hacks around re-using the same VMs.
2. Following from 1., `vmoverlay` now takes as input a base image,
overlays the built rpm-ostree bits, then creates a new base image. Of
course, we don't have to do this in CI, because we build FCOS with
the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay`
then will be more for the developer case where one doesn't want to
iterate via `cosa build` to test rpm-ostree changes. I say "will"
because the functionality doesn't exist yet; I'd like to enhance
`cosa dev-overlay` to do this. (Note `vmsync` should still works just
as before too.)
3. `vmcheck` can be run without building the tree first, as
`tests/vmcheck.sh`. The `make vmcheck` target still exists though for
finger compatibility and better meshing with `vmoverlay` in the
developer case.
What's really nice about using kola spawn is that it takes care of a lot
of things for us, such as the qemu command, journal and console
gathering, and SSH.
Similarly to the compose testsuites, we're using parallel here to run
multiple vmcheck tests at once. (On developer laptops, we cap
parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
stage("Run vmcheck") {
def nhosts = 6
def mem = (nhosts * 1024) + 512
coreos.pod(image: COSA_IMAGE, runAsUser: 0, kvm: true, memory: "${mem}Mi", cpu: "${nhosts}") {
2019-07-22 19:50:18 +03:00
checkout scm
unstash 'rpms'
sh """
set -euo pipefail
Rework vmcheck to use `kola spawn`, move off of PAPR
There's a lot going on here, but essentially:
1. We change the `vmcheck` model so that it always operates on an
immutable base image. It takes that image and dynamically launches a
separate VM for each test using `kola spawn`. This means we can drop
a lot of hacks around re-using the same VMs.
2. Following from 1., `vmoverlay` now takes as input a base image,
overlays the built rpm-ostree bits, then creates a new base image. Of
course, we don't have to do this in CI, because we build FCOS with
the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay`
then will be more for the developer case where one doesn't want to
iterate via `cosa build` to test rpm-ostree changes. I say "will"
because the functionality doesn't exist yet; I'd like to enhance
`cosa dev-overlay` to do this. (Note `vmsync` should still works just
as before too.)
3. `vmcheck` can be run without building the tree first, as
`tests/vmcheck.sh`. The `make vmcheck` target still exists though for
finger compatibility and better meshing with `vmoverlay` in the
developer case.
What's really nice about using kola spawn is that it takes care of a lot
of things for us, such as the qemu command, journal and console
gathering, and SSH.
Similarly to the compose testsuites, we're using parallel here to run
multiple vmcheck tests at once. (On developer laptops, we cap
parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
ci/installdeps.sh # really, we just need test deps, but meh...
2019-07-22 19:50:18 +03:00
# install our built rpm-ostree
find packaging/ ! -name '*.src.rpm' -name '*.rpm' | xargs dnf install -y
rm -rf packaging
"""
Rework vmcheck to use `kola spawn`, move off of PAPR
There's a lot going on here, but essentially:
1. We change the `vmcheck` model so that it always operates on an
immutable base image. It takes that image and dynamically launches a
separate VM for each test using `kola spawn`. This means we can drop
a lot of hacks around re-using the same VMs.
2. Following from 1., `vmoverlay` now takes as input a base image,
overlays the built rpm-ostree bits, then creates a new base image. Of
course, we don't have to do this in CI, because we build FCOS with
the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay`
then will be more for the developer case where one doesn't want to
iterate via `cosa build` to test rpm-ostree changes. I say "will"
because the functionality doesn't exist yet; I'd like to enhance
`cosa dev-overlay` to do this. (Note `vmsync` should still works just
as before too.)
3. `vmcheck` can be run without building the tree first, as
`tests/vmcheck.sh`. The `make vmcheck` target still exists though for
finger compatibility and better meshing with `vmoverlay` in the
developer case.
What's really nice about using kola spawn is that it takes care of a lot
of things for us, such as the qemu command, journal and console
gathering, and SSH.
Similarly to the compose testsuites, we're using parallel here to run
multiple vmcheck tests at once. (On developer laptops, we cap
parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
unstash 'fcos'
try {
timeout(time: 30, unit: 'MINUTES') {
sh """
set -xeuo pipefail
fcos=\$(ls builds/latest/*/*.qcow2) # */
ln -sf "\$(realpath \${fcos})" tests/vmcheck/image.qcow2
NHOSTS=${nhosts} tests/vmcheck.sh
"""
}
} finally {
sh """
if [ -d vmcheck-logs ]; then
tar -C vmcheck-logs -cf- . | xz -c9 > vmcheck-logs.tar.xz
fi
"""
archiveArtifacts allowEmptyArchive: true, artifacts: 'vmcheck-logs.tar.xz'
}
2019-07-22 19:50:18 +03:00
}
Rework vmcheck to use `kola spawn`, move off of PAPR
There's a lot going on here, but essentially:
1. We change the `vmcheck` model so that it always operates on an
immutable base image. It takes that image and dynamically launches a
separate VM for each test using `kola spawn`. This means we can drop
a lot of hacks around re-using the same VMs.
2. Following from 1., `vmoverlay` now takes as input a base image,
overlays the built rpm-ostree bits, then creates a new base image. Of
course, we don't have to do this in CI, because we build FCOS with
the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay`
then will be more for the developer case where one doesn't want to
iterate via `cosa build` to test rpm-ostree changes. I say "will"
because the functionality doesn't exist yet; I'd like to enhance
`cosa dev-overlay` to do this. (Note `vmsync` should still works just
as before too.)
3. `vmcheck` can be run without building the tree first, as
`tests/vmcheck.sh`. The `make vmcheck` target still exists though for
finger compatibility and better meshing with `vmoverlay` in the
developer case.
What's really nice about using kola spawn is that it takes care of a lot
of things for us, such as the qemu command, journal and console
gathering, and SSH.
Similarly to the compose testsuites, we're using parallel here to run
multiple vmcheck tests at once. (On developer laptops, we cap
parallelism at `$(nproc) - 1`).
2019-12-13 20:23:41 +03:00
}