Commit Graph

89 Commits

Author SHA1 Message Date
Colin Walters
86ce9ea1f5 ci: Make msrv test do full build + unit tests
I think we did this at some point, but then stopped.
Prep for https://github.com/coreos/rpm-ostree/pull/2413
because we'll need a full build of the C++ side too in order
to `cargo test`.
2021-01-26 10:31:57 +01:00
Luca BRUNO
85640d44f6 ci: fix install-extra logic
This fixes `install-extra-builddeps.sh` helper, by letting cargo
detect whether the target binary is already present in the
environment with the expected version.
This is in order to avoid mismatches in generated code when the
library version is bumped, and stale binaries are present on
the system.
2021-01-25 17:05:53 +01:00
Colin Walters
08c414f897 Rework bindgen/cxx.rs usage and CI build
cxx.rs (aka cxxbridge) and cbindgen are
both generating source code.  Since the last release
we've introduced the former, and we need to ensure
that the generated cxx.rs source ends up in release tarballs
the same way as the cbindgen code.

Rationalize and clean up the binding infrastructure.
Drop support for the vendored cbindgen which we
weren't actually using:
Closes: https://github.com/coreos/rpm-ostree/issues/2392

Move the cxx-rs and cbindgen bits into the same place,
and update our CoreOS CI build to use a separate `Makefile.bindings`
that just generates the code, so our CI still "works like"
a main Koji RPM build.
2021-01-04 13:17:35 +01:00
Jonathan Lebon
c96ad53dcf ci: Add ~/.cargo/bin to PATH in Prow path as well
This matches what we do in CoreOS CI.
2020-12-23 22:52:40 +01:00
Jonathan Lebon
8461c053c1 ci/install-extra-builddeps: Drop --root=/usr arg
AFAICT, we just need `cxxbridge` to be in the `PATH` of the building
user. Let's avoid targeting privileged paths so devs can just run this
script directly without `sudo`.
2020-12-23 22:52:40 +01:00
Colin Walters
9565c19ef0 Use cxx-rs for core.rs
This is much better than bindgen because it's fully safe.  It's
much more ergonomic too:

 - Invoke Rust methods-on-structs just like C++ methods-on-structs
 - Rust `Result<>` is translated automatically to exceptions

See https://cxx.rs/context.html for more.
2020-12-23 17:45:29 +01:00
Colin Walters
9f19ed2ac8 ci: Introduce install-extra-builddeps.sh
We need to cleanly split off "test dependencies" that we
install inside the cosa pod from builds (where we won't
have `cargo`) from the build time where we use the cosa
buildroot image.

Prep for using https://cxx.rs
2020-12-23 17:45:29 +01:00
Colin Walters
a2c5d1e659 Run rustfmt, and fix CI check for it
Moving the crate to the toplevel broke the CI check for `rustfmt`.
Fix it and run `cargo fmt`.
2020-12-21 19:59:31 -05:00
Colin Walters
b3b4dd3d22 msrv: Bump to Rust 1.48.0
We need this for https://cxx.rs

While we're here:

 - Add some more comments/links
 - Since the Rust bits are now at the toplevel, we can explicitly
   invoke `cargo`
 - And since we can do that, use the `+` syntax to specify the
   toolchain explicitly
2020-12-15 16:17:44 +00:00
Luca BRUNO
b4e4005f56 sh: fix bash shebang
This updates a couple of shebangs for scripts using bash `pipefail`
option.
2020-11-11 16:50:43 +01:00
Jonathan Lebon
b91e6bc9a3 ci: Run C unit tests too
We lost this during the transition from PAPR to CoreOS CI. We don't have
a lot of new tests there since new unit tests tend to be in Rust, though
we should still run what we do have.

Repurpose the `rust` branch to more generically run all unit tests
and not just the Rust ones. It still also checks that compilation
against the MSRV works fine.
2020-10-01 06:08:37 -04:00
Jonathan Lebon
31799978c1 ci: Bump MSRV to 1.45.2
This is in el8 now and it unblocks a bump to the latest systemd crate
version.
2020-08-24 16:45:35 -04:00
Jonathan Lebon
1c9ea5dab3 ci: Bump MSRV to 1.43.1
This is available in the RHEL 8.2 buildroot now. Also fixes compilation
with latest `openat-ext` crate which uses `std::usize::MAX`.
2020-07-27 21:14:43 +02:00
Jonathan Lebon
fda0be62ce ci: Constrain parallel build jobs
The default `_NPROCESSORS_ONLN` heuristic we have isn't cgroups aware.
So it thinks it has e.g. 40 CPUs when running in a k8s pod. This can
then blow through our allocated resource limits.

Declare some modest amount of RAM and CPU resources and override `make`
parallelism.

This matches what ostree does in
https://github.com/ostreedev/ostree/pull/2151.
2020-07-16 15:46:06 -04:00
Jonathan Lebon
2441751d90 ci/installdeps: add distribution-gpg-keys
This is needed by the compose tests. This is part of cosa already, which
is why CI isn't hitting this, but we want to support users not
developing on top of the cosa buildroot container that can just run
`ci/installdeps.sh`.
2020-06-24 16:45:28 -04:00
Jonathan Lebon
fc5825a8f9 ci-commitmessage-submodules: Ignore empty merge commits
Jenkins does its own `git merge` when testing PRs. Doing a naive
`git diff ${merge_commit}^..${merge_commit}` won't work right because
it might perform a diff across multiple commits.

What we want to do here is to just skip trivial merge commits or
otherwise error out on them if they're non-trivial (since it likely
means that one did conflict resolution manually instead of rebasing,
which we should encourage).

The `origin/master..$HEAD` range will correctly still contain all the
parents of any merge commit which is not yet in `origin/master`.
2020-05-07 22:21:16 +02:00
Colin Walters
8a172a2e05 rust: rustfmt(*) and (re)add a CI check for it
We haven't been consistent about doing this; I personally
think rustfmt is a big aggressive with the line wrapping
but eh, consistency is more important.

And heh so I tried to `git push --set-upstream cgwalters` and
that failed because there was an already extant `rustfmt`
branch from a while ago...looking at that code it got lost
in the CI refactoring - we're not running `build-check.sh`
at the moment.

Move the rustfmt bits into `codestyle.sh` which is closer
to where it should be anyways.
2020-04-08 02:52:30 +02:00
Jonathan Lebon
e6907d209b ci: drop FAHC repo in favour of f31-coreos-continuous
FAHC is super out of date now. The way to have access to newer packages
is via the continuous tag, which is still manual for now, but at least
targets the right Fedora release.
2020-03-13 23:13:44 +01:00
Jonathan Lebon
69e3af4331 ci/installdeps: Only builddep from spec file
Doing builddep once based on the baked config and then once more from
the spec file can cause issues sometimes. For example, right now the
latest rpm-ostree release uses libmodulemd1, but we want to rebase to
libmodulemd (2.0). And `dnf` will get confused trying to move from one
to the other.

Really, we don't need to builddep from the last release at all, so just
drop that and rely only on the spec file.

Adapt `pkg_install_builddeps` to allow no args to mean only installing
the basic buildroot stuff like `dnf builddep` and `@buildsys-build`.
2020-02-25 16:48:15 +01:00
Jonathan Lebon
4b15c59b77 ci: Move cargo test into ci/msrv.sh
We need `cargo` in our `PATH` and we already do the `PATH=...` dance in
`ci/msrv.sh`. This only worked before because we were inadvertedly
re-installing cargo when calling `ci/build.sh`, which was fixed in the
previous commit.
2020-02-25 16:48:15 +01:00
Jonathan Lebon
2290999251 ci/msrv.sh: Use SKIP_INSTALLDEPS=1 before building
We've already manually installed dependencies higher up. This saves us
some time, but also we don't want the script to e.g. re-install cargo.
(This also works as a short term hack we need to adapt to libdnf moving
to `libmodulemd-2.0` due to `ci/installdeps.sh` not being entirely
idempotent).
2020-02-25 16:48:15 +01:00
Jonathan Lebon
7501b39cbf spec: Bump libmodulemd version requirement
This matches the latest bump in libdnf:
763ccc3e33
2020-02-25 16:48:15 +01:00
Jonathan Lebon
a40014f5a7 ci: Allow submodule bumps from Dependabot
Tweak the logic in `ci-commitmessage-submodules.sh` to allow Dependabot
bumps.
2020-01-28 13:14:47 -08:00
Jonathan Lebon
13f554ea5c ci: Bump MSRV to 1.39.0
This is in el8 now.
2019-12-20 21:16:24 +01:00
Colin Walters
75a2f8bb33 ci/prow: Use cosa buildroot
See https://github.com/coreos/coreos-assembler/pull/730

- Skip installing deps, hooray!
- Also skip the C test suite for now because of the linked fuse-overlayfs issue
2019-10-02 08:16:10 -07:00
Colin Walters
226b939d29 ci/prow: Use build-check.sh
We want to run the unit tests after a build, so do that.
Furthermore, this actually installs the binaries, which
means if we do it as part of a `Dockerfile` build, we
can use that image for further testing.
2019-10-02 08:16:10 -07:00
Jonathan Lebon
677c3c8b29 ci: Also bump MSRV to 1.37.0 for CCI Jenkins
Just split it out into a separate script for easier sharing.
2019-10-01 11:26:29 -04:00
Jonathan Lebon
46ab7d1ae8 ci: Add Jenkins pipeline
This is an experiment in using Jenkins pipelines for our CI. See similar
initiatives in coreos-assembler[1] and fedora-coreos-config[2].

For now, this only does the following testing:
- checks commit for unintended submodule bumps
- checks the minimum Rust version
- builds RPMs
- builds FCOS (with the new RPMs both for executing the build
  itself, as well as included in the built OS)

There are dummy placeholders for where we'd actually run the vmcheck
and the compose testsuites. Let's address those trickier parts as
follow-ups.

[1] https://github.com/coreos/coreos-assembler/pull/667
[2] https://github.com/coreos/fedora-coreos-config/pull/131

Closes: #1899
Approved by: cgwalters
2019-09-18 15:15:28 +00:00
Jonathan Lebon
e11d426f99 packaging: Move canonical spec file here
So this is a somewhat significant change, but I'd like to try having the
canonical spec file upstream. A few reasons for this:

1. We integrate tightly with the distros we're destined for, and so
   we're in a pretty good position for knowing how the software should
   be packaged.
2. We can atomically change packaging along with the rest of the code.
   This has important ramifications, including that it'll be easier to
   integrate with continuous build services like Packit, but releases
   will also be less fraught with last-minute packaging fixes.
3. I'm playing with Jenkins pipelines and there I'd like to make RPMs
   the "artifact" that gets moved down the pipeline into later stages
   (e.g. `cosa build`). We could even eventually make it an actual
   external artifact so that anyone can easily download RPMs from any
   random PR for testing. (And in fact, with a thin yumrepo layer on
   top, it could be used to replace Packit/rdgo entirely).

Not that this approach doesn't have issues as well (e.g. on the dist-git
side, we'll need some minimal tooling to merge in the changelog), though
I think it's worth trying out.

Closes: #1900
Approved by: cgwalters
2019-09-09 23:50:32 +00:00
Jonathan Lebon
e589161c56 ci/installdeps.sh: Pull ostree from rdgo on f30 too
This is a bit of a hack, but does the trick now. Eventually, we'll need
to either bump rdgo to f30, or set up continuous builds some other way
(e.g. Packit or Jenkins pipelines).

Closes: #1900
Approved by: cgwalters
2019-09-09 23:50:32 +00:00
Jonathan Lebon
bc50cacde3 ci/installdeps.sh: Nuke temporary libdnf dep workaround
These are part of the latest release now, so we can nuke them and skip
one `dnf install` invocation.

Closes: #1900
Approved by: cgwalters
2019-09-09 23:50:32 +00:00
Jonathan Lebon
c7814dd0ed ci: Allow passing HEAD commit to ci-commitmessage-submodules.sh
Jenkins' `checkout scm` gives us the exact commit that we're testing.
Allow passing that to the submodule commit message checker so it doesn't
use `HEAD` (which is likely GitHub's merge commit).

Closes: #1900
Approved by: cgwalters
2019-09-09 23:50:32 +00:00
Jonathan Lebon
6d017644d7 ci: Some more el7 path pruning
Use `dnf` consistently. Drop a couple of other conditionals.

Closes: #1900
Approved by: cgwalters
2019-09-09 23:50:32 +00:00
Colin Walters
2232edb6b0 ci: Add prow Dockerfile
This is an exact copy of https://github.com/ostreedev/ostree/pull/1906
for now.  From that commit message:

I'd like to add OpenShift's prow to this repository. Let's start
by adding a Dockerfile - it doesn't really do anything besides build.

However...I've lately been thinking about e.g. shipping the ostree tests
as an image, and then e.g. we could test FCOS by running that container
(which would orchestrate the host's ostree).

Anyways, not doing that right now but this is a start.
2019-08-28 16:28:22 -04:00
Colin Walters
0d841b2e2f ci: More fixes for cosa build
Use --entrypoint and also rename it while we're here.

Closes: #1889
Approved by: rfairley
2019-08-24 00:35:44 +00:00
Colin Walters
1027ade4db ci: Just use cosa as a buildroot too
Since we have to pull it anyways to do a build, let's use it
as a buildroot.  This should fix CI which broke because we were
doing a f29 build but cosa is f30, and libostree differed.
2019-08-22 20:13:41 +00:00
Jonathan Lebon
035ac2eaa6 tests/vmcheck: Fully drop python 3 requirement
Drop the use of Ansible everywhere. In the few cases where we really
Python, just spawn a container instead.

This is required to be able to hack on Fedora CoreOS.

Closes: #1850
Approved by: jlebon
2019-06-10 15:36:06 +00:00
Jonathan Lebon
206ae24d4e tests: Bump to Python 3 only
This bumps the requirement on the controlling host to Python 3 only.
It also bumps the requirement on the target host to Python 3 as well
since FCOS doesn't ship Python 2 right now.

Though we'll need to eventually drop all Python usage anyway, but at
least let's get tests passing on FCOS first. (See related previous
patch).

Closes: #1828
Approved by: cgwalters
2019-05-08 19:02:32 +00:00
Jonathan Lebon
ad1451fac2 Rebase to latest libdnf
This brings us back in sync with the latest libdnf git master. This
required a bunch of work both on the libdnf and rpm-ostree side to get
working. See e.g.
https://github.com/rpm-software-management/libdnf/issues/645.

A few things to adapt to:

- soname bump to `libdnf.so.2`
- `DnfAdvisory` is no longer a `GObject` (annoyingly it's not replaced
  by something we can keep a ref on, so this requires some hacks to
  steal from the `GPtrArray` -- could enhance libdnf for this later)
- disable SWDB history writing
- use new reldep public API
- update for latest `hy_subject_get_best_selector()` API

This now unlocks the possibility to add support for modules. (One can
see hints of this in the diff by the fact that `libdnf` links to
`libmodulemd1`.)

Update submodule: libdnf

Closes: #1404
Approved by: cgwalters
2019-03-19 14:29:15 +00:00
Jonathan Lebon
6cdcd474b6 Makefile-libs.am: Work around g-ir-scanner issues with clang
Right now there's an issue in Fedora with `g-ir-scanner` picking up
`-fstack-clash-protection` from the `sysconfig` Python module and
passing it to `clang`, which doesn't understand this flag yet.

Just work around this by (1) not even building GIR bindings for our
bundled libdnf since there's no need, and (2) overridding the compiler
used by `g-ir-scanner` so it's always `gcc`.

See: https://github.com/projectatomic/rpm-ostree/pull/1787#issuecomment-473971585

Closes: #1787
Approved by: cgwalters
2019-03-19 12:19:38 +00:00
Jonathan Lebon
d34f24581a ci: Fix repo editing in installdeps.sh
- Use `IFS=` so that we don't drop leading whitespace when reading in,
  which can be used to specify multiple baseurls on multiple lines.
- Use `-r` as good practice so we don't try to interpret anything.
- Use `echo "$line"` so that we don't drop leading whitespace when
  writing out.

Closes: #1787
Approved by: cgwalters
2019-03-19 12:19:38 +00:00
Jonathan Lebon
d0fdd9ac1d ci: Drop c7 support for installdeps.sh
Closes: #1787
Approved by: cgwalters
2019-03-19 12:19:38 +00:00
Jonathan Lebon
4ae3b174f5 ci: Bump to f29
Better late than never!

Closes: #1787
Approved by: cgwalters
2019-03-19 12:19:38 +00:00
Jonathan Lebon
6b928e8f0e ci: Build the whole tree, not just rust bits
Otherwise, we're not testing everything. E.g. the cbindgen bits.

Closes: #1778
Approved by: cgwalters
2019-03-07 21:57:08 +00:00
Colin Walters
b63f3720b8 ci: Add a f29 FCOS build
Let's start doing testing using coreos-assembler + Fedora CoreOS.

Closes: #1774
Approved by: jlebon
2019-03-07 19:16:12 +00:00
Colin Walters
387da3a420 ci: Verify rustfmt
Split out the command into a separate `Makefile` that doesn't
use Automake so we can invoke it early.

Closes: #1674
Approved by: dustymabe
2018-11-21 21:16:03 +00:00
Jonathan Lebon
dbdbaf87ee ci: Make adduser invocation idempotent
This is a bit of a selfish request, though I also really like
`ci/build.sh` for hacking in a throwaway environment. This patch allows
one to rerun `ci/build.sh` without erroring out on `testuser` already
existing.

Closes: #1621
Approved by: cgwalters
2018-10-16 16:03:06 +00:00
Jonathan Lebon
cc69168aaf ci: Drop CI_PKGS and epel-release in .papr.yml
Move specific package mentions from the `.papr.yml` file to
`ci/installdeps.sh`. I find the latter script super useful for
bootstrapping a pet container for rpm-ostree development and this helps
with that.

Closes: #1621
Approved by: cgwalters
2018-10-16 16:03:06 +00:00
Jonathan Lebon
e15baebc68 ci: Split out script to install deps
This should also make it useful for bootstrapping a test container
locally.

Closes: #1606
Approved by: cgwalters
2018-10-11 20:39:57 +00:00
Jonathan Lebon
8f504ec5ce ci/build.sh: Drop unnecessary pkg_install cargo
The latest releases in the stable repos now include the new cargo BR, so
we can drop this now.

Closes: #1606
Approved by: cgwalters
2018-10-11 20:39:57 +00:00