Commit Graph

76 Commits

Author SHA1 Message Date
Jonathan Lebon
fda0be62ce ci: Constrain parallel build jobs
The default `_NPROCESSORS_ONLN` heuristic we have isn't cgroups aware.
So it thinks it has e.g. 40 CPUs when running in a k8s pod. This can
then blow through our allocated resource limits.

Declare some modest amount of RAM and CPU resources and override `make`
parallelism.

This matches what ostree does in
https://github.com/ostreedev/ostree/pull/2151.
2020-07-16 15:46:06 -04:00
Jonathan Lebon
2441751d90 ci/installdeps: add distribution-gpg-keys
This is needed by the compose tests. This is part of cosa already, which
is why CI isn't hitting this, but we want to support users not
developing on top of the cosa buildroot container that can just run
`ci/installdeps.sh`.
2020-06-24 16:45:28 -04:00
Jonathan Lebon
fc5825a8f9 ci-commitmessage-submodules: Ignore empty merge commits
Jenkins does its own `git merge` when testing PRs. Doing a naive
`git diff ${merge_commit}^..${merge_commit}` won't work right because
it might perform a diff across multiple commits.

What we want to do here is to just skip trivial merge commits or
otherwise error out on them if they're non-trivial (since it likely
means that one did conflict resolution manually instead of rebasing,
which we should encourage).

The `origin/master..$HEAD` range will correctly still contain all the
parents of any merge commit which is not yet in `origin/master`.
2020-05-07 22:21:16 +02:00
Colin Walters
8a172a2e05 rust: rustfmt(*) and (re)add a CI check for it
We haven't been consistent about doing this; I personally
think rustfmt is a big aggressive with the line wrapping
but eh, consistency is more important.

And heh so I tried to `git push --set-upstream cgwalters` and
that failed because there was an already extant `rustfmt`
branch from a while ago...looking at that code it got lost
in the CI refactoring - we're not running `build-check.sh`
at the moment.

Move the rustfmt bits into `codestyle.sh` which is closer
to where it should be anyways.
2020-04-08 02:52:30 +02:00
Jonathan Lebon
e6907d209b ci: drop FAHC repo in favour of f31-coreos-continuous
FAHC is super out of date now. The way to have access to newer packages
is via the continuous tag, which is still manual for now, but at least
targets the right Fedora release.
2020-03-13 23:13:44 +01:00
Jonathan Lebon
69e3af4331 ci/installdeps: Only builddep from spec file
Doing builddep once based on the baked config and then once more from
the spec file can cause issues sometimes. For example, right now the
latest rpm-ostree release uses libmodulemd1, but we want to rebase to
libmodulemd (2.0). And `dnf` will get confused trying to move from one
to the other.

Really, we don't need to builddep from the last release at all, so just
drop that and rely only on the spec file.

Adapt `pkg_install_builddeps` to allow no args to mean only installing
the basic buildroot stuff like `dnf builddep` and `@buildsys-build`.
2020-02-25 16:48:15 +01:00
Jonathan Lebon
4b15c59b77 ci: Move cargo test into ci/msrv.sh
We need `cargo` in our `PATH` and we already do the `PATH=...` dance in
`ci/msrv.sh`. This only worked before because we were inadvertedly
re-installing cargo when calling `ci/build.sh`, which was fixed in the
previous commit.
2020-02-25 16:48:15 +01:00
Jonathan Lebon
2290999251 ci/msrv.sh: Use SKIP_INSTALLDEPS=1 before building
We've already manually installed dependencies higher up. This saves us
some time, but also we don't want the script to e.g. re-install cargo.
(This also works as a short term hack we need to adapt to libdnf moving
to `libmodulemd-2.0` due to `ci/installdeps.sh` not being entirely
idempotent).
2020-02-25 16:48:15 +01:00
Jonathan Lebon
7501b39cbf spec: Bump libmodulemd version requirement
This matches the latest bump in libdnf:
763ccc3e33
2020-02-25 16:48:15 +01:00
Jonathan Lebon
a40014f5a7 ci: Allow submodule bumps from Dependabot
Tweak the logic in `ci-commitmessage-submodules.sh` to allow Dependabot
bumps.
2020-01-28 13:14:47 -08:00
Jonathan Lebon
13f554ea5c ci: Bump MSRV to 1.39.0
This is in el8 now.
2019-12-20 21:16:24 +01:00
Colin Walters
75a2f8bb33 ci/prow: Use cosa buildroot
See https://github.com/coreos/coreos-assembler/pull/730

- Skip installing deps, hooray!
- Also skip the C test suite for now because of the linked fuse-overlayfs issue
2019-10-02 08:16:10 -07:00
Colin Walters
226b939d29 ci/prow: Use build-check.sh
We want to run the unit tests after a build, so do that.
Furthermore, this actually installs the binaries, which
means if we do it as part of a `Dockerfile` build, we
can use that image for further testing.
2019-10-02 08:16:10 -07:00
Jonathan Lebon
677c3c8b29 ci: Also bump MSRV to 1.37.0 for CCI Jenkins
Just split it out into a separate script for easier sharing.
2019-10-01 11:26:29 -04:00
Jonathan Lebon
46ab7d1ae8 ci: Add Jenkins pipeline
This is an experiment in using Jenkins pipelines for our CI. See similar
initiatives in coreos-assembler[1] and fedora-coreos-config[2].

For now, this only does the following testing:
- checks commit for unintended submodule bumps
- checks the minimum Rust version
- builds RPMs
- builds FCOS (with the new RPMs both for executing the build
  itself, as well as included in the built OS)

There are dummy placeholders for where we'd actually run the vmcheck
and the compose testsuites. Let's address those trickier parts as
follow-ups.

[1] https://github.com/coreos/coreos-assembler/pull/667
[2] https://github.com/coreos/fedora-coreos-config/pull/131

Closes: #1899
Approved by: cgwalters
2019-09-18 15:15:28 +00:00
Jonathan Lebon
e11d426f99 packaging: Move canonical spec file here
So this is a somewhat significant change, but I'd like to try having the
canonical spec file upstream. A few reasons for this:

1. We integrate tightly with the distros we're destined for, and so
   we're in a pretty good position for knowing how the software should
   be packaged.
2. We can atomically change packaging along with the rest of the code.
   This has important ramifications, including that it'll be easier to
   integrate with continuous build services like Packit, but releases
   will also be less fraught with last-minute packaging fixes.
3. I'm playing with Jenkins pipelines and there I'd like to make RPMs
   the "artifact" that gets moved down the pipeline into later stages
   (e.g. `cosa build`). We could even eventually make it an actual
   external artifact so that anyone can easily download RPMs from any
   random PR for testing. (And in fact, with a thin yumrepo layer on
   top, it could be used to replace Packit/rdgo entirely).

Not that this approach doesn't have issues as well (e.g. on the dist-git
side, we'll need some minimal tooling to merge in the changelog), though
I think it's worth trying out.

Closes: #1900
Approved by: cgwalters
2019-09-09 23:50:32 +00:00
Jonathan Lebon
e589161c56 ci/installdeps.sh: Pull ostree from rdgo on f30 too
This is a bit of a hack, but does the trick now. Eventually, we'll need
to either bump rdgo to f30, or set up continuous builds some other way
(e.g. Packit or Jenkins pipelines).

Closes: #1900
Approved by: cgwalters
2019-09-09 23:50:32 +00:00
Jonathan Lebon
bc50cacde3 ci/installdeps.sh: Nuke temporary libdnf dep workaround
These are part of the latest release now, so we can nuke them and skip
one `dnf install` invocation.

Closes: #1900
Approved by: cgwalters
2019-09-09 23:50:32 +00:00
Jonathan Lebon
c7814dd0ed ci: Allow passing HEAD commit to ci-commitmessage-submodules.sh
Jenkins' `checkout scm` gives us the exact commit that we're testing.
Allow passing that to the submodule commit message checker so it doesn't
use `HEAD` (which is likely GitHub's merge commit).

Closes: #1900
Approved by: cgwalters
2019-09-09 23:50:32 +00:00
Jonathan Lebon
6d017644d7 ci: Some more el7 path pruning
Use `dnf` consistently. Drop a couple of other conditionals.

Closes: #1900
Approved by: cgwalters
2019-09-09 23:50:32 +00:00
Colin Walters
2232edb6b0 ci: Add prow Dockerfile
This is an exact copy of https://github.com/ostreedev/ostree/pull/1906
for now.  From that commit message:

I'd like to add OpenShift's prow to this repository. Let's start
by adding a Dockerfile - it doesn't really do anything besides build.

However...I've lately been thinking about e.g. shipping the ostree tests
as an image, and then e.g. we could test FCOS by running that container
(which would orchestrate the host's ostree).

Anyways, not doing that right now but this is a start.
2019-08-28 16:28:22 -04:00
Colin Walters
0d841b2e2f ci: More fixes for cosa build
Use --entrypoint and also rename it while we're here.

Closes: #1889
Approved by: rfairley
2019-08-24 00:35:44 +00:00
Colin Walters
1027ade4db ci: Just use cosa as a buildroot too
Since we have to pull it anyways to do a build, let's use it
as a buildroot.  This should fix CI which broke because we were
doing a f29 build but cosa is f30, and libostree differed.
2019-08-22 20:13:41 +00:00
Jonathan Lebon
035ac2eaa6 tests/vmcheck: Fully drop python 3 requirement
Drop the use of Ansible everywhere. In the few cases where we really
Python, just spawn a container instead.

This is required to be able to hack on Fedora CoreOS.

Closes: #1850
Approved by: jlebon
2019-06-10 15:36:06 +00:00
Jonathan Lebon
206ae24d4e tests: Bump to Python 3 only
This bumps the requirement on the controlling host to Python 3 only.
It also bumps the requirement on the target host to Python 3 as well
since FCOS doesn't ship Python 2 right now.

Though we'll need to eventually drop all Python usage anyway, but at
least let's get tests passing on FCOS first. (See related previous
patch).

Closes: #1828
Approved by: cgwalters
2019-05-08 19:02:32 +00:00
Jonathan Lebon
ad1451fac2 Rebase to latest libdnf
This brings us back in sync with the latest libdnf git master. This
required a bunch of work both on the libdnf and rpm-ostree side to get
working. See e.g.
https://github.com/rpm-software-management/libdnf/issues/645.

A few things to adapt to:

- soname bump to `libdnf.so.2`
- `DnfAdvisory` is no longer a `GObject` (annoyingly it's not replaced
  by something we can keep a ref on, so this requires some hacks to
  steal from the `GPtrArray` -- could enhance libdnf for this later)
- disable SWDB history writing
- use new reldep public API
- update for latest `hy_subject_get_best_selector()` API

This now unlocks the possibility to add support for modules. (One can
see hints of this in the diff by the fact that `libdnf` links to
`libmodulemd1`.)

Update submodule: libdnf

Closes: #1404
Approved by: cgwalters
2019-03-19 14:29:15 +00:00
Jonathan Lebon
6cdcd474b6 Makefile-libs.am: Work around g-ir-scanner issues with clang
Right now there's an issue in Fedora with `g-ir-scanner` picking up
`-fstack-clash-protection` from the `sysconfig` Python module and
passing it to `clang`, which doesn't understand this flag yet.

Just work around this by (1) not even building GIR bindings for our
bundled libdnf since there's no need, and (2) overridding the compiler
used by `g-ir-scanner` so it's always `gcc`.

See: https://github.com/projectatomic/rpm-ostree/pull/1787#issuecomment-473971585

Closes: #1787
Approved by: cgwalters
2019-03-19 12:19:38 +00:00
Jonathan Lebon
d34f24581a ci: Fix repo editing in installdeps.sh
- Use `IFS=` so that we don't drop leading whitespace when reading in,
  which can be used to specify multiple baseurls on multiple lines.
- Use `-r` as good practice so we don't try to interpret anything.
- Use `echo "$line"` so that we don't drop leading whitespace when
  writing out.

Closes: #1787
Approved by: cgwalters
2019-03-19 12:19:38 +00:00
Jonathan Lebon
d0fdd9ac1d ci: Drop c7 support for installdeps.sh
Closes: #1787
Approved by: cgwalters
2019-03-19 12:19:38 +00:00
Jonathan Lebon
4ae3b174f5 ci: Bump to f29
Better late than never!

Closes: #1787
Approved by: cgwalters
2019-03-19 12:19:38 +00:00
Jonathan Lebon
6b928e8f0e ci: Build the whole tree, not just rust bits
Otherwise, we're not testing everything. E.g. the cbindgen bits.

Closes: #1778
Approved by: cgwalters
2019-03-07 21:57:08 +00:00
Colin Walters
b63f3720b8 ci: Add a f29 FCOS build
Let's start doing testing using coreos-assembler + Fedora CoreOS.

Closes: #1774
Approved by: jlebon
2019-03-07 19:16:12 +00:00
Colin Walters
387da3a420 ci: Verify rustfmt
Split out the command into a separate `Makefile` that doesn't
use Automake so we can invoke it early.

Closes: #1674
Approved by: dustymabe
2018-11-21 21:16:03 +00:00
Jonathan Lebon
dbdbaf87ee ci: Make adduser invocation idempotent
This is a bit of a selfish request, though I also really like
`ci/build.sh` for hacking in a throwaway environment. This patch allows
one to rerun `ci/build.sh` without erroring out on `testuser` already
existing.

Closes: #1621
Approved by: cgwalters
2018-10-16 16:03:06 +00:00
Jonathan Lebon
cc69168aaf ci: Drop CI_PKGS and epel-release in .papr.yml
Move specific package mentions from the `.papr.yml` file to
`ci/installdeps.sh`. I find the latter script super useful for
bootstrapping a pet container for rpm-ostree development and this helps
with that.

Closes: #1621
Approved by: cgwalters
2018-10-16 16:03:06 +00:00
Jonathan Lebon
e15baebc68 ci: Split out script to install deps
This should also make it useful for bootstrapping a test container
locally.

Closes: #1606
Approved by: cgwalters
2018-10-11 20:39:57 +00:00
Jonathan Lebon
8f504ec5ce ci/build.sh: Drop unnecessary pkg_install cargo
The latest releases in the stable repos now include the new cargo BR, so
we can drop this now.

Closes: #1606
Approved by: cgwalters
2018-10-11 20:39:57 +00:00
Colin Walters
08ad9d49c8 ci: Stop installing python-{sphinx,devel}
We should no longer need it. Just noticed this while I was in the file recently.

Closes: #1551
Approved by: jlebon
2018-09-11 15:35:32 +00:00
Colin Walters
6a274b831d build-sys: Hard require Rust
As something that manages your base operating system, we care
about reliability, predictability, as well as performance and
low-level access to native operating system facilities.  The
C programming language is great for the latter two, but fails
at providing a truly memory-safe environment.  Rust is fairly
unique in providing a language that doesn't carry a runtime,
so we can gradually "oxidize" and convert our C code without
imposing additional overhead.  It's also got a lot of modern
design niceties, like not having a null pointer.

Let's pull the trigger here and hard require Rust.  It's the
programming language I personally want to be primarily writing in for
years to come.

This is also in line with a recent trend of reducing our
experimental/optional matrix.

Closes: #1509
Approved by: jlebon
2018-08-21 14:49:26 +00:00
Jonathan Lebon
b66337e0cb ci: Pepper in date markers to help debug timing
This should help out with determining what steps take the most time.

Closes: #1459
Approved by: cgwalters
2018-07-21 14:43:48 +00:00
Colin Walters
0cc002fd8d build-sys: Use python3 for libdnf by default if available
Probably at some point libdnf will drop py2 support, but the
main reason I'm doing this is avoids a python2 dependency
for rpm-ostree for distributions that don't want that.

Note of course rpm-ostree itself doesn't use python, libdnf does.
And only for the python bindings, which we don't use either.  So
this is mostly just to DTRT automatically for the libdnf bits; down
the line we could probably add a patch to make the python fully
conditional.

Closes: #1460
Approved by: jlebon
2018-07-20 22:27:34 +00:00
Jonathan Lebon
201b020f60 ci: Bump version_id match to pick up ostree FAHC
Closes: #1433
Approved by: jlebon
2018-06-28 19:33:11 +00:00
Jonathan Lebon
592d6052b9 ci: Bump to F28
Closes: #1358
Approved by: cgwalters
2018-05-23 14:18:41 +00:00
Jonathan Lebon
169fab6f4f ci: Drop BDB1539 hack
This shouldn't be needed anymore.

Closes: #1365
Approved by: cgwalters
2018-05-14 18:14:54 +00:00
Colin Walters
d25588722b ci: Ensure we exclude ostree from base fedora too
rdgo uses an earlier git tag due to branches, until we have proper
repo priorities that don't care about versions, let's do this
hack.

Same thing as https://pagure.io/fedora-atomic-host-continuous/c/a7145410fbc7e73188fde449628fa0343b05e4a4?branch=master

Closes: #1326
Approved by: jlebon
2018-04-05 15:26:23 +00:00
Colin Walters
bb86912de9 tests: Dip our toes into using Ansible
This adds a shell primitive to make it easy to execute a playbook
task list.

The big picture idea is to sync with https://github.com/ostreedev/ostree/pull/1462
and rewrite some of the libvm shell stuff as playbooks, allowing easier
code sharing with a-h-t and just in general being a better library for
talking ssh and executing commnads.

Closes: #1297
Approved by: jlebon
2018-03-09 19:04:32 +00:00
Jonathan Lebon
0729487ae5 Check and display pending security advisories
Pick up security advisories when checking for pending updates and
include them in the `cached-update` property. On the client-side,
display them in the output of `status`.

This was part of the original vision for how useful a smart `check` mode
could be. It directly impacts how one manages their individual system
(e.g. when to reboot), and paves the way for integration into
higher-level apps that act at the cluster level.

Closes: #1249
Approved by: cgwalters
2018-02-15 15:30:26 +00:00
Colin Walters
b85ae9e1d6 jigdo: V4: Use archful provides for jigdoRPM Requires
When I tried to use my WIP client patches to do:
`rpm-ostree rebase rojig://fahc:fedora-atomic-host`,
I got a missing file object which turned out to
be the client importing the i686 RPMs.

This was passing in the test suite because we don't mirror i686 of course, but
on the client side right now we end up using all enabled repos, and since Fedora
is multiarch, the behavior is going to be...not predictable.

Thinking a bit about on this problem I actually happened to recall
the RPM `%{_isa}` macro which is used in Fedora in various places;
for example to "arch bind" `-devel` packages to their base.  See
for example [this case](33c7dc02bc/f/ostree.spec (_79)) in libostree.

As I noted at first, the core problem here is that the "final"
RPM architecture field is not symmetric in any way with the definition
of that `%{_isa}` macro.  See:

d9d47e0114/installplatform (L25)

The *third* solution I ended up on here is to iterate over the
`Provides` on the server side and we take the first thing
that matches `Provides: %{name}(whatever)`.

I briefly thought about trying to somehow drive into libsolv the
logic to prefer the jigdoRPM's native architecture...IIRC yum did
something like that in the past but it was never done in libsolv?
Anyways the dependencies here are now more correct, so other tools
will also handle it.

Closes: #1213
Approved by: jlebon
2018-01-19 14:18:19 +00:00
Jonathan Lebon
83fb40593f ci: explicitly pass --sysconfdir=/etc to configure
I spent some time trying to figure out why on CentOS my changes to
`/etc/rpm-ostreed.conf` didn't seem to have any effect. It turns out
that for some reason I didn't spend too much time looking into, the
autoconf in CentOS defaults to `${prefix}/etc` when `--prefix` is given.

This is also why I was under the erroneous assumption that D-Bus config
files were in `/usr/etc` in CentOS. This patch fixes that as well.

Closes: #1215
Approved by: cgwalters
2018-01-18 20:44:07 +00:00
Jonathan Lebon
5c9a9c30f7 ci/build-check: add -Wno-error=deprecated-declarations
For libdnf.

Closes: #1114
Approved by: jlebon
2018-01-16 14:14:33 +00:00