Commit Graph

2707 Commits

Author SHA1 Message Date
Jonathan Lebon
28d3fe428f app/compose: Support multiple --add-metadata-from-json
No reason we can't do this. I initially was going to make use of this
for tests, but ended up not needing it.
2019-12-20 21:16:24 +01:00
Jonathan Lebon
0e84776519 libpriv/rojig: Fix unref'ing using wrong function
We were trying to unref a `GHashTable` using `g_ptr_array_unref`.
Something something... rewrite it in Rust.
2019-12-20 21:16:24 +01:00
Jonathan Lebon
7d36ea7ebc rust: Wrap parent directory handling for Path
The parent of a `Path` of `"foobar"` is actually `""`, because
`parent()` wants to return a slice of its buffer:
https://github.com/rust-lang/rust/issues/36861

Just do a trivial wrapper around it to correctly return `"."` instead.

Hit this when trying to pass a treefile in the working directory.
2019-12-20 21:16:24 +01:00
Jonathan Lebon
82b27de843 Release 2019.7
Not the biggest release, but there are some decent targeted fixes and
enhancements and it's been a while.
2019-12-19 16:22:11 +01:00
Jonathan Lebon
774d0171d5 libdnf: Bump to cc36cb7492275e34c10148176824a35a3a67a461
Let's try to do regular-ish bumps again. Both for fixes and features,
but also so we eat regressions as they come rather than all at once when
we actually need something new there really fast.

Update submodule: libdnf
2019-12-19 15:48:49 +01:00
Colin Walters
4881435663 tests: Misc tweaks
- Have libvm.sh inherit libtest, otherwise we don't have `fatal`
- Add `error: ` prefix to `fatal` messages for clarity
- Add missing plural
2019-12-16 15:17:01 +01:00
Jonathan Lebon
c7a9c3b1dd Rework vmcheck to use kola spawn, move off of PAPR
There's a lot going on here, but essentially:

1. We change the `vmcheck` model so that it always operates on an
   immutable base image. It takes that image and dynamically launches a
   separate VM for each test using `kola spawn`. This means we can drop
   a lot of hacks around re-using the same VMs.
2. Following from 1., `vmoverlay` now takes as input a base image,
   overlays the built rpm-ostree bits, then creates a new base image. Of
   course, we don't have to do this in CI, because we build FCOS with
   the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay`
   then will be more for the developer case where one doesn't want to
   iterate via `cosa build` to test rpm-ostree changes. I say "will"
   because the functionality doesn't exist yet; I'd like to enhance
   `cosa dev-overlay` to do this. (Note `vmsync` should still works just
   as before too.)
3. `vmcheck` can be run without building the tree first, as
   `tests/vmcheck.sh`. The `make vmcheck` target still exists though for
   finger compatibility and better meshing with `vmoverlay` in the
   developer case.

What's really nice about using kola spawn is that it takes care of a lot
of things for us, such as the qemu command, journal and console
gathering, and SSH.

Similarly to the compose testsuites, we're using parallel here to run
multiple vmcheck tests at once. (On developer laptops, we cap
parallelism at `$(nproc) - 1`).
2019-12-13 19:18:30 +01:00
Jonathan Lebon
d2a4372d4d app/override: Don't include rpmostree-ex-builtins.h
Likely leftover from when it was an experimental feature. It's not
anymore.
2019-12-13 19:18:30 +01:00
Jonathan Lebon
571bd3a558 tests: Add hidden testutils subcommand
This is a hack to allow using `inject-pkglist` without having to build
the tree first.

Higher-level, I think we can split this back out again if we have a
`-tests` subpackage where we ship the vmcheck testsuite.
2019-12-13 19:18:30 +01:00
Jonathan Lebon
9d73458f0c ci: Add the built RPMs as cosa overrides
So that the built FCOS has them. This is a prereq for actually testing
what we built in `vmcheck`.
2019-12-13 19:18:30 +01:00
Jonathan Lebon
07dfb8dc3e ci: Archive built RPMs
That way, anyone can easily download the latest built RPMs from master
or a specific PR. This isn't a replacement for automated builds in Koji
though since it's not multi-arch.

Also fetch the tags so that the NEVRA derived from `git describe` is
nicer.
2019-12-13 19:18:30 +01:00
Jonathan Lebon
f673305920 ci: re-use variable for container images
Makes it less repetitive and allows controlling the images from a
central place.
2019-12-13 19:18:30 +01:00
Colin Walters
e726d008fc compose: Add an automatic-version-suffix key
This allows replacing the `.` in automatic version increments
with whatever one wants (as long as it's a single ASCII character)
right now.

The specific motivation here is for at least RHEL CoreOS to use
`version-suffix: "-"` so that its versions can become valid
semantic versions.

Related: https://github.com/coreos/rpm-ostree/issues/1954
2019-12-13 17:11:16 +01:00
Colin Walters
75c676715a daemon: Use MountFlags=slave and opt-in to OSTree read-only /sysroot
This is all we need to tell libostree that we support a read-only
`/sysroot` and `/boot`.

See https://github.com/ostreedev/ostree/issues/1265
PR in https://github.com/ostreedev/ostree/pull/1767
2019-12-13 01:44:56 +01:00
Colin Walters
4e3c41be9f kernel: Append /dev/{u,}random to initrd instead of dracut caps
Rather than giving dracut `cap_mknod` which won't work in
unprivileged scenarios, append a tiny static pre-generated CPIO
blob with `/dev/random` and `/dev/urandom` to the output of
dracut.

This is a hack until dracut does this itself.  But the problem
is patches to dracut will take eleven billion years to ship
in RHCOS.

Closes: https://github.com/coreos/rpm-ostree/issues/1950
2019-12-10 22:11:10 +01:00
Colin Walters
f295f54306 kargs: Support --append and --delete simultaneously
Code I wrote for the machine-config-operator expected it to
work, and I don't see a reason not to support it.

See https://github.com/openshift/machine-config-operator/issues/1265
2019-12-10 20:27:57 +01:00
Jonathan Lebon
3b8a1ec6c4 libpriv/kernel: add cap_mknod to dracut run
A lot of history with this. But essentially, dracut tries to `mknod` a
few character devices like `/dev/random` and `/dev/urandom` and fails.

We originally blocked `cap_mknod` because, well, `%post` scripts don't
really need to do that, and it would get wiped anyway. But there is a
use case for dracut's CPIO: we want `/dev/*random` to be available in
early boot *before* systemd even mounts `devtmpfs` because libgcrypt as
part of its constructor-time selftests in FIPS mode wants to read from
there.

For more fun, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1778940
https://bugzilla.redhat.com/show_bug.cgi?id=1401444
https://bugzilla.redhat.com/show_bug.cgi?id=1380866
2019-12-05 09:43:20 -08:00
Jonathan Lebon
2589cd1f92 rust/lockfile: Add more metadata to generated lockfiles
E.g. the generation timestamp, repos that were enabled, and their
generation timestamps.

This is just generally useful, though I'd like to make use specifically
of the new `metadata.generated` key in FCOS to drive versioning:

https://github.com/coreos/fedora-coreos-releng-automation/pull/50
2019-11-12 16:17:03 +01:00
Jonathan Lebon
fdaf99327c core: Split out function to get enabled rpmmd repos
Prep for next patch.
2019-11-12 16:17:03 +01:00
Jonathan Lebon
fec61ce577 libpriv/kernel: Hack around vmlinuz path in HMAC file
As mentioned in the comment block:

```
If there's an HMAC file, fix the path to the kernel in it to be
relative. Right now, the kernel spec encodes `/boot/vmlinux-$kver`,
which of course not going to work for us. We should work towards making
this change directly into the kernel spec.
```

For background, see this comment and following:
https://github.com/ostreedev/ostree/pull/1962#issuecomment-547488164
2019-10-31 14:55:27 +01:00
Jonathan Lebon
6aa496e312 libpriv/kernel: Use g_build_filename instead of g_strconcat
It's much easier to mess up with the latter than the former when
building filenames. There's a bunch more all over the codebase; just did
this bit to be consistent with the next commit which also uses it.
2019-10-31 14:55:27 +01:00
Colin Walters
122811a1ba tree-wide: [scan-build] Add some not-null assertions
I don't understand why this doesn't happen in more places;
it may have to do with the depth of the call chain?
Anyways, add some more asserts that values aren't NULL.
2019-10-18 18:00:16 +02:00
Colin Walters
e0102550d7 tree-wide: [scan-build] Initialize some variables
These are all false positives, but:

1) We might as well be safe
2) Quieting the scanner is worth it since it does find real bugs
2019-10-18 18:00:16 +02:00
Colin Walters
8e5baf5ca7 Detect whether zchunk (zck) is available at build time
We don't *actually* use this ourself, but librepo does, and libdnf gets confused
if librepo doesn't support it.  This is the case in RHEL8 currently.

Basically what breaks is trying to use the Fedora EPEL repo (has zchunk metadata)
on RHEL CoreOS.  And we have a test in kola that does this today.
2019-10-18 17:13:55 +02:00
Colin Walters
bc36d0a95e HACKING.md: Document libdnf vendoring rationale
Per discussion in https://github.com/coreos/rpm-ostree/pull/1929
2019-10-18 15:27:12 +02:00
Jonathan Lebon
9ff9d43822 core: Filter locked packages by checksums before depsolving
Don't just filter down packages by NEVRA, but also filter out those that
don't match the checksum too. We were enforcing checksum matches already
before this, but only *after* depsolving and simply erroring out if they
didn't match.

However, because of how RPM signing is implemented in Fedora, it is
possible to have the same NEVRA in two different repos, each with two
different hashes. E.g. right now for example, `efivar-libs` wasn't
rebuilt for f31, and so f31 is just shipping the f30 RPM, but signed
with the f31 key. And of course, we also had the f30 version in the
pool.

This patch allows us to transition over to the f31 version with
everything else by not getting thrown off by the f30 version already in
the pool. (Still need to investigate how the pool will deal with this.)
2019-10-16 20:39:11 +02:00
Colin Walters
11ee20c1cd unpacker: Build with older libarchive without zstd
It's not in RHEL8.1, and I'm trying to rebase rpm-ostree.
2019-10-15 18:17:46 +02:00
Jonathan Lebon
450948a9f6 ci/papr: Drop required commit status context
This was useful in combination with Homu since it only had to watch one
context. Since we're not using Homu anymore (and Tide instead looks at
all statuses by default), let's just drop it. This brings down the
number of statuses on PRs by one more (and so one less context to
override when needed).

Relatedly, also just test on `master` now.
2019-10-09 10:22:58 -07:00
Jonathan Lebon
45623a9b54 tests/vmcheck: Fix test-misc-1.sh syntax
The `EOF` needs to be alone on a line to be valid. The way to redirect
the output is unintuitively to do it at the beginning of the line
instead.
2019-10-08 14:10:53 -07:00
Colin Walters
c8113bde32 Add hidden coreos-rootfs seal command
All this does is put the immutable bit on the target directory.
The intention is to replace this bit to start:
8b205bfbb9/src/create_disk.sh (L229)

However, the real goal here is to add code in this file
to handle redeploying the rootfs for Fedora CoreOS which
combines OSTree+Ignition:
https://github.com/coreos/fedora-coreos-tracker/issues/94

Basically doing this in proper Rust is going to be a lot
nicer than shell script in dracut modules.  Among other
details, coreutils `mv` doesn't seem to do the right thing
for SELinux labels when policy isn't loaded.
2019-10-04 08:03:03 -07:00
Jonathan Lebon
68750b6894 ci/papr: Drop f29-codestyle and rust-min-version-check
These are also already covered by the Jenkins pipeline.
2019-10-03 13:39:11 -07:00
Jonathan Lebon
289af613a9 ci/jenkins: don't pass GIT_COMMIT to ci-commitmessage-submodules.sh
Jenkins is tricky: it does an initial checkout, merges the PR head into
the target branch, then creates the pod. Once in the pod, we do a
`checkout scm` which *also* merges the PR head into the target branch.
However, the `change.GIT_COMMIT` variable we get from that is set to the
SHA of the first merge, not the second one. Which... yeah is super
confusing since we explicitly assign `change` from that `checkout scm`
operation. So that's probably a valid bug.

This was then throwing off `ci-commitmessage-submodules.sh` since it
didn't find the merge commit in the graph.

Anyway, not going to spend more time on this. Let's just not pass any
commit at all. The git range `origin/master..HEAD` already does what we
want (go through all the commits in HEAD *not* in master).
2019-10-03 13:39:11 -07:00
Colin Walters
010f269492 ci/papr: Drop cosa build
It's currently covered by the Jenkins job.
2019-10-03 10:32:40 -07:00
Rafael Fonseca
4d7af0b49b rust/utils: move common code to a function
Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>
2019-10-02 13:45:40 -07:00
Rafael Fonseca
6dab08b646 rust/lockfile: Add YAML support
Fixes #1904

Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>
2019-10-02 13:45:40 -07:00
Colin Walters
75a2f8bb33 ci/prow: Use cosa buildroot
See https://github.com/coreos/coreos-assembler/pull/730

- Skip installing deps, hooray!
- Also skip the C test suite for now because of the linked fuse-overlayfs issue
2019-10-02 08:16:10 -07:00
Colin Walters
226b939d29 ci/prow: Use build-check.sh
We want to run the unit tests after a build, so do that.
Furthermore, this actually installs the binaries, which
means if we do it as part of a `Dockerfile` build, we
can use that image for further testing.
2019-10-02 08:16:10 -07:00
Jonathan Lebon
ea28cf95c5 app/reset: Fix argc range check
Otherwise, e.g. `rpm-ostree reset foobar` doesn't actually error out.
Some users have hit this as a typo for
`rpm-ostree override reset foobar`.
2019-10-02 04:36:13 -07:00
Jonathan Lebon
677c3c8b29 ci: Also bump MSRV to 1.37.0 for CCI Jenkins
Just split it out into a separate script for easier sharing.
2019-10-01 11:26:29 -04:00
Colin Walters
8b6cfd6f23 OWNERS: New file for Prow integration
This will help us integrate with OpenShift Prow:
https://github.com/coreos/fedora-coreos-tracker/issues/263
2019-09-27 14:58:21 -04:00
Colin Walters
553be6e165 rust: cargo update
Been a while.  I think we may be clear to do this now.  A lot
of stuff like newer rand, etc.

Closes: #1912
Approved by: jlebon
2019-09-26 13:44:27 +00:00
Colin Walters
a846a25dc9 Bump minimum rust to 1.37
We should be clear to do this now.

Closes: #1912
Approved by: jlebon
2019-09-26 13:44:27 +00:00
Jonathan Lebon
8615a964e7 app/status: Fix printf format string for 32-bit
Hit this when compiling in Koji.

Closes: #1910
Approved by: cgwalters
2019-09-25 17:48:47 +00:00
Jonathan Lebon
ccace4c56c Release 2019.6
Last release was almost two months ago. Not a lot of groundbreaking
features, though let's get the various fixes out.

Closes: #1907
Approved by: lucab
2019-09-24 16:23:27 +00:00
Jonathan Lebon
10755592ea Add new ex history command
This is the rpm-ostree equivalent of `dnf history`. As opposed to the
history of the refspec (i.e. `ostree log`), this shows the history of
the system, i.e. the refspecs the host deployed, checksums, versions,
layered packages, etc... The amount of details remembered is similar to
what shows up in `status`.

There's definitely some further enhancements possible (e.g. printing
package diffs, displaying rollbacks), though this seems in good enough
shape as a first cut.

Closes: #1489

Closes: #1813
Approved by: cgwalters
2019-09-24 14:17:12 +00:00
Jonathan Lebon
9e2ceca06f app/deploy: Gate 77 exit behind --unchanged-exit-77
This has a bit of history, but essentially in 1c01141e, we made both
`upgrade` and `deploy` automatically exit 77 if there were no changes.
Then in c3f1e7c8, we only changed `upgrade` so that it became gated
behind `--upgrade-unchanged-exit-77`.

I think we should carry this forward into `deploy` as well. The way I
look at this is: the default UX shouldn't require users to care about
special exit codes. That's something scripts care about. In its vanilla
form, either a command should error out or succeed.

This patch tries to add some consistency by introducing a new
`--unchanged-exit-77` in both `deploy` and `upgrade` (where it just
replaces the previous switch). The naming here matches what `install`
has too.

So... this does break backwards compatibility for any scripts which
relied on that behaviour. Though the only app I know today which wants
deploy semantics and doesn't use the D-Bus API is Zincati, which
actually hit this issue. There's also RHCOS, though the `pivot` there
uses `rebase`, not `deploy`. So overall, I think this is worth breaking
now while we're still in a transitionary period in the downstreams?

Closes: #1906
Approved by: cgwalters
2019-09-24 08:07:04 +00:00
Jonathan Lebon
2943bcaf3a rust/treefile: Various treefile merging tweaks
We were merging some keys with list values by overriding the full list
instead of appending. Fix those.

Also add some missing keys we weren't merging at all.

Closes: #1908
Approved by: cgwalters
2019-09-23 14:28:57 +00:00
Jonathan Lebon
46ab7d1ae8 ci: Add Jenkins pipeline
This is an experiment in using Jenkins pipelines for our CI. See similar
initiatives in coreos-assembler[1] and fedora-coreos-config[2].

For now, this only does the following testing:
- checks commit for unintended submodule bumps
- checks the minimum Rust version
- builds RPMs
- builds FCOS (with the new RPMs both for executing the build
  itself, as well as included in the built OS)

There are dummy placeholders for where we'd actually run the vmcheck
and the compose testsuites. Let's address those trickier parts as
follow-ups.

[1] https://github.com/coreos/coreos-assembler/pull/667
[2] https://github.com/coreos/fedora-coreos-config/pull/131

Closes: #1899
Approved by: cgwalters
2019-09-18 15:15:28 +00:00
worldofpeace
a8ee788e61 Move D-Bus conf file to $(datadir)/dbus-1/system.d
Since D-Bus 1.9.18 configuration files installed by third-party should
go in share/dbus-1/system.d. The old location is for sysadmin overrides.

Closes: #1903
Approved by: jlebon
2019-09-17 04:37:59 +00:00
Colin Walters
13377d4bab postprocess: Ensure dirs are 0755 regardless of umask
`umask` is one of those really evil Unix things...it's pretty
crazy actually there's still no threadsafe way to "`mkdir` ignoring umask".

This surfaced in someone using coreos-assembler with a working
directory of mode `0750` and having that surface in the target
rootfs.

Ref: https://github.com/coreos/fedora-coreos-tracker/issues/272

Closes: #1902
Approved by: jlebon
2019-09-11 20:56:52 +00:00