Commit Graph

126 Commits

Author SHA1 Message Date
Jonathan Lebon
2589cd1f92 rust/lockfile: Add more metadata to generated lockfiles
E.g. the generation timestamp, repos that were enabled, and their
generation timestamps.

This is just generally useful, though I'd like to make use specifically
of the new `metadata.generated` key in FCOS to drive versioning:

https://github.com/coreos/fedora-coreos-releng-automation/pull/50
2019-11-12 16:17:03 +01:00
Colin Walters
c8113bde32 Add hidden coreos-rootfs seal command
All this does is put the immutable bit on the target directory.
The intention is to replace this bit to start:
8b205bfbb9/src/create_disk.sh (L229)

However, the real goal here is to add code in this file
to handle redeploying the rootfs for Fedora CoreOS which
combines OSTree+Ignition:
https://github.com/coreos/fedora-coreos-tracker/issues/94

Basically doing this in proper Rust is going to be a lot
nicer than shell script in dracut modules.  Among other
details, coreutils `mv` doesn't seem to do the right thing
for SELinux labels when policy isn't loaded.
2019-10-04 08:03:03 -07:00
Rafael Fonseca
4d7af0b49b rust/utils: move common code to a function
Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>
2019-10-02 13:45:40 -07:00
Rafael Fonseca
6dab08b646 rust/lockfile: Add YAML support
Fixes #1904

Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>
2019-10-02 13:45:40 -07:00
Colin Walters
553be6e165 rust: cargo update
Been a while.  I think we may be clear to do this now.  A lot
of stuff like newer rand, etc.

Closes: #1912
Approved by: jlebon
2019-09-26 13:44:27 +00:00
Jonathan Lebon
10755592ea Add new ex history command
This is the rpm-ostree equivalent of `dnf history`. As opposed to the
history of the refspec (i.e. `ostree log`), this shows the history of
the system, i.e. the refspecs the host deployed, checksums, versions,
layered packages, etc... The amount of details remembered is similar to
what shows up in `status`.

There's definitely some further enhancements possible (e.g. printing
package diffs, displaying rollbacks), though this seems in good enough
shape as a first cut.

Closes: #1489

Closes: #1813
Approved by: cgwalters
2019-09-24 14:17:12 +00:00
Jonathan Lebon
2943bcaf3a rust/treefile: Various treefile merging tweaks
We were merging some keys with list values by overriding the full list
instead of appending. Fix those.

Also add some missing keys we weren't merging at all.

Closes: #1908
Approved by: cgwalters
2019-09-23 14:28:57 +00:00
Colin Walters
d34e41cc05 Clarify license situation to include GPLv2, relicense Rust code
We have contacted all contributors to the code in `rust/` and
that code is now all relicensed under the "standard Rust license"
of `Apache 2.0 OR MIT`.

[Due to an accident](https://github.com/projectatomic/rpm-ostree/issues/1890),
some GPLv2+ code was imported in the C side, and we're unlikely to
easily change that now.  Make this more official by adding the GPLv2.

I'd like to go through the C code and add SPDX and possibly investigate
relicensing some of the GPLv2+ code to LGPLv2+ but, not right now.

For a bit more about Rust and SPDX, see [this issue](https://github.com/rust-lang/cargo/issues/2039).

Closes: https://github.com/projectatomic/rpm-ostree/issues/1890

Closes: #1897
Approved by: jlebon
2019-09-05 20:49:18 +00:00
Colin Walters
3d6b33b6d4 rust: Add nix as a dependency
I have multiple outstanding PRs that use it and they conflict on
this, so let's just upstream it now.

It helps avoid some use of `unsafe` talking to libc directly.
2019-08-30 10:29:23 -04:00
Colin Walters
acdd04dac6 Move openat_utils to its own crate
I wanted to use this in another project, and I don't think
we're the only ones.  I extracted the code, added better docs
and tests, and published to crates.io:

https://crates.io/crates/openat-ext
2019-08-21 16:44:04 -04:00
Colin Walters
7db30fec13
treefile: Support arch-include (#1886)
* treefile: Rework test helper

Have each test clearly hold its tempdir; this way we can have a simple
helper function rather than a struct with an unreferenced parameter.

Also use `utils::write_file`.

Prep for further test work.

* treefile: Add a basic test for includes

Prep for adding an arch include test.

* treefile: Support `arch-include`

A long time ago we added architecture-specific package lists
via e.g. `packages-ppc64le`.  Much more recently we added
support for having the `include` key be a list - multiple includes.

By combining these two and supporting architecture-conditional includes,
we've effectively added architecture-conditionals to *all* keys.

Notably we want this for Fedora CoreOS today which is using
`remove-from-packages` on `grub2-tools` which isn't present on
s390x.

* ci: Bump cosa build timeout

Looks like another OpenStack perf regression.
2019-08-21 15:25:08 -04:00
Colin Walters
00bd491fe2 treefile: Support multiple includes
I'm working on having Silverblue inherit from Fedora CoreOS.  But
conceptually it also inherits from (parts of) Workstation.
It is just easier if we support multiple inheritance, then I don't
need to think too hard about how to make it a single inheritance chain.

Closes: #1870
Approved by: jlebon
2019-07-18 22:40:59 +00:00
Jonathan Lebon
4418589ca1 rust: Move open_file test to utils module
We had essentially the same test in the `lockfile` and `treefile`
modules. Just dedupe those and move it to `utils`.

Closes: #1867
Approved by: cgwalters
2019-07-18 18:54:27 +00:00
Jonathan Lebon
68432e461c lockfile: Allow omitting the digest
When manually writing lockfile overrides (see previous commit), it's
sometimes easier to not have to specify the SHA256 of the package. For
example, in FCOS, all packages on development and production streams
will be sourced uniquely from coreos-pool, so there's no question of
where the package will come from. It's of course also easier in the
context of local development.

Another motivation for this though is a subtle interaction between
Fedora infra and the way we'd like to implement lockfile management: we
want the override process to be PR-based, with a privileged bot in the
backend tagging new overrides into the pool as necessary on merge.
However, packages built in Koji are initially unsigned, and so we can't
actually *know* what the SHA256 of the package will be until it's signed
and tagged into the pool by the bot.

Closes: #1867
Approved by: cgwalters
2019-07-18 18:54:27 +00:00
Jonathan Lebon
1b43ad04be lockfile: Allow specifying multiple lockfiles
Teach `rpm-ostree compose tree` to accept multiple `--ex-lockfile`
arguments. In this case, later lockfiles can override the NEVRA for
packages specified in previous lockfiles.

This will be used in the FCOS pipeline, where we want to be able to have
a shared "base lockfile" and then stream-specific "override lockfiles".

I contemplated making this an `include: ...` key instead similar to the
manifests, but I'm not sure that paradigm fits as nicely for lockfiles.
Making it separate switches instead also makes it trivial to implement
in cosa.

(And of course, this is all still prefixed with `--ex` which means we
are at liberty of changing this interface later on after gaining some
experience with it).

Closes: #1867
Approved by: cgwalters
2019-07-18 18:54:27 +00:00
Jonathan Lebon
b670ab37c2 lockfile: Switch packages JSON spec to an object
There are two reasons for this:
1. I'd like to add overrides semantics to lockfiles, and keying by the
   package name only makes this much easier.
2. I'd like to make the digest optional, and keeping it as a tuple makes
   this awkward.

A map seems natural too since it makes it more clear that we don't
expect multiple specifications for the same package name.

Another tiny advantage is that it's easier to process with e.g. `jq`.

Closes: #1867
Approved by: cgwalters
2019-07-18 18:54:27 +00:00
Jonathan Lebon
5e2aeb4793 lockfile: Switch to writing it from Rust
I wanted to modify the lockfile specification, but then remembered that
it currently lives in two places right now: once on the Rust side where
it's deserialized, and once more on the C side where it's serialized.

If we could write the lockfile from the Rust side, then we wouldn't have
to deal with the `GVariantBuild` and `json-glib` goop, and instead
we could consistently use serde against the same struct for both
serialization and deserialization.

But there isn't an easy way to do this given that the state to be
serialized is intrinsically linked to libdnf.

So this patch takes the next step in our oxidation process by adding a
minimal `libdnf_sys` module which allows us to call `libdnf` functions
from Rust. This is not the prettiest code I've written, and there's
definitely some polishing that could be done. But I think overall it's a
move in the right general direction: as we oxidize more things, we'll at
some point *have* to integrate more tightly with the C side in a
bidirectional way, instead of the "one-way" approach we've been using so
far.

For this patch specifically, in exchange we get a unique source of truth
for the lockfile spec, just like the treefile, and we drop a lot of C
code in the process.

Closes: #1867
Approved by: cgwalters
2019-07-18 18:54:27 +00:00
Jonathan Lebon
69979bf722 lockfile: Rename function to ror_lockfile_read
Let's just make it really obvious that this is the function that reads
the lockfile.

Prep for also having a symmetric `ror_lockfile_write` function.

Closes: #1867
Approved by: cgwalters
2019-07-18 18:54:27 +00:00
Jonathan Lebon
b381e0294f compose: Hash all treefile externals and flattened manifest
Move hashing to the Rust side so that we can easily hash over the final
set of inputs after parsing. This means that we now hash over all the
externals, like `add-files` references, any `postprocess-script` script,
and `passwd` and `group` files.

The original motivation for this was that hashing over a reserialized
version of the treefile was not deterministic now that treefiles include
hash tables (i.e. `add-commit-metadata`). So I initially included each
individual treefile as part of the hash.

I realized afterwards that just switching to `BTreeMap` fixes this, so
we can keep hashing only the final flattened reserialized treefile so we
ignore comments and whitespace too. But since I already wrote the patch,
and it fixes a real issue today... here we are.

One notable change though is that we now hash the treefile in non-pretty
mode to increase the chances that the serialized form remains stable.
Ironically, this change is likely to cause a no-op commit once it gets
to pipelines which iterate quickly. All for the greater good though.

Closes: #1865
Approved by: cgwalters
2019-07-09 20:28:18 +00:00
Jonathan Lebon
fc49fb0089 treefile: Add new add-commit-metadata key
Add support for a new `add-commit-metadata` key in the treefile so that
we can directly specify commit metadata we want to inject from there.

This will be useful in Fedora CoreOS, where we'll have separate
treefiles for each streams, each with stream-specific metadata values
required.

Closes: #1865
Approved by: cgwalters
2019-07-09 20:28:18 +00:00
Rafael Fonseca
c94bd08b02 treefile: allow ${releasever} in more keys
Besides allowing ${releasever}, only do the substitution as the final
pass after merging the treefiles for all the keys (currently ${basearch}
and ${releasever}) instead of doing it per parse. This way we have the
expected semantics where one could do:

```
include: "fedora-coreos.yaml"
releasever: "42"
```

and have that releasever used.

Fixes #1809

Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>

Closes: #1848
Approved by: cgwalters
2019-06-18 18:19:42 +00:00
Rafael Fonseca
dbf28ac255 compose: Move json lockfile parsing to Rust
Using glib_json to parse the lockfile yields some oddities like
everything being wrapped in a GVariant. Let's leave the parsing to serde
in the Rust side of things. Hopefully that'll make the lockfile easier
to extend in the future.

Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>

Closes: #1851
Approved by: jlebon
2019-06-17 16:28:44 +00:00
Rafael Fonseca
e4d61b6b09 rust: Move open_file to utils.rs
Prep for next patch.

Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>

Closes: #1851
Approved by: jlebon
2019-06-17 16:28:44 +00:00
Colin Walters
5f6578ef3c compose: Add ostree-layers and ostree-override-layers
The use case for `ostree-layers` is to support injecting non-RPM
content in a more flexible way than can be done with `add-files`,
and also without dropping all the way to split composes.

This starts with support on the `compose tree` side but down the
line I'd like to make it more convenient to do *client* side too.

For `ostree-override-layers` this is mainly a development thing
for tools like coreos-assembler.  Rather than building an RPM
we just `make install DESTDIR` then commit and add to
`ostree-override-layers`.

Closes: #1830
Approved by: jlebon
2019-06-07 17:42:32 +00:00
Jonathan Lebon
6b2ac5885f rust: Stop using extern crate and #[macro_use]
Rust 2018 makes this much nicer now.

Closes: #1812
Approved by: lucab
2019-04-15 13:03:21 +00:00
Jonathan Lebon
278ba77774 rust: Reduce visibility of items
Use `pub(crate)` on the ffiutils functions since it's just for us. Drop
unnecessary `pub` fields on `TreefileExternals`.

Closes: #1812
Approved by: lucab
2019-04-15 13:03:21 +00:00
Jonathan Lebon
60e71dfc66 rust: Update to 2018 edition
This mostly involves adjusting our path uses to be anchored, e.g. at
`self::` or `crate::`. I took the opportunity to also tweak `use`
ordering so that we consistently list modules in external crates first
before our own.

Closes: #1812
Approved by: lucab
2019-04-15 13:03:21 +00:00
Jonathan Lebon
749facba86 rust: Bump dependencies
Now that our minimum Rust toolchain version was bumped, we can bump all
our deps to the latest and drop the frozen `rand` version that was done
for #1777.

Closes: #1810
Approved by: jlebon
2019-04-05 19:28:39 +00:00
Jonathan Lebon
95d1c6ea29 ci: Bump minimum Rust version to v1.31.0
Now that we no longer support el7, let's peg our minimum `rustc` version
to the latest Rust module available in el8 instead. That's 1.31.0 right
now.

And drop use of deprecated `trim_left_matches()` (it was deprecated in
1.33.0, which is currently used in the other testsuites that pull
`rustc` from the Fedora repos; we might want to eventually just use the
`RUST_MIN_VERSION` in all the testsuites to not get caught in the middle
like this again).

Closes: #1787
Approved by: cgwalters
2019-03-19 12:19:38 +00:00
Colin Walters
adff1e98b6 compose: Add boot-location: modules
And this (for now at least) completes the epic journey of the
"where's the kernel"?  With this it's found solely in
`/usr/lib/modules/$kver`.

There are a few reasons to do this; most prominent is that
it avoids duplicating the content as the locations may have
different SELinux labels.

Closes: #1773
Approved by: jlebon
2019-03-08 22:49:40 +00:00
Colin Walters
f37426f1ad compose: Remove support for boot-location: both
I'd like to add a new `boot-location: modules`.  In prep
for that, let's remove the legacy `both` which drops into
`/boot`.

The libostree support for handling `/usr/lib/ostree-boot` has
existed for over 4 years:

```
commit 37a059925f6b96d30190b65bee6bdde0ae1c6915
Commit:     Colin Walters <walters@verbum.org>
CommitDate: Sun Nov 30 23:14:05 2014 -0500

    deploy: Ensure that we can deploy using only /usr/lib/ostree-boot

```

I think we assume now that no one is now making *new* treecomposes and needs
a newer rpm-ostree and that they expect people to be able to use as an
upgrade target from a libostree that predates that.

Closes: #1773
Approved by: jlebon
2019-03-08 22:49:40 +00:00
Jonathan Lebon
e49f625b61 rust: Freeze rand crate to v0.6.1
The recent dep bump caused rand to jump to v0.6.5, which pulled in
`fuchsia-cprng`. Sadly, the cargo version we're frozen on chokes on it
because its manifest makes use of the `edition` keyword, which wasn't
stabilized yet.

Let's just freeze the problematic crate for now. We should be able to
bump our Rust toolchain soon-ish.

(Note this commit also does some other minor crate bumps as a result of
regenerating the lockfile to prune away `fuchsia-cprng`.)

Closes: #1777

Closes: #1778
Approved by: cgwalters
2019-03-07 21:57:08 +00:00
Jonathan Lebon
8b1bc9894e rust: Pass through rustfmt
Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Jonathan Lebon
1fe66e91d0 rust/treefile: Support dash convention for all options
Now that we support YAML, it's a gotcha/eyesore that some of our options
use underscores rather than dashes. Let's be nice and switch those few
options over, while of course still supporting the previous name.

Co-authored-by: Colin Walters <walters@verbum.org>

Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Jonathan Lebon
0ebfb6e4ff rust/treefile: Don't hardcode list of architectures
Just try to match the current architecture against any
`packages-${arch}` keys in the treefile.

Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Jonathan Lebon
24335bdf1a rust/treefile: Use HashMap to collect extra fields
Instead of the `Strict` and `Permissive` variants of
`TreeComposeConfig`, just collapse all the excess fields in a new
`extra` member and check for any keys there in the strict YAML path.
This will also allow us to drop the hardcoded list of architectures in
the next patch.

Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Jonathan Lebon
82594b5052 rust: Bump dependencies
Notably, the latest indicatif compiles fine again. This is now the
latest of everything, except the glib stack:

```
$ cargo outdated -R
Name      Project  Compat  Latest  Kind    Platform
----      -------  ------  ------  ----    --------
gio-sys   0.7.0    ---     0.8.0   Normal  ---
glib      0.6.1    ---     0.7.1   Normal  ---
glib-sys  0.7.0    ---     0.8.0   Normal  ---
```

because it requires a newer rustc than we're currently locked at.

Closes: #1770
Approved by: cgwalters
2019-03-01 15:51:07 +00:00
Jonathan Lebon
2be532c9e9 rust/treefile: Add basearch key
Add a `basearch` key to the manifest. This can be used at compose time
to assert the architecture the compose is running on. Though my
motivation is for the common case where it gets omitted from the input
manifest and gets automatically added by rpm-ostree into
`/usr/share/rpm-ostree/treefile.json` for introspection on the client.

(The crucial part here is that the treefile created by rpm-ostree
remains deserializable into a `TreeComposeConfig`).

Closes: https://github.com/coreos/fedora-coreos-tracker/issues/154

Closes: #1766
Approved by: cgwalters
2019-02-27 01:05:43 +00:00
Jonathan Lebon
6a57db1c36 rust/treefile: Rename arch -> basearch
We're really using this variable to substitute `${basearch}` and find
basearch-specific packages. Let's rename the variable to make that more
obvious.

Closes: #1766
Approved by: cgwalters
2019-02-27 01:05:43 +00:00
Colin Walters
3f722b5005 rust: Bump crate dependencies
Prep for adding other crate deps for sysusers, multiprocessing work.

Closes: #1765
Approved by: jlebon
2019-02-26 16:42:42 +00:00
Colin Walters
d1eedbbd96 rust/openat: Add helper to retrieve file type
Will be used by sysusers code.

Closes: #1764
Approved by: jlebon
2019-02-25 20:56:44 +00:00
Colin Walters
579f017fc2 rust: Add clap
A few of my outstanding Rust PRs use this (multiprocess, sysusers).
Let's add it now.

Closes: #1764
Approved by: jlebon
2019-02-25 20:56:44 +00:00
Jonathan Lebon
0e5044ca92 postprocess: Add /home -> /var/home SELinux substitution
Manually patch `file_contexts.subs_dist` so that `/home` is equivalent
to `/var/home`. This is required now that the generated homedirs rules
use `/var/home`. Otherwise, `matchpathcon` for example will return wrong
results.

This patch also includes the *removal* of `/var/home -> /home` so that
we're not dependent on this selinux-policy patch making it at the same
time as downstream:

https://src.fedoraproject.org/rpms/selinux-policy/pull-request/14

(See the conversation there for more information.)

Closes: #1754
Approved by: cgwalters
2019-02-14 17:24:09 +00:00
Colin Walters
85ab9c3181 treefile: Fix octal mode for rojig spec too
Going to need to see if clippy or something has a way for us
to reject leading `0`.

Closes: #1756
Approved by: jlebon
2019-02-11 18:21:51 +00:00
Colin Walters
07fbd195fb composepost: Fix permissions on /usr/etc/default/useradd
Switching between programming languages can be a pain.  Unfortunate
that Rust happily accepts a leading `0` too.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1752

Closes: #1753
Approved by: jlebon
2019-02-04 18:37:58 +00:00
Colin Walters
65afbec3b1 rust: Split openat helpers into module
Will help use them in other parts of our code.

Closes: #1737
Approved by: jlebon
2019-01-23 16:00:16 +00:00
Jonathan Lebon
25d0213d15 rust/treefile: Include filename in more error msgs
This uses the `Context` feature of the failure crate to make error
messages more useful when we fail to open a file. The difference with
`map_err` is that one can still obtain the underlying error from the
context if need be. Though surprisingly, the normal `Display` for a
`Context` doesn't include the original error, so we essentially have to
do a prefix here (see [1]).

Before:

```
error: Failed to load YAML treefile: No such file or directory (os error 2)
```

After:

```
error: Failed to load YAML treefile: Can't open file "treecompose-post.sh": No such file or directory (os error 2)
```

[1] https://github.com/rust-lang-nursery/failure/issues/182

Closes: #1735
Approved by: cgwalters
2019-01-22 15:49:34 +00:00
Jonathan Lebon
1594140a33 rust/treefile: Convert more functions to Fallible
Slightly prep for next patch (was in the area so just did the whole
module).

Closes: #1735
Approved by: cgwalters
2019-01-22 15:49:34 +00:00
Colin Walters
89cdeb5131 Add rayon dependency, do postprocessing in parallel
Obviously doing these trivial functions in parallel doesn't matter,
but I wanted to dip my toes into Rayon (and Rust parallelism in
general).

`try_for_each()` is just...staggeringly simple and beautiful.

Closes: #1727
Approved by: jlebon
2019-01-07 18:50:49 +00:00
Colin Walters
255dc7027e compose: Minor further oxidation of postprocessing
Take a baby step into further oxidation by migrating the bit
to write our presets.

Closes: #1727
Approved by: jlebon
2019-01-07 18:50:49 +00:00