Commit Graph

106 Commits

Author SHA1 Message Date
Rafael Fonseca
c94bd08b02 treefile: allow ${releasever} in more keys
Besides allowing ${releasever}, only do the substitution as the final
pass after merging the treefiles for all the keys (currently ${basearch}
and ${releasever}) instead of doing it per parse. This way we have the
expected semantics where one could do:

```
include: "fedora-coreos.yaml"
releasever: "42"
```

and have that releasever used.

Fixes #1809

Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>

Closes: #1848
Approved by: cgwalters
2019-06-18 18:19:42 +00:00
Rafael Fonseca
dbf28ac255 compose: Move json lockfile parsing to Rust
Using glib_json to parse the lockfile yields some oddities like
everything being wrapped in a GVariant. Let's leave the parsing to serde
in the Rust side of things. Hopefully that'll make the lockfile easier
to extend in the future.

Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>

Closes: #1851
Approved by: jlebon
2019-06-17 16:28:44 +00:00
Rafael Fonseca
e4d61b6b09 rust: Move open_file to utils.rs
Prep for next patch.

Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>

Closes: #1851
Approved by: jlebon
2019-06-17 16:28:44 +00:00
Colin Walters
5f6578ef3c compose: Add ostree-layers and ostree-override-layers
The use case for `ostree-layers` is to support injecting non-RPM
content in a more flexible way than can be done with `add-files`,
and also without dropping all the way to split composes.

This starts with support on the `compose tree` side but down the
line I'd like to make it more convenient to do *client* side too.

For `ostree-override-layers` this is mainly a development thing
for tools like coreos-assembler.  Rather than building an RPM
we just `make install DESTDIR` then commit and add to
`ostree-override-layers`.

Closes: #1830
Approved by: jlebon
2019-06-07 17:42:32 +00:00
Jonathan Lebon
6b2ac5885f rust: Stop using extern crate and #[macro_use]
Rust 2018 makes this much nicer now.

Closes: #1812
Approved by: lucab
2019-04-15 13:03:21 +00:00
Jonathan Lebon
278ba77774 rust: Reduce visibility of items
Use `pub(crate)` on the ffiutils functions since it's just for us. Drop
unnecessary `pub` fields on `TreefileExternals`.

Closes: #1812
Approved by: lucab
2019-04-15 13:03:21 +00:00
Jonathan Lebon
60e71dfc66 rust: Update to 2018 edition
This mostly involves adjusting our path uses to be anchored, e.g. at
`self::` or `crate::`. I took the opportunity to also tweak `use`
ordering so that we consistently list modules in external crates first
before our own.

Closes: #1812
Approved by: lucab
2019-04-15 13:03:21 +00:00
Jonathan Lebon
749facba86 rust: Bump dependencies
Now that our minimum Rust toolchain version was bumped, we can bump all
our deps to the latest and drop the frozen `rand` version that was done
for #1777.

Closes: #1810
Approved by: jlebon
2019-04-05 19:28:39 +00:00
Jonathan Lebon
95d1c6ea29 ci: Bump minimum Rust version to v1.31.0
Now that we no longer support el7, let's peg our minimum `rustc` version
to the latest Rust module available in el8 instead. That's 1.31.0 right
now.

And drop use of deprecated `trim_left_matches()` (it was deprecated in
1.33.0, which is currently used in the other testsuites that pull
`rustc` from the Fedora repos; we might want to eventually just use the
`RUST_MIN_VERSION` in all the testsuites to not get caught in the middle
like this again).

Closes: #1787
Approved by: cgwalters
2019-03-19 12:19:38 +00:00
Colin Walters
adff1e98b6 compose: Add boot-location: modules
And this (for now at least) completes the epic journey of the
"where's the kernel"?  With this it's found solely in
`/usr/lib/modules/$kver`.

There are a few reasons to do this; most prominent is that
it avoids duplicating the content as the locations may have
different SELinux labels.

Closes: #1773
Approved by: jlebon
2019-03-08 22:49:40 +00:00
Colin Walters
f37426f1ad compose: Remove support for boot-location: both
I'd like to add a new `boot-location: modules`.  In prep
for that, let's remove the legacy `both` which drops into
`/boot`.

The libostree support for handling `/usr/lib/ostree-boot` has
existed for over 4 years:

```
commit 37a059925f6b96d30190b65bee6bdde0ae1c6915
Commit:     Colin Walters <walters@verbum.org>
CommitDate: Sun Nov 30 23:14:05 2014 -0500

    deploy: Ensure that we can deploy using only /usr/lib/ostree-boot

```

I think we assume now that no one is now making *new* treecomposes and needs
a newer rpm-ostree and that they expect people to be able to use as an
upgrade target from a libostree that predates that.

Closes: #1773
Approved by: jlebon
2019-03-08 22:49:40 +00:00
Jonathan Lebon
e49f625b61 rust: Freeze rand crate to v0.6.1
The recent dep bump caused rand to jump to v0.6.5, which pulled in
`fuchsia-cprng`. Sadly, the cargo version we're frozen on chokes on it
because its manifest makes use of the `edition` keyword, which wasn't
stabilized yet.

Let's just freeze the problematic crate for now. We should be able to
bump our Rust toolchain soon-ish.

(Note this commit also does some other minor crate bumps as a result of
regenerating the lockfile to prune away `fuchsia-cprng`.)

Closes: #1777

Closes: #1778
Approved by: cgwalters
2019-03-07 21:57:08 +00:00
Jonathan Lebon
8b1bc9894e rust: Pass through rustfmt
Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Jonathan Lebon
1fe66e91d0 rust/treefile: Support dash convention for all options
Now that we support YAML, it's a gotcha/eyesore that some of our options
use underscores rather than dashes. Let's be nice and switch those few
options over, while of course still supporting the previous name.

Co-authored-by: Colin Walters <walters@verbum.org>

Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Jonathan Lebon
0ebfb6e4ff rust/treefile: Don't hardcode list of architectures
Just try to match the current architecture against any
`packages-${arch}` keys in the treefile.

Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Jonathan Lebon
24335bdf1a rust/treefile: Use HashMap to collect extra fields
Instead of the `Strict` and `Permissive` variants of
`TreeComposeConfig`, just collapse all the excess fields in a new
`extra` member and check for any keys there in the strict YAML path.
This will also allow us to drop the hardcoded list of architectures in
the next patch.

Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Jonathan Lebon
82594b5052 rust: Bump dependencies
Notably, the latest indicatif compiles fine again. This is now the
latest of everything, except the glib stack:

```
$ cargo outdated -R
Name      Project  Compat  Latest  Kind    Platform
----      -------  ------  ------  ----    --------
gio-sys   0.7.0    ---     0.8.0   Normal  ---
glib      0.6.1    ---     0.7.1   Normal  ---
glib-sys  0.7.0    ---     0.8.0   Normal  ---
```

because it requires a newer rustc than we're currently locked at.

Closes: #1770
Approved by: cgwalters
2019-03-01 15:51:07 +00:00
Jonathan Lebon
2be532c9e9 rust/treefile: Add basearch key
Add a `basearch` key to the manifest. This can be used at compose time
to assert the architecture the compose is running on. Though my
motivation is for the common case where it gets omitted from the input
manifest and gets automatically added by rpm-ostree into
`/usr/share/rpm-ostree/treefile.json` for introspection on the client.

(The crucial part here is that the treefile created by rpm-ostree
remains deserializable into a `TreeComposeConfig`).

Closes: https://github.com/coreos/fedora-coreos-tracker/issues/154

Closes: #1766
Approved by: cgwalters
2019-02-27 01:05:43 +00:00
Jonathan Lebon
6a57db1c36 rust/treefile: Rename arch -> basearch
We're really using this variable to substitute `${basearch}` and find
basearch-specific packages. Let's rename the variable to make that more
obvious.

Closes: #1766
Approved by: cgwalters
2019-02-27 01:05:43 +00:00
Colin Walters
3f722b5005 rust: Bump crate dependencies
Prep for adding other crate deps for sysusers, multiprocessing work.

Closes: #1765
Approved by: jlebon
2019-02-26 16:42:42 +00:00
Colin Walters
d1eedbbd96 rust/openat: Add helper to retrieve file type
Will be used by sysusers code.

Closes: #1764
Approved by: jlebon
2019-02-25 20:56:44 +00:00
Colin Walters
579f017fc2 rust: Add clap
A few of my outstanding Rust PRs use this (multiprocess, sysusers).
Let's add it now.

Closes: #1764
Approved by: jlebon
2019-02-25 20:56:44 +00:00
Jonathan Lebon
0e5044ca92 postprocess: Add /home -> /var/home SELinux substitution
Manually patch `file_contexts.subs_dist` so that `/home` is equivalent
to `/var/home`. This is required now that the generated homedirs rules
use `/var/home`. Otherwise, `matchpathcon` for example will return wrong
results.

This patch also includes the *removal* of `/var/home -> /home` so that
we're not dependent on this selinux-policy patch making it at the same
time as downstream:

https://src.fedoraproject.org/rpms/selinux-policy/pull-request/14

(See the conversation there for more information.)

Closes: #1754
Approved by: cgwalters
2019-02-14 17:24:09 +00:00
Colin Walters
85ab9c3181 treefile: Fix octal mode for rojig spec too
Going to need to see if clippy or something has a way for us
to reject leading `0`.

Closes: #1756
Approved by: jlebon
2019-02-11 18:21:51 +00:00
Colin Walters
07fbd195fb composepost: Fix permissions on /usr/etc/default/useradd
Switching between programming languages can be a pain.  Unfortunate
that Rust happily accepts a leading `0` too.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1752

Closes: #1753
Approved by: jlebon
2019-02-04 18:37:58 +00:00
Colin Walters
65afbec3b1 rust: Split openat helpers into module
Will help use them in other parts of our code.

Closes: #1737
Approved by: jlebon
2019-01-23 16:00:16 +00:00
Jonathan Lebon
25d0213d15 rust/treefile: Include filename in more error msgs
This uses the `Context` feature of the failure crate to make error
messages more useful when we fail to open a file. The difference with
`map_err` is that one can still obtain the underlying error from the
context if need be. Though surprisingly, the normal `Display` for a
`Context` doesn't include the original error, so we essentially have to
do a prefix here (see [1]).

Before:

```
error: Failed to load YAML treefile: No such file or directory (os error 2)
```

After:

```
error: Failed to load YAML treefile: Can't open file "treecompose-post.sh": No such file or directory (os error 2)
```

[1] https://github.com/rust-lang-nursery/failure/issues/182

Closes: #1735
Approved by: cgwalters
2019-01-22 15:49:34 +00:00
Jonathan Lebon
1594140a33 rust/treefile: Convert more functions to Fallible
Slightly prep for next patch (was in the area so just did the whole
module).

Closes: #1735
Approved by: cgwalters
2019-01-22 15:49:34 +00:00
Colin Walters
89cdeb5131 Add rayon dependency, do postprocessing in parallel
Obviously doing these trivial functions in parallel doesn't matter,
but I wanted to dip my toes into Rayon (and Rust parallelism in
general).

`try_for_each()` is just...staggeringly simple and beautiful.

Closes: #1727
Approved by: jlebon
2019-01-07 18:50:49 +00:00
Colin Walters
255dc7027e compose: Minor further oxidation of postprocessing
Take a baby step into further oxidation by migrating the bit
to write our presets.

Closes: #1727
Approved by: jlebon
2019-01-07 18:50:49 +00:00
Colin Walters
56e6ddb8e5 compose: Change /etc/default/useradd to use HOME=/var/home
For a long time I've resisted encoding "policy" into rpm-ostree
as much as possible.  Doing so makes it more distribution specific
for example.  That said, for `/var/home` there argument for doing
this in rpm-ostree is that we already make that symlink in our
hardcoded rootfs.  So we might as well do the other fixups for it.

https://github.com/coreos/fedora-coreos-config/pull/18
https://pagure.io/workstation-ostree-config/pull-request/121
https://discussion.fedoraproject.org/t/adapting-user-home-in-etc-passwd/487/6
https://github.com/justjanne/powerline-go/issues/94

Closes: #1726
Approved by: jlebon
2019-01-07 15:35:37 +00:00
Colin Walters
bb4a17d0ba compose: Add a stub "postprocess" entrypoint in Rust
Prep for more code.

Closes: #1726
Approved by: jlebon
2019-01-07 15:35:37 +00:00
Jonathan Lebon
fa64ebfa72 rust: Bump dependencies
This leaves only indicatif outdated (see link in diff).

Closes: #1720
Approved by: cgwalters
2018-12-19 21:25:13 +00:00
Jonathan Lebon
9a3cd0782b rust: Add Cargo.lock
We've been seeing more breakage recently due to much of the Rust
ecosystem sticking to minimum rustc versions more recent than ours. A
lockfile now means we have to periodically update it, but at least we
have much better control over our CI system and when we want to spend
time resolving package bump conflicts vs. just trying to get a small
patch in.

Closes: #1719
Approved by: cgwalters
2018-12-19 16:00:00 +00:00
Jonathan Lebon
f6de8f3dfc rust: Drop backtrace lock
We can build against the latest now that we bumped our minimum Rust
version.

Closes: #1716
Approved by: cgwalters
2018-12-17 22:06:49 +00:00
Jonathan Lebon
24d401072a Cargo.toml: Fix backtrace crate to compatibile version
The latest backtrace release isn't compatible with 1.26.2. A patch for
this is underway in:

https://github.com/alexcrichton/backtrace-rs/pull/137

Though for now, let's just restrict the version since we should be
bumping our minimum rustc requirement soon-ish anyway.

Closes: #1713
Approved by: cgwalters
2018-12-13 16:34:56 +00:00
Colin Walters
f811828543 rust/treefile: Add container key
This was omitted since in practice we aren't actually testing it,
the container path is mostly via `ex container` which uses keyfiles.

Closes: #1701

Closes: #1702
Approved by: jlebon
2018-12-06 19:03:06 +00:00
Colin Walters
09e7caef4e compose: Add a CUtf8Buf copy of rojig_name
Avoids passing an allocated buffer from Rust to C; there's
controversy in the PR I sent to rust-lang around defining this as
supported.

Closes: #1691
Approved by: jlebon
2018-11-28 16:53:00 +00:00
Colin Walters
62744449dc rust: Use Fallible<> more
Since use of the `failure` crate has been a success, let's use
it a bit more.  The big thing to convert left is `treefile.rs` which
does need a custom error so we can stop abusing `io::ErrorKind::InvalidInput`.

Closes: #1690
Approved by: jlebon
2018-11-28 16:00:56 +00:00
Colin Walters
a177f551b0 rust/ffi: Convert String creation API, extend docs
Create a new `_new_` naming convention, and extend the FFI
documentation to describe the new state as well as background assumptions.

Closes: #1688
Approved by: jlebon
2018-11-28 09:14:41 +00:00
Colin Walters
a2f5b7c1c0 rust/ffi: More "view" changes, for bytestrings
Change our existing "view as [u8] API", and also add one
that does a view as `OsStr`.  The motivation for the latter
is I noticed ithat `OsStr::from_bytes()` *doesn't* copy,
or rather it just copies the pointer value.  Rust's lifetime
inference ensures that the returned lifetime matches the input array.

I think the previous code in `treefile.rs` was confused about this.

Closes: #1688
Approved by: jlebon
2018-11-28 09:14:41 +00:00
Colin Walters
bfe637a877 rust/ffi: Convert str func to new "view" naming convention
More obvious one shouldn't leak the pointer.

Closes: #1688
Approved by: jlebon
2018-11-28 09:14:41 +00:00
Colin Walters
db0319beac rust/ffi: Change dfd handling to unwrap, create naming convention
I was going to add another usage of this function, and I think the
gerror stuff is unnecessary - if we are handed a bad file descriptor
(or a fd pointing to a regular file) that's something where we should
just abort.

While we're here, I'd like to codify expected usage in the function
names here.  If you like this I'll e.g. also change `str_from_nullable`
to `ffi_view_nullable_str`.

Closes: #1685
Approved by: jlebon
2018-11-27 15:42:08 +00:00
Colin Walters
e0a976386a rust: Use failure crate for errors
In a lot of places we're abusing `io::Error(io::ErrorKind::InvalidInput)`
which is both verbose and inaccurate really.  Maybe in some
places we should be defining custom errors, but eh.

I like the `failure` crate.  Use it in just `utils.rs` for now.
Tweak our error handling FFI wrappers to accept `Display` since
all we do is convert the error to a string.

Closes: #1675
Approved by: lucab
2018-11-21 22:03:53 +00:00
Colin Walters
6b5e475607 rust: Rerun rustfmt
Going to add a CI step next.

Closes: #1674
Approved by: dustymabe
2018-11-21 21:16:03 +00:00
Colin Walters
1dee43319c Use indicatif for progress
This turned out to be messier than I thought, because of two primary
factors; the biggest mess here of course is the indirection
through the DBus API.

The other problem is that previously we passed the string to render
each time, and with current indicatif that'd trigger a rerender.
Since (usually) don't change the "prefix string", rework the API.

Change the "percent/n_items" bits to use autocleanups as well, and
to take the prefix string as an initial argument.

Since the state expands to multiple components, also change the
API to use the `0-initialized` pattern rather than trying to
return an aggregate.

We also gain a "sub message" which we use to display e.g.
package names as we're doing checkouts.  Note this ends up
at the end, since otherwise everything else jumps around.

Closes: #1661
Approved by: rfairley
2018-11-09 13:59:43 +00:00
Colin Walters
ef587b5c5b build-sys: Turn Rust LTO off by default, add --enable-lto flag
For us, this is primarily right now a size issue.  See:
https://internals.rust-lang.org/t/rust-staticlibs-and-optimizing-for-size/5746

For more information, there are these two issues:
https://github.com/rust-lang/cargo/issues/4349
https://bugzilla.mozilla.org/show_bug.cgi?id=1386371

The basic issue here is that a build with LTO off (and a trivial
change to add a `println!` takes 14s here, and with it on takes 38s.
However, with LTO off the stripped size of `librpmostree_rust.a` is
`6M`, with LTO on it's `1.1M`.

I named this `--enable-lto` as I'd like to investigate doing this
for the C code too.

Closes: #1664
Approved by: jlebon
2018-11-05 18:41:09 +00:00
Colin Walters
50b255a8a9 Move varsubst code into Rust, use it in treefile parsing
External tools often want to parse the ref; for example coreos-assembler
currently does so.  Let's ensure `${basearch}` is expanded with
`--print-only` so they can parse that JSON to get the expanded version
reliably.

Implementation note: this is the first Rust code which exposes a
"GLib-like" C API, notably with GHashTable, so we're making more use
of the glib-rs bindings.

Closes: #1653

Closes: #1655
Approved by: jlebon
2018-11-02 20:36:28 +00:00
Colin Walters
88ffdc0ae2 Add compose rojig command
This currently requires a `--i-know-this-is-experimental` flag;
I know it'd be a bit more consistent to have it under `ex`, but
what feels weird about that is *most* of the `ex` commands people
use are client side.  This is where we want it to ultimately end
up.

We've landed a lot of prep patches, but I know there's still
a notable amount of code duplication with `compose tree`.  What's
left is about ~700 lines but it's mostly not hard/complex code
anymore.

In the future, I'd like to extract more of the compose code
to a `rust/src/compose.rs` or so, but I think this is sustainable
fow now.

My high level goal is to get this into coreos-assembler and stand
up a Silverblue build that uses it.

Closes: #1512
Approved by: jlebon
2018-10-31 17:57:45 +00:00
Colin Walters
c044f59726 rust: Remove pub from modules
Conceptually we're only exporting the FFI code right now.

Closes: #1649
Approved by: jlebon
2018-10-30 16:16:22 +00:00