Commit Graph

76 Commits

Author SHA1 Message Date
Colin Walters
56e6ddb8e5 compose: Change /etc/default/useradd to use HOME=/var/home
For a long time I've resisted encoding "policy" into rpm-ostree
as much as possible.  Doing so makes it more distribution specific
for example.  That said, for `/var/home` there argument for doing
this in rpm-ostree is that we already make that symlink in our
hardcoded rootfs.  So we might as well do the other fixups for it.

https://github.com/coreos/fedora-coreos-config/pull/18
https://pagure.io/workstation-ostree-config/pull-request/121
https://discussion.fedoraproject.org/t/adapting-user-home-in-etc-passwd/487/6
https://github.com/justjanne/powerline-go/issues/94

Closes: #1726
Approved by: jlebon
2019-01-07 15:35:37 +00:00
Colin Walters
bb4a17d0ba compose: Add a stub "postprocess" entrypoint in Rust
Prep for more code.

Closes: #1726
Approved by: jlebon
2019-01-07 15:35:37 +00:00
Jonathan Lebon
fa64ebfa72 rust: Bump dependencies
This leaves only indicatif outdated (see link in diff).

Closes: #1720
Approved by: cgwalters
2018-12-19 21:25:13 +00:00
Jonathan Lebon
9a3cd0782b rust: Add Cargo.lock
We've been seeing more breakage recently due to much of the Rust
ecosystem sticking to minimum rustc versions more recent than ours. A
lockfile now means we have to periodically update it, but at least we
have much better control over our CI system and when we want to spend
time resolving package bump conflicts vs. just trying to get a small
patch in.

Closes: #1719
Approved by: cgwalters
2018-12-19 16:00:00 +00:00
Jonathan Lebon
f6de8f3dfc rust: Drop backtrace lock
We can build against the latest now that we bumped our minimum Rust
version.

Closes: #1716
Approved by: cgwalters
2018-12-17 22:06:49 +00:00
Jonathan Lebon
24d401072a Cargo.toml: Fix backtrace crate to compatibile version
The latest backtrace release isn't compatible with 1.26.2. A patch for
this is underway in:

https://github.com/alexcrichton/backtrace-rs/pull/137

Though for now, let's just restrict the version since we should be
bumping our minimum rustc requirement soon-ish anyway.

Closes: #1713
Approved by: cgwalters
2018-12-13 16:34:56 +00:00
Colin Walters
f811828543 rust/treefile: Add container key
This was omitted since in practice we aren't actually testing it,
the container path is mostly via `ex container` which uses keyfiles.

Closes: #1701

Closes: #1702
Approved by: jlebon
2018-12-06 19:03:06 +00:00
Colin Walters
09e7caef4e compose: Add a CUtf8Buf copy of rojig_name
Avoids passing an allocated buffer from Rust to C; there's
controversy in the PR I sent to rust-lang around defining this as
supported.

Closes: #1691
Approved by: jlebon
2018-11-28 16:53:00 +00:00
Colin Walters
62744449dc rust: Use Fallible<> more
Since use of the `failure` crate has been a success, let's use
it a bit more.  The big thing to convert left is `treefile.rs` which
does need a custom error so we can stop abusing `io::ErrorKind::InvalidInput`.

Closes: #1690
Approved by: jlebon
2018-11-28 16:00:56 +00:00
Colin Walters
a177f551b0 rust/ffi: Convert String creation API, extend docs
Create a new `_new_` naming convention, and extend the FFI
documentation to describe the new state as well as background assumptions.

Closes: #1688
Approved by: jlebon
2018-11-28 09:14:41 +00:00
Colin Walters
a2f5b7c1c0 rust/ffi: More "view" changes, for bytestrings
Change our existing "view as [u8] API", and also add one
that does a view as `OsStr`.  The motivation for the latter
is I noticed ithat `OsStr::from_bytes()` *doesn't* copy,
or rather it just copies the pointer value.  Rust's lifetime
inference ensures that the returned lifetime matches the input array.

I think the previous code in `treefile.rs` was confused about this.

Closes: #1688
Approved by: jlebon
2018-11-28 09:14:41 +00:00
Colin Walters
bfe637a877 rust/ffi: Convert str func to new "view" naming convention
More obvious one shouldn't leak the pointer.

Closes: #1688
Approved by: jlebon
2018-11-28 09:14:41 +00:00
Colin Walters
db0319beac rust/ffi: Change dfd handling to unwrap, create naming convention
I was going to add another usage of this function, and I think the
gerror stuff is unnecessary - if we are handed a bad file descriptor
(or a fd pointing to a regular file) that's something where we should
just abort.

While we're here, I'd like to codify expected usage in the function
names here.  If you like this I'll e.g. also change `str_from_nullable`
to `ffi_view_nullable_str`.

Closes: #1685
Approved by: jlebon
2018-11-27 15:42:08 +00:00
Colin Walters
e0a976386a rust: Use failure crate for errors
In a lot of places we're abusing `io::Error(io::ErrorKind::InvalidInput)`
which is both verbose and inaccurate really.  Maybe in some
places we should be defining custom errors, but eh.

I like the `failure` crate.  Use it in just `utils.rs` for now.
Tweak our error handling FFI wrappers to accept `Display` since
all we do is convert the error to a string.

Closes: #1675
Approved by: lucab
2018-11-21 22:03:53 +00:00
Colin Walters
6b5e475607 rust: Rerun rustfmt
Going to add a CI step next.

Closes: #1674
Approved by: dustymabe
2018-11-21 21:16:03 +00:00
Colin Walters
1dee43319c Use indicatif for progress
This turned out to be messier than I thought, because of two primary
factors; the biggest mess here of course is the indirection
through the DBus API.

The other problem is that previously we passed the string to render
each time, and with current indicatif that'd trigger a rerender.
Since (usually) don't change the "prefix string", rework the API.

Change the "percent/n_items" bits to use autocleanups as well, and
to take the prefix string as an initial argument.

Since the state expands to multiple components, also change the
API to use the `0-initialized` pattern rather than trying to
return an aggregate.

We also gain a "sub message" which we use to display e.g.
package names as we're doing checkouts.  Note this ends up
at the end, since otherwise everything else jumps around.

Closes: #1661
Approved by: rfairley
2018-11-09 13:59:43 +00:00
Colin Walters
ef587b5c5b build-sys: Turn Rust LTO off by default, add --enable-lto flag
For us, this is primarily right now a size issue.  See:
https://internals.rust-lang.org/t/rust-staticlibs-and-optimizing-for-size/5746

For more information, there are these two issues:
https://github.com/rust-lang/cargo/issues/4349
https://bugzilla.mozilla.org/show_bug.cgi?id=1386371

The basic issue here is that a build with LTO off (and a trivial
change to add a `println!` takes 14s here, and with it on takes 38s.
However, with LTO off the stripped size of `librpmostree_rust.a` is
`6M`, with LTO on it's `1.1M`.

I named this `--enable-lto` as I'd like to investigate doing this
for the C code too.

Closes: #1664
Approved by: jlebon
2018-11-05 18:41:09 +00:00
Colin Walters
50b255a8a9 Move varsubst code into Rust, use it in treefile parsing
External tools often want to parse the ref; for example coreos-assembler
currently does so.  Let's ensure `${basearch}` is expanded with
`--print-only` so they can parse that JSON to get the expanded version
reliably.

Implementation note: this is the first Rust code which exposes a
"GLib-like" C API, notably with GHashTable, so we're making more use
of the glib-rs bindings.

Closes: #1653

Closes: #1655
Approved by: jlebon
2018-11-02 20:36:28 +00:00
Colin Walters
88ffdc0ae2 Add compose rojig command
This currently requires a `--i-know-this-is-experimental` flag;
I know it'd be a bit more consistent to have it under `ex`, but
what feels weird about that is *most* of the `ex` commands people
use are client side.  This is where we want it to ultimately end
up.

We've landed a lot of prep patches, but I know there's still
a notable amount of code duplication with `compose tree`.  What's
left is about ~700 lines but it's mostly not hard/complex code
anymore.

In the future, I'd like to extract more of the compose code
to a `rust/src/compose.rs` or so, but I think this is sustainable
fow now.

My high level goal is to get this into coreos-assembler and stand
up a Silverblue build that uses it.

Closes: #1512
Approved by: jlebon
2018-10-31 17:57:45 +00:00
Colin Walters
c044f59726 rust: Remove pub from modules
Conceptually we're only exporting the FFI code right now.

Closes: #1649
Approved by: jlebon
2018-10-30 16:16:22 +00:00
Colin Walters
87441d9ec2 rust: Move FFI bits into ffi submodule for each file
The `lib.rs` file was starting to accumulate, and I want to do this
before adding more.  I just made this up, but an `ffi` submodule
in each file seems to work well.  It isolates the FFI consumption
there still (so e.g. if we want pure Rust unit tests, we don't
need to deal with FFI).

Closes: #1646
Approved by: jlebon
2018-10-30 14:24:42 +00:00
Jonathan Lebon
940fc1364a compose: Check that add-files are compatible after parsing
While serde gives us type checking, it of course doesn't understand
semantics beyond that. One example is checking the compatibility of
`add-files` entries with the OSTree model. This is something we can do
upfront early on to avoid surprises for users.

Also tweak the docs to reflect this new check.

Related: #1642

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Colin Walters
202192d61e rust: Create generic ref_from_raw_ptr function
I need to do some hunting for a "ffi helper" crate.  I kind of
understand why these things aren't in the std library.

Anyways this is easy for now, prep for adding more types.

Closes: #1641
Approved by: jlebon
2018-10-26 20:04:08 +00:00
Colin Walters
92d207de5d rust: Merge glibutils with utility FFI helpers
Create `ffiutil.rs` from the merger.  This way all of our FFI
helpers are in one place.  Prep for further refactoring.

Closes: #1641
Approved by: jlebon
2018-10-26 20:04:08 +00:00
Jonathan Lebon
a87486e18e rust: Drop crates-io patch and use 0.4.0
The latest release of the crate includes the features and fixes we need.

Closes: #1639
Approved by: cgwalters
2018-10-25 19:01:39 +00:00
Colin Walters
fe5adaf7ab build-sys: Fix make rustfmt, and run it
When we added the bindgen config this broke as it's not a `.rs`.
Let's keep the source formatted canonically.

Closes: #1635
Approved by: jlebon
2018-10-23 19:05:06 +00:00
Colin Walters
f1fa436c8a rust/treefile: Use macros to reduce redundancy in treefile merging
On the plus side, when submitting a patch to Github, no one knows
how long it took you to figure out...

Anyways so this reduces redundancy.  I double-checked the list.
I was inspired to pick this back up after seeing a Rust code
snippet somewhere noting that macros defined inside a function
can capture variables, which simplifies this even more.

Closes: #1631
Approved by: jlebon
2018-10-22 18:53:16 +00:00
Jonathan Lebon
67a831b043 rust/journal: Adapt for el7 platforms
Supporting ancient systemd is painful, though given that it may end up
in RHCOS at least, it seems worth the effort.

Basically, the big changes here are:
- avoid using `_SYSTEMD_UNIT` since it might be missing in some entries
- also grep for the el7 version of the systemd msg when a service fails
- use `_TRANSPORT=stdout` for the error msg in case of ordering issues

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
676519e5a5 status: Print systemd or ostree staged failure msg
Try to tease out a bit more info from the journal by looking at the
systemd message when the service transitions to the dead state or even
looking at the OSTree output itself.

Example outputs:

```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         error: opendir(ostree/deploy/fedora-atomic/deploy/887c95887a3047a60372016a0d84536530755b60df3cca33c819f7606e220adf.0): No such file or directory
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         ostree-finalize-staged.service: Failed with result 'timeout'.
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Colin Walters
808a80bfd7 rust/lib: Remove unnecessary integer casts
The compiler will infer the correct thing.

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
f6c3616148 status: Detect if staging failed in previous boot
Sample output:

```
$ rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

(Though open to tweaking it).

I also played with directly invoking `journalctl` for the user, but that
can get really spammy with e.g. `os-prober` output and such.

I wrote this in Rust using journal API wrappers because I also plan to
implement the `history` command in Rust and will also enhance that new
`journal` module there for that.

Requires: https://github.com/ostreedev/ostree/pull/1750
Requires: https://github.com/jmesmon/rust-systemd/pull/54
(Though I've pointed the manifest at my branch for now for CI).

Closes: #1567

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Colin Walters
fca7c4bd35 rust/treefile: Include filename in error when parsing
"Entity not found" is not super useful.

Closes: #1607
Approved by: jlebon
2018-10-12 14:31:55 +00:00
Colin Walters
096f8dec65 rust/lib: Rework fd helpers to consistently lseek()
This way the fds are always known to be at the start.  For e.g.
`add-files` we need to both checksum them and then copy them
later.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
6846fe11fa compose: Move opening passwd/group files into Rust
This was I think the last place where we were looking at the parent
directory of the primary treefile.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
c6cdda2c91 lib/rust: Add a helper to convert the raw pointer
So there's fewer uses of `unsafe`.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
0e5c1cde96 compose: Move "serialized treefile" into Rust ownership
Now that we have `CUtf8`, let's just store the serialized JSON
as a string, owned by the Rust side.  This way we can drop the
`serialized_treefile` buffer we were passing around and simplify
various bits of code.  Most notably, we only serialize the JSON
once (via Serde) rather than also doing it again in the C side.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
e3be475566 compose+rust: Parse includes via Rust too
This follows up to https://github.com/projectatomic/rpm-ostree/pull/1576
AKA commit 2e567840ca - we now process
treefile inheritance in Rust code.  Previously for elements which
reference external files (`postprocess-script` and `add-files`)
we'd hardcoded things to only look in the first context dir.

Now we open file descriptors in the Rust side for these "externals"
as we're parsing, and load them C side.  Hence we'll correctly handle
a `postprocess-script` from an included config.

Other advantages are that the include handling was ugly un-typesafe C code
with no unit tests, now it's memory safe Rust with unit tests.

The downside here is I ended up spelling out the list of fields
again - there's probably a way to unify this via macros but
for now I think this is OK.

Closes: #1574
Approved by: jlebon
2018-10-04 13:17:47 +00:00
Colin Walters
363294c3ef rust/bindgen: Use #pragma once
Same as we do for our other headers.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Colin Walters
1cf8283904 rust: Make io::Read a generic again
No reason to do virtual dispatch here.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Colin Walters
7595bce228 rust/treefile: Use the c_utf8 crate
The advantage of this over CStr is that Rust knows it's UTF-8
too.  I also tweaked our path code to use String, and only
view it as a `Path`.  This avoids having to `unwrap()` later
back to a `str`.

Closes: #1588
Approved by: jlebon
2018-09-28 16:21:18 +00:00
Colin Walters
64d02e541c rust/treefile: Fix the type of add-files.
It's really an array of pairs.  Tripped over this while working
on the pure-Rust treefile parsing on the `misc-tweaks` compose test.

Closes: #1581
Approved by: jlebon
2018-09-26 20:15:30 +00:00
Colin Walters
8df07a3b1e rust/treefile: Add support for parsing JSON too
Prep for moving more of our parsing into Rust.  The main
thing here is that for JSON, we need to continue to ignore
unknown fields.  It took me a little while but I eventually
figured out that using `#[serde(flatten)]` works for this.
Seriously: serde is freaking amazingly awesome.

Closes: #1580
Approved by: jlebon
2018-09-26 16:40:40 +00:00
Colin Walters
58e7c34823 rust: Run rustfmt
Prep for further changes.

Closes: #1580
Approved by: jlebon
2018-09-26 16:40:40 +00:00
Colin Walters
f50f9e8d7e Split cbindgen to separate build, support external version
The problem is building bindgen as part of our single run
locks serde to way old versions, and I want to use newer versions.

Since Fedora will now again ship a `cbindgen` package, let's
also support using it if we find it, saving ourselves
the cost of building it.

For distros that don't ship it (e.g. CentOS) for CI purposes
we build it.  For downstream builds that are offline, rather
than vendor the cbindgen sources like we do with our main Rust,
let's just vendor the `rpmostree-rust.h` file as was suggested
in https://bugzilla.redhat.com/show_bug.cgi?id=1608670

Closes: https://github.com/projectatomic/rpm-ostree/issues/1557

Closes: #1573
Approved by: jlebon
2018-09-25 20:29:21 +00:00
Colin Walters
304f9f6a64 treefile: Redo rojig_spec to be Box<CStr>
Apparently I was just totally confused into thinking `OsStr`
was `NUL` terminated.  I don't know where I got that idea
but a quick inspection of the Rust source code makes this
clear.  I may submit a PR to the docs.

So we have two choices: allocate, or internally use a `NUL`-terminated
type.  Let's just use `Box<CStr>` internally.  Note I chose that
over `CString` as the latter is theoretically mutable, and when
passing data back over to C we don't have ownership/mutability
guarantees.

Closes: #1563
Approved by: jlebon
2018-09-17 15:51:53 +00:00
Colin Walters
02b597b690 rust/bindgen: Add an autoptr cleanup for our struct
I'm going to add a new user soon.  This should probably
be a best practice.

Ideally...cbindgen would have a way to add some wrapper bits
around any types it defines or something?

Closes: #1559
Approved by: jlebon
2018-09-13 21:49:06 +00:00
Colin Walters
36bf3c9164 build: Regenerate rpmostree-rust.h if cbindgen.toml changes
So when I edit it I get the changes.  Figuring this out
took me a little while, we were missing two dependencies.

Closes: #1559
Approved by: jlebon
2018-09-13 21:49:06 +00:00
Colin Walters
23698a8aa3 rust: Use debug=true for release builds
Because our primary delivery vechicle uses RPM which
supports separate debuginfo, there's no reason not to build release
builds with debuginfo, so that one can use gdb on them just
like we do with C.

Closes: #1550
Approved by: jlebon
2018-09-11 16:28:15 +00:00
Colin Walters
d2b0e42bfc Use cbindgen
If we're going to scale out our oxidation, let's follow
the path of Firefox (and other projects) further and use
cbindgen: https://github.com/eqrion/cbindgen

It's actually nice that `cbindgen` is packaged today in Fedora,
but I doubt it is elsewhere; we may end up needing to push
that forward, or just vendor it via a `build.rs` script and Cargo.

I chose to rename things to `ROR`/`ror_` since it's shorter.  I
am tempted a bit to rename our internal functions to just `ro_` to
or so.

Closes: #1516
Approved by: jlebon
2018-09-06 13:34:32 +00:00
Colin Walters
279e7c4f1b treefile: Support inline postprocess element (for inheritance)
I'm trying to have a more opinionated model where custom builds
use inheritance, and currently one can only have a single
`postprocess-script`.

Further, in YAML it's very convenient to use inline vs external
data.

Closes: #1527
Approved by: jlebon
2018-09-05 15:27:07 +00:00