Commit Graph

55 Commits

Author SHA1 Message Date
Jonathan Lebon
940fc1364a compose: Check that add-files are compatible after parsing
While serde gives us type checking, it of course doesn't understand
semantics beyond that. One example is checking the compatibility of
`add-files` entries with the OSTree model. This is something we can do
upfront early on to avoid surprises for users.

Also tweak the docs to reflect this new check.

Related: #1642

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Colin Walters
202192d61e rust: Create generic ref_from_raw_ptr function
I need to do some hunting for a "ffi helper" crate.  I kind of
understand why these things aren't in the std library.

Anyways this is easy for now, prep for adding more types.

Closes: #1641
Approved by: jlebon
2018-10-26 20:04:08 +00:00
Colin Walters
92d207de5d rust: Merge glibutils with utility FFI helpers
Create `ffiutil.rs` from the merger.  This way all of our FFI
helpers are in one place.  Prep for further refactoring.

Closes: #1641
Approved by: jlebon
2018-10-26 20:04:08 +00:00
Jonathan Lebon
a87486e18e rust: Drop crates-io patch and use 0.4.0
The latest release of the crate includes the features and fixes we need.

Closes: #1639
Approved by: cgwalters
2018-10-25 19:01:39 +00:00
Colin Walters
fe5adaf7ab build-sys: Fix make rustfmt, and run it
When we added the bindgen config this broke as it's not a `.rs`.
Let's keep the source formatted canonically.

Closes: #1635
Approved by: jlebon
2018-10-23 19:05:06 +00:00
Colin Walters
f1fa436c8a rust/treefile: Use macros to reduce redundancy in treefile merging
On the plus side, when submitting a patch to Github, no one knows
how long it took you to figure out...

Anyways so this reduces redundancy.  I double-checked the list.
I was inspired to pick this back up after seeing a Rust code
snippet somewhere noting that macros defined inside a function
can capture variables, which simplifies this even more.

Closes: #1631
Approved by: jlebon
2018-10-22 18:53:16 +00:00
Jonathan Lebon
67a831b043 rust/journal: Adapt for el7 platforms
Supporting ancient systemd is painful, though given that it may end up
in RHCOS at least, it seems worth the effort.

Basically, the big changes here are:
- avoid using `_SYSTEMD_UNIT` since it might be missing in some entries
- also grep for the el7 version of the systemd msg when a service fails
- use `_TRANSPORT=stdout` for the error msg in case of ordering issues

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
676519e5a5 status: Print systemd or ostree staged failure msg
Try to tease out a bit more info from the journal by looking at the
systemd message when the service transitions to the dead state or even
looking at the OSTree output itself.

Example outputs:

```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         error: opendir(ostree/deploy/fedora-atomic/deploy/887c95887a3047a60372016a0d84536530755b60df3cca33c819f7606e220adf.0): No such file or directory
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         ostree-finalize-staged.service: Failed with result 'timeout'.
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Colin Walters
808a80bfd7 rust/lib: Remove unnecessary integer casts
The compiler will infer the correct thing.

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
f6c3616148 status: Detect if staging failed in previous boot
Sample output:

```
$ rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

(Though open to tweaking it).

I also played with directly invoking `journalctl` for the user, but that
can get really spammy with e.g. `os-prober` output and such.

I wrote this in Rust using journal API wrappers because I also plan to
implement the `history` command in Rust and will also enhance that new
`journal` module there for that.

Requires: https://github.com/ostreedev/ostree/pull/1750
Requires: https://github.com/jmesmon/rust-systemd/pull/54
(Though I've pointed the manifest at my branch for now for CI).

Closes: #1567

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Colin Walters
fca7c4bd35 rust/treefile: Include filename in error when parsing
"Entity not found" is not super useful.

Closes: #1607
Approved by: jlebon
2018-10-12 14:31:55 +00:00
Colin Walters
096f8dec65 rust/lib: Rework fd helpers to consistently lseek()
This way the fds are always known to be at the start.  For e.g.
`add-files` we need to both checksum them and then copy them
later.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
6846fe11fa compose: Move opening passwd/group files into Rust
This was I think the last place where we were looking at the parent
directory of the primary treefile.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
c6cdda2c91 lib/rust: Add a helper to convert the raw pointer
So there's fewer uses of `unsafe`.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
0e5c1cde96 compose: Move "serialized treefile" into Rust ownership
Now that we have `CUtf8`, let's just store the serialized JSON
as a string, owned by the Rust side.  This way we can drop the
`serialized_treefile` buffer we were passing around and simplify
various bits of code.  Most notably, we only serialize the JSON
once (via Serde) rather than also doing it again in the C side.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
e3be475566 compose+rust: Parse includes via Rust too
This follows up to https://github.com/projectatomic/rpm-ostree/pull/1576
AKA commit 2e567840ca - we now process
treefile inheritance in Rust code.  Previously for elements which
reference external files (`postprocess-script` and `add-files`)
we'd hardcoded things to only look in the first context dir.

Now we open file descriptors in the Rust side for these "externals"
as we're parsing, and load them C side.  Hence we'll correctly handle
a `postprocess-script` from an included config.

Other advantages are that the include handling was ugly un-typesafe C code
with no unit tests, now it's memory safe Rust with unit tests.

The downside here is I ended up spelling out the list of fields
again - there's probably a way to unify this via macros but
for now I think this is OK.

Closes: #1574
Approved by: jlebon
2018-10-04 13:17:47 +00:00
Colin Walters
363294c3ef rust/bindgen: Use #pragma once
Same as we do for our other headers.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Colin Walters
1cf8283904 rust: Make io::Read a generic again
No reason to do virtual dispatch here.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Colin Walters
7595bce228 rust/treefile: Use the c_utf8 crate
The advantage of this over CStr is that Rust knows it's UTF-8
too.  I also tweaked our path code to use String, and only
view it as a `Path`.  This avoids having to `unwrap()` later
back to a `str`.

Closes: #1588
Approved by: jlebon
2018-09-28 16:21:18 +00:00
Colin Walters
64d02e541c rust/treefile: Fix the type of add-files.
It's really an array of pairs.  Tripped over this while working
on the pure-Rust treefile parsing on the `misc-tweaks` compose test.

Closes: #1581
Approved by: jlebon
2018-09-26 20:15:30 +00:00
Colin Walters
8df07a3b1e rust/treefile: Add support for parsing JSON too
Prep for moving more of our parsing into Rust.  The main
thing here is that for JSON, we need to continue to ignore
unknown fields.  It took me a little while but I eventually
figured out that using `#[serde(flatten)]` works for this.
Seriously: serde is freaking amazingly awesome.

Closes: #1580
Approved by: jlebon
2018-09-26 16:40:40 +00:00
Colin Walters
58e7c34823 rust: Run rustfmt
Prep for further changes.

Closes: #1580
Approved by: jlebon
2018-09-26 16:40:40 +00:00
Colin Walters
f50f9e8d7e Split cbindgen to separate build, support external version
The problem is building bindgen as part of our single run
locks serde to way old versions, and I want to use newer versions.

Since Fedora will now again ship a `cbindgen` package, let's
also support using it if we find it, saving ourselves
the cost of building it.

For distros that don't ship it (e.g. CentOS) for CI purposes
we build it.  For downstream builds that are offline, rather
than vendor the cbindgen sources like we do with our main Rust,
let's just vendor the `rpmostree-rust.h` file as was suggested
in https://bugzilla.redhat.com/show_bug.cgi?id=1608670

Closes: https://github.com/projectatomic/rpm-ostree/issues/1557

Closes: #1573
Approved by: jlebon
2018-09-25 20:29:21 +00:00
Colin Walters
304f9f6a64 treefile: Redo rojig_spec to be Box<CStr>
Apparently I was just totally confused into thinking `OsStr`
was `NUL` terminated.  I don't know where I got that idea
but a quick inspection of the Rust source code makes this
clear.  I may submit a PR to the docs.

So we have two choices: allocate, or internally use a `NUL`-terminated
type.  Let's just use `Box<CStr>` internally.  Note I chose that
over `CString` as the latter is theoretically mutable, and when
passing data back over to C we don't have ownership/mutability
guarantees.

Closes: #1563
Approved by: jlebon
2018-09-17 15:51:53 +00:00
Colin Walters
02b597b690 rust/bindgen: Add an autoptr cleanup for our struct
I'm going to add a new user soon.  This should probably
be a best practice.

Ideally...cbindgen would have a way to add some wrapper bits
around any types it defines or something?

Closes: #1559
Approved by: jlebon
2018-09-13 21:49:06 +00:00
Colin Walters
36bf3c9164 build: Regenerate rpmostree-rust.h if cbindgen.toml changes
So when I edit it I get the changes.  Figuring this out
took me a little while, we were missing two dependencies.

Closes: #1559
Approved by: jlebon
2018-09-13 21:49:06 +00:00
Colin Walters
23698a8aa3 rust: Use debug=true for release builds
Because our primary delivery vechicle uses RPM which
supports separate debuginfo, there's no reason not to build release
builds with debuginfo, so that one can use gdb on them just
like we do with C.

Closes: #1550
Approved by: jlebon
2018-09-11 16:28:15 +00:00
Colin Walters
d2b0e42bfc Use cbindgen
If we're going to scale out our oxidation, let's follow
the path of Firefox (and other projects) further and use
cbindgen: https://github.com/eqrion/cbindgen

It's actually nice that `cbindgen` is packaged today in Fedora,
but I doubt it is elsewhere; we may end up needing to push
that forward, or just vendor it via a `build.rs` script and Cargo.

I chose to rename things to `ROR`/`ror_` since it's shorter.  I
am tempted a bit to rename our internal functions to just `ro_` to
or so.

Closes: #1516
Approved by: jlebon
2018-09-06 13:34:32 +00:00
Colin Walters
279e7c4f1b treefile: Support inline postprocess element (for inheritance)
I'm trying to have a more opinionated model where custom builds
use inheritance, and currently one can only have a single
`postprocess-script`.

Further, in YAML it's very convenient to use inline vs external
data.

Closes: #1527
Approved by: jlebon
2018-09-05 15:27:07 +00:00
Colin Walters
3d60a31aaa Fix include: with machineid-compat and a few other keys
It turns out we basically have to slap an `Option<T>` around
everything, (in particular `bool` etc.) we need to be able
to distinguish in (I believe) all the cases between
"value unspecified" and "value provided".

Concretely it didn't work to try to set `machineid-compat: false`
in an included yaml treefile becuase it was just defaulted to `true`
by the toplevel.

Down the line we should move all of the parsing into Rust
and have two different `struct` types for "YAML we load" versus
"verified treefile".

Closes: https://github.com/projectatomic/rpm-ostree/issues/1524

Closes: #1525
Approved by: lucab
2018-08-29 12:59:34 +00:00
Jonathan Lebon
04c0678fa6 app: Add support for passing URLs to RPMs
This teaches the client to fetch packages from URLs directly so that one
doesn't have to `curl` first and then install. Supported anywhere
package filenames are allowed (notably: `install` and
`override replace`).

One neat things about this is that we download the file into an
`O_TMPFILE` and then pass on ownership of that fd directly to the
daemon. So at no point are the packages actually laying visible on the
system. (Assuming the filesystem supports `O_TMPFILE` that is).

This adds direct linking to libcurl and openssl, two libraries which we
were already pulling in indirectly.

Closes: #1508
Approved by: cgwalters
2018-08-23 11:16:15 +00:00
Colin Walters
40be3fb1cf Add recommends to treefile (default on)
This is for: https://github.com/projectatomic/rpm-ostree/issues/718
But I'm not going to close that issue as this only does the server
side, and I think we should support it client side too.

Since I wrote that issue, we ended up skipping the `dnf_transaction_depsolve()`
API, and hence we don't need to block on a libdnf change.  So
this was quite simple.

Closes: #1513
Approved by: jlebon
2018-08-23 04:24:07 +00:00
Colin Walters
2032da9b21 treefile: Make ref optional in YAML
It currently works to convert *toplevel* configs to YAML, but
if one wants to make use of `include:`, today you'd have to specify
dummy `ref` entries on the includes.

Further, for rojig-only mode, one doesn't need `ref`.

Closes: #1511
Approved by: jlebon
2018-08-21 01:04:18 +00:00
Jonathan Lebon
1263886c4c compose: Check for packages after processing includes
Otherwise we risk rejecting perfectly valid treefiles. E.g.
fedora-atomic only defines packages in the `-base` file. Let's just move
the check to after having processed all the includes, right where we
collate packages from all the various entries.

The FAHC treecompose is hitting this right now.

Closes: #1500
Approved by: cgwalters
2018-08-13 19:18:38 +00:00
Colin Walters
344aee1d76 rust: Add support for inline rojig spec files
The rojig spec is almost entirely rpm-ostree implementation details;
let's not have lots of people fork/duplicate it.  Rather add the bits
of rojig to the treefile that people need to define (most notably
the name).

Prep for stabilizing rojig.

I had a few false starts with this PR; managing ownership/lifetimes
across C/Rust is just complicated.  I got bit hard by the fact that
the workdir in `--unified-core` is really dfd-relative, and had to
do a dance to propagate the dfd into rust, as well as down into
the rojig builder.

Closes: #1484
Approved by: jlebon
2018-08-03 16:54:47 +00:00
Colin Walters
16be1a0bad rust: Rework treefile to be an object
In a later patch I'm going to add more API; basically rather
than doing the JSON parsing from C, we can add APIs to directly
access the treefile object.  This also demonstrates how we
can do more extensive APIs, in particular implement an "object"
in Rust.

The ownership across the FFI boundary becomes nicer here too,
we don't need to do a dance with the fd.

For writing this I found
http://jakegoulding.com/rust-ffi-omnibus/objects/
quite useful, as well as
https://github.com/rust-lang/regex/blob/master/regex-capi/src/rure.rs

Closes: #1474
Approved by: jlebon
2018-07-30 18:54:48 +00:00
Colin Walters
5e5cdb4e0d rust: Add ptr error mapping API
Builds on the previous commit.

Closes: #1474
Approved by: jlebon
2018-07-30 18:54:48 +00:00
Colin Walters
ba28adf337 rust: Rework GError mapping API
I was going to add a new API that returns a pointer, and that just
wasn't going to work with the trait AFAICS; we'd need to be generic
across many different types.  Let's use plain functions; it's slightly
less elegant but is clear and works.

Closes: #1474
Approved by: jlebon
2018-07-30 18:54:48 +00:00
Colin Walters
fa29f7acfa compose: Support arch-specific packages in YAML (and in JSON again)
Follow up to: https://github.com/projectatomic/rpm-ostree/pull/1459

We now honor arch-specific packages in YAML, and reject unknown
architectures.  I looked a little bit at how to avoid having hardcoded
arch lists, but it doesn't seem worth it right now.

Closes: #1468
Approved by: jlebon
2018-07-24 22:05:06 +00:00
Colin Walters
6ac6f3d086 treefile.rs: Deny unknown fields by default
Let's not make the same mistake we did with JSON where typoing a
field means it's silently ignored.  This actually caught a bug
in a YAML usage we had:

```
error: Failed to load YAML treefile: unknown field `install_langs`, expected one of ... `install-langs` ...
```

Yes, this is a compatibility break with the feature we just announced
but...I seriously doubt anyone (that isn't known to me) has converted
yet, and if they are excited enough to start using a two-week-old feature
they can adjust.

Closes: #1459
Approved by: cgwalters
2018-07-21 14:43:48 +00:00
Colin Walters
2e237513b0 rust/treefile: Avoid unnecessary dup() of fd
We weren't checking for an error from `dup()`, and further it shouldn't
be necessary.  This is the best I could come up with, though it's
obviously not pretty.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
d7b4046599 rust: Add a make rustfmt, run it
It is actually really nice that there's One Canonical Style, even
if I sometimes don't like some details of what rustfmt does.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
e731f59007 rust: Split out treefile into its own module
Further prep for adding more code here.  The `lib.rs` then is the
collection of glue functions; perhaps in some ideal future it could
be generated even.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
2894d61fbf build-sys: Make Rust library naming/entrypoint more generic
Prep for adding more functionality beyond treefile bits.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
bac6d634bb build-sys: Move Rust include header under rust/ directory
It makes more sense to have the include live next to the associated
code, just like we do with C, even though the `cargo build` doesn't
touch it.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
d7342731ab Add a treefile option to stop creating /etc/machine-id
We actually want systemd's `ConditionFirstBoot` to fire.  The
primary rationale here is that we're adopting Ignition for Fedora CoreOS,
and having `ConditionFirstBoot=` function will help a lot, as the idea
is it only runs once.

However, I discovered that this breaks the `units` directive for example,
as systemd blows away all the unit state in `/etc`.  The correct thing
to do from the start is to use presets.  We could add an implementation of
`units` which works with this on and instead writes a preset file but...eh.
My plan is to at some point introduce an "epoch" and flip various defaults,
this one, `tmp-is-dir`, the passwd file handling, etc.

See: https://github.com/dustymabe/bootengine/pull/11

Closes: #1425
Approved by: jlebon
2018-06-25 16:19:00 +00:00
Colin Walters
a4a8c7cc96 rust/treefile: Fix mutate-os-release typo
It's tempting to enable this by default in YAML...one step at a
time though.
2018-06-21 11:53:06 -04:00
Jonathan Lebon
293baa222d treefile.rs: Add missing #[serde(rename)] attributes
I was surprised when I fired up GDB and found out that my
`etc-group-members` modifications weren't being obeyed. It turns out
serde was looking for the wrong field name. I did a quick sanity check
and found a few others.

Closes: #1417
Approved by: cgwalters
2018-06-15 18:48:34 +00:00
Colin Walters
85a39dd923 rust/treefile: Buffer writes
Not doing buffering is a perf footgun.  Thanks to @lucab for
the PR on the original code that was copied here.

Equivalent to https://github.com/cgwalters/coreos-assembler/pull/3

Closes: #1405
Approved by: jlebon
2018-06-12 21:25:07 +00:00
Colin Walters
5fec2b1af3 build-sys: Add some gitignore for Rust bits
Just keeping status tidy.

Closes: #1397
Approved by: jlebon
2018-06-11 13:08:41 +00:00