Commit Graph

2628 Commits

Author SHA1 Message Date
Colin Walters
e0a976386a rust: Use failure crate for errors
In a lot of places we're abusing `io::Error(io::ErrorKind::InvalidInput)`
which is both verbose and inaccurate really.  Maybe in some
places we should be defining custom errors, but eh.

I like the `failure` crate.  Use it in just `utils.rs` for now.
Tweak our error handling FFI wrappers to accept `Display` since
all we do is convert the error to a string.

Closes: #1675
Approved by: lucab
2018-11-21 22:03:53 +00:00
Colin Walters
387da3a420 ci: Verify rustfmt
Split out the command into a separate `Makefile` that doesn't
use Automake so we can invoke it early.

Closes: #1674
Approved by: dustymabe
2018-11-21 21:16:03 +00:00
Colin Walters
6b5e475607 rust: Rerun rustfmt
Going to add a CI step next.

Closes: #1674
Approved by: dustymabe
2018-11-21 21:16:03 +00:00
Jonathan Lebon
17359bc0cc core: Tweak "Importing" messages
To make it more obvious what the difference between "Importing metadata"
and "Importing" is, add "rpm-md" to the first and "packages" to the
second.

Closes: #1681
Approved by: cgwalters
2018-11-21 13:54:35 +00:00
Colin Walters
da8be3f7b8 scripts: Add a hack to handle RHEL7's glibc-common.post
While we had an override for both the Fedora `glibc-all-langpacks.posttrans`
version and the RHEL7-era `glibc-common.post`, there were two
problems.

First, the RHEL7 version's lua calls `rpm.expand()` internally
rather than flagging the script itself for expansion.

Second, we also need to disable rofiles-fuse for it.

Closes: #1678
Approved by: jlebon
2018-11-21 08:14:24 +00:00
Jonathan Lebon
c9cbad94a3 daemon/utils: Finish OstreeAsyncProgress after pull
Otherwise, the object might still own an idle source on the main
context, which will cause issues if another pull operation happens
again. This wasn't causing issues before because in places where we did
do multiple pull operations, we would reuse the same
`OstreeAsyncProgress` object, and the second pull operation *did* do
`ostree_async_progress_finish()`. But that's no longer the case now with
66761916.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Jonathan Lebon
c7ce2564dc daemon/transaction: Emit progress end signal
This is not strictly necessary since the progress is considered ended on
the client side when the transaction is finished, but let's be nice.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Jonathan Lebon
71f0a30d83 ci: Go back to centos/7/atomic/smoketested
We should prefer the smoketested ref in general since it's more stable.
There is an updated glib2 there now. I also tested that layering-relabel
works (at least locally).

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
ce339dff9b vmcheck/multitest: Print when we're waiting
To help debug.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
11ae25792c build-sys: Add bindgen bits to .gitignore
Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
dcc78cc2f1 build-sys: Fix uid check when we don't have /usr/bin/cbindgen
Make this match the other case.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
c08e9620d6 vmcheck/multitest: Time execution of each test
For some reason c7 is consistently timing out, let's gather
data on how long our tests take to execute.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
33178cb621 vmcheck: Sync over libsolv and zchunk-libs too
In general our current CI/test system is susceptible to drift
between the container and AH.  The direction we should be
going is to have coreos-assembler solve this problem with
a SDK, but for now, let's ensure that the container's libsolv
makes it to the host, same thing we do for libostree.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
89486eace1 core: Typo in indicatif commit: metdatametadata
A typo.

Closes: #1673
Approved by: arithx
2018-11-13 20:08:59 +00:00
Colin Walters
1dee43319c Use indicatif for progress
This turned out to be messier than I thought, because of two primary
factors; the biggest mess here of course is the indirection
through the DBus API.

The other problem is that previously we passed the string to render
each time, and with current indicatif that'd trigger a rerender.
Since (usually) don't change the "prefix string", rework the API.

Change the "percent/n_items" bits to use autocleanups as well, and
to take the prefix string as an initial argument.

Since the state expands to multiple components, also change the
API to use the `0-initialized` pattern rather than trying to
return an aggregate.

We also gain a "sub message" which we use to display e.g.
package names as we're doing checkouts.  Note this ends up
at the end, since otherwise everything else jumps around.

Closes: #1661
Approved by: rfairley
2018-11-09 13:59:43 +00:00
Colin Walters
0f5fc1afca daemon/transaction: Clear out libostree progress consistently
Prep for indicatif, the new progress implementation, which is now
more strict about overwriting tasks.  The `OstreeAsyncProgress`
object lingered on and could own tasks on the mainloop.  Narrow
the scope and avoid having one that crosses multiple pull requests.

Closes: #1661
Approved by: rfairley
2018-11-09 13:59:43 +00:00
Marcel-Lambacher
2c4231b376 app/status: Output corresponding status message if GPG key is missing
The previous behaviour was to simply return "Invalid signature"
if the corresponding GPG public key wasn't found.
This status message wasn't clear enough that the key is missing.
If the GPG public key is now missing, a corresponding status message will be issued.

Closes: #1650
Approved by: rfairley
2018-11-07 14:56:16 +00:00
Jonathan Lebon
c59a136ba7 compose: Initialize out_changed var before --print-only
Otherwise we can get fun undefined behaviour like the caller thinking
there was a change when we didn't even install anything.

Closes: #1669
Approved by: cgwalters
2018-11-06 19:04:47 +00:00
Jonathan Lebon
dd2134e54f compose: Drop 'ex' prefix for --unified-core in msg
That was the last bit in which we referred to the old unified core
option. Also change it in the source itself for completeness.

Closes: #1668
Approved by: cgwalters
2018-11-06 14:21:35 +00:00
Jonathan Lebon
c884a72033 compose: Allow --print-only without bwrap support
We don't need to run the bwrap self test if we just want to print the
manifest. I played with putting the self test in `impl_install`, though
e.g. `postprocess` also needs this so it wasn't quite right.

Closes: #1666
Approved by: cgwalters
2018-11-05 22:12:06 +00:00
Jonathan Lebon
dfb7bd8487 compose: Support all target repos in unified mode
Previously, we were limiting the target repo in unified mode to be a
bare-user repo located on the same filesystem (see message of previous
commit). This patch lifts this restriction by making a distinction
between the *build repo* and the *final* target repo.

To do this, we create a bare-user repo located near the pkgcache to
take advantage of hardlinks and devino caching at commit time. And only
after committing do we essentially `pull-local` into the final target
repo. This of course allows us to avoid potentially pulling across the
two filesystems file objects that are already present in the target
repo.

This will be used by coreos-assembler:
https://github.com/coreos/coreos-assembler/pull/190

Closes: #1490

Closes: #1657
Approved by: cgwalters
2018-11-05 21:29:31 +00:00
Jonathan Lebon
21886fc936 compose: Always put workdir on same filesystem as pkgcache
This ensures that we always get hardlinks when checking out of the
pkgcache. This works right now because we indirectly require the target
bare-user repo and the pkgcache to be on the same filesystem by setting
`no_copy_fallback` in the core (I say "indirectly" because that setting
only enforces the workdir to be on the same filesystem as the pkgcache
repo, but since the workdir is currently placed inside the bare-user
repo...).

However, I'd like to change the requirement of a bare-user repo so that
one can commit into a repo on a different file system or a repo of a
different type (e.g. archive repo). This is prep for that.

Closes: #1657
Approved by: cgwalters
2018-11-05 21:29:31 +00:00
Colin Walters
5d9e25653c libvm: Don't sync rust target/
Since it gets enormous, mine is 1.5GB right now.

Closes: #1665
Approved by: jlebon
2018-11-05 19:59:58 +00:00
Colin Walters
bd4d0d8e0a build-sys: With external cbindgen, drop Rust lib dependency
Perhaps an unexpected side benefit of slow compilation processes
is that one has an opportunity to reflect and ponder.

I realized during exactly such a moment that since we moved
`cbindgen` out of our library build, there's no need to wait
for the library to be built before we can start building the C
code.

This is a notable local quality-of-life development improvement.

Closes: #1665
Approved by: jlebon
2018-11-05 19:59:58 +00:00
Jonathan Lebon
55696824df compose: Use previous sepolicy if available
Rather than defaulting to the host system's SELinux policy, we can be
much more efficient here if we instead use the policy of the last commit
if available. Likely, the pkgcache is currently labeled with that
policy, which means we skip the relabeling phase before checkout. But
also, if the policy didn't change at all in the new rootfs, we also skip
the second relabeling phase after assembly.

Closes: #1659
Approved by: cgwalters
2018-11-05 19:20:40 +00:00
Colin Walters
ef587b5c5b build-sys: Turn Rust LTO off by default, add --enable-lto flag
For us, this is primarily right now a size issue.  See:
https://internals.rust-lang.org/t/rust-staticlibs-and-optimizing-for-size/5746

For more information, there are these two issues:
https://github.com/rust-lang/cargo/issues/4349
https://bugzilla.mozilla.org/show_bug.cgi?id=1386371

The basic issue here is that a build with LTO off (and a trivial
change to add a `println!` takes 14s here, and with it on takes 38s.
However, with LTO off the stripped size of `librpmostree_rust.a` is
`6M`, with LTO on it's `1.1M`.

I named this `--enable-lto` as I'd like to investigate doing this
for the C code too.

Closes: #1664
Approved by: jlebon
2018-11-05 18:41:09 +00:00
Jonathan Lebon
0abfcecd38 compose: Print devino cache hits on commit
Requires: https://github.com/ostreedev/ostree/pull/1772

Closes: #1660
Approved by: cgwalters
2018-11-05 17:21:10 +00:00
Colin Walters
50b255a8a9 Move varsubst code into Rust, use it in treefile parsing
External tools often want to parse the ref; for example coreos-assembler
currently does so.  Let's ensure `${basearch}` is expanded with
`--print-only` so they can parse that JSON to get the expanded version
reliably.

Implementation note: this is the first Rust code which exposes a
"GLib-like" C API, notably with GHashTable, so we're making more use
of the glib-rs bindings.

Closes: #1653

Closes: #1655
Approved by: jlebon
2018-11-02 20:36:28 +00:00
Jonathan Lebon
74db30872b compose: Check for error when opening policy
Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
c3be38af72 postprocess: Minor glnx_console interaction fixes
Make sure we always pass an integer and not a float to
`glnx_console_progress_text_percent`. And also make sure we clamp the
value we pass to 100. Not sure how I hit this, somehow the byte counting
we do upfront before committing is *less* than the addition of all the
`g_file_info_get_size()`s?

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
a2d67f658a app/tree: Check upfront if repo is writable
Didn't hit this but this will be useful for a following commit which
delays committing to the final repo until the end.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
82ceef78ce passwd-util: Drop unnecessary OstreeRepo arg
We don't actually need the repo in `rpmostree_passwd_compose_prep()`
since we're already passed the OstreeRepoFile root of the previous
commit as a separate arg.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
8adeb970e3 postprocess: Rename xattr callback
Just a minor readability tweak. Wasn't clear what `read_xattrs_cb`
meant. Also inline an indexing var into the loop.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
5c4e88da9c core: use DEVINO_CANONICAL regardless of repo type
If the policy didn't change, then it's safe to use the devino cache
regardless of the repo type. This dates from #1123; I think the
rationale there was so that it only affects the pkglayering path, where
the system repo is bare. But really, it should be fine to generalize
this since we don't have any xattr callbacks.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
38c2ab3397 app/compose: Minor comment tweak
This is slight bikeshed, though this comment took me a second to parse
because it says "not do it" right before we do it. Let's just move it up
to right outside where the condition is actually tested.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
868bdf167a app/compose: Use g_printerr for note
For consistency with the other note we print lower down.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
37932cf19a app/compose: Check conflicting opts earlier
No reason to do this sanity check so late. Let's bring it up to near the
top of `impl_install_tree()` so we avoid useless work if possible.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
46edef0017 app/compose: Drop helper cachedir_dfd() function
We don't actually need this helper here since we always initialize
`self->cachedir_dfd` nowadays. I took a closer look at this initially
because it was confusing ctags due to the function being named the same
as the struct member it returns.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
6d39cfa78f compose: Don't require SELinux policy in legacy path
In #1630, we lowered SELinux policy loading into the core. However, this
also enabled SELinux policy loading from the host system even in the
legacy (non-unified) compose path. This meant that compose systems now
needed to have the policy installed even though we didn't need it at
all. This caused regressions in pungi:

https://pagure.io/dusty/failed-composes/issue/956

Just make the binding of the "selinux" member conditional on whether or
not we're in unified mode (which is really when we even care about
having it loaded from the start for pkgcache purposes).

Closes: #1656
Approved by: cgwalters
2018-11-02 14:49:32 +00:00
Jonathan Lebon
79d4e52d99 app: Wrap Version in --version in quotes
Wrap the `Version` key in the YAML-compatible output of
`rpm-ostree --version` with quotes so that it's parsed as a string. The
issues with the previous approach in a nutshell:

```
In [5]: yaml.load("asdf: 2018.10")
Out[5]: {'asdf': 2018.1}
```

It's treating the version number as a floating-point. Now, this is
technically a backwards incompatible change, but given that the previous
approach is inherently broken for our needs, I don't see a way around
breaking it now.

See also: 759b099861

Closes: #1654
Approved by: cgwalters
2018-11-01 23:32:19 +00:00
Jonathan Lebon
ff962491a3 man: Fix references to automatic timer and service
There is no `rpm-ostreed-automatic` man page, only
`rpm-ostreed-automatic.service` and `rpm-ostreed-automatic.timer`. Fix
references from the conf man page to those pages.

Closes: #1651

Closes: #1652
Approved by: miabbott
2018-11-01 15:26:06 +00:00
Colin Walters
88ffdc0ae2 Add compose rojig command
This currently requires a `--i-know-this-is-experimental` flag;
I know it'd be a bit more consistent to have it under `ex`, but
what feels weird about that is *most* of the `ex` commands people
use are client side.  This is where we want it to ultimately end
up.

We've landed a lot of prep patches, but I know there's still
a notable amount of code duplication with `compose tree`.  What's
left is about ~700 lines but it's mostly not hard/complex code
anymore.

In the future, I'd like to extract more of the compose code
to a `rust/src/compose.rs` or so, but I think this is sustainable
fow now.

My high level goal is to get this into coreos-assembler and stand
up a Silverblue build that uses it.

Closes: #1512
Approved by: jlebon
2018-10-31 17:57:45 +00:00
Colin Walters
c044f59726 rust: Remove pub from modules
Conceptually we're only exporting the FFI code right now.

Closes: #1649
Approved by: jlebon
2018-10-30 16:16:22 +00:00
Colin Walters
5b8b37658f compose: Make --print-only happen earlier and be quiet
Don't print the version or other warnings; ensure we output clean JSON.

Prep for fixing https://github.com/coreos/fedora-coreos-config/pull/24
(This helps, but we still need to expand `${basearch}`)

Closes: #1648
Approved by: jlebon
2018-10-30 15:04:21 +00:00
Colin Walters
87441d9ec2 rust: Move FFI bits into ffi submodule for each file
The `lib.rs` file was starting to accumulate, and I want to do this
before adding more.  I just made this up, but an `ffi` submodule
in each file seems to work well.  It isolates the FFI consumption
there still (so e.g. if we want pure Rust unit tests, we don't
need to deal with FFI).

Closes: #1646
Approved by: jlebon
2018-10-30 14:24:42 +00:00
Colin Walters
99776be162 core: Disable librpm SELinux plugin when writing rpmdb
OSTree was designed from the very beginning of its existence
to support SELinux well instead of being something wedged on.
rpm-ostree builds on that foundation.  We don't want to have anything
to do with librpm's SELinux code.  And with unified core,
we usually don't, but that `rpm-plugin-selinux` code does get loaded.

Disable it here.  The main reason I'm submitting this patch
is to help an effort in coreos-assembler to use a "supermin" virtual
machine: https://github.com/coreos/coreos-assembler/pull/124

Closes: #1647
Approved by: jlebon
2018-10-29 18:42:10 +00:00
Colin Walters
b5d34b7268 compose: Factor out helper for writing composejson
Prep for sharing this code with rojig.

Closes: #1636
Approved by: jlebon
2018-10-29 01:35:20 +00:00
Jonathan Lebon
f3b8ef043e libpriv: Sanity check that paths are OSTree compatible
I originally was going to add the check here and error out to the user,
but `add-files` is handled all the way near the end of the compose,
which meant that users would have to wait through it all before getting
an error. Though the check is enforced at parsing time now, I think it's
still useful at postprocess time as well as a sanity check.

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Jonathan Lebon
940fc1364a compose: Check that add-files are compatible after parsing
While serde gives us type checking, it of course doesn't understand
semantics beyond that. One example is checking the compatibility of
`add-files` entries with the OSTree model. This is something we can do
upfront early on to avoid surprises for users.

Also tweak the docs to reflect this new check.

Related: #1642

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Jonathan Lebon
f8f47480f1 docs/treefile: Add info about paths for add-files
Add a note that only certain paths actually make sense in the
`add-files` directive and link to the OSTree docs. Let's also the
examples given to more realistic and correct ones.

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00