Commit Graph

2615 Commits

Author SHA1 Message Date
Colin Walters
89486eace1 core: Typo in indicatif commit: metdatametadata
A typo.

Closes: #1673
Approved by: arithx
2018-11-13 20:08:59 +00:00
Colin Walters
1dee43319c Use indicatif for progress
This turned out to be messier than I thought, because of two primary
factors; the biggest mess here of course is the indirection
through the DBus API.

The other problem is that previously we passed the string to render
each time, and with current indicatif that'd trigger a rerender.
Since (usually) don't change the "prefix string", rework the API.

Change the "percent/n_items" bits to use autocleanups as well, and
to take the prefix string as an initial argument.

Since the state expands to multiple components, also change the
API to use the `0-initialized` pattern rather than trying to
return an aggregate.

We also gain a "sub message" which we use to display e.g.
package names as we're doing checkouts.  Note this ends up
at the end, since otherwise everything else jumps around.

Closes: #1661
Approved by: rfairley
2018-11-09 13:59:43 +00:00
Colin Walters
0f5fc1afca daemon/transaction: Clear out libostree progress consistently
Prep for indicatif, the new progress implementation, which is now
more strict about overwriting tasks.  The `OstreeAsyncProgress`
object lingered on and could own tasks on the mainloop.  Narrow
the scope and avoid having one that crosses multiple pull requests.

Closes: #1661
Approved by: rfairley
2018-11-09 13:59:43 +00:00
Marcel-Lambacher
2c4231b376 app/status: Output corresponding status message if GPG key is missing
The previous behaviour was to simply return "Invalid signature"
if the corresponding GPG public key wasn't found.
This status message wasn't clear enough that the key is missing.
If the GPG public key is now missing, a corresponding status message will be issued.

Closes: #1650
Approved by: rfairley
2018-11-07 14:56:16 +00:00
Jonathan Lebon
c59a136ba7 compose: Initialize out_changed var before --print-only
Otherwise we can get fun undefined behaviour like the caller thinking
there was a change when we didn't even install anything.

Closes: #1669
Approved by: cgwalters
2018-11-06 19:04:47 +00:00
Jonathan Lebon
dd2134e54f compose: Drop 'ex' prefix for --unified-core in msg
That was the last bit in which we referred to the old unified core
option. Also change it in the source itself for completeness.

Closes: #1668
Approved by: cgwalters
2018-11-06 14:21:35 +00:00
Jonathan Lebon
c884a72033 compose: Allow --print-only without bwrap support
We don't need to run the bwrap self test if we just want to print the
manifest. I played with putting the self test in `impl_install`, though
e.g. `postprocess` also needs this so it wasn't quite right.

Closes: #1666
Approved by: cgwalters
2018-11-05 22:12:06 +00:00
Jonathan Lebon
dfb7bd8487 compose: Support all target repos in unified mode
Previously, we were limiting the target repo in unified mode to be a
bare-user repo located on the same filesystem (see message of previous
commit). This patch lifts this restriction by making a distinction
between the *build repo* and the *final* target repo.

To do this, we create a bare-user repo located near the pkgcache to
take advantage of hardlinks and devino caching at commit time. And only
after committing do we essentially `pull-local` into the final target
repo. This of course allows us to avoid potentially pulling across the
two filesystems file objects that are already present in the target
repo.

This will be used by coreos-assembler:
https://github.com/coreos/coreos-assembler/pull/190

Closes: #1490

Closes: #1657
Approved by: cgwalters
2018-11-05 21:29:31 +00:00
Jonathan Lebon
21886fc936 compose: Always put workdir on same filesystem as pkgcache
This ensures that we always get hardlinks when checking out of the
pkgcache. This works right now because we indirectly require the target
bare-user repo and the pkgcache to be on the same filesystem by setting
`no_copy_fallback` in the core (I say "indirectly" because that setting
only enforces the workdir to be on the same filesystem as the pkgcache
repo, but since the workdir is currently placed inside the bare-user
repo...).

However, I'd like to change the requirement of a bare-user repo so that
one can commit into a repo on a different file system or a repo of a
different type (e.g. archive repo). This is prep for that.

Closes: #1657
Approved by: cgwalters
2018-11-05 21:29:31 +00:00
Colin Walters
5d9e25653c libvm: Don't sync rust target/
Since it gets enormous, mine is 1.5GB right now.

Closes: #1665
Approved by: jlebon
2018-11-05 19:59:58 +00:00
Colin Walters
bd4d0d8e0a build-sys: With external cbindgen, drop Rust lib dependency
Perhaps an unexpected side benefit of slow compilation processes
is that one has an opportunity to reflect and ponder.

I realized during exactly such a moment that since we moved
`cbindgen` out of our library build, there's no need to wait
for the library to be built before we can start building the C
code.

This is a notable local quality-of-life development improvement.

Closes: #1665
Approved by: jlebon
2018-11-05 19:59:58 +00:00
Jonathan Lebon
55696824df compose: Use previous sepolicy if available
Rather than defaulting to the host system's SELinux policy, we can be
much more efficient here if we instead use the policy of the last commit
if available. Likely, the pkgcache is currently labeled with that
policy, which means we skip the relabeling phase before checkout. But
also, if the policy didn't change at all in the new rootfs, we also skip
the second relabeling phase after assembly.

Closes: #1659
Approved by: cgwalters
2018-11-05 19:20:40 +00:00
Colin Walters
ef587b5c5b build-sys: Turn Rust LTO off by default, add --enable-lto flag
For us, this is primarily right now a size issue.  See:
https://internals.rust-lang.org/t/rust-staticlibs-and-optimizing-for-size/5746

For more information, there are these two issues:
https://github.com/rust-lang/cargo/issues/4349
https://bugzilla.mozilla.org/show_bug.cgi?id=1386371

The basic issue here is that a build with LTO off (and a trivial
change to add a `println!` takes 14s here, and with it on takes 38s.
However, with LTO off the stripped size of `librpmostree_rust.a` is
`6M`, with LTO on it's `1.1M`.

I named this `--enable-lto` as I'd like to investigate doing this
for the C code too.

Closes: #1664
Approved by: jlebon
2018-11-05 18:41:09 +00:00
Jonathan Lebon
0abfcecd38 compose: Print devino cache hits on commit
Requires: https://github.com/ostreedev/ostree/pull/1772

Closes: #1660
Approved by: cgwalters
2018-11-05 17:21:10 +00:00
Colin Walters
50b255a8a9 Move varsubst code into Rust, use it in treefile parsing
External tools often want to parse the ref; for example coreos-assembler
currently does so.  Let's ensure `${basearch}` is expanded with
`--print-only` so they can parse that JSON to get the expanded version
reliably.

Implementation note: this is the first Rust code which exposes a
"GLib-like" C API, notably with GHashTable, so we're making more use
of the glib-rs bindings.

Closes: #1653

Closes: #1655
Approved by: jlebon
2018-11-02 20:36:28 +00:00
Jonathan Lebon
74db30872b compose: Check for error when opening policy
Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
c3be38af72 postprocess: Minor glnx_console interaction fixes
Make sure we always pass an integer and not a float to
`glnx_console_progress_text_percent`. And also make sure we clamp the
value we pass to 100. Not sure how I hit this, somehow the byte counting
we do upfront before committing is *less* than the addition of all the
`g_file_info_get_size()`s?

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
a2d67f658a app/tree: Check upfront if repo is writable
Didn't hit this but this will be useful for a following commit which
delays committing to the final repo until the end.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
82ceef78ce passwd-util: Drop unnecessary OstreeRepo arg
We don't actually need the repo in `rpmostree_passwd_compose_prep()`
since we're already passed the OstreeRepoFile root of the previous
commit as a separate arg.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
8adeb970e3 postprocess: Rename xattr callback
Just a minor readability tweak. Wasn't clear what `read_xattrs_cb`
meant. Also inline an indexing var into the loop.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
5c4e88da9c core: use DEVINO_CANONICAL regardless of repo type
If the policy didn't change, then it's safe to use the devino cache
regardless of the repo type. This dates from #1123; I think the
rationale there was so that it only affects the pkglayering path, where
the system repo is bare. But really, it should be fine to generalize
this since we don't have any xattr callbacks.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
38c2ab3397 app/compose: Minor comment tweak
This is slight bikeshed, though this comment took me a second to parse
because it says "not do it" right before we do it. Let's just move it up
to right outside where the condition is actually tested.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
868bdf167a app/compose: Use g_printerr for note
For consistency with the other note we print lower down.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
37932cf19a app/compose: Check conflicting opts earlier
No reason to do this sanity check so late. Let's bring it up to near the
top of `impl_install_tree()` so we avoid useless work if possible.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
46edef0017 app/compose: Drop helper cachedir_dfd() function
We don't actually need this helper here since we always initialize
`self->cachedir_dfd` nowadays. I took a closer look at this initially
because it was confusing ctags due to the function being named the same
as the struct member it returns.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
6d39cfa78f compose: Don't require SELinux policy in legacy path
In #1630, we lowered SELinux policy loading into the core. However, this
also enabled SELinux policy loading from the host system even in the
legacy (non-unified) compose path. This meant that compose systems now
needed to have the policy installed even though we didn't need it at
all. This caused regressions in pungi:

https://pagure.io/dusty/failed-composes/issue/956

Just make the binding of the "selinux" member conditional on whether or
not we're in unified mode (which is really when we even care about
having it loaded from the start for pkgcache purposes).

Closes: #1656
Approved by: cgwalters
2018-11-02 14:49:32 +00:00
Jonathan Lebon
79d4e52d99 app: Wrap Version in --version in quotes
Wrap the `Version` key in the YAML-compatible output of
`rpm-ostree --version` with quotes so that it's parsed as a string. The
issues with the previous approach in a nutshell:

```
In [5]: yaml.load("asdf: 2018.10")
Out[5]: {'asdf': 2018.1}
```

It's treating the version number as a floating-point. Now, this is
technically a backwards incompatible change, but given that the previous
approach is inherently broken for our needs, I don't see a way around
breaking it now.

See also: 759b099861

Closes: #1654
Approved by: cgwalters
2018-11-01 23:32:19 +00:00
Jonathan Lebon
ff962491a3 man: Fix references to automatic timer and service
There is no `rpm-ostreed-automatic` man page, only
`rpm-ostreed-automatic.service` and `rpm-ostreed-automatic.timer`. Fix
references from the conf man page to those pages.

Closes: #1651

Closes: #1652
Approved by: miabbott
2018-11-01 15:26:06 +00:00
Colin Walters
88ffdc0ae2 Add compose rojig command
This currently requires a `--i-know-this-is-experimental` flag;
I know it'd be a bit more consistent to have it under `ex`, but
what feels weird about that is *most* of the `ex` commands people
use are client side.  This is where we want it to ultimately end
up.

We've landed a lot of prep patches, but I know there's still
a notable amount of code duplication with `compose tree`.  What's
left is about ~700 lines but it's mostly not hard/complex code
anymore.

In the future, I'd like to extract more of the compose code
to a `rust/src/compose.rs` or so, but I think this is sustainable
fow now.

My high level goal is to get this into coreos-assembler and stand
up a Silverblue build that uses it.

Closes: #1512
Approved by: jlebon
2018-10-31 17:57:45 +00:00
Colin Walters
c044f59726 rust: Remove pub from modules
Conceptually we're only exporting the FFI code right now.

Closes: #1649
Approved by: jlebon
2018-10-30 16:16:22 +00:00
Colin Walters
5b8b37658f compose: Make --print-only happen earlier and be quiet
Don't print the version or other warnings; ensure we output clean JSON.

Prep for fixing https://github.com/coreos/fedora-coreos-config/pull/24
(This helps, but we still need to expand `${basearch}`)

Closes: #1648
Approved by: jlebon
2018-10-30 15:04:21 +00:00
Colin Walters
87441d9ec2 rust: Move FFI bits into ffi submodule for each file
The `lib.rs` file was starting to accumulate, and I want to do this
before adding more.  I just made this up, but an `ffi` submodule
in each file seems to work well.  It isolates the FFI consumption
there still (so e.g. if we want pure Rust unit tests, we don't
need to deal with FFI).

Closes: #1646
Approved by: jlebon
2018-10-30 14:24:42 +00:00
Colin Walters
99776be162 core: Disable librpm SELinux plugin when writing rpmdb
OSTree was designed from the very beginning of its existence
to support SELinux well instead of being something wedged on.
rpm-ostree builds on that foundation.  We don't want to have anything
to do with librpm's SELinux code.  And with unified core,
we usually don't, but that `rpm-plugin-selinux` code does get loaded.

Disable it here.  The main reason I'm submitting this patch
is to help an effort in coreos-assembler to use a "supermin" virtual
machine: https://github.com/coreos/coreos-assembler/pull/124

Closes: #1647
Approved by: jlebon
2018-10-29 18:42:10 +00:00
Colin Walters
b5d34b7268 compose: Factor out helper for writing composejson
Prep for sharing this code with rojig.

Closes: #1636
Approved by: jlebon
2018-10-29 01:35:20 +00:00
Jonathan Lebon
f3b8ef043e libpriv: Sanity check that paths are OSTree compatible
I originally was going to add the check here and error out to the user,
but `add-files` is handled all the way near the end of the compose,
which meant that users would have to wait through it all before getting
an error. Though the check is enforced at parsing time now, I think it's
still useful at postprocess time as well as a sanity check.

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Jonathan Lebon
940fc1364a compose: Check that add-files are compatible after parsing
While serde gives us type checking, it of course doesn't understand
semantics beyond that. One example is checking the compatibility of
`add-files` entries with the OSTree model. This is something we can do
upfront early on to avoid surprises for users.

Also tweak the docs to reflect this new check.

Related: #1642

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Jonathan Lebon
f8f47480f1 docs/treefile: Add info about paths for add-files
Add a note that only certain paths actually make sense in the
`add-files` directive and link to the OSTree docs. Let's also the
examples given to more realistic and correct ones.

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Colin Walters
202192d61e rust: Create generic ref_from_raw_ptr function
I need to do some hunting for a "ffi helper" crate.  I kind of
understand why these things aren't in the std library.

Anyways this is easy for now, prep for adding more types.

Closes: #1641
Approved by: jlebon
2018-10-26 20:04:08 +00:00
Colin Walters
92d207de5d rust: Merge glibutils with utility FFI helpers
Create `ffiutil.rs` from the merger.  This way all of our FFI
helpers are in one place.  Prep for further refactoring.

Closes: #1641
Approved by: jlebon
2018-10-26 20:04:08 +00:00
Colin Walters
245d238563 Use autocleanup for task output
I was playing with https://crates.io/crates/indicatif
and this is prep for using it.

By using autocleanups here we ensure that the output state
ends even if we encounter an error.  We previously had a few
explicit `failed` prints in cases where failure was more common
such as dependency resolution, but that was far from consistent.

Now we will always ensure a newline is printed even if an error
occurs, and we won't have to worry about missing calls to
`_task_done()` in the success case.

The downside of this is that we lose the `done` bit in the success
path...I thought about maybe taking a pointer to the error and
printing `done` or `failed` depending on whether it's non-`NULL`
but eh...the `done` is implied by us going on to the next line.
Also that issue will be fixed with the indicatif work.

Closes: #1640
Approved by: jlebon
2018-10-26 19:20:25 +00:00
Jonathan Lebon
a87486e18e rust: Drop crates-io patch and use 0.4.0
The latest release of the crate includes the features and fixes we need.

Closes: #1639
Approved by: cgwalters
2018-10-25 19:01:39 +00:00
Jonathan Lebon
537a040b00 Release 2018.9
Closes: #1633
Approved by: cgwalters
2018-10-25 18:12:37 +00:00
Christian Glombek
20acf6a2f5 compose: Print version number
Users doing builds that are using newer features
want to be sure they have the right version.

Closes: #1556

Closes: #1637
Approved by: jlebon
2018-10-25 17:07:10 +00:00
Colin Walters
fe5adaf7ab build-sys: Fix make rustfmt, and run it
When we added the bindgen config this broke as it's not a `.rs`.
Let's keep the source formatted canonically.

Closes: #1635
Approved by: jlebon
2018-10-23 19:05:06 +00:00
Michael Nguyen
50ef17ed95 app/status: line wrap RemovedBasePackages
Wrap RemovedBasePackages like other package lists in the status
command so it looks prettier.

fixup! app/status: line wrap RemovedBasePackages

fixup! app/status: line wrap RemovedBasePackages

fixup! app/status: line wrap RemovedBasePackages

Resolves: #1613

Closes: #1632
Approved by: jlebon
2018-10-23 14:28:19 +00:00
Colin Walters
fca01e70b5 Lower initial SELinux policy load from compose to core
Add a `selinux` verb to treespec, and bind it from treefile.  If
set, use it in the core to load an initial policy before import,
if we didn't already set a policy.

In practice right now this is only used from the compose path
since the SysrootUpgrader uses the policy from the merge deployment.

Unset the policy if rojig mode is enabled.

Now, non-SELinux use cases are required to set `selinux: false`
in the treespec. For `ex container` I just set it in our example
specs.  Probably that should forcibly disable it in the
treespec but eh, it's experimental.

The other case I can think of is client-side layering; before
we would create a policy using the target root, but it
wasn't a *hard* requirement, i.e. we didn't error out if
`policy_get_name() == NULL`.  Let's preserve that semantic by
hooking off of whether `_new_system()` was used.

Prep for sharing code with `compose rojig`.

Closes: #1630
Approved by: jlebon
2018-10-23 13:40:46 +00:00
Colin Walters
95c1ed3695 tests/compose: Use workdir in tmpdir
This way when debugging if I want to preserve the state so I can
go there and rerun command under gdb for exmaple, all I need
to do is set `env TEST_SKIP_CLEANUP=1` rather than also needing
to find a separate dir and also set `env RPMOSTREE_PRESERVE_TMPDIR=1`.

Closes: #1630
Approved by: jlebon
2018-10-23 13:40:46 +00:00
Colin Walters
f1fa436c8a rust/treefile: Use macros to reduce redundancy in treefile merging
On the plus side, when submitting a patch to Github, no one knows
how long it took you to figure out...

Anyways so this reduces redundancy.  I double-checked the list.
I was inspired to pick this back up after seeing a Rust code
snippet somewhere noting that macros defined inside a function
can capture variables, which simplifies this even more.

Closes: #1631
Approved by: jlebon
2018-10-22 18:53:16 +00:00
Jonathan Lebon
6583a557ae app/livefs: Require --i-like-danger switch
We've had multiple reports by now of folks using plain `ex livefs` and
getting their bootloader wrecked:

https://github.com/projectatomic/rpm-ostree/issues/1495
https://github.com/projectatomic/rpm-ostree/issues/1504
https://github.com/ostreedev/ostree/issues/1459

Let's require a scary switch for now to emphasize this.

Closes: #1622
Approved by: cgwalters
2018-10-17 20:10:11 +00:00
Colin Walters
7c41188476 Switch to ISO8601 timestamps in more places
I was doing this for the "generated" bit on the compose side,
but let's be consistent elsewhere.

Closes: #1624
Approved by: jlebon
2018-10-17 19:32:12 +00:00