Commit Graph

2486 Commits

Author SHA1 Message Date
Colin Walters
a2f5b7c1c0 rust/ffi: More "view" changes, for bytestrings
Change our existing "view as [u8] API", and also add one
that does a view as `OsStr`.  The motivation for the latter
is I noticed ithat `OsStr::from_bytes()` *doesn't* copy,
or rather it just copies the pointer value.  Rust's lifetime
inference ensures that the returned lifetime matches the input array.

I think the previous code in `treefile.rs` was confused about this.

Closes: #1688
Approved by: jlebon
2018-11-28 09:14:41 +00:00
Colin Walters
bfe637a877 rust/ffi: Convert str func to new "view" naming convention
More obvious one shouldn't leak the pointer.

Closes: #1688
Approved by: jlebon
2018-11-28 09:14:41 +00:00
Colin Walters
db0319beac rust/ffi: Change dfd handling to unwrap, create naming convention
I was going to add another usage of this function, and I think the
gerror stuff is unnecessary - if we are handed a bad file descriptor
(or a fd pointing to a regular file) that's something where we should
just abort.

While we're here, I'd like to codify expected usage in the function
names here.  If you like this I'll e.g. also change `str_from_nullable`
to `ffi_view_nullable_str`.

Closes: #1685
Approved by: jlebon
2018-11-27 15:42:08 +00:00
Jonathan Lebon
140704c105 daemon/transaction-types: Fix dnf cache override
Minor regression from #1587. There were places that were still doing
`dnf_context_set_cache_age()` manually, but those calls didn't exactly
have the intended effect since the core now handled caching itself.

The actual result was that the metadata was still being updated, but not
during the `dnf_repo_check` pass that the core does, but rather the
`Importing rpm-md` pass it does right after. So then, we were
incorrectly printing `(cached)` even though we'd update it afterwards.

Switch to the new way of doing things.

Closes: #1686
Approved by: cgwalters
2018-11-26 23:39:44 +00:00
Jonathan Lebon
1164b7b047 daemon/transaction-types: Drop unneeded arg
Drop the `force_refresh` boolean parameter since we only ever call it
with `TRUE`. I think this dates from an earlier implementation where we
did call it with differing values.

Small prep for next patch.

Closes: #1686
Approved by: cgwalters
2018-11-26 23:39:44 +00:00
Jonathan Lebon
2cc06ce89a configure.ac: Add "GitHub release" step to release workflow
Let's make use of the GitHub release feature to make it more prominent
on the "Releases" tab, but more importantly so that we can attach
vendored tarballs for downstreams. E.g. this will allow us to have a
correct `Source0` field in the Fedora spec file.

Related: #1683

Closes: #1684
Approved by: rfairley
2018-11-26 22:59:50 +00:00
Jonathan Lebon
ed2fefad85 Move reset out of experimental
This is relatively uncontroversial functionality that has already proved
useful when helping folks debug their stuff. Let's promote it to the
stable interface.

Closes: #1682
Approved by: rfairley
2018-11-22 20:47:50 +00:00
Jonathan Lebon
03a92c0981 app/ex: Nuke ex kargs compatibilty alias
It said so right there; "nuke in next version". No need to carry this
stuff forever. It's the final phase of feature promotion, like when one
officially moves out of their parents'.

Closes: #1682
Approved by: rfairley
2018-11-22 20:47:50 +00:00
Colin Walters
e0a976386a rust: Use failure crate for errors
In a lot of places we're abusing `io::Error(io::ErrorKind::InvalidInput)`
which is both verbose and inaccurate really.  Maybe in some
places we should be defining custom errors, but eh.

I like the `failure` crate.  Use it in just `utils.rs` for now.
Tweak our error handling FFI wrappers to accept `Display` since
all we do is convert the error to a string.

Closes: #1675
Approved by: lucab
2018-11-21 22:03:53 +00:00
Colin Walters
387da3a420 ci: Verify rustfmt
Split out the command into a separate `Makefile` that doesn't
use Automake so we can invoke it early.

Closes: #1674
Approved by: dustymabe
2018-11-21 21:16:03 +00:00
Colin Walters
6b5e475607 rust: Rerun rustfmt
Going to add a CI step next.

Closes: #1674
Approved by: dustymabe
2018-11-21 21:16:03 +00:00
Jonathan Lebon
17359bc0cc core: Tweak "Importing" messages
To make it more obvious what the difference between "Importing metadata"
and "Importing" is, add "rpm-md" to the first and "packages" to the
second.

Closes: #1681
Approved by: cgwalters
2018-11-21 13:54:35 +00:00
Colin Walters
da8be3f7b8 scripts: Add a hack to handle RHEL7's glibc-common.post
While we had an override for both the Fedora `glibc-all-langpacks.posttrans`
version and the RHEL7-era `glibc-common.post`, there were two
problems.

First, the RHEL7 version's lua calls `rpm.expand()` internally
rather than flagging the script itself for expansion.

Second, we also need to disable rofiles-fuse for it.

Closes: #1678
Approved by: jlebon
2018-11-21 08:14:24 +00:00
Jonathan Lebon
c9cbad94a3 daemon/utils: Finish OstreeAsyncProgress after pull
Otherwise, the object might still own an idle source on the main
context, which will cause issues if another pull operation happens
again. This wasn't causing issues before because in places where we did
do multiple pull operations, we would reuse the same
`OstreeAsyncProgress` object, and the second pull operation *did* do
`ostree_async_progress_finish()`. But that's no longer the case now with
66761916.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Jonathan Lebon
c7ce2564dc daemon/transaction: Emit progress end signal
This is not strictly necessary since the progress is considered ended on
the client side when the transaction is finished, but let's be nice.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Jonathan Lebon
71f0a30d83 ci: Go back to centos/7/atomic/smoketested
We should prefer the smoketested ref in general since it's more stable.
There is an updated glib2 there now. I also tested that layering-relabel
works (at least locally).

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
ce339dff9b vmcheck/multitest: Print when we're waiting
To help debug.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
11ae25792c build-sys: Add bindgen bits to .gitignore
Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
dcc78cc2f1 build-sys: Fix uid check when we don't have /usr/bin/cbindgen
Make this match the other case.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
c08e9620d6 vmcheck/multitest: Time execution of each test
For some reason c7 is consistently timing out, let's gather
data on how long our tests take to execute.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
33178cb621 vmcheck: Sync over libsolv and zchunk-libs too
In general our current CI/test system is susceptible to drift
between the container and AH.  The direction we should be
going is to have coreos-assembler solve this problem with
a SDK, but for now, let's ensure that the container's libsolv
makes it to the host, same thing we do for libostree.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
89486eace1 core: Typo in indicatif commit: metdatametadata
A typo.

Closes: #1673
Approved by: arithx
2018-11-13 20:08:59 +00:00
Colin Walters
1dee43319c Use indicatif for progress
This turned out to be messier than I thought, because of two primary
factors; the biggest mess here of course is the indirection
through the DBus API.

The other problem is that previously we passed the string to render
each time, and with current indicatif that'd trigger a rerender.
Since (usually) don't change the "prefix string", rework the API.

Change the "percent/n_items" bits to use autocleanups as well, and
to take the prefix string as an initial argument.

Since the state expands to multiple components, also change the
API to use the `0-initialized` pattern rather than trying to
return an aggregate.

We also gain a "sub message" which we use to display e.g.
package names as we're doing checkouts.  Note this ends up
at the end, since otherwise everything else jumps around.

Closes: #1661
Approved by: rfairley
2018-11-09 13:59:43 +00:00
Colin Walters
0f5fc1afca daemon/transaction: Clear out libostree progress consistently
Prep for indicatif, the new progress implementation, which is now
more strict about overwriting tasks.  The `OstreeAsyncProgress`
object lingered on and could own tasks on the mainloop.  Narrow
the scope and avoid having one that crosses multiple pull requests.

Closes: #1661
Approved by: rfairley
2018-11-09 13:59:43 +00:00
Marcel-Lambacher
2c4231b376 app/status: Output corresponding status message if GPG key is missing
The previous behaviour was to simply return "Invalid signature"
if the corresponding GPG public key wasn't found.
This status message wasn't clear enough that the key is missing.
If the GPG public key is now missing, a corresponding status message will be issued.

Closes: #1650
Approved by: rfairley
2018-11-07 14:56:16 +00:00
Jonathan Lebon
c59a136ba7 compose: Initialize out_changed var before --print-only
Otherwise we can get fun undefined behaviour like the caller thinking
there was a change when we didn't even install anything.

Closes: #1669
Approved by: cgwalters
2018-11-06 19:04:47 +00:00
Jonathan Lebon
dd2134e54f compose: Drop 'ex' prefix for --unified-core in msg
That was the last bit in which we referred to the old unified core
option. Also change it in the source itself for completeness.

Closes: #1668
Approved by: cgwalters
2018-11-06 14:21:35 +00:00
Jonathan Lebon
c884a72033 compose: Allow --print-only without bwrap support
We don't need to run the bwrap self test if we just want to print the
manifest. I played with putting the self test in `impl_install`, though
e.g. `postprocess` also needs this so it wasn't quite right.

Closes: #1666
Approved by: cgwalters
2018-11-05 22:12:06 +00:00
Jonathan Lebon
dfb7bd8487 compose: Support all target repos in unified mode
Previously, we were limiting the target repo in unified mode to be a
bare-user repo located on the same filesystem (see message of previous
commit). This patch lifts this restriction by making a distinction
between the *build repo* and the *final* target repo.

To do this, we create a bare-user repo located near the pkgcache to
take advantage of hardlinks and devino caching at commit time. And only
after committing do we essentially `pull-local` into the final target
repo. This of course allows us to avoid potentially pulling across the
two filesystems file objects that are already present in the target
repo.

This will be used by coreos-assembler:
https://github.com/coreos/coreos-assembler/pull/190

Closes: #1490

Closes: #1657
Approved by: cgwalters
2018-11-05 21:29:31 +00:00
Jonathan Lebon
21886fc936 compose: Always put workdir on same filesystem as pkgcache
This ensures that we always get hardlinks when checking out of the
pkgcache. This works right now because we indirectly require the target
bare-user repo and the pkgcache to be on the same filesystem by setting
`no_copy_fallback` in the core (I say "indirectly" because that setting
only enforces the workdir to be on the same filesystem as the pkgcache
repo, but since the workdir is currently placed inside the bare-user
repo...).

However, I'd like to change the requirement of a bare-user repo so that
one can commit into a repo on a different file system or a repo of a
different type (e.g. archive repo). This is prep for that.

Closes: #1657
Approved by: cgwalters
2018-11-05 21:29:31 +00:00
Colin Walters
5d9e25653c libvm: Don't sync rust target/
Since it gets enormous, mine is 1.5GB right now.

Closes: #1665
Approved by: jlebon
2018-11-05 19:59:58 +00:00
Colin Walters
bd4d0d8e0a build-sys: With external cbindgen, drop Rust lib dependency
Perhaps an unexpected side benefit of slow compilation processes
is that one has an opportunity to reflect and ponder.

I realized during exactly such a moment that since we moved
`cbindgen` out of our library build, there's no need to wait
for the library to be built before we can start building the C
code.

This is a notable local quality-of-life development improvement.

Closes: #1665
Approved by: jlebon
2018-11-05 19:59:58 +00:00
Jonathan Lebon
55696824df compose: Use previous sepolicy if available
Rather than defaulting to the host system's SELinux policy, we can be
much more efficient here if we instead use the policy of the last commit
if available. Likely, the pkgcache is currently labeled with that
policy, which means we skip the relabeling phase before checkout. But
also, if the policy didn't change at all in the new rootfs, we also skip
the second relabeling phase after assembly.

Closes: #1659
Approved by: cgwalters
2018-11-05 19:20:40 +00:00
Colin Walters
ef587b5c5b build-sys: Turn Rust LTO off by default, add --enable-lto flag
For us, this is primarily right now a size issue.  See:
https://internals.rust-lang.org/t/rust-staticlibs-and-optimizing-for-size/5746

For more information, there are these two issues:
https://github.com/rust-lang/cargo/issues/4349
https://bugzilla.mozilla.org/show_bug.cgi?id=1386371

The basic issue here is that a build with LTO off (and a trivial
change to add a `println!` takes 14s here, and with it on takes 38s.
However, with LTO off the stripped size of `librpmostree_rust.a` is
`6M`, with LTO on it's `1.1M`.

I named this `--enable-lto` as I'd like to investigate doing this
for the C code too.

Closes: #1664
Approved by: jlebon
2018-11-05 18:41:09 +00:00
Jonathan Lebon
0abfcecd38 compose: Print devino cache hits on commit
Requires: https://github.com/ostreedev/ostree/pull/1772

Closes: #1660
Approved by: cgwalters
2018-11-05 17:21:10 +00:00
Colin Walters
50b255a8a9 Move varsubst code into Rust, use it in treefile parsing
External tools often want to parse the ref; for example coreos-assembler
currently does so.  Let's ensure `${basearch}` is expanded with
`--print-only` so they can parse that JSON to get the expanded version
reliably.

Implementation note: this is the first Rust code which exposes a
"GLib-like" C API, notably with GHashTable, so we're making more use
of the glib-rs bindings.

Closes: #1653

Closes: #1655
Approved by: jlebon
2018-11-02 20:36:28 +00:00
Jonathan Lebon
74db30872b compose: Check for error when opening policy
Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
c3be38af72 postprocess: Minor glnx_console interaction fixes
Make sure we always pass an integer and not a float to
`glnx_console_progress_text_percent`. And also make sure we clamp the
value we pass to 100. Not sure how I hit this, somehow the byte counting
we do upfront before committing is *less* than the addition of all the
`g_file_info_get_size()`s?

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
a2d67f658a app/tree: Check upfront if repo is writable
Didn't hit this but this will be useful for a following commit which
delays committing to the final repo until the end.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
82ceef78ce passwd-util: Drop unnecessary OstreeRepo arg
We don't actually need the repo in `rpmostree_passwd_compose_prep()`
since we're already passed the OstreeRepoFile root of the previous
commit as a separate arg.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
8adeb970e3 postprocess: Rename xattr callback
Just a minor readability tweak. Wasn't clear what `read_xattrs_cb`
meant. Also inline an indexing var into the loop.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
5c4e88da9c core: use DEVINO_CANONICAL regardless of repo type
If the policy didn't change, then it's safe to use the devino cache
regardless of the repo type. This dates from #1123; I think the
rationale there was so that it only affects the pkglayering path, where
the system repo is bare. But really, it should be fine to generalize
this since we don't have any xattr callbacks.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
38c2ab3397 app/compose: Minor comment tweak
This is slight bikeshed, though this comment took me a second to parse
because it says "not do it" right before we do it. Let's just move it up
to right outside where the condition is actually tested.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
868bdf167a app/compose: Use g_printerr for note
For consistency with the other note we print lower down.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
37932cf19a app/compose: Check conflicting opts earlier
No reason to do this sanity check so late. Let's bring it up to near the
top of `impl_install_tree()` so we avoid useless work if possible.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
46edef0017 app/compose: Drop helper cachedir_dfd() function
We don't actually need this helper here since we always initialize
`self->cachedir_dfd` nowadays. I took a closer look at this initially
because it was confusing ctags due to the function being named the same
as the struct member it returns.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
6d39cfa78f compose: Don't require SELinux policy in legacy path
In #1630, we lowered SELinux policy loading into the core. However, this
also enabled SELinux policy loading from the host system even in the
legacy (non-unified) compose path. This meant that compose systems now
needed to have the policy installed even though we didn't need it at
all. This caused regressions in pungi:

https://pagure.io/dusty/failed-composes/issue/956

Just make the binding of the "selinux" member conditional on whether or
not we're in unified mode (which is really when we even care about
having it loaded from the start for pkgcache purposes).

Closes: #1656
Approved by: cgwalters
2018-11-02 14:49:32 +00:00
Jonathan Lebon
79d4e52d99 app: Wrap Version in --version in quotes
Wrap the `Version` key in the YAML-compatible output of
`rpm-ostree --version` with quotes so that it's parsed as a string. The
issues with the previous approach in a nutshell:

```
In [5]: yaml.load("asdf: 2018.10")
Out[5]: {'asdf': 2018.1}
```

It's treating the version number as a floating-point. Now, this is
technically a backwards incompatible change, but given that the previous
approach is inherently broken for our needs, I don't see a way around
breaking it now.

See also: 759b099861

Closes: #1654
Approved by: cgwalters
2018-11-01 23:32:19 +00:00
Jonathan Lebon
ff962491a3 man: Fix references to automatic timer and service
There is no `rpm-ostreed-automatic` man page, only
`rpm-ostreed-automatic.service` and `rpm-ostreed-automatic.timer`. Fix
references from the conf man page to those pages.

Closes: #1651

Closes: #1652
Approved by: miabbott
2018-11-01 15:26:06 +00:00
Colin Walters
88ffdc0ae2 Add compose rojig command
This currently requires a `--i-know-this-is-experimental` flag;
I know it'd be a bit more consistent to have it under `ex`, but
what feels weird about that is *most* of the `ex` commands people
use are client side.  This is where we want it to ultimately end
up.

We've landed a lot of prep patches, but I know there's still
a notable amount of code duplication with `compose tree`.  What's
left is about ~700 lines but it's mostly not hard/complex code
anymore.

In the future, I'd like to extract more of the compose code
to a `rust/src/compose.rs` or so, but I think this is sustainable
fow now.

My high level goal is to get this into coreos-assembler and stand
up a Silverblue build that uses it.

Closes: #1512
Approved by: jlebon
2018-10-31 17:57:45 +00:00