Commit Graph

2501 Commits

Author SHA1 Message Date
Colin Walters
50b255a8a9 Move varsubst code into Rust, use it in treefile parsing
External tools often want to parse the ref; for example coreos-assembler
currently does so.  Let's ensure `${basearch}` is expanded with
`--print-only` so they can parse that JSON to get the expanded version
reliably.

Implementation note: this is the first Rust code which exposes a
"GLib-like" C API, notably with GHashTable, so we're making more use
of the glib-rs bindings.

Closes: #1653

Closes: #1655
Approved by: jlebon
2018-11-02 20:36:28 +00:00
Jonathan Lebon
74db30872b compose: Check for error when opening policy
Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
c3be38af72 postprocess: Minor glnx_console interaction fixes
Make sure we always pass an integer and not a float to
`glnx_console_progress_text_percent`. And also make sure we clamp the
value we pass to 100. Not sure how I hit this, somehow the byte counting
we do upfront before committing is *less* than the addition of all the
`g_file_info_get_size()`s?

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
a2d67f658a app/tree: Check upfront if repo is writable
Didn't hit this but this will be useful for a following commit which
delays committing to the final repo until the end.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
82ceef78ce passwd-util: Drop unnecessary OstreeRepo arg
We don't actually need the repo in `rpmostree_passwd_compose_prep()`
since we're already passed the OstreeRepoFile root of the previous
commit as a separate arg.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
8adeb970e3 postprocess: Rename xattr callback
Just a minor readability tweak. Wasn't clear what `read_xattrs_cb`
meant. Also inline an indexing var into the loop.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
5c4e88da9c core: use DEVINO_CANONICAL regardless of repo type
If the policy didn't change, then it's safe to use the devino cache
regardless of the repo type. This dates from #1123; I think the
rationale there was so that it only affects the pkglayering path, where
the system repo is bare. But really, it should be fine to generalize
this since we don't have any xattr callbacks.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
38c2ab3397 app/compose: Minor comment tweak
This is slight bikeshed, though this comment took me a second to parse
because it says "not do it" right before we do it. Let's just move it up
to right outside where the condition is actually tested.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
868bdf167a app/compose: Use g_printerr for note
For consistency with the other note we print lower down.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
37932cf19a app/compose: Check conflicting opts earlier
No reason to do this sanity check so late. Let's bring it up to near the
top of `impl_install_tree()` so we avoid useless work if possible.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
46edef0017 app/compose: Drop helper cachedir_dfd() function
We don't actually need this helper here since we always initialize
`self->cachedir_dfd` nowadays. I took a closer look at this initially
because it was confusing ctags due to the function being named the same
as the struct member it returns.

Closes: #1658
Approved by: cgwalters
2018-11-02 19:26:31 +00:00
Jonathan Lebon
6d39cfa78f compose: Don't require SELinux policy in legacy path
In #1630, we lowered SELinux policy loading into the core. However, this
also enabled SELinux policy loading from the host system even in the
legacy (non-unified) compose path. This meant that compose systems now
needed to have the policy installed even though we didn't need it at
all. This caused regressions in pungi:

https://pagure.io/dusty/failed-composes/issue/956

Just make the binding of the "selinux" member conditional on whether or
not we're in unified mode (which is really when we even care about
having it loaded from the start for pkgcache purposes).

Closes: #1656
Approved by: cgwalters
2018-11-02 14:49:32 +00:00
Jonathan Lebon
79d4e52d99 app: Wrap Version in --version in quotes
Wrap the `Version` key in the YAML-compatible output of
`rpm-ostree --version` with quotes so that it's parsed as a string. The
issues with the previous approach in a nutshell:

```
In [5]: yaml.load("asdf: 2018.10")
Out[5]: {'asdf': 2018.1}
```

It's treating the version number as a floating-point. Now, this is
technically a backwards incompatible change, but given that the previous
approach is inherently broken for our needs, I don't see a way around
breaking it now.

See also: 759b099861

Closes: #1654
Approved by: cgwalters
2018-11-01 23:32:19 +00:00
Jonathan Lebon
ff962491a3 man: Fix references to automatic timer and service
There is no `rpm-ostreed-automatic` man page, only
`rpm-ostreed-automatic.service` and `rpm-ostreed-automatic.timer`. Fix
references from the conf man page to those pages.

Closes: #1651

Closes: #1652
Approved by: miabbott
2018-11-01 15:26:06 +00:00
Colin Walters
88ffdc0ae2 Add compose rojig command
This currently requires a `--i-know-this-is-experimental` flag;
I know it'd be a bit more consistent to have it under `ex`, but
what feels weird about that is *most* of the `ex` commands people
use are client side.  This is where we want it to ultimately end
up.

We've landed a lot of prep patches, but I know there's still
a notable amount of code duplication with `compose tree`.  What's
left is about ~700 lines but it's mostly not hard/complex code
anymore.

In the future, I'd like to extract more of the compose code
to a `rust/src/compose.rs` or so, but I think this is sustainable
fow now.

My high level goal is to get this into coreos-assembler and stand
up a Silverblue build that uses it.

Closes: #1512
Approved by: jlebon
2018-10-31 17:57:45 +00:00
Colin Walters
c044f59726 rust: Remove pub from modules
Conceptually we're only exporting the FFI code right now.

Closes: #1649
Approved by: jlebon
2018-10-30 16:16:22 +00:00
Colin Walters
5b8b37658f compose: Make --print-only happen earlier and be quiet
Don't print the version or other warnings; ensure we output clean JSON.

Prep for fixing https://github.com/coreos/fedora-coreos-config/pull/24
(This helps, but we still need to expand `${basearch}`)

Closes: #1648
Approved by: jlebon
2018-10-30 15:04:21 +00:00
Colin Walters
87441d9ec2 rust: Move FFI bits into ffi submodule for each file
The `lib.rs` file was starting to accumulate, and I want to do this
before adding more.  I just made this up, but an `ffi` submodule
in each file seems to work well.  It isolates the FFI consumption
there still (so e.g. if we want pure Rust unit tests, we don't
need to deal with FFI).

Closes: #1646
Approved by: jlebon
2018-10-30 14:24:42 +00:00
Colin Walters
99776be162 core: Disable librpm SELinux plugin when writing rpmdb
OSTree was designed from the very beginning of its existence
to support SELinux well instead of being something wedged on.
rpm-ostree builds on that foundation.  We don't want to have anything
to do with librpm's SELinux code.  And with unified core,
we usually don't, but that `rpm-plugin-selinux` code does get loaded.

Disable it here.  The main reason I'm submitting this patch
is to help an effort in coreos-assembler to use a "supermin" virtual
machine: https://github.com/coreos/coreos-assembler/pull/124

Closes: #1647
Approved by: jlebon
2018-10-29 18:42:10 +00:00
Colin Walters
b5d34b7268 compose: Factor out helper for writing composejson
Prep for sharing this code with rojig.

Closes: #1636
Approved by: jlebon
2018-10-29 01:35:20 +00:00
Jonathan Lebon
f3b8ef043e libpriv: Sanity check that paths are OSTree compatible
I originally was going to add the check here and error out to the user,
but `add-files` is handled all the way near the end of the compose,
which meant that users would have to wait through it all before getting
an error. Though the check is enforced at parsing time now, I think it's
still useful at postprocess time as well as a sanity check.

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Jonathan Lebon
940fc1364a compose: Check that add-files are compatible after parsing
While serde gives us type checking, it of course doesn't understand
semantics beyond that. One example is checking the compatibility of
`add-files` entries with the OSTree model. This is something we can do
upfront early on to avoid surprises for users.

Also tweak the docs to reflect this new check.

Related: #1642

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Jonathan Lebon
f8f47480f1 docs/treefile: Add info about paths for add-files
Add a note that only certain paths actually make sense in the
`add-files` directive and link to the OSTree docs. Let's also the
examples given to more realistic and correct ones.

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Colin Walters
202192d61e rust: Create generic ref_from_raw_ptr function
I need to do some hunting for a "ffi helper" crate.  I kind of
understand why these things aren't in the std library.

Anyways this is easy for now, prep for adding more types.

Closes: #1641
Approved by: jlebon
2018-10-26 20:04:08 +00:00
Colin Walters
92d207de5d rust: Merge glibutils with utility FFI helpers
Create `ffiutil.rs` from the merger.  This way all of our FFI
helpers are in one place.  Prep for further refactoring.

Closes: #1641
Approved by: jlebon
2018-10-26 20:04:08 +00:00
Colin Walters
245d238563 Use autocleanup for task output
I was playing with https://crates.io/crates/indicatif
and this is prep for using it.

By using autocleanups here we ensure that the output state
ends even if we encounter an error.  We previously had a few
explicit `failed` prints in cases where failure was more common
such as dependency resolution, but that was far from consistent.

Now we will always ensure a newline is printed even if an error
occurs, and we won't have to worry about missing calls to
`_task_done()` in the success case.

The downside of this is that we lose the `done` bit in the success
path...I thought about maybe taking a pointer to the error and
printing `done` or `failed` depending on whether it's non-`NULL`
but eh...the `done` is implied by us going on to the next line.
Also that issue will be fixed with the indicatif work.

Closes: #1640
Approved by: jlebon
2018-10-26 19:20:25 +00:00
Jonathan Lebon
a87486e18e rust: Drop crates-io patch and use 0.4.0
The latest release of the crate includes the features and fixes we need.

Closes: #1639
Approved by: cgwalters
2018-10-25 19:01:39 +00:00
Jonathan Lebon
537a040b00 Release 2018.9
Closes: #1633
Approved by: cgwalters
2018-10-25 18:12:37 +00:00
Christian Glombek
20acf6a2f5 compose: Print version number
Users doing builds that are using newer features
want to be sure they have the right version.

Closes: #1556

Closes: #1637
Approved by: jlebon
2018-10-25 17:07:10 +00:00
Colin Walters
fe5adaf7ab build-sys: Fix make rustfmt, and run it
When we added the bindgen config this broke as it's not a `.rs`.
Let's keep the source formatted canonically.

Closes: #1635
Approved by: jlebon
2018-10-23 19:05:06 +00:00
Michael Nguyen
50ef17ed95 app/status: line wrap RemovedBasePackages
Wrap RemovedBasePackages like other package lists in the status
command so it looks prettier.

fixup! app/status: line wrap RemovedBasePackages

fixup! app/status: line wrap RemovedBasePackages

fixup! app/status: line wrap RemovedBasePackages

Resolves: #1613

Closes: #1632
Approved by: jlebon
2018-10-23 14:28:19 +00:00
Colin Walters
fca01e70b5 Lower initial SELinux policy load from compose to core
Add a `selinux` verb to treespec, and bind it from treefile.  If
set, use it in the core to load an initial policy before import,
if we didn't already set a policy.

In practice right now this is only used from the compose path
since the SysrootUpgrader uses the policy from the merge deployment.

Unset the policy if rojig mode is enabled.

Now, non-SELinux use cases are required to set `selinux: false`
in the treespec. For `ex container` I just set it in our example
specs.  Probably that should forcibly disable it in the
treespec but eh, it's experimental.

The other case I can think of is client-side layering; before
we would create a policy using the target root, but it
wasn't a *hard* requirement, i.e. we didn't error out if
`policy_get_name() == NULL`.  Let's preserve that semantic by
hooking off of whether `_new_system()` was used.

Prep for sharing code with `compose rojig`.

Closes: #1630
Approved by: jlebon
2018-10-23 13:40:46 +00:00
Colin Walters
95c1ed3695 tests/compose: Use workdir in tmpdir
This way when debugging if I want to preserve the state so I can
go there and rerun command under gdb for exmaple, all I need
to do is set `env TEST_SKIP_CLEANUP=1` rather than also needing
to find a separate dir and also set `env RPMOSTREE_PRESERVE_TMPDIR=1`.

Closes: #1630
Approved by: jlebon
2018-10-23 13:40:46 +00:00
Colin Walters
f1fa436c8a rust/treefile: Use macros to reduce redundancy in treefile merging
On the plus side, when submitting a patch to Github, no one knows
how long it took you to figure out...

Anyways so this reduces redundancy.  I double-checked the list.
I was inspired to pick this back up after seeing a Rust code
snippet somewhere noting that macros defined inside a function
can capture variables, which simplifies this even more.

Closes: #1631
Approved by: jlebon
2018-10-22 18:53:16 +00:00
Jonathan Lebon
6583a557ae app/livefs: Require --i-like-danger switch
We've had multiple reports by now of folks using plain `ex livefs` and
getting their bootloader wrecked:

https://github.com/projectatomic/rpm-ostree/issues/1495
https://github.com/projectatomic/rpm-ostree/issues/1504
https://github.com/ostreedev/ostree/issues/1459

Let's require a scary switch for now to emphasize this.

Closes: #1622
Approved by: cgwalters
2018-10-17 20:10:11 +00:00
Colin Walters
7c41188476 Switch to ISO8601 timestamps in more places
I was doing this for the "generated" bit on the compose side,
but let's be consistent elsewhere.

Closes: #1624
Approved by: jlebon
2018-10-17 19:32:12 +00:00
Colin Walters
0f813c1758 scripts: Fully neuter microcode_ctl scripts everywhere
I swear I tested the previous PR here but it looks like they
did a new build recently whose release ends in `el7_5` which no
longer matches.

Since the Fedora spec doesn't have any scripts, let's go back
to using the gperf list.

Closes: #1623
Approved by: jlebon
2018-10-17 18:48:38 +00:00
Jonathan Lebon
cdda3a01ba postprocess: Add ostree-finalize-staged.path
In preparations for https://github.com/ostreedev/ostree/pull/1740, just
hard enable this path unit for now since centrally-maintained distro
presets still need to be updated.

Closes: #1617
Approved by: cgwalters
2018-10-16 20:05:41 +00:00
Colin Walters
05cb34d6b4 compose: Lower metadata logic for read/generate into a helper
Prep for sharing code with `compose rojig`.

Closes: #1618
Approved by: jlebon
2018-10-16 19:18:17 +00:00
Colin Walters
c102a710a4 compose; Lower add-files read into composeutil
Make this function operate like all of the others, on the pair of
(treefile_rs, treefile).  That way the "parsing" is co-located
with usage.

Prep for `compose rojig`.

Closes: #1618
Approved by: jlebon
2018-10-16 19:18:17 +00:00
Colin Walters
10a60d2833 compose: Clean up passwd/group injection API a bit
Rename the function dealing with this, document it a bit,
and move all of the logic related to this inside there.

Prep for sharing more with `compose rojig`.

Closes: #1618
Approved by: jlebon
2018-10-16 19:18:17 +00:00
Jonathan Lebon
67a831b043 rust/journal: Adapt for el7 platforms
Supporting ancient systemd is painful, though given that it may end up
in RHCOS at least, it seems worth the effort.

Basically, the big changes here are:
- avoid using `_SYSTEMD_UNIT` since it might be missing in some entries
- also grep for the el7 version of the systemd msg when a service fails
- use `_TRANSPORT=stdout` for the error msg in case of ordering issues

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Colin Walters
d24f21343f build-sys: Just error out if building with multiple uids
The `--frozen` stuff ends up being annoying when switching
branches.  What we're really trying to protect against here
is the `sudo make install` problem, so let's test for that
more directly by verifying the uids.

(The previous code was also totally broken as it used `$` where
`$$` should have been in multiple places)

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
676519e5a5 status: Print systemd or ostree staged failure msg
Try to tease out a bit more info from the journal by looking at the
systemd message when the service transitions to the dead state or even
looking at the OSTree output itself.

Example outputs:

```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         error: opendir(ostree/deploy/fedora-atomic/deploy/887c95887a3047a60372016a0d84536530755b60df3cca33c819f7606e220adf.0): No such file or directory
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         ostree-finalize-staged.service: Failed with result 'timeout'.
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Colin Walters
808a80bfd7 rust/lib: Remove unnecessary integer casts
The compiler will infer the correct thing.

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
5fb5effbd2 packaging: Nuke more vendored sources
Nuke systemd source files for the same reasons as libcurl (see #1554).
Also noticed that libz-sys was doing this, though it's not new to this
patch.

For reference, see: https://github.com/projectatomic/rpm-ostree/pull/1554

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
f6c3616148 status: Detect if staging failed in previous boot
Sample output:

```
$ rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

(Though open to tweaking it).

I also played with directly invoking `journalctl` for the user, but that
can get really spammy with e.g. `os-prober` output and such.

I wrote this in Rust using journal API wrappers because I also plan to
implement the `history` command in Rust and will also enhance that new
`journal` module there for that.

Requires: https://github.com/ostreedev/ostree/pull/1750
Requires: https://github.com/jmesmon/rust-systemd/pull/54
(Though I've pointed the manifest at my branch for now for CI).

Closes: #1567

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
dbdbaf87ee ci: Make adduser invocation idempotent
This is a bit of a selfish request, though I also really like
`ci/build.sh` for hacking in a throwaway environment. This patch allows
one to rerun `ci/build.sh` without erroring out on `testuser` already
existing.

Closes: #1621
Approved by: cgwalters
2018-10-16 16:03:06 +00:00
Jonathan Lebon
cc69168aaf ci: Drop CI_PKGS and epel-release in .papr.yml
Move specific package mentions from the `.papr.yml` file to
`ci/installdeps.sh`. I find the latter script super useful for
bootstrapping a pet container for rpm-ostree development and this helps
with that.

Closes: #1621
Approved by: cgwalters
2018-10-16 16:03:06 +00:00
Jonathan Lebon
3a6315aac9 libpriv/passwd-util: Fix leaks on error path
Allocate new memory *after* the potential error exit, not before.

Closes: #1620
Approved by: cgwalters
2018-10-16 00:34:09 +00:00