Commit Graph

498 Commits

Author SHA1 Message Date
Colin Walters
88ffdc0ae2 Add compose rojig command
This currently requires a `--i-know-this-is-experimental` flag;
I know it'd be a bit more consistent to have it under `ex`, but
what feels weird about that is *most* of the `ex` commands people
use are client side.  This is where we want it to ultimately end
up.

We've landed a lot of prep patches, but I know there's still
a notable amount of code duplication with `compose tree`.  What's
left is about ~700 lines but it's mostly not hard/complex code
anymore.

In the future, I'd like to extract more of the compose code
to a `rust/src/compose.rs` or so, but I think this is sustainable
fow now.

My high level goal is to get this into coreos-assembler and stand
up a Silverblue build that uses it.

Closes: #1512
Approved by: jlebon
2018-10-31 17:57:45 +00:00
Colin Walters
5b8b37658f compose: Make --print-only happen earlier and be quiet
Don't print the version or other warnings; ensure we output clean JSON.

Prep for fixing https://github.com/coreos/fedora-coreos-config/pull/24
(This helps, but we still need to expand `${basearch}`)

Closes: #1648
Approved by: jlebon
2018-10-30 15:04:21 +00:00
Jonathan Lebon
940fc1364a compose: Check that add-files are compatible after parsing
While serde gives us type checking, it of course doesn't understand
semantics beyond that. One example is checking the compatibility of
`add-files` entries with the OSTree model. This is something we can do
upfront early on to avoid surprises for users.

Also tweak the docs to reflect this new check.

Related: #1642

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Colin Walters
fca01e70b5 Lower initial SELinux policy load from compose to core
Add a `selinux` verb to treespec, and bind it from treefile.  If
set, use it in the core to load an initial policy before import,
if we didn't already set a policy.

In practice right now this is only used from the compose path
since the SysrootUpgrader uses the policy from the merge deployment.

Unset the policy if rojig mode is enabled.

Now, non-SELinux use cases are required to set `selinux: false`
in the treespec. For `ex container` I just set it in our example
specs.  Probably that should forcibly disable it in the
treespec but eh, it's experimental.

The other case I can think of is client-side layering; before
we would create a policy using the target root, but it
wasn't a *hard* requirement, i.e. we didn't error out if
`policy_get_name() == NULL`.  Let's preserve that semantic by
hooking off of whether `_new_system()` was used.

Prep for sharing code with `compose rojig`.

Closes: #1630
Approved by: jlebon
2018-10-23 13:40:46 +00:00
Colin Walters
95c1ed3695 tests/compose: Use workdir in tmpdir
This way when debugging if I want to preserve the state so I can
go there and rerun command under gdb for exmaple, all I need
to do is set `env TEST_SKIP_CLEANUP=1` rather than also needing
to find a separate dir and also set `env RPMOSTREE_PRESERVE_TMPDIR=1`.

Closes: #1630
Approved by: jlebon
2018-10-23 13:40:46 +00:00
Jonathan Lebon
6583a557ae app/livefs: Require --i-like-danger switch
We've had multiple reports by now of folks using plain `ex livefs` and
getting their bootloader wrecked:

https://github.com/projectatomic/rpm-ostree/issues/1495
https://github.com/projectatomic/rpm-ostree/issues/1504
https://github.com/ostreedev/ostree/issues/1459

Let's require a scary switch for now to emphasize this.

Closes: #1622
Approved by: cgwalters
2018-10-17 20:10:11 +00:00
Jonathan Lebon
cdda3a01ba postprocess: Add ostree-finalize-staged.path
In preparations for https://github.com/ostreedev/ostree/pull/1740, just
hard enable this path unit for now since centrally-maintained distro
presets still need to be updated.

Closes: #1617
Approved by: cgwalters
2018-10-16 20:05:41 +00:00
Jonathan Lebon
676519e5a5 status: Print systemd or ostree staged failure msg
Try to tease out a bit more info from the journal by looking at the
systemd message when the service transitions to the dead state or even
looking at the OSTree output itself.

Example outputs:

```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         error: opendir(ostree/deploy/fedora-atomic/deploy/887c95887a3047a60372016a0d84536530755b60df3cca33c819f7606e220adf.0): No such file or directory
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         ostree-finalize-staged.service: Failed with result 'timeout'.
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
f6c3616148 status: Detect if staging failed in previous boot
Sample output:

```
$ rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

(Though open to tweaking it).

I also played with directly invoking `journalctl` for the user, but that
can get really spammy with e.g. `os-prober` output and such.

I wrote this in Rust using journal API wrappers because I also plan to
implement the `history` command in Rust and will also enhance that new
`journal` module there for that.

Requires: https://github.com/ostreedev/ostree/pull/1750
Requires: https://github.com/jmesmon/rust-systemd/pull/54
(Though I've pointed the manifest at my branch for now for CI).

Closes: #1567

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Colin Walters
b3f6f25637 core,scripts: When no cachedir+unified-core, disable rofiles-fuse
This is prep for running inside (unprivileged) Kube containers
as they exist today: https://github.com/projectatomic/rpm-ostree/issues/1329

Sadly FUSE today uses a suid binary that ends up wanting CAP_SYS_ADMIN.
I think there's some work on FUSE-in-containers but I'm not sure of
the current status.

What rofiles-fuse here is doing here is protecting is the hardlinked
repo imports.  But if `--cachedir` isn't specified, that repository
gets thrown away anyways.  So there's no real value to using FUSE
here.

Also since nothing is cached, disable the devino cache.

We also make use of --force-copy-zerosized that just landed
in libostree: https://github.com/ostreedev/ostree/pull/1752

Down the line ideally we gain the capability to detect if either
unprivileged overlayfs/FUSE are available.  Then if `--cachedir`
is specified we can make things work.

Closes: #1591
Approved by: jlebon
2018-10-12 19:03:26 +00:00
Colin Walters
f7f2cfeb61 tests: Add validation for provided passwd/group files
Surprising we didn't have this yet.  Prep for oxidizing.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
a86ad96669 compose: Support not specifying a ref
Split out of supporting "pure rojig" work.  We also want
to support this for doing "oscontainers" as is planned for
Red Hat CoreOS.  The user experience in both cases is oriented
around versioning of the external wrapper, not the inner ref/commit.

Note for users/builders who want to make use of this feature:
You probably want to mirror the changes in our test suite here to
use the compose JSON and parse the resulting `ostree-commit` out of that.

Closes: #1603
Approved by: jlebon
2018-10-09 19:47:25 +00:00
Colin Walters
638fab02bd tests: Add a test for inheritance order of postprocess
Since I got it backwards when rewriting it in Rust.

Closes: #1574
Approved by: jlebon
2018-10-04 13:17:47 +00:00
Colin Walters
3c9e212f71 compose-tests: Use yaml.safe_dump
Otherwise we end up with weird `!!python/unicode` stuff:
https://stackoverflow.com/questions/20352794/pyyaml-is-producing-undesired-python-unicode-output/20369984

Closes: #1597
Approved by: jlebon
2018-10-02 18:03:22 +00:00
Jonathan Lebon
71588f9744 daemon/utils: Filter out rpmostree.rpmdb.pkglist
Drop the `rpmostree.rpmdb.pkglist` keys from the deployment metadata
since users shouldn't need it and it greatly increases the size of the
output.

Closes: #1577
Approved by: cgwalters
2018-09-25 15:20:57 +00:00
Colin Walters
c6de759cc0 compose: Drop combined ostree+rojig compose tree
Currently `compose tree` supports writing to both an OSTree
repository and optionally simultaneously generating a rojig
RPM or rojig set.

We are instead going to split off a separate `compose rojig` verb,
since we want to support a "pure rojig" mode, and having all
3 options would make this code far too complex.

For now anyone who wants *both* formats can use `compose tree` then
follow up with `ex commit2rojig`.

This is also inline with the new coreos-assembler push, where
we'll treat `rpm-ostree compose` as more of a "low level" API,
and wrap it in a more opinionated model there.

Closes: #1570
Approved by: jlebon
2018-09-21 14:42:14 +00:00
Jonathan Lebon
ea5bec6127 app/db-diff: Diff against rollback if no pending
Often, after rebooting from an upgrade, I want to check what was just
updated. This patch makes `db diff` do the right thing in those cases.
Specifically, before `db diff` without arguments would default to
diff'ing the pending deployment with the booted deployment and error out
otherwise. This patch extends the logic so that if there's a rollback
deployment, we default to diff'ing against that.

Closes: #1565
Approved by: cgwalters
2018-09-20 16:12:14 +00:00
Colin Walters
16f48afd5e commit2rojig: Take a (YAML) manifest, not a spec file
This completes the removal of externally-provided rojig spec files.

In the current flow then `compose tree --ex-rojig-output-*` will
end up parsing the manifest twice.  But this is prep for defaulting
rojig to "pure" mode without a separate ostree repo.  Then
anyone who wants to do both can run `commit2rojig` after.

Closes: #1561
Approved by: jlebon
2018-09-17 23:45:57 +00:00
Colin Walters
e208383526 tests: Add rojig spec into base config
This is prep for fully dropping the separate spec file, which
is now just used by `commit2rojig`.

A compose test can now specify it wants YAML instead of JSON.

Closes: #1561
Approved by: jlebon
2018-09-17 23:45:57 +00:00
Jonathan Lebon
f098d22e03 tests/vmcheck: Create correct base commits from layered
Our trick of using layered commits as base commit updates doesn't jive
well with the new opportunistic reuse of the base rpmdb. The issue is
that a layered commit includes the rpmdb of *its* base commit at the
`/usr/lib/sysimage` location. So to convert it into a proper base commit
means that the layered rpmdb should move there.

Closes: #1502
Approved by: cgwalters
2018-09-17 20:07:45 +00:00
Jonathan Lebon
a75460f538 upgrader: Reuse existing rpmdb checkout if available
Check if we can reuse the base rpmdb from the pending deployment if it
matches the base rev we're targeting. This allows us to avoid checking
out the tree early only to later on discard it. Such cases include
layering existing packages and inactive requests.

Closes: #1502
Approved by: cgwalters
2018-09-17 20:07:45 +00:00
Robert Fairley
b82db2b501 initramfs: Return error when --arg specified without --enable
This change gives an error when --arg is specified on its own.

Previously, if adding --arg without --enable, no changes occur and
only the initramfs regeneration status is printed. Since --arg only
takes effect when used with --enable, this should also give an error
(or let the user know in some way).

Closes: #1548
Approved by: jlebon
2018-09-12 20:45:19 +00:00
Robert Fairley
be7d87e4b9 initramfs: Print regeneration status for pending deployment
Previously, after enabling initramfs regeneration, the initramfs
command would indicate that initramfs regeneratin is disabled,
suggesting that the initramfs had not actually been enabled. This
happened because the initramfs regeneration status for the current
deployment was printed; initramfs regeneration is only enabled for a
pending deployment.

This change prints the regeneration for the pending deployment,
rather than the current, when no options are given to the initramfs
command.

Fixes: #1526

Closes: #1548
Approved by: jlebon
2018-09-12 20:45:19 +00:00
Micah Abbott
bce966a981 vmcheck/misc-1: skip the overlay check when needed
It's possible to run the `vmcheck` tests against an existing host that
has `rpm-ostree` already present.  We don't overlay the built binaries
in this situation, so we should not check for the presence of the
overlay in the commit meta.

Closes: #1555
Approved by: cgwalters
2018-09-12 14:48:02 +00:00
Jonathan Lebon
807f21788e Hard require staging
This removes the logic around supporting opting out of the staging
feature. We don't want to support multiple configurations here, and at
this point, staging should be considered stable.

Closes: #1546
Approved by: cgwalters
2018-09-11 20:55:48 +00:00
Colin Walters
5927d85192 tests: Rename one libcomposetest.sh
We had two `libcomposetest.sh` which I always found confusing.
Fix the naming of the one that's shared with `ex-container`
to be more obvious.

Closes: #1543
Approved by: jlebon
2018-09-10 17:06:10 +00:00
Jonathan Lebon
8976e502fc daemon: Make inactive requests a hard error
Flip the switch and make `rpm-ostree install glibc` a hard error if
`--allow-inactive` is not provided.

Closes: #1545
Approved by: cgwalters
2018-09-10 16:17:14 +00:00
Colin Walters
fe87ad4f29 tests: Drop conditionals for Rust enablement
We now hard require it, so change these tests to do so as
well.

Closes: #1544
Approved by: jlebon
2018-09-10 13:19:51 +00:00
Jonathan Lebon
651a9e01f4 core: Accumulate all missing pkgs into a single error
Accumulate the list of requested packages that were not found and error
out with the full list rather than failing early. This fixes a small UX
papercut in certain situations.

Closes: #1540

Closes: #1541
Approved by: cgwalters
2018-09-08 23:32:13 +00:00
Colin Walters
6f6728b992 compose: Add --write-composejson-to
For higher level tools driving rpm-ostree, the command line arguments
are mostly OK as inputs, but the addition of `--write-commitid-to`
shows that we really want structured data that these tools can parse.
JSON is a good enough interchange format, let's use that.

Most notably this does a pkgdiff in the JSON which several higher
level tools do too.

Closes: #1529
Approved by: jlebon
2018-09-07 18:52:54 +00:00
Colin Walters
b9e3bfd7b3 importer: Don't generate var-tmpfiles.d for rpm itself
This caused `/var/lib/rpm` to be a directory in my Fedora CoreOS
builds.

Closes: #1532
Approved by: sinnykumari
2018-09-07 15:10:10 +00:00
Jonathan Lebon
86f2b67d2b vmcheck: Store temporary files in /var/tmp instead
Since `/tmp` might be on tmpfs, so we'd lose it on reboot. But we have
tests that need it to persist across reboots.

Closes: #1531
Approved by: miabbott
2018-09-07 14:20:01 +00:00
Micah Abbott
c9b4a8195e tests/vmcheck: handle missing /etc/yum.repos.d
It's possible to run the `vmcheck` tests against an existing VM which
may not have any existing `/etc/yum.repos.d` directory.  Since the
tests are providing their own repo files, we should be able to run
them in this uncommon configuration.

Closes: #1530
Approved by: jlebon
2018-09-06 20:50:25 +00:00
Colin Walters
961482f088 Add inputhash Provides to rojig RPMs
Prep for rojig-only compose support.  We want to be able
to quickly detect whether or not there were changes in the
input, just like we do when targeting an OSTree repository.

Closes: #1528
Approved by: jlebon
2018-09-06 14:33:48 +00:00
Colin Walters
279e7c4f1b treefile: Support inline postprocess element (for inheritance)
I'm trying to have a more opinionated model where custom builds
use inheritance, and currently one can only have a single
`postprocess-script`.

Further, in YAML it's very convenient to use inline vs external
data.

Closes: #1527
Approved by: jlebon
2018-09-05 15:27:07 +00:00
Ruixin Bao
9920094cfa tests: add test for sorting + conversion of sysuser
As title, added tests to check both conversion + sorting for sysuser
entries, and verifies its correctness.

Closes: #1519
Approved by: cgwalters
2018-08-30 17:37:27 +00:00
Ruixin Bao
4a942acdce passwd-util: adds conversion of group entries to sysusers
Similar to the passwd conversion, this commit
implements conversion of group entries into sysuser struct entries

Tests are also included in the commit

Closes: #1519
Approved by: cgwalters
2018-08-30 17:37:27 +00:00
Ruixin Bao
b4d301836d test: add unit test for passwd to sysuser conversion
Add initial implementation for unit test to test
converting passwd entries -> sysusers

Closes: #1519
Approved by: cgwalters
2018-08-30 17:37:27 +00:00
Colin Walters
3d60a31aaa Fix include: with machineid-compat and a few other keys
It turns out we basically have to slap an `Option<T>` around
everything, (in particular `bool` etc.) we need to be able
to distinguish in (I believe) all the cases between
"value unspecified" and "value provided".

Concretely it didn't work to try to set `machineid-compat: false`
in an included yaml treefile becuase it was just defaulted to `true`
by the toplevel.

Down the line we should move all of the parsing into Rust
and have two different `struct` types for "YAML we load" versus
"verified treefile".

Closes: https://github.com/projectatomic/rpm-ostree/issues/1524

Closes: #1525
Approved by: lucab
2018-08-29 12:59:34 +00:00
Jonathan Lebon
04c0678fa6 app: Add support for passing URLs to RPMs
This teaches the client to fetch packages from URLs directly so that one
doesn't have to `curl` first and then install. Supported anywhere
package filenames are allowed (notably: `install` and
`override replace`).

One neat things about this is that we download the file into an
`O_TMPFILE` and then pass on ownership of that fd directly to the
daemon. So at no point are the packages actually laying visible on the
system. (Assuming the filesystem supports `O_TMPFILE` that is).

This adds direct linking to libcurl and openssl, two libraries which we
were already pulling in indirectly.

Closes: #1508
Approved by: cgwalters
2018-08-23 11:16:15 +00:00
Colin Walters
40be3fb1cf Add recommends to treefile (default on)
This is for: https://github.com/projectatomic/rpm-ostree/issues/718
But I'm not going to close that issue as this only does the server
side, and I think we should support it client side too.

Since I wrote that issue, we ended up skipping the `dnf_transaction_depsolve()`
API, and hence we don't need to block on a libdnf change.  So
this was quite simple.

Closes: #1513
Approved by: jlebon
2018-08-23 04:24:07 +00:00
Colin Walters
5126b2774e Stabilize stage automatic update policy
Now that staging is the default, there's no reason for this
policy not to be stable too.

Closes: #1510
Approved by: jlebon
2018-08-21 01:01:45 +00:00
Colin Walters
b6d07487d5 Turn staged deployments on by default
We've put a lot of work into staged deployments, it's time
to pull the trigger and turn them on by default.  This is
a key step for enabling `stage` mode automatic updates by
default in e.g. Fedora CoreOS/Silverblue.

We add a new `--disable-staged` build-time option to flip
things back.

Closes: #1430
Approved by: jlebon
2018-08-20 20:32:00 +00:00
Colin Walters
44b39a7340 livefs: Require deployment staging
Staging fixes the `/etc` bug for livefs.  There's actually more
we could do here around taking advantage of staging for livefs;
for example, I think once the livefs is complete, we could just delete
the staged deployment.  And then we don't need to render on the next
boot the live status, etc.

Anyways, all that can come in the future.  This is prep for
enabling staging by default.

Closes: #1430
Approved by: jlebon
2018-08-20 20:32:00 +00:00
Colin Walters
e1b62d0f7d tests/vmcheck: Display human-readable status on jq failure
So it's easier to debug.

This inlines the helper into the only function that uses it.

Closes: #1430
Approved by: jlebon
2018-08-20 20:32:00 +00:00
Colin Walters
2278b60a15 tests: Remove dead assert_status_jq
Nothing calls it today; looks like it was last used in
283b915ecf

Closes: #1430
Approved by: jlebon
2018-08-20 20:32:00 +00:00
Jonathan Lebon
4ccad2f64b app/status: Tweak output with --booted
Fix `--booted` was printing an extra line after `Deployments:`. While
we're there, also update the section name to `BootedDeployment:` in that
case to be more correct. (We're not printing all the deployments, only
the booted deployment.)

Closes: #1503
Approved by: cgwalters
2018-08-14 12:32:50 +00:00
Jonathan Lebon
45e162fb69 ci: Split compose test into two
The `f28-compose` test keeps timing out. Some time recently, I/O
performance of the internal OpenStack instance used for testing has
degraded. I have a ticket open to investigate the regression though
haven't had any luck so far.

Let's just take the easy way out and split the test into two testsuites.
This is obviously hacky, and sad, and unfortunate. But the PRs must keep
flowing until we finally wean off of OpenStack.

Closes: #1498
Approved by: cgwalters
2018-08-13 21:06:18 +00:00
Colin Walters
15aecff36a livefs: Rename --replace to --dangerous-do-not-use-replace
And don't mention "replace" in the error texts.
This is (now) known broken with kernel updates; making that not
dangerous isn't going to be easy.  I debated entirely removing it
but for now let's just make it harder to use.

Ref: https://github.com/projectatomic/rpm-ostree/issues/1495

Closes: #1497
Approved by: jlebon
2018-08-09 15:04:11 +00:00
Jonathan Lebon
7c87a553e4 tests/compose: Fix wrong assert in test-rojig-e2e
We should be expecting testpkg-1.1-1 here, not 1.0-1. This was passing
before because of the nondeterministic `find` output (fixed in the
previous commit) which could spit out the older rojig RPM.

Closes: #1491
Approved by: cgwalters
2018-08-07 16:00:13 +00:00