Commit Graph

2433 Commits

Author SHA1 Message Date
Colin Walters
b6d07487d5 Turn staged deployments on by default
We've put a lot of work into staged deployments, it's time
to pull the trigger and turn them on by default.  This is
a key step for enabling `stage` mode automatic updates by
default in e.g. Fedora CoreOS/Silverblue.

We add a new `--disable-staged` build-time option to flip
things back.

Closes: #1430
Approved by: jlebon
2018-08-20 20:32:00 +00:00
Colin Walters
44b39a7340 livefs: Require deployment staging
Staging fixes the `/etc` bug for livefs.  There's actually more
we could do here around taking advantage of staging for livefs;
for example, I think once the livefs is complete, we could just delete
the staged deployment.  And then we don't need to render on the next
boot the live status, etc.

Anyways, all that can come in the future.  This is prep for
enabling staging by default.

Closes: #1430
Approved by: jlebon
2018-08-20 20:32:00 +00:00
Colin Walters
e1b62d0f7d tests/vmcheck: Display human-readable status on jq failure
So it's easier to debug.

This inlines the helper into the only function that uses it.

Closes: #1430
Approved by: jlebon
2018-08-20 20:32:00 +00:00
Colin Walters
2278b60a15 tests: Remove dead assert_status_jq
Nothing calls it today; looks like it was last used in
283b915ecf

Closes: #1430
Approved by: jlebon
2018-08-20 20:32:00 +00:00
Colin Walters
f2a871dfeb build-sys: Stop checking for Python
We don't need it anymore at build time.  The test suite
still uses it, but we detect it dynamically there.

Closes: #1505
Approved by: jlebon
2018-08-14 13:51:09 +00:00
Jonathan Lebon
fb66070806 app/status: Use ├─ glyph when printing origin repos
And only use `└─` for the last repo in the list.

Before:

```
...
Commit: 106a7095fd46741948ba60dbd0586668ae7abe6336671444ec790d7541a88778
        └─ repo-0 (2018-07-23 00:27:05)
        └─ repo-1 (2018-04-25 04:27:32)
...
```

After:

```
...
Commit: 106a7095fd46741948ba60dbd0586668ae7abe6336671444ec790d7541a88778
        ├─ repo-0 (2018-07-23 00:27:05)
        └─ repo-1 (2018-04-25 04:27:32)
...
```

Closes: #1503
Approved by: cgwalters
2018-08-14 12:32:50 +00:00
Jonathan Lebon
4ccad2f64b app/status: Tweak output with --booted
Fix `--booted` was printing an extra line after `Deployments:`. While
we're there, also update the section name to `BootedDeployment:` in that
case to be more correct. (We're not printing all the deployments, only
the booted deployment.)

Closes: #1503
Approved by: cgwalters
2018-08-14 12:32:50 +00:00
Jonathan Lebon
45e162fb69 ci: Split compose test into two
The `f28-compose` test keeps timing out. Some time recently, I/O
performance of the internal OpenStack instance used for testing has
degraded. I have a ticket open to investigate the regression though
haven't had any luck so far.

Let's just take the easy way out and split the test into two testsuites.
This is obviously hacky, and sad, and unfortunate. But the PRs must keep
flowing until we finally wean off of OpenStack.

Closes: #1498
Approved by: cgwalters
2018-08-13 21:06:18 +00:00
Jonathan Lebon
1263886c4c compose: Check for packages after processing includes
Otherwise we risk rejecting perfectly valid treefiles. E.g.
fedora-atomic only defines packages in the `-base` file. Let's just move
the check to after having processed all the includes, right where we
collate packages from all the various entries.

The FAHC treecompose is hitting this right now.

Closes: #1500
Approved by: cgwalters
2018-08-13 19:18:38 +00:00
Jonathan Lebon
e2fe8b1cf1 Release 2018.7
Closes: #1496
Approved by: cgwalters
2018-08-09 16:06:24 +00:00
Colin Walters
15aecff36a livefs: Rename --replace to --dangerous-do-not-use-replace
And don't mention "replace" in the error texts.
This is (now) known broken with kernel updates; making that not
dangerous isn't going to be easy.  I debated entirely removing it
but for now let's just make it harder to use.

Ref: https://github.com/projectatomic/rpm-ostree/issues/1495

Closes: #1497
Approved by: jlebon
2018-08-09 15:04:11 +00:00
Jonathan Lebon
eae3a40f3b libpriv/core: Always call rpmtsSetRootDir()
Call `rpmtsSetRootDir()` even in the `JUSTDB` run. This brings us in
line with libdnf and the rpm CLI itself, which both unconditionally call
`rpmtsSetRootDir()` regardless of whether we're targeting the system
root or a subdir.

This doesn't cause a `chroot()` to happen since librpm only does this
if the target root dir is not `/`.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1613517

Closes: #1492
Approved by: cgwalters
2018-08-08 18:29:45 +00:00
Jonathan Lebon
7c87a553e4 tests/compose: Fix wrong assert in test-rojig-e2e
We should be expecting testpkg-1.1-1 here, not 1.0-1. This was passing
before because of the nondeterministic `find` output (fixed in the
previous commit) which could spit out the older rojig RPM.

Closes: #1491
Approved by: cgwalters
2018-08-07 16:00:13 +00:00
Jonathan Lebon
3891aa2563 tests/compose: Simplify test-rojig-e2e file search
This test was relying on the order in which `find` reports matching path
names to find the right RPM. This was failing for me locally sometimes
because it matched the wrong RPM file. Fix this by just directly
referencing the full path name since we can.

Closes: #1491
Approved by: cgwalters
2018-08-07 16:00:13 +00:00
Jonathan Lebon
dce98e4a53 tests/compose: Split out machineid-compat test
My fix to the testsuite in #1488 in which I made the `machineid-compat`
test part of `test-basic.sh` wasn't correct since the basic tests in
`libbasic-test.sh` also check that the default behaviour without the
`machineid-compat` option is to include it.

Let's just do this right and split out the `machineid-compat` test into
its own run.

Closes: #1491
Approved by: cgwalters
2018-08-07 16:00:13 +00:00
Jonathan Lebon
391a684b5f tests/compose: Fix run-test.sh regression
Regression from hasty hack in #1488. We want to return nonzero if the
test failed so that `parallel` fails too.

Closes: #1491
Approved by: cgwalters
2018-08-07 16:00:13 +00:00
Jonathan Lebon
e94f8c9b5f compose: Fix mutate-os-release handling
I noticed that the latest Fedora Atomic Host 28 and Silverblue did not
have an `OSTREE_VERSION` line in `/etc/os-release` even though both
specified `mutate-os-release` in their manifests. This turned out to be
due to the fact that `/usr/lib/os-release` is now a symlink to a
variant-specific file (e.g. `os-release-atomichost`), so we would
fallback to mutating `/usr/lib/os.release.d/os-release-fedora` instead.

Fix this by just taking the nuclear option of running `realpath` in the
rootfs directly. This is more maintainable than trying to keep up with
changes in variants/naming/etc. There's related discussions to this in
the original [PR](https://github.com/projectatomic/rpm-ostree/pull/410)
which introduced the feature re. resolving symlinks within the rootfs.

Closes: #1481
Approved by: jlebon
2018-08-06 01:32:49 +00:00
Jonathan Lebon
5219df0875 libpriv/bwrap: Add bwrap_run_captured()
The `GSubprocess` API has a great `g_subprocess_communicate` function
that makes it easy to capture stdout and/or stderr. Let's expose that.

Closes: #1481
Approved by: jlebon
2018-08-06 01:32:49 +00:00
Jonathan Lebon
1c899c5e9e compose-tests: Rename dir based on failure
Should make it easier to tell which compose test failed rather than
going through the verbose output of parallel.

Closes: #1488
Approved by: jlebon
2018-08-06 00:33:56 +00:00
Colin Walters
588a0327db postprocess: Error if units with machineid-compat: false
Rather than silently ignoring it.

In theory...we could write to /usr/lib/systemd instead of `/etc`
but eh...I feel like what we really want to do is make it convenient
to write a preset file from the YAML.

(We could have an `add-files` content that takes values literally
 which would be nice in YAML and suck in JSON)

A general thread running through this is that for people making
*derivatives* of a CoreOS-like system, having to create their
own `exampleos-release` package is an annoying hurdle.

Anyways for now we're fixing the bug that we were silently ignoring
it.

Closes: #1488
Approved by: jlebon
2018-08-06 00:33:56 +00:00
Jonathan Lebon
be055179db docs/treefile: Mention rw in for machineid-compat option
See discussions in https://github.com/systemd/systemd/issues/8268 for
more information.

Closes: #1487
Approved by: cgwalters
2018-08-03 16:54:55 +00:00
Colin Walters
344aee1d76 rust: Add support for inline rojig spec files
The rojig spec is almost entirely rpm-ostree implementation details;
let's not have lots of people fork/duplicate it.  Rather add the bits
of rojig to the treefile that people need to define (most notably
the name).

Prep for stabilizing rojig.

I had a few false starts with this PR; managing ownership/lifetimes
across C/Rust is just complicated.  I got bit hard by the fact that
the workdir in `--unified-core` is really dfd-relative, and had to
do a dance to propagate the dfd into rust, as well as down into
the rojig builder.

Closes: #1484
Approved by: jlebon
2018-08-03 16:54:47 +00:00
Jonathan Lebon
a6e4994bf6 app/compose: Fix g_propagate_error ownership
We weren't actually transferring ownership of the error, which meant
that the error message was freed before we could even print it to the
user.

Closes: #1486
Approved by: cgwalters
2018-08-01 23:27:12 +00:00
Colin Walters
dcbd2d91ec compose: Stop calling fchdir in --unified-core mode
Changing the process' working directory is evil in potentially
multi-threaded code, and at this point we really should be using
`openat()`/fd-relative bits most everywhere.  But let's be
conservative and only stop doing it in `--unified-core` mode.

Closes: #1485
Approved by: jlebon
2018-08-01 22:22:47 +00:00
Colin Walters
23badcd288 compose: Write a preset file to enable ostree-remount.service
I've lost count now of how many times people have hit variants
of https://github.com/projectatomic/centos-release-atomic-host-devel/pull/6
Let's just bake it in.

Closes: #1482
Approved by: jlebon
2018-08-01 20:11:38 +00:00
Colin Walters
628a3aa22f tests/compose: Write logs directly
Make logging work the same as it does for the vmcheck-STI work
(at some point I'll try to unify the 3 parallel+script implementions
 we have).  This fixes the problem that when the test times out,
the filename won't have `.txt` and S3 won't have the right MIME type.

Closes: #1479
Approved by: jlebon
2018-08-01 18:05:09 +00:00
Colin Walters
800402b00a tests/compose: Add some time logging
To help us profile.

Closes: #1479
Approved by: jlebon
2018-08-01 18:05:09 +00:00
Colin Walters
da27b94b29 core: Use new rpmtsSetVfyLevel() API for writing rpmdb
Newer librpm defaults to doing a full payload checksum, which we can't
do at this point (writing the db) because we imported the RPMs into
ostree commits, saving just the header in metadata - we don't have the
exact original content to provide again.

Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1607223

Closes: #1469
Approved by: jlebon
2018-07-31 19:22:30 +00:00
Colin Walters
261dd99a69 Update libdnf, disable Python bindings
This entirely drops out Sphinx as well as python-devel from our
builds, makes builds faster, and silences a lot of warnings too.

Update submodule: libdnf

Closes: #1480
Approved by: jlebon
2018-07-31 17:13:50 +00:00
Colin Walters
3446caa190 importer: Prefix error with package name
I think we have issues with librepo not handling being forcibly
interrupted; I saw an lzma error from what I think was a partial
download.

Let's add an error prefix here so this is easier to debug.  I
think we used to have a prefix but lot it when we moved to async.

Closes: #1477
Approved by: jlebon
2018-07-31 15:18:45 +00:00
Jonathan Lebon
67860c5ece app/pkg-builtins: Add --unchanged-exit-77
This is analogous to `upgrade --upgrade-unchanged-exit-77`, but for
`install`/`uninstall`. This way, one can determine whether the command
truly had an effect on the default deployment or not.

Since this works by comparing against the previous default deployment,
this works correctly even if we already had a pending deployment at the
beginning of the transaction.

Closes: #1478
Approved by: cgwalters
2018-07-31 08:53:15 +00:00
Jonathan Lebon
d35fbb665e app/pkg-builtins: Add --idempotent
Add a new `install/uninstall --idempotent` option to make it easier to
interact with the CLI through scripts. E.g. one doesn't have to check
first if a request has already been installed/uninstalled.

Closes: #1467

Closes: #1478
Approved by: cgwalters
2018-07-31 08:53:15 +00:00
Jonathan Lebon
a9e1de0276 daemon/dbus: Document a few missing newer options
Closes: #1478
Approved by: cgwalters
2018-07-31 08:53:15 +00:00
Jonathan Lebon
fcb061b19a app/status: Add --pending-exit-77 switch
This makes it easier for scripts to determine whether there is a pending
deployment instead of using `--json/--jsonpath`.

Closes: #1478
Approved by: cgwalters
2018-07-31 08:53:15 +00:00
Colin Walters
16be1a0bad rust: Rework treefile to be an object
In a later patch I'm going to add more API; basically rather
than doing the JSON parsing from C, we can add APIs to directly
access the treefile object.  This also demonstrates how we
can do more extensive APIs, in particular implement an "object"
in Rust.

The ownership across the FFI boundary becomes nicer here too,
we don't need to do a dance with the fd.

For writing this I found
http://jakegoulding.com/rust-ffi-omnibus/objects/
quite useful, as well as
https://github.com/rust-lang/regex/blob/master/regex-capi/src/rure.rs

Closes: #1474
Approved by: jlebon
2018-07-30 18:54:48 +00:00
Colin Walters
5e5cdb4e0d rust: Add ptr error mapping API
Builds on the previous commit.

Closes: #1474
Approved by: jlebon
2018-07-30 18:54:48 +00:00
Colin Walters
ba28adf337 rust: Rework GError mapping API
I was going to add a new API that returns a pointer, and that just
wasn't going to work with the trait AFAICS; we'd need to be generic
across many different types.  Let's use plain functions; it's slightly
less elegant but is clear and works.

Closes: #1474
Approved by: jlebon
2018-07-30 18:54:48 +00:00
Colin Walters
716b60b7d7 scripts: Add Lua overrides for fedora-release and also -workstation
Dusty hit this when working on Fedora CoreOS.  IMO we don't need
to do the same for a future e.g. `fedora-release-silverblue` as
we don't support someone installing it on a non-ostree system.

(Note this all only really matters for `--unified-core` on the compose
 side)

Closes: #1473
Approved by: jlebon
2018-07-30 13:12:11 +00:00
Jonathan Lebon
7911b14f49 daemon: Fix cached-update including no-op diffs
The `cached-update` variant would mark a bunch of RPMs as upgraded even
if they didn't actually change. The issue turned out to be we were doing
the diff all wrong in the staged deployment case. I'm not sure what I
was thinking in #1344, but essentially, we were marking all layered RPMs
in the staged deployment as updates instead of only marking those
layered RPMs which were actually changed EVR.

We just simplify the approach here by directly doing a pkglist diff
between the booted and staged deployments and consuming that. That's
really all there is to it! Reduces the code quite a bit too.

Closes: #1446

Closes: #1455
Approved by: cgwalters
2018-07-28 06:53:40 +00:00
Jonathan Lebon
a17b4b9be0 tests/utils: Drop empty inject-pkglist.py
This one snuck in as part of #1205. I had initially started writing
`inject-pkglist.c` in Python and somehow inadvertedly made it part of
a commit.

Closes: #1455
Approved by: cgwalters
2018-07-28 06:53:40 +00:00
Colin Walters
831507b0f0 scripts: Add RPMOSTREE_SCRIPT_TRACE debugging envvar
I used `env RPMOSTREE_SCRIPT_TRACE='strace -f -s2048' rpm-ostree compose ...`
to debug https://bugzilla.redhat.com/show_bug.cgi?id=1548050

Closes: #1472
Approved by: jlebon
2018-07-27 13:09:49 +00:00
Colin Walters
e858a30eca compose: Stabilize --unified-core
In line with the recent trend of marking things stable, and in
preparation for stabilizing `rojig://` - Let's stabilize the `--unified-core`
option for `compose tree`.

I'm not sure we could make it the default anytime soon; today it trips
over bugs in the PAM package in RHEL7 for example.  But it
works fine for Fedora, and I think the code/design are good enough to be stable.

Closes: #1465
Approved by: jlebon
2018-07-24 23:06:50 +00:00
Colin Walters
fa29f7acfa compose: Support arch-specific packages in YAML (and in JSON again)
Follow up to: https://github.com/projectatomic/rpm-ostree/pull/1459

We now honor arch-specific packages in YAML, and reject unknown
architectures.  I looked a little bit at how to avoid having hardcoded
arch lists, but it doesn't seem worth it right now.

Closes: #1468
Approved by: jlebon
2018-07-24 22:05:06 +00:00
Jonathan Lebon
1a82f210ae ci: Bump f28-compose by 10 minutes
We're consistently seeing timeouts on the `f28-compose` test. It's
likely due to a performance regression in the backend rather than
anything else. I filed a ticket to get an investigation going, though
for now at least, let's bump the timeout some more to keep patches
flowing.

Closes: #1463
Approved by: jlebon
2018-07-24 20:15:10 +00:00
Jonathan Lebon
c016509548 libdnf: Bump and disable html and man pages
Skip building man pages and HTML docs for our embedded libdnf to speed
up builds.

This bump also pulls in a fix to ensure we never try to install src
packages from `dnf_context_install()`.

See: https://github.com/projectatomic/libdnf/pull/3

Update submodule: libdnf

Closes: #1463
Approved by: jlebon
2018-07-24 20:15:10 +00:00
Jonathan Lebon
b66337e0cb ci: Pepper in date markers to help debug timing
This should help out with determining what steps take the most time.

Closes: #1459
Approved by: cgwalters
2018-07-21 14:43:48 +00:00
Colin Walters
6ac6f3d086 treefile.rs: Deny unknown fields by default
Let's not make the same mistake we did with JSON where typoing a
field means it's silently ignored.  This actually caught a bug
in a YAML usage we had:

```
error: Failed to load YAML treefile: unknown field `install_langs`, expected one of ... `install-langs` ...
```

Yes, this is a compatibility break with the feature we just announced
but...I seriously doubt anyone (that isn't known to me) has converted
yet, and if they are excited enough to start using a two-week-old feature
they can adjust.

Closes: #1459
Approved by: cgwalters
2018-07-21 14:43:48 +00:00
Colin Walters
1b4a73f071 tests: Make two more tests compatible with deployment staging
Unlike the kernel args one, these two were simple.

Closes: #1458
Approved by: jlebon
2018-07-21 13:58:21 +00:00
Colin Walters
0cc002fd8d build-sys: Use python3 for libdnf by default if available
Probably at some point libdnf will drop py2 support, but the
main reason I'm doing this is avoids a python2 dependency
for rpm-ostree for distributions that don't want that.

Note of course rpm-ostree itself doesn't use python, libdnf does.
And only for the python bindings, which we don't use either.  So
this is mostly just to DTRT automatically for the libdnf bits; down
the line we could probably add a patch to make the python fully
conditional.

Closes: #1460
Approved by: jlebon
2018-07-20 22:27:34 +00:00
Jonathan Lebon
ec19d67d74 ci: Stop rebasing Fedora onto updates ref
We can drop this hack now since there have been updates since GA. Also
add a note that trying to drop the rebase to continuous causes
`layering-relabel` to fail.

Closes: #1406
Approved by: jlebon
2018-07-20 18:47:51 +00:00