Commit Graph

2084 Commits

Author SHA1 Message Date
Colin Walters
5037c85330 tests: Extend staging to cover vmcheck/upgrades
This now works after https://github.com/ostreedev/ostree/pull/1570

However, I was hitting an issue with the temp httpd running the
tests serially, let's split them up.

Really the only sane thing is to run each of these tests in a fresh
VM; we'll do the VM-in-container pattern here hopefully soon.

Closes: #1355
Approved by: jlebon
2018-05-02 19:20:26 +00:00
Colin Walters
0dfe0de0e0 daemon: Add some error prefixing around reloading
I was hitting an error during sysroot loading while playing
with deployment staging, and it wasn't initially
clear which phase was causing it (a client-initated reload versus
one we do before or after deploying).  Add some error prefixing.

Closes: #1355
Approved by: jlebon
2018-05-02 19:20:26 +00:00
Colin Walters
506910d930 Add an experimental option to use libostree's "staging" API
Now that infrastructure for this has landed in libostree,
let's make it easy for people to opt-in to testing it.  This is a distinct first
step for adding it as an update policy.

Closes: #1352
Approved by: jlebon
2018-05-02 13:46:37 +00:00
Micah Abbott
b12899a7f2 man/rpm-ostree: --check always updates RPM metadata
Add a short note explaining that the RPM metadata is always refreshed
when using the `--check` flag.

Closes: #1353
Approved by: jlebon
2018-05-01 14:25:08 +00:00
Colin Walters
30e29dfc0e syscore: Only do a single repo prune for base+pkgcache
Since we unified the pkgcache repo in 7056e6b726
there's no longer a reason to perform two repository prunes.

Change things so that the first phase is regenerating all of the
refs (in a single libostree txn), then perform a single prune.

This is preparation for reworking how we do prunes, which is
going to be useful for staged deployments.

Closes: #1351
Approved by: jlebon
2018-04-30 19:01:30 +00:00
Jonathan Lebon
ee458c3c50 daemon/upgrade: Write out new cached update
Right now, cached updates generated during "check" policy runs are
completely decoupled from upgrade operations. This can lead
to the surprising situation where the "Available update" is *older* than
a freshly deployed pending tree with `rpm-ostree upgrade`.

We should just generate a cached update after upgrade operations. This
is also prep for staged deployments, where we'll want to do this as
well.

Note that we write out the cached update here even if automatic updates
are turned off since it's essentially free. I've been thinking about
always displaying that information after an `rpm-ostree upgrade` in
`status`. Though not sure if we should keep it in a separate "Available
update" section, or somehow morph it as part of the pending deployment
output.

Closes: #1344
Approved by: cgwalters
2018-04-26 21:16:52 +00:00
Colin Walters
8726be65e3 core: Finish implementing `override replace ./kernel*.x86_64.rpm
Previously we merged: #1228 AKA 12dc565b00
My recollection is that was working on it the background, while doing
something else, and I clearly didn't get to the point of testing it "for real".

There are many interlocking issues here to make this work.  For example,
the "remove RPM" logic needs special handling for the kernel, because
we also inject content into `/usr/lib/ostree-boot` and also generate
the initramfs, etc.

The architecture I chose is to have the core *detect* when a kernel
is changed, and also call into the kernel processing code when removing
a kernel package.  But the logic for doing kernel reinstallation client-side
is best alongside the initramfs generation logic which already existed
in the sysroot upgrader.

I extended the test suite to cover what was failing before, and I
tested this interactively.  But I'm uncertain about adding a test
for actually *booting* into the GA kernel as it's quite possible
some bits in userspace rely on a newer kernel.  Fixing this properly
really wants some infrastructure to better "re-version" an existing
package without changing its content.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1334

Closes: #1346
Approved by: jlebon
2018-04-26 16:49:53 +00:00
Colin Walters
9df2428a1a tests: Move upgrade/rebase tests from unit to vmcheck/test-upgrades
Now we stop running rpm-ostreed as non-root, which is going to be
a maintenance pain going forward.  If we do introduce non-VM based
tests I think we should look to doing in-container testing.

Closes: #1339
Approved by: cgwalters
2018-04-25 20:28:56 +00:00
Colin Walters
a50a6cb959 tests: Move more update-generation bits to libvm.sh
The autoupdate test had a lot of useful infrastructure.  Prep
for removing `tests/check/test-upgrade-rebase.sh`.

Closes: #1339
Approved by: cgwalters
2018-04-25 20:28:56 +00:00
Colin Walters
39186e3330 docs/compose: Mention caching, privileged containers, example code
And also drop the `-z2` I saw show up in another project.

Closes: #1347
Approved by: jlebon
2018-04-25 18:43:36 +00:00
Colin Walters
1521dc32c9 status: Show rpmmd-repo data in verbose mode
Building on:

 - 9cbec27d4c
 - e7a42f70a9

I was looking at a rpm-ostree run that imports a variety of rpmmd-repos,
and information about the source repositories is really useful for determining
the up-to-dateness.  We've been capturing this data for a while, it's
about time we started showing it somewhere.

This does make `status --verbose` notably more verbose, but eh, that
seems fine for now.

See also https://github.com/projectatomic/rpm-ostree/issues/774

Closes: #1345
Approved by: jlebon
2018-04-25 14:52:23 +00:00
Colin Walters
e34f17fc3e core: Don't rm-rf directories when doing overrides
It's very normal for base packages to own directories with
dependent packages installing files there.  Doing an rm-rf
for directories was just wrong.  Concretely this fixes
an `override replace ./systemd-*.rpm`.

librpm is also pretty conservative here (for good reason)
and just ignores `ENOTEMPTY`, so let's match that.

I opted to split things up so we remove not-directories
in a first pass, then remove all directories we can in the
second.  This should maximize our chances of removing what we can
in a scenario where e.g. two co-dependent packages install files
to a directory one of them owns.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1340

Closes: #1341
Approved by: jlebon
2018-04-20 16:07:44 +00:00
Colin Walters
4c3d9cdbce tests: Split unit test-basic.sh into introspection + upgrade-rebase
Following up to: https://github.com/projectatomic/rpm-ostree/pull/1336

It makes sense to keep the library tests as unit tests (although
we should also support doing them installed).

The upgrade-rebase tests will move into vmcheck/ soon.

Closes: #1338
Approved by: jlebon
2018-04-17 13:08:24 +00:00
Colin Walters
1c8c755e81 tests: Split test-basic into misc-{1,2}
Our test suite originated when package layering was still being
developed, but now that that's mature, the logic where layering
tests are distinct makes less sense.

The `basic` test had grown to really be a collection of many
miscellaneous things.  Let's make that more explicit.  Further,
let's avoid having each test suite grow too large; when a single
test fails we don't have an easy way to rerun just that test,
so a crude way to have faster local iteration is to split into groups.

My plan is to reintroduce a `basic` test that covers the basics
of all functionality - update, deploy, layering, etc.  The advanced/corner
cases of layering like the `rm -rf /` test would still live in a
`test-layering.sh` or so.

Closes: #1336
Approved by: jlebon
2018-04-16 17:53:20 +00:00
Colin Walters
73fa64eb12 tests: Move ostree-update creation functions into libvm.sh
So they can be reused outside of just the autoupdate test.

Closes: #1336
Approved by: jlebon
2018-04-16 17:53:20 +00:00
Colin Walters
4f42b94da7 tests: Migrate some basic unit tests that start daemon to vmcheck
The unit tests run an rpm-ostree daemon as non-root, which worked
surprisingly well for quite a while.  But it started failing when
working on a patch which adds caching that writes to `/var`.

Since we have the vmcheck system now, let's switch over to that.
This PR moves the random "basic" tests we'd accumulated like
one to verify `StateRoot` is only in `status --verbose`, but not
the tests for the `rebase` command etc.

Closes: #1336
Approved by: jlebon
2018-04-16 17:53:20 +00:00
Matthew Leeds
b11d1fc1e3 README.md: Fix youtube link
Closes: #1335
Approved by: cgwalters
2018-04-14 15:24:54 +00:00
Colin Walters
0d0b9c85ee daemon: Simplify how we load deployments
I was going to add a `StagedDeployment` property and
found the code here confusing in the way we were walking
the whole deployment list.  Let's use the
`ostree_sysroot_query_deployments_for()` API for all
cases here.  Then the special case is just:
"if no booted deployment, default == pending".

Also change the code style to declare-and-initialize.

Prep for staging deployments.

Closes: #1327
Approved by: jlebon
2018-04-11 21:00:36 +00:00
Colin Walters
2b624acff0 core,scripts: Move creation of /var/tmp into core
In prep for cleaning up how we manage mounts.  If we're looking
at a real existing `/var`, then it must have the directory.  The
only case where we don't is in pkglayering, so move the special
case there.

Prep for https://github.com/projectatomic/rpm-ostree/issues/1329

Closes: #1333
Approved by: jlebon
2018-04-11 13:09:17 +00:00
Colin Walters
abcaf42d55 compose: Add environment variable to skip tmpdir cleanup
This was useful for me to directly invoke bwrap against the target
rootfs.

Part of working on: https://github.com/projectatomic/rpm-ostree/issues/1329

Closes: #1333
Approved by: jlebon
2018-04-11 13:09:17 +00:00
Colin Walters
e38c64c330 bwrap: Split out a "base" constructor
The bwrap code has some "opinionated" setup around e.g. /etc and
such that I'd like to centralize even more.  However, the dracut
case of taking the host's `/etc` is the unusual standout.  Let's
split out a base constructor.

Prep for https://github.com/projectatomic/rpm-ostree/issues/1329

Closes: #1333
Approved by: jlebon
2018-04-11 13:09:16 +00:00
Colin Walters
4be214a978 bin/status: Factor out deployment printing into helper function
Exactly like: https://github.com/ostreedev/ostree/pull/1504

Prep for staged deployments, where it's a new deployment that isn't
in the deployment array (since it's not in the bootloader order).

Closes: #1328
Approved by: jlebon
2018-04-09 18:48:50 +00:00
Jonathan Lebon
60a59cbbe4 vmcheck/override-replace: Fix status jq test
We should be checking that there are no requested base replacements
here, not removals. This was likely a copy-paste slip-up from copying
the override-remove tests. I still kept those checks for sanity, and
just tacked on the correct ones.

Closes: #1323
Approved by: cgwalters
2018-04-05 15:26:46 +00:00
Jonathan Lebon
bb07a29bc2 overrides: Allow resetting inactive overrides
This fixes a small regression from #852 which prevented inactive
overrides to be reset. Which is funny, because that's exactly the most
likely time when you would want to reset an override.

Basically, the `!is_layered --> no overrides` doesn't make sense for
inactive overrides. I suspect most people worked around this by just
using `reset --all`.

Closes: #1323
Approved by: cgwalters
2018-04-05 15:26:46 +00:00
Jonathan Lebon
cb99fb44b2 buildutils: Add libglnx.m4 to .gitignore
Not sure why this one isn't being auto-generated.

Closes: #1323
Approved by: cgwalters
2018-04-05 15:26:46 +00:00
Colin Walters
d25588722b ci: Ensure we exclude ostree from base fedora too
rdgo uses an earlier git tag due to branches, until we have proper
repo priorities that don't care about versions, let's do this
hack.

Same thing as https://pagure.io/fedora-atomic-host-continuous/c/a7145410fbc7e73188fde449628fa0343b05e4a4?branch=master

Closes: #1326
Approved by: jlebon
2018-04-05 15:26:23 +00:00
Jonathan Lebon
28a00f8a13 daemon: Factor out function for retrieving DnfSack
Factor out a helper function to create a `DnfSack` for the booted
deployment. Prep for more work.

Closes: #1324
Approved by: cgwalters
2018-04-04 20:48:39 +00:00
Jonathan Lebon
29826564d4 daemon: Create cache dir before writing cached update
If the user never ran e.g. `rpm-ostree install`, then the cache dir will
not even exist yet. Let's make sure it's created before writing the
cached update into it.

Closes: #1324
Approved by: cgwalters
2018-04-04 20:48:39 +00:00
Jonathan Lebon
3a1fd2c139 tests/libvm: Support SSH_CONFIG env var
When juggling multiple test VMs for different purposes, it's useful to
be able to easily e.g. `make vmcheck` a specific one by overriding the
ssh-config file to use, rather than editing the latter each time.

Closes: #1324
Approved by: cgwalters
2018-04-04 20:48:39 +00:00
Jonathan Lebon
bd2920ad01 daemon/upgrader: Return new deployment in deploy()
Enhance `rpmostree_sysroot_upgrader_deploy` to also return the newly
created deployment on success. Prep for more work.

Closes: #1324
Approved by: cgwalters
2018-04-04 20:48:39 +00:00
Colin Walters
2b7f33ca9d core: Bound number of async imports by CPUs
I was doing a `--ex-unified-core` compose of FAW, and started hitting
the fd limit (1024), since we created hundreds of threads.  Since
it doesn't make sense to do that, let's bound the number of concurrent
imports by the number of CPUs.

This is another implementation of the "bounded async" pattern, like
c18628ecb8

Closes: #1317
Approved by: jlebon
2018-04-03 14:39:24 +00:00
Colin Walters
defa1dc38a core/scripts: Support /var/lib/rpm-state
I was trying a `--ex-unified-core` compose of FAW, and things fell over
on `urw-base35-fonts` which does a dance of setting a stamp file in
`%post` and checking it in `%posttrans`.

This whole pattern should be considered deprecated by file triggers. But let's
support it for now.

Note there's a lot of parameter passing as we need a single directory which is
held across multiple script invocations.

Closes: #1319
Approved by: jlebon
2018-03-28 18:37:17 +00:00
Colin Walters
f05b980094 build-sys: Mark libpriv symbols as private
This was caught by the abicheck in Fedora; since we were building with default
visibility for `librpmostreepriv.la` which was linked statically into the public
library, we'd end up with lots of internals as public ABI.

Fix this by using `-fvisibility=private` for the libpriv build and for good
measure elsewhere so we remember to use it by default.

Closes: #1320
Approved by: jlebon
2018-03-28 15:18:11 +00:00
Colin Walters
84f09172e4 core: Prefix error when loading commit for cache branch
Somehow my dev container's pkgcache repo lost a commit.  Prefix
the error so this is easier to debug.

Closes: #1314
Approved by: jlebon
2018-03-26 18:27:38 +00:00
Micah Abbott
0399e185e4 docs: add additional details about 'tmp-is-dir'
Expanded the details around using 'tmp-is-dir' and what happens when
it is used.

Closes: #1315
Approved by: dustymabe
2018-03-26 17:26:54 +00:00
Jonathan Lebon
835c3ad303 Release 2018.4
It's been more than a month, let's get a release out since the tree is
in a fairly good condition now.

Closes: #1313
Approved by: cgwalters
2018-03-26 14:13:35 +00:00
Colin Walters
779f764fcd app: Call Reload right after getting sysroot proxy
A final followup to our pile of "add reload" PRs.  This one is
necessary to fix the `status` race.

Closes: #1312
Approved by: jlebon
2018-03-26 13:05:40 +00:00
Jonathan Lebon
706506b2d1 app/dbus: Always call Reload() after transaction
Specifically in this case, this allows us to close a race condition
during `upgrade --check` where the `CachedUpdate` property might not
have been updated yet when we read it after finishing the transaction.

Closes: #1311
Approved by: cgwalters
2018-03-23 21:50:33 +00:00
Jonathan Lebon
37306e36ee daemon: Add new Reload D-Bus method
Add a new `Reload` method as a softer alternative to `ReloadConfig`.
We'll also make use of this on the client side to sync with the daemon.

Closes: #1311
Approved by: cgwalters
2018-03-23 21:50:33 +00:00
Jonathan Lebon
747236d593 daemon: Rename sysroot_reload to be more descriptive
Follow-up to previous commit. Since we have so many concepts called
"sysroot" and "configs", let's be explicit with our naming here to be
sure we know what we're calling.

Closes: #1311
Approved by: cgwalters
2018-03-23 21:50:33 +00:00
Colin Walters
60aa9a4a33 daemon: Automatically reload sysroot before txn
In this PR: https://github.com/projectatomic/rpm-ostree/pull/1309
I was hitting race conditions running `ostree admin pin` then
`rpm-ostree cleanup` as it was possible that the daemon hadn't handled
the inotify on the sysroot and reloaded the deployment state before
the txn request came in.

Close this race by doing an implicit `reload` before starting a txn.
This is a pretty efficient operation because for the sysroot we're
just doing a `stat()` and comparing mtime.

Implementation wise, change the external API to drop the "did change"
boolean as nothing outside of the `sysroot.c` file used it.

A followup to this would be changing the `status` CLI to call a
(new) DBus API like `RequestReload` that at least did the sysroot
reload if the daemon was otherwise idle or so?  And it'd be available
to unprivileged users.

Closes: #1311
Approved by: cgwalters
2018-03-23 21:50:33 +00:00
Colin Walters
9ddf65aade daemon: Retain pinned deployments
Followup to: https://github.com/ostreedev/ostree/pull/1464

Ideally, we'd delegate more logic around these things to libostree, but we're
not there yet.

This means e.g. `rpm-ostree cleanup -r` for a pinned rollback will just silently
skip it.  It'd be nicer to emit an error probably, but that'd be quite a bit
more work.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1293

Closes: #1309
Approved by: jlebon
2018-03-23 13:25:33 +00:00
Jonathan Lebon
4291500368 daemon: Also generate CachedUpdate in DownloadUpdateRpmDiff
Moving to caching the GVariant to disk rather than during RPMOSTreeOS
reloads broke the legacy `DownloadUpdateRpmDiff` path because it relied
on the implied recalculations that occur on reloads to update
`CachedUpdate`.

This patch series was initially going to be about just migrating all the
legacy APIs to make use of the new metadata, which would have fixed this
properly. But we first need some real coverage in that aread, which is
very poor right now. I'd like to investigate integrating the Cockpit
tests (at least the ostree-specific parts) into our CI to remedy this.

Anyways, for now at least, let's fix Cockpit.

Closes: #1300

Closes: #1303
Approved by: cgwalters
2018-03-17 20:17:32 +00:00
Jonathan Lebon
e4dd2c4d23 daemon: Write to journal on successful transaction
We would write to the journal when we initiated a new transaction as
well as when that transaction errored. This just completes the picture
by also writing when the transaction finished successfully. My ulterior
motive is to make testing easier.

Also make sure to print the method name, which is more telling than the
object path.

Closes: #1303
Approved by: cgwalters
2018-03-17 20:17:32 +00:00
Jonathan Lebon
bc97c3aa53 app/db-list: Use pkglist metadata when possible
This patch teaches `db list` to also use the pkglist metadata when it
can, just like we did for `db diff`. To spell it out: this then allows
`db list` to work on commits for which we only have the commit object.

I went for the surgical incision here and didn't try to support
invocations which use fnmatch patterns for now. Definitely possible,
though it didn't feel like it was worth the effort given that the common
case is just a raw `db list` (I'd wager most people are probably
hard-wired to pipe to `grep` anyway for filtering).

Also fix the usage string, which had the arguments flipped.

Closes: #1299
Approved by: cgwalters
2018-03-15 15:49:44 +00:00
Jonathan Lebon
d6172ddee2 app/db-diff: Add --base switch
Support a `--base` switch to perform the diff on the base layer if
deployment checksum was locally composed. This is useful to filter out
from the diff changes due to layered packages.

Closes: #1299
Approved by: cgwalters
2018-03-15 15:49:44 +00:00
Jonathan Lebon
4e99052d07 Fix GitHub issue template formatting
The extra spaces were preventing proper bolding.

Closes: #1306
Approved by: cgwalters
2018-03-14 21:54:16 +00:00
Jonathan Lebon
45fb100826 daemon/deploy: Factor out update variant generation
No functional change. Prep for next patch.

Closes: #1304
Approved by: cgwalters
2018-03-14 21:49:16 +00:00
Jonathan Lebon
2eb424b9ce vmcheck: Add vm_shell_inline
Initially suggested in #1298. Wanted to try this out for an upcoming
patch.

Closes: #1304
Approved by: cgwalters
2018-03-14 21:49:16 +00:00
Jonathan Lebon
3e39fda4aa vmcheck: Make vm_ansible_inline more verbose
Because otherwise, there's no way to see the output of the script.

Also, turn off `gather_facts` since in the majority of cases, we don't
need it, so let's avoid the overhead. We can make this an opt-in flag
later on if needed.

Closes: #1304
Approved by: cgwalters
2018-03-14 21:49:16 +00:00