Commit Graph

1968 Commits

Author SHA1 Message Date
Colin Walters
c5939b7ca7 tests: Add RPMOSTREE_TEST_NO_OVERLAY to skip overlay
The use case here is to run our tests against the shipped tree;
this could be used by a-h-t for example:
https://github.com/projectatomic/atomic-host-tests/issues/74

I tried this with just `TESTS=basic` and it failed for `usroverlay`. So we'd
have to start adding feature detection to the test suite to make this truly
useful, but let's at least start with the basic bits now to play with it.

Closes: #1251
Approved by: jlebon
2018-02-14 20:00:16 +00:00
Colin Walters
f669e5157c jigdo: Clean up client update output a bit
I added the "commit already written" print for the very special case
when doing a `rebase --experimental rojig://` and the target jigdoRPM
is for the same commit you're on.  The semantics today of that are
weird because it's the *next* update where you download all the RPMs...

Anyways that case aside, let's only print when something did change,
and drop the checksum (same rationale as status PR) and also drop
the "from %u packages" since one sees that in the downloader.

Closes: #1244
Approved by: jlebon
2018-02-14 19:32:58 +00:00
Jonathan Lebon
8b72dcbde2 tests/libtest: Fix bash syntax error
I think the logic there just worked because we don't actually use
`noarch` RPMs in the vmcheck suite.

Closes: #1248
Approved by: cgwalters
2018-02-14 19:02:02 +00:00
Jonathan Lebon
57ab9844b9 daemon/autoupdates: Don't insert rpm-diff if empty
Canonicalize to "absent" to make it easier for clients.

Closes: #1248
Approved by: cgwalters
2018-02-14 19:02:02 +00:00
Colin Walters
f0e381392e README.md: Add arch diagram, tweak wording, add talk links
There's a lot more we could do here but I think the architecture
diagram in particular helps a lot.

Closes: #1245
Approved by: jlebon
2018-02-14 17:36:29 +00:00
Colin Walters
12dc565b00 core: Set installonly to ensure we only have one kernel
The goal here is to support `override replace kernel.x86_64.rpm`.

There's a whole lot of logic in libdnf to support having multiple
kernel packages installed.  AIUI, that was implemented because:

1) The kernel is the biggest source of regressions
2) It'd be quite noticeable if all of your kernel modules were deleted

Of course point 2) applies to a lot of userspace too...that's something
rpm-ostree fixes of course.

Anyways, in some testing all we need to do really is just turn that
logic off unconditionally.

Closes: https://github.com/projectatomic/rpm-ostree/issues/946

Closes: #1228
Approved by: jlebon
2018-02-14 16:43:08 +00:00
Colin Walters
40b010a7ea util: Initialize sack pointer to NULL
Otherwise we'll be freeing uninitialized memory on error (this happened in CI
somehow).

Closes: #1246
Approved by: jlebon
2018-02-14 16:17:52 +00:00
Jonathan Lebon
99aa6f89d2 ci: expand rootfs on CentOS
The new kernel tests require a little bit more space.

Closes: #1247
Approved by: cgwalters
2018-02-14 16:16:52 +00:00
Jonathan Lebon
75f4636320 core/commit: clean up rpmdb leftover files
While debugging the recent `BDB0087 DB_RUNRECOVERY` issues that cropped
up recently, I came upon the fact that we're leaving leftover rpmdb
files in the rootfs on client-side assemblies. Let's clean those up too.

We do this in the `commit` phase, *after* the final time we access the
rpmdb to generate the pkglist.

Closes: #1247
Approved by: cgwalters
2018-02-14 16:16:52 +00:00
Colin Walters
0a4f6bdab4 jigdo: Use separate rpmostree/jigdo/... refs
The way we import packages in jigdo mode is different from package layering; we
may only import a subset of files for example. In general, we need to treat
jigdo differently.

Related: https://github.com/projectatomic/rpm-ostree/issues/1197

Closes: #1238
Approved by: jlebon
2018-02-13 20:08:13 +00:00
Colin Walters
cd9e0c8ec7 util: Don't do libsolv cache for temporary sacks
We should really only generate a cache for the current root's rpmdb; this
function can be used for e.g. `db diff` and it doesn't make sense to generate a
libsolv cache for arbitrary commits since it's unlikely they get reused.

Closes: #1238
Approved by: jlebon
2018-02-13 20:08:13 +00:00
Colin Walters
782c49b748 upgrader: In jigdo mode, don't check rpm-md repos twice for layers
Typing `rpm-ostree upgrade` was quite verbose with layered packages, we'd see
the rpm-md repos twice. The better fix would be to pass the context/sack from
one stage to the other, but this is a quick simple fix to at least reduce
verbosity (and potentially avoid extra network requests).

Closes: #1241
Approved by: jlebon
2018-02-13 18:24:47 +00:00
Jonathan Lebon
aa87fe7d1a app/status: also check for timer state
I spent quite some time trying to figure out why the last run timestamp
of the autoupdate service wasn't showing up in `status`. It turned out
to be that systemd only keeps that information around if an associated
timer unit is active.

Thinking more, I think it makes sense to split out the case where the
timer isn't even running explicitly, rather than just reporting "no runs
since boot".

Closes: #1242
Approved by: cgwalters
2018-02-13 17:30:12 +00:00
Jonathan Lebon
92029de009 daemon: fix automatic update service switch
I missed this one when renaming the switch during the review of the
initial code drop.

Closes: #1242
Approved by: cgwalters
2018-02-13 17:30:12 +00:00
Jonathan Lebon
dc63bba78d daemon/dbus: remove reboot mode from comments
This is no longer supported.

Closes: #1242
Approved by: cgwalters
2018-02-13 17:30:12 +00:00
Colin Walters
852b574d01 tests/compose: f is valid in a hex checksum
I *think* this is why our tests started failing recently. It seems somehow very
unlikely to me though that we'd somehow managed to avoid `f` in the boot
checksums until now, but without doing some math...it seems plausible.

Closes: #1243
Approved by: jlebon
2018-02-13 17:26:54 +00:00
Colin Walters
f1e9f53480 scripts: Buffer output in non-journal case, prefix print with id
In the server-side `compose tree` case, this makes it obvious which package is
causing error spew. No particular burning reason, this was just bothering me.

Closes: #1235
Approved by: jlebon
2018-02-12 19:07:11 +00:00
Colin Walters
f26dcd59a2 Add rpm-ostree usroverlay
I saw kalev's slides reference `rpm-ostree unlock`; this patch makes it exist.
In general, people have a hard time (understandably) grasping the distinction
between ostree and rpm-ostree; along with the goal of making ostree really
"libostree", let's start wrapping more commands where it makes sense.

I also took this opportunity to have a more descriptive name; it's important
to note that it *doesn't* overlay `/etc`, `/var`, or `/boot` for example.

Closes: #1233
Approved by: jlebon
2018-02-11 18:03:06 +00:00
Colin Walters
d0e322fb69 postprocess: Special case NFS state files
Currently in RHELAH 7.4, `systemctl start nfs` fails because we've dropped
`/var/lib/nfs/etab` at least:
https://bugzilla.redhat.com/show_bug.cgi?id=1427537

Things appear to work in Fedora 27 Atomic Host; there's been a lot of changes in
upstream nfs-utils in this area. I didn't fully try to analyze all of them, but
I am guessing it's
http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commit;h=8e98eed42b64aa388c09716e3276a53028a839bf
that made things work here.

For now let's just special case these.  I debated having it in a RHEL-only
build but I often do RHELAH treecomposes from my Fedora dev container, and
eh...I think let's ask the RHEL nfs-utils maintainer to backport the patches
to make this work, then we can drop our hack.

Closes: #1229
Approved by: jlebon
2018-02-08 21:59:02 +00:00
Jonathan Lebon
4686e15b94 core: handle shared files and multilib
Not all files from an RPM are necessarily removed during pkg erasure.
For example, files which are shared between pkgs shouldn't be deleted.
Similarly, not all files in an RPM are necessarily copied during pkg
installs. This is the case for multilib handling, which is a mess in its
own right. But such is the cost of trying to replace major parts of a
long-standing foundational project like RPM.

This patch adds some smarts to the way we do overlays and overrides to
handle these cases by calculating beforehand which files we *should't*
checkout/delete.

Closes: #1217
Closes: #1145

Closes: #1227
Approved by: cgwalters
2018-02-08 21:58:55 +00:00
Jonathan Lebon
9a4d780d32 libpriv/importer: factor out pathname translator
This is useful to have in a more global location since we deal with RPM
and OSTree paths in various places.

Closes: #1227
Approved by: cgwalters
2018-02-08 21:58:55 +00:00
Jonathan Lebon
d6b04a038b core: move utility function higher up
Split out to reduce noise in upcoming patch.

Closes: #1227
Approved by: cgwalters
2018-02-08 21:58:55 +00:00
Jonathan Lebon
6c933bbf3c lib/package: fix package diffs for multilib
Our complicated heuristics for handling multiple packages of the same
name comes back to bite us. In the multilib case, we can have packages
of the same NEVR, but different arch, sitting in the same tree.

Previously, even if the arch was different, we would still mark it as an
upgrade or downgrade. But that complicates things in the case of
multiple packages of the same name in the same tree.

We greatly simplify things here by making the diff algorithm dumber. We
now only consider a package as "modified" (i.e. upgraded/downgraded) if
it has the same NA (but different EVR). This makes handling multilib
cases natural and seems worth it overall vs trying to handle the odd
e.g. noarch <--> archful pkg transitions that could happen.

Closes: #1230
Approved by: cgwalters
2018-02-06 21:30:48 +00:00
Colin Walters
b881f33ba7 Add support for rebase rojig://
This is an initial drop of support for:
`rpm-ostree rebase rojig://fahc:fedora-atomic-host`.  We also
then support `rpm-ostree upgrade` from that.

There's a lot that could be improved here; the test coverage is relatively
minimal. A blocking issue there is having a realistic jigdo setup, and that's
going to require changing how we do testing. For now, this means that if we want
to e.g. change the format we'll have to temporarily disable this test, get the
format change in, update FAHC, then re-enable the test.

Closes: #1166
Approved by: jlebon
2018-02-02 20:18:58 +00:00
Colin Walters
2f04450b74 tests/basic: set -x after setup
So it's easier to debug when things fail.

Closes: #1166
Approved by: jlebon
2018-02-02 20:18:58 +00:00
Colin Walters
f6dee42496 sysroot: LOG_ERR failure to read state, not g_critical()
If we can't read the system state, that's an *external* problem with e.g. files
most likely, not a situation in which we should abort.

This came up while playing with `rojig://` where we seem to write
the origin file incorrectly.

Closes: #1166
Approved by: jlebon
2018-02-02 20:18:58 +00:00
Colin Walters
a20fac3e9c core: Fix override-remove from UsrMove compat paths /lib,/bin etc
I'm working on supporting `override replace ./kernel-*.rpm`:
https://github.com/projectatomic/rpm-ostree/issues/946

But after battling the "installonly" logic in libdnf, I was confused why we
still had the files in `/usr/lib/modules`. It turned out to be because we only
remove files in `/usr`, but the code didn't handle UsrMove compat links.

There are a variety of approaches to fix this.  Obviously a lot
of things get nicer in jigdo mode as we do UsrMove canonicalization
on import, and we could change this code to walk the imported pkg
ostree ref.

Another approach would be to walk the initial symlink, and check whether or not
it's the same as `/usr` (i.e. let the kernel do it).

For now though, what I chose to do was to scan the rootfs and find the UsrMove
compat links (i.e. we avoid hardcoding them again here).  This is
fewer syscalls than the above and works well in practice.

Closes: #1226
Approved by: jlebon
2018-01-31 13:09:49 +00:00
Colin Walters
d3c1003079 client: Don't output progress bars on non-ttys
This is what a lot of other tools do. It can get very verbose, with a
potentially huge amount of output if things are trickling in.  This way
we're at least more friendly to someone running `cmd: rpm-ostree upgrade`
via Ansible or equivalent.

The slight hack here is that we *do* output `100%` on non-ttys to ensure we
print the result of the task.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1183

Closes: #1225
Approved by: jlebon
2018-01-29 21:28:07 +00:00
Jonathan Lebon
553a68c32a libpriv: add new rpmostree-types.h
There are types that we want to share between the daemon and the client
for deduplication. Those are, as expected, related to D-Bus things like
formats and enums. Let's create a new file for it rather than shove it
in `rpmostree-util.h`. As mentioned in the file, some of these probably
belong better directly in the public API.

Closes: #1147
Approved by: cgwalters
2018-01-27 23:52:43 +00:00
Jonathan Lebon
51fb641305 Initial support for automatic updates
This patch introduces a new `AutomaticUpdatePolicy` configuration. This
was a long time coming for rpm-ostree, given that its update model makes
it extremely apt for such a feature.

The config supports a `check` mode, which should be very useful to
Atomic Workstation users, as well as a `reboot` mode, which could be
used in its present form in simple single node Atomic Host situations.

There is still a lot of work to be done, including integrating
advisories, and supporting a `deploy` mode. This feature hopefully will
be leveraged as well by higher-level projects like GNOME Software and
Cockpit.

Closes: #1147
Approved by: cgwalters
2018-01-27 23:52:43 +00:00
Colin Walters
3e9c6cf230 Fix "releasever" option, test it by default
In #875 AKA b46fc35901 we
added support for the `releasever` option in treefiles.  I am
pretty sure it worked at the time...but I didn't add tests.

Either it never worked or some refactoring broke it. The whole chain of
`GKeyFile` → `GVariant` is so confusing. Anyways fix it by copying the string.
Now let's use it by default in the compose tests, and while we're here bump
those to F27.

I'm doing this patch now as I was playing with doing a compose from
the `/usr/share/rpm-ostree/treefile.json` and wanted to use the stock
`.repo` files.

Closes: #1220
Approved by: jlebon
2018-01-23 15:18:52 +00:00
Jonathan Lebon
4f025a6609 tree: nuke TODO file
We don't do a good job of keeping this file up to date. It might end up
doing more harm than good if it confuses newcomers.

Anyway, many of these are either no longer relevant or already
implemented. Let's agree that the canonical source of TODOs is the issue
tracker?

Closes: #1222
Approved by: cgwalters
2018-01-22 17:49:25 +00:00
Kalev Lember
79da2e2468 lib: Fix the library name to link to in the pkg-config file
This lets other programs actually link with the shared library.

Closes: #1221
Approved by: jlebon
2018-01-22 16:05:52 +00:00
Colin Walters
48e0cac9fb scripts: Ignore %verifyscript
It obviously shouldn't block the ability to install, and anyways
the single use in `ksh` is not compelling.  If someone comes up with
one we can revisit supporting this.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1216

Closes: #1218
Approved by: jlebon
2018-01-19 15:32:09 +00:00
Colin Walters
b85ae9e1d6 jigdo: V4: Use archful provides for jigdoRPM Requires
When I tried to use my WIP client patches to do:
`rpm-ostree rebase rojig://fahc:fedora-atomic-host`,
I got a missing file object which turned out to
be the client importing the i686 RPMs.

This was passing in the test suite because we don't mirror i686 of course, but
on the client side right now we end up using all enabled repos, and since Fedora
is multiarch, the behavior is going to be...not predictable.

Thinking a bit about on this problem I actually happened to recall
the RPM `%{_isa}` macro which is used in Fedora in various places;
for example to "arch bind" `-devel` packages to their base.  See
for example [this case](33c7dc02bc/f/ostree.spec (_79)) in libostree.

As I noted at first, the core problem here is that the "final"
RPM architecture field is not symmetric in any way with the definition
of that `%{_isa}` macro.  See:

d9d47e0114/installplatform (L25)

The *third* solution I ended up on here is to iterate over the
`Provides` on the server side and we take the first thing
that matches `Provides: %{name}(whatever)`.

I briefly thought about trying to somehow drive into libsolv the
logic to prefer the jigdoRPM's native architecture...IIRC yum did
something like that in the past but it was never done in libsolv?
Anyways the dependencies here are now more correct, so other tools
will also handle it.

Closes: #1213
Approved by: jlebon
2018-01-19 14:18:19 +00:00
Colin Walters
4008bcb27c build-sys: Fix trailing comma causing syntax error
Amazingly things seemed to stumble on from here, but this
started actively breaking when I was trying to change the code below.
It took me a while to realize it wasn't my changes introducing
the error message 😭.

Also indent so it's clearer where the conditional ends.

Closes: #1213
Approved by: jlebon
2018-01-19 14:18:19 +00:00
Jonathan Lebon
500dc02182 vmcheck/overlay: always use --consume --no-bindings
For CI purposes at least, these features are already part of FAH and the
latest CAH smoketested. Anyway, we're already hacking the latest ostree
into place for `--keep-metadata`. If it already supports the latter,
then clearly it'll support those.

Closes: #1215
Approved by: cgwalters
2018-01-18 20:44:07 +00:00
Jonathan Lebon
183cb952e3 daemon: print warning only if val couldn't be parsed
We don't want to print a warning if the setting is missing from the
config file. That's totally normal (e.g. the config we ship has all its
configs commented out). We *do* want to print a warning if the config is
provided, but it couldn't be parsed as a proper `uint64`.

Closes: #1215
Approved by: cgwalters
2018-01-18 20:44:07 +00:00
Jonathan Lebon
9e24e9c793 Revert "tests/libvm: write rpm build logs to file"
This reverts commit 1ef259ed76.

Basically, for historical POSIX compatibility, `errexit` doesn't work at
all inside functions called from an if-statement/boolean context. This
is something I had already learned (and forgotten) when hacking on PAPR.

There are ways around this, but it's just not worth the added complex
shell goop for what it gives us.

http://mywiki.wooledge.org/BashFAQ/105

Closes: #1215
Approved by: cgwalters
2018-01-18 20:44:07 +00:00
Jonathan Lebon
83fb40593f ci: explicitly pass --sysconfdir=/etc to configure
I spent some time trying to figure out why on CentOS my changes to
`/etc/rpm-ostreed.conf` didn't seem to have any effect. It turns out
that for some reason I didn't spend too much time looking into, the
autoconf in CentOS defaults to `${prefix}/etc` when `--prefix` is given.

This is also why I was under the erroneous assumption that D-Bus config
files were in `/usr/etc` in CentOS. This patch fixes that as well.

Closes: #1215
Approved by: cgwalters
2018-01-18 20:44:07 +00:00
Jonathan Lebon
c01d4a8d2b vmcheck: always use base config file
Let's standardize on the default config file when running tests. We copy
the original out of the way and install the default one so tests can do
whatever they want with it.

This also strengthens the post-test cleanup to make sure we rebase back
to the local vmcheck branch, in case we're somehow on a different branch
with the exact same commit.

Closes: #1212
Approved by: cgwalters
2018-01-17 22:45:24 +00:00
Jonathan Lebon
1ef259ed76 tests/libvm: write rpm build logs to file
The output from `rpmbuild` makes test results harder to comb through
when debugging. Let's just dump all that to file and only output it if
something goes wrong.

Closes: #1212
Approved by: cgwalters
2018-01-17 22:45:24 +00:00
Jonathan Lebon
812daf3902 libpriv/util: add function to get deployment root
Prep for auto-updates.

Closes: #1212
Approved by: cgwalters
2018-01-17 22:45:24 +00:00
Jonathan Lebon
688f28c072 daemon: tweak transaction titles for deploy and refresh_md
Prep for auto-updates. Factor out a bunch of flags into nice booleans,
and tweak the transaction title a bit to provide more information about
flags provided.

This will be useful for when the user does a `status` while the daemon
is running due to the automatic updates trigger firing.

Closes: #1212
Approved by: cgwalters
2018-01-17 22:45:24 +00:00
Jonathan Lebon
a70fd4e960 libpriv/rpm-util: add path param for pkglist creation
Prep for auto-updates.

Closes: #1212
Approved by: cgwalters
2018-01-17 22:45:24 +00:00
Jonathan Lebon
2697af3135 app/status: no longer print pending info
Prep for auto-updates. Let's not print the "Pending*" keys anymore. With
automatic updates, we'll have a whole other section of the output just
for that.

Closes: #1212
Approved by: cgwalters
2018-01-17 22:45:24 +00:00
Jonathan Lebon
a211e4670d app/status: separate state from deployments printing
Prep for auto-updates. Use different functions for printing the daemon
state and printing the deployments themselves.

Closes: #1212
Approved by: cgwalters
2018-01-17 22:45:24 +00:00
Colin Walters
2c270a6644 rebase: Add support for providing ostree:// prefix
This fixes an asymmetry we introduced when rendering using the `ostree://`
prefix; we now support parsing it as well.  We reject the `rojig://` prefix
explicitly for now.

On the implementation side...things are messy. My thought is that we do our best
to canonicalize at the entrypoints for both the client side as well as the
server side. However, for backwards compatibility we can't go all the way to
writing out `ostree://` to disk in the origin files.

I'm actually uncertain if Cockpit will deal with this...need to test that.

Closes: #1210
Approved by: jlebon
2018-01-17 14:19:39 +00:00
Colin Walters
567bedf035 core,status: Add API to parse refspecs, use in status
In preparation for rojig (rpm-ostree jigdo) client work.  Changing
all of the code where we currently use refspecs (i.e. pure ostree)
to also learn about jigdo would be really painful - basically adding
new DBus APIs and all of the places we pass through data.

A simpler path then is to overload the meaning of refspecs to have an (optional)
URI scheme. If we don't recognize the scheme, assume it's ostree for now.

Change the status command to parse and then render as a way of "canonicalizing"
so that we have the same behavior as before of rendering with `ostree://`.

Closes: #1208
Approved by: jlebon
2018-01-16 17:43:46 +00:00
Colin Walters
6df154e2e3 daemon: Pass GVariant deployment modifier down another level
Redo the internal API to pass GVariant down another level; this will be more
obviously useful as we move it much farther along through the layers.

Closes: #1170
Approved by: jlebon
2018-01-16 17:16:51 +00:00