Commit Graph

1888 Commits

Author SHA1 Message Date
Colin Walters
a3e0b47e6b ci/papr: Bump compose tests to 4 min vCPUs
Compose is a slow test right now.  Down the line what I'd
like to do is: https://github.com/projectatomic/papr/pull/70
Since this job can be scheduled as a container, not a VM.  There's
no reason to grab a whole 8GB of RAM for it, but we *do* want multiple
CPUs.  Containers do that by default.

Closes: #1187
Approved by: jlebon
2018-01-09 16:26:32 +00:00
Colin Walters
84f0b017d9 bwrap: Use rofiles-fuse --copyup by default
This fixes a large swath of compatibility issues, for the same reasons as
overlayfs makes a lot of things Just Work.  The ugly part of course is
doing hidden copyups inside the filesystem.

We've gone quite a long time with the "pure rofiles" mode, and have made changes
to various bits of userspace to be compatible with it. But what finally made me
give up on that is glibc's locale-archive; there's a patch for it that
is stalled, but even if it was applied we would still need to work with
older glibc.

This issue comes to the fore in unified core 🌐 mode, as without this
we won't get a correct locale archive.

Closes: #1171
Approved by: jlebon
2018-01-09 15:36:08 +00:00
Colin Walters
0e16be8273 tests: Run compose tests in parallel ∥
Over a year later, the "opening the host rpmdb" bug is fixed,
so we can do composes in parallel ∥, hooray!

I'm dusting this off since we were running into CI (PAPR) timeouts
when I was adding more to the compose tests.

Closes: #545
Approved by: jlebon
2018-01-09 14:19:31 +00:00
Colin Walters
02c945cd4f core: Say "Installing %u packages", not "overlays" for containers
Minor, but it was annoying me perhaps somewhat unreasonably.  There's
no "overlay" going on in `ex container` runs, so let's just use the
term "installing".

Closes: #1186
Approved by: jlebon
2018-01-08 19:22:46 +00:00
Colin Walters
e834f4dd27 scripts: Only append "run journalctl" message if using journal
For the `ex container` case we don't log to the journal by default,
so this message was misleading.

Closes: #1181
Approved by: jlebon
2018-01-08 15:14:45 +00:00
Colin Walters
7fa55f7e5f lib/core: Add API to retrieve all pkgs
Will be used for `compose tree --ex-output-jigdo-set`. Also probably more things
could use this. I first tried just doing another query in the higher level code
but that fails for this case as `sort_packages() reimplements
`dnf_transaction_ensure_repo_list()` which we need to get the `DnfRepo` (which
itself feels like a hack, we should be maintaining that dynamically or have a
hash accessor).

Closes: #1184
Approved by: jlebon
2018-01-08 14:41:34 +00:00
Colin Walters
032c1556f4 core,util: Lower some DnfPackage* bits into utils
Ideally we'd be fixing upstream libdnf but that's a bit blocked
right now.  I need these at a higher level to implement
`rpm-ostree compose tree --ex-output-jigdo-set` which needs to
link/copy the input RPMs.

Closes: #1184
Approved by: jlebon
2018-01-08 14:41:34 +00:00
Colin Walters
7ffb544e35 tests/compose: Rename jigdo.sh to jigdo-e2e.sh
I plan to add another jigdo test, and the naming is clearer.

Closes: #1184
Approved by: jlebon
2018-01-08 14:41:34 +00:00
Colin Walters
855d1293ab cli: Rework exit status processing
We've had many bugs from internal helpers using `return EXIT_FAILURE` rather
than `return FALSE`.  The reason we need exit codes is to handle the
`RPM_OSTREE_EXIT_UNCHANGED` case. I realized recently that we had the handy
`RpmOstreeCommandInvocation` which we can use to signal back this special case.
Then all of our functions otherwise are just normal `GError`.

One minor wart here is the two cases of "usage error" versus "command
invocation" in `main.c`, but IMO the general cleanup is well worth that.

Closes: #1169
Approved by: jlebon
2018-01-06 15:30:34 +00:00
Colin Walters
affa50fbc1 Lower jigdo client logic into core
Introduce a new `rpmostree_context_execute_jigdo()` that fills the same role as
`ostree_repo_pull_with_options()`. This will be used by the sysroot upgrader.

I didn't change the jigdo client code much yet; as a TODO says there's a lot
more we can do to improve things. Some of the public APIs we added to the core
no longer need to be public, such as `rpmostree_context_set_packages()`. But
let's try to do things incrementally.

I did at least change the `g_print()`s to `rpmostree_output_message()`.  I
dropped the `commit_and_print()`; at some point will come back and clean things
up so we consistently journal/print stats.

Closes: #1168
Approved by: jlebon
2018-01-05 17:02:53 +00:00
Colin Walters
a55e923eae core: Split out a -private.h header with struct
Prep for doing the "single object, multiple .c files" pattern like is done in
e.g. libostree with OstreeRepo and `ostree-repo-{refs,commit.c}` etc. For jigdo
we'll need to split things up (see what I did there?).

Closes: #1168
Approved by: jlebon
2018-01-05 17:02:52 +00:00
Colin Walters
8f9be2e54b scripts: Suppress microcode_ctl posttrans
Only applies to CentOS7.  Currently the package unconditionally
runs `dracut -f` which we need to handle internally.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1199582

Closes: #1174
Approved by: jlebon
2018-01-05 16:27:26 +00:00
Colin Walters
80e3857720 tests/compose: Pull in Fedora updates
In particular this gets us the selinux-policy fix for:
https://github.com/projectatomic/rpm-ostree/pull/1173#issuecomment-355014583

We might as well do updates since they have to work anyways.

Closes: #1177
Approved by: jlebon
2018-01-05 15:20:42 +00:00
Micah Abbott
f4877419ac docs: update branch examples for 'rebase' command
The old branch examples use Fedora 26 which is almost EOL.  The new
Fedora 27 examples show off the various `testing` and `updates`
branches, as well as the support for different arches.

Closes: #1175
Approved by: cgwalters
2018-01-03 16:05:15 +00:00
Jonathan Lebon
37273705dd app/db-diff: make use of new db API
This is the first step towards unifying how we introspect packages from
a specific commit. We currently do this in three ways: libdnf, librpm,
and now `rpmostree.rpmdb.pkglist`. I'd like to get to a point where we
only have `rpmostree.rpmdb.pkglist` and libdnf, the latter only when
more complex queries are required.

This patch teaches the `db diff` command to make use of the new db diff
API so that it can work even on metadata-only commits. This is relevant
for use cases mentioned in #558.

I didn't get rid of the `rpmhdrs_diff` functions right now because of
the `--changelogs` option: libdnf currently does not expose this, so we
fall back to the previous API in that case. OTOH, I wonder how much it's
actually used in the wild; maybe we could just nix it?

Closes: #1162
Approved by: cgwalters
2017-12-30 11:32:38 +00:00
Jonathan Lebon
def21255ac lib/db: use new pkglist metadata API if possible
Add a function that can smartly perform diff operations on sorted
RpmOstreePackage arrays and make the db API use that. This allows us to
immediately take advantage of the benefits in a few places where diffs
are performed, including post-deployment tree diffs, and the legacy db
diff variant API. The upcoming `CachedUpdate` rework will also make use
of this (but with the notable difference of setting `allow_noent` to
`TRUE`).

Note this introduces a new `rpm_ostree_db_diff_ext` public API which has
the same interface as `rpm_ostree_db_diff` but also takes flags.

Closes: #1162
Approved by: cgwalters
2017-12-30 11:32:38 +00:00
Jonathan Lebon
18d36afdf9 lib/package: teach about rpmostree.rpmdb.pkglist
In order to make use the new pkglist metadata more consummable, let's
add a function to create RpmOstreePackage arrays from it. Since some
APIs need to keep working as before, we use a `allow_noent` parameter to
distinguish between the two cases.

This is prep trying our best to make use of this metadata when possible
rather than checking out the rpmdb and initializing a `DnfSack`.

Closes: #1162
Approved by: cgwalters
2017-12-30 11:32:38 +00:00
Colin Walters
5a6c779850 tests: Make use of commit --no-bindings if available
The way we alias a commit with e.g. `vmcheck_orig` trips
up ref bindings.  Perhaps we should rework the test suite
but for now let's just mark `vmcheck` as aliasable.

See:
https://github.com/ostreedev/ostree/pull/1380
https://github.com/ostreedev/ostree/pull/1379

Closes: #1167
Approved by: jlebon
2017-12-22 14:07:07 +00:00
Jonathan Lebon
e20768dded app/status: support JSONPath queries
The underlying library already supports this. Let's expose it to make it
easier for humans to query the metadata.

Closes: #1164
Approved by: cgwalters
2017-12-21 03:59:54 +00:00
Jonathan Lebon
a3b5f54fbd app/status: wrap package lists nicely if too long
Rather than letting the terminal wrap our line and go unindented on the
next line, let's do the wrapping ourselves to tidy up the output.

Closes: #1159
Approved by: cgwalters
2017-12-20 17:06:31 +00:00
Colin Walters
07bb4339c3 compose: Move txn+ref to toplevel, out of postprocess/commit code
Prep for `compose tree --ex-jigdo`; basically we want to generate the jigdoRPM
and only then set the ref to help bind the two things together.  The main
thing to achieve is that if generating the jigdoRPM fails, the ref isn't
updated.

Closes: #1161
Approved by: jlebon
2017-12-20 15:39:47 +00:00
Colin Walters
2e3ff041c7 tests/compose: Add a test for --write-commitid-to
This is used in critical paths like pungi, so let's be sure it works;
the semantics are a bit subtle as it overrides setting the ref.

Closes: #1161
Approved by: jlebon
2017-12-20 15:39:47 +00:00
Jonathan Lebon
b7fa00e4a6 libpriv/rpm-util: operate on DnfSack directly
Basically reduce these functions to no-frills `DnfSack` utility
functions with the same implied warranty as
`rpmostree_get_matching_packages` and `rpmostree_sack_get_by_pkgname`,
which also operate directly on the sack.

These functions will be used from more places in the upcoming
auto-update patches.

Closes: #1158
Approved by: cgwalters
2017-12-20 13:10:36 +00:00
Jonathan Lebon
b3f19cdc4e libpriv/rpm-util: insert pkglist metadata sorted
To make the new pkglist metadata even more usable, let's insert it
sorted. This ensures that we can bsearch the GVariant on the
client-side.

Enhance the bsearch utility function we have to deal with duplicate key
names. Although this is not the case today in rpm-ostree-managed streams
I know, some packages are allowed to have multiple versions installed,
so let's make sure we handle that deterministically by always returning
the first (earliest) version of the package.

Closes: #1158
Approved by: cgwalters
2017-12-20 13:10:36 +00:00
Jonathan Lebon
addc41ad12 libpriv/rpm-util: drop get_pkglist_for_root
This function returned both an RpmOstreeRefsack and a pkglist bound to
the refsack. Interestingly, there were only two users of it, and one of
them didn't even make use of the pkglist functionality. Since the
lifetime semantics of this function are tricky, let's drop it and
introduce a dedicated function just for returning package lists.

I also dropped the `GCancellable` argument, since it isn't/can't easily
be used by those code paths.

Closes: #1158
Approved by: cgwalters
2017-12-20 13:10:36 +00:00
Jonathan Lebon
d9e870be88 libpriv/core: include pkglist for client commits
The new `rpmdb.pkglist` metadata is a cheap way of retrieving the set of
packages in a commit. I'd like to make use of it as much as possible vs.
checking out the rpmdb and setting up a DnfSack.

Of course, in the case of layered commits, it doesn't matter *as* much,
because a layered commit being present in the repo should mean that a
deployment is currently using it, and we should learn to reuse the rpmdb
checkout of that deployment. Though keeping it consistent across both
server and client commits makes implementing `OstreeDeployment`-agnostic
things like `db diff` more efficient too. I also plan to use this in the
upcoming auto-update code.

Closes: #1158
Approved by: cgwalters
2017-12-20 13:10:36 +00:00
Jonathan Lebon
97133bd028 daemon: avoid using floating GVariant refs
It makes it harder to keep track of ownership except in the trivial
`_new ()` case for which they were designed (i.e. shoving straight into
another glib GVariant function that takes ownership of it right away).

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00
Jonathan Lebon
608ab302e8 daemon/sysroot: reload repo before deployments
In the case of a rebase to a fresh remote, we want to make sure our
`OstreeRepo` picks up the new remote before we try to parse its
deployment. Otherwise, we'll error out when trying to fetch configs for
it.

This was picked up after turning on strict errors in the GPG
verification process.

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00
Jonathan Lebon
5fb153601a daemon/deployment-utils: some new style porting
Prep for more work.

We actually pass a `GError` to the gpg results function and fail for any
other failures than gpg signature verification. (In that case, we want
to make sure that information gets to the D-Bus API level, so killing
the daemon would be the wrong choice).

I also factored out all the logic in
`rpmostreed_commit_generate_cached_details_variant` into a separate
function. This is prep for reusing it elsewhere.

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00
Jonathan Lebon
9ad0829330 daemon/os: reorder iface functions
This is minor, though the list of methods we support is getting long and
will get longer. Let's just add some order to it to make it easier for
humans to parse.

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00
Jonathan Lebon
8e1a769734 daemon/package-variant: fix off-by-one error
We're definitely interested in the rpmdb diff even if there's just one
package changed.

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00
Jonathan Lebon
5018583637 daemon/package-variant: some more porting to new style
Prep for more work.

This also involved changing the functions from pointer return values to
`gboolean`.

I also made functions that didn't really need to be public become
private and fixed some indentation.

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00
Colin Walters
f872ecdd13 scripts: Make use of SYSTEMD_OFFLINE environment variable
More info at:
https://github.com/systemd/systemd/pull/7631 AKA
"systemctl,verbs: Introduce SYSTEMD_OFFLINE environment variable"
f38951a628
and in the docs added.

Closes: #1157
Approved by: jlebon
2017-12-15 17:05:37 +00:00
Colin Walters
f63589d37c bwrap: Hoist subprocess data into struct
This allows our bwrap API to really be more of a wrapper around
`GSubprocessLauncher` rather than having to hold data like the
environment itself.

Closes: #1157
Approved by: jlebon
2017-12-15 17:05:37 +00:00
Colin Walters
937ad1b1fc build-sys: Sync CFLAGS with libostree
In particular this pulls in `-fno-strict-aliasing`.  See
https://github.com/ostreedev/ostree/pull/1384

Closes: #1156
Approved by: jlebon
2017-12-15 16:32:39 +00:00
Colin Walters
8d176e1f92 Use #ifdef for BUILDOPT instead of #if
More prep for `-Werror=undef`. Ideally we'd actually have these be defined but
that seems painful with autotools; for a later date.

Closes: #1156
Approved by: jlebon
2017-12-15 16:32:39 +00:00
Colin Walters
6d1918581b build-sys: Bump glib dependency
This matches reality; I picked what's in CentOS 7 mainline today. More
importantly this also fixes a build error with `-Werror=undef` because we had a
trailing underscore `_` at the end and never noticed.

Closes: #1156
Approved by: jlebon
2017-12-15 16:32:39 +00:00
Colin Walters
b4d0254581 build-sys: Fix use of libglnx configure bits
This fixes the build with `-Werror=undef` like we did in libostree.
And man is this obscure black magic.

Closes: #1156
Approved by: jlebon
2017-12-15 16:32:39 +00:00
Colin Walters
9843c93439 core,passwd: Use new libostree hardlink-break API
The code moved into libostree here:
https://github.com/ostreedev/ostree/pull/1378

Closes: #1154
Approved by: jlebon
2017-12-15 15:55:10 +00:00
Jonathan Lebon
5ff69e10db daemon/os: check for download-only and dry-run conflict
Minor follow-up to previous commit. It seems a bit confusing to allow
specifying `download-only` and `dry-run`. The former already includes
all the steps in the latter but goes further, as documented. Let's check
for this combination.

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Jonathan Lebon
183399e1cd daemon/deploy: fix --dry-run downloads packages
Minor regression that crept in during the `--download-only` work. We
would download and import packages even when `--dry-run` was given.
Make sure we stop right after printing the transaction.

Related: #1128

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Jonathan Lebon
864129c5de libpriv/rpm-util: fix returning FALSE instead of NULL
The compiler doesn't complain, but it throws off the reader.

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Jonathan Lebon
7f0737e7fc daemon/os: fix complete function for SetInitramfsState()
We were calling the wrong completer function for `SetInitramfsState()`.
Not that it mattered much in compiled form since both ways ended up
calling the same internal gdbus function with the same arguments.

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Jonathan Lebon
d5e92df5c1 rpm-ostreed.service: tweak description and add doc
This is highly dependent of the outcome of [1], though until that's
settled there, let's at least update the description to something a
little more apt. It feels more appropriate to consider rpm-ostree as a
"system manager" than just a "package manager" (which it certainly is
too of course). Also use Title Case convention which seems more popular
overall and looks nicer.

[1] https://github.com/projectatomic/rpm-ostree/issues/405

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Jonathan Lebon
be21f6f244 core: remove extra newline between repo downloads
Minor polish item; `output_message` already prints a newline at the end.
Looks nicer without the double empty lines when refreshing metadata, and
no lines when cached.

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Colin Walters
a30af4052e jigdo2commit: Make use of new libostree commitpartial API
This helps jigdo behave more closely to how `ostree_repo_pull_with_options()`
does.

Closes: #1152
Approved by: jlebon
2017-12-15 14:25:16 +00:00
Jonathan Lebon
9aaf195549 Makefile-daemon.am: work around gdbus-codegen path logic
I just rebased my pet container to F27, and this was the only hiccup
when trying to build rpm-ostree. Basically, gdbus-codegen is sensitive
to how it's called when trying to find its own Python modules. Calling
it with the explicit `/usr` prefix works around that. This was fixed
upstream in [1], but hasn't made its way down the metaphorical stream
yet. See [2] for more information.

[1] b9f2ea4235
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1485853

Closes: #1153
Approved by: cgwalters
2017-12-15 02:05:35 +00:00
Colin Walters
fcb2a9d11f postprocess: Use same rigor for bwrap as scripts in unified core mode
This would have prevented corruption I saw when using unified core 🌐 mode; we
ended up appending repeatedly to the version in the imported pkgcache repo
where fedora-atomic does:
`echo 'Storage=persistent' >> /etc/systemd/journald.conf`

See also https://pagure.io/fedora-atomic/pull-request/97

Closes: #1151
Approved by: jlebon
2017-12-14 22:13:27 +00:00
Colin Walters
9f2db12329 core: Fix fcaps (e.g. ping) in unified core mode
This fixes another thing broken with `compose --ex-unified-core`;
for e.g. `/usr/bin/ping` from `iputils`, the classic example of a filecaps
binary.

As I'm writing this commit message I realize it will actually also
take effect for package layering unnecessarily; we'll pointlessly
break the hardlink.  But eh, it doesn't matter right now, we can
optimize that later.

Closes: #1151
Approved by: jlebon
2017-12-14 22:13:27 +00:00
Colin Walters
8b41718d60 Move most jigdo build logic into internal library
Prep for adding `compose tree --ex-jigdo` to do both at the same time.

Changes other than code motion were minimized; the main thing was tweaks around
the initial option processing to call the API.

Closes: #1146
Approved by: jlebon
2017-12-14 19:37:38 +00:00