Commit Graph

1906 Commits

Author SHA1 Message Date
Jonathan Lebon
699cc89198 libpriv/rpm-util: encode epoch as a string
This is essentially a revert of #1190. I initially changed from storing
the epoch, version, and release separately to together to avoid endian
issues with epoch. This works around that instead by just encoding it as
a string.

As is done in our custom NEVRA printer (which before #1190 we used
here), we make sure that epochs of "0" are not printed as part of the
`nevra` or `evr` members.

Closes: #1198
Approved by: cgwalters
2018-01-11 21:58:49 +00:00
Colin Walters
bc5237ebf8 tests: Add a test case for epoch 0 in rpmdb pkglist
Basically we're choosing the libdnf behavior of omitting `Epoch: 0` which IMO is
the sane thing to do even though in *theory* they're different RPMs but
who would ever ship both without incrementing release/etc?

More information in: https://github.com/projectatomic/rpm-ostree/pull/187

Closes: #1198
Approved by: cgwalters
2018-01-11 21:58:49 +00:00
Jonathan Lebon
738fbc633b tests/libtest-core: support multiple literal checks
`grep` supports checking multiple fixed strings separated by newlines,
but it's mostly just easier to pass them as separate arguments, so let's
support that. This is now at parity with the similar
`assert_file_has_content`.

Will upstream this to ostree as well once reviewed.

Closes: #1200
Approved by: cgwalters
2018-01-11 20:17:50 +00:00
Jonathan Lebon
9f7511a5d8 libpriv/rpm-util: add autoptr support to RpmHeaders
Prep for follow up patches.

Closes: #1200
Approved by: cgwalters
2018-01-11 20:17:50 +00:00
Jonathan Lebon
542e4c153b libpriv/rpm-util: factor refts from rootfs helper
This will be used later. Also make it take either a `GLnxTmpDir` or
a path to the rootfs.

Closes: #1200
Approved by: cgwalters
2018-01-11 20:17:50 +00:00
Jonathan Lebon
436f7a1d4c app/status: factor out some print functions
With automatic updates, I'd like to make the outputs of `upgrade
--check/--preview` very similar to parts of `status`. E.g. timestamp,
version, GPG signature printing. So let's factor those out into
`libbuiltin`.

Note that unlike the others, `rpmostree_print_gpg_info` deals directly
with a `GVariant` received from D-Bus, so it felt more appropriate to
place it in `dbus-helpers`. Plus, `rpmostree_print_signatures` was
already there, so it simplified things a bit.

This also sneaks in a nicer way of printing padded strings using the
argument field width syntax.

Closes: #1196
Approved by: cgwalters
2018-01-11 15:37:57 +00:00
Jonathan Lebon
778a78bb9f app/status: factor out escape chars logic
Make use of the new `glnx_stdout_is_console ()` rather than caching our
own result of `isatty()`. Add helper functions in `libbuiltin.h` to
retrieve escape characters. Prep for using them from other files.

Update submodule: libglnx

Closes: #1196
Approved by: cgwalters
2018-01-11 15:37:57 +00:00
Jonathan Lebon
c80a4168de dbus: document missing key from CachedUpdate
Closes: #1196
Approved by: cgwalters
2018-01-11 15:37:57 +00:00
Colin Walters
7e44c032c8 experiments-and-demos: New subdir with skopeo2ostree Dockerfile
Before I went all in on jigdo I was playing around with doing
a build from `Dockerfile` of something that looked like Atomic Host,
and then importing that build into an ostree repo.

For a lot of reasons, I think this is not the way to go in general
right now, but let's keep this demo around so we can pick
things up back later if we want to (or someone else does).

Closes: #1193
Approved by: jlebon
2018-01-11 14:07:17 +00:00
Colin Walters
c0cb60bd5e core: Don't try to relabel if no SELinux policy
This tripped up some local `ex container` work after:
https://github.com/projectatomic/rpm-ostree/pull/1173

Would be nice to have tests but...let's slide this one through for now since I
plan to get a lot more coverage as `ex container` morphs into a `mock`
replacement.

Closes: #1194
Approved by: jlebon
2018-01-10 22:10:13 +00:00
Colin Walters
9f777c26ee compose: Add --ex-jigdo-output-set
Following up on `--ex-jigdo-output-rpm`, add support for writing the entire set
to an output directory. This is intended for use cases like FAHC, where we're
generating data outside of the upstream Fedora infrastructure. Further, we want
to support having our own history stored reliably, even if upstream prunes RPMs.

Now, this can be interesting even for upstreams like Fedora, as it naturally
captures just the subset of RPMs; doing full history support for that would
likely be a lot more palatable than for Everything.

Closes: #1165
Approved by: jlebon
2018-01-10 19:18:40 +00:00
Colin Walters
79f4b4b861 compose: Add --ex-jigdo-output-rpm
Part of the goal of jigdo ♲📦 is to support organizations switching to *only*
providing RPMs. An intermediate step there is to "lock" the repo and jigdo
together; we don't want to update the ref if building the jigdoRPM fails.

Add an option to perform `rpm-ostree compose tree` and `rpm-ostree ex
commit2jigdo` together; notably we generate a commit, but only update the ref
once the jigdoRPM is built.

Closes: #1165
Approved by: jlebon
2018-01-10 19:18:40 +00:00
Jonathan Lebon
4ad627f69b daemon: bump idle exit timeout to 60s
Since it usually takes more than 10s for users to enter consecutive
commands, let's bump the timeout to 60s. That way, we avoid the churn of
starting up twice and e.g. polluting the journal.

On a user interface level, this doesn't make a big difference: a
`status` from cold takes around 100ms, whereas with the daemon running,
it takes slightly less than 50ms. Slightly noticeable, but a non-issue.

However, auto-update will require some more work at startup, and a cold
`status` will bump to about 350ms, which is definitely more noticeable.
Bumping the timeout will ensure that at least within the span of one
"interaction" (multiple commands), we only do this work once.

Closes: #1192
Approved by: cgwalters
2018-01-10 18:53:50 +00:00
Jonathan Lebon
7f946130ef app/main: also return the bus type on option parsing
This is prep for automatic updates. There, we want to know which D-Bus
we're connected to and e.g. only try to reach out to other services like
systemd if we're on the system bus.

Closes: #1191
Approved by: cgwalters
2018-01-10 17:20:33 +00:00
Jonathan Lebon
370679717b libpriv/rpm-util: just encode evr in pkglist
I initially wanted to include all of `epoch`, `version`, and `release`
separately in the rpmdb.pkglist metadata in case we needed them
separately. Thinking more on this, I can't think of a really good reason
to have them, so since we're not public yet, let's just encode the `evr`
as a single string.

The reason I'm revisiting this is because it took me a while to hunt
down issues when handling epoch, which turned out to be the fact that we
weren't consistently committing as big endian (i.e. we did so in the
`compose tree` case, but not the layering case), and neither were we
consistently converting back from big endian. It's doable of course, but
the added gymnastics doesn't feel justified for the gains here.

This also nicely cleans up the `RpmOstreePackage` implementation to be
leaner and faster.

Closes: #1190
Approved by: cgwalters
2018-01-10 17:18:09 +00:00
Colin Walters
b1c92a6a1d tests/compose: Various fixes
- Actually use separate `${test_tmpdir}` for test setup (closes a race)
 - Merge stdout/stderr (more readable)
 - Ensure logs are renamed to `.txt` even on failure
 - Use `--progress` for some feedback
 - Use `-j +1` so that even on unicore machines we get at least 2
   jobs (and in general NCPUS+1)

Closes: #1188
Approved by: jlebon
2018-01-10 15:16:18 +00:00
Colin Walters
c82a7b53a0 man: Various tweaks/updates to the man page
I initially started this since I wanted to have the client-side
commands first, but ended up splitting them into a separate section.

Various other things:

 - Update the intro
 - Add jlebon to authors

Closes: #1189
Approved by: jlebon
2018-01-09 22:38:19 +00:00
Colin Walters
054b48d55b core,compose: Fix unified core pkgcache labeling
Basically the `rpmostree_context_relabel()` call we had in the treecompose path
for unified core didn't actually have any effect as the core code did a relabel
and unset the array.

I think this may actually be a regression from: https://github.com/projectatomic/rpm-ostree/pull/1137
though I didn't verify.

Anyways looking at this, the code is a lot simpler if we change the API so that
the "normal" relabeling is folded into `rpmostree_context_assemble()`. Then we
change the public relabel API to be "force relabel" which we use in the unified
core 🌐 treecompose path.

This shrinks the jigdoRPM for FAH from 90MB to 68MB.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1172

Closes: #1173
Approved by: jlebon
2018-01-09 16:59:19 +00:00
Colin Walters
a3e0b47e6b ci/papr: Bump compose tests to 4 min vCPUs
Compose is a slow test right now.  Down the line what I'd
like to do is: https://github.com/projectatomic/papr/pull/70
Since this job can be scheduled as a container, not a VM.  There's
no reason to grab a whole 8GB of RAM for it, but we *do* want multiple
CPUs.  Containers do that by default.

Closes: #1187
Approved by: jlebon
2018-01-09 16:26:32 +00:00
Colin Walters
84f0b017d9 bwrap: Use rofiles-fuse --copyup by default
This fixes a large swath of compatibility issues, for the same reasons as
overlayfs makes a lot of things Just Work.  The ugly part of course is
doing hidden copyups inside the filesystem.

We've gone quite a long time with the "pure rofiles" mode, and have made changes
to various bits of userspace to be compatible with it. But what finally made me
give up on that is glibc's locale-archive; there's a patch for it that
is stalled, but even if it was applied we would still need to work with
older glibc.

This issue comes to the fore in unified core 🌐 mode, as without this
we won't get a correct locale archive.

Closes: #1171
Approved by: jlebon
2018-01-09 15:36:08 +00:00
Colin Walters
0e16be8273 tests: Run compose tests in parallel ∥
Over a year later, the "opening the host rpmdb" bug is fixed,
so we can do composes in parallel ∥, hooray!

I'm dusting this off since we were running into CI (PAPR) timeouts
when I was adding more to the compose tests.

Closes: #545
Approved by: jlebon
2018-01-09 14:19:31 +00:00
Colin Walters
02c945cd4f core: Say "Installing %u packages", not "overlays" for containers
Minor, but it was annoying me perhaps somewhat unreasonably.  There's
no "overlay" going on in `ex container` runs, so let's just use the
term "installing".

Closes: #1186
Approved by: jlebon
2018-01-08 19:22:46 +00:00
Colin Walters
e834f4dd27 scripts: Only append "run journalctl" message if using journal
For the `ex container` case we don't log to the journal by default,
so this message was misleading.

Closes: #1181
Approved by: jlebon
2018-01-08 15:14:45 +00:00
Colin Walters
7fa55f7e5f lib/core: Add API to retrieve all pkgs
Will be used for `compose tree --ex-output-jigdo-set`. Also probably more things
could use this. I first tried just doing another query in the higher level code
but that fails for this case as `sort_packages() reimplements
`dnf_transaction_ensure_repo_list()` which we need to get the `DnfRepo` (which
itself feels like a hack, we should be maintaining that dynamically or have a
hash accessor).

Closes: #1184
Approved by: jlebon
2018-01-08 14:41:34 +00:00
Colin Walters
032c1556f4 core,util: Lower some DnfPackage* bits into utils
Ideally we'd be fixing upstream libdnf but that's a bit blocked
right now.  I need these at a higher level to implement
`rpm-ostree compose tree --ex-output-jigdo-set` which needs to
link/copy the input RPMs.

Closes: #1184
Approved by: jlebon
2018-01-08 14:41:34 +00:00
Colin Walters
7ffb544e35 tests/compose: Rename jigdo.sh to jigdo-e2e.sh
I plan to add another jigdo test, and the naming is clearer.

Closes: #1184
Approved by: jlebon
2018-01-08 14:41:34 +00:00
Colin Walters
855d1293ab cli: Rework exit status processing
We've had many bugs from internal helpers using `return EXIT_FAILURE` rather
than `return FALSE`.  The reason we need exit codes is to handle the
`RPM_OSTREE_EXIT_UNCHANGED` case. I realized recently that we had the handy
`RpmOstreeCommandInvocation` which we can use to signal back this special case.
Then all of our functions otherwise are just normal `GError`.

One minor wart here is the two cases of "usage error" versus "command
invocation" in `main.c`, but IMO the general cleanup is well worth that.

Closes: #1169
Approved by: jlebon
2018-01-06 15:30:34 +00:00
Colin Walters
affa50fbc1 Lower jigdo client logic into core
Introduce a new `rpmostree_context_execute_jigdo()` that fills the same role as
`ostree_repo_pull_with_options()`. This will be used by the sysroot upgrader.

I didn't change the jigdo client code much yet; as a TODO says there's a lot
more we can do to improve things. Some of the public APIs we added to the core
no longer need to be public, such as `rpmostree_context_set_packages()`. But
let's try to do things incrementally.

I did at least change the `g_print()`s to `rpmostree_output_message()`.  I
dropped the `commit_and_print()`; at some point will come back and clean things
up so we consistently journal/print stats.

Closes: #1168
Approved by: jlebon
2018-01-05 17:02:53 +00:00
Colin Walters
a55e923eae core: Split out a -private.h header with struct
Prep for doing the "single object, multiple .c files" pattern like is done in
e.g. libostree with OstreeRepo and `ostree-repo-{refs,commit.c}` etc. For jigdo
we'll need to split things up (see what I did there?).

Closes: #1168
Approved by: jlebon
2018-01-05 17:02:52 +00:00
Colin Walters
8f9be2e54b scripts: Suppress microcode_ctl posttrans
Only applies to CentOS7.  Currently the package unconditionally
runs `dracut -f` which we need to handle internally.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1199582

Closes: #1174
Approved by: jlebon
2018-01-05 16:27:26 +00:00
Colin Walters
80e3857720 tests/compose: Pull in Fedora updates
In particular this gets us the selinux-policy fix for:
https://github.com/projectatomic/rpm-ostree/pull/1173#issuecomment-355014583

We might as well do updates since they have to work anyways.

Closes: #1177
Approved by: jlebon
2018-01-05 15:20:42 +00:00
Micah Abbott
f4877419ac docs: update branch examples for 'rebase' command
The old branch examples use Fedora 26 which is almost EOL.  The new
Fedora 27 examples show off the various `testing` and `updates`
branches, as well as the support for different arches.

Closes: #1175
Approved by: cgwalters
2018-01-03 16:05:15 +00:00
Jonathan Lebon
37273705dd app/db-diff: make use of new db API
This is the first step towards unifying how we introspect packages from
a specific commit. We currently do this in three ways: libdnf, librpm,
and now `rpmostree.rpmdb.pkglist`. I'd like to get to a point where we
only have `rpmostree.rpmdb.pkglist` and libdnf, the latter only when
more complex queries are required.

This patch teaches the `db diff` command to make use of the new db diff
API so that it can work even on metadata-only commits. This is relevant
for use cases mentioned in #558.

I didn't get rid of the `rpmhdrs_diff` functions right now because of
the `--changelogs` option: libdnf currently does not expose this, so we
fall back to the previous API in that case. OTOH, I wonder how much it's
actually used in the wild; maybe we could just nix it?

Closes: #1162
Approved by: cgwalters
2017-12-30 11:32:38 +00:00
Jonathan Lebon
def21255ac lib/db: use new pkglist metadata API if possible
Add a function that can smartly perform diff operations on sorted
RpmOstreePackage arrays and make the db API use that. This allows us to
immediately take advantage of the benefits in a few places where diffs
are performed, including post-deployment tree diffs, and the legacy db
diff variant API. The upcoming `CachedUpdate` rework will also make use
of this (but with the notable difference of setting `allow_noent` to
`TRUE`).

Note this introduces a new `rpm_ostree_db_diff_ext` public API which has
the same interface as `rpm_ostree_db_diff` but also takes flags.

Closes: #1162
Approved by: cgwalters
2017-12-30 11:32:38 +00:00
Jonathan Lebon
18d36afdf9 lib/package: teach about rpmostree.rpmdb.pkglist
In order to make use the new pkglist metadata more consummable, let's
add a function to create RpmOstreePackage arrays from it. Since some
APIs need to keep working as before, we use a `allow_noent` parameter to
distinguish between the two cases.

This is prep trying our best to make use of this metadata when possible
rather than checking out the rpmdb and initializing a `DnfSack`.

Closes: #1162
Approved by: cgwalters
2017-12-30 11:32:38 +00:00
Colin Walters
5a6c779850 tests: Make use of commit --no-bindings if available
The way we alias a commit with e.g. `vmcheck_orig` trips
up ref bindings.  Perhaps we should rework the test suite
but for now let's just mark `vmcheck` as aliasable.

See:
https://github.com/ostreedev/ostree/pull/1380
https://github.com/ostreedev/ostree/pull/1379

Closes: #1167
Approved by: jlebon
2017-12-22 14:07:07 +00:00
Jonathan Lebon
e20768dded app/status: support JSONPath queries
The underlying library already supports this. Let's expose it to make it
easier for humans to query the metadata.

Closes: #1164
Approved by: cgwalters
2017-12-21 03:59:54 +00:00
Jonathan Lebon
a3b5f54fbd app/status: wrap package lists nicely if too long
Rather than letting the terminal wrap our line and go unindented on the
next line, let's do the wrapping ourselves to tidy up the output.

Closes: #1159
Approved by: cgwalters
2017-12-20 17:06:31 +00:00
Colin Walters
07bb4339c3 compose: Move txn+ref to toplevel, out of postprocess/commit code
Prep for `compose tree --ex-jigdo`; basically we want to generate the jigdoRPM
and only then set the ref to help bind the two things together.  The main
thing to achieve is that if generating the jigdoRPM fails, the ref isn't
updated.

Closes: #1161
Approved by: jlebon
2017-12-20 15:39:47 +00:00
Colin Walters
2e3ff041c7 tests/compose: Add a test for --write-commitid-to
This is used in critical paths like pungi, so let's be sure it works;
the semantics are a bit subtle as it overrides setting the ref.

Closes: #1161
Approved by: jlebon
2017-12-20 15:39:47 +00:00
Jonathan Lebon
b7fa00e4a6 libpriv/rpm-util: operate on DnfSack directly
Basically reduce these functions to no-frills `DnfSack` utility
functions with the same implied warranty as
`rpmostree_get_matching_packages` and `rpmostree_sack_get_by_pkgname`,
which also operate directly on the sack.

These functions will be used from more places in the upcoming
auto-update patches.

Closes: #1158
Approved by: cgwalters
2017-12-20 13:10:36 +00:00
Jonathan Lebon
b3f19cdc4e libpriv/rpm-util: insert pkglist metadata sorted
To make the new pkglist metadata even more usable, let's insert it
sorted. This ensures that we can bsearch the GVariant on the
client-side.

Enhance the bsearch utility function we have to deal with duplicate key
names. Although this is not the case today in rpm-ostree-managed streams
I know, some packages are allowed to have multiple versions installed,
so let's make sure we handle that deterministically by always returning
the first (earliest) version of the package.

Closes: #1158
Approved by: cgwalters
2017-12-20 13:10:36 +00:00
Jonathan Lebon
addc41ad12 libpriv/rpm-util: drop get_pkglist_for_root
This function returned both an RpmOstreeRefsack and a pkglist bound to
the refsack. Interestingly, there were only two users of it, and one of
them didn't even make use of the pkglist functionality. Since the
lifetime semantics of this function are tricky, let's drop it and
introduce a dedicated function just for returning package lists.

I also dropped the `GCancellable` argument, since it isn't/can't easily
be used by those code paths.

Closes: #1158
Approved by: cgwalters
2017-12-20 13:10:36 +00:00
Jonathan Lebon
d9e870be88 libpriv/core: include pkglist for client commits
The new `rpmdb.pkglist` metadata is a cheap way of retrieving the set of
packages in a commit. I'd like to make use of it as much as possible vs.
checking out the rpmdb and setting up a DnfSack.

Of course, in the case of layered commits, it doesn't matter *as* much,
because a layered commit being present in the repo should mean that a
deployment is currently using it, and we should learn to reuse the rpmdb
checkout of that deployment. Though keeping it consistent across both
server and client commits makes implementing `OstreeDeployment`-agnostic
things like `db diff` more efficient too. I also plan to use this in the
upcoming auto-update code.

Closes: #1158
Approved by: cgwalters
2017-12-20 13:10:36 +00:00
Jonathan Lebon
97133bd028 daemon: avoid using floating GVariant refs
It makes it harder to keep track of ownership except in the trivial
`_new ()` case for which they were designed (i.e. shoving straight into
another glib GVariant function that takes ownership of it right away).

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00
Jonathan Lebon
608ab302e8 daemon/sysroot: reload repo before deployments
In the case of a rebase to a fresh remote, we want to make sure our
`OstreeRepo` picks up the new remote before we try to parse its
deployment. Otherwise, we'll error out when trying to fetch configs for
it.

This was picked up after turning on strict errors in the GPG
verification process.

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00
Jonathan Lebon
5fb153601a daemon/deployment-utils: some new style porting
Prep for more work.

We actually pass a `GError` to the gpg results function and fail for any
other failures than gpg signature verification. (In that case, we want
to make sure that information gets to the D-Bus API level, so killing
the daemon would be the wrong choice).

I also factored out all the logic in
`rpmostreed_commit_generate_cached_details_variant` into a separate
function. This is prep for reusing it elsewhere.

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00
Jonathan Lebon
9ad0829330 daemon/os: reorder iface functions
This is minor, though the list of methods we support is getting long and
will get longer. Let's just add some order to it to make it easier for
humans to parse.

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00
Jonathan Lebon
8e1a769734 daemon/package-variant: fix off-by-one error
We're definitely interested in the rpmdb diff even if there's just one
package changed.

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00
Jonathan Lebon
5018583637 daemon/package-variant: some more porting to new style
Prep for more work.

This also involved changing the functions from pointer return values to
`gboolean`.

I also made functions that didn't really need to be public become
private and fixed some indentation.

Closes: #1160
Approved by: cgwalters
2017-12-20 13:09:42 +00:00