Commit Graph

1854 Commits

Author SHA1 Message Date
Colin Walters
937ad1b1fc build-sys: Sync CFLAGS with libostree
In particular this pulls in `-fno-strict-aliasing`.  See
https://github.com/ostreedev/ostree/pull/1384

Closes: #1156
Approved by: jlebon
2017-12-15 16:32:39 +00:00
Colin Walters
8d176e1f92 Use #ifdef for BUILDOPT instead of #if
More prep for `-Werror=undef`. Ideally we'd actually have these be defined but
that seems painful with autotools; for a later date.

Closes: #1156
Approved by: jlebon
2017-12-15 16:32:39 +00:00
Colin Walters
6d1918581b build-sys: Bump glib dependency
This matches reality; I picked what's in CentOS 7 mainline today. More
importantly this also fixes a build error with `-Werror=undef` because we had a
trailing underscore `_` at the end and never noticed.

Closes: #1156
Approved by: jlebon
2017-12-15 16:32:39 +00:00
Colin Walters
b4d0254581 build-sys: Fix use of libglnx configure bits
This fixes the build with `-Werror=undef` like we did in libostree.
And man is this obscure black magic.

Closes: #1156
Approved by: jlebon
2017-12-15 16:32:39 +00:00
Colin Walters
9843c93439 core,passwd: Use new libostree hardlink-break API
The code moved into libostree here:
https://github.com/ostreedev/ostree/pull/1378

Closes: #1154
Approved by: jlebon
2017-12-15 15:55:10 +00:00
Jonathan Lebon
5ff69e10db daemon/os: check for download-only and dry-run conflict
Minor follow-up to previous commit. It seems a bit confusing to allow
specifying `download-only` and `dry-run`. The former already includes
all the steps in the latter but goes further, as documented. Let's check
for this combination.

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Jonathan Lebon
183399e1cd daemon/deploy: fix --dry-run downloads packages
Minor regression that crept in during the `--download-only` work. We
would download and import packages even when `--dry-run` was given.
Make sure we stop right after printing the transaction.

Related: #1128

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Jonathan Lebon
864129c5de libpriv/rpm-util: fix returning FALSE instead of NULL
The compiler doesn't complain, but it throws off the reader.

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Jonathan Lebon
7f0737e7fc daemon/os: fix complete function for SetInitramfsState()
We were calling the wrong completer function for `SetInitramfsState()`.
Not that it mattered much in compiled form since both ways ended up
calling the same internal gdbus function with the same arguments.

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Jonathan Lebon
d5e92df5c1 rpm-ostreed.service: tweak description and add doc
This is highly dependent of the outcome of [1], though until that's
settled there, let's at least update the description to something a
little more apt. It feels more appropriate to consider rpm-ostree as a
"system manager" than just a "package manager" (which it certainly is
too of course). Also use Title Case convention which seems more popular
overall and looks nicer.

[1] https://github.com/projectatomic/rpm-ostree/issues/405

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Jonathan Lebon
be21f6f244 core: remove extra newline between repo downloads
Minor polish item; `output_message` already prints a newline at the end.
Looks nicer without the double empty lines when refreshing metadata, and
no lines when cached.

Closes: #1155
Approved by: cgwalters
2017-12-15 15:23:17 +00:00
Colin Walters
a30af4052e jigdo2commit: Make use of new libostree commitpartial API
This helps jigdo behave more closely to how `ostree_repo_pull_with_options()`
does.

Closes: #1152
Approved by: jlebon
2017-12-15 14:25:16 +00:00
Jonathan Lebon
9aaf195549 Makefile-daemon.am: work around gdbus-codegen path logic
I just rebased my pet container to F27, and this was the only hiccup
when trying to build rpm-ostree. Basically, gdbus-codegen is sensitive
to how it's called when trying to find its own Python modules. Calling
it with the explicit `/usr` prefix works around that. This was fixed
upstream in [1], but hasn't made its way down the metaphorical stream
yet. See [2] for more information.

[1] b9f2ea4235
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1485853

Closes: #1153
Approved by: cgwalters
2017-12-15 02:05:35 +00:00
Colin Walters
fcb2a9d11f postprocess: Use same rigor for bwrap as scripts in unified core mode
This would have prevented corruption I saw when using unified core 🌐 mode; we
ended up appending repeatedly to the version in the imported pkgcache repo
where fedora-atomic does:
`echo 'Storage=persistent' >> /etc/systemd/journald.conf`

See also https://pagure.io/fedora-atomic/pull-request/97

Closes: #1151
Approved by: jlebon
2017-12-14 22:13:27 +00:00
Colin Walters
9f2db12329 core: Fix fcaps (e.g. ping) in unified core mode
This fixes another thing broken with `compose --ex-unified-core`;
for e.g. `/usr/bin/ping` from `iputils`, the classic example of a filecaps
binary.

As I'm writing this commit message I realize it will actually also
take effect for package layering unnecessarily; we'll pointlessly
break the hardlink.  But eh, it doesn't matter right now, we can
optimize that later.

Closes: #1151
Approved by: jlebon
2017-12-14 22:13:27 +00:00
Colin Walters
8b41718d60 Move most jigdo build logic into internal library
Prep for adding `compose tree --ex-jigdo` to do both at the same time.

Changes other than code motion were minimized; the main thing was tweaks around
the initial option processing to call the API.

Closes: #1146
Approved by: jlebon
2017-12-14 19:37:38 +00:00
Colin Walters
6a2962fd8b core: Process systemctl preset in %post
Rather than entirely symlinking `systemctl` → `/bin/true`, in order
to e.g. have NetworkManager be enabled, we need to process presets.

This is one of the things that's breaking FAHC where I did a
`--ex-unified-core` deployment.

(Actually it's a bit tempting to run a mass preset pass at the end,
 but for now let's do this)

Implementation note: this is our first use of GResources, which
is a handy way to embed data into our final binary.

Closes: https://github.com/projectatomic/rpm-ostree/issues/550

Closes: #1148
Approved by: jlebon
2017-12-14 19:05:25 +00:00
Colin Walters
9ff9f6c997 core: Make use of "n items" progress for filesystem checkouts
This one was really begging to be done after the previous uses IMO.

Closes: #1150
Approved by: jlebon
2017-12-14 14:18:14 +00:00
Colin Walters
dac5ccc76e Improve progress output
This rolls up several libglnx changes: https://github.com/GNOME/libglnx/pull/101

Now of course things are trickier here because we have an internal
abstraction over directly emitting to a console versus sending the
result over DBus.  Further complicating things is that some things
call into libdnf and thus *require* use of `DnfState` which does
not give us the "n items" information, versus other parts which
we implement and can do what we want.

Even *further* complicating things is that we have to take care around non-CLI
callers like Cockpit; so I didn't try to pass the "n items" over DBus, rather
just reimplemented the "insert into text" that libglnx is doing.

Anyways overall this looks better IMO for all cases.

Update submodule: libglnx

Closes: #1143
Approved by: jlebon
2017-12-14 03:32:22 +00:00
Colin Walters
1793480155 Start moving some jigdo logic into core
The jigdo ♲📦 effort really throws a spanner into the logic behind our whole
code layout; so far I mostly sidestepped that by having a lot of the new logic
in the CLI, with just some `_jigdo_xxx()` methods in core code.

But in order to start on having the "sysroot" side use jigdo, let's start
moving some bits into core.

Closes: #1144
Approved by: jlebon
2017-12-13 19:05:01 +00:00
Colin Walters
879c5afefc compose: Write rpmdb to /usr/lib/sysimage/rpm-ostree-base-db
This is prep for a rework of
https://github.com/projectatomic/rpm-ostree/pull/621

For a no-op `rpm-ostree upgrade` (i.e. no updates available), as long as
layering is enabled, we pay the cost of checking out the base tree, *mostly*
only to get the base rpmdb.

This is prep for fixing that down the line by knowing we always have the "base"
tree's rpmdb checked out. Then in the layering case we only modify
`/usr/share/rpm` (eventually that will point to `/usr/lib/sysimage/rpm`).

Teaching `rpmostree-core.c` about this can follow on later.

Closes: #1142
Approved by: jlebon
2017-12-13 16:12:36 +00:00
Colin Walters
e9e08c5b1f jigdo: v3: Make installSet=jigdoSet
This came out of discussion in: https://github.com/projectatomic/rpm-ostree/issues/1132

Let's simplify some logic and download/import even unused packages;
i.e. packages which do not provide any content objects.

A lot of the higher level logic wants to reference what I'm going to start
calling the "installSet" i.e. the packages in the rpmdb in the commit. So it's
simpler if the "jigdoSet" is exactly the same thing as the "installSet", and the
cost is pretty small.

Closes: #1140
Approved by: jlebon
2017-12-12 19:31:36 +00:00
Colin Walters
d3e50e9a5d jigdo: v2: Use jigdoset in Requires, and commit hash in Provides
Now that we have the jigdoset in `Requires`, let's make a hard
switch to using it and drop the jigdoset from the jigdoRPM data.

One lingering concern here is that the `Requires` are not quite
as strict as what we had before; for example one apparently can't
add a `Requires:` that refers to an architecture (x86_64 vs noarch).
And a lot more strongly than that we had the repodata checksums
in the old format.  I'm still thinking of a way to use those.

But moving on, this allows us to rework the client side to do a lot more
up-front calculation before downloading the jigdoRPM. In the spirit of that, at
the same time let's add a `Provides: rpmostree-jigdo-commit(e7bdb7443d8...)` so
that we can determine ahead of time whether or not we have the actual commit.

A major change we could now take would be to download the jigdoRPM
in parallel with the jigdo set, but doing that would require
driving a lot more of the jigdo logic into the core; it'd need
to know to specially handle the jigdoRPM download.

Closes: #1140
Approved by: jlebon
2017-12-12 19:31:36 +00:00
Colin Walters
9e146f2752 commit2jigdo: Fix destructor ordering
I was looking at this while chasing what turned out to be an entirely different
bug. Since we're referencing `checksum`, let's call the interator removal last.

Closes: #1140
Approved by: jlebon
2017-12-12 19:31:36 +00:00
Colin Walters
853110f80f commit2jigdo: Print size of non-jigdoSet packages
The idea is to see how much "waste" there is in downloading the set subtraction
of "installSet - jigdoSet".

At the moment I'm actually seeing e.g. `emacs-filesystem-1:25.3-3.fc27.noarch (0
bytes)` where I expected the download size, but that's a separate bug probably
in libdnf which I'll look at later.

Closes: #1140
Approved by: jlebon
2017-12-12 19:31:36 +00:00
Colin Walters
94befb7d74 jigdo: Verify checksum of commit upfront
While we do this when writing the final object, let's do it early on
for better security.  Was just thinking about this while redoing
how we parse the jigdoRPM.

Closes: #1140
Approved by: jlebon
2017-12-12 19:31:36 +00:00
Colin Walters
ca2a651619 commit2jigdo: Synthesize Requires: for jigdo set
Having the "jigdo set" in repodata makes it so we can parallel download the
jigdo RPM with the set. However for now, I kept the jigdo set in the jigdoRPM,
since that way it'll be covered by the signature.

Also, this changes the way we inject metadata to use a magic comment string,
since trying to pass a gigantic macro to `rpmbuild` via its argv didn't work out
so well (it looks like rpmbuild eats newlines). This approach is more robust.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1132

Closes: #1140
Approved by: jlebon
2017-12-12 19:31:36 +00:00
Colin Walters
7a03fd1bc1 core: Implement parallel relabeling
This is another big task just like importing that greatly benefits
from being parallel.  While here I hit the issue that on error
we didn't wait for pending async tasks to complete; I changed things
for importing so that we do that, and used it here too.

This was almost straightforward except I spent a *lot* of time
debugging what turned out to be calling `dnf_package_get_nevra()`
in the worker threads 😢.

I'm mostly writing this to speed up unified core/jigdo, but it's also obviously
relevant on the client side.

Closes: #1137
Approved by: jlebon
2017-12-12 19:21:07 +00:00
Colin Walters
a16422484e txn: Let first error win vs cancellation
Basically since we're doing internal async ops which set the cancellable on
failure, we still want the first error to win since it'll be more useful. See
the docs for `g_task_set_check_cancellable()` for more.

Closes: #1137
Approved by: jlebon
2017-12-12 19:21:07 +00:00
Colin Walters
88cb736404 daemon/sysroot: Drop internal cancellable
I believe this is a leftover vestige, and it was adding confusion when I was
debugging `rpmostree-core.c` async ops and cancellation.

Now the only cancellables in the daemon are created by transaction ops.

Closes: #1137
Approved by: jlebon
2017-12-12 19:21:07 +00:00
Colin Walters
52ce16d794 importer: Check cancellable early on
To exit earlier if we've been cancelled. Came up in review for parallel
relabeling.

Closes: #1137
Approved by: jlebon
2017-12-12 19:21:07 +00:00
Colin Walters
5f03181c0a core: c++ build: Use enum-NONE value instead of zero
In the whole libdnf/C++ discussion I experimented with trying to build
rpm-ostree as C++. There's a whole ton of stuff there. I'm going to punt for
now, but let's land this one change so some progress was made.

Closes: #1141
Approved by: jlebon
2017-12-12 17:26:19 +00:00
Colin Walters
802c1fcb90 compose: Fix --ex-unified-core devino cache (suid bits lost)
With unified core 🌐 we are *relying* on the devino cache
for correctness when using `bare-user` repos.  Otherwise lots
of bad things will happen as we won't hit the happy path from
[this libostree PR](https://github.com/ostreedev/ostree/pull/1297)
(I should probably add an assertion there that we aren't trying to commit
 `user.ostreemeta`).

It looks like I had this working in some of the old unified core WIP patches,
but it was lost when rebasing 🏄.

We noticed this when I was trying to deploy jigdo in FAHC and the system
wouldn't boot as various things rely on those suid transitions.

Closes: #1139
Approved by: jlebon
2017-12-12 14:03:55 +00:00
Jonathan Lebon
752166ce3b app/compose: include rpmdb pkglist in compose
We don't want to have to download all of `/usr/share/rpm` just to get
the list of packages used to compose the tree. This is fundamental
information that needs to be easier to discover. So let's stick it right
in the commit metadata. There's various use cases for this information,
including easily checking for and displaying updates and a pkglist-aware
version of `ostree log`.

Closes: #1134
Approved by: cgwalters
2017-12-08 17:39:15 +00:00
Jonathan Lebon
718596eace libpriv/util: add precondition to catch potential trap
Basically, it doesn't make sense for the caller to only want the
pkglist, but not the refsack because the former has a more limited
lifetime than the latter. Check for that to make sure nobody falls in
this trap like I did.

Closes: #1134
Approved by: cgwalters
2017-12-08 17:39:15 +00:00
Jonathan Lebon
25767893a4 app/compose: fix bad return values
We should be returning `FALSE` here, not `EXIT_FAILURE`.

Closes: #1134
Approved by: cgwalters
2017-12-08 17:39:15 +00:00
Colin Walters
7c99809a71 core: Change relabeling to use libostree's SELinux support
This function is now basically reimplementing all of the intelligence
we have in libostree today.  It wouldn't have worked before we
fixed https://github.com/ostreedev/ostree/issues/1165 but now
that's done, conceptually we just need to call
`ostree_repo_commit_modifier_set_sepolicy()`.  However, I had
to drop the "number of files changed" since currently the libostree
API doesn't support that.

Also, in order to detect the case that content changes at all (so
we still have some useful information in the journal), implement
a "content hash" check.

See also https://github.com/projectatomic/rpm-ostree/pull/1123
(And note we *don't* use `DEVINO_CANONICAL` here)

Closes: #1138
Approved by: jlebon
2017-12-08 15:01:32 +00:00
Colin Walters
0a347fefe8 core: Rework internal checkout API and relabeling error message
In preparation for multithreading relabeling; the problem I hit badly was the
`dnf_package_get_nevra()` isn't safe[1].  Since we were only using that for error
prefixing, move the error prefixing to the (currently non-MT) caller.

[1] Write it in C/C++, where multithreading is doomed to endlessly debug
    data races?
    Write it in Python, where MT isn't useful at all?
    Write it in Golang, where you face deep hostility to in-process C/C++ libs
    or really any other language?
    Or write it in Rust, where we spend a lot of time fighting the borrow checker
    and (to some degree) debugging the FFI glue to C/C++?

Closes: #1138
Approved by: jlebon
2017-12-08 15:01:32 +00:00
Colin Walters
58639683a6 jigdo2commit: Print download size
Since we can, and it's useful information.

Closes: #1131
Approved by: jlebon
2017-12-07 20:41:40 +00:00
Colin Walters
95b423afe9 Add cancel verb and DBus API
Right now the fact that one can only cancel via `Ctrl-C` of an existing client
process is rather frustrating if for example one's ssh connection to a machine
drops. Now, upon reconnecting, one can easily `rpm-ostree cancel` a hung update
or whatever rather than doing the more forcible `systemctl stop rpm-ostreed`
(which is safe of course, unless livefs is involved).

Closes: #1019
Approved by: jlebon
2017-12-07 20:12:52 +00:00
Colin Walters
51c5591ced core: Implement parallel importing
Depends: https://github.com/ostreedev/ostree/pull/1358

For jigdo ♲📦 in order to get true image speed like libostree has we need to
interleave and parallelize downloading and importing.

The messy part about this is having sync API do the "invoke and wait on various
async tasks" pattern. It's the same thing in `ostree_repo_pull_with_options()`.

Importing is pretty dramatically faster with this, I can only imagine the speed
win if we actually interleaved with downloads. However doing that requires
libdnf/librepo work.

Closes: #1124
Approved by: jlebon
2017-12-07 19:44:19 +00:00
Colin Walters
1c0e354571 importer: Rework API
Now that the importer *only* imports into OSTree repos, let's
clean up the API so that the `OstreeRepo` and `OstreeSePolicy`
are passed as constructor args.

Also rework things so there's only one constructor API that
steals the fd.

This is prep for adding another async import API.

Closes: #1124
Approved by: jlebon
2017-12-07 19:44:19 +00:00
Colin Walters
c3b152f3ee jigdo: Add Provides: rpmostree-jigdo(v1), require it on client
We are going to want versioning on the jigdo RPMs, since it's
highly likely things change.

This is done via new magic '-D rpmostree_jigdo_meta` macro, which we can also
use for other things down the line.

Closes: #1135
Approved by: jlebon
2017-12-07 18:32:49 +00:00
Colin Walters
44b938a6ac commit2jigdo: Query objsize from correct repo when building objid map
I have hit multiple times the bug with `bare-user` repos:
https://github.com/ostreedev/ostree/pull/1297

(Need to rebuild my dev container rather than relying on an updated
 `sudo make install` from ostree git)

Anyways if things are out of sync, this bug would manifest as the
main repo wouldn't have the object.

Closes: #1135
Approved by: jlebon
2017-12-07 18:32:49 +00:00
Colin Walters
25cf4589e1 commit2jigdo: Add some error prefixing
As usual doing this after I hit an error.

Closes: #1135
Approved by: jlebon
2017-12-07 18:32:49 +00:00
Colin Walters
d99175f664 status: Prefix ostree refspecs with ostree://
In preparation for jigdo, which would be like `jigdo://`.

Closes: #1136
Approved by: jlebon
2017-12-07 13:48:08 +00:00
Colin Walters
906a8a14e0 jigdo2commit: Change input to repoid:name
Let's "repo bind" the OIRPM by default; this makes the rpm-md repo feel a lot
more like an ostree remote, and IMO is just a really good idea in general to
increase predictabilty.

Closes: #1130
Approved by: jlebon
2017-12-05 13:52:21 +00:00
Colin Walters
90f9fe80e4 scripts: Drop most capabilities
Note this PR requires [bubblewrap 0.2.0](https://github.com/projectatomic/bubblewrap/releases/tag/v0.2.0).

Change our bwrap invocations drop truly dangerous capabilities like
`cap_sys_admin` and `cap_sys_module` just like Docker does today. Because of the
popularity of Docker, we can be pretty sure that most RPM scripts should have
adapted to this (although a problematic area here is that traditional librpm
doesn't actually error out if scripts fail).

There are two reasons to do this:

 - We want "offline" updates by default; updates shouldn't affect the
   running system.  If we prepare the new root in the background, a
   %post shouldn't restart a service for example.  We already "handle"
   this by making `systemctl` a symlink to `/bin/true`, but this approach
   also shuts off `%post`s that do e.g. `insmod`.
 - Protection against accidental system damage

Closes: #1099
Approved by: jlebon
2017-12-05 02:54:23 +00:00
Colin Walters
b21d0ffc95 Release 2017.11
Time to cut a new one with the jigdo work, perf improvements etc.

Closes: #1129
Approved by: jlebon
2017-12-04 21:20:28 +00:00
Colin Walters
dded5c0fdc core: Use DEVINO_CANONICAL for pkglayer if policy unchanged
There's a lot more details in the libostree PR:
https://github.com/ostreedev/ostree/pull/1357

Basically loading the xattrs is slow; let's only do it if we need to, and "need
to" is defined by "SELinux policy changed". On my test F27AH VM, the difference
between a stat() + hash table lookup versus the full xattr load on my test case
of rpm-ostree install ./tree-1.7.0-10.fc27.x86_64.rpm is absolutely dramatic;
consistently on the order of 10s without this support, and <1s with (800ms).

Closes: #1123
Approved by: jlebon
2017-12-04 20:13:34 +00:00