Commit Graph

151 Commits

Author SHA1 Message Date
Colin Walters
7c99809a71 core: Change relabeling to use libostree's SELinux support
This function is now basically reimplementing all of the intelligence
we have in libostree today.  It wouldn't have worked before we
fixed https://github.com/ostreedev/ostree/issues/1165 but now
that's done, conceptually we just need to call
`ostree_repo_commit_modifier_set_sepolicy()`.  However, I had
to drop the "number of files changed" since currently the libostree
API doesn't support that.

Also, in order to detect the case that content changes at all (so
we still have some useful information in the journal), implement
a "content hash" check.

See also https://github.com/projectatomic/rpm-ostree/pull/1123
(And note we *don't* use `DEVINO_CANONICAL` here)

Closes: #1138
Approved by: jlebon
2017-12-08 15:01:32 +00:00
Colin Walters
95b423afe9 Add cancel verb and DBus API
Right now the fact that one can only cancel via `Ctrl-C` of an existing client
process is rather frustrating if for example one's ssh connection to a machine
drops. Now, upon reconnecting, one can easily `rpm-ostree cancel` a hung update
or whatever rather than doing the more forcible `systemctl stop rpm-ostreed`
(which is safe of course, unless livefs is involved).

Closes: #1019
Approved by: jlebon
2017-12-07 20:12:52 +00:00
Colin Walters
90f9fe80e4 scripts: Drop most capabilities
Note this PR requires [bubblewrap 0.2.0](https://github.com/projectatomic/bubblewrap/releases/tag/v0.2.0).

Change our bwrap invocations drop truly dangerous capabilities like
`cap_sys_admin` and `cap_sys_module` just like Docker does today. Because of the
popularity of Docker, we can be pretty sure that most RPM scripts should have
adapted to this (although a problematic area here is that traditional librpm
doesn't actually error out if scripts fail).

There are two reasons to do this:

 - We want "offline" updates by default; updates shouldn't affect the
   running system.  If we prepare the new root in the background, a
   %post shouldn't restart a service for example.  We already "handle"
   this by making `systemctl` a symlink to `/bin/true`, but this approach
   also shuts off `%post`s that do e.g. `insmod`.
 - Protection against accidental system damage

Closes: #1099
Approved by: jlebon
2017-12-05 02:54:23 +00:00
Jonathan Lebon
7056e6b726 daemon: Unify pkgcache with system repo
We originally needed the pkgcache to be a separate repo due to ostree's
overzealous pruning policies. The idea was to maintain multiple commits
in each pkg branch for different SELinux policies. In practice, there's
not much use in maintaining old copies and it's just easier to always
relabel on the fly. So then, the need for a separate repo completely
melts away.

This helps simplify the mental model a bit and allows us to avoid subtle
issues like #1047. Note however that the core is still capable of
handling split repos for the `--ex-unified-core` compose use case. Once
that and the jigdo work are a bit more settled, we can have a clearer
picture of how to simplify the core further.

The tricky bit is migrating the cache. When deploying, we check if a
pkgcache repo exists and migrate its refs if so. We then leave behind a
symlink to the system repo to remain compatible with older rpm-ostrees.

Closes: #1055
Approved by: cgwalters
2017-12-01 15:41:40 +00:00
Colin Walters
b68fe099ea tests: Use ostree commit --consume
Just noticed this while working on something else; we should use it where
possible.

Closes: #1117
Approved by: jlebon
2017-11-29 15:07:25 +00:00
Colin Walters
1c5c4a457e tests: Correctly error out if some part of install.sh fails
It took me way, way, way too long to debug that my dev container somehow missing
`ostree-grub2` caused the script to fail early, but we'd still continue trying
to sync, leading to us not actually changing the installed `rpm-ostree` binary,
leading to test suite failures.

Also add the chmod hammer at the top so we can delete even if we fail after
doing an rsync.

Closes: #1115
Approved by: jlebon
2017-11-29 14:42:25 +00:00
Colin Walters
94a1776fe7 unpacker: Error out when trying to install RPM in /usr/local
This came up with `https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-license-9-0-9.0.176-1.x86_64.rpm`.
Reported by @dustymabe on IRC.

Closes: #1090
Approved by: jlebon
2017-11-08 03:56:20 +00:00
Colin Walters
407bfd5bc1 Lift 'override' out of experimental
We have a *lot* of experimental functionality.  I think the
`override` bits are fleshed out enough now that we can lift
the `ex` designation.  For example, jlebon fixed SELinux
labeling in the presence of override-replace.

Closes: #1089
Approved by: jlebon
2017-11-08 03:35:08 +00:00
Ruixin Bao
078d82e8d0 tests/vmcheck: add deploy-index and proc-cmd test for kargs
Refined a bit for the previous written kargs test.( mainly for
checking kargs after rollback )

Added tests for import-proc-cmdline and deploy-index option

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
27de772c4f libpriv/vmcheck: support replace option for karg command
Refactor the similar code portion from delete option into
a new function, and reuse it in the replace logic.

Add the replace functionality to allow kernel arguments
replacement. Now support two different types of replacement:

1: replace by key=value when there is only one key existing
e.g we have 'hi=hello' as our one of the arguments (hi only has
one value associated with it) , and 'rpm-ostree ex kargs --replace hi=new'
will replace the value and change hi=hello into hi=new.

2: replace by key=value=new_value for all other replacements
( this will work for swapping single value pair too !)

Some tests for rpm-ostree ex kargs --replace are added
for future regression.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
09cf438f04 vmcheck: Add tests for rpm-ostree ex kargs delete and append
Tests are introduced in this commit to test the basic
functionality for rpm-ostree ex kargs --append,
and rpm-ostree ex kargs --delete.

Those tests are added for future regression.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Colin Walters
cb9889204c lib/unpacker: Add repo.timestamp to metadata
Related to: https://github.com/projectatomic/rpm-ostree/issues/774

We aren't yet trying to render this in any way on the client side, but let's
start capturing the timestamp data now so that we can make use of it later
in e.g. `rpm-ostree status -v`.

Closes: #1072
Approved by: jlebon
2017-10-25 18:18:10 +00:00
Colin Walters
1f31cbb99a status: Render ostree metadata source-title key
Depends: https://github.com/ostreedev/ostree/pull/1296

As I mention in the commit there, I see two uses for this in rpm-ostree; first
in our test suite, and second for OCI-built image imports.

I also took a step further here and inject an `original-origin` metadata
key, though we aren't actually using that yet.  The problem I'm trying
to solve there is that repeated `make vmoverlay` starts chaining things up,
but that gets very confusing.  I think we should always have `vmoverlay` unwind
back to the base ref.  (Or at least do that by default)

Closes: #1069
Approved by: jlebon
2017-10-25 03:37:05 +00:00
Colin Walters
9ec45d3c89 daemon: Add commit metadata to deployment properties
In another PR I did the manual bridging of commit metadata to deployment
property, but that's annoying. Let's just bridge all commit metadata.

Closes: #1069
Approved by: jlebon
2017-10-25 03:37:05 +00:00
Jonathan Lebon
e49f7cdd81 app: add --download-only option
To complement the new `--cache-only` option, add a `--download-only`
option. This does exactly what it says: we download the ostree, download
and import packages, but don't actually commit & deploy. This can be
used to effectively prime a follow-up `--cache-only` operation that can
be done during a more convenient/safer maintenance window.

I debated naming the two options `--pull-only` and `--deploy-only` like
the ostree equivalents. Though "pull" felt like the wrong word given
that it's associated more with ostree pulling but rpm-ostree also
downloads & imports RPMs. As for `--deploy-only` vs `--cache-only`, it
seems like `--cache-only` is a more accurate description of the
functionality (i.e. rather than describing an action, it describes a
mode). I also considered `--no-download` to make the synergy with
`--download-only` more obvious. Maybe that's better? Naming is hard...

Closes: #713

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
24c184af03 app: imply cache-only mode for pkgs removals and uninstalls
Now that we have a strong notion of `cache-only` mode, make use of it
when performing an `uninstall` or `ex override remove/reset`.

Closes: #944

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
b811eb61c0 app: support full offline operations with --cache-only
As Colin mentioned in #1035, the new `--cache-only` implemented only the
rpmmd half of the story. Here we complete that story by also ensuring
that when in cache-only mode, we don't download new ostree data nor new
packages. We try to complete the requested operation with what we have.

To do this, we add support for the same `SYNTHETIC` pull that was added
in ostree[1] so that we don't actually pull, but still perform timestamp
checking.

On the pkgcache side, we disable all remote repos and instead insert all
our cached RPMs into the `DnfSack`. Care is taken to still perform
SHA256 verification for local pkg installs/replacements.

[1] https://github.com/ostreedev/ostree/pull/642

Closes: #687

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
e5e9f1f0b8 libvm: factor out helpers to start httpd
This will be used in other tests. Plus, it makes it much nicer to use.

Closes: #1053
Approved by: cgwalters
2017-10-13 15:00:25 +00:00
Jonathan Lebon
74eaa2ba59 daemon: add RPMOSTREE_DEBUG_DISABLE_DAEMON_IDLE_EXIT
The new idle exit behaviour is nice, but it makes debugging it harder
because you have to be fast enough to attach or place your breakpoints
and trigger it before it auto-exits. Add a compile-time flag that
developers can easily turn on to disable the auto-exit behaviour.

Closes: #1052
Approved by: cgwalters
2017-10-12 15:35:44 +00:00
Colin Walters
4564646629 tests/vmcheck: Wait for HTTP server to start
Should fix a flake.

Closes: #1051
Approved by: jlebon
2017-10-11 21:22:19 +00:00
Jonathan Lebon
5867d2a2af vmcheck: use systemd-run for running post-that-hangs
Just taking what I learned from #1035 and applying it here. What's nice
about this is that there's no cleanup needed. Once the process is killed
(or worst case, we reboot the VM), there's no traces left at all.

Also added a few extra "ok" outputs.

Closes: #1043
Approved by: cgwalters
2017-10-06 20:17:36 +00:00
Jonathan Lebon
dd7c748ce1 tests/vmcheck: add test for refresh-md and --cache-only
Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
dc08ebda0e tests/libvm: use rsync and add yumrepo mode
I've been lazy about actually using using rsync instead of scp when
copying new RPMs over to the VM. We do this here. Also make
`vm_send_test_repo` take a mode argument that allows callers to
completely skip the sending of the repo file itself. This will be needed
for the `makecache` test, in which we *don't* want the repo to be local.
It looks cleaner anyway for the gpgcheck use case as well.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Colin Walters
f920f36093 tests: Use --parent=none rather than deleting the ref
This is atomic. Not that it matters for this test, but let's have our tests use
ostree well.

Closes: #1037
Approved by: jlebon
2017-10-05 15:03:06 +00:00
Colin Walters
95227f079a WIP: Implement ex livefs --replace
Lots of tradeoffs in this.  See the comments in the code for
more info.  WIP for now.

Closes: #1028
Approved by: jlebon
2017-10-05 13:19:20 +00:00
Colin Walters
3f367dbce2 scripts+bwrap: Make script execution cancellable
Prep for implementing `rpm-ostree cancel`, but this works with the way we handle
`Ctrl-C` interactively on a client as well. Being able to cancel a script
execution is quite nice; some of them are expensive, and having one loop forever
has been known to happen.

Closes: #1025
Approved by: jlebon
2017-10-03 14:24:07 +00:00
Jonathan Lebon
a9c38d33b8 vmcheck: add SELinux labeling tests
There's a lot of paths in the core related to SELinux policy changes and
relabeling packages. We currently have no test coverage for them. We add
support in the test libraries here to build such packages.

We also add a test that checks both that we correctly relabel RPMs when
the policy changes and that we handle layered packages that install
SELinux packages properly.

Closes: #999
Approved by: cgwalters
2017-10-03 01:01:14 +00:00
Jonathan Lebon
8b1001c9b5 tests/vmcheck: turn on strict bash mode
Closes: #1032
Approved by: cgwalters
2017-10-02 20:36:07 +00:00
Jonathan Lebon
cb7e84c4a6 libpriv/scripts: redirect scriptlet output to journal
Rather than just letting the scriptlets inherit the daemon's
stdout/stderr, redirect their outputs so that we can set a customized
identifier to make it easier to distinguish from the daemon output.

Also print out the `journalctl` command needed so that users can
investigate the output themselves.

Closes: #998
Approved by: cgwalters
2017-09-27 19:01:45 +00:00
Jonathan Lebon
c92ff926f6 app/status: cleaner diff output for replaced pkgs
I suspect a common pattern with local replacement overrides is to
simultaneously replace a group of packages that depend on each other in
one shot, as is the case with docker, docker-common, and
docker-rhel-push-plugin currently in Fedora Atomic Host. In such cases,
we can print a cleaner diff in the status to make it easier to grok.

Before:

     ReplacedBasePackages: strace 4.18-1.fc26 -> 4.19-1.fc26, docker-common 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, docker 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, docker-rhel-push-plugin 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26

After:

     ReplacedBasePackages: docker-common docker docker-rhel-push-plugin 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, strace 4.18-1.fc26 -> 4.19-1.fc26

Closes: #1004
Approved by: cgwalters
2017-09-27 16:22:50 +00:00
Colin Walters
13f307da36 livefs: Update /usr/lib/{passwd,group}, run systemd-tmpfiles
This gets us to supporting live installs of `httpd` and many
other services.  I also tried `postgresql-server` for example.

Closes: https://github.com/projectatomic/rpm-ostree/issues/962

Closes: #996
Approved by: jlebon
2017-09-25 15:10:15 +00:00
Jonathan Lebon
167fd4c1dd libpriv/scripts: use rofiles-fuse on /etc too
We were directly bind mounting the checked out `/usr/etc` onto `/etc`
which was exposing us to corruption from scriptlets. Since we already
have an rofiles-fuse mount for `/usr`, let's just re-use its `etc/`
subdir and bind mount that instead.

Closes: #1003
Approved by: cgwalters
2017-09-24 13:11:21 +00:00
Jonathan Lebon
f089b8de1f libpriv/postprocess: also delete semanage lock files
We don't need those in the tree, so let's nuke them. This also fixes
subtle compatibility issues between hardlinks and lock files (see #999).

Closes: #1002
Approved by: cgwalters
2017-09-21 15:28:55 +00:00
Colin Walters
80de17623f tests/livefs: Make sure /usr/bin/ls is executable
Oh man was I confused how this happened; thought it was a bug in some livefs
changes I was working on.

Closes: #995
Approved by: jlebon
2017-09-15 19:10:24 +00:00
Ruixin
56ef177235 tests/core: allow merging identical files during installation
This PR uses https://github.com/ostreedev/ostree/pull/1156
to ensure that when installing packages containing files with
exact same content, the files will be merged.

When installing packages containing conflicting files,
the error will still be detected at the ostree side.

The checkout overwrite option at rpm-ostree side is also
modified to accomodate the changes made in ostree side.

A test is added for regression
2017-09-13 20:41:15 +00:00
Jonathan Lebon
4ad3ea58c6 app: smarter deployment change detection
Commands like `upgrade` and `deploy` need to know if a new deployment
was actually laid down so that it may print a pkg diff if so. This is
implemented by listening for changes to the DefaultDeployment D-Bus
property. D-Bus emits a signal when the deployment variant changes
value.

However, in #595, with the introduction of `pending-*` related keys, the
deployment variant no longer represents data solely tied to that
specific deployment. In this case, because `deploy` operations currently
set the ref to the resolved checksum, it can happen that deploying the
same base commit when the current refspec *isn't* pointing to that base
commit will result in the `pending-*` keys dropping out and a default
deployment change notification going out.

In this patch, we strengthen how we determine whether a new deployment
was laid down by actually looking at the deployment id, rather than just
assuming that a change to the property implies a new deployment.

Closes: #981

Closes: #984
Approved by: cgwalters
2017-09-12 17:23:25 +00:00
Jonathan Lebon
077d7c1a9e daemon/deploy: allow redeploying livefs booted commit
Also something I noticed while working on #981. When sitting on a livefs
commit, once a user does `rpm-ostree cleanup --pending --rollback`, it's
impossible to redeploy the same booted commit. Let's allow users to do
this.

Closes: #984
Approved by: cgwalters
2017-09-12 17:23:25 +00:00
Colin Walters
150de0087e scripts: Support overriding lua, use for glibc-all-langpacks
Today in Fedora the `glibc-all-langpacks.posttrans` is implemented
in lua, for no good reason.  See:
https://bugzilla.redhat.com/show_bug.cgi?id=1367585

Since that's stalled out, let's add support for overrides.  This
is obviously a much bigger step with more long term maintenance
implications over our current "ignore scripts" list.  But we can't
block either.

This is needed for unified core work:
https://github.com/projectatomic/rpm-ostree/issues/729

(We also override `fedora-release-atomichost` but I'll likely
 submit a patch for that upstream)

Closes: #980
Approved by: jlebon
2017-09-08 18:02:22 +00:00
Colin Walters
e40eaebbe0 core: Use fd-relative access to rpmdb
I was linking to this code from elsewhere and noticed that
for our hardlink breaks we were not using fd-relative even
though we can.  Down the line if we fork librpm into a separate
process and do e.g. `--dbpath=.` it'll do it too.

(Side note, I verified that commenting out the hardlink breaking
 here was caught by the `ostree fsck` I added to the test suite)

Closes: #979
Approved by: jlebon
2017-09-07 22:54:40 +00:00
Colin Walters
b5c98ec4f1 daemon: Add a policy for reload-config
This fixes `rpm-ostree reload` as root, and supports configuring
it to be enabled for other users as well.  This was overlooked
in the polkit work originally.

Closes: https://github.com/projectatomic/rpm-ostree/issues/976

Closes: #977
Approved by: jlebon
2017-09-07 15:11:00 +00:00
Jonathan Lebon
b0c75ae900 vmcheck: use --selinux-policy when committing
Let's make SELinux and atomic-host-tests happy with trees we concoct
ourselves by using the new --selinux-policy. (Specifically, we want our
sync'ed binaries to have install_exec_t).

Closes: #953
Approved by: cgwalters
2017-09-07 14:41:19 +00:00
Jonathan Lebon
50ecc04cb6 overlay.sh: drop hack for backup passwd files
We shouldn't need this anymore.

Closes: #953
Approved by: cgwalters
2017-09-07 14:41:19 +00:00
Jonathan Lebon
b22d7d764d vmcheck: rename build.sh to install.sh
I had meant for this to be in the other PR#968. I originally did both
`make` and `make install` there, but now it only does `make install`, so
let's just rename it to make that more obvious.

Closes: #953
Approved by: cgwalters
2017-09-07 14:41:19 +00:00
Colin Walters
c093a587a2 tests: Add a test case for lua scripts
This is currently unsupported, let's add a test case. Prep for support for
script overrides.

Closes: #973
Approved by: jlebon
2017-09-07 14:09:47 +00:00
Colin Walters
20abc85243 unpacker: Translate /boot → /usr/lib/ostree-boot
At one point `rpm-ostree install libvirt` dragged in libguestfs which in turn
brought in `syslinux-extlinux-nonlinux` which has files in `/boot/extlinux`,
which we rejected.  (That dependency chain appears to have been fixed currently)

For the general case, this is just a partial fix in that we haven't nailed down
the semantics of how updates for `/boot` work.  But in this particular case,
we'll just break libguestfs' `extlinux` verb, which I'm OK with.

Another case is `fwupdate-efi` - we require manual intervention to copy the
data into `/boot` after installing the package.

This is also preparation for [unified core](https://github.com/projectatomic/rpm-ostree/issues/729)
in that we now ensure imported kernels don't end up in `/boot` unless
explicitly configured.

Closes: https://github.com/projectatomic/rpm-ostree/issues/853

Closes: #969
Approved by: jlebon
2017-09-07 13:30:36 +00:00
Ruixin Bao
d6b3217a86 lib/core: Detect file name conflict before rpmdb
This PR uses https://github.com/ostreedev/ostree/pull/1116
to ensure that when installing packages containing conflicting files, the
error will be detected at the ostree side, rather than when
writing the rpm database.

Closes: https://github.com/projectatomic/rpm-ostree/issues/365

Closes: #974
Approved by: cgwalters
2017-09-07 13:13:12 +00:00
Jonathan Lebon
7fbf9f32d4 hacking: make it easier to use a custom tree
Let's make using a custom install tree easier and document the process.
We split out the insttree step into `build.sh` so that we no longer have
to `flock(1)` around it, and also share between `overlay.sh` and
`sync.sh`.

Closes: #968
Approved by: cgwalters
2017-09-01 19:58:55 +00:00
Colin Walters
3047513509 core: Run %post before %posttrans
While working on unified core and the Fedora Atomic Host content set, I hit a
dependency between `docker.posttrans` which tries to read `/etc/os-release`, and
`fedora-release-atomichost.post` which creates that symlink.

It seems best practice to me to run `%post`s strictly before
`%posttrans`; we're not likely to do parallelization anytime
soon anyways.

While here I cleaned things up by having an enum for the script kind,
rather than multiple functions, otherwise we would have had another
wrapper in core.c.

Closes: #963
Approved by: jlebon
2017-08-30 15:50:09 +00:00
Colin Walters
82f95e7761 daemon/upgrader: Use new libostree timestamp-check option
Since we have a copy of this libostree code, pick up the new
changes from <https://github.com/ostreedev/ostree/pull/1055>.

Note the added test doesn't really test our logic since
we're only doing local pulls, but at least we have something.

Closes: #932
Approved by: jlebon
2017-08-22 16:08:21 +00:00
Colin Walters
b6705f3feb daemon: Check for updated rpms when upgrading
This closes a longstanding bug - since package layering first
landed, we only checked for newer RPMs if the base tree changed.
In some scenarios like RHELAH, this doesn't matter much by default
since they move at the same cadence.  Except if you use EPEL for example.
In Fedora, today the FAH releases are async of the rpm-md repos, and
there's also COPR which can update more than once a day even.

We should check for both update sources. Luckily we'd already introduced logic
for this in the treecompose case (checksumming the depsolved package sack). We
just need to start using it for client side assembly too.

Closes: https://github.com/projectatomic/rpm-ostree/issues/391

Closes: #911
Approved by: jlebon
2017-08-16 15:33:11 +00:00