Commit Graph

255 Commits

Author SHA1 Message Date
Jonathan Lebon
b811eb61c0 app: support full offline operations with --cache-only
As Colin mentioned in #1035, the new `--cache-only` implemented only the
rpmmd half of the story. Here we complete that story by also ensuring
that when in cache-only mode, we don't download new ostree data nor new
packages. We try to complete the requested operation with what we have.

To do this, we add support for the same `SYNTHETIC` pull that was added
in ostree[1] so that we don't actually pull, but still perform timestamp
checking.

On the pkgcache side, we disable all remote repos and instead insert all
our cached RPMs into the `DnfSack`. Care is taken to still perform
SHA256 verification for local pkg installs/replacements.

[1] https://github.com/ostreedev/ostree/pull/642

Closes: #687

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
e5e9f1f0b8 libvm: factor out helpers to start httpd
This will be used in other tests. Plus, it makes it much nicer to use.

Closes: #1053
Approved by: cgwalters
2017-10-13 15:00:25 +00:00
Jonathan Lebon
74eaa2ba59 daemon: add RPMOSTREE_DEBUG_DISABLE_DAEMON_IDLE_EXIT
The new idle exit behaviour is nice, but it makes debugging it harder
because you have to be fast enough to attach or place your breakpoints
and trigger it before it auto-exits. Add a compile-time flag that
developers can easily turn on to disable the auto-exit behaviour.

Closes: #1052
Approved by: cgwalters
2017-10-12 15:35:44 +00:00
Colin Walters
4564646629 tests/vmcheck: Wait for HTTP server to start
Should fix a flake.

Closes: #1051
Approved by: jlebon
2017-10-11 21:22:19 +00:00
Colin Walters
334f0b89be ex-container: Make /usr/etc/{,g}shadow user-readable
For the `ex container` case, there's no security issues here; one shouldn't be
doing user management in these roots at all.

This is for work on exporting `ex container` roots to OCI as non-root. Without
this fix, libostree just tries to `openat()` the object for export to tar, and
fails.

See also https://github.com/projectatomic/rpm-ostree/issues/1045

Closes: #1046
Approved by: jlebon
2017-10-10 20:42:55 +00:00
Jonathan Lebon
5867d2a2af vmcheck: use systemd-run for running post-that-hangs
Just taking what I learned from #1035 and applying it here. What's nice
about this is that there's no cleanup needed. Once the process is killed
(or worst case, we reboot the VM), there's no traces left at all.

Also added a few extra "ok" outputs.

Closes: #1043
Approved by: cgwalters
2017-10-06 20:17:36 +00:00
Jonathan Lebon
dd7c748ce1 tests/vmcheck: add test for refresh-md and --cache-only
Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
dc08ebda0e tests/libvm: use rsync and add yumrepo mode
I've been lazy about actually using using rsync instead of scp when
copying new RPMs over to the VM. We do this here. Also make
`vm_send_test_repo` take a mode argument that allows callers to
completely skip the sending of the repo file itself. This will be needed
for the `makecache` test, in which we *don't* want the repo to be local.
It looks cleaner anyway for the gpgcheck use case as well.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Colin Walters
f920f36093 tests: Use --parent=none rather than deleting the ref
This is atomic. Not that it matters for this test, but let's have our tests use
ostree well.

Closes: #1037
Approved by: jlebon
2017-10-05 15:03:06 +00:00
Colin Walters
95227f079a WIP: Implement ex livefs --replace
Lots of tradeoffs in this.  See the comments in the code for
more info.  WIP for now.

Closes: #1028
Approved by: jlebon
2017-10-05 13:19:20 +00:00
Colin Walters
3f367dbce2 scripts+bwrap: Make script execution cancellable
Prep for implementing `rpm-ostree cancel`, but this works with the way we handle
`Ctrl-C` interactively on a client as well. Being able to cancel a script
execution is quite nice; some of them are expensive, and having one loop forever
has been known to happen.

Closes: #1025
Approved by: jlebon
2017-10-03 14:24:07 +00:00
Jonathan Lebon
a9c38d33b8 vmcheck: add SELinux labeling tests
There's a lot of paths in the core related to SELinux policy changes and
relabeling packages. We currently have no test coverage for them. We add
support in the test libraries here to build such packages.

We also add a test that checks both that we correctly relabel RPMs when
the policy changes and that we handle layered packages that install
SELinux packages properly.

Closes: #999
Approved by: cgwalters
2017-10-03 01:01:14 +00:00
Jonathan Lebon
8b1001c9b5 tests/vmcheck: turn on strict bash mode
Closes: #1032
Approved by: cgwalters
2017-10-02 20:36:07 +00:00
Jonathan Lebon
2c94b11b68 libvm: don't rsync $VMCHECK_INSTTREE
We pretty standardized on insttree/ being the installation tree, and
we don't actually support $VMCHECK_INSTTREE anymore.

Closes: #1032
Approved by: cgwalters
2017-10-02 20:36:07 +00:00
Colin Walters
1acd834104 Add test infra for ex container, and one test for bash
We have some unit-style tests that run `ex container`, but
they aren't "real"; they don't use scripts for example.  Let's
add tests for this similar to `tests/compose`.

We use a 26 base, but the target repos need to be 27
to pick up the fix for: https://bugzilla.redhat.com/show_bug.cgi?id=1478172

Add some bits to share infra between `tests/compose` and `tests/ex-container`;
basically handling the rpmmd repos. I tweaked things to be more streamlined
there between the `.papr.yml` and the test script.

Right now this is just one test for `bash`, but lays some of the infrastructure
for doing more. One thing that we need to do to improve more here is to better
cache RPMs, a bit like the compose tests do.

Closes: #1024
Approved by: jlebon
2017-10-02 18:04:24 +00:00
Colin Walters
57937d0e63 libtest: Squash a race in temp webserver setup
Saw this in one of the PR contexts.

Closes: #1031
Approved by: jlebon
2017-10-02 18:02:40 +00:00
Jonathan Dieter
fd6109af1d compose: Copy in additional files before postprocessing
Sometimes it's useful to have access to the additional files when running
the post script, so this re-orders the compose process to copy the
additional files in before the post script runs

Signed-off-by: Jonathan Dieter <jdieter@lesbg.com>

Closes: #997
Approved by: jlebon
2017-09-28 15:38:55 +00:00
Colin Walters
2e59a2748d postprocess: Allow add-files into /etc
Even though it's really `/usr/etc`. This is for greater consistency with
`postprocess-script` where it appears as `/etc`.

Closes: #997
Approved by: jlebon
2017-09-28 15:38:55 +00:00
Jonathan Lebon
cb7e84c4a6 libpriv/scripts: redirect scriptlet output to journal
Rather than just letting the scriptlets inherit the daemon's
stdout/stderr, redirect their outputs so that we can set a customized
identifier to make it easier to distinguish from the daemon output.

Also print out the `journalctl` command needed so that users can
investigate the output themselves.

Closes: #998
Approved by: cgwalters
2017-09-27 19:01:45 +00:00
Jonathan Lebon
c92ff926f6 app/status: cleaner diff output for replaced pkgs
I suspect a common pattern with local replacement overrides is to
simultaneously replace a group of packages that depend on each other in
one shot, as is the case with docker, docker-common, and
docker-rhel-push-plugin currently in Fedora Atomic Host. In such cases,
we can print a cleaner diff in the status to make it easier to grok.

Before:

     ReplacedBasePackages: strace 4.18-1.fc26 -> 4.19-1.fc26, docker-common 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, docker 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, docker-rhel-push-plugin 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26

After:

     ReplacedBasePackages: docker-common docker docker-rhel-push-plugin 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, strace 4.18-1.fc26 -> 4.19-1.fc26

Closes: #1004
Approved by: cgwalters
2017-09-27 16:22:50 +00:00
Colin Walters
58f8f892fb lib/postprocess: Use O_TMPFILE, not O_APPEND for tmpfiles.d writing
The comment here was wrong; we don't rely on `O_APPEND` here for package
layering since we convert on import.  I noticed this while I was doing
a grep for `O_APPEND` in the codebase as part of unified core work.

Fix this by converting to `O_TMPFILE`+`GLNX_LINK_TMPFILE_NOREPLACE`.

Prep for unified core.

Closes: #1009
Approved by: jlebon
2017-09-26 18:27:00 +00:00
Colin Walters
13f307da36 livefs: Update /usr/lib/{passwd,group}, run systemd-tmpfiles
This gets us to supporting live installs of `httpd` and many
other services.  I also tried `postgresql-server` for example.

Closes: https://github.com/projectatomic/rpm-ostree/issues/962

Closes: #996
Approved by: jlebon
2017-09-25 15:10:15 +00:00
Colin Walters
e066dc93c8 tests: Skip running introspection tests if built with ASAN
This is the rpm-ostree equivalent of
<47b4dd1b38>

Unfortunately, introspection uses `dlopen(), which doesn't quite
work when the DSO is compiled with ASAN but the outer executable
isn't.

Prep for syncing PAPR config with ostree.

Closes: #1000
Approved by: jlebon
2017-09-24 13:32:00 +00:00
Jonathan Lebon
167fd4c1dd libpriv/scripts: use rofiles-fuse on /etc too
We were directly bind mounting the checked out `/usr/etc` onto `/etc`
which was exposing us to corruption from scriptlets. Since we already
have an rofiles-fuse mount for `/usr`, let's just re-use its `etc/`
subdir and bind mount that instead.

Closes: #1003
Approved by: cgwalters
2017-09-24 13:11:21 +00:00
Jonathan Lebon
f089b8de1f libpriv/postprocess: also delete semanage lock files
We don't need those in the tree, so let's nuke them. This also fixes
subtle compatibility issues between hardlinks and lock files (see #999).

Closes: #1002
Approved by: cgwalters
2017-09-21 15:28:55 +00:00
Colin Walters
80de17623f tests/livefs: Make sure /usr/bin/ls is executable
Oh man was I confused how this happened; thought it was a bug in some livefs
changes I was working on.

Closes: #995
Approved by: jlebon
2017-09-15 19:10:24 +00:00
Ruixin
56ef177235 tests/core: allow merging identical files during installation
This PR uses https://github.com/ostreedev/ostree/pull/1156
to ensure that when installing packages containing files with
exact same content, the files will be merged.

When installing packages containing conflicting files,
the error will still be detected at the ostree side.

The checkout overwrite option at rpm-ostree side is also
modified to accomodate the changes made in ostree side.

A test is added for regression
2017-09-13 20:41:15 +00:00
Jonathan Lebon
4ad3ea58c6 app: smarter deployment change detection
Commands like `upgrade` and `deploy` need to know if a new deployment
was actually laid down so that it may print a pkg diff if so. This is
implemented by listening for changes to the DefaultDeployment D-Bus
property. D-Bus emits a signal when the deployment variant changes
value.

However, in #595, with the introduction of `pending-*` related keys, the
deployment variant no longer represents data solely tied to that
specific deployment. In this case, because `deploy` operations currently
set the ref to the resolved checksum, it can happen that deploying the
same base commit when the current refspec *isn't* pointing to that base
commit will result in the `pending-*` keys dropping out and a default
deployment change notification going out.

In this patch, we strengthen how we determine whether a new deployment
was laid down by actually looking at the deployment id, rather than just
assuming that a change to the property implies a new deployment.

Closes: #981

Closes: #984
Approved by: cgwalters
2017-09-12 17:23:25 +00:00
Jonathan Lebon
077d7c1a9e daemon/deploy: allow redeploying livefs booted commit
Also something I noticed while working on #981. When sitting on a livefs
commit, once a user does `rpm-ostree cleanup --pending --rollback`, it's
impossible to redeploy the same booted commit. Let's allow users to do
this.

Closes: #984
Approved by: cgwalters
2017-09-12 17:23:25 +00:00
Jonathan Lebon
c08ca8f922 daemon/upgrader: make use of override-commit-ids
Currently, when setting the `override-commit` key in the origin, the
upgrader pulls that commit checksum directly and then updates the
refspec to point to it. This behaviour was inherited from its ostree
version; at the time it was implemented, the pull code didn't support
passing a specific commit for a given refspec. However, we now have
the override-commit-ids option, which will make libostree update the ref
for us.

We change the code here to make use of it and simplify the function.
This also fixes the corner case of local branches: we shouldn't change
the ref if we're on a local branch. This is actually what drove me to
this patch as I was debugging #981.

(Aside: I'm still not convinced updating the refspec is always the
correct thing to do even in the remote case, though it's a bit messy to
disentangle).

Closes: #984
Approved by: cgwalters
2017-09-12 17:23:25 +00:00
Colin Walters
f113fc5e27 Rework treecompose kernel processing
Prep for changing `boot_location: new` to use `/usr/lib/ostree-boot`
and `/usr/lib/modules`.  Rework our kernel postprocessing
so that we unify the `boot_location` handling with initramfs generation.

Instead of doing the initramfs first in postprocessing, we do it nearly last,
after e.g. `etc` is renamed to `usr/etc`. This has some consequences, such as
the fact that `run_bwrap_mutably()` is now called in both situations. In
general, our handling of `etc` is inconsistent, although understandably so.

As part of this, I finally got around to implementing the bit from
https://github.com/systemd/systemd/pull/4174 however suboptimal it is; need the
unified core so we can cleanly ignore the posttrans like we do others.  We
intentionally keep the file around in the generated tree so that installing a
kernel RPM per client doesn't try to do any of this either.

This all gets folded together so that the logic for handling the bootloader gets
simpler - in the Fedora case, we now know to find kernels in `/usr/lib/modules`
and can ignore `/boot`.

Closes: #959
Approved by: jlebon
2017-09-12 14:26:44 +00:00
Colin Walters
93d3fc6617 tests/compose: Add env variables to force caching and fast fail
This helped me debug/fix the tests faster.

Closes: #959
Approved by: jlebon
2017-09-12 14:26:44 +00:00
Colin Walters
150de0087e scripts: Support overriding lua, use for glibc-all-langpacks
Today in Fedora the `glibc-all-langpacks.posttrans` is implemented
in lua, for no good reason.  See:
https://bugzilla.redhat.com/show_bug.cgi?id=1367585

Since that's stalled out, let's add support for overrides.  This
is obviously a much bigger step with more long term maintenance
implications over our current "ignore scripts" list.  But we can't
block either.

This is needed for unified core work:
https://github.com/projectatomic/rpm-ostree/issues/729

(We also override `fedora-release-atomichost` but I'll likely
 submit a patch for that upstream)

Closes: #980
Approved by: jlebon
2017-09-08 18:02:22 +00:00
Colin Walters
e40eaebbe0 core: Use fd-relative access to rpmdb
I was linking to this code from elsewhere and noticed that
for our hardlink breaks we were not using fd-relative even
though we can.  Down the line if we fork librpm into a separate
process and do e.g. `--dbpath=.` it'll do it too.

(Side note, I verified that commenting out the hardlink breaking
 here was caught by the `ostree fsck` I added to the test suite)

Closes: #979
Approved by: jlebon
2017-09-07 22:54:40 +00:00
Colin Walters
b5c98ec4f1 daemon: Add a policy for reload-config
This fixes `rpm-ostree reload` as root, and supports configuring
it to be enabled for other users as well.  This was overlooked
in the polkit work originally.

Closes: https://github.com/projectatomic/rpm-ostree/issues/976

Closes: #977
Approved by: jlebon
2017-09-07 15:11:00 +00:00
Jonathan Lebon
b0c75ae900 vmcheck: use --selinux-policy when committing
Let's make SELinux and atomic-host-tests happy with trees we concoct
ourselves by using the new --selinux-policy. (Specifically, we want our
sync'ed binaries to have install_exec_t).

Closes: #953
Approved by: cgwalters
2017-09-07 14:41:19 +00:00
Jonathan Lebon
50ecc04cb6 overlay.sh: drop hack for backup passwd files
We shouldn't need this anymore.

Closes: #953
Approved by: cgwalters
2017-09-07 14:41:19 +00:00
Jonathan Lebon
b22d7d764d vmcheck: rename build.sh to install.sh
I had meant for this to be in the other PR#968. I originally did both
`make` and `make install` there, but now it only does `make install`, so
let's just rename it to make that more obvious.

Closes: #953
Approved by: cgwalters
2017-09-07 14:41:19 +00:00
Colin Walters
c093a587a2 tests: Add a test case for lua scripts
This is currently unsupported, let's add a test case. Prep for support for
script overrides.

Closes: #973
Approved by: jlebon
2017-09-07 14:09:47 +00:00
Colin Walters
20abc85243 unpacker: Translate /boot → /usr/lib/ostree-boot
At one point `rpm-ostree install libvirt` dragged in libguestfs which in turn
brought in `syslinux-extlinux-nonlinux` which has files in `/boot/extlinux`,
which we rejected.  (That dependency chain appears to have been fixed currently)

For the general case, this is just a partial fix in that we haven't nailed down
the semantics of how updates for `/boot` work.  But in this particular case,
we'll just break libguestfs' `extlinux` verb, which I'm OK with.

Another case is `fwupdate-efi` - we require manual intervention to copy the
data into `/boot` after installing the package.

This is also preparation for [unified core](https://github.com/projectatomic/rpm-ostree/issues/729)
in that we now ensure imported kernels don't end up in `/boot` unless
explicitly configured.

Closes: https://github.com/projectatomic/rpm-ostree/issues/853

Closes: #969
Approved by: jlebon
2017-09-07 13:30:36 +00:00
Ruixin Bao
d6b3217a86 lib/core: Detect file name conflict before rpmdb
This PR uses https://github.com/ostreedev/ostree/pull/1116
to ensure that when installing packages containing conflicting files, the
error will be detected at the ostree side, rather than when
writing the rpm database.

Closes: https://github.com/projectatomic/rpm-ostree/issues/365

Closes: #974
Approved by: cgwalters
2017-09-07 13:13:12 +00:00
Jonathan Lebon
7fbf9f32d4 hacking: make it easier to use a custom tree
Let's make using a custom install tree easier and document the process.
We split out the insttree step into `build.sh` so that we no longer have
to `flock(1)` around it, and also share between `overlay.sh` and
`sync.sh`.

Closes: #968
Approved by: cgwalters
2017-09-01 19:58:55 +00:00
Colin Walters
213a656ddf postprocess: Unlink our treecompose-post out of the final /bin
Another variant of <https://github.com/projectatomic/rpm-ostree/pull/822> 😭

Closes: #964
Approved by: jlebon
2017-08-31 03:06:11 +00:00
Colin Walters
3047513509 core: Run %post before %posttrans
While working on unified core and the Fedora Atomic Host content set, I hit a
dependency between `docker.posttrans` which tries to read `/etc/os-release`, and
`fedora-release-atomichost.post` which creates that symlink.

It seems best practice to me to run `%post`s strictly before
`%posttrans`; we're not likely to do parallelization anytime
soon anyways.

While here I cleaned things up by having an enum for the script kind,
rather than multiple functions, otherwise we would have had another
wrapper in core.c.

Closes: #963
Approved by: jlebon
2017-08-30 15:50:09 +00:00
Jonathan Lebon
2f6f8a910d libvm: set up ControlPath socket in /var/tmp
It turns out there's a much smaller limit than PATH_MAX for Unix
sockets. On Linux, it's 108 characters. It took me some time to figure
out why `vmcheck` would sometimes fail depending on where the src
directory is and how ${topsrcdir} is defined. Let's just make things
safer by just using /var/tmp.

Closes: #949
Approved by: cgwalters
2017-08-24 22:12:17 +00:00
Colin Walters
82f95e7761 daemon/upgrader: Use new libostree timestamp-check option
Since we have a copy of this libostree code, pick up the new
changes from <https://github.com/ostreedev/ostree/pull/1055>.

Note the added test doesn't really test our logic since
we're only doing local pulls, but at least we have something.

Closes: #932
Approved by: jlebon
2017-08-22 16:08:21 +00:00
Colin Walters
932520a0c7 tests/compose: Be a bit more verbose
This would have helped me debug an issue that caused the first compose to fail.

Closes: #931
Approved by: jlebon
2017-08-22 01:02:52 +00:00
Colin Walters
8bb604ce67 tree-wide: Port to ostree_repo_{open,create}_at()
A lot of code gets nicer.

Closes: #922
Approved by: jlebon
2017-08-17 15:28:14 +00:00
Colin Walters
b6705f3feb daemon: Check for updated rpms when upgrading
This closes a longstanding bug - since package layering first
landed, we only checked for newer RPMs if the base tree changed.
In some scenarios like RHELAH, this doesn't matter much by default
since they move at the same cadence.  Except if you use EPEL for example.
In Fedora, today the FAH releases are async of the rpm-md repos, and
there's also COPR which can update more than once a day even.

We should check for both update sources. Luckily we'd already introduced logic
for this in the treecompose case (checksumming the depsolved package sack). We
just need to start using it for client side assembly too.

Closes: https://github.com/projectatomic/rpm-ostree/issues/391

Closes: #911
Approved by: jlebon
2017-08-16 15:33:11 +00:00
Jonathan Lebon
8d92273f3d vmcheck: create vmcheck branch and rebase onto it
I'm working on getting the vmcheck suite working as part of Fedora's new
CI pipeline. In that context, we want to test the deliverable as it is,
i.e. with SKIP_VMOVERLAY=1. For compatibility with the testsuite, we
ensure that the machine is on the vmcheck branch before starting the
tests.

Eventually, we should try to make the vmcheck suite runnable outside of
a configured build directory to make it easier to re-use in such
contexts.

Closes: #917
Approved by: cgwalters
2017-08-15 15:40:35 +00:00