Commit Graph

1825 Commits

Author SHA1 Message Date
Jonathan Lebon
8b1001c9b5 tests/vmcheck: turn on strict bash mode
Closes: #1032
Approved by: cgwalters
2017-10-02 20:36:07 +00:00
Jonathan Lebon
2c94b11b68 libvm: don't rsync $VMCHECK_INSTTREE
We pretty standardized on insttree/ being the installation tree, and
we don't actually support $VMCHECK_INSTTREE anymore.

Closes: #1032
Approved by: cgwalters
2017-10-02 20:36:07 +00:00
Jonathan Lebon
5c25945dd3 Makefile-tests.am: actually support SKIP_INSTALL
This was supposed to be part of #968. `HACKING.md` already refers to
a `SKIP_INSTALL` variable, which basically ensures that we use the
install tree as-is.

Closes: #1032
Approved by: cgwalters
2017-10-02 20:36:07 +00:00
Colin Walters
699d80ba2d bin/compose: Drop workdir variable
Nothing actually uses it, we've been all fd-relative for a while. Just noticed
this while looking at the compose code for further work.

Closes: #1030
Approved by: jlebon
2017-10-02 18:23:54 +00:00
Colin Walters
1acd834104 Add test infra for ex container, and one test for bash
We have some unit-style tests that run `ex container`, but
they aren't "real"; they don't use scripts for example.  Let's
add tests for this similar to `tests/compose`.

We use a 26 base, but the target repos need to be 27
to pick up the fix for: https://bugzilla.redhat.com/show_bug.cgi?id=1478172

Add some bits to share infra between `tests/compose` and `tests/ex-container`;
basically handling the rpmmd repos. I tweaked things to be more streamlined
there between the `.papr.yml` and the test script.

Right now this is just one test for `bash`, but lays some of the infrastructure
for doing more. One thing that we need to do to improve more here is to better
cache RPMs, a bit like the compose tests do.

Closes: #1024
Approved by: jlebon
2017-10-02 18:04:24 +00:00
Colin Walters
57937d0e63 libtest: Squash a race in temp webserver setup
Saw this in one of the PR contexts.

Closes: #1031
Approved by: jlebon
2017-10-02 18:02:40 +00:00
Colin Walters
d4af84d428 Change ex-container to use BARE_USER_ONLY, update core creation API
Switch `ex container` to `OSTREE_REPO_MODE_BARE_USER_ONLY`; this is a good
match, for the same reasons as flatpak. We don't want suid binaries, we don't
want to chown anything, no SELinux labeling, etc.  There's no reason to write the
metadata to the `BARE_USER` xattrs.

The "unprivileged" context as used for `ex container` was a flag that was passed
down, but I think it's a lot easier if things just infer this state by looking
at whether the target repo is `bare-user-only`. Now we just have an
`rpmostree_context_new_tree()` API that handles both container and compose.

Closes: #1026
Approved by: jlebon
2017-10-02 16:13:52 +00:00
Colin Walters
d16d53ecae ex-container: Add --cache-only
Like the compose path; mostly doing this right now since I'm on a plane ✈.

Closes: #1026
Approved by: jlebon
2017-10-02 16:13:52 +00:00
Jonathan Lebon
70caa3b2da tree: add vimrc and editorconfig
Imported from ostree.

Closes: #1029
Approved by: cgwalters
2017-10-02 14:36:44 +00:00
Colin Walters
acbbda8214 tree-wide: Some more misc libglnx porting
More use of the `glnx_unlinkat()`, `glnx_fstatat()`, and `glnx_renameat()`
wrappers.

Closes: #1027
Approved by: jlebon
2017-10-02 13:41:01 +00:00
Jonathan Lebon
2cf87ba897 tree: some more glnx_fstatat_allow_noent porting
Closes: #1022
Approved by: cgwalters
2017-09-28 21:15:10 +00:00
Colin Walters
d6de31d90d libpriv/scripts: Only log to journal if uid == 0
The previous change to log to the journal broke running `ex container` as
non-root with scripts. Yes, I really desperately need to set up real tests for
that, like we have for composes. And obviously checking `getuid() == 0` is a bit
crude but doing better would require plumbing through quite a bit of of
knowledge from the toplevel down into script execution.

Closes: #1021
Approved by: jlebon
2017-09-28 20:59:18 +00:00
Colin Walters
6ad113ca46 daemon: Some misc new style porting
Was just reading the initramfs bits while talking to rubao about kargs work, and
noticed we had a lot of "old style" declarations.

Closes: #1020
Approved by: jlebon
2017-09-28 19:29:31 +00:00
Colin Walters
c0ef5eec4f libpriv/core: Check cancellable before/after downloading rpm-md
This is a quick band-aid to improve Ctrl-C handling in this area:
https://github.com/projectatomic/rpm-ostree/issues/897

At least this way we'll exit earlier rather than waiting until all the repos are
downloaded (and imported into libsolv, which is itself slow).

Closes: #1018
Approved by: jlebon
2017-09-28 17:55:18 +00:00
Jonathan Dieter
fd6109af1d compose: Copy in additional files before postprocessing
Sometimes it's useful to have access to the additional files when running
the post script, so this re-orders the compose process to copy the
additional files in before the post script runs

Signed-off-by: Jonathan Dieter <jdieter@lesbg.com>

Closes: #997
Approved by: jlebon
2017-09-28 15:38:55 +00:00
Colin Walters
2e59a2748d postprocess: Allow add-files into /etc
Even though it's really `/usr/etc`. This is for greater consistency with
`postprocess-script` where it appears as `/etc`.

Closes: #997
Approved by: jlebon
2017-09-28 15:38:55 +00:00
Colin Walters
45287828e5 lib/compose: Maintain /etc as /usr/etc more consistently
Lots of confusion in the codebase about this. The basic problem is that in
*most* cases, our code doesn't care; it's conceptually operating on `/usr/etc`,
which we could maintain as `/etc` and just rename it back at the very end.

The exceptions though are the `/etc/passwd` handling and livefs. And of course
libostree needs to handle `/usr/etc` vs `/etc` for config merging.

I considered trying to keep things the other way, but while I think we have some
ugly added here in this patch for things where we need to maintain an external
view (`remove-files` and `remove-from-packages`, and boy am I glad we had tests
for those), this ends up being mostly more consistent elsewhere.

One thing that might help is to maintain a fd for it; but that'd be an even more
invasive change.

This also ends up rolling in some unified core prep from
https://github.com/projectatomic/rpm-ostree/pull/940 in the form of
`rename_if_exists()` - basically for some minimal rootfs we may not have
`/boot`, or for that matter potentially even `etc`.

Prep for https://github.com/projectatomic/rpm-ostree/pull/997

Closes: #997
Approved by: jlebon
2017-09-28 15:38:55 +00:00
Colin Walters
11d9a78094 libpriv/core: Add some comments (and a bit of style porting)
I was reading some of the core code recently, and in general I think we need
more comments. I made a pass from the top of the file and got halfway down.

Closes: #1017
Approved by: jlebon
2017-09-28 13:04:29 +00:00
Jonathan Lebon
cb7e84c4a6 libpriv/scripts: redirect scriptlet output to journal
Rather than just letting the scriptlets inherit the daemon's
stdout/stderr, redirect their outputs so that we can set a customized
identifier to make it easier to distinguish from the daemon output.

Also print out the `journalctl` command needed so that users can
investigate the output themselves.

Closes: #998
Approved by: cgwalters
2017-09-27 19:01:45 +00:00
Jonathan Lebon
c92ff926f6 app/status: cleaner diff output for replaced pkgs
I suspect a common pattern with local replacement overrides is to
simultaneously replace a group of packages that depend on each other in
one shot, as is the case with docker, docker-common, and
docker-rhel-push-plugin currently in Fedora Atomic Host. In such cases,
we can print a cleaner diff in the status to make it easier to grok.

Before:

     ReplacedBasePackages: strace 4.18-1.fc26 -> 4.19-1.fc26, docker-common 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, docker 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, docker-rhel-push-plugin 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26

After:

     ReplacedBasePackages: docker-common docker docker-rhel-push-plugin 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, strace 4.18-1.fc26 -> 4.19-1.fc26

Closes: #1004
Approved by: cgwalters
2017-09-27 16:22:50 +00:00
Colin Walters
b7733510bd ci: Reduce metadata expiration for rdgo
At some point I'll fix rdgo to retain multiple, but I think this may help CI
reliability in the short term.

Closes: #1015
Approved by: jlebon
2017-09-26 19:41:08 +00:00
Colin Walters
58f8f892fb lib/postprocess: Use O_TMPFILE, not O_APPEND for tmpfiles.d writing
The comment here was wrong; we don't rely on `O_APPEND` here for package
layering since we convert on import.  I noticed this while I was doing
a grep for `O_APPEND` in the codebase as part of unified core work.

Fix this by converting to `O_TMPFILE`+`GLNX_LINK_TMPFILE_NOREPLACE`.

Prep for unified core.

Closes: #1009
Approved by: jlebon
2017-09-26 18:27:00 +00:00
Colin Walters
57fcd9e1b0 libpriv: Remove unused OSTREE_GIO_FAST_QUERYINFO
Hooray, it's dead (here, but not in ostree). Noticed this while working on
<https://github.com/projectatomic/rpm-ostree/pull/997>

Closes: #1012
Approved by: jlebon
2017-09-26 14:54:52 +00:00
Colin Walters
5eab29846a libpriv/passwd: Use GLNX_FILE_COPY_NOXATTRS
Prep for unified core work. This was failing due to a bug in libglnx
<https://github.com/GNOME/libglnx/pull/80> but I think this change is also
correct. There's no good reason for us to copy xattrs like the SELinux label
here - rather we want the labels to be reset during commit.

I did a tree-wide grep for other users and the only other case that is odd is
the treecompose `add-files`; I'd say we should change this but out of (a likely
excess of) conservatism I just left a "FIXME" for now.

Closes: #1008
Approved by: jlebon
2017-09-26 13:07:59 +00:00
Jonathan Lebon
c10a45aa5e libpriv/core: squash a -Wmaybe-uninitialized
Noticed this pass by while doing a `fedpkg local`. It doesn't actually
matter much since there are no exits possible between the point it's
declared and when it's actually assigned, which I will assume is why our
clang suite didn't pick it up, maybe? Not gonna look into that right
now.

Closes: #1010
Approved by: cgwalters
2017-09-25 17:39:42 +00:00
Colin Walters
eacbcc34e0 Release 2017.9
Closes: #1007
Approved by: jlebon
2017-09-25 15:39:57 +00:00
Colin Walters
13f307da36 livefs: Update /usr/lib/{passwd,group}, run systemd-tmpfiles
This gets us to supporting live installs of `httpd` and many
other services.  I also tried `postgresql-server` for example.

Closes: https://github.com/projectatomic/rpm-ostree/issues/962

Closes: #996
Approved by: jlebon
2017-09-25 15:10:15 +00:00
Colin Walters
2ebf43f4a9 ci: Sync with ostree
We have some drift; keep this more in sync so we maintain the rpm/yum
hackarounds for example in one place.

I backed out the ASAN bits though pending at least a newer gpgme:
<https://github.com/projectatomic/rpm-ostree/pull/1000#issuecomment-331278758>

Closes: #1000
Approved by: jlebon
2017-09-24 13:32:00 +00:00
Colin Walters
e066dc93c8 tests: Skip running introspection tests if built with ASAN
This is the rpm-ostree equivalent of
<47b4dd1b38>

Unfortunately, introspection uses `dlopen(), which doesn't quite
work when the DSO is compiled with ASAN but the outer executable
isn't.

Prep for syncing PAPR config with ostree.

Closes: #1000
Approved by: jlebon
2017-09-24 13:32:00 +00:00
Jonathan Lebon
167fd4c1dd libpriv/scripts: use rofiles-fuse on /etc too
We were directly bind mounting the checked out `/usr/etc` onto `/etc`
which was exposing us to corruption from scriptlets. Since we already
have an rofiles-fuse mount for `/usr`, let's just re-use its `etc/`
subdir and bind mount that instead.

Closes: #1003
Approved by: cgwalters
2017-09-24 13:11:21 +00:00
Jonathan Lebon
f089b8de1f libpriv/postprocess: also delete semanage lock files
We don't need those in the tree, so let's nuke them. This also fixes
subtle compatibility issues between hardlinks and lock files (see #999).

Closes: #1002
Approved by: cgwalters
2017-09-21 15:28:55 +00:00
Jonathan Lebon
25ba196c27 libpriv/core: also sort update/downgrade packages
Now that we support overrides, we need the sorter to make sure it goes
through the override packages in case they need to be relabeled.

Closes: #961

Closes: #993
Approved by: cgwalters
2017-09-18 14:53:50 +00:00
Jonathan Lebon
e78807143f libpriv/core: fix tmpdir making logic
Even if we're already initialized, we still need to create the subdir
requested. This would cause relabeling to fail because we try to create
the "relabel" subdir, which never actually gets created.

Closes: #993
Approved by: cgwalters
2017-09-18 14:53:50 +00:00
Colin Walters
80de17623f tests/livefs: Make sure /usr/bin/ls is executable
Oh man was I confused how this happened; thought it was a bug in some livefs
changes I was working on.

Closes: #995
Approved by: jlebon
2017-09-15 19:10:24 +00:00
Colin Walters
cc4844ff02 daemon/upgrader: Clear core ctx after committing
Amazingly we had 3 variations of this bug in the different
layers above using the `RpmOstreeContext` API (in compose, container,
and layering).

We fixed the first two already, this fixes the last one.  We
get a warning if librpm holds a ref to a deleted rpmdb.

Closes: https://github.com/projectatomic/rpm-ostree/issues/987

Closes: #994
Approved by: jlebon
2017-09-15 14:17:59 +00:00
Colin Walters
4a9aebcc02 core: Don't search for base filetriggers if no base rpmdb
Otherwise librpm whines on stderr about a missing database.

Related: https://github.com/projectatomic/rpm-ostree/issues/987

Closes: #989
Approved by: jlebon
2017-09-15 12:03:12 +00:00
Colin Walters
c30e63f283 compose: Clear libdnf context before postprocessing
This ensures we close the rpm database before we start moving
things around during postprocessing, and avoids a warning about
the rpmdb going missing being printed at the end of a compose.

Related: https://github.com/projectatomic/rpm-ostree/issues/987

Closes: #989
Approved by: jlebon
2017-09-15 12:03:12 +00:00
Jonathan Lebon
2c514f446c ci: don't run clang on CentOS
The CentOS tester is currently failing because clang is being passed an
argument it doesn't know about:

  clang: error: unknown argument: '-fstack-protector-strong'

We get our version of clang from EPEL, which currently has 3.4.2. The
Fedora clang does have support for the flag (4.0.1).

Anyway, the point of this check is just to find unused/uninitialized
vars and other gotchas. In that sense, we're more interested in what the
latest version of clang has to say.

Closes: #992
Approved by: cgwalters
2017-09-14 21:02:46 +00:00
Ruixin
56ef177235 tests/core: allow merging identical files during installation
This PR uses https://github.com/ostreedev/ostree/pull/1156
to ensure that when installing packages containing files with
exact same content, the files will be merged.

When installing packages containing conflicting files,
the error will still be detected at the ostree side.

The checkout overwrite option at rpm-ostree side is also
modified to accomodate the changes made in ostree side.

A test is added for regression
2017-09-13 20:41:15 +00:00
Jonathan Lebon
f62f6b7086 livefs: tweak warning re. /etc
Be more explicit about what the warning means.

Closes: #988
Approved by: dustymabe
2017-09-13 13:51:06 +00:00
Jonathan Lebon
2a5ecf7a3e livefs: add warning that /etc won't be preserved
Since the whole premise of livefs is that you stay in your deployment
just a little longer, it becomes much more likely for people to run into
https://github.com/projectatomic/rpm-ostree/issues/40 (myself included).

Printing such a notice was discussed in the initial livefs design
discussions: https://github.com/projectatomic/rpm-ostree/issues/639.

Closes: #986
Approved by: cgwalters
2017-09-12 19:51:48 +00:00
Jonathan Lebon
4ad3ea58c6 app: smarter deployment change detection
Commands like `upgrade` and `deploy` need to know if a new deployment
was actually laid down so that it may print a pkg diff if so. This is
implemented by listening for changes to the DefaultDeployment D-Bus
property. D-Bus emits a signal when the deployment variant changes
value.

However, in #595, with the introduction of `pending-*` related keys, the
deployment variant no longer represents data solely tied to that
specific deployment. In this case, because `deploy` operations currently
set the ref to the resolved checksum, it can happen that deploying the
same base commit when the current refspec *isn't* pointing to that base
commit will result in the `pending-*` keys dropping out and a default
deployment change notification going out.

In this patch, we strengthen how we determine whether a new deployment
was laid down by actually looking at the deployment id, rather than just
assuming that a change to the property implies a new deployment.

Closes: #981

Closes: #984
Approved by: cgwalters
2017-09-12 17:23:25 +00:00
Jonathan Lebon
077d7c1a9e daemon/deploy: allow redeploying livefs booted commit
Also something I noticed while working on #981. When sitting on a livefs
commit, once a user does `rpm-ostree cleanup --pending --rollback`, it's
impossible to redeploy the same booted commit. Let's allow users to do
this.

Closes: #984
Approved by: cgwalters
2017-09-12 17:23:25 +00:00
Jonathan Lebon
c08ca8f922 daemon/upgrader: make use of override-commit-ids
Currently, when setting the `override-commit` key in the origin, the
upgrader pulls that commit checksum directly and then updates the
refspec to point to it. This behaviour was inherited from its ostree
version; at the time it was implemented, the pull code didn't support
passing a specific commit for a given refspec. However, we now have
the override-commit-ids option, which will make libostree update the ref
for us.

We change the code here to make use of it and simplify the function.
This also fixes the corner case of local branches: we shouldn't change
the ref if we're on a local branch. This is actually what drove me to
this patch as I was debugging #981.

(Aside: I'm still not convinced updating the refspec is always the
correct thing to do even in the remote case, though it's a bit messy to
disentangle).

Closes: #984
Approved by: cgwalters
2017-09-12 17:23:25 +00:00
Colin Walters
f113fc5e27 Rework treecompose kernel processing
Prep for changing `boot_location: new` to use `/usr/lib/ostree-boot`
and `/usr/lib/modules`.  Rework our kernel postprocessing
so that we unify the `boot_location` handling with initramfs generation.

Instead of doing the initramfs first in postprocessing, we do it nearly last,
after e.g. `etc` is renamed to `usr/etc`. This has some consequences, such as
the fact that `run_bwrap_mutably()` is now called in both situations. In
general, our handling of `etc` is inconsistent, although understandably so.

As part of this, I finally got around to implementing the bit from
https://github.com/systemd/systemd/pull/4174 however suboptimal it is; need the
unified core so we can cleanly ignore the posttrans like we do others.  We
intentionally keep the file around in the generated tree so that installing a
kernel RPM per client doesn't try to do any of this either.

This all gets folded together so that the logic for handling the bootloader gets
simpler - in the Fedora case, we now know to find kernels in `/usr/lib/modules`
and can ignore `/boot`.

Closes: #959
Approved by: jlebon
2017-09-12 14:26:44 +00:00
Colin Walters
93d3fc6617 tests/compose: Add env variables to force caching and fast fail
This helped me debug/fix the tests faster.

Closes: #959
Approved by: jlebon
2017-09-12 14:26:44 +00:00
Colin Walters
3ff48589b3 compose: Make boot_location: legacy be an alias for both
We have 3 locations to find kernels now; I can't think of
a reason to support placing kernels *only* in `/boot`.  The
original commit
15ecaacd36
doesn't give a reason, and I certainly can't think of one now.

This makes `legacy` be an alias for `both`, which should be fully compatible.

Prep for further refactoring towards changing `new` to mean both
`/usr/lib/ostree-boot` *and* `/usr/lib/modules`.

Closes: #959
Approved by: jlebon
2017-09-12 14:26:44 +00:00
Colin Walters
aab7571fc1 ci: Fixate centos container image to 7.3 until rpm-md repos update
Sigh.

Closes: #985
Approved by: jlebon
2017-09-12 13:45:50 +00:00
Jonathan Lebon
bed10f650a Makefile-tests.am: fail on first command
The shell invocations from make don't have errexit on, so just chain
commands with && manually.

Closes: #983
Approved by: cgwalters
2017-09-11 18:38:43 +00:00
Jonathan Lebon
bae4faea2b app/upgrade: use signal to determine if changed
Make `upgrade` use the same trick as `deploy` to determine if a new
deployment was laid down. Apart from those two, all other operations
that can lay down a new deployment always do so in the happy path.

Prep for further work.

Closes: #983
Approved by: cgwalters
2017-09-11 18:38:43 +00:00