Commit Graph

2009 Commits

Author SHA1 Message Date
Colin Walters
45287828e5 lib/compose: Maintain /etc as /usr/etc more consistently
Lots of confusion in the codebase about this. The basic problem is that in
*most* cases, our code doesn't care; it's conceptually operating on `/usr/etc`,
which we could maintain as `/etc` and just rename it back at the very end.

The exceptions though are the `/etc/passwd` handling and livefs. And of course
libostree needs to handle `/usr/etc` vs `/etc` for config merging.

I considered trying to keep things the other way, but while I think we have some
ugly added here in this patch for things where we need to maintain an external
view (`remove-files` and `remove-from-packages`, and boy am I glad we had tests
for those), this ends up being mostly more consistent elsewhere.

One thing that might help is to maintain a fd for it; but that'd be an even more
invasive change.

This also ends up rolling in some unified core prep from
https://github.com/projectatomic/rpm-ostree/pull/940 in the form of
`rename_if_exists()` - basically for some minimal rootfs we may not have
`/boot`, or for that matter potentially even `etc`.

Prep for https://github.com/projectatomic/rpm-ostree/pull/997

Closes: #997
Approved by: jlebon
2017-09-28 15:38:55 +00:00
Colin Walters
11d9a78094 libpriv/core: Add some comments (and a bit of style porting)
I was reading some of the core code recently, and in general I think we need
more comments. I made a pass from the top of the file and got halfway down.

Closes: #1017
Approved by: jlebon
2017-09-28 13:04:29 +00:00
Jonathan Lebon
cb7e84c4a6 libpriv/scripts: redirect scriptlet output to journal
Rather than just letting the scriptlets inherit the daemon's
stdout/stderr, redirect their outputs so that we can set a customized
identifier to make it easier to distinguish from the daemon output.

Also print out the `journalctl` command needed so that users can
investigate the output themselves.

Closes: #998
Approved by: cgwalters
2017-09-27 19:01:45 +00:00
Jonathan Lebon
c92ff926f6 app/status: cleaner diff output for replaced pkgs
I suspect a common pattern with local replacement overrides is to
simultaneously replace a group of packages that depend on each other in
one shot, as is the case with docker, docker-common, and
docker-rhel-push-plugin currently in Fedora Atomic Host. In such cases,
we can print a cleaner diff in the status to make it easier to grok.

Before:

     ReplacedBasePackages: strace 4.18-1.fc26 -> 4.19-1.fc26, docker-common 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, docker 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, docker-rhel-push-plugin 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26

After:

     ReplacedBasePackages: docker-common docker docker-rhel-push-plugin 2:1.13.1-22.gitb5e3294.fc26 -> 2:1.13.1-21.git27e468e.fc26, strace 4.18-1.fc26 -> 4.19-1.fc26

Closes: #1004
Approved by: cgwalters
2017-09-27 16:22:50 +00:00
Colin Walters
b7733510bd ci: Reduce metadata expiration for rdgo
At some point I'll fix rdgo to retain multiple, but I think this may help CI
reliability in the short term.

Closes: #1015
Approved by: jlebon
2017-09-26 19:41:08 +00:00
Colin Walters
58f8f892fb lib/postprocess: Use O_TMPFILE, not O_APPEND for tmpfiles.d writing
The comment here was wrong; we don't rely on `O_APPEND` here for package
layering since we convert on import.  I noticed this while I was doing
a grep for `O_APPEND` in the codebase as part of unified core work.

Fix this by converting to `O_TMPFILE`+`GLNX_LINK_TMPFILE_NOREPLACE`.

Prep for unified core.

Closes: #1009
Approved by: jlebon
2017-09-26 18:27:00 +00:00
Colin Walters
57fcd9e1b0 libpriv: Remove unused OSTREE_GIO_FAST_QUERYINFO
Hooray, it's dead (here, but not in ostree). Noticed this while working on
<https://github.com/projectatomic/rpm-ostree/pull/997>

Closes: #1012
Approved by: jlebon
2017-09-26 14:54:52 +00:00
Colin Walters
5eab29846a libpriv/passwd: Use GLNX_FILE_COPY_NOXATTRS
Prep for unified core work. This was failing due to a bug in libglnx
<https://github.com/GNOME/libglnx/pull/80> but I think this change is also
correct. There's no good reason for us to copy xattrs like the SELinux label
here - rather we want the labels to be reset during commit.

I did a tree-wide grep for other users and the only other case that is odd is
the treecompose `add-files`; I'd say we should change this but out of (a likely
excess of) conservatism I just left a "FIXME" for now.

Closes: #1008
Approved by: jlebon
2017-09-26 13:07:59 +00:00
Jonathan Lebon
c10a45aa5e libpriv/core: squash a -Wmaybe-uninitialized
Noticed this pass by while doing a `fedpkg local`. It doesn't actually
matter much since there are no exits possible between the point it's
declared and when it's actually assigned, which I will assume is why our
clang suite didn't pick it up, maybe? Not gonna look into that right
now.

Closes: #1010
Approved by: cgwalters
2017-09-25 17:39:42 +00:00
Colin Walters
eacbcc34e0 Release 2017.9
Closes: #1007
Approved by: jlebon
2017-09-25 15:39:57 +00:00
Colin Walters
13f307da36 livefs: Update /usr/lib/{passwd,group}, run systemd-tmpfiles
This gets us to supporting live installs of `httpd` and many
other services.  I also tried `postgresql-server` for example.

Closes: https://github.com/projectatomic/rpm-ostree/issues/962

Closes: #996
Approved by: jlebon
2017-09-25 15:10:15 +00:00
Colin Walters
2ebf43f4a9 ci: Sync with ostree
We have some drift; keep this more in sync so we maintain the rpm/yum
hackarounds for example in one place.

I backed out the ASAN bits though pending at least a newer gpgme:
<https://github.com/projectatomic/rpm-ostree/pull/1000#issuecomment-331278758>

Closes: #1000
Approved by: jlebon
2017-09-24 13:32:00 +00:00
Colin Walters
e066dc93c8 tests: Skip running introspection tests if built with ASAN
This is the rpm-ostree equivalent of
<47b4dd1b38>

Unfortunately, introspection uses `dlopen(), which doesn't quite
work when the DSO is compiled with ASAN but the outer executable
isn't.

Prep for syncing PAPR config with ostree.

Closes: #1000
Approved by: jlebon
2017-09-24 13:32:00 +00:00
Jonathan Lebon
167fd4c1dd libpriv/scripts: use rofiles-fuse on /etc too
We were directly bind mounting the checked out `/usr/etc` onto `/etc`
which was exposing us to corruption from scriptlets. Since we already
have an rofiles-fuse mount for `/usr`, let's just re-use its `etc/`
subdir and bind mount that instead.

Closes: #1003
Approved by: cgwalters
2017-09-24 13:11:21 +00:00
Jonathan Lebon
f089b8de1f libpriv/postprocess: also delete semanage lock files
We don't need those in the tree, so let's nuke them. This also fixes
subtle compatibility issues between hardlinks and lock files (see #999).

Closes: #1002
Approved by: cgwalters
2017-09-21 15:28:55 +00:00
Jonathan Lebon
25ba196c27 libpriv/core: also sort update/downgrade packages
Now that we support overrides, we need the sorter to make sure it goes
through the override packages in case they need to be relabeled.

Closes: #961

Closes: #993
Approved by: cgwalters
2017-09-18 14:53:50 +00:00
Jonathan Lebon
e78807143f libpriv/core: fix tmpdir making logic
Even if we're already initialized, we still need to create the subdir
requested. This would cause relabeling to fail because we try to create
the "relabel" subdir, which never actually gets created.

Closes: #993
Approved by: cgwalters
2017-09-18 14:53:50 +00:00
Colin Walters
80de17623f tests/livefs: Make sure /usr/bin/ls is executable
Oh man was I confused how this happened; thought it was a bug in some livefs
changes I was working on.

Closes: #995
Approved by: jlebon
2017-09-15 19:10:24 +00:00
Colin Walters
cc4844ff02 daemon/upgrader: Clear core ctx after committing
Amazingly we had 3 variations of this bug in the different
layers above using the `RpmOstreeContext` API (in compose, container,
and layering).

We fixed the first two already, this fixes the last one.  We
get a warning if librpm holds a ref to a deleted rpmdb.

Closes: https://github.com/projectatomic/rpm-ostree/issues/987

Closes: #994
Approved by: jlebon
2017-09-15 14:17:59 +00:00
Colin Walters
4a9aebcc02 core: Don't search for base filetriggers if no base rpmdb
Otherwise librpm whines on stderr about a missing database.

Related: https://github.com/projectatomic/rpm-ostree/issues/987

Closes: #989
Approved by: jlebon
2017-09-15 12:03:12 +00:00
Colin Walters
c30e63f283 compose: Clear libdnf context before postprocessing
This ensures we close the rpm database before we start moving
things around during postprocessing, and avoids a warning about
the rpmdb going missing being printed at the end of a compose.

Related: https://github.com/projectatomic/rpm-ostree/issues/987

Closes: #989
Approved by: jlebon
2017-09-15 12:03:12 +00:00
Jonathan Lebon
2c514f446c ci: don't run clang on CentOS
The CentOS tester is currently failing because clang is being passed an
argument it doesn't know about:

  clang: error: unknown argument: '-fstack-protector-strong'

We get our version of clang from EPEL, which currently has 3.4.2. The
Fedora clang does have support for the flag (4.0.1).

Anyway, the point of this check is just to find unused/uninitialized
vars and other gotchas. In that sense, we're more interested in what the
latest version of clang has to say.

Closes: #992
Approved by: cgwalters
2017-09-14 21:02:46 +00:00
Ruixin
56ef177235 tests/core: allow merging identical files during installation
This PR uses https://github.com/ostreedev/ostree/pull/1156
to ensure that when installing packages containing files with
exact same content, the files will be merged.

When installing packages containing conflicting files,
the error will still be detected at the ostree side.

The checkout overwrite option at rpm-ostree side is also
modified to accomodate the changes made in ostree side.

A test is added for regression
2017-09-13 20:41:15 +00:00
Jonathan Lebon
f62f6b7086 livefs: tweak warning re. /etc
Be more explicit about what the warning means.

Closes: #988
Approved by: dustymabe
2017-09-13 13:51:06 +00:00
Jonathan Lebon
2a5ecf7a3e livefs: add warning that /etc won't be preserved
Since the whole premise of livefs is that you stay in your deployment
just a little longer, it becomes much more likely for people to run into
https://github.com/projectatomic/rpm-ostree/issues/40 (myself included).

Printing such a notice was discussed in the initial livefs design
discussions: https://github.com/projectatomic/rpm-ostree/issues/639.

Closes: #986
Approved by: cgwalters
2017-09-12 19:51:48 +00:00
Jonathan Lebon
4ad3ea58c6 app: smarter deployment change detection
Commands like `upgrade` and `deploy` need to know if a new deployment
was actually laid down so that it may print a pkg diff if so. This is
implemented by listening for changes to the DefaultDeployment D-Bus
property. D-Bus emits a signal when the deployment variant changes
value.

However, in #595, with the introduction of `pending-*` related keys, the
deployment variant no longer represents data solely tied to that
specific deployment. In this case, because `deploy` operations currently
set the ref to the resolved checksum, it can happen that deploying the
same base commit when the current refspec *isn't* pointing to that base
commit will result in the `pending-*` keys dropping out and a default
deployment change notification going out.

In this patch, we strengthen how we determine whether a new deployment
was laid down by actually looking at the deployment id, rather than just
assuming that a change to the property implies a new deployment.

Closes: #981

Closes: #984
Approved by: cgwalters
2017-09-12 17:23:25 +00:00
Jonathan Lebon
077d7c1a9e daemon/deploy: allow redeploying livefs booted commit
Also something I noticed while working on #981. When sitting on a livefs
commit, once a user does `rpm-ostree cleanup --pending --rollback`, it's
impossible to redeploy the same booted commit. Let's allow users to do
this.

Closes: #984
Approved by: cgwalters
2017-09-12 17:23:25 +00:00
Jonathan Lebon
c08ca8f922 daemon/upgrader: make use of override-commit-ids
Currently, when setting the `override-commit` key in the origin, the
upgrader pulls that commit checksum directly and then updates the
refspec to point to it. This behaviour was inherited from its ostree
version; at the time it was implemented, the pull code didn't support
passing a specific commit for a given refspec. However, we now have
the override-commit-ids option, which will make libostree update the ref
for us.

We change the code here to make use of it and simplify the function.
This also fixes the corner case of local branches: we shouldn't change
the ref if we're on a local branch. This is actually what drove me to
this patch as I was debugging #981.

(Aside: I'm still not convinced updating the refspec is always the
correct thing to do even in the remote case, though it's a bit messy to
disentangle).

Closes: #984
Approved by: cgwalters
2017-09-12 17:23:25 +00:00
Colin Walters
f113fc5e27 Rework treecompose kernel processing
Prep for changing `boot_location: new` to use `/usr/lib/ostree-boot`
and `/usr/lib/modules`.  Rework our kernel postprocessing
so that we unify the `boot_location` handling with initramfs generation.

Instead of doing the initramfs first in postprocessing, we do it nearly last,
after e.g. `etc` is renamed to `usr/etc`. This has some consequences, such as
the fact that `run_bwrap_mutably()` is now called in both situations. In
general, our handling of `etc` is inconsistent, although understandably so.

As part of this, I finally got around to implementing the bit from
https://github.com/systemd/systemd/pull/4174 however suboptimal it is; need the
unified core so we can cleanly ignore the posttrans like we do others.  We
intentionally keep the file around in the generated tree so that installing a
kernel RPM per client doesn't try to do any of this either.

This all gets folded together so that the logic for handling the bootloader gets
simpler - in the Fedora case, we now know to find kernels in `/usr/lib/modules`
and can ignore `/boot`.

Closes: #959
Approved by: jlebon
2017-09-12 14:26:44 +00:00
Colin Walters
93d3fc6617 tests/compose: Add env variables to force caching and fast fail
This helped me debug/fix the tests faster.

Closes: #959
Approved by: jlebon
2017-09-12 14:26:44 +00:00
Colin Walters
3ff48589b3 compose: Make boot_location: legacy be an alias for both
We have 3 locations to find kernels now; I can't think of
a reason to support placing kernels *only* in `/boot`.  The
original commit
15ecaacd36
doesn't give a reason, and I certainly can't think of one now.

This makes `legacy` be an alias for `both`, which should be fully compatible.

Prep for further refactoring towards changing `new` to mean both
`/usr/lib/ostree-boot` *and* `/usr/lib/modules`.

Closes: #959
Approved by: jlebon
2017-09-12 14:26:44 +00:00
Colin Walters
aab7571fc1 ci: Fixate centos container image to 7.3 until rpm-md repos update
Sigh.

Closes: #985
Approved by: jlebon
2017-09-12 13:45:50 +00:00
Jonathan Lebon
bed10f650a Makefile-tests.am: fail on first command
The shell invocations from make don't have errexit on, so just chain
commands with && manually.

Closes: #983
Approved by: cgwalters
2017-09-11 18:38:43 +00:00
Jonathan Lebon
bae4faea2b app/upgrade: use signal to determine if changed
Make `upgrade` use the same trick as `deploy` to determine if a new
deployment was laid down. Apart from those two, all other operations
that can lay down a new deployment always do so in the happy path.

Prep for further work.

Closes: #983
Approved by: cgwalters
2017-09-11 18:38:43 +00:00
Jonathan Lebon
df515fd2f1 app/libbuiltin: convert to new style
Pretty straightforward stuff. Prep for further work.

Closes: #983
Approved by: cgwalters
2017-09-11 18:38:43 +00:00
Colin Walters
150de0087e scripts: Support overriding lua, use for glibc-all-langpacks
Today in Fedora the `glibc-all-langpacks.posttrans` is implemented
in lua, for no good reason.  See:
https://bugzilla.redhat.com/show_bug.cgi?id=1367585

Since that's stalled out, let's add support for overrides.  This
is obviously a much bigger step with more long term maintenance
implications over our current "ignore scripts" list.  But we can't
block either.

This is needed for unified core work:
https://github.com/projectatomic/rpm-ostree/issues/729

(We also override `fedora-release-atomichost` but I'll likely
 submit a patch for that upstream)

Closes: #980
Approved by: jlebon
2017-09-08 18:02:22 +00:00
Colin Walters
e40eaebbe0 core: Use fd-relative access to rpmdb
I was linking to this code from elsewhere and noticed that
for our hardlink breaks we were not using fd-relative even
though we can.  Down the line if we fork librpm into a separate
process and do e.g. `--dbpath=.` it'll do it too.

(Side note, I verified that commenting out the hardlink breaking
 here was caught by the `ostree fsck` I added to the test suite)

Closes: #979
Approved by: jlebon
2017-09-07 22:54:40 +00:00
Colin Walters
38d1876270 Update libglnx, do some fstatat-noent porting
Started on porting to the new `glnx_fstatat_allow_noent()`.  The
usage varies a lot and it felt easy to screw up, so I'm just
starting by doing a few of them.

Update submodule: libglnx

Closes: #978
Approved by: jlebon
2017-09-07 22:30:34 +00:00
Colin Walters
79086f8c3d core: Use a single tmpdir/ostreetxn for relabeling
I'd like to cut down on the number of tempdirs we make; when
debugging with gdb I often use e.g. `r` to restart forcibly, and
that leaves the dirs around.

Previously we'd make a tmpdir per relabeled package; this hoists it into a
well-known "relabel/" subdir of the per-core tmpdir we make. (A next step here
is to dedup the core tmpdir with the tmpdir that the upgrader makes)

Further while I'm here, let's use a single OSTree transaction for all of the
relabeling; the rationale here is that it's better to do fewer `syncfs()`
operations.

Closes: #970
Approved by: jlebon
2017-09-07 19:04:08 +00:00
Colin Walters
b5c98ec4f1 daemon: Add a policy for reload-config
This fixes `rpm-ostree reload` as root, and supports configuring
it to be enabled for other users as well.  This was overlooked
in the polkit work originally.

Closes: https://github.com/projectatomic/rpm-ostree/issues/976

Closes: #977
Approved by: jlebon
2017-09-07 15:11:00 +00:00
Jonathan Lebon
b0c75ae900 vmcheck: use --selinux-policy when committing
Let's make SELinux and atomic-host-tests happy with trees we concoct
ourselves by using the new --selinux-policy. (Specifically, we want our
sync'ed binaries to have install_exec_t).

Closes: #953
Approved by: cgwalters
2017-09-07 14:41:19 +00:00
Jonathan Lebon
50ecc04cb6 overlay.sh: drop hack for backup passwd files
We shouldn't need this anymore.

Closes: #953
Approved by: cgwalters
2017-09-07 14:41:19 +00:00
Jonathan Lebon
b22d7d764d vmcheck: rename build.sh to install.sh
I had meant for this to be in the other PR#968. I originally did both
`make` and `make install` there, but now it only does `make install`, so
let's just rename it to make that more obvious.

Closes: #953
Approved by: cgwalters
2017-09-07 14:41:19 +00:00
Colin Walters
c093a587a2 tests: Add a test case for lua scripts
This is currently unsupported, let's add a test case. Prep for support for
script overrides.

Closes: #973
Approved by: jlebon
2017-09-07 14:09:47 +00:00
Colin Walters
f69717bf69 libpriv+daemon: Use autocleanup to abort ostree txn
This allows porting one function each in the unpacker and sysroot to new style.
There were also two cases in the core where we were missing an
`abort_transaction()` invocation.

libostree version in: https://github.com/ostreedev/ostree/pull/1132
Prep for: https://github.com/projectatomic/rpm-ostree/pull/970

Closes: #972
Approved by: jlebon
2017-09-07 13:50:19 +00:00
Colin Walters
20abc85243 unpacker: Translate /boot → /usr/lib/ostree-boot
At one point `rpm-ostree install libvirt` dragged in libguestfs which in turn
brought in `syslinux-extlinux-nonlinux` which has files in `/boot/extlinux`,
which we rejected.  (That dependency chain appears to have been fixed currently)

For the general case, this is just a partial fix in that we haven't nailed down
the semantics of how updates for `/boot` work.  But in this particular case,
we'll just break libguestfs' `extlinux` verb, which I'm OK with.

Another case is `fwupdate-efi` - we require manual intervention to copy the
data into `/boot` after installing the package.

This is also preparation for [unified core](https://github.com/projectatomic/rpm-ostree/issues/729)
in that we now ensure imported kernels don't end up in `/boot` unless
explicitly configured.

Closes: https://github.com/projectatomic/rpm-ostree/issues/853

Closes: #969
Approved by: jlebon
2017-09-07 13:30:36 +00:00
Ruixin Bao
d6b3217a86 lib/core: Detect file name conflict before rpmdb
This PR uses https://github.com/ostreedev/ostree/pull/1116
to ensure that when installing packages containing conflicting files, the
error will be detected at the ostree side, rather than when
writing the rpm database.

Closes: https://github.com/projectatomic/rpm-ostree/issues/365

Closes: #974
Approved by: cgwalters
2017-09-07 13:13:12 +00:00
Jonathan Lebon
48ddca9280 ci: workaround broken fedora:26 image
This is essentially the same workaround as
https://github.com/ostreedev/ostree/pull/1143.

See https://bugzilla.redhat.com/show_bug.cgi?id=1483553.

Closes: #975
Approved by: cgwalters
2017-09-07 12:27:54 +00:00
Jonathan Lebon
7fbf9f32d4 hacking: make it easier to use a custom tree
Let's make using a custom install tree easier and document the process.
We split out the insttree step into `build.sh` so that we no longer have
to `flock(1)` around it, and also share between `overlay.sh` and
`sync.sh`.

Closes: #968
Approved by: cgwalters
2017-09-01 19:58:55 +00:00
Jonathan Lebon
5294c1ea69 scripts: squash a -Wmaybe-uninitialized
Closes: #968
Approved by: cgwalters
2017-09-01 19:58:55 +00:00