Commit Graph

1203 Commits

Author SHA1 Message Date
Jonathan Lebon
5860897092 tree: account for ephemeral nevra string
When writing this code, I made the false assumption that the nevra
string lives as long as the pool does, i.e. as long as we have a
reference to its `DnfSack`.

In fact, they have undefined lifetimes. Notably any place in which one
calls `dnf_package_get_nevra` a lot may result in the invalidation of
previously returned nevras.

This patch ensures that we copy the string in the few places where we
are susceptible to this.

There is a related libdnf patch[1] which tightens the definition here so
that we can assume the string at least lives as long as its
`DnfPackage`. It turns out that the callsites addressed in this patch
are also those in which we would break that assumption. IOW, this patch
is needed regardless of how [1] goes.

[1] https://github.com/rpm-software-management/libdnf/pull/388

Closes: #1119
Approved by: cgwalters
2017-11-29 20:05:39 +00:00
Jonathan Lebon
8f6f2dd9e4 core: fix minor memory leak
Closes: #1119
Approved by: cgwalters
2017-11-29 20:05:39 +00:00
Colin Walters
9c004e1833 Change unpacking to use a single ostree txn
Right now each ostree txn incurs a `syncfs()`; see
https://github.com/ostreedev/ostree/issues/1184
And before this patch, we were doing a txn per package import.

We can really do better in libostree - we'll fix that, but in the short term
let's use a bigger txn for every package. However, the obvious change here of
simply hoisting up the txn is that on failure for imports, we'd discard all
downloaded packages.  We fix that by changing the auto txn API to have
a `commit_on_failure` boolean, and use it in cases where we're doing
imports.

This is prep work for jigdo, where we'll be using the import path all the time.
My bigger plan is to do multithreaded imports.

Closes: #1116
Approved by: jlebon
2017-11-29 16:48:19 +00:00
Colin Walters
a822722a5b ex-unpack: Delete
I think the `ex container` path supercedes this; it was really just a demo, and
having it around is annoying since I want to change the importer API and I have
to change it here too.

Closes: #1116
Approved by: jlebon
2017-11-29 16:48:19 +00:00
Colin Walters
f58cb44fb2 txn: For --download-only, print 'changed' message if changed
I was playing with `--download-only` a bit with an eye to
having something like this be used by Cockpit/gnome-software instead
of what it's doing now, but a problem is that at the moment we
don't have a way to reflect the "changed" state back to clients.

This is a first step towards that by simply printing a different
message.

I think really to make all of this work more nicely though, including
supporting e.g. rpm database diffs, we are going to have to instead
work on the [pending deployment](https://github.com/ostreedev/ostree/issues/545)
path.  That way we'll have done the depsolve, stored repo timestamps
etc.; we'll be able to accurately show what *did* change rather than
try to recreate what will happen on the next `rpm-ostree upgrade --cache-only`.

Closes: #1118
Approved by: jlebon
2017-11-29 15:29:49 +00:00
Ruixin Bao
2ef240d401 app/libpriv: add editor option for karg command
Add editor option for kernel arguments. The logic
was taken from the ostree-commit editor.

The user now can edit the kernel arguments from an editor.

The synchronization related issue is resolved by reloading the
sysroot and see if the sysroot has been changed since the call
of the editor prompt.

Closes: #1097
Approved by: jlebon
2017-11-28 18:29:42 +00:00
Colin Walters
b5c8915ea2 postprocess: Add envvar option, and detect NFS, skip ostree txn
This is for: https://pagure.io/atomic-wg/issue/387

Right now the way libostree stages objects into `${repo}/tmp` is basically an
anti-pattern for (possibly concurrent) operations on NFS. Having multiple
processes try to clean the tmpdir invites races, and there's really no reason to
"stage" all of the content.

(Unfortunately unless NFS supports `O_TMPFILE` we still need temp files,
 but that's a separate issue)

In this patch we auto-detect NFS which should make the Fedora pungi runs "just
work", but I also added an environment variable to opt-in.

Closes: #1111
Approved by: jlebon
2017-11-22 16:30:56 +00:00
Colin Walters
366826132a libpriv: Rename "unpacker" to "importer", split off "unpacker-core"
In the jigdo path we don't actually want to import the OIRPM literally
into ostree.  I considered adding jigdo logic into `rpmostree-unpacker.c`
but it'd be a mess as the functionality is quite logically separate
from importing.

So split off an `unpacker-core.c` file which has the bare libarchive+RPM
helpers, and rename `RpmOstreeUnpacker` to `RpmOstreeImporter`.

Closes: #1110
Approved by: jlebon
2017-11-20 20:57:33 +00:00
Colin Walters
3c0da8864b compose: Do relabel before downloading in --ex-unified-core
We removed this in review, but I rediscovered why I added it.  We
fail the `g_assert (sepolicy_matches)` if we already had packages
done with the final label when we go to reuse the cache.

(Basically, if we use the cache multiple times it's hard to avoid
 relabeling all the time which is unfortunate...gets back a bit to
 a way to annotate pkgcache commits as supporting multiple policies)

Closes: #1109
Approved by: jlebon
2017-11-20 15:35:21 +00:00
Colin Walters
6bf4206223 compose: Add --ex-unified-core
The "--ex" prefix here means it's an experimental option. A tremendous change
here is that start to support non-uid 0, but there are various things to fix there;
the unpacker for example needs to learn to set imported objects fully based
on the rpmfi information (i.e. default to uid 0, since libarchive gives the
current uid by default).

And even when run as uid 0, there are some bugs, though I'm not sure
of any showstoppers yet.  For example, dracut's `dracut-install` calls
`cp --preserve=xattrs` which fails to copy the `user.ostreemeta` xattrs
from a checkout (it shouldn't be copying that anyways...)

Nevertheless, the infrastructure behind this really helps (is almost a hard
requirement for) the [jigdo effort](https://github.com/projectatomic/rpm-ostree/issues/1081).
Which is really only true due to SELinux - we need to import the packages,
then generate the final tree to get the final policy, then use that policy
to relabel all of the packages.

Closes: #940
Approved by: jlebon
2017-11-17 18:59:34 +00:00
Colin Walters
94e52abb03 core: Don't try to apply non-root uid/gid when run as non-root
In an unprivileged case, we can't do this on the real filesystem. For
`ex container`, we want to completely ignore uid/gid.  I added a test
installing `httpd` which failed previously.

TODO: For non-root `--ex-unified-core` we need to do it as a commit modifier.

Closes: #940
Approved by: jlebon
2017-11-17 18:59:34 +00:00
Colin Walters
be89b68873 unpacker: Handle uid/gid when running unprivileged (non-root)
It seems that libarchive ends up returning `getuid()` actually,
possibly because the cpio doesn't actually have ownership information?
Anyways, what we really want here is to set `0/0`, which is what
happens for the `ex container` path via
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS`.

Prep for unified core 🌐.

Closes: #940
Approved by: jlebon
2017-11-17 18:59:34 +00:00
Colin Walters
9d4b760754 daemon,libpriv: Some porting to decl-after-stmt
Not immediate prep for anything, I'd just been looking at this
code recently.

(Also a bit of `glnx_throw_errno` porting)

Closes: #1107
Approved by: jlebon
2017-11-17 16:27:53 +00:00
Colin Walters
7ab8869cdd tree-wide: s/hif/dnf/
No real particular reason, just decided to do this while the outstanding patches
don't conflict too much.

Closes: #1106
Approved by: jlebon
2017-11-17 14:46:57 +00:00
Colin Walters
1240d8df2d unpacker,core: Honor nodocs flag
More work for unified core; this was one of the biggest parts of the `ostree
diff` when comparing a unified core treecompose of Fedora Atomic Host versus
the libdnf path.

This is also the first case where we're teaching the pkgcache to redownload
based on state.

Closes: #1105
Approved by: jlebon
2017-11-16 15:13:37 +00:00
Colin Walters
8c4ffc3340 unpacker,core: Handle selinux-policy-targeted at unpack time
Previously in 4a86bdd196 "postprocess: Undo Fedora SELinux policy move"
we fixed things for SELinux policy to move it out of /var and back
into /etc.

But for unified core where we're importing the RPM, the default "/var →
tmpfiles.d" kicks in before this can run. So teach the unpacker to do it too.
This is even less nice than what we had in the postprocessing code, as there we
went to some effort to handle non-targeted policy (e.g. MLS). That said I am not
super concerned about that right now...we can always deal with MLS or other
policies later. I'd be more inclined really to try to take the issue of moving
back to /etc (or implementing /usr with /etc overrides) to SELinux upstream.

Closes: #1105
Approved by: jlebon
2017-11-16 15:13:37 +00:00
Colin Walters
775e6f87ae core: (refactor) Load pkgcache commit object earlier
For unified core work, we'll be caching the "has docs" flag in the pkgcache
commit. So let's load the commit object earlier rather than having each function
load it.

Closes: #1104
Approved by: jlebon
2017-11-15 23:27:06 +00:00
Colin Walters
2e5848c5f4 core: (refactor) Style cleanup for pkg cache state load
Avoid the `goto` by initializing the output variables directly early, so we can
do a full new style consistently. Prep for unified core work.

Closes: #1104
Approved by: jlebon
2017-11-15 23:27:06 +00:00
Colin Walters
b17af329bb passwd: Always break hardlinks for shadow,gshadow, handle no merge
Prep for unified core 🌐.  First, right now `shadow-utils` always opens those with
`O_RDWR`, so let's break those links even if we don't have `usr/lib/passwd`.

Also, while not strictly requried, let's also handle the `merge_passwd_dir`
being `NULL`, as that happened when I was trying a variant of this patch.

Closes: #1098
Approved by: jlebon
2017-11-14 16:35:28 +00:00
Colin Walters
78bd4d6c64 core: Always process usr/etc/passwd
Prep for unified core 🌐. Here we don't have `/usr/lib/passwd` at the start, but
we still want to load the `usr/etc` copies for processing rpmfi overrides.

Closes: #1098
Approved by: jlebon
2017-11-14 16:35:28 +00:00
Colin Walters
5735e23a0f core: Use ADD_FILES checkout mode for "setup"
Prep for unified core 🌐.  On the treecompose path we inject
`/usr/etc/passwd`, and trying to overlay the setup package on top
with the default `UNION_IDENTICAL` fails.

This is obviously a hack; but I can't think of a way to do this significantly
better. (We could do a DB query for the owner of `/etc/passwd`, but eh.)

Closes: #1098
Approved by: jlebon
2017-11-14 16:35:28 +00:00
Jonathan Lebon
00fab1ec23 compose CLI: fix --repo consuming two arguments
I was trying to figure out why:

    rpm-ostree compose tree --repo repo/ manifest.json

would result in:

    error: opendir(manifest.json): No such file or directory

It turned out to be because we had `--repo` in *both* the `install`
options and the `commit` options. This makes sense since both these
subcommands need to be given a repo. However, in the `tree` case, we
were adding both arrays, which meant we inherited two `GOptionEntry`s
for `--repo`. This confused glib2 which consumed not one but two CLI
arguments when looking for the argument associated with `--repo`.

Our CI didn't notice this because it uses the `--repo=foo` notation,
which doesn't throw off glib2.

Fix this by factoring out the `--repo` option into a separate array so
that in the `tree` case, it only gets added once. Exercise the fix in CI
by using the two argument notation for `--repo`.

Closes: #1101
Approved by: cgwalters
2017-11-13 14:13:27 +00:00
Colin Walters
d637bedf3e passwd: Don't use alloca() in a loop
Even if it's a very small loop.

Closes: #1088
Approved by: jlebon
2017-11-10 20:04:16 +00:00
Colin Walters
5a762dd458 lib: Hoist "break single hardlink" API to util.c, use in passwd
The passwd code can reuse the "break hardlink" function we had in the core; lift
it to `util.c`.

Closes: #1088
Approved by: jlebon
2017-11-10 20:04:16 +00:00
Colin Walters
bc49ec2e1c lib/db: Port to new style
No functional changes, and not prep for anything. Was just reading this code
during my jigdo work.

Closes: #1100
Approved by: jlebon
2017-11-10 18:03:14 +00:00
Colin Walters
07969136f2 lib/scripts: Only log to journal if stdout is already the journal
The jigdo work is turning into "partial unified core 🌐" (mostly due to SELinux,
but that's an aside); we made a change here previously to avoid using
the journal if `uid != 0`, but it's more correct to check whether or
not we're already using the journal.

Concretely this fixes calling `rpmostree_context_assemble_tmprootfs()` as uid 0
inside my dev container, running `rpm-ostree compose commit2jigdo`.

Closes: #1084
Approved by: jlebon
2017-11-09 03:36:23 +00:00
Colin Walters
516588d4f7 core: Use tmpdir in ${repo}/tmp for SELinux relabeling
Needed for unified core work for the treecompose side; there
`/var` may not actually be on the same mount as the repo.

Do what we do in a ton of other places (and should probably push
into ostree somehow): allocate a tmpdir in `${repo}/tmp`.

Closes: #1095
Approved by: jlebon
2017-11-09 03:15:26 +00:00
Colin Walters
b3267eb336 scripts: Ignore nfs-utils.post
The maintainer can't apparently be bothered to review my patch,
and we don't need the `%post` anyways - and this is now blocking
my jigdo work.

Closes: #1092
Approved by: jlebon
2017-11-08 16:00:20 +00:00
Colin Walters
94a1776fe7 unpacker: Error out when trying to install RPM in /usr/local
This came up with `https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-license-9-0-9.0.176-1.x86_64.rpm`.
Reported by @dustymabe on IRC.

Closes: #1090
Approved by: jlebon
2017-11-08 03:56:20 +00:00
Colin Walters
407bfd5bc1 Lift 'override' out of experimental
We have a *lot* of experimental functionality.  I think the
`override` bits are fleshed out enough now that we can lift
the `ex` designation.  For example, jlebon fixed SELinux
labeling in the presence of override-replace.

Closes: #1089
Approved by: jlebon
2017-11-08 03:35:08 +00:00
Colin Walters
7ea2a5afce (refactor) Move pkgcache helper function to sysroot-core
Trying to drain the `util.c` file; this one lives more logically
there.

Closes: #1086
Approved by: jlebon
2017-11-08 03:11:36 +00:00
Colin Walters
8b4c5e0901 tree-wide: Use glnx_close_fd()
It's cleaner, and we get `EBADF` checks, etc.

Closes: #1085
Approved by: jlebon
2017-11-08 02:50:12 +00:00
Colin Walters
a42c45f726 lib/core: Synthesize tmpdir for commit if no base provided
The `ex container` path was already doing this - lower the logic
into the core so I can use it for the jigdo path.  I think it's just
clearer as well even for the sysroot pkglayering path.

Closes: #1083
Approved by: jlebon
2017-11-08 02:12:52 +00:00
Colin Walters
170f74c282 lib/core: Add API to set devino cache
Rather than passing it to the assemble functions.

Prep for jigdo work.  I'm trying to move more logic
from the system upgrader path to the core, since some of
it is needed for jigdo and "unified core".

Closes: #1083
Approved by: jlebon
2017-11-08 02:12:52 +00:00
Colin Walters
fe0f268bcc lib/postprocess: Create OstreeSePolicy more directly
We already called the `workaround_selinux_cross_labeling_recurse()`
in the postprocessing path, there's no need to do it again during
commit.

Just making this change as I was going to do some SePolicy stuff
for the [jigdo work](https://github.com/projectatomic/rpm-ostree/issues/1081)
and stumbled again into the ugly mess that is the cross-labeling
hack.

Closes: #1082
Approved by: jlebon
2017-11-08 02:02:43 +00:00
Colin Walters
1857a6a1ef ex-unpack: Open OstreeSePolicy more directly
We should definitely not be calling the "postprocess" version here
as that may *change* the source filesystem.

Closes: #1082
Approved by: jlebon
2017-11-08 02:02:43 +00:00
Ruixin Bao
5ca9c731d7 tests/check: Add unit tests for kargs options
Added unit tests for rpm-ostree ex kargs --delete,
--append and --replace.

Also exposed two getter functions for kargs table
and array so people can retrieve information from
kargs.

Also includes a minor fix for a bug caught by the unit
test.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
27de772c4f libpriv/vmcheck: support replace option for karg command
Refactor the similar code portion from delete option into
a new function, and reuse it in the replace logic.

Add the replace functionality to allow kernel arguments
replacement. Now support two different types of replacement:

1: replace by key=value when there is only one key existing
e.g we have 'hi=hello' as our one of the arguments (hi only has
one value associated with it) , and 'rpm-ostree ex kargs --replace hi=new'
will replace the value and change hi=hello into hi=new.

2: replace by key=value=new_value for all other replacements
( this will work for swapping single value pair too !)

Some tests for rpm-ostree ex kargs --replace are added
for future regression.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
09cf438f04 vmcheck: Add tests for rpm-ostree ex kargs delete and append
Tests are introduced in this commit to test the basic
functionality for rpm-ostree ex kargs --append,
and rpm-ostree ex kargs --delete.

Those tests are added for future regression.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
df3b5c0c4e daemon/libpriv: add handlation for kargs delete option
Some new functions are added to handle delete operations
for kargs. We are now able to do the following delete operations kargs:

1: delete by key, if there is only one value associated with the key
we directly remove it
2: delete by key=value, it will find that specific key/value pair
and remove it from the kernel arguments

Similar to append functionality introduced, delete will
also create a new pending deployment that is rollbackable

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
50863c370d daemon: add handlation for kargs append
API functions from ostree-kernel-args.c
are copied to libpriv. The append functionality
reuses  _ostree_kernel_args_append_argv() for
collecting added kernel arguments.

Also added handlation in rpm-ostree upgrader
to allow deployments happen with kernel arguments.

Now, the user is able to add kernel arguments via
'rpm-ostree ex kargs --append key=value'
or 'rpm-ostree ex kargs --append key' if they
want to have an empty value with key.

The user is also able to display the current
kernel arguments via 'rpm-ostree ex kargs'

In addition, this functionality will create a pending deployment,
and will update the conf file in /boot/loader/entries/ostree-$osname-0.conf upon success.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
fa593afd89 app/daemon: add groundwork for rpm-ostree kargs command
This is initial groundwork for https://github.com/projectatomic/rpm-ostree/issues/594.

This commit sets up most of the required
front end logic( arg parsing, transaction handling), and will
be used in the following commits.

There is nothing really fancy in this commit, as most of the code
shares the similar style between other dbus related commands.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Colin Walters
e7a42f70a9 compose: Add rpmostree.rpmmd-repo metadata to commits by default
This is a revisit of a PR for client-side layering: https://github.com/projectatomic/rpm-ostree/pull/1072
Here though we're doing this by default for server-side composes.
There are a few reasons to do this; first, I'm seeing an issue
in some of our Jenkins jobs for Fedora that hit "mirror roulette"
and end up creating commits that "revert" to older versions temporarily.

While I've [certainly pitched](https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/IMPE6KCRBHCEJH5VBE6ZFIRLPAD743JT/) this as a feature, I think
we really want something like `--force-older-timestamp` - basically
error out if the timestamps on one or more input repos were older.
Not doing that in this patch, but it paves the way to do so.

Second, I'd like to use this data in the `ostree.source-title`
metadata key down the line.  Something like:

`└ rpmmd: fedora-26 (20170310), fedora-26-updates (20171101)`

(This could be a lot nicer if we drive versioning in to the rpm-md repo info,
 and e.g. there's some friendly "week number" style versioning for the updates
 repo now that it's batched...for now we have timestamps)

For CentOS/RHELAH this gets interesting and potentially more verbose,
to the point where we may want to render it more explicitly.

But anyways, let's do this now, as it will be useful even without
an explicit rendering, since users can do e.g. `ostree show` on
a base commit hash to dump the data.

I had a concern that some users may not want to emit this metadata;
they can currently do `--add-metadata-string rpmostree.rpmmd-repos ''`
and that will "win".

Closes: #1079
Approved by: jlebon
2017-11-04 21:02:56 +00:00
Colin Walters
d6218e0e16 postprocess: Use names (not ids) in synthesized tmpfiles.d files
Related to: https://github.com/projectatomic/rpm-ostree/issues/49

We want to support "name binding" per client system, rather than
having a hardcoded mapping in our tree.  Currently if e.g. a new
daemon is added as a dependency (or as part of e.g. systemd) it's
easy to silently miss it.

This is prep for doing that binding client side consistently, which is what we
do with package layering.

Closes: #1077
Approved by: jlebon
2017-10-27 19:46:26 +00:00
Colin Walters
669031f63b bin/compose: Print correct final rootfs path
This was a thinko; `workdir_tmp` won't be set in this path. I think the code
changed here at some point.

Closes: #1076
Approved by: jlebon
2017-10-26 21:20:40 +00:00
Colin Walters
964ab1f8bc bin/compose: Do passwd checking in commit, not install
We won't have done the postprocessing, so `/usr/lib/passwd` won't exist. Trying
to use `compose install` with current fedora-atomic failed (I *really* should
have tested that at least manually with the final patchset). Add `check-passwd`
to the test suite so this gets coverage too.

Closes: #1076
Approved by: jlebon
2017-10-26 21:20:40 +00:00
Colin Walters
2285f0f6db Inline assemble_commit() wrapper into ex-container, the only user
This is a cleanup I was going to do before, but had dropped. It's a general
followup to the `compose` rework; there's not much point to having a high level
wrapper, given that we generally need to do some postprocessing. Inline it into
the `ex container` code, which makes the core more "core".

Closes: #1074
Approved by: jlebon
2017-10-26 15:44:58 +00:00
Colin Walters
59fec100bf bin/compose: Fix use of EXIT_FAILURE for error paths
I got a critical from an nonexistent `DESTDIR` in `rpm-ostree compose install`.

(I'm probably going to do a patch to add `rpmostree_command_invocation_set_exit_code()`
 so the "exit 77" case can use that and everything else can `s/return EXIT_FAILURE/return FALSE/`
 and we'll be free of these bugs)

Closes: #1075
Approved by: jlebon
2017-10-26 14:24:47 +00:00
Colin Walters
cb9889204c lib/unpacker: Add repo.timestamp to metadata
Related to: https://github.com/projectatomic/rpm-ostree/issues/774

We aren't yet trying to render this in any way on the client side, but let's
start capturing the timestamp data now so that we can make use of it later
in e.g. `rpm-ostree status -v`.

Closes: #1072
Approved by: jlebon
2017-10-25 18:18:10 +00:00
Colin Walters
63af4bbdda bin/compose: Expose phases as [install, postprocess, commit] cmds
Right now `rpm-ostree compose tree` is very prescriptive about how things work.
Trying to add anything that isn't an RPM is absolutely fighting the system. Our
postprocessing system *enforces* no network access (good for reproducibilty, but
still prescriptive).

There's really a logical split between three phases:

 - install: "build a rootfs that installs packages"
 - postprocess: "run magical ostree postprocessing like kernel"
 - commit: "commit result to ostree"

So there are two high level flows I'd like to enable here. First is to allow
people to do *arbitrary* postprocessing between `install` and `commit`. For
example, run Ansible and change `/etc`. This path basically is like what we have
today with `postprocess-script.sh`, except the builder can do anything they want
with network access enabled.

Going much farther, this helps us support a "build with Dockerfile" style flow.
We can then provide tooling to extract the container image, and combine
`postprocess` and `commit`.

Or completely the other way - if for example someone wants to use `rpm-ostree
compose install`, they could tar up the result as a Docker/OCI image. That's now
easier; an advantage of this flow over e.g. `yum --installroot` is the "change
detection" code we have.

Related issues/PRs:

 - https://github.com/projectatomic/rpm-ostree/pull/96
 - https://github.com/projectatomic/rpm-ostree/issues/471

One disadvantage of this approach right now is that if one *does* go for
the split approach, we lose the "input hash" metadata for example.  And
down the line, I'd like to add even more metadata, like the input rpm repos,
which could also be rendered on the client side.

But, I think we can address that later by e.g. caching the metadata in a file in
the install root and picking it back up or something.

Closes: #1039
Approved by: jlebon
2017-10-25 17:43:09 +00:00
Colin Walters
1f31cbb99a status: Render ostree metadata source-title key
Depends: https://github.com/ostreedev/ostree/pull/1296

As I mention in the commit there, I see two uses for this in rpm-ostree; first
in our test suite, and second for OCI-built image imports.

I also took a step further here and inject an `original-origin` metadata
key, though we aren't actually using that yet.  The problem I'm trying
to solve there is that repeated `make vmoverlay` starts chaining things up,
but that gets very confusing.  I think we should always have `vmoverlay` unwind
back to the base ref.  (Or at least do that by default)

Closes: #1069
Approved by: jlebon
2017-10-25 03:37:05 +00:00
Colin Walters
9ec45d3c89 daemon: Add commit metadata to deployment properties
In another PR I did the manual bridging of commit metadata to deployment
property, but that's annoying. Let's just bridge all commit metadata.

Closes: #1069
Approved by: jlebon
2017-10-25 03:37:05 +00:00
Colin Walters
aaf0d978b8 postprocess: Also move RPM database if not done already
Prep for split compose.  The current Docker/OCI base images we
built at least for Fedora don't have this move done, so let's
ensure our postprocessing does it.

Closes: #1070
Approved by: jlebon
2017-10-23 20:35:41 +00:00
Colin Walters
21167bea72 postprocess: Deal with an already empty /etc/machine-id
Prep for split compose. The Fedora base container image already empties
`/etc/machine-id`; let's handle that case rather than throwing an error. This
makes it easier to run our postprocessing on an image built on top of that base
image.

Closes: #1070
Approved by: jlebon
2017-10-23 20:35:41 +00:00
Colin Walters
d02bc4b03c compose: Accept NULL treefile for "use defaults" postprocessing
This is prep for split-compose.  We have some options in the
treefile, like `boot_location` and `tmp-is-dir` etc.  While those
are useful options, I don't want to force everyone using
`rpm-ostree compose postprocess` to write a treefile.

Change the code then to accept a `NULL` treefile to mean
"use the defaults".

Closes: #1070
Approved by: jlebon
2017-10-23 20:35:41 +00:00
Colin Walters
5da2894823 ex-container: Always use canonical permissions for commit
Fixes a fsck issue that I saw while submitting another patch.

This is the second time in the last week I've fixed an issue like this; will
probably do a libostree patch to just turn it on by default for
`bare-user-only`, but let's fix this bug anyways.

Closes: #1071
Approved by: peterbaouoft
2017-10-23 14:12:10 +00:00
Colin Walters
a9c8b1fae1 ex-container: Make /etc/shadow 0400 on import, not post-checkout
Switching to the `_CONSUME` flag revealed an "oh god how did I write that"
bug in the previous patch in https://github.com/projectatomic/rpm-ostree/pull/1046
AKA commit: 334f0b89be

The way that actually fixed the bug before was because we were using
hardlink checkouts, and we were operating outside an `rofiles-fuse`
context, we simply directly changed the on-disk object mode.

But with the `_CONSUME` flag we started deleting the files as we write,
meaning that stopped working.

I *initially* wrote a patch to do the same split "prepare/processing/commit"
flow that treecompose and package layering do, but that can't really fix this
bug - we need to do it on import.

So do the chmod on import and drop the postprocessing bits.

Closes: #1067
Approved by: jlebon
2017-10-18 17:07:54 +00:00
Colin Walters
d60cc0248a ex-container: Use shared helper for download+import+rootfs+commit
Prep for fixing a bug in the last (git) commit here for `/etc/shadow`.

Closes: #1067
Approved by: jlebon
2017-10-18 17:07:54 +00:00
Jonathan Lebon
c107a05b8e tree: pass NULL to glnx_fstatat_allow_noent when needed
Now that libglnx allows it, we can skip declaring a `struct stat` we
don't actually need just to check if a file exists.

Closes: #1066
Approved by: cgwalters
2017-10-17 20:48:13 +00:00
Colin Walters
3be6dbae15 Make use of new _CONSUME flag to ostree commit
This is used for both `ex container` and client-side operations, but not most
uses of `compose tree` (which should go into `bare-user` or `archive`).
We'll circle back and do `bare-user` for `_CONSUME` in libostree though
soon.

Closes: #1065
Approved by: jlebon
2017-10-17 18:49:11 +00:00
Colin Walters
c3a5aef80b bin/compose: Clean up ostree version metadata handling a bit
Just noticed this FIXME while working on split compose.  Use
the `#define` we have, and inline the duplicated function which
boils down to `g_variant_lookup()` - I simply changed `&s` to `s`
to inline the `strdup`.

Closes: #1064
Approved by: jlebon
2017-10-17 18:44:31 +00:00
Colin Walters
94527ab69c tree-wide: Use glnx_autofd
It's a better name.  Already ported ostree.

Closes: #1063
Approved by: jlebon
2017-10-17 16:01:34 +00:00
Colin Walters
583c626821 bin/compose: Hide no-op --workdir-tmpfs
We made it a no-op a bit ago in commit 162b8d4898
Let's hide it like we did for the `status --pretty` option.

Closes: #1061
Approved by: jlebon
2017-10-16 20:16:36 +00:00
Jonathan Lebon
4d1b5b4636 libpriv/core: use cached repodata checksum for goal
An interesting behaviour arises from the new `--cache-only` option. We
would essentially consider two goals with the same exact pkglist as
different depending on whether a package came from a yum repo or our
pkgcache repo. This meant that `rpm-ostree upgrade` followed by
`rpm-ostree upgrade -C` was never detected as a no-op and always
resulted in a relayer.

This was due to the fact that the checksum libsolv calculated was based
on our chopped off pkgcache meta-RPMs. This of course won't match the
checksum from the repodata, which is of the whole RPM.

Thankfully, we already save the repodata checksum when unpacking RPMs.
This patch simply detects the case where we're trying to checksum a pkg
originating from the pkgcache, and reads in the saved repodata checksum
rather than the wrong libsolv-derived one.

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
e49f7cdd81 app: add --download-only option
To complement the new `--cache-only` option, add a `--download-only`
option. This does exactly what it says: we download the ostree, download
and import packages, but don't actually commit & deploy. This can be
used to effectively prime a follow-up `--cache-only` operation that can
be done during a more convenient/safer maintenance window.

I debated naming the two options `--pull-only` and `--deploy-only` like
the ostree equivalents. Though "pull" felt like the wrong word given
that it's associated more with ostree pulling but rpm-ostree also
downloads & imports RPMs. As for `--deploy-only` vs `--cache-only`, it
seems like `--cache-only` is a more accurate description of the
functionality (i.e. rather than describing an action, it describes a
mode). I also considered `--no-download` to make the synergy with
`--download-only` more obvious. Maybe that's better? Naming is hard...

Closes: #713

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
24c184af03 app: imply cache-only mode for pkgs removals and uninstalls
Now that we have a strong notion of `cache-only` mode, make use of it
when performing an `uninstall` or `ex override remove/reset`.

Closes: #944

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
b811eb61c0 app: support full offline operations with --cache-only
As Colin mentioned in #1035, the new `--cache-only` implemented only the
rpmmd half of the story. Here we complete that story by also ensuring
that when in cache-only mode, we don't download new ostree data nor new
packages. We try to complete the requested operation with what we have.

To do this, we add support for the same `SYNTHETIC` pull that was added
in ostree[1] so that we don't actually pull, but still perform timestamp
checking.

On the pkgcache side, we disable all remote repos and instead insert all
our cached RPMs into the `DnfSack`. Care is taken to still perform
SHA256 verification for local pkg installs/replacements.

[1] https://github.com/ostreedev/ostree/pull/642

Closes: #687

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
cd3da57453 app/status: make deprecated --pretty flag hidden
Minor detail. We still need to support the `--pretty` flag, but it
doesn't do anything anymore, so let's just hide it to clean up the help
output.

Closes: #1060
Approved by: cgwalters
2017-10-16 16:46:02 +00:00
Colin Walters
aea688fa01 Update libglnx
I want to use `glnx_tmpdir_unset()` in the `compose commit` PR.

Update submodule: libglnx

Closes: #1056
Approved by: jlebon
2017-10-16 13:22:30 +00:00
Colin Walters
854c7db029 scripts: Fix close() → EBADF
Caught by bumping libglnx and its more widespread checking for `EBADF`. We
should probably add a `glnx_fdopen()` that steals ownership of the fd.

Closes: #1056
Approved by: jlebon
2017-10-16 13:22:30 +00:00
Colin Walters
43d52e7196 postprocess: (refactor) Split out "init rootfs" from "final postprocess"
Prep for [compose split](https://github.com/projectatomic/rpm-ostree/pull/1039).
The next step here will be to basically have `postprocess_final()` + `ostree commit`
be `rpm-ostree compose commit`.

This reworks the order for a few things - we move `/usr` first right after
initializing the rootfs, which works now because from previous commits `/etc`
is really `/usr/etc`.

We also change things so that we grab `/boot` and `/var` from the src root,
then `postprocess_final()` handles them in place.

One detail here is that I had to do the `rpmostree_rootfs_prepare_links()`
after the `convert_var_to_tmpfiles_d()` since it writes into var; before
this worked because we were using different roots.

Closes: #1054
Approved by: jlebon
2017-10-13 15:29:11 +00:00
Jonathan Lebon
2b4a65bca5 libpriv/core: also checksum package action
When calculating the goal checksum, we only accounted for package
installs. But with override support, there are now other package actions
we need to pick up on.

In practice, the fact that we also checksum the treespec made this hard
to actually cause an issue. Although, since the actual final goal is
also dependent on the underlying rpmdb, I can imagine e.g. a replacement
override be considered a `DNF_PACKAGE_INFO_UPDATE` for one rpmdb appear
as a `DNF_PACKAGE_INFO_DOWNGRADE` for another.

While we're here, do some light code style porting and streamline the
checksumming process to avoid creating a separate `GPtrArray`.

Closes: #1053
Approved by: cgwalters
2017-10-13 15:00:25 +00:00
Jonathan Lebon
dbceb2aa51 daemon/deploy: remove duplicate call to set_origin
We already set the origin unconditionally right after this block. No
point in `dup`'ing twice.

Closes: #1053
Approved by: cgwalters
2017-10-13 15:00:25 +00:00
Jonathan Lebon
24cff4f5e0 app/uninstall: remove --cache-only option
The `--cache-only` option was added to the generic `option_entries`
array, which is shared by both `install` and `uninstall`. But really, it
only makes sense for `install`. Move it to the `install` specific array
instead.

I also renamed the option entry arrays to make more sense. Before, the
`install_option_entries` referred to the additional options that the
`uninstall` command took. Now the names refer to the command to which
they're associated.

Closes: #1053
Approved by: cgwalters
2017-10-13 15:00:25 +00:00
Jonathan Lebon
74eaa2ba59 daemon: add RPMOSTREE_DEBUG_DISABLE_DAEMON_IDLE_EXIT
The new idle exit behaviour is nice, but it makes debugging it harder
because you have to be fast enough to attach or place your breakpoints
and trigger it before it auto-exits. Add a compile-time flag that
developers can easily turn on to disable the auto-exit behaviour.

Closes: #1052
Approved by: cgwalters
2017-10-12 15:35:44 +00:00
Colin Walters
b2350acf69 bin/compose: (refactor) Split out a constructor function
This moves the JSON parsing and reorders a few things, and
building on the previous commits, adds a constructor function.

Prep for [compose commit](https://github.com/projectatomic/rpm-ostree/pull/1039)

Closes: #1050
Approved by: jlebon
2017-10-12 13:49:42 +00:00
Colin Walters
05698b7c2e bin/compose: (refactor) Move rootfs_dfd into compose data struct
Prep for [compose commit](https://github.com/projectatomic/rpm-ostree/pull/1039).

Closes: #1050
Approved by: jlebon
2017-10-12 13:49:42 +00:00
Colin Walters
973a99895b bin/compose: (refactor) Hoist metadata hash into compose data struct
Prep for [rpmostree compose commit](https://github.com/projectatomic/rpm-ostree/pull/1039).
We still want to write the metadata in that path, so having the data
in the struct will help make things common between the two.

Closes: #1050
Approved by: jlebon
2017-10-12 13:49:42 +00:00
Colin Walters
334f0b89be ex-container: Make /usr/etc/{,g}shadow user-readable
For the `ex container` case, there's no security issues here; one shouldn't be
doing user management in these roots at all.

This is for work on exporting `ex container` roots to OCI as non-root. Without
this fix, libostree just tries to `openat()` the object for export to tar, and
fails.

See also https://github.com/projectatomic/rpm-ostree/issues/1045

Closes: #1046
Approved by: jlebon
2017-10-10 20:42:55 +00:00
Colin Walters
4a4f9952c2 tree-wide: Use a #define for /usr/share/rpm location
In prep for potentially changing it:
http://lists.rpm.org/pipermail/rpm-maint/2017-October/006681.html

Of course actually doing a transition would be harder than this, as we'd need to
add a compat symlink, and even that wouldn't quite be enough as e.g. the
"preview" code would need to learn how to follow the symlink (or just try both
locations).

In practice I think we'd need to land the code to handle both locations, let
that trickle out for e.g. 3 months, then make the switch in our treecomposes.

But, might as well make this change now; using a `#define` makes it slightly
easier to find places that need changing later.

Closes: #1048
Approved by: jlebon
2017-10-10 19:59:21 +00:00
Colin Walters
d0b225c0ab postprocess: Init rootfs earlier
There's a lot going on in the final postprocessing; and some
potentially tricky dependencies.  But we can cleanly move up
the "init rootfs" bits, which helps make the rest of the function
closer to the "final postprocessing" we need for
[split compose](https://github.com/projectatomic/rpm-ostree/pull/1039).

Closes: #1044
Approved by: jlebon
2017-10-10 13:06:35 +00:00
Colin Walters
236dfd723a compose: Clean up internal final postprocess API
It's really way saner if we create the target root dir in the
compose high level.  That way everything relating to actual file names
(and the concept of the "workdir") rather than the "library".

This is prep for [compose split](https://github.com/projectatomic/rpm-ostree/pull/1039)
in that it will be easier to tease out an API for less "opinionated"
postprocessing that just works on `target_rootfs_dfd`.

Closes: #1044
Approved by: jlebon
2017-10-10 13:06:35 +00:00
Colin Walters
b383ef894d compose/postprocess: Fix memleak in error path, minor style update
Using an autoptr for the strbuf not only fixes a memleak in the error path, it's
a bit more efficient since we can just pass `buf->len` rather than running
`strlen()`.

Closes: #1042
Approved by: jlebon
2017-10-06 19:43:39 +00:00
Colin Walters
7c4c2c6640 compose/postprocess: Unlink files in /var as we go
This is prep for compose splitting:
https://github.com/projectatomic/rpm-ostree/issues/471

We'll need to run some postprocessing types in both phases, and it's a lot
easier to reason about if the postprocessing is "idempotent", i.e. it has no
effect a second time.

Closes: #1042
Approved by: jlebon
2017-10-06 19:43:39 +00:00
Jonathan Lebon
4442a0c362 app: add -C flag to always use cached metadata
This is the equivalent version of `yum/dnf -C`. It goes together with
the new `makecache` command to allow completely asynchronous cache
update and usage.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
8285ef2a15 app: add 'refresh-md' command
This is essentially the `dnf/yum makecache` equivalent for rpm-ostree.
To complete the picture, this goes hand in hand with the `-C`
equivalent, which is added in the next patch.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
5601a60398 libpriv/core: log repo info when downloading rpmmd
Move the logging of yum repo information from `prepare` to
`download_metadata`, since the latter could be called without
necessarily calling the former, as is the case with `makecache`.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Colin Walters
a2862f5473 bin/compose: (minor) Rename treefile variable for clarity
We had two things called `treefile`, make one of them the path so things are
less confusing.

Closes: #1041
Approved by: jlebon
2017-10-06 15:36:04 +00:00
Jonathan Lebon
33cb342b6b libpriv/core: allow NULL for treespec
Let callers pass `NULL` for the `@spec` argument. This essentially means
that they're not interested in actually doing any installation/commit
operations, but just the semantics that the core uses when setting up
libdnf wrt e.g. install and source root and enabled repos = required,
etc...

Prep for `makecache`.

Closes: #1038
Approved by: cgwalters
2017-10-05 18:00:54 +00:00
Jonathan Lebon
8fae56d120 libpriv/core: teach core to use deployment for yum repos
Add a function in the core that knows how to take a deployment and point
its passwd dir and libdnf's repos dir to it. We call it the
"configuration deployment", because those are configuration files which
should be inherited from the config merge, not the origin merge.

Prep for `makecache`.

Closes: #1038
Approved by: cgwalters
2017-10-05 18:00:54 +00:00
Colin Walters
95227f079a WIP: Implement ex livefs --replace
Lots of tradeoffs in this.  See the comments in the code for
more info.  WIP for now.

Closes: #1028
Approved by: jlebon
2017-10-05 13:19:20 +00:00
Jonathan Lebon
4501791dc2 daemon: fix uint negative comparison
Closes: #1036
Approved by: cgwalters
2017-10-04 21:10:55 +00:00
Colin Walters
f3c63b6850 app: Unify some cmdline txn processing
We had duplicated code across the cmdline entrypoints for transaction
processing; things like "print pkg diff only if !opt_reboot".

This doesn't dedup all of them - there are some corner cases around
the preview logic in `upgrade`, and `initramfs` also need special
handling.  I'll likely enhance this further down the line for that.

But one reason I'm doing this now is prep for:
[rpm-ostree cancel](https://github.com/projectatomic/rpm-ostree/pull/1019)

Basically, I want to add a `-B/--background` option we honor consistently, and
that'd be a lot easier if we have a combined "start/monitor txn" with the
post-txn option processing in one place.

Closes: #1034
Approved by: jlebon
2017-10-04 13:23:49 +00:00
Colin Walters
025633abed app: Add a more reliable helper for finding active transaction
There are (somewhat hard to avoid) race conditions for the client
to find the active transaction and connect to it.  This approach
adds a retry loop, and teaches the status builtin to use it.

The status code becomes a lot less ugly.  Prep for:
[rpm-ostree cancel](https://github.com/projectatomic/rpm-ostree/pull/1019)

Closes: #1033
Approved by: peterbaouoft
2017-10-03 21:40:20 +00:00
Colin Walters
cafe89cab7 daemon: Exit on idle after ~10-15 seconds
Now that we have the ability to both track clients and our active transaction,
and the `RegisterClient` call acts "atomically", let's start doing exit-on-idle
and return the RAM to the people.

Closes: #606
Approved by: jlebon
2017-10-03 17:56:21 +00:00
Colin Walters
d31c9da236 app: Call RegisterClient with well-known name, use retry loop
This closes a race condition for having the daemon idle exit.  After
the daemon has released its bus name, the dbus-daemon will no longer
allow messages through that targeted its unique name.

Since the intention of the `RegisterClient` method is to be the "knock on the
door", fix this by directly sending a message to the well-known name.

Second, we need to handle the case where the daemon exits without
replying; @jlebon added a `sleep(10)` invocation after the daemon
mainloop quit but before we `ReleaseName`, and I verified these two
things combine to fix that case.

Closes: #606
Approved by: jlebon
2017-10-03 17:56:21 +00:00
Colin Walters
3f367dbce2 scripts+bwrap: Make script execution cancellable
Prep for implementing `rpm-ostree cancel`, but this works with the way we handle
`Ctrl-C` interactively on a client as well. Being able to cancel a script
execution is quite nice; some of them are expensive, and having one loop forever
has been known to happen.

Closes: #1025
Approved by: jlebon
2017-10-03 14:24:07 +00:00
Jonathan Lebon
a9c38d33b8 vmcheck: add SELinux labeling tests
There's a lot of paths in the core related to SELinux policy changes and
relabeling packages. We currently have no test coverage for them. We add
support in the test libraries here to build such packages.

We also add a test that checks both that we correctly relabel RPMs when
the policy changes and that we handle layered packages that install
SELinux packages properly.

Closes: #999
Approved by: cgwalters
2017-10-03 01:01:14 +00:00
Jonathan Lebon
4ad5747b96 libpriv/core: use final sepolicy when doing final commit
It's possible that a layered package installed an SELinux package which
can change the labeling of some files. The problem is that imported
packages will not have used the updated policy and thus may have some of
its paths mislabeled. We fix this here by always loading the SELinux
policy from the tmprootfs and using that for the final commit.

Closes: #999
Approved by: cgwalters
2017-10-03 01:01:14 +00:00
Colin Walters
699d80ba2d bin/compose: Drop workdir variable
Nothing actually uses it, we've been all fd-relative for a while. Just noticed
this while looking at the compose code for further work.

Closes: #1030
Approved by: jlebon
2017-10-02 18:23:54 +00:00