Commit Graph

1888 Commits

Author SHA1 Message Date
Colin Walters
a822722a5b ex-unpack: Delete
I think the `ex container` path supercedes this; it was really just a demo, and
having it around is annoying since I want to change the importer API and I have
to change it here too.

Closes: #1116
Approved by: jlebon
2017-11-29 16:48:19 +00:00
Colin Walters
f58cb44fb2 txn: For --download-only, print 'changed' message if changed
I was playing with `--download-only` a bit with an eye to
having something like this be used by Cockpit/gnome-software instead
of what it's doing now, but a problem is that at the moment we
don't have a way to reflect the "changed" state back to clients.

This is a first step towards that by simply printing a different
message.

I think really to make all of this work more nicely though, including
supporting e.g. rpm database diffs, we are going to have to instead
work on the [pending deployment](https://github.com/ostreedev/ostree/issues/545)
path.  That way we'll have done the depsolve, stored repo timestamps
etc.; we'll be able to accurately show what *did* change rather than
try to recreate what will happen on the next `rpm-ostree upgrade --cache-only`.

Closes: #1118
Approved by: jlebon
2017-11-29 15:29:49 +00:00
Colin Walters
b68fe099ea tests: Use ostree commit --consume
Just noticed this while working on something else; we should use it where
possible.

Closes: #1117
Approved by: jlebon
2017-11-29 15:07:25 +00:00
Colin Walters
1c5c4a457e tests: Correctly error out if some part of install.sh fails
It took me way, way, way too long to debug that my dev container somehow missing
`ostree-grub2` caused the script to fail early, but we'd still continue trying
to sync, leading to us not actually changing the installed `rpm-ostree` binary,
leading to test suite failures.

Also add the chmod hammer at the top so we can delete even if we fail after
doing an rsync.

Closes: #1115
Approved by: jlebon
2017-11-29 14:42:25 +00:00
Ruixin Bao
2ef240d401 app/libpriv: add editor option for karg command
Add editor option for kernel arguments. The logic
was taken from the ostree-commit editor.

The user now can edit the kernel arguments from an editor.

The synchronization related issue is resolved by reloading the
sysroot and see if the sysroot has been changed since the call
of the editor prompt.

Closes: #1097
Approved by: jlebon
2017-11-28 18:29:42 +00:00
Jonathan Lebon
e35f867494 papr: fix sanity test and bump to f27
It was recently changed to use virtualenv to install and invoke ansible.

Closes: #1112
Approved by: miabbott
2017-11-22 16:55:17 +00:00
Colin Walters
b5c8915ea2 postprocess: Add envvar option, and detect NFS, skip ostree txn
This is for: https://pagure.io/atomic-wg/issue/387

Right now the way libostree stages objects into `${repo}/tmp` is basically an
anti-pattern for (possibly concurrent) operations on NFS. Having multiple
processes try to clean the tmpdir invites races, and there's really no reason to
"stage" all of the content.

(Unfortunately unless NFS supports `O_TMPFILE` we still need temp files,
 but that's a separate issue)

In this patch we auto-detect NFS which should make the Fedora pungi runs "just
work", but I also added an environment variable to opt-in.

Closes: #1111
Approved by: jlebon
2017-11-22 16:30:56 +00:00
Colin Walters
366826132a libpriv: Rename "unpacker" to "importer", split off "unpacker-core"
In the jigdo path we don't actually want to import the OIRPM literally
into ostree.  I considered adding jigdo logic into `rpmostree-unpacker.c`
but it'd be a mess as the functionality is quite logically separate
from importing.

So split off an `unpacker-core.c` file which has the bare libarchive+RPM
helpers, and rename `RpmOstreeUnpacker` to `RpmOstreeImporter`.

Closes: #1110
Approved by: jlebon
2017-11-20 20:57:33 +00:00
Colin Walters
3c0da8864b compose: Do relabel before downloading in --ex-unified-core
We removed this in review, but I rediscovered why I added it.  We
fail the `g_assert (sepolicy_matches)` if we already had packages
done with the final label when we go to reuse the cache.

(Basically, if we use the cache multiple times it's hard to avoid
 relabeling all the time which is unfortunate...gets back a bit to
 a way to annotate pkgcache commits as supporting multiple policies)

Closes: #1109
Approved by: jlebon
2017-11-20 15:35:21 +00:00
Colin Walters
d87970966b ci/papr: Update to f27-primary
I didn't try porting some of the other contexts yet...I want to see
if this works.

Note that FAHC was switched to 27, so it's only luck that things
work on 26.

Closes: #1108
Approved by: jlebon
2017-11-17 19:46:38 +00:00
Colin Walters
6bf4206223 compose: Add --ex-unified-core
The "--ex" prefix here means it's an experimental option. A tremendous change
here is that start to support non-uid 0, but there are various things to fix there;
the unpacker for example needs to learn to set imported objects fully based
on the rpmfi information (i.e. default to uid 0, since libarchive gives the
current uid by default).

And even when run as uid 0, there are some bugs, though I'm not sure
of any showstoppers yet.  For example, dracut's `dracut-install` calls
`cp --preserve=xattrs` which fails to copy the `user.ostreemeta` xattrs
from a checkout (it shouldn't be copying that anyways...)

Nevertheless, the infrastructure behind this really helps (is almost a hard
requirement for) the [jigdo effort](https://github.com/projectatomic/rpm-ostree/issues/1081).
Which is really only true due to SELinux - we need to import the packages,
then generate the final tree to get the final policy, then use that policy
to relabel all of the packages.

Closes: #940
Approved by: jlebon
2017-11-17 18:59:34 +00:00
Colin Walters
94e52abb03 core: Don't try to apply non-root uid/gid when run as non-root
In an unprivileged case, we can't do this on the real filesystem. For
`ex container`, we want to completely ignore uid/gid.  I added a test
installing `httpd` which failed previously.

TODO: For non-root `--ex-unified-core` we need to do it as a commit modifier.

Closes: #940
Approved by: jlebon
2017-11-17 18:59:34 +00:00
Colin Walters
be89b68873 unpacker: Handle uid/gid when running unprivileged (non-root)
It seems that libarchive ends up returning `getuid()` actually,
possibly because the cpio doesn't actually have ownership information?
Anyways, what we really want here is to set `0/0`, which is what
happens for the `ex container` path via
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS`.

Prep for unified core 🌐.

Closes: #940
Approved by: jlebon
2017-11-17 18:59:34 +00:00
Colin Walters
71d04a44b8 tests/ex-container: Disable parallelism for now
Adding the 2nd test revealed this didn't actually work, will fix at some point.

Closes: #940
Approved by: jlebon
2017-11-17 18:59:34 +00:00
Colin Walters
9d4b760754 daemon,libpriv: Some porting to decl-after-stmt
Not immediate prep for anything, I'd just been looking at this
code recently.

(Also a bit of `glnx_throw_errno` porting)

Closes: #1107
Approved by: jlebon
2017-11-17 16:27:53 +00:00
Colin Walters
7ab8869cdd tree-wide: s/hif/dnf/
No real particular reason, just decided to do this while the outstanding patches
don't conflict too much.

Closes: #1106
Approved by: jlebon
2017-11-17 14:46:57 +00:00
Colin Walters
1240d8df2d unpacker,core: Honor nodocs flag
More work for unified core; this was one of the biggest parts of the `ostree
diff` when comparing a unified core treecompose of Fedora Atomic Host versus
the libdnf path.

This is also the first case where we're teaching the pkgcache to redownload
based on state.

Closes: #1105
Approved by: jlebon
2017-11-16 15:13:37 +00:00
Colin Walters
8c4ffc3340 unpacker,core: Handle selinux-policy-targeted at unpack time
Previously in 4a86bdd196 "postprocess: Undo Fedora SELinux policy move"
we fixed things for SELinux policy to move it out of /var and back
into /etc.

But for unified core where we're importing the RPM, the default "/var →
tmpfiles.d" kicks in before this can run. So teach the unpacker to do it too.
This is even less nice than what we had in the postprocessing code, as there we
went to some effort to handle non-targeted policy (e.g. MLS). That said I am not
super concerned about that right now...we can always deal with MLS or other
policies later. I'd be more inclined really to try to take the issue of moving
back to /etc (or implementing /usr with /etc overrides) to SELinux upstream.

Closes: #1105
Approved by: jlebon
2017-11-16 15:13:37 +00:00
Colin Walters
775e6f87ae core: (refactor) Load pkgcache commit object earlier
For unified core work, we'll be caching the "has docs" flag in the pkgcache
commit. So let's load the commit object earlier rather than having each function
load it.

Closes: #1104
Approved by: jlebon
2017-11-15 23:27:06 +00:00
Colin Walters
2e5848c5f4 core: (refactor) Style cleanup for pkg cache state load
Avoid the `goto` by initializing the output variables directly early, so we can
do a full new style consistently. Prep for unified core work.

Closes: #1104
Approved by: jlebon
2017-11-15 23:27:06 +00:00
Colin Walters
b17af329bb passwd: Always break hardlinks for shadow,gshadow, handle no merge
Prep for unified core 🌐.  First, right now `shadow-utils` always opens those with
`O_RDWR`, so let's break those links even if we don't have `usr/lib/passwd`.

Also, while not strictly requried, let's also handle the `merge_passwd_dir`
being `NULL`, as that happened when I was trying a variant of this patch.

Closes: #1098
Approved by: jlebon
2017-11-14 16:35:28 +00:00
Colin Walters
78bd4d6c64 core: Always process usr/etc/passwd
Prep for unified core 🌐. Here we don't have `/usr/lib/passwd` at the start, but
we still want to load the `usr/etc` copies for processing rpmfi overrides.

Closes: #1098
Approved by: jlebon
2017-11-14 16:35:28 +00:00
Colin Walters
5735e23a0f core: Use ADD_FILES checkout mode for "setup"
Prep for unified core 🌐.  On the treecompose path we inject
`/usr/etc/passwd`, and trying to overlay the setup package on top
with the default `UNION_IDENTICAL` fails.

This is obviously a hack; but I can't think of a way to do this significantly
better. (We could do a DB query for the owner of `/etc/passwd`, but eh.)

Closes: #1098
Approved by: jlebon
2017-11-14 16:35:28 +00:00
Jonathan Lebon
00fab1ec23 compose CLI: fix --repo consuming two arguments
I was trying to figure out why:

    rpm-ostree compose tree --repo repo/ manifest.json

would result in:

    error: opendir(manifest.json): No such file or directory

It turned out to be because we had `--repo` in *both* the `install`
options and the `commit` options. This makes sense since both these
subcommands need to be given a repo. However, in the `tree` case, we
were adding both arrays, which meant we inherited two `GOptionEntry`s
for `--repo`. This confused glib2 which consumed not one but two CLI
arguments when looking for the argument associated with `--repo`.

Our CI didn't notice this because it uses the `--repo=foo` notation,
which doesn't throw off glib2.

Fix this by factoring out the `--repo` option into a separate array so
that in the `tree` case, it only gets added once. Exercise the fix in CI
by using the two argument notation for `--repo`.

Closes: #1101
Approved by: cgwalters
2017-11-13 14:13:27 +00:00
Jonathan Lebon
bace3168ec build: allow git describe versioning in worktrees
In git worktrees, `.git` is not a directory but a regular file
containing the path to the real git directory. This was causing us to
not use `git describe` in the versioning info, which is really useful to
sanity check that you're running the code you think you're running. Fix
this by just checking that a `.git` entry exists.

Closes: #1102
Approved by: cgwalters
2017-11-13 13:48:58 +00:00
Colin Walters
d637bedf3e passwd: Don't use alloca() in a loop
Even if it's a very small loop.

Closes: #1088
Approved by: jlebon
2017-11-10 20:04:16 +00:00
Colin Walters
5a762dd458 lib: Hoist "break single hardlink" API to util.c, use in passwd
The passwd code can reuse the "break hardlink" function we had in the core; lift
it to `util.c`.

Closes: #1088
Approved by: jlebon
2017-11-10 20:04:16 +00:00
Colin Walters
bc49ec2e1c lib/db: Port to new style
No functional changes, and not prep for anything. Was just reading this code
during my jigdo work.

Closes: #1100
Approved by: jlebon
2017-11-10 18:03:14 +00:00
Colin Walters
07969136f2 lib/scripts: Only log to journal if stdout is already the journal
The jigdo work is turning into "partial unified core 🌐" (mostly due to SELinux,
but that's an aside); we made a change here previously to avoid using
the journal if `uid != 0`, but it's more correct to check whether or
not we're already using the journal.

Concretely this fixes calling `rpmostree_context_assemble_tmprootfs()` as uid 0
inside my dev container, running `rpm-ostree compose commit2jigdo`.

Closes: #1084
Approved by: jlebon
2017-11-09 03:36:23 +00:00
Colin Walters
516588d4f7 core: Use tmpdir in ${repo}/tmp for SELinux relabeling
Needed for unified core work for the treecompose side; there
`/var` may not actually be on the same mount as the repo.

Do what we do in a ton of other places (and should probably push
into ostree somehow): allocate a tmpdir in `${repo}/tmp`.

Closes: #1095
Approved by: jlebon
2017-11-09 03:15:26 +00:00
Colin Walters
b3267eb336 scripts: Ignore nfs-utils.post
The maintainer can't apparently be bothered to review my patch,
and we don't need the `%post` anyways - and this is now blocking
my jigdo work.

Closes: #1092
Approved by: jlebon
2017-11-08 16:00:20 +00:00
Colin Walters
94a1776fe7 unpacker: Error out when trying to install RPM in /usr/local
This came up with `https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-license-9-0-9.0.176-1.x86_64.rpm`.
Reported by @dustymabe on IRC.

Closes: #1090
Approved by: jlebon
2017-11-08 03:56:20 +00:00
Colin Walters
407bfd5bc1 Lift 'override' out of experimental
We have a *lot* of experimental functionality.  I think the
`override` bits are fleshed out enough now that we can lift
the `ex` designation.  For example, jlebon fixed SELinux
labeling in the presence of override-replace.

Closes: #1089
Approved by: jlebon
2017-11-08 03:35:08 +00:00
Colin Walters
7ea2a5afce (refactor) Move pkgcache helper function to sysroot-core
Trying to drain the `util.c` file; this one lives more logically
there.

Closes: #1086
Approved by: jlebon
2017-11-08 03:11:36 +00:00
Colin Walters
8b4c5e0901 tree-wide: Use glnx_close_fd()
It's cleaner, and we get `EBADF` checks, etc.

Closes: #1085
Approved by: jlebon
2017-11-08 02:50:12 +00:00
Colin Walters
a42c45f726 lib/core: Synthesize tmpdir for commit if no base provided
The `ex container` path was already doing this - lower the logic
into the core so I can use it for the jigdo path.  I think it's just
clearer as well even for the sysroot pkglayering path.

Closes: #1083
Approved by: jlebon
2017-11-08 02:12:52 +00:00
Colin Walters
170f74c282 lib/core: Add API to set devino cache
Rather than passing it to the assemble functions.

Prep for jigdo work.  I'm trying to move more logic
from the system upgrader path to the core, since some of
it is needed for jigdo and "unified core".

Closes: #1083
Approved by: jlebon
2017-11-08 02:12:52 +00:00
Colin Walters
fe0f268bcc lib/postprocess: Create OstreeSePolicy more directly
We already called the `workaround_selinux_cross_labeling_recurse()`
in the postprocessing path, there's no need to do it again during
commit.

Just making this change as I was going to do some SePolicy stuff
for the [jigdo work](https://github.com/projectatomic/rpm-ostree/issues/1081)
and stumbled again into the ugly mess that is the cross-labeling
hack.

Closes: #1082
Approved by: jlebon
2017-11-08 02:02:43 +00:00
Colin Walters
1857a6a1ef ex-unpack: Open OstreeSePolicy more directly
We should definitely not be calling the "postprocess" version here
as that may *change* the source filesystem.

Closes: #1082
Approved by: jlebon
2017-11-08 02:02:43 +00:00
Ruixin Bao
5ca9c731d7 tests/check: Add unit tests for kargs options
Added unit tests for rpm-ostree ex kargs --delete,
--append and --replace.

Also exposed two getter functions for kargs table
and array so people can retrieve information from
kargs.

Also includes a minor fix for a bug caught by the unit
test.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
078d82e8d0 tests/vmcheck: add deploy-index and proc-cmd test for kargs
Refined a bit for the previous written kargs test.( mainly for
checking kargs after rollback )

Added tests for import-proc-cmdline and deploy-index option

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
27de772c4f libpriv/vmcheck: support replace option for karg command
Refactor the similar code portion from delete option into
a new function, and reuse it in the replace logic.

Add the replace functionality to allow kernel arguments
replacement. Now support two different types of replacement:

1: replace by key=value when there is only one key existing
e.g we have 'hi=hello' as our one of the arguments (hi only has
one value associated with it) , and 'rpm-ostree ex kargs --replace hi=new'
will replace the value and change hi=hello into hi=new.

2: replace by key=value=new_value for all other replacements
( this will work for swapping single value pair too !)

Some tests for rpm-ostree ex kargs --replace are added
for future regression.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
09cf438f04 vmcheck: Add tests for rpm-ostree ex kargs delete and append
Tests are introduced in this commit to test the basic
functionality for rpm-ostree ex kargs --append,
and rpm-ostree ex kargs --delete.

Those tests are added for future regression.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
df3b5c0c4e daemon/libpriv: add handlation for kargs delete option
Some new functions are added to handle delete operations
for kargs. We are now able to do the following delete operations kargs:

1: delete by key, if there is only one value associated with the key
we directly remove it
2: delete by key=value, it will find that specific key/value pair
and remove it from the kernel arguments

Similar to append functionality introduced, delete will
also create a new pending deployment that is rollbackable

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
50863c370d daemon: add handlation for kargs append
API functions from ostree-kernel-args.c
are copied to libpriv. The append functionality
reuses  _ostree_kernel_args_append_argv() for
collecting added kernel arguments.

Also added handlation in rpm-ostree upgrader
to allow deployments happen with kernel arguments.

Now, the user is able to add kernel arguments via
'rpm-ostree ex kargs --append key=value'
or 'rpm-ostree ex kargs --append key' if they
want to have an empty value with key.

The user is also able to display the current
kernel arguments via 'rpm-ostree ex kargs'

In addition, this functionality will create a pending deployment,
and will update the conf file in /boot/loader/entries/ostree-$osname-0.conf upon success.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
fa593afd89 app/daemon: add groundwork for rpm-ostree kargs command
This is initial groundwork for https://github.com/projectatomic/rpm-ostree/issues/594.

This commit sets up most of the required
front end logic( arg parsing, transaction handling), and will
be used in the following commits.

There is nothing really fancy in this commit, as most of the code
shares the similar style between other dbus related commands.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Colin Walters
e7a42f70a9 compose: Add rpmostree.rpmmd-repo metadata to commits by default
This is a revisit of a PR for client-side layering: https://github.com/projectatomic/rpm-ostree/pull/1072
Here though we're doing this by default for server-side composes.
There are a few reasons to do this; first, I'm seeing an issue
in some of our Jenkins jobs for Fedora that hit "mirror roulette"
and end up creating commits that "revert" to older versions temporarily.

While I've [certainly pitched](https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/IMPE6KCRBHCEJH5VBE6ZFIRLPAD743JT/) this as a feature, I think
we really want something like `--force-older-timestamp` - basically
error out if the timestamps on one or more input repos were older.
Not doing that in this patch, but it paves the way to do so.

Second, I'd like to use this data in the `ostree.source-title`
metadata key down the line.  Something like:

`└ rpmmd: fedora-26 (20170310), fedora-26-updates (20171101)`

(This could be a lot nicer if we drive versioning in to the rpm-md repo info,
 and e.g. there's some friendly "week number" style versioning for the updates
 repo now that it's batched...for now we have timestamps)

For CentOS/RHELAH this gets interesting and potentially more verbose,
to the point where we may want to render it more explicitly.

But anyways, let's do this now, as it will be useful even without
an explicit rendering, since users can do e.g. `ostree show` on
a base commit hash to dump the data.

I had a concern that some users may not want to emit this metadata;
they can currently do `--add-metadata-string rpmostree.rpmmd-repos ''`
and that will "win".

Closes: #1079
Approved by: jlebon
2017-11-04 21:02:56 +00:00
Colin Walters
cb86194e2c Release v2017.10
Now that there's a new libostree out, let's cut one here too, since we have a
fair bit accumulated, and I think it'd be good to have the oustanding kargs PR
to soak in master for a bit after landing.

Closes: #1080
Approved by: jlebon
2017-11-02 18:01:58 +00:00
Colin Walters
d6218e0e16 postprocess: Use names (not ids) in synthesized tmpfiles.d files
Related to: https://github.com/projectatomic/rpm-ostree/issues/49

We want to support "name binding" per client system, rather than
having a hardcoded mapping in our tree.  Currently if e.g. a new
daemon is added as a dependency (or as part of e.g. systemd) it's
easy to silently miss it.

This is prep for doing that binding client side consistently, which is what we
do with package layering.

Closes: #1077
Approved by: jlebon
2017-10-27 19:46:26 +00:00
Colin Walters
669031f63b bin/compose: Print correct final rootfs path
This was a thinko; `workdir_tmp` won't be set in this path. I think the code
changed here at some point.

Closes: #1076
Approved by: jlebon
2017-10-26 21:20:40 +00:00