Commit Graph

1964 Commits

Author SHA1 Message Date
Jonathan Lebon
bace3168ec build: allow git describe versioning in worktrees
In git worktrees, `.git` is not a directory but a regular file
containing the path to the real git directory. This was causing us to
not use `git describe` in the versioning info, which is really useful to
sanity check that you're running the code you think you're running. Fix
this by just checking that a `.git` entry exists.

Closes: #1102
Approved by: cgwalters
2017-11-13 13:48:58 +00:00
Colin Walters
d637bedf3e passwd: Don't use alloca() in a loop
Even if it's a very small loop.

Closes: #1088
Approved by: jlebon
2017-11-10 20:04:16 +00:00
Colin Walters
5a762dd458 lib: Hoist "break single hardlink" API to util.c, use in passwd
The passwd code can reuse the "break hardlink" function we had in the core; lift
it to `util.c`.

Closes: #1088
Approved by: jlebon
2017-11-10 20:04:16 +00:00
Colin Walters
bc49ec2e1c lib/db: Port to new style
No functional changes, and not prep for anything. Was just reading this code
during my jigdo work.

Closes: #1100
Approved by: jlebon
2017-11-10 18:03:14 +00:00
Colin Walters
07969136f2 lib/scripts: Only log to journal if stdout is already the journal
The jigdo work is turning into "partial unified core 🌐" (mostly due to SELinux,
but that's an aside); we made a change here previously to avoid using
the journal if `uid != 0`, but it's more correct to check whether or
not we're already using the journal.

Concretely this fixes calling `rpmostree_context_assemble_tmprootfs()` as uid 0
inside my dev container, running `rpm-ostree compose commit2jigdo`.

Closes: #1084
Approved by: jlebon
2017-11-09 03:36:23 +00:00
Colin Walters
516588d4f7 core: Use tmpdir in ${repo}/tmp for SELinux relabeling
Needed for unified core work for the treecompose side; there
`/var` may not actually be on the same mount as the repo.

Do what we do in a ton of other places (and should probably push
into ostree somehow): allocate a tmpdir in `${repo}/tmp`.

Closes: #1095
Approved by: jlebon
2017-11-09 03:15:26 +00:00
Colin Walters
b3267eb336 scripts: Ignore nfs-utils.post
The maintainer can't apparently be bothered to review my patch,
and we don't need the `%post` anyways - and this is now blocking
my jigdo work.

Closes: #1092
Approved by: jlebon
2017-11-08 16:00:20 +00:00
Colin Walters
94a1776fe7 unpacker: Error out when trying to install RPM in /usr/local
This came up with `https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-license-9-0-9.0.176-1.x86_64.rpm`.
Reported by @dustymabe on IRC.

Closes: #1090
Approved by: jlebon
2017-11-08 03:56:20 +00:00
Colin Walters
407bfd5bc1 Lift 'override' out of experimental
We have a *lot* of experimental functionality.  I think the
`override` bits are fleshed out enough now that we can lift
the `ex` designation.  For example, jlebon fixed SELinux
labeling in the presence of override-replace.

Closes: #1089
Approved by: jlebon
2017-11-08 03:35:08 +00:00
Colin Walters
7ea2a5afce (refactor) Move pkgcache helper function to sysroot-core
Trying to drain the `util.c` file; this one lives more logically
there.

Closes: #1086
Approved by: jlebon
2017-11-08 03:11:36 +00:00
Colin Walters
8b4c5e0901 tree-wide: Use glnx_close_fd()
It's cleaner, and we get `EBADF` checks, etc.

Closes: #1085
Approved by: jlebon
2017-11-08 02:50:12 +00:00
Colin Walters
a42c45f726 lib/core: Synthesize tmpdir for commit if no base provided
The `ex container` path was already doing this - lower the logic
into the core so I can use it for the jigdo path.  I think it's just
clearer as well even for the sysroot pkglayering path.

Closes: #1083
Approved by: jlebon
2017-11-08 02:12:52 +00:00
Colin Walters
170f74c282 lib/core: Add API to set devino cache
Rather than passing it to the assemble functions.

Prep for jigdo work.  I'm trying to move more logic
from the system upgrader path to the core, since some of
it is needed for jigdo and "unified core".

Closes: #1083
Approved by: jlebon
2017-11-08 02:12:52 +00:00
Colin Walters
fe0f268bcc lib/postprocess: Create OstreeSePolicy more directly
We already called the `workaround_selinux_cross_labeling_recurse()`
in the postprocessing path, there's no need to do it again during
commit.

Just making this change as I was going to do some SePolicy stuff
for the [jigdo work](https://github.com/projectatomic/rpm-ostree/issues/1081)
and stumbled again into the ugly mess that is the cross-labeling
hack.

Closes: #1082
Approved by: jlebon
2017-11-08 02:02:43 +00:00
Colin Walters
1857a6a1ef ex-unpack: Open OstreeSePolicy more directly
We should definitely not be calling the "postprocess" version here
as that may *change* the source filesystem.

Closes: #1082
Approved by: jlebon
2017-11-08 02:02:43 +00:00
Ruixin Bao
5ca9c731d7 tests/check: Add unit tests for kargs options
Added unit tests for rpm-ostree ex kargs --delete,
--append and --replace.

Also exposed two getter functions for kargs table
and array so people can retrieve information from
kargs.

Also includes a minor fix for a bug caught by the unit
test.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
078d82e8d0 tests/vmcheck: add deploy-index and proc-cmd test for kargs
Refined a bit for the previous written kargs test.( mainly for
checking kargs after rollback )

Added tests for import-proc-cmdline and deploy-index option

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
27de772c4f libpriv/vmcheck: support replace option for karg command
Refactor the similar code portion from delete option into
a new function, and reuse it in the replace logic.

Add the replace functionality to allow kernel arguments
replacement. Now support two different types of replacement:

1: replace by key=value when there is only one key existing
e.g we have 'hi=hello' as our one of the arguments (hi only has
one value associated with it) , and 'rpm-ostree ex kargs --replace hi=new'
will replace the value and change hi=hello into hi=new.

2: replace by key=value=new_value for all other replacements
( this will work for swapping single value pair too !)

Some tests for rpm-ostree ex kargs --replace are added
for future regression.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
09cf438f04 vmcheck: Add tests for rpm-ostree ex kargs delete and append
Tests are introduced in this commit to test the basic
functionality for rpm-ostree ex kargs --append,
and rpm-ostree ex kargs --delete.

Those tests are added for future regression.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
df3b5c0c4e daemon/libpriv: add handlation for kargs delete option
Some new functions are added to handle delete operations
for kargs. We are now able to do the following delete operations kargs:

1: delete by key, if there is only one value associated with the key
we directly remove it
2: delete by key=value, it will find that specific key/value pair
and remove it from the kernel arguments

Similar to append functionality introduced, delete will
also create a new pending deployment that is rollbackable

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
50863c370d daemon: add handlation for kargs append
API functions from ostree-kernel-args.c
are copied to libpriv. The append functionality
reuses  _ostree_kernel_args_append_argv() for
collecting added kernel arguments.

Also added handlation in rpm-ostree upgrader
to allow deployments happen with kernel arguments.

Now, the user is able to add kernel arguments via
'rpm-ostree ex kargs --append key=value'
or 'rpm-ostree ex kargs --append key' if they
want to have an empty value with key.

The user is also able to display the current
kernel arguments via 'rpm-ostree ex kargs'

In addition, this functionality will create a pending deployment,
and will update the conf file in /boot/loader/entries/ostree-$osname-0.conf upon success.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Ruixin Bao
fa593afd89 app/daemon: add groundwork for rpm-ostree kargs command
This is initial groundwork for https://github.com/projectatomic/rpm-ostree/issues/594.

This commit sets up most of the required
front end logic( arg parsing, transaction handling), and will
be used in the following commits.

There is nothing really fancy in this commit, as most of the code
shares the similar style between other dbus related commands.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Colin Walters
e7a42f70a9 compose: Add rpmostree.rpmmd-repo metadata to commits by default
This is a revisit of a PR for client-side layering: https://github.com/projectatomic/rpm-ostree/pull/1072
Here though we're doing this by default for server-side composes.
There are a few reasons to do this; first, I'm seeing an issue
in some of our Jenkins jobs for Fedora that hit "mirror roulette"
and end up creating commits that "revert" to older versions temporarily.

While I've [certainly pitched](https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/IMPE6KCRBHCEJH5VBE6ZFIRLPAD743JT/) this as a feature, I think
we really want something like `--force-older-timestamp` - basically
error out if the timestamps on one or more input repos were older.
Not doing that in this patch, but it paves the way to do so.

Second, I'd like to use this data in the `ostree.source-title`
metadata key down the line.  Something like:

`└ rpmmd: fedora-26 (20170310), fedora-26-updates (20171101)`

(This could be a lot nicer if we drive versioning in to the rpm-md repo info,
 and e.g. there's some friendly "week number" style versioning for the updates
 repo now that it's batched...for now we have timestamps)

For CentOS/RHELAH this gets interesting and potentially more verbose,
to the point where we may want to render it more explicitly.

But anyways, let's do this now, as it will be useful even without
an explicit rendering, since users can do e.g. `ostree show` on
a base commit hash to dump the data.

I had a concern that some users may not want to emit this metadata;
they can currently do `--add-metadata-string rpmostree.rpmmd-repos ''`
and that will "win".

Closes: #1079
Approved by: jlebon
2017-11-04 21:02:56 +00:00
Colin Walters
cb86194e2c Release v2017.10
Now that there's a new libostree out, let's cut one here too, since we have a
fair bit accumulated, and I think it'd be good to have the oustanding kargs PR
to soak in master for a bit after landing.

Closes: #1080
Approved by: jlebon
2017-11-02 18:01:58 +00:00
Colin Walters
d6218e0e16 postprocess: Use names (not ids) in synthesized tmpfiles.d files
Related to: https://github.com/projectatomic/rpm-ostree/issues/49

We want to support "name binding" per client system, rather than
having a hardcoded mapping in our tree.  Currently if e.g. a new
daemon is added as a dependency (or as part of e.g. systemd) it's
easy to silently miss it.

This is prep for doing that binding client side consistently, which is what we
do with package layering.

Closes: #1077
Approved by: jlebon
2017-10-27 19:46:26 +00:00
Colin Walters
669031f63b bin/compose: Print correct final rootfs path
This was a thinko; `workdir_tmp` won't be set in this path. I think the code
changed here at some point.

Closes: #1076
Approved by: jlebon
2017-10-26 21:20:40 +00:00
Colin Walters
964ab1f8bc bin/compose: Do passwd checking in commit, not install
We won't have done the postprocessing, so `/usr/lib/passwd` won't exist. Trying
to use `compose install` with current fedora-atomic failed (I *really* should
have tested that at least manually with the final patchset). Add `check-passwd`
to the test suite so this gets coverage too.

Closes: #1076
Approved by: jlebon
2017-10-26 21:20:40 +00:00
Colin Walters
2285f0f6db Inline assemble_commit() wrapper into ex-container, the only user
This is a cleanup I was going to do before, but had dropped. It's a general
followup to the `compose` rework; there's not much point to having a high level
wrapper, given that we generally need to do some postprocessing. Inline it into
the `ex container` code, which makes the core more "core".

Closes: #1074
Approved by: jlebon
2017-10-26 15:44:58 +00:00
Colin Walters
59fec100bf bin/compose: Fix use of EXIT_FAILURE for error paths
I got a critical from an nonexistent `DESTDIR` in `rpm-ostree compose install`.

(I'm probably going to do a patch to add `rpmostree_command_invocation_set_exit_code()`
 so the "exit 77" case can use that and everything else can `s/return EXIT_FAILURE/return FALSE/`
 and we'll be free of these bugs)

Closes: #1075
Approved by: jlebon
2017-10-26 14:24:47 +00:00
Colin Walters
cb9889204c lib/unpacker: Add repo.timestamp to metadata
Related to: https://github.com/projectatomic/rpm-ostree/issues/774

We aren't yet trying to render this in any way on the client side, but let's
start capturing the timestamp data now so that we can make use of it later
in e.g. `rpm-ostree status -v`.

Closes: #1072
Approved by: jlebon
2017-10-25 18:18:10 +00:00
Colin Walters
63af4bbdda bin/compose: Expose phases as [install, postprocess, commit] cmds
Right now `rpm-ostree compose tree` is very prescriptive about how things work.
Trying to add anything that isn't an RPM is absolutely fighting the system. Our
postprocessing system *enforces* no network access (good for reproducibilty, but
still prescriptive).

There's really a logical split between three phases:

 - install: "build a rootfs that installs packages"
 - postprocess: "run magical ostree postprocessing like kernel"
 - commit: "commit result to ostree"

So there are two high level flows I'd like to enable here. First is to allow
people to do *arbitrary* postprocessing between `install` and `commit`. For
example, run Ansible and change `/etc`. This path basically is like what we have
today with `postprocess-script.sh`, except the builder can do anything they want
with network access enabled.

Going much farther, this helps us support a "build with Dockerfile" style flow.
We can then provide tooling to extract the container image, and combine
`postprocess` and `commit`.

Or completely the other way - if for example someone wants to use `rpm-ostree
compose install`, they could tar up the result as a Docker/OCI image. That's now
easier; an advantage of this flow over e.g. `yum --installroot` is the "change
detection" code we have.

Related issues/PRs:

 - https://github.com/projectatomic/rpm-ostree/pull/96
 - https://github.com/projectatomic/rpm-ostree/issues/471

One disadvantage of this approach right now is that if one *does* go for
the split approach, we lose the "input hash" metadata for example.  And
down the line, I'd like to add even more metadata, like the input rpm repos,
which could also be rendered on the client side.

But, I think we can address that later by e.g. caching the metadata in a file in
the install root and picking it back up or something.

Closes: #1039
Approved by: jlebon
2017-10-25 17:43:09 +00:00
Colin Walters
1f31cbb99a status: Render ostree metadata source-title key
Depends: https://github.com/ostreedev/ostree/pull/1296

As I mention in the commit there, I see two uses for this in rpm-ostree; first
in our test suite, and second for OCI-built image imports.

I also took a step further here and inject an `original-origin` metadata
key, though we aren't actually using that yet.  The problem I'm trying
to solve there is that repeated `make vmoverlay` starts chaining things up,
but that gets very confusing.  I think we should always have `vmoverlay` unwind
back to the base ref.  (Or at least do that by default)

Closes: #1069
Approved by: jlebon
2017-10-25 03:37:05 +00:00
Colin Walters
9ec45d3c89 daemon: Add commit metadata to deployment properties
In another PR I did the manual bridging of commit metadata to deployment
property, but that's annoying. Let's just bridge all commit metadata.

Closes: #1069
Approved by: jlebon
2017-10-25 03:37:05 +00:00
Colin Walters
cdfba8e131 ci: Update CentOS container to current 7
The rpmmd repo issues seem to be fixed, and the old container was (for some
reason) apparently removed.

Closes: #1073
Approved by: jlebon
2017-10-24 16:26:32 +00:00
Colin Walters
aaf0d978b8 postprocess: Also move RPM database if not done already
Prep for split compose.  The current Docker/OCI base images we
built at least for Fedora don't have this move done, so let's
ensure our postprocessing does it.

Closes: #1070
Approved by: jlebon
2017-10-23 20:35:41 +00:00
Colin Walters
21167bea72 postprocess: Deal with an already empty /etc/machine-id
Prep for split compose. The Fedora base container image already empties
`/etc/machine-id`; let's handle that case rather than throwing an error. This
makes it easier to run our postprocessing on an image built on top of that base
image.

Closes: #1070
Approved by: jlebon
2017-10-23 20:35:41 +00:00
Colin Walters
d02bc4b03c compose: Accept NULL treefile for "use defaults" postprocessing
This is prep for split-compose.  We have some options in the
treefile, like `boot_location` and `tmp-is-dir` etc.  While those
are useful options, I don't want to force everyone using
`rpm-ostree compose postprocess` to write a treefile.

Change the code then to accept a `NULL` treefile to mean
"use the defaults".

Closes: #1070
Approved by: jlebon
2017-10-23 20:35:41 +00:00
Colin Walters
5da2894823 ex-container: Always use canonical permissions for commit
Fixes a fsck issue that I saw while submitting another patch.

This is the second time in the last week I've fixed an issue like this; will
probably do a libostree patch to just turn it on by default for
`bare-user-only`, but let's fix this bug anyways.

Closes: #1071
Approved by: peterbaouoft
2017-10-23 14:12:10 +00:00
Colin Walters
a9c8b1fae1 ex-container: Make /etc/shadow 0400 on import, not post-checkout
Switching to the `_CONSUME` flag revealed an "oh god how did I write that"
bug in the previous patch in https://github.com/projectatomic/rpm-ostree/pull/1046
AKA commit: 334f0b89be

The way that actually fixed the bug before was because we were using
hardlink checkouts, and we were operating outside an `rofiles-fuse`
context, we simply directly changed the on-disk object mode.

But with the `_CONSUME` flag we started deleting the files as we write,
meaning that stopped working.

I *initially* wrote a patch to do the same split "prepare/processing/commit"
flow that treecompose and package layering do, but that can't really fix this
bug - we need to do it on import.

So do the chmod on import and drop the postprocessing bits.

Closes: #1067
Approved by: jlebon
2017-10-18 17:07:54 +00:00
Colin Walters
d60cc0248a ex-container: Use shared helper for download+import+rootfs+commit
Prep for fixing a bug in the last (git) commit here for `/etc/shadow`.

Closes: #1067
Approved by: jlebon
2017-10-18 17:07:54 +00:00
Jonathan Lebon
c107a05b8e tree: pass NULL to glnx_fstatat_allow_noent when needed
Now that libglnx allows it, we can skip declaring a `struct stat` we
don't actually need just to check if a file exists.

Closes: #1066
Approved by: cgwalters
2017-10-17 20:48:13 +00:00
Colin Walters
3be6dbae15 Make use of new _CONSUME flag to ostree commit
This is used for both `ex container` and client-side operations, but not most
uses of `compose tree` (which should go into `bare-user` or `archive`).
We'll circle back and do `bare-user` for `_CONSUME` in libostree though
soon.

Closes: #1065
Approved by: jlebon
2017-10-17 18:49:11 +00:00
Colin Walters
c3a5aef80b bin/compose: Clean up ostree version metadata handling a bit
Just noticed this FIXME while working on split compose.  Use
the `#define` we have, and inline the duplicated function which
boils down to `g_variant_lookup()` - I simply changed `&s` to `s`
to inline the `strdup`.

Closes: #1064
Approved by: jlebon
2017-10-17 18:44:31 +00:00
Colin Walters
94527ab69c tree-wide: Use glnx_autofd
It's a better name.  Already ported ostree.

Closes: #1063
Approved by: jlebon
2017-10-17 16:01:34 +00:00
Colin Walters
583c626821 bin/compose: Hide no-op --workdir-tmpfs
We made it a no-op a bit ago in commit 162b8d4898
Let's hide it like we did for the `status --pretty` option.

Closes: #1061
Approved by: jlebon
2017-10-16 20:16:36 +00:00
Jonathan Lebon
4d1b5b4636 libpriv/core: use cached repodata checksum for goal
An interesting behaviour arises from the new `--cache-only` option. We
would essentially consider two goals with the same exact pkglist as
different depending on whether a package came from a yum repo or our
pkgcache repo. This meant that `rpm-ostree upgrade` followed by
`rpm-ostree upgrade -C` was never detected as a no-op and always
resulted in a relayer.

This was due to the fact that the checksum libsolv calculated was based
on our chopped off pkgcache meta-RPMs. This of course won't match the
checksum from the repodata, which is of the whole RPM.

Thankfully, we already save the repodata checksum when unpacking RPMs.
This patch simply detects the case where we're trying to checksum a pkg
originating from the pkgcache, and reads in the saved repodata checksum
rather than the wrong libsolv-derived one.

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
e49f7cdd81 app: add --download-only option
To complement the new `--cache-only` option, add a `--download-only`
option. This does exactly what it says: we download the ostree, download
and import packages, but don't actually commit & deploy. This can be
used to effectively prime a follow-up `--cache-only` operation that can
be done during a more convenient/safer maintenance window.

I debated naming the two options `--pull-only` and `--deploy-only` like
the ostree equivalents. Though "pull" felt like the wrong word given
that it's associated more with ostree pulling but rpm-ostree also
downloads & imports RPMs. As for `--deploy-only` vs `--cache-only`, it
seems like `--cache-only` is a more accurate description of the
functionality (i.e. rather than describing an action, it describes a
mode). I also considered `--no-download` to make the synergy with
`--download-only` more obvious. Maybe that's better? Naming is hard...

Closes: #713

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
24c184af03 app: imply cache-only mode for pkgs removals and uninstalls
Now that we have a strong notion of `cache-only` mode, make use of it
when performing an `uninstall` or `ex override remove/reset`.

Closes: #944

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
b811eb61c0 app: support full offline operations with --cache-only
As Colin mentioned in #1035, the new `--cache-only` implemented only the
rpmmd half of the story. Here we complete that story by also ensuring
that when in cache-only mode, we don't download new ostree data nor new
packages. We try to complete the requested operation with what we have.

To do this, we add support for the same `SYNTHETIC` pull that was added
in ostree[1] so that we don't actually pull, but still perform timestamp
checking.

On the pkgcache side, we disable all remote repos and instead insert all
our cached RPMs into the `DnfSack`. Care is taken to still perform
SHA256 verification for local pkg installs/replacements.

[1] https://github.com/ostreedev/ostree/pull/642

Closes: #687

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
cd3da57453 app/status: make deprecated --pretty flag hidden
Minor detail. We still need to support the `--pretty` flag, but it
doesn't do anything anymore, so let's just hide it to clean up the help
output.

Closes: #1060
Approved by: cgwalters
2017-10-16 16:46:02 +00:00