Commit Graph

1732 Commits

Author SHA1 Message Date
Colin Walters
9ec45d3c89 daemon: Add commit metadata to deployment properties
In another PR I did the manual bridging of commit metadata to deployment
property, but that's annoying. Let's just bridge all commit metadata.

Closes: #1069
Approved by: jlebon
2017-10-25 03:37:05 +00:00
Colin Walters
cdfba8e131 ci: Update CentOS container to current 7
The rpmmd repo issues seem to be fixed, and the old container was (for some
reason) apparently removed.

Closes: #1073
Approved by: jlebon
2017-10-24 16:26:32 +00:00
Colin Walters
aaf0d978b8 postprocess: Also move RPM database if not done already
Prep for split compose.  The current Docker/OCI base images we
built at least for Fedora don't have this move done, so let's
ensure our postprocessing does it.

Closes: #1070
Approved by: jlebon
2017-10-23 20:35:41 +00:00
Colin Walters
21167bea72 postprocess: Deal with an already empty /etc/machine-id
Prep for split compose. The Fedora base container image already empties
`/etc/machine-id`; let's handle that case rather than throwing an error. This
makes it easier to run our postprocessing on an image built on top of that base
image.

Closes: #1070
Approved by: jlebon
2017-10-23 20:35:41 +00:00
Colin Walters
d02bc4b03c compose: Accept NULL treefile for "use defaults" postprocessing
This is prep for split-compose.  We have some options in the
treefile, like `boot_location` and `tmp-is-dir` etc.  While those
are useful options, I don't want to force everyone using
`rpm-ostree compose postprocess` to write a treefile.

Change the code then to accept a `NULL` treefile to mean
"use the defaults".

Closes: #1070
Approved by: jlebon
2017-10-23 20:35:41 +00:00
Colin Walters
5da2894823 ex-container: Always use canonical permissions for commit
Fixes a fsck issue that I saw while submitting another patch.

This is the second time in the last week I've fixed an issue like this; will
probably do a libostree patch to just turn it on by default for
`bare-user-only`, but let's fix this bug anyways.

Closes: #1071
Approved by: peterbaouoft
2017-10-23 14:12:10 +00:00
Colin Walters
a9c8b1fae1 ex-container: Make /etc/shadow 0400 on import, not post-checkout
Switching to the `_CONSUME` flag revealed an "oh god how did I write that"
bug in the previous patch in https://github.com/projectatomic/rpm-ostree/pull/1046
AKA commit: 334f0b89be

The way that actually fixed the bug before was because we were using
hardlink checkouts, and we were operating outside an `rofiles-fuse`
context, we simply directly changed the on-disk object mode.

But with the `_CONSUME` flag we started deleting the files as we write,
meaning that stopped working.

I *initially* wrote a patch to do the same split "prepare/processing/commit"
flow that treecompose and package layering do, but that can't really fix this
bug - we need to do it on import.

So do the chmod on import and drop the postprocessing bits.

Closes: #1067
Approved by: jlebon
2017-10-18 17:07:54 +00:00
Colin Walters
d60cc0248a ex-container: Use shared helper for download+import+rootfs+commit
Prep for fixing a bug in the last (git) commit here for `/etc/shadow`.

Closes: #1067
Approved by: jlebon
2017-10-18 17:07:54 +00:00
Jonathan Lebon
c107a05b8e tree: pass NULL to glnx_fstatat_allow_noent when needed
Now that libglnx allows it, we can skip declaring a `struct stat` we
don't actually need just to check if a file exists.

Closes: #1066
Approved by: cgwalters
2017-10-17 20:48:13 +00:00
Colin Walters
3be6dbae15 Make use of new _CONSUME flag to ostree commit
This is used for both `ex container` and client-side operations, but not most
uses of `compose tree` (which should go into `bare-user` or `archive`).
We'll circle back and do `bare-user` for `_CONSUME` in libostree though
soon.

Closes: #1065
Approved by: jlebon
2017-10-17 18:49:11 +00:00
Colin Walters
c3a5aef80b bin/compose: Clean up ostree version metadata handling a bit
Just noticed this FIXME while working on split compose.  Use
the `#define` we have, and inline the duplicated function which
boils down to `g_variant_lookup()` - I simply changed `&s` to `s`
to inline the `strdup`.

Closes: #1064
Approved by: jlebon
2017-10-17 18:44:31 +00:00
Colin Walters
94527ab69c tree-wide: Use glnx_autofd
It's a better name.  Already ported ostree.

Closes: #1063
Approved by: jlebon
2017-10-17 16:01:34 +00:00
Colin Walters
583c626821 bin/compose: Hide no-op --workdir-tmpfs
We made it a no-op a bit ago in commit 162b8d4898
Let's hide it like we did for the `status --pretty` option.

Closes: #1061
Approved by: jlebon
2017-10-16 20:16:36 +00:00
Jonathan Lebon
4d1b5b4636 libpriv/core: use cached repodata checksum for goal
An interesting behaviour arises from the new `--cache-only` option. We
would essentially consider two goals with the same exact pkglist as
different depending on whether a package came from a yum repo or our
pkgcache repo. This meant that `rpm-ostree upgrade` followed by
`rpm-ostree upgrade -C` was never detected as a no-op and always
resulted in a relayer.

This was due to the fact that the checksum libsolv calculated was based
on our chopped off pkgcache meta-RPMs. This of course won't match the
checksum from the repodata, which is of the whole RPM.

Thankfully, we already save the repodata checksum when unpacking RPMs.
This patch simply detects the case where we're trying to checksum a pkg
originating from the pkgcache, and reads in the saved repodata checksum
rather than the wrong libsolv-derived one.

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
e49f7cdd81 app: add --download-only option
To complement the new `--cache-only` option, add a `--download-only`
option. This does exactly what it says: we download the ostree, download
and import packages, but don't actually commit & deploy. This can be
used to effectively prime a follow-up `--cache-only` operation that can
be done during a more convenient/safer maintenance window.

I debated naming the two options `--pull-only` and `--deploy-only` like
the ostree equivalents. Though "pull" felt like the wrong word given
that it's associated more with ostree pulling but rpm-ostree also
downloads & imports RPMs. As for `--deploy-only` vs `--cache-only`, it
seems like `--cache-only` is a more accurate description of the
functionality (i.e. rather than describing an action, it describes a
mode). I also considered `--no-download` to make the synergy with
`--download-only` more obvious. Maybe that's better? Naming is hard...

Closes: #713

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
24c184af03 app: imply cache-only mode for pkgs removals and uninstalls
Now that we have a strong notion of `cache-only` mode, make use of it
when performing an `uninstall` or `ex override remove/reset`.

Closes: #944

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
b811eb61c0 app: support full offline operations with --cache-only
As Colin mentioned in #1035, the new `--cache-only` implemented only the
rpmmd half of the story. Here we complete that story by also ensuring
that when in cache-only mode, we don't download new ostree data nor new
packages. We try to complete the requested operation with what we have.

To do this, we add support for the same `SYNTHETIC` pull that was added
in ostree[1] so that we don't actually pull, but still perform timestamp
checking.

On the pkgcache side, we disable all remote repos and instead insert all
our cached RPMs into the `DnfSack`. Care is taken to still perform
SHA256 verification for local pkg installs/replacements.

[1] https://github.com/ostreedev/ostree/pull/642

Closes: #687

Closes: #1049
Approved by: cgwalters
2017-10-16 17:49:51 +00:00
Jonathan Lebon
cd3da57453 app/status: make deprecated --pretty flag hidden
Minor detail. We still need to support the `--pretty` flag, but it
doesn't do anything anymore, so let's just hide it to clean up the help
output.

Closes: #1060
Approved by: cgwalters
2017-10-16 16:46:02 +00:00
Colin Walters
aea688fa01 Update libglnx
I want to use `glnx_tmpdir_unset()` in the `compose commit` PR.

Update submodule: libglnx

Closes: #1056
Approved by: jlebon
2017-10-16 13:22:30 +00:00
Colin Walters
854c7db029 scripts: Fix close() → EBADF
Caught by bumping libglnx and its more widespread checking for `EBADF`. We
should probably add a `glnx_fdopen()` that steals ownership of the fd.

Closes: #1056
Approved by: jlebon
2017-10-16 13:22:30 +00:00
Colin Walters
43d52e7196 postprocess: (refactor) Split out "init rootfs" from "final postprocess"
Prep for [compose split](https://github.com/projectatomic/rpm-ostree/pull/1039).
The next step here will be to basically have `postprocess_final()` + `ostree commit`
be `rpm-ostree compose commit`.

This reworks the order for a few things - we move `/usr` first right after
initializing the rootfs, which works now because from previous commits `/etc`
is really `/usr/etc`.

We also change things so that we grab `/boot` and `/var` from the src root,
then `postprocess_final()` handles them in place.

One detail here is that I had to do the `rpmostree_rootfs_prepare_links()`
after the `convert_var_to_tmpfiles_d()` since it writes into var; before
this worked because we were using different roots.

Closes: #1054
Approved by: jlebon
2017-10-13 15:29:11 +00:00
Jonathan Lebon
2b4a65bca5 libpriv/core: also checksum package action
When calculating the goal checksum, we only accounted for package
installs. But with override support, there are now other package actions
we need to pick up on.

In practice, the fact that we also checksum the treespec made this hard
to actually cause an issue. Although, since the actual final goal is
also dependent on the underlying rpmdb, I can imagine e.g. a replacement
override be considered a `DNF_PACKAGE_INFO_UPDATE` for one rpmdb appear
as a `DNF_PACKAGE_INFO_DOWNGRADE` for another.

While we're here, do some light code style porting and streamline the
checksumming process to avoid creating a separate `GPtrArray`.

Closes: #1053
Approved by: cgwalters
2017-10-13 15:00:25 +00:00
Jonathan Lebon
e5e9f1f0b8 libvm: factor out helpers to start httpd
This will be used in other tests. Plus, it makes it much nicer to use.

Closes: #1053
Approved by: cgwalters
2017-10-13 15:00:25 +00:00
Jonathan Lebon
dbceb2aa51 daemon/deploy: remove duplicate call to set_origin
We already set the origin unconditionally right after this block. No
point in `dup`'ing twice.

Closes: #1053
Approved by: cgwalters
2017-10-13 15:00:25 +00:00
Jonathan Lebon
24cff4f5e0 app/uninstall: remove --cache-only option
The `--cache-only` option was added to the generic `option_entries`
array, which is shared by both `install` and `uninstall`. But really, it
only makes sense for `install`. Move it to the `install` specific array
instead.

I also renamed the option entry arrays to make more sense. Before, the
`install_option_entries` referred to the additional options that the
`uninstall` command took. Now the names refer to the command to which
they're associated.

Closes: #1053
Approved by: cgwalters
2017-10-13 15:00:25 +00:00
Jonathan Lebon
74eaa2ba59 daemon: add RPMOSTREE_DEBUG_DISABLE_DAEMON_IDLE_EXIT
The new idle exit behaviour is nice, but it makes debugging it harder
because you have to be fast enough to attach or place your breakpoints
and trigger it before it auto-exits. Add a compile-time flag that
developers can easily turn on to disable the auto-exit behaviour.

Closes: #1052
Approved by: cgwalters
2017-10-12 15:35:44 +00:00
Colin Walters
b2350acf69 bin/compose: (refactor) Split out a constructor function
This moves the JSON parsing and reorders a few things, and
building on the previous commits, adds a constructor function.

Prep for [compose commit](https://github.com/projectatomic/rpm-ostree/pull/1039)

Closes: #1050
Approved by: jlebon
2017-10-12 13:49:42 +00:00
Colin Walters
05698b7c2e bin/compose: (refactor) Move rootfs_dfd into compose data struct
Prep for [compose commit](https://github.com/projectatomic/rpm-ostree/pull/1039).

Closes: #1050
Approved by: jlebon
2017-10-12 13:49:42 +00:00
Colin Walters
973a99895b bin/compose: (refactor) Hoist metadata hash into compose data struct
Prep for [rpmostree compose commit](https://github.com/projectatomic/rpm-ostree/pull/1039).
We still want to write the metadata in that path, so having the data
in the struct will help make things common between the two.

Closes: #1050
Approved by: jlebon
2017-10-12 13:49:42 +00:00
Colin Walters
4564646629 tests/vmcheck: Wait for HTTP server to start
Should fix a flake.

Closes: #1051
Approved by: jlebon
2017-10-11 21:22:19 +00:00
Colin Walters
334f0b89be ex-container: Make /usr/etc/{,g}shadow user-readable
For the `ex container` case, there's no security issues here; one shouldn't be
doing user management in these roots at all.

This is for work on exporting `ex container` roots to OCI as non-root. Without
this fix, libostree just tries to `openat()` the object for export to tar, and
fails.

See also https://github.com/projectatomic/rpm-ostree/issues/1045

Closes: #1046
Approved by: jlebon
2017-10-10 20:42:55 +00:00
Colin Walters
4a4f9952c2 tree-wide: Use a #define for /usr/share/rpm location
In prep for potentially changing it:
http://lists.rpm.org/pipermail/rpm-maint/2017-October/006681.html

Of course actually doing a transition would be harder than this, as we'd need to
add a compat symlink, and even that wouldn't quite be enough as e.g. the
"preview" code would need to learn how to follow the symlink (or just try both
locations).

In practice I think we'd need to land the code to handle both locations, let
that trickle out for e.g. 3 months, then make the switch in our treecomposes.

But, might as well make this change now; using a `#define` makes it slightly
easier to find places that need changing later.

Closes: #1048
Approved by: jlebon
2017-10-10 19:59:21 +00:00
Colin Walters
d0b225c0ab postprocess: Init rootfs earlier
There's a lot going on in the final postprocessing; and some
potentially tricky dependencies.  But we can cleanly move up
the "init rootfs" bits, which helps make the rest of the function
closer to the "final postprocessing" we need for
[split compose](https://github.com/projectatomic/rpm-ostree/pull/1039).

Closes: #1044
Approved by: jlebon
2017-10-10 13:06:35 +00:00
Colin Walters
236dfd723a compose: Clean up internal final postprocess API
It's really way saner if we create the target root dir in the
compose high level.  That way everything relating to actual file names
(and the concept of the "workdir") rather than the "library".

This is prep for [compose split](https://github.com/projectatomic/rpm-ostree/pull/1039)
in that it will be easier to tease out an API for less "opinionated"
postprocessing that just works on `target_rootfs_dfd`.

Closes: #1044
Approved by: jlebon
2017-10-10 13:06:35 +00:00
Jonathan Lebon
5867d2a2af vmcheck: use systemd-run for running post-that-hangs
Just taking what I learned from #1035 and applying it here. What's nice
about this is that there's no cleanup needed. Once the process is killed
(or worst case, we reboot the VM), there's no traces left at all.

Also added a few extra "ok" outputs.

Closes: #1043
Approved by: cgwalters
2017-10-06 20:17:36 +00:00
Colin Walters
b383ef894d compose/postprocess: Fix memleak in error path, minor style update
Using an autoptr for the strbuf not only fixes a memleak in the error path, it's
a bit more efficient since we can just pass `buf->len` rather than running
`strlen()`.

Closes: #1042
Approved by: jlebon
2017-10-06 19:43:39 +00:00
Colin Walters
7c4c2c6640 compose/postprocess: Unlink files in /var as we go
This is prep for compose splitting:
https://github.com/projectatomic/rpm-ostree/issues/471

We'll need to run some postprocessing types in both phases, and it's a lot
easier to reason about if the postprocessing is "idempotent", i.e. it has no
effect a second time.

Closes: #1042
Approved by: jlebon
2017-10-06 19:43:39 +00:00
Jonathan Lebon
dd7c748ce1 tests/vmcheck: add test for refresh-md and --cache-only
Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
dc08ebda0e tests/libvm: use rsync and add yumrepo mode
I've been lazy about actually using using rsync instead of scp when
copying new RPMs over to the VM. We do this here. Also make
`vm_send_test_repo` take a mode argument that allows callers to
completely skip the sending of the repo file itself. This will be needed
for the `makecache` test, in which we *don't* want the repo to be local.
It looks cleaner anyway for the gpgcheck use case as well.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
4442a0c362 app: add -C flag to always use cached metadata
This is the equivalent version of `yum/dnf -C`. It goes together with
the new `makecache` command to allow completely asynchronous cache
update and usage.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
8285ef2a15 app: add 'refresh-md' command
This is essentially the `dnf/yum makecache` equivalent for rpm-ostree.
To complete the picture, this goes hand in hand with the `-C`
equivalent, which is added in the next patch.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
5601a60398 libpriv/core: log repo info when downloading rpmmd
Move the logging of yum repo information from `prepare` to
`download_metadata`, since the latter could be called without
necessarily calling the former, as is the case with `makecache`.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Colin Walters
a2862f5473 bin/compose: (minor) Rename treefile variable for clarity
We had two things called `treefile`, make one of them the path so things are
less confusing.

Closes: #1041
Approved by: jlebon
2017-10-06 15:36:04 +00:00
Jonathan Lebon
235f2945bf ci: don't use CentOS Alpha anymore
It's no longer being built and is now older than the latest CentOS AH
release. This should help us no longer see messages like:

(rpm-ostree pkg-add:5662): GLib-CRITICAL **: g_variant_dict_lookup:
assertion 'is_valid_dict (dict)' failed

which happen because in #1034, we started using `G_VARIANT_DICT_INIT`,
whose special magic values only make sense in glib2 >= 2.50. (The alpha
image stopped at 2.46).

Saw this while debugging #1035.

Closes: #1040
Approved by: cgwalters
2017-10-06 00:42:55 +00:00
Jonathan Lebon
33cb342b6b libpriv/core: allow NULL for treespec
Let callers pass `NULL` for the `@spec` argument. This essentially means
that they're not interested in actually doing any installation/commit
operations, but just the semantics that the core uses when setting up
libdnf wrt e.g. install and source root and enabled repos = required,
etc...

Prep for `makecache`.

Closes: #1038
Approved by: cgwalters
2017-10-05 18:00:54 +00:00
Jonathan Lebon
8fae56d120 libpriv/core: teach core to use deployment for yum repos
Add a function in the core that knows how to take a deployment and point
its passwd dir and libdnf's repos dir to it. We call it the
"configuration deployment", because those are configuration files which
should be inherited from the config merge, not the origin merge.

Prep for `makecache`.

Closes: #1038
Approved by: cgwalters
2017-10-05 18:00:54 +00:00
Colin Walters
f920f36093 tests: Use --parent=none rather than deleting the ref
This is atomic. Not that it matters for this test, but let's have our tests use
ostree well.

Closes: #1037
Approved by: jlebon
2017-10-05 15:03:06 +00:00
Colin Walters
95227f079a WIP: Implement ex livefs --replace
Lots of tradeoffs in this.  See the comments in the code for
more info.  WIP for now.

Closes: #1028
Approved by: jlebon
2017-10-05 13:19:20 +00:00
Jonathan Lebon
4501791dc2 daemon: fix uint negative comparison
Closes: #1036
Approved by: cgwalters
2017-10-04 21:10:55 +00:00
Jonathan Lebon
5db2389fbb ci: run clang build with -Werror
Start running `clang` with `-Werror` like ostree. We can only run this
on Fedora right now because the CentOS `clang` doesn't support
`-Wno-error=macro-redefined`, though that's fine.

Closes: #1036
Approved by: cgwalters
2017-10-04 21:10:55 +00:00