Commit Graph

1906 Commits

Author SHA1 Message Date
Colin Walters
b2350acf69 bin/compose: (refactor) Split out a constructor function
This moves the JSON parsing and reorders a few things, and
building on the previous commits, adds a constructor function.

Prep for [compose commit](https://github.com/projectatomic/rpm-ostree/pull/1039)

Closes: #1050
Approved by: jlebon
2017-10-12 13:49:42 +00:00
Colin Walters
05698b7c2e bin/compose: (refactor) Move rootfs_dfd into compose data struct
Prep for [compose commit](https://github.com/projectatomic/rpm-ostree/pull/1039).

Closes: #1050
Approved by: jlebon
2017-10-12 13:49:42 +00:00
Colin Walters
973a99895b bin/compose: (refactor) Hoist metadata hash into compose data struct
Prep for [rpmostree compose commit](https://github.com/projectatomic/rpm-ostree/pull/1039).
We still want to write the metadata in that path, so having the data
in the struct will help make things common between the two.

Closes: #1050
Approved by: jlebon
2017-10-12 13:49:42 +00:00
Colin Walters
4564646629 tests/vmcheck: Wait for HTTP server to start
Should fix a flake.

Closes: #1051
Approved by: jlebon
2017-10-11 21:22:19 +00:00
Colin Walters
334f0b89be ex-container: Make /usr/etc/{,g}shadow user-readable
For the `ex container` case, there's no security issues here; one shouldn't be
doing user management in these roots at all.

This is for work on exporting `ex container` roots to OCI as non-root. Without
this fix, libostree just tries to `openat()` the object for export to tar, and
fails.

See also https://github.com/projectatomic/rpm-ostree/issues/1045

Closes: #1046
Approved by: jlebon
2017-10-10 20:42:55 +00:00
Colin Walters
4a4f9952c2 tree-wide: Use a #define for /usr/share/rpm location
In prep for potentially changing it:
http://lists.rpm.org/pipermail/rpm-maint/2017-October/006681.html

Of course actually doing a transition would be harder than this, as we'd need to
add a compat symlink, and even that wouldn't quite be enough as e.g. the
"preview" code would need to learn how to follow the symlink (or just try both
locations).

In practice I think we'd need to land the code to handle both locations, let
that trickle out for e.g. 3 months, then make the switch in our treecomposes.

But, might as well make this change now; using a `#define` makes it slightly
easier to find places that need changing later.

Closes: #1048
Approved by: jlebon
2017-10-10 19:59:21 +00:00
Colin Walters
d0b225c0ab postprocess: Init rootfs earlier
There's a lot going on in the final postprocessing; and some
potentially tricky dependencies.  But we can cleanly move up
the "init rootfs" bits, which helps make the rest of the function
closer to the "final postprocessing" we need for
[split compose](https://github.com/projectatomic/rpm-ostree/pull/1039).

Closes: #1044
Approved by: jlebon
2017-10-10 13:06:35 +00:00
Colin Walters
236dfd723a compose: Clean up internal final postprocess API
It's really way saner if we create the target root dir in the
compose high level.  That way everything relating to actual file names
(and the concept of the "workdir") rather than the "library".

This is prep for [compose split](https://github.com/projectatomic/rpm-ostree/pull/1039)
in that it will be easier to tease out an API for less "opinionated"
postprocessing that just works on `target_rootfs_dfd`.

Closes: #1044
Approved by: jlebon
2017-10-10 13:06:35 +00:00
Jonathan Lebon
5867d2a2af vmcheck: use systemd-run for running post-that-hangs
Just taking what I learned from #1035 and applying it here. What's nice
about this is that there's no cleanup needed. Once the process is killed
(or worst case, we reboot the VM), there's no traces left at all.

Also added a few extra "ok" outputs.

Closes: #1043
Approved by: cgwalters
2017-10-06 20:17:36 +00:00
Colin Walters
b383ef894d compose/postprocess: Fix memleak in error path, minor style update
Using an autoptr for the strbuf not only fixes a memleak in the error path, it's
a bit more efficient since we can just pass `buf->len` rather than running
`strlen()`.

Closes: #1042
Approved by: jlebon
2017-10-06 19:43:39 +00:00
Colin Walters
7c4c2c6640 compose/postprocess: Unlink files in /var as we go
This is prep for compose splitting:
https://github.com/projectatomic/rpm-ostree/issues/471

We'll need to run some postprocessing types in both phases, and it's a lot
easier to reason about if the postprocessing is "idempotent", i.e. it has no
effect a second time.

Closes: #1042
Approved by: jlebon
2017-10-06 19:43:39 +00:00
Jonathan Lebon
dd7c748ce1 tests/vmcheck: add test for refresh-md and --cache-only
Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
dc08ebda0e tests/libvm: use rsync and add yumrepo mode
I've been lazy about actually using using rsync instead of scp when
copying new RPMs over to the VM. We do this here. Also make
`vm_send_test_repo` take a mode argument that allows callers to
completely skip the sending of the repo file itself. This will be needed
for the `makecache` test, in which we *don't* want the repo to be local.
It looks cleaner anyway for the gpgcheck use case as well.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
4442a0c362 app: add -C flag to always use cached metadata
This is the equivalent version of `yum/dnf -C`. It goes together with
the new `makecache` command to allow completely asynchronous cache
update and usage.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
8285ef2a15 app: add 'refresh-md' command
This is essentially the `dnf/yum makecache` equivalent for rpm-ostree.
To complete the picture, this goes hand in hand with the `-C`
equivalent, which is added in the next patch.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
5601a60398 libpriv/core: log repo info when downloading rpmmd
Move the logging of yum repo information from `prepare` to
`download_metadata`, since the latter could be called without
necessarily calling the former, as is the case with `makecache`.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Colin Walters
a2862f5473 bin/compose: (minor) Rename treefile variable for clarity
We had two things called `treefile`, make one of them the path so things are
less confusing.

Closes: #1041
Approved by: jlebon
2017-10-06 15:36:04 +00:00
Jonathan Lebon
235f2945bf ci: don't use CentOS Alpha anymore
It's no longer being built and is now older than the latest CentOS AH
release. This should help us no longer see messages like:

(rpm-ostree pkg-add:5662): GLib-CRITICAL **: g_variant_dict_lookup:
assertion 'is_valid_dict (dict)' failed

which happen because in #1034, we started using `G_VARIANT_DICT_INIT`,
whose special magic values only make sense in glib2 >= 2.50. (The alpha
image stopped at 2.46).

Saw this while debugging #1035.

Closes: #1040
Approved by: cgwalters
2017-10-06 00:42:55 +00:00
Jonathan Lebon
33cb342b6b libpriv/core: allow NULL for treespec
Let callers pass `NULL` for the `@spec` argument. This essentially means
that they're not interested in actually doing any installation/commit
operations, but just the semantics that the core uses when setting up
libdnf wrt e.g. install and source root and enabled repos = required,
etc...

Prep for `makecache`.

Closes: #1038
Approved by: cgwalters
2017-10-05 18:00:54 +00:00
Jonathan Lebon
8fae56d120 libpriv/core: teach core to use deployment for yum repos
Add a function in the core that knows how to take a deployment and point
its passwd dir and libdnf's repos dir to it. We call it the
"configuration deployment", because those are configuration files which
should be inherited from the config merge, not the origin merge.

Prep for `makecache`.

Closes: #1038
Approved by: cgwalters
2017-10-05 18:00:54 +00:00
Colin Walters
f920f36093 tests: Use --parent=none rather than deleting the ref
This is atomic. Not that it matters for this test, but let's have our tests use
ostree well.

Closes: #1037
Approved by: jlebon
2017-10-05 15:03:06 +00:00
Colin Walters
95227f079a WIP: Implement ex livefs --replace
Lots of tradeoffs in this.  See the comments in the code for
more info.  WIP for now.

Closes: #1028
Approved by: jlebon
2017-10-05 13:19:20 +00:00
Jonathan Lebon
4501791dc2 daemon: fix uint negative comparison
Closes: #1036
Approved by: cgwalters
2017-10-04 21:10:55 +00:00
Jonathan Lebon
5db2389fbb ci: run clang build with -Werror
Start running `clang` with `-Werror` like ostree. We can only run this
on Fedora right now because the CentOS `clang` doesn't support
`-Wno-error=macro-redefined`, though that's fine.

Closes: #1036
Approved by: cgwalters
2017-10-04 21:10:55 +00:00
Colin Walters
f3c63b6850 app: Unify some cmdline txn processing
We had duplicated code across the cmdline entrypoints for transaction
processing; things like "print pkg diff only if !opt_reboot".

This doesn't dedup all of them - there are some corner cases around
the preview logic in `upgrade`, and `initramfs` also need special
handling.  I'll likely enhance this further down the line for that.

But one reason I'm doing this now is prep for:
[rpm-ostree cancel](https://github.com/projectatomic/rpm-ostree/pull/1019)

Basically, I want to add a `-B/--background` option we honor consistently, and
that'd be a lot easier if we have a combined "start/monitor txn" with the
post-txn option processing in one place.

Closes: #1034
Approved by: jlebon
2017-10-04 13:23:49 +00:00
Colin Walters
025633abed app: Add a more reliable helper for finding active transaction
There are (somewhat hard to avoid) race conditions for the client
to find the active transaction and connect to it.  This approach
adds a retry loop, and teaches the status builtin to use it.

The status code becomes a lot less ugly.  Prep for:
[rpm-ostree cancel](https://github.com/projectatomic/rpm-ostree/pull/1019)

Closes: #1033
Approved by: peterbaouoft
2017-10-03 21:40:20 +00:00
Colin Walters
cafe89cab7 daemon: Exit on idle after ~10-15 seconds
Now that we have the ability to both track clients and our active transaction,
and the `RegisterClient` call acts "atomically", let's start doing exit-on-idle
and return the RAM to the people.

Closes: #606
Approved by: jlebon
2017-10-03 17:56:21 +00:00
Colin Walters
d31c9da236 app: Call RegisterClient with well-known name, use retry loop
This closes a race condition for having the daemon idle exit.  After
the daemon has released its bus name, the dbus-daemon will no longer
allow messages through that targeted its unique name.

Since the intention of the `RegisterClient` method is to be the "knock on the
door", fix this by directly sending a message to the well-known name.

Second, we need to handle the case where the daemon exits without
replying; @jlebon added a `sleep(10)` invocation after the daemon
mainloop quit but before we `ReleaseName`, and I verified these two
things combine to fix that case.

Closes: #606
Approved by: jlebon
2017-10-03 17:56:21 +00:00
Colin Walters
3f367dbce2 scripts+bwrap: Make script execution cancellable
Prep for implementing `rpm-ostree cancel`, but this works with the way we handle
`Ctrl-C` interactively on a client as well. Being able to cancel a script
execution is quite nice; some of them are expensive, and having one loop forever
has been known to happen.

Closes: #1025
Approved by: jlebon
2017-10-03 14:24:07 +00:00
Jonathan Lebon
a9c38d33b8 vmcheck: add SELinux labeling tests
There's a lot of paths in the core related to SELinux policy changes and
relabeling packages. We currently have no test coverage for them. We add
support in the test libraries here to build such packages.

We also add a test that checks both that we correctly relabel RPMs when
the policy changes and that we handle layered packages that install
SELinux packages properly.

Closes: #999
Approved by: cgwalters
2017-10-03 01:01:14 +00:00
Jonathan Lebon
4ad5747b96 libpriv/core: use final sepolicy when doing final commit
It's possible that a layered package installed an SELinux package which
can change the labeling of some files. The problem is that imported
packages will not have used the updated policy and thus may have some of
its paths mislabeled. We fix this here by always loading the SELinux
policy from the tmprootfs and using that for the final commit.

Closes: #999
Approved by: cgwalters
2017-10-03 01:01:14 +00:00
Jonathan Lebon
8b1001c9b5 tests/vmcheck: turn on strict bash mode
Closes: #1032
Approved by: cgwalters
2017-10-02 20:36:07 +00:00
Jonathan Lebon
2c94b11b68 libvm: don't rsync $VMCHECK_INSTTREE
We pretty standardized on insttree/ being the installation tree, and
we don't actually support $VMCHECK_INSTTREE anymore.

Closes: #1032
Approved by: cgwalters
2017-10-02 20:36:07 +00:00
Jonathan Lebon
5c25945dd3 Makefile-tests.am: actually support SKIP_INSTALL
This was supposed to be part of #968. `HACKING.md` already refers to
a `SKIP_INSTALL` variable, which basically ensures that we use the
install tree as-is.

Closes: #1032
Approved by: cgwalters
2017-10-02 20:36:07 +00:00
Colin Walters
699d80ba2d bin/compose: Drop workdir variable
Nothing actually uses it, we've been all fd-relative for a while. Just noticed
this while looking at the compose code for further work.

Closes: #1030
Approved by: jlebon
2017-10-02 18:23:54 +00:00
Colin Walters
1acd834104 Add test infra for ex container, and one test for bash
We have some unit-style tests that run `ex container`, but
they aren't "real"; they don't use scripts for example.  Let's
add tests for this similar to `tests/compose`.

We use a 26 base, but the target repos need to be 27
to pick up the fix for: https://bugzilla.redhat.com/show_bug.cgi?id=1478172

Add some bits to share infra between `tests/compose` and `tests/ex-container`;
basically handling the rpmmd repos. I tweaked things to be more streamlined
there between the `.papr.yml` and the test script.

Right now this is just one test for `bash`, but lays some of the infrastructure
for doing more. One thing that we need to do to improve more here is to better
cache RPMs, a bit like the compose tests do.

Closes: #1024
Approved by: jlebon
2017-10-02 18:04:24 +00:00
Colin Walters
57937d0e63 libtest: Squash a race in temp webserver setup
Saw this in one of the PR contexts.

Closes: #1031
Approved by: jlebon
2017-10-02 18:02:40 +00:00
Colin Walters
d4af84d428 Change ex-container to use BARE_USER_ONLY, update core creation API
Switch `ex container` to `OSTREE_REPO_MODE_BARE_USER_ONLY`; this is a good
match, for the same reasons as flatpak. We don't want suid binaries, we don't
want to chown anything, no SELinux labeling, etc.  There's no reason to write the
metadata to the `BARE_USER` xattrs.

The "unprivileged" context as used for `ex container` was a flag that was passed
down, but I think it's a lot easier if things just infer this state by looking
at whether the target repo is `bare-user-only`. Now we just have an
`rpmostree_context_new_tree()` API that handles both container and compose.

Closes: #1026
Approved by: jlebon
2017-10-02 16:13:52 +00:00
Colin Walters
d16d53ecae ex-container: Add --cache-only
Like the compose path; mostly doing this right now since I'm on a plane ✈.

Closes: #1026
Approved by: jlebon
2017-10-02 16:13:52 +00:00
Jonathan Lebon
70caa3b2da tree: add vimrc and editorconfig
Imported from ostree.

Closes: #1029
Approved by: cgwalters
2017-10-02 14:36:44 +00:00
Colin Walters
acbbda8214 tree-wide: Some more misc libglnx porting
More use of the `glnx_unlinkat()`, `glnx_fstatat()`, and `glnx_renameat()`
wrappers.

Closes: #1027
Approved by: jlebon
2017-10-02 13:41:01 +00:00
Jonathan Lebon
2cf87ba897 tree: some more glnx_fstatat_allow_noent porting
Closes: #1022
Approved by: cgwalters
2017-09-28 21:15:10 +00:00
Colin Walters
d6de31d90d libpriv/scripts: Only log to journal if uid == 0
The previous change to log to the journal broke running `ex container` as
non-root with scripts. Yes, I really desperately need to set up real tests for
that, like we have for composes. And obviously checking `getuid() == 0` is a bit
crude but doing better would require plumbing through quite a bit of of
knowledge from the toplevel down into script execution.

Closes: #1021
Approved by: jlebon
2017-09-28 20:59:18 +00:00
Colin Walters
6ad113ca46 daemon: Some misc new style porting
Was just reading the initramfs bits while talking to rubao about kargs work, and
noticed we had a lot of "old style" declarations.

Closes: #1020
Approved by: jlebon
2017-09-28 19:29:31 +00:00
Colin Walters
c0ef5eec4f libpriv/core: Check cancellable before/after downloading rpm-md
This is a quick band-aid to improve Ctrl-C handling in this area:
https://github.com/projectatomic/rpm-ostree/issues/897

At least this way we'll exit earlier rather than waiting until all the repos are
downloaded (and imported into libsolv, which is itself slow).

Closes: #1018
Approved by: jlebon
2017-09-28 17:55:18 +00:00
Jonathan Dieter
fd6109af1d compose: Copy in additional files before postprocessing
Sometimes it's useful to have access to the additional files when running
the post script, so this re-orders the compose process to copy the
additional files in before the post script runs

Signed-off-by: Jonathan Dieter <jdieter@lesbg.com>

Closes: #997
Approved by: jlebon
2017-09-28 15:38:55 +00:00
Colin Walters
2e59a2748d postprocess: Allow add-files into /etc
Even though it's really `/usr/etc`. This is for greater consistency with
`postprocess-script` where it appears as `/etc`.

Closes: #997
Approved by: jlebon
2017-09-28 15:38:55 +00:00
Colin Walters
45287828e5 lib/compose: Maintain /etc as /usr/etc more consistently
Lots of confusion in the codebase about this. The basic problem is that in
*most* cases, our code doesn't care; it's conceptually operating on `/usr/etc`,
which we could maintain as `/etc` and just rename it back at the very end.

The exceptions though are the `/etc/passwd` handling and livefs. And of course
libostree needs to handle `/usr/etc` vs `/etc` for config merging.

I considered trying to keep things the other way, but while I think we have some
ugly added here in this patch for things where we need to maintain an external
view (`remove-files` and `remove-from-packages`, and boy am I glad we had tests
for those), this ends up being mostly more consistent elsewhere.

One thing that might help is to maintain a fd for it; but that'd be an even more
invasive change.

This also ends up rolling in some unified core prep from
https://github.com/projectatomic/rpm-ostree/pull/940 in the form of
`rename_if_exists()` - basically for some minimal rootfs we may not have
`/boot`, or for that matter potentially even `etc`.

Prep for https://github.com/projectatomic/rpm-ostree/pull/997

Closes: #997
Approved by: jlebon
2017-09-28 15:38:55 +00:00
Colin Walters
11d9a78094 libpriv/core: Add some comments (and a bit of style porting)
I was reading some of the core code recently, and in general I think we need
more comments. I made a pass from the top of the file and got halfway down.

Closes: #1017
Approved by: jlebon
2017-09-28 13:04:29 +00:00
Jonathan Lebon
cb7e84c4a6 libpriv/scripts: redirect scriptlet output to journal
Rather than just letting the scriptlets inherit the daemon's
stdout/stderr, redirect their outputs so that we can set a customized
identifier to make it easier to distinguish from the daemon output.

Also print out the `journalctl` command needed so that users can
investigate the output themselves.

Closes: #998
Approved by: cgwalters
2017-09-27 19:01:45 +00:00