Commit Graph

1194 Commits

Author SHA1 Message Date
Jonathan Lebon
4f0be52dca compose-tests: add more mutate-os-release cases
Check that the logic added in the previous commit is respected.

Closes: #603
Approved by: cgwalters
2017-02-01 22:25:54 +00:00
Jonathan Lebon
c756b26521 compose: fix mutate-os-release handling
- Let --add-metadata-string=version=val override any automatic version
  prefixing.
- Don't error out if mutate-os-release is given, but no new version is
  given by --add-metadata-string or automatic version prefixing.

Checking keys in parse_keyvalue_strings() is slightly hacky. I initially
wanted to just inspect the GVariantBuilder, but AFAICT, there is no way
to actually look up values from a builder (plus, we need that info early
to know whether automatic_version_prefix should itself inject in the
builder or not).

Closes: #603
Approved by: cgwalters
2017-02-01 22:25:54 +00:00
Colin Walters
1b7e35abec upgrader: Fix use of abspath
In the previous commit I changed to abspath, since when reworking things the
relative path broke (I'm still not entirely sure why, but we were basically
lucky before). However I then later realized this wouldn't be right if
rpm-ostree was started with a `--sysroot` argument from Anaconda.

What we can do instead (at least until libdnf grows fd-relative APIs)
is to make it a fd-relative abspath.

Closes: #596
Approved by: jlebon
2017-01-28 22:48:08 +00:00
Colin Walters
7c7806f831 daemon: Fold pkgchange txn into deploy
The next step will be to design a DBus + commandline API for this. In the
meantime, there are some small subtle new features like honoring the dry-run
option for upgrade. I'm not testing that explicitly yet, but I think that's OK.

Closes: #593
Approved by: jlebon
2017-01-28 22:16:09 +00:00
Colin Walters
c7d44e67d1 daemon: Create a flags type for behaviors rather than 3 bools
3 bools in parameters is too many; time to do flags. This also will lead to
deduplicating with the pkg txn flag type when we merge that in. Also, we dedup
the gvariant parsing code.

Closes: #593
Approved by: jlebon
2017-01-28 22:16:09 +00:00
Colin Walters
d7e709b11c daemon/upgrader: Move repo member into object
Following on the previous commit, this drops the last bit of state that we
threaded through various calls. We only need one reference.

Closes: #590
Approved by: jlebon
2017-01-28 13:58:16 +00:00
Colin Walters
84343ae208 daemon/upgrader: Move some state into the object
I was going to do some work here, and adding another parameter to thread down
through these function calls got painful. We already have an object to store
state in, so do it. This makes us more consistent.

Closes: #590
Approved by: jlebon
2017-01-28 13:58:16 +00:00
Jonathan Lebon
e351cb39a1 main.c: check for root after parsing options
So that e.g `rpm-ostree compose tree --help` doesn't need root.

Closes: #592
Approved by: cgwalters
2017-01-26 15:32:27 +00:00
Colin Walters
74afb02a90 daemon: Fold rebase and deploy txn code together
More txn unification.

Closes: #589
Approved by: jlebon
2017-01-26 14:53:04 +00:00
Colin Walters
7ff54daa89 daemon: Fold rebase/upgrade txn code together
Upgrade is just a special case of rebase conceptually where:
  - We fetch the latest tip
  - We enable downgrade protection

This is part of a larger effort where I am aiming to have a *single* transaction
type for system mutation, so we can do both upgrades and package layering and
enable initramfs all in one go.

Closes: #589
Approved by: jlebon
2017-01-26 14:53:04 +00:00
Colin Walters
98be9cef4e Release 2017.1
Closes: #588
Approved by: cgwalters
2017-01-22 16:25:11 +00:00
Jonathan Lebon
7f173fcdc3 compose: fix bad baseurl
Closes: #587
Approved by: cgwalters
2017-01-21 15:27:11 +00:00
Colin Walters
3703e5fa8e libdnf: Bump to latest master
```
$ git shortlog fe5a08bca7e2599798af7778917da2cc31f1460e..ac7825a234268fb057f1b3050103d2ff8c9f2e61
Colin Walters (1):
      repo: Capture last mirrorlist failure for repo metadata too
```

Closes: #586
Approved by: jlebon
2017-01-21 02:31:05 +00:00
Colin Walters
cac4522e5b Support "system/regenerate-initramfs=true" flag in origin
Currently we push for a model where the initramfs is
generated (in non-hostonly mode), and merely replicated.

However, to support a few unfortunate corner cases like dm-multipath which wants
to inject a config file into the initramfs, we need to support regenerating it
client side too.

Down the line, we'll need this to support overriding the kernel too.

This changes things in the core to add the concept of an "empty"
`RpmOstreeContext`. I initially tried skipping it, but that was too much
duplication. We still want all of the core ostree-related logic that lives in
that code too.

The treespec bits barfed if the spec didn't have a `tree/packages` key. It was
simplest to change that to allow it - and because that was the only case where
we errored out in parsing, I dropped the error handling.

There was another place in the upgrader that now needed to be fixed to handle
transitioning from just regenerating initramfs to not.

Closes: #574
Approved by: jlebon
2017-01-20 16:17:51 +00:00
Colin Walters
b099680d1d build: Capture git describe on build if available, add to --version
Many projects do this, and it really helps debugging to know the
exact hash.

(Of course this is broken in traditional rpm builds from a tarball,
 and rpmdistro-gitoverlay injects it into the Version field,
 but it will help me for vmcheck debugging)

Closes: #584
Approved by: jlebon
2017-01-19 19:49:58 +00:00
Colin Walters
d277205c13 tests: Actually install the new binaries
I think since I landed a change to `--enable-new-name`, `/usr/bin/rpm-ostree`
became a symlink, and without the `-l` switch, rsync skips over them.

Hence, we have only been testing in vmcheck the old binaries, not new ones.
Oops.

Closes: #585
Approved by: jlebon
2017-01-19 18:55:15 +00:00
Colin Walters
02166aeecd Move SIGINT handling into transaction helper
Not being able to `Ctrl-C` treecompose kept irritating me and I
finally looked it into it.  I'd thought it was rpm or librepo's
fault, but nope, it's ours!

We had `SIG{INT,HUP,TERM}` handling globally, but unfortunately
right now some things in libdnf don't respect the cancellable
(It's hard to do without threading it down all the way into libcurl
 and rpm).

Really for treecompose we don't need a `SIGINT` (or other) handlers - we should
just take the default action of immediate process exit.

Now, for the command line dbus calls, *only* when we go to execute a txn should
we catch `SIGINT` so that we can forward it to the daemon to cancel there.

Closes: https://github.com/projectatomic/rpm-ostree/issues/489

Closes: #583
Approved by: jlebon
2017-01-19 16:56:48 +00:00
Colin Walters
d0a4a258a8 tests: Make failure of assert_file_empty() actually fatal
I just noticed this while reading the tests.  The original
ostree version looks OK, so not sure how that happened.

Closes: #580
Approved by: jlebon
2017-01-18 20:24:31 +00:00
Colin Walters
f02130b19e tests: Hard require jq
I don't see a reason not to require more stuff to execute tests,
and having it be optional reduces our coverage unnecessarily.

Closes: #580
Approved by: jlebon
2017-01-18 20:24:31 +00:00
Colin Walters
30bed29230 daemon/upgrader: Rework layer tracking
Working on initramfs, I hit a subtle issue with the fact that
I was trying to "redeploy", but with the origin file changed
during the process.

Previously, it was a bit unclear which parts of the upgrader logic are operating
on the *new* origin versus the "original origin".

The package layering code in the upgrader explicitly carries a delta on top in
the "add/remove" hash sets, which means it isn't visible to
`rpmostree_origin_is_locally_assembled()`.

Whereas for initramfs, I set a new origin. This broke things since we were
expecting to find a parent commit, but the original origin wasn't locally
assembled.

When looking more at this, I realized there's a far simpler model -
rather than keeping track of commit + origin, and using the origin
to try to determine whether or not the commit is layered, we can
keep track of `base_revision` and `final_revision`, and the latter
is only set if we're doing layering.

The diff speaks for itself here - a lot of fragile logic looking at the origin
drops away.

The next step here is probably to drop away the package layering hash sets, but
I'm trying to not change everything at once.

Closes: #579
Approved by: jlebon
2017-01-18 20:05:19 +00:00
Colin Walters
389cba6453 ci: Add jq
Prep for requiring it.

Closes: #581
Approved by: jlebon
2017-01-18 17:15:36 +00:00
Colin Walters
59f34e8a03 tests: Disable ASAN leak checks
Right now we're far from leak free, and I want to use ASAN
as a sanity checker, not a leak checker.

Closes: #576
Approved by: jlebon
2017-01-18 16:52:46 +00:00
Colin Walters
c6f3c43295 tests: Brutally hack in copying in sanitizer shared libraries
Yes, there are better ways to do this, but there are also
better things to do...

Closes: #576
Approved by: jlebon
2017-01-18 16:52:46 +00:00
Colin Walters
99bca8ad3f Suppress uid == 0 check in unit tests
Our `make check` runs an unprivileged rpm-ostreed if run as non-root; this is a
feature. We didn't notice in the CI tests since those run as "docker-uid0". It
does break my local workflow though.

Closes: #573
Approved by: jlebon
2017-01-16 19:03:02 +00:00
Colin Walters
b3e16a3313 bwrap: Turn on all namespaces
The fact we weren't doing this is an oversight.  We should *really*
be using the PID namespace at a minimum, but I decided to just turn
them all on.

The one that seems most likely to potentially introduce a regression is turning
on the netns (i.e. disabling networking). But I can't really think of what we'd
be running in a script today that would break in practice.

Closes: #572
Approved by: jlebon
2017-01-16 14:40:47 +00:00
Colin Walters
6ce3baf77c libdnf: Bump to current master
Mostly for the compilation warning fixes, but everything else is good too.

```
Colin Walters (2):
      build: Propagate CFLAGS to g-ir-scanner
      tree-wide: Squash -Werror=incompatible-pointer-type warnings

Igor Gnatenko (2):
      add RHSM enrollment support
      dnf-context: initialize repos after calling setup() from plugins

Jan Šilhan (1):
      README: reflect changes libdnf renaming and adding links

Jaroslav Mracek (4):
      Change conflict dnf version
      Add support for selectors for goal.downgrade_to
      Increase minor version
      Enable glob usage in query.filter for release (RhBug:1267895)

Kalev Lember (12):
      repo: Move enabled and enabled_metadata parsing here
      repo: Correctly handle "True" and "False" in .repo files
      repo: Parse skip_if_unavailable
      repo: Use same error enum for both gpg key and regular download errors
      repo: Avoid leaking checksum_str for each download
      tests: Fix a memory leak in test_goal_get_solution()
      goal: Fix memory management for hy_goal_describe_problem_rules()
      goal: Fix a memory leak in add_preferred_provide()
      goal: Fix a memory leak in filter_pkg2job()
      repo: Remove leftover .tmp dir on failure
      repo: Fix metadata signature checking in dnf_repo_check()
      Fix -Wformat warnings on i686
```

Closes: #571
Approved by: jlebon
2017-01-13 20:23:15 +00:00
Colin Walters
f8edd59d5f daemon: Introduce an origin structure
In prep for adding "regenerates initramfs" to "is locally assembled",
we need more abstraction.  I previously introduced a helper function, but
I'd still need to touch every call site when changing that.

Rather than having each caller re-parse the origin, let's parse it once into a
structure, and then have error-free accessors for it (that also don't malloc).
This way when adding a new flag, I don't need to touch every call site.

Notes:

 - Some places in the code (like the deployments -> variant bits) tried to
   handle deployments without an origin gracefully.  That's no
   longer true.  I'm not sure how much we care - do we?
 - There are a few places where I changed `packagelist.len() > 0` to
   `is_locally_assembled()`.  I think this is right, but we'll need
   to be sure the "no packages to overlay, just initramfs" case
   works when that lands.

Closes: #566
Approved by: jlebon
2017-01-13 20:08:24 +00:00
Colin Walters
f215acb64f libpriv: Move kernel finalization into a helper function
This will need to be callable outside of postprocess for client side initramfs
generation.

Closes: #566
Approved by: jlebon
2017-01-13 20:08:24 +00:00
Colin Walters
4c98943a64 postprocess: Clean up kernel/initramfs checksum calculation
This avoids having an intermediate file path, and makes the code easier to
follow.

Closes: #566
Approved by: jlebon
2017-01-13 20:08:24 +00:00
Colin Walters
16efa12ae0 libpriv: Factor out "find kernel/initramfs" code into kernel.[ch]
To support running dracut on the client side, the dracut code
needs this, and it makes more sense in kernel.[ch] anyways.

I chose to use a GVariant instead of making a custom structure to avoid having
to manage custom free funcs.

Closes: #566
Approved by: jlebon
2017-01-13 20:08:24 +00:00
Colin Walters
178ec03154 tests: Honor TEST_SKIP_CLEANUP like ostree
We seem to have substantial infrastructure drift here between the two, and I was
*really* confused why `TEST_SKIP_CLEANUP` wasn't being honored. Turned out in
rpm-ostree we have tmpdir code in two places, not entirely sure why. Punting
full cleanup.

Closes: #566
Approved by: jlebon
2017-01-13 20:08:24 +00:00
Colin Walters
4a511fa011 app: Check for root privileges where required early
Rather than sending a dbus message that gets denied, which
is ugly.

Closes: #565

Closes: #570
Approved by: jlebon
2017-01-13 19:46:24 +00:00
Colin Walters
185fba0551 .dir-locals.el: Global Emacs style settings
This ensures things get applied consistently.  I was noticing
some weird indentation in Spacemacs.

Closes: #567
Approved by: jlebon
2017-01-12 16:09:16 +00:00
Colin Walters
e1cdfa1712 libpriv: Introduce an rpmostree-kernel.[ch] for dracut
Since the dracut run is now separated from the rest of postprocess, we change it
to use the bwrap API diretly, and this lets use the new _IMMUTABLE bwrap type.
This will make it easier to reuse for client-side initramfs regeneration.

Splitting this off makes it also easier to simplify the remaining mutable
usage in postprocess.c.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:26 +00:00
Colin Walters
f0ec738376 libpriv: Enhance internal bwrap API
Make it a real struct with methods.  This noticeably increases
the ergonomics and design of the API.

The main goal here is to introduce the enum which defines whether or not the
rootfs is mutable or not. We move the "rofiles" mode from the RPM script code
down into the bwrap layer, which will make it easier to reuse for treecompose.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:26 +00:00
Colin Walters
50ab2983ab postprocess: Use O_TMPFILE for dracut, merge reproducible bits
I was planning to do some further changes here, and I really don't like the
manual fork/exec stuff on in the --reproducible checks. Our subprocess code
should basically be all bwrap.  Synchronous code execution while not reading from the
pipe is a recipe for deadlocks.

What simplifies things a lot is to write to an `O_TMPFILE` fd (or a tempfile on
legacy kernels), and slightly extend our bwrap-executing code to support a child
setup function, so we can set the tmpfile fd to be stdout.

Now that we have a shell script wrapper we inject, it's trivial to reimplement
the "detect reproducibility" in shell script there, rather than C.

This doesn't matter much for treecompose today, but it will matter more when
we're supporting client side initramfs regeneration, since now the dracut
container can be fully immutable.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
Colin Walters
7a421443f9 core: Split up overlay/commit API
In a future change I want to optionally rerun dracut between these. Keep the old
API as a trivial wrapper around the two.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
Colin Walters
f766b59fbf daemon: Add a helper function for "is locally assembled"
This is all two callers need, and in the future I want to add
local initramfs generation.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
Colin Walters
1c3b03af8d daemon: Log into the journal for txn failures
We should really be logging more into the journal in general,
but particularly this will make it easier to find errors
in the test suite.

Right now we only dump script stderr into the journal.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
Colin Walters
0ed86d7475 daemon: Use autoptr for cancellable, initialize inline
This drops one line + whitespace line from each txn, and updates us to use
autoptr. Just a bit less duplication.

I noticed while doing this that not every txn initializes a cancellable, but not
shaving that yak right now.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
Colin Walters
26909a44c1 daemon: Dedup transaction-merging code a bit
We had a lot of copy/paste of this, it's hard to entirely get rid of, but this
is at least smaller.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
Colin Walters
533ff648a6 vmcheck: Note how many tests were skipped
I was initially confused as to the pattern matching, and this will make things a
bit clearer.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
Colin Walters
dee0807147 vmcheck: Scrape out the journal on failure
So we can debug things more easily.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
Colin Walters
d3bf60d373 tests: Add a pkg with rofiles violation that should fail
To ensure we don't accidentally allow mutations.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
Colin Walters
55caec269d libvm: Set -o User=root for rsync too
Without fuse-sshfs, we also need to set the root user, which
is necessary for the Vagrant case.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
Colin Walters
be7dd69b7b libvm: Make it easier to debug login failures
First try to log in, and if we fail, retry with SSH debugging,
so we have a better idea what might be going wrong.  This helped
me figure out that vmcheck's `-o User=root` assumption wasn't
working with my vagrant setup.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
petervo
4fc3589dc0 daemon: Really allow downgrades when pulling a rebase diff
Closes: #563
Approved by: jlebon
2017-01-10 14:09:38 +00:00
Jonathan Lebon
e71e7219d3 redhat-ci: switch compose over to Fedora 25 for now
https://github.com/CentOS/sig-atomic-buildscripts/issues/210

Closes: #561
Approved by: cgwalters
2017-01-08 21:05:06 +00:00
Jonathan Lebon
e32bbf16d1 passwd_prepare_rpm_layering: account for local entries
On Fedora 25, systemd adds a sysuser config file for multiple users. It
also explicitly creates those same users in its %pre, except for one:
systemd-coredump. This means that the tree's /usr/lib/passwd doesn't
contain systemd-coredump. Of course, on first boot, it gets created and
added to /etc/passwd.

During package layering, we map /usr/lib/passwd to the container's
/etc/passwd. If the %pre calls useradd/groupadd without passing an
explicit uid/gid, it's possible that the allocated id is already in use
by an entry in the deployment's /etc/{passwd,group} (such as
systemd-coredump, but the same holds for any manually-added entry).

We resolve this by taking the switcheroo a step further: we map
/usr/lib/passwd to /usr/etc/passwd, and then also map /etc/passwd to
/usr/lib/passwd. That way, useradd in %pre will account for already
allocated local uids and react accordingly.

Closes: #561
Approved by: cgwalters
2017-01-08 21:05:06 +00:00
Jonathan Lebon
9f8136e097 vmcheck: add test-layering-non-root-caps.sh
This new test exercises the new support for non-root file paths and
files with capabilities.

Closes: #561
Approved by: cgwalters
2017-01-08 21:05:06 +00:00