Commit Graph

1349 Commits

Author SHA1 Message Date
Colin Walters
be7dd69b7b libvm: Make it easier to debug login failures
First try to log in, and if we fail, retry with SSH debugging,
so we have a better idea what might be going wrong.  This helped
me figure out that vmcheck's `-o User=root` assumption wasn't
working with my vagrant setup.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
petervo
4fc3589dc0 daemon: Really allow downgrades when pulling a rebase diff
Closes: #563
Approved by: jlebon
2017-01-10 14:09:38 +00:00
Jonathan Lebon
e71e7219d3 redhat-ci: switch compose over to Fedora 25 for now
https://github.com/CentOS/sig-atomic-buildscripts/issues/210

Closes: #561
Approved by: cgwalters
2017-01-08 21:05:06 +00:00
Jonathan Lebon
e32bbf16d1 passwd_prepare_rpm_layering: account for local entries
On Fedora 25, systemd adds a sysuser config file for multiple users. It
also explicitly creates those same users in its %pre, except for one:
systemd-coredump. This means that the tree's /usr/lib/passwd doesn't
contain systemd-coredump. Of course, on first boot, it gets created and
added to /etc/passwd.

During package layering, we map /usr/lib/passwd to the container's
/etc/passwd. If the %pre calls useradd/groupadd without passing an
explicit uid/gid, it's possible that the allocated id is already in use
by an entry in the deployment's /etc/{passwd,group} (such as
systemd-coredump, but the same holds for any manually-added entry).

We resolve this by taking the switcheroo a step further: we map
/usr/lib/passwd to /usr/etc/passwd, and then also map /etc/passwd to
/usr/lib/passwd. That way, useradd in %pre will account for already
allocated local uids and react accordingly.

Closes: #561
Approved by: cgwalters
2017-01-08 21:05:06 +00:00
Jonathan Lebon
9f8136e097 vmcheck: add test-layering-non-root-caps.sh
This new test exercises the new support for non-root file paths and
files with capabilities.

Closes: #561
Approved by: cgwalters
2017-01-08 21:05:06 +00:00
Jonathan Lebon
670f2c5e79 Makefile-tests.am: add SKIP_VMOVERLAY
This is useful when you want to rerun vmcheck after changing the test
case (rather than any compiled code), so no new overlay is actually
required.

Closes: #561
Approved by: cgwalters
2017-01-08 21:05:06 +00:00
Jonathan Lebon
0bbba38727 core: add support for layering non-root files
We lift the restriction from the unpacker on non-root owners. The
unpacker now imports all files as root:root.

We split the running of rpm scripts between %pre and %post/posttrans as
two different stages. During assembly, we first run the %pre scripts,
collect users and groups, then chown as needed (and then finally run the
%post scripts).

Closes: #462

Closes: #561
Approved by: cgwalters
2017-01-08 21:05:06 +00:00
Jonathan Lebon
69e75debc2 rpmostree-passwd: make passwd/group parsers public
We're going to need to access these functions later.

Closes: #561
Approved by: cgwalters
2017-01-08 21:05:06 +00:00
Jonathan Lebon
0e431a8f83 core: add RPMOSTREE_USE_CACHED_METADATA
Being able to just reuse metadata is especially helpful when trying to
debug things lower down that path, as well as cuts about 2 mins from the
compose test.

Closes: #561
Approved by: cgwalters
2017-01-08 21:05:06 +00:00
Jonathan Lebon
b68209b6d4 daemon: start with one commit only when resolving versions
During a deploy operation, we would fetch commit objects from the remote
to resolve the version string. If gpg-verify was turned on, we would
fail to pull them if some of the commits were not signed. This is
because we pulled them in batches. We partially address this by only
fetching the HEAD commit on the first pass. This allows `upgrade`
operations to work just as well as `deploy` operations.

Though there is still an issue if we have to traverse farther back than
when signed commits become unsigned (unless they happen to fall on a
batch boundary). We leave that unsolved for now, since that would likely
require a more complex solution and it's not clear whether it's a real
world issue (signers can just retroactively sign commits).

Copy the gpghome from ostree so that we can test GPG-related cases in
our suite.

Closes: #527

Closes: #557
Approved by: cgwalters
2016-12-24 12:28:48 +00:00
Jonathan Lebon
e10c97007f rebase: add support for rebasing to a specific rev
Expand the available options in the Rebase() D-Bus method to also have a
"revision" key. Its value has the same semantics as the "revision" key
in the Deploy() method (e.g. the "revision=" and "version=" prefixes are
also supported). Also expand the rebase CLI to allow for specifying the
revision as an additional argument.

This allows users to rebase to a specific version or checksum, rather
than only to the latest. Conceptually, this is the equivalent of doing a
rebase followed by a deploy. I.e. we specify an override-commit in the
origin and expect the same behaviours that apply after a deploy to also
apply here.

Closes: #212

Closes: #555
Approved by: cgwalters
2016-12-21 20:00:43 +00:00
Jonathan Lebon
c5fa202378 sync.sh: tweak for container workflow
In the container build + vagrant test workflow, we have to run `make
vmsync` from the env in which autoconf was run, which is the container.

However, when using git worktrees (which is useful in this scenario to
avoid having to overwrite your local builds) and a CentOS build
container, the git inside is too old to understand worktrees, so let's
avoid invoking git in any of the test harness scripts.

Also make sure to restorecon after installing the new rpm-ostreed.

Closes: #555
Approved by: cgwalters
2016-12-21 20:00:43 +00:00
Jonathan Lebon
2f68c02e1a Vagrantfile: specify full path to using_sshfs
When running `vagrant ssh-config` from ansible, Vagrant for some reason
still wants to execute the File.write() operation, but fails to do so
because the $PWD is $topsrcdir/vagrant, in which .vagrant does not
exist.

Switch to using the absolute path.

Closes: #555
Approved by: cgwalters
2016-12-21 20:00:43 +00:00
petervo
9a2788c7b6 daemon: Allow downgrades when pulling a rebase diff
Running the actual rebase already does this automatically.

Closes: #552
Approved by: jlebon
2016-12-20 20:04:25 +00:00
Colin Walters
3a409342e6 postprocessing: Add a g_prefix_error for kernel
Saw this un-prefixed error path in a build; having error prefixes makes for
easier debugging.

Closes: #554
Approved by: giuseppe
2016-12-20 16:24:25 +00:00
Colin Walters
9b51b7f652 Release 2016.13
Closes: #549
Approved by: jlebon
2016-12-12 17:20:32 +00:00
Jonathan Lebon
68acb1d2be compose-tests: add more tests
Add a few more tests to exercise some of the treefile options. We do
need to also expand test-basic.sh itself to sanity-check the structure
of a normal ostree compose. That's up next on the list.

Closes: #548
Approved by: cgwalters
2016-12-09 16:18:11 +00:00
Jonathan Lebon
0fa6f0be82 treefile.md: fix wrong field name copy-files
Closes: #548
Approved by: cgwalters
2016-12-09 16:18:11 +00:00
Colin Walters
69cace4ce8 [ASAN] Squash a few minor misc leaks
This is part of the leak set when doing a treecompose.  There's
some that go down into libdnf...will get to that over time.

Closes: #547
Approved by: jlebon
2016-12-08 16:27:24 +00:00
Colin Walters
23198fd553 build: Sync up warnings with ostree
Mostly to pick up `-Werror=incompatible-pointer-types`.

Closes: #542
Approved by: jlebon
2016-12-07 22:57:10 +00:00
Jonathan Lebon
a21c05ab10 vmcheck: disable default repos to speed it up
Closes: #543
Approved by: cgwalters
2016-12-07 21:47:58 +00:00
Jonathan Lebon
5bf7a040a5 libvm.sh: strengthen reboot
Unsurprisingly, rebooting machines that are running in OpenStack is
not as reliable or as fast as a local VM, which is what vmcheck was
originally written for and tested against.

Replace the:

  sleep 2 # give time for port to go down

which is rife with raciness, with a stronger boot_id-based check to
ensure we're in a new boot. Run "sync" before rebooting which sometimes
helps (though I didn't fully investigate why or whether it always helps,
there's probably something more subtle going on underneath). Increase
the timeout to 120s.

Closes: #543
Approved by: cgwalters
2016-12-07 21:47:58 +00:00
Colin Walters
9c11b057f7 Delete .travis.yml
It was always a stub, and we're definitely not using it.

Closes: #544
Approved by: jlebon
2016-12-07 21:21:37 +00:00
Colin Walters
6fb29775a3 [ASAN] libpriv: Squash misc small leaks
Nothing big here, but this seems to be the last of the ASAN hits
in the unit tests.

Closes: #539
Approved by: jlebon
2016-12-07 20:22:17 +00:00
Colin Walters
1209942fc2 [ASAN] unpacker: Clean up leak of rpmts
This was 800 bytes, which could add up.

Closes: #539
Approved by: jlebon
2016-12-07 20:22:17 +00:00
Colin Walters
977835fd41 unpacker: Fix logic error regression in previous commit
I spotted this warning go by in the build logs.  Let's make it
fatal from now on.

Closes: #536
Approved by: jlebon
2016-12-07 20:16:32 +00:00
Colin Walters
90fc84fb24 ci: --enable-new-name in clang test
So we have some coverage of this before merging.  I didn't make a
whole new context for this though, and it's only mildly useful
because we really want to test the daemon...but this is a start.

Closes: #535
Approved by: jlebon
2016-12-07 20:11:31 +00:00
Jonathan Lebon
56d11de141 redhat-ci: make vmcheck not required
There's issues right now with vmcheck since we switched it over to use
Fedora 25. I'll look into it, but in the meantime, let's make it not
required so we can merge in pending PRs.

Closes: #541
Approved by: cgwalters
2016-12-07 20:04:15 +00:00
Jonathan Lebon
68a9719696 libvm.sh: increase reboot timeout
Closes: #538
Approved by: cgwalters
2016-12-06 21:00:23 +00:00
Jonathan Lebon
c870ac58dc redhat-ci: fix Dockerfile
This was supposed to be part of the last PR (#537). But GitHub somehow
used an out-of-date PR on that branch when I opened the PR.

Closes: #538
Approved by: cgwalters
2016-12-06 21:00:23 +00:00
Jonathan Lebon
56aa6644cf redhat-ci: use dedicated build container image
Make a dedicated container image for building rpm-ostree so that we save
time and bandwidth on building in all the testsuites. Simplify the new
compose test.

Closes: #537
Approved by: cgwalters
2016-12-06 20:37:22 +00:00
Colin Walters
b81c0cdfda tests: Add ./tests/compose
Our current compose tests only use a synthetic `empty.rpm`, but
this really limits usefulness.

Let's make a test suite that requires an internet connection and
downloads Fedora RPMs and does "real" tree composes.

See the updated `tests/README.md` for more information.

This is still a WIP.

Closes: #531
Approved by: jlebon
2016-12-06 19:05:05 +00:00
Jonathan Lebon
c9e890e894 libdnf: bump to fix provides bug
https://github.com/rpm-software-management/libhif/issues/220

Closes: #534
Approved by: cgwalters
2016-12-06 17:27:19 +00:00
Colin Walters
0f4ef0d5a1 [ASAN] unpacker: Fix leaks of header/fi
We weren't freeing the Header struct in finalize if we happened
to own it.  Also define and use a cleanup for `rpmfi`.

Closes: #533
Approved by: jlebon
2016-12-06 16:47:26 +00:00
Colin Walters
12a6999d02 [ASAN] tree-wide: Squash a few string leaks
Just a few random string leaks.

Closes: #533
Approved by: jlebon
2016-12-06 16:47:26 +00:00
Colin Walters
975c0703c3 [ASAN] tree-wide: Clean up gvariant-related leaks
I kept thinking `^a&s` was sufficient for allocation-free reading
of `as`, but it's not, we need to free the outer buffer.  Other
minor cases were using `s` instead of `&s`.

Closes: #533
Approved by: jlebon
2016-12-06 16:47:26 +00:00
Colin Walters
7095c313d8 [ASAN] tree-wide: Clean up leaks of option context
Not all of these were leaks, but let's be consistent and use
`g_autoptr()` for `GOptionContext` everywhere.

Closes: #533
Approved by: jlebon
2016-12-06 16:47:26 +00:00
Colin Walters
88b8f1a655 unpacker: Don't error out layering fcaps-only binaries
In f25 now, `kubernetes-master` is changed to be owned by `root:root`
but still has file capabilities.  Our "non-root owned" check was
bombing out because we need to explicitly ignore "root" ownership.

(If there were no fcaps, then there's no override to check)

Closes: #532
Approved by: jlebon
2016-12-05 19:35:07 +00:00
Colin Walters
7149e9fee5 unpacker: Reorganize code for non-root checks
No functional changes, but I find this more legible - fewer
interacting conditionals.

Closes: #532
Approved by: jlebon
2016-12-05 19:35:07 +00:00
Colin Walters
74c43a0209 core: Error out if we're trying to do an empty install set
This is a partial fix for
https://github.com/projectatomic/rpm-ostree/issues/529

We could drop into this codepath via multiple ways (containers,
etc.), so it makes sense to have a last ditch error here, even
if we should really give an error earlier.

Closes: #530
Approved by: jlebon
2016-12-01 21:15:38 +00:00
Colin Walters
fec4b885ea docs: Document mutate-os-release
Since I actually forgot the semantics when working on
CentOS AH, and noticed we were missing docs.

Closes: #528
Approved by: jlebon
2016-11-30 17:19:57 +00:00
Colin Walters
351832f29e Fix install-langs support
We're looking at changing Atomic Host to use multiple locales (but not all);
See: https://bugzilla.redhat.com/show_bug.cgi?id=1186757

This revealed our `install-langs` support didn't really work.  Our
`treecompose-post.sh` was deleting the extraneous translations anyways,
which masked this.  And for other cases like workstations where
we drag along all the translations anyways, it was fine too.

There were two bugs:
 - In the keyfile spec it's `instlangs`
 - We were setting the macro a bit too late, it should be before
   `dnf_context_setup()`.

Closes: #525
Approved by: jlebon
2016-11-24 14:38:16 +00:00
Jonathan Lebon
05c59d11fa .redhat-ci.yml: run vmcheck tests
Use the newly added support for cluster tests in Red Hat CI to run the
vmcheck testsuite.

Closes: #524
Approved by: cgwalters
2016-11-22 17:15:20 +00:00
Jonathan Lebon
12c4b5b16f vmcheck: adapt for non-vagrant hosts
This is just the final bit required to make sure the vagrant and
non-vagrant paths can work happily together. It's mostly minor fixes,
though the most major change which also affects vagrant is that we now
sync to the root home dir, rather than ~vagrant.

Closes: #524
Approved by: cgwalters
2016-11-22 17:15:20 +00:00
Yunchih Chen
2d12d436ec hacking: add builddep command
Closes: #523
Approved by: cgwalters
2016-11-21 16:07:35 +00:00
Colin Walters
6074a6d728 daemon: Use new OSTree API to verify commits by remotes
Otherwise, we would show remotes using the new `gpgkeypath`
as unknown.

See: https://github.com/ostreedev/ostree/pull/575
See: https://github.com/ostreedev/ostree/issues/573
See: https://bugzilla.redhat.com/show_bug.cgi?id=1393545

Closes: #521
Approved by: jlebon
2016-11-17 16:32:11 +00:00
Colin Walters
2f9d586bdc Redo vmcheck to abstract over ssh-config, drop all building
My development environment is now using "pet" docker containers.
I use VMs for testing things that require that (like rpm-ostree).

This patch builds on work from @jlebon in
https://github.com/projectatomic/rpm-ostree/pull/509
to rework `vmcheck` such that it can work on any `ssh-config`.  By
default we expect this to be Vagrant.

However, I go a lot farther and delete the `vmbuild` code that was
trying to do builds in a container on the target VM.  I think this is
still worth pursuing at some point, but for now I think it's
reasonable to assume that the rpm-ostree developer audience uses Linux
as their host workstation and hence has containers.

(There's another important point here in that for developing lower
 level things like rpm-ostree, there's a strong push to make the VM
 disposable and not a pet)

Closes: #516
Approved by: jlebon
2016-11-16 18:14:23 +00:00
Colin Walters
1577afd85c vagrant: Switch to c7alpha, set up sshfs
The default C7 box gets old, I'd like to have Alpha be a focus
for development.

Let's use sshfs if available.

Closes: #516
Approved by: jlebon
2016-11-16 18:14:23 +00:00
Jonathan Lebon
313a832d7c vagrant: generate ssh-config
We further split libvm from vagrant. It no longer does 'vagrant
ssh-config'. Instead, it always assumes that an ssh-config is provided.
We now have complete separation of libvm from vagrant.

We change the ansible provisioner as follows:
  - Allow passing in a VAGRANT_BOX env var to override the default
    CentOS box.
  - No longer assume that the root user account is unlocked and has a
    valid 'vagrant' password. This worked for the centos box but isn't
    sure to work on every box. Instead, we now just run ansible as the
    default vagrant user, and during provisioning set up the root
    account and generate an ssh-config so that libvm can connect
    directly as root.
  - No longer build the buildimg during provisioning. This actually
    stopped working a while ago since the default rsync is disabled. We
    can just let the buildimg get created on the first compilation. In
    practice, the bigger issue isn't creating the buildimg, but being
    able to easily update the host and buildimg pkgs.

Closes: #516
Approved by: jlebon
2016-11-16 18:14:23 +00:00
Colin Walters
ae108e08a5 Add rpmostree.clientlayer metadata to derived commits
In the future we may add more commands that take as input commit
IDs.  However, we really want to distinguish between server and
client generated commits, as some of these operations won't
make sense for derived commits.

This changes the API to have callers say which type of commit
they're generating, which also fixes a FIXME, and helps get us
a bit closer to the "unified core".

Closes: #520
Approved by: jlebon
2016-11-15 17:11:11 +00:00