Commit Graph

482 Commits

Author SHA1 Message Date
Colin Walters
359ea9b58f tests: Tweak installed tests to deal with ASAN
We need to disable readdir-rand there too.

Closes: #622
Approved by: jlebon
2016-12-09 18:05:53 +00:00
Colin Walters
17f264a487 repo: Add unconfigured-state to remote config options
This is a migration from the origin version.  It's
nicer to have it in the remote, since that's what one
needs to change.  Then tools don't need to mess with
the origin file.o

In fact in this scenario one can keep the "media source" like
`file:///install/repo` or whatever, since conceptually that's where it
came from.  We're just providing a better error.

Closes: https://github.com/ostreedev/ostree/issues/626

Closes: #627
Approved by: jlebon
2016-12-09 17:46:54 +00:00
Colin Walters
66da1199f0 tests/keyfile-utils: Drop tests covering preconditions
The spam in stderr was bothering me, and further at some eventual
point in the future we want to annotate the functions with
`__attribute__((nonnull))` which would then cause tests like these to
become undefined behavior.

The coverage of this isn't worth the log spam basically.

Closes: #611
Approved by: jlebon
2016-12-06 14:50:08 +00:00
Colin Walters
e1c48c8adb [ASAN] tests: Cleanup all current remaining leaks
We now run fully through ASAN here.

Closes: #609
Approved by: jlebon
2016-12-05 15:20:56 +00:00
Colin Walters
7bb0ff46a4 Define and use cleanup helpers for libarchive
This should fix some of the ASAN leaks around libarchive usage,
and is generally better.

Closes: #609
Approved by: jlebon
2016-12-05 15:20:56 +00:00
Colin Walters
62b94635bc [ASAN] tests: Fix leaks
Just for cleaner sanitizer output.

Closes: #598
Approved by: jlebon
2016-11-30 18:51:26 +00:00
Simon McVittie
d1c7eba82c travis-ci: document parameter variables
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
3d6269ebfb travis-ci: Run make distcheck too
ostree is now actively using that mode.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
a1ab6b50f9 travis-ci: cat the test log after successful test runs
This lets us see which tests were skipped.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
db691b5520 travis-ci: Move helper function to before we start building anything
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
49b8ccd8cb travis-ci: Use a non-ostree-specific name for the Docker image
This reduces the diff when comparing these scripts with similar glue
in dbus or elsewhere.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
0473fb8523 travis-ci: put an explicit copyright/license on the scripts
This is deliberately permissive: a lot of it is generic, and I'm
using similar scripts in dbus.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
9026d5806f ci-install: add ci_distro
Otherwise, we'll fail (due to set -u) if this parameter variable isn't
passed.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
54655795c9 ci-build: consistently use yes/no for booleans, not yes/empty
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Jasper St. Pierre
fd6ba80d07 ostree-repo: Make the lock with a long-lasting FD
glnx_make_lock_file requires that the dfd passed in survives the
lifetime of the lock. Since dfd_iter.fd gets cleaned up after the
function returns, this isn't the case. dfd_iter.fd should be equivalent
to tmpdir_dfd, since we iter on ".", and that survives past the
function, so just use that instead.

Closes: #591
Approved by: cgwalters
2016-11-22 02:32:33 +00:00
Colin Walters
eb7ba645af [ASAN] tests: Fix some memleaks in libarchive importer
Caught by `-fsanitize=address`.

Closes: #587
Approved by: jlebon
2016-11-21 16:34:06 +00:00
Colin Walters
41ef2aeb38 pull: Do GPG verify commit objects when using deltas
The fact that we weren't doing this is at best an oversight, and
for some deployment models a security vulnerability.  Having both
`gpg-verify` and `gpg-verify-summary` shows that we were intending
them to be orthogonal/independent.

Lately I've been advocating moving towards pinned TLS instead of
gpg-signed summaries, and if we follow that path, performing GPG
verification of commit objects even if using deltas is more important,
as it provides an at-rest verifiable authenticity and integrity
mechanism.

Content providers which are signing their summary files and/or using
TLS (particularly pinned TLS) for transport should treat this as a
nice-to-have.  However, for providers which are serving content over
plain HTTP and relying on GPG, this is a critical update.

Closes: https://github.com/ostreedev/ostree/issues/517

Closes: #589
Approved by: jlebon
2016-11-21 15:55:09 +00:00
William Manley
0ee9e221be ostree commit: Fix combining trees with multiple --tree=ref arguments
You'd expect

    ostree commit --tree=ref=A --tree=ref=B

to produce a commit with the union of the trees given.  Instead you'd get
a commit with the contents of just the latter commit.  This was due to an
optimisation where we'd skip filling out the `files` and `subdirs`
members of the mtree, just filling in the metadata instead.  This backfires
becuase this same code relies on checking the `files` and `subdirs` members
itself to work out whether the mtree is empty.

This commit removes the optimisation, fixing the bug.  Maybe there's a way
to keep the optimisation and still fix the bug but it's not obvious to
me.

Closes: #581
Approved by: cgwalters
2016-11-17 14:45:55 +00:00
Colin Walters
24bf257ee9 lib: Add an API to GPG verify a commit given a remote
Conceptually we've been moving towards having our GPG verification
paths be per-remote.  The code internally supports this, but we
didn't expose an API to use it conveniently.

This came up when trying to add a new `gpgkeypath` option, since
right now rpm-ostree manually finds keyrings for the remote, and
hence it wasn't looking at the keypath, and said "Unknown key"
in status.

Adding an API fixes this nicely.

Closes: #576
Approved by: giuseppe
2016-11-17 11:33:41 +00:00
Colin Walters
f244c70277 Add "gpgkeypath" option to remotes
For Project Atomic, we already have RPM signatures which use files in
`/etc/pki/rpm-gpg`.  It's convenient to simply bind the OSTree remote
configuration to those file paths, rather than having duplicate key
data.

This does mean that we need to parse the files for verification, so we
end up importing them into the verifier's temporary keyring, which is
a bit ugly, but it's what other projects do.

Closes: https://github.com/ostreedev/ostree/issues/573

Closes: #575
Approved by: giuseppe
2016-11-17 09:44:07 +00:00
Alexander Larsson
bd45e7ac19 commit: Fix reading xattrs from OstreeRepoFile:s
When doing commit --tree=ref=XXX while at the same time applying some
form of modifier, ostree dies trying to read the xattrs using the
raw syscalls. We fix this by falling back to ostree_repo_file_get_xattrs()
in this case.

Also adds a testcase for this.

Closes: #577
Approved by: cgwalters
2016-11-16 22:30:33 +00:00
Colin Walters
37c07d2f1c pull: Add support for http-headers option
Some deployments may want to gate access to content based on things
like OAuth.  In this model, the client system would normally compute a
token and pass it to the server via an API.

We could theoretically support this in the remote config too, but
that'd be a bit weird for OAuth as the information is dynamic.
Therefore this cleans up the code a little bit to more clearly handle
the case that the fetcher is initialized from both remote config
data plus pull options.

Closes: #574
Approved by: giuseppe
2016-11-16 10:04:22 +00:00
Sjoerd Simons
6af8db6fc4 tests: Add test for the cookie jar handling
Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Colin Walters
2139f0e437 pull: Don't do deltas with --commit-metadata-only
We should just download the commit objects directly, as it's
obviously a lot more efficient than deltas.

I had to generate a summary file in more places in the tests,
since once created, it needs to be updated.

Closes: https://github.com/ostreedev/ostree/issues/528

Closes: #566
Approved by: jlebon
2016-11-04 16:04:23 +00:00
Sjoerd Simons
f0e493bf29 Filter bootloader supplied kernel cmdline options
Various bootloader add kernel commandline options dynamically, filter
these out when grabbing boot options from /proc/cmdline. Specifically
grub adds BOOT_IMAGE and systemd-boot adds initrd.

Closes: #560
Approved by: cgwalters
2016-11-01 18:00:37 +00:00
Simon McVittie
7091d288f7 Force C.UTF-8 or C locale for tests
Otherwise several tests fail, for example in this build done in a French
locale by Debian's reproducible builds initiative, to check whether
the resulting binaries are identical to what was produced in an
English locale:

<https://tests.reproducible-builds.org/debian/logs/unstable/amd64/ostree_2016.11-1.build2.log.gz>

(test-basic)
# error: Cannot write to repository: Permission non accordée
...
File 'error-message' doesn't match regexp 'Permission denied'

(test-help)
# Utilisation :
#   ostree [OPTION...] COMMAND
...
File 'out' doesn't match regexp '[Uu]sage'

(test-pull-metalink)
# error: Erreur à la ligne 1, caractère 1 : Le document doit commencer avec un élément (par ex. <book>)
...
File 'err.txt' doesn't match regexp 'Document must begin with an element'

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #558
Approved by: cgwalters
2016-10-29 18:12:42 +00:00
Colin Walters
b77edf24a3 tree-wide: Remove unused variables detected by CLang
CLang finds these, whereas GCC treats having
`__attribute__((cleanup))` as a use.

This obsoletes https://github.com/ostreedev/ostree/pull/411

Closes: #548
Approved by: jlebon
2016-10-27 17:02:01 +00:00
Colin Walters
82a4f56593 tests: Skip libarchive/selinux tests if in container without SELinux
I'm doing builds and `make check` inside a Docker container, with
selinux on as a build-time option, but no policy in the container.
This currently aborts.  Let's not do that.

(This type of thing is why installed tests are a better model)

Closes: #546
Approved by: jlebon
2016-10-27 16:48:34 +00:00
Alexander Larsson
d0e74cf3af Fix pruning of partial commits
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.

In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.

Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.

This fixes https://github.com/ostreedev/ostree/issues/541

Closes: #542
Approved by: cgwalters
2016-10-24 17:48:19 +00:00
Alexander Larsson
ee484697cd pull: Support multiple specifications of --subpath
I need this in flatpak to avoid doing multiple pulls when doing
locale subsetting.

Closes: #523
Approved by: cgwalters
2016-10-11 16:41:03 +00:00
Jonathan Lebon
12e916466c static-delta: add some error handling
We make _ostree_parse_delta_name() a bit more defensive since it handles
user input.

Closes: #504

Closes: #505
Approved by: cgwalters
2016-09-09 19:06:11 +00:00
Colin Walters
845dc65196 repo: Revert default timestamp from 1 back to 0
Quoting Dan Nicholson in

  <https://github.com/ostreedev/ostree/pull/330#issuecomment-245499099>

  mtime of 0 has been the semantics of ostree deployments from basically
  the beginning of the project. We (and others, see
  flatpak/flatpak@b5204c9) rely on that fact when generating trees.

  In particular, this affects caches that use the mtime of the
  associated file or directory to determine if the cache is valid. By
  arbitrarily changing the mtime of the files to something else, all
  the caches we setup in the build are now invalidated. Preseeding
  caches is really important to the user experience as it avoids
  having the user wait while they're regenerated on first run.

  Now, we could change our build infrastructure to preset all the
  mtimes to 1 to match this change, but what does that do for our
  existing users who are on an ostree that deploys with mtimes of 0?
  We could just revert this change at Endless (and the associated one
  in Flatpak), and that would be fine for our users. However, if we
  point non-Endless users to our apps, they'll have the great
  experience of waiting 10 seconds the first time they launch it while
  the fontconfig cache is rebuilt unnecessarily.

Closes: #495
Approved by: jlebon
2016-09-08 13:35:59 +00:00
William Manley
2aacc6912b ostree-prepare-root: Fix running with musl
musl libc's implementation of `realpath` works by opening the path and then
doing a lookup in `/proc/self/fd` to find the canonical path.  This fails
if `/proc` is not mounted.  This causes problems for us if
`ostree-prepare-root` is `init` as `/proc` won't be mounted.

We have to mount `/proc` anyway for `/proc/cmdline` so this fix just
expands the scope over which `/proc` is mounted to include both our
`realpath` calls.

See also:

* http://www.openwall.com/lists/musl/2016/06/08/2 and
* http://git.musl-libc.org/cgit/musl/tree/src/misc/realpath.c?id=e738b8cbe64b6dd3ed9f47b6d4cd7eb2c422b38d

Closes: #485
Approved by: cgwalters
2016-09-01 20:17:58 +00:00
William Manley
598e3240ff switchroot: Fix test-switchroot now autotools can build static
This test previously depended on manually building ostree-prepare-root.
Since 42dab85 we've been able to build static binaries with the usual
autotools build-system.  This change reflects the fact that
`ostree-prepare-root` is built into $srcdir rather than `src/switchroot`
where I was building manually.

This test now passes with `./configure --with-static-compiler=gcc` (glibc)
but still fails with `./configure --with-static-compiler=musl-gcc` (musl).

Closes: #485
Approved by: cgwalters
2016-09-01 20:17:58 +00:00
Jonathan Lebon
661e4636f5 tests: add tests for contenturl and mirrorlist
Closes: #469
Approved by: cgwalters
2016-08-31 16:52:12 +00:00
Jonathan Lebon
c4c030cc79 libtest: add has_gpgme() helper function
Closes: #469
Approved by: cgwalters
2016-08-31 16:52:12 +00:00
Gatis Paeglis
b6ec7526b5 u-boot: Merge ostree's and systems uEnv.txt
This is a proper fix for:
https://bugzilla.gnome.org/show_bug.cgi?id=755787

With this patch, an admin (system builder) can now:

1) Edit /usr/lib/ostree-boot/uEnv.txt
2) Deploy the new tree. OSTree will append system's uEnv.txt
   to the OSTree's managed uEnv.txt (loader/uEnv.txt).

It is common for u-boot systems to read in an extra env
from external /uEnv.txt. The same file OSTree uses to pass
in its env. With this patch /uEnv.txt now contains OSTree's
env + custom env added by system builders.

Closes: #466
Approved by: cgwalters
2016-08-27 13:11:22 +00:00
Colin Walters
3ef4cc2e5b lib: Add an API to list only "our" objects, fix prune to use it
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail.  There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.

Closes: https://github.com/ostreedev/ostree/issues/467

Closes: #471
Approved by: jlebon
2016-08-25 20:01:37 +00:00
Colin Walters
3469161e41 test-rofiles-fuse: Actually check out via hardlinks
The test suite was actually doing something before, just
not quite what I intended.  Without `-U` for bare-user checkouts
we end up doing a copy.

Now, a future commit will change how rofiles work, which would cause
the test suite to permit inplace mutation for non-hardlinked files.
So let's ensure they *are* hardlinked.

Closes: #462
Approved by: jlebon
2016-08-16 21:22:28 +00:00
Dan Nicholson
7ed2f1e5af tests: Test that local pulls do not use deltas
Pulls from local repos now default to disabling static deltas so that
objects are copied. To check this is the case, see if the object files
are hardlinked after pulling.

Closes: #447

Closes: #448
Approved by: cgwalters
2016-08-10 10:54:46 +00:00
Dan Nicholson
2a810fbef9 tests: Ensure deltas for pulling when needed
When testing pulling of deltas, use the new --require-static-deltas
option to pull-local to ensure that deltas are actually used. To support
the require-static-deltas mode, the summary in the remote repo must be
generated.

Closes: #447

Closes: #448
Approved by: cgwalters
2016-08-10 10:54:46 +00:00
Colin Walters
56a922280f tests: Port sysroot.js away from libgsystem
My case-sensitive `git grep` didn't find this one originally,
but the Travis+Debian setup was configured to run it.

Closes: #451
Approved by: jlebon
2016-08-09 19:01:35 +00:00
Colin Walters
a29bb0ab6b Final excision of libgsystem dependency
Lots and lots of preparation led to this moment - when nothing
apparent changes for users!  Woo!

But seriously, having the extra dependency is a minor annoyance, and
in the big picture I think the libgsystem idea was wrong - we need to
land things in GLib, and use git submodules for API-unstable or
Linux-specific sharing.  For a lot of OSTree, the libgsystem `GFile*`
orientation was also wrong, we really want fd-relative.

Closes: #444
Approved by: jlebon
2016-08-09 15:36:09 +00:00
Colin Walters
fd9536bb19 prune: Retain the tip of each ref even with date pruning
I hit an error with [CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel)
where we were doing time-based pruning.  `ostree summary -u` started failing,
and it took me a bit to realize it was because we were pruning
even the tip of old branches, which I was not at all expecting,
and I don't think users will too.

Perhaps in the future we could add some sort of --prune-ref-tips or
something if people wanted it, but I doubt it.

Closes: #445
Approved by: jlebon
2016-08-09 15:12:35 +00:00
Simon McVittie
46e118ee84 travis: run the test suite on various distributions
.travis.yml is obviously still Travis-specific, but tests/ci-* are
designed to be shareable with other CI environments if there is interest
in doing so.

At the moment I'm only testing on Debian and Ubuntu. In principle we
could try a non-Debian-derived Docker container such as Fedora or CentOS
inside travis-ci's Ubuntu environment, similar to what I'm doing
for Debian, but I don't know the correct setup commands to use there.

Closes: #438
Approved by: cgwalters
2016-08-05 23:58:19 +00:00
Simon McVittie
0de51bfc45 Skip tests that use whiteouts under Docker/aufs
ostree's naming convention for whiteouts is similar to what is
done in aufs, which means we can't compose the trees to test this
feature under Docker with the aufs storage driver, as used on
travis-ci.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #437
Approved by: cgwalters
2016-08-05 20:32:57 +00:00
William Manley
52acb9d8e3 ostree-prepare-root: Add test that overlayfs over /usr works.
Closes: #403
Approved by: cgwalters
2016-08-02 19:07:25 +00:00
William Manley
485a374b21 ostree-prepare-root: Use pivot_root if real sysroot is already mounted at /
This allows ostree-prepare-root outside of the initramfs context where the
real rootfs is already mounted at /.  We can't use `mount --move` in this
case because we would be trying to move / into a subdirectory of itself.

Closes: #403
Approved by: cgwalters
2016-08-02 19:07:25 +00:00
William Manley
94e50ca5aa tests: Add basic tests for ostree-prepare-root
These tests use unshare and mount to prepare a fake initrd/early boot
directory structure so we can then test ostree-prepare-root.

Things that are tested:

* Running in an initrd environment
* Running without initrd
* /var and /sysroot being mounted correctly
* /usr being mounted read-only

Things not tested (yet):

* Running as init - this could be accomplished by unsharing the pid
  namespace too.
* mounting/unmounting `/proc` if `/proc/cmdline` isn't available
* Persistent overlayfs for `/usr`
* Probably other things

The tests are basic but can be extended in the future as we do more work
on `ostree-prepare-root`.

These tests must be run as root as they require the ability to `mount`
and to `unshare` the mount namespace. Perhaps in the future we can use
user namespaces for this test once they are more widely available.

Closes: #403
Approved by: cgwalters
2016-08-02 19:07:25 +00:00
Colin Walters
5334758ba7 repo: Make ostree_repo_create() nonfatal on existing repos
In general we want to support "idempotentcy" or "state
synchronization" across interruption.  If a repo is only partially
created due to a crash or whatever, it's hard for a user to know that.
Let's just make `ostree_repo_create()` idempotent. Since all we're
doing is a set of `mkdirat()` invocations, it's quite simple.

This also involved porting to fd-relative, which IMO makes the
code a lot clearer.

Closes: #422
Approved by: 14rcole
2016-08-01 15:12:14 +00:00