Commit Graph

641 Commits

Author SHA1 Message Date
Colin Walters
5c940987e7 Add support for more selective pruning
There are use cases for having a single repo with branches
with different lifecycles; a simple example of what I was
trying to do in CentOS Atomic Host work is have "stable"
and "devel" branches, were we want to prune devel, but
retain *all* of stable.

This patch is split into two parts - first we add a low level "delete all
objects not in this set" API, and change the current prune API
to use this.

Next, we move more logic into the "ostree prune" command. This paves the way for
demonstrating how more sophisticated algorithms/logic could be developed outside
of the ostree core.

Also, the --keep-younger-than logic already lived in the commandline, so it
makes sense to keep extending it there.

Closes: https://github.com/ostreedev/ostree/issues/604

Closes: #646
Approved by: jlebon
2017-01-19 16:28:00 +00:00
Simon McVittie
9d94fc40c8 Make corrupt-repo-ref.js executable
Debian's Lintian packaging consistency check complains that it isn't
executable but has a #! line. In fact it's reasonable to run this
script directly, so make it executable, and put it in a _scripts
variable so it will be installed executable.

Closes: #652
Approved by: cgwalters
2017-01-19 13:54:59 +00:00
Simon McVittie
9a3f82caae Sourced test snippets: remove shebang and make non-executable
They are installed non-executable, which makes Debian's Lintian
packaging consistency check complain that #! is only useful
in executable scripts. But in fact they are not useful to execute
directly (they rely on setup being done in the script that sources
them), so just chmod them -x.

Closes: #652
Approved by: cgwalters
2017-01-19 13:54:59 +00:00
Simon McVittie
01fb30b839 Fix TAP syntax in test-basic-user.sh, and run it
In its initial commit, Alexander Larsson wrote

    This works standalone, but unfortunately it breaks in
    gnome-desktop-testing-runner as /tmp doesn't support
    xattrs, so it is not installed atm.

but we now (a) use /var/tmp, and (b) explicitly skip the test if
xattr support is unavailable. So it should be OK to run now.

Closes: #652
Approved by: cgwalters
2017-01-19 13:54:59 +00:00
Colin Walters
b260fa764c libtest: Enable web server logs
Now that we're daemonizing, it's useful to have the logs.  I
wanted this while debugging cookies.

Closes: #651
Approved by: giuseppe
2017-01-19 10:47:15 +00:00
Colin Walters
686f91062f tests: Loosen error regexp
libcurl AFAICS doesn't have an API to convert HTTP code ➡️ error
string, so let's make the test regexp operate on both.

Closes: #651
Approved by: giuseppe
2017-01-19 10:47:15 +00:00
Colin Walters
56891f9d48 tests: Don't inject newline in URL
It turns out libsoup strips all whitespace even *inside* a URL. We could do that
for libcurl too but...really, people shouldn't do that. In this test we were
adding the trailing newline into the URL. If someone complains who is using the
libcurl code we can deal with it then.

Closes: #651
Approved by: giuseppe
2017-01-19 10:47:15 +00:00
Colin Walters
2f71136eec tests: Alias assert_not_reached() -> fatal()
We had a lot of copies of the "echo something 1>&2; exit 1" code even though
`assert_not_reached()` was it.  Hence, I think we need a shorter alias for that.

Doing this particularly since I noticed a missing `1` in an `exit 1` call in the
rpm-ostree copy of this.

Closes: #648
Approved by: jlebon
2017-01-18 14:28:29 +00:00
Colin Walters
ced22f6c9b Split trivial-httpd into separate binary
Working on the libcurl backend, I hit the issue that the trivial-httpd program
depends on libsoup. I briefly considered having two versions, but libcurl is
client only, and moreover trivial-httpd is no longer trivial - it has various
features which are used by the test suite extensively.

Hence, what we'll do is build it as a separate binary which links to libsoup,
and use it during the tests. We *also* currently still provide `ostree
trivial-httpd` since some things use it like `rpm-ostree-toolbox` and the
Cockpit tests.

After those are ported to use some other webserver, I plan to add a build-time
option to drop it.

Closes: #636
Approved by: jlebon
2017-01-04 16:32:11 +00:00
Colin Walters
359ea9b58f tests: Tweak installed tests to deal with ASAN
We need to disable readdir-rand there too.

Closes: #622
Approved by: jlebon
2016-12-09 18:05:53 +00:00
Colin Walters
17f264a487 repo: Add unconfigured-state to remote config options
This is a migration from the origin version.  It's
nicer to have it in the remote, since that's what one
needs to change.  Then tools don't need to mess with
the origin file.o

In fact in this scenario one can keep the "media source" like
`file:///install/repo` or whatever, since conceptually that's where it
came from.  We're just providing a better error.

Closes: https://github.com/ostreedev/ostree/issues/626

Closes: #627
Approved by: jlebon
2016-12-09 17:46:54 +00:00
Colin Walters
66da1199f0 tests/keyfile-utils: Drop tests covering preconditions
The spam in stderr was bothering me, and further at some eventual
point in the future we want to annotate the functions with
`__attribute__((nonnull))` which would then cause tests like these to
become undefined behavior.

The coverage of this isn't worth the log spam basically.

Closes: #611
Approved by: jlebon
2016-12-06 14:50:08 +00:00
Colin Walters
e1c48c8adb [ASAN] tests: Cleanup all current remaining leaks
We now run fully through ASAN here.

Closes: #609
Approved by: jlebon
2016-12-05 15:20:56 +00:00
Colin Walters
7bb0ff46a4 Define and use cleanup helpers for libarchive
This should fix some of the ASAN leaks around libarchive usage,
and is generally better.

Closes: #609
Approved by: jlebon
2016-12-05 15:20:56 +00:00
Colin Walters
62b94635bc [ASAN] tests: Fix leaks
Just for cleaner sanitizer output.

Closes: #598
Approved by: jlebon
2016-11-30 18:51:26 +00:00
Simon McVittie
d1c7eba82c travis-ci: document parameter variables
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
3d6269ebfb travis-ci: Run make distcheck too
ostree is now actively using that mode.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
a1ab6b50f9 travis-ci: cat the test log after successful test runs
This lets us see which tests were skipped.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
db691b5520 travis-ci: Move helper function to before we start building anything
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
49b8ccd8cb travis-ci: Use a non-ostree-specific name for the Docker image
This reduces the diff when comparing these scripts with similar glue
in dbus or elsewhere.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
0473fb8523 travis-ci: put an explicit copyright/license on the scripts
This is deliberately permissive: a lot of it is generic, and I'm
using similar scripts in dbus.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
9026d5806f ci-install: add ci_distro
Otherwise, we'll fail (due to set -u) if this parameter variable isn't
passed.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
54655795c9 ci-build: consistently use yes/no for booleans, not yes/empty
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Jasper St. Pierre
fd6ba80d07 ostree-repo: Make the lock with a long-lasting FD
glnx_make_lock_file requires that the dfd passed in survives the
lifetime of the lock. Since dfd_iter.fd gets cleaned up after the
function returns, this isn't the case. dfd_iter.fd should be equivalent
to tmpdir_dfd, since we iter on ".", and that survives past the
function, so just use that instead.

Closes: #591
Approved by: cgwalters
2016-11-22 02:32:33 +00:00
Colin Walters
eb7ba645af [ASAN] tests: Fix some memleaks in libarchive importer
Caught by `-fsanitize=address`.

Closes: #587
Approved by: jlebon
2016-11-21 16:34:06 +00:00
Colin Walters
41ef2aeb38 pull: Do GPG verify commit objects when using deltas
The fact that we weren't doing this is at best an oversight, and
for some deployment models a security vulnerability.  Having both
`gpg-verify` and `gpg-verify-summary` shows that we were intending
them to be orthogonal/independent.

Lately I've been advocating moving towards pinned TLS instead of
gpg-signed summaries, and if we follow that path, performing GPG
verification of commit objects even if using deltas is more important,
as it provides an at-rest verifiable authenticity and integrity
mechanism.

Content providers which are signing their summary files and/or using
TLS (particularly pinned TLS) for transport should treat this as a
nice-to-have.  However, for providers which are serving content over
plain HTTP and relying on GPG, this is a critical update.

Closes: https://github.com/ostreedev/ostree/issues/517

Closes: #589
Approved by: jlebon
2016-11-21 15:55:09 +00:00
William Manley
0ee9e221be ostree commit: Fix combining trees with multiple --tree=ref arguments
You'd expect

    ostree commit --tree=ref=A --tree=ref=B

to produce a commit with the union of the trees given.  Instead you'd get
a commit with the contents of just the latter commit.  This was due to an
optimisation where we'd skip filling out the `files` and `subdirs`
members of the mtree, just filling in the metadata instead.  This backfires
becuase this same code relies on checking the `files` and `subdirs` members
itself to work out whether the mtree is empty.

This commit removes the optimisation, fixing the bug.  Maybe there's a way
to keep the optimisation and still fix the bug but it's not obvious to
me.

Closes: #581
Approved by: cgwalters
2016-11-17 14:45:55 +00:00
Colin Walters
24bf257ee9 lib: Add an API to GPG verify a commit given a remote
Conceptually we've been moving towards having our GPG verification
paths be per-remote.  The code internally supports this, but we
didn't expose an API to use it conveniently.

This came up when trying to add a new `gpgkeypath` option, since
right now rpm-ostree manually finds keyrings for the remote, and
hence it wasn't looking at the keypath, and said "Unknown key"
in status.

Adding an API fixes this nicely.

Closes: #576
Approved by: giuseppe
2016-11-17 11:33:41 +00:00
Colin Walters
f244c70277 Add "gpgkeypath" option to remotes
For Project Atomic, we already have RPM signatures which use files in
`/etc/pki/rpm-gpg`.  It's convenient to simply bind the OSTree remote
configuration to those file paths, rather than having duplicate key
data.

This does mean that we need to parse the files for verification, so we
end up importing them into the verifier's temporary keyring, which is
a bit ugly, but it's what other projects do.

Closes: https://github.com/ostreedev/ostree/issues/573

Closes: #575
Approved by: giuseppe
2016-11-17 09:44:07 +00:00
Alexander Larsson
bd45e7ac19 commit: Fix reading xattrs from OstreeRepoFile:s
When doing commit --tree=ref=XXX while at the same time applying some
form of modifier, ostree dies trying to read the xattrs using the
raw syscalls. We fix this by falling back to ostree_repo_file_get_xattrs()
in this case.

Also adds a testcase for this.

Closes: #577
Approved by: cgwalters
2016-11-16 22:30:33 +00:00
Colin Walters
37c07d2f1c pull: Add support for http-headers option
Some deployments may want to gate access to content based on things
like OAuth.  In this model, the client system would normally compute a
token and pass it to the server via an API.

We could theoretically support this in the remote config too, but
that'd be a bit weird for OAuth as the information is dynamic.
Therefore this cleans up the code a little bit to more clearly handle
the case that the fetcher is initialized from both remote config
data plus pull options.

Closes: #574
Approved by: giuseppe
2016-11-16 10:04:22 +00:00
Sjoerd Simons
6af8db6fc4 tests: Add test for the cookie jar handling
Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Colin Walters
2139f0e437 pull: Don't do deltas with --commit-metadata-only
We should just download the commit objects directly, as it's
obviously a lot more efficient than deltas.

I had to generate a summary file in more places in the tests,
since once created, it needs to be updated.

Closes: https://github.com/ostreedev/ostree/issues/528

Closes: #566
Approved by: jlebon
2016-11-04 16:04:23 +00:00
Sjoerd Simons
f0e493bf29 Filter bootloader supplied kernel cmdline options
Various bootloader add kernel commandline options dynamically, filter
these out when grabbing boot options from /proc/cmdline. Specifically
grub adds BOOT_IMAGE and systemd-boot adds initrd.

Closes: #560
Approved by: cgwalters
2016-11-01 18:00:37 +00:00
Simon McVittie
7091d288f7 Force C.UTF-8 or C locale for tests
Otherwise several tests fail, for example in this build done in a French
locale by Debian's reproducible builds initiative, to check whether
the resulting binaries are identical to what was produced in an
English locale:

<https://tests.reproducible-builds.org/debian/logs/unstable/amd64/ostree_2016.11-1.build2.log.gz>

(test-basic)
# error: Cannot write to repository: Permission non accordée
...
File 'error-message' doesn't match regexp 'Permission denied'

(test-help)
# Utilisation :
#   ostree [OPTION...] COMMAND
...
File 'out' doesn't match regexp '[Uu]sage'

(test-pull-metalink)
# error: Erreur à la ligne 1, caractère 1 : Le document doit commencer avec un élément (par ex. <book>)
...
File 'err.txt' doesn't match regexp 'Document must begin with an element'

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #558
Approved by: cgwalters
2016-10-29 18:12:42 +00:00
Colin Walters
b77edf24a3 tree-wide: Remove unused variables detected by CLang
CLang finds these, whereas GCC treats having
`__attribute__((cleanup))` as a use.

This obsoletes https://github.com/ostreedev/ostree/pull/411

Closes: #548
Approved by: jlebon
2016-10-27 17:02:01 +00:00
Colin Walters
82a4f56593 tests: Skip libarchive/selinux tests if in container without SELinux
I'm doing builds and `make check` inside a Docker container, with
selinux on as a build-time option, but no policy in the container.
This currently aborts.  Let's not do that.

(This type of thing is why installed tests are a better model)

Closes: #546
Approved by: jlebon
2016-10-27 16:48:34 +00:00
Alexander Larsson
d0e74cf3af Fix pruning of partial commits
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.

In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.

Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.

This fixes https://github.com/ostreedev/ostree/issues/541

Closes: #542
Approved by: cgwalters
2016-10-24 17:48:19 +00:00
Alexander Larsson
ee484697cd pull: Support multiple specifications of --subpath
I need this in flatpak to avoid doing multiple pulls when doing
locale subsetting.

Closes: #523
Approved by: cgwalters
2016-10-11 16:41:03 +00:00
Jonathan Lebon
12e916466c static-delta: add some error handling
We make _ostree_parse_delta_name() a bit more defensive since it handles
user input.

Closes: #504

Closes: #505
Approved by: cgwalters
2016-09-09 19:06:11 +00:00
Colin Walters
845dc65196 repo: Revert default timestamp from 1 back to 0
Quoting Dan Nicholson in

  <https://github.com/ostreedev/ostree/pull/330#issuecomment-245499099>

  mtime of 0 has been the semantics of ostree deployments from basically
  the beginning of the project. We (and others, see
  flatpak/flatpak@b5204c9) rely on that fact when generating trees.

  In particular, this affects caches that use the mtime of the
  associated file or directory to determine if the cache is valid. By
  arbitrarily changing the mtime of the files to something else, all
  the caches we setup in the build are now invalidated. Preseeding
  caches is really important to the user experience as it avoids
  having the user wait while they're regenerated on first run.

  Now, we could change our build infrastructure to preset all the
  mtimes to 1 to match this change, but what does that do for our
  existing users who are on an ostree that deploys with mtimes of 0?
  We could just revert this change at Endless (and the associated one
  in Flatpak), and that would be fine for our users. However, if we
  point non-Endless users to our apps, they'll have the great
  experience of waiting 10 seconds the first time they launch it while
  the fontconfig cache is rebuilt unnecessarily.

Closes: #495
Approved by: jlebon
2016-09-08 13:35:59 +00:00
William Manley
2aacc6912b ostree-prepare-root: Fix running with musl
musl libc's implementation of `realpath` works by opening the path and then
doing a lookup in `/proc/self/fd` to find the canonical path.  This fails
if `/proc` is not mounted.  This causes problems for us if
`ostree-prepare-root` is `init` as `/proc` won't be mounted.

We have to mount `/proc` anyway for `/proc/cmdline` so this fix just
expands the scope over which `/proc` is mounted to include both our
`realpath` calls.

See also:

* http://www.openwall.com/lists/musl/2016/06/08/2 and
* http://git.musl-libc.org/cgit/musl/tree/src/misc/realpath.c?id=e738b8cbe64b6dd3ed9f47b6d4cd7eb2c422b38d

Closes: #485
Approved by: cgwalters
2016-09-01 20:17:58 +00:00
William Manley
598e3240ff switchroot: Fix test-switchroot now autotools can build static
This test previously depended on manually building ostree-prepare-root.
Since 42dab85 we've been able to build static binaries with the usual
autotools build-system.  This change reflects the fact that
`ostree-prepare-root` is built into $srcdir rather than `src/switchroot`
where I was building manually.

This test now passes with `./configure --with-static-compiler=gcc` (glibc)
but still fails with `./configure --with-static-compiler=musl-gcc` (musl).

Closes: #485
Approved by: cgwalters
2016-09-01 20:17:58 +00:00
Jonathan Lebon
661e4636f5 tests: add tests for contenturl and mirrorlist
Closes: #469
Approved by: cgwalters
2016-08-31 16:52:12 +00:00
Jonathan Lebon
c4c030cc79 libtest: add has_gpgme() helper function
Closes: #469
Approved by: cgwalters
2016-08-31 16:52:12 +00:00
Gatis Paeglis
b6ec7526b5 u-boot: Merge ostree's and systems uEnv.txt
This is a proper fix for:
https://bugzilla.gnome.org/show_bug.cgi?id=755787

With this patch, an admin (system builder) can now:

1) Edit /usr/lib/ostree-boot/uEnv.txt
2) Deploy the new tree. OSTree will append system's uEnv.txt
   to the OSTree's managed uEnv.txt (loader/uEnv.txt).

It is common for u-boot systems to read in an extra env
from external /uEnv.txt. The same file OSTree uses to pass
in its env. With this patch /uEnv.txt now contains OSTree's
env + custom env added by system builders.

Closes: #466
Approved by: cgwalters
2016-08-27 13:11:22 +00:00
Colin Walters
3ef4cc2e5b lib: Add an API to list only "our" objects, fix prune to use it
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail.  There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.

Closes: https://github.com/ostreedev/ostree/issues/467

Closes: #471
Approved by: jlebon
2016-08-25 20:01:37 +00:00
Colin Walters
3469161e41 test-rofiles-fuse: Actually check out via hardlinks
The test suite was actually doing something before, just
not quite what I intended.  Without `-U` for bare-user checkouts
we end up doing a copy.

Now, a future commit will change how rofiles work, which would cause
the test suite to permit inplace mutation for non-hardlinked files.
So let's ensure they *are* hardlinked.

Closes: #462
Approved by: jlebon
2016-08-16 21:22:28 +00:00
Dan Nicholson
7ed2f1e5af tests: Test that local pulls do not use deltas
Pulls from local repos now default to disabling static deltas so that
objects are copied. To check this is the case, see if the object files
are hardlinked after pulling.

Closes: #447

Closes: #448
Approved by: cgwalters
2016-08-10 10:54:46 +00:00
Dan Nicholson
2a810fbef9 tests: Ensure deltas for pulling when needed
When testing pulling of deltas, use the new --require-static-deltas
option to pull-local to ensure that deltas are actually used. To support
the require-static-deltas mode, the summary in the remote repo must be
generated.

Closes: #447

Closes: #448
Approved by: cgwalters
2016-08-10 10:54:46 +00:00
Colin Walters
56a922280f tests: Port sysroot.js away from libgsystem
My case-sensitive `git grep` didn't find this one originally,
but the Travis+Debian setup was configured to run it.

Closes: #451
Approved by: jlebon
2016-08-09 19:01:35 +00:00
Colin Walters
a29bb0ab6b Final excision of libgsystem dependency
Lots and lots of preparation led to this moment - when nothing
apparent changes for users!  Woo!

But seriously, having the extra dependency is a minor annoyance, and
in the big picture I think the libgsystem idea was wrong - we need to
land things in GLib, and use git submodules for API-unstable or
Linux-specific sharing.  For a lot of OSTree, the libgsystem `GFile*`
orientation was also wrong, we really want fd-relative.

Closes: #444
Approved by: jlebon
2016-08-09 15:36:09 +00:00
Colin Walters
fd9536bb19 prune: Retain the tip of each ref even with date pruning
I hit an error with [CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel)
where we were doing time-based pruning.  `ostree summary -u` started failing,
and it took me a bit to realize it was because we were pruning
even the tip of old branches, which I was not at all expecting,
and I don't think users will too.

Perhaps in the future we could add some sort of --prune-ref-tips or
something if people wanted it, but I doubt it.

Closes: #445
Approved by: jlebon
2016-08-09 15:12:35 +00:00
Simon McVittie
46e118ee84 travis: run the test suite on various distributions
.travis.yml is obviously still Travis-specific, but tests/ci-* are
designed to be shareable with other CI environments if there is interest
in doing so.

At the moment I'm only testing on Debian and Ubuntu. In principle we
could try a non-Debian-derived Docker container such as Fedora or CentOS
inside travis-ci's Ubuntu environment, similar to what I'm doing
for Debian, but I don't know the correct setup commands to use there.

Closes: #438
Approved by: cgwalters
2016-08-05 23:58:19 +00:00
Simon McVittie
0de51bfc45 Skip tests that use whiteouts under Docker/aufs
ostree's naming convention for whiteouts is similar to what is
done in aufs, which means we can't compose the trees to test this
feature under Docker with the aufs storage driver, as used on
travis-ci.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #437
Approved by: cgwalters
2016-08-05 20:32:57 +00:00
William Manley
52acb9d8e3 ostree-prepare-root: Add test that overlayfs over /usr works.
Closes: #403
Approved by: cgwalters
2016-08-02 19:07:25 +00:00
William Manley
485a374b21 ostree-prepare-root: Use pivot_root if real sysroot is already mounted at /
This allows ostree-prepare-root outside of the initramfs context where the
real rootfs is already mounted at /.  We can't use `mount --move` in this
case because we would be trying to move / into a subdirectory of itself.

Closes: #403
Approved by: cgwalters
2016-08-02 19:07:25 +00:00
William Manley
94e50ca5aa tests: Add basic tests for ostree-prepare-root
These tests use unshare and mount to prepare a fake initrd/early boot
directory structure so we can then test ostree-prepare-root.

Things that are tested:

* Running in an initrd environment
* Running without initrd
* /var and /sysroot being mounted correctly
* /usr being mounted read-only

Things not tested (yet):

* Running as init - this could be accomplished by unsharing the pid
  namespace too.
* mounting/unmounting `/proc` if `/proc/cmdline` isn't available
* Persistent overlayfs for `/usr`
* Probably other things

The tests are basic but can be extended in the future as we do more work
on `ostree-prepare-root`.

These tests must be run as root as they require the ability to `mount`
and to `unshare` the mount namespace. Perhaps in the future we can use
user namespaces for this test once they are more widely available.

Closes: #403
Approved by: cgwalters
2016-08-02 19:07:25 +00:00
Colin Walters
5334758ba7 repo: Make ostree_repo_create() nonfatal on existing repos
In general we want to support "idempotentcy" or "state
synchronization" across interruption.  If a repo is only partially
created due to a crash or whatever, it's hard for a user to know that.
Let's just make `ostree_repo_create()` idempotent. Since all we're
doing is a set of `mkdirat()` invocations, it's quite simple.

This also involved porting to fd-relative, which IMO makes the
code a lot clearer.

Closes: #422
Approved by: 14rcole
2016-08-01 15:12:14 +00:00
Dan Nicholson
a312d8fd8c tests: Add test for delta with empty parts
Closes: #420
Approved by: cgwalters
2016-07-31 00:01:52 +00:00
Colin Walters
402020b4ca glib.supp: Suppress g_task -> thread leaks
There are variants of this, but right now because `GTask` has its
own pool which isn't necessarily cleaned up on exit, we need
to suppress all of this.

Closes: #410
Approved by: giuseppe
2016-07-28 10:10:17 +00:00
Colin Walters
544365f834 glib.supp: Suppress worker context -> thread bits
The worker context isn't cleaned up now.

Closes: #410
Approved by: giuseppe
2016-07-28 10:10:17 +00:00
Colin Walters
d13cf37b3f glib.supp: Suppress some dynamic type registrations too
These mirror the static ones, and we need the dynamic versions for
glib-networking -> gnutls stuff.

Closes: #410
Approved by: giuseppe
2016-07-28 10:10:17 +00:00
Colin Walters
bac1fa1839 pull: Don't execute static deltas when mirroring
We don't presently support this, since the static delta code assumes
it can just `mmap()` file objects.  We could at some point implement
this, but for now just skip executing deltas when doing
`archive -> archive` mirroring.

I noticed this when trying to mirror a repo in Jenkins in
[CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel).

Closes: #408
Approved by: giuseppe
2016-07-25 06:50:51 +00:00
Colin Walters
87c9e227cb Revert "tests/libtest.sh: Print non-matching file on failure"
This reverts commit 71301d1824.

I was confused by the fact that the non-matching content was
empty and forgot we already did print it.

Reported-by: smcv

Closes: #390
Approved by: jlebon
2016-07-11 19:30:31 +00:00
Colin Walters
35b4131bd7 tests: Fix karg tests on ostree-booted system
https://github.com/ostreedev/ostree/pull/372 caused these tests to
start failing when the host system is managed using ostree - since the
tests *do* replace the `ostree=` kernel argument.

Closes: #384
Approved by: cgwalters
2016-07-11 16:46:07 +00:00
Colin Walters
d7629d33a4 tests: Add some test coverage of repeated pulls w/HTTP 500s
Systems like pulp may want to keep retrying in a loop if the server
throws a (hopefully transient) 500, and we need test coverage of
handling these errors versus our existing 404 and 206 coverage.

Closes: #383
Approved by: mbarnes
2016-07-08 18:38:11 +00:00
Colin Walters
71301d1824 tests/libtest.sh: Print non-matching file on failure
We clean up the temporary directory on failure, which means it's hard
to know *why* a regex didn't match.  Print it when we hit an error.

Closes: #383
Approved by: mbarnes
2016-07-08 18:38:11 +00:00
Colin Walters
0d07c7ecde delta: Add --if-not-exists option
I often want to have "idempotent" systems that iterate to a known
state.  If after generating a commit, the system is interrupted, I'd
like the next run to still generate a delta.  But we don't want to
regenerate if one exists, hence this option.

Closes: #375
Approved by: jlebon
2016-07-04 13:27:06 +00:00
Dan Nicholson
a94530111a tests: Improve check for /proc/cmdline kargs
On some systems there may be no root= argument, so the tests for
appending /proc/cmdline arguments will fail. Instead, loop over each of
the arguments in the host's /proc/cmdline and test that they're in the
constructed config file. That will actually test if ostree added all of
the system's /proc/cmdline args correctly. The regex isn't perfect here,
but it's probably good enough for this test.

Closes: #372
Approved by: cgwalters
2016-06-27 20:20:21 +00:00
Dan Nicholson
99a76c9b34 tests: Remove extra $CMD_PREFIX from test-auto-summary.sh
$OSTREE already has $CMD_PREFIX in it, so adding it again causes you to
call env twice with LD_PRELOAD.

Closes: #372
Approved by: cgwalters
2016-06-27 20:20:21 +00:00
Dan Nicholson
8b397301c4 tests: Ensure mutable deployments from libostreetest
When creating sysroots with libostreetest, we don't get the benefit of
the OSTREE_SYSROOT_DEBUG setting in libtest.sh. That means we'll get
immutable deployments that can't be easily cleaned up.

Ensure the environment variable is set before creating new sysroots. It
would be nice to set the debug flags directly, but that's private API
that isn't currently pulled into libostreetest.

Closes: #372
Approved by: cgwalters
2016-06-27 20:20:21 +00:00
Colin Walters
439069b2bb checkout: Add an option to require hardlinks
I've seen a few people hit this and wonder why checkouts are slow/take
space.  Really, ensuring this happens is the *point* of OSTree.
Physical copies should be a last resort fallback for very unusual
situations (one of those is rpm-ostree checking out the db since
librpm doesn't know how to read from libostree).

Even I hit the fact that `/var` is a mountpoint disallowing hardlinks
with `/ostree` once and was confused.  =)

Add this to the rofiles-fuse test case because it creates a mount
point.

Closes: #368
Approved by: jlebon
2016-06-27 13:08:46 +00:00
Simon McVittie
f38c33fec8 entry_pathname_test_helper: these tests need extended attributes
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #366
Approved by: cgwalters
2016-06-26 21:15:36 +00:00
Alexander Larsson
3640725439 tests: Test partial commits for local remotes
This was broken before, fixed in the previous commit.

Closes: #324
Approved by: yuqi-zhang
2016-06-24 15:21:38 +00:00
Yu Qi Zhang
02a2b689dd refs: resolve conflict between local/remote repos
Add the functionality to use the same name for refs in local and remote
repos. This helps users keep track of local refs of remote origin, much
like local and remote git branches.

Previously, when a local ref is specified, resolve_refspec would fall
back to searching through remote repos if the ref is not found locally.
This function now takes an extra flag to specify whether it should
search through remote repos. Additionally, ostree_repo_resove_rev_ext
was added to call resolve_refspec with fallback_remote being false, so
refs --create would no longer complain when trying to create a local
ref of the same name as a remote one.

Fix remote repo parsing not being handled correctly on refs --create.

Closes: #363
Approved by: jlebon
2016-06-23 19:52:26 +00:00
Yu Qi Zhang
744543110e refs: allow overwrite of empty folders
We noticed that once a ref folder is created, there is no existing
command that can remove it. For example, once "foo/bar" is created,
even if the user deletes foo or all the refs under foo, the folder
will persist.

Now when the user attempts to create a ref "foo" either through commit
or refs --create, if a folder "foo" exists but is empty of refs, the
folder is removed and the new ref "foo" is created.

New unit tests in tests-ref.sh verify this functionality.

Closes: #354
Approved by: cgwalters
2016-06-21 20:20:15 +00:00
Colin Walters
d262fc2e0f tests: Support OT_SKIP_READDIR_RAND
For some reason I'm really not inclined to debug right now, `libtool`
is eating the `-fsanitize=address` argument when linking
`libreaddir-rand.so`, which causes failures since we're trying to
`LD_PRELOAD`.

Closes: #351
Approved by: jlebon
2016-06-17 14:43:44 +00:00
Colin Walters
71e768d230 build: Fix libreaddir-rand to honor global CFLAGS
It's unfortunate that in automake one has to explicitly include the globa
`$(AM_CFLAGS)` if one sets `CFLAGS`, and similarly for other variables.

I'm trying to use `-fsanitize=address`, and not including it was
causing linker failures.

We also weren't inheriting the global warnings etc., so I had to fix a
decl-after-statement.

Closes: #351
Approved by: jlebon
2016-06-17 14:43:44 +00:00
Colin Walters
ab47a8a030 Add "archive" as an alias for "archive-z2"
I find the "-z2" is really a long ago relic of the past when I changed
the format.  We no longer have anything to do with the original
`archive`, so let's start allowing people to type `--mode=archive`
which just looks saner.

At some point later I'll update the docs too, but it'll be an annoying
transition period as we'll have to say "On older OSTree, use -z2" etc.

Closes: #346
Approved by: giuseppe
2016-06-16 14:34:23 +00:00
Yu Qi Zhang
f8fcdba0a5 refs: add "ostree refs --create" and unit tests
Added the ability to create a ref (much like a git tag) for an
existing commit through "ostree refs EXISTING --create=NEWREF".
Previously the only way to create a new ref was by creating a new commit,
but refs --create allows multiple refs to point to the same commit.

The command will fail if:
 - None/more than one existing ref is specified
 - The specified EXISTING tag does not exist, or was not specified
 - The specified NEWREF already exists, or is the name of a folder

Add unit tests in tests-ref.sh to verify above functionality

Closes: #340
Approved by: jlebon
2016-06-15 20:42:30 +00:00
Simon McVittie
4801e9b30b libostreetest: include libtest.sh from srcdir, not builddir
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #337
Approved by: cgwalters
2016-06-13 00:23:04 +00:00
Simon McVittie
8011d8c627 libtest: make gpghome in tmpdir writeable
Otherwise, during "make distcheck" it will be read-only (because it
is a copy of the read-only ${srcdir} with permissions preserved), and
deletion will fail during cleanup.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #337
Approved by: cgwalters
2016-06-13 00:23:04 +00:00
Simon McVittie
0cf0fa8154 test-sysroot.js: set "strict mode" when sourcing libtest.sh
As with the C tests in commit 08580118, this makes sure the test
fails as soon as something goes wrong.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #335
Approved by: giuseppe
2016-06-12 11:26:58 +00:00
Alexander Larsson
d258b124a2 tests: Exit valgrind tests if a leak is detected
This fails the test if OT_TESTS_VALGRIND is set and valgrind
detects a leak.

Closes: #334
Approved by: cgwalters
2016-06-10 12:14:40 +00:00
Colin Walters
70af1d26b1 tests: Modernize valgrind infrastructure
The recent memleak fixes motivated me to look at the bitrotted code to
run invocations of `ostree` in the test suite underneath valgrind.

There are a few things here.  First, update suppressions file from
libhif, since I recently worked on it.

When running *uninstalled* as we now support, we need
`libtool --mode=execute` in the mix so it expands out to
the uninstalled binary and we don't valgrind the intermediate shell.

However, it's harder than that because we chdir into a tmpdir,
which defeats the libtool logic.  AFAICS, the only fix for this
is to determine the realbin path before we chdir, and then unfortunately
we need to change every use of `ostree` to `${OSTREE}` =(

Then this immediately breaks for me on RHEL7 because my ancient
copy of `valgrind-3.10.0-16.el7.x86_64` is unaware of syscall 306, i.e.
`syncfs`.

But let's do this first before I dive into that.

Closes: #292
Approved by: krnowak
2016-06-09 21:10:35 +00:00
Mathnerd314
0e9a875393 repo: use OSTREE_TIMESTAMP (=1) for checked-out files
1 is a better choice than 0 because some programs use 0
as a special value; for example, GNU Tar warns of an
"implausibly old timestamp" with 0.

Closes: #330
Approved by: cgwalters
2016-06-09 18:04:55 +00:00
Colin Walters
7fb49037ab tests/test-pull-c: New test that runs through the pull API via C
We have had in the past issues with running `ostree_repo_pull()`
multiple times in the same process, embarassingly enough.  Nothing in
the current test suite covers this, so let's start.

Closes: #322
Approved by: jlebon
2016-06-07 15:58:37 +00:00
Colin Walters
0858011851 tests: Use strict mode by default for C tests
I was extending the C-based test suite to cover more, and noticed that
we weren't aborting if a command failed.  That made the tests somewhat
less useful, so let's fix it.

Closes: #321
Approved by: jlebon
2016-06-06 20:24:21 +00:00
Krzesimir Nowak
569e43c280 core: Add a function creating an archive-z2 content stream
It is quite similar to the already existing
ostree_raw_file_to_content_stream function, so I factored the common
part to a separate function. The difference is that we cannot report
the size of the resulting stream.

Can be useful for serving a "bare" repository as a faked "archive-z2"
repository.

Closes: #308
Approved by: cgwalters
2016-05-26 16:53:08 +00:00
Mathnerd314
80774680e8 commit: Don't require a subject
In practice, a lot of subjects are empty, because the commit date
and branch are sufficient identification. For example, rpm-ostree
does not use subjects. It also doesn't use the command-line ostree
commit tool, so this was not a problem there, but this makes the
behavior consistent.

Also adds a test that empty subjects and omitting the subject
are equivalent. The --timestamp is so that the commits do not
have different timestamps.

Closes: #305
Approved by: cgwalters
2016-05-25 18:37:47 +00:00
Alexander Larsson
86ac8062f5 tests: Fix build with old glib
test-checksum.c was using g_autofree without including libglnx.h which
has the backport for that.

Closes: #304
Approved by: cgwalters
2016-05-20 11:48:02 +00:00
Jonathan Lebon
65e7969b0b OstreeRepo: add extensions directory
It's very useful for third-party applications to have someplace to store
their data guaranteed to be on the same device as the repo (thus
ensuring hardlinks) while still being shielded away from any of OSTree's
timely garbage collections.

We create a new "extensions/" subdirectory where apps can include
whatever they wish in "extensions/myapp/". This subdirectory is
completely unmanaged by ostree.

NB: I didn't bother making it a member of the OstreeRepo proper since we
don't really use it for anything else yet.

Closes: #286
Approved by: cgwalters
2016-05-09 14:43:16 +00:00
Jonathan Lebon
dfbb61270a tests/test-libarchive-import.c: add tests
- Do a bit of refactoring
- Add test for use_ostree_convention
- Add test for xattr_callback
- Add test for SELinux labeling

Closes: #275
Approved by: cgwalters
2016-05-06 14:44:55 +00:00
Jonathan Lebon
5ef4898afa tests/test-libarchive.sh: add more test
- Test both tar and cpio archives
- Test more hardlink corner cases
- Test symlinks more rigorously
- Test stat override
- Test skip list

Closes: #275
Approved by: cgwalters
2016-05-06 14:44:55 +00:00
Jonathan Lebon
b717fd2c18 ot-builtin-commit.c: add --skip-list option
This was already supported by the commit modifier API, just needed to
expose it. This will also be used to test the libarchive API in a future
test.

Closes: #275
Approved by: cgwalters
2016-05-06 14:44:55 +00:00
Colin Walters
a56ba6081a repo: Clean up staging directory for previous boot IDs
We had a policy of cleaning up all files in `$repo/tmp` older
than one day, but we should really clean up previous bootid staging
directories too, as they can potentially take up a lot of disk space.

https://bugzilla.gnome.org/show_bug.cgi?id=760531

Closes: #170
Approved by: jlebon
2016-05-02 18:44:44 +00:00
Colin Walters
e3ec83a934 repo: Add OSTREE_BOOTID override envvar for debugging
It's useful for test cases to be able to influence this.

Conflicts:
	src/libostree/ostree-repo.c

Closes: #170
Approved by: jlebon
2016-05-02 18:44:44 +00:00
Colin Walters
15b3cab65e repo: Add OSTREE_REPO_TEST_ERROR=pre-commit env var
Setting this causes commit to error out.  There are other ways we
could do this in a more sophisticated fashion, such as via SystemTap
etc.  But this has low-tech applicablity, works as non-root.

The reason I'm adding this is so that we can add test cases for
cleanup of the `tmp/staging-` directory.

Closes: #170
Approved by: jlebon
2016-05-02 18:44:44 +00:00
Simon McVittie
542df553e2 test-parent: this test requires user xattrs
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #278
Approved by: cgwalters
2016-04-25 20:46:11 +00:00
Krzesimir Nowak
c2931583d3 tests: Add a test for ostree pull --url
Closes: #271
Approved by: cgwalters
2016-04-22 14:27:06 +00:00
Alexander Larsson
5595664e47 ostree export: Add --prefix option
This lets you set a prefix for the resulting archive patsh.
Especially useful in combination with --subpath, for instance
--subpath=subdir --prefix=subdir to extract just subdir.

Closes: #265
Approved by: cgwalters
2016-04-19 12:28:06 +00:00
Alexander Larsson
5079f70ec0 ostree export: Add --subpath support
This lets you export a subset of a commit.

Closes: #265
Approved by: cgwalters
2016-04-19 12:28:06 +00:00
Krzesimir Nowak
9db2f43b14 tests, lib: Remove unused variables
Closes: #267
Approved by: cgwalters
2016-04-19 12:16:16 +00:00
Colin Walters
d5498a8b16 libtest: Skip FUSE if CAP_SYS_ADMIN not in bounding set
I'm not sure if something changed or there was a race condition , or
if extraterrestrials intervened to make the tests pass before, but
in the Docker container `/dev/fuse` does exist and since we appear
to be running as uid 0 it'll be writable.

We previously discussed that another requirement for FUSE (and most
setuid binaries) is having `CAP_SYS_ADMIN`, so let's check for that in
the bounding set too.

Closes: #266
Approved by: jlebon
2016-04-18 19:53:13 +00:00
Simon McVittie
24cab773a2 libtest: show files' contents when assertions about them fail
I've seen an intermittent test failure in an autobuilder (sbuild)
environment where logs from failed builds cannot be retrieved,
but I can no longer reproduce it. Put the contents of the offending
file in the test's failing output so that if the failure comes back,
it can be debugged.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #264
Approved by: cgwalters
2016-04-17 14:16:34 +00:00
Colin Walters
e9a1809c4d Rename test-abi to test-symbols, start verifying symbols are documented
We keep forgetting to update `apidoc/ostree-sections.txt`, so let's
start enforcing it.  Of course it turns out we had some bugs here
like symbols marked as public but never implemented, etc.  Those
are fixed in the prior commits.

Closes: #263
Approved by: giuseppe
2016-04-17 13:52:07 +00:00
Alexander Larsson
9e7e594907 Add OstreeRepo option for an out-of-band cache dir
This allows you to have a writable cache dir even for a system-owned
repository.

Closes: #250
Approved by: cgwalters
2016-04-14 15:55:08 +00:00
Alexander Larsson
ddda8e5b8b Add support for ostree static-delta delete
Closes: #245
Approved by: giuseppe
2016-04-08 13:42:43 +00:00
Alexander Larsson
6aa447a917 Add test case for inheriting remote options
Closes: #236
Approved by: cgwalters
2016-04-07 20:02:46 +00:00
Alexander Larsson
91734a8a18 Support pathnames for --subpath=...
This allows you to pull a single file, rather than just a directory.

Closes: #244
Approved by: cgwalters
2016-04-07 15:15:39 +00:00
Giuseppe Scrivano
7763c452dd tests: add missing ${CMD_PREFIX} before ostree
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #242
Approved by: cgwalters
2016-04-07 12:25:24 +00:00
Alexander Larsson
ace0d4650b pull-local: Support --gpg-verify and --gpg-verify-summary
Force the otherwise disabled gpg verifications on.
Note: You need to pass --remote=foo so we know what gpg keys to verify
against.

Closes: #237
Approved by: cgwalters
2016-04-05 19:12:09 +00:00
Alexander Larsson
87298c84b8 Fix local-pull test
This was not being run for whatever reason. Also, the test
count was wrong.

Closes: #237
Approved by: cgwalters
2016-04-05 19:12:09 +00:00
Gatis Paeglis
4e81548447 Introducing ostree-grub-generator
ostree-grub-generator can be used to customize
the generated grub.cfg file. Compile time
decision ostree-grub-generator vs grub2-mkconfig
can be overwritten with the OSTREE_GRUB2_EXEC
envvar - useful for auto tests and OS installers.

Why this alternative approach:

1) The current approach is less flexible than using a
   custom 'ostree-grub-generator' script. Each system can
   adjust this script for its needs, instead of using the
   hardcoded values from ostree-bootloader-grub2.c.

2) Too much overhead on embedded to generate grub.cfg
   via /etc/grub.d/ configuration files. It is still
   possible to do so, even with this patch applied.
   No need to install grub2 package on a target device.

3) The grub2-mkconfig code path has other issues:
   https://bugzilla.gnome.org/show_bug.cgi?id=761180

Task: https://bugzilla.gnome.org/show_bug.cgi?id=762220

Closes: #228
Approved by: cgwalters
2016-04-04 14:19:35 +00:00
Simon McVittie
18e9169d7a libtest.sh: only check whether $(pwd) is empty once
test-sysroot.js runs libtestExec() twice, one of which is after
creating non-hidden directories in $(pwd), so this check needs to be
skipped the second time.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #234
Approved by: cgwalters
2016-03-31 18:54:13 +00:00
Simon McVittie
9c4babf316 test-xattrs: sync how this is skipped with test-rofiles-fuse
If we touch test-xattrs before sourcing libtest.sh, we get:

test tmpdir=/tmp/test-tmp-ostree_test-xattrs.sh.test-HSEXEY is not
empty; run this test via `make check TESTS=`, not directly

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #234
Approved by: cgwalters
2016-03-31 18:54:13 +00:00
Simon McVittie
839628b3fa test-abi: use G_TEST_SRCDIR, G_TEST_BUILDDIR
There's no need to invent new variables for these.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #234
Approved by: cgwalters
2016-03-31 18:54:13 +00:00
Simon McVittie
efb86ba9d3 libtest.sh: use G_TEST_SRCDIR, G_TEST_BUILDDIR to find resources
This fixes the bug that in installed-tests that run testlib.sh under
"bash -c" (i.e. the C and JS tests), $(dirname $0) is "." and we can't do
the LD_PRELOAD correctly:

ERROR: ld.so: object './libreaddir-rand.so' from LD_PRELOAD cannot be
preloaded (cannot open shared object file): ignored.

Similarly, those tests can't copy gpghome correctly.

This also removes the confusing situation that SRCDIR in libtest.sh
(which is ${top_srcdir}/tests) does not mean the same thing as SRCDIR
in test-abi.sh (which is just ${top_srcdir}).

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #234
Approved by: cgwalters
2016-03-31 18:54:13 +00:00
Colin Walters
bdf24cdc04 tests: Make failing to kill the GPG agent non-fatal
It's not working for me in `make check` on a RHEL 7 Workstation,
apparently because no GPG agent is spawned.  I'm guessing this has
something to do with the GPG version?

The downside of this is we will be less likely to notice if GPG
changes again and we start leaking agents like we're in The Matrix
Reloaded.  But the real solution to that is containers anyways.

Closes: #233
Approved by: smcv
2016-03-31 18:43:31 +00:00
Simon McVittie
9dafc82024 Probe for GNU parallel more accurately
moreutils parallel isn't compatible with the command-line syntax
used here. Because it doesn't implement GNU-style --help and exits 1
when that option is given, this test was correctly skipped when
using moreutils parallel, but only by mistake.

moreutils parallel might conceivably gain --help and --version in
future, but hopefully nothing incompatible with GNU parallel is going
to gain a --gnu option.

Also use the --gnu option to force the new command-line semantics;
some versions optionally supported an incompatible command-line syntax
taken from moreutils parallel.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #232
Approved by: cgwalters
2016-03-31 14:22:06 +00:00
Jonathan Lebon
8cda8b6866 basic-test: commit with a non-empty subject
[smcv: split out from a larger commit, part of PR #231; add commit message]
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #232
Approved by: cgwalters
2016-03-31 14:22:06 +00:00
Jonathan Lebon
7835fcdc68 test-pull-untrusted.sh: always corrupt a regular file, not a symlink
test-pull-untrusted.sh would pass when run as root, but fail when run
as testuser. It turned out that the way the files were stored in the
repo when running as a testuser were different, which meant that a
different .file object was chosen for corruption. Except that file
turned out to be a symlink, so the echo "broke" actually just wrote
to the no_such_file symlink target, thus keeping the actual symlink
file's checksum the same and causing the pull-local to pass when it
should have failed.

[smcv: split this out of a larger commit, part of PR #231]
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #232
Approved by: cgwalters
2016-03-31 14:22:06 +00:00
Simon McVittie
e1ce859368 Skip tests that run rofiles-fuse if /dev/fuse or /etc/mtab unavailable
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #232
Approved by: cgwalters
2016-03-31 14:22:06 +00:00
Simon McVittie
1dd85513e5 tests/admin-test.sh: this is a bash script, not a POSIX sh script
The "function foo()" syntax is bash-specific, and Colin indicated in
PR #226 that he prefers to require bash rather than trying to support
every POSIX shell.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #232
Approved by: cgwalters
2016-03-31 14:22:06 +00:00
Simon McVittie
b25ddd29ab In tests that use gpg, terminate the gpg-agent after testing
Otherwise we leak those processes.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #232
Approved by: cgwalters
2016-03-31 14:22:06 +00:00
Simon McVittie
3e3755c497 various tests: skip if temp directory lacks xattr support
Some autobuilder environments place the entire build chroot on tmpfs, so
even /var/tmp might not have this.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #232
Approved by: cgwalters
2016-03-31 14:22:06 +00:00
Simon McVittie
c276025466 test-xattrs: use TAP syntax to skip test
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #232
Approved by: cgwalters
2016-03-31 14:22:06 +00:00
Colin Walters
23d26d5f65 commit: Support writing orphans
The API supports this, and it's not hard for us to do in the command
line as well.  One possible use case is separating "content
generation" in a separate server.

Related: https://github.com/ostreedev/ostree/pull/223

Closes: #227
Approved by: jlebon
2016-03-30 03:10:50 +00:00
Colin Walters
c6b4ecd474 commit: Support generating commits with no parent, or a custom one
When I'm doing local development builds, it's quite common for me not
to want to accumulate history.  There are also use cases for this on
build servers as well.

In particular, using this, one could write a build system that didn't
necessarily need to have access to (a copy of) the OSTree repository.
Instead, the build system would determine the last commit ID on the
branch, and pass that to a worker node, then sync the generated
content back.

The API supported generating custom commits that don't necessarily
reference the previous commit on the same branch, let's just expose
this in the command line for convenience.

I plan to also support this rpm-ostree.

Closes: #223
Approved by: jlebon
2016-03-29 14:31:29 +00:00
Alexander Larsson
456f515522 Add --untrusted option to pull and pull-local
https://bugzilla.gnome.org/show_bug.cgi?id=764125

Closes: #221
Approved by: cgwalters
2016-03-25 12:56:55 +00:00
Colin Walters
f84c198006 tests: Add a test-abi
This would have caught a potential ABI regression in
https://bugzilla.gnome.org/show_bug.cgi?id=764056
(If we ran this test while building `--without-libarchive`)

Pull request: #218
Approved by: jlebon
2016-03-24 16:23:12 +00:00
Colin Walters
7232b2940b tests: Add a commitpartial + prune test
Followup for previous commit.
2016-03-23 15:34:17 -04:00
Simon McVittie
0a7e128782 Skip test_libarchive_ignore_device_file if we cannot write xattrs
The test tries to get a filesystem that supports xattrs by writing
to /var/tmp, but in some automated build environments the entire
build chroot is on a tmpfs.

Signed-off-by: Simon McVittie <smcv@debian.org>
2016-03-21 13:02:49 -04:00
Simon McVittie
35f412c86a tests/admin-test.sh: add #!/bin/sh
This is executable and appears to be intended to be executed directly,
so it should have a #! line.
2016-03-21 13:02:25 -04:00
Colin Walters
684029daa3 libglnx porting: gs_free -> g_autofree
There were surprisingly few uses left.
2016-03-18 12:08:19 -04:00
Colin Walters
a02b425fc2 tests: Strengthen test tmpdir sanity check, be compat with ginsttest saving
GNOME Continuous uses ginstest-runner --report-directory, which causes
the tests to save their tmpdirs persistently.  This also means the
result directories didn't match the `/(var/)?tmp` regexp, which broke
the ostree tests in GContinuous.

Fix this by simply asserting that the tmpdir either has `.tmpdir` or
nothing.
2016-03-18 08:45:25 -04:00
Colin Walters
f51d651b10 docs: Add a section on writing buildsystems
And add a test that is a demo buildsystem.
2016-03-16 17:15:58 -04:00
Giuseppe Scrivano
2bb25a05b0 tests: add test for summary file caching
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-03-15 09:48:47 +01:00
Colin Walters
e2234e854d tests/basic: Fix race in timestamp test
%Z only uses seconds, so it's possible that we did the commit
in the same second, which made this test racy.

- Switch to full nanosecond precision using '%.Y' so it always differs
- Fix the inverted `cmp` usage
- Add a missing `ok`
2016-03-04 10:53:47 -05:00
Colin Walters
fea786cb2d lib: Add ostree_sysroot_load_if_changed() API
This will allow daemons like rpm-ostree to detect if there are any new
deployments efficiently, in combination with using inotify.  If there
are any changes, rpm-ostree wants publish them on DBus.

While we're here, add some changes to start doing unit C testing of
the sysroot API.
2016-03-03 21:56:23 -05:00
Colin Walters
ebd0370976 tests: Unify some tmpdir code, add ability for C to use libtest.sh
I want to be able to easily test the C API on actual data in an OSTree
repo.  The shell `libtest.sh` has code to generate it.  Bridge the two
worlds by introducing a little `libostreetest` library which has a C
API which spawns a shell that runs things in `libtest.sh`.

Yes, this is about as beautiful as it sounds, which is to say, it's
not.  But it works!

Note while we were here, I realized we were actually now creating
*two* tmpdirs per test in `make check` because the tap driver was
already doing that.  Unify it so we know the C code can rely on it.
2016-03-03 18:03:32 -05:00
Colin Walters
12db46e5af tests: More TAP fixups
Hopefully getting closer now.
2016-03-03 18:00:54 -05:00
Colin Walters
0174224a2d tests: Convert two more exit 77 instances into TAP-compatible SKIP
Followup to https://github.com/GNOME/ostree/pull/194
2016-03-03 14:19:10 -05:00
Colin Walters
d49718a619 deploy: Bump the mtime on ostree/deploy after deployments finish
This allows other processes (e.g. rpm-ostreed) to monitor for external
changes (e.g. if someone does `ostree admin undeploy`) in a relatively
sane fashion.

Specifically, I'm trying to fix:
https://github.com/projectatomic/rpm-ostree/issues/220
2016-03-03 08:52:42 -05:00
Colin Walters
d25212f04a tests: Port to glib-tap.mk, make make check run all of the tests
OSTree's code for testing predates the `glib-tap.mk` making its
way into GLib.  Let's switch to it, as it provides a number
of advantages.

By far the biggest advantage is that `make check` can start to run
most of the tests *in addition* to having them work installed.

This commit keeps the installed tests working, but `make check` turns
out to be really broken because...our TAP usage has bitrotted to say
the least.  Fix that all up.

Do some hacks so that the tests work uninstalled as well - in
particular, `glib-tap.mk` and the bits encoded into
`g_test_build_filename()` assume *recursive* Automake (blah).  Work
around that by creating a symlink when installed to loop back.
2016-03-03 08:50:19 -05:00
Giuseppe Scrivano
6d73a620e1 refs: Add argument --list to print the full ref name
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-03-02 14:53:14 -05:00
Giuseppe Scrivano
c9b02ae1f3 refs: add tests
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-03-02 14:52:02 -05:00
Giuseppe Scrivano
2badf36cfd test-rofiles-fuse: skip when fusermount is not present
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-03-01 10:08:25 -05:00
Giuseppe Scrivano
82d4e7fe68 Fix make syntax-check
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-03-01 10:08:25 -05:00
Colin Walters
7fdf072710 deltas: Heuristically detect endianness for older deltas
If the average object size is greater than 4GiB, let's assume we're
dealing with opposite endianness.  I'm fairly confident no one is
going to be shipping peta- or exa- byte size ostree deltas, period.
Past the gigabyte scale you really want bittorrent or something.
2016-02-26 08:19:01 -05:00
Colin Walters
04d77da005 deltas: Use endianness marker when parsing
Extend the `static-delta show` and `pull` commands to use the
endianness information (if available).
2016-02-26 08:19:01 -05:00
Colin Walters
277220aaa6 deltas: Include an endianness marker
We screwed up and had delta integers use host endianness.  Start
digging out by at least annotating the endianness.

https://bugzilla.gnome.org/show_bug.cgi?id=762515
2016-02-26 08:19:01 -05:00
Colin Walters
30c5fb1a51 libarchive: Make autocreate_parents imply autocreating root dir
Some Docker layers are just metadata in the `layer.json`.  If one is
mapping Docker layers to OSTree commits, one needs to create a dummy
root directory, because OSTree doesn't support metadata-only commits.

Let's just push that logic down here because it's easier than special
casing it in higher levels.
2016-02-22 17:02:31 -05:00
Colin Walters
4beb5f4eaf pull: Add a --dry-run option for static deltas
One of the design goals with deltas was not just wire efficiency,
but also having all the data up front about how much data would
be transferred before starting.

Let's expose that better by adding a `dry-run` option to the pull API.
This requires static deltas to be useful.  Basically we simply call
the progress callback once with the data from the superblock.
2016-02-19 15:21:13 -05:00
Colin Walters
f2c5ecb996 pull: Add require-static-deltas pull option
For a production release repository, most OS vendors would want
to just always use static deltas.  Add the ability for the pulls to
require it.

(I think I'll also add a summary key for this actually in addition,
 so the repo manager can force it too)
2016-02-19 15:21:13 -05:00
Jonathan Lebon
eba7df0da9 ostree-sysroot: add debug option to help testing
If ostree is run in a test setup where it operates as root in a tmp
directory, it might cause issues to flag the deployments as immutable.
The test harness might simply be doing an `rm -rf` (effectively the case
for gnome-desktop-testing-runner), which will then fail.

We add a new debug option to the ostree_sysroot object using GLib's
GDebugKey functionality to allow our tests to communicate to ostree that
we don't want immutable deployments.
2016-02-19 11:58:08 -05:00
Colin Walters
3a555114bc repo: Add ostree_repo_import_archive_to_mtree
This is a more flexible version of the previous
ostree_repo_write_archive_to_mtree() which took a file reference.
This has an extensible options structure, and in particular
now supports `ignore_unsupported_content`.

I plan to use this for importing Docker images which contain device
nodes.  (There's no reason for container images to have those, so
we'll just ignore them).

Also here, just like the export variant, the caller is responsible for
setting up libarchive.
2016-02-19 08:54:04 -05:00
Colin Walters
baaf7450da Support Docker-style whiteouts
This is to enable importing Docker layers as ostree commits, then
checking them out in a union.

The prototype work for this is in:
https://github.com/cgwalters/dlayer-ostree

Though it will likely ultimately end up in:
https://github.com/projectatomic/atomic
2016-02-16 10:54:59 -05:00
Colin Walters
42c60effbe pull: Support specifying exact commit to pull via branch@commit
I don't know why we didn't do this a long time ago.  This extends the
pull API to allow grabbing a specific commit, and will set the branch
to it.  There's some support for this in the deploy engine, but there
are a lot of reasons to support it for raw pulls (such as subset
mirroring cases).

In fact I'm thinking we should also have the override-version logic
here too.

NOTE: One thing I debated here is inventing a new syntax on the
command line.  Git doesn't seem to have this functionality (probably
because it'd be rarely used). The '@' character at least doesn't
conflict with anything.

Anyways, I wanted this for some other test cases.  Without this,
writing tests that go between different commits is more awkward as one
must generate the content in one repo, then pull downstream, then
generate more content, then pull again.  But now I can just keep track
of commit IDs and do exactly what I want without synchronizing the
tests.
2016-02-14 10:12:26 -05:00
Colin Walters
355f8438ef Add an export builtin, and API to write to libarchive
At the moment I'm looking at using rpm-ostree to manage RPM inputs
which can then be converted into Docker images.  It's most convenient
if we can stream directly out of libostree rather than doing a
checkout + tar combination.

There are also backup/debugging etc. reasons to implement `export` as
well.
2016-02-14 09:53:01 -05:00
Colin Walters
e9ccdd2d00 Import rofiles-fuse
While it's not strictly tied to OSTree, let's move
https://github.com/cgwalters/rofiles-fuse in here because:

 - It's *very* useful in concert with OSTree
 - It's tiny
 - We can reuse OSTree's test, documentation, etc. infrastructure

One thing to consider also is that at some point we could experiment
with writing a FUSE filesystem for OSTree.  This could internalize a
better equivalent of `--link-checkout-speedup`, but on the other hand,
the cost of walking filesystem trees for these types of operations is
really quite small.

But if we did decide to do more FUSE things in OSTree, this is a step
towards that too.
2016-02-10 13:11:25 +01:00
Colin Walters
ca57ec4aa5 repo: Port -refs.c to openat()
I'd like to incrementally convert all of `ostree-repo*.c` to
fd-relative usage, so that we can sanely introduce
`ostree_repo_new_at()` which doesn't involve GFile.

This one is medium risk, but passes the test suite.
2016-01-28 14:57:13 -05:00
Colin Walters
5ebe43859d tests: Use "bash strict mode"
I noticed in the static deltas tests, there were some tests that
should have been under `-o pipefail` to ensure we properly propagate
errors.

There were a few places where we were referencing undefined variables.

Overall, this is clearly a good idea IMO.
2016-01-27 11:44:10 -05:00
Giuseppe Scrivano
6f96571679 tests: add tests for prune --static-deltas-only
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-12-18 11:21:58 +01:00
Matthew Barnes
460a4b2852 remote: Add "ostree remote summary" command
Downloads and prints a remote summary file and any signatures in an
easy-to-read format, or alternatively with the --raw option, prints
the summary GVariant data directly.

https://bugzilla.gnome.org/show_bug.cgi?id=759250
2015-12-17 15:49:51 -05:00
Giuseppe Scrivano
f88a9733ce tests: add missing ${CMD_PREFIX}
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-12-04 11:10:51 -05:00
Giuseppe Scrivano
335ea3f339 parse-datetime: use the module from gnulib
Use the parse-datetime module from gnulib, and adapt it to not require
other modules as portability is not really an issue for us.

DATE can be specified in different formats, such as: "-1 week", "last
monday", "1 week ago".

Include the generated .c file in the repository so to not add another
dependency to Bison.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-12-04 11:10:46 -05:00
Matthew Barnes
df75fc232a repo: Never delete .commitmeta files
Do not delete a .commitmeta file after removing the last metadata entry.
This way a client will pull the empty .commitmeta file and overwrite old
metadata as expected.

https://bugzilla.gnome.org/750459
2015-11-23 14:22:45 -05:00
Giuseppe Scrivano
45cee1bd70 tests: prefix invocation of ostree with where missing
And add a syntax rule to avoid this in future.

Fixed by:

sed -i -e 's|^ostree |${CMD_PREFIX} ostree |g' tests/*.sh

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-11-16 11:07:55 +01:00
Giuseppe Scrivano
11bc9e58c1 tests: add tests for prune --keep-younger-than=DATE
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-11-16 11:07:55 +01:00
Giuseppe Scrivano
daa6f150c4 tests: add test for ostree prune --delete-commit
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-11-16 10:57:33 +01:00
Alexander Larsson
ee70ffbbcb Add tests for inline static deltas 2015-11-10 08:56:13 +01:00
Giuseppe Scrivano
438e21e858 tests: add tests for prune and tombstones commits
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-11-03 09:53:38 +01:00
Matthew Barnes
2210a79bfa upgrade: Add --override-commit=CHECKSUM option
Adds an entry to the origin file to force the OstreeSysrootUpgrader to
pull and deploy the given checksum.

  [origin]
  override-commit=CHECKSUM

If the option is not given, any such entry is explicitly removed from
the origin file to ensure we upgrade to the latest available commit.
2015-10-13 12:34:57 -04:00
Giuseppe Scrivano
442dcec43b tests: fix LZMA test to really compress/decompress
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-10-13 15:24:09 +02:00
Colin Walters
b6e2eaf212 tests: Add a test for pull+deploy of specific "bare" commit
This is the "pull an older version" that's now supported since
36d65b3cfc

https://github.com/GNOME/ostree/pull/145
2015-10-07 15:20:38 -04:00
Colin Walters
bab6503b69 sysroot: Cleanup refs and prune even on last undeployment
I was working on a different test, and ended up being very confused at
the behavior where removing the last deployment didn't remove the last
`ostree/X/X/X` ref pointing to its commit.

There's no reason to special case the last undeployment AFAIK, and the
existing code handles this.
2015-09-24 12:25:23 -04:00
Colin Walters
27fcf68e7c tests: Add a test for static-delta apply-offline
Obviously it was broken...
2015-09-13 22:23:56 -04:00
Giuseppe Scrivano
bddb25f79e pull: honor gpg-verify-summary=false when a summary signature is present
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-09-10 08:59:01 -04:00
Colin Walters
0110183675 fetcher: Use 0666 (-umask) for temporary files
There's no reason to keep them hidden.  I have a hard policy that
OSTree should *not* be used to carry secrets.  Things like host ssh
private keys should be set up out of band by an OS-external
configuration mechanism such as kickstart, cloud-init, etc.

We also assume that hiding binaries is not very useful as most
attackers would be able to find them on the Internet or (for
subscribed content) acting as a customer.

This fixes a bug with mirroring because we changed to take the
unmodified upstream objects rather than uncompress <-> recompress.

https://bugzilla.gnome.org/show_bug.cgi?id=748959
2015-08-27 11:36:48 -04:00
Jasper St. Pierre
8e8bfa8708 Update .gitignore 2015-08-26 12:16:29 -07:00
Giuseppe Scrivano
863f5d8598 tests: Add tests for rollsum 2015-08-25 09:27:56 -04:00
Giuseppe Scrivano
9b9f4b04b4 tests: rename test-rollsum to test-rollsum-cli
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-08-24 11:07:36 +02:00
Giuseppe Scrivano
3861b79efd tests: add tests for LZMA compressor and decompressor
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-08-12 07:55:50 +02:00
Jasper St. Pierre
3c152fb38b Update .gitignore
https://bugzilla.gnome.org/show_bug.cgi?id=753336
2015-08-08 21:53:43 -04:00
Giuseppe Scrivano
75cdbb8e82 tests: skip tests using gjs/parallel if they are not installed
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes #131
2015-08-05 10:58:00 -04:00
Giuseppe Scrivano
8b9effea56 tests: add tests for --disable-bsdiff and --max-bsdiff-size
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-07-29 17:34:21 -04:00
Jasper St. Pierre
8034a3842b Update .gitignore
https://bugzilla.gnome.org/show_bug.cgi?id=752950
2015-07-28 14:23:02 -04:00
Giuseppe Scrivano
11ba9056ea pull: new option --commit-metadata-only
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-07-24 12:43:35 -04:00
Giuseppe Scrivano
a917c96976 tests: do not commit from the working directory
It fixes this problem:

```
error: Not a regular file or symlink: S.gpg-agent
```

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-07-21 10:02:30 -04:00
Matthew Barnes
df0cae4375 tests: Export OSTREE_SYSROOT in setup_os_repository
Eliminates the need for constantly passing --sysroot=sysroot, but
also makes ostree place remote configs for sysroot/ostree/repo in
sysroot/etc/ostree/remotes.d where they should have been all along.
2015-07-16 12:49:34 -04:00
Colin Walters
530631376e tests: Check error messages instead of "expected-fail", handle old parallel 2015-06-29 13:35:07 -04:00
Colin Walters
720e3b5f83 pull: Error if gpg=true and summary is 404, add more tests
I did a quick audit pass through the pull code.  What I focused on the
most is the case where `gpg-verify-summary=true`, and in particular
where `gpg-verify=false` too.  This should be a valid and secure
configuration.

The primary change here is to error out very quickly if either
`summary` or `summary.sig` are 404.  Previously, we'd only error out
if we were processing deltas.

Expand the existing test case to cover this, plus invalid summary and
invalid sig.  (The test case was failing with current git master too).
2015-06-27 12:04:18 -04:00
Giuseppe Scrivano
0bd10eb6e2 tests: add test for check for remote add --set=gpg-verify-summary=true
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-06-26 11:02:25 +02:00
Giuseppe Scrivano
19ce011e1f pull: fail if GPG is enabled and the summary is not signed
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-06-26 11:02:25 +02:00
Matthew Barnes
6284beb2b6 ostree: Add a "remote refs" command
Works like "ostree refs" but fetches refs from a remote repo.

This depends on the remote repo having a summary file, but any repo
being served over HTTP *ought* to have one.
2015-06-26 11:02:24 +02:00
Colin Walters
43cbe8c1b1 tests/metalink: Add a case with nested unknown elements 2015-06-15 21:20:52 -04:00
Dan Nicholson
5af7e8e8c2 pull-local: Support --depth option
Like pull, allow pull-local to mirror another another repository by
specifying how many parents to traverse.

https://bugzilla.gnome.org/show_bug.cgi?id=750581
2015-06-14 08:49:35 -04:00
Giuseppe Scrivano
91d7f3fa0d tests/test-pull-mirror-summary.sh: remove empty newline
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-06-12 13:10:02 +02:00