Commit Graph

500 Commits

Author SHA1 Message Date
Colin Walters
36b28cb4d2 upgrade: Add support for --pull-only and --deploy-only
This makes it easier to script downloading updates in the background,
and only do deployments just before rebooting.

Partially addresses https://github.com/ostreedev/ostree/issues/640

Closes: #642
Approved by: jlebon
2017-02-27 20:57:23 +00:00
Colin Walters
09b392675a main: Make ostree --version output YAML (and add gitrev)
I learned today that `docker version` does this and I really like
the idea.  While we have the patient open, also add the gitrev
with code taken from https://github.com/projectatomic/rpm-ostree/pull/584

Closes: #691
Approved by: giuseppe
2017-02-22 18:57:18 +00:00
Colin Walters
f02dcc4997 libtest: Re-enable quiet mode for building fs tree
I think I commented this out while debugging something, and forgot to re-enable
it. Reading the log files should be a better again after this.

Closes: #699
Approved by: giuseppe
2017-02-21 15:35:23 +00:00
Colin Walters
b5c5003ff6 pull: Fold together deltapart+fallback count for display
It's just simpler, and I'm not sure people are going to care
much about the difference by default.

We already folded in the fallback sizes into the download totals, so folding in
the count makes things consistent; previously you could see e.g.
`3/3 parts, 100MB/150MB` and be confused.

Closes: #678
Approved by: giuseppe
2017-02-17 14:58:25 +00:00
Colin Walters
46544f5b4d commit: Support -F/--body-file, like git
This is more convenient to script for projects which haven't
yet made the leap to using the API.

Closes: https://github.com/ostreedev/ostree/issues/674

Closes: #681
Approved by: jlebon
2017-02-14 14:15:08 +00:00
Colin Walters
361aa449fb libcurl backend
For rpm-ostree, we already link to libcurl indirectly via librepo, and
only having one HTTP library in process makes sense.

Further, libcurl is (I think) more popular in the embedded space.  It
also supports HTTP/2.0 today, which is a *very* nice to have for OSTree.

This seems to be working fairly well for me in my local testing, but it's
obviously brand new nontrivial code, so it's going to need some soak time.

The ugliest part of this is having to vendor in the soup-url code. With
Oxidation we could follow the path of Firefox and use the
[Servo URL parser](https://github.com/servo/rust-url).  Having to redo
cookie parsing also sucked, and that would also be a good oxidation target.

But that's for the future.

Closes: #641
Approved by: jlebon
2017-02-09 16:37:45 +00:00
Colin Walters
ecf5c079ea tests: Add setup for more realistic repo, change pull-many to use
As OSTree has evolved over time, the tests grew with it.  We
didn't start out with static deltas or a summary file, and the
tests reflect this.

What I really want to do is change more of the pull tests, from
corruption/proxying/mirroring etc. to use this more realistic
repo rather than the tiny one the other test creates.

We start by using some of the code from `test-pull-many.sh`, and change that
test to use `--disable-static-deltas` for pull, since the point of that test is
to *not* test deltas.

Still TODO is investigate changing other tests to use this.

Closes: #658
Approved by: jlebon
2017-02-02 17:04:31 +00:00
Colin Walters
5eea1cdad8 lib: Move the bupsplit selftest into our test framework
We weren't running it before. Also I switched it to use GLib. Preparation for
some oxidation work (having an implementation of bupsplit in Rust).

I exported another function to do the raw rollsum operation which is what this
test suite uses.

Closes: #655
Approved by: jlebon
2017-02-02 16:51:36 +00:00
Colin Walters
d1f4d94720 tests: Add a big (many objects) pull
This would be more likely to tickle things like
https://github.com/ostreedev/ostree/issues/601
reliably.

Also, while working on the curl backend, I hit on the fact that curl doesn't
queue (by default, you can enable) and will happily create 20000+ concurrent TCP
connections if you try. Having this test would have made that more likely to
fail.

Closes: #650
Approved by: giuseppe
2017-01-23 17:29:02 +00:00
Colin Walters
5c940987e7 Add support for more selective pruning
There are use cases for having a single repo with branches
with different lifecycles; a simple example of what I was
trying to do in CentOS Atomic Host work is have "stable"
and "devel" branches, were we want to prune devel, but
retain *all* of stable.

This patch is split into two parts - first we add a low level "delete all
objects not in this set" API, and change the current prune API
to use this.

Next, we move more logic into the "ostree prune" command. This paves the way for
demonstrating how more sophisticated algorithms/logic could be developed outside
of the ostree core.

Also, the --keep-younger-than logic already lived in the commandline, so it
makes sense to keep extending it there.

Closes: https://github.com/ostreedev/ostree/issues/604

Closes: #646
Approved by: jlebon
2017-01-19 16:28:00 +00:00
Simon McVittie
9d94fc40c8 Make corrupt-repo-ref.js executable
Debian's Lintian packaging consistency check complains that it isn't
executable but has a #! line. In fact it's reasonable to run this
script directly, so make it executable, and put it in a _scripts
variable so it will be installed executable.

Closes: #652
Approved by: cgwalters
2017-01-19 13:54:59 +00:00
Simon McVittie
9a3f82caae Sourced test snippets: remove shebang and make non-executable
They are installed non-executable, which makes Debian's Lintian
packaging consistency check complain that #! is only useful
in executable scripts. But in fact they are not useful to execute
directly (they rely on setup being done in the script that sources
them), so just chmod them -x.

Closes: #652
Approved by: cgwalters
2017-01-19 13:54:59 +00:00
Simon McVittie
01fb30b839 Fix TAP syntax in test-basic-user.sh, and run it
In its initial commit, Alexander Larsson wrote

    This works standalone, but unfortunately it breaks in
    gnome-desktop-testing-runner as /tmp doesn't support
    xattrs, so it is not installed atm.

but we now (a) use /var/tmp, and (b) explicitly skip the test if
xattr support is unavailable. So it should be OK to run now.

Closes: #652
Approved by: cgwalters
2017-01-19 13:54:59 +00:00
Colin Walters
b260fa764c libtest: Enable web server logs
Now that we're daemonizing, it's useful to have the logs.  I
wanted this while debugging cookies.

Closes: #651
Approved by: giuseppe
2017-01-19 10:47:15 +00:00
Colin Walters
686f91062f tests: Loosen error regexp
libcurl AFAICS doesn't have an API to convert HTTP code ➡️ error
string, so let's make the test regexp operate on both.

Closes: #651
Approved by: giuseppe
2017-01-19 10:47:15 +00:00
Colin Walters
56891f9d48 tests: Don't inject newline in URL
It turns out libsoup strips all whitespace even *inside* a URL. We could do that
for libcurl too but...really, people shouldn't do that. In this test we were
adding the trailing newline into the URL. If someone complains who is using the
libcurl code we can deal with it then.

Closes: #651
Approved by: giuseppe
2017-01-19 10:47:15 +00:00
Colin Walters
2f71136eec tests: Alias assert_not_reached() -> fatal()
We had a lot of copies of the "echo something 1>&2; exit 1" code even though
`assert_not_reached()` was it.  Hence, I think we need a shorter alias for that.

Doing this particularly since I noticed a missing `1` in an `exit 1` call in the
rpm-ostree copy of this.

Closes: #648
Approved by: jlebon
2017-01-18 14:28:29 +00:00
Colin Walters
ced22f6c9b Split trivial-httpd into separate binary
Working on the libcurl backend, I hit the issue that the trivial-httpd program
depends on libsoup. I briefly considered having two versions, but libcurl is
client only, and moreover trivial-httpd is no longer trivial - it has various
features which are used by the test suite extensively.

Hence, what we'll do is build it as a separate binary which links to libsoup,
and use it during the tests. We *also* currently still provide `ostree
trivial-httpd` since some things use it like `rpm-ostree-toolbox` and the
Cockpit tests.

After those are ported to use some other webserver, I plan to add a build-time
option to drop it.

Closes: #636
Approved by: jlebon
2017-01-04 16:32:11 +00:00
Colin Walters
359ea9b58f tests: Tweak installed tests to deal with ASAN
We need to disable readdir-rand there too.

Closes: #622
Approved by: jlebon
2016-12-09 18:05:53 +00:00
Colin Walters
17f264a487 repo: Add unconfigured-state to remote config options
This is a migration from the origin version.  It's
nicer to have it in the remote, since that's what one
needs to change.  Then tools don't need to mess with
the origin file.o

In fact in this scenario one can keep the "media source" like
`file:///install/repo` or whatever, since conceptually that's where it
came from.  We're just providing a better error.

Closes: https://github.com/ostreedev/ostree/issues/626

Closes: #627
Approved by: jlebon
2016-12-09 17:46:54 +00:00
Colin Walters
66da1199f0 tests/keyfile-utils: Drop tests covering preconditions
The spam in stderr was bothering me, and further at some eventual
point in the future we want to annotate the functions with
`__attribute__((nonnull))` which would then cause tests like these to
become undefined behavior.

The coverage of this isn't worth the log spam basically.

Closes: #611
Approved by: jlebon
2016-12-06 14:50:08 +00:00
Colin Walters
e1c48c8adb [ASAN] tests: Cleanup all current remaining leaks
We now run fully through ASAN here.

Closes: #609
Approved by: jlebon
2016-12-05 15:20:56 +00:00
Colin Walters
7bb0ff46a4 Define and use cleanup helpers for libarchive
This should fix some of the ASAN leaks around libarchive usage,
and is generally better.

Closes: #609
Approved by: jlebon
2016-12-05 15:20:56 +00:00
Colin Walters
62b94635bc [ASAN] tests: Fix leaks
Just for cleaner sanitizer output.

Closes: #598
Approved by: jlebon
2016-11-30 18:51:26 +00:00
Simon McVittie
d1c7eba82c travis-ci: document parameter variables
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
3d6269ebfb travis-ci: Run make distcheck too
ostree is now actively using that mode.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
a1ab6b50f9 travis-ci: cat the test log after successful test runs
This lets us see which tests were skipped.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
db691b5520 travis-ci: Move helper function to before we start building anything
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
49b8ccd8cb travis-ci: Use a non-ostree-specific name for the Docker image
This reduces the diff when comparing these scripts with similar glue
in dbus or elsewhere.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
0473fb8523 travis-ci: put an explicit copyright/license on the scripts
This is deliberately permissive: a lot of it is generic, and I'm
using similar scripts in dbus.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
9026d5806f ci-install: add ci_distro
Otherwise, we'll fail (due to set -u) if this parameter variable isn't
passed.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
54655795c9 ci-build: consistently use yes/no for booleans, not yes/empty
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Jasper St. Pierre
fd6ba80d07 ostree-repo: Make the lock with a long-lasting FD
glnx_make_lock_file requires that the dfd passed in survives the
lifetime of the lock. Since dfd_iter.fd gets cleaned up after the
function returns, this isn't the case. dfd_iter.fd should be equivalent
to tmpdir_dfd, since we iter on ".", and that survives past the
function, so just use that instead.

Closes: #591
Approved by: cgwalters
2016-11-22 02:32:33 +00:00
Colin Walters
eb7ba645af [ASAN] tests: Fix some memleaks in libarchive importer
Caught by `-fsanitize=address`.

Closes: #587
Approved by: jlebon
2016-11-21 16:34:06 +00:00
Colin Walters
41ef2aeb38 pull: Do GPG verify commit objects when using deltas
The fact that we weren't doing this is at best an oversight, and
for some deployment models a security vulnerability.  Having both
`gpg-verify` and `gpg-verify-summary` shows that we were intending
them to be orthogonal/independent.

Lately I've been advocating moving towards pinned TLS instead of
gpg-signed summaries, and if we follow that path, performing GPG
verification of commit objects even if using deltas is more important,
as it provides an at-rest verifiable authenticity and integrity
mechanism.

Content providers which are signing their summary files and/or using
TLS (particularly pinned TLS) for transport should treat this as a
nice-to-have.  However, for providers which are serving content over
plain HTTP and relying on GPG, this is a critical update.

Closes: https://github.com/ostreedev/ostree/issues/517

Closes: #589
Approved by: jlebon
2016-11-21 15:55:09 +00:00
William Manley
0ee9e221be ostree commit: Fix combining trees with multiple --tree=ref arguments
You'd expect

    ostree commit --tree=ref=A --tree=ref=B

to produce a commit with the union of the trees given.  Instead you'd get
a commit with the contents of just the latter commit.  This was due to an
optimisation where we'd skip filling out the `files` and `subdirs`
members of the mtree, just filling in the metadata instead.  This backfires
becuase this same code relies on checking the `files` and `subdirs` members
itself to work out whether the mtree is empty.

This commit removes the optimisation, fixing the bug.  Maybe there's a way
to keep the optimisation and still fix the bug but it's not obvious to
me.

Closes: #581
Approved by: cgwalters
2016-11-17 14:45:55 +00:00
Colin Walters
24bf257ee9 lib: Add an API to GPG verify a commit given a remote
Conceptually we've been moving towards having our GPG verification
paths be per-remote.  The code internally supports this, but we
didn't expose an API to use it conveniently.

This came up when trying to add a new `gpgkeypath` option, since
right now rpm-ostree manually finds keyrings for the remote, and
hence it wasn't looking at the keypath, and said "Unknown key"
in status.

Adding an API fixes this nicely.

Closes: #576
Approved by: giuseppe
2016-11-17 11:33:41 +00:00
Colin Walters
f244c70277 Add "gpgkeypath" option to remotes
For Project Atomic, we already have RPM signatures which use files in
`/etc/pki/rpm-gpg`.  It's convenient to simply bind the OSTree remote
configuration to those file paths, rather than having duplicate key
data.

This does mean that we need to parse the files for verification, so we
end up importing them into the verifier's temporary keyring, which is
a bit ugly, but it's what other projects do.

Closes: https://github.com/ostreedev/ostree/issues/573

Closes: #575
Approved by: giuseppe
2016-11-17 09:44:07 +00:00
Alexander Larsson
bd45e7ac19 commit: Fix reading xattrs from OstreeRepoFile:s
When doing commit --tree=ref=XXX while at the same time applying some
form of modifier, ostree dies trying to read the xattrs using the
raw syscalls. We fix this by falling back to ostree_repo_file_get_xattrs()
in this case.

Also adds a testcase for this.

Closes: #577
Approved by: cgwalters
2016-11-16 22:30:33 +00:00
Colin Walters
37c07d2f1c pull: Add support for http-headers option
Some deployments may want to gate access to content based on things
like OAuth.  In this model, the client system would normally compute a
token and pass it to the server via an API.

We could theoretically support this in the remote config too, but
that'd be a bit weird for OAuth as the information is dynamic.
Therefore this cleans up the code a little bit to more clearly handle
the case that the fetcher is initialized from both remote config
data plus pull options.

Closes: #574
Approved by: giuseppe
2016-11-16 10:04:22 +00:00
Sjoerd Simons
6af8db6fc4 tests: Add test for the cookie jar handling
Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Colin Walters
2139f0e437 pull: Don't do deltas with --commit-metadata-only
We should just download the commit objects directly, as it's
obviously a lot more efficient than deltas.

I had to generate a summary file in more places in the tests,
since once created, it needs to be updated.

Closes: https://github.com/ostreedev/ostree/issues/528

Closes: #566
Approved by: jlebon
2016-11-04 16:04:23 +00:00
Sjoerd Simons
f0e493bf29 Filter bootloader supplied kernel cmdline options
Various bootloader add kernel commandline options dynamically, filter
these out when grabbing boot options from /proc/cmdline. Specifically
grub adds BOOT_IMAGE and systemd-boot adds initrd.

Closes: #560
Approved by: cgwalters
2016-11-01 18:00:37 +00:00
Simon McVittie
7091d288f7 Force C.UTF-8 or C locale for tests
Otherwise several tests fail, for example in this build done in a French
locale by Debian's reproducible builds initiative, to check whether
the resulting binaries are identical to what was produced in an
English locale:

<https://tests.reproducible-builds.org/debian/logs/unstable/amd64/ostree_2016.11-1.build2.log.gz>

(test-basic)
# error: Cannot write to repository: Permission non accordée
...
File 'error-message' doesn't match regexp 'Permission denied'

(test-help)
# Utilisation :
#   ostree [OPTION...] COMMAND
...
File 'out' doesn't match regexp '[Uu]sage'

(test-pull-metalink)
# error: Erreur à la ligne 1, caractère 1 : Le document doit commencer avec un élément (par ex. <book>)
...
File 'err.txt' doesn't match regexp 'Document must begin with an element'

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #558
Approved by: cgwalters
2016-10-29 18:12:42 +00:00
Colin Walters
b77edf24a3 tree-wide: Remove unused variables detected by CLang
CLang finds these, whereas GCC treats having
`__attribute__((cleanup))` as a use.

This obsoletes https://github.com/ostreedev/ostree/pull/411

Closes: #548
Approved by: jlebon
2016-10-27 17:02:01 +00:00
Colin Walters
82a4f56593 tests: Skip libarchive/selinux tests if in container without SELinux
I'm doing builds and `make check` inside a Docker container, with
selinux on as a build-time option, but no policy in the container.
This currently aborts.  Let's not do that.

(This type of thing is why installed tests are a better model)

Closes: #546
Approved by: jlebon
2016-10-27 16:48:34 +00:00
Alexander Larsson
d0e74cf3af Fix pruning of partial commits
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.

In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.

Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.

This fixes https://github.com/ostreedev/ostree/issues/541

Closes: #542
Approved by: cgwalters
2016-10-24 17:48:19 +00:00
Alexander Larsson
ee484697cd pull: Support multiple specifications of --subpath
I need this in flatpak to avoid doing multiple pulls when doing
locale subsetting.

Closes: #523
Approved by: cgwalters
2016-10-11 16:41:03 +00:00
Jonathan Lebon
12e916466c static-delta: add some error handling
We make _ostree_parse_delta_name() a bit more defensive since it handles
user input.

Closes: #504

Closes: #505
Approved by: cgwalters
2016-09-09 19:06:11 +00:00
Colin Walters
845dc65196 repo: Revert default timestamp from 1 back to 0
Quoting Dan Nicholson in

  <https://github.com/ostreedev/ostree/pull/330#issuecomment-245499099>

  mtime of 0 has been the semantics of ostree deployments from basically
  the beginning of the project. We (and others, see
  flatpak/flatpak@b5204c9) rely on that fact when generating trees.

  In particular, this affects caches that use the mtime of the
  associated file or directory to determine if the cache is valid. By
  arbitrarily changing the mtime of the files to something else, all
  the caches we setup in the build are now invalidated. Preseeding
  caches is really important to the user experience as it avoids
  having the user wait while they're regenerated on first run.

  Now, we could change our build infrastructure to preset all the
  mtimes to 1 to match this change, but what does that do for our
  existing users who are on an ostree that deploys with mtimes of 0?
  We could just revert this change at Endless (and the associated one
  in Flatpak), and that would be fine for our users. However, if we
  point non-Endless users to our apps, they'll have the great
  experience of waiting 10 seconds the first time they launch it while
  the fontconfig cache is rebuilt unnecessarily.

Closes: #495
Approved by: jlebon
2016-09-08 13:35:59 +00:00