513 Commits

Author SHA1 Message Date
Colin Walters
d3385a3014 checkout: Provide useful error with checkout -H and incompat mode
Previously we'd assert and dump core if one used `checkout -H` without
`-U` on a bare-user repo, because we'd hit the bare-user symlink case.

Rework the code to handle this, and add tests. I hit this when I was going to
suggest to someone to use `-H` to ensure they were getting hardlinks.

Closes: #779
Approved by: jlebon
2017-04-12 17:06:44 +00:00
Colin Walters
c937305c0e core: Fix default value of disable_xattrs
Sigh.  Rather awful regression from https://github.com/ostreedev/ostree/pull/759

Closes: #775
Approved by: jlebon
2017-04-04 15:54:46 +00:00
Colin Walters
b74e4e79cc ci: Add a check that submodule changes include "Update submodule: "
To prevent repeats of https://github.com/ostreedev/ostree/pull/693

I tested this script in https://github.com/cgwalters/playground/pull/48

Closes: #770
Approved by: jlebon
2017-04-04 15:44:19 +00:00
Colin Walters
305db981d4 Add Coccinelle usage: one for blacklisting, one for patch collection
This is inspired by the [Coccinelle](http://coccinelle.lip6.fr/) usage
in systemd.  I also took it a bit further and added infrastructure
to have spatches which should never apply.  This acts as a blacklist.

The reason to do the latter is that coccinelle is *way* more powerful than the
regular expresssions we have in `make syntax-check`.

I started with blacklisting `g_error_free()` directly. The reason that's bad is
it leaves a dangling pointer.

Closes: #754
Approved by: jlebon
2017-03-30 19:19:54 +00:00
Alexander Larsson
7c8f95c86f Add basic tests for bare-user-only repo modes
This is somewhat complicated by such repos only properly supporting
some subset of file metadata (uid/gid 0, etc). We fix this by
always commiting with filters that make it work.

Closes: #750
Approved by: cgwalters
2017-03-27 13:48:41 +00:00
Colin Walters
455cc5e892 repo+tests: Add [core]disable-xattrs=true, use it on overlayfs
There are a lot of things suboptimal about this approach, but
on the other hand we need to get our CI back up and running.

The basic approach is to - in the test suite, detect if we're on overlayfs. If
so, set a flag in the repo, which gets picked up by a few strategic places in
the core to turn on "ignore xattrs".

I also had to add a variant of this for the sysroot work.

The core problem here is while overlayfs will let us read and
see the SELinux labels, it won't let us write them.

Down the line, we should improve this so that we can selectively ignore e.g.
`security.*` attributes but not `user.*` say.

Closes: https://github.com/ostreedev/ostree/issues/758

Closes: #759
Approved by: jlebon
2017-03-24 22:16:43 +00:00
Erik Larsson
e665e51408 diff: Add ostree_diff_dirs_with_options(), expose via cmdline
The first options are owner_uid/owner_gid, which makes it possible to use diff
on local files where --owner-uid/gid have been passed to commit.

Closes: #740
Approved by: cgwalters
2017-03-21 13:38:04 +00:00
André Klitzing
f72726c7d8 Avoid unnecessary includes
"return" will do the same here. Also that style is used at the
end of the function.

Closes: #732
Approved by: jlebon
2017-03-10 21:46:50 +00:00
Colin Walters
94948e3522 checkout: Support a "pure addition" mode
I plan to use this for `rpm-ostree livefs`.
https://github.com/projectatomic/rpm-ostree/issues/639

Closes: #714
Approved by: jlebon
2017-03-06 20:58:04 +00:00
Colin Walters
ff34810097 repo/checkout: Verify early if src/destination are on same device
At least in all Linux kernels up to today, one can never `link()` across
devices, so we might as well verify that up front. This will help for a future
patch to add a new type of union-add checkout, since Linux checks for `EEXIST`
before `EXDEV`.

Closes: #714
Approved by: jlebon
2017-03-06 20:58:04 +00:00
Colin Walters
a787e0c072 pull: Use all available commits for delta sources
The previous logic for static deltas was to use as a FROM
revision the current branch tip.  However, we want
to support deltas between branches in an automatic
fashion.

If a summary file is available, we already have an
enumerated list of deltas - so the logic introduced
here is to search it, and find the newest commit
we have locally that matches the TO revision target.

This builds on some thoughts from
https://github.com/ostreedev/ostree/pull/151#issuecomment-232390232

Closes: https://github.com/ostreedev/ostree/pull/151

Closes: #710
Approved by: giuseppe
2017-03-04 10:11:14 +00:00
Jonathan Lebon
199646ccfa pull: don't use static deltas if archive repo
In https://github.com/ostreedev/ostree/pull/408, we disabled the use of
static deltas when mirroring. Later,
https://github.com/ostreedev/ostree/pull/506 loosened this up again so
that we could use static deltas when mirroring into bare{-user} repos.

However, the issue which originally spurrred #408 is even more generic
than that: we want to avoid static deltas for any archive repo, not just
when doing a mirror pull. This patch tightens this up, and also
relocates the decision code to make it easier to read.

Closes: #715
Approved by: cgwalters
2017-03-02 20:35:24 +00:00
Gatis Paeglis
64422a7d0b deltas: Expose the filename parameter
The C API (ostree_repo_static_delta_generate) knows what to do
with it, but this parameter was never exposed via command line
tool.

Closes: https://github.com/ostreedev/ostree/issues/695

Closes: #703
Approved by: jlebon
2017-02-27 22:10:26 +00:00
Colin Walters
36b28cb4d2 upgrade: Add support for --pull-only and --deploy-only
This makes it easier to script downloading updates in the background,
and only do deployments just before rebooting.

Partially addresses https://github.com/ostreedev/ostree/issues/640

Closes: #642
Approved by: jlebon
2017-02-27 20:57:23 +00:00
Colin Walters
09b392675a main: Make ostree --version output YAML (and add gitrev)
I learned today that `docker version` does this and I really like
the idea.  While we have the patient open, also add the gitrev
with code taken from https://github.com/projectatomic/rpm-ostree/pull/584

Closes: #691
Approved by: giuseppe
2017-02-22 18:57:18 +00:00
Colin Walters
f02dcc4997 libtest: Re-enable quiet mode for building fs tree
I think I commented this out while debugging something, and forgot to re-enable
it. Reading the log files should be a better again after this.

Closes: #699
Approved by: giuseppe
2017-02-21 15:35:23 +00:00
Colin Walters
b5c5003ff6 pull: Fold together deltapart+fallback count for display
It's just simpler, and I'm not sure people are going to care
much about the difference by default.

We already folded in the fallback sizes into the download totals, so folding in
the count makes things consistent; previously you could see e.g.
`3/3 parts, 100MB/150MB` and be confused.

Closes: #678
Approved by: giuseppe
2017-02-17 14:58:25 +00:00
Colin Walters
46544f5b4d commit: Support -F/--body-file, like git
This is more convenient to script for projects which haven't
yet made the leap to using the API.

Closes: https://github.com/ostreedev/ostree/issues/674

Closes: #681
Approved by: jlebon
2017-02-14 14:15:08 +00:00
Colin Walters
361aa449fb libcurl backend
For rpm-ostree, we already link to libcurl indirectly via librepo, and
only having one HTTP library in process makes sense.

Further, libcurl is (I think) more popular in the embedded space.  It
also supports HTTP/2.0 today, which is a *very* nice to have for OSTree.

This seems to be working fairly well for me in my local testing, but it's
obviously brand new nontrivial code, so it's going to need some soak time.

The ugliest part of this is having to vendor in the soup-url code. With
Oxidation we could follow the path of Firefox and use the
[Servo URL parser](https://github.com/servo/rust-url).  Having to redo
cookie parsing also sucked, and that would also be a good oxidation target.

But that's for the future.

Closes: #641
Approved by: jlebon
2017-02-09 16:37:45 +00:00
Colin Walters
ecf5c079ea tests: Add setup for more realistic repo, change pull-many to use
As OSTree has evolved over time, the tests grew with it.  We
didn't start out with static deltas or a summary file, and the
tests reflect this.

What I really want to do is change more of the pull tests, from
corruption/proxying/mirroring etc. to use this more realistic
repo rather than the tiny one the other test creates.

We start by using some of the code from `test-pull-many.sh`, and change that
test to use `--disable-static-deltas` for pull, since the point of that test is
to *not* test deltas.

Still TODO is investigate changing other tests to use this.

Closes: #658
Approved by: jlebon
2017-02-02 17:04:31 +00:00
Colin Walters
5eea1cdad8 lib: Move the bupsplit selftest into our test framework
We weren't running it before. Also I switched it to use GLib. Preparation for
some oxidation work (having an implementation of bupsplit in Rust).

I exported another function to do the raw rollsum operation which is what this
test suite uses.

Closes: #655
Approved by: jlebon
2017-02-02 16:51:36 +00:00
Colin Walters
d1f4d94720 tests: Add a big (many objects) pull
This would be more likely to tickle things like
https://github.com/ostreedev/ostree/issues/601
reliably.

Also, while working on the curl backend, I hit on the fact that curl doesn't
queue (by default, you can enable) and will happily create 20000+ concurrent TCP
connections if you try. Having this test would have made that more likely to
fail.

Closes: #650
Approved by: giuseppe
2017-01-23 17:29:02 +00:00
Colin Walters
5c940987e7 Add support for more selective pruning
There are use cases for having a single repo with branches
with different lifecycles; a simple example of what I was
trying to do in CentOS Atomic Host work is have "stable"
and "devel" branches, were we want to prune devel, but
retain *all* of stable.

This patch is split into two parts - first we add a low level "delete all
objects not in this set" API, and change the current prune API
to use this.

Next, we move more logic into the "ostree prune" command. This paves the way for
demonstrating how more sophisticated algorithms/logic could be developed outside
of the ostree core.

Also, the --keep-younger-than logic already lived in the commandline, so it
makes sense to keep extending it there.

Closes: https://github.com/ostreedev/ostree/issues/604

Closes: #646
Approved by: jlebon
2017-01-19 16:28:00 +00:00
Simon McVittie
9d94fc40c8 Make corrupt-repo-ref.js executable
Debian's Lintian packaging consistency check complains that it isn't
executable but has a #! line. In fact it's reasonable to run this
script directly, so make it executable, and put it in a _scripts
variable so it will be installed executable.

Closes: #652
Approved by: cgwalters
2017-01-19 13:54:59 +00:00
Simon McVittie
9a3f82caae Sourced test snippets: remove shebang and make non-executable
They are installed non-executable, which makes Debian's Lintian
packaging consistency check complain that #! is only useful
in executable scripts. But in fact they are not useful to execute
directly (they rely on setup being done in the script that sources
them), so just chmod them -x.

Closes: #652
Approved by: cgwalters
2017-01-19 13:54:59 +00:00
Simon McVittie
01fb30b839 Fix TAP syntax in test-basic-user.sh, and run it
In its initial commit, Alexander Larsson wrote

    This works standalone, but unfortunately it breaks in
    gnome-desktop-testing-runner as /tmp doesn't support
    xattrs, so it is not installed atm.

but we now (a) use /var/tmp, and (b) explicitly skip the test if
xattr support is unavailable. So it should be OK to run now.

Closes: #652
Approved by: cgwalters
2017-01-19 13:54:59 +00:00
Colin Walters
b260fa764c libtest: Enable web server logs
Now that we're daemonizing, it's useful to have the logs.  I
wanted this while debugging cookies.

Closes: #651
Approved by: giuseppe
2017-01-19 10:47:15 +00:00
Colin Walters
686f91062f tests: Loosen error regexp
libcurl AFAICS doesn't have an API to convert HTTP code ➡️ error
string, so let's make the test regexp operate on both.

Closes: #651
Approved by: giuseppe
2017-01-19 10:47:15 +00:00
Colin Walters
56891f9d48 tests: Don't inject newline in URL
It turns out libsoup strips all whitespace even *inside* a URL. We could do that
for libcurl too but...really, people shouldn't do that. In this test we were
adding the trailing newline into the URL. If someone complains who is using the
libcurl code we can deal with it then.

Closes: #651
Approved by: giuseppe
2017-01-19 10:47:15 +00:00
Colin Walters
2f71136eec tests: Alias assert_not_reached() -> fatal()
We had a lot of copies of the "echo something 1>&2; exit 1" code even though
`assert_not_reached()` was it.  Hence, I think we need a shorter alias for that.

Doing this particularly since I noticed a missing `1` in an `exit 1` call in the
rpm-ostree copy of this.

Closes: #648
Approved by: jlebon
2017-01-18 14:28:29 +00:00
Colin Walters
ced22f6c9b Split trivial-httpd into separate binary
Working on the libcurl backend, I hit the issue that the trivial-httpd program
depends on libsoup. I briefly considered having two versions, but libcurl is
client only, and moreover trivial-httpd is no longer trivial - it has various
features which are used by the test suite extensively.

Hence, what we'll do is build it as a separate binary which links to libsoup,
and use it during the tests. We *also* currently still provide `ostree
trivial-httpd` since some things use it like `rpm-ostree-toolbox` and the
Cockpit tests.

After those are ported to use some other webserver, I plan to add a build-time
option to drop it.

Closes: #636
Approved by: jlebon
2017-01-04 16:32:11 +00:00
Colin Walters
359ea9b58f tests: Tweak installed tests to deal with ASAN
We need to disable readdir-rand there too.

Closes: #622
Approved by: jlebon
2016-12-09 18:05:53 +00:00
Colin Walters
17f264a487 repo: Add unconfigured-state to remote config options
This is a migration from the origin version.  It's
nicer to have it in the remote, since that's what one
needs to change.  Then tools don't need to mess with
the origin file.o

In fact in this scenario one can keep the "media source" like
`file:///install/repo` or whatever, since conceptually that's where it
came from.  We're just providing a better error.

Closes: https://github.com/ostreedev/ostree/issues/626

Closes: #627
Approved by: jlebon
2016-12-09 17:46:54 +00:00
Colin Walters
66da1199f0 tests/keyfile-utils: Drop tests covering preconditions
The spam in stderr was bothering me, and further at some eventual
point in the future we want to annotate the functions with
`__attribute__((nonnull))` which would then cause tests like these to
become undefined behavior.

The coverage of this isn't worth the log spam basically.

Closes: #611
Approved by: jlebon
2016-12-06 14:50:08 +00:00
Colin Walters
e1c48c8adb [ASAN] tests: Cleanup all current remaining leaks
We now run fully through ASAN here.

Closes: #609
Approved by: jlebon
2016-12-05 15:20:56 +00:00
Colin Walters
7bb0ff46a4 Define and use cleanup helpers for libarchive
This should fix some of the ASAN leaks around libarchive usage,
and is generally better.

Closes: #609
Approved by: jlebon
2016-12-05 15:20:56 +00:00
Colin Walters
62b94635bc [ASAN] tests: Fix leaks
Just for cleaner sanitizer output.

Closes: #598
Approved by: jlebon
2016-11-30 18:51:26 +00:00
Simon McVittie
d1c7eba82c travis-ci: document parameter variables
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
3d6269ebfb travis-ci: Run make distcheck too
ostree is now actively using that mode.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
a1ab6b50f9 travis-ci: cat the test log after successful test runs
This lets us see which tests were skipped.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
db691b5520 travis-ci: Move helper function to before we start building anything
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
49b8ccd8cb travis-ci: Use a non-ostree-specific name for the Docker image
This reduces the diff when comparing these scripts with similar glue
in dbus or elsewhere.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
0473fb8523 travis-ci: put an explicit copyright/license on the scripts
This is deliberately permissive: a lot of it is generic, and I'm
using similar scripts in dbus.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
9026d5806f ci-install: add ci_distro
Otherwise, we'll fail (due to set -u) if this parameter variable isn't
passed.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
54655795c9 ci-build: consistently use yes/no for booleans, not yes/empty
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Jasper St. Pierre
fd6ba80d07 ostree-repo: Make the lock with a long-lasting FD
glnx_make_lock_file requires that the dfd passed in survives the
lifetime of the lock. Since dfd_iter.fd gets cleaned up after the
function returns, this isn't the case. dfd_iter.fd should be equivalent
to tmpdir_dfd, since we iter on ".", and that survives past the
function, so just use that instead.

Closes: #591
Approved by: cgwalters
2016-11-22 02:32:33 +00:00
Colin Walters
eb7ba645af [ASAN] tests: Fix some memleaks in libarchive importer
Caught by `-fsanitize=address`.

Closes: #587
Approved by: jlebon
2016-11-21 16:34:06 +00:00
Colin Walters
41ef2aeb38 pull: Do GPG verify commit objects when using deltas
The fact that we weren't doing this is at best an oversight, and
for some deployment models a security vulnerability.  Having both
`gpg-verify` and `gpg-verify-summary` shows that we were intending
them to be orthogonal/independent.

Lately I've been advocating moving towards pinned TLS instead of
gpg-signed summaries, and if we follow that path, performing GPG
verification of commit objects even if using deltas is more important,
as it provides an at-rest verifiable authenticity and integrity
mechanism.

Content providers which are signing their summary files and/or using
TLS (particularly pinned TLS) for transport should treat this as a
nice-to-have.  However, for providers which are serving content over
plain HTTP and relying on GPG, this is a critical update.

Closes: https://github.com/ostreedev/ostree/issues/517

Closes: #589
Approved by: jlebon
2016-11-21 15:55:09 +00:00
William Manley
0ee9e221be ostree commit: Fix combining trees with multiple --tree=ref arguments
You'd expect

    ostree commit --tree=ref=A --tree=ref=B

to produce a commit with the union of the trees given.  Instead you'd get
a commit with the contents of just the latter commit.  This was due to an
optimisation where we'd skip filling out the `files` and `subdirs`
members of the mtree, just filling in the metadata instead.  This backfires
becuase this same code relies on checking the `files` and `subdirs` members
itself to work out whether the mtree is empty.

This commit removes the optimisation, fixing the bug.  Maybe there's a way
to keep the optimisation and still fix the bug but it's not obvious to
me.

Closes: #581
Approved by: cgwalters
2016-11-17 14:45:55 +00:00
Colin Walters
24bf257ee9 lib: Add an API to GPG verify a commit given a remote
Conceptually we've been moving towards having our GPG verification
paths be per-remote.  The code internally supports this, but we
didn't expose an API to use it conveniently.

This came up when trying to add a new `gpgkeypath` option, since
right now rpm-ostree manually finds keyrings for the remote, and
hence it wasn't looking at the keypath, and said "Unknown key"
in status.

Adding an API fixes this nicely.

Closes: #576
Approved by: giuseppe
2016-11-17 11:33:41 +00:00