3076 Commits

Author SHA1 Message Date
Colin Walters
1bb6e51486 [ASAN] sysroot: Fix leak/double free of keyfile origin
Use autoptr rather than manual cleanup.  The double free isn't a
security problem, since we trust origin files.

Closes: #598
Approved by: jlebon
2016-11-30 18:51:26 +00:00
Colin Walters
3a459bac2d traverse: Use g_hash_table_add
And "move semantics" via `g_steal_pointer()`.  Just a minor code
cleanup I noticed when I was hunting for a leak, which ended up being
elsewhere.

Closes: #598
Approved by: jlebon
2016-11-30 18:51:26 +00:00
Colin Walters
a1224e2de7 [ASAN] cmdline: Fix minor leak in delta cmdline entrypoint
Small, but it's important to stay clean.

Closes: #598
Approved by: jlebon
2016-11-30 18:51:26 +00:00
Colin Walters
ab0400b722 [ASAN] deltas: Fix minor memory leak
We were leaking the checksum, ensure we free it in both normal and
error paths.

Closes: #598
Approved by: jlebon
2016-11-30 18:51:26 +00:00
Colin Walters
79cb421ee2 [ASAN] delta compilation: More leak fixes
Now that I remembered to do `env G_SLICE=always-malloc`, lots more
leaks become apparent.  Nothing major.

Closes: #598
Approved by: jlebon
2016-11-30 18:51:26 +00:00
Mario Sanchez Prada
340376e7b1 man: Mention bare-user in manpages, along with the other modes
Closes: #602

Closes: #603
Approved by: cgwalters
2016-11-30 15:47:24 +00:00
Simon McVittie
d1c7eba82c travis-ci: document parameter variables
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
9cbe23f4b7 travis-ci: Enable stretch (the future Debian 9), replacing unstable
My goal in building ostree for Debian unstable was that we would
have good coverage of "new code" paths. However, it was removed
for #571 as too much of a moving target. Debian testing is less of
a moving target, and in particular is always internally consistent
(packages are co-installable), which Debian unstable is not guaranteed
to be.

Debian 'stretch' is the future Debian 9, which should be released
next year.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
0261560b35 travis-ci: Use "slim" Debian image for testing
Documentation and similar files are stripped from this image, making
it quicker to install.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
3d6269ebfb travis-ci: Run make distcheck too
ostree is now actively using that mode.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
a1ab6b50f9 travis-ci: cat the test log after successful test runs
This lets us see which tests were skipped.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
db691b5520 travis-ci: Move helper function to before we start building anything
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
49b8ccd8cb travis-ci: Use a non-ostree-specific name for the Docker image
This reduces the diff when comparing these scripts with similar glue
in dbus or elsewhere.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
0473fb8523 travis-ci: put an explicit copyright/license on the scripts
This is deliberately permissive: a lot of it is generic, and I'm
using similar scripts in dbus.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
9026d5806f ci-install: add ci_distro
Otherwise, we'll fail (due to set -u) if this parameter variable isn't
passed.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
54655795c9 ci-build: consistently use yes/no for booleans, not yes/empty
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Simon McVittie
a73fef3eaa build: clean up ostree-remount if building without systemd
This is necessary for "make distcheck" on Travis-CI.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Closes: #600
Approved by: cgwalters
2016-11-29 14:11:50 +00:00
Alexander Larsson
d57036f6a2 delta compilation: Fix leak
We need to ref-sik the new varian for g_autoptr to work

Closes: #597
Approved by: cgwalters
2016-11-28 16:25:29 +00:00
Alexander Larsson
36f7824cb0 pull: Don't leak delta superblock variants
Closes: #596
Approved by: cgwalters
2016-11-28 15:56:22 +00:00
Alexander Larsson
239cb1494f pull_with_options: Don't leak csum_v
Closes: #596
Approved by: cgwalters
2016-11-28 15:56:22 +00:00
Alexander Larsson
fc107e5bb4 ostree-repo-traverse: Don't leak floating GVariant
ostree_object_name_serialize returns a floating ref, so we need
to sink it before putting in the hashtable.

Closes: #595
Approved by: cgwalters
2016-11-28 14:53:50 +00:00
Alexander Larsson
c9b158e9f2 pull: scan_commit_object() - don't load variant twice
ostree_repo_load_commit already loaded the object, no need
to load it twice.

Closes: #595
Approved by: cgwalters
2016-11-28 14:53:50 +00:00
Jasper St. Pierre
70d140d61c ostree-repo-traverse: Remove an accidental print statement
Closes: #594
Approved by: jlebon
2016-11-23 22:21:16 +00:00
Colin Walters
7584dc0f25 Release 2016.14
Closes: #593
Approved by: jlebon
v2016.14
2016-11-23 16:20:49 +00:00
Abhay Kadam
13360cb1c4 Fix broken link in docs/CONTRIBUTING.md
The link for git-rebase manual contains stray character (])
at the end.

Closes: #592
Approved by: jlebon
2016-11-22 14:00:15 +00:00
Jasper St. Pierre
fd6ba80d07 ostree-repo: Make the lock with a long-lasting FD
glnx_make_lock_file requires that the dfd passed in survives the
lifetime of the lock. Since dfd_iter.fd gets cleaned up after the
function returns, this isn't the case. dfd_iter.fd should be equivalent
to tmpdir_dfd, since we iter on ".", and that survives past the
function, so just use that instead.

Closes: #591
Approved by: cgwalters
2016-11-22 02:32:33 +00:00
Jasper St. Pierre
fbce608177 ostree-repo-static-delta-processing: Don't close(-1)
Ultimately harmless, but causes somewhat scary strace messages.

Closes: #591
Approved by: cgwalters
2016-11-22 02:32:33 +00:00
Jasper St. Pierre
a8f5c20209 ostree-repo: Fix parameter name
Closes: #591
Approved by: cgwalters
2016-11-22 02:32:33 +00:00
Colin Walters
49b8a72622 [ASAN] lib: Squash various leaks in library and commandline
The pull one is the most likely to affect users.  Otherwise mostly
just cleaning up `-fsanitize=address`.

Closes: #587
Approved by: jlebon
2016-11-21 16:34:06 +00:00
Colin Walters
eb7ba645af [ASAN] tests: Fix some memleaks in libarchive importer
Caught by `-fsanitize=address`.

Closes: #587
Approved by: jlebon
2016-11-21 16:34:06 +00:00
Colin Walters
a6eb8bbcf6 tests: Support TEST_SKIP_CLEANUP=err
I find myself often wanting to debug interactively failing tests.
This makes it more convenient to keep around the temporary directories
just for those tests, rather than accumulating tons of tempdirs from
the successful tests as well.

Closes: #588
Approved by: jlebon
2016-11-21 16:11:55 +00:00
Colin Walters
41ef2aeb38 pull: Do GPG verify commit objects when using deltas
The fact that we weren't doing this is at best an oversight, and
for some deployment models a security vulnerability.  Having both
`gpg-verify` and `gpg-verify-summary` shows that we were intending
them to be orthogonal/independent.

Lately I've been advocating moving towards pinned TLS instead of
gpg-signed summaries, and if we follow that path, performing GPG
verification of commit objects even if using deltas is more important,
as it provides an at-rest verifiable authenticity and integrity
mechanism.

Content providers which are signing their summary files and/or using
TLS (particularly pinned TLS) for transport should treat this as a
nice-to-have.  However, for providers which are serving content over
plain HTTP and relying on GPG, this is a critical update.

Closes: https://github.com/ostreedev/ostree/issues/517

Closes: #589
Approved by: jlebon
2016-11-21 15:55:09 +00:00
Colin Walters
cb57338a12 pull: Use new per-remote API for GPG verification
Trivial change, but makes things more obvious.  And we get test
coverage of the new API for free.

Closes: #589
Approved by: jlebon
2016-11-21 15:55:09 +00:00
Colin Walters
abbd7acaf3 pull: Dedup code for checking for > 0 valid results
We have a public API for this, let's use it internally.

Closes: #589
Approved by: jlebon
2016-11-21 15:55:09 +00:00
Colin Walters
c1c70bceb7 [TSAN] Rework assertions to always access refcount atomically
`-fsanitize=address` complained that the `refcount > 0` assertions
were reading without atomics.  We can fix this by reworking them
to read the previous value.

Closes: #582
Approved by: jlebon
2016-11-17 19:41:57 +00:00
Colin Walters
f0519e541f [TSAN] main: Stop calling g_set_prgname()
It turns out this is basically racy with the presence of other
threads.  It was really cosmetic so let's stop doing it and make
`-fsanitize=thread` happy.

Closes: #582
Approved by: jlebon
2016-11-17 19:41:57 +00:00
Colin Walters
b5c4e6d99a [UBSAN] deltas: Don't call memset(NULL, NULL, 0) with no xattrs
This is actually fine in practice, but it triggers this
`-fsanitize=undefined` warning I saw in the test suite log:

```
src/libostree/ostree-repo-static-delta-compilation.c:160:10: runtime error: null pointer passed as argument 1, which is declared to never be null
```

Closes: #584
Approved by: jlebon
2016-11-17 19:20:20 +00:00
William Manley
0ee9e221be ostree commit: Fix combining trees with multiple --tree=ref arguments
You'd expect

    ostree commit --tree=ref=A --tree=ref=B

to produce a commit with the union of the trees given.  Instead you'd get
a commit with the contents of just the latter commit.  This was due to an
optimisation where we'd skip filling out the `files` and `subdirs`
members of the mtree, just filling in the metadata instead.  This backfires
becuase this same code relies on checking the `files` and `subdirs` members
itself to work out whether the mtree is empty.

This commit removes the optimisation, fixing the bug.  Maybe there's a way
to keep the optimisation and still fix the bug but it's not obvious to
me.

Closes: #581
Approved by: cgwalters
2016-11-17 14:45:55 +00:00
Colin Walters
24bf257ee9 lib: Add an API to GPG verify a commit given a remote
Conceptually we've been moving towards having our GPG verification
paths be per-remote.  The code internally supports this, but we
didn't expose an API to use it conveniently.

This came up when trying to add a new `gpgkeypath` option, since
right now rpm-ostree manually finds keyrings for the remote, and
hence it wasn't looking at the keypath, and said "Unknown key"
in status.

Adding an API fixes this nicely.

Closes: #576
Approved by: giuseppe
2016-11-17 11:33:41 +00:00
Colin Walters
f244c70277 Add "gpgkeypath" option to remotes
For Project Atomic, we already have RPM signatures which use files in
`/etc/pki/rpm-gpg`.  It's convenient to simply bind the OSTree remote
configuration to those file paths, rather than having duplicate key
data.

This does mean that we need to parse the files for verification, so we
end up importing them into the verifier's temporary keyring, which is
a bit ugly, but it's what other projects do.

Closes: https://github.com/ostreedev/ostree/issues/573

Closes: #575
Approved by: giuseppe
2016-11-17 09:44:07 +00:00
Colin Walters
3cd5e6b41a lib: Split out helper function to create GPG context
In prep for future work.

Closes: #575
Approved by: giuseppe
2016-11-17 09:44:07 +00:00
Colin Walters
a6cfe62eb8 lib: Define and use cleanup functions for gpgme
Just a cleanup in preparation for future work.

Closes: #575
Approved by: giuseppe
2016-11-17 09:44:07 +00:00
Alexander Larsson
bd45e7ac19 commit: Fix reading xattrs from OstreeRepoFile:s
When doing commit --tree=ref=XXX while at the same time applying some
form of modifier, ostree dies trying to read the xattrs using the
raw syscalls. We fix this by falling back to ostree_repo_file_get_xattrs()
in this case.

Also adds a testcase for this.

Closes: #577
Approved by: cgwalters
2016-11-16 22:30:33 +00:00
Colin Walters
814aa96825 pull: Redo logic for "scanning"
What in the code is called "scanning" is ensuring (potentially
recursively) have an object, and if not, fetching it.  And then if
it's metadata, parsing it and finding new objects to fetch.

This logic has grown fairly complex.  What I'm trying to fix
right now is that if we're doing a pull-local to a remote repository
via `sshfs` (FUSE) we still end up scanning, which is inefficient.

We can take advantage of the "commitpartial" logic here - if a commit
isn't partial, it's complete, hence we don't need to scan it.

At the same time, I'm changing the logic here to *always* do scans for
dirtree objects.  This will fix cases where multiple commits share
dirtree objects.  We have "commitpartial" metadata, but no such concept
of partial/complete for dirtrees.

But, we'll only ever scan dirtrees if we scan commits, which is
what the section above fixes.

Closes: https://github.com/ostreedev/ostree/issues/543

Closes: #564
Approved by: alexlarsson
2016-11-16 22:17:25 +00:00
Colin Walters
37c07d2f1c pull: Add support for http-headers option
Some deployments may want to gate access to content based on things
like OAuth.  In this model, the client system would normally compute a
token and pass it to the server via an API.

We could theoretically support this in the remote config too, but
that'd be a bit weird for OAuth as the information is dynamic.
Therefore this cleans up the code a little bit to more clearly handle
the case that the fetcher is initialized from both remote config
data plus pull options.

Closes: #574
Approved by: giuseppe
2016-11-16 10:04:22 +00:00
Jonathan Lebon
4b7ab5167c .redhat-ci.yml: no longer install libubsan & clang
Since they're now part of the auto-built image.

Closes: #572
Approved by: cgwalters
2016-11-10 18:10:58 +00:00
Colin Walters
676f0a2797 travis: Drop debian unstable since we can't fetch packages reliably
I don't know what's going on, I suspect mirror churn.  Anyways,
it seems to be consistently failing now, so let's drop it.

Closes: #571
Approved by: jlebon
2016-11-10 17:39:06 +00:00
Colin Walters
36c8946870 Release 2016.13
Closes: #570
Approved by: jlebon
v2016.13
2016-11-10 17:24:39 +00:00
Jonathan Lebon
39e7293e3f .redhat-ci.yml: use new build key
This allows us to more concisely separate building from testing, which
in turn gives us a nicer inheritance pattern in our case.

See also: https://github.com/jlebon/redhat-ci/issues/11

Closes: #569
Approved by: cgwalters
2016-11-09 17:35:40 +00:00
Colin Walters
24ac4ff190 deltas: Only keep one file open at a time during compilation
Otherwise it's possible for us to exhaust available file descriptors
or (on 32 bit) run up against mmap limits.

In the rollsum case, we didn't need to hold open the "from" object
at all.  And in the bsdiff case, we weren't even looking at either of
the files until we started processing.

Also, while we have the patient open, switch to using O_TMPFILE
if available.

Closes: #567
Approved by: giuseppe
2016-11-07 11:17:59 +00:00