2829 Commits

Author SHA1 Message Date
Colin Walters
36c8946870 Release 2016.13
Closes: #570
Approved by: jlebon
v2016.13
2016-11-10 17:24:39 +00:00
Jonathan Lebon
39e7293e3f .redhat-ci.yml: use new build key
This allows us to more concisely separate building from testing, which
in turn gives us a nicer inheritance pattern in our case.

See also: https://github.com/jlebon/redhat-ci/issues/11

Closes: #569
Approved by: cgwalters
2016-11-09 17:35:40 +00:00
Colin Walters
24ac4ff190 deltas: Only keep one file open at a time during compilation
Otherwise it's possible for us to exhaust available file descriptors
or (on 32 bit) run up against mmap limits.

In the rollsum case, we didn't need to hold open the "from" object
at all.  And in the bsdiff case, we weren't even looking at either of
the files until we started processing.

Also, while we have the patient open, switch to using O_TMPFILE
if available.

Closes: #567
Approved by: giuseppe
2016-11-07 11:17:59 +00:00
Sjoerd Simons
2b150f52f8 Update documentation for cookie handling commands
Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Sjoerd Simons
6af8db6fc4 tests: Add test for the cookie jar handling
Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Sjoerd Simons
6303e2d67b trivial-httpd: Add support for checking cookies
Allow passsing a list of cookie key/values to trivial-httpd which should
be provided to allow downloads

Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Sjoerd Simons
be9a3a7a19 OsreeFetcher: Treat 403 as not found
Private Cloudfront instances return 403 for objects which don't exist
rather then a 404.

Change the fetcher to assume 403 is ok for download that are "optional"
rather then erroring out at that step (e.g. trying to download a static
delta if the remote repo doesn't have those)

Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Sjoerd Simons
0613b4a479 remote: Add commands to add and remove cookies for a remote
Add commands to add and remove cookies to a remotes cookie jar.

Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Sjoerd Simons
6b467b9dbc remote: Add command to list cookies
Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Sjoerd Simons
96356aa192 pull: Add per-remote cookie jar
Optionally read cookie jars for a remote to be used when downloading
data. This can be used for private repositories which require specific
cookies to be present, e.g. repositories hosted on Amazon cloudfront
using signed cookies.

Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Colin Walters
2139f0e437 pull: Don't do deltas with --commit-metadata-only
We should just download the commit objects directly, as it's
obviously a lot more efficient than deltas.

I had to generate a summary file in more places in the tests,
since once created, it needs to be updated.

Closes: https://github.com/ostreedev/ostree/issues/528

Closes: #566
Approved by: jlebon
2016-11-04 16:04:23 +00:00
Colin Walters
730f723869 repo: Don't put remote refs in the summary file
I was doing a chain of mirroring like A -> B -> C

And repo B had A as a remote.  When I added B as
a remote to C, the summary file of B had a ref
upstream:foo/bar/baz, which caused all pulls from
B to C to fail, since the summary file is only
expected to have refs, not refspecs.

Closes: https://github.com/ostreedev/ostree/issues/561

Closes: #565
Approved by: jlebon
2016-11-04 15:16:09 +00:00
Sjoerd Simons
f0e493bf29 Filter bootloader supplied kernel cmdline options
Various bootloader add kernel commandline options dynamically, filter
these out when grabbing boot options from /proc/cmdline. Specifically
grub adds BOOT_IMAGE and systemd-boot adds initrd.

Closes: #560
Approved by: cgwalters
2016-11-01 18:00:37 +00:00
Simon McVittie
8ae03d6497 load_metadata_internal: don't leak GBytes
Found by valgrind memcheck. g_variant_new_from_bytes takes a ref to the
bytes, so we need to release the original ref.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #556
Approved by: cgwalters
2016-10-30 22:11:15 +00:00
Simon McVittie
24af123c53 Fix some leaks of floating GVariants
ostree_repo_pull_with_options() and ostree_repo_remote_change() don't
sink floating GVariant arguments, and doing so now would be an
ABI change; so don't rely on them to do so.

Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #556
Approved by: cgwalters
2016-10-30 22:11:15 +00:00
Simon McVittie
53f1fabfbe ot_remote_builtin_show_url: autofree context
Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #556
Approved by: cgwalters
2016-10-30 22:11:15 +00:00
Simon McVittie
1fc2a1202e ostree_repo_pull_with_options: clear dirs array
Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #556
Approved by: cgwalters
2016-10-30 22:11:15 +00:00
Simon McVittie
4739709742 keyfile_set_from_vardict: free the string array
g_variant_get_strv is (transfer container): the caller is expected to
free the array, but not the individual strings.

Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #556
Approved by: cgwalters
2016-10-30 22:11:15 +00:00
Simon McVittie
c8a6b037ef _ostree_sysroot_write_deployments_internal: stop leaking hash table
It appears the result of assign_bootserials() is never actually used,
but I haven't changed it to return void right now.

Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #556
Approved by: cgwalters
2016-10-30 22:11:15 +00:00
Simon McVittie
ff28ac4a30 ostree_sysroot_upgrader_finalize: free new_revision
Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #556
Approved by: cgwalters
2016-10-30 22:11:15 +00:00
Simon McVittie
4c32344b8e ostree_admin_option_context_parse: explicitly clean up when exiting early
The cleanup attribute doesn't clean up before calling a noreturn
function like exit(). Explicitly clean up the pointer variables
(but don't assume that a simple g_object_unref() would be OK either,
in case the behaviour of the cleanup attribute changes).

This isn't a real leak since we're about to exit anyway, but if we
don't fix it then valgrind memcheck will make the tests fail.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #559
Approved by: cgwalters
2016-10-30 21:37:38 +00:00
Simon McVittie
213d5013ce ostree_builtin_pull: consistently set free-function on refs_to_fetch
We are relying on the GPtrArray to free its contents, but we only
give it a free-function on one code path.

Found by valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #559
Approved by: cgwalters
2016-10-30 21:37:38 +00:00
Simon McVittie
22ed96d23c ot_admin_builtin_set_origin: don't leak options GVariant
Found by valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #559
Approved by: cgwalters
2016-10-30 21:37:38 +00:00
Simon McVittie
e757f736e7 _ostree_kernel_args_replace_take: don't leak when replacing
If !existed, then we add arg to kargs->order, where it will be freed
by that array's free-function. However, if the kernel argument did
already exist, we have to either free arg ourselves (and make sure
the old key is what appears in the hash table), or do a linear search
on kargs->order to replace the old key with the new.

Leak found by valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #559
Approved by: cgwalters
2016-10-30 21:37:37 +00:00
Simon McVittie
a0e7d411c1 Distribute valgrind suppressions in tarballs
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #557
Approved by: cgwalters
2016-10-29 18:41:56 +00:00
Simon McVittie
4eb55a687d Distribute test scripts even if we wouldn't run them
This fixes a "make dist" tarball produced on a minimal system and run on a
non-minimal system. Automake knows that files that are only conditionally
included in dist_whatever_WHATEVER are to be distributed, but it does not
do the same for files that are only conditionally included in EXTRA_DIST,
which is how glib-tap.mk's various variables like dist_test_scripts work.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #557
Approved by: cgwalters
2016-10-29 18:41:56 +00:00
Simon McVittie
7091d288f7 Force C.UTF-8 or C locale for tests
Otherwise several tests fail, for example in this build done in a French
locale by Debian's reproducible builds initiative, to check whether
the resulting binaries are identical to what was produced in an
English locale:

<https://tests.reproducible-builds.org/debian/logs/unstable/amd64/ostree_2016.11-1.build2.log.gz>

(test-basic)
# error: Cannot write to repository: Permission non accordée
...
File 'error-message' doesn't match regexp 'Permission denied'

(test-help)
# Utilisation :
#   ostree [OPTION...] COMMAND
...
File 'out' doesn't match regexp '[Uu]sage'

(test-pull-metalink)
# error: Erreur à la ligne 1, caractère 1 : Le document doit commencer avec un élément (par ex. <book>)
...
File 'err.txt' doesn't match regexp 'Document must begin with an element'

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #558
Approved by: cgwalters
2016-10-29 18:12:42 +00:00
Colin Walters
a15dc7f191 ci: Use -fsanitize=undefined by default
It's fast enough to use for CI testing by default, and it
can catch a lot of bad things.

Closes: #552
Approved by: jlebon
2016-10-28 17:47:50 +00:00
Colin Walters
05dc77d7e5 remote-refs: Add NULL terminator to options array
Caught by `-fsanitize=undefined`.

Closes: #552
Approved by: jlebon
2016-10-28 17:47:50 +00:00
Colin Walters
396563e7e9 libglnx: Bump to master (for -fsanitize fixes)
Closes: #552
Approved by: jlebon
2016-10-28 17:47:50 +00:00
Giuseppe Scrivano
c4c8937b20 static-delta: find a similar filename using what is before '.' or '-'
Improve the heuristic to use only the part before the first '.' when
looking for a similar file in the current directory.

last versions of dracut generate reproducible initramfs files, but we
still fallback to the full file download if there is any minimal
change that causes a different checksum and file name.

This change extends that case to deal better with similar files that
have a different suffix.

This is the difference generating a static delta from
fedora-atomic/f24/x86_64/docker-host to fedora-atomic/f24/x86_64/testing/docker-host

before the patch:

fallback for 111ec866aa7ce3688407fa4a1ae7c9fca93dcee0b851fc9434c59ff947830cc7 (47.0 MB)
fallback for c6a898265de22b02c89ea2f35d132628d0ee1c0a058052ed14fee5799c17904c (47.0 MB)
fallback for fbce656249ece77260887ed873e445561b9d43bcb28a32e759c0b1bab89e7137 (6.6 MB)
fallback for cfdb51457e47e0a0fe0bac38991a21279d2646ff2f019630c7b52a0cd3451397 (6.6 MB)
part 0 n:1972 compressed:11239809 uncompressed:33747412
part 1 n:1079 compressed:9683681 uncompressed:55641397
part 2 n:1507 compressed:15050265 uncompressed:44448838
part 3 n:101 compressed:1865881 uncompressed:31896086
part 4 n:278 compressed:2452585 uncompressed:52811323
part 5 n:18 compressed:67621 uncompressed:100220
uncompressed=218645276 compressed=40359842 loose=545102
rollsum=49 objects, 2117254 bytes
bsdiff=4067 objects

after the patch:

part 0 n:843 compressed:19844109 uncompressed:95443178
part 1 n:1223 compressed:11188609 uncompressed:33330401
part 2 n:990 compressed:15762905 uncompressed:61214132
part 3 n:1441 compressed:20614573 uncompressed:31534195
part 4 n:163 compressed:2734997 uncompressed:51356423
part 5 n:285 compressed:2480813 uncompressed:52902904
part 6 n:14 compressed:59125 uncompressed:75341
uncompressed=325856574 compressed=72685131 loose=533283
rollsum=51 objects, 57235332 bytes
bsdiff=4073 objects

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #554
Approved by: cgwalters
2016-10-28 17:04:25 +00:00
Jonathan Lebon
0333260559 delta: return valid enum member
If we can't figure out what endianness a delta is, we should just throw
ENDIAN_INVALID.

Resolves: #550

Closes: #553
Approved by: cgwalters
2016-10-27 22:16:32 +00:00
Colin Walters
7f2960db43 Define an initializer for GVariant{Builder,Dict}
So we build warning-free on GLib (< 2.50, >= 2.50).  This
is a band aid until we hard-require >= 2.50.

Closes: #547
Approved by: jlebon
2016-10-27 18:49:15 +00:00
Jonathan Lebon
cde7928496 .redhat-ci.yml: add clang
Clang has better detection for unused vars when using auto cleanup
functions. We should eventually just fold this back into the first
testsuite. But let's just turn it on for now, at least until it's
satisfied with the whole codebase.

Closes: #549
Approved by: cgwalters
2016-10-27 18:12:05 +00:00
Colin Walters
835d97d659 deploy: Suppress unused variable warning for fscreatecon cleanup
Fixes the clang build.

Closes: #551
Approved by: jlebon
2016-10-27 17:50:56 +00:00
Colin Walters
cbbfb5369f parse-datetime: Use labs() for long input value
Fixes a compliation warning with CLang, I didn't study it to
see whether this was a major issue or not.

Closes: #548
Approved by: jlebon
2016-10-27 17:02:01 +00:00
Colin Walters
21ca60f987 otutil: Note that ot_log_structured takes a printf format
This notably fixes compilation with CLang.

Closes: #548
Approved by: jlebon
2016-10-27 17:02:01 +00:00
Colin Walters
b77edf24a3 tree-wide: Remove unused variables detected by CLang
CLang finds these, whereas GCC treats having
`__attribute__((cleanup))` as a use.

This obsoletes https://github.com/ostreedev/ostree/pull/411

Closes: #548
Approved by: jlebon
2016-10-27 17:02:01 +00:00
Colin Walters
82a4f56593 tests: Skip libarchive/selinux tests if in container without SELinux
I'm doing builds and `make check` inside a Docker container, with
selinux on as a build-time option, but no policy in the container.
This currently aborts.  Let's not do that.

(This type of thing is why installed tests are a better model)

Closes: #546
Approved by: jlebon
2016-10-27 16:48:34 +00:00
Colin Walters
27f37b55bc docs: Link to releng-scripts
Now that the repo starts to implement some of this stuff.

Closes: #544
Approved by: jlebon
2016-10-27 16:28:55 +00:00
Alexander Larsson
d0e74cf3af Fix pruning of partial commits
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.

In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.

Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.

This fixes https://github.com/ostreedev/ostree/issues/541

Closes: #542
Approved by: cgwalters
2016-10-24 17:48:19 +00:00
Colin Walters
d3f14f02e3 Release 2016.12
Closes: #540
Approved by: cgwalters
v2016.12
2016-10-21 19:24:31 +00:00
Alexander Larsson
67bddf76f7 detached metadata: Put these in transaction
If there is a transaction active, then we put writes to detached
metadata into the staging dir, and when reading it we look there
first. This allows transactions to be aborted half-way without
writing the detached metadata into the repository (possibly
overwriting any old metadata from there).

This fixes https://github.com/ostreedev/ostree/issues/526

Closes: #539
Approved by: giuseppe
2016-10-21 10:50:41 +00:00
Alexander Larsson
d43c121675 ostree_repo_read_commit_detached_metadata: Handle parent repo
If the detached metadata is not in the repo, try in the parent
repo if that is set.

Without this a commit will not gpg validate in the child repo

Closes: #539
Approved by: giuseppe
2016-10-21 10:50:41 +00:00
Alexander Larsson
cf6ec1bbbc Fix regression for symlinks in bare-user repos
Commit 1d4f1b8878addd28059c3a3928640491755cd615 started using hardlinks
checkouts of symlinks. However, symlinks are not stored as symlink in the
repo for bare-user repos, so this breaks user-mode checkouts of such repos.

We fix this by checking for !is_symlink in the bare-user case.

This fixes:
     https://github.com/ostreedev/ostree/issues/537

Closes: #538
Approved by: giuseppe
2016-10-21 09:24:27 +00:00
Jonathan Lebon
64568bc706 .redhat-ci.yml: use projectatomic/ostree-tester
Same Dockerfile, but automated to rebuild on pushes.

Closes: #536
Approved by: giuseppe
2016-10-20 08:35:13 +00:00
Jonathan Lebon
e62defa1a4 add .redhat-ci.yml and .redhat-ci.Dockerfile
Add a YAML file for the new Red Hat CI framework.

Rather than re-installing all the build deps everytime, which takes
time, I added a Dockerfile that we can wire up to the Docker Hub. For
now it lives at `jlebon/ostree-tester:rhci`, but we can move it under
the `projectatomic` org (or a new `ostree` org).

Closes: #535
Approved by: cgwalters
2016-10-19 20:35:23 +00:00
Colin Walters
1d4f1b8878 core: Do create hardlinks to symlinks for checkouts
I was noticing a recent performance issue with checkouts
which seemed to be mostly us going back to doing a `fsync()` on
directories.

Regardless, while looking at that, I saw we were spending time
creating new symlinks. Even though symlinks are small, it's still
better to hardlink them.

Going way back in time, the reason we weren't doing this is
because we were hitting `EMFILE` on ext4, but that was for
gnome-continuous which creates *many* build roots.  Even
there though, they're just a cache, and we handle `EMFILE`.

For ostree-for-host-system, we don't expect to have many roots (just 3
at most transiently), so hardlinking symlinks does make sense.

Closes: #521
Approved by: jlebon
2016-10-17 20:40:15 +00:00
Jonathan Lebon
a660e5650d OstreeFetcher: provide proxy credentials if needed
There seems to be an issue in libsoup which causes basic auth
credentials to not be passed to the proxy during requests. We thus have
to handle PROXY_UNAUTHORIZED responses and provide the auth ourselves.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1370558
Related: https://bugzilla.gnome.org/show_bug.cgi?id=772932

Closes: #529
Approved by: cgwalters
2016-10-14 16:06:08 +00:00
Jonathan Lebon
a0598cb494 ostree-sysroot-deploy.c: delete redundant check
Just noticed this while inspecting the code. The deployments retrieved
by `_ostree_sysroot_list_deployment_dirs_for_os` will forcibly already
have a matching osname since it indirectly uses that same variable to
construct them. Having a check there makes it look like there may be
subtle corner cases, when there aren't.

Closes: #529
Approved by: cgwalters
2016-10-14 16:06:08 +00:00