3368 Commits

Author SHA1 Message Date
Colin Walters
ded6417aee lib/gpg: Switch to GLib autocleanups for gpgme types
Prep for dropping `GLNX_DEFINE_CLEANUP_FUNCTION` from libglnx
in favor of using GLib's `G_DEFINE_AUTO_CLEANUP_FREE_FUNC()`.

Closes: #1042
Approved by: jlebon
2017-08-02 16:55:38 +00:00
Colin Walters
6347c0fb88 lib/curl: Only check individual request errors
It looks like `curl_multi_socket_action()` will return an error
if *one* of the requests has an error, but we already check
for that explicitly by iterating over each handle.

In libcurl, the "easy" layer doesn't really make use of this
return value.  I did a bit of looking elsewhere; systemd
does check it as a runtime error, not an assertion.  librepo
doesn't use the multi interface.

Closes: https://github.com/ostreedev/ostree/issues/1035

Closes: #1038
Approved by: jlebon
2017-08-02 15:44:00 +00:00
Colin Walters
2f0707a054 lib/commit: Rework a conditional set for clarity and Coverity
Coverity complained that the `else if (bytes_read == 0)` was dead
code if we happened to find it was already false when testing
`else if (G_UNLIKELY (bytes_read == 0 ...`.

There was nothing wrong with the logic, but let's rework it to
only test the value once; I think it does end up nicer anyways.

Coverity CID: 1452186

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters
fe8938227e tests/lzma: Fix off-by-one in buffer size
Coverity spotted that we had an off-by-one here since we were using
`i+1`.  Fix this by adding a `-1` to the bounds check.  Also use
`sizeof()` to ensure the data and size are coupled.

Coverity CID: 1452207

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters
e70db67c01 rofiles-fuse: Add missing error handling for fcntl()
Spotted by Coverity.

Coverity CID: 1452201

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters
5daae8629e tests/lzma: Use GRand over random()
It's designed for test suites and non-critical random uses like this. This
silences a Coverity warning about weak randomness.

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters
78348fa296 bin/refs: Fix error handling logic
Spotted by Coverity.

Coverity CID: 1452202

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters
36ab8cd0e1 tests: Add some return value checking to pacify Coverity
No real problems here, but Coverity likes to see consistent checking of return
values, and I agree with it.

Coverity CID: 1452213
Coverity CID: 1452211

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters
fbefe875c7 docs/pull: Finish incomplete docs for pull@hash syntax
Closes: https://github.com/ostreedev/ostree/issues/1043

Closes: #1045
Approved by: guyshapiro
2017-08-01 21:51:17 +00:00
Robert McQueen
e8a9172daa gpg-verify-result: try to display the primary key ID from signatures
The fingerprint associated with each signature can be different to
the primary key ID (the normal one that people use to identify a
GPG key) if the signature is from a signing subkey. Try to find the
primary key and print this ID in preference to the subkey signature.

https://github.com/ostreedev/ostree/issues/608

Closes: #1036
Approved by: cgwalters
2017-07-31 14:37:09 +00:00
Robert McQueen
156cf23576 gpg-verify-result: canonicalise key when looking up signatures
Use gpgme_get_key to find the primary key for the key we are
looking for, and the primary key for each signature, and
compare these when looking up signatures.

The primary key is the first in the list of subkeys, which is
the normal key ID people use when referring to a GPG key as an
identity.

If the key has a signing subkey, signature->fpr will not match
the provided key_id, so looking up both keys and comparing the
primary key fingerprints ensures they are both canonicalised.

https://github.com/ostreedev/ostree/issues/608

Closes: #1036
Approved by: cgwalters
2017-07-31 14:37:09 +00:00
Colin Walters
d7f953aa3a lib/pull: Journal fetch results
This is a continuation of addition of journaling to libostree; see
e.g. <https://github.com/ostreedev/ostree/pull/708>.

I wanted more information at the end of fetches; in particular
some details about the delta execution (what opcodes etc.), but
this is a first step: we log things like the transferred data
as well as whether or not GPG was enabled, etc.

One awkward thing about this is how we map the fetcher options like
`tls-ca-path` back out into an enum for the code to log. But eh, hard to fix
without a bigger refactoring.

Closes: #1032
Approved by: jlebon
2017-07-28 13:50:22 +00:00
Colin Walters
d24b259cc5 grub2: Handle aarch64 (at least on Fedora/RHEL)
See: https://github.com/projectatomic/rpm-ostree-toolbox/issues/102#issuecomment-316483554

The logic here should now match what grubby does.  I have no
idea whether this applies to other distributions though.

Closes: #1021
Approved by: jlebon
2017-07-27 18:21:00 +00:00
Colin Walters
79a88ac249 build-sys: Post-release version bump
Closes: #1029
Approved by: jlebon
2017-07-26 21:04:08 +00:00
Colin Walters
fa3e07e5d7 Release 2017.9
Closes: #1029
Approved by: jlebon
v2017.9
2017-07-26 21:04:08 +00:00
Colin Walters
f1f199578e ci: Enable libcurl by default on Fedora
The insttest fell over since its build used libsoup, but that just dropped out
of FAH.

Closes: #1030
Approved by: jlebon
2017-07-26 20:54:35 +00:00
Colin Walters
e09fc83ab3 lib/core: Add #defines for ref/collection binding
These were previously private, but since we expect people to use them, let's add
`#define`s like we did for some of the other commit metadata.

Closes: #1028
Approved by: jlebon
2017-07-26 18:52:04 +00:00
Colin Walters
0985158be7 Update libglnx, port some uses to newer APIs
Mostly for the latest `-Wmaybe-uninitialized` fix, but while here also port some
places to newer APIs.

Update submodule: libglnx

Closes: #1027
Approved by: jlebon
2017-07-24 18:43:57 +00:00
Colin Walters
8456fd5057 build: Turn off default warnings if we find -Werror specified
Our CI runs use `-Werror`; there's no point to our default warning set kicking
in, it just bloats the command line output.

Closes: #1023
Approved by: jlebon
2017-07-21 20:45:47 +00:00
Colin Walters
0ded552cf1 ci: Enable -Werror for clang
I hit an unused-variable warning with `GLNX_AUTO_PREFIX_ERROR` for
rpm-ostree and led me to wonder why ostree didn't fail, then I noticed
we had lost the special `-Werror=unused-variable` bit.  Let's go
ahead and use `-Werror` for clang too.

Closes: #1023
Approved by: jlebon
2017-07-21 20:45:47 +00:00
Colin Walters
ed99b4169e tests: More fixes for gjs tests
Previous to this commit, the gjs tests were installed-only; and our
logic for handling the "--enable-installed-tests=exclusive" logic
actually also meant they weren't installed.

It did work for me locally with `--enable-installed-tests`.

However, to make things fully symmetric, let's enable the js tests to also be
run under `make check`.

Also remove `corrupt-repo-ref.js` from the PAPR invocation since it's not
actually a unit test, it's a utility helper.

Closes: #1022
Approved by: jlebon
2017-07-21 15:45:27 +00:00
Ruixin
ef6f6bc688 lib: Add #define for endoflife metadata key
It is a continuation of https://github.com/ostreedev/ostree/pull/1024
It adds documentation for endoflife metadata key.

Closes: #1025
Approved by: cgwalters
2017-07-21 15:27:06 +00:00
Colin Walters
6430207e47 lib: Add #defines for current well-known metadata keys
This came up in https://github.com/projectatomic/rpm-ostree/issues/142

Let's add `#define`s for our metadata keys, with documentation so
that, well, they're documented.

Closes: #1024
Approved by: peterbaouoft
2017-07-21 14:53:33 +00:00
Colin Walters
cf6c15a6c5 ci/papr: Add a suite to run introspection-based tests without ASAN
Unfortunately we can't do gobject-introspection based tests
while compiling with `-fsanitize=address`, since it needs to hook
`malloc` early on.

Add a new suite which just runs the introspection-based tests without ASAN.

Closes: #1016
Approved by: jlebon
2017-07-20 14:01:11 +00:00
Colin Walters
f9f7d55e79 lib/commit: Fix EBADF with GENERATE_SIZES option for commit
Regression from previous tmpfile refactoring; unfortunately
the `OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` option
only has coverage via gjs currently.

Might expose it via the cmdline in a later option, but in the big picture the
idea was that this data is better kept in static deltas.

Closes: https://github.com/ostreedev/ostree/issues/1014

Closes: #1016
Approved by: jlebon
2017-07-20 14:01:11 +00:00
Colin Walters
ebbd0b3ce6 ci/papr: Switch primary to libcurl, add libsoup context
Sometime in the next few releases I think we should make libcurl
the default.

Prep for more CI work.

Closes: #1016
Approved by: jlebon
2017-07-20 14:01:11 +00:00
Colin Walters
779f125cbe lib/repo: Auto-recreate repo/tmp if it's deleted
We can accumulate a lot of space there; let's be nice to people who delete the
whole directory.

Closes: https://github.com/ostreedev/ostree/issues/1018

Closes: #1020
Approved by: jlebon
2017-07-19 15:01:19 +00:00
Colin Walters
c740b7f6d2 core: Sanitize error text validating refs (e.g. against HTML)
See: https://github.com/projectatomic/rpm-ostree/issues/885

If we get a successful Apache directory listing HTML when fetching what we
intend to be a ref, we'd dump the HTML into the error.

I did some scanning of the pull code, and this was the only case
I saw offhand where we were dumping text out into an error.  Which
makes sense, since most of our formats are binary, the exeptions I
think are just `repo/config` and `repo/refs/`.

Closes: #1015
Approved by: mbarnes
2017-07-19 14:45:57 +00:00
Simon McVittie
bed931c91f build: Don't distribute generated man pages
We build them in "make" and clean them in "make clean", so there
doesn't seem much point in shipping them pre-generated in the tarball.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1013
Approved by: cgwalters
2017-07-18 21:51:43 +00:00
Colin Walters
2a9689b76a Update libglnx, port various bits to new API
Using the error prefixing in the delta processing allows us to
do new code style.  Also strip trailing whitespace.

Use error prefixing in a few other random places.  I didn't
hunt for all of them, just testing out the new API.

Use `glnx_fchmod()`.  Also note I dropped one `fchmod (tmpf, 0600)`
which is no longer necessary.

Update submodule: libglnx

Closes: #1011
Approved by: jlebon
2017-07-18 19:18:38 +00:00
Colin Walters
efd460782a lib/pull: Drop direct use of ->repodir
Prep for `ostree_repo_new_at()`.  Down the line perhaps
we should extend libcurl to accept a file descriptor for cookies,
but this works OK for now.

Closes: #1010
Approved by: jlebon
2017-07-18 19:07:56 +00:00
Colin Walters
9430b8ad75 bin/cookies: Drop libsoup code, fix fd-relative issues, new style
Prep for `ostree_repo_new_at()`. These commands were directly accessing
`repo->repodir`, which it turns out was unnecessary since the the APIs they then
used were fd-relative. Except actually there were bugs there, so fix all of the
cookie util code to actually use the passed `dfd` and not just hardcode
`AT_FDCWD`.

Also, libsoup can't handle this (its APIs require fully qualifed paths), and
there's not a really good reason to have two implementations now; historically
it was useful to cross-check them, but I don't think we need that.

While I'm here, port to new style.

Closes: #1010
Approved by: jlebon
2017-07-18 19:07:56 +00:00
Colin Walters
e0346c1494 Add a notion of "physical" sysroot, use for remote writing
(Note this PR was reverted in <https://github.com/ostreedev/ostree/pull/902>;
 this version should be better)

Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).

In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.

This second iteration of this patch fixes the bug we hit the first time;
embarassingly enough I broke `ostree remote list` finding system remotes.
The fix is to have `ostree_repo_open()` figure out whether it's the same
as `/ostree/repo` for now.

Down the line...we might consider having the `ostree remote` command line itself
instatiate an `OstreeSysroot` by default, but this maximizes compatibility; we
just have to pay a small cost that `ostree` usage outside of that case like
`ostree static-delta` in a releng Jenkins job or whatever will do this `stat()`
too.

Closes: https://github.com/ostreedev/ostree/issues/892

Closes: #1008
Approved by: mbarnes
2017-07-18 18:58:06 +00:00
Colin Walters
8b1f1c4428 lib/pull: Do local content imports async too
This came up in <https://github.com/ostreedev/ostree/pull/982>; when
we added more direct local importing, we did it synchronously.

This was actually quite a regression when doing local pulls between different
modes; in particular between a bare mode and `archive`, as we were suddenly
doing gzip {de,}compression in the main thread.

Down the line actually...a simpler fix is probably to change things so that the
local path is really only used when we know we can hardlink; everything else
would go though the fetcher codepath but with `file://`.

But this isn't a lot more code, and the speed/interactivity win is large.

Note we're only doing content async with this patch. We could do metadata as
well; we have the object already local. But the metadata code path is messier,
and metadata objects are smaller.

Another area where this comes up is that in e.g. Fedora releng, most operations
talk to a NetApp via NFS. So this has the classic network filesystem problem
that operations that are normally cheap like `stat()` can actually have
nontrivial latency. Doing as much as possible in threads is better there too.

Closes: #1006
Approved by: jlebon
2017-07-18 17:03:13 +00:00
Colin Walters
96e49a67f9 ci/papr: Update to F26
In particular F25AH will stop getting updates.

Closes: #1012
Approved by: jlebon
2017-07-18 14:01:54 +00:00
Emmanuele Bassi
47a54bf876 Move the include directive to the enum template
There is no actual written guarantee in glib-mkenums that the template
line specified using --fhead will be added after the templates specified
inside the template file. Since the template file is only used once, we
can simply move the `#include` directive inside the template, so that it
is guaranteed to be in the right place.

Closes: #1007
Approved by: cgwalters
2017-07-17 15:14:44 +00:00
Colin Walters
620a90ebfa lib/pull: Avoid journaling 404s for optional content
Currently in Fedora we don't sign summaries, and every use of
`rpm-ostree` would emit to the journal an error when we failed
to fetch it.

Fix this by having `OSTREE_FETCHER_REQUEST_OPTIONAL_CONTENT` tell the fetcher
not to journal 404 errors. While fixing this, we had a mix of two booleans vs
the flags; fix things so we consistently use the flags in the fetcher and pull
code.

Closes: #1004
Approved by: mbarnes
2017-07-13 21:41:00 +00:00
Philip Withnall
82024f161b build: Ensure all experimental tests are distributed in tarballs
As with the previous commit, ensure that tests which are run when
configured with --enable-experimental-api, are always distributed; even
when running `make dist` from a source tree configured with
--disable-experimental-api.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1002
Approved by: cgwalters
2017-07-12 14:23:44 +00:00
Philip Withnall
f6ba20d9d6 build: Ensure all .sym files are distributed in tarballs
Since we’re using a custom variable for listing the .sym files,
automake’s magic support for automatically distributing all files in
conditionals doesn’t work, and the devel and experimental .sym files
were only being distributed if `make dist` was run on a source tree
which had been configured with --enable-experimental-api or not a
release flag.

Fix that by explicitly listing all the .sym files in EXTRA_DIST.

Specifically, this fixes the case of trying to compile with
--enable-experimental-api from a release tarball which was disted with
--disable-experimental-api.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1001
Approved by: cgwalters
2017-07-12 13:04:32 +00:00
Philip Withnall
acb14648d7 lib/repo: Add OSTREE_REPO_METADATA_REF as a well-known metadata store
As discussed in https://github.com/ostreedev/ostree/pull/946, the
summary file is becoming an unsigned cache of ref information; any
additional metadata for the repository needs to move elsewhere in order
to remain signed. Introduce OSTREE_REPO_METADATA_REF as the well-known
name of a ref where such metadata can live, as the metadata on
contentless commits.

Don’t yet update the documentation for summary-related methods to
mention this, since it’s still hidden behind the
--enable-experimental-api configure option.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #946
Approved by: cgwalters
2017-07-11 19:50:32 +00:00
Philip Withnall
7d57459e83 lib/repo-commit: Fix types of content size cache entries
Use goffset rather than gsize for file sizes. More importantly, get the
unpacked_size from g_file_info_get_size() (goffset) rather than from the
splice return value, which has type gssize.

This will make a difference on 32-bit systems, where goffset is defined
as off64_t, but gsize is 32 bits.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #999
Approved by: cgwalters
2017-07-11 14:55:55 +00:00
Colin Walters
0aa20df20e tests: Run pull tests for bare/bare-user
We have variants of `test-basic` for all 4 modes, but not for pull-test, which
for some reason was named `pull-archive`, but mostly pulls *into* bare repos.
The test code was structured like the basic one where it called into a
`pull-test.sh`, so let's actually use it for 2/3 bare modes. (I tried to extend
it to `bare-user-only` but it failed, going to look at that after this).

This is related to https://github.com/ostreedev/ostree/issues/991

Closes: #998
Approved by: jlebon
2017-07-10 14:18:18 +00:00
Colin Walters
9d941dcebb checkout: Don't set dir mtime to 0 when doing a force copy checkout
When we [switched to using checkout + force_copy](e8efd1c8dc),
a side effect that went unnoticed at the time is that we started
setting directory mtimes to zero.

See the below bug where we long ago set the file times to zero, which got fixed,
so let's not regress things by setting the directory times to zero either. (Even
though AFAICS GNU tar doesn't complain about those)

This semantic is somewhat "overloaded" onto `force_copy`, but it avoids adding
yet another boolean; we don't have that many reserved boolean slots left. I
can't really think of many good use cases for `force_copy` *other* than the
`/etc` merge anyways.

https://bugzilla.redhat.com/show_bug.cgi?id=1229160

Closes: https://github.com/ostreedev/ostree/issues/995

Closes: #997
Approved by: jlebon
2017-07-07 15:01:51 +00:00
Colin Walters
d2a05e5a09 deploy: Port some functions to new style
There are a number of simple ports here.  Prep for further work
in `/etc` merge.

I also stripped trailing whitespace globally.

Closes: #996
Approved by: jlebon
2017-07-07 14:39:47 +00:00
Krzesimir Nowak
7fa534ac17 tests: New tests for creating commits with bindings and pulling them
Closes: #972
Approved by: cgwalters
2017-07-06 19:08:14 +00:00
Krzesimir Nowak
cc9a0386c4 lib/pull: Collection and ref bindings verification
This verifies the collection and ref bindings in the commit metadata
against the collection ID we have stored in the remote config and ref
we want to pull from. For the HEAD commits, we also check if the
checksum of the commit we just fetched agrees with the checksum we
really wanted to pull from the ref.

For commits with explicitly specified checksums and without specified
refs, we only verify if the commit has the bindings. We are able to
only verify the collection binding, though.

Closes: #972
Approved by: cgwalters
2017-07-06 19:08:14 +00:00
Krzesimir Nowak
d91f6a0f61 lib/pull: Pass the ref together with the request
We will want to use the requested ref later for the binding
verification.

Closes: #972
Approved by: cgwalters
2017-07-06 19:08:14 +00:00
Krzesimir Nowak
cf16805a2f ostree: Add collection and ref bindings to metadata on commit
The collection and ref bindings are stored in the commit metadata
under ostree.collection-binding and ostree.ref-binding,
respectively. They will be used to verify if the commit really comes
from the collection and ref we wanted to pull from.

Closes: #972
Approved by: cgwalters
2017-07-06 19:08:14 +00:00
Colin Walters
28ec43c41a build-sys: Post-release version bump
Closes: #994
Approved by: jlebon
2017-07-06 14:45:02 +00:00
Colin Walters
5a5e465492 Release 2017.8
Closes: #994
Approved by: jlebon
v2017.8
2017-07-06 14:45:02 +00:00