3262 Commits

Author SHA1 Message Date
Philip Withnall
ae335f24dc lib/repo-finder: Add mount based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking for them on any currently mounted
removable storage volumes. The idea is to support OS and app updates via
USB stick.

Unit tests are included.

This bumps libostree’s maximum GLib dependency from 2.44 to 2.50 for
g_drive_is_removable(). If GLib 2.50 is not available, the call which
needs it will be omitted and the OstreeRepoFinderMount implementation
will scan all volumes (not just removable ones); this is a performance
hit, but not a functionality hit.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
d15f83c922 lib/repo-finder: Add config-file based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking their collection IDs up in the local
configuration of remotes who have their collection-id key set.

Unit tests are included.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
292230301d lib/repo-finder: Add basic support for finding remote URIs by ref name
Add an initial OstreeRepoFinder interface (but no implementations),
which will find remote URIs by ref names and collection IDs, the
combination of which is globally unique.

The new API is used in a new ostree_repo_find_updates() function, which
resolves a list of ref names to update into a set of remote URIs to pull
them from, which can be treated as mirrors. It is an attempt to
generalise resolution of the URIs to pull from, and to generalise
determination of the order and parallelisation which they should be
downloaded from in.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
7607d94713 lib/pull: Add collection support to ostree_repo_pull_with_options()
This adds a new collection-refs option of type a(sss), giving a list of
(collection ID, ref name, checksum) tuples to pull from the given remote.
This option is intended to supersede the refs and override-commit-ids
options, so is mutually exclusive with them.

This includes support for resolving the refs from the remote’s summary
file, or from its refs/heads and refs/mirrors directories.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
fbf8df8829 lib/refs: Add methods for setting/listing collection–refs
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
4de736fdfa lib/repo: Add collection ID support to OstreeRepo
Add {get,set}_collection_id() methods to OstreeRepo and some documentation
about the concept of a collection ID which globally identifies an
upstream repository. See the documentation for more details.

This will be used in future commits. For now, the new API is marked as
experimental (--enable-experimental-api).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
0a20e7d43c lib/ref: Add OstreeCollectionRef type for globally unique refs
This is a type representing the tuple (collection ID, ref name), which is
guaranteed to be globally unique. It will be used in upcoming commits.

It introduces the concept of a ‘collection’ which is a unique, curated
set of refs which lie in the same trust domain (i.e. all signed by the
same key and validated by the same developer). Flathub might be a
collection, for example; or the set of OS refs coming from a particular
OS vendor.

It includes a function for validating collection IDs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
bf1f8eb0fa lib/repo: Split out ref handling from regenerate_summary()
This will make some future additions to regenerate_summary() easier.
This commit introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Colin Walters
da0791f484 tests: add a syntax-check rule for glnx_prefix_error()
Same as the errno variant; the colon-space `: ` thing got me in a different
patch.

Closes: #956
Approved by: jlebon
2017-06-26 15:09:12 +00:00
Colin Walters
553b99642c cmd/fsck: Port to new style
Happened to look at this code too, it's a straightforward port.

Closes: #955
Approved by: jlebon
2017-06-26 14:54:32 +00:00
Colin Walters
e23071dc39 lib/deltas: Port to more to new code style
Looking at the uses of `ostree_repo_load_file()` here.

Closes: #953
Approved by: jlebon
2017-06-26 14:42:56 +00:00
Colin Walters
612c8a5fa8 lib/repo: More cleanup of load_file() internals
This is followon work from previous cleanups.  Basically
`stat_bare_content_object()` was the `fstatat()` logic
and `ostree_repo_read_bare_fd()` was the `openat()` implementation;
they duplicated some bits to find the object in staging, recurse
into parent etc.

Further, I wanted an internal-only version of this API which didn't allocate
`GFileInfo`/`GInputStream` but used a plain `fd` and `struct stat` to avoid
mallocs.

The end version here I think looks a lot nicer, since we deduplicate the various
`open()` calls in the different cases for example.

Closes: #952
Approved by: jlebon
2017-06-23 18:29:51 +00:00
Colin Walters
63ad289a9c lib/repo: Split archive/bare file parsing
Prep for future cleanup patches (in particular I want an internal-only
version at first that uses a fd+`struct stat`) to avoid allocations.

The new version avoids lots of deep nesting of conditionals as well
by hoisting the "not found" handling to an early return.

There's a bit of code duplication between the two cases but it's
quite worth the result.

Closes: #951
Approved by: jlebon
2017-06-23 14:11:36 +00:00
Colin Walters
aafda9073a lib/core: Avoid NULL deref in content_file_parse() if out variable unset
Prep for a change in `ostree_repo_load_file()`.  We would crash if a
caller had `out_file_info = NULL`, because we deref `ret_file_info`
below it.

Closes: #951
Approved by: jlebon
2017-06-23 14:11:36 +00:00
Simon McVittie
90b587e2c8 build: Always include ostree-trivial-httpd.xml in tarballs
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #949
Approved by: cgwalters
2017-06-22 13:33:39 +00:00
Philip Withnall
9a79d13ce3 lib/remote: Fix ‘Since’ line for OstreeRemote
There was a typo in it when it was first introduced. Let’s not live in
the past.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall
20829a0582 lib/sym: Fix symbol versions for 2017.7 experimental symbols
There was a typo in the group name. It should be OK to change the
version since this is all hidden behind the --enable-experimental-api
configure option.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall
333d264c5b lib/core: Fix ‘Since’ line for ostree_validate_remote_name()
This was missed when cherry-picking it out of
https://github.com/ostreedev/ostree/pull/924#discussion_r123097919.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Colin Walters
027c77c621 ci: Enable -Werror=maybe-uninitialized
We don't want to inject this warning by default for every build
like the other ones in `configure.ac`, since it can be spruriously
wrong.  But there's no reason not to have a more extended set of
warnings for well-known toolchains (e.g. f25).

Closes: #943
Approved by: jlebon
2017-06-20 22:27:18 +00:00
Colin Walters
46e7f08daa repo: Squash a gcc -Wmaybe-uninitialized warning
It's spurious, but unfortunately GCC doesn't currently understand that it will
always be set.

Closes: #943
Approved by: jlebon
2017-06-20 22:27:18 +00:00
Philip Withnall
20dc9454b3 lib/core: Add ostree_validate_remote_name() for remote names
There are a few places in the code where ad-hoc validation was being
performed. Might as well formalise it a bit more.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #948
Approved by: cgwalters
2017-06-20 21:52:22 +00:00
Philip Withnall
fd50c97322 build: Add ‘devel’ or ‘release’ to OSTREE_FEATURES for test-symbols.sh
test-symbols.sh was looking for the DevelBuild string, which is actually
part of the output from `ostree --version`, not $OSTREE_FEATURES.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #948
Approved by: cgwalters
2017-06-20 21:52:22 +00:00
Jonathan Lebon
82410f0e59 ci: add ci-release-build.sh
Add a check that verifies that `is_release_build` is `yes` only for
release commits. And also verify that the commit message has the correct
version.

Closes: #945
Approved by: cgwalters
2017-06-19 16:16:23 +00:00
Colin Walters
b69c6ba992 build-sys: Post-release version bump
Closes: #944
Approved by: jlebon
2017-06-19 15:51:27 +00:00
Colin Walters
2a082b20b8 main: DevelBuild=yes to ostree --version for devel builds
And use it in `test-symbols.sh`, to fix the `distcheck` case;
the previous change stopped distributing `libostree-devel.sym`
in release builds.

Closes: #944
Approved by: jlebon
2017-06-19 15:51:27 +00:00
Colin Walters
6729b7c264 Release 2017.7
Closes: #942
Approved by: jlebon
v2017.7
2017-06-19 14:54:59 +00:00
Colin Walters
a45dc0fd0b build-sys: Add "release build" flag, use for symbol versioning
I was trying to do a release and move the symbols from `-devel.sym` into
`-release.sym`, but it turns out that at least GNU binutils `ld` treats an empty
version script as a syntax error.  Fix this by adding a "release build"
flag, and only include `-devel` in non-release builds.

This would also make it easier to inject that flag into our `.pc` and
`ostree-version.h` and `ostree --version` metadata, but I didn't do that yet.

EDIT: Turns out a simpler fix is just to add an empty section. However I kept
this commit since it's a useful sanity check for whether we should include
`-devel.sym` in builds, and we may want to inject the metadata later.

Closes: #942
Approved by: jlebon
2017-06-19 14:54:59 +00:00
David Shea
22e753176e lib/repo: Fix annotations for out parameters
Change the annotation of the out parameters on ostree_repo_load_file
from `(allow-none)` to `(optional) (nullable)`. `allow-none` is
ambiguous, since these parameters can be both NULL on input and set to
NULL on return.

Closes: #939
Approved by: cgwalters
2017-06-19 13:15:19 +00:00
Alexander Larsson
07dc33ca4a static delta apply: Work on bare-user-only repos
Flatpak make check is failing when applying a static delta
to a bare-user-only repo due to an assert. The fix is to add
bare-user-only to the assert check.

Closes: #940
Approved by: giuseppe
2017-06-19 09:09:17 +00:00
Colin Walters
fb2c3c1db3 tests: Fix previous commit for selinux testing
I only checked the test passed, I didn't read the output closely, and
made it succeed without testing anything.

Fix the absolute/relative `/etc` references.

Closes: #937
Approved by: jlebon
2017-06-16 15:46:07 +00:00
Alexander Larsson
73ba3eb686 pull: When mirroring, only replace summary if we're doing a full mirror
We're hitting this in flathub, where we have a bunch of local builds,
but we also mirror a few refs from the gnome runtime repo into it.
Its fixable by re-doing the summary, but for a short time the
wrong version is visible.

Fixes https://github.com/ostreedev/ostree/issues/846

Closes: #935
Approved by: cgwalters
2017-06-16 15:08:39 +00:00
Colin Walters
371b4a5e7e checkout: Fix SELinux policy labeling when recursing
The code here tried to truncate the string to the previous length,
but that doesn't work when recursing, since further calls change the
length.

What actually ended up happening was the string would get corrupted
after the first level of recursion.

Closes: #936
Approved by: jlebon
2017-06-16 14:54:29 +00:00
Colin Walters
2bab43fb22 lib: Split symbol versioning into -released and -devel
So far a lot of submitted PR have added symbols into the first
section.  Split the file into `-released` and `-devel` to make
this more obvious.

To further enforce things, we hardcode a checksum of the `-released`
file in `test-symbols.sh`.  Only release commits should update that
checksum.

Did you notice I like checksums?

Closes: #931
Approved by: pwithnall
2017-06-16 09:31:16 +00:00
Tristan Van Berkom
3e3a0f0766 ostreee-version.h.in: Added Since: version annotations
This is especially interesting for the versioning symbols themselves,
as it is an indicator of when applications using introspection information
can start to use a symbol in the library to check if they have a
recent enough version of OSTree to use.

Closes: #932
Approved by: cgwalters
2017-06-15 12:59:59 +00:00
Colin Walters
9529e8d435 lib/pull: Extend BAREUSERONLY_FILES flag to HTTP requests
For the flatpak PR: https://github.com/flatpak/flatpak/pull/849

It's really more convenient if this works for HTTP pulls as well, since flatpak
does various types of pulling, and we can just set the flag everywhere.

Further, we might as well reject the content as early as possible.

Closes: #930
Approved by: alexlarsson
2017-06-15 07:10:22 +00:00
Colin Walters
0e6d23835b lib/sysroot: Add some g_prefix_error() for ostree_sysroot_cleanup()
We saw this fail in a CI run. We've been trying to add strategic error prefixing
as a general rule, and this specific instance may help debug.

Closes: #929
Approved by: pwithnall
2017-06-15 00:08:29 +00:00
Colin Walters
584735b1c9 build-sys: post-release version bump
Per request by flatpak.

Closes: #928
Approved by: alexlarsson
2017-06-14 14:51:25 +00:00
Anton Gerasimov
64ab8334b7 lib/sysroot: Add API to get pending/rollback for given stateroot
This imports a function that is used in rpm-ostree, and it's also intended for
use by https://github.com/advancedtelematic/aktualizr to display
what deployment we're going to boot next after the reboot.

Updated-by: Colin Walters <walters@verbum.org>

Closes: #897
Approved by: OYTIS
2017-06-14 09:56:01 +00:00
Colin Walters
0635fcbfd9 lib/checkout: Add bareuseronly_dirs option
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.

See: https://github.com/flatpak/flatpak/issues/845

This option suppresses mode bits outside of `0775` for directory
checkouts.  I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.

Closes: #927
Approved by: alexlarsson
2017-06-13 20:05:31 +00:00
Colin Walters
6ed824bf00 lib/pull: Add OSTREE_REPO_PULL_FLAGS_BAREUSERONLY_FILES
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845

We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909

This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.

Closes: #926
Approved by: alexlarsson
2017-06-13 18:44:28 +00:00
Colin Walters
74e3581ed6 lib/repo: Support hardlink conversions from bare-user to bu-only
Thinking about the problem of flatpak converting from `bare-user` to `bare-user-only`
"in place" by creating a new repo and doing a `pull-local`, I realized
that we can optimize this process by doing hardlinks for both metadata
and regular files.  The repo formats are *almost* compatible, the
exception being symlinks.

An earlier patch caused us to do hardlinks for metadata, this patch takes things
to the next step and special cases this specific conversion. In this case we
need to parse the source object to determine whether or not it's a symlink.

Closes: #922
Approved by: alexlarsson
2017-06-13 12:02:12 +00:00
Colin Walters
b614c65eab lib/repo: Import metadata via hardlink even for distinct repo modes
Our previous logic for import-via-hardlink only tried if the repo modes match,
but we *can* hardlink metadata between e.g. `archive` and `bare-user` repos, and
that's quite useful thing to do. Our documentation encourages converting to/from
those repo modes locally for build systems.

Closes: #922
Approved by: alexlarsson
2017-06-13 12:02:12 +00:00
Colin Walters
695771667c lib/repo: Skip import via hardlink if repo owners don't match
Before this, if one had repos of matching mode but different owners,
which could happen if one e.g. makes a `bare` non-root repo in
`/ostree/deploy/$stateroot/var/tmp`, every time we tried to call `linkat()`
we'd get `EPERM` and fall back to a copy.

Fix this by saving the repo owner uid, and avoid trying to call `linkat()` if we
know it's going to fail. Of course most commonly in this scenario we'll
immediately fail trying to `chown` the files to `0`, but this is prep for a
future patch to improve `bare-user` → `bare-user-only` imports where we'll be a
bit more sophisticated.

Closes: #922
Approved by: alexlarsson
2017-06-13 12:02:12 +00:00
Philip Withnall
21eec96bfd lib/pull: Fix construction of a refspec to use the correct separator
This code looks like it was supposed to build a refspec, but it used a
slash as a separator rather than a colon. The following code does
recover by supporting prefix matching with slashes, but it seems like
this was perhaps not the intention.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #912
Approved by: cgwalters
2017-06-13 01:03:49 +00:00
Alexander Larsson
7159bed8e1 lib/repo: Always look in staging directory for objects
Its often the case that we want to look at objects inside a commit,
before the objects the transaction is finished. For instance:
  https://github.com/flatpak/flatpak/pull/837
Which tries to verify the file permissions before committing the
transaction.

And:
  1e5ffa926a
Which collects the storage size of the objects so that we can
put the total download size in the commit metadata.

I tried to find all the places where we did reads from the
object directories, and in particular this fixes:

 - `ostree_repo_load_file()` for `bare` repos (`archive` was already working).
 - `ostree_repo_query_object_storage_size()`
 - Applying deltas that reference not-yet-commited objects

Closes: #916
Approved by: cgwalters
2017-06-13 00:47:42 +00:00
Colin Walters
848b7c0201 lib/repo: Refactor object copy import function
This came up in: https://github.com/ostreedev/ostree/pull/881

Basically doing streaming for metadata is dumb. Split up the metadata/content
paths so we pass metadata around as `GVariant`. This drops the last internal
caller of `ostree_repo_write_metadata_stream_trusted()` which was the dumb
function mentioned.

Closes: #923
Approved by: jlebon
2017-06-12 21:13:23 +00:00
Colin Walters
8edb5161db lib/checkout: Ignore world-writable dirs for bare-user-only checkout
See https://github.com/ostreedev/ostree/pull/909 for more information on the
rationale. Basically there's no reason for flatpak (which uses `bare-user-only`)
to have world-writable dirs. Particularly with the presence of the system
helper.

An approach I considered instead was to parse and validate directory metadata
objects at commit time. We still may do that in addition; for file objects we *had*
to do it that way because the actual files would be laid down suid.  But directories
live only as inert `.dirmeta` objects until we do a checkout (i.e. `mkdir()`), so
we can solve the problem at checkout time.

Closes: #914
Approved by: alexlarsson
2017-06-12 14:24:22 +00:00
Colin Walters
18ae8e5267 lib/commit: Drop some conditionals/clarify code in content path
Both callers of `commit_loose_object_trusted()` were passing
`OSTREE_OBJECT_TYPE_FILE`, so drop that parameter.  This in turn
allows us to drop lots of checking of that inside the function.

Add a doc comment, and rename to `commit_loose_content_object()` for clarity.

Closes: #914
Approved by: alexlarsson
2017-06-12 14:24:22 +00:00
Colin Walters
aed8a6b09a lib/commit: Port final object writing function to new code style
I noticed my previous patches incorrectly started doing `return glnx_throw*`
inside a `goto out;` function. Fix this by porting forward consistently to new
style. We just do the error prefixing in the caller.

Closes: #914
Approved by: alexlarsson
2017-06-12 14:24:22 +00:00
Colin Walters
5de201df26 repo: Fix leak of superblock fds when generating summary
Related: https://github.com/ostreedev/ostree/issues/920

Closes: #921
Approved by: alexlarsson
2017-06-12 14:15:14 +00:00