3319 Commits

Author SHA1 Message Date
Philip Withnall
b7b79fa78d refs: Add a --collections argument to the built-in refs command
Rather than change the output format used by the existing refs command
to output collection IDs in addition to ref names, this functionality has
been hidden behind an --collections argument. If it’s not specified `ostree
refs` will output the same content as before for a given repository. If
it is specified, the collection ID for each ref will be included in the
output as (collection ID, ref name).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
205a84b36e remote-add: Add a --collection-id argument to the built-in add command
This allows newly configured remotes to have their collection ID specified,
so that refs from them can be downloaded from peers as well as the
upstream collection, using the remote’s configuration.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
144e325579 init: Add a --collection-id argument to the built-in init command
This allows new repositories to be configured with a collection ID which
can be used to uniquely identify refs which originated from this
repository.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
37fc49f36d find-remotes: Add pull support to the find-remotes built-in command
This will pull the remotes after finding them. This potentially needs to
go in its own pull-from-remotes built-in command, but it will be fine
here for now.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
6453203f54 find-remotes: Add a find-remotes built-in command
This is a wrapper around the new ostree_repo_find_remotes() method; it
tries to find available remotes which can serve updates for the
user-provided refs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
e3d4eeacbc lib/repo-finder: Add Avahi based OstreeRepoFinder implementation
This is a more complex implementation of OstreeRepoFinder which resolves
ref names to remote URIs by looking for refs advertised by peers on the
local network using DNS-SD records and mDNS (Avahi). The idea is to
allow OS and app updates to be propagated over local networks, without
the internet.

It requires an OSTree server and code to generate the DNS-SD adverts in
order to be fully functional — support for this will be added
separately.

Unit tests are included.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
7ee4e1295a lib/bloom: Add an internal bloom filter implementation
This will be used in an upcoming commit. It adds a basic bloom filter
implementation, using the SipHash family of hash functions.

The implementation (including its parameter choices and hash functions)
will become a protocol detail in future, so must not be changed so that
its output is bitwise incompatible between OSTree versions.

Unit tests are included.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
ae335f24dc lib/repo-finder: Add mount based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking for them on any currently mounted
removable storage volumes. The idea is to support OS and app updates via
USB stick.

Unit tests are included.

This bumps libostree’s maximum GLib dependency from 2.44 to 2.50 for
g_drive_is_removable(). If GLib 2.50 is not available, the call which
needs it will be omitted and the OstreeRepoFinderMount implementation
will scan all volumes (not just removable ones); this is a performance
hit, but not a functionality hit.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
d15f83c922 lib/repo-finder: Add config-file based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking their collection IDs up in the local
configuration of remotes who have their collection-id key set.

Unit tests are included.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
292230301d lib/repo-finder: Add basic support for finding remote URIs by ref name
Add an initial OstreeRepoFinder interface (but no implementations),
which will find remote URIs by ref names and collection IDs, the
combination of which is globally unique.

The new API is used in a new ostree_repo_find_updates() function, which
resolves a list of ref names to update into a set of remote URIs to pull
them from, which can be treated as mirrors. It is an attempt to
generalise resolution of the URIs to pull from, and to generalise
determination of the order and parallelisation which they should be
downloaded from in.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
7607d94713 lib/pull: Add collection support to ostree_repo_pull_with_options()
This adds a new collection-refs option of type a(sss), giving a list of
(collection ID, ref name, checksum) tuples to pull from the given remote.
This option is intended to supersede the refs and override-commit-ids
options, so is mutually exclusive with them.

This includes support for resolving the refs from the remote’s summary
file, or from its refs/heads and refs/mirrors directories.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
fbf8df8829 lib/refs: Add methods for setting/listing collection–refs
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
4de736fdfa lib/repo: Add collection ID support to OstreeRepo
Add {get,set}_collection_id() methods to OstreeRepo and some documentation
about the concept of a collection ID which globally identifies an
upstream repository. See the documentation for more details.

This will be used in future commits. For now, the new API is marked as
experimental (--enable-experimental-api).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
0a20e7d43c lib/ref: Add OstreeCollectionRef type for globally unique refs
This is a type representing the tuple (collection ID, ref name), which is
guaranteed to be globally unique. It will be used in upcoming commits.

It introduces the concept of a ‘collection’ which is a unique, curated
set of refs which lie in the same trust domain (i.e. all signed by the
same key and validated by the same developer). Flathub might be a
collection, for example; or the set of OS refs coming from a particular
OS vendor.

It includes a function for validating collection IDs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
bf1f8eb0fa lib/repo: Split out ref handling from regenerate_summary()
This will make some future additions to regenerate_summary() easier.
This commit introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Colin Walters
da0791f484 tests: add a syntax-check rule for glnx_prefix_error()
Same as the errno variant; the colon-space `: ` thing got me in a different
patch.

Closes: #956
Approved by: jlebon
2017-06-26 15:09:12 +00:00
Colin Walters
553b99642c cmd/fsck: Port to new style
Happened to look at this code too, it's a straightforward port.

Closes: #955
Approved by: jlebon
2017-06-26 14:54:32 +00:00
Colin Walters
e23071dc39 lib/deltas: Port to more to new code style
Looking at the uses of `ostree_repo_load_file()` here.

Closes: #953
Approved by: jlebon
2017-06-26 14:42:56 +00:00
Colin Walters
612c8a5fa8 lib/repo: More cleanup of load_file() internals
This is followon work from previous cleanups.  Basically
`stat_bare_content_object()` was the `fstatat()` logic
and `ostree_repo_read_bare_fd()` was the `openat()` implementation;
they duplicated some bits to find the object in staging, recurse
into parent etc.

Further, I wanted an internal-only version of this API which didn't allocate
`GFileInfo`/`GInputStream` but used a plain `fd` and `struct stat` to avoid
mallocs.

The end version here I think looks a lot nicer, since we deduplicate the various
`open()` calls in the different cases for example.

Closes: #952
Approved by: jlebon
2017-06-23 18:29:51 +00:00
Colin Walters
63ad289a9c lib/repo: Split archive/bare file parsing
Prep for future cleanup patches (in particular I want an internal-only
version at first that uses a fd+`struct stat`) to avoid allocations.

The new version avoids lots of deep nesting of conditionals as well
by hoisting the "not found" handling to an early return.

There's a bit of code duplication between the two cases but it's
quite worth the result.

Closes: #951
Approved by: jlebon
2017-06-23 14:11:36 +00:00
Colin Walters
aafda9073a lib/core: Avoid NULL deref in content_file_parse() if out variable unset
Prep for a change in `ostree_repo_load_file()`.  We would crash if a
caller had `out_file_info = NULL`, because we deref `ret_file_info`
below it.

Closes: #951
Approved by: jlebon
2017-06-23 14:11:36 +00:00
Simon McVittie
90b587e2c8 build: Always include ostree-trivial-httpd.xml in tarballs
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #949
Approved by: cgwalters
2017-06-22 13:33:39 +00:00
Philip Withnall
9a79d13ce3 lib/remote: Fix ‘Since’ line for OstreeRemote
There was a typo in it when it was first introduced. Let’s not live in
the past.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall
20829a0582 lib/sym: Fix symbol versions for 2017.7 experimental symbols
There was a typo in the group name. It should be OK to change the
version since this is all hidden behind the --enable-experimental-api
configure option.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall
333d264c5b lib/core: Fix ‘Since’ line for ostree_validate_remote_name()
This was missed when cherry-picking it out of
https://github.com/ostreedev/ostree/pull/924#discussion_r123097919.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Colin Walters
027c77c621 ci: Enable -Werror=maybe-uninitialized
We don't want to inject this warning by default for every build
like the other ones in `configure.ac`, since it can be spruriously
wrong.  But there's no reason not to have a more extended set of
warnings for well-known toolchains (e.g. f25).

Closes: #943
Approved by: jlebon
2017-06-20 22:27:18 +00:00
Colin Walters
46e7f08daa repo: Squash a gcc -Wmaybe-uninitialized warning
It's spurious, but unfortunately GCC doesn't currently understand that it will
always be set.

Closes: #943
Approved by: jlebon
2017-06-20 22:27:18 +00:00
Philip Withnall
20dc9454b3 lib/core: Add ostree_validate_remote_name() for remote names
There are a few places in the code where ad-hoc validation was being
performed. Might as well formalise it a bit more.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #948
Approved by: cgwalters
2017-06-20 21:52:22 +00:00
Philip Withnall
fd50c97322 build: Add ‘devel’ or ‘release’ to OSTREE_FEATURES for test-symbols.sh
test-symbols.sh was looking for the DevelBuild string, which is actually
part of the output from `ostree --version`, not $OSTREE_FEATURES.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #948
Approved by: cgwalters
2017-06-20 21:52:22 +00:00
Jonathan Lebon
82410f0e59 ci: add ci-release-build.sh
Add a check that verifies that `is_release_build` is `yes` only for
release commits. And also verify that the commit message has the correct
version.

Closes: #945
Approved by: cgwalters
2017-06-19 16:16:23 +00:00
Colin Walters
b69c6ba992 build-sys: Post-release version bump
Closes: #944
Approved by: jlebon
2017-06-19 15:51:27 +00:00
Colin Walters
2a082b20b8 main: DevelBuild=yes to ostree --version for devel builds
And use it in `test-symbols.sh`, to fix the `distcheck` case;
the previous change stopped distributing `libostree-devel.sym`
in release builds.

Closes: #944
Approved by: jlebon
2017-06-19 15:51:27 +00:00
Colin Walters
6729b7c264 Release 2017.7
Closes: #942
Approved by: jlebon
v2017.7
2017-06-19 14:54:59 +00:00
Colin Walters
a45dc0fd0b build-sys: Add "release build" flag, use for symbol versioning
I was trying to do a release and move the symbols from `-devel.sym` into
`-release.sym`, but it turns out that at least GNU binutils `ld` treats an empty
version script as a syntax error.  Fix this by adding a "release build"
flag, and only include `-devel` in non-release builds.

This would also make it easier to inject that flag into our `.pc` and
`ostree-version.h` and `ostree --version` metadata, but I didn't do that yet.

EDIT: Turns out a simpler fix is just to add an empty section. However I kept
this commit since it's a useful sanity check for whether we should include
`-devel.sym` in builds, and we may want to inject the metadata later.

Closes: #942
Approved by: jlebon
2017-06-19 14:54:59 +00:00
David Shea
22e753176e lib/repo: Fix annotations for out parameters
Change the annotation of the out parameters on ostree_repo_load_file
from `(allow-none)` to `(optional) (nullable)`. `allow-none` is
ambiguous, since these parameters can be both NULL on input and set to
NULL on return.

Closes: #939
Approved by: cgwalters
2017-06-19 13:15:19 +00:00
Alexander Larsson
07dc33ca4a static delta apply: Work on bare-user-only repos
Flatpak make check is failing when applying a static delta
to a bare-user-only repo due to an assert. The fix is to add
bare-user-only to the assert check.

Closes: #940
Approved by: giuseppe
2017-06-19 09:09:17 +00:00
Colin Walters
fb2c3c1db3 tests: Fix previous commit for selinux testing
I only checked the test passed, I didn't read the output closely, and
made it succeed without testing anything.

Fix the absolute/relative `/etc` references.

Closes: #937
Approved by: jlebon
2017-06-16 15:46:07 +00:00
Alexander Larsson
73ba3eb686 pull: When mirroring, only replace summary if we're doing a full mirror
We're hitting this in flathub, where we have a bunch of local builds,
but we also mirror a few refs from the gnome runtime repo into it.
Its fixable by re-doing the summary, but for a short time the
wrong version is visible.

Fixes https://github.com/ostreedev/ostree/issues/846

Closes: #935
Approved by: cgwalters
2017-06-16 15:08:39 +00:00
Colin Walters
371b4a5e7e checkout: Fix SELinux policy labeling when recursing
The code here tried to truncate the string to the previous length,
but that doesn't work when recursing, since further calls change the
length.

What actually ended up happening was the string would get corrupted
after the first level of recursion.

Closes: #936
Approved by: jlebon
2017-06-16 14:54:29 +00:00
Colin Walters
2bab43fb22 lib: Split symbol versioning into -released and -devel
So far a lot of submitted PR have added symbols into the first
section.  Split the file into `-released` and `-devel` to make
this more obvious.

To further enforce things, we hardcode a checksum of the `-released`
file in `test-symbols.sh`.  Only release commits should update that
checksum.

Did you notice I like checksums?

Closes: #931
Approved by: pwithnall
2017-06-16 09:31:16 +00:00
Tristan Van Berkom
3e3a0f0766 ostreee-version.h.in: Added Since: version annotations
This is especially interesting for the versioning symbols themselves,
as it is an indicator of when applications using introspection information
can start to use a symbol in the library to check if they have a
recent enough version of OSTree to use.

Closes: #932
Approved by: cgwalters
2017-06-15 12:59:59 +00:00
Colin Walters
9529e8d435 lib/pull: Extend BAREUSERONLY_FILES flag to HTTP requests
For the flatpak PR: https://github.com/flatpak/flatpak/pull/849

It's really more convenient if this works for HTTP pulls as well, since flatpak
does various types of pulling, and we can just set the flag everywhere.

Further, we might as well reject the content as early as possible.

Closes: #930
Approved by: alexlarsson
2017-06-15 07:10:22 +00:00
Colin Walters
0e6d23835b lib/sysroot: Add some g_prefix_error() for ostree_sysroot_cleanup()
We saw this fail in a CI run. We've been trying to add strategic error prefixing
as a general rule, and this specific instance may help debug.

Closes: #929
Approved by: pwithnall
2017-06-15 00:08:29 +00:00
Colin Walters
584735b1c9 build-sys: post-release version bump
Per request by flatpak.

Closes: #928
Approved by: alexlarsson
2017-06-14 14:51:25 +00:00
Anton Gerasimov
64ab8334b7 lib/sysroot: Add API to get pending/rollback for given stateroot
This imports a function that is used in rpm-ostree, and it's also intended for
use by https://github.com/advancedtelematic/aktualizr to display
what deployment we're going to boot next after the reboot.

Updated-by: Colin Walters <walters@verbum.org>

Closes: #897
Approved by: OYTIS
2017-06-14 09:56:01 +00:00
Colin Walters
0635fcbfd9 lib/checkout: Add bareuseronly_dirs option
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.

See: https://github.com/flatpak/flatpak/issues/845

This option suppresses mode bits outside of `0775` for directory
checkouts.  I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.

Closes: #927
Approved by: alexlarsson
2017-06-13 20:05:31 +00:00
Colin Walters
6ed824bf00 lib/pull: Add OSTREE_REPO_PULL_FLAGS_BAREUSERONLY_FILES
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845

We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909

This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.

Closes: #926
Approved by: alexlarsson
2017-06-13 18:44:28 +00:00
Colin Walters
74e3581ed6 lib/repo: Support hardlink conversions from bare-user to bu-only
Thinking about the problem of flatpak converting from `bare-user` to `bare-user-only`
"in place" by creating a new repo and doing a `pull-local`, I realized
that we can optimize this process by doing hardlinks for both metadata
and regular files.  The repo formats are *almost* compatible, the
exception being symlinks.

An earlier patch caused us to do hardlinks for metadata, this patch takes things
to the next step and special cases this specific conversion. In this case we
need to parse the source object to determine whether or not it's a symlink.

Closes: #922
Approved by: alexlarsson
2017-06-13 12:02:12 +00:00
Colin Walters
b614c65eab lib/repo: Import metadata via hardlink even for distinct repo modes
Our previous logic for import-via-hardlink only tried if the repo modes match,
but we *can* hardlink metadata between e.g. `archive` and `bare-user` repos, and
that's quite useful thing to do. Our documentation encourages converting to/from
those repo modes locally for build systems.

Closes: #922
Approved by: alexlarsson
2017-06-13 12:02:12 +00:00
Colin Walters
695771667c lib/repo: Skip import via hardlink if repo owners don't match
Before this, if one had repos of matching mode but different owners,
which could happen if one e.g. makes a `bare` non-root repo in
`/ostree/deploy/$stateroot/var/tmp`, every time we tried to call `linkat()`
we'd get `EPERM` and fall back to a copy.

Fix this by saving the repo owner uid, and avoid trying to call `linkat()` if we
know it's going to fail. Of course most commonly in this scenario we'll
immediately fail trying to `chown` the files to `0`, but this is prep for a
future patch to improve `bare-user` → `bare-user-only` imports where we'll be a
bit more sophisticated.

Closes: #922
Approved by: alexlarsson
2017-06-13 12:02:12 +00:00