Commit Graph

3450 Commits

Author SHA1 Message Date
Colin Walters
ea15025c19 lib/pull: Some small style porting
I'd mostly been avoiding this file since there's always patches outstanding, but
these few functions shouldn't conflict much.

Closes: #979
Approved by: jlebon
2017-06-29 15:21:06 +00:00
Colin Walters
250e305f73 lib/repo: Port bareuser-conversion stat to bare load
I noticed this is a simple call that's useful to port to the new internal-only
non-allocating API.

Closes: #977
Approved by: jlebon
2017-06-29 15:07:49 +00:00
Colin Walters
064d7bffef lib/deltas: More porting to new code style
Just noticed some of this while working on the previous tmpfile bits.

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Colin Walters
ab9fef5279 lib/commit: Refactor non-failable size indexing function
It can't throw, so remove the `GError` machinery.

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Colin Walters
6f2ea23e8a libutil: Add a helper for O_TMPFILE + mmap()
I added `glnx_open_anonymous_tmpfile()`, but then later noticed
that the usage of this was really to be combined with `mmap()`,
and we had two versions of that in the delta code.  Add a helper.

(Bigger picture...how is this different from glibc's "mmap() of /dev/zero"
 approach for large chunks? One advantage is the storage can be "swapped" to
 `/var/tmp`, but still deleted automatically, rather than requiring swap space)

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Jonathan Lebon
9d10bdfd0d ci: unconditionally turn on -Werror
Closes: #971
Approved by: cgwalters
2017-06-28 16:37:15 +00:00
Jonathan Lebon
373dc4b66c codebase: start using GLNX_HASH_TABLE_FOREACH macros
Use the new macros introduced recently in libglnx to make iterating over
hash tables cleaner. This is just a start, it does not migrate the whole
tree.

Update submodule: libglnx

Closes: #971
Approved by: cgwalters
2017-06-28 16:37:15 +00:00
Colin Walters
ba918e49c5 tree-wide: Misc porting to newer libglnx APIs
- Use the new tmpfile bits
 - `glnx_try_fallocate`
 - `glnx_renameat()`

Depends: https://github.com/GNOME/libglnx/pull/57

Update submodule: libglnx

Closes: #970
Approved by: jlebon
2017-06-28 15:27:56 +00:00
Colin Walters
5776d5dcc0 Port to GLnxTmpfile
There's lots of mechanically replacing `OtTmpFile` with `GLnxTmpfile`;
the biggest changes are in the commit path.  Symlink commits are now
very clearly separated from regular files.  Symlinks are `OtCleanupUnlinkat`,
and regular files are `GLnxTmpfile`.

The commit codepath separates those as `_ostree_repo_commit_path_final()` and
`_ostree_repo_commit_tmpf_final()`. A nice aspect of all of this is that they
both *consume* the temporary on success. This avoids an extra spurious
`unlink()` call.

One of the biggest bits of code motion is in `commit_loose_regfile_object()`,
which no longer needs to care about symlinks. For the most parth though it's
just removing conditionals.

Update submodule: libglnx

Closes: #958
Approved by: jlebon
2017-06-27 22:02:14 +00:00
Jonathan Lebon
79f285d188 test-switchroot.sh: skip if no busybox
Closes: #968
Approved by: cgwalters
2017-06-27 21:45:34 +00:00
Jonathan Lebon
48e49df7f7 papr: build and test on c7
Start testing on CentOS 7 as well to cover kernel differences (e.g.
O_TMPFILE support).

Closes: #968
Approved by: cgwalters
2017-06-27 21:45:34 +00:00
Colin Walters
7d0e6fb63f ci: Actually run installed tests again
We did it *again* 😭

Closes: #969
Approved by: jlebon
2017-06-27 21:21:18 +00:00
Philip Withnall
a03f0447cb lib/pull: Use ostree_repo_verify_summary() to verify summary on pull
Rather than duplicating the code. This introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall
4c238c350d lib/pull: Check whether summary is in normal form when loading it
Check that it’s in normal form and has the correct type when loading it,
since it could come from an untrusted source.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall
eb4887d619 lib/pull: Don’t cache summary file until its signature is verified
This makes no difference to the validity of the code, since any summary
file loaded from the cache will be verified before being read anyway;
but it will make some upcoming changes a little simpler.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall
e0ad9b2266 ostree/summary: Add support for adding additional metadata
When updating a summary file, parse additional arguments to the `ostree
summary` command as additional metadata to be put into the summary.

Add some tests for this.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall
a432a2b420 tests: Fix incorrect summary --update usage in test-local-pull.sh
Previously the spurious ‘update’ atom was ignored; now that `ostree
summary --update` accepts extra arguments as additional metadata, it’s
causing an error. Drop it.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall
2f297ba4d3 lib/repo: Fix a typo in a documentation comment
Looks like a copy-paste error.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Colin Walters
4796cd0454 pull: Check free space when pulling deltas
Computing download/storage size for `archive` pulls is hard; there's
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` which was from a
pre-static-deltas effort by Endless, but we aren't currently making use of this
much.

Static deltas were designed to solve this problem; we have the total
uncompressed size. Let's check free space before doing a delta pull.

Related: https://github.com/ostreedev/ostree/issues/962

Closes: #963
Approved by: jlebon
2017-06-27 18:42:00 +00:00
Colin Walters
54db9ecab3 build: Don't scan ostree-remote.h for introspection if !experimental-api
It's a bit unfortunate that the scanner doesn't error on this, but
anyways

Closes: https://github.com/ostreedev/ostree/issues/966

Closes: #967
Approved by: jlebon
2017-06-27 17:33:08 +00:00
Colin Walters
c2b6afe5b9 lib/deltas: Some style porting
Just a few functions to keep up momentum.

Closes: #964
Approved by: jlebon
2017-06-27 17:19:21 +00:00
Colin Walters
21cb4d1715 ci: Make introspection warnings fatal
Closes: #965
Approved by: pwithnall
2017-06-27 11:56:44 +00:00
Colin Walters
7871bc3051 lib/ref: Suppress more collection ref methods from introspection
This squashes some warnings.

Closes: #965
Approved by: pwithnall
2017-06-27 11:56:44 +00:00
Colin Walters
5effceeba8 lib/commit: Fix fallocate size for bare-user symlinks
We need to account for the trailing NUL.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Colin Walters
af3a96755b lib: Use OtTmpFile for static delta processing
The `OstreeRepoContentBareCommit` struct was basically an `OtTmpFile`, so let's
make it one. I moved the "convert to `GOutputStream`" logic into the callers,
since that bit can't fail; it makes the implementation much simpler since we can
just return the result of `ot_open_tmpfile_linkable_at()`.

Prep for `GLnxTmpfile` porting.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Colin Walters
4dee1984dc lib: Hoist unlinkat() cleanup API to fsutil, use in pull
The pull code also could make use of this in both the metadata and content
paths. I changed it to own the tempfile malloc (just like `GLnxTmpFile`), since
there's no reason to have different lifetimes for the filename and the file, and
that way we only have one variable rather than two.

The content path turns out to be a special case though, where
at least for mirroring archives, we directly pass the file *path*
down into `_ostree_repo_commit_loose_final()`.

This is prep for `GLnxTmpFile` porting.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Colin Walters
1147267e4d lib/commit: Clean up commit file type handling variables
The variables here were duplicative; we don't need two booleans to distinguish
between symlinks and regular files. What we do need to handle is the "physical"
state versus the "object" state. Symlinks objects are stored as regular files in
`bare-user` and `archive`.

Prep for more cleanup.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Philip Withnall
d0d5f54ce2 lib/refs: Add runtime error checking for collection ID validity
Instead of treating it as a programming error — given that it’s user
input, that’s not really appropriate. This modifies write_ref() and
list_collection_refs() to implement validation.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
11ad3ec529 tests: Add integration tests for collections
Test various pieces of core and command line utility functionality
relating to the newly-introduced concept of collections. Mostly focussed
around the find-remotes utility, and around handling of collection–refs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
3dd4848c96 ostree/builtins: Add support for collection–refs to a few utilities
These utilities were not needed for the initial port to support
OstreeCollectionRef, so have been delayed a bit and, in some cases, left
as FIXME comments for follow up later.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
18456d25fb ostree/dump: Include collection IDs and mirrored refs in summary dumps
If a repository’s summary file includes a collection ID, output that. If
it includes refs from other collections (in the ‘collection map’), output
those and include the same metadata detail as for refs in the summary
file’s main refs map.

If collection IDs are specified in the summary file, this changes the
output format from `ostree summary -v` to use (collection ID, ref name)
tuples.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
b7b79fa78d refs: Add a --collections argument to the built-in refs command
Rather than change the output format used by the existing refs command
to output collection IDs in addition to ref names, this functionality has
been hidden behind an --collections argument. If it’s not specified `ostree
refs` will output the same content as before for a given repository. If
it is specified, the collection ID for each ref will be included in the
output as (collection ID, ref name).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
205a84b36e remote-add: Add a --collection-id argument to the built-in add command
This allows newly configured remotes to have their collection ID specified,
so that refs from them can be downloaded from peers as well as the
upstream collection, using the remote’s configuration.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
144e325579 init: Add a --collection-id argument to the built-in init command
This allows new repositories to be configured with a collection ID which
can be used to uniquely identify refs which originated from this
repository.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
37fc49f36d find-remotes: Add pull support to the find-remotes built-in command
This will pull the remotes after finding them. This potentially needs to
go in its own pull-from-remotes built-in command, but it will be fine
here for now.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
6453203f54 find-remotes: Add a find-remotes built-in command
This is a wrapper around the new ostree_repo_find_remotes() method; it
tries to find available remotes which can serve updates for the
user-provided refs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
e3d4eeacbc lib/repo-finder: Add Avahi based OstreeRepoFinder implementation
This is a more complex implementation of OstreeRepoFinder which resolves
ref names to remote URIs by looking for refs advertised by peers on the
local network using DNS-SD records and mDNS (Avahi). The idea is to
allow OS and app updates to be propagated over local networks, without
the internet.

It requires an OSTree server and code to generate the DNS-SD adverts in
order to be fully functional — support for this will be added
separately.

Unit tests are included.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
7ee4e1295a lib/bloom: Add an internal bloom filter implementation
This will be used in an upcoming commit. It adds a basic bloom filter
implementation, using the SipHash family of hash functions.

The implementation (including its parameter choices and hash functions)
will become a protocol detail in future, so must not be changed so that
its output is bitwise incompatible between OSTree versions.

Unit tests are included.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
ae335f24dc lib/repo-finder: Add mount based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking for them on any currently mounted
removable storage volumes. The idea is to support OS and app updates via
USB stick.

Unit tests are included.

This bumps libostree’s maximum GLib dependency from 2.44 to 2.50 for
g_drive_is_removable(). If GLib 2.50 is not available, the call which
needs it will be omitted and the OstreeRepoFinderMount implementation
will scan all volumes (not just removable ones); this is a performance
hit, but not a functionality hit.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
d15f83c922 lib/repo-finder: Add config-file based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking their collection IDs up in the local
configuration of remotes who have their collection-id key set.

Unit tests are included.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
292230301d lib/repo-finder: Add basic support for finding remote URIs by ref name
Add an initial OstreeRepoFinder interface (but no implementations),
which will find remote URIs by ref names and collection IDs, the
combination of which is globally unique.

The new API is used in a new ostree_repo_find_updates() function, which
resolves a list of ref names to update into a set of remote URIs to pull
them from, which can be treated as mirrors. It is an attempt to
generalise resolution of the URIs to pull from, and to generalise
determination of the order and parallelisation which they should be
downloaded from in.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
7607d94713 lib/pull: Add collection support to ostree_repo_pull_with_options()
This adds a new collection-refs option of type a(sss), giving a list of
(collection ID, ref name, checksum) tuples to pull from the given remote.
This option is intended to supersede the refs and override-commit-ids
options, so is mutually exclusive with them.

This includes support for resolving the refs from the remote’s summary
file, or from its refs/heads and refs/mirrors directories.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
fbf8df8829 lib/refs: Add methods for setting/listing collection–refs
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
4de736fdfa lib/repo: Add collection ID support to OstreeRepo
Add {get,set}_collection_id() methods to OstreeRepo and some documentation
about the concept of a collection ID which globally identifies an
upstream repository. See the documentation for more details.

This will be used in future commits. For now, the new API is marked as
experimental (--enable-experimental-api).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
0a20e7d43c lib/ref: Add OstreeCollectionRef type for globally unique refs
This is a type representing the tuple (collection ID, ref name), which is
guaranteed to be globally unique. It will be used in upcoming commits.

It introduces the concept of a ‘collection’ which is a unique, curated
set of refs which lie in the same trust domain (i.e. all signed by the
same key and validated by the same developer). Flathub might be a
collection, for example; or the set of OS refs coming from a particular
OS vendor.

It includes a function for validating collection IDs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
bf1f8eb0fa lib/repo: Split out ref handling from regenerate_summary()
This will make some future additions to regenerate_summary() easier.
This commit introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Colin Walters
da0791f484 tests: add a syntax-check rule for glnx_prefix_error()
Same as the errno variant; the colon-space `: ` thing got me in a different
patch.

Closes: #956
Approved by: jlebon
2017-06-26 15:09:12 +00:00
Colin Walters
553b99642c cmd/fsck: Port to new style
Happened to look at this code too, it's a straightforward port.

Closes: #955
Approved by: jlebon
2017-06-26 14:54:32 +00:00
Colin Walters
e23071dc39 lib/deltas: Port to more to new code style
Looking at the uses of `ostree_repo_load_file()` here.

Closes: #953
Approved by: jlebon
2017-06-26 14:42:56 +00:00
Colin Walters
612c8a5fa8 lib/repo: More cleanup of load_file() internals
This is followon work from previous cleanups.  Basically
`stat_bare_content_object()` was the `fstatat()` logic
and `ostree_repo_read_bare_fd()` was the `openat()` implementation;
they duplicated some bits to find the object in staging, recurse
into parent etc.

Further, I wanted an internal-only version of this API which didn't allocate
`GFileInfo`/`GInputStream` but used a plain `fd` and `struct stat` to avoid
mallocs.

The end version here I think looks a lot nicer, since we deduplicate the various
`open()` calls in the different cases for example.

Closes: #952
Approved by: jlebon
2017-06-23 18:29:51 +00:00