5046 Commits

Author SHA1 Message Date
OpenShift Merge Robot
c48e7e1060
Merge pull request #2236 from lucab/ups/travis-xenial
ci/travis: move to newer base distro
2020-11-11 17:01:13 +01:00
Luca BRUNO
1c65498d77
ci/travis: move to newer base distro
This removes the old pinned distro (Ubuntu Trusty 14.04) from Travis,
moving to the newer default distro (Ubuntu Xenial 16.04).
2020-11-10 13:06:34 +00:00
OpenShift Merge Robot
34390a53b8
Merge pull request #2233 from cgwalters/repo-annotations
sysroot: Fix up some GI nullable annotations
2020-11-09 18:56:24 +00:00
Colin Walters
bf8c4c7e32 sysroot: Fix up some GI nullable annotations
Hit `ostree_sysroot_repo()` shouldn't be nullable while using
the ostree Rust bindings.
2020-11-06 20:06:26 +00:00
OpenShift Merge Robot
e43d445b5b
Merge pull request #2228 from jlebon/pr/drop-volatile
Drop use of `volatile`
2020-11-03 17:36:14 -05:00
OpenShift Merge Robot
f62c7bae2e
Merge pull request #2230 from alexlarsson/pull-dont-save-passed-in-summary
pull: Don't save into cache passed in GByte summaries
2020-11-03 12:35:06 -05:00
OpenShift Merge Robot
932dcf72c4
Merge pull request #2229 from dbnicholson/static-delta-generate-annotations
lib/deltas: Annotate from checksum as nullable
2020-11-03 15:32:54 +01:00
Alexander Larsson
52463686af pull: Don't save into cache passed in GByte summaries
The cache shouldn't be affected by the user passing in some other
summary as it may not be the "official one".

I ran into this in flatpak where the passed summary was correct, but
the re-saving of the cache updated the mtime of the cached file which
led to later http If-Modified-Since calls failing to update.
2020-11-03 11:48:33 +01:00
Dan Nicholson
3e527d9447 lib/deltas: Annotate from checksum as nullable
Without this you can't create a scratch delta from GI. While here,
switch the deprecated allow-none annotations to nullable.
2020-11-02 16:42:30 -07:00
Jonathan Lebon
f895cf4fd2 Drop use of volatile
As detailed in
gitlab.gnome.org/GNOME/glib/-/issues/600#note_877282, volatile
isn't actually needed in these contexts because the atomic operations
already give us strong enough guarantees. In GCC 11, this triggers a
diagnostic due to the volatile qualifier getting dropped anyway.

There is a WIP to do the same in glib:
https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1719

This obsoletes this downstream patch:
https://src.fedoraproject.org/rpms/ostree/c/b8c5a6fb
2020-11-02 14:53:26 -05:00
OpenShift Merge Robot
64e09f46b8
Merge pull request #2226 from alexlarsson/fix-gir-arg
ostree_repo_gpg_sign_data: Fix API doc argument name
2020-10-28 13:31:50 -04:00
Alexander Larsson
2f78441bea ostree_repo_gpg_sign_data: Fix API doc argument name
I got:
src/libostree/ostree-repo.c:5232: Warning: OSTree: ostree_repo_gpg_sign_data: unknown parameter 'out_signature' in documentation comment, should be 'out_signatures'
2020-10-28 15:53:18 +01:00
OpenShift Merge Robot
7bc53f0063
Merge pull request #2224 from stb-tester/bootloader-refactorings
Bootloader probing and construction refactoring
2020-10-28 06:39:57 -04:00
OpenShift Merge Robot
83eca85ad4
Merge pull request #2225 from jlebon/pr/clear-timer
lib/fetch-curl: Unref timeout source
2020-10-27 15:10:58 -04:00
Jonathan Lebon
8717608c7e lib/fetch-curl: Unref timeout source
The timeout timer should always be one-shot, so let's just always
destroy it in the callback. The main context has its own ref on it, so
it won't be freed behind its back.

This *should* fix a leak that was brought up in
https://bugzilla.redhat.com/show_bug.cgi?id=1891761.

Reported-by: Milan Crha <mcrha@redhat.com>
2020-10-27 12:01:01 -04:00
Jonathan Lebon
40edc33ef3 lib/fetcher-curl: Use G_SOURCE_REMOVE instead of FALSE
They're equivalent, though I prefer the former because it's more
descriptive and it makes it really obvious that it's a `GSource`
callback.
2020-10-27 11:57:00 -04:00
William Manley
663c5b41a3 fixup! Refactor ostree_sysroot_query_bootloader 2020-10-27 13:24:46 +00:00
William Manley
631528c87b fixup! Refactor: Centralise choosing the appropriate bootloader 2020-10-27 12:35:29 +00:00
William Manley
2a6c0b21db Tests: Refactor bootloader-entries-crosscheck
I've made this use functions to make it easier to add support for more
bootloaders.  Seeing as there will be a big diff anyway I've also adjusted
the formatting to make it pep8 compliant.
2020-10-26 23:54:19 +00:00
William Manley
a8dce46b5f Refactor ostree_sysroot_query_bootloader
This is more regular, so will make it easier to add more bootloader types
in the future.
2020-10-26 23:51:11 +00:00
William Manley
31acd2ef99 Add support for explicitly requesting any specific bootloader type
...with the `sysroot.bootloader` configuration option.  This can be useful
when converting a system to use `ostree` which doesn't currently have a
bootloader configuration that `ostree` can automatically detect, and is
also useful in combination with the `--sysroot` option when provisioning a
rootfs for systems other than the one you're running `ostree admin deploy`
on.
2020-10-26 23:51:11 +00:00
William Manley
9482ecfe5a Refactor: sysroot.bootloader: Store enum value rather than string
It's easier to extend and it centralises the config parsing.  In other
places we will no longer need to use `g_str_equal` to match these values,
a `switch` statement will be sufficient.
2020-10-26 23:51:11 +00:00
William Manley
062df6ee81 Refactor: Centralise choosing the appropriate bootloader
In preparation for enhancing `_ostree_sysroot_query_bootloader`
2020-10-26 23:51:11 +00:00
William Manley
5e223f2962 ostree_repo_get_bootloader: Document transfer none
I think this may affect bindings too.
2020-10-26 23:51:11 +00:00
OpenShift Merge Robot
6d64477c8d
Merge pull request #2188 from alexlarsson/delta-indexes
Add indexes for deltas outside of the summary
2020-10-23 09:05:24 -04:00
Alexander Larsson
654f3d959a ostree pull: Add more g_debug spew around fetching deltas
This is useful to debug what is happening when downloading via deltas.
2020-10-23 13:55:33 +02:00
Alexander Larsson
8cd796f3f1 Add ostree_repo_gpg_sign_data()
This is similar to ostree_sign_data() but for the old gpg code.
Flatpak will need this to reproduce a signed summary.
2020-10-23 13:55:33 +02:00
Alexander Larsson
bc924ff870 tests: Add a testcase to ensure we're not using the summary if we don't need it
With deltas outside the summary, if a commit is specified when pulling
we don't download the summary. Verify this.
2020-10-23 13:55:33 +02:00
Alexander Larsson
125ed2b199 pull: Only download summary if we need it for the pull operation
If we have a commit id for all the refs we're pulling, and if we
don't need the summary to list all the refs when mirroring then the
only reason to download the summary is for the list of deltas.

With the new "indexed-deltas" property in the config file (and mirrored
to the summary file) we can detect when we don't need the summary for
deltas and completely avoid downloading it then.
2020-10-23 13:55:33 +02:00
Alexander Larsson
6c8e6539e2 deltas: Set indexed-deltas key in the config and summary
Clients can use these during pull and avoid downloading the summary if
needed, or use the indexed-deltas instead of relying on the ones in
the summary which may be left out.
2020-10-23 13:06:46 +02:00
Alexander Larsson
0984ff8471 deltas: Take a shared repo lock while reindexing deltas
This ensures we're not racing with a prune operation that can be removing
the delta indexes we're relying on.
2020-10-23 13:06:46 +02:00
Alexander Larsson
e8a7485458 deltas: Add tests for delta indexes
This tests generation of the index as well as using it when pulling
2020-10-23 13:06:46 +02:00
Alexander Larsson
df7f07fc6c deltas: Use delta indexes when pulling
If there is no delta index in the summary, try to fetch the
delta index for the commit we're going to and use that to find the
delta (if any).
2020-10-23 13:06:42 +02:00
Alexander Larsson
625606a7ec deltas: Add CLI ops to list and reindex delta-indexes 2020-10-23 12:30:08 +02:00
Alexander Larsson
c304703e1d deltas: Make ostree_repo_static_delta_reindex() public
It is useful to be able to trigger this without having to regenerate
the summary. For example, if you are not using summaries, or ar generating
the summaries yourself.
2020-10-23 12:30:08 +02:00
Alexander Larsson
024ef1d756 deltas: Add and document no-deltas-in-summary config option
By default this is FALSE to keep existing clients working.
2020-10-23 12:30:08 +02:00
Alexander Larsson
effde3d513 deltas: Update delta indexes when updating summary
When we update the summary file (and its list of deltas) we also update
all delta indexes. The index format is a single `a{sv}` variant identical
to the metadata-part of the summary with (currently) only the
`ostree.static-deltas` key.

Since we expect most delta indexes to change rarely, we avoid
unnecessary writes when reindexing. New indexes are compared to
existing ones and only the changed ones are written to disk.  This
avoids unnecessary write load and mtime changes on the repo server.
2020-10-23 12:30:08 +02:00
Alexander Larsson
8e1f199dd4 deltas: Add ostree_repo_list_static_delta_indexes() function
This lists all the available delta indexes.
2020-10-23 12:30:08 +02:00
Alexander Larsson
87e564eeaa deltas: Add _ostree_get_relative_static_delta_index_path()
This gets the subpath for a delta index file, which is of the form
"delta-indexes/$commit.index", that contains all the deltas going
to the particular commit.
2020-10-23 12:30:08 +02:00
OpenShift Merge Robot
f8f6464580
Merge pull request #2205 from pwithnall/etags-and-last-modified
Add support for ETag and Last-Modified headers for summary and summary.sig
2020-10-22 18:20:23 -04:00
Philip Withnall
0974a7faf1 tests: Split RFC 2616 date parsing code out and add tests
This makes it testable, and increases its test coverage too 100% of
lines, as measured by `make coverage`.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
92215025aa ostree/trivial-httpd: Add Last-Modified/ETag support
This is basic support for the
Last-Modified/ETag/If-Modified-Since/If-None-Match headers. It’s not
high performance, and doesn’t support all of the related caching
features (like the If-Match header, etc.).

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
a522bf7628 tests: Add simple test for summary file caching
This test would have actually passed before the summary file caching
changes (in the previous few commits) were added, as the `summary.sig`
essentially acted as the ETag for the summary file, and itself wasn’t
updated on disk if it didn’t change when querying the server.

Actually testing that the HTTP caching headers are working to reduce
HTTP traffic would require test hooks into the pull code or the
trivial-httpd server, neither of which I have the time to add at the
moment.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
0e3add8d40 lib/pull: Hook up HTTP caching headers for summary and summary.sig
As `summary` and `summary.sig` aren’t immutable, HTTP requests to
download them can be optimised by sending the `If-None-Match` and
`If-Modified-Since` headers to avoid unnecessarily re-downloading them
if they haven’t changed since last being checked.

Hook them up to the new support for that in the fetcher.

The `ETag` and `Last-Modified` for each file in the cache are stored as
the `user.etag` xattr and the mtime, respectively. For flatpak, for
example, this affects the cached files in
`~/.local/share/flatpak/repo/tmp/cache/summaries`.

If xattrs aren’t supported, or if the server doesn’t support the caching
headers, the pull behaviour is unchanged from before.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
407c683524 libostree: Add support for ETag and Last-Modified headers
Add support in the soup and curl fetchers to send the `If-None-Match`
and `If-Modified-Since` request headers, and pass on the `ETag` and
`Last-Modified` response headers.

This currently introduces no functional changes, but once call sites
provide the appropriate integration, this will allow HTTP caching to
happen with requests (typically with metadata requests, where the data
is not immutable due to being content-addressed). That should reduce
bandwidth requirements.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
OpenShift Merge Robot
95a6d15145
Merge pull request #2222 from fkrull/patch-v2020.7
lib: add missing GI annotation
2020-10-20 16:48:37 -04:00
OpenShift Merge Robot
588fb19cd7
Merge pull request #2221 from kelvinfan001/fix-docs
docs: Fix various typos
2020-10-19 14:11:50 -04:00
Kelvin Fan
a89d011c0b docs: Fix various typos 2020-10-19 10:55:49 -04:00
Felix Krull
4183b68e70 lib: fix GI parameter tags 2020-10-17 22:14:09 +02:00
OpenShift Merge Robot
e2d5e3874d
Merge pull request #2219 from jlebon/pr/fix-inode-print-format
ostree-prepare-root: print st_dev and st_ino as 64-bit ints
2020-10-15 16:15:37 -04:00