Commit Graph

5073 Commits

Author SHA1 Message Date
William Manley
5e223f2962 ostree_repo_get_bootloader: Document transfer none
I think this may affect bindings too.
2020-10-26 23:51:11 +00:00
OpenShift Merge Robot
6d64477c8d
Merge pull request #2188 from alexlarsson/delta-indexes
Add indexes for deltas outside of the summary
2020-10-23 09:05:24 -04:00
Alexander Larsson
654f3d959a ostree pull: Add more g_debug spew around fetching deltas
This is useful to debug what is happening when downloading via deltas.
2020-10-23 13:55:33 +02:00
Alexander Larsson
8cd796f3f1 Add ostree_repo_gpg_sign_data()
This is similar to ostree_sign_data() but for the old gpg code.
Flatpak will need this to reproduce a signed summary.
2020-10-23 13:55:33 +02:00
Alexander Larsson
bc924ff870 tests: Add a testcase to ensure we're not using the summary if we don't need it
With deltas outside the summary, if a commit is specified when pulling
we don't download the summary. Verify this.
2020-10-23 13:55:33 +02:00
Alexander Larsson
125ed2b199 pull: Only download summary if we need it for the pull operation
If we have a commit id for all the refs we're pulling, and if we
don't need the summary to list all the refs when mirroring then the
only reason to download the summary is for the list of deltas.

With the new "indexed-deltas" property in the config file (and mirrored
to the summary file) we can detect when we don't need the summary for
deltas and completely avoid downloading it then.
2020-10-23 13:55:33 +02:00
Alexander Larsson
6c8e6539e2 deltas: Set indexed-deltas key in the config and summary
Clients can use these during pull and avoid downloading the summary if
needed, or use the indexed-deltas instead of relying on the ones in
the summary which may be left out.
2020-10-23 13:06:46 +02:00
Alexander Larsson
0984ff8471 deltas: Take a shared repo lock while reindexing deltas
This ensures we're not racing with a prune operation that can be removing
the delta indexes we're relying on.
2020-10-23 13:06:46 +02:00
Alexander Larsson
e8a7485458 deltas: Add tests for delta indexes
This tests generation of the index as well as using it when pulling
2020-10-23 13:06:46 +02:00
Alexander Larsson
df7f07fc6c deltas: Use delta indexes when pulling
If there is no delta index in the summary, try to fetch the
delta index for the commit we're going to and use that to find the
delta (if any).
2020-10-23 13:06:42 +02:00
Alexander Larsson
625606a7ec deltas: Add CLI ops to list and reindex delta-indexes 2020-10-23 12:30:08 +02:00
Alexander Larsson
c304703e1d deltas: Make ostree_repo_static_delta_reindex() public
It is useful to be able to trigger this without having to regenerate
the summary. For example, if you are not using summaries, or ar generating
the summaries yourself.
2020-10-23 12:30:08 +02:00
Alexander Larsson
024ef1d756 deltas: Add and document no-deltas-in-summary config option
By default this is FALSE to keep existing clients working.
2020-10-23 12:30:08 +02:00
Alexander Larsson
effde3d513 deltas: Update delta indexes when updating summary
When we update the summary file (and its list of deltas) we also update
all delta indexes. The index format is a single `a{sv}` variant identical
to the metadata-part of the summary with (currently) only the
`ostree.static-deltas` key.

Since we expect most delta indexes to change rarely, we avoid
unnecessary writes when reindexing. New indexes are compared to
existing ones and only the changed ones are written to disk.  This
avoids unnecessary write load and mtime changes on the repo server.
2020-10-23 12:30:08 +02:00
Alexander Larsson
8e1f199dd4 deltas: Add ostree_repo_list_static_delta_indexes() function
This lists all the available delta indexes.
2020-10-23 12:30:08 +02:00
Alexander Larsson
87e564eeaa deltas: Add _ostree_get_relative_static_delta_index_path()
This gets the subpath for a delta index file, which is of the form
"delta-indexes/$commit.index", that contains all the deltas going
to the particular commit.
2020-10-23 12:30:08 +02:00
OpenShift Merge Robot
f8f6464580
Merge pull request #2205 from pwithnall/etags-and-last-modified
Add support for ETag and Last-Modified headers for summary and summary.sig
2020-10-22 18:20:23 -04:00
Philip Withnall
0974a7faf1 tests: Split RFC 2616 date parsing code out and add tests
This makes it testable, and increases its test coverage too 100% of
lines, as measured by `make coverage`.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
92215025aa ostree/trivial-httpd: Add Last-Modified/ETag support
This is basic support for the
Last-Modified/ETag/If-Modified-Since/If-None-Match headers. It’s not
high performance, and doesn’t support all of the related caching
features (like the If-Match header, etc.).

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
a522bf7628 tests: Add simple test for summary file caching
This test would have actually passed before the summary file caching
changes (in the previous few commits) were added, as the `summary.sig`
essentially acted as the ETag for the summary file, and itself wasn’t
updated on disk if it didn’t change when querying the server.

Actually testing that the HTTP caching headers are working to reduce
HTTP traffic would require test hooks into the pull code or the
trivial-httpd server, neither of which I have the time to add at the
moment.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
0e3add8d40 lib/pull: Hook up HTTP caching headers for summary and summary.sig
As `summary` and `summary.sig` aren’t immutable, HTTP requests to
download them can be optimised by sending the `If-None-Match` and
`If-Modified-Since` headers to avoid unnecessarily re-downloading them
if they haven’t changed since last being checked.

Hook them up to the new support for that in the fetcher.

The `ETag` and `Last-Modified` for each file in the cache are stored as
the `user.etag` xattr and the mtime, respectively. For flatpak, for
example, this affects the cached files in
`~/.local/share/flatpak/repo/tmp/cache/summaries`.

If xattrs aren’t supported, or if the server doesn’t support the caching
headers, the pull behaviour is unchanged from before.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
407c683524 libostree: Add support for ETag and Last-Modified headers
Add support in the soup and curl fetchers to send the `If-None-Match`
and `If-Modified-Since` request headers, and pass on the `ETag` and
`Last-Modified` response headers.

This currently introduces no functional changes, but once call sites
provide the appropriate integration, this will allow HTTP caching to
happen with requests (typically with metadata requests, where the data
is not immutable due to being content-addressed). That should reduce
bandwidth requirements.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
OpenShift Merge Robot
95a6d15145
Merge pull request #2222 from fkrull/patch-v2020.7
lib: add missing GI annotation
2020-10-20 16:48:37 -04:00
OpenShift Merge Robot
588fb19cd7
Merge pull request #2221 from kelvinfan001/fix-docs
docs: Fix various typos
2020-10-19 14:11:50 -04:00
Kelvin Fan
a89d011c0b docs: Fix various typos 2020-10-19 10:55:49 -04:00
Felix Krull
4183b68e70 lib: fix GI parameter tags 2020-10-17 22:14:09 +02:00
OpenShift Merge Robot
e2d5e3874d
Merge pull request #2219 from jlebon/pr/fix-inode-print-format
ostree-prepare-root: print st_dev and st_ino as 64-bit ints
2020-10-15 16:15:37 -04:00
OpenShift Merge Robot
5d565790d4
Merge pull request #2218 from cgwalters/travis-32bit
travis: Add a 32 bit build
2020-10-15 15:27:48 -04:00
Jonathan Lebon
dec9eab203 ostree-prepare-root: print st_dev and st_ino as 64-bit ints
This matches what systemd does and should work fine on all platforms.

Possibly resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1888436
2020-10-15 12:26:47 -04:00
Colin Walters
53b6bbbdf2 travis: Add a 32 bit build
In the past we've had 32 bit bugs that were caught by
the compiler, let's add this to Travis.
2020-10-15 10:50:22 -04:00
OpenShift Merge Robot
26b98ebc56
Merge pull request #2215 from cgwalters/fd-transfer
deploy: Remove (transfer none) from fd arg
2020-10-14 00:45:13 +02:00
OpenShift Merge Robot
cfa25837dc
Merge pull request #2214 from cgwalters/release-2020-7
Release 2020.7
2020-10-14 00:18:49 +02:00
Colin Walters
b4e7ab014e deploy: Remove (transfer none) from fd arg
GI complains.  And in general one needs to assume that file
descriptors aren't stolen.
2020-10-13 17:40:11 -04:00
Colin Walters
326b8b13c6 Post-release version bump 2020-10-13 17:38:19 -04:00
Colin Walters
32a3a12973 Release 2020.7 2020-10-13 14:31:26 -04:00
OpenShift Merge Robot
a13509ee7d
Merge pull request #2213 from alexlarsson/summary-bugfixes
Various fixes from the indexed-summaries branch
2020-10-08 15:18:34 -04:00
Alexander Larsson
2e9db809b9 signatures: Fix leak in _sign_detached_metadata_append()
This needs to ref_sink the returned variant, as it is used with g_autoptr
in the callers.
2020-10-08 14:10:04 +02:00
Alexander Larsson
f821cdb89e fetch_summary_with_options: Fix n-network-retries option parsing
"&u" is not a valid gvariant format string, it should just be "u".
2020-10-08 14:09:16 +02:00
Alexander Larsson
6ed1066ef0 ostree_repo_find_remotes_async: Fix leak of summary
We were creating a GVariant from a GBytes and storing it in an
g_autoptr without ref_sinking it.
2020-10-08 14:08:51 +02:00
OpenShift Merge Robot
cc1c26f55f
Merge pull request #2212 from jlebon/pr/packit-tweak
ci: Make Packit ignore downstream patches
2020-10-07 15:34:22 -04:00
OpenShift Merge Robot
8eb2d5f820
Merge pull request #2211 from jlebon/pr/close-fd
lib/deploy: Don't leak fd when checksumming dtbs
2020-10-07 15:23:43 -04:00
Jonathan Lebon
ee632e4968 ci: Make Packit ignore downstream patches
We don't really carry "Fedora-only" patches in dist-git. So we want to
nuke all the patches which exist there.

Follow-up to #2210.
2020-10-07 15:04:17 -04:00
Jonathan Lebon
b3dc074f5e lib/deploy: Don't leak fd when checksumming dtbs
Likely the root of https://bugzilla.redhat.com/show_bug.cgi?id=1886149.
2020-10-07 14:47:34 -04:00
OpenShift Merge Robot
84f8de6daa
Merge pull request #2210 from jlebon/pr/packit
Add Packit integration
2020-10-07 09:46:14 -04:00
Jonathan Lebon
9c51aa3090 Add Packit integration
This is a basic `.packit.yaml` integration file which will allow us have
continuous builds of OSTree in cosa and upstream CI. If things go well,
we'll likely deploy this in other build tools like rpm-ostree.

Prompted by wanting to get #2155 out to unblock
https://github.com/coreos/rpm-ostree/pull/2170.
2020-10-06 14:20:57 -04:00
OpenShift Merge Robot
9f98aa9223
Merge pull request #2198 from cgwalters/no-hardlink-zerosize
checkout: Don't hardlink zero sized files
2020-10-05 22:06:38 +02:00
OpenShift Merge Robot
0462de63a1
Merge pull request #2209 from travier/docs-fixup
docs: Fix URL in Jekyll _config.yml
2020-10-05 21:50:00 +02:00
Timothée Ravier
0bbd89e326
docs: Fix URL in Jekyll _config.yml 2020-10-05 21:10:31 +02:00
OpenShift Merge Robot
13290a070c
Merge pull request #2208 from cgwalters/bump-libglnx
libglnx: Bump to master
2020-10-05 20:56:35 +02:00
OpenShift Merge Robot
de5704fbe5
Merge pull request #2207 from travier/docs
docs: Add GitHub Pages support
2020-10-05 20:19:29 +02:00