Commit Graph

3627 Commits

Author SHA1 Message Date
Dan Nicholson
814e481fff fixup! bin/remote: Add list-gpg-keys subcommand 2021-07-15 16:25:13 -06:00
Dan Nicholson
30c054b521 fixup! lib/repo: Add ostree_repo_remote_get_gpg_keys() 2021-07-15 16:24:36 -06:00
Dan Nicholson
90a3bda1f8 bin/remote: Include update URLs in list-gpg-keys 2021-07-15 15:50:04 -06:00
Dan Nicholson
27dc5d7d38 lib/repo: Include WKD update URLs in GPG key listing
If the key UID contains a valid email address, include the GPG WKD
update URLs in GVariant returned by ostree_repo_remote_get_gpg_keys().
2021-07-15 15:50:04 -06:00
Dan Nicholson
4fa403aee5 libotutil: Add helper for GPG WKD update URLs
Calculate the advanced and direct update URLs for the key discovery
portion[1] of the OpenPGP Web Key Directory specification, and include
the URLs in the key listing in ostree_repo_remote_get_gpg_keys(). These
URLs can be used to locate updated GPG keys for the remote.

1. https://datatracker.ietf.org/doc/html/draft-koch-openpgp-webkey-service#section-3.1
2021-07-15 15:50:04 -06:00
Dan Nicholson
fbff05e28d libotutil: Import implementation of zbase32 encoding
This will be used to implement the PGP Web Key Directory (WKD) URL
generation. This is a slightly cleaned up implementation[1] taken from
the zbase32 author's original implementation[2]. It provides a single
zbase32_encode API to convert a set of bytes to the zbase32 encoding.

I believe this should be acceptable for inclusion in ostree. The license
in the source files is BSD style while the original repo LICENSE file
claims the Creative Commons CC0 1.0 Universal license, which is public
domain.

1. https://github.com/dbnicholson/libbase32/tree/for-ostree
2. https://github.com/zooko/libbase32
2021-07-15 15:50:04 -06:00
Dan Nicholson
74fb0c5f78 bin/remote: Add list-gpg-keys subcommand
This provides a wrapper for the `ostree_repo_remote_get_gpg_keys`
function to show the GPG keys associated with a remote. This is
particularly useful for validating that GPG key updates have been
applied. Tests are added, which checks the
`ostree_repo_remote_get_gpg_keys` API by extension.
2021-07-15 15:50:04 -06:00
Dan Nicholson
a50f6d0b9f lib/repo: Add ostree_repo_remote_get_gpg_keys()
This function enumerates the trusted GPG keys for a remote and returns
an array of `GVariant`s describing them. This is useful to see which
keys are collected by ostree for a particular remote. The same
information can be gathered with `gpg`. However, since ostree allows
multiple keyring locations, that's only really useful if you have
knowledge of how ostree collects GPG keyrings.

The format of the variants is documented in
`OSTREE_GPG_KEY_GVARIANT_FORMAT`. This format is primarily a copy of
selected fields within `gpgme_key_t` and its subtypes. The fields are
placed within vardicts rather than using a more efficient tuple of
concrete types. This will allow flexibility if more components of
`gpgme_key_t` are desired in the future.
2021-07-15 15:50:04 -06:00
Dan Nicholson
fc073654dc lib/repo: Allow preparing GPG verifier without global keyrings
Currently the verifier decides whether to include the global keyrings
based on whether the specified remote has its own keyring or not. Allow
callers to exclude the global keyrings even when that's not the case.
This will be used in a subsequent commit in order to get the GPG keys
only associated with a remote.
2021-07-15 15:50:04 -06:00
Dan Nicholson
c8715c123e lib/repo: Factor out GPG verifier preparation
In order to use the GPG verifier, it needs to be seeded with GPG keys
after instantation. Currently this is only used for verifying data, but
it will also be used for getting a list of trusted GPG keys in a
subsequent commit.
2021-07-15 15:50:04 -06:00
Dan Nicholson
dba2cdcbac lib/repo: Factor out GPG verifier key imports
Currently the verifier only imports all the GPG keys when verifying
data, but it would also be useful for inspecting the trusted keys.
2021-07-15 15:50:04 -06:00
Luca BRUNO
38c14b3745
Release 2021.3 2021-07-12 08:38:38 +00:00
Jonathan Lebon
301b52704e ostree-remount: Order before systemd-rfkill.*
The `systemd-rfkill.*` service falls in the category of early things
that need write access to `/var`, so we need to make sure we run before
or it might hit the read-only sysroot.

The long-term fix for this is
https://github.com/ostreedev/ostree/issues/2115.

Closes: https://github.com/coreos/fedora-coreos-tracker/issues/746
2021-06-22 11:22:47 -04:00
Luca BRUNO
70a8f56ce1
lib/commit: respect SOURCE_DATE_EPOCH for commit timestamp
This tweaks `ostree_repo_write_commit` so that it checks for the
envinroment variable `SOURCE_DATE_EPOCH` as a way to override
the current time, which is used as the commit timestamp.

Ref: https://reproducible-builds.org/docs/source-date-epoch/
Ref: https://reproducible-builds.org/specs/source-date-epoch/
2021-06-22 12:15:18 +00:00
Dan Nicholson
d9ef76a598 Don't fail build when systemd unit path not defined
In configure the systemd unit path is optional, but in the code it's
assumed to be defined. Add an `#ifdef` that throws an error when it's
not defined like the handling of `HAVE_LIBMOUNT` below it.
2021-06-18 12:35:41 -06:00
Jonathan Lebon
c9a318faf4
Merge pull request #2375 from cgwalters/generator-remount
Use generator to enable ostree-remount.service and ostree-finalize-staged.path
2021-06-18 10:25:35 -04:00
Colin Walters
73e3ccc401 Use generator to enable ostree-remount.service and ostree-finalize-staged.path
We struggled for a long time with enablement of our "internal units",
trying to follow the philosophy that units should only be enabled
by explicit preset.

See https://bugzilla.redhat.com/show_bug.cgi?id=1451458
and https://github.com/coreos/rpm-ostree/pull/1482
etc.

And I just saw chat (RH internal on a proprietary system sadly) where
someone hit `ostree-remount.service` not being enabled in CentOS8.

Thinking about this more, I realized we've shipped a systemd generator
for a long time and while its only role until now was to generate `var.mount`,
but by using it to force on our internal units, we don't require
people to deal with presets anymore.

Basically we're inverting things so that "if ostree= is on the kernel
cmdline, then enable our units" and not "enable our units, but have
them use ConditionKernelCmdline=ostree to skip".

Drop the weird gyrations we were doing around `ostree-finalize-staged.path`
too; forking `systemctl start` is just asking for bugs.

So after this, hopefully we won't ever again have to think about
distribution presets and our units.
2021-06-16 09:40:28 -04:00
Christian Kellner
f653545833 Fix small typo in ostree-sysroot.c 2021-06-15 23:02:10 +02:00
Colin Walters
edf7477ee9 deploy: Warn if we find content in the deployment's /var
This will be ignored, so let's make it very clear
people are doing something wrong.  Motivated by a bug
in a build pipeline that injected `/var/lib/rpm` into an ostree
commit which ended up crashing rpm-ostree because it was an empty db
which it wasn't expecting.

It *also* turns out rpm-ostree is incorrectly dumping content in the
deployment `/var` today, which is another bug.
2021-06-10 07:33:17 -04:00
Dan Nicholson
0cd9dfe815 repo: Use g_new for OstreeRepoAutoLock
GSlice is effectively deprecated and has little to no advantage over
using the system allocator on Linux.
2021-06-07 13:14:05 -06:00
Dan Nicholson
89f4ce2c1d repo: Make locking precondition failures fatal
Use `g_error` and `g_assert*` rather than `g_return*` when checking the
locking preconditions so that failures result in the program
terminating. Since this code is protecting filesystem data, we'd rather
crash than delete or corrupt data unexpectedly.

`g_error` is used when the error is due to the caller requesting an
invalid transition like attempting to pop a lock type that hasn't been
taken. It also provides a semi-useful message about what happened.
2021-06-05 09:15:34 -06:00
Dan Nicholson
ccef9784d7 repo: Make locking per-OstreeRepo
Previously each thread maintained its own lock file descriptor
regardless of whether the thread was using the same `OstreeRepo` as
another thread. This was very safe but it made certain multithreaded
procedures difficult. For example, if a main thread took an exclusive
lock and then spawned worker threads, it would deadlock if one of the
worker threads tried to acquire the lock.

This moves the file descriptor from thread local storage to the
`OstreeRepo` structure so that threads using the same `OstreeRepo` can
share the lock. A mutex guards against threads altering the lock state
concurrently.

Fixes: #2344
2021-06-05 09:15:32 -06:00
Dan Nicholson
c3ada6fa7a repo: Require lock type in ostree_repo_lock_pop
This simplifies the lock state management considerably since the
previously pushed type doesn't need to be tracked. Instead, 2 counters
are kept to track how many times each lock type has been pushed. When
the number of exclusive locks drops to 0, the lock transitions back to
shared.
2021-06-05 09:07:39 -06:00
Colin Walters
0f36d8c221 repo: Make locking APIs public
Doing anything even somewhat sophisticated requires this;
turns out our own `ostree prune` CLI wants this, e.g.
https://github.com/ostreedev/ostree/issues/2337

Closes: https://github.com/ostreedev/ostree/issues/2286
2021-06-05 09:00:21 -06:00
Timothée Ravier
02527f115e *: rename master to main in tests & examples 2021-05-07 16:55:03 +02:00
Colin Walters
0f3bccf640 pull: Cleanly error out on unknown schemes
Previous to this we'd trip an assertion `abort()` deep in the curl code if e.g.
a user did `ostree remote add foo htttp://...` etc.

Motivated by considering supporting "external remotes" where code outside
ostree does a pull, but we want to reuse the signing verification infrastructure.
2021-04-27 13:11:18 -04:00
Dan Nicholson
e660855796 tests: Test without a cache directory by default
Several tests generate summaries and then expect to use the generated
summary immediately. However, this can cause intermittent test failures
when they inadvertantly get a cached summary file. This typically
happens when the test is run on a filesystem that doesn't support user
extended attributes. In that case, the caching code can only use the
last modified time, which only has 1 second granularity. If tests don't
carefully manage the summary modification times or the repo cache then
they are likely subject to races in some test environments.

This introduces an environment variable `OSTREE_SKIP_CACHE` that
prevents the repo from using a cache directory. This is enabled by
default in tests and disabled for tests that are a explicitly trying to
test the caching behavior.

Fixes: #2313
Fixes: #2351
2021-04-19 11:09:44 -06:00
Colin Walters
6a72674ec6 Release 2021.2 2021-04-15 13:02:48 -04:00
Colin Walters
9b4bffc454 repo: Ensure we set the size for regfile inline
Need to clean up the internal APIs for this.
2021-04-09 22:29:55 +00:00
Colin Walters
6f84aff0ae repo: Add ostree_repo_write_regfile
This API is push rather than pull, which makes it much more
suitable to use cases like parsing a tar file from external
code.

Now, we have a large mess in this area internally because
the original file writing code was pull based, but static
deltas hit the same problem of wanting a push API, so I added
this special `OstreeRepoBareContent` just for writing regular
files from a push API.

Eventually...I'd like to deprecate the pull based API,
and rework things so that for regular files the push API
is the default, and then `write_content_object()` would
be split up into archive/bare cases.

In this world the `ostree_repo_write_content()` API would
then need to hackily bridge pull to push and it'd be
less efficient.

Anyways for now due to this bifurcation, this API only
works on non-archive repositories, but that's fine for
now because that's what I want for the `ostree-ext-container`
bits.
2021-04-09 21:54:44 +00:00
Colin Walters
fce69cdf70 repo: Add ostree_repo_write_symlink
Continuation of the addition of `ostree_repo_write_regfile_inline()`.
This will be helpful for ostree-rs-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.
2021-04-08 21:10:00 +00:00
Colin Walters
9332955b5f
Merge pull request #2327 from cgwalters/writing-apis
repo: Add ostree_repo_write_regfile_inline
2021-04-08 17:09:11 -04:00
Colin Walters
4e2a14eb0c repo: Add ostree_repo_write_regfile_inline
When working on ostree-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.

Plus the gtk-rs binding API to map from `impl Read` to Gio
https://docs.rs/gio/0.9.1/gio/struct.ReadInputStream.html
requires that the input stream is `Send` but the Rust `tar` API
isn't.

This is only 1/3 of the problem; we also need similar APIs
to directly create a symlink, and to stream large objects via
a push-based API.
2021-04-08 14:57:33 +00:00
Colin Walters
dfaf314c42 core: Drop unused error handling from object stream helper
I was going to add some new API and I noticed that this function
never returns an error; presumably at one point it did, but
not anymore.  It simplifies the code flow noticeably
to remove that.
2021-04-07 20:01:07 +00:00
Colin Walters
b5c21defe9 core: Fix lgtm.com warning about always true if (bits > 0)
Since we're not going to change this, let's constant fold the logic
here.
2021-04-07 11:48:35 -04:00
Stefan Berger
81d3017463 rofiles-fuse: Enable support for setting and getting xattrs
Enable support for setting and getting xattrs. Allow modifications
to xattrs only on user.ima xattr.

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
Co-authored-by: Colin Walters <walters@verbum.org>
2021-04-05 17:01:58 -04:00
Colin Walters
57270db9a1 repo: Ensure load_variant_if_exists sets NULL value
Spotted by @jlebon - we usually expect callers to zero, but
let's be explicit.
2021-04-03 16:44:39 -04:00
Colin Walters
0c0e58e06c repo: Fix load_variant_if_exists to return a nullable value
Another introspection fix I hit when trying to use this in Rust.
2021-04-03 16:44:39 -04:00
Felix Krull
47bf29fed3 lib: fix some version tags 2021-03-26 15:59:07 -04:00
Colin Walters
e9e4b91120 Release 2021.1 2021-03-23 15:23:55 -04:00
Jonathan Lebon
36a543ceb1 lib/pull: Add some error-prefixing in dirtree scanning
I think these are the paths involved in the error message at
https://github.com/coreos/rpm-ostree/issues/2250.
2021-03-22 16:32:11 -04:00
Kenneth J. Miller
cfe313afae Add configure option for unsuffixed GRUB2 commands
GRUB starting with version 2.02 permits the use of the linux and
initrd commands for both EFI boot in *-efi installations, as
well as 32-bit BIOS boot in i386-pc installations.

This makes the use of the -16 and -efi suffixes for BIOS and EFI
boot obsolete on systems with a modern GRUB installation.

The --with-modern-grub configure flag makes ostree use the
unsuffixed linux/initrd commands when generating a GRUB
configuration, while defaulting to the previous behaviour for
users not wanted this option.
2021-03-18 14:31:28 +01:00
Colin Walters
1b28e6041c sysroot: Add _require_booted_deployment() API
This is a common pattern that is replicated both in our code
and in rpm-ostree a lot.  Let's add a canonical API.
2021-03-17 19:55:56 +00:00
Colin Walters
857587615d Add an API+CLI to inject metadata for bootable OSTree commits
I was doing some rpm-ostree work and I wanted to compare two
OSTree commits to see if the kernel has changed.  I think
this should be a lot more natural.

Add `ostree commit --bootable` which calls into a new generic
library API `ostree_commit_metadata_for_bootable()` that
discovers the kernel version and injects it as an `ostree.linux`
metadata key.  And for extra clarity, add an `ostree.bootable`
key.

It's interesting because the "core" OSTree layer is all about
generic files, but this is adding special APIs around bootable
OSTree commits (as opposed to e.g. flatpak as well as
things like rpm-ostree's pkgcache refs).

Eventually, I'd like to ensure everyone is using this and
hard require this metadata key for the `ostree admin deploy`
flow - mainly to prevent accidents.
2021-03-12 19:01:42 +00:00
Phaedrus Leeds
19577522f8 Fix translation of file:// URIs into paths
Currently if a file path contains a special character such as '\', and
that character is encoded into a file:// URI that is passed to
ostree_repo_pull_with_options(), the percent encoding will remain in the
path passed to g_file_new() (in the case of backslash %5C) and the pull
will then fail with a file not found error. This is an important edge
case to handle because by default on many Linux distributions a
filesystem with no label is mounted at a path based on its UUID, and
this is then passed to systemd-escape by Flatpak (when
--enable-auto-sideloading was used at compile time) to create a symbolic
link such as this which contains backslashes:

$ ls -l /run/flatpak/sideload-repos/
total 0
lrwxrwxrwx 1 mwleeds mwleeds 55 Mar  9 14:21
'automount-run-media-mwleeds-29419e8f\x2dc680\x2d4e95\x2d9a31\x2d2cc907d421cb'
-> /run/media/mwleeds/29419e8f-c680-4e95-9a31-2cc907d421cb

And Flatpak then passes libostree a file:// URI containing that path, to
implement sideloading (pulling content from the USB drive).

This results in an error like:

Error: While pulling app/org.videolan.VLC/x86_64/stable from remote
flathub:
/run/flatpak/sideload-repos/automount-run-media-mwleeds-29419e8f%5Cx2dc680%5Cx2d4e95%5Cx2d9a31%5Cx2d2cc907d421cb/.ostree/repo:
opendir(/run/flatpak/sideload-repos/automount-run-media-mwleeds-29419e8f%5Cx2dc680%5Cx2d4e95%5Cx2d9a31%5Cx2d2cc907d421cb/.ostree/repo):
No such file or directory

This patch avoids such errors by using g_file_new_for_uri() instead of
g_file_new_for_path(), so that GLib handles the %-decoding for us.

Bug report by user:
https://community.endlessos.com/t/can-not-install-vlc-from-usb-drive-3-9-3/16353
2021-03-10 10:11:06 -08:00
Phaedrus Leeds
2709da4360 pull: Fix some whitespace and a comment 2021-03-10 10:01:04 -08:00
Philip Withnall
60881b75ec ostree-repo-pull: Fix a leak of the summary data if loading from cache
If the `summary_sig_not_modified` branch is taken above, both
`signatures` and `summary` are loaded from the cache. This makes the
`_ostree_repo_load_cache_summary_if_same_sig()` call below redundant (it
checks `signatures` matches the file it was just loaded from, and then
loads `summary` again) — but that call also currently overwrites
`summary` without clearing the old value.

Fix this by only making that call if `signatures` was retrieved, but the
server said the local `summary` cache was invalid.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2021-03-09 11:47:23 +00:00
OpenShift Merge Robot
ba407750b1
Merge pull request #2287 from jlebon/pr/deployment-parsing-comments
lib/sysroot: Add comments and debug statements around sysroot parsing
2021-03-03 14:36:52 -05:00
Jonathan Lebon
02b6197924 lib/sysroot: Add comments and debug statements around sysroot parsing
Was looking at this code more closely today to investigate issues
related to bootlink mismatches (#2283).
2021-03-02 15:36:27 -05:00
Colin Walters
975496d241 deploy: Add subbootversion to journal
To help debug an issue we've seen where `/boot` isn't
in sync with the `/ostree/boot` dir, let's log to the journal
what we're doing.
2021-02-28 14:47:58 +00:00
Colin Walters
093c63cd46 refs: Make ostree_repo_resolve_rev{,_ext}() use (nullable)
We have an `allow_noent` boolean that controls this, but
were missing the `(nullable)` annotation, so the Rust bindings
panic when the ref doesn't exist instead of being `Option<GString>`.
2021-02-19 01:11:43 +00:00
Colin Walters
5a5f54a459 deltas: Fix leak of matches
Found by ASAN.
2021-02-02 21:26:43 +00:00
OpenShift Merge Robot
c216a438b8
Merge pull request #2267 from dbnicholson/pull-depth-fixes
Pull depth fixes
2021-01-12 17:34:03 -05:00
Dan Nicholson
d7f2955f37 pull: Fix local pull with depth and truncated source history
The local pull path was erroring on any missing commit, but that
prevents a depth pull where the source repo has truncated history. As in
the remote case, this also tries to pull in a tombstone commit if the
source repo supports it.

Fixes: #2266
2021-01-12 14:19:01 -07:00
Dan Nicholson
20047ff1fe pull: Error on depth pull with missing head commit
When pulling with depth, missing parent commits are ignored. However,
the check was applying to any commit, which means that it would succeed
even if the requested commit was missing. This might happen on a
corrupted remote repo or when using ref data from a stale summary.

To achieve this, the semantics of the `commit_to_depth` hash table is
changed slightly to only ever includes parent commits. This makes it
easy to detect when a parent commit is being referenced (although there
is a minor bug there when multiple refs are being pulled) while keeping
references to commits that need their `commitpartial` files cleaned up.
It also means that the table is only populated on depth pulls, which
saves some memory and processing in the common depth=0 case.

Fixes: #2265
2021-01-12 14:19:01 -07:00
OpenShift Merge Robot
bdca64340b
Merge pull request #2263 from cgwalters/createat-nullable
repo: Make ostree_repo_create_at take nullable options
2021-01-12 03:38:46 -05:00
Colin Walters
125c83850a repo: Make ostree_repo_create_at take nullable options
Hit this when trying to use the Rust bindings.
2021-01-12 01:20:23 +00:00
Colin Walters
441233b51c repo: Move fsverity bits to ostree-repo-verity.c
This file will get larger when we start doing more with fsverity.
2021-01-11 14:36:40 +00:00
Colin Walters
9a526bbaa5 sysroot: Handle ro /boot but rw /sysroot
The recent change in https://github.com/coreos/fedora-coreos-config/pull/659
broke some of our tests that do `mount -o remount,rw /sysroot` but
leave `/boot` read-only.

We had code for having `/boot` read-only before `/sysroot` but
in practice we had a file descriptor for `/sysroot` that we opened
before the remount that would happen later on.

Clean things up here so that in the library, we also remount
`/boot` at the same time we remount `/sysroot` if either are readonly.

Delete the legacy code for remounting `/boot` rw if we're not in
a mount namespace.  I am fairly confident most users are either
using the `ostree` CLI, or they're using the mount namespace.
2021-01-10 13:49:44 +00:00
Colin Walters
a1c0cffeb3 sysroot: Also maintain canonical boot_fd
Just like we hold a fd for `/sysroot`, also do so for `/boot`
instead of opening and closing it in a few places.

This is a preparatory cleanup for further work.
2021-01-10 13:46:11 +00:00
Colin Walters
10556a95b4 main: Unconditionally set up mount namespace
I was being very conservative initially here, but I think it's
really safe to just unconditionally set up the mount namespace.

This avoids having to check twice for a read-only `/sysroot`
(once in the binary and once in the library).
2021-01-10 13:40:52 +00:00
Dan Nicholson
92a484d278 pull: Use GNU coding style 2020-12-18 09:13:38 -07:00
Dan Nicholson
4db2ba0eb1 pull: Allow disabling commit binding verification
In some cases such as backups or mirroring you may want to pull commits
from one repo to another even if there commits that have incorrect
bindings. Fixing the commits in the source repository to have correct
bindings may not be feasible, so provide a pull option to disable
verification.

For Endless we have several repositories that predate collection IDs and
ref bindings. Later these repositories gained collection IDs to support
the features they provide and ref bindings as the ostree tooling was
upgraded. These repositories contain released commits that were valid to
the clients they were targeting at the time. Correcting the bindings is
not really an option as it would mean invalidating the repository
history.
2020-12-17 14:07:08 -07:00
OpenShift Merge Robot
874f2ca625
Merge pull request #2242 from stb-tester/tarball-import-xattrs
ostree commit --tree=tar: Import xattrs from tarballs
2020-11-17 22:38:32 +01:00
William Manley
a88d2f5f7b ostree commit --tree=tar: Import xattrs from tarballs
If you specify an `xattr_callback` the xattrs will still be taken from
there for now.
2020-11-17 16:54:32 +00:00
Luca BRUNO
3e289b1934
Release 2020.8 2020-11-17 10:32:57 +00:00
OpenShift Merge Robot
fdd3f7fcdf
Merge pull request #2232 from cgwalters/deploy-docs
deployment: Add a bunch of docs and fix annotations
2020-11-12 19:40:28 +01:00
OpenShift Merge Robot
d046631a0f
Merge pull request #2235 from cgwalters/checkout-some-new-style
bin/checkout: Port some to new style
2020-11-12 09:34:44 +01:00
Colin Walters
8fbf2c5b80 deployment: Ensure query_deployments_for returns nullable values
Since that's a common case; hit this while working on rpm-ostree
code using the ostree-rs bindings.
2020-11-11 22:01:39 +00:00
Colin Walters
43913178a7 deployment: Add a bunch of docs and fix annotations
We were missing docs for these, also add some nullability annotations.
Motivated by using these from the Rust bindings.
2020-11-11 22:01:11 +00:00
Colin Walters
f7be2a3e4a bin/checkout: Port some to new style
I was reading this code for unrelated reasons and noticed it
was still old style; port most (but not all) to new style.
2020-11-11 21:57:36 +00:00
OpenShift Merge Robot
34390a53b8
Merge pull request #2233 from cgwalters/repo-annotations
sysroot: Fix up some GI nullable annotations
2020-11-09 18:56:24 +00:00
Colin Walters
bf8c4c7e32 sysroot: Fix up some GI nullable annotations
Hit `ostree_sysroot_repo()` shouldn't be nullable while using
the ostree Rust bindings.
2020-11-06 20:06:26 +00:00
OpenShift Merge Robot
e43d445b5b
Merge pull request #2228 from jlebon/pr/drop-volatile
Drop use of `volatile`
2020-11-03 17:36:14 -05:00
OpenShift Merge Robot
f62c7bae2e
Merge pull request #2230 from alexlarsson/pull-dont-save-passed-in-summary
pull: Don't save into cache passed in GByte summaries
2020-11-03 12:35:06 -05:00
Alexander Larsson
52463686af pull: Don't save into cache passed in GByte summaries
The cache shouldn't be affected by the user passing in some other
summary as it may not be the "official one".

I ran into this in flatpak where the passed summary was correct, but
the re-saving of the cache updated the mtime of the cached file which
led to later http If-Modified-Since calls failing to update.
2020-11-03 11:48:33 +01:00
Dan Nicholson
3e527d9447 lib/deltas: Annotate from checksum as nullable
Without this you can't create a scratch delta from GI. While here,
switch the deprecated allow-none annotations to nullable.
2020-11-02 16:42:30 -07:00
Jonathan Lebon
f895cf4fd2 Drop use of volatile
As detailed in
gitlab.gnome.org/GNOME/glib/-/issues/600#note_877282, volatile
isn't actually needed in these contexts because the atomic operations
already give us strong enough guarantees. In GCC 11, this triggers a
diagnostic due to the volatile qualifier getting dropped anyway.

There is a WIP to do the same in glib:
https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1719

This obsoletes this downstream patch:
https://src.fedoraproject.org/rpms/ostree/c/b8c5a6fb
2020-11-02 14:53:26 -05:00
OpenShift Merge Robot
64e09f46b8
Merge pull request #2226 from alexlarsson/fix-gir-arg
ostree_repo_gpg_sign_data: Fix API doc argument name
2020-10-28 13:31:50 -04:00
Alexander Larsson
2f78441bea ostree_repo_gpg_sign_data: Fix API doc argument name
I got:
src/libostree/ostree-repo.c:5232: Warning: OSTree: ostree_repo_gpg_sign_data: unknown parameter 'out_signature' in documentation comment, should be 'out_signatures'
2020-10-28 15:53:18 +01:00
OpenShift Merge Robot
7bc53f0063
Merge pull request #2224 from stb-tester/bootloader-refactorings
Bootloader probing and construction refactoring
2020-10-28 06:39:57 -04:00
Jonathan Lebon
8717608c7e lib/fetch-curl: Unref timeout source
The timeout timer should always be one-shot, so let's just always
destroy it in the callback. The main context has its own ref on it, so
it won't be freed behind its back.

This *should* fix a leak that was brought up in
https://bugzilla.redhat.com/show_bug.cgi?id=1891761.

Reported-by: Milan Crha <mcrha@redhat.com>
2020-10-27 12:01:01 -04:00
Jonathan Lebon
40edc33ef3 lib/fetcher-curl: Use G_SOURCE_REMOVE instead of FALSE
They're equivalent, though I prefer the former because it's more
descriptive and it makes it really obvious that it's a `GSource`
callback.
2020-10-27 11:57:00 -04:00
William Manley
663c5b41a3 fixup! Refactor ostree_sysroot_query_bootloader 2020-10-27 13:24:46 +00:00
William Manley
631528c87b fixup! Refactor: Centralise choosing the appropriate bootloader 2020-10-27 12:35:29 +00:00
William Manley
a8dce46b5f Refactor ostree_sysroot_query_bootloader
This is more regular, so will make it easier to add more bootloader types
in the future.
2020-10-26 23:51:11 +00:00
William Manley
31acd2ef99 Add support for explicitly requesting any specific bootloader type
...with the `sysroot.bootloader` configuration option.  This can be useful
when converting a system to use `ostree` which doesn't currently have a
bootloader configuration that `ostree` can automatically detect, and is
also useful in combination with the `--sysroot` option when provisioning a
rootfs for systems other than the one you're running `ostree admin deploy`
on.
2020-10-26 23:51:11 +00:00
William Manley
9482ecfe5a Refactor: sysroot.bootloader: Store enum value rather than string
It's easier to extend and it centralises the config parsing.  In other
places we will no longer need to use `g_str_equal` to match these values,
a `switch` statement will be sufficient.
2020-10-26 23:51:11 +00:00
William Manley
062df6ee81 Refactor: Centralise choosing the appropriate bootloader
In preparation for enhancing `_ostree_sysroot_query_bootloader`
2020-10-26 23:51:11 +00:00
William Manley
5e223f2962 ostree_repo_get_bootloader: Document transfer none
I think this may affect bindings too.
2020-10-26 23:51:11 +00:00
Alexander Larsson
654f3d959a ostree pull: Add more g_debug spew around fetching deltas
This is useful to debug what is happening when downloading via deltas.
2020-10-23 13:55:33 +02:00
Alexander Larsson
8cd796f3f1 Add ostree_repo_gpg_sign_data()
This is similar to ostree_sign_data() but for the old gpg code.
Flatpak will need this to reproduce a signed summary.
2020-10-23 13:55:33 +02:00
Alexander Larsson
125ed2b199 pull: Only download summary if we need it for the pull operation
If we have a commit id for all the refs we're pulling, and if we
don't need the summary to list all the refs when mirroring then the
only reason to download the summary is for the list of deltas.

With the new "indexed-deltas" property in the config file (and mirrored
to the summary file) we can detect when we don't need the summary for
deltas and completely avoid downloading it then.
2020-10-23 13:55:33 +02:00
Alexander Larsson
6c8e6539e2 deltas: Set indexed-deltas key in the config and summary
Clients can use these during pull and avoid downloading the summary if
needed, or use the indexed-deltas instead of relying on the ones in
the summary which may be left out.
2020-10-23 13:06:46 +02:00
Alexander Larsson
0984ff8471 deltas: Take a shared repo lock while reindexing deltas
This ensures we're not racing with a prune operation that can be removing
the delta indexes we're relying on.
2020-10-23 13:06:46 +02:00
Alexander Larsson
df7f07fc6c deltas: Use delta indexes when pulling
If there is no delta index in the summary, try to fetch the
delta index for the commit we're going to and use that to find the
delta (if any).
2020-10-23 13:06:42 +02:00
Alexander Larsson
625606a7ec deltas: Add CLI ops to list and reindex delta-indexes 2020-10-23 12:30:08 +02:00
Alexander Larsson
c304703e1d deltas: Make ostree_repo_static_delta_reindex() public
It is useful to be able to trigger this without having to regenerate
the summary. For example, if you are not using summaries, or ar generating
the summaries yourself.
2020-10-23 12:30:08 +02:00
Alexander Larsson
024ef1d756 deltas: Add and document no-deltas-in-summary config option
By default this is FALSE to keep existing clients working.
2020-10-23 12:30:08 +02:00
Alexander Larsson
effde3d513 deltas: Update delta indexes when updating summary
When we update the summary file (and its list of deltas) we also update
all delta indexes. The index format is a single `a{sv}` variant identical
to the metadata-part of the summary with (currently) only the
`ostree.static-deltas` key.

Since we expect most delta indexes to change rarely, we avoid
unnecessary writes when reindexing. New indexes are compared to
existing ones and only the changed ones are written to disk.  This
avoids unnecessary write load and mtime changes on the repo server.
2020-10-23 12:30:08 +02:00
Alexander Larsson
8e1f199dd4 deltas: Add ostree_repo_list_static_delta_indexes() function
This lists all the available delta indexes.
2020-10-23 12:30:08 +02:00
Alexander Larsson
87e564eeaa deltas: Add _ostree_get_relative_static_delta_index_path()
This gets the subpath for a delta index file, which is of the form
"delta-indexes/$commit.index", that contains all the deltas going
to the particular commit.
2020-10-23 12:30:08 +02:00
OpenShift Merge Robot
f8f6464580
Merge pull request #2205 from pwithnall/etags-and-last-modified
Add support for ETag and Last-Modified headers for summary and summary.sig
2020-10-22 18:20:23 -04:00
Philip Withnall
0974a7faf1 tests: Split RFC 2616 date parsing code out and add tests
This makes it testable, and increases its test coverage too 100% of
lines, as measured by `make coverage`.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
92215025aa ostree/trivial-httpd: Add Last-Modified/ETag support
This is basic support for the
Last-Modified/ETag/If-Modified-Since/If-None-Match headers. It’s not
high performance, and doesn’t support all of the related caching
features (like the If-Match header, etc.).

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
0e3add8d40 lib/pull: Hook up HTTP caching headers for summary and summary.sig
As `summary` and `summary.sig` aren’t immutable, HTTP requests to
download them can be optimised by sending the `If-None-Match` and
`If-Modified-Since` headers to avoid unnecessarily re-downloading them
if they haven’t changed since last being checked.

Hook them up to the new support for that in the fetcher.

The `ETag` and `Last-Modified` for each file in the cache are stored as
the `user.etag` xattr and the mtime, respectively. For flatpak, for
example, this affects the cached files in
`~/.local/share/flatpak/repo/tmp/cache/summaries`.

If xattrs aren’t supported, or if the server doesn’t support the caching
headers, the pull behaviour is unchanged from before.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
407c683524 libostree: Add support for ETag and Last-Modified headers
Add support in the soup and curl fetchers to send the `If-None-Match`
and `If-Modified-Since` request headers, and pass on the `ETag` and
`Last-Modified` response headers.

This currently introduces no functional changes, but once call sites
provide the appropriate integration, this will allow HTTP caching to
happen with requests (typically with metadata requests, where the data
is not immutable due to being content-addressed). That should reduce
bandwidth requirements.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
OpenShift Merge Robot
95a6d15145
Merge pull request #2222 from fkrull/patch-v2020.7
lib: add missing GI annotation
2020-10-20 16:48:37 -04:00
Felix Krull
4183b68e70 lib: fix GI parameter tags 2020-10-17 22:14:09 +02:00
Jonathan Lebon
dec9eab203 ostree-prepare-root: print st_dev and st_ino as 64-bit ints
This matches what systemd does and should work fine on all platforms.

Possibly resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1888436
2020-10-15 12:26:47 -04:00
OpenShift Merge Robot
26b98ebc56
Merge pull request #2215 from cgwalters/fd-transfer
deploy: Remove (transfer none) from fd arg
2020-10-14 00:45:13 +02:00
Colin Walters
b4e7ab014e deploy: Remove (transfer none) from fd arg
GI complains.  And in general one needs to assume that file
descriptors aren't stolen.
2020-10-13 17:40:11 -04:00
Colin Walters
326b8b13c6 Post-release version bump 2020-10-13 17:38:19 -04:00
Colin Walters
32a3a12973 Release 2020.7 2020-10-13 14:31:26 -04:00
OpenShift Merge Robot
a13509ee7d
Merge pull request #2213 from alexlarsson/summary-bugfixes
Various fixes from the indexed-summaries branch
2020-10-08 15:18:34 -04:00
Alexander Larsson
2e9db809b9 signatures: Fix leak in _sign_detached_metadata_append()
This needs to ref_sink the returned variant, as it is used with g_autoptr
in the callers.
2020-10-08 14:10:04 +02:00
Alexander Larsson
f821cdb89e fetch_summary_with_options: Fix n-network-retries option parsing
"&u" is not a valid gvariant format string, it should just be "u".
2020-10-08 14:09:16 +02:00
Alexander Larsson
6ed1066ef0 ostree_repo_find_remotes_async: Fix leak of summary
We were creating a GVariant from a GBytes and storing it in an
g_autoptr without ref_sinking it.
2020-10-08 14:08:51 +02:00
Jonathan Lebon
b3dc074f5e lib/deploy: Don't leak fd when checksumming dtbs
Likely the root of https://bugzilla.redhat.com/show_bug.cgi?id=1886149.
2020-10-07 14:47:34 -04:00
OpenShift Merge Robot
9f98aa9223
Merge pull request #2198 from cgwalters/no-hardlink-zerosize
checkout: Don't hardlink zero sized files
2020-10-05 22:06:38 +02:00
OpenShift Merge Robot
06a77bfd69
Merge pull request #2155 from jlebon/pr/add-initrds
lib/deploy: Add support for overlay initrds
2020-10-02 16:41:01 -04:00
Colin Walters
558720e7aa checkout: Don't hardlink zero sized files
Alternative to https://github.com/ostreedev/ostree/pull/2197

Python's (usually) zero-sized `__init__.py` files can provoke
us hitting the hardlink limits on some filesystems (`EMLINK`).
At least one Fedora rpm-ostree user hit this.

The benefits of hardlinking here are quite marginal; lots
of hardlinks can behave suboptimally in particular filesystems
like BTRFS too.

This builds on prior code which made this an option, introduced
in 673cacd633
Now we just do it uncondtionally.

Also this provoked a different bug in a very obscure user mode checkout
case; when the "real" permissions were different from the "physical"
permissions, we would still hardlink.  Fix the test case for this.
2020-10-01 16:47:07 -04:00
Philip Withnall
23bdc4e5df ostree/dump: Fix a memory leak
Re-using the `refs` variable for the main list of refs, plus the
iterated lists, meant that the main list was never freed (although all
the iterated ones were freed correctly).

Fix this by using two variables rather than reusing the one.

Signed-off-by: Philip Withnall <withnall@endlessm.com>
2020-10-01 11:06:56 +01:00
Philip Withnall
8d09a1a8ea lib/pull: Read mode and tombstone options from summary file if possible
Otherwise, fall back to downloading and reading them from the `config`
file. See the previous commit for details.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Fixes: #2165
2020-10-01 11:06:56 +01:00
Philip Withnall
206f1d3a13 lib/repo: Add mode and tombstone config options to the summary file
Currently, they are set in the `config` file and cause that to be
downloaded on every pull. Given that the client is already pulling the
`summary` file, it makes sense to avoid an additional network round trip
and cache those options in the `summary` file.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Helps: #2165
2020-10-01 11:06:56 +01:00
Jonathan Lebon
81b13da8e3 lib/deploy: Add support for overlay initrds
In FCOS and RHCOS, the need to configure software in the initramfs has
come up multiple times. Sometimes, using kernel arguments suffices.
Other times, it really must be a configuration file. Rebuilding the
initramfs on the client-side however is a costly operation. Not only
does it add complexity to the update workflow, it also erodes a lot of
the value obtained from using the baked "blessed" initramfs from the
tree itself.

One elegant way to address this is to allow specifying multiple
initramfses. This is supported by most bootloaders (notably GRUB) and
results in each initrd being overlayed on top of each other.

This patch allows libostree clients to leverage this so that they can
avoid regenerating the initramfs entirely. libostree itself is agnostic
as to what kind and how much data overlay initrds contain. It's up to
the clients to enforce such boundaries.

To implement this, we add a new ostree_sysroot_stage_overlay_initrd
which takes a file descriptor and returns a checksum. Then users can
pass these checksums when calling the deploy APIs via the new array
option `overlay_initrds`. We copy these files into `/boot` and add them
to the BLS as another `initrd` entry.
2020-09-30 13:29:32 -04:00
Jonathan Lebon
40fea4c443 lib/deploy: Add deploy/stage APIs with options
And make the `override_kernel_argv` one of those options. This is mostly
a mechanical move here, no functional change otherwise.

Prep for adding a new option.
2020-09-30 13:29:32 -04:00
Jonathan Lebon
f7500bb703 lib/bootconfig: Add support for multiple initrd keys
Prep for actually teaching the rest of the codebase about this.

We keep the primary initrd in the `options` hash table for backwards
compatibility.
2020-09-30 13:29:32 -04:00
Jonathan Lebon
f04e5d047d lib: Minor versioning related fixes
Fix/add the `Since` marker to the new static delta APIs, and update the
symbol versioning templates/comments.
2020-09-25 15:30:20 -04:00
OpenShift Merge Robot
fad95167ce
Merge pull request #2203 from cgwalters/delta-sign-cleanups
delta: Some minor code style fixups
2020-09-25 13:57:33 +00:00
OpenShift Merge Robot
e6f673a8ed
Merge pull request #2202 from cgwalters/bootcsum-dtb
deploy: Remove deployment bootcsum assertion
2020-09-24 22:14:31 +00:00
Colin Walters
9198fa040c delta: Some minor code style fixups
- Remove some unused variables
- Switch to declare-and-initialize with others
- Fix some indentation from 4 spaces to 2 (GNU style)
2020-09-24 22:12:56 +00:00
Colin Walters
aa2a2783ea deploy: Remove deployment bootcsum assertion
When support for devicetree was added, it created a problem
because old and new ostree versions would compute different
checksums for the "boot data".  The scenario here is:

- Have system with ostree < 2020.4
- Reboot into system with ostree 2020.5
- Try to perform an operation that would retain
  that previous booted deployment (common)

Currently ostree iterates over all the deployments
that will be retained and calls `install_deployment_kernel()`,
even for the booted one (which is a bit silly), but
just to verify that all boot data for the targeted
deployments are installed.

This then re-computes the checksum and we'd trip this
assertion.

In practice though, we don't strictly require them to match;
the only thing that will happen if they don't is that we'll
end up with another copy of the kernel/initramfs - and
that only temporarily until the previous deployment
gets GC'd.

Longer term, I think what we really want to do anyways
is probably closer to like a little ostree repo for `/boot`
so that we can e.g. still hardlink kernels there even if
the initramfs changes, or hardlink both kernel/initramfs
if just the devicetree changes, etc.

Closes: https://github.com/ostreedev/ostree/issues/2154
2020-09-24 21:24:48 +00:00
OpenShift Merge Robot
00df896550
Merge pull request #1985 from fdanis-oss/wip/fda/sign_delta_metadata
Static-delta's superblock signature support
2020-09-24 14:21:16 +00:00
Colin Walters
3441a48c58 checkout: Ensure copies of unreadable usermode checkouts are readable
The extreme special case of "zero mode" files like `/etc/shadow`
comes up again.  What we want is for "user mode" checkouts to
override it to make the file readable; otherwise when operating
as non-root without `CAP_DAC_OVERRIDE` it becomes very difficult
to work with.

Previously, we were hardlinking these files, but then it intersects
with *another* special case around zero sized files, which is
*also* true for `/etc/shadow`.

Trying to avoid hardlinking there unveiled this bug - when
we go to do a copy checkout, we need to override the mode.
2020-09-16 15:14:54 +00:00
Frédéric Danis
2e97f5659f bin/static-delta: Add signature parameters to apply-offline
This allows to check the delta signature before applying it.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Frédéric Danis
fb1faf17d6 lib/deltas: Check signed delta in execute_offline
Add a new function `ostree_repo_static_delta_execute_offline_with_signature`
which takes a signature engine to verify the delta before applying it.
The `ostree_repo_static_delta_execute_offline` is just a wrapper to this
new function, passing a NULL signature engine.
When this function is called without signature engine, but with a sign
delta, it will only fails if `sign-verify-deltas` is set to true in repo
core options.

This commits move signature existence check and delta signature
verification to share common parts between existing APIs and the new
function.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Frédéric Danis
bf0c09ffe1 lib/deltas: Support signed delta in dump
This checks if the static delta file is signed or not to be able to
correctly get the superblock to dump.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Frédéric Danis
0c48423c26 lib/deltas: Support signed delta in execute_offline
This checks if the static delta file is signed or not to be able to
correctly get the superblock to apply.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Frédéric Danis
512db0435c bin/static-delta: Add command to verify delta signature
Add new "static-delta verify" sub-command.
This supports multiple keys to verify the static-delta file.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Frédéric Danis
02a19b2c96 lib/deltas: Add signature check API for static-delta superblock
This retrieves the signatures and pass the static delta block as an array
of bytes to ostree_sign_data_verify().

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Frédéric Danis
92efbc00d8 bin/static-delta: Add support to sign superblock
Add signing ability to "static-delta generate" builtin.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Frédéric Danis
46667567c5 lib/deltas: Add inline signature for static-delta superblock
While the commits contained in the single static-delta file are signed so
we can check them and operate on trusted data, the superblock isn't signed
in any way, so it end up operating on untrusted data to:
 1. actually find where the trusted data is, and
 2. check whether the update is fit for the current device by looking at
    the collection id stored in the metadata

This commit generates signatures of all static data, and concatenate them
to the existing static delta format, i.e. as a GVariant layout `a{sv}ay`
where
 - a{sv}: signatures
 - ay: existing delta variant

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Phaedrus Leeds
c4f26bfdc8 Avoid shadowing local variables
This should help with code readability.

Fixes https://github.com/ostreedev/ostree/issues/2194
2020-09-13 10:08:02 -07:00
Colin Walters
779a901780 commit: Tighten scope of two variables
Prep for adding `-Wshadow` fixes.
2020-09-13 14:58:24 +00:00
Alexander Larsson
b7d1a9746b Update the symbols files to match that we're now on 2020.6 2020-09-11 12:35:32 +02:00
Alexander Larsson
da853a1783 ostree-repo-pull.c: Extract mirrorlist generation to helper
This code was duplicated in 3 places, so move it to a single place
to clean things up.
2020-09-11 12:03:29 +02:00
Alexander Larsson
155b215cd8 Minor cleanup of _ostree_repo_remote_new_fetcher()
Instead of open coding the extra_headers and append_user_agent
setting everywhere we do this in the constructor.
2020-09-11 12:03:28 +02:00
Alexander Larsson
3957bff0cb Inline repo_remote_fetch_summary
This was only used in one place, and (especially with the simplification
with GMainContextPopDefault) and the one caller doesn't really do
much more than call the helper. Additionally, what little it does (saving
the result in the cache) is inherently tied to how the helper work,
and will become even more so when we support summary indexes.

This is a preparatory cleanup for supporting summary indexes. It
doesn't change any behaviour and passes make check on its own.
2020-09-11 12:03:28 +02:00
Alexander Larsson
4b9e712e82 repo_remote_fetch_summary: Use GMainContextPopDefault
This allows us to drop the "goto out" use and clean up this function.
2020-09-11 12:03:28 +02:00
Alexander Larsson
c7df4317bd Add g_autoptr helper for pushing a thread default main context
This happens in a bunch of places, and currently each time it does
we have to use "goto out" style cleanups, which just isn't looking
very nice.
2020-09-11 12:03:28 +02:00
Alexander Larsson
f74bc8dd3d fetch_summary_with_options: drop unnecessary "goto out" use 2020-09-11 12:03:28 +02:00
Alexander Larsson
1f1ef4c989 Break out the signature verification code into a helper function
This changes nothing in the behaviour, but we want to later re-use
this when we also verify the summary index.
2020-09-11 12:03:28 +02:00
Alexander Larsson
598adc457f deltas: Break out _ostree_repo_static_delta_superblock_digest() helper
This loads and makes a digest for a delta superblock. The previous
code was used when generating the deltas section in the summary
file. This changes nothing, but is in preparation for using similar
formats in a separate delta index file.
2020-09-11 12:03:28 +02:00
Alexander Larsson
32014d99e6 Add and use ot_checksum_bytes helper
This removes some duplicated code (and will be use even more later).
2020-09-11 12:00:05 +02:00
Alexander Larsson
dddb449d2c pull: Actually mmap summary files
The change in cbf1aca1d5 actually
only mmaps the signature file, not the summary. This change makes
use mmap both, as well as extract the cache loading into a helper
function that we will later use in more places.
2020-09-11 12:00:02 +02:00
Alexander Larsson
85accb84e8 pull: Break out _ostree_repo_save_cache_summary_file() helper
This is a minor cleanup as its just called twice from
_ostree_repo_cache_summary(). However, later code will need it in more
places.
2020-09-11 11:53:38 +02:00
Alexander Larsson
bb2649a8c0 Fix leak when signing
_ostree_detached_metadata_append_gpg_sig() was returning a floating
ref, but all users were using g_autoptr. Fix it by adding a ref-sink.
2020-09-11 11:45:33 +02:00
Alexander Larsson
74bae256fe list-deltas: Don't break on non-subdir entries
ostree_repo_list_static_delta_names() tried to validate that
any second-level directory element was a directory, but there was
a cut-and-paste issue, and it used `dent->d_type` instead
of `sub_dent->d_type`.

This fixes the code, but all old ostree versions will break if
there are non-directories in a subdirectory of the deltas directory
in the repo, so be wary.
2020-09-11 11:45:24 +02:00
Colin Walters
a1bd29f245 deploy: Add some error prefixing around xattr setting
Looking at
https://github.com/coreos/coreos-assembler/issues/1703
a user is getting a bare:
`error: fsetxattr: Permission denied`

I don't think it's these code paths since a deploy
isn't happening but on inspection I noticed we didn't
have error prefixing here.
2020-09-09 12:34:44 +00:00
Jonathan Lebon
a7a751b69f ostree-remount: Remount /etc rw if needed
When we remount `/sysroot` as read-only, we also make `/etc` read-only.
This is usually OK because we then remount `/var` read-write, which also
flips `/etc` back to read-write... unless `/var` is a separate
filesystem and not a bind-mount to the stateroot `/var`.

Fix this by just remounting `/etc` read-write in the read-only sysroot
case.

Eventually, I think we should rework this to set everything up the way
we want from the initramfs (#2115). This would also eliminate the window
during which `/etc` is read-only while `ostree-remount` runs.
2020-08-28 14:16:46 -04:00
Jonathan Lebon
b3c7b059ea ostree-prepare-root: Fix /etc bind mount
We were bind-mounting the initramfs' `/etc` (to itself) instead of the
target deployment `/etc` (to itself). Since we're already `chdir`'ed
into it, we can just drop the leading slash.
2020-08-28 14:16:29 -04:00
Felix Krull
f4d0b17080 lib: mark out parameters as out parameters 2020-08-26 22:32:47 +02:00
Felix Krull
d5b8929017 lib: add some missing version tags 2020-08-26 22:32:47 +02:00
Matt Bilker
dac2ad288f Fix mkinitcpio with newer systemd versions
- Fixes systemd failing to determine if `/sysroot` is valid because of
  `/etc/os-release` not being available yet.

- Related: #1759
2020-08-25 18:12:55 -04:00
OpenShift Merge Robot
2596a718ce
Merge pull request #2179 from cgwalters/ioctl-fix
linuxfsutil: Pass int to ioctl, not long
2020-08-21 20:13:51 -04:00
Colin Walters
06ed04a816 linuxfsutil: Pass int to ioctl, not long
Otherwise it will fail on big-endian architectures like s390x.
Ref https://bugzilla.redhat.com/show_bug.cgi?id=1867601
2020-08-21 17:41:32 +00:00
Simon McVittie
d3fadf14b7 boot: Replace deprecated StandardOutput=syslog with journal, etc.
systemd deprecated this in v246.

Resolves: #2169
Signed-off-by: Simon McVittie <smcv@collabora.com>
2020-08-21 09:58:05 +01:00
OpenShift Merge Robot
c61ff03304
Merge pull request #2175 from cgwalters/coverity-2020.5
Two small Coverity fixes
2020-08-19 16:46:49 +02:00
OpenShift Merge Robot
56f00586dd
Merge pull request #2176 from cgwalters/pin-str
admin/pin: Enforce that index is a number
2020-08-19 15:45:40 +02:00
Colin Walters
22a445c189 admin/pin: Enforce that index is a number
Validate that we're parsing a number; we want to guard
against typos.

Closes: https://github.com/ostreedev/ostree/issues/2171
2020-08-19 13:11:55 +00:00
Colin Walters
95a7512622 prepare-root: Remove unused variable
Should quiet Coverity.
2020-08-18 23:35:38 +00:00
Colin Walters
1eab48363b pull: Assign idle_src variable before calling unref()
This should pacify Coverity, and also just "reads" better too.
2020-08-18 23:34:57 +00:00
OpenShift Merge Robot
199562fc14
Merge pull request #2149 from stb-tester/boot-self-symlink
sysroot: Support /boot on root or as seperate filesystem for syslinux and u-boot
2020-08-19 01:08:30 +02:00
Jonathan Lebon
10a68cd26b lib/deploy: Clarify comment re. staging API
Don't mention deprecation in the description for
`ostree_sysroot_deploy_tree` since there are legitimate use cases for it
(e.g. to create the first deployment via `ostree admin deploy`).

Instead, make the comment clearly redirect to the staging API when
booted into the sysroot.
2020-08-17 09:48:57 -04:00
Jonathan Lebon
5de3a9759f lib/deploy: Drop unneccessary function arg 2020-08-17 09:48:57 -04:00
Jonathan Lebon
e4fb7d3bb1 lib/cleanup: Drop unnecessary GEqualFunc cast 2020-08-17 09:48:57 -04:00
Jonathan Lebon
74bd136286 lib/deploy: Simplify deployment creation
Minor cleanup; we were declaring a superfluous variable.
2020-08-17 09:48:57 -04:00
Jonathan Lebon
61c544df1b lib/deploy: Avoid shadowing variable
There's already a `boot_relpath` variable in the outside scope.
2020-08-17 09:48:57 -04:00
Jonathan Lebon
52a6224606 lib/deploy: Clean up kargs override handling
Tighten up how we handle kargs here so it's more clear. When we call
`sysroot_finalize_deployment`, any karg overrides have already been set
on the bootconfig object of the deployment. So re-setting it here is
redundant and confusing.
2020-08-17 09:48:57 -04:00
João Paulo Rechi Vita
7cf1fb38b0 dracut: Create reproducible images
Without reproducible images, a rebuild of the initrd will create a
different image file (due to things like creation time of the files in
the cpio archive) even if the actual contents in it are exactly the
same, adding an unnecessary download during updates.

Adding 'reproducible=yes' avoids this and creates the same image files
for the same content.
2020-08-13 08:32:18 -07:00
Colin Walters
f2773c1b55 Add "transient" unlock
I was thinking a bit more recently about the "live" changes
stuff https://github.com/coreos/rpm-ostree/issues/639
(particularly since https://github.com/coreos/rpm-ostree/pull/2060 )
and I realized reading the last debates in that issue that
there's really a much simpler solution; do exactly the same
thing we do for `ostree admin unlock`, except mount it read-only
by default.

Then, anything that wants to modify it does the same thing
libostree does for `/sysroot` and `/boot` as of recently; create
a new mount namespace and do the modifications there.

The advantages of this are numerous.  First, we already have
all of the code, it's basically just plumbing through a new
entry in the state enumeration and passing `MS_RDONLY` into
the `mount()` system call.

"live" changes here also naturally don't persist, unlike what
we are currently doing in rpm-ostree.
2020-08-07 18:57:56 +00:00
Philip Withnall
f5da67d78a pull: Add summary-{,sig-}bytes options to ostree_repo_pull()
These allow the `summary` and `summary.sig` files to be cached at a
higher layer (for example, flatpak) between related pull operations (for
example, within a single flatpak transaction). This avoids
re-downloading `summary.sig` multiple times throughout a transaction,
which increases the transaction’s latency and introduces the possibility
for inconsistency between parts of the transaction if the server changes
its `summary` file part-way through.

In particular, this should speed up flatpak transactions on machines
with high latency network connections, where network round trips have a
high impact on the latency of an overall operation.

Signed-off-by: Philip Withnall <withnall@endlessm.com>
2020-08-07 16:12:15 +01:00
Philip Withnall
bd68c7dfd7 pull: Improve formatting of pull options in documentation
Backticks improve all things.

Signed-off-by: Philip Withnall <withnall@endlessm.com>
2020-08-07 16:11:44 +01:00
OpenShift Merge Robot
1dedc69a46
Merge pull request #2159 from agners/show-parent
Show commit checksum of parent, if present
2020-08-04 21:54:28 +02:00
OpenShift Merge Robot
9832a36a8c
Merge pull request #2160 from cgwalters/sysroot-ro
remount: Still remount /sysroot writable if not configured ro
2020-08-02 20:53:41 -04:00
Colin Walters
33eeb7b9eb remount: Still remount /sysroot writable if not configured ro
Regression from 3564225917
BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1862568

We still need to remount writable if it's not configured on;
because it may need OS adjustments it needs to be opt-in.
2020-08-01 17:27:18 +00:00
Stefan Agner
512b4e6313 Show commit checksum of parent, if present
This is useful for ostree log on client side where often not the
full history of a branch is available. It is also helpful for
ostree show to show if a particular commit has a parent.
2020-07-31 21:46:58 +02:00
Jonathan Lebon
af140266d5 app: Fix various CLI metavariable names
- Use `REV` instead of `REF` in places where we meant it.
- Fix `commit --parent` actually taking a commit checksum and not a ref.
- Fix `ostree admin switch` using `REF` instead of `REFSPEC`.
2020-07-31 15:02:47 -04:00
Colin Walters
0d91206a62 Post-release version bump 2020-07-22 14:09:26 +00:00
Colin Walters
901747f985 Release 2020.4
A lot of stuff here, new signing API is the biggest.  Let's
get a release out.
2020-07-21 21:48:51 +00:00
OpenShift Merge Robot
12a9161a79
Merge pull request #2152 from cgwalters/pull-fdatasync
pull: Add --per-object-fsync
2020-07-20 13:55:26 -04:00
Colin Walters
a615d35762 pull: Add --per-object-fsync
This is the opposite of
https://github.com/ostreedev/ostree/issues/1184

Motivated by OpenShift seeing etcd performance issues during
OS updates: https://github.com/openshift/machine-config-operator/issues/1897

Basically, if we switch to invoking `fsync()` as we go, it makes
ostree performance worse (in my tests, 31s to write 2G versus 7s if we
delay sync) but it avoids *huge* outliers in `fsync()` time for etcd.
2020-07-18 14:59:01 +00:00
Colin Walters
5b75358357 pull: Also append bytes written
This is very useful information that we get from the transaction
stats.  Append it to the final display if we're not inheriting
the transaction.
2020-07-17 17:03:08 +00:00
William Manley
0ced9fde76 sysroot: Support /boot on root or as seperate filesystem for syslinux and u-boot
We use a similar trick to having a `sysroot -> .` symlink on the real root
here to support both /boot on root as well as on a separate filesystem.  No
matter how it's mounted `/boot/xyz` will always refer to the file you'd
expect.

This is nicer than my previous attempts at this because there's no
configuration nor auto-detection required.
2020-07-15 20:37:49 +01:00
Frédéric Danis
892e9acf25 lib/deltas: convert ostree_repo_static_delta_generate to new style
The "new style" code generally avoids `goto err` because it conflicts
with `__attribute__((cleanup))`.  This fixes a compiler warning.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-07-03 13:35:39 +00:00
Jonathan Lebon
1d755f62af lib/repo: Handle EACCES for POSIX locking
If `glnx_make_lock_file` falls back to `flock`, on NFS this uses POSIX
locks (`F_SETLK`). As such, we need to be able to handle `EACCES` as
well as `EAGAIN` (see `fnctl(2)`).

I think this is what coreos-ostree-importer has been hitting, which runs
on RHEL7 in the Fedora infra and does locking over an NFS share where
multiple apps could concurrently pull things into the repo.
2020-06-30 15:09:41 -04:00
OpenShift Merge Robot
fd8ecdf047
Merge pull request #2131 from cgwalters/sign-success
signapi: Change API to also return a success message
2020-06-24 18:24:41 -04:00
OpenShift Merge Robot
adcd261596
Merge pull request #2001 from agners/multi-device-tree
deploy: support devicetree directory
2020-06-24 03:02:18 -04:00
OpenShift Merge Robot
9067c1ca49
Merge pull request #2136 from cgwalters/doc-timestamp
core: Add documentation for ostree_commit_get_timestamp()
2020-06-19 13:57:33 -04:00
Colin Walters
935f879542 sysroot: Remove unimplemented ostree_sysroot_lock_with_mount_namespace
This came in with 5af403be0c but
was never implemented.

I noticed this now because the Rust ostree bindings generate a
wrapper for it which the linker tries to use.
2020-06-18 21:19:43 +00:00
Colin Walters
c87a4c71bc core: Add documentation for ostree_commit_get_timestamp()
Working on some rpm-ostree bits and was going to pass
this to the `chrono` crate and I forgot the format, went
to look at the docs.  Oops.
2020-06-18 20:57:53 +00:00
OpenShift Merge Robot
33c95812bb
Merge pull request #2132 from cgwalters/remove-unused-symbol
libostree-devel.sym: Remove nonexistent stub symbol
2020-06-17 16:18:26 -04:00
Colin Walters
9336837194 libostree-devel.sym: Remove nonexistent stub symbol
This should have been removed when we added symbols to this list.
2020-06-17 19:38:26 +00:00
Denis Pynkin
ede93dc2ef sign/ed25519: fix return value if no correct keys in file
Fix the return value if file doesn't contains correct public key(s).

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-06-17 19:45:05 +03:00
Denis Pynkin
ce73876389 sign/ed25519: fix the abort in case of incorrect public key
We need to check the size of public key before trying to use it.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-06-17 19:44:56 +03:00
Colin Walters
36258036ae signapi: Change API to also return a success message
This is the dual of 1f3c8c5b3d
where we output more detail when signapi fails to validate.

Extend the API to return a string for success, which we output
to stdout.

This will help the test suite *and* end users validate that the expected
thing is happening.

In order to make this cleaner, split the "verified commit" set
in the pull code into GPG and signapi verified sets, and have
the signapi verified set contain the verification string.

We're not doing anything with the verification string in the
pull code *yet* but I plan to add something like
`ostree pull --verbose` which would finally print this.
2020-06-17 00:33:47 +00:00
Colin Walters
1f3c8c5b3d sign/ed25519: Output failed signatures in error message
To aid debuggability, when we find a commit that isn't signed
by our expected key, output a specific error message with the
key.

(And then add code to switch to just printing the count beyond 3
 because the test suite injects 100 keys and hopefully no one
 ever actually does that)
2020-06-16 18:20:54 +03:00
OpenShift Merge Robot
bd9b4ea731
Merge pull request #2121 from cgwalters/arch-key
core: Add OSTREE_COMMIT_META_KEY_ARCH
2020-06-09 02:58:59 +02:00
Stefan Agner
832ca09891 checkout: use FILE as option argument string for --skip-list
Align with --from-file and use 'FILE' instead of 'PATH' as option
argument string. No functional change, this is only cosmetics.

Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
2020-06-08 15:43:32 +02:00
Colin Walters
29dddf38d3 core: Add OSTREE_COMMIT_META_KEY_ARCHITECTURE
Add a standard key for this.  We actually had a case in OpenShift
builds recently where a `ppc64le` image was pushed over an `x86_64`
one and this started failing at runtime with a not immediately
obvious error.

I'll probably end up changing rpm-ostree at least to use
the RPM architecture for this key and fail if it doesn't match
the booted value.

Possibly that should live in ostree but it would involve adding
architecture schema here, which gets into a big mess.  Let's
just standardize the key.

xref e02ef2683d
2020-06-05 17:52:42 +00:00
Colin Walters
b137895022 commit: Note in help that --base takes an argument
I was trying to use this in some testing work and was confused for a minute.
2020-06-04 21:06:52 +00:00
Stefan Agner
5f08649f51 deploy: support devicetree directory
Add support for a devicetree directory at /usr/lib/modules/$kver/dtb/.
In ARM world a general purpose distribution often suppports multiple
boards with a single operating system. However, OSTree currently only
supports a single device tree, which does not allow to use the same
OSTree on different ARM machines. In this scenario typically the boot
loader selects the effective device tree.

This adds device tree directory support for the new boot artefact
location under /usr/lib/modules. If the file `devicetree` does not
exist, then the folder dtb will be checked. All devicetrees are hashed
into the deployment hash. This makes sure that even a single devicetree
change leads to a new deployment and hence can be rolled back.

The loader configuration has a new key "devicetreepath" which contains
the path where devicetrees are stored. This is also written to the
U-Boot variable "fdtdir". The boot loader is expected to use this path
to load a particular machines device tree from.

Closes: #1900
Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
2020-05-29 19:56:11 +02:00
Colin Walters
167edbe63a pull: Add error prefixing with specific object when parsing
One OpenShift user saw this from rpm-ostree:
```
client(id:cli dbus:1.583 unit:machine-config-daemon-host.service uid:0) added; new total=1
Initiated txn UpdateDeployment for client(id:cli dbus:1.583 unit:machine-config-daemon-host.service uid:0): /org/projectatomic/rpmostree1/rhcos
Txn UpdateDeployment on /org/projectatomic/rpmostree1/rhcos failed: File header size 4294967295 exceeds size 0
```

which isn't very helpful.  Let's add some error
prefixing here which would at least tell us which
object was corrupted.
2020-05-29 13:12:02 +00:00
Colin Walters
68ebf743cd remote-add: Default to explicit sign-verify backends
In 588f42e8c6
we added a way to add keys for sign types when doing
a `remote add`, and in https://github.com/ostreedev/ostree/pull/2105
we extended `sign-verify` to support *limiting* to an explicit
set.

This PR changes the *default* for `remote add` to combine
the two - when providing an explicit `--sign-verify=type`,
we now limit the accepted types to only those.
2020-05-28 00:41:34 +00:00
OpenShift Merge Robot
936301608a
Merge pull request #2113 from cgwalters/prepare-root-sysroot-ro
Move ro /sysroot bind mount of /etc into initramfs
2020-05-26 11:38:42 -04:00
Colin Walters
3564225917 Move ro /sysroot bind mount of /etc into initramfs
We recently disabled the read-only /sysroot handling:
e35b82fb89

The core problem was that a lot of services run early in the
real root and want write access to things like `/var` and `/etc`.

In trying to do remounts while the system is running we introduce
too many race conditions.

Instead, just make the `/etc` bind mount in the initramfs right
after we set up the main root.  This is much more natural really,
and avoids all race conditions since nothing is running in the
sysroot yet.

The main awkward part is that since we're not linking
`ostree-prepare-root` to GLib (yet) we have a hacky parser
for the config file.  But, this is going to be fine I think.

In order to avoid parsing the config twice, pass state from
`ostree-prepare-root` to `ostree-remount` via a file in `/run`.
2020-05-24 18:46:28 +00:00
Colin Walters
5cb9d0df38 pull: Add support for sign-verify=<list>
The goal here is to move the code towards a model
where the *client* can explicitly specify which signature types
are acceptable.

We retain support for `sign-verify=true` for backwards compatibility.
But in that configuration, a missing public key is just "no signatures found".

With `sign-verify=ed25519` and no key configured, we can
explicitly say `No keys found for required signapi type ed25519`
which is much, much clearer.

Implementation side, rather than maintaining `gboolean sign_verify` *and*
`GPtrArray sign_verifiers`, just have the array.  If it's `NULL` that means
not to verify.

Note that currently, an explicit list is an OR of signatures, not AND.
In practice...I think most people are going to be using a single entry
anyways.
2020-05-22 19:10:32 +00:00
Jonathan Lebon
e35b82fb89 switchroot/remount: Neuter sysroot.readonly for now
We're hitting issues with the read-only remounts racing with various
services coming up. Let's neuter it for now until we rework how it
works.

See: https://github.com/coreos/fedora-coreos-tracker/issues/488
2020-05-20 16:23:59 -04:00
Colin Walters
5a47c926c1 pull: Only have API to disable signapi for local pulls
There's a lot of historical baggage associated with GPG verification
and `ostree pull` versus `ostree pull-local`.  In particular nowadays,
if you use a `file://` remote things are transparently optimized
to e.g. use reflinks if available.

So for anyone who doesn't trust the "remote" repository, you should
really go through through the regular
`ostree remote add --sign-verify=X file://`
path for example.

Having a mechanism to say "turn on signapi verification" *without*
providing keys goes back into the "global state" debate I brought
up in https://github.com/ostreedev/ostree/issues/2080

It's just much cleaner architecturally if there is exactly one
path to find keys: from a remote config.

So here in contrast to the GPG code, for `pull-local` we explictily
disable signapi validation, and the `ostree_repo_pull()` API just
surfaces flags to disable it, not enable it.
2020-05-17 13:52:24 +00:00
Jonathan Lebon
79079c2657 lib/upgrader: Pull with timestamp-check-from-rev
For the same reason as https://github.com/coreos/rpm-ostree/pull/2094.
What we care most about is that the new commit we pull is newer than the
one we're currently sitting on, not necessarily that it's newer than the
branch itself, which it might not be if e.g. we're trying to deploy a
commit older than the tip but still newer than the deployment (via
`--override-commit`).
2020-05-14 15:27:29 -04:00
Jonathan Lebon
c8efce0656 lib/pull: Add timestamp-check-from-rev
The way `timestamp-check` works might be too restrictive in some
situations. Essentially, we need to support the case where users want to
pull an older commit than the current tip, but while still guaranteeing
that it is newer than some even older commit.

This will be used in Fedora CoreOS. For more information see:
https://github.com/coreos/rpm-ostree/pull/2094
https://github.com/coreos/fedora-coreos-tracker/issues/481
2020-05-14 14:00:42 -04:00
Colin Walters
82679ce834 finalize: Add RequiresMountsFor=/boot too
In https://bugzilla.redhat.com/show_bug.cgi?id=1827712
some OpenShift CI is seeing `/boot` being unmounted before
`ostree-finalize-staged.service` runs or completes.

We finally tracked this down to a bug elsewhere, but
I think we should add this because it clearly shows
our requirements.
2020-05-13 18:52:45 +00:00
Colin Walters
9509a4bc94 pull: Further cleanup signapi verification
Previously in the pull code, every time we went to verify
a commit we would re-initialize an `OstreeSign` instance
of each time, re-parse the remote configuration
and re-load its public keys etc.

In most cases this doesn't matter really because we're
pulling one commit, but if e.g. pulling a commit with
history would get a bit silly.

This changes things so that the pull code initializes the
verifiers once, and reuses them thereafter.

This is continuing towards changing the code to support
explicitly configured verifiers, xref
https://github.com/ostreedev/ostree/issues/2080
2020-05-12 15:20:26 +00:00
Colin Walters
a9a81f3a29 signing: Change API to create instances directly
This cleans up the verification code; it was weird how
we'd get the list of known names and then try to create
an instance from it (and throw an error if that failed, which
couldn't happen).
2020-05-10 14:18:55 +00:00
Colin Walters
588f42e8c6 remote-add: Add --sign-verify=KEYTYPE=[inline|file]:PUBKEYREF
Per https://github.com/ostreedev/ostree/issues/2080#issuecomment-623614483

A huge benefit of ed25519 (and ECC in general) is that keys are very
short - short enough that it's completely reasonable to inline
them into a command line argument.

And I think that's a good model; it makes the keys very visible.

For example, someone could easily copy-paste a commandline
argument from a webpage (secured via TLS) that says to run
`ostree remote add --sign-verify=ed25519=inline:KEY`.
2020-05-09 20:07:44 +00:00
Colin Walters
e1b6398787 ostree-prepare-root: Requires=sysroot.mount
With just `After=` we'll still try to run in the scenario
where `sysroot.mount` fails because the rootfs didn't appear.
And this will end up spewing an error which can confuse people
into thinking something is wrong at the ostree level.

This has come up numerous times w/{Fedora,RHEL} CoreOS, most
recently while looking at
https://bugzilla.redhat.com/show_bug.cgi?id=1803130
2020-05-08 12:50:55 +00:00
Colin Walters
7392259332 commit: Add --mode-ro-executables option
I think we should encourage removing the writable bits from
executables.  This has happened to me:
https://thomask.sdf.org/blog/2019/11/09/take-care-editing-bash-scripts.html

And not having the writable bit may help prevent hardlink
corruption with OSTree in some cases.

We can't do this by default, but add a convenient CLI flag
for it.
2020-05-06 19:41:27 +00:00
Colin Walters
811082063c signing: Add #define OSTREE_SIGN_NAME_ED25519
Using `#define` or constants instead of strings helps avoid
typos and encourages documentation.
2020-05-03 17:04:27 +00:00
Dan Nicholson
dc69f56de6 lib: Coerce flags enums to GIR bitfields
The GI scanner decides if an `enum` is really a `bitfield` if it finds
any values that have left shifts. With an `enumeration`, the
introspecting language may error or convert to a different type if the
user tries to combine values. Change all Flags `enum`s to use
left-shifted values so that they're represented as `bitfield`s in the
GIR.

The primary bug here is that you can't combine `REFS_ONLY` and
`NO_PRUNE` when calling `OSTree.Repo.prune()` from an introspected
language.

This is an IABI break since the typelib will change from `enumeration`
to `bitfield`. `OstreeRepoImportFlags` is internal but the change is
included here to prepare for a subsequent name that would require bit
shifting to operate correctly as a flag.
2020-04-30 15:13:31 -06:00
Denis Pynkin
ee5d0f350f signapi: expose metadata format and key
Explicitly expose functions for querying the metadata format
and key name used by OstreeSign object:
 - ostree_sign_metadata_format
 - ostree_sign_metadata_key

This allows to use the same metadata format and key name
by 3-rd party applications using signapi.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-04-24 16:34:50 +00:00
OpenShift Merge Robot
66527efcb8
Merge pull request #2082 from cgwalters/finalize-sandbox
finalize-staged: Add ProtectHome=yes and ReadOnlyPaths=/etc
2020-04-22 18:51:15 +02:00
Colin Walters
47a3096ab8 finalize-staged: Add ProtectHome=yes and ReadOnlyPaths=/etc
Same motivation as
https://github.com/coreos/rpm-ostree/pull/2060

I tried `InaccessiblePaths=/var` first and was very sad to find
out we have one tiny exception that breaks it.  Otherwise it'd
be so elegant.  Maybe in the future we split out that one thing
to a separate `ostree-finalized-stage-var.service` that's just
`ExecStart=/bin/rm -vf /var/.updated` and is otherwise
`ProtectSystem=strict` etc.
2020-04-22 14:28:33 +00:00
Colin Walters
368574b657 deploy: Add --no-merge
All of the underlying libostree APIs have supported passing `NULL`
for a merge deployment for...a long time.  But we never plumbed
it up into the CLI.

Add a `--no-merge` option to aid people who want to do a "factory reset":
https://github.com/ostreedev/ostree/issues/1793
2020-04-22 14:09:54 +00:00
Colin Walters
1b8fed247c lib: Move gpg/signapi bits into ostree-repo-pull-verify.c
`ostree-repo-pull.c` is huge; separate some of the GPG/signing
verification functions into their own file so they're more easily seen.
2020-04-18 13:35:36 +00:00
Colin Walters
12984ddef2 lib: Move pull struct definition into repo-pull-private.h
Prep for further splitting up `ostree-repo-pull.c`.
2020-04-18 13:34:57 +00:00
Jonathan Lebon
8ece36c28a lib/commit: Check that dirent is a directory before cleaning
I've only noticed this by inspection. But I think it's possible for
`cleanup_txn_dir` to get called with the `staging-...-lock` file since
it matches the prefix.

Make the checking here stronger by verifying that it's a directory. If
it's not a directory (lockfile), then follow the default pruning expiry
logic so that we still cleanup stray lockfiles eventually.
2020-04-17 14:29:13 -04:00
Jonathan Lebon
df065ad766 lib: Rename function for staging dir check
Rename the function to more accurately reflect what it does, which is to
check whether the filename has the `staging-` prefix.
2020-04-17 14:28:20 -04:00
Jonathan Lebon
bb8fd5a2c4 lib/commit: Add more error prefixing
We think we're hitting an error in that function in the Fedora infra.
Add some more error prefixing to help debugging.
2020-04-17 14:27:16 -04:00
Colin Walters
b7f48a74c0 lib: Move internal binding verification API to repo.c
`ostree-repo-pull.c` is rather monstrous; I plan to split it
up a bit.  There's actually already a `pull-private.h` but
that's just for the binding verification API.  I think that one
isn't really pull specific.  Let's move it into the "catchall"
`repo.c`.
2020-04-17 01:16:51 +00:00
Colin Walters
8e7aea4473 Change signature opts to include type, cleanup error handling
Previously we would pass the `verification-key` and `verification-file`
to all backends, ignoring errors from loading keys until we
found one that worked.

Instead, change the options to be `verification-<engine>-key`
and `verification-<engine>-file`, and then
rework this to use standard error handling; barf explicitly if
we can't load the public keys for example.  Preserve
the semantics of accepting the first valid signature.  The
first signature error is captured, the others are currently
compressed into a `(and %d more)` prefix.

And now that I look at this more closely there's a lot of
duplication between the two code paths in pull.c for verifying;
will dedup this next.
2020-04-15 22:07:11 +00:00
Colin Walters
e035230c6b lib/pull: Two cosmetic internal function renames
I'm mainly doing this to sanity check the CI state right now.

However, I also want to more cleanly/clearly distinguish
the "sign" code from the "gpg" code.

Rename one function to include `gpg`.

For the other...I think what it's really doing is using the remote
config, so change it to include `remote` in its name.
2020-04-15 17:06:14 +00:00
Colin Walters
b6040143e5 Only enable "dummy" signature type with opt-in env variable
I don't want to even have to think about people using
this in production.
2020-04-14 22:18:13 +00:00
OpenShift Merge Robot
8baee5cd04
Merge pull request #2058 from cgwalters/signing-cleanups
pull: Update key loading function to match error style
2020-04-07 20:49:04 +02:00
OpenShift Merge Robot
dd44f6f941
Merge pull request #2060 from d4s/ostree_repo_pull_with_options-gpg_check
lib/repo-pull: fix GPG check while pulling remote
2020-04-07 16:04:22 +02:00
OpenShift Merge Robot
4a57204c60
Merge pull request #2044 from martinezjavier/duplicated-entries
grub2: Don't add menu entries if GRUB supports parsing BLS snippets
2020-04-07 15:54:08 +02:00
Denis Pynkin
ef1aeb7458 lib/repo-pull: fix GPG check while pulling remote
If GPG support is disabled in a build time we should to check if any of
options "gpg_verify" or "gpg_verify_summary" is set to TRUE instead
of checking if they are passed via options while pulling from remote.

Fixed the failure with assertion of `ostree find-remotes --pull --mirror`
calling (`tests/test-pull-collections.sh`) if libostree has been compiled
without GPG support.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-04-07 01:11:29 +03:00
OpenShift Merge Robot
1b0eec02d4
Merge pull request #2059 from cgwalters/commit-from-base-tree
commit: Add --base argument
2020-04-06 22:18:49 +02:00
OpenShift Merge Robot
eefb5241c8
Merge pull request #2055 from cgwalters/gtkdoc-warnings
lib: Squash two gtk-doc warnings
2020-04-06 21:54:13 +02:00
Colin Walters
329a82c57e commit: Add --base argument
I was trying to followup the `--selinux-policy-from-base` work
to add a `cosa build --fast=overlay` for coreos-assembler,
but hit on the fact that using e.g. `--owner-uid` disables
commit optimizations.

A while ago, https://github.com/ostreedev/ostree/pull/1643 landed
which optimized this for the case where no modifications are provided.
But, we really need the SELinux policy bits, and it's super convenient
to run `ostree commit` as non-root.

It's fairly surprising actually that it's taken us so long to
iterate on a good interface for this "commit changes on top of a base"
model.  In practice, many nontrivial cases really end up needing
to do a (hardlink) checkout, and that case is optimized.

But for this coreos-assembler work I want to directly overlay onto
a commit object another commit object.

That previous PR above added exactly the API we need, so let's
expose it in the CLI.

What you can see happening in the test is that we provide
`--owner-uid 42`, but that only applies to directories/files
that were added in the commit.

And now that I look at this, I think what we really want here
is to avoid changing directories that exist in the base, but
eh; in practice the main use here is for `--owner-uid 0` while
committing as non-root; and that works fine with this since
the baseline uid will be zero as well.
2020-04-06 19:28:51 +00:00