Commit Graph

4519 Commits

Author SHA1 Message Date
Denis Pynkin
bc4488692c lib/sign: public API optimisation
Removed from public `ostree_sign_detached_metadata_append` function.
Renamed `metadata_verify` into `data_verify` to fit to real
functionality.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
95ab57c17e lib/sign-ed25519: cleanup unneeded code
Removed unused code.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
ea291a0605 builtin/sign: allow to use multiple public keys for verification
`ostree sign` is able to use several public keys provided via arguments
and via file with keys.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
557f423609 sign: fix memory leaks and code cleanup
Return `const char *` instead of copy of the string -- this allow to
avoid unneeded copying and memory leaks in some constructions.
Minor code cleanup and optimisations.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
5fc2ddff30 tests/gpg: skip test in JS if GPG is not supported
Skip the single JS test which throws an error if GPG support
is disabled in a build time.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
36e4667973 builtin/sign: allow to sign with keys from secret file
Read keys from secret file provided by `--keys-file=` option.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
94447617db lib/sign: read ed25519 public keys from well known places
If not provided key of file name with keys for remote, then try to use
system defaults:
- /etc/ostree/trusted.ed25519
- /etc/ostree/trusted.ed25519.d/*
- /usr/share/ostree/trusted.ed25519
- /usr/share/ostree/trusted.ed25519.d/*

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
073876d9b2 lib/sign: add support of file with valid keys for remote
Allow to use custom file with public keys for remote.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
91cc294d05 lib/sign: disable mandatory signature check
Do not check the signature check by default.
Need to enable it explicitly.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
82c773710c tests/sign: add initial test for pulling
Test if we pull signed commits from remote.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
3386893deb sign: check signatures for pulled commits
If `verification-key` is set for remote it is used as a public key for
checking the commit pulled from that remote.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
2831028c41 sign: fixes for ed25519 for loading public keys from files
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
fe3a839ae7 builtin/sign: remove libsodium dependency
Now do not need to compile/link builtin with external dependencies.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
a8521a7c3b sign: fix error return for dummy module
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
2d39126628 sign: fix unneeded objects creation
Do not create objects just for supported modules list.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
4b9232b1fe builtin/sign: remove libsodium-specific code
Use only common sign API without libsoduim parts.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
0b55db9b2f tests/sign: check public keys load from file
Test ed25519 public keys load from file and verify signed commit
against that file.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
06cfcd9a8a builtin/sign: allow to provide the file with public keys
Added option `--keys-file` for `ostree sign`.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
2303202c86 sign: API changes for public keys and CLI keys format
API changes:
- added function `ostree_sign_add_pk()` for multiple public keys using.
- `ostree_sign_set_pk()` now substitutes all previously added keys.
- added function `ostree_sign_load_pk()` allowed to load keys from file.
- `ostree_sign_ed25519_load_pk()` able to load the raw keys list from file.
- use base64 encoded public and private ed25519 keys for CLI and keys file.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
9e8f0f4ca0 tests: add test for commits sign/verification
Add tests checking:
- sign mechanism is in working state
- module 'dummy' is able to sign/verify commit
- module 'ed25519' is able to sign/verify commit
- both modules could be used for the same commit
- 'ostree sign' builtin works with commits
- 'ostree commit' builtin able to sign commits

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
c09df18454 lib/sign: enable verification for pulling
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
e133cb7b74 sign: allow to sign commits from CLI
Add signing ability to commit builtin.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
c3608aa56a sign: add new builtin for signing
This builtin allows to sign and verify commit with new signature
mechanism. At the moment it is possible to use 'dummy' and 'ed25519'
signing modules.

'dummy' module use any ASCII string from command line as a key for
commit's signing or verification.

Support of ed25519 signature is implemented with `libsoium` library.
Secret and public key should be provided in hex presentation via
command line.

Based on 'gpg-sign' source.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
edbbe1c4f2 lib/sign: initial implementation
Added the initial version of signing interface allowing to allowing to
sign and verify commits.
Implemented initial signing modules:
 - dummy -- simple module allowing to sign/verify with ASCII string
 - ed25519 -- module allowing to sign/verify commit with ed25519
   (EdDSA) signature scheme provided by libsodium library.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
84c8164610 Add libsodium dependency
Allow to configure with libsodium flag.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:22:41 +03:00
OpenShift Merge Robot
7a95929867
Merge pull request #2041 from cgwalters/auto-sepolicy
repo/commit: Add support for --selinux-policy-from-base
2020-03-24 20:06:55 +01:00
Colin Walters
b3bbbd1542 repo/commit: Add support for --selinux-policy-from-base
The [dev-overlay](332c6ab3b9/src/cmd-dev-overlay)
script shipped in coreos-assembler mostly exists to deal
with the nontrivial logic around SELinux policy.  Let's make
the use case of "commit some binaries overlaying a base tree, using
the base's selinux policy" just require a magical
`--selinux-policy-from-base` argument to `ostree commit`.

A new C API was added to implement this in the case of `--tree=ref`;
when the base directory is already checked out, we can just reuse
the existing logic that `--selinux-policy` was using.

Requires: https://github.com/ostreedev/ostree/pull/2039
2020-03-24 16:34:26 +00:00
OpenShift Merge Robot
bdccbb16d4
Merge pull request #2042 from cgwalters/pull-repeated-no-flake
tests/pull-repeated: Bump up retries to match max fails
2020-03-24 16:49:14 +01:00
Colin Walters
ea16f7603d tests/pull-repeated: Bump up retries to match max fails
This test keeps occasionally failing in CI - as expected, because
we retry 8 times for an object but it's completely possible for
us to hit the <0.5% chance of 50% failure 8 times in a row.

Since the max errors from the server is 100, set retries to the
same thing.
2020-03-24 12:24:46 +00:00
OpenShift Merge Robot
41d3dfa7b8
Merge pull request #2039 from cgwalters/commit-cli-cleanup
main/commit: Rework control flow to use --tree=X path
2020-03-24 02:22:22 +01:00
Colin Walters
a717535490 main/commit: Rework control flow to use --tree=X path
Rework the simple cases of "commit ." and "commit argv[1]" to
generate the more general "--tree=X --tree=Y" path, so that we
only have one primary control flow here.

Prep for a future patch around loading SELinux policy from
the first argument.
2020-03-23 19:03:16 +00:00
OpenShift Merge Robot
d87301b111
Merge pull request #2040 from cgwalters/itest-require-writable
tests/kola: Two test fixes
2020-03-23 20:01:04 +01:00
Colin Walters
635fd0aa9c tests/kola: Two test fixes
Now that we're actually running this in CI.
2020-03-21 20:48:15 +00:00
OpenShift Merge Robot
89fdbf1b0e
Merge pull request #2037 from mwleeds/retry-on-partial-input
lib/fetcher-util: retry download on G_IO_ERROR_PARTIAL_INPUT
2020-03-20 14:35:14 +01:00
Matthew Leeds
53d1efbaab lib/fetcher-util: retry download on G_IO_ERROR_PARTIAL_INPUT
Add G_IO_ERROR_PARTIAL_INPUT to the list of error codes caused by
transient networking errors which lead us to retry the request. When
attempting to install the spotify flatpak you often get the error
message "Connection terminated unexpectedly" and the download of the deb
file fails. In this case, libsoup is setting G_IO_ERROR_PARTIAL_INPUT
and sometimes a subsequent download attempt is successful, so we should
treat it as transient.

Ideally we would behave as wget does in this case and retry the download
picking up where we left off in the file rather than starting over, but
that would require changes to libsoup I think.

Sadly this patch does not fix the flatpak installation of spotify in the
face of such errors, because flatpak doesn't use libostree to download
extra data, but presumably it's possible we could encounter such an
error pulling from an ostree repo, so the patch is still correct.
2020-03-19 15:28:08 -07:00
OpenShift Merge Robot
d0b3e76fa8
Merge pull request #2036 from jlebon/pr/partial-parent
lib/pull: Don't leave commits pulled by depth as partial
2020-03-19 18:05:00 +01:00
OpenShift Merge Robot
371a327dc2
Merge pull request #2025 from cgwalters/use-kola-run-ext
tests: Rework tests/installed → tests/kola
2020-03-19 17:32:23 +01:00
Colin Walters
b93180a4d3 tests: Rework tests/installed → tests/kola
Previously we made an effort to use the [Fedora Standard Test Interface](https://docs.fedoraproject.org/en-US/ci/standard-test-interface/).
This effort was not very successful; the primary thing that
it really died on is Ansible just didn't support rebooting
very well.  I think that's since gotten better, but even
then, Ansible wasn't the best thing for a test framework
for us anyways.

In the meantime Fedora CoreOS happened emphasizing Ignition
and not "post-hoc reconciliation" models like Ansible over
ssh.

And, [coreos-assembler](https://github.com/coreos/coreos-assembler) happened too.

Furthermore, we really need to test OSTree's interaction
with Ignition as we've invented several special things there.

Then most recently, I've been working on having
cosa/kola support running externally defined tests:
https://github.com/coreos/coreos-assembler/pull/1215

There's a lot of things to clean up after this but at least this
works for me:

```
$ cd /srv/fcos
$ cosa kola run -- --parallel 4 --output-dir tmp/kola -E ~/src/github/ostreedev/ostree/ 'ext.ostree.*'
```

NOTE: This *does not* drop ostree binaries into the target.  See:
https://github.com/coreos/coreos-assembler/pull/1252#issuecomment-600623315

This drops our dependency on Python in the installed tests, and
also fixes a few bugs that came up.

I disabled the `itest-bare-user-root.sh` one because it's
entangled with the shell script infrastructure for the unit tests.
2020-03-19 16:01:57 +00:00
Jonathan Lebon
fd822a8dfd lib/pull: Don't leave commits pulled by depth as partial
When pulling full parent commits via e.g. `--depth N`, we weren't
unmarking them as partial in the out path.

Closes: #2035
2020-03-19 10:10:50 -04:00
OpenShift Merge Robot
ab22e9dacb
Merge pull request #2034 from jlebon/pr/diff-docs
bin/diff: Clarify documentation around REV and DIR syntax
2020-03-19 01:06:15 +01:00
Jonathan Lebon
f7a8617c01 bin/diff: Clarify documentation around REV and DIR syntax
Related: #2032
2020-03-18 10:32:52 -04:00
OpenShift Merge Robot
f7582ce87e
Merge pull request #2033 from cgwalters/pin-error-handling
main/pin: Fix usage of GError
2020-03-18 14:42:42 +01:00
Colin Walters
0b6ac72b7f main/pin: Fix usage of GError
This regressed in 2db79fb398
I noticed this while finally getting the installed tests to run
in FCOS via kola and `ostree admin pin 0` is now aborting because
we were returning TRUE, but no error set.

I don't see a reason to try to continue on if we hit an error;
the original reporter was requesting support for multiple arguments,
but not "ignore invalid requests".
2020-03-18 02:15:34 +00:00
OpenShift Merge Robot
35c8fd3722
Merge pull request #2030 from jlebon/pr/2020.3
Release 2020.3
2020-03-14 16:17:21 +01:00
Jonathan Lebon
8f2c240326 Post-release version bump 2020-03-13 21:59:35 -04:00
Jonathan Lebon
6ed48234ba Release 2020.3
Let's do another release to get the `sysroot.readonly` fixes into FCOS
and unpin ostree and rpm-ostree there.
2020-03-13 21:59:05 -04:00
OpenShift Merge Robot
5164dcbe78
Merge pull request #2027 from cgwalters/pull-auto-mnt-namespace
main: Also automatically remount rw /sysroot for `ostree pull` etc.
2020-03-14 02:52:43 +01:00
Colin Walters
6a27dff941 main: Also automatically remount rw /sysroot for ostree pull etc.
See https://github.com/coreos/fedora-coreos-tracker/issues/343
When we added the read-only sysroot support it broke using "raw"
`ostree pull` and `ostree refs --create` and all of the core repo
CLIs that just operate on a repo and not a sysroot.

Fixing this is a bit ugly as it "layer crosses" things even more.
Extract a helper function that works in both cases.
2020-03-13 20:53:26 +00:00
OpenShift Merge Robot
2d901957b8
Merge pull request #2029 from jlebon/pr/upgrade-tests
ci: use `fcosKola` for running kola tests
2020-03-13 21:53:04 +01:00
Jonathan Lebon
cbb3f0a8c2 ci: use fcosKola for running kola tests
We were using `--no-test-exit-error` for upgrade tests but weren't
actually checking for test failures after.

Instead of running kola directly, just use the `fcosKola` custom step
which automatically takes care of e.g. running tests in parallel and
archiving results.
2020-03-13 14:35:53 -04:00