Commit Graph

4445 Commits

Author SHA1 Message Date
Colin Walters
6d104f6257 ci: Replace PAPR with CoreOS CI
Move the alternative builds into the Jenkinsfile.

Update it to do a FCOS build + kola run.

We drop the flatpak/rpm-ostree runs for now; the former
will needs some work to do the automatic virt bits.  The
latter I think we can circle back to when we e.g. figure
out how to include rpm-ostree's tests in kola runs.
2020-02-06 23:14:42 +00:00
OpenShift Merge Robot
6b9282fa45
Merge pull request #1998 from cgwalters/cci-jenkinsfile
Add .cci.jenkinsfile
2020-02-03 10:42:55 -08:00
Colin Walters
7febd9d36e Add .cci.jenkinsfile
See https://github.com/jlebon/coreos-ci

This is just a start to test, cut down from the rpm-ostree version.
2020-02-03 16:57:24 +00:00
OpenShift Merge Robot
96fb1decf4
Merge pull request #1959 from cgwalters/ostree-verity
Initial fs-verity support
2020-01-28 13:01:40 -08:00
OpenShift Merge Robot
8a9a496501
Merge pull request #1957 from dbnicholson/commit-sizes
Upstream Endless sizes metadata changes
2020-01-27 08:25:59 -08:00
Colin Walters
58fa579b1b Initial fs-verity support
Using fs-verity is natural for OSTree because it's file-based,
as opposed to block based (like dm-verity).  This only covers
files - not symlinks or directories.  And we clearly need to
have integrity for the deployment directories at least.

Also, what we likely need is an API that supports signing files
as they're committed.

So making this truly secure would need a lot more work.  Nevertheless,
I think it's time to start experimenting with it.  Among other things,
it does *finally* add an API that makes files immutable, which will
help against some accidental damage.

This is basic enablement work that is being driven by
Fedora CoreOS; see also https://github.com/coreos/coreos-assembler/pull/876
2020-01-27 15:44:16 +00:00
OpenShift Merge Robot
08e292bc14
Merge pull request #1877 from dbnicholson/specific-gpg-errors
More specific GPG errors
2020-01-26 09:20:51 -08:00
OpenShift Merge Robot
570de2989c
Merge pull request #1996 from akiernan/us-with-systemd
build: fix systemd feature advertisement
2020-01-26 04:19:31 -08:00
Alex Kiernan
562b60022b build: fix systemd feature advertisement
17db0f15a7 ("configure: add option for libsystemd") exposed
--without-libsystemd to allow systemd to be disabled even if the systemd
pkgconfig script was present, introducing a new variable
with_libsystemd; there are now three, almost identical variables:

- with_libsystemd [yes, no, maybe] - controlled by --without-libsystemd,
  resolved into yes/no by the initial checks
- have_libsystemd [yes, no, <undefined>] - only set if with_libsystemd
  is yes/maybe, otherwise undefined
- with_systemd [yes, <undefined>] - yes if have_systemd is yes,
  otherwise undefined

with_systemd is the earliest variable and was previously set by a set of
checks for dracut and mkinitcpio. These checks were changed for a
systemd check in 9e2763106b ("lib: Use sd_journal directly
(optionally)"). This commit also introduced BUILDOPT_LIBSYSTEMD, which
will always match BUILDOPT_SYSTEMD.

Fix the confusion by removing with_systemd which will always be yes when
with_libsystemd=yes, or undefined if with_libsystemd=no. We can ignore
the with_libsystemd=maybe case because it will always be resolved into
yes/no before with_systemd is set.

And replace all uses of BUILDOPT_SYSTEMD with BUILDOPT_LIBSYSTEMD, since
they again always match.

This fixes both the advertised features and the summary output when
systemd is disabled by using with_libsystemd which is always defined.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Fixes: 5c62a7e4d0 ("build: Expose systemd in OSTREE_FEATURES")
Fixes: 17db0f15a7 ("configure: add option for libsystemd")
Supersedes: #1992
2020-01-25 13:12:41 +00:00
OpenShift Merge Robot
c97bdd6d28
Merge pull request #1948 from dbnicholson/trivial-httpd-autoexit
ostree/trivial-httpd: Fix --autoexit with --daemonize and --log-file
2020-01-24 12:42:59 -08:00
Dan Nicholson
11ad68647a ostree/trivial-httpd: Add log message for autoexit
This is useful when checking if the daemon actually exited since we
don't store the child PID anywhere.
2020-01-24 13:11:54 -07:00
Dan Nicholson
fbf5a94e0a ostree/trivial-httpd: Fix --autoexit with --daemonize and --log-file
When --autoexit is used with --daemonize and --log-file, the program
never exits when the root directory is deleted. I believe what happens
is that g_file_new_for_path triggers the glib worker context to be
started to talk to GVfs. Once the program forks, the parent exits and
the thread iterating the worker context is gone. The file monitor then
never receives any events because the inotify helper also runs from the
worker context.

Move the fork earlier just after parsing and validating the command line
arguments. In order to handle setup errors in the child, a pipe is
opened and the parents waits until the child writes a status byte to it.
If the byte is 0, the parent considers the child setup successful and
exits and the child carries on as a daemon. Notably, the child doesn't
reopen stderr to /dev/null until after this so that it can send error
messages there.

Fixes: #1941
2020-01-24 13:11:54 -07:00
Dan Nicholson
68a11d4eeb ci/flatpak: Patch GPG error assertions from OSTree
Some of the flatpak tests assert on GPG error strings that come from
OSTree. Those are being changed here, so patch the cloned flatpak 1.4.1
to accommodate the new error strings. When this work lands, I'll send a
patch upstream to flatpak that will eventually trickle back here in a
tagged build.
2020-01-24 13:03:30 -07:00
Dan Nicholson
b81a6b4ab2 tests/gpg: Add tests for importing updated remote GPG keys
This checks whether gpg-import will properly update the keyring for a
key that already exists. In particular, we check that changing the key
expiration time or revoking it results in commit verification failure
after re-importing the keys.
2020-01-24 13:02:17 -07:00
Dan Nicholson
01da2371c5 tests/gpg: Test ostree_gpg_verify_result_require_valid_signature
Add explicit tests for
`ostree_gpg_verify_result_require_valid_signature` in addition to the
implicit tests via `ostree pull` and others. This allows checking the
error code raised.
2020-01-24 13:02:17 -07:00
Dan Nicholson
0fbfc0b207 lib/gpg: Add more specific OstreeGpgError codes
Currently `ostree_gpg_verify_result_require_valid_signature` always
returns an error that the key used for the signature is missing from the
keyring. However, all that's been determined is that there are no valid
signatures. The error could also be from an expired signature, an
expired key, a revoked key or an invalid signature.

Provide values for these missing errors and return them from
`ostree_gpg_verify_result_require_valid_signature`. The description of
each result is appended to the error message, but since the result can
contain more than one signature but only a single error can be returned,
the status of the last signature is used for the error code. See the
comment for rationale.

Related: flatpak/flatpak#1450
2020-01-24 13:02:17 -07:00
Dan Nicholson
2c24f28ce4 tests/test-gpg-verify-result: Allow specifying signature files
Currently tests are always run against the full lgpl2.sig file with all
signatures, but it should also be possible to specify one or more of the
individual lgpgl2.sig<N> files.

Drop the current usage of passing the signature index in the test data
since it's always specific to the test function and instead provide an
optional array of signature files for the test fixture to sign with.
2020-01-24 13:02:09 -07:00
Dan Nicholson
b786d1b4bc tests/gpg-verify-data: Empty out trustdb.gpg
When the private keys were generated, gpg added an ultimate trust entry
since you normally want to trust your own keys. However, this throws off
the expired signature testing since gpgme considers it valid if the key
is fully or ultimately trusted.

The use of a trustdb for the test-gpg-verify-result is unlike any other
GPG verification in ostree. Under normal circumstances, a temporary GPG
homedir is created without any trust information, so all keys are
treated as having unknown trust.

Regenerate an empty trustdb.gpg in gpg-verify-data so that the tests
behave as ostree normally operates. After this the expired signature
testing correctly shows up as a non-valid signature. The trustdb was
regenerated by simply removing it and running any gpg operation with the
gpg-verify-data directory as the homedir.
2020-01-24 13:02:09 -07:00
Dan Nicholson
7fe265b087 tests/gpg-verify-data: Split out signature data
The full block with all 5 signatures remains, but this allows passing
individual signatures through the GPG verification APIs. The split was
done with `gpgsplit`, and looking at the output of `gpg --list-packets`
of the split and unsplit files appears correct.
2020-01-24 13:02:09 -07:00
Dan Nicholson
b825083549 tests/gpghome: Create revocation certificates for keys
These can then be imported during a test to revoke a key without trying
to go through the gpg --generate-revocation dialog. Note that these need
to go in a subdirectory of the homedir since `gpgkeypath` will try to
import every regular file in the homedir.
2020-01-24 13:02:09 -07:00
Dan Nicholson
63414e85c3 tests/libtest: Make temporary gpghome private
gpg prints a warning about unsafe permissions if the homedir is group or
world readable. This is just noise in the test logs, so appease it by
making the homedir 700.
2020-01-24 13:02:09 -07:00
Dan Nicholson
7f04c5d764 tests/libtest: Record long GPG key IDs and fingerprints
Use long GPG key IDs as it's safer and matches the format used by gpg
and gpgme. Add the associated fingerprints since these are needed by gpg
when manipulating keys.
2020-01-24 13:02:09 -07:00
Dan Nicholson
c54a3b5daa lib/gpg: Prefer declare-and-initialize style
As noted in
https://github.com/ostreedev/ostree/pull/1872#discussion_r295408768.
2020-01-24 13:02:09 -07:00
OpenShift Merge Robot
8cc81126a1
Merge pull request #1993 from dbnicholson/f29-rpmostree-fixes
ci/rpmostree: Bump to 2019.4
2020-01-22 15:01:25 +01:00
Dan Nicholson
a6994459c1 ci/rpmostree: Bump to 2019.4
The vmcheck tests in 2019.3 fail because of an SSH control socket issue
on overlayfs. This is fixed in 2019.4[1]. That has some other changes
such as using Python 3 in tests. The package dependencies have been
synced from the rpm-ostree CI for that.

Unfortunately, this is no longer a totally representative test of f29
since it has 2019.3 in updates. But that's the price you pay for
exercising someone else's CI from your own CI.

1. c89f81c138

Fixes: #1994
2020-01-21 20:33:58 -07:00
OpenShift Merge Robot
859dd00bfa
Merge pull request #1991 from dbnicholson/test-fixes
tests/core: Really pick C.UTF-8 locale
2020-01-21 21:13:02 +01:00
Dan Nicholson
5135a1e58a tests/core: Really pick C.UTF-8 locale
The case-ignoring regex `^(C|en_US)` will match any locale that starts
with `c`. On my system this is `ca_AD.utf8`, which breaks the test
suite. Instead, use a single regex that includes the joining `.` rather
than 2 separate regexes. This also changes `head` to use the `-n`
option, which has been preferred for at least 10 years in the coreutils
version and is supported by busybox as well.
2020-01-21 10:29:41 -07:00
Dan Nicholson
97c831dd5f bin/show: Add --print-sizes option to show sizes metadata
Use the new `ostree_commit_get_object_sizes()` API to read the
`ostree.sizes` commit metadata and print a summary.
2020-01-20 20:46:31 -07:00
Dan Nicholson
260bcd1193 core: Add ostree_commit_get_object_sizes API
This function parses the object listing in the `ostree.sizes` metadata
and returns an array of `OstreeCommitSizesEntry` structures.

Unfortunately, for reasons I don't understand, the linker wants to
resolve `_ostree_read_varuint64` from `ostree-core.c` even though it's
not used by `test-checksum.c` at all.
2020-01-20 20:46:29 -07:00
Dan Nicholson
fcbb453443 core: Add OstreeCommitSizesEntry type
This will be used when reading out entries in the `ostree.sizes`
metadata. Each entry corresponds to an object in the metadata array.
2020-01-20 20:44:12 -07:00
Dan Nicholson
1bbe674d91 libarchive: Support commit sizes metadata
Call the helper to set the generate_sizes boolean so that object size
data is stored while writing the mtree.
2020-01-20 20:42:27 -07:00
John Hiesey
291e9da258 lib/commit: Include object type in sizes metadata
Append a byte encoding the OSTree object type for each object in the
metadata. This allows the commit metadata to be fetched and then for the
program to see which objects it already has for an accurate calculation
of which objects need to be downloaded.

This slightly breaks the `ostree.sizes` `ay` metadata entries. However,
it's unlikely anyone was asserting the length of the entries since the
array currently ends in 2 variable length integers. As far as I know,
the only users of the sizes metadata are the ostree test suite and
Endless' eos-updater[1]. The former is updated here and the latter
already expects this format.

1. https://github.com/endlessm/eos-updater/
2020-01-20 20:42:27 -07:00
Dan Nicholson
a4592678aa tests/sizes: Check duplicate file doesn't add sizes entry
A duplicate file will resolve to the same object, so it shouldn't add
any entries to the sizes metadata.
2020-01-20 20:42:27 -07:00
Dan Nicholson
4f1b991246 tests/sizes: Test that sizes metadata is not reused
Ensure that the object sizes hash table is cleared after a commit and
not only when the repo is closed.
2020-01-20 20:42:27 -07:00
Dan Nicholson
1ea719b76b tests/sizes: Test sizes metadata with existing objects
Repeat the commit to make sure that the files are enumerated again for
the size metadata.
2020-01-20 20:42:27 -07:00
Dan Nicholson
44fb5e72a1 lib/commit: Make size entries for existing objects
If the object was already in the repo then the sizes metadata entry was
skipped. Move the sizes entry creation after the data has been computed
but before the early return for an existing object.
2020-01-20 20:42:27 -07:00
Dan Nicholson
8ec7d6322f lib/commit: Fix object sizes metadata for multiple commits
The object sizes hash table was only being cleared when the repo was
finalized. That means that performing multiple commits while the repo
was open would reuse all the object sizes metadata for each commit.

Clear the hash table when the sizes metadata is setup and when it's
added to a commit. This still does not fix the issue all the way since
it does nothing to prevent the program from constructing multiple
commits simultaneously. To handle that, the object sizes hash table
should be attached to the MutableTree since that has the commit state.
However, the MutableTree is gone when the commit is actually created.
The hash table would have to be transferred to the root file when
writing the MutableTree. That would be an awkward addition to
OstreeRepoFile, though. Add a FIXME to capture that.
2020-01-20 20:42:27 -07:00
Dan Nicholson
694b741a36 tests/sizes: Improve metadata validation
Ensure all 3 of the checksum, compressed size and uncompressed size are
correct. For repeatable objects, skip xattrs and use canonical
permissions for the commit. For the sizes, read a varint rather than
assuming they will be a single byte. To work around bugs in gjs with
byte array unpacking, manually build the array byte by byte. Split out
some helper functions to use in subsequent tests.
2020-01-20 20:42:27 -07:00
Dan Nicholson
37045b4b46 lib/commit: Only set generate_sizes for archive repos
Rather than checking throughout the code, only set the boolean when
appropriate.
2020-01-20 20:42:27 -07:00
OpenShift Merge Robot
20d84f40fa
Merge pull request #1987 from akiernan/us-switchroot-tests
Skip /var test if running with systemd and libmount
2020-01-16 16:47:41 +01:00
OpenShift Merge Robot
c0d760bb0c
Merge pull request #1988 from akiernan/us-switchroot-installed
test-switchroot.sh: Find ostree-prepare-root in installed tests
2020-01-14 20:30:33 +01:00
Alex Kiernan
d61183ce43 fixup! test-switchroot.sh: Find ostree-prepare-root in installed tests 2019-12-30 13:28:35 +00:00
Alex Kiernan
e4db245bec test-switchroot.sh: Find ostree-prepare-root in installed tests
When running with installed tests, ostree-prepare-root (probably)
exists in /usr/lib. Add heuristics to look for it based on the directory
we're running from.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
2019-12-29 14:24:30 +00:00
Alex Kiernan
87ccb400a2 tests: Skip /var test if running with systemd and libmount
If running with systemd and libmount then /var mounting is deferred for
systemd. Skip the relevant tests in this case as it will always fail.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
2019-12-29 14:18:36 +00:00
Alex Kiernan
5c62a7e4d0 build: Expose systemd in OSTREE_FEATURES
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
2019-12-29 13:27:29 +00:00
Alex Kiernan
f6867358e2 test-switchroot.sh: Exclude /proc from file list
Since we're not interested in any file inside /proc, exclude it from the
file listing in our fake root thus avoiding failures when processes die
during our execution and find(1) can't then look inside those
directories.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
2019-12-29 12:37:10 +00:00
OpenShift Merge Robot
b9a95afacc
Merge pull request #1767 from cgwalters/sysroot-mnt-namespace
Support mounting /sysroot (and /boot) read-only
2019-12-11 21:34:36 +01:00
Colin Walters
5af403be0c Support mounting /sysroot (and /boot) read-only
We want to support extending the read-only state to cover `/sysroot`
and `/boot`, since conceptually all of the data there should only
be written via libostree.  Or at least for `/boot` should *mostly*
just be written by ostree.

This change needs to be opt-in though to avoid breaking anyone.

Add a `sysroot/readonly` key to the repository config which instructs
`ostree-remount.service` to ensure `/sysroot` is read-only.  This
requires a bit of a dance because `/sysroot` is actually the same
filesystem as `/`; so we make `/etc` a writable bind mount in this case.

We also need to handle `/var` in the "OSTree default" case of a bind
mount; the systemd generator now looks at the writability state of
`/sysroot` and uses that to determine whether it should have the
`var.mount` unit happen before or after `ostree-remount.service.`

Also add an API to instruct the libostree shared library
that the caller has created a new mount namespace.  This way
we can freely remount read-write.

This approach extends upon in a much better way previous work
we did to support remounting `/boot` read-write.

Closes: https://github.com/ostreedev/ostree/issues/1265
2019-12-11 15:33:57 +00:00
Colin Walters
20daf96880 finalize-staged: Use the core option parsing to load sysroot
Prep for using the default mount namespace handling there that
will land as part of the read-only `/sysroot` and `/boot` work.
See https://github.com/ostreedev/ostree/issues/1265
2019-12-11 14:39:17 +00:00
OpenShift Merge Robot
8f6da76f0f
Merge pull request #1981 from cgwalters/release-2019.6
Release 2019.6
2019-12-09 14:26:55 -05:00