Commit Graph

1089 Commits

Author SHA1 Message Date
Colin Walters
73e3ccc401 Use generator to enable ostree-remount.service and ostree-finalize-staged.path
We struggled for a long time with enablement of our "internal units",
trying to follow the philosophy that units should only be enabled
by explicit preset.

See https://bugzilla.redhat.com/show_bug.cgi?id=1451458
and https://github.com/coreos/rpm-ostree/pull/1482
etc.

And I just saw chat (RH internal on a proprietary system sadly) where
someone hit `ostree-remount.service` not being enabled in CentOS8.

Thinking about this more, I realized we've shipped a systemd generator
for a long time and while its only role until now was to generate `var.mount`,
but by using it to force on our internal units, we don't require
people to deal with presets anymore.

Basically we're inverting things so that "if ostree= is on the kernel
cmdline, then enable our units" and not "enable our units, but have
them use ConditionKernelCmdline=ostree to skip".

Drop the weird gyrations we were doing around `ostree-finalize-staged.path`
too; forking `systemctl start` is just asking for bugs.

So after this, hopefully we won't ever again have to think about
distribution presets and our units.
2021-06-16 09:40:28 -04:00
Colin Walters
edf7477ee9 deploy: Warn if we find content in the deployment's /var
This will be ignored, so let's make it very clear
people are doing something wrong.  Motivated by a bug
in a build pipeline that injected `/var/lib/rpm` into an ostree
commit which ended up crashing rpm-ostree because it was an empty db
which it wasn't expecting.

It *also* turns out rpm-ostree is incorrectly dumping content in the
deployment `/var` today, which is another bug.
2021-06-10 07:33:17 -04:00
Dan Nicholson
06bb56be6d tests: Add single process repo locking tests
The semantics of multiple process locking are covered by
test-concurrency.py, but the semantics of the repository locking from a
single process aren't handled there.

This checks how the repository locking is handled from a single thread
with one OstreeRepo, a single thread with multiple OstreeRepos, and
multiple threads sharing an OstreeRepo.
2021-06-05 09:15:34 -06:00
Dan Nicholson
055b263dee test-concurrency: Lower lock timeout
If there's a locking issue in this test, then it's likely not going to
resolve after a few seconds of serializing access. Lower the default 30
second lock timeout to 5 seconds to prevent the test from hanging
unnecessarily.
2021-06-05 09:15:34 -06:00
Dan Nicholson
c3ada6fa7a repo: Require lock type in ostree_repo_lock_pop
This simplifies the lock state management considerably since the
previously pushed type doesn't need to be tracked. Instead, 2 counters
are kept to track how many times each lock type has been pushed. When
the number of exclusive locks drops to 0, the lock transitions back to
shared.
2021-06-05 09:07:39 -06:00
Colin Walters
0f36d8c221 repo: Make locking APIs public
Doing anything even somewhat sophisticated requires this;
turns out our own `ostree prune` CLI wants this, e.g.
https://github.com/ostreedev/ostree/issues/2337

Closes: https://github.com/ostreedev/ostree/issues/2286
2021-06-05 09:00:21 -06:00
Dan Nicholson
9c20d162a4 tests/gpg: Don't assert subkey expiration when only primary expired
In gnupg 2.3.0[1], if a primary key is expired and a subkey does not
have an expiration or its expiration is older than the primary key, the
subkey's expiration will be reported as the primary's. Previously a
subkey without an expiration would not report one regardless of the
primary key's expiration.

This caused a regression in a test setting an expiration on a primary
key. The test was checking that the subkey was not expired by asserting
that there was no `Key expired` line in the signature verification
output. With gnupg 2.3.0+, it will show as expired, causing the test to
fail.

Remove the assertion since it's not consistent across gnupg versions. In
practice we don't care whether the subkey is considered expired or not
as long as the signature verification fails when the primary key is
expired.

1. https://dev.gnupg.org/T3343

Fixes: #2359
2021-05-28 15:35:58 -06:00
Colin Walters
6664ee4ed4 ci: Fix staged-delay to work with newer systemd
Yeah, we should stop parsing the text; I need to dig at that
at some point.
2021-05-25 16:28:32 -04:00
Timothée Ravier
a709d4f846 *: rename master branch to main (external repos) 2021-05-07 16:55:03 +02:00
Timothée Ravier
02527f115e *: rename master to main in tests & examples 2021-05-07 16:55:03 +02:00
Colin Walters
0f3bccf640 pull: Cleanly error out on unknown schemes
Previous to this we'd trip an assertion `abort()` deep in the curl code if e.g.
a user did `ostree remote add foo htttp://...` etc.

Motivated by considering supporting "external remotes" where code outside
ostree does a pull, but we want to reuse the signing verification infrastructure.
2021-04-27 13:11:18 -04:00
Dan Nicholson
e660855796 tests: Test without a cache directory by default
Several tests generate summaries and then expect to use the generated
summary immediately. However, this can cause intermittent test failures
when they inadvertantly get a cached summary file. This typically
happens when the test is run on a filesystem that doesn't support user
extended attributes. In that case, the caching code can only use the
last modified time, which only has 1 second granularity. If tests don't
carefully manage the summary modification times or the repo cache then
they are likely subject to races in some test environments.

This introduces an environment variable `OSTREE_SKIP_CACHE` that
prevents the repo from using a cache directory. This is enabled by
default in tests and disabled for tests that are a explicitly trying to
test the caching behavior.

Fixes: #2313
Fixes: #2351
2021-04-19 11:09:44 -06:00
Simon McVittie
9d48d39a4b libtest: On failure, make it clearer what has happened
If we fail as a result of `set -x`, It's often not completely obvious
which command failed or how. Use a trap on ERR to show the command that
failed, and its exit status.

Signed-off-by: Simon McVittie <smcv@collabora.com>
2021-04-18 12:17:00 +01:00
Colin Walters
e4d42be31f
Merge pull request #2343 from cgwalters/cargo-workspace
build-sys: Add toplevel workspace Cargo.toml
2021-04-15 19:32:11 -04:00
Colin Walters
6a72674ec6 Release 2021.2 2021-04-15 13:02:48 -04:00
Colin Walters
7310203ce9 tests/inst: Make nondestructive tests runnable as unit tests
Ideally in the future we change more of our unit tests to
support running installed; we've tried this in the past with
https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests

I'd like to pick that back up again.  This takes a step
towards that by having our Rust tests.

To make this even easier, add a `tests/run-installed`
which runs the installed tests (uninstalled, confusingly
but conveniently for now).
2021-04-15 12:33:34 -04:00
Colin Walters
46a0911c6d build-sys: Add toplevel workspace Cargo.toml
rust-analyzer is happier with this because it understands
the project structure out of the box.

We aren't actually again adding a dependency on Rust/cargo in the core,
this is only used to make `cargo build` work out of the box to build
the Rust test code.
2021-04-15 12:33:34 -04:00
Colin Walters
2cc11b7808 tests: More tests for inline writing
I'm hitting a bug, add more tests.
2021-04-09 21:54:44 +00:00
Colin Walters
6f84aff0ae repo: Add ostree_repo_write_regfile
This API is push rather than pull, which makes it much more
suitable to use cases like parsing a tar file from external
code.

Now, we have a large mess in this area internally because
the original file writing code was pull based, but static
deltas hit the same problem of wanting a push API, so I added
this special `OstreeRepoBareContent` just for writing regular
files from a push API.

Eventually...I'd like to deprecate the pull based API,
and rework things so that for regular files the push API
is the default, and then `write_content_object()` would
be split up into archive/bare cases.

In this world the `ostree_repo_write_content()` API would
then need to hackily bridge pull to push and it'd be
less efficient.

Anyways for now due to this bifurcation, this API only
works on non-archive repositories, but that's fine for
now because that's what I want for the `ostree-ext-container`
bits.
2021-04-09 21:54:44 +00:00
Colin Walters
fce69cdf70 repo: Add ostree_repo_write_symlink
Continuation of the addition of `ostree_repo_write_regfile_inline()`.
This will be helpful for ostree-rs-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.
2021-04-08 21:10:00 +00:00
Colin Walters
9332955b5f
Merge pull request #2327 from cgwalters/writing-apis
repo: Add ostree_repo_write_regfile_inline
2021-04-08 17:09:11 -04:00
Colin Walters
4e2a14eb0c repo: Add ostree_repo_write_regfile_inline
When working on ostree-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.

Plus the gtk-rs binding API to map from `impl Read` to Gio
https://docs.rs/gio/0.9.1/gio/struct.ReadInputStream.html
requires that the input stream is `Send` but the Rust `tar` API
isn't.

This is only 1/3 of the problem; we also need similar APIs
to directly create a symlink, and to stream large objects via
a push-based API.
2021-04-08 14:57:33 +00:00
Luca Bruno
6dca0e0e29
Merge pull request #2330 from cgwalters/silence-libarchive-warning
tests: Silence a gcc warning
2021-04-08 14:54:42 +00:00
Colin Walters
5d68a99a2f tests: Silence a gcc warning
We can't mix `goto` and `__attribute__((cleanup))`.
2021-04-08 14:45:28 +00:00
Luca Bruno
9154f41fd4
Merge pull request #2324 from cgwalters/test-use-ex
Various patches for tests/inst
2021-04-08 08:10:05 +00:00
Colin Walters
a90d59cf2d build-sys: Remove --enable-experimental-api
It was added for the collections bits, but we made that stable.
It's now just cruft and we're very unlikely to reuse the infrastructure
again.

Motivated by a unit test failure when running from a tarball:
https://github.com/ostreedev/ostree/issues/2313
2021-04-07 19:19:05 +00:00
Colin Walters
4d9e6de46b tests/inst: Update tokio, hyper and nix 2021-04-07 18:00:57 +00:00
Colin Walters
2628637a78 tests/inst: Update rpm-ostree client 2021-04-07 18:00:57 +00:00
Colin Walters
642dcd10ef tests/inst: Update ostree crate 2021-04-07 18:00:57 +00:00
Colin Walters
35d4e657e7 tests: Drop openat override
No longer needed.
2021-04-07 18:00:57 +00:00
Stefan Berger
81d3017463 rofiles-fuse: Enable support for setting and getting xattrs
Enable support for setting and getting xattrs. Allow modifications
to xattrs only on user.ima xattr.

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
Co-authored-by: Colin Walters <walters@verbum.org>
2021-04-05 17:01:58 -04:00
Colin Walters
6d0bb10bf0 cci: Update for buildroot changes
See latest in https://github.com/coreos/fedora-coreos-tracker/blob/master/docs/ci-and-builds.md
2021-03-31 15:11:34 -04:00
Colin Walters
cb0fd2bf93 tests/pull-test: Use mv -f in a few cases
For some reason I'm hitting this locally because some files are mode `0`.
Which is clearly a different bug but anyways we don't want to prompt.
2021-03-31 15:11:34 -04:00
Colin Walters
e9e4b91120 Release 2021.1 2021-03-23 15:23:55 -04:00
Colin Walters
b69a4180b8 tests/inst: Patch to use my PR for openat
Fixes the build.
2021-03-19 21:45:54 +00:00
Colin Walters
407477f191
Merge pull request #2301 from cgwalters/api-require-booted
sysroot: Add _require_booted_deployment() API
2021-03-18 08:46:35 -04:00
Colin Walters
1b28e6041c sysroot: Add _require_booted_deployment() API
This is a common pattern that is replicated both in our code
and in rpm-ostree a lot.  Let's add a canonical API.
2021-03-17 19:55:56 +00:00
Colin Walters
c52a2ff52e tests/inst: cargo fmt 2021-03-17 18:45:17 +00:00
Colin Walters
d11dd7a37b tests/inst: Fix lots of cargo clippy warnings
Prep for doing this in CI.
2021-03-17 17:13:52 +00:00
Colin Walters
857587615d Add an API+CLI to inject metadata for bootable OSTree commits
I was doing some rpm-ostree work and I wanted to compare two
OSTree commits to see if the kernel has changed.  I think
this should be a lot more natural.

Add `ostree commit --bootable` which calls into a new generic
library API `ostree_commit_metadata_for_bootable()` that
discovers the kernel version and injects it as an `ostree.linux`
metadata key.  And for extra clarity, add an `ostree.bootable`
key.

It's interesting because the "core" OSTree layer is all about
generic files, but this is adding special APIs around bootable
OSTree commits (as opposed to e.g. flatpak as well as
things like rpm-ostree's pkgcache refs).

Eventually, I'd like to ensure everyone is using this and
hard require this metadata key for the `ostree admin deploy`
flow - mainly to prevent accidents.
2021-03-12 19:01:42 +00:00
OpenShift Merge Robot
c97aafcda3
Merge pull request #2285 from cgwalters/log-bootloader-stuff
deploy: Add subbootversion to journal
2021-02-28 16:21:15 +01:00
Colin Walters
975496d241 deploy: Add subbootversion to journal
To help debug an issue we've seen where `/boot` isn't
in sync with the `/ostree/boot` dir, let's log to the journal
what we're doing.
2021-02-28 14:47:58 +00:00
OpenShift Merge Robot
ee490bf9ba
Merge pull request #2284 from cgwalters/use-rpmostree-client
tests/inst: Switch to rpmostree-client from git
2021-02-26 01:59:56 +01:00
Colin Walters
dc10bdfb0c tests/inst: Switch to rpmostree-client from git
See discussion in https://github.com/coreos/rpm-ostree/pull/2569#issuecomment-780569188
Currently pinned to a hash, but after the next stable release let's switch to tags
2021-02-23 15:20:54 +00:00
Colin Walters
d49f3291ad Add --enable-sanitizers, fix make check with it
It's cleaner if this is an build option rather than being
kludged into the CI layer.

Notably we can't use `LD_PRELOAD` anymore with ASAN, so update
our tests to check for `ASAN_OPTIONS`.
2021-02-03 20:33:19 +00:00
Luca BRUNO
bf2c23ca06
tests/ext/destructive: enhance test logic
This enhances external-tests logic, ensuring that destructive tests
have retries and some context to pinpoint failures, and that failed-state
services are reset between iterations.
2021-02-03 12:25:02 +00:00
Dan Nicholson
d7f2955f37 pull: Fix local pull with depth and truncated source history
The local pull path was erroring on any missing commit, but that
prevents a depth pull where the source repo has truncated history. As in
the remote case, this also tries to pull in a tombstone commit if the
source repo supports it.

Fixes: #2266
2021-01-12 14:19:01 -07:00
Dan Nicholson
20047ff1fe pull: Error on depth pull with missing head commit
When pulling with depth, missing parent commits are ignored. However,
the check was applying to any commit, which means that it would succeed
even if the requested commit was missing. This might happen on a
corrupted remote repo or when using ref data from a stale summary.

To achieve this, the semantics of the `commit_to_depth` hash table is
changed slightly to only ever includes parent commits. This makes it
easy to detect when a parent commit is being referenced (although there
is a minor bug there when multiple refs are being pulled) while keeping
references to commits that need their `commitpartial` files cleaned up.
It also means that the table is only populated on depth pulls, which
saves some memory and processing in the common depth=0 case.

Fixes: #2265
2021-01-12 14:19:01 -07:00
Dan Nicholson
b4f06b47a3 tests: Ensure no dangling commit partials on remote depth pull
This was already being done on the local depth pull test, so this just
adds the matching logic to the remote depth pull test.
2021-01-11 16:07:29 -07:00
Dan Nicholson
4db2ba0eb1 pull: Allow disabling commit binding verification
In some cases such as backups or mirroring you may want to pull commits
from one repo to another even if there commits that have incorrect
bindings. Fixing the commits in the source repository to have correct
bindings may not be feasible, so provide a pull option to disable
verification.

For Endless we have several repositories that predate collection IDs and
ref bindings. Later these repositories gained collection IDs to support
the features they provide and ref bindings as the ostree tooling was
upgraded. These repositories contain released commits that were valid to
the clients they were targeting at the time. Correcting the bindings is
not really an option as it would mean invalidating the repository
history.
2020-12-17 14:07:08 -07:00