IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Introduces an intermediate format for overlayfs storage, where
.wh-ostree. prefixed files will be converted into char 0:0
whiteout devices used by overlayfs to mark deletions across layers.
The CI scripts now uses a volume for the scratch directories
previously in /var/tmp otherwise we cannot create whiteout
devices into an overlayfs mounted filesystem.
Related-Issue: #2712
This is a follow up to
0ced9fde76
"sysroot: Support /boot on root or as seperate filesystem for syslinux and u-boot"
What we should have done at the time is changed our bootloader entries
to be prefixed with `/boot`. This means that the GRUB2 BLS support
will Just Work.
For now, I'm making this option default to off out of a lot of
conservatism. I think in the future we should flip this on by default.
If `/boot` is an automount, then the unit will be stopped as soon as the
automount expires. That's would defeat the purpose of using systemd to
delay finalizing the deployment until shutdown. This is not uncommon as
`systemd-gpt-auto-generator` will create an automount unit for `/boot`
when it's the EFI System Partition and there's no fstab entry.
To ensure that systemd doesn't stop the service early when the `/boot`
automount expires, introduce a new unit that holds `/boot` open until
it's sent `SIGTERM`. This uses a new `--hold` option for
`finalize-staged` that loads but doesn't lock the sysroot. A separate
unit is used since we want the process to remain active throughout the
finalization run in `ExecStop`. That wouldn't work if it was specified
in `ExecStart` in the same unit since it would be killed before the
`ExecStop` action was run.
Fixes: #2543
are pending
This is to support pending deployments instead of rasing assertion.
For example:
```
$ sudo rpm-ostree kargs --append=foo=bar
$ sudo ostree admin kargs edit-in-place --append-if-missing=foobar
```
After reboot we get both `foo=bar foobar`.
Fix https://github.com/ostreedev/ostree/issues/2679
In the current "ostree native container" flow, we're inserting
a commit object into the repo but with no refs.
We have hacks in a few places to find the commit digest via e.g.
`find repo/objects -name *.commit` but that's a horrible hack.
Add `ostree rev-parse --single` which will print the single commit,
and error out if there is not exactly one commit.
Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>
Don't parse `rpm-ostree status` output, it's not meant for that. Use
`--json` output instead.
While we're here, fix an obsolete reference to Ansible.
Related: https://github.com/coreos/rpm-ostree/pull/3938
https://github.com/coreos/coreos-assembler/pull/2921 broke this
test which is intentionally causing a systemd unit to fail.
As they say, necessity is the mother of invention. They don't
say though that need always causes particularly *beautiful* things
to be invented...
update all existing deployments in place
Example:
$ sudo ostree admin kargs edit-in-place --append-if-missing=rw
See https://github.com/ostreedev/ostree/issues/2617
This will not add duplicate key, if there is `TESTARG=VAL1` in the
kernel arguments, `--append-if-missing=TESTARG=VAL2` will be ignored.
Not all filesystems support extended attributes. This test uses
/var/tmp to try to get an extended-attributes-capable filesystem,
but that might not succeed.
Signed-off-by: Simon McVittie <smcv@debian.org>
No real changes.
```
$ cargo fix --edition
note: Switching to Edition 2021 will enable the use of the version 2 feature resolver in Cargo.
This may cause some dependencies to be built with fewer features enabled than previously.
More information about the resolver changes may be found at https://doc.rust-lang.org/nightly/edition-guide/rust-2021/default-cargo-resolver.html
When building the following dependencies, the given features will no longer be used:
libc v0.2.126 removed features: extra_traits
The following differences only apply when building with dev-dependencies:
getrandom v0.2.6 removed features: std
```
which looks OK to me.
I messed this up; the last release should inherit from the previous
release (N-1) and not the previous to that (N-2).
I think (hope) this isn't an ABI break...
Just noticed this when I was going to add a new symbol.
I'm aiming to do some more work on the Rust side around `fsck`
like functionality, and this is a useful primitive. There isn't
a great Rust crate for xattrs, and I think it's better to share this
code.
Fix up the paths for the crates now that the Rust bindings are in
`rust/`.
We can't today include the test suite because it depends on `ostree-rs-ext`
which would make everything circular.
(Building that now requires a separate `cd tests/inst && cargo build`)
Quite a while ago we added staged deployments, which solved
a bunch of issues around the `/etc` merge. However...a persistent
problem since then is that any failures in that process that
happened in the *previous* boot are not very visible.
We ship custom code in `rpm-ostree status` to query the previous
journal. But that has a few problems - one is that on systems
that have been up a while, that failure message may even get
rotated out. And second, some systems may not even have a persistent
journal at all.
A general thing we do in e.g. Fedora CoreOS testing is to check
for systemd unit failures. We do that both in our automated tests,
and we even ship code that displays them on ssh logins. And beyond
that obviously a lot of other projects do the same; it's easy via
`systemctl --failed`.
So to make failures more visible, change our `ostree-finalize-staged.service`
to have an internal wrapper around the process that "catches" any
errors, and copies the error message into a file in `/boot/ostree`.
Then, a new `ostree-boot-complete.service` looks for this file on
startup and re-emits the error message, and fails.
It also deletes the file. The rationale is to avoid *continually*
warning. For example we need to handle the case when an upgrade
process creates a new staged deployment. Now, we could change the
ostree core code to delete the warning file when that happens instead,
but this is trying to be a conservative change.
This should make failures here much more visible as is.
On systems where `coreutils` are built with `--enable-single-binary=symlinks` like Nix,
`/usr/bin/env` is symlinked to `/usr/bin/coreutils` and uses `argv[0]` to determine which program to run.
Since the `test-cli-extensions.sh` created a new symlink named `ostree-env`,
coreutils would be confused about the utility to choose, so running it would fail:
ostree-env: unknown program ‘ostree-env’
Try 'ostree-env --help' for more information.
Fixes: https://github.com/ostreedev/ostree/issues/2553
I was reading this thread
https://internals.rust-lang.org/t/from-life-before-main-to-common-life-in-main/16006/30
and that reminded me about this code, which it turns out actually
doesn't compile with my default local cargo config:
```
$ cat ~/.cargo/config
[target.x86_64-unknown-linux-gnu]
rustflags = ["-Ctarget-cpu=native", "-C", "link-arg=-fuse-ld=lld"]
[profile.release]
incremental = true
$ cargo b
...
error: linking with `cc` failed: exit status: 1
|
= note: "cc" "-m64" "/var/srv/walters/src/github/ostreedev/ostree/target/debug/deps/ostree_test-4ca8e730f9dc6ffc.10325uqlhkyr5uol.rcgu.o" "/var/srv/walte"
= note: ld.lld: error: undefined symbol: __start_linkme_NONDESTRUCTIVE_TESTS
>>> referenced by 22nn09lfsklfqvyy
>>> /var/srv/walters/src/github/ostreedev/ostree/target/debug/deps/ostree_test-4ca8e730f9dc6ffc.22nn09lfsklfqvyy.rcgu.o:(ostree_tes)
```
For now let's just go back to having a static list of functions.
We don't have *too* many of those.
If we use head(1) to take only the first two lines, then cut(1) and
earlier pipeline entries are killed by SIGPIPE (if they have not already
terminated), and that's flagged as an error under `set -o pipefail`.
Use an equivalent sed command to take exactly the second line, but
without SIGPIPE.
Signed-off-by: Simon McVittie <smcv@debian.org>
Gbp-Pq: Name test-prune-Read-to-the-end-of-cut-1-output.patch
This prevents writing content into 'bare-split-xattrs` repository,
while carving some space for experimenting via a temporary
`OSTREE_EXP_WRITE_BARE_SPLIT_XATTRS` environment flag.
The commit metadata `version` key is well established but getting it for
a remote commit is cumbersome since the commit object needs to be
fetched and loaded. Including it in the summary additional metadata
allows a much more convenient view of what each of the remote refs
represents.
`kola` now follows symlinks when archiving an external test's `data/`
dir. So the recursive `data` symlink we have here breaks it.
Let's just move the shared files in its own directory and update the
symlinks.
Support for that file was added previously, but the testing lived in
rpm-ostree only. Let's add it here too.
In the process add a hidden `--lock-finalization` to `ostree admin
deploy` to make testing easier (though it could also be useful to update
managers driving OSTree via the CLI).
In fixing https://github.com/coreos/rpm-ostree/pull/3323
I felt that it was a bit ugly we're installing `/usr/bin/ostree-container`.
It's kind of an implementation detail. We want users to use
`ostree container`.
Let's support values outside of $PATH too.
For example, this also ensures that TAB completion for `ost` expands
to `ostree ` with a space.
Basically due to the glib structured logging rework we lost the
`noreturn` attribute on `g_error()`.
This is fixed in glib as of f97ff20adf
But we might as well just throw an error here.
This updates the test logic for CLI extensions, actually checking
for functional output from the subcommand.
It also cleans up some environmental leftover.
This adds some logic to detect and dispatch unknown subcommands to
extensions available in `$PATH`. Additional commands can be
implemented by adding relevant `ostree-$verb` binaries to the system.
As an example, if a `/usr/bin/ostree-extcommand` extension is provided,
the execution of `ostree extcommand --help` will be dispatched to that
as `ostree-extcommand extcommand --help`.
This reworks the var-mount destructive test in order to properly use
the datadir for the current stateroot instead of a duplicated one.
In turn, it ensures that the resulting `var.mount` after reboot is
correctly pointing to the same location which hosted `/var` on the
previous boot.
https://bugzilla.redhat.com/show_bug.cgi?id=1945274 is an issue where a privileged
kubernetes daemonset is writing a socket into `/etc`. This makes ostree upgrades barf.
Now, they should clearly move it to `/run`. However, one option is for us to
just ignore it instead of erroring out. Some brief investigation shows that
e.g. `git add somesocket` is a silent no-op, which is an argument in favor of ignoring it.
Closes: https://github.com/ostreedev/ostree/issues/2446
The logic for `--selinux-policy` ended up in the `--tree=dir`
path, but there's no reason for that. Fix the imported
labeling with `--tree=tar`. Prep for use with containers.
We had this bug because the previous logic was trying to avoid
duplicating the code for generic `--selinux-policy` and
the case of `--selinux-policy-from-base --tree=dir`.
It's a bit more code, but it's cleaner if we dis-entangle them.
Having to touch a global test counter when adding tests is
a recipe for conflicts between PRs.
The TAP protocol allows *ending* with the expected number of
tests, so the best way to do this is to have an explicit
API like our `tap_ok` which bumps a counter, then end with `tap_end`.
I ported one test as a demo.
This will be helpful for the "ostree native container" work in
https://github.com/ostreedev/ostree-rs-ext/
Basically in order to reuse GPG/signapi verification, we need
to support adding a remote, even though it can't be used via
`ostree pull`. (At least, not until we merge ostree-rs-ext into ostree, but
even then I think the principle stands)
We're waaay overdue for this, it's been the default
in rpm-ostree for years, and solves several important bugs
around not capturing `/etc` while things are running.
Also, `ostree admin upgrade --stage` (should) become idempotent.
Closes: https://github.com/ostreedev/ostree/issues/2389
There are some existing issues around fsck in unprivileged bare mode,
so this test does not really work at the moment. Leaving it as a FIXME
for the moment.
It cannot work to use `--no-xattrs` when SELinux is enabled
because we get a `security.selinux` attribute on created files
regardless. So just skip this test if true.
Also add some `ostree fsck`s in here which helped me debug
this.
We have a bunch of APIs to do GPG verification of a commit,
but that doesn't generalize to signapi. Further, they
require the caller to check the signature status explicitly
which seems like a trap.
This much higher level API works with both GPG and signapi.
The intention is to use this in things that are doing "external
pulls" like the ostree-ext tar import support. There we will
get the commitmeta from the tarball and we want to verify it
at the same time we import the commit.
As pointed out in the original review, `gpg-list-keys` fits better
alongside the existing `gpg-import`.
Changes were done with:
```
git grep -l list-gpg-keys | xargs sed -i 's/list-gpg-keys/gpg-list-keys/'
for src in $(git ls-files '*list-gpg-keys*'); do
dst=${src/list-gpg-keys/gpg-list-keys}
git mv "$src" "$dst"
done
```
Some distributions set this during build in order to have reproducible
builds from the same source code: for example, Debian uses the date
from debian/changelog.
However, some of our tests assume that `ostree commit` will result in
a commit with the current date/time, and SOURCE_DATE_EPOCH breaks that
assumption. Unset it for our build-time tests.
Resolves: https://github.com/ostreedev/ostree/issues/2405
Signed-off-by: Simon McVittie <smcv@collabora.com>
This provides a wrapper for the `ostree_repo_remote_get_gpg_keys`
function to show the GPG keys associated with a remote. This is
particularly useful for validating that GPG key updates have been
applied. Tests are added, which checks the
`ostree_repo_remote_get_gpg_keys` API by extension.
If we fail as a result of `set -x`, It's often not completely obvious
which command failed or how. Use a trap on ERR to show the command that
failed, and its exit status.
Signed-off-by: Simon McVittie <smcv@collabora.com>
[Originally from bubblewrap commits c5c999a7 "tests: test --userns"
and 3e5fe1bf "tests: Better error message if assert_files_equal fails";
separated into this commit by Simon McVittie.]
We struggled for a long time with enablement of our "internal units",
trying to follow the philosophy that units should only be enabled
by explicit preset.
See https://bugzilla.redhat.com/show_bug.cgi?id=1451458
and https://github.com/coreos/rpm-ostree/pull/1482
etc.
And I just saw chat (RH internal on a proprietary system sadly) where
someone hit `ostree-remount.service` not being enabled in CentOS8.
Thinking about this more, I realized we've shipped a systemd generator
for a long time and while its only role until now was to generate `var.mount`,
but by using it to force on our internal units, we don't require
people to deal with presets anymore.
Basically we're inverting things so that "if ostree= is on the kernel
cmdline, then enable our units" and not "enable our units, but have
them use ConditionKernelCmdline=ostree to skip".
Drop the weird gyrations we were doing around `ostree-finalize-staged.path`
too; forking `systemctl start` is just asking for bugs.
So after this, hopefully we won't ever again have to think about
distribution presets and our units.
This will be ignored, so let's make it very clear
people are doing something wrong. Motivated by a bug
in a build pipeline that injected `/var/lib/rpm` into an ostree
commit which ended up crashing rpm-ostree because it was an empty db
which it wasn't expecting.
It *also* turns out rpm-ostree is incorrectly dumping content in the
deployment `/var` today, which is another bug.
The semantics of multiple process locking are covered by
test-concurrency.py, but the semantics of the repository locking from a
single process aren't handled there.
This checks how the repository locking is handled from a single thread
with one OstreeRepo, a single thread with multiple OstreeRepos, and
multiple threads sharing an OstreeRepo.
If there's a locking issue in this test, then it's likely not going to
resolve after a few seconds of serializing access. Lower the default 30
second lock timeout to 5 seconds to prevent the test from hanging
unnecessarily.
This simplifies the lock state management considerably since the
previously pushed type doesn't need to be tracked. Instead, 2 counters
are kept to track how many times each lock type has been pushed. When
the number of exclusive locks drops to 0, the lock transitions back to
shared.
In gnupg 2.3.0[1], if a primary key is expired and a subkey does not
have an expiration or its expiration is older than the primary key, the
subkey's expiration will be reported as the primary's. Previously a
subkey without an expiration would not report one regardless of the
primary key's expiration.
This caused a regression in a test setting an expiration on a primary
key. The test was checking that the subkey was not expired by asserting
that there was no `Key expired` line in the signature verification
output. With gnupg 2.3.0+, it will show as expired, causing the test to
fail.
Remove the assertion since it's not consistent across gnupg versions. In
practice we don't care whether the subkey is considered expired or not
as long as the signature verification fails when the primary key is
expired.
1. https://dev.gnupg.org/T3343Fixes: #2359
Previous to this we'd trip an assertion `abort()` deep in the curl code if e.g.
a user did `ostree remote add foo htttp://...` etc.
Motivated by considering supporting "external remotes" where code outside
ostree does a pull, but we want to reuse the signing verification infrastructure.
Several tests generate summaries and then expect to use the generated
summary immediately. However, this can cause intermittent test failures
when they inadvertantly get a cached summary file. This typically
happens when the test is run on a filesystem that doesn't support user
extended attributes. In that case, the caching code can only use the
last modified time, which only has 1 second granularity. If tests don't
carefully manage the summary modification times or the repo cache then
they are likely subject to races in some test environments.
This introduces an environment variable `OSTREE_SKIP_CACHE` that
prevents the repo from using a cache directory. This is enabled by
default in tests and disabled for tests that are a explicitly trying to
test the caching behavior.
Fixes: #2313Fixes: #2351
If we fail as a result of `set -x`, It's often not completely obvious
which command failed or how. Use a trap on ERR to show the command that
failed, and its exit status.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Ideally in the future we change more of our unit tests to
support running installed; we've tried this in the past with
https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests
I'd like to pick that back up again. This takes a step
towards that by having our Rust tests.
To make this even easier, add a `tests/run-installed`
which runs the installed tests (uninstalled, confusingly
but conveniently for now).
rust-analyzer is happier with this because it understands
the project structure out of the box.
We aren't actually again adding a dependency on Rust/cargo in the core,
this is only used to make `cargo build` work out of the box to build
the Rust test code.
This API is push rather than pull, which makes it much more
suitable to use cases like parsing a tar file from external
code.
Now, we have a large mess in this area internally because
the original file writing code was pull based, but static
deltas hit the same problem of wanting a push API, so I added
this special `OstreeRepoBareContent` just for writing regular
files from a push API.
Eventually...I'd like to deprecate the pull based API,
and rework things so that for regular files the push API
is the default, and then `write_content_object()` would
be split up into archive/bare cases.
In this world the `ostree_repo_write_content()` API would
then need to hackily bridge pull to push and it'd be
less efficient.
Anyways for now due to this bifurcation, this API only
works on non-archive repositories, but that's fine for
now because that's what I want for the `ostree-ext-container`
bits.
Continuation of the addition of `ostree_repo_write_regfile_inline()`.
This will be helpful for ostree-rs-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.
When working on ostree-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.
Plus the gtk-rs binding API to map from `impl Read` to Gio
https://docs.rs/gio/0.9.1/gio/struct.ReadInputStream.html
requires that the input stream is `Send` but the Rust `tar` API
isn't.
This is only 1/3 of the problem; we also need similar APIs
to directly create a symlink, and to stream large objects via
a push-based API.
It was added for the collections bits, but we made that stable.
It's now just cruft and we're very unlikely to reuse the infrastructure
again.
Motivated by a unit test failure when running from a tarball:
https://github.com/ostreedev/ostree/issues/2313
Enable support for setting and getting xattrs. Allow modifications
to xattrs only on user.ima xattr.
Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
Co-authored-by: Colin Walters <walters@verbum.org>
I was doing some rpm-ostree work and I wanted to compare two
OSTree commits to see if the kernel has changed. I think
this should be a lot more natural.
Add `ostree commit --bootable` which calls into a new generic
library API `ostree_commit_metadata_for_bootable()` that
discovers the kernel version and injects it as an `ostree.linux`
metadata key. And for extra clarity, add an `ostree.bootable`
key.
It's interesting because the "core" OSTree layer is all about
generic files, but this is adding special APIs around bootable
OSTree commits (as opposed to e.g. flatpak as well as
things like rpm-ostree's pkgcache refs).
Eventually, I'd like to ensure everyone is using this and
hard require this metadata key for the `ostree admin deploy`
flow - mainly to prevent accidents.
It's cleaner if this is an build option rather than being
kludged into the CI layer.
Notably we can't use `LD_PRELOAD` anymore with ASAN, so update
our tests to check for `ASAN_OPTIONS`.
This enhances external-tests logic, ensuring that destructive tests
have retries and some context to pinpoint failures, and that failed-state
services are reset between iterations.
The local pull path was erroring on any missing commit, but that
prevents a depth pull where the source repo has truncated history. As in
the remote case, this also tries to pull in a tombstone commit if the
source repo supports it.
Fixes: #2266
When pulling with depth, missing parent commits are ignored. However,
the check was applying to any commit, which means that it would succeed
even if the requested commit was missing. This might happen on a
corrupted remote repo or when using ref data from a stale summary.
To achieve this, the semantics of the `commit_to_depth` hash table is
changed slightly to only ever includes parent commits. This makes it
easy to detect when a parent commit is being referenced (although there
is a minor bug there when multiple refs are being pulled) while keeping
references to commits that need their `commitpartial` files cleaned up.
It also means that the table is only populated on depth pulls, which
saves some memory and processing in the common depth=0 case.
Fixes: #2265
In some cases such as backups or mirroring you may want to pull commits
from one repo to another even if there commits that have incorrect
bindings. Fixing the commits in the source repository to have correct
bindings may not be feasible, so provide a pull option to disable
verification.
For Endless we have several repositories that predate collection IDs and
ref bindings. Later these repositories gained collection IDs to support
the features they provide and ref bindings as the ostree tooling was
upgraded. These repositories contain released commits that were valid to
the clients they were targeting at the time. Correcting the bindings is
not really an option as it would mean invalidating the repository
history.
If this is not done, the test can fail when the temporary directory is
a tmpfs: for example this happens during build-time testing with /var/tmp
on tmpfs or TEST_TMPDIR pointing to a tmpfs, or installed-tests with
gnome-desktop-testing-runner allocating the test directory on a tmpfs.
In particular, many of Debian's official autobuilders now do the entire
build and test procedure in a chroot hosted on a tmpfs, to improve build
performance and prevent fsync overhead.
In this situation, it appears that overwriting summary.sig with a copy
of summary.sig.2 is not sufficient for the web server to tell the
libostree client that it needs to be re-downloaded. I'm not completely
sure why, because tmpfs does appear to have sub-second-resolution
timestamps, but forcing a distinct mtime is certainly enough to
resolve it.
Resolves: https://github.com/ostreedev/ostree/issues/2245
Bug-Debian: https://bugs.debian.org/975418
Signed-off-by: Simon McVittie <smcv@collabora.com>
I've made this use functions to make it easier to add support for more
bootloaders. Seeing as there will be a big diff anyway I've also adjusted
the formatting to make it pep8 compliant.
This makes it testable, and increases its test coverage too 100% of
lines, as measured by `make coverage`.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
This test would have actually passed before the summary file caching
changes (in the previous few commits) were added, as the `summary.sig`
essentially acted as the ETag for the summary file, and itself wasn’t
updated on disk if it didn’t change when querying the server.
Actually testing that the HTTP caching headers are working to reduce
HTTP traffic would require test hooks into the pull code or the
trivial-httpd server, neither of which I have the time to add at the
moment.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
Alternative to https://github.com/ostreedev/ostree/pull/2197
Python's (usually) zero-sized `__init__.py` files can provoke
us hitting the hardlink limits on some filesystems (`EMLINK`).
At least one Fedora rpm-ostree user hit this.
The benefits of hardlinking here are quite marginal; lots
of hardlinks can behave suboptimally in particular filesystems
like BTRFS too.
This builds on prior code which made this an option, introduced
in 673cacd633
Now we just do it uncondtionally.
Also this provoked a different bug in a very obscure user mode checkout
case; when the "real" permissions were different from the "physical"
permissions, we would still hardlink. Fix the test case for this.
Currently, they are set in the `config` file and cause that to be
downloaded on every pull. Given that the client is already pulling the
`summary` file, it makes sense to avoid an additional network round trip
and cache those options in the `summary` file.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Helps: #2165
In FCOS and RHCOS, the need to configure software in the initramfs has
come up multiple times. Sometimes, using kernel arguments suffices.
Other times, it really must be a configuration file. Rebuilding the
initramfs on the client-side however is a costly operation. Not only
does it add complexity to the update workflow, it also erodes a lot of
the value obtained from using the baked "blessed" initramfs from the
tree itself.
One elegant way to address this is to allow specifying multiple
initramfses. This is supported by most bootloaders (notably GRUB) and
results in each initrd being overlayed on top of each other.
This patch allows libostree clients to leverage this so that they can
avoid regenerating the initramfs entirely. libostree itself is agnostic
as to what kind and how much data overlay initrds contain. It's up to
the clients to enforce such boundaries.
To implement this, we add a new ostree_sysroot_stage_overlay_initrd
which takes a file descriptor and returns a checksum. Then users can
pass these checksums when calling the deploy APIs via the new array
option `overlay_initrds`. We copy these files into `/boot` and add them
to the BLS as another `initrd` entry.
When support for devicetree was added, it created a problem
because old and new ostree versions would compute different
checksums for the "boot data". The scenario here is:
- Have system with ostree < 2020.4
- Reboot into system with ostree 2020.5
- Try to perform an operation that would retain
that previous booted deployment (common)
Currently ostree iterates over all the deployments
that will be retained and calls `install_deployment_kernel()`,
even for the booted one (which is a bit silly), but
just to verify that all boot data for the targeted
deployments are installed.
This then re-computes the checksum and we'd trip this
assertion.
In practice though, we don't strictly require them to match;
the only thing that will happen if they don't is that we'll
end up with another copy of the kernel/initramfs - and
that only temporarily until the previous deployment
gets GC'd.
Longer term, I think what we really want to do anyways
is probably closer to like a little ostree repo for `/boot`
so that we can e.g. still hardlink kernels there even if
the initramfs changes, or hardlink both kernel/initramfs
if just the devicetree changes, etc.
Closes: https://github.com/ostreedev/ostree/issues/2154
The extreme special case of "zero mode" files like `/etc/shadow`
comes up again. What we want is for "user mode" checkouts to
override it to make the file readable; otherwise when operating
as non-root without `CAP_DAC_OVERRIDE` it becomes very difficult
to work with.
Previously, we were hardlinking these files, but then it intersects
with *another* special case around zero sized files, which is
*also* true for `/etc/shadow`.
Trying to avoid hardlinking there unveiled this bug - when
we go to do a copy checkout, we need to override the mode.
And I made a few more API tweaks, such as supporting `Path`
objects directly and also not needing e.g. `commit = commit`, see
- cfa7c71126
- 679bce4cc7
This adds infrastructure to the Rust test suite for destructive
tests, and adds a new `transactionality` test which runs
rpm-ostree in a loop (along with `ostree-finalize-staged`) and
repeatedly uses either `kill -9`, `reboot` and `reboot -ff`.
The main goal here is to flush out any "logic errors".
So far I've validated that this passes a lot of cycles
using
```
$ kola run --qemu-image=fastbuild-fedora-coreos-ostree-qemu.qcow2 ext.ostree.destructive-rs.transactionality --debug --multiply 8 --parallel 4
```
a number of times.
I was thinking a bit more recently about the "live" changes
stuff https://github.com/coreos/rpm-ostree/issues/639
(particularly since https://github.com/coreos/rpm-ostree/pull/2060 )
and I realized reading the last debates in that issue that
there's really a much simpler solution; do exactly the same
thing we do for `ostree admin unlock`, except mount it read-only
by default.
Then, anything that wants to modify it does the same thing
libostree does for `/sysroot` and `/boot` as of recently; create
a new mount namespace and do the modifications there.
The advantages of this are numerous. First, we already have
all of the code, it's basically just plumbing through a new
entry in the state enumeration and passing `MS_RDONLY` into
the `mount()` system call.
"live" changes here also naturally don't persist, unlike what
we are currently doing in rpm-ostree.
We were committing the whole tempdir, which seems to fail
in Travis because the GPG agent Unix domain socket ends up there too,
and ostree refuses to commit sockets.
We use a similar trick to having a `sysroot -> .` symlink on the real root
here to support both /boot on root as well as on a separate filesystem. No
matter how it's mounted `/boot/xyz` will always refer to the file you'd
expect.
This is nicer than my previous attempts at this because there's no
configuration nor auto-detection required.
This is the dual of 1f3c8c5b3d
where we output more detail when signapi fails to validate.
Extend the API to return a string for success, which we output
to stdout.
This will help the test suite *and* end users validate that the expected
thing is happening.
In order to make this cleaner, split the "verified commit" set
in the pull code into GPG and signapi verified sets, and have
the signapi verified set contain the verification string.
We're not doing anything with the verification string in the
pull code *yet* but I plan to add something like
`ostree pull --verbose` which would finally print this.
To aid debuggability, when we find a commit that isn't signed
by our expected key, output a specific error message with the
key.
(And then add code to switch to just printing the count beyond 3
because the test suite injects 100 keys and hopefully no one
ever actually does that)
I'm thinking about adding an implementation of ed25519 signatures
with OpenSSL (so we can ship the feature with Fedora CoreOS
without requiring an additional library) and in preparation for
that it's essential that we validate that libsodium-generated
signatures and OpenSSL-generated signatures are compatible.
I don't know if they are yet actually, but the goal of this
new test is to add a pre-generated repository with a signed
commit generated by libsodium.
This will catch if e.g. there's ever a change in libsodium,
or if existing libsodium implementation versions (e.g. the
one in Debian) might differ from what we ship here.
Add support for a devicetree directory at /usr/lib/modules/$kver/dtb/.
In ARM world a general purpose distribution often suppports multiple
boards with a single operating system. However, OSTree currently only
supports a single device tree, which does not allow to use the same
OSTree on different ARM machines. In this scenario typically the boot
loader selects the effective device tree.
This adds device tree directory support for the new boot artefact
location under /usr/lib/modules. If the file `devicetree` does not
exist, then the folder dtb will be checked. All devicetrees are hashed
into the deployment hash. This makes sure that even a single devicetree
change leads to a new deployment and hence can be rolled back.
The loader configuration has a new key "devicetreepath" which contains
the path where devicetrees are stored. This is also written to the
U-Boot variable "fdtdir". The boot loader is expected to use this path
to load a particular machines device tree from.
Closes: #1900
Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
