IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
See https://github.com/ostreedev/ostree/issues/1593
Basically this makes it easier for people packaging, as the trivial-httpd
is only for tests, and this way the binary will live with the tests.
Also at this point nothing should depend on `ostree trivial-httpd`.
I think this just accidentally was never enabled.
While looking at the code, add a sleep here to be resilient to
filesystems with only second mtime granularity.
I am not sure why this is failing on older Debian systems,
but I'm wildly guessing that something being done in `libtest.sh`
is setting up automake in a way that we need. This is done
in other tests.
Or maybe it's the missing `$CMD_PREFIX`? Let's see...
I think there's never been a real race condition here. Instead
the problem is:
- We have a timeout on the lock, after which we error out (30s)
- This test actually *lowers* the timeout
Looking through the failures for test-concurrency what I see
is
`error: Locking repo exclusive failed: Resource temporarily unavailable`
which is us hitting the timeout.
Hardcoded timeouts are just going to be subject to race conditions.
I understand not wanting to block forever in some cases, but any
arbitrary timeout is just going to get hit in real world conditions
too.
Anyways for now, stop shooting ourselves in the foot and at least
keep the timeout at the default.
Closes: https://github.com/ostreedev/ostree/issues/2038
The `f_bfree` member of the `statvfs` struct is documented as the
"number of free blocks". However, different filesystems have different
interpretations of this. E.g. on XFS, this is truly the number of blocks
free for allocating data. On ext4 however, it includes blocks that
are actually reserved by the filesystem and cannot be used for file
data. (Note this is separate from the distinction between `f_bfree` and
`f_bavail` which isn't relevant to us here since we're privileged.)
If a kernel and initrd is sized just right so that it's still within the
`f_bfree` limit but above what we can actually allocate, the early prune
code won't kick in since it'll think that there is enough space. So we
end up hitting `ENOSPC` when we actually copy the files in.
Rework the early prune code to instead use `fallocate` which guarantees
us that a file of a certain size can fit on the filesystem. `fallocate`
requires filesystem support, but all the filesystems we care about for
the bootfs support it (including even FAT).
(There's technically a TOCTOU race here that existed also with the
`statvfs` code where free space could change between when we check
and when we copy. Ideally we'd be able to pass down that fd to the
copying bits, but anyway in practice the bootfs is pretty much owned by
libostree and one doesn't expect concurrent writes during a finalization
operation.)
I've deprecated sh-inline; in the end I think it is better
to minimize the amount of bash code we have. xshell solves
the core convenience problem of taking local variables and mapping
them to command arguments.
A full port would be nontrivial; this just starts the ball
rolling.
During the early design of FCOS and RHCOS, we chose a value of 384M
for the boot partition. This turned out to be too small: some arches
other than x86_64 have larger initrds, kernel binaries, or additional
artifacts (like device tree blobs). We'll likely bump the boot partition
size in the future, but we don't want to abandon all the nodes deployed
with the current size.[[1]]
Because stale entries in `/boot` are cleaned up after new entries are
written, there is a window in the update process during which the bootfs
temporarily must host all the `(kernel, initrd)` pairs for the union of
current and new deployments.
This patch determines if the bootfs is capable of holding all the
pairs. If it can't but it could hold all the pairs from just the new
deployments, the outgoing deployments (e.g. rollbacks) are deleted
*before* new deployments are written. This is done by updating the
bootloader in two steps to maintain atomicity.
Since this is a lot of new logic in an important section of the
code, this feature is gated for now behind an environment variable
(`OSTREE_ENABLE_AUTO_EARLY_PRUNE`). Once we gain more experience with
it, we can consider turning it on by default.
This strategy increases the fallibility of the update system since one
would no longer be able to rollback to the previous deployment if a bug
is present in the bootloader update logic after auto-pruning (see [[2]]
and following). This is however mitigated by the fact that the heuristic
is opportunistic: the rollback is pruned *only if* it's the only way for
the system to update.
[1]: https://github.com/coreos/fedora-coreos-tracker/issues/1247
[2]: https://github.com/ostreedev/ostree/issues/2670#issuecomment-1179341883Closes: #2670
When hacking and testing locally with `cosa build-fast` and `kola run`,
I prefer to leave testing framework stuff within the work directory
rather than installed in my pet container. Add a `localinstall` target
for this which puts the tests in `tests/kola`. Then a simple `kola run`
will pick it up.
AFAICT, I don't see how `runkola.sh` or the Makefile in `tests/kolainst`
can create files in `tests/kola` since it's geared towards installing
under `/usr`.
g_steal_fd() exists in GLib since 2.70, and libglnx has a backport for
older GLib versions, equivalent to the libglnx-specific glnx_steal_fd().
Signed-off-by: Simon McVittie <smcv@collabora.com>
XFS now seems to want filesystems larger than 300MB, so switch
to ext4. Also use `20MiB` so we align to 512b sectors to squash
a `losetup` warning.
Also tweak some of the numbers to still work.
The only other way to get the remote ref revision from the CLI is to
scrape the output of `ostree remote summary` or pull the commit. The
revision is already there in the summary's ref map, so might as well add
an option to show it.
Allow printing the revision along with the ref. This is very convenient
for looping over the refs in a shell as well as for quickly seeing which
refs are pointed to the same commit.
My editor started following the configuration in .editorconfig and is
applying this rule to many files I'm editing. Let's just get this over
with and strip everything. This was done like so:
git ls-files | grep '\.[ch]$' | xargs sed -ri 's/\s+$//'
When we calculate the reachability set in `ostree prune`, we do this
without any locking. This means that between the time we build the set
and when we call `ostree_repo_prune_from_reachable`, new content
might've been added. This then causes us to immediately prune that
content since it's not in the now outdated set.
Fix this by calculating the set under an exclusive lock.
I think this is what happened in
https://github.com/fedora-silverblue/issue-tracker/issues/405. While
the pruner was running, the `new-updates-sync` script[1] was importing
content into the repo. The newly imported commits were immediately
deleted by the many `ostree prune --commit-only` calls the pruner does,
breaking the refs.
[1] https://pagure.io/fedora-infra/ansible/blob/35b35127e444/f/roles/bodhi2/backend/files/new-updates-sync#_18
musl uses "Result not representable" for ERANGE, support this in
addition to glibc's "Numerical result out of range".
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
IMO, tools invoking `sudo` automatically is a huge anti-pattern.
Require this to be opt-in.
In practice also...these tests really want to be *integration*
and not unit tests because involving `sudo` alongside uninstalled
binaries just creates a giant mess.
Closes: https://github.com/ostreedev/ostree/issues/2761
g_memdup() is subject to an integer overflow on 64-bit machines if the
object being copied is larger than UINT_MAX bytes. I suspect none of
these objects can actually be that large in practice, but it's easier
to replace all the calls than it is to assess whether we need to
replace them.
A backport in libglnx is used on systems where GLib is older than 2.68.x.
Signed-off-by: Simon McVittie <smcv@collabora.com>
This was only there to reproduce a bug we hit long ago
with bootloader file ordering. We're extremely unlikely
to reintroduce such a bug, and it's not worth carrying around
this code.
As noted in readdir(3), in the presence of long filenames it is
possible for a directory entry to be larger than `sizeof (struct dirent)`.
Copy the full length instead.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Introduces an intermediate format for overlayfs storage, where
.wh-ostree. prefixed files will be converted into char 0:0
whiteout devices used by overlayfs to mark deletions across layers.
The CI scripts now uses a volume for the scratch directories
previously in /var/tmp otherwise we cannot create whiteout
devices into an overlayfs mounted filesystem.
Related-Issue: #2712
This is a follow up to
0ced9fde76
"sysroot: Support /boot on root or as seperate filesystem for syslinux and u-boot"
What we should have done at the time is changed our bootloader entries
to be prefixed with `/boot`. This means that the GRUB2 BLS support
will Just Work.
For now, I'm making this option default to off out of a lot of
conservatism. I think in the future we should flip this on by default.
If `/boot` is an automount, then the unit will be stopped as soon as the
automount expires. That's would defeat the purpose of using systemd to
delay finalizing the deployment until shutdown. This is not uncommon as
`systemd-gpt-auto-generator` will create an automount unit for `/boot`
when it's the EFI System Partition and there's no fstab entry.
To ensure that systemd doesn't stop the service early when the `/boot`
automount expires, introduce a new unit that holds `/boot` open until
it's sent `SIGTERM`. This uses a new `--hold` option for
`finalize-staged` that loads but doesn't lock the sysroot. A separate
unit is used since we want the process to remain active throughout the
finalization run in `ExecStop`. That wouldn't work if it was specified
in `ExecStart` in the same unit since it would be killed before the
`ExecStop` action was run.
Fixes: #2543
are pending
This is to support pending deployments instead of rasing assertion.
For example:
```
$ sudo rpm-ostree kargs --append=foo=bar
$ sudo ostree admin kargs edit-in-place --append-if-missing=foobar
```
After reboot we get both `foo=bar foobar`.
Fix https://github.com/ostreedev/ostree/issues/2679
In the current "ostree native container" flow, we're inserting
a commit object into the repo but with no refs.
We have hacks in a few places to find the commit digest via e.g.
`find repo/objects -name *.commit` but that's a horrible hack.
Add `ostree rev-parse --single` which will print the single commit,
and error out if there is not exactly one commit.
Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>
Don't parse `rpm-ostree status` output, it's not meant for that. Use
`--json` output instead.
While we're here, fix an obsolete reference to Ansible.
Related: https://github.com/coreos/rpm-ostree/pull/3938
https://github.com/coreos/coreos-assembler/pull/2921 broke this
test which is intentionally causing a systemd unit to fail.
As they say, necessity is the mother of invention. They don't
say though that need always causes particularly *beautiful* things
to be invented...
update all existing deployments in place
Example:
$ sudo ostree admin kargs edit-in-place --append-if-missing=rw
See https://github.com/ostreedev/ostree/issues/2617
This will not add duplicate key, if there is `TESTARG=VAL1` in the
kernel arguments, `--append-if-missing=TESTARG=VAL2` will be ignored.
Not all filesystems support extended attributes. This test uses
/var/tmp to try to get an extended-attributes-capable filesystem,
but that might not succeed.
Signed-off-by: Simon McVittie <smcv@debian.org>
No real changes.
```
$ cargo fix --edition
note: Switching to Edition 2021 will enable the use of the version 2 feature resolver in Cargo.
This may cause some dependencies to be built with fewer features enabled than previously.
More information about the resolver changes may be found at https://doc.rust-lang.org/nightly/edition-guide/rust-2021/default-cargo-resolver.html
When building the following dependencies, the given features will no longer be used:
libc v0.2.126 removed features: extra_traits
The following differences only apply when building with dev-dependencies:
getrandom v0.2.6 removed features: std
```
which looks OK to me.
I messed this up; the last release should inherit from the previous
release (N-1) and not the previous to that (N-2).
I think (hope) this isn't an ABI break...
Just noticed this when I was going to add a new symbol.
I'm aiming to do some more work on the Rust side around `fsck`
like functionality, and this is a useful primitive. There isn't
a great Rust crate for xattrs, and I think it's better to share this
code.
Fix up the paths for the crates now that the Rust bindings are in
`rust/`.
We can't today include the test suite because it depends on `ostree-rs-ext`
which would make everything circular.
(Building that now requires a separate `cd tests/inst && cargo build`)
Quite a while ago we added staged deployments, which solved
a bunch of issues around the `/etc` merge. However...a persistent
problem since then is that any failures in that process that
happened in the *previous* boot are not very visible.
We ship custom code in `rpm-ostree status` to query the previous
journal. But that has a few problems - one is that on systems
that have been up a while, that failure message may even get
rotated out. And second, some systems may not even have a persistent
journal at all.
A general thing we do in e.g. Fedora CoreOS testing is to check
for systemd unit failures. We do that both in our automated tests,
and we even ship code that displays them on ssh logins. And beyond
that obviously a lot of other projects do the same; it's easy via
`systemctl --failed`.
So to make failures more visible, change our `ostree-finalize-staged.service`
to have an internal wrapper around the process that "catches" any
errors, and copies the error message into a file in `/boot/ostree`.
Then, a new `ostree-boot-complete.service` looks for this file on
startup and re-emits the error message, and fails.
It also deletes the file. The rationale is to avoid *continually*
warning. For example we need to handle the case when an upgrade
process creates a new staged deployment. Now, we could change the
ostree core code to delete the warning file when that happens instead,
but this is trying to be a conservative change.
This should make failures here much more visible as is.
On systems where `coreutils` are built with `--enable-single-binary=symlinks` like Nix,
`/usr/bin/env` is symlinked to `/usr/bin/coreutils` and uses `argv[0]` to determine which program to run.
Since the `test-cli-extensions.sh` created a new symlink named `ostree-env`,
coreutils would be confused about the utility to choose, so running it would fail:
ostree-env: unknown program ‘ostree-env’
Try 'ostree-env --help' for more information.
Fixes: https://github.com/ostreedev/ostree/issues/2553
I was reading this thread
https://internals.rust-lang.org/t/from-life-before-main-to-common-life-in-main/16006/30
and that reminded me about this code, which it turns out actually
doesn't compile with my default local cargo config:
```
$ cat ~/.cargo/config
[target.x86_64-unknown-linux-gnu]
rustflags = ["-Ctarget-cpu=native", "-C", "link-arg=-fuse-ld=lld"]
[profile.release]
incremental = true
$ cargo b
...
error: linking with `cc` failed: exit status: 1
|
= note: "cc" "-m64" "/var/srv/walters/src/github/ostreedev/ostree/target/debug/deps/ostree_test-4ca8e730f9dc6ffc.10325uqlhkyr5uol.rcgu.o" "/var/srv/walte"
= note: ld.lld: error: undefined symbol: __start_linkme_NONDESTRUCTIVE_TESTS
>>> referenced by 22nn09lfsklfqvyy
>>> /var/srv/walters/src/github/ostreedev/ostree/target/debug/deps/ostree_test-4ca8e730f9dc6ffc.22nn09lfsklfqvyy.rcgu.o:(ostree_tes)
```
For now let's just go back to having a static list of functions.
We don't have *too* many of those.
If we use head(1) to take only the first two lines, then cut(1) and
earlier pipeline entries are killed by SIGPIPE (if they have not already
terminated), and that's flagged as an error under `set -o pipefail`.
Use an equivalent sed command to take exactly the second line, but
without SIGPIPE.
Signed-off-by: Simon McVittie <smcv@debian.org>
Gbp-Pq: Name test-prune-Read-to-the-end-of-cut-1-output.patch
This prevents writing content into 'bare-split-xattrs` repository,
while carving some space for experimenting via a temporary
`OSTREE_EXP_WRITE_BARE_SPLIT_XATTRS` environment flag.
The commit metadata `version` key is well established but getting it for
a remote commit is cumbersome since the commit object needs to be
fetched and loaded. Including it in the summary additional metadata
allows a much more convenient view of what each of the remote refs
represents.
`kola` now follows symlinks when archiving an external test's `data/`
dir. So the recursive `data` symlink we have here breaks it.
Let's just move the shared files in its own directory and update the
symlinks.
Support for that file was added previously, but the testing lived in
rpm-ostree only. Let's add it here too.
In the process add a hidden `--lock-finalization` to `ostree admin
deploy` to make testing easier (though it could also be useful to update
managers driving OSTree via the CLI).
In fixing https://github.com/coreos/rpm-ostree/pull/3323
I felt that it was a bit ugly we're installing `/usr/bin/ostree-container`.
It's kind of an implementation detail. We want users to use
`ostree container`.
Let's support values outside of $PATH too.
For example, this also ensures that TAB completion for `ost` expands
to `ostree ` with a space.
Basically due to the glib structured logging rework we lost the
`noreturn` attribute on `g_error()`.
This is fixed in glib as of f97ff20adf
But we might as well just throw an error here.
This updates the test logic for CLI extensions, actually checking
for functional output from the subcommand.
It also cleans up some environmental leftover.
This adds some logic to detect and dispatch unknown subcommands to
extensions available in `$PATH`. Additional commands can be
implemented by adding relevant `ostree-$verb` binaries to the system.
As an example, if a `/usr/bin/ostree-extcommand` extension is provided,
the execution of `ostree extcommand --help` will be dispatched to that
as `ostree-extcommand extcommand --help`.
This reworks the var-mount destructive test in order to properly use
the datadir for the current stateroot instead of a duplicated one.
In turn, it ensures that the resulting `var.mount` after reboot is
correctly pointing to the same location which hosted `/var` on the
previous boot.
https://bugzilla.redhat.com/show_bug.cgi?id=1945274 is an issue where a privileged
kubernetes daemonset is writing a socket into `/etc`. This makes ostree upgrades barf.
Now, they should clearly move it to `/run`. However, one option is for us to
just ignore it instead of erroring out. Some brief investigation shows that
e.g. `git add somesocket` is a silent no-op, which is an argument in favor of ignoring it.
Closes: https://github.com/ostreedev/ostree/issues/2446
The logic for `--selinux-policy` ended up in the `--tree=dir`
path, but there's no reason for that. Fix the imported
labeling with `--tree=tar`. Prep for use with containers.
We had this bug because the previous logic was trying to avoid
duplicating the code for generic `--selinux-policy` and
the case of `--selinux-policy-from-base --tree=dir`.
It's a bit more code, but it's cleaner if we dis-entangle them.
Having to touch a global test counter when adding tests is
a recipe for conflicts between PRs.
The TAP protocol allows *ending* with the expected number of
tests, so the best way to do this is to have an explicit
API like our `tap_ok` which bumps a counter, then end with `tap_end`.
I ported one test as a demo.
This will be helpful for the "ostree native container" work in
https://github.com/ostreedev/ostree-rs-ext/
Basically in order to reuse GPG/signapi verification, we need
to support adding a remote, even though it can't be used via
`ostree pull`. (At least, not until we merge ostree-rs-ext into ostree, but
even then I think the principle stands)
We're waaay overdue for this, it's been the default
in rpm-ostree for years, and solves several important bugs
around not capturing `/etc` while things are running.
Also, `ostree admin upgrade --stage` (should) become idempotent.
Closes: https://github.com/ostreedev/ostree/issues/2389
There are some existing issues around fsck in unprivileged bare mode,
so this test does not really work at the moment. Leaving it as a FIXME
for the moment.
It cannot work to use `--no-xattrs` when SELinux is enabled
because we get a `security.selinux` attribute on created files
regardless. So just skip this test if true.
Also add some `ostree fsck`s in here which helped me debug
this.
We have a bunch of APIs to do GPG verification of a commit,
but that doesn't generalize to signapi. Further, they
require the caller to check the signature status explicitly
which seems like a trap.
This much higher level API works with both GPG and signapi.
The intention is to use this in things that are doing "external
pulls" like the ostree-ext tar import support. There we will
get the commitmeta from the tarball and we want to verify it
at the same time we import the commit.
As pointed out in the original review, `gpg-list-keys` fits better
alongside the existing `gpg-import`.
Changes were done with:
```
git grep -l list-gpg-keys | xargs sed -i 's/list-gpg-keys/gpg-list-keys/'
for src in $(git ls-files '*list-gpg-keys*'); do
dst=${src/list-gpg-keys/gpg-list-keys}
git mv "$src" "$dst"
done
```
Some distributions set this during build in order to have reproducible
builds from the same source code: for example, Debian uses the date
from debian/changelog.
However, some of our tests assume that `ostree commit` will result in
a commit with the current date/time, and SOURCE_DATE_EPOCH breaks that
assumption. Unset it for our build-time tests.
Resolves: https://github.com/ostreedev/ostree/issues/2405
Signed-off-by: Simon McVittie <smcv@collabora.com>
This provides a wrapper for the `ostree_repo_remote_get_gpg_keys`
function to show the GPG keys associated with a remote. This is
particularly useful for validating that GPG key updates have been
applied. Tests are added, which checks the
`ostree_repo_remote_get_gpg_keys` API by extension.
