Commit Graph

1245 Commits

Author SHA1 Message Date
Colin Walters
82d934916b commit: Add --sign-from-file
Passing the private key via a direct command line argument
is just a bad idea because it's highly likely to get logged
or appear in `ps`.
Spotted in review of work for composefs signatures.
2023-07-15 09:50:40 -04:00
Colin Walters
93699cc546 prepare-root: Add metadata for composefs to /run/ostree-booted
Particularly for the signature case, having this metadata
acts as a reliable "proof of execution" of the signature verification
code (as opposed to parsing a log file or so).

Besides that, this is also just a stronger check for "we're using
composefs" instead of checking for "overlayfs on /".
2023-07-13 07:28:39 -04:00
Alexander Larsson
62e4f37653 tests: Fix composefs test
- Was using the wrong metadata key
- We were missing setting the canonical commit args which assigns
  e.g. owner uid 0, which is important for reproducibility
- Use the new --print-hex to make things easier to read
2023-07-07 09:57:00 -04:00
Colin Walters
25120bd7ad
Merge pull request #2912 from cgwalters/itest-transactionality-debug
tests/inst: A few small patches
2023-07-05 05:47:05 -04:00
Colin Walters
0c36e8143d Drop "ostree trivial-httpd" CLI, move to tests directory
See https://github.com/ostreedev/ostree/issues/1593

Basically this makes it easier for people packaging, as the trivial-httpd
is only for tests, and this way the binary will live with the tests.

Also at this point nothing should depend on `ostree trivial-httpd`.
2023-07-04 08:18:24 -04:00
Colin Walters
8bba482bc8 tests: Enable mtime test
I think this just accidentally was never enabled.

While looking at the code, add a sleep here to be resilient to
filesystems with only second mtime granularity.
2023-06-29 02:11:09 -04:00
Colin Walters
0b519c2573 tests: Drop unused alias 2023-06-29 02:09:51 -04:00
Colin Walters
54c731554e tests/transactionality: Port a bit to xshell
This will give us more useful error messages which should
help debug a flake.
2023-06-29 02:08:56 -04:00
Colin Walters
aeef8221c4 test-composefs: Sync flow with other tests
I am not sure why this is failing on older Debian systems,
but I'm wildly guessing that something being done in `libtest.sh`
is setting up automake in a way that we need.  This is done
in other tests.

Or maybe it's the missing `$CMD_PREFIX`?  Let's see...
2023-06-28 11:58:20 -04:00
Joseph Marrero Corchado
24dd9bbe02
Merge pull request #2895 from cgwalters/release
Release 2023.4
2023-06-20 12:25:17 -04:00
Joseph Marrero Corchado
f07c93e167
Merge pull request #2893 from cgwalters/update-prow-config
ci: Add some composefs testing
2023-06-20 12:22:49 -04:00
Colin Walters
0c35149c7e Release 2023.4 2023-06-20 09:39:04 -04:00
Colin Walters
3c7e256cee tests: Add a sanity check for composefs
Prep for adding some coverage of this flow when booting with
composefs.
2023-06-19 19:02:20 -04:00
Colin Walters
a6f2d053c8 lib: Rework composefs metadata, drop custom signatures
We will be switching to handling signature verification of
the target ostree commit.
2023-06-17 17:08:46 -04:00
Colin Walters
7892818fb1 test-concurrency: Don't lower timeout
I think there's never been a real race condition here.  Instead
the problem is:

- We have a timeout on the lock, after which we error out (30s)
- This test actually *lowers* the timeout

Looking through the failures for test-concurrency what I see
is
`error: Locking repo exclusive failed: Resource temporarily unavailable`
which is us hitting the timeout.

Hardcoded timeouts are just going to be subject to race conditions.
I understand not wanting to block forever in some cases, but any
arbitrary timeout is just going to get hit in real world conditions
too.

Anyways for now, stop shooting ourselves in the foot and at least
keep the timeout at the default.

Closes: https://github.com/ostreedev/ostree/issues/2038
2023-06-13 16:17:16 -04:00
Dan Nicholson
9244518896
Merge pull request #2865 from barthalion/increase-metadata-size
Increase the metadata size limit to 128MB
2023-06-13 09:02:48 -06:00
Bartłomiej Piotrowski
40c07362c1 Update big metadata size in test-pull-large-metadata 2023-06-13 15:46:56 +02:00
Jonathan Lebon
193ef29f3f lib/deploy: Use fallocate for early prune space check
The `f_bfree` member of the `statvfs` struct is documented as the
"number of free blocks". However, different filesystems have different
interpretations of this. E.g. on XFS, this is truly the number of blocks
free for allocating data. On ext4 however, it includes blocks that
are actually reserved by the filesystem and cannot be used for file
data. (Note this is separate from the distinction between `f_bfree` and
`f_bavail` which isn't relevant to us here since we're privileged.)

If a kernel and initrd is sized just right so that it's still within the
`f_bfree` limit but above what we can actually allocate, the early prune
code won't kick in since it'll think that there is enough space. So we
end up hitting `ENOSPC` when we actually copy the files in.

Rework the early prune code to instead use `fallocate` which guarantees
us that a file of a certain size can fit on the filesystem. `fallocate`
requires filesystem support, but all the filesystems we care about for
the bootfs support it (including even FAT).

(There's technically a TOCTOU race here that existed also with the
`statvfs` code where free space could change between when we check
and when we copy. Ideally we'd be able to pass down that fd to the
copying bits, but anyway in practice the bootfs is pretty much owned by
libostree and one doesn't expect concurrent writes during a finalization
operation.)
2023-05-29 12:17:05 -04:00
Jonathan Lebon
a3c0d6a3fe lib/deploy: Log case when auto-pruning is hopeless
For easier diagnostics.
2023-05-28 18:38:53 -04:00
Bartłomiej Piotrowski
99f6356b5b Use a value based on OSTREE_MAX_METADATA_SIZE 2023-05-26 12:09:13 +02:00
Colin Walters
88e8b671ce tests: A bit more xshell porting
Part of https://github.com/ostreedev/ostree/issues/2857
2023-05-18 08:14:50 -04:00
Colin Walters
dc23b9389b tests/inst: Add xshell and use it in one place
I've deprecated sh-inline; in the end I think it is better
to minimize the amount of bash code we have.  xshell solves
the core convenience problem of taking local variables and mapping
them to command arguments.

A full port would be nontrivial; this just starts the ball
rolling.
2023-05-10 14:02:09 -04:00
Colin Walters
453aed97f6 tree-wide: Run clang-format
This is a one-time tree wide reformatting to ensure consistency
going forward.
2023-05-02 08:42:19 -04:00
Jonathan Lebon
c561e6179e lib/sysroot-deploy: Add experimental support for automatic early prune
During the early design of FCOS and RHCOS, we chose a value of 384M
for the boot partition. This turned out to be too small: some arches
other than x86_64 have larger initrds, kernel binaries, or additional
artifacts (like device tree blobs). We'll likely bump the boot partition
size in the future, but we don't want to abandon all the nodes deployed
with the current size.[[1]]

Because stale entries in `/boot` are cleaned up after new entries are
written, there is a window in the update process during which the bootfs
temporarily must host all the `(kernel, initrd)` pairs for the union of
current and new deployments.

This patch determines if the bootfs is capable of holding all the
pairs. If it can't but it could hold all the pairs from just the new
deployments, the outgoing deployments (e.g. rollbacks) are deleted
*before* new deployments are written. This is done by updating the
bootloader in two steps to maintain atomicity.

Since this is a lot of new logic in an important section of the
code, this feature is gated for now behind an environment variable
(`OSTREE_ENABLE_AUTO_EARLY_PRUNE`). Once we gain more experience with
it, we can consider turning it on by default.

This strategy increases the fallibility of the update system since one
would no longer be able to rollback to the previous deployment if a bug
is present in the bootloader update logic after auto-pruning (see [[2]]
and following). This is however mitigated by the fact that the heuristic
is opportunistic: the rollback is pruned *only if* it's the only way for
the system to update.

[1]: https://github.com/coreos/fedora-coreos-tracker/issues/1247
[2]: https://github.com/ostreedev/ostree/issues/2670#issuecomment-1179341883

Closes: #2670
2023-05-01 12:12:03 -04:00
Jonathan Lebon
fd3304e987 tests/kolainst: Add make localinstall
When hacking and testing locally with `cosa build-fast` and `kola run`,
I prefer to leave testing framework stuff within the work directory
rather than installed in my pet container. Add a `localinstall` target
for this which puts the tests in `tests/kola`. Then a simple `kola run`
will pick it up.
2023-04-14 15:19:17 -04:00
Jonathan Lebon
771deb55d1 tests/kola: delete unused .gitignore
AFAICT, I don't see how `runkola.sh` or the Makefile in `tests/kolainst`
can create files in `tests/kola` since it's geared towards installing
under `/usr`.
2023-04-14 15:19:17 -04:00
Colin Walters
cafe12cf1f treegen: Require at least one mutation
Since a later assertion would otherwise trigger.  We saw
this happen in CI.
2023-03-23 16:48:55 -04:00
Colin Walters
7b258b2499
Merge pull request #2837 from smcv/g-steal-fd
Use g_steal_fd() in preference to glnx_steal_fd()
2023-03-17 11:27:03 -04:00
Joseph Marrero Corchado
4f0c13fb8b
Merge pull request #2705 from cgwalters/always-boot
Add `sysroot.bootprefix` option
2023-03-17 11:09:19 -04:00
Simon McVittie
7b02fdfdf8 Use g_steal_fd() in preference to glnx_steal_fd()
g_steal_fd() exists in GLib since 2.70, and libglnx has a backport for
older GLib versions, equivalent to the libglnx-specific glnx_steal_fd().

Signed-off-by: Simon McVittie <smcv@collabora.com>
2023-03-17 13:32:26 +00:00
Colin Walters
813c439352 tests: Set size on fileinfo
Closes: https://github.com/ostreedev/ostree/issues/2827
2023-03-16 18:31:03 -04:00
Colin Walters
e8fbc756cb itest-pull-space: Use mkfs.ext4, align to at least 512b
XFS now seems to want filesystems larger than 300MB, so switch
to ext4.  Also use `20MiB` so we align to 512b sectors to squash
a `losetup` warning.
Also tweak some of the numbers to still work.
2023-03-14 17:07:01 -04:00
Simon McVittie
3500ad0443 tests: Ensure non-root users have access to libcap tools
On Debian systems, by default only root has /{usr/,}sbin in PATH.

Signed-off-by: Simon McVittie <smcv@collabora.com>
2023-02-27 12:40:05 +00:00
Colin Walters
cd100df413 Release 2023.1 2023-02-17 11:31:27 -05:00
Colin Walters
8025a83b6d
Merge pull request #1946 from dbnicholson/ostree-metadata-api
ostree-metadata commit API
2023-02-09 08:24:58 -05:00
Colin Walters
cff0a1730a
Merge pull request #2811 from dbnicholson/cli-polish
Various CLI improvements
2023-02-09 08:10:36 -05:00
Dan Nicholson
70ec92e1cd bin/remote-summary: Add options to show metadata 2023-02-07 22:59:30 -07:00
Dan Nicholson
b49f93aa95 bin/summary: Add options to show metadata 2023-02-07 22:59:30 -07:00
Dan Nicholson
86701f0f84 bin/show: Add options to list metadata keys
While `--print-metadata-key` is very useful, it's not that helpful if
you don't know what the keys are.
2023-02-07 22:59:30 -07:00
Dan Nicholson
008c5a9540 bin/remote-refs: Add option to print revisions
The only other way to get the remote ref revision from the CLI is to
scrape the output of `ostree remote summary` or pull the commit. The
revision is already there in the summary's ref map, so might as well add
an option to show it.
2023-02-07 22:59:30 -07:00
Dan Nicholson
90dd45fb27 bin/refs: Add option to print revisions
Allow printing the revision along with the ref. This is very convenient
for looping over the refs in a shell as well as for quickly seeing which
refs are pointed to the same commit.
2023-02-07 22:59:30 -07:00
Dan Nicholson
6b0f42ae37 bin/refs: Sort listing
The output is much more readable sorted. I can't think of any reason
you'd want it unsorted (which is essentially dentry order).
2023-02-07 22:59:30 -07:00
Dan Nicholson
b54f486489 Strip trailing whitespace on all C files
My editor started following the configuration in .editorconfig and is
applying this rule to many files I'm editing. Let's just get this over
with and strip everything. This was done like so:

  git ls-files | grep '\.[ch]$' | xargs sed -ri 's/\s+$//'
2023-02-07 14:50:08 -07:00
Jonathan Lebon
fb63f7aba8 ostree/prune: Calculate reachability under exclusive lock
When we calculate the reachability set in `ostree prune`, we do this
without any locking. This means that between the time we build the set
and when we call `ostree_repo_prune_from_reachable`, new content
might've been added. This then causes us to immediately prune that
content since it's not in the now outdated set.

Fix this by calculating the set under an exclusive lock.

I think this is what happened in
https://github.com/fedora-silverblue/issue-tracker/issues/405. While
the pruner was running, the `new-updates-sync` script[1] was importing
content into the repo. The newly imported commits were immediately
deleted by the many `ostree prune --commit-only` calls the pruner does,
breaking the refs.

[1] https://pagure.io/fedora-infra/ansible/blob/35b35127e444/f/roles/bodhi2/backend/files/new-updates-sync#_18
2023-01-30 15:08:27 -05:00
Dan Nicholson
4fd36337b1 tests/inst: Update hyper-staticfile to 0.9.4
Due to some vulnerabilities[1][2], all versions prior to 0.9.4 have been
yanked from crates.io. There have been some API changes since 0.6.0, but
none of them affect the simple `hyper_staticfile::Static` usage here.

1. https://rustsec.org/advisories/RUSTSEC-2022-0069.html
2. https://rustsec.org/advisories/RUSTSEC-2022-0072.html
2023-01-24 13:20:10 -07:00
Alex Kiernan
bd182584a1 tests: Handle musl's ERANGE mapping
musl uses "Result not representable" for ERANGE, support this in
addition to glibc's "Numerical result out of range".

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
2023-01-02 10:40:17 +00:00
Colin Walters
e2f8ea6957 Release 2022.7 2022-11-23 09:02:52 -05:00
Joseph Marrero Corchado
52b12e71ee
Merge pull request #2782 from cgwalters/testinst-updates 2022-11-22 14:22:48 -05:00
Colin Walters
647f7fb7f1 tests: Require OSTREE_TEST_SUDO to be set to use sudo
IMO, tools invoking `sudo` automatically is a huge anti-pattern.
Require this to be opt-in.

In practice also...these tests really want to be *integration*
and not unit tests because involving `sudo` alongside uninstalled
binaries just creates a giant mess.

Closes: https://github.com/ostreedev/ostree/issues/2761
2022-11-22 10:27:28 -05:00
Colin Walters
2dc92b25cd tests/inst: Update sh-inline 2022-11-22 10:14:48 -05:00