Commit Graph

1257 Commits

Author SHA1 Message Date
Colin Walters
f4e56b910f admin-deploy: Add --stateroot as alias for --os
To further help deprecate the confusing "osname" terminology.
2023-08-17 16:49:09 -04:00
Colin Walters
9d5ccfefff Add ostree admin stateroot-init as alias for os-init
To further help deprecate the confusing "osname" terminology.
2023-08-17 16:46:02 -04:00
Eric Curtin
2cc6b53199
Merge pull request #2966 from cgwalters/ostree-admin-edit
Add `admin set-default`
2023-08-08 14:24:35 +01:00
Eric Curtin
c57c00569c
Merge pull request #2958 from cgwalters/deploy-loosen-etc-usretc
deploy: Support an empty `/etc` and populated `/usr/etc`
2023-08-05 16:48:28 +01:00
Colin Walters
3cd3251aa1 Add admin set-default
A core underlying primitive in the C library is the ability
to arbitrarily reorder bootloader entries.

Let's expose the basic functionality here with the ability to pick
an arbitrarily deployment for the next boot.

Closes: https://github.com/ostreedev/ostree/issues/2965
2023-08-03 15:09:57 -04:00
Colin Walters
ac42e29d66 os-init: Create a mount namespace
Today on anything using readonly sysroot `os-init` fails, because
we don't create a mount namespace if the `UNLOCKED` flag is specified
because we assume it's a readonly operation.

Since technically this is a mutation, let's just lock the sysroot
and use the tested path.
2023-08-02 14:32:22 -04:00
Colin Walters
0406fd3912 deploy: Support an empty /etc and populated /usr/etc
In preparation for support for a transient `/etc`:
https://github.com/ostreedev/ostree/issues/2868
particularly in combination with composefs.

Basically it's just much more elegant if we can directly mount
an overlayfs on the *empty* `etc` directory, using `usr/etc` as
the lower.

In the composefs case, we'd have to mount the composefs overlayfs
itself writable (and call `mkdir`) *just* so we can make that
empty `etc` directory which is ugly.
2023-07-29 08:47:54 -04:00
Colin Walters
6470429b2b tests/destructive: Turn off global sync()
Let's verify that things work with that off, as they should.

Previously:
cb73129483
"deploy: Add a 5s max timeout on global filesystem sync()"

But we may still have problems even with that, see
https://issues.redhat.com/browse/OCPBUGS-15917
where it might be that even a thread doesn't work because
we're locked in the kernel.
2023-07-28 17:59:28 -04:00
Eric Curtin
250c40a671
Merge pull request #2948 from cgwalters/composefs-more-cleanups
composefs: Use lowerdir in /run
2023-07-25 10:25:29 +01:00
Colin Walters
d7fe9e54ed kernel-args: Move private functions out of public header
External users can't call `_` prefixed APIs, so move them
to a `-private.h`.
2023-07-23 13:22:14 -04:00
Colin Walters
64afbcdeb0 composefs: Use lowerdir in /run
I just noticed that this was another constant string duplicated
between prepare-root.c and libostree-1.so, and I went to make
it a common `#define` in libotcore.la.

But then I thought "it's ugly to have this directory mixed into
the deployment namespace" because in some theoretical world
it could also be in the ostree commit, which would cause weird
behavior.

I think this is transient state that is better in `/run`, so move
it there.
2023-07-22 16:46:17 -04:00
Colin Walters
1a2fac371b tests: Remove dead references to "SEED"
Nothing uses this.
2023-07-18 07:55:14 -04:00
Colin Walters
82d934916b commit: Add --sign-from-file
Passing the private key via a direct command line argument
is just a bad idea because it's highly likely to get logged
or appear in `ps`.
Spotted in review of work for composefs signatures.
2023-07-15 09:50:40 -04:00
Colin Walters
93699cc546 prepare-root: Add metadata for composefs to /run/ostree-booted
Particularly for the signature case, having this metadata
acts as a reliable "proof of execution" of the signature verification
code (as opposed to parsing a log file or so).

Besides that, this is also just a stronger check for "we're using
composefs" instead of checking for "overlayfs on /".
2023-07-13 07:28:39 -04:00
Alexander Larsson
62e4f37653 tests: Fix composefs test
- Was using the wrong metadata key
- We were missing setting the canonical commit args which assigns
  e.g. owner uid 0, which is important for reproducibility
- Use the new --print-hex to make things easier to read
2023-07-07 09:57:00 -04:00
Colin Walters
25120bd7ad
Merge pull request #2912 from cgwalters/itest-transactionality-debug
tests/inst: A few small patches
2023-07-05 05:47:05 -04:00
Colin Walters
0c36e8143d Drop "ostree trivial-httpd" CLI, move to tests directory
See https://github.com/ostreedev/ostree/issues/1593

Basically this makes it easier for people packaging, as the trivial-httpd
is only for tests, and this way the binary will live with the tests.

Also at this point nothing should depend on `ostree trivial-httpd`.
2023-07-04 08:18:24 -04:00
Colin Walters
8bba482bc8 tests: Enable mtime test
I think this just accidentally was never enabled.

While looking at the code, add a sleep here to be resilient to
filesystems with only second mtime granularity.
2023-06-29 02:11:09 -04:00
Colin Walters
0b519c2573 tests: Drop unused alias 2023-06-29 02:09:51 -04:00
Colin Walters
54c731554e tests/transactionality: Port a bit to xshell
This will give us more useful error messages which should
help debug a flake.
2023-06-29 02:08:56 -04:00
Colin Walters
aeef8221c4 test-composefs: Sync flow with other tests
I am not sure why this is failing on older Debian systems,
but I'm wildly guessing that something being done in `libtest.sh`
is setting up automake in a way that we need.  This is done
in other tests.

Or maybe it's the missing `$CMD_PREFIX`?  Let's see...
2023-06-28 11:58:20 -04:00
Joseph Marrero Corchado
24dd9bbe02
Merge pull request #2895 from cgwalters/release
Release 2023.4
2023-06-20 12:25:17 -04:00
Joseph Marrero Corchado
f07c93e167
Merge pull request #2893 from cgwalters/update-prow-config
ci: Add some composefs testing
2023-06-20 12:22:49 -04:00
Colin Walters
0c35149c7e Release 2023.4 2023-06-20 09:39:04 -04:00
Colin Walters
3c7e256cee tests: Add a sanity check for composefs
Prep for adding some coverage of this flow when booting with
composefs.
2023-06-19 19:02:20 -04:00
Colin Walters
a6f2d053c8 lib: Rework composefs metadata, drop custom signatures
We will be switching to handling signature verification of
the target ostree commit.
2023-06-17 17:08:46 -04:00
Colin Walters
7892818fb1 test-concurrency: Don't lower timeout
I think there's never been a real race condition here.  Instead
the problem is:

- We have a timeout on the lock, after which we error out (30s)
- This test actually *lowers* the timeout

Looking through the failures for test-concurrency what I see
is
`error: Locking repo exclusive failed: Resource temporarily unavailable`
which is us hitting the timeout.

Hardcoded timeouts are just going to be subject to race conditions.
I understand not wanting to block forever in some cases, but any
arbitrary timeout is just going to get hit in real world conditions
too.

Anyways for now, stop shooting ourselves in the foot and at least
keep the timeout at the default.

Closes: https://github.com/ostreedev/ostree/issues/2038
2023-06-13 16:17:16 -04:00
Dan Nicholson
9244518896
Merge pull request #2865 from barthalion/increase-metadata-size
Increase the metadata size limit to 128MB
2023-06-13 09:02:48 -06:00
Bartłomiej Piotrowski
40c07362c1 Update big metadata size in test-pull-large-metadata 2023-06-13 15:46:56 +02:00
Jonathan Lebon
193ef29f3f lib/deploy: Use fallocate for early prune space check
The `f_bfree` member of the `statvfs` struct is documented as the
"number of free blocks". However, different filesystems have different
interpretations of this. E.g. on XFS, this is truly the number of blocks
free for allocating data. On ext4 however, it includes blocks that
are actually reserved by the filesystem and cannot be used for file
data. (Note this is separate from the distinction between `f_bfree` and
`f_bavail` which isn't relevant to us here since we're privileged.)

If a kernel and initrd is sized just right so that it's still within the
`f_bfree` limit but above what we can actually allocate, the early prune
code won't kick in since it'll think that there is enough space. So we
end up hitting `ENOSPC` when we actually copy the files in.

Rework the early prune code to instead use `fallocate` which guarantees
us that a file of a certain size can fit on the filesystem. `fallocate`
requires filesystem support, but all the filesystems we care about for
the bootfs support it (including even FAT).

(There's technically a TOCTOU race here that existed also with the
`statvfs` code where free space could change between when we check
and when we copy. Ideally we'd be able to pass down that fd to the
copying bits, but anyway in practice the bootfs is pretty much owned by
libostree and one doesn't expect concurrent writes during a finalization
operation.)
2023-05-29 12:17:05 -04:00
Jonathan Lebon
a3c0d6a3fe lib/deploy: Log case when auto-pruning is hopeless
For easier diagnostics.
2023-05-28 18:38:53 -04:00
Bartłomiej Piotrowski
99f6356b5b Use a value based on OSTREE_MAX_METADATA_SIZE 2023-05-26 12:09:13 +02:00
Colin Walters
88e8b671ce tests: A bit more xshell porting
Part of https://github.com/ostreedev/ostree/issues/2857
2023-05-18 08:14:50 -04:00
Colin Walters
dc23b9389b tests/inst: Add xshell and use it in one place
I've deprecated sh-inline; in the end I think it is better
to minimize the amount of bash code we have.  xshell solves
the core convenience problem of taking local variables and mapping
them to command arguments.

A full port would be nontrivial; this just starts the ball
rolling.
2023-05-10 14:02:09 -04:00
Colin Walters
453aed97f6 tree-wide: Run clang-format
This is a one-time tree wide reformatting to ensure consistency
going forward.
2023-05-02 08:42:19 -04:00
Jonathan Lebon
c561e6179e lib/sysroot-deploy: Add experimental support for automatic early prune
During the early design of FCOS and RHCOS, we chose a value of 384M
for the boot partition. This turned out to be too small: some arches
other than x86_64 have larger initrds, kernel binaries, or additional
artifacts (like device tree blobs). We'll likely bump the boot partition
size in the future, but we don't want to abandon all the nodes deployed
with the current size.[[1]]

Because stale entries in `/boot` are cleaned up after new entries are
written, there is a window in the update process during which the bootfs
temporarily must host all the `(kernel, initrd)` pairs for the union of
current and new deployments.

This patch determines if the bootfs is capable of holding all the
pairs. If it can't but it could hold all the pairs from just the new
deployments, the outgoing deployments (e.g. rollbacks) are deleted
*before* new deployments are written. This is done by updating the
bootloader in two steps to maintain atomicity.

Since this is a lot of new logic in an important section of the
code, this feature is gated for now behind an environment variable
(`OSTREE_ENABLE_AUTO_EARLY_PRUNE`). Once we gain more experience with
it, we can consider turning it on by default.

This strategy increases the fallibility of the update system since one
would no longer be able to rollback to the previous deployment if a bug
is present in the bootloader update logic after auto-pruning (see [[2]]
and following). This is however mitigated by the fact that the heuristic
is opportunistic: the rollback is pruned *only if* it's the only way for
the system to update.

[1]: https://github.com/coreos/fedora-coreos-tracker/issues/1247
[2]: https://github.com/ostreedev/ostree/issues/2670#issuecomment-1179341883

Closes: #2670
2023-05-01 12:12:03 -04:00
Jonathan Lebon
fd3304e987 tests/kolainst: Add make localinstall
When hacking and testing locally with `cosa build-fast` and `kola run`,
I prefer to leave testing framework stuff within the work directory
rather than installed in my pet container. Add a `localinstall` target
for this which puts the tests in `tests/kola`. Then a simple `kola run`
will pick it up.
2023-04-14 15:19:17 -04:00
Jonathan Lebon
771deb55d1 tests/kola: delete unused .gitignore
AFAICT, I don't see how `runkola.sh` or the Makefile in `tests/kolainst`
can create files in `tests/kola` since it's geared towards installing
under `/usr`.
2023-04-14 15:19:17 -04:00
Colin Walters
cafe12cf1f treegen: Require at least one mutation
Since a later assertion would otherwise trigger.  We saw
this happen in CI.
2023-03-23 16:48:55 -04:00
Colin Walters
7b258b2499
Merge pull request #2837 from smcv/g-steal-fd
Use g_steal_fd() in preference to glnx_steal_fd()
2023-03-17 11:27:03 -04:00
Joseph Marrero Corchado
4f0c13fb8b
Merge pull request #2705 from cgwalters/always-boot
Add `sysroot.bootprefix` option
2023-03-17 11:09:19 -04:00
Simon McVittie
7b02fdfdf8 Use g_steal_fd() in preference to glnx_steal_fd()
g_steal_fd() exists in GLib since 2.70, and libglnx has a backport for
older GLib versions, equivalent to the libglnx-specific glnx_steal_fd().

Signed-off-by: Simon McVittie <smcv@collabora.com>
2023-03-17 13:32:26 +00:00
Colin Walters
813c439352 tests: Set size on fileinfo
Closes: https://github.com/ostreedev/ostree/issues/2827
2023-03-16 18:31:03 -04:00
Colin Walters
e8fbc756cb itest-pull-space: Use mkfs.ext4, align to at least 512b
XFS now seems to want filesystems larger than 300MB, so switch
to ext4.  Also use `20MiB` so we align to 512b sectors to squash
a `losetup` warning.
Also tweak some of the numbers to still work.
2023-03-14 17:07:01 -04:00
Simon McVittie
3500ad0443 tests: Ensure non-root users have access to libcap tools
On Debian systems, by default only root has /{usr/,}sbin in PATH.

Signed-off-by: Simon McVittie <smcv@collabora.com>
2023-02-27 12:40:05 +00:00
Colin Walters
cd100df413 Release 2023.1 2023-02-17 11:31:27 -05:00
Colin Walters
8025a83b6d
Merge pull request #1946 from dbnicholson/ostree-metadata-api
ostree-metadata commit API
2023-02-09 08:24:58 -05:00
Colin Walters
cff0a1730a
Merge pull request #2811 from dbnicholson/cli-polish
Various CLI improvements
2023-02-09 08:10:36 -05:00
Dan Nicholson
70ec92e1cd bin/remote-summary: Add options to show metadata 2023-02-07 22:59:30 -07:00
Dan Nicholson
b49f93aa95 bin/summary: Add options to show metadata 2023-02-07 22:59:30 -07:00