Commit Graph

1298 Commits

Author SHA1 Message Date
Simon McVittie
b42f053db9 tests: Skip composefs test if /var/tmp does not support user xattrs
Otherwise, this test fails on Debian 12 (Linux 6.1) kernels if /var/tmp
is a tmpfs. Some autobuilders put the entire build chroot on a tmpfs,
to speed up builds.

Signed-off-by: Simon McVittie <smcv@debian.org>
2024-01-24 14:03:18 +00:00
Jonathan Lebon
92b1a27202 Add concept of state overlays
In the OSTree model, executables go in `/usr`, state in `/var` and
configuration in `/etc`. Software that lives in `/opt` however messes
this up because it often mixes code *and* state, making it harder to
manage.

More generally, it's sometimes useful to have the OSTree commit contain
code under a certain path, but still allow that path to be writable by
software and the sysadmin at runtime (`/usr/local` is another instance).

Add the concept of state overlays. A state overlay is an overlayfs
mount whose upper directory, which contains unmanaged state, is carried
forward on top of a lower directory, containing OSTree-managed files.

In the example of `/usr/local`, OSTree commits can ship content there,
all while allowing users to e.g. add scripts in `/usr/local/bin` when
booted into that commit.

Some reconciliation logic is executed whenever the base is updated so
that newer files in the base are never shadowed by a copied up version
in the upper directory. This matches RPM semantics when upgrading
packages whose files may have been modified.

For ease of integration, this is exposed as a systemd template unit which
any downstream distro/user can enable. The instance name is the mountpath
in escaped systemd path notation (e.g.
`ostree-state-overlay@usr-local.service`).

See discussions in https://github.com/ostreedev/ostree/issues/3113 for
more details.
2024-01-09 23:20:41 -05:00
Jonathan Lebon
4982306e67 lib/deploy: Round to block size in early prune space check
When we estimate how much space a new bootcsum dir will use, we
weren't accounting for the space overhead from files not using the
last filesystem block completely. This doesn't matter much if counting
a few files, but e.g. on FCOS aarch64, we include lots of small
devicetree blobs in the bootfs. That loss can add up to enough for the
`fallocate()` check to pass but copying still hitting `ENOSPC` later on.

I think a better fix here is to change approach entirely and instead
refactor `install_deployment_kernel()` so that we can call just the
copying bits of it as part of the early prune logic. We'll get a more
accurate assessment and it's not lost work since we won't need to
recopy later on. Also this would not require having to keep in sync the
estimator and the install bits.

That said, this is blocking FCOS releases, so I went with a more tactical
fix for now.

Fixes: https://github.com/coreos/fedora-coreos-tracker/issues/1637
2024-01-04 12:57:43 -05:00
Colin Walters
5b23804a1a Release 2023.8 2023-12-05 13:37:04 -05:00
Colin Walters
c3dff62e1c commit: Try reflinks for local commits by default
I think we originally used to do this, but at some point in a
code refactoring, this optimization got lost.

It's a quite important optimization for the case of writing content
generated by an external system into an ostree repository.
2023-12-04 20:45:08 -05:00
Colin Walters
bab87cbd9d tests: Drop bare-user on tmpfs negative test
As of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2daf18a7884dc03d5164ab9c7dc3f2ea70638469
the restriction is lifted, FINALLY.
2023-12-03 09:32:36 -05:00
Colin Walters
66064d960d tmpfiles: Copy /usr/share/factory/var to /var
This is a pattern we want to encourage.  It's honestly just
way simpler than what rpm-ostree is doing today in auto-synthesizing
individual tmpfiles.d snippets.
2023-11-30 18:11:33 -05:00
Colin Walters
d3de3a0a88 tests: Add a dedicated finalization test
This one covers `admin lock-finalization --unlock`.
2023-11-28 18:45:11 -05:00
Colin Walters
28cc761806 sysroot: Stabilize deployment finalization, add API and CLI
It's about time we do this; deployment finalization locking
is a useful feature.  An absolutely key thing here is that
we've slowly been moving towards the deployments as the primary
"source of truth".

Specifically in bootc for example, we will GC container images
not referenced by a deployment.

This is then neecessary to support a "pull but don't apply automatically" model.

This stabilizes the existing `ostree admin deploy --lock-finalization`
CLI, and adds a new `ostree admin unlock-finalization`.

We still check the old lock file path, but there's a new boolean
value as part of the staged deployment data which is intended
to be the source of truth in the future.  At some point then we
can drop the rpm-ostree lockfile handling.

Closes: https://github.com/ostreedev/ostree/issues/3025
2023-11-27 10:59:56 -05:00
Colin Walters
b6f435fe11 tests: Turn off gpg verification for dev builds
Right now `ostree admin status` errors out in this case, but
`rpm-ostree status` doesn't.  The former behavior is probably
more of a bug, work around it for now.
2023-11-27 10:59:56 -05:00
Eric Curtin
29973774ea
Merge pull request #3091 from cgwalters/fix-soup3
build-sys: Enable libsoup3 by default if installed
2023-11-10 16:39:13 +00:00
Colin Walters
bc62fd5196 unlock: Don't pass options again to overlayfs
There seems to be a tricky regression here with the util-linux
support for the new mount API, plus overlays support for it.

```
[2023-11-09T21:05:30.633Z] Nov 09 21:05:26 qemu0 kola-runext-unlock-transient.sh[2108]: + unshare -m -- /bin/sh -c 'mount -o remount,rw /usr && echo hello from transient unlock >/usr/share/writable-usr-test'
[2023-11-09T21:05:30.633Z] Nov 09 21:05:26 qemu0 kola-runext-unlock-transient.sh[2148]: mount: /usr: mount point not mounted or bad option.
[2023-11-09T21:05:30.633Z] Nov 09 21:05:26 qemu0 kola-runext-unlock-transient.sh[2148]:        dmesg(1) may have more information after failed mount system call.
```

OK this seems related to the new mount API support in util-linux and overlayfs.  From a strace:

```
2095  open_tree(AT_FDCWD, "/usr", OPEN_TREE_CLOEXEC) = 3
2095  mount_setattr(-1, NULL, 0, NULL, 0) = -1 EINVAL (Invalid argument)
...
2095  fspick(3, "", FSPICK_NO_AUTOMOUNT|FSPICK_EMPTY_PATH) = 4
2095  fsconfig(4, FSCONFIG_SET_FLAG, "seclabel", NULL, 0) = 0
2095  fsconfig(4, FSCONFIG_SET_STRING, "lowerdir", "usr", 0) = -1 EINVAL (Invalid argument)
```

I think the core problem here is it's trying to reconfigure the mount with existing options,
but in the new mount namespace we can't see the lowerdir.

Here we really really just want to remount writable.  Telling
util-linux to not pass existing options fixes it.
2023-11-09 20:53:52 -05:00
Colin Walters
c668a8c531 tests: Work around systemd regression for boot id parsing
cc https://github.com/systemd/systemd/issues/29275
2023-11-09 15:10:33 -05:00
Colin Walters
c3cf5b237d undeploy: Parse integer more properly
`atoi` doesn't offer any error checking.

Closes: https://github.com/ostreedev/ostree/issues/3088
2023-11-08 16:15:01 -05:00
Colin Walters
b9e73a3813
Merge pull request #3078 from HuijingHei/karg-delete-array
karg-delete: support multiple times
2023-10-17 09:44:24 -04:00
HuijingHei
003cb99c32 karg-delete: support multiple times
Fix https://github.com/ostreedev/ostree/pull/2612#issuecomment-1764412141
2023-10-17 20:22:08 +08:00
Colin Walters
d788c5b5c9
Merge pull request #3075 from cgwalters/reenable-composefs
tests: Use ext4, re-enable composefs test
2023-10-16 10:24:10 -04:00
Colin Walters
ac69c70750 deploy: Remove lock when re-staging
This closes the biggest foot-gun when doing e.g.
`rpm-ostree rebase` when zincati is running on a FCOS system.

Previously if zincati happened to have staged + locked a deployment,
we'd keep around the lock which is definitely not what is desired.
2023-10-13 13:45:04 -04:00
Colin Walters
833251cb0d tests: Use ext4, re-enable composefs test
Until the XFS fsverity stuff lands.
2023-10-13 12:00:05 -04:00
Colin Walters
265b7f9ecc deploy: Improve error message for nonexistent stateroot
Came up on an internal chat; previously we were only erroring
out when trying to do the SELinux labeling for `/var` which
was really misleading.

Add some other error prefixing while we have the patient open.
2023-10-12 13:22:35 -04:00
Colin Walters
cd0fc9f4f8
Merge pull request #3072 from alexlarsson/fix-whiteout-test
tests: Fix whiteout test
2023-10-12 10:25:40 -04:00
Alexander Larsson
397a11762e tests: Fix whiteout test
This test was always skipped, because the check:

 if touch overlay/baz/.wh.cow &&
    touch overlay/.wh.deeper &&
    touch overlay/baz/another/.wh..wh..opq; then

always fails due to the missing overlay/baz/another directory.
Fix by creating the directory.
2023-10-12 10:18:36 +02:00
Colin Walters
81c08746c6 repo: Add an option to label /usr/etc as /etc
This will be very useful for enabling a "transient /etc" option
because we won't have to do hacks relabling in the initramfs, or
forcing it on just for composefs.
2023-10-11 13:25:08 -04:00
Colin Walters
734ea3ee37 ci: Disable composefs test for now
Need to change the framework to boot with an Ignition config
that switches to ext4 soon.
2023-10-11 09:16:45 -04:00
Owen W. Taylor
3b2fd6e9ff When exporting, use hardlinks for duplicated files
For ostree_repo_export_tree_to_archive(), and 'ostree export', when the
exported tree contains multiple files with the same checksum, write an
archive with hard links.

Without this, importing a tree, then exporting it again breaks
hardlinks.

As an example of savings: this reduces the (compressed) size of the
Fedora Flatpak Runtime image from 1345MiB to 712MiB.

Resolves: #2925
2023-09-29 13:45:29 -04:00
Colin Walters
372cbd7a64 tests: Add an integration test for composefs signatures
Ensure we have some automated test coverage for this.
2023-09-21 07:53:19 -04:00
Colin Walters
e3291ccce9 tests: Rework detection of trivial-httpd
Because it's now at the toplevel.
2023-09-19 16:52:13 -04:00
Jonathan Lebon
fc5aef6fe6 lib/pull: Don't scan commit objects we fetch via deltas
When we're fetching a commit via static delta, we already take care of
fetching the full commit, so there's no need to also scan it using the
regular object workflow.

Closes: #2053
2023-09-08 17:49:25 -04:00
Jonathan Lebon
c8ed1c7a87 lib/pull: Fix miscounting of missing metadata
If e.g. detached metadata or a parent commit is missing, we
don't consider it an error but we erroneously still increment the
`n_fetched_metadata` counter, causing it to be higher than it should.

This will implicitly be tested by a test added in the next patch.
2023-09-08 17:49:25 -04:00
Colin Walters
e93ebd7385
Merge pull request #3018 from ostreedev/dependabot/submodules/composefs-597a766
build(deps): bump composefs from `1aed878` to `597a766`
2023-08-31 12:16:50 -04:00
Eric Curtin
8d838230f1
Merge pull request #3012 from cgwalters/default-early-prune
sysroot: Promote the "early prune" behavior to default
2023-08-31 16:30:26 +01:00
Colin Walters
d5cfbed5bc sysroot: Promote the "early prune" behavior to default
I think we have enough testing for this, let's default it to
on, and change the variable to allow opt-out in case it does
introduce a regression.
2023-08-31 10:25:59 -04:00
dependabot[bot]
0d7b8ebc6b build(deps): bump composefs from 1aed878 to 597a766
Bumps [composefs](https://github.com/containers/composefs) from `1aed878` to `597a766`.
- [Release notes](https://github.com/containers/composefs/releases)
- [Commits](1aed8781d6...597a766095)

---
updated-dependencies:
- dependency-name: composefs
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-08-30 13:23:13 -04:00
Colin Walters
d36bf35c10 switchroot: Lower config parser to otcore, add unit tests
Part of the continuation of unit testing coverage.
2023-08-25 15:59:34 -04:00
Colin Walters
02d419079d deploy: Add bootloader-naming-2 opt-init
I've verified that this fixes compatibility with GRUB, which
parses the filename:
https://github.com/ostreedev/ostree/issues/2961

However, out of a large degree of conservatism I've made this
an opt-in behavior for now.

My plan is to test it out in the FCOS development streams first.
2023-08-24 09:40:54 -04:00
Colin Walters
20b8cb174c Move prepare-root karg helpers into otcore, add unit tests
Add long overdue unit testing coverage for this, which
at least slightly closes out the android boot CI gap.

Actually, this *copies* the karg parsing code into otcore because
it now uses glib, which we're not yet using in the static
prepare-root.  It's pretty tempting to drop support for the
static prepare root entirely.  But for now we'll live with some
code duplication.
2023-08-23 17:11:10 -04:00
Joseph Marrero Corchado
4451949213
Merge pull request #2997 from cgwalters/test-cleanups-1
tests/destructive: Port more to xshell
2023-08-23 10:38:16 -04:00
Joseph Marrero Corchado
37f247da02
Merge pull request #2999 from cgwalters/add-oscore-units
tests: Add otcore unit tests
2023-08-23 10:37:48 -04:00
Colin Walters
facb9a1a60 tests/inst: Update to latest ostree-ext
(And glib, etc.)
2023-08-22 14:23:56 -04:00
Colin Walters
e3f0c4d49c tests: Add otcore unit tests
This just stubs out the basic infrastructure, to be expanded upon.
2023-08-22 13:04:46 -04:00
Colin Walters
578c87e7ef tests/destructive: Port more to xshell
Just keeping up momentum.
2023-08-22 09:52:43 -04:00
Colin Walters
f4e56b910f admin-deploy: Add --stateroot as alias for --os
To further help deprecate the confusing "osname" terminology.
2023-08-17 16:49:09 -04:00
Colin Walters
9d5ccfefff Add ostree admin stateroot-init as alias for os-init
To further help deprecate the confusing "osname" terminology.
2023-08-17 16:46:02 -04:00
Eric Curtin
2cc6b53199
Merge pull request #2966 from cgwalters/ostree-admin-edit
Add `admin set-default`
2023-08-08 14:24:35 +01:00
Eric Curtin
c57c00569c
Merge pull request #2958 from cgwalters/deploy-loosen-etc-usretc
deploy: Support an empty `/etc` and populated `/usr/etc`
2023-08-05 16:48:28 +01:00
Colin Walters
3cd3251aa1 Add admin set-default
A core underlying primitive in the C library is the ability
to arbitrarily reorder bootloader entries.

Let's expose the basic functionality here with the ability to pick
an arbitrarily deployment for the next boot.

Closes: https://github.com/ostreedev/ostree/issues/2965
2023-08-03 15:09:57 -04:00
Colin Walters
ac42e29d66 os-init: Create a mount namespace
Today on anything using readonly sysroot `os-init` fails, because
we don't create a mount namespace if the `UNLOCKED` flag is specified
because we assume it's a readonly operation.

Since technically this is a mutation, let's just lock the sysroot
and use the tested path.
2023-08-02 14:32:22 -04:00
Colin Walters
0406fd3912 deploy: Support an empty /etc and populated /usr/etc
In preparation for support for a transient `/etc`:
https://github.com/ostreedev/ostree/issues/2868
particularly in combination with composefs.

Basically it's just much more elegant if we can directly mount
an overlayfs on the *empty* `etc` directory, using `usr/etc` as
the lower.

In the composefs case, we'd have to mount the composefs overlayfs
itself writable (and call `mkdir`) *just* so we can make that
empty `etc` directory which is ugly.
2023-07-29 08:47:54 -04:00
Colin Walters
6470429b2b tests/destructive: Turn off global sync()
Let's verify that things work with that off, as they should.

Previously:
cb73129483
"deploy: Add a 5s max timeout on global filesystem sync()"

But we may still have problems even with that, see
https://issues.redhat.com/browse/OCPBUGS-15917
where it might be that even a thread doesn't work because
we're locked in the kernel.
2023-07-28 17:59:28 -04:00
Eric Curtin
250c40a671
Merge pull request #2948 from cgwalters/composefs-more-cleanups
composefs: Use lowerdir in /run
2023-07-25 10:25:29 +01:00