Commit Graph

1321 Commits

Author SHA1 Message Date
Dan Nicholson
60f4592b2c repo: Make summary and signature mtime match
HTTP servers derive Last-Modified from the modification time of the
file. When used in combination with a Cache-Control max-age value,
having the modification times match means that caches will consider them
expired at the same time. This helps make it more likely that clients
won't receive a cached summary and fresh signature or vice versa.

This makes more sense to do now that the summary and signature are
created in a temporary directory and renamed into place. In the old days
where they were created directly in the repo root, it would be strange
to change the summary mtime when it wasn't actually modified.
2024-04-25 09:38:50 -06:00
Dan Nicholson
ffbeff6cd7 tests: Correctly skip single fsverity test
The skip shell function is for skipping an entire test plan. To skip a
single test result, a directive is needed[1]. Without this change, the
test suite errors claiming that 2 test plans were provided when fsverity
isn't available.

1. https://testanything.org/tap-specification.html#skipping-tests
2024-04-24 21:57:23 -06:00
Colin Walters
99ef9806e2
Merge pull request #3230 from cgwalters/initfs-epoch-2
init-fs: Add --epoch=2
2024-04-12 19:56:35 -04:00
Colin Walters
3647395fe5 tests: Skip composefs tests without the feature
Previously this was masked by us shipping composefs vendored.
2024-04-12 16:16:21 -04:00
Colin Walters
74a3d2da9c init-fs: Add --epoch=2
We want to start switching things so that the toplevel `/ostree`
repository is mode 0700, to close off unprivileged code
from being able to access it.  Previous deployment roots
may have setuid binaries, etc.  The `/var/lib/containers/storage`
directory is mode 0700 for this reason I believe.

Closes: https://github.com/ostreedev/ostree/issues/3211
2024-04-12 16:10:36 -04:00
Colin Walters
38f6b9b3ab init-fs: Add --epoch
I want to add another variant here, and `--modern` is now old.  Let's
acknowledge that we may want to make even more changes in the
future.  So `--modern == --epoch=1` but I will add `--epoch=2` after
this.
2024-04-12 08:56:13 -04:00
Colin Walters
bb6dfd3ab0 checkout: Always replace existing content in overlay mode
The combination of the "honor whiteout" and "union" flags
are intended to basically be "merge trees like overlayfs does".
But we were missing this case in order to support e.g. replacing
a symlink with a directory.
2024-03-13 14:07:54 -04:00
HuijingHei
abc7d5b9a0 kargs: parse spaces in kargs input and keep quotes
According to Jonathan's suggestion, should fix the code from
ostree repo.

With this patch:
- kargs input like "init_on_alloc=1 init_on_free=1", will be
parsed as 2 seperated args `init_on_alloc=1` and `init_on_free=1`,
instead of whole;
- According to https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html,
need to keep spaces in double-quotes, like `param="spaces in here"`
will be parsed as whole instead of 3.

Fixes https://github.com/coreos/rpm-ostree/issues/4821
2024-03-08 10:01:06 +08:00
Colin Walters
2fb7a04838 sysroot: Turn on bootloader-naming-2 by default
I think it's about time we flipped this on by default;
like the bootprefix I was a bit too chicken.  We still have
a `bootloader-naming-1` that can be flipped on in case of
some regression.

Closes: https://github.com/ostreedev/ostree/issues/2961
2024-02-28 09:40:34 -05:00
Colin Walters
e47b370963 bootloader/grub2: Don't do anything if we have static configs
This builds on top of fa9924e4fe
(But in a very hacky way because we don't currently link to a JSON library)

Basically, bootupd supports injecting static configs, and this
is the currently least hacky way for us to detect this and understand
that we shouldn't try to run `grub2-mkconfig`.

A further patch I'd like to do here is also change the probing
logic to gracefully no-op if `grub2-mkconfig` doesn't exist,
but that has a bit more risk and involvement.
2024-02-27 14:21:26 -05:00
Eric Curtin
e48cdb90d3 sysroot: Reword comment and use gboolean over bool, error handling
Be more explicit in the comment, and use gboolean over bool. Less header
inclusions when we use gboolean. Although bool is used in some places.
Write a separate _ostree_sysroot_parse_bootlink_aboot function for
aboot. Make is_aboot optional. Handle invalid androidboot karg and no
ostree and androidboot kargs differently.

Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>
Signed-off-by: Eric Curtin <ecurtin@redhat.com>
2024-02-23 14:58:48 +00:00
Eric Curtin
4a71845b12 generator: Fixes for Android Boot environment
In Android Boot environment we do not parse ostree= karg to determine
what directory to boot into, alternatively we do this based on the
androidboot.slot_suffix= karg. But we do set ostree=true karg to denote
that we are indeed booting an ostree environment (required for some
systemd unit files). This change accounts for this approach in the
systemd generator. In this case androidboot.slot_suffix= points you to
/ostree/root.[a|b] and then that points you to the directory to boot
into in /ostree/deploy... Here is what a cmdline may look like in this
type of environment:

androidboot.slot_suffix=_a androidboot.bootdevice=*.ufshc root=PARTLABEL=system_a root=UUID=76a22bf4-f153-4541-b6c7-0332c0dfaeac rw ostree=true loglevel=4 acpi=off console=ttyAMA0 systemd.show_status=auto libahci.ignore_sss=1 slub_debug=FPZ fsck.mode=skip rcupdate.rcu_normal_after_boot=0 rcupdate.rcu_expedited=1

Signed-off-by: Eric Curtin <ecurtin@redhat.com>
2024-02-21 20:54:55 +00:00
Simon McVittie
035b2c1647 test-admin-deploy-var: Don't rely on OSTREE_FEATURES
This is set during build-time testing, but unset during "as-installed"
tests.

Resolves: https://github.com/ostreedev/ostree/issues/3183
Signed-off-by: Simon McVittie <smcv@debian.org>
2024-02-19 21:01:15 +00:00
Simon McVittie
37466ec35b tests: Use skip_without_ostree_feature to detect libarchive, composefs
This avoids false negatives from `ostree --version | grep -q ...`
exiting with failure under `set -o pipefail` because `grep -q` can exit
as soon as it sees the desired string, leaving `ostree --version` to be
terminated by `SIGPIPE` next time it writes to stdout.

Signed-off-by: Simon McVittie <smcv@collabora.com>
2024-02-19 21:01:07 +00:00
Simon McVittie
a84e56d603 tests: Generalize has_gpgme, has_sign_ed25519 into has_ostree_feature
Signed-off-by: Simon McVittie <smcv@collabora.com>
2024-02-19 21:01:05 +00:00
Justus Winter
9b5a373adb
tests: Use long key IDs
Short key IDs are not secure, and may be rejected by OpenPGP
implementations.  See https://evil32.com/

Signed-off-by: Justus Winter <justus@sequoia-pgp.org>
2024-02-16 16:24:18 +01:00
Justus Winter
ad8c9f9817
tests: Use long key IDs
Short key IDs are not secure, and may be rejected by OpenPGP
implementations.  See https://evil32.com/

Signed-off-by: Justus Winter <justus@sequoia-pgp.org>
2024-02-16 13:34:34 +01:00
Colin Walters
ed4bd88a3e rofiles-fuse: Check fsverity flag for copyup
We need to do a copyup if fsverity is enabled.
Sadly to do this we can't just use ostree_break_hardlink
as is.
2024-02-15 08:03:16 -05:00
Colin Walters
87dcc801a2 ostree-tmpfiles.conf: Drop var entry
We are backing away from this semantic, and moving towards
`/var` only being initialized at initial provisioning.
2024-02-12 13:12:09 -05:00
Colin Walters
f81b9fa166 sysroot: Rework /var handling to act like Docker VOLUME /var
We've long struggled with semantics for `/var`.  Our stance of
"/var should start out empty and be managed by the OS" is a strict
one, that pushes things closer to the original systemd upstream
ideal of the "OS state is in /usr".

However...well, a few things.  First, we had some legacy bits
here which were always populating the deployment `/var`.  I don't
think we need that if systemd is in use, so detect if the tree
has `usr/lib/tmpfiles.d`, and don't create that stuff at
`ostree admin stateroot-init` time if so.

Building on that then, we have the stateroot `var` starting out
actually empty.

When we do a deployment, if the stateroot `var` is empty,
make a copy (reflink if possible of course) of the commit's `/var`
into it.

This matches the semantics that Docker created with volumes,
and this is sufficiently simple and easy to explain that I think
it's closer to the right thing to do.

Crucially...it's just really handy to have some pre-existing
directories in `/var` in container images, because Docker (and podman/kube/etc)
don't run systemd and hence don't run `tmpfiles.d` on startup.

I really hit on the fact that we need `/var/tmp` in our container
images by default for example.

So there's still some overlap here with e.g. `/usr/lib/tmpfiles.d/var.conf`
as shipped by systemd, but that's fine - they don't actually conflict
per se.
2024-02-09 17:46:12 -05:00
Colin Walters
cae4ceb6c5 deploy: Honor prepare-root.conf at deploy time
I want to try to get away from the "repository global" configuration
in the repo config.

A major problem is that there's not an obvious way to configure
it as part of an ostree commit/container build - it needs
to be managed "out of band".

With this change, we parse the `usr/lib/ostree/prepare-root.conf`
in the deployment root, and if composefs is enabled there,
then we honor it.

We do still honor `ex-integrity.composefs` but that I think
we can schedule to remove.
2024-02-08 19:53:23 -05:00
Colin Walters
4c813f3221 Revert "Enable sysroot.bootprefix by default"
This reverts commit 8627c8afa1.

See discussion in https://github.com/ostreedev/ostree/pull/3156 ;
we think this breaks s390x in some cases at least, and that warrants
further investigation.
2024-02-07 15:58:06 -05:00
Colin Walters
8627c8afa1 Enable sysroot.bootprefix by default
I've been testing this in various places and not seen any fallout,
so let's finally enable this by default and have the situation where
`/boot` is on the root `/` filesystem work out of the box.
2024-02-06 16:25:33 -05:00
Simon McVittie
b42f053db9 tests: Skip composefs test if /var/tmp does not support user xattrs
Otherwise, this test fails on Debian 12 (Linux 6.1) kernels if /var/tmp
is a tmpfs. Some autobuilders put the entire build chroot on a tmpfs,
to speed up builds.

Signed-off-by: Simon McVittie <smcv@debian.org>
2024-01-24 14:03:18 +00:00
Jonathan Lebon
92b1a27202 Add concept of state overlays
In the OSTree model, executables go in `/usr`, state in `/var` and
configuration in `/etc`. Software that lives in `/opt` however messes
this up because it often mixes code *and* state, making it harder to
manage.

More generally, it's sometimes useful to have the OSTree commit contain
code under a certain path, but still allow that path to be writable by
software and the sysadmin at runtime (`/usr/local` is another instance).

Add the concept of state overlays. A state overlay is an overlayfs
mount whose upper directory, which contains unmanaged state, is carried
forward on top of a lower directory, containing OSTree-managed files.

In the example of `/usr/local`, OSTree commits can ship content there,
all while allowing users to e.g. add scripts in `/usr/local/bin` when
booted into that commit.

Some reconciliation logic is executed whenever the base is updated so
that newer files in the base are never shadowed by a copied up version
in the upper directory. This matches RPM semantics when upgrading
packages whose files may have been modified.

For ease of integration, this is exposed as a systemd template unit which
any downstream distro/user can enable. The instance name is the mountpath
in escaped systemd path notation (e.g.
`ostree-state-overlay@usr-local.service`).

See discussions in https://github.com/ostreedev/ostree/issues/3113 for
more details.
2024-01-09 23:20:41 -05:00
Jonathan Lebon
4982306e67 lib/deploy: Round to block size in early prune space check
When we estimate how much space a new bootcsum dir will use, we
weren't accounting for the space overhead from files not using the
last filesystem block completely. This doesn't matter much if counting
a few files, but e.g. on FCOS aarch64, we include lots of small
devicetree blobs in the bootfs. That loss can add up to enough for the
`fallocate()` check to pass but copying still hitting `ENOSPC` later on.

I think a better fix here is to change approach entirely and instead
refactor `install_deployment_kernel()` so that we can call just the
copying bits of it as part of the early prune logic. We'll get a more
accurate assessment and it's not lost work since we won't need to
recopy later on. Also this would not require having to keep in sync the
estimator and the install bits.

That said, this is blocking FCOS releases, so I went with a more tactical
fix for now.

Fixes: https://github.com/coreos/fedora-coreos-tracker/issues/1637
2024-01-04 12:57:43 -05:00
Colin Walters
5b23804a1a Release 2023.8 2023-12-05 13:37:04 -05:00
Colin Walters
c3dff62e1c commit: Try reflinks for local commits by default
I think we originally used to do this, but at some point in a
code refactoring, this optimization got lost.

It's a quite important optimization for the case of writing content
generated by an external system into an ostree repository.
2023-12-04 20:45:08 -05:00
Colin Walters
bab87cbd9d tests: Drop bare-user on tmpfs negative test
As of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2daf18a7884dc03d5164ab9c7dc3f2ea70638469
the restriction is lifted, FINALLY.
2023-12-03 09:32:36 -05:00
Colin Walters
66064d960d tmpfiles: Copy /usr/share/factory/var to /var
This is a pattern we want to encourage.  It's honestly just
way simpler than what rpm-ostree is doing today in auto-synthesizing
individual tmpfiles.d snippets.
2023-11-30 18:11:33 -05:00
Colin Walters
d3de3a0a88 tests: Add a dedicated finalization test
This one covers `admin lock-finalization --unlock`.
2023-11-28 18:45:11 -05:00
Colin Walters
28cc761806 sysroot: Stabilize deployment finalization, add API and CLI
It's about time we do this; deployment finalization locking
is a useful feature.  An absolutely key thing here is that
we've slowly been moving towards the deployments as the primary
"source of truth".

Specifically in bootc for example, we will GC container images
not referenced by a deployment.

This is then neecessary to support a "pull but don't apply automatically" model.

This stabilizes the existing `ostree admin deploy --lock-finalization`
CLI, and adds a new `ostree admin unlock-finalization`.

We still check the old lock file path, but there's a new boolean
value as part of the staged deployment data which is intended
to be the source of truth in the future.  At some point then we
can drop the rpm-ostree lockfile handling.

Closes: https://github.com/ostreedev/ostree/issues/3025
2023-11-27 10:59:56 -05:00
Colin Walters
b6f435fe11 tests: Turn off gpg verification for dev builds
Right now `ostree admin status` errors out in this case, but
`rpm-ostree status` doesn't.  The former behavior is probably
more of a bug, work around it for now.
2023-11-27 10:59:56 -05:00
Eric Curtin
29973774ea
Merge pull request #3091 from cgwalters/fix-soup3
build-sys: Enable libsoup3 by default if installed
2023-11-10 16:39:13 +00:00
Colin Walters
bc62fd5196 unlock: Don't pass options again to overlayfs
There seems to be a tricky regression here with the util-linux
support for the new mount API, plus overlays support for it.

```
[2023-11-09T21:05:30.633Z] Nov 09 21:05:26 qemu0 kola-runext-unlock-transient.sh[2108]: + unshare -m -- /bin/sh -c 'mount -o remount,rw /usr && echo hello from transient unlock >/usr/share/writable-usr-test'
[2023-11-09T21:05:30.633Z] Nov 09 21:05:26 qemu0 kola-runext-unlock-transient.sh[2148]: mount: /usr: mount point not mounted or bad option.
[2023-11-09T21:05:30.633Z] Nov 09 21:05:26 qemu0 kola-runext-unlock-transient.sh[2148]:        dmesg(1) may have more information after failed mount system call.
```

OK this seems related to the new mount API support in util-linux and overlayfs.  From a strace:

```
2095  open_tree(AT_FDCWD, "/usr", OPEN_TREE_CLOEXEC) = 3
2095  mount_setattr(-1, NULL, 0, NULL, 0) = -1 EINVAL (Invalid argument)
...
2095  fspick(3, "", FSPICK_NO_AUTOMOUNT|FSPICK_EMPTY_PATH) = 4
2095  fsconfig(4, FSCONFIG_SET_FLAG, "seclabel", NULL, 0) = 0
2095  fsconfig(4, FSCONFIG_SET_STRING, "lowerdir", "usr", 0) = -1 EINVAL (Invalid argument)
```

I think the core problem here is it's trying to reconfigure the mount with existing options,
but in the new mount namespace we can't see the lowerdir.

Here we really really just want to remount writable.  Telling
util-linux to not pass existing options fixes it.
2023-11-09 20:53:52 -05:00
Colin Walters
c668a8c531 tests: Work around systemd regression for boot id parsing
cc https://github.com/systemd/systemd/issues/29275
2023-11-09 15:10:33 -05:00
Colin Walters
c3cf5b237d undeploy: Parse integer more properly
`atoi` doesn't offer any error checking.

Closes: https://github.com/ostreedev/ostree/issues/3088
2023-11-08 16:15:01 -05:00
Colin Walters
b9e73a3813
Merge pull request #3078 from HuijingHei/karg-delete-array
karg-delete: support multiple times
2023-10-17 09:44:24 -04:00
HuijingHei
003cb99c32 karg-delete: support multiple times
Fix https://github.com/ostreedev/ostree/pull/2612#issuecomment-1764412141
2023-10-17 20:22:08 +08:00
Colin Walters
d788c5b5c9
Merge pull request #3075 from cgwalters/reenable-composefs
tests: Use ext4, re-enable composefs test
2023-10-16 10:24:10 -04:00
Colin Walters
ac69c70750 deploy: Remove lock when re-staging
This closes the biggest foot-gun when doing e.g.
`rpm-ostree rebase` when zincati is running on a FCOS system.

Previously if zincati happened to have staged + locked a deployment,
we'd keep around the lock which is definitely not what is desired.
2023-10-13 13:45:04 -04:00
Colin Walters
833251cb0d tests: Use ext4, re-enable composefs test
Until the XFS fsverity stuff lands.
2023-10-13 12:00:05 -04:00
Colin Walters
265b7f9ecc deploy: Improve error message for nonexistent stateroot
Came up on an internal chat; previously we were only erroring
out when trying to do the SELinux labeling for `/var` which
was really misleading.

Add some other error prefixing while we have the patient open.
2023-10-12 13:22:35 -04:00
Colin Walters
cd0fc9f4f8
Merge pull request #3072 from alexlarsson/fix-whiteout-test
tests: Fix whiteout test
2023-10-12 10:25:40 -04:00
Alexander Larsson
397a11762e tests: Fix whiteout test
This test was always skipped, because the check:

 if touch overlay/baz/.wh.cow &&
    touch overlay/.wh.deeper &&
    touch overlay/baz/another/.wh..wh..opq; then

always fails due to the missing overlay/baz/another directory.
Fix by creating the directory.
2023-10-12 10:18:36 +02:00
Colin Walters
81c08746c6 repo: Add an option to label /usr/etc as /etc
This will be very useful for enabling a "transient /etc" option
because we won't have to do hacks relabling in the initramfs, or
forcing it on just for composefs.
2023-10-11 13:25:08 -04:00
Colin Walters
734ea3ee37 ci: Disable composefs test for now
Need to change the framework to boot with an Ignition config
that switches to ext4 soon.
2023-10-11 09:16:45 -04:00
Owen W. Taylor
3b2fd6e9ff When exporting, use hardlinks for duplicated files
For ostree_repo_export_tree_to_archive(), and 'ostree export', when the
exported tree contains multiple files with the same checksum, write an
archive with hard links.

Without this, importing a tree, then exporting it again breaks
hardlinks.

As an example of savings: this reduces the (compressed) size of the
Fedora Flatpak Runtime image from 1345MiB to 712MiB.

Resolves: #2925
2023-09-29 13:45:29 -04:00
Colin Walters
372cbd7a64 tests: Add an integration test for composefs signatures
Ensure we have some automated test coverage for this.
2023-09-21 07:53:19 -04:00
Colin Walters
e3291ccce9 tests: Rework detection of trivial-httpd
Because it's now at the toplevel.
2023-09-19 16:52:13 -04:00