Commit Graph

6609 Commits

Author SHA1 Message Date
Colin Walters
05faa1d890
Merge pull request #2877 from ericcurtin/ostree-aboot
Add ostree=aboot for signed Android Boot Images
2023-06-09 07:56:25 -04:00
Alexander Larsson
bb4a89e23c Update submodule: composefs
We will need the new fsverity computation helpers.
2023-06-08 19:14:50 +02:00
Eric Curtin
adf60e26ce Fix read_proc_cmdline_key ("ot-composefs") memory leak
Make it an autofree_char rather than a char
2023-06-08 15:27:16 +01:00
Eric Curtin
aa72caffb5 Add ostree=aboot for signed Android Boot Images
Some kernel images are delivered in a signed kernel + cmdline +
initramfs + dtb blob. When this is added to the commit server side, only
after this do you know what the cmdline is, this creates a recursion
issue. To avoid this, in the case where we have ostree=aboot karg
set, create a symlink after deploy to the correct ostree target in the
rootfs, as the cmdline can't be malleable and secured client-side at
the same time.
2023-06-08 15:27:07 +01:00
Colin Walters
59a653c9dd
Merge pull request #2875 from cgwalters/tempf-always-repo
fetcher: Always open tmpfiles in repo location
2023-06-07 13:34:56 -04:00
Alexander Larsson
0a3dd22f83
Merge pull request #2872 from cgwalters/composefs-followups
Composefs followups
2023-06-07 10:52:06 +02:00
Colin Walters
f7f6f87c51 fetcher: Always open tmpfiles in repo location
In an installation environment (like a live ISO) we may
not have significant space outside of the target installation
repository.

There's no reason not to always open a linkable tempfile.  In
the future we should fix the pull path to verify the checksum
and then just directly link in the object instead of copying.

Closes: https://github.com/ostreedev/ostree/issues/2571
2023-06-06 17:09:30 -04:00
Colin Walters
8762062648
Merge pull request #2874 from aospan/inode64-fix
commit: fix ostree deployment on 64-bit inode fs
2023-06-06 12:37:41 -04:00
Abylay Ospan
de6fddc6ad commit: fix ostree deployment on 64-bit inode fs
This commit addresses a bug that was causing ostree deployment
to become corrupted on the large fs, when any package was installed using
'rpm-ostree install'.

In such instances, multiple files were assigned the same inode. For
example, the '/home' directory and a regular file 'pkg-get' were
assigned the same inode (2147484070), making the deployment unusable.

A root cause analysis was performed, running the process under gdb,
which revealed a lossy conversion from guint64 to guint32, for example
6442451366 converted to 2147484070:

(gdb) p name
$10 = 0x7fe9224d2d70 "home"

(gdb) p inode
$73 = 6442451366

(gdb) s
    device=66311, modifier=0x7fe914791840) at
src/libostree/ostree-repo-commit.c:1590

The conversion resulted in entirely independent files potentially
receiving the same inode.

The issue was discovered on PoC machine equipped with a large NVME
(3.4TB), but the bug can be easily reproduced using `cosa run -m 4000
--qemu-size +3TB', followed by installation of any package using
`rpm-ostree install`. The resulting deployment will be unusable due to
many files being "corrupted" by the aforementioned issue.
2023-06-06 13:37:47 +00:00
Colin Walters
12b7f328e2 prepare-root: More logging in composefs, minor cleanup
- Hoist the `.ostree.cfs` to a shared constant
- Add more logging in general for extra visibility
2023-06-05 13:17:58 -04:00
Colin Walters
624b1084ab prepare-root: Add another missing O_CLOEXEC 2023-06-02 13:10:12 -04:00
Colin Walters
624512f9a7 composefs: Factor out a shared helper for setting error
To keep the error messages consistent.
2023-06-02 10:59:34 -04:00
Colin Walters
1f9607a8e9 mount-util: Add missing O_CLOEXEC
Seen in review.
2023-06-02 09:28:19 -04:00
Colin Walters
b6c054e1fa
Merge pull request #2640 from alexlarsson/composefs
Add initial composefs integration
2023-06-02 09:26:04 -04:00
Colin Walters
67929db1dc
Merge pull request #2871 from dustymabe/dusty-fallocate-einval
lib/deploy: skip fallocate call when requested size is 0
2023-06-01 10:34:13 -04:00
Dusty Mabe
68d1d9a7fc
lib/deploy: skip fallocate call when requested size is 0
If the requested size is 0 then of course we have enough room 🙂

This avoids the fallocate call returning an EINVAL.

Closes: #2869
2023-06-01 09:25:35 -04:00
Colin Walters
5eacc96df3
Merge pull request #2870 from dustymabe/dusty-log-messages
lib/deploy: Disambiguate error messages for early prune space check
2023-06-01 08:24:48 -04:00
Dusty Mabe
a51535b0cd
lib/deploy: Disambiguate error messages for early prune space check
Having the same error message in multiple places means it's not
clear which case failed. Let's make them unique.
2023-06-01 00:00:28 -04:00
Alexander Larsson
7333803949 composefs: When using signatures, delay application until first boot
We can't safely apply the fs-verity with signature until we have
booted with the new initrd, because the public key that matches the
signature is loaded from it. So, instead we save the .sig file next
to the compoosefs, and on the first boot we detect that it is there, and
the composefs file isn't fs-verity, so we apply it.

Things get a bit more complex due to having to temporarily make
/sysroot read-write for the fsverity operation too.
2023-05-31 18:35:44 +02:00
Alexander Larsson
6d2dc95968 CI: Build with composefs on some versions
This enables --with-composefs on:
 * Fedora Latest
 * Debian Testing
 * Ubuntu Latest

These all should have new enough version of dependencies.
2023-05-31 10:57:37 +02:00
Alexander Larsson
e3be4ee52a Update submodule: composefs
Instead of using pkg-config, etc we just include composefs.
In the end the library is just 5 c source files, and it is set up
to be easy to use as a submodule.

For now, composefs support is disabled by default.
2023-05-31 10:57:37 +02:00
Alexander Larsson
f9bdc66649 ostree-remount: Don't skip remount if root is composefs
When using composefs the root fs will always be read-only, but in this
case we should still continue remounting /sysroot. So, we record a
/run/ostree-composefs-root.stamp file in ostree-prepare-root if composefs
is used, and then react to it in ostree-remount.
2023-05-31 10:57:37 +02:00
Alexander Larsson
d47a90347b sysroot: Ensure deployment detection works when using composefs
In the case of composefs, we cannot compare the devino of the rootfs
and the deploy dir, because the root is the composefs mount, not a
bind mount. Instead we check the devino of the etc subdir of the
deploy, because this is a bind mount even when using composefs.
2023-05-31 10:57:37 +02:00
Alexander Larsson
11d7587e40 prepare-root: Support using composefs as root filesystem
This changes ostree-prepare-root to use the .ostree.cfs image as a
composefs filesystem, instead of the checkout.

By default, composefs is used if support is built in and the .ostree.cfs
file exists in the deploy dir, otherwise we fall back to the old
method. However, if the ot-composefs kernel option is specified this
can be tweaked as per:
 * off: Never use composefsz
 * maybe: Use if possible
 * on: Fail if not possible
 * signed: Fail if the cfs image is not fs-verity signed with
   a key in the keyring.
 * digest=....: Fail if the cfs image does not match the specified
   digest.

The final layout when composefs is active is:

 /        ro overlayfs mount for composefs
 /sysroot "real" root
 /etc     rw bind mount to $deploydir/etc
 /var     rw bind mount to $vardir

We also specify the $deploydir/.ostree-mnt directory as the (internal)
mountpoint for the erofs mount for composefs. This can be used to map
the root fs back to the deploy id/dir in use,

A further note: I didn't test the .usr-ovl-work overlayfs case, but a
comment mentions that you can't mount overlayfs on top of a readonly
mount. That seems incompatible with composefs. If this is needed we
have to merge that with the overlayfs that composefs itself sets up,
which is possible with the libcomposefs APIs.
2023-05-31 10:57:37 +02:00
Alexander Larsson
bba3109fe2 switchroot: Make read_proc_cmdline_ostree() take a key argument
This changes it into read_proc_cmdline_key(), as this will later be
used to read additional keys.
2023-05-31 10:57:37 +02:00
Alexander Larsson
3fcebe454e composefs deploy: Store cfs signature in .ostree.cfs.sig file
In many cases, such as when using osbuild, we are not preparing the final
deployment but rather a rootfs tree that will eventually be copied to the
final location. In that case we don't want to apply the signature directly
but when the deployment is copied in place.

To make this situateion workable we also write the signature to a file
next to the composefs image file. Then whatever mechanism that does
the final copy can apply the signature.
2023-05-31 10:57:37 +02:00
Alexander Larsson
c988ff7938 deploy: Write a .ostree.cfs composefs image in the deploy dir
This can be used as a composefs source for the root fs instead of
the checkout by pointing the basedir to /ostree/repo/objects.

We only write the file is `composefs` is enabled.

We enable ensure_rootfs_dirs when building the image which adds the
required root dirs to the image. In particular, this includes /etc
which often isn't in ostree commits in use.

We also create an (empty) .ostree.mnt directory, where composefs
will mount the erofs image that will be used as overlayfs lowerdir
for the root overlayfs mount. This way we can find the deploy
dir from the root overlayfs mount options.

If the commit has composefs digests recorded we verify those with the
created file. It also applies the fs-verity signature if it is
recorded, unless this is disabled with the
ex-integrity.composefs-apply-sign=false option.
2023-05-31 10:55:14 +02:00
Alexander Larsson
0c3d9894be Commit: Add composefs digest and sig to the commit metadata
If `composefs-apply-sig` is enabled (default no) we add an
ostree.composefs digest to the commit metadata. This can be verified
on deploy.

This is a separate option from the generic `composefs` option which
controls whether composefs is used during deploy. It is separate
because we want to not have to force use of fs-verity, etc during the
build.

If the `composefs-certfile` and `composefs-keyfile` keys in the
ex-integrity group are set, then the commit metadata also gets a
ostree.composefs-sig containing the signature of the composefs file.
2023-05-31 10:55:14 +02:00
Alexander Larsson
e2956e2c08 lib: Add (private) API for checking out commits into a composefs image
This supports checking out a commit into a tree which is then
converted into a composefs image containing fs-verity digests for all
the regular files, and payloads that are relative to a the
`repo/objects` directory of a bare ostree repo.

Some specal files are always created in the image. This ensures that
various directories (usr, etc, boot, var, sysroot) exists in the
created image, even if they were not in the source commit. These are
needed (as bindmount targets) if you want to boot from the image. In
the non-composefs case these are just created as needed in the checked
out deploydir, but we can't do that here.

This is all controlled by the new ex-integrity config section, which
has the following layout:

```
[ex-integrity]
fsverity=yes/no/maybe
composefs=yes/no/maybe
composefs-apply-sig=yes/no
composefs-add-metadata=yes/no
composefs-keyfiile=/a/path
composefs-certfile=/a/path
```

The `fsverity` key overrides the old `ex-fsverity` section if
specified.  The default for all these is for the new behaviour to be
disabled. Additionally, enabling composefs implies fsverity defaults
to `maybe`, to avoid having to set both.
2023-05-31 10:55:14 +02:00
Alexander Larsson
9ba98cd8e9 fsverity: Support passing a signature when enabling fs-verity
The composefs code will need this.
2023-05-31 10:55:14 +02:00
Alexander Larsson
c6ed5cc7b2 fsverity: Add _ostree_fsverity_sign helper
This code signs a fsverity digest (using openssl) such that the
resulting signature can be used with the FS_IOC_ENABLE_VERITY ioctl.
2023-05-31 10:55:14 +02:00
Alexander Larsson
02d24d2a38 Add ot_keyfile_get_tristate_with_default() helper
This parses keys like yes/no/maybe. The introduced OtTristate type
is compatible with the existing _OstreeFeatureSupport type.
2023-05-31 10:55:14 +02:00
Colin Walters
0dd2788410
Merge pull request #2864 from cgwalters/prepare-root-prepare-composefs
prepare-root: Move sysroot.tmp creation earlier
2023-05-30 09:15:40 -04:00
Colin Walters
f903d6af67
Merge pull request #2866 from jlebon/pr/autoprune-tweaks
lib/deploy: Use `fallocate` for early prune space check
2023-05-30 08:38:16 -04:00
Jonathan Lebon
193ef29f3f lib/deploy: Use fallocate for early prune space check
The `f_bfree` member of the `statvfs` struct is documented as the
"number of free blocks". However, different filesystems have different
interpretations of this. E.g. on XFS, this is truly the number of blocks
free for allocating data. On ext4 however, it includes blocks that
are actually reserved by the filesystem and cannot be used for file
data. (Note this is separate from the distinction between `f_bfree` and
`f_bavail` which isn't relevant to us here since we're privileged.)

If a kernel and initrd is sized just right so that it's still within the
`f_bfree` limit but above what we can actually allocate, the early prune
code won't kick in since it'll think that there is enough space. So we
end up hitting `ENOSPC` when we actually copy the files in.

Rework the early prune code to instead use `fallocate` which guarantees
us that a file of a certain size can fit on the filesystem. `fallocate`
requires filesystem support, but all the filesystems we care about for
the bootfs support it (including even FAT).

(There's technically a TOCTOU race here that existed also with the
`statvfs` code where free space could change between when we check
and when we copy. Ideally we'd be able to pass down that fd to the
copying bits, but anyway in practice the bootfs is pretty much owned by
libostree and one doesn't expect concurrent writes during a finalization
operation.)
2023-05-29 12:17:05 -04:00
Jonathan Lebon
76649127d1 lib/deploy: Rename variable for clarity
`size_to_remove` looks cryptic in contrast to
`new_new_bootcsum_dirs_total_size`. Rename it in the style of the latter
for easier reading.
2023-05-28 18:39:03 -04:00
Jonathan Lebon
a3c0d6a3fe lib/deploy: Log case when auto-pruning is hopeless
For easier diagnostics.
2023-05-28 18:38:53 -04:00
Jonathan Lebon
115d5cf073 lib/deploy: Drop unused variable
Noticed this diagnostic in my editor with clangd hooked up.
2023-05-28 18:38:38 -04:00
Jonathan Lebon
632ffa4302 lib/deploy: Initialize var to pacify gcc static analysis
Classic case of analysis getting confused by variables initialized by
a function.
2023-05-27 10:38:14 -04:00
Bartłomiej Piotrowski
99f6356b5b Use a value based on OSTREE_MAX_METADATA_SIZE 2023-05-26 12:09:13 +02:00
Bartłomiej Piotrowski
4bac96a8c8 Increase the metadata size limit to 128MB
Flathub has hit the 10MB limit in 2022, and we had to drop less popular
CPU architectures from the main summary to subsummaries, effectively
cutting off users running too old Flatpak version. Despite that, the
main summary containing only x86_64 is already at 7MB. As this is
eventually going to happen to subsummaries as well, preemptively bump
the limit 12 times.

It takes between 2 and 3 years for a change like this to roll out across
Linux distributions so the best time for this was yesterday.

fixes #2715
2023-05-25 11:49:54 +02:00
Colin Walters
c22576c41d prepare-root: Move sysroot.tmp creation earlier
Main motivation is prep for composefs in
https://github.com/ostreedev/ostree/pull/2640
In the interest of that, we add a `bool using_composefs` but
it's currently always `false`.

Co-authored-by: Alexander Larsson <alexl@redhat.com>
2023-05-24 15:50:38 -04:00
Colin Walters
50790b285e
Merge pull request #2860 from cgwalters/xshell2
tests: A bit more xshell porting
2023-05-22 13:37:10 -04:00
Joseph Marrero Corchado
b5b3ef78af
Merge pull request #2859 from jmarrero/release-2023.3
Release 2023.3
2023-05-18 17:09:04 -04:00
Colin Walters
88e8b671ce tests: A bit more xshell porting
Part of https://github.com/ostreedev/ostree/issues/2857
2023-05-18 08:14:50 -04:00
Joseph Marrero
88fe600ff8 configure: post-release version bump 2023-05-17 16:32:43 -04:00
Joseph Marrero
bf3495dbbf Release 2023.3 2023-05-17 16:29:04 -04:00
Colin Walters
16cb47489e
Merge pull request #2856 from cgwalters/port-to-xshell
tests/inst: Add xshell and use it in one place
2023-05-10 15:16:24 -04:00
Colin Walters
dc23b9389b tests/inst: Add xshell and use it in one place
I've deprecated sh-inline; in the end I think it is better
to minimize the amount of bash code we have.  xshell solves
the core convenience problem of taking local variables and mapping
them to command arguments.

A full port would be nontrivial; this just starts the ball
rolling.
2023-05-10 14:02:09 -04:00
Colin Walters
8a2993a9d0
Merge pull request #2565 from cgwalters/clang-format
Add .clang-format file + tree-wide: Run `clang-format`
2023-05-03 08:24:21 -04:00