Commit Graph

6646 Commits

Author SHA1 Message Date
Colin Walters
4e61e6f7d0 Revert "fetcher: Always open tmpfiles in repo location"
This reverts commit f7f6f87c51.

This seems to have broken flatpak, so we'll revert and then
investigate.

Closes: https://github.com/ostreedev/ostree/issues/2900
2023-06-26 21:03:10 -06:00
Dan Nicholson
7a91035c09
Merge pull request #2903 from dbnicholson/composefs-digest-fix
prepare-root: Adjust to composefs mount struct changes
2023-06-26 17:08:48 -06:00
Dan Nicholson
94b49b41e7 prepare-root: Adjust to composefs mount struct changes
This fixes a regression from the latest composefs submodule update in
1582edd1d4. In composefs commit
7560a4fd388481f479c0b3fc2e6d20c6321d9b74 the struct field was changed
from the generic `expected_digest` with the thought that there may be
other signatures or digests in the future.
2023-06-26 09:47:00 -06:00
Eric Curtin
4429923681 bootloader: Pass "options" to aboot bootloader backend
aboot-deploy must know where the next root filesystem to boot is to set
up a symlink /ostree/root.a or /ostree/root.b , this location is in the
ostree= part of these passed in options.

Signed-off-by: Eric Curtin <ecurtin@redhat.com>
2023-06-26 11:44:56 +01:00
Colin Walters
363a1f14a8
Merge pull request #2896 from akiernan/main
lib/deploy: Use off_t not __off_t
2023-06-21 08:06:52 -04:00
Alex Kiernan
e0342d2da1 lib/deploy: Use off_t not __off_t
Not clear why this was __off_t which is the sole appearance in the code
base, but it breaks musl builds.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
2023-06-21 07:31:14 +01:00
Joseph Marrero Corchado
24dd9bbe02
Merge pull request #2895 from cgwalters/release
Release 2023.4
2023-06-20 12:25:17 -04:00
Joseph Marrero Corchado
f07c93e167
Merge pull request #2893 from cgwalters/update-prow-config
ci: Add some composefs testing
2023-06-20 12:22:49 -04:00
Colin Walters
ebb9624204
Merge pull request #2892 from ostreedev/dependabot/submodules/composefs-08bdb03
build(deps): bump composefs from `c9188cd` to `08bdb03`
2023-06-20 11:01:57 -04:00
Colin Walters
a6035125d5 configure: post-release version bump 2023-06-20 09:39:43 -04:00
Colin Walters
0c35149c7e Release 2023.4 2023-06-20 09:39:04 -04:00
Colin Walters
a113b2be3f
Merge pull request #2891 from cgwalters/commit-compute-composefs-digest
lib: Rework composefs metadata, drop custom signatures
2023-06-20 07:44:38 -04:00
Colin Walters
6df69880de ci: Sync prow config with rpm-ostree, enable composefs there
I want to gain testing over the composefs path; but without
yet changing the main Jenkins build.  Because we have duplicate/overlapping
CI systems, we can take advantage of this by testing the composefs
flow via Prow.

Sync the Prow build scripts with what's in coreos/rpm-ostree
right now.

Then let's test the composefs flow a bit more e2e there.
2023-06-19 19:07:01 -04:00
Colin Walters
3c7e256cee tests: Add a sanity check for composefs
Prep for adding some coverage of this flow when booting with
composefs.
2023-06-19 19:02:20 -04:00
Colin Walters
b56da3409d docs/composefs: Updates
- fix URL
- Document requirements
- Document kernel argument
- Adjust for recent changes
2023-06-19 17:29:31 -04:00
dependabot[bot]
1582edd1d4
build(deps): bump composefs from c9188cd to 08bdb03
Bumps [composefs](https://github.com/containers/composefs) from `c9188cd` to `08bdb03`.
- [Release notes](https://github.com/containers/composefs/releases)
- [Commits](c9188cd1f8...08bdb030fc)

---
updated-dependencies:
- dependency-name: composefs
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-19 12:59:34 +00:00
Colin Walters
a6f2d053c8 lib: Rework composefs metadata, drop custom signatures
We will be switching to handling signature verification of
the target ostree commit.
2023-06-17 17:08:46 -04:00
Colin Walters
dd47090e2f
Merge pull request #2890 from cgwalters/composefs-err-frees
composefs: Avoid double free in error path
2023-06-17 11:30:49 -04:00
Colin Walters
79adccee38 composefs: Avoid double unref
The interaction of `iter_loop` and autofree is way too subtle;
I happened to be reading this code and noticed we did the NULL
reset in one path but not another.

The real fix is Rust...
2023-06-17 09:11:51 -04:00
Joseph Marrero Corchado
1fe2bb9f5a
Merge pull request #2885 from cgwalters/doc-composefs
composefs: Add some basic docs
2023-06-16 14:26:01 -04:00
Colin Walters
95f2c72c9f
Merge pull request #2886 from daissi/wip/daissi/disable-http2
ostree-fetcher-curl: explicitly use HTTP1.1 when HTTP2 is disabled
2023-06-15 17:35:45 -04:00
Dan Nicholson
186055d6dd
Merge pull request #2888 from cgwalters/more-error-prefixing
Add more error prefixing when parsing commit objects
2023-06-15 13:25:12 -06:00
Colin Walters
c71fc3d994 Add more error prefixing when parsing commit objects
I've got more debug information in the error case that motivated
bae4347abe
"pull: Add error prefixing for corrupt checksums"
where the sole error is

`error: Invalid checksum of length 0 expected 32`

This must be coming from the pull code in the case where we've
already fetched the commit object.

- Add some error prefixing here in the core commit validation code
- Ensure that we do the validation immediately after loading, including
  of the parent commit reference where I think this error must be coming
  from
- Then the pull code can just safely call `ostree_commit_get_parent`
  which already does the hex conversion etc.
2023-06-15 14:39:02 -04:00
Dylan Aïssi
d1ec6cdf86 ostree-fetcher-curl: explicitly use HTTP1.1 when HTTP2 is disabled
The current logic to select the HTTP version to be use relies on
the fact that curl don't use by default HTTP2. This assumption seems wrong with
recent versions of curl.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>
2023-06-15 15:41:16 +02:00
Colin Walters
91c6b1a6d2 composefs: Add some basic docs
Let's describe the state of things at a high level, independent
of the tracking issue which has a lot more detail (and hence noise).

This document keeps things at a high level and describes how to
enable things today.
2023-06-14 16:41:48 -04:00
Dan Nicholson
ff805f1c4f
Merge pull request #2880 from kraj/main
fix build with lld linker
2023-06-14 12:09:47 -06:00
Khem Raj
c3bd439d3e libostree: Link with libgpg-error for gpg_strerror_r API
With f461c02bb5 use of gpg_strerror_r
was added this symbol comes from libgpg-error however, therefore its
needed to add -lgpg-error to cmdline to resolve this symbol especially
with gold and lld linker. Fixes

aarch64-yoe-linux-ld.lld: error: undefined reference due to --no-allow-shlib-undefined: gpg_strerror_r
>>> referenced by ./.libs/libostree-1.so
2023-06-14 09:39:34 -07:00
Dan Nicholson
c555ad7556
Merge pull request #2884 from cgwalters/errprefix-pull-checksums
pull: Add error prefixing for corrupt checksums
2023-06-14 10:29:46 -06:00
Colin Walters
bae4347abe pull: Add error prefixing for corrupt checksums
I got a bug report with the bare error
```
error: Invalid checksum of length 0 expected 32
```

And I'm pretty sure it's from here.  Add error prefixing so we
know exactly which metadata object was corrupt.
2023-06-14 11:19:44 -04:00
Colin Walters
8971bd05ee
Merge pull request #2881 from ostreedev/dependabot/submodules/composefs-c9188cd
build(deps): bump composefs from `af8e1a7` to `c9188cd`
2023-06-14 09:36:06 -04:00
Dan Nicholson
e79591501c
Merge pull request #2882 from cgwalters/more-test-concurrency
test-concurrency: Don't lower timeout
2023-06-13 16:36:32 -06:00
Colin Walters
7892818fb1 test-concurrency: Don't lower timeout
I think there's never been a real race condition here.  Instead
the problem is:

- We have a timeout on the lock, after which we error out (30s)
- This test actually *lowers* the timeout

Looking through the failures for test-concurrency what I see
is
`error: Locking repo exclusive failed: Resource temporarily unavailable`
which is us hitting the timeout.

Hardcoded timeouts are just going to be subject to race conditions.
I understand not wanting to block forever in some cases, but any
arbitrary timeout is just going to get hit in real world conditions
too.

Anyways for now, stop shooting ourselves in the foot and at least
keep the timeout at the default.

Closes: https://github.com/ostreedev/ostree/issues/2038
2023-06-13 16:17:16 -04:00
Dan Nicholson
9244518896
Merge pull request #2865 from barthalion/increase-metadata-size
Increase the metadata size limit to 128MB
2023-06-13 09:02:48 -06:00
Bartłomiej Piotrowski
40c07362c1 Update big metadata size in test-pull-large-metadata 2023-06-13 15:46:56 +02:00
dependabot[bot]
f142b7deea
build(deps): bump composefs from af8e1a7 to c9188cd
Bumps [composefs](https://github.com/containers/composefs) from `af8e1a7` to `c9188cd`.
- [Release notes](https://github.com/containers/composefs/releases)
- [Commits](af8e1a7cf6...c9188cd1f8)

---
updated-dependencies:
- dependency-name: composefs
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-13 12:58:56 +00:00
Alexander Larsson
c4591c2d28
Merge pull request #2879 from alexlarsson/composefs-new-signature-approach
composefs: Change how we do signatures
2023-06-10 17:36:03 +02:00
Alexander Larsson
2d476611a0 composefs: Change how we do signatures
Currently we generate a signature for the actual composefs image, and
then we apply that when we enable fsverity on the composefs
image. However, there are some issues with this.

First of all, such a signed fs-verity image file can only be read if
the corresponding puiblic keyring is loaded into the fs-verity
keyring. In a typical secure setup we will have a per-commit key that
is loaded from the initrd. Additionally, the keyring is often sealed
to avoid loading more keys later.

This means you can only ever mount (or even look at) composefs images
from the current boot. While this is not a huge issue it is something
of a pain for example when debugging things.

Secondly, and more problematic, during a deploy we can't enable
fs-verity on the newly created composefs file, because and at that
point you need to pass in the signature. Unfortunately this will fail
if the matching public key is not in the keyring, which will fail for
similar reasons as the first issue.

The current workaround is to *not* enable fs-verity during deploy, but
write the signature to a file. Then the first time the particular
commit is booted we apply the signature to the iamge. This works
around issue two, but not issue one. But it causes us to do a lot of
writes and computation during the first boot as we need to write the
fs-verity merkle tree to disk. It would be much better and robust if
the merkle tree could be written during the deployment of the update
(i.e. before boot).

The new apporach is to always deploy an unsigned, but fs-verity
enabled composefs image. Then we create separate files that contain
the expected digest, and a signature of that file. On the first boot
we sign the digest file, and on further boots we can just verify
that it is signed before using it.

This fixes issue 1, since all deploys are always readable, and it
makes the workaround for issue 2 much less problematic, as we only
need to change a much smaller file on the first boot.

Long term I would like to avoid the first-boot writing totally, and
I've been chatting with David Howells (kernel keyring maintainer) and
he proposed adding a new keyring syscall that verifies a PKCS#7
signature from userspace directly. This would be exactly what
fs-verity does, except we wouldn't have to write the digest to disk
during boot, we would just read the digest file and the signature file
each boot and ask the kernel to verify it.
2023-06-10 17:13:33 +02:00
Colin Walters
05faa1d890
Merge pull request #2877 from ericcurtin/ostree-aboot
Add ostree=aboot for signed Android Boot Images
2023-06-09 07:56:25 -04:00
Alexander Larsson
bb4a89e23c Update submodule: composefs
We will need the new fsverity computation helpers.
2023-06-08 19:14:50 +02:00
Eric Curtin
adf60e26ce Fix read_proc_cmdline_key ("ot-composefs") memory leak
Make it an autofree_char rather than a char
2023-06-08 15:27:16 +01:00
Eric Curtin
aa72caffb5 Add ostree=aboot for signed Android Boot Images
Some kernel images are delivered in a signed kernel + cmdline +
initramfs + dtb blob. When this is added to the commit server side, only
after this do you know what the cmdline is, this creates a recursion
issue. To avoid this, in the case where we have ostree=aboot karg
set, create a symlink after deploy to the correct ostree target in the
rootfs, as the cmdline can't be malleable and secured client-side at
the same time.
2023-06-08 15:27:07 +01:00
Colin Walters
59a653c9dd
Merge pull request #2875 from cgwalters/tempf-always-repo
fetcher: Always open tmpfiles in repo location
2023-06-07 13:34:56 -04:00
Alexander Larsson
0a3dd22f83
Merge pull request #2872 from cgwalters/composefs-followups
Composefs followups
2023-06-07 10:52:06 +02:00
Colin Walters
f7f6f87c51 fetcher: Always open tmpfiles in repo location
In an installation environment (like a live ISO) we may
not have significant space outside of the target installation
repository.

There's no reason not to always open a linkable tempfile.  In
the future we should fix the pull path to verify the checksum
and then just directly link in the object instead of copying.

Closes: https://github.com/ostreedev/ostree/issues/2571
2023-06-06 17:09:30 -04:00
Colin Walters
8762062648
Merge pull request #2874 from aospan/inode64-fix
commit: fix ostree deployment on 64-bit inode fs
2023-06-06 12:37:41 -04:00
Abylay Ospan
de6fddc6ad commit: fix ostree deployment on 64-bit inode fs
This commit addresses a bug that was causing ostree deployment
to become corrupted on the large fs, when any package was installed using
'rpm-ostree install'.

In such instances, multiple files were assigned the same inode. For
example, the '/home' directory and a regular file 'pkg-get' were
assigned the same inode (2147484070), making the deployment unusable.

A root cause analysis was performed, running the process under gdb,
which revealed a lossy conversion from guint64 to guint32, for example
6442451366 converted to 2147484070:

(gdb) p name
$10 = 0x7fe9224d2d70 "home"

(gdb) p inode
$73 = 6442451366

(gdb) s
    device=66311, modifier=0x7fe914791840) at
src/libostree/ostree-repo-commit.c:1590

The conversion resulted in entirely independent files potentially
receiving the same inode.

The issue was discovered on PoC machine equipped with a large NVME
(3.4TB), but the bug can be easily reproduced using `cosa run -m 4000
--qemu-size +3TB', followed by installation of any package using
`rpm-ostree install`. The resulting deployment will be unusable due to
many files being "corrupted" by the aforementioned issue.
2023-06-06 13:37:47 +00:00
Colin Walters
12b7f328e2 prepare-root: More logging in composefs, minor cleanup
- Hoist the `.ostree.cfs` to a shared constant
- Add more logging in general for extra visibility
2023-06-05 13:17:58 -04:00
Colin Walters
624b1084ab prepare-root: Add another missing O_CLOEXEC 2023-06-02 13:10:12 -04:00
Colin Walters
624512f9a7 composefs: Factor out a shared helper for setting error
To keep the error messages consistent.
2023-06-02 10:59:34 -04:00
Colin Walters
1f9607a8e9 mount-util: Add missing O_CLOEXEC
Seen in review.
2023-06-02 09:28:19 -04:00