3139 Commits

Author SHA1 Message Date
Dan Nicholson
260bcd1193 core: Add ostree_commit_get_object_sizes API
This function parses the object listing in the `ostree.sizes` metadata
and returns an array of `OstreeCommitSizesEntry` structures.

Unfortunately, for reasons I don't understand, the linker wants to
resolve `_ostree_read_varuint64` from `ostree-core.c` even though it's
not used by `test-checksum.c` at all.
2020-01-20 20:46:29 -07:00
Dan Nicholson
fcbb453443 core: Add OstreeCommitSizesEntry type
This will be used when reading out entries in the `ostree.sizes`
metadata. Each entry corresponds to an object in the metadata array.
2020-01-20 20:44:12 -07:00
Dan Nicholson
1bbe674d91 libarchive: Support commit sizes metadata
Call the helper to set the generate_sizes boolean so that object size
data is stored while writing the mtree.
2020-01-20 20:42:27 -07:00
John Hiesey
291e9da258 lib/commit: Include object type in sizes metadata
Append a byte encoding the OSTree object type for each object in the
metadata. This allows the commit metadata to be fetched and then for the
program to see which objects it already has for an accurate calculation
of which objects need to be downloaded.

This slightly breaks the `ostree.sizes` `ay` metadata entries. However,
it's unlikely anyone was asserting the length of the entries since the
array currently ends in 2 variable length integers. As far as I know,
the only users of the sizes metadata are the ostree test suite and
Endless' eos-updater[1]. The former is updated here and the latter
already expects this format.

1. https://github.com/endlessm/eos-updater/
2020-01-20 20:42:27 -07:00
Dan Nicholson
a4592678aa tests/sizes: Check duplicate file doesn't add sizes entry
A duplicate file will resolve to the same object, so it shouldn't add
any entries to the sizes metadata.
2020-01-20 20:42:27 -07:00
Dan Nicholson
44fb5e72a1 lib/commit: Make size entries for existing objects
If the object was already in the repo then the sizes metadata entry was
skipped. Move the sizes entry creation after the data has been computed
but before the early return for an existing object.
2020-01-20 20:42:27 -07:00
Dan Nicholson
8ec7d6322f lib/commit: Fix object sizes metadata for multiple commits
The object sizes hash table was only being cleared when the repo was
finalized. That means that performing multiple commits while the repo
was open would reuse all the object sizes metadata for each commit.

Clear the hash table when the sizes metadata is setup and when it's
added to a commit. This still does not fix the issue all the way since
it does nothing to prevent the program from constructing multiple
commits simultaneously. To handle that, the object sizes hash table
should be attached to the MutableTree since that has the commit state.
However, the MutableTree is gone when the commit is actually created.
The hash table would have to be transferred to the root file when
writing the MutableTree. That would be an awkward addition to
OstreeRepoFile, though. Add a FIXME to capture that.
2020-01-20 20:42:27 -07:00
Dan Nicholson
37045b4b46 lib/commit: Only set generate_sizes for archive repos
Rather than checking throughout the code, only set the boolean when
appropriate.
2020-01-20 20:42:27 -07:00
Colin Walters
5af403be0c Support mounting /sysroot (and /boot) read-only
We want to support extending the read-only state to cover `/sysroot`
and `/boot`, since conceptually all of the data there should only
be written via libostree.  Or at least for `/boot` should *mostly*
just be written by ostree.

This change needs to be opt-in though to avoid breaking anyone.

Add a `sysroot/readonly` key to the repository config which instructs
`ostree-remount.service` to ensure `/sysroot` is read-only.  This
requires a bit of a dance because `/sysroot` is actually the same
filesystem as `/`; so we make `/etc` a writable bind mount in this case.

We also need to handle `/var` in the "OSTree default" case of a bind
mount; the systemd generator now looks at the writability state of
`/sysroot` and uses that to determine whether it should have the
`var.mount` unit happen before or after `ostree-remount.service.`

Also add an API to instruct the libostree shared library
that the caller has created a new mount namespace.  This way
we can freely remount read-write.

This approach extends upon in a much better way previous work
we did to support remounting `/boot` read-write.

Closes: https://github.com/ostreedev/ostree/issues/1265
2019-12-11 15:33:57 +00:00
Colin Walters
20daf96880 finalize-staged: Use the core option parsing to load sysroot
Prep for using the default mount namespace handling there that
will land as part of the read-only `/sysroot` and `/boot` work.
See https://github.com/ostreedev/ostree/issues/1265
2019-12-11 14:39:17 +00:00
Colin Walters
978bd19f97 Release 2019.6
Nothing really big here, but let's get a release out
so some bigger things like ro-sysroot, signing, sizes can
bake in master for a bit.
2019-12-09 16:18:01 +00:00
Colin Walters
36039759e2 pull: Add support for basic auth
This has been requested a few times for people delivering
OSTree content and wanting to do access control.
2019-12-02 22:48:33 +00:00
Philip Chimento
54639c03ca libostree: Add ostree_async_progress_copy_state()
This allows copying the state from one OstreeAsyncProgress object to
another, atomically, without invoking the callback. This is needed in
libflatpak, in order to chain OstreeAsyncProgress objects so that you
can still receive progress updates when iterating a different
GMainContext than the one that the OstreeAsyncProgress object was
created under.

See https://github.com/flatpak/flatpak/pull/3211 for the application of
this API.
2019-11-20 12:24:26 -08:00
Philip Chimento
382ad59822 Bump version in symbols file
There were no new symbols in 2019.5 and this version didn't get bumped
when 2019.5 was released.
2019-11-20 11:18:36 -08:00
Robert Fairley
aadc4db012 lib/kernel-args: Store kernel args as key/value entries
Define an `OstreeKernelArgsEntry` structure, which holds
both the key and the value. The kargs order array stores
entries for each key/value pair, instead of just the keys.
The hash table is used to locate entries, by storing
entries in a pointer array for each key. The same public
interface is preserved, while maintaining ordering
information of each key/value pair when
appending/replacing/deleting kargs.

Fixes: #1859
2019-11-07 23:39:10 -05:00
Colin Walters
58980a717a lib/keyfile: Treat "group not found" the same as "key not found"
Prep for fsverity, where I want to create a new group
`[fsverity]` in the keyfile that has default values.  We should
treat the absence of a group the same as absence of a key
in these "with defaults" APIs.
2019-11-07 21:15:41 +00:00
Colin Walters
980ca07b03 Release 2019.5 2019-10-30 17:08:00 +00:00
Jonathan Lebon
7ae8da08b9 lib/deploy: Also install HMAC file into /boot
To allow for FIPS mode, we need to also install the HMAC file from
`/usr/lib/modules` to `/boot` alongside the kernel image where the
`fips` dracut module will find it. For details, see:

https://github.com/coreos/fedora-coreos-tracker/issues/302

Note I didn't include the file in the boot checksum since it's itself a
checksum of the kernel, so we don't really gain much here other than
potentially causing an unnecessary bootcsum bump.
2019-10-29 16:45:29 -04:00
OpenShift Merge Robot
476f375cfc
Merge pull request #1953 from uajain/uajain/fix-memory-leak
async-progress: Plug memory leak while destroying GSource
2019-10-29 06:26:17 +01:00
Jonathan Lebon
74936f98d8 lib/pull: Tweak update_timeout logic again
I was hitting `SIGSEGV` when running `cosa build` and narrowed it down
to #1954. What's happening here is that because we're using the default
context, when we unref it in the out path, it may not actually destroy
the `GSource` if it (the context) is still ref'ed elsewhere. So then,
we'd still get events from it if subsequent operations iterated the
context.

This patch is mostly a revert of #1954, except that we still keep a ref
on the `GSource`. That way it is always safe to destroy it afterwards.
(And I've also added a comment to explain this better.)
2019-10-28 15:26:45 -04:00
Umang Jain
78c8c25d64 async-progress: Plug memory leak while destroying GSource
See https://gitlab.gnome.org/GNOME/glib/commit/71973c722
2019-10-25 20:59:15 +05:30
Colin Walters
650d6252af Revert "grub2: Exit gracefully if the configuration has BLS enabled"
This reverts commit 985a14100295c99d0c6d712bfbee0ec52a3a1601.
It turned out that some people have old bootloaders, and hence
get the "no entries" problem.  That's much, much much worse
than double entries.
2019-10-24 17:49:53 +00:00
Jonathan Lebon
d4a186e80e lib/pull: Avoid calling destroy on unref'ed GSource
We're creating the timer source and then passing ownership to the
context, but because we didn't free the pointer, we would still call
`g_source_destroy` in the exit path. We'd do this right after doing
`unref` on the context too, which would have already destroyed and
unref'ed the source.

Drop that and just restrict the scope of that variable down to make
things more obvious.

Just noticed this after reviewing #1953.
2019-10-23 11:15:02 -04:00
OpenShift Merge Robot
1645416aac
Merge pull request #1947 from cgwalters/more-scan-build-2
More scan build 2
2019-10-18 20:59:06 +02:00
Dan Nicholson
4df90d4012 repo: Stop using deprecated G_GNUC_FUNCTION
In glib 2.62 this has been changed to emitting a warning. Use G_STRFUNC
instead, which has been available for a long time and is already used in
other places in ostree.
2019-10-18 11:10:44 -06:00
Colin Walters
fbed380483 lib/checksum-utils: Use g_memdup()
This is clearer and silences a scan-build warning.
2019-10-18 15:07:39 +00:00
Colin Walters
57bb06419a lib: Port variant-builder.c to new style
Seeing `scan-build` warning here, prep for fixing it.
2019-10-18 15:07:39 +00:00
Colin Walters
25c5ae5d08 lib/pull: [scan-build] Silence a dead store warning
This one was actual duplicate code.
2019-10-18 15:07:39 +00:00
Colin Walters
bc1980ca38 lib/repo: [scan-build] Quiet a dead store warning
False positive, just add a pacifier.
2019-10-18 15:07:39 +00:00
OpenShift Merge Robot
3b58d89368
Merge pull request #1943 from cgwalters/more-scan-build
More scan build
2019-10-16 20:44:15 +02:00
OpenShift Merge Robot
38729a4f90
Merge pull request #1937 from cgwalters/zipl-backend
bootloader: Add a zipl bootloader backend
2019-10-16 17:21:10 +02:00
Colin Walters
c61234a428 bootloader: Add a zipl bootloader backend
zipl is a bit special in that it parses the BLS config files
directly *but* we need to run the command to update the "boot block".

Hence, we're not generating a separate config file like the other
backends.  Instead, extend the bootloader interface with a `post_bls_sync`
method that is run in the same place we swap the `boot/loader` symlink.

We write a "stamp file" in `/boot` that says we need to run this command.
The reason we use stamp file is to prevent the case where the system is
interrupted after BLS file is updated, but before zipl is triggered,
then zipl boot records are not updated.
This opens the door to making things eventually-consistent/reconcilable
by later adding a systemd unit to run `zipl` if we're interrupted via
a systemd unit - I think we should eventually take this approach
everywhere rather than requiring `/boot/loader` to be a symlink.

Author: Colin Walters <walters@verbum.org>
Tested-by: Tuan Hoang <tmhoang@linux.ibm.com>
Co-Authored-By: Tuan Hoang <tmhoang@linux.ibm.com>
2019-10-16 14:22:02 +00:00
Colin Walters
946659aacf prune: [scan-build] Initialize a variable
Another false positive because we only read this if `opt_keep_younger_than` is `TRUE`,
but let's initialize variables on general principle.
2019-10-16 13:48:20 +00:00
Colin Walters
a8dc90b02f tree-wide: [scan-build]: Add some asserts that pointers are non-NULL
More "scan-build doesn't understand GError and our out-param conventions"
AKA "these errors would be impossible with Rust's sum type Result<> approach".
2019-10-16 13:44:46 +00:00
Colin Walters
4a38b11159 commit: [scan-build] Remove a dead assignment
The `write_commit()` API defaults to current time, and
this assignment became dead in:
8ba90a3341
2019-10-16 13:38:29 +00:00
Colin Walters
810f24d897 libotutil: Port keyfile-utils.c to new style
I was trying to fix a clang `scan-build` error that jlebon
ended up tracking down in
9344de1ce1

But in the process of tracing through this I found it
way easier to read as "new style" code, so this also ports the
code.

I added a `g_assert()` in there too to help assert that
`g_key_file_get_value` won't leak in the error path.
2019-10-16 13:30:21 +00:00
OpenShift Merge Robot
347e234aaa
Merge pull request #1932 from cgwalters/covscan-fixes-2
libostree: Add an assert to pacify clang-analyzer
2019-10-15 19:56:54 +02:00
Jonathan Lebon
9344de1ce1 src/libotutil: Fix strv memory leak
We were only freeing the array and not the members.

Caught by `clang-analyzer` in:
https://github.com/ostreedev/ostree/pull/1931
2019-10-15 11:59:45 -04:00
Colin Walters
806206fac2 repo: [scan-build]: Mark a variable used
We're just using this to auto-free, quiet the static analysis.
2019-10-15 12:41:27 +00:00
Colin Walters
51d9aa35a9 sysroot: [scan-build] Remove a dead assignment
Just quieting the scan.
2019-10-15 12:41:27 +00:00
Colin Walters
f1fdd885ab sysroot: [scan-build]: Remove a dead assignment
Clarify the conditionals here and remove a dead assignment.
2019-10-15 12:41:27 +00:00
Colin Walters
9032182e3c repo: [scan-build] Initialize a variable
Another GLib error convention issue; but eh, we might as
well be conservative and always initialize variables.
2019-10-15 12:41:27 +00:00
Colin Walters
aa7795d08d libostree: Add an assert to pacify clang-analyzer
Got this error when trying to rebase libostree in RHEL:

```
Error: CLANG_WARNING: [#def1]
libostree-2019.2/src/libostree/ostree-repo-checkout.c:375:21: warning: Access to field 'disable_xattrs' results in a dereference of a null pointer (loaded from variable 'repo')
```

I think what's happening is it sees us effectively testing
`if (repo == NULL)` via the `while (current_repo)`.  Let's
tell it we're sure it's non-null right after the loop.
2019-10-15 12:40:54 +00:00
Javier Martinez Canillas
985a141002 grub2: Exit gracefully if the configuration has BLS enabled
Since Fedora 30 grub2 has support to populate its menu entries from the
BootLoaderSpec fragments in /boot/loader/entries, so there's no need to
generate menu entries anymore using the /etc/grub.d/15_ostree script.

But since ostree doesn't update the bootloader, it may be that the grub2
installed is an old one that doesn't have BLS support.

For new installs, GRUB_ENABLE_BLSCFG=true is set in /etc/default/grub to
tell the /etc/grub.d/10_linux script if a blscfg command has to be added
to the generated grub2 config file.

So check if BLS is enabled in /etc/default/grub and only add the entries
if that's not the case. Otherwise the menu entries will be duplicated.

The approach has the drawback that if a user sets GRUB_ENABLE_BLSCFG=true
in /etc/default/grub without updating grub2, they will get an empty menu.
Since there won't be any entries created by the 30_ostree script and the
blscfg command won't work on the older grub2.

Unfortunately there is no way to know if the installed grub2 already has
BLS support or not.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1751272#c27

Closes: #1929
Approved by: jlebon
2019-09-26 17:59:37 +00:00
Jonathan Lebon
c6c640f3ae Post-release version bump
Closes: #1927
Approved by: cgwalters
2019-09-25 13:43:28 +00:00
Jonathan Lebon
9d39e7d91e Release 2019.4
Tiny release. Just want to get out the important bugfixes instead of
backporting patches (notably the gpg-agent stuff and
`ostree-finalize-staged.service` ordering).

Closes: #1927
Approved by: cgwalters
2019-09-25 13:43:28 +00:00
Jonathan Lebon
88182635ab boot/finalize-staged: Run after systemd-journal-flush.service
In Fedora 31, `systemd-journal-flush.service` uses a new
`--smart-relinquish-var` switch which fixes the
`umount: /var: target is busy` bug by telling journald to stop logging
to `/var` and back to `/run` again during shutdown.

This interacted with `ostree-finalize-staged.service` in a tricky way:
since we weren't strongly ordered against it, when we happened to
finalize after `/var` is relinquished, we never persisted the output
from that service to disk. This then threw off `rpm-ostree status` when
trying to find the completion message to know that finalization went
well.

Just fix this by adding an explicit `After=` on that unit. That way we
shut down *before* `systemd-journal-flush.service` (the `/var`
relinquish bit happens in its `ExecStop=`).

For more info, see:
3ff7a50d66
1e187d2dd5
https://bugzilla.redhat.com/show_bug.cgi?id=1751272

Closes: #1926
Approved by: cgwalters
2019-09-24 21:39:10 +00:00
Philip Withnall
94fcba96e0 lib/repo-pull: Add more debugging on pull failure
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1925
Approved by: cgwalters
2019-09-24 13:36:59 +00:00
Benjamin Gilbert
653fc6a125 prepare-root: remember to remove /sysroot.tmp
Without this, rerunning ostree-prepare-root will fail in mkdir()
because /sysroot.tmp already exists, which complicates debugging from
the dracut emergency shell.

Closes: #1919
Approved by: cgwalters
2019-09-13 12:40:57 +00:00
Colin Walters
ca701f69c2 fsck: Fix version in docs, tweak error text
I think this error message is clearer.

Closes: #1918
Approved by: jlebon
2019-09-09 15:18:50 +00:00