Commit Graph

132 Commits

Author SHA1 Message Date
Jonathan Lebon
92b1a27202 Add concept of state overlays
In the OSTree model, executables go in `/usr`, state in `/var` and
configuration in `/etc`. Software that lives in `/opt` however messes
this up because it often mixes code *and* state, making it harder to
manage.

More generally, it's sometimes useful to have the OSTree commit contain
code under a certain path, but still allow that path to be writable by
software and the sysadmin at runtime (`/usr/local` is another instance).

Add the concept of state overlays. A state overlay is an overlayfs
mount whose upper directory, which contains unmanaged state, is carried
forward on top of a lower directory, containing OSTree-managed files.

In the example of `/usr/local`, OSTree commits can ship content there,
all while allowing users to e.g. add scripts in `/usr/local/bin` when
booted into that commit.

Some reconciliation logic is executed whenever the base is updated so
that newer files in the base are never shadowed by a copied up version
in the upper directory. This matches RPM semantics when upgrading
packages whose files may have been modified.

For ease of integration, this is exposed as a systemd template unit which
any downstream distro/user can enable. The instance name is the mountpath
in escaped systemd path notation (e.g.
`ostree-state-overlay@usr-local.service`).

See discussions in https://github.com/ostreedev/ostree/issues/3113 for
more details.
2024-01-09 23:20:41 -05:00
Colin Walters
28cc761806 sysroot: Stabilize deployment finalization, add API and CLI
It's about time we do this; deployment finalization locking
is a useful feature.  An absolutely key thing here is that
we've slowly been moving towards the deployments as the primary
"source of truth".

Specifically in bootc for example, we will GC container images
not referenced by a deployment.

This is then neecessary to support a "pull but don't apply automatically" model.

This stabilizes the existing `ostree admin deploy --lock-finalization`
CLI, and adds a new `ostree admin unlock-finalization`.

We still check the old lock file path, but there's a new boolean
value as part of the staged deployment data which is intended
to be the source of truth in the future.  At some point then we
can drop the rpm-ostree lockfile handling.

Closes: https://github.com/ostreedev/ostree/issues/3025
2023-11-27 10:59:56 -05:00
Alexander Larsson
accb1f0602 Add ostree admin post-copy command
This command will apply fs-verity on all objects that need it and
needs to be called when an ostree deployment has been copied on a
file-by-file basis, which would loose information such as fs-verity.

This is needed by osbuild which works by creating the final image in a
rootfs, and then separately copying that rootfs file-by-file to a
loopback mounted filesystem image.
2023-11-14 10:15:19 +01:00
Colin Walters
3cd3251aa1 Add admin set-default
A core underlying primitive in the C library is the ability
to arbitrarily reorder bootloader entries.

Let's expose the basic functionality here with the ability to pick
an arbitrarily deployment for the next boot.

Closes: https://github.com/ostreedev/ostree/issues/2965
2023-08-03 15:09:57 -04:00
Colin Walters
265cf7d786 build-sys: Add libsodium to OT_DEP_CRYPTO
There's no reason to have these distinct really.  If we're using
libsodium, we want it in the same places we're using openssl.

Prep for further refactoring.
2023-07-08 15:42:12 -04:00
Colin Walters
0c36e8143d Drop "ostree trivial-httpd" CLI, move to tests directory
See https://github.com/ostreedev/ostree/issues/1593

Basically this makes it easier for people packaging, as the trivial-httpd
is only for tests, and this way the binary will live with the tests.

Also at this point nothing should depend on `ostree trivial-httpd`.
2023-07-04 08:18:24 -04:00
Daniel Kolesa
d0ea2db430 fetcher: add libsoup3 backend
The default is still soup2, you can use --with-soup3 to enable
the soup3 backend instead.
2023-04-12 22:33:08 -06:00
Huijing Hei
3bc59a5206 RFE: Add a hidden option to ostree admin kargs edit-in-place to
update all existing deployments in place

Example:
$ sudo ostree admin kargs edit-in-place --append-if-missing=rw
See https://github.com/ostreedev/ostree/issues/2617

This will not add duplicate key, if there is `TESTARG=VAL1` in the
kernel arguments, `--append-if-missing=TESTARG=VAL2` will be ignored.
2022-06-23 22:31:39 +08:00
Colin Walters
d3d3e4ea13 Add an ostree-boot-complete.service to propagate staging failures
Quite a while ago we added staged deployments, which solved
a bunch of issues around the `/etc` merge.  However...a persistent
problem since then is that any failures in that process that
happened in the *previous* boot are not very visible.

We ship custom code in `rpm-ostree status` to query the previous
journal.  But that has a few problems - one is that on systems
that have been up a while, that failure message may even get
rotated out.  And second, some systems may not even have a persistent
journal at all.

A general thing we do in e.g. Fedora CoreOS testing is to check
for systemd unit failures.  We do that both in our automated tests,
and we even ship code that displays them on ssh logins.  And beyond
that obviously a lot of other projects do the same; it's easy via
`systemctl --failed`.

So to make failures more visible, change our `ostree-finalize-staged.service`
to have an internal wrapper around the process that "catches" any
errors, and copies the error message into a file in `/boot/ostree`.

Then, a new `ostree-boot-complete.service` looks for this file on
startup and re-emits the error message, and fails.

It also deletes the file.  The rationale is to avoid *continually*
warning.  For example we need to handle the case when an upgrade
process creates a new staged deployment.  Now, we could change the
ostree core code to delete the warning file when that happens instead,
but this is trying to be a conservative change.

This should make failures here much more visible as is.
2022-04-26 13:02:46 -04:00
Joseph Marrero
581a58067b Update FSF license notices to use URL instead of address 2021-12-07 08:34:25 -05:00
Dan Nicholson
58a683f8f0 bin/remote: Rename list-gpg-keys to gpg-list-keys
As pointed out in the original review, `gpg-list-keys` fits better
alongside the existing `gpg-import`.

Changes were done with:

```
git grep -l list-gpg-keys | xargs sed -i 's/list-gpg-keys/gpg-list-keys/'
for src in $(git ls-files '*list-gpg-keys*'); do
  dst=${src/list-gpg-keys/gpg-list-keys}
  git mv "$src" "$dst"
done
```
2021-08-23 15:30:31 -06:00
Dan Nicholson
74fb0c5f78 bin/remote: Add list-gpg-keys subcommand
This provides a wrapper for the `ostree_repo_remote_get_gpg_keys`
function to show the GPG keys associated with a remote. This is
particularly useful for validating that GPG key updates have been
applied. Tests are added, which checks the
`ostree_repo_remote_get_gpg_keys` API by extension.
2021-07-15 15:50:04 -06:00
Colin Walters
a90d59cf2d build-sys: Remove --enable-experimental-api
It was added for the collections bits, but we made that stable.
It's now just cruft and we're very unlikely to reuse the infrastructure
again.

Motivated by a unit test failure when running from a tarball:
https://github.com/ostreedev/ostree/issues/2313
2021-04-07 19:19:05 +00:00
Denis Pynkin
c3608aa56a sign: add new builtin for signing
This builtin allows to sign and verify commit with new signature
mechanism. At the moment it is possible to use 'dummy' and 'ed25519'
signing modules.

'dummy' module use any ASCII string from command line as a key for
commit's signing or verification.

Support of ed25519 signature is implemented with `libsoium` library.
Secret and public key should be provided in hex presentation via
command line.

Based on 'gpg-sign' source.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
edbbe1c4f2 lib/sign: initial implementation
Added the initial version of signing interface allowing to allowing to
sign and verify commits.
Implemented initial signing modules:
 - dummy -- simple module allowing to sign/verify with ASCII string
 - ed25519 -- module allowing to sign/verify commit with ed25519
   (EdDSA) signature scheme provided by libsodium library.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
84c8164610 Add libsodium dependency
Allow to configure with libsodium flag.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:22:41 +03:00
Stefan Agner
77b27c07df Avoid race condition when building outside of source tree
When building outside of source tree it can happen that src/ostree/
does not exist (yet) when bison is called. This leads to an build
error like so:
  bison: src/ostree/parse-datetime.c: cannot open: No such file or directory

Make sure that src/ostree/ exists when parse-datetime.c is built.
2019-11-26 09:22:30 +01:00
Alex Kiernan
8f0b225d60 Revert "Gate ostree-trivial-httpd on BUILDOPT_TRIVIAL_HTTPD"
This reverts commit 83d44ac20a.
2019-10-19 22:20:25 +01:00
Alex Kiernan
83d44ac20a Gate ostree-trivial-httpd on BUILDOPT_TRIVIAL_HTTPD
When building without --enable-trivial-httpd-cmdline, don't build or install
the ostree-trivial-httpd binary.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
2019-10-09 06:04:54 +01:00
Denis Pynkin
ca8752e01b build: Conditionally build GPGME-related sources
Do not build GPGME-related sources if flag USE_GPGME is not defined.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Closes: #1889
Approved by: cgwalters
2019-08-01 02:06:47 +00:00
Allen Bai
be2572bf68 lib/kargs: Make API public and upstream new rpm-ostree APIs
This change makes public the current kargs API in src/libostree/ostree-kernel-args.c
and adds documentations.

Upstreams the new kargs API from rpm-ostree/src/libpriv/rpmostree-kargs-process.c

Merges libostree_kernel_args_la_SOURCES to libostree_1_la_SOURCES in Makefile-libostree.am

Upstreams tests/check/test-kargs.c from rpm-ostree.

Closes: #1833

Closes: #1869
Approved by: jlebon
2019-06-18 14:29:14 +00:00
Matthew Leeds
8fbf19c9f5 Make P2P API public (no longer experimental)
Currently the API that allows P2P operations (e.g. pulling an ostree ref
from a LAN or USB source) is hidden behind the configure flag
--enable-experimental-api. This commit makes the API public and makes
that flag essentially a no-op (leaving it in place in case we want to
use it again in the future). The P2P API has been tested over the last
several months and proven to work.

This means that since we're no longer using the "experimental" feature
flag, P2P builds of Flatpak will fail when using versions of OSTree from
this commit onwards, until Flatpak is patched in the near future. If you
want to build Flatpak < 0.11.8 with P2P enabled and link against OSTree
2018.6, you'll have to patch Flatpak.  However, since Flatpak won't yet
have a hard dependency on OSTree 2018.6, it needs a new way to determine
if the P2P API in OSTree is available, so this commit adds a "p2p"
feature flag. This way the feature set is more semantically correct than
if we had continued to use the "experimental" feature flag.

In addition to making the P2P API public, this commit makes the P2P unit
tests run by default, removes the f27-experimental CI instance that's no
longer needed, changes a few man pages to reflect the changes, and
updates the bash completion script to accept the new commands and
options.

Closes: #1596
Approved by: cgwalters
2018-06-04 19:20:10 +00:00
Simon McVittie
f0708b3441 build: Don't distribute Bison-generated parser in dist tarballs
Signed-off-by: Simon McVittie <smcv@collabora.com>

Closes: #1563
Approved by: cgwalters
2018-04-30 18:31:24 +00:00
Colin Walters
eb506c759c Add concept of "staged" deployment
Add API to write a deployment state to `/run/ostree/staged-deployment`,
along with a systemd service which runs at shutdown time.

This is a big change to the ostree model for hosts,
but it closes a longstanding set of bugs; many, many people have
hit the "losing changes in /etc" problem.  It also avoids
the other problem of racing with programs that modify `/etc`
such as LVM backups:
https://bugzilla.redhat.com/show_bug.cgi?id=1365297

We need this in particular to go to a full-on model for
automatically updated host systems where (like a dual-partition model)
everything is fully prepared and the reboot can be taken
asynchronously.

Closes: https://github.com/ostreedev/ostree/issues/545

Closes: #1503
Approved by: jlebon
2018-04-12 14:55:12 +00:00
Colin Walters
7f88fddcd4 sysroot: Add concept of deployment "pinning" 📌
Example user story: Jane rebases her OS to a new major version N, and wants to
keep around N-1 even after a few upgrades for a while so she can easily roll
back. I plan to add `rpm-ostree rebase --pin` to opt-in to this for example.

Builds on the new `libostree-transient` group to store pinning state there.

Closes: https://github.com/ostreedev/ostree/issues/1460

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Marcus Folkesson
6bf4b3e1d8 Add SPDX-License-Identifier to source files
SPDX License List is a list of (common) open source
licenses that can be referred to by a “short identifier”.
It has several advantages compared to the common "license header texts"
usually found in source files.

Some of the advantages:
* It is precise; there is no ambiguity due to variations in license header
  text
* It is language neutral
* It is easy to machine process
* It is concise
* It is simple and can be used without much cost in interpreted
  environments like java Script, etc.
* An SPDX license identifier is immutable.
* It provides simple guidance for developers who want to make sure the
  license for their code is respected

See http://spdx.org for further reading.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>

Closes: #1439
Approved by: cgwalters
2018-01-30 20:03:42 +00:00
Philip Withnall
9546e6795e create-usb: Add a create-usb command to complement OstreeRepoFinderMount
This can be used to put OSTree repositories on USB sticks in a format
recognised by OstreeRepoFinderMount.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1182
Approved by: cgwalters
2017-09-27 14:44:00 +00:00
Philip Withnall
6453203f54 find-remotes: Add a find-remotes built-in command
This is a wrapper around the new ostree_repo_find_remotes() method; it
tries to find available remotes which can serve updates for the
user-provided refs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Georges Basile Stavracas Neto
fda4a47cae libostree: add versioning macros
OSTree currently provides no way to inspect the versioning
information at run time, being only available at compile
time through pkg-config.

This is a problem for e.g. Flatpak, that needs to check
whether the 'update-frequency' option is available. Checking
at compile time isn't great since it's not looking for new
symbols, but only if an optional feature is present.

This commit, then, adds a new header that is generated
at compile time, exposing OSTree's versioning information.

Closes: #728
Approved by: cgwalters
2017-03-11 15:38:56 +00:00
Colin Walters
361aa449fb libcurl backend
For rpm-ostree, we already link to libcurl indirectly via librepo, and
only having one HTTP library in process makes sense.

Further, libcurl is (I think) more popular in the embedded space.  It
also supports HTTP/2.0 today, which is a *very* nice to have for OSTree.

This seems to be working fairly well for me in my local testing, but it's
obviously brand new nontrivial code, so it's going to need some soak time.

The ugliest part of this is having to vendor in the soup-url code. With
Oxidation we could follow the path of Firefox and use the
[Servo URL parser](https://github.com/servo/rust-url).  Having to redo
cookie parsing also sucked, and that would also be a good oxidation target.

But that's for the future.

Closes: #641
Approved by: jlebon
2017-02-09 16:37:45 +00:00
Colin Walters
ced22f6c9b Split trivial-httpd into separate binary
Working on the libcurl backend, I hit the issue that the trivial-httpd program
depends on libsoup. I briefly considered having two versions, but libcurl is
client only, and moreover trivial-httpd is no longer trivial - it has various
features which are used by the test suite extensively.

Hence, what we'll do is build it as a separate binary which links to libsoup,
and use it during the tests. We *also* currently still provide `ostree
trivial-httpd` since some things use it like `rpm-ostree-toolbox` and the
Cockpit tests.

After those are ported to use some other webserver, I plan to add a build-time
option to drop it.

Closes: #636
Approved by: jlebon
2017-01-04 16:32:11 +00:00
Colin Walters
c051ee4a35 build-sys: Minor makefile tweaks
I'm introducing a new binary in a later patch, and it makes
sense to move more things to be common into the common section.

Also I noticed we were missing an inclusion of common `$(AM_LDFLAGS)`, though
AFAIK this doesn't break anything right now.

Closes: #636
Approved by: jlebon
2017-01-04 16:32:11 +00:00
Colin Walters
7f2830cb4e build: Make libsoup optional again
The "remote cookies" code broke this.  While I'm not sure anyone is
actually using ostree-without-http, it isn't too hard to keep the
build time conditional going.  Further, this work is preparatory for
libcurl porting.

Closes: #621
Approved by: jlebon
2016-12-08 16:04:16 +00:00
Sjoerd Simons
0613b4a479 remote: Add commands to add and remove cookies for a remote
Add commands to add and remove cookies to a remotes cookie jar.

Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Sjoerd Simons
6b467b9dbc remote: Add command to list cookies
Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Colin Walters
9e2763106b lib: Use sd_journal directly (optionally)
This was the last caller of libgsystem that isn't
`gs_file_get_path_cached()`.  I think the use case ostree has where
the same code can be called via command line and via a shared library
*and* via a daemon is rather unusual, so let's just copy the code for
logging from libgsystem into here.

For example rpm-ostree hard depends on a daemon mode, so it'll just
use `sd_journal` directly.

Closes: #341
Approved by: jlebon
2016-06-21 18:24:17 +00:00
Colin Walters
09238da065 admin: Add an unlock command, and libostree API
I'm trying to improve the developer experience on OSTree-managed
systems, and I had an epiphany the other day - there's no reason we
have to be absolutely against mutating the current rootfs live.  The
key should be making it easy to rollback/reset to a known good state.

I see this command as useful for two related but distinct workflows:

 - `ostree admin unlock` will assume you're doing "development".  The
   semantics hare are that we mount an overlayfs on `/usr`, but the
   overlay data is in `/var/tmp`, and is thus discarded on reboot.
 - `ostree admin unlock --hotfix` first clones your current deployment,
   then creates an overlayfs over `/usr` persistent
   to this deployment.  Persistent in that now the initramfs switchroot
   tool knows how to mount it as well.  In this model, if you want
   to discard the hotfix, at the moment you roll back/reboot into
   the clone.

Note originally, I tried using `rofiles-fuse` over `/usr` for this,
but then everything immediately explodes because the default (at least
CentOS 7) SELinux policy denies tons of things (including `sshd_t`
access to `fusefs_t`).  Sigh.

So the switch to `overlayfs` came after experimentation.  It still
seems to have some issues...specifically `unix_chkpwd` is broken,
possibly because it's setuid?  Basically I can't ssh in anymore.

But I *can* `rpm -Uvh strace.rpm` which is handy.

NOTE: I haven't tested the hotfix path fully yet, specifically
the initramfs bits.
2016-03-23 11:09:09 -04:00
Joaquim Rocha
6821ca1029 build: Link ostree with libarchive
libarchive (when available) is being used in ot-builtin-export.c so it
is necessary to link ostree with it.

https://bugzilla.gnome.org/show_bug.cgi?id=762457
2016-02-22 11:22:24 -05:00
Colin Walters
355f8438ef Add an export builtin, and API to write to libarchive
At the moment I'm looking at using rpm-ostree to manage RPM inputs
which can then be converted into Docker images.  It's most convenient
if we can stream directly out of libostree rather than doing a
checkout + tar combination.

There are also backup/debugging etc. reasons to implement `export` as
well.
2016-02-14 09:53:01 -05:00
Giuseppe Scrivano
b0163d6182 build: 'make clean' removes parse-datetime.c
and fix make dist while at it.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-02-07 14:56:21 +01:00
Matthew Barnes
460a4b2852 remote: Add "ostree remote summary" command
Downloads and prints a remote summary file and any signatures in an
easy-to-read format, or alternatively with the --raw option, prints
the summary GVariant data directly.

https://bugzilla.gnome.org/show_bug.cgi?id=759250
2015-12-17 15:49:51 -05:00
Colin Walters
76f4507557 build: Delete generated parse-datetime.c file, use AM_V_GEN
Bison is a well known external dependency, so just require it.
Including the generated content in git means it may or may not
be regenerated based randomly on timestamps, etc.

Also use `$(AM_V_GEN)` so we get prettier output.
2015-12-07 10:33:23 -05:00
Giuseppe Scrivano
335ea3f339 parse-datetime: use the module from gnulib
Use the parse-datetime module from gnulib, and adapt it to not require
other modules as portability is not really an issue for us.

DATE can be specified in different formats, such as: "-1 week", "last
monday", "1 week ago".

Include the generated .c file in the repository so to not add another
dependency to Bison.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-12-04 11:10:46 -05:00
Giuseppe Scrivano
f56688da71 prune: add --keep-younger-than=DATE
The format used for DATE is "%Y-%m-%d %H:%M:%S %z"

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-11-16 11:07:38 +01:00
Matthew Barnes
6284beb2b6 ostree: Add a "remote refs" command
Works like "ostree refs" but fetches refs from a remote repo.

This depends on the remote repo having a summary file, but any repo
being served over HTTP *ought* to have one.
2015-06-26 11:02:24 +02:00
Matthew Barnes
c287a7419c ostree: Add a "remote gpg-import" command
Imports GPG keys into a remote-specific keyring.
2015-05-13 13:08:49 -04:00
Matthew Barnes
e7246e6d64 ostree: Split up "remote" subcommands
To make room for "remote gpg-import", which will be non-trivial.
ot-builtin-remote.c was already a little too crowded anyway.

Also while we're at it, port this bit of code away from libgsystem.
2015-05-01 14:38:17 -04:00
Giuseppe Scrivano
c70abfd43c src: Move ot-tool-util from ostree/ to libotutil/
These utilities are not actually specific to the ostree commandline.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-03-06 18:45:38 +01:00
Giuseppe Scrivano
a705d9cf29 Add bsdiff submodule
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-03-03 12:16:17 -05:00
Matthew Barnes
a5b002dae6 ostree: Add gpg-sign command
Signs a commit with one or more GPG keys.
2015-02-26 12:53:01 -05:00