Operating system and container binary deployment and upgrades
Go to file
Colin Walters f81b9fa166 sysroot: Rework /var handling to act like Docker VOLUME /var
We've long struggled with semantics for `/var`.  Our stance of
"/var should start out empty and be managed by the OS" is a strict
one, that pushes things closer to the original systemd upstream
ideal of the "OS state is in /usr".

However...well, a few things.  First, we had some legacy bits
here which were always populating the deployment `/var`.  I don't
think we need that if systemd is in use, so detect if the tree
has `usr/lib/tmpfiles.d`, and don't create that stuff at
`ostree admin stateroot-init` time if so.

Building on that then, we have the stateroot `var` starting out
actually empty.

When we do a deployment, if the stateroot `var` is empty,
make a copy (reflink if possible of course) of the commit's `/var`
into it.

This matches the semantics that Docker created with volumes,
and this is sufficiently simple and easy to explain that I think
it's closer to the right thing to do.

Crucially...it's just really handy to have some pre-existing
directories in `/var` in container images, because Docker (and podman/kube/etc)
don't run systemd and hence don't run `tmpfiles.d` on startup.

I really hit on the fact that we need `/var/tmp` in our container
images by default for example.

So there's still some overlap here with e.g. `/usr/lib/tmpfiles.d/var.conf`
as shipped by systemd, but that's fine - they don't actually conflict
per se.
2024-02-09 17:46:12 -05:00
.copr copr: only use libostree tags 2022-11-11 14:05:56 +00:00
.github ci: Use BOOTC_SKIP_SELINUX_HOST_CHECK, test labeling of /etc 2024-02-08 15:54:23 +01:00
apidoc sysroot: Stabilize deployment finalization, add API and CLI 2023-11-27 10:59:56 -05:00
bash bin/remote-summary: Add options to show metadata 2023-02-07 22:59:30 -07:00
bsdiff@b817e9491c bsdiff: bump submodule, pick up fix for CVE-2014-9862 2022-01-03 16:13:01 +00:00
build-aux Add infrastructure for "make syntax-check" 2015-01-30 15:27:36 +01:00
buildutil buildutil/glibtests.m4: fix bashism 2022-08-23 23:38:20 +01:00
ci ci: Add a bootc/c9s workflow 2024-01-31 14:03:25 -05:00
coccinelle tree-wide: Add+run spatch to use glnx_throw() 2017-05-26 19:27:11 +00:00
composefs@2d5cdcb917 Update submodule: composefs 2024-01-31 12:09:20 +01:00
docs sysroot: Rework /var handling to act like Docker VOLUME /var 2024-02-09 17:46:12 -05:00
libglnx@b415d04688 build(deps): bump libglnx from aff1eea to b415d046 2024-01-10 15:41:14 -05:00
man admin/pin: Add commands to pin booted, pending and rollback deployments 2024-01-29 11:44:20 +00:00
manual-tests Update FSF license notices to use URL instead of address 2021-12-07 08:34:25 -05:00
rust-bindings sysroot: Add a method to borrow sysroot fd 2023-12-01 14:35:41 -05:00
src sysroot: Rework /var handling to act like Docker VOLUME /var 2024-02-09 17:46:12 -05:00
tests sysroot: Rework /var handling to act like Docker VOLUME /var 2024-02-09 17:46:12 -05:00
.cci.jenkinsfile ci: cancel previous build on PR update 2023-11-21 15:33:20 -05:00
.clang-format clang-format: Don't align backslashes 2023-05-02 08:42:19 -04:00
.dir-locals.el .dir-locals.el: Standard Emacs indentation config 2017-01-12 16:09:34 +00:00
.editorconfig Add a .vimrc and .editorconfig 2017-09-21 22:03:11 +00:00
.gitmodules gitmodules: Use github GNOME mirror 2023-09-20 07:58:44 -04:00
.lgtm.yml .lgtm.yml: Fix gpgme dependency 2022-02-17 10:16:13 -07:00
.vimrc Add a .vimrc and .editorconfig 2017-09-21 22:03:11 +00:00
autogen.sh Update submodule: composefs 2023-05-31 10:57:37 +02:00
Cargo.toml rust: Add missing feature versions 2023-12-19 21:02:59 -05:00
cfg.mk ci: Add a codestyle.sh 2022-05-31 11:42:42 -04:00
configure.ac sysroot: Rework /var handling to act like Docker VOLUME /var 2024-02-09 17:46:12 -05:00
CONTRIBUTING.md Rewrite manual in mkdocs 2016-01-28 09:31:37 -05:00
COPYING COPYING: Update to latest FSF with current address 2014-01-16 10:22:30 -05:00
deny.toml deny.toml: Add Unicode-DFS-2016 2022-08-01 14:43:31 -04:00
git.mk Use git.mk 2016-04-07 12:49:40 +00:00
GNUmakefile build: Drop make syntax-check 2023-07-11 14:17:05 -04:00
Makefile-bash.am Update FSF license notices to use URL instead of address 2021-12-07 08:34:25 -05:00
Makefile-boot.am Add concept of state overlays 2024-01-09 23:20:41 -05:00
Makefile-decls.am Update FSF license notices to use URL instead of address 2021-12-07 08:34:25 -05:00
Makefile-libostree-defines.am lib: use ostree-content-writer header 2022-01-03 14:20:58 +00:00
Makefile-libostree.am configure: post-release version bump 2023-12-05 14:03:56 -05:00
Makefile-man.am Add concept of state overlays 2024-01-09 23:20:41 -05:00
Makefile-ostree.am Add concept of state overlays 2024-01-09 23:20:41 -05:00
Makefile-otcore.am Move prepare-root karg helpers into otcore, add unit tests 2023-08-23 17:11:10 -04:00
Makefile-otutil.am libotutil: Link to crypto libs 2023-07-07 20:19:59 +02:00
Makefile-switchroot.am Support transient /etc 2023-10-12 17:03:22 +02:00
Makefile-tests.am sysroot: Rework /var handling to act like Docker VOLUME /var 2024-02-09 17:46:12 -05:00
Makefile.am Factor out a libotcore 2023-07-11 14:08:32 -04:00
mkdocs.yml docs: Add Contributing Tutorial to Mkdocs pages 2018-08-21 14:05:14 +00:00
ostree.doap doap category infrastructure 2014-07-31 11:26:32 +02:00
README.md Link to gardenlinux/ostree-image-builder in README 2023-12-18 14:12:45 +01:00
TODO Fix repeated words. 2015-01-30 15:27:36 +01:00

libostree

This project is now known as "libostree", though it is still appropriate to use the previous name: "OSTree" (or "ostree"). The focus is on projects which use libostree's shared library, rather than users directly invoking the command line tools (except for build systems). However, in most of the rest of the documentation, we will use the term "OSTree", since it's slightly shorter, and changing all documentation at once is impractical. We expect to transition to the new name over time.

As implied above, libostree is both a shared library and suite of command line tools that combines a "git-like" model for committing and downloading bootable filesystem trees, along with a layer for deploying them and managing the bootloader configuration.

The core OSTree model is like git in that it checksums individual files and has a content-addressed-object store. It's unlike git in that it "checks out" the files via hardlinks, and they thus need to be immutable to prevent corruption. Therefore, another way to think of OSTree is that it's just a more polished version of Linux VServer hardlinks.

Features:

  • Transactional upgrades and rollback for the system
  • Replicating content incrementally over HTTP via GPG signatures and "pinned TLS" support
  • Support for parallel installing more than just 2 bootable roots
  • Binary history on the server side (and client)
  • Introspectable shared library API for build and deployment systems
  • Flexible support for multiple branches and repositories, supporting projects like Flatpak which use libostree for applications, rather than hosts.

Documentation

For more information, see the project documentation or the project documentation website.

Operating systems and distributions using OSTree

Apertis uses libostree for their host system as well as Flatpak. See update documentation and apertis-update-manager

Endless OS uses libostree for their host system as well as Flatpak. See their eos-updater and deb-ostree-builder projects.

For Debian/apt, see also https://github.com/stb-tester/apt2ostree and the LWN article Merkle trees and build systems.

Fedora derivatives use rpm-ostree (noted below); there are 4 variants using OSTree:

Red Hat Enterprise Linux CoreOS is a derivative of Fedora CoreOS, used in OpenShift 4. The machine-config-operator manages upgrades. RHEL CoreOS is also the successor to RHEL Atomic Host, which uses rpm-ostree as well.

GNOME Continuous is where OSTree was born - as a high performance continuous delivery/testing system for GNOME.

GNOME OS is a testing OS that uses libostree for their host system as well as Flatpak.

Liri OS has the option to install their distribution using ostree.

Torizon OS is a Linux distribution for embedded systems that updates via OSTree images delivered via Uptane and aktualizr.

Distribution build tools

meta-updater is a layer available for OpenEmbedded systems.

QtOTA is Qt's over-the-air update framework which uses libostree.

The BuildStream build and integration tool supports importing and exporting from libostree repos.

fedora-iot/otto is a tool that helps ship ostree commits inside Docker/OCI containers and run a webserver to serve the commits.

Fedora coreos-assembler is the build tool used to generate Fedora CoreOS derivatives.

debos is a tool-chain for simplifying the process of building a Debian-based OS image.

gardenlinux/ostree-image-builder is a sample for building Debian-based OS images. It is not production ready but it might be useful to get started.

Projects linking to libostree

rpm-ostree is used by the Fedora-derived operating systems listed above. It is a full hybrid image/package system. By default it uses libostree to atomically replicate a base OS (all dependency resolution is done on the server), but it supports "package layering", where additional RPMs can be layered on top of the base. This brings a "best of both worlds"" model for image and package systems.

eos-updater is a daemon that implements updates on EndlessOS.

Flatpak uses libostree for desktop application containers. Unlike most of the other systems here, Flatpak does not use the "libostree host system" aspects (e.g. bootloader management), just the "git-like hardlink dedup". For example, Flatpak supports a per-user OSTree repository.

aktualizr is an Uptane-conformant software update client library intended for use in automotive and other security-sensitive embedded devices. It uses OSTree to manage the OS of the host device by default.

Language bindings

libostree is accessible via GObject Introspection; any language which has implemented the GI binding model should work. For example, Both pygobject and gjs are known to work and further are actually used in libostree's test suite today.

Some bindings take the approach of using GI as a lower level and write higher level manual bindings on top; this is more common for statically compiled languages. Here's a list of such bindings:

Building

Releases are available as GPG signed git tags, and most recent versions support extended validation using git-evtag.

However, in order to build from a git clone, you must update the submodules. If you're packaging OSTree and want a tarball, I recommend using a "recursive git archive" script. There are several available online; this code in OSTree is an example.

Once you have a git clone or recursive archive, building is the same as almost every autotools project:

git submodule update --init
env NOCONFIGURE=1 ./autogen.sh
./configure --prefix=...
make
make install DESTDIR=/path/to/dest

Contact and discussion forums

There is also an #ostree channel on Libera.Chat as well as enabled Github discussions.

Contributing

See Contributing.

Licensing

The licensing for the code of libostree can be canonically found in the individual files; and the overall status in the COPYING file in the source. Currently, that's LGPLv2+. This also covers the man pages and API docs.

The license for the manual documentation in the doc/ directory is: SPDX-License-Identifier: (CC-BY-SA-3.0 OR GFDL-1.3-or-later) This is intended to allow use by Wikipedia and other projects.

In general, files should have a SPDX-License-Identifier and that is canonical.