fedostree

An instance of rpm-ostree for Fedora. This project takes multiple RPM package sets from Fedora, assembles them on the build server side, and stores these trees in an OSTree repository. Client systems can them atomically upgrade and switch between these trees.

Trying it out

See installation.

Background

Fedora today is an extremely flexible system. One can find Fedora builds running on everything from hobbyist ARM devices, to workstations, to testing servers.

This flexibility derives in large part from the fact that from a technological point of view, Fedora is a collection of packages. While pre-assembled "deliverables" such as the Live CDs are distributed by the project, they are only a temporary state.

For example, as soon as they are installed, upgrading involves having a package manager that dynamically reassemble the system from newer parts in the Fedora package collection. One cannot file a bug against the "default offering" as a whole - a package must be chosen.

Furthermore, nearly every aspect of the Fedora infrastructure (and documentation) is structured in terms of packages, from user-facing tools such as Bugzilla and Bodhi, to developer tools such as Koji. The announced security updates are based on package names.

In contrast for example, ChromeOS is delivered and updated as an pre-assembled atomic unit. ChromeOS is (compared to Fedora) completely inflexible, but fulfills a targeted role clearly well.

How OSTree allows a middle ground

Fundamentally, packages are partial filesystem trees with metadata - they are assembled by a package manager into a complete bootable tree. It's important to emphasize that it is only these complete trees that users run.

OSTree allows an OS distributor to ship multiple complete bootable filesystem trees, and furthermore, client machines can atomically switch between them, or even track multiple trees independently.

This allows a middle ground between the two extremes of a combinatorial explosion of packages, and a singular OS.

For example, these are some of the trees the current prototype generates:

Initial goals

The first goal of this project is to be an additional deployment option built in the Fedora infrastructure; possibly only for Fedora rawhide. Developers and testers can use OSTree to atomically replicate, upgrade to newer versions of, and transition between the pre-assembled trees produced by this build server.

Notably in this phase, no common mechanism for additional software installation is provided. That said, individual trees can do so; for example server/docker-io tree can use Docker to install and run server container applications independent of OSTree.

This phase does include basic integration testing on the build server side, which will be a major benefit to the Fedora project and its downstreams.

Required changes in Fedora/RPM for initial deployment

/usr/lib/passwd and nss-altfiles

A change to include /usr/lib/passwd is required; this is implemented currently by the nss-altfiles package. See also this patch for shadow-utils.

/var should be dynamically populated

All RPMs should stop shipping files and directories in /var. See this section of the OSTree documentation.

RPM should cope with its database living in /usr/share/rpm and being immutable.

SELinux support

OSTree was designed from the very beginning to work with SELinux; it just needs some bugfixes and integration on the rpm-ostree side.

Anaconda support

Anaconda should have an OSTree backend in addition to RPM. A basic UI that provides a listview of shipped trees and allows picking them would be quite sufficient initially.

Dracut

OSTree, when replicating content from a build server, effectively reverts the Dracut host-only mode. Furthermore, at the moment we hardcode /etc/machine-id, which is a definite bug that needs to be fixed. Possibly systemd should support reading the machine ID from the kernel commandline, as it's the only host-writable area available in early boot.

Development area: OSTree Layering

This phase would be allowing "layering" of trees. For example, if one installs the base/minimal tree, one could imagine taking the strace package, and computing a new filesystem tree which is the union of the two.

While plain standalone ELF executables would work with no modification, a generalization of this kind of dynamic layering implies a higher level above OSTree that is aware of things like ldconfig and gtk-update-icon-cache and how to trigger them when layers are combined

Conceptually, this is a step back towards combinatorics. For example, if libvirt is a layer that could be applied on top of the base server layer as well as the workstation layer, then there would need to be some notion of dependencies.

Development area: Local package assembly

There is absolutely no reason one could not just use the package manager on the client side to download and assemble packages - but rather than operating live on your current root, OSTree allows setting up the chosen tree for the next boot atomically.

The problem is making this sort of thing efficient and scalable; it would require careful integration of the local OSTree repository and the package manager caching to operate at a speed comparable to traditional package management.

Development area: Live updates

If one is using OSTree in a model without a separate application mechanism (as is the case for rpm-ostree), it is potentially painful to reboot just to upgrade applications.

It would be quite easy to do a bind mount of the new /usr over top of the old. This would avoid some of the problems dpkg/rpm have in creating an partial view. But even this model would still break many real world desktop applications such as Firefox. Few applications handle files from an arbitrary new version replacing their current ones.

Starting from an safe basis, it should be possible to engineer userspace so that many classes of upgrades can be applied both live and safely, without race conditions.

OSTree example: Bisecting Mesa

OSTree allows not just dual booting - one can just as easily have 50 or more trees locally. Suppose that you're tracking Fedora rawhide, and an upgrade breaks Mesa+GNOME (or sound, or something else). You can not only easily revert to a last known good tree, you can use OSTree to download intermediate builds from the build server and bisect across them. Given the ability to do local builds from git, automating bisection across source code is entirely possible as well.

OSTree example: Server side fast CI/CD

OSTree mandates that operating system updates work entirely offline. Work that has been prototyped already in the gnome-continuous codebase then takes updated content from an OSTree repository, and incrementally updates a cached VM disk image from the host side. This could allow booting updated systemd RPMs within minutes after Koji builds. This is fundamentally different from RPM %check as we are booting the tree as it will be shipped to the user - whereas RPM checks may run in e.g. a RHEL6 kernel.

OSTree example: Parallel installing Red Hat Enterprise Linux and Fedora

Many contributors to Fedora are also Red Hat engineers working on Red Hat Enterprise Linux. An example way to use OSTree is to have EL7 installed in the physical /, and install Fedora in /ostree/deploy/fedora. One can choose whether or not to share /home.

OSTree example: Trying rawhide safely

This is an obvious use case - you can run a stable release, and periodically try the development release on bare metal with a great deal of safety, particularly if you choose not to share /home. In this model, the only major risk is the newer kernel containing filesystem corrupting bugs.

OSTree example: "Recovery partition"

You can keep your OS installed in /, manage it with a traditional package manager, but have a small (e.g. base/minimal) tree installed via OSTree to use as a "recovery partition" when something goes wrong. Except unlike traditional recovery partitions, you can easily delete it if you want space, upgrade it offline, etc.

OSTree example: Reliable safe upgrades of a server cluster

OSTree allows taking a "cloud" like approach to a cluster of traditional servers. Every upgrade is atomic and (relatively) efficient, and can be served by any plain HTTP server.