ostree/README.md

OSTree
========

Problem statement
-----------------

Hacking on the core operating system is painful - this includes most
of GNOME from upower and NetworkManager up to gnome-shell.  I want a
system that matches these requirements:

1. Does not disturb your existing OS
2. Is not terribly slow to use
3. Shares your $HOME - you have your data
4. Allows easy rollback
5. Ideally allows access to existing apps

Comparison with existing tools
------------------------------

 - Virtualization

    Fails on points 2) and 3).  Actually qemu-kvm can be pretty fast,
    but in a lot of cases there is no substitute for actually booting
    on bare metal; GNOME 3 really needs some hardware GPU
    acceleration.

 - Rebuilding distribution packages

    Fails on points 1) and 4).  Is also just very annoying: dpkg/rpm
    both want tarballs, which you don't have since you're working from
    git.  The suggested "mock/pbuilder" type chroot builds are *slow*.
    And even with non-chroot builds there is lots of pointless build
    wrapping going on.  Both dpkg and rpm also are poor at helping you
    revert back to the original system.

    All of this can be scripted to be less bad of course - and I have
    worked on such scripts.  But fundamentally you're still fighting
    the system, and if you're hacking on a lowlevel library like say
    glib, you can easily get yourself to the point where you need a
    recovery CD - at that point your edit/compile/debug cycle is just
    too long.

 - "sudo make install"

    Now your system is in an undefined state.  You can use e.g. rpm
    -qV to try to find out what you overwrote, but neither dpkg nor
    rpm will help clean up any files left over that aren't shipped by
    the old package.  

    This is most realistic option for people hacking on system
    components currently, but ostree will be better.

 - LXC / containers

   Fails on 3), and 4) is questionable.  Also shares the annoying part
   of rebuilding distribution packages. LXC is focused on running
   multiple server systems at the *same time*, which isn't what we
   want (at least, not right now), and honestly even trying to support
   that for a graphical desktop would be a lot of tricky work, for
   example getting two GDM instances not to fight over VT
   allocations. But some bits of the technology may make sense to use.

 - jhbuild + distribution packages

    The state of the art in GNOME - but can only build non-root things -
    this means you can't build NetworkManager, and thus are permanently
    stuck on whatever the distro provides.

Who is ostree for?
------------------------------

First - operating system developers and testers.  I specifically keep
a few people in mind - Dan Williams and Eric Anholt, as well as myself
obviously.  For Eric Anholt, a key use case for him is being able to
try out the latest gnome-shell, and combine it with his work on Mesa,
and see how it works/performs - while retaining the ability to roll
back if one or both breaks.

The rollback concept is absolutely key for shipping anything to
enthusiasts or knowledable testers.  With a system like this, a tester
can easily perform a local rollback - something just not well
supported by dpkg/rpm.  (What about Conary?  See below.)

Also, distributing operating system trees (instead of packages) gives
us a sane place to perform automated QA **before** we ship it to
testers.  We should never be wasting these people's time.

Even better, this system would allow testers to *bisect* across
operating system builds, and do so very efficiently.

The core idea - chroots
-----------------------

chroots are the original lightweight "virtualization".  Let's use
them.  So basically, you install a mainstream distribution (say
Debian).  It has a root filesystem like this:

		/usr
		/etc
		/home
		...

Now, what we can do is have a system that installs chroots as a subdirectory
of the root, like:

		/ostree/gnomeos-3.0-opt-393a4555/{usr,etc,sbin,...}
		/ostree/gnomeos-3.2-opt-7e9788a2/{usr,etc,sbin,...}

These live in the same root filesystem as your regular distribution
(Note though, the root partition should be reasonably sized, or
hopefully you've used just one big partition).

You should be able to boot into one of these roots.  Since ostree
lives inside a distro created partition, a tricky part here is that we
need to know how to interact with the installed distribution's grub.
This is an annoying but tractable problem.

OSTree will allow efficiently parallel installing and downloading OS
builds.

An important note here is that we explicitly link /home in each root
to the real /home.  This means you have your data.  This also implies
we share uid/gid, so /etc/passwd will have to be in sync.  Probably
what we'll do is have a script to pull the data from the "host" OS.

Other shared directories are /var and /root.  Note that /etc is
explicitly NOT shared!

On a pure OSTree system, the filesystem layout will look like this:

		.
		|-- boot
		|-- home
		|-- ostree
		|   |-- current -> gnomeos-3.2-opt-7e9788a2
		|   |-- gnomeos-3.0-opt-393a4555
		|   |   |-- etc
		|   |   |-- lib
		|   |   |-- mnt
		|   |   |-- proc
		|   |   |-- run
		|   |   |-- sbin
		|   |   |-- srv
		|   |   |-- sys
		|   |   `-- usr
		|   `-- gnomeos-3.2-opt-7e9788a2
		|       |-- etc
		|       |-- lib
		|       |-- mnt
		|       |-- proc
		|       |-- run
		|       |-- sbin
		|       |-- srv
		|       |-- sys
		|       `-- usr
		|-- root
		`-- var
		

Making this efficient
---------------------

One of the first things you'll probably ask is "but won't that use a
lot of disk space"?  Indeed, it will, if you just unpack a set of RPMs
or .debs into each root.

Besides chroots, there's another old Unix idea we can take advantage
of - hard links.  These allow sharing the underlying data of a file,
with the tradeoff that changing any one file will change all names
that point to it.  This mutability means that we have to either:

 1. Make sure everything touching the operating system breaks hard links
    This is probably tractable over a long period of time, but if anything
    has a bug, then it corrupts the file effectively.
 2. Make the core OS read-only, with a well-defined mechanism for mutating
    under the control of ostree.

I chose 2.

A userspace content-addressed versioning filesystem
---------------------------------------------------

At its very core, that's what ostree is.  Just like git.  If you
understand git, you know it's not like other revision control systems.
git is effectively a specialized, userspace filesystem, and that is a
very powerful idea.

At the core of git is the idea of "content-addressed objects".  For
background on this, see <http://book.git-scm.com/7_how_git_stores_objects.html>

Why not just use git?  Basically because git is designed mainly for
source trees - it goes to effort to be sure it's compressing text for
example, under the assumption that you have a lot of text.  Its
handling of binaries is very generic and unoptimized.

In contrast, ostree is explicitly designed for binaries, and in
particular one type of binary - ELF executables (or it will be once we
start using bsdiff).  

Another big difference versus git is that ostree uses hard links
between "checkouts" and the repository.  This means each checkout uses
almost no additional space, and is *extremely* fast to check out.  We
can do this because again each checkout is designed to be read-only.

So we mentioned above there are:

		/gnomeos/root-3.0-opt
		/gnomeos/root-3.2-opt

There is also a "repository" that looks like this:

		.ht/objects/17/a95e8ca0ba655b09cb68d7288342588e867ee0.file
		.ht/objects/17/68625e7ff5a8db77904c77489dc6f07d4afdba.meta
		.ht/objects/17/cc01589dd8540d85c0f93f52b708500dbaa5a9.file
		.ht/objects/30
		.ht/objects/30/6359b3ca7684358a3988afd005013f13c0c533.meta
		.ht/objects/30/8f3c03010cedd930b1db756ce659c064f0cd7f.meta
		.ht/objects/30/8cf0fd8e63dfff6a5f00ba5a48f3b92fb52de7.file
		.ht/objects/30/6cad7f027d69a46bb376044434bbf28d63e88d.file

Each object is either metadata (like a commit or tree), or a hard link
to a regular file.

Note that also unlike git, the checksum here includes *metadata* such
as uid, gid, permissions, and extended attributes.  (It does not include
file access times, since those shouldn't matter for the OS)

This is another important component to allowing the hardlinks.  We
wouldn't want say all empty files to be shared necessarily.  (Though
maybe this is wrong, and since the OS is readonly, we can make all
files owned by root without loss of generality).

However this tradeoff means that we probably need a second index by
content, so we don't have to redownload the entire OS if permissions
change =)

Atomic upgrades, rollback
-------------------------

OSTree is designed to atomically swap operating systems - such that
during an upgrade and reboot process, you either get the full new
system, or the old one.  There is no "Please don't turn off your
computer".  We do this by simply using a symbolic link like:

/gnomeos -> /gnomeos-e3b0c4429

Where /gnomeos-e3b0c4429/ has the full regular filesystem tree with
usr/ etc/ directories as above.  To upgrade or rollback (there is no
difference internally), we simply check out a new tree into
/gnomeos-b90ae4763 for example, then swap the symbolic link, then
remove the old tree.

But does this mean you have to "reboot" for OS upgrades?  Very likely,
yes - and this is no different from RPM/deb or whatever.  They just
typically lie to you about it =) But read on.

Let's consider a security update to a shared library.  We can download
the update to the repository, build a new tree and atomically swap it
as above, but what if a process has the old shared library in use?

Here's where we will probably need to inspect which processes are
using the library - if any are, then we need to trigger either a
logout/login if it's just the desktop shell and/or apps, or a
"fastboot" if not.  A fastboot is dropping to "init 1" effectively,
then going back to "init 5".

Configuration Management
------------------------

By now if you've thought about this problem domain before, you're wondering
about configuration management.  In other words, if the OS is read only,
how do I edit /etc/sudoers?

Well, have you ever been a system administrator on a zypper/yum
system, done an RPM update, which then drops .rpmnew files in your
/etc/ that you have to go and hunt for with "find" or something, and
said to yourself, "Wow, this system is awesome!!!" ?  Right, that's
what I thought.

Configuration (and systems) management is a tricky problem, and I
certainly don't have a magic bullet.  However, one large conceptual
improvement I think is defaulting to "rebase" versus "merge".

This means that we won't permit direct modification of /etc - instead,
you HAVE to write a script which accomplishes your goals.  To generate
a tree, we check out a new copy, then run your script on top.

If the script fails, we can roll back the update, or drop to a shell
if interactive.

However, we also need to consider cases where the administrator
modifies state indirectly by a program.  Think "adduser" for example.

Possible approaches:

 1. Patch all of these programs to know how to write to the writable
    location, instead of the R/O bind mount overlay.
 2. Move the data to /var

What about "packages"?
----------------------

Basically I think they're a broken idea.  There are several different
classes of things that demand targeted solutions:

 1. Managing and upgrading the core OS (ostree)
 2. Managing and upgrading desktop applications (gnome-shell, glick?)
 3. System extensions - these are arbitrary RPMs like say the nVidia driver.
    We apply them after constructing each root.  Media codecs also fall
    into this category.

How one might install say Apache on top of ostree is an open
question - I think it probably makes sense honestly to ship services
like this with no configuration - just the binaries.  Then admins can
do whatever they want.

Downloads
---------

I'm pretty sure ostree should be significantly better than RPM with
deltarpms.  Note we only download changed objects.  If say just one
translation changes, we only download that new translation!  One
problem we will have to hunt down is programs that inject
e.g. timestamps into generated files.  "gzip" is the canonical
offender here.

Upstream branches
----------------

Note that this system will make it easy to have multiple *upstream* roots too.
For example, something like:

 - builds

   A filesystem tree generated after every time an artifact is built.

 - fastqa

   After each root is built, a very quick test suite is run in it;
   probably this is booting to GDM.  If that works, a new commit is
   made here.  Hopefully we can get fastqa under 2 minutes.

 - dailyqa

   Much more extensive tests, let's say they take 24 hours.

RPM Compatibility
-----------------

We should be able to install LSB rpms.  This implies providing "rpm".
The tricky part here is since the OS itself is not assembled via RPMs,
we need to fake up a database of "provides" as if we were.  Even
harder would be maintaining binary compatibilty with any arbitrary
%post scripts that may be run.

Note these RPMs act like local configuration - they would be
reinstalled every time you switch roots.

What about BTRFS?  Doesn't it solve everything?
-----------------------------------------------

In short, BTRFS is not a magic bullet, but yes - it helps
significantly.  The obvious thing to do is layer BTRFS under dpkg/rpm,
and have a separate subvolume for /home so rollbacks don't lose your
data.  See e.g.
<http://fedoraproject.org/wiki/Features/SystemRollbackWithBtrfs>

As a general rule an issue with the BTRFS is that it can't roll back
just changes to things installed by RPM (i.e. what's in rpm -qal).

For example, it's possible to e.g. run yum update, then edit something
in /etc, reboot and notice things are broken, then roll back and have
silently lost your changes to /etc.

Another example is adding a new binary in /usr/local.  You could say,
"OK, we'll use subvolumes for those!".  But then what about /var (and
your VM images that live in /var/lib/libvirt ?)

Finally, probably the biggest disadvantage of the rpm/dpkg + BTRFS
approach is it doesn't solve the race conditions that happen when
unpacking packages into the live system.  This problem is really
important to me.

Note though ostree can definitely take advantage of BTRFS features!
In particular, we could use "reflink"
<http://lwn.net/Articles/331808/> instead of hard links, and avoid
having the object store corrupted if somehow the files are modified
directly.

Other systems
-------------

I've spent a long time thinking about this problem, and here are some
of the other possible solutions out there I looked at, and why I
didn't use them:

 - Git: <http://git-scm.com/>

    Really awesome, and the core inspiration here.  But like I mentioned
    above, not at all designed for binaries - we can make different tradeoffs.

 - bup: <https://github.com/apenwarr/bup>

    bup is cool.  But it shares the negative tradeoffs with git, though it
    does add positives of its own.  It also inspired me.

 - NixOS: <http://nixos.org/>

    The NixOS people have a lot of really good ideas, and they've definitely
    thought about the problem space.  However, their approach of checksumming
    all inputs to a package is pretty wacky.  I don't see the point, and moreover
    it uses gobs of disk space.

 - Conary: <http://wiki.rpath.com/wiki/Conary:Updates_and_Rollbacks>

   If rpm/dpkg are like CVS, Conary is closer to Subversion.  It's not
   bad, but ostree is better than it for the exact same reasons git
   is better than Subversion.

 - BTRFS: <http://en.wikipedia.org/wiki/Btrfs>

   See above.

 - Jhbuild: <https://live.gnome.org/Jhbuild>

   What we've been using in GNOME, and has the essential property of allowing you
   to "fall back" to a stable system.  But ostree will blow it out of the water.