ostree/DESIGN

Hacktree

This is a combined build system and runtime daemon that builds and
manages root filesystems.  They both share an executable name
"hacktree", but are conceptually split up into two parts:

hacktree-build
hacktree-root-manager

== Problem statement ==

Hacking on the core operating system is painful - this includes most
of GNOME from upower and NetworkManager up to gnome-shell.  I want a
system that matches these requirements:

0) Does not disturb your existing OS
1) Is not terribly slow to use
2) Shares your $HOME - you have your data
3) Allows easy rollback
4) Ideally allows access to existing apps

== Comparison with existing tools ==

Virtualization:
 Fails on points 1) and 2).

Rebuilding OS packages:
 Fails on points 0) and 3).  Is also just very annoying.

jhbuild + OS packages:
  The state of the art in GNOME - but can only build non-root things -
  this means you can't build NetworkManager, and thus are permanently
  stuck on whatever the distro provides.

== Who is hacktree for? ==

First - operating system developers and testers.  I specifically keep
a few people in mind - Dan Williams and Eric Anholt, as well as myself
obviously.  For Eric Anholt, a key use case for him is being able to
try out the latest gnome-shell, and combine it with his work on Mesa,
and see how it works/performs - while retaining the ability to roll
back if one or both breaks.

The rollback concept is absolutely key for shipping anything to
enthusiasts or knowledable testers.  With a system like this, a tester
can easily perform a local rollback - something just not well
supported by dpkg/rpm.  (Why not Conary?  AIUI Conary is targeted at
individual roots, so while you could roll back a given root, it would
use significantly more disk space than hacktree)

Also, distributing operating system trees (instead of packages) gives
us a sane place to perform automated QA **before** we ship it to
testers.  We should never be wasting these people's time.

Even better, this system would allow testers to bisect across
operating system builds efficiently.

== The core idea ==

chroots are the original lightweight "virtualization".  Let's use
them.  So basically, you install a mainstream distribution (say
Debian).  It has a root filesystem with a regular layout, /etc, /usr,
/lib etc.

Now, what we can do is have a system that installs chroots, like:

/gnomeos/root-3.0-opt
/gnomeos/root-3.2-opt

These live in the same root filesystem as your regular distribution
(Note though, the root partition should be reasonably sized, or
 hopefully you've used just one big partition).

You should be able to boot into one of these roots.  Since hacktree
lives inside a distro created partition, a tricky part here is that we
need to know how to interact with the installed distribution's grub.
This is an annoying but tractable problem.

Hacktree will allow efficiently parallel installing and downloading OS
builds.

== The recipe set ==

A recipe is similar to Bitbake's format, except just have metadata -
we don't allow arbitrary Python scripts.  Also, we assume
autotools. Example:

SUMMARY = "The basic file, shell and text manipulation utilities."
HOMEPAGE = "http://www.gnu.org/software/coreutils/"
BUGTRACKER = "http://debbugs.gnu.org/coreutils"
LICENSE = "GPLv3+"
LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504\
                    file://src/ls.c;startline=5;endline=16;md5=e1a509558876db58fb6667ba140137ad"
SRC_URI = "${GNU_MIRROR}/coreutils/${BP}.tar.gz \
           file://remove-usr-local-lib-from-m4.patch \
          "
DEPENDS = "gmp foo"

Each recipe will output one or more artifacts.

== The Root Repository ==

A root repository is simply a git repository (at first).

Keep reading.

== A Root ==

A root is a filesystem tree, stored in the repository using git.  The
filesystem tree is built using a set of artifacts.  Roots are actually
read-only bind-mounts on top of a real writable location, which isn't
"public".

Each root is checkout of a branch in the repo.

TODO can we modify git to add the concept of a "raw" object type only
used for blobs?  The metadata for it could be stored in extended
attributes.  Then we could simply hard link the object for regular
files to their checkout, and almost totally deduplicate.
http://thread.gmane.org/gmane.comp.version-control.git/102001/focus=102094

In GNOME, we will have a root per:
 - major version (3.0, 3.2)
 - "runtime", "sdk", and "devel"
 - Build type (opt, debug)
 - Architecture (ia32, x86_64)

/gnome/root-3.2-runtime-opt-x86_64/{etc,bin,share,usr,lib}
/gnome/root-3.2-devel-debug-x86_64/{etc,bin,share,usr,lib}
/gnome/.real/root-3.2-runtime-opt-x86_64
/gnome/.real/root-3.2-devel-debug-x86_64

A "runtime" root is what's necessary to run applications.  A SDK root
is that, plus all the command line developer tools (gcc, gdb, make,
strace).  And finally the "devel" root has all the API-unstable
headers not necessary for applications (NetworkManager.h etc.)

Hmm, maybe we should punt developer tools into a Unix app framework.

== Artifact ==

An artifact is a binary result of compiling a recipe (there may be
multiple).  Think of an artifact as like a Linux distribution
"package", except there are no runtime dependencies, conflicts, or
pre/post scripts.  It's basically just a gzipped tarball, and we
encode metadata in the filename.

Example:

gdk-pixbuf-runtime,o=master,r=3.2-opt-x86_64,b=opt,v=2.24.0-10-g1d39095,.tar.gz

This is an artifact from the gdk-pixbuf component.  Here's a decoding of the key/value pairs:

o: The origin of the build - there are just "master" and "local"
r: The name of the root this artifact was compiled against
b: The name of the build flavor (known values are "opt" and "debug")
v: The output of "git describe".

To build a root, we simply unpack the artifacts that compose it, and
run "git commit".

hacktree will default to splitting up shared libraries' unversioned .so
link and header files into -devel, and the rest into -runtime.

All binaries default to runtime.

== Configuration Management ==

Have you ever been a system administrator on a zypper/yum system, done
an RPM update, which then drops .rpmnew files in your /etc/ that you
have to go and hunt for with "find" or something, and said to
yourself, "Wow, this system is awesome!!!" ?  Right, that's what I
thought.

Configuration (and systems) management is a tricky problem, and I
certainly don't have a magic bullet.  However, one large conceptual
improvement I think is defaulting to "rebase" versus "merge".

This means that we won't permit direct modification of /etc - instead,
you HAVE to write a script which accomplishes your goals.  It's
recommended to make this script revision controlled.  We just execute
/etc/gnomeos/config.d/* in alphabetical order, while the rootfs is
still mounted read-write.  

Hmm, probably we need to define it to operate on a shadow tree, which
we then diff?

If the script fails, we can roll back the update, or drop to a shell
if interactive.

== Local modifications ==

A key point of this whole endeavour is that we want developers to be
able to do local builds.  This is surprisingly something not well
supported by the Debian/Fedora's tools at least.

=== Updating a root with a new local artifact ===

Whenever you install a local artifact, if no "local" branch exists for
that root, it's created.

Let's say we're debugging gdk-pixbuf, tracking down a memory
corruption bug.  We've added a few printfs, and want to rerun things.
GCC optimization is screwing us, so we build it in debug mode (-O0).
The active root is root-3.2-opt.

$ pwd
~/src/gdk-pixbufroot
$ echo $HACKTREE_ROOT
/gnome/root-3.2-opt
<hack hack hack>
$ hacktree make debug
<time passes, hopefully not too much>
$ ls gdk-pixbuf*.tar.gz
gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
gdk-pixbuf-devel,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
gdk-pixbuf-manifests,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
$ hacktree install gdk-pixbuf*,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
<policykit auth dialog>

Now here's where the cool stuff happens.  hacktree takes
/gnome/root-3.2-opt (the which is given in the r= above), and looks
for the corresponding git branch (root-3.2-opt).  Now hacktree notices
there's no corresponding "local" branch, i.e. local-3.2-opt.  One is
created and checked out:

# pwd
/gnome/repo.git
# git branch local-3.2-opt root-3.2-opt
# git clone --branch local-3.2-opt /gnome/repo.git /gnome/.real-local-3.2-opt

Now, the artifacts specified are overlaid:

# cd /gnome/.real-local-3.2-opt
# tar xvf

Ok, now we need to remove old no longer shipped files from the root.
Thus, we need a list of files corresponding to each original artifact,
and to know which artifacts are in a root.  Note above that one of the
artifacts produced was "manifests".  This contains files like:

/meta/manifests/gdk-pixbuf-runtime.list
/meta/manifests/gdk-pixbuf-devel.list

Thus we diff the manifests, and clean up any leftover files.

# git commit -a -m "Install artifact gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz"
# git checkout 


== Updating roots from master ==

We need to split this into two phases - download, and apply.  Luckily,
git exposes exactly the operations we need, namely "git fetch" and
"git checkout".

hacktree fetch

== Many roots on build master ==

builds.git
  Generated after every time an artifact is built.
fastqa.git
  After each root is built, a very quick test suite is run in it;
  probably this is booting to GDM.  If that works, the latest build
  is committed here.  Hopefully we can get fastqa under 2 minutes.
dailyqa.git
  Much more extensive tests, let's say they take 24 hours.


== TODO figure out ==

* devel/runtime split for things with binaries (buildapi)
* xserver uses cpp (ugh)
 -> try to move artifact split into upstream git?
 -> $#@$!$ fix it 
* Language packs?
 -> System extension
* what about optional random ui tools that are built with tracker?
 -> don't build them by default (--disable-ui-tools), make them an app if cared


== RPM Compatibility ==

We should be able to install LSB rpms.  This implies providing "rpm".
The tricky part here is since the OS itself is not assembled via RPMs,
we need to fake up a database of "provides" as if we were.  Even
harder would be maintaining binary compatibilty with any arbitrary
%post scripts that may be run.