shaba/ostree
1
0
Fork 0
mirror of https://github.com/ostreedev/ostree.git synced 2024-12-22 17:35:55 +03:00
Code Issues Projects Releases Wiki Activity

Update READMEs a bit

Browse Source
This commit is contained in:
Colin Walters 2012-02-03 08:42:07 -05:00
parent 9fb842443e
commit 6647050978
3 changed files with 66 additions and 191 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

128
README-build.md
View File

@ -1,128 +0,0 @@
NOTE THIS STUFF IS OUT OF DATE! I'm working on merging some of these
ideas into jhbuild for now.
== The recipe set ==
A recipe is similar to Bitbake's format, except just have metadata -
we don't allow arbitrary Python scripts. Also, we assume
autotools. Example:
SUMMARY = "The basic file, shell and text manipulation utilities."
HOMEPAGE = "http://www.gnu.org/software/coreutils/"
BUGTRACKER = "http://debbugs.gnu.org/coreutils"
LICENSE = "GPLv3+"
LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504\
file://src/ls.c;startline=5;endline=16;md5=e1a509558876db58fb6667ba140137ad"
SRC_URI = "${GNU_MIRROR}/coreutils/${BP}.tar.gz \
file://remove-usr-local-lib-from-m4.patch \
"
DEPENDS = "gmp foo"
Each recipe will output one or more artifacts.
In GNOME, we will have a root per:
- major version (3.0, 3.2)
- "runtime", "sdk", and "devel"
- Build type (opt, debug)
- Architecture (ia32, x86_64)
/gnome/root-3.2-runtime-opt-x86_64/{etc,bin,share,usr,lib}
/gnome/root-3.2-devel-debug-x86_64/{etc,bin,share,usr,lib}
/gnome/.real/root-3.2-runtime-opt-x86_64
/gnome/.real/root-3.2-devel-debug-x86_64
A "runtime" root is what's necessary to run applications. A SDK root
is that, plus all the command line developer tools (gcc, gdb, make,
strace). And finally the "devel" root has all the API-unstable
headers not necessary for applications (NetworkManager.h etc.)
Hmm, maybe we should punt developer tools into a Unix app framework.
== Artifact ==
An artifact is a binary result of compiling a recipe (there may be
multiple). Think of an artifact as like a Linux distribution
"package", except there are no runtime dependencies, conflicts, or
pre/post scripts. It's basically just a gzipped tarball, and we
encode metadata in the filename.
Example:
gdk-pixbuf-runtime,o=master,r=3.2-opt-x86_64,b=opt,v=2.24.0-10-g1d39095,.tar.gz
This is an artifact from the gdk-pixbuf component. Here's a decoding of the key/value pairs:
o: The origin of the build - there are just "master" and "local"
r: The name of the root this artifact was compiled against
b: The name of the build flavor (known values are "opt" and "debug")
v: The output of "git describe".
To build a root, we simply unpack the artifacts that compose it, and
run "git commit".
hacktree will default to splitting up shared libraries' unversioned .so
link and header files into -devel, and the rest into -runtime.
All binaries default to runtime.
Local modifications ==
A key point of this whole endeavour is that we want developers to be
able to do local builds. This is surprisingly something not well
supported by the Debian/Fedora's tools at least.
=== Updating a root with a new local artifact ===
Whenever you install a local artifact, if no "local" branch exists for
that root, it's created.
Let's say we're debugging gdk-pixbuf, tracking down a memory
corruption bug. We've added a few printfs, and want to rerun things.
GCC optimization is screwing us, so we build it in debug mode (-O0).
The active root is root-3.2-opt.
$ pwd
~/src/gdk-pixbufroot
$ echo $HACKTREE_ROOT
/gnome/root-3.2-opt
<hack hack hack>
$ hacktree make debug
<time passes, hopefully not too much>
$ ls gdk-pixbuf*.tar.gz
gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
gdk-pixbuf-devel,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
gdk-pixbuf-manifests,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
$ hacktree install gdk-pixbuf*,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
<policykit auth dialog>
Now here's where the cool stuff happens. hacktree takes
/gnome/root-3.2-opt (the which is given in the r= above), and looks
for the corresponding git branch (root-3.2-opt). Now hacktree notices
there's no corresponding "local" branch, i.e. local-3.2-opt. One is
created and checked out:
# pwd
/gnome/repo.git
# git branch local-3.2-opt root-3.2-opt
# git clone --branch local-3.2-opt /gnome/repo.git /gnome/.real-local-3.2-opt
Now, the artifacts specified are overlaid:
# cd /gnome/.real-local-3.2-opt
# tar xvf
Ok, now we need to remove old no longer shipped files from the root.
Thus, we need a list of files corresponding to each original artifact,
and to know which artifacts are in a root. Note above that one of the
artifacts produced was "manifests". This contains files like:
/meta/manifests/gdk-pixbuf-runtime.list
/meta/manifests/gdk-pixbuf-devel.list
Thus we diff the manifests, and clean up any leftover files.
# git commit -a -m "Install artifact gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz"
# git checkout

87
README.md
View File

@ -1,5 +1,5 @@
OSTree OSTree
======== ======
Problem statement Problem statement
----------------- -----------------
@ -67,7 +67,7 @@ Comparison with existing tools
stuck on whatever the distro provides. stuck on whatever the distro provides.
Who is ostree for? Who is ostree for?
------------------------------ ------------------
First - operating system developers and testers. I specifically keep First - operating system developers and testers. I specifically keep
a few people in mind - Dan Williams and Eric Anholt, as well as myself a few people in mind - Dan Williams and Eric Anholt, as well as myself
@ -115,16 +115,22 @@ lives inside a distro created partition, a tricky part here is that we
need to know how to interact with the installed distribution's grub. need to know how to interact with the installed distribution's grub.
This is an annoying but tractable problem. This is an annoying but tractable problem.
OSTree will allow efficiently parallel installing and downloading OS First, we install a kernel+initramfs alongside the distribution's.
builds. Then, we have a "trampoline" ostree-init binary which is statically
linked, and boot the kernel with init=/ostree/ostree-init. This then
takes care of chrooting and running the init binary.
An important note here is that we explicitly link /home in each root An important note here is that we bind mount the real /home. This
to the real /home. This means you have your data. This also implies means you have your data. This also implies we share uid/gid, so
we share uid/gid, so /etc/passwd will have to be in sync. Probably /etc/passwd will have to be in sync. Probably what we'll do is have a
what we'll do is have a script to pull the data from the "host" OS. script to pull the data from the "host" OS.
Other shared directories are /var and /root. Note that /etc is I've decided for now to move /var into /ostree to avoid sharing it
explicitly NOT shared! with the "host" distribution, because in practice we're likely
to hit incompatibilities.
Do note however /etc lives *inside* the OSTree; it's presently
versioned and readonly like everything else.
On a pure OSTree system, the filesystem layout will look like this: On a pure OSTree system, the filesystem layout will look like this:
@ -132,6 +138,7 @@ On a pure OSTree system, the filesystem layout will look like this:
|-- boot |-- boot
|-- home |-- home
|-- ostree |-- ostree
| |-- var
| |-- current -> gnomeos-3.2-opt-7e9788a2 | |-- current -> gnomeos-3.2-opt-7e9788a2
| |-- gnomeos-3.0-opt-393a4555 | |-- gnomeos-3.0-opt-393a4555
| | |-- etc | | |-- etc
@ -154,7 +161,6 @@ On a pure OSTree system, the filesystem layout will look like this:
| |-- sys | |-- sys
| `-- usr | `-- usr
|-- root |-- root
`-- var
Making this efficient Making this efficient
@ -204,19 +210,19 @@ can do this because again each checkout is designed to be read-only.
So we mentioned above there are: So we mentioned above there are:
/gnomeos/root-3.0-opt /ostree/gnomeos-3.2-opt-7e9788a2
/gnomeos/root-3.2-opt /ostree/gnomeos-3.2-opt-393a4555
There is also a "repository" that looks like this: There is also a "repository" that looks like this:
.ht/objects/17/a95e8ca0ba655b09cb68d7288342588e867ee0.file /ostree/repo/objects/17/a95e8ca0ba655b09cb68d7288342588e867ee0.file
.ht/objects/17/68625e7ff5a8db77904c77489dc6f07d4afdba.meta /ostree/repo/objects/17/68625e7ff5a8db77904c77489dc6f07d4afdba.meta
.ht/objects/17/cc01589dd8540d85c0f93f52b708500dbaa5a9.file /ostree/repo/objects/17/cc01589dd8540d85c0f93f52b708500dbaa5a9.file
.ht/objects/30 /ostree/repo/objects/30
.ht/objects/30/6359b3ca7684358a3988afd005013f13c0c533.meta /ostree/repo/objects/30/6359b3ca7684358a3988afd005013f13c0c533.meta
.ht/objects/30/8f3c03010cedd930b1db756ce659c064f0cd7f.meta /ostree/repo/objects/30/8f3c03010cedd930b1db756ce659c064f0cd7f.meta
.ht/objects/30/8cf0fd8e63dfff6a5f00ba5a48f3b92fb52de7.file /ostree/repo/objects/30/8cf0fd8e63dfff6a5f00ba5a48f3b92fb52de7.file
.ht/objects/30/6cad7f027d69a46bb376044434bbf28d63e88d.file /ostree/repo/objects/30/6cad7f027d69a46bb376044434bbf28d63e88d.file
Each object is either metadata (like a commit or tree), or a hard link Each object is either metadata (like a commit or tree), or a hard link
to a regular file. to a regular file.
@ -242,27 +248,36 @@ during an upgrade and reboot process, you either get the full new
system, or the old one. There is no "Please don't turn off your system, or the old one. There is no "Please don't turn off your
computer". We do this by simply using a symbolic link like: computer". We do this by simply using a symbolic link like:
/gnomeos -> /gnomeos-e3b0c4429 /ostree/current -> /ostree/gnomeos-3.4-opt-e3b0c4429
Where /gnomeos-e3b0c4429/ has the full regular filesystem tree with Where gnomeos-e3b0c4429 has the full regular filesystem tree with usr/
usr/ etc/ directories as above. To upgrade or rollback (there is no etc/ directories as above. To upgrade or rollback (there is no
difference internally), we simply check out a new tree into difference internally), we simply check out a new tree into
/gnomeos-b90ae4763 for example, then swap the symbolic link, then gnomeos-b90ae4763 for example, then swap the "current" symbolic link,
remove the old tree. then remove the old tree.
But does this mean you have to "reboot" for OS upgrades? Very likely, But does this mean you have to reboot for OS upgrades? Very likely,
yes - and this is no different from RPM/deb or whatever. They just yes - and this is no different from RPM/deb or whatever. They just
typically lie to you about it =) But read on. typically lie to you about it =)
Let's consider a security update to a shared library. We can download A typical model with RPM/deb is to unpack the new files, then use some
the update to the repository, build a new tree and atomically swap it IPC mechanism (SIGHUP, a control binary like /usr/sbin/apachectl) to
as above, but what if a process has the old shared library in use? signal the running process to reload. There are multiple problems
with this - one is that in the new state, daemon A may depend on the
updated configuration in daemon B. This may not be particularly
common in default configurations, but it's highly likely that that
some deployments will have e.g. apache talking to a local MySQL
instance. So you really want to do is only apply the updated
configuration when all the files are in place; not after each RPM or
.deb is installed.
Here's where we will probably need to inspect which processes are What's even harder is the massive set of race conditions that are
using the library - if any are, then we need to trigger either a possible while RPM/deb are in the process of upgrading. Cron jobs are
logout/login if it's just the desktop shell and/or apps, or a very likely to hit this. If we want the ability to apply updates to a
"fastboot" if not. A fastboot is dropping to "init 1" effectively, live system, we could first pause execution of non-upgrade userspace
then going back to "init 5". tasks. This could be done via SIGSTOP for example. Then, we can swap
around the filesystem tree, and then finally attempt to apply updates
via SIGHUP, and if possible, restart processes.
Configuration Management Configuration Management
------------------------ ------------------------

42
gnomeos/README
View File

@ -16,19 +16,18 @@ base "runtime", and one "devel" with all of the development tools like
gcc. We then import that into an OSTree branch gcc. We then import that into an OSTree branch
e.g. "bases/gnomeos-3.4-yocto-i686-devel". e.g. "bases/gnomeos-3.4-yocto-i686-devel".
Now we also assume that you have ostree installed on the host build We also have a Yocto recipe "ostree-native" which generates (as you
system via e.g. jhbuild or RPM if doing a cross build. The core might guess) a native binary of ostree. That binary is used to import
ostbuild tool can then chroot into a checkout of the Yocto base, and into an "archive mode" OSTree repository. You can see it in
start generating artifacts. $builddir/tmp/deploy/images/repo.
Each generated artifact is committed to an OSTree branch like Now that we have an OSTree repository storing a base filesystem, we
"artifacts/gnomeos-3.4-i686-devel/libxslt/master/runtime". Then, a can use "ostbuild" which uses "linux-user-chroot" to chroot inside,
"compose" process merges together the individual filesystem trees into run a build on a source tree, and outputs binaries, which we then add
the final branches (e.g. gnomeos-3.4-i686-devel), and the process to the build tree for the next module, and so on.
repeats.
ostbuild details ostbuild details
------------------- ----------------
The simple goal of ostbuild is that it only takes as input a The simple goal of ostbuild is that it only takes as input a
"manifest" which is basically just a list of components to build. A "manifest" which is basically just a list of components to build. A
@ -39,24 +38,13 @@ There is no support for building from "tarballs" - I want the ability
to review all of the code that goes in, and to efficiently store to review all of the code that goes in, and to efficiently store
source code updates. source code updates.
For GNOME, tarballs are mostly pointless - it's easy enough to just The result of a build of a component is an OSTree branch like
run autogen.sh. However there are two challenges: "artifacts/gnomeos-3.4-i686-devel/libxslt/master". Then, a "compose"
process merges together the individual filesystem trees into the final
branches (e.g. gnomeos-3.4-i686-devel).
1) Tarballs for modules which self-build-depend may include Doing a full build on your system
pre-generated files. For example - flex's tarball includes a ---------------------------------
generated .c file for the parser. For these, we can either move
the module build to the Yocto level (thus giving a convenient way
to pull in host files), or possibly add the ability to
hardlink/copy in host binaries to ostbuild.
2) Tarballs which include translations pulled from a different
location. For example - bison. For these, we basically have to
maintain our own git repositories.
building
--------
srcdir=/src srcdir=/src
builddir=/src/build builddir=/src/build