mirror of
https://github.com/ostreedev/ostree.git
synced 2024-12-22 17:35:55 +03:00
Update READMEs a bit
This commit is contained in:
Colin Walters
parent
9fb842443e
commit
6647050978
128
README-build.md
128
README-build.md
@ -1,128 +0,0 @@
|
||||
NOTE THIS STUFF IS OUT OF DATE! I'm working on merging some of these
|
||||
ideas into jhbuild for now.
|
||||
|
||||
== The recipe set ==
|
||||
|
||||
A recipe is similar to Bitbake's format, except just have metadata -
|
||||
we don't allow arbitrary Python scripts. Also, we assume
|
||||
autotools. Example:
|
||||
|
||||
SUMMARY = "The basic file, shell and text manipulation utilities."
|
||||
HOMEPAGE = "http://www.gnu.org/software/coreutils/"
|
||||
BUGTRACKER = "http://debbugs.gnu.org/coreutils"
|
||||
LICENSE = "GPLv3+"
|
||||
LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504\
|
||||
file://src/ls.c;startline=5;endline=16;md5=e1a509558876db58fb6667ba140137ad"
|
||||
SRC_URI = "${GNU_MIRROR}/coreutils/${BP}.tar.gz \
|
||||
file://remove-usr-local-lib-from-m4.patch \
|
||||
"
|
||||
DEPENDS = "gmp foo"
|
||||
|
||||
Each recipe will output one or more artifacts.
|
||||
|
||||
|
||||
In GNOME, we will have a root per:
|
||||
- major version (3.0, 3.2)
|
||||
- "runtime", "sdk", and "devel"
|
||||
- Build type (opt, debug)
|
||||
- Architecture (ia32, x86_64)
|
||||
|
||||
/gnome/root-3.2-runtime-opt-x86_64/{etc,bin,share,usr,lib}
|
||||
/gnome/root-3.2-devel-debug-x86_64/{etc,bin,share,usr,lib}
|
||||
/gnome/.real/root-3.2-runtime-opt-x86_64
|
||||
/gnome/.real/root-3.2-devel-debug-x86_64
|
||||
|
||||
A "runtime" root is what's necessary to run applications. A SDK root
|
||||
is that, plus all the command line developer tools (gcc, gdb, make,
|
||||
strace). And finally the "devel" root has all the API-unstable
|
||||
headers not necessary for applications (NetworkManager.h etc.)
|
||||
|
||||
Hmm, maybe we should punt developer tools into a Unix app framework.
|
||||
|
||||
== Artifact ==
|
||||
|
||||
An artifact is a binary result of compiling a recipe (there may be
|
||||
multiple). Think of an artifact as like a Linux distribution
|
||||
"package", except there are no runtime dependencies, conflicts, or
|
||||
pre/post scripts. It's basically just a gzipped tarball, and we
|
||||
encode metadata in the filename.
|
||||
|
||||
Example:
|
||||
|
||||
gdk-pixbuf-runtime,o=master,r=3.2-opt-x86_64,b=opt,v=2.24.0-10-g1d39095,.tar.gz
|
||||
|
||||
This is an artifact from the gdk-pixbuf component. Here's a decoding of the key/value pairs:
|
||||
|
||||
o: The origin of the build - there are just "master" and "local"
|
||||
r: The name of the root this artifact was compiled against
|
||||
b: The name of the build flavor (known values are "opt" and "debug")
|
||||
v: The output of "git describe".
|
||||
|
||||
To build a root, we simply unpack the artifacts that compose it, and
|
||||
run "git commit".
|
||||
|
||||
hacktree will default to splitting up shared libraries' unversioned .so
|
||||
link and header files into -devel, and the rest into -runtime.
|
||||
|
||||
All binaries default to runtime.
|
||||
|
||||
Local modifications ==
|
||||
|
||||
A key point of this whole endeavour is that we want developers to be
|
||||
able to do local builds. This is surprisingly something not well
|
||||
supported by the Debian/Fedora's tools at least.
|
||||
|
||||
=== Updating a root with a new local artifact ===
|
||||
|
||||
Whenever you install a local artifact, if no "local" branch exists for
|
||||
that root, it's created.
|
||||
|
||||
Let's say we're debugging gdk-pixbuf, tracking down a memory
|
||||
corruption bug. We've added a few printfs, and want to rerun things.
|
||||
GCC optimization is screwing us, so we build it in debug mode (-O0).
|
||||
The active root is root-3.2-opt.
|
||||
|
||||
$ pwd
|
||||
~/src/gdk-pixbufroot
|
||||
$ echo $HACKTREE_ROOT
|
||||
/gnome/root-3.2-opt
|
||||
<hack hack hack>
|
||||
$ hacktree make debug
|
||||
<time passes, hopefully not too much>
|
||||
$ ls gdk-pixbuf*.tar.gz
|
||||
gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
|
||||
gdk-pixbuf-devel,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
|
||||
gdk-pixbuf-manifests,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
|
||||
$ hacktree install gdk-pixbuf*,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
|
||||
<policykit auth dialog>
|
||||
|
||||
Now here's where the cool stuff happens. hacktree takes
|
||||
/gnome/root-3.2-opt (the which is given in the r= above), and looks
|
||||
for the corresponding git branch (root-3.2-opt). Now hacktree notices
|
||||
there's no corresponding "local" branch, i.e. local-3.2-opt. One is
|
||||
created and checked out:
|
||||
|
||||
# pwd
|
||||
/gnome/repo.git
|
||||
# git branch local-3.2-opt root-3.2-opt
|
||||
# git clone --branch local-3.2-opt /gnome/repo.git /gnome/.real-local-3.2-opt
|
||||
|
||||
Now, the artifacts specified are overlaid:
|
||||
|
||||
# cd /gnome/.real-local-3.2-opt
|
||||
# tar xvf
|
||||
|
||||
Ok, now we need to remove old no longer shipped files from the root.
|
||||
Thus, we need a list of files corresponding to each original artifact,
|
||||
and to know which artifacts are in a root. Note above that one of the
|
||||
artifacts produced was "manifests". This contains files like:
|
||||
|
||||
/meta/manifests/gdk-pixbuf-runtime.list
|
||||
/meta/manifests/gdk-pixbuf-devel.list
|
||||
|
||||
Thus we diff the manifests, and clean up any leftover files.
|
||||
|
||||
# git commit -a -m "Install artifact gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz"
|
||||
# git checkout
|
||||
|
||||
|
||||
87
README.md
87
README.md
@ -1,5 +1,5 @@
|
||||
OSTree
|
||||
========
|
||||
======
|
||||
|
||||
Problem statement
|
||||
-----------------
|
||||
@ -67,7 +67,7 @@ Comparison with existing tools
|
||||
stuck on whatever the distro provides.
|
||||
|
||||
Who is ostree for?
|
||||
------------------------------
|
||||
------------------
|
||||
|
||||
First - operating system developers and testers. I specifically keep
|
||||
a few people in mind - Dan Williams and Eric Anholt, as well as myself
|
||||
@ -115,16 +115,22 @@ lives inside a distro created partition, a tricky part here is that we
|
||||
need to know how to interact with the installed distribution's grub.
|
||||
This is an annoying but tractable problem.
|
||||
|
||||
OSTree will allow efficiently parallel installing and downloading OS
|
||||
builds.
|
||||
First, we install a kernel+initramfs alongside the distribution's.
|
||||
Then, we have a "trampoline" ostree-init binary which is statically
|
||||
linked, and boot the kernel with init=/ostree/ostree-init. This then
|
||||
takes care of chrooting and running the init binary.
|
||||
|
||||
An important note here is that we explicitly link /home in each root
|
||||
to the real /home. This means you have your data. This also implies
|
||||
we share uid/gid, so /etc/passwd will have to be in sync. Probably
|
||||
what we'll do is have a script to pull the data from the "host" OS.
|
||||
An important note here is that we bind mount the real /home. This
|
||||
means you have your data. This also implies we share uid/gid, so
|
||||
/etc/passwd will have to be in sync. Probably what we'll do is have a
|
||||
script to pull the data from the "host" OS.
|
||||
|
||||
Other shared directories are /var and /root. Note that /etc is
|
||||
explicitly NOT shared!
|
||||
I've decided for now to move /var into /ostree to avoid sharing it
|
||||
with the "host" distribution, because in practice we're likely
|
||||
to hit incompatibilities.
|
||||
|
||||
Do note however /etc lives *inside* the OSTree; it's presently
|
||||
versioned and readonly like everything else.
|
||||
|
||||
On a pure OSTree system, the filesystem layout will look like this:
|
||||
|
||||
@ -132,6 +138,7 @@ On a pure OSTree system, the filesystem layout will look like this:
|
||||
|-- boot
|
||||
|-- home
|
||||
|-- ostree
|
||||
| |-- var
|
||||
| |-- current -> gnomeos-3.2-opt-7e9788a2
|
||||
| |-- gnomeos-3.0-opt-393a4555
|
||||
| | |-- etc
|
||||
@ -154,7 +161,6 @@ On a pure OSTree system, the filesystem layout will look like this:
|
||||
| |-- sys
|
||||
| `-- usr
|
||||
|-- root
|
||||
`-- var
|
||||
|
||||
|
||||
Making this efficient
|
||||
@ -204,19 +210,19 @@ can do this because again each checkout is designed to be read-only.
|
||||
|
||||
So we mentioned above there are:
|
||||
|
||||
/gnomeos/root-3.0-opt
|
||||
/gnomeos/root-3.2-opt
|
||||
/ostree/gnomeos-3.2-opt-7e9788a2
|
||||
/ostree/gnomeos-3.2-opt-393a4555
|
||||
|
||||
There is also a "repository" that looks like this:
|
||||
|
||||
.ht/objects/17/a95e8ca0ba655b09cb68d7288342588e867ee0.file
|
||||
.ht/objects/17/68625e7ff5a8db77904c77489dc6f07d4afdba.meta
|
||||
.ht/objects/17/cc01589dd8540d85c0f93f52b708500dbaa5a9.file
|
||||
.ht/objects/30
|
||||
.ht/objects/30/6359b3ca7684358a3988afd005013f13c0c533.meta
|
||||
.ht/objects/30/8f3c03010cedd930b1db756ce659c064f0cd7f.meta
|
||||
.ht/objects/30/8cf0fd8e63dfff6a5f00ba5a48f3b92fb52de7.file
|
||||
.ht/objects/30/6cad7f027d69a46bb376044434bbf28d63e88d.file
|
||||
/ostree/repo/objects/17/a95e8ca0ba655b09cb68d7288342588e867ee0.file
|
||||
/ostree/repo/objects/17/68625e7ff5a8db77904c77489dc6f07d4afdba.meta
|
||||
/ostree/repo/objects/17/cc01589dd8540d85c0f93f52b708500dbaa5a9.file
|
||||
/ostree/repo/objects/30
|
||||
/ostree/repo/objects/30/6359b3ca7684358a3988afd005013f13c0c533.meta
|
||||
/ostree/repo/objects/30/8f3c03010cedd930b1db756ce659c064f0cd7f.meta
|
||||
/ostree/repo/objects/30/8cf0fd8e63dfff6a5f00ba5a48f3b92fb52de7.file
|
||||
/ostree/repo/objects/30/6cad7f027d69a46bb376044434bbf28d63e88d.file
|
||||
|
||||
Each object is either metadata (like a commit or tree), or a hard link
|
||||
to a regular file.
|
||||
@ -242,27 +248,36 @@ during an upgrade and reboot process, you either get the full new
|
||||
system, or the old one. There is no "Please don't turn off your
|
||||
computer". We do this by simply using a symbolic link like:
|
||||
|
||||
/gnomeos -> /gnomeos-e3b0c4429
|
||||
/ostree/current -> /ostree/gnomeos-3.4-opt-e3b0c4429
|
||||
|
||||
Where /gnomeos-e3b0c4429/ has the full regular filesystem tree with
|
||||
usr/ etc/ directories as above. To upgrade or rollback (there is no
|
||||
Where gnomeos-e3b0c4429 has the full regular filesystem tree with usr/
|
||||
etc/ directories as above. To upgrade or rollback (there is no
|
||||
difference internally), we simply check out a new tree into
|
||||
/gnomeos-b90ae4763 for example, then swap the symbolic link, then
|
||||
remove the old tree.
|
||||
gnomeos-b90ae4763 for example, then swap the "current" symbolic link,
|
||||
then remove the old tree.
|
||||
|
||||
But does this mean you have to "reboot" for OS upgrades? Very likely,
|
||||
But does this mean you have to reboot for OS upgrades? Very likely,
|
||||
yes - and this is no different from RPM/deb or whatever. They just
|
||||
typically lie to you about it =) But read on.
|
||||
typically lie to you about it =)
|
||||
|
||||
Let's consider a security update to a shared library. We can download
|
||||
the update to the repository, build a new tree and atomically swap it
|
||||
as above, but what if a process has the old shared library in use?
|
||||
A typical model with RPM/deb is to unpack the new files, then use some
|
||||
IPC mechanism (SIGHUP, a control binary like /usr/sbin/apachectl) to
|
||||
signal the running process to reload. There are multiple problems
|
||||
with this - one is that in the new state, daemon A may depend on the
|
||||
updated configuration in daemon B. This may not be particularly
|
||||
common in default configurations, but it's highly likely that that
|
||||
some deployments will have e.g. apache talking to a local MySQL
|
||||
instance. So you really want to do is only apply the updated
|
||||
configuration when all the files are in place; not after each RPM or
|
||||
.deb is installed.
|
||||
|
||||
Here's where we will probably need to inspect which processes are
|
||||
using the library - if any are, then we need to trigger either a
|
||||
logout/login if it's just the desktop shell and/or apps, or a
|
||||
"fastboot" if not. A fastboot is dropping to "init 1" effectively,
|
||||
then going back to "init 5".
|
||||
What's even harder is the massive set of race conditions that are
|
||||
possible while RPM/deb are in the process of upgrading. Cron jobs are
|
||||
very likely to hit this. If we want the ability to apply updates to a
|
||||
live system, we could first pause execution of non-upgrade userspace
|
||||
tasks. This could be done via SIGSTOP for example. Then, we can swap
|
||||
around the filesystem tree, and then finally attempt to apply updates
|
||||
via SIGHUP, and if possible, restart processes.
|
||||
|
||||
Configuration Management
|
||||
------------------------
|
||||
|
||||
@ -16,19 +16,18 @@ base "runtime", and one "devel" with all of the development tools like
|
||||
gcc. We then import that into an OSTree branch
|
||||
e.g. "bases/gnomeos-3.4-yocto-i686-devel".
|
||||
|
||||
Now we also assume that you have ostree installed on the host build
|
||||
system via e.g. jhbuild or RPM if doing a cross build. The core
|
||||
ostbuild tool can then chroot into a checkout of the Yocto base, and
|
||||
start generating artifacts.
|
||||
We also have a Yocto recipe "ostree-native" which generates (as you
|
||||
might guess) a native binary of ostree. That binary is used to import
|
||||
into an "archive mode" OSTree repository. You can see it in
|
||||
$builddir/tmp/deploy/images/repo.
|
||||
|
||||
Each generated artifact is committed to an OSTree branch like
|
||||
"artifacts/gnomeos-3.4-i686-devel/libxslt/master/runtime". Then, a
|
||||
"compose" process merges together the individual filesystem trees into
|
||||
the final branches (e.g. gnomeos-3.4-i686-devel), and the process
|
||||
repeats.
|
||||
Now that we have an OSTree repository storing a base filesystem, we
|
||||
can use "ostbuild" which uses "linux-user-chroot" to chroot inside,
|
||||
run a build on a source tree, and outputs binaries, which we then add
|
||||
to the build tree for the next module, and so on.
|
||||
|
||||
ostbuild details
|
||||
-------------------
|
||||
----------------
|
||||
|
||||
The simple goal of ostbuild is that it only takes as input a
|
||||
"manifest" which is basically just a list of components to build. A
|
||||
@ -39,24 +38,13 @@ There is no support for building from "tarballs" - I want the ability
|
||||
to review all of the code that goes in, and to efficiently store
|
||||
source code updates.
|
||||
|
||||
For GNOME, tarballs are mostly pointless - it's easy enough to just
|
||||
run autogen.sh. However there are two challenges:
|
||||
The result of a build of a component is an OSTree branch like
|
||||
"artifacts/gnomeos-3.4-i686-devel/libxslt/master". Then, a "compose"
|
||||
process merges together the individual filesystem trees into the final
|
||||
branches (e.g. gnomeos-3.4-i686-devel).
|
||||
|
||||
1) Tarballs for modules which self-build-depend may include
|
||||
pre-generated files. For example - flex's tarball includes a
|
||||
generated .c file for the parser. For these, we can either move
|
||||
the module build to the Yocto level (thus giving a convenient way
|
||||
to pull in host files), or possibly add the ability to
|
||||
hardlink/copy in host binaries to ostbuild.
|
||||
|
||||
2) Tarballs which include translations pulled from a different
|
||||
location. For example - bison. For these, we basically have to
|
||||
maintain our own git repositories.
|
||||
|
||||
|
||||
|
||||
building
|
||||
--------
|
||||
Doing a full build on your system
|
||||
---------------------------------
|
||||
|
||||
srcdir=/src
|
||||
builddir=/src/build
|
||||
|
||||
Loading…
Reference in New Issue
Block a user