Commit Graph

1084 Commits

Author SHA1 Message Date
Colin Walters
d1ddfc973c libotutil: Drop dead prototype
Was never implemented.
2012-11-29 15:56:17 -05:00
Colin Walters
f2b1be6a1e deploy: With --no-kernel, do copy host kernel modules
This is used by the qemu scripts, and there we *do* need the kernel
modules inside the target system.
2012-11-21 12:38:29 -05:00
Colin Walters
05e7b6d596 pull-local: Make multithreaded
We were blocking for easily 1/10 or 1/5 of a second in fdatasync(),
which drastically slows down the whole process.

This threading isn't quite as good as the ostree-pull command, but it
lets us avoid the dependency on libsoup everywhere, and it's simpler.
2012-11-19 12:08:41 -05:00
Colin Walters
de1ce843f1 core: Drop old GLIB_CHECK_VERSION(2.32) bits
We hard require 2.34 now; if it doesn't exist on the system, that's
what embedded dependencies are for.
2012-11-18 15:44:21 -05:00
Colin Walters
3f4d223361 admin prune: New builtin for cleaning up deployments and repo
After a while of pull-deploy cycles, you start to accumulate a lot of
them.  While the deployment read-only part is hardlinked, the -etc
space adds up.

Additionally, the repository itself just gets large.

The new command "ostree admin prune" deletes everything except the
"current" and "previous" deployments.
2012-11-16 17:41:46 -05:00
Colin Walters
05895780a3 deploy: Write refs into repo for current/previous deployments
We don't want this data to be gc'd by default by a prune.
2012-11-16 17:14:23 -05:00
Colin Walters
ffd774e2e6 prune: Report how much disk space we freed
Just nicer to see...
2012-11-16 15:45:05 -05:00
Colin Walters
7861b98673 prune: Add option to traverse refs only
The previous code (unintentionally) only traversed from refs; so data
only reachable from previous commits would be deleted.  That shouldn't
be the default, but we do want to offer it as an option.

So add a --refs-only option.
2012-11-16 12:50:13 -05:00
Colin Walters
51b1dd7cbd traverse: Skip already traversed commits
This is just more efficient, since e.g. "ostree prune" may invoke
ostree_traverse_commit() multiple times for the same commit.
2012-11-16 12:35:52 -05:00
Colin Walters
4169d628c5 prune: Default to deleting loose objects, code cleanup
We're getting closer to matching 'git gc'.
2012-11-16 10:58:35 -05:00
Colin Walters
ee533d6ab9 admin: code cleanup
Refactor internal functions so they can be used by a later patch.
2012-11-16 10:40:49 -05:00
Colin Walters
8f624ab51b ostree.spec.in: Update from Fedora review comments
See https://bugzilla.redhat.com/show_bug.cgi?id=819951
2012-11-16 10:11:39 -05:00
Colin Walters
2769ff36ff Release 2012.12 2012-10-31 22:02:35 -04:00
Jasper St. Pierre
cda5b28bab ot-main: Ignore lt- prefixes in binary names
libtool is the best

https://bugzilla.gnome.org/show_bug.cgi?id=682438
2012-10-24 12:20:15 -04:00
Colin Walters
cc4df4f3c2 admin: Add new pull-deploy command
Fetch the latest for the current tree, and deploy it in one go.
2012-10-22 20:41:49 -04:00
Colin Walters
f405737164 libgsystem: Update 2012-10-22 18:24:31 -04:00
Matthias Clasen
a0449ed246 Make ostree admin --help output more useful
At a minimum, it should list the available subcommands. This is
still not perfect, since there is no way to get at the help output
of the subcommands - getting that right needs more refactoring.

Signed-off-by: Colin Walters <walters@verbum.org>
2012-10-22 18:10:56 -04:00
Colin Walters
5a51bee024 core: quiet a compiler warning 2012-10-17 20:54:52 -04:00
Colin Walters
0c3e901dae core: Check out files before directories
Otherwise we pretty easily hit maximum FD limits =(
2012-10-15 20:03:13 -04:00
Colin Walters
bdbf76b6eb core: Ensure correct file data is synced in non-overwrite case 2012-10-15 18:44:55 -04:00
Colin Walters
1642310486 core: Use linkat() rather than link() in threaded checkouts
This seems to work around a likely Linux kernel VFS bug, where I
randomly see ENOENT on link() when we *definitely* called mkdir() at
an earlier point in time.
2012-10-15 10:49:48 -04:00
Colin Walters
d6956bfc19 core: And one more race in multithreaded mkdir -p
Sigh =(
2012-10-15 10:49:48 -04:00
Colin Walters
c0a6b5da71 core: Fix devino speedup for archive-z 2012-10-14 17:19:30 -04:00
Colin Walters
0d04738801 core: Rework archive-z mode to have header be uncompressed
This is an incompatible change to archive-z, thus it is now renamed to
archive-z2 and ostree will no longer parse archive-z.

I noticed in perf that we were spending some time zlib-decompressing
file headers, which is just inefficient.  Rather than do this, keep
the headers uncompressed, and just zlib-compress content.
2012-10-14 17:10:57 -04:00
Colin Walters
2b90e987bf core: Call fdatasync() before we rename()
Just noticed this from strace, doesn't really matter, but looks nicer.
2012-10-14 15:34:33 -04:00
Colin Walters
b0b409f82e checkout: Drop uncompressed object GC for now
It's racy when multiple processes are involved, and needs a bit more
thought.
2012-10-13 19:22:29 -04:00
Colin Walters
5b512d4c78 core: Fix case where we hardlinked successfully but thought we didn't
We need to return hardlink_supported even if we had to do an unlink
first.
2012-10-13 18:41:47 -04:00
Colin Walters
ad26c7a80b core: A few more error-prefixings 2012-10-13 18:02:39 -04:00
Colin Walters
c159188b81 core: One more minor tweak to recursive directory creation
We would fail in the case where we were trying to create a
subdirectory of /; this didn't actually happen, just noticed
via code inspection.
2012-10-13 17:54:19 -04:00
Colin Walters
a284075731 core: Prefix some error messages
To help debug why I still have this race condition...
2012-10-13 17:46:24 -04:00
Colin Walters
6b09f90122 core: Threadsafe recursive direction creation is harder than it appears 2012-10-13 10:47:03 -04:00
Colin Walters
b4d408f483 core: Fix regression in previous commit 2012-10-12 15:25:25 -04:00
Colin Walters
4436ed34d8 core: Make mkdir -p function safer
Recursing here is just a more obvious way to do it, rather than
relying on the semantics of g_file_make_directory_with_parents().
2012-10-12 15:10:39 -04:00
Colin Walters
238da603b8 core: Ensure file data is synced to disk when checking out via non-hardlinks
Otherwise we aren't crash-safe.
2012-10-12 13:09:10 -04:00
Colin Walters
e9dd22dc86 repo: Avoid race condition in threaded checkout with symbolic links
We were inconsistently putting symbolic links into the uncompressed
object cache, which could cause the threaded checkout to get confused.
2012-10-12 11:24:06 -04:00
Colin Walters
ee73c0fa34 core: Add dynamic uncompressed object cache for archive-z
This gives us something closer to the advantages of archive and
archive-z when using the latter.  Concretely we get deduplication
among multiple checkouts, along with the "devino" hash table trick
during commits to avoid checksumming content again.

This is enabled by default.
2012-10-11 18:33:03 -04:00
Colin Walters
3c0fe01152 pull: Print elapsed time 2012-10-05 17:20:40 -04:00
Colin Walters
52a0b7bdb1 core: Only do devino scan on commit (speeds up pull)
When fetching data remotely, there's no point to the devino scan
because we're not going to be committing local files.

Only do it for "commit".
2012-10-05 17:12:39 -04:00
Colin Walters
efe27f1b05 pull: Ensure worker queue finishes if we unhold an empty queue too 2012-10-04 20:32:14 -04:00
Colin Walters
9618232f4d pull: Stage content asynchronously
For similar reasons as metadata, this avoids having the main thread
blocked in fdatasync(), and even better - we can achieve much higher
parallelism if we have multiple threads blocked on fdatasync().
2012-10-04 20:00:00 -04:00
Colin Walters
5b8e833351 pull: Stage metadata objects asynchronously
This avoids the main thread being blocked on fdatasync(); also as a
bonus we checksum metadata in a separate thread too.
2012-10-04 18:24:37 -04:00
Colin Walters
f1b4db15a2 pull: Ensure queued filemeta requests don't starve everything else
We need this hack for "archive mode" repositories; otherwise,
what ends up happening is that we get 10000+ requests pending
for .filemeta files, which we can't process until we also get
the .filecontent.

Note this hack is unneccessary when fetching from archive-z
repositories.
2012-10-04 18:23:18 -04:00
Colin Walters
4e51701bea pull: Merge metadata and content fetch phases
This is a notable speedup when the metadata scanner is working, and we
don't have a lot of traffic; we can pull down data at the same time.
2012-10-04 15:37:23 -04:00
Colin Walters
5c1dc390ae pull: Asynchronous metadata fetch
Create a worker thread for processing metadata, reserving the main
thread for HTTP requests.

This can create a very significant efficiency win for large pull
requests since we are much more likely to keep a full pipeline open.

The status display is also nicer now.
2012-10-03 22:35:33 -04:00
Colin Walters
5f25762122 admin: Install grub2 config file if we detect /etc/grub.d
Decouple this from the kernel postinst one, since it's possible to
have one but not the other.
2012-10-01 19:34:25 -04:00
Colin Walters
8153799121 admin: Prepare /var/log/dracut.log when creating initramfs
Otherwise dracut complains.
2012-10-01 10:25:36 -04:00
Colin Walters
758d6132e3 Release 2012.11 2012-09-25 22:25:06 -04:00
Colin Walters
3e9c358628 Update libgsystem
This ensures we're disting the README, which makes the autogen.sh do
the right thing.
2012-09-25 22:25:06 -04:00
Colin Walters
044881b60f build: Add one header file missing from dist 2012-09-25 19:13:05 -04:00
Colin Walters
8922871733 build: Add autogen.sh, the COPYING.{GPL,LGPL} files to EXTRA_DIST
We want them in tarballs too.
2012-09-25 18:44:08 -04:00