Commit Graph

2148 Commits

Author SHA1 Message Date
Colin Walters
152dcf89b5 repo: detached sigs: Use error prefixing instead of overwriting
Noted when "rpm-ostree compose sign" failed to write to a repo due to
permissions.
2015-02-25 10:39:15 -05:00
Colin Walters
ace982b51c prepare-root: Move /sysroot instead of unmounting it
I originally thought this would fix a regression, but it turns out
this wasn't the bug.  But pushing anyways as it's just cleaner.

https://bugzilla.gnome.org/show_bug.cgi?id=743891
2015-02-24 15:06:35 -05:00
Giuseppe Scrivano
0a7a70ed67 build: build libbupsplit separately
It avoids gobject-introspection warnings:

src/libostree/bupsplit.h:42: Warning: OSTree: symbol='bupsplit_find_ofs': Unknown namespace for symbol 'bupsplit_find_ofs'
src/libostree/bupsplit.h:43: Warning: OSTree: symbol='bupsplit_selftest': Unknown namespace for symbol 'bupsplit_selftest'
src/libostree/bupsplit.h:33: Warning: OSTree: symbol='BUP_BLOBBITS': Unknown namespace for symbol 'BUP_BLOBBITS'
src/libostree/bupsplit.h:34: Warning: OSTree: symbol='BUP_BLOBSIZE': Unknown namespace for symbol 'BUP_BLOBSIZE'
src/libostree/bupsplit.h:35: Warning: OSTree: symbol='BUP_WINDOWBITS': Unknown namespace for symbol 'BUP_WINDOWBITS'
src/libostree/bupsplit.h:36: Warning: OSTree: symbol='BUP_WINDOWSIZE': Unknown namespace for symbol 'BUP_WINDOWSIZE'

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-23 17:07:13 -05:00
Giuseppe Scrivano
22af1d9633 ostree-repo-static-delta-processing: initialize "modev"
Can be gs_unref_variant'ed on an early error.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-23 16:41:42 -05:00
Giuseppe Scrivano
52822efd32 Fix GObject introspection annotation
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-23 16:41:42 -05:00
Giuseppe Scrivano
50cd3e971a ostree-repo-traverse.c: Fix documentation parameter name
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-23 16:41:42 -05:00
Colin Walters
2f205331f4 tests: Move test gpg keyring into writable tmpdir
At least the version of gpgme in RHEL7 wants to create a lock file,
work around this by copying it into the writable test temporary
directory.
2015-02-23 15:21:23 -05:00
Matthew Barnes
12c8ef1425 Add explicit zlib dependency
Needed for CRC32 computations.
2015-02-23 14:48:52 -05:00
Colin Walters
3b3708c312 repo: Port APIs used by prune to fd-relative *at calls
More of the general trend away from GFile * to the faster and more
secure world of *at().
2015-02-22 23:20:12 -05:00
Colin Walters
474b071055 libglnx: Use git.gnome.org's copy
Not the master from my laptop.
2015-02-22 22:30:53 -05:00
Colin Walters
46bd4657b2 checkout: Drop internal use of GFile *
Checkout was one of the first complex code paths I tried to convert to
*at().  I ended up keeping both, because I hit the "xattrs for a
symlink" problem.  Later, Florian gave me a workaround, and we started
using it here, but the GFile * parameters weren't deleted.  They're
not used, so do so now.
2015-02-22 21:30:23 -05:00
Colin Walters
9539408bb3 deploy: Also look for /usr/lib/os-release
With newer versions of systemd, the file has moved there.
2015-02-22 21:02:27 -05:00
Colin Walters
965a304a17 Use libglnx
Starting down the path of not using libgsystem.  The main win here
will be code sharing between ostree/rpm-ostree as well as going down
the path of not using GFile * for local files.
2015-02-22 21:02:27 -05:00
Giuseppe Scrivano
ef53e0100b tests: do not run tests/test-rollsum as part of make check
it is not really a unit test.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Conflicts:
	Makefile-tests.am
2015-02-21 17:19:20 -05:00
Matthew Barnes
5aa0d51d7a repo: Check for OSTREE_REPO in ostree_repo_new_default()
Convenience feature to avoid having to pass --repo options repeatedly.

Before falling back to the default system repository path, check for a
repository path defined by the OSTREE_REPO environment variable.
2015-02-19 20:44:34 -05:00
Daniel Drake
7c267d6089 libostree: set directory mtimes to 0 on checkout
We already set all file mtimes to 0 so that they are constant
over all checkouts, and can be made constant with a known value from
the system where the ostree was created.

However, this was not happening for directories. Zero their mtimes too.

This is important for shipping a fontconfig cache in the ostree;
the fontconfig cache files embed a directory mtime.
2015-02-18 18:27:18 -05:00
Colin Walters
c2d439f115 tests: Restore accidentally deleted Makefile bit
Not sure how this got deleted...
2015-02-16 10:29:14 -05:00
Colin Walters
c54df89771 deltas: Search for similar objects (possibly renamed across directories)
The previous diff algorithm was file tree based, and only looked
at modified files that lived at the same path.

However, components like the Linux kernel have versioned
subdirectories, e.g. /usr/lib/modules/$kver/.../ext4.ko.  We want to
be able to detect these "modified renames" so that we can compute
diffs (rollsum, bsdiff).
2015-02-16 10:10:35 -05:00
Colin Walters
0f74ed62b7 repo: Add a new iterator traversal API for commits
This is a more optimized version of the GFile * APIs, and is now used
internally by the previous ostree_repo_traverse().
2015-02-16 10:10:35 -05:00
Colin Walters
08476ce254 deltas: Prune deltas when the corresponding "to" commit vanishes
We want prune to actually give you back disk space when using deltas.
2015-02-16 10:10:35 -05:00
Colin Walters
dbad2f6e19 deltas: Make syntax-check happy 2015-02-16 10:10:35 -05:00
Colin Walters
9aa7e30b38 deltas: Implement rollsums
This does an rsync-style prepared delta basically.  On my test data,
it shaves ~6MB of uncompressed data.  Not a huge amount, but I expect
this to be more useful for things like binaries which embed data, etc.
2015-02-16 10:10:35 -05:00
Colin Walters
345754a564 deltas: Initial code to copy content from existing objects
This is preparatory work for implementing rollsum support.
2015-02-16 10:10:35 -05:00
Colin Walters
7900c82a36 deltas: Flesh out the open/write/close opcodes
Refactor open-splice-and-close to call open/close.  We can't just call
write as that would require duplicating the object size parameter.
2015-02-16 10:10:35 -05:00
Colin Walters
96181da26a deltas: Use the new internal streaming APIs
This is significantly more efficient.
2015-02-16 10:10:35 -05:00
Colin Walters
7aea18cf0d deltas: Stub out a few more opcodes 2015-02-16 10:10:35 -05:00
Colin Walters
8fb29c9e5d deltas: Print total size of rollsums we would use 2015-02-16 10:10:35 -05:00
Colin Walters
3df8be0d92 deltas: Compute rollsum targets 2015-02-16 10:10:35 -05:00
Colin Walters
d749932f6b deltas: Rework format to allow streaming
There's still some silliness here, but there is now only one opcode
open-splice-and-close, that writes a single chunk from the payload.
This is really all we need for metadata, and small content objects are
also fine with this.

We get some deduplication between content objects by creating a
dictionary for (uid,gid,mode) tuples and xattrs.

This still keeps the operation/payload code in, so we could do
rollsums in a future update easily.
2015-02-16 10:10:35 -05:00
Colin Walters
513d47a90c deltas: Add _V0 to part #define
To make more explicitly clear that this is the version that matches
the version in the metadata.
2015-02-16 10:10:35 -05:00
Colin Walters
3c2a36eab0 deltas: Remove support for gzipped delta parts
XZ is really, really good.
2015-02-16 10:10:35 -05:00
Colin Walters
92cc3b5968 deltas: Use base64 for csums, add version to parts 2015-02-16 10:10:35 -05:00
Colin Walters
ee4e393fa1 repo: Store pending objects in prefixed subdirectory
I was hitting a bug in libguestfs/guestmount/FUSE where it blew up
with EINVAL on directories containing lots of files (more than
32000?).  We really want to use prefixed subdirs just like the real
objects/ directory does.

This allows us to share more code between the paths, is more
efficient, etc.
2015-02-15 15:30:19 -05:00
Colin Walters
49bdbf1db0 repo: Fix major performance regression with --scan-hardlinks
gnome-continuous uses the ostree_repo_scan_hardlinks() mode to
avoid re-checksumming everything.  However, when I ported the commit
code to use openat() and friends, this optimization was lost.

Re add it.  The difference is about 15s versus 5 minutes.
2015-02-13 16:36:07 -05:00
Colin Walters
64363c26ac pull-local: Fix regression with absolute paths
Don't add cwd unless the path is relative.
2015-02-13 11:39:36 -05:00
Colin Walters
9cc9804195 Change pull-local to just be a wrapper for pull with file:///
This follows up from the previous commit; now that pull knows how to
do the efficient link() or copy for local files, we can just have
pull-local call into ostree_repo_pull().

As part of this:
 - pull() can also accept a file:/// URI instead
   of a remote name (since pull local supports anonymous pulls)
 - pull() knows an "override-remote-name" option, since pull-local
   supported writing a ref out even if there wasn't a remote with
   that name
2015-02-08 04:43:06 -05:00
Colin Walters
ab3bf493dd pull: Optimize file:/// URIs to skip libsoup and hardlink if possible
It's always been suboptimal to have both pull and pull-local; as we go
beyond the raw object data into things like deltas and summary files,
the logic to perform e.g. mirroring should only be in one place.

This will be used by Pulp's OSTree content plugin at least to perform
promotions.
2015-02-06 10:02:57 -05:00
Colin Walters
be4ad54800 Add an internal API to get a read fd for a content object
This will be used by the static deltas work.
2015-02-05 21:50:26 -05:00
Colin Walters
1fdecbd263 pull: Copy the upstream summary file when doing a pull --mirror
While it could be regenerated downstream, there might be other
metadata upstream, and the goal here is a mirror.

https://bugzilla.gnome.org/show_bug.cgi?id=739377
2015-02-05 21:24:21 -05:00
Colin Walters
fab1e113db When mirroring, write content directly, do not verify
When doing a pull --mirror from an archive-z2 repository into another
archive-z2 repository, currently we gunzip/checksum/gzip each content
object.  The re-gzip process in particular is fairly expensive.

This does assume that the upstream content is trusted and correct.
It'd be nice in the future to do at least a CRC check, if not the full
checksum.  (Could we append CRC data to the end of filez objects?)

We could also choose to only do this optimization if fetching over
TLS.

before: 1626 metadata, 20320 content objects fetched; 299634 KiB transferred in 62 seconds
after : 1626 metadata, 20320 content objects fetched; 299634 KiB transferred in 11 seconds
2015-02-05 21:24:21 -05:00
Colin Walters
247866a9bc Add an internal API to stream content objects
For future delta work where we do more interesting things than just
"tar of new objects", this lays the groundwork for doing streaming
writes into content objects.

It's also more efficient, as we avoid many intermediate allocations
and virtual calls.  Just a single `g_output_stream_write_all` for the
splice case.

Conflicts:
	src/libostree/ostree-repo-private.h
	src/libostree/ostree-repo-static-delta-processing.c
2015-02-05 19:15:27 -05:00
Colin Walters
65afe1110d util: Add an API to atomic-replace a file, dirfd relative, optional fsync
This can be used in other places too, but I plan to use it to write
the summary file.
2015-02-05 16:59:52 -05:00
Colin Walters
3b8ed12ab2 repo: Hold an fd "repo_dir_fd" open for the toplevel too
We could just make everything relative to this, but the objects/ and
tmp/ are accessed very often, so I think it's worth holding individual
fds.

This fd can cover everything else: refs, deltas, etc.
2015-02-05 14:15:34 -05:00
Colin Walters
e739677ea1 prepare-root: Update comments 2015-02-04 05:32:45 -05:00
Daniel Drake
4f75d4ea0b prepare-root: avoid double-stacked /sysroot mount
prepare-root works with the mount that has been set up at /sysroot.
It creates a bind-mount within /sysroot (the deployment) and then moves
that mount to /sysroot.

Now we have 2 mounts both at /sysroot, and once we do switch_root, we will
never be able to unmount both of them. I'm not sure if this is ultimately
a kernel bug, but either way, ostree could do a bit more tidying up
after itself.
http://thread.gmane.org/gmane.linux.file-systems/92411

Easy way to reproduce:
1. Boot with rd.break param
2. At initramfs shell, run: ostree-prepare-root /sysroot
3. Observe two /sysroot mounts in /proc/mounts

Fix this by setting up the mounts at /sysroot.tmp, and unmounting the
original /sysroot before our new mount is MS_MOVEd on top of it.
2015-02-03 20:28:37 -05:00
Colin Walters
89a8b9b85a packaging: Add man5 pages 2015-02-03 20:28:37 -05:00
Giuseppe Scrivano
cfc344fbb0 tests: Add tests for ot-unix-utils
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-02 17:42:41 +01:00
Giuseppe Scrivano
a7b362998d configure.ac: Enable option subdir-objects for automake
It silences an automake warning and keep the rootdir cleaner.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-02 17:15:35 +01:00
Giuseppe Scrivano
53122dd2f9 tests: Move test-varint and test-rollsum under "make check"
"make check" tests are always enabled and they do not require to be
installed.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-02 17:14:52 +01:00
Giuseppe Scrivano
6a3959c895 syntax-check: Remove empty lines at the end of file
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-02 15:07:56 +01:00