Commit Graph

2334 Commits

Author SHA1 Message Date
Colin Walters
96d12b3f68 docs: Add a blurb on the summary file
We expect people to use it now, so let's actually describe what it is.
2016-02-27 11:28:54 -05:00
Colin Walters
3d7098bc5e Release 2016.3 2016-02-26 11:58:05 -05:00
Colin Walters
5345573470 deltas: Add a compression size heuristic for endianness detection
I see when analyzing a delta here that due to byteswapping a negative
compression ratio of 540%, 66%, and 28%.  Let's arbitrarily pick 20%
as a threshold for detecting byetswapping.
2016-02-26 08:19:01 -05:00
Colin Walters
7fdf072710 deltas: Heuristically detect endianness for older deltas
If the average object size is greater than 4GiB, let's assume we're
dealing with opposite endianness.  I'm fairly confident no one is
going to be shipping peta- or exa- byte size ostree deltas, period.
Past the gigabyte scale you really want bittorrent or something.
2016-02-26 08:19:01 -05:00
Colin Walters
04d77da005 deltas: Use endianness marker when parsing
Extend the `static-delta show` and `pull` commands to use the
endianness information (if available).
2016-02-26 08:19:01 -05:00
Colin Walters
277220aaa6 deltas: Include an endianness marker
We screwed up and had delta integers use host endianness.  Start
digging out by at least annotating the endianness.

https://bugzilla.gnome.org/show_bug.cgi?id=762515
2016-02-26 08:19:01 -05:00
Alexander Larsson
6c285d2e4a rofiles-fuse: Handle operations on the root
ENSURE_RELPATH breaks when path is "/". In that case we need
to return "." instead of "".

https://bugzilla.gnome.org/show_bug.cgi?id=762596
2016-02-24 20:43:23 +01:00
Alexander Larsson
1833142b14 deltas: Fix regression in ostree_repo_static_delta_execute_offline
xdg-app passed this a filename directly, and in this case it should be
used as is. This regressed to always look for "superblock" in the same
directory as the passed in filename.

https://bugzilla.gnome.org/show_bug.cgi?id=762617
2016-02-24 15:54:09 +01:00
Colin Walters
ed7266b10a deltas: Fix some more 32 bit warnings 2016-02-22 22:07:02 -05:00
Colin Walters
61b4f26862 lib: Two more compiler warning fixes 2016-02-22 21:11:10 -05:00
Colin Walters
88ae6f1b9e libarchive: Fix a 32 bit format warning
This is actually an error by default with our default CFLAGS.
2016-02-22 20:52:33 -05:00
Colin Walters
f4d494a0a4 Release 2016.2 2016-02-22 18:08:01 -05:00
Colin Walters
f461c02bb5 gpg: Use gpg_strerror_r for threadsafety
These APIs are rather painful...this is why GError exists.
2016-02-22 18:07:30 -05:00
Alexander Larsson
9f9d713a56 build: Use threadsafe GPGME
https://bugzilla.gnome.org/show_bug.cgi?id=761863
2016-02-22 18:07:30 -05:00
Colin Walters
30c5fb1a51 libarchive: Make autocreate_parents imply autocreating root dir
Some Docker layers are just metadata in the `layer.json`.  If one is
mapping Docker layers to OSTree commits, one needs to create a dummy
root directory, because OSTree doesn't support metadata-only commits.

Let's just push that logic down here because it's easier than special
casing it in higher levels.
2016-02-22 17:02:31 -05:00
Colin Walters
11b3050fd7 docs: Add a new formats section, move static deltas in there
The `src/libostree/README-deltas.md` was rather hidden - let's move
this into the manual.
2016-02-22 14:06:20 -05:00
Joaquim Rocha
6821ca1029 build: Link ostree with libarchive
libarchive (when available) is being used in ot-builtin-export.c so it
is necessary to link ostree with it.

https://bugzilla.gnome.org/show_bug.cgi?id=762457
2016-02-22 11:22:24 -05:00
Colin Walters
4beb5f4eaf pull: Add a --dry-run option for static deltas
One of the design goals with deltas was not just wire efficiency,
but also having all the data up front about how much data would
be transferred before starting.

Let's expose that better by adding a `dry-run` option to the pull API.
This requires static deltas to be useful.  Basically we simply call
the progress callback once with the data from the superblock.
2016-02-19 15:21:13 -05:00
Colin Walters
f2c5ecb996 pull: Add require-static-deltas pull option
For a production release repository, most OS vendors would want
to just always use static deltas.  Add the ability for the pulls to
require it.

(I think I'll also add a summary key for this actually in addition,
 so the repo manager can force it too)
2016-02-19 15:21:13 -05:00
Jonathan Lebon
eba7df0da9 ostree-sysroot: add debug option to help testing
If ostree is run in a test setup where it operates as root in a tmp
directory, it might cause issues to flag the deployments as immutable.
The test harness might simply be doing an `rm -rf` (effectively the case
for gnome-desktop-testing-runner), which will then fail.

We add a new debug option to the ostree_sysroot object using GLib's
GDebugKey functionality to allow our tests to communicate to ostree that
we don't want immutable deployments.
2016-02-19 11:58:08 -05:00
Colin Walters
3a555114bc repo: Add ostree_repo_import_archive_to_mtree
This is a more flexible version of the previous
ostree_repo_write_archive_to_mtree() which took a file reference.
This has an extensible options structure, and in particular
now supports `ignore_unsupported_content`.

I plan to use this for importing Docker images which contain device
nodes.  (There's no reason for container images to have those, so
we'll just ignore them).

Also here, just like the export variant, the caller is responsible for
setting up libarchive.
2016-02-19 08:54:04 -05:00
Colin Walters
b08b1abccd Rename libarchive write API to "export", matching command line
I was going to add new API for importing, and it was really confusing
that what I think of now as import and export both had "write" in the
name.  It's just clearer to talk about the direction.

At the same time, include `Export` in the options structure.

This isn't an ABI break as the API isn't in a release.
2016-02-19 08:54:04 -05:00
Colin Walters
707c14aeeb manual: Note that the bare-user mode exists 2016-02-18 18:21:32 -05:00
Colin Walters
d3eee15f84 rofiles-fuse: Fix truncate call to not use O_CREAT
I was getting a compilation error with the GCC hardening flags which
look for a missing mode with `O_CREAT`.  The right fix here is to drop
`O_CREAT`, as truncate() should throw `ENOENT` if the file doesn't
exist.
2016-02-17 11:06:53 -05:00
Colin Walters
2c2fd658ee packaging: Add a fuse subpackage
For rofiles-fuse.  Eventually what we should really do is split out
the shared library from the binaries?  A minimal system shouldn't need
rofiles-fuse, it's mainly for doing layered packages and that sort of
thing.
2016-02-17 11:06:53 -05:00
Colin Walters
baaf7450da Support Docker-style whiteouts
This is to enable importing Docker layers as ostree commits, then
checking them out in a union.

The prototype work for this is in:
https://github.com/cgwalters/dlayer-ostree

Though it will likely ultimately end up in:
https://github.com/projectatomic/atomic
2016-02-16 10:54:59 -05:00
Colin Walters
c18193bd19 man/ostree-export.xml: Add to git
Forgot to `git add`.
2016-02-15 06:39:49 -05:00
Colin Walters
f0b6b358cd manual-tests: New static-delta-generate-crosscheck.sh
I wanted to test static deltas vs a repo with "real" content
in it from Fedora Atomic.
2016-02-14 10:28:17 -05:00
Colin Walters
42c60effbe pull: Support specifying exact commit to pull via branch@commit
I don't know why we didn't do this a long time ago.  This extends the
pull API to allow grabbing a specific commit, and will set the branch
to it.  There's some support for this in the deploy engine, but there
are a lot of reasons to support it for raw pulls (such as subset
mirroring cases).

In fact I'm thinking we should also have the override-version logic
here too.

NOTE: One thing I debated here is inventing a new syntax on the
command line.  Git doesn't seem to have this functionality (probably
because it'd be rarely used). The '@' character at least doesn't
conflict with anything.

Anyways, I wanted this for some other test cases.  Without this,
writing tests that go between different commits is more awkward as one
must generate the content in one repo, then pull downstream, then
generate more content, then pull again.  But now I can just keep track
of commit IDs and do exactly what I want without synchronizing the
tests.
2016-02-14 10:12:26 -05:00
Colin Walters
355f8438ef Add an export builtin, and API to write to libarchive
At the moment I'm looking at using rpm-ostree to manage RPM inputs
which can then be converted into Docker images.  It's most convenient
if we can stream directly out of libostree rather than doing a
checkout + tar combination.

There are also backup/debugging etc. reasons to implement `export` as
well.
2016-02-14 09:53:01 -05:00
Colin Walters
e9ccdd2d00 Import rofiles-fuse
While it's not strictly tied to OSTree, let's move
https://github.com/cgwalters/rofiles-fuse in here because:

 - It's *very* useful in concert with OSTree
 - It's tiny
 - We can reuse OSTree's test, documentation, etc. infrastructure

One thing to consider also is that at some point we could experiment
with writing a FUSE filesystem for OSTree.  This could internalize a
better equivalent of `--link-checkout-speedup`, but on the other hand,
the cost of walking filesystem trees for these types of operations is
really quite small.

But if we did decide to do more FUSE things in OSTree, this is a step
towards that too.
2016-02-10 13:11:25 +01:00
Matthew Barnes
5adafd7674 fetcher: Fix hung GTlsInteraction
The GTlsInteraction instance must be created in the session thread
so it uses the correct GMainContext.
2016-02-09 00:58:17 +00:00
Colin Walters
0481389afd lib: Expand ostree static-delta show to show part stats
Now we display stats on the individual parts, such as the blob size
and the number of each type of opcode.  Most interesting to me is
things like how many bsdiff opcodes there are vs new objects, etc.
2016-02-08 14:46:13 +01:00
Colin Walters
56fc249d08 lib: Create an internal static delta parsing/opening function
We had code to deal with opening/checksumming/decompressing static
deltas in a few places.  I'd like to teach `ostree static-delta show`
how to display more information, and this will allow it to just use
`_ostree_static_delta_part_open()` too.
2016-02-08 14:46:13 +01:00
Colin Walters
8702ec7b3e build: Remove --disable-static-deltas option
I'm confident now we can declare them stable.
2016-02-08 14:46:13 +01:00
Colin Walters
0c15c9d6dd README.md: Update to link to Read The Docs, describe a bit better 2016-02-08 14:35:24 +01:00
Colin Walters
371856b6e4 docs-md: Delete (obsoleted by docs/)
This was just a prototype start at markdown docs that I never picked
back up.  Now that we have the Markdown/mkdocs done, delete this.
2016-02-08 14:24:24 +01:00
Giuseppe Scrivano
b0163d6182 build: 'make clean' removes parse-datetime.c
and fix make dist while at it.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-02-07 14:56:21 +01:00
Colin Walters
ba272008a7 packaging: Sync spec file with Fedora 2016-02-03 10:22:01 -05:00
Colin Walters
98d5f6e3db static-delta: Add show subcommand
Right now though, almost all of the details of deltas are private, so
we can't do the "honest thing" and have the command line just use the
shared library.

Eventually some of this should appear in the API, but for now add
command line which is useful for debugging.
2016-01-28 15:38:09 -05:00
Colin Walters
944b3abab1 build: Hoist man conditional higher
So we actually build with `--disable-man`.
2016-01-28 15:34:38 -05:00
Colin Walters
fa9e547e09 lib: Add a #define OSTREE_SHA256_DIGEST_LEN 32
And use it internally.  This way it's a bit less magical.
2016-01-28 15:24:16 -05:00
Colin Walters
c74ea8b075 build: Add --disable-man
Gnome Continuous doesn't have docbook, so copy what we do for glib.
2016-01-28 15:22:10 -05:00
Colin Walters
ca57ec4aa5 repo: Port -refs.c to openat()
I'd like to incrementally convert all of `ostree-repo*.c` to
fd-relative usage, so that we can sanely introduce
`ostree_repo_new_at()` which doesn't involve GFile.

This one is medium risk, but passes the test suite.
2016-01-28 14:57:13 -05:00
Colin Walters
5bab946b80 apidoc: Remove unnecessary srcdir != builddir workaround
It seems to be fine with `gtk-doc-1.19-3.el7.noarch`, so let's drop
this workaround, as it causes `make` warnings.
2016-01-28 09:31:37 -05:00
Colin Walters
64ebe2b82a Rewrite manual in mkdocs
I don't much like Docbook (and am considering converting the man pages
too), but let's start with the manual.

I looked at various documentation generators (there are a lot), and
I had a few requirements:

 - Markdown
 - Packaged in Fedora
 - Suitable for upload to a static webserver

`mkdocs` seems to fit the bill.
2016-01-28 09:31:37 -05:00
Colin Walters
32c360b5a0 build: Rename doc/ -> apidoc/
This is preparation for introducing a `mkdocs` manual under `doc/`
which should be significantly more useful for the world at large than
the minimal manual that exists there now.
2016-01-28 09:31:34 -05:00
Colin Walters
313b4720e8 build: Move man pages into man/
This is preparation for having 3 separate doc build systems (whee):

 - xsltproc for the man pages
 - gtk-doc for the API docs
 - mkdocs for a real manual
2016-01-28 09:31:34 -05:00
Colin Walters
5ebe43859d tests: Use "bash strict mode"
I noticed in the static deltas tests, there were some tests that
should have been under `-o pipefail` to ensure we properly propagate
errors.

There were a few places where we were referencing undefined variables.

Overall, this is clearly a good idea IMO.
2016-01-27 11:44:10 -05:00
Colin Walters
3d2322db7a grub2: Don't delete grub2.cfg.old file we just copied
The original intention here was that we'd keey around a copy of the
file so that grub2 could eventually learn how to do atomic updates by
checking for a "fully written" marker in the *new* file, and if it
didn't exist, falling back to grub2.cfg.old.

I haven't yet proposed that upstream, but we might as well stop
deleting the file since it's useful as a backup at least.

Reported-by: Gatis Paeglis
2016-01-27 10:46:12 -05:00