Commit Graph

2501 Commits

Author SHA1 Message Date
Colin Walters
56fc249d08 lib: Create an internal static delta parsing/opening function
We had code to deal with opening/checksumming/decompressing static
deltas in a few places.  I'd like to teach `ostree static-delta show`
how to display more information, and this will allow it to just use
`_ostree_static_delta_part_open()` too.
2016-02-08 14:46:13 +01:00
Colin Walters
8702ec7b3e build: Remove --disable-static-deltas option
I'm confident now we can declare them stable.
2016-02-08 14:46:13 +01:00
Colin Walters
0c15c9d6dd README.md: Update to link to Read The Docs, describe a bit better 2016-02-08 14:35:24 +01:00
Colin Walters
371856b6e4 docs-md: Delete (obsoleted by docs/)
This was just a prototype start at markdown docs that I never picked
back up.  Now that we have the Markdown/mkdocs done, delete this.
2016-02-08 14:24:24 +01:00
Giuseppe Scrivano
b0163d6182 build: 'make clean' removes parse-datetime.c
and fix make dist while at it.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-02-07 14:56:21 +01:00
Colin Walters
ba272008a7 packaging: Sync spec file with Fedora 2016-02-03 10:22:01 -05:00
Colin Walters
98d5f6e3db static-delta: Add show subcommand
Right now though, almost all of the details of deltas are private, so
we can't do the "honest thing" and have the command line just use the
shared library.

Eventually some of this should appear in the API, but for now add
command line which is useful for debugging.
2016-01-28 15:38:09 -05:00
Colin Walters
944b3abab1 build: Hoist man conditional higher
So we actually build with `--disable-man`.
2016-01-28 15:34:38 -05:00
Colin Walters
fa9e547e09 lib: Add a #define OSTREE_SHA256_DIGEST_LEN 32
And use it internally.  This way it's a bit less magical.
2016-01-28 15:24:16 -05:00
Colin Walters
c74ea8b075 build: Add --disable-man
Gnome Continuous doesn't have docbook, so copy what we do for glib.
2016-01-28 15:22:10 -05:00
Colin Walters
ca57ec4aa5 repo: Port -refs.c to openat()
I'd like to incrementally convert all of `ostree-repo*.c` to
fd-relative usage, so that we can sanely introduce
`ostree_repo_new_at()` which doesn't involve GFile.

This one is medium risk, but passes the test suite.
2016-01-28 14:57:13 -05:00
Colin Walters
5bab946b80 apidoc: Remove unnecessary srcdir != builddir workaround
It seems to be fine with `gtk-doc-1.19-3.el7.noarch`, so let's drop
this workaround, as it causes `make` warnings.
2016-01-28 09:31:37 -05:00
Colin Walters
64ebe2b82a Rewrite manual in mkdocs
I don't much like Docbook (and am considering converting the man pages
too), but let's start with the manual.

I looked at various documentation generators (there are a lot), and
I had a few requirements:

 - Markdown
 - Packaged in Fedora
 - Suitable for upload to a static webserver

`mkdocs` seems to fit the bill.
2016-01-28 09:31:37 -05:00
Colin Walters
32c360b5a0 build: Rename doc/ -> apidoc/
This is preparation for introducing a `mkdocs` manual under `doc/`
which should be significantly more useful for the world at large than
the minimal manual that exists there now.
2016-01-28 09:31:34 -05:00
Colin Walters
313b4720e8 build: Move man pages into man/
This is preparation for having 3 separate doc build systems (whee):

 - xsltproc for the man pages
 - gtk-doc for the API docs
 - mkdocs for a real manual
2016-01-28 09:31:34 -05:00
Colin Walters
5ebe43859d tests: Use "bash strict mode"
I noticed in the static deltas tests, there were some tests that
should have been under `-o pipefail` to ensure we properly propagate
errors.

There were a few places where we were referencing undefined variables.

Overall, this is clearly a good idea IMO.
2016-01-27 11:44:10 -05:00
Colin Walters
3d2322db7a grub2: Don't delete grub2.cfg.old file we just copied
The original intention here was that we'd keey around a copy of the
file so that grub2 could eventually learn how to do atomic updates by
checking for a "fully written" marker in the *new* file, and if it
didn't exist, falling back to grub2.cfg.old.

I haven't yet proposed that upstream, but we might as well stop
deleting the file since it's useful as a backup at least.

Reported-by: Gatis Paeglis
2016-01-27 10:46:12 -05:00
Colin Walters
91a1f91440 refs: Add a missing goto out for error handling
If the `refs/remotes` directory doesn't exist, we'd trip an assertion.
2016-01-26 16:48:37 -05:00
Colin Walters
cd0a9d3435 Add a checkout option to skip fsync
This is a better followup to dc9239dd7b
since I wanted to do fsync-less checkouts in rpm-ostree too, and
replicating the "turn off fsync temporarily" was in retrospect just a
hack.

We can simply add a boolean to the checkout options.

https://github.com/GNOME/ostree/pull/172
2016-01-26 15:39:46 -05:00
Giuseppe Scrivano
a13b56f91c diff: do not traverse parent commits
The object count displayed included also the number of parent
commits.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-01-20 08:59:10 +01:00
Colin Walters
dc9239dd7b sysroot: Don't individually fsync dirs in checkout, rely on syncfs
Originally, a lot of the `fsync()` calls here were added for the
wrong reason - I was chasing a bug that ended up being the extlinux
bootloader not parsing 64 bit ext4 filesystems.  But since it looked
like corruption, I tried adding a lot more `fsync()` calls.

All we should have to do is use `syncfs()`.  If that doesn't work,
it's a kernel bug.

I'm making this change because skipping the individual fsyncs can be a
major performance win - it's easier for the FS to optimize, we do more
in parallel, etc.

https://bugzilla.gnome.org/show_bug.cgi?id=757117
2016-01-13 13:15:08 -05:00
Colin Walters
46c3fc5d76 repo: Note global transaction resume is legacy
See docs for details.

https://github.com/GNOME/ostree/pull/169
2016-01-13 13:09:20 -05:00
Colin Walters
587fc5a5a1 Release 2016.1 2016-01-12 08:55:06 -05:00
Colin Walters
5929ce9e0e repo: Add APIs for devino optimization between checkout -> commit
A fast way to generate new OSTree content using an existing
tree is to checkout (as hard links), add/replace files, then
call `ostree_repo_scan_hardlinks()`, then commit.

But `ostree_repo_scan_hardlinks()` scans the entire repo, which
can be slow if you have a lot of content.

All we really need is a mapping of (device,inode) -> checksum
just for the objects we checked out, then use that mapping
for commits.

This patch adds API so that callers can create a mapping via
`ostree_repo_devino_cache_new()`, then pass it to
`ostree_repo_checkout_tree_at()` which will populate it, and then
`ostree_repo_write_directory_to_mtree()` can consume it.

I plan to use this in rpm-ostree for package layering work.

Notes:
 - The old `ostree_repo_scan_hardlinks()` API still works.
 - I tweaked the cache to be a set with the checksum colocated with
   the key, to avoid a separate malloc block per entry.

https://github.com/GNOME/ostree/pull/167
2016-01-07 14:19:12 -05:00
Colin Walters
21fbc16bc3 repo: Expose dfd-relative mtree writes as public API
For use in rpm-ostree.  We already had all of the internals for this.
2016-01-05 17:43:08 -05:00
Jonathan Lebon
5e7ed2dd8b grub2_generate: load sysroot before using it
The logic for checking which bootversion to use tries to access
sysroot->bootversion if the user didn't specify an explicit bootversion
on the command-line nor through the env var. However, at that point, the
sysroot object is not yet initialized, so it will always return 0, even
when it's 1.

This would cause e.g. `grub2-mkconfig` to have no output for the BLS
entries whenever the entries were under `/boot/loader.1`.

Related: RHBZ1293986
2015-12-24 12:41:22 -05:00
Matthew Barnes
1f1bfbf711 fetcher: Lazily create tmp directory
The tmp directory is lazily created for each fetcher instance, since
it may require superuser permissions and some instances only need
_ostree_fetcher_request_uri_to_membuf() which keeps everything in
memory buffers.
2015-12-19 09:21:22 -05:00
Colin Walters
1810de2b51 deploy: Change large parts to be fd-relative, drop fsync
This is a continuation of earlier work to drop the individual fsync on
files/directories in favor of relying on `syncfs()` for speed.

As part of that cleanup, I'm porting it to be fd-relative.

I feel relatively confident about this change given that this area of
the code has notable test suite coverage, although that code runs as
non-root.
2015-12-19 08:24:51 -05:00
Colin Walters
3a4f90cf8e bootconfig: Add ostree_bootconfig_parser_write_at
This fd-relative API will be used by later libostree porting in the
deploy code path.
2015-12-19 08:24:51 -05:00
Colin Walters
506a891e36 deploy: Find kernel/initramfs consistently from filesystem
I'm porting the deployment code to be fd-relative, but part of the
logic was using `GFile` to talk to `OstreeRepoFile` to determine the
"bootcsum" (boot config checksum) before checking out the file tree.

We can avoid having both code paths by checking out the tree first,
then looking at it on the filesystem.
2015-12-19 08:24:51 -05:00
Giuseppe Scrivano
6f96571679 tests: add tests for prune --static-deltas-only
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-12-18 11:21:58 +01:00
Giuseppe Scrivano
44c6197b0a prune: add new flag --static-deltas-only
When specified, only the static deltas files are pruned.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-12-18 11:21:57 +01:00
Giuseppe Scrivano
6b1e495a7a repo: new function ostree_repo_prune_static_deltas
Extract existing code from ostree_repo_prune and add an argument COMMIT,
that controls which commit purge.  If not set, the old behavior is kept.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-12-18 11:21:57 +01:00
Matthew Barnes
460a4b2852 remote: Add "ostree remote summary" command
Downloads and prints a remote summary file and any signatures in an
easy-to-read format, or alternatively with the --raw option, prints
the summary GVariant data directly.

https://bugzilla.gnome.org/show_bug.cgi?id=759250
2015-12-17 15:49:51 -05:00
Matthew Barnes
1df16a7675 repo: Add ostree_repo_verify_summary()
Verifies signatures on a summary -- both taken as GBytes inputs -- and
returns an OstreeGpgVerifyResult.
2015-12-17 15:49:44 -05:00
Colin Walters
403e05af24 build: Fix srcdir != builddir
Caught by GContinuous.  Also change the file writes to be atomic,
otherwise we're not Ctrl-c safe.
2015-12-14 14:19:35 -05:00
Matthew Barnes
f0b143ca8a pull: Push a temporary main context for sync requests
Given the previous commit, which isolates SoupSession in a separate
thread, it should be safe to start pushing a temporary main context
for synchronous requests again.

This partially reverts 84fe2ff, which partially reverted 9f3d586.

Related to https://bugzilla.gnome.org/show_bug.cgi?id=753336
2015-12-14 11:11:34 -05:00
Matthew Barnes
54066420cf fetcher: Move the SoupSession to a separate thread
Move the SoupSession to a separate thread with its own isolated main
context and main loop.  All interaction with the SoupSession occurs
by way of idle sources attached to the session's main context, which
execute on the session's thread.

This should solve the problem of running an asynchronous fetch request
synchronously by pushing a new thread-default main context and iterating
a main loop until the request completes.  Prior to this, the new thread-
default main context would interfere with the SoupSession's own async
processing.
2015-12-14 11:11:29 -05:00
Matthew Barnes
af30fc764a fetcher: Add "config-flags" construct-only property
A lot of effort here just to avoid touching SoupSession directly in
ostree_fetcher_new().  The reason will become apparent in subsequent
commits.

Note this introduces generated enum/flags GTypes using glib-mkenums.
I could have just made the property type as plain integer, but doing
properties right will henceforth be easier now that the automake-fu
is established.
2015-12-14 09:41:29 -05:00
Alexander Larsson
96eed95720 repo: Allocate a tmpdir for each OstreeFetcher to isolate concurrent downloads
This way two pulls will not use the same tmpdir and accidentally
overwrite each other. However, consecutive OstreeFetchers will reuse
the tmpdirs, so that we can properly resume downloading large objects.

https://bugzilla.gnome.org/show_bug.cgi?id=757611
2015-12-14 08:39:11 +01:00
Alexander Larsson
f771461b4a repo: Use per-transaction staging dir
Concurrent pulls break since we're sharing the staging directory for
all transactions in the repo. This makes us use a per-transaction directory.

However, in order for resumes to work we first look for existing
staging directories and try to aquire an exclusive lock for them. If
we can't find any staging directory or they are all already locked,
then we create a new one.

https://bugzilla.gnome.org/show_bug.cgi?id=757611
2015-12-14 08:38:51 +01:00
Alexander Larsson
be19c88861 repo: Add _ostree_repo_allocate_tmpdir helper
This creates a subdirectory of the tmp dir with a selected prefix,
and takes a lockfile to ensure that nobody else is using the same directory.
However, if a directory with the same prefix already exists and is
not locked that is used instead.

The later is useful if you want to support some kind of resumed operation
on the tmpdir.

touch reused dirs

https://bugzilla.gnome.org/show_bug.cgi?id=757611
2015-12-14 08:37:55 +01:00
Alexander Larsson
211ca6795c Update to latest libglnx 2015-12-14 08:37:17 +01:00
Colin Walters
73d77a2193 build: Also add a configure check for YACC/bison
So we error out more nicely if not found.
2015-12-07 10:39:37 -05:00
Colin Walters
76f4507557 build: Delete generated parse-datetime.c file, use AM_V_GEN
Bison is a well known external dependency, so just require it.
Including the generated content in git means it may or may not
be regenerated based randomly on timestamps, etc.

Also use `$(AM_V_GEN)` so we get prettier output.
2015-12-07 10:33:23 -05:00
Colin Walters
8ba90a3341 cmdline: Fatally error if the timestamp in a commit is invalid
Previously we were just ignoring this, which hid a bug in
an earlier commit that generated them.

Also change the `commit` program to use both APIs - this
involves extra code, but not too much.

This way, reverting the fix with this on top caused the test suite to
fail.  Adding an active test for this would need a custom test program
using the C API, or adding a cmdline flag to the client, neither of
which quite seemed worth it.
2015-12-04 11:26:07 -05:00
Giuseppe Scrivano
f88a9733ce tests: add missing ${CMD_PREFIX}
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-12-04 11:10:51 -05:00
Giuseppe Scrivano
335ea3f339 parse-datetime: use the module from gnulib
Use the parse-datetime module from gnulib, and adapt it to not require
other modules as portability is not really an issue for us.

DATE can be specified in different formats, such as: "-1 week", "last
monday", "1 week ago".

Include the generated .c file in the repository so to not add another
dependency to Bison.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-12-04 11:10:46 -05:00
Colin Walters
6335dadb49 Release 2015.11 2015-12-04 09:53:03 -05:00
Colin Walters
ed9f5ce031 gpg-verifier: Fix compiler warning 2015-12-04 09:51:14 -05:00