2989 Commits

Author SHA1 Message Date
Alexander Larsson
d0e74cf3af Fix pruning of partial commits
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.

In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.

Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.

This fixes https://github.com/ostreedev/ostree/issues/541

Closes: #542
Approved by: cgwalters
2016-10-24 17:48:19 +00:00
Colin Walters
d3f14f02e3 Release 2016.12
Closes: #540
Approved by: cgwalters
v2016.12
2016-10-21 19:24:31 +00:00
Alexander Larsson
67bddf76f7 detached metadata: Put these in transaction
If there is a transaction active, then we put writes to detached
metadata into the staging dir, and when reading it we look there
first. This allows transactions to be aborted half-way without
writing the detached metadata into the repository (possibly
overwriting any old metadata from there).

This fixes https://github.com/ostreedev/ostree/issues/526

Closes: #539
Approved by: giuseppe
2016-10-21 10:50:41 +00:00
Alexander Larsson
d43c121675 ostree_repo_read_commit_detached_metadata: Handle parent repo
If the detached metadata is not in the repo, try in the parent
repo if that is set.

Without this a commit will not gpg validate in the child repo

Closes: #539
Approved by: giuseppe
2016-10-21 10:50:41 +00:00
Alexander Larsson
cf6ec1bbbc Fix regression for symlinks in bare-user repos
Commit 1d4f1b8878addd28059c3a3928640491755cd615 started using hardlinks
checkouts of symlinks. However, symlinks are not stored as symlink in the
repo for bare-user repos, so this breaks user-mode checkouts of such repos.

We fix this by checking for !is_symlink in the bare-user case.

This fixes:
     https://github.com/ostreedev/ostree/issues/537

Closes: #538
Approved by: giuseppe
2016-10-21 09:24:27 +00:00
Jonathan Lebon
64568bc706 .redhat-ci.yml: use projectatomic/ostree-tester
Same Dockerfile, but automated to rebuild on pushes.

Closes: #536
Approved by: giuseppe
2016-10-20 08:35:13 +00:00
Jonathan Lebon
e62defa1a4 add .redhat-ci.yml and .redhat-ci.Dockerfile
Add a YAML file for the new Red Hat CI framework.

Rather than re-installing all the build deps everytime, which takes
time, I added a Dockerfile that we can wire up to the Docker Hub. For
now it lives at `jlebon/ostree-tester:rhci`, but we can move it under
the `projectatomic` org (or a new `ostree` org).

Closes: #535
Approved by: cgwalters
2016-10-19 20:35:23 +00:00
Colin Walters
1d4f1b8878 core: Do create hardlinks to symlinks for checkouts
I was noticing a recent performance issue with checkouts
which seemed to be mostly us going back to doing a `fsync()` on
directories.

Regardless, while looking at that, I saw we were spending time
creating new symlinks. Even though symlinks are small, it's still
better to hardlink them.

Going way back in time, the reason we weren't doing this is
because we were hitting `EMFILE` on ext4, but that was for
gnome-continuous which creates *many* build roots.  Even
there though, they're just a cache, and we handle `EMFILE`.

For ostree-for-host-system, we don't expect to have many roots (just 3
at most transiently), so hardlinking symlinks does make sense.

Closes: #521
Approved by: jlebon
2016-10-17 20:40:15 +00:00
Jonathan Lebon
a660e5650d OstreeFetcher: provide proxy credentials if needed
There seems to be an issue in libsoup which causes basic auth
credentials to not be passed to the proxy during requests. We thus have
to handle PROXY_UNAUTHORIZED responses and provide the auth ourselves.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1370558
Related: https://bugzilla.gnome.org/show_bug.cgi?id=772932

Closes: #529
Approved by: cgwalters
2016-10-14 16:06:08 +00:00
Jonathan Lebon
a0598cb494 ostree-sysroot-deploy.c: delete redundant check
Just noticed this while inspecting the code. The deployments retrieved
by `_ostree_sysroot_list_deployment_dirs_for_os` will forcibly already
have a matching osname since it indirectly uses that same variable to
construct them. Having a check there makes it look like there may be
subtle corner cases, when there aren't.

Closes: #529
Approved by: cgwalters
2016-10-14 16:06:08 +00:00
Jonathan Lebon
3943284dad docs: amend vmlinuz & initramfs naming convention
I was confused while reading the docs how this could work, since in at
least the Fedora/CentOS/RHEL distros, they're named e.g.
initramfs-`uname -r`-$checksum.

Closes: #529
Approved by: cgwalters
2016-10-14 16:06:08 +00:00
Alexander Larsson
ee484697cd pull: Support multiple specifications of --subpath
I need this in flatpak to avoid doing multiple pulls when doing
locale subsetting.

Closes: #523
Approved by: cgwalters
2016-10-11 16:41:03 +00:00
Alexander Larsson
0446cdb0d3 pull: Support inherit-transaction
If this is true, don't initiate, abort of commit a transaction, instead
it is assumed that the caller initiated the transaction, and that it
will eventually be commited.

This allows you to do multiple pulls or a combination of pulls and
commits in a single transaction.

Closes: #525
Approved by: cgwalters
2016-10-07 19:34:47 +00:00
Colin Walters
a0e1344cf8 Release 2016.11
Closes: #524
Approved by: cgwalters
v2016.11
2016-10-07 19:21:41 +00:00
Colin Walters
54621d9e53 libglnx: Update to latest
This pulls in a new compilation flag for wrpseudo compatibility.  Also
note we need to add some includes since glnx-libcontainer went away,
and with it some includes for `sys/mount.h` etc.

Closes: #522
Approved by: cgwalters
2016-10-06 11:54:39 +00:00
Colin Walters
524d2d5cb2 trivial-httpd: Port mostly to fd-relative
We were seeing some weird potential memory corruption in this code
when using it for `rpm-ostree-toolbox installer`, which is almost
certainly not its fault, but let's use it as an excuse to port
(mostly) to fd-relative and away from GFile.

Dropping the last GFile use here is a bit tricky as it does have a
nice high level wrapper around inotify.

Closes: #512
Approved by: jlebon
2016-10-05 21:48:41 +00:00
Colin Walters
a981e5fd76 checkout: Fix fsync defaults for new API to be off for real
My previous change in https://github.com/ostreedev/ostree/pull/425
actually broke things so we basically used the repository defaults =(

This is a subtle mess since we're only trying to flip things off
for the *new* API.

Clean this up so that the "default repo inheritance" lives only in one
place - in the compat layer for the old checkout API.  The new
checkout API defaults to off period, so the repository state is
irrelevant.

Closes: #520
Approved by: jlebon
2016-10-05 21:34:57 +00:00
Simon McVittie
2eae12220e Fix spelling of "repository"
Detected by Debian's Lintian tool.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #519
Approved by: cgwalters
2016-10-01 18:50:41 +00:00
Dan Nicholson
6c84fa4366 admin: Allow running status unlocked
It's useful to let non-root see the current system status.

Closes: #515

Closes: #516
Approved by: jlebon
2016-09-30 18:29:26 +00:00
Jonathan Lebon
056ca71a3b docs: add mention of rpm-ostree package layering
Closes: #514
Approved by: cgwalters
2016-09-20 18:45:44 +00:00
Jonathan Lebon
318430dc70 ostree_sysroot_init_osname: also create /var/log
/var/log is another one of those core directories that should be made
available and properly labeled during early boot before tmpfiles.d
starts up.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1265295

Closes: #513
Approved by: cgwalters
2016-09-20 18:03:57 +00:00
Colin Walters
450361d89b boot: Ensure we remount /var writable before systemd does journal flush
Otherwise, we may not get a persistent journal for the first boot.
https://bugzilla.redhat.com/show_bug.cgi?id=1265295

Closes: #511
Approved by: jlebon
2016-09-15 19:41:44 +00:00
Colin Walters
f2b6afd2df sysroot: Drop an unnecessary fsync
While looking at a slow update issue (which I'm guessing is
unpredictable I/O latency in an OpenStack instance), I noticed
in one of the traces we were inside a fsync here.

Dropping the fsync here is just another of a long series of unwinding
them - we `syncfs()` the sysroot fd and `/boot` and we have a big
`sync()` anyways.

Closes: #508
Approved by: jlebon
2016-09-14 19:14:46 +00:00
Colin Walters
4f736ac33e sysroot: Drop an fsync for origin file when writing deployments
More fsync pruning.  Since we have a public API for writing the origin
file and it did a fsync before, let's preserve that.  But when writing
deployments as part of a full transaction, we rely on the global
`syncfs()`, so add an internal function for origin file writing that
doesn't.

Closes: #509
Approved by: giuseppe
2016-09-14 10:26:39 +00:00
Colin Walters
fcffb73280 sysroot: Port origin writing code to fd-relative
Just preparatory cleanup for a next patch which makes
the fsyncing configurable.

Closes: #509
Approved by: giuseppe
2016-09-14 10:26:39 +00:00
Colin Walters
84a9d61e15 sysroot: Port some small cleanup code to fd-relative
Just a quick patch since I saw this function scroll by in Emacs and it
was too ugly not to be rewritten.

Closes: #510
Approved by: giuseppe
2016-09-14 10:13:12 +00:00
Owen W. Taylor
a8301b909c ostree_sysroot.c: Don't close sysroot_fd twice.
If ostree_sysroot_unload() was called explicitly, then sysroot_fd
would be closed again at finalization time, possibly closing a
random file descriptor belonging to some other part of the
application.

Closes: #507
Approved by: cgwalters
2016-09-13 13:19:10 +00:00
Owen W. Taylor
a5af1cb688 ostree-repo.c: Fix file descriptor cleanup
0 was used as an "unset" flag for tmp_dir_fd, which is technically
incorrect. For cache_dir_fd, -1 was used as the sentinal but 0
was checked for, resulting in close(-1).

Closes: #507
Approved by: cgwalters
2016-09-13 13:19:10 +00:00
Colin Walters
5893b68ef7 pull: Do allow executing deltas when mirroring into bare{,-user}
In https://github.com/ostreedev/ostree/pull/408 we fixed a
bug where we would crash when trying to execute deltas into
an archive repo (which isn't presently supported).

But that was overly aggressive - we obviously *can* execute deltas
when mirroring into a bare repo.  This should fix a regression with
the way flatpak uses mirroring to pull from a user repo into the
system.

Closes: #506
Approved by: alexlarsson
2016-09-12 12:50:34 +00:00
Jonathan Lebon
12e916466c static-delta: add some error handling
We make _ostree_parse_delta_name() a bit more defensive since it handles
user input.

Closes: #504

Closes: #505
Approved by: cgwalters
2016-09-09 19:06:11 +00:00
Colin Walters
36e8ba124e Release 2016.10
Closes: #502
Approved by: cgwalters
v2016.10
2016-09-09 00:28:02 +00:00
Colin Walters
916684ba0d core: Make OSTREE_TIMESTAMP public API
This way e.g. flatpak can detect which timestamp it should use.
See `flatpak/common/flatpak-utils.c:flatpak_zero_mtime`.

Closes: #501
Approved by: cgwalters
2016-09-09 00:10:21 +00:00
Colin Walters
30aaffa995 sysroot: Avoid double cleanup, and ensure no cleanup if specified
Since forever, we've been doing two cleanups.  In
8ece4d6d51
I thought we were doing just one and wanted to go to zero (if specified),
but I actually just dropped one cleanup.

In https://github.com/projectatomic/rpm-ostree/pull/452
@jlebon pointed out the duplication.  Fix this by creating a new internal
deploy wrapper that takes cleanup flags.

(Since we already had the "piecemeal cleanup" API internally, let's
 frame it in terms of that, rather than passing down a boolean).

Closes: #500
Approved by: jlebon
2016-09-08 20:42:09 +00:00
Colin Walters
82b756783b fetcher: Fix another finalization deadlock
If the current repo is already up to date (we have no content to
fetch), it's possible for the fetcher to not request any URIs.  So
create and then finalize it quickly.

Finalization involves calling `g_main_loop_quit()` +
`g_thread_wait()`.  However, if `g_main_loop_quit()` is run *before*
`g_main_loop_run()`, we'll deadlock because GMainLoop assumes in
`_run()` to start things.

This is a common trap - ideally, GMainLoop would record if `_quit()`
was called before `_run()` or something, but doing that now would
likely break people who are expecting quit() -> run() to restart.

In general, we've moved in various GLib-consuming apps to an
explicit "main context iteration with termination condition" model;
see `pull_termination_condition()` in the pull code.

This fixes this race condition.

I verified that an assertion in `_finalize` that more than
zero URIs were requested was hit in multiple test cases, and this patch
has survived a while of make check loops.

Closes: https://github.com/ostreedev/ostree/issues/496

Closes: #499
Approved by: jlebon
2016-09-08 20:09:07 +00:00
Colin Walters
23b4b58d57 delta: Unreference files we've processed
This should help avoid address space exhaustion on 32 bit systems,
and in general is obviously going to improve efficiency.

Closes: #498
Approved by: alexlarsson
2016-09-08 15:21:41 +00:00
Colin Walters
e127070550 repo: Only use mmap() for metadata > 16k
See http://stackoverflow.com/questions/258091/when-should-i-use-mmap-for-file-access
and
https://lwn.net/Articles/591978/

I didn't really notice much performance difference in some small
tests, but I happened to be stracing and realized we were `mmap()`ing
even for 50 bytes which is not very useful, so let's not do it.

Closes: #489
Approved by: alexlarsson
2016-09-08 14:56:30 +00:00
Colin Walters
8dbb104cdc delta: Add missing goto out for failure to mmap()
This was hit in practice when generating a delta for a flatpak app on ARM
it looks like.

Closes: #497
Approved by: alexlarsson
2016-09-08 14:34:04 +00:00
Colin Walters
845dc65196 repo: Revert default timestamp from 1 back to 0
Quoting Dan Nicholson in

  <https://github.com/ostreedev/ostree/pull/330#issuecomment-245499099>

  mtime of 0 has been the semantics of ostree deployments from basically
  the beginning of the project. We (and others, see
  flatpak/flatpak@b5204c9) rely on that fact when generating trees.

  In particular, this affects caches that use the mtime of the
  associated file or directory to determine if the cache is valid. By
  arbitrarily changing the mtime of the files to something else, all
  the caches we setup in the build are now invalidated. Preseeding
  caches is really important to the user experience as it avoids
  having the user wait while they're regenerated on first run.

  Now, we could change our build infrastructure to preset all the
  mtimes to 1 to match this change, but what does that do for our
  existing users who are on an ostree that deploys with mtimes of 0?
  We could just revert this change at Endless (and the associated one
  in Flatpak), and that would be fine for our users. However, if we
  point non-Endless users to our apps, they'll have the great
  experience of waiting 10 seconds the first time they launch it while
  the fontconfig cache is rebuilt unnecessarily.

Closes: #495
Approved by: jlebon
2016-09-08 13:35:59 +00:00
Giuseppe Scrivano
f760a4612a gpg: do not segfault when the algorithm name is not known
Reported by: Patrick Uiterwijk
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #494
Approved by: cgwalters
2016-09-08 11:19:20 +00:00
William Manley
2aacc6912b ostree-prepare-root: Fix running with musl
musl libc's implementation of `realpath` works by opening the path and then
doing a lookup in `/proc/self/fd` to find the canonical path.  This fails
if `/proc` is not mounted.  This causes problems for us if
`ostree-prepare-root` is `init` as `/proc` won't be mounted.

We have to mount `/proc` anyway for `/proc/cmdline` so this fix just
expands the scope over which `/proc` is mounted to include both our
`realpath` calls.

See also:

* http://www.openwall.com/lists/musl/2016/06/08/2 and
* http://git.musl-libc.org/cgit/musl/tree/src/misc/realpath.c?id=e738b8cbe64b6dd3ed9f47b6d4cd7eb2c422b38d

Closes: #485
Approved by: cgwalters
2016-09-01 20:17:58 +00:00
William Manley
5424404813 ostree-prepare-root: Error if realpath fails
I've seen it fail with musl which needs `/proc` to be mounted for it to
work.  The error messages we're rather confusing before.  At least this
now points to the right location.

Closes: #485
Approved by: cgwalters
2016-09-01 20:17:58 +00:00
William Manley
598e3240ff switchroot: Fix test-switchroot now autotools can build static
This test previously depended on manually building ostree-prepare-root.
Since 42dab85 we've been able to build static binaries with the usual
autotools build-system.  This change reflects the fact that
`ostree-prepare-root` is built into $srcdir rather than `src/switchroot`
where I was building manually.

This test now passes with `./configure --with-static-compiler=gcc` (glibc)
but still fails with `./configure --with-static-compiler=musl-gcc` (musl).

Closes: #485
Approved by: cgwalters
2016-09-01 20:17:58 +00:00
William Manley
4d3b93e2ad switchroot: Fix build on Ubuntu
Was failing with error:

    src/switchroot/ostree-prepare-root.c:30:20: fatal error: config.h: No such file or directory
    compilation terminated.

Reported by and fix provided by @gatispaeglis.

Closes: #485
Approved by: cgwalters
2016-09-01 20:17:58 +00:00
Dan Nicholson
39918179d1 build: Actually distribute man page XML source
The make substitution pattern was wrong. The source files are
"ostree.xml", not "ostree.1.xml", for instance.

Closes: #488
Approved by: cgwalters
2016-09-01 19:23:49 +00:00
Dan Nicholson
7adc943596 build: Distribute man page XML source
Without this, the manual pages can't actually be regenerated from a dist
tarball, and running make clean will remove all traces of them.

Closes: #486
Approved by: cgwalters
2016-09-01 19:03:23 +00:00
Dan Nicholson
973ec467ce build: Set --enable-man during distcheck
If xsltproc is not installed, then ENABLE_MAN will be false and the
generated man pages won't be distributed. Pass --enable-man to enforce
that the man pages will be generated and distributed.

Closes: #486
Approved by: cgwalters
2016-09-01 19:03:23 +00:00
Jonathan Lebon
3b55db9661 pull code: clean up mirrorlist hack
While converting the mirrorlist code from using GSList to GPtrArray, I
completely missed the fact that there is now a much cleaner way to do
this.

Closes: #484
Approved by: cgwalters
2016-08-31 18:54:59 +00:00
Jonathan Lebon
661e4636f5 tests: add tests for contenturl and mirrorlist
Closes: #469
Approved by: cgwalters
2016-08-31 16:52:12 +00:00
Jonathan Lebon
c4c030cc79 libtest: add has_gpgme() helper function
Closes: #469
Approved by: cgwalters
2016-08-31 16:52:12 +00:00
Jonathan Lebon
157d878ce1 pull: add mirrorlist support
This commit adds mirrorlist support to the fetcher. Users can now
prepend url or/and contenturl by mirrorlist= to interpret the link as a
mirrorlist.

If an object is not found, the fetcher will automatically try the next
mirror in the order given in the list (assuming the order returned by
the server is significant).

Closes: #469
Approved by: cgwalters
2016-08-31 16:52:12 +00:00