Commit Graph

462 Commits

Author SHA1 Message Date
Alexander Larsson
bd45e7ac19 commit: Fix reading xattrs from OstreeRepoFile:s
When doing commit --tree=ref=XXX while at the same time applying some
form of modifier, ostree dies trying to read the xattrs using the
raw syscalls. We fix this by falling back to ostree_repo_file_get_xattrs()
in this case.

Also adds a testcase for this.

Closes: #577
Approved by: cgwalters
2016-11-16 22:30:33 +00:00
Colin Walters
37c07d2f1c pull: Add support for http-headers option
Some deployments may want to gate access to content based on things
like OAuth.  In this model, the client system would normally compute a
token and pass it to the server via an API.

We could theoretically support this in the remote config too, but
that'd be a bit weird for OAuth as the information is dynamic.
Therefore this cleans up the code a little bit to more clearly handle
the case that the fetcher is initialized from both remote config
data plus pull options.

Closes: #574
Approved by: giuseppe
2016-11-16 10:04:22 +00:00
Sjoerd Simons
6af8db6fc4 tests: Add test for the cookie jar handling
Closes: #531
Approved by: cgwalters
2016-11-05 17:34:09 +00:00
Colin Walters
2139f0e437 pull: Don't do deltas with --commit-metadata-only
We should just download the commit objects directly, as it's
obviously a lot more efficient than deltas.

I had to generate a summary file in more places in the tests,
since once created, it needs to be updated.

Closes: https://github.com/ostreedev/ostree/issues/528

Closes: #566
Approved by: jlebon
2016-11-04 16:04:23 +00:00
Sjoerd Simons
f0e493bf29 Filter bootloader supplied kernel cmdline options
Various bootloader add kernel commandline options dynamically, filter
these out when grabbing boot options from /proc/cmdline. Specifically
grub adds BOOT_IMAGE and systemd-boot adds initrd.

Closes: #560
Approved by: cgwalters
2016-11-01 18:00:37 +00:00
Simon McVittie
7091d288f7 Force C.UTF-8 or C locale for tests
Otherwise several tests fail, for example in this build done in a French
locale by Debian's reproducible builds initiative, to check whether
the resulting binaries are identical to what was produced in an
English locale:

<https://tests.reproducible-builds.org/debian/logs/unstable/amd64/ostree_2016.11-1.build2.log.gz>

(test-basic)
# error: Cannot write to repository: Permission non accordée
...
File 'error-message' doesn't match regexp 'Permission denied'

(test-help)
# Utilisation :
#   ostree [OPTION...] COMMAND
...
File 'out' doesn't match regexp '[Uu]sage'

(test-pull-metalink)
# error: Erreur à la ligne 1, caractère 1 : Le document doit commencer avec un élément (par ex. <book>)
...
File 'err.txt' doesn't match regexp 'Document must begin with an element'

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #558
Approved by: cgwalters
2016-10-29 18:12:42 +00:00
Colin Walters
b77edf24a3 tree-wide: Remove unused variables detected by CLang
CLang finds these, whereas GCC treats having
`__attribute__((cleanup))` as a use.

This obsoletes https://github.com/ostreedev/ostree/pull/411

Closes: #548
Approved by: jlebon
2016-10-27 17:02:01 +00:00
Colin Walters
82a4f56593 tests: Skip libarchive/selinux tests if in container without SELinux
I'm doing builds and `make check` inside a Docker container, with
selinux on as a build-time option, but no policy in the container.
This currently aborts.  Let's not do that.

(This type of thing is why installed tests are a better model)

Closes: #546
Approved by: jlebon
2016-10-27 16:48:34 +00:00
Alexander Larsson
d0e74cf3af Fix pruning of partial commits
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.

In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.

Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.

This fixes https://github.com/ostreedev/ostree/issues/541

Closes: #542
Approved by: cgwalters
2016-10-24 17:48:19 +00:00
Alexander Larsson
ee484697cd pull: Support multiple specifications of --subpath
I need this in flatpak to avoid doing multiple pulls when doing
locale subsetting.

Closes: #523
Approved by: cgwalters
2016-10-11 16:41:03 +00:00
Jonathan Lebon
12e916466c static-delta: add some error handling
We make _ostree_parse_delta_name() a bit more defensive since it handles
user input.

Closes: #504

Closes: #505
Approved by: cgwalters
2016-09-09 19:06:11 +00:00
Colin Walters
845dc65196 repo: Revert default timestamp from 1 back to 0
Quoting Dan Nicholson in

  <https://github.com/ostreedev/ostree/pull/330#issuecomment-245499099>

  mtime of 0 has been the semantics of ostree deployments from basically
  the beginning of the project. We (and others, see
  flatpak/flatpak@b5204c9) rely on that fact when generating trees.

  In particular, this affects caches that use the mtime of the
  associated file or directory to determine if the cache is valid. By
  arbitrarily changing the mtime of the files to something else, all
  the caches we setup in the build are now invalidated. Preseeding
  caches is really important to the user experience as it avoids
  having the user wait while they're regenerated on first run.

  Now, we could change our build infrastructure to preset all the
  mtimes to 1 to match this change, but what does that do for our
  existing users who are on an ostree that deploys with mtimes of 0?
  We could just revert this change at Endless (and the associated one
  in Flatpak), and that would be fine for our users. However, if we
  point non-Endless users to our apps, they'll have the great
  experience of waiting 10 seconds the first time they launch it while
  the fontconfig cache is rebuilt unnecessarily.

Closes: #495
Approved by: jlebon
2016-09-08 13:35:59 +00:00
William Manley
2aacc6912b ostree-prepare-root: Fix running with musl
musl libc's implementation of `realpath` works by opening the path and then
doing a lookup in `/proc/self/fd` to find the canonical path.  This fails
if `/proc` is not mounted.  This causes problems for us if
`ostree-prepare-root` is `init` as `/proc` won't be mounted.

We have to mount `/proc` anyway for `/proc/cmdline` so this fix just
expands the scope over which `/proc` is mounted to include both our
`realpath` calls.

See also:

* http://www.openwall.com/lists/musl/2016/06/08/2 and
* http://git.musl-libc.org/cgit/musl/tree/src/misc/realpath.c?id=e738b8cbe64b6dd3ed9f47b6d4cd7eb2c422b38d

Closes: #485
Approved by: cgwalters
2016-09-01 20:17:58 +00:00
William Manley
598e3240ff switchroot: Fix test-switchroot now autotools can build static
This test previously depended on manually building ostree-prepare-root.
Since 42dab85 we've been able to build static binaries with the usual
autotools build-system.  This change reflects the fact that
`ostree-prepare-root` is built into $srcdir rather than `src/switchroot`
where I was building manually.

This test now passes with `./configure --with-static-compiler=gcc` (glibc)
but still fails with `./configure --with-static-compiler=musl-gcc` (musl).

Closes: #485
Approved by: cgwalters
2016-09-01 20:17:58 +00:00
Jonathan Lebon
661e4636f5 tests: add tests for contenturl and mirrorlist
Closes: #469
Approved by: cgwalters
2016-08-31 16:52:12 +00:00
Jonathan Lebon
c4c030cc79 libtest: add has_gpgme() helper function
Closes: #469
Approved by: cgwalters
2016-08-31 16:52:12 +00:00
Gatis Paeglis
b6ec7526b5 u-boot: Merge ostree's and systems uEnv.txt
This is a proper fix for:
https://bugzilla.gnome.org/show_bug.cgi?id=755787

With this patch, an admin (system builder) can now:

1) Edit /usr/lib/ostree-boot/uEnv.txt
2) Deploy the new tree. OSTree will append system's uEnv.txt
   to the OSTree's managed uEnv.txt (loader/uEnv.txt).

It is common for u-boot systems to read in an extra env
from external /uEnv.txt. The same file OSTree uses to pass
in its env. With this patch /uEnv.txt now contains OSTree's
env + custom env added by system builders.

Closes: #466
Approved by: cgwalters
2016-08-27 13:11:22 +00:00
Colin Walters
3ef4cc2e5b lib: Add an API to list only "our" objects, fix prune to use it
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail.  There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.

Closes: https://github.com/ostreedev/ostree/issues/467

Closes: #471
Approved by: jlebon
2016-08-25 20:01:37 +00:00
Colin Walters
3469161e41 test-rofiles-fuse: Actually check out via hardlinks
The test suite was actually doing something before, just
not quite what I intended.  Without `-U` for bare-user checkouts
we end up doing a copy.

Now, a future commit will change how rofiles work, which would cause
the test suite to permit inplace mutation for non-hardlinked files.
So let's ensure they *are* hardlinked.

Closes: #462
Approved by: jlebon
2016-08-16 21:22:28 +00:00
Dan Nicholson
7ed2f1e5af tests: Test that local pulls do not use deltas
Pulls from local repos now default to disabling static deltas so that
objects are copied. To check this is the case, see if the object files
are hardlinked after pulling.

Closes: #447

Closes: #448
Approved by: cgwalters
2016-08-10 10:54:46 +00:00
Dan Nicholson
2a810fbef9 tests: Ensure deltas for pulling when needed
When testing pulling of deltas, use the new --require-static-deltas
option to pull-local to ensure that deltas are actually used. To support
the require-static-deltas mode, the summary in the remote repo must be
generated.

Closes: #447

Closes: #448
Approved by: cgwalters
2016-08-10 10:54:46 +00:00
Colin Walters
56a922280f tests: Port sysroot.js away from libgsystem
My case-sensitive `git grep` didn't find this one originally,
but the Travis+Debian setup was configured to run it.

Closes: #451
Approved by: jlebon
2016-08-09 19:01:35 +00:00
Colin Walters
a29bb0ab6b Final excision of libgsystem dependency
Lots and lots of preparation led to this moment - when nothing
apparent changes for users!  Woo!

But seriously, having the extra dependency is a minor annoyance, and
in the big picture I think the libgsystem idea was wrong - we need to
land things in GLib, and use git submodules for API-unstable or
Linux-specific sharing.  For a lot of OSTree, the libgsystem `GFile*`
orientation was also wrong, we really want fd-relative.

Closes: #444
Approved by: jlebon
2016-08-09 15:36:09 +00:00
Colin Walters
fd9536bb19 prune: Retain the tip of each ref even with date pruning
I hit an error with [CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel)
where we were doing time-based pruning.  `ostree summary -u` started failing,
and it took me a bit to realize it was because we were pruning
even the tip of old branches, which I was not at all expecting,
and I don't think users will too.

Perhaps in the future we could add some sort of --prune-ref-tips or
something if people wanted it, but I doubt it.

Closes: #445
Approved by: jlebon
2016-08-09 15:12:35 +00:00
Simon McVittie
46e118ee84 travis: run the test suite on various distributions
.travis.yml is obviously still Travis-specific, but tests/ci-* are
designed to be shareable with other CI environments if there is interest
in doing so.

At the moment I'm only testing on Debian and Ubuntu. In principle we
could try a non-Debian-derived Docker container such as Fedora or CentOS
inside travis-ci's Ubuntu environment, similar to what I'm doing
for Debian, but I don't know the correct setup commands to use there.

Closes: #438
Approved by: cgwalters
2016-08-05 23:58:19 +00:00
Simon McVittie
0de51bfc45 Skip tests that use whiteouts under Docker/aufs
ostree's naming convention for whiteouts is similar to what is
done in aufs, which means we can't compose the trees to test this
feature under Docker with the aufs storage driver, as used on
travis-ci.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #437
Approved by: cgwalters
2016-08-05 20:32:57 +00:00
William Manley
52acb9d8e3 ostree-prepare-root: Add test that overlayfs over /usr works.
Closes: #403
Approved by: cgwalters
2016-08-02 19:07:25 +00:00
William Manley
485a374b21 ostree-prepare-root: Use pivot_root if real sysroot is already mounted at /
This allows ostree-prepare-root outside of the initramfs context where the
real rootfs is already mounted at /.  We can't use `mount --move` in this
case because we would be trying to move / into a subdirectory of itself.

Closes: #403
Approved by: cgwalters
2016-08-02 19:07:25 +00:00
William Manley
94e50ca5aa tests: Add basic tests for ostree-prepare-root
These tests use unshare and mount to prepare a fake initrd/early boot
directory structure so we can then test ostree-prepare-root.

Things that are tested:

* Running in an initrd environment
* Running without initrd
* /var and /sysroot being mounted correctly
* /usr being mounted read-only

Things not tested (yet):

* Running as init - this could be accomplished by unsharing the pid
  namespace too.
* mounting/unmounting `/proc` if `/proc/cmdline` isn't available
* Persistent overlayfs for `/usr`
* Probably other things

The tests are basic but can be extended in the future as we do more work
on `ostree-prepare-root`.

These tests must be run as root as they require the ability to `mount`
and to `unshare` the mount namespace. Perhaps in the future we can use
user namespaces for this test once they are more widely available.

Closes: #403
Approved by: cgwalters
2016-08-02 19:07:25 +00:00
Colin Walters
5334758ba7 repo: Make ostree_repo_create() nonfatal on existing repos
In general we want to support "idempotentcy" or "state
synchronization" across interruption.  If a repo is only partially
created due to a crash or whatever, it's hard for a user to know that.
Let's just make `ostree_repo_create()` idempotent. Since all we're
doing is a set of `mkdirat()` invocations, it's quite simple.

This also involved porting to fd-relative, which IMO makes the
code a lot clearer.

Closes: #422
Approved by: 14rcole
2016-08-01 15:12:14 +00:00
Dan Nicholson
a312d8fd8c tests: Add test for delta with empty parts
Closes: #420
Approved by: cgwalters
2016-07-31 00:01:52 +00:00
Colin Walters
402020b4ca glib.supp: Suppress g_task -> thread leaks
There are variants of this, but right now because `GTask` has its
own pool which isn't necessarily cleaned up on exit, we need
to suppress all of this.

Closes: #410
Approved by: giuseppe
2016-07-28 10:10:17 +00:00
Colin Walters
544365f834 glib.supp: Suppress worker context -> thread bits
The worker context isn't cleaned up now.

Closes: #410
Approved by: giuseppe
2016-07-28 10:10:17 +00:00
Colin Walters
d13cf37b3f glib.supp: Suppress some dynamic type registrations too
These mirror the static ones, and we need the dynamic versions for
glib-networking -> gnutls stuff.

Closes: #410
Approved by: giuseppe
2016-07-28 10:10:17 +00:00
Colin Walters
bac1fa1839 pull: Don't execute static deltas when mirroring
We don't presently support this, since the static delta code assumes
it can just `mmap()` file objects.  We could at some point implement
this, but for now just skip executing deltas when doing
`archive -> archive` mirroring.

I noticed this when trying to mirror a repo in Jenkins in
[CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel).

Closes: #408
Approved by: giuseppe
2016-07-25 06:50:51 +00:00
Colin Walters
87c9e227cb Revert "tests/libtest.sh: Print non-matching file on failure"
This reverts commit 71301d1824.

I was confused by the fact that the non-matching content was
empty and forgot we already did print it.

Reported-by: smcv

Closes: #390
Approved by: jlebon
2016-07-11 19:30:31 +00:00
Colin Walters
35b4131bd7 tests: Fix karg tests on ostree-booted system
https://github.com/ostreedev/ostree/pull/372 caused these tests to
start failing when the host system is managed using ostree - since the
tests *do* replace the `ostree=` kernel argument.

Closes: #384
Approved by: cgwalters
2016-07-11 16:46:07 +00:00
Colin Walters
d7629d33a4 tests: Add some test coverage of repeated pulls w/HTTP 500s
Systems like pulp may want to keep retrying in a loop if the server
throws a (hopefully transient) 500, and we need test coverage of
handling these errors versus our existing 404 and 206 coverage.

Closes: #383
Approved by: mbarnes
2016-07-08 18:38:11 +00:00
Colin Walters
71301d1824 tests/libtest.sh: Print non-matching file on failure
We clean up the temporary directory on failure, which means it's hard
to know *why* a regex didn't match.  Print it when we hit an error.

Closes: #383
Approved by: mbarnes
2016-07-08 18:38:11 +00:00
Colin Walters
0d07c7ecde delta: Add --if-not-exists option
I often want to have "idempotent" systems that iterate to a known
state.  If after generating a commit, the system is interrupted, I'd
like the next run to still generate a delta.  But we don't want to
regenerate if one exists, hence this option.

Closes: #375
Approved by: jlebon
2016-07-04 13:27:06 +00:00
Dan Nicholson
a94530111a tests: Improve check for /proc/cmdline kargs
On some systems there may be no root= argument, so the tests for
appending /proc/cmdline arguments will fail. Instead, loop over each of
the arguments in the host's /proc/cmdline and test that they're in the
constructed config file. That will actually test if ostree added all of
the system's /proc/cmdline args correctly. The regex isn't perfect here,
but it's probably good enough for this test.

Closes: #372
Approved by: cgwalters
2016-06-27 20:20:21 +00:00
Dan Nicholson
99a76c9b34 tests: Remove extra $CMD_PREFIX from test-auto-summary.sh
$OSTREE already has $CMD_PREFIX in it, so adding it again causes you to
call env twice with LD_PRELOAD.

Closes: #372
Approved by: cgwalters
2016-06-27 20:20:21 +00:00
Dan Nicholson
8b397301c4 tests: Ensure mutable deployments from libostreetest
When creating sysroots with libostreetest, we don't get the benefit of
the OSTREE_SYSROOT_DEBUG setting in libtest.sh. That means we'll get
immutable deployments that can't be easily cleaned up.

Ensure the environment variable is set before creating new sysroots. It
would be nice to set the debug flags directly, but that's private API
that isn't currently pulled into libostreetest.

Closes: #372
Approved by: cgwalters
2016-06-27 20:20:21 +00:00
Colin Walters
439069b2bb checkout: Add an option to require hardlinks
I've seen a few people hit this and wonder why checkouts are slow/take
space.  Really, ensuring this happens is the *point* of OSTree.
Physical copies should be a last resort fallback for very unusual
situations (one of those is rpm-ostree checking out the db since
librpm doesn't know how to read from libostree).

Even I hit the fact that `/var` is a mountpoint disallowing hardlinks
with `/ostree` once and was confused.  =)

Add this to the rofiles-fuse test case because it creates a mount
point.

Closes: #368
Approved by: jlebon
2016-06-27 13:08:46 +00:00
Simon McVittie
f38c33fec8 entry_pathname_test_helper: these tests need extended attributes
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #366
Approved by: cgwalters
2016-06-26 21:15:36 +00:00
Alexander Larsson
3640725439 tests: Test partial commits for local remotes
This was broken before, fixed in the previous commit.

Closes: #324
Approved by: yuqi-zhang
2016-06-24 15:21:38 +00:00
Yu Qi Zhang
02a2b689dd refs: resolve conflict between local/remote repos
Add the functionality to use the same name for refs in local and remote
repos. This helps users keep track of local refs of remote origin, much
like local and remote git branches.

Previously, when a local ref is specified, resolve_refspec would fall
back to searching through remote repos if the ref is not found locally.
This function now takes an extra flag to specify whether it should
search through remote repos. Additionally, ostree_repo_resove_rev_ext
was added to call resolve_refspec with fallback_remote being false, so
refs --create would no longer complain when trying to create a local
ref of the same name as a remote one.

Fix remote repo parsing not being handled correctly on refs --create.

Closes: #363
Approved by: jlebon
2016-06-23 19:52:26 +00:00
Yu Qi Zhang
744543110e refs: allow overwrite of empty folders
We noticed that once a ref folder is created, there is no existing
command that can remove it. For example, once "foo/bar" is created,
even if the user deletes foo or all the refs under foo, the folder
will persist.

Now when the user attempts to create a ref "foo" either through commit
or refs --create, if a folder "foo" exists but is empty of refs, the
folder is removed and the new ref "foo" is created.

New unit tests in tests-ref.sh verify this functionality.

Closes: #354
Approved by: cgwalters
2016-06-21 20:20:15 +00:00
Colin Walters
d262fc2e0f tests: Support OT_SKIP_READDIR_RAND
For some reason I'm really not inclined to debug right now, `libtool`
is eating the `-fsanitize=address` argument when linking
`libreaddir-rand.so`, which causes failures since we're trying to
`LD_PRELOAD`.

Closes: #351
Approved by: jlebon
2016-06-17 14:43:44 +00:00
Colin Walters
71e768d230 build: Fix libreaddir-rand to honor global CFLAGS
It's unfortunate that in automake one has to explicitly include the globa
`$(AM_CFLAGS)` if one sets `CFLAGS`, and similarly for other variables.

I'm trying to use `-fsanitize=address`, and not including it was
causing linker failures.

We also weren't inheriting the global warnings etc., so I had to fix a
decl-after-statement.

Closes: #351
Approved by: jlebon
2016-06-17 14:43:44 +00:00