Commit Graph

40 Commits

Author SHA1 Message Date
Colin Walters
618617d68b lib/pull: Add support for timestamp-check option, use in upgrader
For both flatpak and ostree-as-host, we really want to verify up front during
pulls that we're not being downgraded. Currently both flatpak and
`OstreeSysrootUpgrader` do this before deployments, but at that point we've
already downloaded all the data, which is annoying.

Closes: https://github.com/ostreedev/ostree/issues/687

Closes: #1055
Approved by: jlebon
2017-08-21 18:24:07 +00:00
Colin Walters
48f8f32921 bin/pull: Fix @ override syntax when pulling multiple refs
Coverity spotted an infloop here since we were incrementing `i++`
instead of `j++`.  But adding a test revealed other bugs - we need
to keep the arrays in sync.

Coverity CID: 1452204

Closes: #1041
Approved by: pwithnall
2017-08-07 16:10:08 +00:00
Colin Walters
c740b7f6d2 core: Sanitize error text validating refs (e.g. against HTML)
See: https://github.com/projectatomic/rpm-ostree/issues/885

If we get a successful Apache directory listing HTML when fetching what we
intend to be a ref, we'd dump the HTML into the error.

I did some scanning of the pull code, and this was the only case
I saw offhand where we were dumping text out into an error.  Which
makes sense, since most of our formats are binary, the exeptions I
think are just `repo/config` and `repo/refs/`.

Closes: #1015
Approved by: mbarnes
2017-07-19 14:45:57 +00:00
Colin Walters
0aa20df20e tests: Run pull tests for bare/bare-user
We have variants of `test-basic` for all 4 modes, but not for pull-test, which
for some reason was named `pull-archive`, but mostly pulls *into* bare repos.
The test code was structured like the basic one where it called into a
`pull-test.sh`, so let's actually use it for 2/3 bare modes. (I tried to extend
it to `bare-user-only` but it failed, going to look at that after this).

This is related to https://github.com/ostreedev/ostree/issues/991

Closes: #998
Approved by: jlebon
2017-07-10 14:18:18 +00:00
Colin Walters
3234295324 pull: Cleanly error when doing local pulls of remote-prefixed refs
In the storage PR I was trying to do a `pull-local` of the whole
`/ostree/repo` on the system, which ended up triggering a `g_critical()`
in the collections code, since we tried to parse a remote-prefixed ref
`fedora:fedora/26/x86_64/atomic-host` as a ref.

I'm not sure offhand what our behavior in this case *should* be.  I
think git only clones local refs, but I need to check.

This corner case arises only with `pull-local`.  But in any case,
while we were previously saying this is programmer error, since it's
so easy to pass various unchecked input into the pull machinery,
make invalid refs an explicit error.

Closes: #992
Approved by: jlebon
2017-07-05 13:37:22 +00:00
Colin Walters
1a9a473580 cmdline/pull: Print final status even if noninteractive
Previously, `ostree pull` was silent if not on a tty.  I don't
see a reason not to print the final status line at least.  This
is prep for more work in the test suite, so I can write assertions
on the output.

But it should also be nicer for people who e.g. do an `ostree pull` in a Jenkins
job or whatever.

Closes: #981
Approved by: jlebon
2017-06-29 16:16:25 +00:00
Colin Walters
90e0d56332 tree-wide: Replace various uses of archive-z2archive
The `-z2` is annoying now since it's really a legacy; we've long
since supported typing `archive`.  Convert the docs fully and
explain that.

Also do some (but not all) of the tests just to encourage newer tests to use
`archive` too.

Closes: #980
Approved by: jlebon
2017-06-29 16:00:13 +00:00
Alexander Larsson
73ba3eb686 pull: When mirroring, only replace summary if we're doing a full mirror
We're hitting this in flathub, where we have a bunch of local builds,
but we also mirror a few refs from the gnome runtime repo into it.
Its fixable by re-doing the summary, but for a short time the
wrong version is visible.

Fixes https://github.com/ostreedev/ostree/issues/846

Closes: #935
Approved by: cgwalters
2017-06-16 15:08:39 +00:00
Colin Walters
9529e8d435 lib/pull: Extend BAREUSERONLY_FILES flag to HTTP requests
For the flatpak PR: https://github.com/flatpak/flatpak/pull/849

It's really more convenient if this works for HTTP pulls as well, since flatpak
does various types of pulling, and we can just set the flag everywhere.

Further, we might as well reject the content as early as possible.

Closes: #930
Approved by: alexlarsson
2017-06-15 07:10:22 +00:00
Colin Walters
6ed824bf00 lib/pull: Add OSTREE_REPO_PULL_FLAGS_BAREUSERONLY_FILES
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845

We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909

This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.

Closes: #926
Approved by: alexlarsson
2017-06-13 18:44:28 +00:00
Colin Walters
ad119aece9 pull-test: Add some 404 tests
See: https://github.com/flatpak/flatpak/issues/816

Closes: #887
Approved by: jlebon
2017-05-26 14:33:59 +00:00
Jonathan Lebon
a8fd37b6a0 pull: complete detached meta fetch before scanning
If somehow a repo has gpg verification on but doesn't have signatures
present for the existing commit, ostree would error out if it needs to
scan the commit object (e.g. if there are no updates available).

An instance of this is currently happening in Fedora AH, in which
signatures are not shipped in the ISO due to filesystem restrictions.
Another possible scenario is if a content provider switches from not
signing commits to signing them; even if older commits are retroactively
signed, clients' local commit objects would error out if they needed
scanning.

This patch adds a check to ensure that we always attempt to fetch the
detached metadata and wait for its result (whether it exists or not)
before moving on to scan their corresponding commit objects.

See also: https://github.com/projectatomic/rpm-ostree/issues/630

Closes: #873
Approved by: cgwalters
2017-05-18 01:14:15 +00:00
Colin Walters
b9df96db8b pull: Support deltas for explicit commits
I think the majority of OSTree usage calls pull with refs, not
explicit commits.  We even added special "override syntax" with
`@` (e.g. `ostree pull foo@ab12c34`) as a hybrid.

However, some users may want to still pull explicit commits
for whatever reason.  The old static delta logic looked at
the previous commit of the ref.  However, in https://github.com/ostreedev/ostree/pull/710
we enhanced the logic to look at all local commits.

It's now a lot more natural to teach the delta logic
to support revisions, e.g. `ostree pull someorigin ab12c34`.

This also fixes the problem that before, `--require-static-deltas`
was completely ignored when processing revisions.

This is a nontrivial refactoring of the logic, but the end
result feels a lot more readable to me.

Closes: https://github.com/ostreedev/ostree/issues/783

Closes: #787
Approved by: cgwalters
2017-04-12 21:30:33 +00:00
Colin Walters
455cc5e892 repo+tests: Add [core]disable-xattrs=true, use it on overlayfs
There are a lot of things suboptimal about this approach, but
on the other hand we need to get our CI back up and running.

The basic approach is to - in the test suite, detect if we're on overlayfs. If
so, set a flag in the repo, which gets picked up by a few strategic places in
the core to turn on "ignore xattrs".

I also had to add a variant of this for the sysroot work.

The core problem here is while overlayfs will let us read and
see the SELinux labels, it won't let us write them.

Down the line, we should improve this so that we can selectively ignore e.g.
`security.*` attributes but not `user.*` say.

Closes: https://github.com/ostreedev/ostree/issues/758

Closes: #759
Approved by: jlebon
2017-03-24 22:16:43 +00:00
Jonathan Lebon
199646ccfa pull: don't use static deltas if archive repo
In https://github.com/ostreedev/ostree/pull/408, we disabled the use of
static deltas when mirroring. Later,
https://github.com/ostreedev/ostree/pull/506 loosened this up again so
that we could use static deltas when mirroring into bare{-user} repos.

However, the issue which originally spurrred #408 is even more generic
than that: we want to avoid static deltas for any archive repo, not just
when doing a mirror pull. This patch tightens this up, and also
relocates the decision code to make it easier to read.

Closes: #715
Approved by: cgwalters
2017-03-02 20:35:24 +00:00
Colin Walters
b5c5003ff6 pull: Fold together deltapart+fallback count for display
It's just simpler, and I'm not sure people are going to care
much about the difference by default.

We already folded in the fallback sizes into the download totals, so folding in
the count makes things consistent; previously you could see e.g.
`3/3 parts, 100MB/150MB` and be confused.

Closes: #678
Approved by: giuseppe
2017-02-17 14:58:25 +00:00
Simon McVittie
9a3f82caae Sourced test snippets: remove shebang and make non-executable
They are installed non-executable, which makes Debian's Lintian
packaging consistency check complain that #! is only useful
in executable scripts. But in fact they are not useful to execute
directly (they rely on setup being done in the script that sources
them), so just chmod them -x.

Closes: #652
Approved by: cgwalters
2017-01-19 13:54:59 +00:00
Colin Walters
17f264a487 repo: Add unconfigured-state to remote config options
This is a migration from the origin version.  It's
nicer to have it in the remote, since that's what one
needs to change.  Then tools don't need to mess with
the origin file.o

In fact in this scenario one can keep the "media source" like
`file:///install/repo` or whatever, since conceptually that's where it
came from.  We're just providing a better error.

Closes: https://github.com/ostreedev/ostree/issues/626

Closes: #627
Approved by: jlebon
2016-12-09 17:46:54 +00:00
Colin Walters
2139f0e437 pull: Don't do deltas with --commit-metadata-only
We should just download the commit objects directly, as it's
obviously a lot more efficient than deltas.

I had to generate a summary file in more places in the tests,
since once created, it needs to be updated.

Closes: https://github.com/ostreedev/ostree/issues/528

Closes: #566
Approved by: jlebon
2016-11-04 16:04:23 +00:00
Colin Walters
bac1fa1839 pull: Don't execute static deltas when mirroring
We don't presently support this, since the static delta code assumes
it can just `mmap()` file objects.  We could at some point implement
this, but for now just skip executing deltas when doing
`archive -> archive` mirroring.

I noticed this when trying to mirror a repo in Jenkins in
[CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel).

Closes: #408
Approved by: giuseppe
2016-07-25 06:50:51 +00:00
Colin Walters
d25212f04a tests: Port to glib-tap.mk, make make check run all of the tests
OSTree's code for testing predates the `glib-tap.mk` making its
way into GLib.  Let's switch to it, as it provides a number
of advantages.

By far the biggest advantage is that `make check` can start to run
most of the tests *in addition* to having them work installed.

This commit keeps the installed tests working, but `make check` turns
out to be really broken because...our TAP usage has bitrotted to say
the least.  Fix that all up.

Do some hacks so that the tests work uninstalled as well - in
particular, `glib-tap.mk` and the bits encoded into
`g_test_build_filename()` assume *recursive* Automake (blah).  Work
around that by creating a symlink when installed to loop back.
2016-03-03 08:50:19 -05:00
Giuseppe Scrivano
82d4e7fe68 Fix make syntax-check
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-03-01 10:08:25 -05:00
Colin Walters
04d77da005 deltas: Use endianness marker when parsing
Extend the `static-delta show` and `pull` commands to use the
endianness information (if available).
2016-02-26 08:19:01 -05:00
Colin Walters
4beb5f4eaf pull: Add a --dry-run option for static deltas
One of the design goals with deltas was not just wire efficiency,
but also having all the data up front about how much data would
be transferred before starting.

Let's expose that better by adding a `dry-run` option to the pull API.
This requires static deltas to be useful.  Basically we simply call
the progress callback once with the data from the superblock.
2016-02-19 15:21:13 -05:00
Colin Walters
f2c5ecb996 pull: Add require-static-deltas pull option
For a production release repository, most OS vendors would want
to just always use static deltas.  Add the ability for the pulls to
require it.

(I think I'll also add a summary key for this actually in addition,
 so the repo manager can force it too)
2016-02-19 15:21:13 -05:00
Colin Walters
42c60effbe pull: Support specifying exact commit to pull via branch@commit
I don't know why we didn't do this a long time ago.  This extends the
pull API to allow grabbing a specific commit, and will set the branch
to it.  There's some support for this in the deploy engine, but there
are a lot of reasons to support it for raw pulls (such as subset
mirroring cases).

In fact I'm thinking we should also have the override-version logic
here too.

NOTE: One thing I debated here is inventing a new syntax on the
command line.  Git doesn't seem to have this functionality (probably
because it'd be rarely used). The '@' character at least doesn't
conflict with anything.

Anyways, I wanted this for some other test cases.  Without this,
writing tests that go between different commits is more awkward as one
must generate the content in one repo, then pull downstream, then
generate more content, then pull again.  But now I can just keep track
of commit IDs and do exactly what I want without synchronizing the
tests.
2016-02-14 10:12:26 -05:00
Colin Walters
5ebe43859d tests: Use "bash strict mode"
I noticed in the static deltas tests, there were some tests that
should have been under `-o pipefail` to ensure we properly propagate
errors.

There were a few places where we were referencing undefined variables.

Overall, this is clearly a good idea IMO.
2016-01-27 11:44:10 -05:00
Giuseppe Scrivano
f88a9733ce tests: add missing ${CMD_PREFIX}
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-12-04 11:10:51 -05:00
Giuseppe Scrivano
45cee1bd70 tests: prefix invocation of ostree with where missing
And add a syntax rule to avoid this in future.

Fixed by:

sed -i -e 's|^ostree |${CMD_PREFIX} ostree |g' tests/*.sh

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-11-16 11:07:55 +01:00
Alexander Larsson
ee70ffbbcb Add tests for inline static deltas 2015-11-10 08:56:13 +01:00
Giuseppe Scrivano
4282ef65bf tests: add new test for pull --disable-static-deltas
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-05-13 09:38:04 +02:00
Colin Walters
ab3bf493dd pull: Optimize file:/// URIs to skip libsoup and hardlink if possible
It's always been suboptimal to have both pull and pull-local; as we go
beyond the raw object data into things like deltas and summary files,
the logic to perform e.g. mirroring should only be in one place.

This will be used by Pulp's OSTree content plugin at least to perform
promotions.
2015-02-06 10:02:57 -05:00
Colin Walters
ca678224be Static deltas support
https://bugzilla.gnome.org/show_bug.cgi?id=721799
2014-12-18 12:48:47 +01:00
Colin Walters
6a5e66b152 pull: Support --mirror option
There's several use cases for calling into ostree itself to do
mirroring, instead of using bare rsync.  For example, it's a bit more
efficient as it doesn't require syncing the objects/ directory.

https://bugzilla.gnome.org/show_bug.cgi?id=728351
2014-06-16 17:14:13 -04:00
Colin Walters
390b781d3a pull: Add remotename:ref syntax
This is really the common case.
2014-01-21 08:57:34 +00:00
Colin Walters
0a9f246016 tests: Small tweaks to pull corruption test
This one is failing here, I suspect it's the generic pull race
condition, but these fixes should make it slightly more reliable.
2014-01-19 11:48:27 -05:00
Jeremy Whiting
7f9eefb62d pull: Verify commits with gpg signatures from detached metadata
This uses gpgv for verification against DATADIR/ostree/pubring.gpg by
default.  The keyring can be overridden by specifying OSTREE_GPG_HOME.

Add a unit test for commit signing with gpg key and verifying on pull;
to implement this we ship a test GPG key generated with no password
for Ostree Tester <test@test.com>.

Change all of the existing tests to disable GPG verification.
2013-09-29 14:49:47 -04:00
Colin Walters
7d72a4b941 pull: Also fetch detached commitmeta files
These will contain GPG signatures and the like in the future, so we
should fetch them now.
2013-09-26 14:36:06 -04:00
Colin Walters
bea4a7538f Fix test reference to obsolete ostree-pull
It was leftover in my buildroot =/
2013-07-09 09:28:02 -04:00
Colin Walters
7aa620109e Add tests missing from previous commit 2012-09-23 19:25:09 -04:00