Commit Graph

62 Commits

Author SHA1 Message Date
Ruixin
f07432d4ce checkout: add an extra checkout_overwrite mode
This is for issue projectatomic/rpm-ostree#365,
an extra option of overwrite mode is added to the checkout command
so that when there is "non-directory" file already exist
during checkout, the error will be handled.

Some tests are added for regression

Closes: #1116
Approved by: cgwalters
2017-09-01 15:42:50 +00:00
Simon McVittie
41e6871e51 basic-test: Skip explicit uses of bare-user if no user xattrs
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1120
Approved by: cgwalters
2017-08-29 19:08:59 +00:00
Colin Walters
779f125cbe lib/repo: Auto-recreate repo/tmp if it's deleted
We can accumulate a lot of space there; let's be nice to people who delete the
whole directory.

Closes: https://github.com/ostreedev/ostree/issues/1018

Closes: #1020
Approved by: jlebon
2017-07-19 15:01:19 +00:00
Krzesimir Nowak
cf16805a2f ostree: Add collection and ref bindings to metadata on commit
The collection and ref bindings are stored in the commit metadata
under ostree.collection-binding and ostree.ref-binding,
respectively. They will be used to verify if the commit really comes
from the collection and ref we wanted to pull from.

Closes: #972
Approved by: cgwalters
2017-07-06 19:08:14 +00:00
Colin Walters
cd7d35945a bin/commit: Add '=' to --statoverride
Previously, we only supported additions in the statoverride file;
it was mainly for adding the setuid bit without having that physically
on disk.

However, for testing a change to `bare-user` handling around *unreadable*
files (which happens for `/etc/shadow` in host content), I need a way
to write that into a repo in the test suite.

I'm not actually aware of a non-test-suite use case for this; a more
sophisticated user is going to be using the API directly, which can already do
this. But we need it for tests at least.

Closes: #989
Approved by: jlebon
2017-06-30 21:23:48 +00:00
Colin Walters
90e0d56332 tree-wide: Replace various uses of archive-z2archive
The `-z2` is annoying now since it's really a legacy; we've long
since supported typing `archive`.  Convert the docs fully and
explain that.

Also do some (but not all) of the tests just to encourage newer tests to use
`archive` too.

Closes: #980
Approved by: jlebon
2017-06-29 16:00:13 +00:00
Philip Withnall
fbf8df8829 lib/refs: Add methods for setting/listing collection–refs
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Colin Walters
0635fcbfd9 lib/checkout: Add bareuseronly_dirs option
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.

See: https://github.com/flatpak/flatpak/issues/845

This option suppresses mode bits outside of `0775` for directory
checkouts.  I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.

Closes: #927
Approved by: alexlarsson
2017-06-13 20:05:31 +00:00
Colin Walters
6ed824bf00 lib/pull: Add OSTREE_REPO_PULL_FLAGS_BAREUSERONLY_FILES
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845

We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909

This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.

Closes: #926
Approved by: alexlarsson
2017-06-13 18:44:28 +00:00
Colin Walters
b614c65eab lib/repo: Import metadata via hardlink even for distinct repo modes
Our previous logic for import-via-hardlink only tried if the repo modes match,
but we *can* hardlink metadata between e.g. `archive` and `bare-user` repos, and
that's quite useful thing to do. Our documentation encourages converting to/from
those repo modes locally for build systems.

Closes: #922
Approved by: alexlarsson
2017-06-13 12:02:12 +00:00
Colin Walters
5913b22944 lib/repo: For bare-user, mask content object modes with 0775
Having every object in a bare-user repo (and checkouts) be executable
is ugly.  I can't think of a good reason to do that; they should only
be executable if their input is.  This does
for `bare-user` what we did for `bare-user-only` in
https://github.com/ostreedev/ostree/pull/909
It's also a stronger version of what we do with `checkout -U` in suppressing
suid - here we also strip world-writable files and the sticky bit (even though
that's meaningless today, it might not be in the future).

Closes: https://github.com/ostreedev/ostree/issues/907

Closes: #908
Approved by: alexlarsson
2017-06-08 06:50:16 +00:00
Colin Walters
d3900f90f4 lib/repofile: Follow symlinks for g_file_read()
This avoids `ostree cat /path/to/symlink` crashing, a longstanding embarassing
issue.

Closes: #915
Approved by: jlebon
2017-06-07 20:54:32 +00:00
Colin Walters
0c4b3a2b6d Canonicalize bare-user-only perms with 0755 mask
For the flatpak use case where bare-user-only was introduced, we actually
don't want to support s{u,g} id files in particular.

Actually, I can't think of a reason to have anything outside of the
`0755 i.e. (u=rwx,g=rx,o=rx)` mask, so that's what we do here.

This will have the effect of treating existing `bare-user-only` repositories as
corrupted if they have files outside of that mask, but I think we should do this
now; most of the flatpak users will still be on `bare-user`, and we haven't
changed the semantics of that mode yet.

Note that in this patch we will also *reject* file content that doesn't
match this.  This is somewhat asymmetric, since we aren't similarly rejecting
e.g. directory metadata.  But, this will close off the biggest source
of the problem for flatpak (setuid binaries).

See: https://github.com/ostreedev/ostree/pull/908
See: https://github.com/flatpak/flatpak/pull/837

Closes: #909
Approved by: alexlarsson
2017-06-07 15:13:55 +00:00
Jonathan Lebon
f813ae74ad basic-test.sh: explicitly check for uncompressed objects
It's not enough to check that the dir exists, since that's done by
default when we open the repo. We want to actually check that
uncompressed objects were cached (i.e. the opposite of the earlier error
path).

Closes: #903
Approved by: cgwalters
2017-06-02 17:46:16 +00:00
Jonathan Lebon
a32c6d2c70 checkout: also chmod in the user checkout case
When falling back to copying, we previously would only chmod checked out
files in the non-user-checkout mode. Fix this by always doing chmod.
The file_mode was being prepared but never actually applied.

Add a basic test in the archive-z2 --> usermode checkout case in which
we're guaranteed to always fall back to copy mode.

Closes: #633

Closes: #903
Approved by: cgwalters
2017-06-02 17:46:16 +00:00
Colin Walters
a195888b0f lib/checkout: Fix regression in subpath for regular files
This is what caused the merge of
https://github.com/projectatomic/rpm-ostree/pull/652
to blow up, since https://github.com/ostreedev/ostree/pull/848
landed right before we tried to merge it.

When I was writing that PR I remember having an uncertain feeling
since we were doing a `mkdirat` above, but at the time I thought
we'd have test suite coverage...turns out we didn't.

For backwards compatibility, we need to continue to do a `mkdirat` here of the
parent. However...I can't think of a reason anyone would *want* that behavior.
Hence, let's add a special trick - if the destination name is `.`, we skip
`mkdirat()`. That way rpm-ostree for example can open a dfd for `/etc` and avoid
the `mkdir`.

Fold the subpath tests into `test-basic.sh` since it's not worth a separate
file. Add a test case for checking out a file.

Closes: #854
Approved by: jlebon
2017-05-12 14:00:20 +00:00
Colin Walters
6060abbb4b repo: Add a "force copy" flag to checkout
This is intended to be used for copying `/usr/etc` → `/etc` for
deployments.

A TODO here is to use `glnx_file_copy_at()` if the repo mode allows
it - then we'd use reflinks if available.

Closes: #804
Approved by: jlebon
2017-04-24 15:26:11 +00:00
Colin Walters
08964d595d checkout: Fix bare-user symlink checkouts
Logic error introduced after refactoring; we hoisted the
`is_bare_user_symlink` variable to the top, but its computation
below.  But the `is_bare` symlink depended on it.

Closes: https://github.com/ostreedev/ostree/issues/798

Closes: #799
Approved by: jlebon
2017-04-18 14:35:45 +00:00
Colin Walters
d3385a3014 checkout: Provide useful error with checkout -H and incompat mode
Previously we'd assert and dump core if one used `checkout -H` without
`-U` on a bare-user repo, because we'd hit the bare-user symlink case.

Rework the code to handle this, and add tests. I hit this when I was going to
suggest to someone to use `-H` to ensure they were getting hardlinks.

Closes: #779
Approved by: jlebon
2017-04-12 17:06:44 +00:00
Alexander Larsson
7c8f95c86f Add basic tests for bare-user-only repo modes
This is somewhat complicated by such repos only properly supporting
some subset of file metadata (uid/gid 0, etc). We fix this by
always commiting with filters that make it work.

Closes: #750
Approved by: cgwalters
2017-03-27 13:48:41 +00:00
Colin Walters
455cc5e892 repo+tests: Add [core]disable-xattrs=true, use it on overlayfs
There are a lot of things suboptimal about this approach, but
on the other hand we need to get our CI back up and running.

The basic approach is to - in the test suite, detect if we're on overlayfs. If
so, set a flag in the repo, which gets picked up by a few strategic places in
the core to turn on "ignore xattrs".

I also had to add a variant of this for the sysroot work.

The core problem here is while overlayfs will let us read and
see the SELinux labels, it won't let us write them.

Down the line, we should improve this so that we can selectively ignore e.g.
`security.*` attributes but not `user.*` say.

Closes: https://github.com/ostreedev/ostree/issues/758

Closes: #759
Approved by: jlebon
2017-03-24 22:16:43 +00:00
Erik Larsson
e665e51408 diff: Add ostree_diff_dirs_with_options(), expose via cmdline
The first options are owner_uid/owner_gid, which makes it possible to use diff
on local files where --owner-uid/gid have been passed to commit.

Closes: #740
Approved by: cgwalters
2017-03-21 13:38:04 +00:00
Colin Walters
94948e3522 checkout: Support a "pure addition" mode
I plan to use this for `rpm-ostree livefs`.
https://github.com/projectatomic/rpm-ostree/issues/639

Closes: #714
Approved by: jlebon
2017-03-06 20:58:04 +00:00
Colin Walters
09b392675a main: Make ostree --version output YAML (and add gitrev)
I learned today that `docker version` does this and I really like
the idea.  While we have the patient open, also add the gitrev
with code taken from https://github.com/projectatomic/rpm-ostree/pull/584

Closes: #691
Approved by: giuseppe
2017-02-22 18:57:18 +00:00
Colin Walters
46544f5b4d commit: Support -F/--body-file, like git
This is more convenient to script for projects which haven't
yet made the leap to using the API.

Closes: https://github.com/ostreedev/ostree/issues/674

Closes: #681
Approved by: jlebon
2017-02-14 14:15:08 +00:00
Simon McVittie
9a3f82caae Sourced test snippets: remove shebang and make non-executable
They are installed non-executable, which makes Debian's Lintian
packaging consistency check complain that #! is only useful
in executable scripts. But in fact they are not useful to execute
directly (they rely on setup being done in the script that sources
them), so just chmod them -x.

Closes: #652
Approved by: cgwalters
2017-01-19 13:54:59 +00:00
Colin Walters
2f71136eec tests: Alias assert_not_reached() -> fatal()
We had a lot of copies of the "echo something 1>&2; exit 1" code even though
`assert_not_reached()` was it.  Hence, I think we need a shorter alias for that.

Doing this particularly since I noticed a missing `1` in an `exit 1` call in the
rpm-ostree copy of this.

Closes: #648
Approved by: jlebon
2017-01-18 14:28:29 +00:00
William Manley
0ee9e221be ostree commit: Fix combining trees with multiple --tree=ref arguments
You'd expect

    ostree commit --tree=ref=A --tree=ref=B

to produce a commit with the union of the trees given.  Instead you'd get
a commit with the contents of just the latter commit.  This was due to an
optimisation where we'd skip filling out the `files` and `subdirs`
members of the mtree, just filling in the metadata instead.  This backfires
becuase this same code relies on checking the `files` and `subdirs` members
itself to work out whether the mtree is empty.

This commit removes the optimisation, fixing the bug.  Maybe there's a way
to keep the optimisation and still fix the bug but it's not obvious to
me.

Closes: #581
Approved by: cgwalters
2016-11-17 14:45:55 +00:00
Alexander Larsson
bd45e7ac19 commit: Fix reading xattrs from OstreeRepoFile:s
When doing commit --tree=ref=XXX while at the same time applying some
form of modifier, ostree dies trying to read the xattrs using the
raw syscalls. We fix this by falling back to ostree_repo_file_get_xattrs()
in this case.

Also adds a testcase for this.

Closes: #577
Approved by: cgwalters
2016-11-16 22:30:33 +00:00
Colin Walters
845dc65196 repo: Revert default timestamp from 1 back to 0
Quoting Dan Nicholson in

  <https://github.com/ostreedev/ostree/pull/330#issuecomment-245499099>

  mtime of 0 has been the semantics of ostree deployments from basically
  the beginning of the project. We (and others, see
  flatpak/flatpak@b5204c9) rely on that fact when generating trees.

  In particular, this affects caches that use the mtime of the
  associated file or directory to determine if the cache is valid. By
  arbitrarily changing the mtime of the files to something else, all
  the caches we setup in the build are now invalidated. Preseeding
  caches is really important to the user experience as it avoids
  having the user wait while they're regenerated on first run.

  Now, we could change our build infrastructure to preset all the
  mtimes to 1 to match this change, but what does that do for our
  existing users who are on an ostree that deploys with mtimes of 0?
  We could just revert this change at Endless (and the associated one
  in Flatpak), and that would be fine for our users. However, if we
  point non-Endless users to our apps, they'll have the great
  experience of waiting 10 seconds the first time they launch it while
  the fontconfig cache is rebuilt unnecessarily.

Closes: #495
Approved by: jlebon
2016-09-08 13:35:59 +00:00
Simon McVittie
0de51bfc45 Skip tests that use whiteouts under Docker/aufs
ostree's naming convention for whiteouts is similar to what is
done in aufs, which means we can't compose the trees to test this
feature under Docker with the aufs storage driver, as used on
travis-ci.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #437
Approved by: cgwalters
2016-08-05 20:32:57 +00:00
Colin Walters
5334758ba7 repo: Make ostree_repo_create() nonfatal on existing repos
In general we want to support "idempotentcy" or "state
synchronization" across interruption.  If a repo is only partially
created due to a crash or whatever, it's hard for a user to know that.
Let's just make `ostree_repo_create()` idempotent. Since all we're
doing is a set of `mkdirat()` invocations, it's quite simple.

This also involved porting to fd-relative, which IMO makes the
code a lot clearer.

Closes: #422
Approved by: 14rcole
2016-08-01 15:12:14 +00:00
Colin Walters
ab47a8a030 Add "archive" as an alias for "archive-z2"
I find the "-z2" is really a long ago relic of the past when I changed
the format.  We no longer have anything to do with the original
`archive`, so let's start allowing people to type `--mode=archive`
which just looks saner.

At some point later I'll update the docs too, but it'll be an annoying
transition period as we'll have to say "On older OSTree, use -z2" etc.

Closes: #346
Approved by: giuseppe
2016-06-16 14:34:23 +00:00
Mathnerd314
0e9a875393 repo: use OSTREE_TIMESTAMP (=1) for checked-out files
1 is a better choice than 0 because some programs use 0
as a special value; for example, GNU Tar warns of an
"implausibly old timestamp" with 0.

Closes: #330
Approved by: cgwalters
2016-06-09 18:04:55 +00:00
Mathnerd314
80774680e8 commit: Don't require a subject
In practice, a lot of subjects are empty, because the commit date
and branch are sufficient identification. For example, rpm-ostree
does not use subjects. It also doesn't use the command-line ostree
commit tool, so this was not a problem there, but this makes the
behavior consistent.

Also adds a test that empty subjects and omitting the subject
are equivalent. The --timestamp is so that the commits do not
have different timestamps.

Closes: #305
Approved by: cgwalters
2016-05-25 18:37:47 +00:00
Jonathan Lebon
65e7969b0b OstreeRepo: add extensions directory
It's very useful for third-party applications to have someplace to store
their data guaranteed to be on the same device as the repo (thus
ensuring hardlinks) while still being shielded away from any of OSTree's
timely garbage collections.

We create a new "extensions/" subdirectory where apps can include
whatever they wish in "extensions/myapp/". This subdirectory is
completely unmanaged by ostree.

NB: I didn't bother making it a member of the OstreeRepo proper since we
don't really use it for anything else yet.

Closes: #286
Approved by: cgwalters
2016-05-09 14:43:16 +00:00
Jonathan Lebon
b717fd2c18 ot-builtin-commit.c: add --skip-list option
This was already supported by the commit modifier API, just needed to
expose it. This will also be used to test the libarchive API in a future
test.

Closes: #275
Approved by: cgwalters
2016-05-06 14:44:55 +00:00
Colin Walters
e3ec83a934 repo: Add OSTREE_BOOTID override envvar for debugging
It's useful for test cases to be able to influence this.

Conflicts:
	src/libostree/ostree-repo.c

Closes: #170
Approved by: jlebon
2016-05-02 18:44:44 +00:00
Colin Walters
15b3cab65e repo: Add OSTREE_REPO_TEST_ERROR=pre-commit env var
Setting this causes commit to error out.  There are other ways we
could do this in a more sophisticated fashion, such as via SystemTap
etc.  But this has low-tech applicablity, works as non-root.

The reason I'm adding this is so that we can add test cases for
cleanup of the `tmp/staging-` directory.

Closes: #170
Approved by: jlebon
2016-05-02 18:44:44 +00:00
Jonathan Lebon
8cda8b6866 basic-test: commit with a non-empty subject
[smcv: split out from a larger commit, part of PR #231; add commit message]
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #232
Approved by: cgwalters
2016-03-31 14:22:06 +00:00
Colin Walters
23d26d5f65 commit: Support writing orphans
The API supports this, and it's not hard for us to do in the command
line as well.  One possible use case is separating "content
generation" in a separate server.

Related: https://github.com/ostreedev/ostree/pull/223

Closes: #227
Approved by: jlebon
2016-03-30 03:10:50 +00:00
Colin Walters
c6b4ecd474 commit: Support generating commits with no parent, or a custom one
When I'm doing local development builds, it's quite common for me not
to want to accumulate history.  There are also use cases for this on
build servers as well.

In particular, using this, one could write a build system that didn't
necessarily need to have access to (a copy of) the OSTree repository.
Instead, the build system would determine the last commit ID on the
branch, and pass that to a worker node, then sync the generated
content back.

The API supported generating custom commits that don't necessarily
reference the previous commit on the same branch, let's just expose
this in the command line for convenience.

I plan to also support this rpm-ostree.

Closes: #223
Approved by: jlebon
2016-03-29 14:31:29 +00:00
Colin Walters
e2234e854d tests/basic: Fix race in timestamp test
%Z only uses seconds, so it's possible that we did the commit
in the same second, which made this test racy.

- Switch to full nanosecond precision using '%.Y' so it always differs
- Fix the inverted `cmp` usage
- Add a missing `ok`
2016-03-04 10:53:47 -05:00
Colin Walters
12db46e5af tests: More TAP fixups
Hopefully getting closer now.
2016-03-03 18:00:54 -05:00
Colin Walters
d25212f04a tests: Port to glib-tap.mk, make make check run all of the tests
OSTree's code for testing predates the `glib-tap.mk` making its
way into GLib.  Let's switch to it, as it provides a number
of advantages.

By far the biggest advantage is that `make check` can start to run
most of the tests *in addition* to having them work installed.

This commit keeps the installed tests working, but `make check` turns
out to be really broken because...our TAP usage has bitrotted to say
the least.  Fix that all up.

Do some hacks so that the tests work uninstalled as well - in
particular, `glib-tap.mk` and the bits encoded into
`g_test_build_filename()` assume *recursive* Automake (blah).  Work
around that by creating a symlink when installed to loop back.
2016-03-03 08:50:19 -05:00
Giuseppe Scrivano
c9b02ae1f3 refs: add tests
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-03-02 14:52:02 -05:00
Giuseppe Scrivano
82d4e7fe68 Fix make syntax-check
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-03-01 10:08:25 -05:00
Colin Walters
baaf7450da Support Docker-style whiteouts
This is to enable importing Docker layers as ostree commits, then
checking them out in a union.

The prototype work for this is in:
https://github.com/cgwalters/dlayer-ostree

Though it will likely ultimately end up in:
https://github.com/projectatomic/atomic
2016-02-16 10:54:59 -05:00
Colin Walters
5ebe43859d tests: Use "bash strict mode"
I noticed in the static deltas tests, there were some tests that
should have been under `-o pipefail` to ensure we properly propagate
errors.

There were a few places where we were referencing undefined variables.

Overall, this is clearly a good idea IMO.
2016-01-27 11:44:10 -05:00
Giuseppe Scrivano
a917c96976 tests: do not commit from the working directory
It fixes this problem:

```
error: Not a regular file or symlink: S.gpg-agent
```

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-07-21 10:02:30 -04:00