Commit Graph

3211 Commits

Author SHA1 Message Date
Colin Walters
a98faa9113 ci: Update to match current rpm-ostree
This copies the `ci/` directory from rpm-ostree, with much the same rationale;
among other things we don't want to depend on the Docker hub.

The specific reason I'm doing this is that I want to add a CentOS7 build, but
that means we can't use `projectatomic/ostree-tester`, and at that point we
might as well unwind it all.

Closes: #917
Approved by: jlebon
2017-06-08 21:08:31 +00:00
Alexander Larsson
2a3f17c7aa repo: After renaming in all loose objects, ensure metadata is stable
When a transaction is finished and we have moved all the staged loose
objects into the repo we fsync all the object directory, to ensure the
filenames are stable before we update the refs files to point to the
new commits.

With out this an unclean shutdown after the transaction is finished
could result in a refs file that points to an incomplete commit.

https://bugzilla.gnome.org/show_bug.cgi?id=759442

Closes: #918
Approved by: cgwalters
2017-06-08 20:03:18 +00:00
Colin Walters
c81252c1e0 repo/commit: Support group-writable files for bare-user-only
These exist in the wild for flatpak, and aren't really a problem. The canonical
permissions are still either `0755` or `0644`, we just support the additional
writable bit for the group (i.e. extend the set to include `0775` and `0664`)
now to avoid breaking some flatpak content.

Closes: #913
Approved by: alexlarsson
2017-06-08 06:58:54 +00:00
Colin Walters
81e1f7761f tests: Add a test for bare-user-only failing to commit suid content
We didn't have coverage of this before, and adding the test infrastructure will
help ensure we have coverage for more changes here.

Closes: #913
Approved by: alexlarsson
2017-06-08 06:58:54 +00:00
Colin Walters
5913b22944 lib/repo: For bare-user, mask content object modes with 0775
Having every object in a bare-user repo (and checkouts) be executable
is ugly.  I can't think of a good reason to do that; they should only
be executable if their input is.  This does
for `bare-user` what we did for `bare-user-only` in
https://github.com/ostreedev/ostree/pull/909
It's also a stronger version of what we do with `checkout -U` in suppressing
suid - here we also strip world-writable files and the sticky bit (even though
that's meaningless today, it might not be in the future).

Closes: https://github.com/ostreedev/ostree/issues/907

Closes: #908
Approved by: alexlarsson
2017-06-08 06:50:16 +00:00
Colin Walters
d3900f90f4 lib/repofile: Follow symlinks for g_file_read()
This avoids `ostree cat /path/to/symlink` crashing, a longstanding embarassing
issue.

Closes: #915
Approved by: jlebon
2017-06-07 20:54:32 +00:00
Colin Walters
807a804b16 lib/repofile: Port mostly to new code style
Prep for a bugfix.

Closes: #915
Approved by: jlebon
2017-06-07 20:54:32 +00:00
Colin Walters
6b402e53f4 builtins/cat: Port to new code style
Definitely better.  Prep for another fix.

Closes: #915
Approved by: jlebon
2017-06-07 20:54:32 +00:00
Philip Withnall
4418ab7fa9 lib/fetcher: Add cleanup function for OstreeFetcher
This is only used internally (the header is not public), so it doesn’t
have to go in ostree-autocleanups.h. It will be used in some following
commits.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:36 +00:00
Philip Withnall
59ffce73c8 lib/repo: Omit deltas from the summary file if there are none
If there are no deltas to be listed in the summary file, don’t bother
including the key for them in the additional metadata section of the
file. This saves a few bytes in some cases.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall
7ce6777028 ostree/dump: Improve formatting for well-known commit metadata keys
This follows on from commit a946c3d4, which added formatting for
well-known summary metadata keys. This commit adds it for commits.

Currently, the only well-known commit metadata key is
ostree.commit.timestamp. Formatting this correctly is especially
important, since it’s a big-endian uint64, which is completely unusable
for mere mortals when presented as a number rather than a date.

Non-formatted output can still be retrieved using the OSTREE_DUMP_RAW
flag, and the non-formatted key name is always printed for clarity.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall
87413ee63b lib/pull: Fix an over-indented block
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall
c968c12ec3 lib/pull: Drop some trailing whitespace
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall
14082e6b03 lib/pull: Simplify a for-loop initialisation
It’s a bit neater to initialise the loop iterator and maximum in the
same place.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall
52ede69df8 lib/pull: Fix a typo in a documentation comment
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall
12479d8b05 lib/repo: Reindent some code in regenerate_summary() for clarity
This makes it a bit more easily separable from the rest of the code in
the function. No functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Colin Walters
0c4b3a2b6d Canonicalize bare-user-only perms with 0755 mask
For the flatpak use case where bare-user-only was introduced, we actually
don't want to support s{u,g} id files in particular.

Actually, I can't think of a reason to have anything outside of the
`0755 i.e. (u=rwx,g=rx,o=rx)` mask, so that's what we do here.

This will have the effect of treating existing `bare-user-only` repositories as
corrupted if they have files outside of that mask, but I think we should do this
now; most of the flatpak users will still be on `bare-user`, and we haven't
changed the semantics of that mode yet.

Note that in this patch we will also *reject* file content that doesn't
match this.  This is somewhat asymmetric, since we aren't similarly rejecting
e.g. directory metadata.  But, this will close off the biggest source
of the problem for flatpak (setuid binaries).

See: https://github.com/ostreedev/ostree/pull/908
See: https://github.com/flatpak/flatpak/pull/837

Closes: #909
Approved by: alexlarsson
2017-06-07 15:13:55 +00:00
Owen W. Taylor
25696b3fb0 lib/repo: Don't copy xattrs when manipulating the GPG keyring
Copying xattrs when manipulating the GPG keyring for a repository
causes errors when the underlying filesystem doesn't support writing
xattrs - overlayfs is a common example. It also causes the selinux
attributes of the keyring files to be copied from the temporary
location instead of properly inherited from the destination directory
(ending up, for example, as unconfined_u:object_r:user_tmp_t:s0, rather
than unconfined_u:object_r:data_home_t:s0)

Closes: #910
Approved by: cgwalters
2017-06-07 13:43:51 +00:00
Colin Walters
e18cacb06e Don't install trivial-httpd man page if not enabled
I just noticed this scroll by in a file listing.

Closes: #905
Approved by: jlebon
2017-06-02 19:36:56 +00:00
Jonathan Lebon
f813ae74ad basic-test.sh: explicitly check for uncompressed objects
It's not enough to check that the dir exists, since that's done by
default when we open the repo. We want to actually check that
uncompressed objects were cached (i.e. the opposite of the earlier error
path).

Closes: #903
Approved by: cgwalters
2017-06-02 17:46:16 +00:00
Jonathan Lebon
ab7c3fd800 manual: document bare-user-only repo mode
Closes: #903
Approved by: cgwalters
2017-06-02 17:46:16 +00:00
Jonathan Lebon
a32c6d2c70 checkout: also chmod in the user checkout case
When falling back to copying, we previously would only chmod checked out
files in the non-user-checkout mode. Fix this by always doing chmod.
The file_mode was being prepared but never actually applied.

Add a basic test in the archive-z2 --> usermode checkout case in which
we're guaranteed to always fall back to copy mode.

Closes: #633

Closes: #903
Approved by: cgwalters
2017-06-02 17:46:16 +00:00
Jonathan Lebon
3ec2b5773e checkout: don't apply SELinux labeling in user mode
If the user requested a user checkout, we don't want to set the SELinux
label xattr.

Closes: #903
Approved by: cgwalters
2017-06-02 17:46:16 +00:00
Brian C. Lane
c651982929 Remove the OSTREE_MAX_RECURSION limit on metadata depth
This was making it impossible to pull or mirror a large ostree repo, and
according to Colin is no longer necessary. It works fine with a test
against a repo with 2741 commit and 451468 objects in it.

Closes: #899

Closes: #904
Approved by: jlebon
2017-06-02 16:18:28 +00:00
Colin Walters
cad42d9601 Revert "Add a notion of "physical" sysroot, use for remote writing"
This reverts commit 1eff3e8343. There
are a few issues with it.  It's not a critical thing for now, so
let's ugly up the git history and revisit when we have time to
debug it and add more tests.

Besides the below issue, I noticed that the simple `ostree remote add`
now writes to `/ostree/repo/config` because we *aren't* using the
`--sysroot` argument.

Closes: https://github.com/ostreedev/ostree/issues/901

Closes: #902
Approved by: mike-nguyen
2017-06-02 14:11:58 +00:00
Krzesimir Nowak
2fdbdd4b2f lib/sysroot: Document the NO_CLEAN flag
Closes: #900
Approved by: jlebon
2017-06-02 13:19:51 +00:00
Colin Walters
a094879f3a lib/repo: Delete unused private prototypes
The implementations were removed in: 6ffcb24d22
I noticed this while looking at the commit code.

Closes: #898
Approved by: jlebon
2017-06-01 21:30:51 +00:00
Colin Walters
f4f1330789 repo/commit: Split up metadata/content commit paths
There was a lot of conditionals inside `write_object()` differentating
between metadata/content, and then for content, on the different repo
types.  Further, in the metadata path since the logic is simpler, can
present a non-streaming API, and further use `OtTmpfile`, etc.

Splitting them up helps drop a lot of conditionals. We introduce a small
`CleanupUnlinkat` that allows us to fully convert to the new code style in both
functions.

This itself is still prep for fully switching to `GLnxTmpfile`.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters
ec1964dd44 repo/commit: Don't renormalize trusted metadata
As the comment in the code says; in the expected checksum case, the caller
really has to have a normal form already.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters
6ba4dac6f2 repo/commit: In the expected checksum case, check existence early
If we have an expected checksum, call `fstatat(repo_dfd, checksum)`
early on before we do much else.  This actually duplicates code,
but future work here is going to split up the metadata/content
commit paths, so they'll need to diverge anyways.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters
d2a92df155 repo/commit: Dedup content writing API implementation
Similar to metadata, for `write_content_trusted()` we can just
call `_write_content()` with a `NULL` output checksum.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters
22b1234f52 repo/commit: Dedup metadata writing API implementations
First, the streaming metadata API is pretty dumb, since metadata
should be small.  Really we should have supported a `GBytes`
version.  Currently, this API *is* used when we do local pulls,
so this commit has test coverage.  However, I plan to change
the object import to avoid using this.  But that's fine, since
I can't think of why someone would use this API.

Next, the only difference between `ostree_repo_write_metadata()` and
`ostree_repo_write_metadata_trusted()` is whether or not we pass
an output checksum; so just dedup the implementations.

Also while I'm here break out the input length validation and do
it early in the streaming case.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters
1eff3e8343 Add a notion of "physical" sysroot, use for remote writing
Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).

In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.

Closes: https://github.com/ostreedev/ostree/issues/892

Closes: #896
Approved by: jlebon
2017-06-01 18:15:56 +00:00
Colin Walters
2414704609 lib/util: Some style conversion
I saw a few instances of `glnx_set_error_from_errno() + return FALSE`,
and fixed them and did a bit of style conversion.

Closes: #895
Approved by: jlebon
2017-05-30 20:33:28 +00:00
Colin Walters
9a3555a74b cmd: Use autoptr for GKeyFile
Prep for code style conversion.

Closes: #891
Approved by: jlebon
2017-05-29 14:09:08 +00:00
Colin Walters
88a1fc92a9 tree-wide: Add+run spatch to use glnx_throw()
I had to run a sed job to add whitespace after, but otherwise this was easy.

Closes: #890
Approved by: jlebon
2017-05-26 19:27:11 +00:00
Colin Walters
9bf8a8503a lib/sysroot: Add non-failable ostree_sysroot_repo()
Having a failable accessor is annoying, since it's really common
to reference both.  Instead, open the repo once when we load
the sysroot, and provide a non-failable accessor.

This is also prep for `ostree_repo_open_at()`, which collapses the separation
between `ostree_repo_new()` and `ostree_repo_open()`.

Closes: #886
Approved by: jlebon
2017-05-26 19:17:59 +00:00
Colin Walters
ed430b45de lib: Add an "is_system" member to OstreeRepo
This is prep for introducing a fd-relative `ostree_repo_new_at()`.
Previously, `ostree_repo_is_system()` compared `GFile` paths, but
there's a much simpler check we can do first - if this repository
was created via `OstreeSysroot`, it must be a system repo.

Closes: #886
Approved by: jlebon
2017-05-26 19:17:59 +00:00
Colin Walters
2f834968c6 lib/fsutil: Port to new code style
Pretty trivial.

Closes: #889
Approved by: jlebon
2017-05-26 19:05:53 +00:00
Colin Walters
ff2b881275 lib/fsutil: Delete unused GFile ioctl method
All the deployment code uses fds.

Closes: #889
Approved by: jlebon
2017-05-26 19:05:53 +00:00
Colin Walters
ad119aece9 pull-test: Add some 404 tests
See: https://github.com/flatpak/flatpak/issues/816

Closes: #887
Approved by: jlebon
2017-05-26 14:33:59 +00:00
Jonathan Lebon
07acb5b82c PAPR: migrate to the new name
The redhat-ci service has been renamed to PAPR. Previous values are
still supported though should be considered deprecated.

Closes: #885
Approved by: cgwalters
2017-05-25 16:31:08 +00:00
Colin Walters
822ade62c6 tests: Add some C tests for object writing
Prep for https://github.com/ostreedev/ostree/pull/881

Closes: #884
Approved by: jlebon
2017-05-25 14:24:05 +00:00
Colin Walters
c6960e63b2 lib/deploy: Port config merge logic to new code style
This is a de-scoping of work I did in preparation for
rpm-ostree [live updates](https://github.com/projectatomic/rpm-ostree/pull/652).
Originally I was going to expose this as a public API.

However, I decided to do things differently, but the cleanup here for new code
style and fd-relative is nice to have anyways.

We rework things to use `OstreeDeployment*`, which the caller is expected to
already have, rather than `GFile*`s pointing to the config directories.

Closes: #741
Approved by: jlebon
2017-05-24 16:31:55 +00:00
Alexander Larsson
be0c02d4f9 fetcher: Send Accept-Encoding: gzip when downloading summary
The summary file can get large, but it compresses well (something
which is not true of other files in the ostree repo which are
already compressed). By sending Accept-Encoding: gzip (and
handling the compressed results) we send a lot less data.

I set up the flathub repo (http://flathub.org/repo) to enable
gzip for the summary file (only), and the result is that the
331514 byte large summary was transferred in 122889 bytes.
On my (fast) network this decreased the time i took to do
"flatpak remote-ls flathub" by about 100msec.

This fixes https://github.com/ostreedev/ostree/issues/802

Closes: #882
Approved by: cgwalters
2017-05-24 13:52:17 +00:00
Philip Withnall
56188808b4 build: Use AM_TESTS_ENVIRONMENT rather than TESTS_ENVIRONMENT
TESTS_ENVIRONMENT is reserved for the user to be able to set when
running the tests. AM_TESTS_ENVIRONMENT is for the tests’ Makefile to
set itself.

https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #880
Approved by: cgwalters
2017-05-23 17:51:27 +00:00
Colin Walters
e99777e8d2 Add stub for new libglnx tmpfile API, port simpler callers to it
It's hard right now to do a full port to the new libglnx tmpfile
API since there are complex cases in the commit path which deal
with symlinks as well.

Let's make things more gradual by introducing the important part (struct with
autocleanup) here in libotutil, port what we can. This will make a future
complete port easier.

Closes: #871
Approved by: jlebon
2017-05-23 14:06:24 +00:00
Colin Walters
db00c9591f bin/cookies: Delete dead tmpfile code in cookie list command
This was a copy-paste-o.

Closes: #871
Approved by: jlebon
2017-05-23 14:06:24 +00:00
Daniel Drake
c1ed9a15c1 Allow commits to mark refs as EOL, replaced by others
A commit can now include a "ostree.endoflife-rebase" metadata key
pointing to a new ref.

When updating, the sysroot upgrader will see this and proceed to
pull and deploy the new ref instead. The origin file in the new
deployment will point to the new ref.

This functionality is planned to be used in Endless OS. We will create
a lesser tested branch for brand new, cutting edge hardware support,
and ship that on hardware platforms that require the latest drivers.
However, once our slower-moving official release is later updated to
support the new hardware, we will use this functionality to migrate
those bleeding-edge users over to the official release.

Closes: #874
Approved by: cgwalters
2017-05-22 20:18:07 +00:00
Daniel Drake
cd65f85dcb libtest: allow committing to alternative branches
This will be used by the upcoming test-admin-upgrade-endoflife.sh

Closes: #874
Approved by: cgwalters
2017-05-22 20:18:07 +00:00