Commit Graph

3855 Commits

Author SHA1 Message Date
Will Thompson
d04debb4fb build: fix "executible" typo
Closes: #1345
Approved by: jlebon
2017-11-16 14:25:53 +00:00
Colin Walters
23db56f9c3 bin: Port a few commands (diff,remote,static-delta) to new style
No functional changes, not prep for anything, just keeping up some momentum.

Closes: #1344
Approved by: jlebon
2017-11-15 23:31:26 +00:00
Carlos Alberto Lopez Perez
20996d0da3 grub-generator: If OSTREE_BOOT_PARTITION is not set, default to /boot
Closes: #1326
Approved by: cgwalters
2017-11-15 14:03:24 +00:00
Philip Withnall
4a58364cfa lib/repo: Fix a memory leak of options in ostree_repo_create()
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1341
Approved by: dbnicholson
2017-11-14 23:13:14 +00:00
Philip Withnall
f7568dbfc8 lib/repo: Add (transfer) annotations to various GHashTable arguments
By default, unless it’s const, an (out) GHashTable will be assumed to be
(transfer full). That means the binding needs to free all the items in
the hash table, plus the table itself.

However, all the GHashTables we use have free functions set already, so
freeing the hash table will free its items. This results in a
double-free.

Fix that by ensuring we annotate such (out) hash tables as (transfer
container). Also annotate some other hash tables as (transfer none)
where appropriate, for clarity.

This fixes OSTree.Repo.list_collection_refs() in the Python bindings.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1341
Approved by: dbnicholson
2017-11-14 23:13:14 +00:00
Colin Walters
6b9ce9d35d lib/traverse: Port to new style
Not prep for anything, was just reading this code a bit while
working on rpm-ostree jigdo.

Closes: #1338
Approved by: jlebon
2017-11-13 03:33:25 +00:00
Dan Nicholson
3e8b7e29fa Revert "lib/pull: Skip ostree_repo_resolve_keyring_for_collection for bindings"
This reverts commit 519b30b7e1. Now that
the experimental GIR is being built correctly and OstreeRemote is a real
boxed type, this can be exposed again.

Closes: #1337
Approved by: pwithnall
2017-11-10 10:03:44 +00:00
Dan Nicholson
ed242cdd3b lib: Include OstreeRemote and OstreeCollectionRef in GIR
Now that g-ir-scanner is being told about ENABLE_EXPERIMENTAL_API, it
can include these types correctly. Drop the __GI_SCANNER__ guards in the
header files so that all the declarations are found.

After this, you can actually construct the types normally:

>>> OSTree.CollectionRef.new('com.example.Foo', 'bar')
<OSTree.CollectionRef object at 0x7f2bba4c7528 (OstreeCollectionRef at 0x55c033ff2f30)>

Closes: #1337
Approved by: pwithnall
2017-11-10 10:03:44 +00:00
Dan Nicholson
a256b2d1a3 lib/remote: Export ostree_remote_get_type symbol
Without this, you can't really use OstreeRemote as a GObject, which is a
requirement for bindings.

This was found when attempting to include OstreeRemote in the GIR, and
g-ir-scanner wasn't able to link it's temporary object due to an
"undefined reference to `ostree_remote_get_type'" error.

Closes: #1337
Approved by: pwithnall
2017-11-10 10:03:44 +00:00
Dan Nicholson
374f7fc973 bin/summary: Fix --raw option
I wanted to inspect a summary file the other day and was saddened to
find it was broken:

  $ ostree summary --raw
  error: No option specified; use -u to update summary

Fix the test to do the normal thing of passing just --raw without
--view. It's legal to pass --raw and --view, but it shouldn't be a
requirement.

Closes: #1336
Approved by: cgwalters
2017-11-09 17:15:59 +00:00
Colin Walters
9856ed3840 deltas: Don't try to rollsum/bsdiff .xz files
Fedora switched to 'xz' compress kernel modules, and recently
[RHEL7 did too](https://bugzilla.redhat.com/show_bug.cgi?id=1367496).
This compression defeats bsdiff.

While we have a "rollsum-able" test, we don't have a "bsdiff-able" test as it'd
be very expensive (we'd have to bsdiff, then apply it and compare the result).

Let's do the tactical quick fix here and just not try to delta files ending in
`.xz.`. This avoids us using bsdiff pointlessly for over 4000 files, which is
quite a notable speed increase for generating deltas.

Closes: #1333
Approved by: jlebon
2017-11-09 03:10:49 +00:00
Philip Withnall
3cf53f7c58 lib/repo: Add gtk-doc comment to OstreeRepoCommitState
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1335
Approved by: cgwalters
2017-11-08 15:47:11 +00:00
Philip Withnall
9c4870b5e1 lib/repo: Add OSTREE_REPO_COMMIT_STATE_NORMAL to represent most commits
This allows more explicit handling of commit state in code using
libostree, rather than hard-coding a commit state of 0 for ‘normal’.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1335
Approved by: cgwalters
2017-11-08 15:47:11 +00:00
Kalev Lember
176a7b4778 fetcher/curl: Fix invalid memory access in finalize()
Reorder cleanup functions so that curl_multi_cleanup() runs before
self->sockets is destroyed. This avoids an assert and invalid memory
access in sock_cb where self->sockets is dereferenced during
curl_multi_cleanup().

Closes: https://github.com/ostreedev/ostree/issues/1331

Closes: #1332
Approved by: cgwalters
2017-11-07 15:37:20 +00:00
Philip Withnall
7d2ad351c4 build: Add a TODO comment about improving glib-mkenums usage in future
Signed-off-by: Philip Withnall <withnall@endlessm.com>

https://github.com/ostreedev/ostree/pull/1329

Closes: #1330
Approved by: cgwalters
2017-11-07 15:14:56 +00:00
Colin Walters
9f4b5c0cc5 build: Also fix -Werror=undef for old libsoup
Same change as before, but for libsoup.

Closes: #1328
Approved by: jlebon
2017-11-06 21:32:30 +00:00
Colin Walters
a16dddcaff build: Work around -Wundef with older GLib
This previous PR caused our `-Werror=undef` to trigger when building
with older GLib; this should work around it.

Closes: #1327
Approved by: jlebon
2017-11-06 20:03:06 +00:00
Colin Walters
015513b8f9 lib/pull: Avoid error if current with --require-static-deltas
A tricky thing here that caused this to go past a lot of our tests
is that the code was mostly OK if there was an available delta from
an older commit.  But this case broke if we e.g. had a new OS
deployment and did a `--require-static-deltas` pull, i.e. the initial
state.

I cleaned up our "find static delta state" function to return an enumeration,
and extended it with an "already have the commit" state.  A problem
I then hit is that we've historically fetched detached metadata for
non-delta pulls, even if the commit hasn't changed.  I decided not to
do that for `--require-static-deltas` pulls for now; otherwise the
code gets notably more complex.

Closes: https://github.com/ostreedev/ostree/issues/1321

Closes: #1323
Approved by: jlebon
2017-11-06 19:41:07 +00:00
Colin Walters
7296bf3dcc build: Add -Werror=undef by default, fix fallout
The main thing here is that a ton of stuff has happened in gnulib since we
imported `parse-datetime.y`. I cherry-picked a little bit of it, but that
upstream doesn't seem to build with `-Wundef`, so I just deleted some hunks.

(Note I reindented the warnings consistently)

Update submodule: libglnx

Closes: #1320
Approved by: jlebon
2017-11-06 15:30:14 +00:00
Dan Nicholson
2e9fea8fa5 tests: Don't symlink rofiles-fuse if it's disabled
Creating the symlink will cause make to try to build rofiles-fuse, which
will fail if it's disabled. Normally I wouldn't disable rofiles-fuse,
but it's triggering a hang in our ARM Xenial builder's kernel in splice.
I'm sure that's fuse's fault, but for now I just need to disable
rofiles-fuse there and found --disable-rofiles-fuse didn't actually
work.

Closes: #1325
Approved by: cgwalters
2017-11-03 22:20:00 +00:00
Dan Nicholson
519b30b7e1 lib/pull: Skip ostree_repo_resolve_keyring_for_collection for bindings
Since ostree_remote_get_type is not made available to g-ir-scanner, it
treats OstreeRemote as a bare struct. That's not kosher for bindings and
it issues the following warning:

  src/libostree/ostree-repo-pull.c:5560: Warning: OSTree:
  ostree_repo_resolve_keyring_for_collection: return value: Invalid
  non-constant return of bare structure or union; register as boxed type
  or (skip)

For now, just skip this API for bindings.

Closes: #1322
Approved by: pwithnall
2017-11-03 22:05:47 +00:00
Dan Nicholson
03bbe45530 lib/core: Fix documentation comment in ostree_validate_collection_id
g-ir-scanner was spitting this warning:

  src/libostree/ostree-core.c:281: Warning: OSTree:
  ostree_validate_collection_id: unknown parameter 'rev' in
  documentation comment, should be 'collection_id'

Closes: #1322
Approved by: pwithnall
2017-11-03 22:05:47 +00:00
Dan Nicholson
69c4737491 build: Define OSTREE_ENABLE_EXPERIMENTAL_API for g-ir-scanner
When compiling libostree, OSTREE_ENABLE_EXPERIMENTAL_API is managed via
config.h. However, g-ir-scanner can't use that since it gets confused
about the namespace of all the random macros. It won't include the
experimental APIs unless the macro is defined through another means.
Without this, none of the experimental APIs were being included in the
gir data.

Closes: #1322
Approved by: pwithnall
2017-11-03 22:05:47 +00:00
Dan Nicholson
6ec4cd3eb5 build: Ensure enumtypes.h is built before enumtypes.c
ostree-enumtypes.c includes ostree-enumtypes.h, so make needs to be told
about the dependency. Without it, parallel make could try to build
ostree-enumtypes.c before the header file exists.

I hit this when running `make -j OSTree-1.0.gir`.

Closes: #1322
Approved by: pwithnall
2017-11-03 22:05:47 +00:00
Colin Walters
758be138be build-sys: Post-release version bump
Closes: #1324
Approved by: pwithnall
2017-11-03 20:35:47 +00:00
Colin Walters
ae61321046 Release 2017.13
We've accumulated a fair bit, time for a new release before we
e.g. try to land the locking PR.

Closes: #1319
Approved by: jlebon
2017-11-02 13:53:41 +00:00
Colin Walters
d1b5997d8c Update libglnx
Mostly for https://github.com/GNOME/libglnx/pull/94
since I keep seeing it in the terminal.

Update submodule: libglnx

Closes: #1318
Approved by: jlebon
2017-11-01 19:14:17 +00:00
Shaun Taheri
370cbd006b static deltas: Set optional flag for superblock
Closes: #1317
Approved by: cgwalters
2017-11-01 18:26:38 +00:00
Colin Walters
9d82d258ba tests: Add a "pull-test2" that uses slightly more realistic content
This one actually exercises the delta paths more; I pulled the
data from some local F26AH builds.  One thing I noticed while doing
that is that we try bsdiff on xz'd data, but that won't be useful.
https://github.com/projectatomic/rpm-ostree/issues/470
https://bugzilla.redhat.com/show_bug.cgi?id=1367496

Anyways I verified this test fails without the fixup for `pread()`
in https://github.com/ostreedev/ostree/pull/1312

Closes: #1314
Approved by: jlebon
2017-11-01 17:38:26 +00:00
Colin Walters
aec1191fa9 deltas: Add an if0'd bit to print bsdiff objects
I used this to find the bsdiff'd objects.

Closes: #1314
Approved by: jlebon
2017-11-01 17:38:26 +00:00
Colin Walters
e885d029aa bin/delta-compilation: Add a "progress bar" for bsdiff generation
It's the slowest part, let's show admins something. This "update every 10%" code
was copied from the fsck command; obviously a better approach would be "progress
every N seconds" but doing that somewhat accurately requires making things
async; not worth it here yet.

Closes: #1314
Approved by: jlebon
2017-11-01 17:38:26 +00:00
Colin Walters
62445166fc lib/pull: Delete unused "exampleos" code
We never ended up using this, and I'm going to revisit this in another patch
with a different approach that has useful *content* and not just a lot of files.

Closes: #1314
Approved by: jlebon
2017-11-01 17:38:26 +00:00
Colin Walters
80ff73ba26 lib/pull: When --require-static-deltas, use them even for file:/// repos
I didn't fully spelunk this, but from what `static-delta-generate-crosscheck.sh`
had, we appeared to be doing this before, and it's clearly useful for local
testing rather than needing to spin up a HTTP server.

Closes: #1313
Approved by: jlebon
2017-10-31 14:20:20 +00:00
Colin Walters
7f8ff5756e lib: Minor static delta fixes
First, the manual crosscheck script bitrotted; it got caught up
in the "use libtest repo creation wrapper" bit, and also it
seems like at some point `pull --require-static-deltas` changed
meaning when dealing with `file:///` repos.  I have more work to
unwind that.

Next, I'm seeing a delta failure which looks like a static delta
miscompilation with rollsums; change the compiler to print out
the source object too, which helped me debug this.

And finally in the processing code, fix incorrect error prefixing, which was
misleading.

Closes: #1311
Approved by: ashcrow
2017-10-30 22:52:02 +00:00
Colin Walters
0d259ac401 lib/deltas: Fix change to use pread() in write opcode
Fixes: 93457071cb "lib/deltas: Use pread() instead of lseek()+read()"

Caught this when trying to test alex's patch locally. I am going to review our
static delta pulls and try to get something more comprehensive locally. But in
the meantime this patch is clearly right.

Closes: #1312
Approved by: jlebon
2017-10-30 19:03:16 +00:00
Alexander Larsson
de0e015908 static deltas: Process each part as soon as its done
Directly when we allocate a new part we finish the old one,
writing the compressed data to a temporary file and generating
the delta header for it.

When all these are done we loop over them and collect the headers,
sizes and either copy the tempfile data into the inlined superblock
or link the tempfiles to disk with the proper names.

Closes: #1309
Approved by: cgwalters
2017-10-27 21:49:26 +00:00
Alexander Larsson
cbbd159a5d static-delta-compilation: Move some things around
We will do some changes later that need these earliers, so move them up.

Closes: #1309
Approved by: cgwalters
2017-10-27 21:49:26 +00:00
Alexander Larsson
74a5df7bd7 static deltas: Use OtVariantBuilder to create deltas
This allows us to create the final delta desciptor directly on disk
rather than having it all in memory. This is nice because it can
become quite large if inlined parts are used.

Note however, that we currently generate all the delta parts in
memory before adding them to the delta, so we still keep all individual
parts in memory. Fixing that is the next step.

Closes: #1309
Approved by: cgwalters
2017-10-27 21:49:26 +00:00
Alexander Larsson
72bb1a6b17 Add OtVariantBuilder
This is similar to GVariantBuilder in that it constructs variant
containers, but it writes it directly to a file descriptor rather
than keep the entier thing in memory. This is useful to create large
variants without using a lot of memory.

Closes: #1309
Approved by: cgwalters
2017-10-27 21:49:26 +00:00
Simon McVittie
74e00d32be ostree-system-generator: Include <libglnx.h> for autocleanups
g_autoptr was new in GLib 2.44, but we officially only require 2.40,
so we need to use the backport in libglnx.

Signed-off-by: Simon McVittie <smcv@collabora.com>

Closes: #1310
Approved by: cgwalters
2017-10-27 18:54:11 +00:00
Colin Walters
90ebd48f6a lib/repo: Fix loading commitstate with parent repos
This makes the code nicer too. Properly unit testing this though really wants
like a whole set of stuff around parent repos...but we do have coverage of the
non-parent path in the current pull tests.

Closes: https://github.com/ostreedev/ostree/issues/1306

Closes: #1308
Approved by: alexlarsson
2017-10-26 07:06:50 +00:00
Philip Withnall
0760ce1281 lib/repo-finder-mount: Ignore mounts which have a ‘system’ file system
For example, tmpfs or a cgroup file system. This is basically an
optimisation of the list of file systems we check for repositories,
since we would never expect any of these file systems to be capable of
containing a repository.

Depends on the new API from
https://bugzilla.gnome.org/show_bug.cgi?id=788927.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1307
Approved by: cgwalters
2017-10-26 01:34:14 +00:00
Colin Walters
d76840d0c1 tree-wide: Use autoptr for OstreeKernelArgs
Much nicer looking.  Prep for more cleanup from
https://github.com/projectatomic/rpm-ostree/pull/1013

Closes: #1302
Approved by: peterbaouoft
2017-10-25 01:48:56 +00:00
Dan Nicholson
63ce86d597 lib/repo: Properly handle NULL homedir when signing commit
Without this, ostree_repo_sign_commit throws a critical message when no
homedir is provided:

(ostree gpg-sign:5034): GLib-GIO-CRITICAL **: g_file_new_for_path: assertion 'path != NULL' failed

Closes: #1305
Approved by: cgwalters
2017-10-24 19:58:07 +00:00
Philip Withnall
886a5d7928 lib/repo-finder-mount: Fix propagating NULL errors
These two code paths tried to propagate errors which had never been set.
Set new errors instead.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1303
Approved by: cgwalters
2017-10-24 16:59:54 +00:00
Colin Walters
b11d1aa3f2 ci: Update centos container to current again
Repo sync issue should be cleared now, and for some reason the previous
container is already GC'd.

Closes: #1304
Approved by: peterbaouoft
2017-10-24 14:59:21 +00:00
Philip Withnall
90680e1b29 lib/repo-finder-avahi: Fix memory corruption of a GVariantIter
A GVariantIter* was being passed to a GVariant format string varargs,
rather than a GVariantIter**. This resulted in memory corruption.

So we can continue to reuse ref_map throughout the function, make it a
GVariantIter* rather than a stack-allocated GVariantIter.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1301
Approved by: cgwalters
2017-10-24 12:55:24 +00:00
Colin Walters
ed15723cd1 lib/commit: Fix hardlink checkout commit with bare-user + mod xattrs
This is more subtle fallout from:
https://github.com/ostreedev/ostree/pull/1170
AKA commit: 8fe4536257

Before, if we found a devino cache hit, we'd use it unconditionally.

Recall that `bare-user` repositories are very special in that they're the only
mode where the on disk state ("physical state") is not the "real" state. The
latter is stored in the `user.ostreemeta` xattr. (`bare-user` repos are also
highly special in that symlinks are regular files physically, but that's not
immediately relevant here).

Since we now have `bare-user-only` for the "pure unprivileged container" case,
`bare-user` should just be used for "OS builds" which have nonzero uids (and
possibly SELinux labels etc.)

In an experimental tool I'm writing "skopeo2ostree" which imports OCI images
into refs, then squashes them together into a single final commit, we lost the
the `81` group ID for `/usr/libexec/dbus-1/dbus-daemon-launch-helper`.

This happened because the commit code was loading the "physical" disk state,
where the uid/gid are zero because that's the uid I happened to be using. We
didn't just directly do the link speedup because I was using `--selinux-policy`
which caused the xattrs to change, which caused us to re-commit objects from the
physical state.

The unit test I added actually doesn't quite trigger this, but I left
it because "why not".  Really testing this requires the installed test
which uses SELinux policy from `/`.

The behavior without this fix looks like:

```
-00755 0 0     12 { [(b'user.ostreemeta', [byte 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x51, 0x00, 0x00, 0x81, 0xed]), (b'security.selinux', b'system_u:object_r:lib_t:s0')] } /usr/lib/dbus-daemon-helper
```

which was obviously totally broken - we shouldn't be picking up the
`user.ostreemeta` xattr and actually committing it of course.

Closes: #1297
Approved by: jlebon
2017-10-23 17:02:28 +00:00
Philip Withnall
4c0f67be0c lib/sysroot: Expand docs for ostree_sysroot_get_repo()
Add some missing annotations and clarify that it always returns an open
repository on success.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1300
Approved by: cgwalters
2017-10-23 16:17:20 +00:00
Colin Walters
795a953a36 core: Add standard SOURCE_TITLE metadata key
This is a freeform string useful to track/display when a commit is "derived"
from some other format.  For example, in the rpm-ostree test we make a
`vmcheck` ref that conceptually overlays the default ref like
`fedora-atomic:fedora/26/x86_64/atomic-host`.

My current patch sets the source title to e.g.
"Dev overlay on fedora-atomic:fedora/26/x86_64/atomic-host".

Another case I'm working on now is importing OCI images to use
as host images.  For that case, the source title is
With this patch we could then set the original OCI image name + tag
as the source name, like:
"oci:cgwalters/demo-custom-fedora-atomic-host:26".

Closes: #1296
Approved by: jlebon
2017-10-23 14:19:41 +00:00