4007 Commits

Author SHA1 Message Date
Matthew Leeds
7dc3e45b3a lib/prune: Don't modify dirent->d_name in place
Currently when I run `ostree prune` it hits a seg fault when the
hash_func is used (in this case g_str_hash) from the call stack
_ostree_repo_prune_tmp() -> g_hash_table_contains() ->
g_hash_table_lookup_node(). So the key, in this case dent->d_name, must
be corrupt in some way.

glnx_dirfd_iterator_next_dent() uses readdir() to get the dirent struct.
And according to the man page for readdir(3), "POSIX.1 explicitly notes
that this field should not be used as an lvalue" (in reference to
d_name). So this commit avoids modifying d_name in place and copies it
instead. This seems to avoid the seg fault.

Closes: #1627
Approved by: jlebon
2018-06-15 19:01:46 +00:00
Alexander Larsson
677e181025 ostree_repo_resolve_rev: Resolve refs set in the transaction
This allows you to get at the current commit for a ref pending in the transaction.

Closes: #1624
Approved by: cgwalters
2018-06-14 17:58:47 +00:00
Philip Withnall
2d2f218669 lib/repo-commit: Delay propagation of errors from abort_transaction()
If there’s a problem while aborting a transaction, store the error but
don’t report it until the end of the function — do a best effort at
clearing the rest of the transaction state first (since most of it
cannot fail).

If cleanup_tmpdir() fails (which, arguably, should not be a
showstopper), this allows a caller to recover and start a new
transaction in future.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1626
Approved by: jlebon
2018-06-14 17:13:43 +00:00
Umang Jain
31809d32f2 lib/repo: Add min-free-space-size option
Similar to min-free-space-percent but it supports specific sizes
(in MB, GB or TB). Also, making min-free-space-percent and -size
mutually exclusive.

min-free-space-percent does not give a fine tuning of the free disk
space that a user might decide to keep. It can translate to very large
size (e.g. 1% = ~10GB on 1TB HDD) or very small (e.g. 1% = ~330MB on 32GB
system like Endless devices). Hence, it makes sense to introduce a config
option to honor specific size as per the user.

Closes: #1616
Approved by: jlebon
2018-06-13 18:57:37 +00:00
Matthew Leeds
c767f7b739 admin: Fix list of subcommands in help and manpage
This adds subcommands that were missing from the ostree-admin man page,
and makes cosmetic fixes there and in the --help output to ensure
alphabetical order and remove trailing whitespace.

Closes: #1621
Approved by: jlebon
2018-06-12 14:36:24 +00:00
Jonathan Lebon
ecdebeb20e switchroot: Allow letting ostree-prepare-root mount /var
In some scenarios, it might make sense to let `ostree-prepare-root` do
the `/var` mount from the state root as before. For example, one may
want to do some system configuration before the switch root. This of
course comes at the expense of supporting `/var` as a mount point in
`/etc/fstab`.

Closes: #1617
Approved by: cgwalters
2018-06-07 21:41:32 +00:00
Matthew Leeds
6f3b5620de tests: Add test for including ostree.h
Closes: #1615
Approved by: jlebon
2018-06-07 17:28:49 +00:00
Matthew Leeds
6ea21696a3 Fix building against old glib versions
We need to include libglnx.h in places where ostree-autocleanups.h is
included, so that we get backports of G_DEFINE_AUTOPTR_CLEANUP_FUNC and
friends.

Closes: #1615
Approved by: jlebon
2018-06-07 17:28:49 +00:00
Matthew Leeds
1d830c1792 Revert "lib: Fix building against old glib versions"
This reverts commit f1d9196076d4aea1f64e0d2cbd17bfa2891b8c4c.

Since libglnx.h does not get installed, it can't be included in
ostree-autocleanups.h, which is included by ostree.h.

Closes: #1615
Approved by: jlebon
2018-06-07 17:28:49 +00:00
Matthew Leeds
8fbf19c9f5 Make P2P API public (no longer experimental)
Currently the API that allows P2P operations (e.g. pulling an ostree ref
from a LAN or USB source) is hidden behind the configure flag
--enable-experimental-api. This commit makes the API public and makes
that flag essentially a no-op (leaving it in place in case we want to
use it again in the future). The P2P API has been tested over the last
several months and proven to work.

This means that since we're no longer using the "experimental" feature
flag, P2P builds of Flatpak will fail when using versions of OSTree from
this commit onwards, until Flatpak is patched in the near future. If you
want to build Flatpak < 0.11.8 with P2P enabled and link against OSTree
2018.6, you'll have to patch Flatpak.  However, since Flatpak won't yet
have a hard dependency on OSTree 2018.6, it needs a new way to determine
if the P2P API in OSTree is available, so this commit adds a "p2p"
feature flag. This way the feature set is more semantically correct than
if we had continued to use the "experimental" feature flag.

In addition to making the P2P API public, this commit makes the P2P unit
tests run by default, removes the f27-experimental CI instance that's no
longer needed, changes a few man pages to reflect the changes, and
updates the bash completion script to accept the new commands and
options.

Closes: #1596
Approved by: cgwalters
2018-06-04 19:20:10 +00:00
Matthew Leeds
f1d9196076 lib: Fix building against old glib versions
This commit includes libglnx.h in ostree-autocleanups.h, so we get the
g_autoptr backports wherever they're needed. Also, remove the "#include
libglnx.h" lines elsewhere that are no longer needed.

Closes: #1596
Approved by: cgwalters
2018-06-04 19:20:09 +00:00
Jan Tojnar
f200efdb8a tests: Fix locale detection
When a locale with C.utf8 in its name (e.g. es_EC.utf8) was installed
on a system, the C.utf8 locale was chosen, even when it was not available.

This patch fixes the grep pattern to match whole lines returned by locale -a.

See: #1592

Closes: #1611
Approved by: cgwalters
2018-06-03 21:25:07 +00:00
Matthew Leeds
bf3525adcb ci: Use master branch as flatpak tag
Closes: #1607
Approved by: jlebon
2018-05-31 19:27:04 +00:00
Matthew Leeds
6a7620b58d ci: Bump flatpak build to F28
Closes: #1607
Approved by: jlebon
2018-05-31 19:27:04 +00:00
Philip Withnall
bf1af263d6 lib/repo-pull: Support retries for delta superblocks
Use the recently introduced architecture for retrying network requests
on transient failure to do the same for delta superblock requests, now
that they’re queued.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1600
Approved by: jlebon
2018-05-30 19:57:13 +00:00
Philip Withnall
f342e66c11 lib/repo-pull: Support queuing delta superblock requests
Just like all the other requests made for delta parts and objects by the
pull code, use a queue for delta superblocks. Currently this doesn’t do
any prioritisation or retries after transient failures, but it could do
in future.

This means that delta superblocks are now subject to the parallel
request limit in the fetcher, which was a problem highlighted here:
https://github.com/ostreedev/ostree/pull/1453#discussion_r168321706.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1600
Approved by: jlebon
2018-05-30 19:57:13 +00:00
Philip Withnall
197644c406 lib/fetcher: Factor out HTTP status code handling from soup and curl
Use the same G_IO_ERROR_* values for HTTP status codes in both fetchers.
The libsoup fetcher still handles a few more internal error codes than
the libcurl one; this could be built on in future.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1594
Approved by: jlebon
2018-05-30 16:23:57 +00:00
Philip Withnall
78f40136db lib/repo-pull: Add some missing assertions for progress statistics
Various of the counters already have assertions like this; add some more
for total paranoia.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1594
Approved by: jlebon
2018-05-30 16:23:57 +00:00
Philip Withnall
c9619a4904 tests: Test pull behaviour when network timeouts occur
Extend test-pull-repeated.sh to test error 408 as well as error 500, to
ensure that the new retry-on-network-timeout code in ostree-repo-pull.c
correctly retries.

Rather than the 200 iterations needed for the error 500 tests, only do 5
iterations. The pull code internally does 5 retries (by default), which
means a full iteration count of 25. That seems to be sufficient to make
the tests reliably pass, in my testing — we can always bump it up to 200
/ 5 = 40 in future if needed (to put it in parity with the error 500
tests).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1594
Approved by: jlebon
2018-05-30 16:23:57 +00:00
Philip Withnall
97c348d65b ostree/trivial-httpd: Add --random-408s command line option
This is exactly like the --random-500s option, except that it will cause
error 408 (request timeout) to be returned, rather than error 500
(internal server error).

This will be used in a following commit to test pull behaviour when
timeouts occur.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1594
Approved by: jlebon
2018-05-30 16:23:57 +00:00
Philip Withnall
224f3cdd24 lib/fetcher-soup: Map more SoupStatus codes to known GIOErrors
This allows the retry code in ostree-repo-pull.c to recover from (for
example) timeouts at the libsoup layer in the stack, as well as from the
GSocket layer in the stack.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1594
Approved by: jlebon
2018-05-30 16:23:57 +00:00
Philip Withnall
938055392f lib/repo-pull: Support retrying requests on transient network errors
Allow network requests to be re-queued if they failed with a transient
error, such as a socket timeout. Retry each request up to a limit
(default: 5), and only then fail the entire pull and propagate the error
to the caller.

Add a new ostree_repo_pull_with_options() option, n-network-retries, to
control the number of retries (including setting it back to the old
default of 0, if the caller wants).

Currently, retries are not supported for FetchDeltaSuperData requests,
as they are not queued. Once they are queued, adding support for retries
should be trivial. A FIXME comment has been left for this.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1594
Approved by: jlebon
2018-05-30 16:23:57 +00:00
Matthew Leeds
f31087137e Fix building P2P code against old glib versions
When building the OstreeBloom code against old versions of glib, we have
to have the libglnx headers included so that it defines
G_DEFINE_AUTOPTR_CLEANUP_FUNC and friends for us.

This is similarly true for test-repo-finder-mount.c which indirectly
includes ostree-autocleanups.h.

Closes: #1605
Approved by: cgwalters
2018-05-30 14:19:25 +00:00
Matthew Leeds
6ec19a6953 find-remotes: Fix unused variable warnings
Closes: #1605
Approved by: cgwalters
2018-05-30 14:19:25 +00:00
Matthew Leeds
7412e65f7d Allow compiling OstreeRepoFinderAvahi without libsoup
Closes: #1605
Approved by: cgwalters
2018-05-30 14:19:25 +00:00
Matthew Leeds
2b32de38cf Allow compiling with experimental API but without curl/soup
This commit rearranges a few things in ostree-repo-pull.c so that OSTree
will successfully compile with experimental API enabled and without
libsoup, libcurl, or avahi:

./autogen.sh --enable-experimental-api --without-soup --without-curl
--without-avahi

This is accomplished with two sets of changes:

1. Move ostree_repo_resolve_keyring_for_collection() so it can be used
even without libsoup or libcurl.
2. Add stub functions for ostree_repo_find_remotes_async() and
ostree_repo_pull_from_remotes_async(), and their _finish() counterparts,
so they return an error when libsoup or libcurl isn't available.

Closes: #1605
Approved by: cgwalters
2018-05-30 14:19:25 +00:00
Jonathan Lebon
4f8944b719 .gitmodules: Update URL for libglnx
Same thing as https://github.com/projectatomic/rpm-ostree/pull/1374.
Otherwise we get an annoying "warning: redirecting to" message.

Update submodule: libglnx

Closes: #1604
Approved by: cgwalters
2018-05-30 13:48:48 +00:00
Matthew Leeds
ee0d9bbd5b Update libglnx to get g_autoptr backports
Update submodule: libglnx

Closes: #1603
Approved by: jlebon
2018-05-29 19:24:20 +00:00
Jonathan Lebon
517140f110 ci: Drop gpgme and libgcrypt hacks
These shouldn't be needed anymore in F28.

Closes: #1601
Approved by: cgwalters
2018-05-29 19:24:08 +00:00
Jonathan Lebon
0f124e2584 ci: Bump to f28, except flatpak
We successfully switched to F28 in rpm-ostree. Time to follow suit here.
We exclude flatpak for now since we're still hitting test issues there
on F28.

Closes: #1601
Approved by: cgwalters
2018-05-29 19:24:08 +00:00
Jonathan Lebon
f6e4543a24 ci: Stop layering oci-kvm-hook
Since we're explicitly using `--device /dev/kvm`, there's no need to
install `oci-kvm-hook`.

Closes: #1601
Approved by: cgwalters
2018-05-29 19:24:08 +00:00
Philip Withnall
23d4beead1 lib/repo-pull: Factor out enqueue function for FetchStaticDeltaData
This introduces no functional changes, but will make upcoming support
for retrying downloads easier to add.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall
dc1fde704d lib/repo-pull: Factor out enqueue function for FetchObjectData
This introduces no functional changes, but will make upcoming support
for retrying downloads easier to add.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall
8b8952ebee lib/repo-pull: Factor out enqueue function for ScanObjectQueueData
This introduces no functional changes, but will make upcoming support
for retrying downloads easier to add.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall
6d21f83813 lib/repo-pull: Rename a variable
Rename from `fdata` to `fetch_data` to clarify things and make it
consistent with other similar functionality in the file.

This introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall
fc10354404 lib/repo-pull: Factor out free function for FetchDeltaSuperData
This introduces no functional changes, but does make the code a little
cleaner.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall
66777b1b75 lib/repo-pull: Use GCC ?: ternary shortcut where appropriate
This introduces no functional changes; just makes the code a bit shorter
in a few places.

https://gcc.gnu.org/onlinedocs/gcc/Conditionals.html

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall
8593357902 lib/repo-pull: Use values from struct in enqueue_one_object_request()
This introduces no functional changes, but will make some upcoming
refactoring a little easier.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Alexander Larsson
5f82503a01 fsck: Add --all to print all corrupted object
Stopping on the first error is nice if you just want to know if everything is ok, but
if you want to figure out all that is wrong its nice to be able to continue and
print all corruptions.

Closes: #1591
Approved by: cgwalters
2018-05-24 21:00:52 +00:00
Alexander Larsson
ef86abe228 fsck: Include the commits that contain the corruption in errors
This makes it easier to figure out what the corruption affects.

Closes: #1591
Approved by: cgwalters
2018-05-24 21:00:52 +00:00
Matthew Leeds
0d53e5d309 bash-completion: Don't add a space after files and directories
Currently if ostree is completing a file or directory for you it adds a
space to the end, but this is an inconvenience when it's an intermediate
directory in a tree. It's better to let the user add the space after the
final directory, so this commit changes the bash completion to avoid
adding a space when completing files or directories.

Closes: #1598
Approved by: jlebon
2018-05-24 12:59:49 +00:00
Colin Walters
9131d8a4cc lib/sysroot: Add wrapper API to prune system repository
The initial motivation for this is that the "staging" code currently
didn't rewrite the deployment refs, meaning that the staged commit
could be pruned.

Hence first, this new API ensures that deployments also
hold a strong ref to their commit, without relying on the magical
"deployment refs" that we inject.  That has always been a weird
artifact of the strict layering separation between OstreeSysroot
and OstreeRepo.

I also plan to change rpm-ostree to start using this API to
hold references to base layers for client-side layering; it also
today generates various refs.

That said, if we still want to support multiple processes
writing to a single repo (as happens on EndlessOS today) we
still need to write refs; perhaps later we could add a concept
of "generators" or something that create refs based on whatever
logic?

Another minor thing this fixes is that we had a printf inside
the library; this propagates the pruned data to the higher level
which can log however it likes.

Closes: #1566
Approved by: jlebon
2018-05-24 12:56:11 +00:00
Colin Walters
371081d123 lib: Add a public helper method for pruning to find all ref'd commits
Prep for reworking how we do sysroot cleanup.  We're going to
start doing more lowlevel pruning work there, and I wanted to avoid
duplicating the ref enumeration.

Closes: #1566
Approved by: jlebon
2018-05-24 12:56:11 +00:00
Andrea Galbusera
ce2995e1dc u-boot: add support for devicetree
Likewise the corresponding support for syslinux introduced by commit
c5112c25e4519835c4cd53f4350c1b2f2a477746, this one enables writing devicetree
filename into the uEnv.txt environment file for u-boot.

Since u-boot does not strictly defines variable names, here 'fdt_file' was
chosen as it appear to be one the most frequently adopted names in u-boot
default environments. Outer boot logic should of course comply with this choice
and use $fdt_file as the device tree file name to pass to boot commands.

This was tested on a custom board booting with u-boot.

Closes: #1590
Approved by: cgwalters
2018-05-23 12:25:38 +00:00
Arnaud Rebillout
4b330feed6 docs: var/local -> var/usrlocal
Signed-off-by: Arnaud Rebillout <arnaud.rebillout@collabora.com>

Closes: #1588
Approved by: jlebon
2018-05-15 16:01:05 +00:00
Jonathan Lebon
7a1ac8ef15 ci: Add .papr-ex.yaml
I'd like to start working towards migrating the OSTree CI towards the
new PAPR model running in PACI. In order to take it for a test drive
(pun intended) without yet replacing our current CI, the new PAPR now
knows to look for a `.papr-ex.yaml` file. So as to not conflict with the
current PAPR, no GitHub statuses will be output other than a
`required-ex` label, which plays the same role as `required`. For more
information, see:

85b1d0a91e

The `.papr-ex.yaml` version is essentially the same except for:

1. the `FAH27-insttests` suite runs directly in the scheduled container
   (notice the new `kvm: true` key)
2. the `host:` based contexts (rpm-ostree and flatpak) are missing since
   we don't support provisioning a host just yet

Keeping them in sync could get annoying, though building confidence in
the new approach will allow us to migrate faster.

Closes: #1587
Approved by: cgwalters
2018-05-11 20:26:09 +00:00
Colin Walters
79c2d76e9f Post-release version bump
Closes: #1586
Approved by: jlebon
2018-05-11 15:52:05 +00:00
Colin Walters
986f176e7c Release 2018.5
Lots of stuff queued up.

Closes: #1585
Approved by: jlebon
v2018.5
2018-05-11 13:06:19 +00:00
Colin Walters
6873650cae lib/sysroot: Add OSTREE_EX_STAGE_DEPLOYMENTS environment variable
I feel like I'm drowning in a pile of experimental-but-almost-stable
features...

Anyways, since we made the feature opt-in in rpm-ostree in
https://github.com/projectatomic/rpm-ostree/pull/1352
let's mirror that a bit here with an environment variable so people
can play with it more easily.

The tests needed some tweaks; specifically we need to reload the
status fact after making changes.  I'm still a bit uncertain
about the Ansible-as-tests.

But we add an upgrade test that uses the new environment variable.

Closes: #1583
Approved by: jlebon
2018-05-11 12:39:17 +00:00
Colin Walters
605d313206 tests/installed: Disable all rpmmd repos
For the same reason we do in the rpm-ostree tests.  This also
made sure the test run worked when I was offline on a plane.

Closes: #1583
Approved by: jlebon
2018-05-11 12:39:17 +00:00