Commit Graph

899 Commits

Author SHA1 Message Date
Dan Nicholson
b33a4e9b1e remote-add: Add --force option to add or replace remote
This uses the OSTREE_REPO_REMOTE_CHANGE_REPLACE operation to add a
remote or replace an existing one. This is roughly the opposite of
--if-not-exists and will raise an error if both options are passed.

Closes: #1166
Approved by: cgwalters
2019-02-08 14:36:41 +00:00
Dan Nicholson
15ba2007a1 repo: Add OstreeRepoRemoteChange replace operation
Add the OSTREE_REPO_REMOTE_CHANGE_REPLACE operation to the
OstreeRepoRemoteChange enum. This operation will add a remote or replace
an existing one. It respects the location of the remote configuration
file when replacing and the remotes config dir settings when adding a
new remote.

Closes: #1166
Approved by: cgwalters
2019-02-08 14:36:41 +00:00
Rafael Fonseca
6e8377cbd1 admin/status: test GPG signature check
Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>

Closes: #1797
Approved by: cgwalters
2019-01-08 14:47:08 +00:00
Umang Jain
963395217a lib/repo-commit: Relax min-free-space check in prepare_transaction()
We want a case where we can disable the min-free-space check. Initially,
it felt like to add a OSTREE_REPO_PULL_FLAGS_DISABLE_FREE_SPACE_CHECK but
the problem is prepare_transaction() does not have a OstreeRepoPullFlags
parameter which we can parse right here. On top of it, prepare_transaction()
enforces min-free-space check and won't let the transaction proceed if
the check failed.

This is pretty bad in conjunction with "inherit-transaction" as what
Flatpak uses. There is no way to disable this check unless we remove
it altogether from prepare_transaction.

This issue came out to light when flatpak wasn't able to write metadata
after fetching from remote:

[uajain@localhost ~]$ flatpak remote-info flathub org.kde.Platform//5.9
error: min-free-space-size 500MB would be exceeded

Metadata objects helps in housekeeping and restricting them means
restricting crucial UX (like search, new updates) functionalities
in clients like gnome-software. The error banners originated from
these issues are also abrupt and not much helpful to the user. This
is the specific instance of the issue this patches tries to address.

See https://github.com/flatpak/flatpak/issues/2139 for discussion.

Closes: #1779
Approved by: mwleeds
2018-12-10 20:04:17 +00:00
Luca Bruno
3ecbdd8197 lib/kargs: allow empty-list arguments
This adds support for empty-list arguments (e.g. `acpi_osi=`), which
are semantically different from simple-keyword arguments.

Ref: https://github.com/projectatomic/rpm-ostree/issues/1706

Closes: #1785
Approved by: cgwalters
2018-12-10 19:39:23 +00:00
Colin Walters
dc2a7de217 lib/commit: Try checksum+hardlink for untrusted local same-uid repos
This mainly helps flatpak for enabling a hardlink-able local pull
during deploy in the --system case.  We assume the files are immutable
when owned by the same uid.

See https://github.com/ostreedev/ostree/issues/1723
and https://github.com/flatpak/flatpak/pull/2342

Closes: #1776
Approved by: uajain
2018-12-04 20:38:41 +00:00
rfairley
05e8c7ef6a lib/repo: Search a list of paths in gpgkeypath for gpg keys
This allows specifying gpgpath as list of
paths that can point to a file or a directory. If a directory path
is given, paths to all regular files in the directory are added
to the remote as gpg ascii keys. If the path is not a directory,
the file is directly added (whether regular file, empty - errors
will be reported later when verifying gpg keys e.g. when pulling).

Adding the gpgkeypath property looks like:

ostree --repo=repo remote add --set=gpgpath="/path/key1.asc,/path/keys.d" R1 https://example.com/some/remote/ostree/repo

Closes #773

Closes: #1773
Approved by: cgwalters
2018-11-21 17:03:10 +00:00
Jonathan Lebon
8eac5be030 lib/commit: Add devino_cache_hits to txn stats
I found this useful while hacking on rpm-ostree but I think it might be
useful enough to upstream. This stat is really helpful for validating
that a pipeline is hitting the devino cache sweet spot.

Closes: #1772
Approved by: cgwalters
2018-11-05 14:08:54 +00:00
Jonathan Lebon
1dcebe8846 bin/pull-local: Add --commit-metadata-only
It might be "local", but e.g. we may be crossing filesystems. So there
are valid use cases for only wanting to pull the commit metadata with
`pull-local`.

Closes: #1769
Approved by: cgwalters
2018-11-01 14:51:09 +00:00
Jonathan Lebon
34a8867749 bin/refs: Disallow creating broken aliases
This is the alias version of #1749. I.e. we want to make sure that one
can't even create an alias which would end up dangling.

See also: https://pagure.io/releng/issue/7891

Closes: #1768
Approved by: sinnykumari
2018-10-31 12:32:30 +00:00
Giuseppe Scrivano
51752baf0e checkout: honor opaque checkouts
if a file ".wh..wh..opq" is present in a directory, delete anything
from lower layers that is already in that directory.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #1486
Approved by: cgwalters
2018-10-26 18:26:51 +00:00
Umang Jain
f3eba6bcec Release 2018.9
Closes: #1761
Approved by: cgwalters
2018-10-24 18:46:39 +00:00
Jonathan Lebon
ac1a919ffd boot: Add ostree-finalize-staged.path
Rather than manually starting the `ostree-finalize-staged.service` unit,
we can leverage systemd's path units for this. It fits quite nicely too,
given that we already have a path we drop iif we have a staged
deployment.

To give some time for the preset to make it to systems, we don't yet
drop the explicit call to `systemctl start`. Though we do make it
conditional based on a DEBUG env var so that we can actually test it in
CI for now. Once we're sure this has propagated, we can drop the
`systemctl start` path and the env var together.

Closes: #1740
Approved by: cgwalters
2018-10-23 13:10:49 +00:00
Colin Walters
9367a1befe checkout: Support --union-identical and --force-copy{,--zerosized}
Actually testing the patch to add `--force-copy-zerosized` to
rpm-ostree tripped over the fact that it uses `--union-identical`,
and we just hit an assertion failure with that combination.

Fix this by copying over the logic we have for the hardlink case.

Closes: #1753
Approved by: jlebon
2018-10-11 20:49:54 +00:00
Colin Walters
673cacd633 repo: Add a checkout option to not hardlink zero-sized files
In rpm-ostree we've hit a few cases where hardlinking zero-sized
files causes us problems.  The most prominent is lock files in
`/usr/etc`, such as `/usr/etc/selinux/semanage.LOCK`.  If there
are two zero-sized lock files to grab, but they're hardlinked,
then locking will fail.

Another case here is if one is using ostree inside a container
and don't have access to FUSE (i.e. `rofiles-fuse`), then the
ostree hardlinking can cause files that aren't ordinarily hardlinked
to become so, and mutation of one mutates all.  An example where
this is concerning is Python `__init__.py` files.

Now, these lock files should clearly not be in the tree to begin
with, but - we're not gaining a huge amount by hardlinking these
files either, so let's add an option to disable it.

Closes: #1752
Approved by: jlebon
2018-10-11 16:32:25 +00:00
Sinny Kumari
c70526841e src/ostree: Don't delete refs having aliases
Deleting a ref with aliases makes them dangling. In such
cases, display an error message to the user.

Fixes #1597

Signed-off-by: Sinny Kumari <sinny@redhat.com>

Closes: #1749
Approved by: cgwalters
2018-10-11 13:41:32 +00:00
Colin Walters
2c55bc6997 Only verify OSTREE_MAX_METADATA_SIZE for HTTP fetches
There are use cases for libostree as a local content store
for content derived or delivered via other mechanisms (e.g. OCI
images, RPMs, etc.).  rpm-ostree today imports RPMs into OSTree
branches, and puts the RPM header value as commit metadata.
Some of these can be quite large because the header includes
permissions for each file.  Similarly, some OCI metadata is large.

Since there's no security issues with this, support committing
such content.

We still by default limit the size of metadata fetches, although
for good measure we make this configurable too via a new
`max-metadata-size` value.

Closes: https://github.com/ostreedev/ostree/issues/1721

Closes: #1744
Approved by: jlebon
2018-10-01 13:23:50 +00:00
Colin Walters
c141fe610b lib/commit: Don't copy xattrs for metadata objects
Copying the xattrs on metadata objects is wrong in general, we
don't "own" them.  Notably this would fail in the situation of
doing a pull from e.g. a `bare-user` source to a destination
that was on a different mount point (so we couldn't hardlink),
and the source had e.g. a `security.selinux` attribute.

Closes: #1734

Closes: #1736
Approved by: jlebon
2018-09-25 14:49:22 +00:00
Umang Jain
5d2e62affb tests: Update tests for ostree_repo_get_min_free_space_bytes()
https://github.com/ostreedev/ostree/issues/1720

Closes: #1722
Approved by: pwithnall
2018-09-21 15:09:12 +00:00
Umang Jain
a70d2f6731 Add tests for ostree_repo_get_min_free_space_bytes
https://phabricator.endlessm.com/T23694

Closes: #1715
Approved by: cgwalters
2018-09-04 21:31:34 +00:00
Colin Walters
7aa242c34c Release 2018.8
Closes: #1705
Approved by: jlebon
2018-08-22 13:53:24 +00:00
Sinny Kumari
dde3f1c0fb src/ostree: Add --group option to ostree config
Fetching value from a repo config using 'ostree config
get SECTIONNAME.KEYNAME' didn't work in some cases like
when having dots in Group Name entry.
As per Desktop entry file specification, Group Name
may contain all ASCII characters except for [ and ]
and control characters.
Link - https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-1.1.html

Having --group option will help user to clearly specify
Group Name and get desired result.

It also adds test for ostree config get|set and bash
completion for --group option

Fixes https://github.com/ostreedev/ostree/issues/1565

Closes: #1696
Approved by: cgwalters
2018-08-20 14:31:15 +00:00
Dan Nicholson
4e6b13e8b6 repo: Add OSTREE_REPO_TEST_ERROR=invalid-cache env var
Add an invalid-cache test error flag to ensure that the code that checks
for and recovers from a corrupted summary cache is hit. This helps make
sure that the recovery path is actually used without resorting to
G_MESSAGES_DEBUG.

Closes: #1698
Approved by: cgwalters
2018-08-14 13:38:11 +00:00
Dan Nicholson
1b5cb52da2 tests: Test for recovery from corrupted summary cache
Check that recovery from a corrupted summary cache (cached summary
doesn't match cached signature) works.

Closes: #1698
Approved by: cgwalters
2018-08-14 13:38:11 +00:00
Dan Nicholson
ce4eb12ffb tests: Add tests for remote summary update races
There have been subtle bugs in the past when a client pulls while the
remote server is updating the summary. The client may get the old
summary and new signature or vice versa. Add tests to simulate this
behavior to make sure there aren't regressions in the future.

Closes: #1698
Approved by: cgwalters
2018-08-14 13:38:11 +00:00
Jonathan Lebon
786ee6bdec lib/config: Rename change-update-summary to auto-...
Mildly bikeshed, though I find the name `auto-update-summary` to be
easier to grok than `change-update-summary`. I think it's because it can
be read as "verb-verb-noun" rather than "noun-verb-noun".

Closes: #1693
Approved by: mwleeds
2018-08-01 19:59:07 +00:00
Matthew Leeds
6869bada49 config: Add a core/change-update-summary option
This commits adds and implements a boolean repo config option called
"change-update-summary" which updates the summary file every time a ref
changes (additions, updates, and deletions).

The main impetus for this feature is that the `ostree create-usb` and
`flatpak create-usb` commands depend on the repo summary being up to
date. On the command line you can work around this by asking the user to
run `ostree summary --update` but in the case of GNOME Software calling
out to `flatpak create-usb` this wouldn't work because it's running as a
user and the repo is owned by root. That strategy also means flatpak
can't update the repo metadata refs for fear of invalidating the
summary.

Another use case for this relates to LAN updates. Specifically, the
component of eos-updater that generates DNS-SD records advertising ostree
refs depends on the repo summary being up to date.

Since ostree_repo_regenerate_summary() now takes an exclusive lock, this
should be safe to enable. However it's not enabled by default because of
the performance cost, and because it's more useful on clients than
servers (which likely have another mechanism for updating the summary).

Fixes https://github.com/ostreedev/ostree/issues/1664

Closes: #1681
Approved by: jlebon
2018-07-30 17:19:12 +00:00
Umang Jain
21318bbc1f Release 2018.7
Request via flatpak: mainly to port min-free-space-size

Closes: #1683
Approved by: cgwalters
2018-07-20 19:43:18 +00:00
Colin Walters
7468600029 deploy: Retain staged by default
For `rpm-ostree ex livefs` we have a use case of pushing a rollback
deployment.  There's no reason this should require deleting the staged
deployment (and doing so actually breaks livefs which tries to access
it as a data source).

I was initially very conservative here, but I think it ends up
being fairly easy to retain the staged deployment.  We need to handle
two cases:

First, when the staged is *intentionally* deleted; here, we just need
to unlink the `/run` file, and then everything will be sync'd up after
reloading.

Second, (as in the livefs case) where we're retaining it,
e.g. adding a deployment to the end.  What I realized here is that
we can have the code keep `new_deployments` as view without staged,
and then when we do the final reload we'll end up re-reading it from
disk anyways.

Closes: #1672
Approved by: jlebon
2018-07-06 15:23:52 +00:00
Robert Fairley
10c2fc33f6 tests: Run network retries test for many retries directly
This runs a test involving many retries for the --network-retries
option directly rather than inside a conditional statement, so that
the command does not silently fail and allow the test to continue
running.

Closes: #1673
Approved by: jlebon
2018-07-05 20:11:42 +00:00
Robert Fairley
7baf167881 ostree/pull: Add network-retries command line option
This exposes a way to specify from the command line the number
of times to retry each download after a network error. If a negative
value is given, then the default number of retries (5) is used. If 0
is given, then errors are returned without retrying.

closes #1659

Closes: #1669
Approved by: jlebon
2018-07-05 17:59:09 +00:00
Robert Fairley
4f096c8f11 tests: Move assert_fail function to tests/libtest.sh
This moves the assert_fail function definition which was defined
and called in tests/test-remote-headers.sh. Done in preparation
for use of the assert_fail function in other test files.

Closes: #1669
Approved by: jlebon
2018-07-05 17:59:09 +00:00
Jonathan Lebon
2cb2571127 tests/installed: Add NOTE when re-using RPMs
One gotcha here is that we don't invalidate the RPMs if we're not
sitting on the same commit anymore. Shouldn't be too hard to fix, though
let's at least make a note of it for now.

Closes: #1668
Approved by: cgwalters
2018-07-04 16:06:38 +00:00
Colin Walters
61ba4e7e5a tests/installed: Add var-mount.yml destructive test
Closes: #1668
Approved by: cgwalters
2018-07-04 16:06:38 +00:00
William Manley
488365f9bf OstreeMutableTree: Invalidate parent contents checksum when metadata changes
This bug has existed before the previous commit, but thanks to the previous
commit it is now easy to fix.

Closes: #1655
Approved by: cgwalters
2018-06-29 21:31:08 +00:00
Colin Walters
7ead3c1aa8 sysroot: Reject attempts to pin the staged deployment
From https://github.com/projectatomic/rpm-ostree/pull/1434#discussion_r198936674

To support it we'd have to actually write it to disk, which...let's
not try that right now.

Closes: #1660
Approved by: jlebon
2018-06-29 01:52:30 +00:00
William Manley
bab3b2bd4c tests: Save corefiles back to tests/ directory if one exists
Makes it easier to debug failures from the tests.

Closes: #1657
Approved by: cgwalters
2018-06-28 15:07:14 +00:00
Javier Martinez Canillas
9f48e212a3 deploy: Change BootLoaderSpec filenames so they can be used for sorting
Currently the BLS snippets are named ostree-$ID-$VARIANT_ID-$index.conf,
but the BLS config files are actually sorted by using the version field
which is the inverse of the index.

In most places, _ostree_sysroot_read_boot_loader_configs() is used to
get the BLS files and this function already returns them sorted by the
version field. The only place where the index trailing number is used is
in the ostree-grub-generator script that lists the BLS files to populate
the grub config file.

But for some bootloaders the BLS filename is the criteria for sorting by
taking the filename as a string version. So on these bootloaders the BLS
entries will be listed in the reverse order.

To avoid that, change the BLS snippets filename to have the version field
instead of the index and also to have the version before deployment name.

Make the filenames to be of the form ostree-$version-$ID-$VARIANT_ID.conf
so the version is before the deployment name.

Signed-off-by: Javier Martinez Canillas <javierm@redhat.com>

Closes: #1654
Approved by: cgwalters
2018-06-27 18:08:28 +00:00
Jonathan Lebon
8dff04601b tests/installed: Wait a bit more for http.server
And also print out the output if it still didn't start up in case there
are error messages hidden in there.

This should hopefully help with diagnosing the flakes we've been seeing
in starting it up.

Closes: #1652
Approved by: cgwalters
2018-06-27 13:29:55 +00:00
William Manley
5d031ae78b Add test for composing trees in different ways
In preparation for adding `ostree commit` optimisations.

Closes: #1645
Approved by: jlebon
2018-06-25 16:22:22 +00:00
Umang Jain
8d97b55241 tests: Add tests for space checks during deltas codepath
Closes: #1614
Approved by: jlebon
2018-06-22 21:01:56 +00:00
Colin Walters
31a356dca9 Release 2018.6 2018-06-19 15:48:14 -04:00
Colin Walters
0f88a2a72d tests/installed: Add a free-space success path test
We implicitly test the success percent path a lot, but not the
absolute path.

Closes: #1632
Approved by: jlebon
2018-06-19 18:29:31 +00:00
Dan Nicholson
ce58307757 deploy: Delete .updated file from /etc and /var on new deployments
Systemd units using ConditionNeedsUpdate run if the mtime of .updated in
the specified directory is newer than /usr. Since /usr has an mtime of
0, there's no way to have an older .updated file. Systemd units
typically specify ConditionNeedsUpdate=/etc or ConditionNeedsUpdate=/var
to support stateless systems like ostree.

Remove the file from the new deployment's /etc and the OS's /var
regardless of where they came from to ensure that these systemd units
run when booting new deployments. This will provide a method to run
services only on upgrade.

Closes: #1628
https://bugzilla.gnome.org/show_bug.cgi?id=752950

Closes: #1631
Approved by: cgwalters
2018-06-18 13:21:52 +00:00
Umang Jain
31809d32f2 lib/repo: Add min-free-space-size option
Similar to min-free-space-percent but it supports specific sizes
(in MB, GB or TB). Also, making min-free-space-percent and -size
mutually exclusive.

min-free-space-percent does not give a fine tuning of the free disk
space that a user might decide to keep. It can translate to very large
size (e.g. 1% = ~10GB on 1TB HDD) or very small (e.g. 1% = ~330MB on 32GB
system like Endless devices). Hence, it makes sense to introduce a config
option to honor specific size as per the user.

Closes: #1616
Approved by: jlebon
2018-06-13 18:57:37 +00:00
Matthew Leeds
6f3b5620de tests: Add test for including ostree.h
Closes: #1615
Approved by: jlebon
2018-06-07 17:28:49 +00:00
Matthew Leeds
6ea21696a3 Fix building against old glib versions
We need to include libglnx.h in places where ostree-autocleanups.h is
included, so that we get backports of G_DEFINE_AUTOPTR_CLEANUP_FUNC and
friends.

Closes: #1615
Approved by: jlebon
2018-06-07 17:28:49 +00:00
Matthew Leeds
1d830c1792 Revert "lib: Fix building against old glib versions"
This reverts commit f1d9196076.

Since libglnx.h does not get installed, it can't be included in
ostree-autocleanups.h, which is included by ostree.h.

Closes: #1615
Approved by: jlebon
2018-06-07 17:28:49 +00:00
Matthew Leeds
f1d9196076 lib: Fix building against old glib versions
This commit includes libglnx.h in ostree-autocleanups.h, so we get the
g_autoptr backports wherever they're needed. Also, remove the "#include
libglnx.h" lines elsewhere that are no longer needed.

Closes: #1596
Approved by: cgwalters
2018-06-04 19:20:09 +00:00
Jan Tojnar
f200efdb8a tests: Fix locale detection
When a locale with C.utf8 in its name (e.g. es_EC.utf8) was installed
on a system, the C.utf8 locale was chosen, even when it was not available.

This patch fixes the grep pattern to match whole lines returned by locale -a.

See: #1592

Closes: #1611
Approved by: cgwalters
2018-06-03 21:25:07 +00:00
Philip Withnall
c9619a4904 tests: Test pull behaviour when network timeouts occur
Extend test-pull-repeated.sh to test error 408 as well as error 500, to
ensure that the new retry-on-network-timeout code in ostree-repo-pull.c
correctly retries.

Rather than the 200 iterations needed for the error 500 tests, only do 5
iterations. The pull code internally does 5 retries (by default), which
means a full iteration count of 25. That seems to be sufficient to make
the tests reliably pass, in my testing — we can always bump it up to 200
/ 5 = 40 in future if needed (to put it in parity with the error 500
tests).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1594
Approved by: jlebon
2018-05-30 16:23:57 +00:00
Matthew Leeds
f31087137e Fix building P2P code against old glib versions
When building the OstreeBloom code against old versions of glib, we have
to have the libglnx headers included so that it defines
G_DEFINE_AUTOPTR_CLEANUP_FUNC and friends for us.

This is similarly true for test-repo-finder-mount.c which indirectly
includes ostree-autocleanups.h.

Closes: #1605
Approved by: cgwalters
2018-05-30 14:19:25 +00:00
Alexander Larsson
5f82503a01 fsck: Add --all to print all corrupted object
Stopping on the first error is nice if you just want to know if everything is ok, but
if you want to figure out all that is wrong its nice to be able to continue and
print all corruptions.

Closes: #1591
Approved by: cgwalters
2018-05-24 21:00:52 +00:00
Alexander Larsson
ef86abe228 fsck: Include the commits that contain the corruption in errors
This makes it easier to figure out what the corruption affects.

Closes: #1591
Approved by: cgwalters
2018-05-24 21:00:52 +00:00
Colin Walters
986f176e7c Release 2018.5
Lots of stuff queued up.

Closes: #1585
Approved by: jlebon
2018-05-11 13:06:19 +00:00
Colin Walters
6873650cae lib/sysroot: Add OSTREE_EX_STAGE_DEPLOYMENTS environment variable
I feel like I'm drowning in a pile of experimental-but-almost-stable
features...

Anyways, since we made the feature opt-in in rpm-ostree in
https://github.com/projectatomic/rpm-ostree/pull/1352
let's mirror that a bit here with an environment variable so people
can play with it more easily.

The tests needed some tweaks; specifically we need to reload the
status fact after making changes.  I'm still a bit uncertain
about the Ansible-as-tests.

But we add an upgrade test that uses the new environment variable.

Closes: #1583
Approved by: jlebon
2018-05-11 12:39:17 +00:00
Colin Walters
605d313206 tests/installed: Disable all rpmmd repos
For the same reason we do in the rpm-ostree tests.  This also
made sure the test run worked when I was offline on a plane.

Closes: #1583
Approved by: jlebon
2018-05-11 12:39:17 +00:00
Colin Walters
a3295ee584 ci: Drop str hotpatch
The change we want is in the current Fedora repos, and git master
is broken:

> qemu-system-x86_64: -vnc :1: Failed to start VNC server: Failed to bind socket: Address already in use

 https://pagure.io/standard-test-roles/pull-request/186#comment-52440

Closes: #1582
Approved by: jlebon
2018-05-08 12:57:35 +00:00
Colin Walters
579faf92fd tests: Fix installed tests more
OK so I noticed that something was failing and we were missing
`set -xeuo pipefail` in our shells.  That of course revealed
the ansible tests didn't actually work - my only defense
here is spending so much time fighting to get it through CI
and trying something new.

Anyways, to make the staged-deploy tests work we need a task
that actually uses `rpm-ostree override` rather than `usroverlay`.

Let's make this a bit saner and have a clean split between
tests that are "shell-script+usroverlay" and "ansible+override".

Closes: #1577
Approved by: jlebon
2018-05-04 19:25:32 +00:00
Colin Walters
956618a188 tests: Tweak staged-deploy test to be faster
Work around https://github.com/ostreedev/ostree/issues/1569

Closes: #1573
Approved by: jlebon
2018-05-04 13:06:37 +00:00
Colin Walters
5337ba51b2 lib/deploy: Do post-ops when removing staged commit
These are further fixes based on running more of the rpm-ostree
test suite.

When dropping the staged deployment, we do need to do the
"post operations" such as bumping the sysroot mtime, so that
clients know something changed.  We also need to regenerate
the deployment refs.  And of course do a sysroot reload.

Also, add a "base cleanup" after creating a staged deployment
which also regenerates the refs.

Closes: #1570
Approved by: jlebon
2018-05-02 16:22:34 +00:00
Colin Walters
25ba8db987 lib/deploy: Throw an error if trying to stage when not ostree-booted
There's no reason to do this.  I didn't actually hit this problem,
but it's a corner case that just occurred to me while working on
the code.

I think callers should be adapted to skip trying to use staging
if there's no booted deployment.

Closes: #1568
Approved by: jlebon
2018-05-02 13:27:01 +00:00
Colin Walters
36ac768fb7 repo: Add checksum to error message opening unreadable object
This would have debugged trying to do a pull as non-root from
a sysroot repository.  See for example:
https://github.com/ostreedev/ostree/issues/1562

Closes: #1564
Approved by: jlebon
2018-05-01 19:22:10 +00:00
Colin Walters
8c1542134c lib/repo: Enable locking by default, but drop external API
The code has been sitting around for a while but since I disabled
it by default, I doubt anyone is really using it or relying on it.

This patch and turns on locking by default, and also drops the
API which was only public in the experimental API builds.
Conceptually these are two distinct things, and we
may actually want to split up the patches.

I don't think this will break anyone, but it's hard to say for sure.
It's also going to be hard to find out until we actually release
I suspect...

But anyone who is broken should be able to add `locking=false` into
their repo config.  On the flip side Endless has been shipping with
this enabled and it is reported to help.

The reason to drop the APIs: I'm a bit concerned about the interactions over time
between libostree's use of the API and any apps that start using it.
For example, if an app specifies a SHARED lock in their code, then
later internally we decide to temporarily grab an `EXCLUSIVE`, but the
app had a second thread/process that was `EXCLUSIVE` already, and
that process was waiting on the first bit of code, then we could
deadlock. I can't think of a real world situation where this would happen
yet though.

We are likely to in the future have say `fsck` take an external lock,
`checkout` grab a shared one, etc.

Closes: #1555
Approved by: jlebon
2018-04-30 17:24:51 +00:00
Colin Walters
06d39efcb5 bin/deploy: Avoid loading merge deployment kargs unless necessary
The fact that `ostree admin deploy` always itself loaded the
merge kargs masked a bug in the core.  Let's change our tests
to not pass any kernel arguments to ensure we cover this.

The new logic in the CLI is a bit subtle, but if you read
carefully is a lot clearer I believe.  Basically we have one
of a few "starting points" in the first section, which can
then be further augmented.

Closes: #1558
Approved by: jlebon
2018-04-27 17:48:21 +00:00
Jonathan Lebon
dc4aa346a3 lib/deploy: Also compare deployment csum versions
When comparing deployments to determine whether we need a new
bootversion, we should also check whether the commit "version" metadata
is the same. Otherwise, we may end up with the a bootconfig whose
`title` includes a version that doesn't match the one from the
deployment checksum.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1343

Closes: #1556
Approved by: cgwalters
2018-04-24 17:04:27 +00:00
Jonathan Lebon
8cfef187c5 tests: Port the last python2 script to python3
This is not used by any test, nor is it packaged. Though let's just port
it over to py3 to certify our codebase completely py2-free. I've
manually checked that the script is still functional.

Closes: #1546
Approved by: cgwalters
2018-04-23 18:33:45 +00:00
Jonathan Lebon
d56058bc9a tests/installed: Prefer python3 over python2
In a world progressively unapproving of python2, let's be a bit smarter
and support testing on platforms that only have python3 installed.

Closes: #1546
Approved by: cgwalters
2018-04-23 18:33:45 +00:00
Colin Walters
d4282720c7 tests: Disable itest-pull.sh since it is too slow
Hopefully we'll fix this soon.

Closes: #1548
Approved by: cgwalters
2018-04-23 17:23:40 +00:00
Colin Walters
76f3e603f9 tests: Lower retry timeout to 5s
No need to poll every second, there's going to be some latency
here and we want to avoid the overhead of polling.

Closes: #1548
Approved by: cgwalters
2018-04-23 17:23:40 +00:00
Colin Walters
41b97e9c12 fsck: Only print "marking commit partial" once
Let's only print if the commit isn't already partial; this
addresses a spam of "marking commit partial" from fsck.

Closes: #1548
Approved by: cgwalters
2018-04-23 17:23:40 +00:00
Jonathan Lebon
6d01d82b9b tests/installed: increase async retries to 500
It seems like 240 retries is just not long enough for all the
non-destructive tests running in parallel to finish. Let's crank that up
to 500 retries.

Closes: #1548
Approved by: cgwalters
2018-04-23 17:23:40 +00:00
Colin Walters
e5f6c9d1e2 tests/installed: Make reboot task less racy
This took a whole lot of experimentation.  I hit upon the idea
of doing a `systemctl stop sshd` to avoid the situation where we
might ssh back into the system while it's in the process of shutting
down.

Ultimately the other fix is disabling `ControlMaster`; see
for example: https://github.com/ansible/ansible/issues/17935

Closes: #1548
Approved by: cgwalters
2018-04-23 17:23:40 +00:00
Matthew Leeds
cf78888401 create-usb: Update summary in destination repo
Currently the create-usb command only generates a summary file in the
destination repo if one doesn't already exist, which means if one does
exist it becomes out of date after the new refs are pulled. This commit
makes ostree regenerate the summary regardless of whether it exists, so
that consumers such as ostree_repo_find_remotes_async() (and at a higher
level, GNOME Software) get an accurate picture of the refs available on
the mount. This commit also updates one of the unit tests to check that
the summary is accurate after a second pull into the same repo.

Since any user of the create-usb command is using collection IDs they
are new enough to be using the unsigned summary support. While it would
technically be possible to use summary signatures on a repo and use the
create-usb command on it (a scenario broken by this commit), the
create-usb command is designed for P2P distribution of refs, which
requires use of unsigned summary support. So this is a legitimate
narrowing of the tool.

Fixes https://github.com/ostreedev/ostree/issues/1465

Closes: #1543
Approved by: cgwalters
2018-04-19 13:35:15 +00:00
Matthew Leeds
653be1556d tests: Fix typo in unit test
Closes: #1543
Approved by: cgwalters
2018-04-19 13:35:15 +00:00
Colin Walters
16d3359bf8 lib/sysroot: Move staged into deployment list, rework handling
Followup to: https://github.com/ostreedev/ostree/pull/1503
After starting some more work on on this in rpm-ostree, it is
actually simpler if the staged deployment just shows up in the list.

It's effectively opt-in today; down the line we may make it the default,
but I worry about breaking things that e.g. assume they can mutate
the deployment before rebooting and have `/etc` already merged.

There's not that many things in libostree that iterate over the deployment
list.  The biggest change here is around the
`ostree_sysroot_write_deployments_with_options` API.  I initially
tried hard to support a use case like "push a rollback" while retaining
the staged deployment, but everything gets very messy because that
function truly is operating on the bootloader list.

For now what I settled on is to just discard the staged deployment;
down the line we can enhance things.

Where we then have some new gymnastics is around implementing
the finalization; we need to go to some effort to pull the staged
deployment out of the list and mark it as unstaged, and then pass
it down to `write_deployments()`.

Closes: #1539
Approved by: jlebon
2018-04-18 18:59:15 +00:00
Alexander Larsson
474556b955 fsck: Mark commits with missing or deleted object partial
This means we can later use various operations to heal the repository
because ostree does not assume all objects are there.

This the begining of a fix for https://github.com/ostreedev/ostree/pull/345

Closes: #1533
Approved by: cgwalters
2018-04-14 15:36:21 +00:00
Giuseppe Scrivano
cdaf7cd838 commit, payload-reflink: do not write to the parent repo
reintroduce the feature that was reverted with commit:

28c7bc6d0e

Differently than the original implementation, now we don't attempt any
test for reflinks support on the parent repository, since the test
requires write access to the repository.

Additionally, also check that the two repositories are on the same
device before attempting any reflink.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #1525
Approved by: cgwalters
2018-04-13 21:52:53 +00:00
Colin Walters
eb506c759c Add concept of "staged" deployment
Add API to write a deployment state to `/run/ostree/staged-deployment`,
along with a systemd service which runs at shutdown time.

This is a big change to the ostree model for hosts,
but it closes a longstanding set of bugs; many, many people have
hit the "losing changes in /etc" problem.  It also avoids
the other problem of racing with programs that modify `/etc`
such as LVM backups:
https://bugzilla.redhat.com/show_bug.cgi?id=1365297

We need this in particular to go to a full-on model for
automatically updated host systems where (like a dual-partition model)
everything is fully prepared and the reboot can be taken
asynchronously.

Closes: https://github.com/ostreedev/ostree/issues/545

Closes: #1503
Approved by: jlebon
2018-04-12 14:55:12 +00:00
Colin Walters
b9fc3eaa15 tests/installed: Move tasks into tasks/ directory
It's cleaner to separate test playbooks from their tasks.

Closes: #1535
Approved by: jlebon
2018-04-11 19:11:07 +00:00
Colin Walters
7357d346c4 tests: Better error message if target is not a symlink
I broke the code for this and ended up adding this to debug it.

Closes: #1535
Approved by: jlebon
2018-04-11 19:11:07 +00:00
Colin Walters
ab61f812c4 tests/installed: Move auto-build logic to playbook-run.sh
This makes it a bit more convenient to make a code change, then
`rm -rf build && ./playbook-run.sh ...`.

Closes: #1535
Approved by: jlebon
2018-04-11 19:11:07 +00:00
Colin Walters
5215f24e68 tests: Merge installed/ and fedora-str/ directories
Let's be opinionated now, and our installed/ test story *is*
Ansible/STR.  Merge `tests/fedora-str` into `tests/installed/`.

Rework the nondestructive tests into a separate playbook run, and parallelize
them for more efficiency.

The destructive tests are also changed to use Ansible more.

Add a higher level `run.sh` entrypoint and update the `README.md`
with some useful tips.

Closes: #1513
Approved by: jlebon
2018-04-05 20:59:23 +00:00
Matthew Leeds
cb3360fca6 lib/repo: Add timestamps to OstreeRepoFinderResult
Currently OstreeRepoFinderResult, a data structure used by pull code
that supports P2P operations, has a hash table mapping refs to checksums
but doesn't include timestamp information. This means that clients have
no way of knowing just from the OstreeRepoFinderResult information if a
commit being offered by a peer remote is an update or downgrade until
they start pulling it. The client could check the summary or the commit
metadata for the timestamps, but this requires adding the temporary
remotes to the repo config, and ostree is already checking timestamps
before returning the results, so I think it makes more sense for them to
be returned rather than leaving it to the client. This limitation is
especially important for offline computers, because for online computers
the latest commit available from any remote is the latest commit,
period.

This commit adds a "ref_to_timestamp" hash table to
OstreeRepoFinderResult that is symmetric to "ref_to_checksum" in that it
shares the same keys. This is an API break, but it's part of the
experimental API, and none of the current users of that (flatpak,
eos-updater, and gnome-software) are affected. See the documentation for
more details on "ref_to_timestamp". One thing to note is the data
structure currently gets initialized in find_remotes_cb(), so only users
of ostree_repo_find_remotes_async() will get them, not users of, say,
ostree_repo_finder_resolve_all_async(). This is because the individual
OstreeRepoFinder implementations don't currently access the timestamps
(but I think this could be changed in the future if there's a need).

This commit will allow P2P support to be added to
flatpak_installation_list_installed_refs_for_update, which will allow
GNOME Software to update apps from USB drives while offline (it's
already possible online).

Closes: #1518
Approved by: cgwalters
2018-04-03 15:50:40 +00:00
Matthew Leeds
925772eb09 tests: Fix unit test for ref-binding metadata
The `ostree show` command is currently failing due to incorrect syntax,
but we want to check that it fails because the metadata isn't there.

Closes: #1520
Approved by: cgwalters
2018-03-28 23:41:55 +00:00
Colin Walters
d379f87405 tests/installed: Fix TESTS= being empty
I broke this in 9b55aaea6f
I'd obviously tested *setting* it locally worked, but I didn't test that
not having it set ran all the tests.

I don't understand why we were doing the `+ ` pattern before; let's
just check if it's empty.

Closes: #1516
Approved by: jlebon
2018-03-26 19:02:54 +00:00
Colin Walters
246a7a5cc2 tests/payload-link: Just test a single duplicate object
We were previously assuming that the host content had duplicates,
which...hopefully it doesn't!  We shouldn't rely on that.

Also this test is slow in production and flaky.  Let's just test
a single duplicate object.

Closes: #1509
Approved by: jlebon
2018-03-22 19:01:52 +00:00
Colin Walters
d2c7c550c1 tests: Small tweaks for local iteration
Support e.g. `-e tests=payload-link`, to choose specific tests for more rapid
iteration, and allow skipping tmpdir cleanup to be able to debug.

Closes: #1509
Approved by: jlebon
2018-03-22 19:01:52 +00:00
Colin Walters
bb9cd1eb72 tests/installed: Use temporary directories more consistently
This is prep for splitting off "nondestructive" tests which
we can run in parallel from the destructive/invasive ones which
e.g. change the host refspec, do deployments.

The `cd` invocation in `prepare_tmpdir` wasn't working because we were running
it in a subshell. Fix this by dropping the subshell.

Closes: #1509
Approved by: jlebon
2018-03-22 19:01:52 +00:00
Colin Walters
dca1cfa60f tests/str: Verify standard-test-roles is installed
This tripped me up when regenerating my dev container.

Closes: #1509
Approved by: jlebon
2018-03-22 19:01:52 +00:00
Colin Walters
d48ccbcabf tests/str: Rework sysinstall-tests to be an entrypoint
Prep for creating more types of tests.

Move copying of `tests/` into the sysinstall-tests rather than `overlay-git`
as not all test types may need that.

Factored out of https://github.com/ostreedev/ostree/pull/1501

Closes: #1509
Approved by: jlebon
2018-03-22 19:01:52 +00:00
Colin Walters
671b026fcc Release 2018.3
It's been over a month since 2018.2; we have a few features and various fixes,
and the "stage" work pending which is pretty invasive. Time for a new release!

Closes: #1506
Approved by: jlebon
2018-03-21 16:01:41 +00:00
Jonathan Lebon
4e4436beec lib/fetcher: Allow clients to append to User-Agent
We do already have `http-headers`, which potentially could be used to
allow clients to completely override the field, but it seems like the
more common use case is simply to append.

Closes: #1496
Approved by: cgwalters
2018-03-16 19:21:31 +00:00
Jonathan Lebon
296ef25e12 lib/core: Support <remote>: syntax when listing refs
Allow users to pass `<remote>:` to list all refs we have locally
belonging to `<remote>`. Also (re-)allow the similar `<remote>:.` syntax
for backwards compatibility with flatpak.

Closes: #1500
Approved by: cgwalters
2018-03-16 19:18:08 +00:00
Colin Walters
10fb74025d tests/str: Rework invocation
Let's make our `run.sh` generically support any playbook. This is prep for
writing further tests in Ansible. Along with that, rework the Ansible so that
`tests.yml` is a playbook, and then the other bits are just task lists. It's
easier to read.

I also started to add a `use_git_build` variable with the idea that we'll be
able to run these same tests against an upstream image by setting that variable
off.

Closes: #1493
Approved by: jlebon
2018-03-16 13:26:22 +00:00
Colin Walters
79fc506284 tests: Avoid generating lots of output in itest-payload-link
We noticed this in a recent PR.  While I'm here, also only do
the `find` once, add `-type l` for good measure, and use our
built in `libtest.sh` assertion functions.

Closes: #1494
Approved by: giuseppe
2018-03-13 13:05:27 +00:00
Colin Walters
6e9d00dbeb ci: Rework installed tests to use Fedora Standard Test interface
Reusing the way `standard-test-roles` has support for booting
a qcow2 actually gets us to the "VM-in-container" flow.  Plus
Ansible over shell script is sometimes nicer.

https://fedoraproject.org/wiki/CI/Tests#Testing_an_Atomic_Host

It's better than what we were doing before for installed tests,
and moreover using Ansible more broadly for testing is going
to align us better with Fedora's CI.

As part of this I split off a "libpaprci" which I intend to maintain
as a "copylib" for a little bit between ostree/rpm-ostree, and then
we'll figure out how to expand from there (maybe some of the patterns
get "baked in" to PAPR for example).

Note the `FAH27-insttests` context moves to the top since it's now
of primary importance, and I expect that we start expanding it.

Closes: #1462
Approved by: jlebon
2018-03-08 20:51:42 +00:00
Giuseppe Scrivano
127d8bb846 commit: add logic for .payload-link
When a new object is added to the repository, create a
$PAYLOAD-SHA256.payload-link symlink file as well.  The target of the
symlink is the checksum of the object that was added the repository.

Whenever we add a new object file, in addition to lookup if the file is
already present with the same checksum we also check if an object with
the same payload is in the repository.

If a file with the same payload is already present in the repository, we
copy it with `glnx_regfile_copy_bytes` that internally attempts to
create a reflink (ioctl (..., FICLONE, ..)) to the target file if the
file system supports it.  This enables to have objects that share the
payload but have a different inode and xattrs.

By default the payload-link-threshold value is G_MAXUINT64 that disables
the feature.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #1443
Approved by: cgwalters
2018-03-07 18:28:59 +00:00
Colin Walters
969e4eb72e repo/refs: Clean up error prefixing
Add some "function global" prefixing in line with what we do in
other places now, and drop the "manual filename" prefixing that
is no longer necessary since
23f7df1500

Closes: https://github.com/ostreedev/ostree/issues/1467

Closes: #1485
Approved by: jlebon
2018-03-06 14:01:39 +00:00
Colin Walters
79d6f635df prune: Error if --static-deltas-only without --delete-commit
The original changes here apparently had the *idea* that `--static-deltas-only`
would be useful in general, but we never implemented that.  The current
situation where it's ignored unless `--delete-commit` is specified is
very misleading and I can easily see it leading to data loss for people.

Let's error out until we have a chance to make it actually useful.

Related: https://github.com/ostreedev/ostree/issues/1479

Closes: #1482
Approved by: giuseppe
2018-03-05 17:39:35 +00:00