Commit Graph

3871 Commits

Author SHA1 Message Date
Colin Walters
22cd178aa6 Post-release version bump
Closes: #1506
Approved by: jlebon
2018-03-21 16:01:41 +00:00
Colin Walters
671b026fcc Release 2018.3
It's been over a month since 2018.2; we have a few features and various fixes,
and the "stage" work pending which is pretty invasive. Time for a new release!

Closes: #1506
Approved by: jlebon
2018-03-21 16:01:41 +00:00
Colin Walters
9ca3f76cd2 lib/deploy: Have internal origin writing API take sepolicy
Ensures it's labeled consistently. Prep for staged deployments which reworks the
logic around when the origin file is written.

Closes: #1505
Approved by: jlebon
2018-03-19 18:42:13 +00:00
Colin Walters
ce2449ad2e lib/deploy: Use in-function error prefixing more
Pulling some of this out of stage deploy work. It's generally better as it's
easier to change functions to have multiple callers.

Closes: #1505
Approved by: jlebon
2018-03-19 18:42:13 +00:00
Colin Walters
01717d7dfc main, status: Factor out deployment printing into helper
Prep for staged deployments; they won't be in the primary deployment
list, and we want to print them first.

Also pull in some code from rpm-ostree for the red/bold bits and use
that tree-wide.

Update submodule: libglnx

Closes: #1504
Approved by: jlebon
2018-03-19 17:15:27 +00:00
Colin Walters
d4d193495f lib/deploy: Port various functions to declare-and-initialize
Just noticed this while working on the code.

Closes: #1499
Approved by: jlebon
2018-03-17 20:36:04 +00:00
Luca Bruno
52f7ba1878 rust/bupsplit: minor idiomatic fixes
This fixes a few un-idiomatic bits in Rust bupsplit code, getting rid
of some unchecked casts and an assert statement.

Closes: #1502
Approved by: cgwalters
2018-03-17 19:59:06 +00:00
Jonathan Lebon
4e4436beec lib/fetcher: Allow clients to append to User-Agent
We do already have `http-headers`, which potentially could be used to
allow clients to completely override the field, but it seems like the
more common use case is simply to append.

Closes: #1496
Approved by: cgwalters
2018-03-16 19:21:31 +00:00
Jonathan Lebon
296ef25e12 lib/core: Support <remote>: syntax when listing refs
Allow users to pass `<remote>:` to list all refs we have locally
belonging to `<remote>`. Also (re-)allow the similar `<remote>:.` syntax
for backwards compatibility with flatpak.

Closes: #1500
Approved by: cgwalters
2018-03-16 19:18:08 +00:00
Colin Walters
10fb74025d tests/str: Rework invocation
Let's make our `run.sh` generically support any playbook. This is prep for
writing further tests in Ansible. Along with that, rework the Ansible so that
`tests.yml` is a playbook, and then the other bits are just task lists. It's
easier to read.

I also started to add a `use_git_build` variable with the idea that we'll be
able to run these same tests against an upstream image by setting that variable
off.

Closes: #1493
Approved by: jlebon
2018-03-16 13:26:22 +00:00
Colin Walters
5b3f79d4bb sysroot: Rework how we find booted deployment
I was looking at this code in prep for "staging" deployments,
and there are several cleanups to be made here.  The first
thing I noticed is that we look for the `ostree=` kernel argument,
but the presence of that should be exactly equivalent to having
`/run/ostree-booted` exist.  We just added a member variable for
that, so let's make use of it.

Related to this, we were erroring out if we had the karg but
didn't find a deployment.  But this can happen if e.g. one is
using `ostree admin --sysroot` from an ostree-booted system!  It's
actually a bit surprising no one has reported this so far; I guess
in the end people are either using non-ostree systems or running
from containers.

Let's add a member variable `root_is_sysroot` that we can use
to determine if we're looking at `/`.  Then, our more precise
"should find a booted deployment" state is when both `ostree_booted`
and `root_is_sysroot` are TRUE.

Next, rather than walking all of the deployments after parsing,
we can inline the `fstatat()` while parsing.  The mild ugly
thing about this is assigning to the sysroot member variable while
parsing, but I will likely clean that up later, just wanted to avoid
rewriting everything in one go.

Closes: #1497
Approved by: jlebon
2018-03-15 17:43:19 +00:00
Colin Walters
bb9cc8912a sysroot: Track whether /run/ostree-booted exists
Prep for further work around deployment staging.

Closes: #1497
Approved by: jlebon
2018-03-15 17:43:19 +00:00
Colin Walters
792c190a44 lib/deploy: Port deployment checkout func to new style
Not sure how we missed this one before.  No functional changes,
just prep for further work.

Closes: #1497
Approved by: jlebon
2018-03-15 17:43:19 +00:00
Rasmus Thomsen
17db0f15a7 configure: add option for libsystemd
Until now ostree checked for libsystemd and enabled
support for it if it found it. This commit changes that
behavior by adding an option to enable/disable libsystemd.
This is especially useful if one uses a source based distro
(like Gentoo/Exherbo), where one wants to avoid such automagic
detection of dependencies and prefers switches for that instead.

Closes: #1490
Approved by: cgwalters
2018-03-13 21:49:47 +00:00
Colin Walters
79fc506284 tests: Avoid generating lots of output in itest-payload-link
We noticed this in a recent PR.  While I'm here, also only do
the `find` once, add `-type l` for good measure, and use our
built in `libtest.sh` assertion functions.

Closes: #1494
Approved by: giuseppe
2018-03-13 13:05:27 +00:00
Joaquim Rocha
591f8a68b1 pull: Ignore the cancellable when aborting a transaction
In ostree_repo_abort_transaction, if we pass a cancellable and it gets
canceled, then the function may fail to fully clean up the transaction
state. This was happening e.g. when the ostree_repo_pull_with_options
call got cancelled.

To fix this, as suggested by Colin Walters, we set the passed
cancellable as NULL, in order for it to be ignored.

https://github.com/ostreedev/ostree/issues/1491

Closes: #1492
Approved by: jlebon
2018-03-12 19:18:57 +00:00
Colin Walters
6e9d00dbeb ci: Rework installed tests to use Fedora Standard Test interface
Reusing the way `standard-test-roles` has support for booting
a qcow2 actually gets us to the "VM-in-container" flow.  Plus
Ansible over shell script is sometimes nicer.

https://fedoraproject.org/wiki/CI/Tests#Testing_an_Atomic_Host

It's better than what we were doing before for installed tests,
and moreover using Ansible more broadly for testing is going
to align us better with Fedora's CI.

As part of this I split off a "libpaprci" which I intend to maintain
as a "copylib" for a little bit between ostree/rpm-ostree, and then
we'll figure out how to expand from there (maybe some of the patterns
get "baked in" to PAPR for example).

Note the `FAH27-insttests` context moves to the top since it's now
of primary importance, and I expect that we start expanding it.

Closes: #1462
Approved by: jlebon
2018-03-08 20:51:42 +00:00
Jeremy Hiatt
3b7044f45e lib/repo: Fix multi-signature support when generating summary files
Ensure that the metadata object is built up with the signatures from all keys
passed to ostree_repo_add_gpg_signature_summary(). Previously only the signature
from the last key would end up in the metadata.

Closes: #1488

Closes: #1489
Approved by: jlebon
2018-03-08 19:58:41 +00:00
Giuseppe Scrivano
127d8bb846 commit: add logic for .payload-link
When a new object is added to the repository, create a
$PAYLOAD-SHA256.payload-link symlink file as well.  The target of the
symlink is the checksum of the object that was added the repository.

Whenever we add a new object file, in addition to lookup if the file is
already present with the same checksum we also check if an object with
the same payload is in the repository.

If a file with the same payload is already present in the repository, we
copy it with `glnx_regfile_copy_bytes` that internally attempts to
create a reflink (ioctl (..., FICLONE, ..)) to the target file if the
file system supports it.  This enables to have objects that share the
payload but have a different inode and xattrs.

By default the payload-link-threshold value is G_MAXUINT64 that disables
the feature.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #1443
Approved by: cgwalters
2018-03-07 18:28:59 +00:00
Giuseppe Scrivano
118f1f7e40 ostree: introduce PAYLOAD_LINK object type
It will be used by successive commits to keep track of the payload
checksum for objects stored in the repository.

The goal is that files having the same payload but different xattrs can
take advantage of reflinks where supported.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #1443
Approved by: cgwalters
2018-03-07 18:28:59 +00:00
Giuseppe Scrivano
418e4545de ostree-repo-private: remove declaration for _ostree_repo_find_object
it was removed with:

commit 8609cb036b
Author: Colin Walters <walters@verbum.org>
Date:   Thu Apr 21 15:14:51 2016 -0400

    repo: Simplify internal has_object() lookup code

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #1443
Approved by: cgwalters
2018-03-07 18:28:59 +00:00
Colin Walters
1c9baad8a8 docs/prune: Document that --static-deltas-only isn't that useful
This is the documentation followup to: https://github.com/ostreedev/ostree/pull/1482

See also https://github.com/ostreedev/ostree/issues/1481

Closes: #1484
Approved by: jlebon
2018-03-06 14:02:32 +00:00
Colin Walters
969e4eb72e repo/refs: Clean up error prefixing
Add some "function global" prefixing in line with what we do in
other places now, and drop the "manual filename" prefixing that
is no longer necessary since
23f7df1500

Closes: https://github.com/ostreedev/ostree/issues/1467

Closes: #1485
Approved by: jlebon
2018-03-06 14:01:39 +00:00
Matthew Leeds
fe6ae92ebc lib: Fix memory leaks of OstreeRemote
The _ostree_repo_get_remote() and _ostree_repo_get_remote_inherited()
methods transfer ownership of the returned OstreeRemote to the caller,
so this commit fixes a few call sites that weren't properly freeing it.

Closes: #1478
Approved by: cgwalters
2018-03-05 20:00:16 +00:00
Colin Walters
79d6f635df prune: Error if --static-deltas-only without --delete-commit
The original changes here apparently had the *idea* that `--static-deltas-only`
would be useful in general, but we never implemented that.  The current
situation where it's ignored unless `--delete-commit` is specified is
very misleading and I can easily see it leading to data loss for people.

Let's error out until we have a chance to make it actually useful.

Related: https://github.com/ostreedev/ostree/issues/1479

Closes: #1482
Approved by: giuseppe
2018-03-05 17:39:35 +00:00
Colin Walters
733c0498dc lib/repo: Do account for size with prune --no-prune
I think this got changed in a refactor.  We definitely want
to total up the amount of space that *would* be freed even
with `--no-prune` AKA `OSTREE_REPO_PRUNE_FLAGS_NO_PRUNE`.

It's actually a bit terrifying this is apparently the first test case for
the `--no-prune` option...

Closes: https://github.com/ostreedev/ostree/issues/1480

Closes: #1483
Approved by: jlebon
2018-03-05 16:58:12 +00:00
Matthew Leeds
1214395f0e lib/repo-finder-mount: Improve debug message
This makes it easier to tell which mount is being checked when repos are
found.

Closes: #1477
Approved by: cgwalters
2018-03-02 19:16:33 +00:00
Matthew Leeds
7727fdd9f7 lib/remote: Fix memory leak
Closes: #1476
Approved by: cgwalters
2018-03-02 18:49:44 +00:00
Matthew Leeds
d0e4a4f03b lib/repo-finder-mount: Update comment about paths
This updates the gtk-doc comment for OstreeRepoFinderMount to match the
correct flatpak repo path, which was fixed in commit 6db6268df.

Closes: #1473
Approved by: cgwalters
2018-03-01 14:10:49 +00:00
Matthew Leeds
2381ca0aa4 lib/repo-pull: Fix free function for hash table
The "ref_original_commits" hash table uses string values, not variants,
so fix the free function passed to g_hash_table_new_full (). Since
g_variant_unref isn't NULL safe, this prevents an assertion failure when
a NULL value is inserted.

Dan Nicholson suggested this patch; I'm just submitting it because he's
busy.

Fixes https://github.com/ostreedev/ostree/issues/1433

Closes: #1474
Approved by: cgwalters
2018-03-01 14:10:24 +00:00
Jonathan Lebon
530043fcf6 lib/sysroot: Fix retrieving non-booted pending deployment
If we're booted into a deployment, then any queries for the pending
merge deployment of a non-booted OS will fail due all of them being
considered rollback.

Fix this by filtering by `osname` *before* determining if we've crossed
the booted deployment yet.

Closes: #1472
Approved by: cgwalters
2018-02-28 18:09:13 +00:00
Matthew Leeds
6db6268dfd lib/repo-finder-mount: Fix path to flatpak repo
OstreeRepoFinderMount checks mounts for a few well-known directories
such as "ostree/repo" and ".ostree/repo" to try to find remotes. One of
the hard-coded directories is "var/lib/flatpak" but that's the flatpak
directory, not the ostree repo used by flatpak, which is at
"var/lib/flatpak/repo". So this commit changes the path so the repo can
be found.

For recent versions of Endless, flatpak uses /ostree/repo as its
repository, so this commit won't make a difference there. But it may
help on other operating systems.

Closes: #1471
Approved by: cgwalters
2018-02-27 20:53:46 +00:00
Colin Walters
ee1f6b2315 bash-completion: Remove admin completions
The `admin` commandline should be considered a demo; I just added
the `pin` command *mostly* so we could use it for unit tests, although
I can imagine other people using it.

But maintaining completions is a lot of overhead right now, let's not
do it for `admin`.

The other command line options that operate on repos we will definitely maintain
since they're used in releng contexts.

Closes: #1468
Approved by: jlebon
2018-02-27 13:56:11 +00:00
Colin Walters
7f88fddcd4 sysroot: Add concept of deployment "pinning" 📌
Example user story: Jane rebases her OS to a new major version N, and wants to
keep around N-1 even after a few upgrades for a while so she can easily roll
back. I plan to add `rpm-ostree rebase --pin` to opt-in to this for example.

Builds on the new `libostree-transient` group to store pinning state there.

Closes: https://github.com/ostreedev/ostree/issues/1460

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Colin Walters
c40a47e965 sysroot: Add API to clean up transient keys in origin files
The `origin/unlocked` and `origin/override-commit` keys are examples of state
that's really transient; we don't want to maintain them across upgrades. Right
now there are bits for this in both `ostree admin upgrade` as well as in
rpm-ostree.

This new API will slightly clean up both cases, but it's really prep for adding
a concept of deployment "pinning" that will live in the new
`libostree-transient` group.

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Colin Walters
2f5a34bed9 sysroot: Bump mtime when writing an origin file
This ensures that e.g. `rpm-ostreed` will get notified of the changes.

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Simon McVittie
971265e4e4 Use Python 3 for tests
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1463
Approved by: cgwalters
2018-02-23 22:49:06 +00:00
Simon McVittie
6cabeaed3f tests/bootloader-entries-crosscheck: Use Python 3-friendly sorting
This is a little clearer than a strcmp()-style negative/zero/positive
return, and also works in Python 2.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1457
Approved by: cgwalters
2018-02-22 19:24:47 +00:00
Simon McVittie
98b597b465 test-concurrency: Explicitly use floor division
Python 3 is pickier about this. Python 2.7 has Python 3-compatible
semantics for division when the division feature is imported from the
future.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1457
Approved by: cgwalters
2018-02-22 19:24:47 +00:00
Simon McVittie
02dc5e2dd4 test-concurrency: Replace range with xrange
range in Python 3 does what xrange did in Python 2. This still works in
Python 2, it just uses a bit more memory.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1457
Approved by: cgwalters
2018-02-22 19:24:47 +00:00
Simon McVittie
9933de2323 test-concurrency: Use Python 3 syntax for octal
This also works in Python 2.7, and is a little clearer.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1457
Approved by: cgwalters
2018-02-22 19:24:47 +00:00
Colin Walters
779df11c9e build-sys: Post-release version bump
Closes: #1455
Approved by: jlebon
2018-02-15 14:44:11 +00:00
Colin Walters
81560cada6 Release 2018.2
There are enough fixes here, and there are some potentially larger patches
incoming like wmanley's checkout speedups and the payload link that will need
soak time in master.

Closes: #1455
Approved by: jlebon
2018-02-15 14:44:11 +00:00
Colin Walters
96eec98f39 fetcher: Drop max queue size assertion in libsoup/libcurl backends
Since f4d1334e19 the primary pull code maintains a
maximum queue. In that commit message I said `Note that I kept an assertion.`.
But I think this is wrong since while it covers a lot of the normal cases, if
one is e.g. trying to fetch a ton of refs, the primary pull code doesn't yet
queue those.  While it'd be nice to queue those, it isn't worth carrying
extra assertions in the backends that can still trigger.

Closes: https://github.com/ostreedev/ostree/issues/1451

Closes: #1453
Approved by: dbnicholson
2018-02-14 23:15:09 +00:00
Alex Kiernan
8dd68fb9f9 Fix static-compiler when CC includes args
Ensure arguments are quoted so that if you include args in CC that
they're handled as a whole.

Closes: #1454
Approved by: cgwalters
2018-02-14 21:09:46 +00:00
Colin Walters
0041a7a1ed core: Add API (and standard concept for) content checksum
There are a few cases for knowing whether a commit has identical
content to another commit.  Some people want to do a "promotion workflow",
where the content of a commit on a tesitng branch is then "promoted"
to a production branch with `ostree commit --tree=ref`.

Another use case I just hit in rpm-ostree deals with
[jigdo](https://github.com/projectatomic/rpm-ostree/issues/1081) where we're
importing RPMs on both the client and server, and will be using the
content checksum, since the client/server cases inject different metadata
into the commit object.

Closes: https://github.com/ostreedev/ostree/issues/1315

Closes: #1449
Approved by: jlebon
2018-02-12 19:03:18 +00:00
Matthew Leeds
5848de93a4 lib/pull: Properly remove temporary remotes
For P2P pulls ostree adds temporary remotes and removes them in
find_remotes_cb(). However, if an OstreeRepoFinderResult gets freed
during the course of that function, the OstreeRemote in the result is
freed but a pointer to it remains in the remotes_to_remove array. This
means that when _ostree_repo_remove_remote() gets called on it at the
end of the function it will fail. In my case the resulting error was
"OSTree-CRITICAL **: _ostree_repo_remove_remote: assertion 'remote->name
!= NULL' failed" but I think it could also seg fault.

This commit adds a reference to the remote so it can be properly removed
when we're finished with it.

Closes: #1450
Approved by: giuseppe
2018-02-09 22:30:51 +00:00
Colin Walters
88d27fb3f1 repo: Create uncompressed-object-cache dir dynamically
Having the `uncompressed-object-cache` directory in `archive` repos by default
is clutter; the functionality should be considered deprecated.

Now we only create the directory if we're doing a checkout with the cache
enabled.

Closes: #1446
Approved by: jlebon
2018-02-08 21:27:11 +00:00
Jonathan Lebon
2e95e06616 lib/checkout: add filter API to skip over files
This is analogous to the filtering support for the commit API: we allow
library users to skip over checking out specific files. This is useful
in some tricky situations where we *know* that the files to be checked
out will conflict with existing files in subtle ways.

One such example is in rpm-ostree support for multilib. There, we want
to allow checking out a package onto an existing tree, but skipping over
files that are not coloured to our preferred value (e.g. not overwriting
an i686 version of `ldconfig` if we already have the `x86_64` version).
See https://github.com/projectatomic/rpm-ostree/pull/1227 for details.

Closes: #1441
Approved by: cgwalters
2018-02-06 15:38:20 +00:00
Alex Kiernan
6bed647648 switchroot: Fix split source/build directory
If you have split source and build directories, then building
static ostree-prepare-root fails to find the source files.

https://github.com/ostreedev/ostree/issues/1429

Closes: #1445
Approved by: cgwalters
2018-02-05 15:26:22 +00:00