Commit Graph

3841 Commits

Author SHA1 Message Date
Jonathan Lebon
530043fcf6 lib/sysroot: Fix retrieving non-booted pending deployment
If we're booted into a deployment, then any queries for the pending
merge deployment of a non-booted OS will fail due all of them being
considered rollback.

Fix this by filtering by `osname` *before* determining if we've crossed
the booted deployment yet.

Closes: #1472
Approved by: cgwalters
2018-02-28 18:09:13 +00:00
Matthew Leeds
6db6268dfd lib/repo-finder-mount: Fix path to flatpak repo
OstreeRepoFinderMount checks mounts for a few well-known directories
such as "ostree/repo" and ".ostree/repo" to try to find remotes. One of
the hard-coded directories is "var/lib/flatpak" but that's the flatpak
directory, not the ostree repo used by flatpak, which is at
"var/lib/flatpak/repo". So this commit changes the path so the repo can
be found.

For recent versions of Endless, flatpak uses /ostree/repo as its
repository, so this commit won't make a difference there. But it may
help on other operating systems.

Closes: #1471
Approved by: cgwalters
2018-02-27 20:53:46 +00:00
Colin Walters
ee1f6b2315 bash-completion: Remove admin completions
The `admin` commandline should be considered a demo; I just added
the `pin` command *mostly* so we could use it for unit tests, although
I can imagine other people using it.

But maintaining completions is a lot of overhead right now, let's not
do it for `admin`.

The other command line options that operate on repos we will definitely maintain
since they're used in releng contexts.

Closes: #1468
Approved by: jlebon
2018-02-27 13:56:11 +00:00
Colin Walters
7f88fddcd4 sysroot: Add concept of deployment "pinning" 📌
Example user story: Jane rebases her OS to a new major version N, and wants to
keep around N-1 even after a few upgrades for a while so she can easily roll
back. I plan to add `rpm-ostree rebase --pin` to opt-in to this for example.

Builds on the new `libostree-transient` group to store pinning state there.

Closes: https://github.com/ostreedev/ostree/issues/1460

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Colin Walters
c40a47e965 sysroot: Add API to clean up transient keys in origin files
The `origin/unlocked` and `origin/override-commit` keys are examples of state
that's really transient; we don't want to maintain them across upgrades. Right
now there are bits for this in both `ostree admin upgrade` as well as in
rpm-ostree.

This new API will slightly clean up both cases, but it's really prep for adding
a concept of deployment "pinning" that will live in the new
`libostree-transient` group.

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Colin Walters
2f5a34bed9 sysroot: Bump mtime when writing an origin file
This ensures that e.g. `rpm-ostreed` will get notified of the changes.

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Simon McVittie
971265e4e4 Use Python 3 for tests
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1463
Approved by: cgwalters
2018-02-23 22:49:06 +00:00
Simon McVittie
6cabeaed3f tests/bootloader-entries-crosscheck: Use Python 3-friendly sorting
This is a little clearer than a strcmp()-style negative/zero/positive
return, and also works in Python 2.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1457
Approved by: cgwalters
2018-02-22 19:24:47 +00:00
Simon McVittie
98b597b465 test-concurrency: Explicitly use floor division
Python 3 is pickier about this. Python 2.7 has Python 3-compatible
semantics for division when the division feature is imported from the
future.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1457
Approved by: cgwalters
2018-02-22 19:24:47 +00:00
Simon McVittie
02dc5e2dd4 test-concurrency: Replace range with xrange
range in Python 3 does what xrange did in Python 2. This still works in
Python 2, it just uses a bit more memory.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1457
Approved by: cgwalters
2018-02-22 19:24:47 +00:00
Simon McVittie
9933de2323 test-concurrency: Use Python 3 syntax for octal
This also works in Python 2.7, and is a little clearer.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1457
Approved by: cgwalters
2018-02-22 19:24:47 +00:00
Colin Walters
779df11c9e build-sys: Post-release version bump
Closes: #1455
Approved by: jlebon
2018-02-15 14:44:11 +00:00
Colin Walters
81560cada6 Release 2018.2
There are enough fixes here, and there are some potentially larger patches
incoming like wmanley's checkout speedups and the payload link that will need
soak time in master.

Closes: #1455
Approved by: jlebon
2018-02-15 14:44:11 +00:00
Colin Walters
96eec98f39 fetcher: Drop max queue size assertion in libsoup/libcurl backends
Since f4d1334e19 the primary pull code maintains a
maximum queue. In that commit message I said `Note that I kept an assertion.`.
But I think this is wrong since while it covers a lot of the normal cases, if
one is e.g. trying to fetch a ton of refs, the primary pull code doesn't yet
queue those.  While it'd be nice to queue those, it isn't worth carrying
extra assertions in the backends that can still trigger.

Closes: https://github.com/ostreedev/ostree/issues/1451

Closes: #1453
Approved by: dbnicholson
2018-02-14 23:15:09 +00:00
Alex Kiernan
8dd68fb9f9 Fix static-compiler when CC includes args
Ensure arguments are quoted so that if you include args in CC that
they're handled as a whole.

Closes: #1454
Approved by: cgwalters
2018-02-14 21:09:46 +00:00
Colin Walters
0041a7a1ed core: Add API (and standard concept for) content checksum
There are a few cases for knowing whether a commit has identical
content to another commit.  Some people want to do a "promotion workflow",
where the content of a commit on a tesitng branch is then "promoted"
to a production branch with `ostree commit --tree=ref`.

Another use case I just hit in rpm-ostree deals with
[jigdo](https://github.com/projectatomic/rpm-ostree/issues/1081) where we're
importing RPMs on both the client and server, and will be using the
content checksum, since the client/server cases inject different metadata
into the commit object.

Closes: https://github.com/ostreedev/ostree/issues/1315

Closes: #1449
Approved by: jlebon
2018-02-12 19:03:18 +00:00
Matthew Leeds
5848de93a4 lib/pull: Properly remove temporary remotes
For P2P pulls ostree adds temporary remotes and removes them in
find_remotes_cb(). However, if an OstreeRepoFinderResult gets freed
during the course of that function, the OstreeRemote in the result is
freed but a pointer to it remains in the remotes_to_remove array. This
means that when _ostree_repo_remove_remote() gets called on it at the
end of the function it will fail. In my case the resulting error was
"OSTree-CRITICAL **: _ostree_repo_remove_remote: assertion 'remote->name
!= NULL' failed" but I think it could also seg fault.

This commit adds a reference to the remote so it can be properly removed
when we're finished with it.

Closes: #1450
Approved by: giuseppe
2018-02-09 22:30:51 +00:00
Colin Walters
88d27fb3f1 repo: Create uncompressed-object-cache dir dynamically
Having the `uncompressed-object-cache` directory in `archive` repos by default
is clutter; the functionality should be considered deprecated.

Now we only create the directory if we're doing a checkout with the cache
enabled.

Closes: #1446
Approved by: jlebon
2018-02-08 21:27:11 +00:00
Jonathan Lebon
2e95e06616 lib/checkout: add filter API to skip over files
This is analogous to the filtering support for the commit API: we allow
library users to skip over checking out specific files. This is useful
in some tricky situations where we *know* that the files to be checked
out will conflict with existing files in subtle ways.

One such example is in rpm-ostree support for multilib. There, we want
to allow checking out a package onto an existing tree, but skipping over
files that are not coloured to our preferred value (e.g. not overwriting
an i686 version of `ldconfig` if we already have the `x86_64` version).
See https://github.com/projectatomic/rpm-ostree/pull/1227 for details.

Closes: #1441
Approved by: cgwalters
2018-02-06 15:38:20 +00:00
Alex Kiernan
6bed647648 switchroot: Fix split source/build directory
If you have split source and build directories, then building
static ostree-prepare-root fails to find the source files.

https://github.com/ostreedev/ostree/issues/1429

Closes: #1445
Approved by: cgwalters
2018-02-05 15:26:22 +00:00
Colin Walters
85c31647cb docs: Dual license under CC BY-SA and the GFDL
This will allow the text to be used in Wikipedia for example; it
also just makes more sense for documentation than the LGPLv2+.

Closes: #1431

Closes: #1432
Approved by: jlebon
2018-02-05 14:31:16 +00:00
Jonathan Lebon
81748857bd bin/checkout: add --selinux-policy switch
This was already supported by the underlying API. Expose it so that we
can test it.

Closes: #1442
Approved by: cgwalters
2018-02-02 22:36:49 +00:00
Jonathan Lebon
6b95d51132 tests/installed: support TESTS filter
Lifted from rpm-ostree. Makes iterating on a single test much faster.
Example use:

    TESTS=label-selinux ./ostree/tests/installed/run.sh

Closes: #1442
Approved by: cgwalters
2018-02-02 22:36:49 +00:00
Jonathan Lebon
5cba67520e libotutil: factor out utility to parse file by line
This will be used in the checkout CLI as well.

Closes: #1442
Approved by: cgwalters
2018-02-02 22:36:49 +00:00
Colin Walters
4a98a86b72 deploy: SELinux-relabel installed kernel/initramfs data
When we changed around the kernel location in rpm-ostree, we
started installing the kernel into `/boot` as `modules_object_t`,
and the current policy didn't permit that.  For maximum compatibility,
relabel installed kernel/initramfs/dtb as `boot_t`.

https://bugzilla.redhat.com/show_bug.cgi?id=1536991

Closes: #1444
Approved by: jlebon
2018-02-02 22:32:49 +00:00
Marcus Folkesson
6bf4b3e1d8 Add SPDX-License-Identifier to source files
SPDX License List is a list of (common) open source
licenses that can be referred to by a “short identifier”.
It has several advantages compared to the common "license header texts"
usually found in source files.

Some of the advantages:
* It is precise; there is no ambiguity due to variations in license header
  text
* It is language neutral
* It is easy to machine process
* It is concise
* It is simple and can be used without much cost in interpreted
  environments like java Script, etc.
* An SPDX license identifier is immutable.
* It provides simple guidance for developers who want to make sure the
  license for their code is respected

See http://spdx.org for further reading.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>

Closes: #1439
Approved by: cgwalters
2018-01-30 20:03:42 +00:00
Colin Walters
2b8d586c5a switchroot: Ensure /sysroot is set to "private" propagation
Downstream BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1498281

This came up as a problem with `oci-umount` which was trying to ensure some host
mounts like `/var/lib/containers` don't leak into privileged containers.  But
since our `/sysroot` mount wasn't private we also got a copy there.

We should have done this from the very start - it makes `findmnt` way, way less
ugly and is just the obviously right thing to do, will possibly create world
peace etc.

Closes: #1438
Approved by: rhvgoyal
2018-01-30 15:05:37 +00:00
Philip Withnall
785da8d5a6 lib/core: Expand documentation for ostree_parse_refspec()
The old documentation had outdated and incomplete annotations, and
didn’t make it very clear that out_remote could legitimately return
NULL.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1437
Approved by: cgwalters
2018-01-29 18:21:00 +00:00
Colin Walters
794d1d98d6 ci: Bump flatpak version (and build to f27)
Just keeping this updated.

Closes: #1400
Approved by: jlebon
2018-01-24 23:24:58 +00:00
Matthew Leeds
660bc588ee lib/pull: Allow specific commits in P2P updates
Currently users of the find_remotes_async()/pull_from_remotes_async()
functions have no way to specify a commit hash to use instead of the
latest one available. This commit implements an "override-commit-ids"
option analogous to the one used by ostree_repo_pull_with_options().
It's accomplished by returning OstreeRepoFinderResult objects pointing
to the given commit checksum(s) regardless of which ones were available
from the remotes, but in the future this implementation could be
improved to take into account the commits advertised by the remotes.

One effect of this is that flatpak will have the ability to downgrade
apps that use collection IDs
(https://github.com/flatpak/flatpak/issues/1309).

Closes: #1425
Approved by: pwithnall
2018-01-24 14:15:53 +00:00
Colin Walters
42eea23864 bin/delta: Fix compilation with relative subdirs --filename
Currently we were parsing `opt_filename` twice...I dug through
the history a bit and it looks like it may have been an accident
from refactoring.

What we're fixing here concretely is that using relative subdirectories
like `--filename somesubdir/foo` broke because we were incorrectly
passing the `somesubdir/` again.

Closes: #1423

Closes: #1427
Approved by: jlebon
2018-01-22 14:02:34 +00:00
Colin Walters
661ec2b1a4 ci: Run a subset ⊂ of rpm-ostree's tests
This is a quick hack to get us more than unit testing, albeit indirectly.

See: https://github.com/projectatomic/rpm-ostree/issues/662

Closes: #771
Approved by: jlebon
2018-01-19 14:16:51 +00:00
Colin Walters
6e4306ea10 ci/papr: Update most contexts to f27
Many of them actually already *were* because they
were inherting.

An exception is flatpak which is being worked on in
https://github.com/ostreedev/ostree/pull/1400

Closes: #1426
Approved by: jlebon
2018-01-19 07:41:18 +00:00
Colin Walters
654a9177d0 lib/pull: Port a few functions to new style
Prep for further work here. This diff is a bit noisy for the delta bits because
the identation was off originally as well.

Closes: #1424
Approved by: jlebon
2018-01-19 07:41:13 +00:00
William Manley
720e2ec9bc Add support for devicetree files alongside the kernel and initramfs
Much like the (optional) initramfs at
`/usr/lib/ostree-boot/initramfs-<SHA256>` or
`/usr/lib/modules/$kver/initramfs` you can now optionally include a
flattened devicetree (.dtb) file alongside the kernel at
`/usr/lib/ostree-boot/devicetree-<SHA256>` or
`/usr/lib/modules/$kver/devicetree`.

This is useful for embedded ARM systems which need the devicetree file
loaded by the bootloader for the kernel to discover and initialise
hardware.  See https://en.wikipedia.org/wiki/Device_tree for more
information.

This patch was mostly produced by copy-pasting code for initramfs handling
and renaming `s/initramfs/devicetree/g`.  It's not beautiful, but it is
fairly straightforward.

It may be useful to extend device-tree support in a number ways in the
future.  Device trees dependant on many details of the hardware they
support.  This makes them unlike kernels, which may support many different
hardware variants as long as the instruction-set matches.  This means that
a ostree tree created with a device-tree in this manner will only boot on
a single model of hardware.  This is sufficient for my purposes, but may
not be for others'.

I've tested this on my NVidia Tegra TK1 device which has u-boot running
in syslinux-compatible mode.

Closes: #1411
Approved by: cgwalters
2018-01-16 22:54:53 +00:00
William Manley
c5112c25e4 syslinux: Add support for DEVICETREE from bootloader spec
The bootloader spec says:

> `devicetree` refers to the binary device tree to use when executing the
> kernel. This also shall be a path relative to the `$BOOT` directory. This
> key is optional. Example:
> `6a9857a393724b7a981ebb5b8495b9ea/3.8.0-2.fc19.armv7hl/tegra20-paz00.dtb`

This is necessary for booting my NVidia Tegra TK1 device.  It uses u-boot
with syslinux compatibility.  In the syslinux files that come with the
device this is called `FDT`, but u-boot treats `FDT and `DEVICETREE` as
synonyms.

See also: [f43c401 in u-boot].

[f43c401 in u-boot]: http://git.denx.de/?p=u-boot.git;a=commit;h=f43c401b72bb0db43ab0b55c4a79e1f4889d3aa2

Closes: #1411
Approved by: cgwalters
2018-01-16 22:54:53 +00:00
William Manley
2098f75c2f ostree admin deploy: Add --no-prune option
If you want cleanup, but don't want to prune the repo.  Pruning can
be quite expensive so ostree admin deploy can be much faster without
pruning.

Closes: #1418
Approved by: cgwalters
2018-01-16 18:59:06 +00:00
William Manley
c5d6725d91 ostree admin deploy: Refactor bringing cleaning into main
In the next commit I will add --no-prune which will affect cleaning.  By
doing this refactor we avoid having to add a NO_PRUNE flag.

Closes: #1418
Approved by: cgwalters
2018-01-16 18:59:06 +00:00
Matthew Leeds
ebc104d3c7 find-remotes: Minor fixes to --finders code
This introduces no functional changes, only cleanups.

Closes: #1414
Approved by: jlebon
2018-01-16 14:15:10 +00:00
Matthew Leeds
4c2205276c man: Add man page for find-remotes
Closes: #1410
Approved by: pwithnall
2018-01-15 19:26:43 +00:00
Matthew Leeds
792b32eb96 man: Update ostree-summary.xml
Update the man page for the summary command to add the undocumented
options, make the syntax clear, and add examples.

Closes: #1416
Approved by: pwithnall
2018-01-15 17:33:06 +00:00
Colin Walters
52dd4f282c build-sys: Post-release version bump
I'm still doing release, then versionbump as separate PRs to ensure
the release commit is tested by itself.

Closes: #1417
Approved by: pwithnall
2018-01-15 15:18:59 +00:00
Colin Walters
d3fa95023e Release 2018.1
In particular I'd like to get the `--copyup` changes out for an rpm-ostree
release that will use them. But there are other good changes here, and let's
keep up a regular release train 🚄 in general.

Closes: #1413
Approved by: jlebon
2018-01-15 14:10:37 +00:00
Colin Walters
8e6e64a5ad lib: Validate metadata structure more consistently during pull
Previously we were doing e.g. `ot_util_filename_validate()` specifically inline
in dirtree objects, but only *after* writing them into the staging directory (by
default). In (non-default) cases such as not using a transaction, such an object
could be written directly into the repo.

A notable gap here is that `pull-local --untrusted` was *not* doing
this verification, just checksums.  We harden that (and also the
static delta writing path, really *everything* that calls
`ostree_repo_write_metadata()` to also do "structure" validation
which includes path traversal checks.  Basically, let's try hard
to avoid having badly structured objects even in the repo.

One thing that sucks in this patch is that we need to allocate a "bounce buffer"
for metadata in the static delta path, because GVariant imposes alignment
requirements, which I screwed up and didn't fulfill when designing deltas. It
actually didn't matter before because we weren't parsing them, but now we are.
In theory we could check alignment but ...eh, not worth it, at least not until
we change the delta compiler to emit aligned metadata which actually may be
quite tricky.  (Big picture I doubt this really matters much right now
but I'm not going to pull out a profiler yet for this)

The pull test was extended to check we didn't even write a dirtree
with path traversal into the staging directory.

There's a bit of code motion in extracting
`_ostree_validate_structureof_metadata()` from `fsck_metadata_object()`.

Then `_ostree_verify_metadata_object()` builds on that to do checksum
verification too.

Closes: #1412
Approved by: jlebon
2018-01-12 19:38:34 +00:00
Colin Walters
f3ae36ff43 lib/checkout: Validate pathnames during checkout
While we do protect against path traversal during pull, let's also validate
during checkout; it's a cheap operation and provides good last-mile protection.

Closes: #1412
Approved by: jlebon
2018-01-12 19:38:34 +00:00
Colin Walters
2b78df25f4 tests: Add a test case for path traversal in a dirtree
I was reading about a recent security issue with both EMC and VMWare:
https://arstechnica.com/information-technology/2018/01/emc-vmware-security-bugs-throw-gasoline-on-cloud-security-fire/

It's a classic path traversal problem, and that made me think more about our
handling of this in libostree.  Fortunately of course, not being new to
this rodeo, long ago I *did* consider path traversal.  Inside the pull
code, we call `ot_util_filename_validate()`.  Also, `fsck` does this too.

I have further followups here, but let's add some test cases for this. I crafted
a repository with a `../` in a dirtree object by patching libostree to inject
it, and that's included as a tarball.

This patch covers the two cases where we do already have checks; pulling
via HTTP, and in `fsck`.

Closes: #1412
Approved by: jlebon
2018-01-12 19:38:34 +00:00
Jonathan Lebon
854a823e05 tests/libtest-core: support multiple literal checks
`grep` supports checking multiple fixed strings separated by newlines,
but it's mostly just easier to pass them as separate arguments, so let's
support that. This is now at parity with the similar
`assert_file_has_content`.

Closes: #1409
Approved by: cgwalters
2018-01-11 21:30:22 +00:00
Colin Walters
fdf7e2c560 lib/fetcher: Add version to USER_AGENT string
This came up in allowing Fedora infrastructure to work around a libcurl bug with
HTTP2: https://pagure.io/atomic-wg/issue/405

Closes: https://github.com/ostreedev/ostree/issues/1405

Closes: #1406
Approved by: jlebon
2018-01-11 14:06:16 +00:00
Matthew Leeds
0be95ded99 tests: Use --finders option for find-remotes
All the current uses of the find-remotes command in the tests use it to
find configured remotes or mounted (USB) remotes, so using
--finders=config and --finders=mount in the tests respectively shouldn't
affect the correctness of the tests. It does however allow the tests to
be run in an environment that doesn't have an Avahi daemon.

Closes: #1407
Approved by: cgwalters
2018-01-11 02:19:07 +00:00
Matthew Leeds
2c932d9721 find-remotes: Add --finders option
It can be helpful to be able to choose which OstreeRepoFinder instances
to use when using the find-remotes command. For example, if the tests
need to run in an environment that can't have an Avahi daemon, this
allows you to disable the Avahi (LAN) finder. This commit adds the
--finders option for this purpose.

Closes: #1407
Approved by: cgwalters
2018-01-11 02:19:07 +00:00