Commit Graph

226 Commits

Author SHA1 Message Date
Colin Walters
a6f2d053c8 lib: Rework composefs metadata, drop custom signatures
We will be switching to handling signature verification of
the target ostree commit.
2023-06-17 17:08:46 -04:00
Daniel Kolesa
d0ea2db430 fetcher: add libsoup3 backend
The default is still soup2, you can use --with-soup3 to enable
the soup3 backend instead.
2023-04-12 22:33:08 -06:00
Dan Nicholson
3ec7b5db18 tests: Ensure real GIO backends aren't used
Set a few environment variables during tests to ensure fake GIO backends
are used. This is particularly important with the soup fetcher backend
as it can cause strange test errors in containerized test environments.
Upstream soup has been setting these 3 environment variables for their
tests since 2015.
2023-04-12 21:42:33 -06:00
Joseph Marrero Corchado
4f0c13fb8b
Merge pull request #2705 from cgwalters/always-boot
Add `sysroot.bootprefix` option
2023-03-17 11:09:19 -04:00
Dan Nicholson
008c5a9540 bin/remote-refs: Add option to print revisions
The only other way to get the remote ref revision from the CLI is to
scrape the output of `ostree remote summary` or pull the commit. The
revision is already there in the summary's ref map, so might as well add
an option to show it.
2023-02-07 22:59:30 -07:00
Colin Walters
23446a0218 Remove readdir-rand
This was only there to reproduce a bug we hit long ago
with bootloader file ordering.  We're extremely unlikely
to reintroduce such a bug, and it's not worth carrying around
this code.
2022-11-01 16:29:17 -04:00
Miguel Angel Ajo
e234b630f8 Support overlayfs whiteouts on checkout
Introduces an intermediate format for overlayfs storage, where
.wh-ostree. prefixed files will be converted into char 0:0
whiteout devices used by overlayfs to mark deletions across layers.

The CI scripts now uses a volume for the scratch directories
previously in /var/tmp otherwise we cannot create whiteout
devices into an overlayfs mounted filesystem.

Related-Issue: #2712
2022-09-28 12:26:31 +02:00
Colin Walters
10e465c3c4 Add sysroot.bootprefix option
This is a follow up to
0ced9fde76
"sysroot: Support /boot on root or as seperate filesystem for syslinux and u-boot"

What we should have done at the time is changed our bootloader entries
to be prefixed with `/boot`.  This means that the GRUB2 BLS support
will Just Work.

For now, I'm making this option default to off out of a lot of
conservatism.  I think in the future we should flip this on by default.
2022-08-31 16:30:31 -04:00
Huijing Hei
3bc59a5206 RFE: Add a hidden option to ostree admin kargs edit-in-place to
update all existing deployments in place

Example:
$ sudo ostree admin kargs edit-in-place --append-if-missing=rw
See https://github.com/ostreedev/ostree/issues/2617

This will not add duplicate key, if there is `TESTARG=VAL1` in the
kernel arguments, `--append-if-missing=TESTARG=VAL2` will be ignored.
2022-06-23 22:31:39 +08:00
Luca BRUNO
7e83726faa
tests: move fixture to fix installed tests
This moves around the packed archive for bare-split-xattrs mode,
in order to fix installed tests.
2022-04-12 09:53:54 +00:00
Luca BRUNO
aca5671eb1
tests/basic-bare-split-xattrs: add fixture, check read logic 2022-03-03 11:11:59 +00:00
Luca BRUNO
08e98e9042
lib/core: introduce 'bare-split-xattrs' mode 2022-03-02 16:45:00 +00:00
Luca BRUNO
513b3c09a5
main: add support for CLI extensions via external binaries
This adds some logic to detect and dispatch unknown subcommands to
extensions available in `$PATH`. Additional commands can be
implemented by adding relevant `ostree-$verb` binaries to the system.

As an example, if a `/usr/bin/ostree-extcommand` extension is provided,
the execution of `ostree extcommand --help` will be dispatched to that
as `ostree-extcommand extcommand --help`.
2021-12-20 10:00:02 +00:00
Joseph Marrero
581a58067b Update FSF license notices to use URL instead of address 2021-12-07 08:34:25 -05:00
Colin Walters
359435de84 Add an API to verify a commit signature explicitly
We have a bunch of APIs to do GPG verification of a commit,
but that doesn't generalize to signapi.  Further, they
require the caller to check the signature status explicitly
which seems like a trap.

This much higher level API works with both GPG and signapi.
The intention is to use this in things that are doing "external
pulls" like the ostree-ext tar import support.  There we will
get the commitmeta from the tarball and we want to verify it
at the same time we import the commit.
2021-08-30 13:27:38 -04:00
Dan Nicholson
58a683f8f0 bin/remote: Rename list-gpg-keys to gpg-list-keys
As pointed out in the original review, `gpg-list-keys` fits better
alongside the existing `gpg-import`.

Changes were done with:

```
git grep -l list-gpg-keys | xargs sed -i 's/list-gpg-keys/gpg-list-keys/'
for src in $(git ls-files '*list-gpg-keys*'); do
  dst=${src/list-gpg-keys/gpg-list-keys}
  git mv "$src" "$dst"
done
```
2021-08-23 15:30:31 -06:00
Dan Nicholson
74fb0c5f78 bin/remote: Add list-gpg-keys subcommand
This provides a wrapper for the `ostree_repo_remote_get_gpg_keys`
function to show the GPG keys associated with a remote. This is
particularly useful for validating that GPG key updates have been
applied. Tests are added, which checks the
`ostree_repo_remote_get_gpg_keys` API by extension.
2021-07-15 15:50:04 -06:00
Luca BRUNO
70a8f56ce1
lib/commit: respect SOURCE_DATE_EPOCH for commit timestamp
This tweaks `ostree_repo_write_commit` so that it checks for the
envinroment variable `SOURCE_DATE_EPOCH` as a way to override
the current time, which is used as the commit timestamp.

Ref: https://reproducible-builds.org/docs/source-date-epoch/
Ref: https://reproducible-builds.org/specs/source-date-epoch/
2021-06-22 12:15:18 +00:00
Colin Walters
7310203ce9 tests/inst: Make nondestructive tests runnable as unit tests
Ideally in the future we change more of our unit tests to
support running installed; we've tried this in the past with
https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests

I'd like to pick that back up again.  This takes a step
towards that by having our Rust tests.

To make this even easier, add a `tests/run-installed`
which runs the installed tests (uninstalled, confusingly
but conveniently for now).
2021-04-15 12:33:34 -04:00
Colin Walters
a90d59cf2d build-sys: Remove --enable-experimental-api
It was added for the collections bits, but we made that stable.
It's now just cruft and we're very unlikely to reuse the infrastructure
again.

Motivated by a unit test failure when running from a tarball:
https://github.com/ostreedev/ostree/issues/2313
2021-04-07 19:19:05 +00:00
Philip Withnall
0974a7faf1 tests: Split RFC 2616 date parsing code out and add tests
This makes it testable, and increases its test coverage too 100% of
lines, as measured by `make coverage`.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
Philip Withnall
a522bf7628 tests: Add simple test for summary file caching
This test would have actually passed before the summary file caching
changes (in the previous few commits) were added, as the `summary.sig`
essentially acted as the ETag for the summary file, and itself wasn’t
updated on disk if it didn’t change when querying the server.

Actually testing that the HTTP caching headers are working to reduce
HTTP traffic would require test hooks into the pull code or the
trivial-httpd server, neither of which I have the time to add at the
moment.

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-22 21:03:34 +01:00
OpenShift Merge Robot
e6f673a8ed
Merge pull request #2202 from cgwalters/bootcsum-dtb
deploy: Remove deployment bootcsum assertion
2020-09-24 22:14:31 +00:00
Colin Walters
aa2a2783ea deploy: Remove deployment bootcsum assertion
When support for devicetree was added, it created a problem
because old and new ostree versions would compute different
checksums for the "boot data".  The scenario here is:

- Have system with ostree < 2020.4
- Reboot into system with ostree 2020.5
- Try to perform an operation that would retain
  that previous booted deployment (common)

Currently ostree iterates over all the deployments
that will be retained and calls `install_deployment_kernel()`,
even for the booted one (which is a bit silly), but
just to verify that all boot data for the targeted
deployments are installed.

This then re-computes the checksum and we'd trip this
assertion.

In practice though, we don't strictly require them to match;
the only thing that will happen if they don't is that we'll
end up with another copy of the kernel/initramfs - and
that only temporarily until the previous deployment
gets GC'd.

Longer term, I think what we really want to do anyways
is probably closer to like a little ostree repo for `/boot`
so that we can e.g. still hardlink kernels there even if
the initramfs changes, or hardlink both kernel/initramfs
if just the devicetree changes, etc.

Closes: https://github.com/ostreedev/ostree/issues/2154
2020-09-24 21:24:48 +00:00
Frédéric Danis
869dbc037e tests/delta: new tests for 'ed25519' signed deltas
Add tests to generate signed deltas and verify them using 'ed25519'
signature engine.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Frédéric Danis
c98a993c99 tests/delta: new tests for signed deltas
Add tests to generate signed deltas and verify them using 'dummy'
signature engine.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Colin Walters
40d6f6b5ee tests: Add a pre-signed-pull.sh test
I'm thinking about adding an implementation of ed25519 signatures
with OpenSSL (so we can ship the feature with Fedora CoreOS
without requiring an additional library) and in preparation for
that it's essential that we validate that libsodium-generated
signatures and OpenSSL-generated signatures are compatible.

I don't know if they are yet actually, but the goal of this
new test is to add a pre-generated repository with a signed
commit generated by libsodium.

This will catch if e.g. there's ever a change in libsodium,
or if existing libsodium implementation versions (e.g. the
one in Debian) might differ from what we ship here.
2020-06-11 18:56:35 +00:00
Colin Walters
48f5a1885e ci: Install kola tests
This builds on
https://github.com/coreos/coreos-assembler/pull/1441
to install our tests rather than running them from the source
directory.  This model will more cleanly allow us to ship
our tests along with a test container or elsewhere, separate
from the source directory.

Also prep for https://github.com/ostreedev/ostree/pull/2048
2020-05-15 19:40:54 +00:00
Colin Walters
368574b657 deploy: Add --no-merge
All of the underlying libostree APIs have supported passing `NULL`
for a merge deployment for...a long time.  But we never plumbed
it up into the CLI.

Add a `--no-merge` option to aid people who want to do a "factory reset":
https://github.com/ostreedev/ostree/issues/1793
2020-04-22 14:09:54 +00:00
Denis Pynkin
1de2efa2ed tests/sign: new test for summary file verification
Add test for signature verification of summary file during the pull.
Adopted version of GPG tests from `test-pull-summary-sigs.sh`.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:55 +03:00
Denis Pynkin
82c773710c tests/sign: add initial test for pulling
Test if we pull signed commits from remote.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
9e8f0f4ca0 tests: add test for commits sign/verification
Add tests checking:
- sign mechanism is in working state
- module 'dummy' is able to sign/verify commit
- module 'ed25519' is able to sign/verify commit
- both modules could be used for the same commit
- 'ostree sign' builtin works with commits
- 'ostree commit' builtin able to sign commits

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
OpenShift Merge Robot
8a9a496501
Merge pull request #1957 from dbnicholson/commit-sizes
Upstream Endless sizes metadata changes
2020-01-27 08:25:59 -08:00
Dan Nicholson
7fe265b087 tests/gpg-verify-data: Split out signature data
The full block with all 5 signatures remains, but this allows passing
individual signatures through the GPG verification APIs. The split was
done with `gpgsplit`, and looking at the output of `gpg --list-packets`
of the split and unsplit files appears correct.
2020-01-24 13:02:09 -07:00
Dan Nicholson
b825083549 tests/gpghome: Create revocation certificates for keys
These can then be imported during a test to revoke a key without trying
to go through the gpg --generate-revocation dialog. Note that these need
to go in a subdirectory of the homedir since `gpgkeypath` will try to
import every regular file in the homedir.
2020-01-24 13:02:09 -07:00
Dan Nicholson
97c831dd5f bin/show: Add --print-sizes option to show sizes metadata
Use the new `ostree_commit_get_object_sizes()` API to read the
`ostree.sizes` commit metadata and print a summary.
2020-01-20 20:46:31 -07:00
Dan Nicholson
260bcd1193 core: Add ostree_commit_get_object_sizes API
This function parses the object listing in the `ostree.sizes` metadata
and returns an array of `OstreeCommitSizesEntry` structures.

Unfortunately, for reasons I don't understand, the linker wants to
resolve `_ostree_read_varuint64` from `ostree-core.c` even though it's
not used by `test-checksum.c` at all.
2020-01-20 20:46:29 -07:00
Colin Walters
36039759e2 pull: Add support for basic auth
This has been requested a few times for people delivering
OSTree content and wanting to do access control.
2019-12-02 22:48:33 +00:00
Jason Wessel
bdbce9d042 fsck: Add test for --delete corruption, fix repair, and partial commit checks
The ostree fsck test is aimed to check that it will still fail an fsck
if the repository has been repaired by fsck.  It also checks that a
pull operation corrects the error and ostree fsck will exit with zero.

The test was modeled after the following script:

rm -rf ./f1
mkdir -p ./f1
./ostree --repo=./f1 init --mode=archive-z2
mkdir -p ./trial
echo test > ./trial/test
./ostree --repo=./f1 commit --tree=dir=./trial --skip-if-unchanged --branch=exp1 --subject="test Commit"

rm -rf ./f2
mkdir -p ./f2
./ostree --repo=./f2 init
./ostree --repo=./f2 pull-local  ./f1

echo whoops > `find ./f2 |grep objects |grep \\.file `
./ostree fsck --repo=./f2 ; echo Exit: $?
./ostree fsck --delete --repo=./f2 ; echo Exit: $?
./ostree fsck --repo=./f2 ; echo Exit: $?
./ostree --repo=./f2 pull-local  ./f1
./ostree fsck --repo=./f2 ; echo Exit: $?

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>

fsck: Update test so that it will pass on fs without xattrs

The fsck test does not require xattrs to prove that it works.  It is
simple enough to change it to use an archvie instead of a bare type
repository.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>

Closes: #1910
Approved by: cgwalters
2019-09-09 13:40:36 +00:00
Denis Pynkin
e187f240fc build: Conditionally build the GPG-related tests
Do not build GPG signing/verification tests if `--without-gpgme`
option is used during configuration.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Closes: #1889
Approved by: cgwalters
2019-08-01 02:06:47 +00:00
Allen Bai
be2572bf68 lib/kargs: Make API public and upstream new rpm-ostree APIs
This change makes public the current kargs API in src/libostree/ostree-kernel-args.c
and adds documentations.

Upstreams the new kargs API from rpm-ostree/src/libpriv/rpmostree-kargs-process.c

Merges libostree_kernel_args_la_SOURCES to libostree_1_la_SOURCES in Makefile-libostree.am

Upstreams tests/check/test-kargs.c from rpm-ostree.

Closes: #1833

Closes: #1869
Approved by: jlebon
2019-06-18 14:29:14 +00:00
Robert Fairley
21ebc7d21e Add sysroot.bootloader repo config key
The sysroot.bootloader key configures the bootloader
that OSTree uses when deploying a sysroot. Having this key
allows specifying behavior not to use the default bootloader
backend code, which is preferable when creating a first
deployment from the sysroot (#1774).

As of now, the key can take the values "auto" or "none". If
the key is not given, the value defaults to "auto".

"auto" causes _ostree_sysroot_query_bootloader() to be used
when writing a new deployment, which is the original behavior
that dynamically detects which bootloader to use.

"none" avoids querying the bootloader dynamically. The BLS
config fragments are still written to
sysroot/boot/loader/entries for use by higher-level software.

More values can be supported in future to specify a single
bootloader, different behavior for the bootloader code, or
a list of bootloaders to try.

Resolves: #1774

Closes: #1814
Approved by: jlebon
2019-03-01 21:20:35 +00:00
Rafael Fonseca
6e8377cbd1 admin/status: test GPG signature check
Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>

Closes: #1797
Approved by: cgwalters
2019-01-08 14:47:08 +00:00
Sinny Kumari
dde3f1c0fb src/ostree: Add --group option to ostree config
Fetching value from a repo config using 'ostree config
get SECTIONNAME.KEYNAME' didn't work in some cases like
when having dots in Group Name entry.
As per Desktop entry file specification, Group Name
may contain all ASCII characters except for [ and ]
and control characters.
Link - https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-1.1.html

Having --group option will help user to clearly specify
Group Name and get desired result.

It also adds test for ostree config get|set and bash
completion for --group option

Fixes https://github.com/ostreedev/ostree/issues/1565

Closes: #1696
Approved by: cgwalters
2018-08-20 14:31:15 +00:00
Stefan Agner
4a389b3082 Avoid race condition in case tests directory does not exist
Make sure the tests directory exists before symlinking files
into it.

Closes: #1703

Closes: #1704
Approved by: cgwalters
2018-08-14 12:50:11 +00:00
Dan Nicholson
ce58307757 deploy: Delete .updated file from /etc and /var on new deployments
Systemd units using ConditionNeedsUpdate run if the mtime of .updated in
the specified directory is newer than /usr. Since /usr has an mtime of
0, there's no way to have an older .updated file. Systemd units
typically specify ConditionNeedsUpdate=/etc or ConditionNeedsUpdate=/var
to support stateless systems like ostree.

Remove the file from the new deployment's /etc and the OS's /var
regardless of where they came from to ensure that these systemd units
run when booting new deployments. This will provide a method to run
services only on upgrade.

Closes: #1628
https://bugzilla.gnome.org/show_bug.cgi?id=752950

Closes: #1631
Approved by: cgwalters
2018-06-18 13:21:52 +00:00
Matthew Leeds
6f3b5620de tests: Add test for including ostree.h
Closes: #1615
Approved by: jlebon
2018-06-07 17:28:49 +00:00
Matthew Leeds
8fbf19c9f5 Make P2P API public (no longer experimental)
Currently the API that allows P2P operations (e.g. pulling an ostree ref
from a LAN or USB source) is hidden behind the configure flag
--enable-experimental-api. This commit makes the API public and makes
that flag essentially a no-op (leaving it in place in case we want to
use it again in the future). The P2P API has been tested over the last
several months and proven to work.

This means that since we're no longer using the "experimental" feature
flag, P2P builds of Flatpak will fail when using versions of OSTree from
this commit onwards, until Flatpak is patched in the near future. If you
want to build Flatpak < 0.11.8 with P2P enabled and link against OSTree
2018.6, you'll have to patch Flatpak.  However, since Flatpak won't yet
have a hard dependency on OSTree 2018.6, it needs a new way to determine
if the P2P API in OSTree is available, so this commit adds a "p2p"
feature flag. This way the feature set is more semantically correct than
if we had continued to use the "experimental" feature flag.

In addition to making the P2P API public, this commit makes the P2P unit
tests run by default, removes the f27-experimental CI instance that's no
longer needed, changes a few man pages to reflect the changes, and
updates the bash completion script to accept the new commands and
options.

Closes: #1596
Approved by: cgwalters
2018-06-04 19:20:10 +00:00
Marcus Folkesson
6bf4b3e1d8 Add SPDX-License-Identifier to source files
SPDX License List is a list of (common) open source
licenses that can be referred to by a “short identifier”.
It has several advantages compared to the common "license header texts"
usually found in source files.

Some of the advantages:
* It is precise; there is no ambiguity due to variations in license header
  text
* It is language neutral
* It is easy to machine process
* It is concise
* It is simple and can be used without much cost in interpreted
  environments like java Script, etc.
* An SPDX license identifier is immutable.
* It provides simple guidance for developers who want to make sure the
  license for their code is respected

See http://spdx.org for further reading.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>

Closes: #1439
Approved by: cgwalters
2018-01-30 20:03:42 +00:00
Colin Walters
2b78df25f4 tests: Add a test case for path traversal in a dirtree
I was reading about a recent security issue with both EMC and VMWare:
https://arstechnica.com/information-technology/2018/01/emc-vmware-security-bugs-throw-gasoline-on-cloud-security-fire/

It's a classic path traversal problem, and that made me think more about our
handling of this in libostree.  Fortunately of course, not being new to
this rodeo, long ago I *did* consider path traversal.  Inside the pull
code, we call `ot_util_filename_validate()`.  Also, `fsck` does this too.

I have further followups here, but let's add some test cases for this. I crafted
a repository with a `../` in a dirtree object by patching libostree to inject
it, and that's included as a tarball.

This patch covers the two cases where we do already have checks; pulling
via HTTP, and in `fsck`.

Closes: #1412
Approved by: jlebon
2018-01-12 19:38:34 +00:00