4835 Commits

Author SHA1 Message Date
Colin Walters
97cda7ff4d bupsplit: rustfmt(*)
Let's use the standard rustfmt style.
Also remove unused parenthesis which rust-analyzer was complaining
about.

Also add a `.gitignore`.
2020-06-01 13:53:55 +00:00
OpenShift Merge Robot
d6ce1b09cf
Merge pull request #2118 from cgwalters/error-prefix-parsing
lib: Add error prefixing with specific object when loading
2020-05-29 18:52:23 -04:00
Stefan Agner
5f08649f51 deploy: support devicetree directory
Add support for a devicetree directory at /usr/lib/modules/$kver/dtb/.
In ARM world a general purpose distribution often suppports multiple
boards with a single operating system. However, OSTree currently only
supports a single device tree, which does not allow to use the same
OSTree on different ARM machines. In this scenario typically the boot
loader selects the effective device tree.

This adds device tree directory support for the new boot artefact
location under /usr/lib/modules. If the file `devicetree` does not
exist, then the folder dtb will be checked. All devicetrees are hashed
into the deployment hash. This makes sure that even a single devicetree
change leads to a new deployment and hence can be rolled back.

The loader configuration has a new key "devicetreepath" which contains
the path where devicetrees are stored. This is also written to the
U-Boot variable "fdtdir". The boot loader is expected to use this path
to load a particular machines device tree from.

Closes: #1900
Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
2020-05-29 19:56:11 +02:00
Colin Walters
167edbe63a pull: Add error prefixing with specific object when parsing
One OpenShift user saw this from rpm-ostree:
```
client(id:cli dbus:1.583 unit:machine-config-daemon-host.service uid:0) added; new total=1
Initiated txn UpdateDeployment for client(id:cli dbus:1.583 unit:machine-config-daemon-host.service uid:0): /org/projectatomic/rpmostree1/rhcos
Txn UpdateDeployment on /org/projectatomic/rpmostree1/rhcos failed: File header size 4294967295 exceeds size 0
```

which isn't very helpful.  Let's add some error
prefixing here which would at least tell us which
object was corrupted.
2020-05-29 13:12:02 +00:00
OpenShift Merge Robot
c6eade5ce5
Merge pull request #2117 from cgwalters/pull-signapi-default-explicit
remote-add: Default to explicit sign-verify backends
2020-05-28 13:52:48 -04:00
Colin Walters
68ebf743cd remote-add: Default to explicit sign-verify backends
In 588f42e8c6
we added a way to add keys for sign types when doing
a `remote add`, and in https://github.com/ostreedev/ostree/pull/2105
we extended `sign-verify` to support *limiting* to an explicit
set.

This PR changes the *default* for `remote add` to combine
the two - when providing an explicit `--sign-verify=type`,
we now limit the accepted types to only those.
2020-05-28 00:41:34 +00:00
Colin Walters
1f637bf341 Add new Rust-based tests
There's a lot going on here.  First, this is intended to run
nicely as part of the new [cosa/kola ext-tests](https://github.com/coreos/coreos-assembler/pull/1252).

With Rust we can get one big static binary that we can upload,
and include a webserver as part of the binary.  This way we don't
need to do the hack of running a container with Python or whatever.

Now, what's even better about Rust for this is that it has macros,
and specifically we are using [commandspec](https://github.com/tcr/commandspec/)
which allows us to "inline" shell script.  I think the macros
could be even better, but this shows how we can intermix
pure Rust code along with using shell safely enough.

We're using my fork of commandspec because the upstream hasn't
merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+).

This model is intended to replace *both* some of our
`make check` tests as well.

Oh, and this takes the obvious step of using the Rust OSTree bindings
as part of our tests.  Currently the "commandspec tests" and "API tests"
are separate, but nothing stops us from intermixing them if we wanted.

I haven't yet tried to write destructive tests with this but
I think it will go well.
2020-05-27 21:59:23 +00:00
OpenShift Merge Robot
854099802f
Merge pull request #2116 from cgwalters/kolainst
tests/kola: Move to tests/kolainst
2020-05-27 11:43:22 -04:00
Colin Walters
718cca8055 tests/kola: Move to tests/kolainst
Follow the precedent set in https://github.com/coreos/rpm-ostree/pull/2106
and rename the directory, to more clearly move away from the
"uninstalled" test model.  Prep for Rust-based tests.
2020-05-27 15:16:50 +00:00
OpenShift Merge Robot
936301608a
Merge pull request #2113 from cgwalters/prepare-root-sysroot-ro
Move ro /sysroot bind mount of /etc into initramfs
2020-05-26 11:38:42 -04:00
OpenShift Merge Robot
8e025978e1
Merge pull request #2105 from cgwalters/pull-signapi-explicit
pull: Add support for sign-verify=<list>
2020-05-24 14:53:46 -04:00
Colin Walters
3564225917 Move ro /sysroot bind mount of /etc into initramfs
We recently disabled the read-only /sysroot handling:
e35b82fb89

The core problem was that a lot of services run early in the
real root and want write access to things like `/var` and `/etc`.

In trying to do remounts while the system is running we introduce
too many race conditions.

Instead, just make the `/etc` bind mount in the initramfs right
after we set up the main root.  This is much more natural really,
and avoids all race conditions since nothing is running in the
sysroot yet.

The main awkward part is that since we're not linking
`ostree-prepare-root` to GLib (yet) we have a hacky parser
for the config file.  But, this is going to be fine I think.

In order to avoid parsing the config twice, pass state from
`ostree-prepare-root` to `ostree-remount` via a file in `/run`.
2020-05-24 18:46:28 +00:00
Colin Walters
5cb9d0df38 pull: Add support for sign-verify=<list>
The goal here is to move the code towards a model
where the *client* can explicitly specify which signature types
are acceptable.

We retain support for `sign-verify=true` for backwards compatibility.
But in that configuration, a missing public key is just "no signatures found".

With `sign-verify=ed25519` and no key configured, we can
explicitly say `No keys found for required signapi type ed25519`
which is much, much clearer.

Implementation side, rather than maintaining `gboolean sign_verify` *and*
`GPtrArray sign_verifiers`, just have the array.  If it's `NULL` that means
not to verify.

Note that currently, an explicit list is an OR of signatures, not AND.
In practice...I think most people are going to be using a single entry
anyways.
2020-05-22 19:10:32 +00:00
OpenShift Merge Robot
8801e38bba
Merge pull request #2110 from jlebon/pr/fix-admin-tests2
tests/admin-test: Fix --allow-downgrade check
2020-05-22 20:27:43 +02:00
Jonathan Lebon
6730acc350 tests/admin-test: Fix --allow-downgrade check
We were doing a check to verify that `ostree admin upgrade` wouldn't
accept a downgrade without `--allow-downgrade`. However, there's no
guarantee that the commit it's upgrading from is older than HEAD^ (what
we're upgrading to). Specifically, if the test runs fast enough, the
timestamps could be equal, since the lowest resolution is seconds.

Rework the test so that we first upgrade to HEAD, which we're sure is at
least 1 second apart from HEAD^, and *then* check that downgrade
protection is enforced.

We also can't use `rev-parse testos/buildmaster/x86_64-runtime` as a way
to know what commit the host is sitting on since the ref might've gone
ahead. Instead, just use `ostree admin status | head -n1`. (I played
with using the `ostree/I/J/K` refs, but those depend on what the
boot/subbootversion is and can easily change if we change previous
tests).
2020-05-22 13:59:36 -04:00
OpenShift Merge Robot
6be014daa8
Merge pull request #2111 from strugee/patch-1
Fix typo
2020-05-22 16:36:21 +02:00
AJ Jordan
0487b498ad
Fix typo 2020-05-21 23:22:44 -07:00
OpenShift Merge Robot
0c8701b896
Merge pull request #2108 from jlebon/pr/back-out-ro-sysroot
switchroot/remount: Neuter sysroot.readonly for now
2020-05-21 17:58:04 +02:00
Jonathan Lebon
e35b82fb89 switchroot/remount: Neuter sysroot.readonly for now
We're hitting issues with the read-only remounts racing with various
services coming up. Let's neuter it for now until we rework how it
works.

See: https://github.com/coreos/fedora-coreos-tracker/issues/488
2020-05-20 16:23:59 -04:00
OpenShift Merge Robot
be62a01bff
Merge pull request #2106 from jlebon/pr/fix-admin-test
tests/admin-test: Ensure that commits are 1s apart
2020-05-20 19:05:35 +02:00
OpenShift Merge Robot
6586d240c2
Merge pull request #2107 from cgwalters/more-commit-clocking
admin-test: Show err.txt on unexpected failure
2020-05-20 18:33:51 +02:00
Colin Walters
191ce95ca1 admin-test: Show err.txt on unexpected failure
In a CI run I think one of these `ostree show` commands is failing.
While that output would be useful, the actual `err.txt` usually
has what we want too.
2020-05-20 15:57:10 +00:00
Jonathan Lebon
70ebb91aa3 tests/admin-test: Ensure that commits are 1s apart
Otherwise the new check we added there to verify that upgrading without
`--allow-downgrade` fails itself fails.

See: https://github.com/ostreedev/ostree/pull/2099#issuecomment-629805840
2020-05-20 10:48:44 -04:00
OpenShift Merge Robot
5a82d75adb
Merge pull request #2102 from cgwalters/test-clockskew-check
ci: Test for clock skew
2020-05-18 21:39:30 +02:00
Colin Walters
f14aa894d3 ci: Test for clock skew
I saw `tests/test-admin-deploy.none.sh` fail in one CI run, and
I want to check if it was because of clock skew, so fail
fast if we detect that.

xref https://github.com/ostreedev/ostree/pull/2099#issuecomment-629805375
2020-05-18 18:48:28 +00:00
OpenShift Merge Robot
0d7132ec79
Merge pull request #2101 from cgwalters/signapi-no-explicit-on
pull: Only have API to disable signapi for local pulls
2020-05-18 00:06:04 -07:00
Colin Walters
5a47c926c1 pull: Only have API to disable signapi for local pulls
There's a lot of historical baggage associated with GPG verification
and `ostree pull` versus `ostree pull-local`.  In particular nowadays,
if you use a `file://` remote things are transparently optimized
to e.g. use reflinks if available.

So for anyone who doesn't trust the "remote" repository, you should
really go through through the regular
`ostree remote add --sign-verify=X file://`
path for example.

Having a mechanism to say "turn on signapi verification" *without*
providing keys goes back into the "global state" debate I brought
up in https://github.com/ostreedev/ostree/issues/2080

It's just much cleaner architecturally if there is exactly one
path to find keys: from a remote config.

So here in contrast to the GPG code, for `pull-local` we explictily
disable signapi validation, and the `ostree_repo_pull()` API just
surfaces flags to disable it, not enable it.
2020-05-17 13:52:24 +00:00
OpenShift Merge Robot
b5e51d8c08
Merge pull request #2100 from cgwalters/make-install-kola-tests
ci: Install kola tests
2020-05-15 14:38:00 -07:00
Colin Walters
48f5a1885e ci: Install kola tests
This builds on
https://github.com/coreos/coreos-assembler/pull/1441
to install our tests rather than running them from the source
directory.  This model will more cleanly allow us to ship
our tests along with a test container or elsewhere, separate
from the source directory.

Also prep for https://github.com/ostreedev/ostree/pull/2048
2020-05-15 19:40:54 +00:00
OpenShift Merge Robot
11b09ec3cf
Merge pull request #2099 from jlebon/pr/timestamp-check-from-rev
lib/pull: Add `timestamp-check-from-rev`
2020-05-15 08:16:38 -07:00
Jonathan Lebon
79079c2657 lib/upgrader: Pull with timestamp-check-from-rev
For the same reason as https://github.com/coreos/rpm-ostree/pull/2094.
What we care most about is that the new commit we pull is newer than the
one we're currently sitting on, not necessarily that it's newer than the
branch itself, which it might not be if e.g. we're trying to deploy a
commit older than the tip but still newer than the deployment (via
`--override-commit`).
2020-05-14 15:27:29 -04:00
Jonathan Lebon
c8efce0656 lib/pull: Add timestamp-check-from-rev
The way `timestamp-check` works might be too restrictive in some
situations. Essentially, we need to support the case where users want to
pull an older commit than the current tip, but while still guaranteeing
that it is newer than some even older commit.

This will be used in Fedora CoreOS. For more information see:
https://github.com/coreos/rpm-ostree/pull/2094
https://github.com/coreos/fedora-coreos-tracker/issues/481
2020-05-14 14:00:42 -04:00
OpenShift Merge Robot
f145d18369
Merge pull request #2098 from cgwalters/finalize-requires-mounts
finalize: Add RequiresMountsFor=/boot too
2020-05-13 12:21:14 -07:00
Colin Walters
82679ce834 finalize: Add RequiresMountsFor=/boot too
In https://bugzilla.redhat.com/show_bug.cgi?id=1827712
some OpenShift CI is seeing `/boot` being unmounted before
`ostree-finalize-staged.service` runs or completes.

We finally tracked this down to a bug elsewhere, but
I think we should add this because it clearly shows
our requirements.
2020-05-13 18:52:45 +00:00
OpenShift Merge Robot
4b8354d478
Merge pull request #2097 from cgwalters/sign-verifier
pull: Further cleanup signapi verification
2020-05-12 11:38:58 -07:00
OpenShift Merge Robot
371955549e
Merge pull request #2096 from cgwalters/test-staged-delay
tests/staged-delay.sh: New test
2020-05-12 10:41:29 -07:00
Colin Walters
9509a4bc94 pull: Further cleanup signapi verification
Previously in the pull code, every time we went to verify
a commit we would re-initialize an `OstreeSign` instance
of each time, re-parse the remote configuration
and re-load its public keys etc.

In most cases this doesn't matter really because we're
pulling one commit, but if e.g. pulling a commit with
history would get a bit silly.

This changes things so that the pull code initializes the
verifiers once, and reuses them thereafter.

This is continuing towards changing the code to support
explicitly configured verifiers, xref
https://github.com/ostreedev/ostree/issues/2080
2020-05-12 15:20:26 +00:00
Colin Walters
474e1e2a33 tests/staged-delay.sh: New test
Attempting to reproduce
https://bugzilla.redhat.com/show_bug.cgi?id=1827712
but no dice yet.
2020-05-11 21:31:13 +00:00
OpenShift Merge Robot
4293c36188
Merge pull request #2095 from cgwalters/sign-get-all
signing: Change API to create instances directly
2020-05-11 18:41:35 +02:00
Colin Walters
a9a81f3a29 signing: Change API to create instances directly
This cleans up the verification code; it was weird how
we'd get the list of known names and then try to create
an instance from it (and throw an error if that failed, which
couldn't happen).
2020-05-10 14:18:55 +00:00
OpenShift Merge Robot
f572206f15
Merge pull request #2092 from cgwalters/sign-verify-ed25519-explicit
remote-add: Add --sign-verify=KEYTYPE=[inline|file]:PUBKEYREF
2020-05-10 14:22:41 +02:00
Colin Walters
588f42e8c6 remote-add: Add --sign-verify=KEYTYPE=[inline|file]:PUBKEYREF
Per https://github.com/ostreedev/ostree/issues/2080#issuecomment-623614483

A huge benefit of ed25519 (and ECC in general) is that keys are very
short - short enough that it's completely reasonable to inline
them into a command line argument.

And I think that's a good model; it makes the keys very visible.

For example, someone could easily copy-paste a commandline
argument from a webpage (secured via TLS) that says to run
`ostree remote add --sign-verify=ed25519=inline:KEY`.
2020-05-09 20:07:44 +00:00
OpenShift Merge Robot
4c01c0385f
Merge pull request #2093 from cgwalters/sysroot-requires
ostree-prepare-root: Requires=sysroot.mount
2020-05-08 21:54:51 +02:00
Colin Walters
e1b6398787 ostree-prepare-root: Requires=sysroot.mount
With just `After=` we'll still try to run in the scenario
where `sysroot.mount` fails because the rootfs didn't appear.
And this will end up spewing an error which can confuse people
into thinking something is wrong at the ostree level.

This has come up numerous times w/{Fedora,RHEL} CoreOS, most
recently while looking at
https://bugzilla.redhat.com/show_bug.cgi?id=1803130
2020-05-08 12:50:55 +00:00
OpenShift Merge Robot
59173c9604
Merge pull request #2091 from cgwalters/commit-w-xor-x
commit: Add --mode-ro-executables
2020-05-06 22:11:45 +02:00
Colin Walters
7392259332 commit: Add --mode-ro-executables option
I think we should encourage removing the writable bits from
executables.  This has happened to me:
https://thomask.sdf.org/blog/2019/11/09/take-care-editing-bash-scripts.html

And not having the writable bit may help prevent hardlink
corruption with OSTree in some cases.

We can't do this by default, but add a convenient CLI flag
for it.
2020-05-06 19:41:27 +00:00
OpenShift Merge Robot
ba874c525f
Merge pull request #2090 from cgwalters/sign-default-type
signing: Add #define OSTREE_SIGN_NAME_ED25519
2020-05-04 14:55:23 +02:00
Colin Walters
811082063c signing: Add #define OSTREE_SIGN_NAME_ED25519
Using `#define` or constants instead of strings helps avoid
typos and encourages documentation.
2020-05-03 17:04:27 +00:00
OpenShift Merge Robot
654983ee31
Merge pull request #2089 from dbnicholson/flags-enums-as-bitfields
lib: Coerce flags enums to GIR bitfields
2020-05-01 22:29:37 +02:00
OpenShift Merge Robot
e6bc24804c
Merge pull request #2087 from cgwalters/test-staged-deploy-cleanup
tests/staged-deploy: Cleanup initial state
2020-05-01 22:10:53 +02:00