IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
This adds support for a new `rpm-ostree compose extensions` command`
which takes a treefile, a new extensions YAML file, and an OSTree repo
and ref. It performs a depsolve and downloads the extensions to a
provided output directory.
This is intended to replace cosa's `download-extensions`:
https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions
The input YAML schema matches the one accepted by that script.
Some differences from the script:
- We have a guaranteed depsolve match and thus can avoid silly issues
we've hit in RHCOS (like downloading the wrong `libprotobuf` for
`usbguard` -- rhbz#1889694).
- We seamlessly re-use the same repos defined in the treefile, whereas
the cosa script uses `reposdir=$dir` which doesn't have the same
semantics (repo enablement is in that case purely based on the
`enabled` flag in those repos, which may be different than what the
rpm-ostree compose ran with).
- We perform more sanity-checks against the requested extensions, such
as whether the extension is already in the base.
- We support no-change detection via a state SHA512 file for better
integration in cosa and pipelines.
- We support a `match-base-evr` key, which forces the extension to have
the same EVR as the one from a base package: this is helpful in the
case of extensions which complement a base package, esp. those which
may not have strong enough reldeps to enforce matching EVRs by
depsolve alone (`kernel-headers` is an example of this).
- We don't try to organize the RPMs into separate directories by
extension because IMO it's not at the right level. Instead, we should
work towards higher-level metadata to represent extensions (see
https://github.com/openshift/os/issues/409 which is related to this).
Closes: #2055
The service where this is hosted was intended to be temporary; support
overriding it so if it goes down in the future people can at
least use a systemd unit file override to change it easily.
This is a workaround for the non-customizability of the cxx-rs
propagation of Rust result to C++ exception. Right now we're
losing context. Work around this by formatting on the Rust
side at exit points, explicitly converting an `anyhow::Error`
by printing it in "single line context".
Since we're likely to gain more things like this, unify
this with `cxx_bridge_gobject::` into a single `cxxrsutil::`.
Gather the current diff of `/etc`, and filter out changes in
the tree which would overwrite it.
There is an OSTree API for diffs but it's a bit awkward, missing
some APIs in the Rust bindings and also `GFile` based unfortunately.
Doing this in Rust is nicer. The dirdiff code obviously needs
a lot more testing, but I think it's right.
This adds support for e.g.:
```
$ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2020-2908628031
```
This will find the Koji builds from the listed update, download
all the RPMs (that aren't debuginfo) and pass them for overrides
in the same way we support `override replace http://somewebserver/foo.rpm`
now.
We also support directly linking a Koji build:
```
$ rpm-ostree override replace https://koji.fedoraproject.org/koji/buildinfo?buildID=1625029
```
Bodhi has a modern HTTP+JSON API, and the lack of a Koji equivalent
drove me to create https://github.com/cgwalters/koji-sane-json-api
and we currently depend on an instance set up in the OpenShift CI
cluster.
I hope it shouldn't take long to deploy this in Fedora Infra,
but I don't want to block on it.
Also notably this still downloads *all* the other RPMs even
ones that aren't installed. Handling that truly correctly
would require moving this logic to the daemon and core.
All of this functionality is keyed off a `cfg(feature = "fedora-integration")`
that is detected by a Rust `build.rs` which parses the build environment's
`/etc/os-release` for now.
This is cleaner if we move the "catch and print errors" into
the Rust side. We lose the "print errors as red" part but
eh we can readd that later Rust side.
This makes the C++ side a bit uglier because the "variable mapping"
is more Rust-native but we only use it Rust side anyways.
(We can't yet move the basearch bits to rust because it depends on
libdnf, which requires buildsystem unification)
But all the unsafe FFI conversion drops out, as do the duplicated
C unit tests.
Prep for the [fedora-integration PR](https://github.com/coreos/rpm-ostree/pull/2420).
This also generalizes the "fetch URL to tempfile" code into
supporting multiple at once - it's much more efficient to do
it that way because we can reuse a TCP connection to servers,
parsed certificates etc.
(I'm going to start calling it `live-apply` instead of "livefs")
On one hand, this drops the unsafe bindgen glue. On the other
hand it demonstrates a notable current ergonomic shortfall of
cxx-rs in that it doesn't support `Option<T>`, so we represent
optional strings by converting them to empty strings.
(Relatedly I discovered in C++ there is `std::optional<>` but it's in C++17.
I assume we can depend on that but let's for now match libdnf's
usage of `-std=c++14` because we know that works everywhere)
cxx-rs has support for bridging types, but it's more awkward
for us because those types are defined in other crates, so
we need to do a newtype dance. Further cxx-rs doesn't currently
support automatically generating wrappers, so add a custom
`gobj_wrap()` for now.
Rust has a nice crate for doing perfect hashing. Move that
code into Rust and drop the dependency on `gperf`. This also
helps move away from Autotools.
The way gresources work using a constructor function started
failing when I was refactoring the build system, and I couldn't
figure it out. It's just easier to use Rust for this which
has nice toolchain-integrated functionality for this.
This is much better than bindgen because it's fully safe. It's
much more ergonomic too:
- Invoke Rust methods-on-structs just like C++ methods-on-structs
- Rust `Result<>` is translated automatically to exceptions
See https://cxx.rs/context.html for more.
There won't be any support for writing to the bdb backend in f34, so
e.g. pkglayering won't work (and obviously even composes wouldn't work
once the buildroot moves to f34).
Instead of requiring the whole world to add an `rpmdb` key in their
manifests, let's just add a compile flag for it, and tweak the spec file
to use this flag on f34.
I was looking at the output of `ostree admin config-diff`
on a base FCOS boot. It'd be really nice to trim that down
as much as possible, so we can cleanly capture the difference
between user config and system config.
Let's use static enablement rather than presets.
The `Journal::open` API has been deprecated in favour of the new
`OpenOptions` builder pattern.
We could dedupe this a bit more, though the mock journal in the history
code makes it trickier and there's little value in mocking the builder
pattern too.
Now always based on an overlayfs:
f2773c1b55
This fixes a whole swath of problems with the previous design,
including the danger in replacing `/usr/lib/ostree-boot` which
broke booting for some people.
Further, we don't need to push a rollback deployment; the livefs
changes are always transient. So now we store livefs state
in `/run` instead of in the origin file.
Since we're doing a rewrite, it's now in Rust for much more safety.
We also always work in terms of incremental diffs between commits;
the previous huge hammer of swapping `/usr` was way too dangerous.
This effectively reverts commit: c8113bde32
We never ended up using it; instead the `rdcore` bits from
`coreos-installer` have the rootfs reprovisioning logic.
See https://github.com/coreos/rpm-ostree/pull/2206#issuecomment-721372634
The commit 7f579a55d3fb7ec1cb9f74f8ec6bc36675df2ccc broke hashing
of overlay commits; this is a super evil bug because it causes us
to silently do the wrong thing.
The cause here is the GLib bindings don't (AFAICS) support getting
a `&mut` for a GLib boxed value.
Move all of the treefile checksum code into one place - this is
far saner. The reason I didn't do this before is that it
will cause a spurious rebuild when one updates rpm-ostree, but...eh.
Particularly in places like the lockfile code where we were iterating
on a list of packages, validating UTF-8 and `memcpy()`ing strings
from C is...well, unnecessary.
I don't think there's any actual real performance concerns right
now but let's use this as a best practice because the patterns
we establish *will* be copy+pasted or at least used as inspiration
for other places where performance might matter.
I happened to scroll past this code while doing something
else and noticed what we were doing here was silly - no need
to re-validate UTF-8 etc when serde can hand us a `String` directly.
Clippy reports this as an error; I think in practice Linux
doesn't return short writes for regular files, but it's still
a good idea to do the right thing here.
Let's just use the GLib translation bits rather than rolling our own;
this applies primarily to `ffi_new_string()`.
However, I think in most cases performance here doesn't
matter enough to have an even more special case that avoids duplicating
the string. Let's remove the `ffi_view_str()` optimization too
in favor of consistently using GLib translation.
In the future perhaps we should argue for adding a `from_glib_str_unchecked()`
to the upstream GLib bindings.
This starts bridging parts of the daemon syscore logic to Rust
plumbing, moving the livefs detection logic over there as a first
consumer. That was the simplest logic available for wiring, and
mostly meant as a sanity check.
It's tempting to flip the default, but let's encourage
people to adapt to the `modules` version now.
Plus infrastructure to add deprecation warnings here
will come in handy in the future.
The next/previous functions were renamed, and the journal is no longer
seeked to the head on open, so we have to do it manually. We did that
already in most places, except one.
For more information, see:
https://github.com/jmesmon/rust-systemd/pull/121
The design of https://fedoraproject.org/wiki/Changes/Sqlite_Rpmdb
is problematic for us for multiple reasons. The first big reason
is that rpm-ostree is designed for "cross" builds and e.g. today
we use a Fedora-derived container to build RHEL CoreOS images.
However the default database lives inside the `rpm` package which
means that if we e.g. upgrade the coreos-assembler container to F33
it will suddenly try to use sqlite for RHCOS which is obviously broken.
Related to this, rebases from f32 to f33 w/layered packages
are broken: https://bugzilla.redhat.com/show_bug.cgi?id=1876194#c3
With this we can configure things to continue to use bdb for f33
for ostree-based systems, so that by enforcing an upgrade order
f32 → f33 [bdb] → f34 [sqlite] ... the intermediate f33 w/bdb
still understands sqlite and hence rebases will work.
The main goal here is simply to start the ball rolling
in having rpm-ostree use the ostree-rs crate, so we can
use it for other code in the future. We also bump the
glib crate to a new major, but no code changes were
required for that.
I spent a bit of time looking through our C code for
an example to oxidize, but almost all of our C code using
libostree also has calls back into other nontrivial rpm-ostree
C code. To make a lot more oxidation progress I think we need
to invent an internal `librpmostree-sys` crate so we can have
Rust -> C calls.
I found this bit of treefile code computing the checksum that
we can oxidize - and as soon as we have more libdnf bindings
we can move all of the checksumming into Rust too.
We want to test upgrades that actually change files as a general
rule; in some cases we want to test "large" upgrades to validate
performance.
This code generates a "synthetic" upgrade that adds an ELF note
to a percentage of ELF files (randomly selected). By doing
it this way we are only actually testing one version of the code.
Migrated from https://github.com/coreos/coreos-assembler/pull/1635/
using the Rust code from https://github.com/ostreedev/ostree/pull/2127
Fix https://github.com/coreos/rpm-ostree/issues/2068: `remove-from-packages`
deleting files that it shouldn't.
Filter out files that user wants removed at `checkout_package_into_root()`,
instead of at the `handle_remove_files_from_package()` function that does
not check whether files are used by other rpms before removing them.
Add a helper function for whitespace_split_packages() so that it now
splits a String by whitespace only if it is not wrapped between single
quotes.
This should allow RHCOS to use syntax like podman > 1.4 in the treefile.
Also add new unit tests and tweak existing compose tests to test this
functionality.
Same motivation as
7392259332
I think we should encourage removing the writable bits from
executables. This has happened to me:
https://thomask.sdf.org/blog/2019/11/09/take-care-editing-bash-scripts.html
And not having the writable bit may help prevent hardlink
corruption with OSTree in some cases.
We can't do this by default, but add a convenient treefile option
for it.
This starts out by just doing this for RPMs, but I'll add
a secondary pass which does it during postprocessing soon too.
We are have been pretty inconsistent about this; I think while it's
true the project is called "OSTree", "RPM-OSTree" is just annoying
to type and looks weird. `rpm-ostree` requires much less
"shift key gymnastics".
Also, the "proper name" style like RPM/OSTree is best when
something can be viewed more "abstractly" - there are multiple
RPM implementations (historically) and OSTree also is a *concept*
in addition to an implementation.
rpm-ostree though is much more of a "concrete" thing so it
makes more sense to use it as a "project name".
In Fedora CoreOS, we have a "coreos-pool" repo from which all packages
in lockfiles are tagged for reproducible builds. This repo is shared
across all streams, including those on f31 and f32.
Thus, it makes no sense for composes to ever pick packages unconstrained
from the pool without being guided by a lockfile. Otherwise, one can
easily end up with e.g. f32 packages in an f31 compose.
Add a new `lockfile-repos` for this which is only used for fetching
lockfile packages and nothing else. For example, this will allow
`cosa fetch --update-lockfile` to Just Work as expected by only fetching
new packages from regular yum repos.
We need to be friendlier to people who are transitioning from
"traditional" yum managed systems. This patchset starts to lay
out the groundwork for supporting "intercepting" binaries that
are in the tree.
For backwards compatibility, this feature is disabled by default,
to enable it, one can add `cliwrap: true` to the manifest.
To start with for example, we wrap `/usr/bin/rpm` and cause it
to drop privileges. This way it can't corrupt anything; we're
not just relying on the read-only bind mount. For example nothing
will accidentally get written to `/var/lib/rpm`.
Now a tricky thing with this one is we *do* want it to write if
we're in an unlocked state.
There are various other examples of binaries we want to intercept,
among them:
- `grubby` -> `rpm-ostree kargs`
- `dracut` -> `rpm-ostree initramfs`
- `yum` -> well...we'll talk about that later
We haven't been consistent about doing this; I personally
think rustfmt is a big aggressive with the line wrapping
but eh, consistency is more important.
And heh so I tried to `git push --set-upstream cgwalters` and
that failed because there was an already extant `rustfmt`
branch from a while ago...looking at that code it got lost
in the CI refactoring - we're not running `build-check.sh`
at the moment.
Move the rustfmt bits into `codestyle.sh` which is closer
to where it should be anyways.
This is just a cleaner arrangement to make the separation more explicit.
It also matches what most other wrapper crates do.
One advantage of this is that we can tell cbindgen directly that we
don't want it to ever export symbols from `libdnf-sys`.
Related discussions in:
https://github.com/coreos/rpm-ostree/pull/2047
When we ran rustfmt, it converted our bare `extern` blocks to
`extern "C"` which has a different meaning apparently.
This caused cbindgen to try to interpret the structs, and it barfed
on the newtype void wrappers.
Looking at libgit2-rs, it seems to use these "uninstantiable types"
instead.
Prep for using `rustfmt`.
I previously ran out of steam in the switch and wanted to
get the PR out for feedback before continuing, but it turns
out I basically stopped 2 meters from the finish line. Completing
the switch from `failure` → `anyhow` was quite easy.
The failure crate isn't actively developed anymore. The
main benefit of `anyhow` is it uses the standard error type.
More info:
https://docs.rs/crate/anyhow/1.0.28
Start the porting process.
Note that the `envsubst` crate has a public dependency on
`failure`, so we need to start mapping its errors.
The ostree test suite was creating deployments manually
(skipping the rpm-ostree upgrader layer which would write history)
and then calling `rpm-ostree cleanup` which tried to open the
history dir and failed.
Just return early if there's no history directory when we're
asked to clean up.
In FCOS we have a kola test that basically does `rpm -q python`.
It's...a bit silly to spawn a whole VM for this. Ensuring that
some specific packages don't get included has come up in a few
cases.
I think FCOS/RHCOS at least will want to blacklist `dnf` for example.
And as noted above, FCOS could blacklist `python`.
One major benefit of doing this inside rpm-ostree is that one
gets the full "libsolv error message experience" when dependency
resolution fails, e.g. blacklisting `glibc` I get:
```
Problem 79: conflicting requests
- package coreos-installer-systemd-0.1.2-1.fc31.x86_64 requires coreos-installer = 0.1.2-1.fc31, but none of the providers can be installed
- package coreos-installer-0.1.2-1.fc31.x86_64 requires rtld(GNU_HASH), but none of the providers can be installed
- package glibc-2.30-10.fc31.x86_64 is filtered out by exclude filtering
- package glibc-2.30-7.fc31.x86_64 is filtered out by exclude filtering
- package glibc-2.30-8.fc31.x86_64 is filtered out by exclude filtering
- package glibc-2.30-5.fc31.i686 is filtered out by exclude filtering
- package glibc-2.30-5.fc31.x86_64 is filtered out by exclude filtering
- package glibc-2.30-10.fc31.i686 is filtered out by exclude filtering
```
This makes it easier to flatten a treefile. To do this, we make having a
workdir optional on the Rust side, and on the C side, just move
`--print-only` handling up front.
The parent of a `Path` of `"foobar"` is actually `""`, because
`parent()` wants to return a slice of its buffer:
https://github.com/rust-lang/rust/issues/36861
Just do a trivial wrapper around it to correctly return `"."` instead.
Hit this when trying to pass a treefile in the working directory.
This allows replacing the `.` in automatic version increments
with whatever one wants (as long as it's a single ASCII character)
right now.
The specific motivation here is for at least RHEL CoreOS to use
`version-suffix: "-"` so that its versions can become valid
semantic versions.
Related: https://github.com/coreos/rpm-ostree/issues/1954
E.g. the generation timestamp, repos that were enabled, and their
generation timestamps.
This is just generally useful, though I'd like to make use specifically
of the new `metadata.generated` key in FCOS to drive versioning:
https://github.com/coreos/fedora-coreos-releng-automation/pull/50
All this does is put the immutable bit on the target directory.
The intention is to replace this bit to start:
8b205bfbb9/src/create_disk.sh (L229)
However, the real goal here is to add code in this file
to handle redeploying the rootfs for Fedora CoreOS which
combines OSTree+Ignition:
https://github.com/coreos/fedora-coreos-tracker/issues/94
Basically doing this in proper Rust is going to be a lot
nicer than shell script in dracut modules. Among other
details, coreutils `mv` doesn't seem to do the right thing
for SELinux labels when policy isn't loaded.
This is the rpm-ostree equivalent of `dnf history`. As opposed to the
history of the refspec (i.e. `ostree log`), this shows the history of
the system, i.e. the refspecs the host deployed, checksums, versions,
layered packages, etc... The amount of details remembered is similar to
what shows up in `status`.
There's definitely some further enhancements possible (e.g. printing
package diffs, displaying rollbacks), though this seems in good enough
shape as a first cut.
Closes: #1489Closes: #1813
Approved by: cgwalters
We were merging some keys with list values by overriding the full list
instead of appending. Fix those.
Also add some missing keys we weren't merging at all.
Closes: #1908
Approved by: cgwalters
We have contacted all contributors to the code in `rust/` and
that code is now all relicensed under the "standard Rust license"
of `Apache 2.0 OR MIT`.
[Due to an accident](https://github.com/projectatomic/rpm-ostree/issues/1890),
some GPLv2+ code was imported in the C side, and we're unlikely to
easily change that now. Make this more official by adding the GPLv2.
I'd like to go through the C code and add SPDX and possibly investigate
relicensing some of the GPLv2+ code to LGPLv2+ but, not right now.
For a bit more about Rust and SPDX, see [this issue](https://github.com/rust-lang/cargo/issues/2039).
Closes: https://github.com/projectatomic/rpm-ostree/issues/1890Closes: #1897
Approved by: jlebon
I wanted to use this in another project, and I don't think
we're the only ones. I extracted the code, added better docs
and tests, and published to crates.io:
https://crates.io/crates/openat-ext
* treefile: Rework test helper
Have each test clearly hold its tempdir; this way we can have a simple
helper function rather than a struct with an unreferenced parameter.
Also use `utils::write_file`.
Prep for further test work.
* treefile: Add a basic test for includes
Prep for adding an arch include test.
* treefile: Support `arch-include`
A long time ago we added architecture-specific package lists
via e.g. `packages-ppc64le`. Much more recently we added
support for having the `include` key be a list - multiple includes.
By combining these two and supporting architecture-conditional includes,
we've effectively added architecture-conditionals to *all* keys.
Notably we want this for Fedora CoreOS today which is using
`remove-from-packages` on `grub2-tools` which isn't present on
s390x.
* ci: Bump cosa build timeout
Looks like another OpenStack perf regression.
I'm working on having Silverblue inherit from Fedora CoreOS. But
conceptually it also inherits from (parts of) Workstation.
It is just easier if we support multiple inheritance, then I don't
need to think too hard about how to make it a single inheritance chain.
Closes: #1870
Approved by: jlebon
We had essentially the same test in the `lockfile` and `treefile`
modules. Just dedupe those and move it to `utils`.
Closes: #1867
Approved by: cgwalters
When manually writing lockfile overrides (see previous commit), it's
sometimes easier to not have to specify the SHA256 of the package. For
example, in FCOS, all packages on development and production streams
will be sourced uniquely from coreos-pool, so there's no question of
where the package will come from. It's of course also easier in the
context of local development.
Another motivation for this though is a subtle interaction between
Fedora infra and the way we'd like to implement lockfile management: we
want the override process to be PR-based, with a privileged bot in the
backend tagging new overrides into the pool as necessary on merge.
However, packages built in Koji are initially unsigned, and so we can't
actually *know* what the SHA256 of the package will be until it's signed
and tagged into the pool by the bot.
Closes: #1867
Approved by: cgwalters
Teach `rpm-ostree compose tree` to accept multiple `--ex-lockfile`
arguments. In this case, later lockfiles can override the NEVRA for
packages specified in previous lockfiles.
This will be used in the FCOS pipeline, where we want to be able to have
a shared "base lockfile" and then stream-specific "override lockfiles".
I contemplated making this an `include: ...` key instead similar to the
manifests, but I'm not sure that paradigm fits as nicely for lockfiles.
Making it separate switches instead also makes it trivial to implement
in cosa.
(And of course, this is all still prefixed with `--ex` which means we
are at liberty of changing this interface later on after gaining some
experience with it).
Closes: #1867
Approved by: cgwalters
There are two reasons for this:
1. I'd like to add overrides semantics to lockfiles, and keying by the
package name only makes this much easier.
2. I'd like to make the digest optional, and keeping it as a tuple makes
this awkward.
A map seems natural too since it makes it more clear that we don't
expect multiple specifications for the same package name.
Another tiny advantage is that it's easier to process with e.g. `jq`.
Closes: #1867
Approved by: cgwalters
I wanted to modify the lockfile specification, but then remembered that
it currently lives in two places right now: once on the Rust side where
it's deserialized, and once more on the C side where it's serialized.
If we could write the lockfile from the Rust side, then we wouldn't have
to deal with the `GVariantBuild` and `json-glib` goop, and instead
we could consistently use serde against the same struct for both
serialization and deserialization.
But there isn't an easy way to do this given that the state to be
serialized is intrinsically linked to libdnf.
So this patch takes the next step in our oxidation process by adding a
minimal `libdnf_sys` module which allows us to call `libdnf` functions
from Rust. This is not the prettiest code I've written, and there's
definitely some polishing that could be done. But I think overall it's a
move in the right general direction: as we oxidize more things, we'll at
some point *have* to integrate more tightly with the C side in a
bidirectional way, instead of the "one-way" approach we've been using so
far.
For this patch specifically, in exchange we get a unique source of truth
for the lockfile spec, just like the treefile, and we drop a lot of C
code in the process.
Closes: #1867
Approved by: cgwalters
Let's just make it really obvious that this is the function that reads
the lockfile.
Prep for also having a symmetric `ror_lockfile_write` function.
Closes: #1867
Approved by: cgwalters
Move hashing to the Rust side so that we can easily hash over the final
set of inputs after parsing. This means that we now hash over all the
externals, like `add-files` references, any `postprocess-script` script,
and `passwd` and `group` files.
The original motivation for this was that hashing over a reserialized
version of the treefile was not deterministic now that treefiles include
hash tables (i.e. `add-commit-metadata`). So I initially included each
individual treefile as part of the hash.
I realized afterwards that just switching to `BTreeMap` fixes this, so
we can keep hashing only the final flattened reserialized treefile so we
ignore comments and whitespace too. But since I already wrote the patch,
and it fixes a real issue today... here we are.
One notable change though is that we now hash the treefile in non-pretty
mode to increase the chances that the serialized form remains stable.
Ironically, this change is likely to cause a no-op commit once it gets
to pipelines which iterate quickly. All for the greater good though.
Closes: #1865
Approved by: cgwalters
Add support for a new `add-commit-metadata` key in the treefile so that
we can directly specify commit metadata we want to inject from there.
This will be useful in Fedora CoreOS, where we'll have separate
treefiles for each streams, each with stream-specific metadata values
required.
Closes: #1865
Approved by: cgwalters
Besides allowing ${releasever}, only do the substitution as the final
pass after merging the treefiles for all the keys (currently ${basearch}
and ${releasever}) instead of doing it per parse. This way we have the
expected semantics where one could do:
```
include: "fedora-coreos.yaml"
releasever: "42"
```
and have that releasever used.
Fixes#1809
Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>
Closes: #1848
Approved by: cgwalters
Using glib_json to parse the lockfile yields some oddities like
everything being wrapped in a GVariant. Let's leave the parsing to serde
in the Rust side of things. Hopefully that'll make the lockfile easier
to extend in the future.
Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>
Closes: #1851
Approved by: jlebon
The use case for `ostree-layers` is to support injecting non-RPM
content in a more flexible way than can be done with `add-files`,
and also without dropping all the way to split composes.
This starts with support on the `compose tree` side but down the
line I'd like to make it more convenient to do *client* side too.
For `ostree-override-layers` this is mainly a development thing
for tools like coreos-assembler. Rather than building an RPM
we just `make install DESTDIR` then commit and add to
`ostree-override-layers`.
Closes: #1830
Approved by: jlebon
Use `pub(crate)` on the ffiutils functions since it's just for us. Drop
unnecessary `pub` fields on `TreefileExternals`.
Closes: #1812
Approved by: lucab
This mostly involves adjusting our path uses to be anchored, e.g. at
`self::` or `crate::`. I took the opportunity to also tweak `use`
ordering so that we consistently list modules in external crates first
before our own.
Closes: #1812
Approved by: lucab
Now that we no longer support el7, let's peg our minimum `rustc` version
to the latest Rust module available in el8 instead. That's 1.31.0 right
now.
And drop use of deprecated `trim_left_matches()` (it was deprecated in
1.33.0, which is currently used in the other testsuites that pull
`rustc` from the Fedora repos; we might want to eventually just use the
`RUST_MIN_VERSION` in all the testsuites to not get caught in the middle
like this again).
Closes: #1787
Approved by: cgwalters
And this (for now at least) completes the epic journey of the
"where's the kernel"? With this it's found solely in
`/usr/lib/modules/$kver`.
There are a few reasons to do this; most prominent is that
it avoids duplicating the content as the locations may have
different SELinux labels.
Closes: #1773
Approved by: jlebon
I'd like to add a new `boot-location: modules`. In prep
for that, let's remove the legacy `both` which drops into
`/boot`.
The libostree support for handling `/usr/lib/ostree-boot` has
existed for over 4 years:
```
commit 37a059925f6b96d30190b65bee6bdde0ae1c6915
Commit: Colin Walters <walters@verbum.org>
CommitDate: Sun Nov 30 23:14:05 2014 -0500
deploy: Ensure that we can deploy using only /usr/lib/ostree-boot
```
I think we assume now that no one is now making *new* treecomposes and needs
a newer rpm-ostree and that they expect people to be able to use as an
upgrade target from a libostree that predates that.
Closes: #1773
Approved by: jlebon
Now that we support YAML, it's a gotcha/eyesore that some of our options
use underscores rather than dashes. Let's be nice and switch those few
options over, while of course still supporting the previous name.
Co-authored-by: Colin Walters <walters@verbum.org>
Closes: #1749
Approved by: cgwalters
Instead of the `Strict` and `Permissive` variants of
`TreeComposeConfig`, just collapse all the excess fields in a new
`extra` member and check for any keys there in the strict YAML path.
This will also allow us to drop the hardcoded list of architectures in
the next patch.
Closes: #1749
Approved by: cgwalters
Add a `basearch` key to the manifest. This can be used at compose time
to assert the architecture the compose is running on. Though my
motivation is for the common case where it gets omitted from the input
manifest and gets automatically added by rpm-ostree into
`/usr/share/rpm-ostree/treefile.json` for introspection on the client.
(The crucial part here is that the treefile created by rpm-ostree
remains deserializable into a `TreeComposeConfig`).
Closes: https://github.com/coreos/fedora-coreos-tracker/issues/154Closes: #1766
Approved by: cgwalters
We're really using this variable to substitute `${basearch}` and find
basearch-specific packages. Let's rename the variable to make that more
obvious.
Closes: #1766
Approved by: cgwalters
Manually patch `file_contexts.subs_dist` so that `/home` is equivalent
to `/var/home`. This is required now that the generated homedirs rules
use `/var/home`. Otherwise, `matchpathcon` for example will return wrong
results.
This patch also includes the *removal* of `/var/home -> /home` so that
we're not dependent on this selinux-policy patch making it at the same
time as downstream:
https://src.fedoraproject.org/rpms/selinux-policy/pull-request/14
(See the conversation there for more information.)
Closes: #1754
Approved by: cgwalters
This uses the `Context` feature of the failure crate to make error
messages more useful when we fail to open a file. The difference with
`map_err` is that one can still obtain the underlying error from the
context if need be. Though surprisingly, the normal `Display` for a
`Context` doesn't include the original error, so we essentially have to
do a prefix here (see [1]).
Before:
```
error: Failed to load YAML treefile: No such file or directory (os error 2)
```
After:
```
error: Failed to load YAML treefile: Can't open file "treecompose-post.sh": No such file or directory (os error 2)
```
[1] https://github.com/rust-lang-nursery/failure/issues/182Closes: #1735
Approved by: cgwalters
Obviously doing these trivial functions in parallel doesn't matter,
but I wanted to dip my toes into Rayon (and Rust parallelism in
general).
`try_for_each()` is just...staggeringly simple and beautiful.
Closes: #1727
Approved by: jlebon
This was omitted since in practice we aren't actually testing it,
the container path is mostly via `ex container` which uses keyfiles.
Closes: #1701Closes: #1702
Approved by: jlebon
Avoids passing an allocated buffer from Rust to C; there's
controversy in the PR I sent to rust-lang around defining this as
supported.
Closes: #1691
Approved by: jlebon
Since use of the `failure` crate has been a success, let's use
it a bit more. The big thing to convert left is `treefile.rs` which
does need a custom error so we can stop abusing `io::ErrorKind::InvalidInput`.
Closes: #1690
Approved by: jlebon
Create a new `_new_` naming convention, and extend the FFI
documentation to describe the new state as well as background assumptions.
Closes: #1688
Approved by: jlebon
Change our existing "view as [u8] API", and also add one
that does a view as `OsStr`. The motivation for the latter
is I noticed ithat `OsStr::from_bytes()` *doesn't* copy,
or rather it just copies the pointer value. Rust's lifetime
inference ensures that the returned lifetime matches the input array.
I think the previous code in `treefile.rs` was confused about this.
Closes: #1688
Approved by: jlebon
I was going to add another usage of this function, and I think the
gerror stuff is unnecessary - if we are handed a bad file descriptor
(or a fd pointing to a regular file) that's something where we should
just abort.
While we're here, I'd like to codify expected usage in the function
names here. If you like this I'll e.g. also change `str_from_nullable`
to `ffi_view_nullable_str`.
Closes: #1685
Approved by: jlebon
In a lot of places we're abusing `io::Error(io::ErrorKind::InvalidInput)`
which is both verbose and inaccurate really. Maybe in some
places we should be defining custom errors, but eh.
I like the `failure` crate. Use it in just `utils.rs` for now.
Tweak our error handling FFI wrappers to accept `Display` since
all we do is convert the error to a string.
Closes: #1675
Approved by: lucab
This turned out to be messier than I thought, because of two primary
factors; the biggest mess here of course is the indirection
through the DBus API.
The other problem is that previously we passed the string to render
each time, and with current indicatif that'd trigger a rerender.
Since (usually) don't change the "prefix string", rework the API.
Change the "percent/n_items" bits to use autocleanups as well, and
to take the prefix string as an initial argument.
Since the state expands to multiple components, also change the
API to use the `0-initialized` pattern rather than trying to
return an aggregate.
We also gain a "sub message" which we use to display e.g.
package names as we're doing checkouts. Note this ends up
at the end, since otherwise everything else jumps around.
Closes: #1661
Approved by: rfairley
External tools often want to parse the ref; for example coreos-assembler
currently does so. Let's ensure `${basearch}` is expanded with
`--print-only` so they can parse that JSON to get the expanded version
reliably.
Implementation note: this is the first Rust code which exposes a
"GLib-like" C API, notably with GHashTable, so we're making more use
of the glib-rs bindings.
Closes: #1653Closes: #1655
Approved by: jlebon
This currently requires a `--i-know-this-is-experimental` flag;
I know it'd be a bit more consistent to have it under `ex`, but
what feels weird about that is *most* of the `ex` commands people
use are client side. This is where we want it to ultimately end
up.
We've landed a lot of prep patches, but I know there's still
a notable amount of code duplication with `compose tree`. What's
left is about ~700 lines but it's mostly not hard/complex code
anymore.
In the future, I'd like to extract more of the compose code
to a `rust/src/compose.rs` or so, but I think this is sustainable
fow now.
My high level goal is to get this into coreos-assembler and stand
up a Silverblue build that uses it.
Closes: #1512
Approved by: jlebon
The `lib.rs` file was starting to accumulate, and I want to do this
before adding more. I just made this up, but an `ffi` submodule
in each file seems to work well. It isolates the FFI consumption
there still (so e.g. if we want pure Rust unit tests, we don't
need to deal with FFI).
Closes: #1646
Approved by: jlebon
While serde gives us type checking, it of course doesn't understand
semantics beyond that. One example is checking the compatibility of
`add-files` entries with the OSTree model. This is something we can do
upfront early on to avoid surprises for users.
Also tweak the docs to reflect this new check.
Related: #1642Closes: #1643
Approved by: cgwalters
I need to do some hunting for a "ffi helper" crate. I kind of
understand why these things aren't in the std library.
Anyways this is easy for now, prep for adding more types.
Closes: #1641
Approved by: jlebon
On the plus side, when submitting a patch to Github, no one knows
how long it took you to figure out...
Anyways so this reduces redundancy. I double-checked the list.
I was inspired to pick this back up after seeing a Rust code
snippet somewhere noting that macros defined inside a function
can capture variables, which simplifies this even more.
Closes: #1631
Approved by: jlebon
Supporting ancient systemd is painful, though given that it may end up
in RHCOS at least, it seems worth the effort.
Basically, the big changes here are:
- avoid using `_SYSTEMD_UNIT` since it might be missing in some entries
- also grep for the el7 version of the systemd msg when a service fails
- use `_TRANSPORT=stdout` for the error msg in case of ordering issues
Closes: #1601
Approved by: cgwalters
Try to tease out a bit more info from the journal by looking at the
systemd message when the service transitions to the dead state or even
looking at the OSTree output itself.
Example outputs:
```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
error: opendir(ostree/deploy/fedora-atomic/deploy/887c95887a3047a60372016a0d84536530755b60df3cca33c819f7606e220adf.0): No such file or directory
check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```
```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
ostree-finalize-staged.service: Failed with result 'timeout'.
check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```
Closes: #1601
Approved by: cgwalters
Sample output:
```
$ rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```
(Though open to tweaking it).
I also played with directly invoking `journalctl` for the user, but that
can get really spammy with e.g. `os-prober` output and such.
I wrote this in Rust using journal API wrappers because I also plan to
implement the `history` command in Rust and will also enhance that new
`journal` module there for that.
Requires: https://github.com/ostreedev/ostree/pull/1750
Requires: https://github.com/jmesmon/rust-systemd/pull/54
(Though I've pointed the manifest at my branch for now for CI).
Closes: #1567Closes: #1601
Approved by: cgwalters
This way the fds are always known to be at the start. For e.g.
`add-files` we need to both checksum them and then copy them
later.
Closes: #1600
Approved by: jlebon
Now that we have `CUtf8`, let's just store the serialized JSON
as a string, owned by the Rust side. This way we can drop the
`serialized_treefile` buffer we were passing around and simplify
various bits of code. Most notably, we only serialize the JSON
once (via Serde) rather than also doing it again in the C side.
Closes: #1600
Approved by: jlebon
This follows up to https://github.com/projectatomic/rpm-ostree/pull/1576
AKA commit 2e567840ca - we now process
treefile inheritance in Rust code. Previously for elements which
reference external files (`postprocess-script` and `add-files`)
we'd hardcoded things to only look in the first context dir.
Now we open file descriptors in the Rust side for these "externals"
as we're parsing, and load them C side. Hence we'll correctly handle
a `postprocess-script` from an included config.
Other advantages are that the include handling was ugly un-typesafe C code
with no unit tests, now it's memory safe Rust with unit tests.
The downside here is I ended up spelling out the list of fields
again - there's probably a way to unify this via macros but
for now I think this is OK.
Closes: #1574
Approved by: jlebon
The advantage of this over CStr is that Rust knows it's UTF-8
too. I also tweaked our path code to use String, and only
view it as a `Path`. This avoids having to `unwrap()` later
back to a `str`.
Closes: #1588
Approved by: jlebon
It's really an array of pairs. Tripped over this while working
on the pure-Rust treefile parsing on the `misc-tweaks` compose test.
Closes: #1581
Approved by: jlebon
Prep for moving more of our parsing into Rust. The main
thing here is that for JSON, we need to continue to ignore
unknown fields. It took me a little while but I eventually
figured out that using `#[serde(flatten)]` works for this.
Seriously: serde is freaking amazingly awesome.
Closes: #1580
Approved by: jlebon
Apparently I was just totally confused into thinking `OsStr`
was `NUL` terminated. I don't know where I got that idea
but a quick inspection of the Rust source code makes this
clear. I may submit a PR to the docs.
So we have two choices: allocate, or internally use a `NUL`-terminated
type. Let's just use `Box<CStr>` internally. Note I chose that
over `CString` as the latter is theoretically mutable, and when
passing data back over to C we don't have ownership/mutability
guarantees.
Closes: #1563
Approved by: jlebon
If we're going to scale out our oxidation, let's follow
the path of Firefox (and other projects) further and use
cbindgen: https://github.com/eqrion/cbindgen
It's actually nice that `cbindgen` is packaged today in Fedora,
but I doubt it is elsewhere; we may end up needing to push
that forward, or just vendor it via a `build.rs` script and Cargo.
I chose to rename things to `ROR`/`ror_` since it's shorter. I
am tempted a bit to rename our internal functions to just `ro_` to
or so.
Closes: #1516
Approved by: jlebon
I'm trying to have a more opinionated model where custom builds
use inheritance, and currently one can only have a single
`postprocess-script`.
Further, in YAML it's very convenient to use inline vs external
data.
Closes: #1527
Approved by: jlebon
It turns out we basically have to slap an `Option<T>` around
everything, (in particular `bool` etc.) we need to be able
to distinguish in (I believe) all the cases between
"value unspecified" and "value provided".
Concretely it didn't work to try to set `machineid-compat: false`
in an included yaml treefile becuase it was just defaulted to `true`
by the toplevel.
Down the line we should move all of the parsing into Rust
and have two different `struct` types for "YAML we load" versus
"verified treefile".
Closes: https://github.com/projectatomic/rpm-ostree/issues/1524Closes: #1525
Approved by: lucab
This teaches the client to fetch packages from URLs directly so that one
doesn't have to `curl` first and then install. Supported anywhere
package filenames are allowed (notably: `install` and
`override replace`).
One neat things about this is that we download the file into an
`O_TMPFILE` and then pass on ownership of that fd directly to the
daemon. So at no point are the packages actually laying visible on the
system. (Assuming the filesystem supports `O_TMPFILE` that is).
This adds direct linking to libcurl and openssl, two libraries which we
were already pulling in indirectly.
Closes: #1508
Approved by: cgwalters
This is for: https://github.com/projectatomic/rpm-ostree/issues/718
But I'm not going to close that issue as this only does the server
side, and I think we should support it client side too.
Since I wrote that issue, we ended up skipping the `dnf_transaction_depsolve()`
API, and hence we don't need to block on a libdnf change. So
this was quite simple.
Closes: #1513
Approved by: jlebon
It currently works to convert *toplevel* configs to YAML, but
if one wants to make use of `include:`, today you'd have to specify
dummy `ref` entries on the includes.
Further, for rojig-only mode, one doesn't need `ref`.
Closes: #1511
Approved by: jlebon
Otherwise we risk rejecting perfectly valid treefiles. E.g.
fedora-atomic only defines packages in the `-base` file. Let's just move
the check to after having processed all the includes, right where we
collate packages from all the various entries.
The FAHC treecompose is hitting this right now.
Closes: #1500
Approved by: cgwalters
The rojig spec is almost entirely rpm-ostree implementation details;
let's not have lots of people fork/duplicate it. Rather add the bits
of rojig to the treefile that people need to define (most notably
the name).
Prep for stabilizing rojig.
I had a few false starts with this PR; managing ownership/lifetimes
across C/Rust is just complicated. I got bit hard by the fact that
the workdir in `--unified-core` is really dfd-relative, and had to
do a dance to propagate the dfd into rust, as well as down into
the rojig builder.
Closes: #1484
Approved by: jlebon
In a later patch I'm going to add more API; basically rather
than doing the JSON parsing from C, we can add APIs to directly
access the treefile object. This also demonstrates how we
can do more extensive APIs, in particular implement an "object"
in Rust.
The ownership across the FFI boundary becomes nicer here too,
we don't need to do a dance with the fd.
For writing this I found
http://jakegoulding.com/rust-ffi-omnibus/objects/
quite useful, as well as
https://github.com/rust-lang/regex/blob/master/regex-capi/src/rure.rsCloses: #1474
Approved by: jlebon
I was going to add a new API that returns a pointer, and that just
wasn't going to work with the trait AFAICS; we'd need to be generic
across many different types. Let's use plain functions; it's slightly
less elegant but is clear and works.
Closes: #1474
Approved by: jlebon
Follow up to: https://github.com/projectatomic/rpm-ostree/pull/1459
We now honor arch-specific packages in YAML, and reject unknown
architectures. I looked a little bit at how to avoid having hardcoded
arch lists, but it doesn't seem worth it right now.
Closes: #1468
Approved by: jlebon
Let's not make the same mistake we did with JSON where typoing a
field means it's silently ignored. This actually caught a bug
in a YAML usage we had:
```
error: Failed to load YAML treefile: unknown field `install_langs`, expected one of ... `install-langs` ...
```
Yes, this is a compatibility break with the feature we just announced
but...I seriously doubt anyone (that isn't known to me) has converted
yet, and if they are excited enough to start using a two-week-old feature
they can adjust.
Closes: #1459
Approved by: cgwalters
We weren't checking for an error from `dup()`, and further it shouldn't
be necessary. This is the best I could come up with, though it's
obviously not pretty.
Closes: #1444
Approved by: jlebon
It is actually really nice that there's One Canonical Style, even
if I sometimes don't like some details of what rustfmt does.
Closes: #1444
Approved by: jlebon
Further prep for adding more code here. The `lib.rs` then is the
collection of glue functions; perhaps in some ideal future it could
be generated even.
Closes: #1444
Approved by: jlebon
We actually want systemd's `ConditionFirstBoot` to fire. The
primary rationale here is that we're adopting Ignition for Fedora CoreOS,
and having `ConditionFirstBoot=` function will help a lot, as the idea
is it only runs once.
However, I discovered that this breaks the `units` directive for example,
as systemd blows away all the unit state in `/etc`. The correct thing
to do from the start is to use presets. We could add an implementation of
`units` which works with this on and instead writes a preset file but...eh.
My plan is to at some point introduce an "epoch" and flip various defaults,
this one, `tmp-is-dir`, the passwd file handling, etc.
See: https://github.com/dustymabe/bootengine/pull/11Closes: #1425
Approved by: jlebon
I was surprised when I fired up GDB and found out that my
`etc-group-members` modifications weren't being obeyed. It turns out
serde was looking for the wrong field name. I did a quick sanity check
and found a few others.
Closes: #1417
Approved by: cgwalters
Automake was warning about duplicate `clean-local` definitions, let's
do the nonrecursive automake dance.
And while I'm here, let's add some Rust unit tests that actually run
on `make check` too, since the whole unit testing bits of Rust are
awesome.
(I also tweaked the propagate bits to use the nicer `is_null()` method)
Closes: #1389
Approved by: jlebon
Let's modernize and start supporting YAML treefiles. I'll dare make the
sweeping generalization that most people would prefer reading and
writing YAML over JSON.
This takes bits from coreos-assembler[1] that know how to serialize a
YAML file and spit it back out as a JSON and makes it into a shared lib
that we can link against. We could use this eventually for JSON inputs
as well to force a validation check before composing.
If we go this route, we could then turn on `--enable-rust` in FAHC for
now and drop the duplicate code in coreos-assembler.
[1] https://github.com/cgwalters/coreos-assemblerCloses: #1377
Approved by: cgwalters
