IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
This moves the users/groups validation logic to Rust, taking care of
all the treefile check-passwd/check-groups knobs.
As all the passwd-handling logic has been ported to Rust, it also
drops the stale C source/headers/imports.
In preparation for publishing this to crates.io and moving
into e.g. github.com/ostreedev/ostree-host-rs
So that ostree upstream can move forward with more Rust on the
"top level".
This tweaks the data and types parsed from `check-passwd` and
`check-groups`, so that it can be more easily iterated and
consumed by passwd-handling logic.
It's a bit silly to have a two bridged functions here; instead
have just a single one on the C++ side that calls multiple
on the Rust side.
Prep for moving more to Rust.
The manual `std::mem::drop()` bits are ugly; while we can do
function pointers from Rust to C++, let's just add the obvious
high level wrapper in Rust that accepts a `FnOnce()`.
Note in one instance we directly pass a function pointer which
is quite clean.
I was thinking about privilege separation today with
systemd units, and that led me to the problem of "lifecycle binding".
We really want e.g. `systemctl stop rpm-ostreed` to kill any
separate systemd units we're managing.
systemd already has a mechanism for this with `BindsTo=`.
And then I realized we weren't doing this for the systemd-tmpfiles
invocations in the `live.rs` code.
Generalize this into a small `isolation` module that fixes this
and several other things at the same time. I'd like to build
on this to further improve our multi-process isolation story
later.
This code really makes sense as a method on the treefile.
And when that's done, we no longer need to expose
`get_postprocess_script()` via cxx, so we can return a nicely
Rust native `Option<&mut File>`.
Port add-files handling to Rust.
Note that there's one very magical line of diff here worth calling out:
We dropped an interface from the cxxrs bridge, because both sides
are now Rust! The treefile code can directly return an `&mut File` reference
instead of needing to pass the raw fd as `i32`.
The ugly C code for this turns into shorter Rust with a unit
test, a lot less allocation (notice how we don't malloc `NUL` terminated
strings in so many places).
It turns out there's a naming clash between `to_string()` here in
Rust *and* introspection is incorrectly associating the method
with `ostree::Deployment` because of the naming prefix.
I don't like the use of `HY_GLOB` in the lockfile package matching. We
have all the information in the Rust object, so it's silly to condense
that to a single string in a hashmap.
Fix this by returning the `LockfileConfig` object itself and then adding
a function to fetch the list of locked packages. This allows the C++
side to see all the individual fields which makes filtering trivial.
The next step is moving all the code which needs the lockfile to Rust.
Then we can drop the shared `LockedPackage` type.
(I did start on converting `find_locked_packages`, though it requires
adding bindings for all the `HyQuery` stuff, which... isn't great (and
also runs into the fact that `hy_query_run` needs to return a
GPtrArray). I think instead of a 1:1 mapping, we'll probably want the
libdnf-sys API wrappers to provide some sugar for the common paths.)
In FCOS, we use "override" lockfiles to pin packages to certain
versions. Right now, we have separate overrides for each base arch we
(eventually want to) support. But that makes maintaining the overrides
cumbersome because of all the duplication.
Let's allow lockfiles to specify only the `evr` of a package, which is
just as good for FCOS, and means that we'll only have to maintain a
single override file for all the architectures.
Same motivation as https://github.com/coreos/bootupd/pull/163
Effectively what we're doing here is creating a human-readable subset
of the stack trace. This is nicer than having the calling functions
add with_context() because it's more verbose (gets duplicative at
each call site), easy to forget, etc.
Prep for more oxidation work. One notable improvement here is that about half
of the callers of the mega function `rpmostree_deployment_get_layered_info`
only wanted the base information, not the layered package lists
for example - so we were passing 4 `NULL`s to ignore those.
This Rust API returns a simple shared struct instead for those
cases. I also changed things so that `base_commit` is always
set, avoiding the callers needing to do that.
There's a huge difference between live updates that change
existing things, versus simply adding new packages (files).
The latter is really quite safe, and live layering is one
of the most requested features.
In prep for adding more methods, require the caller to identify
themselves.
For now this is `CliClient` - one could imagine in the future
we actually do direct DBus, but there's a whole other world
of stuff there.
Add more metadata that zincati needs, like `base-commit-meta`
which includes the `fedora-coreos.stream` key and the cosa basearch,
etc.
Also `Derive(Debug)` since it's used in a cache struct that also
derives debug, and that's a friendly thing to do in general.
This adds sufficient infrastructure to fully port the
`rpmostree-builtin-applylive.cxx` client code to Rust.
We just keep a stub entrypoint for now until we port
the rest of `rpmostree-builtin-ex.cxx`, at which point
a lot of C++ files go away.
The "finish" bits move from the daemon-oriented `live.rs`
into a new `rust/src/builtins` directory. I'd like
to try to more cleanly split up the Rust sources along
core(shared)/client/daemon directories in the future.
This stubs out sufficient infrastructure for us to register
as a client and call the Moo API.
A glaring problem here is the lack of extensive `glib::Variant`
bindings; that's covered in the next gtk-rs release.
My real goal was to try porting the `rpmostree-builtin-apply-live.cxx`
code entirely to Rust, but there's more to do to expose the
transaction helper APIs we have.
cxx-rs only supports a few basic types in `Vec<T>`/`CxxVector<T>`
and we need to pass an array of GObjects in a few cases.
Add a wrapper class hack instead of using `u64` so we at least
have some basic safety here and have a convenient place to
grep for later when we want to improve this.
This moves the `ror_lockfile_write` to cxx.rs, which brings us closer to
getting rid of cbindgen now.
There's one massive hack this uses, which is that we pass an array of
pointers to `DnfPackage` and `DnfRepo` objects as u64. We'll want to
circle back and fix that up once either cxx.rs supports natively arrays
of pointers, or we just come up with our own wrapper type for it.
But for now at least, this unblocks the cbindgen transition and hacking
on the lockfile code.
Came out of discussion in https://github.com/coreos/rpm-ostree/pull/2581
around some racy code for checking for the live commit object.
The reliability of apply-live depends on the
underlying commits not being garbage collected. Our diff logic
is in terms of ostree commits, not the physical filesystem (this
allows us to make various optimizations too).
Ultimately I think we should drive some of the live-apply
logic into libostree itself; we can more easily have an atomic
state file instead of the two split refs.
(Or perhaps what we should add to ostree is like a refs.d model
where a single atomic file can refer to multiple commits)
For now though let's rework the code here to write refs. We
retain the file in `/run` as just a "stamp file" that signals
that a deployment has had `apply-live` run.
More prep for https://github.com/coreos/rpm-ostree/pull/2388
This was actually also my first time really trying out the
latest gtk-rs `glib::Variant` API, which is one of the major
things we need to use to progress oxidation more.
This moves passwd/group compose preparation logic to Rust,
dropping all the remaining minor helpers related to JSON parsing,
file stream creation, and entries deduplication.
We should really have safety comments around our use of `.unwrap()`.
This Google kubernetes rpm-md repo appears to use seconds-since-epoch.
We could try to detect that, probably best done in libdnf.
I think it'd be better to get them to match the informal schema
for this actually though instead.
For now let's just log an error and not crash.
Closes: https://github.com/coreos/rpm-ostree/issues/2600
It turns out we accidentally added GPL'd code into the Rust
side, which wasn't intentional on my part and I think it's since
been copied around.
Honestly I think half of the problem is the gigantic
"blah blah blah GNU General blah blah" just makes people's eyes
glaze over. In contrast the `SPDX-License-Identifier` is short
and obvious.
So let's validate that in CI.
This follows a similar change in ostree:
https://github.com/ostreedev/ostree/pull/1439
If we merge this I'll do the C/C++ side too after that.
A lot of our output is outside of a "task"; the Rust binding
incorrectly made it a method on `Progress`. This is really
just a `println!()` that is backed by our dispatch system.
When we started using cxxrs, most of the glib-rs objects like
`OstreeRepo`/`OstreeSysroot` were owned by C++ and passed
down into Rust. That motivated the addition of the special
bridging infrastructure to re-create a glib-rs wrapper
type from what cxxrs wants (a `Pin<&mut T>`).
But now that we're adding more in Rust, we have the need
to pass these objects back into C++. In fact this will
hopefully soon because the default case as more of the
binary entrypoint becomes Rust.
Add another trait with a method `gobj_rewrap()` that converts
in the other direction. This implementation took me a number
of tries before I finally settled on simply using `mem::transmute()`.
There are a *lot* of caveats listed on the docs for that function,
but I think it really is what we want here. See the link for pending work
on a Rust RFC to enable safe transmutes for some cases, and I believe
that would cover this use case:
https://internals.rust-lang.org/t/pre-rfc-v2-safe-transmute/11431
I've verified this works in a separate patch, but this commit
also adds a simple test case - this goes all the way from:
Rust glib-rs `ostree::Repo` (holding strong ref)
-> Rust `Pin<&mut ostree_sys::OstreeRepo>`
-> (internal cxx-rs C bridge)
-> C++ `OstreeRepo&` reference
-> C `OstreeRepo*` pointer
Which is quite the dance if you think about it!
clippy complains about `to_*` not taking `&self` - I think
we can simplify this more by using our `write_file_with()` API
that we're using in other places. It was explicitly designed
with serde in mind. It's just more efficient and nicer to
serialize to a `BufWriter` instead of to a string, then
writing the string.
Specifically motivated by adding some docs in `treefile.rs` around
how to add a field, but I decided to just do a pass and
document everything at least a little.
View with e.g. `cargo doc --document-private-items`.
Pretty straightforward. Haven't moved `ror_lockfile_write` yet because
that's trickier to do and I'm still figuring out the most elegant way to
do this within cxx.rs' constraints.
Right now, we're using libdnf APIs from Rust via hand-crafted `extern C`
interfaces, which is extra dangerous because there is no signature
checking that happens at compile-time.
Until either we can automate libdnf bindings or use its C++ API directly
via cxx.rs, let's do some basic wrapping in C++ ourselves and use libdnf
through that API only instead. That gives us a lot more confidence and
makes the libdnf API feel more natural to use in Rust.
We trigger a librpm macro file load in many of our paths. Since the
default value shipped by rpm's macro file sets `_dbpath` to
`/var/lib/rpm`, we have to explicitly set that back to `/usr/share/rpm`
in those paths.
This became more problematic recently with libsolv v0.7.17 which fully
keys off of `_dbpath` to find the rpmdb path to load:
04d4d036b2
And it's not technically wrong; we really should make that macro not
lie. This is what this patch does by injecting an RPM macro file in our
composes which sets it to /usr/share/rpm. So then e.g. the `rpm` CLI
doesn't actually need the `/var/lib/rpm` backcompat link anymore, though
there's no harm in leaving it.
In the future, we should be able to drop this once we move all of Fedora
to `/usr/lib/sysimage/rpm` (see
https://github.com/coreos/fedora-coreos-tracker/issues/639).
Closes: #2548
Originally the Rust apply-live code was exposed from Rust to C
via bindgen. But when working on that, I hit the problem
that our output infrastructure was C...and the "reverse direction"
binding stuff was just ugly.
This PR again IMO shows the value of the investment in cxx-rs
because we can now seamlessly call back from the Rust side
into a "C++-ish" progress API, which the C++ side is updated
to use.
The level of indirection here is obviously pretty silly
because the main thing on the C++ output side is basically
a function dispatcher, but...I didn't want to try to rework
that into Rust fully yet. (But, the moment we do this
whole area will get a *lot* cleaner)
Anyways, in the end this makes it easy for the apply-live
code to output progress to the user which was sorely
needed.
Our output system is very confusing in that we bridge over
DBus in some cases and not others. In preparation for allowing
Rust code to call into the C++ progress system which contains
that delegation layer, rename the Rust progress to `console_`
to clearly show that it should only be invoked by code that
knows it's writing to a tty.
WITH_SWDB: Removed in 99309fbe04
WITH_GIR Removed in e2f2862bed
Also, most importantly: don't always reconfigure libdnf
This is a questionable default for the cargo `cmake` crate.
Building in Koji is failing I think due to timestamp issues
causing cmake to run twice.
This is now further migration towards Cargo/Rust possible
because we switched our main binary. We've had an internal
`libdnf-sys` crate for a while, but now it can take over
the build of the underlying library too (like many `-sys`
crates support).
This itself is just an incremental step towards migrating
the main rpm-ostree build system to e.g. cmake too (or
perhaps directly with the `cc` crate, not sure yet) and
driving it via `cargo` too.
First explicitly state that we're a workspace. AIUI
this is actually implicit today via our use of a `path`
dependency, but in the future we may have other sub-crates.
So let's make it explicit now.
Also move the libdnf dependencies directly to that sub-crate.
In RHCOS, we ship kernel development-related packages as an extension.
Those aren't really extensions that are meant to be layered onto the
host. They're meant to be used in a build environment somewhere to
compile kernel modules.
This makes it very different from "OS extensions" in at least two
drastic ways:
1. we don't want to do any depsolving (e.g. we don't want to pull in
`gcc` or something)
2. some of those packages may be present in the base already, but we
still want to redownload them
Hesitated putting this functionality in rpm-ostree, but I think in the
end it cuts from the benefit of moving this code to rpm-ostree if we
can't entirely get rid of the Python script it obsoletes. Plus, being
able to use the `match-base-evr` is still really useful for this use
case.
Let's add a new `kind` key to support this. The traditional extensions
are called "OS extensions" and these new extensions are called
"development extensions".
The latter is not yet part of the state checksum, so change detection
doesn't work there. I think that's fine for now though because the
primary use case is the kernel, and there we want to match the base
version. So if the kernel changes, the base would change too. (Though
there's the corner case of adding a new package to the list while at the
same version...)
We want to be able to enable more repos than those in the treefile when
downloading extensions. In RHCOS for example, the `kernel-rt` packages
come from a separate repo.
But also, once we support "development" extensions, we want to support
the case where devel packages come from another repo.
We now have bidirectional calling between Rust and C++,
but we are generating two static libraries that we then
link together with a tiny C++ `main.cxx`.
Let's make another huge leap towards oxdiation by
having Rust be the entrypoint. This way cargo natively
takes care of linking the internal Rust library, and
our C++ internals become the library.
In other words, we've now fully inverted from
"C app with internal Rust library"
to "Rust binary with internal C++ library".
In order to make this work though we have to finally
kill the C unit tests. But mostly everything covered
there is either being converted to Rust, or covered
elsewhere anyways.
Now as the doc comments in `main.rs` say...this is
a bit awkward because all the CLI code is still in C++.
Porting stuff to use e.g. `structopt` natively would
be a bit of a slog. For now, we basically rely on
the fact that the Rust-native CLIs are all hidden
commands.
Update submodule: libdnf
This demonstrates well the strength of the cxx-rs approach;
we can keep an API in C++ but add unit tests in Rust which
just works much more nicely.
Prep for https://github.com/coreos/rpm-ostree/pull/2502
which wants to drop the C++ unit tests.
Let's include the final extensions file in JSON format as part of the
output directory. A key difference from the input file (apart from YAML
vs JSON) is that this is post-filtering, so any extensions which were
removed because the architecture does not match are not present.
This JSON file will be used by cosa and the MCO. See discussions in:
https://github.com/openshift/os/issues/409
Until now with cxx-rs we'd been using it effectively as a better
cbindgen - we're exposing Rust code to C++ safely. This is
the first case of having Rust calling back into C++ using cxx-rs.
Mostly straightforward stuff. It taught me about the `matches!` macro,
which looks really useful.
Wanted to turn this on in CI, but there's still a bunch of clippy
warnings coming from the `cxx.rs` stuff and some of our unsafe blocks.
For example, it wants the `files` arg in `initramfs_overlay_generate` to
be `&[String]` instead of `&Vec<String>` but that would break cxx.rs (it
looks like cxx.rs does support slices, but it would require creating one
from the vector we have to create anyway).
This adds support for a new `rpm-ostree compose extensions` command`
which takes a treefile, a new extensions YAML file, and an OSTree repo
and ref. It performs a depsolve and downloads the extensions to a
provided output directory.
This is intended to replace cosa's `download-extensions`:
https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions
The input YAML schema matches the one accepted by that script.
Some differences from the script:
- We have a guaranteed depsolve match and thus can avoid silly issues
we've hit in RHCOS (like downloading the wrong `libprotobuf` for
`usbguard` -- rhbz#1889694).
- We seamlessly re-use the same repos defined in the treefile, whereas
the cosa script uses `reposdir=$dir` which doesn't have the same
semantics (repo enablement is in that case purely based on the
`enabled` flag in those repos, which may be different than what the
rpm-ostree compose ran with).
- We perform more sanity-checks against the requested extensions, such
as whether the extension is already in the base.
- We support no-change detection via a state SHA512 file for better
integration in cosa and pipelines.
- We support a `match-base-evr` key, which forces the extension to have
the same EVR as the one from a base package: this is helpful in the
case of extensions which complement a base package, esp. those which
may not have strong enough reldeps to enforce matching EVRs by
depsolve alone (`kernel-headers` is an example of this).
- We don't try to organize the RPMs into separate directories by
extension because IMO it's not at the right level. Instead, we should
work towards higher-level metadata to represent extensions (see
https://github.com/openshift/os/issues/409 which is related to this).
Closes: #2055
The service where this is hosted was intended to be temporary; support
overriding it so if it goes down in the future people can at
least use a systemd unit file override to change it easily.
This is a workaround for the non-customizability of the cxx-rs
propagation of Rust result to C++ exception. Right now we're
losing context. Work around this by formatting on the Rust
side at exit points, explicitly converting an `anyhow::Error`
by printing it in "single line context".
Since we're likely to gain more things like this, unify
this with `cxx_bridge_gobject::` into a single `cxxrsutil::`.
Gather the current diff of `/etc`, and filter out changes in
the tree which would overwrite it.
There is an OSTree API for diffs but it's a bit awkward, missing
some APIs in the Rust bindings and also `GFile` based unfortunately.
Doing this in Rust is nicer. The dirdiff code obviously needs
a lot more testing, but I think it's right.
This adds support for e.g.:
```
$ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2020-2908628031
```
This will find the Koji builds from the listed update, download
all the RPMs (that aren't debuginfo) and pass them for overrides
in the same way we support `override replace http://somewebserver/foo.rpm`
now.
We also support directly linking a Koji build:
```
$ rpm-ostree override replace https://koji.fedoraproject.org/koji/buildinfo?buildID=1625029
```
Bodhi has a modern HTTP+JSON API, and the lack of a Koji equivalent
drove me to create https://github.com/cgwalters/koji-sane-json-api
and we currently depend on an instance set up in the OpenShift CI
cluster.
I hope it shouldn't take long to deploy this in Fedora Infra,
but I don't want to block on it.
Also notably this still downloads *all* the other RPMs even
ones that aren't installed. Handling that truly correctly
would require moving this logic to the daemon and core.
All of this functionality is keyed off a `cfg(feature = "fedora-integration")`
that is detected by a Rust `build.rs` which parses the build environment's
`/etc/os-release` for now.
This is cleaner if we move the "catch and print errors" into
the Rust side. We lose the "print errors as red" part but
eh we can readd that later Rust side.
This makes the C++ side a bit uglier because the "variable mapping"
is more Rust-native but we only use it Rust side anyways.
(We can't yet move the basearch bits to rust because it depends on
libdnf, which requires buildsystem unification)
But all the unsafe FFI conversion drops out, as do the duplicated
C unit tests.
Prep for the [fedora-integration PR](https://github.com/coreos/rpm-ostree/pull/2420).
This also generalizes the "fetch URL to tempfile" code into
supporting multiple at once - it's much more efficient to do
it that way because we can reuse a TCP connection to servers,
parsed certificates etc.
(I'm going to start calling it `live-apply` instead of "livefs")
On one hand, this drops the unsafe bindgen glue. On the other
hand it demonstrates a notable current ergonomic shortfall of
cxx-rs in that it doesn't support `Option<T>`, so we represent
optional strings by converting them to empty strings.
(Relatedly I discovered in C++ there is `std::optional<>` but it's in C++17.
I assume we can depend on that but let's for now match libdnf's
usage of `-std=c++14` because we know that works everywhere)
cxx-rs has support for bridging types, but it's more awkward
for us because those types are defined in other crates, so
we need to do a newtype dance. Further cxx-rs doesn't currently
support automatically generating wrappers, so add a custom
`gobj_wrap()` for now.
Rust has a nice crate for doing perfect hashing. Move that
code into Rust and drop the dependency on `gperf`. This also
helps move away from Autotools.
The way gresources work using a constructor function started
failing when I was refactoring the build system, and I couldn't
figure it out. It's just easier to use Rust for this which
has nice toolchain-integrated functionality for this.
This is much better than bindgen because it's fully safe. It's
much more ergonomic too:
- Invoke Rust methods-on-structs just like C++ methods-on-structs
- Rust `Result<>` is translated automatically to exceptions
See https://cxx.rs/context.html for more.
There won't be any support for writing to the bdb backend in f34, so
e.g. pkglayering won't work (and obviously even composes wouldn't work
once the buildroot moves to f34).
Instead of requiring the whole world to add an `rpmdb` key in their
manifests, let's just add a compile flag for it, and tweak the spec file
to use this flag on f34.
I think we should have done this as soon as it was clear that
Rust was sticking and not just an optional thing.
Reasons to make this change now:
- More clear that Rust is going to be the majority of code in the future
- `cargo build` and `cargo test` in a fresh git clone Just Work
- Paves the way for using `cargo` to build C/C++ instead of Automake
