Commit Graph

3516 Commits

Author SHA1 Message Date
Colin Walters
49c55089d2 lockfile: Be robust to non-seconds-since-epoch repodata
We should really have safety comments around our use of `.unwrap()`.
This Google kubernetes rpm-md repo appears to use seconds-since-epoch.
We could try to detect that, probably best done in libdnf.

I think it'd be better to get them to match the informal schema
for this actually though instead.

For now let's just log an error and not crash.

Closes: https://github.com/coreos/rpm-ostree/issues/2600
2021-02-19 15:56:29 -05:00
Colin Walters
f882d6ddff rust: Add SPDX-License-Identifier and validate it in ci/codestyle.sh
It turns out we accidentally added GPL'd code into the Rust
side, which wasn't intentional on my part and I think it's since
been copied around.

Honestly I think half of the problem is the gigantic
"blah blah blah GNU General blah blah" just makes people's eyes
glaze over.  In contrast the `SPDX-License-Identifier` is short
and obvious.

So let's validate that in CI.

This follows a similar change in ostree:
https://github.com/ostreedev/ostree/pull/1439

If we merge this I'll do the C/C++ side too after that.
2021-02-19 15:56:23 -05:00
Colin Walters
a62ace8d5b codestyle: Remove tabdamage.txt in OK case
Avoid leaving cruft in the srcdir.
2021-02-19 15:56:23 -05:00
dependabot[bot]
1421dcbc14 build(deps): bump openat-ext from 0.1.11 to 0.1.12
Bumps [openat-ext](https://github.com/cgwalters/openat-ext) from 0.1.11 to 0.1.12.
- [Release notes](https://github.com/cgwalters/openat-ext/releases)
- [Commits](https://github.com/cgwalters/openat-ext/compare/v0.1.11...v0.1.12)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-19 15:05:23 -05:00
Colin Walters
27f9734300 apply-live: Delete code to check for live commit post-txn
Per discussion in
https://github.com/coreos/rpm-ostree/pull/2581#discussion_r578621458
it is inherently racy and we'll do a followup fix to write a
ref.
2021-02-19 09:08:22 -05:00
Colin Walters
246f02fd47 apply-live: Move client-side finish to Rust
Now that we have an RPM diff+printing binding, we can
move the client side postprocessing for `apply-live`
to Rust.
2021-02-19 09:08:22 -05:00
Colin Walters
04e0c4e01a Fix progress API to have "output message" separate from task
A lot of our output is outside of a "task"; the Rust binding
incorrectly made it a method on `Progress`.  This is really
just a `println!()` that is backed by our dispatch system.
2021-02-19 09:08:22 -05:00
Colin Walters
1d242ddb59 Add a C++ rpmdb-diff API wrapping the C one, bind in Rust
I'd like to compute diffs in apply-live to differentiate
between "pure layering" versus modifications/removals.
2021-02-19 09:08:22 -05:00
Colin Walters
29a2304fe2 daemon: Rename transaction_livefs → transaction_apply_live
Continuing the renaming.
2021-02-19 09:08:22 -05:00
Colin Walters
ffe72d28ec app: Rename livefs.cxx → apply-live.cxx
Continuing the renaming.
2021-02-19 09:08:22 -05:00
Timothée Ravier
790e709360 countme: Persist last timer trigger to handle system suspends
Set `Persistent=yes` in the timer unit to make sure that systems that
are not running 24/7 gets counted when they wake up if they missed the
previous timer run / counting window.

This is mostly for Fedora Silverblue and IoT like systems as Fedora
CoreOS systems usually run 24/7.

https://www.freedesktop.org/software/systemd/man/systemd.timer.html#Persistent=
2021-02-18 15:14:34 -05:00
Colin Walters
0c9f7ab7eb Add daemon.rs with one helper function, use it from C++
Trying to port to use origin Rust code, I think actually
it gets simpler if we move more to Rust to start.
2021-02-18 10:22:39 -05:00
Colin Walters
3910e77ae8 daemon: Refactor deployment variant generation code
Changes to this ad-hoc `a{sv}` code have been scattered
around over time.  Move the "core" bits to the top of
the function, then have the more complex stuff that depends
on the origin (and refspec/layering status) later.

Prep for further cleanups.
2021-02-18 10:22:39 -05:00
dependabot[bot]
2306c919ec build(deps): bump system-deps from 2.0.3 to 3.0.0
Bumps [system-deps](https://github.com/gdesmott/system-deps) from 2.0.3 to 3.0.0.
- [Release notes](https://github.com/gdesmott/system-deps/releases)
- [Commits](https://github.com/gdesmott/system-deps/compare/v2.0.3...v3.0.0)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-18 08:06:41 -05:00
Colin Walters
9eb217bd89 ci: Add Prow e2e testing
Our "CoreOS CI Jenkins" flow is great, but the reliability/debuggabilty
of the system isn't the best.  This is an experient to *add* another CI
flow that can be run in Prow too.

I'd like to at least have a basic e2e (compose tree, run client tests)
flow that can operate in Prow so that if CoreOS CI Jenkins is down or
flaky, we at least have some confidence in our commits.

Longer term I think we can figure out how to better share CI flows
across these:
https://github.com/coreos/fedora-coreos-tracker/issues/263#issuecomment-772903759
2021-02-18 06:52:42 -05:00
Colin Walters
8a62ad9f3d Add gobj_rewrap() API to pass glib-rs objects back to C++
When we started using cxxrs, most of the glib-rs objects like
`OstreeRepo`/`OstreeSysroot` were owned by C++ and passed
down into Rust.  That motivated the addition of the special
bridging infrastructure to re-create a glib-rs wrapper
type from what cxxrs wants (a `Pin<&mut T>`).

But now that we're adding more in Rust, we have the need
to pass these objects back into C++.  In fact this will
hopefully soon because the default case as more of the
binary entrypoint becomes Rust.

Add another trait with a method `gobj_rewrap()` that converts
in the other direction.  This implementation took me a number
of tries before I finally settled on simply using `mem::transmute()`.
There are a *lot* of caveats listed on the docs for that function,
but I think it really is what we want here.  See the link for pending work
on a Rust RFC to enable safe transmutes for some cases, and I believe
that would cover this use case:
https://internals.rust-lang.org/t/pre-rfc-v2-safe-transmute/11431

I've verified this works in a separate patch, but this commit
also adds a simple test case - this goes all the way from:
   Rust glib-rs `ostree::Repo` (holding strong ref)
   -> Rust `Pin<&mut ostree_sys::OstreeRepo>`
   -> (internal cxx-rs C bridge)
   -> C++ `OstreeRepo&` reference
   -> C `OstreeRepo*` pointer
Which is quite the dance if you think about it!
2021-02-18 06:29:41 -05:00
Colin Walters
8d9e113e8f rust/countme: Fix clippy lint by refactoring serialization
clippy complains about `to_*` not taking `&self` - I think
we can simplify this more by using our `write_file_with()` API
that we're using in other places.  It was explicitly designed
with serde in mind.  It's just more efficient and nicer to
serialize to a `BufWriter` instead of to a string, then
writing the string.
2021-02-18 05:27:41 -05:00
Colin Walters
9854a3aa8e rust/extensions: Fix two clippy lints
These are easy fixes.
2021-02-18 05:27:41 -05:00
Colin Walters
c027b638db rust: Quiet clippy Vec<Foo> arg
cxx.rs only supports `Vec<String>` and not `&[String]` right now.
2021-02-18 05:27:41 -05:00
Colin Walters
35fcf3eac0 rust: Fix some misc clippy lints
Nothing important here really, but clippy is useful so
let's try to keep it quiet.
2021-02-18 05:27:41 -05:00
dependabot[bot]
5f3b692e60 build(deps): bump libc from 0.2.84 to 0.2.86
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.84 to 0.2.86.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.84...0.2.86)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-18 05:13:41 -05:00
dependabot[bot]
3e7ad771fb build(deps): bump memfd from 0.3.0 to 0.4.0
Bumps [memfd](https://github.com/lucab/memfd-rs) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/lucab/memfd-rs/releases)
- [Commits](https://github.com/lucab/memfd-rs/compare/v0.3.0...v0.4.0)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-17 18:42:32 -05:00
dependabot[bot]
f343146efd build(deps): bump cxx from 1.0.30 to 1.0.32
Bumps [cxx](https://github.com/dtolnay/cxx) from 1.0.30 to 1.0.32.
- [Release notes](https://github.com/dtolnay/cxx/releases)
- [Commits](https://github.com/dtolnay/cxx/compare/1.0.30...1.0.32)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-17 17:42:33 -05:00
Colin Walters
047f931746 Release 2021.2 2021-02-17 15:17:33 -05:00
Colin Walters
8a1b2f3739 libdnf-sys: Include gio.h before libdnf.h
See b77f710cfb

Alternative fix to rpm-software-management/libdnf#1139
aka https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1935

This way libdnf's `extern "C"` over the glib headers doesn't apply
because we already processed that header.
2021-02-17 14:02:34 -05:00
Colin Walters
f8dc856aa9 compose: Add missing #include <optional>
This only fails on newer gcc/libstdc++ apparently in current
Fedora rawhide: https://kojipkgs.fedoraproject.org//work/tasks/4372/62164372/build.log
2021-02-17 14:02:34 -05:00
Kelvin Fan
626d021edb app/upgrade: Do not upgrade if updates driver registered
Do not perform an upgrade if detected that an updates driver has
been registered.

Add --bypass-driver option to force an upgrade regardless of whether an
updates driver has been registered.
2021-02-17 11:43:34 -05:00
Kelvin Fan
261b9cbb87 app/status: factor out get_sd_unit_objpath() function
Factor out a `get_sd_unit_objpath()` function into `libbuiltin`
so we can reuse the code in `builtin_upgrade`.
2021-02-17 11:43:34 -05:00
Colin Walters
e6c045cada Add an rpmostree-client sub-crate
This is intended to be published to https://crates.io/crates/rpmostree-client
Part of https://github.com/coreos/rpm-ostree/issues/2389

This directly imports the code from
5551c54c6e/tests/inst/src/rpmostree.rs

Once merged and released I'll try converting the ostree test suite
over as well as Zincati.

Internally add a testutils helper to validate it works.
2021-02-16 19:22:26 -05:00
Colin Walters
7bf885ecf7 tests: Fix bodhi link
From review in https://github.com/coreos/rpm-ostree/pull/2585#discussion_r577149232
2021-02-16 18:18:27 -05:00
Colin Walters
c48e8bfad3 ci: Rework build/test dependency install
Now that `ci/installdeps.sh` gracefully exits if run as non-root,
we can fold the cargo bits into the our build scripts and avoid
invoking both of them.

However, now we need to split test deps to separate file because
we won't have `cargo` in the main cosa pod.  This also fixes a FIXME.

Steal the `grep` invocation from cosa and make it a declarative
text file so we can have comments per package etc.
2021-02-16 18:18:27 -05:00
Colin Walters
d7b75f8459 build-sys: Ignore .cosa
This dir was added recently in coreos-assembler.
2021-02-16 18:18:27 -05:00
Colin Walters
836315084e ci: Add time prefixing before most commands
We don't have timestamps set up right now, and including
timing information is easy and useful to debug CI speed.
2021-02-16 18:18:27 -05:00
Colin Walters
d5227781de ci: Drop: distro-sync, continuous repo
Doing the distro-sync costs ~3 minutes per execution, and it
happens multiple times.  Let's just ensure our images are up
to date instead.

Also drop the continuous repo (for now) - we added this
to test bleeding edge ostree, but I think we need to reintroduce
"git master" builds as whole separate CI flow (layered container)
instead.
2021-02-16 18:18:27 -05:00
Colin Walters
c1ff933bc3 tests/layering-non-root-caps: Adapt to new libcap output
I didn't deep dive on this, just observed that the new output
matches what we laid down.  There's ~2.5 years of changes
and ~200 commits between 2.28 and 2.48.
2021-02-16 17:08:27 -05:00
Colin Walters
04839e200f Misc doc tweaks/cleanups
- Link to osbuild blog
- Drop CentOS 7 stuff
- clarify goals a bit
2021-02-12 16:45:46 -05:00
Colin Walters
35342dabca build-sys: Always run makecargo build
When working on a PR to add a sub-crate I hit the fact
that our `find` bit wasn't fully accurate and spent some
time debugging the fact that the code I got after `make`
wasn't up to date.

Since cargo is smart in general, let's stop trying to
second guess its dependencies and just run `cargo build`
every time `make` is run.

(I'm not sure why we didn't do this from the start)
2021-02-12 15:18:45 -05:00
Colin Walters
118178cef1 tree-wide: Fix C++ range iteration to use references
This is actually a perfect example of the tradeoffs involved
in our team of C/Rust programmers trying to use C++ so we
can use cxx-rs =)

It turns out that "for (auto foo : bar)" is really yet another
one of those C++ sharp edges just waiting to cut.  That
version does a copy of each value; one almost always wants to use
"for (auto & foo : bar)" which like Rust's `.iter()`.

In most of our code we were OK making copies, they were just
a performance hit, but in another place I was relying on the
"side effect" of `.c_str()` but that meant our value got
destroyed.
2021-02-12 12:50:02 -05:00
Kelvin Fan
84065c48df tests/common/libvm: Print error if bad tests/vmcheck/image.qcow2
Print error and exit if cannot find `tests/vmcheck/image.qcow2` or
bad symlink causing failure to spawn a VM when calling the
`vm_kola_spawn()` function.
2021-02-11 21:11:37 -05:00
Colin Walters
764de41cc6 Switch to using cxx-rs for treefile
This is one half of https://github.com/coreos/rpm-ostree/issues/2544
which aims to drop our use of `cbindgen`.
2021-02-11 15:44:38 -05:00
Colin Walters
094be6c469 rust: Add module doc headers
Specifically motivated by adding some docs in `treefile.rs` around
how to add a field, but I decided to just do a pass and
document everything at least a little.

View with e.g. `cargo doc --document-private-items`.
2021-02-11 14:16:38 -05:00
Colin Walters
20178bb25a tree-wide: Include GLib headers before libdnf
Alternative fix to https://github.com/rpm-software-management/libdnf/pull/1139
aka https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1935

This way libdnf's `extern "C"` over the glib headers doesn't apply
because we already processed that header.
2021-02-10 17:05:42 -05:00
Colin Walters
f065c0dd32 Add --enable-sanitizers (not on by default yet)
This way we at least get unit test coverage (which...
our unit test coverage doesn't do much because our
main code paths require privileges or virt).

One main blocker to this is that rustc doesn't expose
first-class support for this yet:
https://github.com/rust-lang/rust/issues/39699

At a practical level this works when building in release
mode but fails with `cargo test` for some reason; linker
arguments being pruned?  Not sure.

So I was able to use this when composing to find a bug,
but then for some other reason the client
side apparently infinite loops inside libsolv.

So we're not enabling this yet for those reasons, but
let's land the build infrastructure now.

```
(lldb) thread backtrace
* thread #4, name = 'pool-/usr/bin/r'
  * frame #0: 0x00007fd61b97200f libc.so.6`__memcpy_sse2_unaligned_erms + 623
    frame #1: 0x00007fd61cbc88e6 libasan.so.6`__asan::asan_realloc(void*, unsigned long, __sanitizer::BufferedStackTrace*) + 214
    frame #2: 0x00007fd61cc4b725 libasan.so.6`__interceptor_realloc + 245
    frame #3: 0x00007fd61baec43e libsolv.so.1`solv_realloc + 30
    frame #4: 0x00007fd61baf0414 libsolv.so.1`repodata_add_dirstr + 276
    frame #5: 0x00007fd61bb6f755 libsolvext.so.1`end_element + 53
    frame #6: 0x00007fd61b05855d libxml2.so.2`xmlParseEndTag1.constprop.0 + 317
    frame #7: 0x00007fd61b063548 libxml2.so.2`xmlParseTryOrFinish.isra.0 + 888
    frame #8: 0x00007fd61af7ed20 libxml2.so.2`xmlParseChunk + 560
    frame #9: 0x00007fd61bb727e7 libsolvext.so.1`solv_xmlparser_parse + 183
    frame #10: 0x00007fd61bb5ea0e libsolvext.so.1`repo_add_rpmmd + 254
    frame #11: 0x000055a4fce7a5f5 rpm-ostree`::load_filelists_cb(repo=<unavailable>, fp=<unavailable>) at dnf-sack.cpp:444:23
    frame #12: 0x000055a4fce7cad6 rpm-ostree`load_ext(_DnfSack*, libdnf::Repo*, _hy_repo_repodata, char const*, char const*, int (*)(s_Repo*, _IO_FILE*), _GError**) at dnf-sack.cpp:430:13
    frame #13: 0x000055a4fce7df60 rpm-ostree`dnf_sack_load_repo at dnf-sack.cpp:1789:26
    frame #14: 0x000055a4fce7eee9 rpm-ostree`dnf_sack_add_repo at dnf-sack.cpp:2217:28
    frame #15: 0x000055a4fce7f0fb rpm-ostree`dnf_sack_add_repos at dnf-sack.cpp:2271:32
    frame #16: 0x000055a4fce870ee rpm-ostree`dnf_context_setup_sack_with_flags at dnf-context.cpp:1796:29
    frame #17: 0x000055a4fcdf757f rpm-ostree`rpmostree_context_download_metadata at rpmostree-core.cxx:1206:44
    frame #18: 0x000055a4fcdf95c3 rpm-ostree`rpmostree_context_prepare at rpmostree-core.cxx:2001:48
    frame #19: 0x000055a4fce54ab7 rpm-ostree`rpmostree_sysroot_upgrader_prep_layering at rpmostree-sysroot-upgrader.cxx:1018:38
    frame #20: 0x000055a4fcdcb143 rpm-ostree`deploy_transaction_execute(_RpmostreedTransaction*, _GCancellable*, _GError**) at rpmostreed-transaction-types.cxx:1445:49
    frame #21: 0x000055a4fcdba4cd rpm-ostree`transaction_execute_thread(_GTask*, void*, void*, _GCancellable*) at rpmostreed-transaction.cxx:340:34
    frame #22: 0x00007fd61c58f7e2 libgio-2.0.so.0`g_task_thread_pool_thread + 114
    frame #23: 0x00007fd61c3d7e54 libglib-2.0.so.0`g_thread_pool_thread_proxy.lto_priv.0 + 116
    frame #24: 0x00007fd61c3d52b2 libglib-2.0.so.0`g_thread_proxy + 82
    frame #25: 0x00007fd61b8af3f9 libpthread.so.0`start_thread + 233
    frame #26: 0x00007fd61b9c9903 libc.so.6`__clone + 67
(lldb)
```
2021-02-10 14:30:37 -05:00
Colin Walters
e643ca8113 build: Drop introspection+clang hack
Forcing use of GCC breaks trying to use `clang -fsanitize=address`.
The real fix here anyways should have been avoiding passing
compiler flags to a compiler that doesn't understand them.
2021-02-10 14:30:37 -05:00
Jonathan Lebon
49fe650124 lockfile: Move ror_lockfile_read to cxx.rs
Pretty straightforward. Haven't moved `ror_lockfile_write` yet because
that's trickier to do and I'm still figuring out the most elegant way to
do this within cxx.rs' constraints.
2021-02-10 13:39:38 -05:00
Jonathan Lebon
b640892f04 libdnf-sys: Drop C API, replace with cxx.rs bridge
Right now, we're using libdnf APIs from Rust via hand-crafted `extern C`
interfaces, which is extra dangerous because there is no signature
checking that happens at compile-time.

Until either we can automate libdnf bindings or use its C++ API directly
via cxx.rs, let's do some basic wrapping in C++ ourselves and use libdnf
through that API only instead. That gives us a lot more confidence and
makes the libdnf API feel more natural to use in Rust.
2021-02-10 13:39:38 -05:00
dependabot[bot]
5cd9e8f5e8 build(deps): bump serde_yaml from 0.8.15 to 0.8.16
Bumps [serde_yaml](https://github.com/dtolnay/serde-yaml) from 0.8.15 to 0.8.16.
- [Release notes](https://github.com/dtolnay/serde-yaml/releases)
- [Commits](https://github.com/dtolnay/serde-yaml/compare/0.8.15...0.8.16)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-10 10:46:38 -05:00
Jonathan Lebon
56f95cc5c1 extensions: Don't try to chown RPM packages
We don't need it and it won't work in the unprivileged path where we're
running this in a cosa supermin (of course, this is all a bit silly
because we don't actually need privileges to begin with for this, but
there's a lot of momentum in sticking with that workflow).

Update submodule: libglnx
2021-02-10 10:20:38 -05:00
dependabot[bot]
e7c744e8b2 build(deps): bump serde_json from 1.0.61 to 1.0.62
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.61 to 1.0.62.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.61...v1.0.62)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-09 19:44:35 -05:00
Colin Walters
6886e44425 scripts: Bind /usr/share/empty over /usr/share/rpm
Now that we inject the `%_dbpath /usr/share/rpm` macro, `rpm -q`
will start using it.  But in RPM script invocation, we don't
want them to see any RPM database at all - trying to query it
should be a clean failure.
2021-02-09 18:36:35 -05:00