Commit Graph

3366 Commits

Author SHA1 Message Date
Jonathan Lebon
fb8e2da9d6 ci: Re-add CARGO_BUILD_JOBS
We lost this at some point during the CI re-shuffle. We need to
constrain cargo builds too to respect our CPU allocation.

This doesn't totally keep all jobs under 5 since e.g. we could have 5
make jobs and 5 cargo codegen builds going at once, but I think as long
as it's not something ridiculous like 40, it should be fine. Otherwise
we'll tighten it more.
2021-02-09 18:36:35 -05:00
Jonathan Lebon
667fdc9ff4 libpriv/rpm-util: Use /usr/share/rpm for base rpmdb query
Follow-up to previous commit: we had another path where we made a
temporary rootfs and symlinked `/var/lib/rpm` to the base rpmdb. That of
course broke now that we inject a macro to point the rpmdb to
`/usr/share/rpm`.

Rework this to use `/usr/share/rpm` since that's our canonical location
for now, but also add the compat symlinks so that this logic should keep
working even on trees without the injected macro yet.
2021-02-09 18:36:35 -05:00
Jonathan Lebon
60215ae865 libpriv/rpm-util: Add /usr/lib/sysimage/rpm symlink in rpmdb checkout
We don't technically need this yet, but it mirrors how it's set up in
our composes so that if there's code that wants to use the new location
too, it'll just work.
2021-02-09 18:36:35 -05:00
Jonathan Lebon
aa1a0f8719 lib/package: Fix comment about rpmdb checkout
We do fallback here.
2021-02-09 18:36:35 -05:00
Jonathan Lebon
99486a75e8 Add /usr/lib/rpm/macros.d/macros.rpm-ostree to set %_dbpath to /usr/share/rpm
We trigger a librpm macro file load in many of our paths. Since the
default value shipped by rpm's macro file sets `_dbpath` to
`/var/lib/rpm`, we have to explicitly set that back to `/usr/share/rpm`
in those paths.

This became more problematic recently with libsolv v0.7.17 which fully
keys off of `_dbpath` to find the rpmdb path to load:

04d4d036b2

And it's not technically wrong; we really should make that macro not
lie. This is what this patch does by injecting an RPM macro file in our
composes which sets it to /usr/share/rpm. So then e.g. the `rpm` CLI
doesn't actually need the `/var/lib/rpm` backcompat link anymore, though
there's no harm in leaving it.

In the future, we should be able to drop this once we move all of Fedora
to `/usr/lib/sysimage/rpm` (see
https://github.com/coreos/fedora-coreos-tracker/issues/639).

Closes: #2548
2021-02-09 18:36:35 -05:00
Jonathan Lebon
8ab604d098 ci: Temporarily use libsolv-0.7.17
We need to make sure that we can work with newer libsolv, which changed
how the rpmdb is found (see #2548).
2021-02-09 18:36:35 -05:00
Luca BRUNO
8a2f281143 libpriv/postprocess: get rid of goto statements
This removes all goto statements in the postprocess module,
replacing them with an exception catcher instead.
2021-02-09 16:43:32 +01:00
Colin Walters
2f6b5a654d Bind output core into Rust, use in apply-live
Originally the Rust apply-live code was exposed from Rust to C
via bindgen.  But when working on that, I hit the problem
that our output infrastructure was C...and the "reverse direction"
binding stuff was just ugly.

This PR again IMO shows the value of the investment in cxx-rs
because we can now seamlessly call back from the Rust side
into a "C++-ish" progress API, which the C++ side is updated
to use.

The level of indirection here is obviously pretty silly
because the main thing on the C++ output side is basically
a function dispatcher, but...I didn't want to try to rework
that into Rust fully yet.  (But, the moment we do this
whole area will get a *lot* cleaner)

Anyways, in the end this makes it easy for the apply-live
code to output progress to the user which was sorely
needed.
2021-02-09 04:43:29 -05:00
Colin Walters
c9e9269770 Rename internal Rust progress to console_
Our output system is very confusing in that we bridge over
DBus in some cases and not others.  In preparation for allowing
Rust code to call into the C++ progress system which contains
that delegation layer, rename the Rust progress to `console_`
to clearly show that it should only be invoked by code that
knows it's writing to a tty.
2021-02-09 04:43:29 -05:00
Colin Walters
8f71dcfafe build-sys: Statically link binary against shlib code
Having our binary depend on the shared library, which in
turn depends on the binary (at runtime) is messy.
Instead, statically compile the shlib code into our binary.
This duplicates the text a bit, but it's not a lot of code.

The goal is to more easily in the future to e.g. move the
shared library out into a separate git repository entirely
that runs on a separate lifecycle - that would still build
using Automake for example while the main git repository
switches to purely cargo.

Another motivation is avoiding linker issues I had with other
patches due to this semi-cyclical dependency.
2021-02-09 03:37:31 -05:00
Colin Walters
36566e406e build-sys: Add --enable-werror, rework compiler flags
So...at some point we somehow lost `-Wall` in our
default compiler flags which means we were missing some
potentially important warnings.  And
we used to have `-Werror` on in CI which combined
with the above was strongly opinionated about not
landing warnings in git master.

Our default stance here remains the same; we have
an opinionated set of `-Werror=` that applies in
*all* configurations.   However that set moves
into Automake - I don't think we need to do
compiler version detection anymore, we can assume
a modern compiler.

We also add back in `-Wall` by default now.

Further in CI, add `-Werror`.  The implementation
here is in our buildsystem rather than
`export CXXFLAGS=-Werror` because unfortunately
we have to fix things in libdnf too, and I don't
want to block entirely on that.
2021-02-08 16:50:09 -05:00
Colin Walters
a8fe6cf32b transaction: Fix missing return in error case
Hooray for compiler warnings.
2021-02-08 14:38:31 -05:00
Colin Walters
dba43e201e tree-wide: Fix clang -Wgnu-designator/-Wunused-variable
clang warns about this:
https://clang.llvm.org/docs/DiagnosticsReference.html#wgnu-designator
Also fix unused variables and hard error on them.
2021-02-08 14:38:31 -05:00
Colin Walters
b942bd8cf7 ci/clang-analyzer: Install deps
This flow was missing from here which broke in CI.
2021-02-08 13:21:16 -05:00
dependabot[bot]
4493fb00b7 build(deps): bump cxx from 1.0.29 to 1.0.30
Bumps [cxx](https://github.com/dtolnay/cxx) from 1.0.29 to 1.0.30.
- [Release notes](https://github.com/dtolnay/cxx/releases)
- [Commits](https://github.com/dtolnay/cxx/compare/1.0.29...1.0.30)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-08 12:07:16 -05:00
Colin Walters
8ddaf0bbd6 Make failure to find packages fatal, add more error prefixing
To help debug https://bugzilla.redhat.com/show_bug.cgi?id=1925584
2021-02-08 11:20:50 -05:00
Jonathan Lebon
d22d241754 core: Don't allow noent when resolving pkgcache rev
If we get here, it's that we expect the pkgcache to be there. So don't
allow ENOENT (we weren't even checking for the ENOENT case here, which
shows that this was the intent).

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1925584
2021-02-08 11:20:50 -05:00
Colin Walters
5cefee81ff tree-wide: Squash some clang-analyzer found unused variables
We weren't using the return values, so just drop them.
2021-02-05 12:39:07 -05:00
Colin Walters
4d2a6e6de0 tree-wide: Pacify some clang-analyzer "Dead nested assignment"
It doesn't understand the "$x_owned" pattern (which is really
much like Rust's `std::borrow::Cow`).
2021-02-05 12:39:07 -05:00
Colin Walters
18c29fa5d6 libmain: Refactor to fix analyzer warning
Previously this function was in `goto out` style
and so we had this awkward "track return value in pointer"
thing.  But `clang-analyzer` correctly points out that
we don't need this anymore because we never read the value
initially stored.
2021-02-05 12:39:07 -05:00
Colin Walters
6a594dbe6b util: Annotate our "throw" wrappers as [[ noreturn ]]
This way the compiler and `clang-analyzer` understand and won't
issue an error if we are missing a `return` after an unconditional
`throw_gerror()`.
2021-02-05 12:39:07 -05:00
Colin Walters
a36f9716c6 tree-wide: Fix some "Dead assignment" from clang-analyzer
The `have_rpmdb` one was a leftover looks like.  In the `disabled_all_repos`
case it was clearly there for symmetry, but eh; it seems somewhat
unlikely that we add a *3rd* case there.  Also while we're
here change it to C++ `bool` so tools like analyzers know it really
is a boolean.
2021-02-05 12:39:07 -05:00
Colin Walters
7b5b35210b tree-wide: Fix some spurious "Dead assignment" from clang-analyzer
This fixes some spurious warnings from clang-analyzer (aka `scan-build`) around
"Dead assignment".  Unfortunately the analyzer doesn't understand
the side effects of `__attribute__((cleanup))` here.

More info on the `(void)` pattern: https://clang-analyzer.llvm.org/faq.html#dead_store
2021-02-05 12:39:07 -05:00
Colin Walters
6dd12a8175 ci: Add clang-analyzer.sh
There are really no excuses for any C/C++ project not to use
both ASAN+UBSAN and static analysis in CI.
2021-02-05 12:39:07 -05:00
Colin Walters
47c60eb6ce libdnf: Various buildsys fixes
WITH_SWDB: Removed in 99309fbe04
WITH_GIR Removed in e2f2862bed

Also, most importantly: don't always reconfigure libdnf

This is a questionable default for the cargo `cmake` crate.
Building in Koji is failing I think due to timestamp issues
causing cmake to run twice.
2021-02-04 17:57:33 -05:00
Jonathan Lebon
f5fde004dc Makefiles: Drop unnecessary cd rust/ and CARGO_TARGET_DIR
We can simplify these now that Rust is at the root dir.
2021-02-04 17:00:33 -05:00
Jonathan Lebon
df7adbb5f0 Makefile-rpm-ostree: Fix setting of cargo_target_dir
We don't want tabs here otherwise it doesn't work.
2021-02-04 17:00:33 -05:00
Jonathan Lebon
dd6b805fce Makefile-rpm-ostree: Drop hardcoded release path
And actually, we can just use `rpm-ostree` here, which symlinks to the
correct one (and `install` follows links).
2021-02-04 17:00:33 -05:00
Colin Walters
5060cdf7fc ci: Drop clang and unit tests from here
We're running them in Prow now and we're hitting capacity
issues in CentOS CI.  Bigger picture, the "just build and unit test"
stuff runs in any cluster, so let's save our bare metal capacity
for our compose/VM testing.
2021-02-04 15:19:36 -05:00
Luca BRUNO
1932979059 make/rust-test: fix linking and loading
This fixes link dependencies and build-libraries path, in order to
make Rust tests work.
It also introduces an additional wildcard target to allow specifying
a test filter to cargo.
2021-02-04 13:55:00 -05:00
Colin Walters
ae3392ffaf build-sys: Cleanly split up deps of public shlib vs internals
First, the public shared library only depends on a few
things (not the libdnf dependencies) so let's ensure we
only link it to those libraries.

And then, I realized we don't actually need the libdnf
dependencies here - I think I only added those back here
when trying vainly to keep the C unit tests working.  But
we don't have those anymore!  So we can delete the duplication
and fully rely on Cargo taking care of libdnf.

Conceptually for a static library we don't "link" it against
anything in Automake, that happens at the final stage with
the Rust linker.
2021-02-04 10:59:20 -05:00
Colin Walters
19c7fbb7c9 build-sys: Delete duplicate Rust pkg-config dependencies
We didn't need this after switching to a Rust main.
2021-02-04 10:59:20 -05:00
Colin Walters
c383c9511f build-sys: Drop Fedora 25+ rpm version check
I think we can just assume this nowadays.
2021-02-04 10:59:20 -05:00
Colin Walters
588541c60d Move libdnf build over to Cargo
This is now further migration towards Cargo/Rust possible
because we switched our main binary.  We've had an internal
`libdnf-sys` crate for a while, but now it can take over
the build of the underlying library too (like many `-sys`
crates support).

This itself is just an incremental step towards migrating
the main rpm-ostree build system to e.g. cmake too (or
perhaps directly with the `cc` crate, not sure yet) and
driving it via `cargo` too.
2021-02-04 10:59:20 -05:00
Colin Walters
2128f5784a build-sys: Explicit workspace, move libdnf deps to crate
First explicitly state that we're a workspace.  AIUI
this is actually implicit today via our use of a `path`
dependency, but in the future we may have other sub-crates.
So let's make it explicit now.

Also move the libdnf dependencies directly to that sub-crate.
2021-02-04 10:59:20 -05:00
Colin Walters
0a05e467e6 ci: Propagate make jobs to clang build too
Otherwise we try to run 40 processes.
2021-02-03 19:25:26 -05:00
Colin Walters
7268ac9875 ci: Consistently source libbuild
Since we need to set HOME and PATH, let's do that in a central
place rather than scattering it around by having all of
our entrypoint scripts source the `libbuild.sh` shell "library".

Move the CoreOS CI entrypoint into a script like the others.
2021-02-03 19:25:26 -05:00
Colin Walters
6fcfaee948 ci: Don't assume HEAD exists
For some reason the way Prow clones the repo it doesn't exist;
the git default of `..` should work though.
2021-02-03 19:25:26 -05:00
Luca BRUNO
9c3864b97e libpriv/passwd: move entries deduplication logic to Rust
This moves `group` and `passwd` merging/deduplication to Rust.
2021-02-03 15:59:38 -05:00
Colin Walters
fe342c30ed ci: Also test for HOME being writable
Actually it seems OpenShift sets HOME=/ for some reason; probably
related to the non-root uid default.

And whole lot of the Prow jobs do `export HOME=$(mktemp -d)` today.

I am tempted to add a `cosa entrypoint` command or something
that sanitizes the environment setup.
2021-02-03 15:21:42 -05:00
Colin Walters
79f07957f5 ci: Ensure HOME is set
Prow doesn't set this and it breaks our `cargo install`.
2021-02-03 14:07:39 -05:00
Jonathan Lebon
57edf7e638 tests/compose.sh: Always rebuild supermin appliance
We always want the latest rpm-ostree binaries tested, so we need to
always rerun supermin.

Patch better viewed with whitespace ignored.
2021-02-03 12:22:38 -05:00
Jonathan Lebon
317b920667 extensions: Add support for development extensions
In RHCOS, we ship kernel development-related packages as an extension.
Those aren't really extensions that are meant to be layered onto the
host.  They're meant to be used in a build environment somewhere to
compile kernel modules.

This makes it very different from "OS extensions" in at least two
drastic ways:
1. we don't want to do any depsolving (e.g. we don't want to pull in
   `gcc` or something)
2. some of those packages may be present in the base already, but we
   still want to redownload them

Hesitated putting this functionality in rpm-ostree, but I think in the
end it cuts from the benefit of moving this code to rpm-ostree if we
can't entirely get rid of the Python script it obsoletes. Plus, being
able to use the `match-base-evr` is still really useful for this use
case.

Let's add a new `kind` key to support this. The traditional extensions
are called "OS extensions" and these new extensions are called
"development extensions".

The latter is not yet part of the state checksum, so change detection
doesn't work there. I think that's fine for now though because the
primary use case is the kernel, and there we want to match the base
version. So if the kernel changes, the base would change too. (Though
there's the corner case of adding a new package to the list while at the
same version...)
2021-02-03 12:22:38 -05:00
Jonathan Lebon
20ab52adaa rust/treefile: Drop unnecessary #[serde(rename)]
The key is already called `rpmdb`.
2021-02-03 12:22:38 -05:00
Jonathan Lebon
f4bee1352d app/compose: Add comment about pkgcache
Gives a bit more info about how the extensions path is different from
the base treecompose.
2021-02-03 12:22:38 -05:00
Jonathan Lebon
9d07286f3d extensions: Fix missing CxxResult 2021-02-03 12:22:38 -05:00
Jonathan Lebon
f7090358be extensions: Support enabling additional repos
We want to be able to enable more repos than those in the treefile when
downloading extensions. In RHCOS for example, the `kernel-rt` packages
come from a separate repo.

But also, once we support "development" extensions, we want to support
the case where devel packages come from another repo.
2021-02-03 12:22:38 -05:00
Jonathan Lebon
bbc72cbf6b core: Fix handling of local packages when downloading
In the core context, this is redundant with `sort_packages` because it
won't put local packages in the `pkgs_to_download` array anyway, but we
want this check even if we call `rpmostree_download_packages` directly
and pass some packages which may be local.
2021-02-03 12:22:38 -05:00
Jonathan Lebon
90c546c5dc core: Factor out function to download pkgs
I want to be able to use this function without an `RpmOstreeContext`.

Prep for future patch.
2021-02-03 12:22:38 -05:00
Jonathan Lebon
e85c86be79 core: Factor out function to set repos on pkgs
And use a hash table to make it more efficient.

Prep for future patch.
2021-02-03 12:22:38 -05:00