IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
The inevitable followup to https://github.com/coreos/rpm-ostree/pull/2278
that I was too cowardly to do at the time. But it's time to admit
the 2 months or so of work on this was wasted. We have too much
tech debt and this is a large chunk of C/C++ code that touches everything
in the codebase in a nontrivial way.
Bigger picture, I'm going to work on
https://github.com/coreos/fedora-coreos-tracker/issues/828
which will strongly orient rpm-ostree towards the container world instead.
We'll still obviously keep the rpm package world around, but only
as a secondary layer. What rojig was trying to do in putting "images"
inside an RPM was conflating layers. It would have had a lot of
benefits probably if we'd truly pushed it over the edge into completion,
but that didn't happen. Let's focus on containers instead.
There's still a lot more rojig code to delete but this first patch removes
the bulk of it. Touching everything that references e.g. `RPMOSTREE_REFSPEC_TYPE_ROJIG`
etc. can come as a 3rd phase.
This addresses the server compose side of
https://github.com/coreos/rpm-ostree/issues/2584.
One tricky bit is handling overrides across included treefiles (or
really, even within a single treefile): as usual, higher-level treefiles
should override lowel-level ones. Rust makes it pretty nice to handle.
For now this just supports a `repo` field, but one could imagine e.g.
`repos` (which takes an array of repoids instead), or e.g.
`exclude-repos`.
The actual core implementation otherwise is pretty straightforward.
This should help a lot in RHCOS where we currently use many `exclude=`
directives in repo files to get it to do what we want.
This is also kind of a requirement for modularity support because as
soon as rpm-ostree becomes modules-aware, modular filtering logic will
break composes which assume rpm-ostree treats modular and non-modular
packages the same.
Right now if we want to lock e.g. systemd, we need to specify every
subpackage of systemd that we use. This is a lot of duplication because
in the majority of cases, what we really mean is "lock at this build of
systemd".
Since RPMs bake in the source RPM they were built from, we can use this
to lock packages more succinctly. See the testcase and #2676 for
examples of how this looks.
Closes: https://github.com/coreos/rpm-ostree/issues/2676
The compose tests are expensive; each run involves running
all the `%post` scripts and `dracut` etc. This is definitely
a source of timeouts in CCI.
Remove `test-boot-location-modules.sh` - it's the default
now and is used by FCOS. Add dedicated script where we can
test all these things by default after a `cosa build`.
This aims to move the compose tests to only cover bits *not*
in cosa like the non-unified-core path.
In FCOS, we use "override" lockfiles to pin packages to certain
versions. Right now, we have separate overrides for each base arch we
(eventually want to) support. But that makes maintaining the overrides
cumbersome because of all the duplication.
Let's allow lockfiles to specify only the `evr` of a package, which is
just as good for FCOS, and means that we'll only have to maintain a
single override file for all the architectures.
We trigger a librpm macro file load in many of our paths. Since the
default value shipped by rpm's macro file sets `_dbpath` to
`/var/lib/rpm`, we have to explicitly set that back to `/usr/share/rpm`
in those paths.
This became more problematic recently with libsolv v0.7.17 which fully
keys off of `_dbpath` to find the rpmdb path to load:
04d4d036b2
And it's not technically wrong; we really should make that macro not
lie. This is what this patch does by injecting an RPM macro file in our
composes which sets it to /usr/share/rpm. So then e.g. the `rpm` CLI
doesn't actually need the `/var/lib/rpm` backcompat link anymore, though
there's no harm in leaving it.
In the future, we should be able to drop this once we move all of Fedora
to `/usr/lib/sysimage/rpm` (see
https://github.com/coreos/fedora-coreos-tracker/issues/639).
Closes: #2548
In RHCOS, we ship kernel development-related packages as an extension.
Those aren't really extensions that are meant to be layered onto the
host. They're meant to be used in a build environment somewhere to
compile kernel modules.
This makes it very different from "OS extensions" in at least two
drastic ways:
1. we don't want to do any depsolving (e.g. we don't want to pull in
`gcc` or something)
2. some of those packages may be present in the base already, but we
still want to redownload them
Hesitated putting this functionality in rpm-ostree, but I think in the
end it cuts from the benefit of moving this code to rpm-ostree if we
can't entirely get rid of the Python script it obsoletes. Plus, being
able to use the `match-base-evr` is still really useful for this use
case.
Let's add a new `kind` key to support this. The traditional extensions
are called "OS extensions" and these new extensions are called
"development extensions".
The latter is not yet part of the state checksum, so change detection
doesn't work there. I think that's fine for now though because the
primary use case is the kernel, and there we want to match the base
version. So if the kernel changes, the base would change too. (Though
there's the corner case of adding a new package to the list while at the
same version...)
Let's include the final extensions file in JSON format as part of the
output directory. A key difference from the input file (apart from YAML
vs JSON) is that this is post-filtering, so any extensions which were
removed because the architecture does not match are not present.
This JSON file will be used by cosa and the MCO. See discussions in:
https://github.com/openshift/os/issues/409
This adds support for a new `rpm-ostree compose extensions` command`
which takes a treefile, a new extensions YAML file, and an OSTree repo
and ref. It performs a depsolve and downloads the extensions to a
provided output directory.
This is intended to replace cosa's `download-extensions`:
https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions
The input YAML schema matches the one accepted by that script.
Some differences from the script:
- We have a guaranteed depsolve match and thus can avoid silly issues
we've hit in RHCOS (like downloading the wrong `libprotobuf` for
`usbguard` -- rhbz#1889694).
- We seamlessly re-use the same repos defined in the treefile, whereas
the cosa script uses `reposdir=$dir` which doesn't have the same
semantics (repo enablement is in that case purely based on the
`enabled` flag in those repos, which may be different than what the
rpm-ostree compose ran with).
- We perform more sanity-checks against the requested extensions, such
as whether the extension is already in the base.
- We support no-change detection via a state SHA512 file for better
integration in cosa and pipelines.
- We support a `match-base-evr` key, which forces the extension to have
the same EVR as the one from a base package: this is helpful in the
case of extensions which complement a base package, esp. those which
may not have strong enough reldeps to enforce matching EVRs by
depsolve alone (`kernel-headers` is an example of this).
- We don't try to organize the RPMs into separate directories by
extension because IMO it's not at the right level. Instead, we should
work towards higher-level metadata to represent extensions (see
https://github.com/openshift/os/issues/409 which is related to this).
Closes: #2055
I was looking at the output of `ostree admin config-diff`
on a base FCOS boot. It'd be really nice to trim that down
as much as possible, so we can cleanly capture the difference
between user config and system config.
Let's use static enablement rather than presets.
We're seeing some CI failures that I think are a bug in rojig.
In the bigger picture...we never actually started using this,
and I think longer term shipping os updates via containers
probably makes more sense.
I put a *lot* of effort into this code and it's pretty cool
so it's hard to just delete it. And *maybe* someone out there
is using it (but I doubt it). So rather than just deleting
it entirely let's make it a build-time option.
I verified that it builds at least.
In f32, ping is no longer privileged since it ships with the sysctl for
`ping_group_range` which allows unpriv pings. Check the file caps of
arping instead, which does still use them.
The design of https://fedoraproject.org/wiki/Changes/Sqlite_Rpmdb
is problematic for us for multiple reasons. The first big reason
is that rpm-ostree is designed for "cross" builds and e.g. today
we use a Fedora-derived container to build RHEL CoreOS images.
However the default database lives inside the `rpm` package which
means that if we e.g. upgrade the coreos-assembler container to F33
it will suddenly try to use sqlite for RHCOS which is obviously broken.
Related to this, rebases from f32 to f33 w/layered packages
are broken: https://bugzilla.redhat.com/show_bug.cgi?id=1876194#c3
With this we can configure things to continue to use bdb for f33
for ostree-based systems, so that by enforcing an upgrade order
f32 → f33 [bdb] → f34 [sqlite] ... the intermediate f33 w/bdb
still understands sqlite and hence rebases will work.
Fix https://github.com/coreos/rpm-ostree/issues/2068: `remove-from-packages`
deleting files that it shouldn't.
Filter out files that user wants removed at `checkout_package_into_root()`,
instead of at the `handle_remove_files_from_package()` function that does
not check whether files are used by other rpms before removing them.
Add a helper function for whitespace_split_packages() so that it now
splits a String by whitespace only if it is not wrapped between single
quotes.
This should allow RHCOS to use syntax like podman > 1.4 in the treefile.
Also add new unit tests and tweak existing compose tests to test this
functionality.
Now that cosa and FCOS have moved to f32, a bunch of tests are breaking.
Let's make them more resistant to releasever changes.
While we're here though, bump the container image we use on the target
host to f32, and update the systemd example output.
Same motivation as
7392259332
I think we should encourage removing the writable bits from
executables. This has happened to me:
https://thomask.sdf.org/blog/2019/11/09/take-care-editing-bash-scripts.html
And not having the writable bit may help prevent hardlink
corruption with OSTree in some cases.
We can't do this by default, but add a convenient treefile option
for it.
This starts out by just doing this for RPMs, but I'll add
a secondary pass which does it during postprocessing soon too.
In Fedora CoreOS, we have a "coreos-pool" repo from which all packages
in lockfiles are tagged for reproducible builds. This repo is shared
across all streams, including those on f31 and f32.
Thus, it makes no sense for composes to ever pick packages unconstrained
from the pool without being guided by a lockfile. Otherwise, one can
easily end up with e.g. f32 packages in an f31 compose.
Add a new `lockfile-repos` for this which is only used for fetching
lockfile packages and nothing else. For example, this will allow
`cosa fetch --update-lockfile` to Just Work as expected by only fetching
new packages from regular yum repos.
Today, lockfiles only restrict the NEVRA of specifc package names from
which libsolv can pick. But nothing stops libsolv from picking entirely
different packages which still satisfy the manifest requests.
This was mostly a theoretical issue in Fedora CoreOS, but became reality
with the addition of Fedora 32 packages in the pool. libsolv would
happily try to pick e.g. `libcurl-minimal` from f32 instead of sticking
with the f31 `libcurl` from the lockfiles:
https://github.com/coreos/fedora-coreos-streams/issues/75#issuecomment-610734584
(But more generally, see
https://github.com/coreos/fedora-coreos-tracker/issues/454).
Let's add a `--ex-lockfile-strict` mode, which in CI and production
pipeline build contexts will require that (1) *only* locked packages are
considered by libsolv, and (2) *all* locked packages were marked for
install.
One important thing to note here is that we don't short-circuit libsolv
and manually `hy_goal_install` lockfile packages. We want to make sure
the treefile is still canonical. Strict mode simply ensures that the
result agrees with the lockfile.
That said, even in developer contexts, we don't want the
`libcurl-minimal` issue that happened to be triggered. But we still want
to allow flexibility in adding and removing packages to make hacking
easier. I have some follow-up patches which will enable this.
In FCOS we have a kola test that basically does `rpm -q python`.
It's...a bit silly to spawn a whole VM for this. Ensuring that
some specific packages don't get included has come up in a few
cases.
I think FCOS/RHCOS at least will want to blacklist `dnf` for example.
And as noted above, FCOS could blacklist `python`.
One major benefit of doing this inside rpm-ostree is that one
gets the full "libsolv error message experience" when dependency
resolution fails, e.g. blacklisting `glibc` I get:
```
Problem 79: conflicting requests
- package coreos-installer-systemd-0.1.2-1.fc31.x86_64 requires coreos-installer = 0.1.2-1.fc31, but none of the providers can be installed
- package coreos-installer-0.1.2-1.fc31.x86_64 requires rtld(GNU_HASH), but none of the providers can be installed
- package glibc-2.30-10.fc31.x86_64 is filtered out by exclude filtering
- package glibc-2.30-7.fc31.x86_64 is filtered out by exclude filtering
- package glibc-2.30-8.fc31.x86_64 is filtered out by exclude filtering
- package glibc-2.30-5.fc31.i686 is filtered out by exclude filtering
- package glibc-2.30-5.fc31.x86_64 is filtered out by exclude filtering
- package glibc-2.30-10.fc31.i686 is filtered out by exclude filtering
```
Translate RPM paths under `/var/run` to `/run` automatically; this
quiets down systemd. Since we end up running `systemd-tmpfiles`
a few times in FCOS reducing spew here is particularly valuable.
The bug is really in the packages here but...we don't have an
agile process for fixing them.
Note that for this fix to take effect, if you have a `cache/pkgcache-repo`
you'll need to remove it.
This is a follow-up hack to #1797 to force libdnf to let us use modular
packages as if they were regular packages until we actually support
modules correctly (#1435).
A repo marked as a modular hotfix means that libdnf doesn't try to
filter out modular RPMs from the repo as it usually does.
Resolves: https://pagure.io/releng/failed-composes/issue/717
Again, a lot going on here, but essentially, we adapt the compose tests
to run either privileged or fully unprivileged via supermin, just like
cosa.
I actually got more than halfway through this initially using `cosa
build` directly for testing. But in the end, we simply need more
flexibility than that. We want to be able to manipulate exactly how
rpm-ostree is called, and cosa is very opinionated about this (and may
also change from under us in the future).
(Another big difference for example is that cosa doesn't care about
non-unified mode, whereas we *need* to have coverage for this until we
fully kill it.)
Really, the most important bit we want from there is the
unprivileged-via-supermin bits. So we copy and adapt that here. One
obvious improvement then is sharing this code more easily (e.g. a
`cosa runasroot` or something?)
However, we still use the FCOS manifest (frozen at a specific tag). It's
a realistic example, and because of the lockfiles and pool, we get good
reproducibility.
Prep for making `--unified-core` the only path. It turns
out our compose testsuite has a lot of hardcoded ideas about
how the two paths work. The rojig tests in particular need
cached RPMs, so we can't just rely on caching the pkgcache repo.
Add a `--download-only-rpms` that always returns RPMs, and doesn't
import into the pkgcache repo.
Closes: #1798
Approved by: jlebon
We had two `libcomposetest.sh` which I always found confusing.
Fix the naming of the one that's shared with `ex-container`
to be more obvious.
Closes: #1543
Approved by: jlebon
The `f28-compose` test keeps timing out. Some time recently, I/O
performance of the internal OpenStack instance used for testing has
degraded. I have a ticket open to investigate the regression though
haven't had any luck so far.
Let's just take the easy way out and split the test into two testsuites.
This is obviously hacky, and sad, and unfortunate. But the PRs must keep
flowing until we finally wean off of OpenStack.
Closes: #1498
Approved by: cgwalters
Make logging work the same as it does for the vmcheck-STI work
(at some point I'll try to unify the 3 parallel+script implementions
we have). This fixes the problem that when the test times out,
the filename won't have `.txt` and S3 won't have the right MIME type.
Closes: #1479
Approved by: jlebon
- Actually use separate `${test_tmpdir}` for test setup (closes a race)
- Merge stdout/stderr (more readable)
- Ensure logs are renamed to `.txt` even on failure
- Use `--progress` for some feedback
- Use `-j +1` so that even on unicore machines we get at least 2
jobs (and in general NCPUS+1)
Closes: #1188
Approved by: jlebon
Over a year later, the "opening the host rpmdb" bug is fixed,
so we can do composes in parallel ∥, hooray!
I'm dusting this off since we were running into CI (PAPR) timeouts
when I was adding more to the compose tests.
Closes: #545
Approved by: jlebon
When we added the `--ex-unified-core` option our caching story got
very messy because the non-unified core caches RPMs, but unified
does ostree repo caching.
For jigdo, we want the RPMs. Fix this by mirroring the RPMs using
`--download-only` and pointing the tests consistently at that.
Closes: #1122
Approved by: jlebon
We have some unit-style tests that run `ex container`, but
they aren't "real"; they don't use scripts for example. Let's
add tests for this similar to `tests/compose`.
We use a 26 base, but the target repos need to be 27
to pick up the fix for: https://bugzilla.redhat.com/show_bug.cgi?id=1478172
Add some bits to share infra between `tests/compose` and `tests/ex-container`;
basically handling the rpmmd repos. I tweaked things to be more streamlined
there between the `.papr.yml` and the test script.
Right now this is just one test for `bash`, but lays some of the infrastructure
for doing more. One thing that we need to do to improve more here is to better
cache RPMs, a bit like the compose tests do.
Closes: #1024
Approved by: jlebon
We can be a bit less wasteful here by merging the check and vmcheck
suites into a single suite. The check suite today takes a negligible
amount of time to run, so we're not gaining much by parallelizing them.
It's more of a sanity check at this point before we start vmcheck.
Also start running vmcheck on CentOS 7. We adapt the ci scripts to
accomodate both Fedora and CentOS target machines.
This commit also switches to Fedora 26 as the primary test base.
Closes: #871
Approved by: cgwalters
