IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
There's a lot going on here, but essentially:
1. We change the `vmcheck` model so that it always operates on an
immutable base image. It takes that image and dynamically launches a
separate VM for each test using `kola spawn`. This means we can drop
a lot of hacks around re-using the same VMs.
2. Following from 1., `vmoverlay` now takes as input a base image,
overlays the built rpm-ostree bits, then creates a new base image. Of
course, we don't have to do this in CI, because we build FCOS with
the freshly built RPMs (so it uses `SKIP_VMOVERLAY=1`). `vmoverlay`
then will be more for the developer case where one doesn't want to
iterate via `cosa build` to test rpm-ostree changes. I say "will"
because the functionality doesn't exist yet; I'd like to enhance
`cosa dev-overlay` to do this. (Note `vmsync` should still works just
as before too.)
3. `vmcheck` can be run without building the tree first, as
`tests/vmcheck.sh`. The `make vmcheck` target still exists though for
finger compatibility and better meshing with `vmoverlay` in the
developer case.
What's really nice about using kola spawn is that it takes care of a lot
of things for us, such as the qemu command, journal and console
gathering, and SSH.
Similarly to the compose testsuites, we're using parallel here to run
multiple vmcheck tests at once. (On developer laptops, we cap
parallelism at `$(nproc) - 1`).
This is a hack to allow using `inject-pkglist` without having to build
the tree first.
Higher-level, I think we can split this back out again if we have a
`-tests` subpackage where we ship the vmcheck testsuite.
Drop the use of Ansible everywhere. In the few cases where we really
Python, just spawn a container instead.
This is required to be able to hack on Fedora CoreOS.
Closes: #1850
Approved by: jlebon
In the app, rebuild the exact command-line that the client used and pass
that to the daemon to be used as the transaction title. Especially in
transactions like `UpdateDeployment()`, we can avoid reverse-engineering
what the original command used was.
This will be used by the upcoming history feature to record the
command-line used in the journal.
Closes: #1824
Approved by: rfairley
This bumps the requirement on the controlling host to Python 3 only.
It also bumps the requirement on the target host to Python 3 as well
since FCOS doesn't ship Python 2 right now.
Though we'll need to eventually drop all Python usage anyway, but at
least let's get tests passing on FCOS first. (See related previous
patch).
Closes: #1828
Approved by: cgwalters
In Fedora 29, and Fedora 30 Silverblue, I have come across the
following error when executing `make vmsync` from my build container
(also on Fedora 29 and Fedora 30 images respectively):
```
...
Failed to connect to new control master
...
Control socket connect(/var/tmp/ssh-vmcheck-1556768111752693879.sock): Connection refused
Failed to connect to new control master
...
```
Previously this worked with Fedora 28 as the host.
After changing the socket to be in /dev/shm, the SSH connection to
the `vmcheck` VM is successful and the sources sync over.
The cause of this seems to be a problem with overlayfs and unix
sockets: https://github.com/moby/moby/issues/12080
Since overlayfs is the default graph driver in Fedora now, work
around this by switching the socket to be in /dev/shm.
Closes: #1827
Approved by: jlebon
One question I often have when looking at the output of `status -a`:
```
AvailableUpdate:
Version: 29.20181202.0 (2018-12-02T08:37:50Z)
Commit: dece5737a087d5c6038efdb86cb4512f867082ccfc6eb0fa97b2734c1f6d99c3
GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
SecAdvisories: FEDORA-2018-042156f164 Unknown net-snmp-libs-1:5.8-3.fc29.x86_64
FEDORA-2018-87ba0312c2 Moderate kernel-4.19.5-300.fc29.x86_64
FEDORA-2018-87ba0312c2 Moderate kernel-core-4.19.5-300.fc29.x86_64
FEDORA-2018-87ba0312c2 Moderate kernel-modules-4.19.5-300.fc29.x86_64
FEDORA-2018-87ba0312c2 Moderate kernel-modules-extra-4.19.5-300.fc29.x86_64
FEDORA-2018-f467c36c2b Moderate git-core-2.19.2-1.fc29.x86_64
Diff: 67 upgraded, 1 removed, 16 added
```
is "How serious and relevant are these advisories to me? How soon should
I reboot?". For the packages that I'm most familiar with, e.g. `kernel`
and `git-core`, I usually look up the advisory and check why it was
marked as a security update, mentioned CVEs, and how those affect me.
The updateinfo metadata includes a wealth of information that could be
useful here. In Fedora, CVEs treated by the security response team
result in RHBZs, which end up attached to the advisories and thus make
it into that metadata.
This patch tries to reduce friction in answering some of those questions
above by checking for those CVEs and printing a short description in the
output of `status -a`. Example:
```
AvailableUpdate:
Version: 29.20181202.0 (2018-12-02T08:37:50Z)
Commit: dece5737a087d5c6038efdb86cb4512f867082ccfc6eb0fa97b2734c1f6d99c3
GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
SecAdvisories: FEDORA-2018-042156f164 Unknown net-snmp-libs-1:5.8-3.fc29.x86_64
CVE-2018-18065 CVE-2018-18066 net-snmp: various flaws [fedora-all]
https://bugzilla.redhat.com/show_bug.cgi?id=1637573
FEDORA-2018-87ba0312c2 Moderate kernel-4.19.5-300.fc29.x86_64
FEDORA-2018-87ba0312c2 Moderate kernel-core-4.19.5-300.fc29.x86_64
FEDORA-2018-87ba0312c2 Moderate kernel-modules-4.19.5-300.fc29.x86_64
FEDORA-2018-87ba0312c2 Moderate kernel-modules-extra-4.19.5-300.fc29.x86_64
CVE-2018-16862 kernel: cleancache: Infoleak of deleted files after reuse of old inodes
https://bugzilla.redhat.com/show_bug.cgi?id=1649017
CVE-2018-19407 kernel: kvm: NULL pointer dereference in vcpu_scan_ioapic in arch/x86/kvm/x86.c
https://bugzilla.redhat.com/show_bug.cgi?id=1652656
FEDORA-2018-f467c36c2b Moderate git-core-2.19.2-1.fc29.x86_64
CVE-2018-19486 git: Improper handling of PATH allows for commands to executed from current directory
https://bugzilla.redhat.com/show_bug.cgi?id=1653143
Diff: 67 upgraded, 1 removed, 16 added
```
Including the CVE name and RHBZ link also makes it easier to look for
more details if desired.
Closes: #1695
Approved by: rfairley
Our trick of using layered commits as base commit updates doesn't jive
well with the new opportunistic reuse of the base rpmdb. The issue is
that a layered commit includes the rpmdb of *its* base commit at the
`/usr/lib/sysimage` location. So to convert it into a proper base commit
means that the layered rpmdb should move there.
Closes: #1502
Approved by: cgwalters
Since `/tmp` might be on tmpfs, so we'd lose it on reboot. But we have
tests that need it to persist across reboots.
Closes: #1531
Approved by: miabbott
First, split it into its own section; it's important enough to merit it.
Second, explicitly reference the systemd timer/service units. For
example, a question I often have is "when is the next run" and of course
you can get that rpm `systemctl status rpm-ostreed-automatic.timer` but
you have to know that, and the reminder helps.
(I briefly looked at implementing the `Trigger` line from `systemctl status`
but it's not entirely trivial...tempting to just fork off a `systemctl status | grep `)
Prep for unifying this text with the message we print when one does
`rpm-ostree upgrade` when auto-updates are enabled.
Closes: #1432
Approved by: jlebon
First the pinning tests would try to pin a staged deployment,
and some of the later tests here depend on a subtle way on the
state of the system. It's tempting to do a `reset` before each one
and reboot but this makes things work.
There's some additional assertions here as I went through and
was debugging.
Prep for making staging the default.
Closes: #1438
Approved by: jlebon
Following up to https://github.com/projectatomic/rpm-ostree/pull/1352
AKA 506910d930
which added an experimental flag to globally enable deployment
staging, let's add an `ex-stage` automatic update policy.
I chose to create a new `test-autoupdate-stage.sh` and rename
the previous one to `test-autoupdate-check.sh` in going with
the previous theme of smaller test files; it's
way faster to iterate on new tests when it's a new file. And adding
staging at the top would have been weird.
This was all quite straightforward, just plumbing through lots
of layers.
Closes: #1321
Approved by: jlebon
Now we stop running rpm-ostreed as non-root, which is going to be
a maintenance pain going forward. If we do introduce non-VM based
tests I think we should look to doing in-container testing.
Closes: #1339
Approved by: cgwalters
The unit tests run an rpm-ostree daemon as non-root, which worked
surprisingly well for quite a while. But it started failing when
working on a patch which adds caching that writes to `/var`.
Since we have the vmcheck system now, let's switch over to that.
This PR moves the random "basic" tests we'd accumulated like
one to verify `StateRoot` is only in `status --verbose`, but not
the tests for the `rebase` command etc.
Closes: #1336
Approved by: jlebon
When juggling multiple test VMs for different purposes, it's useful to
be able to easily e.g. `make vmcheck` a specific one by overriding the
ssh-config file to use, rather than editing the latter each time.
Closes: #1324
Approved by: cgwalters
Because otherwise, there's no way to see the output of the script.
Also, turn off `gather_facts` since in the majority of cases, we don't
need it, so let's avoid the overhead. We can make this an opt-in flag
later on if needed.
Closes: #1304
Approved by: cgwalters
This adds a shell primitive to make it easy to execute a playbook
task list.
The big picture idea is to sync with https://github.com/ostreedev/ostree/pull/1462
and rewrite some of the libvm shell stuff as playbooks, allowing easier
code sharing with a-h-t and just in general being a better library for
talking ssh and executing commnads.
Closes: #1297
Approved by: jlebon
We would error out when trying to start the transient httpd service if
it already exists, e.g. from a previous test.
Depending on how we exit, the `vm_stop_httpd` trap for the previous test
might not have been able to kick in. I think this happens when we exit
using `fatal`, which just does an `exit 1`. It's not strictly an error,
so doesn't trip the `ERR` handler.
Let's just go the extra mile and explicitly delete transient services if
they already exist.
Closes: #1284
Approved by: cgwalters
Pick up security advisories when checking for pending updates and
include them in the `cached-update` property. On the client-side,
display them in the output of `status`.
This was part of the original vision for how useful a smart `check` mode
could be. It directly impacts how one manages their individual system
(e.g. when to reboot), and paves the way for integration into
higher-level apps that act at the cluster level.
Closes: #1249
Approved by: cgwalters
This patch introduces a new `AutomaticUpdatePolicy` configuration. This
was a long time coming for rpm-ostree, given that its update model makes
it extremely apt for such a feature.
The config supports a `check` mode, which should be very useful to
Atomic Workstation users, as well as a `reboot` mode, which could be
used in its present form in simple single node Atomic Host situations.
There is still a lot of work to be done, including integrating
advisories, and supporting a `deploy` mode. This feature hopefully will
be leveraged as well by higher-level projects like GNOME Software and
Cockpit.
Closes: #1147
Approved by: cgwalters
This reverts commit 1ef259ed76.
Basically, for historical POSIX compatibility, `errexit` doesn't work at
all inside functions called from an if-statement/boolean context. This
is something I had already learned (and forgotten) when hacking on PAPR.
There are ways around this, but it's just not worth the added complex
shell goop for what it gives us.
http://mywiki.wooledge.org/BashFAQ/105Closes: #1215
Approved by: cgwalters
The output from `rpmbuild` makes test results harder to comb through
when debugging. Let's just dump all that to file and only output it if
something goes wrong.
Closes: #1212
Approved by: cgwalters
This is the first step towards unifying how we introspect packages from
a specific commit. We currently do this in three ways: libdnf, librpm,
and now `rpmostree.rpmdb.pkglist`. I'd like to get to a point where we
only have `rpmostree.rpmdb.pkglist` and libdnf, the latter only when
more complex queries are required.
This patch teaches the `db diff` command to make use of the new db diff
API so that it can work even on metadata-only commits. This is relevant
for use cases mentioned in #558.
I didn't get rid of the `rpmhdrs_diff` functions right now because of
the `--changelogs` option: libdnf currently does not expose this, so we
fall back to the previous API in that case. OTOH, I wonder how much it's
actually used in the wild; maybe we could just nix it?
Closes: #1162
Approved by: cgwalters
We originally needed the pkgcache to be a separate repo due to ostree's
overzealous pruning policies. The idea was to maintain multiple commits
in each pkg branch for different SELinux policies. In practice, there's
not much use in maintaining old copies and it's just easier to always
relabel on the fly. So then, the need for a separate repo completely
melts away.
This helps simplify the mental model a bit and allows us to avoid subtle
issues like #1047. Note however that the core is still capable of
handling split repos for the `--ex-unified-core` compose use case. Once
that and the jigdo work are a bit more settled, we can have a clearer
picture of how to simplify the core further.
The tricky bit is migrating the cache. When deploying, we check if a
pkgcache repo exists and migrate its refs if so. We then leave behind a
symlink to the system repo to remain compatible with older rpm-ostrees.
Closes: #1055
Approved by: cgwalters
I've been lazy about actually using using rsync instead of scp when
copying new RPMs over to the VM. We do this here. Also make
`vm_send_test_repo` take a mode argument that allows callers to
completely skip the sending of the repo file itself. This will be needed
for the `makecache` test, in which we *don't* want the repo to be local.
It looks cleaner anyway for the gpgcheck use case as well.
Closes: #1035
Approved by: cgwalters
Prep for implementing `rpm-ostree cancel`, but this works with the way we handle
`Ctrl-C` interactively on a client as well. Being able to cancel a script
execution is quite nice; some of them are expensive, and having one loop forever
has been known to happen.
Closes: #1025
Approved by: jlebon
There's a lot of paths in the core related to SELinux policy changes and
relabeling packages. We currently have no test coverage for them. We add
support in the test libraries here to build such packages.
We also add a test that checks both that we correctly relabel RPMs when
the policy changes and that we handle layered packages that install
SELinux packages properly.
Closes: #999
Approved by: cgwalters
We pretty standardized on insttree/ being the installation tree, and
we don't actually support $VMCHECK_INSTTREE anymore.
Closes: #1032
Approved by: cgwalters
Rather than just letting the scriptlets inherit the daemon's
stdout/stderr, redirect their outputs so that we can set a customized
identifier to make it easier to distinguish from the daemon output.
Also print out the `journalctl` command needed so that users can
investigate the output themselves.
Closes: #998
Approved by: cgwalters
It turns out there's a much smaller limit than PATH_MAX for Unix
sockets. On Linux, it's 108 characters. It took me some time to figure
out why `vmcheck` would sometimes fail depending on where the src
directory is and how ${topsrcdir} is defined. Let's just make things
safer by just using /var/tmp.
Closes: #949
Approved by: cgwalters
Now that we require clients to have an active session to RegisterClient,
we can't use runuser to check for non-root functionality. Add a new
vm_cmd_as() function to allow connecting as a different user. While
we're there, do some minor cleanups to consistently use `local` when
possible.
Closes: #894
Approved by: cgwalters
Add experimental support for replacing packages from the base layer with
local RPMs. This is useful for example, to cherry pick a fixed package,
or to roll back to a previous package version. Like with pkg removals,
only files in /usr are actually replaced.
This patch also contains a few usability improvements as well, e.g.
showing the full NEVRA of removed packages rather than just their names,
and support for resetting overrides using either the pkgname or NEVRA.
Closes: #852
Approved by: cgwalters
Having to prepare RPMs in advance separately from the tests that use
them severely limits our ability to test various cases and to iterate
quickly when creating tests.
Add a new `build_rpm` function which can basically build the RPM on the
fly and update the yum repo afterwards. It makes it trivial to test
things like package updates:
build_rpm foo 1.0 1
<stuff>
build_rpm foo 1.0 2
<stuff>
The RPMs are all created inside the temporary test directory and thus
cleaned up on exit.
I'm doing this in a separate commit because it's the most important diff
of the transition and might be easy to lose in the larger diff where we
move all the tests to make use of this.
Closes: #854
Approved by: cgwalters
There are a few different use cases here. First, for layering new packages,
there's no good reason for us to force a reboot. Second, we want some support
for cherry-picking security updates and allowing admins to restart services. Finally,
at some point we should offer support for entirely replacing the running tree
if that's what the user wants.
Until now we've been very conservative, but there's a spectrum here. In
particular, this patch changes things so we push a rollback before we start
doing anything live. I think in practice, many use cases would be totally fine
with doing most changes live, and falling back to the rollback if something went
wrong.
This initial code drop *only* supports live layering of new packages. However,
a lot of the base infrastructure is laid for future work.
For now, this will be classified as an experimental feature, hence `ex livefs`.
Part of: https://github.com/projectatomic/rpm-ostree/issues/639Closes: #652
Approved by: jlebon
I need to build ostree from git too. So now my workflow is:
```
export insttree=/srv/walters/tmp/rootfs
cd ostree
make && make install DESTDIR=${insttree}
cd rpm-ostree
make && make install DESTDIR=${insttree}
env VMCHECK_INSTTREE=${insttree} make vmoverlay
```
Closes: #705
Approved by: jlebon
1. Don't require an ssh-config
In the case of redhat-ci, the VMs are already fully configured for the
system (injected in the hosts file, host key accepted, etc...). So
there's no need to have an ssh-config there. In general, it should be
acceptable to run the vmcheck suite against a resolvable host without
having to create an ssh-config for it.
2. Make the host name configurable
Rather than hardcoding "vmcheck" as the hostname, allow overridding it
by specifying a VM env var directly. We also prepare the various scripts
to make use of the $VM variable whenever host-specific dirs/files are
created so that parallel runs won't step on each other.
Closes: #675
Approved by: cgwalters
While reading a recent conversation about GPG checking at treecompose
time, I had a sudden thought - were we actually doing verification
client side? Turned out, we aren't. That happens as part of
`dnf_transaction_commit()` which we don't use.
That function verifies every package at one go, but for us I think it's better
to do it before "importing". We shouldn't have untrusted bits that we've
unpacked (they might have suid binaries, for one thing).
This is an embarassing problem, but it's worth emphasizing that everyone should
be retrieving repodata at a minimum over TLS, which sets a baseline. On RHEL, we
already do pinned TLS, and there are discussions about extending that elsewhere.
See: https://bugzilla.redhat.com/show_bug.cgi?id=1422157Closes: #656
Approved by: jlebon
