IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
At one point `rpm-ostree install libvirt` dragged in libguestfs which in turn
brought in `syslinux-extlinux-nonlinux` which has files in `/boot/extlinux`,
which we rejected. (That dependency chain appears to have been fixed currently)
For the general case, this is just a partial fix in that we haven't nailed down
the semantics of how updates for `/boot` work. But in this particular case,
we'll just break libguestfs' `extlinux` verb, which I'm OK with.
Another case is `fwupdate-efi` - we require manual intervention to copy the
data into `/boot` after installing the package.
This is also preparation for [unified core](https://github.com/projectatomic/rpm-ostree/issues/729)
in that we now ensure imported kernels don't end up in `/boot` unless
explicitly configured.
Closes: https://github.com/projectatomic/rpm-ostree/issues/853Closes: #969
Approved by: jlebon
Let's make using a custom install tree easier and document the process.
We split out the insttree step into `build.sh` so that we no longer have
to `flock(1)` around it, and also share between `overlay.sh` and
`sync.sh`.
Closes: #968
Approved by: cgwalters
While working on unified core and the Fedora Atomic Host content set, I hit a
dependency between `docker.posttrans` which tries to read `/etc/os-release`, and
`fedora-release-atomichost.post` which creates that symlink.
It seems best practice to me to run `%post`s strictly before
`%posttrans`; we're not likely to do parallelization anytime
soon anyways.
While here I cleaned things up by having an enum for the script kind,
rather than multiple functions, otherwise we would have had another
wrapper in core.c.
Closes: #963
Approved by: jlebon
It turns out there's a much smaller limit than PATH_MAX for Unix
sockets. On Linux, it's 108 characters. It took me some time to figure
out why `vmcheck` would sometimes fail depending on where the src
directory is and how ${topsrcdir} is defined. Let's just make things
safer by just using /var/tmp.
Closes: #949
Approved by: cgwalters
Since we have a copy of this libostree code, pick up the new
changes from <https://github.com/ostreedev/ostree/pull/1055>.
Note the added test doesn't really test our logic since
we're only doing local pulls, but at least we have something.
Closes: #932
Approved by: jlebon
This closes a longstanding bug - since package layering first
landed, we only checked for newer RPMs if the base tree changed.
In some scenarios like RHELAH, this doesn't matter much by default
since they move at the same cadence. Except if you use EPEL for example.
In Fedora, today the FAH releases are async of the rpm-md repos, and
there's also COPR which can update more than once a day even.
We should check for both update sources. Luckily we'd already introduced logic
for this in the treecompose case (checksumming the depsolved package sack). We
just need to start using it for client side assembly too.
Closes: https://github.com/projectatomic/rpm-ostree/issues/391Closes: #911
Approved by: jlebon
I'm working on getting the vmcheck suite working as part of Fedora's new
CI pipeline. In that context, we want to test the deliverable as it is,
i.e. with SKIP_VMOVERLAY=1. For compatibility with the testsuite, we
ensure that the machine is on the vmcheck branch before starting the
tests.
Eventually, we should try to make the vmcheck suite runnable outside of
a configured build directory to make it easier to re-use in such
contexts.
Closes: #917
Approved by: cgwalters
The rebase command syntax has confused people a lot. Let's follow
git here and add a `-b/--branch` option and encourage people to use
that. The case of switching remotes is `-m/--remote`; it's definitely
unfortunate that `-r` is already taken for `--reboot`.
One thing I'm a little bit unhappy about is how we're doing logic
on the client side here. Changing the DBus API for this would
also be awkward though.
Closes: https://github.com/projectatomic/rpm-ostree/issues/886Closes: #890
Approved by: jlebon
The new API to find pending and rollback deployments do so relative to
the booted deployment. This caused an interesting behaviour: the first
time a user uses "rpm-ostree rollback", it would (as expected) move the
previous deployment first. but the second call to "rpm-ostree rollback"
would fail since there were now no more rollback deployments.
We fine tune the logic here to allow this, as well as the more general
case of putting the booted deployment back on top.
This fixes a subtle regression from b7cf58e
(https://github.com/projectatomic/rpm-ostree/pull/767).
Closes: https://github.com/projectatomic/rpm-ostree/issues/906Closes: #907
Approved by: cgwalters
Fix regression from https://github.com/projectatomic/rpm-ostree/pull/870
which caused `vmsync` and `vmoverlay` to no longer actually overlay
ostree bits.
We go back to using `--files-from`, but just make sure to filter out the
stuff that we don't need (and which previously caused issues).
Closes: #907
Approved by: cgwalters
When commit metadata contains ostree.endoflife attribute,
its information will be added to the deployment Variant,
which will later be shown as a red & bold message when
'rpm-ostree status' command is called.
A test is added for future regression
Closes: #889
Approved by: cgwalters
File triggers are a post-RHEL7 thing; more information at
http://rpm.org/user_doc/file_triggers.html
There are two notable users I've been testing this with;
`glib2` and `vagrant`. The `vagrant` one is more immediately urgent,
since it makes `vagrant-libvirt` work, which I currently rely on
for my workstation dev.
I've tested things successfully with `vagrant`, and I did verify that we run the
`glib2` ones when doing `rpm-ostree ex container`.
Long term, more transaction file triggers are likely to live in
"base" packages like `glib2`. We don't implement those yet, but
extending this to do that shouldn't be too hard.
There was *significant* what I'd call reverse engineering of the
implementation in librpm. The file triggers code there is spread out
and abstracted in a few different places in the code. I found
trying to understand what header values were involved to be quite
tricky.
There are some corner cases like multiple patterns that I *think*
this does correctly, but could use more validation. The main
question I had was - is it required that the patterns for e.g.
`%transfiletriggerin` and `%transfiletriggerun` be identical?
Closes: https://github.com/projectatomic/rpm-ostree/issues/648Closes: #869
Approved by: jlebon
Follow-up tweak to #894. Make the client smarter so we only register
when we know we can. We could be more sophisticated here and e.g.
introduce the concept of "read-only" clients in the daemon to only allow
access to non-mutating methods, though let's delay that discussion at
least until the daemon learns to auto-exit.
Closes: #898Closes: #900
Approved by: cgwalters
This is a followup to https://github.com/projectatomic/rpm-ostree/pull/888
but more comprehensive; in the layering case, the sanitycheck runs
after all the `%posttrans` scripts, so we'll get a consistent error message
for the `rm -rf /` test.
We also do the sanitycheck for the "pure ostree" case, as well as cases
where we didn't actually layer packages (including `ex override remove` as
well as simply regenerating an initrd).
There's obviously a lot more we could do in a sanitycheck; as I say in the
comment it's tempting to consider trying to boot systemd (in a fully volatile
config), but for now let's do this. In the end of course the admin has rollback
too.
Closes: #892
Approved by: jlebon
Now that we require clients to have an active session to RegisterClient,
we can't use runuser to check for non-root functionality. Add a new
vm_cmd_as() function to allow connecting as a different user. While
we're there, do some minor cleanups to consistently use `local` when
possible.
Closes: #894
Approved by: cgwalters
This test would actually fail even if the bin user were allowed to
install a package because there are no enabled repos to install. Fix it
so that we know we have foo there and explicitly check that the error
message is what we expect.
Closes: #894
Approved by: cgwalters
The version checking function in particular is really useful for people doing
`from gi.repository import RpmOstree`, which we'd like at least some things like
Anaconda and Pungi to do.
Closes: #891
Approved by: jlebon
For https://pagure.io/atomic-wg/issue/299 we need to make it
more convenient to substitute the architecture in an installation
context. I plan to use this API inside `rpmostreepayload` in Anaconda,
so we can substitute the same value of `${basearch}` we use in treefiles
since https://github.com/projectatomic/rpm-ostree/pull/305
Now, you might wonder - why do we need an API wrapping libdnf? It's because
libdnf is not API stable yet. We're just exposing a tiny subset. In theory we
could use the Python dnf bindings in Anaconda, but things get slightly weird if
rpmostreepayload depends on dnf. Perhaps we'll do that down the road, but for
now this a small API surface to maintain (forever).
This change reworks the internal `varsubst` bits to take a pure `DnfContext`,
since we don't want to spin up a whole `RpmOstreeContext` just to do some
string substitutions.
Closes: #877
Approved by: jlebon
I was thinking today about our script handling, and I realized
an excellent way to showcase the advancement rpm-ostree makes
over traditional package managers is the fact that we survive a
`%post` script that does `rm -rf /`!
See e.g. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858521
It's been this way ever since we first implemented package layering;
the fact that we construct a new root and use bubblewrap to sandbox
makes us very resilient to this type of thing.
But, let's add a test case for this to be sure we preserve this behavior; for
example, if in the future we for some reason we decide to leak some host state
into the scripts.
Closes: #888
Approved by: jlebon
This is required for glibc-all-langpacks at least:
https://bugzilla.redhat.com/show_bug.cgi?id=1367585
Otherwise, its usage is...extraordinarily rare. In fact looking at a snapshot of
`rpm-specs-20170518.tar.xz` from Fedora, the only other use is in
`postfix.spec`, and it appears bogus (the value is already expanded at build
time).
But the glibc case is special, as the value of `install_langs` is indeed
potentially dynamic per system.
Closes: #873
Approved by: jlebon
We can be a bit less wasteful here by merging the check and vmcheck
suites into a single suite. The check suite today takes a negligible
amount of time to run, so we're not gaining much by parallelizing them.
It's more of a sanity check at this point before we start vmcheck.
Also start running vmcheck on CentOS 7. We adapt the ci scripts to
accomodate both Fedora and CentOS target machines.
This commit also switches to Fedora 26 as the primary test base.
Closes: #871
Approved by: cgwalters
The `install` command in CentOS 7 is too old to understand that
`-Dt foo/bar` means creating both `foo` and `bar`, which is useful so
that we avoid an explicit `mkdir` before. But we can't do that here.
Closes: #871
Approved by: cgwalters
Add experimental support for replacing packages from the base layer with
local RPMs. This is useful for example, to cherry pick a fixed package,
or to roll back to a previous package version. Like with pkg removals,
only files in /usr are actually replaced.
This patch also contains a few usability improvements as well, e.g.
showing the full NEVRA of removed packages rather than just their names,
and support for resetting overrides using either the pkgname or NEVRA.
Closes: #852
Approved by: cgwalters
Make sure that we wipe out any leftover configuration files from a
previous run before layering the test pkg, or we'll get false positive.
Also make sure to correctly clean up the VM in the case the livefs test
errors out.
Closes: #859
Approved by: cgwalters
Seen in the wild with `vagrant`'s use of `%post -p /usr/bin/ruby`. This was a
very easy fix, and actually makes the code a little bit nicer, as we no longer
need to explicitly make the script executable, since we now pass it as
`argv[1]`, the same way librpm does. That in turn would make it possible to fix
the TODO and use `bwrap --file`, but that can come later.
Closes: https://github.com/projectatomic/rpm-ostree/issues/856Closes: #858
Approved by: jlebon
Nuke all the previous goop that was used to create RPMs at `make check`
time and transition all the tests to use the new `build_rpm` function.
It definitely feels cleaner to use. It's also really nice to have the
spec live in the same file as the test that uses it.
Closes: #854
Approved by: cgwalters
Having to prepare RPMs in advance separately from the tests that use
them severely limits our ability to test various cases and to iterate
quickly when creating tests.
Add a new `build_rpm` function which can basically build the RPM on the
fly and update the yum repo afterwards. It makes it trivial to test
things like package updates:
build_rpm foo 1.0 1
<stuff>
build_rpm foo 1.0 2
<stuff>
The RPMs are all created inside the temporary test directory and thus
cleaned up on exit.
I'm doing this in a separate commit because it's the most important diff
of the transition and might be easy to lose in the larger diff where we
move all the tests to make use of this.
Closes: #854
Approved by: cgwalters
Import libostreetest.c from ostreedev/ostree as libtest.c. This is just
a really useful and outrageous way of using libtest.sh from C.
Closes: #854
Approved by: cgwalters
Minor tweak; use g_assert_no_error() before using g_assert(ret) so that
we actually get a printout of the error the test fails.
Closes: #854
Approved by: cgwalters
It seems silly that to find out more detailed information about the
NEVRA of a cached pkg, we have to resort to write out the header to
disk, then reading it back in with librpm in order to tease out the info
we want. Let's just encode that information directly in the commit
metadata and provide a helper to fetch it.
Closes: #847
Approved by: cgwalters
I didn't realize at the time I wrote the cache_branch_to_nevra test that
the already existing test-utils.c would be the perfect place to add this
test. Merge the two together now.
Closes: #847
Approved by: cgwalters
The property of removal overrides dropping out if the package was
removed from the base layer felt a bit too magical and hacky. We really
should remember that wish and re-apply it if the pkg comes back. This is
similar to package layering: requests can become inactive (seems like a
better word than "dormant") if the package is already part of the base
layer, but they don't really go away.
This patch reworks the logic so that removal overrides work the same
way. In the status output, we now have both "RemovedBasePackages" and
"InactiveBaseRemovals" (which is only printed in verbose mode),
similarly to how we have "LayeredPackages" and "InactiveRequests". And
similarly, we also print out in the upgrader during a transaction all
the inactive base removals.
Another cool thing is that we now allow any pattern to be specified at
the CLI. E.g. `ex override remove /usr/bin/strace` will resolve to
strace.
Closes: #836
Approved by: cgwalters
In the error path when trying to remove a base package, we would try to
print a DnfPackage as char*, which of course didn't result in any
coherent output.
Closes: #833
Approved by: cgwalters
To catch things like https://github.com/projectatomic/rpm-ostree/issues/826
during testing.
(Really IMO this should be the system-wide default except GNOME and
lots of apps would start dumping core all over the place... 😢)
Closes: #828
Approved by: jlebon
Copy the bits from `overlay.sh` that takes the ostree files from the
build container so that we also get updated ostree bits when we
`vmsync`.
Also make sure to `daemon-reload` before restarting the daemon.
Closes: #803
Approved by: cgwalters
During provisioning, PAPR injects a fedora.repo pointing at a much
better & faster mirror than dl.fp.o. Let's use that to make the compose
test less flaky. Hoping to make these sorts of optimizations more
discoverable in upstream PAPR.
Closes: #799
Approved by: cgwalters
There are a few reasons to do this. First, systemd changed to refuse mounts on
symlinks, and hence if one *wants* "/tmp-on-tmpfs", one would need to write a
different `sysroot-tmp.mount` unit.
Second, the original rationale for having this symlink was that if you had
multiple ostree stateroots ("osnames"), it's nicer if they had the same `/tmp`
to avoid duplication. But in practice today that's already an issue due to
`/var/tmp`, and further the multiple-stateroot case is pretty unusual. And that
case is *further* broken by SELinux (if one wanted to have e.g. an Ubuntu and
Fedora) stateroots. So let's fully decouple this and make `/tmp` a plain
old directory by default, so systemd's `tmp.mount` can become useful.
Now, things get interesting for the case where someone wants a physical `/tmp`
that *does* persist across reboots. Right now, if one just did a `systemctl mask
tmp.mount` as we do in Fedora Atomic Host's cloud images, you'd get a semantic
where `/tmp` stays per-deployment, which is weird. Our recommendation for
that should likely be to set up a bind mount for `/tmp` → `/var/tmp`.
For now, this stays an option to ensure compatibility; if FAH Cloud images
want to stay with "physical /tmp", then we'd have to change the kickstart.
Closes: https://github.com/projectatomic/rpm-ostree/issues/669Closes: #778
Approved by: jlebon
This was a kind of last-minute bug introduced when I tweaked the
checkout to use `.` to avoid a `mkdir()` for files. But there were
multiple bugs with that; for files that are in subdirectories of `/etc`
we obviously need to get the right subdir and not use `/etc`.
Second, we need to handle the case where we're adding new subdirectories.
This change fixes `rpm-ostree install firewalld + rpm-ostree ex livefs`.
Closes: #783
Approved by: jlebon
This is a valid case when layering local RPMs. Otherwise (e.g. if
specific non-local packages are requested), give the nicer error rather
than letting libdnf throw something less useful.
Closes: #780Closes: #781
Approved by: cgwalters
First, we need to preserve whitespace in the regexps. Second,
in the error path, all of our variables were wrong.
Commit 376a2cc3f5 is an excellent
poster child for how many bugs one can introduce in a single commit
for a ~5 line function.
Closes: #777
Approved by: jlebon
I was about to add another one of these but it feels like a bit
overkill to run through a recompose for trivial tweaks like turning off docs.
We can do a compose with multiple options at once and test the result as a unit,
at least for the smaller/less invasive options.
This change is prep for adding a switch to do `/tmp` as a regular dir.
Closes: #777
Approved by: jlebon
There are a few different use cases here. First, for layering new packages,
there's no good reason for us to force a reboot. Second, we want some support
for cherry-picking security updates and allowing admins to restart services. Finally,
at some point we should offer support for entirely replacing the running tree
if that's what the user wants.
Until now we've been very conservative, but there's a spectrum here. In
particular, this patch changes things so we push a rollback before we start
doing anything live. I think in practice, many use cases would be totally fine
with doing most changes live, and falling back to the rollback if something went
wrong.
This initial code drop *only* supports live layering of new packages. However,
a lot of the base infrastructure is laid for future work.
For now, this will be classified as an experimental feature, hence `ex livefs`.
Part of: https://github.com/projectatomic/rpm-ostree/issues/639Closes: #652
Approved by: jlebon
Now that rpm-ostree operations are cumulative, we need to make sure we
also clean up pending deployments. Also clean up the rollback for
consistency.
We also strengthen our ref handling. Always nuke the vmcheck ref (which
might not even exist anymore if a test rebased without --skip-purge) and
recreate it from the original ref.
Closes: #769
Approved by: cgwalters
This is a follow-up to commit 77acf62. There, we added support for
rebasing from a local branch to another local branch. But in testing,
it's also really useful to be able to rebase from a remote-based refspec
to a local branch. We allow this here by slightly expanding the syntax
of allowed refspecs.
Now, we can use rpm-ostree all the time rather than fallback to `ostree
admin deploy`, which isn't pkg-aware.
Closes: #764
Approved by: cgwalters
Conceptually: we're going to move rpm-ostree and ostree at the same
cadence most of the time; for both releases *and* for git master.
The problem so far has been the latter part. Reusing FAHC
for the build gets us half of the problem.
The other trick I realized we can do - just pull ostree out from the build
container. This avoids fetching it from the internet, and makes my workflow for
hacking on both nicer - I just `sudo make install` in my build container for
ostree.
It's tempting to make the whole thing symmetric and require `sudo make install`
for rpm-ostree and not do the insttree thing but...perhaps after.
Closes: #758
Approved by: jlebon
This was argued on list a while ago: http://lists.rpm.org/pipermail/rpm-ecosystem/2016-August/000391.html
Basically, every use of `%pretrans` is a workaround for some in-place
upgrade problem. But we never do inplace updates, we always assemble
a new tree. Hence, there is no point for us to ever execute these.
Let's stop erroring out on them. If there does turn out to be some problem with
a particular package (e.g. the `openjdk-copy-configs` or whatever), we would
likely end up needing to fix that package's `%post`, not start executing its
`%pretrans`.
Closes: #763
Approved by: jlebon
`Commit` is useful in cases where one wants to e.g. run `rpm-ostree db diff`.
`StateRoot` however is going to be in the vast majority of cases identical
across deployments.
Closes: #743
Approved by: jlebon
This is preparatory for future changes which will make use the newer sysroot
writing API. But in this change, we can drop all of our version ifdef'd code.
Closes: #740
Approved by: jlebon
I want to be able to debug a test by sshing in with the same setup;
right now the suite resets and reboots by default.
This also required fixing how we deal with `/etc/yum.repos.d`, since at the
start of the test it could be either pristine or not.
Closes: #578
Approved by: jlebon
I need to build ostree from git too. So now my workflow is:
```
export insttree=/srv/walters/tmp/rootfs
cd ostree
make && make install DESTDIR=${insttree}
cd rpm-ostree
make && make install DESTDIR=${insttree}
env VMCHECK_INSTTREE=${insttree} make vmoverlay
```
Closes: #705
Approved by: jlebon
Just like `rpm-ostree ex`, for things like `ex livefs` that have DBus
interfaces, we should segregate these off so that people know they're unstable.
And conversely that they can test for the presence of the method on the main
interface for stability.
I initially tried having the same `RpmostreeOS` object implement both
but couldn't work out how to do that; see https://mail.gnome.org/archives/gtk-app-devel-list/2017-March/msg00161.htmlCloses: #701
Approved by: jlebon
I originally thought there was a bug here, but missed we were skipping
symlinks earlier above. See [previous discussion][1].
Now, I'm not aware right now of something this patch actively fixes, but I think
it makes sense to be consistent in our handling of things here with respect to
symlinks.
1: 29dd1bd801..8158dcfb47 (r95017893)Closes: #689
Approved by: jlebon
There's no point to shipping these backup files in the base tree. We already had
code to delete them for the package layering case where they caused active harm.
At the point we added that code we really should have *also* changed treecompose
to delete them. Better late than never.
The reason I'm doing this now is because having them in the base tree causes `ex
livefs` to spuriously think that layering a package that *doesn't* change `/etc`
as if it does, because the layering code deletes the backup files.
Closes: #693
Approved by: jlebon
I built+installed ostree git master with
https://github.com/ostreedev/ostree/pull/723 in my dev container,
which broke the rpm-ostree tests. Kind of embarrassing that
I forgot to check rpm-ostree for usage of `ostree trivial-httpd`.
This is another thing that really wants a shared test container.
Anyways let's just use Python for this.
Closes: #684
Approved by: jlebon
I'd like to embed structured metadata about the originating git
repository. See [this example](https://pagure.io/fedora-atomic-host-continuous/c/142b12020d7efe18b56d039304efea102a210790?branch=master). However, I think what we really
want here is a *single* value which has subkeys.
One thing in the back of my mind too is...we could use this to
enhance our "change detection". Right now we checksum the sack,
treefile, and treecompose-post. But down the line, I'd
like to support more sophisticated postprocessing, where the
script might reference external files or the like.
In that case, we could stop checksumming the post script, and rely on whether or
not the git repo changed. (This would conversely mean we would do a build even
if e.g. the repo's `README.md` changed, but we can address that with a
post-assemble content check).
Anyways though, for now, this gets us the ability to more easily drop more
structured metadata in the commit, whether it's input git repos, tests that
passed, etc.
Note a trap that bit me here: since the metadata we write here is *host* endian,
but `ostree show --raw` byteswaps (it needs to since the core ostree variant
is always big endian), we get inverted numbers if the host is little.
I think we should probably canonicalize our metadata to big endian; this should
be pretty backwards compatible since I doubt anyone has been adding raw numbers
so far.
Closes: #676
Approved by: jlebon
Allow the `make vmcheck` target to take a HOSTS var, which is simply a
space-separated list of hosts on which we can run testsuites. Add a
multitest.py script that takes care of monitoring and scheduling the
tests onto the nodes.
The script itself is "dumb": we don't know how long each test can take,
so we can't do any smart/heuristic scheduling that could save more time.
Closes: #675
Approved by: cgwalters
1. Don't require an ssh-config
In the case of redhat-ci, the VMs are already fully configured for the
system (injected in the hosts file, host key accepted, etc...). So
there's no need to have an ssh-config there. In general, it should be
acceptable to run the vmcheck suite against a resolvable host without
having to create an ssh-config for it.
2. Make the host name configurable
Rather than hardcoding "vmcheck" as the hostname, allow overridding it
by specifying a VM env var directly. We also prepare the various scripts
to make use of the $VM variable whenever host-specific dirs/files are
created so that parallel runs won't step on each other.
Closes: #675
Approved by: cgwalters
We start by adding support in the core for installing packages strictly
from the cache repo. We fool the libdnf stack by re-exporting the header
as an RPM, and explicitly marking it for install. The treefile format
supports specifying the expected SHA-256 of the metadata header, in case
the cache for a specific NEVRA changed.
Closes: #657
Approved by: cgwalters
Pull #646 introduced a subtle regression: we went from always including
a "packages" entry to only including it if there are packages present.
Albeit it's easy to guard against, though to be nice, let's make it
easier for consumers by always including it.
Reported-by: Micah Abbott <miabbott@redhat.com>
Closes: #670
Approved by: cgwalters
While reading a recent conversation about GPG checking at treecompose
time, I had a sudden thought - were we actually doing verification
client side? Turned out, we aren't. That happens as part of
`dnf_transaction_commit()` which we don't use.
That function verifies every package at one go, but for us I think it's better
to do it before "importing". We shouldn't have untrusted bits that we've
unpacked (they might have suid binaries, for one thing).
This is an embarassing problem, but it's worth emphasizing that everyone should
be retrieving repodata at a minimum over TLS, which sets a baseline. On RHEL, we
already do pinned TLS, and there are discussions about extending that elsewhere.
See: https://bugzilla.redhat.com/show_bug.cgi?id=1422157Closes: #656
Approved by: jlebon
We don't want to expose the host's `/tmp` since that means scripts could
potentially find things like the X11 socket or whatever.
To debug things better, add a quick bash script to run bwrap like the C code
does. Perhaps down the line we can add `rpm-ostree internals run-bwrap` or so.
Closes: #647
Approved by: jlebon
There are two main issues right now; first, we don't pick up manual changes to
`.origin` files, which occurs when one needs to sed it to remove `unconfigured`
for example. Second, we need to reload changes to the remotes.
Closes: #598
Approved by: jlebon
Clean up the output of vmcheck a bit by printing out ssh debug details
and rpm-ostree status output to the log file. This will help make it
easier to quickly see which tests failed (as well as helping satisfy my
OCD for a nice output).
Closes: #645
Approved by: cgwalters
On the first call to vmsync/vmoverlay, we do an immediate vm_ssh_wait()
to check if we have a live VM. However, we don't necessarily have a
working rpm-ostree in there if we're hacking on stuff. It gets annoying
to wait for the timeout there.
Let's just work around that by instead only calling status if we got
there through vm_reboot_cmd(), which is most likely when we're most
interested in the status output anyway.
Closes: #645
Approved by: cgwalters
See https://github.com/projectatomic/rpm-ostree/issues/233 - for RPMs which
place files in e.g. `/opt`, we have different behavior in the treecompose case
(silently drop it) versus package layering (does the wrong thing).
Since the unpacker right now is only used in the layering case, this just
ensures we'll get a consistent error there.
Closes: #624
Approved by: jlebon
As part of an earlier cleanup of origin parsing, we started checking
the origin `unconfigured-state` even just starting the daemon, which
is kind of bad.
It's tempting to flip the default for the parser so that we *only* check
unconfigured state if we go to upgrade, but let's not do that in this patch.
Closes: #626
Approved by: jlebon
I'm not sure why we weren't doing this before, but we need to also
support files in /var and /run that are owned by root.
Related: RHBZ#1421781
Closes: #622
Approved by: cgwalters
We sometimes talk about using `ostree admin undeploy`, but that
doesn't know about the pkgcache, and hence space there leaks
until the next rpm-ostree operation.
Just for this, we need to expose a cleanup command (and API). But
we also need to support cleaning:
- repomd
- downloads (repo/tmp)
So let's start implementing that.
Closes: #614
Approved by: jlebon
I debated just putting this in the supported list, but decided against
it in the end. This really should be something that happens
transparently, and if it doesn't then something else is probably wrong.
Closes: #617
Approved by: cgwalters
Until now, we always used the booted deployment, and would
garbage collect the "pending" deployment. This is the
way OSTree was designed, but I think for rpm-ostree given
how mutable we are on the client side, there's a much stronger
argument for being more stateful too.
This is a relatively simple code change to split the "merge deployment"
concept into two. There's now the "config merge deployment" and the
"origin merge deployment".
Basically, `rpm-ostree install foo; rpm-ostree install bar` will
now install both `foo` and `bar`. But we will still use the booted
deployment for `/etc`.
Down the line, I think I'd like to drive into OSTree the concept of
a "staged" deployment, that has the hardlink checkout done, but doesn't
have the config merge.
But we don't need to change the OSTree core for this yet; we can
do it here in rpm-ostree, and this relatively simple code change
fixes many issues at once.
For example, `rpm-ostree upgrade && rpm-ostree install foo` now
does what you expect as well.
Obviously, we want to enable doing multiple things in *one* transaction,
and we're not far away, but I think this is also the right thing to do now.
I'm relatively confident it won't break anyone's workflow, as what
we did before wasn't generally that useful. However, people will
need to learn to `ostree admin undeploy 0` if they *don't* want
this behavior. (We need to have `rpm-ostree cleanup`).
Closes: https://github.com/projectatomic/rpm-ostree/issues/406Closes: #611
Approved by: jlebon
I'm watching https://github.com/rpm-software-management/libdnf/pull/199 and I
really don't like it. We already have a place to put out-of-rpmdb metadata,
which is in the ostree commit for imported packages. No need to involve a
relational database for this (and further, one that would need to learn about
multiple ostrees).
We're not yet *using* this information in the UI, but we could; imagine
changing the `status` `Packages:` to show packages-per-repo or so. We
could also expose an `rpm-ostree pkg-info foo`.
But for now, let's just start recording this.
Closes: #610
Approved by: jlebon
One thing that's very confusing about OSTree is there are two layers -
deployments and the refs/commits. If one does an `rpm-ostree upgrade`, but then
e.g. `ostree admin undeploy 0`, you still have the new revision in the repo.
We don't do a good job of displaying this state, or helping people clean
it up.
Down the line, I also want to better support something like `rpm-ostree pull` to
cache updates explicitly *without* deploying.
This commit just adds a bit of information to the status display. We might want
to have better formatting, but I think this an OK start.
Closes: #595
Approved by: jlebon
The actual problem I am trying to fix with this is fallout from the
introduction of `/usr/libexec/rpm-ostreed`, which required a SELinux
policy change. Specifically for CentOS, the base policy is rev'd
slowly.
My hope was that by merging the daemon code back into `/usr/bin/rpm-ostree`
which is labeled `install_exec_t`, starting via systemd would do
the right thing. It turns out that doesn't happen.
Now later, I'm picking this patch back up because I want to do multprocessing in
the daemon (and in the core), and it makes sense to share code between them,
because multiprocessing will need to go through a re-exec path.
Another benefit is we avoid duplicated text (libglnx, internal helpers) between
the two binaries.
Closes: #292
Approved by: jlebon
When we checked out the base tree for package layering, we would create
the directory in which ostree did the checkout. This meant however that
ostree wouldn't apply xattrs on the root directory itself. This would
cause the directory to be mislabeled (as system_conf_t instead of
root_t), which in turn cause SELinux violations on reboot when systemd
tried to make the root mount shared.
This patch fixes this by first settling on a permanent directory in
which to do checkouts -- really, we'll never have multiple package
layering operations going on at the same time. Once we know that we have
a reserved path, we can safely let ostree create it for us with the
proper xattrs.
Resolves: RHBZ#1318547
Closes: #605
Approved by: cgwalters
Currently we push for a model where the initramfs is
generated (in non-hostonly mode), and merely replicated.
However, to support a few unfortunate corner cases like dm-multipath which wants
to inject a config file into the initramfs, we need to support regenerating it
client side too.
Down the line, we'll need this to support overriding the kernel too.
This changes things in the core to add the concept of an "empty"
`RpmOstreeContext`. I initially tried skipping it, but that was too much
duplication. We still want all of the core ostree-related logic that lives in
that code too.
The treespec bits barfed if the spec didn't have a `tree/packages` key. It was
simplest to change that to allow it - and because that was the only case where
we errored out in parsing, I dropped the error handling.
There was another place in the upgrader that now needed to be fixed to handle
transitioning from just regenerating initramfs to not.
Closes: #574
Approved by: jlebon
Many projects do this, and it really helps debugging to know the
exact hash.
(Of course this is broken in traditional rpm builds from a tarball,
and rpmdistro-gitoverlay injects it into the Version field,
but it will help me for vmcheck debugging)
Closes: #584
Approved by: jlebon
I think since I landed a change to `--enable-new-name`, `/usr/bin/rpm-ostree`
became a symlink, and without the `-l` switch, rsync skips over them.
Hence, we have only been testing in vmcheck the old binaries, not new ones.
Oops.
Closes: #585
Approved by: jlebon
I don't see a reason not to require more stuff to execute tests,
and having it be optional reduces our coverage unnecessarily.
Closes: #580
Approved by: jlebon
Working on initramfs, I hit a subtle issue with the fact that
I was trying to "redeploy", but with the origin file changed
during the process.
Previously, it was a bit unclear which parts of the upgrader logic are operating
on the *new* origin versus the "original origin".
The package layering code in the upgrader explicitly carries a delta on top in
the "add/remove" hash sets, which means it isn't visible to
`rpmostree_origin_is_locally_assembled()`.
Whereas for initramfs, I set a new origin. This broke things since we were
expecting to find a parent commit, but the original origin wasn't locally
assembled.
When looking more at this, I realized there's a far simpler model -
rather than keeping track of commit + origin, and using the origin
to try to determine whether or not the commit is layered, we can
keep track of `base_revision` and `final_revision`, and the latter
is only set if we're doing layering.
The diff speaks for itself here - a lot of fragile logic looking at the origin
drops away.
The next step here is probably to drop away the package layering hash sets, but
I'm trying to not change everything at once.
Closes: #579
Approved by: jlebon
Our `make check` runs an unprivileged rpm-ostreed if run as non-root; this is a
feature. We didn't notice in the CI tests since those run as "docker-uid0". It
does break my local workflow though.
Closes: #573
Approved by: jlebon
We seem to have substantial infrastructure drift here between the two, and I was
*really* confused why `TEST_SKIP_CLEANUP` wasn't being honored. Turned out in
rpm-ostree we have tmpdir code in two places, not entirely sure why. Punting
full cleanup.
Closes: #566
Approved by: jlebon
First try to log in, and if we fail, retry with SSH debugging,
so we have a better idea what might be going wrong. This helped
me figure out that vmcheck's `-o User=root` assumption wasn't
working with my vagrant setup.
Closes: #560
Approved by: jlebon
Being able to just reuse metadata is especially helpful when trying to
debug things lower down that path, as well as cuts about 2 mins from the
compose test.
Closes: #561
Approved by: cgwalters
During a deploy operation, we would fetch commit objects from the remote
to resolve the version string. If gpg-verify was turned on, we would
fail to pull them if some of the commits were not signed. This is
because we pulled them in batches. We partially address this by only
fetching the HEAD commit on the first pass. This allows `upgrade`
operations to work just as well as `deploy` operations.
Though there is still an issue if we have to traverse farther back than
when signed commits become unsigned (unless they happen to fall on a
batch boundary). We leave that unsolved for now, since that would likely
require a more complex solution and it's not clear whether it's a real
world issue (signers can just retroactively sign commits).
Copy the gpghome from ostree so that we can test GPG-related cases in
our suite.
Closes: #527Closes: #557
Approved by: cgwalters
Expand the available options in the Rebase() D-Bus method to also have a
"revision" key. Its value has the same semantics as the "revision" key
in the Deploy() method (e.g. the "revision=" and "version=" prefixes are
also supported). Also expand the rebase CLI to allow for specifying the
revision as an additional argument.
This allows users to rebase to a specific version or checksum, rather
than only to the latest. Conceptually, this is the equivalent of doing a
rebase followed by a deploy. I.e. we specify an override-commit in the
origin and expect the same behaviours that apply after a deploy to also
apply here.
Closes: #212Closes: #555
Approved by: cgwalters
In the container build + vagrant test workflow, we have to run `make
vmsync` from the env in which autoconf was run, which is the container.
However, when using git worktrees (which is useful in this scenario to
avoid having to overwrite your local builds) and a CentOS build
container, the git inside is too old to understand worktrees, so let's
avoid invoking git in any of the test harness scripts.
Also make sure to restorecon after installing the new rpm-ostreed.
Closes: #555
Approved by: cgwalters
Add a few more tests to exercise some of the treefile options. We do
need to also expand test-basic.sh itself to sanity-check the structure
of a normal ostree compose. That's up next on the list.
Closes: #548
Approved by: cgwalters
Unsurprisingly, rebooting machines that are running in OpenStack is
not as reliable or as fast as a local VM, which is what vmcheck was
originally written for and tested against.
Replace the:
sleep 2 # give time for port to go down
which is rife with raciness, with a stronger boot_id-based check to
ensure we're in a new boot. Run "sync" before rebooting which sometimes
helps (though I didn't fully investigate why or whether it always helps,
there's probably something more subtle going on underneath). Increase
the timeout to 120s.
Closes: #543
Approved by: cgwalters
Our current compose tests only use a synthetic `empty.rpm`, but
this really limits usefulness.
Let's make a test suite that requires an internet connection and
downloads Fedora RPMs and does "real" tree composes.
See the updated `tests/README.md` for more information.
This is still a WIP.
Closes: #531
Approved by: jlebon
This is just the final bit required to make sure the vagrant and
non-vagrant paths can work happily together. It's mostly minor fixes,
though the most major change which also affects vagrant is that we now
sync to the root home dir, rather than ~vagrant.
Closes: #524
Approved by: cgwalters
My development environment is now using "pet" docker containers.
I use VMs for testing things that require that (like rpm-ostree).
This patch builds on work from @jlebon in
https://github.com/projectatomic/rpm-ostree/pull/509
to rework `vmcheck` such that it can work on any `ssh-config`. By
default we expect this to be Vagrant.
However, I go a lot farther and delete the `vmbuild` code that was
trying to do builds in a container on the target VM. I think this is
still worth pursuing at some point, but for now I think it's
reasonable to assume that the rpm-ostree developer audience uses Linux
as their host workstation and hence has containers.
(There's another important point here in that for developing lower
level things like rpm-ostree, there's a strong push to make the VM
disposable and not a pet)
Closes: #516
Approved by: jlebon
We further split libvm from vagrant. It no longer does 'vagrant
ssh-config'. Instead, it always assumes that an ssh-config is provided.
We now have complete separation of libvm from vagrant.
We change the ansible provisioner as follows:
- Allow passing in a VAGRANT_BOX env var to override the default
CentOS box.
- No longer assume that the root user account is unlocked and has a
valid 'vagrant' password. This worked for the centos box but isn't
sure to work on every box. Instead, we now just run ansible as the
default vagrant user, and during provisioning set up the root
account and generate an ssh-config so that libvm can connect
directly as root.
- No longer build the buildimg during provisioning. This actually
stopped working a while ago since the default rsync is disabled. We
can just let the buildimg get created on the first compilation. In
practice, the bigger issue isn't creating the buildimg, but being
able to easily update the host and buildimg pkgs.
Closes: #516
Approved by: jlebon
Thought it'd be fun to write a test for verifying proper handling of
scriptlets during package layering. There's obviously a lot more that
could go in here (patches welcome!), but it's a start.
Closes: #434
Approved by: cgwalters
This is a useless test for now since we should be able to test on a VM
even if there are updates available.
I initially had it just as an example of the vmcheck test harness,
without actually thinking too much into what it was testing. But we do
need a nice test-basic.sh that will give rpm-ostree a good general
workout.
Closes: #394
Approved by: cgwalters
We don't have a lot of outstanding changes to the C code, so now seems
like a good time to do this. I implemented this with some highly
sophisticated sed commands like:
```
find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done
```
Part of dropping the dependency on libgsystem, same as what we're
doing in ostree.
Closes: #393
Approved by: jlebon
- Rename test-layering.sh to test-layering-basic.sh and make it test
both pkg-add and pkg-remove.
- Add test-layering-relayer.sh, which verifies that pkgs are properly
relayered during the creation of new deployments (e.g. upgrades,
rebases, deploys).
- Add test-layering-rpmdb.sh, which verifies that packages respect the
rpm requirements before being overlayed.
Closes: #371
Approved by: cgwalters
Print out a better summary that sums up the passes, failures, and skips.
Silence the restoring step at the end of the testsuite by only echo'ing
to the log file. Finally, add some colouring because colours are
awesome.
Closes: #371
Approved by: cgwalters
The test-basic.sh test was failing on machines that had jq installed.
Will have to make sure to add it to the PR tester.
Closes: #360
Approved by: cgwalters
We now make the test harness handle restoring the VM to the original
state. The wonderful thing about ostree here is that it's a perfect
shoo-in for this. We make a 'backup' of the current ref, and just have
to make sure that the VM is back on that ref after running each test.
This will allow us to write tests without worrying as much about
cleaning up in the event of an error.
Closes: #360
Approved by: cgwalters
This is a preliminary package layering test which simply installs the
foo package and verifies that it functions properly. A bunch of
primitives are added to libvm.sh to facilitate this and future tests.
Closes: #344
Approved by: cgwalters
We're going to be create many different packages in the future. Let's
refactor and generalize the targets so that they can build anything.
We also add a package foo, which will be used soon for a basic test of
package layering.
Closes: #344
Approved by: cgwalters
This is a very simple test harness inspired by the atomic one. It's a
simple bash script that sets up a permanent ssh connection to the host
and runs the test scripts. Also add a "demo" test-basic.sh test to make
sure that it works.
Closes: #344
Approved by: cgwalters
If the dir that was rsync'ed already contains build artifacts, we don't
want those contaminating our build process.
Closes: #336
Approved by: cgwalters
I really want a command to cancel the running transaction, but as a
preparatory step, let's display what's going on in `status`.
The text output here is really a demonstration - I'll improve it but I
want to rework the internals first.
Closes: #320
Approved by: jlebon
According to tmpfiles.d(5), files should follow the convention
<package>.conf or <package>-<part>.conf. So we rename
tmpfiles-ostree-integration.conf to rpm-ostree-0-integration.conf.
The 0 index is so that the autovar conf created by postprocess is
sourced *after* this one, so that `integration.conf` has higher
precedence if there are duplicate entries.
Closes: #325
Approved by: cgwalters
The `make vmshell` target makes it even easier to immediately try out
your changes in a live running Atomic Host. It will automatically
provision the VM, sync your latest changes, build, install in a new
deployment onto which the VM is rebooted, and drop you in the shell.
Closes: #321
Approved by: cgwalters
In order for vmcheck to be useful for testing, we need the machine to be
in a "clean" state. That is, sitting on a commit, ready for being
manipulated.
This is a small step towards this goal. Instead of overwriting the
files, we use rofiles-fuse to safely install and create a new
deployment.
Closes: #321
Approved by: cgwalters
It's really simple to reflect the DBus API into JSON, which tools like
Ansible or `jq` from the command line or whatever can parse to find
interesting things like the current version.
Closes: #301Closes: #315
Approved by: jlebon
I find myself not liking the name rpm-ostree anymore; it's
descriptive, but unfortunately we compete with other projects with
easier and sexier names.
Also, people continually find the ostree and rpm-ostree layering
unclear. It's *much* easier to say "nxs depends on ostree", even
though textually it's obvious "rpm-ostree depends on ostree".
Anyways, just an experiment for now.
Closes: #307
Approved by: giuseppe
This is infrastructure that's oriented around Vagrant, we do builds
inside the target VM (actually inside a Docker container), but then
directly `make install DESTDIR=/host/`.
The goal here is to have a convenient workflow for:
- dev -> hand testing
- dev -> run destructive tests
Closes: #303
Approved by: jlebon
Since we now run everything uninstalled, we can't expect the tmpfiles
conf file to be installed. We add an env var that will allow us to tell
rpm-ostree to look elsewhere. This is then used in test-compose.sh.
Closes: #304
Approved by: cgwalters
- Move all the uninstalled, non-destructive tests to check/
- Add a README
- Move utilities to utils/
- Move common files to common/
Closes: #304
Approved by: cgwalters
For now, we don't remove the ability to install tests (e.g. the
necessary blurbs in glib-tap.mk are still there), but we take out all
our previously installed tests from the list.
All the previously installed tests are then converted to work
uninstalled. This also allows us to do some simplifications in the test
cases.
Also do some cleanups in the Makefile to improve legibility.
Closes: #304
Approved by: cgwalters
This works around a potential issue with libsolv if we go down the
rpmostree_get_pkglist_for_root() path. Though rpm has been using the
/usr/share/rpm location (since the RpmOstreeContext set the _dbpath
macro), the /var/lib/rpm directory will still exist, but be empty.
libsolv gets confused because it sees the /var/lib/rpm dir and doesn't
even try the /usr/share/rpm location, and eventually dies when it tries
to load the data.
So we set the symlink now. This is also what we do on boot anyway for
compatibility reasons using tmpfiles.
This also means we don't have to do the /var/lib/rpm --> /usr/share/rpm
transition during the rootfs postprocess (but we still have to clean up
db and lock files).
Also get rid of the unused pkglist variable.
NB: I used the GFile & gs APIs to mesh with the surrounding code.
Closes: #290
Approved by: cgwalters
This comes from the tradition of yum repo files. It's significantly
easier for rpm-ostree users building multiple architectures if some
core templating for the treefiles is built in.
Otherwise, everyone needs to learn about a new wrapper tool for
rpm-ostree, and said tool would need to re-do the same "basearch"
evaluation that is already occuring inside libhif.
This commit also paves the way for introducing `${releasever}`
substitution.
NOTE: This depends on pending changes to libhif git.
Closes: #274
Approved by: jlebon
If one does `rpm-ostree foo --bar`, one would expect to see
`Unknown subcommand foo`, not `Unknown option --bar`.
Closes: #267Closes: #270
Approved by: jlebon
awk is already used by the tap driver, so it won't be an additional
dependency.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #263
Approved by: cgwalters
I was tracking another regression where we seem to have lost
`/usr/etc` contents which manifested as `Labeling with... (null)`
which was clearly wrong.
Now this change actually impacts the test suite - we now (again IMO
correctly) error out if `selinux: true`. The `no-selinux-tag` test
no longer makes sense, so delete it.
We do need more "real" tests that use selinux on and off.
Pull request: #243
Approved by: jlebon
When running the tests by hand, I wanted to be able to source
`libtest.sh` from a running shell. Make this slightly easier by
allowing one to set `SRCDIR`, since `$0` won't work.
The libtest.sh script is aware of whether a tmpdir needs to be created
or not for the test. Make use of this in setup-session.sh so that we're
sure we're in the right directory before creating the sysroot dir.
Running `make check` would fail because test-ucontainer.sh uses
test-repo, which points directly to the installed_testdir, where it is
not installed yet.
We make another processed version of test-repo.repo.in which instead
points to the repo in the srcdir and make use of that in
test-ucontainer.sh.
There are probably other ways of solving this, but this is the one that
jumped out at me.
For the same reasons as described in GNOME/ostree#187. In summary: we
want to make it easy for testers to clean up after we're done by not
setting the immutable flag.
Note that I had to also add it to setup-session.sh so that the daemon
inherited the env var. The libtest.sh hunk is redundant in that case,
but still necessary if the tests are run directly.
The assert[_not]_file_has_content functions used grep to check the
pattern against the content. This meant that the '.' characters are
interpreted as "any char". Yesterday, the date was 20150910 and thus the
otheros' tree's version was labelled as such. This date also happens to
match the 1.0.10 pattern, and thus caused the test to fail.
This patch makes sure this doesn't happen again by escaping all the dots
to make them literal.
Unfortunately RHEL 7 has an older version of dbus, and I use it as a
workstation. It's not a lot of code and only used for tests. We can
make it build time conditional down the line or something.
This is a *third* implementation of rpm database diffs in the code,
but it is now a public introspectable shared library API.
Further commits will change the command line tools to use this, and
then after that we'll further deduplicate the `db diff` from this
code.
The `QueryResult` class ended up being too awkward; having NEVRA
strings meant for example that clients would have to parse them. It
would be harder to present something like the current `rpm-ostree
upgrade` package diff output.
Now...I debated quite a while before doing this patch. The thing
that's really awful about creating this library is there are *SO MANY*
layers. rpm-ostree → libhif → hawkey → libsolv → librpm. It's enough
to make one question whether one is actually accomplishing anything or
just contributing to a collective insanity...
Let's pretend for now it's the former.
Closes: https://github.com/projectatomic/rpm-ostree/pull/136
This will help build release engineering and other types of tools;
for example, rather than parsing the output of `db diff`, one
should be able to call an API.
Initially, this adds the generic infrastructure for a public shared
library, with a new function call to do the equivalent of `rpm -qa` on
a particular OSTree commit.
Closes: https://github.com/projectatomic/rpm-ostree/issues/117
Closes: https://github.com/projectatomic/rpm-ostree/pull/124
A local yum repository, which contains a dummy .rpm package adding
support for programs needed by the post-process phase, is created as
part of the test.
Since no libc is present, the empty.c program (which is a no-op)
directly invokes the interrupt 0x80 to exit.
The test works only on x86_64, it is skipped on other architectures.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
