Commit Graph

1274 Commits

Author SHA1 Message Date
Jonathan Lebon
9d9042d835 vmcheck: prep for parallel runs
1. Don't require an ssh-config

In the case of redhat-ci, the VMs are already fully configured for the
system (injected in the hosts file, host key accepted, etc...). So
there's no need to have an ssh-config there. In general, it should be
acceptable to run the vmcheck suite against a resolvable host without
having to create an ssh-config for it.

2. Make the host name configurable

Rather than hardcoding "vmcheck" as the hostname, allow overridding it
by specifying a VM env var directly. We also prepare the various scripts
to make use of the $VM variable whenever host-specific dirs/files are
created so that parallel runs won't step on each other.

Closes: #675
Approved by: cgwalters
2017-03-13 00:42:13 +00:00
Jonathan Lebon
058d7e356e bump libdnf for chksum fix
Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00
Jonathan Lebon
fd64084b9b vmcheck: add basic test for local RPM installs
Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00
Jonathan Lebon
81c43e81fb PkgChange: add support for layering local RPMs
Allow users to directly specify an RPM file on the command-line. The
"packages_added" array of the PkgChange() method can now contain
absolute paths to RPM files.

Grow the origin format to have a new "requested-local" key. This is
similar to the "requested" key, except that the packages are always
installed from cache. The "requested-local" array values also embed the
SHA-256 of the header we expect.

There is now a new "LocalPackages" line in the status. These packages
are a subset of the "packages" element (which are printed as
"LayeredPackages") and represent the packages that are explicitly marked
for installing from cache.

Interesting design choices/notes:
  - Just as before, even with foo-1.0-1.x86_64 installed from RPM, a
    user can still request "/usr/bin/foo": it will be made dormant. As
    soon as foo stops being explicitly layered from the RPM, it will try
    to fulfill the request by going to the repos. This allows users to
    "pin" a layered package to a certain RPM, and then unpin it.
  - The strings/NEVRAs in "requested" and "requested-local" are strictly
     distinct. This allows us to be able to tell what the user means
     exactly when they do "rpm-ostree uninstall".

Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00
Jonathan Lebon
5c5ebc1b0a core: add support for installing from cache
We start by adding support in the core for installing packages strictly
from the cache repo. We fool the libdnf stack by re-exporting the header
as an RPM, and explicitly marking it for install. The treefile format
supports specifying the expected SHA-256 of the metadata header, in case
the cache for a specific NEVRA changed.

Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00
Colin Walters
e86dc13c49 bwrap: Don't use --unshare-net in nspawn by default
This will fix rpm-ostree-in-mock-in-koji. The drawback is minor: post scripts
will have network access. But we're going to be testing the no-network case in
our Docker-based builds, so that's fine.

Closes: #672
Approved by: jlebon
2017-03-10 17:27:56 +00:00
ankit01ojha
98a78628c4 Add issue template, move PR template to .github
Closes: https://github.com/projectatomic/rpm-ostree/issues/673

Closes: #674
Approved by: cgwalters
2017-03-10 16:31:27 +00:00
Jonathan Lebon
49cbdb739a status: always include the packages entries
Pull #646 introduced a subtle regression: we went from always including
a "packages" entry to only including it if there are packages present.
Albeit it's easy to guard against, though to be nice, let's make it
easier for consumers by always including it.

Reported-by: Micah Abbott <miabbott@redhat.com>

Closes: #670
Approved by: cgwalters
2017-03-10 14:54:45 +00:00
Colin Walters
14a9135ea0 main: Output YAML for --version
Equivalent of https://github.com/ostreedev/ostree/pull/691
Basically, it's nicer to parse.

Closes: #665
Approved by: jlebon
2017-03-09 14:23:34 +00:00
Chen Fan
afbf0de286 spec: Update from fedora dist-git
We'll keep a copy here for now, though the canonical
version should be viewed as the Fedora dist-git.

Updated-by: Colin Walters <walters@verbum.org>

Closes: #667
Approved by: jlebon
2017-03-08 19:52:58 +00:00
Colin Walters
7cf366497f daemon: Log txn initiation
Down the line we should have a saner stringification, and also
log things like the invoking `loginuid` etc.

But this gets us something.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
fd6b30746e daemon: Move mainctx iteration out of main() into object
Preparation for idle exit - the daemon knows when it should idle exit. It's
tempting to move all of the mainloop logic there, but it'd be more code churn
than I want to do right now.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
927d9d061a daemon: Track transaction status, render it in systemd status
This is part of the idle exit work - the higher level daemon logic needs to
track running transaction state, so we can combine that with whether we have `> 0` clients.

(Yes, we need a better stringification of txns)

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
33e9f2a4cd daemon: use sd_notify() when we receieve a signal
Just to keep the status display up to date.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
d3772ffc4f daemon: Use sd_notify(STATUS=) to print how many clients are active
Now `systemctl status rpm-ostreed` will show it. In future patches I plan to add
whether we have an active txn and our pending idle exit timeout.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
98c1f43326 daemon: Add RegisterClient API
This is like what bluez does. With this, we have a stronger mechanism to avoid
races with future work to auto-exit on idle.  Registered clients
hold a reference to the daemon effectively.

Note that calling `UnregisterClient` is optional if the calling process is going
to exit soon - as is the case for using the command line binary via e.g.
`rpm-ostree status`.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
d4789e0886 daemon: use systemd notify(STOPPING=1) + ReleaseName combo
More info here:

 - https://lists.freedesktop.org/archives/dbus/2015-May/016671.html
 - https://github.com/cgwalters/test-exit-on-idle

This is preparatory work for doing exit-on-idle, which we want to do because we
currently leak memory badly. Even if we didn't leak, it's still better to not
have services sit around resident.

While I had the patient open, I did some code cleanup such as switching to
direct `g_main_context_iteration()` and using `sd_journal_print()` instead of
`g_warning()` etc.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
fb8f857967 ci: Sync Dockerfile with ostree
Down the line we should share these.  Currently this is
required for
https://github.com/projectatomic/rpm-ostree/pull/665

Closes: #668
Approved by: jlebon
2017-03-07 15:01:26 +00:00
Colin Walters
7b8e05ffc2 Allow and start using C99 declaration-after-statement
The equivalent of https://github.com/ostreedev/ostree/pull/718
but for this codebase.

I just picked one example at random, there's plenty of others, but I don't want
to do any kind of tree-wide conversion since we have lots of outstanding
patches.

Closes: #664
Approved by: jlebon
2017-03-06 23:16:30 +00:00
Jonathan Lebon
5dc69a554e .redhat-ci.yml: increase vmcheck timeout
https://github.com/projectatomic/rpm-ostree/pull/656 added some more
tests to vmcheck that made it more likely to run overtime.

Closes: #661
Approved by: jlebon
2017-03-06 22:19:10 +00:00
Colin Walters
79ee224709 core: Log commit stats into journal
It's useful to see how many new objects were added and how much storage they
use. Just part of a general trend where I'd like to make a lot more extensive
use of the journal, particularly with structured logging.

Closes: #661
Approved by: jlebon
2017-03-06 22:19:10 +00:00
Colin Walters
e1e2bf9fbe compose-tree: Print nsolvables, consolidate disabled repos
This should help us if we ever find corrupt repodata; I thought
that might be happening in https://pagure.io/releng/issue/6602
Also, consolidate the disabled repos printing, since I think
we don't need to be verbose about those.

Closes: #613
Approved by: jlebon
2017-03-06 21:21:10 +00:00
Colin Walters
6003779450 Release 2017.3
This release includes a fix for [CVE-2017-2623](https://bugzilla.redhat.com/show_bug.cgi?id=1422157).

There are a few new features, such as `systemctl reload rpm-ostreed` now being
supported. Some bugfixes such as memory leak fixes. Besides that, there's a lot
of internal refactoring going on in preparation for work on local RPM
installation.

Closes: #663
Approved by: jlebon
2017-03-06 17:13:10 +00:00
Colin Walters
f9944e6d96 daemon: Remove several unused instance variables
These were leftovers from the very original code which did support exit-on-idle;
we're going to do it differently now.

Closes: #659
Approved by: jlebon
2017-03-06 16:14:38 +00:00
Colin Walters
a00b9f94fb daemon: Tell systemd when we're shutting down
This won't actually help with much right now; just splitting out this change to
highlight it as part of future work for exit-on-idle.

Closes: #659
Approved by: jlebon
2017-03-06 16:14:38 +00:00
Colin Walters
66425c3161 core: Do GPG verification before importing
While reading a recent conversation about GPG checking at treecompose
time, I had a sudden thought - were we actually doing verification
client side?  Turned out, we aren't.  That happens as part of
`dnf_transaction_commit()` which we don't use.

That function verifies every package at one go, but for us I think it's better
to do it before "importing". We shouldn't have untrusted bits that we've
unpacked (they might have suid binaries, for one thing).

This is an embarassing problem, but it's worth emphasizing that everyone should
be retrieving repodata at a minimum over TLS, which sets a baseline. On RHEL, we
already do pinned TLS, and there are discussions about extending that elsewhere.

See: https://bugzilla.redhat.com/show_bug.cgi?id=1422157

Closes: #656
Approved by: jlebon
2017-03-06 15:19:43 +00:00
Colin Walters
127275fe0e build: Depend on ostree 2017.2, drop check for gpg symbol
Since somehow it's breaking in [Fedora's Koji](https://kojipkgs.fedoraproject.org//packages/rpm-ostree/2017.2/2.fc25/data/logs/x86_64/build.log)
and I don't feel like debugging it.  Let's just take a hard dependency
now and kill the conditional.

Closes: https://github.com/projectatomic/rpm-ostree/issues/653

Closes: #655
Approved by: jlebon
2017-03-03 14:34:57 +00:00
Colin Walters
3568ed9a83 core: Plug leak of pkgcache repo
This showed up as a number of duplicate fds to the pkgcache root in
`/proc/<pid>/fd` for the daemon.

Closes: #654
Approved by: jlebon
2017-03-03 13:38:40 +00:00
Jonathan Lebon
89693c0385 core: no longer embed treespec
In the case of client layering, we hackily use the treespec because
that's what the core understands (for now), but it really shouldn't be
part of the final commit, nor should we rely on it.

This patch starts the path towards moving us away from the treespec by
not embedding it in client layers, and instead directly inserting
layered packages under the "rpmostree.packages" key.

The SERVER_BASE case still embeds the treespec, since only the container
path uses it for now and it needs it.

Down the line, we'll want to make the treespec just one of the methods
by which we initialize RpmOstreeContext. But nothing stops us from
hiding that detail already.

Closes: #650
Approved by: cgwalters
2017-03-01 17:42:42 +00:00
Colin Walters
22048b25a7 scripts: Use tmpfs for /var/tmp, not the host's /tmp
We don't want to expose the host's `/tmp` since that means scripts could
potentially find things like the X11 socket or whatever.

To debug things better, add a quick bash script to run bwrap like the C code
does. Perhaps down the line we can add `rpm-ostree internals run-bwrap` or so.

Closes: #647
Approved by: jlebon
2017-02-28 23:37:15 +00:00
Jonathan Lebon
b364357f60 origin: fix indentation
Closes: #651
Approved by: cgwalters
2017-02-28 22:45:07 +00:00
Jonathan Lebon
7fd474f8e1 vmcheck: adjust for new behaviour
Closes: #646
Approved by: cgwalters
2017-02-27 19:53:19 +00:00
Jonathan Lebon
051492b3e6 upgrader: switch to stateless model
This is the culmination of the work paved by the previous commits. We no
longer handle package set mutations in the upgrader, but rather let
clients modify the origin as needed for that. As such, the upgrader is
now completely memoryless. We no longer try to error out if a package is
already in the base and we no longer drop out packages previously
layered packages that are now in the base. *Which* packages actually get
layered is determined during deployment and not coded in the origin.

At an API level, the deployment variant returned (and thus what gets
printed by `status --json`) now has both "packages" and
"requested-packages". The former retains its original semantic: it's the
set of packages that are *actually* layered. The latter contains
everything from the origin, which includes the former set + dormant
packages.

Accordingly, the output of `rpm-ostree status` also now distinguishes
between "RequestedPackages" and "LayeredPackages". Extra: we now quote
provides that contain spaces.

Closes: #646
Approved by: cgwalters
2017-02-27 19:53:19 +00:00
Jonathan Lebon
6a5d4a2699 RpmOstreeOrigin: also cache initramfs args
Ideally, we want to parse out everything in the origin on init and then
cache the values to make it even easier to inspect it. We make this more
explicit by renaming the struct members to "cache_*" and also caching
the initramfs args. This doesn't actually save any allocations right
now (though it might in the future), but we do the same for everything
else, and it's nice to be consistent.

Closes: #646
Approved by: cgwalters
2017-02-27 19:53:19 +00:00
Jonathan Lebon
d0c8dddf99 RpmOstreeOrigin: remove is_locally_assembled()
There are multiple reasons for this. The first is that with the
introduction of "dormant" packages, it becomes impossible from simply
looking at a deployment's origin whether it was layered or not. We
*could* make the upgrader turn around and insert back the "real" final
packages that were layered. But we're really trying to reinforce the
concept that the origin is mutable and represents an input for the
upgrader. Derived data shouldn't live there.

Whether a deployment is actually layered or not instead is determined by
the deployment commit itself. The embedded treespec records the exact
packages that were layered, which we will later use to enhance status
output.

Closes: #646
Approved by: cgwalters
2017-02-27 19:53:19 +00:00
Colin Walters
b034381506 daemon: Implement "reload"
There are two main issues right now; first, we don't pick up manual changes to
`.origin` files, which occurs when one needs to sed it to remove `unconfigured`
for example. Second, we need to reload changes to the remotes.

Closes: #598
Approved by: jlebon
2017-02-24 21:36:45 +00:00
Colin Walters
229fdfa7f9 scripts: Bind rather than symlink /usr/etc → /etc
This fixes resolution of relative symlinks, which fixes installation
of `firewalld` in Fedora 25.

Closes: https://github.com/projectatomic/rpm-ostree/issues/638

Closes: #640
Approved by: jlebon
2017-02-24 20:43:05 +00:00
Jonathan Lebon
a3ee87d67d vmcheck/test.sh: output reboot details to log
Clean up the output of vmcheck a bit by printing out ssh debug details
and rpm-ostree status output to the log file. This will help make it
easier to quickly see which tests failed (as well as helping satisfy my
OCD for a nice output).

Closes: #645
Approved by: cgwalters
2017-02-24 19:50:25 +00:00
Jonathan Lebon
88548f2e1a libvm: don't call rpm-ostree status on first time
On the first call to vmsync/vmoverlay, we do an immediate vm_ssh_wait()
to check if we have a live VM. However, we don't necessarily have a
working rpm-ostree in there if we're hacking on stuff. It gets annoying
to wait for the timeout there.

Let's just work around that by instead only calling status if we got
there through vm_reboot_cmd(), which is most likely when we're most
interested in the status output anyway.

Closes: #645
Approved by: cgwalters
2017-02-24 19:50:25 +00:00
Colin Walters
0f795cae3f daemon: Squash a memleak of parent commit
Closes: #644
Approved by: jlebon
2017-02-24 18:55:36 +00:00
Colin Walters
cc20f32fb7 tree-wide: Use autoptr cleanup for HyQuery
This was a noticeable source of leaks.

Closes: #644
Approved by: jlebon
2017-02-24 18:55:36 +00:00
Jonathan Lebon
c0b4fbc0f5 util: drop custom set_prefix_error_from_errno
We can use libglnx functions for this now.

Closes: #643
Approved by: cgwalters
2017-02-23 20:57:51 +00:00
Colin Walters
f573354324 ci: Hotfix ostree temporarily
For https://github.com/ostreedev/ostree/pull/705

Closes: #642
Approved by: jlebon
2017-02-23 20:03:06 +00:00
Jonathan Lebon
b7613dcc81 RpmOstreeOrigin: don't error on unconfigured state
This is a follow-up to 775c781 (#626). Really, RpmOstreeOrigin shouldn't
concern itself with whether the origin is unconfigured. Its main goal
should be to parse it out and make it easy for users to modify it. That
sort of business logic lives in the upgrader, which *should* be
concerned if the origin in unconfigured.

Closes: #634
Approved by: cgwalters
2017-02-21 21:12:09 +00:00
Jonathan Lebon
62c7242a98 upgrader: move origin mutation outside of the upgrader
This is the beginning of a move towards making the upgrader more
stateless and less hysteretic. Ideally, the only input required should
be an origin file. Users of the upgrader are in charge of modifying the
origin as needed. The main goal of this is to simplify the upgrader
model and a better separation of concerns.

This was already done for the initramfs-related entries. This patch
formalises this for most of the other inputs, except packages, which
will be done soon.

Closes: #634
Approved by: cgwalters
2017-02-21 21:12:09 +00:00
Jonathan Lebon
c437f90a83 upgrader: drop support for ignore scripts
This functionality is disabled right now, and if we want to re-enable
it, we'll want to do it in a way that ensures the ignore scripts are
permanent.

Closes: #634
Approved by: cgwalters
2017-02-21 21:12:09 +00:00
Jonathan Lebon
3858913c98 package_diff: make use of apply_revision_override
Dedup some more code. Might be worth it to just completely unify
package_diff with deploy at some point.

Closes: #634
Approved by: cgwalters
2017-02-21 21:12:09 +00:00
Colin Walters
84f5b21a2e ci: Install ostree too
Since this is the second case I've seen tripping over this, maybe
we should add an rpm-ostree → ostree dependency?

Closes: #637
Approved by: jlebon
2017-02-21 20:38:24 +00:00
Chen Fan
7ee41f75b7 remove duplicate words in comment
Signed-off-by: Chen Fan <fan.chen@easystack.cn>

Closes: #635
Approved by: cgwalters
2017-02-21 14:47:59 +00:00
Jonathan Lebon
8eb746f533 test-initramfs.sh: fix for centos
Fetch the osname rather than hardcoding it.

Closes: #636
Approved by: cgwalters
2017-02-21 14:13:42 +00:00