Commit Graph

1347 Commits

Author SHA1 Message Date
Colin Walters
a05e256177 compose: Delete /usr/etc/passwd- (and the other variants)
There's no point to shipping these backup files in the base tree. We already had
code to delete them for the package layering case where they caused active harm.
At the point we added that code we really should have *also* changed treecompose
to delete them. Better late than never.

The reason I'm doing this now is because having them in the base tree causes `ex
livefs` to spuriously think that layering a package that *doesn't* change `/etc`
as if it does, because the layering code deletes the backup files.

Closes: #693
Approved by: jlebon
2017-03-20 16:35:17 +00:00
Colin Walters
70176029a0 libvm: Fix typo in "using ssh-config" case
This patch clearly underscores the issue.

Closes: #691
Approved by: mike-nguyen
2017-03-17 19:09:21 +00:00
Jonathan Lebon
bde8456da6 compose: consistent error messages across subcommands
Make `rpm-ostree compose bad-command` and `rpm-ostree ex bad-command`
have the same error message formatting.

Closes: #688
Approved by: cgwalters
2017-03-16 21:39:33 +00:00
Jonathan Lebon
152e7bf1a6 CLI: use proper command name in experimental notice
This is a follow-up to #683. All the experimental commands will be under
`rpm-ostree ex`, so we remove the flag and instead print the notice
before dispatching the subcommand, where we can print the correct name.

Closes: #688
Approved by: cgwalters
2017-03-16 21:39:33 +00:00
Colin Walters
710ea30612 main: Move "rpm-ostree container" → "rpm-ostree ex container"
Now all of our "experimental" bits are under one roof, so we get consistent
handling for them.

Closes: #683
Approved by: jlebon
2017-03-16 19:57:14 +00:00
Colin Walters
8dfed46e26 app: Rename internals → ex
The goal is to consolidate our "experimental" functionality under one
subcommand.  This makes it easier to determine when things "graduate"
to permanent-stability status under the main command line.

Closes: https://github.com/projectatomic/rpm-ostree/issues/682

Closes: #683
Approved by: jlebon
2017-03-16 19:57:14 +00:00
Colin Walters
7226b4135a app: Make hidden/experimental flags for command parsing
I think this ends up being cleaner - the properties of
a command are now consistently represented as flags.

Prep work for https://github.com/projectatomic/rpm-ostree/issues/682

Closes: #683
Approved by: jlebon
2017-03-16 19:57:14 +00:00
Colin Walters
7453c955df main: Remove unused "preview" commands
Closes: #683
Approved by: jlebon
2017-03-16 19:57:14 +00:00
Colin Walters
12c34bb249 Move flags into command struct, pass down through builtins
This is prep work for https://github.com/projectatomic/rpm-ostree/issues/682

Closes: #683
Approved by: jlebon
2017-03-16 19:57:14 +00:00
Colin Walters
62a93c7850 postprocess: Handle f26 /etc/nsswitch.conf configuration
F26 put sss first, which broke our regexp. When we switch to sysusers, man it'll
be nice to dump ♲ this.

Closes: https://github.com/projectatomic/rpm-ostree/issues/685

Closes: #686
Approved by: jlebon
2017-03-16 18:39:38 +00:00
Colin Walters
cc23957a2c tests: Stop using "ostree trivial-httpd"
I built+installed ostree git master with
https://github.com/ostreedev/ostree/pull/723 in my dev container,
which broke the rpm-ostree tests.  Kind of embarrassing that
I forgot to check rpm-ostree for usage of `ostree trivial-httpd`.

This is another thing that really wants a shared test container.

Anyways let's just use Python for this.

Closes: #684
Approved by: jlebon
2017-03-16 13:55:38 +00:00
Colin Walters
fc9396372f app: Dedup code for subcommand parsing
We had lots of copies of this code, and while I'm not planning on changing it, I
decided to dedup on general principle.

Closes: #681
Approved by: jlebon
2017-03-15 21:11:42 +00:00
Colin Walters
7e4f56a862 app: Switch subcommands to use "base" RpmOstreeCommand struct
They were just duplicates.  Prepartory cleanup for future patches.

Closes: #681
Approved by: jlebon
2017-03-15 21:11:42 +00:00
Colin Walters
24faf942df compose: Add --add-metadata-from-json
I'd like to embed structured metadata about the originating git
repository.  See [this example](https://pagure.io/fedora-atomic-host-continuous/c/142b12020d7efe18b56d039304efea102a210790?branch=master).  However, I think what we really
want here is a *single* value which has subkeys.

One thing in the back of my mind too is...we could use this to
enhance our "change detection".  Right now we checksum the sack,
treefile, and treecompose-post.  But down the line, I'd
like to support more sophisticated postprocessing, where the
script might reference external files or the like.

In that case, we could stop checksumming the post script, and rely on whether or
not the git repo changed. (This would conversely mean we would do a build even
if e.g. the repo's `README.md` changed, but we can address that with a
post-assemble content check).

Anyways though, for now, this gets us the ability to more easily drop more
structured metadata in the commit, whether it's input git repos, tests that
passed, etc.

Note a trap that bit me here: since the metadata we write here is *host* endian,
but `ostree show --raw` byteswaps (it needs to since the core ostree variant
is always big endian), we get inverted numbers if the host is little.

I think we should probably canonicalize our metadata to big endian; this should
be pretty backwards compatible since I doubt anyone has been adding raw numbers
so far.

Closes: #676
Approved by: jlebon
2017-03-14 18:17:07 +00:00
Colin Walters
479667807c compose: Canonicalize user-provided metadata to big endian
Should have no effect right now, since there's actually
no way we expose at the moment to write anything other
than strings.

But it will be useful when we add a way to write arbitrary metadata; see
https://github.com/projectatomic/rpm-ostree/pull/676

Closes: #679
Approved by: jlebon
2017-03-14 15:27:47 +00:00
Colin Walters
5142b8b4c2 compose-tree: Clean up metadata parsing
Preparatory work for a future patch; by having a hash it's easier to handle
duplicates. Note I started trying to use decl-after-stmt but hit the fact we're
still using `goto out` here.

Closes: #679
Approved by: jlebon
2017-03-14 15:27:47 +00:00
Colin Walters
05a59e5af5 core: Don't try to chmod() symlinks
I just noticed this while reading the code. Right now we're actually following
links (since `AT_SYMLINK_NOFOLLOW` isn't implemented), which could definitely
lead to bad things if e.g. one has a symlink pointing to a file which shouldn't
be executable or world-readable etc.

(Also I noticed our hardlink-breaking-only-if-regular logic is now broken;
 I added a TODO)

Closes: #677
Approved by: jlebon
2017-03-13 17:53:51 +00:00
Colin Walters
77fa10cfbe core: Skip (and warn on) rpmfi overrides outside of /{usr,etc,var}
I still need to query whether Fedora has anything like this; maybe `/boot`? But
things are not likely to work right now if we do, so let's ignore it.

Closes: #677
Approved by: jlebon
2017-03-13 17:53:50 +00:00
Colin Walters
ab7431c458 core: Apply rpmfi overrides for /etc to /usr/etc
This fixes installing `mock`, which for some reason has files in `/etc/mock`
owned by `root:mock`.

Closes: https://github.com/projectatomic/rpm-ostree/issues/671

Closes: #677
Approved by: jlebon
2017-03-13 17:53:50 +00:00
Colin Walters
d8b96a8531 core: Include filename in error messages for rpmfi overrides
This gives me now:

```
error: While applying overrides for pkg mock: fstatat: etc/mock/custom-1-aarch64.cfg: No such file or directory
```

Which is definitely more useful for debugging 💣.

Closes: #677
Approved by: jlebon
2017-03-13 17:53:50 +00:00
Colin Walters
62320b1920 rebase: Update man page for syntax, print if just changing remote
The rationale for just printing if changing remote is I think
it's really only that version that feels "magical".

Closes: https://github.com/projectatomic/rpm-ostree/issues/569

Closes: #678
Approved by: jlebon
2017-03-13 14:45:30 +00:00
Jonathan Lebon
c65ac2955c redhat-ci: provision more nodes for vmcheck
Make use of the new support for parallel execution. In the best case,
this should bring down the execution time to about a third.

Closes: #675
Approved by: cgwalters
2017-03-13 00:42:13 +00:00
Jonathan Lebon
29840bcecb vmcheck: add support for parallel runs
Allow the `make vmcheck` target to take a HOSTS var, which is simply a
space-separated list of hosts on which we can run testsuites. Add a
multitest.py script that takes care of monitoring and scheduling the
tests onto the nodes.

The script itself is "dumb": we don't know how long each test can take,
so we can't do any smart/heuristic scheduling that could save more time.

Closes: #675
Approved by: cgwalters
2017-03-13 00:42:13 +00:00
Jonathan Lebon
9d9042d835 vmcheck: prep for parallel runs
1. Don't require an ssh-config

In the case of redhat-ci, the VMs are already fully configured for the
system (injected in the hosts file, host key accepted, etc...). So
there's no need to have an ssh-config there. In general, it should be
acceptable to run the vmcheck suite against a resolvable host without
having to create an ssh-config for it.

2. Make the host name configurable

Rather than hardcoding "vmcheck" as the hostname, allow overridding it
by specifying a VM env var directly. We also prepare the various scripts
to make use of the $VM variable whenever host-specific dirs/files are
created so that parallel runs won't step on each other.

Closes: #675
Approved by: cgwalters
2017-03-13 00:42:13 +00:00
Jonathan Lebon
058d7e356e bump libdnf for chksum fix
Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00
Jonathan Lebon
fd64084b9b vmcheck: add basic test for local RPM installs
Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00
Jonathan Lebon
81c43e81fb PkgChange: add support for layering local RPMs
Allow users to directly specify an RPM file on the command-line. The
"packages_added" array of the PkgChange() method can now contain
absolute paths to RPM files.

Grow the origin format to have a new "requested-local" key. This is
similar to the "requested" key, except that the packages are always
installed from cache. The "requested-local" array values also embed the
SHA-256 of the header we expect.

There is now a new "LocalPackages" line in the status. These packages
are a subset of the "packages" element (which are printed as
"LayeredPackages") and represent the packages that are explicitly marked
for installing from cache.

Interesting design choices/notes:
  - Just as before, even with foo-1.0-1.x86_64 installed from RPM, a
    user can still request "/usr/bin/foo": it will be made dormant. As
    soon as foo stops being explicitly layered from the RPM, it will try
    to fulfill the request by going to the repos. This allows users to
    "pin" a layered package to a certain RPM, and then unpin it.
  - The strings/NEVRAs in "requested" and "requested-local" are strictly
     distinct. This allows us to be able to tell what the user means
     exactly when they do "rpm-ostree uninstall".

Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00
Jonathan Lebon
5c5ebc1b0a core: add support for installing from cache
We start by adding support in the core for installing packages strictly
from the cache repo. We fool the libdnf stack by re-exporting the header
as an RPM, and explicitly marking it for install. The treefile format
supports specifying the expected SHA-256 of the metadata header, in case
the cache for a specific NEVRA changed.

Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00
Colin Walters
e86dc13c49 bwrap: Don't use --unshare-net in nspawn by default
This will fix rpm-ostree-in-mock-in-koji. The drawback is minor: post scripts
will have network access. But we're going to be testing the no-network case in
our Docker-based builds, so that's fine.

Closes: #672
Approved by: jlebon
2017-03-10 17:27:56 +00:00
ankit01ojha
98a78628c4 Add issue template, move PR template to .github
Closes: https://github.com/projectatomic/rpm-ostree/issues/673

Closes: #674
Approved by: cgwalters
2017-03-10 16:31:27 +00:00
Jonathan Lebon
49cbdb739a status: always include the packages entries
Pull #646 introduced a subtle regression: we went from always including
a "packages" entry to only including it if there are packages present.
Albeit it's easy to guard against, though to be nice, let's make it
easier for consumers by always including it.

Reported-by: Micah Abbott <miabbott@redhat.com>

Closes: #670
Approved by: cgwalters
2017-03-10 14:54:45 +00:00
Colin Walters
14a9135ea0 main: Output YAML for --version
Equivalent of https://github.com/ostreedev/ostree/pull/691
Basically, it's nicer to parse.

Closes: #665
Approved by: jlebon
2017-03-09 14:23:34 +00:00
Chen Fan
afbf0de286 spec: Update from fedora dist-git
We'll keep a copy here for now, though the canonical
version should be viewed as the Fedora dist-git.

Updated-by: Colin Walters <walters@verbum.org>

Closes: #667
Approved by: jlebon
2017-03-08 19:52:58 +00:00
Colin Walters
7cf366497f daemon: Log txn initiation
Down the line we should have a saner stringification, and also
log things like the invoking `loginuid` etc.

But this gets us something.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
fd6b30746e daemon: Move mainctx iteration out of main() into object
Preparation for idle exit - the daemon knows when it should idle exit. It's
tempting to move all of the mainloop logic there, but it'd be more code churn
than I want to do right now.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
927d9d061a daemon: Track transaction status, render it in systemd status
This is part of the idle exit work - the higher level daemon logic needs to
track running transaction state, so we can combine that with whether we have `> 0` clients.

(Yes, we need a better stringification of txns)

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
33e9f2a4cd daemon: use sd_notify() when we receieve a signal
Just to keep the status display up to date.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
d3772ffc4f daemon: Use sd_notify(STATUS=) to print how many clients are active
Now `systemctl status rpm-ostreed` will show it. In future patches I plan to add
whether we have an active txn and our pending idle exit timeout.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
98c1f43326 daemon: Add RegisterClient API
This is like what bluez does. With this, we have a stronger mechanism to avoid
races with future work to auto-exit on idle.  Registered clients
hold a reference to the daemon effectively.

Note that calling `UnregisterClient` is optional if the calling process is going
to exit soon - as is the case for using the command line binary via e.g.
`rpm-ostree status`.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
d4789e0886 daemon: use systemd notify(STOPPING=1) + ReleaseName combo
More info here:

 - https://lists.freedesktop.org/archives/dbus/2015-May/016671.html
 - https://github.com/cgwalters/test-exit-on-idle

This is preparatory work for doing exit-on-idle, which we want to do because we
currently leak memory badly. Even if we didn't leak, it's still better to not
have services sit around resident.

While I had the patient open, I did some code cleanup such as switching to
direct `g_main_context_iteration()` and using `sd_journal_print()` instead of
`g_warning()` etc.

Closes: #660
Approved by: jlebon
2017-03-07 22:33:58 +00:00
Colin Walters
fb8f857967 ci: Sync Dockerfile with ostree
Down the line we should share these.  Currently this is
required for
https://github.com/projectatomic/rpm-ostree/pull/665

Closes: #668
Approved by: jlebon
2017-03-07 15:01:26 +00:00
Colin Walters
7b8e05ffc2 Allow and start using C99 declaration-after-statement
The equivalent of https://github.com/ostreedev/ostree/pull/718
but for this codebase.

I just picked one example at random, there's plenty of others, but I don't want
to do any kind of tree-wide conversion since we have lots of outstanding
patches.

Closes: #664
Approved by: jlebon
2017-03-06 23:16:30 +00:00
Jonathan Lebon
5dc69a554e .redhat-ci.yml: increase vmcheck timeout
https://github.com/projectatomic/rpm-ostree/pull/656 added some more
tests to vmcheck that made it more likely to run overtime.

Closes: #661
Approved by: jlebon
2017-03-06 22:19:10 +00:00
Colin Walters
79ee224709 core: Log commit stats into journal
It's useful to see how many new objects were added and how much storage they
use. Just part of a general trend where I'd like to make a lot more extensive
use of the journal, particularly with structured logging.

Closes: #661
Approved by: jlebon
2017-03-06 22:19:10 +00:00
Colin Walters
e1e2bf9fbe compose-tree: Print nsolvables, consolidate disabled repos
This should help us if we ever find corrupt repodata; I thought
that might be happening in https://pagure.io/releng/issue/6602
Also, consolidate the disabled repos printing, since I think
we don't need to be verbose about those.

Closes: #613
Approved by: jlebon
2017-03-06 21:21:10 +00:00
Colin Walters
6003779450 Release 2017.3
This release includes a fix for [CVE-2017-2623](https://bugzilla.redhat.com/show_bug.cgi?id=1422157).

There are a few new features, such as `systemctl reload rpm-ostreed` now being
supported. Some bugfixes such as memory leak fixes. Besides that, there's a lot
of internal refactoring going on in preparation for work on local RPM
installation.

Closes: #663
Approved by: jlebon
2017-03-06 17:13:10 +00:00
Colin Walters
f9944e6d96 daemon: Remove several unused instance variables
These were leftovers from the very original code which did support exit-on-idle;
we're going to do it differently now.

Closes: #659
Approved by: jlebon
2017-03-06 16:14:38 +00:00
Colin Walters
a00b9f94fb daemon: Tell systemd when we're shutting down
This won't actually help with much right now; just splitting out this change to
highlight it as part of future work for exit-on-idle.

Closes: #659
Approved by: jlebon
2017-03-06 16:14:38 +00:00
Colin Walters
66425c3161 core: Do GPG verification before importing
While reading a recent conversation about GPG checking at treecompose
time, I had a sudden thought - were we actually doing verification
client side?  Turned out, we aren't.  That happens as part of
`dnf_transaction_commit()` which we don't use.

That function verifies every package at one go, but for us I think it's better
to do it before "importing". We shouldn't have untrusted bits that we've
unpacked (they might have suid binaries, for one thing).

This is an embarassing problem, but it's worth emphasizing that everyone should
be retrieving repodata at a minimum over TLS, which sets a baseline. On RHEL, we
already do pinned TLS, and there are discussions about extending that elsewhere.

See: https://bugzilla.redhat.com/show_bug.cgi?id=1422157

Closes: #656
Approved by: jlebon
2017-03-06 15:19:43 +00:00
Colin Walters
127275fe0e build: Depend on ostree 2017.2, drop check for gpg symbol
Since somehow it's breaking in [Fedora's Koji](https://kojipkgs.fedoraproject.org//packages/rpm-ostree/2017.2/2.fc25/data/logs/x86_64/build.log)
and I don't feel like debugging it.  Let's just take a hard dependency
now and kill the conditional.

Closes: https://github.com/projectatomic/rpm-ostree/issues/653

Closes: #655
Approved by: jlebon
2017-03-03 14:34:57 +00:00