Commit Graph

1370 Commits

Author SHA1 Message Date
Colin Walters
eba8e55543 build: Fix Makefile dependency for gdbus-codegen
Also add a comment about enabling autoptr in the future.

Closes: #705
Approved by: jlebon
2017-03-27 17:48:39 +00:00
Colin Walters
e666a5b350 vmcheck: Support VMCHECK_INSTTREE
I need to build ostree from git too.  So now my workflow is:

```
export insttree=/srv/walters/tmp/rootfs
cd ostree
make && make install DESTDIR=${insttree}
cd rpm-ostree
make && make install DESTDIR=${insttree}
env VMCHECK_INSTTREE=${insttree} make vmoverlay
```

Closes: #705
Approved by: jlebon
2017-03-27 17:48:39 +00:00
Colin Walters
fa0af8b317 ci: Build ostree from git temporarily
See https://github.com/ostreedev/ostree/issues/758

Closes: #707
Approved by: jlebon
2017-03-27 16:35:43 +00:00
Jonathan Lebon
2672dc1255 pkg-builtins: factor out sorting function
Rework pkg_change() so that the core functionality is implemented by a
helper function. Also convert it to the new style.

I also renamed the "local-packages" to "install-local-packages".
Contrast with the future values: "install-packages" and
"uninstall-packages".

Closes: #704
Approved by: cgwalters
2017-03-23 21:49:50 +00:00
Jonathan Lebon
071aa93f03 app: convert many builtins to new return style
One interesting note here was that I couldn't use glnx_throw in as many
places as I wanted because EXIT_SUCCESS/EXIT_FAILURE are the exact
opposite of that semantic. It was still convenient to use though as a
short-form for g_set_error().

Closes: #704
Approved by: cgwalters
2017-03-23 21:49:50 +00:00
Jonathan Lebon
d592c1874c D-Bus: enable fd passing for deploy/rebase/upgrade
This is the same change that was done for PkgChange(). It will be
required if we want to support specifying local packages through e.g.
--install switches.

Closes: #704
Approved by: cgwalters
2017-03-23 21:49:50 +00:00
Jonathan Lebon
60fcb729e0 libglnx: bump for glnx_null_throw* wrappers
Closes: #704
Approved by: cgwalters
2017-03-23 21:49:50 +00:00
Colin Walters
3e833659b7 daemon: Add a separate OSExperimental interface
Just like `rpm-ostree ex`, for things like `ex livefs` that have DBus
interfaces, we should segregate these off so that people know they're unstable.
And conversely that they can test for the presence of the method on the main
interface for stability.

I initially tried having the same `RpmostreeOS` object implement both
but couldn't work out how to do that; see https://mail.gnome.org/archives/gtk-app-devel-list/2017-March/msg00161.html

Closes: #701
Approved by: jlebon
2017-03-23 19:24:41 +00:00
Colin Walters
1969b96f91 libpriv/origin: Convert to new code style
This is a small one, but I like the momentum. Also note my use of the `,`
operator here. A little unusual maybe but I think it makes sense.

Closes: #700
Approved by: jlebon
2017-03-22 20:42:38 +00:00
Jonathan Lebon
702ccadb64 core: convert some functions to new style
Closes: #699
Approved by: cgwalters
2017-03-22 17:07:10 +00:00
Colin Walters
d33807437f libglnx: declare TESTS earlier
Otherwise libglnx won't be able to add to it.

Closes: #699
Approved by: cgwalters
2017-03-22 17:07:10 +00:00
Jonathan Lebon
100021aa09 libglnx: bump for new error API
I'd like to use it in the stuff I'm currently hacking on!

Closes: #699
Approved by: cgwalters
2017-03-22 17:07:10 +00:00
Colin Walters
1c421029d5 vmcheck: Drop hack for updating ostree
The version in the current release has the fix.

Closes: #698
Approved by: jlebon
2017-03-22 15:35:46 +00:00
Colin Walters
eaced1be7c vmcheck/overlay: Delete passwd backup files here for now
Until we land <https://github.com/projectatomic/rpm-ostree/pull/693> and it
makes its way into all of our streams.

Necessary for livefs.

Closes: #698
Approved by: jlebon
2017-03-22 15:35:46 +00:00
Colin Walters
8e8c26f9fb vmcheck: Don't throw if test.sh fails early
If test.sh failed even before writing to the log file, opening will fail.

Closes: #698
Approved by: jlebon
2017-03-22 15:35:46 +00:00
Colin Walters
8b6c55ca60 core: Also apply mode overrides to symlinks
I originally thought there was a bug here, but missed we were skipping
symlinks earlier above.  See [previous discussion][1].

Now, I'm not aware right now of something this patch actively fixes, but I think
it makes sense to be consistent in our handling of things here with respect to
symlinks.

1: 29dd1bd801..8158dcfb47 (r95017893)

Closes: #689
Approved by: jlebon
2017-03-21 21:35:37 +00:00
Jonathan Lebon
d2ea6cf1b1 rpm-ostree install: pass local packages by fd
This is a follow-up to commit 81c43e8 (#657). That commit extended the
definition of "packages_added" to also support local RPMs. We revert
that here, and instead open the RPM file ourselves and send the fd over
D-Bus. We add support for a "local-packages" option containing the fd
indices to process.

It's just cleaner and safer overall; the daemon and client might not
even be sharing the same view of the filesystem!

Closes: #696
Approved by: cgwalters
2017-03-21 21:09:34 +00:00
Jonathan Lebon
64b7dfcfb0 dbus: enable fd list passing for PkgChange
Add an annotation so that the generated D-Bus code allows rpm-ostree to
pass fds through messages. Note that this doesn't change the API itself,
only how rpm-ostree uses the GDBus library. Clients should still be able
to invoke PkgChange() as before.

Closes: #696
Approved by: cgwalters
2017-03-21 21:09:34 +00:00
Colin Walters
35436827af status: Only show one of BaseCommit/Commit
Following up on the previous commit regarding timestamps - the checksum of the
derived commit isn't very useful. It's not going to be the same between any two
people's systems. What we should show instead is the rpm-md repository
timestamps.

In the short term though, I think omitting this even without the rpm-md
timestamps is better.  Less visual clutter.

Closes: #697
Approved by: jlebon
2017-03-21 20:33:24 +00:00
Colin Walters
77548d06ee status: Use the timestamp from the base, not layered
Currently we propagate the version field from the base to derived,
but the status UI shows the timestamp from the derived commit.

I found this *extremely* confusing when I was trying to reason about the
up-to-dateness of my system. The version and timestamp should be locked
together so we display them consistently.

When we generated the local commit isn't really useful information; conceptually
we have the base timestamp, plus timestamps for the rpm-md repos. We should
display the latter instead.

(This also adds `base-timestamp` and `base-verison` to the JSON; only
 the former is new information, but the latter helps us be consistent)

Closes: #697
Approved by: jlebon
2017-03-21 20:33:24 +00:00
Jonathan Lebon
df8bd53845 pkg-builtins: remove more code for no scripts
This takes commit c437f90 a bit further. We delete more #ifdef 0 blocks
which can just be revived later on if we need it.

Closes: #695
Approved by: cgwalters
2017-03-21 16:53:56 +00:00
Colin Walters
19b093e4a0 vmcheck: Honor TESTS=
Minor regression from the multitest reimplementation, but it's really handy for
the "debug and fix a test" case.

Closes: #692
Approved by: jlebon
2017-03-20 20:41:30 +00:00
Jonathan Lebon
7233db8a74 multitest.py: don't use os.wait()
We can't use os.wait(), since it will conflict with the subprocess'
module's poll() and wait().

Closes: #694
Approved by: cgwalters
2017-03-20 16:59:00 +00:00
Colin Walters
a05e256177 compose: Delete /usr/etc/passwd- (and the other variants)
There's no point to shipping these backup files in the base tree. We already had
code to delete them for the package layering case where they caused active harm.
At the point we added that code we really should have *also* changed treecompose
to delete them. Better late than never.

The reason I'm doing this now is because having them in the base tree causes `ex
livefs` to spuriously think that layering a package that *doesn't* change `/etc`
as if it does, because the layering code deletes the backup files.

Closes: #693
Approved by: jlebon
2017-03-20 16:35:17 +00:00
Colin Walters
70176029a0 libvm: Fix typo in "using ssh-config" case
This patch clearly underscores the issue.

Closes: #691
Approved by: mike-nguyen
2017-03-17 19:09:21 +00:00
Jonathan Lebon
bde8456da6 compose: consistent error messages across subcommands
Make `rpm-ostree compose bad-command` and `rpm-ostree ex bad-command`
have the same error message formatting.

Closes: #688
Approved by: cgwalters
2017-03-16 21:39:33 +00:00
Jonathan Lebon
152e7bf1a6 CLI: use proper command name in experimental notice
This is a follow-up to #683. All the experimental commands will be under
`rpm-ostree ex`, so we remove the flag and instead print the notice
before dispatching the subcommand, where we can print the correct name.

Closes: #688
Approved by: cgwalters
2017-03-16 21:39:33 +00:00
Colin Walters
710ea30612 main: Move "rpm-ostree container" → "rpm-ostree ex container"
Now all of our "experimental" bits are under one roof, so we get consistent
handling for them.

Closes: #683
Approved by: jlebon
2017-03-16 19:57:14 +00:00
Colin Walters
8dfed46e26 app: Rename internals → ex
The goal is to consolidate our "experimental" functionality under one
subcommand.  This makes it easier to determine when things "graduate"
to permanent-stability status under the main command line.

Closes: https://github.com/projectatomic/rpm-ostree/issues/682

Closes: #683
Approved by: jlebon
2017-03-16 19:57:14 +00:00
Colin Walters
7226b4135a app: Make hidden/experimental flags for command parsing
I think this ends up being cleaner - the properties of
a command are now consistently represented as flags.

Prep work for https://github.com/projectatomic/rpm-ostree/issues/682

Closes: #683
Approved by: jlebon
2017-03-16 19:57:14 +00:00
Colin Walters
7453c955df main: Remove unused "preview" commands
Closes: #683
Approved by: jlebon
2017-03-16 19:57:14 +00:00
Colin Walters
12c34bb249 Move flags into command struct, pass down through builtins
This is prep work for https://github.com/projectatomic/rpm-ostree/issues/682

Closes: #683
Approved by: jlebon
2017-03-16 19:57:14 +00:00
Colin Walters
62a93c7850 postprocess: Handle f26 /etc/nsswitch.conf configuration
F26 put sss first, which broke our regexp. When we switch to sysusers, man it'll
be nice to dump ♲ this.

Closes: https://github.com/projectatomic/rpm-ostree/issues/685

Closes: #686
Approved by: jlebon
2017-03-16 18:39:38 +00:00
Colin Walters
cc23957a2c tests: Stop using "ostree trivial-httpd"
I built+installed ostree git master with
https://github.com/ostreedev/ostree/pull/723 in my dev container,
which broke the rpm-ostree tests.  Kind of embarrassing that
I forgot to check rpm-ostree for usage of `ostree trivial-httpd`.

This is another thing that really wants a shared test container.

Anyways let's just use Python for this.

Closes: #684
Approved by: jlebon
2017-03-16 13:55:38 +00:00
Colin Walters
fc9396372f app: Dedup code for subcommand parsing
We had lots of copies of this code, and while I'm not planning on changing it, I
decided to dedup on general principle.

Closes: #681
Approved by: jlebon
2017-03-15 21:11:42 +00:00
Colin Walters
7e4f56a862 app: Switch subcommands to use "base" RpmOstreeCommand struct
They were just duplicates.  Prepartory cleanup for future patches.

Closes: #681
Approved by: jlebon
2017-03-15 21:11:42 +00:00
Colin Walters
24faf942df compose: Add --add-metadata-from-json
I'd like to embed structured metadata about the originating git
repository.  See [this example](https://pagure.io/fedora-atomic-host-continuous/c/142b12020d7efe18b56d039304efea102a210790?branch=master).  However, I think what we really
want here is a *single* value which has subkeys.

One thing in the back of my mind too is...we could use this to
enhance our "change detection".  Right now we checksum the sack,
treefile, and treecompose-post.  But down the line, I'd
like to support more sophisticated postprocessing, where the
script might reference external files or the like.

In that case, we could stop checksumming the post script, and rely on whether or
not the git repo changed. (This would conversely mean we would do a build even
if e.g. the repo's `README.md` changed, but we can address that with a
post-assemble content check).

Anyways though, for now, this gets us the ability to more easily drop more
structured metadata in the commit, whether it's input git repos, tests that
passed, etc.

Note a trap that bit me here: since the metadata we write here is *host* endian,
but `ostree show --raw` byteswaps (it needs to since the core ostree variant
is always big endian), we get inverted numbers if the host is little.

I think we should probably canonicalize our metadata to big endian; this should
be pretty backwards compatible since I doubt anyone has been adding raw numbers
so far.

Closes: #676
Approved by: jlebon
2017-03-14 18:17:07 +00:00
Colin Walters
479667807c compose: Canonicalize user-provided metadata to big endian
Should have no effect right now, since there's actually
no way we expose at the moment to write anything other
than strings.

But it will be useful when we add a way to write arbitrary metadata; see
https://github.com/projectatomic/rpm-ostree/pull/676

Closes: #679
Approved by: jlebon
2017-03-14 15:27:47 +00:00
Colin Walters
5142b8b4c2 compose-tree: Clean up metadata parsing
Preparatory work for a future patch; by having a hash it's easier to handle
duplicates. Note I started trying to use decl-after-stmt but hit the fact we're
still using `goto out` here.

Closes: #679
Approved by: jlebon
2017-03-14 15:27:47 +00:00
Colin Walters
05a59e5af5 core: Don't try to chmod() symlinks
I just noticed this while reading the code. Right now we're actually following
links (since `AT_SYMLINK_NOFOLLOW` isn't implemented), which could definitely
lead to bad things if e.g. one has a symlink pointing to a file which shouldn't
be executable or world-readable etc.

(Also I noticed our hardlink-breaking-only-if-regular logic is now broken;
 I added a TODO)

Closes: #677
Approved by: jlebon
2017-03-13 17:53:51 +00:00
Colin Walters
77fa10cfbe core: Skip (and warn on) rpmfi overrides outside of /{usr,etc,var}
I still need to query whether Fedora has anything like this; maybe `/boot`? But
things are not likely to work right now if we do, so let's ignore it.

Closes: #677
Approved by: jlebon
2017-03-13 17:53:50 +00:00
Colin Walters
ab7431c458 core: Apply rpmfi overrides for /etc to /usr/etc
This fixes installing `mock`, which for some reason has files in `/etc/mock`
owned by `root:mock`.

Closes: https://github.com/projectatomic/rpm-ostree/issues/671

Closes: #677
Approved by: jlebon
2017-03-13 17:53:50 +00:00
Colin Walters
d8b96a8531 core: Include filename in error messages for rpmfi overrides
This gives me now:

```
error: While applying overrides for pkg mock: fstatat: etc/mock/custom-1-aarch64.cfg: No such file or directory
```

Which is definitely more useful for debugging 💣.

Closes: #677
Approved by: jlebon
2017-03-13 17:53:50 +00:00
Colin Walters
62320b1920 rebase: Update man page for syntax, print if just changing remote
The rationale for just printing if changing remote is I think
it's really only that version that feels "magical".

Closes: https://github.com/projectatomic/rpm-ostree/issues/569

Closes: #678
Approved by: jlebon
2017-03-13 14:45:30 +00:00
Jonathan Lebon
c65ac2955c redhat-ci: provision more nodes for vmcheck
Make use of the new support for parallel execution. In the best case,
this should bring down the execution time to about a third.

Closes: #675
Approved by: cgwalters
2017-03-13 00:42:13 +00:00
Jonathan Lebon
29840bcecb vmcheck: add support for parallel runs
Allow the `make vmcheck` target to take a HOSTS var, which is simply a
space-separated list of hosts on which we can run testsuites. Add a
multitest.py script that takes care of monitoring and scheduling the
tests onto the nodes.

The script itself is "dumb": we don't know how long each test can take,
so we can't do any smart/heuristic scheduling that could save more time.

Closes: #675
Approved by: cgwalters
2017-03-13 00:42:13 +00:00
Jonathan Lebon
9d9042d835 vmcheck: prep for parallel runs
1. Don't require an ssh-config

In the case of redhat-ci, the VMs are already fully configured for the
system (injected in the hosts file, host key accepted, etc...). So
there's no need to have an ssh-config there. In general, it should be
acceptable to run the vmcheck suite against a resolvable host without
having to create an ssh-config for it.

2. Make the host name configurable

Rather than hardcoding "vmcheck" as the hostname, allow overridding it
by specifying a VM env var directly. We also prepare the various scripts
to make use of the $VM variable whenever host-specific dirs/files are
created so that parallel runs won't step on each other.

Closes: #675
Approved by: cgwalters
2017-03-13 00:42:13 +00:00
Jonathan Lebon
058d7e356e bump libdnf for chksum fix
Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00
Jonathan Lebon
fd64084b9b vmcheck: add basic test for local RPM installs
Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00
Jonathan Lebon
81c43e81fb PkgChange: add support for layering local RPMs
Allow users to directly specify an RPM file on the command-line. The
"packages_added" array of the PkgChange() method can now contain
absolute paths to RPM files.

Grow the origin format to have a new "requested-local" key. This is
similar to the "requested" key, except that the packages are always
installed from cache. The "requested-local" array values also embed the
SHA-256 of the header we expect.

There is now a new "LocalPackages" line in the status. These packages
are a subset of the "packages" element (which are printed as
"LayeredPackages") and represent the packages that are explicitly marked
for installing from cache.

Interesting design choices/notes:
  - Just as before, even with foo-1.0-1.x86_64 installed from RPM, a
    user can still request "/usr/bin/foo": it will be made dormant. As
    soon as foo stops being explicitly layered from the RPM, it will try
    to fulfill the request by going to the repos. This allows users to
    "pin" a layered package to a certain RPM, and then unpin it.
  - The strings/NEVRAs in "requested" and "requested-local" are strictly
     distinct. This allows us to be able to tell what the user means
     exactly when they do "rpm-ostree uninstall".

Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00