Commit Graph

2504 Commits

Author SHA1 Message Date
Jonathan Lebon
1fe66e91d0 rust/treefile: Support dash convention for all options
Now that we support YAML, it's a gotcha/eyesore that some of our options
use underscores rather than dashes. Let's be nice and switch those few
options over, while of course still supporting the previous name.

Co-authored-by: Colin Walters <walters@verbum.org>

Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Jonathan Lebon
0ebfb6e4ff rust/treefile: Don't hardcode list of architectures
Just try to match the current architecture against any
`packages-${arch}` keys in the treefile.

Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Jonathan Lebon
24335bdf1a rust/treefile: Use HashMap to collect extra fields
Instead of the `Strict` and `Permissive` variants of
`TreeComposeConfig`, just collapse all the excess fields in a new
`extra` member and check for any keys there in the strict YAML path.
This will also allow us to drop the hardcoded list of architectures in
the next patch.

Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Chris Weeks
90790cabaf Add support for bash completion
This stand-alone file doesn't impact any part of the build process and
is designed to be a convenience for end-users.

Closes: #1499
Approved by: jlebon
2019-03-01 21:36:39 +00:00
Jonathan Lebon
82594b5052 rust: Bump dependencies
Notably, the latest indicatif compiles fine again. This is now the
latest of everything, except the glib stack:

```
$ cargo outdated -R
Name      Project  Compat  Latest  Kind    Platform
----      -------  ------  ------  ----    --------
gio-sys   0.7.0    ---     0.8.0   Normal  ---
glib      0.6.1    ---     0.7.1   Normal  ---
glib-sys  0.7.0    ---     0.8.0   Normal  ---
```

because it requires a newer rustc than we're currently locked at.

Closes: #1770
Approved by: cgwalters
2019-03-01 15:51:07 +00:00
Jonathan Lebon
27e4c353b2 daemon/sysroot: Fix ActiveTransaction property
Minor regression from #1755. Noticed this after seeing the following in
the journal:

```
Mar 01 02:41:03 f29-ros rpm-ostree[3931]: Failed to GetConnectionUnixUser for client /org/projectatomic/rpmostree1/fedora_atomic: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Could not get UID of name '/org/projectatomic/rpmostree1/fedora_atomic': no such name
Mar 01 02:41:03 f29-ros rpm-ostree[3931]: Failed to GetConnectionUnixProcessID for client /org/projectatomic/rpmostree1/fedora_atomic: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Could not get PID of name '/org/projectatomic/rpmostree1/fedora_atomic': no such name
```

Closes: #1769
Approved by: cgwalters
2019-03-01 15:50:33 +00:00
Colin Walters
5c69bcb4fe compose: Make initramfs.img world-readable
There's nothing secret there, it's server side generated.
Ref: https://github.com/coreos/coreos-assembler/pull/372#issuecomment-467620937

Closes: #1767
Approved by: jlebon
2019-02-27 14:33:05 +00:00
Jonathan Lebon
2be532c9e9 rust/treefile: Add basearch key
Add a `basearch` key to the manifest. This can be used at compose time
to assert the architecture the compose is running on. Though my
motivation is for the common case where it gets omitted from the input
manifest and gets automatically added by rpm-ostree into
`/usr/share/rpm-ostree/treefile.json` for introspection on the client.

(The crucial part here is that the treefile created by rpm-ostree
remains deserializable into a `TreeComposeConfig`).

Closes: https://github.com/coreos/fedora-coreos-tracker/issues/154

Closes: #1766
Approved by: cgwalters
2019-02-27 01:05:43 +00:00
Jonathan Lebon
6a57db1c36 rust/treefile: Rename arch -> basearch
We're really using this variable to substitute `${basearch}` and find
basearch-specific packages. Let's rename the variable to make that more
obvious.

Closes: #1766
Approved by: cgwalters
2019-02-27 01:05:43 +00:00
Jonathan Lebon
75f415f09f tests/libtest.sh: Lift assert_jq from libvm.sh
We already had this logic, but it was in `libvm.sh`. Prep for using it
elsewhere.

Closes: #1766
Approved by: cgwalters
2019-02-27 01:05:43 +00:00
Colin Walters
3f722b5005 rust: Bump crate dependencies
Prep for adding other crate deps for sysusers, multiprocessing work.

Closes: #1765
Approved by: jlebon
2019-02-26 16:42:42 +00:00
Colin Walters
07d621bac9 compose: Pass treefile directly to core
A lot of history here.  Long before the YAML treefiles, I was
dissatisfied with the JSON treefile syntax, and also wanted
to create a format that could be used by the (still experimental)
`container` builtin, and chose to use GKeyFile.

I don't think that really worked out; the Rust YAML is just way
better.

In the future we probably want to more cleanly split off the
"compose only" aspects of the treefile.  But for now, it's rather
tedious to pass down flags from the treefile into the keyfile/gvariant.
Let's just give direct access to the treefile to the core.

Prep for sysusers.

Closes: #1764
Approved by: jlebon
2019-02-25 20:56:44 +00:00
Colin Walters
d1eedbbd96 rust/openat: Add helper to retrieve file type
Will be used by sysusers code.

Closes: #1764
Approved by: jlebon
2019-02-25 20:56:44 +00:00
Colin Walters
579f017fc2 rust: Add clap
A few of my outstanding Rust PRs use this (multiprocess, sysusers).
Let's add it now.

Closes: #1764
Approved by: jlebon
2019-02-25 20:56:44 +00:00
Colin Walters
e7f87b03ef tests: Don't make system user with differently-named group
Sysusers doesn't really support this, and no RPMs do it.

Closes: #1763
Approved by: jlebon
2019-02-25 14:21:28 +00:00
Colin Walters
a2044f8923 main: Move cancellable creation later
We should only be calling into GIO once we've set our preparatory
environment variables etc.  Not fixing anything today that I know
of, just noticed while looking at the code.

Closes: #1763
Approved by: jlebon
2019-02-25 14:21:28 +00:00
Colin Walters
13e4976f59 bwrap: Add an API to take a file descriptor
Will be used by the sysusers code to pass down a pipe that
the child writes to.

Closes: #1763
Approved by: jlebon
2019-02-25 14:21:28 +00:00
Colin Walters
406cc51fdc compose: Support RPMOSTREE_PRESERVE_TMPDIR=on-fail
This is easy to set in the environment for local development
and avoid leaking space in the success cases.

Closes: #1763
Approved by: jlebon
2019-02-25 14:21:28 +00:00
Colin Walters
aea374710e daemon: Time out client txn connections after 30s
This removes the "transaction monitor".  Originally the idea
here was things would queue, but we never really made that
work. Since transactions started grabbing the
libostree sysroot lock in particular.

Rip out the transaction monitor code with the queue and have
a singleton txn ref in the sysroot object.  This should ensure
that if a txn is active, one always gets an error message about
which one.

Next, add a 30s timeout for connections to the transaction DBus
sockets after they're complete.

This should address https://github.com/projectatomic/rpm-ostree/issues/1692
which is a case where gnome-software was leaking the txn DBus
connection and keeping it alive.

Closes: #1755
Approved by: cgwalters
2019-02-22 14:47:02 +00:00
Colin Walters
e58ee20996 cli/cancel: Close txn bus connection after cancelling
There's no reason to keep this around, we aren't going to use it.
Motivation here is the lifetime of transactions is currently
entangled with the bus connections from clients, and we don't
want to hold them open unnecessarily.

Also, we were adding a notify on the wrong object.  I think it
was just working before because something else ran the mainloop
and we got the property notification anyways.

Closes: #1759
Approved by: jlebon
2019-02-19 16:01:50 +00:00
Jonathan Lebon
97a60f90bd Release 2019.2
Let's get the SELinux fix out (#1754).

Closes: #1757
Approved by: cgwalters
2019-02-14 18:23:37 +00:00
Jonathan Lebon
0e5044ca92 postprocess: Add /home -> /var/home SELinux substitution
Manually patch `file_contexts.subs_dist` so that `/home` is equivalent
to `/var/home`. This is required now that the generated homedirs rules
use `/var/home`. Otherwise, `matchpathcon` for example will return wrong
results.

This patch also includes the *removal* of `/var/home -> /home` so that
we're not dependent on this selinux-policy patch making it at the same
time as downstream:

https://src.fedoraproject.org/rpms/selinux-policy/pull-request/14

(See the conversation there for more information.)

Closes: #1754
Approved by: cgwalters
2019-02-14 17:24:09 +00:00
Jonathan Lebon
677c083f24 libpriv: Rebuild policy during postprocessing
It's possible for some postprocessing scripts to affect the final
SELinux policy. This is the case for the new `/etc/default/useradd` edit
we now do (#1726), but it could've been the case beforehand too with
user scripts modifying e.g. booleans (though ideally all these
modifications would be part of RPMs).

Do a final `semodule -nB` during postprocessing so that the final policy
we commit is "up to date". Otherwise, users may only see changes take
effect if they layer packages that trigger a rebuild.

The motivation for this is specifically for `/etc/default/useradd`.
There is magic in `selinux-policy` that parses the file and generates
templated rules from the value of `HOME`.

For more info, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1669982
https://src.fedoraproject.org/rpms/selinux-policy/pull-request/14

Closes: #1754
Approved by: cgwalters
2019-02-14 17:24:09 +00:00
Jonathan Lebon
58a79056a8 libpriv/scripts: Add /run/ostree-booted for scriptlets
RPM-OSTree has been pretty good so far at consuming the exact same RPMs
used for traditional OSes without modifications. This is important,
because shielding RPMs from the OSTree abstraction means we remain
compatible with a large portion of the ecosystem.

However, there are some apps that definitely require rethinking their
approach. The example right now is akmods, which has a patch proposed to
build kmods at `%post` time on OSTree systems instead of from the
daemon.[1]

In such situations, scriptlets need something to key off of for the
OSTree-specific approach. The `/run/ostree-booted` file is the de facto
API to determine if we're running on an OSTree system or not. This patch
simply extends this API so that scriptlets can naturally make use of
them.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1667014

Closes: #1750
Approved by: cgwalters
2019-02-12 14:24:48 +00:00
Colin Walters
85ab9c3181 treefile: Fix octal mode for rojig spec too
Going to need to see if clippy or something has a way for us
to reject leading `0`.

Closes: #1756
Approved by: jlebon
2019-02-11 18:21:51 +00:00
Colin Walters
9ca74d0c31 compose: Fix install --unified-core
There's lots of gyrations here for unified-core vs not; it's
been broken in the case of `--unified-core` for a while I think.
In that case our workdir is tmpdir, so rename that directory.

Closes: #1743
Approved by: jlebon
2019-02-05 22:26:11 +00:00
Jonathan Lebon
7cceb35219 app/rebase: Support local repo remotes
Teach rpm-ostree to interpret rebases where the remote component is a
path to a local repo, e.g.:

    rpm-ostree rebase /mnt/ostree/repo:my/target/ref

Essentially, the local remote in this case is considered "ephemeral".
It's kind of the equivalent of, on traditional systems:

    dnf install --repofrompath repo,/path/to/repodata ...

The use case for this is in OpenShift v4, in which upgrades are done
from containers containing the OSTree commit. There, we want to point
RPM-OSTree directly at the repo in the mounted container and rebase to
the checksum.

For now, the option is marked experimental. One major reason for this is
that the way we pass the repo differs on RHEL7 vs other platforms. (See
comment block in `rpmostree-dbus-helpers.c` for details).

Related: https://github.com/openshift/machine-config-operator/issues/314

Co-authored-by: Colin Walters <walters@verbum.org>

Closes: #1732
Approved by: cgwalters
2019-02-05 18:03:02 +00:00
Colin Walters
07fbd195fb composepost: Fix permissions on /usr/etc/default/useradd
Switching between programming languages can be a pain.  Unfortunate
that Rust happily accepts a leading `0` too.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1752

Closes: #1753
Approved by: jlebon
2019-02-04 18:37:58 +00:00
Jonathan Lebon
3fda356dd5 docs/treefile: Add some more details to mutate-os-release
Make it easier to understand what the field actually represents.

Closes: #1751
Approved by: ashcrow
2019-02-01 18:43:29 +00:00
Jonathan Lebon
fa5be441b1 libpriv/rpm-util: Group by SRPMs when printing changelogs
I've become quite fond of the `--changelogs` switch lately on my
Silverblue. I find it useful as a first step when nailing down a
regression, or just to peruse the churn of changes after an update.

One annoyance is that the same changelog gets printed multiple times for
each RPM that shares the same SRPM. This patch does a naive lookahead to
dedupe them. Example output:

```
  NetworkManager 1:1.12.6-4.fc29.x86_64 -> 1:1.12.6-5.fc29.x86_64
  NetworkManager-adsl 1:1.12.6-4.fc29.x86_64 -> 1:1.12.6-5.fc29.x86_64
  NetworkManager-bluetooth 1:1.12.6-4.fc29.x86_64 -> 1:1.12.6-5.fc29.x86_64
  NetworkManager-config-connectivity-fedora 1:1.12.6-4.fc29.noarch -> 1:1.12.6-5.fc29.noarch
  NetworkManager-libnm 1:1.12.6-4.fc29.x86_64 -> 1:1.12.6-5.fc29.x86_64
  NetworkManager-ppp 1:1.12.6-4.fc29.x86_64 -> 1:1.12.6-5.fc29.x86_64
  NetworkManager-wifi 1:1.12.6-4.fc29.x86_64 -> 1:1.12.6-5.fc29.x86_64
  NetworkManager-wwan 1:1.12.6-4.fc29.x86_64 -> 1:1.12.6-5.fc29.x86_64
    * Mon Jan 14 2019 Beniamino Galvani <bgalvani@redhat.com> - 1:1.12.6-5
    - improve Wi-Fi PMF support (rh #1665694)
```

I say "naive" because this doesn't catch the case where a binary RPM is
named differently enough that they're not consecutive. Anyway, this
simple heuristic cleans up the output nicely.

Closes: #1738
Approved by: cgwalters
2019-01-26 14:36:52 +00:00
Colin Walters
65afbec3b1 rust: Split openat helpers into module
Will help use them in other parts of our code.

Closes: #1737
Approved by: jlebon
2019-01-23 16:00:16 +00:00
Jonathan Lebon
25d0213d15 rust/treefile: Include filename in more error msgs
This uses the `Context` feature of the failure crate to make error
messages more useful when we fail to open a file. The difference with
`map_err` is that one can still obtain the underlying error from the
context if need be. Though surprisingly, the normal `Display` for a
`Context` doesn't include the original error, so we essentially have to
do a prefix here (see [1]).

Before:

```
error: Failed to load YAML treefile: No such file or directory (os error 2)
```

After:

```
error: Failed to load YAML treefile: Can't open file "treecompose-post.sh": No such file or directory (os error 2)
```

[1] https://github.com/rust-lang-nursery/failure/issues/182

Closes: #1735
Approved by: cgwalters
2019-01-22 15:49:34 +00:00
Jonathan Lebon
1594140a33 rust/treefile: Convert more functions to Fallible
Slightly prep for next patch (was in the area so just did the whole
module).

Closes: #1735
Approved by: cgwalters
2019-01-22 15:49:34 +00:00
Jonathan Lebon
d9577618a1 Release 2019.1
Not a lot of changes, but a couple of important bugfixes.

Closes: #1736
Approved by: dustymabe
2019-01-21 17:54:25 +00:00
Jonathan Lebon
34bdffcbd2 packaging: Don't include checksums for libtool.m4 and configure
These are more files that get mangled at `%configure` time. These two
new ones specifically, I tripped on while building on ppc64le for RHEL7.
See https://bugzilla.redhat.com/show_bug.cgi?id=1113618 for more info.

Closes: #1731
Approved by: cgwalters
2019-01-15 19:20:36 +00:00
rfairley
a20d10e3d5 libpriv/util: Add date field in auto-versioning
This adds an optional date field to the prefix
passed by automatic_version_prefix. An example of specifying
the field is as follows:

10.<date:%Y>

And the fields progress like:

10.2018.0
10.2018.1
10.2018.2
10.2019.0

The date format creates a new "current date" string using
valid date directives passed into g_date_time_format().

If there is a problem reading the given date format,
an error is given and the next version is returned as NULL.

If no <date:...> tag is detected in the auto version prefix,
the same behavior as before (appending .1 and incrementing) occurs.

This may be helpful to avoid writing glue code to auto-update
the version if a date string in the commit version is desired.
Otherwise, --add-metadata-string=version= is an alternative for
complete customization.

Fixes: #1712

Closes: #1721
Approved by: jlebon
2019-01-14 18:49:41 +00:00
Colin Walters
89cdeb5131 Add rayon dependency, do postprocessing in parallel
Obviously doing these trivial functions in parallel doesn't matter,
but I wanted to dip my toes into Rayon (and Rust parallelism in
general).

`try_for_each()` is just...staggeringly simple and beautiful.

Closes: #1727
Approved by: jlebon
2019-01-07 18:50:49 +00:00
Colin Walters
255dc7027e compose: Minor further oxidation of postprocessing
Take a baby step into further oxidation by migrating the bit
to write our presets.

Closes: #1727
Approved by: jlebon
2019-01-07 18:50:49 +00:00
Colin Walters
56e6ddb8e5 compose: Change /etc/default/useradd to use HOME=/var/home
For a long time I've resisted encoding "policy" into rpm-ostree
as much as possible.  Doing so makes it more distribution specific
for example.  That said, for `/var/home` there argument for doing
this in rpm-ostree is that we already make that symlink in our
hardcoded rootfs.  So we might as well do the other fixups for it.

https://github.com/coreos/fedora-coreos-config/pull/18
https://pagure.io/workstation-ostree-config/pull-request/121
https://discussion.fedoraproject.org/t/adapting-user-home-in-etc-passwd/487/6
https://github.com/justjanne/powerline-go/issues/94

Closes: #1726
Approved by: jlebon
2019-01-07 15:35:37 +00:00
Colin Walters
bb4a17d0ba compose: Add a stub "postprocess" entrypoint in Rust
Prep for more code.

Closes: #1726
Approved by: jlebon
2019-01-07 15:35:37 +00:00
Colin Walters
43bd9f4329 core: Fix rpm-md repo caching
I'd been seeing hangs sometimes in "Importing metadata" and had
thought it was libsolv being slow, but actually the problem is
that while we explicitly check the libdnf repos,
`dnf_context_setup_sack_with_flags()` checks them again using
its `cache_age`.

Set the libdnf context's expiry to "never" after we've done the
checking so it's not checked twice.

This PR brought to you by airplane 🛫 travel and not wanting
to pay for expensive WiFi 💸.

Closes: #1723
Approved by: lucab
2018-12-25 08:39:16 +00:00
Jonathan Lebon
fa64ebfa72 rust: Bump dependencies
This leaves only indicatif outdated (see link in diff).

Closes: #1720
Approved by: cgwalters
2018-12-19 21:25:13 +00:00
Jonathan Lebon
514f4d2473 core: Explain a bit the base/layered split mismatch issue
This is a very commonly hit issue in Fedora land. Add some explanation
so users aren't just confused, as well as the `cleanup -m` pointer. Also
link back to the main issue.

Closes: #1717
Approved by: cgwalters
2018-12-19 20:40:17 +00:00
Jonathan Lebon
9a3cd0782b rust: Add Cargo.lock
We've been seeing more breakage recently due to much of the Rust
ecosystem sticking to minimum rustc versions more recent than ours. A
lockfile now means we have to periodically update it, but at least we
have much better control over our CI system and when we want to spend
time resolving package bump conflicts vs. just trying to get a small
patch in.

Closes: #1719
Approved by: cgwalters
2018-12-19 16:00:00 +00:00
Jonathan Lebon
f6de8f3dfc rust: Drop backtrace lock
We can build against the latest now that we bumped our minimum Rust
version.

Closes: #1716
Approved by: cgwalters
2018-12-17 22:06:49 +00:00
Jonathan Lebon
1f72fbb055 ci: Bump minimum Rust version to 1.29.2
This matches the latest DTS release on el7.

Closes: #1716
Approved by: cgwalters
2018-12-17 22:06:49 +00:00
Colin Walters
c60043c2db compose: Use dracut tmpdir under target root
The problem here is that bare-user has `user.ostreemeta` xattrs,
`tmpfs` (which bwrap uses for `/var/tmp` by default) doesn't support that.

Pass through the tmpdir to the target rootfs, which is the same place
as the repo in unified-core mode.

Ref: https://github.com/coreos/coreos-assembler/issues/254

Closes: #1716
Approved by: cgwalters
2018-12-17 22:06:49 +00:00
Jonathan Lebon
5202ce06c2 packaging: Don't include checksums for ltmain.sh
And `config.guess` and `config.sub`. These files get mangled by the
`%configure` macro when it tries to insert hardening compile flags and
so the checksums no longer match. This is an ugly hack akin to #1554
that requires an incision in the cargo vendor JSON.

Fedora does package a lot of these crate sources now which we
could use to drop these hacks, but not all the crates are packaged (I
counted 4 unpackaged top-level crates), and I'm not sure what their
states are in RHEL7/8 either.

Closes: #1715
Approved by: cgwalters
2018-12-15 14:21:19 +00:00
Jonathan Lebon
b889a51056 Release 2018.10
It's been a while and we have lots of goodies. Likely the last one
before the new year. 🎉

Closes: #1713
Approved by: cgwalters
2018-12-13 16:34:56 +00:00
Jonathan Lebon
24d401072a Cargo.toml: Fix backtrace crate to compatibile version
The latest backtrace release isn't compatible with 1.26.2. A patch for
this is underway in:

https://github.com/alexcrichton/backtrace-rs/pull/137

Though for now, let's just restrict the version since we should be
bumping our minimum rustc requirement soon-ish anyway.

Closes: #1713
Approved by: cgwalters
2018-12-13 16:34:56 +00:00