Commit Graph

2387 Commits

Author SHA1 Message Date
Colin Walters
88ffdc0ae2 Add compose rojig command
This currently requires a `--i-know-this-is-experimental` flag;
I know it'd be a bit more consistent to have it under `ex`, but
what feels weird about that is *most* of the `ex` commands people
use are client side.  This is where we want it to ultimately end
up.

We've landed a lot of prep patches, but I know there's still
a notable amount of code duplication with `compose tree`.  What's
left is about ~700 lines but it's mostly not hard/complex code
anymore.

In the future, I'd like to extract more of the compose code
to a `rust/src/compose.rs` or so, but I think this is sustainable
fow now.

My high level goal is to get this into coreos-assembler and stand
up a Silverblue build that uses it.

Closes: #1512
Approved by: jlebon
2018-10-31 17:57:45 +00:00
Colin Walters
c044f59726 rust: Remove pub from modules
Conceptually we're only exporting the FFI code right now.

Closes: #1649
Approved by: jlebon
2018-10-30 16:16:22 +00:00
Colin Walters
5b8b37658f compose: Make --print-only happen earlier and be quiet
Don't print the version or other warnings; ensure we output clean JSON.

Prep for fixing https://github.com/coreos/fedora-coreos-config/pull/24
(This helps, but we still need to expand `${basearch}`)

Closes: #1648
Approved by: jlebon
2018-10-30 15:04:21 +00:00
Colin Walters
87441d9ec2 rust: Move FFI bits into ffi submodule for each file
The `lib.rs` file was starting to accumulate, and I want to do this
before adding more.  I just made this up, but an `ffi` submodule
in each file seems to work well.  It isolates the FFI consumption
there still (so e.g. if we want pure Rust unit tests, we don't
need to deal with FFI).

Closes: #1646
Approved by: jlebon
2018-10-30 14:24:42 +00:00
Colin Walters
99776be162 core: Disable librpm SELinux plugin when writing rpmdb
OSTree was designed from the very beginning of its existence
to support SELinux well instead of being something wedged on.
rpm-ostree builds on that foundation.  We don't want to have anything
to do with librpm's SELinux code.  And with unified core,
we usually don't, but that `rpm-plugin-selinux` code does get loaded.

Disable it here.  The main reason I'm submitting this patch
is to help an effort in coreos-assembler to use a "supermin" virtual
machine: https://github.com/coreos/coreos-assembler/pull/124

Closes: #1647
Approved by: jlebon
2018-10-29 18:42:10 +00:00
Colin Walters
b5d34b7268 compose: Factor out helper for writing composejson
Prep for sharing this code with rojig.

Closes: #1636
Approved by: jlebon
2018-10-29 01:35:20 +00:00
Jonathan Lebon
f3b8ef043e libpriv: Sanity check that paths are OSTree compatible
I originally was going to add the check here and error out to the user,
but `add-files` is handled all the way near the end of the compose,
which meant that users would have to wait through it all before getting
an error. Though the check is enforced at parsing time now, I think it's
still useful at postprocess time as well as a sanity check.

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Jonathan Lebon
940fc1364a compose: Check that add-files are compatible after parsing
While serde gives us type checking, it of course doesn't understand
semantics beyond that. One example is checking the compatibility of
`add-files` entries with the OSTree model. This is something we can do
upfront early on to avoid surprises for users.

Also tweak the docs to reflect this new check.

Related: #1642

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Jonathan Lebon
f8f47480f1 docs/treefile: Add info about paths for add-files
Add a note that only certain paths actually make sense in the
`add-files` directive and link to the OSTree docs. Let's also the
examples given to more realistic and correct ones.

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Colin Walters
202192d61e rust: Create generic ref_from_raw_ptr function
I need to do some hunting for a "ffi helper" crate.  I kind of
understand why these things aren't in the std library.

Anyways this is easy for now, prep for adding more types.

Closes: #1641
Approved by: jlebon
2018-10-26 20:04:08 +00:00
Colin Walters
92d207de5d rust: Merge glibutils with utility FFI helpers
Create `ffiutil.rs` from the merger.  This way all of our FFI
helpers are in one place.  Prep for further refactoring.

Closes: #1641
Approved by: jlebon
2018-10-26 20:04:08 +00:00
Colin Walters
245d238563 Use autocleanup for task output
I was playing with https://crates.io/crates/indicatif
and this is prep for using it.

By using autocleanups here we ensure that the output state
ends even if we encounter an error.  We previously had a few
explicit `failed` prints in cases where failure was more common
such as dependency resolution, but that was far from consistent.

Now we will always ensure a newline is printed even if an error
occurs, and we won't have to worry about missing calls to
`_task_done()` in the success case.

The downside of this is that we lose the `done` bit in the success
path...I thought about maybe taking a pointer to the error and
printing `done` or `failed` depending on whether it's non-`NULL`
but eh...the `done` is implied by us going on to the next line.
Also that issue will be fixed with the indicatif work.

Closes: #1640
Approved by: jlebon
2018-10-26 19:20:25 +00:00
Jonathan Lebon
a87486e18e rust: Drop crates-io patch and use 0.4.0
The latest release of the crate includes the features and fixes we need.

Closes: #1639
Approved by: cgwalters
2018-10-25 19:01:39 +00:00
Jonathan Lebon
537a040b00 Release 2018.9
Closes: #1633
Approved by: cgwalters
2018-10-25 18:12:37 +00:00
Christian Glombek
20acf6a2f5 compose: Print version number
Users doing builds that are using newer features
want to be sure they have the right version.

Closes: #1556

Closes: #1637
Approved by: jlebon
2018-10-25 17:07:10 +00:00
Colin Walters
fe5adaf7ab build-sys: Fix make rustfmt, and run it
When we added the bindgen config this broke as it's not a `.rs`.
Let's keep the source formatted canonically.

Closes: #1635
Approved by: jlebon
2018-10-23 19:05:06 +00:00
Michael Nguyen
50ef17ed95 app/status: line wrap RemovedBasePackages
Wrap RemovedBasePackages like other package lists in the status
command so it looks prettier.

fixup! app/status: line wrap RemovedBasePackages

fixup! app/status: line wrap RemovedBasePackages

fixup! app/status: line wrap RemovedBasePackages

Resolves: #1613

Closes: #1632
Approved by: jlebon
2018-10-23 14:28:19 +00:00
Colin Walters
fca01e70b5 Lower initial SELinux policy load from compose to core
Add a `selinux` verb to treespec, and bind it from treefile.  If
set, use it in the core to load an initial policy before import,
if we didn't already set a policy.

In practice right now this is only used from the compose path
since the SysrootUpgrader uses the policy from the merge deployment.

Unset the policy if rojig mode is enabled.

Now, non-SELinux use cases are required to set `selinux: false`
in the treespec. For `ex container` I just set it in our example
specs.  Probably that should forcibly disable it in the
treespec but eh, it's experimental.

The other case I can think of is client-side layering; before
we would create a policy using the target root, but it
wasn't a *hard* requirement, i.e. we didn't error out if
`policy_get_name() == NULL`.  Let's preserve that semantic by
hooking off of whether `_new_system()` was used.

Prep for sharing code with `compose rojig`.

Closes: #1630
Approved by: jlebon
2018-10-23 13:40:46 +00:00
Colin Walters
95c1ed3695 tests/compose: Use workdir in tmpdir
This way when debugging if I want to preserve the state so I can
go there and rerun command under gdb for exmaple, all I need
to do is set `env TEST_SKIP_CLEANUP=1` rather than also needing
to find a separate dir and also set `env RPMOSTREE_PRESERVE_TMPDIR=1`.

Closes: #1630
Approved by: jlebon
2018-10-23 13:40:46 +00:00
Colin Walters
f1fa436c8a rust/treefile: Use macros to reduce redundancy in treefile merging
On the plus side, when submitting a patch to Github, no one knows
how long it took you to figure out...

Anyways so this reduces redundancy.  I double-checked the list.
I was inspired to pick this back up after seeing a Rust code
snippet somewhere noting that macros defined inside a function
can capture variables, which simplifies this even more.

Closes: #1631
Approved by: jlebon
2018-10-22 18:53:16 +00:00
Jonathan Lebon
6583a557ae app/livefs: Require --i-like-danger switch
We've had multiple reports by now of folks using plain `ex livefs` and
getting their bootloader wrecked:

https://github.com/projectatomic/rpm-ostree/issues/1495
https://github.com/projectatomic/rpm-ostree/issues/1504
https://github.com/ostreedev/ostree/issues/1459

Let's require a scary switch for now to emphasize this.

Closes: #1622
Approved by: cgwalters
2018-10-17 20:10:11 +00:00
Colin Walters
7c41188476 Switch to ISO8601 timestamps in more places
I was doing this for the "generated" bit on the compose side,
but let's be consistent elsewhere.

Closes: #1624
Approved by: jlebon
2018-10-17 19:32:12 +00:00
Colin Walters
0f813c1758 scripts: Fully neuter microcode_ctl scripts everywhere
I swear I tested the previous PR here but it looks like they
did a new build recently whose release ends in `el7_5` which no
longer matches.

Since the Fedora spec doesn't have any scripts, let's go back
to using the gperf list.

Closes: #1623
Approved by: jlebon
2018-10-17 18:48:38 +00:00
Jonathan Lebon
cdda3a01ba postprocess: Add ostree-finalize-staged.path
In preparations for https://github.com/ostreedev/ostree/pull/1740, just
hard enable this path unit for now since centrally-maintained distro
presets still need to be updated.

Closes: #1617
Approved by: cgwalters
2018-10-16 20:05:41 +00:00
Colin Walters
05cb34d6b4 compose: Lower metadata logic for read/generate into a helper
Prep for sharing code with `compose rojig`.

Closes: #1618
Approved by: jlebon
2018-10-16 19:18:17 +00:00
Colin Walters
c102a710a4 compose; Lower add-files read into composeutil
Make this function operate like all of the others, on the pair of
(treefile_rs, treefile).  That way the "parsing" is co-located
with usage.

Prep for `compose rojig`.

Closes: #1618
Approved by: jlebon
2018-10-16 19:18:17 +00:00
Colin Walters
10a60d2833 compose: Clean up passwd/group injection API a bit
Rename the function dealing with this, document it a bit,
and move all of the logic related to this inside there.

Prep for sharing more with `compose rojig`.

Closes: #1618
Approved by: jlebon
2018-10-16 19:18:17 +00:00
Jonathan Lebon
67a831b043 rust/journal: Adapt for el7 platforms
Supporting ancient systemd is painful, though given that it may end up
in RHCOS at least, it seems worth the effort.

Basically, the big changes here are:
- avoid using `_SYSTEMD_UNIT` since it might be missing in some entries
- also grep for the el7 version of the systemd msg when a service fails
- use `_TRANSPORT=stdout` for the error msg in case of ordering issues

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Colin Walters
d24f21343f build-sys: Just error out if building with multiple uids
The `--frozen` stuff ends up being annoying when switching
branches.  What we're really trying to protect against here
is the `sudo make install` problem, so let's test for that
more directly by verifying the uids.

(The previous code was also totally broken as it used `$` where
`$$` should have been in multiple places)

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
676519e5a5 status: Print systemd or ostree staged failure msg
Try to tease out a bit more info from the journal by looking at the
systemd message when the service transitions to the dead state or even
looking at the OSTree output itself.

Example outputs:

```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         error: opendir(ostree/deploy/fedora-atomic/deploy/887c95887a3047a60372016a0d84536530755b60df3cca33c819f7606e220adf.0): No such file or directory
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

```
[root@f28-ros ~]# rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         ostree-finalize-staged.service: Failed with result 'timeout'.
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Colin Walters
808a80bfd7 rust/lib: Remove unnecessary integer casts
The compiler will infer the correct thing.

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
5fb5effbd2 packaging: Nuke more vendored sources
Nuke systemd source files for the same reasons as libcurl (see #1554).
Also noticed that libz-sys was doing this, though it's not new to this
patch.

For reference, see: https://github.com/projectatomic/rpm-ostree/pull/1554

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
f6c3616148 status: Detect if staging failed in previous boot
Sample output:

```
$ rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
...
```

(Though open to tweaking it).

I also played with directly invoking `journalctl` for the user, but that
can get really spammy with e.g. `os-prober` output and such.

I wrote this in Rust using journal API wrappers because I also plan to
implement the `history` command in Rust and will also enhance that new
`journal` module there for that.

Requires: https://github.com/ostreedev/ostree/pull/1750
Requires: https://github.com/jmesmon/rust-systemd/pull/54
(Though I've pointed the manifest at my branch for now for CI).

Closes: #1567

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Jonathan Lebon
dbdbaf87ee ci: Make adduser invocation idempotent
This is a bit of a selfish request, though I also really like
`ci/build.sh` for hacking in a throwaway environment. This patch allows
one to rerun `ci/build.sh` without erroring out on `testuser` already
existing.

Closes: #1621
Approved by: cgwalters
2018-10-16 16:03:06 +00:00
Jonathan Lebon
cc69168aaf ci: Drop CI_PKGS and epel-release in .papr.yml
Move specific package mentions from the `.papr.yml` file to
`ci/installdeps.sh`. I find the latter script super useful for
bootstrapping a pet container for rpm-ostree development and this helps
with that.

Closes: #1621
Approved by: cgwalters
2018-10-16 16:03:06 +00:00
Jonathan Lebon
3a6315aac9 libpriv/passwd-util: Fix leaks on error path
Allocate new memory *after* the potential error exit, not before.

Closes: #1620
Approved by: cgwalters
2018-10-16 00:34:09 +00:00
Jonathan Lebon
700ab613ee core: Fix NULL pointer dereference
This is so obvious now, it's painful. We should fallback to the empty
string `""`, not `\0` which is of course semantically equivalent to
`NULL`. I think in practice we've never hit a `SIGSEGV` because librpm
itself has the same `?: ""` logic as well.

Closes: #1620
Approved by: cgwalters
2018-10-16 00:34:09 +00:00
Jonathan Lebon
58ab822569 daemon: Check if outvar is provided before assigning
Closes: #1620
Approved by: cgwalters
2018-10-16 00:34:09 +00:00
Colin Walters
0a050d833f compose: Add a helper to convert treefile → treespec
At some point perhaps we'll define a sane format that unifies
treefile/treespec.  This is not that day, but let's extract
the code that converts the two into a clean helper.  Then the
compose context holds onto the treespec, and e.g. things like the
ref just point to its data.

Prep for sharing this code with rojig.

Closes: #1616
Approved by: jlebon
2018-10-15 17:47:26 +00:00
Colin Walters
800e718573 compose: Drop duplicate addition of arch- and bootstrap pkgs
Since we parse the treefile consistently in Rust now, drop this
duplicate code.

Closes: #1616
Approved by: jlebon
2018-10-15 17:47:26 +00:00
Colin Walters
40ce70f0c2 scripts: Correctly override RHEL7 glibc-common.post
This bit moved to glibc-all-langpacks.posttrans at some point
in Fedora.  The code is the same, use the same override.  There
isn't a Fedora glibc-common.post.

So far we didn't really try `--unified-core` with RHEL7 content,
this fixes the last of the problems for me.

Closes: #1612
Approved by: jlebon
2018-10-15 16:39:12 +00:00
Colin Walters
cc3d3a4a19 scripts: Neuter RHEL7 pam and microcode_ctl
This gets me close to doing a build with `--unified-core`, but
now I'm tripping over a weird issue with the locale archive.  I think
it's related to the lua override.

Closes: #1612
Approved by: jlebon
2018-10-15 16:39:12 +00:00
Colin Walters
13c7bcaccb postprocess: Add missing newline to os-release message
Just noticed this in a build.

Closes: #1611
Approved by: jlebon
2018-10-12 22:25:30 +00:00
Colin Walters
fcbea6a67a compose: Also port one passwd bit to using Rust treefile
I missed this use before in the passwd code which was also parsing
the "filename" parameter.  Teach this to use the fd that was opened
Rust side too.

It's really tempting to try oxidizing this whole file but...baby steps.

Closes: #1610
Approved by: jlebon
2018-10-12 21:18:10 +00:00
Colin Walters
b3f6f25637 core,scripts: When no cachedir+unified-core, disable rofiles-fuse
This is prep for running inside (unprivileged) Kube containers
as they exist today: https://github.com/projectatomic/rpm-ostree/issues/1329

Sadly FUSE today uses a suid binary that ends up wanting CAP_SYS_ADMIN.
I think there's some work on FUSE-in-containers but I'm not sure of
the current status.

What rofiles-fuse here is doing here is protecting is the hardlinked
repo imports.  But if `--cachedir` isn't specified, that repository
gets thrown away anyways.  So there's no real value to using FUSE
here.

Also since nothing is cached, disable the devino cache.

We also make use of --force-copy-zerosized that just landed
in libostree: https://github.com/ostreedev/ostree/pull/1752

Down the line ideally we gain the capability to detect if either
unprivileged overlayfs/FUSE are available.  Then if `--cachedir`
is specified we can make things work.

Closes: #1591
Approved by: jlebon
2018-10-12 19:03:26 +00:00
Colin Walters
8ba5bda821 core: Apply s{u,g}id consistently on checkout
This is basically overriding what happens with `bare-user` mode
OSTree repositories.  I put a lot of thought into avoiding creating
suid files with that mode.

But today this creates a situation where if we don't have a devino
cache, the file will lose its suid bits.

In the end, since we're using the "inaccessible directory" pattern
anyways for rpm-ostree on the host, we don't need to really worry
about transient suid binaries.  And similarly when we're run inside
an existing container, that's also fine.

Closes: #1591
Approved by: jlebon
2018-10-12 19:03:26 +00:00
Colin Walters
4b6040b740 importer: Don't import libselinux .LOCK files
These shouldn't be in the package; the fact that they're empty
files causes libostree to hardlink them which breaks things.
See also https://github.com/projectatomic/rpm-ostree/pull/1002

Closes: #1591
Approved by: jlebon
2018-10-12 19:03:26 +00:00
Colin Walters
fca7c4bd35 rust/treefile: Include filename in error when parsing
"Entity not found" is not super useful.

Closes: #1607
Approved by: jlebon
2018-10-12 14:31:55 +00:00
Jonathan Lebon
7c2b9c3276 ci: Add rust-min-version-check
Add a check to make sure we stay within the minimum version required to
build with DTS, which is updated frequently, but may still lag behind in
comparison to Fedora stable.

Closes: #1606
Approved by: cgwalters
2018-10-11 20:39:57 +00:00
Jonathan Lebon
e15baebc68 ci: Split out script to install deps
This should also make it useful for bootstrapping a test container
locally.

Closes: #1606
Approved by: cgwalters
2018-10-11 20:39:57 +00:00