Commit Graph

2348 Commits

Author SHA1 Message Date
Colin Walters
800e718573 compose: Drop duplicate addition of arch- and bootstrap pkgs
Since we parse the treefile consistently in Rust now, drop this
duplicate code.

Closes: #1616
Approved by: jlebon
2018-10-15 17:47:26 +00:00
Colin Walters
40ce70f0c2 scripts: Correctly override RHEL7 glibc-common.post
This bit moved to glibc-all-langpacks.posttrans at some point
in Fedora.  The code is the same, use the same override.  There
isn't a Fedora glibc-common.post.

So far we didn't really try `--unified-core` with RHEL7 content,
this fixes the last of the problems for me.

Closes: #1612
Approved by: jlebon
2018-10-15 16:39:12 +00:00
Colin Walters
cc3d3a4a19 scripts: Neuter RHEL7 pam and microcode_ctl
This gets me close to doing a build with `--unified-core`, but
now I'm tripping over a weird issue with the locale archive.  I think
it's related to the lua override.

Closes: #1612
Approved by: jlebon
2018-10-15 16:39:12 +00:00
Colin Walters
13c7bcaccb postprocess: Add missing newline to os-release message
Just noticed this in a build.

Closes: #1611
Approved by: jlebon
2018-10-12 22:25:30 +00:00
Colin Walters
fcbea6a67a compose: Also port one passwd bit to using Rust treefile
I missed this use before in the passwd code which was also parsing
the "filename" parameter.  Teach this to use the fd that was opened
Rust side too.

It's really tempting to try oxidizing this whole file but...baby steps.

Closes: #1610
Approved by: jlebon
2018-10-12 21:18:10 +00:00
Colin Walters
b3f6f25637 core,scripts: When no cachedir+unified-core, disable rofiles-fuse
This is prep for running inside (unprivileged) Kube containers
as they exist today: https://github.com/projectatomic/rpm-ostree/issues/1329

Sadly FUSE today uses a suid binary that ends up wanting CAP_SYS_ADMIN.
I think there's some work on FUSE-in-containers but I'm not sure of
the current status.

What rofiles-fuse here is doing here is protecting is the hardlinked
repo imports.  But if `--cachedir` isn't specified, that repository
gets thrown away anyways.  So there's no real value to using FUSE
here.

Also since nothing is cached, disable the devino cache.

We also make use of --force-copy-zerosized that just landed
in libostree: https://github.com/ostreedev/ostree/pull/1752

Down the line ideally we gain the capability to detect if either
unprivileged overlayfs/FUSE are available.  Then if `--cachedir`
is specified we can make things work.

Closes: #1591
Approved by: jlebon
2018-10-12 19:03:26 +00:00
Colin Walters
8ba5bda821 core: Apply s{u,g}id consistently on checkout
This is basically overriding what happens with `bare-user` mode
OSTree repositories.  I put a lot of thought into avoiding creating
suid files with that mode.

But today this creates a situation where if we don't have a devino
cache, the file will lose its suid bits.

In the end, since we're using the "inaccessible directory" pattern
anyways for rpm-ostree on the host, we don't need to really worry
about transient suid binaries.  And similarly when we're run inside
an existing container, that's also fine.

Closes: #1591
Approved by: jlebon
2018-10-12 19:03:26 +00:00
Colin Walters
4b6040b740 importer: Don't import libselinux .LOCK files
These shouldn't be in the package; the fact that they're empty
files causes libostree to hardlink them which breaks things.
See also https://github.com/projectatomic/rpm-ostree/pull/1002

Closes: #1591
Approved by: jlebon
2018-10-12 19:03:26 +00:00
Colin Walters
fca7c4bd35 rust/treefile: Include filename in error when parsing
"Entity not found" is not super useful.

Closes: #1607
Approved by: jlebon
2018-10-12 14:31:55 +00:00
Jonathan Lebon
7c2b9c3276 ci: Add rust-min-version-check
Add a check to make sure we stay within the minimum version required to
build with DTS, which is updated frequently, but may still lag behind in
comparison to Fedora stable.

Closes: #1606
Approved by: cgwalters
2018-10-11 20:39:57 +00:00
Jonathan Lebon
e15baebc68 ci: Split out script to install deps
This should also make it useful for bootstrapping a test container
locally.

Closes: #1606
Approved by: cgwalters
2018-10-11 20:39:57 +00:00
Jonathan Lebon
8f504ec5ce ci/build.sh: Drop unnecessary pkg_install cargo
The latest releases in the stable repos now include the new cargo BR, so
we can drop this now.

Closes: #1606
Approved by: cgwalters
2018-10-11 20:39:57 +00:00
Colin Walters
886a0e4986 composeutil: Test for ability to open new /dev/null
If `nodev` is set, we should fail fast.  See also
https://github.com/containers/storage/pull/208

Closes: #1604
Approved by: jlebon
2018-10-11 17:31:55 +00:00
Colin Walters
aa485b7625 Fix cache_age handling for compose and client-side
PR: https://github.com/projectatomic/rpm-ostree/pull/1562
AKA commit: a7bbf5bc14
introduced two regressions.  First one for `compose tree`.  The intention is
the default there is to *always* immediately check for updated
rpm-md - a bit like `yum clean expire-cache`.  However due
to bugs in the stack we end up downloading it again anyways, but
that's not the topic here.

When we made that change we basically stopped using `DnfContext`'s
`cache_age`, which is what `compose tree` was setting.

Introduce a new explicit API to do what we want for `compose tree`.

Secondly, we were mistakenly always caching on the client
side.

This also fixes an issue that `--cache-only` didn't work, it basically
just made `compose tree` use the default `metadata_expire`.  Now we
really don't expire it.

Closes: #1587
Approved by: jlebon
2018-10-11 16:49:09 +00:00
Colin Walters
096f8dec65 rust/lib: Rework fd helpers to consistently lseek()
This way the fds are always known to be at the start.  For e.g.
`add-files` we need to both checksum them and then copy them
later.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
6846fe11fa compose: Move opening passwd/group files into Rust
This was I think the last place where we were looking at the parent
directory of the primary treefile.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
f7f2cfeb61 tests: Add validation for provided passwd/group files
Surprising we didn't have this yet.  Prep for oxidizing.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
c6cdda2c91 lib/rust: Add a helper to convert the raw pointer
So there's fewer uses of `unsafe`.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
0e5c1cde96 compose: Move "serialized treefile" into Rust ownership
Now that we have `CUtf8`, let's just store the serialized JSON
as a string, owned by the Rust side.  This way we can drop the
`serialized_treefile` buffer we were passing around and simplify
various bits of code.  Most notably, we only serialize the JSON
once (via Serde) rather than also doing it again in the C side.

Closes: #1600
Approved by: jlebon
2018-10-11 16:03:02 +00:00
Colin Walters
1d031fe51e core: Maintain /usr/etc as /etc when running scripts
In preparation for running in default Docker permissions where
we can `chroot()` and `makedev()` but not e.g. create bind mounts,
move `/usr/etc` to `/etc` when running scripts.

The script processing is also entangled with our passwd/group
file handling, so change those functions called from the core too.

It's tempting to basically maintain `/usr/etc` as `/etc` all
the way from immediately after checkout to just before commit.

We can't change how we do imports now; perhaps importing
RPMs into ostree as `usr/etc` was just a mistake in retrospect,
but oh well.

Closes: #1592
Approved by: jlebon
2018-10-10 13:24:47 +00:00
Colin Walters
a86ad96669 compose: Support not specifying a ref
Split out of supporting "pure rojig" work.  We also want
to support this for doing "oscontainers" as is planned for
Red Hat CoreOS.  The user experience in both cases is oriented
around versioning of the external wrapper, not the inner ref/commit.

Note for users/builders who want to make use of this feature:
You probably want to mirror the changes in our test suite here to
use the compose JSON and parse the resulting `ostree-commit` out of that.

Closes: #1603
Approved by: jlebon
2018-10-09 19:47:25 +00:00
Micah Abbott
51e90eaf8d man: add note about pinned deployments when using cleanup
Closes: #1599
Approved by: cgwalters
2018-10-04 16:01:07 +00:00
Colin Walters
e3be475566 compose+rust: Parse includes via Rust too
This follows up to https://github.com/projectatomic/rpm-ostree/pull/1576
AKA commit 2e567840ca - we now process
treefile inheritance in Rust code.  Previously for elements which
reference external files (`postprocess-script` and `add-files`)
we'd hardcoded things to only look in the first context dir.

Now we open file descriptors in the Rust side for these "externals"
as we're parsing, and load them C side.  Hence we'll correctly handle
a `postprocess-script` from an included config.

Other advantages are that the include handling was ugly un-typesafe C code
with no unit tests, now it's memory safe Rust with unit tests.

The downside here is I ended up spelling out the list of fields
again - there's probably a way to unify this via macros but
for now I think this is OK.

Closes: #1574
Approved by: jlebon
2018-10-04 13:17:47 +00:00
Colin Walters
638fab02bd tests: Add a test for inheritance order of postprocess
Since I got it backwards when rewriting it in Rust.

Closes: #1574
Approved by: jlebon
2018-10-04 13:17:47 +00:00
Jonathan Lebon
edb3edf9d4 upgrader: Output msg when staging deployment
It still takes a noticeable amount of time to stage a deployment, so
just give some feedback so we don't just hang there.

Closes: #1594
Approved by: cgwalters
2018-10-02 18:44:38 +00:00
Jonathan Lebon
efbe161d65 app/start-daemon: Drop unused callbacks
Noticed these while perusing the codebase. We don't do anything in those
handlers, and they're optional, so just nix them.

Closes: #1594
Approved by: cgwalters
2018-10-02 18:44:38 +00:00
Colin Walters
3c9e212f71 compose-tests: Use yaml.safe_dump
Otherwise we end up with weird `!!python/unicode` stuff:
https://stackoverflow.com/questions/20352794/pyyaml-is-producing-undesired-python-unicode-output/20369984

Closes: #1597
Approved by: jlebon
2018-10-02 18:03:22 +00:00
Jonathan Lebon
2e3d4b72a1 libpriv/scripts: Add rpmdb query sanity checks
Make sure we can open and query the rpmdb when creating new deployments.
This should help filter out cases where somehow librpm failed to
actually write the rpmdb but didn't error out.

This requires splitting the sanity checking in two so that we still get
that nice error first on scripts that do `rm -rf`.

See: #1566

Closes: #1584
Approved by: cgwalters
2018-10-02 14:54:53 +00:00
Jonathan Lebon
16b005bd56 ci: Use continuous image instead of rebasing to it
That should save us a bunch of time. We're also seeing some flakes right
now from rebasing.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Jonathan Lebon
0b14c18a7b ci/compose: Use host repos for building rpm-ostree too
We want to use the host repos not just for downloading the packages
we'll compose trees with, but also for the rpm-ostree build itself
leading up to the tests.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Jonathan Lebon
640c1d74c9 ci: Stop rebasing FAH on updates branch
The stable branch should have the updated librpm now.
Also bump slightly the compose tests timeout.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Colin Walters
363294c3ef rust/bindgen: Use #pragma once
Same as we do for our other headers.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Colin Walters
5d57025e16 build-sys: Add a rust-test target
I often am editing just the Rust code, and want the fast iteration
feedback of a `cargo test` - don't want to pay the cost of full
optimization (particularly LTO) for the release build, and I
*just* want to run the Rust tests.

Basically if you're editing our Rust code a lot, this target is
your friend.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Colin Walters
1cf8283904 rust: Make io::Read a generic again
No reason to do virtual dispatch here.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Colin Walters
426e16ee8b bwrap: Also drop constructor direct bwrap arguments
Following on to the previous change which added an explicit bind
mount API and also removed the API to append bwrap arguments, let's
also change the constructor to disallow the latter.

There was just one non-bind-mount argument being used
for "ro /var + tmpfs /var/tmp" - change that to an explicit new
API.

Looking at all of this, perhaps what we really want is to move
the "mutability" for `/var` underneath our API too, but let's
do this smaller incremental step first.

Closes: #1593
Approved by: jlebon
2018-09-28 21:07:49 +00:00
Colin Walters
87d934d421 bwrap: Add APIs to set up bind mounts
This is prep for adding a "backend" to this that uses just plain
`chroot` - we want things to avoid talking to bwrap's commandline
directly.

Closes: #1590
Approved by: jlebon
2018-09-28 18:03:19 +00:00
Jonathan Lebon
8e31c202da libpriv/postprocess: Handle ENOTEMPTY from renameat
While debugging test failures in #1584, I was perplexed to find that the
ex-container tests didn't work on current git master. It turns out we
were only checking for one of the two possible error codes in the case
where we rename to a non-empty dir. So why is CI getting `EEXIST` while
locally I get `ENOTEMPTY`? Doing some diving kernel-side revealed it's
due to xfs vs tmpfs.

Closes: #1589
Approved by: cgwalters
2018-09-28 17:19:00 +00:00
Colin Walters
7595bce228 rust/treefile: Use the c_utf8 crate
The advantage of this over CStr is that Rust knows it's UTF-8
too.  I also tweaked our path code to use String, and only
view it as a `Path`.  This avoids having to `unwrap()` later
back to a `str`.

Closes: #1588
Approved by: jlebon
2018-09-28 16:21:18 +00:00
Colin Walters
1966167839 build-sys: Inject Automake's default CXXFLAGS to libdnf by default
Today I was trying to use gdb and noticed my libdnf build didn't
have debuginfo.  Now, I thought that's what `-DCMAKE_BUILD_TYPE=RelWithDebugInfo`
was doing but...I have no idea right now where I got that.

I looked at RPM builds, and the way this works is it exports CXXFLAGS.
Now for our C code, our defaults actually come from Autoconf.  Let's
do the beautiful hack of telling Autoconf we're going to use C++ so
it sets `CXXFLAGS` for us.

Closes: #1586
Approved by: jlebon
2018-09-28 16:21:08 +00:00
Colin Walters
64d02e541c rust/treefile: Fix the type of add-files.
It's really an array of pairs.  Tripped over this while working
on the pure-Rust treefile parsing on the `misc-tweaks` compose test.

Closes: #1581
Approved by: jlebon
2018-09-26 20:15:30 +00:00
Colin Walters
c96f93f0ad build-sys: Propagate Automake verbose mode to Rust/cargo
This is analogous to commit c62058e548
which propagated `V=1` into `cmake`.  Except if the build *isn't*
verbose (for local development), let's not force `--verbose` on
for Rust.

Closes: #1583
Approved by: jlebon
2018-09-26 17:07:19 +00:00
Colin Walters
8df07a3b1e rust/treefile: Add support for parsing JSON too
Prep for moving more of our parsing into Rust.  The main
thing here is that for JSON, we need to continue to ignore
unknown fields.  It took me a little while but I eventually
figured out that using `#[serde(flatten)]` works for this.
Seriously: serde is freaking amazingly awesome.

Closes: #1580
Approved by: jlebon
2018-09-26 16:40:40 +00:00
Colin Walters
58e7c34823 rust: Run rustfmt
Prep for further changes.

Closes: #1580
Approved by: jlebon
2018-09-26 16:40:40 +00:00
Colin Walters
f50f9e8d7e Split cbindgen to separate build, support external version
The problem is building bindgen as part of our single run
locks serde to way old versions, and I want to use newer versions.

Since Fedora will now again ship a `cbindgen` package, let's
also support using it if we find it, saving ourselves
the cost of building it.

For distros that don't ship it (e.g. CentOS) for CI purposes
we build it.  For downstream builds that are offline, rather
than vendor the cbindgen sources like we do with our main Rust,
let's just vendor the `rpmostree-rust.h` file as was suggested
in https://bugzilla.redhat.com/show_bug.cgi?id=1608670

Closes: https://github.com/projectatomic/rpm-ostree/issues/1557

Closes: #1573
Approved by: jlebon
2018-09-25 20:29:21 +00:00
Jonathan Lebon
71588f9744 daemon/utils: Filter out rpmostree.rpmdb.pkglist
Drop the `rpmostree.rpmdb.pkglist` keys from the deployment metadata
since users shouldn't need it and it greatly increases the size of the
output.

Closes: #1577
Approved by: cgwalters
2018-09-25 15:20:57 +00:00
Colin Walters
b2408a466f core: Fix segfault if no selinux-policy package
In `--unified-core` mode, we would segfault if `selinux: true`
but no policy was found.

I hit this while hacking up the manifest for some Red Hat CoreOS
work.

Closes: #1578
Approved by: jlebon
2018-09-24 17:44:49 +00:00
Colin Walters
2e567840ca compose: Have first-one-wins semantics for rojig parsing
The core bug here is that previously if we had multiple YAML files
in include, we ended up overwriting self->treefile_rs for the
last one. Handling inheritance worked, but it broke rojig since
we generate the specfile Rust side.

Let's have first-one-wins semantics for now.  I have a bigger
fix incoming in https://github.com/projectatomic/rpm-ostree/pull/1574

Closes: #1576
Approved by: jlebon
2018-09-24 14:37:18 +00:00
Colin Walters
65bdc08917 postprocess: Make autovar.conf world-readable
No reason why not to; follows on the general principle that no
content in the OS should be secret.

https://bugzilla.redhat.com/show_bug.cgi?id=1631794

Closes: #1571
Approved by: jlebon
2018-09-21 15:56:52 +00:00
Colin Walters
c6de759cc0 compose: Drop combined ostree+rojig compose tree
Currently `compose tree` supports writing to both an OSTree
repository and optionally simultaneously generating a rojig
RPM or rojig set.

We are instead going to split off a separate `compose rojig` verb,
since we want to support a "pure rojig" mode, and having all
3 options would make this code far too complex.

For now anyone who wants *both* formats can use `compose tree` then
follow up with `ex commit2rojig`.

This is also inline with the new coreos-assembler push, where
we'll treat `rpm-ostree compose` as more of a "low level" API,
and wrap it in a more opinionated model there.

Closes: #1570
Approved by: jlebon
2018-09-21 14:42:14 +00:00
Colin Walters
42af11a6d6 compose: Add the timestamp to JSON metadata as ISO8601
In coreos-assembler I want to sort our builds by timestamp,
and having the ostree timestamp in the metadata is convenient.

Closes: #1569
Approved by: jlebon
2018-09-21 13:33:11 +00:00