Commit Graph

2615 Commits

Author SHA1 Message Date
Colin Walters
1cf8283904 rust: Make io::Read a generic again
No reason to do virtual dispatch here.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Colin Walters
426e16ee8b bwrap: Also drop constructor direct bwrap arguments
Following on to the previous change which added an explicit bind
mount API and also removed the API to append bwrap arguments, let's
also change the constructor to disallow the latter.

There was just one non-bind-mount argument being used
for "ro /var + tmpfs /var/tmp" - change that to an explicit new
API.

Looking at all of this, perhaps what we really want is to move
the "mutability" for `/var` underneath our API too, but let's
do this smaller incremental step first.

Closes: #1593
Approved by: jlebon
2018-09-28 21:07:49 +00:00
Colin Walters
87d934d421 bwrap: Add APIs to set up bind mounts
This is prep for adding a "backend" to this that uses just plain
`chroot` - we want things to avoid talking to bwrap's commandline
directly.

Closes: #1590
Approved by: jlebon
2018-09-28 18:03:19 +00:00
Jonathan Lebon
8e31c202da libpriv/postprocess: Handle ENOTEMPTY from renameat
While debugging test failures in #1584, I was perplexed to find that the
ex-container tests didn't work on current git master. It turns out we
were only checking for one of the two possible error codes in the case
where we rename to a non-empty dir. So why is CI getting `EEXIST` while
locally I get `ENOTEMPTY`? Doing some diving kernel-side revealed it's
due to xfs vs tmpfs.

Closes: #1589
Approved by: cgwalters
2018-09-28 17:19:00 +00:00
Colin Walters
7595bce228 rust/treefile: Use the c_utf8 crate
The advantage of this over CStr is that Rust knows it's UTF-8
too.  I also tweaked our path code to use String, and only
view it as a `Path`.  This avoids having to `unwrap()` later
back to a `str`.

Closes: #1588
Approved by: jlebon
2018-09-28 16:21:18 +00:00
Colin Walters
1966167839 build-sys: Inject Automake's default CXXFLAGS to libdnf by default
Today I was trying to use gdb and noticed my libdnf build didn't
have debuginfo.  Now, I thought that's what `-DCMAKE_BUILD_TYPE=RelWithDebugInfo`
was doing but...I have no idea right now where I got that.

I looked at RPM builds, and the way this works is it exports CXXFLAGS.
Now for our C code, our defaults actually come from Autoconf.  Let's
do the beautiful hack of telling Autoconf we're going to use C++ so
it sets `CXXFLAGS` for us.

Closes: #1586
Approved by: jlebon
2018-09-28 16:21:08 +00:00
Colin Walters
64d02e541c rust/treefile: Fix the type of add-files.
It's really an array of pairs.  Tripped over this while working
on the pure-Rust treefile parsing on the `misc-tweaks` compose test.

Closes: #1581
Approved by: jlebon
2018-09-26 20:15:30 +00:00
Colin Walters
c96f93f0ad build-sys: Propagate Automake verbose mode to Rust/cargo
This is analogous to commit c62058e548
which propagated `V=1` into `cmake`.  Except if the build *isn't*
verbose (for local development), let's not force `--verbose` on
for Rust.

Closes: #1583
Approved by: jlebon
2018-09-26 17:07:19 +00:00
Colin Walters
8df07a3b1e rust/treefile: Add support for parsing JSON too
Prep for moving more of our parsing into Rust.  The main
thing here is that for JSON, we need to continue to ignore
unknown fields.  It took me a little while but I eventually
figured out that using `#[serde(flatten)]` works for this.
Seriously: serde is freaking amazingly awesome.

Closes: #1580
Approved by: jlebon
2018-09-26 16:40:40 +00:00
Colin Walters
58e7c34823 rust: Run rustfmt
Prep for further changes.

Closes: #1580
Approved by: jlebon
2018-09-26 16:40:40 +00:00
Colin Walters
f50f9e8d7e Split cbindgen to separate build, support external version
The problem is building bindgen as part of our single run
locks serde to way old versions, and I want to use newer versions.

Since Fedora will now again ship a `cbindgen` package, let's
also support using it if we find it, saving ourselves
the cost of building it.

For distros that don't ship it (e.g. CentOS) for CI purposes
we build it.  For downstream builds that are offline, rather
than vendor the cbindgen sources like we do with our main Rust,
let's just vendor the `rpmostree-rust.h` file as was suggested
in https://bugzilla.redhat.com/show_bug.cgi?id=1608670

Closes: https://github.com/projectatomic/rpm-ostree/issues/1557

Closes: #1573
Approved by: jlebon
2018-09-25 20:29:21 +00:00
Jonathan Lebon
71588f9744 daemon/utils: Filter out rpmostree.rpmdb.pkglist
Drop the `rpmostree.rpmdb.pkglist` keys from the deployment metadata
since users shouldn't need it and it greatly increases the size of the
output.

Closes: #1577
Approved by: cgwalters
2018-09-25 15:20:57 +00:00
Colin Walters
b2408a466f core: Fix segfault if no selinux-policy package
In `--unified-core` mode, we would segfault if `selinux: true`
but no policy was found.

I hit this while hacking up the manifest for some Red Hat CoreOS
work.

Closes: #1578
Approved by: jlebon
2018-09-24 17:44:49 +00:00
Colin Walters
2e567840ca compose: Have first-one-wins semantics for rojig parsing
The core bug here is that previously if we had multiple YAML files
in include, we ended up overwriting self->treefile_rs for the
last one. Handling inheritance worked, but it broke rojig since
we generate the specfile Rust side.

Let's have first-one-wins semantics for now.  I have a bigger
fix incoming in https://github.com/projectatomic/rpm-ostree/pull/1574

Closes: #1576
Approved by: jlebon
2018-09-24 14:37:18 +00:00
Colin Walters
65bdc08917 postprocess: Make autovar.conf world-readable
No reason why not to; follows on the general principle that no
content in the OS should be secret.

https://bugzilla.redhat.com/show_bug.cgi?id=1631794

Closes: #1571
Approved by: jlebon
2018-09-21 15:56:52 +00:00
Colin Walters
c6de759cc0 compose: Drop combined ostree+rojig compose tree
Currently `compose tree` supports writing to both an OSTree
repository and optionally simultaneously generating a rojig
RPM or rojig set.

We are instead going to split off a separate `compose rojig` verb,
since we want to support a "pure rojig" mode, and having all
3 options would make this code far too complex.

For now anyone who wants *both* formats can use `compose tree` then
follow up with `ex commit2rojig`.

This is also inline with the new coreos-assembler push, where
we'll treat `rpm-ostree compose` as more of a "low level" API,
and wrap it in a more opinionated model there.

Closes: #1570
Approved by: jlebon
2018-09-21 14:42:14 +00:00
Colin Walters
42af11a6d6 compose: Add the timestamp to JSON metadata as ISO8601
In coreos-assembler I want to sort our builds by timestamp,
and having the ostree timestamp in the metadata is convenient.

Closes: #1569
Approved by: jlebon
2018-09-21 13:33:11 +00:00
Jonathan Lebon
ea5bec6127 app/db-diff: Diff against rollback if no pending
Often, after rebooting from an upgrade, I want to check what was just
updated. This patch makes `db diff` do the right thing in those cases.
Specifically, before `db diff` without arguments would default to
diff'ing the pending deployment with the booted deployment and error out
otherwise. This patch extends the logic so that if there's a rollback
deployment, we default to diff'ing against that.

Closes: #1565
Approved by: cgwalters
2018-09-20 16:12:14 +00:00
Jonathan Lebon
e76c057b0c app/db-diff: Factor out helper printing function
Factor out a function for reading checksums from deployments and
printing their diffs. Also slightly rework the error-handling logic.
Prep for next commit.

Closes: #1565
Approved by: cgwalters
2018-09-20 16:12:14 +00:00
Jonathan Lebon
b27339f1bd app/db-diff: Drop redundant check
By the way `ostree_sysroot_query_deployments_for()` is defined, the
pending deployment is never the same as the booted deployment, so drop
that redundant check.

Closes: #1565
Approved by: cgwalters
2018-09-20 16:12:14 +00:00
Colin Walters
ca6f32371a compose: Split off composeutil.[ch]
Prep for adding a new toplevel rojig command.  It's also just
cleaner to avoid cluttering up the main compose logic with
distractions like the `_legacy_prep_dev()` bits.

Closes: #1564
Approved by: jlebon
2018-09-20 12:35:24 +00:00
Colin Walters
16f48afd5e commit2rojig: Take a (YAML) manifest, not a spec file
This completes the removal of externally-provided rojig spec files.

In the current flow then `compose tree --ex-rojig-output-*` will
end up parsing the manifest twice.  But this is prep for defaulting
rojig to "pure" mode without a separate ostree repo.  Then
anyone who wants to do both can run `commit2rojig` after.

Closes: #1561
Approved by: jlebon
2018-09-17 23:45:57 +00:00
Colin Walters
e208383526 tests: Add rojig spec into base config
This is prep for fully dropping the separate spec file, which
is now just used by `commit2rojig`.

A compose test can now specify it wants YAML instead of JSON.

Closes: #1561
Approved by: jlebon
2018-09-17 23:45:57 +00:00
Jonathan Lebon
f098d22e03 tests/vmcheck: Create correct base commits from layered
Our trick of using layered commits as base commit updates doesn't jive
well with the new opportunistic reuse of the base rpmdb. The issue is
that a layered commit includes the rpmdb of *its* base commit at the
`/usr/lib/sysimage` location. So to convert it into a proper base commit
means that the layered rpmdb should move there.

Closes: #1502
Approved by: cgwalters
2018-09-17 20:07:45 +00:00
Jonathan Lebon
a75460f538 upgrader: Reuse existing rpmdb checkout if available
Check if we can reuse the base rpmdb from the pending deployment if it
matches the base rev we're targeting. This allows us to avoid checking
out the tree early only to later on discard it. Such cases include
layering existing packages and inactive requests.

Closes: #1502
Approved by: cgwalters
2018-09-17 20:07:45 +00:00
Colin Walters
304f9f6a64 treefile: Redo rojig_spec to be Box<CStr>
Apparently I was just totally confused into thinking `OsStr`
was `NUL` terminated.  I don't know where I got that idea
but a quick inspection of the Rust source code makes this
clear.  I may submit a PR to the docs.

So we have two choices: allocate, or internally use a `NUL`-terminated
type.  Let's just use `Box<CStr>` internally.  Note I chose that
over `CString` as the latter is theoretically mutable, and when
passing data back over to C we don't have ownership/mutability
guarantees.

Closes: #1563
Approved by: jlebon
2018-09-17 15:51:53 +00:00
Colin Walters
d17381265e build-sys: Rework Rust --frozen logic
For local development, I want to be able to e.g. update `Cargo.toml`
or switch branches and not have to `rm -rf target`.

Let's tweak the logic here so we only pass `--frozen` if the ownership
of the dir is different.

Probably an even better fix would be to just error out, but
this is a conservative tweak.

Closes: #1563
Approved by: jlebon
2018-09-17 15:51:53 +00:00
Colin Walters
a7bbf5bc14 core: Stop overriding metadata_expire
See:
https://github.com/rpm-software-management/libdnf/pull/308
https://github.com/rpm-software-management/libdnf/pull/320

Closes: #1553

Closes: #1562
Approved by: cjao
2018-09-17 05:06:16 +00:00
Jonathan Lebon
7c570a2e8e libpriv/util: Add getter for layer version
Prep for another patch. Some callers will need to know whether some
layers contain certain fixes before proceeding. We already have that
information handy so just expose it.

We should probably do something about that massive argument list... will
return to that at some point. But at least with the new base layer
accessor from the previous patch, there aren't many callers affected by
the added argument.

Closes: #1560
Approved by: cgwalters
2018-09-14 16:49:41 +00:00
Jonathan Lebon
87ab4fafb6 libpriv: Add simpler base layer getter
In the majority of cases, we just want to know if a deployment is
layered and what the base commit is. Make a simpler accessor for this.
We can further simplify the logic on the client side here by relying on
the fact that there is only a base layer iif we're layered, so we don't
need two output variables.

I find it also makes things much easier to grok in some places, e.g. in
`rpmostreed-deployment-utils.c`, we now have:

    const char *current_base_checksum = current_base_checksum_owned ?: current_checksum;

which better crystallizes what we want to get at.

Or e.g. in the sysroot upgrader, we can drop a comment which states
something that is now completely obvious.

Closes: #1560
Approved by: cgwalters
2018-09-14 16:49:41 +00:00
Jonathan Lebon
897dd2839a postprocess: Don't copy base rpmdb when layering
We only want to copy our own rpmdb to the base rpmdb location in the
server-side paths. OTOH, when creating layered commits client-side, we
want to make sure that we *don't* touch the base rpmdb location.
Accordingly, move the logic that does the hardlinking into
`postprocess_final`.

This can cause issues if we want to make use of the base rpmdb (as in a
forthcoming patch) of commits that were not created by a recent enough
rpm-ostree which inject the base rpmdb. When layering on such base
commits, we would unwittingly copy the new rpmdb (with the layered
packages) into the base rpmdb location. This then of course throws off
any logic that assumes the rpmdb at that location is from the base
layer.

There's a slight compatibility issue here: there are still older layered
commits out there where we did this and which might throw off new logic
that relies on this. We bump the clientlayer version so that we can just
check for that version to know whether it's safe to use the base rpmdb.

Closes: #1560
Approved by: cgwalters
2018-09-14 16:49:41 +00:00
Colin Walters
02b597b690 rust/bindgen: Add an autoptr cleanup for our struct
I'm going to add a new user soon.  This should probably
be a best practice.

Ideally...cbindgen would have a way to add some wrapper bits
around any types it defines or something?

Closes: #1559
Approved by: jlebon
2018-09-13 21:49:06 +00:00
Colin Walters
36bf3c9164 build: Regenerate rpmostree-rust.h if cbindgen.toml changes
So when I edit it I get the changes.  Figuring this out
took me a little while, we were missing two dependencies.

Closes: #1559
Approved by: jlebon
2018-09-13 21:49:06 +00:00
Colin Walters
8df776e367 compose: Remove support for ex-rojig-spec in treefile
In https://github.com/projectatomic/rpm-ostree/pull/1484
AKA commit 235e8f82da
we introduced a new `rojig` treefile entry that is nicer and
more maintainable than a split spec file.

In prep for more "pure rojig" work, drop support for this key.
Note we can also only now do this because we hard require Rust, and
the new model is only implemented in Rust.

The only known user of this today is FAHC, but that is now
[pinned to an assembler build](https://pagure.io/fedora-atomic-host-continuous/c/cdcb6ae943bee643983d4bd22d00030f2eb930d2?branch=master).

Closes: #1558
Approved by: jlebon
2018-09-13 21:10:21 +00:00
Robert Fairley
b82db2b501 initramfs: Return error when --arg specified without --enable
This change gives an error when --arg is specified on its own.

Previously, if adding --arg without --enable, no changes occur and
only the initramfs regeneration status is printed. Since --arg only
takes effect when used with --enable, this should also give an error
(or let the user know in some way).

Closes: #1548
Approved by: jlebon
2018-09-12 20:45:19 +00:00
Robert Fairley
be7d87e4b9 initramfs: Print regeneration status for pending deployment
Previously, after enabling initramfs regeneration, the initramfs
command would indicate that initramfs regeneratin is disabled,
suggesting that the initramfs had not actually been enabled. This
happened because the initramfs regeneration status for the current
deployment was printed; initramfs regeneration is only enabled for a
pending deployment.

This change prints the regeneration for the pending deployment,
rather than the current, when no options are given to the initramfs
command.

Fixes: #1526

Closes: #1548
Approved by: jlebon
2018-09-12 20:45:19 +00:00
Micah Abbott
bce966a981 vmcheck/misc-1: skip the overlay check when needed
It's possible to run the `vmcheck` tests against an existing host that
has `rpm-ostree` already present.  We don't overlay the built binaries
in this situation, so we should not check for the presence of the
overlay in the commit meta.

Closes: #1555
Approved by: cgwalters
2018-09-12 14:48:02 +00:00
Jonathan Lebon
807f21788e Hard require staging
This removes the logic around supporting opting out of the staging
feature. We don't want to support multiple configurations here, and at
this point, staging should be considered stable.

Closes: #1546
Approved by: cgwalters
2018-09-11 20:55:48 +00:00
Colin Walters
0d2b819e19 compose: Fix double-parsing of first included treefile
This is [brown paper bag bug](http://www.catb.org/jargon/html/B/brown-paper-bag-bug.html)
for sure.  But it's existed forever.  Basically our recursion
when parsing `include:` ended up parsing the first included treefile
twice.

I noticed this with the new `postprocess` element - luckily
my script was idempotent.  Historically we've gotten away with
this probably because the main thing is the `packages` array
and duplicates don't matter there.

Closes: #1552
Approved by: jlebon
2018-09-11 18:53:35 +00:00
Colin Walters
66d127ddd7 compose: Update to decl-and-init coding style
No functional changes, prep for more work.

Closes: #1552
Approved by: jlebon
2018-09-11 18:53:35 +00:00
Colin Walters
23698a8aa3 rust: Use debug=true for release builds
Because our primary delivery vechicle uses RPM which
supports separate debuginfo, there's no reason not to build release
builds with debuginfo, so that one can use gdb on them just
like we do with C.

Closes: #1550
Approved by: jlebon
2018-09-11 16:28:15 +00:00
Colin Walters
08ad9d49c8 ci: Stop installing python-{sphinx,devel}
We should no longer need it. Just noticed this while I was in the file recently.

Closes: #1551
Approved by: jlebon
2018-09-11 15:35:32 +00:00
Jonathan Lebon
57c4f8acff packaging: Don't vendor bundled libcurl
The `curl-sys` crate includes with it a bundled copy of libcurl which is
used if `pkgconfig` doesn't find libcurl configuration files. In our
case, we always want to use the system libcurl. So filter it out. This
also drops our *compressed* tarball by 2.5M.

One tricky bit is that cargo crates include a checksum JSON that's read
by `cargo build` later on to validate the crate. So we need to do some
JSON surgery.

What made me look into this was that Koji builds were failing due to the
`%configure` macro including hardening bits that sub out e.g. all
`config.sub` and `ltmain.sh` files which then caused the checksum to
fail validation. This completely sidesteps that issue.

Closes: #1554
Approved by: cgwalters
2018-09-11 14:44:44 +00:00
Jonathan Lebon
7467952c96 packaging: Use standard strict mode header
The script already turned on `-e` later on. Make it conform to the
standard strict mode with the rest of the options right at the
beginning.

Closes: #1554
Approved by: cgwalters
2018-09-11 14:44:44 +00:00
Jonathan Lebon
772f29282e Release 2018.8
Closes: #1547
Approved by: cgwalters
2018-09-10 17:59:21 +00:00
Colin Walters
5927d85192 tests: Rename one libcomposetest.sh
We had two `libcomposetest.sh` which I always found confusing.
Fix the naming of the one that's shared with `ex-container`
to be more obvious.

Closes: #1543
Approved by: jlebon
2018-09-10 17:06:10 +00:00
Jonathan Lebon
8976e502fc daemon: Make inactive requests a hard error
Flip the switch and make `rpm-ostree install glibc` a hard error if
`--allow-inactive` is not provided.

Closes: #1545
Approved by: cgwalters
2018-09-10 16:17:14 +00:00
Colin Walters
fe87ad4f29 tests: Drop conditionals for Rust enablement
We now hard require it, so change these tests to do so as
well.

Closes: #1544
Approved by: jlebon
2018-09-10 13:19:51 +00:00
Jonathan Lebon
651a9e01f4 core: Accumulate all missing pkgs into a single error
Accumulate the list of requested packages that were not found and error
out with the full list rather than failing early. This fixes a small UX
papercut in certain situations.

Closes: #1540

Closes: #1541
Approved by: cgwalters
2018-09-08 23:32:13 +00:00
Jonathan Lebon
f6c0788b4e ci: Use FAH updates branch for newer librpm
Right now, we're compiling against librpm 4.14.2, but running against
4.14.1, so we're hitting missing symbol issues from #1469. Just use the
updates ref for now until the next stable FAH update.

Closes: #1538
Approved by: cgwalters
2018-09-08 21:51:22 +00:00