Commit Graph

2500 Commits

Author SHA1 Message Date
Colin Walters
c6de759cc0 compose: Drop combined ostree+rojig compose tree
Currently `compose tree` supports writing to both an OSTree
repository and optionally simultaneously generating a rojig
RPM or rojig set.

We are instead going to split off a separate `compose rojig` verb,
since we want to support a "pure rojig" mode, and having all
3 options would make this code far too complex.

For now anyone who wants *both* formats can use `compose tree` then
follow up with `ex commit2rojig`.

This is also inline with the new coreos-assembler push, where
we'll treat `rpm-ostree compose` as more of a "low level" API,
and wrap it in a more opinionated model there.

Closes: #1570
Approved by: jlebon
2018-09-21 14:42:14 +00:00
Colin Walters
42af11a6d6 compose: Add the timestamp to JSON metadata as ISO8601
In coreos-assembler I want to sort our builds by timestamp,
and having the ostree timestamp in the metadata is convenient.

Closes: #1569
Approved by: jlebon
2018-09-21 13:33:11 +00:00
Jonathan Lebon
ea5bec6127 app/db-diff: Diff against rollback if no pending
Often, after rebooting from an upgrade, I want to check what was just
updated. This patch makes `db diff` do the right thing in those cases.
Specifically, before `db diff` without arguments would default to
diff'ing the pending deployment with the booted deployment and error out
otherwise. This patch extends the logic so that if there's a rollback
deployment, we default to diff'ing against that.

Closes: #1565
Approved by: cgwalters
2018-09-20 16:12:14 +00:00
Jonathan Lebon
e76c057b0c app/db-diff: Factor out helper printing function
Factor out a function for reading checksums from deployments and
printing their diffs. Also slightly rework the error-handling logic.
Prep for next commit.

Closes: #1565
Approved by: cgwalters
2018-09-20 16:12:14 +00:00
Jonathan Lebon
b27339f1bd app/db-diff: Drop redundant check
By the way `ostree_sysroot_query_deployments_for()` is defined, the
pending deployment is never the same as the booted deployment, so drop
that redundant check.

Closes: #1565
Approved by: cgwalters
2018-09-20 16:12:14 +00:00
Colin Walters
ca6f32371a compose: Split off composeutil.[ch]
Prep for adding a new toplevel rojig command.  It's also just
cleaner to avoid cluttering up the main compose logic with
distractions like the `_legacy_prep_dev()` bits.

Closes: #1564
Approved by: jlebon
2018-09-20 12:35:24 +00:00
Colin Walters
16f48afd5e commit2rojig: Take a (YAML) manifest, not a spec file
This completes the removal of externally-provided rojig spec files.

In the current flow then `compose tree --ex-rojig-output-*` will
end up parsing the manifest twice.  But this is prep for defaulting
rojig to "pure" mode without a separate ostree repo.  Then
anyone who wants to do both can run `commit2rojig` after.

Closes: #1561
Approved by: jlebon
2018-09-17 23:45:57 +00:00
Colin Walters
e208383526 tests: Add rojig spec into base config
This is prep for fully dropping the separate spec file, which
is now just used by `commit2rojig`.

A compose test can now specify it wants YAML instead of JSON.

Closes: #1561
Approved by: jlebon
2018-09-17 23:45:57 +00:00
Jonathan Lebon
f098d22e03 tests/vmcheck: Create correct base commits from layered
Our trick of using layered commits as base commit updates doesn't jive
well with the new opportunistic reuse of the base rpmdb. The issue is
that a layered commit includes the rpmdb of *its* base commit at the
`/usr/lib/sysimage` location. So to convert it into a proper base commit
means that the layered rpmdb should move there.

Closes: #1502
Approved by: cgwalters
2018-09-17 20:07:45 +00:00
Jonathan Lebon
a75460f538 upgrader: Reuse existing rpmdb checkout if available
Check if we can reuse the base rpmdb from the pending deployment if it
matches the base rev we're targeting. This allows us to avoid checking
out the tree early only to later on discard it. Such cases include
layering existing packages and inactive requests.

Closes: #1502
Approved by: cgwalters
2018-09-17 20:07:45 +00:00
Colin Walters
304f9f6a64 treefile: Redo rojig_spec to be Box<CStr>
Apparently I was just totally confused into thinking `OsStr`
was `NUL` terminated.  I don't know where I got that idea
but a quick inspection of the Rust source code makes this
clear.  I may submit a PR to the docs.

So we have two choices: allocate, or internally use a `NUL`-terminated
type.  Let's just use `Box<CStr>` internally.  Note I chose that
over `CString` as the latter is theoretically mutable, and when
passing data back over to C we don't have ownership/mutability
guarantees.

Closes: #1563
Approved by: jlebon
2018-09-17 15:51:53 +00:00
Colin Walters
d17381265e build-sys: Rework Rust --frozen logic
For local development, I want to be able to e.g. update `Cargo.toml`
or switch branches and not have to `rm -rf target`.

Let's tweak the logic here so we only pass `--frozen` if the ownership
of the dir is different.

Probably an even better fix would be to just error out, but
this is a conservative tweak.

Closes: #1563
Approved by: jlebon
2018-09-17 15:51:53 +00:00
Colin Walters
a7bbf5bc14 core: Stop overriding metadata_expire
See:
https://github.com/rpm-software-management/libdnf/pull/308
https://github.com/rpm-software-management/libdnf/pull/320

Closes: #1553

Closes: #1562
Approved by: cjao
2018-09-17 05:06:16 +00:00
Jonathan Lebon
7c570a2e8e libpriv/util: Add getter for layer version
Prep for another patch. Some callers will need to know whether some
layers contain certain fixes before proceeding. We already have that
information handy so just expose it.

We should probably do something about that massive argument list... will
return to that at some point. But at least with the new base layer
accessor from the previous patch, there aren't many callers affected by
the added argument.

Closes: #1560
Approved by: cgwalters
2018-09-14 16:49:41 +00:00
Jonathan Lebon
87ab4fafb6 libpriv: Add simpler base layer getter
In the majority of cases, we just want to know if a deployment is
layered and what the base commit is. Make a simpler accessor for this.
We can further simplify the logic on the client side here by relying on
the fact that there is only a base layer iif we're layered, so we don't
need two output variables.

I find it also makes things much easier to grok in some places, e.g. in
`rpmostreed-deployment-utils.c`, we now have:

    const char *current_base_checksum = current_base_checksum_owned ?: current_checksum;

which better crystallizes what we want to get at.

Or e.g. in the sysroot upgrader, we can drop a comment which states
something that is now completely obvious.

Closes: #1560
Approved by: cgwalters
2018-09-14 16:49:41 +00:00
Jonathan Lebon
897dd2839a postprocess: Don't copy base rpmdb when layering
We only want to copy our own rpmdb to the base rpmdb location in the
server-side paths. OTOH, when creating layered commits client-side, we
want to make sure that we *don't* touch the base rpmdb location.
Accordingly, move the logic that does the hardlinking into
`postprocess_final`.

This can cause issues if we want to make use of the base rpmdb (as in a
forthcoming patch) of commits that were not created by a recent enough
rpm-ostree which inject the base rpmdb. When layering on such base
commits, we would unwittingly copy the new rpmdb (with the layered
packages) into the base rpmdb location. This then of course throws off
any logic that assumes the rpmdb at that location is from the base
layer.

There's a slight compatibility issue here: there are still older layered
commits out there where we did this and which might throw off new logic
that relies on this. We bump the clientlayer version so that we can just
check for that version to know whether it's safe to use the base rpmdb.

Closes: #1560
Approved by: cgwalters
2018-09-14 16:49:41 +00:00
Colin Walters
02b597b690 rust/bindgen: Add an autoptr cleanup for our struct
I'm going to add a new user soon.  This should probably
be a best practice.

Ideally...cbindgen would have a way to add some wrapper bits
around any types it defines or something?

Closes: #1559
Approved by: jlebon
2018-09-13 21:49:06 +00:00
Colin Walters
36bf3c9164 build: Regenerate rpmostree-rust.h if cbindgen.toml changes
So when I edit it I get the changes.  Figuring this out
took me a little while, we were missing two dependencies.

Closes: #1559
Approved by: jlebon
2018-09-13 21:49:06 +00:00
Colin Walters
8df776e367 compose: Remove support for ex-rojig-spec in treefile
In https://github.com/projectatomic/rpm-ostree/pull/1484
AKA commit 235e8f82da
we introduced a new `rojig` treefile entry that is nicer and
more maintainable than a split spec file.

In prep for more "pure rojig" work, drop support for this key.
Note we can also only now do this because we hard require Rust, and
the new model is only implemented in Rust.

The only known user of this today is FAHC, but that is now
[pinned to an assembler build](https://pagure.io/fedora-atomic-host-continuous/c/cdcb6ae943bee643983d4bd22d00030f2eb930d2?branch=master).

Closes: #1558
Approved by: jlebon
2018-09-13 21:10:21 +00:00
Robert Fairley
b82db2b501 initramfs: Return error when --arg specified without --enable
This change gives an error when --arg is specified on its own.

Previously, if adding --arg without --enable, no changes occur and
only the initramfs regeneration status is printed. Since --arg only
takes effect when used with --enable, this should also give an error
(or let the user know in some way).

Closes: #1548
Approved by: jlebon
2018-09-12 20:45:19 +00:00
Robert Fairley
be7d87e4b9 initramfs: Print regeneration status for pending deployment
Previously, after enabling initramfs regeneration, the initramfs
command would indicate that initramfs regeneratin is disabled,
suggesting that the initramfs had not actually been enabled. This
happened because the initramfs regeneration status for the current
deployment was printed; initramfs regeneration is only enabled for a
pending deployment.

This change prints the regeneration for the pending deployment,
rather than the current, when no options are given to the initramfs
command.

Fixes: #1526

Closes: #1548
Approved by: jlebon
2018-09-12 20:45:19 +00:00
Micah Abbott
bce966a981 vmcheck/misc-1: skip the overlay check when needed
It's possible to run the `vmcheck` tests against an existing host that
has `rpm-ostree` already present.  We don't overlay the built binaries
in this situation, so we should not check for the presence of the
overlay in the commit meta.

Closes: #1555
Approved by: cgwalters
2018-09-12 14:48:02 +00:00
Jonathan Lebon
807f21788e Hard require staging
This removes the logic around supporting opting out of the staging
feature. We don't want to support multiple configurations here, and at
this point, staging should be considered stable.

Closes: #1546
Approved by: cgwalters
2018-09-11 20:55:48 +00:00
Colin Walters
0d2b819e19 compose: Fix double-parsing of first included treefile
This is [brown paper bag bug](http://www.catb.org/jargon/html/B/brown-paper-bag-bug.html)
for sure.  But it's existed forever.  Basically our recursion
when parsing `include:` ended up parsing the first included treefile
twice.

I noticed this with the new `postprocess` element - luckily
my script was idempotent.  Historically we've gotten away with
this probably because the main thing is the `packages` array
and duplicates don't matter there.

Closes: #1552
Approved by: jlebon
2018-09-11 18:53:35 +00:00
Colin Walters
66d127ddd7 compose: Update to decl-and-init coding style
No functional changes, prep for more work.

Closes: #1552
Approved by: jlebon
2018-09-11 18:53:35 +00:00
Colin Walters
23698a8aa3 rust: Use debug=true for release builds
Because our primary delivery vechicle uses RPM which
supports separate debuginfo, there's no reason not to build release
builds with debuginfo, so that one can use gdb on them just
like we do with C.

Closes: #1550
Approved by: jlebon
2018-09-11 16:28:15 +00:00
Colin Walters
08ad9d49c8 ci: Stop installing python-{sphinx,devel}
We should no longer need it. Just noticed this while I was in the file recently.

Closes: #1551
Approved by: jlebon
2018-09-11 15:35:32 +00:00
Jonathan Lebon
57c4f8acff packaging: Don't vendor bundled libcurl
The `curl-sys` crate includes with it a bundled copy of libcurl which is
used if `pkgconfig` doesn't find libcurl configuration files. In our
case, we always want to use the system libcurl. So filter it out. This
also drops our *compressed* tarball by 2.5M.

One tricky bit is that cargo crates include a checksum JSON that's read
by `cargo build` later on to validate the crate. So we need to do some
JSON surgery.

What made me look into this was that Koji builds were failing due to the
`%configure` macro including hardening bits that sub out e.g. all
`config.sub` and `ltmain.sh` files which then caused the checksum to
fail validation. This completely sidesteps that issue.

Closes: #1554
Approved by: cgwalters
2018-09-11 14:44:44 +00:00
Jonathan Lebon
7467952c96 packaging: Use standard strict mode header
The script already turned on `-e` later on. Make it conform to the
standard strict mode with the rest of the options right at the
beginning.

Closes: #1554
Approved by: cgwalters
2018-09-11 14:44:44 +00:00
Jonathan Lebon
772f29282e Release 2018.8
Closes: #1547
Approved by: cgwalters
2018-09-10 17:59:21 +00:00
Colin Walters
5927d85192 tests: Rename one libcomposetest.sh
We had two `libcomposetest.sh` which I always found confusing.
Fix the naming of the one that's shared with `ex-container`
to be more obvious.

Closes: #1543
Approved by: jlebon
2018-09-10 17:06:10 +00:00
Jonathan Lebon
8976e502fc daemon: Make inactive requests a hard error
Flip the switch and make `rpm-ostree install glibc` a hard error if
`--allow-inactive` is not provided.

Closes: #1545
Approved by: cgwalters
2018-09-10 16:17:14 +00:00
Colin Walters
fe87ad4f29 tests: Drop conditionals for Rust enablement
We now hard require it, so change these tests to do so as
well.

Closes: #1544
Approved by: jlebon
2018-09-10 13:19:51 +00:00
Jonathan Lebon
651a9e01f4 core: Accumulate all missing pkgs into a single error
Accumulate the list of requested packages that were not found and error
out with the full list rather than failing early. This fixes a small UX
papercut in certain situations.

Closes: #1540

Closes: #1541
Approved by: cgwalters
2018-09-08 23:32:13 +00:00
Jonathan Lebon
f6c0788b4e ci: Use FAH updates branch for newer librpm
Right now, we're compiling against librpm 4.14.2, but running against
4.14.1, so we're hitting missing symbol issues from #1469. Just use the
updates ref for now until the next stable FAH update.

Closes: #1538
Approved by: cgwalters
2018-09-08 21:51:22 +00:00
Colin Walters
af5c74f9ea app/main: Give an immediate error if no /run/ostree-booted
Let's be more graceful here for people who `yum install rpm-ostree`
and expect magical[1] things to happen.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1537

[1] Understandably if one doesn't know lots of background

Closes: #1538
Approved by: cgwalters
2018-09-08 21:51:22 +00:00
Colin Walters
6f6dab1092 compose: Do import with --unified-core and --download-only
No reason to defer it to build time.

Closes: #1535
Approved by: jlebon
2018-09-07 20:24:11 +00:00
Colin Walters
326a9e0391 importer: Don't generate var-tmpfiles.d for rpm itself
This caused `/var/lib/rpm` to be a directory in my Fedora CoreOS
builds.

Closes: #1533
Approved by: jlebon
2018-09-07 19:41:48 +00:00
Colin Walters
6f6728b992 compose: Add --write-composejson-to
For higher level tools driving rpm-ostree, the command line arguments
are mostly OK as inputs, but the addition of `--write-commitid-to`
shows that we really want structured data that these tools can parse.
JSON is a good enough interchange format, let's use that.

Most notably this does a pkgdiff in the JSON which several higher
level tools do too.

Closes: #1529
Approved by: jlebon
2018-09-07 18:52:54 +00:00
Colin Walters
d2aee36f8f compose: Don't print message about --force-nocache with --dry-run
I'm splitting up coreos-assembler to have separate fetch/build
stages, with the first using `--download-only` and the second
using `--cache-only`, and the message from `fetch` is basically
wrong.

Closes: #1534
Approved by: jlebon
2018-09-07 18:09:31 +00:00
Colin Walters
b9e3bfd7b3 importer: Don't generate var-tmpfiles.d for rpm itself
This caused `/var/lib/rpm` to be a directory in my Fedora CoreOS
builds.

Closes: #1532
Approved by: sinnykumari
2018-09-07 15:10:10 +00:00
Colin Walters
1ee2a62343 scripts: Workaround Lua in fedora-release-coreos too
Just like the other ones from commit: 716b60b7d7

Yes, we probably should just patch the package but then
we'd have to wait for all of the koji/bodhi pain.

Closes: #1532
Approved by: sinnykumari
2018-09-07 15:10:10 +00:00
Jonathan Lebon
86f2b67d2b vmcheck: Store temporary files in /var/tmp instead
Since `/tmp` might be on tmpfs, so we'd lose it on reboot. But we have
tests that need it to persist across reboots.

Closes: #1531
Approved by: miabbott
2018-09-07 14:20:01 +00:00
Micah Abbott
c9b4a8195e tests/vmcheck: handle missing /etc/yum.repos.d
It's possible to run the `vmcheck` tests against an existing VM which
may not have any existing `/etc/yum.repos.d` directory.  Since the
tests are providing their own repo files, we should be able to run
them in this uncommon configuration.

Closes: #1530
Approved by: jlebon
2018-09-06 20:50:25 +00:00
Colin Walters
961482f088 Add inputhash Provides to rojig RPMs
Prep for rojig-only compose support.  We want to be able
to quickly detect whether or not there were changes in the
input, just like we do when targeting an OSTree repository.

Closes: #1528
Approved by: jlebon
2018-09-06 14:33:48 +00:00
Colin Walters
0a82e39052 compose: Fix workdir tmp dfd reference
If a `--workdir` was explicitly provided, we would pass `-1`
down into the Rust libs.

I found this while working on rojig-only which is reworking
the workdir bits.

Closes: #1528
Approved by: jlebon
2018-09-06 14:33:48 +00:00
Colin Walters
d2b0e42bfc Use cbindgen
If we're going to scale out our oxidation, let's follow
the path of Firefox (and other projects) further and use
cbindgen: https://github.com/eqrion/cbindgen

It's actually nice that `cbindgen` is packaged today in Fedora,
but I doubt it is elsewhere; we may end up needing to push
that forward, or just vendor it via a `build.rs` script and Cargo.

I chose to rename things to `ROR`/`ror_` since it's shorter.  I
am tempted a bit to rename our internal functions to just `ro_` to
or so.

Closes: #1516
Approved by: jlebon
2018-09-06 13:34:32 +00:00
Colin Walters
279e7c4f1b treefile: Support inline postprocess element (for inheritance)
I'm trying to have a more opinionated model where custom builds
use inheritance, and currently one can only have a single
`postprocess-script`.

Further, in YAML it's very convenient to use inline vs external
data.

Closes: #1527
Approved by: jlebon
2018-09-05 15:27:07 +00:00
Ruixin Bao
9920094cfa tests: add test for sorting + conversion of sysuser
As title, added tests to check both conversion + sorting for sysuser
entries, and verifies its correctness.

Closes: #1519
Approved by: cgwalters
2018-08-30 17:37:27 +00:00
Ruixin Bao
6facef9733 passwd-util: adds sorting logic for sysuser_entry
Per tile, added logic for sysusers_sorting. The comparison
for each sysuer_ent is like the following:
1: g > u > m for sysent_type and this takes precedence
2: if type matches, we compare the name for sysent using strcmp
3: If both type and name matches, we want to error out because it indicates a duplicate entry

Closes: #1519
Approved by: cgwalters
2018-08-30 17:37:27 +00:00