Commit Graph

2214 Commits

Author SHA1 Message Date
Colin Walters
588a0327db postprocess: Error if units with machineid-compat: false
Rather than silently ignoring it.

In theory...we could write to /usr/lib/systemd instead of `/etc`
but eh...I feel like what we really want to do is make it convenient
to write a preset file from the YAML.

(We could have an `add-files` content that takes values literally
 which would be nice in YAML and suck in JSON)

A general thread running through this is that for people making
*derivatives* of a CoreOS-like system, having to create their
own `exampleos-release` package is an annoying hurdle.

Anyways for now we're fixing the bug that we were silently ignoring
it.

Closes: #1488
Approved by: jlebon
2018-08-06 00:33:56 +00:00
Jonathan Lebon
be055179db docs/treefile: Mention rw in for machineid-compat option
See discussions in https://github.com/systemd/systemd/issues/8268 for
more information.

Closes: #1487
Approved by: cgwalters
2018-08-03 16:54:55 +00:00
Colin Walters
344aee1d76 rust: Add support for inline rojig spec files
The rojig spec is almost entirely rpm-ostree implementation details;
let's not have lots of people fork/duplicate it.  Rather add the bits
of rojig to the treefile that people need to define (most notably
the name).

Prep for stabilizing rojig.

I had a few false starts with this PR; managing ownership/lifetimes
across C/Rust is just complicated.  I got bit hard by the fact that
the workdir in `--unified-core` is really dfd-relative, and had to
do a dance to propagate the dfd into rust, as well as down into
the rojig builder.

Closes: #1484
Approved by: jlebon
2018-08-03 16:54:47 +00:00
Jonathan Lebon
a6e4994bf6 app/compose: Fix g_propagate_error ownership
We weren't actually transferring ownership of the error, which meant
that the error message was freed before we could even print it to the
user.

Closes: #1486
Approved by: cgwalters
2018-08-01 23:27:12 +00:00
Colin Walters
dcbd2d91ec compose: Stop calling fchdir in --unified-core mode
Changing the process' working directory is evil in potentially
multi-threaded code, and at this point we really should be using
`openat()`/fd-relative bits most everywhere.  But let's be
conservative and only stop doing it in `--unified-core` mode.

Closes: #1485
Approved by: jlebon
2018-08-01 22:22:47 +00:00
Colin Walters
23badcd288 compose: Write a preset file to enable ostree-remount.service
I've lost count now of how many times people have hit variants
of https://github.com/projectatomic/centos-release-atomic-host-devel/pull/6
Let's just bake it in.

Closes: #1482
Approved by: jlebon
2018-08-01 20:11:38 +00:00
Colin Walters
628a3aa22f tests/compose: Write logs directly
Make logging work the same as it does for the vmcheck-STI work
(at some point I'll try to unify the 3 parallel+script implementions
 we have).  This fixes the problem that when the test times out,
the filename won't have `.txt` and S3 won't have the right MIME type.

Closes: #1479
Approved by: jlebon
2018-08-01 18:05:09 +00:00
Colin Walters
800402b00a tests/compose: Add some time logging
To help us profile.

Closes: #1479
Approved by: jlebon
2018-08-01 18:05:09 +00:00
Colin Walters
da27b94b29 core: Use new rpmtsSetVfyLevel() API for writing rpmdb
Newer librpm defaults to doing a full payload checksum, which we can't
do at this point (writing the db) because we imported the RPMs into
ostree commits, saving just the header in metadata - we don't have the
exact original content to provide again.

Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1607223

Closes: #1469
Approved by: jlebon
2018-07-31 19:22:30 +00:00
Colin Walters
261dd99a69 Update libdnf, disable Python bindings
This entirely drops out Sphinx as well as python-devel from our
builds, makes builds faster, and silences a lot of warnings too.

Update submodule: libdnf

Closes: #1480
Approved by: jlebon
2018-07-31 17:13:50 +00:00
Colin Walters
3446caa190 importer: Prefix error with package name
I think we have issues with librepo not handling being forcibly
interrupted; I saw an lzma error from what I think was a partial
download.

Let's add an error prefix here so this is easier to debug.  I
think we used to have a prefix but lot it when we moved to async.

Closes: #1477
Approved by: jlebon
2018-07-31 15:18:45 +00:00
Jonathan Lebon
67860c5ece app/pkg-builtins: Add --unchanged-exit-77
This is analogous to `upgrade --upgrade-unchanged-exit-77`, but for
`install`/`uninstall`. This way, one can determine whether the command
truly had an effect on the default deployment or not.

Since this works by comparing against the previous default deployment,
this works correctly even if we already had a pending deployment at the
beginning of the transaction.

Closes: #1478
Approved by: cgwalters
2018-07-31 08:53:15 +00:00
Jonathan Lebon
d35fbb665e app/pkg-builtins: Add --idempotent
Add a new `install/uninstall --idempotent` option to make it easier to
interact with the CLI through scripts. E.g. one doesn't have to check
first if a request has already been installed/uninstalled.

Closes: #1467

Closes: #1478
Approved by: cgwalters
2018-07-31 08:53:15 +00:00
Jonathan Lebon
a9e1de0276 daemon/dbus: Document a few missing newer options
Closes: #1478
Approved by: cgwalters
2018-07-31 08:53:15 +00:00
Jonathan Lebon
fcb061b19a app/status: Add --pending-exit-77 switch
This makes it easier for scripts to determine whether there is a pending
deployment instead of using `--json/--jsonpath`.

Closes: #1478
Approved by: cgwalters
2018-07-31 08:53:15 +00:00
Colin Walters
16be1a0bad rust: Rework treefile to be an object
In a later patch I'm going to add more API; basically rather
than doing the JSON parsing from C, we can add APIs to directly
access the treefile object.  This also demonstrates how we
can do more extensive APIs, in particular implement an "object"
in Rust.

The ownership across the FFI boundary becomes nicer here too,
we don't need to do a dance with the fd.

For writing this I found
http://jakegoulding.com/rust-ffi-omnibus/objects/
quite useful, as well as
https://github.com/rust-lang/regex/blob/master/regex-capi/src/rure.rs

Closes: #1474
Approved by: jlebon
2018-07-30 18:54:48 +00:00
Colin Walters
5e5cdb4e0d rust: Add ptr error mapping API
Builds on the previous commit.

Closes: #1474
Approved by: jlebon
2018-07-30 18:54:48 +00:00
Colin Walters
ba28adf337 rust: Rework GError mapping API
I was going to add a new API that returns a pointer, and that just
wasn't going to work with the trait AFAICS; we'd need to be generic
across many different types.  Let's use plain functions; it's slightly
less elegant but is clear and works.

Closes: #1474
Approved by: jlebon
2018-07-30 18:54:48 +00:00
Colin Walters
716b60b7d7 scripts: Add Lua overrides for fedora-release and also -workstation
Dusty hit this when working on Fedora CoreOS.  IMO we don't need
to do the same for a future e.g. `fedora-release-silverblue` as
we don't support someone installing it on a non-ostree system.

(Note this all only really matters for `--unified-core` on the compose
 side)

Closes: #1473
Approved by: jlebon
2018-07-30 13:12:11 +00:00
Jonathan Lebon
7911b14f49 daemon: Fix cached-update including no-op diffs
The `cached-update` variant would mark a bunch of RPMs as upgraded even
if they didn't actually change. The issue turned out to be we were doing
the diff all wrong in the staged deployment case. I'm not sure what I
was thinking in #1344, but essentially, we were marking all layered RPMs
in the staged deployment as updates instead of only marking those
layered RPMs which were actually changed EVR.

We just simplify the approach here by directly doing a pkglist diff
between the booted and staged deployments and consuming that. That's
really all there is to it! Reduces the code quite a bit too.

Closes: #1446

Closes: #1455
Approved by: cgwalters
2018-07-28 06:53:40 +00:00
Jonathan Lebon
a17b4b9be0 tests/utils: Drop empty inject-pkglist.py
This one snuck in as part of #1205. I had initially started writing
`inject-pkglist.c` in Python and somehow inadvertedly made it part of
a commit.

Closes: #1455
Approved by: cgwalters
2018-07-28 06:53:40 +00:00
Colin Walters
831507b0f0 scripts: Add RPMOSTREE_SCRIPT_TRACE debugging envvar
I used `env RPMOSTREE_SCRIPT_TRACE='strace -f -s2048' rpm-ostree compose ...`
to debug https://bugzilla.redhat.com/show_bug.cgi?id=1548050

Closes: #1472
Approved by: jlebon
2018-07-27 13:09:49 +00:00
Colin Walters
e858a30eca compose: Stabilize --unified-core
In line with the recent trend of marking things stable, and in
preparation for stabilizing `rojig://` - Let's stabilize the `--unified-core`
option for `compose tree`.

I'm not sure we could make it the default anytime soon; today it trips
over bugs in the PAM package in RHEL7 for example.  But it
works fine for Fedora, and I think the code/design are good enough to be stable.

Closes: #1465
Approved by: jlebon
2018-07-24 23:06:50 +00:00
Colin Walters
fa29f7acfa compose: Support arch-specific packages in YAML (and in JSON again)
Follow up to: https://github.com/projectatomic/rpm-ostree/pull/1459

We now honor arch-specific packages in YAML, and reject unknown
architectures.  I looked a little bit at how to avoid having hardcoded
arch lists, but it doesn't seem worth it right now.

Closes: #1468
Approved by: jlebon
2018-07-24 22:05:06 +00:00
Jonathan Lebon
1a82f210ae ci: Bump f28-compose by 10 minutes
We're consistently seeing timeouts on the `f28-compose` test. It's
likely due to a performance regression in the backend rather than
anything else. I filed a ticket to get an investigation going, though
for now at least, let's bump the timeout some more to keep patches
flowing.

Closes: #1463
Approved by: jlebon
2018-07-24 20:15:10 +00:00
Jonathan Lebon
c016509548 libdnf: Bump and disable html and man pages
Skip building man pages and HTML docs for our embedded libdnf to speed
up builds.

This bump also pulls in a fix to ensure we never try to install src
packages from `dnf_context_install()`.

See: https://github.com/projectatomic/libdnf/pull/3

Update submodule: libdnf

Closes: #1463
Approved by: jlebon
2018-07-24 20:15:10 +00:00
Jonathan Lebon
b66337e0cb ci: Pepper in date markers to help debug timing
This should help out with determining what steps take the most time.

Closes: #1459
Approved by: cgwalters
2018-07-21 14:43:48 +00:00
Colin Walters
6ac6f3d086 treefile.rs: Deny unknown fields by default
Let's not make the same mistake we did with JSON where typoing a
field means it's silently ignored.  This actually caught a bug
in a YAML usage we had:

```
error: Failed to load YAML treefile: unknown field `install_langs`, expected one of ... `install-langs` ...
```

Yes, this is a compatibility break with the feature we just announced
but...I seriously doubt anyone (that isn't known to me) has converted
yet, and if they are excited enough to start using a two-week-old feature
they can adjust.

Closes: #1459
Approved by: cgwalters
2018-07-21 14:43:48 +00:00
Colin Walters
1b4a73f071 tests: Make two more tests compatible with deployment staging
Unlike the kernel args one, these two were simple.

Closes: #1458
Approved by: jlebon
2018-07-21 13:58:21 +00:00
Colin Walters
0cc002fd8d build-sys: Use python3 for libdnf by default if available
Probably at some point libdnf will drop py2 support, but the
main reason I'm doing this is avoids a python2 dependency
for rpm-ostree for distributions that don't want that.

Note of course rpm-ostree itself doesn't use python, libdnf does.
And only for the python bindings, which we don't use either.  So
this is mostly just to DTRT automatically for the libdnf bits; down
the line we could probably add a patch to make the python fully
conditional.

Closes: #1460
Approved by: jlebon
2018-07-20 22:27:34 +00:00
Jonathan Lebon
ec19d67d74 ci: Stop rebasing Fedora onto updates ref
We can drop this hack now since there have been updates since GA. Also
add a note that trying to drop the rebase to continuous causes
`layering-relabel` to fail.

Closes: #1406
Approved by: jlebon
2018-07-20 18:47:51 +00:00
Colin Walters
096004426c rebase: Add support for "custom origin" descriptions
We're looking to embed an ostree commit inside a container image,
to make it easier to transport around with other images.

Conceptually here the host system is tracking a container (just
like for rojig we're tracking an RPM).  This is the first step
towards making that support nicer; tooling can do
`rebase --custom-origin-url oscontainer://quay.io/exampleos@sha256:...`
and have that show up in `rpm-ostree status`.

There are two values, one intended to be machine readable (like
the `ostree://` and `rojig://` and one for humans which we
display when an admin types `rpm-ostree upgrade`.

This builds on prior work in
27bd7b97bb from #1396 .

Closes: #1406
Approved by: jlebon
2018-07-20 18:47:51 +00:00
Colin Walters
95de58c29d Update libglnx
For `renameat2()` fix to build with latest glibc (e.g. Fedora rawhide).

Update submodule: libglnx

Closes: #1461
Approved by: jlebon
2018-07-17 18:35:38 +00:00
Colin Walters
d1839378d6 compose: Add some assertions around treefile context directory
Seeing a crash on in one of our builds, not sure yet what's
going wrong as I haven't gotten a core out, but this makes things
clearer.

($kingdom for `T`/`Option<T>` instead of nullable pointers...)

Closes: #1454
Approved by: jlebon
2018-07-12 18:30:55 +00:00
Colin Walters
035a2a72ba tests/kernel-args: Prep for staged
This test was hard to adapt for staging.  I did the obvious change
of just rebooting after most of the `kargs` invocations, but much
later in the suite there was a test that compared vs the "pristine"
kargs.  Supporting that would be a bit hard...I briefly thought
about pinning etc. but it gets so messy to do in both staged and not-staged
defaults.  So I just deleted those bits.

Closes: #1453
Approved by: jlebon
2018-07-12 17:44:15 +00:00
Ruixin Bao
9373f13b0a doc: add repo_structure section
This commit plans to add repo_structure document to better explain
what is the meaning for each folder. A similar approach can be found
in https://github.com/openshift/openshift-ansible/blob/master/docs/repo_structure.md

Closes: #1452
Approved by: jlebon
2018-07-11 20:40:59 +00:00
Jonathan Lebon
caf66d6e45 vmcheck/autoupdate-check: Fix minor typos
Closes: #1450
Approved by: cgwalters
2018-07-11 13:56:37 +00:00
Jonathan Lebon
cfb5e7c04b libvm: add vm_get_journal_after_cursor
The cursor argument was getting munged up by SSH. Use a helper to make
sure we always get quoting right.

Closes: #1450
Approved by: cgwalters
2018-07-11 13:56:37 +00:00
Jonathan Lebon
fbee05bcc5 vmcheck/autoupdate-check.sh: Fix --check/--preview tests
Noticed this while looking at the logs for #1432.

Because --check and --preview exit with rc=77 when there are no updates,
we would actually stop early on in the test and marking it as SKIPPED.
Fix this by making sure we explicitly check for the $rc we expected when
using those switches.

I also added a final grep pass to make it easy to inspect whether we
skipped any tests. I was about to do this nicely in `multitest.py`
instead, though it may not be of this world much longer, so meh...

Closes: #1450
Approved by: cgwalters
2018-07-11 13:56:37 +00:00
Colin Walters
bfd39aecf3 status: Rework auto-update status display
First, split it into its own section; it's important enough to merit it.
Second, explicitly reference the systemd timer/service units.  For
example, a question I often have is "when is the next run" and of course
you can get that rpm `systemctl status rpm-ostreed-automatic.timer` but
you have to know that, and the reminder helps.

(I briefly looked at implementing the `Trigger` line from `systemctl status`
 but it's not entirely trivial...tempting to just fork off a `systemctl status | grep `)

Prep for unifying this text with the message we print when one does
`rpm-ostree upgrade` when auto-updates are enabled.

Closes: #1432
Approved by: jlebon
2018-07-10 13:26:37 +00:00
Colin Walters
7fa15cf078 app: Add an updateupgrade alias
Same as `dnf` for example, and `flatpak` today uses `update`, so
let's do both.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1448

Closes: #1449
Approved by: jlebon
2018-07-09 14:21:32 +00:00
Colin Walters
2e237513b0 rust/treefile: Avoid unnecessary dup() of fd
We weren't checking for an error from `dup()`, and further it shouldn't
be necessary.  This is the best I could come up with, though it's
obviously not pretty.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
d7b4046599 rust: Add a make rustfmt, run it
It is actually really nice that there's One Canonical Style, even
if I sometimes don't like some details of what rustfmt does.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
e731f59007 rust: Split out treefile into its own module
Further prep for adding more code here.  The `lib.rs` then is the
collection of glue functions; perhaps in some ideal future it could
be generated even.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
30ccaa4db6 build-sys: Use --frozen if Rust library was already built
The main rationale here is avoiding any Cargo internet access during
`sudo make install` type flows.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
2894d61fbf build-sys: Make Rust library naming/entrypoint more generic
Prep for adding more functionality beyond treefile bits.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
bac6d634bb build-sys: Move Rust include header under rust/ directory
It makes more sense to have the include live next to the associated
code, just like we do with C, even though the `cargo build` doesn't
touch it.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Yoginski
bcbd68d9c8 Explain how to remove packages from the base layer
Haven't seen it being mentioned in any manuals.
Closes: #1441
Approved by: cgwalters
2018-07-03 21:14:28 +00:00
Colin Walters
b1d7ef15d8 sysroot-core: Use new libostree sysroot pruning API
Use the new API introduced in
9131d8a4cc
which helps avoid relying on the magical deployment refs.  I thought
this would help with a livefs+staging issue, but it doesn't.  But we might
as well do it, it's adding another level of safety.

Closes: #1439
Approved by: jlebon
2018-07-03 20:18:00 +00:00
Colin Walters
222d730cce vmcheck/misc-2: Make compatible with staged default
First the pinning tests would try to pin a staged deployment,
and some of the later tests here depend on a subtle way on the
state of the system.  It's tempting to do a `reset` before each one
and reboot but this makes things work.

There's some additional assertions here as I went through and
was debugging.

Prep for making staging the default.

Closes: #1438
Approved by: jlebon
2018-07-03 19:14:54 +00:00