Commit Graph

115 Commits

Author SHA1 Message Date
Jonathan Lebon
571bd3a558 tests: Add hidden testutils subcommand
This is a hack to allow using `inject-pkglist` without having to build
the tree first.

Higher-level, I think we can split this back out again if we have a
`-tests` subpackage where we ship the vmcheck testsuite.
2019-12-13 19:18:30 +01:00
Colin Walters
c8113bde32 Add hidden coreos-rootfs seal command
All this does is put the immutable bit on the target directory.
The intention is to replace this bit to start:
8b205bfbb9/src/create_disk.sh (L229)

However, the real goal here is to add code in this file
to handle redeploying the rootfs for Fedora CoreOS which
combines OSTree+Ignition:
https://github.com/coreos/fedora-coreos-tracker/issues/94

Basically doing this in proper Rust is going to be a lot
nicer than shell script in dracut modules.  Among other
details, coreutils `mv` doesn't seem to do the right thing
for SELinux labels when policy isn't loaded.
2019-10-04 08:03:03 -07:00
Jonathan Lebon
83a267409d Add new D-Bus APIs for deployment finalization
Teach `UpdateDeployment` to make use of libostree's staging lock and
then add a `FinalizeDeployment` API to perform the final unlock &
reboot.

I also added a hidden CLI to make testing this easier, but also because
it's likely the FCOS-agent-yet-to-be-named will just end up using the
CLI to keep it simple.

Closes: #1748

Closes: #1814
Approved by: lucab
2019-04-30 08:36:28 +00:00
Colin Walters
ccc244879c build-sys: Always enable compose tooling
There's no reason to separate it anymore.

Closes: #1794
Approved by: jlebon
2019-03-25 13:46:29 +00:00
Colin Walters
387da3a420 ci: Verify rustfmt
Split out the command into a separate `Makefile` that doesn't
use Automake so we can invoke it early.

Closes: #1674
Approved by: dustymabe
2018-11-21 21:16:03 +00:00
Colin Walters
11ae25792c build-sys: Add bindgen bits to .gitignore
Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
dcc78cc2f1 build-sys: Fix uid check when we don't have /usr/bin/cbindgen
Make this match the other case.

Closes: #1676
Approved by: cgwalters
2018-11-20 18:06:34 +00:00
Colin Walters
bd4d0d8e0a build-sys: With external cbindgen, drop Rust lib dependency
Perhaps an unexpected side benefit of slow compilation processes
is that one has an opportunity to reflect and ponder.

I realized during exactly such a moment that since we moved
`cbindgen` out of our library build, there's no need to wait
for the library to be built before we can start building the C
code.

This is a notable local quality-of-life development improvement.

Closes: #1665
Approved by: jlebon
2018-11-05 19:59:58 +00:00
Colin Walters
88ffdc0ae2 Add compose rojig command
This currently requires a `--i-know-this-is-experimental` flag;
I know it'd be a bit more consistent to have it under `ex`, but
what feels weird about that is *most* of the `ex` commands people
use are client side.  This is where we want it to ultimately end
up.

We've landed a lot of prep patches, but I know there's still
a notable amount of code duplication with `compose tree`.  What's
left is about ~700 lines but it's mostly not hard/complex code
anymore.

In the future, I'd like to extract more of the compose code
to a `rust/src/compose.rs` or so, but I think this is sustainable
fow now.

My high level goal is to get this into coreos-assembler and stand
up a Silverblue build that uses it.

Closes: #1512
Approved by: jlebon
2018-10-31 17:57:45 +00:00
Colin Walters
fe5adaf7ab build-sys: Fix make rustfmt, and run it
When we added the bindgen config this broke as it's not a `.rs`.
Let's keep the source formatted canonically.

Closes: #1635
Approved by: jlebon
2018-10-23 19:05:06 +00:00
Colin Walters
d24f21343f build-sys: Just error out if building with multiple uids
The `--frozen` stuff ends up being annoying when switching
branches.  What we're really trying to protect against here
is the `sudo make install` problem, so let's test for that
more directly by verifying the uids.

(The previous code was also totally broken as it used `$` where
`$$` should have been in multiple places)

Closes: #1601
Approved by: cgwalters
2018-10-16 17:41:09 +00:00
Colin Walters
5d57025e16 build-sys: Add a rust-test target
I often am editing just the Rust code, and want the fast iteration
feedback of a `cargo test` - don't want to pay the cost of full
optimization (particularly LTO) for the release build, and I
*just* want to run the Rust tests.

Basically if you're editing our Rust code a lot, this target is
your friend.

Closes: #1585
Approved by: jlebon
2018-10-02 13:14:08 +00:00
Colin Walters
c96f93f0ad build-sys: Propagate Automake verbose mode to Rust/cargo
This is analogous to commit c62058e548
which propagated `V=1` into `cmake`.  Except if the build *isn't*
verbose (for local development), let's not force `--verbose` on
for Rust.

Closes: #1583
Approved by: jlebon
2018-09-26 17:07:19 +00:00
Colin Walters
f50f9e8d7e Split cbindgen to separate build, support external version
The problem is building bindgen as part of our single run
locks serde to way old versions, and I want to use newer versions.

Since Fedora will now again ship a `cbindgen` package, let's
also support using it if we find it, saving ourselves
the cost of building it.

For distros that don't ship it (e.g. CentOS) for CI purposes
we build it.  For downstream builds that are offline, rather
than vendor the cbindgen sources like we do with our main Rust,
let's just vendor the `rpmostree-rust.h` file as was suggested
in https://bugzilla.redhat.com/show_bug.cgi?id=1608670

Closes: https://github.com/projectatomic/rpm-ostree/issues/1557

Closes: #1573
Approved by: jlebon
2018-09-25 20:29:21 +00:00
Colin Walters
ca6f32371a compose: Split off composeutil.[ch]
Prep for adding a new toplevel rojig command.  It's also just
cleaner to avoid cluttering up the main compose logic with
distractions like the `_legacy_prep_dev()` bits.

Closes: #1564
Approved by: jlebon
2018-09-20 12:35:24 +00:00
Colin Walters
d17381265e build-sys: Rework Rust --frozen logic
For local development, I want to be able to e.g. update `Cargo.toml`
or switch branches and not have to `rm -rf target`.

Let's tweak the logic here so we only pass `--frozen` if the ownership
of the dir is different.

Probably an even better fix would be to just error out, but
this is a conservative tweak.

Closes: #1563
Approved by: jlebon
2018-09-17 15:51:53 +00:00
Colin Walters
36bf3c9164 build: Regenerate rpmostree-rust.h if cbindgen.toml changes
So when I edit it I get the changes.  Figuring this out
took me a little while, we were missing two dependencies.

Closes: #1559
Approved by: jlebon
2018-09-13 21:49:06 +00:00
Colin Walters
d2b0e42bfc Use cbindgen
If we're going to scale out our oxidation, let's follow
the path of Firefox (and other projects) further and use
cbindgen: https://github.com/eqrion/cbindgen

It's actually nice that `cbindgen` is packaged today in Fedora,
but I doubt it is elsewhere; we may end up needing to push
that forward, or just vendor it via a `build.rs` script and Cargo.

I chose to rename things to `ROR`/`ror_` since it's shorter.  I
am tempted a bit to rename our internal functions to just `ro_` to
or so.

Closes: #1516
Approved by: jlebon
2018-09-06 13:34:32 +00:00
Colin Walters
884427d948 core: Disable --as-needed for -ldl -lm linkage
https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
broke our build, since Rust doesn't yet have a way to express
the fact that the static library has dynamic dependencies.
(AIUI this is actually something libtool can handle with `.la` but eh)

Closes: #1522
Approved by: cgwalters
2018-08-28 21:08:15 +00:00
Jonathan Lebon
04c0678fa6 app: Add support for passing URLs to RPMs
This teaches the client to fetch packages from URLs directly so that one
doesn't have to `curl` first and then install. Supported anywhere
package filenames are allowed (notably: `install` and
`override replace`).

One neat things about this is that we download the file into an
`O_TMPFILE` and then pass on ownership of that fd directly to the
daemon. So at no point are the packages actually laying visible on the
system. (Assuming the filesystem supports `O_TMPFILE` that is).

This adds direct linking to libcurl and openssl, two libraries which we
were already pulling in indirectly.

Closes: #1508
Approved by: cgwalters
2018-08-23 11:16:15 +00:00
Jonathan Lebon
8c301401af build: Fix building rust in debug mode
When building in `debug` mode, `RUST_DEBUG` was still turned off because
`rust_debug_release` was set to `yes`, not `debug`.

Fix this by tweaking how `--enable-rust-debug` works: when it's *not*
provided, we default to the `$CFLAGS` detection logic. Otherwise, it
overrides it.

Closes: #1514
Approved by: cgwalters
2018-08-23 01:35:57 +00:00
Colin Walters
6a274b831d build-sys: Hard require Rust
As something that manages your base operating system, we care
about reliability, predictability, as well as performance and
low-level access to native operating system facilities.  The
C programming language is great for the latter two, but fails
at providing a truly memory-safe environment.  Rust is fairly
unique in providing a language that doesn't carry a runtime,
so we can gradually "oxidize" and convert our C code without
imposing additional overhead.  It's also got a lot of modern
design niceties, like not having a null pointer.

Let's pull the trigger here and hard require Rust.  It's the
programming language I personally want to be primarily writing in for
years to come.

This is also in line with a recent trend of reducing our
experimental/optional matrix.

Closes: #1509
Approved by: jlebon
2018-08-21 14:49:26 +00:00
Colin Walters
d7b4046599 rust: Add a make rustfmt, run it
It is actually really nice that there's One Canonical Style, even
if I sometimes don't like some details of what rustfmt does.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
30ccaa4db6 build-sys: Use --frozen if Rust library was already built
The main rationale here is avoiding any Cargo internet access during
`sudo make install` type flows.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
2894d61fbf build-sys: Make Rust library naming/entrypoint more generic
Prep for adding more functionality beyond treefile bits.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Colin Walters
bac6d634bb build-sys: Move Rust include header under rust/ directory
It makes more sense to have the include live next to the associated
code, just like we do with C, even though the `cargo build` doesn't
touch it.

Closes: #1444
Approved by: jlebon
2018-07-05 13:58:22 +00:00
Jonathan Lebon
07ecfca3d9 Add new reset command
Add a new `reset` command that makes it easy to blow away all
customizations: overlays, overrides, and initramfs. One can use flags to
only reset some of the customizations.

I placed this under `ex` out of conservatism. It's a pretty simple
command with simple behaviour, though the features it relies on
(no-layering, no-initramfs) are brand new. We can move it out of there
in a release or two?

Closes: #1387

Closes: #1419
Approved by: cgwalters
2018-06-26 20:32:28 +00:00
Jonathan Lebon
479406e6a5 Add support for YAML treefiles
Let's modernize and start supporting YAML treefiles. I'll dare make the
sweeping generalization that most people would prefer reading and
writing YAML over JSON.

This takes bits from coreos-assembler[1] that know how to serialize a
YAML file and spit it back out as a JSON and makes it into a shared lib
that we can link against. We could use this eventually for JSON inputs
as well to force a validation check before composing.

If we go this route, we could then turn on `--enable-rust` in FAHC for
now and drop the duplicate code in coreos-assembler.

[1] https://github.com/cgwalters/coreos-assembler

Closes: #1377
Approved by: cgwalters
2018-06-05 13:08:33 +00:00
Colin Walters
f05b980094 build-sys: Mark libpriv symbols as private
This was caught by the abicheck in Fedora; since we were building with default
visibility for `librpmostreepriv.la` which was linked statically into the public
library, we'd end up with lots of internals as public ABI.

Fix this by using `-fvisibility=private` for the libpriv build and for good
measure elsewhere so we remember to use it by default.

Closes: #1320
Approved by: jlebon
2018-03-28 15:18:11 +00:00
Colin Walters
36071d3c69 rojig-rename: (almost) Everything else
This renames the remaining C files, tests, etc.  There are only
a few hits for `jigdo` left; changing them would be a format break,
so let's wait to do that until we need to.

Closes: #1279
Approved by: jlebon
2018-03-01 22:35:46 +00:00
Colin Walters
df86d81ad7 rojig-rename: Rename the "build" side files
Closes: #1279
Approved by: jlebon
2018-03-01 22:35:46 +00:00
Colin Walters
f26dcd59a2 Add rpm-ostree usroverlay
I saw kalev's slides reference `rpm-ostree unlock`; this patch makes it exist.
In general, people have a hard time (understandably) grasping the distinction
between ostree and rpm-ostree; along with the goal of making ostree really
"libostree", let's start wrapping more commands where it makes sense.

I also took this opportunity to have a more descriptive name; it's important
to note that it *doesn't* overlay `/etc`, `/var`, or `/boot` for example.

Closes: #1233
Approved by: jlebon
2018-02-11 18:03:06 +00:00
Colin Walters
95b423afe9 Add cancel verb and DBus API
Right now the fact that one can only cancel via `Ctrl-C` of an existing client
process is rather frustrating if for example one's ssh connection to a machine
drops. Now, upon reconnecting, one can easily `rpm-ostree cancel` a hung update
or whatever rather than doing the more forcible `systemctl stop rpm-ostreed`
(which is safe of course, unless livefs is involved).

Closes: #1019
Approved by: jlebon
2017-12-07 20:12:52 +00:00
Colin Walters
694b798c73 Introduce experimental "rpm-ostree jigdo"
Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081

To briefly recap: Let's experiment with doing ostree-in-RPM, basically the
"compose" process injects additional data (SELinux labels for example) in an
"ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree
image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we
need need to download. For updates, like `yum update` we only download changed
RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still
done server side, and we still have a reliable OSTree commit checksum.

This is a lot like [Jigdo](http://atterer.org/jigdo/)

Here we fully demonstrate the concept working end-to-end; we use the
"traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which
has a checksum, version etc. Then the new `ex commit2jigdo` generates the
"oirpm". This is the "server side" operation. Next simulating the client side,
`jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs,
fully regenerating *bit for bit* the final OSTree commit.

If you want to play with this, I'd take a look at the `test-jigdo.sh`; from
there you can find other useful bits like the example `fedora-atomic-host.spec`
file (though the canonical copy of this will likely land in the
[fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo.

Closes: #1103
Approved by: jlebon
2017-12-04 14:24:53 +00:00
Colin Walters
a822722a5b ex-unpack: Delete
I think the `ex container` path supercedes this; it was really just a demo, and
having it around is annoying since I want to change the importer API and I have
to change it here too.

Closes: #1116
Approved by: jlebon
2017-11-29 16:48:19 +00:00
Ruixin Bao
fa593afd89 app/daemon: add groundwork for rpm-ostree kargs command
This is initial groundwork for https://github.com/projectatomic/rpm-ostree/issues/594.

This commit sets up most of the required
front end logic( arg parsing, transaction handling), and will
be used in the following commits.

There is nothing really fancy in this commit, as most of the code
shares the similar style between other dbus related commands.

Closes: #1013
Approved by: cgwalters
2017-11-06 18:49:36 +00:00
Jonathan Lebon
8285ef2a15 app: add 'refresh-md' command
This is essentially the `dnf/yum makecache` equivalent for rpm-ostree.
To complete the picture, this goes hand in hand with the `-C`
equivalent, which is added in the next patch.

Closes: #1035
Approved by: cgwalters
2017-10-06 18:20:59 +00:00
Jonathan Lebon
577aef98e2 app: spawn polkit authentication agent
Spawn pkttyagent when trying to call a method that may require
authentication to give users a chance to provide auth right from the
terminal.

Since we're now relying on polkit for authorizing most of the OS
interface methods, let's drop the root check on those.

Closes: #894
Approved by: cgwalters
2017-07-27 17:10:41 +00:00
Jonathan Lebon
cde3295e26 app: add experimental support for pkg removals
This is one more step towards making rpm-ostree more powerful in its
quest to be the ultimate *hybrid* image/package system. Package layering
allows us to add packages on top of the base package set received from
the content provider. However, we're not able to remove or replace
packages in the base set itself.

This patch introduces a new `override` command, which is for now nested
under the experimental `ex` command. The `override` command will allow
users to modify the base package set itself. The first implemented
subcommands are `remove` and `reset`.

A stub has been provided for the more useful `replace` subcommand,
though much of the needed logic for that operation are implemented in
this patch as part of the `remove` subcommand.

Part of: https://github.com/projectatomic/rpm-ostree/issues/485

Closes: #797
Approved by: cgwalters
2017-06-05 20:48:50 +00:00
Colin Walters
95e9aa4284 Introduce ex livefs
There are a few different use cases here. First, for layering new packages,
there's no good reason for us to force a reboot. Second, we want some support
for cherry-picking security updates and allowing admins to restart services.  Finally,
at some point we should offer support for entirely replacing the running tree
if that's what the user wants.

Until now we've been very conservative, but there's a spectrum here. In
particular, this patch changes things so we push a rollback before we start
doing anything live. I think in practice, many use cases would be totally fine
with doing most changes live, and falling back to the rollback if something went
wrong.

This initial code drop *only* supports live layering of new packages.  However,
a lot of the base infrastructure is laid for future work.

For now, this will be classified as an experimental feature, hence `ex livefs`.

Part of: https://github.com/projectatomic/rpm-ostree/issues/639

Closes: #652
Approved by: jlebon
2017-05-12 15:00:59 +00:00
Colin Walters
8dfed46e26 app: Rename internals → ex
The goal is to consolidate our "experimental" functionality under one
subcommand.  This makes it easier to determine when things "graduate"
to permanent-stability status under the main command line.

Closes: https://github.com/projectatomic/rpm-ostree/issues/682

Closes: #683
Approved by: jlebon
2017-03-16 19:57:14 +00:00
Colin Walters
b034381506 daemon: Implement "reload"
There are two main issues right now; first, we don't pick up manual changes to
`.origin` files, which occurs when one needs to sed it to remove `unconfigured`
for example. Second, we need to reload changes to the remotes.

Closes: #598
Approved by: jlebon
2017-02-24 21:36:45 +00:00
Colin Walters
a66d27230d Add a cleanup command and DBus API
We sometimes talk about using `ostree admin undeploy`, but that
doesn't know about the pkgcache, and hence space there leaks
until the next rpm-ostree operation.

Just for this, we need to expose a cleanup command (and API).  But
we also need to support cleaning:

 - repomd
 - downloads (repo/tmp)

So let's start implementing that.

Closes: #614
Approved by: jlebon
2017-02-13 15:28:05 +00:00
Jonathan Lebon
b0727d88fd start-daemon: make hidden
I debated just putting this in the supported list, but decided against
it in the end. This really should be something that happens
transparently, and if it doesn't then something else is probably wrong.

Closes: #617
Approved by: cgwalters
2017-02-10 20:11:27 +00:00
Colin Walters
5f7c5305ff Delete "compose sign"
We want people to use the libostree API for things like this. Further, the
`rpm-sign` tool that this calls is Red Hat internal, so it doesn't make sense to
have a public wrapper for it.

Closes: https://github.com/projectatomic/rpm-ostree/pull/152

Closes: #607
Approved by: jlebon
2017-02-07 21:43:22 +00:00
Colin Walters
687567d3ee Merge daemon binary into main binary
The actual problem I am trying to fix with this is fallout from the
introduction of `/usr/libexec/rpm-ostreed`, which required a SELinux
policy change.  Specifically for CentOS, the base policy is rev'd
slowly.

My hope was that by merging the daemon code back into `/usr/bin/rpm-ostree`
which is labeled `install_exec_t`, starting via systemd would do
the right thing.  It turns out that doesn't happen.

Now later, I'm picking this patch back up because I want to do multprocessing in
the daemon (and in the core), and it makes sense to share code between them,
because multiprocessing will need to go through a re-exec path.

Another benefit is we avoid duplicated text (libglnx, internal helpers) between
the two binaries.

Closes: #292
Approved by: jlebon
2017-02-07 16:07:09 +00:00
Colin Walters
cac4522e5b Support "system/regenerate-initramfs=true" flag in origin
Currently we push for a model where the initramfs is
generated (in non-hostonly mode), and merely replicated.

However, to support a few unfortunate corner cases like dm-multipath which wants
to inject a config file into the initramfs, we need to support regenerating it
client side too.

Down the line, we'll need this to support overriding the kernel too.

This changes things in the core to add the concept of an "empty"
`RpmOstreeContext`. I initially tried skipping it, but that was too much
duplication. We still want all of the core ostree-related logic that lives in
that code too.

The treespec bits barfed if the spec didn't have a `tree/packages` key. It was
simplest to change that to allow it - and because that was the only case where
we errored out in parsing, I dropped the error handling.

There was another place in the upgrader that now needed to be fixed to handle
transitioning from just regenerating initramfs to not.

Closes: #574
Approved by: jlebon
2017-01-20 16:17:51 +00:00
Colin Walters
328b85d396 Add infrastructure for alternative name (currently nts)
See https://github.com/projectatomic/rpm-ostree/issues/405

This patch adds an (off by default) `--enable-new-name` build option
which currently defaults to `nts`.  This is purely additive, and
the intention is that we'll support the rpm-ostree name in
perpetuity most likely.

At the moment, we add a new name for:
  - /usr/bin/$name
  - The systemd unit file

But we notably *don't* attempt to add a new name to the DBus API,
as it'd be a lot more invasive of a patch, and less payoff (it's
mostly just programs/scripts that interact with the DBus).

Closes: #497
Approved by: jlebon
2016-10-20 14:22:28 +00:00
Colin Walters
125c482b1d Switch to using libhif as a git submodule
So I was trying to hack on my host's copy of rpm-ostree inside a pet
docker container, but ran into a conflict with libhif since dnf uses
it.  I think we basically need to *always* build the bundled path,
rather than what I'm doing with CAHC and FADC where it's built as a
regular RPM.

It's not really sustainable right now for us to have both bundled and
not-bundled build paths - and we need to support co-installation with
dnf.

Another major issue is that we want to version lock with libhif -
right now our CI and both CAHC/FADC track libhif master, but that
means everything breaks if libhif breaks and we don't immediately
port.

git submodules solve all of these problems - the same as we're doing
with libglnx.

libglnx is *designed* for use as a git submodule, where as libhif
needs to support being both bundled and not-bundled.  So we end up
with some hacks on our side, but I think it's all not too bad.  I've
marked build rules with `# bundled libhif` so we know where to find
them later when libhif is stable.

Closes: #357
Approved by: jlebon
2016-06-30 14:27:55 +00:00
Colin Walters
9d473df11e build: Remove --enable-usrbin-atomic
We haven't used this in a long time now that the atomic command
exists.

Closes: #361
Approved by: jlebon
2016-06-29 16:44:08 +00:00