Commit Graph

74 Commits

Author SHA1 Message Date
Prajna Sariputra
3412108ad0 docs: Update links to the OSTree documentation
Those got moved to GitHub from readthedocs.io, meaning links pointing
to specific pages return a 404, so this fixes that along with
skipping the redirect for the links that just point to the main page.
2020-11-17 14:14:23 +01:00
Timothée Ravier
d71a50796d docs: Import Release page 2020-10-01 12:01:25 -04:00
Timothée Ravier
9e43141994 docs: Import Hacking page 2020-10-01 12:01:25 -04:00
Timothée Ravier
5e17d6fc24 docs: Move and update Treefile page 2020-10-01 12:01:25 -04:00
Timothée Ravier
7383d074bd docs: Move and update Compose server page 2020-10-01 12:01:25 -04:00
Timothée Ravier
33c893bbd0 docs: Move and update Client administration page 2020-10-01 12:01:25 -04:00
Timothée Ravier
75708f9d77 docs: Move and update Background page 2020-10-01 12:01:25 -04:00
Timothée Ravier
86e41df561 docs: Update Repo Structure page 2020-10-01 12:01:25 -04:00
Timothée Ravier
dff488b8dd docs: Update Contributing page 2020-10-01 12:01:25 -04:00
Timothée Ravier
a3080a42bf docs: Unify and update README and Index page 2020-10-01 12:01:25 -04:00
Timothée Ravier
ad4f0272e3 docs: Add Jekyll and theme config 2020-10-01 12:01:25 -04:00
Colin Walters
456a3ec7c2 compose: Add rpmdb option, default to bdb
The design of https://fedoraproject.org/wiki/Changes/Sqlite_Rpmdb
is problematic for us for multiple reasons.  The first big reason
is that rpm-ostree is designed for "cross" builds and e.g. today
we use a Fedora-derived container to build RHEL CoreOS images.

However the default database lives inside the `rpm` package which
means that if we e.g. upgrade the coreos-assembler container to F33
it will suddenly try to use sqlite for RHCOS which is obviously broken.

Related to this, rebases from f32 to f33 w/layered packages
are broken: https://bugzilla.redhat.com/show_bug.cgi?id=1876194#c3

With this we can configure things to continue to use bdb for f33
for ostree-based systems, so that by enforcing an upgrade order
f32 → f33 [bdb] → f34 [sqlite] ... the intermediate f33 w/bdb
still understands sqlite and hence rebases will work.
2020-09-11 10:06:28 -04:00
Colin Walters
70462a9a05 compose: Add a "warn and sleep" if --unified-core is not provided
Start adding some pain if `--unified-core` isn't provided
to help flush out anyone relying on it.  (And I think today
pungi is not passing it, so e.g. Fedora IoT/Silverblue are impacted)

Prep for merging https://github.com/coreos/rpm-ostree/pull/1793

Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>
2020-05-20 17:17:28 +02:00
Doug Hellmann
d160ed181c docs/background: Fix benefits and drawbacks formatting
The lists of pros and cons for each approach in the background section
of the manual use markdown formatting that is rendered as nested
lists. Separate the lists of benefits and drawbacks so that readers of
https://rpm-ostree.readthedocs.io/en/latest/manual/background/ and the
rendered markdown on github can understand which items are pros and
which are cons.

Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
2020-05-15 12:58:17 -07:00
Colin Walters
1bd69619eb treefile: Add readonly-executables
Same motivation as
7392259332

I think we should encourage removing the writable bits from
executables.  This has happened to me:
https://thomask.sdf.org/blog/2019/11/09/take-care-editing-bash-scripts.html

And not having the writable bit may help prevent hardlink
corruption with OSTree in some cases.

We can't do this by default, but add a convenient treefile option
for it.

This starts out by just doing this for RPMs, but I'll add
a secondary pass which does it during postprocessing soon too.
2020-05-12 08:39:37 -07:00
Colin Walters
860e0f018a s/RPM-OSTree/rpm-ostree/
We are have been pretty inconsistent about this; I think while it's
true the project is called "OSTree", "RPM-OSTree" is just annoying
to type and looks weird.  `rpm-ostree` requires much less
"shift key gymnastics".

Also, the "proper name" style like RPM/OSTree is best when
something can be viewed more "abstractly" - there are multiple
RPM implementations (historically) and OSTree also is a *concept*
in addition to an implementation.

rpm-ostree though is much more of a "concrete" thing so it
makes more sense to use it as a "project name".
2020-05-07 21:55:50 +02:00
Jonathan Lebon
3ec5e2878d manifest: Add lockfile-repos field
In Fedora CoreOS, we have a "coreos-pool" repo from which all packages
in lockfiles are tagged for reproducible builds. This repo is shared
across all streams, including those on f31 and f32.

Thus, it makes no sense for composes to ever pick packages unconstrained
from the pool without being guided by a lockfile. Otherwise, one can
easily end up with e.g. f32 packages in an f31 compose.

Add a new `lockfile-repos` for this which is only used for fetching
lockfile packages and nothing else. For example, this will allow
`cosa fetch --update-lockfile` to Just Work as expected by only fetching
new packages from regular yum repos.
2020-04-19 09:17:17 -04:00
Colin Walters
e41a8ab26f Add support for wrapping binaries (rpm, dracut, grubby)
We need to be friendlier to people who are transitioning from
"traditional" yum managed systems.  This patchset starts to lay
out the groundwork for supporting "intercepting" binaries that
are in the tree.

For backwards compatibility, this feature is disabled by default,
to enable it, one can add `cliwrap: true` to the manifest.

To start with for example, we wrap `/usr/bin/rpm` and cause it
to drop privileges.  This way it can't corrupt anything; we're
not just relying on the read-only bind mount.  For example nothing
will accidentally get written to `/var/lib/rpm`.

Now a tricky thing with this one is we *do* want it to write if
we're in an unlocked state.

There are various other examples of binaries we want to intercept,
among them:

 - `grubby` -> `rpm-ostree kargs`
 - `dracut` -> `rpm-ostree initramfs`
 - `yum` -> well...we'll talk about that later
2020-04-15 16:22:57 +02:00
Colin Walters
5a0d3356ef treefile: Add exclude-packages
In FCOS we have a kola test that basically does `rpm -q python`.
It's...a bit silly to spawn a whole VM for this.  Ensuring that
some specific packages don't get included has come up in a few
cases.

I think FCOS/RHCOS at least will want to blacklist `dnf` for example.
And as noted above, FCOS could blacklist `python`.

One major benefit of doing this inside rpm-ostree is that one
gets the full "libsolv error message experience" when dependency
resolution fails, e.g. blacklisting `glibc` I get:

```
 Problem 79: conflicting requests
  - package coreos-installer-systemd-0.1.2-1.fc31.x86_64 requires coreos-installer = 0.1.2-1.fc31, but none of the providers can be installed
  - package coreos-installer-0.1.2-1.fc31.x86_64 requires rtld(GNU_HASH), but none of the providers can be installed
  - package glibc-2.30-10.fc31.x86_64 is filtered out by exclude filtering
  - package glibc-2.30-7.fc31.x86_64 is filtered out by exclude filtering
  - package glibc-2.30-8.fc31.x86_64 is filtered out by exclude filtering
  - package glibc-2.30-5.fc31.i686 is filtered out by exclude filtering
  - package glibc-2.30-5.fc31.x86_64 is filtered out by exclude filtering
  - package glibc-2.30-10.fc31.i686 is filtered out by exclude filtering
```
2020-02-05 21:02:06 +01:00
Colin Walters
e726d008fc compose: Add an automatic-version-suffix key
This allows replacing the `.` in automatic version increments
with whatever one wants (as long as it's a single ASCII character)
right now.

The specific motivation here is for at least RHEL CoreOS to use
`version-suffix: "-"` so that its versions can become valid
semantic versions.

Related: https://github.com/coreos/rpm-ostree/issues/1954
2019-12-13 17:11:16 +01:00
Colin Walters
80fd2e883b README.md: Add a "why" section, update background.md
Point at FCOS and not Project Atomic.  Add an inline "Why"
section since people will want to know that right away.

(An great thing about Github is the prevalence it gives to `README.md`;
 projects should use that as an "elevator pitch")

Drop outdated bits in `background.md`.

Closes: #1895
Approved by: jlebon
2019-09-04 20:09:13 +00:00
Colin Walters
7db30fec13
treefile: Support arch-include (#1886)
* treefile: Rework test helper

Have each test clearly hold its tempdir; this way we can have a simple
helper function rather than a struct with an unreferenced parameter.

Also use `utils::write_file`.

Prep for further test work.

* treefile: Add a basic test for includes

Prep for adding an arch include test.

* treefile: Support `arch-include`

A long time ago we added architecture-specific package lists
via e.g. `packages-ppc64le`.  Much more recently we added
support for having the `include` key be a list - multiple includes.

By combining these two and supporting architecture-conditional includes,
we've effectively added architecture-conditionals to *all* keys.

Notably we want this for Fedora CoreOS today which is using
`remove-from-packages` on `grub2-tools` which isn't present on
s390x.

* ci: Bump cosa build timeout

Looks like another OpenStack perf regression.
2019-08-21 15:25:08 -04:00
Colin Walters
00bd491fe2 treefile: Support multiple includes
I'm working on having Silverblue inherit from Fedora CoreOS.  But
conceptually it also inherits from (parts of) Workstation.
It is just easier if we support multiple inheritance, then I don't
need to think too hard about how to make it a single inheritance chain.

Closes: #1870
Approved by: jlebon
2019-07-18 22:40:59 +00:00
Jonathan Lebon
fc49fb0089 treefile: Add new add-commit-metadata key
Add support for a new `add-commit-metadata` key in the treefile so that
we can directly specify commit metadata we want to inject from there.

This will be useful in Fedora CoreOS, where we'll have separate
treefiles for each streams, each with stream-specific metadata values
required.

Closes: #1865
Approved by: cgwalters
2019-07-09 20:28:18 +00:00
Colin Walters
5f6578ef3c compose: Add ostree-layers and ostree-override-layers
The use case for `ostree-layers` is to support injecting non-RPM
content in a more flexible way than can be done with `add-files`,
and also without dropping all the way to split composes.

This starts with support on the `compose tree` side but down the
line I'd like to make it more convenient to do *client* side too.

For `ostree-override-layers` this is mainly a development thing
for tools like coreos-assembler.  Rather than building an RPM
we just `make install DESTDIR` then commit and add to
`ostree-override-layers`.

Closes: #1830
Approved by: jlebon
2019-06-07 17:42:32 +00:00
Colin Walters
adff1e98b6 compose: Add boot-location: modules
And this (for now at least) completes the epic journey of the
"where's the kernel"?  With this it's found solely in
`/usr/lib/modules/$kver`.

There are a few reasons to do this; most prominent is that
it avoids duplicating the content as the locations may have
different SELinux labels.

Closes: #1773
Approved by: jlebon
2019-03-08 22:49:40 +00:00
Colin Walters
f37426f1ad compose: Remove support for boot-location: both
I'd like to add a new `boot-location: modules`.  In prep
for that, let's remove the legacy `both` which drops into
`/boot`.

The libostree support for handling `/usr/lib/ostree-boot` has
existed for over 4 years:

```
commit 37a059925f6b96d30190b65bee6bdde0ae1c6915
Commit:     Colin Walters <walters@verbum.org>
CommitDate: Sun Nov 30 23:14:05 2014 -0500

    deploy: Ensure that we can deploy using only /usr/lib/ostree-boot

```

I think we assume now that no one is now making *new* treecomposes and needs
a newer rpm-ostree and that they expect people to be able to use as an
upgrade target from a libostree that predates that.

Closes: #1773
Approved by: jlebon
2019-03-08 22:49:40 +00:00
Jonathan Lebon
1fe66e91d0 rust/treefile: Support dash convention for all options
Now that we support YAML, it's a gotcha/eyesore that some of our options
use underscores rather than dashes. Let's be nice and switch those few
options over, while of course still supporting the previous name.

Co-authored-by: Colin Walters <walters@verbum.org>

Closes: #1749
Approved by: cgwalters
2019-03-02 19:20:21 +00:00
Jonathan Lebon
3fda356dd5 docs/treefile: Add some more details to mutate-os-release
Make it easier to understand what the field actually represents.

Closes: #1751
Approved by: ashcrow
2019-02-01 18:43:29 +00:00
rfairley
a20d10e3d5 libpriv/util: Add date field in auto-versioning
This adds an optional date field to the prefix
passed by automatic_version_prefix. An example of specifying
the field is as follows:

10.<date:%Y>

And the fields progress like:

10.2018.0
10.2018.1
10.2018.2
10.2019.0

The date format creates a new "current date" string using
valid date directives passed into g_date_time_format().

If there is a problem reading the given date format,
an error is given and the next version is returned as NULL.

If no <date:...> tag is detected in the auto version prefix,
the same behavior as before (appending .1 and incrementing) occurs.

This may be helpful to avoid writing glue code to auto-update
the version if a date string in the commit version is desired.
Otherwise, --add-metadata-string=version= is an alternative for
complete customization.

Fixes: #1712

Closes: #1721
Approved by: jlebon
2019-01-14 18:49:41 +00:00
Jonathan Lebon
f8f47480f1 docs/treefile: Add info about paths for add-files
Add a note that only certain paths actually make sense in the
`add-files` directive and link to the OSTree docs. Let's also the
examples given to more realistic and correct ones.

Closes: #1643
Approved by: cgwalters
2018-10-26 20:48:07 +00:00
Colin Walters
8df776e367 compose: Remove support for ex-rojig-spec in treefile
In https://github.com/projectatomic/rpm-ostree/pull/1484
AKA commit 235e8f82da
we introduced a new `rojig` treefile entry that is nicer and
more maintainable than a split spec file.

In prep for more "pure rojig" work, drop support for this key.
Note we can also only now do this because we hard require Rust, and
the new model is only implemented in Rust.

The only known user of this today is FAHC, but that is now
[pinned to an assembler build](https://pagure.io/fedora-atomic-host-continuous/c/cdcb6ae943bee643983d4bd22d00030f2eb930d2?branch=master).

Closes: #1558
Approved by: jlebon
2018-09-13 21:10:21 +00:00
Colin Walters
279e7c4f1b treefile: Support inline postprocess element (for inheritance)
I'm trying to have a more opinionated model where custom builds
use inheritance, and currently one can only have a single
`postprocess-script`.

Further, in YAML it's very convenient to use inline vs external
data.

Closes: #1527
Approved by: jlebon
2018-09-05 15:27:07 +00:00
Colin Walters
40be3fb1cf Add recommends to treefile (default on)
This is for: https://github.com/projectatomic/rpm-ostree/issues/718
But I'm not going to close that issue as this only does the server
side, and I think we should support it client side too.

Since I wrote that issue, we ended up skipping the `dnf_transaction_depsolve()`
API, and hence we don't need to block on a libdnf change.  So
this was quite simple.

Closes: #1513
Approved by: jlebon
2018-08-23 04:24:07 +00:00
Colin Walters
588a0327db postprocess: Error if units with machineid-compat: false
Rather than silently ignoring it.

In theory...we could write to /usr/lib/systemd instead of `/etc`
but eh...I feel like what we really want to do is make it convenient
to write a preset file from the YAML.

(We could have an `add-files` content that takes values literally
 which would be nice in YAML and suck in JSON)

A general thread running through this is that for people making
*derivatives* of a CoreOS-like system, having to create their
own `exampleos-release` package is an annoying hurdle.

Anyways for now we're fixing the bug that we were silently ignoring
it.

Closes: #1488
Approved by: jlebon
2018-08-06 00:33:56 +00:00
Jonathan Lebon
be055179db docs/treefile: Mention rw in for machineid-compat option
See discussions in https://github.com/systemd/systemd/issues/8268 for
more information.

Closes: #1487
Approved by: cgwalters
2018-08-03 16:54:55 +00:00
Colin Walters
344aee1d76 rust: Add support for inline rojig spec files
The rojig spec is almost entirely rpm-ostree implementation details;
let's not have lots of people fork/duplicate it.  Rather add the bits
of rojig to the treefile that people need to define (most notably
the name).

Prep for stabilizing rojig.

I had a few false starts with this PR; managing ownership/lifetimes
across C/Rust is just complicated.  I got bit hard by the fact that
the workdir in `--unified-core` is really dfd-relative, and had to
do a dance to propagate the dfd into rust, as well as down into
the rojig builder.

Closes: #1484
Approved by: jlebon
2018-08-03 16:54:47 +00:00
Ruixin Bao
9373f13b0a doc: add repo_structure section
This commit plans to add repo_structure document to better explain
what is the meaning for each folder. A similar approach can be found
in https://github.com/openshift/openshift-ansible/blob/master/docs/repo_structure.md

Closes: #1452
Approved by: jlebon
2018-07-11 20:40:59 +00:00
Yoginski
bcbd68d9c8 Explain how to remove packages from the base layer
Haven't seen it being mentioned in any manuals.
Closes: #1441
Approved by: cgwalters
2018-07-03 21:14:28 +00:00
Colin Walters
d7342731ab Add a treefile option to stop creating /etc/machine-id
We actually want systemd's `ConditionFirstBoot` to fire.  The
primary rationale here is that we're adopting Ignition for Fedora CoreOS,
and having `ConditionFirstBoot=` function will help a lot, as the idea
is it only runs once.

However, I discovered that this breaks the `units` directive for example,
as systemd blows away all the unit state in `/etc`.  The correct thing
to do from the start is to use presets.  We could add an implementation of
`units` which works with this on and instead writes a preset file but...eh.
My plan is to at some point introduce an "epoch" and flip various defaults,
this one, `tmp-is-dir`, the passwd file handling, etc.

See: https://github.com/dustymabe/bootengine/pull/11

Closes: #1425
Approved by: jlebon
2018-06-25 16:19:00 +00:00
Michael Scherer
14b104d9f1 Fix the link to Fedora Atomic json file
Closes: #1423
Approved by: jlebon
2018-06-22 14:13:06 +00:00
Micah Abbott
de42ef701f docs: improve install/uninstall help
This adds some additional help to the man page and the RTD page around
using local packages with the install/uninstall commands.

Closes: #1388
Approved by: cgwalters
2018-06-05 15:59:52 +00:00
Colin Walters
39186e3330 docs/compose: Mention caching, privileged containers, example code
And also drop the `-z2` I saw show up in another project.

Closes: #1347
Approved by: jlebon
2018-04-25 18:43:36 +00:00
Micah Abbott
0399e185e4 docs: add additional details about 'tmp-is-dir'
Expanded the details around using 'tmp-is-dir' and what happens when
it is used.

Closes: #1315
Approved by: dustymabe
2018-03-26 17:26:54 +00:00
Colin Walters
3dd132285a Initial renaming pass of "jigdo" to "rojig"
I noticed that Ubuntu also uses the original "jigdo", so let's start
pulling off the band-aid here and do a mass rename.

For this first pass I'm focusing on CLI entrypoints and docs, as that's what
people are going to see; renaming all of the internal C functions, structure
variables etc. can come later.

Closes: #1269
Approved by: jlebon
2018-02-26 15:32:50 +00:00
Colin Walters
79f4b4b861 compose: Add --ex-jigdo-output-rpm
Part of the goal of jigdo ♲📦 is to support organizations switching to *only*
providing RPMs. An intermediate step there is to "lock" the repo and jigdo
together; we don't want to update the ref if building the jigdoRPM fails.

Add an option to perform `rpm-ostree compose tree` and `rpm-ostree ex
commit2jigdo` together; notably we generate a commit, but only update the ref
once the jigdoRPM is built.

Closes: #1165
Approved by: jlebon
2018-01-10 19:18:40 +00:00
Micah Abbott
f4877419ac docs: update branch examples for 'rebase' command
The old branch examples use Fedora 26 which is almost EOL.  The new
Fedora 27 examples show off the various `testing` and `updates`
branches, as well as the support for different arches.

Closes: #1175
Approved by: cgwalters
2018-01-03 16:05:15 +00:00
Colin Walters
d02bc4b03c compose: Accept NULL treefile for "use defaults" postprocessing
This is prep for split-compose.  We have some options in the
treefile, like `boot_location` and `tmp-is-dir` etc.  While those
are useful options, I don't want to force everyone using
`rpm-ostree compose postprocess` to write a treefile.

Change the code then to accept a `NULL` treefile to mean
"use the defaults".

Closes: #1070
Approved by: jlebon
2017-10-23 20:35:41 +00:00
Colin Walters
f113fc5e27 Rework treecompose kernel processing
Prep for changing `boot_location: new` to use `/usr/lib/ostree-boot`
and `/usr/lib/modules`.  Rework our kernel postprocessing
so that we unify the `boot_location` handling with initramfs generation.

Instead of doing the initramfs first in postprocessing, we do it nearly last,
after e.g. `etc` is renamed to `usr/etc`. This has some consequences, such as
the fact that `run_bwrap_mutably()` is now called in both situations. In
general, our handling of `etc` is inconsistent, although understandably so.

As part of this, I finally got around to implementing the bit from
https://github.com/systemd/systemd/pull/4174 however suboptimal it is; need the
unified core so we can cleanly ignore the posttrans like we do others.  We
intentionally keep the file around in the generated tree so that installing a
kernel RPM per client doesn't try to do any of this either.

This all gets folded together so that the logic for handling the bootloader gets
simpler - in the Fedora case, we now know to find kernels in `/usr/lib/modules`
and can ignore `/boot`.

Closes: #959
Approved by: jlebon
2017-09-12 14:26:44 +00:00
Colin Walters
3ff48589b3 compose: Make boot_location: legacy be an alias for both
We have 3 locations to find kernels now; I can't think of
a reason to support placing kernels *only* in `/boot`.  The
original commit
15ecaacd36
doesn't give a reason, and I certainly can't think of one now.

This makes `legacy` be an alias for `both`, which should be fully compatible.

Prep for further refactoring towards changing `new` to mean both
`/usr/lib/ostree-boot` *and* `/usr/lib/modules`.

Closes: #959
Approved by: jlebon
2017-09-12 14:26:44 +00:00