Commit Graph

2603 Commits

Author SHA1 Message Date
Jonathan Lebon
035ac2eaa6 tests/vmcheck: Fully drop python 3 requirement
Drop the use of Ansible everywhere. In the few cases where we really
Python, just spawn a container instead.

This is required to be able to hack on Fedora CoreOS.

Closes: #1850
Approved by: jlebon
2019-06-10 15:36:06 +00:00
Colin Walters
5f6578ef3c compose: Add ostree-layers and ostree-override-layers
The use case for `ostree-layers` is to support injecting non-RPM
content in a more flexible way than can be done with `add-files`,
and also without dropping all the way to split composes.

This starts with support on the `compose tree` side but down the
line I'd like to make it more convenient to do *client* side too.

For `ostree-override-layers` this is mainly a development thing
for tools like coreos-assembler.  Rather than building an RPM
we just `make install DESTDIR` then commit and add to
`ostree-override-layers`.

Closes: #1830
Approved by: jlebon
2019-06-07 17:42:32 +00:00
Jonathan Lebon
c1cc0827a3 core: Strengthen how we enforce lockfiles
One problem with how we use lockfiles right now is that we don't enforce
them for dependencies. That is, if `foo` requires `bar`, but only `foo`
is in the manifest, then while `foo` will be locked, `bar` will never
be checked against the lockfile because it was never explicitly
requested.

Higher-level though, I don't like how indirect the locking here feels.
See some comments about that in:

https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r288772527
https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r289419017

Essentially, the manifest is an input file of patterns, and all we
really know from the lockfile output is that the set of packages in
there satisfies this input in some way. But:

1. there are multiple ways to satisfy the same input (hence why hints
   like `SOLVER_FAVOR` exist)
2. the solution is dependent on how the solver is implemented (i.e.
   different libsolv versions might yield different solutions)
3. the solution is dependent on flags fed to the solver (i.e. different
   libdnf versions might yield different solutions)

So any attempt at cross-checking between the input file and the lockfile
is going to be very hard. Using a stricter mode as I suggested in #1745
of only allowing pure pkgnames or NEVRAs would help, but it wouldn't
address the dependency issue. (Though I'm still thinking about possibly
doing this anyway.)

The solution I propose here is instead to take the nuclear approach: we
completely exclude from the sack all packages of the same name as
packages in our lockfiles, but which do not match the NEVRA. Therefore,
any possible solution has to also satisfy our lockfile (or error out).

Closes: #1849
Approved by: cgwalters
2019-06-06 14:06:07 +00:00
Jonathan Lebon
022622be46 core: Fix outdated misleading comment
We no longer tell libdnf to add the pkg to the goal until later (because
of https://github.com/rpm-software-management/libdnf/issues/700). Just
tweak those comments to be more explicit about that.

Closes: #1849
Approved by: cgwalters
2019-06-06 14:06:07 +00:00
Jonathan Lebon
9cbf62998b core: Use variable for sack higher up
It's nicer to read/less verbose.

Closes: #1849
Approved by: cgwalters
2019-06-06 14:06:07 +00:00
Rafael Fonseca
79dfcea42f compose: Add --ex-lockfile and --ex-write-lockfile-to
Fixes #1670

This patch introduces a new `compose tree
--ex-write-lockfile-to=manifest.lock` argument and a new `compose tree
--ex-lockfile=manifest.lock` to read it back for subsequent invocations.

Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>

Closes: #1745
Approved by: jlebon
2019-06-04 15:14:47 +00:00
Jonathan Lebon
68c416fa9c app/deploy: Add --lock-finalization switch
Of course, update agents driving rpm-ostree know exactly to which commit
they want the system to upgrade, so `upgrade --lock-finalization` is not
helpful. Teach `deploy` the `--lock-finalization` switch too.

Closes: #1846
Approved by: lucab
2019-05-27 17:18:36 +00:00
Colin Walters
bee8ba4e48 composeutil: Add error prefixing to legacy mkdev path
Had to track down via strace that it was this that was failing
in my toolbox container.

Really need to merge the unified-core-only PR.

Closes: #1845
Approved by: jlebon
2019-05-27 16:30:48 +00:00
Jonathan Lebon
7dcfb2d24a app/db-diff: Rename "old" -> "from" and "new" -> "to"
The `old` and `new` naming is odd. It implies a temporal relationship
between the two commits. Just rename those to the more apt "from" and
"to".

The difference is mostly cosmetic, but I didn't want to inherit this in
the new JSON interface. It does technically breaks the `diff` output
which is a somewhat machine-compatible interface, though the "ostree
diff commit" headers have been pretty freeform anyway, so I doubt anyone
is actually trying to read those.

Closes: #1844
Approved by: cgwalters
2019-05-23 20:24:11 +00:00
Jonathan Lebon
ae354e87ab app/db-diff: Check upfront for invalid format arg
And avoid more costly operations.

Closes: #1844
Approved by: cgwalters
2019-05-23 20:24:11 +00:00
Jonathan Lebon
c866bdfac9 daemon/package-variants: Fix docstring arguments
Some argument names were wrong, others missing.

Closes: #1844
Approved by: cgwalters
2019-05-23 20:24:11 +00:00
Jonathan Lebon
12fbe89aba app/db-diff: Add --format=json output
Add a new "json" output format. The "diff" format is also a mostly
machine-compatible one. But JSON is much more ubiquitous and easier to
consume.

Closes: #1844
Approved by: cgwalters
2019-05-23 20:24:11 +00:00
Jonathan Lebon
1ac29f25b7 Release 2019.4
It's been almost two months!

Closes: #1841
Approved by: cgwalters
2019-05-22 01:17:08 +00:00
Jonathan Lebon
a27d1a2153 daemon: Bump sysroot mtime to force reload
Right now, after calling `rpm-ostree finalize-deployment`, we update the
`DefaultDeployment` property so that its `finalization-locked` key is
updated. This allows update agents like zincati to correctly understand
the current state if the reboot is locking/inhibited.

The issue though is that this property is accessible through D-Bus only,
and current plans for zincati is to just use the CLI for now.
Unfortunately, the output of `status --json` doesn't correctly get
updated since the deployments array comes from the sysroot interface.

Just use the nuclear mtime bump instead to force a reload. Another
approach long term is to formalize the set of paths/attributes libostree
clients should be monitoring, though having a single API is nice too.

Closes: #1842
Approved by: cgwalters
2019-05-21 19:56:06 +00:00
Jonathan Lebon
461a4f792d libpriv/kargs: Tweak delete restrictions
1. Allow deleting keys without values (e.g. `nosmt`) if such a key
   variant exists (i.e. this won't work if there are only e.g.
   `nosmt=foo` and `nosmt=bar` variants).
2. Allow deleting duplicate `keys[=val]` kargs.

Closes: #1834

Closes: #1835
Approved by: cgwalters
2019-05-17 13:44:06 +00:00
Colin Walters
0c80aa9666 core: Split posttrans into separate output status
Prep for adding support for injecting native ostree layers; we still
want to run posttrans scripts (and file triggers) after these so
that adding a shared library will still have `ldconfig` run.

Closes: #1836
Approved by: jlebon
2019-05-16 20:39:11 +00:00
Colin Walters
cdd482f80b kargs: Just print args, don't add additional text
The Unix tradition is generally not to add English text unless
necessary.

This makes the output of this command more obviously parsable,
although I'm not entirely sure we should do this versus adding
`--json` or so, but eh, it's also not wrong.

Closes: #1833
Approved by: jlebon
2019-05-15 21:27:53 +00:00
Robert Fairley
d829311a3a vagrant: Add header noting coreos-assembler
Follow-up to #1826 to reflect new development occurring for
Fedora CoreOS using coreos-assembler.

Closes: #1831
Approved by: jlebon
2019-05-13 19:50:58 +00:00
Jonathan Lebon
4d845952c8 tests/compose: Add RPMOSTREE_TMPDIR_LOCATION
This allows one to run the tests from a container using overlay +
SELinux protection by running the actual compose into a non-overlay
bind-mount. Otherwise, we'll hit `ENOTSUP` when trying to set labels on
various checkouts.

Closes: #1829
Approved by: cgwalters
2019-05-09 02:10:27 +00:00
Jonathan Lebon
c035c2b574 compose: Add --no-parent option
There are cases where we do want all the things that specifying a ref
provides (e.g. change detection, version incrementing, SELinux labeling
optimizations, and of course writing the ref) but we *don't* want the
new commit to have a parent. Add a new `--no-parent` option to
accommodate this.

This will be used by coreos-assembler. See discussions at
https://github.com/coreos/coreos-assembler/issues/159.

Closes: #1829
Approved by: cgwalters
2019-05-09 02:10:27 +00:00
Jonathan Lebon
44594ca4d2 tests/compose: Fix subtest scoping
We had a subtest that wasn't actually part of the `basic_test()` and so
was being executed when the file gets sourced instead of the function
being explicitly called.

Closes: #1829
Approved by: cgwalters
2019-05-09 02:10:27 +00:00
Robert Fairley
be1be526ff vagrant: Use a Fedora 29 container
Switch the Docker + Vagrant development docs to use a Fedora 29
build container, and a Fedora 29 Atomic Host Vagrant box. CentOS
7-based testing was recently removed (#1785) - let's have the
documented development pattern reflect this.

Also no longer enables the EPEL7 repo in the Vagrant VM, as needed
dependencies are available in Fedora Atomic Host.

A note is left to later switch to Fedora CoreOS as the documented
Vagrant box to use, once Fedora CoreOS boxes are produced.
Alternatively, one may use [cosa](https://github.com/coreos/coreos-assembler).

A few notes are also added to vagrant/README.md in places where
the reader may hit problems.

squash

Closes: #1826
Approved by: cgwalters
2019-05-09 00:08:14 +00:00
Robert Fairley
03a94e21b8 HACKING: Add link to tests README
Add a link to tests/README.md to point readers to additional
information on where to find different types of tests.

Closes: #1826
Approved by: cgwalters
2019-05-09 00:08:14 +00:00
Jonathan Lebon
4c99cc765f Copy used command-line and set as transaction title
In the app, rebuild the exact command-line that the client used and pass
that to the daemon to be used as the transaction title. Especially in
transactions like `UpdateDeployment()`, we can avoid reverse-engineering
what the original command used was.

This will be used by the upcoming history feature to record the
command-line used in the journal.

Closes: #1824
Approved by: rfairley
2019-05-08 21:09:08 +00:00
Jonathan Lebon
01fbaa7ba4 libpriv/util: Expand set of safe shell chars
Expand the set to include '/' and '=' since those don't need escaping in
shells.

Closes: #1824
Approved by: rfairley
2019-05-08 21:09:08 +00:00
Jonathan Lebon
d113b6a913 app/status: Make --json output pretty JSON
Yes, it's mostly for machines, but in practice it's super useful for
humans to look at esp. when debugging.

Closes: #1828
Approved by: cgwalters
2019-05-08 19:02:32 +00:00
Jonathan Lebon
206ae24d4e tests: Bump to Python 3 only
This bumps the requirement on the controlling host to Python 3 only.
It also bumps the requirement on the target host to Python 3 as well
since FCOS doesn't ship Python 2 right now.

Though we'll need to eventually drop all Python usage anyway, but at
least let's get tests passing on FCOS first. (See related previous
patch).

Closes: #1828
Approved by: cgwalters
2019-05-08 19:02:32 +00:00
Jonathan Lebon
9f618d0acc vmcheck/overlay: Drop dependency on host Python
Also switch to using `jq` on the controlling host instead of Python.

This is also prep for switching CI to FCOS which is likely to not ship
Python at all. There are still spots a bit everywhere where we currently
assume Python on the target host. We'll have to address those soon.

Closes: #1828
Approved by: cgwalters
2019-05-08 19:02:32 +00:00
Robert Fairley
c89f81c138 libvm: Use shared memory for SSH control socket instead of /var/tmp
In Fedora 29, and Fedora 30 Silverblue, I have come across the
following error when executing `make vmsync` from my build container
(also on Fedora 29 and Fedora 30 images respectively):

```
...
Failed to connect to new control master
...
Control socket connect(/var/tmp/ssh-vmcheck-1556768111752693879.sock): Connection refused
Failed to connect to new control master
...
```

Previously this worked with Fedora 28 as the host.

After changing the socket to be in /dev/shm, the SSH connection to
the `vmcheck` VM is successful and the sources sync over.

The cause of this seems to be a problem with overlayfs and unix
sockets: https://github.com/moby/moby/issues/12080

Since overlayfs is the default graph driver in Fedora now, work
around this by switching the socket to be in /dev/shm.

Closes: #1827
Approved by: jlebon
2019-05-02 20:43:28 +00:00
Jonathan Lebon
aaccd48ef1 daemon/deploy: Set transaction title earlier
Instead of doing a bunch of work before setting the transaction title,
set it upfront.

Also make more explicit how we determine whether we're doing an upgrade.

Closes: #1825
Approved by: rfairley
2019-05-02 20:43:17 +00:00
Jonathan Lebon
e0b0ff0066 daemon: Rework kargs transaction
Make it use the `deploy()` function like the others instead of having a
separate function that sets kargs and then automatically deploys.

Prep for future patches.

Closes: #1825
Approved by: rfairley
2019-05-02 20:43:17 +00:00
Jonathan Lebon
4c2fab8f5e libpriv/util: Factor out shell quoting function
Will be used elsewhere in an upcoming patch.

Closes: #1825
Approved by: rfairley
2019-05-02 20:43:17 +00:00
Jonathan Lebon
4499ac4ce0 libpriv/util: Factor out function to get version
Will be used by the history work.

Closes: #1823
Approved by: mike-nguyen
2019-05-01 15:12:15 +00:00
Jonathan Lebon
fcd2afaade daemon/utils: Allow unfiltered deployment variant
For the history work, I'd like to be able to retrieve the full GVariant,
which includes the whole unfiltered layered and base commit metadata. So
let's add an argument to allow not filtering those.

Closes: #1823
Approved by: mike-nguyen
2019-05-01 15:12:15 +00:00
Jonathan Lebon
0aded8a85b app/status: Strengthen deployment printing
Allow `out_printed_cached_update` and `sysroot_proxy` to be `NULL`. This
will be the case for the history feature, which will reuse the same code
for printing.

Closes: #1823
Approved by: mike-nguyen
2019-05-01 15:12:15 +00:00
Jonathan Lebon
41673e921e app/compose: Add ostree-content-checksum to composejson
This is useful for tracking OSTree content across a pipeline.

See related discussions in
https://github.com/coreos/coreos-assembler/issues/159.

Closes: #1822
Approved by: cgwalters
2019-04-30 20:37:30 +00:00
Jonathan Lebon
83a267409d Add new D-Bus APIs for deployment finalization
Teach `UpdateDeployment` to make use of libostree's staging lock and
then add a `FinalizeDeployment` API to perform the final unlock &
reboot.

I also added a hidden CLI to make testing this easier, but also because
it's likely the FCOS-agent-yet-to-be-named will just end up using the
CLI to keep it simple.

Closes: #1748

Closes: #1814
Approved by: lucab
2019-04-30 08:36:28 +00:00
Jonathan Lebon
9cb1f612de libdnf: Bump for GPG subkeys fix
See https://github.com/projectatomic/rpm-ostree/issues/1094#issuecomment-480826305
and https://github.com/rpm-software-management/libdnf/pull/711.

Update submodule: libdnf

Closes: #1819
Approved by: cgwalters
2019-04-28 14:15:11 +00:00
Jonathan Lebon
3d18c417b4 core: Also suggest upgrade for base/layered split
In the case of Fedora Silverblue which has daily composes, doing
`rpm-ostree cleanup -m && rpm-ostree upgrade` would work around the
majority of incidences here.

It doesn't apply if one is pinning to a specific version for whatever
reason, but that's not the common case.

Closes: #1818
Approved by: rfairley
2019-04-26 18:29:21 +00:00
Jonathan Lebon
97bb57d7a7 daemon: Carry option GVariant into transaction type
Rather than using flags, use the new approach of just carrying the
GVariant from the D-Bus message all the way to inside the transaction
type.

Closes: #1816
Approved by: cgwalters
2019-04-26 12:56:37 +00:00
Jonathan Lebon
6334aebca1 app: Simplify option variant creation
Use the new style of creating option variants directly instead of
through a helper.

Closes: #1816
Approved by: cgwalters
2019-04-26 12:56:37 +00:00
Jonathan Lebon
27c4e40368 app: Drop ex reset prototype
This was long promoted out of experimental.

Closes: #1816
Approved by: cgwalters
2019-04-26 12:56:37 +00:00
Jonathan Lebon
d2df455238 app/main: Make rpmostree_subcommand_parse local
It's not used elsewhere. Other commands with subcommands use
`rpmostree_handle_subcommand` instead which makes use of it.

Also drop the unused `invocation` arg.

Closes: #1816
Approved by: cgwalters
2019-04-26 12:56:37 +00:00
Jonathan Lebon
6b2ac5885f rust: Stop using extern crate and #[macro_use]
Rust 2018 makes this much nicer now.

Closes: #1812
Approved by: lucab
2019-04-15 13:03:21 +00:00
Jonathan Lebon
278ba77774 rust: Reduce visibility of items
Use `pub(crate)` on the ffiutils functions since it's just for us. Drop
unnecessary `pub` fields on `TreefileExternals`.

Closes: #1812
Approved by: lucab
2019-04-15 13:03:21 +00:00
Jonathan Lebon
60e71dfc66 rust: Update to 2018 edition
This mostly involves adjusting our path uses to be anchored, e.g. at
`self::` or `crate::`. I took the opportunity to also tweak `use`
ordering so that we consistently list modules in external crates first
before our own.

Closes: #1812
Approved by: lucab
2019-04-15 13:03:21 +00:00
Jonathan Lebon
0da9f9972b tests/vmcheck: Adapt to kernel v5.0
Closes: #1810
Approved by: jlebon
2019-04-05 19:28:39 +00:00
Jonathan Lebon
749facba86 rust: Bump dependencies
Now that our minimum Rust toolchain version was bumped, we can bump all
our deps to the latest and drop the frozen `rand` version that was done
for #1777.

Closes: #1810
Approved by: jlebon
2019-04-05 19:28:39 +00:00
Colin Walters
633f173484 scripts: Update fedora-release-workstation lua override
Prep for us to switch to unified-core only.  Note this is
dead code now in f30+.

Closes: #1808
Approved by: jlebon
2019-04-01 14:06:00 +00:00
Colin Walters
fc27c6ba2b core: Add a hack for /etc/selinux/config and %pre
I'm not very proud of this code, but I'm also not yet convinced that
we should rework things to re-interleave pre/post for just this
one thing.  Particularly when there's a patch to fix it
that just needs to be applied:
https://src.fedoraproject.org/rpms/selinux-policy/pull-request/16#

But practically speaking this fixes using `coreos-assembler`
for me with Silverblue and `tpm2-abrmd-selinux`.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1804

Closes: #1806
Approved by: jlebon
2019-03-29 20:28:13 +00:00