Commit Graph

1132 Commits

Author SHA1 Message Date
Colin Walters
436a8116f2 status: Sort the output package list
I am thinking of indenting into multiple lines if it gets long too,
but this is a small visual improvement.

Closes: #440
Approved by: jlebon
2016-08-30 20:09:24 +00:00
Colin Walters
363e213f18 upgrader: Avoid cleaning system repo twice
We need to manage our baselayer refs independently of the deployment
being written by libostree, so suppress the default clean; we'll do
it in rpm-ostree.

I noticed the `Freed objects:` message twice and always wondered why
until today.

Closes: #437
Approved by: jlebon
2016-08-30 19:53:24 +00:00
Colin Walters
2bc8d7cccb upgrader: Prune pkgcache repo
Until now, we weren't pruning the pkgcache repo at all.  I ran out of
space in the root partition in my CAHC vagrant test box, so it's time
to fix this.

The basic algorithm is to walk over the full rpmdb contents of each
root, generate a set of "currently referenced" cached refs, then delete
any refs in the pkgcache repo which aren't included.  Then, do a prune
of the pkgcache repo.

While we're here, factor out a `sysroot_upgrader_cleanup()` function
which does all of the cleanup.  The idea is at some point we need to
introduce an `rpm-ostree cleanup` command or so which calls this, to
handle the case where the system is interrupted post-deploy but
pre-clean.

Closes: https://github.com/projectatomic/rpm-ostree/issues/428

Closes: #437
Approved by: jlebon
2016-08-30 19:53:24 +00:00
Colin Walters
334cec56a0 upgrader: Use a temporary baseref, rather than walking twice
In preparation for future changes here, rework things so
that we only walk the set of deployments once after the new
deployment has been generated.

To avoid a race where if we're interrupted after the deployment write
but before the regeneration, create a special `rpmostree/base/tmp` ref
that we delete after all the bases are regenerated.

Closes: #437
Approved by: jlebon
2016-08-30 19:53:24 +00:00
Colin Walters
35d84f7507 pkglayering: Commit in base repo, not pkgcache
Currently, we do the final commit into the pkgcache repo, then
pull it to the base.  The problem with this is that, combined
with the fact that we're not presently pruning the pkgcache repo,
we leak space.

In preparation for a cleaner fix for this, rework things so that the
core infra can know about *both* a base repo and a pkgcache repo.  If
they're separate (as is the case for rpm-ostree-on-host), whenever we
are doing layering, explicitly pull just the `.file` objects that are
referenced into the parent repo.  We do the final commit in the base
repo.

Closes: #437
Approved by: jlebon
2016-08-30 19:53:24 +00:00
Jonathan Lebon
accd2b9f6f vmcheck: add a test for RPM scripts
Thought it'd be fun to write a test for verifying proper handling of
scriptlets during package layering. There's obviously a lot more that
could go in here (patches welcome!), but it's a start.

Closes: #434
Approved by: cgwalters
2016-08-17 21:28:01 +00:00
Jonathan Lebon
83ee4ee638 testenv: add builddir to PATH
Closes: #434
Approved by: cgwalters
2016-08-17 21:28:01 +00:00
Jonathan Lebon
0fe12fe6a3 mutate-os-release: skip VERSION_ID
I hit this with librepo subbing out the $releasever with e.g. 7.2016.1
when trying to pull various URLs. It should be enough for the user to
see the ostree version in VERSION and PRETTY_NAME. For applications,
there's OSTREE_VERSION if they need just that.

Closes: #433
Approved by: cgwalters
2016-08-17 20:45:29 +00:00
Colin Walters
589a8e0974 core: Neuter systemctl during script execution
`systemctl start/stop/try-restart` are all useless for us in
scripts, since changes should only affect the *next* boot.

`systemctl enable` is also wrong - one should use presets instead.

Currently, systemd has code to detect whether it's inside
a chroot, which works for mock, but *not* for Docker or bubblewrap.
(We should teach systemd a nicer way to disable itself, but
 even if we did that we'd have to support old scripts)

So, this fixes layering `glusterfs` in CAHC.

Closes: #432
Approved by: jlebon
2016-08-17 20:41:36 +00:00
Colin Walters
6900c616ec scripts: Execute %pre
Now that we handle `useradd`, which is most of what anyone wants
to do in `%pre`, let's turn it on.

Closes: #432
Approved by: jlebon
2016-08-17 20:41:36 +00:00
Colin Walters
550adecb47 core: Support scripts which call useradd
When executing scripts, map `/usr/lib/passwd` temporarily back to
`/etc/passwd` from the POV of the script, then move it back.  This
allows client-side layered packages be merged with the base tree
version, while still being distinct from the `/etc/passwd`.

(In the future we'll likely rewrite all of this to use
`systemd-sysusers(8)`, but for now let's be incremental.)

Note in this commit it isn't very useful, because pretty much all RPMs
only call `useradd` in `%pre`, which is the next commit.

Closes: #432
Approved by: jlebon
2016-08-17 20:41:36 +00:00
Colin Walters
dbd2c0ee7a unpacker: Synthesize tmpfiles.d for nonroot-owned /run and /var files
Because OSTree includes numeric uid/gid, but package layering (and
traditional full-client-side RPM) expects to allocate UIDs on the
machine, we need to handle non-root owned files specially.

For files in /run and /var, we can convert them to
`systemd-tmpfiles(8)` snippets which will be executed at the next
boot, after a uid/gid has been assigned.

Closes: #432
Approved by: jlebon
2016-08-17 20:41:36 +00:00
Colin Walters
3ad4e6c72b bwrap/compose: Add a workaround for Fedora's use of rpm-ostree-in-mock
Decided to test this on Sunday evening.  Of course it was broken =(
(Actually I tested mock-in-Docker but it should be the same)

The core problem is that mock does `chroot()` without using `/`
as a mount point.  This breaks an assumption in bwrap that it is.
Now, in theory we could move this same logic down into bwrap to
work around this situation, but for now let's hack it here.

Mock is old, legacy container code that doesn't really do anything
in a modern way - in fact our goal should be to replace it
with a combination of rpm-ostree and bwrap.  So carrying this
hack here to get us to that future should be OK for now.

Closes: #431
Approved by: jlebon
2016-08-15 18:31:58 +00:00
Colin Walters
09c5f9a4fe Release 2016.7
Closes: #430
Approved by: cgwalters
2016-08-13 14:34:38 +00:00
Colin Walters
7455e26273 bwrap: Add a selftest
I want a better error message if the user happens to execute
inside e.g. a Docker container without sufficient privileges
for recursive containerization.

Closes: #429
Approved by: jlebon
2016-08-12 23:34:33 +00:00
Colin Walters
551e4c91f9 postprocess: Switch to using bwrap for script execution
The previous commit https://github.com/projectatomic/rpm-ostree/pull/422
introduced a regression in the "outside of a container" path - we
get `EINVAL` trying to `mount("proc",...)` and honestly I'm not sure why.

We can either back up or plow forward, and it turns out to be
pretty straightforward to complete the port to using bwrap.

I extracted the bwrap-execution code out of the RPM script engine,
because the treecompose model is currently different (no hardlinks
yet).

NOTE: A *very* important side effect of this is that we now
require "privileged" containers on hosts without user namespaces,
and on userns hosts, require `CLONE_NEWUSER` to be exported to the
container host.

In general though, the previous path of blindly executing scripts as root
without e.g. `proc` mounted was just bad.

Closes: #429
Approved by: jlebon
2016-08-12 23:34:33 +00:00
Colin Walters
88a130da09 lib: Extract bwrap-executing internal API
The treecompose code will learn how to use bwrap instead of
libcontainer in libglnx, since the latter is a buggy copy of a subset
of the former.

Closes: #429
Approved by: jlebon
2016-08-12 23:34:33 +00:00
Colin Walters
873f4766d7 Release 2016.6
Closes: #427
Approved by: jlebon
2016-08-11 20:22:54 +00:00
Colin Walters
a299083b09 core: Fix an error-handling logic error in relabel path
If next_dent returns `FALSE`, we need to also return `FALSE` from
the function.

I just noticed this as I happened to be reading the code randomly.

Closes: #426
Approved by: jlebon
2016-08-11 19:40:29 +00:00
Colin Walters
19490c593e Update to latest libdnf
I want the excludes= fix, and the memleak fixes are nice.

Closes: #425
Approved by: jlebon
2016-08-11 19:26:06 +00:00
Colin Walters
56a4f73901 scripts: Error out if we encounter lua
Otherwise we try to execute as shell script which obviously doesn't
work.  This was noticed when @dustymabe was trying to assemble
a Fedora container with `glibc-all-langpacks`.

Closes: #424
Approved by: jlebon
2016-08-11 15:32:49 +00:00
Colin Walters
aa2295db26 compose: Add --cache-only
When trying to test treecomposes, it's annoying to hit the
metadata servers each time.  Add a `--cache-only` option which
is kind of like yum's `-C` option (but not quite because
if the metadata doesn't exist we will update it).

Closes: #423
Approved by: jlebon
2016-08-09 15:18:14 +00:00
Colin Walters
3d1040a780 compose: Port bits of postprocessing to fd-relative
I was looking at starting to unify the treecompose path to the
"new world" infrastructure in package layering.  An initial
step here is to port to fd-relative, which cleans up the code.

Note this depends on a libglnx pull.

Closes: #422
Approved by: jlebon
2016-08-08 18:11:42 +00:00
Colin Walters
ff8e2992da libglnx porting: Switch to glnx_file_get_contents_utf8_at()
A small one, but a start.

Closes: #421
Approved by: jlebon
2016-08-08 18:07:46 +00:00
Colin Walters
0c8f6b982e build: Import attributes.m4 copy from systemd, update warnings
This matches https://github.com/ostreedev/ostree/pull/431
I also merged in the `-Wint-conversion` here since we build fine
with it now.

Closes: #420
Approved by: jlebon
2016-08-08 14:26:28 +00:00
Colin Walters
dee758e111 gitmodules: Update for rename
rpmdistro-gitoverlay doesn't like when the submodule name differs
from its path, so let's make it happy.

Closes: #419
Approved by: giuseppe
2016-08-04 22:10:59 +00:00
Colin Walters
a852ad3459 Fix accidental double __ in previous commit
My sed powers got overly enthusiastic.

Closes: #417
Approved by: giuseppe
2016-08-04 12:33:07 +00:00
Colin Walters
e18d43bad3 Port to new ostree_repo_checkout_at()
Squashes some deprecation warnings, and now with the latest master we
can stop setting `disable_fsync`.

Closes: #418
Approved by: giuseppe
2016-08-04 12:21:38 +00:00
Colin Walters
39913a2c25 Update for libhif -> libdnf
Still not many PRs outstanding, so it's a good time to take this plunge.
Mostly automated via `sed`.

Closes: #416
Approved by: Conan-Kudo
2016-08-04 01:10:08 +00:00
Colin Walters
c4e98f3f47 compose: Print progress during ostree commit
Until we finally do the "store packages as ostree commits and union"
thing, doing commits can be slow, particularly into archive repos
where we pay lots of cost in gzip.

Let's show a progress bar.  The implementation here uses a background
thread which communicates with the "UI" via atomics.  The UI uses a
timer - and if stdout isn't a tty, we assume it's Jenkins or something
and dial updates back to every 5 seconds to avoid spamming output.

Closes: #409
Approved by: giuseppe
2016-08-01 15:55:12 +00:00
Colin Walters
563fcd5750 compose: Print stats at end of commit
`ostree commit --table-output` can already do this, and since we're
super noisy for `compose tree` right now, we might as well print this.
It's useful to see how many new objects were created.

Closes: #409
Approved by: giuseppe
2016-08-01 15:55:12 +00:00
Colin Walters
c9c1483f58 postprocess: Also pick up user.pax.flags
This is in use by [PaX](https://en.wikipedia.org/wiki/PaX); see also
the [Arch Linux wiki](https://wiki.archlinux.org/index.php/PaX).

Closes: #412

Closes: #414
Approved by: copumpkin
2016-07-29 18:39:26 +00:00
Jonathan Lebon
b3c3c029f0 compose: add "mutate-os-release" treefile key
Users/administrators can now enable the "mutate-os-release" string
key to tell rpm-ostree to modify /etc/os-release in the following ways:

- All occurrences of the key's value found in VERSION, VERSION_ID, and
  PRETTY_NAME will be replaced by the version string of new compose.
- A new OSTREE_VERSION key is appended containing the version string of
  the new compose.

This provides an easy way for clients and third-party applications to
find out what version they are currently running.

Also bump libglnx so that we can use the latest version of
glnx_file_replace_contents_at() which supports AT_FDCWD.

Closes: #410
Approved by: cgwalters
2016-07-28 00:29:41 +00:00
Colin Walters
89f7e481b4 Release 2016.5
Closes: #408
Approved by: cgwalters
2016-07-25 17:01:08 +00:00
Jonathan Lebon
8d844c2c39 pkg-layering: print transaction on dry run
Commit d153ece removes redundant transaction printing, but we do still
want to print it manually when we're doing a dry run.

Closes: #407
Approved by: cgwalters
2016-07-25 15:06:39 +00:00
Colin Walters
d153ecec9d Don't print transaction in middle of upgrade overlay output
...but still do print it for `compose tree` and `container`.  This
is more followup to #403, where I noticed we were printing
just the overlay transaction in the middle of `Overlaying...`.

Since we're now printing the package diff again at the end, let's just
rely on that rather than show the user over and over which packages
they requested.

Closes: #403

Closes: #404
Approved by: jlebon
2016-07-25 14:37:34 +00:00
Colin Walters
8be495273e upgrader: Drop "Need to overlay" output
This came up in #403, but looking at this more, I don't see a need to
print out the input package set here.  The list of overlaid packages
is very clear in `rpm-ostree status`, and the user knows what they
typed on the command line.

Closes: https://github.com/projectatomic/rpm-ostree/issues/403

Closes: #404
Approved by: jlebon
2016-07-25 14:37:34 +00:00
Colin Walters
4ccee066a0 Add gpg-enabled to DBus data, hide GpgSignature field if disabled
It's really in-your-face for admins to see `(unsigned)` which originally
was intentional, but in practice right now at least several major
users (Fedora, RHEL) don't enable GPG signatures.

So let's just hide it if not enabled.

Perhaps in the future we should add a `-v` option or something which
would show all of the fields even if not enabled.

Closes: #399

Closes: #402
Approved by: jlebon
2016-07-21 20:52:28 +00:00
Colin Walters
d2d0b21bff sysroot: Reload when repo changes (not just sysroot)
Since some of the DBus data like `CachedUpdate/commit` derives solely
from the ostree repo state, and `upgrade --check` only writes to the
repo, we need to watch for and reload from changes there too.

Closes: #392

Closes: #400
Approved by: jlebon
2016-07-21 20:44:28 +00:00
Colin Walters
803ebebc21 daemon: Do resolve rev for deployment variant details
Previously in #353 we changed this to not resolve the rev, but
that breaks the `CachedUpdateDetails` which actually wants it
to be resolved, to show pending updates.

This effectively reverts 749a5f03a8586392cafff949075c45a040d4e984
and changes things so we set `allow_noent=TRUE` and if there is
no ref, just return the deployment checksum.

Fixes: 749a5f03a8586392cafff949075c45a040d4e984
Closes: #392

Closes: #398
Approved by: jlebon
2016-07-18 18:12:29 +00:00
Colin Walters
ce73e4d9cc daemon: Clean up CachedUpdate GError usage
In looking at this code, I noticed we were incorrectly mixing
`Option(GVariant)` with GError handling.  If the deployment
had no refspec we'd return `NULL` but not set `error`.

Fix this by requiring a respec, and change the single caller that
relied on this to check outside.

Closes: #398
Approved by: jlebon
2016-07-18 18:12:29 +00:00
Colin Walters
454139d41b upgrade: Print changed packages again
First I noticed we needed to query the new default *after* the
transaction had run.  Then, I noticed the obvious embarassing logic
error.

Fixes: 3f1e7c85a18182669899f4065d23fec6ac0962c
Closes: #396

Closes: #397
Approved by: jlebon
2016-07-18 14:10:50 +00:00
Colin Walters
64c6f8d906 docs: Separate README.md and docs index, elaborate more
Have the index page be rationale/comparison for now.

Closes: #370
Approved by: jlebon
2016-07-16 02:16:09 +00:00
Jonathan Lebon
40a44a561c vagrant/Makefile: drop sudo and depend on buildimg
Closes: #394
Approved by: cgwalters
2016-07-15 19:14:19 +00:00
Jonathan Lebon
52b325970d vmcheck: move vmbuild and vmoverlay to scripts
That way we can make use of libvm instead of relying on vagrant.

Closes: #394
Approved by: cgwalters
2016-07-15 19:14:19 +00:00
Jonathan Lebon
194c2bf5cd libvm.sh: add vm_setup and vm_rsync
In preparation for being vagrant independent, let's factor out some
things directly into the libvm.

Closes: #394
Approved by: cgwalters
2016-07-15 19:14:19 +00:00
Jonathan Lebon
8942268ecb test.sh: write test result to log for easier grepping
Closes: #394
Approved by: cgwalters
2016-07-15 19:14:19 +00:00
Jonathan Lebon
ab6fe314aa vmcheck/test.sh: support external ssh-config
Closes: #394
Approved by: cgwalters
2016-07-15 19:14:19 +00:00
Jonathan Lebon
ce2a434104 libtest.sh: output file in assert_file_has_content
This is the same patch as:
https://github.com/ostreedev/ostree/pull/264

Closes: #394
Approved by: cgwalters
2016-07-15 19:14:19 +00:00
Jonathan Lebon
2da585023d vmcheck: remove test-basic.sh
This is a useless test for now since we should be able to test on a VM
even if there are updates available.

I initially had it just as an example of the vmcheck test harness,
without actually thinking too much into what it was testing. But we do
need a nice test-basic.sh that will give rpm-ostree a good general
workout.

Closes: #394
Approved by: cgwalters
2016-07-15 19:14:19 +00:00