Commit Graph

72 Commits

Author SHA1 Message Date
Jonathan Lebon
5c5ebc1b0a core: add support for installing from cache
We start by adding support in the core for installing packages strictly
from the cache repo. We fool the libdnf stack by re-exporting the header
as an RPM, and explicitly marking it for install. The treefile format
supports specifying the expected SHA-256 of the metadata header, in case
the cache for a specific NEVRA changed.

Closes: #657
Approved by: cgwalters
2017-03-10 18:35:11 +00:00
Jonathan Lebon
6a5d4a2699 RpmOstreeOrigin: also cache initramfs args
Ideally, we want to parse out everything in the origin on init and then
cache the values to make it even easier to inspect it. We make this more
explicit by renaming the struct members to "cache_*" and also caching
the initramfs args. This doesn't actually save any allocations right
now (though it might in the future), but we do the same for everything
else, and it's nice to be consistent.

Closes: #646
Approved by: cgwalters
2017-02-27 19:53:19 +00:00
Colin Walters
cc20f32fb7 tree-wide: Use autoptr cleanup for HyQuery
This was a noticeable source of leaks.

Closes: #644
Approved by: jlebon
2017-02-24 18:55:36 +00:00
Chen Fan
7ee41f75b7 remove duplicate words in comment
Signed-off-by: Chen Fan <fan.chen@easystack.cn>

Closes: #635
Approved by: cgwalters
2017-02-21 14:47:59 +00:00
Jonathan Lebon
c756b26521 compose: fix mutate-os-release handling
- Let --add-metadata-string=version=val override any automatic version
  prefixing.
- Don't error out if mutate-os-release is given, but no new version is
  given by --add-metadata-string or automatic version prefixing.

Checking keys in parse_keyvalue_strings() is slightly hacky. I initially
wanted to just inspect the GVariantBuilder, but AFAICT, there is no way
to actually look up values from a builder (plus, we need that info early
to know whether automatic_version_prefix should itself inject in the
builder or not).

Closes: #603
Approved by: cgwalters
2017-02-01 22:25:54 +00:00
Colin Walters
cac4522e5b Support "system/regenerate-initramfs=true" flag in origin
Currently we push for a model where the initramfs is
generated (in non-hostonly mode), and merely replicated.

However, to support a few unfortunate corner cases like dm-multipath which wants
to inject a config file into the initramfs, we need to support regenerating it
client side too.

Down the line, we'll need this to support overriding the kernel too.

This changes things in the core to add the concept of an "empty"
`RpmOstreeContext`. I initially tried skipping it, but that was too much
duplication. We still want all of the core ostree-related logic that lives in
that code too.

The treespec bits barfed if the spec didn't have a `tree/packages` key. It was
simplest to change that to allow it - and because that was the only case where
we errored out in parsing, I dropped the error handling.

There was another place in the upgrader that now needed to be fixed to handle
transitioning from just regenerating initramfs to not.

Closes: #574
Approved by: jlebon
2017-01-20 16:17:51 +00:00
Colin Walters
f215acb64f libpriv: Move kernel finalization into a helper function
This will need to be callable outside of postprocess for client side initramfs
generation.

Closes: #566
Approved by: jlebon
2017-01-13 20:08:24 +00:00
Colin Walters
4c98943a64 postprocess: Clean up kernel/initramfs checksum calculation
This avoids having an intermediate file path, and makes the code easier to
follow.

Closes: #566
Approved by: jlebon
2017-01-13 20:08:24 +00:00
Colin Walters
16efa12ae0 libpriv: Factor out "find kernel/initramfs" code into kernel.[ch]
To support running dracut on the client side, the dracut code
needs this, and it makes more sense in kernel.[ch] anyways.

I chose to use a GVariant instead of making a custom structure to avoid having
to manage custom free funcs.

Closes: #566
Approved by: jlebon
2017-01-13 20:08:24 +00:00
Colin Walters
e1cdfa1712 libpriv: Introduce an rpmostree-kernel.[ch] for dracut
Since the dracut run is now separated from the rest of postprocess, we change it
to use the bwrap API diretly, and this lets use the new _IMMUTABLE bwrap type.
This will make it easier to reuse for client-side initramfs regeneration.

Splitting this off makes it also easier to simplify the remaining mutable
usage in postprocess.c.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:26 +00:00
Colin Walters
f0ec738376 libpriv: Enhance internal bwrap API
Make it a real struct with methods.  This noticeably increases
the ergonomics and design of the API.

The main goal here is to introduce the enum which defines whether or not the
rootfs is mutable or not. We move the "rofiles" mode from the RPM script code
down into the bwrap layer, which will make it easier to reuse for treecompose.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:26 +00:00
Colin Walters
50ab2983ab postprocess: Use O_TMPFILE for dracut, merge reproducible bits
I was planning to do some further changes here, and I really don't like the
manual fork/exec stuff on in the --reproducible checks. Our subprocess code
should basically be all bwrap.  Synchronous code execution while not reading from the
pipe is a recipe for deadlocks.

What simplifies things a lot is to write to an `O_TMPFILE` fd (or a tempfile on
legacy kernels), and slightly extend our bwrap-executing code to support a child
setup function, so we can set the tmpfile fd to be stdout.

Now that we have a shell script wrapper we inject, it's trivial to reimplement
the "detect reproducibility" in shell script there, rather than C.

This doesn't matter much for treecompose today, but it will matter more when
we're supporting client side initramfs regeneration, since now the dracut
container can be fully immutable.

Closes: #560
Approved by: jlebon
2017-01-11 18:39:25 +00:00
Jonathan Lebon
0e431a8f83 core: add RPMOSTREE_USE_CACHED_METADATA
Being able to just reuse metadata is especially helpful when trying to
debug things lower down that path, as well as cuts about 2 mins from the
compose test.

Closes: #561
Approved by: cgwalters
2017-01-08 21:05:06 +00:00
Colin Walters
3a409342e6 postprocessing: Add a g_prefix_error for kernel
Saw this un-prefixed error path in a build; having error prefixes makes for
easier debugging.

Closes: #554
Approved by: giuseppe
2016-12-20 16:24:25 +00:00
Colin Walters
69cace4ce8 [ASAN] Squash a few minor misc leaks
This is part of the leak set when doing a treecompose.  There's
some that go down into libdnf...will get to that over time.

Closes: #547
Approved by: jlebon
2016-12-08 16:27:24 +00:00
Colin Walters
a347ff5f69 postprocess: Port remaining libgsystem call sites
Except for `gs_file_get_path_cached()`.  We have to carry
the recursive hardlinking code, oh well.

Closes: #511
Approved by: jlebon
2016-11-08 14:38:54 +00:00
Colin Walters
5c8d80042c postprocess: More fd-relative-ification
I hit a bit of a wall here around `gs_shutil_cp_al_or_fallback()`.
I guess we'll have to reimplement it here.

In the meantime, this gets us closer.

Closes: #510
Approved by: jlebon
2016-11-04 14:59:27 +00:00
Colin Walters
634a475ae4 postprocess: Drop a libgsystem use in add-files
I'm cowardly just using gs_file_get_path_cached() for now.

Closes: #510
Approved by: jlebon
2016-11-04 14:59:27 +00:00
Colin Walters
e0357f41d1 postprocess: Port tmpfiles.d-copying code to fd-relative
Drops libgsystem use.

Closes: #510
Approved by: jlebon
2016-11-04 14:59:27 +00:00
Colin Walters
68898217ed postprocess: Drop use of gs_file_rename(), use fd-relative
Another libgsystem use gone.

Closes: #510
Approved by: jlebon
2016-11-04 14:59:27 +00:00
Colin Walters
7ee4edf8e3 tree-wide: Remove more unused variables
Now that the other PRs are merged, kill of the rest of the unused
variables.

Closes: #504
Approved by: jlebon
2016-10-28 18:25:33 +00:00
Colin Walters
5100506909 compose: Port a large chunk of postprocessing to fd-relative
Part of the de-libgsystem-ification porting.

Closes: #498
Approved by: jlebon
2016-10-27 14:57:43 +00:00
Colin Walters
1c2908cbef tree-wide: Port away from gs_transfer_out_value()
Using `g_steal_pointer()` is a little bit more typing, but less
magical which is good.

Closes: #498
Approved by: jlebon
2016-10-27 14:57:42 +00:00
Colin Walters
49fffad471 Run bwrap with fixed environment (PATH + LANG)
This is basically a re-implementation of
85c9dd5c07
for the same reasons: it makes things work for rpm-ostree running
inside NixOS.  But there are other cases where if e.g. an unprivileged
user runs rpm-ostree and may not have `/usr/sbin` in PATH, we still
want the container to pick it up, etc.

While changing things I nuked the hacky `RPMOSTREE_DEBUG_SCRIPT` env
var, we can re-add it later in a better way if needed.

Closes: #490
Approved by: jlebon
2016-10-17 20:29:35 +00:00
Patrick Uiterwijk
98332a3be4 compose: Add option to write commitid to file instead of updating ref
Closes: #492
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>

Closes: #494
Approved by: cgwalters
2016-10-17 20:02:55 +00:00
Colin Walters
4542778648 tree-wide: Port to g_file_enumerator_iterate()
Unlike OSTree, I'm going to assert we can take a hard dependency
on the version of GLib that has it.

Closes: #483
Approved by: jlebon
2016-10-17 19:59:40 +00:00
Colin Walters
4ccd31fcb5 tree-wide: Port to libglnx for shutil_rm_rf
Closes: #483
Approved by: jlebon
2016-10-17 19:59:40 +00:00
Colin Walters
b47301b3e1 tree-wide: Port to libglnx autocleanup macros
Closes: #483
Approved by: jlebon
2016-10-17 19:59:40 +00:00
Colin Walters
a15364c185 compose: Respect absolute paths for postprocess-script again
Regressed from some code refactoring.

Closes: #491
Approved by: jlebon
2016-10-14 17:29:43 +00:00
Colin Walters
4a86bdd196 postprocess: Undo Fedora SELinux policy move
Basically, this brings us back to where we are with RHEL/CentOS 7.

For lots more information, see
https://bugzilla.redhat.com/show_bug.cgi?id=1290659

Closes: #479
Approved by: jlebon
2016-10-07 15:57:15 +00:00
Colin Walters
1281434cec postprocess: Print absolute path for ignored files in /var
For ease of debugging.  Maybe in the future we try to look them up in
the RPM database too.

Closes: #473
Approved by: jlebon
2016-10-04 13:28:11 +00:00
Colin Walters
0cd584ccf6 postprocess: Silently clean up known-unnecessary files in /var
Per OSTree design, /var can start out empty.  However, our warning
spam here is annoying.  Let's first delete some known files -
obviously this won't be exhaustive, but it's way faster than trying to
fix all of this in the packages right now.

The major one is the SELinux policy, which resulted in a lot of spam.

Closes: #473
Approved by: jlebon
2016-10-04 13:28:11 +00:00
Jonathan Lebon
0fe12fe6a3 mutate-os-release: skip VERSION_ID
I hit this with librepo subbing out the $releasever with e.g. 7.2016.1
when trying to pull various URLs. It should be enough for the user to
see the ostree version in VERSION and PRETTY_NAME. For applications,
there's OSTREE_VERSION if they need just that.

Closes: #433
Approved by: cgwalters
2016-08-17 20:45:29 +00:00
Colin Walters
551e4c91f9 postprocess: Switch to using bwrap for script execution
The previous commit https://github.com/projectatomic/rpm-ostree/pull/422
introduced a regression in the "outside of a container" path - we
get `EINVAL` trying to `mount("proc",...)` and honestly I'm not sure why.

We can either back up or plow forward, and it turns out to be
pretty straightforward to complete the port to using bwrap.

I extracted the bwrap-execution code out of the RPM script engine,
because the treecompose model is currently different (no hardlinks
yet).

NOTE: A *very* important side effect of this is that we now
require "privileged" containers on hosts without user namespaces,
and on userns hosts, require `CLONE_NEWUSER` to be exported to the
container host.

In general though, the previous path of blindly executing scripts as root
without e.g. `proc` mounted was just bad.

Closes: #429
Approved by: jlebon
2016-08-12 23:34:33 +00:00
Colin Walters
3d1040a780 compose: Port bits of postprocessing to fd-relative
I was looking at starting to unify the treecompose path to the
"new world" infrastructure in package layering.  An initial
step here is to port to fd-relative, which cleans up the code.

Note this depends on a libglnx pull.

Closes: #422
Approved by: jlebon
2016-08-08 18:11:42 +00:00
Colin Walters
ff8e2992da libglnx porting: Switch to glnx_file_get_contents_utf8_at()
A small one, but a start.

Closes: #421
Approved by: jlebon
2016-08-08 18:07:46 +00:00
Colin Walters
39913a2c25 Update for libhif -> libdnf
Still not many PRs outstanding, so it's a good time to take this plunge.
Mostly automated via `sed`.

Closes: #416
Approved by: Conan-Kudo
2016-08-04 01:10:08 +00:00
Colin Walters
c4e98f3f47 compose: Print progress during ostree commit
Until we finally do the "store packages as ostree commits and union"
thing, doing commits can be slow, particularly into archive repos
where we pay lots of cost in gzip.

Let's show a progress bar.  The implementation here uses a background
thread which communicates with the "UI" via atomics.  The UI uses a
timer - and if stdout isn't a tty, we assume it's Jenkins or something
and dial updates back to every 5 seconds to avoid spamming output.

Closes: #409
Approved by: giuseppe
2016-08-01 15:55:12 +00:00
Colin Walters
563fcd5750 compose: Print stats at end of commit
`ostree commit --table-output` can already do this, and since we're
super noisy for `compose tree` right now, we might as well print this.
It's useful to see how many new objects were created.

Closes: #409
Approved by: giuseppe
2016-08-01 15:55:12 +00:00
Colin Walters
c9c1483f58 postprocess: Also pick up user.pax.flags
This is in use by [PaX](https://en.wikipedia.org/wiki/PaX); see also
the [Arch Linux wiki](https://wiki.archlinux.org/index.php/PaX).

Closes: #412

Closes: #414
Approved by: copumpkin
2016-07-29 18:39:26 +00:00
Jonathan Lebon
b3c3c029f0 compose: add "mutate-os-release" treefile key
Users/administrators can now enable the "mutate-os-release" string
key to tell rpm-ostree to modify /etc/os-release in the following ways:

- All occurrences of the key's value found in VERSION, VERSION_ID, and
  PRETTY_NAME will be replaced by the version string of new compose.
- A new OSTREE_VERSION key is appended containing the version string of
  the new compose.

This provides an easy way for clients and third-party applications to
find out what version they are currently running.

Also bump libglnx so that we can use the latest version of
glnx_file_replace_contents_at() which supports AT_FDCWD.

Closes: #410
Approved by: cgwalters
2016-07-28 00:29:41 +00:00
Colin Walters
cc2551dc06 Switch to glib cleanup macros
We don't have a lot of outstanding changes to the C code, so now seems
like a good time to do this.  I implemented this with some highly
sophisticated sed commands like:

```
find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done
```

Part of dropping the dependency on libgsystem, same as what we're
doing in ostree.

Closes: #393
Approved by: jlebon
2016-07-15 13:08:58 +00:00
Jonathan Lebon
8016319bcb postprocess: fix typo in error msg
Closes: #381
Approved by: cgwalters
2016-07-07 21:54:57 +00:00
Colin Walters
c68ea0a29a core: Add /var/lib/vagrant -> /usr/lib/vagrant
I'm thinking this list of things actually needs to live outside of the
source code and be more obvious (and we should file bugs to get the
packages fixed), but in the short term my demo today uses Vagrant and
it'd be nice if it was installable.

Closes: #364
Approved by: jlebon
2016-06-30 15:02:42 +00:00
Colin Walters
f35f5cf468 core: Initial implementation of %posttrans using bwrap+rofiles-fuse
In order to make many things work, we need to run scripts.  Short version:
For now, we:

 - Run `%posttrans`
 - Treat most `%post` as the same as `%posttrans`
 - Ignore `%preun` and such since we never uninstall

Most importantly though, we start to build up an "override" list
for script handling.  Currently it's just a blacklist of scripts
we don't need.

Significant work here would be needed to run Lua scripts, so far I've
been able to just skip them.

Closes: #338
Approved by: jlebon
2016-06-24 16:08:08 +00:00
Jonathan Lebon
3073a42868 tmpfiles.d: rename and re-order
According to tmpfiles.d(5), files should follow the convention
<package>.conf or <package>-<part>.conf. So we rename
tmpfiles-ostree-integration.conf to rpm-ostree-0-integration.conf.

The 0 index is so that the autovar conf created by postprocess is
sourced *after* this one, so that `integration.conf` has higher
precedence if there are duplicate entries.

Closes: #325
Approved by: cgwalters
2016-06-15 20:41:45 +00:00
Jonathan Lebon
6b07755d68 add RPMOSTREE_UNINSTALLED_PKGLIBDIR
Since we now run everything uninstalled, we can't expect the tmpfiles
conf file to be installed. We add an env var that will allow us to tell
rpm-ostree to look elsewhere. This is then used in test-compose.sh.

Closes: #304
Approved by: cgwalters
2016-06-06 21:14:07 +00:00
Jonathan Lebon
ead1ecdd23 package layering: major rework
- Move the package layering logic away from pkg-add and into the
  upgrader
- Add pkg-delete
- Add dry-run option

Closes: #289
Approved by: cgwalters
2016-05-27 22:18:44 +00:00
Colin Walters
beb026f701 pkg-add: New builtin to layer additional packages
This builds upon the earlier prototype in
https://github.com/cgwalters/atomic-pkglayer

The `.origin` file says for a replicated installation:

    [origin]
    refspec=local:rhel-atomic-host/7/x86_64/standard

If you then run `rpm-ostree pkg-add strace`, it will result in a new tree with:

    [origin]
    baserefspec=local:rhel-atomic-host/7/x86_64/standard

    [packages]
    requested=strace;

Work still remaining here is to teach `rpm-ostree status` and
`rpm-ostree upgrade` about this.

Closes: #289
Approved by: cgwalters
2016-05-27 22:18:44 +00:00
Jonathan Lebon
d9e8535bcf treecompose: fix crash when "remove-from-packages" used
This works around a potential issue with libsolv if we go down the
rpmostree_get_pkglist_for_root() path. Though rpm has been using the
/usr/share/rpm location (since the RpmOstreeContext set the _dbpath
macro), the /var/lib/rpm directory will still exist, but be empty.
libsolv gets confused because it sees the /var/lib/rpm dir and doesn't
even try the /usr/share/rpm location, and eventually dies when it tries
to load the data.

So we set the symlink now. This is also what we do on boot anyway for
compatibility reasons using tmpfiles.

This also means we don't have to do the /var/lib/rpm --> /usr/share/rpm
transition during the rootfs postprocess (but we still have to clean up
db and lock files).

Also get rid of the unused pkglist variable.

NB: I used the GFile & gs APIs to mesh with the surrounding code.

Closes: #290
Approved by: cgwalters
2016-05-27 17:06:08 +00:00