Commit Graph

273 Commits

Author SHA1 Message Date
Colin Walters
0578e13de8 core: Set a useragent
See https://github.com/rpm-software-management/libhif/pull/140

I just noticed this while debugging HTTP requests.

Closes: #318
Approved by: giuseppe
2016-06-12 16:20:33 +00:00
Colin Walters
20e69a9692 status: Import systemd bits to use UTF-8 circle
It's slightly prettier, but this is just laying some
groundwork/precedent for importing more systemd code and using it for
our formatting.

Closes: #295
Approved by: jlebon
2016-06-09 19:33:19 +00:00
Jonathan Lebon
d99df468c7 libhif: always prefix include directives
This is in response to:
https://github.com/rpm-software-management/libhif/pull/138

Closes: #310
Approved by: cgwalters
2016-06-09 12:27:53 +00:00
Jonathan Lebon
6b07755d68 add RPMOSTREE_UNINSTALLED_PKGLIBDIR
Since we now run everything uninstalled, we can't expect the tmpfiles
conf file to be installed. We add an env var that will allow us to tell
rpm-ostree to look elsewhere. This is then used in test-compose.sh.

Closes: #304
Approved by: cgwalters
2016-06-06 21:14:07 +00:00
Colin Walters
3a93fc162f core: Checksum package checksums, not just NEVRA for change detection
GPG signing an RPM doesn't change its NEVRA, and we need to support
detecting the case when RPMs change from unsigned to signed (or vice
versa).

It's also quite common for local developers to rev RPMs without
bumping the release or whatever, so this will fix that too.

Closes: #291

Closes: #296
Approved by: jlebon
2016-06-02 18:40:32 +00:00
Colin Walters
93e4039f60 core: Dedup hardlink/tempfile code
I wanted to avoid yet another copy of the "generate tempfile name"
code, so moved it to libglnx:

  https://github.com/GNOME/libglnx/pull/14

This also closes the TODO about deduping the "break one hardlink" code
with the "break all links in one dir" code.  The core observation here
is that it's simpler to copy to a tempfile and rename over the
existing, rather than rename, create, unlink.

Closes: #293
Approved by: jlebon
2016-06-01 15:15:11 +00:00
Jonathan Lebon
ead1ecdd23 package layering: major rework
- Move the package layering logic away from pkg-add and into the
  upgrader
- Add pkg-delete
- Add dry-run option

Closes: #289
Approved by: cgwalters
2016-05-27 22:18:44 +00:00
Jonathan Lebon
a25cdde25c core: major rework
This patch prepares RpmOstreeContext for supporting package layering. A
relabel operation is added as well to support relabeling imported
packages if the sepolicy of the rootfs we're overlaying onto is
different from during import.

Closes: #289
Approved by: cgwalters
2016-05-27 22:18:44 +00:00
Jonathan Lebon
1ccc560c26 output: support printf type usage
Closes: #289
Approved by: cgwalters
2016-05-27 22:18:44 +00:00
Jonathan Lebon
ca162dece2 unpacker: major rework
- Delete unpack_to_dfd path
- Get rid of copynpaste stuff and use the newly reworked ostree
  libarchive API which now supports the callbacks we need

Closes: #289
Approved by: cgwalters
2016-05-27 22:18:44 +00:00
Jonathan Lebon
2a036cf8b4 package-layering fixups
This patch fixes up errors in the parent commit which added pkg-add.
Some of them are due to the rebase on top of the unprivileged
infrastructure.

- fix compile errors (due to libhif changes after rebase)
- delete duplicate prototype for rpmostree_sysroot_upgrader_deploy
- include allow-older in flags type
- fix change_upgrader_refspec to use g_strdup() (this was causing the
  wrong old refspec to be registered)
- in builtin-status.c, check for NULL before joining the packages array
- sysroot-upgrader: fix gtype function names
- roc_context_prepare_for_root(): delete unused param
- assemble_commit(): delete unused param
- RpmOstreeSysrootUpgraderFlags: fix docs
- Fix sysroot property name and add reboot opt

Closes: #289
Approved by: cgwalters
2016-05-27 22:18:44 +00:00
Colin Walters
beb026f701 pkg-add: New builtin to layer additional packages
This builds upon the earlier prototype in
https://github.com/cgwalters/atomic-pkglayer

The `.origin` file says for a replicated installation:

    [origin]
    refspec=local:rhel-atomic-host/7/x86_64/standard

If you then run `rpm-ostree pkg-add strace`, it will result in a new tree with:

    [origin]
    baserefspec=local:rhel-atomic-host/7/x86_64/standard

    [packages]
    requested=strace;

Work still remaining here is to teach `rpm-ostree status` and
`rpm-ostree upgrade` about this.

Closes: #289
Approved by: cgwalters
2016-05-27 22:18:44 +00:00
Jonathan Lebon
d9e8535bcf treecompose: fix crash when "remove-from-packages" used
This works around a potential issue with libsolv if we go down the
rpmostree_get_pkglist_for_root() path. Though rpm has been using the
/usr/share/rpm location (since the RpmOstreeContext set the _dbpath
macro), the /var/lib/rpm directory will still exist, but be empty.
libsolv gets confused because it sees the /var/lib/rpm dir and doesn't
even try the /usr/share/rpm location, and eventually dies when it tries
to load the data.

So we set the symlink now. This is also what we do on boot anyway for
compatibility reasons using tmpfiles.

This also means we don't have to do the /var/lib/rpm --> /usr/share/rpm
transition during the rootfs postprocess (but we still have to clean up
db and lock files).

Also get rid of the unused pkglist variable.

NB: I used the GFile & gs APIs to mesh with the surrounding code.

Closes: #290
Approved by: cgwalters
2016-05-27 17:06:08 +00:00
Colin Walters
2cab8216b4 core: Adapt to libhif API query change
We'll track git master.

Closes: #286
Approved by: jlebon
2016-05-19 17:44:34 +00:00
Colin Walters
cbadc9bdaa libpriv: Print repository name for each package too
I'm trying to debug why I'm getting an older version of `docker`,
and it's useful to see the repository name we're getting something
from.  Yum does this by default.

(Though we should probably consider column formatted output too
 on a tty)

Closes: #282
Approved by: jlebon
2016-05-11 16:07:50 +00:00
Jonathan Lebon
582650e9c7 daemon: add TaskBegin, TaskEnd, and PercentProgress
Closes: #260
Approved by: cgwalters
2016-05-11 15:03:38 +00:00
Colin Walters
b718e28d79 postprocess: Add --add ostree to dracut invocation
Right now the `ostree.rpm` package always configures dracut to inject
the ostree setup via a conf file.  But it's actually simpler and
cleaner to just have callers specify it explicitly.

https://bugzilla.redhat.com/show_bug.cgi?id=1331369

Closes: #276
Approved by: jlebon
2016-05-06 14:50:42 +00:00
Colin Walters
4fc25d74a3 compose: Introduce variable substitution for ${basearch}
This comes from the tradition of yum repo files.  It's significantly
easier for rpm-ostree users building multiple architectures if some
core templating for the treefiles is built in.

Otherwise, everyone needs to learn about a new wrapper tool for
rpm-ostree, and said tool would need to re-do the same "basearch"
evaluation that is already occuring inside libhif.

This commit also paves the way for introducing `${releasever}`
substitution.

NOTE: This depends on pending changes to libhif git.

Closes: #274
Approved by: jlebon
2016-04-28 15:38:53 +00:00
Colin Walters
e2e754e94c compose: Reuse previous commit lookup in passwd checking
A future commit is going to change our parsing of the "ref" member in
treefiles, so ensure we only load it once early on in compose-tree.
We already looked up the previous commit there, so just pass it down
rather than reloading the ref.

Closes: #274
Approved by: jlebon
2016-04-28 15:38:53 +00:00
Giuseppe Scrivano
94a2804b0a compose: support adding external files
This will allow to copy arbitrary files into the rootfs, specifying something like:

"add-files": [["service.template", "/exports/service.template"],
              ["config.json.template", "/exports/config.json.template"]]

It is quite useful when building a container image.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #253
Approved by: cgwalters
2016-04-27 20:21:12 +00:00
Colin Walters
63761e5127 core: Skip over local packages for downloads
Because `hif_source_get_location()` is actually a reference to the
upstream repo, we shouldn't create a cache directory there.

This is just a two line fix, but I changed some things so that we keep
track of the number of local packages, in order to make the "Need to
download" print accurate.

We still need to add them (confusingly) to the `packages_to_download`
because that's also used for the container path which imports them.

Closes: #255

Closes: #256
Approved by: jlebon
2016-03-31 17:56:19 +00:00
Colin Walters
3350c0e062 postprocess: Fix "compose tree" regression with /etc
The fact that we didn't notice this is a testament to the state of
testing =(

I think I want to change the container build path to also only
grab /usr and /etc rather than having the paths be different,
but for now - the quick fix.

Pull request: #243
Approved by: jlebon
2016-03-22 18:35:33 +00:00
Colin Walters
ed7fc02b04 postprocess: Ensure we error out if selinux is enabled, but no policy found
I was tracking another regression where we seem to have lost
`/usr/etc` contents which manifested as `Labeling with... (null)`
which was clearly wrong.

Now this change actually impacts the test suite - we now (again IMO
correctly) error out if `selinux: true`.  The `no-selinux-tag` test
no longer makes sense, so delete it.

We do need more "real" tests that use selinux on and off.

Pull request: #243
Approved by: jlebon
2016-03-22 18:35:33 +00:00
Jonathan Lebon
f8b87efc1f RpmOstreeContext: add set_repo()
The new context methods recently added for unprivileged infrastructure
would set the ostreerepo member when calling new_unprivileged(). Since
here we will be using the sysroot repo, we need the ability to set the
repo. Just add a new set function for now, maybe move it into
new_system() later.

Also, make sure we dereference the repo on finalize.

Pull request: #240
Approved by: cgwalters
2016-03-21 19:57:06 +00:00
Jonathan Lebon
224e4b31d8 RpmOstreeTreespec: allow missing ref element
There will be cases where we don't actually want to tag a ref on the
final assembled commit. This will be true for package layering.

Pull request: #240
Approved by: cgwalters
2016-03-21 19:57:06 +00:00
Jonathan Lebon
bf44928f40 RpmOstreeContext: remove unused params
Pull request: #240
Approved by: cgwalters
2016-03-21 19:57:06 +00:00
Jonathan Lebon
8594931c67 RpmOstreeTreespec: allow missing repos element
We allow the "repos" key to be missing, which just means that we let
libhif do its job, i.e. automatically use the repos in repodir that are
enabled.

We still do a check in rpmostree_context_setup() to make sure that at
least one repo is functional.

Pull request: #240
Approved by: cgwalters
2016-03-21 19:57:06 +00:00
Colin Walters
f46302c2ab unpacker: Fix a 32 bit format string error 2016-02-23 13:44:14 -05:00
Colin Walters
397fdd8d33 Use a dummy install root for unprivileged composes
Sadly, libhif keeps trying to auto-create it.  It'll need patching
there, and possibly in librpm.
2016-02-22 14:27:46 -05:00
Colin Walters
f577279fac libpriv: Unify /usr/local, /etc -> /usr/etc, rpmdb handling
Right now we're doing the /etc -> /usr/etc inside the RPM import, but
we might as well do the /usr/local bits in both.  Also, use
/usr/share/rpm by default for treecompose too so that is unified.

Other things like systemd unit files and kernel handling are only
going to be used for host side composes.
2016-02-22 14:27:46 -05:00
Colin Walters
58cf2c9403 tests: Add test coverage for rpm-ostree container 2016-02-22 14:27:46 -05:00
Colin Walters
961a036c5b Introduce "treespec" concept as GKeyFile
I debated config file formats a lot.  JSON is fairly awkward for
humans to write, and really painful to parse from C.  YAML is nice,
but also painful from C.

Both are fairly overpowered for what we really need.  Keyfiles
(desktop spec, `GKeyFile`) have a lot of limitations, but at least
it's used by systemd and `.desktop` files, and we already have a
parser.

We still parse the JSON treefiles, but internally convert them to
`GKeyFile` (which is in turn converted to `GVariant` for a canonical
form).
2016-02-22 14:27:46 -05:00
Colin Walters
20170cba4f libpriv: Make the OSTree repo implicit
It's the default for unprivileged composes if it exists.  This is an
incremental step towards always using the ostree repo.
2016-02-22 14:27:46 -05:00
Colin Walters
1eda3a022c Introduce RpmOstreeContext as an object
This helps unify some code more initially between the treecompose and
container bits.
2016-02-22 14:27:46 -05:00
Colin Walters
b3a7b4e218 Rename rpmostree-hif.[ch] -> core
This is really going to be the heart of rpm-ostree, so let's give it a
better name.
2016-02-22 14:27:46 -05:00
Colin Walters
f14d1a3536 build: Port to master libhif
This is a work-in-progress port against the libhif master + some
outstanding PRs.
2016-02-22 14:27:46 -05:00
Colin Walters
8de4f9be27 Add a container builtin
This is just a tech demo.  Example usage:

```
mkdir -p ~/.cache/rpmostree-containers
cd ~/.cache/rpmostree-containers
rpm-ostree container init
cp /etc/yum.repos.d/CentOS-Core.repo rpmmd.repos.d
rpm-ostree container assemble bash
rpm-ostree container assemble httpd
```
2016-02-22 14:27:45 -05:00
Colin Walters
b716959252 Major revamp/extension of libhif/unpacker code
This is in preparation for `rpm-ostree container`, which handles
unpacking RPMs as non-root.

At the moment, I'm copying code in from both ostree's libarchive bits
(fixable...may need to export some utility functions) and some
functions from libhif (harder, see:
http://lists.rpm.org/pipermail/rpm-ecosystem/2016-January/000297.html )

There's lots more cleanup to do here, but I don't want to block on the
resolution of the libhif changes.
2016-02-22 14:27:45 -05:00
Colin Walters
ec4387afba internals: New unpack verb
This is part of taking over from librpm.  The most important high
level goal is fully unprivilged operation.

Right now we're basically starting to do what
http://libguestfs.org/supermin.1.html does, except in C, and
faster.

There's no reason that `compose tree` should require privileges.
However right now, things like `%post` scripts will want to run in the
target root - so we'd have to require `linux-user-chroot`.

Regardless of unprivileged operation though, another major thing we
can do is use our control over the unpacking process to do a lot more
sophisticated caching.  We can build up a precise mapping of (rpm
ENVR, file path, selinux label) -> object and avoid rechecksumming
each time.

And even for files that aren't known, we can parallelize commit with
unpacking, etc.  (Ok assuming treecompose-post won't mutate anything).
2016-01-12 10:22:43 -05:00
Colin Walters
5a530b9c85 Honor --proxy again, tweak internal libhif API
Two nits noticed in review that I accidentally ignored before pushing.

https://github.com/projectatomic/rpm-ostree/pull/202
2016-01-10 09:11:04 -05:00
Colin Walters
236a3c46cd Extract a shared internal rpmostree-hif.[ch]
As we start to do more package things, extract common helper functions
around HifContext * that by default operates on the system root.

Some of these bits should go in libhif, but the immediate plan is to
iterate here, then push downwards later.
2016-01-09 11:07:12 -05:00
Colin Walters
163f50214b postprocess: Delete dead code relating to yumdb
This has been long useless because we now disable history writing in
libhif.
2016-01-04 16:23:54 -05:00
Colin Walters
5cd40e6d86 postprocess: Cleanup more /var -> tmpfiles code to fd relative
Besides porting GFile -> fd, I specifically want it to operate in an
append mode for package layering.  Then given an existing tree, we
ensure we're not deleting the underlying tree's autovar files.
2016-01-04 12:47:31 -05:00
Colin Walters
cca057d24b postprocess: Convert some of the /var -> tmpfiles to fd relative
Part of an ongoing effort to port away from `GFile`.

Conflicts:
	src/libpriv/rpmostree-postprocess.c
2016-01-04 12:47:14 -05:00
Colin Walters
8de94004df libpriv: Change internal API to prepare SELinux
This function will be made public for use in package layering.
2016-01-03 09:45:14 -05:00
Colin Walters
e2fa1675f8 libpriv: Convert some SELinux labeling code to dirfd-relative
This is preparatory work for the package layering branch, which
will also reuse this code.
2016-01-03 08:00:10 -05:00
Jonathan Lebon
033830e0c0 postprocess.c: be more verbose when running script
We should tell the user that we will run the postprocess script *before*
we run it to help them diagnose issues if things go wrong.

Resolves: #188
2015-12-10 14:49:57 -05:00
Colin Walters
b82f7338ea src: Quiet a few gcc -Wmaybe-uninitialized warnings
GCC (at least 5.2.1) isn't smart enough to figure out these are always
initialized.
2015-11-23 12:08:37 -05:00
Giuseppe Scrivano
bb2f32502b postprocess: provide --gzip and --reproducible to dracut if supported
This should help to generate the same initrd when the files didn't
change.

Newer versions of gzip (or pigz when available) can generate rsync
friendly files and if present, Dracut already takes advantage of it.

Also use --reproducible, to instruct Dracut to generate CPIO
reproducible files.  It is required a version of GNU CPIO that
has support for it.

Check that Dracut has --reproducible in its --help output before
setting it.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-11-04 12:33:55 +01:00
Matthew Barnes
463d3676ea daemon: Miscellaneous cleanups 2015-09-09 22:00:05 -04:00
Colin Walters
60ed86228f compose: Don't do container stuff if we're in a container already
This helps avoid some failures.  See:
fbdb15cd95
2015-08-24 16:16:52 -04:00
Colin Walters
84d2d51576 postprocess: Don't use absolute paths for helpers like dracut
It moved from `/usr/sbin` -> `/usr/bin`, which obviously broke
rpm-ostree.

No reason to hardcode absolute paths here.
2015-06-15 13:44:52 -04:00
Colin Walters
b8ac0f7caf libpriv: Introduce an rpmostree_mkdtemp()
- Can also give you a file descriptor
 - Takes a constant string as input, returning a mutated string as a
   separate variable which means that one can check whether the variable
   is `NULL` to know whether or not one needs to `rm -rf` it on error
   paths.
2015-05-23 09:32:28 -04:00
Colin Walters
bfe801b77e libpriv: Fix memory leak 2015-05-23 09:32:23 -04:00
Colin Walters
1aa0e37cc9 libpriv: Teach RpmRevisionData to use new RefTs API
More code de-duplication.
2015-05-23 09:30:57 -04:00
Colin Walters
70f5b535e4 libpriv: Add RpmOstreeRefTs and API to create one from a commit
This API is intended for use by the caching framework to hold open an
rpmdb for the previous commit.
2015-05-23 09:30:57 -04:00
Colin Walters
b378386e98 Prefer rpmostree_ for internal API over _rpm_ostree
The refsack code was using the latter, and it stood out.  Now that
we're making use of explicit export markers, there's no need to uglify
internal APIs with a leading `_`.
2015-05-22 09:38:55 -04:00
Colin Walters
07871a08fd libpriv: Further centralize (commit | root) -> (sack | rpmdb) code
While `rpm-util.c` may not best describe this, it's where most
of this code is ending up.  Let's further centralize things.

We more consistently return an `RpmOstreeRefSack` instead of a
`HySack`, where the former supports refcounting and knows how to clean
up its temporary directory if it was allocated from a commit.
2015-05-22 09:32:03 -04:00
Colin Walters
3c3947564d db: Use internal rpmdb-in-tempdir API
And now, finally the actual goal is achieved.  \o/

Only one code path dealing with extracting the rpm database from an
OSTree commit.

An astute reader would notice that the `root` member of the struct was
actually only necessary as of a few commits ago.  But said astute
reader would also realize it's kind of late in the evening and not
worth rebasing it to where it would properly go.
2015-05-21 21:27:38 -04:00
Colin Walters
0756708f24 db: Create var/lib/rpm -> ../../usr/share/rpm in tempdir
hawkey and libsolv are both patched to look in `usr/share/rpm` if the
db can't be found in `/var/lib/rpm`.  However, librpm itself isn't.
One *can* override it with a macro...which is process global.  Yuck.
Needs fixing.

Anyways, we can just make a symlink.  That's a lot easier than writing
a patch for librpm and waiting a billion years to be able to use it
everywhere we care about.

This will help unify the librpm tempdir code with the hawkey tempdir
code.
2015-05-21 21:25:27 -04:00
Colin Walters
60d68f6e24 db: Make RpmRevisionData an opaque struct
This allows us to more cleanly encapsulate things.
2015-05-21 21:16:21 -04:00
Colin Walters
349dd45679 db: Remove --rpmdbdir option
As far as I can tell, this is basically a way to specify the temporary
directory.  That significantly complicates the code as it now
needs to keep track of whether or not it owns the temporary directory.

This hinders unifying this code with the hawkey query path.

Because of this, and since I'm not aware of a use case for specifying
this tempdir, let's remove it.
2015-05-21 21:16:21 -04:00
Colin Walters
bd9b9eac0a Drop root arguments from internal rpmdb API
It was only used to access the yumdb, which we don't use because:

 - It badly exacerbates the OSTree one-HTTP-request-per-object issue
 - We're assembling multiple repos on the server side, so things like
   who took the action aren't relevant.

But the reason I did this patch at the moment is because I want to
unify the code that's creating tempdirs from commits so we can feed
real files to librpm.
2015-05-21 20:34:01 -04:00
Colin Walters
5de58aae9c libpriv: Split out an internal API to get a tempdir with just the rpmdb
For a future patch, I want to add an API to get an rpmts for a commit,
instead of a hawkey Sack, because libsolv doesn't expose some
optimized queries that we can get by just going directly to librpm,
such as package file owners.
2015-05-21 20:15:08 -04:00
Alexander Larsson
79cdcd1817 Support container: true in treefile
If this is set we don't do anything special for kernel or /boot, as
such things are not needed when running in a container.
2015-05-21 12:00:20 +02:00
Colin Walters
cf89acb74c (cleanup): Move the refsack and root loading code into libpriv
We had `src/lib` having its own little private library; I wanted to
use some of it inside `src/libpriv`, so let's consistently have all
private utility code in `src/libpriv`.

Closes: https://github.com/projectatomic/rpm-ostree/pull/147
2015-05-20 20:51:42 -04:00
Colin Walters
1c3a549ef9 postprocess: Handle Fedora rawhide kernel installation
The vmlinuz binary has moved to /usr/lib/modules, which is a change
mostly for the better, but we need to adapt.

Closes: https://github.com/projectatomic/rpm-ostree/pull/143
2015-05-12 13:51:12 -04:00
Colin Walters
c16058b66a libpriv: Move rpm/hawkey util functions into -rpm-util
`treepkgdiff` no longer describes the function of the file; as a baby
step towards unifying the db and hawkey code, move the functions into
that file.
2015-04-23 16:30:18 -04:00
Colin Walters
e414be8ae2 Move diff printing code into client
This is a step forward to deduplicating; the client tooling now calls
into the public API for diffs, rather than using the older internal
function.

Note: this patch also links the client against the public library.
2015-04-23 16:30:18 -04:00
Colin Walters
7b45f13a62 Link lib/ against libpriv/, use it to share sack loading function
On the plus side, we share some code between the library and the
binary now.  On the downside, because `librpmostreepriv.la` is a
noinst library, its code text is duplicated between the shared library
and binary, at least until we either:

 - Have the binary solely use the public shared library (like ostree does)
 - Install `librpmostreepriv.so` to e.g. `/usr/lib64/rpm-ostree/librpmostreepriv.so`
   without the headers being public
2015-04-23 16:30:17 -04:00
Colin Walters
f3600e922e libpriv: Adjust internal package diff API to be fd-relative
We presently have 3 internal code paths that are doing rpmdb
inspection.  This conversion to fd-relative for one of them is a
generic cleanup preparatory to de-duplicating.

Note this bumps libglnx to include
381ca54ee3
2015-04-23 16:30:17 -04:00
Giuseppe Scrivano
3bf7926adc postprocess: do not hardcode /etc/machine-id
The file is automatically populated by systemd when it is empty.

Apparently it doesn't work when the file is missing (as of
systemd-219-9.fc22).

https://bugzilla.redhat.com/show_bug.cgi?id=1198700

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-04-23 11:37:50 +02:00
Colin Walters
a8a2049443 build: Split up src into app/ and libpriv/
We currently have an internal-only library, but the sources for it are
in the same dir as the app.  For future work on a public shared
library, we'll need a clearer source structure.

Start by just renaming the app files into `src/app/`, and the internal
private library into `src/libpriv/`, with the appropriate
`Makefile.am` changes.

Closes: https://github.com/projectatomic/rpm-ostree/pull/123
2015-04-08 16:17:06 -04:00