keremet/rpm-ostree
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
89f7e481b4
rpm-ostree/src/app/rpmostree-compose-builtin-tree.c

969 lines
32 KiB
C
Raw Normal View History

Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
/* -*- mode: C; c-file-style: "gnu"; indent-tabs-mode: nil; -*-
*
* Copyright (C) 2013,2014 Colin Walters <walters@verbum.org>
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published
* by the Free Software Foundation; either version 2 of the licence or (at
* your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General
* Public License along with this library; if not, write to the
* Free Software Foundation, Inc., 59 Temple Place, Suite 330,
* Boston, MA 02111-1307, USA.
*/
#include "config.h"
#include <string.h>
#include <glib-unix.h>
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
#include <json-glib/json-glib.h>
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
#include <gio/gunixoutputstream.h>
libhif: always prefix include directives This is in response to: https://github.com/rpm-software-management/libhif/pull/138 Closes: #310 Approved by: cgwalters
2016-06-09 05:09:00 +03:00
#include <libhif/libhif.h>
compose: Protect the system during treecompose using container APIs I was looking again at using hawkey/librepo, and realized just how much I'd have to fight all of these libraries to avoid affecting the running system. What we really want to do with librepo/hawkey is run them effectively unprivileged, and to hide the system's RPM database from them. This is a baby step towards that, by confining our existing yum. - /usr, /etc, and /var/lib/rpm are mounted read-only - yum is now run under CLONE_NEWPID, to avoid stray %post scripts affecting system processes
2014-11-10 04:28:10 +03:00
#include <stdio.h>
Start using libglnx The high level goal is to deprecate libgsystem. I was trying to share code between ostree/rpm-ostree, but it was too painful to commit to forver frozen ABI for new utility APIs. The git submodule approach will much more easily allow breaking API/ABI, and iterate on APIs until they either land in GLib or not. Note that libglnx will not use GFile*, so a full port to it will involve also not using that. Thus, it will be necessarily incremental; in the meantime we'll link to both libgsystem and libglnx.
2015-02-15 21:56:14 +03:00
#include <libglnx.h>
[libhif] Implement support for install-langs
2015-01-02 01:44:57 +03:00
#include <rpm/rpmmacro.h>
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Add new "compose" builtin, rename "treecompose" -> "compose tree" And do the same for "sign". This way we can have the compose server utilities cleanly separated from what most people will see, which is the client side tools.
2014-05-26 23:05:08 +04:00
#include "rpmostree-compose-builtins.h"
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
#include "rpmostree-util.h"
Rename rpmostree-hif.[ch] -> core This is really going to be the heart of rpm-ostree, so let's give it a better name.
2016-02-09 18:38:38 +03:00
#include "rpmostree-core.h"
compose: Split out JSON helpers into separate file In preparation for using some of them in the postprocessing code too. Yeah, parsing JSON in C sucks =( Oh well.
2014-11-13 22:53:43 +03:00
#include "rpmostree-json-parsing.h"
Unify rpm-ostree into one binary And start making an internal library.
2014-01-29 23:37:44 +04:00
#include "rpmostree-postprocess.h"
compose: Add check-passwd/group JSON options, fails compose if uids/gids change Verify uid/gid on files, directories and symlinks Just output a msg when user/group is removed with no files json-parsing: Add functions for strictly dealing with ints passwd/json: Add simple scripts to convert passwd/group files to json data docs: Check-passwd/groups and ignore-remove-users/groups JSON config. entries
2014-11-20 17:40:35 +03:00
#include "rpmostree-passwd-util.h"
app: Add rpmostree_usage_error() Convenience function for missing or invalid command-line arguments.
2015-10-26 16:36:22 +03:00
#include "rpmostree-libbuiltin.h"
libpriv: Move rpm/hawkey util functions into -rpm-util `treepkgdiff` no longer describes the function of the file; as a baby step towards unifying the db and hawkey code, move the functions into that file.
2015-04-19 19:10:32 +03:00
#include "rpmostree-rpm-util.h"
Unify rpm-ostree into one binary And start making an internal library.
2014-01-29 23:37:44 +04:00
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
#include "libgsystem.h"
static char *opt_workdir;
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
static gboolean opt_workdir_tmpfs;
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
static char *opt_cachedir;
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
static gboolean opt_force_nocache;
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
static char *opt_proxy;
compose tree: Add --output-repodata-dir This could be used by higher level tools like fedmsg-atomic-composer that want to merge the yum repodata used for input with the tree content.
2014-10-24 01:14:14 +04:00
static char *opt_output_repodata_dir;
treecompose: Add --add-metadata-string argument for versioning This will be used by rpm-ostree-toolbox to inject version numbers for the trees.
2014-10-14 17:32:29 +04:00
static char **opt_metadata_strings;
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
static char *opt_repo;
compose: Add a --add-override-pkg-repo argument As a developer, a workflow I have for testing things is to create an RPM, toss it into a local yum repository, then do a compose. However at the moment to add the local overrides I have to edit the treefile, which is annoying. Let's add a commandline override for this. Note this also deletes the old "repos_data" code which was not being used.
2014-07-11 22:02:45 +04:00
static char **opt_override_pkg_repos;
compose-tree: Add --touch-if-changed This is a convenient way for external tools to integrate rpm-ostree better into automation. With the caching, it's now cheaper to just rerun rpm-ostree if you know an input yum repository changed, or just on a timer. Then one can use this flag to determine whether or not to launch any further processes after rpm-ostree, such as disk image generation, test suites, etc.
2015-01-29 00:34:29 +03:00
static char *opt_touch_if_changed;
compose: Add a --dry-run option This way it's easier for people to inspect the next system state.
2016-03-01 23:38:45 +03:00
static gboolean opt_dry_run;
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
static gboolean opt_print_only;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
static GOptionEntry option_entries[] = {
treecompose: Add --add-metadata-string argument for versioning This will be used by rpm-ostree-toolbox to inject version numbers for the trees.
2014-10-14 17:32:29 +04:00
{ "add-metadata-string", 0, 0, G_OPTION_ARG_STRING_ARRAY, &opt_metadata_strings, "Append given key and value (in string format) to metadata", "KEY=VALUE" },
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
{ "workdir", 0, 0, G_OPTION_ARG_STRING, &opt_workdir, "Working directory", "WORKDIR" },
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
{ "workdir-tmpfs", 0, 0, G_OPTION_ARG_NONE, &opt_workdir_tmpfs, "Use tmpfs for working state", NULL },
compose tree: Add --output-repodata-dir This could be used by higher level tools like fedmsg-atomic-composer that want to merge the yum repodata used for input with the tree content.
2014-10-24 01:14:14 +04:00
{ "output-repodata-dir", 0, 0, G_OPTION_ARG_STRING, &opt_output_repodata_dir, "Save downloaded repodata in DIR", "DIR" },
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
{ "cachedir", 0, 0, G_OPTION_ARG_STRING, &opt_cachedir, "Cached state", "CACHEDIR" },
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
{ "force-nocache", 0, 0, G_OPTION_ARG_NONE, &opt_force_nocache, "Always create a new OSTree commit, even if nothing appears to have changed", NULL },
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
{ "repo", 'r', 0, G_OPTION_ARG_STRING, &opt_repo, "Path to OSTree repository", "REPO" },
compose: Add a --add-override-pkg-repo argument As a developer, a workflow I have for testing things is to create an RPM, toss it into a local yum repository, then do a compose. However at the moment to add the local overrides I have to edit the treefile, which is annoying. Let's add a commandline override for this. Note this also deletes the old "repos_data" code which was not being used.
2014-07-11 22:02:45 +04:00
{ "add-override-pkg-repo", 0, 0, G_OPTION_ARG_STRING_ARRAY, &opt_override_pkg_repos, "Include an additional package repository from DIRECTORY", "DIRECTORY" },
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
{ "proxy", 0, 0, G_OPTION_ARG_STRING, &opt_proxy, "HTTP proxy", "PROXY" },
compose-tree: Add --touch-if-changed This is a convenient way for external tools to integrate rpm-ostree better into automation. With the caching, it's now cheaper to just rerun rpm-ostree if you know an input yum repository changed, or just on a timer. Then one can use this flag to determine whether or not to launch any further processes after rpm-ostree, such as disk image generation, test suites, etc.
2015-01-29 00:34:29 +03:00
{ "touch-if-changed", 0, 0, G_OPTION_ARG_STRING, &opt_touch_if_changed, "Update the modification time on FILE if a new commit was created", "FILE" },
compose: Add a --dry-run option This way it's easier for people to inspect the next system state.
2016-03-01 23:38:45 +03:00
{ "dry-run", 0, 0, G_OPTION_ARG_NONE, &opt_dry_run, "Just print the transaction and exit", NULL },
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
{ "print-only", 0, 0, G_OPTION_ARG_NONE, &opt_print_only, "Just expand any includes and print treefile", NULL },
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{ NULL }
};
compose: Add automatic_version_prefix to json config.
2015-01-07 09:30:27 +03:00
/* FIXME: This is a copy of ot_admin_checksum_version */
static char *
checksum_version (GVariant *checksum)
{
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autoptr(GVariant) metadata = NULL;
compose: Add automatic_version_prefix to json config.
2015-01-07 09:30:27 +03:00
const char *ret = NULL;
metadata = g_variant_get_child_value (checksum, 0);
if (!g_variant_lookup (metadata, "version", "&s", &ret))
return NULL;
return g_strdup (ret);
}
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
typedef struct {
GPtrArray *treefile_context_dirs;
compose: Move 'workdir' to be 'self' member This avoids passing another parameter around.
2014-11-17 04:05:47 +03:00
GFile *workdir;
Start using libglnx The high level goal is to deprecate libgsystem. I was trying to share code between ostree/rpm-ostree, but it was too painful to commit to forver frozen ABI for new utility APIs. The git submodule approach will much more easily allow breaking API/ABI, and iterate on APIs until they either land in GLib or not. Note that libglnx will not use GFile*, so a full port to it will involve also not using that. Thus, it will be necessarily incremental; in the meantime we'll link to both libgsystem and libglnx.
2015-02-15 21:56:14 +03:00
int workdir_dfd;
treecompose: Make use of cachedir again (just for metadata) We need to do selinux first.
2016-02-10 15:47:32 +03:00
int cachedir_dfd;
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
OstreeRepo *repo;
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
char *ref;
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
char *previous_checksum;
compose: Put the expanded treefile content in /usr/share/rpm-ostree/treefile.json Since the treefile format now supports includes, we would need to either include the whole chain, or just the expanded portion we use in the compose. This patch does the latter. This should allow a client to take the same treefile and generate a similar tree (if they want to reproduce with the same RPMs, those can be extracted from the RPM database inside the tree).
2014-09-07 20:38:34 +04:00
GBytes *serialized_treefile;
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
} RpmOstreeTreeComposeContext;
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
static gboolean
compute_checksum_from_treefile_and_goal (RpmOstreeTreeComposeContext *self,
HyGoal goal,
compose: add cache for added files Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #253 Approved by: cgwalters
2016-04-04 14:48:08 +03:00
GFile *contextdir,
JsonArray *add_files,
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
char **out_checksum,
GError **error)
{
gboolean ret = FALSE;
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autofree char *ret_checksum = NULL;
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
GChecksum *checksum = g_checksum_new (G_CHECKSUM_SHA256);
/* Hash in the raw treefile; this means reordering the input packages
* or adding a comment will cause a recompose, but let's be conservative
* here.
*/
{ gsize len;
const guint8* buf = g_bytes_get_data (self->serialized_treefile, &len);
g_checksum_update (checksum, buf, len);
}
compose: add cache for added files Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #253 Approved by: cgwalters
2016-04-04 14:48:08 +03:00
if (add_files)
{
guint i, len = json_array_get_length (add_files);
for (i = 0; i < len; i++)
{
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autoptr(GFile) srcfile = NULL;
compose: add cache for added files Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #253 Approved by: cgwalters
2016-04-04 14:48:08 +03:00
const char *src, *dest;
JsonArray *add_el = json_array_get_array_element (add_files, i);
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autoptr(GFile) child = NULL;
compose: add cache for added files Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #253 Approved by: cgwalters
2016-04-04 14:48:08 +03:00
if (!add_el)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Element in add-files is not an array");
goto out;
}
src = _rpmostree_jsonutil_array_require_string_element (add_el, 0, error);
if (!src)
goto out;
dest = _rpmostree_jsonutil_array_require_string_element (add_el, 1, error);
if (!dest)
goto out;
srcfile = g_file_resolve_relative_path (contextdir, src);
if (!_rpmostree_util_update_checksum_from_file (checksum,
srcfile,
NULL,
error))
goto out;
g_checksum_update (checksum, (const guint8 *) dest, strlen (dest));
}
}
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
/* FIXME; we should also hash the post script */
/* Hash in each package */
Introduce RpmOstreeContext as an object This helps unify some code more initially between the treecompose and container bits.
2016-02-09 21:13:28 +03:00
rpmostree_hif_add_checksum_goal (checksum, goal);
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
ret_checksum = g_strdup (g_checksum_get_string (checksum));
ret = TRUE;
compose: add cache for added files Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #253 Approved by: cgwalters
2016-04-04 14:48:08 +03:00
out:
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
gs_transfer_out_value (out_checksum, &ret_checksum);
if (checksum) g_checksum_free (checksum);
return ret;
}
[libhif]: Add initial progress reporting This requires open-coding the 3 steps that hif_context_run() was doing. Which is fine, because ultimately we want to have more control over that.
2015-01-02 01:32:40 +03:00
static void
on_hifstate_percentage_changed (HifState *hifstate,
guint percentage,
gpointer user_data)
{
const char *text = user_data;
Move console code to libglnx The main goal is to be able to share this with OSTree.
2015-02-18 00:54:54 +03:00
glnx_console_progress_text_percent (text, percentage);
[libhif]: Add initial progress reporting This requires open-coding the 3 steps that hif_context_run() was doing. Which is fine, because ultimately we want to have more control over that.
2015-01-02 01:32:40 +03:00
}
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
static gboolean
set_keyfile_string_array_from_json (GKeyFile *keyfile,
const char *keyfile_group,
const char *keyfile_key,
JsonArray *a,
GError **error)
{
gboolean ret = FALSE;
guint len = json_array_get_length (a);
guint i;
g_autoptr(GPtrArray) instlangs_v = g_ptr_array_new ();
for (i = 0; i < len; i++)
{
const char *elt = _rpmostree_jsonutil_array_require_string_element (a, i, error);
if (!elt)
goto out;
g_ptr_array_add (instlangs_v, (char*)elt);
}
g_key_file_set_string_list (keyfile, keyfile_group, keyfile_key,
(const char*const*)instlangs_v->pdata, instlangs_v->len);
ret = TRUE;
out:
return ret;
}
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
static gboolean
[libhif] compose: Initial port to libhif instead of yum There are a lot of advantages to this. See the linked issue for more details, but briefly: - Lays the groundwork for package layering - Better caching (exit compose after depsolve if no changes) - Better error handling - Potential for unprivileged package downloads - Potential to better containerize installs TODO: - langs handling - progress output Closes: https://github.com/projectatomic/rpm-ostree/issues/53
2015-01-01 02:18:33 +03:00
install_packages_in_root (RpmOstreeTreeComposeContext *self,
compose: Introduce variable substitution for ${basearch} This comes from the tradition of yum repo files. It's significantly easier for rpm-ostree users building multiple architectures if some core templating for the treefiles is built in. Otherwise, everyone needs to learn about a new wrapper tool for rpm-ostree, and said tool would need to re-do the same "basearch" evaluation that is already occuring inside libhif. This commit also paves the way for introducing `${releasever}` substitution. NOTE: This depends on pending changes to libhif git. Closes: #274 Approved by: jlebon
2016-04-28 15:59:20 +03:00
RpmOstreeContext *ctx,
[libhif] compose: Initial port to libhif instead of yum There are a lot of advantages to this. See the linked issue for more details, but briefly: - Lays the groundwork for package layering - Better caching (exit compose after depsolve if no changes) - Better error handling - Potential for unprivileged package downloads - Potential to better containerize installs TODO: - langs handling - progress output Closes: https://github.com/projectatomic/rpm-ostree/issues/53
2015-01-01 02:18:33 +03:00
JsonObject *treedata,
GFile *yumroot,
char **packages,
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
gboolean *out_unmodified,
char **out_new_inputhash,
[libhif] compose: Initial port to libhif instead of yum There are a lot of advantages to this. See the linked issue for more details, but briefly: - Lays the groundwork for package layering - Better caching (exit compose after depsolve if no changes) - Better error handling - Potential for unprivileged package downloads - Potential to better containerize installs TODO: - langs handling - progress output Closes: https://github.com/projectatomic/rpm-ostree/issues/53
2015-01-01 02:18:33 +03:00
GCancellable *cancellable,
GError **error)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
gboolean ret = FALSE;
[libhif]: Add initial progress reporting This requires open-coding the 3 steps that hif_context_run() was doing. Which is fine, because ultimately we want to have more control over that.
2015-01-02 01:32:40 +03:00
guint progress_sigid;
[libhif] compose: Initial port to libhif instead of yum There are a lot of advantages to this. See the linked issue for more details, but briefly: - Lays the groundwork for package layering - Better caching (exit compose after depsolve if no changes) - Better error handling - Potential for unprivileged package downloads - Potential to better containerize installs TODO: - langs handling - progress output Closes: https://github.com/projectatomic/rpm-ostree/issues/53
2015-01-01 02:18:33 +03:00
GFile *contextdir = self->treefile_context_dirs->pdata[0];
Introduce RpmOstreeContext as an object This helps unify some code more initially between the treecompose and container bits.
2016-02-09 21:13:28 +03:00
g_autoptr(RpmOstreeInstall) hifinstall = { 0, };
HifContext *hifctx;
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autofree char *ret_new_inputhash = NULL;
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
g_autoptr(GKeyFile) treespec = g_key_file_new ();
JsonArray *enable_repos = NULL;
compose: add cache for added files Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #253 Approved by: cgwalters
2016-04-04 14:48:08 +03:00
JsonArray *add_files = NULL;
[libhif] compose: Initial port to libhif instead of yum There are a lot of advantages to this. See the linked issue for more details, but briefly: - Lays the groundwork for package layering - Better caching (exit compose after depsolve if no changes) - Better error handling - Potential for unprivileged package downloads - Potential to better containerize installs TODO: - langs handling - progress output Closes: https://github.com/projectatomic/rpm-ostree/issues/53
2015-01-01 02:18:33 +03:00
treecompose: Make use of cachedir again (just for metadata) We need to do selinux first.
2016-02-10 15:47:32 +03:00
/* TODO - uncomment this once we have SELinux working */
#if 0
g_autofree char *cache_repo_pathstr = glnx_fdrel_abspath (self->cachedir_dfd, "repo");
g_autoptr(GFile) cache_repo_path = g_file_new_for_path (cache_repo_pathstr);
glnx_unref_object OstreeRepo *ostreerepo = ostree_repo_new (cache_repo_path);
if (!g_file_test (cache_repo_pathstr, G_FILE_TEST_EXISTS))
{
if (!ostree_repo_create (ostreerepo, OSTREE_REPO_MODE_BARE_USER, cancellable, error))
goto out;
}
#endif
Introduce RpmOstreeContext as an object This helps unify some code more initially between the treecompose and container bits.
2016-02-09 21:13:28 +03:00
hifctx = rpmostree_context_get_hif (ctx);
Honor --proxy again, tweak internal libhif API Two nits noticed in review that I accidentally ignored before pushing. https://github.com/projectatomic/rpm-ostree/pull/202
2016-01-10 17:11:04 +03:00
if (opt_proxy)
hif_context_set_http_proxy (hifctx, opt_proxy);
Introduce RpmOstreeContext as an object This helps unify some code more initially between the treecompose and container bits.
2016-02-09 21:13:28 +03:00
compose: Support RPMOSTREE_RPM_VERBOSITY In the future we'll be taking over pretty much all RPM functionality (unpack/scripts) and will be able to provide a lot more useful information more directly under our control. But in the meantime: - Set the default to "info" which is apparently where things like corrupted packages will show up. It's just info you know? - Allow callers to override this via environment variable, specifially one can use "debug" for lots of info. Closes: #331 Approved by: jlebon
2016-06-18 14:58:02 +03:00
/* Hack this here... see https://github.com/rpm-software-management/libhif/issues/53
* but in the future we won't be using librpm at all for unpack/scripts, so it won't
* matter.
*/
{ const char *debuglevel = getenv ("RPMOSTREE_RPM_VERBOSITY");
if (!debuglevel)
debuglevel = "info";
hif_context_set_rpm_verbosity (hifctx, debuglevel);
rpmlogSetFile(NULL);
}
[libhif] compose: Initial port to libhif instead of yum There are a lot of advantages to this. See the linked issue for more details, but briefly: - Lays the groundwork for package layering - Better caching (exit compose after depsolve if no changes) - Better error handling - Potential for unprivileged package downloads - Potential to better containerize installs TODO: - langs handling - progress output Closes: https://github.com/projectatomic/rpm-ostree/issues/53
2015-01-01 02:18:33 +03:00
hif_context_set_repo_dir (hifctx, gs_file_get_path_cached (contextdir));
compose: With --cachedir, retain packages too Really...we should have done this since day zero. Given that the final integration of package layering/compose/rpm caching is finally over the horizion but not yet here, let's still do this now. I plan to backport this patch to the 2016.3-fixes branch. Closes: #332 Approved by: jlebon
2016-06-18 15:13:26 +03:00
/* By default, retain packages in addition to metadata with --cachedir */
if (opt_cachedir)
hif_context_set_keep_cache (hifctx, TRUE);
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
g_key_file_set_string (treespec, "tree", "ref", self->ref);
g_key_file_set_string_list (treespec, "tree", "packages", (const char *const*)packages, g_strv_length (packages));
[libhif] Implement support for install-langs
2015-01-02 01:44:57 +03:00
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
/* Some awful code to translate between JSON and GKeyFile */
if (json_object_has_member (treedata, "install-langs"))
{
JsonArray *a = json_object_get_array_member (treedata, "install-langs");
if (!set_keyfile_string_array_from_json (treespec, "tree", "install-langs", a, error))
goto out;
}
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
[libhif] compose: Initial port to libhif instead of yum There are a lot of advantages to this. See the linked issue for more details, but briefly: - Lays the groundwork for package layering - Better caching (exit compose after depsolve if no changes) - Better error handling - Potential for unprivileged package downloads - Potential to better containerize installs TODO: - langs handling - progress output Closes: https://github.com/projectatomic/rpm-ostree/issues/53
2015-01-01 02:18:33 +03:00
/* Bind the json \"repos\" member to the hif state, which looks at the
* enabled= member of the repos file. By default we forcibly enable
* only repos which are specified, ignoring the enabled= flag.
*/
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
if (!json_object_has_member (treedata, "repos"))
{
g_set_error_literal (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Treefile is missing required \"repos\" member");
goto out;
}
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
enable_repos = json_object_get_array_member (treedata, "repos");
Introduce RpmOstreeContext as an object This helps unify some code more initially between the treecompose and container bits.
2016-02-09 21:13:28 +03:00
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
if (!set_keyfile_string_array_from_json (treespec, "tree", "repos", enable_repos, error))
goto out;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
[libhif] Update to support "documentation" flag
2015-01-17 01:23:17 +03:00
{ gboolean docs = TRUE;
if (!_rpmostree_jsonutil_object_get_optional_boolean_member (treedata,
"documentation",
&docs,
error))
goto out;
if (!docs)
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
g_key_file_set_boolean (treespec, "tree", "documentation", FALSE);
}
{ g_autoptr(GError) tmp_error = NULL;
g_autoptr(RpmOstreeTreespec) treespec_value = rpmostree_treespec_new_from_keyfile (treespec, &tmp_error);
g_assert_no_error (tmp_error);
core: major rework This patch prepares RpmOstreeContext for supporting package layering. A relabel operation is added as well to support relabeling imported packages if the sepolicy of the rootfs we're overlaying onto is different from during import. Closes: #289 Approved by: cgwalters
2016-05-20 21:16:38 +03:00
if (!rpmostree_context_setup (ctx, gs_file_get_path_cached (yumroot), "/", treespec_value,
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
cancellable, error))
goto out;
[libhif] Update to support "documentation" flag
2015-01-17 01:23:17 +03:00
}
[libhif]: Add initial progress reporting This requires open-coding the 3 steps that hif_context_run() was doing. Which is fine, because ultimately we want to have more control over that.
2015-01-02 01:32:40 +03:00
/* --- Downloading metadata --- */
Introduce RpmOstreeContext as an object This helps unify some code more initially between the treecompose and container bits.
2016-02-09 21:13:28 +03:00
if (!rpmostree_context_download_metadata (ctx, cancellable, error))
Major revamp/extension of libhif/unpacker code This is in preparation for `rpm-ostree container`, which handles unpacking RPMs as non-root. At the moment, I'm copying code in from both ostree's libarchive bits (fixable...may need to export some utility functions) and some functions from libhif (harder, see: http://lists.rpm.org/pipermail/rpm-ecosystem/2016-January/000297.html ) There's lots more cleanup to do here, but I don't want to block on the resolution of the libhif changes.
2016-01-26 22:34:51 +03:00
goto out;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
if (!rpmostree_context_prepare_install (ctx, &hifinstall, cancellable, error))
Major revamp/extension of libhif/unpacker code This is in preparation for `rpm-ostree container`, which handles unpacking RPMs as non-root. At the moment, I'm copying code in from both ostree's libarchive bits (fixable...may need to export some utility functions) and some functions from libhif (harder, see: http://lists.rpm.org/pipermail/rpm-ecosystem/2016-January/000297.html ) There's lots more cleanup to do here, but I don't want to block on the resolution of the libhif changes.
2016-01-26 22:34:51 +03:00
goto out;
[libhif]: Add initial progress reporting This requires open-coding the 3 steps that hif_context_run() was doing. Which is fine, because ultimately we want to have more control over that.
2015-01-02 01:32:40 +03:00
Don't print transaction in middle of upgrade overlay output ...but still do print it for `compose tree` and `container`. This is more followup to #403, where I noticed we were printing just the overlay transaction in the middle of `Overlaying...`. Since we're now printing the package diff again at the end, let's just rely on that rather than show the user over and over which packages they requested. Closes: #403 Closes: #404 Approved by: jlebon
2016-07-24 21:26:21 +03:00
rpmostree_print_transaction (hifctx);
compose: add cache for added files Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #253 Approved by: cgwalters
2016-04-04 14:48:08 +03:00
if (json_object_has_member (treedata, "add-files"))
add_files = json_object_get_array_member (treedata, "add-files");
Major revamp/extension of libhif/unpacker code This is in preparation for `rpm-ostree container`, which handles unpacking RPMs as non-root. At the moment, I'm copying code in from both ostree's libarchive bits (fixable...may need to export some utility functions) and some functions from libhif (harder, see: http://lists.rpm.org/pipermail/rpm-ecosystem/2016-January/000297.html ) There's lots more cleanup to do here, but I don't want to block on the resolution of the libhif changes.
2016-01-26 22:34:51 +03:00
/* FIXME - just do a depsolve here before we compute download requirements */
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
if (!compute_checksum_from_treefile_and_goal (self, hif_context_get_goal (hifctx),
compose: add cache for added files Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #253 Approved by: cgwalters
2016-04-04 14:48:08 +03:00
contextdir, add_files,
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
&ret_new_inputhash, error))
goto out;
[libhif] Actually honor --force-nocache option
2015-01-08 05:40:32 +03:00
/* Only look for previous checksum if caller has passed *out_unmodified */
if (self->previous_checksum && out_unmodified != NULL)
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
{
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autoptr(GVariant) commit_v = NULL;
g_autoptr(GVariant) commit_metadata = NULL;
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
const char *previous_inputhash = NULL;
if (!ostree_repo_load_variant (self->repo, OSTREE_OBJECT_TYPE_COMMIT,
self->previous_checksum,
&commit_v, error))
goto out;
commit_metadata = g_variant_get_child_value (commit_v, 0);
if (g_variant_lookup (commit_metadata, "rpmostree.inputhash", "&s", &previous_inputhash))
{
if (strcmp (previous_inputhash, ret_new_inputhash) == 0)
{
*out_unmodified = TRUE;
ret = TRUE;
goto out;
}
}
else
g_print ("Previous commit found, but without rpmostree.inputhash metadata key\n");
}
compose: Add a --dry-run option This way it's easier for people to inspect the next system state.
2016-03-01 23:38:45 +03:00
if (opt_dry_run)
{
ret = TRUE;
goto out;
}
[libhif]: Add initial progress reporting This requires open-coding the 3 steps that hif_context_run() was doing. Which is fine, because ultimately we want to have more control over that.
2015-01-02 01:32:40 +03:00
/* --- Downloading packages --- */
core: major rework This patch prepares RpmOstreeContext for supporting package layering. A relabel operation is added as well to support relabeling imported packages if the sepolicy of the rootfs we're overlaying onto is different from during import. Closes: #289 Approved by: cgwalters
2016-05-20 21:16:38 +03:00
if (!rpmostree_context_download (ctx, hifinstall, cancellable, error))
Major revamp/extension of libhif/unpacker code This is in preparation for `rpm-ostree container`, which handles unpacking RPMs as non-root. At the moment, I'm copying code in from both ostree's libarchive bits (fixable...may need to export some utility functions) and some functions from libhif (harder, see: http://lists.rpm.org/pipermail/rpm-ecosystem/2016-January/000297.html ) There's lots more cleanup to do here, but I don't want to block on the resolution of the libhif changes.
2016-01-26 22:34:51 +03:00
goto out;
Move console code to libglnx The main goal is to be able to share this with OSTree.
2015-02-18 00:54:54 +03:00
{ g_auto(GLnxConsoleRef) console = { 0, };
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
glnx_unref_object HifState *hifstate = hif_state_new ();
[libhif]: Add initial progress reporting This requires open-coding the 3 steps that hif_context_run() was doing. Which is fine, because ultimately we want to have more control over that.
2015-01-02 01:32:40 +03:00
progress_sigid = g_signal_connect (hifstate, "percentage-changed",
G_CALLBACK (on_hifstate_percentage_changed),
"Installing packages:");
Move console code to libglnx The main goal is to be able to share this with OSTree.
2015-02-18 00:54:54 +03:00
glnx_console_lock (&console);
[libhif]: Add initial progress reporting This requires open-coding the 3 steps that hif_context_run() was doing. Which is fine, because ultimately we want to have more control over that.
2015-01-02 01:32:40 +03:00
if (!hif_transaction_commit (hif_context_get_transaction (hifctx),
hif_context_get_goal (hifctx),
hifstate,
error))
goto out;
g_signal_handler_disconnect (hifstate, progress_sigid);
}
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
ret = TRUE;
[libhif] Actually honor --force-nocache option
2015-01-08 05:40:32 +03:00
if (out_unmodified)
*out_unmodified = FALSE;
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
gs_transfer_out_value (out_new_inputhash, &ret_new_inputhash);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
out:
return ret;
}
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
static gboolean
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
process_includes (RpmOstreeTreeComposeContext *self,
GFile *treefile_path,
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
guint depth,
JsonObject *root,
GCancellable *cancellable,
GError **error)
{
gboolean ret = FALSE;
const char *include_path;
const guint maxdepth = 50;
if (depth > maxdepth)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Exceeded maximum include depth of %u", maxdepth);
goto out;
}
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
{
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autoptr(GFile) parent = g_file_get_parent (treefile_path);
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
gboolean existed = FALSE;
if (self->treefile_context_dirs->len > 0)
{
GFile *prev = self->treefile_context_dirs->pdata[self->treefile_context_dirs->len-1];
if (g_file_equal (parent, prev))
existed = TRUE;
}
if (!existed)
{
g_ptr_array_add (self->treefile_context_dirs, parent);
parent = NULL; /* Transfer ownership */
}
}
compose: Split out JSON helpers into separate file In preparation for using some of them in the postprocessing code too. Yeah, parsing JSON in C sucks =( Oh well.
2014-11-13 22:53:43 +03:00
if (!_rpmostree_jsonutil_object_get_optional_string_member (root, "include", &include_path, error))
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
goto out;
if (include_path)
{
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autoptr(GFile) treefile_dirpath = g_file_get_parent (treefile_path);
g_autoptr(GFile) parent_path = g_file_resolve_relative_path (treefile_dirpath, include_path);
glnx_unref_object JsonParser *parent_parser = json_parser_new ();
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
JsonNode *parent_rootval;
JsonObject *parent_root;
GList *members;
GList *iter;
if (!json_parser_load_from_file (parent_parser,
gs_file_get_path_cached (parent_path),
error))
goto out;
parent_rootval = json_parser_get_root (parent_parser);
if (!JSON_NODE_HOLDS_OBJECT (parent_rootval))
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Treefile root is not an object");
goto out;
}
parent_root = json_node_get_object (parent_rootval);
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
if (!process_includes (self, parent_path, depth + 1, parent_root,
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
cancellable, error))
goto out;
members = json_object_get_members (parent_root);
for (iter = members; iter; iter = iter->next)
{
const char *name = iter->data;
JsonNode *parent_val = json_object_get_member (parent_root, name);
JsonNode *val = json_object_get_member (root, name);
g_assert (parent_val);
if (!val)
json_object_set_member (root, name, json_node_copy (parent_val));
else
{
JsonNodeType parent_type =
json_node_get_node_type (parent_val);
JsonNodeType child_type =
json_node_get_node_type (val);
if (parent_type != child_type)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Conflicting element type of '%s'",
name);
goto out;
}
if (child_type == JSON_NODE_ARRAY)
{
JsonArray *parent_array = json_node_get_array (parent_val);
JsonArray *child_array = json_node_get_array (val);
JsonArray *new_child = json_array_new ();
guint i, len;
len = json_array_get_length (parent_array);
for (i = 0; i < len; i++)
json_array_add_element (new_child, json_node_copy (json_array_get_element (parent_array, i)));
len = json_array_get_length (child_array);
for (i = 0; i < len; i++)
json_array_add_element (new_child, json_node_copy (json_array_get_element (child_array, i)));
json_object_set_array_member (root, name, new_child);
}
}
}
compose: Put the expanded treefile content in /usr/share/rpm-ostree/treefile.json Since the treefile format now supports includes, we would need to either include the whole chain, or just the expanded portion we use in the compose. This patch does the latter. This should allow a client to take the same treefile and generate a similar tree (if they want to reproduce with the same RPMs, those can be extracted from the RPM database inside the tree).
2014-09-07 20:38:34 +04:00
json_object_remove_member (root, "include");
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
}
ret = TRUE;
out:
return ret;
}
treecompose: Add --add-metadata-string argument for versioning This will be used by rpm-ostree-toolbox to inject version numbers for the trees.
2014-10-14 17:32:29 +04:00
static gboolean
parse_keyvalue_strings (char **strings,
compose: Convert metadata handling into builder We're going to start adding our own metadata, so take this initial step of having the user-specified metadata accumulated into a builder.
2015-01-03 05:21:30 +03:00
GVariantBuilder *builder,
treecompose: Add --add-metadata-string argument for versioning This will be used by rpm-ostree-toolbox to inject version numbers for the trees.
2014-10-14 17:32:29 +04:00
GError **error)
{
gboolean ret = FALSE;
char **iter;
for (iter = strings; *iter; iter++)
{
const char *s;
const char *eq;
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autofree char *key = NULL;
treecompose: Add --add-metadata-string argument for versioning This will be used by rpm-ostree-toolbox to inject version numbers for the trees.
2014-10-14 17:32:29 +04:00
s = *iter;
eq = strchr (s, '=');
if (!eq)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Missing '=' in KEY=VALUE metadata '%s'", s);
goto out;
}
key = g_strndup (s, eq - s);
g_variant_builder_add (builder, "{sv}", key,
g_variant_new_string (eq + 1));
}
ret = TRUE;
out:
return ret;
}
compose: Add automatic_version_prefix to json config.
2015-01-07 09:30:27 +03:00
static gboolean
compose_strv_contains_prefix (gchar **strv,
const gchar *prefix)
{
if (!strv)
return FALSE;
while (*strv)
{
if (g_str_has_prefix (*strv, prefix))
return TRUE;
++strv;
}
return FALSE;
}
app: Allow for custom exit status codes Change the command callback return type to integer, so commands can return a custom exit status. Usually it should be EXIT_SUCCESS (0) or EXIT_FAILURE (1).
2015-11-02 23:43:58 +03:00
int
Add new "compose" builtin, rename "treecompose" -> "compose tree" And do the same for "sign". This way we can have the compose server utilities cleanly separated from what most people will see, which is the client side tools.
2014-05-26 23:05:08 +04:00
rpmostree_compose_builtin_tree (int argc,
char **argv,
GCancellable *cancellable,
GError **error)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{
app: Allow for custom exit status codes Change the command callback return type to integer, so commands can return a custom exit status. Usually it should be EXIT_SUCCESS (0) or EXIT_FAILURE (1).
2015-11-02 23:43:58 +03:00
int exit_status = EXIT_FAILURE;
compose: Raise up the code to read the previous (OSTree) commit I'm planning to replace the caching code with something that inspects the previous commit rather than a lookaside cache, so raise this code up to a higher level.
2015-01-03 04:42:25 +03:00
GError *temp_error = NULL;
app: Add rpmostree_usage_error() Convenience function for missing or invalid command-line arguments.
2015-10-26 16:36:22 +03:00
GOptionContext *context = g_option_context_new ("TREEFILE - Run yum and commit the result to an OSTree repository");
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
RpmOstreeTreeComposeContext selfdata = { NULL, };
RpmOstreeTreeComposeContext *self = &selfdata;
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
JsonNode *treefile_rootval = NULL;
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
JsonObject *treefile = NULL;
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autofree char *cachekey = NULL;
g_autofree char *new_inputhash = NULL;
g_autoptr(GFile) previous_root = NULL;
g_autofree char *previous_checksum = NULL;
g_autoptr(GFile) yumroot = NULL;
g_autoptr(GFile) yumroot_varcache = NULL;
glnx_unref_object OstreeRepo *repo = NULL;
g_autoptr(GPtrArray) bootstrap_packages = NULL;
g_autoptr(GPtrArray) packages = NULL;
g_autoptr(GFile) treefile_path = NULL;
g_autoptr(GFile) treefile_dirpath = NULL;
g_autoptr(GFile) repo_path = NULL;
glnx_unref_object JsonParser *treefile_parser = NULL;
compose: Convert metadata handling into builder We're going to start adding our own metadata, so take this initial step of having the user-specified metadata accumulated into a builder.
2015-01-03 05:21:30 +03:00
gs_unref_variant_builder GVariantBuilder *metadata_builder =
g_variant_builder_new (G_VARIANT_TYPE ("a{sv}"));
compose: Introduce variable substitution for ${basearch} This comes from the tradition of yum repo files. It's significantly easier for rpm-ostree users building multiple architectures if some core templating for the treefiles is built in. Otherwise, everyone needs to learn about a new wrapper tool for rpm-ostree, and said tool would need to re-do the same "basearch" evaluation that is already occuring inside libhif. This commit also paves the way for introducing `${releasever}` substitution. NOTE: This depends on pending changes to libhif git. Closes: #274 Approved by: jlebon
2016-04-28 15:59:20 +03:00
g_autoptr(RpmOstreeContext) corectx = NULL;
g_autoptr(GHashTable) varsubsts = NULL;
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
gboolean workdir_is_tmp = FALSE;
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
self->treefile_context_dirs = g_ptr_array_new_with_free_func ((GDestroyNotify)g_object_unref);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
app: Collect D-Bus CLI options into main.c In the style of ostree's CLI, add some option parsing flags RPM_OSTREE_BUILTIN_FLAG_NONE RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD and extend rpmostree_option_context_parse() to handle the --sysroot and --peer options and return an OstreeSysroot proxy object (unless the flag RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD is passed).
2015-08-05 19:39:07 +03:00
if (!rpmostree_option_context_parse (context,
option_entries,
&argc, &argv,
RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD,
cancellable,
NULL,
error))
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
goto out;
core: Use generic "builtin" infrastructure for command handling I want to have more options than just "create".
2014-03-22 23:05:41 +04:00
if (argc < 2)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{
app: Add rpmostree_usage_error() Convenience function for missing or invalid command-line arguments.
2015-10-26 16:36:22 +03:00
rpmostree_usage_error (context, "TREEFILE must be specified", error);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
goto out;
}
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
if (!opt_repo)
{
app: Add rpmostree_usage_error() Convenience function for missing or invalid command-line arguments.
2015-10-26 16:36:22 +03:00
rpmostree_usage_error (context, "--repo must be specified", error);
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
goto out;
}
compose: Require being run as root Someday we'll enhance RPM; see https://github.com/rpm-software-management/rpm/issues/8 But anyways right now at least a few people have tried, and we get all the way past downloading packages and then bomb out when doing the actual transaction with a useless error message. So let's be up front about this.
2015-07-24 05:10:48 +03:00
if (getuid () != 0)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"compose tree must presently be run as uid 0 (root)");
goto out;
}
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
repo_path = g_file_new_for_path (opt_repo);
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
repo = self->repo = ostree_repo_new (repo_path);
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
if (!ostree_repo_open (repo, cancellable, error))
goto out;
treecompose: Add --add-metadata-string argument for versioning This will be used by rpm-ostree-toolbox to inject version numbers for the trees.
2014-10-14 17:32:29 +04:00
core: Use generic "builtin" infrastructure for command handling I want to have more options than just "create".
2014-03-22 23:05:41 +04:00
treefile_path = g_file_new_for_path (argv[1]);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
if (opt_workdir)
{
compose: Move 'workdir' to be 'self' member This avoids passing another parameter around.
2014-11-17 04:05:47 +03:00
self->workdir = g_file_new_for_path (opt_workdir);
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
}
else
{
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autofree char *tmpd = NULL;
libpriv: Introduce an rpmostree_mkdtemp() - Can also give you a file descriptor - Takes a constant string as input, returning a mutated string as a separate variable which means that one can check whether the variable is `NULL` to know whether or not one needs to `rm -rf` it on error paths.
2015-05-22 21:19:34 +03:00
if (!rpmostree_mkdtemp ("/var/tmp/rpm-ostree.XXXXXX", &tmpd, NULL, error))
goto out;
compose: Move 'workdir' to be 'self' member This avoids passing another parameter around.
2014-11-17 04:05:47 +03:00
self->workdir = g_file_new_for_path (tmpd);
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
workdir_is_tmp = TRUE;
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
if (opt_workdir_tmpfs)
{
if (mount ("tmpfs", tmpd, "tmpfs", 0, (const void*)"mode=755") != 0)
{
util: Add helper function to set GError from errno Really this should be in libgsystem (or possibly glib-unix.h), we have a copy in ostree. But we can refactor to use a shared version later.
2014-11-20 02:32:08 +03:00
_rpmostree_set_prefix_error_from_errno (error, errno,
"mount(tmpfs): ");
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
goto out;
}
}
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
}
Start using libglnx The high level goal is to deprecate libgsystem. I was trying to share code between ostree/rpm-ostree, but it was too painful to commit to forver frozen ABI for new utility APIs. The git submodule approach will much more easily allow breaking API/ABI, and iterate on APIs until they either land in GLib or not. Note that libglnx will not use GFile*, so a full port to it will involve also not using that. Thus, it will be necessarily incremental; in the meantime we'll link to both libgsystem and libglnx.
2015-02-15 21:56:14 +03:00
if (!glnx_opendirat (AT_FDCWD, gs_file_get_path_cached (self->workdir),
FALSE, &self->workdir_dfd, error))
goto out;
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
if (opt_cachedir)
compose-tree: Auto-create cache directory It's just more friendly.
2014-06-07 02:25:08 +04:00
{
treecompose: Make use of cachedir again (just for metadata) We need to do selinux first.
2016-02-10 15:47:32 +03:00
if (!glnx_opendirat (AT_FDCWD, opt_cachedir, TRUE, &self->cachedir_dfd, error))
{
g_prefix_error (error, "Opening cachedir '%s': ", opt_cachedir);
goto out;
}
}
else
{
self->cachedir_dfd = fcntl (self->workdir_dfd, F_DUPFD_CLOEXEC, 3);
if (self->cachedir_dfd < 0)
{
glnx_set_error_from_errno (error);
goto out;
}
compose-tree: Auto-create cache directory It's just more friendly.
2014-06-07 02:25:08 +04:00
}
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
treecompose: Add --add-metadata-string argument for versioning This will be used by rpm-ostree-toolbox to inject version numbers for the trees.
2014-10-14 17:32:29 +04:00
if (opt_metadata_strings)
{
if (!parse_keyvalue_strings (opt_metadata_strings,
compose: Convert metadata handling into builder We're going to start adding our own metadata, so take this initial step of having the user-specified metadata accumulated into a builder.
2015-01-03 05:21:30 +03:00
metadata_builder, error))
treecompose: Add --add-metadata-string argument for versioning This will be used by rpm-ostree-toolbox to inject version numbers for the trees.
2014-10-14 17:32:29 +04:00
goto out;
}
Start using libglnx The high level goal is to deprecate libgsystem. I was trying to share code between ostree/rpm-ostree, but it was too painful to commit to forver frozen ABI for new utility APIs. The git submodule approach will much more easily allow breaking API/ABI, and iterate on APIs until they either land in GLib or not. Note that libglnx will not use GFile*, so a full port to it will involve also not using that. Thus, it will be necessarily incremental; in the meantime we'll link to both libgsystem and libglnx.
2015-02-15 21:56:14 +03:00
if (fchdir (self->workdir_dfd) != 0)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{
Start using libglnx The high level goal is to deprecate libgsystem. I was trying to share code between ostree/rpm-ostree, but it was too painful to commit to forver frozen ABI for new utility APIs. The git submodule approach will much more easily allow breaking API/ABI, and iterate on APIs until they either land in GLib or not. Note that libglnx will not use GFile*, so a full port to it will involve also not using that. Thus, it will be necessarily incremental; in the meantime we'll link to both libgsystem and libglnx.
2015-02-15 21:56:14 +03:00
glnx_set_error_from_errno (error);
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
goto out;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
}
core: Make unprivileged case ignore ownership, add "_compose" context I was in the process of trying to support `%post` scripts, and I wanted to use `rpm-ostree container` for convenient and safe testing. However the recent package layering changes broke it to error out on perms like `filesystem`'s `root:mail` on `/var/mail`. I decided to introduce a new `rpmostree_context_new_compose` which had the current behavior, switch `compose tree` to use it, and then change `_new_unprivileged` to *really* be unprivileged. Specifically we ignore file ownership (and fix dir owners) because we assume we'll be operating with `bare-user` repos. Closes: #327 Approved by: jlebon
2016-06-16 15:52:29 +03:00
corectx = rpmostree_context_new_compose (self->cachedir_dfd, cancellable, error);
compose: Introduce variable substitution for ${basearch} This comes from the tradition of yum repo files. It's significantly easier for rpm-ostree users building multiple architectures if some core templating for the treefiles is built in. Otherwise, everyone needs to learn about a new wrapper tool for rpm-ostree, and said tool would need to re-do the same "basearch" evaluation that is already occuring inside libhif. This commit also paves the way for introducing `${releasever}` substitution. NOTE: This depends on pending changes to libhif git. Closes: #274 Approved by: jlebon
2016-04-28 15:59:20 +03:00
if (!corectx)
goto out;
varsubsts = rpmostree_context_get_varsubsts (corectx);
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
treefile_parser = json_parser_new ();
if (!json_parser_load_from_file (treefile_parser,
gs_file_get_path_cached (treefile_path),
error))
goto out;
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
treefile_rootval = json_parser_get_root (treefile_parser);
if (!JSON_NODE_HOLDS_OBJECT (treefile_rootval))
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Treefile root is not an object");
goto out;
}
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
treefile = json_node_get_object (treefile_rootval);
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
if (!process_includes (self, treefile_path, 0, treefile,
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
cancellable, error))
goto out;
if (opt_print_only)
{
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
glnx_unref_object JsonGenerator *generator = json_generator_new ();
g_autoptr(GOutputStream) stdout = g_unix_output_stream_new (1, FALSE);
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
json_generator_set_pretty (generator, TRUE);
json_generator_set_root (generator, treefile_rootval);
(void) json_generator_to_stream (generator, stdout, NULL, NULL);
app: Allow for custom exit status codes Change the command callback return type to integer, so commands can return a custom exit status. Usually it should be EXIT_SUCCESS (0) or EXIT_FAILURE (1).
2015-11-02 23:43:58 +03:00
exit_status = EXIT_SUCCESS;
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
goto out;
}
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
compose: Introduce variable substitution for ${basearch} This comes from the tradition of yum repo files. It's significantly easier for rpm-ostree users building multiple architectures if some core templating for the treefiles is built in. Otherwise, everyone needs to learn about a new wrapper tool for rpm-ostree, and said tool would need to re-do the same "basearch" evaluation that is already occuring inside libhif. This commit also paves the way for introducing `${releasever}` substitution. NOTE: This depends on pending changes to libhif git. Closes: #274 Approved by: jlebon
2016-04-28 15:59:20 +03:00
{ const char *input_ref = _rpmostree_jsonutil_object_require_string_member (treefile, "ref", error);
if (!input_ref)
goto out;
self->ref = _rpmostree_varsubst_string (input_ref, varsubsts, error);
if (!self->ref)
goto out;
}
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
if (!ostree_repo_read_commit (repo, self->ref, &previous_root, &previous_checksum,
compose: Raise up the code to read the previous (OSTree) commit I'm planning to replace the caching code with something that inspects the previous commit rather than a lookaside cache, so raise this code up to a higher level.
2015-01-03 04:42:25 +03:00
cancellable, &temp_error))
{
if (g_error_matches (temp_error, G_IO_ERROR, G_IO_ERROR_NOT_FOUND))
{
g_clear_error (&temp_error);
Introduce "treespec" concept as GKeyFile I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).
2016-02-10 11:25:58 +03:00
g_print ("No previous commit for %s\n", self->ref);
compose: Raise up the code to read the previous (OSTree) commit I'm planning to replace the caching code with something that inspects the previous commit rather than a lookaside cache, so raise this code up to a higher level.
2015-01-03 04:42:25 +03:00
}
else
{
g_propagate_error (error, temp_error);
goto out;
}
}
else
g_print ("Previous commit: %s\n", previous_checksum);
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
self->previous_checksum = previous_checksum;
compose: Raise up the code to read the previous (OSTree) commit I'm planning to replace the caching code with something that inspects the previous commit rather than a lookaside cache, so raise this code up to a higher level.
2015-01-03 04:42:25 +03:00
yumroot = g_file_get_child (self->workdir, "rootfs.tmp");
Start using libglnx The high level goal is to deprecate libgsystem. I was trying to share code between ostree/rpm-ostree, but it was too painful to commit to forver frozen ABI for new utility APIs. The git submodule approach will much more easily allow breaking API/ABI, and iterate on APIs until they either land in GLib or not. Note that libglnx will not use GFile*, so a full port to it will involve also not using that. Thus, it will be necessarily incremental; in the meantime we'll link to both libgsystem and libglnx.
2015-02-15 21:56:14 +03:00
if (!glnx_shutil_rm_rf_at (self->workdir_dfd, "rootfs.tmp", cancellable, error))
compose: Raise up the code to read the previous (OSTree) commit I'm planning to replace the caching code with something that inspects the previous commit rather than a lookaside cache, so raise this code up to a higher level.
2015-01-03 04:42:25 +03:00
goto out;
compose: Add automatic_version_prefix to json config.
2015-01-07 09:30:27 +03:00
if (json_object_has_member (treefile, "automatic_version_prefix") &&
!compose_strv_contains_prefix (opt_metadata_strings, "version="))
{
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autoptr(GVariant) variant = NULL;
g_autofree char *last_version = NULL;
g_autofree char *next_version = NULL;
compose: Add automatic_version_prefix to json config.
2015-01-07 09:30:27 +03:00
const char *ver_prefix;
ver_prefix = _rpmostree_jsonutil_object_require_string_member (treefile,
"automatic_version_prefix",
error);
if (!ver_prefix)
goto out;
if (previous_checksum)
{
if (!ostree_repo_load_variant (repo, OSTREE_OBJECT_TYPE_COMMIT,
previous_checksum, &variant, error))
goto out;
last_version = checksum_version (variant);
}
compose: Create _rpmostree_util_next_version for testing
2015-01-12 08:07:33 +03:00
next_version = _rpmostree_util_next_version (ver_prefix, last_version);
compose: Add automatic_version_prefix to json config.
2015-01-07 09:30:27 +03:00
g_variant_builder_add (metadata_builder, "{sv}", "version",
g_variant_new_string (next_version));
}
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
bootstrap_packages = g_ptr_array_new ();
packages = g_ptr_array_new ();
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
compose: Make 'bootstrap_packages' actually optional now It has in practice been optional since: 827e711eb74135f075f02f5c44685e4e35c1a8b1 Now let's let people write treefiles without it.
2014-11-17 23:48:08 +03:00
if (json_object_has_member (treefile, "bootstrap_packages"))
{
jsonutil: Remove useless GCancellable Spotted by mbarnes from review of https://github.com/projectatomic/rpm-ostree/pull/52/commits
2014-11-18 00:08:03 +03:00
if (!_rpmostree_jsonutil_append_string_array_to (treefile, "bootstrap_packages", packages, error))
compose: Make 'bootstrap_packages' actually optional now It has in practice been optional since: 827e711eb74135f075f02f5c44685e4e35c1a8b1 Now let's let people write treefiles without it.
2014-11-17 23:48:08 +03:00
goto out;
}
jsonutil: Remove useless GCancellable Spotted by mbarnes from review of https://github.com/projectatomic/rpm-ostree/pull/52/commits
2014-11-18 00:08:03 +03:00
if (!_rpmostree_jsonutil_append_string_array_to (treefile, "packages", packages, error))
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
goto out;
g_ptr_array_add (packages, NULL);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
{ glnx_unref_object JsonGenerator *generator = json_generator_new ();
compose: Serialize treefile early This way we don't emit a critical if we exit due to having already cached content.
2014-09-30 00:24:53 +04:00
char *treefile_buf = NULL;
gsize len;
json_generator_set_root (generator, treefile_rootval);
json_generator_set_pretty (generator, TRUE);
treefile_buf = json_generator_to_data (generator, &len);
self->serialized_treefile = g_bytes_new_take (treefile_buf, len);
}
passwd: Use check-* file json config. entries to migrate data across commits
2015-01-20 09:37:22 +03:00
treefile_dirpath = g_file_get_parent (treefile_path);
if (TRUE)
compose: Raise up the code to read the previous (OSTree) commit I'm planning to replace the caching code with something that inspects the previous commit rather than a lookaside cache, so raise this code up to a higher level.
2015-01-03 04:42:25 +03:00
{
gboolean generate_from_previous = TRUE;
compose: Support "preserve-passwd" option (enabled by default) The checking code from #56 landed, and started triggering for me on the `dockerroot` user. It's nice to know it works. Then the issue is... "what now"? It turns out in the case of `dockerroot` it's actually unused, so we could fix this by deleting it. But in general we need to support dynamic uids/gids/. And we can't yet take a hard dep on #49. So this patch changes things so we take a copy of the passwd/group data from the previous commit. Any users subsequently added in the *new* commit will be additive. Closes: https://github.com/projectatomic/rpm-ostree/issues/78
2014-12-24 00:28:53 +03:00
compose: Raise up the code to read the previous (OSTree) commit I'm planning to replace the caching code with something that inspects the previous commit rather than a lookaside cache, so raise this code up to a higher level.
2015-01-03 04:42:25 +03:00
if (!_rpmostree_jsonutil_object_get_optional_boolean_member (treefile,
"preserve-passwd",
&generate_from_previous,
error))
goto out;
compose: Support "preserve-passwd" option (enabled by default) The checking code from #56 landed, and started triggering for me on the `dockerroot` user. It's nice to know it works. Then the issue is... "what now"? It turns out in the case of `dockerroot` it's actually unused, so we could fix this by deleting it. But in general we need to support dynamic uids/gids/. And we can't yet take a hard dep on #49. So this patch changes things so we take a copy of the passwd/group data from the previous commit. Any users subsequently added in the *new* commit will be additive. Closes: https://github.com/projectatomic/rpm-ostree/issues/78
2014-12-24 00:28:53 +03:00
compose: Raise up the code to read the previous (OSTree) commit I'm planning to replace the caching code with something that inspects the previous commit rather than a lookaside cache, so raise this code up to a higher level.
2015-01-03 04:42:25 +03:00
if (generate_from_previous)
{
passwd: Use check-* file json config. entries to migrate data across commits
2015-01-20 09:37:22 +03:00
if (!rpmostree_generate_passwd_from_previous (repo, yumroot,
treefile_dirpath,
previous_root, treefile,
compose: Raise up the code to read the previous (OSTree) commit I'm planning to replace the caching code with something that inspects the previous commit rather than a lookaside cache, so raise this code up to a higher level.
2015-01-03 04:42:25 +03:00
cancellable, error))
goto out;
}
}
compose: Support "preserve-passwd" option (enabled by default) The checking code from #56 landed, and started triggering for me on the `dockerroot` user. It's nice to know it works. Then the issue is... "what now"? It turns out in the case of `dockerroot` it's actually unused, so we could fix this by deleting it. But in general we need to support dynamic uids/gids/. And we can't yet take a hard dep on #49. So this patch changes things so we take a copy of the passwd/group data from the previous commit. Any users subsequently added in the *new* commit will be additive. Closes: https://github.com/projectatomic/rpm-ostree/issues/78
2014-12-24 00:28:53 +03:00
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
{ gboolean unmodified = FALSE;
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
compose: Introduce variable substitution for ${basearch} This comes from the tradition of yum repo files. It's significantly easier for rpm-ostree users building multiple architectures if some core templating for the treefiles is built in. Otherwise, everyone needs to learn about a new wrapper tool for rpm-ostree, and said tool would need to re-do the same "basearch" evaluation that is already occuring inside libhif. This commit also paves the way for introducing `${releasever}` substitution. NOTE: This depends on pending changes to libhif git. Closes: #274 Approved by: jlebon
2016-04-28 15:59:20 +03:00
if (!install_packages_in_root (self, corectx, treefile, yumroot,
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
(char**)packages->pdata,
[libhif] Actually honor --force-nocache option
2015-01-08 05:40:32 +03:00
opt_force_nocache ? NULL : &unmodified,
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
&new_inputhash,
cancellable, error))
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
goto out;
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
if (unmodified)
{
g_print ("No apparent changes since previous commit; use --force-nocache to override\n");
app: Allow for custom exit status codes Change the command callback return type to integer, so commands can return a custom exit status. Usually it should be EXIT_SUCCESS (0) or EXIT_FAILURE (1).
2015-11-02 23:43:58 +03:00
exit_status = EXIT_SUCCESS;
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
goto out;
}
compose: Add a --dry-run option This way it's easier for people to inspect the next system state.
2016-03-01 23:38:45 +03:00
else if (opt_dry_run)
{
g_print ("--dry-run complete, exiting\n");
exit_status = EXIT_SUCCESS;
goto out;
}
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
}
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
if (g_strcmp0 (g_getenv ("RPM_OSTREE_BREAK"), "post-yum") == 0)
goto out;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
compose: Support 'postprocess-script' This is obviously a total cop-out. However, without glibc fixes, we can't do better. See: https://bugzilla.redhat.com/show_bug.cgi?id=156477
2014-11-14 19:53:21 +03:00
if (!rpmostree_treefile_postprocessing (yumroot, self->treefile_context_dirs->pdata[0],
self->serialized_treefile, treefile,
compose: Move content processing code into -postprocess It seems clearer to me if all code which is operating on the content (such as enabling systemd units) is under -postprocess. The "compose tree" code should be tying everything together and calling out to sub-components. This is prep for adding more postprocessing code.
2014-11-13 22:54:33 +03:00
cancellable, error))
goto out;
compose: Move boot-location handling to postprocessing Originally I was trying to have all of the treefile parsing in treecompose, and then call other functions as API. But that turns out to be unrealistic. We'll need finer grained control over this in the future. Therefore, let's move the boot-location handling down, in preparation for further commits which parse the treefile in the commit phase.
2014-11-17 17:18:02 +03:00
if (!rpmostree_prepare_rootfs_for_commit (yumroot, treefile, cancellable, error))
goto out;
compose: Add check-passwd/group JSON options, fails compose if uids/gids change Verify uid/gid on files, directories and symlinks Just output a msg when user/group is removed with no files json-parsing: Add functions for strictly dealing with ints passwd/json: Add simple scripts to convert passwd/group files to json data docs: Check-passwd/groups and ignore-remove-users/groups JSON config. entries
2014-11-20 17:40:35 +03:00
compose: support adding external files This will allow to copy arbitrary files into the rootfs, specifying something like: "add-files": [["service.template", "/exports/service.template"], ["config.json.template", "/exports/config.json.template"]] It is quite useful when building a container image. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #253 Approved by: cgwalters
2016-03-30 16:35:52 +03:00
if (!rpmostree_copy_additional_files (yumroot, self->treefile_context_dirs->pdata[0], treefile, cancellable, error))
goto out;
passwd: Use check-* file json config. entries to migrate data across commits
2015-01-20 09:37:22 +03:00
if (!rpmostree_check_passwd (repo, yumroot, treefile_dirpath, treefile,
compose: Reuse previous commit lookup in passwd checking A future commit is going to change our parsing of the "ref" member in treefiles, so ensure we only load it once early on in compose-tree. We already looked up the previous commit there, so just pass it down rather than reloading the ref. Closes: #274 Approved by: jlebon
2016-04-28 16:09:40 +03:00
previous_checksum,
compose: Add check-passwd/group JSON options, fails compose if uids/gids change Verify uid/gid on files, directories and symlinks Just output a msg when user/group is removed with no files json-parsing: Add functions for strictly dealing with ints passwd/json: Add simple scripts to convert passwd/group files to json data docs: Check-passwd/groups and ignore-remove-users/groups JSON config. entries
2014-11-20 17:40:35 +03:00
cancellable, error))
goto out;
passwd: Use check-* file json config. entries to migrate data across commits
2015-01-20 09:37:22 +03:00
if (!rpmostree_check_groups (repo, yumroot, treefile_dirpath, treefile,
compose: Reuse previous commit lookup in passwd checking A future commit is going to change our parsing of the "ref" member in treefiles, so ensure we only load it once early on in compose-tree. We already looked up the previous commit there, so just pass it down rather than reloading the ref. Closes: #274 Approved by: jlebon
2016-04-28 16:09:40 +03:00
previous_checksum,
compose: Add check-passwd/group JSON options, fails compose if uids/gids change Verify uid/gid on files, directories and symlinks Just output a msg when user/group is removed with no files json-parsing: Add functions for strictly dealing with ints passwd/json: Add simple scripts to convert passwd/group files to json data docs: Check-passwd/groups and ignore-remove-users/groups JSON config. entries
2014-11-20 17:40:35 +03:00
cancellable, error))
goto out;
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
{
const char *gpgkey;
do not crash when no selinux tag is present Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-03-30 17:54:33 +03:00
gboolean selinux = TRUE;
Switch to glib cleanup macros We don't have a lot of outstanding changes to the C code, so now seems like a good time to do this. I implemented this with some highly sophisticated sed commands like: ``` find -name '*.c' | while read name; do sed -i -e 's,gs_unref_object \([A-Za-z]*\) \*,g_autoptr(\1),' ${name}; done ``` Part of dropping the dependency on libgsystem, same as what we're doing in ostree. Closes: #393 Approved by: jlebon
2016-07-15 06:12:37 +03:00
g_autoptr(GVariant) metadata = NULL;
[libhif] Rework compose caching with new packaging APIs With yum, we would have had to make a custom Python app to cleanly separate the fetch metadata/depsolve phases from installation. Now that libhif/hawkey gives us that, make use of it by exiting after depsolve if the previous compose has the same package set, and the treefile is the same. This saves a fairly substantial amount of time and I/O, and makes it much more palatable to simply run the compose tool on demand in response to say repo regeneration notifications. A further important note; --cachedir is no longer used; we store the inputhash in the OSTree commit metadata itself.
2015-01-03 05:54:23 +03:00
g_variant_builder_add (metadata_builder, "{sv}",
"rpmostree.inputhash",
g_variant_new_string (new_inputhash));
metadata = g_variant_ref_sink (g_variant_builder_end (metadata_builder));
compose: Convert metadata handling into builder We're going to start adding our own metadata, so take this initial step of having the user-specified metadata accumulated into a builder.
2015-01-03 05:21:30 +03:00
compose: Split out JSON helpers into separate file In preparation for using some of them in the postprocessing code too. Yeah, parsing JSON in C sucks =( Oh well.
2014-11-13 22:53:43 +03:00
if (!_rpmostree_jsonutil_object_get_optional_string_member (treefile, "gpg_key", &gpgkey, error))
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
goto out;
autobuilder: Avoid critical if there's no gpg_key
2014-03-29 04:15:43 +04:00
do not crash when no selinux tag is present Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-03-30 17:54:33 +03:00
if (!_rpmostree_jsonutil_object_get_optional_boolean_member (treefile,
"selinux",
&selinux,
error))
goto out;
pkg-add: New builtin to layer additional packages This builds upon the earlier prototype in https://github.com/cgwalters/atomic-pkglayer The `.origin` file says for a replicated installation: [origin] refspec=local:rhel-atomic-host/7/x86_64/standard If you then run `rpm-ostree pkg-add strace`, it will result in a new tree with: [origin] baserefspec=local:rhel-atomic-host/7/x86_64/standard [packages] requested=strace; Work still remaining here is to teach `rpm-ostree status` and `rpm-ostree upgrade` about this. Closes: #289 Approved by: cgwalters
2015-02-11 21:06:43 +03:00
{ g_autofree char *new_revision = NULL;
glnx_fd_close int rootfs_fd = -1;
if (!glnx_opendirat (AT_FDCWD, gs_file_get_path_cached (yumroot), TRUE,
&rootfs_fd, error))
goto out;
g_print ("Committing...\n");
if (!rpmostree_commit (rootfs_fd, repo, self->ref, metadata, gpgkey, selinux, NULL,
&new_revision,
cancellable, error))
goto out;
package-layering fixups This patch fixes up errors in the parent commit which added pkg-add. Some of them are due to the rebase on top of the unprivileged infrastructure. - fix compile errors (due to libhif changes after rebase) - delete duplicate prototype for rpmostree_sysroot_upgrader_deploy - include allow-older in flags type - fix change_upgrader_refspec to use g_strdup() (this was causing the wrong old refspec to be registered) - in builtin-status.c, check for NULL before joining the packages array - sysroot-upgrader: fix gtype function names - roc_context_prepare_for_root(): delete unused param - assemble_commit(): delete unused param - RpmOstreeSysrootUpgraderFlags: fix docs - Fix sysroot property name and add reboot opt Closes: #289 Approved by: cgwalters
2016-02-25 20:42:56 +03:00
g_print ("%s => %s\n", self->ref, new_revision);
pkg-add: New builtin to layer additional packages This builds upon the earlier prototype in https://github.com/cgwalters/atomic-pkglayer The `.origin` file says for a replicated installation: [origin] refspec=local:rhel-atomic-host/7/x86_64/standard If you then run `rpm-ostree pkg-add strace`, it will result in a new tree with: [origin] baserefspec=local:rhel-atomic-host/7/x86_64/standard [packages] requested=strace; Work still remaining here is to teach `rpm-ostree status` and `rpm-ostree upgrade` about this. Closes: #289 Approved by: cgwalters
2015-02-11 21:06:43 +03:00
}
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
}
compose-tree: Add --touch-if-changed This is a convenient way for external tools to integrate rpm-ostree better into automation. With the caching, it's now cheaper to just rerun rpm-ostree if you know an input yum repository changed, or just on a timer. Then one can use this flag to determine whether or not to launch any further processes after rpm-ostree, such as disk image generation, test suites, etc.
2015-01-29 00:34:29 +03:00
if (opt_touch_if_changed)
{
gs_fd_close int fd = open (opt_touch_if_changed, O_CREAT|O_WRONLY|O_NOCTTY, 0644);
if (fd == -1)
{
gs_set_error_from_errno (error, errno);
g_prefix_error (error, "Updating '%s': ", opt_touch_if_changed);
goto out;
}
if (futimens (fd, NULL) == -1)
{
gs_set_error_from_errno (error, errno);
goto out;
}
}
app: Allow for custom exit status codes Change the command callback return type to integer, so commands can return a custom exit status. Usually it should be EXIT_SUCCESS (0) or EXIT_FAILURE (1).
2015-11-02 23:43:58 +03:00
exit_status = EXIT_SUCCESS;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
out:
compose: Explicitly clear out context object before removing rootfs This squashes an ugly `/var/tmp/rootfs.XXXX/usr/share/rpm: ENOENT` warning that occurs because we were nuking the temporary rootfs before cleaning up the context object, which was indirectly holding references to it. Closes: #319 Approved by: jlebon
2016-06-13 22:40:22 +03:00
/* Explicitly close this one now as it may have references to files
* we delete below.
*/
g_clear_object (&corectx);
compose: Ensure we've cleaned up references to tmpfs workdir before umount Otherwise the `umount()` will always fail. This hasn't been a problem so far while running in a external container (docker/systemd-nspawn), but is when running in `mock` because it doesn't set its namespace to be private. This should help Fedora's Bodhi, which uses rpm-ostree inside mock.
2015-10-31 04:40:14 +03:00
/* Move back out of the workding directory to ensure unmount works */
(void )chdir ("/");
if (self->workdir_dfd != -1)
(void) close (self->workdir_dfd);
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
compose tree: Cleanup workdir always Particularly if we hit the cached case, but we might as well always do it.
2014-06-11 15:27:31 +04:00
if (workdir_is_tmp)
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
{
if (opt_workdir_tmpfs)
compose: Ensure we've cleaned up references to tmpfs workdir before umount Otherwise the `umount()` will always fail. This hasn't been a problem so far while running in a external container (docker/systemd-nspawn), but is when running in `mock` because it doesn't set its namespace to be private. This should help Fedora's Bodhi, which uses rpm-ostree inside mock.
2015-10-31 04:40:14 +03:00
if (umount (gs_file_get_path_cached (self->workdir)) != 0)
{
fprintf (stderr, "warning: umount failed: %m\n");
}
compose: Move 'workdir' to be 'self' member This avoids passing another parameter around.
2014-11-17 04:05:47 +03:00
(void) gs_shutil_rm_rf (self->workdir, NULL, NULL);
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
}
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
if (self)
{
compose: Move 'workdir' to be 'self' member This avoids passing another parameter around.
2014-11-17 04:05:47 +03:00
g_clear_object (&self->workdir);
compose: Put the expanded treefile content in /usr/share/rpm-ostree/treefile.json Since the treefile format now supports includes, we would need to either include the whole chain, or just the expanded portion we use in the compose. This patch does the latter. This should allow a client to take the same treefile and generate a similar tree (if they want to reproduce with the same RPMs, those can be extracted from the RPM database inside the tree).
2014-09-07 20:38:34 +04:00
g_clear_pointer (&self->serialized_treefile, g_bytes_unref);
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
g_ptr_array_unref (self->treefile_context_dirs);
}
app: Allow for custom exit status codes Change the command callback return type to integer, so commands can return a custom exit status. Usually it should be EXIT_SUCCESS (0) or EXIT_FAILURE (1).
2015-11-02 23:43:58 +03:00
return exit_status;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
}
Reference in New Issue Copy Permalink