keremet/rpm-ostree
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
48e8c126f2
rpm-ostree/src/rpmostree-compose-builtin-tree.c

1068 lines
33 KiB
C
Raw Normal View History

Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
/* -*- mode: C; c-file-style: "gnu"; indent-tabs-mode: nil; -*-
*
* Copyright (C) 2013,2014 Colin Walters <walters@verbum.org>
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published
* by the Free Software Foundation; either version 2 of the licence or (at
* your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General
* Public License along with this library; if not, write to the
* Free Software Foundation, Inc., 59 Temple Place, Suite 330,
* Boston, MA 02111-1307, USA.
*/
#include "config.h"
#include <string.h>
#include <glib-unix.h>
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
#include <json-glib/json-glib.h>
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
#include <sys/mount.h>
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
#include <gio/gunixoutputstream.h>
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Add new "compose" builtin, rename "treecompose" -> "compose tree" And do the same for "sign". This way we can have the compose server utilities cleanly separated from what most people will see, which is the client side tools.
2014-05-26 23:05:08 +04:00
#include "rpmostree-compose-builtins.h"
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
#include "rpmostree-util.h"
Unify rpm-ostree into one binary And start making an internal library.
2014-01-29 23:37:44 +04:00
#include "rpmostree-postprocess.h"
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
#include "libgsystem.h"
static char *opt_workdir;
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
static gboolean opt_workdir_tmpfs;
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
static char *opt_cachedir;
static char *opt_proxy;
static char *opt_repo;
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
static gboolean opt_print_only;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
static GOptionEntry option_entries[] = {
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
{ "workdir", 0, 0, G_OPTION_ARG_STRING, &opt_workdir, "Working directory", "WORKDIR" },
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
{ "workdir-tmpfs", 0, 0, G_OPTION_ARG_NONE, &opt_workdir_tmpfs, "Use tmpfs for working state", NULL },
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
{ "cachedir", 0, 0, G_OPTION_ARG_STRING, &opt_cachedir, "Cached state", "CACHEDIR" },
{ "repo", 'r', 0, G_OPTION_ARG_STRING, &opt_repo, "Path to OSTree repository", "REPO" },
{ "proxy", 0, 0, G_OPTION_ARG_STRING, &opt_proxy, "HTTP proxy", "PROXY" },
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
{ "print-only", 0, 0, G_OPTION_ARG_NONE, &opt_print_only, "Just expand any includes and print treefile", NULL },
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{ NULL }
};
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
typedef struct {
GPtrArray *treefile_context_dirs;
} RpmOstreeTreeComposeContext;
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
static char *
subprocess_context_print_args (GSSubprocessContext *ctx)
{
GString *ret = g_string_new ("");
gs_strfreev char **argv = NULL;
char **strviter;
g_object_get ((GObject*)ctx, "argv", &argv, NULL);
for (strviter = argv; strviter && *strviter; strviter++)
{
gs_free char *quoted = g_shell_quote (*strviter);
treecompose: Add missing space in subprocess printing
2014-05-06 01:57:16 +04:00
g_string_append_c (ret, ' ');
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
g_string_append (ret, quoted);
}
return g_string_free (ret, FALSE);
}
autobuilder: Avoid critical if there's no gpg_key
2014-03-29 04:15:43 +04:00
static gboolean
object_get_optional_string_member (JsonObject *object,
const char *member_name,
const char **out_value,
GError **error)
{
gboolean ret = FALSE;
JsonNode *node = json_object_get_member (object, member_name);
if (node != NULL)
{
*out_value = json_node_get_string (node);
if (!*out_value)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Member '%s' is not a string", member_name);
goto out;
}
}
else
*out_value = NULL;
ret = TRUE;
out:
return ret;
}
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
static const char *
object_require_string_member (JsonObject *object,
const char *member_name,
GError **error)
{
const char *ret;
if (!object_get_optional_string_member (object, member_name, &ret, error))
return NULL;
if (!ret)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Member '%s' not found", member_name);
return NULL;
}
return ret;
}
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
static const char *
array_require_string_element (JsonArray *array,
guint i,
GError **error)
{
const char *ret = json_array_get_string_element (array, i);
if (!ret)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Element at index %u is not a string", i);
return NULL;
}
return ret;
}
static gboolean
append_string_array_to (JsonObject *object,
const char *member_name,
GPtrArray *array,
GError **error)
{
JsonArray *jarray = json_object_get_array_member (object, member_name);
guint i, len;
if (!jarray)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"No member '%s' found", member_name);
return FALSE;
}
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
len = json_array_get_length (jarray);
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
for (i = 0; i < len; i++)
{
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
const char *v = array_require_string_element (jarray, i, error);
if (!v)
return FALSE;
g_ptr_array_add (array, g_strdup (v));
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
}
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
return TRUE;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
}
typedef struct {
GSSubprocess *process;
GFile *tmp_reposdir_path;
GDataOutputStream *stdin;
/* GDataInputStream *stdout; */
} YumContext;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
static gboolean
yum_context_close (YumContext *yumctx,
GCancellable *cancellable,
GError **error)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
gboolean ret = FALSE;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
if (!yumctx)
return TRUE;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
if (yumctx->process)
{
if (yumctx->stdin)
{
if (!g_output_stream_close ((GOutputStream*)yumctx->stdin, cancellable, error))
goto out;
g_clear_object (&yumctx->stdin);
}
/*
if (yumctx->stdout)
{
if (!g_input_stream_close ((GInputStream*)yumctx->stdout, cancellable, error))
goto out;
g_clear_object (&yumctx->stdout);
}
*/
g_print ("Waiting for yum...\n");
if (!gs_subprocess_wait_sync_check (yumctx->process, cancellable, error))
goto out;
g_print ("Waiting for yum [OK]\n");
g_clear_object (&yumctx->process);
}
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
ret = TRUE;
out:
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
return ret;
}
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
static void
yum_context_free (YumContext *yumctx)
{
if (!yumctx)
return;
(void) yum_context_close (yumctx, NULL, NULL);
g_free (yumctx);
}
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
static inline
void cleanup_keyfile_unref (void *loc)
{
GKeyFile *locp = *((GKeyFile**)loc);
if (locp)
g_key_file_unref (locp);
}
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
static gboolean
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
append_repo_and_cache_opts (RpmOstreeTreeComposeContext *self,
JsonObject *treedata,
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
GFile *workdir,
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
GPtrArray *args,
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
GCancellable *cancellable,
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
GError **error)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
gboolean ret = FALSE;
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
JsonArray *enable_repos = NULL;
JsonArray *repos_data = NULL;
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
guint i;
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
gs_unref_object GFile *yumcache_lookaside = NULL;
gs_unref_object GFile *repos_tmpdir = NULL;
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
yumcache_lookaside = g_file_resolve_relative_path (workdir, "yum-cache");
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
if (!gs_file_ensure_directory (yumcache_lookaside, TRUE, cancellable, error))
goto out;
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
repos_tmpdir = g_file_resolve_relative_path (workdir, "tmp-repos");
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
if (!gs_shutil_rm_rf (repos_tmpdir, cancellable, error))
goto out;
if (!gs_file_ensure_directory (repos_tmpdir, TRUE, cancellable, error))
goto out;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
if (g_getenv ("RPM_OSTREE_OFFLINE"))
g_ptr_array_add (args, g_strdup ("-C"));
treecompose: Honor http_proxy environment variable
2014-05-18 17:29:51 +04:00
{
const char *proxy;
if (opt_proxy)
proxy = opt_proxy;
else
proxy = g_getenv ("http_proxy");
if (proxy)
g_ptr_array_add (args, g_strconcat ("--setopt=proxy=", proxy, NULL));
}
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
{
GString *reposdir_value = g_string_new ("--setopt=reposdir=");
gboolean first = TRUE;
for (i = 0; i < self->treefile_context_dirs->len; i++)
{
GFile *contextdir = self->treefile_context_dirs->pdata[i];
if (first)
first = FALSE;
else
g_string_append_c (reposdir_value, ',');
g_string_append (reposdir_value, gs_file_get_path_cached (contextdir));
}
g_ptr_array_add (args, g_string_free (reposdir_value, FALSE));
}
g_ptr_array_add (args, g_strdup ("--disablerepo=*"));
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
if (json_object_has_member (treedata, "repos"))
enable_repos = json_object_get_array_member (treedata, "repos");
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
if (enable_repos)
{
guint i;
guint n = json_array_get_length (enable_repos);
for (i = 0; i < n; i++)
{
const char *reponame = array_require_string_element (enable_repos, i, error);
if (!reponame)
goto out;
g_ptr_array_add (args, g_strconcat ("--enablerepo=", reponame, NULL));
}
}
core: Use --setopt=cachedir rather than copying/moving I don't know why I wasn't doing this before, it's clearly better.
2014-01-31 04:04:58 +04:00
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
if (json_object_has_member (treedata, "repos_data"))
repos_data = json_object_get_array_member (treedata, "repos_data");
else
repos_data = NULL;
if (repos_data)
{
guint i;
guint n = json_array_get_length (repos_data);
if (n > 0)
g_ptr_array_add (args, g_strconcat ("--setopt=reposdir=/etc/yum.repos.d,",
gs_file_get_path_cached (repos_tmpdir),
NULL));
for (i = 0; i < n; i++)
{
const char *repodata = array_require_string_element (repos_data, i, error);
__attribute__ ((cleanup(cleanup_keyfile_unref))) GKeyFile *keyfile = NULL;
gs_strfreev char **groups = NULL;
const char *reponame;
gs_free char *rpmostree_reponame = NULL;
gs_free char *rpmostree_repo_filename = NULL;
gs_unref_object GFile *repo_tmp_file = NULL;
gsize len;
if (!repodata)
goto out;
keyfile = g_key_file_new ();
if (!g_key_file_load_from_data (keyfile, repodata, -1, 0, error))
{
g_prefix_error (error, "Parsing keyfile data in repos_data: ");
goto out;
}
groups = g_key_file_get_groups (keyfile, &len);
if (len == 0)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"No groups found in keyfile data in repos_data");
goto out;
}
reponame = groups[0];
g_assert (strchr (reponame, '/') == NULL);
rpmostree_reponame = g_strconcat (reponame, NULL);
rpmostree_repo_filename = g_strconcat (rpmostree_reponame, ".repo", NULL);
repo_tmp_file = g_file_get_child (repos_tmpdir, rpmostree_repo_filename);
if (!g_file_replace_contents (repo_tmp_file, repodata, strlen (repodata),
NULL, FALSE, 0,
NULL,
cancellable, error))
goto out;
g_ptr_array_add (args, g_strconcat ("--enablerepo=", rpmostree_reponame, NULL));
}
}
compose: Tell yum not to cache packages locally They get deleted anyways since we blow away the tmpdir, but we expect people to be using a local HTTP proxy. We stopped trying to cache packages internally as that caused conflicts with the yum lock with concurrently executing composes.
2014-06-19 19:57:33 +04:00
g_ptr_array_add (args, g_strdup ("--setopt=keepcache=0"));
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
g_ptr_array_add (args, g_strconcat ("--setopt=cachedir=",
gs_file_get_path_cached (yumcache_lookaside),
NULL));
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
ret = TRUE;
out:
return ret;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
}
static YumContext *
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
yum_context_new (RpmOstreeTreeComposeContext *self,
JsonObject *treedata,
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
GFile *yumroot,
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
GFile *workdir,
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
GCancellable *cancellable,
GError **error)
{
gboolean success = FALSE;
YumContext *yumctx = NULL;
GPtrArray *yum_argv = g_ptr_array_new_with_free_func (g_free);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
gs_unref_object GSSubprocessContext *context = NULL;
gs_unref_object GSSubprocess *yum_process = NULL;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
g_ptr_array_add (yum_argv, g_strdup ("yum"));
g_ptr_array_add (yum_argv, g_strdup ("-y"));
core: Use --setopt=cachedir rather than copying/moving I don't know why I wasn't doing this before, it's clearly better.
2014-01-31 04:04:58 +04:00
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
if (!append_repo_and_cache_opts (self, treedata, workdir, yum_argv,
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
cancellable, error))
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
goto out;
core: Use --setopt=cachedir rather than copying/moving I don't know why I wasn't doing this before, it's clearly better.
2014-01-31 04:04:58 +04:00
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
g_ptr_array_add (yum_argv, g_strconcat ("--installroot=",
gs_file_get_path_cached (yumroot),
NULL));
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
g_ptr_array_add (yum_argv, g_strdup ("shell"));
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
g_ptr_array_add (yum_argv, NULL);
context = gs_subprocess_context_new ((char**)yum_argv->pdata);
{
gs_strfreev char **duped_environ = g_get_environ ();
duped_environ = g_environ_setenv (duped_environ, "OSTREE_KERNEL_INSTALL_NOOP", "1", TRUE);
rpm-ostree: Set HARDLINK=no in environment during install This avoids the kernel pointlessly doing its own checksum of content.
2014-02-21 23:56:12 +04:00
/* See fedora's kernel.spec */
duped_environ = g_environ_setenv (duped_environ, "HARDLINK", "no", TRUE);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
gs_subprocess_context_set_environment (context, duped_environ);
}
treecompose: Fix use-before-alloc
2014-05-05 16:34:45 +04:00
gs_subprocess_context_set_stdin_disposition (context, GS_SUBPROCESS_STREAM_DISPOSITION_PIPE);
/* gs_subprocess_context_set_stdout_disposition (context, GS_SUBPROCESS_STREAM_DISPOSITION_PIPE); */
yumctx = g_new0 (YumContext, 1);
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
{
gs_free char *cmdline = subprocess_context_print_args (context);
g_print ("Starting %s\n", cmdline);
}
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
yumctx->process = gs_subprocess_new (context, cancellable, error);
if (!yumctx->process)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
goto out;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
yumctx->stdin = (GDataOutputStream*)g_data_output_stream_new (gs_subprocess_get_stdin_pipe (yumctx->process));
/* yumctx->stdout = (GDataInputStream*)g_data_input_stream_new (gs_subprocess_get_stdout_pipe (yumctx->process)); */
success = TRUE;
out:
if (!success)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
yum_context_free (yumctx);
return NULL;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
}
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
return yumctx;
}
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
static gboolean
yum_context_command (YumContext *yumctx,
const char *cmd,
GPtrArray **out_lines,
GCancellable *cancellable,
GError **error)
{
gboolean ret = FALSE;
gsize bytes_written;
gs_unref_ptrarray GPtrArray *lines = g_ptr_array_new_with_free_func (g_free);
gs_free char *cmd_nl = g_strconcat (cmd, "\n", NULL);
g_print ("yum> %s", cmd_nl);
if (!g_output_stream_write_all ((GOutputStream*)yumctx->stdin,
cmd_nl, strlen (cmd_nl), &bytes_written,
cancellable, error))
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
goto out;
ret = TRUE;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
gs_transfer_out_value (out_lines, &lines);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
out:
return ret;
}
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
static gboolean
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
yuminstall (RpmOstreeTreeComposeContext *self,
JsonObject *treedata,
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
GFile *yumroot,
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
GFile *workdir,
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
char **packages,
GCancellable *cancellable,
GError **error)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{
gboolean ret = FALSE;
char **strviter;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
YumContext *yumctx;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
yumctx = yum_context_new (self, treedata, yumroot, workdir, cancellable, error);
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
if (!yumctx)
goto out;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
for (strviter = packages; strviter && *strviter; strviter++)
{
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
gs_free char *cmd = NULL;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
const char *package = *strviter;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
gs_unref_ptrarray GPtrArray *lines = NULL;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
if (g_str_has_prefix (package, "@"))
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
cmd = g_strconcat ("group install ", package, NULL);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
else
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
cmd = g_strconcat ("install ", package, NULL);
if (!yum_context_command (yumctx, cmd, &lines,
cancellable, error))
goto out;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
}
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
{
gs_unref_ptrarray GPtrArray *lines = NULL;
if (!yum_context_command (yumctx, "run", &lines,
cancellable, error))
goto out;
}
if (!yum_context_close (yumctx, cancellable, error))
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
goto out;
ret = TRUE;
out:
return ret;
}
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
static gboolean
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
process_includes (RpmOstreeTreeComposeContext *self,
GFile *treefile_path,
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
guint depth,
JsonObject *root,
GCancellable *cancellable,
GError **error)
{
gboolean ret = FALSE;
const char *include_path;
const guint maxdepth = 50;
if (depth > maxdepth)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Exceeded maximum include depth of %u", maxdepth);
goto out;
}
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
{
gs_unref_object GFile *parent = g_file_get_parent (treefile_path);
gboolean existed = FALSE;
if (self->treefile_context_dirs->len > 0)
{
GFile *prev = self->treefile_context_dirs->pdata[self->treefile_context_dirs->len-1];
if (g_file_equal (parent, prev))
existed = TRUE;
}
if (!existed)
{
g_ptr_array_add (self->treefile_context_dirs, parent);
parent = NULL; /* Transfer ownership */
}
}
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
if (!object_get_optional_string_member (root, "include", &include_path, error))
goto out;
if (include_path)
{
gs_unref_object GFile *treefile_dirpath = g_file_get_parent (treefile_path);
gs_unref_object GFile *parent_path = g_file_resolve_relative_path (treefile_dirpath, include_path);
gs_unref_object JsonParser *parent_parser = json_parser_new ();
JsonNode *parent_rootval;
JsonObject *parent_root;
GList *members;
GList *iter;
if (!json_parser_load_from_file (parent_parser,
gs_file_get_path_cached (parent_path),
error))
goto out;
parent_rootval = json_parser_get_root (parent_parser);
if (!JSON_NODE_HOLDS_OBJECT (parent_rootval))
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Treefile root is not an object");
goto out;
}
parent_root = json_node_get_object (parent_rootval);
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
if (!process_includes (self, parent_path, depth + 1, parent_root,
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
cancellable, error))
goto out;
members = json_object_get_members (parent_root);
for (iter = members; iter; iter = iter->next)
{
const char *name = iter->data;
JsonNode *parent_val = json_object_get_member (parent_root, name);
JsonNode *val = json_object_get_member (root, name);
g_assert (parent_val);
if (!val)
json_object_set_member (root, name, json_node_copy (parent_val));
else
{
JsonNodeType parent_type =
json_node_get_node_type (parent_val);
JsonNodeType child_type =
json_node_get_node_type (val);
if (parent_type != child_type)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Conflicting element type of '%s'",
name);
goto out;
}
if (child_type == JSON_NODE_ARRAY)
{
JsonArray *parent_array = json_node_get_array (parent_val);
JsonArray *child_array = json_node_get_array (val);
JsonArray *new_child = json_array_new ();
guint i, len;
len = json_array_get_length (parent_array);
for (i = 0; i < len; i++)
json_array_add_element (new_child, json_node_copy (json_array_get_element (parent_array, i)));
len = json_array_get_length (child_array);
for (i = 0; i < len; i++)
json_array_add_element (new_child, json_node_copy (json_array_get_element (child_array, i)));
json_object_set_array_member (root, name, new_child);
}
}
}
}
ret = TRUE;
out:
return ret;
}
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
static char *
cachedir_fssafe_key (const char *primary_key)
{
GString *ret = g_string_new ("");
for (; *primary_key; primary_key++)
{
const char c = *primary_key;
if (!g_ascii_isprint (c) || c == '-')
g_string_append_printf (ret, "\\%02x", c);
else if (c == '/')
g_string_append_c (ret, '-');
else
g_string_append_c (ret, c);
}
return g_string_free (ret, FALSE);
}
static GFile *
cachedir_keypath (GFile *cachedir,
const char *primary_key)
{
gs_free char *fssafe_key = cachedir_fssafe_key (primary_key);
return g_file_get_child (cachedir, fssafe_key);
}
static gboolean
cachedir_lookup_string (GFile *cachedir,
const char *key,
char **out_value,
GCancellable *cancellable,
GError **error)
{
gboolean ret = FALSE;
gs_free char *ret_value = NULL;
if (cachedir)
{
gs_unref_object GFile *keypath = cachedir_keypath (cachedir, key);
if (!_rpmostree_file_load_contents_utf8_allow_noent (keypath, &ret_value,
cancellable, error))
goto out;
}
ret = TRUE;
gs_transfer_out_value (out_value, &ret_value);
out:
return ret;
}
static gboolean
cachedir_set_string (GFile *cachedir,
const char *key,
const char *value,
GCancellable *cancellable,
GError **error)
{
gboolean ret = FALSE;
gs_unref_object GFile *keypath = NULL;
if (!cachedir)
return TRUE;
keypath = cachedir_keypath (cachedir, key);
if (!g_file_replace_contents (keypath, value, strlen (value), NULL,
FALSE, 0, NULL,
cancellable, error))
goto out;
ret = TRUE;
out:
return ret;
}
static gboolean
compute_checksum_for_compose (JsonObject *treefile_rootval,
GFile *yumroot,
char **out_checksum,
GCancellable *cancellable,
GError **error)
{
gboolean ret = FALSE;
gs_free char *ret_checksum = NULL;
GChecksum *checksum = g_checksum_new (G_CHECKSUM_SHA256);
{
JsonNode *treefile_rootnode = json_node_new (JSON_NODE_OBJECT);
gs_unref_object JsonGenerator *generator = json_generator_new ();
gs_free char *treefile_buf = NULL;
gsize len;
json_node_set_object (treefile_rootnode, treefile_rootval);
json_generator_set_root (generator, treefile_rootnode);
treefile_buf = json_generator_to_data (generator, &len);
g_checksum_update (checksum, (guint8*)treefile_buf, len);
json_node_free (treefile_rootnode);
}
tree compose: Delete .dbenv.lock and __db.* files from /usr/share/rpm Currently on an Atomic compose, I'm seeing abrtd trying to write to /usr/share/rpm/.dbenv.lock, which is denied by policy because it's usr_t. There are multiple ways to address this, but there's no good reason to leave the lock files and __db* files around. rpm appears to operate correctly without them if calling process merely gets EROFS.
2014-06-06 22:51:18 +04:00
/* Query the generated rpmdb, to see if anything has changed. */
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
{
int estatus;
tree compose: Delete .dbenv.lock and __db.* files from /usr/share/rpm Currently on an Atomic compose, I'm seeing abrtd trying to write to /usr/share/rpm/.dbenv.lock, which is denied by policy because it's usr_t. There are multiple ways to address this, but there's no good reason to leave the lock files and __db* files around. rpm appears to operate correctly without them if calling process merely gets EROFS.
2014-06-06 22:51:18 +04:00
gs_free char *yumroot_var_lib_rpm =
g_build_filename (gs_file_get_path_cached (yumroot),
"var/lib/rpm",
NULL);
const char *rpmqa_argv[] = { PKGLIBDIR "/rpmqa-sorted-and-clean",
yumroot_var_lib_rpm,
NULL };
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
gs_free char *rpmqa_result = NULL;
if (!g_spawn_sync (NULL, (char**)rpmqa_argv, NULL,
G_SPAWN_SEARCH_PATH, NULL, NULL,
&rpmqa_result, NULL, &estatus, error))
goto out;
if (!g_spawn_check_exit_status (estatus, error))
tree compose: Delete .dbenv.lock and __db.* files from /usr/share/rpm Currently on an Atomic compose, I'm seeing abrtd trying to write to /usr/share/rpm/.dbenv.lock, which is denied by policy because it's usr_t. There are multiple ways to address this, but there's no good reason to leave the lock files and __db* files around. rpm appears to operate correctly without them if calling process merely gets EROFS.
2014-06-06 22:51:18 +04:00
{
g_prefix_error (error, "Executing %s: ",
rpmqa_argv[0]);
goto out;
}
compose-tree: Fix location of rpmdb querying for caching We were just repeatedly hitting the cache...oops. Also add an assertion that the returned data isn't empty.
2014-05-29 22:57:55 +04:00
if (!*rpmqa_result)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
tree compose: Delete .dbenv.lock and __db.* files from /usr/share/rpm Currently on an Atomic compose, I'm seeing abrtd trying to write to /usr/share/rpm/.dbenv.lock, which is denied by policy because it's usr_t. There are multiple ways to address this, but there's no good reason to leave the lock files and __db* files around. rpm appears to operate correctly without them if calling process merely gets EROFS.
2014-06-06 22:51:18 +04:00
"Empty result from %s", rpmqa_argv[0]);
compose-tree: Fix location of rpmdb querying for caching We were just repeatedly hitting the cache...oops. Also add an assertion that the returned data isn't empty.
2014-05-29 22:57:55 +04:00
goto out;
}
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
g_checksum_update (checksum, (guint8*)rpmqa_result, strlen (rpmqa_result));
}
ret_checksum = g_strdup (g_checksum_get_string (checksum));
ret = TRUE;
gs_transfer_out_value (out_checksum, &ret_checksum);
out:
if (checksum) g_checksum_free (checksum);
return ret;
}
core: Use generic "builtin" infrastructure for command handling I want to have more options than just "create".
2014-03-22 23:05:41 +04:00
gboolean
Add new "compose" builtin, rename "treecompose" -> "compose tree" And do the same for "sign". This way we can have the compose server utilities cleanly separated from what most people will see, which is the client side tools.
2014-05-26 23:05:08 +04:00
rpmostree_compose_builtin_tree (int argc,
char **argv,
GCancellable *cancellable,
GError **error)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{
core: Use generic "builtin" infrastructure for command handling I want to have more options than just "create".
2014-03-22 23:05:41 +04:00
gboolean ret = FALSE;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
GOptionContext *context = g_option_context_new ("- Run yum and commit the result to an OSTree repository");
const char *ref;
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
RpmOstreeTreeComposeContext selfdata = { NULL, };
RpmOstreeTreeComposeContext *self = &selfdata;
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
JsonNode *treefile_rootval = NULL;
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
JsonObject *treefile = NULL;
Support "repos_data" in treefiles, use it for automatic overrides/ For convenience, allow users to create an "overrides/" directory which is a repo. If it exists, the builder automatically adds it. (However, we should likely disallow distributing these builds on the network)
2014-02-26 02:07:59 +04:00
JsonArray *units = NULL;
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
guint len;
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
guint i;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
gs_free char *ref_unix = NULL;
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
gs_free char *cachekey = NULL;
gs_free char *cached_compose_checksum = NULL;
gs_free char *new_compose_checksum = NULL;
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
gs_unref_object GFile *workdir = NULL;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
gs_unref_object GFile *cachedir = NULL;
gs_unref_object GFile *yumroot = NULL;
gs_unref_object GFile *targetroot = NULL;
gs_unref_object GFile *yumroot_varcache = NULL;
Unify rpm-ostree into one binary And start making an internal library.
2014-01-29 23:37:44 +04:00
gs_unref_object OstreeRepo *repo = NULL;
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
gs_unref_ptrarray GPtrArray *bootstrap_packages = NULL;
gs_unref_ptrarray GPtrArray *packages = NULL;
gs_unref_object GFile *treefile_path = NULL;
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
gs_unref_object GFile *repo_path = NULL;
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
gs_unref_object JsonParser *treefile_parser = NULL;
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
gboolean workdir_is_tmp = FALSE;
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
self->treefile_context_dirs = g_ptr_array_new_with_free_func ((GDestroyNotify)g_object_unref);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
g_option_context_add_main_entries (context, option_entries, NULL);
if (!g_option_context_parse (context, &argc, &argv, error))
goto out;
core: Use generic "builtin" infrastructure for command handling I want to have more options than just "create".
2014-03-22 23:05:41 +04:00
if (argc < 2)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{
Add new "compose" builtin, rename "treecompose" -> "compose tree" And do the same for "sign". This way we can have the compose server utilities cleanly separated from what most people will see, which is the client side tools.
2014-05-26 23:05:08 +04:00
g_printerr ("usage: rpm-ostree compose tree TREEFILE\n");
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Option processing failed");
goto out;
}
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
if (!opt_repo)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"--repo must be specified");
goto out;
}
repo_path = g_file_new_for_path (opt_repo);
repo = ostree_repo_new (repo_path);
if (!ostree_repo_open (repo, cancellable, error))
goto out;
core: Use generic "builtin" infrastructure for command handling I want to have more options than just "create".
2014-03-22 23:05:41 +04:00
treefile_path = g_file_new_for_path (argv[1]);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
if (opt_workdir)
{
workdir = g_file_new_for_path (opt_workdir);
}
else
{
gs_free char *tmpd = g_mkdtemp (g_strdup ("/var/tmp/rpm-ostree.XXXXXX"));
workdir = g_file_new_for_path (tmpd);
workdir_is_tmp = TRUE;
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
if (opt_workdir_tmpfs)
{
/* Use a private mount namespace to avoid polluting the global
* namespace, and to ensure the mount gets cleaned up if we exit
* unexpectedly.
*/
if (unshare (CLONE_NEWNS) != 0)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"unshare(CLONE_NEWNS): %s", g_strerror (errno));
goto out;
}
if (mount (NULL, "/", "none", MS_PRIVATE | MS_REC, NULL) == -1)
{
int errsv = errno;
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"mount(/, MS_PRIVATE | MS_REC): %s",
g_strerror (errsv));
goto out;
}
if (mount ("tmpfs", tmpd, "tmpfs", 0, (const void*)"mode=755") != 0)
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"mount(tmpfs): %s", g_strerror (errno));
goto out;
}
}
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
}
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
if (opt_cachedir)
compose-tree: Auto-create cache directory It's just more friendly.
2014-06-07 02:25:08 +04:00
{
cachedir = g_file_new_for_path (opt_cachedir);
if (!gs_file_ensure_directory (cachedir, FALSE, cancellable, error))
goto out;
}
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
if (chdir (gs_file_get_path_cached (workdir)) != 0)
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
{
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Failed to chdir to '%s': %s",
opt_workdir, strerror (errno));
goto out;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
}
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
treefile_parser = json_parser_new ();
if (!json_parser_load_from_file (treefile_parser,
gs_file_get_path_cached (treefile_path),
error))
goto out;
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
treefile_rootval = json_parser_get_root (treefile_parser);
if (!JSON_NODE_HOLDS_OBJECT (treefile_rootval))
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
{
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
"Treefile root is not an object");
goto out;
}
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
treefile = json_node_get_object (treefile_rootval);
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
if (!process_includes (self, treefile_path, 0, treefile,
treecompose: Add support for an "include" key This greatly increases the practiality of using treefiles directly instead of the old products.json.
2014-05-03 18:05:43 +04:00
cancellable, error))
goto out;
if (opt_print_only)
{
gs_unref_object JsonGenerator *generator = json_generator_new ();
gs_unref_object GOutputStream *stdout = g_unix_output_stream_new (1, FALSE);
json_generator_set_pretty (generator, TRUE);
json_generator_set_root (generator, treefile_rootval);
(void) json_generator_to_stream (generator, stdout, NULL, NULL);
ret = TRUE;
goto out;
}
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
yumroot = g_file_get_child (workdir, "rootfs.tmp");
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
if (!gs_shutil_rm_rf (yumroot, cancellable, error))
goto out;
treecompose: Remove internal caching, require --repo argument It's a lot clearer if the inputs, outputs, and cache state are cleanly separated. At least the "lorax" tool relies on a local HTTP cache instead of keeping around the yum repos - let's do the same. This commit causes treecompose to require a --repo argument, and it also gains an optional --proxy argument.
2014-05-03 14:55:35 +04:00
targetroot = g_file_get_child (workdir, "rootfs");
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
ref = object_require_string_member (treefile, "ref", error);
if (!ref)
goto out;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
ref_unix = g_strdelimit (g_strdup (ref), "/", '_');
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
bootstrap_packages = g_ptr_array_new ();
packages = g_ptr_array_new ();
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
compose: Migrate content of /etc/{passwd,group} to /usr/lib more sanely I had an epiphany today while working on https://bugzilla.redhat.com/show_bug.cgi?id=1098304 - I realized that I can just do an install, and then copy over everything except the root entries from /etc/passwd into /usr/lib/passwd. No need for a patched shadow-utils. No need to modify the /etc/nsswitch.conf before doing the install root. It totally works. I have no idea why I originally overcomplicated this. The thing that sucks a bit about this code is that I have to drop to the FILE * APIs so that I can use the glibc APIs for processing group/shadow. Also, the way I deduplicated the code paths for processing passwd/group is crappy, but I think it's better than duplicating them (as systemd-sysusers does). The good: We don't need a two-step RPM transaction, we don't need a patch for shadow-utils, it's just saner The bad: Code is not the most beautiful? Not really bad. The ugly: I didn't think of this in the first place and spent months beating my head against the wall of shadow-utils...
2014-07-11 02:28:22 +04:00
if (!append_string_array_to (treefile, "bootstrap_packages", packages, error))
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
goto out;
if (!append_string_array_to (treefile, "packages", packages, error))
goto out;
g_ptr_array_add (packages, NULL);
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
if (!yuminstall (self, treefile, yumroot, workdir,
compose: Migrate content of /etc/{passwd,group} to /usr/lib more sanely I had an epiphany today while working on https://bugzilla.redhat.com/show_bug.cgi?id=1098304 - I realized that I can just do an install, and then copy over everything except the root entries from /etc/passwd into /usr/lib/passwd. No need for a patched shadow-utils. No need to modify the /etc/nsswitch.conf before doing the install root. It totally works. I have no idea why I originally overcomplicated this. The thing that sucks a bit about this code is that I have to drop to the FILE * APIs so that I can use the glibc APIs for processing group/shadow. Also, the way I deduplicated the code paths for processing passwd/group is crappy, but I think it's better than duplicating them (as systemd-sysusers does). The good: We don't need a two-step RPM transaction, we don't need a patch for shadow-utils, it's just saner The bad: Code is not the most beautiful? Not really bad. The ugly: I didn't think of this in the first place and spent months beating my head against the wall of shadow-utils...
2014-07-11 02:28:22 +04:00
(char**)packages->pdata,
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
cancellable, error))
goto out;
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
cachekey = g_strconcat ("treecompose/", ref, NULL);
if (!cachedir_lookup_string (cachedir, cachekey,
&cached_compose_checksum,
cancellable, error))
goto out;
if (!compute_checksum_for_compose (treefile, yumroot,
&new_compose_checksum,
cancellable, error))
goto out;
if (g_strcmp0 (cached_compose_checksum, new_compose_checksum) == 0)
{
g_print ("No changes to input, reusing cached commit\n");
ret = TRUE;
goto out;
}
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
ref_unix = g_strdelimit (g_strdup (ref), "/", '_');
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
if (g_strcmp0 (g_getenv ("RPM_OSTREE_BREAK"), "post-yum") == 0)
goto out;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
if (!rpmostree_postprocess (yumroot, cancellable, error))
goto out;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
if (json_object_has_member (treefile, "units"))
units = json_object_get_array_member (treefile, "units");
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
if (units)
len = json_array_get_length (units);
else
len = 0;
Resolve dependencies before building This allows us to skip the whole install process if the RPMs haven't changed, which is a rather large win. Part of this commit involves some code to attempt to talk to "yum shell" for dependency resolution, in an attempt to avoid depsolving twice, which just isn't really going to work, since "yum shell" isn't an API as it turns out. The only other real alternative is doing something PackageKit-like, which is ~4000 lines of very nontrivial Python.
2014-01-26 19:13:45 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
{
gs_unref_object GFile *multiuser_wants_dir =
g_file_resolve_relative_path (yumroot, "usr/etc/systemd/system/multi-user.target.wants");
core: Add an environment variable to breakpoint post-yum
2014-01-30 03:37:05 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
if (!gs_file_ensure_directory (multiuser_wants_dir, TRUE, cancellable, error))
Add postprocessing infrastructure We remove the root password from all images. The core/docker tree also gets docs and some kernel drivers removed.
2014-01-30 03:12:50 +04:00
goto out;
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
for (i = 0; i < len; i++)
{
const char *unitname = array_require_string_element (units, i, error);
gs_unref_object GFile *unit_link_target = NULL;
gs_free char *symlink_target = NULL;
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
if (!unitname)
goto out;
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
symlink_target = g_strconcat ("/usr/lib/systemd/system/", unitname, NULL);
unit_link_target = g_file_get_child (multiuser_wants_dir, unitname);
core: Don't make multi-user.target.wants link if it already exists
2014-02-28 19:44:43 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
if (g_file_query_file_type (unit_link_target, G_FILE_QUERY_INFO_NOFOLLOW_SYMLINKS, NULL) == G_FILE_TYPE_SYMBOLIC_LINK)
continue;
core: Don't make multi-user.target.wants link if it already exists
2014-02-28 19:44:43 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
g_print ("Adding %s to multi-user.target.wants\n", unitname);
core: Don't make multi-user.target.wants link if it already exists
2014-02-28 19:44:43 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
if (!g_file_make_symbolic_link (unit_link_target, symlink_target,
cancellable, error))
goto out;
}
}
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
{
gs_unref_object GFile *target_treefile_dir_path =
g_file_resolve_relative_path (yumroot, "usr/share/rpm-ostree");
gs_unref_object GFile *target_treefile_path =
g_file_get_child (target_treefile_dir_path, "treefile.json");
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
if (!gs_file_ensure_directory (target_treefile_dir_path, TRUE,
cancellable, error))
goto out;
Switch to taking JSON as input, generate "treefile" from products.json This is somewhat similar to what we've been doing with Continuous; we take the manifest.json, and turn it into a "snapshot". Except here there is a notion of inheritance. This gets stored into the tree as /usr/share/rpm-ostree/treefile.json. Additionally, it goes into the autobuilder directory in products-built.json. Though really we should split up that file, since it will be kind of...large.
2014-02-13 03:26:31 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
g_print ("Copying '%s' to '%s'\n",
gs_file_get_path_cached (treefile_path),
gs_file_get_path_cached (target_treefile_path));
if (!g_file_copy (treefile_path, target_treefile_path,
G_FILE_COPY_TARGET_DEFAULT_PERMS,
cancellable, NULL, NULL, error))
goto out;
}
compose: Add support for "default_target" Currently the systemd RPM ships with default.target -> graphical.target, which is either itself changed by Anaconda (via parsing /etc/sysconfig/desktop, which...anyways let's stop here). Or anaconda might set it directly to multi-user.target. For rpm-ostree, we perform some minimal level of "preconfiguration" per tree, so they are directly usable without an intervening installer. As an example for fedora-atomic/base/core, we just want multi-user.target. Thus, this patch provides the treefile author a declarative mechanism to set it.
2014-06-17 21:31:23 +04:00
{
const char *default_target = NULL;
if (!object_get_optional_string_member (treefile, "default_target",
&default_target, error))
goto out;
if (default_target != NULL)
{
gs_unref_object GFile *default_target_path =
g_file_resolve_relative_path (yumroot, "usr/etc/systemd/system/default.target");
(void) gs_file_unlink (default_target_path, NULL, NULL);
if (!g_file_make_symbolic_link (default_target_path, default_target,
cancellable, error))
goto out;
}
}
autobuilder: Avoid critical if there's no gpg_key
2014-03-29 04:15:43 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
{
const char *gpgkey;
if (!object_get_optional_string_member (treefile, "gpg_key", &gpgkey, error))
goto out;
autobuilder: Avoid critical if there's no gpg_key
2014-03-29 04:15:43 +04:00
treecompose: De-indent unnecessarily indented region
2014-05-16 01:47:47 +04:00
if (!rpmostree_commit (yumroot, repo, ref, gpgkey,
json_object_get_boolean_member (treefile, "selinux"),
cancellable, error))
Unify rpm-ostree into one binary And start making an internal library.
2014-01-29 23:37:44 +04:00
goto out;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
}
treecompose: Add initial caching The way this works is still fairly naive in that we hash in two inputs: 1) The treefile JSON 2) The result of rpm -qa If both of those are a hit, we reuse the existing commit.
2014-05-16 01:46:51 +04:00
if (!cachedir_set_string (cachedir, cachekey,
new_compose_checksum,
cancellable, error))
goto out;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
g_print ("Complete\n");
out:
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
compose tree: Cleanup workdir always Particularly if we hit the cached case, but we might as well always do it.
2014-06-11 15:27:31 +04:00
if (workdir_is_tmp)
compose: Add --workdir-tmpfs This ~halves the write traffic to my SSD when doing local composes. My laptop has 12GiB of RAM, plenty for this.
2014-07-02 06:07:56 +04:00
{
if (opt_workdir_tmpfs)
(void) umount (gs_file_get_path_cached (workdir));
(void) gs_shutil_rm_rf (workdir, NULL, NULL);
}
treecompose: Look for yum repo configs in same directory as treefiles This is significantly easier than requiring mutation of the host yum configuration.
2014-05-18 22:13:31 +04:00
if (self)
{
g_ptr_array_unref (self->treefile_context_dirs);
}
core: Use generic "builtin" infrastructure for command handling I want to have more options than just "create".
2014-03-22 23:05:41 +04:00
return ret;
Rewrite core utilities in C There are two major reasons: 1) I want to do things like process SELinux labels here, and that type of thing is best done in C. 2) There are presently 3 languages in this code, and this takes us down to just two.
2014-01-24 22:34:19 +04:00
}
Reference in New Issue Copy Permalink