From 694b798c73123521ddfcce2fdd828e80a71edf7c Mon Sep 17 00:00:00 2001 From: Colin Walters Date: Thu, 9 Nov 2017 14:54:33 -0500 Subject: [PATCH] Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon --- Makefile-libpriv.am | 5 + Makefile-rpm-ostree.am | 2 + design/jigdo.md | 97 ++ src/app/rpmostree-builtin-ex.c | 4 + src/app/rpmostree-compose-builtins.h | 2 + src/app/rpmostree-ex-builtin-commit2jigdo.c | 1228 +++++++++++++++++ src/app/rpmostree-ex-builtin-jigdo2commit.c | 368 +++++ src/app/rpmostree-ex-builtins.h | 2 + src/libpriv/rpmostree-core.c | 164 ++- src/libpriv/rpmostree-core.h | 21 + src/libpriv/rpmostree-importer.c | 84 +- src/libpriv/rpmostree-importer.h | 4 + src/libpriv/rpmostree-jigdo-assembler.c | 643 +++++++++ src/libpriv/rpmostree-jigdo-assembler.h | 70 + src/libpriv/rpmostree-jigdo-core.h | 77 ++ .../rpmostree-libarchive-input-stream.c | 171 +++ .../rpmostree-libarchive-input-stream.h | 66 + src/libpriv/rpmostree-util.c | 51 + src/libpriv/rpmostree-util.h | 5 + tests/compose-tests/test-jigdo.sh | 97 ++ .../composedata/fedora-atomic-host-oirpm.spec | 23 + 21 files changed, 3140 insertions(+), 44 deletions(-) create mode 100644 design/jigdo.md create mode 100644 src/app/rpmostree-ex-builtin-commit2jigdo.c create mode 100644 src/app/rpmostree-ex-builtin-jigdo2commit.c create mode 100644 src/libpriv/rpmostree-jigdo-assembler.c create mode 100644 src/libpriv/rpmostree-jigdo-assembler.h create mode 100644 src/libpriv/rpmostree-jigdo-core.h create mode 100644 src/libpriv/rpmostree-libarchive-input-stream.c create mode 100644 src/libpriv/rpmostree-libarchive-input-stream.h create mode 100755 tests/compose-tests/test-jigdo.sh create mode 100644 tests/composedata/fedora-atomic-host-oirpm.spec diff --git a/Makefile-libpriv.am b/Makefile-libpriv.am index 4b93a9dd..32e73d71 100644 --- a/Makefile-libpriv.am +++ b/Makefile-libpriv.am @@ -45,6 +45,9 @@ librpmostreepriv_la_SOURCES = \ src/libpriv/rpmostree-rpm-util.h \ src/libpriv/rpmostree-importer.c \ src/libpriv/rpmostree-importer.h \ + src/libpriv/rpmostree-jigdo-assembler.c \ + src/libpriv/rpmostree-jigdo-assembler.h \ + src/libpriv/rpmostree-jigdo-core.h \ src/libpriv/rpmostree-unpacker-core.c \ src/libpriv/rpmostree-unpacker-core.h \ src/libpriv/rpmostree-output.c \ @@ -55,6 +58,8 @@ librpmostreepriv_la_SOURCES = \ src/libpriv/rpmostree-editor.h \ src/libpriv/libsd-locale-util.c \ src/libpriv/libsd-locale-util.h \ + src/libpriv/rpmostree-libarchive-input-stream.c \ + src/libpriv/rpmostree-libarchive-input-stream.h \ $(NULL) librpmostreepriv_la_CFLAGS = \ diff --git a/Makefile-rpm-ostree.am b/Makefile-rpm-ostree.am index 4dc5e183..1e511eb9 100644 --- a/Makefile-rpm-ostree.am +++ b/Makefile-rpm-ostree.am @@ -37,6 +37,8 @@ rpm_ostree_SOURCES = src/app/main.c \ src/app/rpmostree-builtin-status.c \ src/app/rpmostree-builtin-ex.c \ src/app/rpmostree-builtin-container.c \ + src/app/rpmostree-ex-builtin-commit2jigdo.c \ + src/app/rpmostree-ex-builtin-jigdo2commit.c \ src/app/rpmostree-builtin-db.c \ src/app/rpmostree-builtin-start-daemon.c \ src/app/rpmostree-db-builtin-diff.c \ diff --git a/design/jigdo.md b/design/jigdo.md new file mode 100644 index 00000000..b17c991e --- /dev/null +++ b/design/jigdo.md @@ -0,0 +1,97 @@ +Introducing rpm-ostree jigdo +-------- + +In the rpm-ostree project, we're blending an image system (libostree) +with a package system (libdnf). The goal is to gain the +advantages of both. However, the dual nature also brings overhead; +this proposal aims to reduce some of that by adding a new "jigdo" +model to rpm-ostree that makes more operations use the libdnf side. + +To do this, we're reviving an old idea: The [http://atterer.org/jigdo/](Jigdo) +approach to reassembling large "images" by downloading component packages. (We're +not using the code, just the idea). + +In this approach, we're still maintaining the "image" model of libostree. When +one deploys an OSTree commit, it will reliably be bit-for-bit identical. It will +have a checksum and a version number. There will be *no* dependency resolution +on the client by default, etc. + +The change is that we always use libdnf to download RPM packages as they exist +today, storing any additional data inside a new "ostree-image" RPM. In this +proposal, rather than using ostree branches, the system tracks an "ostree-image" +RPM that behaves a bit like a "metapackage". + +Why? +---- + +The "dual" nature of the system appears in many ways; users and administrators +effectively need to understand and manage both systems. + +An example is when one needs to mirror content. While libostree does support +mirroring, and projects like Pulp make use of it, support is not as widespread +as mirroring for RPM. And mirroring is absolutely critical for many +organizations that don't want to depend on Internet availability. + +Related to this is the mapping of libostree "branches" and rpm-md repos. In +Fedora we offer multiple branches for Atomic Host, such as +`fedora/27/x86_64/atomic-host` as well as +`fedora/27/x86_64/testing/atomic-host`, where the latter is equivalent to `yum +--enablerepo=updates-testing update`. In many ways, I believe the way we're +exposing as OSTree branches is actually nicer - it's very clear when you're on +the testing branch. + +However, it's also very *different* from the yum/dnf model. Once package +layering is involved (and for a lot of small scale use cases it will be, +particularly for desktop systems), the libostree side is something that many +users and administrators have to learn *in addition* to their previous "mental model" +of how the libdnf/yum/rpm side works with `/etc/yum.repos.d` etc. + +Finally, for network efficiency; on the wire, libostree has two formats, and the +intention is that most updates hit the network-efficient static delta path, but +there are various cases where this doesn't happen, such as if one is skipping a +few updates, or today when rebasing between branches. In practice, as soon as +one involves libdnf, the repodata is already large enough that it's not worth +trying to optimize fetching content over simply redownloading changed RPMs. + +(Aside: people doing custom systems tend to like the network efficiency of "pure + ostree" where one doesn't pay the "repodata cost" and we will continue to + support that.) + +How? +---- + +We've already stated that a primary design goal is to preserve the "image" +functionality by default. Further, let's assume that we have an OSTree commit, +and we want to point it at a set of RPMs to use as the jigdo source. The source +OSTree commit can have modified, added to, or removed data from the RPM set, and +we will support that. Examples of additional data are the initramfs and RPM +database. + +We're hence treating the RPM set as just data blobs; again, no dependency +resolution, `%post` scripts or the like will be executed on the client. Or again +to state this more strongly, installation will still result in an OSTree commit +with checksum that is bit-for-bit identical. + +A simple approach is to scan over the set of files in the RPMs, then the set +of files in the OSTree commit, and add RPMs which contain files in the OSTree +commit to our "jigdo set". + +However, a major complication is SELinux labeling. It turns out that in a lot of +cases, doing SELinux labeling is expensive; there are thousands of regular +expressions involved. However, RPM packages themselves don't contain labels; +instead the labeling is stored in the `selinux-policy-targeted` package, and +further complicating things is that there are also other packages that add +labeling configuration such as `container-selinux`. In other words there's a +circular dependency: packages have labels, but labels are contained in packages. +We go to great lengths to handle this in rpm-ostree for package layering, and we +need to do the same for jigdo. + +We can address this by having our OIRPM contain a mapping of (package, file +path) to a set of extended attributes (including the key `security.selinux` +one). + +At this point, if we add in the new objects such as the metadata objects from +the OSTree commit and all new content objects that aren't part of a package, +we'll have our OIRPM. (There is +some [further complexity](https://pagure.io/fedora-atomic/issue/94) around +handling the initramfs and SELinux labeling that we'll omit for now). diff --git a/src/app/rpmostree-builtin-ex.c b/src/app/rpmostree-builtin-ex.c index 198bee0b..09f1ba01 100644 --- a/src/app/rpmostree-builtin-ex.c +++ b/src/app/rpmostree-builtin-ex.c @@ -30,6 +30,10 @@ static RpmOstreeCommand ex_subcommands[] = { "Assemble local unprivileged containers", rpmostree_builtin_container }, { "kargs", 0, "Query or Modify the kernel arguments", rpmostree_ex_builtin_kargs }, + { "commit2jigdo", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD, + "Convert an OSTree commit into an rpm-ostree jigdo", rpmostree_ex_builtin_commit2jigdo }, + { "jigdo2commit", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD, + "Convert an rpm-ostree jigdo into an OSTree commit", rpmostree_ex_builtin_jigdo2commit }, { NULL, 0, NULL, NULL } }; diff --git a/src/app/rpmostree-compose-builtins.h b/src/app/rpmostree-compose-builtins.h index 3a7cc0aa..d546b7ee 100644 --- a/src/app/rpmostree-compose-builtins.h +++ b/src/app/rpmostree-compose-builtins.h @@ -30,6 +30,8 @@ gboolean rpmostree_compose_builtin_tree (int argc, char **argv, RpmOstreeCommand gboolean rpmostree_compose_builtin_install (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error); gboolean rpmostree_compose_builtin_postprocess (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error); gboolean rpmostree_compose_builtin_commit (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error); +gboolean rpmostree_compose_builtin_commit2jigdo (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error); +gboolean rpmostree_compose_builtin_jigdo2commit (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error); G_END_DECLS diff --git a/src/app/rpmostree-ex-builtin-commit2jigdo.c b/src/app/rpmostree-ex-builtin-commit2jigdo.c new file mode 100644 index 00000000..352fad45 --- /dev/null +++ b/src/app/rpmostree-ex-builtin-commit2jigdo.c @@ -0,0 +1,1228 @@ +/* -*- mode: C; c-file-style: "gnu"; indent-tabs-mode: nil; -*- + * + * Copyright (C) 2017 Colin Walters + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published + * by the Free Software Foundation; either version 2 of the licence or (at + * your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General + * Public License along with this library; if not, write to the + * Free Software Foundation, Inc., 59 Temple Place, Suite 330, + * Boston, MA 02111-1307, USA. + */ + +#include "config.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "rpmostree-ex-builtins.h" +#include "rpmostree-util.h" +#include "rpmostree-core.h" +#include "rpmostree-jigdo-core.h" +#include "rpmostree-postprocess.h" +#include "rpmostree-passwd-util.h" +#include "rpmostree-libbuiltin.h" +#include "rpmostree-rpm-util.h" + +#include "libglnx.h" + +static char *opt_repo; +static char *opt_pkgcache_repo; +static gboolean opt_only_contentdir; + +static GOptionEntry commit2jigdo_option_entries[] = { + { "repo", 0, 0, G_OPTION_ARG_STRING, &opt_repo, "OSTree repo", "REPO" }, + { "pkgcache-repo", 0, 0, G_OPTION_ARG_STRING, &opt_pkgcache_repo, "Pkgcache OSTree repo", "REPO" }, + { "only-contentdir", 0, 0, G_OPTION_ARG_NONE, &opt_only_contentdir, "Do not generate RPM, only output content directory", NULL }, + { NULL } +}; + +/* Pair of package, and Set, which is either a basename, or a full + * path for non-unique basenames. + */ +typedef struct { + DnfPackage *pkg; + GHashTable *objids; /* Set */ +} PkgObjid; + +static void +pkg_objid_free (PkgObjid *pkgobjid) +{ + g_object_unref (pkgobjid->pkg); + g_hash_table_unref (pkgobjid->objids); + g_free (pkgobjid); +} + +typedef struct { + OstreeRepo *repo; + OstreeRepo *pkgcache_repo; + + guint n_nonunique_objid_basenames; + guint n_objid_basenames; + guint duplicate_big_pkgobjects; + GHashTable *commit_content_objects; /* Set */ + GHashTable *content_object_to_pkg_objid; /* Map */ + guint n_duplicate_pkg_content_objs; + guint n_unused_pkg_content_objs; + GHashTable *objsize_to_object; /* Map */ +} RpmOstreeCommit2JigdoContext; + +static void +rpm_ostree_commit2jigdo_context_free (RpmOstreeCommit2JigdoContext *ctx) +{ + g_clear_object (&ctx->repo); + g_clear_object (&ctx->pkgcache_repo); + g_clear_pointer (&ctx->commit_content_objects, (GDestroyNotify)g_hash_table_unref); + g_clear_pointer (&ctx->content_object_to_pkg_objid, (GDestroyNotify)g_hash_table_unref); + g_clear_pointer (&ctx->objsize_to_object, (GDestroyNotify)g_hash_table_unref); + g_free (ctx); +} +G_DEFINE_AUTOPTR_CLEANUP_FUNC(RpmOstreeCommit2JigdoContext, rpm_ostree_commit2jigdo_context_free) + +/* Initialize a commit2jigdo context */ +static gboolean +rpm_ostree_commit2jigdo_context_new (RpmOstreeCommit2JigdoContext **out_context, + GCancellable *cancellable, + GError **error) +{ + g_autoptr(RpmOstreeCommit2JigdoContext) self = g_new0 (RpmOstreeCommit2JigdoContext, 1); + + self->repo = ostree_repo_open_at (AT_FDCWD, opt_repo, cancellable, error); + if (!self->repo) + return FALSE; + + self->pkgcache_repo = ostree_repo_open_at (AT_FDCWD, opt_pkgcache_repo, cancellable, error); + if (!self->pkgcache_repo) + return FALSE; + + self->commit_content_objects = g_hash_table_new_full (g_str_hash, g_str_equal, (GDestroyNotify)g_free, NULL); + self->content_object_to_pkg_objid = g_hash_table_new_full (g_str_hash, g_str_equal, + (GDestroyNotify)g_free,(GDestroyNotify)pkg_objid_free); + self->objsize_to_object = g_hash_table_new_full (NULL, NULL, NULL, (GDestroyNotify)g_free); + + *out_context = g_steal_pointer (&self); + return TRUE; +} + +/* Add @objid to the set of objectids for @checksum */ +static void +add_objid (GHashTable *object_to_objid, const char *checksum, const char *objid) +{ + GHashTable *objids = g_hash_table_lookup (object_to_objid, checksum); + if (!objids) + { + objids = g_hash_table_new_full (g_str_hash, g_str_equal, g_free, NULL); + g_hash_table_insert (object_to_objid, g_strdup (checksum), objids); + } + g_hash_table_add (objids, g_strdup (objid)); +} + +/* One the main tricky things we need to handle when building the objidmap is + * that we want to compress the xattr map some by using basenames if possible. + * Otherwise we use the full path. + */ +typedef struct { + DnfPackage *package; + GHashTable *seen_nonunique_objid; /* Set */ + GHashTable *seen_objid_to_path; /* Map */ + GHashTable *seen_path_to_object; /* Map */ + char *tmpfiles_d_path; /* Path to tmpfiles.d, which we skip */ +} PkgBuildObjidMap; + +static void +pkg_build_objidmap_free (PkgBuildObjidMap *map) +{ + g_clear_pointer (&map->seen_nonunique_objid, (GDestroyNotify)g_hash_table_unref); + g_clear_pointer (&map->seen_objid_to_path, (GDestroyNotify)g_hash_table_unref); + g_clear_pointer (&map->seen_path_to_object, (GDestroyNotify)g_hash_table_unref); + g_free (map->tmpfiles_d_path); + g_free (map); +} +G_DEFINE_AUTOPTR_CLEANUP_FUNC(PkgBuildObjidMap, pkg_build_objidmap_free) + +/* Recursively walk @dir, building a map of object to Set */ +static gboolean +build_objid_map_for_tree (RpmOstreeCommit2JigdoContext *self, + PkgBuildObjidMap *build, + GHashTable *object_to_objid, + GFile *dir, + GCancellable *cancellable, + GError **error) +{ + g_autoptr(GFileEnumerator) direnum = + g_file_enumerate_children (dir, "standard::name,standard::type", + G_FILE_QUERY_INFO_NOFOLLOW_SYMLINKS, + cancellable, error); + if (!direnum) + return FALSE; + while (TRUE) + { + GFileInfo *info = NULL; + GFile *child = NULL; + if (!g_file_enumerator_iterate (direnum, &info, &child, cancellable, error)) + return FALSE; + if (!info) + break; + OstreeRepoFile *repof = (OstreeRepoFile*)child; + if (!ostree_repo_file_ensure_resolved (repof, NULL)) + g_assert_not_reached (); + GFileType ftype = g_file_info_get_file_type (info); + + /* Handle directories */ + if (ftype == G_FILE_TYPE_DIRECTORY) + { + if (!build_objid_map_for_tree (self, build, object_to_objid, child, + cancellable, error)) + return FALSE; + continue; /* On to the next */ + } + + g_autofree char *path = g_file_get_path (child); + + /* Handling SELinux labeling for the tmpfiles.d would get very tricky. + * Currently the jigdo unpack path is intentionally "dumb" - we won't + * synthesize the tmpfiles.d like we do for layering. So punt these into + * the new object set. + */ + if (g_str_equal (path, build->tmpfiles_d_path)) + continue; + + const char *checksum = ostree_repo_file_get_checksum (repof); + const char *bn = glnx_basename (path); + const gboolean is_known_nonunique = g_hash_table_contains (build->seen_nonunique_objid, bn); + if (is_known_nonunique) + { + add_objid (object_to_objid, checksum, path); + self->n_nonunique_objid_basenames++; + } + else + { + const char *existing_path = g_hash_table_lookup (build->seen_objid_to_path, bn); + if (!existing_path) + { + g_hash_table_insert (build->seen_objid_to_path, g_strdup (bn), g_strdup (path)); + g_hash_table_insert (build->seen_path_to_object, g_strdup (path), g_strdup (checksum)); + add_objid (object_to_objid, checksum, bn); + } + else + { + const char *previous_obj = g_hash_table_lookup (build->seen_path_to_object, existing_path); + g_assert (previous_obj); + /* Replace the previous basename with a full path */ + add_objid (object_to_objid, previous_obj, existing_path); + /* And remove these two hashes which are only needed for transitioning */ + g_hash_table_remove (build->seen_path_to_object, existing_path); + g_hash_table_remove (build->seen_objid_to_path, bn); + /* Add to our nonunique set */ + g_hash_table_add (build->seen_nonunique_objid, g_strdup (bn)); + /* And finally our conflicting entry with a full path */ + add_objid (object_to_objid, checksum, path); + self->n_nonunique_objid_basenames++; + } + } + self->n_objid_basenames++; + } + + return TRUE; +} + +/* For objects bigger than this we'll try to detect identical. + */ +#define BIG_OBJ_SIZE (1024 * 1024) + +/* If someone is shipping > 4GB objects...I don't even know. + * The reason we're doing this is on 32 bit architectures it's + * a pain to put 64 bit numbers in GHashTable. + */ +static gboolean +query_objsize_assert_32bit (OstreeRepo *repo, const char *checksum, + guint32 *out_objsize, + GError **error) +{ + guint64 objsize; + if (!ostree_repo_query_object_storage_size (repo, OSTREE_OBJECT_TYPE_FILE, checksum, + &objsize, NULL, error)) + return FALSE; + if (objsize > G_MAXUINT32) + return glnx_throw (error, "Content object '%s' is %" G_GUINT64_FORMAT " bytes, not supported", + checksum, objsize); + *out_objsize = (guint32) objsize; + return TRUE; +} + +static char * +contentonly_hash_for_object (OstreeRepo *repo, + const char *checksum, + GCancellable *cancellable, + GError **error) +{ + g_autoptr(GInputStream) istream = NULL; + g_autoptr(GFileInfo) finfo = NULL; + if (!ostree_repo_load_file (repo, checksum, &istream, &finfo, NULL, + cancellable, error)) + return FALSE; + + const guint64 size = g_file_info_get_size (finfo); + + g_autoptr(GChecksum) hasher = g_checksum_new (G_CHECKSUM_SHA256); + /* See also https://gist.github.com/cgwalters/0df0d15199009664549618c2188581f0 + * and https://github.com/coreutils/coreutils/blob/master/src/ioblksize.h + * Turns out bigger block size is better; down the line we should use their + * same heuristics. + */ + if (size > 0) + { + gsize bufsize = MIN (size, 128 * 1024); + g_autofree char *buf = g_malloc (bufsize); + gsize bytes_read; + do + { + if (!g_input_stream_read_all (istream, buf, bufsize, &bytes_read, cancellable, error)) + return FALSE; + g_checksum_update (hasher, (const guint8*)buf, bytes_read); + } + while (bytes_read > 0); + } + + return g_strdup (g_checksum_get_string (hasher)); +} + +/* Write a single complete new object (in uncompressed object stream form) + * to the new/ subdir of @tmp_dfd. + */ +static gboolean +write_one_new_object (OstreeRepo *repo, + int tmp_dfd, + OstreeObjectType objtype, + const char *checksum, + GCancellable *cancellable, + GError **error) +{ + GLNX_AUTO_PREFIX_ERROR ("Processing new reachable", error); + g_autoptr(GInputStream) istream = NULL; + guint64 size; + if (!ostree_repo_load_object_stream (repo, objtype, checksum, + &istream, &size, cancellable, error)) + return FALSE; + + g_assert (checksum[0] && checksum[1]); + const char *subdir; + switch (objtype) + { + case OSTREE_OBJECT_TYPE_DIR_META: + subdir = RPMOSTREE_JIGDO_DIRMETA_DIR; + break; + case OSTREE_OBJECT_TYPE_DIR_TREE: + subdir = RPMOSTREE_JIGDO_DIRTREE_DIR; + break; + case OSTREE_OBJECT_TYPE_FILE: + subdir = RPMOSTREE_JIGDO_NEW_DIR; + break; + default: + g_assert_not_reached (); + } + g_autofree char *prefix = g_strdup_printf ("%s/%c%c", subdir, checksum[0], checksum[1]); + + if (!glnx_shutil_mkdir_p_at (tmp_dfd, prefix, 0755, cancellable, error)) + return FALSE; + + g_autofree char *new_obj_path = g_strconcat (prefix, "/", (checksum+2), NULL); + g_auto(GLnxTmpfile) tmpf = { 0, }; + if (!glnx_open_tmpfile_linkable_at (tmp_dfd, ".", O_CLOEXEC | O_WRONLY, + &tmpf, error)) + return FALSE; + g_autoptr(GOutputStream) ostream = g_unix_output_stream_new (tmpf.fd, FALSE); + + if (g_output_stream_splice (ostream, istream, G_OUTPUT_STREAM_SPLICE_CLOSE_SOURCE | G_OUTPUT_STREAM_SPLICE_CLOSE_TARGET, + cancellable, error) < 0) + return FALSE; + if (!glnx_link_tmpfile_at (&tmpf, GLNX_LINK_TMPFILE_NOREPLACE, tmp_dfd, + new_obj_path, error)) + return FALSE; + + return TRUE; +} + +/* Write a set of content-identical objects, with the identical content + * only written once. These go in the new-contentident/ subdirectory. + */ +static gboolean +write_content_identical_set (OstreeRepo *repo, + int tmp_dfd, + guint content_ident_idx, + GPtrArray *identicals, + GCancellable *cancellable, + GError **error) +{ + g_assert_cmpint (identicals->len, >, 1); + GLNX_AUTO_PREFIX_ERROR ("Processing big content-identical", error); + g_autofree char *subdir = g_strdup_printf ("%s/%u", RPMOSTREE_JIGDO_NEW_CONTENTIDENT_DIR, content_ident_idx); + if (!glnx_shutil_mkdir_p_at (tmp_dfd, subdir, 0755, cancellable, error)) + return FALSE; + + /* Write metadata for all of the objects as a single variant */ + g_autoptr(GVariantBuilder) builder = g_variant_builder_new (RPMOSTREE_JIGDO_NEW_CONTENTIDENT_VARIANT_FORMAT); + for (guint i = 0; i < identicals->len; i++) + { + const char *checksum = identicals->pdata[i]; + g_autoptr(GFileInfo) finfo = NULL; + g_autoptr(GVariant) xattrs = NULL; + if (!ostree_repo_load_file (repo, checksum, NULL, &finfo, &xattrs, + cancellable, error)) + return FALSE; + g_variant_builder_add (builder, "(suuu@a(ayay))", + checksum, + GUINT32_TO_BE (g_file_info_get_attribute_uint32 (finfo, "unix::uid")), + GUINT32_TO_BE (g_file_info_get_attribute_uint32 (finfo, "unix::gid")), + GUINT32_TO_BE (g_file_info_get_attribute_uint32 (finfo, "unix::mode")), + xattrs); + } + g_autoptr(GVariant) meta = g_variant_ref_sink (g_variant_builder_end (builder)); + g_autofree char *meta_path = g_strconcat (subdir, "/01meta", NULL); + if (!glnx_file_replace_contents_at (tmp_dfd, meta_path, + g_variant_get_data (meta), + g_variant_get_size (meta), + GLNX_FILE_REPLACE_NODATASYNC, + cancellable, error)) + return FALSE; + + /* Write the content */ + const char *checksum = identicals->pdata[0]; + g_autoptr(GInputStream) istream = NULL; + if (!ostree_repo_load_file (repo, checksum, &istream, + NULL, NULL, cancellable, error)) + return FALSE; + g_autofree char *content_path = g_strconcat (subdir, "/05content", NULL); + g_auto(GLnxTmpfile) tmpf = { 0, }; + if (!glnx_open_tmpfile_linkable_at (tmp_dfd, ".", O_CLOEXEC | O_WRONLY, + &tmpf, error)) + return FALSE; + g_autoptr(GOutputStream) ostream = g_unix_output_stream_new (tmpf.fd, FALSE); + if (g_output_stream_splice (ostream, istream, G_OUTPUT_STREAM_SPLICE_CLOSE_SOURCE | G_OUTPUT_STREAM_SPLICE_CLOSE_TARGET, + cancellable, error) < 0) + return FALSE; + if (!glnx_link_tmpfile_at (&tmpf, GLNX_LINK_TMPFILE_NOREPLACE, tmp_dfd, + content_path, error)) + return FALSE; + + return TRUE; +} + +/* Taken from ostree-repo-static-delta-compilation.c */ +static guint +bufhash (const void *b, gsize len) +{ + const signed char *p, *e; + guint32 h = 5381; + + for (p = (signed char *)b, e = (signed char *)b + len; p != e; p++) + h = (h << 5) + h + *p; + + return h; +} + +/* Taken from ostree-repo-static-delta-compilation.c */ +static guint +xattr_chunk_hash (const void *vp) +{ + GVariant *v = (GVariant*)vp; + gsize n = g_variant_n_children (v); + guint i; + guint32 h = 5381; + + for (i = 0; i < n; i++) + { + const guint8* name; + const guint8* value_data; + g_autoptr(GVariant) value = NULL; + gsize value_len; + + g_variant_get_child (v, i, "(^&ay@ay)", + &name, &value); + value_data = g_variant_get_fixed_array (value, &value_len, 1); + + h += g_str_hash (name); + h += bufhash (value_data, value_len); + } + + return h; +} + +/* Taken from ostree-repo-static-delta-compilation.c */ +static gboolean +xattr_chunk_equals (const void *one, const void *two) +{ + GVariant *v1 = (GVariant*)one; + GVariant *v2 = (GVariant*)two; + gsize l1 = g_variant_get_size (v1); + gsize l2 = g_variant_get_size (v2); + + if (l1 != l2) + return FALSE; + + if (l1 == 0) + return l2 == 0; + + return memcmp (g_variant_get_data (v1), g_variant_get_data (v2), l1) == 0; +} + +static int +cmp_objidxattrs (gconstpointer ap, + gconstpointer bp) +{ + GVariant *a = *((GVariant**)ap); + GVariant *b = *((GVariant**)bp); + const char *a_objid; + g_variant_get_child (a, 0, "&s", &a_objid); + const char *b_objid; + g_variant_get_child (b, 0, "&s", &b_objid); + return strcmp (a_objid, b_objid); +} + +/* Walk @pkg, building up a map of content object hash to "objid". */ +static gboolean +build_objid_map_for_package (RpmOstreeCommit2JigdoContext *self, + DnfPackage *pkg, + GCancellable *cancellable, + GError **error) +{ + g_autofree char *cachebranch = rpmostree_get_cache_branch_pkg (pkg); + g_autofree char *pkg_commit = NULL; + g_autoptr(GFile) commit_root = NULL; + + if (!ostree_repo_read_commit (self->pkgcache_repo, cachebranch, &commit_root, &pkg_commit, + cancellable, error)) + return FALSE; + + /* Maps a content object checksum to a set of "objid", which is either + * a basename (if unique) or a full path. + */ + g_autoptr(GHashTable) object_to_objid = g_hash_table_new_full (g_str_hash, g_str_equal, + g_free, (GDestroyNotify)g_hash_table_unref); + /* Allocate temporary build state (mostly hash tables) just for this call */ + { g_autoptr(PkgBuildObjidMap) build = g_new0 (PkgBuildObjidMap, 1); + build->package = pkg; + build->seen_nonunique_objid = g_hash_table_new_full (g_str_hash, g_str_equal, + g_free, NULL); + build->seen_objid_to_path = g_hash_table_new_full (g_str_hash, g_str_equal, + g_free, g_free); + build->seen_path_to_object = g_hash_table_new_full (g_str_hash, g_str_equal, + g_free, g_free); + build->tmpfiles_d_path = g_strconcat ("/usr/lib/tmpfiles.d/pkg-", + dnf_package_get_name (pkg), ".conf", NULL); + if (!build_objid_map_for_tree (self, build, object_to_objid, + commit_root, cancellable, error)) + return FALSE; + } + + /* Loop over the objects we found in this package */ + GLNX_HASH_TABLE_FOREACH_IT (object_to_objid, it, char *, checksum, + GHashTable *, objid_set) + { + /* See if this is a "big" object. If so, we add a mapping from + * size → checksum, so we can heuristically later try to find + * "content-identical objects" i.e. they differ only in metadata. + */ + guint32 objsize; + if (!query_objsize_assert_32bit (self->repo, checksum, &objsize, error)) + return FALSE; + if (objsize >= BIG_OBJ_SIZE) + { + /* If two big objects that are actually *different* happen + * to have the same size...eh, not too worried about it right + * now. It'll just be a missed optimization. We keep track + * of how many at least to guide future work. + */ + if (g_hash_table_replace (self->objsize_to_object, GUINT_TO_POINTER (objsize), + g_strdup (checksum))) + self->duplicate_big_pkgobjects++; + } + + if (g_hash_table_contains (self->content_object_to_pkg_objid, checksum)) + { + /* We already found an instance of this, just add it to our + * duplicate count as a curiosity. + */ + self->n_duplicate_pkg_content_objs++; + g_hash_table_iter_remove (&it); + } + else if (!g_hash_table_contains (self->commit_content_objects, checksum)) + { + /* This happens a lot for Fedora Atomic Host today where we disable + * documentation. But it will also happen if we modify any files in + * postprocessing. + */ + self->n_unused_pkg_content_objs++; + } + else + { + /* Add object → pkgobjid to the global map */ + g_hash_table_iter_steal (&it); + PkgObjid *pkgobjid = g_new (PkgObjid, 1); + pkgobjid->pkg = g_object_ref (pkg); + pkgobjid->objids = g_steal_pointer (&objid_set); + + g_hash_table_insert (self->content_object_to_pkg_objid, g_steal_pointer (&checksum), pkgobjid); + } + } + + return TRUE; +} + +static int +compare_pkgs (gconstpointer ap, + gconstpointer bp) +{ + DnfPackage **a = (gpointer)ap; + DnfPackage **b = (gpointer)bp; + return dnf_package_cmp (*a, *b); +} + +static gboolean +write_commit2jigdo (RpmOstreeCommit2JigdoContext *self, + const char *commit, + const char *oirpm_spec, + const char *outputdir, + GHashTable *pkgs_with_content, + GHashTable *new_reachable_small, + GHashTable *new_big_content_identical, + GCancellable *cancellable, + GError **error) +{ + g_autoptr(GVariant) commit_obj = NULL; + if (!ostree_repo_load_commit (self->repo, commit, &commit_obj, NULL, error)) + return FALSE; + g_autoptr(GVariant) commit_inline_meta = g_variant_get_child_value (commit_obj, 0); + g_autoptr(GVariant) commit_detached_meta = NULL; + if (!ostree_repo_read_commit_detached_metadata (self->repo, commit, &commit_detached_meta, + cancellable, error)) + return FALSE; + + g_auto(GLnxTmpDir) oirpm_tmpd = { 0, }; + if (!glnx_mkdtemp ("rpmostree-jigdo-XXXXXX", 0700, &oirpm_tmpd, error)) + return FALSE; + + /* The commit object and metadata go first, so that the client can do GPG verification + * early on. + */ + { g_autofree char *commit_dir = g_strdup_printf ("%s/%c%c", RPMOSTREE_JIGDO_COMMIT_DIR, + commit[0], commit[1]); + if (!glnx_shutil_mkdir_p_at (oirpm_tmpd.fd, commit_dir, 0755, cancellable, error)) + return FALSE; + g_autofree char *commit_path = g_strconcat (commit_dir, "/", commit+2, NULL); + if (!glnx_file_replace_contents_at (oirpm_tmpd.fd, commit_path, + g_variant_get_data (commit_obj), + g_variant_get_size (commit_obj), + GLNX_FILE_REPLACE_NODATASYNC, + cancellable, error)) + return FALSE; + + } + { const guint8 *buf = (const guint8*)""; + size_t buflen = 0; + g_autofree char *commit_metapath = g_strconcat (RPMOSTREE_JIGDO_COMMIT_DIR, "/meta", NULL); + if (commit_detached_meta) + { + buf = g_variant_get_data (commit_detached_meta); + buflen = g_variant_get_size (commit_detached_meta); + } + if (!glnx_file_replace_contents_at (oirpm_tmpd.fd, commit_metapath, + buf, buflen, GLNX_FILE_REPLACE_NODATASYNC, + cancellable, error)) + return FALSE; + } + + /* write out the variant containing packages */ + { g_autoptr(GPtrArray) jigdo_packages = g_ptr_array_new (); + GLNX_HASH_TABLE_FOREACH (pkgs_with_content, DnfPackage *, pkg) + { + g_ptr_array_add (jigdo_packages, pkg); + } + g_ptr_array_sort (jigdo_packages, compare_pkgs); + g_autoptr(GVariantBuilder) pkgbuilder = g_variant_builder_new (RPMOSTREE_JIGDO_PKGS_VARIANT_FORMAT); + for (guint i = 0; i < jigdo_packages->len; i++) + { + DnfPackage *pkg = jigdo_packages->pdata[i]; + g_autofree char *repodata_checksum = NULL; + if (!rpmostree_get_repodata_chksum_repr (pkg, &repodata_checksum, error)) + return FALSE; + g_variant_builder_add (pkgbuilder, "(stssss)", + dnf_package_get_name (pkg), + dnf_package_get_epoch (pkg), + dnf_package_get_version (pkg), + dnf_package_get_release (pkg), + dnf_package_get_arch (pkg), + repodata_checksum); + } + g_autoptr(GVariant) jigdo_packages_v = g_variant_ref_sink (g_variant_builder_end (pkgbuilder)); + if (!glnx_file_replace_contents_at (oirpm_tmpd.fd, RPMOSTREE_JIGDO_PKGS, + g_variant_get_data (jigdo_packages_v), + g_variant_get_size (jigdo_packages_v), + GLNX_FILE_REPLACE_NODATASYNC, + cancellable, error)) + return FALSE; + } + + /* dirtree/dirmeta */ + if (!glnx_shutil_mkdir_p_at (oirpm_tmpd.fd, RPMOSTREE_JIGDO_DIRMETA_DIR, 0755, cancellable, error)) + return FALSE; + if (!glnx_shutil_mkdir_p_at (oirpm_tmpd.fd, RPMOSTREE_JIGDO_DIRTREE_DIR, 0755, cancellable, error)) + return FALSE; + /* Traverse the commit again, adding dirtree/dirmeta */ + g_autoptr(GHashTable) commit_reachable = NULL; + if (!ostree_repo_traverse_commit (self->repo, commit, 0, + &commit_reachable, + cancellable, error)) + return FALSE; + GLNX_HASH_TABLE_FOREACH_IT (commit_reachable, it, GVariant *, object, + GVariant *, also_object) + { + OstreeObjectType objtype; + const char *checksum; + ostree_object_name_deserialize (object, &checksum, &objtype); + if (G_IN_SET (objtype, OSTREE_OBJECT_TYPE_DIR_TREE, OSTREE_OBJECT_TYPE_DIR_META)) + { + if (!write_one_new_object (self->repo, oirpm_tmpd.fd, objtype, checksum, + cancellable, error)) + continue; + } + g_hash_table_iter_remove (&it); + } + + GLNX_HASH_TABLE_FOREACH_IT (new_reachable_small, it, const char *, checksum, + void *, unused) + { + if (!write_one_new_object (self->repo, oirpm_tmpd.fd, + OSTREE_OBJECT_TYPE_FILE, checksum, + cancellable, error)) + return FALSE; + g_hash_table_iter_remove (&it); + } + + /* Process large objects, which may only have 1 reference, in which case they also + * go under new/, otherwise new-contentident/. + */ + if (!glnx_shutil_mkdir_p_at (oirpm_tmpd.fd, RPMOSTREE_JIGDO_NEW_CONTENTIDENT_DIR, 0755, cancellable, error)) + return FALSE; + guint content_ident_idx = 0; + GLNX_HASH_TABLE_FOREACH_IT (new_big_content_identical, it, const char *, content_checksum, + GPtrArray *, identicals) + { + g_assert_cmpint (identicals->len, >=, 1); + if (identicals->len == 1) + { + const char *checksum = identicals->pdata[0]; + if (!write_one_new_object (self->repo, oirpm_tmpd.fd, + OSTREE_OBJECT_TYPE_FILE, checksum, + cancellable, error)) + return FALSE; + } + else + { + if (!write_content_identical_set (self->repo, oirpm_tmpd.fd, content_ident_idx, + identicals, cancellable, error)) + return FALSE; + content_ident_idx++; + } + g_hash_table_iter_remove (&it); + } + + GLNX_HASH_TABLE_FOREACH_IT (new_big_content_identical, it, const char *, content_checksum, + GPtrArray *, identicals) + { + g_assert_cmpint (identicals->len, >=, 1); + if (identicals->len == 1) + { + const char *checksum = identicals->pdata[0]; + if (!write_one_new_object (self->repo, oirpm_tmpd.fd, + OSTREE_OBJECT_TYPE_FILE, checksum, + cancellable, error)) + return FALSE; + } + else + { + if (!write_content_identical_set (self->repo, oirpm_tmpd.fd, content_ident_idx, + identicals, cancellable, error)) + return FALSE; + content_ident_idx++; + } + g_hash_table_iter_remove (&it); + } + + /* And finally, the xattr data (usually just SELinux labels, the file caps + * here but *also* in the RPM header; we could optimize that, but it's not + * really worth it) + */ + { g_autoptr(GHashTable) xattr_table_hash = g_hash_table_new_full (xattr_chunk_hash, xattr_chunk_equals, + (GDestroyNotify)g_variant_unref, NULL); + if (!glnx_shutil_mkdir_p_at (oirpm_tmpd.fd, RPMOSTREE_JIGDO_XATTRS_DIR, 0755, cancellable, error)) + return FALSE; + + g_autoptr(GHashTable) pkg_to_objidxattrs = g_hash_table_new_full (NULL, NULL, g_object_unref, + (GDestroyNotify) g_ptr_array_unref); + + /* First, gather the unique set of xattrs from all pkgobjs */ + g_autoptr(GVariantBuilder) xattr_table_builder = + g_variant_builder_new (RPMOSTREE_JIGDO_XATTRS_TABLE_VARIANT_FORMAT); + guint global_xattr_idx = 0; + GLNX_HASH_TABLE_FOREACH_IT (self->commit_content_objects, it, const char *, checksum, + const char *, unused) + { + /* Is this content object associated with a package? If not, it was + * already processed. + */ + PkgObjid *pkgobjid = g_hash_table_lookup (self->content_object_to_pkg_objid, checksum); + if (!pkgobjid) + { + g_hash_table_iter_remove (&it); + continue; + } + + g_autoptr(GVariant) xattrs = NULL; + if (!ostree_repo_load_file (self->repo, checksum, NULL, NULL, + &xattrs, cancellable, error)) + return FALSE; + + /* No xattrs? We're done */ + if (g_variant_n_children (xattrs) == 0) + { + g_hash_table_iter_remove (&it); + continue; + } + + /* Keep track of the unique xattr set */ + void *xattr_idx_p; + guint this_xattr_idx; + if (!g_hash_table_lookup_extended (xattr_table_hash, xattrs, NULL, &xattr_idx_p)) + { + g_variant_builder_add (xattr_table_builder, "@a(ayay)", xattrs); + g_hash_table_insert (xattr_table_hash, g_steal_pointer (&xattrs), + GUINT_TO_POINTER (global_xattr_idx)); + this_xattr_idx = global_xattr_idx; + /* Increment this for the next loop */ + global_xattr_idx++; + } + else + { + this_xattr_idx = GPOINTER_TO_UINT (xattr_idx_p); + } + + /* Add this to our map of pkg → [objidxattrs] */ + DnfPackage *pkg = pkgobjid->pkg; + GPtrArray *pkg_objidxattrs = g_hash_table_lookup (pkg_to_objidxattrs, pkg); + if (!pkg_objidxattrs) + { + pkg_objidxattrs = g_ptr_array_new_with_free_func ((GDestroyNotify)g_variant_unref); + g_hash_table_insert (pkg_to_objidxattrs, g_object_ref (pkg), pkg_objidxattrs); + } + + GLNX_HASH_TABLE_FOREACH (pkgobjid->objids, const char *, objid) + { + g_ptr_array_add (pkg_objidxattrs, + g_variant_ref_sink (g_variant_new ("(su)", objid, this_xattr_idx))); + } + + /* We're done with this object data */ + g_hash_table_iter_remove (&it); + g_hash_table_remove (self->content_object_to_pkg_objid, checksum); + + } + + g_print ("%u unique xattrs\n", g_hash_table_size (xattr_table_hash)); + + /* Write the xattr string table */ + if (!glnx_shutil_mkdir_p_at (oirpm_tmpd.fd, RPMOSTREE_JIGDO_XATTRS_DIR, 0755, cancellable, error)) + return FALSE; + { g_autoptr(GVariant) xattr_table = g_variant_ref_sink (g_variant_builder_end (xattr_table_builder)); + if (!glnx_file_replace_contents_at (oirpm_tmpd.fd, RPMOSTREE_JIGDO_XATTRS_TABLE, + g_variant_get_data (xattr_table), + g_variant_get_size (xattr_table), + GLNX_FILE_REPLACE_NODATASYNC, + cancellable, error)) + return FALSE; + } + + /* Subdirectory for packages */ + if (!glnx_shutil_mkdir_p_at (oirpm_tmpd.fd, RPMOSTREE_JIGDO_XATTRS_PKG_DIR, 0755, cancellable, error)) + return FALSE; + + /* We're done with these maps */ + g_clear_pointer (&self->commit_content_objects, (GDestroyNotify)g_ptr_array_unref); + g_clear_pointer (&self->content_object_to_pkg_objid, (GDestroyNotify)g_ptr_array_unref); + + /* Now that we have a mapping for each package, sort + * the package xattr data by objid, and write it to + * xattrs/${nevra} + */ + GLNX_HASH_TABLE_FOREACH_IT (pkg_to_objidxattrs, it, DnfPackage*, pkg, GPtrArray *,objidxattrs) + { + GLNX_AUTO_PREFIX_ERROR ("Writing xattrs", error); + const char *nevra = dnf_package_get_nevra (pkg); + + /* Ensure the objid array is sorted so we can bsearch it */ + g_ptr_array_sort (objidxattrs, cmp_objidxattrs); + + /* Build up the variant from sorted data */ + g_autoptr(GVariantBuilder) objid_xattr_builder = g_variant_builder_new ((GVariantType*)"a(su)"); + for (guint i = 0; i < objidxattrs->len; i++) + { + GVariant *objidxattr = objidxattrs->pdata[i]; + g_variant_builder_add (objid_xattr_builder, "@(su)", objidxattr); + } + g_autoptr(GVariant) objid_xattrs_final = g_variant_ref_sink (g_variant_builder_end (objid_xattr_builder)); + + g_autofree char *path = g_strconcat (RPMOSTREE_JIGDO_XATTRS_PKG_DIR, "/", nevra, NULL); + if (!glnx_file_replace_contents_at (oirpm_tmpd.fd, path, g_variant_get_data (objid_xattrs_final), + g_variant_get_size (objid_xattrs_final), + GLNX_FILE_REPLACE_NODATASYNC, + cancellable, error)) + return FALSE; + + g_hash_table_iter_remove (&it); + } + } + + if (!opt_only_contentdir) + { + const char *commit_version; + if (!g_variant_lookup (commit_inline_meta, OSTREE_COMMIT_META_KEY_VERSION, "&s", &commit_version)) + commit_version = NULL; + + g_autoptr(GPtrArray) rpmbuild_argv = g_ptr_array_new_with_free_func (g_free); + g_ptr_array_add (rpmbuild_argv, g_strdup ("rpmbuild")); + g_ptr_array_add (rpmbuild_argv, g_strdup ("-bb")); + /* We use --build-in-place to avoid having to compress the data again into a + * Source only to immediately uncompress it. + */ + g_ptr_array_add (rpmbuild_argv, g_strdup ("--build-in-place")); + // Taken from https://github.com/cgwalters/homegit/blob/master/bin/rpmbuild-cwd + const char *rpmbuild_dir_args[] = { "_sourcedir", "_specdir", "_builddir", + "_srcrpmdir", "_rpmdir", }; + for (guint i = 0; i < G_N_ELEMENTS (rpmbuild_dir_args); i++) + { + g_ptr_array_add (rpmbuild_argv, g_strdup ("-D")); + g_ptr_array_add (rpmbuild_argv, g_strdup_printf ("%s %s", rpmbuild_dir_args[i], outputdir)); + } + g_ptr_array_add (rpmbuild_argv, g_strdup ("-D")); + g_ptr_array_add (rpmbuild_argv, g_strconcat ("_buildrootdir ", outputdir, "/.build", NULL)); + if (commit_version) + { + g_ptr_array_add (rpmbuild_argv, g_strdup ("-D")); + g_ptr_array_add (rpmbuild_argv, g_strconcat ("ostree_version ", commit_version, NULL)); + } + g_ptr_array_add (rpmbuild_argv, g_strdup (oirpm_spec)); + g_ptr_array_add (rpmbuild_argv, NULL); + int estatus; + GLNX_AUTO_PREFIX_ERROR ("Running rpmbuild", error); + if (!g_spawn_sync (oirpm_tmpd.path, (char**)rpmbuild_argv->pdata, NULL, G_SPAWN_SEARCH_PATH, + NULL, NULL, NULL, NULL, &estatus, error)) + return FALSE; + if (!g_spawn_check_exit_status (estatus, error)) + return FALSE; + } + else + { + g_print ("Wrote: %s\n", oirpm_tmpd.path); + glnx_tmpdir_unset (&oirpm_tmpd); + } + + + return TRUE; +} + +/* Entrypoint function for turning a commit into an OIRPM. + * + * The basic prerequisite for this: when doing a compose tree, import the + * packages, and after import check out the final tree and SELinux relabel the + * imports so that they're reliably updated (currently depends on some unified + * core 🌐 work). + * + * First, we find the "jigdo set" of packages we need; not all packages that + * live in the tree actually need to be imported; things like `emacs-filesystem` + * or `rootfiles` today don't actually generate any content objects we use. + * + * The biggest "extra data" we need is the SELinux labels for the files in + * each package. To simplify things, we generalize this to "all xattrs". + * + * Besides that, we need the metadata objects like the OSTree commit and the + * referenced dirtree/dirmeta objects. Plus the added content objects like the + * rpmdb, initramfs, etc. + * + * One special optimization made is support for detecting "content-identical" + * added content objects, because right now we have the initramfs 3 times in the + * tree (due to SELinux labels). While we have 3 copies on disk, we can easily + * avoid that on the wire. + * + * Once we've determined all the needed data, we make a temporary directory, and + * start writing out files inside it. This temporary directory is then turned + * into the OIRPM (what looks like a plain old RPM) by invoking `rpmbuild` using + * a `.spec` file. + * + * The resulting "jigdo set" is then that OIRPM, plus the exact NEVRAs - we also + * record the repodata checksum (normally sha256), to ensure that we get the + * *exact* RPMs we require bit-for-bit. + */ +static gboolean +impl_commit2jigdo (RpmOstreeCommit2JigdoContext *self, + const char *rev, + const char *oirpm_spec, + const char *outputdir, + GCancellable *cancellable, + GError **error) +{ + g_assert_cmpint (*outputdir, ==, '/'); + g_autofree char *commit = NULL; + g_autoptr(GFile) root = NULL; + if (!ostree_repo_read_commit (self->repo, rev, &root, &commit, cancellable, error)) + return FALSE; + + g_print ("Finding reachable objects from target %s...\n", commit); + g_autoptr(GHashTable) commit_reachable = NULL; + if (!ostree_repo_traverse_commit (self->repo, commit, 0, + &commit_reachable, + cancellable, error)) + return FALSE; + GLNX_HASH_TABLE_FOREACH_IT (commit_reachable, it, GVariant *, object, + GVariant *, also_object) + { + OstreeObjectType objtype; + const char *checksum; + ostree_object_name_deserialize (object, &checksum, &objtype); + if (objtype == OSTREE_OBJECT_TYPE_FILE) + g_hash_table_add (self->commit_content_objects, g_strdup (checksum)); + g_hash_table_iter_remove (&it); + } + g_print ("%u content objects\n", g_hash_table_size (self->commit_content_objects)); + + g_print ("Finding reachable objects from packages...\n"); + g_autoptr(RpmOstreeRefSack) sack = + rpmostree_get_refsack_for_commit (self->repo, commit, cancellable, error); + if (!sack) + return FALSE; + + hy_autoquery HyQuery hquery = hy_query_create (sack->sack); + hy_query_filter (hquery, HY_PKG_REPONAME, HY_EQ, HY_SYSTEM_REPO_NAME); + g_autoptr(GPtrArray) pkglist = hy_query_run (hquery); + g_print ("Building object map from %u packages\n", pkglist->len); + + g_assert_cmpint (pkglist->len, >, 0); + + for (guint i = 0; i < pkglist->len; i++) + { + DnfPackage *pkg = pkglist->pdata[i]; + if (!build_objid_map_for_package (self, pkg, cancellable, error)) + return FALSE; + } + + g_print ("%u content objects in packages\n", g_hash_table_size (self->content_object_to_pkg_objid)); + g_print (" %u duplicate, %u unused\n", + self->n_duplicate_pkg_content_objs, self->n_unused_pkg_content_objs); + g_print (" %u big sizematches, %u/%u nonunique basenames\n", + self->duplicate_big_pkgobjects, self->n_nonunique_objid_basenames, self->n_objid_basenames); + /* These sets track objects which aren't in the packages */ + g_autoptr(GHashTable) new_reachable_big = g_hash_table_new_full (g_str_hash, g_str_equal, g_free, NULL); + g_autoptr(GHashTable) new_reachable_small = g_hash_table_new_full (g_str_hash, g_str_equal, g_free, NULL); + g_autoptr(GHashTable) pkgs_with_content = g_hash_table_new (NULL, NULL); + guint64 pkg_bytes = 0; + guint64 oirpm_bytes_small = 0; + + /* Loop over every content object in the final commit, and see whether we + * found a package that contains that exact object. We classify new + * objects as either "big" or "small" - for "big" objects we'll try + * to heuristically find a content-identical one. + */ + GLNX_HASH_TABLE_FOREACH (self->commit_content_objects, const char *, checksum) + { + guint32 objsize; + if (!query_objsize_assert_32bit (self->repo, checksum, &objsize, error)) + return FALSE; + const gboolean is_big = objsize >= BIG_OBJ_SIZE; + + PkgObjid *pkgobjid = g_hash_table_lookup (self->content_object_to_pkg_objid, checksum); + if (!pkgobjid) + g_hash_table_add (is_big ? new_reachable_big : new_reachable_small, g_strdup (checksum)); + else + g_hash_table_add (pkgs_with_content, pkgobjid->pkg); + + if (pkgobjid) + pkg_bytes += objsize; + else if (!is_big) + oirpm_bytes_small += objsize; + /* We'll account for new big objects later after more analysis */ + } + + g_print ("Found objects in %u/%u packages; new (unpackaged) objects: %u small + %u large\n", + g_hash_table_size (pkgs_with_content), + pkglist->len, + g_hash_table_size (new_reachable_small), + g_hash_table_size (new_reachable_big)); + if (g_hash_table_size (pkgs_with_content) != pkglist->len) + { + g_print ("Packages without content:\n"); + for (guint i = 0; i < pkglist->len; i++) + { + DnfPackage *pkg = pkglist->pdata[i]; + if (!g_hash_table_contains (pkgs_with_content, pkg)) + { + g_autofree char *tmpfiles_d_path = g_strconcat ("usr/lib/tmpfiles.d/pkg-", + dnf_package_get_name (pkg), + ".conf", NULL); + g_autoptr(GFile) tmpfiles_d_f = g_file_resolve_relative_path (root, tmpfiles_d_path); + const gboolean is_tmpfiles_only = g_file_query_exists (tmpfiles_d_f, cancellable); + /* I added this while debugging missing tmpfiles.d/pkg-$x.conf + * objects; it turns out not to trigger currently, but keeping it + * anyways. + */ + if (is_tmpfiles_only) + { + g_hash_table_add (pkgs_with_content, pkg); + g_print (" %s (tmpfiles only)\n", dnf_package_get_nevra (pkg)); + } + else + g_print (" %s\n", dnf_package_get_nevra (pkg)); + } + } + g_print ("\n"); + } + +#if 0 + /* Maps a new big object hash to an object from a package */ + g_autoptr(GHashTable) new_big_pkgidentical = g_hash_table_new_full (g_str_hash, g_str_equal, g_free, g_free); +#endif + g_print ("Examining large objects more closely for content-identical versions...\n"); + /* Maps a new big object hash to a set of duplicates; yes this happens + * unfortunately for the initramfs right now due to SELinux labeling. + */ + g_autoptr(GHashTable) new_big_content_identical = g_hash_table_new_full (g_str_hash, g_str_equal, + g_free, (GDestroyNotify)g_ptr_array_unref); + + guint64 oirpm_bytes_big = 0; + GLNX_HASH_TABLE_FOREACH_IT (new_reachable_big, it, const char *, checksum, + void *, unused) + { + guint32 objsize; + if (!query_objsize_assert_32bit (self->repo, checksum, &objsize, error)) + return FALSE; + g_assert_cmpint (objsize, >=, BIG_OBJ_SIZE); + + g_autofree char *obj_contenthash = + contentonly_hash_for_object (self->repo, checksum, cancellable, error); + if (!obj_contenthash) + return FALSE; + g_autofree char *objsize_formatted = g_format_size (objsize); + + /* This is complex to implement; it would be useful for the grub2-efi data + * but in the end we should avoid having content-identical objects in the + * OS data anyways. + */ +#if 0 + const char *probable_source = g_hash_table_lookup (self->objsize_to_object, GUINT_TO_POINTER (objsize)); + + if (probable_source) + { + g_autofree char *probable_source_contenthash = + contentonly_hash_for_object (self->pkgcache_repo, probable_source, cancellable, error); + if (!probable_source_contenthash) + return FALSE; + if (g_str_equal (obj_contenthash, probable_source_contenthash)) + { + g_print ("%s %s (pkg content hash hit)\n", checksum, objsize_formatted); + g_hash_table_replace (new_big_pkgidentical, g_strdup (checksum), g_strdup (probable_source)); + g_hash_table_iter_remove (&it); + + pkg_bytes += objsize; + /* We found a package content hash hit, loop to next */ + continue; + } + } +#endif + + /* OK, see if it duplicates another *new* object */ + GPtrArray *identicals = g_hash_table_lookup (new_big_content_identical, obj_contenthash); + if (!identicals) + { + identicals = g_ptr_array_new_with_free_func (g_free); + g_hash_table_insert (new_big_content_identical, g_strdup (obj_contenthash), identicals); + g_print ("%s %s (new, objhash %s)\n", checksum, objsize_formatted, obj_contenthash); + oirpm_bytes_big += objsize; + } + else + { + g_print ("%s (content identical with %u objects)\n", checksum, identicals->len); + } + g_ptr_array_add (identicals, g_strdup (checksum)); + } + + { g_autofree char *pkg_bytes_formatted = g_format_size (pkg_bytes); + g_autofree char *oirpm_bytes_formatted_small = g_format_size (oirpm_bytes_small); + g_autofree char *oirpm_bytes_formatted_big = g_format_size (oirpm_bytes_big); + g_print ("pkg content size: %s\n", pkg_bytes_formatted); + g_print ("oirpm content size (small objs): %s\n", oirpm_bytes_formatted_small); + g_print ("oirpm content size (big objs): %s\n", oirpm_bytes_formatted_big); + } + + if (!write_commit2jigdo (self, commit, oirpm_spec, outputdir, pkgs_with_content, + new_reachable_small, new_big_content_identical, + cancellable, error)) + return FALSE; + + return TRUE; +} + +int +rpmostree_ex_builtin_commit2jigdo (int argc, + char **argv, + RpmOstreeCommandInvocation *invocation, + GCancellable *cancellable, + GError **error) +{ + g_autoptr(GOptionContext) context = g_option_context_new ("REV OIRPM-SPEC OUTPUTDIR"); + if (!rpmostree_option_context_parse (context, + commit2jigdo_option_entries, + &argc, &argv, + invocation, + cancellable, + NULL, NULL, NULL, NULL, + error)) + return EXIT_FAILURE; + + if (argc != 4) + { + rpmostree_usage_error (context, "REV OIRPM-SPEC OUTPUTDIR are required", error); + return EXIT_FAILURE; + } + + if (!(opt_repo && opt_pkgcache_repo)) + { + rpmostree_usage_error (context, "--repo and --pkgcache-repo must be specified", error); + return EXIT_FAILURE; + } + + const char *rev = argv[1]; + const char *oirpm_spec = argv[2]; + const char *outputdir = argv[3]; + if (!g_str_has_prefix (outputdir, "/")) + return glnx_throw (error, "outputdir must be absolute"), EXIT_FAILURE; + + g_autoptr(RpmOstreeCommit2JigdoContext) self = NULL; + if (!rpm_ostree_commit2jigdo_context_new (&self, cancellable, error)) + return EXIT_FAILURE; + if (!impl_commit2jigdo (self, rev, oirpm_spec, outputdir, cancellable, error)) + return EXIT_FAILURE; + + return EXIT_SUCCESS; +} diff --git a/src/app/rpmostree-ex-builtin-jigdo2commit.c b/src/app/rpmostree-ex-builtin-jigdo2commit.c new file mode 100644 index 00000000..79b42989 --- /dev/null +++ b/src/app/rpmostree-ex-builtin-jigdo2commit.c @@ -0,0 +1,368 @@ +/* -*- mode: C; c-file-style: "gnu"; indent-tabs-mode: nil; -*- + * + * Copyright (C) 2017 Colin Walters + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published + * by the Free Software Foundation; either version 2 of the licence or (at + * your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General + * Public License along with this library; if not, write to the + * Free Software Foundation, Inc., 59 Temple Place, Suite 330, + * Boston, MA 02111-1307, USA. + */ + +#include "config.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "rpmostree-ex-builtins.h" +#include "rpmostree-util.h" +#include "rpmostree-core.h" +#include "rpmostree-jigdo-assembler.h" +#include "rpmostree-postprocess.h" +#include "rpmostree-passwd-util.h" +#include "rpmostree-libbuiltin.h" +#include "rpmostree-rpm-util.h" + +#include "libglnx.h" + +static char *opt_repo; +static char *opt_rpmmd_reposdir; +static char *opt_releasever; +static char **opt_enable_rpmmdrepo; +static char *opt_oirpm_version; + +static GOptionEntry jigdo2commit_option_entries[] = { + { "repo", 0, 0, G_OPTION_ARG_STRING, &opt_repo, "OSTree repo", "REPO" }, + { "rpmmd-reposd", 'd', 0, G_OPTION_ARG_STRING, &opt_rpmmd_reposdir, "Path to yum.repos.d (rpmmd) config directory", "PATH" }, + { "enablerepo", 'e', 0, G_OPTION_ARG_STRING_ARRAY, &opt_enable_rpmmdrepo, "Enable rpm-md repo with id ID", "ID" }, + { "releasever", 0, 0, G_OPTION_ARG_STRING, &opt_releasever, "Value for $releasever", "RELEASEVER" }, + { "oirpm-version", 'V', 0, G_OPTION_ARG_STRING, &opt_oirpm_version, "Use this specific version of OIRPM", "VERSION" }, + { NULL } +}; + +typedef struct { + OstreeRepo *repo; + GLnxTmpDir tmpd; + RpmOstreeContext *ctx; +} RpmOstreeJigdo2CommitContext; + +static void +rpm_ostree_jigdo2commit_context_free (RpmOstreeJigdo2CommitContext *ctx) +{ + g_clear_object (&ctx->repo); + (void) glnx_tmpdir_delete (&ctx->tmpd, NULL, NULL); + g_free (ctx); +} +G_DEFINE_AUTOPTR_CLEANUP_FUNC(RpmOstreeJigdo2CommitContext, rpm_ostree_jigdo2commit_context_free) + +/* Initialize a context for converting a jigdo to a commit. + */ +static gboolean +rpm_ostree_jigdo2commit_context_new (RpmOstreeJigdo2CommitContext **out_context, + GCancellable *cancellable, + GError **error) +{ + g_autoptr(RpmOstreeJigdo2CommitContext) self = g_new0 (RpmOstreeJigdo2CommitContext, 1); + + self->repo = ostree_repo_open_at (AT_FDCWD, opt_repo, cancellable, error); + if (!self->repo) + return FALSE; + + /* Our workdir lives in the repo for command line testing */ + if (!glnx_mkdtempat (ostree_repo_get_dfd (self->repo), + "tmp/rpmostree-jigdo-XXXXXX", 0700, &self->tmpd, error)) + return FALSE; + + self->ctx = rpmostree_context_new_tree (self->tmpd.fd, self->repo, cancellable, error); + if (!self->ctx) + return FALSE; + + DnfContext *dnfctx = rpmostree_context_get_dnf (self->ctx); + + if (opt_rpmmd_reposdir) + dnf_context_set_repo_dir (dnfctx, opt_rpmmd_reposdir); + + *out_context = g_steal_pointer (&self); + return TRUE; +} + +static DnfPackage * +query_nevra (DnfContext *dnfctx, + const char *name, + guint64 epoch, + const char *version, + const char *release, + const char *arch, + GError **error) +{ + hy_autoquery HyQuery query = hy_query_create (dnf_context_get_sack (dnfctx)); + hy_query_filter (query, HY_PKG_NAME, HY_EQ, name); + hy_query_filter_num (query, HY_PKG_EPOCH, HY_EQ, epoch); + hy_query_filter (query, HY_PKG_VERSION, HY_EQ, version); + hy_query_filter (query, HY_PKG_RELEASE, HY_EQ, release); + hy_query_filter (query, HY_PKG_ARCH, HY_EQ, arch); + g_autoptr(GPtrArray) pkglist = hy_query_run (query); + if (pkglist->len == 0) + return glnx_null_throw (error, "Failed to find package '%s'", name); + return g_object_ref (pkglist->pdata[0]); +} + +static gboolean +commit_and_print (RpmOstreeJigdo2CommitContext *self, + RpmOstreeRepoAutoTransaction *txn, + GCancellable *cancellable, + GError **error) +{ + OstreeRepoTransactionStats stats; + if (!ostree_repo_commit_transaction (self->repo, &stats, cancellable, error)) + return FALSE; + txn->initialized = FALSE; + + g_print ("Metadata Total: %u\n", stats.metadata_objects_total); + g_print ("Metadata Written: %u\n", stats.metadata_objects_written); + g_print ("Content Total: %u\n", stats.content_objects_total); + g_print ("Content Written: %u\n", stats.content_objects_written); + g_print ("Content Bytes Written: %" G_GUINT64_FORMAT "\n", stats.content_bytes_written); + + return TRUE; +} + +static int +compare_pkgs_reverse (gconstpointer ap, + gconstpointer bp) +{ + DnfPackage **a = (gpointer)ap; + DnfPackage **b = (gpointer)bp; + return dnf_package_cmp (*b, *a); // Reverse +} + +static gboolean +impl_jigdo2commit (RpmOstreeJigdo2CommitContext *self, + const char *oirpm_name, + GCancellable *cancellable, + GError **error) +{ + g_autoptr(GKeyFile) tsk = g_key_file_new (); + + if (opt_releasever) + g_key_file_set_string (tsk, "tree", "releasever", opt_releasever); + if (opt_enable_rpmmdrepo) + g_key_file_set_string_list (tsk, "tree", "repos", + (const char *const*)opt_enable_rpmmdrepo, + g_strv_length (opt_enable_rpmmdrepo)); + g_autoptr(RpmOstreeTreespec) treespec = rpmostree_treespec_new_from_keyfile (tsk, error); + if (!treespec) + return FALSE; + + + if (!rpmostree_context_setup (self->ctx, NULL, NULL, treespec, cancellable, error)) + return FALSE; + if (!rpmostree_context_download_metadata (self->ctx, cancellable, error)) + return FALSE; + + DnfContext *dnfctx = rpmostree_context_get_dnf (self->ctx); + g_autoptr(DnfPackage) oirpm_pkg = NULL; + { hy_autoquery HyQuery query = hy_query_create (dnf_context_get_sack (dnfctx)); + if (opt_oirpm_version) + { + hy_query_filter (query, HY_PKG_NAME, HY_EQ, oirpm_name); + hy_query_filter (query, HY_PKG_VERSION, HY_EQ, opt_oirpm_version); + } + else + { + hy_query_filter (query, HY_PKG_NAME, HY_EQ, oirpm_name); + } + g_autoptr(GPtrArray) pkglist = hy_query_run (query); + if (pkglist->len == 0) + return glnx_throw (error, "Failed to find jigdo OIRPM package '%s'", oirpm_name); + g_ptr_array_sort (pkglist, compare_pkgs_reverse); + if (pkglist->len > 1) + { + g_print ("%u oirpm matches\n", pkglist->len); + } + g_ptr_array_set_size (pkglist, 1); + if (!rpmostree_context_set_packages (self->ctx, pkglist, cancellable, error)) + return FALSE; + oirpm_pkg = g_object_ref (pkglist->pdata[0]); + } + + g_print ("oirpm: %s (%s)\n", dnf_package_get_nevra (oirpm_pkg), + dnf_package_get_reponame (oirpm_pkg)); + + if (!rpmostree_context_download (self->ctx, cancellable, error)) + return FALSE; + + glnx_fd_close int oirpm_fd = -1; + if (!rpmostree_context_consume_package (self->ctx, oirpm_pkg, &oirpm_fd, error)) + return FALSE; + + g_autoptr(RpmOstreeJigdoAssembler) jigdo = rpmostree_jigdo_assembler_new_take_fd (&oirpm_fd, oirpm_pkg, error); + if (!jigdo) + return FALSE; + g_autofree char *checksum = NULL; + g_autoptr(GVariant) commit = NULL; + g_autoptr(GVariant) commit_meta = NULL; + g_autoptr(GVariant) pkgs = NULL; + if (!rpmostree_jigdo_assembler_read_meta (jigdo, &checksum, &commit, &commit_meta, &pkgs, + cancellable, error)) + return FALSE; + + g_print ("OSTree commit: %s\n", checksum); + + { OstreeRepoCommitState commitstate; + gboolean has_commit; + if (!ostree_repo_has_object (self->repo, OSTREE_OBJECT_TYPE_COMMIT, checksum, + &has_commit, cancellable, error)) + return FALSE; + if (has_commit) + { + if (!ostree_repo_load_commit (self->repo, checksum, NULL, &commitstate, error)) + return FALSE; + if (!(commitstate & OSTREE_REPO_COMMIT_STATE_PARTIAL)) + { + g_print ("Commit is already written, nothing to do\n"); + return TRUE; /* 🔚 Early return */ + } + } + } + + g_printerr ("TODO implement GPG verification\n"); + + g_auto(RpmOstreeRepoAutoTransaction) txn = { 0, }; + if (!rpmostree_repo_auto_transaction_start (&txn, self->repo, FALSE, cancellable, error)) + return FALSE; + + if (!rpmostree_jigdo_assembler_write_new_objects (jigdo, self->repo, cancellable, error)) + return FALSE; + + if (!commit_and_print (self, &txn, cancellable, error)) + return FALSE; + + /* Download any packages we don't already have imported */ + g_autoptr(GPtrArray) pkgs_required = g_ptr_array_new_with_free_func (g_object_unref); + const guint n_pkgs = g_variant_n_children (pkgs); + for (guint i = 0; i < n_pkgs; i++) + { + const char *name, *version, *release, *architecture; + const char *repodata_checksum; + guint64 epoch; + g_variant_get_child (pkgs, i, "(&st&s&s&s&s)", + &name, &epoch, &version, &release, &architecture, + &repodata_checksum); + // TODO: use repodata checksum, but probably only if covered by the ostree + // gpg sig? + DnfPackage *pkg = query_nevra (dnfctx, name, epoch, version, release, architecture, error); + // FIXME: We shouldn't require a package to be in the repos if we already + // have it imported otherwise we'll break upgrades for ancient systems + if (!pkg) + return FALSE; + g_ptr_array_add (pkgs_required, g_object_ref (pkg)); + } + + g_print ("Jigdo from %u packages\n", pkgs_required->len); + + if (!rpmostree_context_set_packages (self->ctx, pkgs_required, cancellable, error)) + return FALSE; + + g_autoptr(GHashTable) pkgset_to_import = g_hash_table_new (NULL, NULL); + { g_autoptr(GPtrArray) pkgs_to_import = rpmostree_context_get_packages_to_import (self->ctx); + for (guint i = 0; i < pkgs_to_import->len; i++) + g_hash_table_add (pkgset_to_import, pkgs_to_import->pdata[i]); + g_print ("%u packages to import\n", pkgs_to_import->len); + } + + g_autoptr(GHashTable) pkg_to_xattrs = g_hash_table_new_full (NULL, NULL, + (GDestroyNotify)g_object_unref, + (GDestroyNotify)g_variant_unref); + + for (guint i = 0; i < pkgs_required->len; i++) + { + DnfPackage *pkg = pkgs_required->pdata[i]; + const gboolean should_import = g_hash_table_contains (pkgset_to_import, pkg); + g_autoptr(GVariant) objid_to_xattrs = NULL; + if (!rpmostree_jigdo_assembler_next_xattrs (jigdo, &objid_to_xattrs, cancellable, error)) + return FALSE; + if (!objid_to_xattrs) + return glnx_throw (error, "missing xattr entry: %s", dnf_package_get_name (pkg)); + if (!should_import) + continue; + g_hash_table_insert (pkg_to_xattrs, g_object_ref (pkg), g_steal_pointer (&objid_to_xattrs)); + } + + if (!rpmostree_context_download (self->ctx, cancellable, error)) + return FALSE; + g_autoptr(GVariant) xattr_table = rpmostree_jigdo_assembler_get_xattr_table (jigdo); + if (!rpmostree_context_import_jigdo (self->ctx, xattr_table, pkg_to_xattrs, + cancellable, error)) + return FALSE; + + /* Write commitmeta/commit last since libostree doesn't expose an API to set + * partial state right now. + */ + if (!ostree_repo_write_commit_detached_metadata (self->repo, checksum, commit_meta, + cancellable, error)) + return FALSE; + { g_autofree guint8*csum = NULL; + if (!ostree_repo_write_metadata (self->repo, OSTREE_OBJECT_TYPE_COMMIT, + checksum, commit, &csum, + cancellable, error)) + return FALSE; + } + + return TRUE; +} + +int +rpmostree_ex_builtin_jigdo2commit (int argc, + char **argv, + RpmOstreeCommandInvocation *invocation, + GCancellable *cancellable, + GError **error) +{ + g_autoptr(GOptionContext) context = g_option_context_new ("OIRPM"); + if (!rpmostree_option_context_parse (context, + jigdo2commit_option_entries, + &argc, &argv, + invocation, + cancellable, + NULL, NULL, NULL, NULL, + error)) + return EXIT_FAILURE; + + if (argc != 2) + { + rpmostree_usage_error (context, "OIRPM name is required", error); + return EXIT_FAILURE; + } + + if (!opt_repo) + { + rpmostree_usage_error (context, "--repo must be specified", error); + return EXIT_FAILURE; + } + + const char *oirpm = argv[1]; + + g_autoptr(RpmOstreeJigdo2CommitContext) self = NULL; + if (!rpm_ostree_jigdo2commit_context_new (&self, cancellable, error)) + return EXIT_FAILURE; + if (!impl_jigdo2commit (self, oirpm, cancellable, error)) + return EXIT_FAILURE; + + return EXIT_SUCCESS; +} diff --git a/src/app/rpmostree-ex-builtins.h b/src/app/rpmostree-ex-builtins.h index a201d23f..c3a85f81 100644 --- a/src/app/rpmostree-ex-builtins.h +++ b/src/app/rpmostree-ex-builtins.h @@ -34,6 +34,8 @@ BUILTINPROTO(unpack); BUILTINPROTO(livefs); BUILTINPROTO(override); BUILTINPROTO(kargs); +BUILTINPROTO(commit2jigdo); +BUILTINPROTO(jigdo2commit); #undef BUILTINPROTO diff --git a/src/libpriv/rpmostree-core.c b/src/libpriv/rpmostree-core.c index 5f4933f0..f6e4dabe 100644 --- a/src/libpriv/rpmostree-core.c +++ b/src/libpriv/rpmostree-core.c @@ -864,6 +864,8 @@ rpmostree_find_cache_branch_by_nevra (OstreeRepo *pkgcache, { /* there's no safe way to convert a nevra string to its cache branch, so let's * just do a dumb lookup */ + /* TODO: parse the refs once at core init, this function is itself + * called in loops */ g_autoptr(GHashTable) refs = NULL; if (!ostree_repo_list_refs_ext (pkgcache, "rpmostree/pkg", &refs, @@ -1145,8 +1147,8 @@ append_quoted (GString *r, const char *value) } /* Return the ostree cache branch for a nevra */ -static char * -cache_branch_for_n_evr_a (const char *name, const char *evr, const char *arch) +char * +rpmostree_get_cache_branch_for_n_evr_a (const char *name, const char *evr, const char *arch) { GString *r = g_string_new ("rpmostree/pkg/"); append_quoted (r, name); @@ -1173,16 +1175,16 @@ rpmostree_get_cache_branch_header (Header hdr) g_autofree char *name = headerGetAsString (hdr, RPMTAG_NAME); g_autofree char *evr = headerGetAsString (hdr, RPMTAG_EVR); g_autofree char *arch = headerGetAsString (hdr, RPMTAG_ARCH); - return cache_branch_for_n_evr_a (name, evr, arch); + return rpmostree_get_cache_branch_for_n_evr_a (name, evr, arch); } /* Return the ostree cache branch from a libdnf Package */ char * rpmostree_get_cache_branch_pkg (DnfPackage *pkg) { - return cache_branch_for_n_evr_a (dnf_package_get_name (pkg), - dnf_package_get_evr (pkg), - dnf_package_get_arch (pkg)); + return rpmostree_get_cache_branch_for_n_evr_a (dnf_package_get_name (pkg), + dnf_package_get_evr (pkg), + dnf_package_get_arch (pkg)); } static gboolean @@ -1362,6 +1364,8 @@ find_pkg_in_ostree (RpmOstreeContext *self, * which ones we'll need to import, and which ones we'll need to relabel */ static gboolean sort_packages (RpmOstreeContext *self, + GPtrArray *packages, + GCancellable *cancellable, GError **error) { DnfContext *dnfctx = self->dnfctx; @@ -1374,13 +1378,13 @@ sort_packages (RpmOstreeContext *self, self->pkgs_to_relabel = g_ptr_array_new_with_free_func ((GDestroyNotify)g_object_unref); GPtrArray *sources = dnf_context_get_repos (dnfctx); - g_autoptr(GPtrArray) packages = dnf_goal_get_packages (dnf_context_get_goal (dnfctx), - DNF_PACKAGE_INFO_INSTALL, - DNF_PACKAGE_INFO_UPDATE, - DNF_PACKAGE_INFO_DOWNGRADE, -1); for (guint i = 0; i < packages->len; i++) { DnfPackage *pkg = packages->pdata[i]; + + if (g_cancellable_set_error_if_cancelled (cancellable, error)) + return FALSE; + const char *reponame = dnf_package_get_reponame (pkg); gboolean is_locally_cached = (g_strcmp0 (reponame, HY_CMDLINE_REPO_NAME) == 0); @@ -1787,8 +1791,16 @@ rpmostree_context_prepare (RpmOstreeContext *self, /* XXX: consider a --allow-uninstall switch? */ if (!dnf_goal_depsolve (goal, DNF_INSTALL | DNF_ALLOW_UNINSTALL, error) || - !check_goal_solution (self, removed_pkgnames, replaced_nevras, error) || - !sort_packages (self, error)) + !check_goal_solution (self, removed_pkgnames, replaced_nevras, error)) + { + rpmostree_output_task_end ("failed"); + return FALSE; + } + g_autoptr(GPtrArray) packages = dnf_goal_get_packages (dnf_context_get_goal (dnfctx), + DNF_PACKAGE_INFO_INSTALL, + DNF_PACKAGE_INFO_UPDATE, + DNF_PACKAGE_INFO_DOWNGRADE, -1); + if (!sort_packages (self, packages, cancellable, error)) { rpmostree_output_task_end ("failed"); return FALSE; @@ -1798,6 +1810,29 @@ rpmostree_context_prepare (RpmOstreeContext *self, return TRUE; } +/* Rather than doing a depsolve, directly set which packages + * are required. Will be used by jigdo. + */ +gboolean +rpmostree_context_set_packages (RpmOstreeContext *self, + GPtrArray *packages, + GCancellable *cancellable, + GError **error) +{ + g_clear_pointer (&self->pkgs_to_download, (GDestroyNotify)g_ptr_array_unref); + g_clear_pointer (&self->pkgs_to_import, (GDestroyNotify)g_ptr_array_unref); + g_clear_pointer (&self->pkgs_to_relabel, (GDestroyNotify)g_ptr_array_unref); + return sort_packages (self, packages, cancellable, error); +} + +/* Returns a reference to the set of packages that will be imported */ +GPtrArray * +rpmostree_context_get_packages_to_import (RpmOstreeContext *self) +{ + g_assert (self->pkgs_to_import); + return g_ptr_array_ref (self->pkgs_to_import); +} + static int compare_pkgs (gconstpointer ap, gconstpointer bp) @@ -1993,23 +2028,13 @@ import_one_package (RpmOstreeContext *self, DnfContext *dnfctx, DnfPackage *pkg, OstreeSePolicy *sepolicy, + GVariant *jigdo_xattr_table, + GVariant *jigdo_xattrs, GCancellable *cancellable, GError **error) { - DnfRepo *pkg_repo = dnf_package_get_repo (pkg); - g_autofree char *pkg_path = NULL; - if (pkg_is_local (pkg)) - pkg_path = g_strdup (dnf_package_get_filename (pkg)); - else - { - const char *pkg_location = dnf_package_get_location (pkg); - pkg_path = - g_build_filename (dnf_repo_get_location (pkg_repo), - "packages", glnx_basename (pkg_location), NULL); - } - - /* Verify signatures if enabled */ - if (!dnf_transaction_gpgcheck_package (dnf_context_get_transaction (dnfctx), pkg, error)) + glnx_fd_close int fd = -1; + if (!rpmostree_context_consume_package (self, pkg, &fd, error)) return FALSE; /* Only set SKIP_EXTRANEOUS for packages we know need it, so that @@ -2030,19 +2055,14 @@ import_one_package (RpmOstreeContext *self, } /* TODO - tweak the unpacker flags for containers */ - g_autoptr(RpmOstreeImporter) unpacker = rpmostree_importer_new_at (AT_FDCWD, pkg_path, pkg, flags, error); + g_autoptr(RpmOstreeImporter) unpacker = rpmostree_importer_new_fd (fd, pkg, flags, error); if (!unpacker) return FALSE; - /* And delete it now; this does mean if we fail it'll have been - * deleted and hence more annoying to debug, but in practice people - * should be able to redownload, and if the error was something like - * ENOSPC, deleting it was the right move I'd say. - */ - if (!pkg_is_local (pkg)) + if (jigdo_xattrs) { - if (!glnx_unlinkat (AT_FDCWD, pkg_path, 0, error)) - return FALSE; + g_assert (!sepolicy); + rpmostree_importer_set_jigdo_mode (unpacker, jigdo_xattr_table, jigdo_xattrs); } OstreeRepo *ostreerepo = get_pkgcache_repo (self); @@ -2064,9 +2084,11 @@ dnf_state_assert_done (DnfState *hifstate) } gboolean -rpmostree_context_import (RpmOstreeContext *self, - GCancellable *cancellable, - GError **error) +rpmostree_context_import_jigdo (RpmOstreeContext *self, + GVariant *jigdo_xattr_table, + GHashTable *jigdo_pkg_to_xattrs, + GCancellable *cancellable, + GError **error) { DnfContext *dnfctx = self->dnfctx; const int n = self->pkgs_to_import->len; @@ -2075,6 +2097,7 @@ rpmostree_context_import (RpmOstreeContext *self, OstreeRepo *repo = get_pkgcache_repo (self); g_return_val_if_fail (repo != NULL, FALSE); + g_return_val_if_fail (jigdo_pkg_to_xattrs == NULL || self->sepolicy == NULL, FALSE); if (!dnf_transaction_import_keys (dnf_context_get_transaction (dnfctx), error)) return FALSE; @@ -2094,8 +2117,16 @@ rpmostree_context_import (RpmOstreeContext *self, for (guint i = 0; i < self->pkgs_to_import->len; i++) { DnfPackage *pkg = self->pkgs_to_import->pdata[i]; - if (!import_one_package (self, dnfctx, pkg, - self->sepolicy, cancellable, error)) + GVariant *jigdo_xattrs = NULL; + if (jigdo_pkg_to_xattrs) + { + jigdo_xattrs = g_hash_table_lookup (jigdo_pkg_to_xattrs, pkg); + if (!jigdo_xattrs) + g_error ("Failed to find jigdo xattrs for %s", dnf_package_get_nevra (pkg)); + } + if (!import_one_package (self, dnfctx, pkg, self->sepolicy, + jigdo_xattr_table, jigdo_xattrs, + cancellable, error)) return FALSE; dnf_state_assert_done (hifstate); } @@ -2116,6 +2147,59 @@ rpmostree_context_import (RpmOstreeContext *self, return TRUE; } +gboolean +rpmostree_context_import (RpmOstreeContext *self, + GCancellable *cancellable, + GError **error) +{ + return rpmostree_context_import_jigdo (self, NULL, NULL, cancellable, error); +} + +/* Given a single package, verify its GPG signature (if enabled), open a file + * descriptor for it, and delete the on-disk downloaded copy. + */ +gboolean +rpmostree_context_consume_package (RpmOstreeContext *self, + DnfPackage *pkg, + int *out_fd, + GError **error) +{ + /* Verify signatures if enabled */ + if (!dnf_transaction_gpgcheck_package (dnf_context_get_transaction (self->dnfctx), pkg, error)) + return FALSE; + + DnfRepo *pkg_repo = dnf_package_get_repo (pkg); + g_autofree char *pkg_path = NULL; + const gboolean is_local = pkg_is_local (pkg); + if (is_local) + pkg_path = g_strdup (dnf_package_get_filename (pkg)); + else + { + const char *pkg_location = dnf_package_get_location (pkg); + pkg_path = + g_build_filename (dnf_repo_get_location (pkg_repo), + "packages", glnx_basename (pkg_location), NULL); + } + + glnx_autofd int fd = -1; + if (!glnx_openat_rdonly (AT_FDCWD, pkg_path, TRUE, &fd, error)) + return FALSE; + + /* And delete it now; this does mean if we fail it'll have been + * deleted and hence more annoying to debug, but in practice people + * should be able to redownload, and if the error was something like + * ENOSPC, deleting it was the right move I'd say. + */ + if (!pkg_is_local (pkg)) + { + if (!glnx_unlinkat (AT_FDCWD, pkg_path, 0, error)) + return FALSE; + } + + *out_fd = glnx_steal_fd (&fd); + return TRUE; +} + static gboolean checkout_package (OstreeRepo *repo, DnfPackage *pkg, diff --git a/src/libpriv/rpmostree-core.h b/src/libpriv/rpmostree-core.h index 976c349f..beb2bad9 100644 --- a/src/libpriv/rpmostree-core.h +++ b/src/libpriv/rpmostree-core.h @@ -92,6 +92,7 @@ gboolean rpmostree_context_get_state_sha512 (RpmOstreeContext *self, char **out_checksum, GError **error); +char * rpmostree_get_cache_branch_for_n_evr_a (const char *name, const char *evr, const char *arch); char *rpmostree_get_cache_branch_header (Header hdr); char *rpmostree_get_cache_branch_pkg (DnfPackage *pkg); @@ -130,14 +131,34 @@ gboolean rpmostree_context_prepare (RpmOstreeContext *self, GCancellable *cancellable, GError **error); +/* Alternative to _prepare() for non-depsolve cases like jigdo */ +gboolean rpmostree_context_set_packages (RpmOstreeContext *self, + GPtrArray *packages, + GCancellable *cancellable, + GError **error); + +GPtrArray *rpmostree_context_get_packages_to_import (RpmOstreeContext *self); + gboolean rpmostree_context_download (RpmOstreeContext *self, GCancellable *cancellable, GError **error); +gboolean +rpmostree_context_consume_package (RpmOstreeContext *self, + DnfPackage *package, + int *out_fd, + GError **error); + gboolean rpmostree_context_import (RpmOstreeContext *self, GCancellable *cancellable, GError **error); +gboolean rpmostree_context_import_jigdo (RpmOstreeContext *self, + GVariant *xattr_table, + GHashTable *pkg_to_xattrs, + GCancellable *cancellable, + GError **error); + gboolean rpmostree_context_relabel (RpmOstreeContext *self, GCancellable *cancellable, GError **error); diff --git a/src/libpriv/rpmostree-importer.c b/src/libpriv/rpmostree-importer.c index eb0c93ed..d5735d30 100644 --- a/src/libpriv/rpmostree-importer.c +++ b/src/libpriv/rpmostree-importer.c @@ -35,6 +35,7 @@ #include "rpmostree-unpacker-core.h" #include "rpmostree-importer.h" #include "rpmostree-core.h" +#include "rpmostree-jigdo-assembler.h" #include "rpmostree-rpm-util.h" #include #include @@ -67,6 +68,11 @@ struct RpmOstreeImporter char *hdr_sha256; char *ostree_branch; + + gboolean jigdo_mode; + GVariant *jigdo_xattr_table; + GVariant *jigdo_xattrs; + GVariant *jigdo_next_xattrs; /* passed from filter to xattr cb */ }; G_DEFINE_TYPE(RpmOstreeImporter, rpmostree_importer, G_TYPE_OBJECT) @@ -91,6 +97,10 @@ rpmostree_importer_finalize (GObject *object) g_free (self->hdr_sha256); + g_clear_pointer (&self->jigdo_xattr_table, (GDestroyNotify)g_variant_unref); + g_clear_pointer (&self->jigdo_xattrs, (GDestroyNotify)g_variant_unref); + g_clear_pointer (&self->jigdo_next_xattrs, (GDestroyNotify)g_variant_unref); + G_OBJECT_CLASS (rpmostree_importer_parent_class)->finalize (object); } @@ -300,6 +310,16 @@ rpmostree_importer_new_at (int dfd, const char *path, return g_steal_pointer (&ret); } +void +rpmostree_importer_set_jigdo_mode (RpmOstreeImporter *self, + GVariant *xattr_table, + GVariant *xattrs) +{ + self->jigdo_mode = TRUE; + self->jigdo_xattr_table = g_variant_ref (xattr_table); + self->jigdo_xattrs = g_variant_ref (xattrs); +} + static void get_rpmfi_override (RpmOstreeImporter *self, const char *path, @@ -475,6 +495,13 @@ build_metadata_variant (RpmOstreeImporter *self, g_variant_new_string (chksum_repr)); } + if (self->jigdo_mode) + { + g_variant_builder_add (&metadata_builder, "{sv}", + "rpmostree.jigdo", + g_variant_new_boolean (TRUE)); + } + if (self->doc_files) { g_variant_builder_add (&metadata_builder, "{sv}", @@ -680,6 +707,45 @@ unprivileged_filter_cb (OstreeRepo *repo, return OSTREE_REPO_COMMIT_FILTER_ALLOW; } +static OstreeRepoCommitFilterResult +jigdo_filter_cb (OstreeRepo *repo, + const char *path, + GFileInfo *file_info, + gpointer user_data) +{ + RpmOstreeImporter *self = ((cb_data*)user_data)->self; + GError **error = ((cb_data*)user_data)->error; + const gboolean error_was_set = (error && *error != NULL); + + if (error_was_set) + return OSTREE_REPO_COMMIT_FILTER_SKIP; + + if (g_file_info_get_file_type (file_info) != G_FILE_TYPE_DIRECTORY) + { + self->jigdo_next_xattrs = NULL; + if (!rpmostree_jigdo_assembler_xattr_lookup (self->jigdo_xattr_table, path, + self->jigdo_xattrs, + &self->jigdo_next_xattrs, + error)) + return OSTREE_REPO_COMMIT_FILTER_SKIP; + /* No xattrs means we don't need to import it */ + if (!self->jigdo_next_xattrs) + return OSTREE_REPO_COMMIT_FILTER_SKIP; + } + + return OSTREE_REPO_COMMIT_FILTER_ALLOW; +} + +static GVariant* +jigdo_xattr_cb (OstreeRepo *repo, + const char *path, + GFileInfo *file_info, + gpointer user_data) +{ + RpmOstreeImporter *self = user_data; + return g_steal_pointer (&self->jigdo_next_xattrs); +} + static GVariant* xattr_cb (OstreeRepo *repo, const char *path, @@ -737,7 +803,9 @@ import_rpm_to_repo (RpmOstreeImporter *self, * is unprivileged, anything else is a compose. */ const gboolean unprivileged = ostree_repo_get_mode (repo) == OSTREE_REPO_MODE_BARE_USER_ONLY; - if (unprivileged) + if (self->jigdo_mode) + filter = jigdo_filter_cb; + else if (unprivileged) filter = unprivileged_filter_cb; else filter = compose_filter_cb; @@ -749,9 +817,17 @@ import_rpm_to_repo (RpmOstreeImporter *self, modifier_flags |= OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS; g_autoptr(OstreeRepoCommitModifier) modifier = ostree_repo_commit_modifier_new (modifier_flags, filter, &fdata, NULL); - ostree_repo_commit_modifier_set_xattr_callback (modifier, xattr_cb, - NULL, self); - ostree_repo_commit_modifier_set_sepolicy (modifier, sepolicy); + if (self->jigdo_mode) + { + ostree_repo_commit_modifier_set_xattr_callback (modifier, jigdo_xattr_cb, + NULL, self); + g_assert (sepolicy == NULL); + } + else + { + ostree_repo_commit_modifier_set_xattr_callback (modifier, xattr_cb, NULL, self); + ostree_repo_commit_modifier_set_sepolicy (modifier, sepolicy); + } OstreeRepoImportArchiveOptions opts = { 0 }; opts.ignore_unsupported_content = TRUE; diff --git a/src/libpriv/rpmostree-importer.h b/src/libpriv/rpmostree-importer.h index 58d16e9f..b94a4adc 100644 --- a/src/libpriv/rpmostree-importer.h +++ b/src/libpriv/rpmostree-importer.h @@ -59,6 +59,10 @@ rpmostree_importer_new_at (int dfd, RpmOstreeImporterFlags flags, GError **error); +void rpmostree_importer_set_jigdo_mode (RpmOstreeImporter *self, + GVariant *xattr_table, + GVariant *xattrs); + gboolean rpmostree_importer_read_metainfo (int fd, Header *out_header, diff --git a/src/libpriv/rpmostree-jigdo-assembler.c b/src/libpriv/rpmostree-jigdo-assembler.c new file mode 100644 index 00000000..996103b2 --- /dev/null +++ b/src/libpriv/rpmostree-jigdo-assembler.c @@ -0,0 +1,643 @@ +/* -*- mode: C; c-file-style: "gnu"; indent-tabs-mode: nil; -*- + * + * Copyright (C) 2017 Red Hat, Inc. + * + * Licensed under the GNU Lesser General Public License Version 2.1 + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + + +#include "config.h" + +#include +#include "rpmostree-libarchive-input-stream.h" +#include "rpmostree-unpacker-core.h" +#include "rpmostree-jigdo-assembler.h" +#include "rpmostree-core.h" +#include "rpmostree-rpm-util.h" +#include +#include +#include +#include +#include +#include + +#include +#include + +typedef enum { + STATE_COMMIT, + STATE_DIRMETA, + STATE_DIRTREE, + STATE_NEW_CONTENTIDENT, + STATE_NEW, + STATE_XATTRS_TABLE, + STATE_XATTRS_PKG, +} JigdoAssemblerState; + +static gboolean +throw_libarchive_error (GError **error, + struct archive *a) +{ + return glnx_throw (error, "%s", archive_error_string (a)); +} + +typedef GObjectClass RpmOstreeJigdoAssemblerClass; + +struct RpmOstreeJigdoAssembler +{ + GObject parent_instance; + JigdoAssemblerState state; + DnfPackage *pkg; + GVariant *commit; + GVariant *meta; + GVariant *pkgs; + char *checksum; + GVariant *xattrs_table; + struct archive *archive; + struct archive_entry *next_entry; + int fd; +}; + +G_DEFINE_TYPE(RpmOstreeJigdoAssembler, rpmostree_jigdo_assembler, G_TYPE_OBJECT) + +static void +rpmostree_jigdo_assembler_finalize (GObject *object) +{ + RpmOstreeJigdoAssembler *self = (RpmOstreeJigdoAssembler*)object; + if (self->archive) + archive_read_free (self->archive); + g_clear_pointer (&self->commit, (GDestroyNotify)g_variant_unref); + g_clear_pointer (&self->meta, (GDestroyNotify)g_variant_unref); + g_clear_pointer (&self->pkgs, (GDestroyNotify)g_variant_unref); + g_free (self->checksum); + g_clear_object (&self->pkg); + g_clear_pointer (&self->xattrs_table, (GDestroyNotify)g_variant_unref); + glnx_close_fd (&self->fd); + + G_OBJECT_CLASS (rpmostree_jigdo_assembler_parent_class)->finalize (object); +} + +static void +rpmostree_jigdo_assembler_class_init (RpmOstreeJigdoAssemblerClass *klass) +{ + GObjectClass *gobject_class = G_OBJECT_CLASS (klass); + + gobject_class->finalize = rpmostree_jigdo_assembler_finalize; +} + +static void +rpmostree_jigdo_assembler_init (RpmOstreeJigdoAssembler *self) +{ + self->fd = -1; +} + +/* + * rpmostree_jigdo_assembler_new_take_fd: + * @fd: Fd, ownership is taken + * @pkg: (optional): Package reference, used for metadata + * @error: error + * + * Create a new unpacker instance. The @pkg argument, if + * specified, will be inspected and metadata such as the + * origin repo will be added to the final commit. + */ +RpmOstreeJigdoAssembler * +rpmostree_jigdo_assembler_new_take_fd (int *fd, + DnfPackage *pkg, + GError **error) +{ + glnx_fd_close int owned_fd = glnx_steal_fd (fd); + + struct archive *archive = rpmostree_unpack_rpm2cpio (owned_fd, error); + if (archive == NULL) + return NULL; + + RpmOstreeJigdoAssembler *ret = g_object_new (RPMOSTREE_TYPE_JIGDO_ASSEMBLER, NULL); + ret->archive = g_steal_pointer (&archive); + ret->pkg = pkg ? g_object_ref (pkg) : NULL; + ret->fd = glnx_steal_fd (&owned_fd); + + return g_steal_pointer (&ret); +} + +static GVariant * +jigdo_read_variant (const GVariantType *vtype, + struct archive *a, + struct archive_entry *entry, + GCancellable *cancellable, + GError **error) +{ + const char *path = archive_entry_pathname (entry); + const struct stat *stbuf = archive_entry_stat (entry); + if (!S_ISREG (stbuf->st_mode)) + return glnx_null_throw (error, "Expected regular file for entry: %s", path); + if (stbuf->st_size > OSTREE_MAX_METADATA_SIZE) + { + g_autofree char *max_formatted = g_format_size (OSTREE_MAX_METADATA_SIZE); + g_autofree char *found_formatted = g_format_size (stbuf->st_size); + return glnx_null_throw (error, "Exceeded maximum size %s; %s is of size: %s", max_formatted, found_formatted, path); + } + g_assert_cmpint (stbuf->st_size, >=, 0); + const size_t total = stbuf->st_size; + g_autofree guint8* buf = g_malloc (total); + size_t bytes_read = 0; + while (bytes_read < total) + { + ssize_t r = archive_read_data (a, buf + bytes_read, total - bytes_read); + if (r < 0) + return throw_libarchive_error (error, a), NULL; + if (r == 0) + break; + bytes_read += r; + } + g_assert_cmpint (bytes_read, ==, total) +; + /* Need to take ownership once now, then pass it as a parameter twice */ + guint8* buf_owned = g_steal_pointer (&buf); + return g_variant_new_from_data (vtype, buf_owned, bytes_read, FALSE, g_free, buf_owned); +} + +/* Remove leading prefix */ +static const char * +peel_entry_pathname (struct archive_entry *entry, + GError **error) +{ + const char *pathname = archive_entry_pathname (entry); + static const char prefix[] = "./usr/lib/ostree-jigdo/"; + if (!g_str_has_prefix (pathname, prefix)) + return glnx_null_throw (error, "Entry does not have prefix '%s': %s", prefix, pathname); + pathname += strlen (prefix); + const char *nextslash = strchr (pathname, '/'); + if (!nextslash) + return glnx_null_throw (error, "Missing subdir in %s", pathname); + return nextslash+1; +} + +static gboolean +jigdo_next_entry (RpmOstreeJigdoAssembler *self, + gboolean *out_eof, + struct archive_entry **out_entry, + GCancellable *cancellable, + GError **error) +{ + if (g_cancellable_set_error_if_cancelled (cancellable, error)) + return FALSE; + + *out_eof = FALSE; + + if (self->next_entry) + { + *out_entry = g_steal_pointer (&self->next_entry); + return TRUE; /* 🔚 Early return */ + } + + /* Loop, skipping non-regular files */ + struct archive_entry *entry = NULL; + while (TRUE) + { + int r = archive_read_next_header (self->archive, &entry); + if (r == ARCHIVE_EOF) + { + *out_eof = TRUE; + return TRUE; /* 🔚 Early return */ + } + if (r != ARCHIVE_OK) + return throw_libarchive_error (error, self->archive); + + const struct stat *stbuf = archive_entry_stat (entry); + /* We only have regular files, ignore intermediate dirs */ + if (!S_ISREG (stbuf->st_mode)) + continue; + + /* Otherwise we're done */ + break; + } + + *out_eof = FALSE; + *out_entry = entry; /* Owned by archive */ + return TRUE; +} + +static struct archive_entry * +jigdo_require_next_entry (RpmOstreeJigdoAssembler *self, + GCancellable *cancellable, + GError **error) +{ + gboolean eof; + struct archive_entry *entry; + if (!jigdo_next_entry (self, &eof, &entry, cancellable, error)) + return FALSE; + if (eof) + return glnx_null_throw (error, "Unexpected end of archive"); + return entry; +} + +static char * +parse_checksum_from_pathname (const char *pathname, + GError **error) +{ + /* We have an extra / */ + if (strlen (pathname) != OSTREE_SHA256_STRING_LEN + 1) + return glnx_null_throw (error, "Invalid checksum path: %s", pathname); + g_autoptr(GString) buf = g_string_new (""); + g_string_append_len (buf, pathname, 2); + g_string_append (buf, pathname+3); + return g_string_free (g_steal_pointer (&buf), FALSE); +} + +/* First step: read metadata: the commit object and its metadata, suitable for + * GPG verification, as well as the component package NEVRAs. + */ +gboolean +rpmostree_jigdo_assembler_read_meta (RpmOstreeJigdoAssembler *self, + char **out_checksum, + GVariant **out_commit, + GVariant **out_detached_meta, + GVariant **out_pkgs, + GCancellable *cancellable, + GError **error) +{ + g_assert_cmpint (self->state, ==, STATE_COMMIT); + struct archive_entry *entry = jigdo_require_next_entry (self, cancellable, error); + if (!entry) + return FALSE; + const char *entry_path = peel_entry_pathname (entry, error); + if (!entry_path) + return FALSE; + if (!g_str_has_prefix (entry_path, RPMOSTREE_JIGDO_COMMIT_DIR "/")) + return glnx_throw (error, "Unexpected entry: %s", entry_path); + entry_path += strlen (RPMOSTREE_JIGDO_COMMIT_DIR "/"); + + g_autofree char *checksum = parse_checksum_from_pathname (entry_path, error); + if (!checksum) + return FALSE; + + g_autoptr(GVariant) commit = jigdo_read_variant (OSTREE_COMMIT_GVARIANT_FORMAT, + self->archive, entry, cancellable, error); + g_autoptr(GVariant) meta = NULL; + + entry = jigdo_require_next_entry (self, cancellable, error); + entry_path = peel_entry_pathname (entry, error); + if (!entry_path) + return FALSE; + if (g_str_equal (entry_path, RPMOSTREE_JIGDO_COMMIT_DIR "/meta")) + { + meta = jigdo_read_variant (G_VARIANT_TYPE ("a{sv}"), self->archive, entry, + cancellable, error); + if (!meta) + return FALSE; + } + else + { + self->next_entry = entry; /* Stash for next call */ + } + + /* And the component packages */ + entry = jigdo_require_next_entry (self, cancellable, error); + entry_path = peel_entry_pathname (entry, error); + if (!entry_path) + return FALSE; + if (!g_str_equal (entry_path, RPMOSTREE_JIGDO_PKGS)) + return glnx_throw (error, "Unexpected state for path: %s", entry_path); + g_autoptr(GVariant) pkgs = jigdo_read_variant (RPMOSTREE_JIGDO_PKGS_VARIANT_FORMAT, + self->archive, entry, cancellable, error); + if (!pkgs) + return FALSE; + + self->state = STATE_DIRMETA; + self->checksum = g_strdup (checksum); + self->commit = g_variant_ref (commit); + self->meta = meta ? g_variant_ref (meta) : NULL; + self->pkgs = g_variant_ref (pkgs); + *out_checksum = g_steal_pointer (&checksum); + *out_commit = g_steal_pointer (&commit); + *out_detached_meta = g_steal_pointer (&meta); + *out_pkgs = g_steal_pointer (&pkgs); + return TRUE; +} + +static gboolean +process_contentident (RpmOstreeJigdoAssembler *self, + OstreeRepo *repo, + struct archive_entry *entry, + const char *meta_pathname, + GCancellable *cancellable, + GError **error) +{ + GLNX_AUTO_PREFIX_ERROR ("Processing content-identical", error); + /* Read the metadata variant, which has an array of checksum, metadata + * for regfile objects that have identical content. + */ + if (!g_str_has_suffix (meta_pathname, "/01meta")) + return glnx_throw (error, "Malformed contentident: %s", meta_pathname); + const char *contentident_id_start = meta_pathname + strlen (RPMOSTREE_JIGDO_NEW_CONTENTIDENT_DIR "/"); + const char *slash = strchr (contentident_id_start, '/'); + if (!slash) + return glnx_throw (error, "Malformed contentident: %s", meta_pathname); + // g_autofree char *contentident_id_str = g_strndup (contentident_id_start, slash - contentident_id_start); + + g_autoptr(GVariant) meta = jigdo_read_variant (RPMOSTREE_JIGDO_NEW_CONTENTIDENT_VARIANT_FORMAT, + self->archive, entry, + cancellable, error); + + + /* Read the content */ + // FIXME match contentident_id + entry = jigdo_require_next_entry (self, cancellable, error); + if (!entry) + return FALSE; + + const char *content_pathname = peel_entry_pathname (entry, error); + if (!content_pathname) + return FALSE; + if (!g_str_has_suffix (content_pathname, "/05content")) + return glnx_throw (error, "Malformed contentident: %s", content_pathname); + + const struct stat *stbuf = archive_entry_stat (entry); + + /* Copy the data to a temporary file; a better optimization would be to write + * the data to the first object, then clone it, but that requires some + * more libostree API. As far as I can see, one can't reliably seek with + * libarchive; only some formats support it, and cpio isn't one of them. + */ + g_auto(GLnxTmpfile) tmpf = { 0, }; + if (!glnx_open_anonymous_tmpfile (O_RDWR | O_CLOEXEC, &tmpf, error)) + return FALSE; + + const size_t total = stbuf->st_size; + const size_t bufsize = MIN (128*1024, total); + g_autofree guint8* buf = g_malloc (bufsize); + size_t bytes_read = 0; + while (bytes_read < total) + { + ssize_t r = archive_read_data (self->archive, buf, MIN (bufsize, total - bytes_read)); + if (r < 0) + return throw_libarchive_error (error, self->archive); + if (r == 0) + break; + if (glnx_loop_write (tmpf.fd, buf, r) < 0) + return glnx_throw_errno_prefix (error, "write"); + bytes_read += r; + } + g_assert_cmpint (bytes_read, ==, total); + g_clear_pointer (&buf, g_free); + + const guint n = g_variant_n_children (meta); + for (guint i = 0; i < n; i++) + { + const char *checksum; + guint32 uid,gid,mode; + g_autoptr(GVariant) xattrs = NULL; + g_variant_get_child (meta, i, "(&suuu@a(ayay))", &checksum, &uid, &gid, &mode, &xattrs); + /* See if we already have this object */ + gboolean has_object; + if (!ostree_repo_has_object (repo, OSTREE_OBJECT_TYPE_FILE, checksum, + &has_object, cancellable, error)) + return FALSE; + if (has_object) + continue; + uid = GUINT32_FROM_BE (uid); + gid = GUINT32_FROM_BE (gid); + mode = GUINT32_FROM_BE (mode); + + if (lseek (tmpf.fd, 0, SEEK_SET) < 0) + return glnx_throw_errno_prefix (error, "lseek"); + g_autoptr(GInputStream) istream = g_unix_input_stream_new (tmpf.fd, FALSE); + /* Like _ostree_stbuf_to_gfileinfo() - TODO make that public with a + * better content writing API. + */ + g_autoptr(GFileInfo) finfo = g_file_info_new (); + g_file_info_set_attribute_uint32 (finfo, "standard::type", G_FILE_TYPE_REGULAR); + g_file_info_set_attribute_boolean (finfo, "standard::is-symlink", FALSE); + g_file_info_set_attribute_uint32 (finfo, "unix::uid", uid); + g_file_info_set_attribute_uint32 (finfo, "unix::gid", gid); + g_file_info_set_attribute_uint32 (finfo, "unix::mode", mode); + g_file_info_set_attribute_uint64 (finfo, "standard::size", total); + + g_autoptr(GInputStream) objstream = NULL; + guint64 objlen; + if (!ostree_raw_file_to_content_stream (istream, finfo, xattrs, &objstream, + &objlen, cancellable, error)) + return FALSE; + + g_autofree guchar *csum = NULL; + if (!ostree_repo_write_content (repo, checksum, objstream, objlen, &csum, + cancellable, error)) + return FALSE; + } + + return TRUE; +} + +static gboolean +state_transition (RpmOstreeJigdoAssembler *self, + const char *pathname, + JigdoAssemblerState new_state, + GError **error) +{ + if (self->state > new_state) + return glnx_throw (error, "Unexpected state for path: %s", pathname); + self->state = new_state; + return TRUE; +} + +/* Process new objects included in the OIRPM */ +gboolean +rpmostree_jigdo_assembler_write_new_objects (RpmOstreeJigdoAssembler *self, + OstreeRepo *repo, + GCancellable *cancellable, + GError **error) +{ + GLNX_AUTO_PREFIX_ERROR ("Writing new objects", error); + g_assert_cmpint (self->state, ==, STATE_DIRMETA); + + /* TODO sort objects in order for importing, verify we're not + * importing an unknown object. + */ + while (TRUE) + { + gboolean eof; + struct archive_entry *entry; + if (!jigdo_next_entry (self, &eof, &entry, cancellable, error)) + return FALSE; + if (eof) + break; + const char *pathname = peel_entry_pathname (entry, error); + if (!pathname) + return FALSE; + if (g_str_has_prefix (pathname, RPMOSTREE_JIGDO_DIRMETA_DIR "/")) + { + if (!state_transition (self, pathname, STATE_DIRMETA, error)) + return FALSE; + g_autofree char *checksum = + parse_checksum_from_pathname (pathname + strlen (RPMOSTREE_JIGDO_DIRMETA_DIR "/"), error); + if (!checksum) + return FALSE; + g_autoptr(GVariant) dirmeta = jigdo_read_variant (OSTREE_DIRMETA_GVARIANT_FORMAT, + self->archive, entry, + cancellable, error); + g_autofree guint8*csum = NULL; + if (!ostree_repo_write_metadata (repo, OSTREE_OBJECT_TYPE_DIR_META, + checksum, dirmeta, &csum, cancellable, error)) + return FALSE; + } + else if (g_str_has_prefix (pathname, RPMOSTREE_JIGDO_DIRTREE_DIR "/")) + { + if (!state_transition (self, pathname, STATE_DIRTREE, error)) + return FALSE; + g_autofree char *checksum = + parse_checksum_from_pathname (pathname + strlen (RPMOSTREE_JIGDO_DIRTREE_DIR "/"), error); + if (!checksum) + return FALSE; + g_autoptr(GVariant) dirtree = jigdo_read_variant (OSTREE_TREE_GVARIANT_FORMAT, + self->archive, entry, + cancellable, error); + g_autofree guint8*csum = NULL; + if (!ostree_repo_write_metadata (repo, OSTREE_OBJECT_TYPE_DIR_TREE, + checksum, dirtree, &csum, cancellable, error)) + return FALSE; + } + else if (g_str_has_prefix (pathname, RPMOSTREE_JIGDO_NEW_CONTENTIDENT_DIR "/")) + { + if (!state_transition (self, pathname, STATE_NEW_CONTENTIDENT, error)) + return FALSE; + if (!process_contentident (self, repo, entry, pathname, cancellable, error)) + return FALSE; + } + else if (g_str_has_prefix (pathname, RPMOSTREE_JIGDO_NEW_DIR "/")) + { + if (!state_transition (self, pathname, STATE_NEW, error)) + return FALSE; + g_autofree char *checksum = + parse_checksum_from_pathname (pathname + strlen (RPMOSTREE_JIGDO_NEW_DIR "/"), error); + if (!checksum) + return FALSE; + + const struct stat *stbuf = archive_entry_stat (entry); + g_assert_cmpint (stbuf->st_size, >=, 0); + + g_autoptr(GInputStream) archive_stream = _rpm_ostree_libarchive_input_stream_new (self->archive); + g_autofree guint8*csum = NULL; + if (!ostree_repo_write_content (repo, checksum, archive_stream, + stbuf->st_size, &csum, cancellable, error)) + return FALSE; + } + else if (g_str_has_prefix (pathname, RPMOSTREE_JIGDO_XATTRS_DIR "/")) + { + self->next_entry = g_steal_pointer (&entry); /* Stash for next call */ + break; + } + else + return glnx_throw (error, "Unexpected entry: %s", pathname); + } + + return TRUE; +} + +GVariant * +rpmostree_jigdo_assembler_get_xattr_table (RpmOstreeJigdoAssembler *self) +{ + g_assert (self->xattrs_table); + return g_variant_ref (self->xattrs_table); +} + +/* Loop over each package, returning its xattr set (as indexes into the xattr table) */ +gboolean +rpmostree_jigdo_assembler_next_xattrs (RpmOstreeJigdoAssembler *self, + GVariant **out_objid_to_xattrs, + GCancellable *cancellable, + GError **error) +{ + /* Init output variable for EOF state now */ + *out_objid_to_xattrs = NULL; + + /* If we haven't loaded the xattr string table, do so */ + if (self->state < STATE_XATTRS_TABLE) + { + gboolean eof; + struct archive_entry *entry; + if (!jigdo_next_entry (self, &eof, &entry, cancellable, error)) + return FALSE; + if (eof) + return TRUE; /* 🔚 Early return */ + + const char *pathname = peel_entry_pathname (entry, error); + if (!pathname) + return FALSE; + if (!g_str_has_prefix (pathname, RPMOSTREE_JIGDO_XATTRS_TABLE)) + return glnx_throw (error, "Unexpected entry: %s", pathname); + + g_autoptr(GVariant) xattrs_table = jigdo_read_variant (RPMOSTREE_JIGDO_XATTRS_TABLE_VARIANT_FORMAT, + self->archive, entry, cancellable, error); + if (!xattrs_table) + return FALSE; + g_assert (!self->xattrs_table); + self->xattrs_table = g_steal_pointer (&xattrs_table); + self->state = STATE_XATTRS_TABLE; + } + + /* Look for an xattr entry */ + gboolean eof; + struct archive_entry *entry; + if (!jigdo_next_entry (self, &eof, &entry, cancellable, error)) + return FALSE; + if (eof) + return TRUE; /* 🔚 Early return */ + + const char *pathname = peel_entry_pathname (entry, error); + if (!pathname) + return FALSE; + /* At this point there's nothing left besides xattrs, so throw if it doesn't + * match that filename pattern. + */ + if (!g_str_has_prefix (pathname, RPMOSTREE_JIGDO_XATTRS_PKG_DIR "/")) + return glnx_throw (error, "Unexpected entry: %s", pathname); + // const char *nevra = pathname + strlen (RPMOSTREE_JIGDO_XATTRS_PKG_DIR "/"); + *out_objid_to_xattrs = jigdo_read_variant (RPMOSTREE_JIGDO_XATTRS_PKG_VARIANT_FORMAT, + self->archive, entry, cancellable, error); + return TRUE; +} + +/* Client side lookup for xattrs */ +gboolean +rpmostree_jigdo_assembler_xattr_lookup (GVariant *xattr_table, + const char *path, + GVariant *xattrs, + GVariant **out_xattrs, + GError **error) +{ + int pos; + if (!rpmostree_variant_bsearch_str (xattrs, path, &pos)) + { + const char *bn = glnx_basename (path); + if (!rpmostree_variant_bsearch_str (xattrs, bn, &pos)) + { + // TODO add an "objects to skip" map; currently not found means + // "don't import" + *out_xattrs = NULL; + return TRUE; + /* jigdodata->caught_error = TRUE; */ + /* return glnx_null_throw (&jigdodata->error, "Failed to find jigdo xattrs for path '%s'", path); */ + } + } + guint xattr_idx; + g_variant_get_child (xattrs, pos, "(&su)", NULL, &xattr_idx); + if (xattr_idx >= g_variant_n_children (xattr_table)) + return glnx_throw (error, "Out of range jigdo xattr index %u for path '%s'", xattr_idx, path); + *out_xattrs = g_variant_get_child_value (xattr_table, xattr_idx); + return TRUE; +} diff --git a/src/libpriv/rpmostree-jigdo-assembler.h b/src/libpriv/rpmostree-jigdo-assembler.h new file mode 100644 index 00000000..64d3c45d --- /dev/null +++ b/src/libpriv/rpmostree-jigdo-assembler.h @@ -0,0 +1,70 @@ +/* -*- mode: C; c-file-style: "gnu"; indent-tabs-mode: nil; -*- + * + * Copyright (C) 2015 Colin Walters + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published + * by the Free Software Foundation; either version 2 of the licence or (at + * your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General + * Public License along with this library; if not, write to the + * Free Software Foundation, Inc., 59 Temple Place, Suite 330, + * Boston, MA 02111-1307, USA. + */ + +#pragma once + +#include "libglnx.h" +#include "rpmostree-jigdo-core.h" + +typedef struct RpmOstreeJigdoAssembler RpmOstreeJigdoAssembler; + +#define RPMOSTREE_TYPE_JIGDO_ASSEMBLER (rpmostree_jigdo_assembler_get_type ()) +#define RPMOSTREE_JIGDO_ASSEMBLER(inst) (G_TYPE_CHECK_INSTANCE_CAST ((inst), RPMOSTREE_TYPE_JIGDO_ASSEMBLER, RpmOstreeJigdoAssembler)) +#define RPMOSTREE_IS_JIGDO_ASSEMBLER(inst) (G_TYPE_CHECK_INSTANCE_TYPE ((inst), RPMOSTREE_TYPE_JIGDO_ASSEMBLER)) + +GType rpmostree_jigdo_assembler_get_type (void); + +G_DEFINE_AUTOPTR_CLEANUP_FUNC (RpmOstreeJigdoAssembler, g_object_unref) + +RpmOstreeJigdoAssembler* +rpmostree_jigdo_assembler_new_take_fd (int *fd, + DnfPackage *pkg, /* for metadata */ + GError **error); + +gboolean +rpmostree_jigdo_assembler_read_meta (RpmOstreeJigdoAssembler *jigdo, + char **out_checksum, + GVariant **commit, + GVariant **detached_meta, + GVariant **pkgs, + GCancellable *cancellable, + GError **error); + +gboolean +rpmostree_jigdo_assembler_write_new_objects (RpmOstreeJigdoAssembler *jigdo, + OstreeRepo *repo, + GCancellable *cancellable, + GError **error); + + +GVariant * rpmostree_jigdo_assembler_get_xattr_table (RpmOstreeJigdoAssembler *self); + +gboolean +rpmostree_jigdo_assembler_next_xattrs (RpmOstreeJigdoAssembler *self, + GVariant **out_objid_to_xattrs, + GCancellable *cancellable, + GError **error); + +gboolean +rpmostree_jigdo_assembler_xattr_lookup (GVariant *xattr_table, + const char *path, + GVariant *xattrs, + GVariant **out_xattrs, + GError **error); diff --git a/src/libpriv/rpmostree-jigdo-core.h b/src/libpriv/rpmostree-jigdo-core.h new file mode 100644 index 00000000..3b0aee91 --- /dev/null +++ b/src/libpriv/rpmostree-jigdo-core.h @@ -0,0 +1,77 @@ +/* -*- mode: C; c-file-style: "gnu"; indent-tabs-mode: nil; -*- + * + * Copyright (C) 2017 Colin Walters + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published + * by the Free Software Foundation; either version 2 of the licence or (at + * your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General + * Public License along with this library; if not, write to the + * Free Software Foundation, Inc., 59 Temple Place, Suite 330, + * Boston, MA 02111-1307, USA. + */ + +#pragma once + +#include + +#include "libglnx.h" +#include +#include + +/* An OIRPM is structured as an ordered set of files/directories; we use numeric + * prefixes to ensure ordering. Most of the files are in GVariant format. + * + * An OIRPM starts with the OSTree commit object and its detached metadata, + * so that can be GPG verified first - if that fails, we can then cleanly + * abort. + * + * Next, we have the "jigdo set" - the NEVRAs + repodata checksum of the + * RPM packages we need. So during client side processing, downloads + * can be initiated for those while we continue to process the OIRPM. + * + * The dirmeta/dirtree objects that are referenced by the commit follow. + * + * A special optimization is made for "content-identical" new objects, + * such as the initramfs right now which unfortunately has separate + * SELinux labels and hence different object checksum. + * + * The pure added content objects follow - content objects which won't be + * generated when we import the packages. One interesting detail is right now + * this includes the /usr/lib/tmpfiles.d/pkg-foo.conf objects that we generate + * server side, because we don't generate that client side in jigdo mode. + * + * Finally, we have the xattr data, which is mostly in support of SELinux + * labeling (note this is done on the server side still). In order to + * dedup content, we have an xattr "string table" which is just an array + * of xattrs; then there is a GVariant for each package which contains + * a mapping of "objid" to an unsigned integer index into the xattr table. + * The "objid" can either be a full path, or a basename if that basename is + * unique inside a particular package. + */ + +/* Use a numeric prefix to ensure predictable ordering */ +#define RPMOSTREE_JIGDO_COMMIT_DIR "00commit" +#define RPMOSTREE_JIGDO_PKGS "01pkgs" +#define RPMOSTREE_JIGDO_PKGS_VARIANT_FORMAT (G_VARIANT_TYPE ("a(stssss)")) // NEVRA,repodata checksum +#define RPMOSTREE_JIGDO_DIRMETA_DIR "02dirmeta" +#define RPMOSTREE_JIGDO_DIRTREE_DIR "03dirtree" +//#define RPMOSTREE_JIGDO_NEW_PKGIDENT "04new-pkgident" +//#define RPMOSTREE_JIGDO_NEW_PKGIDENT_VARIANT_FORMAT (G_VARIANT_TYPE ("a{ua{s(sa(uuua(ayay)))}}")) // Map)>> +#define RPMOSTREE_JIGDO_NEW_CONTENTIDENT_DIR "04new-contentident" +#define RPMOSTREE_JIGDO_NEW_CONTENTIDENT_VARIANT_FORMAT (G_VARIANT_TYPE ("a(suuua(ayay))")) // checksum,uid,gid,mode,xattrs +#define RPMOSTREE_JIGDO_NEW_DIR "05new" +#define RPMOSTREE_JIGDO_XATTRS_DIR "06xattrs" +#define RPMOSTREE_JIGDO_XATTRS_TABLE "06xattrs/00table" +#define RPMOSTREE_JIGDO_XATTRS_PKG_DIR "06xattrs/pkg" + +#define RPMOSTREE_JIGDO_XATTRS_TABLE_VARIANT_FORMAT (G_VARIANT_TYPE ("aa(ayay)")) +/* NEVRA + xattr table */ +#define RPMOSTREE_JIGDO_XATTRS_PKG_VARIANT_FORMAT (G_VARIANT_TYPE ("a(su)")) diff --git a/src/libpriv/rpmostree-libarchive-input-stream.c b/src/libpriv/rpmostree-libarchive-input-stream.c new file mode 100644 index 00000000..1e58e814 --- /dev/null +++ b/src/libpriv/rpmostree-libarchive-input-stream.c @@ -0,0 +1,171 @@ +/* + * Copyright (C) 2011 Colin Walters + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General + * Public License along with this library; if not, write to the + * Free Software Foundation, Inc., 59 Temple Place, Suite 330, + * Boston, MA 02111-1307, USA. + */ + +#include "config.h" + + +#include +#include +#include "rpmostree-libarchive-input-stream.h" + +enum { + PROP_0, + PROP_ARCHIVE +}; + +G_DEFINE_TYPE (RpmOstreeLibarchiveInputStream, _rpm_ostree_libarchive_input_stream, G_TYPE_INPUT_STREAM) + +struct _RpmOstreeLibarchiveInputStreamPrivate { + struct archive *archive; +}; + +static void rpm_ostree_libarchive_input_stream_set_property (GObject *object, + guint prop_id, + const GValue *value, + GParamSpec *pspec); +static void rpm_ostree_libarchive_input_stream_get_property (GObject *object, + guint prop_id, + GValue *value, + GParamSpec *pspec); +static gssize rpm_ostree_libarchive_input_stream_read (GInputStream *stream, + void *buffer, + gsize count, + GCancellable *cancellable, + GError **error); +static gboolean rpm_ostree_libarchive_input_stream_close (GInputStream *stream, + GCancellable *cancellable, + GError **error); + +static void +rpm_ostree_libarchive_input_stream_finalize (GObject *object) +{ + G_OBJECT_CLASS (_rpm_ostree_libarchive_input_stream_parent_class)->finalize (object); +} + +static void +_rpm_ostree_libarchive_input_stream_class_init (RpmOstreeLibarchiveInputStreamClass *klass) +{ + GObjectClass *gobject_class = G_OBJECT_CLASS (klass); + GInputStreamClass *stream_class = G_INPUT_STREAM_CLASS (klass); + + g_type_class_add_private (klass, sizeof (RpmOstreeLibarchiveInputStreamPrivate)); + + gobject_class->get_property = rpm_ostree_libarchive_input_stream_get_property; + gobject_class->set_property = rpm_ostree_libarchive_input_stream_set_property; + gobject_class->finalize = rpm_ostree_libarchive_input_stream_finalize; + + stream_class->read_fn = rpm_ostree_libarchive_input_stream_read; + stream_class->close_fn = rpm_ostree_libarchive_input_stream_close; + + /** + * RpmOstreeLibarchiveInputStream:archive: + * + * The archive that the stream reads from. + */ + g_object_class_install_property (gobject_class, + PROP_ARCHIVE, + g_param_spec_pointer ("archive", + "", "", + G_PARAM_READWRITE | + G_PARAM_CONSTRUCT_ONLY | + G_PARAM_STATIC_STRINGS)); + +} + +static void +rpm_ostree_libarchive_input_stream_set_property (GObject *object, + guint prop_id, + const GValue *value, + GParamSpec *pspec) +{ + RpmOstreeLibarchiveInputStream *self = RPM_OSTREE_LIBARCHIVE_INPUT_STREAM (object); + switch (prop_id) + { + case PROP_ARCHIVE: + self->priv->archive = g_value_get_pointer (value); + break; + default: + G_OBJECT_WARN_INVALID_PROPERTY_ID (object, prop_id, pspec); + break; + } +} + +static void +rpm_ostree_libarchive_input_stream_get_property (GObject *object, + guint prop_id, + GValue *value, + GParamSpec *pspec) +{ + RpmOstreeLibarchiveInputStream *self = RPM_OSTREE_LIBARCHIVE_INPUT_STREAM (object); + + switch (prop_id) + { + case PROP_ARCHIVE: + g_value_set_pointer (value, self->priv->archive); + break; + default: + G_OBJECT_WARN_INVALID_PROPERTY_ID (object, prop_id, pspec); + } +} + +static void +_rpm_ostree_libarchive_input_stream_init (RpmOstreeLibarchiveInputStream *self) +{ + self->priv = G_TYPE_INSTANCE_GET_PRIVATE (self, + RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM, + RpmOstreeLibarchiveInputStreamPrivate); + +} + +GInputStream * +_rpm_ostree_libarchive_input_stream_new (struct archive *a) +{ + return G_INPUT_STREAM (g_object_new (RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM, + "archive", a, NULL)); +} + +static gssize +rpm_ostree_libarchive_input_stream_read (GInputStream *stream, + void *buffer, + gsize count, + GCancellable *cancellable, + GError **error) +{ + RpmOstreeLibarchiveInputStream *self = RPM_OSTREE_LIBARCHIVE_INPUT_STREAM (stream); + + if (g_cancellable_set_error_if_cancelled (cancellable, error)) + return -1; + + gssize res = archive_read_data (self->priv->archive, buffer, count); + if (res < 0) + { + g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED, + "%s", archive_error_string (self->priv->archive)); + } + + return res; +} + +static gboolean +rpm_ostree_libarchive_input_stream_close (GInputStream *stream, + GCancellable *cancellable, + GError **error) +{ + return TRUE; +} diff --git a/src/libpriv/rpmostree-libarchive-input-stream.h b/src/libpriv/rpmostree-libarchive-input-stream.h new file mode 100644 index 00000000..4009896f --- /dev/null +++ b/src/libpriv/rpmostree-libarchive-input-stream.h @@ -0,0 +1,66 @@ +/* + * Copyright (C) 2011 Colin Walters + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General + * Public License along with this library; if not, write to the + * Free Software Foundation, Inc., 59 Temple Place, Suite 330, + * Boston, MA 02111-1307, USA. + * + * Author: Alexander Larsson + */ + +#pragma once + +#include +#include +#include + +G_BEGIN_DECLS + +#define RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM (_rpm_ostree_libarchive_input_stream_get_type ()) +#define RPM_OSTREE_LIBARCHIVE_INPUT_STREAM(o) (G_TYPE_CHECK_INSTANCE_CAST ((o), RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM, RpmOstreeLibarchiveInputStream)) +#define RPM_OSTREE_LIBARCHIVE_INPUT_STREAM_CLASS(k) (G_TYPE_CHECK_CLASS_CAST((k), RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM, RpmOstreeLibarchiveInputStreamClass)) +#define RPM_OSTREE_IS_LIBARCHIVE_INPUT_STREAM(o) (G_TYPE_CHECK_INSTANCE_TYPE ((o), RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM)) +#define RPM_OSTREE_IS_LIBARCHIVE_INPUT_STREAM_CLASS(k) (G_TYPE_CHECK_CLASS_TYPE ((k), RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM)) +#define RPM_OSTREE_LIBARCHIVE_INPUT_STREAM_GET_CLASS(o) (G_TYPE_INSTANCE_GET_CLASS ((o), RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM, RpmOstreeLibarchiveInputStreamClass)) + +typedef struct _RpmOstreeLibarchiveInputStream RpmOstreeLibarchiveInputStream; +typedef struct _RpmOstreeLibarchiveInputStreamClass RpmOstreeLibarchiveInputStreamClass; +typedef struct _RpmOstreeLibarchiveInputStreamPrivate RpmOstreeLibarchiveInputStreamPrivate; + +struct _RpmOstreeLibarchiveInputStream +{ + GInputStream parent_instance; + + /*< private >*/ + RpmOstreeLibarchiveInputStreamPrivate *priv; +}; + +struct _RpmOstreeLibarchiveInputStreamClass +{ + GInputStreamClass parent_class; + + /*< private >*/ + /* Padding for future expansion */ + void (*_g_reserved1) (void); + void (*_g_reserved2) (void); + void (*_g_reserved3) (void); + void (*_g_reserved4) (void); + void (*_g_reserved5) (void); +}; + +GType _rpm_ostree_libarchive_input_stream_get_type (void) G_GNUC_CONST; + +GInputStream * _rpm_ostree_libarchive_input_stream_new (struct archive *a); + +G_END_DECLS diff --git a/src/libpriv/rpmostree-util.c b/src/libpriv/rpmostree-util.c index c4621096..e6db92c2 100644 --- a/src/libpriv/rpmostree-util.c +++ b/src/libpriv/rpmostree-util.c @@ -837,3 +837,54 @@ rpmostree_diff_print (GPtrArray *removed, g_print (" %s\n", nevra); } } + +/* Copy of ot_variant_bsearch_str() from libostree + * @array: A GVariant array whose first element must be a string + * @str: Search for this string + * @out_pos: Output position + * + * Binary search in a GVariant array, which must be of the form 'a(s...)', + * where '...' may be anything. The array elements must be sorted. + * + * Returns: %TRUE iff found + */ +gboolean +rpmostree_variant_bsearch_str (GVariant *array, + const char *str, + int *out_pos) +{ + const gsize n = g_variant_n_children (array); + if (n == 0) + return FALSE; + + gsize imax = n - 1; + gsize imin = 0; + gsize imid = -1; + while (imax >= imin) + { + const char *cur; + + imid = (imin + imax) / 2; + + g_autoptr(GVariant) child = g_variant_get_child_value (array, imid); + g_variant_get_child (child, 0, "&s", &cur, NULL); + + int cmp = strcmp (cur, str); + if (cmp < 0) + imin = imid + 1; + else if (cmp > 0) + { + if (imid == 0) + break; + imax = imid - 1; + } + else + { + *out_pos = imid; + return TRUE; + } + } + + *out_pos = imid; + return FALSE; +} diff --git a/src/libpriv/rpmostree-util.h b/src/libpriv/rpmostree-util.h index 0669d3ef..e2eea347 100644 --- a/src/libpriv/rpmostree-util.h +++ b/src/libpriv/rpmostree-util.h @@ -172,3 +172,8 @@ rpmostree_repo_auto_transaction_start (RpmOstreeRepoAutoTransaction *autotxn return TRUE; } G_DEFINE_AUTO_CLEANUP_CLEAR_FUNC (RpmOstreeRepoAutoTransaction, rpmostree_repo_auto_transaction_cleanup) + +gboolean +rpmostree_variant_bsearch_str (GVariant *array, + const char *str, + int *out_pos); diff --git a/tests/compose-tests/test-jigdo.sh b/tests/compose-tests/test-jigdo.sh new file mode 100755 index 00000000..af2f415f --- /dev/null +++ b/tests/compose-tests/test-jigdo.sh @@ -0,0 +1,97 @@ +#!/bin/bash + +set -xeuo pipefail + +dn=$(cd $(dirname $0) && pwd) +. ${dn}/libcomposetest.sh +. ${dn}/../common/libtest.sh + +prepare_compose_test "jigdo" +# Add a local rpm-md repo so we can mutate local test packages +pyappendjsonmember "repos" '["test-repo"]' +build_rpm test-pkg \ + files "/usr/bin/test-pkg" \ + install "mkdir -p %{buildroot}/usr/bin && echo localpkg data > %{buildroot}/usr/bin/test-pkg" +# The test suite writes to pwd, but we need repos in composedata +# Also we need to disable gpgcheck +echo gpgcheck=0 >> yumrepo.repo +ln yumrepo.repo composedata/test-repo.repo +pyappendjsonmember "packages" '["test-pkg"]' +# Need unified core for this, as well as a cachedir +mkdir cache +runcompose --ex-unified-core --cachedir $(pwd)/cache --add-metadata-string version=42.0 +npkgs=$(rpm-ostree --repo=${repobuild} db list ${treeref} |grep -v '^ostree commit' | wc -l) +echo "npkgs=${npkgs}" +rpm-ostree --repo=${repobuild} db list ${treeref} test-pkg >test-pkg-list.txt +assert_file_has_content test-pkg-list.txt 'test-pkg-1.0-1.x86_64' + +rev=$(ostree --repo=${repobuild} rev-parse ${treeref}) +mkdir jigdo-output +do_commit2jigdo() { + targetrev=$1 + rpm-ostree ex commit2jigdo --repo=repo-build --pkgcache-repo cache/pkgcache-repo ${targetrev} $(pwd)/composedata/fedora-atomic-host-oirpm.spec $(pwd)/jigdo-output + (cd jigdo-output && createrepo_c .) +} +do_commit2jigdo ${rev} +find jigdo-output -name '*.rpm' | tee rpms.txt +assert_file_has_content rpms.txt 'fedora-atomic-host-42.0.*x86_64' + +ostree --repo=jigdo-unpack-repo init --mode=bare-user +echo 'fsync=false' >> jigdo-unpack-repo/config +# Technically this isn't part of composedata but eh +cat > composedata/jigdo-test.repo < 0 +assert_file_has_content jigdo2commit-out.txt '[1-9][0-9]* packages to import' +ostree --repo=jigdo-unpack-repo rev-parse ${rev} +ostree --repo=jigdo-unpack-repo fsck +ostree --repo=jigdo-unpack-repo refs > jigdo-refs.txt +assert_file_has_content jigdo-refs.txt 'rpmostree/pkg/test-pkg/1.0-1.x86__64' + +echo "ok jigdo ♲📦 fresh assembly" + +origrev=${rev} +unset rev +# Update test-pkg +build_rpm test-pkg \ + version 1.1 \ + files "/usr/bin/test-pkg" \ + install "mkdir -p %{buildroot}/usr/bin && echo localpkg data 1.1 > %{buildroot}/usr/bin/test-pkg" +# Also add an entirely new package +build_rpm test-newpkg \ + files "/usr/bin/test-newpkg" \ + install "mkdir -p %{buildroot}/usr/bin && echo new localpkg data > %{buildroot}/usr/bin/test-newpkg" +pyappendjsonmember "packages" '["test-newpkg"]' +runcompose --ex-unified-core --cachedir $(pwd)/cache --add-metadata-string version=42.1 +newrev=$(ostree --repo=${repobuild} rev-parse ${treeref}) +rpm-ostree --repo=${repobuild} db list ${treeref} test-newpkg >test-newpkg-list.txt +assert_file_has_content test-newpkg-list.txt 'test-newpkg-1.0-1.x86_64' + +# Jigdo version 42.1 +do_commit2jigdo ${newrev} +find jigdo-output -name '*.rpm' | tee rpms.txt +assert_file_has_content rpms.txt 'fedora-atomic-host-42.1.*x86_64' + +# And pull it; we should download the newer version by default +do_jigdo2commit +# Now we should only download 2 packages +assert_file_has_content jigdo2commit-out.txt '2 packages to import' +for x in ${origrev} ${newrev}; do + ostree --repo=jigdo-unpack-repo rev-parse ${x} +done +ostree --repo=jigdo-unpack-repo fsck +ostree --repo=jigdo-unpack-repo refs > jigdo-refs.txt +# We should have both refs; GC will be handled by the sysroot upgrader +# via deployments, same way it is for pkg layering. +assert_file_has_content jigdo-refs.txt 'rpmostree/pkg/test-pkg/1.0-1.x86__64' +assert_file_has_content jigdo-refs.txt 'rpmostree/pkg/test-pkg/1.1-1.x86__64' + +echo "ok jigdo ♲📦 update!" diff --git a/tests/composedata/fedora-atomic-host-oirpm.spec b/tests/composedata/fedora-atomic-host-oirpm.spec new file mode 100644 index 00000000..4005323c --- /dev/null +++ b/tests/composedata/fedora-atomic-host-oirpm.spec @@ -0,0 +1,23 @@ +# The canonical version of this is in https://pagure.io/fedora-atomic +# Suppress most build root processing we are just carrying +# binary data. +%global __os_install_post /usr/lib/rpm/brp-compress %{nil} +Name: fedora-atomic-host +Version: %{ostree_version} +Release: 1%{?dist} +Summary: Image (rpm-ostree jigdo) for Fedora Atomic Host +License: MIT + +%description +%{summary} + +%prep + +%build + +%install +mkdir -p %{buildroot}%{_prefix}/lib/ostree-jigdo/%{name} +for x in *; do mv ${x} %{buildroot}%{_prefix}/lib/ostree-jigdo/%{name}; done + +%files +%{_prefix}/lib/ostree-jigdo/%{name}