Introduce experimental "rpm-ostree jigdo"
Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
This commit is contained in:
Colin Walters
Atomic Bot
parent
42282c0a84
commit
694b798c73
@ -45,6 +45,9 @@ librpmostreepriv_la_SOURCES = \
|
||||
src/libpriv/rpmostree-rpm-util.h \
|
||||
src/libpriv/rpmostree-importer.c \
|
||||
src/libpriv/rpmostree-importer.h \
|
||||
src/libpriv/rpmostree-jigdo-assembler.c \
|
||||
src/libpriv/rpmostree-jigdo-assembler.h \
|
||||
src/libpriv/rpmostree-jigdo-core.h \
|
||||
src/libpriv/rpmostree-unpacker-core.c \
|
||||
src/libpriv/rpmostree-unpacker-core.h \
|
||||
src/libpriv/rpmostree-output.c \
|
||||
@ -55,6 +58,8 @@ librpmostreepriv_la_SOURCES = \
|
||||
src/libpriv/rpmostree-editor.h \
|
||||
src/libpriv/libsd-locale-util.c \
|
||||
src/libpriv/libsd-locale-util.h \
|
||||
src/libpriv/rpmostree-libarchive-input-stream.c \
|
||||
src/libpriv/rpmostree-libarchive-input-stream.h \
|
||||
$(NULL)
|
||||
|
||||
librpmostreepriv_la_CFLAGS = \
|
||||
|
||||
@ -37,6 +37,8 @@ rpm_ostree_SOURCES = src/app/main.c \
|
||||
src/app/rpmostree-builtin-status.c \
|
||||
src/app/rpmostree-builtin-ex.c \
|
||||
src/app/rpmostree-builtin-container.c \
|
||||
src/app/rpmostree-ex-builtin-commit2jigdo.c \
|
||||
src/app/rpmostree-ex-builtin-jigdo2commit.c \
|
||||
src/app/rpmostree-builtin-db.c \
|
||||
src/app/rpmostree-builtin-start-daemon.c \
|
||||
src/app/rpmostree-db-builtin-diff.c \
|
||||
|
||||
97
design/jigdo.md
Normal file
97
design/jigdo.md
Normal file
@ -0,0 +1,97 @@
|
||||
Introducing rpm-ostree jigdo
|
||||
--------
|
||||
|
||||
In the rpm-ostree project, we're blending an image system (libostree)
|
||||
with a package system (libdnf). The goal is to gain the
|
||||
advantages of both. However, the dual nature also brings overhead;
|
||||
this proposal aims to reduce some of that by adding a new "jigdo"
|
||||
model to rpm-ostree that makes more operations use the libdnf side.
|
||||
|
||||
To do this, we're reviving an old idea: The [http://atterer.org/jigdo/](Jigdo)
|
||||
approach to reassembling large "images" by downloading component packages. (We're
|
||||
not using the code, just the idea).
|
||||
|
||||
In this approach, we're still maintaining the "image" model of libostree. When
|
||||
one deploys an OSTree commit, it will reliably be bit-for-bit identical. It will
|
||||
have a checksum and a version number. There will be *no* dependency resolution
|
||||
on the client by default, etc.
|
||||
|
||||
The change is that we always use libdnf to download RPM packages as they exist
|
||||
today, storing any additional data inside a new "ostree-image" RPM. In this
|
||||
proposal, rather than using ostree branches, the system tracks an "ostree-image"
|
||||
RPM that behaves a bit like a "metapackage".
|
||||
|
||||
Why?
|
||||
----
|
||||
|
||||
The "dual" nature of the system appears in many ways; users and administrators
|
||||
effectively need to understand and manage both systems.
|
||||
|
||||
An example is when one needs to mirror content. While libostree does support
|
||||
mirroring, and projects like Pulp make use of it, support is not as widespread
|
||||
as mirroring for RPM. And mirroring is absolutely critical for many
|
||||
organizations that don't want to depend on Internet availability.
|
||||
|
||||
Related to this is the mapping of libostree "branches" and rpm-md repos. In
|
||||
Fedora we offer multiple branches for Atomic Host, such as
|
||||
`fedora/27/x86_64/atomic-host` as well as
|
||||
`fedora/27/x86_64/testing/atomic-host`, where the latter is equivalent to `yum
|
||||
--enablerepo=updates-testing update`. In many ways, I believe the way we're
|
||||
exposing as OSTree branches is actually nicer - it's very clear when you're on
|
||||
the testing branch.
|
||||
|
||||
However, it's also very *different* from the yum/dnf model. Once package
|
||||
layering is involved (and for a lot of small scale use cases it will be,
|
||||
particularly for desktop systems), the libostree side is something that many
|
||||
users and administrators have to learn *in addition* to their previous "mental model"
|
||||
of how the libdnf/yum/rpm side works with `/etc/yum.repos.d` etc.
|
||||
|
||||
Finally, for network efficiency; on the wire, libostree has two formats, and the
|
||||
intention is that most updates hit the network-efficient static delta path, but
|
||||
there are various cases where this doesn't happen, such as if one is skipping a
|
||||
few updates, or today when rebasing between branches. In practice, as soon as
|
||||
one involves libdnf, the repodata is already large enough that it's not worth
|
||||
trying to optimize fetching content over simply redownloading changed RPMs.
|
||||
|
||||
(Aside: people doing custom systems tend to like the network efficiency of "pure
|
||||
ostree" where one doesn't pay the "repodata cost" and we will continue to
|
||||
support that.)
|
||||
|
||||
How?
|
||||
----
|
||||
|
||||
We've already stated that a primary design goal is to preserve the "image"
|
||||
functionality by default. Further, let's assume that we have an OSTree commit,
|
||||
and we want to point it at a set of RPMs to use as the jigdo source. The source
|
||||
OSTree commit can have modified, added to, or removed data from the RPM set, and
|
||||
we will support that. Examples of additional data are the initramfs and RPM
|
||||
database.
|
||||
|
||||
We're hence treating the RPM set as just data blobs; again, no dependency
|
||||
resolution, `%post` scripts or the like will be executed on the client. Or again
|
||||
to state this more strongly, installation will still result in an OSTree commit
|
||||
with checksum that is bit-for-bit identical.
|
||||
|
||||
A simple approach is to scan over the set of files in the RPMs, then the set
|
||||
of files in the OSTree commit, and add RPMs which contain files in the OSTree
|
||||
commit to our "jigdo set".
|
||||
|
||||
However, a major complication is SELinux labeling. It turns out that in a lot of
|
||||
cases, doing SELinux labeling is expensive; there are thousands of regular
|
||||
expressions involved. However, RPM packages themselves don't contain labels;
|
||||
instead the labeling is stored in the `selinux-policy-targeted` package, and
|
||||
further complicating things is that there are also other packages that add
|
||||
labeling configuration such as `container-selinux`. In other words there's a
|
||||
circular dependency: packages have labels, but labels are contained in packages.
|
||||
We go to great lengths to handle this in rpm-ostree for package layering, and we
|
||||
need to do the same for jigdo.
|
||||
|
||||
We can address this by having our OIRPM contain a mapping of (package, file
|
||||
path) to a set of extended attributes (including the key `security.selinux`
|
||||
one).
|
||||
|
||||
At this point, if we add in the new objects such as the metadata objects from
|
||||
the OSTree commit and all new content objects that aren't part of a package,
|
||||
we'll have our OIRPM. (There is
|
||||
some [further complexity](https://pagure.io/fedora-atomic/issue/94) around
|
||||
handling the initramfs and SELinux labeling that we'll omit for now).
|
||||
@ -30,6 +30,10 @@ static RpmOstreeCommand ex_subcommands[] = {
|
||||
"Assemble local unprivileged containers", rpmostree_builtin_container },
|
||||
{ "kargs", 0,
|
||||
"Query or Modify the kernel arguments", rpmostree_ex_builtin_kargs },
|
||||
{ "commit2jigdo", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD,
|
||||
"Convert an OSTree commit into an rpm-ostree jigdo", rpmostree_ex_builtin_commit2jigdo },
|
||||
{ "jigdo2commit", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD,
|
||||
"Convert an rpm-ostree jigdo into an OSTree commit", rpmostree_ex_builtin_jigdo2commit },
|
||||
{ NULL, 0, NULL, NULL }
|
||||
};
|
||||
|
||||
|
||||
@ -30,6 +30,8 @@ gboolean rpmostree_compose_builtin_tree (int argc, char **argv, RpmOstreeCommand
|
||||
gboolean rpmostree_compose_builtin_install (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error);
|
||||
gboolean rpmostree_compose_builtin_postprocess (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error);
|
||||
gboolean rpmostree_compose_builtin_commit (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error);
|
||||
gboolean rpmostree_compose_builtin_commit2jigdo (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error);
|
||||
gboolean rpmostree_compose_builtin_jigdo2commit (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error);
|
||||
|
||||
G_END_DECLS
|
||||
|
||||
|
||||
1228
src/app/rpmostree-ex-builtin-commit2jigdo.c
Normal file
1228
src/app/rpmostree-ex-builtin-commit2jigdo.c
Normal file
File diff suppressed because it is too large
Load Diff
368
src/app/rpmostree-ex-builtin-jigdo2commit.c
Normal file
368
src/app/rpmostree-ex-builtin-jigdo2commit.c
Normal file
@ -0,0 +1,368 @@
|
||||
/* -*- mode: C; c-file-style: "gnu"; indent-tabs-mode: nil; -*-
|
||||
*
|
||||
* Copyright (C) 2017 Colin Walters <walters@verbum.org>
|
||||
*
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU Lesser General Public License as published
|
||||
* by the Free Software Foundation; either version 2 of the licence or (at
|
||||
* your option) any later version.
|
||||
*
|
||||
* This library is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
* Lesser General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU Lesser General
|
||||
* Public License along with this library; if not, write to the
|
||||
* Free Software Foundation, Inc., 59 Temple Place, Suite 330,
|
||||
* Boston, MA 02111-1307, USA.
|
||||
*/
|
||||
|
||||
#include "config.h"
|
||||
|
||||
#include <string.h>
|
||||
#include <glib-unix.h>
|
||||
#include <gio/gunixoutputstream.h>
|
||||
#include <libdnf/libdnf.h>
|
||||
#include <sys/mount.h>
|
||||
#include <stdio.h>
|
||||
#include <libglnx.h>
|
||||
#include <rpm/rpmmacro.h>
|
||||
|
||||
#include "rpmostree-ex-builtins.h"
|
||||
#include "rpmostree-util.h"
|
||||
#include "rpmostree-core.h"
|
||||
#include "rpmostree-jigdo-assembler.h"
|
||||
#include "rpmostree-postprocess.h"
|
||||
#include "rpmostree-passwd-util.h"
|
||||
#include "rpmostree-libbuiltin.h"
|
||||
#include "rpmostree-rpm-util.h"
|
||||
|
||||
#include "libglnx.h"
|
||||
|
||||
static char *opt_repo;
|
||||
static char *opt_rpmmd_reposdir;
|
||||
static char *opt_releasever;
|
||||
static char **opt_enable_rpmmdrepo;
|
||||
static char *opt_oirpm_version;
|
||||
|
||||
static GOptionEntry jigdo2commit_option_entries[] = {
|
||||
{ "repo", 0, 0, G_OPTION_ARG_STRING, &opt_repo, "OSTree repo", "REPO" },
|
||||
{ "rpmmd-reposd", 'd', 0, G_OPTION_ARG_STRING, &opt_rpmmd_reposdir, "Path to yum.repos.d (rpmmd) config directory", "PATH" },
|
||||
{ "enablerepo", 'e', 0, G_OPTION_ARG_STRING_ARRAY, &opt_enable_rpmmdrepo, "Enable rpm-md repo with id ID", "ID" },
|
||||
{ "releasever", 0, 0, G_OPTION_ARG_STRING, &opt_releasever, "Value for $releasever", "RELEASEVER" },
|
||||
{ "oirpm-version", 'V', 0, G_OPTION_ARG_STRING, &opt_oirpm_version, "Use this specific version of OIRPM", "VERSION" },
|
||||
{ NULL }
|
||||
};
|
||||
|
||||
typedef struct {
|
||||
OstreeRepo *repo;
|
||||
GLnxTmpDir tmpd;
|
||||
RpmOstreeContext *ctx;
|
||||
} RpmOstreeJigdo2CommitContext;
|
||||
|
||||
static void
|
||||
rpm_ostree_jigdo2commit_context_free (RpmOstreeJigdo2CommitContext *ctx)
|
||||
{
|
||||
g_clear_object (&ctx->repo);
|
||||
(void) glnx_tmpdir_delete (&ctx->tmpd, NULL, NULL);
|
||||
g_free (ctx);
|
||||
}
|
||||
G_DEFINE_AUTOPTR_CLEANUP_FUNC(RpmOstreeJigdo2CommitContext, rpm_ostree_jigdo2commit_context_free)
|
||||
|
||||
/* Initialize a context for converting a jigdo to a commit.
|
||||
*/
|
||||
static gboolean
|
||||
rpm_ostree_jigdo2commit_context_new (RpmOstreeJigdo2CommitContext **out_context,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
g_autoptr(RpmOstreeJigdo2CommitContext) self = g_new0 (RpmOstreeJigdo2CommitContext, 1);
|
||||
|
||||
self->repo = ostree_repo_open_at (AT_FDCWD, opt_repo, cancellable, error);
|
||||
if (!self->repo)
|
||||
return FALSE;
|
||||
|
||||
/* Our workdir lives in the repo for command line testing */
|
||||
if (!glnx_mkdtempat (ostree_repo_get_dfd (self->repo),
|
||||
"tmp/rpmostree-jigdo-XXXXXX", 0700, &self->tmpd, error))
|
||||
return FALSE;
|
||||
|
||||
self->ctx = rpmostree_context_new_tree (self->tmpd.fd, self->repo, cancellable, error);
|
||||
if (!self->ctx)
|
||||
return FALSE;
|
||||
|
||||
DnfContext *dnfctx = rpmostree_context_get_dnf (self->ctx);
|
||||
|
||||
if (opt_rpmmd_reposdir)
|
||||
dnf_context_set_repo_dir (dnfctx, opt_rpmmd_reposdir);
|
||||
|
||||
*out_context = g_steal_pointer (&self);
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
static DnfPackage *
|
||||
query_nevra (DnfContext *dnfctx,
|
||||
const char *name,
|
||||
guint64 epoch,
|
||||
const char *version,
|
||||
const char *release,
|
||||
const char *arch,
|
||||
GError **error)
|
||||
{
|
||||
hy_autoquery HyQuery query = hy_query_create (dnf_context_get_sack (dnfctx));
|
||||
hy_query_filter (query, HY_PKG_NAME, HY_EQ, name);
|
||||
hy_query_filter_num (query, HY_PKG_EPOCH, HY_EQ, epoch);
|
||||
hy_query_filter (query, HY_PKG_VERSION, HY_EQ, version);
|
||||
hy_query_filter (query, HY_PKG_RELEASE, HY_EQ, release);
|
||||
hy_query_filter (query, HY_PKG_ARCH, HY_EQ, arch);
|
||||
g_autoptr(GPtrArray) pkglist = hy_query_run (query);
|
||||
if (pkglist->len == 0)
|
||||
return glnx_null_throw (error, "Failed to find package '%s'", name);
|
||||
return g_object_ref (pkglist->pdata[0]);
|
||||
}
|
||||
|
||||
static gboolean
|
||||
commit_and_print (RpmOstreeJigdo2CommitContext *self,
|
||||
RpmOstreeRepoAutoTransaction *txn,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
OstreeRepoTransactionStats stats;
|
||||
if (!ostree_repo_commit_transaction (self->repo, &stats, cancellable, error))
|
||||
return FALSE;
|
||||
txn->initialized = FALSE;
|
||||
|
||||
g_print ("Metadata Total: %u\n", stats.metadata_objects_total);
|
||||
g_print ("Metadata Written: %u\n", stats.metadata_objects_written);
|
||||
g_print ("Content Total: %u\n", stats.content_objects_total);
|
||||
g_print ("Content Written: %u\n", stats.content_objects_written);
|
||||
g_print ("Content Bytes Written: %" G_GUINT64_FORMAT "\n", stats.content_bytes_written);
|
||||
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
static int
|
||||
compare_pkgs_reverse (gconstpointer ap,
|
||||
gconstpointer bp)
|
||||
{
|
||||
DnfPackage **a = (gpointer)ap;
|
||||
DnfPackage **b = (gpointer)bp;
|
||||
return dnf_package_cmp (*b, *a); // Reverse
|
||||
}
|
||||
|
||||
static gboolean
|
||||
impl_jigdo2commit (RpmOstreeJigdo2CommitContext *self,
|
||||
const char *oirpm_name,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
g_autoptr(GKeyFile) tsk = g_key_file_new ();
|
||||
|
||||
if (opt_releasever)
|
||||
g_key_file_set_string (tsk, "tree", "releasever", opt_releasever);
|
||||
if (opt_enable_rpmmdrepo)
|
||||
g_key_file_set_string_list (tsk, "tree", "repos",
|
||||
(const char *const*)opt_enable_rpmmdrepo,
|
||||
g_strv_length (opt_enable_rpmmdrepo));
|
||||
g_autoptr(RpmOstreeTreespec) treespec = rpmostree_treespec_new_from_keyfile (tsk, error);
|
||||
if (!treespec)
|
||||
return FALSE;
|
||||
|
||||
|
||||
if (!rpmostree_context_setup (self->ctx, NULL, NULL, treespec, cancellable, error))
|
||||
return FALSE;
|
||||
if (!rpmostree_context_download_metadata (self->ctx, cancellable, error))
|
||||
return FALSE;
|
||||
|
||||
DnfContext *dnfctx = rpmostree_context_get_dnf (self->ctx);
|
||||
g_autoptr(DnfPackage) oirpm_pkg = NULL;
|
||||
{ hy_autoquery HyQuery query = hy_query_create (dnf_context_get_sack (dnfctx));
|
||||
if (opt_oirpm_version)
|
||||
{
|
||||
hy_query_filter (query, HY_PKG_NAME, HY_EQ, oirpm_name);
|
||||
hy_query_filter (query, HY_PKG_VERSION, HY_EQ, opt_oirpm_version);
|
||||
}
|
||||
else
|
||||
{
|
||||
hy_query_filter (query, HY_PKG_NAME, HY_EQ, oirpm_name);
|
||||
}
|
||||
g_autoptr(GPtrArray) pkglist = hy_query_run (query);
|
||||
if (pkglist->len == 0)
|
||||
return glnx_throw (error, "Failed to find jigdo OIRPM package '%s'", oirpm_name);
|
||||
g_ptr_array_sort (pkglist, compare_pkgs_reverse);
|
||||
if (pkglist->len > 1)
|
||||
{
|
||||
g_print ("%u oirpm matches\n", pkglist->len);
|
||||
}
|
||||
g_ptr_array_set_size (pkglist, 1);
|
||||
if (!rpmostree_context_set_packages (self->ctx, pkglist, cancellable, error))
|
||||
return FALSE;
|
||||
oirpm_pkg = g_object_ref (pkglist->pdata[0]);
|
||||
}
|
||||
|
||||
g_print ("oirpm: %s (%s)\n", dnf_package_get_nevra (oirpm_pkg),
|
||||
dnf_package_get_reponame (oirpm_pkg));
|
||||
|
||||
if (!rpmostree_context_download (self->ctx, cancellable, error))
|
||||
return FALSE;
|
||||
|
||||
glnx_fd_close int oirpm_fd = -1;
|
||||
if (!rpmostree_context_consume_package (self->ctx, oirpm_pkg, &oirpm_fd, error))
|
||||
return FALSE;
|
||||
|
||||
g_autoptr(RpmOstreeJigdoAssembler) jigdo = rpmostree_jigdo_assembler_new_take_fd (&oirpm_fd, oirpm_pkg, error);
|
||||
if (!jigdo)
|
||||
return FALSE;
|
||||
g_autofree char *checksum = NULL;
|
||||
g_autoptr(GVariant) commit = NULL;
|
||||
g_autoptr(GVariant) commit_meta = NULL;
|
||||
g_autoptr(GVariant) pkgs = NULL;
|
||||
if (!rpmostree_jigdo_assembler_read_meta (jigdo, &checksum, &commit, &commit_meta, &pkgs,
|
||||
cancellable, error))
|
||||
return FALSE;
|
||||
|
||||
g_print ("OSTree commit: %s\n", checksum);
|
||||
|
||||
{ OstreeRepoCommitState commitstate;
|
||||
gboolean has_commit;
|
||||
if (!ostree_repo_has_object (self->repo, OSTREE_OBJECT_TYPE_COMMIT, checksum,
|
||||
&has_commit, cancellable, error))
|
||||
return FALSE;
|
||||
if (has_commit)
|
||||
{
|
||||
if (!ostree_repo_load_commit (self->repo, checksum, NULL, &commitstate, error))
|
||||
return FALSE;
|
||||
if (!(commitstate & OSTREE_REPO_COMMIT_STATE_PARTIAL))
|
||||
{
|
||||
g_print ("Commit is already written, nothing to do\n");
|
||||
return TRUE; /* 🔚 Early return */
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
g_printerr ("TODO implement GPG verification\n");
|
||||
|
||||
g_auto(RpmOstreeRepoAutoTransaction) txn = { 0, };
|
||||
if (!rpmostree_repo_auto_transaction_start (&txn, self->repo, FALSE, cancellable, error))
|
||||
return FALSE;
|
||||
|
||||
if (!rpmostree_jigdo_assembler_write_new_objects (jigdo, self->repo, cancellable, error))
|
||||
return FALSE;
|
||||
|
||||
if (!commit_and_print (self, &txn, cancellable, error))
|
||||
return FALSE;
|
||||
|
||||
/* Download any packages we don't already have imported */
|
||||
g_autoptr(GPtrArray) pkgs_required = g_ptr_array_new_with_free_func (g_object_unref);
|
||||
const guint n_pkgs = g_variant_n_children (pkgs);
|
||||
for (guint i = 0; i < n_pkgs; i++)
|
||||
{
|
||||
const char *name, *version, *release, *architecture;
|
||||
const char *repodata_checksum;
|
||||
guint64 epoch;
|
||||
g_variant_get_child (pkgs, i, "(&st&s&s&s&s)",
|
||||
&name, &epoch, &version, &release, &architecture,
|
||||
&repodata_checksum);
|
||||
// TODO: use repodata checksum, but probably only if covered by the ostree
|
||||
// gpg sig?
|
||||
DnfPackage *pkg = query_nevra (dnfctx, name, epoch, version, release, architecture, error);
|
||||
// FIXME: We shouldn't require a package to be in the repos if we already
|
||||
// have it imported otherwise we'll break upgrades for ancient systems
|
||||
if (!pkg)
|
||||
return FALSE;
|
||||
g_ptr_array_add (pkgs_required, g_object_ref (pkg));
|
||||
}
|
||||
|
||||
g_print ("Jigdo from %u packages\n", pkgs_required->len);
|
||||
|
||||
if (!rpmostree_context_set_packages (self->ctx, pkgs_required, cancellable, error))
|
||||
return FALSE;
|
||||
|
||||
g_autoptr(GHashTable) pkgset_to_import = g_hash_table_new (NULL, NULL);
|
||||
{ g_autoptr(GPtrArray) pkgs_to_import = rpmostree_context_get_packages_to_import (self->ctx);
|
||||
for (guint i = 0; i < pkgs_to_import->len; i++)
|
||||
g_hash_table_add (pkgset_to_import, pkgs_to_import->pdata[i]);
|
||||
g_print ("%u packages to import\n", pkgs_to_import->len);
|
||||
}
|
||||
|
||||
g_autoptr(GHashTable) pkg_to_xattrs = g_hash_table_new_full (NULL, NULL,
|
||||
(GDestroyNotify)g_object_unref,
|
||||
(GDestroyNotify)g_variant_unref);
|
||||
|
||||
for (guint i = 0; i < pkgs_required->len; i++)
|
||||
{
|
||||
DnfPackage *pkg = pkgs_required->pdata[i];
|
||||
const gboolean should_import = g_hash_table_contains (pkgset_to_import, pkg);
|
||||
g_autoptr(GVariant) objid_to_xattrs = NULL;
|
||||
if (!rpmostree_jigdo_assembler_next_xattrs (jigdo, &objid_to_xattrs, cancellable, error))
|
||||
return FALSE;
|
||||
if (!objid_to_xattrs)
|
||||
return glnx_throw (error, "missing xattr entry: %s", dnf_package_get_name (pkg));
|
||||
if (!should_import)
|
||||
continue;
|
||||
g_hash_table_insert (pkg_to_xattrs, g_object_ref (pkg), g_steal_pointer (&objid_to_xattrs));
|
||||
}
|
||||
|
||||
if (!rpmostree_context_download (self->ctx, cancellable, error))
|
||||
return FALSE;
|
||||
g_autoptr(GVariant) xattr_table = rpmostree_jigdo_assembler_get_xattr_table (jigdo);
|
||||
if (!rpmostree_context_import_jigdo (self->ctx, xattr_table, pkg_to_xattrs,
|
||||
cancellable, error))
|
||||
return FALSE;
|
||||
|
||||
/* Write commitmeta/commit last since libostree doesn't expose an API to set
|
||||
* partial state right now.
|
||||
*/
|
||||
if (!ostree_repo_write_commit_detached_metadata (self->repo, checksum, commit_meta,
|
||||
cancellable, error))
|
||||
return FALSE;
|
||||
{ g_autofree guint8*csum = NULL;
|
||||
if (!ostree_repo_write_metadata (self->repo, OSTREE_OBJECT_TYPE_COMMIT,
|
||||
checksum, commit, &csum,
|
||||
cancellable, error))
|
||||
return FALSE;
|
||||
}
|
||||
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
int
|
||||
rpmostree_ex_builtin_jigdo2commit (int argc,
|
||||
char **argv,
|
||||
RpmOstreeCommandInvocation *invocation,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
g_autoptr(GOptionContext) context = g_option_context_new ("OIRPM");
|
||||
if (!rpmostree_option_context_parse (context,
|
||||
jigdo2commit_option_entries,
|
||||
&argc, &argv,
|
||||
invocation,
|
||||
cancellable,
|
||||
NULL, NULL, NULL, NULL,
|
||||
error))
|
||||
return EXIT_FAILURE;
|
||||
|
||||
if (argc != 2)
|
||||
{
|
||||
rpmostree_usage_error (context, "OIRPM name is required", error);
|
||||
return EXIT_FAILURE;
|
||||
}
|
||||
|
||||
if (!opt_repo)
|
||||
{
|
||||
rpmostree_usage_error (context, "--repo must be specified", error);
|
||||
return EXIT_FAILURE;
|
||||
}
|
||||
|
||||
const char *oirpm = argv[1];
|
||||
|
||||
g_autoptr(RpmOstreeJigdo2CommitContext) self = NULL;
|
||||
if (!rpm_ostree_jigdo2commit_context_new (&self, cancellable, error))
|
||||
return EXIT_FAILURE;
|
||||
if (!impl_jigdo2commit (self, oirpm, cancellable, error))
|
||||
return EXIT_FAILURE;
|
||||
|
||||
return EXIT_SUCCESS;
|
||||
}
|
||||
@ -34,6 +34,8 @@ BUILTINPROTO(unpack);
|
||||
BUILTINPROTO(livefs);
|
||||
BUILTINPROTO(override);
|
||||
BUILTINPROTO(kargs);
|
||||
BUILTINPROTO(commit2jigdo);
|
||||
BUILTINPROTO(jigdo2commit);
|
||||
|
||||
#undef BUILTINPROTO
|
||||
|
||||
|
||||
@ -864,6 +864,8 @@ rpmostree_find_cache_branch_by_nevra (OstreeRepo *pkgcache,
|
||||
{
|
||||
/* there's no safe way to convert a nevra string to its cache branch, so let's
|
||||
* just do a dumb lookup */
|
||||
/* TODO: parse the refs once at core init, this function is itself
|
||||
* called in loops */
|
||||
|
||||
g_autoptr(GHashTable) refs = NULL;
|
||||
if (!ostree_repo_list_refs_ext (pkgcache, "rpmostree/pkg", &refs,
|
||||
@ -1145,8 +1147,8 @@ append_quoted (GString *r, const char *value)
|
||||
}
|
||||
|
||||
/* Return the ostree cache branch for a nevra */
|
||||
static char *
|
||||
cache_branch_for_n_evr_a (const char *name, const char *evr, const char *arch)
|
||||
char *
|
||||
rpmostree_get_cache_branch_for_n_evr_a (const char *name, const char *evr, const char *arch)
|
||||
{
|
||||
GString *r = g_string_new ("rpmostree/pkg/");
|
||||
append_quoted (r, name);
|
||||
@ -1173,16 +1175,16 @@ rpmostree_get_cache_branch_header (Header hdr)
|
||||
g_autofree char *name = headerGetAsString (hdr, RPMTAG_NAME);
|
||||
g_autofree char *evr = headerGetAsString (hdr, RPMTAG_EVR);
|
||||
g_autofree char *arch = headerGetAsString (hdr, RPMTAG_ARCH);
|
||||
return cache_branch_for_n_evr_a (name, evr, arch);
|
||||
return rpmostree_get_cache_branch_for_n_evr_a (name, evr, arch);
|
||||
}
|
||||
|
||||
/* Return the ostree cache branch from a libdnf Package */
|
||||
char *
|
||||
rpmostree_get_cache_branch_pkg (DnfPackage *pkg)
|
||||
{
|
||||
return cache_branch_for_n_evr_a (dnf_package_get_name (pkg),
|
||||
dnf_package_get_evr (pkg),
|
||||
dnf_package_get_arch (pkg));
|
||||
return rpmostree_get_cache_branch_for_n_evr_a (dnf_package_get_name (pkg),
|
||||
dnf_package_get_evr (pkg),
|
||||
dnf_package_get_arch (pkg));
|
||||
}
|
||||
|
||||
static gboolean
|
||||
@ -1362,6 +1364,8 @@ find_pkg_in_ostree (RpmOstreeContext *self,
|
||||
* which ones we'll need to import, and which ones we'll need to relabel */
|
||||
static gboolean
|
||||
sort_packages (RpmOstreeContext *self,
|
||||
GPtrArray *packages,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
DnfContext *dnfctx = self->dnfctx;
|
||||
@ -1374,13 +1378,13 @@ sort_packages (RpmOstreeContext *self,
|
||||
self->pkgs_to_relabel = g_ptr_array_new_with_free_func ((GDestroyNotify)g_object_unref);
|
||||
|
||||
GPtrArray *sources = dnf_context_get_repos (dnfctx);
|
||||
g_autoptr(GPtrArray) packages = dnf_goal_get_packages (dnf_context_get_goal (dnfctx),
|
||||
DNF_PACKAGE_INFO_INSTALL,
|
||||
DNF_PACKAGE_INFO_UPDATE,
|
||||
DNF_PACKAGE_INFO_DOWNGRADE, -1);
|
||||
for (guint i = 0; i < packages->len; i++)
|
||||
{
|
||||
DnfPackage *pkg = packages->pdata[i];
|
||||
|
||||
if (g_cancellable_set_error_if_cancelled (cancellable, error))
|
||||
return FALSE;
|
||||
|
||||
const char *reponame = dnf_package_get_reponame (pkg);
|
||||
gboolean is_locally_cached =
|
||||
(g_strcmp0 (reponame, HY_CMDLINE_REPO_NAME) == 0);
|
||||
@ -1787,8 +1791,16 @@ rpmostree_context_prepare (RpmOstreeContext *self,
|
||||
|
||||
/* XXX: consider a --allow-uninstall switch? */
|
||||
if (!dnf_goal_depsolve (goal, DNF_INSTALL | DNF_ALLOW_UNINSTALL, error) ||
|
||||
!check_goal_solution (self, removed_pkgnames, replaced_nevras, error) ||
|
||||
!sort_packages (self, error))
|
||||
!check_goal_solution (self, removed_pkgnames, replaced_nevras, error))
|
||||
{
|
||||
rpmostree_output_task_end ("failed");
|
||||
return FALSE;
|
||||
}
|
||||
g_autoptr(GPtrArray) packages = dnf_goal_get_packages (dnf_context_get_goal (dnfctx),
|
||||
DNF_PACKAGE_INFO_INSTALL,
|
||||
DNF_PACKAGE_INFO_UPDATE,
|
||||
DNF_PACKAGE_INFO_DOWNGRADE, -1);
|
||||
if (!sort_packages (self, packages, cancellable, error))
|
||||
{
|
||||
rpmostree_output_task_end ("failed");
|
||||
return FALSE;
|
||||
@ -1798,6 +1810,29 @@ rpmostree_context_prepare (RpmOstreeContext *self,
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
/* Rather than doing a depsolve, directly set which packages
|
||||
* are required. Will be used by jigdo.
|
||||
*/
|
||||
gboolean
|
||||
rpmostree_context_set_packages (RpmOstreeContext *self,
|
||||
GPtrArray *packages,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
g_clear_pointer (&self->pkgs_to_download, (GDestroyNotify)g_ptr_array_unref);
|
||||
g_clear_pointer (&self->pkgs_to_import, (GDestroyNotify)g_ptr_array_unref);
|
||||
g_clear_pointer (&self->pkgs_to_relabel, (GDestroyNotify)g_ptr_array_unref);
|
||||
return sort_packages (self, packages, cancellable, error);
|
||||
}
|
||||
|
||||
/* Returns a reference to the set of packages that will be imported */
|
||||
GPtrArray *
|
||||
rpmostree_context_get_packages_to_import (RpmOstreeContext *self)
|
||||
{
|
||||
g_assert (self->pkgs_to_import);
|
||||
return g_ptr_array_ref (self->pkgs_to_import);
|
||||
}
|
||||
|
||||
static int
|
||||
compare_pkgs (gconstpointer ap,
|
||||
gconstpointer bp)
|
||||
@ -1993,23 +2028,13 @@ import_one_package (RpmOstreeContext *self,
|
||||
DnfContext *dnfctx,
|
||||
DnfPackage *pkg,
|
||||
OstreeSePolicy *sepolicy,
|
||||
GVariant *jigdo_xattr_table,
|
||||
GVariant *jigdo_xattrs,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
DnfRepo *pkg_repo = dnf_package_get_repo (pkg);
|
||||
g_autofree char *pkg_path = NULL;
|
||||
if (pkg_is_local (pkg))
|
||||
pkg_path = g_strdup (dnf_package_get_filename (pkg));
|
||||
else
|
||||
{
|
||||
const char *pkg_location = dnf_package_get_location (pkg);
|
||||
pkg_path =
|
||||
g_build_filename (dnf_repo_get_location (pkg_repo),
|
||||
"packages", glnx_basename (pkg_location), NULL);
|
||||
}
|
||||
|
||||
/* Verify signatures if enabled */
|
||||
if (!dnf_transaction_gpgcheck_package (dnf_context_get_transaction (dnfctx), pkg, error))
|
||||
glnx_fd_close int fd = -1;
|
||||
if (!rpmostree_context_consume_package (self, pkg, &fd, error))
|
||||
return FALSE;
|
||||
|
||||
/* Only set SKIP_EXTRANEOUS for packages we know need it, so that
|
||||
@ -2030,19 +2055,14 @@ import_one_package (RpmOstreeContext *self,
|
||||
}
|
||||
|
||||
/* TODO - tweak the unpacker flags for containers */
|
||||
g_autoptr(RpmOstreeImporter) unpacker = rpmostree_importer_new_at (AT_FDCWD, pkg_path, pkg, flags, error);
|
||||
g_autoptr(RpmOstreeImporter) unpacker = rpmostree_importer_new_fd (fd, pkg, flags, error);
|
||||
if (!unpacker)
|
||||
return FALSE;
|
||||
|
||||
/* And delete it now; this does mean if we fail it'll have been
|
||||
* deleted and hence more annoying to debug, but in practice people
|
||||
* should be able to redownload, and if the error was something like
|
||||
* ENOSPC, deleting it was the right move I'd say.
|
||||
*/
|
||||
if (!pkg_is_local (pkg))
|
||||
if (jigdo_xattrs)
|
||||
{
|
||||
if (!glnx_unlinkat (AT_FDCWD, pkg_path, 0, error))
|
||||
return FALSE;
|
||||
g_assert (!sepolicy);
|
||||
rpmostree_importer_set_jigdo_mode (unpacker, jigdo_xattr_table, jigdo_xattrs);
|
||||
}
|
||||
|
||||
OstreeRepo *ostreerepo = get_pkgcache_repo (self);
|
||||
@ -2064,9 +2084,11 @@ dnf_state_assert_done (DnfState *hifstate)
|
||||
}
|
||||
|
||||
gboolean
|
||||
rpmostree_context_import (RpmOstreeContext *self,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
rpmostree_context_import_jigdo (RpmOstreeContext *self,
|
||||
GVariant *jigdo_xattr_table,
|
||||
GHashTable *jigdo_pkg_to_xattrs,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
DnfContext *dnfctx = self->dnfctx;
|
||||
const int n = self->pkgs_to_import->len;
|
||||
@ -2075,6 +2097,7 @@ rpmostree_context_import (RpmOstreeContext *self,
|
||||
|
||||
OstreeRepo *repo = get_pkgcache_repo (self);
|
||||
g_return_val_if_fail (repo != NULL, FALSE);
|
||||
g_return_val_if_fail (jigdo_pkg_to_xattrs == NULL || self->sepolicy == NULL, FALSE);
|
||||
|
||||
if (!dnf_transaction_import_keys (dnf_context_get_transaction (dnfctx), error))
|
||||
return FALSE;
|
||||
@ -2094,8 +2117,16 @@ rpmostree_context_import (RpmOstreeContext *self,
|
||||
for (guint i = 0; i < self->pkgs_to_import->len; i++)
|
||||
{
|
||||
DnfPackage *pkg = self->pkgs_to_import->pdata[i];
|
||||
if (!import_one_package (self, dnfctx, pkg,
|
||||
self->sepolicy, cancellable, error))
|
||||
GVariant *jigdo_xattrs = NULL;
|
||||
if (jigdo_pkg_to_xattrs)
|
||||
{
|
||||
jigdo_xattrs = g_hash_table_lookup (jigdo_pkg_to_xattrs, pkg);
|
||||
if (!jigdo_xattrs)
|
||||
g_error ("Failed to find jigdo xattrs for %s", dnf_package_get_nevra (pkg));
|
||||
}
|
||||
if (!import_one_package (self, dnfctx, pkg, self->sepolicy,
|
||||
jigdo_xattr_table, jigdo_xattrs,
|
||||
cancellable, error))
|
||||
return FALSE;
|
||||
dnf_state_assert_done (hifstate);
|
||||
}
|
||||
@ -2116,6 +2147,59 @@ rpmostree_context_import (RpmOstreeContext *self,
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
gboolean
|
||||
rpmostree_context_import (RpmOstreeContext *self,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
return rpmostree_context_import_jigdo (self, NULL, NULL, cancellable, error);
|
||||
}
|
||||
|
||||
/* Given a single package, verify its GPG signature (if enabled), open a file
|
||||
* descriptor for it, and delete the on-disk downloaded copy.
|
||||
*/
|
||||
gboolean
|
||||
rpmostree_context_consume_package (RpmOstreeContext *self,
|
||||
DnfPackage *pkg,
|
||||
int *out_fd,
|
||||
GError **error)
|
||||
{
|
||||
/* Verify signatures if enabled */
|
||||
if (!dnf_transaction_gpgcheck_package (dnf_context_get_transaction (self->dnfctx), pkg, error))
|
||||
return FALSE;
|
||||
|
||||
DnfRepo *pkg_repo = dnf_package_get_repo (pkg);
|
||||
g_autofree char *pkg_path = NULL;
|
||||
const gboolean is_local = pkg_is_local (pkg);
|
||||
if (is_local)
|
||||
pkg_path = g_strdup (dnf_package_get_filename (pkg));
|
||||
else
|
||||
{
|
||||
const char *pkg_location = dnf_package_get_location (pkg);
|
||||
pkg_path =
|
||||
g_build_filename (dnf_repo_get_location (pkg_repo),
|
||||
"packages", glnx_basename (pkg_location), NULL);
|
||||
}
|
||||
|
||||
glnx_autofd int fd = -1;
|
||||
if (!glnx_openat_rdonly (AT_FDCWD, pkg_path, TRUE, &fd, error))
|
||||
return FALSE;
|
||||
|
||||
/* And delete it now; this does mean if we fail it'll have been
|
||||
* deleted and hence more annoying to debug, but in practice people
|
||||
* should be able to redownload, and if the error was something like
|
||||
* ENOSPC, deleting it was the right move I'd say.
|
||||
*/
|
||||
if (!pkg_is_local (pkg))
|
||||
{
|
||||
if (!glnx_unlinkat (AT_FDCWD, pkg_path, 0, error))
|
||||
return FALSE;
|
||||
}
|
||||
|
||||
*out_fd = glnx_steal_fd (&fd);
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
static gboolean
|
||||
checkout_package (OstreeRepo *repo,
|
||||
DnfPackage *pkg,
|
||||
|
||||
@ -92,6 +92,7 @@ gboolean rpmostree_context_get_state_sha512 (RpmOstreeContext *self,
|
||||
char **out_checksum,
|
||||
GError **error);
|
||||
|
||||
char * rpmostree_get_cache_branch_for_n_evr_a (const char *name, const char *evr, const char *arch);
|
||||
char *rpmostree_get_cache_branch_header (Header hdr);
|
||||
char *rpmostree_get_cache_branch_pkg (DnfPackage *pkg);
|
||||
|
||||
@ -130,14 +131,34 @@ gboolean rpmostree_context_prepare (RpmOstreeContext *self,
|
||||
GCancellable *cancellable,
|
||||
GError **error);
|
||||
|
||||
/* Alternative to _prepare() for non-depsolve cases like jigdo */
|
||||
gboolean rpmostree_context_set_packages (RpmOstreeContext *self,
|
||||
GPtrArray *packages,
|
||||
GCancellable *cancellable,
|
||||
GError **error);
|
||||
|
||||
GPtrArray *rpmostree_context_get_packages_to_import (RpmOstreeContext *self);
|
||||
|
||||
gboolean rpmostree_context_download (RpmOstreeContext *self,
|
||||
GCancellable *cancellable,
|
||||
GError **error);
|
||||
|
||||
gboolean
|
||||
rpmostree_context_consume_package (RpmOstreeContext *self,
|
||||
DnfPackage *package,
|
||||
int *out_fd,
|
||||
GError **error);
|
||||
|
||||
gboolean rpmostree_context_import (RpmOstreeContext *self,
|
||||
GCancellable *cancellable,
|
||||
GError **error);
|
||||
|
||||
gboolean rpmostree_context_import_jigdo (RpmOstreeContext *self,
|
||||
GVariant *xattr_table,
|
||||
GHashTable *pkg_to_xattrs,
|
||||
GCancellable *cancellable,
|
||||
GError **error);
|
||||
|
||||
gboolean rpmostree_context_relabel (RpmOstreeContext *self,
|
||||
GCancellable *cancellable,
|
||||
GError **error);
|
||||
|
||||
@ -35,6 +35,7 @@
|
||||
#include "rpmostree-unpacker-core.h"
|
||||
#include "rpmostree-importer.h"
|
||||
#include "rpmostree-core.h"
|
||||
#include "rpmostree-jigdo-assembler.h"
|
||||
#include "rpmostree-rpm-util.h"
|
||||
#include <rpm/rpmlib.h>
|
||||
#include <rpm/rpmlog.h>
|
||||
@ -67,6 +68,11 @@ struct RpmOstreeImporter
|
||||
char *hdr_sha256;
|
||||
|
||||
char *ostree_branch;
|
||||
|
||||
gboolean jigdo_mode;
|
||||
GVariant *jigdo_xattr_table;
|
||||
GVariant *jigdo_xattrs;
|
||||
GVariant *jigdo_next_xattrs; /* passed from filter to xattr cb */
|
||||
};
|
||||
|
||||
G_DEFINE_TYPE(RpmOstreeImporter, rpmostree_importer, G_TYPE_OBJECT)
|
||||
@ -91,6 +97,10 @@ rpmostree_importer_finalize (GObject *object)
|
||||
|
||||
g_free (self->hdr_sha256);
|
||||
|
||||
g_clear_pointer (&self->jigdo_xattr_table, (GDestroyNotify)g_variant_unref);
|
||||
g_clear_pointer (&self->jigdo_xattrs, (GDestroyNotify)g_variant_unref);
|
||||
g_clear_pointer (&self->jigdo_next_xattrs, (GDestroyNotify)g_variant_unref);
|
||||
|
||||
G_OBJECT_CLASS (rpmostree_importer_parent_class)->finalize (object);
|
||||
}
|
||||
|
||||
@ -300,6 +310,16 @@ rpmostree_importer_new_at (int dfd, const char *path,
|
||||
return g_steal_pointer (&ret);
|
||||
}
|
||||
|
||||
void
|
||||
rpmostree_importer_set_jigdo_mode (RpmOstreeImporter *self,
|
||||
GVariant *xattr_table,
|
||||
GVariant *xattrs)
|
||||
{
|
||||
self->jigdo_mode = TRUE;
|
||||
self->jigdo_xattr_table = g_variant_ref (xattr_table);
|
||||
self->jigdo_xattrs = g_variant_ref (xattrs);
|
||||
}
|
||||
|
||||
static void
|
||||
get_rpmfi_override (RpmOstreeImporter *self,
|
||||
const char *path,
|
||||
@ -475,6 +495,13 @@ build_metadata_variant (RpmOstreeImporter *self,
|
||||
g_variant_new_string (chksum_repr));
|
||||
}
|
||||
|
||||
if (self->jigdo_mode)
|
||||
{
|
||||
g_variant_builder_add (&metadata_builder, "{sv}",
|
||||
"rpmostree.jigdo",
|
||||
g_variant_new_boolean (TRUE));
|
||||
}
|
||||
|
||||
if (self->doc_files)
|
||||
{
|
||||
g_variant_builder_add (&metadata_builder, "{sv}",
|
||||
@ -680,6 +707,45 @@ unprivileged_filter_cb (OstreeRepo *repo,
|
||||
return OSTREE_REPO_COMMIT_FILTER_ALLOW;
|
||||
}
|
||||
|
||||
static OstreeRepoCommitFilterResult
|
||||
jigdo_filter_cb (OstreeRepo *repo,
|
||||
const char *path,
|
||||
GFileInfo *file_info,
|
||||
gpointer user_data)
|
||||
{
|
||||
RpmOstreeImporter *self = ((cb_data*)user_data)->self;
|
||||
GError **error = ((cb_data*)user_data)->error;
|
||||
const gboolean error_was_set = (error && *error != NULL);
|
||||
|
||||
if (error_was_set)
|
||||
return OSTREE_REPO_COMMIT_FILTER_SKIP;
|
||||
|
||||
if (g_file_info_get_file_type (file_info) != G_FILE_TYPE_DIRECTORY)
|
||||
{
|
||||
self->jigdo_next_xattrs = NULL;
|
||||
if (!rpmostree_jigdo_assembler_xattr_lookup (self->jigdo_xattr_table, path,
|
||||
self->jigdo_xattrs,
|
||||
&self->jigdo_next_xattrs,
|
||||
error))
|
||||
return OSTREE_REPO_COMMIT_FILTER_SKIP;
|
||||
/* No xattrs means we don't need to import it */
|
||||
if (!self->jigdo_next_xattrs)
|
||||
return OSTREE_REPO_COMMIT_FILTER_SKIP;
|
||||
}
|
||||
|
||||
return OSTREE_REPO_COMMIT_FILTER_ALLOW;
|
||||
}
|
||||
|
||||
static GVariant*
|
||||
jigdo_xattr_cb (OstreeRepo *repo,
|
||||
const char *path,
|
||||
GFileInfo *file_info,
|
||||
gpointer user_data)
|
||||
{
|
||||
RpmOstreeImporter *self = user_data;
|
||||
return g_steal_pointer (&self->jigdo_next_xattrs);
|
||||
}
|
||||
|
||||
static GVariant*
|
||||
xattr_cb (OstreeRepo *repo,
|
||||
const char *path,
|
||||
@ -737,7 +803,9 @@ import_rpm_to_repo (RpmOstreeImporter *self,
|
||||
* is unprivileged, anything else is a compose.
|
||||
*/
|
||||
const gboolean unprivileged = ostree_repo_get_mode (repo) == OSTREE_REPO_MODE_BARE_USER_ONLY;
|
||||
if (unprivileged)
|
||||
if (self->jigdo_mode)
|
||||
filter = jigdo_filter_cb;
|
||||
else if (unprivileged)
|
||||
filter = unprivileged_filter_cb;
|
||||
else
|
||||
filter = compose_filter_cb;
|
||||
@ -749,9 +817,17 @@ import_rpm_to_repo (RpmOstreeImporter *self,
|
||||
modifier_flags |= OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS;
|
||||
g_autoptr(OstreeRepoCommitModifier) modifier =
|
||||
ostree_repo_commit_modifier_new (modifier_flags, filter, &fdata, NULL);
|
||||
ostree_repo_commit_modifier_set_xattr_callback (modifier, xattr_cb,
|
||||
NULL, self);
|
||||
ostree_repo_commit_modifier_set_sepolicy (modifier, sepolicy);
|
||||
if (self->jigdo_mode)
|
||||
{
|
||||
ostree_repo_commit_modifier_set_xattr_callback (modifier, jigdo_xattr_cb,
|
||||
NULL, self);
|
||||
g_assert (sepolicy == NULL);
|
||||
}
|
||||
else
|
||||
{
|
||||
ostree_repo_commit_modifier_set_xattr_callback (modifier, xattr_cb, NULL, self);
|
||||
ostree_repo_commit_modifier_set_sepolicy (modifier, sepolicy);
|
||||
}
|
||||
|
||||
OstreeRepoImportArchiveOptions opts = { 0 };
|
||||
opts.ignore_unsupported_content = TRUE;
|
||||
|
||||
@ -59,6 +59,10 @@ rpmostree_importer_new_at (int dfd,
|
||||
RpmOstreeImporterFlags flags,
|
||||
GError **error);
|
||||
|
||||
void rpmostree_importer_set_jigdo_mode (RpmOstreeImporter *self,
|
||||
GVariant *xattr_table,
|
||||
GVariant *xattrs);
|
||||
|
||||
gboolean
|
||||
rpmostree_importer_read_metainfo (int fd,
|
||||
Header *out_header,
|
||||
|
||||
643
src/libpriv/rpmostree-jigdo-assembler.c
Normal file
643
src/libpriv/rpmostree-jigdo-assembler.c
Normal file
@ -0,0 +1,643 @@
|
||||
/* -*- mode: C; c-file-style: "gnu"; indent-tabs-mode: nil; -*-
|
||||
*
|
||||
* Copyright (C) 2017 Red Hat, Inc.
|
||||
*
|
||||
* Licensed under the GNU Lesser General Public License Version 2.1
|
||||
*
|
||||
* This library is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU Lesser General Public
|
||||
* License as published by the Free Software Foundation; either
|
||||
* version 2.1 of the License, or (at your option) any later version.
|
||||
*
|
||||
* This library is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
* Lesser General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU Lesser General Public
|
||||
* License along with this library; if not, write to the Free Software
|
||||
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
|
||||
*/
|
||||
|
||||
|
||||
#include "config.h"
|
||||
|
||||
#include <gio/gunixinputstream.h>
|
||||
#include "rpmostree-libarchive-input-stream.h"
|
||||
#include "rpmostree-unpacker-core.h"
|
||||
#include "rpmostree-jigdo-assembler.h"
|
||||
#include "rpmostree-core.h"
|
||||
#include "rpmostree-rpm-util.h"
|
||||
#include <rpm/rpmlib.h>
|
||||
#include <rpm/rpmlog.h>
|
||||
#include <rpm/rpmfi.h>
|
||||
#include <rpm/rpmts.h>
|
||||
#include <archive.h>
|
||||
#include <archive_entry.h>
|
||||
|
||||
#include <string.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
typedef enum {
|
||||
STATE_COMMIT,
|
||||
STATE_DIRMETA,
|
||||
STATE_DIRTREE,
|
||||
STATE_NEW_CONTENTIDENT,
|
||||
STATE_NEW,
|
||||
STATE_XATTRS_TABLE,
|
||||
STATE_XATTRS_PKG,
|
||||
} JigdoAssemblerState;
|
||||
|
||||
static gboolean
|
||||
throw_libarchive_error (GError **error,
|
||||
struct archive *a)
|
||||
{
|
||||
return glnx_throw (error, "%s", archive_error_string (a));
|
||||
}
|
||||
|
||||
typedef GObjectClass RpmOstreeJigdoAssemblerClass;
|
||||
|
||||
struct RpmOstreeJigdoAssembler
|
||||
{
|
||||
GObject parent_instance;
|
||||
JigdoAssemblerState state;
|
||||
DnfPackage *pkg;
|
||||
GVariant *commit;
|
||||
GVariant *meta;
|
||||
GVariant *pkgs;
|
||||
char *checksum;
|
||||
GVariant *xattrs_table;
|
||||
struct archive *archive;
|
||||
struct archive_entry *next_entry;
|
||||
int fd;
|
||||
};
|
||||
|
||||
G_DEFINE_TYPE(RpmOstreeJigdoAssembler, rpmostree_jigdo_assembler, G_TYPE_OBJECT)
|
||||
|
||||
static void
|
||||
rpmostree_jigdo_assembler_finalize (GObject *object)
|
||||
{
|
||||
RpmOstreeJigdoAssembler *self = (RpmOstreeJigdoAssembler*)object;
|
||||
if (self->archive)
|
||||
archive_read_free (self->archive);
|
||||
g_clear_pointer (&self->commit, (GDestroyNotify)g_variant_unref);
|
||||
g_clear_pointer (&self->meta, (GDestroyNotify)g_variant_unref);
|
||||
g_clear_pointer (&self->pkgs, (GDestroyNotify)g_variant_unref);
|
||||
g_free (self->checksum);
|
||||
g_clear_object (&self->pkg);
|
||||
g_clear_pointer (&self->xattrs_table, (GDestroyNotify)g_variant_unref);
|
||||
glnx_close_fd (&self->fd);
|
||||
|
||||
G_OBJECT_CLASS (rpmostree_jigdo_assembler_parent_class)->finalize (object);
|
||||
}
|
||||
|
||||
static void
|
||||
rpmostree_jigdo_assembler_class_init (RpmOstreeJigdoAssemblerClass *klass)
|
||||
{
|
||||
GObjectClass *gobject_class = G_OBJECT_CLASS (klass);
|
||||
|
||||
gobject_class->finalize = rpmostree_jigdo_assembler_finalize;
|
||||
}
|
||||
|
||||
static void
|
||||
rpmostree_jigdo_assembler_init (RpmOstreeJigdoAssembler *self)
|
||||
{
|
||||
self->fd = -1;
|
||||
}
|
||||
|
||||
/*
|
||||
* rpmostree_jigdo_assembler_new_take_fd:
|
||||
* @fd: Fd, ownership is taken
|
||||
* @pkg: (optional): Package reference, used for metadata
|
||||
* @error: error
|
||||
*
|
||||
* Create a new unpacker instance. The @pkg argument, if
|
||||
* specified, will be inspected and metadata such as the
|
||||
* origin repo will be added to the final commit.
|
||||
*/
|
||||
RpmOstreeJigdoAssembler *
|
||||
rpmostree_jigdo_assembler_new_take_fd (int *fd,
|
||||
DnfPackage *pkg,
|
||||
GError **error)
|
||||
{
|
||||
glnx_fd_close int owned_fd = glnx_steal_fd (fd);
|
||||
|
||||
struct archive *archive = rpmostree_unpack_rpm2cpio (owned_fd, error);
|
||||
if (archive == NULL)
|
||||
return NULL;
|
||||
|
||||
RpmOstreeJigdoAssembler *ret = g_object_new (RPMOSTREE_TYPE_JIGDO_ASSEMBLER, NULL);
|
||||
ret->archive = g_steal_pointer (&archive);
|
||||
ret->pkg = pkg ? g_object_ref (pkg) : NULL;
|
||||
ret->fd = glnx_steal_fd (&owned_fd);
|
||||
|
||||
return g_steal_pointer (&ret);
|
||||
}
|
||||
|
||||
static GVariant *
|
||||
jigdo_read_variant (const GVariantType *vtype,
|
||||
struct archive *a,
|
||||
struct archive_entry *entry,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
const char *path = archive_entry_pathname (entry);
|
||||
const struct stat *stbuf = archive_entry_stat (entry);
|
||||
if (!S_ISREG (stbuf->st_mode))
|
||||
return glnx_null_throw (error, "Expected regular file for entry: %s", path);
|
||||
if (stbuf->st_size > OSTREE_MAX_METADATA_SIZE)
|
||||
{
|
||||
g_autofree char *max_formatted = g_format_size (OSTREE_MAX_METADATA_SIZE);
|
||||
g_autofree char *found_formatted = g_format_size (stbuf->st_size);
|
||||
return glnx_null_throw (error, "Exceeded maximum size %s; %s is of size: %s", max_formatted, found_formatted, path);
|
||||
}
|
||||
g_assert_cmpint (stbuf->st_size, >=, 0);
|
||||
const size_t total = stbuf->st_size;
|
||||
g_autofree guint8* buf = g_malloc (total);
|
||||
size_t bytes_read = 0;
|
||||
while (bytes_read < total)
|
||||
{
|
||||
ssize_t r = archive_read_data (a, buf + bytes_read, total - bytes_read);
|
||||
if (r < 0)
|
||||
return throw_libarchive_error (error, a), NULL;
|
||||
if (r == 0)
|
||||
break;
|
||||
bytes_read += r;
|
||||
}
|
||||
g_assert_cmpint (bytes_read, ==, total)
|
||||
;
|
||||
/* Need to take ownership once now, then pass it as a parameter twice */
|
||||
guint8* buf_owned = g_steal_pointer (&buf);
|
||||
return g_variant_new_from_data (vtype, buf_owned, bytes_read, FALSE, g_free, buf_owned);
|
||||
}
|
||||
|
||||
/* Remove leading prefix */
|
||||
static const char *
|
||||
peel_entry_pathname (struct archive_entry *entry,
|
||||
GError **error)
|
||||
{
|
||||
const char *pathname = archive_entry_pathname (entry);
|
||||
static const char prefix[] = "./usr/lib/ostree-jigdo/";
|
||||
if (!g_str_has_prefix (pathname, prefix))
|
||||
return glnx_null_throw (error, "Entry does not have prefix '%s': %s", prefix, pathname);
|
||||
pathname += strlen (prefix);
|
||||
const char *nextslash = strchr (pathname, '/');
|
||||
if (!nextslash)
|
||||
return glnx_null_throw (error, "Missing subdir in %s", pathname);
|
||||
return nextslash+1;
|
||||
}
|
||||
|
||||
static gboolean
|
||||
jigdo_next_entry (RpmOstreeJigdoAssembler *self,
|
||||
gboolean *out_eof,
|
||||
struct archive_entry **out_entry,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
if (g_cancellable_set_error_if_cancelled (cancellable, error))
|
||||
return FALSE;
|
||||
|
||||
*out_eof = FALSE;
|
||||
|
||||
if (self->next_entry)
|
||||
{
|
||||
*out_entry = g_steal_pointer (&self->next_entry);
|
||||
return TRUE; /* 🔚 Early return */
|
||||
}
|
||||
|
||||
/* Loop, skipping non-regular files */
|
||||
struct archive_entry *entry = NULL;
|
||||
while (TRUE)
|
||||
{
|
||||
int r = archive_read_next_header (self->archive, &entry);
|
||||
if (r == ARCHIVE_EOF)
|
||||
{
|
||||
*out_eof = TRUE;
|
||||
return TRUE; /* 🔚 Early return */
|
||||
}
|
||||
if (r != ARCHIVE_OK)
|
||||
return throw_libarchive_error (error, self->archive);
|
||||
|
||||
const struct stat *stbuf = archive_entry_stat (entry);
|
||||
/* We only have regular files, ignore intermediate dirs */
|
||||
if (!S_ISREG (stbuf->st_mode))
|
||||
continue;
|
||||
|
||||
/* Otherwise we're done */
|
||||
break;
|
||||
}
|
||||
|
||||
*out_eof = FALSE;
|
||||
*out_entry = entry; /* Owned by archive */
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
static struct archive_entry *
|
||||
jigdo_require_next_entry (RpmOstreeJigdoAssembler *self,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
gboolean eof;
|
||||
struct archive_entry *entry;
|
||||
if (!jigdo_next_entry (self, &eof, &entry, cancellable, error))
|
||||
return FALSE;
|
||||
if (eof)
|
||||
return glnx_null_throw (error, "Unexpected end of archive");
|
||||
return entry;
|
||||
}
|
||||
|
||||
static char *
|
||||
parse_checksum_from_pathname (const char *pathname,
|
||||
GError **error)
|
||||
{
|
||||
/* We have an extra / */
|
||||
if (strlen (pathname) != OSTREE_SHA256_STRING_LEN + 1)
|
||||
return glnx_null_throw (error, "Invalid checksum path: %s", pathname);
|
||||
g_autoptr(GString) buf = g_string_new ("");
|
||||
g_string_append_len (buf, pathname, 2);
|
||||
g_string_append (buf, pathname+3);
|
||||
return g_string_free (g_steal_pointer (&buf), FALSE);
|
||||
}
|
||||
|
||||
/* First step: read metadata: the commit object and its metadata, suitable for
|
||||
* GPG verification, as well as the component package NEVRAs.
|
||||
*/
|
||||
gboolean
|
||||
rpmostree_jigdo_assembler_read_meta (RpmOstreeJigdoAssembler *self,
|
||||
char **out_checksum,
|
||||
GVariant **out_commit,
|
||||
GVariant **out_detached_meta,
|
||||
GVariant **out_pkgs,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
g_assert_cmpint (self->state, ==, STATE_COMMIT);
|
||||
struct archive_entry *entry = jigdo_require_next_entry (self, cancellable, error);
|
||||
if (!entry)
|
||||
return FALSE;
|
||||
const char *entry_path = peel_entry_pathname (entry, error);
|
||||
if (!entry_path)
|
||||
return FALSE;
|
||||
if (!g_str_has_prefix (entry_path, RPMOSTREE_JIGDO_COMMIT_DIR "/"))
|
||||
return glnx_throw (error, "Unexpected entry: %s", entry_path);
|
||||
entry_path += strlen (RPMOSTREE_JIGDO_COMMIT_DIR "/");
|
||||
|
||||
g_autofree char *checksum = parse_checksum_from_pathname (entry_path, error);
|
||||
if (!checksum)
|
||||
return FALSE;
|
||||
|
||||
g_autoptr(GVariant) commit = jigdo_read_variant (OSTREE_COMMIT_GVARIANT_FORMAT,
|
||||
self->archive, entry, cancellable, error);
|
||||
g_autoptr(GVariant) meta = NULL;
|
||||
|
||||
entry = jigdo_require_next_entry (self, cancellable, error);
|
||||
entry_path = peel_entry_pathname (entry, error);
|
||||
if (!entry_path)
|
||||
return FALSE;
|
||||
if (g_str_equal (entry_path, RPMOSTREE_JIGDO_COMMIT_DIR "/meta"))
|
||||
{
|
||||
meta = jigdo_read_variant (G_VARIANT_TYPE ("a{sv}"), self->archive, entry,
|
||||
cancellable, error);
|
||||
if (!meta)
|
||||
return FALSE;
|
||||
}
|
||||
else
|
||||
{
|
||||
self->next_entry = entry; /* Stash for next call */
|
||||
}
|
||||
|
||||
/* And the component packages */
|
||||
entry = jigdo_require_next_entry (self, cancellable, error);
|
||||
entry_path = peel_entry_pathname (entry, error);
|
||||
if (!entry_path)
|
||||
return FALSE;
|
||||
if (!g_str_equal (entry_path, RPMOSTREE_JIGDO_PKGS))
|
||||
return glnx_throw (error, "Unexpected state for path: %s", entry_path);
|
||||
g_autoptr(GVariant) pkgs = jigdo_read_variant (RPMOSTREE_JIGDO_PKGS_VARIANT_FORMAT,
|
||||
self->archive, entry, cancellable, error);
|
||||
if (!pkgs)
|
||||
return FALSE;
|
||||
|
||||
self->state = STATE_DIRMETA;
|
||||
self->checksum = g_strdup (checksum);
|
||||
self->commit = g_variant_ref (commit);
|
||||
self->meta = meta ? g_variant_ref (meta) : NULL;
|
||||
self->pkgs = g_variant_ref (pkgs);
|
||||
*out_checksum = g_steal_pointer (&checksum);
|
||||
*out_commit = g_steal_pointer (&commit);
|
||||
*out_detached_meta = g_steal_pointer (&meta);
|
||||
*out_pkgs = g_steal_pointer (&pkgs);
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
static gboolean
|
||||
process_contentident (RpmOstreeJigdoAssembler *self,
|
||||
OstreeRepo *repo,
|
||||
struct archive_entry *entry,
|
||||
const char *meta_pathname,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
GLNX_AUTO_PREFIX_ERROR ("Processing content-identical", error);
|
||||
/* Read the metadata variant, which has an array of checksum, metadata
|
||||
* for regfile objects that have identical content.
|
||||
*/
|
||||
if (!g_str_has_suffix (meta_pathname, "/01meta"))
|
||||
return glnx_throw (error, "Malformed contentident: %s", meta_pathname);
|
||||
const char *contentident_id_start = meta_pathname + strlen (RPMOSTREE_JIGDO_NEW_CONTENTIDENT_DIR "/");
|
||||
const char *slash = strchr (contentident_id_start, '/');
|
||||
if (!slash)
|
||||
return glnx_throw (error, "Malformed contentident: %s", meta_pathname);
|
||||
// g_autofree char *contentident_id_str = g_strndup (contentident_id_start, slash - contentident_id_start);
|
||||
|
||||
g_autoptr(GVariant) meta = jigdo_read_variant (RPMOSTREE_JIGDO_NEW_CONTENTIDENT_VARIANT_FORMAT,
|
||||
self->archive, entry,
|
||||
cancellable, error);
|
||||
|
||||
|
||||
/* Read the content */
|
||||
// FIXME match contentident_id
|
||||
entry = jigdo_require_next_entry (self, cancellable, error);
|
||||
if (!entry)
|
||||
return FALSE;
|
||||
|
||||
const char *content_pathname = peel_entry_pathname (entry, error);
|
||||
if (!content_pathname)
|
||||
return FALSE;
|
||||
if (!g_str_has_suffix (content_pathname, "/05content"))
|
||||
return glnx_throw (error, "Malformed contentident: %s", content_pathname);
|
||||
|
||||
const struct stat *stbuf = archive_entry_stat (entry);
|
||||
|
||||
/* Copy the data to a temporary file; a better optimization would be to write
|
||||
* the data to the first object, then clone it, but that requires some
|
||||
* more libostree API. As far as I can see, one can't reliably seek with
|
||||
* libarchive; only some formats support it, and cpio isn't one of them.
|
||||
*/
|
||||
g_auto(GLnxTmpfile) tmpf = { 0, };
|
||||
if (!glnx_open_anonymous_tmpfile (O_RDWR | O_CLOEXEC, &tmpf, error))
|
||||
return FALSE;
|
||||
|
||||
const size_t total = stbuf->st_size;
|
||||
const size_t bufsize = MIN (128*1024, total);
|
||||
g_autofree guint8* buf = g_malloc (bufsize);
|
||||
size_t bytes_read = 0;
|
||||
while (bytes_read < total)
|
||||
{
|
||||
ssize_t r = archive_read_data (self->archive, buf, MIN (bufsize, total - bytes_read));
|
||||
if (r < 0)
|
||||
return throw_libarchive_error (error, self->archive);
|
||||
if (r == 0)
|
||||
break;
|
||||
if (glnx_loop_write (tmpf.fd, buf, r) < 0)
|
||||
return glnx_throw_errno_prefix (error, "write");
|
||||
bytes_read += r;
|
||||
}
|
||||
g_assert_cmpint (bytes_read, ==, total);
|
||||
g_clear_pointer (&buf, g_free);
|
||||
|
||||
const guint n = g_variant_n_children (meta);
|
||||
for (guint i = 0; i < n; i++)
|
||||
{
|
||||
const char *checksum;
|
||||
guint32 uid,gid,mode;
|
||||
g_autoptr(GVariant) xattrs = NULL;
|
||||
g_variant_get_child (meta, i, "(&suuu@a(ayay))", &checksum, &uid, &gid, &mode, &xattrs);
|
||||
/* See if we already have this object */
|
||||
gboolean has_object;
|
||||
if (!ostree_repo_has_object (repo, OSTREE_OBJECT_TYPE_FILE, checksum,
|
||||
&has_object, cancellable, error))
|
||||
return FALSE;
|
||||
if (has_object)
|
||||
continue;
|
||||
uid = GUINT32_FROM_BE (uid);
|
||||
gid = GUINT32_FROM_BE (gid);
|
||||
mode = GUINT32_FROM_BE (mode);
|
||||
|
||||
if (lseek (tmpf.fd, 0, SEEK_SET) < 0)
|
||||
return glnx_throw_errno_prefix (error, "lseek");
|
||||
g_autoptr(GInputStream) istream = g_unix_input_stream_new (tmpf.fd, FALSE);
|
||||
/* Like _ostree_stbuf_to_gfileinfo() - TODO make that public with a
|
||||
* better content writing API.
|
||||
*/
|
||||
g_autoptr(GFileInfo) finfo = g_file_info_new ();
|
||||
g_file_info_set_attribute_uint32 (finfo, "standard::type", G_FILE_TYPE_REGULAR);
|
||||
g_file_info_set_attribute_boolean (finfo, "standard::is-symlink", FALSE);
|
||||
g_file_info_set_attribute_uint32 (finfo, "unix::uid", uid);
|
||||
g_file_info_set_attribute_uint32 (finfo, "unix::gid", gid);
|
||||
g_file_info_set_attribute_uint32 (finfo, "unix::mode", mode);
|
||||
g_file_info_set_attribute_uint64 (finfo, "standard::size", total);
|
||||
|
||||
g_autoptr(GInputStream) objstream = NULL;
|
||||
guint64 objlen;
|
||||
if (!ostree_raw_file_to_content_stream (istream, finfo, xattrs, &objstream,
|
||||
&objlen, cancellable, error))
|
||||
return FALSE;
|
||||
|
||||
g_autofree guchar *csum = NULL;
|
||||
if (!ostree_repo_write_content (repo, checksum, objstream, objlen, &csum,
|
||||
cancellable, error))
|
||||
return FALSE;
|
||||
}
|
||||
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
static gboolean
|
||||
state_transition (RpmOstreeJigdoAssembler *self,
|
||||
const char *pathname,
|
||||
JigdoAssemblerState new_state,
|
||||
GError **error)
|
||||
{
|
||||
if (self->state > new_state)
|
||||
return glnx_throw (error, "Unexpected state for path: %s", pathname);
|
||||
self->state = new_state;
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
/* Process new objects included in the OIRPM */
|
||||
gboolean
|
||||
rpmostree_jigdo_assembler_write_new_objects (RpmOstreeJigdoAssembler *self,
|
||||
OstreeRepo *repo,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
GLNX_AUTO_PREFIX_ERROR ("Writing new objects", error);
|
||||
g_assert_cmpint (self->state, ==, STATE_DIRMETA);
|
||||
|
||||
/* TODO sort objects in order for importing, verify we're not
|
||||
* importing an unknown object.
|
||||
*/
|
||||
while (TRUE)
|
||||
{
|
||||
gboolean eof;
|
||||
struct archive_entry *entry;
|
||||
if (!jigdo_next_entry (self, &eof, &entry, cancellable, error))
|
||||
return FALSE;
|
||||
if (eof)
|
||||
break;
|
||||
const char *pathname = peel_entry_pathname (entry, error);
|
||||
if (!pathname)
|
||||
return FALSE;
|
||||
if (g_str_has_prefix (pathname, RPMOSTREE_JIGDO_DIRMETA_DIR "/"))
|
||||
{
|
||||
if (!state_transition (self, pathname, STATE_DIRMETA, error))
|
||||
return FALSE;
|
||||
g_autofree char *checksum =
|
||||
parse_checksum_from_pathname (pathname + strlen (RPMOSTREE_JIGDO_DIRMETA_DIR "/"), error);
|
||||
if (!checksum)
|
||||
return FALSE;
|
||||
g_autoptr(GVariant) dirmeta = jigdo_read_variant (OSTREE_DIRMETA_GVARIANT_FORMAT,
|
||||
self->archive, entry,
|
||||
cancellable, error);
|
||||
g_autofree guint8*csum = NULL;
|
||||
if (!ostree_repo_write_metadata (repo, OSTREE_OBJECT_TYPE_DIR_META,
|
||||
checksum, dirmeta, &csum, cancellable, error))
|
||||
return FALSE;
|
||||
}
|
||||
else if (g_str_has_prefix (pathname, RPMOSTREE_JIGDO_DIRTREE_DIR "/"))
|
||||
{
|
||||
if (!state_transition (self, pathname, STATE_DIRTREE, error))
|
||||
return FALSE;
|
||||
g_autofree char *checksum =
|
||||
parse_checksum_from_pathname (pathname + strlen (RPMOSTREE_JIGDO_DIRTREE_DIR "/"), error);
|
||||
if (!checksum)
|
||||
return FALSE;
|
||||
g_autoptr(GVariant) dirtree = jigdo_read_variant (OSTREE_TREE_GVARIANT_FORMAT,
|
||||
self->archive, entry,
|
||||
cancellable, error);
|
||||
g_autofree guint8*csum = NULL;
|
||||
if (!ostree_repo_write_metadata (repo, OSTREE_OBJECT_TYPE_DIR_TREE,
|
||||
checksum, dirtree, &csum, cancellable, error))
|
||||
return FALSE;
|
||||
}
|
||||
else if (g_str_has_prefix (pathname, RPMOSTREE_JIGDO_NEW_CONTENTIDENT_DIR "/"))
|
||||
{
|
||||
if (!state_transition (self, pathname, STATE_NEW_CONTENTIDENT, error))
|
||||
return FALSE;
|
||||
if (!process_contentident (self, repo, entry, pathname, cancellable, error))
|
||||
return FALSE;
|
||||
}
|
||||
else if (g_str_has_prefix (pathname, RPMOSTREE_JIGDO_NEW_DIR "/"))
|
||||
{
|
||||
if (!state_transition (self, pathname, STATE_NEW, error))
|
||||
return FALSE;
|
||||
g_autofree char *checksum =
|
||||
parse_checksum_from_pathname (pathname + strlen (RPMOSTREE_JIGDO_NEW_DIR "/"), error);
|
||||
if (!checksum)
|
||||
return FALSE;
|
||||
|
||||
const struct stat *stbuf = archive_entry_stat (entry);
|
||||
g_assert_cmpint (stbuf->st_size, >=, 0);
|
||||
|
||||
g_autoptr(GInputStream) archive_stream = _rpm_ostree_libarchive_input_stream_new (self->archive);
|
||||
g_autofree guint8*csum = NULL;
|
||||
if (!ostree_repo_write_content (repo, checksum, archive_stream,
|
||||
stbuf->st_size, &csum, cancellable, error))
|
||||
return FALSE;
|
||||
}
|
||||
else if (g_str_has_prefix (pathname, RPMOSTREE_JIGDO_XATTRS_DIR "/"))
|
||||
{
|
||||
self->next_entry = g_steal_pointer (&entry); /* Stash for next call */
|
||||
break;
|
||||
}
|
||||
else
|
||||
return glnx_throw (error, "Unexpected entry: %s", pathname);
|
||||
}
|
||||
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
GVariant *
|
||||
rpmostree_jigdo_assembler_get_xattr_table (RpmOstreeJigdoAssembler *self)
|
||||
{
|
||||
g_assert (self->xattrs_table);
|
||||
return g_variant_ref (self->xattrs_table);
|
||||
}
|
||||
|
||||
/* Loop over each package, returning its xattr set (as indexes into the xattr table) */
|
||||
gboolean
|
||||
rpmostree_jigdo_assembler_next_xattrs (RpmOstreeJigdoAssembler *self,
|
||||
GVariant **out_objid_to_xattrs,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
/* Init output variable for EOF state now */
|
||||
*out_objid_to_xattrs = NULL;
|
||||
|
||||
/* If we haven't loaded the xattr string table, do so */
|
||||
if (self->state < STATE_XATTRS_TABLE)
|
||||
{
|
||||
gboolean eof;
|
||||
struct archive_entry *entry;
|
||||
if (!jigdo_next_entry (self, &eof, &entry, cancellable, error))
|
||||
return FALSE;
|
||||
if (eof)
|
||||
return TRUE; /* 🔚 Early return */
|
||||
|
||||
const char *pathname = peel_entry_pathname (entry, error);
|
||||
if (!pathname)
|
||||
return FALSE;
|
||||
if (!g_str_has_prefix (pathname, RPMOSTREE_JIGDO_XATTRS_TABLE))
|
||||
return glnx_throw (error, "Unexpected entry: %s", pathname);
|
||||
|
||||
g_autoptr(GVariant) xattrs_table = jigdo_read_variant (RPMOSTREE_JIGDO_XATTRS_TABLE_VARIANT_FORMAT,
|
||||
self->archive, entry, cancellable, error);
|
||||
if (!xattrs_table)
|
||||
return FALSE;
|
||||
g_assert (!self->xattrs_table);
|
||||
self->xattrs_table = g_steal_pointer (&xattrs_table);
|
||||
self->state = STATE_XATTRS_TABLE;
|
||||
}
|
||||
|
||||
/* Look for an xattr entry */
|
||||
gboolean eof;
|
||||
struct archive_entry *entry;
|
||||
if (!jigdo_next_entry (self, &eof, &entry, cancellable, error))
|
||||
return FALSE;
|
||||
if (eof)
|
||||
return TRUE; /* 🔚 Early return */
|
||||
|
||||
const char *pathname = peel_entry_pathname (entry, error);
|
||||
if (!pathname)
|
||||
return FALSE;
|
||||
/* At this point there's nothing left besides xattrs, so throw if it doesn't
|
||||
* match that filename pattern.
|
||||
*/
|
||||
if (!g_str_has_prefix (pathname, RPMOSTREE_JIGDO_XATTRS_PKG_DIR "/"))
|
||||
return glnx_throw (error, "Unexpected entry: %s", pathname);
|
||||
// const char *nevra = pathname + strlen (RPMOSTREE_JIGDO_XATTRS_PKG_DIR "/");
|
||||
*out_objid_to_xattrs = jigdo_read_variant (RPMOSTREE_JIGDO_XATTRS_PKG_VARIANT_FORMAT,
|
||||
self->archive, entry, cancellable, error);
|
||||
return TRUE;
|
||||
}
|
||||
|
||||
/* Client side lookup for xattrs */
|
||||
gboolean
|
||||
rpmostree_jigdo_assembler_xattr_lookup (GVariant *xattr_table,
|
||||
const char *path,
|
||||
GVariant *xattrs,
|
||||
GVariant **out_xattrs,
|
||||
GError **error)
|
||||
{
|
||||
int pos;
|
||||
if (!rpmostree_variant_bsearch_str (xattrs, path, &pos))
|
||||
{
|
||||
const char *bn = glnx_basename (path);
|
||||
if (!rpmostree_variant_bsearch_str (xattrs, bn, &pos))
|
||||
{
|
||||
// TODO add an "objects to skip" map; currently not found means
|
||||
// "don't import"
|
||||
*out_xattrs = NULL;
|
||||
return TRUE;
|
||||
/* jigdodata->caught_error = TRUE; */
|
||||
/* return glnx_null_throw (&jigdodata->error, "Failed to find jigdo xattrs for path '%s'", path); */
|
||||
}
|
||||
}
|
||||
guint xattr_idx;
|
||||
g_variant_get_child (xattrs, pos, "(&su)", NULL, &xattr_idx);
|
||||
if (xattr_idx >= g_variant_n_children (xattr_table))
|
||||
return glnx_throw (error, "Out of range jigdo xattr index %u for path '%s'", xattr_idx, path);
|
||||
*out_xattrs = g_variant_get_child_value (xattr_table, xattr_idx);
|
||||
return TRUE;
|
||||
}
|
||||
70
src/libpriv/rpmostree-jigdo-assembler.h
Normal file
70
src/libpriv/rpmostree-jigdo-assembler.h
Normal file
@ -0,0 +1,70 @@
|
||||
/* -*- mode: C; c-file-style: "gnu"; indent-tabs-mode: nil; -*-
|
||||
*
|
||||
* Copyright (C) 2015 Colin Walters <walters@verbum.org>
|
||||
*
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU Lesser General Public License as published
|
||||
* by the Free Software Foundation; either version 2 of the licence or (at
|
||||
* your option) any later version.
|
||||
*
|
||||
* This library is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
* Lesser General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU Lesser General
|
||||
* Public License along with this library; if not, write to the
|
||||
* Free Software Foundation, Inc., 59 Temple Place, Suite 330,
|
||||
* Boston, MA 02111-1307, USA.
|
||||
*/
|
||||
|
||||
#pragma once
|
||||
|
||||
#include "libglnx.h"
|
||||
#include "rpmostree-jigdo-core.h"
|
||||
|
||||
typedef struct RpmOstreeJigdoAssembler RpmOstreeJigdoAssembler;
|
||||
|
||||
#define RPMOSTREE_TYPE_JIGDO_ASSEMBLER (rpmostree_jigdo_assembler_get_type ())
|
||||
#define RPMOSTREE_JIGDO_ASSEMBLER(inst) (G_TYPE_CHECK_INSTANCE_CAST ((inst), RPMOSTREE_TYPE_JIGDO_ASSEMBLER, RpmOstreeJigdoAssembler))
|
||||
#define RPMOSTREE_IS_JIGDO_ASSEMBLER(inst) (G_TYPE_CHECK_INSTANCE_TYPE ((inst), RPMOSTREE_TYPE_JIGDO_ASSEMBLER))
|
||||
|
||||
GType rpmostree_jigdo_assembler_get_type (void);
|
||||
|
||||
G_DEFINE_AUTOPTR_CLEANUP_FUNC (RpmOstreeJigdoAssembler, g_object_unref)
|
||||
|
||||
RpmOstreeJigdoAssembler*
|
||||
rpmostree_jigdo_assembler_new_take_fd (int *fd,
|
||||
DnfPackage *pkg, /* for metadata */
|
||||
GError **error);
|
||||
|
||||
gboolean
|
||||
rpmostree_jigdo_assembler_read_meta (RpmOstreeJigdoAssembler *jigdo,
|
||||
char **out_checksum,
|
||||
GVariant **commit,
|
||||
GVariant **detached_meta,
|
||||
GVariant **pkgs,
|
||||
GCancellable *cancellable,
|
||||
GError **error);
|
||||
|
||||
gboolean
|
||||
rpmostree_jigdo_assembler_write_new_objects (RpmOstreeJigdoAssembler *jigdo,
|
||||
OstreeRepo *repo,
|
||||
GCancellable *cancellable,
|
||||
GError **error);
|
||||
|
||||
|
||||
GVariant * rpmostree_jigdo_assembler_get_xattr_table (RpmOstreeJigdoAssembler *self);
|
||||
|
||||
gboolean
|
||||
rpmostree_jigdo_assembler_next_xattrs (RpmOstreeJigdoAssembler *self,
|
||||
GVariant **out_objid_to_xattrs,
|
||||
GCancellable *cancellable,
|
||||
GError **error);
|
||||
|
||||
gboolean
|
||||
rpmostree_jigdo_assembler_xattr_lookup (GVariant *xattr_table,
|
||||
const char *path,
|
||||
GVariant *xattrs,
|
||||
GVariant **out_xattrs,
|
||||
GError **error);
|
||||
77
src/libpriv/rpmostree-jigdo-core.h
Normal file
77
src/libpriv/rpmostree-jigdo-core.h
Normal file
@ -0,0 +1,77 @@
|
||||
/* -*- mode: C; c-file-style: "gnu"; indent-tabs-mode: nil; -*-
|
||||
*
|
||||
* Copyright (C) 2017 Colin Walters <walters@verbum.org>
|
||||
*
|
||||
* This program is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU Lesser General Public License as published
|
||||
* by the Free Software Foundation; either version 2 of the licence or (at
|
||||
* your option) any later version.
|
||||
*
|
||||
* This library is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
* Lesser General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU Lesser General
|
||||
* Public License along with this library; if not, write to the
|
||||
* Free Software Foundation, Inc., 59 Temple Place, Suite 330,
|
||||
* Boston, MA 02111-1307, USA.
|
||||
*/
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <ostree.h>
|
||||
|
||||
#include "libglnx.h"
|
||||
#include <rpm/rpmlib.h>
|
||||
#include <libdnf/libdnf.h>
|
||||
|
||||
/* An OIRPM is structured as an ordered set of files/directories; we use numeric
|
||||
* prefixes to ensure ordering. Most of the files are in GVariant format.
|
||||
*
|
||||
* An OIRPM starts with the OSTree commit object and its detached metadata,
|
||||
* so that can be GPG verified first - if that fails, we can then cleanly
|
||||
* abort.
|
||||
*
|
||||
* Next, we have the "jigdo set" - the NEVRAs + repodata checksum of the
|
||||
* RPM packages we need. So during client side processing, downloads
|
||||
* can be initiated for those while we continue to process the OIRPM.
|
||||
*
|
||||
* The dirmeta/dirtree objects that are referenced by the commit follow.
|
||||
*
|
||||
* A special optimization is made for "content-identical" new objects,
|
||||
* such as the initramfs right now which unfortunately has separate
|
||||
* SELinux labels and hence different object checksum.
|
||||
*
|
||||
* The pure added content objects follow - content objects which won't be
|
||||
* generated when we import the packages. One interesting detail is right now
|
||||
* this includes the /usr/lib/tmpfiles.d/pkg-foo.conf objects that we generate
|
||||
* server side, because we don't generate that client side in jigdo mode.
|
||||
*
|
||||
* Finally, we have the xattr data, which is mostly in support of SELinux
|
||||
* labeling (note this is done on the server side still). In order to
|
||||
* dedup content, we have an xattr "string table" which is just an array
|
||||
* of xattrs; then there is a GVariant for each package which contains
|
||||
* a mapping of "objid" to an unsigned integer index into the xattr table.
|
||||
* The "objid" can either be a full path, or a basename if that basename is
|
||||
* unique inside a particular package.
|
||||
*/
|
||||
|
||||
/* Use a numeric prefix to ensure predictable ordering */
|
||||
#define RPMOSTREE_JIGDO_COMMIT_DIR "00commit"
|
||||
#define RPMOSTREE_JIGDO_PKGS "01pkgs"
|
||||
#define RPMOSTREE_JIGDO_PKGS_VARIANT_FORMAT (G_VARIANT_TYPE ("a(stssss)")) // NEVRA,repodata checksum
|
||||
#define RPMOSTREE_JIGDO_DIRMETA_DIR "02dirmeta"
|
||||
#define RPMOSTREE_JIGDO_DIRTREE_DIR "03dirtree"
|
||||
//#define RPMOSTREE_JIGDO_NEW_PKGIDENT "04new-pkgident"
|
||||
//#define RPMOSTREE_JIGDO_NEW_PKGIDENT_VARIANT_FORMAT (G_VARIANT_TYPE ("a{ua{s(sa(uuua(ayay)))}}")) // Map<pkgid,Map<path,Set<(checksum,uid,gid,mode,xattrs)>)>>
|
||||
#define RPMOSTREE_JIGDO_NEW_CONTENTIDENT_DIR "04new-contentident"
|
||||
#define RPMOSTREE_JIGDO_NEW_CONTENTIDENT_VARIANT_FORMAT (G_VARIANT_TYPE ("a(suuua(ayay))")) // checksum,uid,gid,mode,xattrs
|
||||
#define RPMOSTREE_JIGDO_NEW_DIR "05new"
|
||||
#define RPMOSTREE_JIGDO_XATTRS_DIR "06xattrs"
|
||||
#define RPMOSTREE_JIGDO_XATTRS_TABLE "06xattrs/00table"
|
||||
#define RPMOSTREE_JIGDO_XATTRS_PKG_DIR "06xattrs/pkg"
|
||||
|
||||
#define RPMOSTREE_JIGDO_XATTRS_TABLE_VARIANT_FORMAT (G_VARIANT_TYPE ("aa(ayay)"))
|
||||
/* NEVRA + xattr table */
|
||||
#define RPMOSTREE_JIGDO_XATTRS_PKG_VARIANT_FORMAT (G_VARIANT_TYPE ("a(su)"))
|
||||
171
src/libpriv/rpmostree-libarchive-input-stream.c
Normal file
171
src/libpriv/rpmostree-libarchive-input-stream.c
Normal file
@ -0,0 +1,171 @@
|
||||
/*
|
||||
* Copyright (C) 2011 Colin Walters <walters@verbum.org>
|
||||
*
|
||||
* This library is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU Lesser General Public
|
||||
* License as published by the Free Software Foundation; either
|
||||
* version 2 of the License, or (at your option) any later version.
|
||||
*
|
||||
* This library is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
* Lesser General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU Lesser General
|
||||
* Public License along with this library; if not, write to the
|
||||
* Free Software Foundation, Inc., 59 Temple Place, Suite 330,
|
||||
* Boston, MA 02111-1307, USA.
|
||||
*/
|
||||
|
||||
#include "config.h"
|
||||
|
||||
|
||||
#include <archive.h>
|
||||
#include <gio/gio.h>
|
||||
#include "rpmostree-libarchive-input-stream.h"
|
||||
|
||||
enum {
|
||||
PROP_0,
|
||||
PROP_ARCHIVE
|
||||
};
|
||||
|
||||
G_DEFINE_TYPE (RpmOstreeLibarchiveInputStream, _rpm_ostree_libarchive_input_stream, G_TYPE_INPUT_STREAM)
|
||||
|
||||
struct _RpmOstreeLibarchiveInputStreamPrivate {
|
||||
struct archive *archive;
|
||||
};
|
||||
|
||||
static void rpm_ostree_libarchive_input_stream_set_property (GObject *object,
|
||||
guint prop_id,
|
||||
const GValue *value,
|
||||
GParamSpec *pspec);
|
||||
static void rpm_ostree_libarchive_input_stream_get_property (GObject *object,
|
||||
guint prop_id,
|
||||
GValue *value,
|
||||
GParamSpec *pspec);
|
||||
static gssize rpm_ostree_libarchive_input_stream_read (GInputStream *stream,
|
||||
void *buffer,
|
||||
gsize count,
|
||||
GCancellable *cancellable,
|
||||
GError **error);
|
||||
static gboolean rpm_ostree_libarchive_input_stream_close (GInputStream *stream,
|
||||
GCancellable *cancellable,
|
||||
GError **error);
|
||||
|
||||
static void
|
||||
rpm_ostree_libarchive_input_stream_finalize (GObject *object)
|
||||
{
|
||||
G_OBJECT_CLASS (_rpm_ostree_libarchive_input_stream_parent_class)->finalize (object);
|
||||
}
|
||||
|
||||
static void
|
||||
_rpm_ostree_libarchive_input_stream_class_init (RpmOstreeLibarchiveInputStreamClass *klass)
|
||||
{
|
||||
GObjectClass *gobject_class = G_OBJECT_CLASS (klass);
|
||||
GInputStreamClass *stream_class = G_INPUT_STREAM_CLASS (klass);
|
||||
|
||||
g_type_class_add_private (klass, sizeof (RpmOstreeLibarchiveInputStreamPrivate));
|
||||
|
||||
gobject_class->get_property = rpm_ostree_libarchive_input_stream_get_property;
|
||||
gobject_class->set_property = rpm_ostree_libarchive_input_stream_set_property;
|
||||
gobject_class->finalize = rpm_ostree_libarchive_input_stream_finalize;
|
||||
|
||||
stream_class->read_fn = rpm_ostree_libarchive_input_stream_read;
|
||||
stream_class->close_fn = rpm_ostree_libarchive_input_stream_close;
|
||||
|
||||
/**
|
||||
* RpmOstreeLibarchiveInputStream:archive:
|
||||
*
|
||||
* The archive that the stream reads from.
|
||||
*/
|
||||
g_object_class_install_property (gobject_class,
|
||||
PROP_ARCHIVE,
|
||||
g_param_spec_pointer ("archive",
|
||||
"", "",
|
||||
G_PARAM_READWRITE |
|
||||
G_PARAM_CONSTRUCT_ONLY |
|
||||
G_PARAM_STATIC_STRINGS));
|
||||
|
||||
}
|
||||
|
||||
static void
|
||||
rpm_ostree_libarchive_input_stream_set_property (GObject *object,
|
||||
guint prop_id,
|
||||
const GValue *value,
|
||||
GParamSpec *pspec)
|
||||
{
|
||||
RpmOstreeLibarchiveInputStream *self = RPM_OSTREE_LIBARCHIVE_INPUT_STREAM (object);
|
||||
switch (prop_id)
|
||||
{
|
||||
case PROP_ARCHIVE:
|
||||
self->priv->archive = g_value_get_pointer (value);
|
||||
break;
|
||||
default:
|
||||
G_OBJECT_WARN_INVALID_PROPERTY_ID (object, prop_id, pspec);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
static void
|
||||
rpm_ostree_libarchive_input_stream_get_property (GObject *object,
|
||||
guint prop_id,
|
||||
GValue *value,
|
||||
GParamSpec *pspec)
|
||||
{
|
||||
RpmOstreeLibarchiveInputStream *self = RPM_OSTREE_LIBARCHIVE_INPUT_STREAM (object);
|
||||
|
||||
switch (prop_id)
|
||||
{
|
||||
case PROP_ARCHIVE:
|
||||
g_value_set_pointer (value, self->priv->archive);
|
||||
break;
|
||||
default:
|
||||
G_OBJECT_WARN_INVALID_PROPERTY_ID (object, prop_id, pspec);
|
||||
}
|
||||
}
|
||||
|
||||
static void
|
||||
_rpm_ostree_libarchive_input_stream_init (RpmOstreeLibarchiveInputStream *self)
|
||||
{
|
||||
self->priv = G_TYPE_INSTANCE_GET_PRIVATE (self,
|
||||
RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM,
|
||||
RpmOstreeLibarchiveInputStreamPrivate);
|
||||
|
||||
}
|
||||
|
||||
GInputStream *
|
||||
_rpm_ostree_libarchive_input_stream_new (struct archive *a)
|
||||
{
|
||||
return G_INPUT_STREAM (g_object_new (RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM,
|
||||
"archive", a, NULL));
|
||||
}
|
||||
|
||||
static gssize
|
||||
rpm_ostree_libarchive_input_stream_read (GInputStream *stream,
|
||||
void *buffer,
|
||||
gsize count,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
RpmOstreeLibarchiveInputStream *self = RPM_OSTREE_LIBARCHIVE_INPUT_STREAM (stream);
|
||||
|
||||
if (g_cancellable_set_error_if_cancelled (cancellable, error))
|
||||
return -1;
|
||||
|
||||
gssize res = archive_read_data (self->priv->archive, buffer, count);
|
||||
if (res < 0)
|
||||
{
|
||||
g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
|
||||
"%s", archive_error_string (self->priv->archive));
|
||||
}
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
static gboolean
|
||||
rpm_ostree_libarchive_input_stream_close (GInputStream *stream,
|
||||
GCancellable *cancellable,
|
||||
GError **error)
|
||||
{
|
||||
return TRUE;
|
||||
}
|
||||
66
src/libpriv/rpmostree-libarchive-input-stream.h
Normal file
66
src/libpriv/rpmostree-libarchive-input-stream.h
Normal file
@ -0,0 +1,66 @@
|
||||
/*
|
||||
* Copyright (C) 2011 Colin Walters <walters@verbum.org>
|
||||
*
|
||||
* This library is free software; you can redistribute it and/or
|
||||
* modify it under the terms of the GNU Lesser General Public
|
||||
* License as published by the Free Software Foundation; either
|
||||
* version 2 of the License, or (at your option) any later version.
|
||||
*
|
||||
* This library is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
* Lesser General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU Lesser General
|
||||
* Public License along with this library; if not, write to the
|
||||
* Free Software Foundation, Inc., 59 Temple Place, Suite 330,
|
||||
* Boston, MA 02111-1307, USA.
|
||||
*
|
||||
* Author: Alexander Larsson <alexl@redhat.com>
|
||||
*/
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <gio/gio.h>
|
||||
#include <archive.h>
|
||||
#include <archive_entry.h>
|
||||
|
||||
G_BEGIN_DECLS
|
||||
|
||||
#define RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM (_rpm_ostree_libarchive_input_stream_get_type ())
|
||||
#define RPM_OSTREE_LIBARCHIVE_INPUT_STREAM(o) (G_TYPE_CHECK_INSTANCE_CAST ((o), RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM, RpmOstreeLibarchiveInputStream))
|
||||
#define RPM_OSTREE_LIBARCHIVE_INPUT_STREAM_CLASS(k) (G_TYPE_CHECK_CLASS_CAST((k), RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM, RpmOstreeLibarchiveInputStreamClass))
|
||||
#define RPM_OSTREE_IS_LIBARCHIVE_INPUT_STREAM(o) (G_TYPE_CHECK_INSTANCE_TYPE ((o), RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM))
|
||||
#define RPM_OSTREE_IS_LIBARCHIVE_INPUT_STREAM_CLASS(k) (G_TYPE_CHECK_CLASS_TYPE ((k), RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM))
|
||||
#define RPM_OSTREE_LIBARCHIVE_INPUT_STREAM_GET_CLASS(o) (G_TYPE_INSTANCE_GET_CLASS ((o), RPM_OSTREE_TYPE_LIBARCHIVE_INPUT_STREAM, RpmOstreeLibarchiveInputStreamClass))
|
||||
|
||||
typedef struct _RpmOstreeLibarchiveInputStream RpmOstreeLibarchiveInputStream;
|
||||
typedef struct _RpmOstreeLibarchiveInputStreamClass RpmOstreeLibarchiveInputStreamClass;
|
||||
typedef struct _RpmOstreeLibarchiveInputStreamPrivate RpmOstreeLibarchiveInputStreamPrivate;
|
||||
|
||||
struct _RpmOstreeLibarchiveInputStream
|
||||
{
|
||||
GInputStream parent_instance;
|
||||
|
||||
/*< private >*/
|
||||
RpmOstreeLibarchiveInputStreamPrivate *priv;
|
||||
};
|
||||
|
||||
struct _RpmOstreeLibarchiveInputStreamClass
|
||||
{
|
||||
GInputStreamClass parent_class;
|
||||
|
||||
/*< private >*/
|
||||
/* Padding for future expansion */
|
||||
void (*_g_reserved1) (void);
|
||||
void (*_g_reserved2) (void);
|
||||
void (*_g_reserved3) (void);
|
||||
void (*_g_reserved4) (void);
|
||||
void (*_g_reserved5) (void);
|
||||
};
|
||||
|
||||
GType _rpm_ostree_libarchive_input_stream_get_type (void) G_GNUC_CONST;
|
||||
|
||||
GInputStream * _rpm_ostree_libarchive_input_stream_new (struct archive *a);
|
||||
|
||||
G_END_DECLS
|
||||
@ -837,3 +837,54 @@ rpmostree_diff_print (GPtrArray *removed,
|
||||
g_print (" %s\n", nevra);
|
||||
}
|
||||
}
|
||||
|
||||
/* Copy of ot_variant_bsearch_str() from libostree
|
||||
* @array: A GVariant array whose first element must be a string
|
||||
* @str: Search for this string
|
||||
* @out_pos: Output position
|
||||
*
|
||||
* Binary search in a GVariant array, which must be of the form 'a(s...)',
|
||||
* where '...' may be anything. The array elements must be sorted.
|
||||
*
|
||||
* Returns: %TRUE iff found
|
||||
*/
|
||||
gboolean
|
||||
rpmostree_variant_bsearch_str (GVariant *array,
|
||||
const char *str,
|
||||
int *out_pos)
|
||||
{
|
||||
const gsize n = g_variant_n_children (array);
|
||||
if (n == 0)
|
||||
return FALSE;
|
||||
|
||||
gsize imax = n - 1;
|
||||
gsize imin = 0;
|
||||
gsize imid = -1;
|
||||
while (imax >= imin)
|
||||
{
|
||||
const char *cur;
|
||||
|
||||
imid = (imin + imax) / 2;
|
||||
|
||||
g_autoptr(GVariant) child = g_variant_get_child_value (array, imid);
|
||||
g_variant_get_child (child, 0, "&s", &cur, NULL);
|
||||
|
||||
int cmp = strcmp (cur, str);
|
||||
if (cmp < 0)
|
||||
imin = imid + 1;
|
||||
else if (cmp > 0)
|
||||
{
|
||||
if (imid == 0)
|
||||
break;
|
||||
imax = imid - 1;
|
||||
}
|
||||
else
|
||||
{
|
||||
*out_pos = imid;
|
||||
return TRUE;
|
||||
}
|
||||
}
|
||||
|
||||
*out_pos = imid;
|
||||
return FALSE;
|
||||
}
|
||||
|
||||
@ -172,3 +172,8 @@ rpmostree_repo_auto_transaction_start (RpmOstreeRepoAutoTransaction *autotxn
|
||||
return TRUE;
|
||||
}
|
||||
G_DEFINE_AUTO_CLEANUP_CLEAR_FUNC (RpmOstreeRepoAutoTransaction, rpmostree_repo_auto_transaction_cleanup)
|
||||
|
||||
gboolean
|
||||
rpmostree_variant_bsearch_str (GVariant *array,
|
||||
const char *str,
|
||||
int *out_pos);
|
||||
|
||||
97
tests/compose-tests/test-jigdo.sh
Executable file
97
tests/compose-tests/test-jigdo.sh
Executable file
@ -0,0 +1,97 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -xeuo pipefail
|
||||
|
||||
dn=$(cd $(dirname $0) && pwd)
|
||||
. ${dn}/libcomposetest.sh
|
||||
. ${dn}/../common/libtest.sh
|
||||
|
||||
prepare_compose_test "jigdo"
|
||||
# Add a local rpm-md repo so we can mutate local test packages
|
||||
pyappendjsonmember "repos" '["test-repo"]'
|
||||
build_rpm test-pkg \
|
||||
files "/usr/bin/test-pkg" \
|
||||
install "mkdir -p %{buildroot}/usr/bin && echo localpkg data > %{buildroot}/usr/bin/test-pkg"
|
||||
# The test suite writes to pwd, but we need repos in composedata
|
||||
# Also we need to disable gpgcheck
|
||||
echo gpgcheck=0 >> yumrepo.repo
|
||||
ln yumrepo.repo composedata/test-repo.repo
|
||||
pyappendjsonmember "packages" '["test-pkg"]'
|
||||
# Need unified core for this, as well as a cachedir
|
||||
mkdir cache
|
||||
runcompose --ex-unified-core --cachedir $(pwd)/cache --add-metadata-string version=42.0
|
||||
npkgs=$(rpm-ostree --repo=${repobuild} db list ${treeref} |grep -v '^ostree commit' | wc -l)
|
||||
echo "npkgs=${npkgs}"
|
||||
rpm-ostree --repo=${repobuild} db list ${treeref} test-pkg >test-pkg-list.txt
|
||||
assert_file_has_content test-pkg-list.txt 'test-pkg-1.0-1.x86_64'
|
||||
|
||||
rev=$(ostree --repo=${repobuild} rev-parse ${treeref})
|
||||
mkdir jigdo-output
|
||||
do_commit2jigdo() {
|
||||
targetrev=$1
|
||||
rpm-ostree ex commit2jigdo --repo=repo-build --pkgcache-repo cache/pkgcache-repo ${targetrev} $(pwd)/composedata/fedora-atomic-host-oirpm.spec $(pwd)/jigdo-output
|
||||
(cd jigdo-output && createrepo_c .)
|
||||
}
|
||||
do_commit2jigdo ${rev}
|
||||
find jigdo-output -name '*.rpm' | tee rpms.txt
|
||||
assert_file_has_content rpms.txt 'fedora-atomic-host-42.0.*x86_64'
|
||||
|
||||
ostree --repo=jigdo-unpack-repo init --mode=bare-user
|
||||
echo 'fsync=false' >> jigdo-unpack-repo/config
|
||||
# Technically this isn't part of composedata but eh
|
||||
cat > composedata/jigdo-test.repo <<eof
|
||||
[jigdo-test]
|
||||
baseurl=file://$(pwd)/jigdo-output
|
||||
enabled=1
|
||||
gpgcheck=0
|
||||
eof
|
||||
do_jigdo2commit() {
|
||||
rpm-ostree ex jigdo2commit -d $(pwd)/composedata -e fedora-local -e test-repo -e jigdo-test --repo=jigdo-unpack-repo fedora-atomic-host | tee jigdo2commit-out.txt
|
||||
}
|
||||
do_jigdo2commit
|
||||
# there will generally be pkgs not in the jigdo set, but let's at least assert it's > 0
|
||||
assert_file_has_content jigdo2commit-out.txt '[1-9][0-9]* packages to import'
|
||||
ostree --repo=jigdo-unpack-repo rev-parse ${rev}
|
||||
ostree --repo=jigdo-unpack-repo fsck
|
||||
ostree --repo=jigdo-unpack-repo refs > jigdo-refs.txt
|
||||
assert_file_has_content jigdo-refs.txt 'rpmostree/pkg/test-pkg/1.0-1.x86__64'
|
||||
|
||||
echo "ok jigdo ♲📦 fresh assembly"
|
||||
|
||||
origrev=${rev}
|
||||
unset rev
|
||||
# Update test-pkg
|
||||
build_rpm test-pkg \
|
||||
version 1.1 \
|
||||
files "/usr/bin/test-pkg" \
|
||||
install "mkdir -p %{buildroot}/usr/bin && echo localpkg data 1.1 > %{buildroot}/usr/bin/test-pkg"
|
||||
# Also add an entirely new package
|
||||
build_rpm test-newpkg \
|
||||
files "/usr/bin/test-newpkg" \
|
||||
install "mkdir -p %{buildroot}/usr/bin && echo new localpkg data > %{buildroot}/usr/bin/test-newpkg"
|
||||
pyappendjsonmember "packages" '["test-newpkg"]'
|
||||
runcompose --ex-unified-core --cachedir $(pwd)/cache --add-metadata-string version=42.1
|
||||
newrev=$(ostree --repo=${repobuild} rev-parse ${treeref})
|
||||
rpm-ostree --repo=${repobuild} db list ${treeref} test-newpkg >test-newpkg-list.txt
|
||||
assert_file_has_content test-newpkg-list.txt 'test-newpkg-1.0-1.x86_64'
|
||||
|
||||
# Jigdo version 42.1
|
||||
do_commit2jigdo ${newrev}
|
||||
find jigdo-output -name '*.rpm' | tee rpms.txt
|
||||
assert_file_has_content rpms.txt 'fedora-atomic-host-42.1.*x86_64'
|
||||
|
||||
# And pull it; we should download the newer version by default
|
||||
do_jigdo2commit
|
||||
# Now we should only download 2 packages
|
||||
assert_file_has_content jigdo2commit-out.txt '2 packages to import'
|
||||
for x in ${origrev} ${newrev}; do
|
||||
ostree --repo=jigdo-unpack-repo rev-parse ${x}
|
||||
done
|
||||
ostree --repo=jigdo-unpack-repo fsck
|
||||
ostree --repo=jigdo-unpack-repo refs > jigdo-refs.txt
|
||||
# We should have both refs; GC will be handled by the sysroot upgrader
|
||||
# via deployments, same way it is for pkg layering.
|
||||
assert_file_has_content jigdo-refs.txt 'rpmostree/pkg/test-pkg/1.0-1.x86__64'
|
||||
assert_file_has_content jigdo-refs.txt 'rpmostree/pkg/test-pkg/1.1-1.x86__64'
|
||||
|
||||
echo "ok jigdo ♲📦 update!"
|
||||
23
tests/composedata/fedora-atomic-host-oirpm.spec
Normal file
23
tests/composedata/fedora-atomic-host-oirpm.spec
Normal file
@ -0,0 +1,23 @@
|
||||
# The canonical version of this is in https://pagure.io/fedora-atomic
|
||||
# Suppress most build root processing we are just carrying
|
||||
# binary data.
|
||||
%global __os_install_post /usr/lib/rpm/brp-compress %{nil}
|
||||
Name: fedora-atomic-host
|
||||
Version: %{ostree_version}
|
||||
Release: 1%{?dist}
|
||||
Summary: Image (rpm-ostree jigdo) for Fedora Atomic Host
|
||||
License: MIT
|
||||
|
||||
%description
|
||||
%{summary}
|
||||
|
||||
%prep
|
||||
|
||||
%build
|
||||
|
||||
%install
|
||||
mkdir -p %{buildroot}%{_prefix}/lib/ostree-jigdo/%{name}
|
||||
for x in *; do mv ${x} %{buildroot}%{_prefix}/lib/ostree-jigdo/%{name}; done
|
||||
|
||||
%files
|
||||
%{_prefix}/lib/ostree-jigdo/%{name}
|
||||
Loading…
Reference in New Issue
Block a user