IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Let's modernize and start supporting YAML treefiles. I'll dare make the
sweeping generalization that most people would prefer reading and
writing YAML over JSON.
This takes bits from coreos-assembler[1] that know how to serialize a
YAML file and spit it back out as a JSON and makes it into a shared lib
that we can link against. We could use this eventually for JSON inputs
as well to force a validation check before composing.
If we go this route, we could then turn on `--enable-rust` in FAHC for
now and drop the duplicate code in coreos-assembler.
[1] https://github.com/cgwalters/coreos-assemblerCloses: #1377
Approved by: cgwalters
This renames the remaining C files, tests, etc. There are only
a few hits for `jigdo` left; changing them would be a format break,
so let's wait to do that until we need to.
Closes: #1279
Approved by: jlebon
We're continuing an incremental renaming process; previously we changed
the most user-visible strings. Now we're doing some internal variables,
and notably the cached refs and the origin files - the latter set is
things that end up on disk.
This leaves the biggest items; renaming APIs, files, and tests.
Closes: #1276
Approved by: jlebon
I noticed that Ubuntu also uses the original "jigdo", so let's start
pulling off the band-aid here and do a mass rename.
For this first pass I'm focusing on CLI entrypoints and docs, as that's what
people are going to see; renaming all of the internal C functions, structure
variables etc. can come later.
Closes: #1269
Approved by: jlebon
Changes in a server-side tree can cause the need for clients to import different
objects from packages. For example, turning on documentation. Another more
subtle case is where an object might "move" from package A to B by being deleted
from A - then the jigdo build process will pick the B version.
We need a "cache validation key"; a way for the server to tell the client that
the objects it should import from the package have changed. Initially I was
thinking of using the libostree "content hash" but that would be awkward as we'd
have to do an import on the server side too.
After more consideration I realized a simple *count* of the number of objects
actually works, because (as I note in a comment) changing a file in the tree
will result in it ending up in the jigdoRPM (and count as a deletion). And
obviously adding or removing objects changes the count too.
In fact we could have done this *without* breaking the format by just having the
client start recording the number of xattr entries, but this adds greater
flexibility down the line since we can in theory change how we do cache
invalidation if we *really* need to (but at the cost of triggering clients to
redownload packages).
Note the client logic got moved around as now we need to parse all the xattrs
before we decide what packages to download.
My test case here is turning on docs - I noticed this actually affects *every*
package which was surprising to me; I expected at least some packages wouldn't
have docs. I'll double check this.
It'd be good to have a "moving object" case too which I may look at.
Closes: https://github.com/projectatomic/rpm-ostree/issues/1197Closes: #1256
Approved by: jlebon
Since we changed things to have `jigdoSet = pkgSet`, we can just require exactly
`${npkgs}` here on import, which is what we found from `db list`.
Closes: #1256
Approved by: jlebon
The way we import packages in jigdo mode is different from package layering; we
may only import a subset of files for example. In general, we need to treat
jigdo differently.
Related: https://github.com/projectatomic/rpm-ostree/issues/1197Closes: #1238
Approved by: jlebon
I *think* this is why our tests started failing recently. It seems somehow very
unlikely to me though that we'd somehow managed to avoid `f` in the boot
checksums until now, but without doing some math...it seems plausible.
Closes: #1243
Approved by: jlebon
In #875 AKA b46fc35901 we
added support for the `releasever` option in treefiles. I am
pretty sure it worked at the time...but I didn't add tests.
Either it never worked or some refactoring broke it. The whole chain of
`GKeyFile` → `GVariant` is so confusing. Anyways fix it by copying the string.
Now let's use it by default in the compose tests, and while we're here bump
those to F27.
I'm doing this patch now as I was playing with doing a compose from
the `/usr/share/rpm-ostree/treefile.json` and wanted to use the stock
`.repo` files.
Closes: #1220
Approved by: jlebon
When I tried to use my WIP client patches to do:
`rpm-ostree rebase rojig://fahc:fedora-atomic-host`,
I got a missing file object which turned out to
be the client importing the i686 RPMs.
This was passing in the test suite because we don't mirror i686 of course, but
on the client side right now we end up using all enabled repos, and since Fedora
is multiarch, the behavior is going to be...not predictable.
Thinking a bit about on this problem I actually happened to recall
the RPM `%{_isa}` macro which is used in Fedora in various places;
for example to "arch bind" `-devel` packages to their base. See
for example [this case](33c7dc02bc/f/ostree.spec (_79)) in libostree.
As I noted at first, the core problem here is that the "final"
RPM architecture field is not symmetric in any way with the definition
of that `%{_isa}` macro. See:
d9d47e0114/installplatform (L25)
The *third* solution I ended up on here is to iterate over the
`Provides` on the server side and we take the first thing
that matches `Provides: %{name}(whatever)`.
I briefly thought about trying to somehow drive into libsolv the
logic to prefer the jigdoRPM's native architecture...IIRC yum did
something like that in the past but it was never done in libsolv?
Anyways the dependencies here are now more correct, so other tools
will also handle it.
Closes: #1213
Approved by: jlebon
Following up on `--ex-jigdo-output-rpm`, add support for writing the entire set
to an output directory. This is intended for use cases like FAHC, where we're
generating data outside of the upstream Fedora infrastructure. Further, we want
to support having our own history stored reliably, even if upstream prunes RPMs.
Now, this can be interesting even for upstreams like Fedora, as it naturally
captures just the subset of RPMs; doing full history support for that would
likely be a lot more palatable than for Everything.
Closes: #1165
Approved by: jlebon
Part of the goal of jigdo ♲📦 is to support organizations switching to *only*
providing RPMs. An intermediate step there is to "lock" the repo and jigdo
together; we don't want to update the ref if building the jigdoRPM fails.
Add an option to perform `rpm-ostree compose tree` and `rpm-ostree ex
commit2jigdo` together; notably we generate a commit, but only update the ref
once the jigdoRPM is built.
Closes: #1165
Approved by: jlebon
- Actually use separate `${test_tmpdir}` for test setup (closes a race)
- Merge stdout/stderr (more readable)
- Ensure logs are renamed to `.txt` even on failure
- Use `--progress` for some feedback
- Use `-j +1` so that even on unicore machines we get at least 2
jobs (and in general NCPUS+1)
Closes: #1188
Approved by: jlebon
Basically the `rpmostree_context_relabel()` call we had in the treecompose path
for unified core didn't actually have any effect as the core code did a relabel
and unset the array.
I think this may actually be a regression from: https://github.com/projectatomic/rpm-ostree/pull/1137
though I didn't verify.
Anyways looking at this, the code is a lot simpler if we change the API so that
the "normal" relabeling is folded into `rpmostree_context_assemble()`. Then we
change the public relabel API to be "force relabel" which we use in the unified
core 🌐 treecompose path.
This shrinks the jigdoRPM for FAH from 90MB to 68MB.
Closes: https://github.com/projectatomic/rpm-ostree/issues/1172Closes: #1173
Approved by: jlebon
This is used in critical paths like pungi, so let's be sure it works;
the semantics are a bit subtle as it overrides setting the ref.
Closes: #1161
Approved by: jlebon
The new `rpmdb.pkglist` metadata is a cheap way of retrieving the set of
packages in a commit. I'd like to make use of it as much as possible vs.
checking out the rpmdb and setting up a DnfSack.
Of course, in the case of layered commits, it doesn't matter *as* much,
because a layered commit being present in the repo should mean that a
deployment is currently using it, and we should learn to reuse the rpmdb
checkout of that deployment. Though keeping it consistent across both
server and client commits makes implementing `OstreeDeployment`-agnostic
things like `db diff` more efficient too. I also plan to use this in the
upcoming auto-update code.
Closes: #1158
Approved by: cgwalters
This fixes another thing broken with `compose --ex-unified-core`;
for e.g. `/usr/bin/ping` from `iputils`, the classic example of a filecaps
binary.
As I'm writing this commit message I realize it will actually also
take effect for package layering unnecessarily; we'll pointlessly
break the hardlink. But eh, it doesn't matter right now, we can
optimize that later.
Closes: #1151
Approved by: jlebon
Rather than entirely symlinking `systemctl` → `/bin/true`, in order
to e.g. have NetworkManager be enabled, we need to process presets.
This is one of the things that's breaking FAHC where I did a
`--ex-unified-core` deployment.
(Actually it's a bit tempting to run a mass preset pass at the end,
but for now let's do this)
Implementation note: this is our first use of GResources, which
is a handy way to embed data into our final binary.
Closes: https://github.com/projectatomic/rpm-ostree/issues/550Closes: #1148
Approved by: jlebon
This is prep for a rework of
https://github.com/projectatomic/rpm-ostree/pull/621
For a no-op `rpm-ostree upgrade` (i.e. no updates available), as long as
layering is enabled, we pay the cost of checking out the base tree, *mostly*
only to get the base rpmdb.
This is prep for fixing that down the line by knowing we always have the "base"
tree's rpmdb checked out. Then in the layering case we only modify
`/usr/share/rpm` (eventually that will point to `/usr/lib/sysimage/rpm`).
Teaching `rpmostree-core.c` about this can follow on later.
Closes: #1142
Approved by: jlebon
Having the "jigdo set" in repodata makes it so we can parallel download the
jigdo RPM with the set. However for now, I kept the jigdo set in the jigdoRPM,
since that way it'll be covered by the signature.
Also, this changes the way we inject metadata to use a magic comment string,
since trying to pass a gigantic macro to `rpmbuild` via its argv didn't work out
so well (it looks like rpmbuild eats newlines). This approach is more robust.
Closes: https://github.com/projectatomic/rpm-ostree/issues/1132Closes: #1140
Approved by: jlebon
With unified core 🌐 we are *relying* on the devino cache
for correctness when using `bare-user` repos. Otherwise lots
of bad things will happen as we won't hit the happy path from
[this libostree PR](https://github.com/ostreedev/ostree/pull/1297)
(I should probably add an assertion there that we aren't trying to commit
`user.ostreemeta`).
It looks like I had this working in some of the old unified core WIP patches,
but it was lost when rebasing 🏄.
We noticed this when I was trying to deploy jigdo in FAHC and the system
wouldn't boot as various things rely on those suid transitions.
Closes: #1139
Approved by: jlebon
We don't want to have to download all of `/usr/share/rpm` just to get
the list of packages used to compose the tree. This is fundamental
information that needs to be easier to discover. So let's stick it right
in the commit metadata. There's various use cases for this information,
including easily checking for and displaying updates and a pkglist-aware
version of `ostree log`.
Closes: #1134
Approved by: cgwalters
Let's "repo bind" the OIRPM by default; this makes the rpm-md repo feel a lot
more like an ostree remote, and IMO is just a really good idea in general to
increase predictabilty.
Closes: #1130
Approved by: jlebon
Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081
To briefly recap: Let's experiment with doing ostree-in-RPM, basically the
"compose" process injects additional data (SELinux labels for example) in an
"ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree
image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we
need need to download. For updates, like `yum update` we only download changed
RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still
done server side, and we still have a reliable OSTree commit checksum.
This is a lot like [Jigdo](http://atterer.org/jigdo/)
Here we fully demonstrate the concept working end-to-end; we use the
"traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which
has a checksum, version etc. Then the new `ex commit2jigdo` generates the
"oirpm". This is the "server side" operation. Next simulating the client side,
`jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs,
fully regenerating *bit for bit* the final OSTree commit.
If you want to play with this, I'd take a look at the `test-jigdo.sh`; from
there you can find other useful bits like the example `fedora-atomic-host.spec`
file (though the canonical copy of this will likely land in the
[fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo.
Closes: #1103
Approved by: jlebon
When we added the `--ex-unified-core` option our caching story got
very messy because the non-unified core caches RPMs, but unified
does ostree repo caching.
For jigdo, we want the RPMs. Fix this by mirroring the RPMs using
`--download-only` and pointing the tests consistently at that.
Closes: #1122
Approved by: jlebon
We removed this in review, but I rediscovered why I added it. We
fail the `g_assert (sepolicy_matches)` if we already had packages
done with the final label when we go to reuse the cache.
(Basically, if we use the cache multiple times it's hard to avoid
relabeling all the time which is unfortunate...gets back a bit to
a way to annotate pkgcache commits as supporting multiple policies)
Closes: #1109
Approved by: jlebon
The "--ex" prefix here means it's an experimental option. A tremendous change
here is that start to support non-uid 0, but there are various things to fix there;
the unpacker for example needs to learn to set imported objects fully based
on the rpmfi information (i.e. default to uid 0, since libarchive gives the
current uid by default).
And even when run as uid 0, there are some bugs, though I'm not sure
of any showstoppers yet. For example, dracut's `dracut-install` calls
`cp --preserve=xattrs` which fails to copy the `user.ostreemeta` xattrs
from a checkout (it shouldn't be copying that anyways...)
Nevertheless, the infrastructure behind this really helps (is almost a hard
requirement for) the [jigdo effort](https://github.com/projectatomic/rpm-ostree/issues/1081).
Which is really only true due to SELinux - we need to import the packages,
then generate the final tree to get the final policy, then use that policy
to relabel all of the packages.
Closes: #940
Approved by: jlebon
I was trying to figure out why:
rpm-ostree compose tree --repo repo/ manifest.json
would result in:
error: opendir(manifest.json): No such file or directory
It turned out to be because we had `--repo` in *both* the `install`
options and the `commit` options. This makes sense since both these
subcommands need to be given a repo. However, in the `tree` case, we
were adding both arrays, which meant we inherited two `GOptionEntry`s
for `--repo`. This confused glib2 which consumed not one but two CLI
arguments when looking for the argument associated with `--repo`.
Our CI didn't notice this because it uses the `--repo=foo` notation,
which doesn't throw off glib2.
Fix this by factoring out the `--repo` option into a separate array so
that in the `tree` case, it only gets added once. Exercise the fix in CI
by using the two argument notation for `--repo`.
Closes: #1101
Approved by: cgwalters
This is a revisit of a PR for client-side layering: https://github.com/projectatomic/rpm-ostree/pull/1072
Here though we're doing this by default for server-side composes.
There are a few reasons to do this; first, I'm seeing an issue
in some of our Jenkins jobs for Fedora that hit "mirror roulette"
and end up creating commits that "revert" to older versions temporarily.
While I've [certainly pitched](https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/IMPE6KCRBHCEJH5VBE6ZFIRLPAD743JT/) this as a feature, I think
we really want something like `--force-older-timestamp` - basically
error out if the timestamps on one or more input repos were older.
Not doing that in this patch, but it paves the way to do so.
Second, I'd like to use this data in the `ostree.source-title`
metadata key down the line. Something like:
`└ rpmmd: fedora-26 (20170310), fedora-26-updates (20171101)`
(This could be a lot nicer if we drive versioning in to the rpm-md repo info,
and e.g. there's some friendly "week number" style versioning for the updates
repo now that it's batched...for now we have timestamps)
For CentOS/RHELAH this gets interesting and potentially more verbose,
to the point where we may want to render it more explicitly.
But anyways, let's do this now, as it will be useful even without
an explicit rendering, since users can do e.g. `ostree show` on
a base commit hash to dump the data.
I had a concern that some users may not want to emit this metadata;
they can currently do `--add-metadata-string rpmostree.rpmmd-repos ''`
and that will "win".
Closes: #1079
Approved by: jlebon
Related to: https://github.com/projectatomic/rpm-ostree/issues/49
We want to support "name binding" per client system, rather than
having a hardcoded mapping in our tree. Currently if e.g. a new
daemon is added as a dependency (or as part of e.g. systemd) it's
easy to silently miss it.
This is prep for doing that binding client side consistently, which is what we
do with package layering.
Closes: #1077
Approved by: jlebon
Right now `rpm-ostree compose tree` is very prescriptive about how things work.
Trying to add anything that isn't an RPM is absolutely fighting the system. Our
postprocessing system *enforces* no network access (good for reproducibilty, but
still prescriptive).
There's really a logical split between three phases:
- install: "build a rootfs that installs packages"
- postprocess: "run magical ostree postprocessing like kernel"
- commit: "commit result to ostree"
So there are two high level flows I'd like to enable here. First is to allow
people to do *arbitrary* postprocessing between `install` and `commit`. For
example, run Ansible and change `/etc`. This path basically is like what we have
today with `postprocess-script.sh`, except the builder can do anything they want
with network access enabled.
Going much farther, this helps us support a "build with Dockerfile" style flow.
We can then provide tooling to extract the container image, and combine
`postprocess` and `commit`.
Or completely the other way - if for example someone wants to use `rpm-ostree
compose install`, they could tar up the result as a Docker/OCI image. That's now
easier; an advantage of this flow over e.g. `yum --installroot` is the "change
detection" code we have.
Related issues/PRs:
- https://github.com/projectatomic/rpm-ostree/pull/96
- https://github.com/projectatomic/rpm-ostree/issues/471
One disadvantage of this approach right now is that if one *does* go for
the split approach, we lose the "input hash" metadata for example. And
down the line, I'd like to add even more metadata, like the input rpm repos,
which could also be rendered on the client side.
But, I think we can address that later by e.g. caching the metadata in a file in
the install root and picking it back up or something.
Closes: #1039
Approved by: jlebon
Sometimes it's useful to have access to the additional files when running
the post script, so this re-orders the compose process to copy the
additional files in before the post script runs
Signed-off-by: Jonathan Dieter <jdieter@lesbg.com>
Closes: #997
Approved by: jlebon
Even though it's really `/usr/etc`. This is for greater consistency with
`postprocess-script` where it appears as `/etc`.
Closes: #997
Approved by: jlebon
The comment here was wrong; we don't rely on `O_APPEND` here for package
layering since we convert on import. I noticed this while I was doing
a grep for `O_APPEND` in the codebase as part of unified core work.
Fix this by converting to `O_TMPFILE`+`GLNX_LINK_TMPFILE_NOREPLACE`.
Prep for unified core.
Closes: #1009
Approved by: jlebon
We don't need those in the tree, so let's nuke them. This also fixes
subtle compatibility issues between hardlinks and lock files (see #999).
Closes: #1002
Approved by: cgwalters
Prep for changing `boot_location: new` to use `/usr/lib/ostree-boot`
and `/usr/lib/modules`. Rework our kernel postprocessing
so that we unify the `boot_location` handling with initramfs generation.
Instead of doing the initramfs first in postprocessing, we do it nearly last,
after e.g. `etc` is renamed to `usr/etc`. This has some consequences, such as
the fact that `run_bwrap_mutably()` is now called in both situations. In
general, our handling of `etc` is inconsistent, although understandably so.
As part of this, I finally got around to implementing the bit from
https://github.com/systemd/systemd/pull/4174 however suboptimal it is; need the
unified core so we can cleanly ignore the posttrans like we do others. We
intentionally keep the file around in the generated tree so that installing a
kernel RPM per client doesn't try to do any of this either.
This all gets folded together so that the logic for handling the bootloader gets
simpler - in the Fedora case, we now know to find kernels in `/usr/lib/modules`
and can ignore `/boot`.
Closes: #959
Approved by: jlebon
We can be a bit less wasteful here by merging the check and vmcheck
suites into a single suite. The check suite today takes a negligible
amount of time to run, so we're not gaining much by parallelizing them.
It's more of a sanity check at this point before we start vmcheck.
Also start running vmcheck on CentOS 7. We adapt the ci scripts to
accomodate both Fedora and CentOS target machines.
This commit also switches to Fedora 26 as the primary test base.
Closes: #871
Approved by: cgwalters
There are a few reasons to do this. First, systemd changed to refuse mounts on
symlinks, and hence if one *wants* "/tmp-on-tmpfs", one would need to write a
different `sysroot-tmp.mount` unit.
Second, the original rationale for having this symlink was that if you had
multiple ostree stateroots ("osnames"), it's nicer if they had the same `/tmp`
to avoid duplication. But in practice today that's already an issue due to
`/var/tmp`, and further the multiple-stateroot case is pretty unusual. And that
case is *further* broken by SELinux (if one wanted to have e.g. an Ubuntu and
Fedora) stateroots. So let's fully decouple this and make `/tmp` a plain
old directory by default, so systemd's `tmp.mount` can become useful.
Now, things get interesting for the case where someone wants a physical `/tmp`
that *does* persist across reboots. Right now, if one just did a `systemctl mask
tmp.mount` as we do in Fedora Atomic Host's cloud images, you'd get a semantic
where `/tmp` stays per-deployment, which is weird. Our recommendation for
that should likely be to set up a bind mount for `/tmp` → `/var/tmp`.
For now, this stays an option to ensure compatibility; if FAH Cloud images
want to stay with "physical /tmp", then we'd have to change the kickstart.
Closes: https://github.com/projectatomic/rpm-ostree/issues/669Closes: #778
Approved by: jlebon
I was about to add another one of these but it feels like a bit
overkill to run through a recompose for trivial tweaks like turning off docs.
We can do a compose with multiple options at once and test the result as a unit,
at least for the smaller/less invasive options.
This change is prep for adding a switch to do `/tmp` as a regular dir.
Closes: #777
Approved by: jlebon
There's no point to shipping these backup files in the base tree. We already had
code to delete them for the package layering case where they caused active harm.
At the point we added that code we really should have *also* changed treecompose
to delete them. Better late than never.
The reason I'm doing this now is because having them in the base tree causes `ex
livefs` to spuriously think that layering a package that *doesn't* change `/etc`
as if it does, because the layering code deletes the backup files.
Closes: #693
Approved by: jlebon
I'd like to embed structured metadata about the originating git
repository. See [this example](https://pagure.io/fedora-atomic-host-continuous/c/142b12020d7efe18b56d039304efea102a210790?branch=master). However, I think what we really
want here is a *single* value which has subkeys.
One thing in the back of my mind too is...we could use this to
enhance our "change detection". Right now we checksum the sack,
treefile, and treecompose-post. But down the line, I'd
like to support more sophisticated postprocessing, where the
script might reference external files or the like.
In that case, we could stop checksumming the post script, and rely on whether or
not the git repo changed. (This would conversely mean we would do a build even
if e.g. the repo's `README.md` changed, but we can address that with a
post-assemble content check).
Anyways though, for now, this gets us the ability to more easily drop more
structured metadata in the commit, whether it's input git repos, tests that
passed, etc.
Note a trap that bit me here: since the metadata we write here is *host* endian,
but `ostree show --raw` byteswaps (it needs to since the core ostree variant
is always big endian), we get inverted numbers if the host is little.
I think we should probably canonicalize our metadata to big endian; this should
be pretty backwards compatible since I doubt anyone has been adding raw numbers
so far.
Closes: #676
Approved by: jlebon
Add a few more tests to exercise some of the treefile options. We do
need to also expand test-basic.sh itself to sanity-check the structure
of a normal ostree compose. That's up next on the list.
Closes: #548
Approved by: cgwalters
Our current compose tests only use a synthetic `empty.rpm`, but
this really limits usefulness.
Let's make a test suite that requires an internet connection and
downloads Fedora RPMs and does "real" tree composes.
See the updated `tests/README.md` for more information.
This is still a WIP.
Closes: #531
Approved by: jlebon
