Commit Graph

13 Commits

Author SHA1 Message Date
Colin Walters
3c9e212f71 compose-tests: Use yaml.safe_dump
Otherwise we end up with weird `!!python/unicode` stuff:
https://stackoverflow.com/questions/20352794/pyyaml-is-producing-undesired-python-unicode-output/20369984

Closes: #1597
Approved by: jlebon
2018-10-02 18:03:22 +00:00
Colin Walters
e208383526 tests: Add rojig spec into base config
This is prep for fully dropping the separate spec file, which
is now just used by `commit2rojig`.

A compose test can now specify it wants YAML instead of JSON.

Closes: #1561
Approved by: jlebon
2018-09-17 23:45:57 +00:00
Colin Walters
800402b00a tests/compose: Add some time logging
To help us profile.

Closes: #1479
Approved by: jlebon
2018-08-01 18:05:09 +00:00
Jonathan Lebon
479406e6a5 Add support for YAML treefiles
Let's modernize and start supporting YAML treefiles. I'll dare make the
sweeping generalization that most people would prefer reading and
writing YAML over JSON.

This takes bits from coreos-assembler[1] that know how to serialize a
YAML file and spit it back out as a JSON and makes it into a shared lib
that we can link against. We could use this eventually for JSON inputs
as well to force a validation check before composing.

If we go this route, we could then turn on `--enable-rust` in FAHC for
now and drop the duplicate code in coreos-assembler.

[1] https://github.com/cgwalters/coreos-assembler

Closes: #1377
Approved by: cgwalters
2018-06-05 13:08:33 +00:00
Colin Walters
b1c92a6a1d tests/compose: Various fixes
- Actually use separate `${test_tmpdir}` for test setup (closes a race)
 - Merge stdout/stderr (more readable)
 - Ensure logs are renamed to `.txt` even on failure
 - Use `--progress` for some feedback
 - Use `-j +1` so that even on unicore machines we get at least 2
   jobs (and in general NCPUS+1)

Closes: #1188
Approved by: jlebon
2018-01-10 15:16:18 +00:00
Colin Walters
d9d8eefdc5 tests/compose: Disable fsync
I've been running these a lot locally and this helps a bit.

Closes: #1103
Approved by: jlebon
2017-12-04 14:24:53 +00:00
Colin Walters
dafb3d6daa tests/compose: Rework caching to cache RPMs
When we added the `--ex-unified-core` option our caching story got
very messy because the non-unified core caches RPMs, but unified
does ostree repo caching.

For jigdo, we want the RPMs. Fix this by mirroring the RPMs using
`--download-only` and pointing the tests consistently at that.

Closes: #1122
Approved by: jlebon
2017-12-01 19:20:40 +00:00
Jonathan Lebon
00fab1ec23 compose CLI: fix --repo consuming two arguments
I was trying to figure out why:

    rpm-ostree compose tree --repo repo/ manifest.json

would result in:

    error: opendir(manifest.json): No such file or directory

It turned out to be because we had `--repo` in *both* the `install`
options and the `commit` options. This makes sense since both these
subcommands need to be given a repo. However, in the `tree` case, we
were adding both arrays, which meant we inherited two `GOptionEntry`s
for `--repo`. This confused glib2 which consumed not one but two CLI
arguments when looking for the argument associated with `--repo`.

Our CI didn't notice this because it uses the `--repo=foo` notation,
which doesn't throw off glib2.

Fix this by factoring out the `--repo` option into a separate array so
that in the `tree` case, it only gets added once. Exercise the fix in CI
by using the two argument notation for `--repo`.

Closes: #1101
Approved by: cgwalters
2017-11-13 14:13:27 +00:00
Colin Walters
63af4bbdda bin/compose: Expose phases as [install, postprocess, commit] cmds
Right now `rpm-ostree compose tree` is very prescriptive about how things work.
Trying to add anything that isn't an RPM is absolutely fighting the system. Our
postprocessing system *enforces* no network access (good for reproducibilty, but
still prescriptive).

There's really a logical split between three phases:

 - install: "build a rootfs that installs packages"
 - postprocess: "run magical ostree postprocessing like kernel"
 - commit: "commit result to ostree"

So there are two high level flows I'd like to enable here. First is to allow
people to do *arbitrary* postprocessing between `install` and `commit`. For
example, run Ansible and change `/etc`. This path basically is like what we have
today with `postprocess-script.sh`, except the builder can do anything they want
with network access enabled.

Going much farther, this helps us support a "build with Dockerfile" style flow.
We can then provide tooling to extract the container image, and combine
`postprocess` and `commit`.

Or completely the other way - if for example someone wants to use `rpm-ostree
compose install`, they could tar up the result as a Docker/OCI image. That's now
easier; an advantage of this flow over e.g. `yum --installroot` is the "change
detection" code we have.

Related issues/PRs:

 - https://github.com/projectatomic/rpm-ostree/pull/96
 - https://github.com/projectatomic/rpm-ostree/issues/471

One disadvantage of this approach right now is that if one *does* go for
the split approach, we lose the "input hash" metadata for example.  And
down the line, I'd like to add even more metadata, like the input rpm repos,
which could also be rendered on the client side.

But, I think we can address that later by e.g. caching the metadata in a file in
the install root and picking it back up or something.

Closes: #1039
Approved by: jlebon
2017-10-25 17:43:09 +00:00
Jonathan Lebon
cf19d83502 ci: unite testsuites and run vmcheck on centos
We can be a bit less wasteful here by merging the check and vmcheck
suites into a single suite. The check suite today takes a negligible
amount of time to run, so we're not gaining much by parallelizing them.
It's more of a sanity check at this point before we start vmcheck.

Also start running vmcheck on CentOS 7. We adapt the ci scripts to
accomodate both Fedora and CentOS target machines.

This commit also switches to Fedora 26 as the primary test base.

Closes: #871
Approved by: cgwalters
2017-07-18 13:58:38 +00:00
Jonathan Lebon
4f0be52dca compose-tests: add more mutate-os-release cases
Check that the logic added in the previous commit is respected.

Closes: #603
Approved by: cgwalters
2017-02-01 22:25:54 +00:00
Jonathan Lebon
68acb1d2be compose-tests: add more tests
Add a few more tests to exercise some of the treefile options. We do
need to also expand test-basic.sh itself to sanity-check the structure
of a normal ostree compose. That's up next on the list.

Closes: #548
Approved by: cgwalters
2016-12-09 16:18:11 +00:00
Colin Walters
b81c0cdfda tests: Add ./tests/compose
Our current compose tests only use a synthetic `empty.rpm`, but
this really limits usefulness.

Let's make a test suite that requires an internet connection and
downloads Fedora RPMs and does "real" tree composes.

See the updated `tests/README.md` for more information.

This is still a WIP.

Closes: #531
Approved by: jlebon
2016-12-06 19:05:05 +00:00