keremet/rpm-ostree
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4c3d3fcb45
rpm-ostree/tests/compose/test-basic.sh

76 lines
2.5 KiB
Bash
Raw Normal View History

tests: Add ./tests/compose Our current compose tests only use a synthetic `empty.rpm`, but this really limits usefulness. Let's make a test suite that requires an internet connection and downloads Fedora RPMs and does "real" tree composes. See the updated `tests/README.md` for more information. This is still a WIP. Closes: #531 Approved by: jlebon
2016-12-03 02:07:08 +03:00
#!/bin/bash
set -xeuo pipefail
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
# XXX: nuke this test once we fully drop non-unified core mode
dn=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=libcomposetest.sh
. "${dn}/libcomposetest.sh"
# Add a local rpm-md repo so we can mutate local test packages
treefile_append "repos" '["test-repo"]'
# test `recommends: false` (test-misc-tweaks tests the true path)
build_rpm foobar recommends foobar-rec
build_rpm foobar-rec
echo gpgcheck=0 >> yumrepo.repo
ln "$PWD/yumrepo.repo" config/yumrepo.repo
treefile_append "packages" '["foobar"]'
treefile_pyedit "tf['add-commit-metadata']['foobar'] = 'bazboo'"
treefile_pyedit "tf['add-commit-metadata']['overrideme'] = 'old var'"
tests: Add ./tests/compose Our current compose tests only use a synthetic `empty.rpm`, but this really limits usefulness. Let's make a test suite that requires an internet connection and downloads Fedora RPMs and does "real" tree composes. See the updated `tests/README.md` for more information. This is still a WIP. Closes: #531 Approved by: jlebon
2016-12-03 02:07:08 +03:00
compose: Add --add-metadata-from-json I'd like to embed structured metadata about the originating git repository. See [this example](https://pagure.io/fedora-atomic-host-continuous/c/142b12020d7efe18b56d039304efea102a210790?branch=master). However, I think what we really want here is a *single* value which has subkeys. One thing in the back of my mind too is...we could use this to enhance our "change detection". Right now we checksum the sack, treefile, and treecompose-post. But down the line, I'd like to support more sophisticated postprocessing, where the script might reference external files or the like. In that case, we could stop checksumming the post script, and rely on whether or not the git repo changed. (This would conversely mean we would do a build even if e.g. the repo's `README.md` changed, but we can address that with a post-assemble content check). Anyways though, for now, this gets us the ability to more easily drop more structured metadata in the commit, whether it's input git repos, tests that passed, etc. Note a trap that bit me here: since the metadata we write here is *host* endian, but `ostree show --raw` byteswaps (it needs to since the core ostree variant is always big endian), we get inverted numbers if the host is little. I think we should probably canonicalize our metadata to big endian; this should be pretty backwards compatible since I doubt anyone has been adding raw numbers so far. Closes: #676 Approved by: jlebon
2017-03-06 19:20:25 +03:00
# Test metadata json with objects, arrays, numbers
cat > metadata.json <<EOF
{
"exampleos.gitrepo": {
"rev": "97ec21c614689e533d294cdae464df607b526ab9",
"src": "https://gitlab.com/exampleos/custom-atomic-host"
},
treefile: Add new `add-commit-metadata` key Add support for a new `add-commit-metadata` key in the treefile so that we can directly specify commit metadata we want to inject from there. This will be useful in Fedora CoreOS, where we'll have separate treefiles for each streams, each with stream-specific metadata values required. Closes: #1865 Approved by: cgwalters
2019-07-08 17:34:59 +03:00
"exampleos.tests": ["smoketested", "e2e"],
"overrideme": "new val"
compose: Add --add-metadata-from-json I'd like to embed structured metadata about the originating git repository. See [this example](https://pagure.io/fedora-atomic-host-continuous/c/142b12020d7efe18b56d039304efea102a210790?branch=master). However, I think what we really want here is a *single* value which has subkeys. One thing in the back of my mind too is...we could use this to enhance our "change detection". Right now we checksum the sack, treefile, and treecompose-post. But down the line, I'd like to support more sophisticated postprocessing, where the script might reference external files or the like. In that case, we could stop checksumming the post script, and rely on whether or not the git repo changed. (This would conversely mean we would do a build even if e.g. the repo's `README.md` changed, but we can address that with a post-assemble content check). Anyways though, for now, this gets us the ability to more easily drop more structured metadata in the commit, whether it's input git repos, tests that passed, etc. Note a trap that bit me here: since the metadata we write here is *host* endian, but `ostree show --raw` byteswaps (it needs to since the core ostree variant is always big endian), we get inverted numbers if the host is little. I think we should probably canonicalize our metadata to big endian; this should be pretty backwards compatible since I doubt anyone has been adding raw numbers so far. Closes: #676 Approved by: jlebon
2017-03-06 19:20:25 +03:00
}
EOF
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
# drop the --unified-core and add --workdir
mkdir -p cache/workdir
export compose_base_argv="${compose_base_argv/--unified-core/--workdir=$PWD/cache/workdir}"
compose: Add --parent option This may seem like a backflip on #1829, but there's a common theme here: in a promotion workflow, the parent (or lack of parent) of a commit is an important parameter, so we need full flexibility in configuring it. But again, like #1829, we still want e.g. change detection, versioning, and various optimizations to happen on whatever the latest commit on that ref is in the build repo. Closes: #1871 Approved by: cgwalters
2019-07-18 18:25:24 +03:00
# Test --parent at the same time (hash is `echo | sha256sum`)
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
rm -rf cache/workdir && mkdir cache/workdir
compose: Add --parent option This may seem like a backflip on #1829, but there's a common theme here: in a promotion workflow, the parent (or lack of parent) of a commit is an important parameter, so we need full flexibility in configuring it. But again, like #1829, we still want e.g. change detection, versioning, and various optimizations to happen on whatever the latest commit on that ref is in the build repo. Closes: #1871 Approved by: cgwalters
2019-07-18 18:25:24 +03:00
runcompose --add-metadata-from-json metadata.json \
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
--parent=01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
compose-tests: add more tests Add a few more tests to exercise some of the treefile options. We do need to also expand test-basic.sh itself to sanity-check the structure of a normal ostree compose. That's up next on the list. Closes: #548 Approved by: cgwalters
2016-12-09 01:31:20 +03:00
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
# shellcheck source=libbasic-test.sh
. "${dn}/libbasic-test.sh"
compose: Add --ex-unified-core The "--ex" prefix here means it's an experimental option. A tremendous change here is that start to support non-uid 0, but there are various things to fix there; the unpacker for example needs to learn to set imported objects fully based on the rpmfi information (i.e. default to uid 0, since libarchive gives the current uid by default). And even when run as uid 0, there are some bugs, though I'm not sure of any showstoppers yet. For example, dracut's `dracut-install` calls `cp --preserve=xattrs` which fails to copy the `user.ostreemeta` xattrs from a checkout (it shouldn't be copying that anyways...) Nevertheless, the infrastructure behind this really helps (is almost a hard requirement for) the [jigdo effort](https://github.com/projectatomic/rpm-ostree/issues/1081). Which is really only true due to SELinux - we need to import the packages, then generate the final tree to get the final policy, then use that policy to relabel all of the packages. Closes: #940 Approved by: jlebon
2017-11-16 05:28:03 +03:00
basic_test
lib/postprocess: Use O_TMPFILE, not O_APPEND for tmpfiles.d writing The comment here was wrong; we don't rely on `O_APPEND` here for package layering since we convert on import. I noticed this while I was doing a grep for `O_APPEND` in the codebase as part of unified core work. Fix this by converting to `O_TMPFILE`+`GLNX_LINK_TMPFILE_NOREPLACE`. Prep for unified core. Closes: #1009 Approved by: jlebon
2017-09-25 19:58:44 +03:00
compose: Add --ex-unified-core The "--ex" prefix here means it's an experimental option. A tremendous change here is that start to support non-uid 0, but there are various things to fix there; the unpacker for example needs to learn to set imported objects fully based on the rpmfi information (i.e. default to uid 0, since libarchive gives the current uid by default). And even when run as uid 0, there are some bugs, though I'm not sure of any showstoppers yet. For example, dracut's `dracut-install` calls `cp --preserve=xattrs` which fails to copy the `user.ostreemeta` xattrs from a checkout (it shouldn't be copying that anyways...) Nevertheless, the infrastructure behind this really helps (is almost a hard requirement for) the [jigdo effort](https://github.com/projectatomic/rpm-ostree/issues/1081). Which is really only true due to SELinux - we need to import the packages, then generate the final tree to get the final policy, then use that policy to relabel all of the packages. Closes: #940 Approved by: jlebon
2017-11-16 05:28:03 +03:00
# This one is done by postprocessing /var
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
ostree --repo="${repo}" cat "${treeref}" /usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf > autovar.txt
lib/postprocess: Use O_TMPFILE, not O_APPEND for tmpfiles.d writing The comment here was wrong; we don't rely on `O_APPEND` here for package layering since we convert on import. I noticed this while I was doing a grep for `O_APPEND` in the codebase as part of unified core work. Fix this by converting to `O_TMPFILE`+`GLNX_LINK_TMPFILE_NOREPLACE`. Prep for unified core. Closes: #1009 Approved by: jlebon
2017-09-25 19:58:44 +03:00
# Picked this one at random as an example of something that won't likely be
# converted to tmpfiles.d upstream. But if it is, we can change this test.
postprocess: Use names (not ids) in synthesized tmpfiles.d files Related to: https://github.com/projectatomic/rpm-ostree/issues/49 We want to support "name binding" per client system, rather than having a hardcoded mapping in our tree. Currently if e.g. a new daemon is added as a dependency (or as part of e.g. systemd) it's easy to silently miss it. This is prep for doing that binding client side consistently, which is what we do with package layering. Closes: #1077 Approved by: jlebon
2017-10-27 19:56:05 +03:00
assert_file_has_content_literal autovar.txt 'd /var/cache 0755 root root - -'
# And this one has a non-root uid
assert_file_has_content_literal autovar.txt 'd /var/log/chrony 0755 chrony chrony - -'
lib/postprocess: Use O_TMPFILE, not O_APPEND for tmpfiles.d writing The comment here was wrong; we don't rely on `O_APPEND` here for package layering since we convert on import. I noticed this while I was doing a grep for `O_APPEND` in the codebase as part of unified core work. Fix this by converting to `O_TMPFILE`+`GLNX_LINK_TMPFILE_NOREPLACE`. Prep for unified core. Closes: #1009 Approved by: jlebon
2017-09-25 19:58:44 +03:00
echo "ok autovar"
Add support for YAML treefiles Let's modernize and start supporting YAML treefiles. I'll dare make the sweeping generalization that most people would prefer reading and writing YAML over JSON. This takes bits from coreos-assembler[1] that know how to serialize a YAML file and spit it back out as a JSON and makes it into a shared lib that we can link against. We could use this eventually for JSON inputs as well to force a validation check before composing. If we go this route, we could then turn on `--enable-rust` in FAHC for now and drop the duplicate code in coreos-assembler. [1] https://github.com/cgwalters/coreos-assembler Closes: #1377 Approved by: cgwalters
2018-05-31 23:31:22 +03:00
compose: Use static enablement for ostree systemd services I was looking at the output of `ostree admin config-diff` on a base FCOS boot. It'd be really nice to trim that down as much as possible, so we can cleanly capture the difference between user config and system config. Let's use static enablement rather than presets.
2020-12-02 20:57:34 +03:00
# Validate this exists
ostree --repo="${repo}" ls "${treeref}" /usr/lib/systemd/system/multi-user.target.wants/ostree-remount.service
compose: Write a preset file to enable ostree-remount.service I've lost count now of how many times people have hit variants of https://github.com/projectatomic/centos-release-atomic-host-devel/pull/6 Let's just bake it in. Closes: #1482 Approved by: jlebon
2018-07-31 23:00:55 +03:00
tests: Bump to Python 3 only This bumps the requirement on the controlling host to Python 3 only. It also bumps the requirement on the target host to Python 3 as well since FCOS doesn't ship Python 2 right now. Though we'll need to eventually drop all Python usage anyway, but at least let's get tests passing on FCOS first. (See related previous patch). Closes: #1828 Approved by: cgwalters
2019-05-08 17:15:48 +03:00
python3 <<EOF
Add support for YAML treefiles Let's modernize and start supporting YAML treefiles. I'll dare make the sweeping generalization that most people would prefer reading and writing YAML over JSON. This takes bits from coreos-assembler[1] that know how to serialize a YAML file and spit it back out as a JSON and makes it into a shared lib that we can link against. We could use this eventually for JSON inputs as well to force a validation check before composing. If we go this route, we could then turn on `--enable-rust` in FAHC for now and drop the duplicate code in coreos-assembler. [1] https://github.com/cgwalters/coreos-assembler Closes: #1377 Approved by: cgwalters
2018-05-31 23:31:22 +03:00
import json, yaml
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
tf=yaml.safe_load(open("$treefile"))
with open("$treefile.json", "w") as f:
json.dump(tf, f)
Add support for YAML treefiles Let's modernize and start supporting YAML treefiles. I'll dare make the sweeping generalization that most people would prefer reading and writing YAML over JSON. This takes bits from coreos-assembler[1] that know how to serialize a YAML file and spit it back out as a JSON and makes it into a shared lib that we can link against. We could use this eventually for JSON inputs as well to force a validation check before composing. If we go this route, we could then turn on `--enable-rust` in FAHC for now and drop the duplicate code in coreos-assembler. [1] https://github.com/cgwalters/coreos-assembler Closes: #1377 Approved by: cgwalters
2018-05-31 23:31:22 +03:00
EOF
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
export treefile=$treefile.json
rm -rf cache/workdir && mkdir cache/workdir
tests: Drop conditionals for Rust enablement We now hard require it, so change these tests to do so as well. Closes: #1544 Approved by: jlebon
2018-09-10 15:50:33 +03:00
runcompose
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
echo "ok json"
compose: Add --no-parent option There are cases where we do want all the things that specifying a ref provides (e.g. change detection, version incrementing, SELinux labeling optimizations, and of course writing the ref) but we *don't* want the new commit to have a parent. Add a new `--no-parent` option to accommodate this. This will be used by coreos-assembler. See discussions at https://github.com/coreos/coreos-assembler/issues/159. Closes: #1829 Approved by: cgwalters
2019-05-08 07:13:43 +03:00
# also check that --no-parent doesn't invalidate change detection
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
rm -rf cache/workdir && mkdir cache/workdir
compose: Add --no-parent option There are cases where we do want all the things that specifying a ref provides (e.g. change detection, version incrementing, SELinux labeling optimizations, and of course writing the ref) but we *don't* want the new commit to have a parent. Add a new `--no-parent` option to accommodate this. This will be used by coreos-assembler. See discussions at https://github.com/coreos/coreos-assembler/issues/159. Closes: #1829 Approved by: cgwalters
2019-05-08 07:13:43 +03:00
runcompose --no-parent |& tee out.txt
assert_file_has_content_literal out.txt "No apparent changes since previous commit"
echo "ok --no-parent"
Reference in New Issue Copy Permalink