keremet/rpm-ostree
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
b7fa99e408
rpm-ostree/tests/compose/test-lockfile.sh

188 lines
6.4 KiB
Bash
Raw Normal View History

compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
#!/bin/bash
set -xeuo pipefail
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
dn=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=libcomposetest.sh
. "${dn}/libcomposetest.sh"
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
# Add a local rpm-md repo so we can mutate local test packages
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
treefile_append "repos" '["test-repo"]'
core: Strengthen how we enforce lockfiles One problem with how we use lockfiles right now is that we don't enforce them for dependencies. That is, if `foo` requires `bar`, but only `foo` is in the manifest, then while `foo` will be locked, `bar` will never be checked against the lockfile because it was never explicitly requested. Higher-level though, I don't like how indirect the locking here feels. See some comments about that in: https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r288772527 https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r289419017 Essentially, the manifest is an input file of patterns, and all we really know from the lockfile output is that the set of packages in there satisfies this input in some way. But: 1. there are multiple ways to satisfy the same input (hence why hints like `SOLVER_FAVOR` exist) 2. the solution is dependent on how the solver is implemented (i.e. different libsolv versions might yield different solutions) 3. the solution is dependent on flags fed to the solver (i.e. different libdnf versions might yield different solutions) So any attempt at cross-checking between the input file and the lockfile is going to be very hard. Using a stricter mode as I suggested in #1745 of only allowing pure pkgnames or NEVRAs would help, but it wouldn't address the dependency issue. (Though I'm still thinking about possibly doing this anyway.) The solution I propose here is instead to take the nuclear approach: we completely exclude from the sack all packages of the same name as packages in our lockfiles, but which do not match the NEVRA. Therefore, any possible solution has to also satisfy our lockfile (or error out). Closes: #1849 Approved by: cgwalters
2019-06-05 22:05:22 +03:00
build_rpm test-pkg-common
build_rpm test-pkg requires test-pkg-common
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
build_rpm another-test-pkg
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
# The test suite writes to pwd, but we need repos together with the manifests
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
# Also we need to disable gpgcheck
echo gpgcheck=0 >> yumrepo.repo
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
ln "$PWD/yumrepo.repo" config/yumrepo.repo
treefile_append "packages" '["test-pkg", "another-test-pkg"]'
runcompose --ex-write-lockfile-to="$PWD/versions.lock"
rpm-ostree --repo=${repo} db list ${treeref} > test-pkg-list.txt
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
assert_file_has_content test-pkg-list.txt 'test-pkg-1.0-1.x86_64'
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
assert_file_has_content test-pkg-list.txt 'another-test-pkg-1.0-1.x86_64'
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
echo "ok compose"
assert_has_file "versions.lock"
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
assert_jq "versions.lock" \
lockfile: Switch packages JSON spec to an object There are two reasons for this: 1. I'd like to add overrides semantics to lockfiles, and keying by the package name only makes this much easier. 2. I'd like to make the digest optional, and keeping it as a tuple makes this awkward. A map seems natural too since it makes it more clear that we don't expect multiple specifications for the same package name. Another tiny advantage is that it's easier to process with e.g. `jq`. Closes: #1867 Approved by: cgwalters
2019-07-10 16:50:30 +03:00
'.packages["test-pkg"].evra = "1.0-1.x86_64"' \
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
'.packages["test-pkg-common"].evra = "1.0-1.x86_64"' \
rust/lockfile: Add more metadata to generated lockfiles E.g. the generation timestamp, repos that were enabled, and their generation timestamps. This is just generally useful, though I'd like to make use specifically of the new `metadata.generated` key in FCOS to drive versioning: https://github.com/coreos/fedora-coreos-releng-automation/pull/50
2019-11-11 16:50:58 +03:00
'.packages["another-test-pkg"].evra = "1.0-1.x86_64"' \
'.metadata.rpmmd_repos|length > 0' \
'.metadata.generated'
lockfile: Switch packages JSON spec to an object There are two reasons for this: 1. I'd like to add overrides semantics to lockfiles, and keying by the package name only makes this much easier. 2. I'd like to make the digest optional, and keeping it as a tuple makes this awkward. A map seems natural too since it makes it more clear that we don't expect multiple specifications for the same package name. Another tiny advantage is that it's easier to process with e.g. `jq`. Closes: #1867 Approved by: cgwalters
2019-07-10 16:50:30 +03:00
echo "ok lockfile created"
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
# Read lockfile back (should be a no-op)
core: Strengthen how we enforce lockfiles One problem with how we use lockfiles right now is that we don't enforce them for dependencies. That is, if `foo` requires `bar`, but only `foo` is in the manifest, then while `foo` will be locked, `bar` will never be checked against the lockfile because it was never explicitly requested. Higher-level though, I don't like how indirect the locking here feels. See some comments about that in: https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r288772527 https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r289419017 Essentially, the manifest is an input file of patterns, and all we really know from the lockfile output is that the set of packages in there satisfies this input in some way. But: 1. there are multiple ways to satisfy the same input (hence why hints like `SOLVER_FAVOR` exist) 2. the solution is dependent on how the solver is implemented (i.e. different libsolv versions might yield different solutions) 3. the solution is dependent on flags fed to the solver (i.e. different libdnf versions might yield different solutions) So any attempt at cross-checking between the input file and the lockfile is going to be very hard. Using a stricter mode as I suggested in #1745 of only allowing pure pkgnames or NEVRAs would help, but it wouldn't address the dependency issue. (Though I'm still thinking about possibly doing this anyway.) The solution I propose here is instead to take the nuclear approach: we completely exclude from the sack all packages of the same name as packages in our lockfiles, but which do not match the NEVRA. Therefore, any possible solution has to also satisfy our lockfile (or error out). Closes: #1849 Approved by: cgwalters
2019-06-05 22:05:22 +03:00
build_rpm test-pkg-common version 2.0
build_rpm test-pkg version 2.0 requires test-pkg-common
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
build_rpm another-test-pkg version 2.0
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
runcompose --ex-lockfile="$PWD/versions.lock" |& tee out.txt
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
rpm-ostree --repo=${repo} db list ${treeref} > test-pkg-list.txt
assert_file_has_content out.txt 'test-pkg-1.0-1.x86_64'
assert_file_has_content out.txt 'test-pkg-common-1.0-1.x86_64'
assert_file_has_content out.txt 'another-test-pkg-1.0-1.x86_64'
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
echo "ok lockfile read"
# now add an override and check that not specifying a digest is allowed
cat > override.lock <<EOF
{
"packages": {
"another-test-pkg": {
"evra": "2.0-1.x86_64"
}
}
}
EOF
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
runcompose \
--ex-lockfile="$PWD/versions.lock" \
--ex-lockfile="$PWD/override.lock" \
compose: Add --ex-lockfile-strict Today, lockfiles only restrict the NEVRA of specifc package names from which libsolv can pick. But nothing stops libsolv from picking entirely different packages which still satisfy the manifest requests. This was mostly a theoretical issue in Fedora CoreOS, but became reality with the addition of Fedora 32 packages in the pool. libsolv would happily try to pick e.g. `libcurl-minimal` from f32 instead of sticking with the f31 `libcurl` from the lockfiles: https://github.com/coreos/fedora-coreos-streams/issues/75#issuecomment-610734584 (But more generally, see https://github.com/coreos/fedora-coreos-tracker/issues/454). Let's add a `--ex-lockfile-strict` mode, which in CI and production pipeline build contexts will require that (1) *only* locked packages are considered by libsolv, and (2) *all* locked packages were marked for install. One important thing to note here is that we don't short-circuit libsolv and manually `hy_goal_install` lockfile packages. We want to make sure the treefile is still canonical. Strict mode simply ensures that the result agrees with the lockfile. That said, even in developer contexts, we don't want the `libcurl-minimal` issue that happened to be triggered. But we still want to allow flexibility in adding and removing packages to make hacking easier. I have some follow-up patches which will enable this.
2020-04-14 23:44:07 +03:00
--ex-write-lockfile-to="$PWD/versions.lock.new" \
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
--dry-run "${treefile}" |& tee out.txt
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
assert_file_has_content out.txt 'test-pkg-1.0-1.x86_64'
assert_file_has_content out.txt 'test-pkg-common-1.0-1.x86_64'
assert_file_has_content out.txt 'another-test-pkg-2.0-1.x86_64'
compose: Add --ex-lockfile-strict Today, lockfiles only restrict the NEVRA of specifc package names from which libsolv can pick. But nothing stops libsolv from picking entirely different packages which still satisfy the manifest requests. This was mostly a theoretical issue in Fedora CoreOS, but became reality with the addition of Fedora 32 packages in the pool. libsolv would happily try to pick e.g. `libcurl-minimal` from f32 instead of sticking with the f31 `libcurl` from the lockfiles: https://github.com/coreos/fedora-coreos-streams/issues/75#issuecomment-610734584 (But more generally, see https://github.com/coreos/fedora-coreos-tracker/issues/454). Let's add a `--ex-lockfile-strict` mode, which in CI and production pipeline build contexts will require that (1) *only* locked packages are considered by libsolv, and (2) *all* locked packages were marked for install. One important thing to note here is that we don't short-circuit libsolv and manually `hy_goal_install` lockfile packages. We want to make sure the treefile is still canonical. Strict mode simply ensures that the result agrees with the lockfile. That said, even in developer contexts, we don't want the `libcurl-minimal` issue that happened to be triggered. But we still want to allow flexibility in adding and removing packages to make hacking easier. I have some follow-up patches which will enable this.
2020-04-14 23:44:07 +03:00
assert_jq versions.lock.new \
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
'.packages["test-pkg"].evra = "1.0-1.x86_64"' \
'.packages["test-pkg-common"].evra = "1.0-1.x86_64"' \
'.packages["another-test-pkg"].evra = "2.0-1.x86_64"'
echo "ok override"
compose: Add --ex-lockfile-strict Today, lockfiles only restrict the NEVRA of specifc package names from which libsolv can pick. But nothing stops libsolv from picking entirely different packages which still satisfy the manifest requests. This was mostly a theoretical issue in Fedora CoreOS, but became reality with the addition of Fedora 32 packages in the pool. libsolv would happily try to pick e.g. `libcurl-minimal` from f32 instead of sticking with the f31 `libcurl` from the lockfiles: https://github.com/coreos/fedora-coreos-streams/issues/75#issuecomment-610734584 (But more generally, see https://github.com/coreos/fedora-coreos-tracker/issues/454). Let's add a `--ex-lockfile-strict` mode, which in CI and production pipeline build contexts will require that (1) *only* locked packages are considered by libsolv, and (2) *all* locked packages were marked for install. One important thing to note here is that we don't short-circuit libsolv and manually `hy_goal_install` lockfile packages. We want to make sure the treefile is still canonical. Strict mode simply ensures that the result agrees with the lockfile. That said, even in developer contexts, we don't want the `libcurl-minimal` issue that happened to be triggered. But we still want to allow flexibility in adding and removing packages to make hacking easier. I have some follow-up patches which will enable this.
2020-04-14 23:44:07 +03:00
# sanity-check that we can remove packages in relaxed mode
treefile_remove "packages" '"another-test-pkg"'
runcompose \
--ex-lockfile="$PWD/versions.lock" \
--ex-lockfile="$PWD/override.lock" \
--ex-write-lockfile-to="$PWD/versions.lock.new" \
--dry-run "${treefile}" |& tee out.txt
assert_file_has_content out.txt 'test-pkg-1.0-1.x86_64'
assert_file_has_content out.txt 'test-pkg-common-1.0-1.x86_64'
assert_not_file_has_content out.txt 'another-test-pkg'
echo "ok relaxed mode can remove pkg"
# test strict mode
# sanity-check that refeeding the output lockfile as input satisfies strict mode
mv versions.lock{.new,}
runcompose \
--ex-lockfile-strict \
--ex-lockfile="$PWD/versions.lock" \
--ex-write-lockfile-to="$PWD/versions.lock.new" \
--dry-run "${treefile}" |& tee out.txt
assert_streq \
"$(jq .packages versions.lock | sha256sum)" \
"$(jq .packages versions.lock.new | sha256sum)"
echo "ok strict mode sanity check"
# check that trying to install a pkg that's not in the lockfiles fails
build_rpm unlocked-pkg
treefile_append "packages" '["unlocked-pkg"]'
if runcompose \
--ex-lockfile-strict \
--ex-lockfile="$PWD/versions.lock" \
--dry-run "${treefile}" &>err.txt; then
fatal "compose unexpectedly succeeded"
fi
assert_file_has_content err.txt 'Packages not found: unlocked-pkg'
echo "ok strict mode no unlocked pkgs"
# check that a locked pkg with unlocked deps causes an error
build_rpm unlocked-pkg version 2.0 requires unlocked-pkg-dep
build_rpm unlocked-pkg-dep version 2.0
# notice we add unlocked-pkg, but not unlocked-pkg-dep
cat > override.lock <<EOF
{
"packages": {
"unlocked-pkg": {
"evra": "2.0-1.x86_64"
}
}
}
EOF
if runcompose \
--ex-lockfile-strict \
--ex-lockfile="$PWD/versions.lock" \
--ex-lockfile="$PWD/override.lock" \
--dry-run "${treefile}" &>err.txt; then
fatal "compose unexpectedly succeeded"
fi
assert_file_has_content err.txt 'Could not depsolve transaction'
assert_file_has_content err.txt 'unlocked-pkg-dep-2.0-1.x86_64 is filtered out by exclude filtering'
treefile_remove "packages" '"unlocked-pkg"'
echo "ok strict mode no unlocked pkg deps"
# check that a locked pkg which isn't actually in the repos causes an error
cat > override.lock <<EOF
{
"packages": {
"unmatched-pkg": {
"evra": "1.0-1.x86_64"
}
}
}
EOF
if runcompose \
--ex-lockfile-strict \
--ex-lockfile="$PWD/versions.lock" \
--ex-lockfile="$PWD/override.lock" \
--dry-run "${treefile}" &>err.txt; then
fatal "compose unexpectedly succeeded"
fi
assert_file_has_content err.txt "Couldn't find locked package 'unmatched-pkg-1.0-1.x86_64'"
echo "ok strict mode locked pkg missing from rpmmd"
manifest: Add `lockfile-repos` field In Fedora CoreOS, we have a "coreos-pool" repo from which all packages in lockfiles are tagged for reproducible builds. This repo is shared across all streams, including those on f31 and f32. Thus, it makes no sense for composes to ever pick packages unconstrained from the pool without being guided by a lockfile. Otherwise, one can easily end up with e.g. f32 packages in an f31 compose. Add a new `lockfile-repos` for this which is only used for fetching lockfile packages and nothing else. For example, this will allow `cosa fetch --update-lockfile` to Just Work as expected by only fetching new packages from regular yum repos.
2020-04-14 23:44:08 +03:00
# test lockfile-repos, i.e. check that a pkg in a lockfile repo with higher
# NEVRA isn't picked unless if it's not in the lockfile
# some file shuffling to get a separate yumrepo-locked/ which has foobar-2.0
build_rpm foobar
mv yumrepo yumrepo.bak
build_rpm foobar version 2.0
mv yumrepo yumrepo-locked
mv yumrepo.bak yumrepo
sed -e 's/test-repo/test-lockfile-repo/g' < yumrepo.repo > yumrepo-locked.repo
sed -e 's/yumrepo/yumrepo-locked/g' < yumrepo-locked.repo > yumrepo-locked.repo.new
mv yumrepo-locked.repo.new yumrepo-locked.repo
ln "$PWD/yumrepo-locked.repo" config/yumrepo-locked.repo
treefile_append "packages" '["foobar"]'
# try first as a regular repo, to make sure it's functional
treefile_append "repos" '["test-lockfile-repo"]'
runcompose \
--ex-lockfile="$PWD/versions.lock" \
--ex-write-lockfile-to="$PWD/versions.lock.new" \
--dry-run "${treefile}" |& tee out.txt
assert_file_has_content out.txt 'foobar-2.0-1.x86_64'
# ok, now as a lockfile repo
treefile_remove "repos" '"test-lockfile-repo"'
treefile_append "lockfile-repos" '["test-lockfile-repo"]'
runcompose \
--ex-lockfile="$PWD/versions.lock" \
--ex-write-lockfile-to="$PWD/versions.lock.new" \
--dry-run "${treefile}" |& tee out.txt
assert_file_has_content out.txt 'foobar-1.0-1.x86_64'
treefile_remove "packages" '"foobar"'
echo "ok lockfile-repos"
Reference in New Issue Copy Permalink