keremet/rpm-ostree
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
462a389b3a
rpm-ostree/tests/compose/test-lockfile.sh

74 lines
2.6 KiB
Bash
Raw Normal View History

compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
#!/bin/bash
set -xeuo pipefail
dn=$(cd $(dirname $0) && pwd)
. ${dn}/libcomposetest.sh
prepare_compose_test "lockfile"
# Add a local rpm-md repo so we can mutate local test packages
pyappendjsonmember "repos" '["test-repo"]'
core: Strengthen how we enforce lockfiles One problem with how we use lockfiles right now is that we don't enforce them for dependencies. That is, if `foo` requires `bar`, but only `foo` is in the manifest, then while `foo` will be locked, `bar` will never be checked against the lockfile because it was never explicitly requested. Higher-level though, I don't like how indirect the locking here feels. See some comments about that in: https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r288772527 https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r289419017 Essentially, the manifest is an input file of patterns, and all we really know from the lockfile output is that the set of packages in there satisfies this input in some way. But: 1. there are multiple ways to satisfy the same input (hence why hints like `SOLVER_FAVOR` exist) 2. the solution is dependent on how the solver is implemented (i.e. different libsolv versions might yield different solutions) 3. the solution is dependent on flags fed to the solver (i.e. different libdnf versions might yield different solutions) So any attempt at cross-checking between the input file and the lockfile is going to be very hard. Using a stricter mode as I suggested in #1745 of only allowing pure pkgnames or NEVRAs would help, but it wouldn't address the dependency issue. (Though I'm still thinking about possibly doing this anyway.) The solution I propose here is instead to take the nuclear approach: we completely exclude from the sack all packages of the same name as packages in our lockfiles, but which do not match the NEVRA. Therefore, any possible solution has to also satisfy our lockfile (or error out). Closes: #1849 Approved by: cgwalters
2019-06-05 22:05:22 +03:00
build_rpm test-pkg-common
build_rpm test-pkg requires test-pkg-common
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
build_rpm another-test-pkg
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
# The test suite writes to pwd, but we need repos in composedata
# Also we need to disable gpgcheck
echo gpgcheck=0 >> yumrepo.repo
ln yumrepo.repo composedata/test-repo.repo
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
pyappendjsonmember "packages" '["test-pkg", "another-test-pkg"]'
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
pysetjsonmember "documentation" 'False'
mkdir cache
# Create lockfile
runcompose --ex-write-lockfile-to=$PWD/versions.lock --cachedir $(pwd)/cache
lockfile: Switch packages JSON spec to an object There are two reasons for this: 1. I'd like to add overrides semantics to lockfiles, and keying by the package name only makes this much easier. 2. I'd like to make the digest optional, and keeping it as a tuple makes this awkward. A map seems natural too since it makes it more clear that we don't expect multiple specifications for the same package name. Another tiny advantage is that it's easier to process with e.g. `jq`. Closes: #1867 Approved by: cgwalters
2019-07-10 16:50:30 +03:00
rpm-ostree --repo=${repobuild} db list ${treeref} > test-pkg-list.txt
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
assert_file_has_content test-pkg-list.txt 'test-pkg-1.0-1.x86_64'
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
assert_file_has_content test-pkg-list.txt 'another-test-pkg-1.0-1.x86_64'
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
echo "ok compose"
assert_has_file "versions.lock"
lockfile: Switch packages JSON spec to an object There are two reasons for this: 1. I'd like to add overrides semantics to lockfiles, and keying by the package name only makes this much easier. 2. I'd like to make the digest optional, and keeping it as a tuple makes this awkward. A map seems natural too since it makes it more clear that we don't expect multiple specifications for the same package name. Another tiny advantage is that it's easier to process with e.g. `jq`. Closes: #1867 Approved by: cgwalters
2019-07-10 16:50:30 +03:00
assert_jq versions.lock \
'.packages["test-pkg"].evra = "1.0-1.x86_64"' \
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
'.packages["test-pkg-common"].evra = "1.0-1.x86_64"' \
rust/lockfile: Add more metadata to generated lockfiles E.g. the generation timestamp, repos that were enabled, and their generation timestamps. This is just generally useful, though I'd like to make use specifically of the new `metadata.generated` key in FCOS to drive versioning: https://github.com/coreos/fedora-coreos-releng-automation/pull/50
2019-11-11 16:50:58 +03:00
'.packages["another-test-pkg"].evra = "1.0-1.x86_64"' \
'.metadata.rpmmd_repos|length > 0' \
'.metadata.generated'
lockfile: Switch packages JSON spec to an object There are two reasons for this: 1. I'd like to add overrides semantics to lockfiles, and keying by the package name only makes this much easier. 2. I'd like to make the digest optional, and keeping it as a tuple makes this awkward. A map seems natural too since it makes it more clear that we don't expect multiple specifications for the same package name. Another tiny advantage is that it's easier to process with e.g. `jq`. Closes: #1867 Approved by: cgwalters
2019-07-10 16:50:30 +03:00
echo "ok lockfile created"
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
# Read lockfile back
core: Strengthen how we enforce lockfiles One problem with how we use lockfiles right now is that we don't enforce them for dependencies. That is, if `foo` requires `bar`, but only `foo` is in the manifest, then while `foo` will be locked, `bar` will never be checked against the lockfile because it was never explicitly requested. Higher-level though, I don't like how indirect the locking here feels. See some comments about that in: https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r288772527 https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r289419017 Essentially, the manifest is an input file of patterns, and all we really know from the lockfile output is that the set of packages in there satisfies this input in some way. But: 1. there are multiple ways to satisfy the same input (hence why hints like `SOLVER_FAVOR` exist) 2. the solution is dependent on how the solver is implemented (i.e. different libsolv versions might yield different solutions) 3. the solution is dependent on flags fed to the solver (i.e. different libdnf versions might yield different solutions) So any attempt at cross-checking between the input file and the lockfile is going to be very hard. Using a stricter mode as I suggested in #1745 of only allowing pure pkgnames or NEVRAs would help, but it wouldn't address the dependency issue. (Though I'm still thinking about possibly doing this anyway.) The solution I propose here is instead to take the nuclear approach: we completely exclude from the sack all packages of the same name as packages in our lockfiles, but which do not match the NEVRA. Therefore, any possible solution has to also satisfy our lockfile (or error out). Closes: #1849 Approved by: cgwalters
2019-06-05 22:05:22 +03:00
build_rpm test-pkg-common version 2.0
build_rpm test-pkg version 2.0 requires test-pkg-common
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
build_rpm another-test-pkg version 2.0
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
runcompose --ex-lockfile=$PWD/versions.lock --cachedir $(pwd)/cache
echo "ok compose with lockfile"
core: Strengthen how we enforce lockfiles One problem with how we use lockfiles right now is that we don't enforce them for dependencies. That is, if `foo` requires `bar`, but only `foo` is in the manifest, then while `foo` will be locked, `bar` will never be checked against the lockfile because it was never explicitly requested. Higher-level though, I don't like how indirect the locking here feels. See some comments about that in: https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r288772527 https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r289419017 Essentially, the manifest is an input file of patterns, and all we really know from the lockfile output is that the set of packages in there satisfies this input in some way. But: 1. there are multiple ways to satisfy the same input (hence why hints like `SOLVER_FAVOR` exist) 2. the solution is dependent on how the solver is implemented (i.e. different libsolv versions might yield different solutions) 3. the solution is dependent on flags fed to the solver (i.e. different libdnf versions might yield different solutions) So any attempt at cross-checking between the input file and the lockfile is going to be very hard. Using a stricter mode as I suggested in #1745 of only allowing pure pkgnames or NEVRAs would help, but it wouldn't address the dependency issue. (Though I'm still thinking about possibly doing this anyway.) The solution I propose here is instead to take the nuclear approach: we completely exclude from the sack all packages of the same name as packages in our lockfiles, but which do not match the NEVRA. Therefore, any possible solution has to also satisfy our lockfile (or error out). Closes: #1849 Approved by: cgwalters
2019-06-05 22:05:22 +03:00
rpm-ostree --repo=${repobuild} db list ${treeref} > test-pkg-list.txt
compose: Add --ex-lockfile and --ex-write-lockfile-to Fixes #1670 This patch introduces a new `compose tree --ex-write-lockfile-to=manifest.lock` argument and a new `compose tree --ex-lockfile=manifest.lock` to read it back for subsequent invocations. Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com> Closes: #1745 Approved by: jlebon
2019-01-23 17:53:08 +03:00
assert_file_has_content test-pkg-list.txt 'test-pkg-1.0-1.x86_64'
core: Strengthen how we enforce lockfiles One problem with how we use lockfiles right now is that we don't enforce them for dependencies. That is, if `foo` requires `bar`, but only `foo` is in the manifest, then while `foo` will be locked, `bar` will never be checked against the lockfile because it was never explicitly requested. Higher-level though, I don't like how indirect the locking here feels. See some comments about that in: https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r288772527 https://github.com/projectatomic/rpm-ostree/pull/1745#discussion_r289419017 Essentially, the manifest is an input file of patterns, and all we really know from the lockfile output is that the set of packages in there satisfies this input in some way. But: 1. there are multiple ways to satisfy the same input (hence why hints like `SOLVER_FAVOR` exist) 2. the solution is dependent on how the solver is implemented (i.e. different libsolv versions might yield different solutions) 3. the solution is dependent on flags fed to the solver (i.e. different libdnf versions might yield different solutions) So any attempt at cross-checking between the input file and the lockfile is going to be very hard. Using a stricter mode as I suggested in #1745 of only allowing pure pkgnames or NEVRAs would help, but it wouldn't address the dependency issue. (Though I'm still thinking about possibly doing this anyway.) The solution I propose here is instead to take the nuclear approach: we completely exclude from the sack all packages of the same name as packages in our lockfiles, but which do not match the NEVRA. Therefore, any possible solution has to also satisfy our lockfile (or error out). Closes: #1849 Approved by: cgwalters
2019-06-05 22:05:22 +03:00
assert_file_has_content test-pkg-list.txt 'test-pkg-common-1.0-1.x86_64'
lockfile: Allow omitting the digest When manually writing lockfile overrides (see previous commit), it's sometimes easier to not have to specify the SHA256 of the package. For example, in FCOS, all packages on development and production streams will be sourced uniquely from coreos-pool, so there's no question of where the package will come from. It's of course also easier in the context of local development. Another motivation for this though is a subtle interaction between Fedora infra and the way we'd like to implement lockfile management: we want the override process to be PR-based, with a privileged bot in the backend tagging new overrides into the pool as necessary on merge. However, packages built in Koji are initially unsigned, and so we can't actually *know* what the SHA256 of the package will be until it's signed and tagged into the pool by the bot. Closes: #1867 Approved by: cgwalters
2019-07-10 16:52:54 +03:00
assert_file_has_content test-pkg-list.txt 'another-test-pkg-1.0-1.x86_64'
echo "ok lockfile read"
# now add an override and check that not specifying a digest is allowed
cat > override.lock <<EOF
{
"packages": {
"another-test-pkg": {
"evra": "2.0-1.x86_64"
}
}
}
EOF
runcompose --dry-run \
--ex-lockfile=$PWD/versions.lock \
--ex-lockfile=$PWD/override.lock \
--ex-write-lockfile-to=$PWD/versions.lock \
--cachedir $(pwd)/cache |& tee out.txt
echo "ok compose with lockfile"
assert_file_has_content out.txt 'test-pkg-1.0-1.x86_64'
assert_file_has_content out.txt 'test-pkg-common-1.0-1.x86_64'
assert_file_has_content out.txt 'another-test-pkg-2.0-1.x86_64'
assert_jq versions.lock \
'.packages["test-pkg"].evra = "1.0-1.x86_64"' \
'.packages["test-pkg-common"].evra = "1.0-1.x86_64"' \
'.packages["another-test-pkg"].evra = "2.0-1.x86_64"'
echo "ok override"
Reference in New Issue Copy Permalink