keremet/rpm-ostree
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
a70da06b71
rpm-ostree/tests/compose/test-rojig-e2e.sh

118 lines
4.7 KiB
Bash
Raw Normal View History

Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
#!/bin/bash
set -xeuo pipefail
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
dn=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=libcomposetest.sh
. "${dn}/libcomposetest.sh"
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
# Add a local rpm-md repo so we can mutate local test packages
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
treefile_append "repos" '["test-repo"]'
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
build_rpm test-pkg \
files "/usr/bin/test-pkg" \
install "mkdir -p %{buildroot}/usr/bin && echo localpkg data > %{buildroot}/usr/bin/test-pkg"
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
# The test suite writes to pwd, but we need repos in config
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
# Also we need to disable gpgcheck
echo gpgcheck=0 >> yumrepo.repo
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
ln yumrepo.repo config/test-repo.repo
treefile_append "packages" '["test-pkg"]'
treefile_set "documentation" 'False'
runcompose --add-metadata-string version=42.0
npkgs=$(rpm-ostree --repo=${repo} db list ${treeref} |grep -v '^ostree commit' | wc -l)
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
echo "npkgs=${npkgs}"
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
rpm-ostree --repo=${repo} db list ${treeref} test-pkg >test-pkg-list.txt
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
assert_file_has_content test-pkg-list.txt 'test-pkg-1.0-1.x86_64'
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
rev=$(ostree --repo=${repo} rev-parse ${treeref})
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
mkdir rojig-output
do_commit2rojig() {
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
targetrev=$1
tests/compose: Add some time logging To help us profile. Closes: #1479 Approved by: jlebon
2018-07-31 16:55:31 +03:00
echo "$(date): starting commit2rojig"
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
runasroot rpm-ostree ex commit2rojig --repo=${repo} --pkgcache-repo cache/pkgcache-repo ${targetrev} ${treefile} $(pwd)/rojig-output
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
(cd rojig-output && createrepo_c .)
tests/compose: Add some time logging To help us profile. Closes: #1479 Approved by: jlebon
2018-07-31 16:55:31 +03:00
echo "$(date): finished commit2rojig"
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
}
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
do_commit2rojig ${rev}
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
test -f rojig-output/x86_64/fedora-coreos-42.0-1.fc31.x86_64.rpm
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
ostree --repo=rojig-unpack-repo init --mode=bare-user
echo 'fsync=false' >> rojig-unpack-repo/config
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
# Technically this isn't part of config/ but eh
cat > config/rojig-test.repo <<eof
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
[rojig-test]
baseurl=file://$(pwd)/rojig-output
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
enabled=1
gpgcheck=0
eof
Initial renaming pass of "jigdo" to "rojig" I noticed that Ubuntu also uses the original "jigdo", so let's start pulling off the band-aid here and do a mass rename. For this first pass I'm focusing on CLI entrypoints and docs, as that's what people are going to see; renaming all of the internal C functions, structure variables etc. can come later. Closes: #1269 Approved by: jlebon
2018-02-22 18:18:24 +03:00
do_rojig2commit() {
tests/compose: Add some time logging To help us profile. Closes: #1479 Approved by: jlebon
2018-07-31 16:55:31 +03:00
echo "$(date): starting rojig2commit"
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
rpm-ostree ex rojig2commit -d $(pwd)/config -e cache -e test-repo -e rojig-test --repo=rojig-unpack-repo rojig-test:fedora-coreos | tee rojig2commit-out.txt
tests/compose: Add some time logging To help us profile. Closes: #1479 Approved by: jlebon
2018-07-31 16:55:31 +03:00
echo "$(date): finished rojig2commit"
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
}
Initial renaming pass of "jigdo" to "rojig" I noticed that Ubuntu also uses the original "jigdo", so let's start pulling off the band-aid here and do a mass rename. For this first pass I'm focusing on CLI entrypoints and docs, as that's what people are going to see; renaming all of the internal C functions, structure variables etc. can come later. Closes: #1269 Approved by: jlebon
2018-02-22 18:18:24 +03:00
do_rojig2commit
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
# there will generally be pkgs not in the rojig set, but let's at least assert it's > 0
assert_file_has_content rojig2commit-out.txt ${npkgs}/${npkgs}' packages to import'
ostree --repo=rojig-unpack-repo rev-parse ${rev}
tests/compose: Add some time logging To help us profile. Closes: #1479 Approved by: jlebon
2018-07-31 16:55:31 +03:00
echo "$(date): starting fsck"
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
ostree --repo=rojig-unpack-repo fsck
tests/compose: Add some time logging To help us profile. Closes: #1479 Approved by: jlebon
2018-07-31 16:55:31 +03:00
echo "$(date): finished fsck"
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
ostree --repo=rojig-unpack-repo refs > rojig-refs.txt
Second renaming pass for "rojig" We're continuing an incremental renaming process; previously we changed the most user-visible strings. Now we're doing some internal variables, and notably the cached refs and the origin files - the latter set is things that end up on disk. This leaves the biggest items; renaming APIs, files, and tests. Closes: #1276 Approved by: jlebon
2018-02-27 19:07:25 +03:00
assert_file_has_content rojig-refs.txt 'rpmostree/rojig/test-pkg/1.0-1.x86__64'
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
Second renaming pass for "rojig" We're continuing an incremental renaming process; previously we changed the most user-visible strings. Now we're doing some internal variables, and notably the cached refs and the origin files - the latter set is things that end up on disk. This leaves the biggest items; renaming APIs, files, and tests. Closes: #1276 Approved by: jlebon
2018-02-27 19:07:25 +03:00
echo "ok rojig ♲📦 fresh assembly"
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
origrev=${rev}
unset rev
# Update test-pkg
build_rpm test-pkg \
version 1.1 \
files "/usr/bin/test-pkg" \
install "mkdir -p %{buildroot}/usr/bin && echo localpkg data 1.1 > %{buildroot}/usr/bin/test-pkg"
# Also add an entirely new package
build_rpm test-newpkg \
files "/usr/bin/test-newpkg" \
install "mkdir -p %{buildroot}/usr/bin && echo new localpkg data > %{buildroot}/usr/bin/test-newpkg"
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
treefile_append "packages" '["test-newpkg"]'
runcompose --add-metadata-string version=42.1
newrev=$(ostree --repo=${repo} rev-parse ${treeref})
rpm-ostree --repo=${repo} db list ${treeref} test-newpkg >test-newpkg-list.txt
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
assert_file_has_content test-newpkg-list.txt 'test-newpkg-1.0-1.x86_64'
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
# Rojig version 42.1
do_commit2rojig ${newrev}
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
path=rojig-output/x86_64/fedora-coreos-42.1-1.fc31.x86_64.rpm
commit2jigdo: Synthesize Requires: for jigdo set Having the "jigdo set" in repodata makes it so we can parallel download the jigdo RPM with the set. However for now, I kept the jigdo set in the jigdoRPM, since that way it'll be covered by the signature. Also, this changes the way we inject metadata to use a magic comment string, since trying to pass a gigantic macro to `rpmbuild` via its argv didn't work out so well (it looks like rpmbuild eats newlines). This approach is more robust. Closes: https://github.com/projectatomic/rpm-ostree/issues/1132 Closes: #1140 Approved by: jlebon
2017-12-08 22:21:19 +03:00
rpm -qp --requires ${path} > requires.txt
jigdo: V4: Use archful provides for jigdoRPM Requires When I tried to use my WIP client patches to do: `rpm-ostree rebase rojig://fahc:fedora-atomic-host`, I got a missing file object which turned out to be the client importing the i686 RPMs. This was passing in the test suite because we don't mirror i686 of course, but on the client side right now we end up using all enabled repos, and since Fedora is multiarch, the behavior is going to be...not predictable. Thinking a bit about on this problem I actually happened to recall the RPM `%{_isa}` macro which is used in Fedora in various places; for example to "arch bind" `-devel` packages to their base. See for example [this case](https://src.fedoraproject.org/rpms/ostree/blob/33c7dc02bc4f14a6acd7679cc4d84e9629294ae4/f/ostree.spec#_79) in libostree. As I noted at first, the core problem here is that the "final" RPM architecture field is not symmetric in any way with the definition of that `%{_isa}` macro. See: https://github.com/rpm-software-management/rpm/blob/d9d47e01146a5d4411691a71916b1030ac7da193/installplatform#L25 The *third* solution I ended up on here is to iterate over the `Provides` on the server side and we take the first thing that matches `Provides: %{name}(whatever)`. I briefly thought about trying to somehow drive into libsolv the logic to prefer the jigdoRPM's native architecture...IIRC yum did something like that in the past but it was never done in libsolv? Anyways the dependencies here are now more correct, so other tools will also handle it. Closes: #1213 Approved by: jlebon
2018-01-18 00:25:39 +03:00
assert_file_has_content requires.txt 'glibc(.*) = '
assert_file_has_content requires.txt 'systemd(.*) = '
tests/compose: Fix wrong assert in test-rojig-e2e We should be expecting testpkg-1.1-1 here, not 1.0-1. This was passing before because of the nondeterministic `find` output (fixed in the previous commit) which could spit out the older rojig RPM. Closes: #1491 Approved by: cgwalters
2018-08-07 17:46:49 +03:00
assert_file_has_content requires.txt 'test-pkg(.*) = 1.1-1'
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
# And pull it; we should download the newer version by default
Initial renaming pass of "jigdo" to "rojig" I noticed that Ubuntu also uses the original "jigdo", so let's start pulling off the band-aid here and do a mass rename. For this first pass I'm focusing on CLI entrypoints and docs, as that's what people are going to see; renaming all of the internal C functions, structure variables etc. can come later. Closes: #1269 Approved by: jlebon
2018-02-22 18:18:24 +03:00
do_rojig2commit
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
# Now we should only download 2 packages
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
assert_file_has_content rojig2commit-out.txt '2/[1-9][0-9]* packages to import'
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
for x in ${origrev} ${newrev}; do
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
ostree --repo=rojig-unpack-repo rev-parse ${x}
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
done
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
ostree --repo=rojig-unpack-repo fsck
ostree --repo=rojig-unpack-repo refs > rojig-refs.txt
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
# We should have both refs; GC will be handled by the sysroot upgrader
# via deployments, same way it is for pkg layering.
Second renaming pass for "rojig" We're continuing an incremental renaming process; previously we changed the most user-visible strings. Now we're doing some internal variables, and notably the cached refs and the origin files - the latter set is things that end up on disk. This leaves the biggest items; renaming APIs, files, and tests. Closes: #1276 Approved by: jlebon
2018-02-27 19:07:25 +03:00
assert_file_has_content rojig-refs.txt 'rpmostree/rojig/test-pkg/1.0-1.x86__64'
assert_file_has_content rojig-refs.txt 'rpmostree/rojig/test-pkg/1.1-1.x86__64'
Introduce experimental "rpm-ostree jigdo" Tracking issue: https://github.com/projectatomic/rpm-ostree/issues/1081 To briefly recap: Let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we need need to download. For updates, like `yum update` we only download changed RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum. This is a lot like [Jigdo](http://atterer.org/jigdo/) Here we fully demonstrate the concept working end-to-end; we use the "traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which has a checksum, version etc. Then the new `ex commit2jigdo` generates the "oirpm". This is the "server side" operation. Next simulating the client side, `jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs, fully regenerating *bit for bit* the final OSTree commit. If you want to play with this, I'd take a look at the `test-jigdo.sh`; from there you can find other useful bits like the example `fedora-atomic-host.spec` file (though the canonical copy of this will likely land in the [fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo. Closes: #1103 Approved by: jlebon
2017-11-09 22:54:33 +03:00
Second renaming pass for "rojig" We're continuing an incremental renaming process; previously we changed the most user-visible strings. Now we're doing some internal variables, and notably the cached refs and the origin files - the latter set is things that end up on disk. This leaves the biggest items; renaming APIs, files, and tests. Closes: #1276 Approved by: jlebon
2018-02-27 19:07:25 +03:00
echo "ok rojig ♲📦 update!"
jigdo V5: Use number of objects as cache invalidation trigger Changes in a server-side tree can cause the need for clients to import different objects from packages. For example, turning on documentation. Another more subtle case is where an object might "move" from package A to B by being deleted from A - then the jigdo build process will pick the B version. We need a "cache validation key"; a way for the server to tell the client that the objects it should import from the package have changed. Initially I was thinking of using the libostree "content hash" but that would be awkward as we'd have to do an import on the server side too. After more consideration I realized a simple *count* of the number of objects actually works, because (as I note in a comment) changing a file in the tree will result in it ending up in the jigdoRPM (and count as a deletion). And obviously adding or removing objects changes the count too. In fact we could have done this *without* breaking the format by just having the client start recording the number of xattr entries, but this adds greater flexibility down the line since we can in theory change how we do cache invalidation if we *really* need to (but at the cost of triggering clients to redownload packages). Note the client logic got moved around as now we need to parse all the xattrs before we decide what packages to download. My test case here is turning on docs - I noticed this actually affects *every* package which was surprising to me; I expected at least some packages wouldn't have docs. I'll double check this. It'd be good to have a "moving object" case too which I may look at. Closes: https://github.com/projectatomic/rpm-ostree/issues/1197 Closes: #1256 Approved by: jlebon
2018-02-15 23:07:39 +03:00
# Add all docs to test https://github.com/projectatomic/rpm-ostree/issues/1197
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
treefile_set "documentation" 'True'
runcompose --add-metadata-string version=42.2
newrev=$(ostree --repo=${repo} rev-parse ${treeref})
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
do_commit2rojig ${newrev}
find rojig-output -name '*.rpm' | tee rpms.txt
tests/compose: Target FCOS 31, move off of PAPR Again, a lot going on here, but essentially, we adapt the compose tests to run either privileged or fully unprivileged via supermin, just like cosa. I actually got more than halfway through this initially using `cosa build` directly for testing. But in the end, we simply need more flexibility than that. We want to be able to manipulate exactly how rpm-ostree is called, and cosa is very opinionated about this (and may also change from under us in the future). (Another big difference for example is that cosa doesn't care about non-unified mode, whereas we *need* to have coverage for this until we fully kill it.) Really, the most important bit we want from there is the unprivileged-via-supermin bits. So we copy and adapt that here. One obvious improvement then is sharing this code more easily (e.g. a `cosa runasroot` or something?) However, we still use the FCOS manifest (frozen at a specific tag). It's a realistic example, and because of the lockfiles and pool, we get good reproducibility.
2019-12-22 01:42:09 +03:00
assert_file_has_content rpms.txt 'fedora-coreos-42.2.*x86_64'
Initial renaming pass of "jigdo" to "rojig" I noticed that Ubuntu also uses the original "jigdo", so let's start pulling off the band-aid here and do a mass rename. For this first pass I'm focusing on CLI entrypoints and docs, as that's what people are going to see; renaming all of the internal C functions, structure variables etc. can come later. Closes: #1269 Approved by: jlebon
2018-02-22 18:18:24 +03:00
do_rojig2commit
jigdo V5: Use number of objects as cache invalidation trigger Changes in a server-side tree can cause the need for clients to import different objects from packages. For example, turning on documentation. Another more subtle case is where an object might "move" from package A to B by being deleted from A - then the jigdo build process will pick the B version. We need a "cache validation key"; a way for the server to tell the client that the objects it should import from the package have changed. Initially I was thinking of using the libostree "content hash" but that would be awkward as we'd have to do an import on the server side too. After more consideration I realized a simple *count* of the number of objects actually works, because (as I note in a comment) changing a file in the tree will result in it ending up in the jigdoRPM (and count as a deletion). And obviously adding or removing objects changes the count too. In fact we could have done this *without* breaking the format by just having the client start recording the number of xattr entries, but this adds greater flexibility down the line since we can in theory change how we do cache invalidation if we *really* need to (but at the cost of triggering clients to redownload packages). Note the client logic got moved around as now we need to parse all the xattrs before we decide what packages to download. My test case here is turning on docs - I noticed this actually affects *every* package which was surprising to me; I expected at least some packages wouldn't have docs. I'll double check this. It'd be good to have a "moving object" case too which I may look at. Closes: https://github.com/projectatomic/rpm-ostree/issues/1197 Closes: #1256 Approved by: jlebon
2018-02-15 23:07:39 +03:00
# Not every package has docs, but there are going to need to be changes
rojig-rename: (almost) Everything else This renames the remaining C files, tests, etc. There are only a few hits for `jigdo` left; changing them would be a format break, so let's wait to do that until we need to. Closes: #1279 Approved by: jlebon
2018-02-27 23:23:59 +03:00
assert_file_has_content rojig2commit-out.txt '[1-9][0-9]*/[1-9][0-9]* packages to import ([1-9][0-9]* changed)'
ostree --repo=rojig-unpack-repo ls -R ${newrev} >/dev/null
Second renaming pass for "rojig" We're continuing an incremental renaming process; previously we changed the most user-visible strings. Now we're doing some internal variables, and notably the cached refs and the origin files - the latter set is things that end up on disk. This leaves the biggest items; renaming APIs, files, and tests. Closes: #1276 Approved by: jlebon
2018-02-27 19:07:25 +03:00
echo "ok rojig ♲📦 updated docs"
Reference in New Issue Copy Permalink