IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Add --output option to ovirt-img, so that
the output format for the progress is set
from the CLI.
Only supported format for the initial version
is "text" output.
Related: #154
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Because we use json filename, we must specify the "host_device" for
block device and "file" for regular file.
Fixes: #168
Thanks: Benny Zlotnik <bzlotnik@redhat.com>
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Add SPDX header to README files that
have no format, just plain text. The header
in these files will be visible in the project
pages.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add SPDX header to certificate example files in
the project. We could add the header to the files,
as long as it remains out of the BEGIN ---- END
block. However, since the certificates get
generated through test/genpki script, headers
would get lost if we re-generate them.
Instead of modifying the script to artificially
add the headers (which defies the purpose of the
script in the first place), add explicit license
files, that will work even if we replace the files.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
When running without systemd (e.g. in a container) we don't need to
notify system. Add new configuration "daemon:systemd_enable" to allow
disabling systemd integration. This removes the runtime dependency on
the systemd.daemon package, simplifying deployment.
Disable systemd integration in all the test and example configurations
system they never start the daemon as a systemd service.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Create new containerized test module focused
in containerized deployment. This test will check
uploading and downloading with a containerized
imageio version.
Initial version only tests that starting a server
with an initial ticket works, and we can upload
and download with that server.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Currently all tests that rely on userstorage employ
the Backend wrapper only to add a couple attributes.
With recent update, context management is natively
supported by userstorage library. On the other
hand, not all test require the Backend additional
attributes.
We can avoid using the Backend wrapper completely
for tests that do not require the attributes.
Otherwise, the Backend can just wrap the
storage from within the userstorage context (i.e.,
after the storage is setup), so we avoid the
__enter__() and __exit__() overrides.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Update userstorage to v0.5.1.
Tests will still use the Backend class
in order to add the url and the
can_detect_sector_size attribute,
but the context can be managed directly
by the userstorage.
Set pytest.xfail as handler for missing
storage.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add an option, only available through config file,
to change the timeout waiting for disk creation.
This is a fine-gran configuration that should be
fine as default for most cases, but may be
interesting to modify it for stress tests,
or slow networks, etc.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Recent HTTPRequestHandler changes avoid urls that
start with '//' to protect against open redirect
attacks (see 4abab6b603).
Server url in the http test have no path,
therefore the handler that is used in the
test receives the root path, resulting in
requests starting with '//' when the Backend
tries to get the extents, which break in the
pipelines for recent Python 3.10 versions.
This is not a realistic scenario and not
something the library needs to handle.
Is better to change the url to a more
realistic scenario, and since we want
to test image handling, have
`/images/ticket-id` in the url path.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add --version option to ovirt-img tool to show current version:
$ ./ovirt-img -h
usage: ovirt-img [-h] [--version] {download-disk,upload-disk} ...
Transfer disk images
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
commands:
{download-disk,upload-disk}
download-disk Download disk
upload-disk Upload disk
$ ./ovirt-img --version
ovirt-img 2.4.6
Fixes: #125
Signed-off-by: Albert Esteve <aesteve@redhat.com>
When we upload to a new disk using qcow2 format or new file we know that
the disk is empty and the entire disk contents is read as zeros by the
guest. In this case we can skip zero extents on the source image instead
of zeroing them on the destination image.
Zeroing on the destination image is usually fast, but when uploading big
images it can slow down the upload a lot. When uploading to qcow2 image,
zeroing creates suboptimal image that will be slower to read and copy
later.
Example upload of empty 8 TiB image:
Before:
./ovirt-img upload-disk -c engine --storage-domain fc-01 empty-8t.qcow2
[ 100% ] 8.00 TiB, 222.82 s, 36.77 GiB/s | upload completed
After:
./ovirt-img upload-disk -c engine --storage-domain fc-01 empty-8t.qcow2
[ 100% ] 8.00 TiB, 11.51 s, 711.88 GiB/s | upload completed
Example upload of 8 TiB Fedora 35 image:
Before:
$ ./ovirt-img upload-disk -c engine --storage-domain fc-01 fedora-35-8t.qcow2
[ 100% ] 8.00 TiB, 317.88 s, 25.77 GiB/s | upload completed
After:
$ ./ovirt-img upload-disk -c engine --storage-domain fc-01 fedora-35-8t.qcow2
[ 100% ] 8.00 TiB, 109.13 s, 75.07 GiB/s | upload completed
Fixes: #76
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Add an additional argument to the options.add_sub_command
function to optionally (True by default) add common
transfer-related options to the command.
Currently these options are "--max-workers" and
"--buffer-size".
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add UUID type for UUID normalization and parameter
validation. Use it to verify disk_id in download-disk.
Output:
ovirt-img download-disk: error: argument disk_id:
invalid UUID value: 'invalid-uuid-argument'
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add block_signals argument to run qemu-nbd with blocked signals. This is
useful for clean termination when qemu-nbd is managed by a client that
handle signals.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The io module is used only on the client side, and we don't have plans
to use it on the server side. On the client side we can teach io.copy()
to abort cleanly when the application is terminated, which will simplify
signal handling.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The io module is used only on the client side, and we don't have plans
to use it on the server side. On the client side we can teach io.copy()
to abort cleanly when the application is terminated, which will simplify
signal handling.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Test that parsing options without arguments is same as --help.
Fixes: 744799e00467 (ovirt-img: Fix running without arguments)
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
In ovit-img log_file defaults to /dev/stderr fd,
and is passed to logging configurator through
filename argument.
However, this is not correct, filename only is
used for files. File descriptions shall go
as stream parameters.
Otherwise the tool breaks with
'OSError: [Errno 29] Illegal seek'
in an internal logging module.
Reproduced with Python 3.6 in CentOS Stream 8.
To solve it, default log_file to None instead, so
that logging configuration does not use a filename
and falls back to the stream argument, which defaults
to stderr.
Fixes: a2bbe122cc85d08dd5ab60e6d2438b54b1f11e6b
Fixes: #113
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Reading the password in _ovirt.connect() does not work with creating a
progress bar before connecting. Change the parser to read the password
before returning the parsed arguments. Callers can consume the password
from args.password.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
When running an operation with a progress bar as context manager, the
code run in the context may fail. In this case we wan to display that
the operation failed in the progress.
Add error_phase argument defaulting to "command failed" to make this
easy.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Transferring an image include several phases that may take several
seconds, and we want to show what the tool is doing. When we finish, we
want to show that the operation was completed. If we failed, we want to
show a short failure message even if we use log errors to file.
Add an optional phase argument to ProgressBar. If the phase is set, it
is displayed at the end:
[ ---- ] 0 bytes, 0.00 s, 0 bytes/s | setting up
The caller can change the phase by setting a new value:
pb.phase = "downloading image"
This will redraw the progress bar:
[ 0% ] 0 bytes, 0.00 s, 0 bytes/s | downloading image
This change only adds the infrastructure, no change yet in the actual
commands.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
With current code we update the progress very quickly after getting the
size, so it it does not matter. But in future versions we will get the
size from OPTIONS response, and when downloading big images this can
update the progress few seconds before the first byte is transferred.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Make it easier to assert about the progress line by separating the
progress text and the assert about the text width and line terminator.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The ovirt-img tool support now optional logging options:
--log-file LOG_FILE Log file name (default: /dev/stderr).
--log-level LOG_LEVEL
Log level (choices: {debug, info, warning, error},
default: warning).
Users can change the default log file and level using the configuration
file:
[engine]
log_file = /home/user/.ovirt-img/engine.log
log_level = info
Like other options, command lines options override values from the
config file.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Add infrastructure to support mixing command line arguments and config
file options.
Parser options are defined now using list of Option named tuple. We use
the list to add arguments to the arguments parser, read and merge config
file options, set defaults, and check required arguments.
We support now:
- command line only arguments (e.g. --config, --password-file)
- config only arguments (e.g. password)
- default values if the command line argument and the config file do not
specify a value.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
When running without arguments we fail after parsing the arguments
because `config` is not set, and required arguments are missing. Fix by
detecting the case when no command was specified and failing with useful
help (like ovirt-img --help). This is a common behavior for tool that
has sub commands (e.g git).
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Previously we updated the progress up to 10 times per second, and we
displayed fractional percent values (e.g. 12.57%). This is too noisy and
creates too many uninteresting updates. It also calls
time.monotonic_time() on every update to check if it is time to update
which is waste of resources for very little benefit.
Change the progress to show integer values (12%) and update the progress
only when the progress value changes. With this change we update the
progress up to 100 times during a transfer.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Add -c,--config option specifying the config section in the ovirt-img
configuration file.
Example file:
$ cat ~/.config/ovirt-img.conf
[engine]
engine_url = https://my.engine
username = admin@internal
password = password
cafile = /home/user/certs/my.engine.pem
The configuration file can contain multiple sections. This can be useful
for people managing multiple oVirt environments.
Using a configuration file downloading is much simpler:
$ ./ovirt-img download-disk -c engine adb9b2c7-32a9-4e87-894c-710de7165086 disk.qcow2
If the same options are specified both in the command line and the
config file, command line options win.
The password option is special case; if --password-file is specified, we
read the password from the password file even if password is specified
in the config file.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The Size validator accept human-size like "1m" and convert it an
integer.
>>> size = _options.Size()
>>> size("2g")
2147483648
The validator supports minimum and maximum values to ensure
that user input in in the valid range:
>>> size = Size(minimum=4*KiB, maximum=1*MiB)
>>> size("42")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "ovirt_imageio/client/_options.py", line 89, in __call__
raise ValueError(f"Size {s!r} < {self.minimum}")
ValueError: Size '42' < 4k
>>> size("2m")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "ovirt_imageio/client/_options.py", line 92, in __call__
raise ValueError(f"Size {s!r} > {self.maximum}")
ValueError: Size '2m' > 1m
Finally the validator limits are printed as size value. This makes it
easy to show the minimum, maximum and default value in online help:
>>> size = Size(minimum=4*KiB, default=256*KiB, maximum=1*MiB)
>>> f"range: {size.minimum}-{size.maximum} default: {size.default}"
'range: 4k-1m default: 256k'
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
In server_test started_imageio fixture
creates servers with built ovirt-imageio,
and terminates the process at the end,
between runs.
However, if runs do happen too fast
it may try to create the next server socket
before the previous is closed, resulting in
an "address already in use" error.
Add a wait() after terminate() to ensure we do
not run into such issue.
Fixes: e6ed8210c19c98f3edb2ba8909bf83ca3571c57b
Signed-off-by: Albert Esteve <aesteve@redhat.com>
In admin_tool_test, srv fixture is parametrized
to create two servers with built ovirt-imageio,
with a terminate at the end, between runs.
However, if runs do happen too fast
it may try to create the next server socket
before the previous is closed, resulting in
an "address already in use" error.
Add a wait() after terminate() to ensure we do
not run into such issue.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
The base image used for backup tests was created by a pytest fixture.
This is useful since the image is created transparently when needed, but
in CI environment creating the image can be very slow and requires very
long timeouts to avoid random failures.
Move the creation of the image to new "images" target. The "check"
target depends on it so the image will be create automatically when
running "make check".
With this change the backup tests run successfully on all tested
distros. Previously they were skipped on centos 9 since they always
timed out.
Testing shows that the backup tests take 19-23 seconds on github, so 60
seconds timeout should be enough.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The backup test tends to time out randomly on github, even with 120
seconds timeout. I think this happens when virt-builder takes too much
time to create the image. Lets give it more time.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Test 10 concurrent downloads, each creating 4 connections, reproducing
the bug with the default backlog size.
Bug-Url: https://bugzilla.redhat.com/2066113
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
