IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Measure is only required for qcow2 sparse images
to block storage. Currently we only consider the
format, but we can also consider sparse
attribute before measuring the image to avoid
unnecessary measures.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
When we upload to a new disk using qcow2 format or new file we know that
the disk is empty and the entire disk contents is read as zeros by the
guest. In this case we can skip zero extents on the source image instead
of zeroing them on the destination image.
Zeroing on the destination image is usually fast, but when uploading big
images it can slow down the upload a lot. When uploading to qcow2 image,
zeroing creates suboptimal image that will be slower to read and copy
later.
Example upload of empty 8 TiB image:
Before:
./ovirt-img upload-disk -c engine --storage-domain fc-01 empty-8t.qcow2
[ 100% ] 8.00 TiB, 222.82 s, 36.77 GiB/s | upload completed
After:
./ovirt-img upload-disk -c engine --storage-domain fc-01 empty-8t.qcow2
[ 100% ] 8.00 TiB, 11.51 s, 711.88 GiB/s | upload completed
Example upload of 8 TiB Fedora 35 image:
Before:
$ ./ovirt-img upload-disk -c engine --storage-domain fc-01 fedora-35-8t.qcow2
[ 100% ] 8.00 TiB, 317.88 s, 25.77 GiB/s | upload completed
After:
$ ./ovirt-img upload-disk -c engine --storage-domain fc-01 fedora-35-8t.qcow2
[ 100% ] 8.00 TiB, 109.13 s, 75.07 GiB/s | upload completed
Fixes: #76
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Add '--name' option to the 'upload-disk' command
to allow users to define a custom alias name
for the new uploaded disk. If ommited, default
behaviour remains, taking the source image filename
as an alias.
usage: ovirt-img upload-disk [-h] [-c CONFIG] [--engine-url ENGINE_URL]
[--username USERNAME]
[--password-file PASSWORD_FILE] [--cafile CAFILE]
[--log-file LOG_FILE] [--log-level LOG_LEVEL]
[--max-workers MAX_WORKERS]
[--buffer-size BUFFER_SIZE] -s STORAGE_DOMAIN
[-f {raw,qcow2}] [--preallocated]
[--disk-id DISK_ID] [--name NAME]
filename
....
--name NAME Alias name for the new disk. If not specified, name
will correspond with the image filename.
Closes: #121
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add upload-disk sub-command for upload disks.
$ ./ovirt-img upload-disk -h
usage: ovirt-img upload-disk [-h] [-c CONFIG] [--engine-url ENGINE_URL]
[--username USERNAME]
[--password-file PASSWORD_FILE] [--cafile CAFILE]
[--log-file LOG_FILE] [--log-level LOG_LEVEL]
[--max-workers MAX_WORKERS]
[--buffer-size BUFFER_SIZE] -s STORAGE_DOMAIN
[-f {raw,qcow2}] [--preallocated] [--disk-id ID]
filename
positional arguments:
filename Path to image to upload. Supported formats: raw,
qcow2, iso.
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
If set, read specified section from the configuration
file.
--engine-url ENGINE_URL
ovirt-engine URL. If not set, read from the specified
config section (required).
--username USERNAME ovirt-engine username. If not set, read from the
specified config section (required).
--password-file PASSWORD_FILE
Read ovirt-engine password from file. If not set, read
password from the specified config section, or prompt
the user for the password.
--cafile CAFILE Path to ovirt-engine CA certificate. If not set, read
from the specified config section
--log-file LOG_FILE Log to file instead of stderr.
--log-level LOG_LEVEL
Log level (choices: {debug, info, warning, error},
default: warning).
--max-workers MAX_WORKERS
Maximum number of workers (range: 1-8, default: 4).
--buffer-size BUFFER_SIZE
Buffer size per worker (range: 64k-16m, default: 4m).
-s STORAGE_DOMAIN, --storage-domain STORAGE_DOMAIN
Name of the storage domain.
-f {raw,qcow2}, --format {raw,qcow2}
Upload image format (default qcow2 for data disks and
raw for iso disks).
--preallocated Create preallocated disk. Required when using raw
format on block based storage domain (iSCSI, FC). ISO
images are always uploaded to preallocated disk.
--disk-id ID A UUID for the new disk. If not specified oVirt will
create a new UUID.
$ ./ovirt-img upload-disk --config engine --storage-domain iscsi-sd disk1.qcow2
[ ---- ] 0 bytes, 0.00 s, 0 bytes/s | inspecting image
[ ---- ] 0 bytes, 0.01 s, 0 bytes/s | creating disk
[ ---- ] 0 bytes, 6.39 s, 0 bytes/s | creating transfer
[ 38% ] 2.29 GiB, 15.99 s, 146.59 MiB/s | uploading image
[ 100% ] 6.00 GiB, 22.88 s, 268.56 MiB/s | finalizing transfer
[ 100% ] 6.00 GiB, 33.02 s, 186.07 MiB/s | upload completed
Possible combinations:
Content | Storage | Backup | Format | Allocation | Allowed
------- | ------- | ------ | ------ | ------------ | -------
data | block | yes | qcow | sparse | ✓
data | block | yes | qcow | preallocated | ✓
data | block | no | raw | sparse | x
data | block | no | raw | preallocated | ✓
data | file | yes | qcow | sparse | ✓
data | file | yes | qcow | preallocated | ✓
data | file | no | raw | sparse | ✓
data | file | no | raw | sparse | ✓
iso | block | no | raw | preallocated | ✓
iso | file | no | raw | preallocated | ✓
Note: Source allocation policies omitted as they make no difference.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add an additional argument to the options.add_sub_command
function to optionally (True by default) add common
transfer-related options to the command.
Currently these options are "--max-workers" and
"--buffer-size".
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add UUID type for UUID normalization and parameter
validation. Use it to verify disk_id in download-disk.
Output:
ovirt-img download-disk: error: argument disk_id:
invalid UUID value: 'invalid-uuid-argument'
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add a new "add_disk" function to the _ovirt
module for the ovirt-img tool. Add disk allows
to create a new disk in the engine through an
open connection.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Register signals handlers before running the command, and handle
TerminatedBySignal exception gracefully.
When terminated by signal, log an error instead of a traceback, since
this is expected condition. Since termination by SIGINT is typically
initiated by a user log info level message.
For any other error, log a traceback to make it easy to report useful
bug reports.
An example run when ovirt-img is interrupted:
$ ./ovirt-img download-disk -c engine adb9b2c7 dl.qcow2
[ 49% ] 3.00 GiB, 4.68 s, 656.51 MiB/s | command failed
An example run when ovirt-img is interrupted with more verbose logging:
$ ./ovirt-img download-disk -c engine adb9b2c7 dl.qcow2 --log-level info
...
[ 70% ] 4.25 GiB, 3.62 s, 1.17 GiB/s | command failed
2022-07-28 18:08:16,562 INFO (MainThread) [tool] Exiting: Terminated by signal 2
An example run when ovirt-img is terminated::
$ ./ovirt-img download-disk -c engine 07b53997 dl.qcow2
[ 26% ] 13.25 GiB, 26.23 s, 517.27 MiB/s | command failed
2022-07-28 18:15:53,518 ERROR (MainThread) [tool] Exiting: Terminated by signal 15
An example run when qemu-nbd is terminated:
$ ./ovirt-img download-disk -c engine 07b53997 dl.qcow2
[ 17% ] 8.75 GiB, 16.74 s, 535.39 MiB/s | command failed
2022-07-28 18:20:23,721 ERROR (MainThread) [tool] Command failed
Traceback (most recent call last):
File "ovirt-imageio/ovirt_imageio/client/_tool.py", line 38, in main
args.command(args)
File "ovirt-imageio/ovirt_imageio/client/_download.py", line 73, in download_disk
_api.download(
File "ovirt-imageio/ovirt_imageio/client/_api.py", line 179, in download
_io.copy(
File "ovirt-imageio/ovirt_imageio/client/_io.py", line 46, in copy
with Executor(name=name) as executor:
File "ovirt-imageio/ovirt_imageio/client/_io.py", line 198, in __exit__
self.stop()
File "ovirt-imageio/ovirt_imageio/client/_io.py", line 175, in stop
raise self._errors[0]
File "ovirt-imageio/ovirt_imageio/client/_io.py", line 248, in _run
handler.copy(req)
File "ovirt-imageio/ovirt_imageio/client/_io.py", line 292, in copy
self._src.write_to(self._dst, req.length, self._buf)
File "ovirt-imageio/ovirt_imageio/_internal/backends/http.py", line 220, in write_to
writer.write(view[:n])
File "ovirt-imageio/ovirt_imageio/_internal/backends/nbd.py", line 119, in write
self._client.write(self._position, buf)
File "ovirt-imageio/ovirt_imageio/_internal/nbd.py", line 445, in write
self._recv_reply(cmd)
File "ovirt-imageio/ovirt_imageio/_internal/nbd.py", line 959, in _recv_reply
magic = self._recv_fmt("!I")[0]
File "ovirt-imageio/ovirt_imageio/_internal/nbd.py", line 1221, in _recv_fmt
data = self._recv(s.size)
File "ovirt-imageio/ovirt_imageio/_internal/nbd.py", line 1231, in _recv
self._recv_into(buf)
File "ovirt-imageio/ovirt_imageio/_internal/nbd.py", line 1241, in _recv_into
raise ProtocolError(
ovirt_imageio._internal.nbd.ProtocolError: Server closed the connection, read 0 bytes, expected 4 bytes
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
If the application is handling signals, run qemu-nbd with SIGINT
blocked. This will allow clean termination when interrupting the
ovirt-img tool.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Add block_signals argument to run qemu-nbd with blocked signals. This is
useful for clean termination when qemu-nbd is managed by a client that
handle signals.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Add an _app module for managing application global state. The module
registers signal handlers for SIGINT and SIGTERM. When signal is
received, it saves the first signal.
The worker copy loop checks now if the application was terminated, and
raises _app.TerminatedBySignal. This error is propagated to main() so
the tool can handle termination by signals gracefully.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The io module is used only on the client side, and we don't have plans
to use it on the server side. On the client side we can teach io.copy()
to abort cleanly when the application is terminated, which will simplify
signal handling.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
We raise the first error in the thread calling io.copy(), so the caller
can log a detailed exception if needed. However we also logged a
traceback for every failed worker, which log the same error twice, and
makes termination by signal a big mess.
Replace the exception log with debug log for cleaner termination.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
When receiving a termination signal (SIGINT, SIGTERM), send the signal
to the entire process group. This will terminate qemu-nbd and trigger
normal cleanup flow.
When handling command errors, suppress errors if the tool was terminated
by signal for cleaner termination.
Example interrupted run:
$ ./ovirt-img download-disk -c engine adb9b2c7-32a9-4e87-894c-710de7165086 dl.qcow2
[ 87% ] 5.27 GiB, 4.84 s, 1.09 GiB/s | command failed
2022-07-28 12:49:15,731 ERROR (MainThread) [tool] Terminated by signal 2
Example run when qemu-nbd is killed during transfer:
$ ./ovirt-img download-disk -c engine 1860d15a-e407-4b52-9b72-bf7555bef48f dl.qcow2
[ 85% ] 8.50 GiB, 16.33 s, 533.12 MiB/s | command failed
2022-07-28 12:53:06,143 ERROR (MainThread) [tool] Command failed
Traceback (most recent call last):
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/client/_tool.py", line 41, in main
args.command(args)
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/client/_download.py", line 73, in download_disk
_api.download(
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/client/_api.py", line 176, in download
io.copy(
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/_internal/io.py", line 44, in copy
with Executor(name=name) as executor:
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/_internal/io.py", line 196, in __exit__
self.stop()
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/_internal/io.py", line 173, in stop
raise self._errors[0]
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/_internal/io.py", line 245, in _run
handler.copy(req)
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/_internal/io.py", line 289, in copy
self._src.write_to(self._dst, req.length, self._buf)
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/_internal/backends/http.py", line 220, in write_to
writer.write(view[:n])
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/_internal/backends/nbd.py", line 119, in write
self._client.write(self._position, buf)
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/_internal/nbd.py", line 444, in write
self._send(data)
File "/home/nsoffer/src/ovirt-imageio/ovirt_imageio/_internal/nbd.py", line 1227, in _send
self._sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The io module is used only on the client side, and we don't have plans
to use it on the server side. On the client side we can teach io.copy()
to abort cleanly when the application is terminated, which will simplify
signal handling.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
We raise the first error in the thread calling io.copy(), so the caller
can log a detailed exception if needed. However we also logged a
traceback for every failed worker, which log the same error twice, and
makes termination by signal a big mess.
Replace the exception log with debug log for cleaner termination.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Test that parsing options without arguments is same as --help.
Fixes: 744799e00467 (ovirt-img: Fix running without arguments)
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
In ovit-img log_file defaults to /dev/stderr fd,
and is passed to logging configurator through
filename argument.
However, this is not correct, filename only is
used for files. File descriptions shall go
as stream parameters.
Otherwise the tool breaks with
'OSError: [Errno 29] Illegal seek'
in an internal logging module.
Reproduced with Python 3.6 in CentOS Stream 8.
To solve it, default log_file to None instead, so
that logging configuration does not use a filename
and falls back to the stream argument, which defaults
to stderr.
Fixes: a2bbe122cc85d08dd5ab60e6d2438b54b1f11e6b
Fixes: #113
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Reading the password in _ovirt.connect() does not work with creating a
progress bar before connecting. Change the parser to read the password
before returning the parsed arguments. Callers can consume the password
from args.password.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
When running an operation with a progress bar as context manager, the
code run in the context may fail. In this case we wan to display that
the operation failed in the progress.
Add error_phase argument defaulting to "command failed" to make this
easy.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Transferring an image include several phases that may take several
seconds, and we want to show what the tool is doing. When we finish, we
want to show that the operation was completed. If we failed, we want to
show a short failure message even if we use log errors to file.
Add an optional phase argument to ProgressBar. If the phase is set, it
is displayed at the end:
[ ---- ] 0 bytes, 0.00 s, 0 bytes/s | setting up
The caller can change the phase by setting a new value:
pb.phase = "downloading image"
This will redraw the progress bar:
[ 0% ] 0 bytes, 0.00 s, 0 bytes/s | downloading image
This change only adds the infrastructure, no change yet in the actual
commands.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
With current code we update the progress very quickly after getting the
size, so it it does not matter. But in future versions we will get the
size from OPTIONS response, and when downloading big images this can
update the progress few seconds before the first byte is transferred.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Make it easier to assert about the progress line by separating the
progress text and the assert about the text width and line terminator.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The ovirt-img tool support now optional logging options:
--log-file LOG_FILE Log file name (default: /dev/stderr).
--log-level LOG_LEVEL
Log level (choices: {debug, info, warning, error},
default: warning).
Users can change the default log file and level using the configuration
file:
[engine]
log_file = /home/user/.ovirt-img/engine.log
log_level = info
Like other options, command lines options override values from the
config file.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Add infrastructure to support mixing command line arguments and config
file options.
Parser options are defined now using list of Option named tuple. We use
the list to add arguments to the arguments parser, read and merge config
file options, set defaults, and check required arguments.
We support now:
- command line only arguments (e.g. --config, --password-file)
- config only arguments (e.g. password)
- default values if the command line argument and the config file do not
specify a value.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
When running without arguments we fail after parsing the arguments
because `config` is not set, and required arguments are missing. Fix by
detecting the case when no command was specified and failing with useful
help (like ovirt-img --help). This is a common behavior for tool that
has sub commands (e.g git).
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The ovirt-imageio-client package installs now the ovirt-img tool, and
requires python3-ovirt-engine-sdk4.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Change ProgressBar private attributes to private names. The only public
attribute is the size which is set only after connecting to the source
and getting the source size.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Previously we updated the progress up to 10 times per second, and we
displayed fractional percent values (e.g. 12.57%). This is too noisy and
creates too many uninteresting updates. It also calls
time.monotonic_time() on every update to check if it is time to update
which is waste of resources for very little benefit.
Change the progress to show integer values (12%) and update the progress
only when the progress value changes. With this change we update the
progress up to 100 times during a transfer.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Add -c,--config option specifying the config section in the ovirt-img
configuration file.
Example file:
$ cat ~/.config/ovirt-img.conf
[engine]
engine_url = https://my.engine
username = admin@internal
password = password
cafile = /home/user/certs/my.engine.pem
The configuration file can contain multiple sections. This can be useful
for people managing multiple oVirt environments.
Using a configuration file downloading is much simpler:
$ ./ovirt-img download-disk -c engine adb9b2c7-32a9-4e87-894c-710de7165086 disk.qcow2
If the same options are specified both in the command line and the
config file, command line options win.
The password option is special case; if --password-file is specified, we
read the password from the password file even if password is specified
in the config file.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
These options are useful mostly for testing performance and choosing
good defaults. They may be useful for users if the defaults are not
optimal for their environment.
Example usage:
./ovirt-img download-disk --buffer-size 256k --max-workers 2 ...
Using the size validator we can show online help using human sizes:
$ ./ovirt-img download-disk -h
...
--max-workers MAX_WORKERS
Maximum number of workers (range: 1-8, default: 4).
--buffer-size BUFFER_SIZE
Buffer size per worker (range: 64k-16m, default: 4m).
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Export also MAX_WORKERS to help users choose a good default value,
similar to BUFFER_SIZE. Users using the client defaults will be updated
automatically when the client changes the default value.
Both names are also exported now from the _api module for internal
usage.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The Size validator accept human-size like "1m" and convert it an
integer.
>>> size = _options.Size()
>>> size("2g")
2147483648
The validator supports minimum and maximum values to ensure
that user input in in the valid range:
>>> size = Size(minimum=4*KiB, maximum=1*MiB)
>>> size("42")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "ovirt_imageio/client/_options.py", line 89, in __call__
raise ValueError(f"Size {s!r} < {self.minimum}")
ValueError: Size '42' < 4k
>>> size("2m")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "ovirt_imageio/client/_options.py", line 92, in __call__
raise ValueError(f"Size {s!r} > {self.maximum}")
ValueError: Size '2m' > 1m
Finally the validator limits are printed as size value. This makes it
easy to show the minimum, maximum and default value in online help:
>>> size = Size(minimum=4*KiB, default=256*KiB, maximum=1*MiB)
>>> f"range: {size.minimum}-{size.maximum} default: {size.default}"
'range: 4k-1m default: 256k'
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
ovirt-img is a tool for transferring disk images, replacing the oVirt
python SDK examples.
This is a minimal version implementing only download disk. More commands
will be added later.
The tool can be used without a configuration file; this will be handy
when running the tool in a container.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The tool package depends on ovirt engine python sdk. This import is
problematic since the python sdk is not packaged for Fedora, but we can
consume it via pip.
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
