IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
New bench tool supports multiple commands, minimizing code duplication.
$ ./bench -h
usage: bench [-h] {send,recv} ...
imaged benchmark tool
optional arguments:
-h, --help show this help message and exit
commands:
{send,recv}
send send image data to stdout
recv receive image data from stdin
$ ./bench send -h
usage: bench send [-h] [-o OFFSET] [-b BUFFERSIZE] path size
positional arguments:
path path to existing image
size amount of data to copy (in MiB)
optional arguments:
-h, --help show this help message and exit
-o OFFSET, --offset OFFSET
offset in image (in bytes)
-b BUFFERSIZE, --buffer-size BUFFERSIZE
copy buffer size (in bytes)
$ ./bench recv -h
usage: bench recv [-h] [-o OFFSET] [-b BUFFERSIZE] path size
positional arguments:
path path to existing image
size amount of data to copy (in MiB)
optional arguments:
-h, --help show this help message and exit
-o OFFSET, --offset OFFSET
offset in image (in bytes)
-b BUFFERSIZE, --buffer-size BUFFERSIZE
copy buffer size (in bytes)
Change-Id: I007a4660ee966d0b9419e57f5c7d8a79e614b0bb
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Allows easy testing with various options (offset, buffersize).
usage: recvfile [-h] [-o OFFSET] [-b BUFFERSIZE] path size
Receive image data from stdin
positional arguments:
path path to existing image
size amount of data to copy (in MiB)
optional arguments:
-h, --help show this help message and exit
-o OFFSET, --offset OFFSET
offset in image (in bytes)
-b BUFFERSIZE, --buffer-size BUFFERSIZE
copy buffer size (in bytes)
Change-Id: Ief49db53449c1915a68a4bbe95e9894d55312b47
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
If caller specifies an offset, we seek to the offset and receive size
bytes after the offset.
If the offset is not aligned to block size, we receive the first
unaligned chunk before the first block without direct I/O. This may be
useful for performing random I/O.
| block | block | block | block |
+-------------+----+--------+-------------+--------+----+
| | | | |
+------ seek ------+-- io --+- direct io -+-- io --+
| | |
+----- offset -----+------------ size -------------+
Change-Id: I4211178d955b7b9a2612ea1f5ec9159e4e545ea3
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
When doing small reads, there is no point in using a big buffer, reading
or writing useless data. Buffer size is using the size of the request
(aligned to next block size), up to suggested buffer size.
Change-Id: I10a8e2be74bfa8339a37260926b33c62c2e124ba
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Allows easy testing with various options (offset, buffersize).
usage: sendfile [-h] [-o OFFSET] [-b BUFFERSIZE] path size
Send image data to stdout
positional arguments:
path path to existing image
size amount of data to copy (in MiB)
optional arguments:
-h, --help show this help message and exit
-o OFFSET, --offset OFFSET
offset in image (in bytes)
-b BUFFERSIZE, --buffer-size BUFFERSIZE
copy buffer size (in bytes)
Change-Id: I25105e637b5578fd700f8d5bfccf60c6e44e28bd
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
If caller specifies an offset, we seek to the offset and send size bytes
after the offset.
If the offset is not aligned to block size, we seek to the nearest block
and skip the reminder bytes in this block. This may be useful for
performing random I/O.
| block | block | block | block |
+-------------+-------------+-------------+-------------+
| | |
+--- seek ----+----------+------ read -------------+----+
| | | |
+-------------+-- skip --+--------- send ----------+
| | |
+--------- offset -------+--------- size ----------+
Change-Id: I2d76ba6ed1811ccd0ab3b8750adf942217a4a63b
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Using new directio operations, we can report the progress, rate and
elapsed time when running the copy-to-image and copy-from-image tests
scripts.
Change-Id: Ie53f74f5787c01a80eeeb4d19ff37b5095fc35ec
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Rename copy-from-image to sendfile and copy-to-image to recvfile,
matching new directio operation classes. Removing the common prefix
also make it easier to invoke them from the command line.
Change-Id: I84443e9677c36efec32cd47d942452645c0bb0fa
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
This constant was used as the buffer size, and is clashing with the
storage block size (512), which should have a constant.
Change-Id: I4584e2e888d675e59c8732403512a95917540f1c
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Convert directio copy functions to operation classes, to allow
cancellation and progress reporting.
copy_from_image was rename to Send, and copy_to_image was renamed to
Receive. The tests were renamed to reflect the new terms.
Change-Id: I9336a35d5b7163daeaf10ee84d9fc87d76539a44
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Use @pytest.mark.parameterize() to eliminate duplicate test code and
improve coverage.
Change-Id: I3aee864ea06a8892b1abf0feaebabc5e6bc1a6bb
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Add new errors module for keeping imaged errors. The first error is
PartialContent, raised when available size is shorter then the requested
size. Add tests verifying that the new error is raised when it should.
The new tests reveal that the reading loop was not checking the partial
content correctly when a file is not aligned to 512 bytes. The last read
returned partial content leaving the file offset at the end of the file.
The next read fails because the file offset is not aligned. Now we check
this state and fail wit the new PartialContent exception.
Change-Id: I9066ec43c71e6ce873abd16cac68c5d283694b3d
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Previously we disabled directio for reading the last block, possibly
returning incorrect data from the buffer cache instead of latest data
from shared storage.
Use io.FileIO instead of built-in file object, which can handle partial
reads at the end of the file when using direct I/O. io.FileIO.readinto()
does not handle interrupted system calls, so we must protect it with
util.uninterruptible().
Previously the reading loop was not handling partial reads. Fixing it to
handle partial reads made the first part (reading full blocks) and the
seconds part (reading last partial block) identical, so we have now
single loop.
For consistency, use also io.FileIO for writing to images. Like
io.FileIO.readinto(), io.FileIO.write() does not handle interrupted
system calls, so we must also protect it with util.uniterruptible().
Previously the writing loop did not handle partial reads from the socket
correctly. A short read from a socket file object means that the socket
was closed; otherwise the file object would block until all requested
data arrive. Now we handle this failure correctly.
The tests were simplified by replacing the sockets with StringIO.
Change-Id: Ie61ecbf831f65f8e3c63f488c902e4f839c43655
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Save about 2 seconds by running imaged server once for all server test,
removing lot of unneeded boilerplate.
Change-Id: I32040aa28961281d93df32acc35d4b8a19c95fe1
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Using pytest.fixture, we create now only one instance of uhttp server
and uhttp secure server, and reuse them for all tests. This save 1
second in the tests.
Change-Id: Ia47422e9b683c4fe659ad9be57b6112b19ff8907
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Previously we use same socket for all uhttp tests at
/tmp/vdsm-uhttp-test.sock. This can cause tests to fail if a server
started by another test is still running and prevent parallel testing.
We also did not remove the socket after the tests.
Keep the socket in per-test temporary directory automatically cleaned up
by pytest.
Change-Id: If2a7deb62f2800fa144f2322051343ef2b4a8d0e
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Adding two generic utilities:
- start_thread - starts named daemonic thread with no boilerplate
- uninterruptible - repeat a function after an interrupted syscall
Change-Id: Iaa64e3f8dfa167fd13323ea07c9af15631d217f5
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Python cannot do directio read properly due to misdesign of built-in
fileobject's readinto(). Add an extension module providing an aligned
buffer and efficient way to read data from a file descriptor or copy
data from strings or other buffers.
The buffer implements the buffer interface so it can be written to
another file object or to a file descriptor without additional copy.
Change-Id: I979a6c2dc00271a8f71b35e56b29ac3c2b0f9f4a
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Python 2.7.9 changed the behavior in an incompatible way, enabling
client certificate by default, breaking the HTTPS tests. Since we trust
our certificates, disable the verification when running on Python
version that verify certificates by default.
For more info see https://www.python.org/dev/peps/pep-0476
Change-Id: Ibda25e04d72c91d81935ad317b9b0db9617f8336
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
On the host, the socket is created in /run/vdsm, and can be accessed
only by the user vdsm, so SSL is not needed.
Change-Id: Ie53aa97ef6d771cbf4d33cf6b1edc1d2014d92dc
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The directio module implements writing data from file object to image or
from image to file object using direct io.
In Vdsm we use dd for this purpose, which requires additional copies and
context switches:
socket -> pipe -> dd -> storage
Here we save one copy and eliminate context switches:
socket -> buffer -> storage
The module does not implement yet reading or writing with offset.
Change-Id: I769a4d2b03c39de237750d5b81c566994be530a0
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
vdsm-imaged provides direct access to oVirt disks using HTTPS protocol.
Together with ovirt-image-proxy, it allows uploading a disk image
directly into an oVirt disk, or downloading an oVirt disk.
See the README for the big picture.
Change-Id: I58c6471e700dfdae6ce643dcc0852935da56a962
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
