IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Bump the required version of libnbd to 1.14 (released Aug 2022). This
allows us to remove a check for older nbdcopy in the code.
RHEL 9 (our new minimum) has libnbd 1.20
nbdinfo --size prints the size on stdout, causing it to appear in
virt-v2v output. Using --can connect instead is silent.
Note that nbdinfo --can connect was added in libnbd 1.9.2 (Jul 2021)
and our previous minimum version of libnbd was 1.9.3 so we're OK.
However since neither of these was a stable version I also updated the
minimum libnbd requirement to 1.10 (Sep 2021), and added a proper
check in ./configure
Fixes: commit fb72e059863a60503b6011b8590c25c3a010a58f
The ocaml-libvirt project (https://gitlab.com/libvirt/libvirt-ocaml)
provides bindings for libvirt. For historical reasons we bundled this
as it was throught ocaml-libvirt wasn't widespread on distros. In
fact Fedora and Debian derivatives all have this. Arch does not
(yet), but we can fix that.
It said in the README file in that directory, "before virt-v2v 1.42 is
released we hope to have unbundled this". That didn't happen, but
let's remove it now.
Previously we used nbdkit-readahead-filter with input modes that used
curl, ssh and vddk nbdkit plugins. We quickly dropped it for curl and
vddk because it caused poor performance and failures with VMware
servers (bugs https://bugzilla.redhat.com/show_bug.cgi?id=1848862https://bugzilla.redhat.com/show_bug.cgi?id=1832805).
This drops it for ssh too (eg for vmx or xen-ssh input modes).
The readahead filter in nbdkit is naively written and implementing a
replacement has proven difficult. In any case it is not needed. When
doing the conversion, readahead is already implemented in the
libguestfs appliance kernel. When doing the copy, nbdcopy is already
able to keep sufficient requests in flight.
Split virt-v2v into several cooperating helper programs. Use disk
image pipelines on both the input and output sides even when accessing
local files. Expose the NBD sockets. Use nbdcopy for the copy step.
Some features have been removed and we intend to add those back later
(see TODO file).
For the original plan to split virt-v2v, see:
https://listman.redhat.com/archives/libguestfs/2020-November/msg00022.html
Thanks: Ming Xie, Tingting Zheng, Nir Soffer, Eric Blake, Martin Kletzander
This change is made up of many separate commits done during
development. The history of those commit messages is preserved below,
but the individual commits do not make too much sense so they have
been squashed into a single large change.
v2v: Move library-ish parts of virt-v2v into lib/ subdirectory
In preparation for splitting virt-v2v, moving library-ish parts of the
code that we wish to reuse in the new helpers into the lib/
subdirectory.
This is neutral code refactoring.
lib: Define format for metadata
In a previous iteration of the virt-v2v split I proposed using an open
format for metadata such as XML, and actually implemented much of it.
However to keep this change simple, and because no one except us is
supposed to be generating or consuming this metadata, this commit
replaces the open format with a simple OCaml serialization of an
opaque version string + the struct (eg. Types.sources). The opaque
version string is there to ensure binary compatibility between the
helpers and to discourage people from trying to write or consume the
metadata.
Note: The metadata is not ABI and will change arbitrarily between
releases. If you need to write or consume the metadata it's best to
talk to us about what you're trying to do.
inputs: Create helper-v2v-input-disk
As part of splitting up virt-v2v create input helpers. This is the
first and simplest input helper which implements the “virt-v2v -i disk”
functionality, ie. being able to drive virt-v2v from a local disk file
without any metadata.
For further details on the virt-v2v split, refer to this plan:
https://listman.redhat.com/archives/libguestfs/2020-November/msg00022.html
outputs: Create helper-v2v-output-disk
This is the simplest possible output helper. It creates the output
disks (really: processes and sockets). Note this does not yet create
the final libvirt XML. This will be added in a later commit.
convert: Create helper-v2v-convert
This commit moves the conversion code into a separate helper program
(helper-v2v-convert) which performs the conversion on the input disks.
The input disks are actually COW overlays over the source disks so
that nothing is changed on the source.
This step creates metadata files: guestcaps, inspect, target_buses and
target_firmware corresponding to the internal data structures. These
will be consumed by the output finalization step.
v2v: Get rid of Modules_list
This functionality will be replaced in the new virt-v2v.
v2v: Rearrange sources into input/ and output/ directories
Rearrange sources for incomplete input and output drivers into the new
directories.
lib: Remove unused input and output objects
These objects are no longer required after creating the modular input
and output helpers.
lib/nbdkit.ml: Add LANG=C for all nbdkit instances
In old virt-v2v this was added through the Nbdkit_sources module to
all instances of nbdkit. Add it unconditionally through Nbdkit module
to get the same effect.
outputs: Create helper-v2v-output-null
This handles -o null conversions.
v2v: Add new virt-v2v command line parser and program
In the newly modular virt-v2v, this program is responsible for
handling compatibility with the old virt-v2v command line. It will
continue to be the main way that people use virt-v2v for the
foreseeable future. This program starts the helper programs and
handles multiplexing of virt-v2v command line parameters to the right
helper.
docs, tests: Adjust --no-copy documentation and tests
Since copying and creating the output are now handled in separate
programs, --no-copy will usually create the output disks (but empty).
Adjust documentation and tests accordingly.
It's probably better to remove this option.
inputs: Create helper-v2v-input-libvirt
This handles "-i libvirtxml" (input from libvirt XML file), and all
"-i libvirt" cases which are not handled by more specific code
(ie. not vcenter-https, not vddk, not xen-ssh).
outputs: Finish finalization code for helper-v2v-output-disk
Create the final libvirt XML.
lib, tests: Don't print unused field in source_disk, fix test.
lib: Remove unused fields in s_disks struct
The fields s_qemu_uri and s_format were no longer used, remove them.
inputs: Create helper-v2v-input-ova
This handles parsing OVA files (-i ova).
outputs: Create helper-v2v-output-glance
This implements -o glance conversions to OpenStack Glance.
outputs: Create helper-v2v-output-json
Implements -o json mode.
outputs: Create helper-v2v-output-qemu
Implements -o qemu mode.
tests/test-v2v-bad-networks-and-bridges.sh: Fix test
This test depended on the specifics of parameter parsing and errors.
Adjust the test so it works with modular virt-v2v.
inputs: Combine all input helpers into one program.
This reduces the duplication of code from the previous plan. There is
now a single helper, and it uses a "hidden" -im parameter (passed by
virt-v2v) to select the input mode, eg:
helper-v2v-input -im libvirtxml v2vdir xmlfile
outputs: Combine all output helpers into one program.
This reduces duplication of code. There is now a single helper, and
it uses a "hidden" -om parameter (passed by virt-v2v) to select the
output mode, eg:
helper-v2v-output -om disk setup v2vdir -os /storage
outputs: Implement -o libvirt
inputs: Implement input from vcenter over HTTPS
This implements virt-v2v -i libvirt when we detect that the libvirt
URI points to a VMware server over HTTPS (without using VDDK).
inputs: Implement input from VMware using VDDK
This implements -i libvirt -it vddk.
inputs: Implement input from VMware via VMX
This implements -i vmx.
v2v: Fix -io ? and -oo ?
inputs: Implement input from Xen over SSH
input: Refactor input helper
Now that we have moved all the input-side code from old virt-v2v,
refactor and generally clean up.
output: Refactor output helper
General refactoring and clean up to improve the quality of the code in
the output helper.
outputs: Implement -o openstack
outputs: Implement -o rhv and -o vdsm
outputs: Implement -o rhv-upload
v2v: Run helpers with --program-name=virt-v2v
This means the helpers will use "virt-v2v" instead of "helper-v2v-..."
in error messages and similar, hopefully reducing confusion.
convert, output: Improve consistency of error messages
Don't use "prog" usually since it is added by the error function.
However occasionally when there's an internal error with virt-v2v
using the wrong arguments to the helper then we can use prog to
display the actual helper having problems.
inputs, outputs: Add cmdline abstract type
Convenient way to pass the multiple command line options as a single
parameter to functions. This is simple refactoring to make the next
change possible.
inputs, outputs: Give an error for invalid option combinations
virt-v2v 1.4x was fussy about reporting errors for options which were
not applicable in certain input or output modes. Replicate that as
much as possible here. Old virt-v2v checked output modes more
thoroughly than input modes, and I have stuck with copying that
behaviour.
This also corrects an error in -o libvirt: In old virt-v2v the output
pool defaulted to "default" rather than giving an error.
inputs, outputs: Choose qemu-nbd PID file named based on socket
Previously we attempted to choose the PID file name randomly.
Although this should never conflict, I saw one case where qemu-nbd
failed to start up, printing only:
qemu-nbd: Cannot lock pid file: Resource temporarily unavailable
My reading of the code is this could be caused by the PID file already
being locked.
Anyway there is a better way to choose PID file names: simply extend
the already unique socket name with ".pid".
v2v: Don't print double error messages
If running helper-v2v-* programs, assume that if these exit on error
then they have already printed an error message. Therefore the main
virt-v2v program does not need to print another error message.
v2v: Set permissions and SELinux labels on all sockets
When running virt-v2v as non-root (the recommended way) this all
worked fine before.
However a problem arises when running virt-v2v as root. Libvirt will
run qemu as a non-root user, so we need to set permissions
appropriately (ironically making everything a bit less secure).
Also set SELinux labels if we detect SELinux is being used.
Reported-by: Tingting Zheng
output: Explicitly shut down the NBD handle
This avoids a warning from qemu-nbd:
qemu-nbd: Disconnect client, due to: Failed to send reply: Unable to write to socket: Broken pipe
For more information about the warning, see:
https://lists.nongnu.org/archive/html/qemu-block/2021-07/msg00703.html
lib/nbdkit: Always set both socket and file labels when using SELinux
We always set the file permissions to 0777 so we might as well always
set the SELinux labels when we detect that we are using SELinux. This
avoids complexity elsewhere in virt-v2v.
inputs, outputs: Label all qemu-nbd sockets when using SELinux
Abstract qemu-nbd into a data type
Add a new module QemuNBD which contains the common code for running
qemu-nbd. Replace existing code in the input and output helpers with
this module.
v2v: In verbose mode, dump nbdinfo about each NBD socket
This could help with debugging, especially understanding if nbdcopy
can use multi-conn.
input: Use the cache filter (if available) with slow plugins
This adds the cache filter to the chain of filters for slow plugins
(curl, ssh, vddk).
There is a potential further enhancement here: using conditional
cache-on-read=/path. However that requires a very new nbdkit and
further changes elsewhere in virt-v2v.
input/nbdkit: Refactor these modules
These modules were made from old virt-v2v by splitting up the old
Nbdkit_sources module, but otherwise the code was virtually
unmodified. This refactoring eliminates code duplication and dead
code left over from the split.
Although this is mostly refactoring, I also got rid of the ability to
use nbdkit-vddk-plugin < 1.17.10, which required the awkward use to
LD_LIBRARY_PATH.
convert: Do not use qemu block layer copyonread
Before this change:
[ 0.0] Opening the source
[ 145.6] Inspecting the source
[ 988.4] Checking for sufficient free disk space in the guest
[ 988.4] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[3892.1] Mapping filesystem data to avoid copying unused and blank areas
[4125.9] Closing the overlay
[4126.6] Assigning disks to buses
[4126.6] Checking if the guest needs BIOS or UEFI to boot
[4126.6] Creating output metadata
[4132.8] Copying disk 1/1
█ 100% [****************************************]
[4205.1] Creating output metadata
[4205.1] Finishing off
After this change:
[ 0.0] Opening the source
[ 8.4] Inspecting the source
[ 14.1] Checking for sufficient free disk space in the guest
[ 14.1] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 83.5] Mapping filesystem data to avoid copying unused and blank areas
[ 87.2] Closing the overlay
[ 87.9] Assigning disks to buses
[ 87.9] Checking if the guest needs BIOS or UEFI to boot
[ 87.9] Creating output metadata
[ 94.0] Copying disk 1/1
█ 100% [****************************************]
[ 165.7] Creating output metadata
[ 165.7] Finishing off
We are now faster than virt-v2v 1.45:
[ 0.0] Opening the source -i libvirt [...]
[ 1.4] Creating an overlay to protect the source from being modified
[ 4.8] Opening the overlay
[ 17.2] Inspecting the overlay
[ 23.7] Checking for sufficient free disk space in the guest
[ 23.7] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 110.0] Mapping filesystem data to avoid copying unused and blank areas
[ 124.5] Closing the overlay
[ 125.1] Assigning disks to buses
[ 125.1] Checking if the guest needs BIOS or UEFI to boot
[ 125.1] Initializing the target -o null
[ 125.2] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw)
(100.00/100%)
[ 764.6] Creating output metadata
[ 764.6] Finishing off
Thanks: Peter Krempa
v2v: Write dir/convert and dir/copy files
During the conversion and copying phases, write files literally called
"convert" and "copy" into the v2v directory. Helpers can use these to
make decisions based on the phase of virt-v2v. In particular we will
use the presence of the "convert" file to determine if we need to
enable copy-on-read.
input: Implement nbdkit-cow-filter cow-on-read (copy on read)
This has considerable performance benefits during the conversion step.
See also:
https://listman.redhat.com/archives/libguestfs/2021-July/msg00054.html
Add list of requirements to the README
todo: Put some items left over from modularization on the backlog
input: -i disk: Always detect input format
If the input format is raw, prefer nbdkit.
v2v: Minor refactoring of the code that runs nbdcopy
convert: Remove bogus "Creating output metadata" message
Left over from virt-v2v 1.45
