IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
This tool can be used to estimate the disk space needed before doing a
virt-v2v conversion. It is a replacement for the old --print-estimate
option which was dropped in virt-v2v 2.0 (commit 5828c9c7d5 "v2v:
Remove --print-estimate option").
In Kubernetes and tools like Kubevirt, it's not possible to create
some disks and attach to them (in order to populate them with data) in
one step. This makes virt-v2v conversions awkward because ideally we
would like the output mode (-o kubevirt) to both create the target
disks and populate them at the same time.
To work around this problem, we need a tool which can inspect the
virt-v2v source hypervisor before we do the conversion in order to
find out how many disks are needed and their sizes. Then we can
create the target disks, and then we can run a second container with
virt-v2v attached to the disks to do the conversion and populate the
output.
This new tool essentially uses the same -i* options as virt-v2v (and
no -o* options) and outputs various useful metadata. Example:
$ virt-v2v-inspector --quiet -i disk fedora-32.img
virt-v2v-inspector: The QEMU Guest Agent will be installed for this guest
at first boot.
virt-v2v-inspector: warning: /files/boot/grub2/device.map/hd0 references
unknown device "vda". You may have to fix this entry manually after
conversion.
<?xml version='1.0' encoding='utf-8'?>
<v2v-inspection>
<!-- generated by virt-v2v-inspector 2.1.9local,libvirt -->
<program>virt-v2v-inspector</program>
<package>virt-v2v</package>
<version>2.1.9</version>
<disks>
<disk index='0'>
<virtual-size>6442450944</virtual-size>
<allocated estimated='true'>1400897536</allocated>
</disk>
</disks>
<operatingsystem>
<name>linux</name>
<distro>fedora</distro>
<osinfo>fedora32</osinfo>
<arch>x86_64</arch>
<major_version>32</major_version>
<minor_version>0</minor_version>
<package_format>rpm</package_format>
<package_management>dnf</package_management>
<product_name>Fedora 32 (Thirty Two)</product_name>
</operatingsystem>
</v2v-inspection>
There should be sufficient information in the <disks> section to
allocate target disks, plus additional information is printed which
might be useful.
Note that we do a full conversion in order to generate this
information. In particular it's not possible to generate the
<allocated/> estimate without this. It's plausible we could have a
--no-convert option, but I'm not sure it's worthwhile: it would only
save a little time, but would make everything less accurate, plus
maybe it is a good idea to find out if conversion is going to work
before we create the target disks?
I chose XML instead of JSON for output. XML allows us to annotate
elements with attributes like "estimated='true'". It also lets us
represent 64 bit number accurately, where JSON cannot represent such
numbers.
The output can be written to stdout (the default, or you can use "-O -"),
but for use from another program it is usually better to write the
output to a file using "-O output.xml".
Acked-by: Laszlo Ersek <lersek at redhat.com>
Inherited from libguestfs, we wrote valgrind log files into the tmp/
directory during tests. However this makes it very difficult to
connect a particular log file to a particular failing test. As with
nbdkit let's just write the valgrind output to stderr, where it will
end up in the test log file (ie. tests/test-suite.log or tests/*.log).
I also adjusted the valgrind parameters so they more closely match
those used by nbdkit.
glibc 2.34 removed MALLOC_CHECK_ and MALLOC_PERTURB_ and replaced them
with GLIBC_TUNABLES with a completely different syntax and requiring a
special library (libc_malloc_debug.so.0) to be preloaded.
Note this works even if the preloaded library is not present since
ld.so ignores missing LD_PRELOAD. So this shouldn't break anything
either for older glibc or if the special library is missing.
Add a new front end called virt-v2v-in-place which implements simple
in-place conversion support for local disks.
Commit 255722cbf3 ("v2v: Modular virt-v2v") temporarily dropped this
feature. This commit adds it back.
In a future commit I will break up the large input/input.ml file,
which is now much simpler to do. However this commit sticks to the
modularization only.
Note the first class module syntax is a somewhat obscure and new OCaml
feature. For more on this topic:
https://dev.realworldocaml.org/first-class-modules.html
Splitting virt-v2v into separate helper programs was an interesting
exercise, but the helpers themselves were not independently useful
(eg. you could not run them on their own). On reflection the value of
the exercise was really twofold:
- providing the v2vdir and input and output pipelines based on NBD
- forcing stages to be independent of each other (because they ran as
separate processes), thus enforcing the split
The drawback of separate processes is:
- need to serialise and then reparse all options as command line
parameters
- various smaller problems with getting accurate timing in messages,
handling errors well, etc.
This commit turns helper-v2v-convert back into an internal module. It
still has the strong separation, but will now be used directly as an
OCaml module.
Split virt-v2v into several cooperating helper programs. Use disk
image pipelines on both the input and output sides even when accessing
local files. Expose the NBD sockets. Use nbdcopy for the copy step.
Some features have been removed and we intend to add those back later
(see TODO file).
For the original plan to split virt-v2v, see:
https://listman.redhat.com/archives/libguestfs/2020-November/msg00022.html
Thanks: Ming Xie, Tingting Zheng, Nir Soffer, Eric Blake, Martin Kletzander
This change is made up of many separate commits done during
development. The history of those commit messages is preserved below,
but the individual commits do not make too much sense so they have
been squashed into a single large change.
v2v: Move library-ish parts of virt-v2v into lib/ subdirectory
In preparation for splitting virt-v2v, moving library-ish parts of the
code that we wish to reuse in the new helpers into the lib/
subdirectory.
This is neutral code refactoring.
lib: Define format for metadata
In a previous iteration of the virt-v2v split I proposed using an open
format for metadata such as XML, and actually implemented much of it.
However to keep this change simple, and because no one except us is
supposed to be generating or consuming this metadata, this commit
replaces the open format with a simple OCaml serialization of an
opaque version string + the struct (eg. Types.sources). The opaque
version string is there to ensure binary compatibility between the
helpers and to discourage people from trying to write or consume the
metadata.
Note: The metadata is not ABI and will change arbitrarily between
releases. If you need to write or consume the metadata it's best to
talk to us about what you're trying to do.
inputs: Create helper-v2v-input-disk
As part of splitting up virt-v2v create input helpers. This is the
first and simplest input helper which implements the “virt-v2v -i disk”
functionality, ie. being able to drive virt-v2v from a local disk file
without any metadata.
For further details on the virt-v2v split, refer to this plan:
https://listman.redhat.com/archives/libguestfs/2020-November/msg00022.html
outputs: Create helper-v2v-output-disk
This is the simplest possible output helper. It creates the output
disks (really: processes and sockets). Note this does not yet create
the final libvirt XML. This will be added in a later commit.
convert: Create helper-v2v-convert
This commit moves the conversion code into a separate helper program
(helper-v2v-convert) which performs the conversion on the input disks.
The input disks are actually COW overlays over the source disks so
that nothing is changed on the source.
This step creates metadata files: guestcaps, inspect, target_buses and
target_firmware corresponding to the internal data structures. These
will be consumed by the output finalization step.
v2v: Get rid of Modules_list
This functionality will be replaced in the new virt-v2v.
v2v: Rearrange sources into input/ and output/ directories
Rearrange sources for incomplete input and output drivers into the new
directories.
lib: Remove unused input and output objects
These objects are no longer required after creating the modular input
and output helpers.
lib/nbdkit.ml: Add LANG=C for all nbdkit instances
In old virt-v2v this was added through the Nbdkit_sources module to
all instances of nbdkit. Add it unconditionally through Nbdkit module
to get the same effect.
outputs: Create helper-v2v-output-null
This handles -o null conversions.
v2v: Add new virt-v2v command line parser and program
In the newly modular virt-v2v, this program is responsible for
handling compatibility with the old virt-v2v command line. It will
continue to be the main way that people use virt-v2v for the
foreseeable future. This program starts the helper programs and
handles multiplexing of virt-v2v command line parameters to the right
helper.
docs, tests: Adjust --no-copy documentation and tests
Since copying and creating the output are now handled in separate
programs, --no-copy will usually create the output disks (but empty).
Adjust documentation and tests accordingly.
It's probably better to remove this option.
inputs: Create helper-v2v-input-libvirt
This handles "-i libvirtxml" (input from libvirt XML file), and all
"-i libvirt" cases which are not handled by more specific code
(ie. not vcenter-https, not vddk, not xen-ssh).
outputs: Finish finalization code for helper-v2v-output-disk
Create the final libvirt XML.
lib, tests: Don't print unused field in source_disk, fix test.
lib: Remove unused fields in s_disks struct
The fields s_qemu_uri and s_format were no longer used, remove them.
inputs: Create helper-v2v-input-ova
This handles parsing OVA files (-i ova).
outputs: Create helper-v2v-output-glance
This implements -o glance conversions to OpenStack Glance.
outputs: Create helper-v2v-output-json
Implements -o json mode.
outputs: Create helper-v2v-output-qemu
Implements -o qemu mode.
tests/test-v2v-bad-networks-and-bridges.sh: Fix test
This test depended on the specifics of parameter parsing and errors.
Adjust the test so it works with modular virt-v2v.
inputs: Combine all input helpers into one program.
This reduces the duplication of code from the previous plan. There is
now a single helper, and it uses a "hidden" -im parameter (passed by
virt-v2v) to select the input mode, eg:
helper-v2v-input -im libvirtxml v2vdir xmlfile
outputs: Combine all output helpers into one program.
This reduces duplication of code. There is now a single helper, and
it uses a "hidden" -om parameter (passed by virt-v2v) to select the
output mode, eg:
helper-v2v-output -om disk setup v2vdir -os /storage
outputs: Implement -o libvirt
inputs: Implement input from vcenter over HTTPS
This implements virt-v2v -i libvirt when we detect that the libvirt
URI points to a VMware server over HTTPS (without using VDDK).
inputs: Implement input from VMware using VDDK
This implements -i libvirt -it vddk.
inputs: Implement input from VMware via VMX
This implements -i vmx.
v2v: Fix -io ? and -oo ?
inputs: Implement input from Xen over SSH
input: Refactor input helper
Now that we have moved all the input-side code from old virt-v2v,
refactor and generally clean up.
output: Refactor output helper
General refactoring and clean up to improve the quality of the code in
the output helper.
outputs: Implement -o openstack
outputs: Implement -o rhv and -o vdsm
outputs: Implement -o rhv-upload
v2v: Run helpers with --program-name=virt-v2v
This means the helpers will use "virt-v2v" instead of "helper-v2v-..."
in error messages and similar, hopefully reducing confusion.
convert, output: Improve consistency of error messages
Don't use "prog" usually since it is added by the error function.
However occasionally when there's an internal error with virt-v2v
using the wrong arguments to the helper then we can use prog to
display the actual helper having problems.
inputs, outputs: Add cmdline abstract type
Convenient way to pass the multiple command line options as a single
parameter to functions. This is simple refactoring to make the next
change possible.
inputs, outputs: Give an error for invalid option combinations
virt-v2v 1.4x was fussy about reporting errors for options which were
not applicable in certain input or output modes. Replicate that as
much as possible here. Old virt-v2v checked output modes more
thoroughly than input modes, and I have stuck with copying that
behaviour.
This also corrects an error in -o libvirt: In old virt-v2v the output
pool defaulted to "default" rather than giving an error.
inputs, outputs: Choose qemu-nbd PID file named based on socket
Previously we attempted to choose the PID file name randomly.
Although this should never conflict, I saw one case where qemu-nbd
failed to start up, printing only:
qemu-nbd: Cannot lock pid file: Resource temporarily unavailable
My reading of the code is this could be caused by the PID file already
being locked.
Anyway there is a better way to choose PID file names: simply extend
the already unique socket name with ".pid".
v2v: Don't print double error messages
If running helper-v2v-* programs, assume that if these exit on error
then they have already printed an error message. Therefore the main
virt-v2v program does not need to print another error message.
v2v: Set permissions and SELinux labels on all sockets
When running virt-v2v as non-root (the recommended way) this all
worked fine before.
However a problem arises when running virt-v2v as root. Libvirt will
run qemu as a non-root user, so we need to set permissions
appropriately (ironically making everything a bit less secure).
Also set SELinux labels if we detect SELinux is being used.
Reported-by: Tingting Zheng
output: Explicitly shut down the NBD handle
This avoids a warning from qemu-nbd:
qemu-nbd: Disconnect client, due to: Failed to send reply: Unable to write to socket: Broken pipe
For more information about the warning, see:
https://lists.nongnu.org/archive/html/qemu-block/2021-07/msg00703.html
lib/nbdkit: Always set both socket and file labels when using SELinux
We always set the file permissions to 0777 so we might as well always
set the SELinux labels when we detect that we are using SELinux. This
avoids complexity elsewhere in virt-v2v.
inputs, outputs: Label all qemu-nbd sockets when using SELinux
Abstract qemu-nbd into a data type
Add a new module QemuNBD which contains the common code for running
qemu-nbd. Replace existing code in the input and output helpers with
this module.
v2v: In verbose mode, dump nbdinfo about each NBD socket
This could help with debugging, especially understanding if nbdcopy
can use multi-conn.
input: Use the cache filter (if available) with slow plugins
This adds the cache filter to the chain of filters for slow plugins
(curl, ssh, vddk).
There is a potential further enhancement here: using conditional
cache-on-read=/path. However that requires a very new nbdkit and
further changes elsewhere in virt-v2v.
input/nbdkit: Refactor these modules
These modules were made from old virt-v2v by splitting up the old
Nbdkit_sources module, but otherwise the code was virtually
unmodified. This refactoring eliminates code duplication and dead
code left over from the split.
Although this is mostly refactoring, I also got rid of the ability to
use nbdkit-vddk-plugin < 1.17.10, which required the awkward use to
LD_LIBRARY_PATH.
convert: Do not use qemu block layer copyonread
Before this change:
[ 0.0] Opening the source
[ 145.6] Inspecting the source
[ 988.4] Checking for sufficient free disk space in the guest
[ 988.4] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[3892.1] Mapping filesystem data to avoid copying unused and blank areas
[4125.9] Closing the overlay
[4126.6] Assigning disks to buses
[4126.6] Checking if the guest needs BIOS or UEFI to boot
[4126.6] Creating output metadata
[4132.8] Copying disk 1/1
█ 100% [****************************************]
[4205.1] Creating output metadata
[4205.1] Finishing off
After this change:
[ 0.0] Opening the source
[ 8.4] Inspecting the source
[ 14.1] Checking for sufficient free disk space in the guest
[ 14.1] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 83.5] Mapping filesystem data to avoid copying unused and blank areas
[ 87.2] Closing the overlay
[ 87.9] Assigning disks to buses
[ 87.9] Checking if the guest needs BIOS or UEFI to boot
[ 87.9] Creating output metadata
[ 94.0] Copying disk 1/1
█ 100% [****************************************]
[ 165.7] Creating output metadata
[ 165.7] Finishing off
We are now faster than virt-v2v 1.45:
[ 0.0] Opening the source -i libvirt [...]
[ 1.4] Creating an overlay to protect the source from being modified
[ 4.8] Opening the overlay
[ 17.2] Inspecting the overlay
[ 23.7] Checking for sufficient free disk space in the guest
[ 23.7] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 110.0] Mapping filesystem data to avoid copying unused and blank areas
[ 124.5] Closing the overlay
[ 125.1] Assigning disks to buses
[ 125.1] Checking if the guest needs BIOS or UEFI to boot
[ 125.1] Initializing the target -o null
[ 125.2] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw)
(100.00/100%)
[ 764.6] Creating output metadata
[ 764.6] Finishing off
Thanks: Peter Krempa
v2v: Write dir/convert and dir/copy files
During the conversion and copying phases, write files literally called
"convert" and "copy" into the v2v directory. Helpers can use these to
make decisions based on the phase of virt-v2v. In particular we will
use the presence of the "convert" file to determine if we need to
enable copy-on-read.
input: Implement nbdkit-cow-filter cow-on-read (copy on read)
This has considerable performance benefits during the conversion step.
See also:
https://listman.redhat.com/archives/libguestfs/2021-July/msg00054.html
Add list of requirements to the README
todo: Put some items left over from modularization on the backlog
input: -i disk: Always detect input format
If the input format is raw, prefer nbdkit.
v2v: Minor refactoring of the code that runs nbdcopy
convert: Remove bogus "Creating output metadata" message
Left over from virt-v2v 1.45
Richard W.M. Jones