IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Commit ade62ebdb5 ("v2v: Add support for virt-customize options")
included --inject* options that covered facilities for injecting QEMU
guest agent etc which are already part of virt-v2v. We shouldn't add
the options for that.
The generator in libguestfs was changed so we can now exclude those
options from virt-v2v.
Update common submodule to include:
Richard W.M. Jones (1):
mlcustomize: Update generated options for virt-v2v
Fixes: commit ade62ebdb50e32a05c336ffcf4c08c6996a3af36
For higher level tools making further arbitrary changes to the guest
post-conversion, especially adding more firstboot scripts, it's useful
if we have various features from virt-customize. Particularly the
ability to upload and run scripts. Since we have the virt-customize
code in the common/ submodule, the easiest thing to do is to add the
virt-customize options, the same way it is done in virt-builder and
virt-sysprep.
Also updates the common submodule to get:
Richard W.M. Jones (1):
mlcustomize: Move virt-customize modules to mlcustomize/
Currently we only print it in debugging output, but we might use it in
future. It costs nothing to get it as it is already collected for
Windows guests (and is empty for other guests).
When we get large virt-v2v logs back from customers it is often
difficult to find relevant information about the guest. Try to fix
this by tagging some important information so that it will match this
regular expression in searches:
^info:
Note I'm not including information about input and output drivers
here. If we suspect a problem there, then more detailed review of the
log is always needed, plus that information is usually much easier to
find (eg. finding the kubevirt yaml or libvirt XML is obvious even in
large log files).
The main complexity here was to ensure that all of the nbdinfo
information appears together. We need to run the command and collect
the output, otherwise it gets mixed in with debugging from nbdkit
which resulted from running nbdinfo.
Also update the common submodule to include:
Richard W.M. Jones (1):
mldrivers/linux_kernels.ml: Prefix general information with ^info:
Even if these appear in the middle of the list, allow them to be
empty. For example this is now permitted whereas previously it was an
error:
virt-v2v --mac <MAC>:ip:<ADDR>,,,nameserver
Reported-by: Arik Hadas
The option takes values of "virtio-scsi", "virtio-blk" (with the latter
being the default). It maps on the convert option with the same name
introduced in the previous commits, thus allowing us to alter the order in
which the VirtIO block drivers are going to be searched for. This is
useful if we want the virtio-scsi driver to be installed during
conversion instead of the default virtio-blk.
Also update the docs accordingly.
Originally-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Message-Id: <20230310175433.781335-6-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
[lersek@redhat.com: fix the docs to placate "test-v2v-docs.sh"]
When injecting a block VirtIO driver during conversion, we rely on the
predefined list of names of such drivers. Order in this list denotes the
priority (see common/mlcustomize/inject_virtio_win.ml).
This commit introduces the "block_driver" convert option, setting its
value to the default Virtio_blk and making sure it's being properly passed
to the conversion functions. Along with the means of altering the drivers
list (introduced in a separate commit in the common submodule), this option
will be brought to command line, giving us the opportunity to choose the
block driver to be used on conversion.
Originally-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Message-Id: <20230310175433.781335-4-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
OCaml 4.08 introduces a stdlib Option module which looks a bit like
ours but has a number of differences. In particular our functions
Option.may and Option.default have no corresponding functions in
stdlib, although there are close enough equivalents.
This change was automated using this command:
$ perl -pi.bak \
-e 's/Option.may/Option.iter/g; s/Option.default /Option.value ~default:/g' \
`git ls-files`
Update common module to include:
commit cffa077323fafcdfcf78e230c022afa891a6b3ff
Author: Richard W.M. Jones <rjones@redhat.com>
Date: Mon Feb 20 12:11:51 2023 +0000
mlstdutils: Rework the Option module to be compatible with stdlib
commit 007d0506c538db0a43fec7e9986a95ecdcd48b56
Author: Richard W.M. Jones <rjones@redhat.com>
Date: Mon Feb 20 12:18:29 2023 +0000
mltools: Replace Option.may with Option.iter
In various places we used PCRE.compile ~anchored:true (PCRE2_ANCHORED)
thinking it means that the regular expression is anchored at both
ends. However this is not what it means (it only anchors the start).
Replace with ^...$
This commit splits up any long lines found in errors, warnings or
other messages. OCaml ignores whitespace following "\<CR>" within a
string, eg:
"long string \
more stuff"
is parsed as:
"long string more stuff"
Thanks: Laszlo Ersek, for working out the OCaml syntax for this
Update common to get this commit:
mldrivers: New directory containing driver detection code
Move the existing code from virt-v2v for handling driver detection.
This is just code motion from virt-v2v, so that we can create a new
tool (in guestfs-tools) called 'virt-drivers'.
This so far only handles Linux (because virt-v2v doesn't do Windows
driver detection), but we will add Windows support in future.
This is just code motion.
The existing function Inspect_source.get_firmware_bootable_device
detected if the guest could boot using UEFI through inspection (rather
than whatever the hypervisor told us).
Move this function to a new module, renaming it as
Firmware.detect_firmware.
Since we intend to move this into common/ (so it can be used in
guestfs-tools), remove the i_firmware field from the inspect struct
and pass it around as a separate argument.
So that we can reuse this useful code in virt-customize, move it to
the common module.
This is mostly code motion, but note:
(a) the module name changes from Windows_virtio to Inject_virtio_win
(b) as there were still some dependencies on types in the virt-v2v
Types module (eg. block driver type) I had to duplicate those.
See also:
https://listman.redhat.com/archives/libguestfs/2022-December/030389.html
This library is needed by Windows_virtio. Since I intend to move that
functionality into common/mlcustomize, I must also move this library
into common/.
This large unit test hadn't been updated for quite a while and it
complicates moving this code into common/. Remove it for now, but
consider revisiting it later, especially if we can make it smaller and
more useful.
This commit adds a new -o kubevirt mode. You can specify where to
place the disks and metadata (guest.yaml) file using -os. Also
allocation mode (-oa) and disk format (-of) may be set. For example:
$ virt-v2v -i disk guest.img -o kubevirt -os /var/tmp
will generate /var/tmp/guest.yaml and /var/tmp/guest-sda
This is not finalized since undoubtedly the way the disks are handled
will have to be changed in future.
The ocaml-libvirt project (https://gitlab.com/libvirt/libvirt-ocaml)
provides bindings for libvirt. For historical reasons we bundled this
as it was throught ocaml-libvirt wasn't widespread on distros. In
fact Fedora and Debian derivatives all have this. Arch does not
(yet), but we can fix that.
It said in the README file in that directory, "before virt-v2v 1.42 is
released we hope to have unbundled this". That didn't happen, but
let's remove it now.
Only after option parsing does the -v (verbose) option take effect,
and so any debug messages emitted before this point are not seen. In
particular, debug messages emitted when creating the v2v directory
were lost. In any case there's no point creating this directory until
nearer the point when we might actually need it.
I have also renamed the directory in the code from "tmpdir" to
"v2vdir" since tmpdir was a bit generic and didn't accurately describe
what this directory is for.
This is simple refactoring.
This test was disabled in commit 255722cbf3 ("v2v: Modular virt-v2v").
This simply enables it again. I don't think the test is very valuable
so maybe we should completely remove it in future, but since the code
was still there and enabling it was trouble-free that's what I did.
Fixes: commit 255722cbf39afc0b012e2ac00d16fa6ba2f8c21f
Add a new front end called virt-v2v-in-place which implements simple
in-place conversion support for local disks.
Commit 255722cbf3 ("v2v: Modular virt-v2v") temporarily dropped this
feature. This commit adds it back.
This flag can be used for certain inputs to control whether we place a
protective overlay over the source. It is only supported for local
file type inputs (-i disk and -i libvirt). For other input types it
cannot work:
* -i ova: These cannot be updated in-place because the OVA embeds
checksums.
* Any input on VMware or Xen: There's no point doing an in-place
update for a guest which his still sitting on a foreign hypervisor.
* -i vmx: We cannot reliably update a disk in VMDK format using qemu
tools.
This flag is always set to true, so there is no effect in the current
code. It allows us to implement a second front end for in-place
support.
Output modules can specify now request_size to override the default
request size in nbdcopy.
The rhv-upload plugin is translating every NBD command to HTTP request,
translated back to NBD command on imageio server. The HTTP client and
server, and the NBD client on the imageio server side are synchronous
and implemented in python, so they have high overhead per request. To
get good performance we need to use larger request size.
Testing shows that request size of 4 MiB speeds up the copy disk phase
from 14.7 seconds to 7.9 seconds (1.8x times faster). Request size of 8
MiB is a little bit (3%) faster so we may want to tune this in the
future.
Here are stats extracted from imageio log when importing Fedora 35 image
with 3 GiB of random data. For each copy, we have 4 connection stats.
Before:
connection 1 ops, 14.767843 s
dispatch 4023 ops, 11.427662 s
zero 38 ops, 0.053840 s, 327.91 MiB, 5.95 GiB/s
write 3981 ops, 8.975877 s, 988.61 MiB, 110.14 MiB/s
flush 4 ops, 0.001023 s
connection 1 ops, 14.770026 s
dispatch 4006 ops, 11.408732 s
zero 37 ops, 0.057205 s, 633.21 MiB, 10.81 GiB/s
write 3965 ops, 8.907420 s, 986.65 MiB, 110.77 MiB/s
flush 4 ops, 0.000280 s
connection 1 ops, 14.768180 s
dispatch 4057 ops, 11.430712 s
zero 42 ops, 0.030011 s, 470.47 MiB, 15.31 GiB/s
write 4011 ops, 9.002055 s, 996.98 MiB, 110.75 MiB/s
flush 4 ops, 0.000261 s
connection 1 ops, 14.770744 s
dispatch 4037 ops, 11.462050 s
zero 45 ops, 0.026668 s, 750.82 MiB, 27.49 GiB/s
write 3988 ops, 9.002721 s, 989.36 MiB, 109.90 MiB/s
flush 4 ops, 0.000282 s
After:
connection 1 ops, 7.940377 s
dispatch 323 ops, 6.695582 s
zero 37 ops, 0.079958 s, 641.12 MiB, 7.83 GiB/s
write 282 ops, 6.300242 s, 1.01 GiB, 164.54 MiB/s
flush 4 ops, 0.000537 s
connection 1 ops, 7.908156 s
dispatch 305 ops, 6.643475 s
zero 36 ops, 0.144166 s, 509.43 MiB, 3.45 GiB/s
write 265 ops, 6.179187 s, 941.23 MiB, 152.32 MiB/s
flush 4 ops, 0.000324 s
connection 1 ops, 7.942349 s
dispatch 325 ops, 6.744800 s
zero 45 ops, 0.185335 s, 622.19 MiB, 3.28 GiB/s
write 276 ops, 6.236819 s, 995.45 MiB, 159.61 MiB/s
flush 4 ops, 0.000369 s
connection 1 ops, 7.955572 s
dispatch 317 ops, 6.721212 s
zero 43 ops, 0.135771 s, 409.68 MiB, 2.95 GiB/s
write 270 ops, 6.326366 s, 988.26 MiB, 156.21 MiB/s
flush 4 ops, 0.001439 s
Before commit 4de22686fe ("output: Turn helper into an OCaml module")
the output mode was a separate program, called twice (once for setup,
once for finalize). That meant that it had to actually parse the
output options on the command line twice. After refactoring to turn
the output mode into a regular OCaml module it still checked and
parsed the output options twice, which is obviously a waste of time.
In addition, we didn't check any of the output options until after
doing the conversion. This means user errors would appear quite late.
Apart from the reduction in work, this is a mostly neutral refactoring.
Updates: commit 4de22686fe74e1711efd9bfed3f663b67e7ad69e
When we decide which output name to use, we start with the source name
(source.s_name), and override it with the output name specified on the
command line (-on option).
This calculation is done in quite a few places.
One place is in the conversion function. This is completely
unnecessary and the commit removes it.
Also all the output functions do it, but they don't do it during other
option parsing code (ie. parse_options). I have rationalized this
slightly by moving the calculation in all output modules to the
parse_options function and passing the computed value around in the
same structure as all the other parsed options. (This change will
make more sense in the context of the following commit which further
rationalizes option parsing across all output modules.)
The change is quite large but is only refactoring and should make no
semantic change.
When running in verbose mode (virt-v2v -vx) debug messages are
supposed to only be written to stderr. This allows virt-p2v and other
wrappers to separate ordinary progress messages and warnings, from
debugging information.
However when we added nbdinfo output (only printed in verbose mode),
the output was wrongly sent to stdout instead of stderr. You can show
this as follows:
$ virt-v2v -vx -i disk /var/tmp/fedora-35.qcow2 -o null 2>/dev/null
[ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2
[ 1.0] Opening the source
... some messages elided ...
protocol: newstyle-fixed without TLS <-- debug output to stdout
export="":
export-size: 6442450944 (6G)
content: DOS/MBR boot sector
uri: nbd+unix:///?socket=/tmp/v2v.Xqvaml/in0
... etc ...
This patch sends the nbdinfo debugging output to stderr instead.
Reported-by: Tingting Zheng
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2044922
Fixes: commit 255722cbf39afc0b012e2ac00d16fa6ba2f8c21f
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Old virt-v2v would print a summary of the input and output options
before connecting to the input/output, looking something like this:
[ 0.2] Opening the source -i libvirt -ic [etc]
This gave reassurance that virt-v2v was doing something in the case
where the source was slow or unreachable. In particular if you use
-i libvirt with a vCenter URL, and the URL is wrong, libvirt hangs for
a few minutes without printing anything.
Modular virt-v2v rearranged things so the connecting phase was silent,
which meant that in the case above virt-v2v appeared to hang for a few
minutes printing nothing at all.
This change adds to_string functions to all the input and output
methods and uses them to print a message like:
[ 0.0] Setting up the source: -i libvirt -ic [etc]
The hang still happens, but at least it's now clear where it's hanging.
Note the old "Opening the source" message now refers to libguestfs
connecting to the NBD source disk pipeline.
Typical full output looks like this:
$ virt-v2v -i disk /var/tmp/fedora-35.img -o disk -os /var/tmp/out
[ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.img
[ 1.1] Opening the source
[ 5.9] Inspecting the source
[ 11.5] Checking for sufficient free disk space in the guest
[ 11.5] Converting Fedora Linux 35 (Thirty Five) to run on KVM
virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown
device "vda". You may have to fix this entry manually after conversion.
virt-v2v: This guest has virtio drivers installed.
[ 57.4] Mapping filesystem data to avoid copying unused and blank areas
[ 61.0] Closing the overlay
[ 61.7] Assigning disks to buses
[ 61.7] Checking if the guest needs BIOS or UEFI to boot
[ 61.7] Setting up the destination: -o disk -os /var/tmp/out
[ 62.8] Copying disk 1/1
█ 100% [****************************************]
[ 81.7] Creating output metadata
[ 81.7] Finishing off
Reported-by: Xiaodai Wang
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2041886
Fixes: commit 255722cbf39afc0b012e2ac00d16fa6ba2f8c21f
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
This was added when we were setting the cow-block-size to 1M. However
since commit 351d61f768 ("input: -it vddk: Reduce cow-block-size to
4K") we stopped doing that so this is no longer needed. Note that the
current nbdcopy request size is 256K (since libnbd 1.10) so requests
will be quite a bit smaller after this change.
This improves copy performance quite markedly in my test:
$ virt-builder [--format=qcow2] fedora-35 --copy-in /var/tmp/1G:/var/tmp
$ ./run virt-v2v -i disk fedora-35.[img|qcow2] -o null
raw qcow2
4M 3.1 5.4 # before this commit
256K 1.3 2.4 # after this commit
Reverts: commit 08e764959ec9dadd71a95d22d3d88d647a18d165
In old virt-v2v, we did (approximately, since the steps were not as
clear):
Input -> Convert -> Output -> Copy -> Finalize
After modularizing virt-v2v we changed this to:
Input -> Output -> Convert -> Copy -> Finalize
However this has a (sort of) problem. For -o rhv-upload when we start
the nbdkit rhv-upload-plugin machinery, it obtains a time-limited
ticket from imageio. This ticket could expire if the conversion step
takes longer than a certain time (60 seconds by default, may be
increased in a future version of oVirt).
I believe this is really a problem in imageio or that the
rhv-upload-plugin should really renew this ticket automatically, but
it does not, instead failing. (The ticket *is* renewed automatically
whenever a request is sent to imageio, but in this case no requests
are being sent).
Anyway the easiest thing is to switch the ordering back to how it was
in old virt-v2v (at top). It doesn't make a difference for any other
output modes.
Reported-by: Nir Soffer
The refactoring in commit 255722cbf39a ("v2v: Modular virt-v2v")
accidentally dropped this code which prints the version of virt-v2v
and libvirt:
f737044190/v2v/v2v.ml (L57-L66)
This is useful for debugging so add it back.
$ ./run virt-v2v -v -x
virt-v2v: virt-v2v 1.45.94local,libvirt (x86_64)
libvirt version: 7.7.0
virt-v2v: error: -i libvirt: expecting a libvirt guest name on the command
line
rm -rf '/tmp/v2v.c6aorj'
Reported-by: Ming Xie
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2032112
Fixes: commit 255722cbf39afc0b012e2ac00d16fa6ba2f8c21f
This file was contenated from the various input modes when I
modularized virt-v2v. It was done this way essentially for
convenience. However it wasn't a great idea because:
- Not very easy to see which file is responsible for each mode.
ie. it's a lot more obvious that input_disk.ml might be responsible
for -i disk.
- It didn't sufficiently isolate each mode. Each mode is logically
separate and does not need to be in the same top level file or
module.
- This single file "open"'d just about every namespace which is a
symptom of poor encapsulation.
In addition the various *_source and *_servers functions which were
called one after another are combined into a single function. (Their
separation was an unusual side effect of earlier modularization, not a
necessary feature).
This is just code movement, there is no change to functionality.
We can use the new [On_exit] module to do this much more cleanly and
reliably. Remove the need for explicit cleanup.
The existing functions were in any case somewhat broken with regard to
signal handling so a further change was made to the common submodule
to fix that.
In a future commit I will break up the large input/input.ml file,
which is now much simpler to do. However this commit sticks to the
modularization only.
Note the first class module syntax is a somewhat obscure and new OCaml
feature. For more on this topic:
https://dev.realworldocaml.org/first-class-modules.html
Splitting virt-v2v into separate helper programs was an interesting
exercise, but the helpers themselves were not independently useful
(eg. you could not run them on their own). On reflection the value of
the exercise was really twofold:
- providing the v2vdir and input and output pipelines based on NBD
- forcing stages to be independent of each other (because they ran as
separate processes), thus enforcing the split
The drawback of separate processes is:
- need to serialise and then reparse all options as command line
parameters
- various smaller problems with getting accurate timing in messages,
handling errors well, etc.
This commit turns helper-v2v-convert back into an internal module. It
still has the strong separation, but will now be used directly as an
OCaml module.
The QXL drivers no longer matter when converting Windows guests; remove
the fake images and their references from the test suite.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1961107
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20211202094637.8020-11-lersek@redhat.com>
A typo ensured that compat output options (which are legacy options
like --vdsm-image-uuid which are turned into -oo options
transparently) were being passed to the input helper subprocess. The
reason this didn't create any test failures is because we don't
actually test those legacy options. Normal -oo options were being
passed around correctly and are not affected by this change.
Fixes: commit 255722cbf39afc0b012e2ac00d16fa6ba2f8c21f
Otherwise the build fails with errors such as
> GEN helper-v2v-output
> /usr/bin/ld: cannot find -lnbd
> collect2: error: ld returned 1 exit status
> File "caml_startup", line 1:
> Error: Error during linking (exit code 1)
in the "convert", "output" and "v2v" modules.
(This patch is similar to guestfs-tools commit 9b0f2e2d5893 ("Makefile.am:
use $(LIBGUESTFS_LIBS) for linking OCaml programs", 2021-09-06).)
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20210927063458.4824-2-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
This was a debug option which was used during the tests and abused by
other callers. With modular virt-v2v we can offer a better solution,
so remove the option completely.
This improves performance in the VDDK case by avoiding splitting up
reads (which is expensive with VDDK). It probably won't affect other
input modes, but we may need to revisit this if there are performance
regressions.
Old virt-v2v (~10 mins):
nbdkit: debug: VixDiskLib_Read 305,838,907
nbdkit: debug: VixDiskLib_QueryAllocatedBlocks 295,215,654
New virt-v2v, cow-block-size=64K (~18 mins):
nbdkit: debug: VixDiskLib_QueryAllocatedBlocks 967,852,565
nbdkit: debug: VixDiskLib_Read 602,845,730
New virt-v2v, cow-block-size=1M (~14 mins):
nbdkit: debug: VixDiskLib_Read 2,168,071,068
nbdkit: debug: VixDiskLib_QueryAllocatedBlocks 457,364,633
New virt-v2v, cow-block-size=1M, nbdcopy --request-size=4M (~10 mins):
nbdkit: debug: VixDiskLib_Read 747,427,040
nbdkit: debug: VixDiskLib_QueryAllocatedBlocks 450,574,928
Split virt-v2v into several cooperating helper programs. Use disk
image pipelines on both the input and output sides even when accessing
local files. Expose the NBD sockets. Use nbdcopy for the copy step.
Some features have been removed and we intend to add those back later
(see TODO file).
For the original plan to split virt-v2v, see:
https://listman.redhat.com/archives/libguestfs/2020-November/msg00022.html
Thanks: Ming Xie, Tingting Zheng, Nir Soffer, Eric Blake, Martin Kletzander
This change is made up of many separate commits done during
development. The history of those commit messages is preserved below,
but the individual commits do not make too much sense so they have
been squashed into a single large change.
v2v: Move library-ish parts of virt-v2v into lib/ subdirectory
In preparation for splitting virt-v2v, moving library-ish parts of the
code that we wish to reuse in the new helpers into the lib/
subdirectory.
This is neutral code refactoring.
lib: Define format for metadata
In a previous iteration of the virt-v2v split I proposed using an open
format for metadata such as XML, and actually implemented much of it.
However to keep this change simple, and because no one except us is
supposed to be generating or consuming this metadata, this commit
replaces the open format with a simple OCaml serialization of an
opaque version string + the struct (eg. Types.sources). The opaque
version string is there to ensure binary compatibility between the
helpers and to discourage people from trying to write or consume the
metadata.
Note: The metadata is not ABI and will change arbitrarily between
releases. If you need to write or consume the metadata it's best to
talk to us about what you're trying to do.
inputs: Create helper-v2v-input-disk
As part of splitting up virt-v2v create input helpers. This is the
first and simplest input helper which implements the “virt-v2v -i disk”
functionality, ie. being able to drive virt-v2v from a local disk file
without any metadata.
For further details on the virt-v2v split, refer to this plan:
https://listman.redhat.com/archives/libguestfs/2020-November/msg00022.html
outputs: Create helper-v2v-output-disk
This is the simplest possible output helper. It creates the output
disks (really: processes and sockets). Note this does not yet create
the final libvirt XML. This will be added in a later commit.
convert: Create helper-v2v-convert
This commit moves the conversion code into a separate helper program
(helper-v2v-convert) which performs the conversion on the input disks.
The input disks are actually COW overlays over the source disks so
that nothing is changed on the source.
This step creates metadata files: guestcaps, inspect, target_buses and
target_firmware corresponding to the internal data structures. These
will be consumed by the output finalization step.
v2v: Get rid of Modules_list
This functionality will be replaced in the new virt-v2v.
v2v: Rearrange sources into input/ and output/ directories
Rearrange sources for incomplete input and output drivers into the new
directories.
lib: Remove unused input and output objects
These objects are no longer required after creating the modular input
and output helpers.
lib/nbdkit.ml: Add LANG=C for all nbdkit instances
In old virt-v2v this was added through the Nbdkit_sources module to
all instances of nbdkit. Add it unconditionally through Nbdkit module
to get the same effect.
outputs: Create helper-v2v-output-null
This handles -o null conversions.
v2v: Add new virt-v2v command line parser and program
In the newly modular virt-v2v, this program is responsible for
handling compatibility with the old virt-v2v command line. It will
continue to be the main way that people use virt-v2v for the
foreseeable future. This program starts the helper programs and
handles multiplexing of virt-v2v command line parameters to the right
helper.
docs, tests: Adjust --no-copy documentation and tests
Since copying and creating the output are now handled in separate
programs, --no-copy will usually create the output disks (but empty).
Adjust documentation and tests accordingly.
It's probably better to remove this option.
inputs: Create helper-v2v-input-libvirt
This handles "-i libvirtxml" (input from libvirt XML file), and all
"-i libvirt" cases which are not handled by more specific code
(ie. not vcenter-https, not vddk, not xen-ssh).
outputs: Finish finalization code for helper-v2v-output-disk
Create the final libvirt XML.
lib, tests: Don't print unused field in source_disk, fix test.
lib: Remove unused fields in s_disks struct
The fields s_qemu_uri and s_format were no longer used, remove them.
inputs: Create helper-v2v-input-ova
This handles parsing OVA files (-i ova).
outputs: Create helper-v2v-output-glance
This implements -o glance conversions to OpenStack Glance.
outputs: Create helper-v2v-output-json
Implements -o json mode.
outputs: Create helper-v2v-output-qemu
Implements -o qemu mode.
tests/test-v2v-bad-networks-and-bridges.sh: Fix test
This test depended on the specifics of parameter parsing and errors.
Adjust the test so it works with modular virt-v2v.
inputs: Combine all input helpers into one program.
This reduces the duplication of code from the previous plan. There is
now a single helper, and it uses a "hidden" -im parameter (passed by
virt-v2v) to select the input mode, eg:
helper-v2v-input -im libvirtxml v2vdir xmlfile
outputs: Combine all output helpers into one program.
This reduces duplication of code. There is now a single helper, and
it uses a "hidden" -om parameter (passed by virt-v2v) to select the
output mode, eg:
helper-v2v-output -om disk setup v2vdir -os /storage
outputs: Implement -o libvirt
inputs: Implement input from vcenter over HTTPS
This implements virt-v2v -i libvirt when we detect that the libvirt
URI points to a VMware server over HTTPS (without using VDDK).
inputs: Implement input from VMware using VDDK
This implements -i libvirt -it vddk.
inputs: Implement input from VMware via VMX
This implements -i vmx.
v2v: Fix -io ? and -oo ?
inputs: Implement input from Xen over SSH
input: Refactor input helper
Now that we have moved all the input-side code from old virt-v2v,
refactor and generally clean up.
output: Refactor output helper
General refactoring and clean up to improve the quality of the code in
the output helper.
outputs: Implement -o openstack
outputs: Implement -o rhv and -o vdsm
outputs: Implement -o rhv-upload
v2v: Run helpers with --program-name=virt-v2v
This means the helpers will use "virt-v2v" instead of "helper-v2v-..."
in error messages and similar, hopefully reducing confusion.
convert, output: Improve consistency of error messages
Don't use "prog" usually since it is added by the error function.
However occasionally when there's an internal error with virt-v2v
using the wrong arguments to the helper then we can use prog to
display the actual helper having problems.
inputs, outputs: Add cmdline abstract type
Convenient way to pass the multiple command line options as a single
parameter to functions. This is simple refactoring to make the next
change possible.
inputs, outputs: Give an error for invalid option combinations
virt-v2v 1.4x was fussy about reporting errors for options which were
not applicable in certain input or output modes. Replicate that as
much as possible here. Old virt-v2v checked output modes more
thoroughly than input modes, and I have stuck with copying that
behaviour.
This also corrects an error in -o libvirt: In old virt-v2v the output
pool defaulted to "default" rather than giving an error.
inputs, outputs: Choose qemu-nbd PID file named based on socket
Previously we attempted to choose the PID file name randomly.
Although this should never conflict, I saw one case where qemu-nbd
failed to start up, printing only:
qemu-nbd: Cannot lock pid file: Resource temporarily unavailable
My reading of the code is this could be caused by the PID file already
being locked.
Anyway there is a better way to choose PID file names: simply extend
the already unique socket name with ".pid".
v2v: Don't print double error messages
If running helper-v2v-* programs, assume that if these exit on error
then they have already printed an error message. Therefore the main
virt-v2v program does not need to print another error message.
v2v: Set permissions and SELinux labels on all sockets
When running virt-v2v as non-root (the recommended way) this all
worked fine before.
However a problem arises when running virt-v2v as root. Libvirt will
run qemu as a non-root user, so we need to set permissions
appropriately (ironically making everything a bit less secure).
Also set SELinux labels if we detect SELinux is being used.
Reported-by: Tingting Zheng
output: Explicitly shut down the NBD handle
This avoids a warning from qemu-nbd:
qemu-nbd: Disconnect client, due to: Failed to send reply: Unable to write to socket: Broken pipe
For more information about the warning, see:
https://lists.nongnu.org/archive/html/qemu-block/2021-07/msg00703.html
lib/nbdkit: Always set both socket and file labels when using SELinux
We always set the file permissions to 0777 so we might as well always
set the SELinux labels when we detect that we are using SELinux. This
avoids complexity elsewhere in virt-v2v.
inputs, outputs: Label all qemu-nbd sockets when using SELinux
Abstract qemu-nbd into a data type
Add a new module QemuNBD which contains the common code for running
qemu-nbd. Replace existing code in the input and output helpers with
this module.
v2v: In verbose mode, dump nbdinfo about each NBD socket
This could help with debugging, especially understanding if nbdcopy
can use multi-conn.
input: Use the cache filter (if available) with slow plugins
This adds the cache filter to the chain of filters for slow plugins
(curl, ssh, vddk).
There is a potential further enhancement here: using conditional
cache-on-read=/path. However that requires a very new nbdkit and
further changes elsewhere in virt-v2v.
input/nbdkit: Refactor these modules
These modules were made from old virt-v2v by splitting up the old
Nbdkit_sources module, but otherwise the code was virtually
unmodified. This refactoring eliminates code duplication and dead
code left over from the split.
Although this is mostly refactoring, I also got rid of the ability to
use nbdkit-vddk-plugin < 1.17.10, which required the awkward use to
LD_LIBRARY_PATH.
convert: Do not use qemu block layer copyonread
Before this change:
[ 0.0] Opening the source
[ 145.6] Inspecting the source
[ 988.4] Checking for sufficient free disk space in the guest
[ 988.4] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[3892.1] Mapping filesystem data to avoid copying unused and blank areas
[4125.9] Closing the overlay
[4126.6] Assigning disks to buses
[4126.6] Checking if the guest needs BIOS or UEFI to boot
[4126.6] Creating output metadata
[4132.8] Copying disk 1/1
█ 100% [****************************************]
[4205.1] Creating output metadata
[4205.1] Finishing off
After this change:
[ 0.0] Opening the source
[ 8.4] Inspecting the source
[ 14.1] Checking for sufficient free disk space in the guest
[ 14.1] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 83.5] Mapping filesystem data to avoid copying unused and blank areas
[ 87.2] Closing the overlay
[ 87.9] Assigning disks to buses
[ 87.9] Checking if the guest needs BIOS or UEFI to boot
[ 87.9] Creating output metadata
[ 94.0] Copying disk 1/1
█ 100% [****************************************]
[ 165.7] Creating output metadata
[ 165.7] Finishing off
We are now faster than virt-v2v 1.45:
[ 0.0] Opening the source -i libvirt [...]
[ 1.4] Creating an overlay to protect the source from being modified
[ 4.8] Opening the overlay
[ 17.2] Inspecting the overlay
[ 23.7] Checking for sufficient free disk space in the guest
[ 23.7] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 110.0] Mapping filesystem data to avoid copying unused and blank areas
[ 124.5] Closing the overlay
[ 125.1] Assigning disks to buses
[ 125.1] Checking if the guest needs BIOS or UEFI to boot
[ 125.1] Initializing the target -o null
[ 125.2] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw)
(100.00/100%)
[ 764.6] Creating output metadata
[ 764.6] Finishing off
Thanks: Peter Krempa
v2v: Write dir/convert and dir/copy files
During the conversion and copying phases, write files literally called
"convert" and "copy" into the v2v directory. Helpers can use these to
make decisions based on the phase of virt-v2v. In particular we will
use the presence of the "convert" file to determine if we need to
enable copy-on-read.
input: Implement nbdkit-cow-filter cow-on-read (copy on read)
This has considerable performance benefits during the conversion step.
See also:
https://listman.redhat.com/archives/libguestfs/2021-July/msg00054.html
Add list of requirements to the README
todo: Put some items left over from modularization on the backlog
input: -i disk: Always detect input format
If the input format is raw, prefer nbdkit.
v2v: Minor refactoring of the code that runs nbdcopy
convert: Remove bogus "Creating output metadata" message
Left over from virt-v2v 1.45
The amount of memory requested depends on libguestfs. In this
libguestfs commit:
224f373043
the size of the appliance was increased to 1280M. This causes
virt-v2v to allocate 3.5G, which is much more than we need.
Previous virt-v2v commit fa8d5419c830eb942679d9e341b2b67f58324a31
adjusted the calculation to bring the total back down to 2G. This
commit adjusts the calculation again, bringing it back down to 2.5G.
Setting/adjusting the number of drive heads in the NTFS header is only
necessary for ancient versions of Windows. Modern versions ignore
this. In addition this operation broke when we added BitLocker
support. Only do this for ancient Windows 2000/XP and skip it for
everything else.
Reported-by: Ming Xie
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1994984
