IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
For higher level tools making further arbitrary changes to the guest
post-conversion, especially adding more firstboot scripts, it's useful
if we have various features from virt-customize. Particularly the
ability to upload and run scripts. Since we have the virt-customize
code in the common/ submodule, the easiest thing to do is to add the
virt-customize options, the same way it is done in virt-builder and
virt-sysprep.
Also updates the common submodule to get:
Richard W.M. Jones (1):
mlcustomize: Move virt-customize modules to mlcustomize/
nbdinfo --size prints the size on stdout, causing it to appear in
virt-v2v output. Using --can connect instead is silent.
Note that nbdinfo --can connect was added in libnbd 1.9.2 (Jul 2021)
and our previous minimum version of libnbd was 1.9.3 so we're OK.
However since neither of these was a stable version I also updated the
minimum libnbd requirement to 1.10 (Sep 2021), and added a proper
check in ./configure
Fixes: commit fb72e059863a60503b6011b8590c25c3a010a58f
Since OCaml 4.07 (released 2018-07-10) the always-loaded standard
library module has been called Stdlib. The old Pervasives module was
finally removed in OCaml 5.
$ perl -pi.bak -e 's/Pervasives\./Stdlib./g' -- `git ls-files`
OCaml >= 4.07 is now required.
Also update the common submodule with:
commit d61cd820b49e403848d15c5deaccbf8dd7045370
Author: Jürgen Hötzel
Date: Sat May 20 18:16:40 2023 +0200
Add support for OCaml 5.0
In commit 1e0d615a88 ("build: Don't write valgrind log files to tmp/
directory") I added this option without really thinking too much about
it. However it makes the output of valgrind much harder to read
because it includes many traces which are not counted as true leaks.
Fixes: commit 1e0d615a883e4056b757d041a1ad1bbd39df3b3c
The ocaml-libvirt project (https://gitlab.com/libvirt/libvirt-ocaml)
provides bindings for libvirt. For historical reasons we bundled this
as it was throught ocaml-libvirt wasn't widespread on distros. In
fact Fedora and Debian derivatives all have this. Arch does not
(yet), but we can fix that.
It said in the README file in that directory, "before virt-v2v 1.42 is
released we hope to have unbundled this". That didn't happen, but
let's remove it now.
Inherited from libguestfs, we wrote valgrind log files into the tmp/
directory during tests. However this makes it very difficult to
connect a particular log file to a particular failing test. As with
nbdkit let's just write the valgrind output to stderr, where it will
end up in the test log file (ie. tests/test-suite.log or tests/*.log).
I also adjusted the valgrind parameters so they more closely match
those used by nbdkit.
The next libguestfs-common submodule update will assume
GUESTFS_HAVE_LUKS_UUID and GUESTFS_HAVE_CRYPTSETUP_OPEN;
guestfs_cryptsetup_open() is available starting with libguestfs-1.44.
Suggested-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220228155717.18077-1-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Warning 6 "labels-omitted" is not useful. It's fine to omit labels on
positional arguments.
Example:
File "perl_edit.ml", line 30, characters 2-13:
30 | c_edit_file (verbose ()) g (Guestfs.c_pointer g) file expr
^^^^^^^^^^^
Warning 6 [labels-omitted]: label verbose was omitted in the application of this function.
The function is specified as:
external c_edit_file : verbose:bool -> Guestfs.t -> int64 -> string -> string -> unit
The complaint is that the verbose: label has been omitted from the
first argument when the function is called, but IMO this is a
stylistic thing, not a bug.
(cherry picked from
guestfs-tools commit 577f7aee4b1c720f4c4826115b49a0c3870b149e)
In OCaml 4.13:
File "perl_edit.ml", line 30, characters 2-13:
30 | c_edit_file (verbose ()) g (Guestfs.c_pointer g) file expr
^^^^^^^^^^^
Error (warning 6 [labels-omitted]): label verbose was omitted in the application of this function.
(cherry picked from
guestfs-tools commit a4930f5fad82e5358d565b8cf3610970e9646259)
This has been in OCaml since 4.02, and we've required 4.02 for a long
time (and now require 4.03). In any case the result of the test is no
longer used anywhere so the test was redundant.
In OCaml 4.13:
Alert ocaml_deprecated_cli: Setting a warning with a sequence of lowercase or uppercase letters,
like 'CDEFLMPSUVYZX', is deprecated.
Use the equivalent signed form: +C+D+E+F+L+M+P+S+U+V+Y+Z+X+52-3.
(cherry picked from
guestfs-tools commit fa4f59e1d99c08d7e0bae2a7cb54f254a6506d67)
Split virt-v2v into several cooperating helper programs. Use disk
image pipelines on both the input and output sides even when accessing
local files. Expose the NBD sockets. Use nbdcopy for the copy step.
Some features have been removed and we intend to add those back later
(see TODO file).
For the original plan to split virt-v2v, see:
https://listman.redhat.com/archives/libguestfs/2020-November/msg00022.html
Thanks: Ming Xie, Tingting Zheng, Nir Soffer, Eric Blake, Martin Kletzander
This change is made up of many separate commits done during
development. The history of those commit messages is preserved below,
but the individual commits do not make too much sense so they have
been squashed into a single large change.
v2v: Move library-ish parts of virt-v2v into lib/ subdirectory
In preparation for splitting virt-v2v, moving library-ish parts of the
code that we wish to reuse in the new helpers into the lib/
subdirectory.
This is neutral code refactoring.
lib: Define format for metadata
In a previous iteration of the virt-v2v split I proposed using an open
format for metadata such as XML, and actually implemented much of it.
However to keep this change simple, and because no one except us is
supposed to be generating or consuming this metadata, this commit
replaces the open format with a simple OCaml serialization of an
opaque version string + the struct (eg. Types.sources). The opaque
version string is there to ensure binary compatibility between the
helpers and to discourage people from trying to write or consume the
metadata.
Note: The metadata is not ABI and will change arbitrarily between
releases. If you need to write or consume the metadata it's best to
talk to us about what you're trying to do.
inputs: Create helper-v2v-input-disk
As part of splitting up virt-v2v create input helpers. This is the
first and simplest input helper which implements the “virt-v2v -i disk”
functionality, ie. being able to drive virt-v2v from a local disk file
without any metadata.
For further details on the virt-v2v split, refer to this plan:
https://listman.redhat.com/archives/libguestfs/2020-November/msg00022.html
outputs: Create helper-v2v-output-disk
This is the simplest possible output helper. It creates the output
disks (really: processes and sockets). Note this does not yet create
the final libvirt XML. This will be added in a later commit.
convert: Create helper-v2v-convert
This commit moves the conversion code into a separate helper program
(helper-v2v-convert) which performs the conversion on the input disks.
The input disks are actually COW overlays over the source disks so
that nothing is changed on the source.
This step creates metadata files: guestcaps, inspect, target_buses and
target_firmware corresponding to the internal data structures. These
will be consumed by the output finalization step.
v2v: Get rid of Modules_list
This functionality will be replaced in the new virt-v2v.
v2v: Rearrange sources into input/ and output/ directories
Rearrange sources for incomplete input and output drivers into the new
directories.
lib: Remove unused input and output objects
These objects are no longer required after creating the modular input
and output helpers.
lib/nbdkit.ml: Add LANG=C for all nbdkit instances
In old virt-v2v this was added through the Nbdkit_sources module to
all instances of nbdkit. Add it unconditionally through Nbdkit module
to get the same effect.
outputs: Create helper-v2v-output-null
This handles -o null conversions.
v2v: Add new virt-v2v command line parser and program
In the newly modular virt-v2v, this program is responsible for
handling compatibility with the old virt-v2v command line. It will
continue to be the main way that people use virt-v2v for the
foreseeable future. This program starts the helper programs and
handles multiplexing of virt-v2v command line parameters to the right
helper.
docs, tests: Adjust --no-copy documentation and tests
Since copying and creating the output are now handled in separate
programs, --no-copy will usually create the output disks (but empty).
Adjust documentation and tests accordingly.
It's probably better to remove this option.
inputs: Create helper-v2v-input-libvirt
This handles "-i libvirtxml" (input from libvirt XML file), and all
"-i libvirt" cases which are not handled by more specific code
(ie. not vcenter-https, not vddk, not xen-ssh).
outputs: Finish finalization code for helper-v2v-output-disk
Create the final libvirt XML.
lib, tests: Don't print unused field in source_disk, fix test.
lib: Remove unused fields in s_disks struct
The fields s_qemu_uri and s_format were no longer used, remove them.
inputs: Create helper-v2v-input-ova
This handles parsing OVA files (-i ova).
outputs: Create helper-v2v-output-glance
This implements -o glance conversions to OpenStack Glance.
outputs: Create helper-v2v-output-json
Implements -o json mode.
outputs: Create helper-v2v-output-qemu
Implements -o qemu mode.
tests/test-v2v-bad-networks-and-bridges.sh: Fix test
This test depended on the specifics of parameter parsing and errors.
Adjust the test so it works with modular virt-v2v.
inputs: Combine all input helpers into one program.
This reduces the duplication of code from the previous plan. There is
now a single helper, and it uses a "hidden" -im parameter (passed by
virt-v2v) to select the input mode, eg:
helper-v2v-input -im libvirtxml v2vdir xmlfile
outputs: Combine all output helpers into one program.
This reduces duplication of code. There is now a single helper, and
it uses a "hidden" -om parameter (passed by virt-v2v) to select the
output mode, eg:
helper-v2v-output -om disk setup v2vdir -os /storage
outputs: Implement -o libvirt
inputs: Implement input from vcenter over HTTPS
This implements virt-v2v -i libvirt when we detect that the libvirt
URI points to a VMware server over HTTPS (without using VDDK).
inputs: Implement input from VMware using VDDK
This implements -i libvirt -it vddk.
inputs: Implement input from VMware via VMX
This implements -i vmx.
v2v: Fix -io ? and -oo ?
inputs: Implement input from Xen over SSH
input: Refactor input helper
Now that we have moved all the input-side code from old virt-v2v,
refactor and generally clean up.
output: Refactor output helper
General refactoring and clean up to improve the quality of the code in
the output helper.
outputs: Implement -o openstack
outputs: Implement -o rhv and -o vdsm
outputs: Implement -o rhv-upload
v2v: Run helpers with --program-name=virt-v2v
This means the helpers will use "virt-v2v" instead of "helper-v2v-..."
in error messages and similar, hopefully reducing confusion.
convert, output: Improve consistency of error messages
Don't use "prog" usually since it is added by the error function.
However occasionally when there's an internal error with virt-v2v
using the wrong arguments to the helper then we can use prog to
display the actual helper having problems.
inputs, outputs: Add cmdline abstract type
Convenient way to pass the multiple command line options as a single
parameter to functions. This is simple refactoring to make the next
change possible.
inputs, outputs: Give an error for invalid option combinations
virt-v2v 1.4x was fussy about reporting errors for options which were
not applicable in certain input or output modes. Replicate that as
much as possible here. Old virt-v2v checked output modes more
thoroughly than input modes, and I have stuck with copying that
behaviour.
This also corrects an error in -o libvirt: In old virt-v2v the output
pool defaulted to "default" rather than giving an error.
inputs, outputs: Choose qemu-nbd PID file named based on socket
Previously we attempted to choose the PID file name randomly.
Although this should never conflict, I saw one case where qemu-nbd
failed to start up, printing only:
qemu-nbd: Cannot lock pid file: Resource temporarily unavailable
My reading of the code is this could be caused by the PID file already
being locked.
Anyway there is a better way to choose PID file names: simply extend
the already unique socket name with ".pid".
v2v: Don't print double error messages
If running helper-v2v-* programs, assume that if these exit on error
then they have already printed an error message. Therefore the main
virt-v2v program does not need to print another error message.
v2v: Set permissions and SELinux labels on all sockets
When running virt-v2v as non-root (the recommended way) this all
worked fine before.
However a problem arises when running virt-v2v as root. Libvirt will
run qemu as a non-root user, so we need to set permissions
appropriately (ironically making everything a bit less secure).
Also set SELinux labels if we detect SELinux is being used.
Reported-by: Tingting Zheng
output: Explicitly shut down the NBD handle
This avoids a warning from qemu-nbd:
qemu-nbd: Disconnect client, due to: Failed to send reply: Unable to write to socket: Broken pipe
For more information about the warning, see:
https://lists.nongnu.org/archive/html/qemu-block/2021-07/msg00703.html
lib/nbdkit: Always set both socket and file labels when using SELinux
We always set the file permissions to 0777 so we might as well always
set the SELinux labels when we detect that we are using SELinux. This
avoids complexity elsewhere in virt-v2v.
inputs, outputs: Label all qemu-nbd sockets when using SELinux
Abstract qemu-nbd into a data type
Add a new module QemuNBD which contains the common code for running
qemu-nbd. Replace existing code in the input and output helpers with
this module.
v2v: In verbose mode, dump nbdinfo about each NBD socket
This could help with debugging, especially understanding if nbdcopy
can use multi-conn.
input: Use the cache filter (if available) with slow plugins
This adds the cache filter to the chain of filters for slow plugins
(curl, ssh, vddk).
There is a potential further enhancement here: using conditional
cache-on-read=/path. However that requires a very new nbdkit and
further changes elsewhere in virt-v2v.
input/nbdkit: Refactor these modules
These modules were made from old virt-v2v by splitting up the old
Nbdkit_sources module, but otherwise the code was virtually
unmodified. This refactoring eliminates code duplication and dead
code left over from the split.
Although this is mostly refactoring, I also got rid of the ability to
use nbdkit-vddk-plugin < 1.17.10, which required the awkward use to
LD_LIBRARY_PATH.
convert: Do not use qemu block layer copyonread
Before this change:
[ 0.0] Opening the source
[ 145.6] Inspecting the source
[ 988.4] Checking for sufficient free disk space in the guest
[ 988.4] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[3892.1] Mapping filesystem data to avoid copying unused and blank areas
[4125.9] Closing the overlay
[4126.6] Assigning disks to buses
[4126.6] Checking if the guest needs BIOS or UEFI to boot
[4126.6] Creating output metadata
[4132.8] Copying disk 1/1
█ 100% [****************************************]
[4205.1] Creating output metadata
[4205.1] Finishing off
After this change:
[ 0.0] Opening the source
[ 8.4] Inspecting the source
[ 14.1] Checking for sufficient free disk space in the guest
[ 14.1] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 83.5] Mapping filesystem data to avoid copying unused and blank areas
[ 87.2] Closing the overlay
[ 87.9] Assigning disks to buses
[ 87.9] Checking if the guest needs BIOS or UEFI to boot
[ 87.9] Creating output metadata
[ 94.0] Copying disk 1/1
█ 100% [****************************************]
[ 165.7] Creating output metadata
[ 165.7] Finishing off
We are now faster than virt-v2v 1.45:
[ 0.0] Opening the source -i libvirt [...]
[ 1.4] Creating an overlay to protect the source from being modified
[ 4.8] Opening the overlay
[ 17.2] Inspecting the overlay
[ 23.7] Checking for sufficient free disk space in the guest
[ 23.7] Converting Fedora 28 (Server Edition) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 110.0] Mapping filesystem data to avoid copying unused and blank areas
[ 124.5] Closing the overlay
[ 125.1] Assigning disks to buses
[ 125.1] Checking if the guest needs BIOS or UEFI to boot
[ 125.1] Initializing the target -o null
[ 125.2] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw)
(100.00/100%)
[ 764.6] Creating output metadata
[ 764.6] Finishing off
Thanks: Peter Krempa
v2v: Write dir/convert and dir/copy files
During the conversion and copying phases, write files literally called
"convert" and "copy" into the v2v directory. Helpers can use these to
make decisions based on the phase of virt-v2v. In particular we will
use the presence of the "convert" file to determine if we need to
enable copy-on-read.
input: Implement nbdkit-cow-filter cow-on-read (copy on read)
This has considerable performance benefits during the conversion step.
See also:
https://listman.redhat.com/archives/libguestfs/2021-July/msg00054.html
Add list of requirements to the README
todo: Put some items left over from modularization on the backlog
input: -i disk: Always detect input format
If the input format is raw, prefer nbdkit.
v2v: Minor refactoring of the code that runs nbdcopy
convert: Remove bogus "Creating output metadata" message
Left over from virt-v2v 1.45
This Debian page explains the upstream situation:
https://wiki.debian.org/genisoimage
On Fedora, xorriso provides a compatibility program called "mkisofs".
However this is not present in Debian. Hence the choice to look for
the program called "xorrisofs".
(cherry picked from commit 2216ab2e328457ef172d6bfa534272edf2f81a3a)
libguestfs 1.45.3 now reads the RPM database using librpm, which means
our old phony database created by db_dump can no longer work. Instead
provide a real (but very minimal) sqlite database.
This commit also fixes the virt-inspector test since the RPM database
contents are now different.
With the Weblate adoption, we let it update the po files from the
catalog template. The po4a behaviour of extracting the template,
merging the existing translations, and creating the translated PODs at
once is problematic. Hence, split the extraction and the translated POD
generation in two.
Use po4a-gettextize to extract the catalog template only, not doing it
anymore automatically at each build. There is no more need for a
po4a.conf file.
Use po4a-translate to create translated PODs from the po files, keeping
the fixup of the generated files (to avoid spurious =encoding, etc).
Add a silent rule to hide the po4a-translate command lines by default.
These changes also allow us to get rid of the POD existance checks with
associated error message pointing to the update-po rule. Now each
translated POD file is generated because of make dependency, and it
depends only on its po file.
In commit 5f355c2952729581f1b988297fe8cabe756c01e2 I incorrectly
rewrote ‘libtool’ from the original libguestfs file as ‘$(LIBTOOL)’.
This was because of a comment that appears in the libguestfs file:
Note we run libtool, not $(LIBTOOL) since the latter expands to
libtool-kill-dependency-libs.sh
Since virt-v2v doesn't have libtool-kill-dependency-libs.sh, I
reasoned that we should change libtool back to $(LIBTOOL) and remove
the comment.
However the result of this was wrong. $(LIBTOOL) expands to
$(top_builddir)/libtool, but this breaks (because of the relative
path) whenever the VG macro is used within pushd...popd pairs, as
happens in several tests.
Restoring the libguestfs version fixes this.
The test for this was missing, so it was always defined wrongly. In
any case nothing in libguestfs-common or virt-v2v actually uses this
flag (it's used by the libguestfs daemon) so the code was dead.
This fails with Fedora 32, and possibly earlier versions:
checking if OCaml ‘-runtime-variant _pic’ works... no
The reason is:
$ ocamlc -runtime-variant _pic test.ml -o test
File "test.ml", line 1:
Error: Cannot find file camlheader_pic
which may even be a packaging error in the Fedora package. However it
makes no sense to test the bytecode compiler since we don't use it on
any architecture we care about and bytecode doesn't even contain a
linked runtime. Changing the test to use ocamlopt instead of ocamlc
fixes the problem.
(cherry picked from
libguestfs commit 711fab172ca2b2cad5d91636cd408dcefc5cea44)
It turns out that
a) the standalone pep8 executable is deprecated in favour of pycodestyle
b) $python-pep8 does not exist in modern distros
and thus the style checks for Python scripts have been skipped for a
long time.
Instead, switch to pycodestyle directly, updating the list of ignored
issues to the two categories left.
Remove various redundant checks for headers, functions, libraries,
tools, and other stuff not actually used neither by virt-v2v nor by its
submodules from common.
Also, stop building the mlaugeas, and windows submodules, as they are
not used.
1.41 was only required to run a single test, but in all other respects
1.40 works fine. Relax the dependency and fix the test instead.
Thanks: Nir Soffer.
