1
0
mirror of https://github.com/OpenNebula/one.git synced 2025-01-07 17:17:41 +03:00
Commit Graph

419 Commits

Author SHA1 Message Date
Pavel Czerný
fa73142e7d
F #6723: Remove onehost importvm (#3231)
* Remove vm_import table from DB
* Remove imported vms actions
* Fix fsck for image and network
* onedb fsck fix running_vms only for non-backup images
2024-11-21 17:31:47 +01:00
Pavel Czerný
e13c329db4
F #2111: Optimize VM history handling (#3243)
- oned load only last 2 history records (not the full list)
- Dump all history records only if needed in VirtualMachine::to_xml.
- Dump conforms XML schecam and removes VM template from history records.

Speed up of onevm show command:
  - for small SQLite DB is for VM with 500 histories: 130 ms down to 5 ms
  - for big MySQL DB VM with 687 histories: 1000 ms down to 200 ms
2024-09-30 14:14:37 +02:00
Pavel Czerný
0cde7d96a2
B OpenNebula/one#6596: Fix Host NUMA nodes after VM migration (#3226)
* Fix Host NUMA nodes after VM migration
* Move template parsing to HostShareCapacity

+ add warning about unsafe pointer usage
2024-09-24 09:11:35 +02:00
Tino Vázquez
05d147c524
M #-: Bump version 6.10.0
Also bump copyright year to 2024
2024-07-29 14:27:11 +02:00
Pavel Czerný
95c51ff87f
F #5861: Search VM using JSON querry (#3102)
Signed-off-by: Pavel Czerny <pczerny@opennebula.io>
Co-authored-by: Alejandro Huertas <ahuertas@opennebula.io>
2024-06-13 12:35:15 +02:00
Pavel Czerný
2fe396ae15
L #-: Add Astyle linting
* Add AStyle to smoke tests
* Format C++ files using AStyle
2024-06-03 11:40:24 +02:00
Pavel Czerný
0f248706de
B #6498: Allow VM recover --recreate in poweroff and suspend (#3067)
* Refactor release vnc port
* Fix VNC port for recover --recreate in stopped state
* Improve error messages
*  Update recover --recreate description

(cherry picked from commit 9ed1fba8b95c1ea5d3984f136e638b973577ad69)
2024-05-21 12:11:06 +02:00
Ruben S. Montero
c509eaa815
F #6053: Renamed oned internal state name PROLOG_RESTORE to RESTORE
Now it is consistent with the API naming
2024-05-10 12:02:38 +02:00
Ruben S. Montero
3dfd53df1c
F #6053: In-place restore of VM backups (shared and ssh drivers)
This commit implements the in-place restore of VM backups. Selected VM disks will
be replaced with the specified backup:

* A new API call has been added to the XML-RPC API (`one.vm.restore`) with
  the following arguments:
    - VM ID to be restored, needs to be in **poweroff** state
    - IMAGE ID of the backup to restore
    - INCREMENT ID, only for incremental backups, the increment to use
      (defults to -1 to use the last increment available)
    - DISK ID of the disk to restore (defaults to -1 to restore all VM
      disks)

* Datastore drivers needs to implemente a new operation `ls`. This new
  operation takes the VM, image information of the backup and datastore
  information and returns the restore URL for the disks in the backup.

* This commit includes the implementation for qcow2 and ssh drivers,
  ceph will be implemented in a separated PR. The new driver action is
  `restore host:vm_dir vm_id img_id inc_id disk_id`

* The restore operation is performed in a new state `PROLOG_RESTORE`
  rendered as `RESTORE` and `rest` in short form. State in in RSuntone.

TODO:
  - Remove any existing VM snapshot (system/disk) in the VM. Note that
    snapshots are not included in a backup.

  - Ceph drivers

  - JAVA, GO Lang API bindings

  - Sunstone interface, new state and new operation. Review new state in
    RSuntone.

co-authored-by: Pavel Czerny <pczerny@opennebula.io>
2024-04-26 12:45:04 +02:00
Pavel Czerný
e4fec11dc4
B OpenNebula/one#6530: Fix max SPICE passwd length (#3009)
(cherry picked from commit be64cbdf04)
2024-03-26 12:05:15 +01:00
Pavel Czerny
57efabf2d5
B #6503: Fix disk size after disk-snapshot-revert
When a disk with snapshots is resized and a subsequent revert operation
is performed, the following actions are now done:

* DISK/SIZE is updated to reflect the active snapshot size
* Quotas are updated to account the space freed after the resize
  operation

(cherry picked from commit 178142105dea30c7deffd1842d0f0ad23d7dc93b)
(cherry picked from commit d2aaf2c8823d4172b1be673f959c4c922266534f)
2024-03-11 18:08:02 +01:00
Ruben S. Montero
174e8999d0
L #-: Removed unused method 2024-03-04 12:36:12 +01:00
Ruben S. Montero
4ce7340381
F #6492: Index PCI passthrough devices with bus
If q35 machine type is detected the slot of the pci device is set to 0
and the bus to pci_id + 1.

Q35 models uses pcie-root-ports to attach PCI devices. Each PCI port is
selected by the bus parameter of the PCI address and it that does not accept a
slot number greater than 0.

Example:

A VM with 2 X710 VFs is defined OpenNebula as:

PCI=[
  ADDRESS="0000:44:0a:0",
  BUS="44",
  CLASS="0200",
  DEVICE="154c",
  DOMAIN="0000",
  FUNCTION="0",
  NUMA_NODE="0",
  PCI_ID="0",
  SHORT_ADDRESS="44:0a.0",
  SLOT="0a",
  VENDOR="8086",
  VM_ADDRESS="01:00.0",
  VM_BUS="0x01",
  VM_DOMAIN="0x0000",
  VM_FUNCTION="0",
  VM_SLOT="0000" ]

PCI=[
  ADDRESS="0000:44:0a:1",
  BUS="44",
  CLASS="0200",
  DEVICE="154c",
  DOMAIN="0000",
  FUNCTION="1",
  NUMA_NODE="0",
  PCI_ID="1",
  SHORT_ADDRESS="44:0a.1",
  SLOT="0a",
  VENDOR="8086",
  VM_ADDRESS="02:00.0",
  VM_BUS="0x02",
  VM_DOMAIN="0x0000",
  VM_FUNCTION="0",
  VM_SLOT="0000" ]

Each PCI VFs is attached to different pcie-root-port, selected with the
VM_BUS parameter:

00:02.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port

The PCI topology is:

-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Cirrus Logic GD 5446
           +-02.0-[01]----00.0  Intel Corporation Ethernet Virtual Function 700 Series
           +-02.1-[02]----00.0  Intel Corporation Ethernet Virtual Function 700 Series
           +-02.2-[03-04]----00.0-[04]--
           +-02.3-[05]----00.0  Red Hat, Inc. Virtio network device
           +-02.4-[06]----00.0  Red Hat, Inc. Virtio SCSI
           +-02.5-[07]----00.0  Red Hat, Inc. QEMU XHCI Host Controller
           +-02.6-[08]----00.0  Red Hat, Inc. Virtio console
           +-02.7-[09]----00.0  Red Hat, Inc. Virtio memory balloon
           +-03.0-[0a]--
           +-03.1-[0b]--
           +-03.2-[0c]--
           +-03.3-[0d]--
           +-03.4-[0e]--
           +-03.5-[0f]--
           +-03.6-[10]--
           +-03.7-[11]--
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
2024-02-08 14:57:28 +01:00
Ruben S. Montero
2e596c0cf5
F #6439: PCI attach/detach operation
* Adds 2 new API calls one.vm.attachpci and one.vm.detachpci
* The operation is only for POWEROFF VMs
* Adds 2 new actions to register the event in the VM history pci-detach/pci-attach
* Ruby OCA bindings
* onevm cli commands

TODO:
- JAVA and GO bindings
- Sunstone implementation

(cherry picked from commit 5c4331a4dd)
2024-01-31 17:50:45 +01:00
Pavel Czerný
442041fd07
F #6341: Generic Quotas
- Generic quotas are defined and read from oned.conf (QUOTA_VM_ATTRIBUTE)
- Generic quotas react to the running mode (i.e. account only when the VM is
  running). For each quota oned also adds a RUNNIN_* metric
- one.system.config API call has been updated so every user can read
  quota configuration:
    . Sensitive information is hidden (***)
    . Oneadmin group has no longer access to sensitive information
      through the API
- CLI has been updated to render generic quotas
- onedb fsck fixes generic quotas reading the configuration from
  oned.conf

Other changes in this PR:
- Refactor Quota metrics to use std::vector instead of C array
Squashed commit of the following:
- New methods to Template and Attribute classes to render hidden
  attributes as "***"
- Update Quota calls to not include unneeded quotas

co-authored-by:Ruben S. Montero <rsmontero@opennebula.org>
2024-01-08 14:07:03 +01:00
Daniel Clavijo Coca
800bfbba0f
M #: Update core updateconf description with allowed changes (#2864) 2023-12-10 11:36:23 +01:00
Pavel Czerný
9cf5f85db8
B #6320: Fix history for VM disk actions (#2806) 2023-11-28 11:08:57 +01:00
Neal Hansen
c1a0e5bfc5
F OpenNebula/one#6286: Add support for VIDEO devices in KVM
- New attribute VIDEO, that includes
  - TYPE: Custom video device (none, vga, cirrus, virtio)
 - IOMMU: Enable IOMMU for the video device
 - ATS: Enable ATS (Address Translation Services) for the video device
 - VRAM: VRAM allocated to the video device (Megabytes)
 - RESOLUTION: Video resolution, in format like: 1280x720 or 1920x1080

- Support for onevm CLI command
- Updated VM xsd file.
2023-09-20 20:00:10 +02:00
Pavel Czerný
534823e7f1
B #6315: Fix remove of restricted vector attributes (#2731) 2023-09-18 16:17:59 +02:00
Ruben S. Montero
156cd04ba5
F OpenNebula/one#6265: Delete of snapshots in the middle of the chain
- Add suport for delete snapshots in the middle of the chain in qcow2
  (ALLOW_ORPHANS=NO). Example, snapshot 2 below:
                     0 <- 1 <- 2 <- 3

- It supports removing the last snapshot and active (blockpull) and
  first one (snapshot 0). The first snapshot cannot be deleted in
  persistent images because of the underlying link setting.

- Snapshots are deleted by blockcommit. For example to delete 2, 3 is
  committed into 2:
                    0 <- 1 <- 2: (2+3)
  A new file 3.current is created in the filesystem to look for the
  actual file name supporting snapshot 3.

- qcow2/shared operations snap_delete, snap_revert and cpds
  (disk-saveas) has been updated to react to the .current files

- The kvm/deploy action has been updated to resolve links in source file
  attributes. This is needed so the <backingStore> of the libvirt domain
  does not contain the sysmlinks that changes on the revert/create
  operations.

- The kvm/migrate action also includes a minor modification to resolv
  the VM disk

- Snapshots with more than 1 child (or their relatives) cannot be deleted to not
  break backing chains.

- It has been found that the snap-delete operation in suspend state is
  insecure as the guest caches may not be updated accordingly. The
  operation is now not allowed in this state.

- The associated state has been removed from oned. It needs to be
  removed in CLI/Sunstone and APIs

- The changes are for VM disk-snapshots. A similar update can be made
  for images in the datastore/fs snap_delete action. This operation is
  not implemented in this commit.

Note: Output of xmllint may or may not include new lines when multiple matches
of an xpath expression occurs. Parsing of xmllint output adds additional new lines
to deal with different versions.
2023-07-20 19:04:28 +02:00
Ruben S. Montero
6bbfbb03e4
F #6063: Backup Jobs for OpenNebula
Includes the following changes:
- xml-schema for Backup Job and Scheduled Actions
- GO, Java api
- Deprecate onevm update-chart, delete-chart
    * The commands are replaced by sched-update and sched-delete
    * Refactor method deprecate_command, it's still possible to run the
    command
    * Delete 'shutdown' and 'delete' commands deprecated years ago
    * Fix --verbose option for sched-update and sched-delete
- Re-implementation of scheduled actions, now are managed and executed
  by oned
- Backup Job objects, API, and CLI commands
2023-07-03 18:15:52 +02:00
Pavel Czerný
03add6d9ac
F #6103: CppCheck version 2.10 - solved issues (#2492)
* Update smoke test checkout action

(cherry picked from commit 466d336d7e)
2023-02-14 09:51:04 +01:00
Pavel Czerný
8e7ff2ed36
F #6103: Last set of CppCheck issues (#2481)
(cherry picked from commit 4a9662a551)
(cherry picked from commit d7b7303f8f)
2023-02-07 08:56:52 +01:00
Pavel Czerný
d7f1e1e0fc
B #6085: Fix trim of VNC/SPICE password (#2460)
(cherry picked from commit 726a6a8547)
2023-01-26 10:03:09 +01:00
Tino Vázquez
3376afdaf0
M #-: Bump year 2023 2023-01-09 12:23:19 +01:00
Pavel Czerný
b25274323e
M #-: Trim VM template error message (#2424) 2022-12-19 15:21:12 +01:00
Pavel Czerný
3beca2d9d3
B #5640: Fix NETRX and NETTX for accounting (#2417) 2022-12-12 09:28:46 +01:00
Ruben S. Montero
d711ea4362
F #5516: Incremental backups not compatible with snapshots
- Qemu will terminate VMs with system snapshots and checkpoints, example:

libvirtd[7446]: internal error: qemu unexpectedly closed the monitor: 2022-12-07T16:06:59.603678Z qemu-kvm-one: Device has active dirty bitmaps.

- Also disk snapshots may render inconsistent bitmaps, example:

'virsh --connect qemu:///system checkpoint-delete 45d76e5f-25fa-416a-97fd-1624380d5e02 one-191-0':   error: Failed to delete checkpoint one-191-0 error: internal error: unable to execute QEMU command 'query-named-block-nodes': Bitmap '' doesn't satisfy the constraints Error preparing disk files

This commits prevents snapshot operations when increment backups are configured, and vice versa.
2022-12-09 17:25:56 +01:00
Ruben S. Montero
865908de1c
F #5989: Live update of Virtual Network attributes
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>
co-authored-by: Frederick Borges <fborges@opennebula.io>
co-authored-by: Christian González <cgonzalez@opennebula.io>

* VNET updates trigger a driver action on running VMs with NICs in the
  network.

* VNET includes a sets with VM status: updated, outdated, error and
  updating. With VMs in each state.

* VNET flags error situations with a new state UPDATE_FAILURE.

* The same procedure is applied when an AR is updated (only VMs in that
  AR are updated).

* A new options in the one.vn.recover API call enable to recover or
  retry this VM update operations.

* The following attributes can be live-updated per VNET driver:
    - PHYDEV (novlan, vlan, ovs driver)
    - MTU (vlan, ovs driver)
    - VLAN_ID (vlan, ovs driver)
    - QINQ_TYPE (ovs driver)
    - CVLANS (ovs driver)
    - VLAN_TAGGED_ID (ovs driver)
    - OUTER_VLAN_ID (ovs driver)
    - INBOUND_AVG_BW (SG, ovs driver + KVM)
    - INBOUND_PEAK_BW (SG, ovs driver + KVM)
    - INBOUND_PEAK_KB (SG, ovs driver + KVM)
    - OUTBOUND_AVG_BW (SG, ovs driver + KVM)
    - OUTBOUND_PEAK_BW (SG, ovs driver + KVM)
    - OUTBOUND_PEAK_KB (SG, ovs driver + KVM)

* New API call one.vm.updatenic, allows to update individual NICs
  without the need of detach/attach (only QoS supported).

* Update operations for: 802.1Q, bridge, fw, ovswitch, ovswitch_vxlan
  and vxlan network drivers.

* VNET attributes (old values) stored in VNET_UPDATE to allow
  implementation of update operations. The attribute is removed after a
  successful update.

* Updates to CLI onevnet (--retry option) / onevm (nicupdate command)

* XSD files updated to reflect the new data model

* Ruby and JAVA bindings updated: new VNET state and recover option, new
  VM API call.

* Suntone and Fireedge implementation (lease status, recover option, new
  states)

TODO: Virtual Functions does not support this functionality

iii
2022-11-16 15:06:49 +01:00
Ruben S. Montero
0a46e21129
F #5516: Incremental backups for qcow2 disk images
- Adds new configuration attribute MODE: FULL or INCREMENTAL for
  BACKUP_CONFIG. FULL backups uses a differen backup image.

- INCREMENTAL backup information is together with the backup image.
  Example:

      <BACKUP_INCREMENTS>
        <INCREMENT>
          <DATE><![CDATA[1667770552]]></DATE>
          <ID><![CDATA[0]]></ID>
          <PARENT_ID><![CDATA[-1]]></PARENT_ID>
          <SIZE><![CDATA[172]]></SIZE>
          <SOURCE><![CDATA[bb828060]]></SOURCE>
          <TYPE><![CDATA[FULL]]></TYPE>
        </INCREMENT>
        <INCREMENT>
          <DATE><![CDATA[1667770604]]></DATE>
          <ID><![CDATA[1]]></ID>
          <PARENT_ID><![CDATA[0]]></PARENT_ID>
          <SIZE><![CDATA[1]]></SIZE>
          <SOURCE><![CDATA[ca0de5f6]]></SOURCE>
          <TYPE><![CDATA[INCREMENT]]></TYPE>
        </INCREMENT>
        <INCREMENT>
          <DATE><![CDATA[1667770700]]></DATE>
          <ID><![CDATA[2]]></ID>
          <PARENT_ID><![CDATA[1]]></PARENT_ID>
          <SIZE><![CDATA[39]]></SIZE>
          <SOURCE><![CDATA[e9897d6a]]></SOURCE>
          <TYPE><![CDATA[INCREMENT]]></TYPE>
        </INCREMENT>
      </BACKUP_INCREMENTS>

    This information only appears on incremental backups

- Internal BACKUP_CONFIG data includes information about the current
  active backup and the last increment id.

- Backup operation includes a new parameter: reset. This "closes" the
  current active incremental chain and creates a new FULL backup.

- Backup drivers store backups with increment index (0 = FULL) e.g.
  disk.0.0.

- Incremental backups are only allowed for VMs using all disks in qcow2
  format.

- Backup configuration cannot be changed while doing a VM backup.

- Downloader strings includes backup chains <inc_id>:<backup_ref>,...

- Restic downloader  has been updated to support backup chains. Disk
  images are rebased across increments.
2022-11-06 22:54:36 +01:00
Ruben S. Montero
e433ccb85b
F #5516: New backup interface for OpenNebula
co-authored-by: Frederick Borges <fborges@opennebula.io>
co-authored-by: Neal Hansen <nhansen@opennebula.io>
co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io>
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>

BACKUP INTERFACE
=================

* Backups are exposed through a a special Datastore (BACKUP_DS) and
  Image (BACKUP) types. These new types can only be used for backup'ing
  up VMs. This approach allows to:

  - Implement tier based backup policies (backups made on different
    locations).

  - Leverage access control and quota systems

  - Support differnt storage and backup technologies

* Backup interface for the VMs:

  - VM configures backups with BACKUP_CONFIG. This attribute can be set
    in the VM template or updated with updateconf API call. It can include:

    + BACKUP_VOLATILE: To backup or not volatile disks

    + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent,
      suspend or none). When possible backups are crash consistent.

    + KEEP_LAST: keep only a given number of backups.

  - Backups are initiated by the one.vm.backup API call that requires
    the target Datastore to perform the backup (one-shot). This is
    exposed by the onevm backup command.

  - Backups can be periodic through scheduled actions.

  - Backup configuration is updated with one.vm.updateconf API call.

* Restore interface:

  - Restores are initiated by the one.image.restore API call. This is
    exposed by oneimage restore command.

  - Restore include configurable options for the VM template

    + NO_IP: to not preserve IP addresses (but keep the NICs and network
      mapping)

    + NO_NIC: to not preserve network mappings

  - Other template attributes:

    + Clean PCI devices, including network configuration in case of TYPE=NIC
    attributes. By default it removes SHORT_ADDRESS and leave the "auto"
    selection attributes.

    + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node

  - It is possible to restore single files stored in the repository by
    using the backup specific URL.

* Sunstone (Ruby version) has been updated to expose this feautres.

BACKUP DRIVERS & IMPLEMENTATION
===============================

* Backup operation is implemented by a combination of 3 driver operations:

  - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate
    backups for RUNNING VMs.

  - TM. This commit introduces 2 new operations (and their
    corresponding _live variants):

    + pre_backup(_live): Prepares the disks to be back'ed up in the
      repository. It is specific to the driver: (i) ceph uses the export
      operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as
      needed.
    + post_backup(_live): Performs cleanning operations, i.e. KVM
      snapshots or tmp dirs.

  - DATASTORE. Each backup technology is represented by its
    corresponfing driver, that needs to implement:

    + backup: it takes the VM disks in file (qcow2) format and stores it
      the backup repository.

    + restore: it takes a backup image and restores the associated disks
      and VM template.

    + monitor: to gather available space in the repository

    + rm: to remove existing backups

    + stat: to return the "restored" size of a disk stored in a backup

    + downloader pseudo-URL handler: in the form
      <backup_proto>://<driver_snapshot_id>/<disk filename>

BACKUP MANAGEMENT
=================

Backup actions may potentially take some time, leaving some vmm_exec threads in
use for a long time, stucking other vmm operations. Backups are planned
by the scheduler through the sched action interface.

Two attributes has been added to sched.conf:
  * MAX_BACKUPS max active backup operations in the cloud. No more
    backups will be started beyond this limit.

  * MAX_BACKUPS_HOST max number of backups per host

* Fix onevm CLI to properly show and manage schedule actions. --schedule
  supports now, as well as relative times +<seconds_from_stime>

  onvm backup --schedule now -d 100 63

* Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs
  to use the batch interface or request specific permissions

Internal restructure of Scheduler:

- All sched_actions interface is now in SchedActionsXML class and files.
  This class uses references to VM XML, and MUST be used in the same
  lifetime scope.

- XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as
  static functions.

- VirtualMachineActionPool includes counters for active backups (total
  and per host).

SUPPORTED PLATFORMS
====================
* hypervisor: KVM
* TM: qcow2/shared/ssh, ceph
* backup: restic, rsync

Notes on Ceph

* Ceph backups are performed in the following steps:
    1. A snapshot of each disk is taken (group snapshots cannot be used as
       it seems we cannot export the disks afterwards)
    2. Disks are export to a file
    3. File is converted to qcow2 format
    4. Disk files are upload to the backup repo

TODO:
  * Confirm crash consistent snapshots cannot be used in Ceph

TODO:
  * Check if using VM dir instead of full path is better to accomodate
    DS migrations i.e.:
    - Current path: /var/lib/one/datastores/100/53/backup/disk.0
    - Proposal: 53/backup/disk.0

RESTIC DRIVER
=============
Developed together with this feature is part of the EE edtion.

* It supports the SFTP protocol, the following attributes are
  supported:

  - RESTIC_SFTP_SERVER
  - RESTIC_SFTP_USER: only if different from oneadmin
  - RESTIC_PASSWORD
  - RESTIC_IONICE: Run restic under a given ionice priority (class 2)
  - RESTIC_NICE: Run restic under a given nice
  - RESTIC_BWLIMIT: Limit restic upload/download BW
  - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes:
    off, auto, max). This requires repositories version 2. By default,
    auto is used (average compression without to much CPU usage)
  - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a
    backend (5 by default). For high-latency backends this number can be
    increased.

* downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name>
  snapshot_id is the restic snapshot hash. To recover single disk images
  from a backup. This URLs support:

  - RESTIC_CONNECTIONS
  - RESTIC_BWLIMIT
  - RESTIC_IONICE
  - RESTIC_NICE

  These options needs to be defined in the associated datastore.

RSYNC DRIVER
=============
A rsync driver is included as part of the CE distribution. It uses the
rsync tool to store backups in a remote server through SSH:

* The following attributes are supported to configure the backup
  datastore:

  - RSYNC_HOST
  - RSYNC_USER
  - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default)

* downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover
  single files from an existing backup. (RSYNC_HOST and RSYN_USER needs
  to be set in ds_id

EMULATOR_CPUS
=============

This commit includes a non related backup feature:

* Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the
  CPU IDs where the emulator threads will be pinned. If this value is
  not defined the allocated CPU wll be used when using a PIN policy.

(cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b)

F OpenNebula/one#5516: adding rsync backup driver

(cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305)

F OpenNebula/one#5516: update install.sh, add vmid to source, some polish

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d)

F OpenNebula/one#5516: cleanup

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340)

F OpenNebula/one#5516: update downloader, default args, size check

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305)

LL

(cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-10-07 22:01:37 +02:00
Ruben S. Montero
74bc751bd7
F #5940: Improve PCI Passthrough and SRIOV support
- SHORT_ADDRESS can be used to select specific devices  (useful on
  homogenous clusters or nic attach operations). This name has been
  selected because of:
    1. It is the attrbute shown host info
    2. It does not collide with the reserved ADDRESS attribute

- New test and add functions that considers both allocation methods: by
  name (VENDOR/CLASS/DEVICE) or address (SHORT_ADDRESS)

- Parameter check on VM creation

- revert and add method use the same pci_attribute function to add info
  to the VM PCI attribute

- Remove well-known attributes when parsing PCI devices (ADDRESS,
  PREV_ADDRES, BUS, FUNCTION, SLOT, NUMA_NODE, UUID)

- Support for attach and detach NIC with PCI attributes
    * onevm_exec.rb looks for PCI devices for ATTACH=YES when attaching/detaching an interface
    * script action are now written in Ruby
    * KVM module with common actions (hostdev/interface device str)
    * Minor changes in xmlparser and OpenNebulaVM classes

- PCI selection options to onevm nic-attach:
    * pci short_address
    * pci_device device ID
    * pci_vendor vendor ID
    * pci_class class ID

- VF can be configured by setting some parameters through IP link (e.g.
  MAC or VLAN_ID). This commit includes a mixin to activate_vf

    * one_vmm_exec.rb looks for PCI VN_MAD drivers
    * VM class (VNM) adds a @pcis array
    * activate_vf should be called in the pre stage. The following drivers
      acticate VFs (VLAN_ID is implemented as 802.1Q tag)
            - 802.1Q
            - bridge
            - fw
            - ovswitch
    * Supported attributes for VF:
      - MAC
      - VLAN_ID
      - spoof checking (SPOOFCHK)
      - trust (TRUST)

- Predictable PCI addresses for guests is only enabled if PCI bus 1
  is present (PCI bridge bus cannot be hotplugged)

- Improve integration with Libvirt/QEMU:

    * When attach, only activate the VF being attached
    * Attach: Use <interface> and not <hostdev> for VF. There seems to be a race
      condition between accessing the vfio device and permission setup.
    * Attach: Remove address on attach as it may fail beacuse PCI controller
      is not present, e.g.:

        ATTACHNIC: Could not attach NIC to 28534240: error: Failed to attach device
        from /dev/fd/63 error: XML error: Invalid PCI address 0000:01:01.0.
        Only PCI buses up to 0 are available ExitCode: 1

    * Detach: Detach always use <hostdev> as libvirt fails to identified the
      device just by address when using <interface>

- Suntone Support: The "Hardware" profile of a network interface (NIC) of a VM can be of three types, now exposed in Sunstone:

    * "Emulated" it includes the hardware model emulated by Qemu
    * "PCI - Automatic" oned hw scheduler will pick the best PCI device for the NIC
    * "PCI - Manual" user can specify the PCI device by its short-address as shown in host information

This commits also enables the attach/detach operations on PCI based NICs in Suntone. *Note*: only for KVM VMs

(reverts commit aafd5f3014)
(reverts commit 671ef0b2d91aba00b1e6c63e19c4a50c3af20f6f)
2022-10-04 20:53:08 +02:00
Ruben S. Montero
7a1a85edb6
B #5867: Fix quotas and fsck
co-author: Pavel Czerny <pczerny@opennebula.systems>
(cherry picked from commit e2d4141599)
2022-06-20 18:35:40 +02:00
Tino Vazquez
f7d53e75ff M #-: Bump version 6.3.85 2022-04-07 19:49:58 +02:00
Pavel Czerný
5591f88a3b
B #5524: Apply SYSTEM_DISK_USAGE for VM snapshots (#1647)
Co-authored-by: Ruben S. Montero <rsmontero@opennebula.org>
2022-01-04 13:03:47 +01:00
Pavel Czerný
b80ee243c0
F #5482: Append flag for onevm updateconf (#1395) 2021-08-09 11:00:58 +02:00
Ruben S. Montero
0f130a1abd
F #5289: Remove unneeded code 2021-06-15 11:24:19 +02:00
Ruben S. Montero
3f171f3b5b
F #5289: New api for sched actions
SchedActions are now a complete managed type. It is in in /TEMPLATE and
not /USER_TEMPLATE and includes dedicated API Calls to manage the
entries:

- one.vm.schedadd (add new sched_action)
- one.vm.schedupdate (update an exiting sched_action)
- one.vm.scheddelete (delete an action)

The scheduler now use this API. This will prevent race conditions
between update operations from different processes.

co-authored-by: Pavel Czerný <pczerny@opennebula.systems>
2021-06-15 08:22:26 +00:00
Ruben S. Montero
0ebda56330
M #-: Reduce attribute copies on updateconf
(cherry picked from commit 99d165856d)
2021-06-10 09:56:02 +02:00
Pavel Czerný
6bca41f0db
M #-: Showback option to calculate only running VM (#1105)
(cherry picked from commit c5ad2698b7)
2021-04-15 11:13:56 +02:00
Pavel Czerný
872939fb26
F #727: Improve shareable disks (#970)
* Shareable only on supported hypervisors
* Fix disk-attach
* Restrict shareable disks to SHARED datastores
* On image clone: copy SHAREABLE depends on DEFAULT_IMAGE_PERSISTENT flag
2021-03-15 16:24:25 +01:00
Ruben S. Montero
59b6e59fdd
M #-: Add comments 2021-03-15 14:23:48 +01:00
Alejandro Huertas
f92c6bbfab
B #5096: fix bug updating VM conf 2021-03-15 14:23:48 +01:00
Ruben S. Montero
cb52559200
WIP 2021-03-15 14:23:48 +01:00
Vlastimil Holer
096754b63f
M #-: Bump year to 2021 (#778) 2021-02-09 16:07:56 +01:00
Pavel Czerný
d6d299e612
F #1112: Allow disk-saveas in undeployed, stopped (#591) 2021-01-08 10:50:12 +01:00
Pavel Czerný
7ba1bbe633
F #1660: Hotplug VCPU and mem for KVM (#392)
Co-authored-by: Ruben S. Montero <rsmontero@opennebula.org>
2020-11-17 11:24:52 +01:00
Pavel Czerný
2808d10373
M #-: Fix VM cloning image disk format (#410) 2020-11-11 15:37:01 +01:00
Ruben S. Montero
26498959bc
B #5131: Track VNC allocation across migration
(cherry picked from commit dd6a7f4cd6)
2020-10-16 12:45:24 +02:00
Pavel Czerný
3ea67d3645
F #4936: Reduce number of new/delete, use unique_ptr (#212)
* On exit stop RequestManager first

* Use unique_ptr for pointers to Template

* Reduce number of new/delete
2020-09-15 11:16:00 +02:00