1
0
mirror of https://github.com/OpenNebula/one.git synced 2024-12-22 13:33:52 +03:00
Commit Graph

248 Commits

Author SHA1 Message Date
Daniel Clavijo Coca
886a106fb7
M OpenNebula/one#6504: Update vm pool view schema (#2990) 2024-03-20 12:21:12 +01:00
Daniel Clavijo Coca
df0b41dec1
M #-: Update vm state reference (#2929) 2024-02-09 11:05:41 +01:00
Pavel Czerný
8d564f5d46
M #-: Add ERROR attribute to backupjob xml schema (#2869)
(cherry picked from commit 6e2c4c623f)
2023-12-12 16:28:32 +01:00
Pavel Czerný
1b3f3d1462
F #6333: Fix xml schema (#2811) 2023-11-14 12:08:43 +01:00
Ruben S. Montero
ea1d5183ea
F #6333: Snapshots for incremental backup mode
- Defines a new a attribute "INCREMENT_MODE" that can be "SNAPSHOT" or
     "CBT"

- SNAPSHOT mode: takes a disk snapshot (virsh snapshot-create-as or
  qemu-img). The current snapshot is saved in the backup store, and
  committed to the base disk image.

  The backup_qcow2.rb util adapt to different VM folder layout for SSH,
  shared, persistent, non persistent disks.

- Lint vxlan.rb

- Update onevm and XSD schema to reflect the new backup attribute
2023-11-10 19:40:26 +01:00
Pavel Czerný
2ada2ec687
M #-: Remove obsolete attribute from oned.conf (#2808) 2023-11-10 11:52:15 +01:00
Neal Hansen
c1a0e5bfc5
F OpenNebula/one#6286: Add support for VIDEO devices in KVM
- New attribute VIDEO, that includes
  - TYPE: Custom video device (none, vga, cirrus, virtio)
 - IOMMU: Enable IOMMU for the video device
 - ATS: Enable ATS (Address Translation Services) for the video device
 - VRAM: VRAM allocated to the video device (Megabytes)
 - RESOLUTION: Video resolution, in format like: 1280x720 or 1920x1080

- Support for onevm CLI command
- Updated VM xsd file.
2023-09-20 20:00:10 +02:00
Pavel Czerný
ac49ab6d30
F #6227: New API for VMGroup roles
* New API for VMGroup roles: add, del, update
* Go and Java api for VM Group roles
2023-09-14 15:36:26 +02:00
Pavel Czerny
3893aa39d1 M #-: Update Backup Job xml schema 2023-08-08 09:42:05 +02:00
Pavel Czerny
9d463a72f1 M #-: Add DISK_TYPE to xml-schema 2023-08-08 09:42:05 +02:00
Ruben S. Montero
6bbfbb03e4
F #6063: Backup Jobs for OpenNebula
Includes the following changes:
- xml-schema for Backup Job and Scheduled Actions
- GO, Java api
- Deprecate onevm update-chart, delete-chart
    * The commands are replaced by sched-update and sched-delete
    * Refactor method deprecate_command, it's still possible to run the
    command
    * Delete 'shutdown' and 'delete' commands deprecated years ago
    * Fix --verbose option for sched-update and sched-delete
- Re-implementation of scheduled actions, now are managed and executed
  by oned
- Backup Job objects, API, and CLI commands
2023-07-03 18:15:52 +02:00
Pavel Czerný
bf36ccf30d
F #6111: Add VCPU to VM short xml (#2655)
(cherry picked from commit 78c584661c)
2023-06-29 09:59:47 +02:00
Ruben S. Montero
0784385caf
B #6027: Fix schema for host.xsd 2023-01-09 18:52:40 +01:00
Pavel Czerný
0696693531
B #6027: Parse host NUMA monitoring (#2440)
Co-authored-by: Ruben S. Montero <rsmontero@opennebula.org>
2023-01-09 18:29:09 +01:00
Jan Orel
a7fd43d9f5
B #5676: Create LDAP user without password (#2399)
* Create LDAP user without password
* Update oneuser CLI tool
* Update opennebula_configuration.xsd

Co-authored-by: Pavel Czerný <pczerny@opennebula.io>
2022-12-07 10:03:15 +01:00
Ruben S. Montero
413665213b
F #5753: Two options for live memory resize
This commit add a new attribute to the VM template MEMORY_RESIZE_MODE.
It can be:

- BALLOONING to increase/decrease the memory balloon
- HOTPLUG to add/remove memory modules to the VM

By default VMs will use BALLOONNING if not specified.
2022-11-28 10:29:01 +01:00
Pavel Czerný
d8d190c5ec
M #-: Fix xml-schema for backups (#2363) 2022-11-23 09:58:40 +01:00
Ruben S. Montero
865908de1c
F #5989: Live update of Virtual Network attributes
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>
co-authored-by: Frederick Borges <fborges@opennebula.io>
co-authored-by: Christian González <cgonzalez@opennebula.io>

* VNET updates trigger a driver action on running VMs with NICs in the
  network.

* VNET includes a sets with VM status: updated, outdated, error and
  updating. With VMs in each state.

* VNET flags error situations with a new state UPDATE_FAILURE.

* The same procedure is applied when an AR is updated (only VMs in that
  AR are updated).

* A new options in the one.vn.recover API call enable to recover or
  retry this VM update operations.

* The following attributes can be live-updated per VNET driver:
    - PHYDEV (novlan, vlan, ovs driver)
    - MTU (vlan, ovs driver)
    - VLAN_ID (vlan, ovs driver)
    - QINQ_TYPE (ovs driver)
    - CVLANS (ovs driver)
    - VLAN_TAGGED_ID (ovs driver)
    - OUTER_VLAN_ID (ovs driver)
    - INBOUND_AVG_BW (SG, ovs driver + KVM)
    - INBOUND_PEAK_BW (SG, ovs driver + KVM)
    - INBOUND_PEAK_KB (SG, ovs driver + KVM)
    - OUTBOUND_AVG_BW (SG, ovs driver + KVM)
    - OUTBOUND_PEAK_BW (SG, ovs driver + KVM)
    - OUTBOUND_PEAK_KB (SG, ovs driver + KVM)

* New API call one.vm.updatenic, allows to update individual NICs
  without the need of detach/attach (only QoS supported).

* Update operations for: 802.1Q, bridge, fw, ovswitch, ovswitch_vxlan
  and vxlan network drivers.

* VNET attributes (old values) stored in VNET_UPDATE to allow
  implementation of update operations. The attribute is removed after a
  successful update.

* Updates to CLI onevnet (--retry option) / onevm (nicupdate command)

* XSD files updated to reflect the new data model

* Ruby and JAVA bindings updated: new VNET state and recover option, new
  VM API call.

* Suntone and Fireedge implementation (lease status, recover option, new
  states)

TODO: Virtual Functions does not support this functionality

iii
2022-11-16 15:06:49 +01:00
Pavel Czerný
ceab1ed159
M #-: Fix xml-schema for backups (#2326) 2022-10-27 12:36:23 +02:00
Ruben S. Montero
e433ccb85b
F #5516: New backup interface for OpenNebula
co-authored-by: Frederick Borges <fborges@opennebula.io>
co-authored-by: Neal Hansen <nhansen@opennebula.io>
co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io>
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>

BACKUP INTERFACE
=================

* Backups are exposed through a a special Datastore (BACKUP_DS) and
  Image (BACKUP) types. These new types can only be used for backup'ing
  up VMs. This approach allows to:

  - Implement tier based backup policies (backups made on different
    locations).

  - Leverage access control and quota systems

  - Support differnt storage and backup technologies

* Backup interface for the VMs:

  - VM configures backups with BACKUP_CONFIG. This attribute can be set
    in the VM template or updated with updateconf API call. It can include:

    + BACKUP_VOLATILE: To backup or not volatile disks

    + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent,
      suspend or none). When possible backups are crash consistent.

    + KEEP_LAST: keep only a given number of backups.

  - Backups are initiated by the one.vm.backup API call that requires
    the target Datastore to perform the backup (one-shot). This is
    exposed by the onevm backup command.

  - Backups can be periodic through scheduled actions.

  - Backup configuration is updated with one.vm.updateconf API call.

* Restore interface:

  - Restores are initiated by the one.image.restore API call. This is
    exposed by oneimage restore command.

  - Restore include configurable options for the VM template

    + NO_IP: to not preserve IP addresses (but keep the NICs and network
      mapping)

    + NO_NIC: to not preserve network mappings

  - Other template attributes:

    + Clean PCI devices, including network configuration in case of TYPE=NIC
    attributes. By default it removes SHORT_ADDRESS and leave the "auto"
    selection attributes.

    + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node

  - It is possible to restore single files stored in the repository by
    using the backup specific URL.

* Sunstone (Ruby version) has been updated to expose this feautres.

BACKUP DRIVERS & IMPLEMENTATION
===============================

* Backup operation is implemented by a combination of 3 driver operations:

  - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate
    backups for RUNNING VMs.

  - TM. This commit introduces 2 new operations (and their
    corresponding _live variants):

    + pre_backup(_live): Prepares the disks to be back'ed up in the
      repository. It is specific to the driver: (i) ceph uses the export
      operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as
      needed.
    + post_backup(_live): Performs cleanning operations, i.e. KVM
      snapshots or tmp dirs.

  - DATASTORE. Each backup technology is represented by its
    corresponfing driver, that needs to implement:

    + backup: it takes the VM disks in file (qcow2) format and stores it
      the backup repository.

    + restore: it takes a backup image and restores the associated disks
      and VM template.

    + monitor: to gather available space in the repository

    + rm: to remove existing backups

    + stat: to return the "restored" size of a disk stored in a backup

    + downloader pseudo-URL handler: in the form
      <backup_proto>://<driver_snapshot_id>/<disk filename>

BACKUP MANAGEMENT
=================

Backup actions may potentially take some time, leaving some vmm_exec threads in
use for a long time, stucking other vmm operations. Backups are planned
by the scheduler through the sched action interface.

Two attributes has been added to sched.conf:
  * MAX_BACKUPS max active backup operations in the cloud. No more
    backups will be started beyond this limit.

  * MAX_BACKUPS_HOST max number of backups per host

* Fix onevm CLI to properly show and manage schedule actions. --schedule
  supports now, as well as relative times +<seconds_from_stime>

  onvm backup --schedule now -d 100 63

* Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs
  to use the batch interface or request specific permissions

Internal restructure of Scheduler:

- All sched_actions interface is now in SchedActionsXML class and files.
  This class uses references to VM XML, and MUST be used in the same
  lifetime scope.

- XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as
  static functions.

- VirtualMachineActionPool includes counters for active backups (total
  and per host).

SUPPORTED PLATFORMS
====================
* hypervisor: KVM
* TM: qcow2/shared/ssh, ceph
* backup: restic, rsync

Notes on Ceph

* Ceph backups are performed in the following steps:
    1. A snapshot of each disk is taken (group snapshots cannot be used as
       it seems we cannot export the disks afterwards)
    2. Disks are export to a file
    3. File is converted to qcow2 format
    4. Disk files are upload to the backup repo

TODO:
  * Confirm crash consistent snapshots cannot be used in Ceph

TODO:
  * Check if using VM dir instead of full path is better to accomodate
    DS migrations i.e.:
    - Current path: /var/lib/one/datastores/100/53/backup/disk.0
    - Proposal: 53/backup/disk.0

RESTIC DRIVER
=============
Developed together with this feature is part of the EE edtion.

* It supports the SFTP protocol, the following attributes are
  supported:

  - RESTIC_SFTP_SERVER
  - RESTIC_SFTP_USER: only if different from oneadmin
  - RESTIC_PASSWORD
  - RESTIC_IONICE: Run restic under a given ionice priority (class 2)
  - RESTIC_NICE: Run restic under a given nice
  - RESTIC_BWLIMIT: Limit restic upload/download BW
  - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes:
    off, auto, max). This requires repositories version 2. By default,
    auto is used (average compression without to much CPU usage)
  - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a
    backend (5 by default). For high-latency backends this number can be
    increased.

* downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name>
  snapshot_id is the restic snapshot hash. To recover single disk images
  from a backup. This URLs support:

  - RESTIC_CONNECTIONS
  - RESTIC_BWLIMIT
  - RESTIC_IONICE
  - RESTIC_NICE

  These options needs to be defined in the associated datastore.

RSYNC DRIVER
=============
A rsync driver is included as part of the CE distribution. It uses the
rsync tool to store backups in a remote server through SSH:

* The following attributes are supported to configure the backup
  datastore:

  - RSYNC_HOST
  - RSYNC_USER
  - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default)

* downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover
  single files from an existing backup. (RSYNC_HOST and RSYN_USER needs
  to be set in ds_id

EMULATOR_CPUS
=============

This commit includes a non related backup feature:

* Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the
  CPU IDs where the emulator threads will be pinned. If this value is
  not defined the allocated CPU wll be used when using a PIN policy.

(cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b)

F OpenNebula/one#5516: adding rsync backup driver

(cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305)

F OpenNebula/one#5516: update install.sh, add vmid to source, some polish

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d)

F OpenNebula/one#5516: cleanup

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340)

F OpenNebula/one#5516: update downloader, default args, size check

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305)

LL

(cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-10-07 22:01:37 +02:00
Pavel Czerný
fecfd03a1a
M #-: Restricted dirs for CONTEXT/FILES (#2243)
* M #-: Restricted dirs for CONTEXT/FILES

* M #-: Fix opennebula_configuration.xsd
2022-07-26 09:18:08 +02:00
Pavel Czerný
37b01901fd
M #-: Fix host.xsd (#2183)
(cherry picked from commit 70dc2b87a1)
2022-06-27 10:24:27 +02:00
Alejandro Huertas Herrero
69a7a8df2d
F #5885: add missing XSD (#2170) 2022-06-21 17:31:33 +02:00
Sergio Betanzos
ef51e3876f
M #~: Add image and vnet to hook state xsd (#2085) 2022-05-25 18:58:36 +02:00
Ruben S. Montero
4b746348b3
f #5725: State for Virtual Networks
* VNETs include state to flag their configuration status
  (READY/ERROR/LOCK)
* Hooks can be program based on VNET state transitions
* Include Java/Go/Ruby bindings
* New vCenter network drivers
* Place-holder to customize actions (create/delete) for other netowrk drivers
* Sunstone integration
* OneFlow syncs flow creation on VNET status
* Remove deprecated vcenter hooks and autoregistration

co-authored-by: Frederick Ernesto Borges Noronha <fborges@opennebula.io>
co-authored-by: Alejandro Huertas <ahuertas@opennebula.io>
co-authored-by: Pavel Czerny <pczerny@opennebula.systems>
2022-04-09 15:20:17 +02:00
Pavel Czerný
1010ae903c
M #-: Fix opennebula_configuration.xsd (#1847) 2022-03-21 09:02:27 +01:00
Ruben S. Montero
289846f67f
M #-: Add host drivers to VM xml (short)
- This will help to identify VM class (e.g. KVM vs VMware) in list views
- Update xsd schema for vm pool (VM_MAD/TM_MAD)
2022-03-16 18:42:23 +01:00
Pavel Czerný
d787313ea1
F #5753: Better live mem resize for KVM (#1820)
(cherry picked from commit c6965c3cfe)
2022-03-01 10:50:25 +01:00
Tino Vazquez
854d586e8f
M #-: refresh doc link to API 2022-01-18 17:32:19 +01:00
Pavel Czerný
8206ace347
F #5661: Update opennebula_configuration.xsd (#1688) 2022-01-10 15:18:05 +01:00
Pavel Czerný
5591f88a3b
B #5524: Apply SYSTEM_DISK_USAGE for VM snapshots (#1647)
Co-authored-by: Ruben S. Montero <rsmontero@opennebula.org>
2022-01-04 13:03:47 +01:00
Pavel Czerný
b98fdfe3d3
F #5431: Update opennebula conf schema (#1666) 2021-12-13 17:53:23 +01:00
Pavel Czerný
38d06a92aa
F #5561: Showback computes running hours (#1527)
(cherry picked from commit af9ea24ec4)
2021-10-15 17:41:02 +02:00
Vlastimil Holer
190b0c6e4f
F OpenNebula/addon-context-linux#86: Update NIC parameters (#1447)
- Support METHOD and IP6_METHOD.
- Update METRIC6 to IP6_METRIC6.
- Deprecate CONTEXT_FORCE_IPV4 (unused for some time)
2021-09-10 12:19:38 +02:00
Jan Orel
decc8c9964
B #5466: Add PCI devices desc to host.xsd (#1371) 2021-07-23 11:16:26 +02:00
Sergio Betanzos
78bb1135d8
M #~: Minor fix in xsd schema files (#1359) 2021-07-13 17:57:16 +02:00
Pavel Czerný
80b21445b9
M #-: Fix vm/monitoring schema (#1346) 2021-07-02 09:50:30 +02:00
Jan Orel
ae83314e63
B #-: Fix ZONE occurs in ZONE_POOL xsd (#1329) 2021-06-24 10:14:27 +02:00
Jan Orel
5c72738bd7
M #-: Add DISK_SIZE to VM/MONITORING xsd (#1319) 2021-06-21 09:41:26 +02:00
Pavel Czerný
8dbe2c3da8
M #-: More details to vm.xsd and vm_pool.xsd (#1316) 2021-06-18 13:24:14 +02:00
Pavel Czerný
61d43e8e50
F #4510: Fix markatplace xml schema (#1271) 2021-06-04 10:11:16 +02:00
Ruben S. Montero
f1a48bdc0d
F #5270, #5272: Support for Google Compute and DigitalOcean
co-authored-by: Alejandro Huertas <ahuertas@opennebula.io>
co-authored-by: Juan Antonio <jescobar@opennebula.io>
co-authored-by: Sergio Betanzos <sbetanzos@opennebula.io>
2021-05-01 21:53:09 +02:00
Pavel Czerný
abe30095c1
M #-: Fix oned config schema (#1153) 2021-04-29 17:56:37 +02:00
Pavel Czerny
69e83eb161
F #5353: Zone standby mode. DB version number to 6.2.0
Includes the following commits from f-5353:

  - 62d3d38d7fb135b82874a5076a7f06c23675e0b7
  - 52f3f066603c478061891175c008a58cfed3dc43
  - 0e82f9e5471c3cf0e21969393299279c6e739138
  - 8c32050940f528ea876e8293e6fbbf56a373bd54
2021-04-27 10:45:58 +02:00
Jan Orel
678fc67f75
B #5305: Add MONITORING_DATA to xsd (#1010) 2021-03-22 18:09:06 +01:00
Alejandro Huertas Herrero
c31316ab94
M #-: add missing XSD IMAGE_ENCRYPTED (#886) 2021-02-26 10:55:33 +01:00
Alejandro Huertas Herrero
532c4b238d
M #-: add missing PREV_STATE to image.xsd (#882) 2021-02-25 12:26:37 +01:00
Ruben S. Montero
3ce190b989
M #-: Fix opennebula_configuration xsd 2021-01-18 17:08:33 +01:00
Ruben S. Montero
ce7aac40a2
M #-: Update opennebula_configuration.xsd (oned.conf) 2021-01-12 17:00:50 +00:00
Jan Orel
d53f0a3d90
M #4089: Update xsd files (#452) 2020-11-20 10:56:39 +01:00