1
0
mirror of https://github.com/OpenNebula/one.git synced 2024-12-22 13:33:52 +03:00
Commit Graph

82 Commits

Author SHA1 Message Date
Ruben S. Montero
3dfd53df1c
F #6053: In-place restore of VM backups (shared and ssh drivers)
This commit implements the in-place restore of VM backups. Selected VM disks will
be replaced with the specified backup:

* A new API call has been added to the XML-RPC API (`one.vm.restore`) with
  the following arguments:
    - VM ID to be restored, needs to be in **poweroff** state
    - IMAGE ID of the backup to restore
    - INCREMENT ID, only for incremental backups, the increment to use
      (defults to -1 to use the last increment available)
    - DISK ID of the disk to restore (defaults to -1 to restore all VM
      disks)

* Datastore drivers needs to implemente a new operation `ls`. This new
  operation takes the VM, image information of the backup and datastore
  information and returns the restore URL for the disks in the backup.

* This commit includes the implementation for qcow2 and ssh drivers,
  ceph will be implemented in a separated PR. The new driver action is
  `restore host:vm_dir vm_id img_id inc_id disk_id`

* The restore operation is performed in a new state `PROLOG_RESTORE`
  rendered as `RESTORE` and `rest` in short form. State in in RSuntone.

TODO:
  - Remove any existing VM snapshot (system/disk) in the VM. Note that
    snapshots are not included in a backup.

  - Ceph drivers

  - JAVA, GO Lang API bindings

  - Sunstone interface, new state and new operation. Review new state in
    RSuntone.

co-authored-by: Pavel Czerny <pczerny@opennebula.io>
2024-04-26 12:45:04 +02:00
Ruben S. Montero
2e596c0cf5
F #6439: PCI attach/detach operation
* Adds 2 new API calls one.vm.attachpci and one.vm.detachpci
* The operation is only for POWEROFF VMs
* Adds 2 new actions to register the event in the VM history pci-detach/pci-attach
* Ruby OCA bindings
* onevm cli commands

TODO:
- JAVA and GO bindings
- Sunstone implementation

(cherry picked from commit 5c4331a4dd)
2024-01-31 17:50:45 +01:00
Pavel Czerný
e2f2782195
B #6387: Fix accounting after onevm resize (#2866)
* Fix accounting after onevm resize
* Fix crash in VM resize if VM has no history

(cherry picked from commit ca934bbde0)
2024-01-03 13:02:29 +01:00
Ruben S. Montero
a9da75db0e
F #6030: Cancel backup operation
Author: Pavel Czerny <pczerny@opennebula.systems>
Author: Michal Opala <mopala@opennebula.io>
(cherry picked from commit f12c365d3e)
2023-03-16 13:11:27 +01:00
Tino Vázquez
3376afdaf0
M #-: Bump year 2023 2023-01-09 12:23:19 +01:00
Ruben S. Montero
865908de1c
F #5989: Live update of Virtual Network attributes
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>
co-authored-by: Frederick Borges <fborges@opennebula.io>
co-authored-by: Christian González <cgonzalez@opennebula.io>

* VNET updates trigger a driver action on running VMs with NICs in the
  network.

* VNET includes a sets with VM status: updated, outdated, error and
  updating. With VMs in each state.

* VNET flags error situations with a new state UPDATE_FAILURE.

* The same procedure is applied when an AR is updated (only VMs in that
  AR are updated).

* A new options in the one.vn.recover API call enable to recover or
  retry this VM update operations.

* The following attributes can be live-updated per VNET driver:
    - PHYDEV (novlan, vlan, ovs driver)
    - MTU (vlan, ovs driver)
    - VLAN_ID (vlan, ovs driver)
    - QINQ_TYPE (ovs driver)
    - CVLANS (ovs driver)
    - VLAN_TAGGED_ID (ovs driver)
    - OUTER_VLAN_ID (ovs driver)
    - INBOUND_AVG_BW (SG, ovs driver + KVM)
    - INBOUND_PEAK_BW (SG, ovs driver + KVM)
    - INBOUND_PEAK_KB (SG, ovs driver + KVM)
    - OUTBOUND_AVG_BW (SG, ovs driver + KVM)
    - OUTBOUND_PEAK_BW (SG, ovs driver + KVM)
    - OUTBOUND_PEAK_KB (SG, ovs driver + KVM)

* New API call one.vm.updatenic, allows to update individual NICs
  without the need of detach/attach (only QoS supported).

* Update operations for: 802.1Q, bridge, fw, ovswitch, ovswitch_vxlan
  and vxlan network drivers.

* VNET attributes (old values) stored in VNET_UPDATE to allow
  implementation of update operations. The attribute is removed after a
  successful update.

* Updates to CLI onevnet (--retry option) / onevm (nicupdate command)

* XSD files updated to reflect the new data model

* Ruby and JAVA bindings updated: new VNET state and recover option, new
  VM API call.

* Suntone and Fireedge implementation (lease status, recover option, new
  states)

TODO: Virtual Functions does not support this functionality

iii
2022-11-16 15:06:49 +01:00
Ruben S. Montero
0a46e21129
F #5516: Incremental backups for qcow2 disk images
- Adds new configuration attribute MODE: FULL or INCREMENTAL for
  BACKUP_CONFIG. FULL backups uses a differen backup image.

- INCREMENTAL backup information is together with the backup image.
  Example:

      <BACKUP_INCREMENTS>
        <INCREMENT>
          <DATE><![CDATA[1667770552]]></DATE>
          <ID><![CDATA[0]]></ID>
          <PARENT_ID><![CDATA[-1]]></PARENT_ID>
          <SIZE><![CDATA[172]]></SIZE>
          <SOURCE><![CDATA[bb828060]]></SOURCE>
          <TYPE><![CDATA[FULL]]></TYPE>
        </INCREMENT>
        <INCREMENT>
          <DATE><![CDATA[1667770604]]></DATE>
          <ID><![CDATA[1]]></ID>
          <PARENT_ID><![CDATA[0]]></PARENT_ID>
          <SIZE><![CDATA[1]]></SIZE>
          <SOURCE><![CDATA[ca0de5f6]]></SOURCE>
          <TYPE><![CDATA[INCREMENT]]></TYPE>
        </INCREMENT>
        <INCREMENT>
          <DATE><![CDATA[1667770700]]></DATE>
          <ID><![CDATA[2]]></ID>
          <PARENT_ID><![CDATA[1]]></PARENT_ID>
          <SIZE><![CDATA[39]]></SIZE>
          <SOURCE><![CDATA[e9897d6a]]></SOURCE>
          <TYPE><![CDATA[INCREMENT]]></TYPE>
        </INCREMENT>
      </BACKUP_INCREMENTS>

    This information only appears on incremental backups

- Internal BACKUP_CONFIG data includes information about the current
  active backup and the last increment id.

- Backup operation includes a new parameter: reset. This "closes" the
  current active incremental chain and creates a new FULL backup.

- Backup drivers store backups with increment index (0 = FULL) e.g.
  disk.0.0.

- Incremental backups are only allowed for VMs using all disks in qcow2
  format.

- Backup configuration cannot be changed while doing a VM backup.

- Downloader strings includes backup chains <inc_id>:<backup_ref>,...

- Restic downloader  has been updated to support backup chains. Disk
  images are rebased across increments.
2022-11-06 22:54:36 +01:00
Ruben S. Montero
e433ccb85b
F #5516: New backup interface for OpenNebula
co-authored-by: Frederick Borges <fborges@opennebula.io>
co-authored-by: Neal Hansen <nhansen@opennebula.io>
co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io>
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>

BACKUP INTERFACE
=================

* Backups are exposed through a a special Datastore (BACKUP_DS) and
  Image (BACKUP) types. These new types can only be used for backup'ing
  up VMs. This approach allows to:

  - Implement tier based backup policies (backups made on different
    locations).

  - Leverage access control and quota systems

  - Support differnt storage and backup technologies

* Backup interface for the VMs:

  - VM configures backups with BACKUP_CONFIG. This attribute can be set
    in the VM template or updated with updateconf API call. It can include:

    + BACKUP_VOLATILE: To backup or not volatile disks

    + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent,
      suspend or none). When possible backups are crash consistent.

    + KEEP_LAST: keep only a given number of backups.

  - Backups are initiated by the one.vm.backup API call that requires
    the target Datastore to perform the backup (one-shot). This is
    exposed by the onevm backup command.

  - Backups can be periodic through scheduled actions.

  - Backup configuration is updated with one.vm.updateconf API call.

* Restore interface:

  - Restores are initiated by the one.image.restore API call. This is
    exposed by oneimage restore command.

  - Restore include configurable options for the VM template

    + NO_IP: to not preserve IP addresses (but keep the NICs and network
      mapping)

    + NO_NIC: to not preserve network mappings

  - Other template attributes:

    + Clean PCI devices, including network configuration in case of TYPE=NIC
    attributes. By default it removes SHORT_ADDRESS and leave the "auto"
    selection attributes.

    + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node

  - It is possible to restore single files stored in the repository by
    using the backup specific URL.

* Sunstone (Ruby version) has been updated to expose this feautres.

BACKUP DRIVERS & IMPLEMENTATION
===============================

* Backup operation is implemented by a combination of 3 driver operations:

  - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate
    backups for RUNNING VMs.

  - TM. This commit introduces 2 new operations (and their
    corresponding _live variants):

    + pre_backup(_live): Prepares the disks to be back'ed up in the
      repository. It is specific to the driver: (i) ceph uses the export
      operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as
      needed.
    + post_backup(_live): Performs cleanning operations, i.e. KVM
      snapshots or tmp dirs.

  - DATASTORE. Each backup technology is represented by its
    corresponfing driver, that needs to implement:

    + backup: it takes the VM disks in file (qcow2) format and stores it
      the backup repository.

    + restore: it takes a backup image and restores the associated disks
      and VM template.

    + monitor: to gather available space in the repository

    + rm: to remove existing backups

    + stat: to return the "restored" size of a disk stored in a backup

    + downloader pseudo-URL handler: in the form
      <backup_proto>://<driver_snapshot_id>/<disk filename>

BACKUP MANAGEMENT
=================

Backup actions may potentially take some time, leaving some vmm_exec threads in
use for a long time, stucking other vmm operations. Backups are planned
by the scheduler through the sched action interface.

Two attributes has been added to sched.conf:
  * MAX_BACKUPS max active backup operations in the cloud. No more
    backups will be started beyond this limit.

  * MAX_BACKUPS_HOST max number of backups per host

* Fix onevm CLI to properly show and manage schedule actions. --schedule
  supports now, as well as relative times +<seconds_from_stime>

  onvm backup --schedule now -d 100 63

* Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs
  to use the batch interface or request specific permissions

Internal restructure of Scheduler:

- All sched_actions interface is now in SchedActionsXML class and files.
  This class uses references to VM XML, and MUST be used in the same
  lifetime scope.

- XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as
  static functions.

- VirtualMachineActionPool includes counters for active backups (total
  and per host).

SUPPORTED PLATFORMS
====================
* hypervisor: KVM
* TM: qcow2/shared/ssh, ceph
* backup: restic, rsync

Notes on Ceph

* Ceph backups are performed in the following steps:
    1. A snapshot of each disk is taken (group snapshots cannot be used as
       it seems we cannot export the disks afterwards)
    2. Disks are export to a file
    3. File is converted to qcow2 format
    4. Disk files are upload to the backup repo

TODO:
  * Confirm crash consistent snapshots cannot be used in Ceph

TODO:
  * Check if using VM dir instead of full path is better to accomodate
    DS migrations i.e.:
    - Current path: /var/lib/one/datastores/100/53/backup/disk.0
    - Proposal: 53/backup/disk.0

RESTIC DRIVER
=============
Developed together with this feature is part of the EE edtion.

* It supports the SFTP protocol, the following attributes are
  supported:

  - RESTIC_SFTP_SERVER
  - RESTIC_SFTP_USER: only if different from oneadmin
  - RESTIC_PASSWORD
  - RESTIC_IONICE: Run restic under a given ionice priority (class 2)
  - RESTIC_NICE: Run restic under a given nice
  - RESTIC_BWLIMIT: Limit restic upload/download BW
  - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes:
    off, auto, max). This requires repositories version 2. By default,
    auto is used (average compression without to much CPU usage)
  - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a
    backend (5 by default). For high-latency backends this number can be
    increased.

* downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name>
  snapshot_id is the restic snapshot hash. To recover single disk images
  from a backup. This URLs support:

  - RESTIC_CONNECTIONS
  - RESTIC_BWLIMIT
  - RESTIC_IONICE
  - RESTIC_NICE

  These options needs to be defined in the associated datastore.

RSYNC DRIVER
=============
A rsync driver is included as part of the CE distribution. It uses the
rsync tool to store backups in a remote server through SSH:

* The following attributes are supported to configure the backup
  datastore:

  - RSYNC_HOST
  - RSYNC_USER
  - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default)

* downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover
  single files from an existing backup. (RSYNC_HOST and RSYN_USER needs
  to be set in ds_id

EMULATOR_CPUS
=============

This commit includes a non related backup feature:

* Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the
  CPU IDs where the emulator threads will be pinned. If this value is
  not defined the allocated CPU wll be used when using a PIN policy.

(cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b)

F OpenNebula/one#5516: adding rsync backup driver

(cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305)

F OpenNebula/one#5516: update install.sh, add vmid to source, some polish

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d)

F OpenNebula/one#5516: cleanup

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340)

F OpenNebula/one#5516: update downloader, default args, size check

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305)

LL

(cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-10-07 22:01:37 +02:00
Tino Vazquez
f7d53e75ff M #-: Bump version 6.3.85 2022-04-07 19:49:58 +02:00
Ruben S. Montero
265d05c118
F #5612: (De)/Attach security groups from VM NICs
Author: Pavel Czerný <pczerny@opennebula.systems>
2022-01-25 20:27:05 +01:00
Vlastimil Holer
096754b63f
M #-: Bump year to 2021 (#778) 2021-02-09 16:07:56 +01:00
Ruben S. Montero
5ce49ed404
F #4936: Refactor pools to use unique_ptr
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>
2020-09-10 09:08:29 +02:00
Ruben S. Montero
764b8a8afd
M #-: Fix merge 2020-08-07 09:54:41 +02:00
Ruben S. Montero
22ff6b50f2
B #5006: Increment running quotas through monitor
(cherry picked from commit aa9be35504)
2020-08-07 00:30:06 +02:00
Ruben S. Montero
325db91bcb
F #4936:Refactor ActionManager and Timers
co-authored-by: Pavel Czerny <pczerny@opennebula.systems>
2020-07-24 16:00:59 +02:00
Ruben S. Montero
565094961a
F #4936: oned refactor
* Remove duplicit methods from NebulaUtil

    * Const correctness

    * Fix minor code quality issues

    * Remove ObjectCollection::clone()

    * Minor SSLUtil interface change

co-authored-by: Pavel Czerný <pczerny@opennebula.systems>
2020-07-05 22:01:32 +02:00
Pavel Czerný
daaf132a43
F #4936: Remove 'using namespace std' from headers (#60) 2020-07-02 22:42:10 +02:00
Vlastimil Holer
f3c50a5d89
M #-: Year bump to 2020 (#4634) 2020-04-30 15:00:02 +02:00
Ruben S. Montero
a6481bb038
F #1764: updateconf for running VM
L #-: Use nullptr, cpplint rules

co-authored-by: Pavel Czerny <pczerny@opennebula.systems>
2019-07-26 13:45:26 +02:00
Vlastimil Holer
441cf1f7f9 Bump version to 5.7.85, year to 2019 2019-01-16 11:47:59 +01:00
Ruben S. Montero
b89642aaab F #2138: Add options to migrate through a power-off life-cycle
Co-authored-by: Christian González <cgonzalez@opennebula.systems>
2018-12-04 12:00:13 +01:00
Ruben S. Montero
1dd6cab6af F #1246 Add recover --delete-db option
co-authored-by: Christian González <cgonzalez@opennebula.systems>
2018-10-15 15:38:54 +02:00
Ruben S. Montero
1235866e51 F #2228: Added new VM quota - RUNNING_MEMORY, RUNNING_CPU and RUNNING_VMS.
These running quotas are for all states included in ACTIVE and VMS being
scheduled (PENDING and HOLD) as those states consume the same resources (in
terms of allocation) as running VMs.

Author: juanmont <juanmont@ucm.es>
2018-09-05 15:14:39 +02:00
Ruben S. Montero
37222d65dc F #1792: Greatly improve concurrency by: Remove big lock on pool control class. Increase number of MySQL connections (new configuration variable). Make all parser reentrants. This commit also simplifies the pool references.
(cherry picked from commit a3d3ceb771034794c4fb453dcd9aaff64c53baed)
2018-03-17 23:35:21 +01:00
Vlastimil Holer
a4c0447ccf Bump year to 2018 (#1623) 2018-01-02 18:27:37 +01:00
juanmont
beae0776c3 B #5405: unified functionality for free vm resources in a single function (#498) 2017-09-26 18:30:13 +02:00
Javi Fontan
8de979e42c Bump version 5.3.80 2017-05-25 16:07:35 +02:00
Ruben S. Montero
04e4991d4d F #5005: Fix action queue. Update new action classes 2017-02-08 12:24:42 +01:00
Ruben S. Montero
d143012eb6 F #5005: Add request information to events and callbacks from API calls. 2017-02-07 17:26:23 +01:00
Ruben S. Montero
7a2face60c F #5005: New interface of ActionManager to accommodate additional arguments. Updates managers to the new interface 2017-02-03 20:39:34 +01:00
Ruben S. Montero
c1c3d0479d F #4393: More work on resize 2016-12-15 21:12:33 +01:00
Carlos Martín
ffdb41bc3e Feature #4597: Allow to create more VR VMs, and terminate existing ones
(cherry picked from commit 94070576d8)
2016-07-08 11:21:59 +02:00
Javi Fontan
e1f6dee180 Update copyright notice year 2016-05-04 12:33:23 +02:00
Ruben S. Montero
058e23c37a feature #3801, bug #3775: Delete operation is now under the recover
interface for admin. New terminate operation can be used in any "final"
state for end-users.
2016-05-02 18:34:42 +02:00
Jaime Melis
e20fb5c4c4 Fix copyright in banners 2015-09-23 16:51:10 +02:00
Carlos Martín
27e3ac289f Feature #3782: Rename disk snapshot tag to name 2015-07-09 13:13:07 +02:00
Javi Fontan
4b08d76fbf Whitespace cleanup 2015-07-01 15:18:28 -04:00
Ruben S. Montero
83ac6188de feature #3782: New state to delete snapshots. Delete action OCA (Ruby),
core and CLI
2015-05-27 16:29:41 +02:00
Ruben S. Montero
98f76cf3fe feature #3782: Create and Revert operations for Disk Snapshots. Core and
OCA levels
2015-05-19 18:41:23 +02:00
Ruben S. Montero
1c828b28d4 Feature #3654: Refactor DM to init pointers to managers only once 2015-04-26 23:34:26 +02:00
Ruben S. Montero
41782d01e5 feature #3654: Retry PROLOGs, EPILOGs, SHUTDOWNs and BOOTs through recover. Remove restart action from oned, removed shutdown retries through shutdown
feature #3214: Redo saveas check.

Revert "Feature #3654: Allow to run a shutdown again from shutdown state". This reverts 36afa9c864

Revert "Feature #3654: Allow to run a poweroff/undeploy again from shutdown_poweroff/undeploy". This reverts 9be8b151bc
2015-04-26 22:54:15 +02:00
Carlos Martín
55dd9304bc Feature #3654: Merge Dispatch Manager reset with reboot 2015-04-24 15:45:54 +02:00
Carlos Martín
ae7fab2ff8 Feature #3654: Merge the cancel and shutdown states 2015-04-24 15:25:47 +02:00
Carlos Martín
402400053f Feature #3654 #1610: Improve reason when an action is not available
(cherry picked from commit 24fa8d779c745cf6d80a75a0aaed28875209bd08)
2015-04-23 15:08:59 +02:00
Ruben S. Montero
44ab9678a3 feature #3564: Remove FAILED and FAILURE states 2015-04-21 14:00:52 +02:00
Jaime Melis
8d00b74177 Change year to 2015 2015-02-24 12:27:59 +01:00
Ruben S. Montero
093f95524e feature #3383: Clean RequestManager code 2015-02-05 22:49:33 +01:00
Jaime Melis
11520021f7 Bump to version 4.5.0 2014-01-09 11:51:20 +01:00
Ruben S. Montero
72e68b0703 Merge branch 'feature-1835' into one-4.0
Conflicts:
	src/lcm/LifeCycleStates.cc
	src/vmm/VirtualMachineManager.cc
2013-04-03 17:11:16 +02:00
Carlos Martín
728d043574 Feature #1839: Rename shutdown-save to undeploy 2013-04-02 17:01:22 +02:00