1
0
mirror of https://github.com/OpenNebula/one.git synced 2025-01-03 01:17:41 +03:00
Commit Graph

2199 Commits

Author SHA1 Message Date
Ruben S. Montero
dca50b2bb9
F #6275: External scheduler API
The connection to an external scheduler module is configured in sched.conf:

EXTERNAL_SCHEDULER = [
   SERVER  = "http://localhost:4567",
   PROXY   = "",
   TIMEOUT = 10
]

The API post on '/' the list of VMs, their pre-selected list of
candidate hosts based on REQUIREMENTS along with the VM information
(CAPACITY, TEMPLATE and USER_TEMPLATE).

Example:
{
  "VMS": [
    {
      "CAPACITY": {
        "CPU": 1.5,
        "DISK_SIZE": 1024,
        "MEMORY": 131072
      },
      "HOST_IDS": [
        3,
        4,
        5
      ],
      "ID": 32,
      "STATE": "PENDING",
      "TEMPLATE": {
        "AUTOMATIC_DS_REQUIREMENTS": "(\"CLUSTERS/ID\" @> 0)",
        "AUTOMATIC_NIC_REQUIREMENTS": "(\"CLUSTERS/ID\" @> 0)",
        "AUTOMATIC_REQUIREMENTS": "(CLUSTER_ID = 0) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED)",
        "CPU": "1.5",
        "MEMORY": "128",
        ...
      },
      "USER_TEMPLATE": {}
    },
    {
      "CAPACITY": {
        "CPU": 1.5,
        "DISK_SIZE": 1024,
        "MEMORY": 131072
      },
      "HOST_IDS": [
        3,
        4,
        5
      ],
      "ID": 33,
      "STATE": "PENDING",
      "TEMPLATE": {
       ...
      },
      "USER_TEMPLATE": {}
    }
  ]
}

The scheduler needs to respond to this post action with a simple list of
the allocation for each VM:

{
  "VMS": [
    {
      "ID": 32,
      "HOST_ID": 2
    },
    {
      "ID": 33,
      "HOST_ID": 0
    }
  ]
}

This commits vendorize Vendorize nlohmann-json (MIT license)
2023-08-29 17:11:01 +02:00
Ruben S. Montero
1e42b638a9
F #6063: Improve Backup Job Operations
* Schedule Action Manager executes only on RAFT leader in HA configurations.
* Relative time for Scheduled Action Backup Jobs returns error.
* Store Backup Job ID to VM and associated backup Image.
* VM can be only in one Backup Job at a time.
* Deny single Backup if the VM is already part of BackupJob to prevent
  inconsistent backup states.

Other fixes:
    - Fix crash on scheduled backup for VM in pending state
    - Fix error messages for Scheduled Actions
    - Remove SCHED_ACTION from Backup Job template on update

co-authored-by: Pavel Czerný <pczerny@opennebula.io>
2023-08-07 11:49:18 +02:00
Ruben S. Montero
156cd04ba5
F OpenNebula/one#6265: Delete of snapshots in the middle of the chain
- Add suport for delete snapshots in the middle of the chain in qcow2
  (ALLOW_ORPHANS=NO). Example, snapshot 2 below:
                     0 <- 1 <- 2 <- 3

- It supports removing the last snapshot and active (blockpull) and
  first one (snapshot 0). The first snapshot cannot be deleted in
  persistent images because of the underlying link setting.

- Snapshots are deleted by blockcommit. For example to delete 2, 3 is
  committed into 2:
                    0 <- 1 <- 2: (2+3)
  A new file 3.current is created in the filesystem to look for the
  actual file name supporting snapshot 3.

- qcow2/shared operations snap_delete, snap_revert and cpds
  (disk-saveas) has been updated to react to the .current files

- The kvm/deploy action has been updated to resolve links in source file
  attributes. This is needed so the <backingStore> of the libvirt domain
  does not contain the sysmlinks that changes on the revert/create
  operations.

- The kvm/migrate action also includes a minor modification to resolv
  the VM disk

- Snapshots with more than 1 child (or their relatives) cannot be deleted to not
  break backing chains.

- It has been found that the snap-delete operation in suspend state is
  insecure as the guest caches may not be updated accordingly. The
  operation is now not allowed in this state.

- The associated state has been removed from oned. It needs to be
  removed in CLI/Sunstone and APIs

- The changes are for VM disk-snapshots. A similar update can be made
  for images in the datastore/fs snap_delete action. This operation is
  not implemented in this commit.

Note: Output of xmllint may or may not include new lines when multiple matches
of an xpath expression occurs. Parsing of xmllint output adds additional new lines
to deal with different versions.
2023-07-20 19:04:28 +02:00
Ruben S. Montero
6bbfbb03e4
F #6063: Backup Jobs for OpenNebula
Includes the following changes:
- xml-schema for Backup Job and Scheduled Actions
- GO, Java api
- Deprecate onevm update-chart, delete-chart
    * The commands are replaced by sched-update and sched-delete
    * Refactor method deprecate_command, it's still possible to run the
    command
    * Delete 'shutdown' and 'delete' commands deprecated years ago
    * Fix --verbose option for sched-update and sched-delete
- Re-implementation of scheduled actions, now are managed and executed
  by oned
- Backup Job objects, API, and CLI commands
2023-07-03 18:15:52 +02:00
Pavel Czerný
167e25077a
F #6239: Update VMAction permissions (#2642)
* Add Sched actions and sg (de)attach to VM_*_OPERATION
* Add Backup cancel action
* For consistency add poweroff migrate, alias (de)attach and nic update to set_auth_ops
2023-06-21 09:40:03 +02:00
Bruno Rodriguez
99d6d46913
B #4045: Add pci root ports config on q35 vms
Signed-off-by: Bruno Rodriguez <brodriguez@opennebula.io>
(cherry picked from commit c7e15fc143eafd0e7b595f5d2d2eea64bb080ea7)
2023-06-19 12:59:55 +02:00
Ruben S. Montero
45f28e15d3
F #6185: Support huge pages without CPU pinning
VM topology can be defined:

- To use a NUMA node and huge pages. Example:
    TOPOLOGY = [ NODE_AFFINITY = 0, HUGEPAGE_SIZE = 2 ]

- To use huge pages without specifying the NUMA node. Example:
    TOPOLOGY = [ HUGEPAGE_SIZE = 2 ]

In any case OpenNebula will:

- look (or check) for a NUMA node with enough free huge pages
- will pick the NUMA node with more free pages (if no NUMA node is specified)
- configure the VM with CPU affinity to the selected NUMA node

(cherry picked from commit 1b3b88ed4e)
2023-04-24 20:45:45 +02:00
Ruben S. Montero
53b9611219
F #5966: Support for huge pages for NUMA affinity 2023-04-24 17:52:15 +02:00
Ruben S. Montero
4efa25c3e5
F #5966: NUMA affinity
* NODE_AFFINITY can be used to pin a VM to all the CPUs of a NUMA node.
  The attribute is part of the TOPOLOGY definition.

* A default virtual topology is defined when no other is set. This
  allows to set only the affinity:

     TOPOLOGY = [ NODE_AFFINITY = 1 ]

* NODE_AFFINITY can only be set for not-pinned VMs to not conflict with
  core allocations for pin VMs. VMs with NODE_AFFINITY set will be
  allocated to regular hosts (i.e. PIN_POLICY = NONE)

(cherry picked from commit fc15d6ebc9)
2023-04-23 00:01:11 +02:00
Ruben S. Montero
a9da75db0e
F #6030: Cancel backup operation
Author: Pavel Czerny <pczerny@opennebula.systems>
Author: Michal Opala <mopala@opennebula.io>
(cherry picked from commit f12c365d3e)
2023-03-16 13:11:27 +01:00
Pavel Czerný
75ee27eb32
M #-: Add more build warnings, const version of vector_value method (#2534)
* M #-: const version of vector_value returns const&
* M #-: Turn on compiler flags -Wextra -Werror
2023-03-08 15:52:20 +01:00
Ruben S. Montero
238e520bc9
F #6029: Signal increment flatten with an attribute
Extending the BACKUP state may lead to unneed wait times to perform VM
management operations

Fixes reset of backup chains for poweroff state

(cherry picked from commit 377db9653e)
2023-02-16 16:13:00 +01:00
Pavel Czerný
03add6d9ac
F #6103: CppCheck version 2.10 - solved issues (#2492)
* Update smoke test checkout action

(cherry picked from commit 466d336d7e)
2023-02-14 09:51:04 +01:00
Ruben S. Montero
c68c81a86e
F #6029: Prevent backup removal VM is in BACKUP
Some backup drivers may not allow delete (forget) operations while an
active backup is being performed. This commits fails the
one.image.delete API call in that case.

A new configuration attribute to describe the related driver behavior has been
added CONCURRENT_FORGET for DS_MAD_CONF.

(cherry picked from commit 940c1d7d63)
2023-02-09 19:40:44 +01:00
Ruben S. Montero
8cb3a360b8
F #6029, #6074: Retention and restore opetions for incremental backups
* Introduce support to follow KEEP_LAST for incremental backups.
  - New increment_flatten action added for backup datastores.
  - increment_flatten will consolidate KEEP_LAST increments into the
    current first increment in the chain.
  - increment_flatten MUST return the new chain (inc1:source1,...) and size
    of the new first increment (FULL) in the chain.

* Downloader logic for restore has been extracted from downloader.sh to
  reuse the increment flatten logic. A new command restic_downloader.rb
  process restic:// pseudo-urls.

* Restore process uses two new attributes to customize the restore
  process:
  - NAME to be used as base name for images and VM Template
  - INCREMENT_ID to restore disks from a given increment (not always the
    last one)

* Common logic has been added to BackupImage class (backup.rb)

* Includes the following fixes:
  - Fix when increment includes blocks larger than max qemu-io request size
  - Fix IMAGE counter for quotas on backup images
  - Fix rsync restore NO_IP / NO_NIC attributes

TODO:
* Mimic increment_flatten logic and restore images on the backup server
* Suntone restore options

co-authored-by:Michal Opala <mopala@opennebula.io>
2023-02-07 13:19:03 +01:00
Pavel Czerný
19f18e2877
M #-: Fix CppCheck issues in 6.6 branch (#2485)
(cherry picked from commit c70a239033)
2023-02-07 12:34:04 +01:00
Pavel Czerný
8e7ff2ed36
F #6103: Last set of CppCheck issues (#2481)
(cherry picked from commit 4a9662a551)
(cherry picked from commit d7b7303f8f)
2023-02-07 08:56:52 +01:00
Pavel Czerný
6ee3a5ef5b
F #6103: More CppCheck issues solved (#2477)
(cherry picked from commit 3fd059c543)
2023-02-02 12:54:45 +01:00
Pavel Czerný
a5c57b69b3
M #-: Add CppCheck to smoke tests (#2464) 2023-01-31 13:46:09 +01:00
Pavel Czerný
d7f1e1e0fc
B #6085: Fix trim of VNC/SPICE password (#2460)
(cherry picked from commit 726a6a8547)
2023-01-26 10:03:09 +01:00
Pavel Czerný
0696693531
B #6027: Parse host NUMA monitoring (#2440)
Co-authored-by: Ruben S. Montero <rsmontero@opennebula.org>
2023-01-09 18:29:09 +01:00
Tino Vázquez
3376afdaf0
M #-: Bump year 2023 2023-01-09 12:23:19 +01:00
Pavel Czerný
0fe9eac8b5
B #5916: Fix onevm instantiate --persistent (#2428) 2023-01-04 16:10:03 +01:00
Pavel Czerný
82d2191dfd
B #6022: Fix lock override and --all flag (#2427)
* This commit syncs oned and API specification for the ALL flag. The internal defines were not consistent with the API specification.
2023-01-04 15:53:50 +01:00
Pavel Czerný
9959ceac06
M #-: Bump DB version to 6.8 (#2425) 2022-12-22 14:20:25 +01:00
Tino Vázquez
4f9e45a57c
M #-: Bump version 6.7.80 2022-12-21 19:57:29 +01:00
Pavel Czerný
b25274323e
M #-: Trim VM template error message (#2424) 2022-12-19 15:21:12 +01:00
Tino Vázquez
238b5d704c
M #-: Bump version to 6.6.0 2022-12-15 18:07:06 +01:00
Jan Orel
4a2f8193a8
Revert "B #5801: Correctly terminate oned, if init fails (#2396)" (#2420)
This reverts commit 0221dc187a.
2022-12-13 15:46:54 +01:00
Ruben S. Montero
4df86811b9
F #5681: Support for block context devices
Add support for block context CD's. System Datastore can include
CONTEXT_DISK_TYPE to specify the type (FILER or BLOCK) of the context
CD's

co-authored-by: Victor Palma
2022-12-12 16:49:25 +01:00
Pavel Czerný
3beca2d9d3
B #5640: Fix NETRX and NETTX for accounting (#2417) 2022-12-12 09:28:46 +01:00
Ruben S. Montero
d711ea4362
F #5516: Incremental backups not compatible with snapshots
- Qemu will terminate VMs with system snapshots and checkpoints, example:

libvirtd[7446]: internal error: qemu unexpectedly closed the monitor: 2022-12-07T16:06:59.603678Z qemu-kvm-one: Device has active dirty bitmaps.

- Also disk snapshots may render inconsistent bitmaps, example:

'virsh --connect qemu:///system checkpoint-delete 45d76e5f-25fa-416a-97fd-1624380d5e02 one-191-0':   error: Failed to delete checkpoint one-191-0 error: internal error: unable to execute QEMU command 'query-named-block-nodes': Bitmap '' doesn't satisfy the constraints Error preparing disk files

This commits prevents snapshot operations when increment backups are configured, and vice versa.
2022-12-09 17:25:56 +01:00
Pavel Czerný
08e0e6c305
B #5728: Fix argument parsing for onemonitord (#2411)
* Fix argument parsing for onemonitord
* Update NebulaTemplate.h
2022-12-08 10:33:45 +01:00
Pavel Czerný
0221dc187a
B #5801: Correctly terminate oned, if init fails (#2396)
+ Improve formating of driver initialization log messages
2022-12-07 11:15:37 +01:00
Pavel Czerný
d8baeb028e
F #5925: Fix quotas after image delete failure (#2408)
* Fix quotas after image delete failure
* Fix set backup state after delete failure
2022-12-07 09:39:03 +01:00
Tino Vázquez
5b525c0c43 M #-: Bump version 6.5.90 2022-12-05 14:07:13 +01:00
Pavel Czerný
ff1f296ed1
M #-: Update Java bindings (#2386) 2022-11-29 16:09:59 +01:00
Pavel Czerný
2bda3d2c81
M #-: Fix minor issues in backups (#2366)
* Handle backup append flag for updateconf
* Fix backup attributes for updateconf
2022-11-23 10:00:24 +01:00
Ruben S. Montero
865908de1c
F #5989: Live update of Virtual Network attributes
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>
co-authored-by: Frederick Borges <fborges@opennebula.io>
co-authored-by: Christian González <cgonzalez@opennebula.io>

* VNET updates trigger a driver action on running VMs with NICs in the
  network.

* VNET includes a sets with VM status: updated, outdated, error and
  updating. With VMs in each state.

* VNET flags error situations with a new state UPDATE_FAILURE.

* The same procedure is applied when an AR is updated (only VMs in that
  AR are updated).

* A new options in the one.vn.recover API call enable to recover or
  retry this VM update operations.

* The following attributes can be live-updated per VNET driver:
    - PHYDEV (novlan, vlan, ovs driver)
    - MTU (vlan, ovs driver)
    - VLAN_ID (vlan, ovs driver)
    - QINQ_TYPE (ovs driver)
    - CVLANS (ovs driver)
    - VLAN_TAGGED_ID (ovs driver)
    - OUTER_VLAN_ID (ovs driver)
    - INBOUND_AVG_BW (SG, ovs driver + KVM)
    - INBOUND_PEAK_BW (SG, ovs driver + KVM)
    - INBOUND_PEAK_KB (SG, ovs driver + KVM)
    - OUTBOUND_AVG_BW (SG, ovs driver + KVM)
    - OUTBOUND_PEAK_BW (SG, ovs driver + KVM)
    - OUTBOUND_PEAK_KB (SG, ovs driver + KVM)

* New API call one.vm.updatenic, allows to update individual NICs
  without the need of detach/attach (only QoS supported).

* Update operations for: 802.1Q, bridge, fw, ovswitch, ovswitch_vxlan
  and vxlan network drivers.

* VNET attributes (old values) stored in VNET_UPDATE to allow
  implementation of update operations. The attribute is removed after a
  successful update.

* Updates to CLI onevnet (--retry option) / onevm (nicupdate command)

* XSD files updated to reflect the new data model

* Ruby and JAVA bindings updated: new VNET state and recover option, new
  VM API call.

* Suntone and Fireedge implementation (lease status, recover option, new
  states)

TODO: Virtual Functions does not support this functionality

iii
2022-11-16 15:06:49 +01:00
Pavel Czerný
de8c7b7869
F #5516: Dummy drivers for backup (#2350)
* Dummy drivers for backup/restore
* Sync backup ADMIN rights with oned.conf
2022-11-16 09:27:03 +01:00
Ruben S. Montero
0a46e21129
F #5516: Incremental backups for qcow2 disk images
- Adds new configuration attribute MODE: FULL or INCREMENTAL for
  BACKUP_CONFIG. FULL backups uses a differen backup image.

- INCREMENTAL backup information is together with the backup image.
  Example:

      <BACKUP_INCREMENTS>
        <INCREMENT>
          <DATE><![CDATA[1667770552]]></DATE>
          <ID><![CDATA[0]]></ID>
          <PARENT_ID><![CDATA[-1]]></PARENT_ID>
          <SIZE><![CDATA[172]]></SIZE>
          <SOURCE><![CDATA[bb828060]]></SOURCE>
          <TYPE><![CDATA[FULL]]></TYPE>
        </INCREMENT>
        <INCREMENT>
          <DATE><![CDATA[1667770604]]></DATE>
          <ID><![CDATA[1]]></ID>
          <PARENT_ID><![CDATA[0]]></PARENT_ID>
          <SIZE><![CDATA[1]]></SIZE>
          <SOURCE><![CDATA[ca0de5f6]]></SOURCE>
          <TYPE><![CDATA[INCREMENT]]></TYPE>
        </INCREMENT>
        <INCREMENT>
          <DATE><![CDATA[1667770700]]></DATE>
          <ID><![CDATA[2]]></ID>
          <PARENT_ID><![CDATA[1]]></PARENT_ID>
          <SIZE><![CDATA[39]]></SIZE>
          <SOURCE><![CDATA[e9897d6a]]></SOURCE>
          <TYPE><![CDATA[INCREMENT]]></TYPE>
        </INCREMENT>
      </BACKUP_INCREMENTS>

    This information only appears on incremental backups

- Internal BACKUP_CONFIG data includes information about the current
  active backup and the last increment id.

- Backup operation includes a new parameter: reset. This "closes" the
  current active incremental chain and creates a new FULL backup.

- Backup drivers store backups with increment index (0 = FULL) e.g.
  disk.0.0.

- Incremental backups are only allowed for VMs using all disks in qcow2
  format.

- Backup configuration cannot be changed while doing a VM backup.

- Downloader strings includes backup chains <inc_id>:<backup_ref>,...

- Restic downloader  has been updated to support backup chains. Disk
  images are rebased across increments.
2022-11-06 22:54:36 +01:00
Ruben S. Montero
57fe4c79e3
M #-: Missing pieces f62cd2c10c (6.6.0 versioning) 2022-10-27 11:40:57 +02:00
Tino Vázquez
f62cd2c10c
M #-: Downgrade version to 6.5.80 (#2314) 2022-10-21 14:27:45 +02:00
Ruben S. Montero
e433ccb85b
F #5516: New backup interface for OpenNebula
co-authored-by: Frederick Borges <fborges@opennebula.io>
co-authored-by: Neal Hansen <nhansen@opennebula.io>
co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io>
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>

BACKUP INTERFACE
=================

* Backups are exposed through a a special Datastore (BACKUP_DS) and
  Image (BACKUP) types. These new types can only be used for backup'ing
  up VMs. This approach allows to:

  - Implement tier based backup policies (backups made on different
    locations).

  - Leverage access control and quota systems

  - Support differnt storage and backup technologies

* Backup interface for the VMs:

  - VM configures backups with BACKUP_CONFIG. This attribute can be set
    in the VM template or updated with updateconf API call. It can include:

    + BACKUP_VOLATILE: To backup or not volatile disks

    + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent,
      suspend or none). When possible backups are crash consistent.

    + KEEP_LAST: keep only a given number of backups.

  - Backups are initiated by the one.vm.backup API call that requires
    the target Datastore to perform the backup (one-shot). This is
    exposed by the onevm backup command.

  - Backups can be periodic through scheduled actions.

  - Backup configuration is updated with one.vm.updateconf API call.

* Restore interface:

  - Restores are initiated by the one.image.restore API call. This is
    exposed by oneimage restore command.

  - Restore include configurable options for the VM template

    + NO_IP: to not preserve IP addresses (but keep the NICs and network
      mapping)

    + NO_NIC: to not preserve network mappings

  - Other template attributes:

    + Clean PCI devices, including network configuration in case of TYPE=NIC
    attributes. By default it removes SHORT_ADDRESS and leave the "auto"
    selection attributes.

    + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node

  - It is possible to restore single files stored in the repository by
    using the backup specific URL.

* Sunstone (Ruby version) has been updated to expose this feautres.

BACKUP DRIVERS & IMPLEMENTATION
===============================

* Backup operation is implemented by a combination of 3 driver operations:

  - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate
    backups for RUNNING VMs.

  - TM. This commit introduces 2 new operations (and their
    corresponding _live variants):

    + pre_backup(_live): Prepares the disks to be back'ed up in the
      repository. It is specific to the driver: (i) ceph uses the export
      operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as
      needed.
    + post_backup(_live): Performs cleanning operations, i.e. KVM
      snapshots or tmp dirs.

  - DATASTORE. Each backup technology is represented by its
    corresponfing driver, that needs to implement:

    + backup: it takes the VM disks in file (qcow2) format and stores it
      the backup repository.

    + restore: it takes a backup image and restores the associated disks
      and VM template.

    + monitor: to gather available space in the repository

    + rm: to remove existing backups

    + stat: to return the "restored" size of a disk stored in a backup

    + downloader pseudo-URL handler: in the form
      <backup_proto>://<driver_snapshot_id>/<disk filename>

BACKUP MANAGEMENT
=================

Backup actions may potentially take some time, leaving some vmm_exec threads in
use for a long time, stucking other vmm operations. Backups are planned
by the scheduler through the sched action interface.

Two attributes has been added to sched.conf:
  * MAX_BACKUPS max active backup operations in the cloud. No more
    backups will be started beyond this limit.

  * MAX_BACKUPS_HOST max number of backups per host

* Fix onevm CLI to properly show and manage schedule actions. --schedule
  supports now, as well as relative times +<seconds_from_stime>

  onvm backup --schedule now -d 100 63

* Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs
  to use the batch interface or request specific permissions

Internal restructure of Scheduler:

- All sched_actions interface is now in SchedActionsXML class and files.
  This class uses references to VM XML, and MUST be used in the same
  lifetime scope.

- XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as
  static functions.

- VirtualMachineActionPool includes counters for active backups (total
  and per host).

SUPPORTED PLATFORMS
====================
* hypervisor: KVM
* TM: qcow2/shared/ssh, ceph
* backup: restic, rsync

Notes on Ceph

* Ceph backups are performed in the following steps:
    1. A snapshot of each disk is taken (group snapshots cannot be used as
       it seems we cannot export the disks afterwards)
    2. Disks are export to a file
    3. File is converted to qcow2 format
    4. Disk files are upload to the backup repo

TODO:
  * Confirm crash consistent snapshots cannot be used in Ceph

TODO:
  * Check if using VM dir instead of full path is better to accomodate
    DS migrations i.e.:
    - Current path: /var/lib/one/datastores/100/53/backup/disk.0
    - Proposal: 53/backup/disk.0

RESTIC DRIVER
=============
Developed together with this feature is part of the EE edtion.

* It supports the SFTP protocol, the following attributes are
  supported:

  - RESTIC_SFTP_SERVER
  - RESTIC_SFTP_USER: only if different from oneadmin
  - RESTIC_PASSWORD
  - RESTIC_IONICE: Run restic under a given ionice priority (class 2)
  - RESTIC_NICE: Run restic under a given nice
  - RESTIC_BWLIMIT: Limit restic upload/download BW
  - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes:
    off, auto, max). This requires repositories version 2. By default,
    auto is used (average compression without to much CPU usage)
  - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a
    backend (5 by default). For high-latency backends this number can be
    increased.

* downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name>
  snapshot_id is the restic snapshot hash. To recover single disk images
  from a backup. This URLs support:

  - RESTIC_CONNECTIONS
  - RESTIC_BWLIMIT
  - RESTIC_IONICE
  - RESTIC_NICE

  These options needs to be defined in the associated datastore.

RSYNC DRIVER
=============
A rsync driver is included as part of the CE distribution. It uses the
rsync tool to store backups in a remote server through SSH:

* The following attributes are supported to configure the backup
  datastore:

  - RSYNC_HOST
  - RSYNC_USER
  - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default)

* downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover
  single files from an existing backup. (RSYNC_HOST and RSYN_USER needs
  to be set in ds_id

EMULATOR_CPUS
=============

This commit includes a non related backup feature:

* Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the
  CPU IDs where the emulator threads will be pinned. If this value is
  not defined the allocated CPU wll be used when using a PIN policy.

(cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b)

F OpenNebula/one#5516: adding rsync backup driver

(cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305)

F OpenNebula/one#5516: update install.sh, add vmid to source, some polish

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d)

F OpenNebula/one#5516: cleanup

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340)

F OpenNebula/one#5516: update downloader, default args, size check

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305)

LL

(cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-10-07 22:01:37 +02:00
Ruben S. Montero
74bc751bd7
F #5940: Improve PCI Passthrough and SRIOV support
- SHORT_ADDRESS can be used to select specific devices  (useful on
  homogenous clusters or nic attach operations). This name has been
  selected because of:
    1. It is the attrbute shown host info
    2. It does not collide with the reserved ADDRESS attribute

- New test and add functions that considers both allocation methods: by
  name (VENDOR/CLASS/DEVICE) or address (SHORT_ADDRESS)

- Parameter check on VM creation

- revert and add method use the same pci_attribute function to add info
  to the VM PCI attribute

- Remove well-known attributes when parsing PCI devices (ADDRESS,
  PREV_ADDRES, BUS, FUNCTION, SLOT, NUMA_NODE, UUID)

- Support for attach and detach NIC with PCI attributes
    * onevm_exec.rb looks for PCI devices for ATTACH=YES when attaching/detaching an interface
    * script action are now written in Ruby
    * KVM module with common actions (hostdev/interface device str)
    * Minor changes in xmlparser and OpenNebulaVM classes

- PCI selection options to onevm nic-attach:
    * pci short_address
    * pci_device device ID
    * pci_vendor vendor ID
    * pci_class class ID

- VF can be configured by setting some parameters through IP link (e.g.
  MAC or VLAN_ID). This commit includes a mixin to activate_vf

    * one_vmm_exec.rb looks for PCI VN_MAD drivers
    * VM class (VNM) adds a @pcis array
    * activate_vf should be called in the pre stage. The following drivers
      acticate VFs (VLAN_ID is implemented as 802.1Q tag)
            - 802.1Q
            - bridge
            - fw
            - ovswitch
    * Supported attributes for VF:
      - MAC
      - VLAN_ID
      - spoof checking (SPOOFCHK)
      - trust (TRUST)

- Predictable PCI addresses for guests is only enabled if PCI bus 1
  is present (PCI bridge bus cannot be hotplugged)

- Improve integration with Libvirt/QEMU:

    * When attach, only activate the VF being attached
    * Attach: Use <interface> and not <hostdev> for VF. There seems to be a race
      condition between accessing the vfio device and permission setup.
    * Attach: Remove address on attach as it may fail beacuse PCI controller
      is not present, e.g.:

        ATTACHNIC: Could not attach NIC to 28534240: error: Failed to attach device
        from /dev/fd/63 error: XML error: Invalid PCI address 0000:01:01.0.
        Only PCI buses up to 0 are available ExitCode: 1

    * Detach: Detach always use <hostdev> as libvirt fails to identified the
      device just by address when using <interface>

- Suntone Support: The "Hardware" profile of a network interface (NIC) of a VM can be of three types, now exposed in Sunstone:

    * "Emulated" it includes the hardware model emulated by Qemu
    * "PCI - Automatic" oned hw scheduler will pick the best PCI device for the NIC
    * "PCI - Manual" user can specify the PCI device by its short-address as shown in host information

This commits also enables the attach/detach operations on PCI based NICs in Suntone. *Note*: only for KVM VMs

(reverts commit aafd5f3014)
(reverts commit 671ef0b2d91aba00b1e6c63e19c4a50c3af20f6f)
2022-10-04 20:53:08 +02:00
Ruben S. Montero
907f26050f
F #5940: Improve PCI Passthrough and SRIOV support
- SHORT_ADDRESS can be used to select specific devices  (useful on
  homogenous clusters or nic attach operations). This name has been
  selected because of:
    1. It is the attrbute shown host info
    2. It does not collide with the reserved ADDRESS attribute

- New test and add functions that considers both allocation methods: by
  name (VENDOR/CLASS/DEVICE) or address (SHORT_ADDRESS)

- Parameter check on VM creation

- revert and add method use the same pci_attribute function to add info
  to the VM PCI attribute

- Remove well-known attributes when parsing PCI devices (ADDRESS,
  PREV_ADDRES, BUS, FUNCTION, SLOT, NUMA_NODE, UUID)

- Support for attach and detach NIC with PCI attributes
    * onevm_exec.rb looks for PCI devices for ATTACH=YES when attaching/detaching an interface
    * script action are now written in Ruby
    * KVM module with common actions (hostdev/interface device str)
    * Minor changes in xmlparser and OpenNebulaVM classes

- PCI selection options to onevm nic-attach:
    * pci short_address
    * pci_device device ID
    * pci_vendor vendor ID
    * pci_class class ID

- VF can be configured by setting some parameters through IP link (e.g.
  MAC or VLAN_ID). This commit includes a mixin to activate_vf.

    * one_vmm_exec.rb looks for PCI VN_MAD drivers
    * VM class (VNM) adds a @pcis array
    * activate_vf should be called in the pre stage. The following drivers
      acticate VFs (VLAN_ID is implemented as 802.1Q tag)
            - 802.1Q
            - bridge
            - fw
            - ovswitch

- Improve integration with Libvirt/QEMU:

    * When attach, only activate the VF being attached
    * Attach: Use <interface> and not <hostdev> for VF. There seems to be a race
      condition between accessing the vfio device and permission setup.
    * Attach: Remove address on attach as it may fail beacuse PCI controller
      is not present, e.g.:

        ATTACHNIC: Could not attach NIC to 28534240: error: Failed to attach device
        from /dev/fd/63 error: XML error: Invalid PCI address 0000:01:01.0.
        Only PCI buses up to 0 are available ExitCode: 1

    * Detach: Detach always use <hostdev> as libvirt fails to identified the
      device just by address when using <interface>
2022-09-07 15:54:38 +02:00
Pavel Czerný
e9e7aa5125
M #-: Fix oned segfault after driver init failed (#2217)
(cherry picked from commit aa54a8ca55)
2022-07-18 18:04:36 +02:00
Ruben S. Montero
7a1a85edb6
B #5867: Fix quotas and fsck
co-author: Pavel Czerny <pczerny@opennebula.systems>
(cherry picked from commit e2d4141599)
2022-06-20 18:35:40 +02:00
Alejandro Huertas Herrero
f63f313ddc
B #5820: add 4,5 debug levels (#2135) 2022-06-08 18:27:38 +02:00
Tino Vazquez
823aa8b6c1
M #-: Bump to 6.99.80 2022-04-20 12:47:55 +02:00
Pavel Czerný
3b0b9f349e
M #-: Bumb DB version (#1962) 2022-04-20 11:30:10 +02:00
Tino Vazquez
3c79648409 M #-: Bump version 6.9.80 2022-04-18 19:45:21 +02:00
Ruben S. Montero
4b746348b3
f #5725: State for Virtual Networks
* VNETs include state to flag their configuration status
  (READY/ERROR/LOCK)
* Hooks can be program based on VNET state transitions
* Include Java/Go/Ruby bindings
* New vCenter network drivers
* Place-holder to customize actions (create/delete) for other netowrk drivers
* Sunstone integration
* OneFlow syncs flow creation on VNET status
* Remove deprecated vcenter hooks and autoregistration

co-authored-by: Frederick Ernesto Borges Noronha <fborges@opennebula.io>
co-authored-by: Alejandro Huertas <ahuertas@opennebula.io>
co-authored-by: Pavel Czerny <pczerny@opennebula.systems>
2022-04-09 15:20:17 +02:00
Tino Vazquez
f7d53e75ff M #-: Bump version 6.3.85 2022-04-07 19:49:58 +02:00
Ruben S. Montero
3970d386b6
F #5792: Set KVM shares based on cgroup version
* New probe to get CGROUP_VERSION to the host
* LibvirtKVM driver set shares based on version (1024/100)
* Limits for min and max are check and enforced
2022-04-02 19:14:37 +02:00
Pavel Czerný
8571ff280d
B #-: Do not print password to oned.log (#1871)
(cherry picked from commit fd55fbb7ab46038426af5c3b5599055a56794275)
2022-03-29 14:21:24 +02:00
Ruben S. Montero
265d05c118
F #5612: (De)/Attach security groups from VM NICs
Author: Pavel Czerný <pczerny@opennebula.systems>
2022-01-25 20:27:05 +01:00
Ruben S. Montero
46b6837def
b #5697: Remove SAVE_AS_HOT attribute
(cherry picked from commit f100729f09)
2022-01-18 12:01:31 +01:00
Pavel Czerný
1242fbca95
B #629: Execute only first scheduled action (#1657)
(cherry picked from commit 22222bfeef)
2022-01-04 13:46:15 +01:00
Pavel Czerný
5591f88a3b
B #5524: Apply SYSTEM_DISK_USAGE for VM snapshots (#1647)
Co-authored-by: Ruben S. Montero <rsmontero@opennebula.org>
2022-01-04 13:03:47 +01:00
Christian González
42b7c9a3d3
M #: increase oned db versions (#1637) 2021-12-01 15:55:08 +01:00
Pavel Czerný
507f2f0c7f
F #5431: User encrypted attributes (#1574)
(cherry picked from commit 9aa26b295d)
2021-11-24 10:59:06 +01:00
Tino Vazquez
7e750c72a6
M #-: Bump version for 6.4 Beta
6.3.80
2021-11-03 13:09:25 +01:00
Tino Vazquez
f4c07fe1a9
M #-: Bump version to 6.2.0 2021-10-21 21:14:12 +02:00
Tino Vazquez
0effec897b
M #-: Bump version to 6.1.90 2021-10-14 18:28:05 +02:00
Pavel Czerný
3c2efafcc5
B #4193: Limit action execution on wilds (#1496) 2021-10-08 19:27:06 +02:00
Tino Vazquez
2436bbbde3
M #-: Bump to 6.1.85 2021-10-06 16:32:52 +02:00
Pavel Czerný
77276366a8
M #-: Method for case insensitive string compare (#1464) 2021-09-17 09:11:16 +02:00
Ruben S. Montero
121b4b096e
F #2481: UEFI Support
co-authored-by: Ricardo Diaz <rdiaz@opennebula.io>
co-authored-by: Pavel Czerny <pczerny@opennebula.systems>
2021-09-14 15:26:21 +02:00
Pavel Czerný
b80ee243c0
F #5482: Append flag for onevm updateconf (#1395) 2021-08-09 11:00:58 +02:00
Pavel Czerný
3e72c08212
B #3610: Fix failure response for disk-attach (#1352)
(cherry picked from commit d4f5a8bdf6)
2021-07-13 17:57:16 +02:00
Pavel Czerný
accdc228ca
M #-: Log messages to VM log (#1339)
Bring back behaviour from one-5.12
2021-06-30 10:05:05 +02:00
Pavel Czerný
de6d7dd150
B #2187: Increase AR size from 32 bit to 64 bit (#1295) 2021-06-24 10:52:46 +02:00
Pavel Czerný
9375625793
M #-: Fix market_mad_conf defaults (#1306)
(cherry picked from commit 5fc94b4992)
2021-06-18 11:22:10 +02:00
Ruben S. Montero
0f130a1abd
F #5289: Remove unneeded code 2021-06-15 11:24:19 +02:00
Ruben S. Montero
3f171f3b5b
F #5289: New api for sched actions
SchedActions are now a complete managed type. It is in in /TEMPLATE and
not /USER_TEMPLATE and includes dedicated API Calls to manage the
entries:

- one.vm.schedadd (add new sched_action)
- one.vm.schedupdate (update an exiting sched_action)
- one.vm.scheddelete (delete an action)

The scheduler now use this API. This will prevent race conditions
between update operations from different processes.

co-authored-by: Pavel Czerný <pczerny@opennebula.systems>
2021-06-15 08:22:26 +00:00
Ruben S. Montero
0ebda56330
M #-: Reduce attribute copies on updateconf
(cherry picked from commit 99d165856d)
2021-06-10 09:56:02 +02:00
Pavel Czerný
8b8d811e82
F #4510: Option to disable Marketplace (#1269) 2021-06-03 17:14:30 +02:00
Jan Orel
86f174d469
F #5381: Join qcow2 + shared TM drivers (#1217) 2021-05-26 18:21:13 +02:00
Pavel Czerný
6c71efc9ad
M #-: Fix occasional crash on exit (#1242)
(cherry picked from commit e16dfecd59)
2021-05-21 23:27:03 +02:00
Pavel Czerný
84d106a57a
M #-: Inherit Nebula from NebulaService (#1160) 2021-05-04 11:38:25 +02:00
Ruben S. Montero
f1a48bdc0d
F #5270, #5272: Support for Google Compute and DigitalOcean
co-authored-by: Alejandro Huertas <ahuertas@opennebula.io>
co-authored-by: Juan Antonio <jescobar@opennebula.io>
co-authored-by: Sergio Betanzos <sbetanzos@opennebula.io>
2021-05-01 21:53:09 +02:00
Pavel Czerny
69e83eb161
F #5353: Zone standby mode. DB version number to 6.2.0
Includes the following commits from f-5353:

  - 62d3d38d7fb135b82874a5076a7f06c23675e0b7
  - 52f3f066603c478061891175c008a58cfed3dc43
  - 0e82f9e5471c3cf0e21969393299279c6e739138
  - 8c32050940f528ea876e8293e6fbbf56a373bd54
2021-04-27 10:45:58 +02:00
Pavel Czerný
6bca41f0db
M #-: Showback option to calculate only running VM (#1105)
(cherry picked from commit c5ad2698b7)
2021-04-15 11:13:56 +02:00
Ruben S. Montero
0ef2364e7d
F #5324: Remove duplicates in Security Group
Author: Pavel Czerny <pczerny@opennebula.systems>
(cherry picked from commit fca2fb32c0)
2021-04-12 10:05:19 +00:00
Vlastimil Holer
23bf3b1534
M #-: Bump version to 6.1.80 (#1073) 2021-04-06 18:14:38 +02:00
Vlastimil Holer
c23c0ee26b
M #-: Bump version to 6.0.0 (#1026) 2021-03-25 14:17:10 +01:00
Vlastimil Holer
341991bce1
M #-: Bump version to 5.13.90 (and year in LXC) (#981) 2021-03-16 16:35:41 +01:00
Pavel Czerný
872939fb26
F #727: Improve shareable disks (#970)
* Shareable only on supported hypervisors
* Fix disk-attach
* Restrict shareable disks to SHARED datastores
* On image clone: copy SHAREABLE depends on DEFAULT_IMAGE_PERSISTENT flag
2021-03-15 16:24:25 +01:00
Ruben S. Montero
59b6e59fdd
M #-: Add comments 2021-03-15 14:23:48 +01:00
Alejandro Huertas
f92c6bbfab
B #5096: fix bug updating VM conf 2021-03-15 14:23:48 +01:00
Ruben S. Montero
cb52559200
WIP 2021-03-15 14:23:48 +01:00
Pavel Czerný
f7f076c638
M #-: Fix possible NIC name collision (#962) 2021-03-12 13:09:04 +01:00
Vlastimil Holer
ba44f49c91
M #-: Bump version to 5.13.85 (#896) 2021-03-01 14:11:26 +01:00
Pavel Czerný
ecd36bcb9a
B #5261: Fix PCI references after vm migrate fails (#877) 2021-02-25 11:45:13 +01:00
Pavel Czerný
ea25922462
M #-: Remove not implemented method do_action (#843) 2021-02-22 20:01:44 +01:00
Ruben S. Montero
60dd3a3cce
F #1345: Support for Image Encrypted attributes 2021-02-17 13:03:37 +00:00
Ruben S. Montero
e1ea0b7968
M #-: Fix Attach auth operation 2021-02-10 18:12:19 +00:00
Ruben S. Montero
ab260f6395
M #-: Fix wrong defaul auth operation level 2021-02-10 17:23:20 +00:00
Vlastimil Holer
096754b63f
M #-: Bump year to 2021 (#778) 2021-02-09 16:07:56 +01:00