1
0
mirror of https://github.com/OpenNebula/one.git synced 2025-01-18 06:03:39 +03:00
one/include/Backups.h

349 lines
8.5 KiB
C
Raw Normal View History

F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
/* -------------------------------------------------------------------------- */
2023-01-09 12:23:19 +01:00
/* Copyright 2002-2023, OpenNebula Project, OpenNebula Systems */
F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
/* */
/* Licensed under the Apache License, Version 2.0 (the "License"); you may */
/* not use this file except in compliance with the License. You may obtain */
/* a copy of the License at */
/* */
/* http://www.apache.org/licenses/LICENSE-2.0 */
/* */
/* Unless required by applicable law or agreed to in writing, software */
/* distributed under the License is distributed on an "AS IS" BASIS, */
/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
/* See the License for the specific language governing permissions and */
/* limitations under the License. */
/* -------------------------------------------------------------------------- */
#ifndef BACKUPS_H_
#define BACKUPS_H_
#include <string>
#include <map>
#include "ObjectCollection.h"
#include "Template.h"
class ObjectXML;
/**
* This class represents the backup information of a VM, it consists of two
* parts, configuration and list of backups
* The schema is as follows:
* <BACKUPS>
* <BACKUP_CONFIG>
* <KEEP_LAST> Just keep the last N backups
* <BACKUP_VOLATILE> Backup volatile disks or not
* <FS_FREEZE> FS freeze operation to perform on the VM
F #5516: Incremental backups for qcow2 disk images - Adds new configuration attribute MODE: FULL or INCREMENTAL for BACKUP_CONFIG. FULL backups uses a differen backup image. - INCREMENTAL backup information is together with the backup image. Example: <BACKUP_INCREMENTS> <INCREMENT> <DATE><![CDATA[1667770552]]></DATE> <ID><![CDATA[0]]></ID> <PARENT_ID><![CDATA[-1]]></PARENT_ID> <SIZE><![CDATA[172]]></SIZE> <SOURCE><![CDATA[bb828060]]></SOURCE> <TYPE><![CDATA[FULL]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770604]]></DATE> <ID><![CDATA[1]]></ID> <PARENT_ID><![CDATA[0]]></PARENT_ID> <SIZE><![CDATA[1]]></SIZE> <SOURCE><![CDATA[ca0de5f6]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770700]]></DATE> <ID><![CDATA[2]]></ID> <PARENT_ID><![CDATA[1]]></PARENT_ID> <SIZE><![CDATA[39]]></SIZE> <SOURCE><![CDATA[e9897d6a]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> </BACKUP_INCREMENTS> This information only appears on incremental backups - Internal BACKUP_CONFIG data includes information about the current active backup and the last increment id. - Backup operation includes a new parameter: reset. This "closes" the current active incremental chain and creates a new FULL backup. - Backup drivers store backups with increment index (0 = FULL) e.g. disk.0.0. - Incremental backups are only allowed for VMs using all disks in qcow2 format. - Backup configuration cannot be changed while doing a VM backup. - Downloader strings includes backup chains <inc_id>:<backup_ref>,... - Restic downloader has been updated to support backup chains. Disk images are rebased across increments.
2022-11-06 22:54:17 +01:00
* <MODE> Backup mode
F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
* <LAST_DATASTORE_ID> The dastore ID used to store the active backups(*)
* <LAST_BACKUP_ID> ID of the active backup(*)
* <LAST_BACKUP_SIZE> SIZE of the active backup(*)
* <ACTIVE_FLATTEN> if true current chain is being flatten
F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
* <BACKUP_IDS>
* <ID> ID of the image with a valid backup
*
* (*) refers to the active backup operation, and are only present while
* a backup is being performed
*
* Configuration attributes defaults
* - BACKUP_VOLATILE "NO"
* - FS_FREEZE "NONE"
* - KEEP_LAST (empty = keep all)
*/
class Backups
{
public:
Backups();
~Backups() = default;
F #5516: Incremental backups for qcow2 disk images - Adds new configuration attribute MODE: FULL or INCREMENTAL for BACKUP_CONFIG. FULL backups uses a differen backup image. - INCREMENTAL backup information is together with the backup image. Example: <BACKUP_INCREMENTS> <INCREMENT> <DATE><![CDATA[1667770552]]></DATE> <ID><![CDATA[0]]></ID> <PARENT_ID><![CDATA[-1]]></PARENT_ID> <SIZE><![CDATA[172]]></SIZE> <SOURCE><![CDATA[bb828060]]></SOURCE> <TYPE><![CDATA[FULL]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770604]]></DATE> <ID><![CDATA[1]]></ID> <PARENT_ID><![CDATA[0]]></PARENT_ID> <SIZE><![CDATA[1]]></SIZE> <SOURCE><![CDATA[ca0de5f6]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770700]]></DATE> <ID><![CDATA[2]]></ID> <PARENT_ID><![CDATA[1]]></PARENT_ID> <SIZE><![CDATA[39]]></SIZE> <SOURCE><![CDATA[e9897d6a]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> </BACKUP_INCREMENTS> This information only appears on incremental backups - Internal BACKUP_CONFIG data includes information about the current active backup and the last increment id. - Backup operation includes a new parameter: reset. This "closes" the current active incremental chain and creates a new FULL backup. - Backup drivers store backups with increment index (0 = FULL) e.g. disk.0.0. - Incremental backups are only allowed for VMs using all disks in qcow2 format. - Backup configuration cannot be changed while doing a VM backup. - Downloader strings includes backup chains <inc_id>:<backup_ref>,... - Restic downloader has been updated to support backup chains. Disk images are rebased across increments.
2022-11-06 22:54:17 +01:00
// *************************************************************************
// Backup modes
// *************************************************************************
enum Mode
{
FULL = 0, /** < Full backups */
INCREMENT = 1, /** < Forward increments */
};
static std::string mode_to_str(Mode bm)
{
switch (bm)
{
case FULL: return "FULL";
case INCREMENT: return "INCREMENT";
default: return "";
}
};
static Mode str_to_mode(std::string& str_mode)
{
Mode mode = FULL;
one_util::toupper(str_mode);
if ( str_mode == "FULL" )
{
mode = FULL;
}
else if ( str_mode == "INCREMENT" )
{
mode = INCREMENT;
}
return mode;
};
F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
// *************************************************************************
// Inititalization functions
// *************************************************************************
/**
* Builds the snapshot list from its XML representation. This function
* is used when importing it from the DB.
* @param node xmlNode for the template
* @return 0 on success
*/
int from_xml(const ObjectXML* xml);
/**
* XML Representation of the Snapshots
*/
std::string& to_xml(std::string& xml) const;
/**
* Gets the BACKUP_CONFIG attribute attribute and parses the associated
* attributes:
* - BACKUP_VOLATILE
* - KEEP_LAST
* - FS_FREEZE
F #5516: Incremental backups for qcow2 disk images - Adds new configuration attribute MODE: FULL or INCREMENTAL for BACKUP_CONFIG. FULL backups uses a differen backup image. - INCREMENTAL backup information is together with the backup image. Example: <BACKUP_INCREMENTS> <INCREMENT> <DATE><![CDATA[1667770552]]></DATE> <ID><![CDATA[0]]></ID> <PARENT_ID><![CDATA[-1]]></PARENT_ID> <SIZE><![CDATA[172]]></SIZE> <SOURCE><![CDATA[bb828060]]></SOURCE> <TYPE><![CDATA[FULL]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770604]]></DATE> <ID><![CDATA[1]]></ID> <PARENT_ID><![CDATA[0]]></PARENT_ID> <SIZE><![CDATA[1]]></SIZE> <SOURCE><![CDATA[ca0de5f6]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770700]]></DATE> <ID><![CDATA[2]]></ID> <PARENT_ID><![CDATA[1]]></PARENT_ID> <SIZE><![CDATA[39]]></SIZE> <SOURCE><![CDATA[e9897d6a]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> </BACKUP_INCREMENTS> This information only appears on incremental backups - Internal BACKUP_CONFIG data includes information about the current active backup and the last increment id. - Backup operation includes a new parameter: reset. This "closes" the current active incremental chain and creates a new FULL backup. - Backup drivers store backups with increment index (0 = FULL) e.g. disk.0.0. - Incremental backups are only allowed for VMs using all disks in qcow2 format. - Backup configuration cannot be changed while doing a VM backup. - Downloader strings includes backup chains <inc_id>:<backup_ref>,... - Restic downloader has been updated to support backup chains. Disk images are rebased across increments.
2022-11-06 22:54:17 +01:00
* - MODE
F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
*
* The following attributes are stored in the configuration and refers
* only to the active backup operation
* - LAST_DATASTORE_ID
* - LAST_BACKUP_ID
* - LAST_BACKUP_SIZE
F #5516: Incremental backups for qcow2 disk images - Adds new configuration attribute MODE: FULL or INCREMENTAL for BACKUP_CONFIG. FULL backups uses a differen backup image. - INCREMENTAL backup information is together with the backup image. Example: <BACKUP_INCREMENTS> <INCREMENT> <DATE><![CDATA[1667770552]]></DATE> <ID><![CDATA[0]]></ID> <PARENT_ID><![CDATA[-1]]></PARENT_ID> <SIZE><![CDATA[172]]></SIZE> <SOURCE><![CDATA[bb828060]]></SOURCE> <TYPE><![CDATA[FULL]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770604]]></DATE> <ID><![CDATA[1]]></ID> <PARENT_ID><![CDATA[0]]></PARENT_ID> <SIZE><![CDATA[1]]></SIZE> <SOURCE><![CDATA[ca0de5f6]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770700]]></DATE> <ID><![CDATA[2]]></ID> <PARENT_ID><![CDATA[1]]></PARENT_ID> <SIZE><![CDATA[39]]></SIZE> <SOURCE><![CDATA[e9897d6a]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> </BACKUP_INCREMENTS> This information only appears on incremental backups - Internal BACKUP_CONFIG data includes information about the current active backup and the last increment id. - Backup operation includes a new parameter: reset. This "closes" the current active incremental chain and creates a new FULL backup. - Backup drivers store backups with increment index (0 = FULL) e.g. disk.0.0. - Incremental backups are only allowed for VMs using all disks in qcow2 format. - Backup configuration cannot be changed while doing a VM backup. - Downloader strings includes backup chains <inc_id>:<backup_ref>,... - Restic downloader has been updated to support backup chains. Disk images are rebased across increments.
2022-11-06 22:54:17 +01:00
*
* Incremental backups include a reference to the last increment and full
* backup:
* - LAST_INCREMENT_ID
* - INCREMENTAL_BACKUP_ID
* @param tmpl Template to parse, the root element must be BACKUP_CONFIG
* @param can_increment VM disks support incremental backup
* @param append Only append new values from tmpl
* @param error_str Returns the error reason, if any
* @return 0 success, -1 error
F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
*/
int parse(Template *tmpl, bool can_increment,
bool append, std::string& error_str);
F #5516: Incremental backups for qcow2 disk images - Adds new configuration attribute MODE: FULL or INCREMENTAL for BACKUP_CONFIG. FULL backups uses a differen backup image. - INCREMENTAL backup information is together with the backup image. Example: <BACKUP_INCREMENTS> <INCREMENT> <DATE><![CDATA[1667770552]]></DATE> <ID><![CDATA[0]]></ID> <PARENT_ID><![CDATA[-1]]></PARENT_ID> <SIZE><![CDATA[172]]></SIZE> <SOURCE><![CDATA[bb828060]]></SOURCE> <TYPE><![CDATA[FULL]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770604]]></DATE> <ID><![CDATA[1]]></ID> <PARENT_ID><![CDATA[0]]></PARENT_ID> <SIZE><![CDATA[1]]></SIZE> <SOURCE><![CDATA[ca0de5f6]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770700]]></DATE> <ID><![CDATA[2]]></ID> <PARENT_ID><![CDATA[1]]></PARENT_ID> <SIZE><![CDATA[39]]></SIZE> <SOURCE><![CDATA[e9897d6a]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> </BACKUP_INCREMENTS> This information only appears on incremental backups - Internal BACKUP_CONFIG data includes information about the current active backup and the last increment id. - Backup operation includes a new parameter: reset. This "closes" the current active incremental chain and creates a new FULL backup. - Backup drivers store backups with increment index (0 = FULL) e.g. disk.0.0. - Incremental backups are only allowed for VMs using all disks in qcow2 format. - Backup configuration cannot be changed while doing a VM backup. - Downloader strings includes backup chains <inc_id>:<backup_ref>,... - Restic downloader has been updated to support backup chains. Disk images are rebased across increments.
2022-11-06 22:54:17 +01:00
/**
* @return true if Backup includes configuration attributes
*/
bool configured()
{
return !config.empty();
F #5516: Incremental backups for qcow2 disk images - Adds new configuration attribute MODE: FULL or INCREMENTAL for BACKUP_CONFIG. FULL backups uses a differen backup image. - INCREMENTAL backup information is together with the backup image. Example: <BACKUP_INCREMENTS> <INCREMENT> <DATE><![CDATA[1667770552]]></DATE> <ID><![CDATA[0]]></ID> <PARENT_ID><![CDATA[-1]]></PARENT_ID> <SIZE><![CDATA[172]]></SIZE> <SOURCE><![CDATA[bb828060]]></SOURCE> <TYPE><![CDATA[FULL]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770604]]></DATE> <ID><![CDATA[1]]></ID> <PARENT_ID><![CDATA[0]]></PARENT_ID> <SIZE><![CDATA[1]]></SIZE> <SOURCE><![CDATA[ca0de5f6]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770700]]></DATE> <ID><![CDATA[2]]></ID> <PARENT_ID><![CDATA[1]]></PARENT_ID> <SIZE><![CDATA[39]]></SIZE> <SOURCE><![CDATA[e9897d6a]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> </BACKUP_INCREMENTS> This information only appears on incremental backups - Internal BACKUP_CONFIG data includes information about the current active backup and the last increment id. - Backup operation includes a new parameter: reset. This "closes" the current active incremental chain and creates a new FULL backup. - Backup drivers store backups with increment index (0 = FULL) e.g. disk.0.0. - Incremental backups are only allowed for VMs using all disks in qcow2 format. - Backup configuration cannot be changed while doing a VM backup. - Downloader strings includes backup chains <inc_id>:<backup_ref>,... - Restic downloader has been updated to support backup chains. Disk images are rebased across increments.
2022-11-06 22:54:17 +01:00
}
F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
/**
* @return true if the backup needs to include volatile disks
*/
bool do_volatile() const;
F #5516: Incremental backups for qcow2 disk images - Adds new configuration attribute MODE: FULL or INCREMENTAL for BACKUP_CONFIG. FULL backups uses a differen backup image. - INCREMENTAL backup information is together with the backup image. Example: <BACKUP_INCREMENTS> <INCREMENT> <DATE><![CDATA[1667770552]]></DATE> <ID><![CDATA[0]]></ID> <PARENT_ID><![CDATA[-1]]></PARENT_ID> <SIZE><![CDATA[172]]></SIZE> <SOURCE><![CDATA[bb828060]]></SOURCE> <TYPE><![CDATA[FULL]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770604]]></DATE> <ID><![CDATA[1]]></ID> <PARENT_ID><![CDATA[0]]></PARENT_ID> <SIZE><![CDATA[1]]></SIZE> <SOURCE><![CDATA[ca0de5f6]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770700]]></DATE> <ID><![CDATA[2]]></ID> <PARENT_ID><![CDATA[1]]></PARENT_ID> <SIZE><![CDATA[39]]></SIZE> <SOURCE><![CDATA[e9897d6a]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> </BACKUP_INCREMENTS> This information only appears on incremental backups - Internal BACKUP_CONFIG data includes information about the current active backup and the last increment id. - Backup operation includes a new parameter: reset. This "closes" the current active incremental chain and creates a new FULL backup. - Backup drivers store backups with increment index (0 = FULL) e.g. disk.0.0. - Incremental backups are only allowed for VMs using all disks in qcow2 format. - Backup configuration cannot be changed while doing a VM backup. - Downloader strings includes backup chains <inc_id>:<backup_ref>,... - Restic downloader has been updated to support backup chains. Disk images are rebased across increments.
2022-11-06 22:54:17 +01:00
/**
* @return true if the backup needs to include volatile disks
*/
Mode mode() const;
F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
/**
* Set of functions to manipulate the LAST_* attributes referring to
* the active backup operation
*/
void last_datastore_id(int ds_id)
{
config.replace("LAST_DATASTORE_ID", ds_id);
}
void last_backup_id(const std::string& id)
{
config.replace("LAST_BACKUP_ID", id);
}
void last_backup_size(const std::string& size)
{
config.replace("LAST_BACKUP_SIZE", size);
}
F #5516: Incremental backups for qcow2 disk images - Adds new configuration attribute MODE: FULL or INCREMENTAL for BACKUP_CONFIG. FULL backups uses a differen backup image. - INCREMENTAL backup information is together with the backup image. Example: <BACKUP_INCREMENTS> <INCREMENT> <DATE><![CDATA[1667770552]]></DATE> <ID><![CDATA[0]]></ID> <PARENT_ID><![CDATA[-1]]></PARENT_ID> <SIZE><![CDATA[172]]></SIZE> <SOURCE><![CDATA[bb828060]]></SOURCE> <TYPE><![CDATA[FULL]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770604]]></DATE> <ID><![CDATA[1]]></ID> <PARENT_ID><![CDATA[0]]></PARENT_ID> <SIZE><![CDATA[1]]></SIZE> <SOURCE><![CDATA[ca0de5f6]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770700]]></DATE> <ID><![CDATA[2]]></ID> <PARENT_ID><![CDATA[1]]></PARENT_ID> <SIZE><![CDATA[39]]></SIZE> <SOURCE><![CDATA[e9897d6a]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> </BACKUP_INCREMENTS> This information only appears on incremental backups - Internal BACKUP_CONFIG data includes information about the current active backup and the last increment id. - Backup operation includes a new parameter: reset. This "closes" the current active incremental chain and creates a new FULL backup. - Backup drivers store backups with increment index (0 = FULL) e.g. disk.0.0. - Incremental backups are only allowed for VMs using all disks in qcow2 format. - Backup configuration cannot be changed while doing a VM backup. - Downloader strings includes backup chains <inc_id>:<backup_ref>,... - Restic downloader has been updated to support backup chains. Disk images are rebased across increments.
2022-11-06 22:54:17 +01:00
void last_increment_id(int id)
{
config.replace("LAST_INCREMENT_ID", id);
}
void incremental_backup_id(int id)
{
config.replace("INCREMENTAL_BACKUP_ID", id);
}
void active_flatten(bool status)
{
config.replace("ACTIVE_FLATTEN", status);
}
void backup_job_id(int id)
{
config.replace("BACKUP_JOB_ID", id);
}
F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
/* ---------------------------------------------------------------------- */
int last_datastore_id() const
{
int dst;
config.get("LAST_DATASTORE_ID", dst);
return dst;
}
std::string last_backup_id() const
{
std::string id;
config.get("LAST_BACKUP_ID", id);
return id;
}
std::string last_backup_size() const
{
std::string sz;
config.get("LAST_BACKUP_SIZE", sz);
return sz;
}
F #5516: Incremental backups for qcow2 disk images - Adds new configuration attribute MODE: FULL or INCREMENTAL for BACKUP_CONFIG. FULL backups uses a differen backup image. - INCREMENTAL backup information is together with the backup image. Example: <BACKUP_INCREMENTS> <INCREMENT> <DATE><![CDATA[1667770552]]></DATE> <ID><![CDATA[0]]></ID> <PARENT_ID><![CDATA[-1]]></PARENT_ID> <SIZE><![CDATA[172]]></SIZE> <SOURCE><![CDATA[bb828060]]></SOURCE> <TYPE><![CDATA[FULL]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770604]]></DATE> <ID><![CDATA[1]]></ID> <PARENT_ID><![CDATA[0]]></PARENT_ID> <SIZE><![CDATA[1]]></SIZE> <SOURCE><![CDATA[ca0de5f6]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> <INCREMENT> <DATE><![CDATA[1667770700]]></DATE> <ID><![CDATA[2]]></ID> <PARENT_ID><![CDATA[1]]></PARENT_ID> <SIZE><![CDATA[39]]></SIZE> <SOURCE><![CDATA[e9897d6a]]></SOURCE> <TYPE><![CDATA[INCREMENT]]></TYPE> </INCREMENT> </BACKUP_INCREMENTS> This information only appears on incremental backups - Internal BACKUP_CONFIG data includes information about the current active backup and the last increment id. - Backup operation includes a new parameter: reset. This "closes" the current active incremental chain and creates a new FULL backup. - Backup drivers store backups with increment index (0 = FULL) e.g. disk.0.0. - Incremental backups are only allowed for VMs using all disks in qcow2 format. - Backup configuration cannot be changed while doing a VM backup. - Downloader strings includes backup chains <inc_id>:<backup_ref>,... - Restic downloader has been updated to support backup chains. Disk images are rebased across increments.
2022-11-06 22:54:17 +01:00
int last_increment_id() const
{
int id;
config.get("LAST_INCREMENT_ID", id);
return id;
}
int incremental_backup_id() const
{
int id;
config.get("INCREMENTAL_BACKUP_ID", id);
return id;
}
int keep_last() const
{
int kl;
if (!config.get("KEEP_LAST", kl))
{
return 0;
}
return kl;
}
bool active_flatten() const
{
bool af = false;
config.get("ACTIVE_FLATTEN", af);
return af;
}
int backup_job_id() const
{
int id;
if (!config.get("BACKUP_JOB_ID", id))
{
return -1;
}
return id;
}
F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
/* ---------------------------------------------------------------------- */
void last_backup_clear()
{
config.erase("LAST_DATASTORE_ID");
config.erase("LAST_BACKUP_ID");
config.erase("LAST_BACKUP_SIZE");
}
/**
* @param riids Return the backups that needs to be removed to conform
* to KEEP_LAST configuration
*/
void remove_last(std::set<int> &riids) const
{
int kl;
riids.clear();
if (!config.get("KEEP_LAST", kl) || kl == 0)
{
return;
}
auto iids = ids.get_collection();
auto it = iids.cbegin();
int to_remove = iids.size() - kl;
for (int i = 0 ; i < to_remove && it != iids.cend() ; ++i, ++it)
{
riids.insert(*it);
}
}
void remove_backup_job_id()
{
config.erase("BACKUP_JOB_ID");
}
F #5516: New backup interface for OpenNebula co-authored-by: Frederick Borges <fborges@opennebula.io> co-authored-by: Neal Hansen <nhansen@opennebula.io> co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io> co-authored-by: Pavel Czerný <pczerny@opennebula.systems> BACKUP INTERFACE ================= * Backups are exposed through a a special Datastore (BACKUP_DS) and Image (BACKUP) types. These new types can only be used for backup'ing up VMs. This approach allows to: - Implement tier based backup policies (backups made on different locations). - Leverage access control and quota systems - Support differnt storage and backup technologies * Backup interface for the VMs: - VM configures backups with BACKUP_CONFIG. This attribute can be set in the VM template or updated with updateconf API call. It can include: + BACKUP_VOLATILE: To backup or not volatile disks + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent, suspend or none). When possible backups are crash consistent. + KEEP_LAST: keep only a given number of backups. - Backups are initiated by the one.vm.backup API call that requires the target Datastore to perform the backup (one-shot). This is exposed by the onevm backup command. - Backups can be periodic through scheduled actions. - Backup configuration is updated with one.vm.updateconf API call. * Restore interface: - Restores are initiated by the one.image.restore API call. This is exposed by oneimage restore command. - Restore include configurable options for the VM template + NO_IP: to not preserve IP addresses (but keep the NICs and network mapping) + NO_NIC: to not preserve network mappings - Other template attributes: + Clean PCI devices, including network configuration in case of TYPE=NIC attributes. By default it removes SHORT_ADDRESS and leave the "auto" selection attributes. + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node - It is possible to restore single files stored in the repository by using the backup specific URL. * Sunstone (Ruby version) has been updated to expose this feautres. BACKUP DRIVERS & IMPLEMENTATION =============================== * Backup operation is implemented by a combination of 3 driver operations: - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate backups for RUNNING VMs. - TM. This commit introduces 2 new operations (and their corresponding _live variants): + pre_backup(_live): Prepares the disks to be back'ed up in the repository. It is specific to the driver: (i) ceph uses the export operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as needed. + post_backup(_live): Performs cleanning operations, i.e. KVM snapshots or tmp dirs. - DATASTORE. Each backup technology is represented by its corresponfing driver, that needs to implement: + backup: it takes the VM disks in file (qcow2) format and stores it the backup repository. + restore: it takes a backup image and restores the associated disks and VM template. + monitor: to gather available space in the repository + rm: to remove existing backups + stat: to return the "restored" size of a disk stored in a backup + downloader pseudo-URL handler: in the form <backup_proto>://<driver_snapshot_id>/<disk filename> BACKUP MANAGEMENT ================= Backup actions may potentially take some time, leaving some vmm_exec threads in use for a long time, stucking other vmm operations. Backups are planned by the scheduler through the sched action interface. Two attributes has been added to sched.conf: * MAX_BACKUPS max active backup operations in the cloud. No more backups will be started beyond this limit. * MAX_BACKUPS_HOST max number of backups per host * Fix onevm CLI to properly show and manage schedule actions. --schedule supports now, as well as relative times +<seconds_from_stime> onvm backup --schedule now -d 100 63 * Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs to use the batch interface or request specific permissions Internal restructure of Scheduler: - All sched_actions interface is now in SchedActionsXML class and files. This class uses references to VM XML, and MUST be used in the same lifetime scope. - XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as static functions. - VirtualMachineActionPool includes counters for active backups (total and per host). SUPPORTED PLATFORMS ==================== * hypervisor: KVM * TM: qcow2/shared/ssh, ceph * backup: restic, rsync Notes on Ceph * Ceph backups are performed in the following steps: 1. A snapshot of each disk is taken (group snapshots cannot be used as it seems we cannot export the disks afterwards) 2. Disks are export to a file 3. File is converted to qcow2 format 4. Disk files are upload to the backup repo TODO: * Confirm crash consistent snapshots cannot be used in Ceph TODO: * Check if using VM dir instead of full path is better to accomodate DS migrations i.e.: - Current path: /var/lib/one/datastores/100/53/backup/disk.0 - Proposal: 53/backup/disk.0 RESTIC DRIVER ============= Developed together with this feature is part of the EE edtion. * It supports the SFTP protocol, the following attributes are supported: - RESTIC_SFTP_SERVER - RESTIC_SFTP_USER: only if different from oneadmin - RESTIC_PASSWORD - RESTIC_IONICE: Run restic under a given ionice priority (class 2) - RESTIC_NICE: Run restic under a given nice - RESTIC_BWLIMIT: Limit restic upload/download BW - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes: off, auto, max). This requires repositories version 2. By default, auto is used (average compression without to much CPU usage) - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a backend (5 by default). For high-latency backends this number can be increased. * downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name> snapshot_id is the restic snapshot hash. To recover single disk images from a backup. This URLs support: - RESTIC_CONNECTIONS - RESTIC_BWLIMIT - RESTIC_IONICE - RESTIC_NICE These options needs to be defined in the associated datastore. RSYNC DRIVER ============= A rsync driver is included as part of the CE distribution. It uses the rsync tool to store backups in a remote server through SSH: * The following attributes are supported to configure the backup datastore: - RSYNC_HOST - RSYNC_USER - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default) * downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover single files from an existing backup. (RSYNC_HOST and RSYN_USER needs to be set in ds_id EMULATOR_CPUS ============= This commit includes a non related backup feature: * Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the CPU IDs where the emulator threads will be pinned. If this value is not defined the allocated CPU wll be used when using a PIN policy. (cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b) F OpenNebula/one#5516: adding rsync backup driver (cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305) F OpenNebula/one#5516: update install.sh, add vmid to source, some polish Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d) F OpenNebula/one#5516: cleanup Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340) F OpenNebula/one#5516: update downloader, default args, size check Signed-off-by: Neal Hansen <nhansen@opennebula.io> (cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305) LL (cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-09-09 11:46:44 +02:00
/**
* Adds / deletes a backup from the list. Each backup is represented by
* an image in the backup datastore. The list holds the ID's of the images
*
* @return 0 on success -1 if an error adding (already present) or deleting
* (not present) occurred
*/
int add(int id)
{
return ids.add(id);
}
int del(int id)
{
return ids.del(id);
}
private:
/**
* Text representation of the backup information of the VM
*/
Template config;
/**
* Backups of the VM as a collection of Image ID
*/
ObjectCollection ids;
};
#endif /*BACKUPS_H_*/