1
0
mirror of https://github.com/OpenNebula/one.git synced 2024-12-23 17:33:56 +03:00
one/include/NebulaUtil.h
Ruben S. Montero e433ccb85b
F #5516: New backup interface for OpenNebula
co-authored-by: Frederick Borges <fborges@opennebula.io>
co-authored-by: Neal Hansen <nhansen@opennebula.io>
co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io>
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>

BACKUP INTERFACE
=================

* Backups are exposed through a a special Datastore (BACKUP_DS) and
  Image (BACKUP) types. These new types can only be used for backup'ing
  up VMs. This approach allows to:

  - Implement tier based backup policies (backups made on different
    locations).

  - Leverage access control and quota systems

  - Support differnt storage and backup technologies

* Backup interface for the VMs:

  - VM configures backups with BACKUP_CONFIG. This attribute can be set
    in the VM template or updated with updateconf API call. It can include:

    + BACKUP_VOLATILE: To backup or not volatile disks

    + FS_FREEZE: How the FS is freeze for running VMs (qemu-agent,
      suspend or none). When possible backups are crash consistent.

    + KEEP_LAST: keep only a given number of backups.

  - Backups are initiated by the one.vm.backup API call that requires
    the target Datastore to perform the backup (one-shot). This is
    exposed by the onevm backup command.

  - Backups can be periodic through scheduled actions.

  - Backup configuration is updated with one.vm.updateconf API call.

* Restore interface:

  - Restores are initiated by the one.image.restore API call. This is
    exposed by oneimage restore command.

  - Restore include configurable options for the VM template

    + NO_IP: to not preserve IP addresses (but keep the NICs and network
      mapping)

    + NO_NIC: to not preserve network mappings

  - Other template attributes:

    + Clean PCI devices, including network configuration in case of TYPE=NIC
    attributes. By default it removes SHORT_ADDRESS and leave the "auto"
    selection attributes.

    + Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node

  - It is possible to restore single files stored in the repository by
    using the backup specific URL.

* Sunstone (Ruby version) has been updated to expose this feautres.

BACKUP DRIVERS & IMPLEMENTATION
===============================

* Backup operation is implemented by a combination of 3 driver operations:

  - VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate
    backups for RUNNING VMs.

  - TM. This commit introduces 2 new operations (and their
    corresponding _live variants):

    + pre_backup(_live): Prepares the disks to be back'ed up in the
      repository. It is specific to the driver: (i) ceph uses the export
      operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as
      needed.
    + post_backup(_live): Performs cleanning operations, i.e. KVM
      snapshots or tmp dirs.

  - DATASTORE. Each backup technology is represented by its
    corresponfing driver, that needs to implement:

    + backup: it takes the VM disks in file (qcow2) format and stores it
      the backup repository.

    + restore: it takes a backup image and restores the associated disks
      and VM template.

    + monitor: to gather available space in the repository

    + rm: to remove existing backups

    + stat: to return the "restored" size of a disk stored in a backup

    + downloader pseudo-URL handler: in the form
      <backup_proto>://<driver_snapshot_id>/<disk filename>

BACKUP MANAGEMENT
=================

Backup actions may potentially take some time, leaving some vmm_exec threads in
use for a long time, stucking other vmm operations. Backups are planned
by the scheduler through the sched action interface.

Two attributes has been added to sched.conf:
  * MAX_BACKUPS max active backup operations in the cloud. No more
    backups will be started beyond this limit.

  * MAX_BACKUPS_HOST max number of backups per host

* Fix onevm CLI to properly show and manage schedule actions. --schedule
  supports now, as well as relative times +<seconds_from_stime>

  onvm backup --schedule now -d 100 63

* Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs
  to use the batch interface or request specific permissions

Internal restructure of Scheduler:

- All sched_actions interface is now in SchedActionsXML class and files.
  This class uses references to VM XML, and MUST be used in the same
  lifetime scope.

- XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as
  static functions.

- VirtualMachineActionPool includes counters for active backups (total
  and per host).

SUPPORTED PLATFORMS
====================
* hypervisor: KVM
* TM: qcow2/shared/ssh, ceph
* backup: restic, rsync

Notes on Ceph

* Ceph backups are performed in the following steps:
    1. A snapshot of each disk is taken (group snapshots cannot be used as
       it seems we cannot export the disks afterwards)
    2. Disks are export to a file
    3. File is converted to qcow2 format
    4. Disk files are upload to the backup repo

TODO:
  * Confirm crash consistent snapshots cannot be used in Ceph

TODO:
  * Check if using VM dir instead of full path is better to accomodate
    DS migrations i.e.:
    - Current path: /var/lib/one/datastores/100/53/backup/disk.0
    - Proposal: 53/backup/disk.0

RESTIC DRIVER
=============
Developed together with this feature is part of the EE edtion.

* It supports the SFTP protocol, the following attributes are
  supported:

  - RESTIC_SFTP_SERVER
  - RESTIC_SFTP_USER: only if different from oneadmin
  - RESTIC_PASSWORD
  - RESTIC_IONICE: Run restic under a given ionice priority (class 2)
  - RESTIC_NICE: Run restic under a given nice
  - RESTIC_BWLIMIT: Limit restic upload/download BW
  - RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes:
    off, auto, max). This requires repositories version 2. By default,
    auto is used (average compression without to much CPU usage)
  - RESTIC_CONNECTIONS: Sets the number of concurrent connections to a
    backend (5 by default). For high-latency backends this number can be
    increased.

* downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name>
  snapshot_id is the restic snapshot hash. To recover single disk images
  from a backup. This URLs support:

  - RESTIC_CONNECTIONS
  - RESTIC_BWLIMIT
  - RESTIC_IONICE
  - RESTIC_NICE

  These options needs to be defined in the associated datastore.

RSYNC DRIVER
=============
A rsync driver is included as part of the CE distribution. It uses the
rsync tool to store backups in a remote server through SSH:

* The following attributes are supported to configure the backup
  datastore:

  - RSYNC_HOST
  - RSYNC_USER
  - RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default)

* downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover
  single files from an existing backup. (RSYNC_HOST and RSYN_USER needs
  to be set in ds_id

EMULATOR_CPUS
=============

This commit includes a non related backup feature:

* Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the
  CPU IDs where the emulator threads will be pinned. If this value is
  not defined the allocated CPU wll be used when using a PIN policy.

(cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b)

F OpenNebula/one#5516: adding rsync backup driver

(cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305)

F OpenNebula/one#5516: update install.sh, add vmid to source, some polish

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d)

F OpenNebula/one#5516: cleanup

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340)

F OpenNebula/one#5516: update downloader, default args, size check

Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305)

LL

(cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
2022-10-07 22:01:37 +02:00

367 lines
10 KiB
C++

/* -------------------------------------------------------------------------- */
/* Copyright 2002-2022, OpenNebula Project, OpenNebula Systems */
/* */
/* Licensed under the Apache License, Version 2.0 (the "License"); you may */
/* not use this file except in compliance with the License. You may obtain */
/* a copy of the License at */
/* */
/* http://www.apache.org/licenses/LICENSE-2.0 */
/* */
/* Unless required by applicable law or agreed to in writing, software */
/* distributed under the License is distributed on an "AS IS" BASIS, */
/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
/* See the License for the specific language governing permissions and */
/* limitations under the License. */
/* -------------------------------------------------------------------------- */
#ifndef _NEBULA_UTIL_H_
#define _NEBULA_UTIL_H_
#include <string>
#include <sstream>
#include <vector>
#include <set>
#include <algorithm>
#include <random>
#include <mutex>
#include <openssl/crypto.h>
namespace one_util
{
std::string& toupper(std::string& st);
std::string& tolower(std::string& st);
/* Case insensitive string compare
* @param str1 First string to compare
* @param str2 Second string to compare
* @return true if the string are equal, false otherwise
*/
bool icasecmp(const std::string& str1, const std::string& str2);
std::string log_time(time_t the_time);
std::string log_time();
/**
* Escape XML entity and character references
* @param in the string to be escaped
* @return a string copy
*/
std::string xml_escape(const std::string& in);
/**
* sha1 digest
* @param in the string to be hashed
* @return sha1 hash of str
*/
std::string sha1_digest(const std::string& in);
/**
* sha256 digest
* @param in the string to be hashed
* @return sha256 hash of str
*/
std::string sha256_digest(const std::string& in);
/**
* AES256 encryption
* @param in the string to encrypt
* @param password to encrypt data
* @return a pointer to the encrypted string (must be freed) or nullptr in case of
* error
*/
std::string * aes256cbc_encrypt(const std::string& in, const std::string& password);
/**
* AES256 decryption
* @param in the base64 string to decrypt
* @param password to decrypt data
* @return a pointer to the decrypted string (must be freed) or nullptr in case of
* error
*/
std::string * aes256cbc_decrypt(const std::string& in, const std::string& password);
/**
* Creates a random number, using time(0) as seed, and performs an sha1 hash
* @return a new random password
*/
std::string random_password();
/**
* Returns random number, default range is <0, Type Max Value>, specialization for integer types
* @param min - minimal potentially generated number, defaults to 0
* @param max - maximal potentially generated number, defaults to type max value
* @return number between min, max
*/
template<typename Integer, typename std::enable_if<std::is_integral<Integer>::value>::type* = nullptr>
Integer random(Integer min = 0, Integer max = std::numeric_limits<Integer>::max())
{
static std::mutex _mutex;
static std::random_device rd;
static std::mt19937_64 rng(rd());
std::uniform_int_distribution<Integer> distribution(min, max);
std::lock_guard<std::mutex> lock(_mutex);
Integer i = distribution(rng);
return i;
}
/**
* Returns random number, default range is <0, Type Max Value>, specialization for floating types
* @param min - minimal potentially generated number, defaults to 0
* @param max - maximal potentially generated number, defaults to type max value
* @return number between min, max
*/
template<typename Floating, typename std::enable_if<std::is_floating_point<Floating>::value>::type* = nullptr>
Floating random(Floating min = 0, Floating max = std::numeric_limits<Floating>::max())
{
static std::mutex _mutex;
static std::random_device rd;
static std::mt19937_64 rng(rd());
std::uniform_real_distribution<Floating> distribution(min, max);
std::lock_guard<std::mutex> lock(_mutex);
Floating f = distribution(rng);
return f;
}
/**
* Splits a string, using the given delimiter
*
* @param st string to split
* @param delim delimiter character
* @param clean_empty true to clean empty split parts.
* Example for st "a::b:c"
* clean_empty true will return ["a", "b", "c"]
* clean_empty fase will return ["a", "", "b", "c"]
*
* @return a vector containing the resulting substrings
*/
template <class T>
void split(const std::string &st, char delim, std::vector<T> &parts)
{
std::string part;
std::stringstream ss(st);
while (getline(ss, part, delim))
{
if (part.empty())
{
continue;
}
std::istringstream iss(part);
T part_t;
iss >> part_t;
if ( iss.fail() )
{
continue;
}
parts.push_back(part_t);
}
}
std::vector<std::string> split(const std::string& st, char delim,
bool clean_empty = true);
/**
* Splits a string, using the given delimiter
*
* @param st string to split
* @param delim delimiter character
* @param result where the result will be saved
*/
template <class T>
void split_unique(const std::string& st, char delim, std::set<T>& result)
{
T elem;
std::vector<std::string> strings = split(st, delim, true);
for (const auto& str : strings)
{
std::istringstream iss(str);
iss >> elem;
if ( iss.fail() )
{
continue;
}
result.insert(elem);
}
}
/**
* Explicit specialization for strings
*/
template <>
void split_unique(const std::string& st, char delim,
std::set<std::string>& result);
/**
* Joins the given element with the delimiter
*
* @param first iterator
* @param last iterator
* @param delim delimiter character
* @return the joined strings
*/
template <class Iterator>
std::string join(Iterator first, Iterator last, char delim)
{
std::ostringstream oss;
for (Iterator it = first; it != last; it++)
{
if (it != first)
{
oss << delim;
}
oss << *it;
}
return oss.str();
}
/**
* Joins the given element with the delimiter
*
* @param values set of values
* @param delim delimiter character
* @return the joined strings
*/
template <class T>
std::string join(const std::set<T>& values, char delim)
{
return join(values.begin(), values.end(), delim);
}
/**
* Creates a string from the given float, using fixed notation. If the
* number has any decimals, they will be truncated to 2.
*
* @param num
* @return
*/
std::string float_to_str(const float &num);
/**
* Returns a scaped version of a value in the from "<op><val><cl>"
* @param v the value to be escaped
* @param op the opening escape string
* @param cl the closing escape string
*/
template <typename ValueType> inline
std::string escape(const ValueType& v, const char * op, const char * cl)
{
std::ostringstream oss;
oss << op << v << cl;
return oss.str();
}
template <typename ValueType> inline
std::string escape_xml(const ValueType &v)
{
return escape(v, "<![CDATA[", "]]>");
}
template <typename ValueType> inline
std::string escape_xml_attr(const ValueType &v)
{
return escape(v, "'", "'");
}
void escape_json(const std::string& str, std::ostringstream& s);
void escape_token(const std::string& str, std::ostringstream& s);
/**
* Checks if a strings matches a regular expression
*
* @param pattern PCRE extended pattern
* @param subject the string to test
* @return 0 on match, another value otherwise
*/
int regex_match(const char *pattern, const char *subject);
/**
* Trim an string using the isspace function
* @param the string
* @return trimed string
*/
std::string trim(const std::string& str);
/**
* Returns a copy of st with the all occurrences of "find" substituted
* for "replacement"
* @param st string input
* @param sfind string to search for
* @param replacement string to replace occurrences with
* @return a string copy
*/
std::string gsub(const std::string& st, const std::string& sfind,
const std::string& replacement);
template <class T>
std::set<T> set_intersection(const std::set<T> &first, const std::set<T>
&second)
{
std::set<T> output;
std::set_intersection(
first.begin(), first.end(), second.begin(), second.end(),
std::inserter(output, output.begin()));
return output;
}
/**
* Generates a new uuid
*/
std::string uuid();
/**
* Reads a generic value from string that supports operator >>.
* @param str Input string
* @param value Numeric value converted from the str, undefined if
* the method fails
* @return true on success, false otherwise
*/
template <class T>
bool str_cast(const std::string str, T& value)
{
std::istringstream iss(str);
iss >> value;
if (iss.fail() || !iss.eof())
{
return false;
}
return true;
}
} // namespace one_util
#endif /* _NEBULA_UTIL_H_ */