5
0
mirror of git://git.proxmox.com/git/proxmox-backup.git synced 2024-12-22 13:34:16 +03:00
Go to file
Stefan Lendl 160948c9c1 tests: simple tests for writing the network config
Simple tests for manual and static configurations.

Signed-off-by: Stefan Lendl <s.lendl@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Folke Gleumes <f.gleumes@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-24 21:48:45 +02:00
.cargo cargo: switch to use packaged crates by default 2020-01-03 09:40:33 +01:00
debian d/control: record proxmox-notify build dependency 2024-04-24 10:46:52 +02:00
docs installation: add section about unattended/automatic installation 2024-04-24 21:39:09 +02:00
etc etc/pbs-enterprise.list: change to bookworm 2023-06-26 22:12:53 +02:00
examples client: backup writer: only borrow http client 2024-04-04 10:51:45 +02:00
pbs-api-types realm sync: add sync job for AD realms 2024-04-24 21:06:14 +02:00
pbs-buildcfg workspace: inherit metadata 2022-12-12 09:05:27 +01:00
pbs-client client: backup writer: fix minor formatting issue 2024-04-21 13:28:38 +02:00
pbs-config tests: simple tests for writing the network config 2024-04-24 21:48:45 +02:00
pbs-datastore api: assert that maintenance mode transitions are valid 2024-04-22 12:19:22 +02:00
pbs-fuse-loop sort dependencies 2022-12-12 09:08:56 +01:00
pbs-key-config move pbs_config::key_config to pbs-key-config 2022-12-12 14:19:52 +01:00
pbs-pxar-fuse pxar-fuse: use ReplyBufState::is_full() when possible 2024-03-25 17:17:12 +01:00
pbs-tape access first element with first() rather than get(0) 2024-02-13 10:10:19 +01:00
pbs-tools proxmox-backup-mgr: gc jobs: pretty-print bytes/duration/timestamps 2024-04-22 13:58:08 +02:00
proxmox-backup-banner switch regular dependencies to workspace ones 2022-12-12 09:07:12 +01:00
proxmox-backup-client client: fix whitespace issue 2024-04-21 13:28:38 +02:00
proxmox-file-restore accept a ref to a HttpClient 2023-11-25 17:07:42 +01:00
proxmox-restore-daemon fix #4975: client: ignore E2BIG error flag 2024-02-15 10:34:10 +01:00
pxar-bin fix #4975: client: ignore E2BIG error flag 2024-02-15 10:34:10 +01:00
src api: tape: don't allow overwriting of ids in changer/drive config 2024-04-24 21:40:36 +02:00
templates server: notifications: send tape notifications via notification system 2024-04-23 23:14:46 +02:00
tests tests: add oneOf schema support 2024-02-02 15:07:38 +01:00
www ui: notifications: pull in UX improvements for match rules creation 2024-04-23 23:14:46 +02:00
zsh-completions zsh: fix completions 2021-09-03 10:29:48 +02:00
.gitignore gitignore: generally ignore generated systemd service files 2024-03-08 08:00:30 +01:00
Cargo.toml bump version to 3.2.0-1 2024-04-23 23:47:30 +02:00
defines.mk docs: add datastore.cfg.5 man page 2021-02-10 11:05:02 +01:00
Makefile buildsys: install templates for test notifications 2024-04-23 23:14:46 +02:00
README.rst docs: updated README.rst build guide 2023-08-08 11:48:50 +02:00
rustfmt.toml bump edition in rustfmt.toml 2022-10-13 15:01:11 +02:00
TODO.rst tape: add/use rust scsi changer implementation using libsgutil2 2021-01-25 13:14:07 +01:00

Build & Release Notes
*********************

``rustup`` Toolchain
====================

We normally want to build with the ``rustc`` Debian package. To do that
you can set the following ``rustup`` configuration:

    # rustup toolchain link system /usr
    # rustup default system


Versioning of proxmox helper crates
===================================

To use current git master code of the proxmox* helper crates, add::

   git = "git://git.proxmox.com/git/proxmox"

or::

   path = "../proxmox/proxmox"

to the proxmox dependency, and update the version to reflect the current,
pre-release version number (e.g., "0.1.1-dev.1" instead of "0.1.0").


Local cargo config
==================

This repository ships with a ``.cargo/config`` that replaces the crates.io
registry with packaged crates located in ``/usr/share/cargo/registry``.

A similar config is also applied building with dh_cargo. Cargo.lock needs to be
deleted when switching between packaged crates and crates.io, since the
checksums are not compatible.

To reference new dependencies (or updated versions) that are not yet packaged,
the dependency needs to point directly to a path or git source (e.g., see
example for proxmox crate above).


Build
=====
on Debian 12 Bookworm

Setup:
  1. # echo 'deb http://download.proxmox.com/debian/devel/ bookworm main' | sudo tee /etc/apt/sources.list.d/proxmox-devel.list
  2. # sudo wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
  3. # sudo apt update
  4. # sudo apt install devscripts debcargo clang
  5. # git clone git://git.proxmox.com/git/proxmox-backup.git
  6. # cd proxmox-backup; sudo mk-build-deps -ir

Note: 2. may be skipped if you already added the PVE or PBS package repository

You are now able to build using the Makefile or cargo itself, e.g.::

  # make deb
  # # or for a non-package build
  # cargo build --all --release

Design Notes
************

Here are some random thought about the software design (unless I find a better place).


Large chunk sizes
=================

It is important to notice that large chunk sizes are crucial for performance.
We have a multi-user system, where different people can do different operations
on a datastore at the same time, and most operation involves reading a series
of chunks.

So what is the maximal theoretical speed we can get when reading a series of
chunks? Reading a chunk sequence need the following steps:

- seek to the first chunk's start location
- read the chunk data
- seek to the next chunk's start location
- read the chunk data
- ...

Lets use the following disk performance metrics:

:AST: Average Seek Time (second)
:MRS: Maximum sequential Read Speed (bytes/second)
:ACS: Average Chunk Size (bytes)

The maximum performance you can get is::

  MAX(ACS) = ACS /(AST + ACS/MRS)

Please note that chunk data is likely to be sequential arranged on disk, but
this it is sort of a best case assumption.

For a typical rotational disk, we assume the following values::

  AST: 10ms
  MRS: 170MB/s

  MAX(4MB)  = 115.37 MB/s
  MAX(1MB)  =  61.85 MB/s;
  MAX(64KB) =   6.02 MB/s;
  MAX(4KB)  =   0.39 MB/s;
  MAX(1KB)  =   0.10 MB/s;

Modern SSD are much faster, lets assume the following::

  max IOPS: 20000 => AST = 0.00005
  MRS: 500Mb/s

  MAX(4MB)  = 474 MB/s
  MAX(1MB)  = 465 MB/s;
  MAX(64KB) = 354 MB/s;
  MAX(4KB)  =  67 MB/s;
  MAX(1KB)  =  18 MB/s;


Also, the average chunk directly relates to the number of chunks produced by
a backup::

  CHUNK_COUNT = BACKUP_SIZE / ACS

Here are some staticics from my developer worstation::

  Disk Usage:       65 GB
  Directories:   58971
  Files:        726314
  Files < 64KB: 617541

As you see, there are really many small files. If we would do file
level deduplication, i.e. generate one chunk per file, we end up with
more than 700000 chunks.

Instead, our current algorithm only produce large chunks with an
average chunks size of 4MB. With above data, this produce about 15000
chunks (factor 50 less chunks).