38 Commits

Author SHA1 Message Date
Dietmar Maurer
0355554905 proxmox-rrd: use a journal to reduce amount of bytes written
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.

Apply the journal every 30 minutes and on daemon startup.

Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Wolfgang Bumiller
e0ce41b03a update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
Wolfgang Bumiller
fa9757e67f bump proxmox dependency to 0.14.0 and proxmox-http to 0.5.0
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-08 11:18:22 +02:00
Dietmar Maurer
071cb7aa8b proxmox-rrd: use correct directory options in create_rrdb_dir 2021-10-07 08:50:50 +02:00
Dietmar Maurer
9c64c09c92 proxmox-rrd: cleanup error handling 2021-10-07 08:01:12 +02:00
Dietmar Maurer
5165bed8c2 proxmox-rrd: use log crate instead of eprintln, avoid duplicate logs 2021-10-06 18:19:22 +02:00
Dietmar Maurer
9c7fd3c936 proxmox-rrd: fix update (do not update) when time is in the past 2021-10-06 18:01:48 +02:00
Dietmar Maurer
881d8f85ea proxmox-rrd: improve developer docs 2021-10-06 12:19:54 +02:00
Dietmar Maurer
54f7a80f97 proxmox-rrd: remove serde dependency 2021-10-06 10:55:46 +02:00
Dietmar Maurer
538e6f66f3 split out RRD api types into proxmox-rrd-api-types crate 2021-10-06 09:49:51 +02:00
Dietmar Maurer
ac17698e4a proxmox-rrd: use create_path instead of std::fs::create_dir_all
To ensure correct file ownership.
2021-10-06 08:37:14 +02:00
Dietmar Maurer
8d1a9d2ec6 move RRD code into proxmox-rrd crate 2021-10-06 08:13:28 +02:00
Dietmar Maurer
a4e56bff60 more api type cleanups: avoid re-exports 2021-09-10 12:25:32 +02:00
Dietmar Maurer
80f7bf3822 start new pbs-config workspace
moved src/config/domains.rs
2021-09-02 12:58:20 +02:00
Fabian Grünbichler
c94f367dec clippy: more misc fixes
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:55 +01:00
Fabian Grünbichler
655ceac1c6 clippy: collapse/rework nested ifs
no semantic changes (intended).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Wolfgang Bumiller
4d8bd987a4 clippy fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 11:20:07 +02:00
Stefan Reiter
6eff0b289e rrd: fix integer underflow
Causes a panic if last_update is smaller than RRD_DATA_ENTRIES*reso,
which (I believe) can happen when inserting the first value for a DB.

Clamp the value to 0 in that case.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-01 14:30:32 +02:00
Dietmar Maurer
110ceff08c avoid chrono dependency, depend on proxmox 0.3.8
- remove chrono dependency

- depend on proxmox 0.3.8

- remove epoch_now, epoch_now_u64 and epoch_now_f64

- remove tm_editor (moved to proxmox crate)

- use new helpers from proxmox 0.3.8
  * epoch_i64 and epoch_f64
  * parse_rfc3339
  * epoch_to_rfc3339_utc
  * strftime_local

- BackupDir changes:
  * store epoch and rfc3339 string instead of DateTime
  * backup_time_to_string now return a Result
  * remove unnecessary TryFrom<(BackupGroup, i64)> for BackupDir

- DynamicIndexHeader: change ctime to i64

- FixedIndexHeader: change ctime to i64
2020-09-15 07:12:57 +02:00
Dominik Csapak
50fa6045b3 api2/status: use new rrd::extract_cached_data
and drop the now unused extract_lists function

this also fixes a bug, where we did not add the datastore to the list at
all when there was no rrd data

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:16 +02:00
Dominik Csapak
11cb8cd008 rrd: move creation of serde value into api
there is now a 'extract_cached_data' which just returns
the data of the specified field, and an api function that converts
a list of fields to the correct serde value

this way we do not have to create a serde value in rrd/cache.rs
(makes for a better interface)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:14 +02:00
Dominik Csapak
9a860821e4 refactor time functions to tools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:10 +02:00
Dominik Csapak
20ada7a08c rrd: add 'extract_lists'
this is an interface to simply get the Vec<Option<f64>> out of rrd
without going through serde values

we return a list of timestamps and a HashMap with the lists we could find
(otherwise it is not in the map)

if no lists could be extracted, the time list is also empty

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-09 12:19:06 +02:00
Dietmar Maurer
1fa3341d68 rrd: reduce io by saving data only once a minute 2020-05-29 09:16:13 +02:00
Dietmar Maurer
a76b0e3c09 src/rrd/rrd.rs: do not wrap error and return ErrorKind::NotFound 2020-05-25 10:30:04 +02:00
Dietmar Maurer
9ce21e005d src/rrd/cache.rs: display/log error when RRD load fails 2020-05-25 10:18:53 +02:00
Dietmar Maurer
207d0c7714 src/rrd/rrd.rs: store/verify magic number 2020-05-25 09:21:54 +02:00
Dietmar Maurer
bfc2f8b7a2 src/rrd/rrd.rs: implement DST_COUNTER 2020-05-25 08:14:30 +02:00
Dietmar Maurer
d261516f2d src/rrd/rrd.rs: correctly compute derived values
use f64 for time.
2020-05-25 07:02:04 +02:00
Dietmar Maurer
f4561029ba src/rrd/rrd.rs: implement DST_DERIVE 2020-05-24 19:02:35 +02:00
Dietmar Maurer
7acdec12e8 src/rrd/rrd.rs: restructure whole code 2020-05-24 16:51:28 +02:00
Dietmar Maurer
777f527603 src/rrd/rrd.rs: reduce size by using f64:NAN as UNKNOWN 2020-05-24 09:09:09 +02:00
Dietmar Maurer
fd2c16f704 src/rrd/rrd.rs: simplify an fix old value deletion 2020-05-24 06:44:06 +02:00
Dietmar Maurer
6357746f2e rrd: fix display interval, try to avoid numeric errors 2020-05-23 16:03:43 +02:00
Dietmar Maurer
69da9baf59 rrd: simplify code 2020-05-23 15:37:17 +02:00
Dietmar Maurer
e073e5c107 rrd: pack multiple rrd values into th estat list 2020-05-23 14:03:44 +02:00
Dietmar Maurer
1c0117790f add experimental rrd api to get cpu stats 2020-05-23 11:50:53 +02:00
Dietmar Maurer
5bc85d2d04 add simple rrd implementation 2020-05-23 10:42:48 +02:00