Commit Graph

260 Commits

Author SHA1 Message Date
Dominik Csapak
2ce3e5fb78 rest-server: use hypers AddrIncoming for proxmox-backup-api
this has a 'from_listener' (tokio::net::TcpListener) since hyper 0.14.5 in
the 'tcp' feature (we use 'full', which includes that; since 0.14.13
it is not behind a feature flag anymore).

this makes it possible to create a hyper server without our
'StaticIncoming' wrapper and thus makes it unnecessary.

The only other thing we have to do is to change the Service impl from
tokio::net::TcpStream to hyper::server::conn::AddStream to fulfill the trait
requirements.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-29 09:38:40 +02:00
Dietmar Maurer
48ba0a2dd5 ExtJsFormatter: use ParameterError to correctly compute 'errors'
By default, 'errors' is now empty.

Depend on proxmox 0.13.5.
2021-09-28 10:19:55 +02:00
Dietmar Maurer
962553d252 proxmox-rest-server: cleanup formatter, improve docs
Use trait for OutputFormatter. This is functionally equivalent,
but more rust-like...
2021-09-28 07:45:50 +02:00
Dietmar Maurer
6e50c7aac3 WorkerTaskContext: add shutdown_requested() and fail_on_shutdown() 2021-09-24 12:04:31 +02:00
Dietmar Maurer
42dae7e1fb cleanup WorkerTaskContext 2021-09-24 11:39:30 +02:00
Dietmar Maurer
020c8e6980 cleanup worker task logging
In order to avoid name conflicts with WorkerTaskContext

- renamed WorkerTask::log to WorkerTask::log_message

Note: Methods have different fuction signatures

Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.

Use the task_log!() and task_warn!() macros more often.
2021-09-24 10:34:11 +02:00
Dietmar Maurer
7a4bb6000e rename TaskState to WorkerTaskContext 2021-09-24 10:33:49 +02:00
Dietmar Maurer
85ec987a48 move src/server/h2service.rs into proxmox-rest-server crate 2021-09-24 10:28:17 +02:00
Dietmar Maurer
e8c124fe1b move worker_task.rs into proxmox-rest-server crate
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
2021-09-24 10:28:17 +02:00
Dietmar Maurer
eb1f23c588 use UPID and systemd helpers from proxmox 0.13.4 2021-09-23 12:01:43 +02:00
Dietmar Maurer
1d60abf9f1 move src/server/rest.rs to proxmox-rest-server crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
7c1bd58a8a rest server: cleanup auth-log handling
Handle auth logs the same way as access log.
- Configure with ApiConfig
- CommandoSocket command to reload auth-logs "api-auth-log-reopen"

Inside API calls, we now access the ApiConfig using the RestEnvironment.

The openid_login api now also logs failed logins and return http_err!(UNAUTHORIZED, ..)
on failed logins.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
48b7a61a21 rest server: do not use pbs_api_types::Authid
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
2ea6f8d01d rest server: return UserInformation from ApiAuth::check_auth
This need impl UserInformation for Arc<CachedUserInfo> which is implemented
with proxmox 0.13.2

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
efeccc11cc make get_index and ApiConfig property (callback)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
3d73529460 rest server: simplify get_index() method signature
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
cc67441662 move normalize_uri_path and extract_cookie to proxmox-rest-server crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
fbe0de85d0 move src/tools/compression.rs to proxmox-rest-server crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
dc28aa1ae7 move src/server/formatter.rs to proxmox-rest-server crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
ba04dfb9b2 move src/server/environment.rs to proxmox-rest-server crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
51d84f9847 move src/tools/daemon.rs to proxmox-rest-server workspace
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
ca7a26166f move ApiConfig, FileLogger and CommandoSocket to proxmox-rest-server workspace
ApiConfig: avoid using  pbs_config::backup_user()
CommandoSocket: avoid using  pbs_config::backup_user()
FileLogger: avoid using  pbs_config::backup_user()
- use atomic_open_or_create_file()

Auth Trait: moved definitions to proxmox-rest-server/src/lib.rs
- removed CachedUserInfo patrameter
- return user as String (not Authid)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
2e426f9df2 start new proxmox-rest-server workspace
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
c983a1d623 avoid type re-exports 2021-09-14 08:35:43 +02:00
Dietmar Maurer
908d87f6cb more api type cleanups: avoid re-exports 2021-09-10 12:25:32 +02:00
Dietmar Maurer
bc6ad1dfce move user configuration to pbs_config workspace
Also moved memcom.rs and cached_user_info.rs
2021-09-10 07:09:04 +02:00
Dietmar Maurer
93dfd7d8e9 start new pbs-config workspace
moved src/config/domains.rs
2021-09-02 12:58:20 +02:00
Wolfgang Bumiller
716101d660 move channel/stream helpers to pbs-tools
pbs_tools
  ::blocking: std/async wrapping with block_in_place
  ::stream: stream <-> AsyncRead/AsyncWrite wrapping

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-20 11:27:40 +02:00
Wolfgang Bumiller
e8349f3d2e move client to pbs-client subcrate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 12:58:43 +02:00
Thomas Lamprecht
1c81fc4cc1 rest: log response: avoid unnecessary mut on variable
a match expresses the fallback slightly nicer and needs no mut,
which is always nice to avoid.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-11 13:05:19 +02:00
Wolfgang Bumiller
223db419a4 test fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-07 12:17:10 +02:00
Wolfgang Bumiller
6759d3ebaf split out pbs-buildcfg module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-06 12:00:14 +02:00
Thomas Lamprecht
7bed6d5c31 REST: set error message extenesion for bad-request response log
We send it already to the user via the response body, but the
log_response does not has, nor wants to have FWIW, access to the
async body stream, so pass it through the ErrorMessageExtension
mechanism like we do else where.

Note that this is not only useful for PBS API proxy/daemon but also
the REST server of the file-restore daemon running inside the restore
VM, and it really is *very* helpful to debug things there..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-03 21:34:03 +02:00
Thomas Lamprecht
3c2492f0ff REST: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-03 21:34:03 +02:00
Wolfgang Bumiller
fa8194e0bb refactor send_command
- refactor the combinators,
- make it take a `&T: Serialize` instead of a Value, and
  allow sending the raw string via `send_raw_command`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-05-11 16:28:08 +02:00
Thomas Lamprecht
610b147ba7 server/rest: fix new type ambiguity
basically the same as commit 5fb15b781e
Will be required once we get to use a newer rustc, at least the
client build for archlinux was broken due to this.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-22 21:24:44 +02:00
Thomas Lamprecht
ac49fbd472 enable tape backup by default
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-12 12:31:56 +02:00
Stefan Reiter
c80251ecca server/rest: add ApiAuth trait to make user auth generic
This allows switching the base user identification/authentication method
in the rest server. Will initially be used for single file restore VMs,
where authentication is based on a ticket file, not the PBS user
backend (PAM/local).

To avoid putting generic types into the RestServer type for this, we
merge the two calls "extract_auth_data" and "check_auth" into a single
one, which can use whatever type it wants internally.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 13:57:57 +02:00
Thomas Lamprecht
1b2a34166d server: rest: collapse nested if for less indentation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-07 17:57:46 +02:00
Thomas Lamprecht
27fca61f1b server: rest: switch from fastest to default deflate compression level
I made some comparision with bombardier[0], the one listed here are
30s looped requests with two concurrent clients:

[ static download of ext-all.js ]:
  lvl                              avg /   stdev  / max
 none        1.98 MiB  100 %    5.17ms /  1.30ms / 32.38ms
 fastest   813.14 KiB   42 %   20.53ms /  2.85ms / 58.71ms
 default   626.35 KiB   30 %   39.70ms /  3.98ms / 85.47ms

[ deterministic (pre-defined data), but real API call ]:
  lvl                              avg /   stdev  / max
 none      129.09 KiB  100 %    2.70ms / 471.58us / 26.93ms
 fastest    42.12 KiB   33 %    3.47ms / 606.46us / 32.42ms
 default    34.82 KiB   27 %    4.28ms / 737.99us / 33.75ms

The reduction is quite better with default, but it's also slower, but
only when testing over unconstrained network. For real world
scenarios where compression actually matters, e.g., when using a
spotty train connection, we will be faster again with better
compression.

A GPRS limited connection (Firefox developer console) requires the
following load (until the DOMContentLoaded event triggered) times:
  lvl        t      x faster
 none      9m 18.6s   x 1.0
 fastest   3m 20.0s   x 2.8
 default   2m 30.0s   x 3.7

So for worst case using sligthly more CPU time on the server has a
tremendous effect on the client load time.

Using a more realistical example and limiting for "Good 2G" gives:

 none      1m  1.8s   x 1.0
 fastest      22.6s   x 2.7
 default      16.6s   x 3.7

16s is somewhat OK, >1m just isn't...

So, use default level to ensure we get bearable load times on
clients, and if we want to improve transmission size AND speed then
we could always use a in-memory cache, only a few MiB would be
required for the compressable static files we server.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-07 17:57:42 +02:00
Dominik Csapak
692de3a903 server/rest: compress static files
compress them on the fly
and refactor the size limit for chunking files

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
376d90679c server/rest: compress api calls
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
c2a1c87039 server/rest: add helper to extract compression headers
for now we only extract 'deflate'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
a06d3892fe tools/compression: add DeflateEncoder and helpers
implements a deflate encoder that can compress anything that implements
AsyncRead + Unpin into a file with the helper 'compress'

if the inner type is a Stream, it implements Stream itself, this way
some streaming data can be streamed compressed

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
d28249c290 tools: add compression module
only contains a basic enum for the different compresssion methods

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Thomas Lamprecht
7334107fae server/rest: drop now unused imports
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-01 11:53:13 +02:00
Stefan Reiter
25a14e8eb6 server/rest: extract auth to seperate module
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-01 11:26:28 +02:00
Thomas Lamprecht
5fb15b781e server/rest: fix type ambiguity
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-29 12:02:30 +02:00
Thomas Lamprecht
45a652c26f server/rest: rust format
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-29 08:17:26 +02:00
Dietmar Maurer
9dbc5f47ca ui: enable experimental tape UI if tape.cfg exists 2021-03-03 09:02:02 +01:00