IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
traverse the usual error counting mechanisms, as then the
select_service_node helper either picks up the right node and it
starts there or it can trigger fencing of that.
Note, in practice this normally can only happen if the admin
butchered around in the node cluster state, but as we only select the
safe nodes from the configured groups, we should be safe in any case.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
so that we don't need to specify all usage stats explicitly for
bigger tests.
Note, we explicitly use two digits for memory as with just one a lot
of services are exactly the same, which gives us flaky tests due to
rounding, or some flakiness in the rust code - so this is a bit of a
stop gap for that too and should be reduced to a single digit once
we fixed it in the future.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
E.g., pve-ha-manager is our current HA manager, so talking about the
"current HA stack" being EOL without mentioning the actually meant
`rgmanager` one, got taken up the wrong way by some potential users.
Correct that and a few other things, but as there are definitively
stuff still out-of-date, or will be in a few months, mention that
this is an older readme and refer to the HA reference docs at the
top.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Pretty safe to do as we recompute everything per round anyway (and
much more often on top of that, but that's another topic).
Actually I'd argue that it's safer as this way a user doesn't need to
actively restart the manager, which grinds much more gears and
watchdog changes than checking periodically and updating it
internally. Plus, a lot of admins won't expect that they need to
restart the current active master and thus they'll complain that
their recently made change to the CRS config had no effect/the CRS
doesn't work at all.
We should codify such a change in test for this though.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
to hint to a potential "code optimizer" that it may not be easily
moved above to the scheduling selection
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
if something goes wrong with the TOPSIS scoring. Not expected to
happen, but it's rather cheap to be on the safe side.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
With the Usage::Static plugin, scoring is not as cheap anymore and
select_service_node() is called for each running service.
This should cover most calls of select_service_node().
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
With the Usage::Static plugin, scoring is not as cheap anymore and
select_service_node() is called for each running service.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Note that recompute_online_node_usage() becomes much slower when the
'static' resource scheduler mode is used. Tested it with ~300 HA
services (minimal containers) running on my virtual test cluster.
Timings with 'basic' mode were between 0.0004 - 0.001 seconds
Timings with 'static' mode were between 0.007 - 0.012 seconds
Combined with the fact that recompute_online_node_usage() is currently
called very often this can lead to a lot of delay during recovery
situations with hundreds of services and low thousands of services
overall and with genereous estimates even run into the watchdog timer.
Ideas to remedy this is using PVE::Cluster's
get_guest_config_properties() instead of load_config() and/or
optimizing how often recompute_online_node_usage() is called.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
The method will be extended to include other HA-relevant settings from
datacenter.cfg.
Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
for calculating node usage of services based upon static CPU and
memory configuration as well as scoring the nodes with that
information to decide where to start a new or recovered service.
For getting the service stats, it's necessary to also consider the
migration target (if present), becuase the configuration file might
have already moved.
It's necessary to update the cluster filesystem upon stealing the
service to be able to always read the moved config right away when
adding the usage.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
no functional change is intended.
One test needs adaptation too, because it created its own version of
$online_node_usage.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
In preparation for scheduling based on static information, where the
scoring of nodes depends on information from the service's
VM/CT configuration file (and the $sid is required to query that).
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
in preparation to also support static resource scheduling via another
such Usage plugin.
The interface is designed in anticipation of the Usage::Static plugin,
the Usage::Basic plugin doesn't require all parameters.
In Usage::Static, the $haenv will necessary for logging and getting
the static node stats. add_service_usage_to_node() and
score_nodes_to_start_service() take the sid, service node and the
former also the optional migration target (during a migration it's not
clear whether the config file has already been moved or not) to be
able to get the static service stats.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
to be used for static resource scheduling.
In container's vmstatus(), the 'cores' option takes precedence over
the 'cpulimit' one, but it felt more accurate to prefer 'cpulimit'
here.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
to be used for static resource scheduling. In the simulation
environment, the information can be added in hardware_status.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
only count up target selection if that node is already in the online
node usage list, to avoid that a offline node is considered online if
its a target from any command
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
currently wrong as online_node_usage doesn't considers counting the
target node if the source node isn't considered online (=
operational) anymore
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
run_workers is responsible for updating the state after workers have
exited. if the current LRM state is 'active', but a shutdown_request was
issued in 'restart' mode (like on package upgrades), this call is the
only one made in the LRM work() loop.
skipping it if there are active services means the following sequence of
events effectively keeps the LRM from restarting or making any progress:
- start HA migration on node A
- reload LRM on node A while migration is still running
even once the migration is finished, the service count is still >= 1
since the LRM never calls run_workers (directly or via
manage_resources), so the service having been migrated is never noticed.
maintenance mode (i.e., rebooting the node with shutdown policy migrate)
does call manage_resources and thus run_workers, and will proceed once
the last worker has exited.
reported by a user:
https://forum.proxmox.com/threads/lrm-hangs-when-updating-while-migration-is-running.108628
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
every LRM round is scheduled to run for 10s but we spend only half
of that to actively trying to run workers (in the max_worker limit).
Raise that to 80% duty cycle.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
If a setup has a lot VMs we may run into the time limit from the
run_worker loop before processing all workers, which can easily
happen if an admin did not increased their default of max_workers in
the setup, but even with a bigger max_worker setting one can run into
it.
That combined with the fact that we sorted just by the $sid
alpha-numerically means that CTs where preferred over VMs (C comes
before V) and additionally lower VMIDs where preferred too.
That means that a set of SIDs had a lower chance of ever get actually
run, which is naturally not ideal at all.
Improve on that behavior by adding a counter to the queued worker and
preferring those that have a higher one, i.e., spent more time
waiting on getting actively run.
Note, due to the way the stop state is enforced, i.e., always
enqueued as new worker, its start-try counter will be reset every
round and thus have a lower priority compared to other request
states. We probably want to differ between a stop request when the
service is/was in another state just before and the time a stop is
just re-requested even if a service was already stopped for a while.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>