IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
We ignored if the cluster state update failed and happily worked with
an empty state, resulting in strange actions, e.g., the removal of
all (not so) "stale" services or changing the all but the masters
node state to unknown.
Check on the update result and if failed, either do not get active,
or, if already active, skip the current round with the knowledge
that we only got here because the update failed but our lock renew
worked => cfs got already in a working and quorate state again -
(probably just a restart)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
We updated the CRM and LRM view of the cluster state only in the PVE2
environment, outside of all regression testing and simulation scope.
Further, we ignored if this update failed and happily worked with an
empty state, resulting in strange actions, e.g., the removal of all
(not so) "stale" services or changing the all but the masters node
state to unknown.
This patch tries to improve this by moving out the update in a own
environment method, cluster_update_state, calling this in the LRM and
CRM and saving its result.
As with our introduced functionallity to simulate cfs rw or update
errors we can also simulate failures of this state update with the RT
system.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
We called them at similar times anyways, and have them under the
regression test cover with this change.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
Mainly addresses a problem where we read the manager status without
catching any possible exceptions.
As this was done only to check if our node has active fencing jobs,
which tells us that it makes no sense to even try to acquire the
manager lock - as we're fenced soon anyway.
Besides this check we always checked if we're quorate and if there
are services configured, so move
both checks in the new 'can_get_active' method, which replaces the
check_pending_fencing and the has_services method.
Move the quorum check in front and catch a possible error from the
following manager status read.
As a side effect the state transition code gets a bit shorter without
hiding the check intention.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
we may get an error here if the cluster filesystem is (temporarily)
unavailable here, this error resulted in stopping the whole CRM
service immediately, which then triggered a node reset (if happened
on the current master), even if we had still time left to retry and
thus, for example, handle a update of pve-cluster gracefully.
Add a method which wraps the status read in an eval and logs an
eventual error, but does not abort the service. Instead we rely on
our get_protected_ha_agent_lock method to detect a problem and switch
to the lost_agent_lock state.
If the pmxcfs outage was really short, so that the manager status
read failed but the lock update worked again we update also always
before doing real work when in the 'active' state. If this update
fails we return from the eval and try next round again, as no point
in doing anything without consistent state.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
Add simulated hardware commands for the cluster file system.
This allows to tell the regression test or simulator system that a
certain nodes calls to methods accessing the CFS should fail, i.e.,
die.
With this we can cover a situation which mainly happen during a
cluster file system update.
For now allow to define if the CFS is read-/writeable (state rw) and
if updates of the CFS (state update) should work or fail.
Add 'can read/write' assertions all over the relevant methods.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
This was introduced for cleaning up an possible left over systemd
watchdog mux enable link, which is gone for good now.
Then it was extended with trigger targets, as the HA Manager services
now restart when the pve-api-update trigger fires.
As the autogenerated postinst does the same unconditionally for the
pve-ha-lrm.service and pve-ha-crm.service already we may remove it
too.
The only difference is that try-restart is used by the auto generated
script, not reload-or-try-restart, but this does not matter, as the
HA services have currently no reload ability.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This was copied by accident when adding the transitional code for
removing the left over of the systemd managed watchdog mux in
commit f8a3fc80af299e613c21c9b67e29aee8cc807018
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This transitional code was added first with
commit f8a3fc80af299e613c21c9b67e29aee8cc807018
and fixed up with
commit ecc145c9724f056549e5458f17d7714ac8c83459
during Proxmox VE 4.1 and 4.2 to remove the problematic systemd
managed watchdog mux socket.
As each system going for an distribution upgrade must first upgrade
to 4.4, where this gets handled, we can remove it now.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
as it's a static unit dh_systemd_starting it is not possible - but it gets
pulled in and started by pve-ha-crm/pve-ha-lrm anyway.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
otherwise it gets confused and enables pve-ha-crm twice in the postinst.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Wrap those calls to the cfs_read_file method, which may now also die
if there was a grave problem reading the file, into eval in all
methods which are used by the ha services.
The ones only used by API calls or CLI helpers are not wrapped, as
there it can be handled more gracefull (i.e., no watchdog is
running) and further, this is more intended to temporarily workaround
until we handle such an exception explicitly in the services - which
is a bit bigger change, so let's just go back to the old behavior for
now.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The check if a service is configured has precedence over the check if
a service is already processed by the manager.
This fixes a bug where a service could be shown as queued even if he
was meant to be ignored.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
In this state the resource will not get touched by us, all commands
(like start/stop/migrate) go directly to the VM/CT itself and not
through the HA stack.
The resource will not get recovered if its node fails.
Achieve that by simply removing the respective service from the
manager_status service status hash if it is in ignored state.
Add the state also to the test and simulator hardware.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
To ensure that the LRM and CRM services get reloaded when
pve-api-update trigger gets activated.
Important, as we directly use perl API modules from qemu-server,
pve-container, pve-common and really want to avoid to run outdated,
possible problematic or deprecated, code.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we must shut all services down when stopping the LRM for a host
shutdown, this can take longer than 95 seconds and should not
get interrupted to ensure a gracefull poweroff.
The watchdog is still active untill all services got stopped so we
still are safe from a freeze or equivalent failure.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Using the nodename in $mailto is not correct and can lead to mails not
forwarding in restrictive mail server configurations.
Also changes $mailfrom to 'root' instead of 'root@localhost', which
results in postfix appending the proper FQDN there, too. As a result the
Delivered-to header reads something like 'root@host.domain.tld' instead
of 'root@localhost', which is much more informational and more
consistent.
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
when a stopped VM managed by HA got backuped the HA stack
continuously tried to shut it down as check_running returns only if a
PID for the VM exists.
As the VM was locked the shutdown tries were blocked, but still a lot
of annoying messages and task spawns happened during the backup
period.
As querying the VM status through the vm monitor is not cheap, check
if the VM is locked with the backup lock first, the config is cached
and so this is quite cheap, only then query the VMs status over qmp,
and check if the VM is in the 'prelaunch' state.
This state gets only set if KVM was started with the `-S` option and
has not yet continued guest operation.
Some performance results, I repeated each check 1000 times, first
number is the total time spent just with the check, second time is
the the time per single check:
old check (vm runs): 87.117 ms/total => 87.117 us/loop
new check (runs, no backup): 107.744 ms/total => 107.744 us/loop
new check (runs, backup): 760.337 ms/total => 760.337 us/loop
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Without syncing the journal could loose logs for a small interval (ca
10-60 seconds), but these last seconds are really interesting for
analyzing the cause of a triggered watchdog.
Also without this often the
> "client did not stop watchdog - disable watchdog updates"
messages wasn't flushed to persistent storage and so some users had a
hard time to figure out why the machine reset.
Use the '--sync' switch of journalctl which - to quote its man page -
"guarantees that any log messages written before its invocation are
safely stored on disk at the time it returns."
Use execl to call `journalctl --sync` in a child process, do not care
for any error checks or recovery as we will be reset anyway. This is
just a hit or miss try to log the situation more consistently, if it
fails we cannot really do anything anyhow.
We call the function on two points:
a) if we exit with active connections, here the watchdog will be
triggered soon and we want to ensure that this is logged.
b) if a client closes the connection without sending the magic close
byte, here the watchdog would trigger while we hang in epoll at
the beginning of the loop, so sync the log here also.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Commit 61ae38eb6fc5ab351fb61f2323776819e20538b7 which ensured that
services get freezed on a node reboot had a side effect where running
services did not get gracefully shutdown on node reboot.
This may lead to data loss as the services then get hard killed, or
they may even prevent a node reboot because a storage cannot get
unmounted as a service still access it.
This commits addresses this issue but does not changes behavior of
the freeze logic for now, but we should evaluate if a freeze makes
really sense here or at least make it configurable.
The changed regression test is a result of the fact that we did not
adapt the correct behavior for the is_node_shutdown command in the
problematic commit. The simulation envrionment returned true
everytime a node shutdown (reboot and poweroff) and the real world
environment just returned true if a poweroff happened but not on a
reboot.
Now the simulation acts the same way as the real environment.
Further I moved the simulation implemenentation to the base class so
that both simulator and regression test system behave the same.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
It is important that all storages stop after pve-ha-lrm.
If the storages stop too early the vm loses disks and can not shutdown.
This can end in fencing the node.
If a service is in error state the single state change command that
can make sense is setting the disabled request state.
Thus abort on all other commands early to enhance user experience.
Change the old enabled/disabled GTK "Switch" element to a ComboBox
one and add all possible service states, so we can simulate the real
world behaviour with its new states better.
As we do not need to map a the boolean swicth value to our states
anymore, we may drop the set_setvice_state method from the RTHardware
class and use the one from the Hardware base class instead.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Most things done by sim_hardware_cmd are already abstracted and
available in both, the TestHardware and the RTHardware class.
Abstract out the CRM and LRM control to allow the unification of both
classes sim_hardware_cmd.
As in the last year mostly the regression test systems TestHardware
class saw new features use it as base.
We return now the current status out of the locked context, this
allows to update the simulators GUI out of the locked context.
This changes increases the power of the HA Simulator, but the new
possible actions must be still implemented in its GUI. This will be
done in future patches.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Do not allocate the HA Environment every time we fork a new CRM or
LRM, but once at the start of the Simulator for all nodes.
This can be done as the Env does not saves any state and thus can be
reused, we use this also in the TestHardware class.
Making the behavior of both Hardware classes more similar allows us
to refactor out some common code in following commits.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
