I reordered the component list to match chronological order: we first install an entry, then boot it, then the checks happen, etc. Before it was ordered by "importance", but that is harder to follow.
9.9 KiB
title | category | layout | SPDX-License-Identifier |
---|---|---|---|
Automatic Boot Assessment | Booting | default | LGPL-2.1-or-later |
Automatic Boot Assessment
systemd provides support for automatically reverting back to the previous version of the OS or kernel in case the system consistently fails to boot. The Boot Loader Specification describes how to annotate boot loader entries with a counter that specifies how many attempts should be made to boot it. This document describes how systemd implements this scheme.
The many different components involved in the implementation may be used independently and in combination with other software to for example support other boot loaders or take actions outside of the boot loader.
Here's a brief overview of the complete set of components:
-
The
kernel-install(8)
script can optionally create boot loader entries that carry an initial boot counter (the initial counter is configurable in/etc/kernel/tries
). -
The
systemd-boot(7)
boot loader optionally maintains a per-boot-loader-entry counter described by the Boot Loader Specification that is decreased by one on each attempt to boot the entry, prioritizing entries that have non-zero counters over those which already reached a counter of zero when choosing the entry to boot. -
The
boot-complete.target
target unit (seesystemd.special(7)
) serves as a generic extension point both for units that are necessary to consider a boot successful (e.g.systemd-boot-check-no-failures.service
described below), and units that want to act only if the boot is successful (e.g.systemd-bless-boot.service
described below). -
The
systemd-boot-check-no-failures.service(8)
service is a simple service health check tool. When enabled it becomes an indirect dependency ofsystemd-bless-boot.service
(by means ofboot-complete.target
, see below), ensuring that the boot will not be considered successful if there are any failed services. -
The
systemd-bless-boot.service(8)
service automatically marks a boot loader entry, for which boot counting as mentioned above is enabled, as "good" when a boot has been determined to be successful, thus turning off boot counting for it. -
The
systemd-bless-boot-generator(8)
generator automatically pulls insystemd-bless-boot.service
when use ofsystemd-boot
with boot counting enabled is detected.
Details
As described in Boot Loader Specification,
the boot counting data is stored in the file name of the boot loader entries as
a plus (+
), followed by a number, optionally followed by -
and another
number, right before the file name suffix (.conf
or .efi
).
The first number is the "tries left" counter encoding how many attempts to boot this entry shall still be made. The second number is the "tries done" counter, encoding how many failed attempts to boot it have already been made. Each time a boot loader entry marked this way is booted the first counter is decremented, and the second one incremented. (If the second counter is missing, then it is assumed to be equivalent to zero.) If the boot attempt completed successfully the entry's counters are removed from the name (entry state "good"), thus turning off boot counting for the future.
Walkthrough
Here's an example walkthrough of how this all fits together.
-
The user runs
echo 3 >/etc/kernel/tries
to enable boot counting. -
A new kernel is installed.
kernel-install
is used to generate a new boot loader entry file for it. Let's say the version string for the new kernel is4.14.11-300.fc27.x86_64
, a new boot loader entry/boot/loader/entries/4.14.11-300.fc27.x86_64+3.conf
is hence created. -
The system is booted for the first time after the new kernel has been installed. The boot loader now sees the
+3
counter in the entry file name. It hence renames the file to4.14.11-300.fc27.x86_64+2-1.conf
indicating that at this point one attempt has started. After the rename completed, the entry is booted as usual. -
Let's say this attempt to boot fails. On the following boot the boot loader will hence see the
+2-1
tag in the name, and hence rename the entry file to4.14.11-300.fc27.x86_64+1-2.conf
, and boot it. -
Let's say the boot fails again. On the subsequent boot the loader hence will see the
+1-2
tag, and rename the file to4.14.11-300.fc27.x86_64+0-3.conf
and boot it. -
If this boot also fails, on the next boot the boot loader will see the tag
+0-3
, i.e. the counter reached zero. At this point the entry will be considered "bad", and ordered after all non-bad entries. The next newest boot entry is now tried, i.e. the system automatically reverted to an earlier version.
The above describes the walkthrough when the selected boot entry continuously fails. Let's have a look at an alternative ending to this walkthrough. In this scenario the first 4 steps are the same as above:
-
as above
-
as above
-
as above
-
as above
-
Let's say the second boot succeeds. The kernel initializes properly, systemd is started and invokes all generators.
-
One of the generators started is
systemd-bless-boot-generator
which detects that boot counting is used. It hence pullssystemd-bless-boot.service
into the initial transaction. -
systemd-bless-boot.service
is ordered after andRequires=
the genericboot-complete.target
unit. This unit is hence also pulled into the initial transaction. -
The
boot-complete.target
unit is ordered after and pulls in various units that are required to succeed for the boot process to be considered successful. One such unit issystemd-boot-check-no-failures.service
. -
The graphical desktop environment installed on the machine starts a service called
graphical-session-good.service
, which is also ordered beforeboot-complete.target
, that registers a D-Bus endpoint. -
systemd-boot-check-no-failures.service
is run after all its own dependencies completed, and assesses that the boot completed successfully. It hence exits cleanly. -
graphical-session-good.service
waits for a user to log in. In the user desktop environment, one minute after the user has logged in and started the first program, a user service is invoked which makes a D-Bus call tographical-session-good.service
. Upon receiving that call,graphical-session-good.service
exits cleanly. -
This allows
boot-complete.target
to be reached. This signifies to the system that this boot attempt shall be considered successful. -
Which in turn permits
systemd-bless-boot.service
to run. It now determines which boot loader entry file was used to boot the system, and renames it dropping the counter tag. Thus4.14.11-300.fc27.x86_64+1-2.conf
is renamed to4.14.11-300.fc27.x86_64.conf
. From this moment boot counting is turned off for this entry. -
On the following boot (and all subsequent boots after that) the entry is now seen with boot counting turned off, no further renaming takes place.
How to adapt this scheme to other setups
Of the stack described above many components may be replaced or augmented. Here are a couple of recommendations.
-
To support alternative boot loaders in place of
systemd-boot
two scenarios are recommended:a. Boot loaders already implementing the Boot Loader Specification can simply implement the same rename logic, and thus integrate fully with the rest of the stack.
b. Boot loaders that want to implement boot counting and store the counters elsewhere can provide their own replacements for
systemd-bless-boot.service
andsystemd-bless-boot-generator
, but should continue to useboot-complete.target
and thus support any services ordered before that. -
To support additional components that shall succeed before the boot is considered successful, simply place them in units (if they aren't already) and order them before the generic
boot-complete.target
target unit, combined withRequires=
dependencies from the target, so that the target cannot be reached when any of the units fail. You may add any number of units like this, and only if they all succeed the boot entry is marked as good. Note that the target unit shall pull in these boot checking units, not the other way around.Depending on the setup, it may be most convenient to pull in such units through normal enablement symlinks, or during early boot using a
generator
, or even during later boot. In the last case, care must be taken to ensure that the start job is created beforeboot-complete.target
has been reached. -
To support additional components that shall only run on boot success, simply wrap them in a unit and order them after
boot-complete.target
, pulling it in.
FAQ
-
I have a service which — when it fails — should immediately cause a reboot. How does that fit in with the above? — That's orthogonal to the above, please use
FailureAction=
in the unit file for this. -
Under some condition I want to mark the current boot loader entry as bad right-away, so that it never is tried again, how do I do that? — You may invoke
/usr/lib/systemd/systemd-bless-boot bad
at any time to mark the current boot loader entry as "bad" right-away so that it isn't tried again on later boots.