mirror of
git://git.proxmox.com/git/pve-docs.git
synced 2025-01-06 13:17:48 +03:00
212 lines
8.4 KiB
Plaintext
212 lines
8.4 KiB
Plaintext
[[chapter_firmware_updates]]
|
|
Firmware Updates
|
|
----------------
|
|
ifdef::wiki[]
|
|
:pve-toplevel:
|
|
endif::wiki[]
|
|
Firmware updates from this chapter should be applied when running {pve} on a
|
|
bare-metal server. Whether configuring firmware updates is appropriate within
|
|
guests, e.g. when using device pass-through, depends strongly on your setup and
|
|
is therefore out of scope.
|
|
|
|
In addition to regular software updates, firmware updates are also important
|
|
for reliable and secure operation.
|
|
|
|
When obtaining and applying firmware updates, a combination of available options
|
|
is recommended to get them as early as possible or at all.
|
|
|
|
The term firmware is usually divided linguistically into microcode (for CPUs)
|
|
and firmware (for other devices).
|
|
|
|
|
|
[[sysadmin_firmware_persistent]]
|
|
Persistent Firmware
|
|
~~~~~~~~~~~~~~~~~~~
|
|
This section is suitable for all devices. Updated microcode, which is usually
|
|
included in a BIOS/UEFI update, is stored on the motherboard, whereas other
|
|
firmware is stored on the respective device. This persistent method is
|
|
especially important for the CPU, as it enables the earliest possible regular
|
|
loading of the updated microcode at boot time.
|
|
|
|
CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the
|
|
device configuration could be reset. Please follow the vendor's instructions
|
|
carefully and back up the current configuration.
|
|
|
|
Please check with your vendor which update methods are available.
|
|
|
|
* Convenient update methods for servers can include Dell's Lifecycle Manager or
|
|
Service Packs from HPE.
|
|
|
|
* Sometimes there are Linux utilities available as well. Examples are
|
|
https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
|
|
ConnectX or
|
|
https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
|
|
for Broadcom network cards.
|
|
|
|
* https://fwupd.org[LVFS] is also an option if there is a cooperation with
|
|
the https://fwupd.org/lvfs/vendors/[hardware vendor] and
|
|
https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical
|
|
requirement for this is that the system was manufactured after 2014 and is
|
|
booted via UEFI.
|
|
|
|
{pve} ships its own version of the `fwupd` package to enable Secure Boot
|
|
Support with the Proxmox signing key. This package consciously dropped the
|
|
dependency recommendation for the `udisks2` package, due to observed issues with
|
|
its use on hypervisors. That means you must explicitly configure the correct
|
|
mount point of the EFI partition in `/etc/fwupd/daemon.conf`, for example:
|
|
|
|
.File `/etc/fwupd/daemon.conf`
|
|
----
|
|
# Override the location used for the EFI system partition (ESP) path.
|
|
EspLocation=/boot/efi
|
|
----
|
|
|
|
TIP: If the update instructions require a host reboot, make sure that it can be
|
|
done safely. See also xref:ha_manager_node_maintenance[Node Maintenance].
|
|
|
|
|
|
[[sysadmin_firmware_runtime_files]]
|
|
Runtime Firmware Files
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
This method stores firmware on the {pve} operating system and will pass it to a
|
|
device if its xref:sysadmin_firmware_persistent[persisted firmware] is less
|
|
recent. It is supported by devices such as network and graphics cards, but not
|
|
by those that rely on persisted firmware such as the motherboard and hard disks.
|
|
|
|
In {pve} the package `pve-firmware` is already installed by default. Therefore,
|
|
with the normal xref:system_software_updates[system updates (APT)], included
|
|
firmware of common hardware is automatically kept up to date.
|
|
|
|
An additional xref:sysadmin_debian_firmware_repo[Debian Firmware Repository]
|
|
exists, but is not configured by default.
|
|
|
|
If you try to install an additional firmware package but it conflicts, APT will
|
|
abort the installation. Perhaps the particular firmware can be obtained in
|
|
another way.
|
|
|
|
|
|
[[sysadmin_firmware_cpu]]
|
|
CPU Microcode Updates
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
Microcode updates are intended to fix found security vulnerabilities and other
|
|
serious CPU bugs. While the CPU performance can be affected, a patched microcode
|
|
is usually still more performant than an unpatched microcode where the kernel
|
|
itself has to do mitigations. Depending on the CPU type, it is possible that
|
|
performance results of the flawed factory state can no longer be achieved
|
|
without knowingly running the CPU in an unsafe state.
|
|
|
|
To get an overview of present CPU vulnerabilities and their mitigations, run
|
|
`lscpu`. Current real-world known vulnerabilities can only show up if the
|
|
{pve} host is xref:system_software_updates[up to date], its version not
|
|
xref:faq-support-table[end of life], and has at least been rebooted since the
|
|
last kernel update.
|
|
|
|
Besides the recommended microcode update via
|
|
xref:sysadmin_firmware_persistent[persistent] BIOS/UEFI updates, there is also
|
|
an independent method via *Early OS Microcode Updates*. It is convenient to use
|
|
and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
|
|
updates. Regardless of the method in use, a reboot is always needed to apply a
|
|
microcode update.
|
|
|
|
|
|
Set up Early OS Microcode Updates
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
To set up microcode updates that are applied early on boot by the Linux kernel,
|
|
you need to:
|
|
|
|
. Enable the xref:sysadmin_debian_firmware_repo[Debian Firmware Repository]
|
|
. Get the latest available packages `apt update` (or use the web interface,
|
|
under Node -> Updates)
|
|
. Install the CPU-vendor specific microcode package:
|
|
- For Intel CPUs: `apt install intel-microcode`
|
|
- For AMD CPUs: `apt install amd64-microcode`
|
|
. Reboot the {pve} host
|
|
|
|
Any future microcode update will also require a reboot to be loaded.
|
|
|
|
Microcode Version
|
|
^^^^^^^^^^^^^^^^^
|
|
To get the current running microcode revision for comparison or debugging
|
|
purposes:
|
|
|
|
----
|
|
# grep microcode /proc/cpuinfo | uniq
|
|
microcode : 0xf0
|
|
----
|
|
|
|
A microcode package has updates for many different CPUs. But updates
|
|
specifically for your CPU might not come often. So, just looking at the date on
|
|
the package won't tell you when the company actually released an update for your
|
|
specific CPU.
|
|
|
|
If you've installed a new microcode package and rebooted your {pve} host, and
|
|
this new microcode is newer than both, the version baked into the CPU and the
|
|
one from the motherboard's firmware, you'll see a message in the system log
|
|
saying "microcode updated early".
|
|
|
|
----
|
|
# dmesg | grep microcode
|
|
[ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
|
|
[ 0.896580] microcode: Microcode Update Driver: v2.2.
|
|
----
|
|
|
|
|
|
[[sysadmin_firmware_troubleshooting]]
|
|
Troubleshooting
|
|
^^^^^^^^^^^^^^^
|
|
For debugging purposes, the set up Early OS Microcode Update applied regularly
|
|
at system boot can be temporarily disabled as follows:
|
|
|
|
1. make sure that the host can be rebooted xref:ha_manager_node_maintenance[safely]
|
|
2. reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
|
|
3. at the desired {pve} boot entry press `E`
|
|
4. go to the line which starts with `linux` and append separated by a space
|
|
*`dis_ucode_ldr`*
|
|
5. press `CTRL-X` to boot this time without an Early OS Microcode Update
|
|
|
|
If a problem related to a recent microcode update is suspected, a package
|
|
downgrade should be considered instead of package removal
|
|
(`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old
|
|
xref:sysadmin_firmware_persistent[persisted] microcode might be loaded, even
|
|
though a more recent one would run without problems.
|
|
|
|
A downgrade is possible if an earlier microcode package version is
|
|
available in the Debian repository, as shown in this example:
|
|
|
|
----
|
|
# apt list -a intel-microcode
|
|
Listing... Done
|
|
intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
|
|
intel-microcode/stable 3.20230512.1 amd64
|
|
----
|
|
----
|
|
# apt install intel-microcode=3.202305*
|
|
...
|
|
Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
|
|
...
|
|
dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
|
|
...
|
|
intel-microcode: microcode will be updated at next boot
|
|
...
|
|
----
|
|
|
|
Make sure (again) that the host can be rebooted
|
|
xref:ha_manager_node_maintenance[safely]. To apply an older microcode
|
|
potentially included in the microcode package for your CPU type, reboot now.
|
|
|
|
[TIP]
|
|
====
|
|
It makes sense to hold the downgraded package for a while and try more recent
|
|
versions again at a later time. Even if the package version is the same in the
|
|
future, system updates may have fixed the experienced problem in the meantime.
|
|
----
|
|
# apt-mark hold intel-microcode
|
|
intel-microcode set on hold.
|
|
----
|
|
----
|
|
# apt-mark unhold intel-microcode
|
|
# apt update
|
|
# apt upgrade
|
|
----
|
|
====
|