A power cut, an empty CMOS battery and two purple screens on a Dell PowerEdge R640 running ESXi 9
TL;DR – A Secure Boot PSOD listing dozens of “unsigned” VIBs is not always a broken image. Read the timestamp in the error first. If it shows a date decades in the past, your RTC/CMOS battery is dead and the fix starts in the BIOS, not in ESXi. And if a second PSOD follows (“A security violation was detected”), the ESXi encryption recovery key is what gets you out – so make sure you have it before you need it.
The setup
| Component | Value |
|---|---|
| Server | Dell PowerEdge R640, 2 × Xeon Gold 6240, 512 GB RAM |
| Hypervisor | VMware ESXi 9.0.2 (build 25148076) |
| Firmware | UEFI boot, Secure Boot enabled, TPM 2.0 enabled |
| Role | Host being prepared for a vSAN ESA cluster |
After a power cut, the host came back up with a purple screen.
PSOD #1 – “UEFI Secure Boot failed”
The relevant part, as text:
The system has found a problem on your machine and cannot continue.
UEFI Secure Boot failed at time 1998-01-10T00:54:12.967Z:
Failed to verify signatures of the following vib(s): [atlantic bcm-mpi3 bmcal bnxtnet
... esx-base esx-update ... nsx-* ... vsan vsanhealth vsipfwlib]. All tardisks validatedAt first glance this looks like a seriously broken image: VMware, Intel, Broadcom and NSX VIBs all failing signature checks at once. Two details say otherwise:
- The timestamp: 1998. A server built in 2019 does not legitimately live in 1998.
- “All tardisks validated”. The payloads themselves are intact. It is the signature validity check that fails – and certificates have a validity period.
Secure Boot checks every VIB signature against the current system time. If the hardware clock jumps back decades, the moment is before the signing certificates became valid, so every perfectly good VIB is rejected. Broadcom documents exactly this failure: a depleted CMOS battery resets the hardware clock to a factory default date, and Secure Boot rejects the VIB signatures because that date predates the certificates’ validity period (KB 445564 – their example shows 2000-01-01; mine showed 1998).
Rule of thumb: don’t start removing VIBs or rebuilding images until you’ve read the date in the PSOD.
Fixing the clock
Reboot, press F2 → System Setup → System BIOS → Miscellaneous Settings and set the system date and time. Use UTC – ESXi treats the hardware clock as UTC.
KB 445564 lists three resolutions: set the UEFI time manually, temporarily disable Secure Boot so ESXi can boot and correct the time via NTP, and replace the CMOS battery. I’d strongly prefer the first one on a TPM host: changing firmware security settings is exactly what can trigger the next problem.
I corrected the time, left Secure Boot enabled and rebooted.
PSOD #2 – “A security violation was detected”
The PSOD even points straight to the Broadcom KB that explains the fix:
The system has found a problem on your machine and cannot continue.
Unable to restore the system configuration. A security violation was detected.
https://knowledge.broadcom.com/external/article/312109This is a different mechanism. On a host with a TPM 2.0, ESXi encrypts its configuration and seals the key to the TPM and the firmware’s security state. If that state doesn’t match what was sealed, ESXi refuses to decrypt its configuration (Broadcom KB 312109).
Fixing the clock didn’t cause this. The first PSOD happened so early that it hid the second one; once the signatures validated, boot progressed to the next security check and failed there.
How to recover: the encryption recovery key
KB 312109 gives the order of checks:
- Make sure the firmware security settings are what they were before: UEFI boot, Secure Boot enabled, TPM enabled. A power loss with a dead battery, or your own troubleshooting in the BIOS, may have changed something.
- If the settings are correct and it still fails, boot with the recovery key: when the ESXi boot loader appears, press Shift+O (the letter O), append
encryptionRecoveryKey=<your-recovery-key>to the existing boot options and press Enter. ESXi decrypts the configuration and boots. - Once up, re-seal and persist the configuration with
esxcli system settings encryption set --mode=TPMfollowed by/sbin/auto-backup.sh.
Tip: the iDRAC9 virtual console clipboard makes pasting that long key much less painful.
And the catch, straight from Broadcom: if you don’t have the recovery key, the only option is to reinstall ESXi.
What I did
This host was being prepared for a new vSAN ESA cluster, with no workloads or configuration worth saving. I performed a clean ESXi 9.0.2 install and it booted normally. On a production host – especially one in a VCF environment, where a reinstall means decommissioning and recommissioning the host – the recovery key would have saved hours.
Root cause: the RTC/CMOS battery
The hardware clock is kept by the real-time clock (RTC), powered by a coin cell on the system board (a CR2032 on the R640) whenever the server has no AC power. A depleted battery is invisible while the server is plugged in. The moment you lose mains power – a power cut, a PDU swap, a move – the clock resets.
Don’t confuse it with the PERC controller’s cache battery: for this problem only the system board battery matters.
Checking the battery
- iDRAC System Event Log / Lifecycle Log – look for a CMOS battery failure event (Dell’s diagnostics report it as
DIAG0213 – System board CMOS battery failure detected). The absence of an event does not prove the battery is healthy. - A simple power test: set the correct date/time in the BIOS (UTC), shut down normally, remove AC power from both PSUs, wait several minutes, reconnect power, press F2 and check the date. If the clock has reverted to an old default date, replace the battery. It’s a few euros and a five-minute job.
Prevention checklist
Before a Dell host with Secure Boot and TPM goes into production or a vSAN cluster:
[ ] UEFI boot mode, Secure Boot enabled, TPM 2.0 enabled
[ ] BIOS date/time correct and set to UTC
[ ] RTC clock survives complete AC power removal (test above)
[ ] ESXi NTP configured and time verified
[ ] Encryption recovery key retrieved and stored OFF the host
[ ] iDRAC NTP, SEL monitoring and email/SNMP/Redfish alerts configured
[ ] Hosts monitored outside ESXi (iDRAC / OpenManage Enterprise)Get the recovery key on every TPM host – today, while it’s healthy:
esxcli system settings encryption get # Mode: TPM means the config is sealed
esxcli system settings encryption recovery list # store the key in your password vaultA few useful checks once ESXi is running:
date # ESXi system time
esxcli hardware clock get # hardware (RTC) clock
esxcli software vib signature verify # VIB signature statusTwo notes on monitoring:
- NTP cannot help a host that doesn’t boot. ESXi only gets to NTP after Secure Boot has passed – and Secure Boot depends on the RTC.
- iDRAC is independent of ESXi. Configure NTP on the iDRAC too (it has its own clock, which keeps its logs and certificates sane – it does not correct the host BIOS clock), and send its hardware alerts somewhere you actually read. For multiple servers, Dell OpenManage Enterprise centralises this.
Troubleshooting flow
PSOD: “UEFI Secure Boot failed … Failed to verify signatures”
- Timestamp wrong? Stop. Fix the BIOS date/time (UTC), check the iDRAC log for a CMOS battery event, plan a battery replacement. Don’t touch the VIBs.
- Timestamp correct? Now investigate the image: recent upgrades, OEM/async drivers, NSX VIBs, firmware changes, Secure Boot configuration.
PSOD: “Unable to restore the system configuration. A security violation was detected”
- Verify UEFI, Secure Boot and TPM are enabled and unchanged.
- Boot with
encryptionRecoveryKey=via Shift+O. - Re-seal with
esxcli system settings encryption set --mode=TPMand run/sbin/auto-backup.sh. - No recovery key → reinstall (and in VCF: decommission/recommission).
Don’t clear the TPM on a production host without understanding the consequences – it destroys the sealed key for good.
Lessons learned
- Read the timestamp on a Secure Boot PSOD. A wall of failed VIB signatures looks like a software problem; the date tells you it’s hardware.
- Secure Boot makes the hardware clock part of your security chain. It’s not just a BIOS checkbox.
- The first PSOD can hide the second. Fixing one security check lets the boot reach the next one.
- The recovery key is the difference between a reboot and a rebuild. Collect it for every TPM host and keep it off the host.
- Test the RTC battery – a €5 coin cell can take a host out after the next power cut.
And if your PSOD says UEFI Secure Boot failed at time 1998…, look at the clock before you blame ESXi.
References
- Broadcom KB 445564 – PSOD with UEFI Secure Boot failed due to incorrect CMOS/system time
- Broadcom KB 312109 – ESXi boot failures due to system configuration issues
- Broadcom KB 428266 – “A security violation was detected” after reinstallation
- Broadcom KB 316424 – TPM-enabled ESXi host preparation for hardware changes
- Dell KB 000220179 – How to recover the secure ESXi configuration
