Why vCenter shows no LLDP neighbours on the Minisforum MS-A2 (and how to fix it)

After: vSphere Client on v-lab-esx01, vmnic2 LLDP tab shows Port 8, port description SFP+ 8, system description USW-Pro-Aggregation and system name V-LAB-SWITCH05-Aggregation

One kernel module parameter on the Intel X710 was all it took to get LLDP neighbour information back in vCenter for my Minisforum MS-A2 hosts.

In my V-LAB the MS-A2s connect to a UniFi switch with their two 10G SFP+ ports. Those ports are the uplinks of the VCF distributed switch. Everything worked, but in vCenter the LLDP tab of every X710 uplink stayed empty. No switch name, no port ID, no VLANs.

Before: vSphere Client on v-lab-esx02, vmnic1 LLDP tab says Link Layer Discovery Protocol is not available on this physical network adapter
Before: on v-lab-esx02 the LLDP tab only says that LLDP is not available on this adapter.

That matters more than it looks. LLDP is how you prove that vmnic2 really lands on the switch port you think it does, before you troubleshoot vSAN or NSX. It also shows the ESXi host and vmnic on the UniFi side. In this post I explain why the information was missing and how I fixed it the VCF way, one host at a time.

Root cause: the X710 LLDP agent eats the frames

The Intel X710 has its own LLDP agent in the NIC firmware, and it is enabled by default. That agent takes every incoming LLDP frame (EtherType 0x88CC) for itself, mainly to negotiate DCB. As a result the frames never reach the ESXi network stack, so the distributed switch has nothing to show.

The i40en driver controls this agent with the module parameter LLDP:

ValueMeaning
1 (default)Firmware LLDP agent on, frames are consumed by the NIC
0Firmware LLDP agent off, frames are passed to ESXi

The parameter is a list with one value per i40en port. When it shows no value at all, the driver runs on its default, which is 1.

There is a second, separate cause with the same symptom. A vSphere Distributed Switch listens for CDP by default, and UniFi switches send LLDP, not CDP. So you need both: the NIC must pass the frames, and the VDS must listen for LLDP.

Step 1: set the distributed switch to LLDP

Start with the free fix, because it needs no reboot. In the vSphere Client:

  1. Go to Networking, select the VCF distributed switch and open Configure → Properties.
  2. Click Edit → Advanced → Discovery protocol.
  3. Set Type to Link Layer Discovery Protocol and Operation to Both.

I use Both instead of Listen. ESXi then also advertises itself, so the UniFi port overview shows the host name and vmnic for each port. With NSX on the VDS this one setting covers the NSX uplinks too.

A standard vSwitch only supports CDP. LLDP is a distributed switch feature.

Step 2: check the i40en ports and parameters

SSH to the host and look at two things first: how many i40en ports the host has, and which module parameters are already set.

esxcli network nic list | grep i40en
esxcli system module parameters list -m i40en | grep -i lldp
esxcli system module parameters list -m i40en
esxcfg-module -g i40en

The first command shows the X710 ports; on the MS-A2 that is two. The second shows LLDP with an empty Value column, which means the firmware agent is on. The third lists all i40en parameters, so you can see whether any Value is filled in. The last one prints the option string that is configured for the module right now; on a host with default settings it is empty.

v-lab-esx03 before the change: two X710 ports vmnic1 and vmnic2 on i40en, the LLDP parameter with an empty Value column, and the full i40en parameter list with every Value empty
v-lab-esx03 before the change: two X710 ports (vmnic1 and vmnic2), an empty LLDP value so the firmware agent is on, and no other i40en parameter set either.

That last check is important. esxcli system module parameters set -p replaces the complete parameter string of the module. It does not add one parameter. If you already set something else on i40en, put it in the same string, or it is gone after the reboot.

Step 3: disable the firmware LLDP agent, one host at a time

The driver reads its parameters only when it loads, so every host needs a reboot. In a vSAN cluster I treat that as a normal rolling maintenance:

  1. Check that vSAN Skyline Health is green and that no resync is running.
  2. Put the host in maintenance mode from vCenter with vSAN data migration set to Ensure accessibility.
  3. Set the parameter, one value per i40en port: esxcli system module parameters set -m i40en -p LLDP=0,0. Check it with esxcli system module parameters list -m i40en | grep -i lldp; the Value column now shows 0,0. With four X710 ports it would be LLDP=0,0,0,0.
  4. Reboot the host.
  5. Exit maintenance mode and wait until the vSAN resync is complete.
  6. Repeat on the next MS-A2.
v-lab-esx03: esxcli system module parameters set -m i40en -p LLDP=0,0, followed by the list command showing LLDP with value 0,0
The change on v-lab-esx03: after the set command the LLDP value reads 0,0 for both X710 ports. It becomes active after the reboot.

Count the values carefully. Broadcom documents a case where an invalid LLDP value stopped i40en from loading at all, so the X710 ports disappeared after a reboot (KB 392205). In a VCF cluster those are your vSAN, vMotion and TEP uplinks, so keep the Comet console or another out-of-band path ready before the first reboot.

The setting survives reboots, because it is stored in the host configuration. It does not survive a reinstall, which is why the last section of this post matters.

Step 4: verify X710 LLDP on both sides

First, check on the host that LLDP frames from the UniFi switch now reach ESXi. Replace vmnic2 with your X710 uplink:

pktcap-uw --uplink vmnic2 --ethtype 0x88cc -c 1 -o - | tcpdump-uw -enr -
pktcap-uw on v-lab-esx03 vmnic2 before the reboot: no LLDP frame arrives, the capture is stopped by hand and reports 0 packets dumped, 0 dropped
Before: v-lab-esx03 has the parameter set but has not rebooted yet, so the firmware agent is still active. No LLDP frame reaches vmnic2, and the capture ends with 0 packets.
pktcap-uw on v-lab-esx01 vmnic2 captures one LLDP frame (ethertype 0x88cc) sent to 01:80:c2:00:00:0e by V-LAB-SWITCH05-Aggregation; 1 packet dumped, 0 dropped
After: on v-lab-esx01 one LLDP frame from V-LAB-SWITCH05-Aggregation now arrives on vmnic2. 01:80:c2:00:00:0e is the standard LLDP multicast address. Same command, now with the firmware agent off.

UniFi sends LLDP every 30 seconds by default, so give it a moment. Then look at what ESXi learned from the neighbour:

vim-cmd hostsvc/net/query_networkhint --pnic-name=vmnic2

In the vSphere Client, go to the host → Configure → Networking → Virtual switches and click the info icon next to the physical adapter. The LLDP tab now shows the UniFi switch name, the port ID and the VLANs.

After: vSphere Client on v-lab-esx01, vmnic2 LLDP tab shows Port 8, port description SFP+ 8, system description USW-Pro-Aggregation and system name V-LAB-SWITCH05-Aggregation
After: v-lab-esx01 vmnic2 now sees its neighbour, port 8 (SFP+ 8) on V-LAB-SWITCH05-Aggregation, a UniFi USW-Pro-Aggregation.

Everything you need to check the cabling is in that one screen. The System Name tells you which switch, the Port ID which port, and the System Description even shows the UniFi model and firmware.

On the UniFi side, open the switch in UniFi Network and look at the port list. With the VDS on Both, each port shows the ESXi host and its vmnic. That is the quickest way I know to check that the cabling of every MS-A2 matches the uplink mapping in VCF.

Make the X710 LLDP setting stick

A setting made by hand on one host is exactly the kind of drift VCF tries to avoid. I make sure it comes back in two places.

vSphere Configuration Profiles. When the cluster is managed with configuration profiles, a module parameter set by hand shows up as drift, and a later remediation can put it back to the default. Export the cluster configuration, add the i40en LLDP setting in the kernel module settings, import it and remediate. From then on the profile is the source of truth, also for new hosts.

Kickstart. My MS-A2s are installed with an automated ESXi Kickstart. I added the parameter to the %firstboot section, so a reinstalled host comes up with the agent already off:

%firstboot --interpreter=busybox
esxcli system module parameters set -m i40en -p LLDP=0,0

The parameter becomes active at the next driver load, so let the Kickstart end with a reboot.

One side effect to be aware of: DCB. Data Center Bridging is a set of Ethernet extensions for lossless traffic. Priority Flow Control pauses one traffic class instead of the whole link, and DCBX lets the NIC and the switch agree on those settings. DCBX travels inside LLDP frames, which is exactly why the X710 has its own LLDP agent. You need DCB for RoCE, for example vSAN over RDMA, not for vSAN over TCP. The X710 is not a RoCE NIC and my vSAN runs over TCP, so turning the agent off costs nothing here. If you use DCB on your X710s, keep the agent on and accept that the VDS won’t see the LLDP neighbours.

Wrap-up

Empty LLDP information on an X710 is not a UniFi problem and not a vCenter problem. The NIC simply keeps the frames for itself. Set the VDS to LLDP / Both, set LLDP=0,... on i40en with one value per port, reboot the hosts one by one in maintenance mode, and put the setting in your configuration profile and Kickstart.

The same fix applies to any host with an Intel 700-series NIC, not only the MS-A2. While you are at it, check the X710 firmware too: I covered that in updating Intel 700 Series NIC firmware on ESXi. With LLDP working, the next step for my lab is end-to-end jumbo frame validation between the MS-A2s, which deserves a post of its own.

Sources

When ESXi Secure Boot Fails Because Your Server Thinks It’s 1998

ESXi 9 PSOD on a Dell PowerEdge R640: UEFI Secure Boot failed at time 1998-01-10, failed to verify signatures of many VIBs, all tardisks validated.

A power cut, an empty CMOS battery and two purple screens on a Dell PowerEdge R640 running ESXi 9

TL;DR – A Secure Boot PSOD listing dozens of “unsigned” VIBs is not always a broken image. Read the timestamp in the error first. If it shows a date decades in the past, your RTC/CMOS battery is dead and the fix starts in the BIOS, not in ESXi. And if a second PSOD follows (“A security violation was detected”), the ESXi encryption recovery key is what gets you out – so make sure you have it before you need it.

The setup

Component Value
Server Dell PowerEdge R640, 2 × Xeon Gold 6240, 512 GB RAM
Hypervisor VMware ESXi 9.0.2 (build 25148076)
Firmware UEFI boot, Secure Boot enabled, TPM 2.0 enabled
Role Host being prepared for a vSAN ESA cluster

After a power cut, the host came back up with a purple screen.

PSOD #1 – “UEFI Secure Boot failed”

ESXi 9 PSOD on a Dell PowerEdge R640: UEFI Secure Boot failed at time 1998-01-10, failed to verify signatures of many VIBs, all tardisks validated.
PSOD #1: Secure Boot rejects almost every VIB – note the date.

The relevant part, as text:

The system has found a problem on your machine and cannot continue.

UEFI Secure Boot failed at time 1998-01-10T00:54:12.967Z:
Failed to verify signatures of the following vib(s): [atlantic bcm-mpi3 bmcal bnxtnet
... esx-base esx-update ... nsx-* ... vsan vsanhealth vsipfwlib]. All tardisks validated

At first glance this looks like a seriously broken image: VMware, Intel, Broadcom and NSX VIBs all failing signature checks at once. Two details say otherwise:

  • The timestamp: 1998. A server built in 2019 does not legitimately live in 1998.
  • “All tardisks validated”. The payloads themselves are intact. It is the signature validity check that fails – and certificates have a validity period.

Secure Boot checks every VIB signature against the current system time. If the hardware clock jumps back decades, the moment is before the signing certificates became valid, so every perfectly good VIB is rejected. Broadcom documents exactly this failure: a depleted CMOS battery resets the hardware clock to a factory default date, and Secure Boot rejects the VIB signatures because that date predates the certificates’ validity period (KB 445564 – their example shows 2000-01-01; mine showed 1998).

Rule of thumb: don’t start removing VIBs or rebuilding images until you’ve read the date in the PSOD.

Fixing the clock

Reboot, press F2 → System Setup → System BIOS → Miscellaneous Settings and set the system date and time. Use UTC – ESXi treats the hardware clock as UTC.

KB 445564 lists three resolutions: set the UEFI time manually, temporarily disable Secure Boot so ESXi can boot and correct the time via NTP, and replace the CMOS battery. I’d strongly prefer the first one on a TPM host: changing firmware security settings is exactly what can trigger the next problem.

I corrected the time, left Secure Boot enabled and rebooted.

PSOD #2 – “A security violation was detected”

ESXi 9 PSOD on a Dell PowerEdge R640: Unable to restore the system configuration. A security violation was detected.
PSOD #2 after fixing the clock: the TPM-sealed configuration can't be restored.

The PSOD even points straight to the Broadcom KB that explains the fix:

The system has found a problem on your machine and cannot continue.

Unable to restore the system configuration. A security violation was detected.
https://knowledge.broadcom.com/external/article/312109

This is a different mechanism. On a host with a TPM 2.0, ESXi encrypts its configuration and seals the key to the TPM and the firmware’s security state. If that state doesn’t match what was sealed, ESXi refuses to decrypt its configuration (Broadcom KB 312109).

Fixing the clock didn’t cause this. The first PSOD happened so early that it hid the second one; once the signatures validated, boot progressed to the next security check and failed there.

How to recover: the encryption recovery key

KB 312109 gives the order of checks:

  1. Make sure the firmware security settings are what they were before: UEFI boot, Secure Boot enabled, TPM enabled. A power loss with a dead battery, or your own troubleshooting in the BIOS, may have changed something.
  2. If the settings are correct and it still fails, boot with the recovery key: when the ESXi boot loader appears, press Shift+O (the letter O), append encryptionRecoveryKey=<your-recovery-key> to the existing boot options and press Enter. ESXi decrypts the configuration and boots.
  3. Once up, re-seal and persist the configuration with esxcli system settings encryption set --mode=TPM followed by /sbin/auto-backup.sh.

Tip: the iDRAC9 virtual console clipboard makes pasting that long key much less painful.

And the catch, straight from Broadcom: if you don’t have the recovery key, the only option is to reinstall ESXi.

What I did

This host was being prepared for a new vSAN ESA cluster, with no workloads or configuration worth saving. I performed a clean ESXi 9.0.2 install and it booted normally. On a production host – especially one in a VCF environment, where a reinstall means decommissioning and recommissioning the host – the recovery key would have saved hours.

Root cause: the RTC/CMOS battery

The hardware clock is kept by the real-time clock (RTC), powered by a coin cell on the system board (a CR2032 on the R640) whenever the server has no AC power. A depleted battery is invisible while the server is plugged in. The moment you lose mains power – a power cut, a PDU swap, a move – the clock resets.

Don’t confuse it with the PERC controller’s cache battery: for this problem only the system board battery matters.

Checking the battery

  • iDRAC System Event Log / Lifecycle Log – look for a CMOS battery failure event (Dell’s diagnostics report it as DIAG0213 – System board CMOS battery failure detected). The absence of an event does not prove the battery is healthy.
  • A simple power test: set the correct date/time in the BIOS (UTC), shut down normally, remove AC power from both PSUs, wait several minutes, reconnect power, press F2 and check the date. If the clock has reverted to an old default date, replace the battery. It’s a few euros and a five-minute job.

Prevention checklist

Before a Dell host with Secure Boot and TPM goes into production or a vSAN cluster:

[ ] UEFI boot mode, Secure Boot enabled, TPM 2.0 enabled
[ ] BIOS date/time correct and set to UTC
[ ] RTC clock survives complete AC power removal (test above)
[ ] ESXi NTP configured and time verified
[ ] Encryption recovery key retrieved and stored OFF the host
[ ] iDRAC NTP, SEL monitoring and email/SNMP/Redfish alerts configured
[ ] Hosts monitored outside ESXi (iDRAC / OpenManage Enterprise)

Get the recovery key on every TPM host – today, while it’s healthy:

esxcli system settings encryption get             # Mode: TPM means the config is sealed
esxcli system settings encryption recovery list # store the key in your password vault

A few useful checks once ESXi is running:

date                                       # ESXi system time
esxcli hardware clock get # hardware (RTC) clock
esxcli software vib signature verify # VIB signature status

Two notes on monitoring:

  • NTP cannot help a host that doesn’t boot. ESXi only gets to NTP after Secure Boot has passed – and Secure Boot depends on the RTC.
  • iDRAC is independent of ESXi. Configure NTP on the iDRAC too (it has its own clock, which keeps its logs and certificates sane – it does not correct the host BIOS clock), and send its hardware alerts somewhere you actually read. For multiple servers, Dell OpenManage Enterprise centralises this.

Troubleshooting flow

PSOD: “UEFI Secure Boot failed … Failed to verify signatures”

  • Timestamp wrong? Stop. Fix the BIOS date/time (UTC), check the iDRAC log for a CMOS battery event, plan a battery replacement. Don’t touch the VIBs.
  • Timestamp correct? Now investigate the image: recent upgrades, OEM/async drivers, NSX VIBs, firmware changes, Secure Boot configuration.

PSOD: “Unable to restore the system configuration. A security violation was detected”

  1. Verify UEFI, Secure Boot and TPM are enabled and unchanged.
  2. Boot with encryptionRecoveryKey= via Shift+O.
  3. Re-seal with esxcli system settings encryption set --mode=TPM and run /sbin/auto-backup.sh.
  4. No recovery key → reinstall (and in VCF: decommission/recommission).

Don’t clear the TPM on a production host without understanding the consequences – it destroys the sealed key for good.

Lessons learned

  1. Read the timestamp on a Secure Boot PSOD. A wall of failed VIB signatures looks like a software problem; the date tells you it’s hardware.
  2. Secure Boot makes the hardware clock part of your security chain. It’s not just a BIOS checkbox.
  3. The first PSOD can hide the second. Fixing one security check lets the boot reach the next one.
  4. The recovery key is the difference between a reboot and a rebuild. Collect it for every TPM host and keep it off the host.
  5. Test the RTC battery – a €5 coin cell can take a host out after the next power cut.

And if your PSOD says UEFI Secure Boot failed at time 1998…, look at the clock before you blame ESXi.

References

Homelab 2.0 – VCF 9.0.1 on Dell PowerEdge R640

Introduction

After a long period of not blogging and a compromised website I take a new start.

This post documents the baseline configuration of my homelab which is built around VMware Cloud Foundation (VCF) 9.0.1 running on Dell PowerEdge R640 servers. All future posts will assume this environment unless explicitly stated otherwise.

The primary goal of this homelab is to provide a realistic, repeatable platform for testing VMware features, validating upgrade paths, and developing automation workflows without impacting production environments. Also this lab is being rebuild regular as this is being used a demo material in The VCF trainings I teach. Sometimes things go crazy and things get broken. Luckily I have another smaller VCF environment that runs as a kind of production environment. This is run on MiniForum MS-A2 devices. I will give you more info how this is setup in another post.


Homelab Goals

This homelab is used for:

  • VMware Cloud Foundation (VCF) lifecycle management
  • vSphere and vCenter feature validation
  • HCX migration testing
  • PowerShell / PowerCLI automation
  • Patch, upgrade, and failure scenario testing
  • General infrastructure experimentation
  • Demo Material during the VCF training classes I teach as Broadcom VMware VCI

Hardware Overview

Physical Hosts

The homelab consists of multiple Dell PowerEdge R640 servers.

ComponentSpecification
Server ModelDell PowerEdge R640
CPUDual Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
Memory512 GB RAM per host
BootBOSS-S1 with mirrored M.2 SSDs
StorageLocal SSD / NVMe for vSAN (2×1.6TB)
NetworkingDual 25GbE SFP+ + Dual 10GbE + 1Gbe iDRAC

Out-of-Band Management

  • iDRAC 9 Enterprise
  • Dedicated management NIC
  • Static IP addressing

Network Architecture

My network infrastructure is based on Ubiquity material.

PurposeDevice
Router / Firewall Dream Machine Pro
SFP+ AggregateUSW Pro Aggregation
28x10GbE + 4x25GbE
Network switchPro Max 48
32x1GbE + 16×2.5GbE + 4x10GbE SFP+

The network is segmented using VLANs to mirror production-style deployments.

VLAN & IP Plan

PurposeVLANSubnet
Management INFRA3192.168.3.0/24
Management VM4192.168.4.0/24
vMotion7192.168.7.0/24
vSAN6192.168.6.0/24
NSX Overlay12192.168.12.0/24
Storage (iSCSI – NFS)5192.168.5.0/24

All ESXi hosts are configured with redundant 10GbE and 25 GBE uplinks connected to a managed switch supporting VLAN trunking. the 25 GbE nics will be used for vSAN, vMotion, NSX Overlay. One 25GbE NIC is really 25GbE connected.


Software Stack & Versions

The following software versions are used as the baseline for this homelab:

ComponentVersion
VMware Cloud Foundation9.0.1
ESXiIncluded with VCF 9.0.1
vCenter ServerIncluded with VCF 9.0.1
NSXIncluded with VCF 9.0.1
vSANIncluded with VCF 9.0.1
Dell OpenManageLatest supported version

Note: Versions may change over time as upgrades are performed. This post will be updated accordingly.


Core Infrastructure Services

DNS

  • Internal DNS hosted on a Microsoft Windows 2025 server
  • Forward and reverse DNS configured for all infrastructure components

NTP

  • Central NTP source configured for all ESXi hosts, vCenter, and VCF components
  • I use 3 enteries, 0.be.pool.ntp.org, 1.be.pool.ntp.org and 2.be.pool.ntp.org
  • Time synchronization validated before any VCF deployment
  • Firewall allows servers to go to internet for the NTP configuration

Certificates

Certificates

  • Initial deployment uses VMware-generated certificates
  • Future posts will cover integration with an internal Certificate Authority on a Windows 2025 Server

Licenses

  • As vExpert I was happy I got a VMUG Advantage Package. And finishing up some VMware Certificates you get licenses in the VMUG Advantage program.

Authentication & Access

  • vCenter authentication via local SSO domain
  • ESXi root access enabled for troubleshooting
  • SSH access restricted and enabled only when required. 1 host has permanent SSH open as this makes it easier when doing demos.
  • API access used extensively for automation and scripting

VCF Deployment Assumptions

All future posts assume the following prerequisites are met:

  • Fully functional DNS with forward and reverse lookup
  • Reliable NTP configuration across all hosts
  • VLAN-backed networking with jumbo frames where applicable
  • iDRAC access for all physical servers
  • Administrative access to the VCF management domain

If your environment differs, adjustments may be required.


What’s Next

Upcoming posts will cover:

  • Preparing Dell PowerEdge R640 servers for VCF 9.0
  • How to get VCF 9.0 License Keys
  • VCF bring-up and management domain deployment
  • NSX configuration and workload domain creation
  • HCX deployment and migration testing
  • PowerShell and API-based automation examples

Change Log

  • v1.0 – Initial publication of homelab baseline

This post will be kept up to date as the homelab evolves.