It’s not a good day when an ESX host goes down…thankfully HA brought the VMs back up on another host, but still. Then when the host doesn’t boot back up after…it gets worse. Turns out there was an issue with the storage controller for the local drives – had to load the foreign config and bring the array healthy again. That worked successfully – and ESX started to boot…then I was greeted with this screen:
…stuck in busybox. I found the following from VMWare:
VMware ESX 4.0 uses a virtual disk to store the service console. The virtual disk is created during the installation phase on a VMFS volume residing on the boot disk or LUN. The VMFS volume containing the service console files (also referred to the esxconsole.vmdk) may be detected as a snapshot LUN. As a result of the VMFS volume being detected as a snapshot LUN, the VMware ESX host may be unable to access the service console files during boot.
The following resolution is taken from this KB article: VMWare KB1012142
-
Provide the necessary credentials to access the busy box.
-
Run this command to enable resignaturing on the VMware ESX machine:
esxcfg-advcfg -s 1 /LVM/EnableResignature
You must get an output similar to:
Value of EnableResignature is 1.
Note: If the root is mounted as read only, run the command mount -o remount / to remount the volumes so that they are in a writable state.
-
Run this command to unload the VMFS drivers:
vmkload_mod -u vmfs3
-
Run this command to load the VMFS drivers:
vmkload_mod vmfs3
-
Run this command to detect new VMFS volumes and resignature the volume:
vmkfstools -V
-
Run this command to identify the full path of the esxconsole.vmdk file:
find /vmfs/volumes/ -name esxconsole.vmdk
The output appears similar to:
/vmfs/volumes/4a14d968-88bf7161-700f-00145ef48f76/esxconsole-4a14d906-2f96-7956-7284-00145ef48f74/esxconsole.vmdkNote: Make a note of this full path.
-
Restart the VMware ESX machine. You see a menu provided by the grub boot loader.
-
Press e to edit the grub entries manually.
-
Scroll down to the line that starts with kernel /vmlinuz (it is indented under the VMware ESX 4.0 heading).
-
Go to the end of the line and include the following entry after a space:
/boot/cosvmdk=<path>/esxconsole.vmdk
Where <path> is the full path identified in step 6.
-
Press Enter to accept the changes.
-
Press b to boot using the modified settings. The ESX host successfully boots.
Note: The changes made to the boot options are not saved. They only apply to the current boot process. The changes need to be made to the boot configuration files as described in the following steps.
-
Log into the console as root.
-
Edit the /etc/vmware/esx.conf file with a text editor and modify the following lines:
/adv/Misc/CosCorefile = “/vmfs/volumes/<path>/core-dumps/cos-core”
/boot/cosvmdk = “/vmfs/volumes/<path>/esxconsole.vmdk”
Where <path> is the full path identified in step 6.
-
Run this command to update the boot configuration files:
esxcfg-boot -b
Long story short, when the config was reloaded onto the local storage controller, the GUID for the volume changed, thus causing ESX to not be able to find the service console. By resignaturing the volume and changing to the correct GUID – the service console is found, and the ESX host boots successfully. You will notice, however, that the local datastore is listed as a snapshot:
