Reconstructing the Cluster

Even if you can get critical VMs up and running again from existing backups on other servers, the ESXi cluster needs to be completely rebuilt. This is sometimes a challenge because infrastructures often grow dynamically. In some cases, the team member who created the initial configuration is no longer with the company, or the documentation is sparse. However, if you have access to scripts prepared for automatic installation of the cluster, also known as kickstart files, the reconstruction work is far easier to handle and faster [4].

If you have an Enterprise Plus license, you can back up your cluster's configuration profiles and the configurations of the virtual switches automatically. If not, you can run a backup manually or automate it with your own tools. Use the following commands at the ESXi command line to synchronize the configuration first and then create a tar.gz file that you can download in your browser:

vim-cmd hostsvc/firmware/sync_config
vim-cmd hostsvc/firmware/backup_config

The output from the second command contains the URL for downloading the file. If you prefer to use the vSphere command-line interface (CLI), or if it is easier for you to automate, open the CLI and navigate to C:\Program Files\VMware\VMware vSphere CLI\bin. Launch the backup program there with the command:

vicfg-cfgbackup.pl --server=esxi --username=root -s latest_backup.tgz

You will then be prompted to enter the root password. To work around this prompt (e.g., in your automated scripts), you can pass in the password directly with --password=<password>. You will then find the backup file in your current working directory.

To restore the configuration, you first need to install a version with the same build ID as the one used to make the backup. Copy the backup file to the system or the attached datastore and then connect to the console. When you get there, switch the system to maintenance mode before starting the recovery:

vim-cmd hostsvc/maintenance_mode_enter
vim-cmd hostsvc/firmware/ restore_config /<directory>/latest_backup.tgz

If the universally unique identifier (UUID) of the host system has changed in the meantime, you need to add 1 before specifying the backup file. This means that the UUID is overwritten, but this only works if the backup was not encrypted. Encryption was introduced in vSphere version 7.0 U2. However, the key remains in the hardware's trusted platform module (TPM), and the file can only be recovered on the same host system. Afterward, you will of course need to install all the recommended updates and restore the backups of the VMs.

Conclusions

This article illustrates the consequences of being hit by a specialized ransomware variant on the ESXi server and how you can at least prepare for recovery, if you can't protect yourself reliably.