Corelan Cybersecurity Research

Today, I will try to explain 2 techniques that will allow you to backup your vmdk files (virtual machines) on a VMWare ESX 3.0.x server. My lab runs on 3.0.2, but this will work on 3.0.1 (and maybe earlier versions) as well. Before continuing, let’s make a couple of assumptions :

1. I will explain how to backup the virtual machines to another location on the SAN. Although one of the scripts that will be explained in this document will allow you to backup using 3^rd party software or copy the files over the network to another server (using NFS, FTP, SMB, …), I’m not going to explain that because I don’t use commercial 3^rd party software myself. Furthermore, I’m sure you can figure out how to backup the backup copy yourself
2. We will backup machines while they are powered on. There’s one exception : Never backup a running Windows AD Domain Controller using the “Powered On VM” technique. Look at the section to backup Domain Controllers instead (which will explain you how to get the status, stop the VM, copy the files, and start the machine again). After all, backing up a live (running) Virtual Machine is based upon snapshots. Windows Domain Controllers don’t like to be snapshotted in a virtual environment. You’ll probably end up with a USN rollback event, which results in the netlogon service to be paused. End result : your DC will screwed and you’ll need to rebuild it! (http://support.microsoft.com/kb/875495)

Back up Powered On VM’s

The basic technique used to back up running VM’s is based upon native VMWare commands, but Massimiliano Daneri from www.vmts.net has built a very nice wrapper around the necessary commands. Go to his website and download the vmbk.pl package. (Current version is 3.0 revision 11). Copy the file to your esx hosts (using scp or Veeam’s FastSCP). Put the file under /tmp.
Next, open a root shell on the esx host and go to the /tmp folder
Unpack the tar.gz archive by running : tar –xvfz vmbk.tar.gz
Go into the extracted folder : cd vmbk
Run the install script : ./install.sh

Note :If you already had an older version running on your server, you can do an inplace upgrade. The install.sh script will not overwrite any config files.

The install script has copied the vmbk.pl script to /usr/local/bin folder, along with the default config file vmbk-default.conf.

There are many ways to set up the config files and schedule the script, but I’d like to do it my way (which does not necessarily mean that my way is better… it’s just somewhat different than the procedure which is provided on the vmts.net website).

First of all, you’ll need to determine the folders on your ESX server that contain the vmdk/vmx files of the machines that need to be backed up. Don’t forget : do NOT use this vmbk.pl script to back up Domain Controllers. The backup will be useless AND you’ll screw up your running DC !

My lab has 3 machines : Venus (which is the DC), Apollo, and Hercules

The live machines are stored under :
/vmfs/volumes/storage1/apollo
/vmfs/volumes/storage1/hercules
/vmfs/volumes/storage1/Venus

The volume on the SAN that will be used to hold the backed up version of the machines is : /vmfs/volumes/SAN_os_Storage_170Gb/backups

Next, copy the default, unmodified, vmbk config file from /usr/local/bin to /vmfs/volumes/storage1/apollo and to /vmfs/volumes/storage1/hercules. I want to make sure I’m not using the wrong config file for a certain machine, so I’ve renamed the config files and included the name of the virtual machine in the name of the config file. This is not a requirement for the script to work. So basically, the config file setup looks like this :

[root@esx01 storage1]# pwd
/vmfs/volumes/storage1
[root@esx01 storage1]# find -name vmbk*.conf
./apollo/vmbk-apollo.conf
./hercules/vmbk-hercules.conf
[root@esx01 storage1]#

Edit both files and go over the parameters. The following parameters needed my attention :

backupESX=true
destination=/vmfs/volumes/SAN_os_Storage_170Gb/backups
Restore=true
BackupVMX=true
DiskFormat=VMDK
usecp=false
email=true
smtpserver=
to=
from=
encode=base64
html=true

(I did not change any of the other parameters. )

After editing the files, let’s make a “master” script that will be used in the crontab, launching the 2 backups. Go to /root and create a folder bin (if it doesn’t exist already). Set the permissions on that bin folder to “root” only (chmod 700 /root/bin –R).

Create a file called backupvms.sh under /root/bin and put the following 2 lines in there (change the paths and filenames to reflect your environment of course):
/usr/bin/perl /usr/local/bin/vmbk.pl -d /vmfs/volumes/SAN_os_Storage_170Gb/backups -1 -4 -c -C /vmfs/volumes/storage1/hercules/vmbk-hercules.conf -x /vmfs/volumes/storage1/hercules/hercules.vmx
/usr/bin/perl /usr/local/bin/vmbk.pl -d /vmfs/volumes/SAN_os_Storage_170Gb/backups -1 -4 -c -C /vmfs/volumes/storage1/apollo/vmbk-apollo.conf -x /vmfs/volumes/storage1/apollo/apollo.vmx

This will first launch the vmbk.pl script, with the following parameters : (first script)
-d : use /vmfs/volumes/SAN_os_Storage_170Gb/backups as destination
-1 : always use vmdk extension (for export)
-4 : backup configuration file and CMOS
-c : create restore command on export directory (doesn’t work with the current version)
-C : use /vmfs/volumes/storage1/hercules/vmbk-hercules.conf as config file
-x : select guest by vmx file, use /vmfs/volumes/storage1/hercules/hercules.vmx

Note : You could leave out the –d and –c parameters, as they are specified in the config file.

Now edit the crontab, from your root shell, using the crontab –e command. Assuming that you want to backup your guests on Sunday morning, 3:30am : Add the following entry :

30 3 * * 0 /bin/sh /root/bin/backupvm.sh > /dev/null 2>&1

Save and exit.

The last thing we need to do is to allow the ESX host to connect to your mailserver. You will have to change the ESX firewall for this. Run the following command to allow outgoing traffic on port 25 (for smtp) : esxcfg-firewall -o 25,tcp,out,smtpclient

Now, when the script runs, a snapshot will be made of the running guests. The machine will be backed up to the backup folder using the vmware tools export functionality, and the snapshot delta will be applied to the running machine. When each script completes, you’ll receive an email that looks more or less like this :

Quick note : the script will only backup running machines. If your machine is not running, no backup will be made.

The script and config files, as configured in this procedure, will allow the backup script to backup ALL disks from the virtual machine. In my setup, I only use vmware based disks for OS partitions, and then use the Microsoft iSCSI initiator to connect to the SAN and mount additional disks within Windows. Those disks will not be backed up as part of this script !

If you don’t want certain (vmdk) disks to be backed up, you need to specify this in the corresponding vmx file. Have a look at http://www.vmts.net/vmbkmanualv3.htm and look for “option in vmx configuration file –> SCSIX:Y”

Back up Windows AD Domain Controllers

There’s one machine left : the domain controller.
As explained earlier, we cannot use the ESX snapshot functionality because it will break your DC. And since we cannot just copy the files while the machine is running, this is what you can do.
First of all, I’ll assume you have multiple DC’s running, so taking down a DC is not a big problem. Another requirement is that you have the vmware tools installed on the guest. This will allow you to properly shut down the machine.

Next, edit the /root/bin/backupvms.sh script and add the following lines :

#first, stop “Venus”
if [ -x /usr/bin/vmware ]
then
ESXV3=`/usr/bin/vmware -v | grep “ESX Server 3″`
if [ -n “$ESXV3” ]
then
# Get the VM list and grep on Venus
VMLIST=`/usr/bin/vmware-cmd -l | grep Venus`
for VM in $VMLIST
do
# Get the VM state
VMSTATE=`/usr/bin/vmware-cmd “$VM” getstate -q`
# Guest OS shutdown if VMSTATE is equal to “on”
if [ “$VMSTATE” == “on” ]
then
echo “Stopping Venus ($VM)”
/usr/bin/vmware-cmd “$VM” stop trysoft
sleep 2
fi
done
# Delay for 1,5 minute to give the VMs more time
# to cleanly shutdown
sleep 90
fi
fi

#see if Venus has shut down
VMLIST=`/usr/bin/vmware-cmd -l | grep Venus`
for VM in $VMLIST
do
VMSTATE=`/usr/bin/vmware-cmd “$VM” getstate -q`
if [ “$VMSTATE” == “off” ]
then
#remove the old backup folder
/bin/rm /vmfs/volumes/SAN_os_Storage_170Gb/backups/Venus -rf >>/var/log/backup.log
#create a new backup folder
/bin/mkdir /vmfs/volumes/SAN_os_Storage_170Gb/backups/Venus >>/var/log/backup.log
#copy all files to the backup folder
/bin/cp /vmfs/volumes/storage1/Venus/* /vmfs/volumes/SAN_os_Storage_170Gb/backups/Venus -R >>/var/log/backup.log
fi
done

#start Venus again
if [ -x /usr/bin/vmware ]
then
ESXV3=`/usr/bin/vmware -v | grep “ESX Server 3″`
if [ -n “$ESXV3” ]
then
# Get the VM list and grep on Venus
VMLIST=`/usr/bin/vmware-cmd -l | grep Venus`
for VM in $VMLIST
do
# Get the VM state
VMSTATE=`/usr/bin/vmware-cmd “$VM” getstate -q`
# Guest OS shutdown if VMSTATE is equal to “off”
if [ “$VMSTATE” == “off” ]
then
echo “Starting Venus ($VM)”
/usr/bin/vmware-cmd “$VM” start
sleep 2
fi
done
fi
fi

Pay attention: when doing a copy/paste of the script, some of the lines in the script may be split into 2 lines – make sure to check every line and reassemble the broken lines. Save and exit. Before you run this script, you’ll need to test. If you have multiple machines that have “Venus” in the name, you’ll need to change the “grep” parameter to ensure that you are only stopping the machine that needs to be stopped.

That’s it. This will backup your running machines using vmbk.pl and backup your DC by stopping the machine, copying the files, and starting the machine again.

The entire script can be downloaded from
backupvms.sh (2.3 KiB)

How to restore ?

Since the current version (rev11) does not support creating the .restore shell script yet, you’ll have to perform the restore manually. See http://www.vmts.net/snitz/topic.asp?TOPIC_ID=316 for more info on restoring without the .restore script