Corelan Cybersecurity Research

In certain distributed AD scenario’s, Domain Admins group membership or local DC admin privileges are restricted to certain people only. This is a good thing to do, but it requires you to think about certain issues before they happen.

One of these issues is backup and restore. Yes, you can schedule a system state backup on a DC and share the folder that contains the bkf file. The file can be put on tape by someone who doesn’t have Domain Admin or DC admin rights. No big deal. Not only a regular backup, but a ASR backup can be scheduled as well (see at the bottom of this document). This type of backup might help you speed up the restore process in case of a failure. I think we all agree that the backup piece is nothing to worry about, even if you don’t allow anyone else other than yourself to be admin in the environment.

Restoring the DC to the same hardware is no problem either. You provide a local person with the AD restore password and that person can do an in-place restore, restoring the entire system state, do authoritative restore and so on… on the freshly installed 2003 server. This works fine, as long as the hardware is kind of identical to the original server hardware that was used to create the backup. After all, if you restore the system state, you basically overwrite the registry as well. So if the new server contains a different network adapter, you’ll lose network connectivity after rebooting, and nobody will be able to authenticate against the restored Active Directory controller. The only person who would be able to solve the issue would need to have local admin rights to the server – which is exactly what the local person doesn’t have. (or you can try booting in AD restore mode and clear up any driver issues)… But there are other ways to bypass the problem. In real life, a restore to the same hardware may be difficult. A Disaster Recovery scenario may require you to get a new server, any server, and get AD up and running again. If you are not prepared for this, you might spend more time trying to recover than building a new DC and migrate all of your clients…

One of the possible solutions that might help preventing situations like that from ruining your day (night, week and even your carreer) is VMWare. But it requires some preparation

Preparation by the Enterprise/Domain Admin

First, you (with admin rights) need to convert a live (physical) machine into a vmware machine. (If your production environment is already running within VMWare, then you can just clone the production server and skip the following procedure and jump right into "When the disaster hits".) Virtualizing a physical server can be done online (hot) or by using a boot cd (cold). I would recommend you doing a coldboot conversion, because Domain Controllers don’t like the agent (live) conversion. Oh and by the way: Domain Controllers don’t like vmware snapshots either, so DON’T do it !). In most cases, the conversion itself works fine. But the real trouble is yet to start. In most cases, when booting, some services on the machine won’t work well. Either there will be driver/hardware issues, or you’ll have a non-working DC. Anyhow, before even booting the virtualized DC, make sure it is isolated from the rest of the network. By the way : the machine that needs to be virtualized should be the first DC in the domain or even forest (if you have a single domain forest).

The first thing you should do is make sure the Windows server boots. Assuming that it does, install VMWare tools, and then focus on setting/correcting IP stack/network interfaces if needed.. You may see the following message when you try to set the IP address on the vmware adapter :

This means that the virtualized system still sees the original (physical) NIC and it’s IP properties, so you’ll have to remove that adapter from the system prior to continuing : Open a Dos prompt and run set devmgr_show_nonpresent_devices=1 (tx Stijn H. for the tip) . Next, open devmgmt.msc, enable "Show hidden devices" and remove the non-existing devices. If necessary, clean up the registry by removing the incorrect keys from the registry under HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Interfaces and from HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Adapters. (Use a different IP address if you’re not sure which GUID corresponds with the physical interface and which one corresponds with the vmware interface). Reboot the server afterwards. If just removing the keys and setting the IP doesn’t work, you may want to try the following things :

Make sure the DC boots and works fine (open AD U&C, see if netlogon share is working, etc…) prior to saving the vmdk file. If the vmdk files are ok, put them on a USB drive or something else and leave the file in the remote location. You can find more information about fixing possible issues with your DC in my next blog post : How to restore a Windows 2003 DC using ASR and vmware. (By the way : that blog post will also explain how to rebuild your DC in VMWare, using ASR. So if a vmware convert (coldboot) doesn’t work, then you can also rebuild the DC in a vmware environment as well.)

If a conversion does not work and you are confident that a regular system restore will work, open VMWare server and create a new custom virtual machine. Note : this procedure assumes that you have a full backup of the DC that you want to rebuild. Make sure the backup contains the volume that contains the system files, and the System State. Before continuing, make sure the machine does NOT have access to the production environment. Make sure the machine is on an isolated network. Next, create a similar disk layout, install Windows 2003 (with same SP and hotfixes, as standalone server), give it the same hostname and IP. Next, put the backup (bkf) file from the live DC on the system. Do not restore it yet. Make sure the machine has the same services running (DNS, …).  Next, restore the entire .bkf file "in-place", overwrite all existing files. This basically overwrites a lot of stuff (registry, ntds, log, sysvol, hal). The hal is a very important component, and unfortunately it is a component that is very sensitive. If you machine doesn’t boot, boot with the Windows 2003 server CD-Rom again. Setup will find an existing installation, and you can allow the setup tools to repair the installation.
Have a look at http://support.microsoft.com/kb/263532/en-us for more help regarding the rebuild of a DC on different hardware (including ways to repair the hal and so on). Reboot and you should be fine.

From this point forward, we’ll assume that you (the Domain Admin) have managed to rebuild a DC, in an isolated vmware environment. Again, verify that the DC works well in that standalone vmware environment and keep a safe copy aside in the remote location, waiting for the disaster to strike.

All steps from this point forward do not require Domain Admin or local DC rights. The only privileges you need to give to a remote admin is the Directory Services Restore Mode password.

When the disaster hits

Now, when the disaster hits and you are left without any DC’s, this is what the local person, without Domain admin rights or local admin rights to the DC, needs to do :

1. set up a fresh machine and install (free) VMWare server. (Don’t forget to register for your free key)
2. Open the vmware (vmdk) Domain Controller using VMWare server
3. Boot the VM machine in Directory Services Restore mode (Press F8 at boot time) 
   

   Log on with the active directory restore mode user & password (usually this is administrator + the password that you’ve set during dcpromo)
   Click "OK" at the warning
   
   This will grant the person admin rights to the machine. If there is still something wrong with drivers or so, he (or she) can now install additional drivers or make other modifications.

4. Copy a recent ntbackup backup file (.bkf) into the VMWare environment
5. Restore the System State to an alternate location (don’t restore the files in-place !)
6. Open a prompt and navigate to the folder that contains the restored system state
7. Copy the ntds.dit file and the .log file from the restored system state (Active Directory folder) on top of the ntds.dit (the one that already exists on the vmware box. Default folder is c:\windows\ntds\ntds.dit)
8. Reboot. If all goes well, you’ll have a working DC. (a lot depends on the quality of your backup, but in some cases, the ntds.dit file is corrupt) .
9. If you are getting an error when Windows server boots again (at the logon page), boot into AD Restore mode again and do this :

(assuming that c:\windows\ntds\ntds.dit is the location where your dit file is)

When the file is repaired, run a "ntdsutil files integrity" to verify that the file is ok again. Additionally, if you know what you are doing, you can run an authoritative restore as well

(click YES to continue the restore)

Exit ntdsutil (quit – quit), reboot the machine, you should have a working DC again.

If this vmware machine was not the first DC in your domain, you’ll need to seize the fsmo roles to this new DC. You can read how this works in one of my future posts.

Remember : if you still have working DC’s at the time the disaster hits you, you should use those DC’s and rebuild a new DC (with a regular dcpromo) instead of using this recovery technique. You may need to seize FSMO roles if the DC that died was holding one or more roles, but that should be the hardest part in that scenario.

If you need to put the vmware server on physical hardware afterwards… well – read http://www.vmware.com/support/v2p/index.html. I haven’t tried it myself, it does look a bit complex, but it can be done

In case you were wondering how to automate/schedule a ASR backup ? As with many Microsoft tools, it is based on existing tools with undocumented features/parameters.
Simply use ntbackup with the "asrbackup" option and you’ll be fine.

Example :

Make sure the floppy drive contains a diskette when the script runs.