Corelan Cybersecurity Research

Today, I’m going to share some ‘notes from the field’ about fixing oab issues in Exchange 2007

In order to fully understand the oab generation and distribution process, I will assume that you are running the Mailbox server role and HUB/CAS server roles on different servers. Of course, this is not a requirement to run a successfull Exchange installation at all, but it will help understanding the various processes. Furthermore, I will sometimes refer to clustered mailbox servers (CCR) and how this setup may influence the oalgen process.

Symptoms

When you have discovered that the Offline Address Book is not updating anymore, you may see the following event log entries in the log on your mailbox server :

Event 9334, MSExchange SA

OALGen encountered error ffffffff while initializing the offline address list generation process. No offline address lists have been generated. Please check the event log for more information.
– /o=Group Messaging Organization/cn=addrlists/cn=oabs/cn=Default Offline Address List

Event 9335, MSExchangeSA

OALGen is running on a cluster continuous replication (CCR) node which does not have registry value ‘SYSTEM\CurrentControlSet\Services\MSExchangeSA\Parameters\CLUSTERNAME\EnableOabGenOnThisNode’ or it is not set to this node name. Offline address book generation will not be performed.

The Offline Address Book files that get downloaded by Outlook clients are stored on the CAS Server(s) in the following folder (assuming that you have performed an Exchange installation using the default folders) : C:\Program Files\Microsoft\Exchange Server\ClientAccess\OAB\

This folder

• should exist on all CAS servers
• should contain a bunch of lzx files and a file called oab.xml. Most of these files should be not older than a couple of hours, and especially oab.xml should have a relatively recent timestamp

If one of these parameters is not ok, you most likely will have a problem with the offline address book. I’m convinced that you will be able to fix the issue, but you can only fix the issue if you know that an issue exists. So you have to monitor this folder and see if the files get updated. In fact, it may be ok to just monitor the timestamp of oab.xml and report back if the file gets older than one day (for example).

Detecting the issue : Monitoring oab.xml file age

The following simple powershell script may help you monitoring the timestamp of oab.xml on your cas servers :

# # Powershell Script written by Peter Van Eeckhoutte # # Purpose : monitor the oab.xml timestamp # and report timestamp issues via email # # http://www.corelan.be:8800 # # ############ Edit the following fields ################################## $emailFrom = "admin@mydomain.com" $emailTo = "admin@mydomain.com" $smtpServer = "localhost" $oabpath = "c:\Program Files\Microsoft\Exchange Server\ClientAccess\OAB\abcdefgh-djdjdjdj-iiiiiii-zieueie" $alerthours=25 ######################################################################### # # Other vars # $subject = "Possible OABGen problem detected on " + $env:ComputerName $body="" # ###################" # # $basetimestamp=[DateTime]::Now.AddHours($alerthours*-1) # # Get timestamp of oab.xml file $oabfile = get-item $oabpath"\oab.xml" write-host "Current oab.xml timestamp : " $oabfile.lastaccesstime write-host "Comparing this timestamp with " $basetimestamp if ($oabfile.lastaccesstime -le $basetimestamp) { write-host "oab.xml file is too old ! There may be an oabgen problem !" $body = "The oab.xml (Offline Address Book) file on " + $env:ComputerName $body += " has a timestamp that is older than " + $alerthours + " hours ago`n`n" $body += "File monitored : " + $oabpath + "\oab.xml`n" $body += "File timestamp : " + $oabfile.lastaccesstime #send email $smtp = new-object Net.Mail.SmtpClient($smtpServer) $smtp.Send($emailFrom, $emailTo, $subject, $body) write-host "Email sent`n" } else { write-host "oab.xml file timestamp is ok" } # #

Simply edit the variables at the top of the script : set email addresses, the folder path to oab.xml (only specify the path, not the file name), set the number of hours you want to allow and schedule the script to run every hour or so. (In my example above, I’m allowing the oab.xml file to be 25 hours old. Check the oabgen schedule on the server and find a good amount of hours that is allowed for your environment).   If you are having issues scheduling the script on your server, let me know…

So far so good, at least now we now if the oab gets updated or not.

Fixing oalgen and oab distribution issues

If the oab.xml (and corresponding lzx files) are not getting updated, or if the folder that should contain these files does not even exist, following this procedure to detecting/fixing the issue.

Make sure there is an Offline Address Book in the environment

First of all, there needs to be a default offline address book configured in the environment.

Run the ‘Get-OfflineAddressBook | FL’ cmdlet in the Exchange Management Shell and verify that

• there is a default offline address book, that has the “IsDefault” parameter set to “true”
• the offline address book server is set to one of the servers running the mailbox server role
• The Schedule field is not empty
• The “IsValid” parameter is true

If one of these requirements is not met, delete the offline address book, wait until Active Directory DC’s have replicated information to each other and create a new one.  If there is no Offline Address Book defined yet, then just create a new one.

(Note : If you are creating a new Offline Address Book, the MailboxDatabases may loose their reference to the default offline address book. This will result in Autodiscover returning a non-working OAB Url to Outlook clients, and they won’t be able to download the OAB anymore.  At the bottom of this page, there is a short note on this and the powershell command to reset the reference to the default offline address book for mailbox databases)

While it is obviously perfectly possible to create a new Offline Address Book using powershell, it may be just easier to do this with the EMC GUI.

Offline Address Books are configured under “Organization Configuration” – “Mailbox” – “Offline Address Book”

Create a new OAB, provide a good name (such as “Default Offline Address List”) and make sure the OAB contains at least the default Global Address List (and optionally other address lists as well). Set the generation server to your mailbox server (of if you are running a cluster, pick the CMS (cluster name)).

Enable “Web based distribution” and select all of the CAS servers (and their corresponding OAB (Default Web Site) Webfolder). As you can see, it is important that the CAS servers have a working OAB webfolder.  If not, the oab may not be available for download by end users. You can test if the oab folder exists by browsing to https://your_cas_server.domain.com/oab    You should get a 403 – Forbidden page

Basically, the OAB Virtual Web folder must be configured correctly as well. Use Get-OABVirtualDirectory and Set-OABVirtualDirectory to find and fix any issues. (Check InternalURL, ExternalURL and RequireSSL parameters). If the IIS folder does not exists, you can create it through the New-OABVirtualWeb cmdlet

Save the new OAB and run the ‘Get-OfflineAddressBook | FL’ command again.

Before doing anything else, verify that the Remote Registry Service is running. If the service is not running, the Exchange Server may encounter problems when doing RPC calls to itself, which could result in OAB (and other) problems.

Make sure the OAB gets generated/populated

Ok, the OAB now exists.

Open adsiedit.msc, open “Configuration”, and drill down to CN=Services, CN=Microsoft Exchange, CN=Messaging Organization, CN=Address Lists Containers, CN=Offline Address Lists

See if there is a child object CN=Default Offline Address Lists. Open the properties and note the objectGUID value.

We’ll talk about this value later on.

Let’s verify that the OAB is getting populated/refreshed ?  First of all, check the properties of the OAB and check the Update Schedule. Make sure the offline address book gets updated let’s say at least once per day.

The OAB will be created on the server that holds the active mailbox server role.  If you only have one mailbox server, that’s the one. If you are operating a cluster, it will be the active node.  That is, if the active Windows cluster node is also the active Exchange node.

So you have to verify what the active node is on both the Windows level and Exchange level :

Windows Cluster  : Open a regular command prompt and issue the “cluster group” command. Find the Cluster name under “Group” and note the name of the server in the “Node” column :

C:\>cluster group Listing status for all available resource groups: Group Node Status -------------------- --------------- ------ Available Storage MBSERVER02 Offline mbclust01 MBSERVER01 Online

In this example, the Windows cluster name is ‘mbclust01’ and the active node is MBSERVER01

In Exchange, the active Exchange cluster node can be found using the following powershell command :

Get-ClusteredMailboxServerStatus | FT Identity,OperationalMachines Identity OperationalMachines -------- ------------------- MBCLUST01 {MBSERVER02 , MBSERVER01}

As you can see in the output above, the Active node is set to MBSERVER02 and it should be MBSERVER01

You can fix this by running the ‘Move-ClusteredMailboxserver’ cmdlet :

Move-ClusteredMailboxServer -id MBCLUST01 -TargetMachine MBSERVER01 -MoveComment "Manual Move"

Verify that the correct node is active now :

Get-ClusteredMailboxServerStatus | FT Identity,OperationalMachines Identity OperationalMachines -------- ------------------- MBCLUST01 {MBSERVER01 , MBSERVER02}

Now open the registry on the mailbox servers, go to

HKLM\System\CurrentControlSet\Services\MSExchangeSA\Parameters\

and verify that there is a registry key value called “EnableOabGenOnThisNode” (Reg_sz) that contains the hostname of the active mailbox node. If that is not the case, create the value (String Value) and enter the hostname (not fqdn) of the active mailbox server.  What could happen, if the active node goes down (instead of being moved with Move-ClusteredMailBoxServerStatus) is that this registry key does not get updated.  Therefore, it is importand to always use Move-ClusteredMailboxServerStatus to move the active node (before you can take it down for maintenance for example).  The Failover Cluster Management console (in 2008, or the corresponding console in 2003) is not Exchange aware, and as you can see, moving the active cluster node to another server requires some Exchange specific tasks…

Anyways, this should take care of Event ID 9335.

Verify that the MS Exchange System Attendant service is running on the active mailbox server. Stop and Start it to make sure settings are re-read.

In the application event log on the mailbox server, you may see a MSExchangeSA Event log entry that indicates that the oabgen dll is started.

Now force the offline address book to get updated. You can use the GUI (right-click on the OAB and choose update) or via Powershell :

update-offlineAddressBook -id "

Wait a little while, open the Application Log and verify that OALGen process has successfully created a new Offline Address Book.  If you don’t like the brief amount of event log information, you may want to increase the verbosity/logging level for the OALGen process by running the following powershell command prior to running the “update-offlineaddressbook” cmdlet

You can enable verbose logging for the Offline Address List generator service by running the Set-EventLogLevel -id "MSExchangeSA\OAL Generator" -Level expert -verbose cmdlet (possible level values are Lowest, Low, Medium, High and Expert) Next, you can look for OAL Generator event log entries by using the following command : Get-EventLog Application | where {$_.Category –eq "OAL Generator"} | ft -Wrap > c:\oallog.txt

Open C:\Program Files\Microsoft\Exchange Server\ExchangeOAB\ and verify that

• a folder is created. This GUID must match with the objectGUID of the Offline Address List (see the actions we did using adsiedit earlier)
• this folder is populated with LZX files and oab.xml

This indicates that the offline address book has been generated/populated.

This folder (C:\Program Files\Microsoft\Exchange Server\ExchangeOAB) should be shared as ‘ExchangeOAB’, and the "Exchange Servers" group should have ‘List Folder Contents’ and ‘Read’ access.  Verify that your CAS server(s) are member of the "Exchange Servers" group, just to be sure.

Make sure the OAB gets distributed to the CAS server(s)

Take note of the GUID folder name on the mailbox server / active cluster node. On the CAS server, open C:\Program Files\Microsoft\Exchange Server\ClientAccess\OAB and verify that a folder with the same name exists. Again, this is the objectGUID of the Offline Address list.  If there are other folders as well but you only have one Offline Address List, feel free to remove the other folders (even if that means you have to remove all folders).

If the folder you are looking for does not exist, restart the MS Exchange File Distribution service on the CAS Server(s) (If you have multiple CAS servers, clean the ClientAccess\OAB folder on all CAS servers and restart the service on all CAS servers).  The MSExchangeFDS Service is responsible for picking up the files from the Mailbox server that holds the OAB (by connecting to the ExchangeOAB share, as explained above) and dropping it into the local OAB folder.

So basically, if you are in a single server setup, the OAB files will be copied over immediately to the OAB directory after the OAB has been generated. If you have a multi-server config (separate CAS and Mailbox server(s)), the OAB files will be copied to the CAS servers when :

• A new OAB is created
• A change to the OAB has happened
• The OAB Generation Server has changed
• The Default Polling Interval is up (this is 8 hrs by default).
• There are new OAB files (new binpatch.oab file has been created).

Look for event id 1008 in the application log (MSExchangeFDS) on the CAS Server : You should see something like this

Event Type: Information
Event Source: MSExchangeFDS
Event Category: FileReplication
Event ID: 1008
Date: 22/05/2009
Time: 2:16
User: N/A
Computer: OABServer
Description:
Process MSExchangeFDS.exe (PID=2484). Offline Address Book data synchronization task has completed successfully. OAB name: "Default Offline Address List", Guid:

If the folder/files are finally created, the OAB generation and update process are working fine.  Don’t forget to start monitoring the oab.xml file age so you can get warned if something goes wrong.  (You will need to monitor the oab.xml file on all CAS servers, of course)

If clients still cannot get the OAB, you will need to check

• the OABVirtualDirectory settings on the CAS servers
• certificates installed on the server vs  clients trusting the issuing/root CA that has issued the certificates, and the certifiicate containing the proper names/Subject Alternate Names of the CAS servers
• autodiscover settings

It may help executing an E-Mail Autoconfiguration test on the Outlook client and verifying that all URL’s that are returned for OAB files are accessible from the client.

If the URL returned by Autodiscover does not match with the guid that is available on the CAS/OAB folder, you may need to

• review mailbox database settings (“EMC” – “Server Configuration” – “Mailbox” – Client Settings tabsheet) and verify that the Default Offline Address Book field is configured and points to the correct OAB

Apply to al mailboxdatabases at once : Get-MailboxDataBase  | Set-MailboxDatabase -OfflineAddressBook “Default Offline Address List”. You may have to wait an hour or two before all settings are corrected. While things are getting updated, the client may not even get a OABURL parameter back from autodiscover. Just give it more time and it will work fine.

If that does not work, rework the entire configuration :

• remove the entire OAB setup and wait until AD is replicated
• create a new OAB and force it to update immediately. Verify that a new GUID folder is created on the mailbox server.
• Restart the File Distribution Service on the CAS servers and verify that the new GUID folder is copied from the mailbox server to the CAS server
• Reset the default offline address list on all mailbox databases.

 

You can find more information about the OAB Web Distribution and OAB Generation process in general at http://msexchangeteam.com/archive/2006/11/15/431502.aspx

© 2009 – 2021, Peter Van Eeckhoutte (corelanc0d3r). All rights reserved.

Similar/Related posts:

Outlook 2007 unable to download Offline Address Book – error 0X8004010F and 0X80190194 Exchange 2007 Administration : Policies and limits Exchange 2007 : Indexing and searching mailboxes Delegating Exchange 2007 Distribution List Management to End Users Exchange 2007/2010 : Renaming attachments ‘on the fly’ – custom transport agent