Corelan Cybersecurity Research

(a.k.a. PVE Exch2007 Rerouter manual)

Keywords : Exchange 2007 outbound SMTP relay redundancy SendConnector outgoing mail high availability

Microsoft Exchange 2007 allows you to consolidate your messaging infrastructure and host more users on one server. Combined with the introduction of new server roles, Exchange 2007 allows for high levels of redundancy and resilience, ensuring uptime and availability.

However, there might be a specific yet not uncommon scenario where availability may not be guaranteed without administrator intervention.  This scenario is related to outgoing SMTP emails, using a third party relay server, and 2 sites that are connected over a (very expensive) WAN connection.

I’ll try to explain the problem and what we can do to automate failover

Note : I have intentionally left out some redundancy components such as clustering, because they are irrelevant in this post.

Let’s imagine this scenario :

Site A has a mailbox server, a HUB/CAS server, a local internet connection, and a third party SMTP relay server that is hosted in the DMZ

Site B has a standby mailbox server, a HUB/CAS server, a local internet connection with a different ISP, and a third party SMTP relay sitting in its own DMZ.

Building redundancy for incoming smtp emails from the internet is not very hard.  Each domain name has 2 MX records. One pointing to the relay server in site A, the other one pointing to the relay server in site B.

Each SMTP relay is configured to transfer incoming emails to the local HUB server(s).

But there is a problem with outgoing emails.  Each HUB server has a SendConnector that points to the local SMTP 3rd party relay server in the local DMZ. But what will happen if the local internet connection in site A goes down, and the SMTP relay server is still up and running ?  The HUB server in Site A is not aware of the internet connection state, and will still continue to route outgoing emails to the local SMTP relay.  Emails will be queued until the connection becomes available again.  But sometimes you don’t know when the connection will be up and running, and you may not even be able to change routing configuration because the internet connection went down at night and you cannot VPN into the office to solve the problem.

There are some ways to overcome this problem, but most of them require manual intervention, or will make your mail routing topology somewhat cluttered and obscure.

So I came up with the idea to build an application that allows you to monitor the internet connection, disable SendConnectors and restart the Transport Service.  That way, you can temporarily disable the SendConnector to the local SMTP relay, so emails are routed to the HUB in SiteB and then transferred to the second SMTP relay, automatically.
And as soon as the internet connection in Site A is back, the connectors can be changed back to their original state.

This application is currently in beta phase, however I have decided to make it available for download already, so you can start testing and see if my concept works for you. 

The application can be downloaded at http://users.telenet.be/internet.activities/freetools/exch2007rerouter

You should install the application on the HUB server that has the SendConnector to your SMTP relay. The HUB server will need to be able to either ping hosts on the internet or connect to websites on the internet (depending on how you set up the internet connection check parameters in the Configuration of the tool). Additionally, the application uses PowerShell and Exchange specific cmdlets, so make sure the Management Shell is installed on the HUB server.

After installing the application, you will find these 2 components :

1. A service, called "PVE Exch2007 Rerouter". The startup state is set to manual, so don’t forget to change the state to "Automatic" yourself.  Don’t try to start the service yet, you will need to run some Configuration settings first.

First of all, change the credentials for running this service from "Local System" to an account that has administrative access to the Exchange environment. (The account should have permissions to disable/enable the SendConnectors)

2. A configuration tool. If all goes well, you should see an icon the the Configuration tool on your Desktop.

Send Connector actions

While logged on with a user account that has access to the Exchange environment, launch the Configuration Tool. While the application starts, the list of SendConnectors will be obtained using PowerShell, so it might take a couple of seconds before the application is displayed.

First, you need to think about the purpose of the Send Connectors. Pick the SendConnector that allows the transfer of emails to the internet towards your local SMTP Relay server. 

When the internet connection is down, you would want to disable that connector. Select the connector in the upper part, right click and choose "Disable Connector".

When the internet connection becomes available again, you’d want to re-enable this connector. Select the connector in the lower part of the window, right click and choose "Enable Connector".

Internet connection detection

Next, you need to specify how the application can determine whether the internet connection is up or down.

Open the "Internet Connection Detection" tabsheet

Right-click in the listview and choose "Add" to create a new connection check. You can add 3 types of tests : either a ping to an IP address or hostname, perform a HTTP connect to a webpage (http or https) or do a tcp connect to an open port. (Make sure to enter the correct syntax for this last type of test. You need to specify the hostname, followed by :, then followed  by the port number)

Make sure to pick target hosts that are thrustworthy and make sure that you have received permission to use these hosts as a target, if applicable.

The set of connection detection checks will determine whether the internet connection is down.

If you get a warning about the fact that the config file cannot be saved, then the service probably has locked the config file. Either wait a couple of seconds and try again, or stop the service, save the changes, and start the service again (which is in fact the recommended way)

General settings

In the third tabsheet, you can change the behaviour of when and how the tool should decide whether a connection check has succeeded or not, and whether the internet connection should be considered up or down.

Ping Settings :

– Set number of pings that should be sent to each target host.
– Define the minimal number of pings that should work, otherwise the test will conclude that the ping has failed
– Set the ping timeout

Internet Connection Detection Behaviour :

– Consider internet connection down when …. or more tests failed.  This setting is related to the entire set of connection checks that are defined in the second tabsheet. Suppose you have defined 3 checks. If you want to have the tool decide that the internet connection is down when 2 or more out of these 3 checks have failed, then set this value to 2
– Number of internet connection check runs before state can be changed : If you want to make sure the internet connection is really down, you can combine the results of multiple ‘runs’/iterations to really make sure the line is down.  The results of each set of connection checks will then be gathered. 
– The next setting "Within those runs, number of failed results…"  allows you to set a threshold within the iterations.
– The 4th setting allows you the specify the frequency (or interval) that is used by the tool to run the connection checks.

Ok, all of this may sound a bit confusing, so let’s create an example.

Suppose you have set the parameters to the same values as in the screenshot above, and suppose you have created 3 connection checks.  So when will the connection state change from "Internet is up and running" to "Internet connection down" ?

Hour 0:00 : Run 1
3 tests are run. If 2 or more of these tests fail, then remember that run 1 out of 2 has failed.
Hour 0:02 : Run 2
3 tests are run. If 2 or more of these tests fail, then remember that run 2 out of 2 has failed.

So 2 runs have been made, which is the minimum to trigger a state change.   2 out of these 2 runs have returned "failed connection", so the state is changed to down.

Maintenance window:

You can prevent the tool from checking connection states during a given interval. Specify the start and end time, and choose the days of the week on which the windows should be active.

Version updates :

It is strongly advised to turn on "Automatically check for updates". If you have enabled admin alerts and specified SMTP settings (see below), then you will be notified when a new version is available.  Version update checks will happen every 10 hours.

Logging and Alerting

If this is not clear yet, you can use the "Logging and Alerting" section, and enable "Log internet connection changes in event log", and enable "Verbose logging in event log" so you can see how the application works.

Open the Application Event Log and you’ll see every step that is taken by the application to determine line state, and will show you the actions it has taken based upon these results.  Make sure to disable the verbose logging again, as this will fill up your event log very fast.

If you want email reports, then enable "Send admin alerts to (email address):" and enter a valid email address. You can specify multiple addresses by separating the addresses with semi-colon.

Next, at the lower part of the tabsheet, you’ll need to specify the SMTP settings. Use the "Test SMTP connection" button to verify that the settings are correct.

Don’t forget to SAVE the changes.  After saving the changes, you should restart the service  (or just start the service if this is the first time configuration). Have a look at the event log and make sure the application has not generated any errors.

Final note : make sure to start the service when the internet connection is up.  When the service is started, the tool assumes that the connection is down. Shortly after starting the service, the status should go from DOWN to UP.  If you have enabled admin alerts, you will get an email stating that the state has changed.  This proves that the tools works and that admin email alerts are configured correctly.

© 2008 – 2021, Peter Van Eeckhoutte (corelanc0d3r). All rights reserved.

Similar/Related posts:

Free tool – PVE SMTP Open Relay Tester Free tool – Attachment filtering with Exchange 2007/2010 (custom transport agent) Free tool – Free POP3 Collector Exchange 2007 Administration : Antispam and Content Filtering Free tool : Windows 2003/2008 Certificate Authority Certificate List Utility for pending requests and about-to-expire certificates