Juniper Screenos : Redundant multi-exitpoint ISP routing failover using multiple vrouters, multiple OSPF areas and eBGP

Published February 6, 2009 |

Introduction

As you most likely already know, Juniper screenOS supports a couple of dynamic routing protocols (OSPF, BGP, RIP). These protocols can be used to build very powerful and redundant networks, however there are some screenos specific issues with these implementations, and these issues may introduce a little bit of complexity in the design and configuration of your Juniper device.

In today’s scenario, we’ll look at setting up the following network topology :

The primary datacenter is the main location where servers, services and resources are hosted. In case of a disaster, the secondary datacenter will take over.

The primary datacenter and the secondary datacenter are interconnected with a direct link. Router-A and Router-B use BGP to exchange routing information. These 2 routers are managed by a third party ISP. The ISP only allows you, the customer, to talk OSPF with these routers. They have agreed with you to set up OSPF area 0 between their router and your Juniper firewall, in each datacenter.

Both the primary and secondary datacenter have an internet connection. The internet routers are managed by the ISP. The ISP uses BGP internally, but they don’t allow you, the customer, to talk BGP with these routers. The internet connection in the primary datacenter must be the first connection used. When the internet connection in the main site goes down (and only when the internet connection in the main site goes down) the internet connection in the secondary datacenter must be used. The ISP has agreed to set up OSPF area 1 between their router and your Juniper firewall. They have committed to advertise a default gateway into OSPF area 1 in each site. In the main site, this default gateway will have cost 20, in the secondary site this default gateway will have a higher cost. If a specific internet connection goes down, they will no longer advertise the default gateway for that link into OSPF.

The internet ISP does not want you to advertise anything back into OSPF area 1.

Each datacenter has a connection to a WAN. At the same time, 2 remote sites are connected to the WAN as well. All of the WAN routers are exchanging routing information with BGP. The goals is to make sure the WAN link in the primary datacenter is used first. When this link is down, the WAN connection in the secondary datacenter must be used as route to reach both the primary LAN and secondary LAN. The ISP has agreed to form eBGP between your firewall and their router, in each site. They do not want you, however, to advertise a default gateway to them, and you don’t want to a receive a default gateway from them.

All of the failover and failback scenario’s must happen dynamically and automatically. You cannot use static routes, all networks must be routed and advertised dynamically.

There are a couple of ways to get this to work, but in this scenario I’ll use do everything purely based on routing and routing protocols.

Scope of work

First, let’s have a closer look at the primary datacenter :

Based on the information from the introduction : this is what we know :

we must set up OSPF (area 1) between Juniper and internet ISP router, eBGP with the WAN provider router and OSPF area 0 with router-A (connection to the other datacenter). So we need to configure two OSPF areas and one BGP AS on the Juniper firewall.

The desired functionality for each area / protocol is :

OSPF area 1 : the ISP will advertise a default gateway when the internet connection is up.

eBGP : remote networks will be advertise to us, and we need to advertise our networks back

OSPF area 0 : the two routers will advertise all networks (connected and anything that is advertised into OSPF area 0) to all neighbors. We need to advertise routes coming from eBGP into OSPF area 0, and we need to advertise the default gateways learned via

OSPF 0 into area 1 as well.

Sound simple

Good news – Juniper screenOS supports all of the above areas and protocols. You can set up multiple areas in one Juniper device and BGP at the same time.

But if you put all of it into one vrouter, you will encounter the following problem :

At a certain point, you will have to redistribute routes learned via eBGP (from the remote locations) into OSPF area 0 and only into area 0. However, you cannot do this. In ScreenOS, you can only redistribute the routes into OSPF without making a differentiation per area. So when you try to add the routes to OSPF area 1, you will add the routes into OSPF area 0 as well, and this is not what we want. In fact, the ISP specifically told us not to do this. (And it would be a bad bad idea to advertise private ranges to an internet provider. Most ISPs block private ranges on their perimeter, but some allow private ranges in their internal networks. And if multiple clients would start advertising internal routes into the ISP backbone, it may become messy. Of course, you could ask your ISP if they could stop routes from being advertised into OSPF area 1, but it is a lot safer not having to rely on the ISP for this.

Solution ?

One way of doing this is by using 2 vrouters. If you had already set up your firewall with one vrouter, you can still change the setup. If you are still in design phase, then you’re good to go.

In this scenario, we’ll use the default “trust-vr” for eBGP and OSPF area 1 (because we need to advertise all internal/remote networks to each other anyway), and we’ll use a new vrouter called “internet-vr” for OSPF area 0.

This will allow us to properly distribute routes from eBGP into area 0 and routes from area 0 into eBGP, limiting the internal/remote networks to eBGP/OSPF Area 0 only.

In addition to this, we have to make sure to get the default gateway advertised as well, in a dynamic way. We cannot use a static default route in trust-vr that points to internet-vr (because it is static); and we cannot use the vrouter option to automatically advertise the default gateway to another vrouter either, because that option will result in putting a static route in trust-vr. The default route would not disappear when the default gateway is not being advertised anymore by the internet ISP. So we’ll export the default gateway manually.

So we need 2 vrouters, and we’ll create 3 zones : Public, WAN and LAN (each with one of the interfaces bound to the corresponding zone. Eth0/0 is LAN, Eth0/1 is WAN, Eth0/2 is Internet)

We need to follow the same steps for datacenter 2, but we need to increase costs/metrics so routes in datacenter 2 are only used as secondary route. (see later)

Setup primary datacenter

First of all, set up the new vrouter, create a zone and put the corresponding interface in the zone and set up IP addresses :

set vrouter name internet-vr

set zone name Public
set zone name WAN
set zone name LAN

set zone WAN vrouter trust-vr
set zone LAN vrouter trust-vr
set zone Public vrouter internet-vr

set int eth0/0 zone LAN
set int eth0/1 zone WAN
set int eth0/2 zone Public

set int eth0/0 ip 192.168.0.254/24
set int eth0/0 route
set int eth0/1 ip 192.168.136.1/24
set int eth0/1 route
set int eth0/2 ip 1.1.1.1/24
set int eth0/2 route

Set up OSPF with the internet provider (on vrouter internet-vr, via interface eth0/2). Despite the fact that I’m not using md5-authentication in this example, I would strongly advise you to use md5 authentication in this area (or all areas for that matter)

set vrouter internet-vr
set preference ospf 20
set router-id 1.1.1.1
set proto ospf
set enable
set area 0.0.0.1
exit
exit

set interface eth0/2 proto ospf area 0.0.0.1
set interface eth0/2 proto ospf enable

If the ISP has set up their side of the OSPF, a neighborship will be formed and the default gateway is now added into the RIB of vrouter internet-vr

Now we need to advertise this default route to trust-vr, making sure that this advertisement is dynamic. In order to do this, we need to use an access-list, route-map and export command :

set vrouter internet-vr set access-list 1 set access-list 1 permit default-route 10 set route-map name "DefGWToTrust" permit 10 set match ip 1 exit set export-to vrouter "trust-vr" route-map "DefGWToTrust" protocol ospf

This will make sure the default gateway is added to vrouter trust-vr and if the ISP does no longer advertise the default gateway, it will no longer get advertised to trust-vr either.

Since we have set the the ospf preference to 20 in the main site (internet-vr), the default gateway advertised by the ISP will have a cost of 20 (in the internet-vr vrouter). If you look at the routing table of internet-vr, you should see the default gateway, a host route (for 1.1.1.1) and a connected subnet route (for 1.1.1.0). If we want the default route to have a cost 20 in trust-vr as well, we need to set the preference for imported routes to 20 in trust-vr :

set vrouter "trust-vr"
unset auto-route-export
set preference imported 20

If you look at the routing table of trust-vr, you should see a default route, towards vrouter internet-vr, with a cost of 20.

get vrouter trust-vr route protocol imported

H: Host C: Connected S: Static A: Auto-Exported
I: Imported R: RIP P: Permanent D: Auto-Discovered
N: NHRP
iB: IBGP eB: EBGP O: OSPF E1: OSPF external type 1
E2: OSPF external type 2 trailing B: backup route

Total 20/max entries

         ID          IP-Prefix      Interface         Gateway   P Pref    Mtr     Vsys
--------------------------------------------------------------------------------------
*        32          0.0.0.0/0            n/a      internet-vr E1I   20     11     Root

Hooray, theoretically, we could start creating policies to allow access to the internet. Keep in mind : all interfaces are in route mode, so don’t forget to specify “nat src” in your policies between LAN and Public (or WAN and Public if that is what you want to do)

(I’ve kept the policies-part out of scope for this scenario, I’m sure you can figure this out yourself.)

Now we are ready to set up eBGP with the WAN provider. The WAN provider has assigned private AS number 65005 to your primary datacenter and private AS 65010 to the secondary datacenter. The WAN provider uses AS 65000 on their routers.

set vrouter trust-vr set router-id 192.168.136.1 set proto bgp 65005 set reject-default-route set enable set neighbor 192.168.136.2 remote-as 65000 set neighbor 192.168.136.2 md5-authentication BlahBlahBlahChooseAStrongMD5String set neighbor 192.168.136.2 reject-default-route set neighbor 192.168.136.2 enable

unset sync exit

Next, enable BGP on the WAN zone interface :

set int e0/1 proto bgp

The eBGP neighbors are now converging. When they are converged, the BGP neighbor should be in state ESTABLISH.

get vrouter trust-vr proto bgp neighbor 

Peer AS Remote IP       Local IP          Wt Status   State     ConnID Up/Down
--------------------------------------------------------------------------------------
  65000 192.168.136.2   192.168.136.1    100 Enabled  ESTABLISH      5 23:45:01

Next, set up OSPF area 1

set vrouter trust-vr
set proto ospf
set enable
set area 0.0.0.1

set int e0/0 proto ospf area 0.0.0.1
set int e0/0 proto ospf enable

Route Redistribution

First, we’ll redistribute routes learned via eBGP into OSPF (in vrouter trust-vr. Since there is only one area in this vrouter, all routes can and will be redistributed into this area). Since we want to keep the number of access-lists and route maps to an absolute minimum, we’ll also advertise the default gateway (imported from internet-vr) into OSPF at the same time

set vrouter trust-vr
set access-list 1 permit 0.0.0.0/0 10
set route-map name “InjectRoutesintoOSPF” permit 10
set match ip 1
exit
set proto osp
set redistribute route-map InjectRoutesintoOSPF proto bgp
set redistribute route-map InjectRoutesIntoOSPF proto imported
exit

Note that we do not need to redistribute the connected network in our case, because the provider that manages router-A and router-B is already advertising the connected routes to each other. If that would not be the case, you could simply add “set redistribute route-map InjectRoutesIntoOSPF proto connected” to the command list as well.

At this point, both the networks learned via BGP and the default gateway is advertised to OSPF area 1.

Now we need to advertise the routes learned via OSPF area 1 into BGP :

We won’t use an access-list or route-map for this, we’ll advertise the local networks to BGP via the “network” statement :

set vrouter trust-vr
set proto bgp
set network 192.168.0.0/24
set network 192.168.1.0/24
exit

(you don’t need an access-list to redistribute the routes, however if you want to alter BGP attributes, you will need an access-list and route-map)

Setup secondary datacenter

The setup in the secondary datacenter is almost exactly the same. The main differences are (assuming that you know that the router-id, neighbor ip’s , interface IP’s, remote-as numbers etc need to be set to the local addresses)

1. routes advertised from eBGP to OSPF area 1 must have a higher cost than the routes that are redistributed from eBGP into OSPF area 1 in the first datacenter. You can do this by setting the OSPF preference in vrouter trust-vr on the firewall in the 2nd datacenter, making sure that it has a higher value than the routes that arrrive from the first datacenter. Review the cost of the routes as they are injected by the provider from BGP to OSPF area 1 in the second datacenter and then make sure the preferences of OSPF area1 is higher than that value.

2. default route advertised must have a higher cost than the default gateway that is export from internet-vr to trust-vr and then redistributed into area 1. This can be achieved by setting the imported preference in vrouter trust-vr to a higher value than the value on the firewall in the first datacenter

3. routes advertised to the WAN provider (redistribution into eBGP) must be used as a backup route. This can be achieved by either asking your ISP to prepend the AS path (so the path is at least one AS longer than the path that is advertised via the first datacenter), or to use an access-list/route map for redistributing the networks into BGP and setting some attributes (a higher metric/med value) in the route-map. Either way should work.

Note : if you want to AS prepend routes yourself, you need to create an as-path-access-list in the bgp instance. so instead of creating an access-list in the vrouter, you need to enter the bgp config and create a as-path-access-list. Suppose you want to prepend 65010 twice to the eBGP redistribution in the secondary datacenter (so the AS_Path becomes larger than the path that is advertised via eBGP in the main datacenter), you can do this :

set vrouter trust-vr
set proto bgp 65010
set as-path-access-list 10 permit "65010 65010"
set neighbor 192.168.137.2 remote-as 65000
set neighbor 192.168.137.2 md5-authentication BlahBlahBlahChooseAStrongMD5String
set neighbor 192.168.137.2 enable
set neighbor 192.168.137.2 route-map "ASPrependBGP" out
set network 192.168.0.0/24
set network 192.168.1.0/24
exit

set vrouter trust-vr
set access-list 10 permit ip 192.168.0.0/24 1
set access-list 10 permit ip 192.168.1.0/24 2
set route-map name "ASPrependBGP" permit 10
set match ip 10
set as-path 10

The last statement refers to the as-path access list.

Make sure only to prepend your own AS number. If you are changing an existing neighbor, you may have to disable and enable the BGP instance before you will see the changes.

When the second datacenter has been set up, there will be 2 default gateways in OSPF area 1 : one with cost 20, pointing to datacenter 1, and one with a higher cost, pointing to datacenter 2. Furthermore, the routes to the remote networks will be in the routing table twice as well : one with a lower cost and one with a higher cost. And on the WAN, the networks from datacenter 1 and 2 will be added twice. Depending on the solution you have implemented, one will either have a higher metric or a longer AS path.

Note : in the example above, the AS prepend will be applied to all outgoing redistributions. You can also apply the route-map on an individual network instead, (e.g. set network 192.168.0.0/24 route-map “ASPrependBGP”)

Test failover

What will happen if

– internet goes down in datacenter 1 ?

The ISP no longer advertises the default gateway to OSPF area 0 (internet-vr). This default gateway is no longer exported to trust-vr and thus no longer redistributed into OSPF area 1. Only a couple of seconds after the internet connection went down, all traffic to the internet is redirected to the firewall in datacenter 2.

– WAN connection goes down in datacenter 1 ?

The 2 local network are no longer advertised via eBGP into the WAN network, and the remote networks cannot be advertised into eBGP to the primary datacenter. The WAN will now use the routes that are advertised from datacenter 2. All traffic between the networks from the two datacenters and the remote networks will use the WAN link in datacenter 2. This failover process takes a little longer (20 to 30 seconds), but it will work automatically.

If one of these connections becomes active again, the route advertisements will take place again, and the routing will be restored automatically. In case of internet traffic, this will happen quite fast. In case of the WAN connection, it will take a little longer. But again, it will work just fine.

Some Notes on track-ip

In case you want to use track-ip, keep in mind

– to specify a manage-ip on the interface that you want to use to monitor a remote IP address with. This IP must be different from the interface IP. If you don’t do this, traffic may failover to the backup route, but it may not get back online when the original link is active again.

– to specify a host route (/32) towards the IP that you want to monitor, and send traffic for this host towards the gateway that you are using when normal routing is up. Set the default route static but not permanent, and set the host route “static and permanent”. Sounds complicated, but it makes a lot of sense :

Suppose eth0/0 is connected to the internet. The default route (static, non-permanent) points to the internet router (1.1.1.1). The firewall has a second default route (with higher cost) that points to another router, in another zone if you want, via another interface. The idea is to bring down the default route when the primary internet link goes down, and to route traffic via the second router. When the primary link is active again, this original default route must be used again.

So you have decided to enable track-ip on ethernet0/0, pinging 3.3.3.3 (a host on the internet).

When the firewall cannot reach the track-ip IP address anymore (3.3.3.3), this interface will go down, and so will the associated default route. The second default route comes active and access to the internet is rerouted via the second link.

At this point, track-ip will also follow the routing table. So you would have 2 possible scenario’s :

– track-ip can now ping the monitored IP again via the backup link, and the interface will come active again (and so will the associated default route). But if the local internet connection is still down, track-ip will put the interface down again. This process will be repeated over and over again until the local internet connection is up again. So you’ll have a flapping default route, causing a lot of problems.

– track-ip can no longer ping the monitored IP, because the second route has another firewall that does not allow the monitored IP to be pinged, or the second route does not know how to handle traffic for the public IP on the firewall eth0/0 interface. EIther way, the original default route is down, the backup default route does not allow ping and track-ip will never be able to bring the interface up again. Result : traffic is rerouted via the backup route, but traffic will not be reverted back to the primary link when the primary link is up. This is a problem as well. This scenario will always occur if you only have one default route in your routing table.

Conclusion : you need to put in a static and permanent route. So if you are monitoring ip 3.3.3.3, and if your primary link uses ethernet0/0, via internet router 1.1.1.1, you need to add a host route :

set route 3.3.3.3/32 interface eth0/0 gateway 1.1.1.1 preference 20 permanent

This will ensure that the local route towards to monitored IP will still be up, even when the interface is down. Internet will be redirected towards the backup link (if any), and as soon as the local internet connection is up again, track-ip will be able to ping 3.3.3.3 and the interface will be brought online again. The default route will be put in place again, and traffic will be routed over the primary link again.

In case you don’t know how to set up track-ip :

First, set up a separate manage-ip on the interface

Next, enable track-ip :

set interface ethernet0/0 monitor track-ip ip 3.3.3.3 interval 5 threshold 7 weight 1
unset interface ethernet0/0 monitor track-ip dynamic
set interface ethernet0/0 monitor track-ip

The threshold is the number of pings that must fail before track-ip is in failed state.

The internal is the number of seconds to wait between two pings.

In our example, a ping will be performed every 5 seconds. When the ping has failed 7 times, the weight is set to 1 and the interface will go down.

You can specify multiple IP’s to ping. The results of all IP’s will be summed.

The default weight is 255. This means that the sum of the failures of all IP’s that are monitored must reach 255 to put the interface in a down state. In my example, the weigth is set to 1.

You can get the status and some statistics on track-ip using

get int e0/0 monitor track-ip

Final note :

This blog post does not intend to be the one-and-only solution for this network scenario. You could use track-ip and some other functionalities within screenOS to build similar redundant networks as well. It’s just an example of what you can do and how you should design your vrouters, routing protocol parameters, etc

Posted in Juniper, Networking | Tagged access-list, advertise, area, as prepend, as-path, as-path-access-list, bgp, ebgp, failover, Juniper, med, metric, multiple, multi_exit_disc, ospf, redistribute, redundant, route-map, screenos, track-ip, vrouter

Corelan Cybersecurity Research

:: Knowledge is not an object, it's a flow ::