Juniper ScreenOS : defeating iBGP full mesh requirement using route reflectors and confederations

Published March 22, 2009 |

As explained in one of my earlier posts, one of the requirements to successfully setup and operate an iBGP configuration is that all iBGP clients need to have a BGP connection to all other iBGP clients. (= full mesh). This is required because an iBGP device only exchanges information about its own networks and it does not pass on BGP updates from other peers to other peers.

Suppose you have 4 devices in the same AS, exchanging prefix information over BGP, then you would need 6 iBGP connections.

If you add one more router in the AS, you need 4 more iBGP connections

In other words, adding more devices increases the number of peers exponentially. It is clear that this does not scale very well. Of course, you can use peer-groups to somewhat simplify the configuration of the peers, but you still need to configure neighbors to all other routers on all devices.

Basically, the setup on each device (screenos) would look something like this :

set vrouter trust-vr
set protocol bgp 65500
set enable
set neighbor peer-group "iBGPPeers"
set neighbor peer-group "iBGPPeers" remote-as 65500
set neighbor peer-group "iBGPPeers" md5-authentication "iBGPTest"
set neighbor 192.168.0.30 peer-group iBGPPeers
set neighbor 192.168.3.10 peer-group iBGPPeers
set neighbor 192.168.21.15 peer-group iBGPPeers

so you basically need to create a peer group, set the parameters that will be applied to all peers in this group (remote-as and md5-authentication parameters) and then you define the peers into the peer group. If you are using route-maps, you can either define the route-map in the peer group, and/or directly on a specific peer. If you define a route-map directly on a peer, it will take precedence over the global route-map in the peer-group.

Nice, but using peer-groups only limits the number of lines of screenos statements, it does not defeat the full mesh requirement. Some critics might even say that using peer-groups makes it more difficult when you are using multiple route-maps.

A lots of work, risk of making mistakes…. there are numerous reasons to consider when designing your network. But if using iBGP is your only option, how can you make your life easier ?

There are 2 solutions. You can use route reflectors and confederations to decrease the number of peer relationships you need to configure and maintain. Both solutions have some drawbacks too, but we’ll talk about that when we get there.

Route reflectors

A route reflector is a router that acts as a hub between various iBGP clients. These clients peer with the RR (Route reflector) and exchange information with it. The RR reflects the information to other BGP peers (iBGP and eBGP)

Route reflectors introduce the concept of ‘clients’ and ‘non-clients’. When a host is connected to a RR, it becomes a client. The RR and the clients now form a cluster. Peers of the RR that are not part of the cluster are non-clients.

Non-clients still need to be configured to form a full mesh with the RR and the other iBGP speakers (but not with the other clients that are part of the RR cluster)

So in other words, if we look back at the example of 5 routers, and now assume that router5 is a RR, and router 2 and 4 are RR clients, then the iBGP setup would look like this :

router 1 and 3 now only have to peer with each other and with the RR. RR clients 2 and 4 only need to peer with router5 (the RR). So instead of having to configure 10 iBGP peers, you only need to set up 5 relationships.

The drawbacks of using a RR are clear : the RR is now a sinle point of failure and the RR process will put somewhat more load on the router. Of course, you can set up multiple RR’s for added redundancy (you just need to set up all clients and non-clients to peer with all RRs)… and as long as you don’t have thousands of routes exchange via BGP, the extra load should not be a big issue either. And if you have more routes, you may need to reconsider the design and split up the network (AS) into smaller networks (ASs) and use eBGP to exchange information between the various networks.

Example :

(Note : a RR will only reflect the routes within the same AS. So if you peer RR’s from multiple ASs with each other using eBGP, you will still need to redistribute prefixes from ebgp into ibgp on the RRs in order to get the information exchanged to the other routers in the AS.)

Route reflectors will preserve all iBGP attributes, however some implementations still allow attributes to be changed or filtering to be applied. I would not recommend changing attributes unless you really know what you are doing. If, for example, you start playing with the next_hop attribute, you may introduce a routing loop. On the other hand, it may be nice to be able to play with the local_pref or med values in order to prefer one path over another.

It’s also good to know that changing from a topology without a RR to a RR topology does not introduce a major design or configuration change. AS numbers still remain the same. You only need to set up the RR, set up the RR clients, and then you can start removing iBGP peer relationships.

To a non-client, the RR just acts as a regular BGP peer, so even if the non-client is not RR aware of cannot handle RR configurations, as long as it can speak iBGP it will work. It will never know that the peer is a RR.

This is how a RR network is set up in screenos :

1. The route reflector

set vrouter trust-vr
set proto bgp
set reflector
set reflector cluster-id 1

That’s it. Enable the reflector and set the cluster-id (to 1 in this example). You can get the reflector settings using the “get vrouter trust-vr proto bgp reflector” statement :

get vrouter trust-vr proto bgp reflector

Route reflector:       enable
Cluster ID

Now change the iBGP relationship with the RR clients (and add a statement to make them reflector client) :

set vrouter trust-vr
set proto bgp
set neighbor  reflector-client

(if you were already using peer-groups, you can set the reflector-client configuration on the peer-group and it will be applied to all neighbors)

2. RR clients

There’s nothing special to configure on the RR clients. Just set up the iBGP peer relationship with the RR and that’s it. You can remove the peer relationships with the other RR clients and with the non-RR clients. RR clients only need to talk to the RR.

When the client has been configured to peer with the RR, it becomes a reflector client. You can get query this info by running the following command on the client:

get vrouter trust-vr protocol bgp neighbor  | include reflector

    reflector client: yes

On the RR, you can also get the same info by running :

get vr trust-vr proto bgp neighbor 192.168.0.127

peer: 192.168.0.127,  remote AS: 65000, admin status: enable, PeerID 2, CBAfi 0
type: IBGP
connection state: ACTIVE, connection id: 0 retry interval: node default(120s), cur retry time 120s
configured hold time: node default(20s), configured keepalive: node default(6s)
designated local IP: n/a
--------------------------------------------------------------------------------------
For address family: IPv4 unicast
route map(ipv4) in name: , route map(ipv4) out name: 
weight(ipv4): 100 (default)
self as next hop(ipv4): disable
send default route to peer(ipv4): disable
ignore default route from peer(ipv4): disable
send community path attribute(ipv4): no
reflector client(ipv4): yes
configured adv-interval(ipv4): default(5s)
--------------------------------------------------------------------------------------
For address family: IPv6 unicast
route map(ipv6) in name: , route map out(ipv6) name: 
weight(ipv6): 100 (default)
self as next hop(ipv6): disable
send default route to peer(ipv6): disable
ignore default route from peer(ipv6): disable
send community path attribute(ipv6): no
reflector client(ipv6): no
configured adv-interval(ipv6): default(5s)
--------------------------------------------------------------------------------------
Neighbor Capabilities: 
  Route refresh: advertised 
  Address family IPv4 Unicast:  advertised 
force reconnect is disable
total messages to peer: 234, from peer: 206 
update messages to peer: 30, from peer: 5
Tx queue length 0, Tx queue HWM: 5
route-refresh messages to peer: 0, from peer: 0 
current packets from peer number: 1, length: 21 
last reset 00:29:24 ago, due to BGP recv Notification(Cease: Admin stopped)(code 6 : subcode 3)
number of total successful connections: 5
peer down elapsed time: 1 minutes 53 seconds

(the RR client is at 192.168.0.127, the RR is 192.168.0.8). In fact, this client is a linux router running quagga.

As opposed to screenOS, on Cisco (or quagga) you do need to specify that the client is a route reflector. You can set up the reflector-client using the following commands :

router bgp 65000
   bgp router-id 1.1.1.1
   neighbor  remote-as 65000
   neighbor  route-reflector-client

In quagga (and Cisco), you can verify that the client is a rr-client using the sh ip bgp neighbors command :

[root@linuxrouter-1 ~]# telnet localhost 2605
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Hello, this is Quagga (version 0.99.10).
Copyright 1996-2005 Kunihiro Ishiguro, et al.


User Access Verification

Password: 
linuxrouter-1> en
Password: 


linuxrouter-1# sh ip bgp neighbors
BGP neighbor is 192.168.0.8, remote AS 65000, local AS 65000, internal link
  BGP version 4, remote router ID 192.168.0.8
  BGP state = Established, up for 00:04:34
  Last read 00:00:03, hold time is 20, keepalive interval is 6 seconds
  Neighbor capabilities:
    4 Byte AS: advertised
    Route refresh: advertised and received(old & new)
    Address family IPv4 Unicast: advertised and received
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                1          6
    Keepalives:            47         45
    Route Refresh:          0          0
    Capability:             0          0
    Total:                 49         52
  Minimum time between advertisement runs is 5 seconds

 For address family: IPv4 Unicast
  Route-Reflector Client
  4 accepted prefixes

  Connections established 1; dropped 0
  Last reset never
Local host: 192.168.0.127, Local port: 53411
Foreign host: 192.168.0.8, Foreign port: 179
Nexthop: 192.168.0.127
Nexthop global: fe80::a00:27ff:fe6f:9ac
Nexthop local: ::
BGP connection: non shared network
Read thread: on  Write thread: off

3. non-RR clients

There’s nothing special to configure. Just remove the peer relationships with the RR clients. non-RR clients only need to peer with each other and with the RR.

Confederations

BGP confederations offer a second way to limit the number of peer relationships in a iBGP environment. This technique is based on the concept of breaking an AS into smaller sub-ASs. Peers in the sub-AS need to form a full mesh. Information between multiple sub-ASs is exchanged via eBGP (so you can dedicate one router in one sub-AS to exchange information with one router in the other sub-AS). To the outside world, the entire set of sub-ASs still look like one AS.

Although eBGP is used between sub-ASs, BGP attributes are preserved while crossing the sub-AS boundary. So attributes such as med, local_pref, next_hop are not changed (as if you were doing iBGP)

There are 2 major drawbacks in this scenario :

1. Migrating from iBGP full mesh to confederations require a major reconfiguration of all routers

2. BGP uses the shortest path to determine the best route. Because an AS can consist of multiple sub-ASs, but looks like one AS path hop to external BGP peers, you can get suboptimal routing. (after all, the sub-ASs do not influence the AS path length to external BGP peers). You can overcome this by manually setting the local_pref and med attributes on the eBGP peers.

This is how confederations are set up in screenos :

If BGP was already running, you need to break it down entirely. After all, you will need to put the bgp instance in a new AS (a sub-AS)

When BGP is entirely gone from the config, you need to set 3 different AS numbers :

Local AS (= the local sub-AS number). This number must be used as the local bgp AS number
Confederation ID = the real (parent or public) AS number (the AS that is used to the outside world)
Confederation peers : list of other sub-ASs within the same confederation.

So if you want to break down AS 65000 into 2 smaller sub-ASs (65100 and 65200), you need to set the following parameters :

set vrouter trust-vr proto bgp 65100
set vrouter trust-vr proto bgp confederation id 65000
set vrouter trust-vr proto bgp confederation peer 65200
set vrouter trust-vr proto bgp enable

You need to perform the same config on all devices in the same confederation and sub-AS

Next, you need to set up the bgp neighbors, using the sub-AS numbers (65100 and 65200). Within the sub-AS, there must be a full mesh.

If you want to interconnect the various sub-ASs, you can use one router in Sub-AS 65100 and one router in Sub-AS 65200 and set up a eBGP relationship between the two.

eBGP hosts (in another AS than confederation AS 65000) can peer with one of the hosts in the confederation using AS number 65000. These external hosts are not aware of the sub-ASs.

Route reflectors versus Confederations

As you probably have deducted by now, it is recommended to use route reflectors over confederations. On the other hand, confederations may help increasing stability. If you are running an IGP (OSPF, …) in one sub-AS, you can run it independently from other sub-ASs and you can use summarization when exchanging routing information between sub-ASs… This will help decreasing the number of route updates between subASs (and thus stabilize the parent AS) in case the networks connected via the OSPF are not 100% stable.

Posted in Juniper, Networking | Tagged as, autonomous system, bgp, bgpd, Cisco, cluster-id, confederation, ebgp, full mesh, ibgp, ios, Juniper, Linux and Unix, neighbor, path, peer-group, quagga, reflector-client, remote-as, route reflector, route-reflector-client, rr, screenos, zebra

Corelan Cybersecurity Research

:: Knowledge is not an object, it's a flow ::