IPSec VPN between Windows Server 2008 and Juniper ScreenOS

Published January 11, 2009 |

In this blog post, I will show you how to set up a IPSec VPN tunnel between a Windows Server and a Juniper ScreenOS based firewall and route traffic between hosts that are located behind these 2 VPN gateways.

The Windows Server will acts as a gateway to build a VPN tunnel towards the Juniper firewall, so the hosts behind the Windows Server can access hosts behind the Juniper firewall.

We’ll assume the following network layout :

Network 192.168.10.0/24 is located behind a multi-homed Windows 2008 Server (2 network interfaces, one with a private IP (192.168.10.254) and one with an internet public IP (I’ll use 192.168.0.0/24 as ‘internet’, so the ‘public’ IP of the Windows Server is 192.168.0.100. Of course, this is not really a public IP, and the Juniper’s public IP will be in the same range, but it is not relevant to my explanation today. Just bear in mind that both the Juniper and the 2008 Server have a public IP. Traffic to the internet, initiated by these hosts, are not subject to address translation (no NAT). They are directly connected to the internet. If you feel not confident enough putting the Windows server directly to the internet, you can also put it in a DMZ. Just make sure it has a public IP which is routable, and no NAT is used.
Network 192.168.2.0/24 is located behind a SSG5 Juniper firewall. Eth0/1 is connected to the internet (192.168.0.30) and eth0/2 is connected to the private network (192.168.2.0/24).
The goal is to allow hosts 192.168.10.1 and 192.168.2.1 to talk to each other, using an IPSec tunnel between the Windows Server and the SSG firewall.

The 2 zones on the screenOS are “Public” (for the internet facing interface) and “Lan”. If you are using Juniper equipment that does not support zones other than the default trust and untrust, feel free to replace “Public” with “untrust” and “Lan” with “trust”

The Juniper firewall is a ssg5, running screenos 6.2 (but this may work with older versions as well)

Shared configuration parameters :

Phase 1 : 3DES – SHA1, DH Group2. PSK : DNCSQLsdjqkl390DNJldapfSDx
Phase 2 : ESP – 3DES – SHA1. We won’t use PFS to start with. (In fact, this option is not available in the Windows 2008 GUI. You can use PFS… Just scroll down to the bottom of this post in order to figure out how you can do this)
If you’re not sure what these Phases or parameters mean, check out this post and this discussion on juniperforum.com (In case you are wondering : my nickname on juniperforum.com is c0d3r)

ScreenOS configuration :

We’ll use a route based VPN for this purpose, so we need to create a tunnel interface. We’ll use an ip unnumbered tunnel interface for this. The tunnel interface will be placed in the Public zone.

set int tunnel.1 zone Public
set int tunnel.1 ip unnumbered interface eth0/1

get int tun.1
Interface tunnel.1:
  description tunnel.1
  number 20, if_info 1768, if_index 1, mode route
  link down
  vsys Root, zone Public, vr trust-vr
  admin mtu 1500, operating mtu 1500, default mtu 1500
  *ip 0.0.0.0/0  unnumbered, source interface ethernet0/1
  *manage ip 0.0.0.0
  pmtu-v4 disabled
  ping disabled, telnet disabled, SSH disabled, SNMP disabled
  web disabled, ident-reset disabled, SSL disabled

  OSPF disabled  BGP disabled  RIP disabled  RIPng disabled  mtrace disabled
  PIM: not configured  IGMP not configured
  MLD not configured
  NHRP disabled
  bandwidth: physical 0kbps, configured egress [gbw 0kbps mbw 0kbps]
             configured ingress mbw 0kbps, current bw 0kbps
             total allocated gbw 0kbps

Next, we need to create a Phase1 definition. We’ll use the GUI for this :

Go to “VPNs”, “AutoKey Advanced”, “Gateway” and create a new gateway. Set a gateway name, fill out the public IP of the other side (192.168.0.100 in our case) and then click advanced

In the advanced screen, select the following options :

– fill out the Preshared Key, set the outgoing interface also to eth0/1, and set the P1 proposal to pre-g2-3des-sha

– make sure the P1 Mode is set to Main

Click “Return” and then “OK” to save the changes

In the menu on the left, go to “VPNs”, “AutoKey IKE” and create a new defintion. This is Phase2

Pick a name, select the predefined “Phase1” that we have created earlier and click “Advanced”

Under advanced :

Set the Phase 2 proposal to nofps-esp-3des-sha, Bind the VPN to the tunnel interface, and set the proxy ID. The Local IP and netmask is the local network (192.168.2.0/24), the Remote IP and netmask indicates the remote network (192.168.10.0/24). Make sure the service is set to ANY. Click “return” and then “OK” to save.

Now we can set up the routing and firewall policy

Add a route to the 192.168.10.0/24 network and send it to the tunnel interface :

set route 192.168.10.0/24 int tunnel.1

Add the necessary policies so all traffic between the two networks is allowed :

set address LAN LocalNetwork 192.168.2.0/24     
set address Public RemoteNetwork 192.168.10.0/24
set policy from LAN to Public LocalNetwork RemoteNetwork any permit
set policy from Public to LAN RemoteNetwork LocalNetwork any permit
save

Configure Windows 2008 Server

First of all, make sure the Windows 2008 Server is configured as a router. (you can configure this via “Routing & Remote Access” – which can be installed as part of the “Network Policy and Access Services” server role) Otherwise, the hosts behind the server would not be able to route traffic via the server, thru the tunnel, to the hosts behind the remote endpoint of the tunnel.

Next, create the VPN definition :

Open the firewall settings on the Windows 2008 Server

(Server Manager – Windows Firewall properties)

Go to the “IPSec Settings” tabsheet and click Customize

Under “Key Exchange (main mode)”, choose Advanced and click the Customize button

Set 3DES-SHA as first Security Method in the list

Verify that the Key Exchange method is set to Diffie-Hellman Group 2 (which should be the case by default)

Verify that the key lifetime is set to 480 minutes (which is 28800 seconds, the default setting on Juniper)

Click “OK” to save

Under “Data Protection (Quick Mode)” (= Phase 2), choose “advanced” and click “customize”

Look at the right hand list (Data integrity and encryption) and make sure ESP – SHA1 – 3DES is set to the first one in the list. Verify that the Key Lifetime is set to 60 minutes ( = 3600 seconds, default setting on Juniper)

As you can see, the Key Lifetime setting in Windows Server indicates both a time (60 minutes) and Kbytes (100000). In Juniper, the default is 3600 seconds, 0 bytes. In other words, if you really want it to match (which is a requirement), you’ll have to create your own P2 proposal on Juniper so the settings match :

Click “new” to create a new proposal

Click OK to save

Now edit the AutoKey Ike definition (Phase 2) on Juniper and select this newly created proposal (under “Advanced”)

Click “Return” and “OK” to save

Let’s continue with the Windows 2008 Server Setup

Enable “Require encryption for all connection security rules that use these settings”

(otherwise, you will see stuff like this in a “debug ike detail” : )

## 2009-01-11 01:30:09 : IKE<192.168.0.100> proto(3), esp(11), auth(2), encap(1), group(0)
## 2009-01-11 01:30:09 : IKE<192.168.0.100> expect [0]:
## 2009-01-11 01:30:09 : IKE<192.168.0.100> atts<00000003 00000000 00000003 00000002 00000001 00000002>
## 2009-01-11 01:30:09 : IKE<192.168.0.100> proto(3), esp(3), auth(2), encap(1), group(2)
## 2009-01-11 01:30:09 : IKE<0.0.0.0        >   Check P2 Proposal
## 2009-01-11 01:30:09 : IKE<192.168.0.100> SA life type = seconds
## 2009-01-11 01:30:09 : IKE<0.0.0.0        >   SA life duration (TLV) = 0x 00 00 0e 10   
## 2009-01-11 01:30:09 : IKE<192.168.0.100> SA life type = kilobytes
## 2009-01-11 01:30:09 : IKE<0.0.0.0        >   SA life duration (TLV) = 0x 00 01 86 a0   
## 2009-01-11 01:30:09 : IKE<0.0.0.0        >   encap mode from peer = 1.
## 2009-01-11 01:30:09 : IKE<0.0.0.0        >   encap mode after converting it to private value = 1.
## 2009-01-11 01:30:09 : IKE<192.168.0.100> Phase 2 received:
## 2009-01-11 01:30:09 : IKE<192.168.0.100> atts<00000002 00000003 00000000 00000002 00000001 00000000>
## 2009-01-11 01:30:09 : IKE<192.168.0.100> proto(2), ah(3), auth(2), encap(1), group(0)
## 2009-01-11 01:30:09 : IKE<192.168.0.100> expect [0]:
## 2009-01-11 01:30:09 : IKE<192.168.0.100> atts<00000003 00000000 00000003 00000002 00000001 00000002>
## 2009-01-11 01:30:09 : IKE<192.168.0.100> proto(3), esp(3), auth(2), encap(1), group(2)
## 2009-01-11 01:30:09 : IKE<0.0.0.0        >   Check P2 Proposal
## 2009-01-11 01:30:09 : IKE<192.168.0.100> SA life type = seconds

Click “OK” to save

These are the new default settings for IPSec rules.

In order to encrypt traffic and set up the tunnel, we need to set up a Connection Security Rule.

Click “New Rule”

Choose “tunnel connection”

Fill out the tunnel parameters :

Computers in Endpoint 1 = the local subnet behind the Windows Server

The local tunnel computer = the public IP of the Windows Server

The remote tunnel computer = the public IP of the Juniper

Computers in Endpoint 2 : the remote subnet behind the Juniper firewall

Set the Preshared Key

“When does this rule apply” : leave all options checked (unless you know what you are doing)

Pick a name for the rule and click Finish

Add a route so traffic is sent to the internet. (the default gateway probably points to the internet, so it may not be required to add a route). In my testlab, I have set a static route pointing directly to the Juniper router, but in real life, you’ll probably want to point the route to your local ISP router.

In my testlab, I had to reboot my Windows 2008 Server before everything started working correctly, but it might not be necessary to do this

From the client behind the Windows Server, which has its default gateway set to 192.168.10.254, try to ping the client behind the Juniper firewall, which has its default gateway set to 192.168.2.8 :

From the client behind the Juniper, ping the host behind the Windows Server :

After the first packets were sent from the host behind the WIndows 2008 Server to the host behind the Juniper, you should see Phase 1 and Phase 2 (active sa) come up on the Juniper firewall :

Phase 1

get ike cookies 

IKEv1 SA -- Active: 1, Dead: 0, Total 1

1102f/0003, 192.168.0.30:500->192.168.0.100:500, PRESHR/grp2/3DES/SHA, xchg(2) (IKE_to_Windows2008/grp-1/usr-1)
resent-tmr 322 lifetime 28800 lt-recv 28800 nxt_rekey 28531 cert-expire 0
initiator, err cnt 0, send dir 0, cond 0x0
nat-traversal map not available
ike heartbeat              : disabled
ike heartbeat last rcv time: 0
ike heartbeat last snd time: 0
XAUTH status: 0
DPD seq local 0, peer 0


IKEv2 SA -- Active: 0, Dead: 0, Total 0

Phase 2

get sa active
Total active sa: 1
total configured sa: 1
HEX ID    Gateway         Port Algorithm     SPI      Life:sec kb Sta   PID vsys
00000001<   192.168.0.100  500 esp:3des/sha1 37703021  3218 unlim A/-    -1 0
00000001>   192.168.0.100  500 esp:3des/sha1 68b2fde4  3218 unlim A/-    -1 0

On the Windows Server, you should see the corresponding Phases :

Phase 1 (Main Mode)

Phase 2 (Quick Mode)

Juniper screenos logs :

Notes :

1. If, for any reason, you get “Received notify message for DOI <1> <1> ” messages in your “debug ike detail” output, clear the ike cookies, and wait some time until the corresponding ike cookies are cleared on the WIndows Server. Generate traffic again and it should work again (I’m still trying to figure out what exactly may be causing this)

2. If the tunnel does not work, you can try disabling policy-checking “unset ike policy-checking” and/or set the device to accept all proposals “set ike accept-all-proposals”. Don’t forget to disable this setting after you’ve figured out which proposals need to be used.

3. Windows 2008 does not support PFS (Phase2) to be configured via the GUI. You can only enable PFS when configuring the Connection Security Rule via command line :

rem Script taken from Peter Van Eeckhoutt’s Blog
rem http://www.corelan.berem Main Mode (Phase 1)
netsh advfirewall set global mainmode mmsecmethods dhgroup2:3des-sha1
rem
rem Quick Mode (Phase 2)
rem
netsh advfirewall consec add rule name="IPSec tunnel to 192.168.2.0 network behind juniper"     
     enable=yes
     mode=tunnel
     localtunnelendpoint=192.168.0.100
     remotetunnelendpoint=129.168.0.30
     endpoint1=192.168.10.0/24
     endpoint2=192.168.2.0/24
     action=requireinrequireout
     auth1=computerpsk
     auth1psk="DNCSQLsdjqkl390DNJldapfSDx"
     qmsecmethods=esp:sha1-3des
     qmpfs=dhgroup2

Don’t forget to enable PFS on the Juniper if you have used this script.

– The second command was put on multiple lines in order to allow you to see all parameters. If you want to copy/paste the command, make sure to put all parameters on one line

– the “qmpfs=” parameter is responsible for enabling PFS. You can either specify a DH Group, or just use “mainmode”, which will enable the use of the DH group defined in main mode (Phase 1)
– You can specify multiple proposals for both Phase1 and Phase2. Just separate them with a comma.

Posted in 001_Security, Juniper, Networking, Windows Server | Tagged 2008, gateway, host-to-host-vpn, ipsec, ipsec-windows-2008-vpn-howto, Juniper, juniper-and-windows-2008-vpn, remote access, route-vpn-traffic-over-tunnel-windows-2008, routing, rras, screenos, server, tunnel, vpn, vpn-windows-2008-gateway, windows, windows-2008-r2-ipsec-settings-cisco-vpn, windows-2008-vpn-ipsec, windows-2008-vpn-server-port-juniper

Corelan Cybersecurity Research

:: Knowledge is not an object, it's a flow ::