One Example

1. Multi-homed network with T1 Internet access
2. Branch office with always-on DSL
3. Branch office with dial-up Internet connection sharing behind NAT

We deployed VPN gateways to secure IP between our offices, using the Internet for site-to-site transport. We also deployed VPN clients for secure mobile access from our existing laptops.

A Nokia CryptoCluster VPN was already in place at the multi-homed site, so we leveraged this to create new IKE and IPsec policies for other sites and users. A WatchGuard FireBox-II was deployed at the DSL-connected site, so we activated Branch Office and Mobile User VPN features on this existing firewall. Because NAT can wreak havoc on VPNs, we replaced our dial-up NAT appliance with an analog router. Behind the router, in front of our privately-addressed LAN, we dropped a new NetScreen-5 VPN/firewall.

For remote access, we could have used VPN client software sold by WatchGuard and NetScreen. But we wanted to connect to both sites from the same laptop, so we ended up using SafeNet's Soft-PK client. On a Macintosh laptop, we used the PGP Desktop VPN client.

Although small, this VPN is complex by design—our objective was to illustrate options, tradeoffs, and issues. Because we started with three ICSA-certified gateways, we thought we had a pretty good shot at getting this multi-vendor VPN to work. We encountered a few speed bumps, but no insurmountable roadblocks. Drawing from this recent experience, here are a few recommendations for debugging multi-vendor VPNs. Consider it your VPN Bible:

1. Document What You Have In Mind: Many VPNs start with a good idea—boxes and interconnecting lines on a white board. At some point, this must be transferred to paper, with all the I's dotted and T's crossed. Many vendors supply planning sheets in product documentation. We started with VPN planning worksheets created by Joel Snyder. If you don't have a planning sheet, start with a blank sheet of paper. Heck, use a placemat. Whatever you do, write it all down.

Identify every router, firewall, security gateway, and subnet by IP address and netmask. Record pertinent network details—interface speed, default gateway, DNS. Then design your security policies. What traffic must each gateway protect? Identify security parameters for IKE and IPsec, including encryption and hash algorithms, Diffie-Helman Group, authentication method, and lifetimes. Specify whether you'll use ESP or AH, in tunnel or transport mode.

Documenting these decisions are important in any VPN—but doubly important in multi-vendor VPNs. Review your proposed design against all product specs, making adjustments to reflect differences in feature support. For example, we started with a policy that protected IKE with 3DES/SHA-1, then scaled back to DES/MD5 where necessary. Vendors may use different terminology, so be prepared for translation from plan to GUI. For example, IPsec parameters defined in our plan were implemented as WatchGuard "Dynamic Security Association Proposals" and NetScreen "AutoKey IKE Configuration Phase 2 Proposals".

2. Verify Connectivity First: After any change to network plumbing, verify connectivity before doing anything else. Don't waste time wondering why your tunnels aren't up when you have a bad cable, netmask, or default route. Use ping and traceroute to test reachability from the public interface on one gateway to the public interface on the other. Don't get thrown off by a firewall configured to ignore pings—ping something else on the same segment.

Make sure there are no filters in your way. Use UDP ping to verify reachability on port 500 (IKE). Check access router ACLs to make sure they allow protocols 50/51 (ESP/AH). Make sure that NAT isn’t being applied somewhere in the path between VPN gateways. If you verify these details early, you won't waste time chasing dead-ends. For example, we avoided most NAT issues by replacing Internet connection sharing with a small public subnet, courtesy of our local ISP FastNet. We debugged routing on this new subnet before we started to configure any VPN policies.

3. Build Incrementally: Begin with a simple case—preshared secrets, single policy for all IP—and then refine. Again, this is a good idea for any VPN—but essential in a multi-vendor VPN. If you start with a policy that reflects ICSA or VPNC test parameters, you may even be successful first time out of the chute. This is precisely what happened for us with our Nokia-NetScreen pairing using 3DES/SHA1/DH2. Of course, as soon as you have a configuration that works, back it up immediately!

Add complexity step-by-step. Over time, we replaced preshared secrets by digital certificates, added alternative proposals, and tightened security policy definitions and access control lists. We started with site-to-site—usually an easier multi-vendor nut to crack. Then we added same-vendor VPN clients. Finally, multi-vendor VPN clients. Introducing these upgrades gradually made it much easier to diagnose problems.

4. Debug Policies: When problems occur, IKE and IPsec logs and query commands can be used to examine security association status and the proposals being exchanged. Even these basic tools can provide the information needed to find basic mismatches. For example, debug let us determine that Diffie-Helman Group differences were causing IKE phase one proposals to be rejected. Once isolated, this problem was easily resolved by creating a matching custom policy.

In multi-vendor networks, be on the lookout for problems that occur on one gateway but not others, or in one direction but not the reverse. When we first upgraded to digital certificates, we discovered that our NetScreen was having trouble checking the CRL supplied in the Nokia's certificate. It turns out that the path specified by our Microsoft CA was not fully qualified and thus could not be resolved outside of the network in which it was issued. In this case, the one way problem turned out to be neither gateway, but how we'd issued certificates from a third vendor's CA.

5. Sniff Out Interoperability Problems: Another one-way problem we found was a true interoperability problem—differences in commit bit processing that surfaced after software upgrade. Again, debug was useful to capture the information needed by customer support. For tough interoperability problems, packet sniffers are invaluable to capture exchanges between affected products. Customer support may not have the environment to reproduce your configuration or it may just be faster and simpler to supply a packet trace for offsite analysis.

Packet sniffers can also be useful to eyeball differences. Comparing one product log to another can be difficult, since products record the same exchanges in different ways. Comparing packet captures generated by different vendor pairings can make it easier to spot where an exchange differs. If you know how to read the packet capture, you may find the problem without knowing how to interpret each product's log.

6. Don't Hesitate To Ask: Many vendors publish tech notes or knowledge bases that describe common multi-vendor combinations—how to configure them, how to debug them. For example, an increasing number of vendors can tell you how to configure their gateway to interoperate with the Windows 2000 VPN client.

Even vendors that don’t publish this information may have it if you ask. Customer support may have previously seen the combination you're trying to create. If not, they may actually be eager to solve engineering riddles posed by a new pairing. Multi-vendor VPN debugging doesn't have to consist of finger-pointing.

7. Be Realistic: Finally, scale back your requirements when you hit a costly obstacle. Can you do without the algorithm or feature causing the interoperability problem? Live with common denominators—or find a replacement product. For example, VPN client software can be hit or miss. The same product that works flawlessly on PCs A, B, and C may never work on laptop D. We were able to circumvent a problem like this by substituting VPN clients.

Bottom line
Multi-vendor VPNs have come a long way from the early days of IPsec. They are still more challenging than single-vendor VPNs, but not the impossible tangle one might imagine. Bakeoffs, certification test programs, and time in the field have improved interoperability. Applying a methodic approach to system integration can make the quest for interoperability in multi-vendor VPNs that much easier.

—End

< Back to to page 1: The Quest for Interoperability