VPN

Part 3: VPN RFP Lab Eval
NetScreen

• Monitoring and Alarms
• Logging and Diagnostics
• Using NetScreen-Global Manager
• Global Manager Real-Time Reports
• For Carriers: NetScreen-Global PRO
• Additional Features
• Adding High Availability
• Our Experience With Tech Support
• Customer Feedback
• Did NetScreen Satisfy Our RFP?

by Lisa Phifer Vice President of Core Competence, Inc. [December 20, 2001]

Several months ago, ISP-Planet issued a Request For Proposal (RFP) for Virtual Private Network (VPN) appliances suitable for Internet Service Providers' (ISPs) deployment to broadband-enabled businesses of 10-200 employees. Using vendor RFP responses to build our short-list, we invited four contenders to submit their solutions to our lab for hands-on evaluation. In previous installments, we evaluated the solutions proposed by SonicWALL and RapidStream.

Here, we publish our third installment of our third evalution of Customer Premise Equipment (CPE) manufactured by NetScreen Technologies. The NetScreen-100 is a firewall/VPN/traffic management appliance for collocation facilities and medium-to-large enterprises. The NetScreen-5XP is an entry-level Internet appliance for small offices and teleworkers. Both can be provisioned and monitored from a NOC with NetScreen's Global Manager or Global PRO.nd contrast these offerings. If you need a brief review of what we've accomplished so far, start with Part One.

Monitoring and Alarms
Monitors, alarms, and SNMP polling/traps can be used to detect problems. Alarms can be e-mailed to two addresses, with or without a traffic log (left). System alarms are automatically generated for events like VIP status change. Thresholds can be set to generate attack, traffic, or VPN alarms. However, custom alarms cannot be created and built-in alarms cannot be disabled. Alarm logs can be viewed, written to file, or cleared in their entirety.

VPN alarms can be generated by monitors for each IPsec tunnel (Phase 2 Security Association (SA)). When VPN monitoring is on, the NetScreen pings the remote VPN gateway every 10 seconds, measuring latency and triggering an alarm when the SA status changes. Managed VPN providers can use monitors to keep site-to-site tunnels active, sending an alert if a router, link, or backbone failure brings a tunnel down.

Traffic alarms are generated by thresholds for each Policy. When counting is enabled, thresholds can be configured per second or minute—for example, an alarm can be triggered when packets to a VIP exceed 200 Kbytes per second. We triggered "per second" alarms, but were unable to trigger "per minute" alarms. Be warned: because the threshold resets each second or minute, sustained high-volume traffic can generate many alarms.

NetScreen appliances also provide SNMP access to the IETF standard MIB-II, an enterprise VPN monitoring Management Information Base (MIB), and enterprise traps for system events, traffic alarms, and VPN tunnel up/down notifications. SNMP write/trap permission is configured per community string—for example, you can let a customer monitor traps without granting SNMP write permission.

WebUI Traffic and Counter panels provide device-level monitoring. (For network monitoring, see NetScreen-Global.) The Traffic tab plots real-time throughput (bits per second) for any Policy with managed bandwidth (right).

The Counters tab displays traffic history (bytes per second, minute, hour, day, or month) for any Policy with counting on (below).

Because counters are maintained per Policy, these tend to apply to VIPs, site-to-site tunnels, or VPN User Groups—but not to individual Users. They offer insight into traffic flow, but not current status. For a list of active IKE, IPsec, or L2TP tunnels, use the CLI. The CLI can also reset individual SAs or tunnels —for example, disconnecting a remote user. WebUI tunnel status and reset would be welcome additions.

(Back to Top)

Logging and Diagnostics
The Event Log viewed from the WebUI includes IKE and L2TP messages—along with device reset, ifStatus changes, and configuration updates. This can be used to diagnose problems or infer status (e.g., IKE Phase 2 Ok means the IPsec SA is up).

An optional Self Log records denied packets to the NetScreen itself, identifying "door knob rattlers". Per-Policy Traffic Logs record sessions and datagrams permitted through the firewall (right).

Event and Traffic Logs are stored on the NetScreen; they can also be forwarded to a SYSLOG server, WebTrends Firewall Suite, or NetScreen-Global. Forwarded logs can (and should) be protected by VPN encryption (above).

Records can be simultaneously viewed from the WebUI and sent to WebTrends Firewall Suite for analysis (right).

Logs let you see what's getting through your NetScreen. Determining why sessions are not getting through can be a little tougher.

Fortunately, the NetScreen CLI includes robust query, diagnostic, and debug tools (left). For example, set IKE debug to view Phase 1/2 messages, including transform proposals. Debug packet flows by configuring a filter to specify source and destination address/port and protocol. Or use Snoop to record packets to a debug buffer, filtered by direction, interface, and flow.

Time and again, these tools have proven invaluable when diagnosing VPN and firewall problems.

(Back to Top)

Using NetScreen-Global Manager
For central administration, we also tested NetScreen-Global Manager 2.6, a Windows NT/2000 program that controls up to 1000 appliances from one PC. Global Manager exchanges configuration, logging, and keep-alive messages with NetScreen appliances over configurable ports.

Unless all appliances are on the trusted LAN, Global Manager traffic should be encrypted by using the VPN option. To enable Global Manager administration, use the CLI or WebUI (right).

When Global Manager is launched, NetScreen appliances are organized into folders displayed on a "network topology" pane. Global Manager does not support discovery. To add a new appliance, you must supply the device name, IP, port, admin login, and password. Thereafter, Global Manager will send keep-alives to the appliance, coloring the device icon to indicate reachability.

Using the topo pane, select a device and menu item to provision individual appliances. Configurable parameters are nearly identical to those available in the WebUI. For example, the Global Manager Show Users button and WebUI Users tab display the same list and can view/edit the same user parameters (above). Similarly, Global Manager can import/export/edit configuration files, update firmware, and reset any single NetScreen appliance.

Global Manager provisioning feels like a Win32 version of the WebUI. Admins will like the ability to jump from one appliance to another, but Global Manager still provisions one appliance at a time, using device-level accounts and passwords. But it adds a few shortcuts. Addresses, services, users, or schedules can be added to an appliance's configuration by dragging them from another appliance. Entire config files can be dragged from one appliance to another. To quickly provision a new appliance by cloning an existing appliance, just edit the IP addresses in a replicated config before dropping it on the new appliance.

(Back to Top)

Global Manager Real-Time Reports
Global Manager monitoring feels quite different from the WebUI. Global Manager can supervise up to 1000 appliances, launching up to 100 Report Viewer windows at once. Each Report Viewer displays real-time Network Activity and Resource Utilization graphs, Log Reports, or Traffic Summaries for one appliance.

Network Activity graphs incoming or outgoing bytes per interface. Resource Utilization graphs CPU, memory, and flash consumption. These real-time graphs come are quite handy and have no direct parallel in the WebUI.

Log Reports are broken down into Event, Traffic, and Self-Deny Logs. The Event Log contains system, configuration, and/or alarm events. These are separate lists in the WebUI; Global Manager conveniently merges them into one. The Global Manager Traffic Log aggregates events that are tracked per policy in the WebUI. In effect, Global Manager is a SYSLOG server, enabling central storage and analysis. Logs can be sent to the Windows Log or e-mailed at regular intervals. A Query panel can search the Traffic Log; adding search to other logs would be nice (right).

Traffic Summaries are tables or bar charts of sessions and bytes for the current day, month, or year. Each report sums inbound and outbound traffic per IP, service, policy, or VPN tunnel, for one entity or for all. For example, this HQ NetScreen-100 Summary Report depicts total sessions and bytes tunneled to branch office and teleworker 5XPs during the current report interval (above).

Summaries offer more than per-policy Counters in the WebUI. They make it easy to eyeball traffic distribution, locating top talkers, bandwidth-hogs, and broken VPNs. If only these reports could be printed or exported to a file, they would be even better.

Global Manager is a report engine, not an alarm surveillance system. Topo icons indicate whether Global Manager keep-alives are getting through, but not whether the device has an active alarm. Although alarms appear in the Event Log, Global Manager cannot acknowledge or clear alarms. ISPs responsible for managed services need more. Large ISPs and carriers should consider NetScreen-Global PRO.

(Back to Top)

For Carriers: NetScreen-Global PRO
Just after our lab eval, NetScreen announced Global PRO 3.0, a next generation management system. PRO's Policy Manager is "rack and go" software, preloaded on a Sun Netra T1 server. PRO's Report Manager ships on a Sun Ultra 10 Workstation. Administrators access PRO from Windows NT or 2000 Consoles; role-based privileges control what each can see and do. Crystal Reports generates comprehensive reports, drawing from PRO's Oracle database. An option integrates PRO with Micromuse Netcool for fault management and root cause analysis.

Global PRO 3.0 (December 2001 release) will offer industrial-strength features not found in its predecessors. For example, PRO can push each centrally-configured policy to as many as 10,000 NetScreen appliances. With NetScreen-Remote 6.0 (early 2002 release), VPN clients will be able to download per-user SafeNet policies from PRO.

Carriers responsible for thousands of CPE really require a robust, distributed management platform like Global PRO. PRO licenses are available from 100 to 10,000 devices. Smaller shops that need a big league manager can use Global PRO Express 3.0, a cut-down version designed for up to 100 devices. PRO Express ships on a Sun Netra 1X server, hosting a combined Policy Manager and Realtime Monitor.

(Back to Top)

Additional Features
Although our lab eval focused on managed VPN, NetScreen appliances provide single-point policy enforcement for additional services.

Traffic Management: All NetScreens implement port- and policy-based traffic shaping and DiffServ marking. Using a prioritized "leaky bucket" algorithm, NetScreen appliances guarantee bandwidth to each policy, up to port capacity. Maximum bandwidths and priorities can be specified per policy, distributing unused bandwidth. For example, the device can be set to let lower priority apps burst to a configured maximum without impeding higher priority constant bit rate traffic. DiffServ bits can be set by priority.

Server Load Balancing: The NetScreen-100 can load-balance sessions by mapping VIPs to server pools with (weighted) round-robin or least connection algorithms. Using ping to verify server status, the NetScreen-100 distributes new sessions to active servers (8 per pool). The NetScreen-5XP maps VIPs to individual servers, but does not offer server load balancing.

These features are included in standard NetScreen products. There are no add-on licenses for anti-virus scanning or authentication services. However, NetScreen appliances can be integrated with these separately-purchased WebTrends products:

URL Filtering: NetScreen appliances can filter HTTP requests by consulting a WebTrends Websense server. Even without Websense, NetScreens can be configured to block Java, ActiveX, and .exe/.zip files.

Vulnerability Testing: NetScreens ship with a WebTrends Security Analyzer trial version. This popular scanner checks the NetScreen for backdoors, open ports, password strength, and other common vulnerabilities, producing a report. ISPs often run scanners like this before and after installing a managed firewall, illustrating the firewall's value and verifying correct implementation of policies.

Firewall Log Analysis: NetScreens also ship with a 14-day trial copy of WebTrends Firewall Suite, upgradeable to a licensed copy at a 10 percent discount. This Windows NT/2000 SYSLOG analyzer records Event and Traffic data from NetScreens in WebTrends Enhanced Log Format, producing usage and trend reports.

(Back to Top)

Adding High Availability
NetScreen-100s can also be deployed in redundant groups, where one is active and others are hot standby(s). Redundant groups must contain the same model. Note that HA is supported on the NetScreen-100, -500, and -1000, but not on the NetScreen-5XP or -10. NetScreen shipped us two NetScreen-100s to evaluate the HA solution they had proposed.

Configuration and state are continuously synchronized between group members using NetScreen's Redundancy Protocol, enabling fail-over from active to standby without losing TCP sessions or VPN tunnels. Authenticated, encrypted sync messages can be sent through any port, but a dedicated cross-over between DMZ ports is recommended. Fail-over can be initiated manually and automatically by heartbeat or path monitor triggers.

Path monitors let each NetScreen independently verify router/server reachability with ping or ARP. Several monitors can be specified in a prioritized list; fail-over occurs when the standby achieves better aggregate results than the active unit. For example, we configured our HA pair to ping our WAN router every second, counting three lost pings as failure (right).

When our standby received no responses, we realized they were going to the virtual MAC associated with the untrusted IP then owned by the active unit. It seems that unique management IPs are required on the port(s) used to send/receive monitor pings. When we assigned public management IPs to both units, responses were received by each unit's physical MAC. In this config, fail-over took about 6 seconds, without breaking connections or tunnels. The cost: three public IPs (one shared, two unique).

Simple heartbeats may detect device-level failure, but path monitors provide greater resiliency to network failure. For even higher availability, NetScreen recommends deploying redundant NetScreens behind redundant Ethernet switches, connected by an 802.1Q trunk, and redundant WAN routers that speak Virtual Router Redundancy Protocol (VRRP).

(Back to Top)

Our Experience With Tech Support
NetScreen hardware is warranteed for one year, with next day replacement. Software upgrades are free for 90 days, maintenance releases for one year. Every NetScreen appliance includes 30 days of "basic" 12x5 phone support, covering installation, configuration, and trouble-shooting. E-mail and Web support are provided at no cost for one year. Annual support contracts provide 24x7 technical assistance with problem analysis and interoperability debugging. Phone support is also available 12x5 without a contract, at an hourly rate. Critical problems affecting customer network performance are escalated for resolution within 4 to 8 hours.

The "Secured by NetScreen" program provides NetScreen ISP partners with access to sales, marketing, shared seminar materials, lead referrals, sales training, a listing on NetScreen's website, and joint PR. Key ISP support staff receive in-depth training, with the cost depending upon the ISP's purchase commitment. ISPs that meet program requirements may be eligible for additional discounts and market development funds.

NetScreen's support site provides software, manuals, top-ten issues, app notes, a searchable knowledge base, and on-line problem reporting. We found step-by-step examples for just about every feature in NetScreen's Concepts and Examples guide or app notes.

Earlier this year, NetScreen support impressed us by resolving a baffling interoperability problem quickly. We were therefore surprised when support during this eval was downright sluggish. Admittedly, we reported no critical issues that deserved escalation. But support took days and weeks to handle questions that were eventually answered by an engineer in just one phone call.

(Back to Top)

Customer Feedback
Lab evaluations can verify feature support and assess overall manageability, but that's just part of the picture. For field experience with installation and support, we consulted an ISP reference supplied by NetScreen: Babak Pasdar, CEO of IGX Global Networks.

IGX delivers comprehensive security infrastructure, including managed VPN services, to midsize companies. According to Pasdar, "We're not [looking for] people that just go through the motions of putting a product in place to make it look like they have security. Typically, we get customers with multiple, redundant connections to the Internet, redundant locations, and complex needs than can't be met by typical VAR solutions." One example: a 700-employee company with international offices. IGX rolled out Internet access to each site, then built a 12-node VPN for branch office access to a central Oracle-based accounting system.

Pasdar is a big believer in ASIC-based firewall/VPN platforms like NetScreen. "I've been using CheckPoint since 1994. They are an extremely frustrating organization to deal with as reseller and their technology isn't all that great," he said. "Multiple platforms increases common vulnerabilities and [requires] more support expertise. PC performance is just not adequate for a VPN device."

IGX systematically tested Cisco PIX, RapidStream, and NetScreen performance. "We look at throughput—but not just raw throughput," said Pasdar. "We break it down into burstable throughput, sustained throughput, TCP vs. UDP, and different size packets. We also look at ramp rate—if your firewall doesn't have a good ramp rate, it is very easy to DDOS." In Pasdar's benchmarks, NetScreen came out on top. "I want NetScreen to have competition, and RapidStream validates NetScreen's [ASIC-based] approach," he said.

IGX NetScreen field installation averages 15 minutes. "A lot of people preconfigure policies in a box and ship it, but we believe that's insecure—putting those policies and keys in someone else's hands," said Pasdar. Instead, IGX ships a network-configured box to someone who can be talked through the install. "We have a confignet—a raw network that lets the [newly installed] device connect to us. We go in and configure [security] policies once we know the box is in trusted hands."

For remote provisioning, IGX uses SSH and NetScreen-Global PRO. "We are the exclusive device managers," said Pasdar. "We provide specific users at specific customers read-only access to their devices. But if we're responsible, we own those configs." IGX analyzes requested changes, advising customers of risk and impact on uptime. "Changes can have radical ramifications on organizations if you don't consider what will start or stop working as result," said Pasdar. "Customers have the option not to engage us on risk assessment, but we remind them that every time they add a VPN node, they're extending security to that site."

IGX also uses Global PRO for remote monitoring. "Sometimes, we use it in situations where the customer doesn't actually need a firewall, but wants statistics. The NetScreen can act as a passive probe, collecting trending and bandwidth usage information," said Pasdar.

Regarding NetScreen support, Pasdar said, "Quite frankly, we don't usually need it. Cisco support is better, but the percent of time we need NetScreen's help is significantly less. Sometimes we touch base with them on very complex VPN environments—more design than support. We put boxes out there and they do what they do." With over 1000 NetScreens deployed, IGX has returned 5 defective units.

"They've been top notch in listening to us in terms of feature requests," said Pasdar. Features added by popular demand include thresholding DOS features to avoid being triggered by real traffic, authenticating administrators with RADIUS, and policy-based NAT. Still on Pasdar's wish list: support for OSPF, BGP, and AES (an emerging standard for advanced encryption).

Pasdar struck us as a demanding, but satisfied, customer. But note that IGX is selling premium security services to midsize companies—the top end of our RFP's target market. While price never came up during our interview with Pasdar, "NetScreen is a bit pricey" appears every so often on ISP-Lists. Smaller businesses may need NetScreen-100 features like 10/100—on a sub-$5000 (NetScreen-10) budget. To address this gap, NetScreen is introducing the NetScreen-25 and -50.

(Back to Top)

Did NetScreen Satisfy Our RFP?
Our objective was to determine how well NetScreen's solution met our RFP's requirements. After hands-on testing, we find that the majority of our requirements are well-satisfied. Functionally, these appliances offer more than we asked for, including integrated traffic management, load balancing, and a MacOS client. Nevertheless, we do have a few concerns:

• NetScreens are not platforms on which to deploy à la carte application services. Hooks enable integration with Websense filtering, but NetScreen believes that "anti-virus and content filtering are better implemented beyond the firewall." The only separately licensed add-on is the 5XP user limit.
  
  
• For central provisioning, NetScreen-Global PRO sounds great—but high end for our RFP's ISP. The entry-level NetScreen-Global really offers device-level provisioning, albeit from a central PC. That meets our RFP, but NetScreen-Global PRO Express 3.0 may be a better fit, depending on price.
  
  
• Our requirements for remote user VPN are not completely satisfied. Customers can manage their own RADIUS database for user authentication, but L2TP/IKE Users must still be configured on the NetScreen. Per-user stats are not available in common configurations. Documentation is needed for configuring L2TP-over-IPsec with Microsoft's W2K client.
  
  
• VPN and traffic monitors are great, but the WebUI and NetScreen-Global do not make it easy to eyeball or disconnect active tunnels—especially those associated with remote users.

Finally, let's consider up-front cost and revenue potential for each of our RFP scenarios:

In the Entry-Level Scenario: (10-25 employees), the NetScreen-5XP Elite ($995) fits quite nicely. At 10 users, the entry-level 5XP is simply too small.

In Scenario Two: (50 employees), we might swap the proposed NetScreen-10 ($3995) for a new NetScreen-25 with three 10/100 ports. Traffic management may help to land this account, but is not an incremental revenue opportunity. Instead, we might sell NetScreen-5XPs ($495) to teleworkers and/or increase throughput with a NetScreen-50. High-availability requires NetScreen-100 or better—but an HA pair runs nearly $20K.

In Scenario Three: our customer can take full advantage of NetScreen-100 HA, traffic management, and load balancing features. We would use a full-mesh VPN to connect branch office 5XP Elites—or NetScreen-25s if 10/100 is required. Policy-based NAT, hub-and-spoke topology, and PPPoE/DHCP make the 5XP a more attractive option for teleworkers. We might offer NetScreen-Global reports or WebTrends vulnerability testing from our NOC as value-added services.

Update: Product upgrades released

(Back to Top)

Tune in next year (just two weeks from now) for our VPN RFP series closer: a point-by-point comparison between SonicWALL, RapidStream, and NetScreen solutions and their suitability for ISP deployment to small businesses.

And when you've read all three evaluations, don't forget to participate in our poll:

—End Part Three—