Intrusion Detection:
Reducing Network Security Risk —continued

by Recourse Technologies [December 24, 2001]

How Are Intrusions Detected?
IDSes use a number of different technologies to detect malicious activity. The three most widely distributed technologies are signature detection, behavioral anomaly detection, and protocol anomaly detection.

Signature Based Detection
The majority of commercial IDS products on the market are based upon a system that examines the network traffic for specific patterns of attack. This means that for every exploit, the IDS vendor must code a signature specifically for that attack in order to detect it, and therefore the attack must be known. Almost all IDS systems are structured around a large signature database and attempt to compare every packet to every signature in the database.

Unfortunately, there are some significant flaws in this approach that render the IDS incapable of recognizing attacks. As network speeds increase, the IDS sensor does not have the resources to look at every packet, so some packets are discarded, allowing attacks to slip by unnoticed by the sensor. Most IDS sensors can only operate effectively up to about 60Mb/sec. Higher data speeds generally decrease their detection rate and increase their false positive rate considerably, thus reducing their effectiveness. Many companies today fully utilize 10/100Mb/sec or even up to 1Gb/sec on their network backbone, where most of their mission critical servers reside. Because a signature-based sensor cannot effectively operate at these data speeds, it leaves these systems vulnerable to attack.

Another known issue with signature-based systems is the time it takes the IDS vendor to identify new attacks, create a signature, and release an update. Attacks like Code Red and Nimda cannot be identified by signature-based systems until the signature is added to the database, leaving a window of opportunity for attacks to penetrate the network unnoticed. Unfortunately, a new attack does the most damage during this window of opportunity.

Behavioral Anomaly Detection
A less prevalent method of Intrusion Detection is the ability to detect statistical anomalies. The framework of statistical anomaly detection is the "baseline" of certain system statistics, or patterns of behavior that are tracked continually by the system. Changes in these patterns are used to indicate an attack. Examples include detection of excessive use, detection of use at unusual hours, and detection of changes in system calls made by user processes.

The benefit of this approach is that it can detect the anomalies without having to understand the underlying cause behind the anomalies; however, legitimate use of the system can trigger anomalies leading to a very high number of false positives.

Protocol Anomaly Detection
Protocol anomaly detection is performed at the application protocol layer. It focuses on the structure and content of the communications. Many attacks target protocols such as Telnet, HTTP, RPC, SMTP, and Rlogin for example.

When protocol rules are modeled directly in the sensors, it is easy to identify traffic that violates the rules, such as unexpected data, extra characters, and invalid characters. That is exactly how some of these attacks can be identified. Protocol-based IDSes, for example, can detect code Red, because they model the HTTP protocol exactly as it is reflected in the RFC. The Code Red attack violates the HTTP protocol specification because it uses a GET request to post and execute malicious code on the victim server. The IDS recognizes this as a violation of the protocol and alerts the system administrator to the violation. While the same kind of attack is making its way past signature-based systems, this attack is recognized by the IDS as a protocol violation and is reported to the system administrators, giving them hours, sometimes even days to respond to the new threat before a signature for the attack is developed and distributed.

A Layered Security Approach
In the layered security approach (right), Intrusion Detection enters the game at the highest level, providing a highly coordinated approach to managing security issues, from identifying threats on the network and gathering additional information on demand, to responding quickly and taking appropriate action. Through the use of distributed sensors, protocol anomaly detection, and high-speed statistical correlation analysis, a layered security approach to Intrusion Detection can identify and respond to both common and novel attacks to protect your network against business interruption, and prevent damage to your network as well as to customer confidence.

Deploying Network IDS
A successful IDS deployment is one that monitors each network segment by installing a sensor on the segment itself or on a segment boundary device, such as a switch, that has the ability to inspect all packets on the subnet. If you are using a signature-based IDS, you must consistently obtain the updated set of attack signatures from all your IDS and firewall vendors and review all security policies weekly to help narrow the window of vulnerability. It may be advisable to use a configuration management tool to track the signature file information on all systems.

IDS sensors are most effective when deployed on the network perimeter, such as on both sides of the firewall, near dial-up servers, and on links to partner networks.

Deploying Host Based IDS
The next step is to deploy host-based intrusion detection mechanisms on all servers identified as mission critical by your security policy, increasing their chances of surviving an attack. Unless you have an unlimited budget, you will want to prioritize your deployment. If your main concern is attack from the Internet, you should concentrate your host-based defenses in the Demilitarized Zone (DMZ).

Deception is perhaps the easiest tool in your arsenal to manage and perhaps the most rewarding tool when identifying malicious activity on the network. As deception hosts log every connection and keystroke entered, it is possible to learn the intentions, motive, and experience of the attacker. A common scheme for deploying deception hosts is based on "The Minefield" principle, and simply involves placing them where an attacker is likely to find them, often with appealing server names (such as "Primary Mail Server" etc.). Deception systems can be placed in your DMZ to attract attackers away from production network assets, and on the internal network to catch snooping employees or hackers that may have bypassed your other defenses.

Establishing a policy for centrally monitored IDS systems and sensors will aid in the correlation and analysis of events. It is also recommended that administrators rotate their logs, a copy of which should always be written to remote, removable media in case an attacker tries to delete or modify the log data.

In Summary
To be effective, a network security solution must be made up of several layers to address the various types of threats faced by today's networks. Intrusion detection systems will not pick up every attack, no matter what kind of system the company has deployed. If only signature IDSes are deployed throughout the network, they will not pick up new attacks. Since protocol anomaly systems can detect many new attacks like Code Red, Code Red II, and Nimda, corporations should, at minimum, be able to strengthen their defenses at the gates to their networks: at the Internet connection, VPN connections, customer network connections, and so on. Thus they bolster the first line of defense at the entry-points so that these attacks can be detected as soon as possible.

The Internet provides a cost effective platform for companies and businesses to sell their products and services to a vast audience that is without geographic constraints. In 1995 there were 50 million people accessing the Internet worldwide, with recent projections suggesting somewhere in the region of 807 million people by 2003. The possibilities for companies with sound business practices and solid security are limitless. Detecting attacks quickly requires advanced warning systems. Most corporations need to improve their detection abilities in order to protect their data, their customers, and their partners' networks. Strengthen your defenses and reduce your network risk.

—End