Why Metro Area EtherLECs Should (Still) Worry about DDOS Attacks

Internet Service providers are all too familiar with Distributed Denial of Service (DDOS) attacks. Most data CLECs understand the concepts of a DDOS attack as well, and appreciate the threats they represent. Equipment failure, business disruption, and erosion of customer confidence top the list. As the stakes increase, it's inevitable that legal action will be taken against organizations and individuals whose equipment is found to have participated in a DDOS attack for failing to meet industry best practices for hardening systems against compromise by malicious software, especially DDOS Trojan Horse programs. Beyond this, Service Providers may soon find it necessary to demonstrate they exercise best operational practices to safeguard network resources against such attacks during service level agreement negotiations with customers.

Why focus attention on EtherLECs when I talk about DDOS attacks? To answer this, it's helpful to look at what DDOS attacks are and how prolific they have become. Thereafter, it's a matter of (my) speculative extrapolation.

DDOS Attacks - a Short Primer
Generally, a denial of service attack attempts to halt or crash a system by exploiting a logic error in software (euphemism for bug), or by exploiting or flooding a system in such a fashion that critical resources (memory or CPU) are exhausted attempting to satisfy bogus requests. Distributed Denials of Service (DDOS attacks) are of the flooding kind, where an attacker's master or handler program activates software "zombies" maliciously installed on a large number of systems the attacker has previously compromised, and instructs them to attack an IP address, address range, or subnet.

Typically, the hundreds of zombies engaged in the attack overwhelm the target system of facility with connection requests that are difficult to distinguish from legitimate traffic. The bogus connection requests that are delivered to and processed by web hosts consume CPU and memory and limit the host's ability to process legitimate requests. Or the sheer volume of traffic overwhelms or congests routers and switches to the point where both good and bad traffic are discarded. Both consequences are debilitating.

Worrisome data on DOS prevalence
If only DDOS attacks were as infrequent events as those deemed as newsworthy - notably, the attacks on Amazon, E*Trade, e-Bay, and Yahoo! in February 2000, and on Microsoft's name services in January 2001 - but this is sadly not the case. A recent UCSD study has revealed that DDOS attacks are astonishingly frequent. The first quantitative data on global DOS prevalence conservatively puts the number of DDOS attacks at more than 2000 per week.

Now the data that captured my attention and makes me think hard about EtherLECs is this: most DDOS attacks average 2000 packets per second, but some generate over 150,000 packets per second. The low number corroborates speculation that many DDOS zombies are installed on woefully under-secured dialup PCs, but the higher number suggests significantly larger numbers of zombies per attack and zombies attached to higher bandwidth media.

Issues for EtherLECs
Now for the speculative extrapolation I promised. If DDOS attacks are damaging when executed from zombies over dialup, residential broadband, and telco WAN services, they have the potential to become absolutely debilitating when launched from large numbers of zombies over 100 Mbps + Ethernet access loops.

DDOS attacks can also devastate an Ethernet access market in its formative stages if directed at EtherLEC infrastructure as well as customer sites. EtherLECs are sweet targets for zombie hosts; they also provide service to top-tier consumers. The Honeynets Project gives us concrete evidence that the motivation and incentive to launch DDOS attacks against infrastructure is escalating: the paper, Know Your Enemy: Motives is plain scary: in a replay of conversations held in private chat rooms (hosted, of course, on compromised systems), black hats discuss "taking out India", compromising then modifying an ISP's billing system, and more. Is it so hard to imagine a black hat community engaged in a head-to-head competition to see who can be the first to throw a million packets per second at a target? A billion?

The UCSD study indicates that even today, "there is a small but significant fraction of attacks directed against network infrastructure… some of these attacks, particularly a few destined towards routers, are comprised of a disproportionately large number of packets. This point is particularly disturbing, since overwhelming a router could deny service to all end hosts that rely upon that router for connectivity."

Routers today; Ethernet and Optical Switches Tomorrow? Preventative Action
Increasingly, customers will expect "service providers" to protect them. Customers will also judge any EtherLEC most critically should its infrastructure fall victim to DDOS attack. If you are switching IP anywhere in your network, or manage customer equipment that switches IP, there are best practices you should implement.

Begin with customer education. While customers will expect you to protect them, they must also assume responsibility for protecting their own systems from being exploited as DDOS masters and zombies.

Point them to online resources that explain DDOS attacks, identify the ports DDOS masters use to communicate with zombies, and encourage them to block these inbound and outbound at firewalls. Emphasize that every customer must take measures to prevent source address spoofing and private network address leakage from their networks.

Encourage customers to enable their own ICMP and TCP SYN rate limiting defenses (Cisco users should read about and implement Committed Access Rate, CAR, and Unicast RPF). If customers express unwillingness to participate, explain that companies have successfully petitioned courts to issue temporary restraining orders against other companies whose equipment had hosted DDOS zombies, with immediate disconnection from the Internet a consequence of those orders.

At the June Internet Security Conference, Dr. Bill Hancock, Chief Security Officer of Exodus communications suggested a number of preventative measures for ISPs in addition to those already mentioned, including router neighbor authentication and selected packet discard. In the same session, Dr. Stefan Savage of Asta Networks recommended that ISPs "construct filters to maximally match attack and minimally match good traffic…closest to ingress, with the lowest forwarding impact".

The Filtering Conundrum
According to Dr. Hancock, ISPs are reluctant to implement filters. "The problem with is not a lack of concern for the customers' dilemma of an attack or the recognized need to stop attacks, but bandwidth constriction due to filters and other activities on backbone routers to mitigate or reduce attacks to insignificance. By placing even simple DDoS filters on backbone routers, the performance of the routers degrades significantly and kills the throughput of the network. It is often easier and less painful to all customers to allow the DDoS attack to proceed over the backbone network than try to kill it off with backbone filters that would kill network performance."

Typically, ISPs will only put filters on edge routers near customer connection points. EtherLECs can implement filters here as well, as these will not degrade backbone resources. However, Dr. Hancock warns that "customers need to understand that while these may stop attacker traffic from reaching the customer's site, the traffic is still on the network and is still in the path. This means there will be performance issues even if the ISP stops the traffic from arriving to the end point of the network."

Emerging DDOS Countermeasures" technologies
Technology beyond filtering is needed. With technology, however, need begets innovation. DDOS attacks are a big and costly problem, big enough to attract investors for a considerable number of startups in a weak economy. Asta Networks, Captus Networks, Cs3, and Mazu Networks are among the companies who promise innovation and solutions. These and other companies remain deep in a stealth mode of operation, seeking to protect innovation in this area. Established IDS companies like Recourse Technologies, Network Flight Recorder, and others are also players in this space.

But anti-DDOS solutions are complex. "DDOS blocking has a lot of the same properties as IDS building," says Marcus Ranum, CTO of Network Flight Recorder, "You need to be very sure that you're looking at a DDOS attack before you take any kind of action. Obviously, it'd be A Bad Thing if you shut off access to legitimate traffic as a result of a DDOS, so the IDS needs to be the most accurate possible so you don't react incorrectly."

Stefan Savage, Chief Scientist and Co-founder of Asta Networks, adds, "There are a variety of ways to manage high-bandwidth DDoS attacks including filtering, rate-limiting, re-provisioning, and re-routing. However, to use these countermeasures effectively requires very quick detection and trace back capabilities plus a good understanding of the underlying network topology and the capabilities of its routers and switches."

Early counter-DDOS solutions will implement proprietary methods of distinguishing good throughput from bad. Like Intrusion Detection Systems (IDS), traffic that matches a signature or profile of a DOS will be acted upon according to policies established by service providers. The objective of near-term solutions will be to contain the traffic onslaught so that servers and infrastructure are able to withstand an attack without suffering a complete failure.

Products like the Captus CaptIO series operate "in line" with traffic and will automatically enforce firewall/filtering rules to block attack traffic, then divert it to a logging device for subsequent forensic analysis. Cs3 has attracted DARPA funding to attempt to solve the source address forgery, and to develop a fair service scheduling approach to allow customers a fair share of Internet resources during a DDOS attack.

Irrespective of the approach, automated containment is likely to follow form from manual intervention performed today - rate limiting of traffic and transaction requests according to customer-assigned thresholds, which will be set below the capacity of systems under attack. Once the attack is contained, trace back, identification of attack sources, and ultimately, identification of the perpetrators begins. This will only be partially automated at best in first generation equipment. This appears temptingly close to the network operations equivalent to treating the symptom rather than eliminating the root cause of the disease. It is. But the patient lives while we search for a cure.

Closing Remarks
EtherLECs are emerging as important bandwidth providers. Their infrastructure is too tempting to be left untested by black hats. A four-pronged pre-emptive response is recommended: (1) Adopt the preventative measures that constitute "best practices" today is essential to avoid a major black eye, and the financial and credibility setbacks an infrastructure failure might incur; (2) investigate and trial early DDOS countermeasure technology. (3) help yourself and partner ISPs by leading a strong campaign to educate your Ethernet access customers on their role and responsibilities in preventing DDOS attacks; and finally, (4) work with local and federal law enforcement for more aggressive incident response, leading to prosecution and financial liabilities.

David Piscitello is president of Core Competence, Inc., a network consulting firm and founder of The Internet Security Conference.