Technology Overview

As the networks become more complex, with greater numbers of elements, more structure is required. Elements become specialized in their applications, management and security become more important, physical location is a consideration, and the capability to handle higher densities of customers is critical.

In a complex network, structure can be imposed on routers by assigning specific jobs to particular routers. A common approach for ISP networks is to divide assignments among routers in the following way:

The differences listed in this table are not absolute, and it often happens that a particular router can fulfill either role. However, as Internet traffic continues to grow, the demands for access routers to handle increased density and backbone routers to handle greater throughput become more important. These requirements can be met most efficiently with platforms designed for the specific application.

The Aggregation Router

This design has a number of desirable characteristics:

• Backbone and access routers are separate, so that backbone router configuration can be kept relatively stable over time. Backbone routers are not affected when individual customers are added to or removed from access routers, or when individual customers add or remove value-added services.

• Two backbone routers are used in each POP to enhance network availability.

• Access routers are equipped with redundant trunk connections, again allowing enhanced network availability.

• Access routers can be added as new subscribers are brought onto the network.

The Cisco 10000 is flexible enough to serve as an access router in a variety of POP architectures.

• Floor space in a POP can be very expensive. Real estate costs are lowered by simply reducing the number of racks required to terminate a large number of customer connections.

• Network management is simplified by deploying a smaller number of larger routers. Fewer individual routers to configure, manage, and monitor results in a more efficient operation.

• Larger numbers of interfaces on the access router improve the statistical performance of a network. Packet networks are usually designed to take advantage of statistical multiplexing, capitalizing on the fact that not all links are busy all of the time. The use of larger numbers of interfaces reduces the chances that a fluke burst of traffic from many sources at once will cause temporary network congestion.

The appropriate degree of statistical multiplexing is not a "one size fits all" proposition. ISPs have different views on the trade-off between a network that is rich in bandwidth resources and does not rely on multiplexing, versus one that uses a lower-cost infrastructure but risks occasional congestion. For this reason, the Cisco 10000 can be configured with different combinations of interface cards for a wide range of multiplexing ratios.

• The high capacity of fiber-optic cables provides a critical advantage in simply reducing the mass of copper wiring that would otherwise have to be deployed.

• SONET/SDH also provides a high-density, industry-wide standard interface between network transport equipment and the equipment that uses the transported signals, such as aggregation routers.

• Compliance with Network Equipment Building Systems (NEBS) standards to make it suitable for central office deployment

• High interface density to minimize rack space requirements

• High port count for subscriber connections to minimize the number of routers that must be monitored and configured

• High-touch features such as extended access lists and QoS classification for value-added services

• Redundant components for high availability

• Optimized performance, including parallel processing, to provide fast throughput despite additional packet analyses required for intelligent services offerings

The remainder of this section discusses the Cisco 10000 and some of the features that provide industry-leading capabilities for leased line aggregation.

Both of the subscriber-side interface cards support full-rate (unchannelized) DS3, as well as channelization to DS1 (1.544 Mbps) and NxDS0 (Nx64 kbps). The interfaces also support "subrate" DS3, in which the rate of data transfer across a DS3 can be reduced to limit peak access rate. Subrate modes are included to interoperate with Cisco port adapters such as the PA-T3 and PA-2T3, and with customer premises DSUs from Quick Eagle Networks, Inc. (formerly Digital Link Corporation), Larscom Incorporated, ADC Telecommunications, Inc. (formerly ADC Kentrox), Verilink Corporation, and ADTRAN, Inc.

Connections to the backbone network can be made with several additional interface cards:

Table 1-2: Overall Density for Line Cards

Subscriber Interface Card Type	DS1 Density (Sessions)
	Per Chassis	Per Rack (12-inch depth)	Per Rack (30-inch depth)
Six-port channelized DS3 card	1008	3024	6048
Channelized OC-12 card	2016	6048	12096

In the past, private networks have been built by enterprise information technology departments either by leasing dedicated circuits between sites, or by the use of a virtual circuit technology such as Frame Relay or ATM. In each case, the enterprise operating the private network incurred substantial expense building and maintaining the network. ISPs often had to build several wide-area networks, one for Internet traffic, another for Frame Relay, and a third for ATM.

An ISP can reduce network costs by moving all of this traffic onto one common network and then creating VPNs, built on top of a common Internet backbone. This allows the ISP to maintain one network instead of several and also makes it economical for the ISP to offer a total private network package to businesses that want to outsource management of their corporate wide-area network.

For example, many ISPs are beginning to offer

• Business class data service (differentiated from economy class)

• Voice over IP services

Successful deployment of a VoIP service requires careful attention to latency through the network, given that small variations in delay caused by network congestion can annoy listeners and can cause high error rates in fax and modem traffic.

The following Cisco 10000 features enable ISPs to offer controlled-latency services:

• Priority queuing mechanisms allow some classes of traffic to be configured for the lowest possible latency in spite of network congestion.

• Careful design throughout the Cisco 10000 ensures that latency-sensitive packets cannot get held up in queues behind long backlogs of latency-insensitive traffic.

For subscribers who are starting to outgrow a single 1.544-Mbps DS1 circuit, this poses a problem: It's a big jump from DS1 at 1.544 Mbps to DS3 at 44.736 Mbps!

The Cisco 10000 offers two features to help bridge this gap:

• High-Performance Multilink PPP

• Subrate DS3

High-Performance Multilink PPP

Although the rate at which bits are clocked across a DS3 is fixed at 44.736 Mbps, vendors of data service units (DSUs) have created various hardware mechanisms to limit the rate of user traffic that can be sent across a DS3. These rate-limiting mechanisms allow ISPs to offer graded rates of access to their networks, along with a flexible pricing structure. These mechanisms are typically simple and reliable, although special hardware is required on both ends of the DS3 link.

• Password encryption

• Authentication, authorization, and accounting (AAA) support

• Router authentication

• Kerberos authentication and client support on Telnet

Some additional special measures are implemented in the Cisco 10000 to resist denial of service attacks. These are discussed in the following sections.

Access Lists

Reverse Path Forwarding Check

The RPF feature does not affect the packet forwarding rate through the Cisco 10000. Hence, network administrators will not be forced to disable it to improve throughput, which is sometimes necessary when a security check impairs performance.

Fast-Path Internet Control Message Protocol

Most denial of service attacks are directed against host computers or web servers and use routers as a means of accessing the target. However, attacks can also be launched against the router itself through operations that are not normally optimized for throughput. The resultant flooding can consume large amounts of router memory or processor cycles.

In addition, messages that are sent from the high-speed forwarding path to the router's internal processor are categorized by priority. This helps ensure that the router cannot become so busy responding to an overload of unimportant traffic that it neglects essential packets, such as keep-alives and route updates, that keep the network itself operating.

Cisco 10000 availability has been increased in the following ways:

The Cisco 10000 incorporates several mechanisms to reduce its downtime:

Many communications devices are based on a shared system bus, to which all circuit cards are attached. Systems based on PCI or similar bus standards are relatively straightforward to design with off-the-shelf components. However, the shared-bus approach has several limiting characteristics for the leased line aggregation application:

The Cisco 10000 eliminates shared buses. The shared bus is replaced by a Cisco-developed line card interconnect that uses point-to-point links between each line card and the PRE. Each line card has its own private path to the PRE, and no backplane resources relating to packet forwarding are shared between line cards.

This plan counters the difficulties of the shared-bus approach in the following ways:

The point-to-point backplane technology also provides scalable bandwidth. Bus modes are defined that allow lower-cost implementations with as little as 800 Mbps in each direction between line card and PRE. Line cards that take advantage of all backplane connectivity with current silicon technology can achieve 3.2 Gbps in each direction across the backplane. Future improvements in silicon technology will allow Cisco to take advantage of the clean electrical environment to boost backplane throughput even more.

• Termination of the physical interface and delineation of packet boundaries

• Restructuring interface-specific packet formats into a generic packet encapsulation used across the backplane links

• Management of physical interface functions, such as monitoring alarms for SONET/SDH, and maintaining statistics required by the T1 facility data link (FDL)

Within the Cisco 10000, the PRE executes most of the intensive packet processing tasks. This frees the line cards for other tasks---providing the highest possible interface density, supplying the unique circuitry required for each physical interface type, and handling interface-specific functions that require low-latency response such as alarms, FDL, and SONET APS.

The two PRE elements have complementary functions:

• The FP executes relatively simple algorithms on every packet at the highest speeds, to provide high overall throughput.

• The RP executes the complicated control plane algorithms required for initialization and management of the forwarding path.

Allocating these two classes of functions to separate processing paths yields the best possible balance between packet throughput and feature set flexibility.

Each of the 16 processors in a PXF network processor is an independent, high-performance processor, customized for packet processing. Each processor, called an eXpress Micro Controller (XMC), provides a sophisticated dual-instruction-issue execution unit, with a variety of special instructions designed to execute packet processing tasks efficiently.

In addition to processing packets, eXpress Micro Controllers have access to various on-chip resources such as register files and timers. They also have shared access to very large off-chip memories for storing state information, such as routing tables and packet queues.

Within a single PXF network processor, the 16 eXpress Micro Controllers are linked together in four parallel pipelines. Each pipeline comprises four microcontrollers arranged as a systolic array, where each processor can efficiently pass its results to its neighboring downstream processor. Four parallel pipelines are used, further increasing throughput.

The PXF network processor architecture allows all 32 independent processors to work efficiently on per-packet feature processing, yielding high throughput while still allowing substantial feature processing. By centralizing packet processing in the PRE, the Cisco 10000 ESR architecture frees up space on line cards, enabling high interface density, yet retaining the compact NEBS transmission equipment form factor.

• A high-speed direct memory access (DMA) channel is provided to send packets back and forth between the FP and the RP. Packets such as route updates that are not processed by the FP are sent via this link to the RP. Similarly, the RP sends packets by passing them to the FP for transmission to line cards.

• The RP also has memory-mapped access to all of the state information used by the eXpress Micro Controllers (XMCs). The RP is responsible for configuring the tables and lists used by the XMCs.