|
|
This chapter details these steps in the process of designing an MPLS network:
Additional design steps are required for Class-of-Service, MPLS VPNs, traffic engineering, and other IP services.
A typical structure for MPLS provider networks (carriers or ISPs) is shown in Figure 3-1. An MPLS network consists of edge label switch routers (Edge LSRs) around a core of label switch routers (LSRs). Customer sites are connected to the provider MPLS network. Figure 3-1 shows nine customer sites and six Edge LSRs, but typically there are several hundred customer sites per Edge LSR.
The customer premises equipment (CPE) runs ordinary IP forwarding and normally does not run MPLS. If the CPE does run MPLS, it uses it independently from the provider. It is important to note that the Edge LSRs are part of the provider network and are controlled by the provider. The Edge LSRs are critical to network operation and are not intended to be CPE under any circumstances. The provider might locate and manage routers at the customer sites but these should be running ordinary IP outside the MPLS network itself.
For details on mixed MPLS and IP-over-ATM networks, see Chapter 2, "Integrating MPLS with IP and ATM."
The simplest MPLS network structure is shown in Figure 3-2 Topology (a). This structure applies to router-only networks that might use MPLS for supporting VPN services or IP Traffic Engineering. In this structure, customer sites are connected directly to router-based Edge LSRs. The Edge LSRs are connected to other LSRs that are also based on router platforms.
The routers are interconnected by virtually any sort of link: serial, Ethernet, packet-over-SONET, and so on, and packets with MPLS headers. The routers involved could be 6400, 7200, 7500, or 12000 series Gigabit Switch Routers. Midrange routers (3600 and 4700 series) could be used in lower-bandwidth applications.
In a variant of this structure, the point-to-point links between the routers are actually ATM PVCs. These may be used during migration to ATM MPLS.
The simplest ATM MPLS network structure is shown in Figure 3-2 Topology (b). As with the previous case, customer sites are connected directly to router-based Edge LSRs, typically 6400, 7200, or 7500 series routers. The Edge LSRs are connected by ATM links to the core ATM LSRs. The ATM LSRs may be BPX 8650 IP+ATM Switches, LS1010, 8500 MSR, and later other ATM switches such as the MGX 8800 with PXM-45.
The ATM switches carry packets with ATM MPLS labels; this means that on each ATM link there is a different MPLS Label VC (LVC) for each label.
It is possible to mix ATM MPLS and packet-based MPLS on one network. A simple example of this is shown in Figure 3-2, Topology (c). In a network such as this, some links run packet-based MPLS, and some links run ATM MPLS. The devices that interface between packet-based MPLS and ATM MPLS are the same routers that act as ATM Edge LSRs: anything from a 3600 up to a 12000.
ATM MPLS networks with router-based Edge LSRs may also use separate access devices, as shown in Figure 3-2 Topology (d). This will occur when access is required through a device that does not support MPLS services. There are three common situations where this is required:
Customer traffic is carried through the access device to the Edge LSR. Between the access device and the Edge LSR, there is a different logical link for each customer. This may be a Frame Relay or ATM PVC, or a PPP link.
The previous type of network can be simplified if the access device supports Edge LSR function as well as Frame Relay, ATM, or other access services. This is shown in Figure 3-2 Topology (b).
In the case of IP+ATM edge switches, a single device gives access to both MPLS services and PVC or SVC services. IP+ATM edge switches include the BPX 8680, MGX 8850, and 6400 universal access concentrator.
MPLS networks can use traditional ATM equipment as a migration step in introducing MPLS to an existing ATM network. Traditional ATM switches can be used in three ways, as shown in Figure 3-3 Topology (f).
These uses of traditional ATM equipment have disadvantages and must be used with care.
Providers may want to keep an existing ATM infrastructure while building a new MPLS infrastructure (either ATM MPLS or packet-based MPLS) alongside the old infrastructure. Cisco IP+ATM edge devices support this well, allowing customers to access both the MPLS network and services, along with the old ATM network, even from a single access link. This is shown in Figure 3-3 Topology (g).
The IP+ATM access devices can be any of those that can be used in Figure 3-3 Topology (e). The network in Figure 3-3 Topology (g) supports the same functions and services as Figure 3-3 Topology (e), but the Topology (g) network requires more equipment.
Also see: Dual Backbones: Traditional ATM and ATM MPS or Packet-over-SONET, page 2-23.
There are four main considerations when choosing ATM MPLS edge equipment:
1. Type of services to be offered: IP+ATM, that is, end-to-end PVC and SVC services as well as IP services, or just IP
2. Type of access lines
3. Number of access lines
4. Requirements for redundancy and reliability. Key issues are:
Redundancy levels for MPLS edge devices can be classified as:
Equipment recommendations based on these requirements are shown in Table 3-1. Cisco will support MPLS on most or all mid-range and high-end routers, most ATM switches, and most access products with routing capability.
| Equipment | Type of Services | Access Lines | Redundancy Support | Comments | ||
|---|---|---|---|---|---|---|
3600 router | IP only | Relatively small numbers of async, modem, serial/Frame Relay, 10 Mbps Ethernet, ISDN BRI & PRI, HSSI, E1/T1 serial, Fast Ethernet, OC-3/STM-1 ATM, voice interfaces, and others. | None | Small number of LVCs supported on ATM cards will lead to limitations on MPLS network size. Not recommended for provider ATM MPLS networks. | ||
4700 router | IP only | Relatively small numbers of serial/Frame Relay, 10 Mbps Ethernet, ISDN BRI, E1/T1 serial, Fast Ethernet, E3, T3 or OC-3/STM-1 ATM, and others. | None | Small number of LVCs supported on ATM cards will lead to limitations on MPLS network size. Not recommended for provider ATM MPLS networks. | ||
7200 router | IP only | Serial/Frame Relay up to E1/T1, 10 Mbps Ethernet and Fast Ethernet, ISDN BRI, HSSI, high-speed serial, E3, T3 or OC-3/STM-1 ATM, packet-over-SONET/SDH and others. | None | Minimum recommended for provider networks. PA-A2 CES-ATM port adaptors do not currently support MPLS. | ||
7505, 7507, 7513, or 7576 router | IP only | Serial/Frame Relay or ISDN up to E1/T1, 10 Mbps Ethernet, Fast Ethernet, and Gigabit Ethernet, HSSI, high-speed serial, ATM, packet-over-SONET/SDH and others. | Warm-Standby Processor Redundancy with dual RSPs. |
| ||
12008/12012 | IP only | POSIP and ATM at OC-3 to OC-48 rates, and Gigabit Ethernet.
| Warm-Standby Processor Redundancy | Suitable for high-speed peerings between providers. | ||
Catalyst 5500 with Route Switch Modules | IP+ATM | 10 Mbps and Fast Ethernet, E3, T3, OC-3/STM-1 and OC-12/STM-4 ATM, and others. | None | The Cat 5500 is primarily a LAN switch, but also has limited Edge LSR capability. The Cat 5500 may be connected only to an ATM MPLS network by tunnelling. | ||
6400 | IP + ATM | ATM at E3/T3 to STM-4 rates, also Ethernet and Fast Ethernet. | Warm-Standby Processor Redundancy | MPLS support is not yet shipping. | ||
MGX 8850 | IP + ATM | High numbers of 56K/64K Frame Relay, T1/E1 Frame Relay, channelized, and ATM, and higher-speed Frame Relay, serial and channelized T3. | Full Warm-to-Hot Standby. | At FCS, the MGX 8800 will have hot-standby1:N redundancy capability for customer access lines, hot standby control for PVCs and hot-standby trunks. 1:N warm standby redundancy for RPMs is scheduled for release in CY2000, | ||
BPX 8650 | IP + ATM | High numbers of 56K/64K Frame Relay, T1/E1 Frame Relay, channelized, and ATM, ATM at E3/T3 to STM-4 rates, and others. | Excellent redundancy in general, but there is a single point of failure for Edge LSR function. | (See BPX 8680.) | ||
BPX 8680 | IP + ATM | High numbers of 56K/64K Frame Relay, T1/E1 ATM, Frame Relay and channelized. Also ATM at E3/T3 to STM-4 rates, and others. Extra 6400, 7200, or 7500 routers (or "label switch controller" packages) may be required to act as Edge LSRs. E3/T3 or faster ATM access lines are used. If IP service is to be supported for large numbers of ATM links at T3/E3 rates and above, it is more cost-effective to use separate stand-alone routers. | Full Warm-to-Hot Standby (but with FCS limitations). | BXM trunk cards must be used. BXM cards are required. MPLS is not supported on BNI cards, except if the BNI cards are used as Feeder Trunks. BCC cards must be BCC3-64 or later. BCC4 cards are strongly recommended. The BPX 8680 can include up to 16 MGX 8850 shelves, with redundancy features as described above. Full redundancy for the combined device relies on redundancy for the label switch controller for the BPX 8600 shelf. A redundant configuration using two simultaneously active controllers is supported in BPX software release 9.3. |
There are some practical issues to bear in mind when examining the product specifications for products to be used as ATM Edge LSRs:
There are five main considerations when choosing ATM LSRs:
1. Type of trunks
2. Number of trunks
3. Number of connections supported
4. Whether VC Merge is required
5. Requirements for redundancy and reliability
Equipment recommendations based on these requirements are shown in Table 3-2. MPLS will be supported on most ATM switches. Consequently, more ATM LSRs will become available in the future. In addition, any traditional ATM switch can be used in a Cisco MPLS network if tunnelling is used, subject to significant limitations.
 |
Note If redundant pairs of trunks are used, the number of trunks supported will be half the numbers shown. |
| Equipment | Type and Number of ATM Trunks | Number of Connections Supported | VC Merge Supported | Redundancy Support | Comments |
|---|---|---|---|---|---|
MGX 8850 with PXM1 | 4xOC-3/STM1 | 8K. | No | Full Warm-to-Hot Standby | The MGX 8850 is intended primarily to be an Edge LSR, but will also has limited ATM LSR capability. This capability will not be available until late in CY 2000. |
LS1010 | 32 x T1/E1 with Inverse Multiplexing over ATM (IMA), 32 x T3/E3, 32 x OC-3/STM-1, 8 x OC-12/STM-4 | 64K | Yes | None |
|
6400 | 16 x T3/E3, 16 x OC-3/ATM-1, | 64K | Yes | Warm-Standby Processor Redundancy | ATM LSR support on this switch is not yet shipping. |
BPX 8650 | 144 x T3/E3, 96 x OC-3/STM-1, | 192K | VC Merge is supported in BPX Software Release 9.3, with Enhanced BXM cards. | Some redundancy features at FCS, full redundancy is possible later. | All MPLS interfaces must be on BXM cards. BCC cards must be BCC3-64 or later. BCC4 cards are strongly recommended. The BPX 8650 has supports hot standby trunks and switching fabrics. Full redundancy relies on redundancy for the label switch controller for the BPX 8600 shelf. A redundant configuration using two simultaneously active controllers is supported in BPX Software Release 9.3. |
8540 MSR | 64 x T1/E1 with IMA, 64 x T3/E3, | 256K | Yes | Warm-Standby Processor Redundancy | Some trunk cards are not yet shipping. |
MGX 8800 with PXM-45 card(s) | 192 x T3/E3, 144 x OC-3/STM-1, | 384K | Yes | Full Warm-to-Hot Standby | PXM-45 cards are not yet shipping |
LSRs other than ATM switches may be also be used. In particular, the following routers may be used as LSRs:
Using these routers, MPLS may be supported over virtually any link type: ATM, packet-over-SONET, Ethernet, and so on. Router-based LSRs do not support native ATM virtual circuit connections, and all except the 12000 series have relatively low capacity compared to Cisco ATM LSRs.
The goal of designing an MPLS network prior to installation is to produce a network that will operate reliably and optimally. Because of the inherently connectionless nature of IP traffic, customers will not be able to tell a carrier exactly which traffic they want to send where. Because of this, it is not possible to perfectly design a network ahead of time.
This section considers these initial design steps result in a working network:
1. Design Points of Presence.
2. Dimension backbone links in the network.
3. Design IP routing.
4. Dimension MPLS Label VC space.
5. Refine the design once the network is operational. This final design step is an ongoing process of optimizing the network design.
The design of Points of Presence (PoP) for an ATM MPLS network is constrained by:
Some typical PoP designs are shown in Figure 3-4.
Where a single Edge LSR device is sufficient for supporting the number and types of access lines in a Point of Presence (PoP) location, then the simple structure shown in Figure 3-4 Topology (a) is sufficient. Numerous access lines (typically tens or hundreds) are brought into a single Edge LSR, which is connected to the rest of the ATM MPLS network. Numbers and types of access lines supported by single Edge LSRs are described in Table 3-1.
A PoP may require more than one Edge LSR because of a large number of access lines to be supported at that location. Alternatively, different types of Edge LSR might be required because of different types of access lines to be supported. Where there are several Edge LSRs in a PoP, it makes sense to also include an ATM LSR. This is shown in Figure 3-4 Topology (b).
The ATM LSR:
Depending on reliability requirements, redundant pairs of links would be used between the Edge LSRs and the ATM LSR.
An extension to the previous model is to use traditional access concentrators in addition to Edge LSRs and an ATM LSRs. Circumstances where this is appropriate are discussed in "Structures for MPLS Networks" on page 1.
One example of this type of PoP uses MGX 8220 access shelves, 6400, 7200, or 7500 Edge LSRs, and BPX 8650. This is shown in Figure 3-4 Topology (c). IP traffic from access concentrators is carried in ATM PVCs to a Edge LSRs. These may be carried through the same BPX 8650 that acts as an ATM LSR because this is an IP+ATM switch.
Edge LSRs in such a PoP may have three different types of configuration:
Note that the label switch controller (LSC) in the BPX 8650 can act as an Edge LSR simultaneously while performing its LSC function. However, use of an LSC as an Edge LSR is not recommended for providers who consider the separation of MPLS control functions from data forwarding functions to be important. As an Edge LSR, an LSC can perform any of the three functions discussed above.
The number of Edge LSRs required in the PoP depends on:
The capacity of a 6400, 7200, or 7500 router running MPLS edge functions is roughly the same as its ordinary IP capacity using Cisco Enhanced Forwarding (CEF). For example, a 7200 router with an NPE 200 processor can support close to 200 Mbps of MPLS edge traffic at normal IP packet sizes.
If additional edge functions such as CAR and WRED are used, then performance may be affected. In these circumstances, the performance of the routers should be verified for the particular combination of features to be used. Note also that voice-over-IP packets are very short and will have significantly lower throughput than indicated here.
The MGX 8850 and Cisco 6400 integrate the functions described in the previous example into a single device, illustrated in Figure 3-4 Topology (d). It consists of:
The 6400 and MGX 8850 also have IP+ATM capability.
Some sites in a network may have a pure switching role. In an ATM MPLS network, these sites will consist of a single ATM LSR, as shown in Figure 3-4 Topology (e), or possibly a redundant pair of ATM LSRs. The ATM LSR would typically be a BPX 8650.
In some networks, it may make sense to use a router-based LSR instead, as shown in Figure 3-4 Topology (f). A 7500 or 12000 series router may be suitable for this application. Core LSRs usually have Edge LSR capability as well. For example, the BPX 8650 has limited Edge LSR capability as part of its label switch controller.
This section is a guideline to dimensioning the links in a small MPLS network, but it is not the only way of doing this. Different providers will have their own unique procedures for designing and running networks, but they all will be similar to this.
It is not the purpose of initial design to produce a perfect network. Several approximations are made in this process leading to a network that works but is not necessarily optimal. The last step in the design process is to optimize the network, discussed in the "Ongoing Network Design" section on page 3-42.
As an illustrative example, see Figure 3-5, which is a large network distributed across Australia.
1. Design edge points of presence and their layout.
3. Estimate the traffic matrix.
4. Estimate bidirectional traffic flows.
| Traffic Source | ||||||
|---|---|---|---|---|---|---|
| Traffic Destination | Distribution Percentage | Adelaide | Brisbane | Melbourne | Perth | Sydney |
Adelaide | 2.5% | 1.25 | 2.5 | 10 | 1.25 | 12.5 |
Brisbane | 5% | 2.5 | 5 | 20 | 2.5 | 25 |
Melbourne | 40% | 20 | 40 | 160 | 20 | 200 |
Perth | 2.5% | 1.25 | 2.5 | 10 | 1.25 | 12.5 |
Sydney | 50% | 25 | 50 | 200 | 25 | 250 |
Total | 100% | 50 Mbps | 100 Mbps | 400 Mbps | 50 Mbps | 500 Mbps |
| Adelaide | Brisbane | Melbourne | Perth | Sydney | |
|---|---|---|---|---|---|
Adelaide | 1.25 |
|
|
|
|
Brisbane | 2.5 | 5 |
|
|
|
Melbourne | 20 | 40 | 160 |
|
|
Perth | 1.25 | 2.5 | 20 | 1.25 |
|
Sydney | 25 | 50 | 200 | 25 | 250 |
5. Design the layout of the backbone network.
6. Estimate link flows based on the estimated traffic.
7. Assign link capacities.
8. Adjust for redundancy.
9. Check whether the selected equipment is adequate.
Assume the Melbourne PoP had used an MGX 8850 instead of a BPX 8680. This PoP needs two STM-4 links and two STM-1 links, which is not yet supported on an MGX 8850. So, in this case, the PoP would need to be redesigned by using a BPX 8680 instead of an MGX 8850.
Note that any such redesigns, if required, are a relatively minor issue. Many different types of Cisco equipment can be used in ATM MPLS PoPs, and it will usually be found that a PoP can be built to meet the requirements of one location, simply by using combinations of equipment used at other locations. A BPX 8680, for example, combines several MGX 8850 shelves.
There are three main ways of achieving change-over for a redundant pair of ATM links:
1. Data link-level change-over
This is the normal link redundancy mechanism in traditional ATM networks. Change-over occurs because of physical and Data Link monitoring in the ATM switch or ATM LSR hardware. The switch hardware also typically sets up a copy of all virtual circuit state on the backup link. In a switch with data link-level redundancy, any single link failure will typically result in close to zero data loss on any virtual circuits.
2. Inverse multiplexing over ATM (IMA)
IMA carries distributes data over a group of links by distributing cells across the links in round-robin fashion. It offers both data-link level load sharing across links, and redundancy. If one of the links in a group fails, cells are no longer sent on that one link, but the others are still used. IMA is available only for low-speed links—groups of T1 or E1 links.
3. Parallel links with network-layer change-over
In this case, a redundant pair of trunks is used, but data-link layer protection is not used at all, and all connection change-over takes place at the network layer. IP or PNNI routing is aware of all link failures and reacts to them.
Where a redundant link is required, recommendations for use of these modes are:
MPLS uses ordinary IP routing protocols—OSPF, IS-IS, and so on—to determine the routes for IP traffic and LVCs. Every LSR runs ordinary IP routing protocols in the same way that ordinary IP routers do.
An important implication of this is that OSPF (or IS-IS, and so on) "sees" an MPLS network as being exactly like an ordinary router network. It is possible to have various viewpoints of an ATM MPLS network:
Using these rules, the routing viewpoint of an MPLS network can be derived. This is shown in Figure 3-9 Topology (b).
Designing IP routing in an MPLS network is almost exactly the same process as designing IP routing for an ordinary IP network. By looking at the routing viewpoint, a network can be divided into areas, route summarization can be designed, and so on.
There are several design guides for IP routing on http://wwwin-cons.cisco.com/~dblack/design-guides/. Cisco training partners provide a number of courses on IP routing design, and your Cisco account or Jumpstart team will be able to provide other assistance. Also, see the book "Internet Routing Architectures," mentioned in the Introduction. A few routing issues specific to MPLS networks are considered in the next section.
The restrictions on summarization exist because summarization stops some types of label-switched paths being set up end-to-end.
For example, assume that an ABR summarizes reachability for 1.1.1.0/24, 1.1.2.0/24 and 1.1.3.0/24 with a single route for 1.1.0.0/16. Now assume that a packet with IP address 1.1.1.23 arrives with a label for 1.1.0.0/16. The ABR cannot label-switch the packet. It must look past the label and examine the IP address to find that the packet should go on to 1.1.1.0/24.
Because ATM LSRs cannot examine IP addresses, they may not do IP route summarization. Some ATM LSRs, such as the BPX 8650, have a limited ability to examine IP addresses by sending the packets to the Edge LSR function in their label switch controller. However this can be done only for a small minority of the traffic flowing through the ATM LSR.
This chapter has shown how many of the issues of designing MPLS networks are similar to those of designing ordinary IP networks. One important exception to this is the dimensioning of MPLS LVC requirements on each link. This design problem is illustrated in Figure 3-11.
In order to complete the design of an ATM MPLS network, a sufficient number of VCs must be reserved for use as LVCs on each link. This can be a problem because any ATM switch can support only a certain number of active VCs. This is particularly important if there are multiple ATM services—MPLS, PNNI, and so on—sharing the resources of the links in an IP+ATM network.
The design problem is to determine the number of LVCs required.
The required number of LVCs depends on:
The number of LVCs used in a particular area of a network depends on the number of IP destination-prefixes known in that area. This follows the normal rules for an IP network:
These destination prefix rules are shown in Figure 3-12.
Each ATM Edge LSR and each label switch controller will ask a neighboring MPLS node for LVCs for the destination-prefixes it knows about. If MPLS Class of Service is used, it may ask for up to four LVCs for each destination-prefix.
The requests for LVCs flow through the network according to the paths chosen by IP routing. With VC merge, the LVCs to each destination will be merged at each ATM LSR. This means that on each link, there is at most one LVC per destination in the Area. This is shown in Figure 3-13 Topology (a).
If MPLS Class-of-Service is used, then this is multiplied by the number of classes. If VC merge is not used, there may be many more LVCs.
For ATM Edge LSRs, the number of LVCs used per link depends on whether VC merge is being used in the network.
Let d be the number of destination-prefixes known in an area, and c be the number of Classes of Service used in the network. If VC merge is used, then the number of LVCs used per link l satisfies
If VC merge is not being used in the network, there are three dependencies:
If de is the number of destinations reachable through a particular ATM Edge LSR (this will often equal 1, due to summarization) and the total number of ATM Edge LSRs and LSCs in the area is n, then the number of LVCs used per link satisfies
A simpler equation applies in the particular case where all these conditions exist:
These conditions will often apply in the core of MPLS networks supporting VPNs, but not using VC Merge.
The number of LVCs used per link on the ATM Edge LSR in this case is given by:
One of the preceding three equations is then used to check whether a sufficient number of LVCs is available on the equipment, as shown in Table 3-5. Table 3-6 shows the LVC capacity of Cisco ATM Edge LSR interfaces.
| Device | Situation | Key Parameter | Equation |
|---|---|---|---|
Edge LSR | The network uses VC merge. | Number of active VCs supported per ATM link. | Equation 1 |
Edge LSR | The network does not use VC merge; there is one destination-prefix per LSR or Edge LSR, all links are unnumbered, and there are no out-of-area routes. | Number of active VCs supported per ATM link. | Equation 3 |
Edge LSR | The network does not use VC merge, all other situations apply. | Number of active VCs supported per ATM link. | Equation 2 |
| Device | Interface Hardware | Number of Active LVCs supported | Notes |
|---|---|---|---|
3600 | NM-1A ATM Network Modules | 1024 |
|
4700 | NP-1A ATM Network Processor Module | 1023 |
|
7200, 7500 | PA-A1 or standard ATM port adaptor | 2048 |
|
Catalyst 5500, 7200, 7500 | PA-A3 ATM port adaptor. | 4096 |
|
6400 | Node Route Processor (NRP) | 2048 | Capacity is reduced by one LVC for each active PVC that terminates on the NRP. |
MGX 8850 IP+ATM switch | Route Processor Module (RPM) | 4096 | Capacity is reduced by one LVC for each active PVC that terminates on the RPM. In addition, the PXM is limited to 16K LVCs. This is unlikely to be a problem unless more than three RPMs are used in an MGX 8850 shelf. |
12000 series routers | 4xOC-3 ATM Line Card | 2047 | The 2047 active VCs are shared between all four ports. Network capacity is reduced by one destination prefix for every second and subsequent route chosen for each destination according to equal-cost multipath routing, if the extra routes are on the same card. |
12000 series routers | 1xOC-12 ATM line card | 2047 |
|
Consider a network where VC merge is being used and one Class of Service is being used. If the Edge LSRs are all 7200 series routers with PA-A3 port adaptors, then how many IP destination prefixes can safely be supported in the area?
VC merge is being used, so Table 3-5 indicates that Equation 1 should be used. One Class of Service is being used, so c = 1. Table 3-6 states that the PA-A3 port adaptor supports 4096 LVCs, so l = 4096. Substituting these into Equation 1 gives
This means that 4096 destination-prefixes are guaranteed to be supported within the area, provided that the ATM LSRs do not impose a tighter limit (this discussion considers only the Edge LSRs).
Consider a network where VC merge is not used, and four Classes of Service are being used. This network is the core of an MPLS VPN service, and there is one destination-prefix per LSR or Edge LSR. All links are unnumbered. No out-of-area routes are injected into the interior routing protocol. The Edge LSRs are 7200 and 7500 series routers with a mixture of PA-A1 and PA-A3 ATM port adaptors.
What is the largest number of LSRs that can be used if the network consists of a single area? Assume that the ATM LSRs support a sufficiently large number of LVCs.
According to the conditions given, Table 3-5 indicates that Equation 3 should be used. Four Classes of Service are used, so c + 4. The Table 3-6 shows ATM interfaces each support 2048 or 4096 LVCs. The interfaces with 2048 LVCs give a tighter restriction, so use 1 = 2048. Substituting these into Equation 3 gives
This means that a maximum of 256 LSRs (Edge LSRs or ATM LSRs) may be used in the area, provided that the IP routing protocol supports that many routers in an area.
Consider a network where:
How many LSRs can be used in the entire network?
There are multiple areas and so there will be out-of-area routes in each area. Table 3-5 indicates that Equation 2 should be used in this case. Four Classes of Service are used, so c = 4. Table 3-6 gives
1 = 4096. There are at most 100 ATM LSRs and LSRs in each area, so use n = 100.
Observe that there is one route per Edge LSR. In the worst case, all out-of-area routes are accessed through a single LSR—this concentrates the LVC requirements on the links to that single LSR. In this case, de = (d - 100). Substituting these into Equation 2 gives
This means that only 109 LSRs can be used in the network.
By comparison with the previous example, we can see that using multiple areas can have major disadvantages in ATM MPLS networks without VC merge. These examples indicated that ATM LSRs without VC merge typically cannot be used in networks of larger than a few hundred nodes. An alternative that works around these limitations is to use the same switches, but to use MPLS-over-PVCs instead of ATM MPLS.
With VC merge, the LVCs to each destination will be merged at each ATM LSR. This means that there is at most one LVC per destination on each link, as shown in Figure 3-13 Topology (a). If MPLS Class-of-Service is used, then this is multiplied by the number of classes.
If d is the number of destination-prefixes known in an area, and c is the number of Classes of Service used, then the number of LVCs used per link l satisfies
Another important issue in switches that support VC merge is the number of LVCs that must be merged together in the switch, m. This depends on the number of links into the switch k. The limit is:
These equations are then used to check whether a sufficient number of LVCs is available on the equipment, as shown in Table 3-7. Both equations must be checked.
| Device | Key Parameters | Check Against |
|---|---|---|
ATM LSR with VC Merge | 1. Number of active VCs supported per ATM link. 2. Number of merging LVCs supported per switch, or per port card, whichever is applicable to the switch architecture. | Equation 4 Equation 5 |
In general, a per-switch limit applies to shared-memory switches such as the LS1010 or 8540 MSR. A per-port card limit applies to crossbar switches such as the BPX 8650.
In Table 3-8, the numbers of active LVCs supported are maximums; the actual limits will be dependent on configurations. In a BPX 8650, for example, the actual number of active LVCs supported per link must be down-rated by a minimum of 270 lines per interface if AutoRoute is enabled on that interface. On all switches, the VC space reserved for PVCs, SVCs, and so on, must be subtracted from the available VC space.
| Device | Interface Hardware | Number of Active LVCs Supported | Number of Active Merging LVCs Supported |
|---|---|---|---|
LS1010 | Any ATM port hardware. | 4096 per OC-3 port, 16K per OC-12 port, 16K per OC-48 port | 64K per switch |
6400 | Any ATM port hardware. | 4096 per OC-3 port, 16K per OC-12 port, 16K per OC-48 port | 256K per switch |
8540 MSR | Any ATM port adaptors. | 4096 per OC-3 port, 16K per OC-12 port, 16K per OC-48 port | 256K per switch |
BPX 8650 or 8680 | BXM-E cards | 32K per BXM, shared amongst up to 12 interfaces | 32K per BXM, with a maximum of 16K per port on OC-3 BXM cards and 2xOC-12 BXM cards. T3/E3 BXM cards and 1xOC-12 BXM cards have a limit of 32K per port. |
MGX 8800 with PXM-45 | AXSM cards | 128K per AXSM, shared amongst up to 16 interfaces | 128K per AXSM |
A network uses BPX 8650 ATM LSRs with VC merge. Two Classes of Service are used. Each BPX 8650 has 4x1-port OC-12/STM-4 BXM cards, with each port used to link to another ATM LSR or Edge LSR.
What limit do these ATM LSRs put on the number of IP-destination-prefixes that can be supported inside an area?
Table 3-7 shows that both Equation 4 and Equation 5 must be checked. Two Classes of Service are used, so c = 2. Each switch has four ports, so k = 4. Looking up the BPX 8650 in Table 3-8 shows that BXM cards support 32K active LVCs. In this case, each BXM card has one port, so each link supports 32K LVCs, or l = 32768. Table 3-8 shows that 32K LVCs can be merged into a 1-port OC-12 BXM card, so m = 32768.
Substituting these parameters into Equation 4 gives
Substituting the parameters into Equation 5 gives
The limit from Equation 5 is tighter, which means that the limit imposed by the ATM LSRs is 5461 destination-prefixes in the area. (The Edge LSRs might impose a tighter limit.)
A network uses 8540 MSR ATM LSRs with VC merge. Four Classes of Service are used. Each 8540 MSR has 8xOC-3/STM-1 with each port used to link to another ATM LSR or Edge LSR.
What limit do these ATM LSRs put on the number of IP-destination-prefixes that can be supported inside an area?
Table 3-7 shows that both Equation 4 and Equation 5 must be checked. Four Classes of Service are used, so c = 4. Each switch has eight ports, so k = 8. Looking up the 8540 MSR in Table 3-8 shows that OC-3 port cards support 4096 LVCs, or l = 4096. Similarly, Table 3-8 shows that the 8540 MSR supports 256K merging VCs, so m = 262144.
Substituting these parameters into Equation 4 gives
Substituting the parameters into Equation 5 gives
The limit from Equation 4 is tighter, which means that the limit imposed by the ATM LSRs is 1024 destination-prefixes in the area. (The Edge LSRs might impose a tighter limit.)
A network uses BPX 8650 ATM LSRs with VC merge. Four Classes of Service are used. Each BPX 8650 has eight ports, on 2x4-port OC-3/STM-1 BXM cards, with each port used to link to another ATM LSR or Edge LSR.
What limit do these ATM LSRs put on the number of IP-destination-prefixes that can be supported inside an area?
Table 3-7 shows that both Equation 4 and Equation 5 must be checked. Four Classes of Service are used, so c = 4. Each switch has eight ports, so k = 8. Looking up the BPX 8650 in Table 3-8 shows that BXM cards support 32K active LVCs. In this case, each BXM card has four ports, so we can assume that each link supports 32K/4 LVCs, or l = 8192. Similarly, Table 3-8 shows that 4xOC-3 BXM cards can support 32K merging VCs, with a maximum of 16K per port. The worst case is when all LVCs try to merge into the same port, so use so m = 16384.
Substituting these parameters into Equation 4 gives
Substituting the parameters into Equation 5 gives
The limit from Equation 5 is tighter, which means that the limit imposed by the ATM LSRs is 585 destination-prefixes in the area. (The Edge LSRs might impose a tighter limit.)
Without VC merge, there may be many VCs per destination on each link, as shown in Figure 3-13 Topology (b). If the total number of ATM Edge LSRs and LSCs in the area is n, then there may be up to c(n-1) LVCs per destination on each link.
The number of LVCs used per link will then satisfy
A tighter limit applies in the particular case where VC merge is not used, and there is one destination prefix per Edge LSR or LSC, and all links are unnumbered, and there are no address prefixes from outside the area. These conditions will often apply in the core of MPLS networks supporting VPNs but not using VC Merge.
The number of LVCs used in this case is given by:
Either Equation 6 or Equation 7 is then used to check whether a sufficient number of LVCs is available on the equipment, as shown in Table 3-9. Table 3-10 shows the limits of Cisco ATM LSRs without VC merge capability.
| Device | Situation | Key Parameter | Check Against |
|---|---|---|---|
ATM LSR without VC Merge | There is one destination-prefix per LSR or Edge LSR, all links are unnumbered, and there are no out-of-area routes. | Number of active VCs supported per ATM link. | Equation 3 |
ATM LSR without VC Merge | All other situations. | Number of active VCs supported per ATM link. | Equation 2 |
| Device | Interface Hardware | Number of Active LVCs Supported per Link |
|---|---|---|
BPX 8650 | Older BXM cards (or pre-9.3.x software) | 16K per BXM, shared amongst up to 12 interfaces |
|
Note The numbers of active LVCs per link are maximums; the actual limits will be dependent on configurations. In a BPX 8650, for example, the actual number of active LVCs supported per link must be down-rated by a minimum of 270 lines per interface if AutoRoute is enabled on that interface. On all switches, the VC space reserved for PVCs, SVCs, and so on must be subtracted from the available VC space. |
A network uses BPX 8650 ATM LSRs without VC merge. One Class of Service is used. Each BPX 8650 has 2x4-port OC-3/STM-1 BXM cards, with each port used to link to another ATM LSR or Edge LSR. There is one destination-prefix per LSR or Edge LSR. All links in the area are unnumbered, and there are no out-of-area routes known.
What limit do these ATM LSRs put on the number of LSRs or Edge LSRs that can be supported inside an area?
Table 3-9 shows that Equation 7 should be checked in this case. One Class of Service is used, so c = 1. Looking up the BPX 8650 in Table 3-8 shows that BXM cards support 16K active LVCs. In this case, each BXM card has four ports, so each link supports 16K/4 LVCs, or l = 4096.
Substituting these parameters into Equation 7 gives
This means that the limit imposed by the ATM LSRs is 90 LSRs or Edge LSRs.
These examples indicate that ATM LSRs without VC merge typically cannot be used in networks of larger than a few hundred nodes. An alternative that works around these limitations is to use the same switches, but to use MPLS-over-PVCs instead of ATM MPLS.
A network uses BPX 8650 ATM LSRs without VC merge. Two Classes of Service are used. Each BPX 8650 has 4x1-port OC-12/STM-4 BXM cards, with each port used to link to another ATM LSR or Edge LSR.
What limit do these ATM LSRs put on the number of IP-destination-prefixes that can be supported inside an area?
No information is given on the relationship between devices and routes, so Table 3-9 shows that Equation 6 should be checked. Two Classes of Service are used, so c = 2. Looking up the BPX 8650 in Table 3-8 shows that BXM cards support 16K active LVCs. In this case, each BXM card has one port, so each link supports 16K LVCs, or l = 16384.
Substituting these parameters into Equation 6 gives
Because n the number of LSRs in the area has not been given, this question cannot be explicitly answered as stated. However, if it is assumed that n = 50 this would give an indicative value of d >167.
In other words, the number of destination prefixes that may be supported depends on the number of LSRs in the area with, for example, a limit of 167 destination prefixes if there are 50 LSRs in the area.
The following sections provide further considerations about the preceding examples.
The limits on destination prefixes indicated in the previous examples are much smaller than the size of the Internet backbone routing table, which is about 70,000 routes. Despite this, ATM MPLS can still be used in networks with full Internet routing, by use of an MPLS feature known as BGP Next-Hop Labelling.
BGP Next-Hop Labelling allows BGP Autonomous System Boundary Routers (ASBRs) to exchange the full Internet routing table with each other by way of BGP, while readvertising only a limited subset of these addresses (or none at all) into the Interior routing protocol (OSPF or ISIS) Areas through which they are connected. Because only a limited set of destination-prefixes is known on OSPF or IS-IS in the MPLS network, the limits discussed here are sufficient even though they are much smaller than the Internet routing table.
Cisco MPLS Virtual Private Networks (VPNs) extend the BGP Next-Hop Labelling technique to deal with address from many different customers' VPNs.
The limits shown in Equations 4 to 7 apply when MPLS Traffic Engineering is not used. If Traffic Engineering is used, then one LVC will be used for each Traffic Engineering Tunnel on each link, in addition to the limits shown above.
VP Tunnels involve several logical links terminating on a single physical interface on an LSR or ATM LSR. When VP Tunnels are terminate on an interface, the LVCs on all VP Tunnels must be taken into account. For example, if four VP Tunnels terminate on a logical interface that supports 4000 LVCs, then an average of only 1000 LVCs will be available per VP Tunnel.
The limits shown in Equations 4 to 7 can be quite loose. The number of LVCs actually used on a particular link may be much less than these limits suggest, particularly if VC merge is not being used. However, it is difficult to calculate exactly how many will be required. This depends on the exact shape and state of the network, and the exact paths chosen by IP routing. If this can be analyzed, taking account of such things as failed links and multipath routing, then fewer LVCs could be safely reserved on each link: a very complex process. In any case, the limits shown above will be safe.
A network that has run out of cross-connects is basically not functioning. There is no way to control which LVCs are created and which are not. This may result in permanent black holes or overloaded LSCs. The network should be designed from the start so that it will not run out of connections.
Two different behaviors might occur when a link runs out of LVCs, depending on the type of traffic:
In case of LVC starvation, it is usually impossible to predict which LVCs are created and which are not. This is because LVCs are created roughly in the order in which routing converges. This is, in turn, dependent on random factors such as which links fail, and the exact timing of link failures compared to routing protocol update timers.
In other words, it is very important to design the network and allocate switch resources so that a sufficient number of LVCs are available on each link.
Network design is an ongoing process. Once an ATM-MPLS network is deployed, ongoing design activities are required to:
The ongoing process will involve the following steps:
1. Measure actual PoP and link traffic, and compare against
2. Based on the comparison between predicted and actual traffic, some modeling assumptions might be changed. For example, the traffic distribution across nodes might be different than that initially predicted. Review the initial design and dimensioning if the modeling assumptions are changed. Lower and higher-bandwidth links might be required.
3. As customers are added, and as traffic increases, review the initial design to:
Posted: Mon Sep 25 11:54:19 PDT 2000
Copyright 1989-2000©Cisco Systems Inc.