|
|
Your Cisco 12016 Gigabit Switch Router (GSR) went through extensive testing and burn-in before leaving the factory. However, if you encounter problems when starting the router for the first time, use the information in this chapter to help isolate the cause of the problems. Problems with the initial startup are most likely caused by the source power or a system card not properly seated in the backplane.
This chapter contains the following sections:
This section describes the methods used in troubleshooting the router. To enable efficient problem solving, the troubleshooting methods are organized according to the major subsystems in the router.
If your system failed its initial attempt at starting up, as described in the"Starting the Router and Observing Initial Conditions" section in "Observing System Startup and Performing a Basic Configuration," verify the following conditions:
When all of these conditions are met, the hardware check is complete, and you should proceed to "Observing System Startup and Performing a Basic Configuration," to perform a basic system configuration, and then to the appropriate companion software configuration documentation to configure the Cisco IOS software, protocols, and individual interfaces in your system.
If the startup sequence again fails before the system software boots successfully, use the procedures in this chapter to isolate and, if possible, resolve the problem.
If you cannot solve a problem on your own, contact a Cisco customer service representative for assistance. When you call, please have the following information ready:
Table 5-1 shows the normal contents of the alphanumeric displays on the GRP and the line cards, as well as the normal LED states on the alarm card, the power modules (AC-input or DC-input), and the blower modules at system startup.
| Component | LED | Display Contents/LED Status and Meaning |
|---|---|---|
GRP | Alphanumeric displays | Top row: MSTR The GRP is enabled and recognized by the system; a valid Cisco IOS software image is running. |
Line card | Alphanumeric displays | Top row: IOS The line card is enabled and ready for use. |
Alarm card | Detected alarm severity Alarm card CSC 0 and 1 SFC 0, 1, and 2 | Critical: Off Enabled: On Enabled: On Enabled: On |
Power module | Power status | PWR OK: On The correct power module voltages are present and no faults have been detected. |
Blower module | Blower status | OK: On |
The key to solving system problems is to try to isolate the problem to a specific subsystem. The first step in solving startup problems is to compare what the router is doing with what it should be doing. Because a startup problem is usually attributable to a single component, it is more efficient to examine each subsystem, rather than to try to troubleshoot each router component.
For purposes of the troubleshooting procedures in this chapter, the Cisco 12016 GSR consists of the following subsystems:
In a normal router startup sequence, the following events and conditions occur:
By checking the status LEDs on the power modules and the alphanumeric displays on the GRP and line cards, you can generally determine when and where the router failed in the startup sequence.
The following sections describe what you should expect to see in the power module LEDs on router startup.
Figure 5-1 shows the location of the LEDs on the power supply faceplate.
If either the AC input voltage to the power supply or the -48 VDC output voltage to the backplane falls outside the required range, the green LED labeled PWR OK will go off and the yellow LED labeled FAULT will go on. Because of the redundancy scheme used in both the standard and optional AC-input power subsystem, a problem with the DC output voltage from only one power supply to the backplane should not affect router operation.
If an AC-input power supply detects an internal overtemperature condition, the power supply sends an overtemperature warning to the system across the MBus, then shuts down. The yellow LEDs labeled FAULT and TEMP are on.
If an AC-input power supply detects a current limit condition, the power supply sends a fault signal to the system across the MBus. The yellow LED labeled ILIM is on.
When the Cisco 12016 GSR is equipped with the standard AC-input power shelf and three power supplies, a problem with the DC output voltage from two power supplies prevents the router from starting up or continuing normal operation.
When the Cisco 12016 GSR is equipped with the optional AC-input power shelf and four power supplies, a problem with the DC output voltage from three power supplies prevents the router from starting up or continuing normal operation.
Figure 5-2 shows the location of the LEDs on the power entry module faceplate.
For a DC-input power entry module, normal operating behavior occurs when the following conditions are met:
If either the DC input voltage to the power entry module or the -48 VDC output voltage to the power shelf backplane falls outside the required range, or the power entry module circuit breaker is switched off, the green LED labeled PWR OK will go off and the yellow LED labeled FAULT will go on.
Because of the 2N redundancy scheme, a problem with the DC output voltage from just one power entry module in a backplane load zone to the backplane should not affect router operation. A problem with the DC output voltage from both power entry modules in a load zone prevents the router from starting up or continuing normal operation.
If a DC-input power entry module detects an internal overtemperature condition, the power entry module sends an overtemperature warning to the system across the MBus, then shuts down. The yellow LEDs labeled FAULT and TEMP are on.
Figure 5-3 shows the location of the LEDs on the alarm card faceplate.
If there are no alarm conditions within the router, the critical, major, and minor alarm LEDs on the alarm card faceplate should be off.
If there are no faults on the alarm card itself, the green LED labeled ENABLED on the alarm card faceplate should be on, and the yellow LED labeled FAIL should be off.
Figure 5-4 shows the location of the alphanumeric LEDs on the GRP faceplate.
When the router is powered on, the four-character alphanumeric displays on the GRP indicate the following:
Status messages are displayed as the boot process continues. (See Table 5-2.)
Figure 5-5 shows the location of the alphanumeric LEDs on the line card faceplate.
When the line card is powered on, the four-character displays indicate the following:
Status messages appear in the alphanumeric LED displays as the boot process continues on the line card. Table 5-3 lists the messages that appear in the line card alphanumeric LED displays. Some messages are displayed only for a few milliseconds; others are displayed for several seconds.
The section includes the following subsections:
AC-input power supplies are monitored for internal temperature, voltage, and current load by the MBus controller module on the power bus board in the power interface panel in the back of the chassis and by the MBus module on the GRP. If the router detects an extreme condition, it generates an alarm on the alarm card and logs the appropriate warning messages on the console.
To begin checking the AC-input power subsystem, examine the four LEDs on the power supply faceplate.
When a AC-input power supply is seated in its power shelf bay and connected to an AC power source supplying power within the required range, the green LED labeled PWR OK on the power supply faceplate should be on and the power supply fan should also be on. This is normal behavior.
If either the AC input voltage to the power supply or the -48 VDC output voltage to the backplane falls outside the required range, the green LED labeled PWR OK will go off and the yellow LED labeled FAULT will go on. Because of the redundancy scheme used in the standard and optional AC-input power subsystems, a problem with the DC output voltage from just one power supply to the backplane should not affect router operation.
When the Cisco 12016 GSR is equipped with the standard AC-input power shelf and three power supplies, a problem with the DC output voltage from two power supplies prevents the router from starting up or continuing normal operation.
When the Cisco 12016 GSR is equipped with the optional AC-input power shelf and four power supplies, a problem with the DC output voltage from three power supplies prevents the router from starting up or continuing normal operation.
Continue checking the AC-input power subsystem by assessing the following:
In a GSR equipped for AC-input operation, the power bus board in the power interface panel inside the GSR chassis is equipped with two user-replaceable fuses. One fuse protects the MBus controller module that monitors the operation of the power subsystem; the other fuse protects the MBus controller module against voltage or current monitoring signals (Vmon and Imon) that exceed the specified voltage threshold.
The voltage and current monitoring signals permit the MBus controller module to monitor the voltage and current from the power supplies in the AC-input power shelf.
You can issue the show environment command at the user EXEC mode prompt on the system console to display operational information from the power bus board MBus controller module.
Because the MBus controller module only monitors the operation of the power shelf and does not control its operation, an open MBus controller module fuse renders the power shelf invisible to the system, but the system continues to operate. No power shelf operational information is displayed in response to a console command.
An open fuse on the Vmon and Imon +5.1 VDC bias line causes the Vmon and IMon signals to go to zero. The power shelf and system continue to operate. These signals are monitored by the MBus controller module.
To replace these fuses, refer to the "Replacing a Power Bus Board Fuse" section in "Maintaining Your Cisco 12016 GSR."
If you are unable to resolve the problem, or if you determine that either the power supply or power cord is faulty, contact your Cisco service representative for assistance.
DC-input power entry modules are monitored for internal temperature, voltage, and current load by the MBus module in the power interface panel in the back of the chassis, and by the MBus module on the GRP. If the router detects an extreme condition, it generates an alarm on the alarm card and logs the appropriate warning messages on the console.
To begin checking the DC-input power subsystem, examine the three LEDs on the power entry module faceplate.
For a DC-input power entry module, normal operating behavior occurs when the following conditions are met:
If either the DC input voltage to the power entry module or the -48 VDC output voltage to the power shelf backplane falls outside the required range, or the power entry module circuit breaker is switched off, the green LED labeled PWR OK will go off and the yellow LED labeled FAULT will go on.
Because of the 2N redundancy scheme, a problem with the DC output voltage from just one power entry module in a backplane load zone to the backplane should not affect router operation. A problem with the DC output voltage from both power entry modules in a load zone prevents the router from starting up or continuing normal operation.
Continue checking the power subsystem by assessing the following:
In a GSR equipped for DC-input operation, the power bus board in the power interface panel inside the GSR chassis is equipped with a fuse to protect the MBus controller module that monitors the operation of the power subsystem.
You can issue the show environment command at the user EXEC mode prompt on the system console to display operational information from the power bus board MBus controller module.
Because the MBus controller module only monitors the operation of the power shelf and does not control its operation, an open MBus controller module fuse renders the power shelf invisible to the system, but the system continues to operate. No power shelf operational information is displayed in response to a console command.
To replace the power bus board MBus controller module fuse, refer to the "Replacing a Power Bus Board Fuse" section in "Maintaining Your Cisco 12016 GSR."
If you are unable to resolve the problem or if you determine that either the power entry module or power cable is faulty, contact your Cisco service representative for assistance.
The DC voltage from the power shelf is transferred to the two load zones on the chassis backplane by two redundant power bus bars. (See Figure 5-6.) System load zone 1 on the chassis backplane distributes a nominal DC source of -48 VDC to the upper blower module and the upper card cage. System load zone 2 distributes a nominal DC source of -48 VDC to the switch fabric card cage, the lower card cage, and the lower blower module.
The -48 VDC from the backplane feeds DC-to-DC converters on each card. The MBus module on each card controls the DC-to-DC converters. When directed by the GRP or by MBus software, the MBus module turns on the DC-to-DC converters, which convert the -48 VDC into the voltages required by the card.
DC-to-DC converters on the fabric cards take the -48 VDC from the chassis backplane and produce +5 VDC to power the MBus modules on those cards. DC-to-DC converters on the alarm cards also take the -48 VDC from the chassis backplane and produce +5 VDC to power the MBus modules on those cards, but also put +5 VDC back onto the backplane for the MBus modules on the GRP, the line cards, the power bus board, and in the blower modules.
The -48 VDC power for the two blower modules is transferred from the chassis backplane to the blower modules through two shielded wiring harnesses and floating connectors built into the back of the chassis. An internal controller card in the blower module converts -48 VDC into a variable DC voltage that powers the blower module fans. The MBus module in each blower module uses +5 VDC from the chassis backplane to power the MBus interface circuitry and the temperature sensor.
Begin troubleshooting the power distribution system by assessing the following:
This section contains the following subsections:
The Cisco 12016 GSR processor subsystem consists of the GRP, the line cards, and the alarm cards.
The GRP and the line cards each have two onboard processors. One processor serves as the card's main processor, and the other serves as the MBus module processor. The MBus module processor monitors the card's environment and controls the card's onboard DC-to-DC converter.
As soon as power is supplied to the system, the MBus module begins to operate. The MBus module determines what type of card it is mounted on, and whether it should turn on the card's DC-to-DC converters. The MBus module on the GRP turns on card power after a brief delay, but the MBus modules on the line card delay turning on power until they receive a command to do so from the GRP MBus module.
To operate at all, a minimally configured Cisco 12016 GSR must have a GRP installed in slot 7 of the upper card cage. If the Cisco 12016 GSR is equipped with an optional, redundant GRP, that GRP must be installed in the far left slot in the lower card cage (slot 8).
While the router cannot operate without a properly installed GRP, it can operate without any line cards installed, as long as no line card is in partial contact with the backplane pins.
A line card that is only partially seated in the backplane sends incomplete signals to the GRP, which can cause the router to halt. First, ensure that the GRP is installed properly and that the system software has initialized successfully. Then, if necessary, you can troubleshoot individual line cards.
To isolate a problem with the GRP, use the alphanumeric LED displays on the GRP.
The GRP alphanumeric LED displays are located at the bottom of the GRP faceplate, near the card ejector lever. (See Figure 5-7.) To clarify the LED display message format, Figure 5-7 shows the LED displays in the head-up orientation in the upper card cage.
To isolate a problem with the GRP, check the GRP alphanumeric displays:
| LED Message | Description |
|---|---|
LMEM | Running low memory test |
LCAH | Initializing lower 15K cache |
BSS | Initializing main memory for ROM |
NVRAM | Initializing NVRAM |
EXPT | Initializing interrupt handlers |
TLB | Initializing TLB |
CACH | Initializing CPU data and instruction cache |
CACH | Enabling CPU cache parity |
MEM | Initializing main memory |
NVRAM | Detecting the NVRAM size |
PCMC | Initializing the PCMCIA |
EXIT | Exiting the initialization sequence |
IOS |
There are eight numbered card slots in the upper card cage and eight numbered card slots in the lower card cage. The master GRP must be installed in slot 7 of the upper card cage. If the router is equipped with an optional, redundant GRP, that GRP must be installed in the far left slot in the lower card cage (slot 8), leaving available slots for up to 14 line cards in the upper and lower card cages. If the router is not equipped with an optional, redundant GRP, up to seven line cards can be installed in slots 0 through 6 in the upper card cage, and up to eight line cards can be installed in slots 8 through 15 in the lower card cage.
As each line card powers up in response to a command issued by the GRP across the MBus, a power-on self-test (POST) is performed on the line card memory. A full set of field diagnostics can also be run on a line card from the system console. The diagnostics provide either a pass or fail message in the line card alphanumeric LED displays, as well as on the system console. (For information on diagnostic testing, refer to "Running Diagnostics on the Cisco 12016 GSR."
The line card alphanumeric LED displays are located at one end of the line card faceplate, near the card ejector lever. Line cards installed in the upper card cage are installed in the head-up orientation, so the alphanumeric LED displays are located at the bottom of the upper card cage. Line cards installed in the lower card cage are installed in the head-down orientation, so the alphanumeric LED displays are located at the top of the lower card cage. (Figure 5-8 shows the LED displays in the head-up orientation to clarify the LED display message format.)
To isolate a problem with a line card, check the line card alphanumeric displays:
| LED Message | Description |
|---|---|
MEM | Running POST memory test |
LROM | After POST memory test |
BSS | Initializing main memory for ROM |
RST | Saving reset reason register |
IO | Resetting the I/O system on the card |
EXPT | Initializing interrupt handlers |
TLB | Initializing TLB |
CACH | Initializing CPU data and instruction cache |
MEM | Initializing main memory |
LROM | Ready to access download |
ROMI | Getting ROM images |
FABL | Waiting for fabric downloader to load |
FABL | Loading fabric downloader |
FABL | Launching fabric downloader |
FABL | Fabric downloader launch complete |
IOS | Downloading the Cisco IOS software |
IOS | Launching the Cisco IOS software |
IOS | Running the Cisco IOS software |
IOS | Line card enabled |
The Cisco 12016 GSR is equipped with two alarm cards. One card occupies the dedicated far left slot of the upper card cage. A second alarm card occupies the dedicated far right slot of the lower card cage. In both card cages, the alarm card slot differs from the rest of the card cage slots in that it is labeled to identify it as an alarm card slot, is physically narrower, and has a different backplane connector.
The Cisco 12016 GSR alarm card (see Figure 5-9) has three primary functions:
The alarm card faceplate is equipped with three pairs of LEDs---labeled Critical, Major, and Minor (see Figure 5-9)---that identify system alarm conditions detected through the MBus.
The alarms can warn of an overtemperature condition on a component in one of the card cages, a fan failure in a blower module, an overcurrent condition in a power supply, or an out-of-tolerance voltage on one of the cards in one of the card cages. The LEDs are driven by MBus software, which sets the threshold levels for triggering the different stages of alarms.
The GRP continuously polls the system for temperature, voltage, current, and fan speed values. If an over-threshold value is detected, the GRP sets the appropriate alarm severity level on the alarm card, which lights one of the LED pairs and energizes the appropriate alarm card relays, activating any external audible or visual alarms wired to the alarm card. The GRP also logs a message about the threshold violation on the system console.
The alarm card faceplate has one pair of LEDs that indicates the operational status of the alarm card.
A green LED labeled ENABLED indicates that the card has been detected by the system and is okay. A yellow LED labeled FAIL indicates that the system has detected a fault in the alarm card.
If no faults have been detected on the alarm card, the green LED labeled ENABLED on the alarm card faceplate should be on, and the yellow LED labeled FAIL should be off.
If there are no alarm conditions within the router, the critical, major, and minor alarm LEDs on the alarm card faceplate should be off.
If there are no faults on the alarm card itself, the green LED labeled ENABLED on the alarm card faceplate should be on, and the yellow LED labeled FAIL should be off.
If there are no faults on the SFCs (SFC 0, SFC 1, or SFC 2), the green LED for each SFC should be on, and the yellow LED for each SFC should be off. If the system detects an SFC fault, it turns off the green LED for the faulty card, turns on the yellow LED, logs a warning message on the system console, but continues operating.
The cooling subsystem of the Cisco 12016 GSR consists of two blower modules in the chassis and a fan in each of the power modules in the power shelf. The blower modules and the power module fans circulate air to maintain acceptable operating temperatures within the router.
The upper and lower blower modules are identical, so they are interchangeable. The blower module is a sheet metal enclosure containing three blowers, a blower controller card, and two faceplate LEDs. (See Figure 1-17.) Both blower modules have snap-on plastic front covers mounted over the blower module faceplates. Two blower module LEDs are visible through the front covers.
When on, the green LED labeled OK indicates that the blower module is functioning normally.
The blower module operates from -48 VDC distributed to it directly from the chassis backplane. A controller card on the blower module converts -48 VDC into a variable DC voltage that powers the blowers, and uses +5 VDC distributed to it directly from the chassis backplane to power the MBus interface circuitry and the temperature sensor.
The three fans in each blower module are variable-speed fans, and all three fans should be operating any time system power is on. The variable-speed feature allows the fans to operate at a lower speed when the internal chassis temperature is within the normal operating range, and at a higher speed if the internal temperature exceeds a specified temperature.
If the air temperature inside the chassis rises, blower speed increases to provide additional cooling air to the internal components. If the internal air temperature continues to rise beyond the specified threshold, the system environmental monitor shuts down all internal power to prevent equipment damage from excessive heat.
To isolate a problem with the chassis cooling system, assess the following conditions:
Queued messages: %ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdown
There are no field-replaceable components in the blower module. If the blower module is faulty, you must replace the entire blower module.
If you are still unable to resolve the problem, contact a Cisco customer service representative for assistance.
Each power module (AC-input power supply or DC-input power entry module) is equipped with a fan that draws cooler air in the front of the power module and forces warmer out the back of the power shelf.
When a power module is seated in its power shelf bay and connected to a power source supplying power within the required range, the power module fan should be on. As long as the power source is within the required range, the fan should remain on.
If the fan in an power module fails, when the power supply detects an internal overtemperature condition, it sends an overtemperature warning to the system across the MBus, then shuts down. The yellow LEDs labeled FAULT and TEMP are on.
You should also look for console messages indicating that the system has detected a power module fan failure or an overtemperature condition inside the power module. Use the show environment command or the show environment all command to display information about the internal system environment, including power supply fan rotation and temperature measurements within the power module.
There are no field-replaceable components in either the AC-input power supply or the DC-input power entry module. If the power module fan fails, you must replace the power module.
If you are still unable to resolve a problem, contact a service representative for assistance.
The following additional reference materials are available for troubleshooting your Cisco 12016 GSR installation:
Cisco documentation and additional literature are available in a CD-ROM package, which ships with your product. The Documentation CD-ROM, a member of the Cisco Connection Family, is updated monthly. Therefore, it might be more current than printed documentation. To order additional copies of the Documentation CD-ROM, contact your local sales representative or call customer service. The CD-ROM package is available as a single package or as an annual subscription. You can also access Cisco documentation on the World Wide Web at http://www.cisco.com, http://www-china.cisco.com, or http://www-europe.cisco.com.
Posted: Mon Jun 5 13:48:47 PDT 2000
Copyright 1989 - 2000©Cisco Systems Inc.
