cc/td/doc/product/core/cis12008
hometocprevnextglossaryfeedbacksearchhelp
PDF

Table of Contents

Troubleshooting the Installation

Troubleshooting the Installation

Your Cisco 12008 was subjected to extensive testing and burn-in before being shipped from the factory. However, if you encounter problems starting up the router, the information in this chapter will help you to isolate the probable cause.

This chapter contains the following sections:

In this chapter, it is assumed that you are troubleshooting the initial Cisco 12008 system startup, and that the system is in the original factory configuration. If you have removed or replaced components or changed any default settings, the recommendations in this chapter might not apply.

At initial system startup, you should verify the following:

If you cannot solve a problem, contact a customer service representative for assistance. When you call, have the following information at hand:

Troubleshooting Overview

This section identifies the normal status of the system at startup. It also describes the methods used in troubleshooting the Cisco 12008. To enable efficient problem solving, the troubleshooting methods are presented organized according to the router's major subsystems.

Normal System Status at Startup

Table 5-1 shows the contents of the alphanumeric displays on the GRP and the line cards at system startup; this table also shows the normal, expected state of the LEDs on the faceplate of the CSC(s) at system startup.

For the layout of the status LEDs on the CSC faceplate, refer to Figure 1-12.


Table  5-1: Status of Alphanumeric Displays and LEDs at System Startup
Card LED Function Display Contents/LED Status/Meaning
GRP LED alphanumeric displays MSTR (top)

RP (bottom)

The GRP is enabled and recognized by the system; a valid Cisco IOS software image is running.

Line card LED alphanumeric displays IOS (top)

RUN (bottom)

The line card is enabled and ready for use.

CSC(s)

System alarm LEDs CRITICAL: OFF

MAJOR: OFF

MINOR: OFF

No system alarm conditions exist.

CSC Status LEDs FAIL: OFF

ENABLED: ON

The CSC is functional.

FAN FAIL Status LEDs LINECARD (left LED): OFF

PWR SPLY (right LED): OFF

The fan trays are operational.

SFC Status LEDs FAIL (top): OFF

ENABLED (bottom): ON

All the SFCs are functional.

AC-input power supply Power status LEDs AC INPUT OK (top): ON

OUTPUT FAIL (bottom): OFF

The power supply voltages are present and within tolerance.

DC-input power supply Power status LEDs INPUT OK (top): ON

OUTPUT FAIL (bottom): OFF

The power supply voltages are present and within tolerance.

Problem Solving Using a Subsystem Approach

The key to solving system problems is to try to isolate the difficulty to a specific subsystem. The first step in solving startup problems, for example, is to compare what the system is doing to what it should be doing.

Since a startup problem is usually attributable to a single component, it is more effective to isolate the problem to a specific subsystem, rather than trying to troubleshoot each system component.

For purposes of the troubleshooting procedures in this chapter, the Cisco 12008 will be regarded as consisting of the following subsystems:

Each fan tray incorporates individual fans that draw +24 VDC from a DC-DC converter on the CSC that operates under control of its onboard MBus module. Both fan trays should begin to operate about two seconds after application of system power.
The fan trays incorporate a variable-speed feature, enabling the fans to run at a slower speed (when the internal chassis temperature remains within the normal operating range) or to run at a higher speed (when the internal temperature of the router exceeds a specified threshold). In the latter case, the voltage being delivered to the fans is increased, causing them to run at maximum speed to force a greater volume of air through the router.
In a noisy, air-conditioned environment, it may be difficult to hear whether or not the fan trays are operating. The first indication of a fan failure comes from the side-by-side fan tray status LEDs on the CSC faceplate (see Figure 1-12).
If the LED on the left is on (amber), a fan failure has occurred in the card cage fan tray. If the LED on the right is on (amber), a fan failure has occurred in the power supply fan tray.
Each fan in a fan tray is monitored separately for failure. A failed fan is not shut off in the usual sense; instead, a current-limiting feature in the faulty fan prevents it from interfering with the operation of other fans. The most common fan failure is that the fan just stops running, but continues to draw current.
If a fan in the card cage fan tray or the power supply fan tray fails, the CSC increases the voltage being delivered to the fans, causing them to run at maximum speed to compensate for the failed fan.
You can make a gross assessment of fan tray operation by placing your hand near the exhaust vents at the top rear of the router enclosure. A reduced flow of air from the card cage exhaust vent or the power supply exhaust vent may indicate that one or more fans on a given fan tray have failed. However, this method of checking fan operation is neither definitive nor reliable.
If you determine that a fan tray is not operating properly, you should immediately consider replacing the faulty fan tray to guard against an overheating condition in the router that could approach the shutdown threshold.
No operating adjustments can be made for either fan tray. An individual fan is either operating normally or it is not operating at all.
A line card or GRP that is not firmly seated in the backplane might cause the system to hang and crash. Two 4-character alphanumeric displays at the bottom of the GRP and each line card provide status and error messages that can be an aid in troubleshooting.

Identifying Startup Problems

Startup problems are commonly due to problems with source power or to a card that is not properly seated in the backplane.

Normal System Startup Sequence

Each card installed in the system incorporates an MBus module and at least one DC-DC converter. The MBus module on each card provides an interface to the GRP via the system's maintenance bus (MBus); the MBus module on each card also controls its own onboard DC-DC converter.

When you first start up the router, the following sequence of events occurs:

If the onboard MBus module determines that it is mounted on a line card or an SFC, the MBus module remains idle for the time being, leaving the card powered down until it receives power up instructions from the master MBus module on the GRP.
The top display is powered by the DC-DC converter on the card; the bottom display is powered by the +5 VDC provided through the backplane by the AC-input or DC-input power supply(ies).

By checking the status LEDs on the power supply(ies) and the alphanumeric displays on the GRP and each line card, you can generally determine when and where the system failed during the startup sequence.

The following section describes what you should expect to see in the power supply LEDs on system startup.

Power Supply Status LEDs

When you start up the system by turning on the rotary power switch on the power supply faceplate, the following should occur:

If this LED does not go on, or if it goes off while system power is still applied, there could be a problem with either the source power itself or the DC output voltages being delivered to the backplane (+5 VDC and -48 VDC). There could also just be a problem with the MBus controller inside the power supply.
The AC INPUT OK LED (AC-input power supply) or the INPUT OK LED (DC-input power supply) stays on when all of the following conditions are satisfied:

  • The power supplies are on and receiving normal source power.

The source power specifications for the AC-input power supply are 180 VAC to 264 VDC, 50 to 60 Hz, single phase.
The source power specifications for the DC-input power supply are -40.5 VDC to -70 VDC.

  • The power supplies are providing the nominal +5.2 VDC and -48 VDC operating voltages to the backplane.

If the AC or DC source power or the DC operating voltages being supplied to the backplane fall outside the allowable tolerances, the OUTPUT FAIL LED on the power supply goes on, because a problem with either of the DC operating voltages being supplied to the backplane prevents the system from starting up or continuing normal operations.
For example, if a problem occurs with the -48 VDC line that supplies power to the router's internal components (see Figure 1-21), the system fails during the boot sequence.
Depending on when the OUTPUT FAIL LED goes on, proceed as follows:

  • If the AC INPUT OK LED (AC-input power supply) or the INPUT OK LED (DC-input power supply) stays off when system power is applied, there is a problem with either the AC or the DC source power or the connection to the power supply. In this case proceed to the section entitled "Troubleshooting the Power Subsystem."

  • If the AC INPUT OK LED (AC-input power supply) or the INPUT OK LED (DC-input power supply) goes on temporarily and then goes off within 30 seconds, the system is probably shutting itself down because it detected an out-of-tolerance power or temperature condition within a power supply. In this case proceed to the section entitled "Troubleshooting the Power Subsystem."

  • If the AC INPUT OK LED (on the AC-input power supply) or the INPUT OK LED (on the DC-input power supply) goes on, and the system starts up as expected but then displays the following message and shuts down after 2 minutes, there may be a problem with one of the fan trays.

If background noise prevents you from hearing fan tray operation, place your hand at the top rear of the router enclosure to determine if air is being exhausted from the fan tray vents. You should feel a steady volume of air coming from these vents.
If the AC INPUT OK LED (AC-input power supply) or the INPUT OK LED (DC-input power supply) is on, but the fan trays are not operating, there might be a problem with the fan trays (or, possibly, the CSC, which powers the fan trays).
To check the operating voltage of the fan trays, issue either the show environment or the show environment all command. (Either command displays information about the system's internal environment.) If the fan tray voltage is okay, the fan trays themselves may be faulty. If the fan tray voltage is bad, there may be a problem with the CSC (which supplies the fan tray operating voltages).
If the system detects that it is overheating due to a fan tray failure, it shuts itself down. In this case, proceed to the section entitled "Troubleshooting the Cooling Subsystem."

Troubleshooting the Power Subsystem

The power subsystem in the Cisco 12008 consists of the following:

Each power supply provides +5.2 VDC and -48 VDC output voltages to the backplane. The +5.2 VDC output powers the MBus modules on each card in the system. The MBus modules, in turn, control the DC-DC converters on each card. The DC-DC converter takes the -48 VDC from the power supply and converts it into the various voltages required to operate card circuitry.

To begin checking the power subsystem, examine the status of the two LEDs on the power supply faceplate at initial system startup for the following conditions:

In systems with a single AC-input power supply or a single DC-input power supply, and in systems with redundant power when both power supplies are being shut down, the OUTPUT FAIL LED goes on momentarily as the system shuts down, but goes off and remains so when the power supply completely shuts down.

Each AC-input or DC-input power supply is monitored by its own onboard MBus module, as well as by the master MBus module on the GRP. Each power supply is monitored for internal temperature conditions, overvoltage conditions, and overcurrent conditions.

Continue checking the power subsystem by assessing the following:

If the internal circuit breaker of the power supply has been tripped, you can reset it by turning the rotary power supply switch OFF then ON again. However, note that the tripping of the internal circuit breaker is a likely indication of an electrical problem in the chassis.

Note Each power supply in the Cisco 12008 should be attached to a separate power source.

If you are unable to resolve the problem, or if you determine that either the power supply or the power cord (AC) or power cable (DC) is faulty, contact your Cisco service representative for assistance.

Troubleshooting the Processor Subsystem

The Cisco 12008 processor subsystem consists of the GRP and the line cards.

The GRP and the line cards each have two onboard processors. One processor serves as the card's main processor, and the other serves as the MBus module processor. The latter monitors the card's environment and controls the card's onboard DC-DC converter.

The MBus module processor begins to operate as soon as power is applied to the system. The MBus processor determines what type of card it is mounted on and whether it should turn on the card's DC-DC converter. The master MBus module on the GRP turns on card power after a brief delay; the MBus modules on the line cards delay turning on power until they receive a command to do so from the GRP master MBus module.

To operate at all, a minimally configured Cisco 12008 must have a GRP. However, the system can operate without any line cards, as long as no line card is in partial contact with the backplane pins.

A line card that is only partially inserted into the backplane sends incomplete signals to the GRP, which can cause the system to hang. First ensure that the GRP is installed properly and that the system software has initialized successfully. Then, if necessary, you can troubleshoot the individual line cards.

A power-on self-test (POST) runs immediately at power-on to determine the condition of GRP memory. The results of this test appear in the alphanumeric display as a pass/fail message.

Troubleshooting the GRP

To isolate a problem with the GRP, assess the following:


Table  5-2:
GRP Alphanumeric Display Messages
Top Display Bottom Display Message
LMEM TEST Low memory test running
LCAH INIT Lower 15K cache initialization
BSS INIT Initialize main memory for ROM
NVRAM INIT Initialize NVRAM
EXPT INIT Initialize interrupt handlers
TLB INIT Initialize TLB
CACH INIT Initialize CPU data and instruction cache
CACH PARY Enable CPU cache parity
MEM INIT Initialize main memory
NVRAM SIZE Size the NVRAM
PCMC INIT Initialize the PCMCIA
EXIT INIT Exit the initialization sequence
IOS UP The Cisco IOS is up and running
Caution The GRP reset switch resets the GRP and the entire system. To prevent system errors and problems, use the GRP reset switch only at the direction of your Cisco-certified service representative.

Troubleshooting the Line Cards

Line cards can be installed in any slot in the upper card cage, except for the two slots in the middle of the card cage, which are reserved for the CSCs. Also, by convention, the left-most slot in the upper card cage is occupied by the GRP. However, you need not follow this recommendation.

As each line card powers up in response to a command issued by the GRP across the MBus, a power-on self-test (POST) is performed on line card memory. A full set of field diagnostics can also be run on a line card from the system console. The diagnostics provide a pass/fail message in the line card alphanumeric displays, as well as on the system console.

To isolate a problem with a line card, make the following assessment:


Table  5-3:
Line Card Alphanumeric Display Messages
Top Display Bottom Display Message
MEM TEST POST memory test running
LROM RUN After POST memory test
BSS INIT Initialize main memory for ROM
RST SAVE Save reset reason register
IO RST Reset the I/O system on the card
EXPT INIT Initialize interrupt handlers
TLB INIT Initialize TLB
CACH INIT Initialize CPU data and instruction cache
MEM INIT Initialize main memory
LROM RDY Ready to access download
ROMI GET Getting ROM images
FABL WAIT Wait for load of fabric downloader
FABL DNLD Loading fabric downloader
FABL STRT Launching fabric downloader
FABL RUN Fabric downloader launch complete
IOS DNLD Downloading the IOS
IOS STRT Launching the IOS
IOS UP IOS is running
IOS RUN Line card enabled

CSC Alarm Functions

The CSC incorporates the following system and component alarm functions:

A 25-pin D-sub connector on the CSC faceplate is connected directly to the critical, major, and minor alarm relays incorporated into the CSC.

Note Only safety extra-low voltage (SELV) external alarm circuits can be attached to the DB-25 connector on the CSC faceplate.
A manual reset switch is also incorporated into the CSC faceplate that you can push to reset (silence) an audible alarm. The visible alarm, however, can be reset only by the system's alarm monitoring software.

Check the CSC for an indication of a critical, major, or minor alarm. If any one of the three system alarm LEDs is on, check the system console for messages describing the fault condition.

Troubleshooting the Cooling Subsystem

The Cisco 12008 incorporates two fan trays (see Figure 1-2) that provide cooling air for other system components:

Each fan tray operates from +24 VDC that is distributed to it through the backplane from a DC-DC converter on the CSC. A recessed, blind-mating connector in the back of each fan tray provides connectivity to the backplane.

Both fan trays are essential; they provide sufficient cooling air for the router's components and electronic circuitry. The card cage fan tray incorporates six variable-speed fans; the power supply fan tray incorporates four variable-speed fans.

To isolate a problem with the router's cooling subsystem, make the following assessments:


Note To determine if the fan trays are operating, first listen for the hum of the fans. In a noisy environment, you might have to place your hand at the top rear of the router enclosure to feel the air being exhausted from the fan tray vents.

  1. Remove the air filter assembly from the chassis.

  2. Loosen the two captive installation screws securing the fan tray to the chassis frame.

  3. Grasp the fan tray carrier by its extraction/insertion ring and pull the unit part way out of the slot.

  4. Firmly reseat the fan tray in the slot.

  5. Tighten the fan tray captive installation screws.

  6. Reinstall the air filter assembly.
    If the left fan tray LED remains on after reseating the card cage fan tray, the unit is faulty and should be replaced.
    If the right fan tray LED on the CSC faceplate is on (amber), reseat the power supply fan tray in the backplane, as follows:

  7. Loosen the captive installation screw on the fan tray faceplate.

  8. Grasp the fan tray by the captive installation screw and partially withdraw the unit from the slot.

  9. Firmly reseat the fan tray in the slot.

  10. Tighten the fan tray captive installation screw.
    If the right fan tray LED remains on after reseating the power supply fan tray, the unit is faulty and should be replaced.
    On system startup, note that the fans operate at maximum speed until IOS is fully booted or if any fans fail or are absent.
    Fan speed is determined by temperature sensors on the cards in the upper and lower card cages and the power supplies. If an air temperature above the normal operating limit is detected within the router, the MBus module on the CSC increases the fan tray operating voltage. This higher voltage results in a corresponding increase in fan speed, forcing more cooling air through the entire router enclosure.
The preceding message could also indicate a faulty component or a faulty temperature sensor. Before the system shuts down, issue the show environment command or the show environment all command. Either command displays information about the internal system environment, including voltage measurements on each card for the +3.3 VDC and +5 VDC from the DC-DC converter, the +5.2 VDC for the MBus module, and the operating voltage for the fan trays. Each command also displays the temperature measurements made by two sensors on each card (one for inlet air temperature and one for the card's hot-spot temperature), as well as a temperature measurement made by a sensor in the power supply(ies).
If an environmental shutdown results from an out-of-tolerance power condition, the OUTPUT FAIL LED on the power supply goes on before the system shuts down. In this case, refer to the section entitled "Troubleshooting the Power Subsystem."

There are no field-replaceable components in the fan trays. If a fan tray is faulty, you must replace the entire unit.

If you are still unable to resolve a problem with the router's cooling subsystem, contact a Cisco service representative for assistance.

Additional Troubleshooting Reference Information

The following additional reference materials are available for troubleshooting your Cisco 12008 installation:

hometocprevnextglossaryfeedbacksearchhelp
Copyright 1989-1998 © Cisco Systems Inc.