Your Cisco 12012 went through extensive testing and burn-in before leaving the factory. However, if you encounter problems starting up the router, use the information in this chapter to help isolate the cause of the problems.
This chapter contains the following sections:
The procedures in this chapter assume that you are troubleshooting the initial Cisco 12012 system startup, and that the system is in the original factory configuration. If you have removed or replaced components or changed any default settings, the recommendations in this chapter might not apply.
At the initial system boot, you should verify the following:
- External power cables are connected, and proper source power is being supplied.
- The system blower modules are operating.
- The Cisco IOS software boots successfully.
- The GRP and line cards are properly installed in their slots, and each initializes (is enabled by the system software) without problems.
If you are unable to easily solve a problem, contact a service representative for assistance and further instructions. Before you call, have the following information ready to help your service provider assist you as quickly as possible:
- The date you received the router and the system serial number.
- The line cards you have installed. (Use the show hardware command to determine this.)
- The type of Cisco IOS software running and its release number. (Use the show version command to determine this.)
- A brief description of the problem you are having and the steps you have already taken to isolate and resolve the problem.
- Any maintenance agreement or warranty information.
This section describes the troubleshooting methods used in this chapter and describes how the Cisco 12012 is divided into subsystems for more efficient problem solving.
The key to solving problems in the system is to try to isolate the problem to a specific subsystem. The first step in solving startup problems is to compare what the system is doing to what it should be doing. Since a startup problem is usually attributable to a single component, it is more efficient to first isolate the problem to a subsystem rather than troubleshoot each component in the system.
For the troubleshooting procedures in this chapter, the Cisco 12012 consists of the following subsystems:
- Power subsystem--consists of the following components:
- AC-input power supplies or DC-input power supplies. The Cisco 12012 can be configured for source AC power or source DC power. A minimally configured Cisco 12012 system operating with source AC power has two AC-input power supplies installed. Adding a third and fourth AC-input power supply provides power supply redundancy and current sharing among the power supplies.
- A Cisco 12012 system operating with source DC power has one DC-input power supply installed. Adding a second DC-input power supply provides power supply redundancy and current sharing between the power supplies.
 | Caution Do not mix AC-input power supplies and DC-input power supplies in a Cisco 12012. |
- DC-DC converters. A DC-DC converter is installed on each card in the upper and lower card cages. The converter is under control of the MBus module, another component on each card. The DC-DC converter takes -48 VDC and converts it into +3.3 VDC and +5 VDC for use by the card circuitry.
- Harnesses. Two harnesses link the backplane with the two blower modules.
- Cooling subsystem--comprises the two blower modules, which includes the individual blowers, the blower speed control card, the faceplate LEDs, and the air filter. The two blower modules should be operating whenever the system power is on.
- The variable speed feature in the Cisco 12012 allows a blower to operate at a slower speed and provide quieter operation when the internal system temperature is within the normal operating range. If the internal temperature exceeds a specific temperature, the blower speed increases to move more cooling air through the system. As a result, it might be difficult to determine whether or not the blower is operating in noisy, air-conditioned rooms. If you determine that the blower is not operating, contact a service representative immediately; there are no installation adjustments that you can make.
- Processor subsystem--includes the Gigabit Route Processor (GRP) and all line cards. Each card has an onboard processor. The GRP downloads a copy of the Cisco IOS image to each line card processor. A line card or GRP that is partially installed in the backplane might cause the system to hang and crash. Two 4-character alphanumeric LED displays at the bottom of each line card or GRP faceplate display status and error messages, which can aid in troubleshooting.
Startup problems are commonly due to source power or to a card not properly seated in the backplane. Although an overtemperature condition is unlikely at initial startup, the environmental monitoring functions are included in this chapter because they also monitor internal voltages.
When you start up the router for the first time, you should observe the startup sequence. This section contains a description of the normal startup sequence as follows:
- Each card in the system has an MBus module and at least one DC-DC converter. Each MBus module controls the DC-DC converter. The MBus module receives +5 VDC directly from the power supplies through the backplane. When the power supply power switches are turned on, each MBus module boots from an onboard electrically erasable programmable read-only memory (EEPROM) device. Each MBus module processor reads a set of identification pins on the card to the backplane connector telling the MBus module processor what kind of card it is mounted on, which in turn, determines how the MBus module will function.
- The clock and scheduler card (CSC), containing the system clock, immediately powers up.
- The MBus module on the GRP monitors the progress of the clock and scheduler card power up. When the CSC has powered up, the MBus module on the GRP turns on its DC-DC converter powering up the GRP.
- The GRP sends the instructions to each line card to power up. Each line card processor begins to perform its own boot process. Each line card, through its MBus module, notifies the GRP when the boot process is complete.
- The GRP sends a command to each switch fabric card to power up. As each switch fabric card powers up, its progress is monitored by its MBus module processor. When the power-up process is complete, the switch fabric card MBus module notifies the GRP that the switch fabric card is online.
As the boot process progresses for each card, the card's status is displayed in the alphanumeric LED displays. The top display is powered by the DC-DC converter on the card; the bottom display is powered by the +5 VDC that powers the MBus module.
By checking the state of the LEDs on the power supplies and the alphanumeric displays on the GRP and line cards, you can determine when and where the system failed in the startup sequence.
When you start up the system by turning on the power supply power switches, the following should occur:
- The power supply green input OK (DC-input power supply) or AC OK (AC-input power supply) LED should go on immediately; it should remain on as long as the system is receiving source power and the power switch is in the ON (|) position. If this LED does not go on, or if it goes off while the power switch is still on, there could be a problem with either the source power, the internal DC voltages (+5 VDC and -48 VDC) that the power supplies distribute to the internal components, or the cooling subsystem.
- The green input OK LED indicates the status of the power supplies and internal DC voltages. This LED stays on when all of the following conditions are met:
- Power supplies are on and receiving 180 to 264 VAC, 50 to 60 Hz (or -48 VDC to -60 VDC) source power.
- Power supplies are providing the +5 and -48 VDC to internal components.
- All internal DC voltages are within tolerance.
- If the AC (or DC) source power or any of the internal DC voltages exceed allowable tolerances, the output OK LED will not go on, or will go off shortly after you turn on the power. Because both the GRP (which use +5 and +3.3 VDC), and the blower modules (which uses -48 VDC) are required for operation, a problem with any of the internal DC lines can prevent the system from starting up or continuing operation.
- For example, if there is a problem with the -48 VDC line that supplies the blower module, the system will start up, but will also recognize that the blower is not operating. The system will initiate a blower failure shutdown sequence, display the appropriate warning messages, then shut down after two minutes. If there is a problem with any of the other DC lines, the GRP will not be able to initialize the system software, so the system might attempt to start up and fail during the boot sequence.
- Depending on when the red output OK LED goes off, proceed as follows:
- If the input OK LED (or AC OK LED on an AC-input power supply) stays off (if it never goes on) when you turn on the power switch, there is a problem with either the DC (or AC) source power or the DC power that is distributed to the internal components. Proceed to the section "Troubleshooting the Power Subsystem."
- If the input OK LED (or AC OK LED on an AC-input power supply) goes on temporarily, then goes off within 30 seconds, the system is most likely shutting itself down because it detected an out-of-tolerance power or temperature condition within the power supplies. Proceed to the section "Troubleshooting the Power Subsystem."
- If the input OK LED (AC OK LED on an AC-input power supply) goes on, and the system starts up as expected but then displays the following message and shuts down after two minutes, there is a problem with one of the blower modules. Proceed to the section "Troubleshooting the Cooling Subsystem."
%ENVM-2-FAN: Fan has failed, shutdown in 2 minutes
- If the red output fail LED (common to both AC-input and DC-input power supplies) stays off, yet the system starts up correctly, displays the preceding message, and then shuts down after about two minutes, there is a problem with the -48 VDC line to the blower module. Proceed to the section "Troubleshooting the Power Subsystem."
- When the system is powered on, the blower modules immediately begin operating. The green fan OK LED should be lit on both blower modules.
- If the input OK LED is on, but a blower module is not operating, there is a problem with the blower. (The system will shut itself down if it detects that one or both of the blower modules is not functioning properly.) Proceed to the section "Troubleshooting the Cooling Subsystem."
- If the input OK LED is on at startup and a blower is operating, but the system shuts down after two minutes, there might be a problem with the blower control board. Proceed to the section "Troubleshooting the Cooling Subsystem."
- When the system is powered on, the alphanumeric LED displays on the GRP indicate the following:
- The top display indicates which GRP software component is running.
- The bottom display indicates the phase of the boot process that is currently occurring.
The power subsystem in the Cisco 12012 consists of the AC-input or DC-input power supplies, the MBus modules, the DC-DC converters, and the power distribution system. The power supplies provide +5 and -48 VDC output. The +5 VDC output from the power supply powers the MBus modules on each card in the system. The MBus modules, in turn, control the DC-DC converters also present on each card in the system. The DC-DC converter takes -48 VDC from the power supply and converts it into +5 and +3.3 VDC, which is distributed to the card circuitry.
Begin checking the power subsystem by first looking at the two LEDs on the power supply faceplate. The input OK LED on a DC-input power supply, or the AC OK LED on an AC-input power supply goes on when the power supply power switch is turned on (|) and the power supply is receiving source AC or DC power.
The red output fail LED is normally off, but goes on if the power supply detects a fault.
In systems with a single DC-input power supply, and in systems with redundant power when both power supplies are being shut down, the output fail LED lights momentarily as the system ramps down, but is off when the power supply has completely shut down.
The AC-input and DC-input power supplies are monitored by the MBus module and the GRP for internal temperature conditions and for overvoltage and overcurrent conditions.
Continue by checking the following to help isolate the problem with the power subsystem:
- Are the power supply input OK LEDs on?
- If yes, the power source is good, and the power supplies are functional.
- If no, but the blower modules are operating and the alphanumeric LED displays on the GRP and line cards are on, suspect a faulty power supply LED. The MBus modules driving the alphanumeric displays are powered by +5 VDC directly from the power supply, and the blower modules use -48 VDC; therefore, if both the GRP and the blower modules are operating, all internal DC voltages are within tolerance. Use the show environment command to check the voltages on each card.
- If no and there is not other obvious activity, first suspect that the power supply power switch is not fully in the ON (|) position. Rotate the power supply power switch clockwise and ensure that it is set completely in the ON (|) position.
- If the power supply power switch is set correctly and the input OK LED remains off, suspect the AC (or DC) source or the power cable. Turn the power supply power switch OFF (STANDBY on an AC-input power supply), and check the source AC (or DC) circuit breaker. Verify that the breaker is on and has not tripped. Verify that the source AC (or DC) circuit breaker has the proper current rating. Each power supply in the Cisco 12012 should be attached to a separate power source. In the case of systems powered by source AC, there might be an uninterruptable power supply (UPS) for each AC-input power supply in the system. Check that the UPS is functioning correctly.
- Check the power cable or power cord from the power source to the Cisco 12012. Verify that it is in good shape and is not damaged. If the insulation appears cracked or broken, or the plugs appear loose, do not use the cable. Immediately replace it with a new cord or cable.
- If the LED fails to go on after you connect the power supply to a new power source, exchange the power cable with a replacement, if one is available, and turn the power switch back on. If the AC (or input) OK LED then goes on, return the first power cable for replacement.
- If the LED still fails to go on when connected to a different power source with a new power cable, the power supply is probably faulty. If a spare power supply is available, replace the existing power supply with the spare and restart the system.
- If the Input OK LED then goes on, return the faulty power supply for replacement.
- If you are unable to resolve the problem or if you determine that either the power supply or power cable is faulty, contact a service representative for assistance.
The Cisco 12012 processor subsystem consists of the GRP, the line cards, and the alarm card. The GRP and the line cards each have two processors: one processor is the card's main processor; and the other processor is a component in the MBus module. The MBus module begins operation as soon as power is applied to the system. The MBus module determines the type of card it is mounted on and whether it should turn on the DC-DC converter. The GRP MBus module turns on card power after a brief delay; the line card MBus modules delay turning on power until they receive a command from the GRP.
A Cisco 12012 requires one GRP be installed. The system cannot operate unless the GRP is installed properly; however, the system can operate without any line cards installed as long as none are in partial contact with the backplane pins. A line card that is partially connected to the backplane will send incomplete signals to the GRP, which could cause the system to hang. Therefore, first ensure that the GRP is installed properly and the system software has initialized successfully. Then, if necessary, you can troubleshoot individual line cards.
A power-on self-test (POST) runs immediately at power-on to determine the condition of the GRP memory. Results are displayed in the alphanumeric LED display as a pass/fail message.
Check the following to help isolate a problem with the GRP:
- Are both the alphanumeric LED displays on?
- The two displays are powered separately. The top display receives power from the DC-DC converter on the GRP. The bottom display is powered directly from the power supply; therefore, even if the GRP has not powered up, its bottom display could be on. If both displays are off, the GRP may not be fully plugged into the backplane connector, there might be a problem with the MBus module on the GRP, or the system power supply might be off.
- If both displays are on, check the message being displayed. As soon as the DC-DC converter is turned on by the MBus module, the processor on the GRP begins the boot process. Status messages are displayed as the boot process continues. (See Table 5-1, which provides a list of messages that can be displayed by the GRP alphanumeric LED display.) If one of the messages appears frozen, the boot process could be halted. Note the message being displayed on a piece of paper. Turn off the system power supply power switches, then turn them back on to reset the system and start the boot process again. If the system halts again, the GRP could be faulty and might need to be replaced.
Table 5-1: GRP Alphanumeric LED Display Messages
| LED Display1
| Indications1
|
|---|
LMEM
TEST
| Low memory test running
|
LCAH
INIT
| Lower 15k cache initialization
|
BSS
INIT
| Initialize main memory for ROM
|
NVRAM
INIT
| Initialize NVRAM
|
EXPT
INIT
| Initialize interrupt handlers
|
TLB
INIT
| Initialize TLB
|
CACH
INIT
| Initialize CPU data and instruction cache
|
CACH
PARY
| Enable CPU cache parity
|
MEM
INIT
| Initialize main memory
|
NVRAM
SIZE
| Size of the NVRAM
|
PCMC
INIT
| Initialize the PCMCIA
|
EXIT
INIT
| Exit the initialization sequence
|
IOS
UP
| The Cisco IOS software is up and running
|
MSTR
RP
| The GRP is enabled and recognized by the system
|
1
Some messages appear briefly (millisecond duration); others last several seconds.
- If the power supplies and blowers appear operational but none of the GRP LEDs or displays are on, suspect that the GRP has not been properly installed or that the +5 VDC output from the power supplies is faulty. Turn the power supply power switch OFF (STANDBY on an AC-input power supply) on each power supply, loosen the two captive screws on the top and bottom of the GRP faceplate, and use the ejector levers to eject and reseat the GRP. Tighten the captive screws, then power up the system by turning the power supply power switches to ON (|).
- Is a critical, major, or minor alarm LED on the alarm card on?
- If any of the three alarm card LEDs is on, a fault has been detected in the system. Check the console for messages indicating the source of the problem.
- There could be a false error indication originating from the GRP. You might want to reseat or replace the GRP.
| Caution The GRP reset switch resets the GRP and the entire system. To prevent system errors and problems, use it only at the direction of a Cisco-certified service representative. |
Line cards can be installed in slots 0 through 11 in the upper card cage. As each line card powers up, a power-on self-test (POST) is performed on the line card memory. A full set of field diagnostics can also be run on a line card from the system console providing a pass/fail message both in the line card alphanumeric LED display and on the system console. (For information on diagnostic testing, refer to the chapter "Running Diagnostics on the Cisco 12012.")
Check the following to help isolate a problem with the line cards:
- Are both the alphanumeric LED displays on?
- The two displays are powered separately. The top display receives power from the DC-DC converter on the line card. The bottom display is powered directly from the power supply. So, even if the line card has not powered up, the bottom display could be on. If both displays are off, the line card might not be fully plugged into the backplane connector, there might be a problem with the MBus module on the line card, or the system power supply may be off.
- If both displays are on, check the message being displayed. As soon as the DC-DC converter is turned on by the MBus module, the processor on the line card begins the boot process. Status messages are displayed in the alphanumeric displays as the boot process continues on the line card.
Table 5-2 provides a list of messages that can be displayed by the line card alphanumeric LED display. Some of these messages are displayed only for a few milliseconds; others can last for several seconds.
Table 5-2: Line Card Alphanumeric LED Display Messages
| LED Display2
| Indications1
|
|---|
MEM
TEST
| POST memory test running
|
LROM
RUN
| After POST memory test
|
BSS
INIT
| Initialize main memory for ROM
|
RST
SAVE
| Save reset reason register
|
IO
RST
| Reset the I/O system on the card
|
EXPT
INIT
| Initialize interrupt handlers
|
TLB
INIT
| Initialize TLB
|
CACH
INIT
| Initialize CPU data and instruction cache
|
MEM
INIT
| Initialize main memory
|
LROM
RDY
| Ready to access download
|
ROMI
GET
| Getting ROM images
|
FABL
WAIT
| Wait for load of fabric downloader
|
FABL
DNLD
| The fabric downloader loads
|
FABL
STRT
| The fabric downloader launches
|
FABL
RUN
| The fabric downloader launch is complete
|
IOS
DNLD
| The Cisco IOS software downloads
|
IOS
STRT
| The Cisco IOS software launches
|
IOS
UP
| The Cisco IOS software runs in DRAM
|
IOS
RUN
| The line card is enabled and ready for use
|
1
Some messages appear briefly (millisecond duration); others last several seconds.
The alarm card is installed in the rightmost slot in the upper card cage. The slot is labeled Alarm card and is slightly narrower than the rest of the slots in the upper card cage. In addition, the alarm card slot backplane connector is different from the rest of the backplane connectors in the upper card cage.
The alarm card has three primary functions:
- Provides a visual display of the three stages of system alarm (critical, major, and minor) detected by the system's environmental monitor. Three pairs of LEDs (two pair of LEDs are red, one pair of LEDs is amber) on the alarm card faceplate are driven by the environmental monitor software in the system.
- When the environmental software detects an abnormal condition (such as overtemperature or overvoltage), the software determines the severity of the condition and sets the appropriate pair of alarm card LEDs. The alarm card LEDs are paired for redundancy.
- Provides a connection point for the system to connect to two site-wide external audio/visual alarm systems. Two redundant 25-pin D-sub connectors on the alarm card faceplate are tied directly to the critical, major, and minor alarm relays on the alarm card. (Only safety extra-low voltage [SELV] external alarm circuits can be attached to the two external alarm connectors.)
- A switch on the alarm card faceplate can reset an audio alarm that signals that a critical, major, or minor error has occurred. The visual alarm must be reset by the environmental monitor software.
- Provides a visual status of the cards in the lower card cage through five pairs of LEDs (one pair for each slot in the lower card cage).
- A green enabled LED indicates the card in the slot has been detected by the system and is OK.
- A yellow fail LED indicates a fault in the card in that lower card cage slot has been detected by the system.
Note Check the alarm card for critical, major, or minor error LEDs that are on. If any of the six LEDs are on, check the system console for messages describing the fault.
The Cisco 12012 has two blower modules that provide cooling air for the other system components. The top blower module is mounted in the frame above the upper card cage. The bottom blower module is mounted in the frame below the power supply bay. Each blower module receives power and signals though harnesses attached to the frame. A connector recessed in the back of the blower module mates with a connector mounted on the frame when the blower module is installed in the frame.
Both blower modules are needed to provide sufficient cooling air through the system. Each blower module contains three variable speed fans and a controller card. There are two LEDs on the blower module faceplate visible through the blower module front cover.
The green LED, when lit, indicates that the blower module is functioning. Check the following to help isolate a problem with the cooling system:
- When you start up the system, do the blower modules go on?
Note To determine if a blower module is operating, visually check the two LEDs. Under normal operation the green fans OK LED should be on. Listen for the blower fans. In noisy environments, place your hand at the back of the frame, behind the blower modules, both top and bottom, to feel for air being forced out the exhaust vents.
- If the blower modules come on, the -48 VDC line from the power supply to the blower is good.
- If one or both of the blower modules do not come on, there could a problem with either the blower module or the -48 VDC power. If both blower modules do not come on, it is likely there is problem with the -48 VDC output from the power supply. Check the red output fail LED on each power supply. If the output fail LED on a power supply is on, the power supply is faulty and should be replaced.
- If one or both of the blower modules do not come on and the power supply output fail LED is off (-48 VDC is OK), ensure that the blower module is seated properly in the frame. Remove the blower module snap-on front cover, loosen the two captive screws on the blower module faceplate, grasp the blower module handle and pull the blower module out and then firmly push to reseat the blower module in the frame. Reinstall the blower module front cover.
- If the blower module does not come on, there could be a problem with the blower module controller card. The controller card controls the speed of each fan. Fan speed is dependent on the air temperature detected on the cards in the upper and lower card cages and the power supplies in the power supply bay. Changes in air temperature in one of those locations changes the fan speed which forces less or more cooling air through both card cages and the power supply bay.
- The following message indicates that the system has detected an overtemperature condition or out-of-tolerance power inside the system:
Queued messages:
%ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdown
- If an environmental shutdown results from an out-of-tolerance power condition, the output fail LED on the power supply will go on before the system shuts down. Refer to the section "Troubleshooting the Power Subsystem."
- Although an overtemperature condition is unlikely at initial startup, ensure that heated exhaust air from other equipment is not entering the air filter, and that there is sufficient clearance (at least 12-inches, 30.5 cm) around the front and rear of the chassis to allow cooling air to enter and hot air to exhaust.
- Check the condition of the air filter located in a hinged tray in front of the lower card cage. If the air filter appears dirty, you should remove the filter and either vacuum it or replace it.
- The preceding message could also indicate a faulty component or temperature sensor. Before the system shuts down, use the show environment all or show environment table commands to display the internal system environment including voltages and temperatures measured at each card.
There are no field replaceable components in the blower module. If the blower module is faulty, you must replace the entire blower module.
If you are still unable to resolve the problem, contact a service representative for assistance.
This section provides a list of additional Cisco reference material for troubleshooting your Cisco 12012 installation:
