Troubleshooting Hardware and Booting Problems

• Netboot---Routers can boot from a server using the Trivial File Transfer Protocol (TFTP), the DEC Maintenance Operation Protocol (MOP), or the Remote Copy Protocol (RCP) across any of the supported media types (such as Ethernet, Token Ring, Fiber Distributed Data Interface [FDDI], High-Speed Serial Interface [HSSI], and serial lines).

• Flash memory---Routers can boot from Flash memory, a nonvolatile storage medium that can be electrically erased and reprogrammed.

• ROM---Routers can boot a system from built-in read-only memory (ROM).

• PC Flash memory card---Routers can boot from a removable Flash memory card.

This section provides general information about router booting.

1. Boot an image from Flash memory.

2. Boot an image using a netboot.

3. Boot from a ROM image.

goriot# configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
goriot(config)# boot system flash gsxx
goriot(config)# boot system gsxx 131.108.1.101
goriot(config)# boot system rom
goriot(config)# ^Z
goriot#
%SYS-5-CONFIG_I: Configured from console by console
goriot# copy running-config startup-config
[ok]
goriot#

Using this strategy, a router has three sources from which to boot: Flash memory, netboot, and ROM. Providing alternative sources can help to mitigate any failure of the TFTP server or the network.

The following examples show console output from netbooting sessions that were successful even though timeouts and out-of-order packets occurred (exclamation points represent successfully received packets):

Booting gs3-bfx from 131.108.1.123: !.!!!!!!!!!!!!!!!!!!!!!!
 
Booting gs3-bfx from 131.108.1.123: !O.O!!!!!!!!!!!!!!!!!!!!!!

• NVRAM configurations for client and adjacent routers.

• Debugging output from adjacent routers using the following privileged exec commands:

  • debug ip packet

  • debug arp

  • debug ip udp

  • debug tftp

For more information about these debug commands, refer to the Debug Command Reference.

• The AC (or DC) OK LED should go on immediately and should remain on as long as the system is receiving power.

• The blower should be operating.

• The Route Switch Processor (RSP) and front-panel Normal LEDs should go on (to indicate normal system operation) and should remain on during system operation; the CPU Halt LED should remain off.

• The Enabled LED on each interface processor should go on (to indicate that the RSP has completed initialization of the interface processor).

When the 7500 series system has initialized successfully, the system banner should be displayed on the console screen. If it is not displayed, make sure that the console terminal is properly connected to the RSP console port and that the terminal is set correctly. The system banner should look similar to the following:

System Bootstrap, Version 4.6(5), SOFTWARE
Copyright (c) 1986-1995 by cisco Systems
RSP2 processor with 16384 Kbytes of memory
### [...] ###
F3: 2012356+47852+194864 at 0x1000
              Restricted Rights Legend
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
              cisco Systems, Inc.
              170 Tasman Drive
              San Jose, CA 95134
GS Software (RSP-K), Version 10.3(571) [fc3], RELEASE SOFTWARE
Copyright (c) 1986-1995 by cisco Systems, Inc.
[...]
Press RETURN to get started!

If a problem occurs, try to isolate the problem to a specific subsystem. The Cisco 7500 series routers have the following subsystems:

Possible Problem Area	Solution
Power subsystem	Step 1 Check to see whether the blower is operating and LEDs on the processor modules are on. If the blower and LEDs are on but the Power Supply LED is off, there is probably a faulty Power Supply LED. Step 2 Make sure the power switch is set correctly to the on position. Step 3 Make sure the power source, power cable, and power supply are functioning correctly. Swap parts to see whether one of the components is faulty. Step 4 Ensure that the blower module is seated properly. Make sure that the blower control board edge connector is inserted fully in the backplane socket.
Cooling subsystem	Step 1 Check to see whether the blower is operating when you start up the system. If the blower is not operating, there might be a problem with the blower or the +24 V DC power: If the Output Fail LED is on, there might be a problem with the +24V DC supply to the blower or fan tray at either the power supply or the blower control board. If the blower is not operating and the Output Fail LED is off, ensure that the blower module is seated properly. Ensure that the blower control board edge connector is inserted fully in the backplane socket. Step 2 If the system and blower start up but shut down after about two minutes, one or more fans might have failed or might be operating out of tolerance. You will probably see an error message similar to the following: %ENVM-2-FAN: Fan has failed, shutdown in 2 minutes If the blower or the blower control board fails, you must replace the blower module. Step 3 If you see the following message at startup, the system has detected an overtemperature condition or out-of-tolerance power inside the chassis: Queued messages: %ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdown If an environmental shutdown results from an out-of-tolerance power condition, the Output Fail LED goes on before the system shuts down. This shutdown message might also indicate a faulty component or temperature sensor. Before the system shuts down, use the show environment or show environment table commands to display the internal chassis environment. Step 4 Ensure that heated exhaust air from other equipment is not entering the inlet vents, and that there is sufficient clearance around the chassis to allow cooling air to flow.
Processor subsystem	Step 1 Check the RSP1 LEDs. If no LEDs come on, ensure that the power supplies and blower are functioning properly. Step 2 Check the seating of the RSP. If the RSP is not seated properly, it will hang the system. Step 3 If the RSP CPU Halt LED is on, the system has detected a processor hardware failure. Contact a technical support representative for instructions. Step 4 Check to see whether the RSP Normal LED is on, indicating that the system software has initialized successfully and the system is operational. Step 5 Check the Enabled LED on each interface processor. This LED should go on when the RSP has initialized the interface processor. Step 6 If the Enabled LED on an individual interface processor is off, the interface processor might have pulled away from the backplane. If the interface processors are not seated properly, they will hang the system.

1RSP = Route Switch Processor

• The DC OK LED should go on and should remain on as long as the system is receiving source power.

• The fans should be operating.

• The Route Processor (RP) Normal LED should go on and stay on to indicate normal system operation; the Halt CPU LED should remain off.

• The Enabled LED on the Switch Processor (SP) or Silicon Switch Processor (SSP) and each interface processor should go on when the RP has completed initialization of the interface processor or SP (or SSP) for operation.

• Power subsystem---Includes power supplies, fans, external power cable, and internal power harness that connects to the backplane

• Cooling subsystem---Depending on your system, the cooling subsystem includes the -following:

  • Cisco 7000: Chassis blower

  • Cisco 7010: Fan tray assembly, including six individual fans, the fan control board, and the tray itself

• Processor subsystem---Includes the RP, SP (or SSP), and all interface processors

Possible Problem Area	Solution
Power subsystem	Step 1 Check to see whether the DC OK LED is on. Step 2 If the LED is not on but the fans are operating and LEDs on the processor modules are on, the Power Supply LED might be faulty. Step 3 If the LED is not on and there is no other activity, make sure the power switch is fully in the on position. Step 4 Make sure the power source, power cable, and power supply are functioning correctly. Swap parts to see whether one of the components is faulty. Step 5 Ensure that the fan tray is seated properly. Make sure the fan control board edge connector is inserted fully in the backplane socket.
Cooling subsystem	Step 1 Check to see whether the fans are operating. Step 2 If the fans are not operating and the DC OK LED is off, there might be a problem with the +24V DC power. Step 3 Ensure that the fan tray is seated properly. Make sure that the fan control board edge connector is inserted fully in the backplane socket. Step 4 If the system and the fans start up but shut down after about two minutes, one or more fans has failed or is operating out of tolerance. You will see an error message similar to the following: %ENVM-2-FAN: Fan array has failed, shutdown in 2 minutes If one or more fans or the fan control board fails, you must replace the fan tray. Step 5 If you see the following error message, the system has detected an overtemperature condition or out-of-tolerance power inside the chassis: Queued messages: %ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdown If an environmental shutdown results from an out-of-tolerance power condition, the DC OK LED will go off before the system shuts down. This shutdown message could also indicate a faulty component or temperature sensor. Use the show environment or show environment table command to display the internal chassis environment. Step 6 Make sure that heated exhaust air from other equipment is not entering the inlet vents, and that there is sufficient clearance around the chassis to allow cooling air to flow.
Processor subsystem	Step 1 Check to see whether the RP1 LEDs come on when system power is turned on. Step 2 If none of the RP LEDs come on, make sure that both the fan and power supply are functioning properly. Step 3 If the power supply and fans appear operational but none of the RP LEDs are on, an improperly connected RP, SP2 (or SSP3), or interface processor might have hung the bus. Step 4 If the SP (or SSP) Enabled LED is off but any of the RP LEDs are on, make sure the SP (or SSP) is seated in its slot properly. Step 5 Check to see whether the Boot Error LED is on. If the LED is on, the system software is unable to start up. If you have a spare RP with the system software ROMs installed, replace the installed RP with the spare to see whether the system will boot. Step 6 Check to see whether the RP CPU Halt LED is on. If it is, the system has detected a processor hardware failure. Contact a technical support representative for more information. Step 7 Check to see whether all interface processor Enabled LEDs are on. Step 8 If the Enabled LED on an individual interface processor is off, make sure that the interface processor has not pulled away from the backplane.

1RP = Route Processor 2SP = Switch Processor 3SSP = Silicon Switch Processor

• The System OK LED should come on and stay on as long as power is supplied.

• The fans should be operating.

• Power subsystem---This subsystem includes the power supply and the wiring.

• Cooling subsystem---This subsystem includes the blower assembly, which should come on when power is applied.

• Network processor modules (NPMs)---This subsystem includes all NPMs installed in the router chassis.

• System cables---This subsystem includes all the external cables that connect the router to the network.

Possible Problem Area	Solution
Power and cooling subsystems	Step 1 Check to see whether the blower is operating. If it is not, check the AC power input, AC power source, router circuit breaker, and power supply cable. Step 2 If the system shuts down after being on a short time, check the power supply. If the power supply appears operational, the router might have shut down due to overheating. Check the console for error messages similar to the following: %SYS-1-OVERTEMP: System detected OVERTEMPERATURE condition. Please resolve cooling problem immediately! Make sure that the fans are working and that there is no air blockage to cooling vents. Step 3 If the system partially boots but LEDs do not light, contact your technical support representative.
NPMs1 and cables	Step 1 Make sure that NPMs are properly connected to the motherboard connector. Step 2 Check the external cables. Step 3 Check the processor or software for proper configuration. Step 4 Check the external console connection and verify that the console baud rate is correct.

1NPM = network processor module

• The System OK LED should come on and stay on as long as power is supplied.

• The fans should be operating.

When the system has initialized successfully, the system banner should be displayed on the console screen. The system banner should look similar to the following:

• Power subsystem---This subsystem includes the power supply and the wiring.

• Cooling subsystem---This subsystem includes the fan, which should go on when power is applied.

• Network interfaces---This subsystem includes all network interfaces, such as Ethernet, Token Ring, serial, or ISDN Basic Rate Interface (BRI).

• System cables---This subsystem includes all the external cables that connect the router to the network.

Possible Problem Area	Solution
Power and cooling subsystems	Step 1 If the Power LED is off, make sure the power supply is plugged in to the wall receptacle and that the cable from the power supply to the router is connected. Step 2 If the system shuts down after being on a short time, there might have been a thermal-induced shutdown caused by a faulty fan, or the power to the system might have been lost. Ensure that the system is receiving power and that the chassis intake and exhaust vents are clear. Step 3 If the system does not boot up but LEDs are on, check the 12V power supply. Step 4 If the system partially boots but LEDs are not on, check the 5V power supply.
Network interfaces and cables	Step 1 If a network interface is not recognized by the system, check the interface cable connection and the LED on the network interface. Step 2 If a network interface is recognized but will not initialize, check the interface cable connection. Step 3 If the system will not boot properly or constantly or intermittently reboots, there might be a processor or software problem. Make sure that DRAM SIMM modules are seated properly. Step 4 If the system boots but the console screen is frozen, check the external console connection and verify that the console baud rate is correct. Step 5 If the system powers on and boots with a particular interface disconnected, check the network interface connection.

• The OK LED should come on and stay on as long as power is supplied.

• The fans should be operating.

When the system has initialized successfully, the system banner should be displayed on the console screen.

Possible Problem

Solution

Bootup problem

Step 1 Check to see whether the fan is operating. If it is not, check the fan or the 12V power supply. Step 2 If the system shuts down after being on for a short time, check the power supply. Step 3 If the power supply appears operational, the router might have shut down due to overheating. Ensure that the chassis intake and exhaust vents are clear. Step 4 If the system does not boot up but the System OK LED is on, check the 12V power supply to make sure it is not faulty. Step 5 If the system partially boots but the System OK LED is not on, check the 5V power supply to make sure it is not faulty.

• The PS1 and PS2 LEDs on the supervisor engine module faceplate should be green.

• The system fan assembly should be operating and the Fan LED on the supervisor engine module should come on.

• The Status LED on the supervisor engine module and all interfaces should be orange until the boot is complete.

When the system boot is complete, the supervisor engine module should initialize the switching modules. The status LED on each switching module goes on when initialization has been completed, and the console screen displays a script and system banner similar to the following:

        ATE0
ATS0=1
Catalyst 5000 Power Up Diagnostics
Init NVRAM Log 
LED Test 
ROM CHKSUM 
DUAL PORT RAM r/w 
RAM r/w 
RAM address test 
Byte/Word Enable test 
RAM r/w 55aa 
RAM r/w aa55 
EARL test 
BOOTROM Version 1.4, Dated Dec  5 1995 16:49:40
BOOT date: 00/00/00 BOOT time: 03:18:57
SIMM RAM address test 
SIMM Ram r/w 55aa 
SIMM Ram r/w aa55 
Start to Uncompress Image ...
IP address for Catalyst not configured
BOOTP will commence after the ports are online
Ports are coming online ...
Cisco Systems Console 

• Power subsystem---This subsystem includes the power supplies and power supply fans.

• Cooling subsystem---This subsystem includes the chassis fan assembly, which should be operating when the system power is on.

• Processor and interface subsystem---This subsystem includes the supervisor engine module (which contains the system operating software), the network interfaces, and all associated cabling.

Possible Problem Area	Solution
Power subsystem	Step 1 Check to see whether the PS1 LED is on. If it is not, ensure that the power supply is connected properly and is flush with the back of the chassis. Make sure that captive installation screws are tight. Step 2 Check the AC source and the power cable. Connect the power cord to another power source if one is available and turn the power back on. If the LED fails to go on after you connect the power supply to a new power source, replace the power cord. Step 3 If the LED fails to go on when the switch is connected to a different power source with a new power cord, the power supply is probably faulty. If a second power supply is available, install it in the second power supply bay and contact a customer service representative for further instructions. Step 4 Repeat these steps for the second power supply if present.
Cooling subsystem	Step 1 Check to see whether the Fan LED on the supervisor engine module is green. If it is not, check the power subsystem to see whether it is operational. Step 2 If the Fan LED is red, the fan assembly might not be seated properly in the backplane. To ensure that the fan assembly is seated properly, loosen the captive installation screws, remove the fan assembly, and reinstall it. Tighten all captive installation screws and restart the system. Step 3 If the Fan LED is still red, the system has probably detected a fan assembly failure. Contact a technical support representative for assistance.
Processor and interface subsystem	Step 1 Check the supervisor engine module Status and Link LEDs. These should both be green if all diagnostic and self-tests were successful and ports are operational. For more information about interpreting the supervisor engine module LEDs, refer to the user guide for your switch. Step 2 Check the LEDs on individual interface modules. In most cases these should be green (or should flicker green, in the case of Transmit and Receive LEDs) if the interface is functioning correctly. For detailed information on interpreting interface module LEDs, refer to the user guide for your switch. Step 3 Check all cabling and connections. Replace any faulty cabling.

• The Power LED should come on.

• The fan should begin operating and should stay on while power is applied to the system.

• On some models, the DIAG LED should come on, stay on for the duration of the system's self-test diagnostics, and then turn off.

While booting, the console screen displays a script and system banner, which should be similar to the following:

• Power subsystem---This subsystem includes the input power, AC power cable, and power supply.

• Cooling subsystem---This subsystem includes the fans, which should be operating when the system power is on.

• Network interfaces and system cables subsystem---This subsystem includes all the network interfaces and the cables that connect the equipment to the network.

Possible Problem Area	Solution
Power and cooling subsystems	Step 1 Check to see whether the Power LED is on. If it is not on and the fans are not running, check the AC power cord and the AC receptacle the cord is plugged in to. Make sure the cord is intact and properly attached. Swap the cord with another cord to see whether the cord is faulty. Make sure the receptacle is receiving power. Step 2 If the Fault LED comes on after power up or after the unit has been running for a period of time, cycle the power to the unit. Step 3 Check to see whether the fans are running and that the chassis intake vents and exhaust ports are clear. Step 4 View the console to check whether a temperature error has been reported. If you continue to experience power problems, or if temperature errors are being reported to the console and you cannot isolate a specific cause and correct it, contact the Cisco TAC.1
Network interfaces and system cables subsystem	Step 1 If a network interface is not recognized by the system, check the interface cable and connection and the LED that corresponds to the network interface. Step 2 If a network interface is recognized but will not communicate properly, check the interface cable and connections. Step 3 If the system will not boot properly or intermittently reboots, the processor may be faulty or the hardware or software setup may be incorrect. Step 4 If the system boots but the console is frozen, check the external console connection and verify the console setup.

1TAC = Technical Assistance Center

• The PS LED on the supervisor engine module faceplate should come on and stay green while power is applied to the system.

• The system fan assembly and Fan LED should come on and stay on while power is applied to the system.

• The Status LED on the supervisor engine module and on each interface should be orange until the boot is complete.

When the system boot is complete, the supervisor engine module initializes the switching modules. The status LED on each switching module goes on when initialization has been completed, and the console screen displays a script and system banner similar to the following:

BOOTROM Version 2.1, Dated May 22 1996 15:17:09
 
Boot date: 05/22/96 BOOT time: 15:17:09
 
Executing from RAM
 
 
 
Cisco Systems Console
 
 
 
Sending RARP request with address 00:40:0b:a0:05:b8
 
Sending bootp request with address 00:40:0b:a0:05:b8
 
Sending RARP request with address 00:40:0b:a0:05:b8
 
Sending bootp request with address 00:40:0b:a0:05:b8
 
No bootp or rarp response received
 
Enter password:

If problems occur, try to isolate the problem to a specific subsystem. The Catalyst 2900 series LAN switches have the following subsystems:

• Power subsystem---This subsystem includes the power supplies and power supply fans.

• Cooling subsystem---This subsystem includes the chassis fan assembly, which should be operating when the system power is on.

• Processor and interface subsystem---This subsystem includes the supervisor engine module (which contains the system operating software), the network interfaces, and all associated cabling.

Possible Problem Area	Solution
Power subsystem	Step 1 Check the Power LED. If it is off, ensure that the power supply cord is not damaged and that it is properly attached to the power supply and to an AC receptacle. Step 2 If the LED is red, the power supply has detected an anomaly or voltage outage and needs to be serviced. Contact your technical support representative for instructions.
Cooling subsystem	Step 1 Check to see whether the Fan LED on the supervisor engine module is green. If it is not, check the power subsystem to see whether it is operational. Step 2 If the Fan LED is red, contact a technical support representative for assistance.
Series processor and interface subsystem	Step 1 Check the supervisor engine module Status and Link LEDs. These should both be green if all diagnostic and self-tests were successful and ports are operational. For more information about interpreting the supervisor engine module LEDs, refer to the user guide for your switch. Step 2 Check the LEDs on individual interface modules. In most cases these should be green (or should flicker green, in the case of transmit and receive LEDs) if the interface is functioning correctly. For detailed information on interpreting interface module LEDs, refer to the user guide for your switch. Step 3 Check all cabling and connections. Replace any faulty cabling.

Possible Problem	Symptom
Low-level test failure	The startup process is halted.
High-level test failure	The self-test program restarts and the number of the failed test is displayed. High-level tests are repeated until the self-test is completed successfully.
Hardware error	Contact your technical support representative. Be sure to note the number of the failed test and the version number of the software containing the self-test program.

Possible Problem	Solution
System initialization problem	Step 1 Make sure power cords and data cables are firmly connected at both ends. Step 2 Make sure all cards (front and back of the chassis) are firmly seated in the midplane and screwed securely to the chassis. Step 3 Make sure power supplies, blowers, and disk drives are properly connected and screwed securely to the chassis.
Blower problem	If you are experiencing any of the following symptoms, you might need to replace the blower: The temperature on one or more cards is out of the recommended range. The system is powered on, but the blower is not turning, is making noise, or is exhausting air. Two minutes after the system is powered on, the blower fails to reduce its speed in a room temperature environment. The system is powered on, but the blower's green LED is off. The LED indicates that the blower is turning at a rate of at least 1500 rotations per minute.
Bulk power tray problems	In a system with one power tray, no power will be present if the power tray is faulty. There might be a problem with the power tray if cycling the system's power has no effect. A system with two power trays can operate normally when only one is working. If you suspect a problem, use the CLI1 command show chassis powersupply. If a status line for an occupied slot says anything other than Good, check the faulty power tray to see that it is properly connected. Replace the power tray if necessary.
Switch card problems	If you are experiencing any of the following symptoms, you might need to replace the switch card: POST2 fails (the FLT3 LED stays lit and POST results indicate a problem). The Switch card fails even when moved to the other slot. (If the card fails in one slot but operates properly in another, suspect a problem with the midplane.) Diagnostics that loop data through the switch card fail on two or more function cards. The Switch card fails to come up or to select a TCS4 hub. Traffic is not passing through the system, but the line cards and NPs5 are operational. The system has data transmission problems that do not go away when you replace the card that appears to be failing or that occur in several cards simultaneously. (Problems of this type may also indicate a faulty midplane.) The switch card cannot be fully inserted into its slot. This probably indicates damage to the connectors on either the card or the midplane. Inspect all the connectors and replace the card or the midplane if you find damage.
Network processor problems	If the NP fails to power up, check its access card at the back of the chassis. An NP requires an NPAC6; it cannot operate with any other kind of access card. If the system fails to boot, it could indicate either a problem with the NP, a problem with the NP's hard disk drive, or a problem with the software on the hard drive. If you are experiencing any of the following symptoms, you might need to replace the NP card: POST fails (the FLT LED stays lit and the POST results indicate a problem). The NP fails even when moved to the other slot. (If the card fails in one slot but operates properly in the other, suspect a problem with the midplane.) Hardware diagnostics fail. You cannot get to the CLI to run the diagnostics. The card fails to load. The NP or its access card cannot be fully inserted into its slot. This probably indicates damage to the connectors on either the card or the midplane. Inspect all the connectors and replace the card or the midplane if you find damage.
Interface module problems	The following tips will help you distinguish between problems in a line card and problems in an access card: Run the manufacturing diagnostics and check the information provided for the access card. Swap another line card of the same type. If the second card has the same problem as the first one, the access card is probably at fault. If the second card works properly, the first line card is likely to be the source of the problem. Faults in the line card are more common than faults in the access card. If you cannot determine which card is causing a problem, try replacing the line card. Use the looping tests described in the LightStream 2020 Network Operations Guide. If you are having trouble bringing up an interface module, check the following: Make sure the access card behind the line card is compatible with the line card. The low-speed line card (LSC) is compatible with the following access card: Low-speed access card (LSAC) The medium-speed line card (MSC) is compatible with the following access cards: T3 medium speed access card (T3 MSAC) E3 G.804 medium speed access card (E3 G.804 MSAC) E3 PLCP medium speed access card (E3 PLCP MSAC) The cell line card (CLC) is compatible with the following access cards: OC-3c single mode access card (OC3AC SM)---1 or 2 ports OC-3c multimode access card (OC3AC MM)---1 or 2 ports T3 access card (T3AC)---4 or 8 ports E3 access card (E3AC)---4 or 8 ports
Interface module problems	The packet line card is compatible with the following access cards: FDDI access card Ethernet access card Fiber Ethernet access card T1 circuit emulation access card E1 circuit emulation access card The NP is compatible with the following access card: NPAC If an FDDI7 module does not pass traffic, make sure the FDDI cables for each port are attached to the proper connectors. If you are bringing up a low-speed module, make sure the interface jumpers on the access card are set to match the physical interfaces marked on the fantails (V.35, X.21 or RS-449). If you are bringing up an E1 circuit emulation module, make sure the interface jumpers on the access card are set properly. If you are having signal quality problems with a physical interface on an access card, check the following: Make sure that cables are within the specifications of your media type. Make sure that connectors are not damaged. Check optical connectors for dirt or scratches on the optical surface. For electrical connectors, check that pins are not bent, broken, or loose.
Interface module problems	If you are experiencing any of the following symptoms, you might need to replace the line card or its access card: POST fails (the FLT LED stays lit and the POST indicates a problem). A card fails even when moved to another slot. (When you move a line card, be sure to pair it with an appropriate access card. When you move an access card, pair it with an appropriate line card.) Hardware diagnostics fail. The line card fails to load. The line card hangs repeatedly. The line card or access card cannot be fully inserted into its slot. This probably indicates damage to the connectors on either the card or the midplane. Inspect all the connectors and replace the card or the midplane if you find damage. Two or more ports are passing no traffic, dropping many cells, or flapping. If only one port has symptoms, there is probably a problem with the line, the external DSU/CSU if one is present, the access card, or the remote device. Use the looping tests described in the LightStream 2020 Network Operations Guide to help isolate the problem.
Disk assembly problems	Disk assembly problems are indicated by the following symptoms: The node fails to boot. Files become corrupted. In a system with two NPs, the primary NP appears to fail and the backup takes over. The failed NP might pass diagnostics. The system fails to read or write floppy disks. In the case of a write failure, check the write protect switch on the disk. If a disk problem is indicated, check the disk assembly connector for bent or broken pins. If any pins are bent or damaged, they are the likely source of the problem. Replace the disk assembly connector. If the connector is in good condition, the problem may be in the disk assembly itself, or in the software on the disk. If you suspect a problem with the software, you should be able to correct it by reinstalling the software as described in the LightStream 2020 Network Operations Guide.
Midplane problems	Midplane problems are indicated by the following symptoms: A card fails in one slot but operates normally in other slots. Data transmission problems do not go away when you replace the FRU8 that appears to be failing, or problems occur in several FRUs simultaneously. (Problems of this type might also indicate a faulty switch card.) Failure of a card to fully insert into its slot. This probably indicates damage to the connectors on either the card or the midplane. Inspect all the connectors and replace the card or the midplane if you find damage. Electrical failure or electrical problems that do not go away when you replace the FRU that appears to be failing, or that occur in several FRUs simultaneously. Electrical problems include out-of-range voltages.

1CLI = command-line interface 2POST = power-on self-test 3FLT = fault 4TCS = test and control system 5NP = network processor 6NPAC = network processor access card 7FDDI = Fiber Distributed Data Interface 8FRU = field-replaceable unit

To test a system, start with a simple hardware configuration and add one card at a time until a failed interface appears or is isolated. Use a simple software configuration and test connectivity using a ping test.

If you determine that a part or card replacement is required, contact your sales or technical support representative. Specific instructions concerning part or card installation are outlined in the configuration note provided with the replacement.

If a part replacement appears to solve a problem, reinstall the suspect part to verify the failure. Always double-check a repair.

Possible Problem	Solution
Incorrect or corrupted image (exec does not function)	Step 1 Check the configuration register using the show version exec command. Set the register to boot from Flash memory. For information about configuration register settings, refer to your hardware installation and maintenance documentation. Step 2 Power cycle the router. Step 3 Within the first 60 seconds of booting, press the Break key to access the ROM monitor. Step 4 At the ROM monitor prompt (>), enter o/r 0x1 to set the configuration register to boot from ROM. Step 5 Enter i to reinitialize the router, which causes the router to enter setup mode. Step 6 Obtain the correct system image. If necessary, contact your technical support representative to determine which image is correct. Step 7 After the correct image is identified, use the copy tftp flash privileged exec command at the router to retrieve the image. Step 8 Check the configuration register using the show version exec command. Set the register to boot from Flash memory. Step 9 Use the show running-config privileged exec command to see whether the router configuration contains the boot system flash global configuration command. Note: Issuing the copy running-config startup-config command at this point on a Cisco 2500, Cisco 3000, Cisco 4000, or Cisco 7000 series will overwrite the configuration. Make sure you have a backup of your configuration file.
Incorrect or corrupted image (exec does not function)	Step 10 Include the boot system flash command if it is not in the configuration. Be sure to use the copy running-config startup-config command after this change. Step 11 Enter the reload privileged exec command to restart the router. Syntax: The following is the syntax for the reload command: reload [text] | [in [hh:]mm [text]] | [at hh:mm [month day | day month] [text]] | [cancel] Examples: The following example illustrates how to use the reload command to immediately reload the software on the router: Router# reload The following example illustrates how to use the reload command to reload the software on the router in 10 -minutes: Router# reload in 10 Router# Reload scheduled for 11:57:08 PDT Fri Apr 21 1996 (in 10 minutes) Proceed with reload? [confirm] Router#
Incorrect or corrupted image (exec functions)	Step 1 Obtain the correct system image. If necessary, contact your technical support representative to determine which image is appropriate. Step 2 Use the copy tftp flash privileged exec command to retrieve the image. Step 3 Check the configuration register using the show version exec command. Set the register to boot from Flash memory. For information about configuration register settings, refer to your hardware installation and maintenance documentation. Step 4 Use the show running-config privileged exec command to determine whether the active configuration contains the boot system flash global configuration command. Use the show startup-config privileged exec command to determine whether the boot system flash command is included in the configuration stored in NVRAM.1 Step 5 Include the boot system flash command if it is not in the configuration. Be sure to use the copy running-config startup-config privileged exec command to save your modification after this change. Step 6 Enter the reload privileged exec command to restart the router. Syntax: The following is the syntax for the reload command: reload [text] | [in [hh:]mm [text]] | [at hh:mm [month day | day month] [text]] | [cancel] Examples: The following example illustrates how to use the reload command to immediately reload the software on the router: Router# reload
Incorrect or corrupted image (exec functions)	The following example illustrates how to use the reload command to reload the software on the router in 10 -minutes: Router# reload in 10 Router# Reload scheduled for 11:57:08 PDT Fri Apr 21 1996 (in 10 minutes) Proceed with reload? [confirm] Router#

1NVRAM = nonvolatile random-access memory

Possible Problem	Solution
No system image in Flash memory	Step 1 Use the show flash exec command to determine whether an image exists in Flash memory. Step 2 If no image exists, use the copy tftp flash privileged exec command to copy the system image from your TFTP1 server to the router's Flash memory.The following is an example of the use of the copy tftp flash command: router# copy flash tftp filename The router asks you for the IP address of the TFTP server and the name of the image file you are copying to the server. A sample of the output for this command using IP address 131.108.10.6 and filename ic92130n follows: IP address of remote host [255.255.255.255]? 131.108.10.6 Name of file to copy []? ic92130n writing ic92130n !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! router#
No system image in Flash memory	Step 3 Enter the reload privileged exec command to reboot the router. Syntax: The following is the syntax for the reload command: reload [text] | [in [hh:]mm [text]] | [at hh:mm [month day | day month] [text]] | [cancel] Examples: The following example illustrates how to use the reload command to immediately reload the software on the router: Router# reload The following example illustrates how to use the reload command to reload the software on the router in 10 minutes: Router# reload in 10 Router# Reload scheduled for 11:57:08 PDT Fri Apr 21 1996 (in 10 minutes) Proceed with reload? [confirm] Router#
Missing boot system flash global configuration command	Step 1 Use the show running-config privileged exec command to determine whether the configuration includes a boot system flash global configuration command entry. Use the show startup-config privileged exec command to determine whether the boot system flash command is included in the configuration stored in NVRAM2. Step 2 Check the order of the boot system commands. For the recommended ordering, refer to the section "Fault-Tolerant Boot Strategies" earlier in this chapter. Step 3 Add the boot system flash command or reorder the boot system commands if necessary. Step 4 Save the configuration change to NVRAM using the copy running-config startup-config privileged exec command. The required syntax is as follows: copy running-config {rcp | startup-config | tftp | file-id} (Cisco 7000, Cisco 7200, and Cisco 7500 series only) Syntax Description: rcp---Specifies a copy operation to a network server using RCP. startup-config---Specifies the configuration used for initialization as the destination of the copy operation. The Cisco 4500 series cannot use this keyword. tftp---Specifies a TFTP server as the destination of the copy operation. file-id---Specifies a device:filename as the destination of the copy operation. The device argument is optional; but when it is used, the colon (:) is required.
Misconfigured configuration register	Use the show version exec command to check the configuration register setting. Make sure it is set to boot from Flash memory. Refer to your hardware installation and maintenance publication for details regarding configuration register settings.

1TFTP = Trivial File Transfer Protocol 2NVRAM = nonvolatile random-access memory

The following output is an example of a failed netboot session:

Booting gs3-bfx..........[failed]

Possible Problem	Solution
Network is disconnected or isolated	Step 1 Boot the router from ROM or Flash memory if possible. Step 2 Use the ping exec command to send a message to the broadcast address (255.255.255.255). Step 3 If there is no response from the server, use the show arp exec command to look for an entry in the ARP table that is associated with the server. Step 4 Use the show ip route exec command to view the IP routing table. Look for an entry in the table for the network or subnet of the server. Sample Display: The following is sample output from the show ip route command when entered without an address: Router# show ip route Codes: I - IGRP derived, R - RIP derived, O - OSPF derived C - connected, S - static, E - EGP derived, B - BGP derived * - candidate default route, IA - OSPF inter area route E1 - OSPF external type 1 route, E2 - OSPF external type 2 route Gateway of last resort is 131.119.254.240 to network 129.140.0.0 O E2 150.150.0.0 [160/5] via 131.119.254.6, 0:01:00, Ethernet2 E 192.67.131.0 [200/128] via 131.119.254.244, 0:02:22, Ethernet2 O E2 192.68.132.0 [160/5] via 131.119.254.6, 0:00:59, Ethernet2 O E2 130.130.0.0 [160/5] via 131.119.254.6, 0:00:59, Ethernet2 E 128.128.0.0 [200/128] via 131.119.254.244, 0:02:22, Ethernet2 E 129.129.0.0 [200/129] via 131.119.254.240, 0:02:22, Ethernet2 E 192.65.129.0 [200/128] via 131.119.254.244, 0:02:22, Ethernet2 If a path to a boot server exists, a disconnected network is not the problem. If no path exists, make sure that a path is available before again attempting to netboot.
TFTP server is down	Step 1 Check the TFTP server to determine whether it is up and running. You can do this by attempting to make a TFTP connection from the boot server to itself. The connection will be successful if the TFTP server is running. Step 2 If the TFTP server is not running, initialize it. The initialization process will vary depending on the type of boot server. For a BSD UNIX server, check the /etc/inetd.conf file. If the TFTP server is not included in this file, add the appropriate line and cause inetd to reload its configuration.
Router image in wrong directory	Step 1 Look at the server configuration file to see whether it points to the directory in which the router image is located. Step 2 Move the router image to the correct directory if necessary. Step 3 Make sure the /tftpboot directory is reachable over the network.
Router system image file permissions are incorrect	Step 1 Check the permissions of the system image file. Step 2 If necessary, change the permissions for the file. On a UNIX boot server, set the permissions for the file to owner read/write, group read, and global read (the UNIX command for setting these permissions is chmod 644 filename).
Bad protocol address	Step 1 Check the server configuration file to make sure the IP address of the host is correct. Step 2 Change the configuration if it is incorrect.
Missing or -misconfigured default gateway specification	Step 1 Use the show running-config privileged exec command to view the router configuration. Check for the ip default-gateway global configuration command, which defines a default gateway. Syntax: ip default-gateway ip-address Syntax Description: ip-address---IP address of the router. Step 2 If the command is missing, add it to the configuration. If the command is present, make sure it specifies the correct IP address.
Misconfigured boot system command	Step 1 Use the show running-config privileged exec command to view the router configuration. Check the boot server address (IP address of a TFTP server or MAC1 address of a MOP2 server) that is configured on the router. Step 2 If the address is specified incorrectly, specify the correct boot server address using the boot system global configuration command.
Wrong filename is specified	Step 1 Use the show running-config privileged exec command to view the router configuration. Check the boot filename that is configured on the router. Step 2 Make sure the filename is specified correctly. Change the filename if necessary. Check the host documentation for details about setting the name of the system image on the TFTP server. Step 3 Some versions of the ROM are case sensitive. Try changing the case of the filename. Contact your technical support representative for more information.
Incorrect configuration register setting	To netboot from a server, you must set the configuration register properly. The specific configuration for netbooting depends on the platform that is being booted. Step 1 Check the configuration register setting for your system. Step 2 Determine whether you want to manually or automatically netboot from a TFTP server. To manually netboot, the configuration register must be set to 0x0; otherwise, you will be netbooting using the default system image name or the image specified by the boot system global configuration command. Refer to the Cisco IOS configuration guides and command references and your hardware installation and maintenance publications for more details about setting the configuration register.

1MAC = Media Access Control 2MOP = Maintenance Operation Protocol

Possible Problem	Solution
Missing or incorrect tftp-server global configuration command	Step 1 Use the show running-config privileged exec command to determine whether the tftp-server system global configuration command is missing or incorrectly specified. Step 2 Add or modify the tftp-server system global configuration command as necessary on the router acting as the TFTP server. Specify the name of a file in Flash memory.
Incomplete image in Flash memory	Use the show flash exec command to determine whether the image is incomplete. This display might show that the image is deleted and indicate the reason. Following is an example of show flash output indicating that the image is deleted: babar# show flash 2048K bytes of flash memory sized on embedded flash. File name/status 0 xx-k.914-0.16 1 xx3-confg 2 xx-k.91-4.2 [deleted] [invalid cksum] [0/2097152 bytes free/total]
Incorrect image in Flash memory	Step 1 A "wrong system software" message is displayed when a router attempts to boot an incorrect image. In this case, the router is being booted from the ROM monitor. Following is an example of the ROM monitor output after an attempt to boot an incorrect image: > b gs3-klingon 131.108.9.40 Booting gs3-klingon from 131.108.9.40:!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!O!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!![OK - 2056792/3394950 bytes] F3: 2011628+45132+192972 at 0x1000 Wrong system software for this hardware Step 2 Obtain the correct image. If necessary, contact your technical support representative to determine which image is correct. Step 3 When you identify the correct image, use the copy tftp flash privileged exec command to retrieve the image.

The following example shows a netbooting session that contains excessive timeouts and out-of-order packets:

Booting gs3-bfx from 131.108.1.123: !O.O!.O..O!!!OOO.O!!.O.O.....

The client router might boot in this situation. However, when excessive timeouts and out-of-order packets occur, there is probably a network problem, and netbooting (as well as network service availability) might be inconsistent.

Possible Problem	Solution
Link is saturated	Step 1 Boot the router from ROM and ping the TFTP server. Determine whether timeouts and out-of-order packets appear. Step 2 Check local network concentrators for excessive collisions on the same network. If there are excessive collisions, reorganizing your network topology might help reduce collisions. Step 3 Use the show interfaces exec command on routers in the path or place a network analyzer between the router and server. Look for dropped packets and output errors. Step 4 If approximately 15%  or more of the traffic is being dropped, or if any output errors occur, congestion might be the problem. Step 5 Wait until the traffic subsides before attempting to netboot the router. If the problem is chronic, increase bandwidth or move the server closer to the router being booted.
Link is down	Step 1 Check the continuity of the path from the booting router to the boot server using ping or trace exec commands. Step 2 If a break is found, restore the link and attempt to netboot again.

Booting gs3-bfx!OOOO..........[timed out]
 
Booting gs3-bfx!.O.O.O.O..........[timed out]
 
Booting gs3-bfx!!!!!!!!!!OOOOOOOOOO..........[timed out]

Booting gs3-bfx!..........[failed]

Possible Problem	Solution
Bad routing paths on neighbor routers	Step 1 Verify that neighbor routers can ping the server. Step 2 Use the trace exec command to determine the path to the server. Step 3 Use the show arp privileged exec command to examine the ARP1 tables or the show ip route privileged exec command to view the IP routing table. Verify that the server is listed and that the routing table entries are appropriate. Step 4 Use the clear arp-cache and clear ip-route privileged exec commands to force the router to repopulate its ARP and routing tables. Step 5 Try to netboot the router again.
Problems caused by multiple paths	Step 1 Shut down all extra interfaces except the one over which you intend to netboot the router. Step 2 Use the no ip proxy-arp interface configuration command on all neighboring routers to disable their ability to provide proxy ARP responses. Make this change with care because it can cause problems for other network traffic. If you do not want to disable proxy ARP, boot the router from ROM and configure the ip default-gateway global configuration command. Step 3 Try to netboot the router again.

1ARP = Address Resolution Protocol

Booting gs3-bfx..........[timed out]

Possible Problem	Solution
Intermediate routers have ARP filtering enabled	Step 1 Boot the router from ROM. Step 2 Make sure you can ping the server from the router. Step 3 Use the copy running-config tftp privileged exec command to test TFTP connectivity to the server. Step 4 If the preceding steps are successful, check the configuration at the intermediate router using the show arp exec command. Step 5 Enable the debug arp privileged exec command to determine whether neighbor proxy ARP responses are being generated. Caution: Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use. Step 6 If the neighbor is not sending proxy ARP responses and its configuration contains the no ip proxy-arp interface configuration command, disable ARP filtering by removing the entry. Note that proxy ARP is enabled by default. Step 7 If you need to have a no ip proxy-arp entry in the neighbor router configurations, use the ip default-gateway global configuration command on the router to specify a default gateway.
Missing or misconfigured IP helper address on intermediate router	Step 1 Check the configurations of all routers in the path. Make sure that all intermediate routers have an IP helper address specified that points to the TFTP server. Syntax: ip helper-address address Syntax Description: address---Destination broadcast or host address to be used when forwarding UDP1 broadcasts. You can have more than one helper address per interface. Step 2 Include helper addresses as required using the ip helper-address interface configuration command. If you are unicasting to your server, you do not need to use the IP helper address, but if you are broadcasting to 255.255.255.255 (by omitting the IP address of the server), add the ip helper-address command on the neighboring router interface used in the netbooting broadcast.

1UDP = User Datagram Protocol

Possible Problem

Solution

Filename mismatch

Step 1 If you are booting manually, refer to the user guide for your router to see the proper command-line format. Step 2 Check the router configuration file. Compare the filename specified in the boot system filename [address] global configuration command entry with the actual router image filename. Make sure they match. Step 3 If the filenames differ, change the name in the configuration file. Remember to use the router image filename in the boot system global configuration command specification and the configuration filename with the boot host and boot network global configuration commands.

Possible Problem	Solution
Incorrect EPROM1 size setting	Step 1 Power down the system. Step 2 Inspect EPROM size jumpers. Refer to the hardware installation and maintenance publication for your router to determine the proper setting. Step 3 Move jumpers as required.
Configuration register is not set correctly	Step 1 Check your configuration settings (boot ROM jumpers and software configuration). If no jumper is set at bit 0, and no other boot field is defined, you must reconfigure your system so that it can boot properly. Step 2 To enable your router to boot properly, do one of the following: Configure the software configuration register of the router using the config-register value global configuration command. (This applies to the IGS, Cisco 2500, Cisco 3000, and Cisco 7000 platforms running Cisco IOS Release 10.0 or later in the EPROM.) Set the boot ROM jumper to permit booting. Include the correct boot system global configuration commands to boot the system. Set bit 0 to a value of 1 to force booting from ROM. Refer to the Cisco IOS configuration guides and command references, as well as your hardware installation and maintenance publications, for more information about configuring your router for the various booting options.

1EPROM = erasable programmable read-only memory

Possible Problem	Solution
Incorrect configuration register setting	Step 1 At the ROM monitor prompt (>), enter b to boot the system. Step 2 If a configuration exists in NVRAM, the system displays the vacant message. Press the Enter key to continue. If a configuration does not exist in NVRAM, the setup menu appears. Skip the setup process. Step 3 Use the show version exec command to determine the configuration register setting. Step 4 Look for an invalid configuration register setting. The default is 0x101, which disables the Break key and forces the router to boot from ROM. A typical "bad" setting has a 0 in the least significant bit (for example 0x100). For details about setting the configuration register, refer to your hardware installation and maintenance publication.
Break key pressed during boot process	At the ROM monitor prompt, enter c to allow the router to continue booting.
Console cable inserted or removed during boot process, or console power-cycled during boot process	Step 1 Press the Enter key and wait for the ROM monitor prompt (>). Step 2 If the ROM monitor prompt appears, enter c at the prompt to continue the booting process.

Possible Problem	Solution
Wrong terminal speed setting	Step 1 Use the monitor setup menu to check the terminal line speed setting for the monitor. Step 2 Check the terminal speed configured on the router as specified in the configuration register setting (default is 9600 baud, 8 databits, 2 stop bits, and no parity). Step 3 If the terminal speed of the monitor and the router do not match, modify as necessary. Refer to your hardware installation and maintenance documentation for details about setting up the monitor.
Router hardware problem	Check all hardware for damage, including cabling (broken wire), adapters (loose pin), router ports, and so forth. For more information, refer to the hardware troubleshooting information discussed earlier in this chapter.

Possible Problem

Solution

EPROM problem

Generally, this problem occurs only if you have just replaced your system EPROMs. Step 1 Power down the system. Step 2 Inspect each EPROM. Make sure each EPROM is correctly positioned in the socket (with notches properly aligned) in the correct socket. Step 3 If a pin is bent, straighten it carefully. Reinstall the EPROM and power up the system. If a pin breaks off, the EPROM must be replaced. Step 4 If an EPROM has been installed backward and power has been applied to it, the EPROM has been damaged and must be replaced. Step 5 If local timeouts persist, contact your technical support representative.

Possible Problem	Solution
Flow control configured on the terminal conflicts with the EIA/TIA-232 control signals supported by the access server console port (RJ-45 to DB-25)	Step 1 Check whether flow control is configured on your terminal. Step 2 Disable all flow control on the terminal. With flow control enabled, the terminal will wait indefinitely for a CTS1 signal because the RJ-45 console port on the access server does not assert CTS. For information on how to check for and disable flow control on your specific terminal, consult the documentation provided by your terminal manufacturer. Step 3 Alternatively, you can "strap," or short, CTS high by providing the proper voltage on the CTS signal lead to make the signal active. Find an unused signal that is known to be active and strap CTS to it. The terminal sees CTS being asserted (indicating that the access server is ready to receive data) and allows input to be entered. Step 4 On an already configured access server, another solution is to connect your terminal to the auxiliary port of the access server. The auxiliary port, unlike the console port, asserts CTS, and the terminal will therefore allow input. However, on a new access server with no configuration, this is not an alternative because the bootup banners and Setup routine are seen only on the console port.
Hardware problem	Step 1 Check all hardware---including cabling (broken wires), adapters (loose pins), access server ports, and the terminal itself---for damage. Step 2 Replace any hardware that is damaged or excessively worn. For more information, refer to the hardware troubleshooting information earlier in this chapter.

1CTS = clear-to-send