cc/td/doc/product/dsl_prod/6160
hometocprevnextglossaryfeedbacksearchhelp
PDF

Table of Contents

Troubleshooting

Troubleshooting

Read this chapter to learn how to isolate faults in the Cisco 6160 digital subscriber line (DSL) access concentrator. Most problems in a Cisco 6160 system can be traced to one of the system's field-replaceable units (FRUs).

Overview

This chapter consists of three major sections:

Hot-swappable FRUs

All Cisco 6160 FRUs are hot swappable, except:

However, hot-swapping some FRUs causes an interruption in service. See Table 4-1.


Table 4-1: Service Interruptions Caused by the Replacing of FRUs
FRU Does Hot Swap Interrupt Service? Notes

NI-2 card

Yes

-

Line card

Yes

Service is interrupted only for subscribers served by that line card.

I/O card

Yes

You must turn off system power to replace the I/O card. Note that the MAC address is stored on the I/O card.

Blower assembly

No

-

Air filter

No

-

PEM

See next column.

You can replace a PEM without interrupting service if the other PEM in the chassis is operating normally.

Basic Checks

Before using the troubleshooting tables in this chapter, make the following basic checks:


Note Check the release notes for more information on how to solve problems.

Contacting the Cisco TAC for Help

In certain situations, the troubleshooting tables in this chapter direct you to contact the Cisco Technical Assistance Center (TAC) for help. If you have a maintenance contract or if your hardware is under warranty, call the TAC at +1 800 553 2447 (North America only), +1 408 526 7209, or visit this URL for a worldwide list of TAC regional telephone numbers:

http://www.cisco.com/warp/public/687/Directory/DirTAC.shtml

System-wide Problems

This section offers suggestions for problems that affect the entire Cisco 6160 system.

Symptoms Steps to Take

System fails to come up.

    1. Check the Power LED on the NI-2, the Status LEDs on the line cards, and the Input OK LED on each PEM. If all LEDs are off, or if the red Out Fail LED on either PEM is on, troubleshoot the PEMs. See the "PEM Problems" section.

    2. If any green LEDs are on, the system has power. Check the Status LED on the NI-2 card. If the Status LED is off, refer to the "NI-2 Card Problems" section.

    3. If the Cisco 6160 is set to boot from a remote device over the network, make sure the remote device is up, that its network connection is solid, and that it contains the boot file. (The Cisco 6160 tries to boot over the network for a configured period, usually 5 to 15 minutes. If it is unable to boot over the network, it will eventually boot from bootflash.)

    4. Try to establish a console connection to the Cisco 6160. If you cannot connect, see the steps for the symptom "You cannot establish a console or Telnet connection to the system."

    5. If you achieve a console connection:

    • Examine the command prompt. If the prompt says rommon 1>, the problem could be in flash memory, in bootflash, in an incorrectly set boot configuration register, or in an incorrect file name in a boot system command in the startup-config file. Refer to the Configuration Guide for Cisco DSLAMs with NI-2 for information on setting and interpreting configuration registers, configuring flash memory, and editing the startup-config file.

    To get out of the ROM monitor, enter dir flash:. In the resulting display, find the name of the software image. Then enter boot flash:imagename, replacing imagename with the name of the software image. The system boots from flash.

You cannot establish a console or Telnet connection to the system.

    1. For a console problem, check the terminal settings against the list of settings in the "Connecting a Console Terminal" section.

    2. For a Telnet problem:

    3. For both console and Telnet problems:

System experiences a critical, major, or minor alarm.

    1. Enter the command show facility-alarm status. Note the affected slot and port, if any, and the description of the problem.

    2. If no slot number is indicated, enter show environment all and examine the results for an indication of which FRU is at fault. Refer to the "FRU-specific Problems" section for instructions on troubleshooting that FRU.

    3. If ATM0/0, ATM0/1, ATM0/2, or ATM0/3 is indicated:

    4. If a line card slot is indicated, consult the "Line Card Problems" section.

A trunk or subtending port fails to come up. (OC-3c or DS3.)

    1. Check the cable connections at both ends. Refer to "Pin Assignments," to check pinouts.

    2. To check the interface status and configuration, enter show interface atm slot#/port#. Check the following in the resulting display:

    • If the port's Admin Status is down, enter the commands below to correct the problem, replacing slot/port ID atm 0/1 with your slot/port ID:

      DSLAM> configure terminal
      Enter configuration commands, one per line.
      End with CNTL/Z.
      DSLAM(config-if)# int atm 0/1
      DSLAM(config-if)# no shut
      DSLAM(config-if)# 
       
      
If you need more information on interface configuration, refer to the Configuration Guide for Cisco DSLAMs with NI-2 and the ATM Switch Router Software Configuration Guide.

    3. For a DS3 interface: enter show controllers atmslot#/port#. Check the following in the resulting display:

    4. Check the status and configuration of the interface at the far end.

    5. If you need to run a loopback test, do the following:

    6. In interface configuration mode, reset the trunk port by executing the shut command followed by the no shut command.

    7. Replace the NI-2 card.

    8. If the problem with a DS3 interface persists, troubleshoot the I/O card. See the "I/O Card Problems" section.

System overheats.

Troubleshoot the blower assembly. See the "Blower Problems" section.

System experiences a clocking problem.

Troubleshoot the NI-2. See the "NI-2 Card Problems" section.

System experiences a power problem.

Troubleshoot the PEMs. See the "PEM Problems" section.

FRU-specific Problems

The following sections describe symptoms that might occur and the steps that you need to take if you experience problems with any Cisco 6160 field replaceable unit (FRU).

NI-2 Card Problems

If you need to remove or replace an NI-2 card, follow the steps below and also refer to the Cisco 6160 NI-2 Card Installation and Configuration document for instructions.

Symptom Steps to Take

Power LED is off.

    1. Check the Status LEDs on the line cards and the Input OK LED on each PEM. If all LEDs are off, or if the red Out Fail LED on either PEM is on, troubleshoot the PEMs---see the "PEM Problems" section.

    2. If the line card Status LEDs and the PEM Input OK LEDs are lit, remove the NI-2 card from its slot and check for bent or broken pins on both the card and the backplane. If you find damaged pins on the card, replace it. If you find damaged pins on the backplane, contact the Cisco TAC.

Status LED is off, indicating that the NI-2 failed to boot or failed its power-on self test.

Press the Reset button on the NI-2. If the problem persists, replace the card.

Critical LED, Major LED, or Minor LED is on.

See the "System-wide Problems" section.

A trunk or subtending port fails to come up.

See the "System-wide Problems" section.

NI-2 cannot be fully inserted into its slot.

Inspect connectors on both the card and the backplane, looking for obstructions, bent pins, or other damage. If you find damage to a line card connector, replace the line card. If you find damage to a backplane connector, contact the Cisco TAC.

Line Card Problems

If you need to remove or replace a line card, follow the steps below.

Symptom Steps to Take

All LEDs are off.

    1. Check the Power LED on the NI-2 card and the Input OK LEDs on the PEMs. If none are lit or if the red Out Fail LED on either PEM is lit, troubleshoot the PEM---see the "PEM Problems" section.

    2. If the NI-2 Power LED and the PEM Input OK LEDs are lit, remove the card from its slot and check for bent or broken pins on both the card and the backplane. If you find damaged pins on the card, replace it. If you find damaged pins on the backplane, contact the Cisco TAC.

All ports on a card fail to come up (modems do not train). Status LED might be red, indicating that the line card failed to boot or failed its power-on self test.

    1. Enter the show ipc nodes command to find out whether there is a communication problem between the line card and the NI-2. There should be an entry in the resulting display for each line card ("SMB IP Slot n") and for the NI-2 ("IPC Master"). If there is an entry for each card, go to step 4.

    2. If one or more cards are not listed, enter show oir status. If the resulting display indicates that the card in question is loading new code, wait a few minutes and reenter the command. (Usually, 2 to 3 minutes is long enough to wait.) The card's status should change to running.

    3. Enter the command show dsl int atm slot#/port#. If the status says Microcode downloading, wait 10 minutes. When the download is complete, the card will reboot and come up normally.

    4. Check the alarm LEDs on the NI-2 card or enter show facility-alarm status to determine the alarm status of the system. If any alarms are indicated, see the "System-wide Problems" section for instructions on how to troubleshoot alarms.

    5. Enter the command show dsl status and examine the results to ensure that the card is configured to be in its current slot. (In the Names column, the slot number appears as part of the port ID. For example, in ATM19/2, the slot number is 19. Nothing is displayed for slots that are not configured.) If necessary, use the slot command to update the configuration, or move the card to the correct slot. If the card is a Flexi, it will not function unless you use the slot command to configure either CAP or DMT operation.

    6. Reset the line card by disconnecting it from the backplane and reseating it in its slot.

    7. Install the card in another slot.

    8. If the problem persists, replace the line card.

Port fails to come up (modems do not train), or port LED flashes continuously.

    1. Enter the command show dsl int atm slot#/port# to display the port's configuration. Check the display to ensure that the port is properly provisioned. Make sure the port is configured to be running ("no shut" or IOS admin state = up). Also check the line status; if this reads "No CPE detected," troubleshoot the CPE device and the loop as described below under the symptom "You suspect a problem with the CPE or the subscriber loop."

    2. In interface configuration mode, reset the port by executing the shut command followed by the no shut command.

    3. Connect the subscriber to another port on the Cisco 6160. If the modems do not train, troubleshoot the CPE device and the loop as described below under the symptom "You suspect a problem with the CPE or the subscriber loop." If the modems train, go to the next step to troubleshoot the port that failed.

    4. To test the modem on the line card, enter dsl test atmslot#/port# self. Results are displayed automatically after a few seconds. If the modem fails the test, replace the line card.

You suspect a problem with the CPE or the subscriber loop.

    1. Disconnect the local loop and replace it with a test setup that consists of a modem or CPE that is known to work and a few thousand feet of wire. If the modems train over the shorter distance, the problem lies in the local loop or in the CPE---see the steps that follow. If the modems do not train, the port is probably at fault. Replace the line card.

    2. Make sure that the CPE at the subscriber site is powered up.

    3. Make sure that the CPE is compatible with the Cisco 6160, and that the software version running on the CPE is compatible with the software version running on the Cisco 6160.

    4. Power cycle the CPE. When you turn the power on, the WAN Link LED on the CPE should blink. If the LED does not blink, check the CPE's configuration---if the interface is shut down, bring it up.

    5. Perform a continuity check to find out if the cabling to the CPE is connected and intact. For example, if there is a phone on the line, check for a dial tone.

    6. If there is no POTS, check DC resistance by shorting tip and ring at the remote end.

    7. Make sure there are no load coils on the local loop.

    8. Is the local loop too long? The maximum length range is 15,000 to 25,000 feet (4572 to 7620 meters). Within that range, wire gauge, crosstalk, and multiple bridge taps reduce the distance over which the modems can train.

    9. Is the local loop too short? DMT modems train best over loops of several thousand feet. In a test situation with a loop just a few feet in length, the modems may fail to train. Add wire to the loop.

    10. Replace the CPE.

Modems train at a low bit rate, or modems retrain continuously, or the line experiences too many errors.

    1. If Flexi cards and SDSL cards are installed on the same side of the chassis, reinstall the cards so that all the Flexi cards are on one side and all the SDSL cards are on the other side. Mixing Flexi and SDSL cards can decrease performance.

    2. Enter the command show dsl int atmslot#/port# to display the port's configuration. Check the display to ensure that the port is properly provisioned. Look in particular for these statistics:

    3. Crosstalk is caused by interference between services in adjacent cables. It affects random bits rather than chunks of data; upstream and downstream traffic can be affected differently. If crosstalk is increasing the bit error rate (BER), you can compensate in several ways:

    4. If you experience impulse noise or clipping, both of which affect chunks of data rather than random bits, you can compensate by turning on or increasing interleaving and error correction. (However, note that this approach adds delay.) Use the dmt interleaving-delay command for interleaving. Use the dmt encoding-trellis command or the dmt check-bytes command for error correction. If you use dmt check-bytes and if the codeword size is not set to auto, you might need to use the dmt codeword-size command to adjust the codeword setting.

    5. If errors or retraining occur while the line is ringing, use the dmt interleaving-delay command to turn on interleaving.

    6. In interface configuration mode, reset the port by executing the shut command followed by the no shut command.

    7. Connect the customer to a different port on the Cisco 6160. If the modems train, replace the line card with the faulty port.

    8. Troubleshoot the CPE device and the loop as described above under the symptom "You suspect a problem with the CPE or the subscriber loop."

    9. Replace the line card.

Card cannot be fully inserted into its slot.

    1. Remove the card and reinsert it, pushing firmly on both the top and the bottom of the faceplate. (The card might jam in the slot if you apply pressure to the top only.) If the chassis is full, you might need to push sideways to insert the card.

    2. Inspect connectors on both the card and the backplane, looking for obstructions, bent pins, or other damage. If you find damage to a line card connector, replace the line card. If you find damage to a backplane connector, contact the Cisco TAC.

Card experiences problems in one slot but operates normally in another.

There may be a fault in your backplane. Contact the Cisco TAC.

I/O Card Problems

If you need to remove or replace the I/O card, follow the steps below and also refer to the Cisco 6160 I/O Card Installation and Configuration document for instructions.

Symptom Steps to Take

A trunk port fails to come up.

See the "System-wide Problems" section.

A subtending port fails to come up.

See the "System-wide Problems" section.

An alarm relay or BITS circuit fails.

    1. Check the connection at the wire wrap connector on the I/O card. Refer to the "Wire Wrap Connector" section for the pin assignments of the wire wrap connector.

    2. If the problem involves the circuit breaker trip alarm, check the connection at the small green terminal block behind each PEM.

    3. Check the connection at the other end of the circuit.

    4. For a BITS problem, troubleshoot the T1 line at the other end of the circuit.

    5. Replace the I/O card (except in the case of a circuit breaker alarm problem; the I/O card does not provide the circuit breaker trip alarm).

This message appears on the console or in the system log file:

00:00:28:%C6100-4-COOKIE:
Corrupt or missing MAC
address cookie using random
base 007e.eb7d.e700

    1. Make sure the software release running on your system is compatible with the hardware. The Cisco 6160 requires Cisco IOS Release 12.1(1)DA or greater.

    2. Enter the command show hardware chassis. In the resulting display, look for data from the I/O card EEPROM. If the system can read the contents of the EEPROM, the I/O card is likely to be healthy.

    3. If the I/O card is missing, install it.

    4. If the I/O card is present, reseat it. First, turn off system power. Use a screwdriver to unfasten the screws that secure the I/O card cover, and pull the card off the connectors. Push it firmly back into place and tighten the screws.

    5. If the problem persists, replace the I/O card.

Blower Problems

If you need to remove or replace the blower assembly, follow the steps below and also refer to the Cisco 6160 Blower Assembly and PEM Installation and Configuration document for instructions.

Symptom Steps to Take

Blowers do not run.

    1. Make sure the blower assembly is fully inserted into the chassis and screwed in place.

    2. Check input power connections. If connections are loose, or if their polarity is reversed, the chassis does not receive power and the blowers do not run.

    3. Replace the blower assembly.

Blowers run but the system overheats.

    1. Make sure that the air intake vents at the bottom front of the chassis and the exhaust vents on the top of the chassis are free of blockages. (See the "Cooling Vents" section.)

    2. Make sure that the ambient temperature and other environmental factors affecting the system are within the ranges specified in the "System Specifications and Compliances" section.

    3. Make sure that all cards, blank faceplates, and covers are in place. The cooling system cannot operate effectively unless the chassis is fully enclosed.

    4. Check the LED on the blower assembly. If the LED is red, one or more blowers have failed. Replace the blower assembly.

    5. Check the air filter at the bottom of the chassis, and if necessary clean or replace it. Refer to "Preventive Maintenance," for instructions on cleaning or replacing the air filter.

    6. Reduce the ambient temperature.

Blower status LED is red.

One or more blowers have failed. Replace the blower assembly.

Blower status LED is off.

The blower assembly has failed or is not receiving power. If other components in the system are receiving power (look for green LEDs), replace the blower assembly.

Blower speed changes unpredictably.

The system changes the blower speed automatically under certain conditions. See the "Blower Assembly and Air Filter" section for a list of those conditions.

Air filter is damaged, dirty, or clogged.

Refer to "Preventive Maintenance," for instructions on cleaning or replacing the air filter.

PEM Problems

If you need to remove or replace a power entry module, follow the steps below and also refer to the Cisco 6160 Blower Assembly and PEM Installation and Configuration document for instructions.

Symptom Steps to Take

Input OK LED on PEM fails to go on.

    1. Make sure the circuit breaker on the PEM is turned to ON (1).

    2. Make sure the PEM is properly seated in the chassis and screwed in place.

    3. Make sure the power leads are properly connected to power terminals on the backplane behind the PEM. If connections are loose or their polarity is reversed, the chassis does not receive power.

    4. Check the power cable for breaks, cracked insulation, loose lugs, and signs of wear. Replace the power cable if it is not in good condition.

    5. Check the power source.

    6. Turn the PEM to OFF (0) and move it to the other PEM slot. If the PEM still fails, replace it.

Out Fail LED on PEM goes on.

    1. Make sure the PEM is fully inserted into its slot and firmly screwed in place.

    2. Replace the PEM.

    3. If the problem persists, there may be a fault in your backplane. Contact the Cisco TAC.

Circuit breaker on PEM trips.

A tripped circuit breaker indicates a serious overcurrent situation. The probable cause is a faulty backplane, a faulty power connector on the backplane, or a faulty PEM. Contact the Cisco TAC for assistance.

PEM has problems in one slot but operates normally in the other.

There might be a fault in your backplane. Contact the Cisco TAC.

Alarms

Tables 4-2 through 4-8 describe alarms for the Cisco 6160. In each table, the text in the Alarm column is the text that appears in the description field of the alarm message. Alarm messages appear on the console screen as the alarms occur; to see a list of current alarms, enter show facility-alarm status.

Refer to the problem tables in this chapter for more detailed troubleshooting instructions.


Table 4-2: Chassis Alarm
Alarm Keyword Severity Description

Chassis temperature too high

OVER_TEMP

Major

An overtemperature condition has been detected. (Temperature is measured on the NI-2 card.)


Table 4-3: Card Slot Alarms
Alarm Keyword Severity Description

Provisioned slot is empty

MODULE_MISMATCH

Major

The card in this slot does not match the type configured for this slot.

Module not equal to provisioning

MODULE_MISSING

Minor

This slot is configured for a card, but no card is present.

Invalid module for this slot

MODULE_INVALID

Minor

The card detected in this slot cannot operate in this slot or is incompatible with the system configuration.

Flex module not provisioned

FLEX_MODULE_NOT_PROVISIONED

Info

You must use the slot command to configure a Flexi card as either CAP or DMT.


Table 4-4: IOS Controller Alarms
Alarm Keyword Severity Description

Loss of active clock sync

LOSS_OF_SYNC

Major

Loss of timing reference. The configured clock source is not available, so the system is using its internal clock.

BITS clock failure

BITS_FAILED

Major

BITS clock failure (LOS or AIS). The configured clock source is not available, so the system is using its internal clock.

The source of the OC-3c alarms in Table 4-5 is one of the following interfaces: ATM0/1 (the trunk) or ATM0/2 (the subtending interface).


Table 4-5: OC-3c/STM-1 Network Interface Alarms
Alarm Keyword Severity Description

Loss of Cell Delineation

SONET_LOCD

Critical

Loss of cell delineation on a SONET line.

Path RDI Received

SONET_PRDI

Critical

Path Remote Defect Indication was received on a SONET line. This is equivalent to Path Far End Receive Failure (FERF).

Path AIS Received

SONET_PAIS

Critical

Path Alarm Indication Signal was received on a SONET line.

Loss of Pointer

SONET_LOP

Critical

Loss of pointer condition on a SONET line.

Line RDI

SONET_LRDI

Critical

Line Remote Defect Indication received on a SONET line. This is equivalent to line Far End Receive Failure (FERF).

Line AIS Received

SONET_LAIS

Critical

Line Alarm Indication Signal received on a SONET line.

Loss of Frame

SONET_LOF

Critical

LOF condition on a SONET line.

Loss of Signal

SONET_LOS

Critical

LOS detected on the SONET line.

Signal Label Mismatch

SONET_SIGNAL_
LABEL

Minor

Incorrect payload type signal label mismatch on a SONET line.

The source of the DS3 alarms in Table 4-6 is one of the following interfaces: ATM0/1 (the trunk), ATM0/2 (subtending interface), or ATM0/2 (subtending interface).


Table 4-6: DS3/E3 Network Interface Alarms
Alarm Keyword Severity Description

Loss of Cell Delineation

DS3/E3_LOCD

Critical

The DS3 line is experiencing a loss of cell delineation.

RAI Received

DS3/E3_RAI

Critical

The DS3 line is receiving a remote alarm indication.

Yellow Alarm Received

DS3/E3_YELLOW

Critical

The DS3 line is receiving a yellow alarm, indicating that another device has detected a failure that might be in this device.

AIS Received

DS3/E3_AIS

Critical

The DS3 line is receiving an Alarm Indication Signal.

OOF Received

DS3/E3_OOF

Critical

The DS3 line has detected an Out of Frame condition.

LOS Detected

DS3/E3_LOS

Critical

The DS3 line has detected Loss of Signal at the framer.

PLCP LOF Detected

DS3_PLCP_LOF

Critical

The DS3 line has detected a Physical Layer Convergence Procedure Loss of Frame error.


Table 4-7: Power Alarms
Alarm Keyword Severity Description

fan voltage out of range

POWER_FAN

Major

The voltage going to the fans from one or both PEMs is outside the acceptable range.

Power supply fault

POWER_FAULT

Major

One or both PEMs are reporting a fault caused by high temperature, output failure, or input failure.

Temperature exceeds limit

PS_OVER_TEMP

Major

The PEM's temperature is above the acceptable range.


Table 4-8: Blower Alarms
Alarm Keyword Severity Description

not detected or missing

FAN_TRAY_REMOVED

Major

The blower assembly has been removed from the chassis or is missing.

Multiple fan failures

FAN_FAIL_MULTI

Major

Two or more blowers (out of four) in the blower assembly have failed. When blowers fail, the remaining blowers run at full speed until the fault is corrected.

Single fan failure

FAN_FAIL_SING

Minor

A single blower (one of four) in the blower assembly has failed. When blowers fail, the remaining blowers run at full speed until the fault is corrected.


hometocprevnextglossaryfeedbacksearchhelp
Posted: Wed May 17 13:13:57 PDT 2000
Copyright 1989 - 2000©Cisco Systems Inc.