Table of Contents

6.1 Before You Begin

6.1.1 Starting MML

6.2 Retrieving and Clearing Alarms

6.2.1 Retrieving All Alarms

6.2.1.1 Retrieving Alarms Continuously

6.2.2 Interpreting Alarm Messages

6.2.2.1 Component ID
6.2.2.2 Alarm Category
6.2.2.3 Severity Level

6.2.3 Acknowledging Alarms
6.2.4 Clearing Alarms

6.2.4.1 Clearing Platform Alarms
6.2.4.2 Clearing Signaling Alarms

6.3 Retrieving System Log Files

6.3.1 Retrieving a Log
6.3.2 Changing the Log Level for Processes

6.4 Performing a Call Trace

6.4.1 Setting the Debug Level
6.4.2 Running the Trace
6.4.3 Creating the SIM file

6.5 Contacting the TAC

6.5.1 Case Classifications
6.5.2 How to Open a Case
6.5.3 Customer Preparation
6.5.4 Getting Help with Your System
6.5.5 TAC Case Initiation
6.5.6 Response Times
6.5.7 TAC Case Resolution

Troubleshooting Your System

This chapter describes three procedures for troubleshooting your system:

• Retrieving, acknowledging, and clearing alarms

• Retrieving and examining system log files

• Performing a call trace

Also this chapter tells you how to contact Cisco's Technical Assistance Center (TAC) and explains what information you will need to gather before contacting them.

6.1 Before You Begin

The telephony controller provides an interface called MML (based on Bellcore TL1) that allows you to retrieve and clear alarms. You should be familiar with MML commands before attempting these procedures. See "MML Commands," for more information about MML commands.

6.1.1 Starting MML

To start an MML session:

Step 1 Log in to the telephony controller and change to any subdirectory under /opt/TransPath (for example, /opt/Transpath/bin).

Step 2 Enter the following command:

mml

If you receive an error message that sessions are already in use, enter the following command:

mml -s session_number

Use any number from 2 to 12 and repeat until you find a vacant session. After successfully entering MML, the prompt changes to:

mml>
 

6.2 Retrieving and Clearing Alarms

The telephony controller generates alarms that let you know if there are problems with your system. You are notified of alarms in one or more ways:

• Your local alarm device (such as a visual alarm box) in conjunction with your alarm relay unit (ARU) receives alarm messages from the telephony controller software and generates a visual or audio notification.

• The telephony controller software's SNMP agent subsystem generates traps that are collected by your SNMP Manager.

• You have an open MML session on the telephony controller and are retrieving alarms continuously.

Although most alarms automatically clear after a problem is corrected, you need to retrieve and identify alarms on the telephony controller. You will also need to clear certain alarms.

The telephony controller stores alarm information in several places. Table 6-1 lists the telephony controller software's alarm information and descriptions.

6.2.1 Retrieving All Alarms

To retrieve all alarms, perform the following steps:

Step 1 Log in to the telephony controller and start an MML session. (See the "Starting MML" section for more information.)

Step 2 Enter the following MML command:

See the "Interpreting Alarm Messages" section for information on how to interpret these alarms.

6.2.1.1 Retrieving Alarms Continuously

If you retrieve alarms as they occur, this can help you determine how often a particular alarm is generated in a specific period of time. To retrieve alarms continuously, log in to the telephony controller and start an MML session. Enter the following command:

RTRV-ALMS::CONT

Alarms and autonomous messages are then continuously reported through the MML interface for the duration of the command.

6.2.2 Interpreting Alarm Messages

Figure 6-1 shows an example of an MML alarm response. The following subsections describe each of the message components.

Figure 6-1: MML Command Line

6.2.2.1 Component ID

This field identifies the system component that generated the alarm. The initial letters refer to the type of component, and the subsequent numbers refer to the timeslot for that element.

• LS---Indicates an SS7 link alarm.

• DC---Indicates a signal channel alarm.

• SC---Indicates a telephony controller alarm.

• SC1- NAS1---Indicates an alarm between telephony controller 1 and network access server 1.

• NAS---Indicates an alarm on the network access server.

• PC---Indicates a point code alarm.

• SP!-IP---Indicates an internet protocol alarm on the telephony controller.

The numbers after the initial letters identify the number of the component or the timeslot of the component. Table 6-2 shows the component descriptions for the alarm messages retrieved in the previous example, on page 6-3.

6.2.2.2 Alarm Category

The ALM identifier means that the messages listed are alarms. The entry after ALM= refers to the category of alarm. From the list retrieved in the previous example, we can identify the following categories of alarms.

6.2.2.3 Severity Level

This field provides the alarm severity level. Table 6-4 shows the four levels of alarm severity and the actions you need to take.

6.2.3 Acknowledging Alarms

If you acknowledge the last alarm of a certain severity (for example, the last Support Fail alarm on channel T-1-16), then the alarm relay associated with the Support Fail alarm relay is turned off. This action also turns off all related alarms in the central office. (In this case, all the Support Fail alarms would be turned off.)

Acknowledging an alarm removes it from an internal processes list maintained by the telephony controller, but does not clear the alarm. It will still be visible if a RTRV-ALM command is used.

To acknowledge an alarm, perform the following steps:

Step 1 Log in to the telephony controller and start an MML session. (See the "Starting MML" section for more information.)

Step 2 Enter the following MML command:

ACK-ALM:componentId:"alarmCategory"

For example, to acknowledge a Support Fail alarm on channel T-1-16, enter the following command:

ACK-ALM:T-1-16:"SUPPORT FAILED"

The system returns a message similar to the following:

CONFIG01/REL1 97-08-28 18:00:32
M COMPLD
;
 

This response displays the software version information (Configuration 01 of Release 1); the date and time (August 28, 1997, 32 seconds after 6:00 p.m.); and that the action was completed (the telephony controller processed the Support Fail alarm acknowledgment for channel T-1-16).

6.2.4 Clearing Alarms

The telephony controller produces platform alarms and signaling alarms. The following subsections describe these types of alarms and how to manage them.

6.2.4.1 Clearing Platform Alarms

Platform alarms are associated with telephony controller performance. This section provides information about the platform alarms with alarm causes and recovery actions.

Table 6-5 lists the platform alarms, alarm causes, recovery actions, and the page number for a flowchart illustrating how to resolve the alarm.

6.2.4.2 Clearing Signaling Alarms

The telephony controller software generates signaling alarms if it has problems transporting data on a signal channel. Threshold Crossing Alerts (TCAs) are caused when the number of errors exceeds the number of errors allowed in a specified time period or the threshold value. The threshold value is set in the thresholds.dat file in /opt/TransPath/etc.

Signaling alarms have three classifications of severity:

• Major

• Minor

• Information

Table 6-6 lists the platform level alarms, alarm causes, recovery actions, and the page number for a flowchart illustrating how to resolve the alarm.

6.3 Retrieving System Log Files

The telephony controller software creates system log files as shown in Table 6-7 and stores them in the /opt/TransPath/var/log directory.

Table 6-8 shows the log numbering system.

6.3.1 Retrieving a Log

To retrieve a log, perform the following steps:

Step 1 Log in to the telephony controller and change to the /opt/TransPath/var/log directory.

Step 2 Enter the ls command to list the available logs.

The system displays a list of the current and the historical logs. (Table 6-8 shows the log naming system.)

Step 3 To view a log, enter the following command:

cat logFileName.logNumber | more

---

Note Because the log files are very large, use the | more parameter to scroll through the file. You may prefer to print the file to find the information you need.

---

For example, to view the platform log from 5 days before yesterday, enter the following command:

cat SyslogPlatform.5 | more

The system returns a response similar to the following:

Mar  2 04:05:09 va-blue : | AlarmManager (PID 29974) {Platform} <Debug> wrote sU
Mar  2 04:05:09 va-blue : | AlarmManager (PID 29974) {Platform} <Info> #0 Operay
Mar  2 04:05:10 va-blue : | MeasurementManager (PID 29977) {Platform} <Debug> m3
Mar  2 04:05:10 va-blue : | MeasurementManager (PID 29977) {Platform} <Debug> m3
Mar  2 04:05:21 va-blue : | ProcessManager (PID 29973) {Platform} <Debug> Creat1
Mar  2 04:05:21 va-blue : | ProcessManager (PID 29973) {Platform} <Debug> Sendi1
Mar  2 04:05:21 va-blue : | ProcessManager (PID 29973) {Platform} <Debug> writi5
Mar  2 04:05:21 va-blue : | ConfigManager (PID 29975) {Platform} <Debug> readin6
Mar  2 04:05:21 va-blue : | ConfigManager (PID 29975) {Platform} <Debug> read i5
Mar  2 04:05:21 va-blue : | ConfigManager (PID 29975) {Platform} <Debug> procEv.
Mar  2 04:05:21 va-blue : | ConfigManager (PID 29975) {Platform} <Debug> writin7
--More--

6.3.2 Changing the Log Level for Processes

The dumper can potentially generate a large number of files to the logSpoolDir area. For example, if the maxTime value were set to 15 minutes, over 200 files would be created in the logSpoolDir daily.

Therefore, you may want to limit the number of logs being created. One method of limiting the logs is to change the logging level or threshold value of the processes you want to limit. The CHG-LOG command sets the logging level for active processes only. This command can be invoked for individual active processes or for all processes. When the ALL parameter is used, a message appears that states that the log level did not change for some processes. This is because the changes are not applied to monitored and passive processes.

See the "CHG-LOGChange Log" section for a more detailed explanation of the elements of this process.

Caution Running in debug mode causes the Syslog.Platform files to become very large. These files are frequently deleted by the automatic system process. Cisco does not recommend logging to another destination (such as Platform_1.log) while running in debug mode, because the telephony controller will not automatically delete these files. The hard disk will become full and cause system crashes.

To change the log level of a single process, perform the following steps:

Step 1 Log in to the telephony controller and change to any subdirectory under /opt/TransPath (for example, /opt/Transpath/bin).

Step 2 Start an MML session as described in the "Starting MML" section.

Step 3 Enter the following MML command:

CHG-LOG:processName:logLevel

For example, to change the log level of the engine, enter the following command:

CHG-LOG:ENG-01:INF

The system returns a message similar to the following:

 CONFIG01/BUILD 13.6 97-05-15 16:35:29
M COMPLD
 ;
 

See the"CHG-LOGChange Log" section for valid log levels and their meanings.

To change the log level of a all processes, perform the following steps:

Step 1 Log in to the telephony controller and start MML.

Step 2 Enter the following MML command:

CHG-LOG:ALL:logLevel

For example, to change the log level of all processes to "warning," enter the following command:

CHG-LOG:ALL:WRN

The system returns a message similar to the following:

 CONFIG01/BUILD 13.6 97-05-15 16:37:43
M COMPLD
 SNSP
 "DSKM-01:Operation not supported"
 /* Status, Operation Not Supported By Component */
 SNSP
 "IOCC-01:Operation not supported"
 /* Status, Operation Not Supported By Component */
 SNSP
 "IOCC-02:Operation not supported"
 /* Status, Operation Not Supported By Component */
 ;
 

This response indicates that the DSKM-01, IOCC_01, and IOCC-02 processes will not allow their log levels to be changed. See the"CHG-LOGChange Log" section for valid log levels and their meanings.

6.4 Performing a Call Trace

A call trace records information in a trace file that shows how the telephony controller processes a call. If you are experiencing problems with call processing and need to contact Cisco for support, you should run a call trace before contacting Cisco's Technical Assistance Center (TAC). You probably won't be able to interpret the trace file, but the Cisco TAC can and this will help them troubleshoot the problem more effectively. For some problems, the TAC will not begin troubleshooting until they receive the trace file, so it is a good practice to create this file before contacting the Cisco TAC.

The following steps present an overview of how to create a trace file:

Step 1 Set the telephony controller software log to debug level.

Step 2 Use MML to run the trace and create a log file.

Step 3 Run the log file through the conversion analyzer and simulator programs to create a SIM file.

The following sections describe how to complete each of the above steps.

6.4.1 Setting the Debug Level

Step 1 Log in to the telephony controller and change to the /opt/TransPath/etc directory.

Step 2 Use a text editor to make the following changes to the XECfgParm.dat file:

• Set the ioChanCtl.logPrio parameter to Debug.

• Set the ProcM.logPrio parameter to Debug.

• Set the engine.logPrio parameter to Debug.

Step 3 Confirm that the logFileFmt parameter is set to ../var/platform_%s.log.

Caution Running in debug mode causes the Syslog.Platform files to become very large. These files are frequently deleted by the automatic system process. Cisco does not recommend logging to another destination (such as Platform_1.log) while running in debug mode, because the telephony controller will not automatically delete these files. The hard disk will become full and cause system crashes. If you use a log other than Syslog.Platform, you need to monitor the size of the files and delete them if the disk is becoming full.

6.4.2 Running the Trace

Step 1 Log in to the telephony controller as a user in the transpath group and change to any subdirectory under /opt/TransPath (for example, /opt/TransPath/bin).

Step 2 Start an MML session as described in the "Starting MML" section.

Step 3 Enter the following MML command:

STA-SC-TRC:sc:log="filename", prd=600

The sc parameter identifies the originating signal channel. For example, if this is an SS7 signaling path, enter the point code. If this is an ISDN connection, enter the name of the IP signal path.

The filename parameter specifies the file to which the trace is captured. If you do not specify a filename, the telephony controller software creates a file using the same name as the signal channel with a .log extension. Ensure that this file does not already exist; if it does, the trace will not run correctly and you will receive an error message.

The prd parameter specifies the trace period in seconds. If you do not specify a time limit, the software continues the trace until an 8 KB boundary is reached or until you manually stop the trace. For example, if you entered STA-SC-TRC: sc:log="filename", prd=600, the telephony controller starts a trace on the signal channel specified, saves the trace to the filename specified, and collects data for 600 seconds (10 minutes).

Step 4 Make the call you want to trace.

Step 5 End the call, and enter the following command to end the trace:

STP-SC-TRC:ALL

Step 6 Quit MML.

6.4.3 Creating the SIM file

Step 1 Change to the opt/TransPath/var/trace directory. This directory contains the filename you specified when you ran the trace.

Step 2 Enter the following command:

ca -f "filename"

You may want to add these parameters:

• | more to view the file in sections.

• /tmp/ca.out to direct the output to another file.

Step 3 Look in the header to locate the call identification number. You need this number to create the SIM file.

Step 4 Change to the/opt/TransPath/lib directory.

Step 5 Enter the following command:

sim -f /opt/TransPath/var/trace/"filename" -y callIdentificationNumber -o "orig.mdo" -t "term.mdo" [-m cc.mdo -l lcm.mdo] -d /opt/TransPath/var/trace/filename.sim

The callIdentificationNumber is the number in the trace header. The first filename parameter specifies the original trace filename; the second filename parameter is the name of the new SIM file you are creating.

The orig.mdo parameter defines the variant for the originating protocol (for example, ANSISS7.mdo). The term.mdo parameter defines the variant for the terminating protocol (for example, Bell_1268_C3.mdo).

Step 6 Compress the SIM file using the UNIX tar command or zip the file and forward to the TAC, who will use it to troubleshoot your system.

6.5 Contacting the TAC

The focal point of all Cisco software and hardware maintenance and support services is the Cisco Technical Assistance Center (TAC). The TAC is staffed by senior customer engineers who have experience with the Cisco product line and all aspects of data communications networking technology. The TAC provides worldwide technical support.

6.5.1 Case Classifications

TimeSaver Do not rely on electronic mail to submit Priority 1 or Priority 2 problems to Cisco. Telephone your servicing technical support organization.

6.5.2 How to Open a Case

Have your maintenance contract number for the system that needs service and the system's serial number ready. Be prepared to give a brief description of the problem so the dispatcher can write a report and dispatch it to the appropriate CERT (Customer Engineering Response Team). It is also a good idea to tell the dispatcher the priority level of the problem (see the "Case Classifications" section). You will either be connected directly to an available CE or a CE will call you back shortly.

There are three ways to open a case with the TAC: online, by phone, or by e-mail.

1. Open a case online at www.cisco.com.

2. Call TAC phone number indicated in Table 6-11.

3. Open a case by sending e-mail to tac@cisco.com.

6.5.3 Customer Preparation

Step 1 If you suspect a problem, call or e-mail the TAC and a case will be opened.

Step 2 Assign a case classification based on the criteria provided in the "Case Classifications" section.

Step 3 If you classify the case as P1 or P2 and send it via e-mail or the Internet, call the TAC to confirm the problem. The phone number is 800 553-2447 for U.S. customers or 408 526-7209 for international customers.

Step 4 A copy of the case is accessible to the customer at www-tac.cisco.com.

6.5.4 Getting Help with Your System

You have 24-hour support for your SS7 dial access solution via Cisco's Technical Assistance Center (TACs). There are 4 TACs worldwide. To initiate a case, contact the closest TAC and tell them your problem. You will be issued a case number that you can check by phone or on the Web.

The Cisco TAC offers support in multiple languages during business hours and in English after business hours. You can send e-mail to the listed address and receive answers in the languages indicated in Table 6-12.

You can also initiate your case online via the Internet at www.cisco.com. Outside the locations listed in Table 6-11 and Table 6-12, contact the Cisco regional sales office nearest you or contact your local authorized Cisco distributor.

6.5.5 TAC Case Initiation

1. TAC opens the case based on information received from the customer and assigns a case number.

2. The case is forwarded to the next support level for action and the case number is communicated to the originator.

3. TAC support reviews the case and ensures that the classification assigned by the customer is appropriate and that the case contains the data and information required to analyze the problem.

6.5.6 Response Times

Based on the case classification, Cisco Systems is committed to the following initial customer response times and problem resolution times. A workaround that returns the customer to operation is considered acceptable as an interim problem resolution.

6.5.7 TAC Case Resolution

The following procedure summarizes how the TAC resolves each case.

1. When a solution has been developed, TAC closes the case and records the elapsed time to measure compliance with the commitments detailed in the "Response Times" section .

2. TAC contacts the customer and informs them of the solution and arranges the appropriate method of transmittal.

3. In most cases, the problem solution is transmitted to the customer using an encrypted data link. If the size of the solution prohibits direct transmission, the solution is sent by the next available flight or Federal Express overnight priority service.

4. Customer confirms with the TAC the receipt and installation of the solution. They also confirm that the solution resolves the problem described on the case.

5. If the problem is not resolved to the customer's satisfaction, the case is reopened by the TAC. The case is then forwarded to the TAC support and the manager of operations is notified.

Posted: Fri Nov 19 13:59:53 PST 1999
Copyright 1989-1999©Cisco Systems Inc.