|
|
The preceding chapters of this guide described maintenance of the various devices in your Cisco Media Gateway Controller (MGC) node. This chapter describes troubleshooting for the Cisco MGC. It includes the following sections to help you isolate system problems:
The Cisco MGC is a Sun Netra UNIX host running Cisco MGC Software Release 7. The Cisco MGC performs real-time call-processing and SS7 layer functions; manages trunk resources, alarms, and call routing; and administers billing information.
Cisco MGC functionality includes
Sun Netra UNIX hosts serve as the platform for the Cisco MGC Software Release 7. They provide full functional redundancy and are designed to recover from any hardware failure within one second with no data loss. In addition, the Sun Netra hosts meet and exceed Network Equipment Building System (NEBS) Level 3 standards.
Using two Cisco MGCs in a continuos service architecture provides system redundancy and reliability. The call-processing application is active on one Cisco MGC and switches to the standby Cisco MGC only under failure conditions.
Each Ethernet NIC for each Sun Netra system is connected using 100BaseT to the LAN switches. The LAN switches connect to the Cisco Signaling Link Terminals (SLTs) using 10BaseT. Together, these three elements form the Cisco MGC node. Figure 8-1 displays the connections between the elements of the Cisco MGC node.
The LEDs and other indicators on the Sun hosts provide some basic information about the status of the system. For more information about the LEDs on the Sun host platforms, refer to Chapter 5 "Maintaining the Cisco MGC."
This chapter uses the alarms and events that appear at the Cisco MGC as the basis for isolating problems within the system. Table 2-2 in the Cisco Media Gateway Controller Software Release 7 Reference Guide lists the Cisco MGC alarm and event categories, provides brief descriptions, lists possible causes, and suggests corrective actions.
Typically, suggested corrective actions start with simple troubleshooting tasks and become increasingly more complex. It is easier, for example, to check LEDs and cabling than to perform a call trace. The suggested corrective actions point to other chapters in this manual, as well as to troubleshooting tools including the Cisco MGC software Release 7, the Cisco WAN Manager, and CiscoWorks.
For information on using CiscoWorks for Catalyst 5500 problems, refer to "Troubleshooting MSR Signaling."
Additionally, you will find examples of troubleshooting typical problems. The examples provide a logical sequence for troubleshooting that you can use as a model.
The following sections contain various equipment failure scenarios for the solution, including
Each Cisco SLT has an Reliable User Datagram Protocol (RUDP)/User Datagram Protocol (UDP)/IP connection to each Cisco MGC for the transfer of MTP3, ISDN User Part (ISUP), and Transaction Capabilities Application Part (TCAP) information. A Cisco SLT platform failure results in the surviving Cisco SLT platforms taking over the distribution of messages to the active Cisco MGC. Cisco SLT platforms should be provisioned so that half of the platforms can support the entire signaling load. The result is that a Cisco SLT platform failure has no significant effect on call processing.
There are several Cisco SLT failure scenarios to consider:
Cisco MGC hosts run in active-standby mode. The call-processing application is active on only one Cisco MGC platform at a time, and the application switches to the standby platform when a critical alarm occurs. The result is that Cisco MGC failure and switchover events are invisible to the SS7 signaling network.
Cisco MGC alarms can be configured as minor, major, or critical. Critical alarms are generated whenever any significant failure occurs. Any critical alarm causes a switchover to occur. For example, if the call engine or I/O channel controller (IOCC)-MTP in the active Cisco MGC should fail, there is a disconnection from the process manager (procM) and a switchover to the standby Cisco MGC.
An operating system (OS) or hardware failure in the active Cisco MGC can also cause a switchover to the standby Cisco MGC. The failover daemon (foverd) in the standby Cisco MGC detects the failure of the active Cisco MGC and instructs procM to initiate a switchover. The standby Cisco MGC then takes over all call-processing functions. The switchover is transparent to all the Cisco SLTs.
The Cisco MGC generates alarms to indicate problems with processes, routes, linksets, signaling links, and bearer channels. The Cisco Media Gateway Controller Software Release 7 Reference Guide lists all of the Cisco MGC alarms and events, and provides descriptions, possible causes, and suggested actions.
Alarm log files reside in /opt/CiscoMGC/var/log. After a specified period, the alarm files become historical logs and reside at /opt/CiscoMGC/var/spool.
The following subsections describe each of the message components for the typical MML alarm response shown below:
"LPC-01: 2000-02-26 09:16:07.806," "LPC-01:ALM=\"SCMGC MTP3 COMM FAIL\",SEV=MJ" "IOCM-01: 2000-02-26 09:17:00.690," "IOCM-01:ALM=\"Config Fail\",SEV=MN" "MGC1alink2: 2000-02-26 09:17:47.224,ALM=\"SC FAIL\",SEV=MJ" "MGC1alink3: 2000-02-26 09:17:47.225,ALM=\"SC FAIL\",SEV=MJ"
The first element of the alarm message identifies the system component that generated the alarm, using the customer-defined description of the component given during system configuration. In our example, these are LPC-01, IOCM-01, MGC1alink2, and MGC1alink3.
All system components are described in the Cisco Media Gateway Controller Software Release 7 Reference Guide.
The second element of the alarm message identifies the time of the alarm by year, month, day, hour, minute, hundredths, and thousandths of a second (milliseconds).
The third element of the alarm message identifies the alarm category. It indicates the MML description of the alarm/event. In our example:
The last element of the alarm message identifies the severity level of the alarm. The four levels are
 |
Note Critical alarms cause the system to switchover. |
To retrieve all active alarms, complete the following steps:
Step 2 Enter the following command:
rtrv-alms
MML returns a response that shows all active alarms:
Media Gateway Controller 2000-02-26 11:41:01 M RTRV "LPC-01: 2000-02-26 09:16:07.806," "LPC-01:ALM=\"SCMGC MTP3 COMM FAIL\",SEV=MJ" "IOCM-01: 2000-02-26 09:17:00.690," "IOCM-01:ALM=\"Config Fail\",SEV=MN" "MGC1alink2: 2000-02-26 09:17:47.224,ALM=\"SC FAIL\",SEV=MJ" "MGC1alink3: 2000-02-26 09:17:47.225,ALM=\"SC FAIL\",SEV=MJ" "MGC1alink4: 2000-02-26 09:17:47.226,ALM=\"SC FAIL\",SEV=MJ" "MGC2alink1: 2000-02-26 09:17:47.227,ALM=\"SC FAIL\",SEV=MJ" "MGC2alink2: 2000-02-26 09:17:47.227,ALM=\"SC FAIL\",SEV=MJ" "MGC2alink4: 2000-02-26 09:17:47.229,ALM=\"SC FAIL\",SEV=MJ" "dpc5: 2000-02-26 09:17:47.271,ALM=\"PC UNAVAIL\",SEV=MJ" "ls3link1: 2000-02-26 09:16:28.174," "ls3link1:ALM=\"Config Fail\",SEV=MN" "ls3link1: 2000-02-26 09:18:59.844,ALM=\"SC FAIL\",SEV=MJ"
If you retrieve alarms continuously, you can determine how often a particular alarm occurs in a specific period of time. To retrieve alarms continuously, log in to the active Cisco MGC, start an MML session, and enter the following command:
rtrv-alms::cont
Alarms and autonomous messages are then continuously reported through the MML interface.
Acknowledging an alarm does not clear the alarm. You can still retrieve it with the rtrv-alm command.
To acknowledge an alarm, perform the following steps:
ack-alm:componentId:"alarmCategory"
Step 2 For example, to acknowledge a signal channel fail alarm on MGC2alink1, enter the command:
ack-alm:MGC2alink1:"SC FAIL"
The system returns a message similar to the following:
Cisco MGC 2000-02-26 11:42:30 M COMPLD ;
This response displays the software version information (Cisco MGC), and the date and time (2000-02-26 11:42:30). It also indicates that the action was completed (COMPLD).
You can clear an alarm category for a component. Clearing the alarm category removes it and any associated alarms from the internal processes list maintained by the Cisco MGC.
clr-alm: componentId:"alarmCategory"
Step 2 The system returns a message similar to the following:
Cisco MGC 2000-02-26 11:45:14 M COMPLD ;
This response displays the software version information (Cisco MGC), and the date and time (2000-02-26 11:42:30). It also indicates that the action was completed (COMPLD).
This section lists the possible causes for specific problems and recommends methods for troubleshooting.
The Cisco MGC node is considered to be a standard Service Switching Point (SSP) in an SS7 network. The SS7 network carries two types of signals:
The signals involved in the setup and teardown of bearer circuits are circuit-related. Non-circuit-related signals are used for all the ancillary services provided by the SS7 network, including database access and network management.
The SS7 protocol is composed of several levels or "parts," including the following:
There are many variations of different parts of the SS7 protocol stack. MTP has ANSI, ITU, Bellcore, and a number of national variations. Each country and each major carrier may have slightly different variations of a part to fit its particular needs.
The SS7 network needs to have the highest degree of reliability. Each switch with access to the SS7 network must be configured to a preconceived set of network parameters. There is some risk that the person configuring a switch will not use the correct set of parameters or values. This is the root cause of most SS7 problems at both the MTP layers and upper layers of the SS7 protocol. A single parameter value, such as an incorrect timer value, can cause SS7 connectivity to act improperly or fail completely.
The first, and most important, step in troubleshooting SS7 related problems is to understand, and fully document, the SS7 network topology and protocols. The protocol documents are used as a reference over the months and years of maintenance on the SS7 network.
The Cisco MGC software generates signaling alarms if it detects problems with the transportation of data on a signal channel or at a signaling destination.
Threshold Crossing Alerts (TCAs) occur when the number of errors exceeds the number of errors allowed in a specified time period. This is the threshold value and is set in the thresholds.dat file in the /opt/CiscoMGC/etc subdirectory.
 |
Caution The values in the thresholds.dat file are preset by Cisco and should not be modified. Changing the values in the thresholds.dat file could degrade Cisco MGC performance. |
Signaling alarms have three classifications of severity:
|
Note Multiple alarms are likely to occur for severe failures. For example, SUPPORT FAIL and SC FAIL would typically occur with LIF LOS. |
Signaling links are the dedicated communication channels that the Cisco MGC uses to transfer signaling information among itself, the Cisco SLTs, and the Signal Transfer Points (STPs). Signaling links provide the necessary delivery reliability for higher-layer SS7 signaling protocols.
You can use the Cisco MGC software and MML commands to manage signaling channels and lines. You can retrieve signaling channel attributes, change the states of signaling channels, and change the state of signaling lines. See "Operating Procedures," for detailed information.
|
Note For more information on MML commands, refer to the Cisco Media Gateway Controller Software Release 7 Reference Guide. |
Because all types of signaling channels have basically the same functionality, they are managed similarly. Unless otherwise noted, all commands, counters, and alarms mentioned here are applicable to all types of signaling channels.
Service states indicate link status. There are two levels of signaling channel service states:
| Link State ID | Link State | Description |
|---|---|---|
INB | Install busy | When a system is first configured, all links default to this state and must be manually set in-service (IS) through the use of the set signaling channel state MML command. |
IS | In-service | The signaling channel is IS and fully operational. This is the normal state of a signaling link. |
OOS | Out-of-Service | The signaling link is out-of-service (OOS) from the remote end. The system is actively trying to restore the link. |
| Link State ID | Link State | Description |
|---|---|---|
SUPPENT | Supporting entity failure | A signaling link is out of service, because a supporting entity has failed. For example, LIF LOS, EQPY FAIL. This condition can generate a "SUPPORT FAIL" alarm. |
INH | Inhibit | In the INH state, the signaling channel is active but does not provide service for call processing. If the signaling channel is the last one in the signal path, the inhibit request is denied and an error is returned. This is for SS7 signaling channels only and fails on other types of signaling channels. |
UNH | Un-inhibit | The UNH state is used to return an inhibited signaling channel to service, rather than the IS option. This is for SS7 signaling channels only and fails on other types of signaling channels. |
PEND | Pending | Used in combination with OOS. OOSPEND indicates that the signaling link has been requested to stop providing service after all active calls are terminated. This state allows the calls to complete, but blocks any new calls. |
FOOS | Forced out-of-service | Forced out-of-service locally. The channel remains in this state until manually set into service. |
CONF | Configuration | Link is unavailable due to a configuration error. Typically, further diagnosis is required regarding the configuration of the failed link. |
The major issues with the physical layer of an SS7 signaling link are related to cabling, clock source, and connector pinouts. The cable should be of high quality (shielded) and the connectors should be attached and crimped solidly. Since SS7 links are synchronous, one side of the link must provide the clock source and the other side must use this clock signal to read the bits.
Finally, the most common mistake is to use the wrong cable pinouts for a specific physical configuration. Make sure that the connector has the correct number of pins (RJ-45, DB-25) and that each pin maps to the correct signal. A number of different physical layers are supported, including ANSI T1, CEPT E1, and V.35. Make sure that the cable complies with the connector and the physical protocol being used.
If the configuration appears to be valid and the cable pinout is good, check that the signal is being sent and received correctly. Use a Bit Error Rate Tester (BERT) or perform a signal loopback on the interface. It is possible that the cable is bad, so try to replace it. Finally, it is possible that the line card is bad, so you might try replacing it too.
The most common mistake in SS7 signal link configuration is to misconfigure the Signal Link Code (SLC) for the SS7 link. This is a preconfigured code on both ends of the link. If the SLC or the point codes do not match, the link does not align and no transmission can take place.
For T1 and E1 connectors, an SS7 signaling link is carried in a single 56- or 64-kbps time slot. The time slot that is used must also agree on both sides of the link.
Make sure the MTP2 timers and thresholds agree with the network defaults. Confirm that the far-end switch or STP has the same values as your system.
When a Cisco SLT is used to terminate MTP2, confirm that the RUDP parameters agree on both sides and are consistent with the documentation.
An SS7 signaling link has a hierarchy of network element entities that must be functioning before the link can function. These include the physical interface (discussed above) and the control software for the link. If any of these fail, the link also fails.
Link failures between the Cisco SLT and the Cisco MGC can be caused by
In each of the above cases, it is impossible to transfer MTP3 signaling messages from the Cisco SLT to the Cisco MGC. Cisco SLT platform failure (which is equivalent to MTP2 failure) causes signaling messages to be unable to go to MTP3. The MTP2 layer on the Cisco SLT is supposed to transmit SIPO messages to the STP mated pair to initiate the changeover procedure. Cisco SLT platform failure on the SS7 network is detected by the mated STP pair, which detects timer expiration and link unavailability.
Signaling destinations refer to the endpoints of a network. Typically, if signaling links are in service, the signaling destinations should also be in service.
For ISDN signaling, the signaling channel is in service if the Cisco MGC can talk to the media gateway and ISDN backhaul is configured. The destination is in service if the signaling channel is in service and the remote ISDN device is up.
Apparent mismatches can occur due to
An SS7 STP is treated as an adjacent point code (APC) to the Cisco MGC. SS7 MTP uses a message exchange called Signaling Link Test Message (SLTM)/Signaling Link Test Acknowledgment (SLTA) to confirm that the far-end point code is the one configured. The SLTM consists of the originating point code (OPC) of the Cisco MGC, an APC number, and an SS7 network indicator. If the values for these parameters match with the values used for these at the far-end switch, an SLTA is returned. If the value for any of these parameters do not match, the far-end switch does not send an SLTA. The Cisco MGC drops the link and tries to realign it. This process continues until the SLTM parameters match on both sides. The problem is manifested by the SS7 links dropping and recovering in roughly 30-second cycles.
Make sure that the MTP3 route timers and thresholds agree with the network defaults. Confirm that the far-end switch or STP has the same values as your Cisco MGCs.
If the SS7 destination point code (DPC) is fully associated, it can have the same SLTM/SLTA problems as described above.
If the SS7 DPC is quasi-associated, the most common cause for failure is a route misconfiguration. Review the route information between the Cisco MGC and the DPC to make sure that the APCs are valid, the route priorities are set correctly, and the route uses the appropriate linkset.
The destination service state is queried from and maintained in the engine process (ENG-01), not the I/O channel manager (IOCM).
To retrieve information about all external point codes and signal paths, complete the following steps:
Step 2 Enter the following command:
rtrv-dest:all
For Release 7.3 of the MGC software, the system returns a message similar to the following:
mml> rtrv-dest:all Media Gateway Controller - MGC-01 2000-04-05 08:30:11 M RTRV "dpc1:PKG=SS7-ANSI,ASSOC=UNK,PST=IS,SST=UNK" "dpc2:PKG=SS7-ANSI,ASSOC=UNK,PST=IS,SST=UNK" "dpc3:PKG=SS7-ANSI,ASSOC=UNK,PST=OOS,SST=UNK" "dpc4:PKG=SS7-ANSI,ASSOC=UNK,PST=OOS,SST=UNK" "dpc5:PKG=SS7-ANSI,ASSOC=UNK,PST=OOS,SST=UNK" "dpc6:PKG=SS7-ANSI,ASSOC=UNK,PST=IS,SST=UNK" "dpc7:PKG=SS7-ANSI,ASSOC=UNK,PST=IS,SST=UNK" "dpc8:PKG=SS7-ANSI,ASSOC=UNK,PST=IS,SST=UNK"
This response displays the software version information (Cisco MGC); the date and time (2000-02-26 11:29:56); the DPCs; the signaling protocol used; and the status of the destination (PST=IS or PST=OOS).
For Release 7.4 and up of the MGC software, the system returns a message similar to the following:
mml> rtrv-dest:all Media Gateway Controller - MGC-04 2000-04-05 08:05:36 M RTRV "dpc1:PKG=SS7-ANSI,ASSOC=SWITCHED,PST=IS,SST=UND" "dpc2:PKG=SS7-ANSI,ASSOC=SWITCHED,PST=IS,SST=UND" "dpc3:PKG=SS7-ANSI,ASSOC=SWITCHED,PST=IS,SST=UND" "dpc4:PKG=SS7-ANSI,ASSOC=SWITCHED,PST=IS,SST=UND" "dpc5:PKG=SS7-ANSI,ASSOC=SWITCHED,PST=IS,SST=UND" "dpc6:PKG=SS7-ANSI,ASSOC=SWITCHED,PST=IS,SST=UND" "eisupftvsc:PKG=EISUP,ASSOC=SWITCHED,PST=OOS,SST=UND" "eisupsvc1:PKG=EISUP,ASSOC=SWITCHED,PST=OOS,SST=UND"
This response displays the software version information (Cisco MGC); the date and time (2000-02-26 11:29:56); the DPCs; the signaling protocol used; and the status of the destination (PST=IS or PST=OOS).
|
Note If this command is entered after a switchover has occurred, the state of some of the destinations might be listed as undefined (UND). UND is the default state for a destination when the system starts. In this instance, UNDs indicate that the Cisco MGC has not received a service state message from the associated destination since the switchover occurred. No user action is required. |
To retrieve information about a specific DPC, log in to the active Cisco MGC, start an MML session, and enter the following command:
rtrv-dest: point_code
Where: point_code is the DPC.
The system returns a message that displays the software version information; the date and time; an indication of whether the action was completed; and the status of the specified DPC.
To retrieve information about a specific signal path, log in to the active Cisco MGC, start an MML session, and enter the following command:
rtrv-dest: sig_path
Where: sig_path is the signaling path number.
The system returns a message that displays the software version information; the date and time; an indication of whether the action was completed; and the status of the specified signaling path.
|
Note If this command is entered after a switchover has occurred, the state of some of the destinations may be listed as undefined (UND). UND is the default state for a destination when the system starts. In this instance, UNDs indicate that the Cisco MGC has not received a service state message from the associated destination since the switchover occurred. No user action is required. |
Make sure that the MTP3 traffic restart timers and thresholds agree with the network defaults. Confirm that the far-end switch or STP also has the same values.
When resolving signaling problems between the Cisco MGC and an associated SS7 network element (such as an STP), you may need to verify that the MTP2 and MTP3 timer settings used by the Cisco MGC conform to settings used by the associated SS7 network element. MML commands are used to retrieve the settings for the MTP2 and MTP3 timers on the Cisco MGC. The following subsections describe methods for verifying the MTP timer settings on the Cisco MGC.
|
Note Refer to the Cisco MGC Software Release 7 Provisioning Guide for more information on the MTP timers. |
The procedure used to verify the settings for MTP2 timers varies based on how SS7 signaling is terminated for your Cisco MGC. If you are using Cisco SLTs to terminate SS7 signaling, refer to the "Verifying MTP2 Timers for Cisco SLTs" section. If you are using I/O cards to terminate SS7 signaling, refer to the "Verifying MTP2 Timers for I/O Cards" section. The procedure to verify MTP3 timers is the same for both SS7 signaling termination methods.
If you find, after you verify the settings, that you need to modify the settings for the MTP timers, proceed to the "Modifying MTP Timer Settings" section.
To verify the values used for the MTP2 timers when you are using Cisco SLTs to terminate SS7 signaling, complete the following steps:
Router #show SS7 mtp2 timer channel
Where: channel specifies a channel, 0 through 3.
The system returns a message similar to the following:
SS7 MTP2 Timers for channel 0 in milliseconds
Protocol version for channel 0 is Japan NTT Q.703 Version 1-1
T1 aligned/ready = 15000
T2 not aligned = 5000
T3 aligned = 3000
T4 Emergency Proving = 3000
T4 Normal Proving = 3000
T5 sending SIB = 200
T6 remote cong = 3000
T7 excess ack delay = 2000
T8 errored int mon = 0
TA SIE timer = 20
TF FISU timer = 20
TO SIO timer = 20
TS SIOS timer = 20
Step 2 Verify the MTP2 timers settings listed for the Cisco SLTs against the MTP2 timers used at the associated destination.
If the MTP2 timers settings match, your signaling problem has different cause. Continue troubleshooting the problem.
If the MTP2 timers settings do not match, perform the procedure in the "Modifying MTP2 Timers for Cisco SLTs" section.
To verify the values used for the MTP2 timers when you are using I/O cards to terminate SS7 signaling, complete the following steps:
|
Note If you use this command to verify the settings for the MTP2 timers when you are using Cisco SLTs to terminate SS7 signaling links, the displayed results reflect the default values for the SS7 variant assigned to the linkset, not the actual values used. Refer to the "Verifying MTP2 Timers for Cisco SLTs" section for the procedure to obtain the actual settings used for these MTP2 timers. |
prov-rtrv:component:name="MML_name"
Where:
The system returns a message similar to the following:
vsc-01 - Media Gateway Controller 2000-07-27 18:33:56
M RTRV
"session=nsite04:sigsvcprop"
/*
mtp2AermEmgThr = 1 mtp2AermNrmThr = 4 mtp2CongDiscard = false mtp2LssuLen = 1 mtp2MaxAlignRetries = 5 mtp2MaxMsuFrmLen = 272 mtp2MaxOutsFrames = 127 mtp2ProvingEmgT4 = 6 mtp2ProvingNormalT4 = 23 mtp2SuermThr = 64 mtp2T1 = 450 mtp2T2 = 250 mtp2T3 = 20 mtp2T5 = 1 mtp2T6 = 60 mtp2T7 = 10
Step 2 Verify the MTP2 timers settings listed for the I/O cards against the MTP2 timers used at the associated destination.
If the MTP2 timers settings match, your signaling problem has different cause. Continue troubleshooting the problem.
If the MTP2 timers settings do not match, perform the procedure in the "Modifying MTP2 Timers for I/O Cards" section.
To verify the values used for the MTP3 timers, complete the following steps:
prov-rtrv:component:name="MML_name"
Where:
The system returns a message similar to the following:
vsc-01 - Media Gateway Controller 2000-07-27 18:33:56
M RTRV
"session=nsite04:sigsvcprop"
/*
mtp3ApcMtpRstrtT28 = 50
mtp3DlnkConnAckT7 = 10
mtp3FrcUnhT13 = 10
mtp3InhAckT14 = 20
mtp3LocInhTstT20 = 900
mtp3MaxSltTries = 2
mtp3MsgPriority = 2
mtp3MtpRstrtT24 = 60
mtp3RepeatRstrtT26 = 150
mtp3TfrUsed = false
mtp3TraSnT29 = 600
mtp3tstSltmT1 = 60
mtp3tstSltmT2 = 600
mtp3UnhAckTl2 = 10
reference = ANSI96
Step 2 Verify the MTP3 timers settings listed against the MTP3 timers used at the associated destination.
If the MTP3 timers settings match, your signaling problem has different cause. Continue troubleshooting the problem.
If the MTP3 timers settings do not match, perform the procedure in the "Modifying MTP3 Timers" section.
When resolving signaling problems between the Cisco MGC and an associated SS7 network element (such as an STP), you may need to modify the MTP2 and MTP3 timer settings on the Cisco MGC, so that they conform to the settings used by that SS7 network element. You use MML commands to modify the settings for the MTP2 and MTP3 timers. The following subsections describe methods for modifying the settings of the MTP timers on the Cisco MGC.
The procedure used to modify the settings for MTP2 timers varies based on how SS7 signaling is terminated for your Cisco MGC. If you are using Cisco SLTs to terminate SS7 signaling, refer to the "Modifying MTP2 Timers for Cisco SLTs" section. If you are using I/O cards to terminate SS7 signaling, refer to the "Modifying MTP2 Timers for I/O Cards" section. The procedure for modifying MTP3 timers is the same for both SS7 signaling termination methods.
|
Note Refer to the Cisco MGC Software Release 7 Provisioning Guide for more information on the MTP timers. |
You might want to verify the new settings after the modification is complete. To do this, refer to the procedure in the "Verifying MTP Timer Settings" section.
Use the following MML command at the Cisco SLT to modify the settings for the MTP2 timers:
Router (config)#ss7 mtp-variant standard channel parameters
Where:
|
Note Refer to the Cisco Signaling Link Terminal Guide for more information on the parameters for this command. |
In the following example, the aligned/ready timer duration on channel 0 is set to 30,000 milliseconds:
Router(config)# ss7 mtp2-variant Bellcore 0 Router(config-Bellcore)# T1 30000
In the following example, the aligned/ready timer is restored to its default value of 13,000 milliseconds:
Router(config)# ss7 mtp2-variant Bellcore 0 Router(config-Bellcore)# no T1
Use the following MML command to modify the settings for the MTP2 timers when you are using I/O cards to provide SS7 signaling links:
prov-ed:component:name="MML_name",param_name=param_value
Where:
|
Note Refer to the Cisco MGC Software Release 7 Provisioning Guide for more information on the parameters for this command. |
In the following example, the MTP2 T2 timer, maximum period in a Not Aligned state before returning to an OOS state, is set to 120 tenths of a second:
prov-ed:sigvcprop:name="ls-sunb7-1",mtp2T2=120
Use the following MML command to modify the settings for the MTP3 timers.
prov-ed:component:name="MML_name",param_name=param_value
Where:
|
Note Refer to the Cisco MGC Software Release 7 Provisioning Guide for more information on the parameters for this command. |
In the following example, the MTP3 T1 timer, waiting for signaling link test acknowledgment message, is set to 65 tenths of a second:
prov-ed:sigvcprop:name="ls-sunb7-1",mtp3tstSltmT1=65
Bearer channels are the focus of everything that the Cisco MGC does. The main function of the
Cisco MGC is to ensure that an ingress bearer channel at one endpoint can be successfully connected to an egress bearer channel at another endpoint.
The state of the bearer channels is often a good indicator of the overall health of the system.
Destinations and bearer channels are fundamentally associated; hence, the state of each channel is maintained by the call engine.
The paragraphs that follow detail methods of identifying problems with your bearer connections.
Use the rtrv-tc and rtrv-cic commands to determine the state of bearer channels, when the configuration is small. The report is limited to a maximum of 24 (T1) or 31 (E1) bearer channels for any single destination.
Key indicators in the output from the rtrv-tc and rtrv-cic commands are described in the paragraphs that follow.
Each DS0 bearer channel is identified by a point code, a network indicator, and a CIC.
The PST field identifies the primary service state of a bearer channel. The current state of a bearer channel is either IS or OOS. Individual bearer channels can be OOS even if the destination is IS, due to signaling events such as Q.931 service messages. OOS bearer channels are not available for calls.
The SST field identifies the secondary service state of a bearer channel. The SST value indicates why the bearer channel is in the primary state.
The CALL field identifies the current call state of the CIC. After a call is initiated, a circuit does not return to the Idle (available) state until all related release signaling is satisfactorily completed (the correct release sequence). In and Out call states indicate that the CIC is not available for new calls. Table 8-3 describes the various call states.
| Call State | Description |
|---|---|
In | Incoming call is in progress. Bearer channel is not available for new call. |
Out | Outgoing call is in progress. Bearer channel is not available for new call. |
Idle | Circuit is available for use. |
The BLK field identifies the type of circuit block that has been placed on the CIC. Blocked circuits are not available for calls. Table 8-4 describes the valid circuit block states.
| Block State | Description |
|---|---|
AUTO | Hardware blocking type—the CIC is blocked by an external message generated by a network element outside the MGW. |
REM | Remotely blocked. |
LOC | Locally blocked. |
NONE | There is no block on the CIC. DS0 is available for use. |
MAN | Manually blocked, for example, by an MML command such as blk-cic. |
GATEWAY | Blocked at the MGW. |
|
Note Block states are additive: for example, LOCMAN (locally, manually blocked) and REMMAN (remotely, manually blocked) can both be active at the same time. |
You can retrieve the number and status of all traffic channels for a specific signaling channel. To retrieve traffic channel states, complete the following steps:
Step 2 Enter the following command:
rtrv-tc:channelId
The system returns a message similar to the following:
Media Gateway Controller 1999-08-06 14:06:36 M RTRV "dpc1:CIC=1,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=2,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=3,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=4,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=5,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=6,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=7,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=8,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=9,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=10,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=11,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=12,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=13,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=14,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=15,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=16,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=17,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=18,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=19,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=20,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=21,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=22,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=23,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=24,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=25,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=26,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=27,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=28,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=29,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=30,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=31,PST=IS,CALL=IDLE,BLK=NONE" "dpc1:CIC=32,PST=IS,CALL=IDLE,BLK=NONE"
Call traces record information in a trace file that shows how the Cisco MGC processed a specific call. Traces are most useful when you can be sure that a problem call is reaching the call engine and starting an instance of a Message Definition Language (MDL) state machine. You can determine whether the problem call is reaching the call engine by looking for the presence of non-idle circuits (rtrv-cic) or "new cmgCall" entries in the debug logs.
After you start a trace, all call-processing activity for calls originating from the specified destination is captured. This allows you to follow the call through the Cisco MGC to see where it fails.
The trace output is in binary format. It shows:
Using call trace logs is easy if you remember how to locate the record of a call:
If you are experiencing problems with call processing and need to contact Cisco for support, you should run a call trace before contacting Cisco's Technical Assistance Center (TAC). The trace file helps the Cisco TAC troubleshoot the problem more effectively. For some problems, the TAC cannot begin troubleshooting the problem until you supply the trace file, so it is a good practice to create this file before contacting them.
After checking all physical connections, signal links, bearer channels, and destinations, the person who is troubleshooting the Cisco MGC begins to suspect that the call engine is part of the problem. Performing a call trace while making a call provides details about what is occurring inside the call engine and indicates where the breakdown is occurring (if it is occurring within the call engine).
The MML commands listed in Table 8-5 allow you to start and stop a call trace. See the MML Command Reference appendix in the Cisco Media Gateway Controller Software Release 7 Reference Guide for detailed information on the MML commands because the commands in Release 7.3 can differ substantially from those in later versions. The UNIX command enables you to view the call trace.
Command | Description |
|---|---|
sta-sc-trc:sig_path, log="filenameprefix" |
|
stp-sc-trc:all |
|
get_trc.sh filename |
|
After running the get_trc.sh script, you can view the call flow using the S command (SimPrint in less). This view allows you to see the direction of messages traversing the Cisco MGC. However, by converting the trace into a .trc file (using the C option of get_trc.sh), you can view the call from a code perspective as each line of code is processed by the engine. This view is most helpful for TAC personnel who need to discover things such as cause codes that help in troubleshooting a problem.
If you are running Release 7.4 and up of the MGC software, you still use the sta-sc-trc MML command to start a call trace, but the granularity of what is traceable has been very much improved. Table 8-6 lists the call trace commands for Release 7.4.
Command | Description |
|---|---|
sta-sc-trc:[sig_path|trkgrp >]: prd=n, confirm |
|
|
|
|
|
get_trc.sh filename |
|
To run the trace on Release 7.3 software, complete the following steps:
sta-sc-trc:sc
Where sc identifies the originating signal channel.
For example, if this is an SS7 signaling path, enter the point code. If this is an ISDN connection, enter the name of the IP signal path. You can run the trace on an OPC (use the rtrv-dest:all command to get a list of valid point codes and signal paths).
|
Note If the rtrv-dest command is entered after a switchover has occurred, the state of some of the destinations might be listed as undefined (UND). UND is the default state for a destination when the system starts. In this instance, UNDs indicate that the Cisco MGC has not received a service state message from the associated destination since the switchover occurred. No user action is required. |
The Cisco MGC software creates a trace file using the same name as the signal channel with a .log extension. Ensure that this file does not already exist; if it does, the trace does not run correctly and you receive an error message.
|
Note Refer to the MML Command Reference appendix in the Cisco Media Gateway Controller Software Release 7 Reference Guide for more information on the sta-sc-trc command. |
Step 2 Make the call.
Step 3 When the call is finished, enter the following command to end the trace:
stp-sc-trc:all
Step 4 Quit MML.
To run the trace on Release 7.4 and up of the MGC software, complete the following steps:
In Release 7.4 and up software, this command can be entered in any one of five different formats:
1. sta-sc-trc:sig_path:[log="filenameprefix"][,prd=n], confirm
2. sta-sc-trc:sig_path:span=x[,rng=y][,log="filenameprefix"][,prd=n]
3. sta-sc-trc:sig_path:span=x[[,tc=z],rng=y][,log="filenameprefix"][,prd=n]
4. sta-sc-trc:trkgrp:[log="filenameprefix"][,prd=n], confirm
5. sta-sc-trc:trkgrp:trk=w[,rng=y][,log="filenameprefix"][,prd=n]
Where:
The following paragraphs present examples of each of the five possible command variations:
1. A signaling path level trace traces all calls occurring on the signaling path. Use this format if the specific traffic channel the call uses is unknown.
sta-sc-trc:sig_path:log="filenameprefix", prd=600, confirm
2. A signaling path/span level trace traces calls at the span level. Use this format to reduce the amount of trace information if you know the span on which the call will be placed.
sta-sc-trc:sig_path:span=x
3. A signaling path/span/traffic channel level trace traces calls at the TC or CIC level. Use this format if the traffic channel on which the call will be placed is known.
sta-sc-trc:sig_path:span=x,tc=y
4. A trunk group level trace traces all calls at a trunk group level. Use this format if the trunk group on which the call will be placed is known.
sta-sc-trc:trkgrp:confirm
5. A trunk group/trunk level trace traces only calls for a given trunk (or CIC). Use this format if the trunk group and trunk on which the call will be placed is known.
sta-sc-trc:trkgrp:trk=w
|
Note Refer to the MML Command Reference appendix in the Cisco Media Gateway Controller Software Release 7 Reference Guide for detailed information on using the sta-sc-trc command. In Release 7.4 and up, the command differs substantially from what it was in earlier versions of the software. |
Step 2 Make the call.
Step 3 You can retrieve active trace information by using the rtrv-sc-trc command.
Refer to the MML Command Reference appendix in the Cisco Media Gateway Controller Software Release 7 Reference Guide for detailed information on the rtrv-sc-trc command.
Step 4 When the call is finished, enter the following command to end the trace:
stp-sc-trc:all
Step 5 Quit MML.
The MML command sta-sc-trc produces .btr (binary trace) files, which cannot be viewed with a text editor. The main part of the file name is set up in the sta-sc-trc command, as explained above, and the Cisco MGC adds the .btr extension to these files. The .btr files can contain tracings from many calls all mixed together. Each tracing record in the file has a specific record type and records information of the type that relates to that record. Each record has a unique call ID that relates it to a specific call and is a recording of the external events that the MDL call model was exposed to while the recording was made. Each tracing record is not a recording of the actual MDL.
You can view the call trace output data using the get_tr.sh UNIX script. Get_trc.sh uses the Conversion Analyzer and SimPrint utilities in combination to give a single common interface to all the trace tools. Get_trc.sh makes considerable use of the UNIX less utility for displaying file output and it is assumed that less is available on the system. In Release 7.3 the command should be entered as follows:
get_tr.sh filename
Where: filename is the name of the call trace output data file (.btr) you want to view.
The Conversion Analyzer is a viewer utility for .btr trace files. The name Conversion Analyzer (or "ca") was given to this utility because, at the time it was developed, it was the only way to view a binary trace (.btr) file and the results of the "conversion" that takes place in the MDL call model. The "ca" utility displays each record from a .btr file in a readable form (ASCII text) that can be viewed with any text editor; however, some useful sorting and display options are also available.
The .btr files serve as source files for .ca files. The .ca files are ASCII text output from the Conversion Analyzer obtained by redirection of the standard output to a file. There are two main sections in a .ca file. The header section contains a list of every SigPath defined on the Cisco MGC and a list of the message definition object (MDO) modules that are loaded. The main body contains a printout of every record. Each record has a record number, a timestamp, a call ID, and the print data that the record contains.
The Simulator is a powerful MDO file processing utility that uses .mdo files to replay the events recorded in a .btr file. The front end of the Simulator reads the .btr file. The interpreter in the Simulator utility that loads the .mdo files and replays the events (.btr files) through the MDO, is the same interpreter used by the Call Engine in the Cisco MGC when .mdo files are used (as opposed to .so files). As the interpreter steps through each line of object code (and the action of each object is interpreted) in the .mdo file, each object's print method is activated, which forms the next line of text in the .trc file.
The print method for each object contains text that directly relates to the appearance of the .mdl source code that produced the object in the .mdo file (through compilation of the .mdl source code with the MDL compiler). The .mdo files used with the Simulator when it is processing a .btr file to create a .trc file, must be the same .mdo files that were in use when the .btr file was originally recorded on the Cisco MGC. This is why the conversion from a .btr file to a .trc file is usually done on the Cisco MGC that originated the .btr file.
The interpreter is not used with .so files because those files interact directly with the call engine in the Cisco MGC, but the tracer can record a .btr file regardless of whether .mdo or .so files were used to process the call. The Simulator can, however, replay .btr files using .so files in place of .mdo files. This is a way of checking that the .so and .mdo files perform in exactly the same way, although .so is faster.
Because .so files do not contain MDO objects, there are no print methods available to the Simulator, so no .trc output is possible. When a .btr file is produced by a Cisco MGC using .so files, the replay in the Simulator must be done with the .mdo files that were used to produce the .so files in order to produce an accurate .trc file.
Trace files (.trc files) are text files that are produced by the Simulator utility. They contain detailed line by line trace information from the MDO code that was run in the simulation replay that produced the file, thus they contain MDL traces. The .trc extension is added by the get_trc.sh script if the source of the trace is a .btr file.
Trace files are source files for the SimPrint (SP) utility. They are text files and can be viewed with a text editor. The .trc file should be sent to Cisco Customer Support if expert analysis is required.
SimPrint (SP) is a viewing utility for .trc files. SP converts a .trc file into a sequence diagram that shows all of the external and internal events that occur in a .trc file. This is useful for getting an overview of what is occurring in the trace.
The following list defines the terms used in the call flow printouts generated by the SimPrint tool:
Starting in Release 7.4 and up, you can use the Trace Viewer to view and navigate through call trace outputs. Clicking Trace Viewer in the Cisco MGC toolkit toolbar creates a list box of trace files for you to choose from. When you select a file, you can click View to run the get_trc.sh UNIX shell script utility, which allows you to view a number of different types of files. Get_trc.sh uses the Conversion Analyzer and SimPrint utilities in combination to give a single common interface to all the trace tools. Get_trc.sh makes considerable use of the UNIX less utility for displaying file output and it is assumed that less is available on the system.
Call trace files can be rather large, and leaving these files on your disk after you no longer require them could raise capacity issues. Call trace files are deleted using UNIX commands. The call trace files can be found in the /opt/CiscoMGC/var/trace directory.
Performing call traces to identify problems can be difficult due to the large amount of data the trace may gather before the error occurs, and the negative impact performing call traces has on system performance. Release 7.4 of the Cisco MGC software has MML commands that can be used to diagnose problems with hung calls and abnormal call termination. The following sections describe those commands.
In Release 7.4 of the Cisco MGC software, there is a command that prints diagnostic information about hung calls to a file. The contents of the file include all of the previous states of the call and a history of occurrences leading up to the call being stuck in its current state.
To print diagnostic information on a hung call, complete the following steps:
Step 2 Enter the following command:
prt-call:sig_path:cic=n [,log=xyz]
or
prt-call:sig_path:span=p, bc=m [,log=xyz]
Where:
|
Note This command allows for the use of wildcards for the sig_path parameter. |
For example, the following MML command prints call data for a signaling path called dms100-pc using a CIC of 124:
prt-call:dms100-pc:cic=124 Media Gateway Controller 2000-06-14 11:02:39 M COMPLD "dms100-pc" ;
The output for this command is stored in a file called pc_20000614110239.prt.
Step 3 To change directories, enter the following command:
cd /opt/CiscoMGC/var/trace
Step 4 Use a text file viewer, such as vi, to view the contents of the log file.
In Release 7.4 of the Cisco MGC software, there is a command that prints the global variable information from the state machine and external event information for a call to a file.
To print this information, complete the following steps:
Step 2 Enter the following command:
sta-abn-trc:sig_path|all[,log=xyz] [,prd=n],confirm
Where:
|
Note This command allows for the use of wildcards for the sig_path parameter. |
For example, the following MML command prints call data for a signaling path called dms100-pc using a CIC of 124:
prt-call:dms100-pc:cic=124 Media Gateway Controller 2000-06-14 11:02:39 M COMPLD "dms100-pc" ;
The output for this command is stored in a file called pc_20000614110239.prt.
Step 3 To change directories, enter the following command:
cd /opt/CiscoMGC/var/trace
Step 4 Use a text file viewer, such as vi, to view the contents of the log file.
System logs provide vital information that you can use in troubleshooting problems; however, you should observe the following caution:
|
Caution Debug level logging provides extremely verbose output and, if misused, can cause serious system performance degradation. |
Table 8-7 lists the logging levels that can be displayed without degrading system performance.
| Process | Lowest Logging Level Without Performance Degradation |
|---|---|
Engine | Info (the debug level causes major performance impacts—do not set). |
All others | Debug, but only a single process can be in debug at any point in time. |
To set a process to the debug level of message logging, enter the command:
mml> set-log:processname:debug, confirm
|
Note Log level and destination can be controlled through settings in the XECfgParm.dat file. |
A log message consists of several fields. Refer to the Cisco Media Gateway Controller Software Release 7 Reference Guide for detailed information on specific fields and valid values in log files.
The Cisco MGC creates system log files and stores them in the /$BASEDIR/var/log directory.
Table 8-8 lists the system log files for Release 7.3.
| Log File Type | Active Name | Archived Name | Description |
|---|---|---|---|
System | platform.log | platform.log.n | Contains log messages of varying severity created by system |
Command | mml.log | mml.log.n | MML command category log messages |
Call Processing | cp.log | cp.log.n | Call processing category log messages |
Protocol | prot.log | prot.log.n | Protocol category log messages |
Management | mgmt.log | mgmt.log.n | Element management category log messages |
IO Subsystem | tios.log | tios.log.n | I/O subsystem category log messages |
Environmental | env.log | env.log.n | Environmental category log messages |
Alarms | alm.csv | alm_yyyymmdd | Alarm category log messages |
Measurements | meas.csv | meas_yyyymmdd | Measurement category log messages |
Table 8-9 lists the system log files for Release 7.4 and up.
| Log File Type | Active Name | Archived Name | Description |
|---|---|---|---|
System | platform.log | platform_yyyymmddhhmmss.log | Contains log messages of varying severity created by system |
Command | mml.log | MML_yyyymm | MML command category log messages |
Alarms | alm.csv | alm_yyyymmdd | Alarm category log messages |
Measurements | meas.csv | meas_yyyymmdd | Measurement category log messages |
CDRs | cdr.bin | cdr_yyyymmdd | Call detail records rotated on a regular basis |
The platform log files can be used for debugging or troubleshooting your system. These logs are archived by the Cisco MGC software log rotation utility. This utility is automatically loaded during installation of the Cisco MGC software and is preset to run daily at 4:05 a.m. (local time).
|
Note You can modify automatic execution of the log rotation utility script (log_rotate.sh) using the UNIX tool, crontab. You must have system administration authority to use crontab. For more information on crontab, enter the command, man crontab, on your Cisco MGC. |
The log rotation utility maintains one current copy and seven historical copies of the system logs. The youngest historic file is logName.0 and the oldest is logName.7.
Each execution of the program deletes the seventh historical log and increments all other files by 1 to make room for the newest file. For example, logName.6 becomes logName.7, then logName.5 becomes logName.6, and so on. Finally, the program changes the current log to logName.1, creates a new logName.0 file, and begins logging in that file.
The Cisco MGC software creates the system log file (platform.log) and stores it in the /opt/CiscoMGC/var/log directory.
To retrieve a log file, complete the following steps:
Step 2 Enter the ls command to list the available logs.
The system displays a list of current and historical logs.
Step 3 To view a specific log, enter the following command:
cat SyslogPlatform.log_number | less
|
Note Because the log files are very large, use the less parameter to scroll through the file. You might prefer to print the file to find the information you need. |
For example, to view the platform log from 5 days before yesterday, enter the following command:
cat SyslogPlatform.5 | less
The system returns a response similar to the following:
Mar 2 04:05:09 va-blue : | AlarmManager (PID 29974) {Platform} <Debug> wrote sU
Mar 2 04:05:09 va-blue : | AlarmManager (PID 29974) {Platform} <Info> #0 Operay
Mar 2 04:05:10 va-blue : | MeasurementManager (PID 29977) {Platform} <Debug> m3
Mar 2 04:05:10 va-blue : | MeasurementManager (PID 29977) {Platform} <Debug> m3
Mar 2 04:05:21 va-blue : | ProcessManager (PID 29973) {Platform} <Debug> Sendi1
Mar 2 04:05:21 va-blue : | ProcessManager (PID 29973) {Platform} <Debug> writi5
Mar 2 04:05:21 va-blue : | ConfigManager (PID 29975) {Platform} <Debug> readin6
Mar 2 04:05:21 va-blue : | ConfigManager (PID 29975) {Platform} <Debug> read i5
Mar 2 04:05:21 va-blue : | ConfigManager (PID 29975) {Platform} <Debug> procEv.
--More--
In order to control the types of log messages being written to the System log file, you can use the chg-log command (Release 7.4 and up) or set-log command (Release 7.3). See the MML Command appendix in the Cisco Media Gateway Controller Software Release 7 Reference Guide for details.
The system can generate a large number of files to the logSpoolDir area. For example, if the maxTime value is set to 15 minutes, over 2000 files are created in the logSpoolDir daily. Therefore, you might want to limit the number of logs being created.
One method of limiting the number of logs is to change the logging level or threshold value of the processes you want to limit. The chg-log command (Release 7.4 and up only) sets the logging level for active processes only. This command can be invoked for individual active processes or for all processes. When the all parameter is used, a message is returned that some processes log levels did not change.
To change the log level of a single process, complete the following steps:
Step 2 Enter the following MML command:
chg-log:process_name:log_level
For example, to change the log level of the engine, enter the following command:
chg-log:eng-01:info
To change the log level of all processes, perform the following steps:
Step 2 Enter the following MML command:
chg-log:all:log_level
For example, to change the log level of all processes to warning, enter the command:
chg-log:all:warn
You can use the log viewer application, which is part of the Cisco MGC toolkit, to retrieve and display log messages from system log files. The log viewer accounts for log rotations and makes new logs available. The log server is responsible for log rotation. The log server closes the current file, and creates a new file for current logging. The log viewer also has an option for exporting the results of a log file search to a UNIX file.
For more information on using the log viewer, refer to the "Log Viewer" section.
The format of each text log message should be consistent, and should contain at least the following information:
Dec 12 01:35:23.047 EST: IOCC-01 1 CGST DC-1-1: Network congestion detected
Text messages should not have embedded carriage-return or line-feed characters.
Use the procedure in this section to recover from a failed switchover operation. You would typically use this procedure when the standby Cisco MGC is unavailable to process calls and a critical alarm occurs on the active Cisco MGC.
To recover from a switchover failure, complete the following steps:
rtrv-alms
Step 2 Identify the critical alarm that caused the switchover attempt. To do this, review the alarm(s) that are listed in the response. There should be at least one critical alarm, and an alarm indicating that a switchover began and another alarm indicating that the switchover failed.
If there is only one critical alarm listed, that alarm caused the switchover attempt.
If there is more than one critical alarm listed, compare the timestamp of the critical alarms with the timestamp of the alarm indicating that a switchover began. The critical alarm that occurred before the switchover was begun is the alarm that caused the switchover attempt.
Step 3 Refer to the Cisco MGC Software Release 7 Reference Guide for descriptions of the steps necessary to resolve the critical alarm that caused the switchover attempt.
Step 4 If performing the steps required to resolve the critical alarm does not stabilize the active Cisco MGC, contact the Cisco TAC. Otherwise, proceed to Step 5.
Step 5 Log in to the standby Cisco MGC, start an MML session, and enter the following command to view the current alarms.
rtrv-alms
Step 6 Resolve the listed alarm(s). Refer to the Cisco MGC Software Release 7 Reference Guide for descriptions of the steps necessary to resolve the alarm(s).
Step 7 If resolving the alarms does not stabilize the standby Cisco MGC, contact the Cisco TAC.
In certain situations, it may be necessary to restore your Cisco MGC software configuration (such as when the Cisco MGC software has become corrupted or you have replaced a disk drive). Use the procedures in this section to restore your Cisco MGC software from a local tape drive (if equipped) or a network drive.
|
Note In these procedures, the assumption is that backup operations have been performed regularly on your Cisco MGC. For more information on backing up your Cisco MGC, refer to the "Cisco MGC Backup Procedures" section. |
This procedure restores everything on a tape in a local tape drive to the Cisco MGC base directory.
|
Caution This procedure overwrites existing files under the Cisco MGC base directory. Current content in the overwritten files will be lost! |
To restore the entire Cisco MGC software directory from a local tape, complete the following steps:
va-blade console login: root Password: Mar 17 10:21:34 va-blade login: ROOT LOGIN /dev/console Last login: Thu Mar 16 10:16:34 on console Sun Microsystems Inc. SunOS 5.6 Generic August 1997 # cd /opt/CiscoMGC/local
Step 2 Ensure that this Cisco MGC is in the standby platform state:
rtrv-ne
The system should respond with a message similar to the following:
Virtual Switch Controller - VSC-01 2000-01-12 12:00:47 M RTRV "Type:VSC" "Hardware platform:sun4u sparc SUNW,Ultra-30" "Vendor:"Cisco Systems, Inc."" "Location:Virtual Switch Controller - VSC-01" "Version:"7.3(10).r2"" "Platform State:STANDBY"
If the response indicates that the Cisco MGC is in the standby platform state, proceed to Step 3.
If the response indicates that the Cisco MGC is in the active platform state, manually change the platform state as specified in the "Manual Switchover" section. Once that procedure is complete, proceed to Step 3.
Step 3 Locate your most recent full backup tape and insert it in the tape drive.
Step 4 Run the restore.sh script:
# ./restore.sh
The system should respond with a message similar to the following:
MGC restore utility ----------------------------- Source currently set to Local tape (/dev/rmt/0h) Enter: <N> set source to remote NFS server <L> set source to Local tape (/dev/rmt/0h) <R> for Restore <Q> to quit Select restore mode:
Step 5 Select R to start the restore.
Select backup mode: R Are you sure you want to restore a backup. Current data in the MGC directory will be overwritten and lost. Answer(Y/N):
Step 6 Select y if you are sure you want to restore from the tape.
Answer(Y/N): y x ., 0 bytes, 0 tape blocks x ./var, 0 bytes, 0 tape blocks x ./var/log, 0 bytes, 0 tape blocks x ./var/log/platform.log, 117 bytes, 1 tape blocks x ./var/log/mml.log, 187 bytes, 1 tape blocks. . . . #
Step 7 When the restore has finished, remove the tape from the tape drive.
Step 8 If you have performed any partial backups since the creation of the full backup tape you have just restored, retrieve the most recent partial backup tape and repeat steps 5 to 7 with that tape in the tape drive.
This procedure restores files to the MGC software base directory from a file on an NFS mountable directory on a remote machine. The remote machine must be set up with an NFS mountable directory that can be written to by the machine being backed up. The NFS setup of the remote machine is beyond the scope of this procedure.
To restore the Cisco MGC software directory from a remote machine, complete the following steps:
va-blade console login: root Password: Mar 17 10:21:34 va-blade login: ROOT LOGIN /dev/console Last login: Thu Mar 16 10:16:34 on console Sun Microsystems Inc. SunOS 5.6 Generic August 1997 # cd /opt/CiscoMGC/local
Step 2 Ensure that this Cisco MGC is in the standby platform state:
rtrv-ne
The system should respond with a message similar to the following:
Virtual Switch Controller - VSC-01 2000-01-12 12:00:47 M RTRV "Type:VSC" "Hardware platform:sun4u sparc SUNW,Ultra-30" "Vendor:"Cisco Systems, Inc."" "Location:Virtual Switch Controller - VSC-01" "Version:"7.3(10).r2"" "Platform State:STANDBY"
If the response indicates that the Cisco MGC is in the standby platform state, proceed to Step 3.
If the response indicates that the Cisco MGC is in the active platform state, manually change the platform state as specified in the "Manual Switchover" section. Once that procedure is complete, proceed to Step 3.
Step 3 Run the restore.sh script:
# ./restore.sh
The system should respond with a message similar to the following:
MGC restore utility ----------------------------- Source currently set to Local tape (/dev/rmt/0h) Enter: <N> set source to remote NFS server <L> set source to Local tape (/dev/rmt/0h) <R> for Restore <Q> to quit Select restore mode:
Step 4 Select N to define the remote NFS server:
Select backup mode: N
Step 5 Enter the name of the remote NFS server:
Enter server name: remote hostname
Step 6 Enter the directory name on the remote NFS server:
Enter remote directory : remote directory name Enter server name: va-panthers Enter remote directory : /backup MGC restore utility ----------------------------- Source currently set to remote NFS server (va-panthers:/backup) Enter: <N> set source to remote NFS server <L> set source to Local tape (/dev/rmt/0h) <R> for Restore <Q> to quit Select restore mode:
Step 7 Select R to start the restore:
Select restore mode: R mount -F nfs -o retry=3 va-panthers:/backup /mnt Available files: va-blade20000317105201P.tar va-blade20000317105337.tar Enter filename to restore from: va-blade20000317105337.tar
Step 8 Enter the filename for the most recent full backup performed on your system.
|
Note Full backups have a file name consisting of the name of the host and the timestamp with a .tar designation. Partial backups have a file name consisting of the name of the host, timestamp, and the letter "P" with a .tar designation: |
Enter filename to restore from: filename Are you sure you want to restore a backup. Current data in the MGC directory will be overwritten and lost. Answer(Y/N):
Step 9 Enter y if you are sure that you want to restore the MGC directory:
x etc, 0 bytes, 0 tape blocks x etc/Copyright, 545 bytes, 2 tape blocks x etc/CONFIG_LIB, 0 bytes, 0 tape blocks x etc/CONFIG_LIB/new, 0 bytes, 0 tape blocks . . restore from va-panthers:/backup/va-blade20000317105337.tar complete #
Step 10 If you have performed any partial backups since the creation of the full backup you have just restored, repeat steps 3 to 9 and restore the most recent partial backup.
Step 11 When the restore has finished, you can log off from the machine.
Posted: Mon Aug 28 10:35:09 PDT 2000
Copyright 1989-2000©Cisco Systems Inc.