VPNSC: MPLS Solution Troubleshooting Guide

Table 9-1: Summary of MPLS VPN Service Request Types

Service Request Type	Description
Broken	While the router is correctly configured, the service is unavailable (due to a broken cable or Layer 2 problem, for example). A service request moves to Broken if the Auditor finds the routing and forwarding tables for this service, but they do not match the service intent.
Closed	A service request moves to Closed if the service request should no longer be used during the provisioning or auditing process. A service request moves to the Closed state only upon a successful audit of a remove request. VPNSC: MPLS Solution does not remove a service request from the database to allow for extended auditing. Only a specific administrator action results in service requests being removed.
Deployed	A service request moves to Deployed if the configlet commands have been verified as found in the router configuration file. Deployed indicates that the configuration file has been downloaded to the router, and the intent of the request has been verified at the configuration level.
Failed Deploy	After provisioning occurred, the service request failed to download the configlets to the router. A service request moves to Failed Deploy if an error was detected during the deployment process by the Cisco IP Manager (CIPM). If CIPM is not being used to download configlets, and the product is simply exporting configlets to a directory, there is no way to distinguish between a service request in the Failed Deploy and Pending states. There are two causes for Failed Deploy status: CIPM reports to VPIM that the download failed (lost connection, bad password, etc.). The object could not establish configuration-level verification of intent. If the configlets are exported to a directory, the service request cannot move into a Failed Deploy state.
Functional	A service request moves to Functional when the Auditor finds the VPN routing and forwarding tables (VRF) for this service and they match with the service intent. This state requires configuration-level verification.
Invalid	Indicates that the service request information is incorrect in some way. A service request moves to Invalid if the request was either internally inconsistent or not consistent with the rest of the existing network/router configurations (for example, no more interfaces were available on the router). The VPN Provisioning Inventory Manager (VPIM) server cannot generate configlets to service this request.
Lost	A service request moves to Lost when the Auditor cannot find a configuration-level verification of intent in the router configuration files. The service request was deployed, but now some or all router configuration information is missing. A service request can move to the Lost state only when the service request had been Deployed or Functional.
Pending	A service request moves to Pending when the VPN Provisioning Inventory Manager (VPIM) server determines that the request looks consistent and was able to generate the required configlets for this request. Pending indicates that the service request has generated the configlets and the configlets are successfully downloaded to the routers. The Auditor regards pending service requests as new requests and begins the audit. If the service has been freshly provisioned and not yet audited, it is not an error (pending audit). However, if an audit is done and the service is still pending, it is in an error state.
Requested	If the service is newly entered and not yet deployed, it is not an error. However, if a Deploy is done and it remains Requested, the service is in an error state.

Table 9-3: State Transition Paths for VPNSC Service Requests (Part 2)

Service Request States	Broken	Invalid	Failed Deploy	Closed
Requested	No transition to Broken	Deploy Service Request error	Deploy failed	No transition to Closed
Pending	Route audit is not successful	Redeploy service request error	Redeploy service request failed	Removal of the service request is successful
Deployed	Route audit is not successful	Redeploy service request error	Redeploy service request failed	No transition to Closed
Functional	Route audit is not successful	Redeploy service request error	Redeploy service request failed	No transition to Closed
Lost	Route audit is not successful	Redeploy service request error	Redeploy service request failed	No transition to Closed
Broken	Route audit is not successful	Redeploy service request error	Redeploy service request failed	No transition to Closed
Invalid	No transition to Broken	Redeploy service request error	Redeploy service request failed	No transition to Closed
Failed Deploy	No transition to Broken	Redeploy service request error	Redeploy service request failed	No transition to Closed
Closed	No transition to Broken	No transition to Invalid	No transition to Failed Deploy	No transition to Closed

3. Question: Which of the error states are due to provisioning and which are due to auditing?

  Answer: Requested (after provisioning), Invalid, and Failed Deploy are due to error conditions in provisioning. Pending (after auditing), Lost, and Broken are due to error conditions in auditing.

• Function 1: Collect the PE router configuration files (PE-upload)

• Function 2: Collect the CE router configuration files (CE-upload)

• Function 3: Provisioning

• Function 4: Write the changed configuration information to the PE (PE-DownLoad)

• Function 5: Write the changed configuration information to the CE (CE-DownLoad)

1. Question: What is the flow of the provisioning operation?

2. Question: Where can I see how the provisioning functions performed in my audit?

  Answer: In the Task Log Summary Table, look at the PE-UpLoad and CE-UpLoad information for that service request. One or both of them should say "Fail." This makes the rest of them "skipped." Thus, the service request remains in the Requested state.

  Answer: Check at the top of the frame for a set of links: Stdout/Stderr Errors. Stdout/Stderr gives you the messages that were displayed to the terminal when the VPIMDownLoadClient program runs. From this, you can see whether and how the client application ran. All abnormal terminations would be reported. Similarly, Errors lists the error messages reported by the client.

5. Question: When I try to access the Log Summary Table, I see a "Fatal Error" message displayed. What happened?

  Answer: Click the Stdout/Stderr link and view the output.

  First, being a CORBA client, did it connect to the CORBA server?

  If it did not, you see a number for "Retrying Connection" lines, followed by these lines (at the very end):

  Caught exception when pinging VPIMManager: SYSTEM_EXCEPTION:10085—Communication failure—no server at host: your_host_name [Completion status: COMPLETED_NO]

  This message means that the CVPIM Server is not running. Invoke wdgui and take the appropriate actions:

a. If disabled, issue the following command:

wdclient start CVPIMServer.

b. If disabled-dependent, see which of the dependent server(s) are disabled.

c. If the lock_manager is not running, restart Watchdog.

7. Question: I see in the wdlog that the Task Scheduler is up and running. Yet my scheduled task is not running. What's wrong?

d. Select the Invalid service request.

e. From the Provisioning drop-down menu, choose Deploy VPN Service.

f. If you have multiple Invalid requests, then either repeat this procedure for all service requests or choose Provisioning > Deploy Service Requests.

g. Be sure to click the Deploy selected service requests radio button.

If the service request fails an Audit New Service Request, it is kept in Pending. If the service request passes this audit and audit routing is enabled (by checking Use VPN routing information during audits in the task window), VPNSC: MPLS Solution tests whether there is adequate data for audit routing. If the audit does not have the data or the Use VPN routing information during audits is not checked, the service request moves to Deployed.

If the routing test passes, the service request moves to Functional. If the routing test fails, the service request is Broken.

  Answer:

a. For a service request stuck in Pending, generate a service request audit report and select the Audit new service requests checkbox. In the report window, select Audit Details.

For a service request stuck in Lost, generate a service request audit report and select the Audit existing service requests checkbox. In the report window, select Audit Details.

This gives you the audit trace for both the PE and CE. There are two columns—one for the PE and other for the CE.

b. Look in the "Audit Status" row.

For the audit to pass, both should say "Success." However, as the service request is stuck in Pending or Lost, one or both of them should say "Failure."

c. Go to the column and read through the audit details.

One or both of the tests would fail and the audit details explains why it failed. (This is usually reported in the next-to-last line.)

d. Read the error information.

The problem is usually missing or wrong configuration information on the router. Has this configuration command been generated in the configlet? The configlet for the router is given below in the same column.

e. See if the command is present, and if present, whether it is correct.

f. If you feel that the command is incorrect and should be changed, contact Cisco Support for help.

g. If the command is correct, press the corresponding router configuration button.

The router configuration on which the auditing was performed is shown.

h. Check the configuration command line in the configlet (the line that is apparently missing or incorrect as discovered by the audit) and see if the command line is present in the router.

i. If it is removed or absent or incorrectly changed in the router, redeploy the service request to get the corrections sent to the router.

j. However, if the configlet seems correct and the configuration commands are present in the router, call Cisco Support for assistance.

k. You can continue troubleshooting by selecting Show Processed Config Text. This presents the configuration file of the model router.

l. Check whether that command is also present in the configuration of the model router.

m. If it is absent or incorrectly modeled, be sure to inform Cisco Support.

  Answer: The audit is for a service request that may or may not be in multiple VPNs. However, VPNSC: MPLS Solution runs an Audit VPN by VPN in alphabetical order. In case a service request is in multiple VPNs, the software audits the service request only once. This takes place in the VPN that comes up first in the alphabetical sequence, which in this case, is Blue. The software does not audit the same service request in other VPNs because (a) it is already been audited and there is nothing new to audit; (b) auditing is a costly process and auditing again for the sake of display in audit reports is a waste of time; and (c) the service request information along with the audit run information is already available in the VPN Console.

  If you have a service request in multiple VPNs, look for the audit report in the VPN that comes first in the alphabetical sequence.

3. Question: What is the Show Processed Config Text button? How does it differ from the configuration file?

  Answer: When performing a deployed test, VPNSC: MPLS Solution software executes an "attribute based" audit. This means that the Auditor builds a software model of the router. To do this, the product takes all the relevant configuration commands from the router. Then the product builds the software model. VPNSC: MPLS Solution software runs the provisioning and auditing functions using this software model as a basis.

  The configuration text is the computed configuration file of the router's software model. The software model does not model all aspects of the router configuration, such as enable passwords and clock commands, so the configuration text does not contain everything that the configuration file contains. It does contain all the relevant information such as interface commands, routing protocol commands, and their dependencies (such as access lists, route-map, and so forth).

  Answer: When you are scheduling an audit task (by choosing Auditing > Generate Service Request Audit Reports), you must check the check box, Use VPN routing information during audits.

  Also be sure to schedule data collection by choosing Monitoring > Collect VPN Routing Information.

  Answer: You must collect VPN routing info to use it. Schedule data collection by choosing Monitoring > Collect VPN Routing Information.

7. Question: I have a VPN with two CEs. One PE-CE Layer 2 connection is down and in a Broken or Lost state. Why does the other PE-CE pair's service request also move to Broken?

  Answer: The routing test checks whether the service request provides a routing level connection of the site to the VPN. For a VPN to exist, there must be at least two sites. Hence, in a VPN with two sites, if connectivity of one of the sites go down, the VPN no longer exists. Therefore, the connectivity of the other site to the VPN also fails—there are no routes across the provider's core network.

  Answer: First read the answer to the previous question (Question 7).

  Until you add at least two sites to the VPN, the VPN is not complete. The first site cannot participate in a VPN because the VPN does not yet exist. Thus, the service request moves to Broken. The service request moves to Functional when the second site is connected to the VPN, assuming core network connectivity exists and is functional.

10. Question: I've scheduled an audit. But I do not see any audit reports. Has it run as yet?

  Answer: First, is there a service request to audit? If not, no reports can be generated.

n. If a service request exists, start by going to the Task Logs (see Question 2 under "Provisioning Problems" for information on accessing the Task Logs).

o. Click GenerateVPN AuditReports And Topologies.

p. Click Stdout/Stderr.

Detailed information is listed on the right pane.

q. Scroll down to the line "about to call run_ngs on baseline baseline_name."

where baseline_name is VPN_BS_vpn_name for a VPN and SP_BS_provider_name for a provider. Below that note a call to "run_ngs" made with n argument.

r. Look for a line such as "run_ngs for baseline VPN_BS_VPN failed with exit code n."

s. Check the value of n.

If the value of n is 6, run_ngs may have crashed.

t. Check the core file in the tmpdir directory (as given in netsys.tmpdir.unix in csm.properties file) and see if it belongs to run_ngs or conn_solver.

u. If the value of n is 1 or another number, go to the repository_path/Baselines/baseline_name directory and read the file run_ngs.log.

The run_ngs.log file provides the reasons for the unexpected termination.

v. In any case, tar the Repository and save the core file (if any) and contact Cisco Support for assistance.

If the run_ngs.log file says something like: "path/.dataroot pointed to by default.ddf does not exist," someone copied over the Repository. Read the next question for the solution for this.

  Answer: Go to repository_path/Baselines and issue this command:

rm -f VPN_BS_*/default.ddf SP_BS_*/default.ddf

  Now the audits will run.

  Answer: This can occur if run_ngs crashed while conn_solver succeeded in writing out state change and audit trail information to the Repository. If this is this case, you do not see any reports, yet the job seems to be done. Be sure to send the core file and the Repository to Cisco Support.

13. Question: The audit seems to have been performed (from audit reports), but the All VPN Service Requests Report shows the service request is still in the prior state.

14. Question: I see an extra row in the Audit Reports: Repository Error. What does this mean?

  Answer: A read/write operation to the Repository failed. Report this error and send your current Repository to Cisco Support.