|
|
This appendix provides the background and technical information you need to start building some Cisco Info Center Automation procedures. The procedures in this appendix can be regarded as best practices to solve some individual problems. There may be various solutions to some situations, this appendix only provides suggested implementations.
This appendix contains the following sections:
There are four types of triggers:
A trigger type is determined by how the action is called. A trigger may have either or both ascent and descent actions.
The following sections describe the four types of triggers in more detail.
This type of trigger allows a one-shot response to a situation.
This type of trigger allows for a multi-shot response to a situation, only stopping when the situation has been resolved.
There are situations where you may not want Automation to start as soon as a trigger is primed, instead, you may wish for the trigger to be primed for a number of sample intervals before calling the ascent action. This type of behavior is called a delayed trigger. The number of sample intervals before triggering is called the threshold.
A delayed edge trigger delays triggering until the threshold has been exceeded. In Figure C-3, the threshold has been set to 5, so the trigger has to be primed for 5 samples in a row before firing the ascent action. It then waits for the trigger to evaluate false, when it then calls its descent action.
This type of trigger allows for the delaying of action until it is absolutely necessary.
When an alert has been marked as clear, it is not automatically deleted. The deletion of alerts is a function of the Automation system. This requires you to define a policy for the deletion of alerts and implement it in the Automation system. The following sections provide several suggestions and their implementations.
The simplest policy is to expire alerts by how old they are. The first question to answer is what is an old alert. There are a number of time fields in an alert which could be used, FirstOccurrence, LastOccurrence, and StateChange. These are times stored in seconds (starting at Midnight January 1st 1970).
The requirement is to measure when "now" is and use that as the basis to determine the age of an alert. The Cisco Info Server SQL statement getdate returns the time in seconds, whenever a Cisco Info Server SQL statement is executed. Using a simple calculation, it is then possible to work out the time five minutes ago (getdate - 300), or the time 24 hours ago (getdate - 86400). For example, to select all alerts that last occurred five minutes or more ago, specify the following command:
select * from alerts.status where LastOccurrence < (getdate - 300);
A typical scenario for clearing old alerts would be to delete any cleared alerts which occurred more than five minutes ago. The following tables contain the trigger and action for this scenario.
Trigger Name | FindOldAlerts |
Sample Rate | 60 |
Ascent Action | DeleteAlert |
Descent Action | Not Applicable |
Condition | select * from alerts.status where Severity=0 and LastOccurrence < (getdate - 300); |
Type | Level |
Threshold | Not Applicable |
Execution Mode | For each |
Action Name | DeleteAlert |
Data Effect | delete from alerts.status where Serial = @Serial; |
External Effect | Not Applicable |
This trigger and action deletes alerts, however, possibly sooner than you desire. Consider the scenario where a single alert comes in at midnight and in the morning, the operations team fix the problem and clear the alert. Because the alert last occurred at midnight, as soon as it is cleared, when the Automation is executed (every sixty seconds) the alert will be deleted even though it has only been cleared for sixty seconds.
In the previous example, alerts may have been deleted too quickly. The reason for this was because the LastOccurrence field was used to test the age of the alert. What may be more useful is testing on the time the alert was last changed. For this you can use the StateChange field, which reflects the time the alert was last modified by an Automation or user.
Trigger Name | FindOldAlertsTwo |
Sample Rate | 60 |
Ascent Action | DeleteAlertTwo |
Descent Action | Not Applicable |
Condition | select * from alerts.status where Severity=0 and StateChange < (getdate - 300); |
Type | Level |
Threshold | Not Applicable |
Execution Mode | For each |
Action Name | DeleteAlertTwo |
Data Effect | delete from alerts.status where Serial = @Serial; |
External Effect | Not Applicable |
Five minutes after an alert is cleared, it is deleted. This Automation still uses a query and multiple deletes and can be improved for performance. This is covered in the next example.
The previous example has one drawback. When many expired alerts exist, many SQL commands are executed to delete the various alerts one at a time. A more efficient solution would be to use one trigger that detects when any alerts need deletion. When so, then a delete command could be used once (by setting the Execution Mode to Once Only). The delete command should then be set up to delete any old alerts with a where clause which is the same as the trigger. The following tables contain the Automation.
Trigger Name | FindOldAlertsThree |
Sample Rate | 60 |
Ascent Action | DeleteAlertThree |
Descent Action | Not Applicable |
Condition | select * from alerts.status where Severity=0 and |
Type | Level |
Threshold | Not Applicable |
Execution Mode | Only once |
Action Name | DeleteAlertThree |
Data Effect | delete from alerts.status where Severity=0 and |
External Effect | Not Applicable |
There is another way to simplify the entire process. In the previous example, the same query is performed twice, once in the trigger, once in the action. Whenever the trigger fires, the same query is executed again. Using a delete command in the trigger condition makes the operation more efficient. The trigger will be unable to fire Ascent and Descent actions. This allows SQL commands to be executed at regular intervals.
Trigger Name | FindOldAlertsFast |
Sample Rate | 60 |
Ascent Action | Not Applicable |
Descent Action | Not Applicable |
Condition | delete from alerts.status where Severity=0 and |
Type | Level |
Threshold | Not Applicable |
Execution Mode | Only once |
With this single trigger, the SQL condition will be executed every sixty seconds, deleting aged alerts. As only one command is ever executed, this is a stable and efficient Automation.
To complete this scenario there are two additional elements required. See the following section "Clearing Journals and Details Tables" for more information.
When you delete an alert in the Cisco Info Server using the Event List, the Event List not only clears the alert, it also deletes related journal and details entries. When a delete is carried out by Automation, the tables are not tidied up in the same way. Left unmanaged, the tables will grow continuously. This would consume space and would eventually have an impact on Cisco Info Server performance.
The most efficient way to handle these tables is to implement housekeeping Automations to perform a clean-up. These Automations are incorporated within the standard product and will be active upon installation. They are reproduced here to show an example of this type of Automation. They also show example usage of an SQL sub-query statement and using tables other than the alerts.status table.
The journals trigger deletes all of the entries in the journals table which have a serial number which is not in the alerts.status table. The details trigger deletes all of the entries in the details table which have an identifier which is not in the alerts.status table.
Trigger Name | CleanJournalTable |
Sample Rate | 3600 |
Ascent Action | Not Applicable |
Descent Action | Not Applicable |
Condition | delete from alerts.status where Serial not in (( |
Type | Level |
Threshold | Not Applicable |
Execution Mode | Only once |
Trigger Name | CleanDetailTable |
Sample Rate | 3600 |
Ascent Action | Not Applicable |
Descent Action | Not Applicable |
Condition | delete from alerts.details where Identifier not in (( |
Type | Level |
Threshold | Not Applicable |
Execution Mode | Only once |
Another time based filtering mechanism would be to make any alert, which occurs more frequently than a given threshold, flash on the Event List to highlight the alert.
This example uses the Poll field to hold a value, which is the threshold for the average number of seconds in between occurrences of the alert. When the average gap between them is less than the Poll field value, the alert item flashes in the Event List. The trigger resets the flashing mechanism before execution. This ensures that when the occurrence of the alert stops, then the flashing also stops (when they pass back across the threshold).
Before using this Automation, a statement should be added to the rules file, so the Cisco Info Mediator sets the poll value. In this example, specify:
@Poll = 6
In this case, the value is set to six, which sets an average of less than six seconds between alerts.
When you do not want to edit the rules file, you should set the poll value for all alerts by changing the first trigger line. The following example shows this trigger. When you set the Poll in the rules file, omit this statement. An advantage of setting the poll value in the rules file is that it allows different intervals for different alerts.
Trigger Name | FlashHighRate |
Sample Rate | 60 |
Ascent Action | FlashHighRate |
Descent Action | none |
Condition | update alerts.status set Poll = 6, Flash = 0; |
Type | Level |
Threshold | none |
Execution Mode | For each matched row |
Action Name | FlashHighRate |
Data Effect | update alerts.status set Flash = 1 where Serial = @Serial and |
External Effect | Not Applicable |
The first trigger condition statement resets the flash status for all alerts and sets the poll values. The second trigger condition statement selects any alerts which have occurred more than once, and are at least one minute old. The last condition is to allow for a reasonable sample rate.
In certain situations, events become irrelevant or redundant when other related events are received. For example, the fact an interface is down can be overshadowed by the fact the whole node is down. This Automation is an example of how to perform an efficient sweep of events and delete events that pertain to any interface failure for which a node failure also exists.
The Automation uses a sub-query, similar to the housekeeping Automation for clearing up journals and details. The use of a sub-query is much more efficient than referencing each alert separately in an action. The subquery does not allow complex field comparisons, and therefore is only suited to certain tasks.
Trigger Name | SuppressIfevents |
Sample Rate | 60 |
Ascent Action | none |
Descent Action | none |
Condition | delete from alerts.status where Summary like `Link Down' and Node in ((select Node from alerts.status where Summary like `Node Down')); |
Type | Level |
Threshold | 0 |
Execution Mode | Only once |
When executed, the statement within the double parenthesis is executed first. This generates a list of node values for which a Node Down event exists. The main statement is then executed, deleting all events reporting Link Down whose node value appears in the list generated by the subquery.
Following are the six generic types of SNMP traps:
0 | Cold Start |
1 | Warm Start |
2 | Link Down |
3 | Link Up |
4 | Authentication |
5 | EGP Neighbor Loss |
6 | Enterprise Specific1 |
| 1This is not a generic trap. It's a flag for an enterprise specific trap. |
For many of them, there are generic ways to handle them within Cisco Info Center Automation. The following sections provide some examples.
In some cases, it may be important to retain the Cold Start message. For example, where you need to know a machine has been power cycled when it should not have been. To achieve this, you need to flag that the Cold Start message has been handled. In the example here, the Grade field is used to show that the Cold Start has been handled. The trigger looks for all Cold Start messages that have a non-clear Severity and a Grade field set to 0.
Trigger Name | FindColdStarts |
Sample Rate | 60 |
Ascent Action | ClearColdStart |
Descent Action | Not applicable |
Condition | select * from alerts.status where Summary like 'Cold Start' and Severity > 0 and Grade = 0; |
Type | Level |
Threshold | Not applicable |
Execution Mode | For each |
Action Name | ClearColdStart |
Data Effect | update alerts.status set Severity = 0 where Node = '@Node' and Serial <> @Serial and LastOccurrence < @LastOccurrence; update alerts.status set Grade=1 where Serial = @Serial; |
External Effect | Not Applicable |
With this trigger and action, when a Cold Start is received, all related alerts are cleared, however, the cold start itself is not; it just has the Grade field set to 1, which stops the alert from setting off the trigger again.
It is important to note that the Grade field was an arbitrary selection of a field. It just happens to be one of the unused core fields in the Cisco Info Center alert record. When you wanted to use the Grade field for any other purpose, it may be worth adding a field to the Cisco Info Center alert records.
When a Link Up occurs, any Link Down messages from that node are implicitly no longer valid. When a Link Up is seen, it is appropriate to clean all the Link Down messages and the Link Up message. The AlertKey field is critical, because it should contain the physical port number of the link. So when clearing the Link Down messages, the only ones to be cleared are the Link Down messages from the Node and physical port that gave the Link Up.
Trigger Name | FindLinkUpOne |
Sample Rate | 60 |
Ascent Action | ClearLinkUpOne |
Descent Action | Not applicable |
Condition | select * from alerts.status where Summary like 'Link Up' and Severity > 0; |
Type | Level |
Threshold | Not applicable |
Execution Mode | For each |
Action Name | ClearLinkUpOne |
Data Effect | update alerts.status set Severity=0 where Serial=@Serial; update alerts.status set Severity=0 where Summary like 'Link Down' and Node='@Node' and AlertKey='@AlertKey'and LastOccurrence <= @LastOccurrence; |
External Effect | Not Applicable |
With this Link Up trigger and action, related Link Down alerts are cleared. However, this example may not be powerful enough for WAN requirements. The next example expands on further requirements.
Not all Link Up alerts close Link Down alerts. For example, in European ISDN lines where you may be more interested in the Link Up messages, as opposed to Link Down messages with traditional permanent links. Take Cisco routers as an example, they report the traditional ifIndex value along with the type of circuit (for example, ethernet) tokenring, BRI (Basic Rate Interface) and this information can be included in the Summary field. To detect ISDN, the string BRI can be used to discriminate between ISDN and non-ISDN link up/down messages. This needs two triggers and actions.
Trigger Name | FindNonISDNLinkUp |
Sample Rate | 60 |
Ascent Action | ClearNonISDNLinkUp |
Descent Action | Not applicable |
Condition | select * from alerts.status where Summary like 'Link Up' and Summary not like 'BRI' and Severity > 0; |
Type | Level |
Threshold | Not applicable |
Execution Mode | For each |
Action Name | ClearNonISDNLinkUp |
Data Effect | update alerts.status set Severity=0 where Serial=@Serial; update alerts.status set Severity=0 where Summary like 'Link Down' and Node='@Node' and AlertKey='@AlertKey'and LastOccurrence <= @LastOccurrence; |
External Effect | Not Applicable |
Trigger Name | FindISDNLinkDown |
Sample Rate | 60 |
Ascent Action | ClearISDNLinkDown |
Descent Action | Not applicable |
Condition | select * from alerts.status where Summary like 'Link Down' and Summary like 'BRI' and Severity > 0; |
Type | Level |
Threshold | Not applicable |
Execution Mode | For each |
Action Name | ClearISDNLinkDown |
Data Effect | update alerts.status set Severity=0 where Serial=@Serial; update alerts.status set Severity=0 where Summary like 'Link Up' and Node='@Node' and AlertKey='@AlertKey'and LastOccurrence <= @LastOccurrence; |
External Effect | Not Applicable |
With this pair of triggers and actions, ISDN Link Down messages clear Link Up messages for the interface and non-ISDN Link Up messages clear Link Down messages for the interface.
When a particular event occurs, it may be useful to generate mail for particular users. With most mail systems, it is possible to have mail forwarded to managers or on-call support staff or even to a text pager. This can be achieved using external actions in Cisco Info Center.
Before installing the Automation, it will be necessary to create a shell script to compose and send the mail. The following example shows the script used:
#!/bin/sh
This script should be installed and made executable. For this example, it resides in:
$OMNIHOME/utils/nco_mail
The host where this file is installed must be running the process control system. In this example, it resides on a host called bester.
Trigger Name | MailOnCritical |
Sample Rate | 60 |
Ascent Action | SendMailNow |
Descent Action | Not applicable |
Condition | select * from alerts.status where Severity=5 and Grade=0; |
Type | Level |
Threshold | Not applicable |
Execution Mode | For each |
Action Name | SendMailNow |
Data Effect | update alerts.status via '@Identifier' set Grade=1; |
External Effect | Executable: Arguments: Host: Effective UID: |
When an alert of a Severity value 5 (Critical) with a Grade field set to 0 occurs, the trigger fires the action. The action itself does two things. The data effect sets the Grade field to 1 to stop the trigger firing again. The external effect calls the shell script with all the parameters it needs to send mail. In this case, the mail is sent to root.
In the previous example, mail was sent as soon as an alert appeared. You may wish to configure the system to let you know after a period, when an alarm is still present. This can be achieved by modifying the trigger to use a delayed edge. In this example, the trigger is configured to perform the action after five minutes.
Delayed triggers depend on the threshold and sample rate. When an alert matches the condition, the threshold comes into play; with a threshold of five, the condition must be fulfilled five times on the next evaluations of the trigger. So where the sample rate is sixty and the threshold is five, the trigger condition must be fulfilled five times, which equates to five minutes (sixty seconds times five equals 300 seconds; five minutes).
Note this trigger does not distinguish between the critical alerts, it just registers that there are critical alerts. Therefore, when an alert arrives as critical in the first minute, followed by a second alert in the second minute, followed by the first alert being deleted, the "clock" will continue ticking after being started by the first alert. This Automation is very much an example, and it is recommended you make the where clause much more specific.
Note also, this trigger uses Once only as its execution mode.
Trigger Name | MailOnCriticalAfter5 |
Sample Rate | 60 |
Ascent Action | SendMailAfter5 |
Descent Action | Not applicable |
Condition | select * from alerts.status where Severity=5 and Grade=0; |
Type | Delayed Level |
Threshold | 5 |
Execution Mode | Once only |
Action Name | SendMailAfter5 |
Data Effect | update alerts.status via '@Identifier' set Grade=1; |
External Effect | Executable: Arguments: Host: Effective UID: |
When any critical alerts have existed in the system for more than five minutes, a mail message is sent. The trigger fires the action as Once only.
When you want to page someone rather than send mail, the principle in the previous two examples is still appropriate. The only change needed is to the shell script, which needs to be modified to send a message to the paging system.
Automation triggers are checked at regular intervals to see when the trigger should fire. The sample rate controls the check. For example, the sample rate could be set to sixty seconds. This means that every sixty seconds, the SQL inside the Automation trigger is executed and the results are evaluated to see when the trigger should fire. Therefore, every sixty seconds, there will be a query on the database.
A sample rate of 0 is a special case, and one to be especially wary of. When set to 0, the sample rate is ignored and the trigger becomes real-time. It is not constantly executed after every insert or update to the alerts.status table, however, the trigger is evaluated in the second after every new alert arrives in the Cisco Info Server. Note, the trigger will be performing an SQL query on the entire database which could adversely affect performance.
For example, when you expect to have hundreds of alerts arriving constantly, a real-time trigger is a bad solution; when one hundred alerts arrive over ten seconds, the SQL will be executed ten times, and it could well be the trigger and action take most of the second to do what they have to do. This means the Cisco Info Server has less time to respond to users and their Event Lists, less time to process other Automations and in the worst case not enough time to finish the action before the next alerts coming in set it off again.
With that kind of alert load it is probably better to use a standard, non-zero sample rate which is set to a value which will give you predictable performance, not dependent on the number of arriving alerts.
There are occasions when real-time triggers are appropriate, however, these are rare occasions. Often people may use a real-time trigger to change incoming data from a Cisco Info Mediator, for example to swap fields around or change severity values; this is an inappropriate use of real-time triggers because the Cisco Info Mediators themselves are configurable by changing the rules file for the Cisco Info Mediator. The Cisco Info Mediator rules file allows great flexibility in what gets sent to the Cisco Info Server and how it is formatted. Changes made to the Cisco Info Mediator rules file do not impact the performance of the Cisco Info Server, and in cases where alerts need to be discarded, can improve Cisco Info Server performance.
You can use a simple rule (derived from Nyquists theorem on sampling which is used in audio recording devices) to work out the rate you want to sample with. For example, when something happens at approximately <n> seconds, sample at <n>/2. Therefore, when something happens every thirty seconds (or at least you want the system to handle it as soon as possible), sample at fifteen seconds. This rule also suggests a sampling rate may be selected by the update rate of a user's Cisco Info Admin Desktop.
Consider, for example, a system where all the user's Event Lists are set to refresh every
sixty seconds. This could be considered as the user's sampling rate and so a generic sample rate for Automations of thirty seconds would ensure users always saw post Automation processed alerts.
Low sample rates have a drawback, similar to the problem of real-time triggers. When you have a one second sample rate for example, you can run out of time for the Cisco Info Server to service users, gates and other connections. Avoid low sample rates, and use the rule above to work out how fast you really need to sample.
The following example shows an Automation required to collect events and forward them to a third party database as SNMP traps in batches of 100. Additionally they need to be ordered by serial number. This example performs the following processing:
1. Event data is acquired by multiple Cisco WAN Manager (formerly known as StrataView) Cisco Info Mediators. The Cisco Info Mediators poll the Informix database underlying each Cisco WAN Manager Cisco Info Mediator.
2. The rules file sets the fields from the Informix database that should be populated in the Cisco Info Server database fields and decides how de-duplication within the Cisco Info Server is achieved. It also creates and sets fields used in the Automation.
3. In this case, Cisco Info Server de-duplication is turned off as this is performed within rules file processing.
4. The Automation sets the field, SendNew to 1 for the lowest Serial numbered event flagged as needing to be sent (FlagNew set to 1), and the next 99. This has the effect of batching the next 100 events to be sent. When ready, the events are forwarded to an SNMP Cisco Info Gateway.
5. The Cisco Info Gateway is configured for the required mapping, writer attributes and performs an AFTER IDUC DO command.
Trigger Name | Ready to send |
Sample Rate | 10 |
Ascent Action | Clear to send |
Descent Action | Not applicable |
Condition SQL | select * from alerts.status where FlagNew = 1 order by Serial asc; |
Type | Level |
Execution Mode | Once only |
Ascent Action SQL | update alerts.status set SendNew = 1 where Serial >= @Serial and Serial <= (@Serial + 99); |
Descent Action SQL | Not applicable |
External Effect | Not applicable |
The complete configuration file has not been replicated here. The following is the relevant extracts from the file:
The following Automation monitors remote host links and plays a soundfile on a Cisco Info Admin Desktop screen when a link to a host is lost. The soundfile will be played every sixty seconds until the alert is acknowledged. Extracts of the rules file, look up table, the Automation and an external action script have been provided below:
Trigger Name | Fire soundfile |
Sample Rate | 60 |
Ascent Action | Soundfile |
Descent Action | Not applicable |
Condition | select * from alerts.status where Soundfile <>" and Acknowledged = 0 and Grade = 0; |
Type | Level |
Threshold | Not applicable |
Execution Mode | For each |
Action Name | Soundfile |
Data Effect | update alerts.status set Grade = 1 where Serial = @Serial; |
External Action | $OMNIHOME/utils/audioplay -f @Soundfile >/dev/null 2>&1 |
The complete rules file has not been replicated here. Following are the relevant extracts. The @Customer field is already populated with a customer short code, for example, MM for Micromuse. The table sounds are stored in:
tablesounds='/nfshome/sounds/sounds.lookup'
MM Link Down MM_linkdown.au
MM Link Up MM_linkup.au
MM Node Down MM_nodedown.au
MM Node Up MM_nodeup.au
MM Threshold on MM_bandthresh.au
MM Authentification MM_authfail.au
MM Warm Start MM_nodewarmstart.au
MM Cold Start MM_coldstart.au
MM IF down on MM_IntfaceDown.au
MM IF up on MM_IntFaceUp.au
MM Rearm Trap MM_CPU_80.au
The following script takes the @SoundFile value passed to it by the Automation and fires the sound file on the host specified within the script. Note, for this example to work, all hosts must have the /nfshome NFS partition mounted.
#!bin/csh -f
#
# audioplay -s hostname -f soundfile
#
# Amend the next line to change the target hosts for sound files
set hosts=(host1 host2 host3 host4)
if (-f/NOAUDIO) then
exit 0
endif
#
if ("x$1"=="x") then
echo "Usage: audioplay -f soundfile"
exit 0
endif
if ("x$1"=="x-f") then
set filename=/nfshome/sounds$2
# shift
# shift
endif
if (! -f${filename} then
echo "Could not find file ${filename}
exit 3
endif
foreach i($hosts)
set X='sh -c "/usr/sbin/ping $i 1 2>&1 | grep alive"'
if ("x$x" !="") then
(rsh ${i} /usr/demo/SOUND/play ${filename} >/dev/null&)&
echo `date @+%d/%m/%y %H:%M:%S" ${filename} root@${i} >>
/usr/Omnibus/log/audioplay.log &
sleep 1
endif
end
echo ` ` >> /usr/Omnibus/log/audioplay.log
Posted: Thu Apr 1 10:42:41 PST 1999
Copyright 1989-1999©Cisco Systems Inc.