|
|
The optional failover configuration features both hardware and software redundancy, which provides added protection against catastrophic system failures. In a failover system, a primary system is paired with a backup system designed to automatically take over from the primary if failure occurs. There may be a very short period of service interruption when switching over to the backup system.
Although it is not a true fault-tolerant system, the failover configuration provides a very high level of up-time at much less cost than a fault-tolerant alternative.
The design of the failover configuration provides the following:
Failover relies on having one system explicitly designated as the Master at any one time. The Master system controls failover processing as long as it remains the Master. Should the Master fail, the current Slave system takes over the role of Master. When the failed system comes back on-line, it becomes the Slave of the new Master.
Major error conditions with the Master triggers automatic failover to the current Slave system. Failure of the Slave triggers fault reporting.
The failover configuration automatically copies changed configuration data from one system to another. This allows either system to operate independently in case the other fails. This function is provided by the replication daemon.
When failover occurs, the failover daemon automatically triggers switchover of communications lines to the current Master system.
The failover configuration reports faults through two mechanisms. The Master system reports faults via the XE Alarm Manager. Both the primary and backup systems are connected to an Alarm Relay Unit (ARU), which automatically triggers site alarms in the event either system goes out-of-service for any reason.
This architecture uses a small daemon process (forevd) to the top layer of the standard TransPath software to provide the required failover functions. A second small daemon process (frepld) keeps the slave system synchronized. These daemons work with the existing Process Manager to control application processes as needed when failover occurs.
The following daemons are used by failover:
foverd)
frepld)
The failover daemon controls overall failover processing.
1. Exchanges heartbeat messages with its counterpart running on the redundant computer system. The daemons use these messages to negotiate master/slave status and to detect failure of the master or slave system.
2. Starts the replication daemon when the system transitions to the slave state.
3. Stops the replication daemon when the system transitions away from the slave state.
4. Starts the XE Process Manager (procM) when a system is ready to become Master.
5. Stops the XE Process Manager when the system transitions away from the master state.
The replication daemon copies any new or changed files from the current Master system to the current Slave system. It runs on the Slave system and is inactive if the system is taken out-of-service.
The failover configuration normally starts automatically when the system is booted. The major difference between start-up of the failover configuration and the standard configuration is that the failover daemon is started in place of the Process Manager.
An entry is made in the system configuration file /etc/inittab for the failover daemon at the default multi-user run level. This insures that the failover daemon is always running when the system is booted multi-user. The setup is configured such that initd always restarts foverd in the event foverd fails. A modified /etc/init.d/transpath start up and/or shutdown script is provided to start the failover daemon automatically at system boot.
Each failover daemon has a state that determines its current functional role. These states are shown in the following table.
| State | Description |
|---|---|
Master | System is performing active call processing. |
Slave | System is standing by to take over from the Master in case the Master should fail. |
Init | System is initializing after re-boot or re-start. |
Out-of-Service | System has been commanded to go out of service. The system will not be available to process calls until it is commanded to go back in service. |
At any one time, one system is normally the "Master" and the other "Slave." Both systems may boot at the same time or one may be down for short periods. The failover daemons on each system negotiate Master/Slave status through the exchange of inter-process communications messages. A summary of these messages follows:
1. Each failover daemon exchanges periodic heartbeat messages with its partner. Messages are sent over two separate pathways which provides redundancy. The primary path is over an asynchronous serial link between the two machines. The secondary path uses datagrams over UDP sockets and a common Ethernet LAN.
2. A remote system is considered to be down when the receiver fails to receive 10 consecutive messages. This results in a delay of approximately 20 seconds before detection of system failure.
3. When each failover daemon receives a message from its counterpart, it compares the sender's role to its own. The result of the comparison determines which daemon becomes the master and which become the slave. The listing in the following table displays these relationships.
| Receivers State | Sender's State | Result |
|---|---|---|
init | init | Daemon with the earlier execution start time becomes the Master, the other becomes the Slave. |
init | Master | Receiver becomes the Slave. |
init | Slave | Daemon with the earlier execution start time becomes the Master, the other becomes the Slave. |
Slave | Master | No change. |
Master | Slave | No change. |
| Receivers State | Sender's State | Result |
|---|---|---|
Slave | Slave | Daemon with the earlier execution start time becomes the Master. |
Master | Master | Daemon with the later execution start time becomes the Slave. (This situation can only occur if the daemons are unable to communication over either of the redundant pathways.) |
init | out-of-service | Receiver becomes master immediately. |
slave | out-of-service | Receiver becomes master immediately. |
1. Each heartbeat message contains a sequence number. When the two failover daemons synchronize correctly, each increments the last received sequence number. Using this sequence number sends the next heartbeat message.
2. The receiver compares the sequence number it receives with its own internal sequence number. The receiver considers the message state and discards it when the sequence number received is smaller. Upon receiving a larger sequence number, the receiver updates the internal sequence number with the received sequence number.
3. If a remote system does not transmit a specified number of heartbeat messages in a reasonable time (two second timeout per message), the remote system is considered to have failed.
4. The initial sequence number when the failover daemon starts is 1.
5. Sequence numbers "roll over" to 1 after they have reached an upper limit (e.g., 32768).
6. When one daemon sends a heartbeat that it has gone out-of-service, the other immediately becomes the master. This provides a mechanism for commanded failover to the spare system.
1. Failover daemon stops data replication if it is active.
2. Failover daemon triggers switchover of the communications lines to the current system.
3. Failover daemon starts the XE Process Manager.
4. As long as the current system is Master, the Process Manager is automatically restarted upon termination. The Process Manager is designed to recover automatically upon restart.
1. Failover daemon commands the XE Process Manager to shut down the entire application, if it is running by sending a UNIX signal(SIGUSR1).
2. The daemon waits until it detects that the Process Manager has terminated.
3. Failover daemon starts Data Replication. Configuration files which are updated on the current Master system are automatically copied to the current Slave system.
1. Failover daemon commands the XE Process Manager to shut down the entire application, if it is running, by sending a UNIX signal(SIGUSR1).
2. The daemon waits until it detects that the Process Manager has terminated.
3. Failover daemon does not start Data Replication.
4. Failover partner system becomes the master immediately.
1. Failover daemon transitions back to the Init state.
2. Failover daemon transitions from the Init state to either the Master state or the Slave state depending on whether its partner is currently Master or not.
As it is not generally feasible to run "dual-port" disk drives on most UNIX systems, such that more than one system can access them simultaneously at the hardware controller level, failover uses NFS (Network File System) for file synchronization. NFS is a protocol developed by SUN Systems for distributing file systems across a network. NFS allows processes residing on one system to access files on another system as though they were local, and through the replication daemon enables configuration data from the Master system to be transferred to the Slave.
Following is the summary of the Data Replication approach:
1. Each system has its own disk drives with their own file systems. Data Replication duplicates, or shadows, certain files from the Master system to the Slave system, based on the last update time of each file. For increased reliability, each system may optionally use an independent RAID or mirrored disk set.
2. The file system structure on each system appears identical. Directories from each system are "cross-mounted" on the other system via NFS. For example, on the local system /opt/TransPath/etc is mounted while on the remote system /opt/remote/etc is mounted.
3. Exported file systems are "soft-mounted," so that if the remote system is not available, the start-up of the local system is not delayed. The installation procedure configures NFS so that the local system attempts to retry mounting the remote file system when the remote system is unavailable.
4. A dedicated daemon process that runs on the slave system performs Replication. Replication stops when the Slave system becomes the Master. This approach allows the Slave system to "catch-up" with the master system whenever the slave comes back on-line.
5. The replication daemon is configured with a list of files to monitor on the master and the slave systems. At regular intervals, the daemon wakes up and looks for files that were added, updated, or deleted on the master system since the last scan.
6. Copying of new files to the slave file system takes place upon the creation of new files. Copying of updated files to the slave system takes place in the same manner. Deleting files causes their removal from the slave system.
7. The replication daemon understands when NFS is unable to mount an exported file system when scanning reveals the file system is not below the expected mount point. It does not attempt to delete every replicated file on the slave system.
8. When a file is being copied, the replication daemon uses a temporary name. After transferring the file completely and successfully, the replication daemon renames the file to its correct name. If an error occurs during the transfer, the daemon deletes the temporary file.
It is the responsibility of the Master system to report any failover-related faults. The Master system uses the XE Alarm Manager to set and clear faults.
Alarms are set under the following conditions:
1. Loading of configuration data fails (informational).
2. Transition to Init State (informational).
3. Transition to Master State (informational).
4. Transition to Slave State (informational).
5. Transition to Out-of-service (severe).
6. Transition of partner system to Out-of-service (severe).
7. Communications with partner system lost (severe).
In addition, the failover configuration connects to an Alarm Relay Unit, furnished with the system. The system sends periodic heartbeat messages to the ARU whenever it is in Slave mode, which prevents any alarms from being relayed by the ARU. If the system is taken Out-of-service, or is shut down, the ARU automatically triggers alarms until the system is returned to operation in either the Master or Slave mode.
The failover daemon and replication daemon log messages to the system log files whenever any major event occurs. If you need to find out more about the current failover status on a system, you can refer to the log files.
Failover is normally started automatically when the system is booted. However, manual start-up is also possible using the following UNIX command:
sh /etc/init.d/transpath start
The failover configuration is normally shut down automatically when the system is shut down. However, manual shutdown is also possible using the following UNIX command:
sh /etc/init.d/transpath stop
The failover configuration can also be shut down manually by executing a provided shell script using the following UNIX command:
/opt/TransPath/local/cmd_failover.sh stop
This command must be executed from an account in the TransPath system group or from the root account. Also note that shut down of the Process Manager from the MML command line will not function the same way for the failover configuration as for the standard configuration. If the Process Manager is shut down manually for any reason, the failover daemon automatically restarts it. Therefore, use one of the shutdown procedures described in this section to shut down the failover configuration properly.
The failover configuration can be taken out-of-service (OOS) by executing a provided shell script using the following UNIX command:
/opt/TransPath/local/cmd_failover.sh oos
This command must be executed from an account in the TransPath system group or from the root account. If the current Master system is taken OOS, this automatically triggers failover to the Slave system. Be sure that the Slave system is fully operational before taking the current Master system OOS, otherwise there will be an interruption of service.
The failover configuration can be put back in-service by executing a provided shell script, using the UNIX command:
/opt/TransPath/local/cmd_failover.sh is
This command must be executed from an account in the TransPath system group or from the root account.
The following are explanations of failover configuration terms and commands:
Commanded Failover | Master of Slave system is controlled by the system administrator with "root" privileges through the use of the following command: |
Out-of-Service | Takes a failover daemon out-of-service from either the master or slave state. If running data replication (Slave) or TransPath (Master) will be halted and a running slave system will take over the role of Master. Use the following command: |
In Service | Brings an out-of-service failover daemon back into the operational state. If another system is running as Master, the reinstated daemon becomes Slave, otherwise it becomes the Master. Use the following command: |
Stop | Stops a failover daemon completely, halting the replication or TransPath application. This should only be restarted by following the proper startup procedures or a system boot. Use the following command: |
|
|