Table of Contents

Overview of the Data Management Package

Data Collection

Event Logs
IP Session Logs

Data Management
Data Analysis

Using the Data Management Package

Setting and Viewing Status Information

Overall Status
Trigger History and Configuration
Directory Summary for /usr/nr/var
Current Configuration of Actions

Generating and Customizing Reports with SQL Queries

Running the Interactive SQL Queries
Configuring the NetRanger SQL Queries

Default Oracle Schemas

nr_log_alarm and nr_log_alarm_1 Table Schemas
nr_log_context Table Schema
nr_log_tcpconn Table Schema
nr_log_error Table Schema
nr_log_command Table Schema

Managing Security Data

This chapter describes the following topics:

• Overview of the Data Management Package

• Using the Data Management Package

Overview of the Data Management Package

The NetRanger Data Management Package (DMP) provides key operational components of NetRanger:

• Data Collection

• Data Management

• Data Analysis

Together, these services allow NetRanger users to manage security data. The relationship among the parts is illustrated in Figure 7-1.

Functions

Data Collection

DMP collects two basic forms of data: high-level event data and detailed binary IP session data (as diagrammed in Figure 7-2). DMP uses two types of log files to capture these types of data:

• Event Logs

• IP Session Logs

Figure 7-2: Basic Log File Data Relationships

Event Logs

Event logs are ASCII files written to the /usr/nr/var directory with the naming convention of log.YYYYMMDDHHMM. By default, level 2-5 events are written to event logs on a Director system, and low-level 1 events are written to an event log on the Sensor system. Log files capture data on alarms, command, and errors.

Event alarm data are unique because the record format contains optional as well as fixed components.

Fixed alarm data include the following information:

• Which sensor detected the event

• When the alarm was generated

• The type of alarm

• The source and target of the event

The optional data field contains information that cannot be described by the fixed portion of the alarm record format. This information may be a string associated with an event, or context data that consists of a snapshot of incoming and outgoing TCP traffic (a 256-byte maximum in both directions).

For example, a typical line written to a log file would look like the following:

4,1025294,1998/04/16,16:58:36,1998/04/16,11:58:36,10008,11,100,OUT,OUT,1,2001,0,TCP/IP,10.1.6.1,10.2.3.5,0,0,0.0.0.0,

Each comma-delimited field represents a different type of data. The first value tells you what kind of record is being logged. The following example log file references values and record types.

Field Value	Record Type
0	Default
1	Command
2	Error
3	Command Log
4	Event
5	IP Log
6	Redirect

In the example log file on the previous page, 4 denotes an Event record. The following table provides a reference for the rest of the fields in the sample Event record.

Sample Field Value	Field Type
4	Record Type
1025294	Record ID
1998/04/16	GMT Datestamp
16:58:36	GMT Timestamp
1998/04/16	Local Datestamp
11:58:36	Local Timestamp
10008	Application ID
11	Host ID
100	Organization ID
OUT	Source Direction
OUT	Destination Direction
1	Alarm Level
2001	SigID
0	SubSigID
TCP/IP	Protocol
10.1.6.1	Source IP Address
10.2.3.5	Destination IP Address
0	Source Port
0	Destination Port
0.0.0.0	Router IP Address

Two other types of records, Error and Command Log, are similar in structure to the Event record. Both of these record types have the same first eight fields contained in Event records (Record Type, Record ID, GMT Date and Timestamp, Local Date and Timestamp, Application ID, Host ID, and Organization ID).

An Error record type has a ninth field denoting the actual Error String generated by a service. The Command Log record type also contains fields for the source Application ID, Host ID, and Organization ID, as well as the Command String. In both cases, you will have a complete record of errors and commands.

IP Session Logs

IP Session logs capture all incoming and outgoing TCP packets associated with a specific connection, and therefore contain binary data. They are written to a Sensor's /usr/nr/var/iplog directory with the naming convention of iplog.source_IP_address.

In addition to detecting attack signatures, sensord and packetd are able to monitor the traffic associated with a specific type of attack. For example, sensord can be configured to monitor all of the packets associated with an IP spoof. sensord or packetd creates a separate log file in /usr/nr/var/iplog for each of these monitoring sessions. The name of each session log file is based on the IP address of the attacking host (for example, iplog.10.145.16.152).

There is a performance penalty associated with the logging of context data. DMP provides two options for controlling overhead:

• You can set the Minimum Context Level in nrConfigure's Data Management dialog box.

• You can set the SAP_EXCLUDE_CONTEXT token in the /usr/nr/bin/sap/load_run.sh script to determine exactly what levels and types of event data should be exported onto a database.

Context data is not loaded if this variable is set.

By default, level 1 Event and IP Session logs are left on a Sensor until they are required. This prevents the relatively large amounts of data associated with these types of events from impeding NetRanger communications during periods of network load.

Even though the DMP is designed to transfer NetRanger data into industrial strength databases, both types of data are initially written to flat files for two reasons: speed and fault tolerance. Data can be written to a flat file much faster than to a database, and access to flat files is guaranteed as long as the host system is operational. Databases, on the other hand, are subject to unpredictable load, and contain too many access layers to be truly fault tolerant---if the database is down, there are no places to turn.

Data Management

NetRanger uses a simple push-pull process to migrate data from flat files to a database or data archive. Figure 7-3 diagrams the push-pull process for both types of log files.

The push process involves the loggerd service to write event, command, and error notifications into a single flat file in /usr/nr/var. This file is serialized based on configurable size and time thresholds set via nrConfigure. Once the current log file exceeds one of these thresholds, it is replaced by a fresh log file and moved to /usr/nr/var/new.

The pull process involves the sapd service, which relies on its own size and time thresholds. sapd pulls the oldest file from /usr/nr/var/new into /usr/nr/var/tmp and executes database load procedures specified by the user. After the log file is successfully loaded into a database, it is placed in /usr/nr/var/dump where it is processed by a user-defined purge process.

This basic push-pull process applies to IP session logs as well as event logs. The primary differences are that IP session logs are written to /usr/nr/var/iplog rather than /usr/nr/var/new, and they are automatically placed in /usr/nr/var/dump after a configurable amount of time.

Figure 7-3: The Push-Pull Process

Data Analysis

After data is loaded into a database, it can be analyzed for patterns and trends. Status reports relating to network activity and vulnerabilities can also be generated. Although any number of different third-party tools can generate these types of output, DMP is shipped with a small but comprehensive collection of Oracle SQL*Plus queries that show how event data can be analyzed relative to different perspectives, such as time versus events.

Using the Data Management Package

This section provides information on the following:

• Setting and Viewing Status Information

• Generating and Customizing Reports with SQL Queries

• Default Oracle Schemas

Setting and Viewing Status Information

This section describes the following topics:

Before generating or customizing reports with SQL queries, you need to learn how to configure and view data management status information. This information gives you insight into the following indicators:

• Overall Status

• Trigger History and Configuration

• Directory Summary for /usr/nr/var

• Current Configuration of Actions

You can view File Management status in Brief or Full mode. Brief mode provides a summarized view of the FileMgmt information, and includes overall status and uptime, a summation of errors, and a trigger history. Full mode is a superset of Brief mode, and also includes information on scheduler status, normal summation, and trigger configuration.

To set for your File Management viewing mode, follow these steps:

Step 1 On the Director interface, click Configure on the Security menu.

nrConfigure opens.

Step 2 Double-click Data Management.

The Data Management dialog box opens (see Figure 7-4).

Step 3 Click the General tab.

Figure 7-4: nrConfigure's Data Management Dialog Box

Overall Status

On the Overall Status Display (see Figure 7-5) an Overall Status line (the 1st line in the status display) indicates that sapd keeps track of its overall status; that is, if a launched action returns indicating an error condition, sapd knows that an error has occurred. If an error occurs, the Overall Status line reads something like the following:

Overall Status = ERROR Oracle_Load ;

Figure 7-5: Database Overall Status Display

Overall Status = Normal

Trigger History and Configuration

The Trigger History and Configuration sections (see Figure 7-6 and Figure 7-7) provide information on recent triggers run, and the configuration status of triggers.

The information summarized in the Trigger History section consists of the following:

• Trigger status (either Normal or ERROR)

• Number of times the trigger has generated a status (if Normal, the number increments in the OK column; if ERROR, the number increments in the Error column).

• Trigger name

• Trigger's corresponding action

• The last time the trigger ran

Figure 7-6: Trigger History Display

Trigger Configuration Display

To find out how to change the configuration of triggers, refer to the "Configuring Data Management" section of "Configuration Management."

Directory Summary for /usr/nr/var

The Directory Summary section (see Figure 7-8) provides information on each directory in /usr/nr/var, the disk partition in which NetRanger security data is logged and staged to third-party database or storage tools. The display summarizes the following information for each directory:

• Directory name

• Number of files in the directory

• Total number of bytes in the directory

• Date- and timestamp of the oldest file in the directory

• Date- and timestamp of the newest file in the directory

Figure 7-8: Directory Summary Display

Current Configuration of Actions

The Action Configuration section (see Figure 7-9) displays the current configuration of all database actions. The display summarizes the following information:

• Action name

• Associated system command, with arguments

Figure 7-9: Action Configuration Display

To find out how to change the configuration of actions, refer to the "Configuring Data Management" section of "Configuration Management."

Generating and Customizing Reports with SQL Queries

NetRanger ships with SQL queries that can be run automatically by NetRanger or interactively by a human operator. The SQL queries themselves are the same in either case, but the entry methods differ.

Interactive queries are run by a human operator in the SQL+ environment, and are distinguished by the "@" character at the start of each query command. The four commands are: @event, @space, @time, and @system.

Automatic queries are run in batch mode by NetRanger. Information on when they are run and what the output looks like can be set in NetRanger's configuration files and in the Oracle report templates.

This section describes the following topics:

• Running the Interactive SQL Queries

• Configuring the NetRanger SQL Queries

Running the Interactive SQL Queries

The queries shipped with the DMP generate basic reports, which are accessed via the following interactive SQL entry points:

• event.sql---The event queries return summary information on what alarm levels, signatures, and string matches were detected by NetRanger.

• space.sql---The space queries build on the event queries, adding summary information about where alarms occurred---the source and destination addresses involved in the events.

• time.sql---The time queries build on the space queries, and summarize when security events occurred. You can vary the level of granularity by choosing the following time segments: monthly, daily, hourly, and every ten minutes.

• system.sql---The system queries return information on records in tables, such as counts and types of signatures and organizations.

In turn, each of these queries provides access to a number of different subqueries, such as event1.sql, event2.sql, and so on. The query you run depends on the answers you give to the initial (event/space/time/system) query. Basically, the greater the number of the subordinate query, the more detail is returned.

Table 7-1 lists the SQL user commands and subordinate queries.

Example 7-1 illustrates the interactive component of an event query.

SQL> @event
EVENT DIMENSION QUERIES
================================================
1    alarm level summary
2    alarm signature summary
3    alarm signature with string data summary
These queries summarize Sensor alarms with a primary focus
on the EVENT dimension.
You can filter the data returned to you with the following
criterion:
ORGANIZATION_NAME
MIN_EVENT_DATE

Please enter your desired value at the following prompts:
select query (1,2,3)>1
organization name (%)>%
minimum event date (YYYY/MM/DD)>1998/05/01
 

In Example 7-1, typing 1 selects an alarm level summary. Typing the wildcard character (%) for the organization name provides a listing of alarms for all NetRanger organization names. Typing an actual organization name at this prompt will narrow the returned information.

The results of this interactive query are illustrated in Example 7-2.

event1: alarm level summary
organization_name = %
minimum_event_date = 1998/5/1
Org Name	Level	From	To	Count	Recent
---------------	--------	----	---	-------	-----------
Data Warehouse	 5	IN	OUT	3	05:02 08:45
Net Systems	 5	IN	IN	15	05:02 11:21
Net Systems	 3	OUT	OUT	4	05:01 19:15
Net Systems	 3	IN	IN	2	05:01 15:03
Net Systems	 3	IN	OUT	79	05:02 09:28
Cisco Systems	 3	IN	IN	231	05:02 09:37
Cisco Systems	 3	IN	OUT	176	05:02 09:37
Cisco Systems	 2	OUT	IN	9110	05:02 12:15
Cisco Systems	 2	IN	IN	18	05:01 14:06
 

Configuring the NetRanger SQL Queries

The NetRanger-controlled SQL queries are set to run in daily, weekly, and monthly batch modes. You can use nrConfigure's Data Management dialog box (see Figure 7-10) to change the scheduling of reports as well as the content and organization of report data.

To change the scheduling of reports, you need to customize the following triggers:

• Report_Day

• Report_Week

• Report_Month

For more information on customizing triggers, refer to the "Setting Triggers" section of "Configuration Management."

Figure 7-10: Setting Report Triggers with nrConfigure

event1 % MIN_EVENT_DATE
 

event9 % MIN_EVENT_DATE

Default Oracle Schemas

The native Oracle schema supported by the DMP are illustrated in Figure 7-11. Each box on the left identifies how event data is grouped prior to being loaded into a database table. Each box on the right represents the target database tables. The schemas for these tables are defined on the next several pages.

Figure 7-11: Grouping of Event Data into Target Database Tables

nr_log_alarm and nr_log_alarm_1 Table Schemas

The nr_log_alarm and nr_log_alarm_1 tables contain information about level 2-5 and level 1 alarms, respectively. The schema for the nr_log_alarm and nr_log_alarm_1 tables are defined in Table 7-2.

nr_log_context Table Schema

The nr_log_context table contains information on alarm contexts. NetRanger buffers
256 bytes of context data in each direction (incoming and outgoing). However, to accommodate a maximum of three escaped characters per context byte, the field size for the DATA_INCOM and DATA_OUTGO fields must accommodate 768 (256 x 3) bytes. The following occurs during translation:

• Normal characters are printed as themselves:

CONTEXT CHAR=4A	PRINTABLE CHAR=J
CONTEXT CHAR=4B	PRINTABLE CHAR=K
 

• The "\" character is escaped in the output as "\\":

CONTEXT CHAR=5C	PRINTABLE CHAR=\\
 

• Non-printable characters are escaped with the "\" character and printed in hexadecimal format:

CONTEXT CHAR=0D	PRINTABLE CHAR=\0D
CONTEXT CHAR=FE	PRINTABLE CHAR=\FE
 

The schema for the nr_log_context table is defined in Table 7-3.

nr_log_tcpconn Table Schema

The nr_log_tcpconn table contains information on TCP connections. The schema for the nr_log_tcpconn table is defined in Table 7-4.

nr_log_error Table Schema

The nr_log_error table contains information on errors in NetRanger data. The schema for the nr_log_error table is defined in Table 7-5.

nr_log_command Table Schema

The nr_log_command table contains information on NetRanger commands executed. The schema for the nr_log_command table is defined in Table 7-6.

Posted: Fri Jan 29 09:18:57 PST 1999
Copyright 1989-1999©Cisco Systems Inc.