Customizing FlowCollector

This chapter describes how to customize FlowCollector operation using thread, filter, and protocol definitions, lists of port and autonomous system numbers, and other FlowCollector configuration parameters.

• Before You Begin

• Understanding FlowCollector Data Collection and Aggregation

• Creating a Filter

• Creating a Thread

• Defining Protocols

• Defining Source and Destination Port Numbers

• Defining Source and Destination Autonomous System Numbers

• Modifying FlowCollector Resources

• Managing Disk Space

• Where to Go from Here

Before You Begin: FlowCollector Configuration and Resource Files

The process of customizing FlowCollector operation involves changes and additions to one or more of the following FlowCollector configuration and resource files:

• nfconfig.file

• nfknown.protocols

• nfknown.srcports

• nfknown.dstports

• nfknown.srcasns

• nfknown.dstasns

• nf.resources

nfconfig.file

The nfconfig.file contains definitions of the aggregation tasks that collect and aggregate data exported from NetFlow export devices in your network. These aggregation tasks, defined in terms of threads and filters, tell FlowCollector how to collect and aggregate the incoming NetFlow export data. Each aggregation task must have a thread defined for it (filters are optional).

For more information about creating or modifying threads and filters, refer to the section "Understanding FlowCollector Data Collection and Aggregation," later in this chapter.

nfknown.protocols

The nfknown.protocols file contains definitions of recognized Application layer protocols (FTP, Telnet, etc.) for use in aggregating data. These definitions are also used for protocol filters. You edit this file to add or remove protocol definitions. FlowCollector scans this file and maintains a list of protocols it finds. FlowCollector searches the protocols in the order they are defined in nfknown.protocols.

For more information about creating or modifying protocols, refer to the section "Defining Protocols," later in this chapter.

nfknown.srcports

The nfknown.srcports file contains Transport layer source TCP or UDP port numbers used in the SourcePort aggregation scheme (or any other aggregation scheme using source port numbers as part of its key). These TCP or UDP port numbers correspond to the port numbers defined in RFC 1700, for example, Telnet = 23, FTP = 20 or 21.

For more information about creating or modifying source port numbers, refer to the section "Defining Source and Destination Port Numbers," later in this chapter.

nfknown.dstports

The nfknown.dstports file contains destination port numbers used in the DestPort aggregation scheme (or any other aggregation scheme using destination port numbers as part of its key).

For more information about creating or modifying destination port numbers, refer to the section "Defining Source and Destination Port Numbers," later in this chapter.

nfknown.srcasns

The nfknown.srcasns file contains source autonomous system numbers, either origin or peer, used in the ASMatrix and DetailASMatrix aggregation schemes (or any other aggregation scheme using source autonomous system numbers as part of its key).

For more information about creating or modifying source autonomous system numbers, refer to the section "Defining Source and Destination Autonomous System Numbers," later in this chapter.

nfknown.dstasns

The nfknown.dstasns file contains destination autonomous system numbers, either origin or peer, used in the ASMatrix and DetailASMatrix aggregation schemes (or any other aggregation scheme using destination autonomous system numbers as part of its key).

For more information about creating or modifying destination autonomous system numbers, refer to the section "Defining Source and Destination Autonomous System Numbers," later in this chapter.

nf.resources

The nf.resources file contains the variables and corresponding directory file path names used to configure your startup FlowCollector environment. Besides the path names, the nf.resources file also includes a number of configuration parameters for tuning FlowCollector performance and behavior.

Understanding FlowCollector Data Collection and Aggregation

FlowCollector collects and summarizes (aggregates) data into data files based on user-defined criteria specified in a FlowCollector thread. A thread is an aggregation task defined by a set of user-configurable attributes that specify how FlowCollector will aggregate the traffic flows stored on the workstation. Two key thread attributes are:

• Aggregation schemes--Define how to aggregate the traffic flows stored on the workstation

• Filters--Define the flow information that is accepted or rejected by the aggregation scheme

Figure 6-1 shows an example of how FlowCollector uses threads and filters. In this example, threadA uses filterA and the SourceNode aggregation scheme; threadB uses both filterA and filterB (filters can be shared among threads) and the DestPort aggregation scheme; threadC doesn't use any filters, but it also uses the DestPort aggregation scheme.

The following nf.config file example contains the thread definitions to accomplish the general data aggregation scheme shown in Figure 6-1. The file contains two filter definitions (filterA and filterB) and three thread definitions (threadA, threadB, and threadC) to accomplish the aggregation. NetFlow export traffic arrives on FlowCollector UDP port 9991. Data is to be aggregated by the following aggregation schemes:

• SourceNode (Source IP address)

• DestPort (Destination port number)

Filter filterA
       permit     nexthop    172.16.23.65        0.0.0.0
Filter filterB
       deny       srcaddr    172.16.0.0          0.0.255.255
       permit     srcaddr    0.0.0.0             0.0.0.0
Thread threadA
       Filter filterA
       Aggregation SourceNode
       Period 10
       Port 9991
       State Active
       DataSetPath /opt/CSConfc/Data
       DiskSpaceLimit 0
       FileRetain 0
Thread threadB
       Filter filterA
       Filter filterB
       Aggregation DestPort
       Period 10
       Port 9991
       State Active
       DataSetPath /opt/CSCOnfc/Data
       DiskSpaceLimit 0
       FileRetain 0
Thread threadC
       Aggregation DestPort
       Period 10
       Port 9991
       State Active
       DataSetPath /opt/CSCOnfc/Data
       DiskSpaceLimit 0
       FileRetain 0

In the example, threadA uses filterA to include only the traffic passing through the export device 172.16.23.65, while threadB uses both filters--filterA to include only the traffic passing through the export device 172.16.23.65, and filterB to exclude traffic from network 172.16.0.0. All three threads flush their aggregated data every 10 minutes into data files saved in the /opt/CSCOnfc/Data directory.

Creating a Filter

A filter defines which flow data is to be included or excluded as FlowCollector aggregates data. The default condition for a filter is to deny (exclude) the flow.

filter filter-name
       {permit|deny} type value mask
                  .
                  .
                  .
       {permit|deny} type value mask

where:

Filter keyword and variable entries are not case-sensitive. Table 6-1 describes the default filter types provided with FlowCollector, the type of input required for the value, and whether the value requires a mask.

When defining a filter, keep in mind the following qualifications:

• You must use an explicit permit statement to permit flows that do not meet defined filtering conditions; otherwise, everything in the flow is denied. (The default condition for a filter is to deny the flow.) For example:
  filter kill-www
         deny Dstport 80
  
  

In this example, all flows going to port 80 and all other flows are denied. If you want to deny flows to port 80 only, but permit all other flows, you need an explicit wildcard entry to permit the other flows. For example:

filter kill-www
       deny Dstport 80
       permit Dstaddr 0.0.0.0 255.255.255.255

• When multiple filter conditions exist, FlowCollector attempts to apply the conditions sequentially, in the order you specify, until a match is found.

For example, the filter named filterA contains four filter conditions.The first condition states that all flows from network 171.69.1.0 are permitted. Also, based on the order in which filter condition definitions appear in this example, filterA allows a flow from Srcaddr = 171.69.1.24 with Srcport = 53.

Filter filterA
       permit     Srcaddr        171.69.1.24       0.0.0.255
       deny       Srcaddr        204.233.0.0       0.0.255.255
       deny       Srcport        53
       permit     Dstaddr        0.0.0.0           255.255.255.255

If you want to permit traffic from network 171.69.1.0, but deny traffic coming from port 53, you should change the order of the filter conditions as follows:

Filter filterA
       deny       Srcaddr        204.233.0.0       0.0.255.255
       deny       Srcport        53
       permit     Srcaddr        171.69.1.24       0.0.0.255
       permit     Srcaddr        0.0.0.0           255.255.255.255

The last filter condition overrides the default behavior, which calls for denying all flows that do not match any of the first three filter conditions.

Creating a Thread

A thread is a set of defined attributes that tells FlowCollector how to aggregate the traffic flows stored on the workstation.

The syntax for a thread definition is as follows:

Thread thread-name
[Filter filter-name]
          .
          .
          .
[Filter filter-name]
Aggregation scheme 
Period minutes
Port value
DataSetPath directory-path
[DiskSpaceLimit Megabytes]
State active|inactive
FileRetain number

The keywords and their arguments are listed on separate lines for legibility. Keyword and argument entries are not case-sensitive. Table 6-2 lists thread attributes and variables.

Thread Alpha
       Aggregation SourceNode 
       Period 30
       Port 9991
       State Active
       DataSetPath /opt/CSCOnfc/Data
       DiskSpaceLimit 0
       FileRetain 24

Aggregation Schemes

FlowCollector provides a library of predefined aggregation schemes (see Table 6-3) that you can use to determine the type of information that is aggregated and stored in the data files.

Each aggregation scheme consists of one or more key fields, which tell FlowCollector what to look for in the exported NetFlow datagram, and one or more value fields, which contain statistical information pulled from the exported NetFlow datagram. The key fields and value fields shown in Table 6-3 correspond to the fields found in Version 1, Version 5, and Version 7 NetFlow export datagrams. Table 6-4 provides brief definitions of each of the key and value fields. For more information about these three versions of the NetFlow export datagram format, refer to the appendix "NetFlow Export Datagram Format," later in this guide.

For example, the SourceNode aggregation scheme uses just one key field, srcaddr (source address), and returns data for three value fields, the total number of packets sent, the total number of bytes sent, and the total number of flows aggregated into this record. Other aggregation schemes offer different combinations of key and value fields (see Table 6-3), and are described individually in the sections that follow.

RawFlows

The output from this aggregation scheme is an exact image of the NetFlow export datagram without aggregation, and is stored in binary data files of n minutes worth of data as specified by the Period attribute in the thread definition.

SourceNode

The output of this aggregation scheme consists of one record for each unique source IP address present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

DestNode

The output of this aggregation scheme consists of one record for each unique destination IP address present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

HostMatrix

The output of this aggregation scheme consists of one record for each unique source and destination IP address pair present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

SourcePort

The output of this aggregation scheme consists of one record for each unique source port present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

DestPort

The output of this aggregation scheme consists of one record for each unique destination port present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Protocol

The output of this aggregation scheme consists of one record for each unique protocol (as defined in the nfknown.protocols file) present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

DetailDestNode

The output of this aggregation scheme consists of one record for each unique combination of destination IP address, source port, destination port, and protocol present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

DetailHostMatrix

The output of this aggregation scheme consists of one record for each unique combination of source IP address, destination IP address, source port, destination port, and protocol present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

DetailInterface

The output of this aggregation scheme consists of one record for each unique combination of source IP address, destination IP address, input, output, and nexthop present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

CallRecord

The output of this aggregation scheme consists of one record for each unique combination of source IP address, destination IP address, source port, destination port, protocol, and type of service present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

ASMatrix

The output of this aggregation scheme consists of one record for each unique source and destination autonomous system number pair present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

NetMatrix

The output of this aggregation scheme consists of one record for each unique combination of input interface, output interface, masked source IP address, masked destination IP address, source mask, and destination mask present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

DetailSourceNode

The output of this aggregation scheme consists of one record for each unique combination of source IP address, source port, destination port, and protocol present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

DetailASMatrix

The output of this aggregation scheme consists of one record for each unique combination of source IP address, destination IP address, source port, destination port, protocol, input interface, output interface, source autonomous system number, and destination autonomous system number present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Defining Protocols

Use the information in this section to define the protocols that you want FlowCollector to recognize as it aggregates data. The protocols FlowCollector recognizes are defined in the nfknown.protocols file, located in the $NFC_DIR/config directory.

• The protocol definition must be added to the nfknown.protocols file

• There must be an active thread calling for this protocol

• There must be NetFlow data traffic that applies to the protocol

Figure 6-2 shows an example of a typical communication session between host A and host B. This example assumes that NetFlow data export is enabled for the export device interfaces to both host A and host B so that exported NetFlow data gives FlowCollector statistics for communication in both directions (A to B; B to A). In this example, FlowCollector aggregates data for two protocols-- Telnet and FTP--between host A (the Telnet server, using port 23) and host B (the Telnet client, using port 9001).

• For Telnet, FlowCollector aggregates data between host A (the Telnet server, using port 23) and host B (the Telnet client, using port 9001).

• For FTP, FlowCollector aggregates data between host A (the FTP server, using port 20) and host B (the FTP client, using port 9002).

Whether the aggregated data is stored in data files for later retrieval depends on how FlowCollector is customized.

Table 6-5 lists the raw data received in the data collection example shown in Figure 6-2.

When you add the Telnet protocol definition to the nfkown.protocols file, the data file produced by the Protocol aggregation scheme will contain a row with 50 packets (20 plus 30) and 3000 bytes (2000 plus 1000).

In this example, no FTP protocol definition was added to the nfkown.protocols file, so the data file will also have another row for "Others" (including the FTP data) containing 70 packets (20 plus 50) and 500 bytes (200 plus 300).

The protocols listed in the nfknown.protocols file are used by the aggregation schemes and protocol filters you define in the nfconfig.file. To configure the protocols that FlowCollector recognizes, you must edit the nfknown.protocols file and add a definition that includes the following information:

The command syntax for a protocol definition is

protocol name
       [[srcport|dstport] number [OR [srcport|dstport] number]]
         prot value

where:

The known protocols (such as WWW, Telnet, FTP) listed in the nfknown.protocols files are similar to the definitions specified in the /etc/services file of a UNIX workstation. For information about the protocols and protocol types supported on your workstation, refer to the protocols file in the /etc directory on your workstation.

Protocol TCP-Telnet
         Dstport 23 OR Srcport 23
         Prot 6
Protocol TCP-FTP
         Srcport 20 OR Srcport 21 OR Dstport 20 OR Dstport 21
         Prot 6
Protocol TCP-WWW
         Dstport 80 OR Srcport 80
         Prot 6
Protocol TCP-SMTP
         Srcport 25 OR Dstport 25
         Prot 6
Protocol TCP-Other
         Prot 6
Protocol UDP-TFTP
         Srcport 69 OR Dstport 69
         Prot 17

Defining Source and Destination Port Numbers

Use the information in this section to specify the source and destination port numbers from which FlowCollector will collect and aggregate data. The port numbers FlowCollector recognizes are defined in the following files, located in the $NFC_DIR/config directory:

• nfknown.srcports--This file contains the source port numbers that you want FlowCollector to recognize.

• nfknown.dstports--This file contains the destination port numbers that you want FlowCollector to recognize.

The command syntax for a port number, range of port numbers, or a range grouped under an assigned label is:

value[,value[:label]]
           .
           .
           .
value[,value[:label]]

where:

50, 100

10000, 19999: 10K_19K_Port_Range

The following example shows the contents of a sample nfknown.srcports file:

21:ftp
88
50, 100
10000, 19999: 10K_19K_Port_Range
20000, 29999: My_Range
40000, 49999: My_Range

Defining Source and Destination Autonomous System Numbers

Use the information in this section to specify the source and destination autonomous systems from which FlowCollector will collect and aggregate data. The autonomous systems FlowCollector recognizes are defined in the following files, located in the $NFC_DIR/config directory:

• nfknown.srcasns--This file contains the source autonomous system numbers, either origin or peer, that you want FlowCollector to recognize.

• nfknown.dstasns--This file contains the destination autonomous system numbers, either origin or peer, that you want FlowCollector to recognize.

1:Your_Network
2
10, 15
20, 30:My_Network
35, 40:My_Network

Modifying FlowCollector Resources

The nf.resources file contains the configuration parameter settings and directory file path names used to configure your startup FlowCollector environment. Besides the path name definitions, the nf.resources file also includes a set of parameters for tuning FlowCollector performance. The nf.resources file is located in $/NFC_DIR/config.

Table 6-6 describes the available configuration parameters and their values.

Increasing UDP Socket Receive Buffer Size

Due to the high volume of NetFlow data export traffic, you might have to increase the normal buffer size associated with the UDP socket on which data is received. To do so, edit the value (in bytes) of the SOCKET_BUFSIZE parameter in the $NFC_DIR/config/nf.resources file.

Mapping a List of IP Addresses to One IP Address or Label

You can substitute a user-specified IP address or label for a set of IP addresses from which FlowCollector receives NetFlow export datagrams. For example, you can specify the label "blab-gateway" as the label representing packets coming from three separate IP addresses: 171.69.1.172, 171.69.1.173, and 191.71.1.25.

To do this, you must edit the ROUTER_GROUPNAME parameter in the nf.resources file. The syntax is

ROUTER_GROUPNAME label {
     a.b.c.d
        .
        .
        .
     w.x.y.z
}

ROUTER_GROUPNAME blab-gateway {
     171.69.1.172
     171.69.1.173
     191.71.1.25
}

Preventing FlowCollector from Accepting Unsolicited Packets

In its default configuration, FlowCollector accepts NetFlow export packets from any IP address. If necessary, you can specify the source IP addresses or defined ROUTER_GROUPNAME labels from which FlowCollector should receive NetFlow export packets, thus preventing FlowCollector from accepting packets from any unspecified sources.

To do this, you must remove the comment character from the beginning of each line in the ACCEPT_PACKETS_FROM parameter in the nf.resources file and edit the parameter to include the source IP addresses or ROUTER_GROUPNAME labels. The syntax of the parameter is

ACCEPT_PACKETS_FROM {
     a.b.c.d
        .
        .
        .
     w.x.y.z
}

ACCEPT_PACKETS_FROM {
     131.108.2.1
     131.108.2.2
     131.108.2.3
     blab_gateway
}

For information on ROUTER_GROUPNAME labels, refer to the section "Mapping a List of IP Addresses to One IP Address or Label," earlier in this chapter.

Retaining Router IP Addresses for Switched Export Packets

If your network includes switching devices that support Version 7 NetFlow export datagrams, you can configure FlowCollector to retain the IP address of the short-cut router as the source of data switched through a Cisco Catalyst 5000 series switch. To do this, you must edit the USE_SHORT_CUT_ADDRESS_AS_SOURCE_IP parameter in the nf.resources file. The syntax of the parameter is

USE_SHORT_CUT_ADDRESS_AS_SOURCE_IP value

Using a User-Defined Script to Process FlowCollector Data Files

You can specify the location of a script file that FlowCollector will execute after it has written a new data file. This capability makes it easier for your client applications to process a new data file without having to poll for it. FlowCollector invokes the script with the absolute path name of the newly written FlowCollector data file. FlowCollector expects the location of your user-supplied script to be defined by the USER_SCRIPT_LOCATION parameter in the nf.resources file. This parameter is read only at startup.

Changing the Output Buffer Size

FlowCollector transfers output data in blocks to optimize performance and ensure the most efficient handling of data files as they are generated and written as disk files. The size of a block is user-configurable, and defined by the OUTPUT_BUFFER_SIZE parameter in the nf.resources file. The syntax of the parameter is

OUTPUT_BUFFER_SIZE size

Managing Disk Space

Depending on the volume of flow data being exported from the export devices, as well as the FlowCollector thread attribute settings you use, FlowCollector can consume large amounts of disk space in a short period of time. FlowCollector provides several thread attributes and features that can help you manage your disk space usage:

• Filters (cannot be used with the RawFlows aggregation scheme)

• Aggregation schemes

• FileRetain thread attribute

• DiskSpaceLimit thread attribute (cannot be used with the RawFlows aggregation scheme)

Filters

As described earlier, a filter can help you discard any flow data that is not of interest to you. By using filters to ensure you are storing only data of interest, you can potentially reduce the amount of disk space used by FlowCollector.

Aggregation Schemes

Aggregation schemes are used to define how you want FlowCollector to summarize the flow data being exported from your export devices. By using only those aggregation schemes required for your application and, when possible, by selecting the aggregation schemes that generate the least amount of data on disk, you can reduce the amount of disk space used by FlowCollector. For example, using the HostMatrix aggregation scheme will result in less disk space usage than would the DetailHostMatrix scheme. Of course, which aggregation schemes you use will be determined primarily by the data you are interested in and how you want to summarize that data. It is important to realize, however, that the different aggregation schemes can greatly affect the amount of disk space used by FlowCollector.

Table 6-7 shows an example of some aggregation schemes with example file sizes and arrival rate.

FileRetain Thread Attribute

The FileRetain thread attribute allows you to limit the number of files FlowCollector will retain on disk per day for the given thread. By setting the FileRetain attribute to a lower value, you can reduce the amount of disk space FlowCollector uses by reducing the number of retained data files.

DiskSpaceLimit Thread Attribute

The DiskSpaceLimit thread attribute allows you to limit the total amount of disk space a thread can use in the partition specified by the DataSetPath attribute. Beyond the limit specified by the DiskSpaceLimit attribute, FlowCollector will no longer write data to the disk for the given thread.

The features and thread attributes mentioned above will help you to manage disk space usage by FlowCollector. You may need to use your own file archival and deletion techniques to save older data files and prevent disk space exhaustion.

Where to Go from Here

The appendixes in this guide provide information on the following topics: