Table of Contents

Before You Begin: FlowCollector Configuration and Resource Files

nfconfig.file
nfknown.protocols
nfknown.srcports
nfknown.dstports
nfknown.srcasns
nfknown.dstasns
nf.resources
nfcd.config

Understanding FlowCollector Data Collection and Aggregation
Creating a Filter
Creating a Thread
Aggregation Schemes

RawFlows
SourceNode
DestNode
HostMatrix
SourcePort
DestPort
Protocol
DetailDestNode
DetailHostMatrix
DetailInterface
CallRecord
ASMatrix
NetMatrix
DetailSourceNode
DetailASMatrix
RouterAS
RouterProtoPort
RouterSrcPrefix
RouterDstPrefix
RouterPrefix
ASHostMatrix
HostMatrixInterface
DetailCallRecord

Defining Protocols
Defining Source and Destination Port Numbers
Defining Source and Destination Autonomous System Numbers
Modifying FlowCollector Resources

Increasing UDP Socket Receive Buffer Size
Mapping a List of IP Addresses to One IP Address or Label
Preventing FlowCollector from Accepting Unsolicited Packets
Retaining Router IP Addresses for Switched Export Packets
Using a User-Defined Script to Process FlowCollector Data Files
Changing the Output Buffer Size

Configuring the Daemon (NFCD)
Managing Disk Space

Filters
Aggregation Schemes
NF_Thread Data File and Disk Space Parameters
Parameters in the nf.resources File
Unsolicited Event Notifications

Customizing FlowCollector

This chapter describes how to customize FlowCollector operation using thread, filter, and protocol definitions, lists of port and autonomous system numbers, and other FlowCollector configuration parameters.

This chapter includes the following sections:

• Before You Begin

• Understanding FlowCollector Data Collection and Aggregation

• Creating a Filter

• Creating a Thread

• Aggregation Schemes

• Defining Protocols

• Defining Source and Destination Port Numbers

• Defining Source and Destination Autonomous System Numbers

• Modifying FlowCollector Resources

• Configuring the Daemon (NFCD)

• Managing Disk Space

Before You Begin: FlowCollector Configuration and Resource Files

The process of customizing FlowCollector operation involves changes and additions to one or more of the following FlowCollector configuration and resource files located in the $NFC_DIR/config directory:

• nfconfig.file

• nfknown.protocols

• nfknown.srcports

• nfknown.dstports

• nfknown.srcasns

• nfknown.dstasns

• nf.resources

• nfcd.config

nfconfig.file

The nfconfig.file contains definitions of the aggregation tasks that collect and aggregate data exported from NetFlow export devices in your network. These aggregation tasks, defined in terms of threads and filters, tell FlowCollector how to collect and aggregate the incoming NetFlow export data. Each aggregation task must have a thread defined for it (filters are optional).

For more information about creating or modifying threads and filters, see the "Understanding FlowCollector Data Collection and Aggregation" section.

nfknown.protocols

The nfknown.protocols file contains definitions of recognized application layer protocols (FTP, Telnet, and so forth) for use in aggregating data. These definitions are also used for protocol filters. You edit this file to add or remove protocol definitions. FlowCollector scans this file and maintains a list of protocols it finds. FlowCollector searches the protocols in the order that they are defined in nfknown.protocols.

For more information about creating or modifying protocols, see the "Defining Protocols" section.

nfknown.srcports

The nfknown.srcports file contains transport layer source TCP or UDP port numbers used in the SourcePort aggregation scheme (or any other aggregation scheme using source port numbers as part of its key). These TCP or UDP port numbers correspond to the port numbers defined in RFC 1700; for example, Telnet = 23, and FTP = 20 or 21.

For more information about creating or modifying source port numbers, see the "Defining Source and Destination Port Numbers" section.

nfknown.dstports

The nfknown.dstports file contains destination port numbers used in any aggregation scheme that includes destination port numbers as part of its key.

For more information about creating or modifying destination port numbers, see the "Defining Source and Destination Port Numbers" section.

nfknown.srcasns

The nfknown.srcasns file contains source autonomous system numbers, either origin or peer, used in any aggregation scheme that includes source autonomous system numbers as part of its key.

For more information about creating or modifying source autonomous system numbers, see the "Defining Source and Destination Autonomous System Numbers" section.

nfknown.dstasns

The nfknown.dstasns file contains destination autonomous system numbers, either origin or peer, used in any aggregation scheme that includes destination autonomous system numbers as part of its key.

For more information about creating or modifying destination autonomous system numbers, see the "Defining Source and Destination Autonomous System Numbers" section.

nf.resources

The nf.resources file contains the variables and corresponding directory file path names used to configure your startup FlowCollector environment. Besides the path names, the nf.resources file also includes a number of configuration parameters for tuning FlowCollector performance and behavior.

For more information about this file, see the "Modifying FlowCollector Resources" section.

nfcd.config

The nfcd.config file contains the parameters needed by FlowCollector to automatically start NFCollector and NFCGW upon system startup, automatically restart NFCollector and NFCGW if they terminate abnormally, and set the scheduling priorities of NFCollector and NFCGW.

For more information, see the "Configuring the Daemon (NFCD)" section.

Understanding FlowCollector Data Collection and Aggregation

FlowCollector collects and summarizes (aggregates) data into data files based on user-defined criteria specified in a FlowCollector thread. A thread is an aggregation task defined by a set of user-configurable attributes that specify how FlowCollector aggregates the traffic flows stored on the workstation. Two key thread attributes are:

• Aggregation schemes---Define how to aggregate the traffic flows stored on the workstation

• Filters---Define the flow information that is accepted or rejected by the aggregation scheme

Figure 5-1 shows an example of how FlowCollector uses threads and filters. In this example, threadA uses filterA and the SourceNode aggregation scheme; threadB uses both filterA and filterB
(filters can be shared among threads) and the DestPort aggregation scheme; threadC does not use any filters, but it also uses the DestPort aggregation scheme.

The following nf.config file example contains the thread definitions to accomplish the general data aggregation scheme shown in Figure 5-1. The file contains two filter definitions (filterA and filterB) and three thread definitions (threadA, threadB, and threadC) to accomplish the aggregation. NetFlow export traffic arrives on FlowCollector UDP port 9991. Data is to be aggregated by the following aggregation schemes:

• SourceNode (source IP address)

• DestPort (destination port number)

FilterfilterA
permitnexthop172.16.23.650.0.0.0
 
FilterfilterB
denysrcaddr172.16.0.00.0.255.255
permitsrcaddr0.0.0.00.0.0.0
 
ThreadthreadA
FilterfilterA
AggregationSourceNode
Period10
Port9991
StateActive
DataSetPath/opt/CSCOnfc/Data
Compression6
BinaryN
MaxUsage500
 
ThreadthreadB
FilterfilterA
Filter filterB
Aggregation DestPort
Period 10
Port 9991
State Active
DataSetPath /opt/CSCOnfc/Data
Compression6
BinaryN
MaxUsage500
 
ThreadthreadC
Aggregation DestPort
Period 10
Port 9991
State Active
DataSetPath /opt/CSCOnfc/Data
Compression6
BinaryN
MaxUsage500
 

In the example, threadA uses filterA to include only the traffic passing through the export device 172.16.23.65, whereas threadB uses both filters---filterA to include only the traffic passing through the export device 172.16.23.65, and filterB to exclude traffic from network 172.16.0.0. All three threads flush their aggregated data every 10 minutes into data files saved in the /opt/CSCOnfc/Data directory.

Creating a Filter

A filter defines which flow data is to be included or excluded as FlowCollector aggregates data. The default condition for a filter is to deny (exclude) the flow.

filterfilter-name
{permit|deny}typevaluemask
.
.
.
{permit|deny}typevaluemask
 

where:

filter	The keyword that identifies the definition as a filter.
filter-name	The unique, user-specified name of the filter. The name can be up to 16 alphanumeric characters.
permit	The keyword that keeps the data that matches the specified filter type and value.
deny	The keyword that rejects the data that matches the specified filter type and value (matching flow data is ignored and not aggregated).
type	The filter type. See Table 5-1 for a description of filter types.
value	The value associated with the filter type. All filter types require a value. See Table 5-1 for a description of filter types and values.
mask	Required if the filter uses the srcaddr, dstaddr, or nexthop type. The IP netmask that qualifies the IP address used as value. See Table 5-1 for a description.

Filter keyword and variable entries are not case sensitive. Table 5-1 describes the default filter types provided with FlowCollector, the type of input required for the value, and whether the value requires a mask.

Table 5-1: Filter Types, Values, and Their Descriptions

Type	Value	Mask Required	Description
srcaddr	Source IP address	Yes	Filter the input data based on the source IP address. If you use this type, you must provide the IP netmask that qualifies the source IP address.
dstaddr	Destination IP address	Yes	Filter the input data based on the destination IP address. If you use this type, you must provide the IP netmask that qualifies the destination IP address.
srcport	Source port number	No	Filter the input data based on the source port number.
destport	Destination port number	No	Filter the input data based on the destination port number.
srcinterface	Source interface number	No	Filter the input data based on the source interface number.
dstinterface	Destination interface number	No	Filter the input data based on the destination interface number.
nexthop	Next hop IP address	Yes	Filter the input data based on the next hop IP address. If you use this type, you must provide the IP netmask that qualifies the next hop IP address.
protocol	Protocol name	No	Filter the input data based on the protocol definitions in the nfknown.protocols file. For more information on protocol definitions, see the "Defining Protocols" section.
prot	Protocol number	No	Filter the input data based on the protocol number in the flow record, where the protocol number corresponds to a protocol specified in the /etc/protocols file of your workstation.
ToS	Type of service	No	Filter the input data based on the type of service (ToS).
srcas	Source AS	No	Filter the input data based on the autonomous system number of the source, either origin or peer.
dstas	Destination AS	No	Filter the input data based on the autonomous system number of the destination, either origin or peer.

When defining a filter, keep in mind the following qualifications:

• You must use an explicit permit statement to permit flows that do not meet defined filtering conditions; otherwise, everything in the flow is denied. (The default condition for a filter is to deny the flow.) For example:

  filter kill-www
  denyDstport80
   
  

In this example, all flows going to port 80 and all other flows are denied. If you want to deny flows to port 80 only, but permit all other flows, you need an explicit wildcard entry to permit the other flows. For example:

filter kill-www
denyDstport80
permitDstaddr10.0.0.0255.255.255.255
 

• When multiple filter conditions exist, FlowCollector attempts to apply the conditions sequentially, in the order you specify, until a match is found, as shown in the following example.

  FilterfilterA
  permitSrcaddr172.16.1.240.0.0.255
  denySrcaddr192.168.0.00.0.255.255
  denySrcport53
  permitDstaddr10.0.0.0255.255.255.255
   
  

If you want to permit traffic from network 172.16.1.0, but deny traffic coming from port 53, you should change the order of the filter conditions as follows:

FilterfilterA
denySrcaddr192.168.0.00.0.255.255
denySrcport53
permitSrcaddr172.16.1.240.0.0.255
permitSrcaddr10.0.0.0255.255.255.255
 

The last filter condition overrides the default behavior, which calls for denying all flows that do not match any of the first three filter conditions.

Creating a Thread

A thread is a set of defined attributes that tells FlowCollector how to aggregate the traffic flows stored on the workstation.

The syntax for a thread definition is as follows:

Thread thread-name
[Filter filter-name]
.
.
.
[Filter filter-name]
Aggregation scheme 
Period minutes
Port value
DataSetPath directory-path
State active|inactive
Compression value (1-9)
Binary yes|no
MaxUsage megabytes

The keywords and their arguments are listed on separate lines for legibility. Keyword and argument entries are not case sensitive. Table 5-2 lists thread attributes and variables.

In the following example, thread Alpha uses the SourceNode aggregation scheme. FlowCollector creates a compressed binary data file in the directory /opt/CSCOnfc/Data every 30 minutes, keeps the last 24 data files per day, and specifies 500 MB in maximum disk space usage:

ThreadAlpha
AggregationSourceNode 
Period30
Port9991
StateActive
DataSetPath/opt/CSCOnfc/Data
Compression 6
Binary yes
MaxUsage 500

Aggregation Schemes

FlowCollector provides a library of predefined aggregation schemes (see Table 5-3) that you can use to determine the type of information that is aggregated and stored in the data files.

Each aggregation scheme consists of one or more key fields, which tell FlowCollector what to look for in the exported NetFlow datagram, and one or more value fields, which contain statistical information pulled from the exported NetFlow datagram. The key fields and value fields shown in Table 5-3 correspond to the fields found in Version 1, Version 5, Version 7, and Version 8 NetFlow export datagrams. Table 5-4 provides brief definitions of each key and value field. For more information about these three versions of the NetFlow export datagram format, see "NetFlow Export Datagram Format."

For example, the SourceNode aggregation scheme uses just one key field, srcaddr (source address), and returns data for three value fields: the total number of packets sent, the total number of bytes sent, and the total number of flows aggregated into this record. Other aggregation schemes offer different combinations of key and value fields (see Table 5-3) and are described individually in the sections that follow.

Table 5-3: FlowCollector Aggregation Schemes, Key Fields, and Value Fields

Aggregation Scheme	Key Fields																Value Fields
	srcaddr	dstaddr	srcport	dstport	protocol	protocol byte (prot)	ToS	input interface	output interface	nexthop	src_as	dst_as	masked srcaddr	masked dstaddr	src_mask	dst_mask	packet count	byte count	flow count	firstTimeStamp	lastTimeStamp	totalActiveTime
RawFlows
SourceNode	·																·	·	·
DestNode		·															·	·	·
HostMatrix	·	·															·	·	·
SourcePort			·														·	·	·
DestPort				·													·	·	·
Protocol					·												·	·	·
DetailDestNode		·	·	·	·												·	·	·
DetailHostMatrix	·	·	·	·	·												·	·	·	·	·
DetailInterface	·	·						·	·	·							·	·	·
CallRecord	·	·	·	·		·	·										·	·	·	·	·	·
ASMatrix1											·	·					·	·	·
NetMatrix1								·	·				·	·	·	·	·	·	·
DetailSourceNode	·		·	·	·												·	·	·
DetailASMatrix1	·	·	·	·	·			·	·		·	·					·	·	·
RouterAS2								·	·		·	·					·	·	·	·	·	·
RouterProtoPort2			·	·	·												·	·	·	·	·	·
RouterSrcPrefix2								·			·		·		·		·	·	·	·	·	·
RouterDstPrefix2									·			·		·		·	·	·	·	·	·	·
RouterPrefix2								·	·		·	·	·	·	·	·	·	·	·	·	·	·
AsHostMatrix1	·	·									·	·					·	·	·	·	·	·
HostMatrixInterface	·	·			·			·	·								·	·	·
DetailCallRecord	·	·	·	·	·		·	·	·								·	·	·	·	·	·

1Compatible only with Version 5 and Version 7 export data. 2Compatible only with Version 8 export data. On-router aggregation is required. Only RawFlows or the same aggregation scheme on the FlowCollector station and the router being monitored is supported.

Table 5-4: Key and Value Field Definitions

Field	Description
srcaddr	Source IP address
dstaddr	Destination IP address
srcport	TCP/UDP source port number or equivalent
dstport	TCP/UDP destination port number or equivalent
protocol	Name or label assigned to a protocol definition in the nfknown.protocols file
protocol byte (prot)	IP protocol type (for example, TCP = 6; UDP = 17)
ToS	IP type of service
input interface	SNMP index of input interface
output interface	SNMP index of output interface
nexthop	IP address of next hop export device
src_as	Autonomous system number of the source, either origin or peer
dst_as	Autonomous system number of the destination, either origin or peer
masked srcaddr	Source IP address masked with the source netmask (src_mask)
masked dstaddr	Destination IP address masked with the destination netmask (dst_mask)
src_mask	Source IP address prefix mask bits
dst_mask	Destination IP address prefix mask bits
packet count	Packets counted as part of this record
byte count	Total number of Layer 3 bytes counted as part of this record
flow count	Total number of flows aggregated into this record
firstTimeStamp	Time, in UTC seconds, of the first packet summarized into this record
lastTimeStamp	Time, in UTC seconds, of the last packet summarized into this record
totalActiveTime	Sum of individual activetime for all the flows summarized into the current record

RawFlows

The output of this aggregation scheme is an exact image of the NetFlow export datagram without aggregation and is stored in binary data files of n minutes' worth of data as specified by the Period attribute in the thread definition.

SourceNode

The output of this aggregation scheme consists of one record for each unique source IP address present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key field:	srcaddr
Value fields:	packet count, byte count, flow count

DestNode

The output of this aggregation scheme consists of one record for each unique destination IP address present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key field:	dstaddr
Value fields:	packet count, byte count, flow count

HostMatrix

The output of this aggregation scheme consists of one record for each unique source and destination IP address pair present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	srcaddr, dstaddr
Value fields:	packet count, byte count, flow count

SourcePort

The output of this aggregation scheme consists of one record for each unique source port present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key field:	srcport
Value fields:	packet count, byte count, flow count

DestPort

The output of this aggregation scheme consists of one record for each unique destination port present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key field:	dstport
Value fields:	packet count, byte count, flow count

Protocol

The output of this aggregation scheme consists of one record for each unique protocol (as defined in the nfknown.protocols file) present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key field:	protocol
Value fields:	packet count, byte count, flow count

DetailDestNode

The output of this aggregation scheme consists of one record for each unique combination of destination IP address, source port, destination port, and protocol present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	dstaddr, srcport, dstport, protocol
Value fields:	packet count, byte count, flow count

DetailHostMatrix

The output of this aggregation scheme consists of one record for each unique combination of source IP address, destination IP address, source port, destination port, and protocol present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	srcaddr, dstaddr, srcport, dstport, protocol
Value fields:	packet count, byte count, flow count, firstTimeStamp, lastTimeStamp

DetailInterface

The output of this aggregation scheme consists of one record for each unique combination of source IP address, destination IP address, input, output, and nexthop present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	srcaddr, dstaddr, input, output, nexthop
Value fields:	packet count, byte count, flow count

CallRecord

The output of this aggregation scheme consists of one record for each unique combination of source IP address, destination IP address, source port, destination port, protocol byte, and type of service present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	srcaddr, dstaddr, srcport, dstport, protocol byte, ToS
Value fields:	packet count, byte count, flow count, firstTimeStamp, lastTimeStamp, totalActiveTime

ASMatrix

The output of this aggregation scheme consists of one record for each unique source and destination autonomous system number pair present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	src_as, dst_as
Value fields:	packet count, byte count, flow count

NetMatrix

The output of this aggregation scheme consists of one record for each unique combination of input interface, output interface, masked source IP address, masked destination IP address, source mask, and destination mask present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	input interface, output interface, masked srcaddr, masked dstaddr, src_mask, dst_mask
Value fields:	packet count, byte count, flow count

DetailSourceNode

The output of this aggregation scheme consists of one record for each unique combination of source IP address, source port, destination port, and protocol present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	srcaddr, srcport, destport, protocol
Value fields:	packet count, byte count, flow count

DetailASMatrix

The output of this aggregation scheme consists of one record for each unique combination of source IP address, destination IP address, source port, destination port, protocol, input interface, output interface, source autonomous system number, and destination autonomous system number present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	srcaddr, dstaddr, srcport, dstport, protocol, input interface, output interface, src_as, dst_as
Value fields:	packet count, byte count, flow count

RouterAS

The output of this aggregation scheme consists of one record for each unique combination of input interface, output interface, source autonomous system number, and destination autonomous system number present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	input interface, output interface, src_as, dst_as
Value fields:	packet count, byte count, flow count, firstTimeStamp, lastTimeStamp, totalActiveTime

RouterProtoPort

The output of this aggregation scheme consists of one record for each unique combination of source port, destination port, and protocol present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	srcport, destport, protocol
Value fields:	packet count, byte count, flow count, firstTimeStamp, lastTimeStamp, totalActiveTime

RouterSrcPrefix

The output of this aggregation scheme consists of one record for each unique combination of input interface, source autonomous system number, masked source IP address, and source mask present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	input interface, src_as, masked srcaddr, src_mask
Value fields:	packet count, byte count, flow count, firstTimeStamp, lastTimeStamp, totalActiveTime

RouterDstPrefix

The output of this aggregation scheme consists of one record for each unique combination of output interface, destination autonomous system number, masked destination IP address, and destination mask present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	output interface, dst_as, masked dstaddr, dst_mask
Value fields:	packet count, byte count, flow count, firstTimeStamp, lastTimeStamp, totalActiveTime

RouterPrefix

The output of this aggregation scheme consists of one record for each unique combination of input interface, output interface, source autonomous system number, destination autonomous system number, masked source IP address, masked destination IP address, source mask, and destination mask present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	input interface, output interface, src_as, dst_as, masked srcaddr, masked dstaddr, src_mask, dst_mask
Value fields:	packet count, byte count, flow count, firstTimeStamp, lastTimeStamp, totalActiveTime

ASHostMatrix

The output of this aggregation scheme consists of one record for each unique combination of source IP address, destination IP address, source autonomous system number, and destination autonomous system number present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	srcaddr, dstaddr, src_as, dst_as
Value fields:	packet count, byte count, flow count, firstTimeStamp, lastTimeStamp, totalActiveTime

HostMatrixInterface

The output of this aggregation scheme consists of one record for each unique combination of source IP address, destination IP address, protocol, input interface, and output interface present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	srcaddr, dstaddr, protocol, input interface, output interface
Value fields:	packet count, byte count, flow count

DetailCallRecord

The output of this aggregation scheme consists of one record for each unique combination of source IP address, destination IP address, source port, destination port, protocol, type of service, input interface, and output interface present in the flow data received by FlowCollector during the current collection period. Each output record contains the following fields:

Key fields:	srcaddr, dstaddr, srcport, dstport, protocol, ToS, input interface, output interface
Value fields:	packet count, byte count, flow count, firstTimeStamp, lastTimeStamp, totalActiveTime

Defining Protocols

Use the information in this section to define the protocols that you want FlowCollector to recognize as it aggregates data. The protocols FlowCollector recognizes are defined in the nfknown.protocols file, located in the $NFC_DIR/config directory.

• The protocol definition must be added to the nfknown.protocols file.

• There must be an active thread calling for this protocol.

• There must be NetFlow data traffic that applies to the protocol.

If you remove the protocol from the nfknown.protocols file, information for that protocol is no longer recognized and is aggregated under "Others."

Figure 5-2 shows an example of a typical communication session between Host A and Host B. This example assumes that NetFlow data export is enabled for the export device interfaces to both Host A and Host B so that exported NetFlow data gives FlowCollector statistics for communication in both directions (A to B; B to A). In this example, FlowCollector aggregates data for two protocols--- Telnet and FTP---between Host A (the Telnet server, using port 23) and Host B (the Telnet client, using port 9001).

• For Telnet, FlowCollector aggregates data between Host A (the Telnet server, using port 23) and Host B (the Telnet client, using port 9001).

• For FTP, FlowCollector aggregates data between Host A (the FTP server, using port 20) and Host B (the FTP client, using port 9002).

Table 5-5 lists the raw data received in the data collection example shown in Figure 5-2.

Table 5-5: Data Received Example

Source Address	Destination Address	Source Port	Destination Port	Protocol Byte	Packets	Bytes
A	B	23	9001	6	20	2000
B	A	9001	23	6	30	1000
A	B	20	9002	6	20	200
B	A	9002	20	6	50	300

When you add the Telnet protocol definition to the nfknown.protocols file, the data file produced by the protocol aggregation scheme contains a row with 50 packets (20 plus 30) and 3000 bytes (2000 plus 1000).

In this example, no FTP protocol definition was added to the nfknown.protocols file, so the data file also has another row for "Others" (including the FTP data) containing 70 packets (20 plus 50) and 500 bytes (200 plus 300).

The protocols listed in the nfknown.protocols file are used by the aggregation schemes and protocol filters you define in the nfconfig.file. To configure the protocols that FlowCollector recognizes, you must edit the nfknown.protocols file and add a definition that includes the following information:

The command syntax for a protocol definition is

protocolname
[[srcport|dstport]number[OR[srcport|dstport]number]]
protvalue
 

where:

protocol	Keyword that identifies the definition as a protocol.
name	Unique, user-specified name of the protocol definition. Can be up to 16 alphanumeric characters.
srcport	Source port.
dstport	Destination port.
number	Port number.
OR	(Optional.) Provides a Boolean OR functionality when you have more than one srcport or dstport.
prot	Protocol byte.
value	Protocol type (similar to those specified in the /etc/protocols file on a UNIX workstation; for example, TCP = 6, UDP = 17).

The known protocols (such as WWW, Telnet, and FTP) listed in the nfknown.protocols files are similar to the definitions specified in the /etc/services file of a UNIX workstation. For information about the protocols and protocol types supported on your workstation, refer to the protocols file in the /etc directory on your workstation.

ProtocolTCP-Telnet
Dstport23 OR Srcport23
Prot6
 
ProtocolTCP-FTP
Srcport 20 OR Srcport 21 OR Dstport 20 OR Dstport 21
Prot 6
 
ProtocolTCP-WWW
Dstport 80 OR Srcport 80
Prot 6
 
ProtocolTCP-SMTP
Srcport 25 OR Dstport 25
Prot 6
 
ProtocolTCP-Other
Prot 6
 
ProtocolUDP-TFTP
Srcport 69 OR Dstport 69
Prot 17
 

Defining Source and Destination Port Numbers

Use the information in this section to specify the source and destination port numbers from which FlowCollector collects and aggregates data. The port numbers FlowCollector recognizes are defined in the following files, located in the $NFC_DIR/config directory:

• nfknown.srcports---This file contains the source port numbers that you want FlowCollector to recognize.

• nfknown.dstports---This file contains the destination port numbers that you want FlowCollector to recognize.

The command syntax for a port number, range of port numbers, or range grouped under an assigned label is:

value[,value[:label]]
.
.
.
value[,value[:label]]
 

where:

value	A number between 0 and 65535.
label	(Optional.) An alphanumeric ASCII string of up to 16 characters.

50, 100
 

10000, 19999: 10K_19K_Pt_Rng
 

The following example shows the contents of a sample nfknown.srcports file:

21:ftp
88
50, 100
10000, 19999: 10K_19K_Pt_Rng
20000, 29999: My_Range
40000, 49999: My_Range
 

Defining Source and Destination Autonomous System Numbers

Use the information in this section to specify the source and destination autonomous systems from which FlowCollector collects and aggregates data. The autonomous systems FlowCollector recognizes are defined in the following files, located in the $NFC_DIR/config directory:

• nfknown.srcasns---This file contains the source autonomous system numbers, either origin or peer, that you want FlowCollector to recognize.

• nfknown.dstasns---This file contains the destination autonomous system numbers, either origin or peer, that you want FlowCollector to recognize.

FlowCollector uses the contents of the nfknown.srcasns and nfknown.dstasns files in aggregation schemes that make use of source or destination autonomous system numbers. When you add an autonomous system definition to either of these files, traffic to or from the autonomous system is counted separately in the data file. Any unrecognized autonomous system numbers (autonomous system numbers not defined in their respective files) are aggregated together and appear as "Others" in the data file.

1: Your_Network
2
10, 15
20, 30: My_Network
35, 40: My_Network
 

Modifying FlowCollector Resources

The nf.resources file contains the configuration parameter settings and directory file path names used to configure your startup FlowCollector environment. Besides the path name definitions, the nf.resources file also includes a set of parameters for tuning FlowCollector performance. Only users with root or .bin directory privileges can modify this file. The nf.resources file is located in $NFC_DIR/config.

Table 5-6 describes the available configuration parameters and their values.

Table 5-6: nf.resources File Configuration Parameters

Flag	Possible Values	Description	Default Value
OUTPUT_DOTTEDADDRESS	Yes No	Writes the IP address to the data files in dotted decimal format, for example, 172.16.3.100. Writes the IP address to the data files in network address format, for example, 8557414940.	Yes
CSV_FORMAT	Yes No	Uses a comma (,) as the delimiter in writing aggregation output. Uses a vertical bar ( | ) as the delimiter.	No
LONG_OUTPUTFILE_SUFFIX	Yes No	Sets the output file extension to add the year, month, and day to the hour and minute, for example, _YYYY_MM_DD.HHMM. Sets the output file extension to include HHMM only.	No
GMT_FLAG	Yes No	Uses the Greenwich Mean Time reference to set date and time. Uses local time as set by system clock. This attribute affects the date and time used in naming the data file directory structure, names of data files, headers in data files, and messages in the log files.	Yes
DEVICE_DOTTEDADDRESS	Yes No	Uses the IP address of the sending export device for storage. Attempts to get the DNS name before using the IP address of the sending export device for storage. If a ROUTER_GROUPNAME label has been defined using the ROUTER_GROUPNAME configuration parameter, that label is used; otherwise, the IP address or the DNS name is used, depending on the setting of the DEVICE_DOTTEDADDRESS configuration parameter. For more information on the ROUTER_GROUPNAME configuration parameter, see the "Mapping a List of IP Addresses to One IP Address or Label" section.	Yes
SOURCENODE_BUCSIZE DESTNODE_BUCSIZE HOSTMATRIX_BUCSIZE NETMATRIX_BUCSIZE DETAILSOURCENODE_BUCSIZE DETAILDESTNODE_BUCSIZE DETAILHOSTMATRIX_BUCSIZE DETAILINTERFACE_BUCSIZE CALLRECORD_BUCSIZE ASMATRIX_BUCSIZE DETAILASMATRIX_BUCSIZE ROUTERAS_BUCSIZE ROUTERPROTOPORT_BUCSIZE ROUTERSRCPREFIX_BUCSIZE ROUTERDSTPREFIX_BUCSIZE ROUTERPREFIX_BUCSIZE ASHOSTMATRIX_BUCSIZE HOSTMATRIXINTERFACE_BUCSIZE DETAILCALLRECORD_BUCSIZE	Varies according to configuration	Controls performance. The term BUCSIZE refers to the number of buffer pages set aside to hold aggregated data for a given aggregation scheme. The general rule of thumb for BUCSIZE values is: If an aggregation scheme produces n records in a collection interval, the corresponding BUCSIZE value should be (approximately) between n/20 and n. The best approach is to sample NetFlow traffic, and then determine whether changes are required.	2000 2000 2000 2000 2000 2000 6000 6000 50000 2000 50000 50000 50000 50000 50000 50000 50000 50000 50000
SOCKET_BUFSIZE	Buffer size (in bytes)	Specifies the size of the UDP socket receive buffer.	900000
ROUTER_GROUPNAME	List of IP addresses or labels	Allows a user-specified IP address or label to be substituted for a list of IP addresses from which FlowCollector can receive NetFlow export datagrams. For more information, see the "Mapping a List of IP Addresses to One IP Address or Label" section.	Disabled
ACCEPT_PACKETS_FROM	List of IP addresses or labels	Allows packets to be filtered by source address (or by defined ROUTER_GROUPNAME label). For more information, see the "Preventing FlowCollector from Accepting Unsolicited Packets" section.	Disabled
USE_SHORT_CUT_ADDRESS_AS_ SOURCE_IP	Yes No	Uses the address of the router being bypassed (shortcut) as the source of the corresponding flow. Uses the address of the export device being bypassed (shortcut) as the source of the corresponding flow. For more information, see the "Retaining Router IP Addresses for Switched Export Packets" section.	No
USER_SCRIPT_LOCATION	Path name and filename	Specifies the location of a user-supplied script. For more information, see the "Using a User-Defined Script to Process FlowCollector Data Files" section.	Disabled
OUTPUT_BUFFER_SIZE	1, 2, 4, 8, 16	Specifies the size (in megabytes) of the memory buffer FlowCollector uses for I/O operations. For more information, see the "Changing the Output Buffer Size" section.	4
NFC_USERNAME	User name	Specifies the registered FlowCollector user. Only root and this specified user can access or modify FlowCollector configuration items and statistics from a remote location.	nfcuser
CLEANUP_INTERVAL	Hours	Specifies the number of hours between each data file CLEANUP_JOB. Setting this parameter to zero disables this feature.	24 hours
CLEANUP_JOB	Path name and filename	Specifies the location of the script or program that is run at the end of the time period specified in the CLEANUP_INTERVAL parameter. This script could be the default shell script, a user-defined script, or a binary program file. The default shell script path and filename is $NFC_DIR/bin/nfc_clean_up_job.sh.	Shell script supplied by FlowCollector that deletes any files older than 7 days
DISK_USAGE_THRESHOLD	Percentage	Specifies a disk usage threshold percentage level. Every time a data file is written to disk, information on disk space usage is obtained. If disk space usage exceeds the specified percentage value, one unsolicited event notification (UEN) is sent. Future UENs are sent only after disk space levels have fallen below the percentage amount specified in the DISK_USAGE_CLEAR_THRESHOLD value. Setting this parameter to zero disables this feature.	85%
DISK_USAGE_CLEAR_THRESHOLD	Percentage	An "all-clear" UEN is sent when disk usage falls below this threshold percentage value once a DISK_USAGE_THRESHOLD UEN has been sent.	75%
NFC20_COMPATIBLE _MODE	Yes No	Instructs FlowCollector 3.0 to generate FlowCollector 2.0-compatible data files, filesready file, and directory structures. When this parameter is set to "Yes," the following FlowCollector 3.0 features are disabled: Version 3.0 directory structure, including thread IDs Version 3.0 filesready file Aggregation definitions in data files MaxUsage parameter available in an NF_Thread FlowCollector 3.0-compatible files and directories are generated.	Yes
UEN_RECIPIENT_LIST	IP Address, port or DNS name, port	Specifies a list of the IP addresses and ports (or DNS system names and ports) of the client programs running to receive unsolicited event notifications (UENs). For example, an entry "flow-ultra1, 12345" implies that a program executing on flow-ultra1 and listening to port 12345 would receive UENs dispatched by the NFCGW.	None

Increasing UDP Socket Receive Buffer Size

Because of the high volume of NetFlow data export traffic, you might have to increase the normal buffer size associated with the UDP socket on which data is received. To do so, edit the value
(in bytes) of the SOCKET_BUFSIZE parameter in the $NFC_DIR/config/nf.resources file.

Mapping a List of IP Addresses to One IP Address or Label

You can substitute a user-specified IP address or label for a set of IP addresses from which FlowCollector receives NetFlow export datagrams. For example, you can specify the label "blab-gateway" as the label representing packets coming from three separate IP addresses: 172.16.1.172, 172.16.1.173, and 192.68.1.25.

To do this, you must edit the ROUTER_GROUPNAME parameter in the nf.resources file. The syntax is

ROUTER_GROUPNAMElabel {
a.b.c.d
.
.
.
w.x.y.z
}
 

ROUTER_GROUPNAMEblab-gateway {
172.16.1.172
172.16.1.173
192.68.1.25
}
 

If applicable, the mapped ROUTER_GROUPNAME parameter is used with all aggregation schemes, but FlowCollector uses the real IP address to report errors involving receipt of an invalid or unsolicited NetFlow export packet.

Preventing FlowCollector from Accepting Unsolicited Packets

In its default configuration, FlowCollector accepts NetFlow export packets from any IP address. If necessary, you can specify the source IP addresses or defined ROUTER_GROUPNAME labels from which FlowCollector should receive NetFlow export packets, thus preventing FlowCollector from accepting packets from any unspecified sources.

To do this, you must remove the comment character from the beginning of each line in the ACCEPT_PACKETS_FROM parameter in the nf.resources file and edit the parameter to include the source IP addresses or ROUTER_GROUPNAME labels. The syntax of the parameter is

ACCEPT_PACKETS_FROM {
a.b.c.d
.
.
.
w.x.y.z
}
 

ACCEPT_PACKETS_FROM {
172.31.2.1
172.31.2.2
172.31.2.3
blab_gateway
}
 

For information on ROUTER_GROUPNAME labels, see the previous section, "Mapping a List of IP Addresses to One IP Address or Label."

Retaining Router IP Addresses for Switched Export Packets

If your network includes switching devices that support Version 7 (Catalyst 5000 series switch with an NFFC) NetFlow export datagrams, you can configure FlowCollector to retain the IP address of the shortcut router as the source of data switched through a Cisco Catalyst 5000 series switch. To do this, you must edit the USE_SHORT_CUT_ADDRESS_AS_SOURCE_IP parameter in the nf.resources file. The syntax of the parameter is

USE_SHORT_CUT_ADDRESS_AS_SOURCE_IP value
 

Using a User-Defined Script to Process FlowCollector Data Files

You can specify the location of a script file that FlowCollector executes after it has written a new data file. This capability makes it easier for your client applications to process a new data file without having to poll for it. FlowCollector invokes the script with the absolute path name of the newly written FlowCollector data file. FlowCollector expects the location of your user-supplied script to be defined by the USER_SCRIPT_LOCATION parameter in the nf.resources file. This parameter is read only at startup.

Changing the Output Buffer Size

FlowCollector transfers output data in blocks to optimize performance and ensure the most efficient handling of data files as they are generated and written as disk files. The size of a block is user-configurable, and defined by the OUTPUT_BUFFER_SIZE parameter in the nf.resources file. The syntax of the parameter is

OUTPUT_BUFFER_SIZE size
 

Configuring the Daemon (NFCD)

The Daemon (NFCD) provides high availability to the FlowCollector system. The Daemon is loaded as a configuration file is called nfcd.config when the system is booted. It is located in the $NFC_DIR/config directory. The Daemon performs the following functions:

See Table 5-7 for nfcd.config parameter descriptions.

The following is an example of an nfcd.config file:

--------------------Begin nfcd.config --------------------
NFCD_LOGFILE   /opt/CSCOnfc/logs/nfcd.log
MAX_RESTART_ATTEMPTS   3
 
 
APPLICATION {
	 	 	 	 	PATH    /opt/CSCOnfc/bin/NFCollector
    PROGRAMFLAGS    
    RESTART Yes
    AUTOSTART   Yes
    TMPFILES /tmp/nfcollector.pid
    SCHEDULING_PRIORITY {
        SCHEDULING_CLASS RealTime
        DEFAULT_PRIORITY -20
	 		 	 		 }
}
APPLICATION {
	 		 	 	 PATH    /opt/CSCOnfc/bin/NFCGW
    PROGRAMFLAGS    -i
    RESTART Yes
    AUTOSTART   Yes
    RETRY_INTERVAL 120
    TMPFILES /tmp/nfcgw.pid /tmp/nfc.events
    SCHEDULING_PRIORITY {
        SCHEDULING_CLASS RealTime
        DEFAULT_PRIORITY -20
	 		 	 		 }
}
--------------- End of nfcd.config ------------------------

Table 5-7: nfcd.config File Parameter Descriptions

Parameter	Variable Type	Description	Default
NFCD_LOGFILE	NFCD Config	Name and location of the nfcd.log file.	$NFC_DIR/logs
MAX_RESTART_ATTEMPTS	NFCD Config	Number of times that NFCD attempts to automatically restart NFCollector and NFCGW if they terminate abnormally.	3
PATH	APPLICATION	Path of NFCollector and NFCGW executable files.	$NFC_DIR/bin
PROGRAMFLAGS	APPLICATION	Flags used in invoking NFCollector and NFCGW. This can be used to pass "-i" to NFCGW, turning off authentication on HP-UX platforms.	Empty string
RESTART (Yes/No)	APPLICATION	Specifies whether NFCollector and NFCGW should be restarted by NFCD upon an abnormal exit.	No
RETRY_INTERVAL	APPLICATION	If RESTART is set to Yes, this parameter specifies the number of seconds to wait before attempting to restart NFCollector and NFCGW.	1 second
AUTOSTART (Yes/No)	APPLICATION	Specifies whether NFCollector and NFCGW should be started when NFCD is started.	No
TMPFILES	APPLICATION	List of temporary files used by NFCollector and NFCGW. Not user- configurable.	NA
SCHEDULING_CLASS	APPLICATION/ SCHEDULING_PRIORITY	The scheduling class of the process executing NFCollector and NFCGW. Values include RealTime or TimeShared. This parameter is supported only on Solaris systems. For more details refer to the "priocntl" or "nice" Solaris man pages.	TimeShared
DEFAULT_PRIORITY	APPLICATION/ SCHEDULING_PRIORITY	The default priority of the process. Acceptable values are in the range of [-20, 20]. This value applies to time-shared scheduling and is used when one of these conditions occurs. 1 . No SCHEDULING_CLASS is specified. 2 . SCHEDULING_CLASS has the value TimeShared. 3 . SCHEDULING_CLASS is set to RealTime and NFCD cannot set the scheduling class of the process to real time.	0

Managing Disk Space

Depending on the volume of flow data being exported from the export devices, as well as the FlowCollector thread attribute settings you use, FlowCollector can consume large amounts of disk space in a short period. FlowCollector provides several thread attributes and features that can help you manage your disk space usage:

• Filters (cannot be used with the RawFlows aggregation scheme)

• Aggregation schemes

• NF_Thread data file and disk space options

• Parameters in the nf.resources file

Filters

As described earlier, a filter can help you discard any flow data that is not of interest to you. By using filters to ensure that you are storing only data of interest, you can potentially reduce the amount of disk space used by FlowCollector.

Aggregation Schemes

Aggregation schemes are used to define how you want FlowCollector to summarize the flow data being exported from your export devices. By using only those aggregation schemes required for your application and, when possible, by selecting the aggregation schemes that generate the least amount of data on disk, you can reduce the amount of disk space used by FlowCollector. For example, using the HostMatrix aggregation scheme results in less disk space usage than using the DetailHostMatrix scheme. Of course, the aggregation schemes you use are determined primarily by the data you are interested in and how you want to summarize that data. It is important to realize, however, that the different aggregation schemes can greatly affect the amount of disk space used by FlowCollector.

NF_Thread Data File and Disk Space Parameters

Optional parameters are available on a per NF_Thread basis to limit disk space and improve system performance at the same time. These parameters are documented in the "Creating a Thread" section. These parameters include:

• Applying gzip compression to data files

• Creating binary data files instead of ASCII format data files

• Using a MaxUsage parameter to specify the amount of disk space used by data files before they are flushed

Parameters in the nf.resources File

The nf.resources file contains parameters that assist in managing disk space. See the "Modifying FlowCollector Resources" section for more details. These parameters include:

• The CLEANUP_INTERVAL parameter specifies the amount of time between each data file CLEANUP_JOB.

• The CLEANUP_JOB parameter specifies the program or script to run at the end of the CLEANUP_INTERVAL parameter to flush data files.

• The DISK_USAGE_THRESHOLD parameter provides a warning message when data files consume the maximum amount of disk space specified in the parameter.

Unsolicited Event Notifications

When FlowCollector encounters an error or needs to issue a warning on some aspect of FlowCollector operations, it generates messages to inform the user that something is wrong and needs to be addressed. These messages are called unsolicited event notifications (UENs). These messages do not appear within the FlowCollector application. Instead, they consist of UDP packets that are sent by NFCollector and received at the IP address and port number specified in the nf.resources file.

UENs are not enabled by default. You must edit the nf.resources file and indicate the
IP address and port (or the DNS system name and port) where a client application is running and listening for these events. This parameter supports multiple entries formatted as a list.

Posted: Fri Jul 9 11:08:55 PDT 1999
Copyright 1989-1999©Cisco Systems Inc.