18 Intelligent NetFlow analyzer

The outcome of this project is the NetFlow Monitor program. It is a modular distributed system for analysis of network traffic based on NetFlow statistics. NetFlow Monitor is able to analyze the traffic almost in real time, including intelligent filtering, aggregation and statistical evaluation of the data. It also provides a number of options for selections based on multiple criteria (e.g., source/destination IP address, protocol, port, etc.) with the granularity down to individual flows.

The entire system consists of three blocks:

executive part - NetFlow Collector,
user interface - NetFlow Monitor,
warning subsystem - NetFlow Event.

18.1 NetFlow Collector

The first part is written in the C programming language and performs the processing of received data. The system currently supports the export formats of NetFlow versions 1, 5, 6 and 7. Other functionality of the NetFlow Collector is available in the form of simple modules.

One module is the NetFlow Forwarder, which is designed for redirection of the data flow to other destination(s) specified by IP addresses or ports. Another module then allows filtering of incoming data according to access lists (ACL) - we can thus define the sources from which NetFlow exports are accepted.

An important module takes care of storing the received and evaluated NetFlow exports into a database. This export module not only stores data from internal buffers into a MySQL database, but also aggregates certain information about data flows. All information about data flows is stored in such a manner that non-aggregated information is stored in hourly tables and from them daily, weekly, monthly and yearly tables with aggregated information are generated.

The number of tables is theoretically limited only by the available disk space, but an explicit limit can also be set in the configuration if necessary. For example, we can instruct the system to keep the hourly tables for the last 3 days, the daily tables with aggregated data for the last 14 days, the weekly tables for the last 6 weeks and the monthly tables for the last two years.

The combination of tables with aggregated and non-aggregated information gives the user a considerable flexibility since the same data can be viewed from many different perspectives. Typically, for a network administrator it is often important to have access to detailed information about network activities of the last hour at the level of particular data flows.

18.2 The NetFlow Monitor

The second part of the system for NetFlow data processing is written in the PHP programming language. Its purpose is to present measured values in a user-friendly form, produce various graphs and statistics and, last but not least, allow an easy configuration of the NetFlow Collector through a web interface.

In the first half of the year 2003 we rewrote the system interface starting from the original core that was created in the previous year. The advantage of the NetFlow Monitor is the simplicity of the web interface, which allows a comfortable manipulation with measured statistics. The new web interface contains six menu items: Main, Profiles, Archive, Statistics, Options and Help.

Figure 18.1: Statistics of transferred data volume example

The Main menu contains two basic search functions (Overview and Full Search), the next two menu items provide information about both communicating sides and their autonomous systems, respectively, and the last item shows a list of generated graphs (which can be exported or printed at user's discretion). The Profiles menu contains user-defined queries. The Archive menu enables the backup of individual tables or the system configuration. The Statistics menu provides information about the status of particular processes, size and state of individual tables, state of the database etc. The Help menu contains brief information about system installation, the distribution version, license conditions etc.

Since the NetFlow system has been designed for a distributed architecture, it is necessary to set up a NetFlow Unit item for each participating NetFlow Collector. The NetFlow Unit means just a single server - more NetFlow Collectors can run under one NetFlow Unit. When setting up a NetFlow Collector, the user has to specify a port on which the NetFlow exports from routers are awaited and the number of tables that are to be stored.

Figure 18.2: Statistics of the distribution of the used L4 ports (applications) example

The NetFlow Monitor supports 18 different statistics such as the summaries of transferred bytes, 10 most active hosts, distribution of protocols and L4 ports. All statistics can be defined for the whole network, a subnetwork, a selected host etc.

18.3 The NetFlow portal

During the year 2003 we finished the new portal, which provides news about NetFlow Monitor development. At present the portal offers the visitors a live test of the software (currently two OSR7600 routers with international connectivity and one GSR12000 backbone router are monitored). After completing the registration form, the entire distribution can be also downloaded directly from the portal. The URL of the portal is netflow.cesnet.cz. In order to establish a wider user base, two system versions are offered, one for the recommended operating system GNU/Debian Linux and another for other Linux distributions.

Figure 18.3: The new portal netflow.cesnet.cz

18.4 Conclusion

During the year 2003 we designed and implemented a usable production version of a network monitoring system - the NetFlow Monitor. It is now being used by more than 1000 organisations from more than 60 countries all over the world. The most frequent users are Internet service providers (ISP), universities and telecommunication companies. The system is distributed under the GNU General Public License (GPL).

contents