For When You Can't Have The Real Thing
[ start | index | login ]
start > Linux > Installing nfsen

Installing nfsen

Created by dave. Last edited by dave, 11 years and 223 days ago. Viewed 26,579 times. #14
[diff] [history] [edit] [rdf]
labels
attachments

Backplot

(Added 20 August 2012)

I'm using this configuration to monitor the uplinks (both to the internet and for TLS links to our partners) for a small hosting company. According to nfsen, we pass in the order of 8 TB of traffic internet monthly, and ~35TB of traffic with our partners monthly.

The current nfsen system is a Sun V60x system with 2GB of memory and mirrored 72GB Ultra-320 disks running CentOS 5.x. It only performs netflow (and netflow accounting) tasks.

I keep 80GB of flows, and that is about a month of data. The vast majority of it (77GB currently) is the internet traffic rather than the partner traffic, but that's due to the nature of the traffic (both internet and partner).

Our primary goal with this tool was to provide some accounting as to which of our customer's IPs were transferring how much with the internet. This information is collected through some perl scripts (currently unreleased because they are very ugly) and presented through dynamically generated graphs. However we have found that keeping the flows around are good for forensic investigations regarding abnormal traffic; we've been able to confirm hacked systems through traffic patterns as well as show customer some undesired behavior by showing them the relevant flow data.

Also, you really want to read the whole thing -- including the parts about CentOS 5 -- before proceeding.

Installing a fprobe/nfsen configuration on CentOS 4

(13 May 2009)

Configure your network:

For this to work you need a smart switch that can mirror/clone traffic from one port to another, a computer to act as the flow generator, and a computer to act as the nfsen station.

In practice this means you will be mirroring the traffic you are interested in to a port to be connected to the monitoring station. I like to dedicate an interface on the monitoring station for this purpose; depending on your switch this might be mandatory as some switches will not let a system transmit through a switch set to "monitoring/mirroring" mode. (Note that 3Com used to call this mode "RMON Roving Port Analysis". Maybe they still do.)

Usually when doing pilots, the flow generating computer is the same as the nfsen computer.

Note that if your firewall/router is suitably advanced, you may be able to generate netflows internally from the device and therefore skip the fprobe steps completely.

This is all left as an exercise for the reader.

Download:

For the purposes of this example I am using:

Install Prerequisites

These are available from rpmforge:

$ sudo yum install rrdtool rrdtool-devel perl-MailTools

You also need stuff like apache, perl, php, and the development tools, but all that stuff is available from base and/or update.

Other Stuff

Turn off selinux.

Install fprobe

I use my own .src.rpm. In practice you can just do it the straight forward way:

$ ./configure
$ sudo make install

Start running fprobe to generate netflows

Invocation:

# fprobe -fip -i device dest_ip:dest_port

Notes:

  • dest_ip can be localhost if you are running this on the same computer.
  • for best results, don't use the interface you are monitoring to send/receive the netflows (because their addition will skew the numbers slightly).
  • You might be interested in -e and -q values depending on how busy your link is.
Operational example:
# fprobe -fip -i eth2 -e 120 -q 10240 10.16.5.45:999

Create a local user

nfsen wants to run as a local user, by default "netflow".

$ sudo useradd -G apache -d /var/lib/netflow netflow

I stick netflow stuff in /var for no good reason.

This local user must be a member of the group that the webserver runs as (on CentOS, it is apache).

Also, you have to make sure that the webserver's group has read-access to the nfsen tree; I usually do it like this:

$ sudo chown netflow:apache ~netflow
$ sudo chmod 750 ~netflow

Build and Install nfdump

$ ./configure --enable-nfprofile
$ make
$ sudo make install

Configure nfsen

I copy etc/nfsen-dist.conf to etc/nfsen.conf and then edit the copy.

Lines I change:

  • $BASEDIR (I change it to live in /var/lib/netflow/nfsen-1.3 so that if I ever have multiple versions I can keep them separate)
  • $WWWUSER/$WWWGROUP (change both to apache)
  • %sources (remove the two samples, add your flow(s)); example to match the fprobe example above:
    %sources = ( 'ns25-lan'    => { 'port'    => '999', 'col' => '#0000ff', 'type' => 'netflow' }, );
Install nfsen:

$ sudo install.pl etc/nfsen.conf

Start nfsen:

$ sudo nfsen-1.3/bin/nfsen start

Configure http to look at nfsen:

This is an example /etc/httpd/conf.d/nfsen.conf

Alias /nfsen /var/www/nfsen
<Directory /var/www/nfsen/>
    DirectoryIndex nfsen.php
    Options -Indexes
    AllowOverride all
    order allow,deny
    allow from all
    AuthType Basic
    AuthUserFile /etc/httpd/conf/htpasswd.nfsen
    AuthName "Access"
    require valid-user
    AddType application/x-httpd-php .php
    php_flag magic_quotes_gpc on
    php_flag track_vars on
</Directory>

Note if you use the Auth stuff above you need to create and populate your relevant password file too:

sudo htpasswd -c /etc/httpd/conf/htpasswd.nfsen admin
New password:
Re-type new password:

...and restart httpd so that the change takes effect:

$ sudo /usr/sbin/service httpd restart

Make it all reasonably available

You'll want to ensure that the following things start on boot:

  • fprobe
  • nfsen
  • httpd
Enjoy your netflows

Point your browser at the system where you installed nfsen and start playing.

If you get ugly messages about not being able to initialize globals (among other problems) then you almost certainly have selinux running (turn it off). Either that or you skipped the directory ownership/permissions step above.

A Brief Word About VLANs

If you are using VLANs on the wire you are sniffing, it depends on the switch type as to whether or not you need to get fancy. When I was hooked up to a 3Com 4900, a single fprobe instance attached to the interface read all VLANs correctly; when the 4900 was replaced by a Dell 3548, the single fprobe instance only read the outbound traffic and didn't read any of the inbound traffic. I had to create un-addressed interfaces for each VLAN and then run a separate fprobe instance for each interface (with an associated separate probe on the nfsen collector too).

eg:

vconfig add eth2 2024
ifconfig eth2.2024 up
fprobe -f"ip" -i eth2.2024 -e 120 -q 10240 172.30.0.159:997
vconfig add eth2 2045
ifconfig eth2.2045 up
fprobe -f"ip" -i eth2.2045 -e 120 -q 10240 172.30.0.159:996

Update: 2 June 2010

Notes about CentOS 5

Quasi-upgrade/new installation: my nfsen system has small, full disks. Instead of migrating the installation from a small disk to a large one, I decided to build a new system, copy the current data (plus assorted other things I've done with the historical data) to the new system, and go. Of course things are not so straight forward.

nfsen has been updated to 1.3.3. I cannot get 1.3.3 to install, it complains about 'docs' not being a valid directory. 1.3.2 works. Interestingly the 1.3.2 tar file is much larger than the 1.3.3 file was.

nfdump has been updated to 1.6.x. The format of the netflow files has changed and by default 1.6.1 won't read 1.5.x dump files. If you have some 1.5.x netflow files around you might want to read, you have to configure nfdump as so:

#  ./configure --enable-compat15 --enable-nfprofile

If you are getting rrdtool and friends from rpmforge, beware that rpmforge has upgraded their latest available rrdtool to be 1.4.x (for EL 5.x anyways); nfsen won't deal with anything higher than the 1.3.x stream. I went to the rpmforge repo site in my web browser (which for me was >>http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/) and downloaded the versions I had been running on my older system, which were:

  • perl-rrdtool-1.2.30-1.el5.rf.i386.rpm
  • rrdtool-1.2.30-1.el5.rf.i386.rpm
  • rrdtool-devel-1.2.30-1.el5.rf.i386.rpm
no comments | post comment
This is a collection of techical information, much of it learned the hard way. Consider it a lab book or a /info directory. I doubt much of it will be of use to anyone else.

Useful:


snipsnap.org | Copyright 2000-2002 Matthias L. Jugel and Stephan J. Schmidt