For When You Can't Have The Real Thing
[ start | index | login ]
start > dave > experiments > Net Flows > 2009-03-04 > 1

2009-03-04 #1

Created by dave. Last edited by dave, 10 years and 47 days ago. Viewed 1,732 times. #1
[edit] [rdf]

90% of the solution

So it turns out that 90% of any problem you have is likely to have been solved already for you, somewhere out on the internet. The problem then is finding that solution; fortunately the 90% of that problem is already solved, and that's Google. The problem then is figuring out where the answer is amongst all the noise that is out there.

In my case, may I present:>>NfSen and >>NFDUMP.

Once I had the fprobe collector business sorted, I installed nfdump and nfsen. This generates interesting graphs based on individual flow collector sources, and can then query stored flows for more detailed information. You can set up multiple filters, and even do historical graph generation for specific targets (ie networks, IPs, protocols, and/or ports).

I estimate that my system can probably store a couple months of detailed flow information before having to prune old records; fortunately the system will keep RRD files which summarize the activity, so the graphs will go back in time much further. The system even automatically prunes old files once you get to a high-water mark, so space management becomes automatic.

The only thing it doesn't do is provide an interface to easily generate historical activity graphs for particular IP addresses, something which was one of the original points of this exercise. However, nfdump is easy to set up to generate machine-readable output:

# nfdump -q -o pipe -r $FILE
One perl script later and I can generate RRDs for each IP address I am interested in.

The next trick is purely a web trick; I need to generate the top-level index pages (which I've done before} and then figure out how to dynamically generate the graphs on demand (because I don't want to generate a million graphs every five minutes that nobody's going to look at).

The downer is that it renders much of the below exercise pointless, although I did end up learning about database programming a bit, which I suppose is good. And some of the shell wrapper scripts which generate graphs from RRD files will probably be useful in a vague way in that I know the mechanics of generating a RRD file; I just have to do it on demand now.

no comments | post comment
This is a collection of techical information, much of it learned the hard way. Consider it a lab book or a /info directory. I doubt much of it will be of use to anyone else.

Useful: | Copyright 2000-2002 Matthias L. Jugel and Stephan J. Schmidt