For When You Can't Have The Real Thing
[ start | index | login ]
start > pcap to netflow

pcap to netflow

Created by dave. Last edited by dave, one year and 346 days ago. Viewed 702 times. #3
[diff] [history] [edit] [rdf]
labels
attachments
(2022-04-06)

Problem

I have a pile of pcap files that I want to generate netflow files for.

Solution

nfdump comes with a suitable utility.

$ for i in `ls $CAPS | sort -V ` ; do 
  echo ">> $i" | tee -a nfpcapd.log
  nfpcapd -e 60,60 -r $CAPS/$i -l . 2>> nfpcapd.log
done

This took about 15 minutes to chew through 200G of pcap files on my tap machine.

Something slightly more robust. This script:

  • can handle "cap.dmp###" files that are in different directories and sorts/processes them by age
  • can be re-run on the same dataset and won't process cap.dmp files already processed
  • excludes the last file in the list of cap.dmp files on the theory that there is probably a tcpdump currently writing to it
#!/bin/bash

OIFS="$IFS" IFS=$'\n' touch .nfpcaps for i in `find /home/capture -type f -name 'cap.dmp*' -printf "%T@ %p\n" | sort -n | head -n -1 | sed -E -e 's/^[0-9\.]+ //' ` ; do echo ">> $i" | tee -a nfpcapd.log grep -q "$i$" .nfpcaps if [ $? -eq 0 ]; then echo "skipping" | tee -a nfpcapd.log else echo "$i" >> .nfpcaps nfpcapd -e 60,60 -r "$i" -l . 2>> nfpcapd.log fi done IFS=$OIFS

no comments | post comment
This is a collection of techical information, much of it learned the hard way. Consider it a lab book or a /info directory. I doubt much of it will be of use to anyone else.

Useful:


snipsnap.org | Copyright 2000-2002 Matthias L. Jugel and Stephan J. Schmidt