(2022-04-06)
Problem
I have a pile of pcap files that I want to generate netflow files for.
Solution
nfdump comes with a suitable utility.
$ for i in `ls $CAPS | sort -V ` ; do
echo ">> $i" | tee -a nfpcapd.log
nfpcapd -e 60,60 -r $CAPS/$i -l . 2>> nfpcapd.log
done
This took about 15 minutes to chew through 200G of pcap files on my tap machine.
Something slightly more robust. This script:
- can handle "cap.dmp###" files that are in different directories and sorts/processes them by age
- can be re-run on the same dataset and won't process cap.dmp files already processed
- excludes the last file in the list of cap.dmp files on the theory that there is probably a tcpdump currently writing to it
#!/bin/bashOIFS="$IFS"
IFS=$'\n'
touch .nfpcaps
for i in `find /home/capture -type f -name 'cap.dmp*' -printf "%T@ %p\n" | sort -n | head -n -1 | sed -E -e 's/^[0-9\.]+ //' ` ; do
echo ">> $i" | tee -a nfpcapd.log
grep -q "$i$" .nfpcaps
if [ $? -eq 0 ]; then
echo "skipping" | tee -a nfpcapd.log
else
echo "$i" >> .nfpcaps
nfpcapd -e 60,60 -r "$i" -l . 2>> nfpcapd.log
fi
done
IFS=$OIFS